Build a support copilot that knows when to stop

The customer brief

This original Role Atlas exercise simulates a small software company whose support team must search multiple policy documents. Build a tool that drafts an answer and shows the evidence an agent needs to review it. Do not send replies automatically.

Create 15 fictional help articles and a synthetic account table. Include different account tiers, an outdated policy, and questions with no answer in the material. Never use confidential employer documents or real customer records in a public demo.

Build the narrow version first

Start with keyword search as your baseline. Add retrieval and answer generation only after the basic interface works. Each answer should link to its supporting passage and document date.

• Filter accessible documents before retrieval using the signed-in test user's permissions.
• When evidence is missing or conflicting, show "Needs human review" rather than guessing.
• Store feedback and timing with a request ID, without exposing document contents in public logs.
• Keep API credentials on the server and set a request limit. A mock response mode is enough to demonstrate the workflow.

Define what passing means

Reserve 30 questions you did not use while tuning. Score answer support, appropriate abstention, and unauthorized-data exposure separately. Report response time and API cost when applicable. Run more than once if outputs vary.

Include a screen recording of one correct answer, one missing-answer case, and one permissions boundary. An evaluation report is more credible than claiming the system is production-ready. Anthropic's evaluation guide is optional background for designing your rubric.

Source / Anthropic: Demystifying evals for AI agents (January 2026) (opens in a new tab)