If you only adopt one practice from this course, make it evals. Without them every change is a guess, and you will not notice regressions until users do.
Watch
Hamel Husain at Arize Observe 2026, arguing that as agents become more capable, evaluation becomes the hard problem.
What an eval is
An eval is a set of inputs, a way to run your agent on them, and a way to score the outputs. That is all. It can start as a spreadsheet and 30 rows.
Build the set from reality
- Start with real traffic or realistic tasks, not invented happy paths. Read 50 actual conversations or traces before writing a single test.
- Cover failure modes you have seen: wrong tool, wrong argument, ignored policy, infinite loop, refusal when it should help.
- Add every production bug as a new case. The set grows from incidents.
- Keep a held-out slice you do not tune against.
Three kinds of scoring
- Code checks. Did it call the right tool with valid arguments? Is the output valid JSON? Did the final database state match the expected one (tau-bench grades this way)? Cheap, deterministic, preferred.
- Reference comparison. Compare against a known answer or key facts.
- LLM-as-judge. A model grades subjective qualities such as helpfulness or policy adherence. Useful, but you must validate the judge: have a human label a sample, measure agreement, and refine the rubric until they match.
Grade the trajectory, not only the answer
For agents, the path matters. Record the tool calls and check for forbidden actions, unnecessary steps and cost. An answer that is right because the agent got lucky is a future incident.
Handle non-determinism
Run each case several times. Report the pass rate and, where consistency matters, the all-k-trials rate from lesson 1. Track variance as well as the mean.
Run a small, fast eval set on every change to prompts, tools or model versions, and a larger one before release. Treat a drop like a failing unit test.
Resources
A practical case for evals and how to build them in layers.
How to build and validate a judge you can trust.
Which metrics to use for classification, summarization, extraction and more.
Guidance on writing measurable success criteria and graders.
An open-source tool for running evals and red-team tests from config files.
Check yourself
- Read 30 real or realistic interactions. Write 30 test cases with expected behavior.
- Score them with code checks where possible and a judge where not. Hand-label 15 to measure judge agreement.
- Run the set five times. Report pass rate, all-five-pass rate and cost per run.