How to Evaluate an AI Agent Before You Trust It
You cannot ship what you cannot measure, and “it seemed good when I tried it” is not measurement. A practical method for testing non-deterministic systems.
Traditional software testing rests on an assumption that does not hold here: the same input produces the same output. Take that away and most of your testing instincts stop working. Assertions become flaky. Snapshots become noise. The whole apparatus assumes determinism you no longer have.
The response to this is not to give up on testing. It is to test differently — measuring distributions instead of asserting equality.
Start with the golden set
Before any agent code, assemble real examples with known-correct outcomes. This is the foundation and there is no substitute for it.
Where the examples come from. Historical records of the work being automated. Cases your team already handled where you know what the right answer was. If the process is entirely new, have a domain expert work through fifty cases manually — that is a week of someone’s time and it is the highest-leverage week in the project.
How many. Fifty is enough to catch obvious regressions. Two hundred gives you reasonable confidence. Beyond about five hundred you get diminishing returns unless you are covering genuinely distinct categories.
What goes in it. Deliberately not just typical cases. You want the boring middle, but you also want the ambiguous ones, the malformed inputs, the cases where the correct answer is “I cannot determine this,” and the adversarial ones if your system is externally facing.
Choose a grading method per task
Different outputs need different graders, and using the wrong one is a common way to get numbers that do not mean anything.
Exact match works for classification, routing, and structured extraction where there is exactly one right answer. Cheap, unambiguous, and the best option whenever it applies.
Structural validation works when the shape matters more than the content: does the output parse, does it satisfy the schema, are required fields present, are the values in range. Fast, deterministic, and it catches a surprising share of real failures.
Rubric grading by a model works for open-ended outputs — summaries, drafted messages, explanations. You define specific criteria and have a model score against them. Correlates reasonably well with human judgement when the rubric is concrete. It degrades badly when the rubric is vague, so criteria like “is it good” are worse than useless.
Human review remains necessary for a sampled subset, permanently. It is the only thing that catches problems your rubric did not anticipate, which is precisely the category you most need to know about.
Outcome measurement is the strongest signal and the one people skip because it is slow: did the downstream thing actually work? Did the extracted data reconcile? Did the message get a reply? Did the human accept the recommendation without editing it? When you can measure this, it dominates every proxy.
Measure the things that break projects
Accuracy is the obvious metric and it is rarely the one that kills you.
Failure mode distribution matters more than the aggregate rate. Ten percent wrong is a very different problem depending on whether the errors are randomly scattered or all concentrated in one input category — the second is a bug you can fix, the first is a capability limit.
Calibration — does the agent’s expressed confidence track its actual accuracy? A system that is right eighty percent of the time and knows which eighty percent is enormously more useful than one that is right ninety percent of the time with uniform confidence, because the first one can route its uncertain cases to a human.
Cost and latency percentiles. The p50 is comfortable and the p99 is what people complain about. Agent loops have long tails by nature.
Consistency across runs. Run the same input several times. High variance on identical inputs means the system is fragile in ways that a single-pass eval will not show you.
Wire it into the workflow
An eval suite that runs when someone remembers is an eval suite that does not run.
- On every change to prompts, tools, models, or retrieval — automatically, in CI.
- Blocking on regression. If accuracy drops below the current baseline, the change does not merge. This is the discipline that makes the whole thing worth having.
- Reported as a diff, not an absolute. “Improved on 12 cases, regressed on 3, here are the 3” is actionable. “Score: 0.87” is not.
- Cheap enough to run constantly. If a full eval costs forty dollars and takes an hour, it will get skipped. Keep a fast subset for every commit and run the full set nightly.
Shadow mode before switchover
The last step before an agent takes real actions: run it alongside the existing process without letting it act.
Both the agent and the current process handle the same inputs. Only the current process’s output is used. You compare them. This does two things — it gives you a genuine production-distribution measurement rather than an eval-set proxy, and it lets the people who will rely on the system watch it work before they have to trust it.
The disagreements are the valuable output. Every case where the agent and the human differ is either a bug or a case where your definition of correct was less settled than you thought. Both are worth knowing before you cut over.
Run it long enough to cover a full cycle of whatever seasonality your work has. Then switch over on the numbers, not on a launch date.
What good enough looks like
There is no universal threshold. It depends entirely on the cost asymmetry.
If a wrong answer is expensive and a delay is cheap — legal, medical, financial, anything regulated — you want high confidence thresholds and generous escalation to humans. Accept lower automation rates in exchange for very few wrong actions.
If a wrong answer is cheap and throughput is the constraint — internal triage, first-pass categorisation, draft generation — you can run at much lower confidence, because the correction cost is a person spending ten seconds.
The question is never “is the agent accurate enough.” It is “is the agent more accurate than the alternative, at a cost that makes sense, with the errors falling in places we can absorb.” Frame it that way and the threshold usually becomes obvious.
Evaluation is the first thing we build on every agent engagement. See our approach, or read about why agents stall in production.