How to Discover the AI Failures Your Evals Don’t Yet Measure

A prospect told an AI leasing assistant that the available option was out of budget, then thanked it for its time. The assistant answered courteously: reach out if circumstances change, and have a great day.

Nothing in that reply was rude. Nothing was obviously untrue. A generic evaluator might even score it well for tone.

But a person reviewing the full exchange could see a different failure. The assistant’s job was not simply to close conversations pleasantly; it was to help match prospects to properties. Faced with a budget objection, it should have explored cheaper units or alternatives at other properties owned by the same company. Instead, it ended the conversation. The source does not say whether a sale was lost. It does show why the exchange was a sales failure worth measuring: the assistant gave up where the product objective called for objection handling.

That reversal answers a question AI product teams often ask too late: how do you find the failures your current evals cannot measure? You begin with human review of real, reconstructable interactions—not with a dashboard of generic metrics. Only after people have seen what “good” means in the product’s actual context can automation help apply that definition at scale.

A polite answer can still be the wrong answer

The leasing exchange is useful precisely because it does not look like a conventional model failure. There is no visible contradiction, nonsensical text, or broken tool call. The response is coherent and socially appropriate. Its failure emerges only when the reviewer brings in the product’s purpose: a leasing assistant should continue helping a prospect whose first option does not fit.

That gap is criteria drift. In plain language, criteria drift is what happens when reviewing real examples changes your understanding of what should count as good or bad. Before seeing this trace, a team might evaluate factual accuracy, helpfulness, or politeness. After seeing it, the team has a new, concrete criterion: when a prospect objects on budget, does the assistant investigate relevant alternatives rather than simply sign off?

An automated evaluator could check for objection handling once that criterion exists. It could not reliably tell the team, in advance, that objection handling was the missing definition of quality. That discovery required a person to connect the trace to the product goal.

The implication is not that automation is weak. It is that automation has a sequence. First discover the standards. Then make them repeatable.

Start with evidence a reviewer can reconstruct

A reviewer cannot discover a hidden failure from a final answer alone. They need the same ingredients that shaped it: the user’s input, the system prompt, retrieved context, tool calls and results, intermediate model calls, and the final user-facing output.

Think of a trace as a complete record of one session, not a transcript fragment. In the leasing case, the budget objection matters differently when the reviewer can see what inventory or property information the system had available, what it was instructed to do, and how it arrived at its final reply. A complete trace lets someone answer two basic questions: what happened, and was it good?

This also changes what teams should capture. The goal is not logging for its own sake. It is preserving enough context for a reviewer to distinguish a bad answer from a bad outcome, an unavailable option from an ignored one, or a genuine limitation from a missed opportunity.

Sample for contrast, not convenience

Reading the first ten traces in a folder is rarely a useful portrait of a product. It may overrepresent one user type, one request pattern, or one unusually noisy day. The source workflow instead creates a deliberately diverse initial sample by mixing representatives from trace clusters with random picks.

That is a practical safeguard: cluster representatives help expose different kinds of interactions, while random examples keep the review from becoming too tidy or too dependent on the system’s grouping. The point is not to claim that a small sample is statistically complete. It is to make limited human attention more likely to encounter meaningful variation.

The workflow adds a second safeguard: reviewers examine 10 traces before the system begins suggesting errors. This pause matters because early automation can make its categories feel more authoritative than they are. Reviewing those traces first gives people raw material to notice failures in their own terms—the assistant gave up instead of offering alternatives, for example—before suggested labels begin shaping the conversation.

A useful early annotation is specific and user-facing. “Bad response” gives neither a colleague nor an automated reviewer anything to work with. “The assistant ended the conversation after a budget objection rather than offering available alternatives” identifies an observable failure. It says what went wrong without pretending to know the internal root cause.

Let people calibrate the product’s definition of good

Human review is not an ornamental approval step. It is where a team turns individual surprises into a defensible criterion.

For the leasing assistant, a reviewer should not turn one awkward interaction immediately into a sweeping rule. Inspect several relevant traces. Are budget objections common? Were alternatives available? What behavior should count as an acceptable attempt to help? The answers shape a criterion that is specific enough to evaluate but faithful to the product’s purpose.

This is calibration: people align on what they are looking for before asking a system to look for more of it. It also guards against confusing the first visible symptom with the important error. At the outset, reviewers can focus on the first or upstream user-facing failure in a trace, where an early mistake may explain what follows.

The result need not be a perfect taxonomy. The immediate aim is to find actionable patterns in the logs. As annotations accumulate, they can be grouped into failure modes and weighed as product issues. But the category comes after the encounters with evidence, not before them.

Automate what the trace can show—and keep proposals reviewable

Once people have supplied examples and criteria, automation becomes useful in a more disciplined way. It can help with sampling, bookkeeping, trace annotation, and checks for failures that are directly visible in the trace. It can also propose additional annotations for a human to accept or reject.

That last distinction is important. A proposed annotation is not a verdict. Early suggestions can be wrong, including cases where a good response is flagged as a failure. Reviewer disagreement is valuable feedback: it shows where the automated interpretation does not yet match the product’s standard.

The boundary is not mysterious: automation is strongest where the evidence is explicit and contained in the trace. It can spot an answer that conflicts with tool output, and it can apply a criterion people have already defined. It is weaker where quality depends on context outside the trace or on a product-specific judgment about what the interaction was supposed to accomplish.

Build the eval after the discovery

The order is the method:

1. Capture reconstructable, complete traces. Preserve the interaction and the system activity needed to judge it.
2. Draw a deliberately diverse sample. Mix varied representatives with random traces rather than reviewing whatever happens to be first.
3. Have people review and calibrate. Write specific notes about observable user-facing failures; inspect multiple instances before treating a pattern as understood.
4. Use automation to make review scalable. Let it find obvious in-trace contradictions and propose annotations, while people accept, reject, and refine those proposals.

This approach does not promise that every failure will become measurable immediately. Nor should it. Some of the most consequential failures become visible only when someone sees an apparently successful interaction against the actual purpose of the product.

The leasing assistant did not fail because it lacked manners. It failed because manners stood in for helping. Once a human reviewer recognized that difference, objection handling could become a criterion, then a reviewable automated check. That is the durable role of evals: not to automate whatever is easiest to count, but to preserve the hard-won judgment about what the product is really for.