"It Works, I Think" Is Not a Test: How to Actually Run Evals on Your AI Agents

Ask a team how their AI agent is doing a month after they shipped it, and the honest answer is usually three words: it works, I think. No measurement system, no regression tests, no idea whether yesterday's prompt change quietly broke today's behaviour. So they ship, they pray, and they wait for a customer to find the bug they could not.

Evals are the way out of that. Not a perfect eval system, which no one ever finishes building, but any eval system, running today. This is the practical version, drawn from how our solutions team at Beam sets it up with enterprise customers.

Why teams skip evals, and what it actually costs

Almost every team that skips evals gives one of four reasons, and each one translates to something less comfortable.

"We'll add evals later" means never, because the eval debt compounds with every prompt change. "Our use case is too subjective" means success was never defined; subjectivity is a missing rubric, not a fact of life. "Evals are for research teams" means the team does not know how to start small. And "we'll just watch production" means the customer becomes the QA team, which is how you lose them quietly.

The cost of waiting is specific. Without evals, a regression can go undetected for four to six weeks before someone, usually a customer, notices. Engineers respond by freezing the prompt, so the agent stops improving out of fear of breaking what works. And internal trust erodes before customers ever complain, which is the damage that is hardest to undo.

Step one: build a golden dataset in a day

A golden dataset is a small set of inputs paired with the correct output. It is the agent's report card, and it is the foundation everything else depends on.

You can build the first version in a day. Pull twenty real inputs from production or pilot data. For each one, write the ideal output yourself. Cover roughly 60% happy path, 30% edge cases, and 10% adversarial inputs. Store them as input-to-expected-output pairs, and run the agent against the whole set on every prompt change.

There is a useful test hiding in this. If you cannot write the ideal output for an example, the agent cannot either. That is a signal to cut the example or fix the specification, not to move on.

The set grows with the agent. Twenty examples is an MVP that catches obvious breaks. Fifty to a hundred gives you meaningful surface area. Two hundred or more, refreshed quarterly, is compliance-ready. A twenty-example golden set you actually run beats a perfect benchmark you will never build.

Step two: turn every bug into a regression test

Every prompt change can break something that already worked. Regression tests are the safety net, and the rule for building them is simple: every fixed bug becomes a regression test case, and every customer complaint becomes one too.

Run the full set automatically before any prompt change merges, and decide from the result. Ship if the pass rate meets target and there are zero regressions. Hold if any high-impact case regressed. Investigate if the pass rate dropped more than a couple of points.

The payoff is cumulative. Three months in, you have a hundred or more tests locking in known-good behaviour. Six months in, your agent stops getting worse every time it gets better. That is how a good agent stays good.

Step three: read the trace like a confession

When an eval fails, the trace tells you why, if you read it properly. There are four places to look, in order.

Start with the input: was the data clean, complete, and in the expected format? Most agent bugs are actually input bugs. Then the reasoning: what did the agent decide to do, and why, watching for hesitation, repeats, and contradictions. Then the tool calls: which tools fired, with what arguments, and what came back, watching for malformed arguments and silent retries. Finally the output: how it differed from the golden answer, whether through hallucinated fields, wrong format, or missing context.

Across thousands of traces, the failures cluster into five patterns, ranked here by how much damage they do.

A trace is a confession. Read it like one.

Step four: match human oversight to risk, not comfort

Human-in-the-loop is not all-or-nothing, and it should not be set by how nervous the team feels. It should be set by confidence and impact together.

Start with more oversight than you think you need, and relax it only as your eval data proves the agent is reliable. Never reduce oversight without eval data to back it up.

The loop that makes every agent better

Put the pieces together and you get a cycle every production agent should run: measure, diagnose, fix, verify.

Measure by running the agent against the golden set and regression tests, producing one accuracy number per category. Diagnose by reading the trace for every failure and categorising the failure mode, because roughly 80% of failures cluster into three to five buckets. Fix the highest-frequency bucket first, and change one thing at a time, whether a prompt, a tool, or a fallback. Verify by re-running the full eval set to confirm the fix worked and nothing else broke, then promote.

Run it weekly during active development, monthly for stable agents, and immediately after any production incident.

The metrics worth tracking

Accuracy is not one number. These are the metrics that tell you whether an agent is fit for production, and where to aim.

Track these over time, not as one-off snapshots. The direction of the line matters more than any single reading.

How Beam makes evals the default

The reason most teams skip evals is friction, so the fix is to remove it. In Beam, evaluation is built into the platform rather than bolted on afterward. Teams upload a golden dataset from a CSV or straight from production traces, get regression tests auto-generated from past failures, and see a trace viewer with a pre and post diff on every prompt change. Human-in-the-loop dashboards ship with the risk-tier presets above already baked in, and every change shows its accuracy delta before it goes live.

That is the difference between evals as a project you never start and evals as the default path. The teams whose AI agents keep improving are not the ones with the most talent. They are the ones who measure.

Common questions about AI agent evals

What is an AI agent eval?

An eval is a repeatable test of an agent's behaviour against known-correct answers. In practice it combines a golden dataset (inputs paired with ideal outputs), regression tests (locked-in cases from past bugs), and trace analysis (reading the agent's step-by-step logs) to produce an accuracy score you can track over time.

How many examples do I need to start evaluating an agent?

Twenty. Pull twenty real inputs, write the ideal output for each, and cover about 60% happy path, 30% edge cases, and 10% adversarial. That MVP catches obvious breaks. Grow to 50 to 100 for meaningful coverage and 200 or more, refreshed quarterly, for compliance-grade evaluation.

How do I stop prompt changes from breaking my agent?

Turn every fixed bug and every customer complaint into a regression test, then run the full suite automatically before any prompt change merges. Ship only if the pass rate meets target with zero regressions on high-impact cases. This is what keeps an agent from getting worse each time you try to make it better.

When should an AI agent require human review?

Match oversight to confidence and impact, not to comfort. High-confidence, low-impact actions can auto-approve; low-confidence or high-impact actions need a human; and any genuinely new pattern should require review for its first several runs. Relax oversight only as eval data proves the agent is reliable.