17 min read

How to Write Eval Criteria for an LLM Judge

By submitting, you consent to our use of your data. Privacy Policy.

Category

AI Agents

Share the article

Why the criteria, not the judge model, decide how many agent errors get caught; what a vague criterion costs at volume; what a working one contains; and how to test it against your own reviewers.

📌 Key takeaways

  • In a published test, the same GPT-4 judge passed a wrong answer in 14 of 20 judgments with a plain prompt and in 3 of 20 once it had the answer key. Only the instructions changed (Zheng et al., NeurIPS 2023).

  • On one step running 10,000 tasks a month, our model puts the gap between those two criteria at 275 errors a month that nobody catches, worth about $69,000 at $250 an error.

  • A panel of three judges multiplies the criteria rather than replacing them: three weak judges voting catch 22% of errors, down from 30% for one; three well-briefed ones reach 94%, up from 85%.

  • A criterion that works is one check, on named fields, with a written 5 and 1, an order of reasoning and a weight. Ten good criteria combined as a plain average can still pass a file with a wrong figure.

  • Validate the judge against your best reviewers on 100 to 200 real cases. Twenty cases cannot tell an 85% judge from a 64% one.

Most teams I talk to now have an AI judge in their agent workflows. A model reads each output, scores it against criteria, and decides whether the work moves on, reruns or goes to a person. That check is what lets an agent run claims, invoices or onboarding at volume.

When the check lets errors through, the first question I hear is which judge model to use. A bigger one? Three of them, so one can catch the others? The evidence points somewhere else. In a published test, the same model went from missing 14 of 20 wrong answers to missing 3, and the only change was the instructions it was given. So instead of comparing judge models, we worked out what makes a judge accurate and where the week of effort should go. My answer: the criteria decide how accurate a judge is, not the judge model. A bigger model or a panel multiplies whatever the criteria give it. In the best-known test, well-briefed judging matched human experts. Vague criteria leave most errors in place, whatever the model.

This piece covers the criteria an LLM judge scores against inside an agent workflow: the evidence that instructions matter more than the model, what weak criteria cost at volume, what a working criterion contains, and how to validate it against your own reviewers. It does not cover which judge method to use (that is in Moving Beyond LLM-as-a-Judge) or where to place the checks in a workflow (that is in How Quality Gates Make AI Agents Reliable in Production). Every number is either published, with a source, or modelled from assumptions stated in the text. Figures as of 15 November 2026.

The same judge, three sets of instructions. With a plain prompt the judge passed 14 of 20 wrong answers; told to solve the problem first, 6 of 20; given the correct answer to compare against, 3 of 20. The model never changed.

Same model, same questions. The misses fell by four fifths when the instructions changed (Zheng et al., 2023, Table 4).

Does the judge model or the criteria decide how accurate an AI judge is?

The criteria do. Hold the model fixed, change only what it is told, and its misses fall by four fifths. By criteria I mean the written checks the judge scores each output against.

The cleanest evidence comes from Zheng and colleagues at NeurIPS 2023. They had GPT-4 judge answers to 10 math questions, each pair in both orders, and counted how often it called a wrong answer correct (Zheng et al., 2023, Table 4):

What the judge was given

Wrong answers it passed

Share of misses

A plain prompt: "which answer is better?"

14 of 20

70%

An instruction to solve the problem first

6 of 20

30%

The correct answer to compare against

3 of 20

15%

The test is small, 20 judgments, so read the direction rather than the decimals. Each step added a piece of what a good criterion contains: an order of reasoning, then an answer key. The model never changed.

The ceiling is a human expert, not perfection. In the same paper GPT-4 agreed with expert reviewers on 85% of judgments in a chat benchmark, ties excluded. Two experts agreed with each other on 81%. Liu and colleagues found the same pattern with G-Eval, a published scoring method our judge builds on: writing out the evaluation steps before scoring raised the judge's agreement with human ratings (Liu et al., EMNLP 2023).

Model size still matters, up to a point. Smaller models can judge well when they are built for it: Prometheus 2, an open model trained to score against criteria the user writes, tracked human and GPT-4 ratings more closely than any other open judge its authors tested (Kim et al., EMNLP 2024). Off the shelf it is harder. On a different test of thirteen judges, Thakur and colleagues found only the largest came close to how often humans agreed, and none reached it (Thakur et al., 2025). A small judge works when it is trained or tightly briefed for the job, not by default.

🧭 How we see it at Beam

We run one judge model per workflow step and put the effort into the criteria. Judging is easier than drafting, so the judge can be smaller than the model that wrote the output, as long as the criteria tell it exactly what to compare.

If a well-briefed judge reaches expert level, the next question is what the gap between a vague and a well-briefed one costs you.

What does a weak criterion cost at production volume?

In our model, the gap between a plain prompt and a criterion with an answer key is 275 errors a month that nobody catches, on a single step. The model takes one step that is right 95% of the time, 10,000 tasks a month, and a judge whose catch rate is the published one from the test above:

The judge checks against

Catches

Errors passed on per 10,000 tasks

Cost at $30 each

Cost at $250 each

No check

0%

500

$15,000

$125,000

A plain prompt

30%

350

$10,500

$87,500

An order of reasoning

70%

150

$4,500

$37,500

The answer key from the source

85%

75

$2,250

$18,750

Modelled: 95% step accuracy, catch rates from the Zheng test, every caught error fixed before the next step. $30 is half an hour of rework; $250 is a customer-facing correction with a supervisor's sign-off.

Errors passed on per 10,000 tasks by what the judge checks against. No check: 500. A plain prompt: 350. An order of reasoning: 150. The answer key from the source: 75.

A plain prompt leaves 350 of 500 errors in place. The answer key leaves 75 (modelled).

A plain prompt leaves 70% of the errors in place, so it costs nearly as much as no check while looking like a control. The model leaves out false alarms, correct work the judge rejects, because the test did not measure them; I come back to them below.

The judge's own bill does not change the decision. At OpenRouter's measured price for an LLM judge, $0.114 per 1,000 judgments (OpenRouter, Sep 2026), ten criteria on 10,000 tasks cost about $11 a month. Even at 100 times that, for long files and a larger model, it is $1,140.

The obvious fix is a panel of judges. It works only when each judge is already good. With three independent judges voting, a 30% catch rate falls to 22%, because a majority of weak judges outvotes the one that caught the error. An 85% judge rises to 94%. A panel multiplies the criteria; it does not replace them. It also costs three times as much, and three copies of one model reading one vague prompt tend to make the same mistake, so the real gain is smaller.

A panel of three judges voting by majority. Three judges with a 30 percent catch rate catch 22 percent together. Three judges with an 85 percent catch rate catch 94 percent together. The panel multiplies what the criteria give it.

A panel amplifies the criteria. Weak judges outvote the one that caught the error (modelled, independent votes).

🧭 How we see it at Beam

Beam runs at 98% accuracy across more than 10 million agent tasks. At that volume the other 2% is still 20 tasks in every 1,000, and the criteria decide how many of them the judge finds.

If the criteria are where the money is, the next question is what separates one that catches 85% from one that catches 30%.

What does a criterion that works look like?

A criterion that works is one check, on named fields, with a written pass and fail, an order of reasoning and a weight. Leave any of those out and the judge fills the gap with its own standard, including the habits Zheng and colleagues found: favouring the answer shown first, the longer answer and its own writing. Six parts:

  1. One check. "The figure matches and the tone is right" hides which one failed.

  2. A question a reviewer would ask. "Does the cited limit equal the requested limit?" is a question. "Ensure quality" is not.

  3. The fields to compare, never "the whole output".

  4. What a 5 is and what a 1 is, as facts someone can point at. On a qualitative check, write a 3 as well, because most real outputs land in the middle.

  5. The order of reasoning, plus one pass and one fail example. Read the source value, read the output value, compare, then score.

  6. A weight. Some failures break every step after them.

Checks with an answer key are binary: a figure matches the source or it does not, so it scores 5 or 1. A close figure is a wrong figure, because the next step will use it.

The same insurance triage step written two ways. Left, what most people write: "Check that the underwriting decision is correct", with no field named, no 1 or 5 defined, no order of reasoning, and the whole output scored. Right, a criterion that works: check requested limit match, compares submission.limit_requested with the output, 5 is the limit the broker asked for, 1 is any other figure including the cap, reason is read source then output then compare then score, fail example is €5M requested and €3M cited, weight highest because the next checks use it.

The vague version lets the judge invent the standard as it scores. The working version names one check, two fields and a written pass and fail.

Here is what that looks like in insurance. Most people write "Check that the underwriting decision is correct" or "Score the whole output from 1 to 5". Neither names a field, defines a 1 or says how to reason. Take one step instead: triage of a commercial submission. The agent reads the broker's submission and the matching row of the insurer's appetite grid, one row per territory and occupancy, then writes a decision. Each rule on the row becomes a criterion. The field names and figures are a teaching example, not an insurer's rules.

#

Check

Compares

5

1

1

Employee count

submission vs output

Same integer

Any difference (5,000 cited as 500)

2

Sum insured

submission vs output

Same after stripping symbols

A shifted digit

3

Requested limit

submission vs output

The limit the broker asked for

The cap cited instead (€3M for €5M asked)

4

Named insured

submission vs output

Same legal name

Parent, broker or trading name

5

Territory

risk location vs output

The risk location

The broker's office address

6

Occupancy

dropdown code vs output

The dropdown code

A word from the narrative

7

Appetite row

territory and code vs row used

That exact row

A neighbouring row or a blend

8

Headcount band

output vs row maximum

Within the band

Outside it; scored only if 1 and 7 pass

9

Limit cap

output vs row cap

Within the cap

Over it; scored only if 3 and 7 pass

10

Decision and reason

decision vs grid action

Right action, rule and both figures named

Quoted despite a hard stop

Check 10 is the one qualitative check, so it has a 3: the right action, but the reason drops a figure. The three number checks carry the highest weight, because a wrong figure makes every appetite check after it meaningless.

Weights alone do not finish the job. A step's ten criterion scores usually combine into one score with a pass mark, and if that score is an average, a wrong figure can still pass. Assume a weighted average with 1 as 0% and 5 as 100%, and a pass mark of 80:

The file

Equal weights

Number checks weighted 3x

All ten checks pass

100%

100%

Wrong sum insured (check 2 fails)

90%, passes

81%, passes

Limit over the cap and quoted anyway (9 and 10 fail)

80%, passes

88%, passes

Modelled, with an assumed way of combining scores.

Ten criteria combined as an average. A file with a wrong sum insured scores 90 percent with equal weights and 81 percent with number checks weighted three times, and passes an 80 pass mark either way. A file quoted over the cap scores 80 or 88 percent and passes too.

Ten good criteria and an average make a vague criterion again. A hard stop has to fail the step on its own (modelled).

Weighting the numbers pulls the wrong-figure file down to 81%, still a pass, and lifts the over-cap file to 88%, because the decision checks now count for less. A hard stop has to fail the step on its own: make it a must-pass check, or set the pass mark per criterion. The full threshold arithmetic, and the setup we recommend for appetite rules, is in Turning Underwriting Rules into Eval Criteria for AI Agents.

Written this way, a criterion looks right. Whether it is right is a separate question, and you answer it with your own reviewers.

How do you know your criteria are good enough to trust?

You check the judge against your best reviewers on 100 to 200 real cases before you let it gate anything. Agreement with your experts is the number that matters, and it needs enough cases to mean something.

Here is how far an 85% agreement result can move by chance alone, depending on how many cases you label (modelled, 95% range):

Cases your reviewers label

85% agreement could really be

20

64% to 95%

50

73% to 92%

100

77% to 91%

200

79% to 89%

400

81% to 88%

How far a measured 85 percent agreement can move by chance, by number of labelled cases. At 20 cases, 64 to 95 percent. At 100, 77 to 91 percent. At 200, 79 to 89 percent. At 400, 81 to 88 percent.

Twenty cases cannot tell an expert-level judge from a weak one. At 100 to 200 the range is narrow enough to decide on (modelled, 95% interval).

Twenty cases cannot tell an expert-level judge from a weak one. At 100 to 200 the range is narrow enough to decide on, for a few days of a reviewer's time.

Then read the result four ways:

  • Read the disagreements, not just the rate. Most point at a criterion that left something open, such as a 3 nobody wrote down.

  • Count false alarms separately: correct work the judge failed. Each one costs a rerun or a person's time, and a judge that fails good work is the one your team stops trusting.

  • Compare it with how often your reviewers agree with each other. If they agree 81% of the time, a judge at 85% is done; pushing further tunes it to one reviewer's habits.

  • Keep labelling a small sample every month, and again whenever the judge's model is updated. Documents and rules change, and a judge that agreed in March can drift by June.

How we run criteria on a step at Beam

Each criterion on a step gets its own score and a written reason, and a low score reruns the step with that reason. Solution engineers write the criteria with the process owner, from that workflow's documents, not from one company-wide rubric.

🧭 How we see it at Beam

Our judge is a variant of G-Eval that sees the task, the input, the output and the criteria, and scores each criterion from 1 to 5 with a written reason. In our standard setup, a score below 80 reruns the step on the judge's feedback, up to three times, and if more than 30% of tasks need a rerun, our docs say to fix the step's instructions instead (Beam docs, Evaluation Framework). A step still failing after the last rerun shows a "Validation Failure" in the task list, and on steps that send, pay or file we recommend Consent required, so a person approves first.

A stable rule goes inside the criteria, so the rerun and the trail of criterion, score and reason come with it. That trail is what you show an auditor: for each task, which rules were checked and why each passed or failed. Failed cases feed Optimize Outputs, where you "Mark what went wrong in failed outputs, and AI uses examples to rewrite prompts with better context" (Beam docs). Today a person improves the criteria; criteria that learn from feedback on their own are on our roadmap.

Where should you start?

Start with the one step where a wrong figure costs the most, and spend a week writing its criteria with the person who owns the process. Pick one judge model that is good enough and leave it.

How do you start small?

  1. Pick one workflow you run at volume and one step in it, usually extraction.

  2. Sit with the process owner and list the rules they check by hand. One rule becomes one criterion.

  3. For each, name the two fields compared, write the 5 and the 1, and add one pass and one fail example from real files.

  4. Weight the checks that break the next step, and make hard stops fail the step on their own.

  5. Have two reviewers label 100 to 200 past cases. Compare the judge with them, count false alarms, and rewrite the criteria behind each disagreement.

  6. Turn on reruns once the judge agrees with your reviewers about as often as they agree with each other. Keep labelling a small sample every month.

What should you ask your team or vendor?

  • Which fields does each criterion compare, and where is its 1 written down?

  • How do criterion scores combine, and can a single check fail the step on its own?

  • How often does the judge agree with our reviewers, on how many cases, and how often does it fail correct work?

  • When a check fails, what does the rerun receive: a score, or the criterion and its reason?

  • Who rewrites the criteria when the business rules change, and how fast?

  • What does the judge cost per task, and what would a second judge add?

What does this look like on the submission triage step?

Take the triage step above at an illustrative 10,000 submissions a month, right 95% of the time. A plain "is the decision correct?" judge passes about 350 wrong decisions a month (modelled). The ten criteria, with the answer key taken from the submission and the appetite row, bring that to about 75. Writing them takes one person a week; labelling 150 past files takes two reviewers a few days. At $250 per wrong decision, the 275 fewer errors are worth about $69,000 a month, against a judge bill of $11 to $1,140. Reruns, reviewer time on flagged files and false alarms come out of that, and the gate model in our quality gates post prices them at cents a task. Making checks 9 and 10 fail the step on their own closes the hole the average leaves.

A judge agrees with your experts when your experts would recognise its criteria. Pick the judge model once per step, then put the week into the criteria, starting with the step where one wrong number costs the most.

FAQ

Does a bigger judge model catch more agent errors?

Only as far as the criteria let it. In the Zheng test the same model went from passing 14 of 20 wrong answers to 3 of 20 when it was given an order of reasoning and the answer key. A larger or trained judge raises the ceiling, but vague criteria leave most errors in place whatever the model.

Should I use a panel of LLM judges?

Not to fix weak criteria. Three independent judges voting take a 30% catch rate down to 22%, because the weak majority outvotes the one that caught the error, and an 85% judge up to 94%. A panel multiplies the criteria at three times the cost. Write the criteria first; add judges later if the ceiling matters.

How many cases do I need to validate an LLM judge?

One hundred to two hundred real cases labelled by two of your own reviewers. With 20 cases a measured 85% agreement could really be anywhere from 64% to 95%. With 200 the range is 79% to 89%, which is narrow enough to decide on. Keep labelling a small sample every month.

What does one good eval criterion contain?

One check, phrased as a question a reviewer would ask, on two named fields, with a written 5 and 1 (and a 3 on qualitative checks), an order of reasoning with one pass and one fail example, and a weight. Checks with an answer key score 5 or 1, since a close figure is still a wrong figure.

Figures as of 15 November 2026. Modelled values are labelled as such; vendor-reported figures are marked as vendor claims.

Start Today

Start building AI agents to automate processes

Join our platform and start building AI agents for various types of automations.

Start Today

Start building AI agents to automate processes

Join our platform and start building AI agents for various types of automations.