
By submitting, you consent to our use of your data. Privacy Policy.
Category
AI Agents
Share the article
Why a rulebook in a PDF or a prompt is not yet a control, how each appetite line becomes one criterion, why averaged scores let out-of-appetite files through, where the rules should live, and how Beam runs the gate.
📌 Key takeaways
A guideline in a PDF or a prompt is not a control, because nothing makes the agent apply it and nothing records whether it did. A control is a check that runs every time, a score, a written reason and a consequence.
Each appetite line becomes one criterion scored 5 or 1. On an illustrative file €2 million over the cap, five equally weighted rules averaged together score 84 and clear an 80 pass mark.
With four or more equally weighted rules, one failed rule can never pull a plain average below 80. Hard rules need a threshold of 100 or a must-pass flag; judgment checks keep the 80 and the middle of the scale.
Number match carries the highest weight, because a rerun that rewrites the client's €5 million as €3 million would otherwise pass every other check.
Keep the rules in the criteria unless they must be read live. In our model, fetching them live without rebuilding the retry sends 36 more files per 1,000 to an underwriter.
Most carriers I talk to already have the hard part of AI governance written down. Their underwriting guidelines say who they will write, where the company can sit, how large it can be and how much cover they will put out. Now they are putting agents on submission intake, and the question from model risk is always the same: how do we know the agent applied the guidelines?
The common answer is to turn the guidelines into eval criteria, which is right. Then most teams average those criteria into a single score and set a pass mark, and that can quietly break it. On an illustrative file I show below, a company asking for €2 million more cover than the carrier writes scores 84 out of 100. With a pass mark of 80, it goes through. So instead of a single score, we map every appetite rule to one criterion, scored as a pass or a fail, and every one of them has to pass on its own. Keep those criteria inside the platform unless the rules have to be read live. Set up that way, a rule becomes a control, because it can stop the task and the record says why.
This piece covers commercial submission intake: how appetite rules become criteria, how their scores should combine, where the rulebook should live, and how the gate runs on Beam. It does not cover how to write a criterion in general or how to validate a judge against your reviewers; both are in How to Write Eval Criteria for an LLM Judge. Every number is either published, with a source, or modelled from assumptions stated in the text. The field names and figures are a teaching example, not a carrier's book. Figures as of 15 November 2026.
Why is a rulebook in a PDF or a prompt not yet a control?
Because nothing makes the agent apply it, and nothing records whether it did. An agent can retrieve a guideline stored as a PDF and still ignore it. A guideline written into the agent's instructions depends on the model choosing to follow it. Neither leaves a record a reviewer can open.
A control has four parts: a check that runs every time, a score, a written reason, and a consequence when it fails. That is also what regulators now ask for. The NAIC's Model Bulletin on insurers' use of AI, adopted on 4 December 2023, expects insurers to document how AI systems are governed, tested and monitored, and the NAIC's implementation map showed 27 states and the District of Columbia had adopted it by November 2026 (NAIC Model Bulletin; adoption map). A sentence in a prompt documents none of that.
🧭 How we see it at Beam
On Beam the check is an evaluation on the step. After the underwriting step runs, a judge model scores its output against written criteria, on every execution. The timing sits in the workflow, not in the model's judgment.
So the next question is what those criteria should look like.
How does each appetite line become one criterion?
Each line of the rulebook becomes exactly one criterion, scored 5 if the file meets it and 1 if it does not. Take an illustrative file. A company in Dallas, Texas, with 5,000 employees asks for €5 million of cover. The carrier's appetite line for Dallas writes up to 5,000 employees and up to €3 million.
A human underwriter stops this file on one line: cover is €2 million over the cap. The agent has to stop it on the same line, against the same numbers, with a reason that names both.

One line of the rulebook, one criterion, one reason that names both figures (illustrative).
Three choices make this mapping work.
Pass or fail, nothing in between. A coverage cap has no meaningful middle, so we score appetite rules 5 or 1. Judgment checks are different. Whether a referral note is usable keeps the middle of the scale, and the criterion spells out what a 1, a 3 and a 5 look like.
Coverage and number match stay separate. They ask different questions. Coverage asks whether the requested limit sits inside appetite. Number match asks whether the agent kept the figures from the submission. On the Dallas file, coverage fails and number match passes, which tells the reviewer the client did ask for €5 million. If the client asked for €3 million and the agent read €5 million, both fail, and the reason says the output does not match the submission. That is the case a rerun exists to fix.
Checks sit where the work happens. Extraction checks (fields present, figures matching the source document) stay on the extraction step. Appetite rules stay on the underwriting step. A formatting miss and an out-of-appetite limit are different events, and the record should show them apart.
Solution engineers write these criteria for each workflow, together with the process owner, from the carrier's own guidelines. Start from your guideline rather than a generic insurance list, because one carrier's Dallas cap is not another's.
Can a judge model be trusted to score them? A pass or fail rule on a figure from the file is a narrow task, and a judge with the answer key in front of it is at its most reliable. Even so, test it on your own past files first. How to do that, and how many files it takes, is in How to Write Eval Criteria for an LLM Judge.
Getting the criteria right is half the job. How their scores combine is the other half, and it is where most setups go wrong.
Why can an averaged score let an out-of-appetite file through?
Because an average lets four passes outvote one fail. Most eval setups combine criteria scores into one number and rerun the step when it falls below a threshold. In our working session the example threshold was 80. Suppose the scores are combined as a plain weighted average and shown as a percentage of the maximum. That is my assumption, not a published Beam formula, so treat these results as modelled.
On the Dallas file, coverage scores 1 and the other four rules score 5:
Setup (modelled) | Score | At a threshold of 80 |
|---|---|---|
Five rules, equal weight | 84.0 | Passes |
Number match at weight 5, others at 3 | 85.9 | Passes |
Coverage and number match at 5, others at 3 | 78.9 | Stopped |
Same weights, but size fails instead of coverage | 87.4 | Passes |

In three of the four setups, a file the carrier would never write goes through (modelled).
In three of the four setups, a file the carrier would never write goes through. Raising coverage's weight saves the coverage rule, and then the size rule leaks instead.
The general rule is short. When one rule fails and the rest pass, the score is 100 × (1 minus 0.8 × that rule's share of the total weight). It only drops below 80 if the failed rule carries more than a quarter of the weight. At most three rules can each carry more than a quarter. So with four or more equally weighted rules, one failed rule never pulls the score below 80. If the platform first rescales scores so that a 1 counts as zero, a failed rule costs its full share of the weight, five equal rules land at exactly 80, and the limit moves to five rules. A real appetite book has more lines than that.

Four or more equally weighted rules, and one failure can never stop the file. Two failures out of eight land exactly on 80 (modelled).
Two wrong rules out of eight, equally weighted, score exactly 80. That still passes a "rerun below 80" setting.
The same maths explains why we put the highest weight, 5, on number match in insurance workflows. Imagine a rerun that rewrites the client's €5 million as €3 million. Coverage now scores 5 and number match scores 1. With number match at weight 3, like every other rule, the edited file scores 84 and passes. At weight 5 it scores 76.5 and is stopped. The heaviest weight goes on the check that keeps every other check honest. Underneath the weight, you want one more control: every retry scored against the original submission, never against a version of it the agent produced.

Number match is the check that keeps every other check honest. At weight 5 the rewritten file is stopped (modelled).
The fix is to stop asking one threshold to do two jobs. Put the hard rules on the underwriting step with a threshold of 100, so every rule must score 5. Judgment checks, like the quality of a referral note, belong on the step that drafts the note, with a threshold of 80 and a scale that uses the middle. Hard rules never average with soft ones, and each threshold means one thing. If your platform cannot set 100, review every flagged file on the appetite step and require a person's consent before quote or bind.

Hard rules at 100 on the underwriting step, judgment checks at 80 on the drafting step, consent before quote or bind.
With the gate set up this way, the last design choice is where the rules themselves are stored.
Where should the rulebook live: in the criteria or fetched live?
In the criteria, unless the rules must be read live from an outside system. The two options feel the same in a workshop and differ the first time a score comes back low.
Option A: rules in the criteria. When appetite changes, someone edits the criterion and the next run uses it. The step reruns on a low score, up to three times. The record is the eval itself: rule, figure, score, reason, attempt.
Option B: rules fetched live. Some carriers will not copy their book into another system, or it changes between runs. Then reading it has to be its own step, because a step can call an outside system and a criterion cannot. The check still runs, but the built-in rerun does not come with it. Rebuilding the retry takes about four extra steps in the workflow, whose only job is retrying, and the record shows them next to every decision.
To see what that difference is worth, I modelled 1,000 submissions, assuming 15% are out of appetite, the agent misreads 5% of the rest, the gate catches 90% of misreads and a rerun fixes 60% of those. None of these are Beam data.
Per 1,000 submissions (modelled) | A: rules in criteria | B: fetched live, no rerun rebuilt |
|---|---|---|
Sent to a person | 152 | 188 |
Misreads fixed by a rerun | 34 | 0 |
Errors that slipped through | 6.5 | 4.3 |

Fetching the rules live without rebuilding the retry sends 36 more files per 1,000 to an underwriter (modelled).
Option B sends 36 more files per 1,000 to an underwriter: about 9 hours at 15 minutes each, or roughly $550 at $60 an hour. Option A's reruns cost about $16 per 1,000 files, at an assumed 2 cents a rerun plus about 1 cent for the judge. Build the four retry steps and Option B gets close to A, at the cost of steps you build and maintain. One honest limit: Option A lets about two more errors per 1,000 slip through, because a rerun can come back wrong in a way the gate misses. That is the price of fixing 34 misreads without a person.
One result surprised me. 89% of Option A's reruns are spent on files that are out of appetite, where no rerun can help. They cost cents, so I would not optimise them away, and the record makes the reason visible. When coverage fails while number match passes, on every attempt, the reviewer knows the figure is real and the decline is right.
Option A has its own risk, which is stale rules. If the criteria are updated late after an appetite change, files are decided on the old rules in between. At 2,000 submissions a month (about 95 a working day), modelled:
Change affects | Updated same day | 1 day late | 5 days late | 10 days late |
|---|---|---|---|---|
5% of files | 0 | 5 | 24 | 48 |
10% of files | 0 | 10 | 48 | 95 |
25% of files | 0 | 24 | 119 | 238 |
So the decision rule is concrete: if your team can update criteria the day appetite changes, keep the rules in the criteria. If the book changes faster than that, or the carrier will not let it leave their system, fetch it live and accept the extra steps.
How does this run on Beam?
Each step carries its own criteria in plain language, a judge model, a pass mark and up to three reruns. The record of every attempt is the eval itself.
🧭 How we see it at Beam
After the last rerun, in our docs' own words, the "Workflow continues with best output", and the task list shows a "Validation Failure" (Beam docs, Evaluation Framework; Task Executions). For underwriting, that means a file that failed appetite three times is flagged but not stopped. That is why we recommend Consent required on the quote or bind step: a setting that makes a person approve before the step acts (Automation Modes). Today, that consent is what turns the flag into a stop.
Across all workflows, we publish 98% accuracy over more than 10 million agent tasks. Even at that rate, 2 tasks in 100 need finding, and the eval record, with its rule, figure and reason, is how a reviewer finds them.
How do you start with your own rulebook?
Start with one line of business and its appetite guide, run the gate in scoring mode next to your underwriters, and turn on reruns only once it agrees with them.
How do you start small?
Pick one line of business you already write at volume, and its appetite guide.
Write one criterion per appetite line, scored 5 or 1, plus number match at weight 5. Keep the carrier's own wording, so a reviewer can set the criterion beside the guideline.
Run the gate in scoring mode, where it scores every file but does not rerun or block, on a few hundred past submissions your underwriters have already decided. Every disagreement is either a badly written criterion or a misread worth knowing about.
Set the appetite step to must-pass, put Consent required on quote or bind, and go live on new submissions.
Track four numbers every week: the share failing an appetite rule, the rerun rate, the share sent to an underwriter, and how often an underwriter overturns the gate. Our docs suggest rewriting the step's instructions, not adding retries, once more than 30% of tasks need a rerun.
What should you ask your team or vendor?
Is each appetite rule its own criterion, with its own reason?
How do the criteria scores combine? Can one failed rule fail the step on its own?
Which weight sits on number match, and is every retry scored against the original submission?
After the last rerun, does the task stop, or continue with a flag?
When appetite changes, who edits the criteria, and how fast does the next run pick it up?
Can I open any single decision and see the rule, the figure, the score and the reason for every attempt?
What does this look like for a mid-size carrier?
An illustrative mid-size carrier writes 2,000 commercial submissions a month across eight appetite lines. The underwriting step gets eight pass or fail criteria plus number match at weight 5, with a threshold of 100; a plain average at 80 would have passed a file with one failed rule, and even two. The referral note gets its own rubric at 80, and quote and bind need consent.
In our model, 300 out-of-appetite files a month stop with a named reason, about 68 misreads are fixed by a rerun, and about 4 more reach an underwriter after three failed attempts. About 13 misreads still slip through, so keep checking a sample of passed files against your underwriters. The reruns cost about $32 a month.
How we use this at Beam
A rulebook already says what a good decision looks like. Turning each rule into its own criterion, letting none be averaged away, and keeping them where the rerun and the record come built in is what makes it a control. That is how we set up underwriting and claims workflows: criteria written from the carrier's own guideline with the process owner, hard rules kept apart from judgment checks, and a person's consent before anything is quoted or bound. For the broader case for checks on every step, read How Quality Gates Make AI Agents Reliable in Production; for where agents fit in insurance operations, AI Agents in Insurance.
If you are deciding where to start, start with the appetite line that costs you most when it is missed.
FAQ
How do you turn underwriting guidelines into AI agent eval criteria?
One line of the appetite guide becomes one criterion, scored 5 if the file meets it and 1 if it does not, with a reason that names the figure from the file and the figure from the guideline. Keep the carrier's wording, keep coverage and number match as separate checks, and put each check on the step where the work happens.
Why does averaging eval criteria let bad files through?
Because passes outvote a fail. With four or more equally weighted rules, one failed rule can never pull a plain average below 80, and two failures out of eight score exactly 80. Hard rules need a threshold of 100 or a must-pass flag, kept apart from judgment checks that use the middle of the scale.
Should the underwriting rulebook live in the agent or be fetched live?
In the criteria, if your team can update them the day appetite changes. The rerun and the audit record come built in. If the book changes faster than that, or cannot leave the carrier's system, fetch it live as its own step and rebuild the retry, which in our model costs about four extra steps and 36 more files per 1,000 sent to an underwriter.
What does the NAIC Model Bulletin require for AI underwriting?
Documented governance, testing and monitoring of AI systems used in insurance decisions. The bulletin was adopted in December 2023 and 27 states plus the District of Columbia had adopted it by November 2026. An eval record that shows, for every file, which rule was checked, the score and the reason is the kind of evidence it asks for.
Figures as of 15 November 2026. Modelled values are labelled as such; vendor-reported figures are marked as vendor claims.





