How We Select the Right Frontier Model for Each Problem

Key takeaways

• Eight leading models sit within six points on one aggregate intelligence index, yet their order changes when the task changes.

• The grader changes the result. Human preference, exact-answer checks and a model's willingness to abstain reward different behaviour.

• In one field study, a smaller model trained on a company's own A/B results increased clicks by 32%. Generic ChatGPT rewrites produced no uplift.

• The same model has scored up to seven points apart on the same coding benchmark under different harnesses.

• Claude Opus 5.5 gained four benchmark points between high and max effort while cost per task rose 3.3 times. Effort belongs in the evaluation, alongside the model.

Which model should run this step?

That question comes up repeatedly when we build an agent. The tempting answer is the model at the top of the latest leaderboard. In practice, the leaderboard is where the work starts.

A model can lead overall and still lose on exact answers, admitting uncertainty, visual judgement or writing. The grader can change the winner. The harness can move a score by more than the published gap between two frontier models.

So we evaluate the model inside the job it will actually do. We define the output, decide how it will be judged, run the candidates inside the real agent setup and compare the cost of an accepted result. Public benchmarks help us decide what to test. The workflow decides what to run.

What should an LLM evaluation measure?

An LLM evaluation should measure whether the model produces an acceptable result for one defined step of the workflow. That requires a specific input, a clear output and a grading method that reflects what success means in production.

Overall model rankings are useful shortlists

Composite leaderboards compress many types of work into one score. Eight models from four labs sit within six points on Artificial Analysis's Intelligence Index, but the finishing order changes across the underlying tasks. Claude Opus 5.5 leads overall and finishes last in this comparison at admitting what it does not know. Gemini 4 Argon ranks fourth overall and first on three individual jobs.

The overall score identifies models with enough general capability to test. The task-level results identify where their behaviour separates. An aggregate lead is weak evidence for a specific agent step unless the work and grading resemble the benchmark.

Define the grader before comparing models

The same output can look strong or weak depending on what the evaluation rewards. In the examples below, Claude leads when people judge the finished deliverable and presentation matters. GPT-6 Astra leads when an answer key checks an exact result. Gemini 4 Argon leads when declining to guess is preferable to returning a plausible error.

This changes the first question in a model evaluation. Instead of asking which model is strongest, ask what a successful result looks like for this step. A refund agent may need a verified state change. A research agent may need source coverage and groundedness. A design task may need an expert to judge whether the output is usable.

The evaluation should reward the behaviour the workflow needs. A benchmark with a different grader can still reveal capability, but it cannot make the deployment decision on its own.

How should you evaluate writing and design models?

Writing and design need a combination of rubric-based and human evaluation because quality includes judgement. Exact-answer scores can check factual requirements, but they cannot decide whether the output has the right voice, visual hierarchy or level of finish.

Design exposes differences in model taste

The same prompt can produce recognisably different design decisions. In the seaplane comparison, four models created different scenes at prices up to four times apart. Claude Opus 5.5 produced the most finished result in this example. Gemini 4 Argon came close at roughly a quarter of the cost.

That preference does not transfer automatically to the next brief. In the Newton's cradle example, the author preferred GPT-6 Astra's treatment of the physics scene to Opus.

Newton's cradle. The author rated GPT-6 Astra's version better: "Opus 5.5 sucks here."

Source: Ivana on X

These are useful probes, rather than general model rankings. They show what to inspect in a larger set: composition, hierarchy, physical behaviour, completeness and the amount of specification the model needs before it produces acceptable work.

Writing needs a voice test and an outcome test

Four models given the same product announcement can preserve the same facts and take four different approaches to the headline. In the comparison below, GPT-6 produced the plainest first draft while Claude reached for a more distinctive line.

That tells us something about default style. It does not tell us which copy will perform. The voice test asks whether the draft fits the company and audience. The outcome test asks whether it improves the metric the step exists to move.

The distinction matters. In one field study, a smaller model trained on a company's previous A/B results increased clicks by 32%, while subject lines rewritten by ChatGPT produced no uplift (Lee et al.). A pleasing draft and a commercially effective draft are separate results.

How should cost factor into LLM evaluation?

Compare cost per accepted result, rather than token price alone. A cheaper model can use more tokens, fail more cases or create additional review work. A more expensive model can finish the same task with fewer attempts.

Bounded tasks are still where smaller models often perform best. On the customer-service tool-use comparison cited here, Claude Haiku 5.5 scores above Claude Opus 5.5 at about one cent per task (OpenRouter). A short policy and a checkable outcome leave the larger model little room to add value.

The reverse can also happen. At a matched benchmark score, Claude Opus 5.5 costs less per completed task than Claude Sonnet 5.5 because the models consume different numbers of tokens. One team's projected 95% saving from a price sheet became 68% after the full agent loop ran in production (Unblocked).

Effort adds another variable. Claude Opus 5.5 gains four points between high and max effort in the comparison below, while cost per task rises 3.3 times.

The engineering question is whether those four points remove failures that matter in this workflow. If the accepted cases already pass at high effort, max buys benchmark performance without improving the product. Anthropic similarly describes effort as a lever to test before switching models.

Our guide to choosing Claude models for enterprise AI agents goes deeper into the model-and-effort grid. For a cross-provider evaluation, the principle is simpler: hold the cases and grader constant, then compare the total cost of passing them.

Why should you test an LLM inside the agent harness?

The evaluation unit is the model plus the system around it. Prompts, tools, context, retry logic, resource limits and provider settings can all change the result.

The same model has scored up to seven points apart on the same coding benchmark depending on who ran it. That spread is wider than the gap between many frontier models. Settings can also carry different meanings across providers. Gemini 4 Argon accepts a temperature parameter but ignores it, according to Google Cloud's model documentation.

A clean API comparison can therefore select a model that performs differently once tools and orchestration are attached. Run the candidate inside the production harness, with the real prompt, tool definitions, context limits and retry policy. Record the trace as well as the final output so a pass can be explained and a failure can be reproduced.

How often should you re-evaluate an LLM?

Re-evaluate whenever the model, prompt, tools, harness or acceptance criteria change. Keep the accepted cases as a regression set so a provider update or migration can be tested before it changes production behaviour.

Model choice has become an operating process because providers change the underlying system. This year, one frontier model was switched off for every customer overnight, another was shelved before release, and some API names began pointing to newer versions. Nearly one in three teams changed their primary lab in September, according to Vercel's AI Gateway Production Index.

A fallback from another provider gives the workflow somewhere to go during an outage or retirement. It still needs to pass the same cases. Provider diversity without tested behavioural compatibility only moves the failure.

A practical framework for evaluating LLMs in an agent workflow

Evaluate at the level of one agent step. A planner, classifier, writer and checker can have different models because their outputs, graders, latency budgets and failure costs differ.

1. Define the output

State what the step produces: a label, a database change, a research report, a reply or a design. Keep the boundary narrow enough that a failure can be attributed and retested.

2. Define how the result will be accepted

Use deterministic checks for exact outputs and state changes. Use structured rubrics for qualities such as groundedness, tone and completeness. Calibrate subjective graders with the people who accept the work in practice.

3. Use public evidence to choose candidates

Select benchmarks that resemble the task and inspect their grading method. Use aggregate rankings as a shortlist. Include a small model when the work is bounded, and include more than one provider where continuity matters.

4. Run the candidates in the real harness

Keep the prompt, tools, context and test cases constant. Compare multiple runs where consistency matters. Record traces, latency, token use, retries and the final state.

5. Compare cost per accepted result

Count the complete workflow, including failed attempts and review. Test effort settings as well as model families. The useful saving is the one that survives the agent loop.

6. Keep the cases and test the fallback

Turn accepted outputs and production failures into a regression set. Rerun it when a model alias, provider version, prompt or tool changes. A fallback earns its place by passing those cases before it is needed.

How we see it at Beam

In Beam, the model is a setting on each step of an agent, with fallback models and an automatic rerun when accuracy drops below a threshold. Every model available to Beam is ranked on our own agent workloads. Public benchmarks tell us which models to test; our own cases decide which model a step runs on.

In my own use, Claude has been noticeably better at design work than OpenAI's models. With GPT-6.1 Sol, I had to spell out basic layout instructions I would expect a model at that level to handle. That is one experience with a particular kind of work, which is exactly why we test the task before generalising the model choice.

The right LLM for an agent step is the one that passes that step's real cases, inside its real harness, at an acceptable completed-task cost.

FAQ: Evaluating LLMs for AI agents

Which LLM benchmark should I use for an AI agent?

Use a benchmark that resembles the agent step and rewards the behaviour you need. Tool-use and coding benchmarks can help shortlist models, but the final evaluation should run accepted cases from your workflow through the actual prompt, tools and harness.

Should every step in an AI agent use the same model?

Different steps can use different models. Planning may need open reasoning, while classification may be bounded and easy to check. Evaluate each step against its own output, latency budget, failure cost and acceptance criteria, then configure the model at that level.

How should I compare LLM costs?

Measure cost per accepted result. Include all tokens, retries, failed attempts and human review created by the step. Also test effort settings, because a lower setting on the same model may preserve quality with fewer variables than a complete model migration.

When should an LLM evaluation be rerun?

Rerun it when the model version or alias changes, and whenever the prompt, tools, context, harness or acceptance criteria change. Keep a regression set of accepted cases and meaningful failures so the new configuration can be compared with the previous one.

Scores, prices and availability as of 19 October 2026. Vendor-reported figures and single-run social comparisons should be labelled in the final Framer version.