Test-Time Compute: Why AI Agents Now Get Smarter by Thinking Longer, and When It Backfires

For most of the last few years, a smarter model meant a bigger one. That axis has quietly rotated. Some of the sharpest recent gains come not from training a larger model but from letting an existing one think longer at the moment you ask it a question. The industry calls this test-time compute, or inference-time compute, and it changes how you should design an agent.

The short version: reasoning is now a dial, not a fixed property. And like any dial, turning it all the way up is sometimes brilliant and sometimes a waste.

What test-time compute actually is

When a model spends extra steps working through a problem before it answers, it is using test-time compute. Those intermediate steps, often called reasoning tokens, cost compute at the moment of inference rather than during training. Most current frontier models expose this as a reasoning-effort setting you can turn from low to high.

The recent launches make the shift concrete. GPT-6 Astra uses a technique its makers call opaque recurrence, retaining reasoning state between turns so it can keep thinking across a task. DeepSeek's V4.1-Flash ships with controllable reasoning effort trained directly into the model. The knob is becoming a standard part of the interface, which means deciding where to set it is now part of building an agent.

The trade-off: accuracy bought with latency and cost

More thinking is more tokens, and more tokens mean more latency and more spend. On hard problems that trade is often worth it. On easy ones it is pure waste, and past a certain point the returns flatten out entirely.

It can also backfire outright. A model given room to over-think can second-guess a correct answer, wander off the task, or loop. The practitioner reports on this year's cheaper models describe exactly that, agents re-reading their own work and spinning without progress. More reasoning is not a free upgrade. It is a setting with a cost and a failure mode.

The counter-intuitive part: more effort can be cheaper

Here is where intuition breaks. When the ARC Prize team evaluated GPT-6 Astra on ARC-AGI-3, they found that at maximum reasoning effort the model solved each puzzle in far fewer moves, needing about 51.7% fewer actions than the human baseline on 96% of levels. Max effort turned out to be the cheapest path to a good score, because thinking harder up front meant acting less.

DeepSeek, meanwhile, built an explicit exponential token penalty into how it controls reasoning effort, a sign of how carefully the trade needs to be managed rather than left wide open. The lesson from both is the same: reasoning effort is not linear with cost, and it is not linear with quality. You have to tune it, not assume it.

What it means for AI agents: reasoning effort is a per-step dial

An agent is a sequence of steps with wildly different difficulty. A classification step and a disputed-decision step do not need the same amount of thinking, and paying for deep reasoning on every call is how an agent gets slow and expensive for no benefit.

The move is to set reasoning effort per step, the same way you assign a model per step. Crank it up on the multi-step judgment and the high-stakes decision. Keep it low on the high-volume, mechanical work. Cap it wherever the agent tends to loop.

And measure the result on cost per completed task, not cost per token. A step that thinks longer but finishes correctly on the first attempt is often cheaper end to end than a fast one that has to be retried.

The frontier's rising bill

Test-time compute is real capability, but it is not free, and the cost is moving somewhere visible. Analyses of model pricing now note that the cost of running the actual frontier models is rising, partly because those models spend more reasoning tokens per task. Training-time scaling was the lab's bill to pay. Test-time scaling is increasingly yours, at runtime, which is exactly why the dial belongs under your control rather than pinned to maximum.

How Beam handles reasoning effort

The same logic that makes model choice a per-step decision applies to how hard each step should think. In Beam, reasoning effort and model selection are set at the step level, with fallback and retry catching the cases that go wrong, and the whole thing measured on completed tasks rather than raw token spend. The goal is not the most thinking. It is the right amount, in the few places it changes the answer.

Common questions about test-time compute for AI agents

What is test-time compute?

Test-time compute, also called inference-time compute, is the extra work a model does while answering, spending reasoning tokens to think through a problem before it responds. It is a way to improve results without training a bigger model, and most current models let you control how much of it to use.

Does more reasoning always make an AI agent better?

No. On hard problems more reasoning helps, but returns flatten out and can reverse, with the model over-thinking, second-guessing, or looping. It also adds latency and cost. The right amount of reasoning is step-dependent, not "as much as possible."

What is reasoning effort and how should I set it?

Reasoning effort is a setting that controls how much a model thinks before answering. Set it low for high-volume, simple, latency-sensitive steps and high for multi-step judgment and high-stakes decisions. Tune it per step and measure the effect on cost per completed task, not per token.

Is test-time compute different from training compute?

Yes. Training compute is spent once, when the model is built. Test-time compute is spent every time the model answers. The recent shift is that a lot of new capability now comes from test-time compute, which means the cost, and the control, moves to inference and to you.