Reasoning Models and Test-Time Compute: When to Pay for Thinking

Every provider now sells a model that thinks before it answers. The thinking is real: the model generates intermediate reasoning tokens, sometimes thousands of them, before the visible response, and on the right tasks accuracy jumps. It is also billed, and it is slow. So every request you send is, implicitly, a decision to pay for thinking or not. Most teams make that decision once, globally, and get it wrong in one direction or the other.

What Test-Time Compute Is

Training compute is spent once. Test-time compute is spent per request, and reasoning models give you a dial. Three forms:

• Longer reasoning. The model thinks for more tokens. Most APIs expose a thinking budget you can set per request.
• Parallel sampling. Generate several answers, take the majority or the one a verifier accepts. Cost multiplies by the sample count; useful when you have a checker.
• Search. Explore candidate solutions, keep promising ones, expand. Expensive; mostly for offline use or agents with a verifier in the loop.

All three trade latency and cost for accuracy. All three have diminishing returns.

Why Returns Diminish

The gain from thinking comes from the model catching its own errors and exploring alternatives. On a task that needs three steps, a budget that allows five is enough; a budget that allows fifty adds nothing but cost. And past a point, longer reasoning can hurt: the model second-guesses a correct answer or drifts. The curve for any given task rises steeply, flattens, and sometimes dips. Your job is to find the flat part.

The Four-Setting Method

Do this once per feature and again when the model or the inputs change.

1. Assemble 100 to 300 real requests with graded answers. Real, not synthetic; the distribution is what you are measuring.
2. Run them at four settings: non-reasoning model, reasoning at a low budget, medium, high.
3. Record accuracy, p50 and p95 latency, and cost per request for each.
4. Plot accuracy against cost. The knee is where the next dollar stops buying accuracy.

setting              accuracy   p95 latency   cost/req
non-reasoning          71%         1.1s        $0.004
reasoning, low         84%         3.8s        $0.019
reasoning, medium      87%         7.2s        $0.041
reasoning, high        88%        15.9s        $0.096

In this table (a real shape, if not your numbers), "low" is the right setting for requests that need reasoning at all, and "high" buys one point for five times the cost.

Route, Don't Choose

The table hides the more important finding: usually a subset of requests gets all the gain. Long inputs, multi-step questions, code, anything with a checkable answer. Short factual lookups and formatting tasks gain nothing from thinking. So the answer is rarely "which setting" and usually "which requests".

pythonCopy
def route(request):
    if request.has_code or request.steps_estimate > 2 or len(request.text) > 1500:
        return REASONING_LOW
    return NON_REASONING

A rule like that, tuned on your eval set, often keeps 60% of traffic on the cheap path with no measurable accuracy loss. A small model can also act as the router, or a first attempt on the cheap path can escalate when a verifier rejects it (a cascade). Log the route on every request; route distribution drifting is an early warning that inputs changed.

Where Reasoning Helps and Where It Doesn't

Gains cluster in verifiable domains, because that is where these models were trained (see the post on GRPO and RLVR). Expect help on math, code generation and repair, multi-step planning, structured extraction from messy input, and constraint satisfaction. Expect little on summarization, open-ended writing, and simple classification, where a non-reasoning model at a fraction of the cost does as well. Measure, but set your expectations from the training recipe.

Reasoning in Agents

Agents multiply the decision: an agent making twenty model calls per task, all on a reasoning model at a high budget, is expensive and slow, and most of those calls are "which tool next", which rarely needs deep thought. Route within the agent: reasoning for planning and for verifying a final result, non-reasoning for routine tool selection. And measure task-level pass rate against cost per task, not per-call accuracy.

What to Practice Next

Run the four-setting method on one feature. Deliver the table, the plot, and a routing rule, and report the blended accuracy and cost of the routed system against "reasoning for everything" and "reasoning for nothing". The module reasoning-models-post-training-and-test-time-compute covers where these models come from and how to fine-tune a small one for the cheap tier.