Reasoning Models, Post-Training, and Test-Time Compute

Every provider now sells two kinds of model: one that answers, and one that thinks first. The thinking kind is better at math, code, planning, and anything with a checkable answer, and it is slower and more expensive per request, sometimes by a lot. As an application engineer you make a "pay for thinking or not" decision on every request, whether you know it or not. This module gives you the background to make it deliberately: where these models come from, why they behave the way they do, and how to measure the tradeoff on your own workload.

The Post-Training Pipeline

A base model is trained to predict the next token on a large corpus. Everything that makes it useful as an assistant or an agent happens afterward, in post-training, which now has three recognizable stages.

Supervised fine-tuning (SFT). Show the model examples of the behavior you want (prompt, ideal response) and train it to imitate them. This is where instruction-following and format come from. It is limited by the quality and quantity of examples, and it cannot teach the model to do better than its examples.

Preference optimization (DPO and relatives). Show the model pairs of responses to the same prompt, one preferred and one rejected, and train it to raise the probability of the preferred one. Direct Preference Optimization does this without training a separate reward model, which made it the default over the older RLHF-with-PPO recipe for most teams. This stage tunes style, helpfulness, and safety.

Reinforcement learning with verifiable rewards (RLVR). For tasks where correctness can be checked automatically (the numeric answer matches, the tests pass, the output parses, the proof verifies), you do not need human labels. Sample responses, check them, reinforce the correct ones. Because nothing constrains how the model reaches the answer, it learns to spend tokens on intermediate steps, backtracking, and self-checking, because those raise its reward. That is a reasoning model.

The algorithm that made RLVR practical at scale is GRPO (Group Relative Policy Optimization). Classic PPO needs a value model to estimate how good a state is, which doubles memory and adds instability. GRPO drops it: for each prompt, sample a group of G responses, score each, and use each response's reward relative to the group's mean as its advantage.

pythonCopy
def grpo_step(policy, ref_policy, prompt, verify, G=8, kl_coef=0.04):
    responses = [policy.sample(prompt) for _ in range(G)]
    rewards = [verify(prompt, r) for r in responses]           # e.g. 1.0 if tests pass else 0.0
    mu = sum(rewards) / G
    sigma = (sum((r - mu) ** 2 for r in rewards) / G) ** 0.5 + 1e-6
    advantages = [(r - mu) / sigma for r in rewards]
    loss = 0.0
    for resp, adv in zip(responses, advantages):
        # raise log-prob of tokens in above-average responses, lower below-average
        loss += -adv * policy.logprob(prompt, resp)
        # stay close to the reference so the policy does not drift into reward hacking
        loss += kl_coef * kl(policy, ref_policy, prompt, resp)
    return loss / G

The group normalization is the whole trick: a response is only "good" relative to its siblings, so the signal stays informative whether the task is easy or hard. The KL term keeps the model from finding degenerate outputs that satisfy the verifier without being useful.

What you should take from this: reasoning is a training outcome, not a prompting trick. Asking a non-reasoning model to "think step by step" borrows the form without the training that made the form reliable.

What RLVR Cannot Do

RLVR needs a verifier. Math, code, structured extraction, logic puzzles, and games have one. "Write a compelling product description" does not. For those tasks the recipe is still preference data plus DPO or a learned reward model, and reasoning models offer a smaller advantage. This is why reasoning-model benchmark wins cluster in math and code, and why you should not assume the gain transfers to your summarization feature.

It also means the frontier of post-training is partly the frontier of verifier design: the more of a task you can make checkable, the more RL can improve it. That is an engineering skill, and it is the same skill as writing a good eval.

Test-Time Compute

Once a model can reason, inference has a new dial: how much to think. Three forms.

• Longer reasoning. More thinking tokens before the answer. Most reasoning APIs expose a budget.
• Parallel sampling. Generate several answers and take the majority, or the one a verifier accepts. Cost multiplies by the sample count.
• Search. Explore candidate solutions, keep the promising ones, expand. Expensive; mostly used offline or in agents with a verifier in the loop.

All three trade latency and cost for accuracy, and all three have diminishing returns. The curve is task-specific: a task that improves from 60% to 85% with 2,000 thinking tokens might reach 87% with 20,000. Somewhere on that curve is the point where the next dollar is wasted, and you need to find it empirically.

Deciding Which Requests Get Reasoning

Do this once per feature; revisit when the model or the inputs change.

1. Build an eval set of 100 to 300 real requests with graded answers.
2. Run it at four settings: non-reasoning model; reasoning model with a low budget; medium; high.
3. For each setting record accuracy, p50 and p95 latency, and cost per request.
4. Plot accuracy against cost. Find the knee.
5. Segment: often a subset of requests (long inputs, multi-step questions, code) gets all the gain. Build a router that sends only those to the reasoning path.

setting              accuracy   p95 latency   cost/req
non-reasoning          71%         1.1s        $0.004
reasoning, low         84%         3.8s        $0.019
reasoning, medium      87%         7.2s        $0.041
reasoning, high        88%        15.9s        $0.096

In a table like this, "low" is the answer for the requests that need reasoning, and a router that keeps 60% of traffic on the non-reasoning path cuts the blended cost further with no measurable loss. That table, for your own workload, is what an interviewer wants to hear you have made.

Synthetic Data and the Feedback Loop

Almost all post-training data today is partly synthetic: a stronger model generates candidate responses, a verifier or a judge filters them, and the survivors become SFT or preference data for the next model. This is how post-training scales without armies of annotators. The risks are inherited quirks and, in the limit, model collapse from training on your own outputs. The mitigation is the same as everywhere in this path: verification before anything enters the training set, and a held-out human-labeled eval that the synthetic pipeline never touches.

If you fine-tune a small open-weight model for a narrow verifiable task (your SQL dialect, your internal API's call patterns), this is the recipe you will use: generate with a large model, filter with your verifier, SFT the small model, and optionally run a short GRPO phase with the same verifier. It is feasible with open tooling and modest GPU budgets, and it is one of the strongest portfolio projects available right now, because it demonstrates the entire pipeline end to end.

Common Mistakes and Bad Instincts

• Reasoning everywhere. Most requests do not need it; the bill says so.
• Trusting benchmark deltas. Gains cluster in verifiable domains; measure on your task.
• "Think step by step" as a substitute. Form without training.
• Max thinking budget by default. Find the knee.
• RL without a verifier. If you cannot write the checker, use preference data.
• Synthetic data without a held-out human set. You will not see the collapse coming.

Where to Go Next

• serving-models-and-llm-systems-in-production: routing, cascades, and where the reasoning path fits in a serving stack
• agent-evals-trajectories-tool-calls-and-regression-suites: the eval set this module's method depends on
• fine-tuning-adaptation-and-when-not-to-fine-tune: the decision framework that says when any of this is worth doing

What to Practice Next

Take one feature with a checkable output. Build a 150-request eval with graded answers. Run the four-setting comparison, produce the accuracy-versus-cost table and plot, and write a routing rule that sends only the requests that benefit to the reasoning path. Report the blended cost and accuracy of the routed system against "reasoning for everything" and "reasoning for nothing". Stretch: fine-tune a small open-weight model on the same task with verifier-filtered synthetic data and add it as the cheapest tier.