τ-bench and SWE-bench: What Agent Benchmarks Actually Measure

Every model release comes with a table of agent benchmark scores, and every table invites the wrong conclusion: that a higher number means the model will do better in your product. Benchmarks measure something narrower and more specific than that. Knowing exactly what makes you a better reader of model announcements and a better designer of your own evals.

SWE-bench: Resolving Real Issues

SWE-bench takes real issues from open-source Python repositories, each paired with the pull request that fixed it and the tests that PR made pass. The agent is given the repository and the issue text and must produce a patch. The patch is graded by running the tests: the ones that should now pass, and the ones that should still pass.

What it measures: an agent's ability to navigate an unfamiliar codebase, localize a bug from a natural-language description, make a correct change, and not break anything else, using a specific scaffold (the harness the benchmark runners used). That last part matters. The same model scores very differently with different scaffolds, which is direct evidence that the harness is part of the agent.

What it does not measure: your codebase, your language, your conventions, your tests. And because the repositories are public and old, models can have seen the fixes during training. Subsets (Verified, Lite, and others) exist to control for quality and contamination; read which one a number refers to.

τ-bench: Tools, Policies, and a Simulated User

τ-bench (tau-bench) puts an agent in a customer-service scenario (airline, retail) with a set of tools, a policy document ("refunds only within 24 hours unless..."), and a simulated user played by another model with a hidden goal. The agent must converse, call tools, follow the policy, and reach the right end state in a database.

What it measures: multi-turn tool use under rules, with an unpredictable counterpart. The grading is on world state (was the right reservation modified in the right way), not on the transcript.

Its most useful contribution is the metric pass^k: the probability that the agent succeeds on all k independent attempts at the same task. A model with 70% pass^1 might have 35% pass^4. That gap is reliability, and it is what users experience, because a user who hits the failing 30% does not get to rerun.

Why Numbers Saturate

Benchmarks are static and public. Three things follow. Models improve on them faster than on the underlying skill, because training data and post-training targets drift toward them. Scaffolds get tuned to them, so a score reflects the scaffold as much as the model. And once the top of the leaderboard is near the ceiling, differences between models are noise plus contamination. A benchmark is most informative in the middle of its life.

How to Read a Result

• Which subset, which scaffold, how many runs? A pass^1 on Lite with a custom scaffold is not comparable to pass^4 on Verified with the reference scaffold.
• Is it the model or the agent? If a vendor reports a score for "our agent", the harness is in the number. That is fine; just do not attribute it to the model alone.
• What is the task's shape? SWE-bench is "modify code to pass tests". If your product is "summarize meetings", the number tells you little.
• Reliability or capability? pass^1 is capability. pass^k is reliability. Products need the second.

What to Take From Benchmarks

Use them to shortlist models. Then build your own suite, with your tools and your harness, and measure pass rates over several runs on tasks drawn from your real traffic. That number is the only one that predicts your product, and it is also the number that improves when you fix your harness rather than swap your model.

What to Practice Next

Find the benchmark table in a recent model announcement. For each agent benchmark, write down the subset, the scaffold, and whether it is pass^1 or pass^k. Then write three tasks from your own product in the same shape as the closest benchmark and run them five times each. Compare your pass^5 to the announced number; the gap is the difference between a benchmark and a system. The module agent-evals-trajectories-tool-calls-and-regression-suites shows how to build the suite that closes it.