The AI Evals Engineer: A New Role and How to Get It

Two years ago "evals" was a task an ML engineer did on Friday afternoons. Now it is a job title, and job boards list it next to AI engineer and agent engineer. The reason is simple: teams shipping LLM features and agents discovered that the hard part is not making the model do something, it is knowing whether it did the right thing, and keeping that true after every change. Someone has to own that, and the someone is an evals engineer.

What the Job Is

An evals engineer designs, builds, and maintains the systems that measure whether AI features and agents are doing their jobs. Concretely, in a typical week:

• Write and curate eval datasets. Turn production failures, support tickets, and product requirements into tasks with graded outcomes. Decide what "correct" means when it is not obvious, and write it down as a rubric.
• Build grading pipelines. Mechanical checks where possible (parses, matches, tests pass, tool called, world state correct); LLM judges where necessary, validated against human labels.
• Run regression suites in CI. Every prompt, tool, harness, or model change runs the suite; the evals engineer owns the gate and the thresholds.
• Own online experimentation. A/B tests comparing versions in production, with metrics that connect to the offline suite.
• Triage and route. When a metric drops, figure out whether it is the model, the prompt, the tools, the harness, or the inputs, and hand it to the right owner with a reproducible case.
• Argue about quality. With product, with engineering, with legal. The evals engineer is the person in the room who can say what "good" means in a way that can be measured.

Why Companies Hire for It

Evals and benchmarking now appear in most AI engineering job postings, ahead of RAG. Three forces:

1. Agents. A chat feature has an answer to grade. An agent has a trajectory, a world state, and a cost, and grading those is a specialty.
2. Change velocity. Models update monthly, prompts change weekly. Without a suite, every change is a gamble.
3. Accountability. When an AI feature fails a customer, someone has to explain what the system was measured against. Increasingly that is a regulatory question as well as a product one.

The Skills

Matter a lot:

• Precise thinking about what "correct" means, and the ability to write it as a rubric a stranger could apply.
• Data wrangling: building, cleaning, deduplicating, and versioning datasets.
• Enough statistics to know when a 2-point difference is noise. Confidence intervals, multiple runs, sample sizes.
• Software engineering: the suite is a codebase with CI, fixtures, and a sandbox.
• Understanding how LLMs and agents fail, so tasks target real failure modes.
• LLM-as-judge design and validation, including the known biases.

Matter less than people think:

• Training models. Most evals engineers never do.
• Deep ML theory. Useful; not the job.
• Prompt engineering. You grade prompts; you do not primarily write them.

The Interview

Expect a take-home or live exercise of the form: "Here are 50 transcripts from our agent. Design an eval." Strong answers name the failure modes they see, propose mechanical checks for the ones that can be mechanized, write a rubric for the rest, explain how they would validate a judge, and say how the suite would run in CI and what would fail a build. Weak answers propose "use GPT to score them 1-10".

Also expect: a statistics question about whether a result is significant; a question about a metric that moved with no code change (the inputs drifted); and a product question about tradeoffs between quality, cost, and latency.

A Portfolio That Works

You do not need permission to build an evals portfolio. Pick a public agent or build a small one, then:

1. Write 30 tasks with graded outcomes, from three sources: failures you observed, representative successes, adversarial cases.
2. Implement mechanical trajectory checks and one validated judge (with the human-agreement numbers).
3. Run it five times per task, report pass rates per metric with confidence intervals.
4. Put it in CI with a threshold, break something deliberately, and show the gate catching it with a trace diff.
5. Write a two-page report on what you learned about the agent's failure modes.

That repository answers every interview question above with evidence, and it takes a weekend or two.

Paths In

From QA or SDET: you already think in test cases and regression; add LLM failure modes and judge validation. From data science: you already think in metrics and significance; add the CI and harness side. From backend engineering: you already build pipelines; add rubric design and statistics. From product or analytics: rarer, but the "what does good mean" instinct is the scarce half of the job.

What to Practice Next

Build the portfolio above. The module agent-evals-trajectories-tool-calls-and-regression-suites is the full method; evaluating-ai-agents-trajectories is the short version to start from today.