Harness Engineering: The Runtime Around the Model

Most agent prototypes die between demo and production, and the reason is rarely the model. The team cannot answer the questions security, finance, and the on-call engineer ask: what can this thing do, what did it do yesterday, how much can one run cost, and can it do that again by accident. Those are not model questions. They are harness questions, and the prototype has no harness.

The framing that fixes this is a formula: agent = model + harness. The model proposes actions. The harness, ordinary deterministic code you write, decides which tools exist, validates every proposed call, authorizes or refuses it, executes it, records what happened, and feeds the result back. Reliability, safety, cost control, and debuggability live in the harness. This module is about building one.

The Five Layers

A production harness has five responsibilities. You can build them incrementally, and the order below is the order of pain if they are missing.

Layer	Owns	If missing
Tool orchestration	Tool registry, schemas, per-task allowlists, execution	Model calls tools that do not exist or should not for this task
Guardrails	Permissions, budgets (turns, tokens, dollars), schema validation, approval gates	Runaway loops, unauthorized side effects, five-figure bills
Verification	Checking outputs before the next step: tests, schema checks, sanity checks, judges	Cascading errors; the model builds on a wrong result
Context and memory	What the model sees each turn; what persists across runs	Incoherent long runs; memory poisoning
Observability	Traces of every proposed and executed call, costs, replayable runs	"What did it do?" has no answer

The model contributes planning and language. Everything in the table is yours.

A Minimal Harness

Here is the skeleton, small enough to read in one sitting and complete enough to run. It wraps any model that supports tool calling.

pythonCopy
from dataclasses import dataclass, field
import json, time, uuid

class NeedsApproval(Exception): ...
class BudgetExceeded(Exception): ...

@dataclass
class Tool:
    name: str
    schema: dict            # JSON schema for args
    fn: callable
    side_effect: bool = False

@dataclass
class RunState:
    run_id: str = field(default_factory=lambda: uuid.uuid4().hex)
    turns: int = 0
    tokens: int = 0
    cost_usd: float = 0.0
    trace: list = field(default_factory=list)
    approvals: set = field(default_factory=set)

class Harness:
    def __init__(self, model, tools: dict[str, Tool], *, max_turns=25, max_usd=2.0):
        self.model, self.tools = model, tools
        self.max_turns, self.max_usd = max_turns, max_usd

    def run(self, task: str, allowed: set[str], state: RunState | None = None) -> str:
        state = state or RunState()
        messages = [{"role": "user", "content": task}]
        visible = [self.tools[n].as_model_schema() for n in allowed]   # scoped surface
        while True:
            self._check_budget(state)
            reply = self.model.complete(messages, tools=visible)
            state.turns += 1; state.tokens += reply.usage.total; state.cost_usd += reply.usage.cost
            if not reply.tool_calls:
                self._log(state, "final", {"text": reply.text})
                return reply.text
            for call in reply.tool_calls:
                result = self._execute(state, call, allowed)
                messages.append(reply.as_message())
                messages.append({"role": "tool", "tool_call_id": call.id, "content": result})

    def _execute(self, state, call, allowed) -> str:
        self._log(state, "proposed", {"tool": call.name, "args": call.args})
        if call.name not in allowed:
            return self._refuse(state, call, "tool not available to this task")
        tool = self.tools[call.name]
        errors = validate(call.args, tool.schema)
        if errors:
            return self._refuse(state, call, f"invalid args: {errors}")
        if tool.side_effect and call.name not in state.approvals:
            self._log(state, "paused_for_approval", {"tool": call.name, "args": call.args})
            raise NeedsApproval(state.run_id, call)          # durable pause; resume later
        started = time.time()
        try:
            out = tool.fn(**call.args)
        except Exception as e:
            out = {"error": str(e)}
        out = reduce_for_context(out)                        # context layer: shrink before it enters
        self._log(state, "executed", {"tool": call.name, "ms": int((time.time()-started)*1000),
                                      "ok": "error" not in out})
        return json.dumps(out)

    def _refuse(self, state, call, reason) -> str:
        self._log(state, "refused", {"tool": call.name, "reason": reason})
        return json.dumps({"error": reason})

    def _check_budget(self, state):
        if state.turns >= self.max_turns or state.cost_usd >= self.max_usd:
            self._log(state, "budget_exceeded", {"turns": state.turns, "usd": state.cost_usd})
            raise BudgetExceeded(state.run_id)

    def _log(self, state, event, data):
        state.trace.append({"t": time.time(), "run": state.run_id, "event": event, **data})

Walk through what each line buys you. allowed scopes the tool surface per task, which is both a context win and a security boundary. validate refuses malformed calls before they reach code. side_effect plus approvals turns irreversible actions into a pause the run can resume from after a human says yes. max_turns and max_usd make runaway loops impossible rather than unlikely. reduce_for_context keeps tool output from flooding the window. And _log records every proposed call, including refused ones, which is the difference between a trace you can debug and a trace that only shows the happy path.

Verification Loops

The model is good at producing things and bad at knowing whether they are right. The harness closes that gap by inserting checks the model cannot skip.

• Structural. Output parses, matches the schema, references IDs that exist.
• Executable. Generated code runs; generated SQL executes against a sandbox and returns a plausible row count; the generated config loads.
• Comparative. The result is consistent with a second source (a different tool, a cached value, a small model's answer).
• Judged. For free-text outputs, a separate model grades against a rubric, and you have validated that judge against human labels on a sample.

The pattern is the same for all four: verify, and on failure feed the specific failure back to the model as a tool result rather than a generic "try again". "Row count was 0; expected roughly 1,200" gets fixed in one turn. "Something went wrong" gets you a loop.

The Never-Again Loop

The discipline that makes harness engineering compound: when the agent fails, do not reach for the prompt first.

1. Reproduce from the trace. You have the exact proposed calls and results.
2. Ask which layer should have made this failure impossible. Wrong tool? Orchestration (scope it out). Bad args? Guardrails (tighten the schema). Built on a wrong result? Verification (add a check). Forgot a constraint? Context (carry it verbatim through compaction). Cannot tell what happened? Observability.
3. Fix that layer.
4. Add the failing case to the eval suite so the fix is permanent.

Teams that do this converge on reliable agents over weeks. Teams that iterate on prompts converge on a 3,000-word prompt and the same failure rate.

Sandboxing

Any tool that executes code, shell commands, or arbitrary network calls runs in a sandbox: a container or microVM with no credentials, an allowlisted network, a read-only mount of what it needs, and a hard timeout. The harness passes inputs in and gets outputs back; nothing the model produces runs on the host. This is not paranoia. A prompt injection in a file the agent read can become a command the agent runs, and the sandbox is the layer that makes that boring.

Durable State

Everything in RunState should be persisted after every step, keyed by run_id. That is what lets NeedsApproval be a pause rather than a crash: a separate process records the approval, adds the tool to state.approvals, and calls run again with the saved state. The same persistence gives you crash recovery and replay. Checkpointing frameworks (LangGraph's checkpointers, durable workflow engines like Temporal) do this for you; the skeleton above shows what they are doing underneath.

Common Mistakes and Bad Instincts

• The model is the agent. Every failure becomes a prompt edit.
• Logging only executed calls. The refused and the paused ones are where the interesting bugs are.
• Budgets as suggestions. "Please do not use more than 20 turns" in the prompt is not a budget.
• Global tool lists. Every task sees every tool; every injection reaches every tool.
• Approval in-process. input("ok?") is a demo. Approval is a state the run waits in.
• No eval for the fix. The failure comes back next prompt change.

Where to Go Next

• mcp-and-agent-protocols-building-tool-servers: the tool orchestration layer, standardized
• agent-evals-trajectories-tool-calls-and-regression-suites: the eval suite the never-again loop feeds
• ai-security-for-agents-prompt-injection-excessive-agency-and-sandboxing: the guardrail and sandbox layers under attack

What to Practice Next

Build the harness above around a real model and four tools, two of which have side effects. Persist RunState to SQLite after every step. Demonstrate, with traces: a refused out-of-scope call, a refused invalid-args call, a run that pauses for approval and resumes in a new process, a run that hits the turn budget, and a verification failure that the model corrects in the next turn. Then run one deliberately bad task and use the never-again loop to fix it in the harness, not the prompt, and add it to a test.