Durable Execution for Agents: Checkpointing, Resume, and Human-in-the-Loop

An agent that runs for twenty minutes, calls forty tools, and waits for a human halfway through is a distributed system with all the usual failure modes: the process restarts, a tool times out, the approval arrives an hour later. Most agent prototypes handle none of this; they run in one process and hold all state in memory, so any interruption means starting over, re-paying for every model call, and sometimes re-executing side effects. Durable execution is the fix, and it is older than LLMs.

The Property

Durable execution means a multi-step workflow survives crashes, restarts, and long waits without losing progress. If step four fails, the workflow resumes at step four with the results of steps one through three intact. If the process dies while waiting for a human, a new process picks up the wait. Every side effect happens exactly once even if a step is retried.

For agents this requires persisting, after every step: which step you are on, every tool result so far, the model's last plan, the token and cost budget consumed, and any pending approval.

Two Ways to Get It

Graph checkpointing

Model the agent as a graph: nodes are steps (call the model, run a tool, verify, ask a human), edges decide what runs next. After every node, save the full graph state to a store keyed by a run ID. LangGraph does this with a checkpointer; the idea is framework-independent.

pythonCopy
graph = build_agent_graph()
app = graph.compile(checkpointer=SqliteSaver.from_conn_string("runs.db"))
cfg = {"configurable": {"thread_id": run_id}}

app.invoke({"task": task}, cfg)     # crashes after node 3
app.invoke(None, cfg)               # resumes at node 4 from the saved state

What you get: resume after crash, a complete history of every state (which doubles as a debugger: you can inspect exactly what the model saw before the step that went wrong), and time travel (fork a run from an earlier state to try a different path).

Workflow engines

Durable workflow engines (Temporal is the best-known) wrap each step as an activity with retries, timeouts, and idempotency, and the workflow code itself is replayed deterministically from an event log after any crash. The agent loop becomes a workflow; each model call and tool call becomes an activity.

pythonCopy
@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, task):
        plan = await workflow.execute_activity(call_model, task, start_to_close_timeout=60)
        results = []
        for step in plan.steps:
            if step.side_effect:
                await workflow.wait_condition(lambda: self.approved.get(step.id))  # durable pause
            results.append(await workflow.execute_activity(run_tool, step, retry_policy=RETRY))
        return await workflow.execute_activity(call_model, (task, results))

    @workflow.signal
    async def approve(self, step_id: str):
        self.approved[step_id] = True

What you get: everything above plus battle-tested retries, exactly-once semantics for side effects, and waits that can last days. The cost is running the engine and adapting to its programming model.

Pick graph checkpointing for a single team's agent with modest reliability needs. Pick a workflow engine when the agent is a product with SLAs, long waits, and side effects that must not repeat.

Human-in-the-Loop as a State

The most common mistake in prototypes is approval as a blocking prompt: input("ok? ") in the same process. It works in a demo and fails the first time a human is not at the keyboard.

In a durable design, "waiting for approval" is a state the run sits in, persisted like any other. A separate process (a web form, a Slack button, an email link) records the approval and resumes the run, possibly hours later, on a different machine. The approval itself (who, when, exactly which tool call with which arguments) is part of the run's history. That is what makes an agent safe to run unattended: every irreversible action has a recorded approval, and the recording is durable.

Idempotency

Retries are only safe if repeating a step does not repeat its effect. Every side-effecting tool call needs an idempotency key derived from the run ID and step ID, and the tool must honor it (the payment provider returns the existing charge; the email service deduplicates; the database upsert is keyed). Without this, "resume after crash" becomes "charge the customer twice".

What It Buys You Beyond Reliability

• Replay for debugging. Load the state before the bad step and inspect it.
• Replay for evals. Re-run a saved trajectory against a new prompt or model and diff the behavior.
• Cost control. The budget consumed is persisted, so a resumed run cannot exceed it.
• Observability for free. The event log is the trace.

What to Practice Next

Take an agent that holds state in memory. Persist its state after every step to SQLite keyed by run ID. Make approval a saved state resumed by a separate script. Add idempotency keys to every side-effect tool. Then kill the process mid-run, resume, and show from the log that no side effect happened twice. The module harness-engineering-the-runtime-around-the-model puts durable state inside a complete harness.