Agent Memory: Working, Episodic, and Long-Term, and Why Compaction Is Lossy

Every agent framework has a "memory" feature and every one means something slightly different by it. Underneath, there are three distinct things, and the bugs that plague long-running agents (forgetting a constraint mid-task, repeating a mistake from last week forever, confidently recalling something that never happened) come from treating them as one.

The Three Stores

Working memory is the context window: what the model sees on this turn. It is fast, complete for the current run, and gone when the run ends. Its problem is size: it fills.

Episodic memory is a record of what happened in past runs: transcripts, summaries, tool results, outcomes. Written at the end of a run, retrieved by similarity or recency when a new run might benefit. Its problem is relevance: most of it is noise for any given new task.

Long-term memory is curated facts that should influence every run: user preferences, project conventions, decisions already made, things that went wrong and must not recur. Small, stable, and deliberately maintained. Its problem is trust: anything in it is believed.

Store	Written by	Read by	Size	Failure
Working	The loop	Every turn	Bounded by the window	Fills; compaction loses detail
Episodic	Harness, at run end	Retrieval, when relevant	Grows unbounded	Irrelevant recall; stale facts
Long-term	Deliberately, with review	Every run, in full	Tens of entries	Poisoning; permanent wrong beliefs

Write Policies

The single most important rule: the model may propose long-term memory writes; the harness or a human commits them.

An agent that writes freely to long-term memory will, sooner or later, record something wrong: a misread instruction, a fact from an injected document, a preference the user expressed once sarcastically. From then on every run starts with that belief. This is memory poisoning, and it is on the OWASP agentic risk list for a reason.

pythonCopy
def end_of_run(state, proposals):
    # episodic: always written, tagged, never trusted as instruction
    episodic.append(summarize(state.trace), tags=state.task_type, run_id=state.run_id)
    # long-term: proposals go to a review queue with provenance
    for p in proposals:
        review_queue.add(p, run_id=state.run_id, sources=state.untrusted_sources_read)

Entries carry provenance (which run, which sources it read). Entries are inspectable and deletable. Runs that read untrusted content have their proposals flagged. Long-term memory stays small enough to read in full at the start of every run, because if you cannot read it, you cannot audit it.

Episodic Retrieval Done Right

The naive design retrieves the most similar past transcript and pastes it in. Two problems: transcripts are long, and similarity is not relevance. Better:

• Store structured summaries (task, tools used, outcome, one-line lesson) alongside the raw transcript.
• Retrieve summaries; load the raw transcript only if the model asks for it.
• Filter by outcome. A past failure is useful as a warning; a past success is useful as a template; both should be labeled as which they are.
• Expire or down-weight by age. A fact about the codebase from six months ago may be false.

Compaction: The Working-Memory Problem

When the context fills, something is dropped or summarized. Summaries are lossy, and the loss is invisible until the model confidently forgets that the user said Thursday, not Friday.

Three practices that make compaction survivable:

1. Keep raw artifacts retrievable by ID. Every tool result is stored outside the context with an ID. The summary says "search results for X (#41)" and the model can re-fetch #41. Nothing is truly deleted.
2. Carry constraints verbatim. The summary carries a short list of commitments word for word: "output must be CSV", "do not modify data/raw", "deadline Thursday". Paraphrasing constraints is how they get lost.
3. Compact tool results eagerly, conversation lazily. A 4,000-token JSON blob should be reduced to what matters before it enters the context. Conversation turns are cheaper to keep and more expensive to lose.

And measure: compaction events per run. Many compactions means the context strategy is wrong upstream (too many tools, tool output not reduced), not that the summarizer needs to be better.

A Note on "Reconstruction"

A newer line of thinking treats the context not as something to summarize but as something to rebuild from durable state on each turn: the task spec, the constraints, the current plan, the last few results, and pointers to everything else. Instead of a lossy rolling summary, the harness assembles a fresh, complete, bounded context every turn from structured state. It is more work to build and dramatically more robust for long runs, and it pairs naturally with checkpointing.

What to Practice Next

Take an agent with a "memory" feature and classify every write it makes into the three stores. Add a review queue for long-term writes with provenance. Add IDs to tool results and make the summary point at them. Then run a 40-turn task and check whether a constraint stated in turn 2 is still honored in turn 38. The module context-engineering-designing-what-the-model-sees covers memory alongside the rest of the context budget.