Agent Memory Deep Dive #0: The State of Agent Memory — Why It's Still Broken

Every agent today has amnesia, a hoarding problem, or both. There is no middle ground.

TLDR: Agent memory is the most visibly broken component of the current AI stack. I spent some time researching every major approach — from Claude Code’s file-based system to Mem0’s vector store to Letta’s virtual memory to brain-inspired architectures. The verdict: we’re in the “flat files era” of memory. The equivalent of storing user data in CSV files before databases existed. Here’s where we actually are, why nothing works well enough, and what the ideal system would look like.

The Problem You Already Feel

You use AI agents daily. You’ve already experienced this:

You told Claude your project uses TypeScript three sessions ago. It suggests Python.
You had a brilliant debugging session last week. Today you can’t find it, and starting fresh takes 30 minutes to rebuild context.
Your agent has been working for 40 messages. It contradicts a decision it made at message 5 — because that context was silently evicted.
You use Claude Code, ChatGPT, and Cursor. Each knows a fragment of your world. None of them can answer: “What did I decide about the auth architecture on Tuesday?”

These aren’t edge cases. They’re the default experience of working with AI in 2026.

Where We Are: The Memory Mechanisms of Major Agents

Let’s look at what the most-used coding agents actually do for memory today. Not marketing — the actual architecture.

Claude Code — File-Based Memory with Conversation Compaction

~/.claude/projects/<project-hash>/memory/
├── MEMORY.md          (index file, always loaded into context)
├── user_role.md       (who you are)
├── feedback_*.md      (behavioral corrections)
└── project_*.md       (project-specific context)

How it works: Markdown files with YAML frontmatter, organized by type (user, feedback, project, reference). MEMORY.md is injected into the system prompt every session — the agent sees its own memories from the first message. The agent can create/update memory files mid-session. Per-project scoped by directory hash.

Within a session: when context approaches limits, Claude Code runs conversation compaction — summarizing earlier turns while preserving key decisions and tool results. This is the within-session durability mechanism.

The architecture decision that matters: Memory is just files on disk. No database, no embedding index, no server process. This means:

Extremely low operational complexity (it’s literally cat + write)
Survives anything — no corruption, no migration, no schema changes
Human-readable and git-controllable

What breaks:

No semantic search — retrieval is “load the index file and hope it’s relevant.” If your memory grows to 50 entries, the agent either reads all of them (token cost) or reads the index and guesses which files matter (retrieval miss).
No cross-project memory — per-project scoping means lessons from project A are invisible in project B. You learn the same things repeatedly.
No decay — stale memories live forever at full weight. You manually prune or suffer pollution.
Compaction is lossy — when conversation context is summarized, the reasoning chain behind decisions disappears. The agent keeps the conclusion but loses the “why,” making it unable to re-evaluate when circumstances change.
In-band extraction competes with the main task — the agent decides what to remember while also trying to solve your problem. Memory management is a tax on every session’s token budget.

For the infra engineer: Think of this as a flat-file database. No indexes, no query planner, linear scan on read. Works at small scale, degrades predictably as it grows. The simplicity is a genuine strength — but it’s the same class of simplicity that made people move from CSV to SQLite.

OpenAI Codex CLI — AGENTS.md + Opt-In Memory System

~/.codex/
├── config.toml           (user preferences)
├── memories/             (auto-extracted cross-session context)
├── history.jsonl         (session transcripts)
└── AGENTS.md             (global instructions)

<project>/.codex/
├── config.toml           (project config)
├── environments/         (env definitions)
└── skills/               (reusable agent skills)

How it works: Two-layer context system.

Layer 1 — Instruction Chain (always active): At session start, Codex walks the directory tree from ~/.codex/AGENTS.md down to the current working directory, concatenating every AGENTS.md it finds. Later files override earlier ones. Capped at 32 KiB total. Rebuilt fresh every session — no stale cache.

Layer 2 — Memories (opt-in, disabled by default): When enabled (memories = true in config), Codex runs a background extraction pass after sessions go idle. It pulls preferences, conventions, workflows, and known pitfalls into ~/.codex/memories/. Future sessions get these auto-injected into the system prompt.

A sub-feature called Chronicle passively observes screen activity to build additional context. Session transcripts persist as JSONL at ~/.codex/history.jsonl.

The architecture decision that matters: Memory is opt-in and post-hoc. The extraction happens after the conversation, not during it. This means memory management never competes with the main task for tokens. The tradeoff: the agent can’t decide in the moment what’s worth remembering — that decision is made by a separate pass later.

What breaks:

Opt-in means most users don’t have it — default is no cross-session learning. Each task starts fresh unless you’ve explicitly enabled the feature.
No search over memories — memories are injected into context, not queried. As they grow, you hit the same “everything in prompt” problem.
No graph, no linking — memories are flat text blobs, not structured knowledge with relationships.
Chronicle is opaque — passive screen observation raises both privacy questions and quality concerns (what does it actually capture? how accurate is extraction from screen content?).
AGENTS.md is manual — the instruction chain is human-authored, not agent-learned. It’s documentation, not memory.

For the infra engineer: Think of this as a write-ahead log (history.jsonl) with a background ETL job (memory extraction) that materializes a flat view (memories/ directory) for downstream consumers (future sessions). No indexes on the WAL. No incremental extraction. The ETL runs full-batch.

OpenClaw (380k stars) — Multi-Backend Semantic Memory with Temporal Decay

OpenClaw is the highest-starred personal AI assistant. Its memory system is one of the most sophisticated native implementations.

Session persistence:

~/.openclaw/agents/<agentId>/sessions/
├── sessions.json          (metadata: timestamps, topics, status)
└── <sessionId>.jsonl      (full conversation transcript, append-only)

Sessions auto-reset daily at 4 AM or after configurable idle timeout. Full conversation history preserved on disk as append-only JSONL.

Memory search architecture (dual-path):

Vector embeddings — supports 10+ embedding providers (OpenAI, Gemini, local GGUF models, Ollama, Bedrock)
BM25 keyword search — traditional information retrieval alongside semantic
Weighted rank fusion — merges both result sets with configurable weights
Temporal decay — 30-day half-life. Older memories rank lower in results unless reinforced by access.
MMR diversity — Maximal Marginal Relevance ensures results aren’t all semantically redundant
Multimodal — images and audio indexed via Gemini embeddings

Context management (the compaction pipeline):

When contextTokens > contextWindow - reserveTokens:

Pre-compaction memory flush — durable state written to disk before summarization
Auto-compaction — LLM summarizes long conversations, creates persistent compaction entries in transcript
Session pruning — in-memory trimming of old tool results (non-destructive to disk)
Workspace re-injection — AGENTS.md, SOUL.md, TOOLS.md re-loaded after compaction

Memory backends:

Builtin: SQLite-based engine indexing MEMORY.md + memory/ tree
QMD (Query Memory Daemon): Local-first sidecar using BM25 + vector + llama.cpp reranking. Indexes workspace files, additional configured paths, and session transcripts. Falls back to builtin if unavailable.

What’s good:

Temporal decay is native — 30-day half-life means old memories naturally fade without manual intervention
Multi-cue retrieval (vector + BM25 + time weighting + MMR) is the most complete retrieval pipeline in any coding agent
Session transcripts are indexed and searchable — past conversations are queryable knowledge, not just archives
Local embedding support (GGUF, Ollama) — can run fully offline

What breaks:

Decay is time-based only — the 30-day half-life doesn’t consider access frequency or relevance reinforcement. A critical architectural decision from January decays just like a casual question from January.
No consolidation pipeline — raw sessions are searchable but never distilled into structured knowledge. The system finds old conversations; it doesn’t extract durable facts from them.
No graph layer — entities aren’t linked. “User → uses → Fastify” isn’t modeled as a relationship, just as text that happens to contain those words.
Compaction is irreversible within session — once context is summarized, the reasoning chain behind early decisions is gone for the remainder of that session.

For the infra engineer: This is the closest to a real database-backed system. Dual-index search (vector + BM25), configurable decay, local sidecar process (QMD) for search offloading. The architecture is what you’d design if you approached agent memory as an information retrieval problem — which is correct but incomplete. It retrieves well but doesn’t learn (no consolidation) or relate (no graph).

Hermes (198k stars) — Frozen Snapshot + Self-Improving Skills

Hermes (by Nous Research) is a self-improving agent that evolved from OpenClaw’s lineage. Its memory architecture has a distinctive “frozen snapshot” design.

Memory stores:

Store	Size limit	Content	Persistence
`MEMORY.md`	~2,200 chars (~800 tokens)	Environment facts, conventions, lessons, techniques	`~/.hermes/memories/`
`USER.md`	~1,375 chars (~500 tokens)	Name, role, timezone, preferences, pet peeves, workflow habits	`~/.hermes/memories/`
Session history	Unlimited	All past conversations, FTS5-indexed	`~/.hermes/state.db` (SQLite)
Skills	Unlimited	Reusable procedural memory, auto-created from complex tasks	`~/.hermes/skills/`

The “frozen snapshot” pattern (performance-critical insight):

At session start: MEMORY.md and USER.md contents are rendered into the system prompt as a frozen block
This frozen block never changes mid-session — it’s static text in the system prompt
Why: LLM prefix caching. If the system prompt is stable, the provider can cache the KV computation for the prefix. Changing it mid-session invalidates that cache and costs re-computation on every subsequent turn.
When the agent modifies memory during a session: changes persist to disk immediately but don’t appear in the active system prompt until next session
Tool responses always show live disk state — so the agent can verify its writes landed

This is an infra-native optimization. The frozen snapshot isn’t a limitation — it’s a deliberate choice to maximize prefix cache hit rates. The tradeoff: memories written in turn 5 of a session won’t influence the agent’s behavior until the next session. But the tokens saved on KV recomputation across 50+ turns are substantial.

Session search (FTS5):

Full-text indexed in SQLite — ~20ms query time, ~1ms scroll time
Returns actual messages from DB (no LLM summarization, no truncation)
Agent can scroll forward/backward inside any found session
Zero LLM cost for retrieval — pure text search on structured data

Skills (procedural memory):

Agent autonomously creates reusable skills after completing complex tasks
Skills self-improve during subsequent use
Background review process can patch skills silently between sessions
This is the only major agent that implements procedural memory — not just “what” and “why” but “how to do it again”

Memory tool actions: add, replace, remove — no explicit read since content is already in the system prompt.

Security: All memory writes are scanned for prompt injection, credential exfiltration, SSH backdoors, and invisible Unicode characters before acceptance. Exact duplicate entries are auto-rejected.

What’s good:

Prefix cache optimization is genuinely clever — most agents waste tokens by mutating context mid-session
FTS5 session search is fast and free (no embedding cost, no LLM cost)
Skills = procedural memory. The agent doesn’t just remember facts — it remembers how to do things
Hard character limits on MEMORY.md force consolidation — the agent must compress when it runs out of space
Write approval gate (opt-in) — users can require approval before any memory mutation

What breaks:

Character limits force premature eviction — 800 tokens for all environment knowledge means useful memories get deleted to make room. There’s no “archive” tier — when MEMORY.md is full, something must be removed.
No semantic search on memories — FTS5 is keyword-only. “What do I know about deployment patterns?” won’t find a memory about “CI/CD pipeline conventions” unless those exact words appear.
Frozen snapshot means stale in-session — if the agent learns something critical at turn 2, it can’t act on that memory until next session. For long sessions (100+ turns), this matters.
No temporal decay — memories persist at equal weight until explicitly removed. Old conventions compete with new ones.
Skills are opaque — auto-generated procedural knowledge that self-improves is powerful but hard to audit. If a skill develops a subtle bug, it propagates silently.

For the infra engineer: Think of this as a read-optimized store with write-back caching. The frozen snapshot is a consistency tradeoff — read performance (prefix caching) over write visibility (immediate effect). FTS5 session search is like having a fast log index — you can grep history efficiently but can’t do similarity queries. The skills system is the most interesting primitive — it’s essentially learned runbooks that the agent maintains itself.

Letta (MemGPT, 23k stars) — Agent-Managed Virtual Memory

How it works: Treats the context window like RAM. The agent explicitly manages what’s “paged in” vs stored externally. Direct analog to virtual memory in operating systems.

Three tiers:

Tier	OS Analogy	Agent control
Core Memory (blocks)	RAM pages	Agent explicitly reads/writes/edits. Always in context.
Recall Storage	Page cache	Agent searches past conversations by query or time range.
Archival Storage	Swap / disk	Agent inserts knowledge, searches by semantic similarity.

Agent memory tools:

core_memory_append(label, content)    — Add to a named block ("human", "persona")
core_memory_replace(label, old, new)  — Edit within a block (surgical update)
conversation_search(query, page)      — Search conversation history
conversation_search_date(start, end)  — Time-range search on history
archival_memory_insert(content)       — Store long-term knowledge
archival_memory_search(query, page)   — Retrieve by semantic similarity

The key insight: Memory management is an action the agent takes, not something the system does behind the scenes. The agent decides what’s important enough for core memory (expensive, always visible) vs archival (cheap, on-demand).

What’s good: Most sophisticated control model — the agent has true agency over its memory. Self-improving: it updates its own persona and knowledge blocks. The only system where the agent can explicitly say “I don’t need this in working memory anymore — page it out.”

What breaks:

Platform lock-in — requires running a Letta server (PostgreSQL-backed). No embedded mode, no “just link a library.”
Needs strong models — weaker models spend more tokens managing memory than doing actual work. The cognitive overhead of memory management is non-trivial.
No cross-tool support — single agent instance, no mechanism for sharing memory across different tools.
No principled decay — archival storage grows unbounded. Old knowledge never weakens.
Recall/archival search is basic vector similarity — no hybrid retrieval, no graph, no temporal weighting.

For the infra engineer: This is the virtual memory / page table model applied literally. Core memory = hot pages pinned in RAM. Archival = cold storage on disk. The page replacement policy is model-driven rather than LRU/LFU — the agent’s judgment replaces the kernel’s eviction heuristics. Elegant in theory, but the “model as page replacement algorithm” has higher overhead than hardware-based approaches.

Mem0 (25k+ stars) — Bolt-On Memory Layer

How it works: A framework-agnostic memory service with a 3-call API: add(), search(), get_all(). Under the hood, an 8-phase pipeline handles extraction, embedding, deduplication, entity linking, and hybrid retrieval.

The pipeline (what actually happens on add()):

Retrieve last 10 messages from session scope (session context)
Embed new messages, search vector store for top-10 similar existing memories (collision detection)
LLM extraction — single call with ~900-line prompt → structured facts with attributed_to and linked_memory_ids
Batch embed all extracted memory texts
MD5 dedup (skip exact hash matches)
Persist to vector store (27 backend options: Qdrant, Pinecone, pgvector, etc.)
Entity linking — extract named entities → separate collection with back-links
Save messages to SQLite for sliding window context

Search (hybrid scoring): BM25 (keyword, sigmoid-normalized) + semantic similarity (vector) + entity boost (graph-like entity linking). Configurable weights, optional reranking.

What’s good: Lowest integration friction of any memory system — genuinely 3 lines to add memory to an existing agent. 27 vector backends means you can run it against whatever infra you already have. Framework-agnostic design.

What breaks:

Append-only — no forgetting, no decay. Paid platform has decay; OSS does not.
No working memory management — can’t control what’s active in context, just what’s stored.
Black-box extraction — the 900-line prompt decides what’s worth remembering. You can’t easily tune this.
LLM required on every ingest — no offline extraction mode. Each add() call costs an LLM call.
No consolidation — related memories never merge. Ten mentions of “uses TypeScript” become ten separate entries, not one strengthened fact.

For the infra engineer: This is an ETL pipeline with a vector warehouse as the sink. Source = conversations. Transform = LLM extraction + embedding. Load = vector store. Retrieval = hybrid search over the warehouse. Clean architecture, well-composed. The gap: no lifecycle management on the warehouse side — data goes in and stays forever at full weight.

The Capability Map: What Each System Bet On

What Each System Bet On

Every system made a different bet on what matters most. None covers the full problem.

System	What it bet on	Episodic	Extraction	Graph	Vector	Decay	Multi-Cue	Consolidation
Claude Code	Simplicity (zero deps)	-	Manual	-	-	-	-	-
Codex CLI	Post-hoc learning	✓	Opt-in	-	-	-	-	-
Hermes	Performance (prefix cache)	✓	Agent-driven	-	-	-	Keyword	Skills
OpenClaw	Retrieval quality	✓	Agent-driven	-	✓	Time-only	✓	-
Mem0	Integration simplicity	-	✓	Partial	✓	-	✓	-
Letta/MemGPT	Agent control	✓	✓	-	✓	-	Partial	-
Hindsight	Biological fidelity	✓	✓	✓	✓	-	✓	Manual
Graphiti (Zep)	Relationships over time	✓	✓	✓	✓	-	✓	Partial

The pattern: Every row has a different strength. The Decay column is almost entirely empty — regardless of what each system prioritized, none invested in forgetting. OpenClaw’s 30-day half-life is the only native decay in a major agent, and it’s time-based only (a critical decision from January fades at the same rate as a trivial question from January).

This isn’t a ranking — each system’s bet is defensible for its use case. The insight is what nobody bet on.

Why It Doesn’t Work: The 5 Structural Failures

1. Append-Only is the Default

Almost every system only adds memories. Nothing fades, nothing gets pruned, nothing weakens. After 6 months of daily use, you have 10,000 stored facts. 8,000 are stale. Retrieval quality is now worse than having no memory.

Human memory solves this: the Ebbinghaus forgetting curve naturally decays memories that aren’t reinforced. Your agents have no equivalent.

2. No Working Memory Control

Your brain holds ~7 items in active working memory. It decides what to load. Current agents mostly either:

Dump everything into context (overflow)
Retrieve reactively with no control (miss relevant context)
Let the system decide (black box, wrong items loaded)

Letta gives agents explicit page-in/page-out control. Hermes takes a different approach — hard character limits on MEMORY.md force the agent to consolidate (overflow → error → agent must compress). OpenClaw uses auto-compaction with pre-flush. But none of these provide true working memory selection — the ability to load specific relevant context for the current task from a larger pool.

3. Retrieval is One-Dimensional

Most systems do vector similarity search — “find text that looks like my query.” But human recall works via spreading activation: multiple cues converge. You remember something because of time + place + emotion + association, not because it’s semantically similar to a query string.

Production memory needs: keyword + embedding + graph traversal + temporal + decay scoring — simultaneously.

4. Every Agent is a Memory Island

You use 3-5 AI tools daily. Each has its own walled memory garden. Insights from Claude Code don’t reach ChatGPT. Decisions made in Cursor don’t inform Gemini. There’s no memory interchange format, no portability standard, no unified search.

MCP standardized tool access. A2A standardized agent communication. Memory has no equivalent protocol.

5. No Consolidation (Episodes → Knowledge)

Humans don’t store raw experiences forever. During sleep, the hippocampus replays episodes while the neocortex extracts patterns. Raw events decay; distilled knowledge persists.

Most agent memory is either:

Raw transcripts (complete but unsearchable at scale)
Extracted facts (lossy, no provenance back to source)
Never both with a bridge between them

What Would an Ideal Agent Memory Look Like?

Based on neuroscience, research papers (FadeMem, AgeMem, ActiveMem, OBLIVION — all 2026), and the gaps in every existing system:

Five-Layer Brain-Inspired Architecture

Five biological properties it must have:

Selective encoding — Don’t store everything. Encode what’s novel, salient, or contradicts existing knowledge. Discard routine chatter.
Cue-based retrieval — Not “memory #4521” but “that conversation about Terraform… last Tuesday… where I was debugging the ECS deploy.” Multiple signals converge.
Offline consolidation — Background process replays episodes, extracts durable facts, builds graph connections. Not on the critical path.
Reconsolidation on recall — Accessing a memory updates its strength. May trigger merges with new information. Memories are living, not frozen.
Decay as a feature — Memories that aren’t accessed lose strength. Prevents retrieval pollution. The system naturally focuses on what matters.

What Would an Ideal Personal Second Brain Look Like?

The agent memory problem and the personal knowledge problem converge. What you actually want:

A system that indexes every AI conversation you’ve ever had — across every tool — and makes that knowledge searchable, connected, and self-maintaining.

Your Second Brain

Why this doesn’t exist yet:

No capture standard — Each AI tool stores conversations differently. No export protocol exists. Browser extensions break on DOM changes.
No memory database — No production-grade DB combines episodic log + semantic store + graph + vector + native decay. Graphiti gets closest (6/7) but no decay.
LLM dependency — Extraction and consolidation need LLM calls. Fully offline operation requires local models that are only now becoming viable.
Security — Cross-tool memory creates a new attack surface. XTHP attacks (NDSS 2026) show 75% of real-world tools are vulnerable to memory poisoning.
Scale — Thousands of sessions × millions of tokens × multi-modal (images, PDFs, code). The data volume is non-trivial.

The Gap: What Needs to Be Built

What exists	What’s missing
Memory libraries (Mem0, A-MEM)	Memory database with decay as primitive
Per-tool memory (Claude Code files, ChatGPT memory)	Cross-tool unified memory
Append-only stores	Principled forgetting
Vector-only retrieval	Multi-cue spreading activation
On-demand extraction	Background consolidation
Tool-specific formats	Memory portability standard
Raw storage OR extracted facts	Both with provenance bridge

What’s Coming Next (This Series)

This overview is Part 0. The series goes deep on each dimension:

Why Your Agent Forgets — The 4 failure modes in detail
The Memory Landscape — 10+ approaches scored on 7 dimensions
The Forgetting Problem — Why append-only breaks and what decay engines exist
The Cross-Tool Gap — Why agents are siloed and what’s emerging
Brain-Inspired Architecture — The 5-layer target system and the missing database

The tools we use daily are getting smarter at reasoning, planning, and coding. But they still can’t remember what we told them yesterday. That’s the gap.

#AgenticAI #AIMemory #SecondBrain #AIInfra #LLM #AgentArchitecture