Agent Memory Deep Dive #4: The Cross-Tool Memory Gap — Why Agents Are Siloed
You use 5 AI tools daily. Each one thinks it’s your only assistant.
TLDR: Every AI tool has its own memory island. Claude Code knows your project preferences. ChatGPT knows your writing style. Cursor knows your codebase. Gemini knows your research questions. None share. The problem isn’t technical (we know how to build unified stores). It’s structural: no capture standard, no interchange format, no incentive for vendors to interoperate. Three projects are trying (Memorix for coding agents, VESTI for web chatbots, Portable Agent Memory for the protocol). None bridge both worlds. This is the “email before SMTP” moment.
The Daily Reality
Here’s someone’s actual Tuesday:
Morning: Claude Code session — debugging a Terraform module. Discover the issue is a provider version incompatibility. Spend 30 minutes finding the fix. Session ends.
Midday: ChatGPT conversation — researching different approaches to blue-green deployments for the same infrastructure. Build mental model. Learn three patterns I didn’t know.
Afternoon: Back in Claude Code — new session, same project. Starting the deployment automation. I need the Terraform findings from this morning AND the deployment patterns from ChatGPT. Neither agent knows about the other session. I re-explain everything.
Evening: Cursor — writing the actual code. I’ve now explained the same context three times today.
This isn’t unique. Anyone using multiple AI tools lives this. The aggregate knowledge from all your interactions is fragmented across walled gardens that will never talk to each other unless something changes.
What Each Tool Knows (And Doesn’t)
| Tool | What it stores | What it misses |
|---|---|---|
| Claude Code | Per-project memory (CLAUDE.md + memory files), session transcripts in JSONL | Other projects, non-code conversations, ChatGPT/Gemini context |
| ChatGPT | ~50 extracted facts (“User likes dark mode”), full conversation history in account | No cross-tool, can’t import, limited extraction quality |
| Gemini | Conversation history in Google account | No persistent memory extraction, no cross-tool |
| Cursor | Per-project rules (.cursorrules), codebase embeddings | Only code context, no conversational memory |
| Codex CLI | AGENTS.md instructions, opt-in Memories (background-extracted), session history in JSONL | Memories disabled by default, no search over history, no cross-tool |
| OpenClaw | Session transcripts, semantic memory (vector+BM25), temporal decay | No cross-tool, local to one agent instance |
| Hermes | MEMORY.md, USER.md, FTS5-indexed sessions, self-improving skills | Hard size limits, no cross-tool, keyword search only |
| Claude.ai (web) | Conversation history, project context | No connection to Claude Code sessions |
The information loss: Across 5 tools × 3 sessions/day × 250 working days = ~3,750 AI interactions/year. The knowledge created in those interactions — debugging solutions, architectural decisions, learning progressions, preference patterns — exists in 5 disconnected silos. No unified view. No cross-reference. No way to ask “what have I learned about X across all my tools?”
Why This Is Architecturally Hard
It’s tempting to say “just build an aggregator.” But the engineering challenges are real:
1. No Capture Standard
Each tool stores conversations differently:
- Claude Code: JSONL files in
~/.claude/with structured messages, tool calls, and results - ChatGPT: REST API accessible via session token (internal, undocumented, breaks periodically)
- Gemini: Google account data, accessible via Takeout or undocumented API
- Cursor: VS Code extension storage, conversation format undocumented
- Claude.ai: Web app state, no export mechanism
There’s no protocol for “give me my conversation history in a standard format.” Every adapter is bespoke, fragile, and subject to breaking changes.
What we learned from chatgpt-exporter: The project has maintained stable capture from ChatGPT for 2.5 years. Only 2 API breaks in that entire period. The internal REST API pattern (reading the same endpoints the web UI uses) is more stable than DOM scraping. But it still requires reverse-engineering undocumented APIs.
2. Different Extraction Quality
Even if you can capture raw conversations, the quality of what’s captured varies wildly:
- Claude Code sessions contain full tool inputs/outputs, file contents, git diffs — extremely rich
- ChatGPT conversations are message text only — no structured metadata about what was done
- Coding agent sessions have clear artifact boundaries (this file was created, this test was run)
- Research conversations have diffuse knowledge (insights spread across 50 messages)
A unified extraction pipeline can’t treat all sources equally. The information density and structure varies by source.
3. Privacy and Security
Session transcripts contain:
- Source code (potentially proprietary)
- API keys accidentally pasted
- Internal architecture details
- Personal information
- Business logic and strategies
This data cannot go to a cloud service for most users. Any cross-tool memory system must be local-first. This constrains the architecture — you can’t rely on cloud-hosted LLMs for extraction unless you’re comfortable sending your transcripts to them.
4. Scale
The math: 5 sessions/day × 50k tokens average per session × 365 days = ~91M tokens/year of raw conversation data. That’s before extraction, embeddings, or graph construction.
Storage for raw text is cheap. But vector embeddings, graph nodes, and LLM-based extraction all have non-trivial costs at this scale. And retrieval over a growing corpus must stay fast (<100ms) to be usable.
5. The XTHP Attack Surface
From NDSS 2026 (Li et al.): 75% of real-world memory-enabled tools are vulnerable to cross-tool memory poisoning.
The attack: Poisoned data from one tool (e.g., a malicious comment in a code file opened in Cursor) gets stored in shared memory. Later, when another tool (Claude Code) queries memory, the poisoned entry redirects its behavior.
Example:
- Attacker embeds hidden instruction in a code comment
- Cursor reads the file, stores the instruction as a “project convention”
- Cross-tool memory surfaces this “convention” to Claude Code
- Claude Code follows the instruction, thinking it’s a legitimate user preference
Cross-tool memory creates a new attack surface that doesn’t exist in siloed systems. Any solution must have provenance (which source contributed this memory?) and trust scoring (how much do I trust memories from each source?).
What Exists Today
Memorix (514 stars) — Cross-Agent Memory for Coding Tools
The only system explicitly designed for cross-tool memory. Uses MCP protocol for standardized access.
Supported agents:
- Core: Claude Code, Cursor, Windsurf (full integration)
- Extended: GitHub Copilot, Kiro, Codex (with caveats)
- Community: Gemini CLI, OpenCode (best-effort)
How it works: An MCP server (stdio or HTTP mode) that any MCP-compatible agent can connect to. Memories are stored locally in SQLite + Orama search index. Project-scoped by default, optional global scope.
The observation/reasoning split: Stores “what” separately from “why.” Observations (“the auth module uses JWT”) are factual. Reasoning (“we chose JWT because session cookies don’t work well with microservices”) captures decision rationale. Different queries route to different substrates.
Limitation: Coding agents only. Doesn’t capture web chatbot conversations (ChatGPT, Claude.ai, Gemini web). Doesn’t handle non-code knowledge. The hardest gap — bridging conversational AI (web) with operational AI (coding) — remains untouched.
VESTI (Gold Award, 2026 AI Hackathon) — Browser Extension for Web Chatbots
Chrome extension that captures conversations from ChatGPT, Claude.ai, and Gemini via browser integration.
Approach: Reads the same REST APIs that the web UIs use (the chatgpt-exporter pattern). Extracts and indexes conversations locally. Provides cross-chatbot search.
Limitation: Web chatbots only. Doesn’t cover coding agents (Claude Code, Cursor, Codex). Doesn’t have the memory sophistication (extraction, graph, decay) of dedicated memory systems. It’s a capture + search tool, not a memory intelligence layer.
Portable Agent Memory (Ravindran, 2026) — The Protocol Attempt
An academic paper with working code (Apache 2.0, 54 tests) proposing a standard for agent memory portability.
5-component model:
- Memory encoding (how facts are represented)
- Storage abstraction (pluggable backends)
- Retrieval interface (standardized query)
- Provenance chain (Merkle-DAG — any fact traces to its source)
- Injection-resistant rehydration (safely loading memory into new contexts)
The Merkle-DAG provenance: Every memory node has a cryptographic hash chain linking it to the conversation that produced it. You can always answer “where did this come from?” and verify it hasn’t been tampered with. This directly addresses the XTHP trust problem.
Limitation: Academic work. The standard needs adoption to matter, and there’s no industry momentum behind it (no vendor has incentive to adopt an interoperability standard for memory — their lock-in is the moat).
The 7 Gaps Nobody Has Closed
| # | Gap | What it means | Who’s closest |
|---|---|---|---|
| 1 | Cross-tool memory | Unified index across web chatbots + coding agents | Nobody (Memorix = coding only, VESTI = web only) |
| 2 | Working + long-term unified | Active context management + persistent knowledge in one system | Letta (full control model), OpenClaw (compaction + search) |
| 3 | Principled forgetting | Strength-based decay, not append-only | OpenClaw (30-day half-life, time-only), nobody does access-frequency-weighted |
| 4 | Offline/local-first | No LLM dependency for storage or retrieval | Claude Code files (but no search sophistication) |
| 5 | Memory portability | Standard format, import/export, migration between systems | Portable Agent Memory (paper, no adoption) |
| 6 | Multi-modal memory | Images, PDFs, diagrams, code files alongside text | Nobody |
| 7 | Trust and provenance | Which conversation produced this fact? Is it tampered? | Memoir (git-like blame), Portable Agent Memory (Merkle-DAG) |
Gap 1 is the killer. Not because it’s the hardest technically — it’s the hardest economically. OpenAI has no incentive to make ChatGPT memories portable. Anthropic has no incentive to index Claude conversations into a competitor-compatible format. Memory lock-in is strategic.
The solution has to come from the user side: a third-party system that captures from multiple sources without vendor cooperation. Like how email clients work — they read your mail from any provider via IMAP, regardless of whether the provider wants interoperability.
The “SMTP for Memory” Problem
MCP standardized how agents access tools. Before MCP, every framework had its own tool protocol. After MCP, one standard.
A2A standardized how agents talk to each other. Before A2A, no interoperability between agents from different vendors.
Memory has no equivalent standard. No “MCP for memory” exists. Each system has its own:
- Storage format (vector IDs, graph nodes, markdown files, SQLite rows)
- Query interface (API calls, MCP tools, file reads)
- Metadata schema (timestamps, strength, provenance, categories)
- Lifecycle model (append-only, TTL, archive, decay)
Until there’s a standard, every tool remains an island. Users who want unified memory must build their own bridge — or wait for one of the emerging projects to mature enough to be that bridge.
What Practitioners Can Do Today
Minimal viable cross-tool memory (the manual approach):
- Use Memorix for coding agent memory (Claude Code, Cursor, Codex share context)
- Use Claude Code’s built-in memory for project-specific knowledge
- Export ChatGPT conversations periodically (chatgpt-exporter) for personal archive
- Accept the gap: web chatbot knowledge stays siloed for now
If you’re building a product in this space:
The winning product captures from everywhere and searches across everything. The moat is breadth of capture + quality of extraction. Specifically:
- Web chatbots via browser extension (VESTI pattern)
- Coding agents via MCP server (Memorix pattern)
- CLI sessions via transcript ingestion (Claude Code JSONL files)
- Unified local index with multi-cue retrieval
- Provenance on every fact (source tool, timestamp, conversation ID)
The user should be able to ask: “What do I know about Kubernetes pod scheduling?” and get answers sourced from a ChatGPT research session, a Claude Code debugging session, and a Cursor code review — merged, ranked by recency and strength, with links back to each source.
That product doesn’t exist. But the pieces do.
Previous: Agent Memory Deep Dive #3: The Forgetting Problem Next: Agent Memory Deep Dive #5: Brain-Inspired Architecture