Two Real Problems in the Agent Domain

Problem A: Multi-Session Orchestration & Persistent Memory

The Problem

I run multiple AI sessions simultaneously across providers:

• Claude Code (CLI sessions, multiple parallel)
• ChatGPT (web conversations)
• Gemini (web + API)
• Other tools (Cursor, Copilot, custom agents)

Pain points:

1. Sessions are siloed. Each provider has its own conversation history. No unified view of “what am I working on across all my AI assistants.”

2. Sessions are ephemeral. Old sessions disappear or become unfindable. A great debugging session from 3 weeks ago? Gone. The research conversation that led to an architectural decision? Buried in pagination.

3. No cross-session memory. Claude Code has per-project memory (CLAUDE.md + memory files). ChatGPT has limited memory. Gemini has none. Nothing connects them — insights from one session don’t flow to others.

4. No orchestration layer. I can’t:

   • Search across all sessions (“when did I solve that DynamoDB throughput issue?”)
   • Resume a session that was lost to context limits
   • Route a task to the best provider for it
   • Track which sessions are active, which are stale, which produced useful artifacts

5. Knowledge decay. Over time, the accumulated knowledge from hundreds of AI interactions just… evaporates. Each new session starts from zero (or near-zero with basic memory features).

What Would a Solution Look Like?

Core idea: A meta-layer that sits above all AI providers and provides:

┌────────────────────────────────────────────────────────────┐
│                    Orchestration Layer                       │
│                                                            │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │
│  │ Session      │  │ Long-Term    │  │ Router /     │    │
│  │ Registry     │  │ Memory       │  │ Dispatcher   │    │
│  │              │  │              │  │              │    │
│  │ All sessions │  │ Cross-session│  │ "Best tool   │    │
│  │ All providers│  │ knowledge    │  │  for this    │    │
│  │ Full history │  │ Searchable   │  │  task"       │    │
│  └──────────────┘  └──────────────┘  └──────────────┘    │
│                                                            │
│  ┌──────────────┐  ┌──────────────┐                       │
│  │ Artifact     │  │ Session      │                       │
│  │ Store        │  │ Resumption   │                       │
│  │              │  │              │                       │
│  │ Code, docs,  │  │ Replay from  │                       │
│  │ decisions    │  │ event log    │                       │
│  └──────────────┘  └──────────────┘                       │
└────────────────────────────────────────────────────────────┘
         │                │                │
    ┌────▼────┐     ┌────▼────┐     ┌────▼────┐
    │ Claude  │     │ ChatGPT │     │ Gemini  │
    │ Code    │     │         │     │         │
    └─────────┘     └─────────┘     └─────────┘

Key components:

1. Session Registry — Unified index of every AI session across every provider. Metadata: provider, start time, topic, status (active/stale/archived), key artifacts produced.

2. Long-Term Memory Store — Structured knowledge extracted from sessions:

   • Decisions made and rationale
   • Patterns discovered
   • Debugging solutions (problem → fix mapping)
   • Architectural knowledge
   • Searchable via semantic similarity AND keyword

3. Session Ingestion — Adapters per provider:

   • Claude Code: read session transcripts (JSONL files in ~/.claude/)
   • ChatGPT: export via API or browser extension
   • Gemini: export via API
   • Custom agents: event log protocol

4. Knowledge Extraction Pipeline — After each session (or periodically), extract:

   • What was accomplished?
   • What was learned?
   • What artifacts were produced?
   • What decisions were made (and why)?

5. Retrieval Interface — “What do I know about X?” searches across ALL past sessions and returns relevant context, regardless of which provider/session it came from.

Existing Partial Solutions

Tool	What it does	What’s missing
Claude Code memory (CLAUDE.md)	Per-project persistent memory	Single-provider, manual, no search across sessions
ChatGPT Memory	Stores facts about user	Extremely limited, no structured retrieval
Mem0	Cross-session memory layer	Requires integration, no session management
Khoj	Personal AI with memory	Focused on personal knowledge, not session orchestration
Pieces	Code snippet + context manager	Not built for full session orchestration

Design Constraints

• Must work locally (privacy — session transcripts contain sensitive code)
• Must be provider-agnostic (no vendor lock-in for the meta-layer itself)
• Must support passive ingestion (I don’t want to manually tag/export every session)
• Must scale to thousands of sessions over months/years
• Retrieval must be fast — if it takes 10 seconds to search, I won’t use it

---

Problem B: No Production-Grade Open-Source Agent Framework

The Problem

After researching dozens of agent frameworks (LangGraph, CrewAI, AutoGen, AgentScope, Semantic Kernel, Mastra, etc.), none handles all production-critical components well enough:

What production agents need:

Component	What it does	Who does it well?
Orchestration	Multi-agent routing, delegation, workflow	LangGraph (graph-based), AgentScope (pipeline)
Durability	Crash recovery, checkpoint/resume, event sourcing	Temporal/Restate (but not agent-native)
Memory	Long-term, cross-session, semantic search	Mem0, Zep (standalone, not integrated)
Sandboxing	Isolated tool execution, security	E2B, Modal (standalone services)
Compute	Scale-to-zero, multi-cloud, cost management	Nobody — manual Docker/K8s/Lambda
Observability	Token costs, latency, traces, debugging	Langfuse, Phoenix (bolt-on)

The core issue: Each framework excels at 1-2 of these but is weak or missing the rest.

AgentScope Analysis

AgentScope (Alibaba/ModelScope) is the closest to a complete solution:

• Multi-agent messaging and pipelines
• Memory (with Mem0/REME integration)
• Tool execution
• MCP + A2A protocol support
• RAG integration
• Tracing

But: It’s tightly coupled to Alibaba Cloud ecosystem:

• DashScope models as default
• OSS (Alibaba Object Storage) for artifacts
• Designed for Chinese cloud infrastructure patterns
• Community and docs are China-focused
• Running it on AWS/GCP requires significant adaptation

What I Tried: clouda-oss

I designed clouda-oss to solve this — a framework where:

• Every infrastructure concern is behind an interface (ABC)
• Same agent code runs on laptop → Docker → AWS → GCP → K8s
• Pluggable backends selected by YAML config
• Full production stack: events, memory, bus, compute, sandbox, credentials

Current state (June 2026):

• Core framework is implemented (575 Python tests passing)
• UI is built (778 TS tests passing)
• AWS bridge works (DynamoDB, S3, Bedrock, Lambda sandbox, OpenSearch)
• Local development flow works (clouda dev)

What’s hard:

1. Scope is enormous. Covering all the production components across all deployment targets is a multi-year effort for a solo developer.
2. The “last mile” problem. Each deployment target (Lambda, Fargate, CloudRun, K8s) has unique constraints that leak through abstractions.
3. Moving target. The AI landscape changes monthly — new protocols (MCP, A2A), new patterns (computer use, browser agents), new models with different capabilities.
4. Community chicken-and-egg. Without users, hard to prioritize. Without features, hard to attract users.

Gap Analysis: What’s Still Missing in clouda-oss

Component	Status	Effort to Complete
ComputeBackend abstraction	Not started	Medium — interface is designed, needs implementation
Checkpoint/Resume	Not started	High — deep integration with event store
CredentialStore	Not started	Medium — well-scoped
CronScheduler	Not started	Medium
SkillStore + learning	Not started	High — extraction is the hard part
Distributed orchestrator	Not started	High — need real message bus routing
GCP bridge	Not started	Medium — mirror AWS bridge patterns
K8s bridge	Not started	Medium-High

---

Connecting the Two Problems

These aren’t independent problems. They’re the same problem at different scales:

• Problem A (personal) = “I need an orchestration layer for MY AI sessions”
• Problem B (platform) = “Developers need a production runtime for THEIR agents”

Insight: The meta-layer from Problem A IS a special case of the platform from Problem B:

• Session Registry = EventStore (tracking all sessions)
• Long-Term Memory = MemoryStore (cross-session knowledge)
• Session Ingestion = adapters that produce Events from external providers
• Knowledge Extraction = post-session hooks (skill extraction, memory consolidation)
• Retrieval Interface = Memory search + session search

The path: Solve Problem A as a vertical slice of Problem B. Build the personal orchestration layer USING clouda-oss primitives, proving them in daily use. The personal tool becomes:

1. A dogfooding vehicle for the framework
2. A demo/showcase for potential users
3. The first “agent” built on the platform (a meta-agent that manages other AI interactions)

---

Next Steps

For Problem A (Personal Orchestration)

1. Build a minimal session ingestion pipeline for Claude Code transcripts (~/.claude/ JSONL files)
2. Add a local memory store (SQLite + vector embeddings) for cross-session knowledge
3. Build a simple CLI/TUI to search across all past sessions
4. Add ChatGPT/Gemini ingestion (export-based initially)

For Problem B (Framework Completion)

1. Prioritize ComputeBackend + Checkpoint/Resume (enables the “invincible agent” story)
2. Ship a working clouda deploy for Docker Compose (lowest friction path to multi-machine)
3. Write one compelling end-to-end example that shows the full stack
4. Consider: focus on the LOCAL + DOCKER tier first, defer cloud bridges until there’s demand

For the Post Series

• Document these as real problems practitioners face
• Show the thought process: how personal pain leads to platform thinking
• Connect to the broader “agent infrastructure” theme (AI Infra pillar)