Agent Scaling

A single agent session is cheap — 1-2 CPU cores, 2 GB RAM, no GPU. But at 10,000 concurrent sessions, that’s 20,000 cores and 20 TB of RAM if naively allocated. Agent scaling is about exploiting the bursty nature of agent workloads to serve many sessions on fewer resources.

---

1. The Burstiness Advantage

Agent sessions spend most of their time waiting — waiting for the LLM to generate, waiting for the user to respond:

Timeline for one agent session:
  |--LLM call (GPU)--|--execute code (CPU)--|--wait for user--|--LLM--|--exec--|--wait--|
  |    2s active     |     0.5s active      |   30s idle      |  1s   | 0.3s   | 60s    |
  
  Active: ~4s out of ~94s total = 4.3% CPU utilization per session

This means 1 CPU core can handle ~20 sessions with reasonable overcommit. The key is managing the 4.3% active time across sessions without collisions.

---

2. Sandbox Pooling

Creating a sandbox per request is slow (200ms-1s). A warm pool keeps pre-created sandboxes ready:

class SandboxPool:
    """Pre-warm pool of ready-to-use sandboxes."""
    
    def __init__(self, provider, pool_size=50, min_ready=10):
        self.provider = provider      # Container/Firecracker/gVisor
        self.pool_size = pool_size
        self.min_ready = min_ready
        self.ready = asyncio.Queue()  # Pre-created, idle sandboxes
        self.active = {}              # session_id → sandbox
        
    async def start(self):
        """Pre-warm the pool."""
        for _ in range(self.pool_size):
            sandbox = await self.provider.create()
            await self.ready.put(sandbox)
        
        # Background task: keep pool topped up
        asyncio.create_task(self._replenish_loop())
    
    async def claim(self, session_id: str) -> Sandbox:
        """Get a ready sandbox. <10ms if pool has availability."""
        try:
            sandbox = self.ready.get_nowait()
        except asyncio.QueueEmpty:
            # Pool exhausted — create on-demand (slower)
            sandbox = await self.provider.create()
            logger.warning("Pool exhausted — cold-starting sandbox")
        
        self.active[session_id] = sandbox
        return sandbox
    
    async def release(self, session_id: str):
        """Return sandbox to pool (reset state) or destroy."""
        sandbox = self.active.pop(session_id, None)
        if sandbox is None:
            return
        
        if await self._can_recycle(sandbox):
            await sandbox.reset()  # Clean filesystem, kill processes
            await self.ready.put(sandbox)
        else:
            await sandbox.destroy()  # Too dirty to recycle
    
    async def _replenish_loop(self):
        """Keep at least min_ready sandboxes in the pool."""
        while True:
            while self.ready.qsize() < self.min_ready:
                sandbox = await self.provider.create()
                await self.ready.put(sandbox)
            await asyncio.sleep(1)
    
    async def _can_recycle(self, sandbox) -> bool:
        """Check if sandbox is clean enough to reuse."""
        # Don't recycle if: too much disk used, modified system files, etc.
        disk_usage = await sandbox.get_disk_usage()
        return disk_usage < 500_000_000  # < 500 MB

Pool Sizing

# How many sandboxes to keep warm?
def calculate_pool_size(
    peak_requests_per_sec: float,
    avg_session_duration_sec: float,
    cold_start_latency_sec: float,
    target_cold_start_rate: float = 0.01,  # 1% cold starts acceptable
) -> int:
    """Calculate warm pool size for target cold-start rate."""
    # Average concurrent sessions
    concurrent = peak_requests_per_sec * avg_session_duration_sec
    
    # Pool should cover burst above average
    # With Poisson arrivals, P(burst > pool) < target
    import scipy.stats
    pool = int(scipy.stats.poisson.ppf(1 - target_cold_start_rate, concurrent))
    
    return pool

# Example: 10 new sessions/sec, average 60s each
# Concurrent: 600 sessions
# Pool for 1% cold starts: ~625 sandboxes

---

3. CPU Overcommit Strategy

Not all sessions are active simultaneously. Overcommit CPU allocation:

Overcommit ratio	Effect	When to use
1× (no overcommit)	Each session gets dedicated CPU	Latency-critical
4×	4 sessions share 1 CPU core	Interactive agents (human-paced)
8×	8 sessions share 1 CPU core	Mostly-idle sessions (long think times)
16×	16 sessions per core	Batch agents (can tolerate delays)

# Kubernetes resource configuration for agent pods
agent_pod_resources = {
    "requests": {
        "cpu": "250m",      # Request 0.25 CPU (allows 4× overcommit)
        "memory": "512Mi",  # Memory can't be overcommitted (OOM risk)
    },
    "limits": {
        "cpu": "2000m",     # Can burst to 2 CPUs when active
        "memory": "2Gi",    # Hard memory limit
    },
}

# On a 64-core node:
# With requests: 64 / 0.25 = 256 sessions per node
# With limits: 64 / 2.0 = 32 sessions can burst simultaneously
# This works because only ~12% of sessions burst at any time

Monitoring Overcommit Health

# Alert if CPU contention causes latency spikes
overcommit_metrics = {
    "cpu_throttle_rate": pct_pods_throttled,  # Should be <5%
    "execution_latency_p99": p99_code_exec,    # Should be <2× baseline
    "scheduler_wait_ms": avg_scheduler_wait,   # Should be <100ms
}

# If throttle rate > 10%, reduce overcommit ratio
if cpu_throttle_rate > 0.10:
    alert("CPU overcommit too aggressive — reduce sessions per node")

---

4. Warm-Start Optimization

The biggest latency cost: installing packages inside the sandbox.

Pre-Built Images

# Base image with common packages pre-installed
FROM python:3.12-slim

# Pre-install the top-50 most-requested packages
RUN pip install --no-cache-dir \
    numpy pandas matplotlib scikit-learn \
    requests beautifulsoup4 flask fastapi \
    torch transformers datasets \
    pytest black ruff mypy

# Pre-install Node.js ecosystem
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - && \
    apt-get install -y nodejs && \
    npm install -g typescript ts-node eslint prettier

Layered Image Strategy

Base layer:     OS + Python + Node.js            (shared, ~500 MB)
Common layer:   Top-50 packages                   (shared, ~2 GB)
Workspace:      User files + session packages     (per-session, <500 MB)

With overlay filesystems, the base and common layers are shared across all sandboxes on a host — 100 sandboxes use ~2.5 GB shared + 100 × 500 MB per-session = 52.5 GB total (not 250 GB).

Package Cache

# Shared package cache across sandboxes
# Mount a read-only pip cache directory
container_config = {
    "volumes": {
        "/cache/pip": {"bind": "/root/.cache/pip", "mode": "ro"},
        "/cache/npm": {"bind": "/root/.npm", "mode": "ro"},
    }
}
# First install on any sandbox populates the cache
# Subsequent sandboxes get instant "installs" from cache

---

5. Autoscaling Architecture

class AgentAutoscaler:
    """Scale sandbox infrastructure based on demand."""
    
    def __init__(self, k8s_client, metrics_client):
        self.k8s = k8s_client
        self.metrics = metrics_client
    
    async def evaluate(self):
        """Check scaling conditions every 10 seconds."""
        pool_utilization = await self.metrics.get("sandbox_pool_utilization")
        queue_depth = await self.metrics.get("sandbox_claim_queue_depth")
        cold_start_rate = await self.metrics.get("sandbox_cold_start_rate")
        
        current_nodes = await self.k8s.get_node_count("sandbox-pool")
        
        # Scale up triggers
        if pool_utilization > 0.80 or queue_depth > 10 or cold_start_rate > 0.05:
            target = min(current_nodes + self._scale_up_increment(queue_depth), MAX_NODES)
            await self.k8s.scale_nodes("sandbox-pool", target)
            logger.info(f"Scaling up: {current_nodes} → {target} nodes")
        
        # Scale down triggers (conservative)
        elif pool_utilization < 0.30 and queue_depth == 0:
            target = max(current_nodes - 1, MIN_NODES)
            await self.k8s.scale_nodes("sandbox-pool", target)
            logger.info(f"Scaling down: {current_nodes} → {target} nodes")
    
    def _scale_up_increment(self, queue_depth):
        """Scale more aggressively when queue is deep."""
        if queue_depth > 50:
            return 5   # Emergency scale
        elif queue_depth > 10:
            return 2
        return 1

Scaling Metrics Summary

Metric	Scale up when	Scale down when
Pool utilization	>80%	<30% for 10 min
Claim queue depth	>10	0 for 5 min
Cold start rate	>5%	<1%
CPU throttle rate	>10%	<2%
Node CPU utilization	>70%	<20% for 15 min

---

6. Cost Optimization at Scale

Technique	Savings	Tradeoff
CPU overcommit (4-8×)	4-8× fewer nodes	Occasional latency spikes
Sandbox pooling	Avoid cold-start costs	Memory for idle sandboxes
Shared base images	80% less storage	Must maintain base image
Spot/preemptible nodes	60-70% compute savings	Session interruption risk
Auto-expire idle sessions	Free memory for active	User must restart session

Spot instance strategy for agents: Agent sandboxes are stateless (or state is checkpointed). Use spot instances for the sandbox pool, and checkpoint active sessions every N turns so interrupted sessions can resume on a new node.

---

Self-Check

• Calculate the overcommit ratio appropriate for interactive agent sessions
• Design a sandbox warm pool with target cold-start rate
• Implement an autoscaler based on pool utilization and queue depth
• Build a layered image strategy to minimize per-sandbox storage
• Describe when to use spot instances for agent infrastructure

---

References

• E2B Architecture: Scaling Firecracker Sandboxes
• Kubernetes Horizontal Pod Autoscaler
• For sandbox types, see Sandbox Types
• For state persistence across interruptions, see Agent State Management