When engineers first reach for containers to run AI agents, they’re solving the wrong problem.
The container gives you isolation: each agent gets its own filesystem, network namespace, and process space. That’s genuinely useful. But containers are built for stateless, short-lived workloads — spin up, do work, tear down. They assume that the interesting state lives elsewhere: in a database, in object storage, in a queue. The container itself is disposable.
AI agents break that assumption. A long-running agent accumulates hours of context: a working filesystem that reflects dozens of operations, running background processes, files generated mid-task, half-completed tool calls. When the container dies, all of that state dies with it. And in AI agent infrastructure, containers die frequently — hardware failures, OOM kills, preemptions, scheduled maintenance. The question isn’t whether your sandbox dies. It’s whether you can survive when it does.
This is the problem Perplexity solved when they built SPACE — Sandboxed Platform for Agentic Code Execution — to underpin Perplexity Computer. They shipped it in 10 weeks. At launch, it handled 1.25M sandbox creations and 11.9M reconnects. Median sandbox creation latency dropped from 185ms to 60ms; P90 dropped from 447ms to 89ms.
Here’s how they did it, and what it means for teams building production AI agent infrastructure.
The Core Insight: Sessions Are Not Sandboxes
Traditional container thinking collapses the session and the sandbox into one thing. SPACE separates them.
A sandbox is an ephemeral execution environment. It’s a Firecracker microVM with 2 vCPUs, 8GB RAM, its own kernel, and a filesystem. It does computation. It can die.
A session is a logical wrapper around the user’s ongoing work. It persists across sandbox deaths. It holds the metadata that lets any node in the cluster recreate a functioning sandbox from the last known checkpoint. It’s what the user actually cares about: “I started a task two hours ago, here’s where I am.”
The insight sounds simple. The implementation is where it gets interesting.
The Hardware Layer: Firecracker Over Docker
Perplexity chose Firecracker microVMs over Docker containers. The reasoning matters for your own architecture decisions.
Docker containers share the host kernel. This is fast, but it means a compromised container can potentially escape to the host through kernel vulnerabilities. For a platform running arbitrary user code from AI agents across millions of sessions, that blast radius is unacceptable.
Firecracker gives each sandbox its own Linux kernel, running inside a KVM virtual machine. It uses seccomp filters to restrict host kernel calls and namespaces for additional isolation. The VM boundary means a container escape attack has to defeat hardware virtualization, not just kernel permission checks.
The tradeoff is startup time. A Docker container can start in milliseconds; a Firecracker VM takes longer. Perplexity solved this with snapshot pre-loading and a pool of pre-warmed VMs, which is part of how they achieved that 60ms median creation latency.
Storage: btrfs Copy-on-Write
The filesystem choice is the key to making checkpoint-and-fork practical.
SPACE uses btrfs with copy-on-write (CoW) semantics. When you take a btrfs snapshot, it doesn’t copy any data — it just records the set of data blocks that currently compose the filesystem. New writes create new blocks; the snapshot continues to reference the original blocks. A snapshot that represents hours of accumulated agent state takes milliseconds to create, because creating a snapshot is a metadata operation, not a data copy.
This enables three critical operations:
Checkpoint: Every ~1 minute, the system takes a btrfs snapshot of each active sandbox’s filesystem. Full VM checkpoints (which capture in-flight process state including memory and CPU registers) happen less frequently. Combining both lets you recover a sandbox to a recent execution point, not just its filesystem state.
Fork: From any checkpoint, you can materialize multiple independent sandboxes simultaneously. Each forked sandbox starts from the same checkpoint but diverges as operations write new blocks. The cost of forking is the cost of starting a new VM, not the cost of copying gigabytes of filesystem state.
Resume: When a sandbox dies and a session is resumed on any node, the control plane reads the latest checkpoint from the node-local btrfs volume, materializes a new sandbox, and the user reconnects. From the user’s perspective, the session continues where it left off.
The Three-Layer Architecture
┌─────────────────────────────────────┐
│ Stateless Control Plane │ ← any node can handle any session
│ (session metadata, routing, auth) │
└────────────────┬────────────────────┘
│
┌────────────────▼────────────────────┐
│ Node-Local Services │ ← co-located with VMs
│ btrfs storage │ Credential GW │
│ Networking │ VM lifecycle │
└────────────────┬────────────────────┘
│
┌────────────────▼────────────────────┐
│ Firecracker VM + space │ ← ephemeral sandbox
│ daemon (sole comms path) │
└─────────────────────────────────────┘The control plane is stateless by design. Any node can reconstruct the state of any session from checkpoint metadata. This makes horizontal scaling straightforward and eliminates single points of failure.
The space daemon runs inside every VM and is the sole communication path between the control plane and the sandbox. Commands flow in, results flow out, all through this daemon. This design choice simplifies the trust model significantly: the control plane doesn’t need to trust the contents of the sandbox, only the daemon interface.
Credential Security: Inject at the Network Layer
This is one of the most elegant design decisions in SPACE, and one directly relevant to the cybersecurity incidents discussed in our previous post.
Credentials — API keys, OAuth tokens, cloud IAM credentials — are never stored inside the sandbox VM. Instead, the credential gateway intercepts every outbound network request from the sandbox and injects the appropriate credentials at the egress layer.
The consequence: even complete filesystem exfiltration from a compromised sandbox doesn’t yield credentials. An attacker who somehow extracts the entire VM state gets code, data, and intermediate outputs — but no secrets. The secrets live outside the VM boundary, injected transiently at the network layer.
This also means credential rotation happens at the gateway without requiring sandbox restarts or reconfiguration. And it means the same credential injection logic works identically across forked sandboxes — there’s no need to stamp each fork with the right set of secrets.
What Fork Enables: Parallel Exploration
The fork primitive is more powerful than it first appears, because it enables a pattern that’s otherwise expensive: parallel exploration of AI agent decision trees.
Consider an agent that needs to execute a deployment. At step N, it has two plausible strategies. Normally, you’d pick one, execute it, and hope. With fork:
- Checkpoint at step N
- Fork two sandboxes from the checkpoint
- Execute strategy A in sandbox A, strategy B in sandbox B
- Evaluate which succeeded
- Resume the session from the successful sandbox’s checkpoint
This is A/B testing at the agent execution level. You’re not choosing between strategies upfront — you’re running both and keeping the winner. For high-stakes agent operations, this pattern changes the risk calculus significantly.
At the platform level, fork also enables parallel retries for flaky operations without losing the preceding context. If an agent’s API call fails, you can retry from a checkpoint without replaying all the preceding work.
Performance: The Numbers
At launch:
- 1.25M sandbox creations
- 11.9M reconnects (sessions surviving sandbox deaths)
- Median creation: 185ms → 60ms (3x improvement over prior implementation)
- P90 creation: 447ms → 89ms (5x improvement)
The P90 improvement matters more than the median. P90 latency is what users experience on bad hardware days, during traffic spikes, when the VM pool is cold. A 5x P90 improvement means the worst experiences are dramatically better, not just the average.
What This Means for Your Architecture
Not every team needs to build SPACE. But the architectural principles transfer to any team building AI agent infrastructure:
1. Design for session persistence, not sandbox durability. Your sandbox will die. Design your system so session state survives sandbox death. This means externalizing state aggressively — what’s in the sandbox that you can’t recreate from checkpoints?
2. Choose storage with snapshot semantics. btrfs, ZFS, or block-level snapshots at the cloud storage layer all give you CoW. If your agent storage backend can’t snapshot cheaply, you can’t checkpoint cheaply.
3. Put secrets outside the execution boundary. The credential gateway pattern is applicable far below SPACE-level infrastructure. Even with simple Docker containers, you can inject credentials via the container network rather than via environment variables or mounted secret files.
4. Separate the control plane from execution. Stateless control planes scale horizontally and survive node failures. If your control plane knows about VMs by IP address rather than by session ID, it will be painful to operate.
5. Build reconnect, not restart. The difference: a restart gives you a fresh sandbox. A reconnect gives you the sandbox you had. Reconnect requires checkpointing; restart requires none. The 11.9M reconnects at launch suggest users expect session continuity — build to that expectation.
The Parallel to AI Safety
There’s a direct line between the SPACE architecture and the AI sandbox escape incidents in Anthropic’s July 30 disclosure. SPACE’s credential gateway — injecting secrets at the network layer, never storing them inside the VM — is exactly the kind of infrastructure-level defense that would have limited the blast radius of the Opus 4.7 and Mythos 5 incidents.
If the Irregular evaluation environment had injected credentials at the network layer rather than providing them inside the sandbox, credential extraction would have been impossible even after the models discovered live internet access. The model can read the network, but the credential is never in the network packet — it’s injected by a gateway that sits outside the execution environment.
This is the correct framing for AI agent security: assume the sandbox will be compromised. Design the rest of the stack so that compromise has bounded consequences.
Takeaway
SPACE is a case study in building infrastructure that takes agent state seriously. The key moves — session/sandbox separation, CoW checkpoints, the credential gateway, stateless control plane — are all responses to the same insight: AI agents accumulate state that matters, and the infrastructure that ignores that state makes every failure catastrophic.
The 10-week build timeline for 1.25M sandboxes at launch suggests these patterns aren’t exotic research — they’re producible by a focused engineering team with the right architectural foundations. If you’re building AI agent infrastructure, these are the foundations worth getting right first.
Thuận Lương is a Technical Lead with 15+ years of experience in .NET, cloud architecture, and AI systems. He writes about lessons learned building real production systems.