Sandman: A Sandbox Service for Production Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) needs stateful environments for tool execution and evaluation. Training jobs request thousands of environments in bursts that can overwhelm shared cluster services, while cancelled or timed-out requests can leave resources running. We present Sandman, a sandbox service built and operated at Amazon for large-scale agentic RL workloads in production on shared Kubernetes. A small daemon mounted into unmodified task images preserves files and processes across calls. Sandman paces creation requests, checks sandbox readiness, and retains cleanup responsibility after callers time out or cancel. Production deployments created 3.1 million sandboxes for 122 coding-agent training runs in two weeks. Peak production concurrency was 11,858 sandboxes. In one 10,000-request multi-client burst, Sandman made half the sandboxes usable in 116 seconds and all in 308 seconds. Full delivery took 44% less time than a tuned configuration of agent-sandbox, an open-source sandbox manager for Kubernetes. Sandman left no sandboxes at client exit. On software-engineering task images, median no-op command latency was 2.0 ms, compared with 14.3 ms for agent-sandbox and 203.5 ms for OpenSandbox. We also measure provisioning costs and sandbox delays within training runs, and report design trade-offs and operational lessons from production use and comparisons with three open-source systems. We will open-source Sandman's daemon, provider and pool.