Sandbox service scales agent reinforcement learning on Kubernetes clusters
Sandman: Building and Operating Sandboxes for Agentic Reinforcement Learning on Shared Kubernetes
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
Summary
Many AI training tasks need to run lots of small, separate computer environments quickly, which can be hard when sharing resources with other tasks. The authors built Sandman, a system at Amazon that creates and manages these environments efficiently on a Kubernetes cluster. Sandman keeps each environment ready to use with files and processes stored until no longer needed. It has operated at very large scale, handling over 11,000 environments at once and millions over time. The paper shares design choices and lessons from running Sandman in real-world settings.
What this means in practice
- •For machine learning platform engineers: Manage large numbers of task-specific agent training environments on shared Kubernetes clusters efficiently and reliably.
- •For cloud infrastructure teams: Operate sandboxed environments that persist state and processes for bursty workloads without disrupting other shared cluster jobs.
Authors
Yiming Yan, Hongyang Wang, Wenbo Chen, Xuan Wang, Jingguo Liu, Xian Li, Chen Qi, Leo Fan, Siwei Li, Qianli Wu, Yichuan Li, Ruochen Wen, Qian Qu, Zheshen Wang, Benoit Dumoulin
Abstract
Agentic reinforcement learning (RL) often requires repeated tool calls in task-specific environments. A training job may create thousands of these environments in bursts while sharing a cluster with other jobs. We present Sandman, a sandbox service built and operated at Amazon for agent tool execution and evaluation. It serves multiple production agentic RL workloads on shared Kubernetes. A training framework supplies a task image and uses commands and file operations in the returned sandbox. Files and processes persist across calls until the framework releases the environment. Sandman manages creation, readiness and cleanup. Each sandbox is a Kubernetes pod with a small daemon mounted into the unmodified image. In a scale test, Sandman held 11,858 concurrent tiny sandboxes. Four production SWE-bench Verified training campaigns created 3.1 million sandboxes in October 2026. We report the design trade-offs and operational lessons from handling creation bursts, preparing task environments, reclaiming abandoned sandboxes, and running evaluation under controlled conditions. We will open-source Sandman's daemon, provider and pool.