betaFeedback

WorksComputer Science

BunshinRL: Whole-Workspace Forking for Agentic Reinforcement Learning Rollouts

Abstract

Agentic reinforcement learning (RL) rollouts interleave model reasoning with tool calls in persistent workspaces. Scaling these rollouts incurs substantial memory costs for concurrent environments, while slow tool calls delay entire rollout batches. We present BunshinRL, an agentic RL rollout system that exploits shared environment state and recurring tool calls through three techniques: memory sharing across environments, tool-prefix caching, and speculative tool execution. These techniques build on BunshinFork, a novel live whole-workspace forking mechanism for containers. BunshinFork reconstructs process trees, shares memory copy-on-write, and preserves file, communication, and GUI state, enabling cached-state reuse and rollback after incorrect speculation. We implement BunshinRL and BunshinFork through Linux kernel module and CRIU modifications. BunshinRL reduces peak memory by up to 43%, total memory consumption by up to 51%, and end-to-end rollout latency by 23%.

Introduction

AI agents are increasingly used to perform computer-operating tasks, with browser agents navigating interactive websites and computer-use agents (CUAs) operating desktop applications through graphical interfaces ([27, 35]). Reinforcement learning (RL) improves their ability to complete multi-step tasks by learning from interaction outcomes ([33]). Unlike text-only RL, where rollouts generate responses without interacting with an external environment, agentic RL rollouts interleave model reasoning with tool execution. Each rollout therefore requires a persistent workspace that maintains application and execution state across tool calls and supplies observations for subsequent decisions.

Agentic RL presents two key new challenges. First, each sample in the training dataset needs an isolated sandbox so that its code execution, file edits, and application interactions cannot corrupt another rollout’s state. A sandbox for a CUA may contain a browser, a display server, desktop applications, and supporting services, all of which consume CPU memory throughout the episode. With 4 GB per sandbox, a batch of 256 rollouts requires 1 TB of CPU memory. Group-based algorithms such as GRPO further multiply environment instances by sampling several trajectories per task ([21]).

Second, tool execution accounts for significant rollout time, as each tool call can take seconds to minutes and there can be a long tool call chain for complex tasks. In synchronous RL, a few slow trajectories can delay the completion of an entire rollout batch, not only delaying the RL rollout process but also leaving other resources underutilized.

These costs present two opportunities for reuse. First, environments within a rollout batch can contain substantial identical memory, especially among trajectories sampled from the same task in a GRPO group. Second, the same set of training tasks across iterations can produce tool calls that are exactly the same as seen in previous iterations. Matching tool call prefixes offer an opportunity to skip tool calls in the beginning of trajectories. Recurring calls later in a trajectory provide history from which we can potentially leverage for tool speculative execution.

Based on these insights, we propose BunshinRL, an agentic RL rollout system that incorporates three complementary techniques. Memory sharing retains one physical copy of common environment state and allocates private pages only as samples diverge in their tool calls. Tool-prefix caching restores workspaces that are snapshotted in previous iterations and maintained in memory- and storage-efficient tree structures. Speculative tool execution uses a small trained tool selector model to predict what tool calls from previous runs to use for speculative execution. Upon reasoning steps completion, we compare tool call signatures and either keep speculating for matched comparison or roll back to the workspace state before wrong tool speculation.

Realizing these techniques requires proper support at the systems layer. Memory sharing requires runtime copy-on-write (CoW) isolation; prefix reuse must restore the state associated with cached tool results; and incorrect speculation must leave no changes in the original workspace. These requirements motivate a common primitive: live whole-workspace forking. A fork shares unchanged parent memory while supporting independent execution. Speculation forks the current workspace before executing a predicted call, whereas caching retains snapshots from which subsequent rollouts can fork.

We realize this primitive with BunshinFork, a live forking mechanism for container workspaces. Containers are lightweight and offer the level of isolation that is sufficient for many types of tasks. However, no existing solution supports live container forking. BunshinFork forks a container by reconstructing the process tree and execution metadata in isolated namespaces, establishing CoW memory mappings, and providing a private writable file-system view. It also preserves internal communication and GUI state while reconstructing resources that require private copies.

On top of BunshinFork, we design several optimizations to reduce the latency and memory consumption of BunshinRL. First, we cache only a selected set of workspace snapshots in memory, while leaving the entire snapshots on disk to avoid on-demand disk read and workspace reconstruction latency. Both the in-memory and on-disk snapshots leverage deduplication to save space. Second, we replay some tool calls even when they hit prefix cache, when replaying them is faster than reconstructing snapshots from disk. Third, we only perform speculative tool execution for tail samples, as tail determines rollout latency and speculation incurs some CPU and memory overhead.

We implement BunshinRL from scratch and implement BunshinFork by changing the Linux kernel and the CRIU container checkpoint and restore system (Checkpoint/Restore Project, [3]). We evaluate BunshinRL by training Qwen3-VL-8B (Qwen Team, [17]) on OSWorld and WebArena, and we compare it with CRIU and AgentENV (kvcache-ai, 2026). On OSWorld, BunshinRL reduces peak memory by 30% compared with CRIU; on WebArena, it reduces peak memory by 43% compared with AgentENV. It reduces total memory consumption by up to 51%. BunshinRL improves end-to-end rollout latency by 23%, while only adding 1 GB of peak memory and maintaining original reward scores.

Overall, this paper makes the following key contributions:

  • We identify memory duplication and tool-execution latency as key challenges in agentic RL rollouts.

  • We propose memory sharing, tool-prefix caching, and speculative tool execution to reduce rollout memory consumption and latency.

  • We build BunshinRL, an integrated agentic RL rollout system that reduces peak memory by up to 43% and rollout latency by 23%.

  • We develop BunshinFork, the live whole-workspace forking mechanism underlying BunshinRL. It also supports container forking beyond RL, including agent inference.

The source code of BunshinRL and BunshinFork is available at https://github.com/WukLab/ForkRL.

Motivation

Efficient agentic RL requires coordinating model generation with many stateful execution environments. We examine the workspace requirements, memory costs, and tool-execution overheads of this workload to motivate the three complementary features of BunshinRL: memory sharing, tool-prefix caching, and speculative tool execution (Figure 3).

Figure 3. Illustration of BunshinRL. Right shows two samples in a simplified rollout step; left shows the on-disk and in-memory workspace snapshot stores for tool call prefix. Reasoning steps’ generated tool call signatures are matched with the tool prefix tree. All initial matched tool calls are skipped until the first miss in tool signature (Tool 3 for R1 and Tool 4 for R2), at which point the corresponding snapshots are used to reconstruct the workspaces. As C3 is not cached in memory, Tool 3 is replayed for R2 and its delta write is added to the C0+C1+C2 snapshot. The samples execute actual Tool 3 for R1 and actual Tool 4 for R2, creating new delta CoW pages. After these calls finish, speculation starts from the resulting workspaces: Tool 4 for R1 and Tools 5 and 6 for R2. The Spec Tool Selector selects tool calls that are likely to happen from previous tool call history. When a reasoning step finishes, its output tool signature is compared with the speculative tool signature. For failed speculation, the system rolls back to the workspace snapshot before speculation.

Background: Agentic RL

Agentic RL trains a policy to complete tasks through interactions with an environment. Training alternates between collecting rollout trajectories and updating the policy using their rewards. Within a trajectory, the policy reasons over the task and previous observations, generates a tool call, and receives an observation from its execution. Group-based algorithms such as GRPO sample multiple trajectories for each task to construct relative reward signals ([21]). Unlike text-only RL, collecting agentic RL rollouts requires not only model inference but also environment execution to carry out tool calls and obtain observations.

Workspace Environments in Rollout

Each rollout needs an isolated workspace whose state persists across tool calls. Systems such as AgentENV (kvcache-ai, 2026) and DSec (Huang et al., [8]) provide sandbox infrastructure for executing agentic RL rollouts. For a computer-use agent, this workspace may include a browser, desktop applications, a display server, and supporting services. A tool call can modify files, leave processes running, or change application memory and GUI state.

These properties constrain both reuse and speculation. Reusing a tool result must also recover the workspace state it produced; executing a predicted call must isolate its effects until the prediction is accepted. Copying files alone misses live application state, while restarting and replaying calls adds work and may produce different outcomes because of timing, background activity, or remote responses. Workspace management must therefore account for processes, memory, files, and communication state. External services require separate handling: restoring a local workspace cannot undo effects outside its boundary.

Memory Cost in Rollout

A rollout batch requires many concurrent environments, especially when group-based training samples several trajectories for each task. Each environment may hold a large application working set, so maintaining independent copies can quickly exhaust host memory. Yet these environments often contain the same libraries, initialized applications, and task data. Sharing immutable container image layers does not eliminate duplicated process memory accumulated during execution.

Figure 1 breaks down memory that is identical across environments into file cache, heap, and shared memory. Environments in the same GRPO group exhibit more common state than environments from different tasks, and substantial overlap remains after ten steps. The heap and shared-memory components highlight opportunities beyond sharing immutable files.

Figure 1. Shareable Memory in Agentic RL Environments. Share of each environment’s memory that is identical in another environment, split into file cache, heap, and shared memory. The bars compare environments of different tasks with environments of the same GRPO group at step 0 and step 10.

Memory sharing can retain one physical copy of common state and allocate private pages only when an environment modifies them. Memory consumption then depends primarily on the shared working set and each trajectory’s changes, rather than a full copy for every rollout. Some resources, such as process metadata, still require private storage, and savings decrease as trajectories overwrite shared state. Reducing duplication allows more rollouts to run concurrently under a fixed memory budget and makes room for cached workspace states and speculative executions.

Figure 2 compares peak CPU memory as rollout concurrency increases. The gap between CRIU and BunshinRL widens because CRIU maintains private memory copies while BunshinRL shares unchanged state. At 256 concurrent rollouts, CRIU uses about 1.8 times as much peak memory, illustrating the capacity benefit of sharing across environments.

Figure 2. Peak Memory as Rollouts Scale. Peak CPU memory of BunshinRL and CRIU as the number of concurrent rollouts grows. Each CRIU fork holds a private copy of memory, while BunshinRL forks share the memory they have not changed.

Tool Calls in Rollout

A rollout spends time both generating model outputs and executing tools. In synchronous training, the batch must wait for its slowest trajectory, so a long browser navigation, test suite, or application operation can delay the next training step even after other rollouts finish. These delays accumulate across tool calls, and faster model inference alone does not remove them.

Repeated training tasks create two opportunities to reduce tool latency. First, trajectories can repeat tool-call prefixes across samples and training iterations. A prefix cache can reuse their observations and resulting workspace states, skipping tool execution while retaining each trajectory’s reasoning ([12]). A valid hit must match the originating task, workspace, and tool-call prefix; identical arguments alone are insufficient.

Second, calls that recur after trajectories diverge can inform predictions of upcoming tool invocations. Executing a predicted call alongside policy generation can hide tool latency ([22]). A matching call can adopt the speculative result and workspace state; a mismatch requires discarding speculative changes and executing the actual call from the prior state. The benefit depends on prediction accuracy, available overlap, and the cost of managing speculative workspaces.

Together, these opportunities motivate BunshinRL as an integrated rollout system: memory sharing increases environment concurrency and reduces the cost of retained states, prefix caching avoids repeated tool execution, and speculation overlaps remaining tool work with reasoning. Efficient whole-workspace management enables all three, while reuse must respect the task’s execution and sampling semantics.

BunshinRL: Agentic Rollout System

BunshinRL manages the execution environments used to collect agentic RL trajectories. It combines three features: memory deduplication across related workspaces, tool-prefix caching that avoids repeated execution, and speculative tool execution that overlaps tool latency with model generation. All three use BunshinFork, the whole-workspace forking mechanism to be described in § 4. Figure 3 illustrates the main components of BunshinRL.

Memory Deduplication

BunshinRL reduces the resident memory required by concurrent rollouts by forking workspaces and performing memory copy-on-write (CoW). Rollouts for samples in the training batch begin from one initialized workspace. When they follow a common sequence of tool calls, they reuse cached workspace snapshots (§ 3.2) and defer creating a writable branch until their calls diverge. A sample that shares a longer prefix can fork a later snapshot and inherit the state accumulated through that prefix.

BunshinFork maps unchanged pages from a source into its descendants with copy-on-write. A branch allocates private pages as it writes, while other descendants continue to reference the shared pages. The memory cost only includes the common memory state, branch-private writes, and reconstruction metadata, rather than a full workspace image for every sample (§ 2.3).

The same sharing applies inside the prefix snapshot store and to speculative branches. A child snapshot retains changes relative to its parent while referencing inherited pages; a speculative fork adds private state only as it diverges. In Figure 3, C3C_3 extends C2C_2 with Δ3\Delta_3. A sample adopting C3C_3 can share the base, Δ1\Delta_1, and Δ2\Delta_2 already mapped by another sample adopting C2C_2.

Tool-Prefix Caching

BunshinRL organizes tool-call prefixes as a tree rooted at the initial workspace state C0C_0. Each edge represents a tool invocation under its parent state. A cache hit requires matching tool call signature prefix, including tool names and arguments at every tool call steps from the initial state. A sample traverses the tree as its policy generates calls. On the first miss (unmatched tool name or argument), BunshinRL reconstructs the cached state (up until the last tool hit) and then uses BunshinFork to create a writable branch.

Snapshot cache management. We record all previous iterations’ workspace states after every tool call for all samples. Storing the raw snapshots of all of them will consume huge amounts of storage space. We leverage a similar deduplication idea from § 3.1 by storing only the deltas of what a tool call changes at every child node of a prefix tree, as shown in Figure 3. The entire prefix tree is then stored on disk. Upon prefix matching, we restore the workspace by reading from the root to the last matched tool call and add deltas of every tree node to the base snapshot. Although this technique makes the snapshot tree small enough to fit on disk, reading tree nodes from disk and reconstructing workspaces can be slow and offset the latency saved by prefix reuse. BunshinRL therefore retains hot workspace snapshots in memory. We design an ancestor-aware LRU policy which maintains an ancestor-closed cache: accessing a snapshot refreshes its recency and retains all its snapshot ancestors in memory. Eviction preserves this invariant, so an ancestor remains resident while any cached descendant depends on it.

Sparse snapshots and replay. To further reduce memory consumption caused by snapshot caches, BunshinRL materializes snapshots at selected tool-call boundaries rather than every step, while retaining tool-call metadata. A prefix without its own snapshot is reconstructed by forking the nearest retained ancestor and replaying the intervening calls. The replay overlaps with model reasoning and thus keeps reconstruction off the critical path, e.g. Tool 3 replay with Reason 4 for R2 in Figure 3. Our sparse snapshot mechanism trades additional tool execution for a smaller memory footprint without adding latency.

Overall, the memory footprint introduced by snapshots is much smaller than the memory saving we get from deduplication (§ 3.1) but largely speeds up the rollout phase.

Speculative Tool Execution

We propose speculative tool execution in BunshinRL to hide tool latency behind model reasoning. Our key insight is that recurring tool calls performed in earlier training iterations provide a basis for predicting future actions in later iterations.

Speculation predictor and mechanism. We train a small classifier model, Spec Tool Selector, which predicts the tool call from all historically recorded tool calls based on the sample’s trajectory so far. BunshinRL forks the current workspace and executes predicted calls in parallel with reasoning. As soon as the policy produces the actual signature, BunshinRL compares it with the prediction. A match adopts the speculative observation and state once execution completes and its dependencies are validated; an unfinished call incurs only its remaining execution time. A mismatch immediately aborts the speculative call, discards its private writes, and starts the actual call from the preceding valid snapshot. Figure 3 shows failed speculation for R1R_1 and successful speculation for R2R_2.

To further overlap tool-call speculation with model reasoning, we predict KK number of tool calls at a time and execute them in the speculation workspace. Each speculated tool call is compared with actual tool signatures as soon as the corresponding reasoning step finishes. The first mismatch aborts the whole speculation chain.

Tail-aware selection. Even when we perform memory-deduplicated fork for speculation execution, it unavoidably still consumes some additional CPU and memory. To reduce this overhead, we perform speculation only to samples causing the long tail in a rollout batch, as that is what determines the rollout iteration’s total latency. We use recorded timings from previous training iterations to identify historically slow samples and long tool calls, and we only perform speculation for those samples. This selection concentrates speculative work on the most valuable opportunities; other samples use ordinary tool execution and prefix caching.

Overall, tool call speculation reduces overall rollout latency while adding only tiny memory overhead thanks to our whole-workspace forking and memory deduplication mechanisms.

BunshinFork: Workspace Fork

This section describes BunshinFork, the whole-workspace forking mechanism underlying BunshinRL (§ 3), and how it preserves isolated execution while sharing unchanged state.

Why Fork Containers

Agentic RL maintains rollout environments, cached prefixes, and speculative branches whose memory footprint limits concurrency. BunshinFork uses containers so the host kernel can directly share anonymous memory copy-on-write and unchanged file-backed pages through its page cache, without duplicating a guest kernel for each environment. This reduces the memory cost of the three optimizations in BunshinRL. Fork creation must also be fast: its overhead can otherwise offset the latency saved by prefix reuse and speculative execution.

MicroVMs provide a stronger isolation boundary but introduce guest-kernel and memory-management overheads [8]. Container isolation is sufficient for our web and GUI rollout tasks, which require no guest-level privileges. Workloads requiring privileged operations, such as nested containers, may instead favor VMs [11, 8].

Fork Overview

BunshinFork captures a consistent workspace boundary and reconstructs its processes, memory mappings, files, and communication resources in an isolated container. Its central principle is to separate online fork creation from durable checkpointing: unchanged memory is shared rather than eagerly copied, while serialization to storage proceeds asynchronously. The source resumes once the logical snapshot is consistent, allowing rollouts to continue without waiting for disk writes.

To fork the entire process tree, BunshinFork briefly freezes the source and records its process hierarchy, threads, open descriptors, signal and synchronization state, and the register state needed to resume execution. It then creates a destination with independent namespaces and restores the recorded topology, preserving application-visible process identifiers and relationships while assigning distinct host identifiers. Separate PID, mount, network, and IPC namespaces isolate the restored resources from those of the source. Corresponding source and destination processes are linked for memory sharing. A single frozen source can supply multiple clones, which subsequently execute independently.

The freeze covers metadata capture, memory snapshot-holder creation (§ 4.4), and a coordinated file-system snapshot. It copies metadata and page tables rather than the application working set. Once these components describe the same point in execution, the source resumes and any durable serialization continues in the background. Figure 4 illustrates the key mechanisms of BunshinFork.

Figure 4. BunshinFork architecture. A brief freeze captures a consistent workspace state. The live fork path reconstructs isolated clones that share unchanged pages copy-on-write and allocate private pages on writes. Retained snapshot holders support asynchronous serialization, allowing the source to resume without waiting for disk writes.

Process Subsystem

Forking each process independently does not reproduce a workspace: Linux fork() makes each new process a child of its source, breaking the original parent–child relationships. As Figure 5 illustrates, BunshinFork instead reconstructs the original process-tree topology inside an isolated container. To do so, BunshinFork reads the parent–child relationships from the frozen snapshot and recreates them among the processes in the new container.

Linux Process Fork vs. BunshinFork Process-Tree Fork

Figure 5. Linux Process Fork vs. BunshinFork Process-Tree Fork. Left: native fork() duplicates one process as a child of the source process. Right: BunshinFork creates a sibling container and reconstructs the whole process tree inside an independent namespace.

Memory Subsystem

Rollouts and speculative branches inherit application state and modify it incrementally. Eagerly copying that state would increase both fork latency and memory consumption. BunshinFork therefore shares inherited memory copy-on-write (CoW), allocating private pages only as branches diverge.

BunshinFork distinguishes anonymous memory from file-backed and shared memory. Anonymous memory has no backing file; file-backed memory includes executable images, libraries, and mapped application data. Shared-memory regions support communication among processes and must preserve those relationships within each restored workspace.

Today’s Linux fork() establishes sharing between a parent and its child within a workspace, but BunshinFork must share memory between corresponding processes in separate containers. To achieve this, we build a Linux kernel module to install write-protected mappings of the source’s anonymous pages in each destination without copying their contents. The kernel’s existing CoW fault handling then allocates a private page on the first write by either side, leaving other branches’ mappings unchanged. Both the source and its clones can therefore continue writing without changing the state observed by another rollout. Multiple clones can share the same source pages, so their memory cost follows private writes rather than full workspace copies.

File-System Subsystem

BunshinFork combines a read-only btrfs snapshot with a private overlayfs layer for each clone. The snapshot is taken while the source is frozen; clones read unchanged files from the shared lower layer and write to their own upper layers. Because siblings read the same underlying files, the host page cache can retain one copy of their unchanged pages.

When a clone is forked again, reflink copies of its upper layer preserve its modifications while sharing existing disk blocks. Overlayfs copies a file into the private upper layer on its first write, so modifying a large file can still incur substantial copy-up overhead.

Network Subsystem

BunshinFork distinguishes connections within a workspace from connections to external services. Internal connections, such as those between a browser and a local server, are part of the captured state. Their TCP state is restored using kernel TCP-repair mechanisms inside each clone’s independent network namespace, avoiding endpoint and port conflicts.

External connections are closed during fork and reestablished independently by each clone: a remote server cannot safely treat multiple branches as the same endpoint. Duplicating a live connection would leave ownership and message ordering ambiguous and could replay operations with externally visible side effects. Speculative branches can restrict external networking to prevent uncommitted tool calls from producing remote effects. Restoring a local snapshot cannot undo effects outside the workspace.

GUI Subsystem

A containerized application that relies on a host X11 ([19]) or Wayland ([7]) server leaves window, input, clipboard, and compositor state outside the workspace. Restoring the application alone can therefore produce an inconsistent GUI.

BunshinRL instead runs the GUI stack inside each workspace using Webtop ([14]) and Selkies ([20]). The Wayland compositor and its clients reside together in the container, with the desktop exported over HTTPS. Keeping this state together ensures that a restored agent observes the desktop associated with its restored applications, rather than a separate host session. BunshinFork reconstructs them together, giving each rollout or speculative branch an independent display session, cursor, windows, clipboard, and input stream.

Evaluation

We implemented BunshinRL from scratch; the rollout algorithm is based on GRPO, and the rollout phase runs on vLLM ((vLLM Team, 2023)). We implemented BunshinFork with the Linux kernel module and extended the CRIU system. We run all experiments on two servers: a GPU server with one NVIDIA H100 GPU serves the policy model during rollouts and trains it synchronously; a workspace server hosts all sample workspaces and stores snapshot caches. The latter has two Intel Xeon Gold 5218 CPUs with 64 hardware threads in total, 256 GB of DDR4 memory, and a 512 GB NVMe SSD.

Figure 6. Memory over an OSWorld Epoch. Working set of all environments on a node over one epoch of OSWorld rollouts with BunshinRL, CRIU, and AgentENV.

Figure 7. Memory over a WebArena Epoch. Working set of all environments on a node over one epoch of WebArena rollouts with BunshinRL, CRIU, and AgentENV.

Experimental Setups and Baselines

We evaluated BunshinRL using OSWorld (Xie et al., [27]) and WebArena (Zhou et al., [35]). OSWorld has 369 desktop tasks that use applications such as a web browser, an office suite, a media player, and a combination of common desktop based CUA tasks. WebArena has web tasks on several self-hosted sites, and we selected its shopping site tasks subset as our training workload. Each WebArena environment runs the shopping site and its database in the same container as the browser. We train Qwen3-VL-8B (Qwen Team, [17]) with a simple synchronous GRPO loop (Shao et al., [21]) on a subset of these two workloads.

Baselines. We compare BunshinRL with two systems that fork environments, using the same GRPO loop, policy, and tasks. CRIU (Checkpoint/Restore Project, [3]) forks a container by writing a checkpoint to disk and restoring each fork from it. Each restored fork holds a private copy of the source’s memory. AgentENV (kvcache-ai, 2026) is a sandbox system designed for the Kimi K3 RL environment. It builds a fork mechanism on top of Firecracker microVM (Agache et al., [1]). We give AgentENV the same workspace images as BunshinRL and CRIU.

Rollout Memory

Figures 6 and 7 show the memory of all environments on the environment server over one training epoch. Memory rises as rollouts start, peaks while most rollouts run at once, and falls as rollouts finish. On OSWorld, BunshinRL reduces peak memory by 30% compared with CRIU and by 26% compared with AgentENV. On WebArena, BunshinRL reduces peak memory by 43% compared with AgentENV.

Peak memory. Containers share the host kernel, so the host kernel sees every page of every environment. BunshinFork uses this to share exactly the pages that a fork has not written (§ 4.4). Forks also read unchanged files through one copy in the host page cache (§ 4.5). CRIU restores every fork with a private copy of the source’s memory, so each fork adds the memory of a whole environment. AgentENV forks share the snapshot’s memory copy-on-write, so AgentENV uses less memory than CRIU at the same batch size. AgentENV also shares file data through the host page cache and returns free guest pages to the host. However, each microVM runs its own guest kernel, whose page tables and other metadata stay private to the microVM. A microVM also needs two levels of page tables, the guest kernel’s and the host’s for guest memory, while a container needs only the host’s.

Epoch length. All three systems replay the same recorded trajectories, so differences in epoch length come from the environments. On OSWorld, CRIU takes 1.3× longer than BunshinRL to finish the epoch. Each CRIU fork writes a checkpoint to disk and restores a private copy of memory, so forking takes longer. On WebArena, a full batch of CRIU environments does not fit in the server’s 256 GB of memory and runs out of memory, so we lower CRIU’s batch size. Smaller batches and slower forks together make CRIU take 2.4× longer than BunshinRL to finish the epoch. Its peak memory is lower than AgentENV’s only because it runs fewer environments at once. Even so, BunshinRL runs the full batch with 36% less peak memory than CRIU. This shows that memory also limits rollout throughput, because environments that do not fit force a smaller batch.

AgentENV forks quickly, but its OSWorld epoch still takes 1.2× longer than BunshinRL’s, and its memory stays near its peak for over half the epoch. An OSWorld environment with Chrome runs about 40 processes, and a batch runs many environments at once on 64 hardware threads, so the CPU is heavily oversubscribed. In a container, the host kernel sees every process and can run it on any free CPU. In a microVM, the host schedules virtual CPUs and the guest kernel schedules processes on them, so every process is scheduled twice. Each switch between virtual CPUs is also a VM context switch, with a VM exit and a VM entry. A microVM’s processes can also run only on its own virtual CPUs, even when other host CPUs are idle. As a result, tool calls take longer in AgentENV than in BunshinRL, and AgentENV’s environments hold their memory longer.

Tool-Prefix Caching and Speculative Tool Execution

All experiments in this subsection run on OSWorld and compare three configurations of BunshinRL. Base BunshinRL shares memory across forks but uses neither tool-prefix caching nor speculative tool execution. Spec, labeled Spec tool call in the figures, adds speculative tool execution to Base. Full BunshinRL adds both tool-prefix caching and speculative tool execution to Base. We run one epoch with each configuration, and we also train with each configuration for five epochs.

Rollout latency. In synchronous RL training, a step waits for its slowest rollout. A few stragglers therefore set the step time and leave the GPU underused while it waits for them. Rollout latency matters both overall and at the tail, and the two techniques target these two parts. Tool-prefix caching skips the tool calls that match a cached prefix, so it shortens many rollouts and lowers latency overall. Speculative tool execution runs a predicted tool call while the policy is still generating, so it hides the latency of slow tool calls and helps most for the stragglers.

Figure 8 shows how many rollouts are still running over the epoch, and the epoch ends when the last rollout finishes. Base finishes the epoch in 20.8 minutes. Spec finishes it in 19.2 minutes, which is 1.1× faster, and Full finishes it in 17.0 minutes, which is 1.2× faster. Spec mainly shortens the tail. Half of its rollouts finish only 0.5 minutes earlier than in Base, but its last rollout finishes 1.6 minutes earlier. Full adds prefix caching and finishes rollouts earlier throughout the epoch. Half of its rollouts finish 1.4 minutes earlier than in Base, and its last rollout finishes 3.8 minutes earlier. Prefix caching skips matched calls entirely, while speculation hides only the part of a tool call that overlaps with generation, and a misprediction still runs the actual call afterward.

Figure 8. Rollout Completion over Time. Number of rollouts still running over one OSWorld epoch for the Base, Full, and Spec configurations of BunshinRL (§ 7.1). A batch that drains sooner means higher rollout throughput.

Memory cost. Keeping prefix snapshots and speculative forks adds little memory, because each one is a copy-on-write fork. A cached snapshot starts with no private copy of its anonymous memory and shares all of it with the environment it came from. The snapshot is frozen and never writes, so it holds a page of its own only when that environment later writes to the page. Its cost therefore grows with how much the environment changes after the snapshot, not with the size of the environment. Snapshots along one prefix also share the pages they inherit from each other (§ 3.1). Beyond these pages, each snapshot or fork holds only its own copy of shared-memory regions, such as display buffers, and its own page tables. A mispredicted speculative fork is discarded as soon as the policy model issues a different call, so it holds its memory only briefly. As Figure 9 shows, Full and Spec raise peak memory by only about 1 GiB over Base, from 69 GiB to 70 GiB. Full also lowers memory late in the epoch, because its rollouts finish sooner and release their environments.

Figure 9. Memory Cost of Environment Reuse. Working set of all environments over one OSWorld epoch for the Base, Full, and Spec configurations of BunshinRL (§ 7.1).

Hit rate. Figure 10 shows the share of tool calls in each training epoch that are served from a cached snapshot or a speculative fork instead of running on the rollout’s critical path. Full starts at 40% in the first epoch, where hits come from rollouts of the same GRPO group that share a prefix. Its hit rate rises to 65% by the third epoch, because the cache now holds snapshots from earlier epochs. Spec stays between 41% and 53%. From the second epoch on, Full hits 10 to 13 points more often than Spec, because prefix-cache hits add to speculation hits. Both hit rates drop in the fifth epoch as model weights drift more significantly and it starts to issue calls that earlier epochs did not make.

Figure 10. Hit Rate over Training. Share of tool calls in each OSWorld training epoch served from a cached snapshot or a speculative fork, for the Full and Spec configurations of BunshinRL (§ 7.1).

Reward. Tool-prefix caching and speculative tool execution do not change the reward. Figure 11 shows the average reward in each epoch of the same three training runs. The three configurations reach the same reward in every epoch.

Figure 11. Training Reward. Average rollout reward in each OSWorld training epoch for the Base, Full, and Spec configurations of BunshinRL (§ 7.1).

Fork Microbenchmarks

Fork memory. Figure 12 forks one OSWorld environment 2 to 16 times and measures the total memory of the source and its forks. At 16 forks, BunshinRL uses 10.0 GB, while CRIU uses 17.8 GB and AgentENV uses 12.3 GB. CRIU adds about 1.0 GB per fork, because each restored fork holds a private copy of the source’s process memory. BunshinRL adds about 0.5 GB per fork, which is the memory a fork cannot share, such as the display buffers that BunshinFork keeps private (§ 4.4) and the kernel metadata of the fork’s processes. AgentENV adds about as much per fork as BunshinRL, because its forks also share the snapshot’s memory copy-on-write. However, AgentENV starts higher, and it uses 1.5 GB to 2.8 GB more than BunshinRL at every fork count.

Figure 12. Memory of Forks. Total memory of a source environment and its forks as the number of forks grows, for BunshinFork, CRIU, and AgentENV.

Related Work

Agentic RL rollout systems. See, VenusRL, and tree-structured RL optimize scheduling, template sharing, and trajectory branching [16, 34] (Ji et al. 2025). BunshinRL instead exploits live workspace state at intermediate rollout steps, combining memory sharing, tool-prefix caching, and tool speculation.

Sandbox infrastructure. AgentENV supports Firecracker snapshots and live-environment forks (kvcache-ai, 2026; Kimi Team, 2026); DSec optimizes sandbox provisioning, image sharing, and memory reclamation ([8]). BunshinFork branches containers through direct CoW sharing of live application memory, without first serializing it. This complements cluster-level provisioning and scheduling.

Checkpointing and branchable workspaces. CRIU, DeltaBox, Crab, and StateFork preserve execution state through checkpoints and layered storage ([3, 4] (Wu et al. 2026; Xu et al. 2026)). Our distinction is not state preservation alone: BunshinFork directly shares unchanged live memory across isolated process trees, reducing concurrent-branch memory costs while persisting snapshots asynchronously.

Tool caching and environment reuse. TVCache reuses tool values and associated state; its Docker backend preserves files and selected shell metadata ([12]). ToolCaching manages tool responses, while tree-search agents can reconstruct prefixes by replay ([32] (Yu et al. 2024)). BunshinRL caches resumable workspaces, including running processes and memory, so subsequent actions inherit the corresponding application state. Unlike CUA-Sandbox’s runtime/application separation ([29]), BunshinFork requires no application-specific decomposition. Reuse must still respect sampling semantics: shared stochastic outcomes can alter policy updates ([6]).

Speculative tool execution. PASTE, draft-model tool prediction, and search speculation overlap predicted calls with generation ([22, 15, 9]). Speculative Actions also considers sandboxing and rollback (Ye et al., 2025). BunshinRL contributes a concrete live CoW runtime for mutable workspaces: matching predictions retain both results and application state; mismatches discard isolated branches. External effects still require confinement or deferral.

Serverless memory sharing. Catalyzer, SEUSS, and Mitosis accelerate sandbox creation through shared execution state ([2, 25] (Du et al. 2020)); Medes and KSM deduplicate existing pages ([18, 23]). BunshinFork targets intermediate states of long-running, multi-process agent workspaces, enabling BunshinRL to combine memory sharing with tool reuse and speculation in RL rollout.

Conclusion

We presented BunshinRL, an agentic RL rollout system, and BunshinFork, its live whole-workspace forking mechanism. Through memory sharing, tool-prefix caching, and speculative tool execution, BunshinRL reduces peak memory by up to 43% and rollout latency by 23%. Together, these techniques make stateful agent training more resource-efficient and provide a reusable foundation for branching execution in other agent workloads.

References

  1. [1]Agache, A., Brooker, M., Florescu, A., Iordache, A., Liguori, A., Neugebauer, R., Piwonka, P., and Popa, D.-M. Firecracker: Lightweight virtualization for serverless applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pp. 419–434, Santa Clara, CA, 2020. USENIX Association. ISBN 978-1-939133-13-7. URL https://www.usenix.org/conference/nsdi20/presentation/agache.
  2. [2]Cadden, J., Unger, T., Awad, Y., Dong, H., Krieger, O., and Appavoo, J. SEUSS: skip redundant paths to make serverless fast. In Proceedings of the Fifteenth European Conference on Computer Systems, pp. 1–15, 2020.DOI
  3. [3]Checkpoint/Restore Project. CRIU: Checkpoint/restore in userspace. https://criu.org/Main_Page, 2025. Accessed: 2026-05-15.
  4. [4]Dong, Y., He, J., Liu, S., Hou, Y., Du, D., Xu, Z., Yu, S., Yang, B., et al. DeltaBox: Scaling stateful AI agents with millisecond-level sandbox checkpoint/rollback. arXiv preprint arXiv:2605.22781, 2026. URL https://arxiv.org/abs/2605.22781.
  5. [6]Gupta, S. Marginally correct tool caches can reverse group-normalized policy updates. arXiv preprint arXiv:2609.26866, 2026. URL https://arxiv.org/abs/2609.26866.
  6. [7]Hoegsberg, K. The Wayland Protocol. Wayland Project, 2012. URL https://wayland.freedesktop.org/docs/html/. Accessed: 2026-05-15.
  7. [8]Huang, J., Tang, H., Chen, J., et al. DeepSeek Elastic Compute (DSeeC): A sandbox infrastructure for effective agentic training at scale. arXiv preprint arXiv:2609.22978, 2026. URL https://arxiv.org/abs/2609.22978.
  8. [9]Huang, Z., Zeng, W., Fu, T., Liu, T., Sun, Y., Hong, K., Yang, X., Liu, C., Li, Y., Zhang, Q., Dai, G., Zhu, Z., and Wang, Y. Reducing latency of LLM search agent via speculation-based algorithm-system co-design. arXiv preprint arXiv:2511.20048, 2025. URL https://arxiv.org/abs/2511.20048.
  9. [11]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. URL https://arxiv.org/abs/2607.24653.
  10. [12]Kumar, A. V., Kataria, B., Oh, B., Manzoor, E., and Singh, R. TVCache: A stateful tool-value cache for post-training LLM agents. arXiv preprint arXiv:2602.10986, 2026. URL https://arxiv.org/abs/2602.10986.
  11. [14]LinuxServer.io. Webtop: Linux in a web browser. https://github.com/linuxserver/docker-webtop, 2026. Accessed: 2026-05-15.
  12. [15]Nichols, D., Singhhania, P., Jekel, C., Bhatele, A., and Menon, H. Optimizing agentic language model inference via speculative tool calls. arXiv preprint arXiv:2512.15834, 2025. URL https://arxiv.org/abs/2512.15834.
  13. [16]Qin, R., He, W., Huang, W., Zhang, Y., Zhao, Y., Pang, B., Xu, X., Shan, Y., Wu, Y., and Zhang, M. Seer: Online context learning for fast synchronous LLM reinforcement learning. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), 2026. URL https://arxiv.org/abs/2511.14617.
  14. [17]Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. URL https://arxiv.org/abs/2511.21631.
  15. [18]Saxena, D., Ji, T., Singhvi, A., Khalid, J., and Akella, A. Memory deduplication for serverless computing with Medes. In Proceedings of the Seventeenth European Conference on Computer Systems (EuroSys), pp. 714–729, 2022. doi: 10.1145/3492321.3524272.DOI
  16. [19]Scheifler, R. W. and Gettys, J. The X window system. ACM Transactions on Graphics, 5(2):79–109, April 1986. doi: 10.1145/22949.24053.DOI
  17. [20]Selkies Project. Selkies: Linux WebRTC HTML5 remote desktop streaming platform. https://github.com/selkies-project/selkies, 2024. Accessed: 2026-05-15.
  18. [21]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024. URL https://arxiv.org/abs/2402.03300.
  19. [22]Sui, Y., Zhao, H., Ma, R., He, Z., Wang, H., Li, J., Xu, K., Chen, K., et al. Parallelizing tool execution and LLM generation for low-latency agent serving. arXiv preprint arXiv:2603.18897, 2026. URL https://arxiv.org/abs/2603.18897.
  20. [23]The Linux Kernel Documentation. Kernel samepage merging. https://docs.kernel.org/admin-guide/mm/ksm.html, 2026. Accessed: 2026-10-25.
  21. [25]Wei, X., Lu, F., Wang, T., Gu, J., Yang, Y., Chen, R., and Chen, H. No provisioned concurrency: Fast RDMA-codesigned remote fork for serverless computing. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pp. 497–517, 2023.arxiv.org/abs/2203.10225
  22. [27]Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024. doi: 10.48550/arXiv.2404.07972. URL https://arxiv.org/abs/2404.07972.
  23. [29]Yan, X., Jiao, Z., Liu, J., Wan, Z., Ma, S., Yu, X., Jiang, T., Zhang, C., et al. CUA-Sandbox: Efficient environments for computer-use agent reinforcement learning. arXiv preprint arXiv:2609.32750, 2026. URL https://arxiv.org/abs/2609.32750.
  24. [32]Zhai, Y., Shen, D., Luo, J., and Yang, B. ToolCaching: Towards efficient caching for LLM tool-calling. arXiv preprint arXiv:2601.15335, 2026. URL https://arxiv.org/abs/2601.15335.
  25. [33]Zhang, G., Geng, H., Yu, X., Yin, Z., Zhang, Z., Tan, Z., Zhou, H., Li, Z., et al. The landscape of agentic reinforcement learning for LLMs: A survey. Transactions on Machine Learning Research, 2026a. URL https://arxiv.org/abs/2509.02547.
  26. [34]Zhang, M., Li, Y., Zhang, M., Zhu, S., Zhang, P., Hu, X., Chen, J., Wang, Z., et al. VenusRL: A fully disaggregated agentic RL system with priority scheduling and scalable interaction. arXiv preprint arXiv:2610.03286, 2026b. URL https://arxiv.org/abs/2610.03286.
  27. [35]Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., Alon, U., and Neubig, G. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2307.13854.

Paper details

vlnc:W_22yamavc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents