betaFeedback

WorksComputer Science

Rose: Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Abstract

Embodied reinforcement learning (RL) coordinates policy inference, simulation, and model training with highly heterogeneous resource demands. Although individual components frequently underutilize GPU compute during execution, existing systems only overlap work at coarse component boundaries (e.g., pipelining inference and simulation across simulator groups), leaving substantial low-activity GPU windows within subcomponent phases and nested loops. Exploiting these fine-grained windows manually requires tedious dependency analysis and workload-specific coordination. Furthermore, even when independent work is scheduled concurrently, cross-process GPU time slicing and streaming multiprocessor (SM) contention can negate the benefits of overlap. We present Rose, a runtime-aware system that automates fine-grained overlap and GPU resource management for embodied RL while preserving synchronous training semantics. Through a lightweight programmable interface that exposes execution hierarchy and state accesses, Rose tracks runtime data readiness and GPU activity to perform a two-stage optimization. First, a window-guided graph optimizer identifies low-activity windows and applies graph transformations to overlap complementary work across components and loop levels. Second, leveraging NVIDIA Multi-Process Service (MPS) for cross-process GPU concurrency, a slack-aware resource manager tunes process-level SM quotas by exploiting branch synchronization slack to mitigate interference on the critical path. Both stages combine local profiling with end-to-end iteration validation to guarantee throughput gains with minimal tuning overhead. Across diverse policies, simulators, RL algorithms, and GPU platforms, Rose achieves 1.20–1.92×\times training throughput improvement and increases average SM activity by 23.8–39.3% compared with the state-of-the-art systems.

Introduction

Reinforcement learning is increasingly used to train embodied policies for complex robotic tasks. Through trial-and-error interaction with simulators, RL can improve task performance and execution robustness [5, 12, 14].

Training these policies with RL requires coordinating multiple components with heterogeneous resource demands across rollout and policy training. As illustrated in Figure 1, rollout combines policy inference, simulation, and reward computation, while policy training performs model updates. Policy training and backbone inference require substantial GPU computation, while simulation combines physics computation and rendering with CPU and GPU demands that vary across simulators. Moreover, a component’s execution can be divided into phases corresponding to its subcomponents. GPU demand varies considerably within these phases, which leave GPU resources underused during parts of their execution.

Figure 1. Embodied RL system and heterogeneous components.

This variety motivates placing components with complementary resource demands on the same GPUs to share compute resources, a placement strategy known as colocation. Existing systems such as RLinf [31] support flexible component placement and pipeline policy inference with simulation across simulator groups to improve utilization for colocation [32]. Nevertheless, profiling RLinf training a π0.5\pi_{0.5} [2] policy on LIBERO [13] with this pipeline reveals substantial remaining windows. Device-wide SM activity stays below 50% for 38.8% of the policy-simulator interaction loop execution time and 59.7% of the complete iteration time (Figure 3). Nsight Systems traces locate the interaction-loop windows in the action-head phase at the end of policy inference.

Figure 3. SM activity over a complete training iteration with simulator-group pipeline. The lower panel shows policy-simulator interaction spread from the upper rollout.

These pipelines schedule entire component invocations, leaving internal phases outside the scheduler’s visibility. Our key insight is that an independent phase that can exploit a low-activity window may belong to the same component, another component, or a different loop level. Here, independence means the phase can be moved into the target window without breaking data and state dependencies. For example, the backbone phase of one pipeline group may overlap with the action-head phase of another group once the former’s observations are ready. Exploiting such opportunities requires exposing phase boundaries, intermediate-data readiness, and state constraints.

However, scheduling independent phases does not by itself ensure concurrent GPU execution. Kernels submitted by separate component processes may still execute through GPU time slicing. NVIDIA Multi-Process Service (MPS) enables concurrent kernel execution across processes [22], but resource contention can offset the benefits of overlap. The resulting slowdowns can affect downstream execution differently. An earlier-finishing branch may have scheduling slack: it can tolerate some slowdown without delaying dependent downstream phases. This slack can guide which process to constrain to reduce interference on the branch that determines downstream progress. Because quotas apply to entire processes, their effects extend beyond individual overlap regions and must be evaluated over the complete iteration.

We present Rose, a runtime-aware system for optimizing execution graphs and GPU resource sharing in colocated embodied RL. Through a programmable interface, developers wrap their existing implementations to expose execution regions and loops and annotate state accesses. Rose records invocation-level dependencies, intermediate-data readiness, and annotated state accesses at runtime. The execution hierarchy provides the structure for locating transformation opportunities, while runtime records supply the constraints for checking their legality. Rose uses this information to construct overlap schedules without requiring developers to implement workload-specific scheduling or coordination.

Rose performs optimization in two stages. First, it identifies low-activity windows from GPU activity measurements and uses the execution hierarchy to locate candidate phases and enclosing loops where graph transformations may expose overlap. Generic transformations partition and reschedule the exposed regions while preserving data and state dependencies and training update boundaries. Local profiling checks whether candidates shorten the completion time of affected regions. Promising transformations are composed into candidate graphs. Second, with the selected graph fixed, measured concurrent execution costs and branch slack guide process-level SM quota tuning. Both stages retain changes only when complete-iteration measurements confirm an improvement. Candidate limits and measurement reuse reduce tuning cost. The evaluated policies use fixed inference steps per action chunk without variable-length autoregressive decoding. This regularity supports keeping the selected graph and quotas fixed during subsequent training.

We evaluate Rose across models, simulators, algorithms, and GPU platforms. Against the strongest evaluated RLinf configuration and the asynchronous baseline RL-VLA3^{3}, Rose achieves 1.20–1.92× higher end-to-end throughput, measured in simulator steps/s over complete training iterations. It also increases SM activity, averaged over time and across GPUs, by 23.8–39.3%. Our contributions are:

  • A hierarchical execution model with composable transformations. We combine developer-exposed execution structure and state annotations with invocation-level dependencies and data readiness to support graph transformations that preserve dependencies and training update boundaries.

  • Window-guided graph optimization. We use low-activity windows and execution hierarchy to discover overlap opportunities, check candidate transformations through local measurements, and select execution graphs through complete-iteration evaluation.

  • Slack-aware GPU resource optimization. We combine branch slack and measured concurrent execution costs to guide process-level SM quota tuning, using a coarse-to-fine search and accounting for quota effects beyond the targeted overlap region.

Background and Related Works

Embodied Reinforcement Learning

Embodied RL combines three main components: (1) policy inference, (2) environment simulation, and (3) policy optimization, denoted as Policy, Simulation, and Training in Figure 2, respectively. The first two components jointly perform rollout collection, i.e., the policy generates actions, while the environment executes them and returns new observations and rewards. Repeated interactions produce trajectories, which policy optimization uses to update the policy toward maximizing rewards. Colocated placement shares GPUs among simulator, policy inference, and training. Disaggregated placement assigns components to separate GPU subsets. Hybrid placement assigns simulation and inference to separate GPU subsets during rollout and reuses GPU resources for training. Simulator-group pipelining interleaves inference and simulation across simulator groups. In an asynchronous configuration, components execute asynchronously on separate GPU subsets, allowing rollout and training to overlap.

Figure 2. Embodied RL execution modes.

Policy Inference.

Policy inference combines a vision-language backbone with an action generation module to map observations and task instructions to actions [2, 3, 9, 10, 20]. Two stages have different GPU demands, and action generation may contain repeated decoding or refinement steps. Fixed output sizes and generation budgets make the execution structure tend to be regular.

Environment Simulation.

Environment simulation combines physics stepping, rendering, reward computation, and episode resets. LIBERO and RoboCasa use CPU physics with GPU rendering [13, 19, 35], while ManiSkill3 and BEHAVIOR-1K support GPU physics and rendering [11, 28]. Environment instances run in parallel, but interaction steps within each instance retain sequential data and state dependencies.

Policy Optimization.

Policy optimization performs repeated minibatch updates over collected trajectories [12, 24, 25], involving GPU-intensive forward and backward computation. Trajectory availability and reuse, policy-version requirements, and parameter-update order constrain its overlap with rollout.

Related Works

RL Systems. OpenRLHF [6], HybridFlow [26], and ReaL [16] improve RL efficiency through flexible placement and parameter resharding between rollout and training, while RLHFuse [34] and AReaL [4] overlap stages via stage fusion and asynchronous rollout, respectively. RLinf [31] extends such flexibility to embodied RL (Figure 2). These systems choose among manually designed execution modes, whereas Rose derives overlap from the workflow graph.

Asynchronous Embodied RL Systems. RL-VLA3 [27] and EBRL [17] modify RL algorithms for asynchronous execution, overlapping simulation, inference, and training, with EBRL further adjusting CPU and GPU quotas at runtime. Such modifications potentially compromise convergence. In contrast, Rose preserves synchronous algorithms and derives overlap via graph transformations, without changing the algorithm’s semantics.

Graph-based Execution and Scheduling. Dataflow systems [1, 7, 18] express computation as dependency graphs, and optimizers [8, 30, 33] and schedulers [15, 29] search for semantics-preserving transformations and map task DAGs onto heterogeneous resources. Rose applies these ideas to embodied RL, where nested loops and policy-version dependencies constrain valid transformations.

Motivation

Low-Activity Windows Across Execution Levels

Whole-component scheduling leaves GPU resources underused within components and across loop levels, as illustrated in Figure 3. During one group’s action-head execution, another group’s backbone inputs may already be ready. Across rollout–epoch boundaries, training may overlap with environment reset when trajectory data are available and state accesses are independent. Exploiting these fine-grained overlap opportunities raises two key challenges:

Challenge-1: Determining legal overlap plans. Reordering must preserve data and state dependencies, the policy versions used for rollout, and training update boundaries.

Challenge-2: Selecting beneficial overlap plans. Partitioning and rescheduling introduce multiple candidate graphs. Downstream readiness can affect the benefits of combined transformations, and exhaustive evaluation is costly.

GPU Concurrency and Resource Interference

GPU time slicing. GPU execution may remain time-sliced even when processes overlap. Figure 4 compares serial and overlapping execution of policy inference and simulation. Without MPS, overlap reduces their joint completion time for LIBERO, but provides no reduction for ManiSkill. For ManiSkill, we observe time-sliced GPU execution across separate CUDA contexts despite process-level overlap, as illustrated in Figure 4(c).

Different execution modes and latency of CPU-based and GPU-based simulators

Figure 4. Different execution modes and latency of CPU-based and GPU-based simulators. (a) π0.5\pi_{0.5} and LIBERO, batch=16. (b) π0.5\pi_{0.5} and ManiSkill, batch=80. (c) Schematic illustration of time-sliced GPU execution across CUDA contexts without MPS.

Interference under GPU concurrency. NVIDIA MPS enables cross-process GPU concurrency [22], but colocated workloads still share execution resources. Table 1 shows that, with MPS enabled, overlapping policy inference and ManiSkill simulation increases their execution time by 1.1% and 44.4%, respectively, relative to isolated execution. As simulation determines their joint completion time, the resulting speedup is limited to 1.07×. Inference finishes earlier than simulation, leaving branch slack before joint completion. To reduce interference raises the following challenge:

SimulatorMPSInf. alone (s)Sim. alone (s)Serial (s)Inf. overlap (s)Sim. overlap (s)Speedup
LIBEROOff0.5680.2890.8540.6270.3161.36×
ManiSkillOn2.6324.9817.6732.6617.1921.07×

Table 1. Policy inference and simulator execution times. The reported speedup is Tserial/max⁡(Tinf,overlap,Tsim,overlap)T_{\mathrm{serial}}/\max(T_{\mathrm{inf,overlap}}, T_{\mathrm{sim,overlap}}).

Challenge-3: Tuning SM quotas under interference. Quota tuning must balance branch slack, concurrent execution costs, and effects beyond the targeted overlap region.

Design

Overview

As illustrated in Figure 5, Rose exposes execution hierarchy and forms execution graphs at runtime (§ 4.2). With the metrics of the execution graph, Rose performs a two-stage optimization: graph-based overlap and slack-aware resource management. The graph optimizer identifies low-activity nodes, searches for complementary work, and transforms the execution graphs to enable overlap (§ 4.3). With the execution graph fixed, the resource manager tunes the quota of a GPU process with slack to reduce interference between overlapping processes (§ 4.5).

Overview of Rose

Figure 5. Overview of Rose. ① Code wrapping to expose hierarchy. ② The original graph registration. ③ Graph execution dispatch and measurement. ④ Collects dependencies and metrics for optimization. ⑤ The graph optimizer retrieves graph, transforms them, and registers back. Validated graphs are retained and this may run for multiple rounds. ⑥ The resource manager retrieves the best graph, tuning the selected process SM quota, and stores the best quota back. ⑦ The buffer outputs final plan for subsequent RL training.

Programmable Execution Model

Rose provides a hierarchical execution graph representation of existing RL programs through a programming interface. Graph construction combines the exposed structure with state annotations and invocation records to establish dependencies and loop independence, enabling overlap across components and execution levels.

Hierarchical Execution Graph Representation

Rose provides ExecutionNode and LoopNode to represent execution as hierarchical execution graphs, as shown in Figure 6a. An ExecutionNode wraps a code block in a component worker with a callable entry and may expose an internal graph of child nodes, data bindings, and execution constraints. Figure 6b illustrates this wrapping for policy code. Parent-child relations form an execution hierarchy rooted at the RL iteration loop, supporting analysis at different granularities. LoopNode extends ExecutionNode: its internal graph describes the loop body, while its iteration count specifies repetition. Loops with conditional termination, such as while loops, are wrapped with ExecutionNode to retain the native loop control. Accordingly, edges represent execution constraints that preserve required data and state dependencies.

Execution node wrapper and wrap example

Figure 6. (a) Execution node wrapper. (b) An example of wrapping policy code, can also wrap new component code.

An invocation is a distinct execution of a node, which recurs across loop iterations or input groups. A node defines the exposed computation and callable entries, while invocation records associate each execution with its parent invocation, loop iteration, data references, and state accesses.

Execution Graph Construction

Rose constructs execution graphs by combining the exposed hierarchy with invocation-level bindings.

Invocation recording. Figure 7 shows invocation recording through ExecutionTrace, using the hooks called in Figure 6a. Each record identifies the structural node, parent invocation, and current iteration. The before hook binds inputs to data references and records state accesses, while after binds outputs. These records link invocations to the hierarchy and state dependency analysis.

Execution trace wrapper for invocation recording

Figure 7. Execution trace wrapper for invocation recording.

Data dependencies. Rose uses data references to identify which invocation produces each input, establishing producer-consumer dependencies. For loops, it records outputs at the end of each iteration, allowing consumers to proceed once their required data are ready without waiting for the entire loop to finish. For example, trajectory data from each rollout epoch can become available before the rollout returns the full collection.

State dependencies. Beyond explicit inputs and outputs, nodes may access shared objects such as model weights and simulator state. Rose provides three annotations to describe such accesses: read existing state, update new state from its previous value, and refresh new state without inheriting its previous value. Each annotation applies to a specific state object. The system binds these annotations to invocation records to derive state dependencies.

Loop independence. Rose analyzes data and state dependencies across iterations to determine loop independence. For a LoopNode, the classification is assigned by dependency analysis, as indicated by its kind field in Figure 6.

The simulator reset annotated as refresh indicates that the new epoch’s episode state does not inherit the previous epoch’s state value. If the remaining data and state constraints also permit independent execution, the system identifies the loop as a free loop. A loop with dependencies across iterations is classified as a dependent loop. Loops with conditional termination, such as while loops, are classified as dependent loops without an iteration count.

The resulting execution graphs and loop independence information provide the graph optimizer with the execution structure and constraints for reorganization.

Window-guided Graph Optimization

Building on the execution graphs, Rose identifies low-activity windows and searches for work that can execute during these windows while respecting data and state dependencies. It constructs overlap schedules through composable graph transformations (Figure 8), uses local measurements to guide candidate selection and composition, and retains only changes validated to reduce end-to-end iteration time. Graph optimization examples are illustrated in Figure 9.

Figure 8. Transformation APIs.

Figure 9. Optimization examples. Black arrows indicate execution dependencies. (a) Split simulator-policy interact loop and interleave them. (b) Split training batch loop and rewire it to overlap simulator reset. (c) Rewire to overlap communication inside simulator-policy interaction node. (d) Rewire inside a component node between independent component graphs. Reward computation is a part of complete simulator computation.

Graph Transformation APIs

Rose provides four execution graph transformation APIs: Rewire, Split, Merge, and Interleave. The transformations reorganize execution graphs while reusing nodes’ callable entries and preserving required data and state dependencies.

Rewire. Rewire changes the execution order of existing nodes while preserving required data and state dependencies. It applies to both loop and non-loop nodes whose inputs and state permit the new placement.

Split. Split constructs separately schedulable executions of a node or its internal graph, each bound to a data partition or an independent group of loop iterations. Spatial split partitions batched computation into groups with separable data and state while preserving each group’s iteration order. Temporal split organizes independent iterations of a free loop into separate execution graphs while preserving the original batch sizes.

Merge. Merge collects results from the groups or execution graphs produced by Split to reconstruct the input required by the next unsplit node.

Interleave. Interleave staggers nodes across the execution graphs produced by Split to enable overlap while preserving dependencies within each graph. For example, a legal Split of a policy-simulator step produces data groups A and B, and Interleave makes simulator(A) overlap policy(B), while each group retains its policy-before-simulator ordering.

Overlap Candidate Discovery

Rose uses the execution intervals of node invocations as candidate overlap windows. To limit candidate trials, it first ranks nodes by duration and SM activity over these windows, then searches for work that can overlap under data and state constraints, and finally orders feasible candidates by duration matching.

Ranking low-activity nodes. For a schedulable node vv, DvD_v is its total invocation duration within one training iteration, and aˉv\bar{a}_v is the time-weighted average SM activity observed over these invocations. Motivated by NVIDIA’s profiling metrics documentation [21], Rose excludes nodes with aˉv≥0.8\bar{a}_v \ge0.8 and prioritizes those with aˉv<0.5\bar{a}_v < 0.5 over those with 0.5≤aˉv<0.80.5 \le\bar{a}_v < 0.8. Within each activity tier, Rose ranks nodes by decreasing S(v)=Dv(1−aˉv)S(v) = D_v(1-\bar{a}_v). The resulting rank π(v)\pi(v) incorporates both the activity-tier priority and the ordering within each tier. Eligible nodes must also satisfy Dv≥min⁡{τa,τrTiter}D_v \ge\min\{\tau_a,\tau_r T_{\mathrm{iter}}\}, where τa\tau_a is an absolute duration threshold and τr\tau_r specifies a fraction of the observed iteration time TiterT_{\mathrm{iter}}.

Candidate search. For each target node vv selected by this ranking, Rose searches for a candidate node ww whose work can overlap with vv through feasible graph transformations. When vv’s parent is a LoopNode, Rose first searches its internal graph and tries Split followed by Interleave. Temporal split uses independent iterations from the parent loop or legal free ancestors, considered from nearest to farthest. If no free loop exists, Rose uses Spatial split. It then searches the loop’s parent’s internal graph in this order: other movable loops, splittable downstream non-loop consumers, and unsplittable non-loop nodes that permit Rewire.

When vv’s parent is non-loop, Rose first checks whether vv can be repositioned, then searches other nodes in the parent’s internal graph. If no feasible candidate exists, it searches the parent’s siblings. Feasibility requires checking transformation applicability, data and state constraints, and estimated capacity.

Candidate ordering. Candidate ordering uses node rank π(v)\pi(v) followed by increasing duration mismatch δ(v,w)=∣T(v)−T^(w)∣\delta(v,w)=|T(v)-\widehat{T}(w)|. Here, T(v)T(v) is the mean invocation duration of vv, representing its average window duration, and T^(w)\widehat{T}(w) estimates the candidate’s duration after any required Split. This ordering determines which alternatives receive measurements under the per-node trial budget.

Candidate Evaluation and Graph Composition

Since local improvements do not guarantee reduced iteration time, Algorithm 1 combines local measurements for graph transformation and complete-iteration validation to accept or reject the result.

Algorithm 1 Window-guided graph optimization

Require: Feasible graph G0G_0; settings MM, τ\tau

Require: Target limit K≥1K \ge1; alternative limit per target L≥1L \ge1

Require: Round limit R≥1R \ge1; improvement threshold ϵ∈(0,1)\epsilon\in(0,1)

Ensure: Graph GG selected for subsequent training

  1. G←G0G \gets G_0

  2. for r=1,…,Rr = 1,\ldots,R do

  3. mref←RunTraining⁡(G)m_{\mathrm{ref}} \gets\operatorname{RunTraining}(G); D←[ ]\mathcal{D} \gets[\ ]

  4. for all v∈TopK⁡(G,S(v),τ)v \in\operatorname{TopK}(G,S(v),\tau) do

  5. Φv←Top⁡LFeasibleTransform⁡(G,v);(π,δ)\Phi_v \gets\operatorname{Top}_L\operatorname{FeasibleTransform}(G,v);(\pi,\delta)

  6. if Φv=∅\Phi_v = \varnothing then continue

  7. Av←CommonGraph⁡(G,v,Φv)A_v \gets\operatorname{CommonGraph}(G,v,\Phi_v)

  8. c0←Avc_0 \gets A_v

  9. Cv←{c0}∪{Transform⁡(Av,ϕ):ϕ∈Φv}C_v \gets\{c_0\} \cup\{\operatorname{Transform}(A_v,\phi):\phi\in\Phi_v\}

  10. for all c∈Cvc \in C_v do

  11. mc←MeasureLocal⁡(c)m_c \gets\operatorname{MeasureLocal}(c)

  12. end for

  13. D[v]←(c↦mc:c∈Cv)\mathcal{D}[v] \gets(c \mapsto m_c:c \in C_v)

  14. end for

  15. Gcomb←Combine⁡(G,D,M)G_{\mathrm{comb}} \gets\operatorname{Combine}(G,\mathcal{D},M)

  16. if Gcomb=GG_{\mathrm{comb}} = G or ¬Valid⁡(Gcomb)\neg\operatorname{Valid}(G_{\mathrm{comb}}) then

  17. break

  18. end if

  19. mcomb←RunOrReuse⁡(Gcomb)m_{\mathrm{comb}} \gets\operatorname{RunOrReuse}(G_{\mathrm{comb}})

  20. if mcomb.time≥(1−ϵ)mref.timem_{\mathrm{comb}}.\mathrm{time} \ge(1-\epsilon)m_{\mathrm{ref}}.\mathrm{time} then

  21. break

  22. end if

  23. G←GcombG \gets G_{\mathrm{comb}}

  24. end for

  25. return GG

All targets in a round use the same reference graph GG, the complete execution graph of an RL iteration. For each target vv, FeasibleTransform⁡(G,v)\operatorname{FeasibleTransform}(G,v) performs the candidate search described above and returns feasible transformation alternatives ϕ(v,w)\phi(v,w) for overlapping vv with ww. The retained alternatives share a comparison graph AvA_v: the internal graph of the lowest common ancestor enclosing all nodes affected by these alternatives. Transform⁡(Av,ϕ)\operatorname{Transform}(A_v,\phi) applies a retained alternative to AvA_v, producing a candidate local graph cc. Both the transformed graphs and the unchanged graph c0=Avc_0=A_v are measured over this common comparison graph. Their completion times account for downstream readiness, reconstruction, and synchronization.

Combine uses the measured reductions in completion time to greedily compose changes that can be applied together, selecting at most one candidate per target. Combined graphs are ranked by estimated cumulative benefit and checked against the memory settings MM before complete-iteration validation. A combined graph is accepted only when a valid observation shows a relative reduction in iteration time exceeding ϵ\epsilon compared with GG. RunOrReuse reuses only end-to-end observations of the same complete graph. Each complete-iteration trial advances training and updates the weights. Its execution trace, time, and validity guide subsequent optimization rounds.

Algorithm 2 Slack-aware GPU resource optimization

Require: Fixed graph GG; target process ff; comparison graph AA

Require: Initial numerator k0k_0; quota size QQ

Require: Coarse-step multiplier ρ≥2\rho\ge2

Ensure: Selected quotas q∗q^* and measurement m∗m^*

  1. kb←k0k_b \gets k_0

  2. H←{kb↦t(kb)}\mathcal{H} \gets\{k_b \mapsto t(k_b)\} {Coarse search}

  3. while kb>1k_b > 1 do

  4. k←max⁡(1,kb−ρ)k \gets\max(1,k_b-\rho); H[k]←t(k)\mathcal{H}[k] \gets t(k)

  5. if H[k]>H[kb]\mathcal{H}[k] > \mathcal{H}[k_b] then break

  6. kb←kk_b \gets k

  7. end while {Fine search around the anchor kbk_b}

  8. for all d∈{−1,+1}d \in\{-1,+1\} do

  9. k←kb+dk \gets k_b+d

  10. while 1≤k≤Q1 \le k \le Q and k∉dom⁡(H)k \notin\operatorname{dom}(\mathcal{H}) do

  11. H[k]←t(k)\mathcal{H}[k] \gets t(k)

  12. if H[k]>H[kb]\mathcal{H}[k] > \mathcal{H}[k_b] then break

  13. k←k+dk \gets k+d

  14. end while

  15. end for {End-to-end validation}

  16. k∗←arg⁡min⁡k∈dom⁡(H) H[k]k^* \gets\underset{k \in\operatorname{dom}(\mathcal{H})}{\arg\min}\ \mathcal{H}[k]

  17. (q∗,m∗)←Validate⁡(k∗)(q^*,m^*) \gets\operatorname{Validate}(k^*)

  18. return (q∗,m∗)(q^*,m^*)

Slack-aware Resource Management

GPU contention can slow overlapping branches unequally while downstream work waits for both. The earlier-ready branch can tolerate some delay without postponing the original synchronization, and this tolerance is its branch slack. Rose tunes the earlier-ready process’s SM quota to reduce interference, using local measurements for quota search and complete-iteration validation for acceptance.

For readiness times C0≥F0C_0 \ge F_0 measured from a common origin, the earlier-ready branch has slack σ=C0−F0\sigma=C_0-F_0. After reducing its process quota, write the resulting readiness times as C0−xC_0-x and F0+yF_0+y, respectively. The reduction in synchronization time is ΔTsync=min⁡{x,σ−y}\Delta T_{\mathrm{sync}}=\min\{x,\sigma-y\}. Synchronization advances when the later-ready branch accelerates (x>0x>0) and the earlier-ready branch’s added delay remains within its original slack (y<σy<\sigma), as illustrated in Figure 10.

Figure 10. SM quota management and latency of overlapping execution on π0.5\pi_{0.5} and ManiSkill. “91/100” denotes the percentage quota of policy and simulator.

With GG fixed to the graph selected by Algorithm 1, Rose uses a feasible observation m0m_0 under quotas q0q_0 to select two overlapping branches on distinct processes sharing a GPU. It tunes only the earlier-ready process ff, keeping this target and all other quotas fixed throughout the search. If no eligible pair exists, Rose retains q0q_0. The comparison graph AA is the internal graph of the lowest common ancestor enclosing both branches and their synchronization.

Uniform MPS provisioning caps the SM resources available to client CUDA contexts. The backend provides QQ validated quota levels (§B). q(k)q(k) retains q0q_0 except that ff’s quota is set to k/Qk/Q, for k∈{1,…,Q}k \in\{1,\ldots,Q\}. Algorithm 2 records in HH the locally measured completion time t(k)t(k) of AA under (G,q(k))(G,q(k)), covering the same effective work across quotas.

Algorithm 2 first reduces the quota in coarse steps of ρ\rho levels until completion time increases or the minimum quota is reached. It then searches in both directions with unit steps around the best coarse-search point, held fixed as the anchor; a direction stops when its measured time exceeds the anchor’s. The measured quota with the smallest t(k)t(k) is passed to Validate. Since a process quota also affects work outside AA, Validate retains the selected quota only if a complete iteration is faster than m0m_0; otherwise, it returns q0q_0. Measurements follow the execution semantics in § 4.4.1.

Evaluation

We evaluate Rose in terms of throughput, scalability, optimization effectiveness and overhead, as well as its impact on policy learning.

Experimental Setup

Rose is built on RLinf with 5K lines of Python code, adding ExecutionNode and ExecutionTrace wrappers, and graph transformation implementation. It applies GPU quotas through MPS provisioning at a granularity of two SMs, as detailed in §B.

Testbeds. We evaluate throughput on two 8-GPU platforms: NVIDIA H100 GPUs with Intel Xeon Platinum 8481C CPUs, and NVIDIA A800 GPUs with Intel Xeon Platinum 8358P CPUs. Each GPU has

ModelSimulatorAverage SM activity
π0.5\pi_{0.5}LIBEROBaselineRose
ManiSkill46.38%58.01%
OpenVLA-OFTLIBERO54.98%68.08%
ManiSkill43.59%56.57%

Table 2. GPU execution efficiency on A800.

80 GB memory, and each node provides 920 GB host memory. Scaling experiments use 4 and 8 A800 GPUs within one node and 16 GPUs across two nodes, connected via NVLink within nodes and 8×200 Gbps RDMA NICs across nodes.

Workloads and metrics. We evaluate π0.5\pi_{0.5} [2], OpenVLA-OFT [9], and GR00T N1.6 [20] on LIBERO-Spatial [13] and ManiSkill [28] (“PutOnPlateInScene25Main-v3”), using PPO [24] and GRPO [25]. Across systems, we match rollout volumes and policy-update budgets using benchmark-recommended configurations. We report end-to-end throughput (simulator steps/s) and average SM activity across GPUs over each iteration.

Baselines. We compare Rose against four RLinf configurations and RL-VLA³ [27]. The RLinf configurations are explained in section 2 and denoted as Colocated, Colocated+Pipe, Hybrid, and Hybrid+Pipe.

End-to-End Performance

Training throughput. Figure 11 reports throughput for the twelve workload configurations on H100 and A800.

Figure 11. Training throughput.

(1) H100. Across PPO and GRPO, Rose achieves up to 1.39×, 1.43×, and 1.44× the RLinf throughput with the best configuration on LIBERO for π0.5\pi_{0.5}, OpenVLA-OFT, and GR00T N1.6, respectively. On ManiSkill, the corresponding speedups are 1.20×, 1.27×, and 1.22×. (2) A800. Across PPO and GRPO, Rose achieves up to 1.42×, 1.27×, and 1.33× the RLinf throughput on LIBERO for the same models. On ManiSkill, the corresponding maximum speedups are 1.29×, 1.20×, and 1.24×. (3) Rose also achieves up to 1.92× and 1.61× the throughput of RL-VLA3 on LIBERO and ManiSkill across platforms respectively.

We observe that enabling simulator-group pipelining reduces throughput in RLinf’s colocated configuration on ManiSkill. Grouping environments reduces the batch size processed by each component. Policy inference time decreases approximately linearly with batch size, whereas the less compute-intensive GPU simulation sees only a modest reduction. Executing the additional simulator group therefore substantially increases cumulative simulation time, making simulation the bottleneck. As discussed in Section 3, this configuration still time-slices GPU execution across policy and simulator processes, thus providing insufficient concurrent GPU execution to offset the increased simulation cost, resulting in lower throughput.

GPU execution efficiency. We plot the SM activity for a complete training iteration after automatic optimization in Figure 12. SM activity remains below 50% for 8.4% of the interaction-loop execution time and 21.3% of the complete iteration time, corresponding to relative reductions of 78.4% and 64.3% in these time fractions, respectively. The remaining low-activity windows persist because no complementary and dispatchable work is available to fill them. Table 2 compares average SM activity between Rose and RLinf with the best configuration for π0.5\pi_{0.5} and OpenVLA-OFT on LIBERO and ManiSkill. Across configurations, Rose improves SM activity by 23.8–39.3% on average.

Figure 12. SM activity after Rose optimization over a training iteration. The lower panel is spread from part of the upper rollout.

Scaling. We evaluate scaling on four, eight, and sixteen A800 GPUs on π0.5\pi_{0.5} PPO training with two simulators. The workload for each rank remains fixed, so the total simulator and training workloads increase with the GPU count under data parallelism. Figure 13 compares Rose with different RLinf configurations for each simulator and GPU count. On LIBERO, Rose improves throughput over the best configuration by 46.5%, 42.2%, and 40.2% on four, eight, and sixteen GPUs, respectively. On ManiSkill, the corresponding improvements are 31.5%, 29.2%, and 29.8%.

Figure 13. Training throughput on 4-16 GPUs.

Ablation Study

We evaluate MPS and quota optimization on ManiSkill using the same workload and component placement: (1) Colocated, the best RLinf configuration; (2) Graph-only, with graph optimization but no MPS; (3) MPS-on, with the optimized graph and default MPS quotas; and (4) Full Rose, which additionally tunes the selected process’s SM quota.

Figure 14 (a) reports the throughput of the four variants. Graph optimization alone achieves 1.08×, 1.10× and 1.06× the throughput of Colocated. With MPS enabled, the speedup becomes 1.12×, 1.13× and 1.10×, increasing by 3.4%, 2.3% and 3.9%. Finally, SM quota optimization brings the overall speedup of Full Rose to 1.29×, 1.20× and 1.24×, increasing by 15.6%, 6.7% and 12.6%. All speedups use Colocated as the baseline. The speedup breakdown of three methods is 33.1%, 13.5%, 53.4% for π0.5\pi_{0.5}, 54.9%, 12.1%, 33.0% for OpenVLA-OFT, and 29.2%, 18.3%, 52.5% for GR00T.

Training throughput ablation

Figure 14. Training throughput ablation.

Figure 15 shows how local latency varies while tuning SM quota for all models. Local latency is measured over an overlap execution window spanning four action chunks. The x-axis denotes the SM quota of the selected process with slack. The full-range sweep shows that overly restrictive quotas substantially increase latency, while the default quota (100%) is suboptimal. Zooming into the 80–100% range reveals a U-shaped latency curve, with minima at 95%, 91%, and 89%, respectively.

Figure 15. Latency during quota tuning.

We additionally evaluate the benefit of using multiple graph optimization rounds. There are three variants: (1) Colocated+Pipe. The best configuration for RLinf on LIBERO. (2) Rose (one round). Execution graph after optimization for one round. (3) Full Rose. Execution graph after completing two rounds of optimization. Figure 14 (b) reports the throughput of the variants. The first and second rounds contribute 60.7% and 39.3% of the total reduction in iteration time for π0.5\pi_{0.5}, 62.9% and 37.1% for OpenVLA-OFT, and 67.6% and 32.4% for GR00T, respectively.

Optimization Overhead

We measure optimization overhead with the target limit set to K=3K=3. For both workloads, tuning includes an end-to-end reference profile and two rounds of graph optimization, each followed by end-to-end profiling. ManiSkill additionally requires seven short MPS quota trials and a final validation. We distinguish total optimization time from net overhead by subtracting training execution whose progress is retained. For π0.5\pi_{0.5} with PPO on LIBERO, two rounds of graph optimization take 1,998 s, including 1,336 s of training execution, leaving 662 s of net overhead. The policy reaches a 90.2% success rate after 252 iterations in 85,727 s, including optimization. Total time and net overhead therefore account for 2.3% and 0.8% of the complete run, respectively. For ManiSkill, the total optimization time is 4,615 s, including 2,290 s of retained training execution and 2,325 s of net overhead. Relative to its complete training time of 100,716 s to reach a 90.0% success rate, total time and net overhead correspond to 4.6% and 2.3%, respectively.

Conclusion

We presented Rose, a runtime-aware system for automatic overlap optimization in embodied RL. Combining a programmable interface with runtime measurements, Rose performs two-stage optimization through a window-guided graph optimizer and a slack-aware resource manager while preserving training semantics. Evaluations demonstrate substantial improvements in training throughput and GPU utilization.

Runtime

Runtime characteristics. The data plane records host-call duration for every invocation and additionally records device time when available. Separately, it maintains a device-wide SM activity ([21]) trace throughout the iteration for the GPU associated with each rank. For an invocation with device activity, the duration extends from the host start to the later of host return and device completion. Parent and child nodes provide alternative observation scopes but their measurements are not added as independent work.

Replacement of callable graph. The runtime implements a node’s transformed internal graph as an alternative callable. A transformed graph must execute the node’s complete effective work while preserving its output interface and state effects. A hook at the node’s invocation entry selects this callable when registered. Each invocation executes either the native graph or its replacement, never both.

Dependency-driven execution. An invocation is submitted only when its input, state, and scheduling requirements are satisfied. Output readiness accounts for required device computation and communication. A host return alone does not establish GPU output readiness. Intermediate buffers remain live until their required consumers complete.

Local validation. Local validation compares the reference and transformed graphs over the same workload with comparable input and mutable state prepared. Preparation is excluded from regional timing but included in optimization overhead. Local trials do not commit their mutations to the ongoing training state.

Implementation

Rose extends RLinf [31] with 5K lines of Python, adding execution-node wrappers, the ExecutionTrace data plane, graph transformations and runtime support.

Execution integration and bindings. Rose adds an invocation hook to the common ExecutionNode wrapper for synchronous and asynchronous execution. At each invocation, the hook selects the alternative callable registered for the node or its native entry if no alternative is registered. The driver installs these callables and their scheduling metadata on participating workers and launches invocations through RLinf’s distributed execution interface.

Each replacement callable returns results through the original output interface. Spatial split binds invocations to subsets of the original inputs. Temporal split preserves data and state bindings for separate graphs. Fragment outputs retain their positions in the complete result, which Merge uses to reconstruct downstream inputs.

Execution scheduling and completion. Independent execution graphs run as separate asynchronous tasks within each worker. Before execution, tasks wait for the required input, state and scheduling dependencies while allowing other runnable tasks to progress. Cross-worker data transfers reuse RLinf’s routed communication.

For asynchronous entries, the wrapper invokes trace.after from the returned future’s completion callback. Output readiness is tracked separately through backend completion events covering device work and required data transfers.

MPS provisioning and local quota trials. Rose applies quotas through uniform MPS provisioning within each GPU worker process [22]. This ensures quota to take effect for simulator backend that manages CUDA contexts internally such as PhysX, which a Python-level Green Context wrapper cannot reliably constrain. Before launching a worker, Rose sets CUDA_MPS_ACTIVE_THREAD_PERCENTAGE. The resulting limit remains fixed for the client’s lifetime, so quota changes require replacing the affected test worker. Quota trials use short workloads on separately initialized workers. We characterize effective quota levels by querying cudaDevAttrMultiProcessorCount in fresh MPS clients under different percentage settings. On the tested Hopper and Ampere GPUs, the smallest observed SM-count increment was two.

Profiling and timing. Local validation measures execution time without Nsight Systems profiling. Within each comparison, reference and candidate runs use the same workload, timing boundaries, and profiling mode.

During profiled iterations, Nsight Systems [23] collects device-wide SM activity throughout the iteration. NVTX ranges mark exposed invocations only in the first and last few steps of inner loops to reduce recording overhead. Invocation identifiers map these ranges to structural nodes, allowing sampled execution intervals to be aligned with the SM-activity trace. For unsampled invocations, device-associated durations are estimated from sampled invocations of the same node.

Completion time of the comparison graph is the wall-clock span from its designated start to its completion event, including computation, data transfer and synchronization. End-to-end timing uses complete-iteration boundaries.

Optimization overhead. Optimization overhead includes worker and model initialization, graph transformation and installation, data and state preparation, local trials and complete iteration validation. Quota tuning additionally incurs worker replacement and reinitialization, while compatible graph trials reuse initialized workers. These costs are reported in Section 5.

References

  1. [1]Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 265–283. https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadiarxiv.org/abs/1605.08695
  2. [2]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, brian ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. 2025. π0.5: a Vision-Language-Action Model with Open-World Generalization. In Proceedings of The 9th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 305), Joseph Lim, Shuran Song, and Hae-Won Park (Eds.). PMLR, 17–40. https://proceedings.mlr.press/v305/black25a.htmlarxiv.org/abs/2504.16054
  3. [3]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelina Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. 2025. π: A Vision-Language-Action Flow Model for General Robot Control. In Proceedings of Robotics: Science and Systems. Los Angeles, CA, USA. doi:10.15607/RSS.2025.XXI.010
  4. [4]Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhan Yuan, and Yi Wu. 2025. AREL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference. Curran Associates, Inc., 36256–36282. doi:10.52202/0895713-1218arxiv.org/abs/2505.24298
  5. [5]Jiacheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto Martín-Martín, Peter Stone, Kuo-Hao Zeng, and Kiana Ehsani. 2025. FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning. In 2025 IEEE International Conference on Robotics and Automation (ICRA). 3617–3624. doi:10.1109/ICRA55743.2025.11127934DOI
  6. [6]Jian Hu, Xibin Wu, Zilin Zhu, Xianyue Weixun Wang, Dehao Zhang, and Yu Cao. 2024. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143 (2024).arxiv.org/abs/2405.11143
  7. [7]Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. 2007. Dryad: distributed data-parallel programs from sequential building blocks. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 (Lisbon, Portugal) (EuroSys ’07). Association for Computing Machinery, New York, NY, USA, 59–72. doi:10.1145/1272996.1273005DOI
  8. [8]Zhiliao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. 2019. TASO: optimizing deep learning computation with automatic generation of graph substitutions. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (Huntsville, Ontario, Canada) (SOSP ’19). Association for Computing Machinery, New York, NY, USA, 47–62. doi:10.1145/3341301.3359630
  9. [9]Moo Jin Kim, Chelsea Finn, and Percy Liang. 2025. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. In Proceedings of Robotics: Science and Systems. Los Angeles, CA, USA. doi:10.15607/RSS.2025.XXI.017DOI
  10. [10]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025. OpenVLA: An Open-Source Vision-Language-Action Model. In Proceedings of The 8th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 270), Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (Eds.). PMLR, 2679–2713. https://proceedings.mlr.press/v270/kim25c.htmlarxiv.org/abs/2406.09246
  11. [11]Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villan Renteria, Jerry Huuayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. 2023. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In Proceedings of The 6th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 205), Karen Liu, Dana Kulić, and Jeff Ichnowski (Eds.). PMLR, 80–93. https://proceedings.mlr.press/v205/li23a.htmlarxiv.org/abs/2403.09227
  12. [12]Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, Dehui Wang, Dingxiang Luo, Yuchen Fan, Youbang Sun, Jia Zeng, Jiangmiao Pang, Shanghang Zhang, Yu Wang, Yao Mu, Bowen Zhou, and Ning Ding. 2025. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. arXiv:2509.09674 [cs.RO] https://arxiv.org/abs/2509.09674
  13. [13]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 44776–44791. doi:10.52202/075280-1939
  14. [14]Jijia Liu, Feng Gao, Bingwen Wei, Xinglei Chen, Qingming Liao, Yi WU, Chao Yu, and Yu Wang. 2025. What Can RL Bring to VLA Generalization? An Empirical Study. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Konusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference. Curran Associates, Inc., 97121–97151. doi:10.52202/085713-3248
  15. [15]Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI’20). USENIX Association, 881–897. https://www.usenix.org/conference/osdi20/presentation/ma
  16. [16]Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. 2025. ReaLeaf: Efficient RLHF Training of Large Language Models with Parameter Reallocation. In Proceedings of Machine Learning and Systems, M. Zaharia, G. Joshi, and Y. Lin (Eds.), Vol. 7. MLSys.arxiv.org/abs/2406.14088
  17. [17]Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao, Han Xiao, Xin Ding, Mingzhe Huang, Xin He, Lu Shi, Hao Wu, Haipeng Dai, Guihai Chen, Yunxin Liu, and Ting Cao. 2026. EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management. arXiv:2609.27547 [cs.LG] https://arxiv.org/abs/2609.27547
  18. [18]Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martín Abadi. 2013. Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles (Farminton, Pennsylvania) (SOSP ’13). Association for Computing Machinery, New York, NY, USA, 439–455. doi:10.1145/2517349.2522738DOI
  19. [19]Shounak Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. 2024. RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots. In Proceedings of Robotics: Science and Systems. Delft, Netherlands. doi:10.15607/RSS.2024.XX.050DOI
  20. [20]NVIDIA, Johan Bjorck, Fernando Castaneda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushik Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhidong Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. 2025. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734 [cs.RO] https://arxiv.org/abs/2503.14734
  21. [21]NVIDIA Corporation. 2026. NVIDIA DCGM User Guide: Profiling Metrics. Accessed: 2026-10-27. https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#profiling-metrics
  22. [22]NVIDIA Corporation. 2026. NVIDIA Multi-Process Service. Accessed: 2026-10-27. https://docs.nvidia.com/deploy/mps/latest/index.html
  23. [23]NVIDIA Corporation. 2026. NVIDIA Nsight Systems user guide. Accessed: 2026-10-27. https://docs.nvidia.com/nsight-systems/UserGuide/index.html
  24. [24]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347
  25. [25]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeek-Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402.03300
  26. [26]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems (Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 1279–1297. doi:10.1145/3689031.3696075DOI
  27. [27]Haoran Sun, Yongjian Guo, Zhong Guan, Shuai Di, Xiaodong Bai, Jing Long, Tianyun Zhao, Mingxi Luo, Hongke Zhao, Likang Wu, Xiaotie Deng, Xu Chu, Xi Xiao, Sheng Wen, Yicheng Gong, and Junwu Xiong. 2026. RL-VLA³: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training. arXiv:2602.05765 [cs.AI] https://arxiv.org/abs/2602.05765
  28. [28]Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xin­son­g Lin, Yulin Liu, Tse-Kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. 2025. Demonstrating GPU Parallelized Robot Simulation and Rendering for Generalizable Embodied AI with ManiSkill3. In Proceedings of Robotics: Science and Systems. LosAngeles, CA, USA. doi:10.15607/RSS.2025.XXI.021DOI
  29. [29]Haluk Topcuoglu, Salim Hariri, and Min-Yin Wu. 2002. Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems 13, 3 (March 2002), 260–274. doi:10.1109/71.993206DOI
  30. [30]Colin Unger, Zhihai Jia, Wei Wu, Sina Lin, Mandeep Bajnes, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaluddin Mohd-Yusof, Xi Luo, Dheeravtha Mudigere, Jongsoo Park, Misha Smelyanskiy, and Alex Aiken. 2022. Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 267–284. https://www.usenix.org/conference/osdi22/presentation/unger
  31. [31]Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, Chunyang Zhu, Junhao Hu, Xiziao Huang, Mingjie Wei, Yuqing Xie, Ke Yang, Bo Dai, Zhexuan Xu, Jiakun Du, Xiangyuan Wang, Xu Fu, Letong Shi, Zhihao Liu, Kang Chen, Weilin Liu, Gang Liu, Boyun Li, Jianlei Yang, Zhi Yang, Guohao Dai, and Yu Wang. 2026. RLinf: Flexible and Efficient Large-Scale Reinforcement Learning via Macro-to-Micro Flow Transformation. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26). USENIX Association, Seattle, WA, 829–846. https://www.usenix.org/conference/osdi26/presentation/yu-chao
  32. [32]Hongzhi Zang, Mingjie Wei, Si Xu, Yongji Wu, Zhen Guo, Yuanqing Wang, Hao Lin, Peihong Wang, Hua Yuan, Yixian Zhang, Liangzhi Shi, Yuqing Xie, Zhexuan Xu, Zihao Liu, Kang Chen, Wenhao Tang, Quanlu Zhang, Weinan Zhang, Chao Yu, and Yu Wang. 2026. RLux-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models. In Proceedings of Robotics: Science and Systems. Sydney, Australia. doi:10.15607/RSS.2026.XXII.089DOI
  33. [33]Lianmin Zheng, Zhounan Li, Hao Zhang, Yonghao Zhuang, Zhening Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 559–578. https://www.usenix.org/conference/osdi22/presentation/zheng-lianminarxiv.org/abs/2201.12023
  34. [34]Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. 2025. Optimizing RLHF Training for Large Language Models with Stage Fusion. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). USENIX Association, Philadelphia, PA, 489–503. https://www.usenix.org/conference/nsdi25/presentation/zhongarxiv.org/abs/2409.13221
  35. [35]Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Madduri, Soroush Nasiriany, and Yifeng Zhu. 2025. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv:2009.12293 [cs.RO] https://arxiv.org/abs/2009.12293

Paper details

vlnc:W_9q9qnhz2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents