WorksComputer Science
AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
Abstract
Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.
Introduction
Recent advances in Vision-Language Models (VLMs) ([1, 6] (Yang et al. 2025)) have established Vision–Language–Action (VLA) ([9, 13, 21]) models as a prominent paradigm for robotic manipulation. Although imitation learning (IL) from large-scale robot demonstrations enables VLA models to acquire diverse manipulation skills, its performance is inherently constrained by the coverage of the demonstration data. Reinforcement learning (RL) overcomes this limitation by directly optimizing policies with task feedback obtained through environment interaction, and recent work ([15] (Xu et al. 2026)) has successfully applied RL to VLA models. However, existing VLA-RL methods still face substantial challenges in long-horizon manipulation tasks.
First, most methods rely on terminal-only success rewards. In an episode that completes the first subtasks but fails at subtask , the absence of a terminal success reward causes generalized advantage estimation (GAE) to propagate failure-related temporal-difference (TD) residuals backward to , suppressing the advantages of correctly executed actions. We refer to this as credit contamination, where useful behavior in failed trajectories is systematically devalued, with the effect worsening as the horizon grows. Prior work ([4, 28] (Yang et al. 2026a; Zhai et al. 2025)) mitigates reward sparsity through stage-level preference optimization or subtask-level intermediate rewards, but these approaches alter reward density without resolving credit contamination. Second, subtask mastery is uneven, with failures increasingly concentrated in a few bottleneck subtasks during later training. Uniform task sampling and fixed exploration then waste interaction budget on saturated subtasks while providing insufficient exploration for the bottlenecks.
To address these challenges, we propose AtomicRL, as illustrated in Figure 1, an RL post-training framework built around atomic task structure. Our key insight is that semantic subtasks provide a unified basis for both the propagation of learning signals and the allocation of exploration resources. AtomicRL couples Semantic-Aware Policy Optimization (SAPO) with Bottleneck-Adaptive Target Exploration (BATE), supported by a lightweight Progress–Completion Critic (PCC) that estimates atomic subtask progress and completion boundaries. PCC also provides progress estimates for constructing dense progress-based shaping rewards (PBSR). SAPO modulates TD-residual propagation in advantage estimation with a soft gate conditioned on semantic boundaries. Within each atomic subtask, it preserves standard GAE trace decay; across subtask boundaries, it maintains propagation between successful subtasks while attenuating residuals from failed subtasks. By leaving one-step TD residuals and value bootstrapping unchanged, SAPO reduces failure interference while preserving intra-subtask credit assignment and successful subtask dependencies. BATE tracks the mastery of each subtask via a moving average of its completion rate, jointly adjusts the sampling weight of full tasks and the strength of subtask-level entropy regularization, directing interaction and exploration resources toward bottleneck subtasks. Finally, to support physical deployment, we develop an automated Real2Sim2Real pipeline. Given a single real-world multimodal demonstration and the required auxiliary information, the pipeline reconstructs an interactive simulation environment. It connects simulation-based RL with real-robot deployment, shifting costly physical trial-and-error to parallel simulation with reproducible resets.
Figure 1. Overview of AtomicRL. Unlike previous RL methods that use standard GAE which may suffer from credit contamination, AtomicRL uses SAPO to promote more reliable credit assignment. Together with BATE and the Real2Sim2Real pipeline, AtomicRL achieves strong performance on long-horizon manipulation tasks in both simulation and the real world.
We conduct extensive experiments in both simulation and the real world. On LIBERO-LONG [18] AtomicRL achieves success rates of 99.0% with [9] and 98.8% with OpenVLA-OFT [13], improving over PPO by 3.6% and 4.0%. On CALVIN ABC-D [24], the average completed sequence length improves from 3.56 to 4.56, while the success rate of completing full five-subtask chain improves by 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further yields an average improvement of 21.3% on real-robot long-horizon tasks. These results demonstrate the effectiveness of AtomicRL for long-horizon VLA reinforcement learning. Overall, our contributions are as follows:
We introduce AtomicRL, an RL post-training framework built around atomic task structure for long-horizon robotic manipulation.
We propose SAPO to mitigate cross-subtask credit interference and BATE to adaptively direct task sampling and exploration toward bottleneck subtasks.
We develop a Real2Sim2Real pipeline that streamlines simulation construction and RL training for efficient real-robot deployment.
We validate the effectiveness of AtomicRL through extensive experiments conducted in both simulated environments and real-world robots.
Related Work
Reinforcement Learning for VLA Models
Reinforcement learning (RL) optimizes policies through environment interaction and task-level feedback. Recent work has explored both offline and online RL paradigms for VLA policies. Offline RL approaches leverage task feedback to enable continued policy improvement [7 a] (Zhang et al. 2025b). [8] integrates expert demonstrations, autonomous interaction data, and human corrections through advantage-conditioned learning. For scalable online training, VLA-RL (Lu et al., 2025) and SimpleVLA-RL [15] explore frameworks for continually improving VLA policies through environment interaction, while RL4VLA [20] systematically investigates how different RL algorithms affect VLA generalization. Furthermore, Flow Q-Learning [26], Flow-GRPO [19] and [2] investigate online RL fine-tuning for flow-matching policies, addressing challenges in action likelihood computation and exploration.
RL for long-horizon manipulation introduces additional challenges due to sparse and delayed task feedback [3, 17, 5]. STARE-VLA (Xu et al., 2025) exploits the semantic stage structure of trajectories to perform stage-level preference optimization and reward shaping, while IG-RFT [27] combines trajectory-level and subtask-level rewards to provide intermediate guidance. However, dense reward shaping provides useful process guidance over long-horizon trajectories, but cannot eliminate the spurious interference caused by later-stage failures. In contrast, AtomicRL gates the propagation of advantage signals at the semantic boundaries of atomic tasks to prevent cross-task interference from unreliable signals, while allocating the exploration budget to bottleneck atomic tasks to improve the efficiency of long-horizon RL.
Online Reinforcement Learning in Real World
Online reinforcement learning for real-world manipulation is constrained by interaction costs and safety risks. To reduce reliance on real-robot trial and error, some research focuses on learning action-conditioned world models and optimizing policies through imagined interactions [29, 14, 11] (Zhang et al. 2026c). WoVR [12] addresses prediction errors and their accumulation to improve the reliability of model-generated rollouts for VLA reinforcement fine-tuning, while PlayWorld (Yin et al., 2026) and RISE (Yang et al., 2026b) learn world models from autonomous robot interactions and transfers policies trained through model-based reinforcement learning to real hardware. Although world models can effectively capture short-term dynamics, long-horizon predictions still exhibit visual distortions and action–state inconsistencies. Other studies adopt an explicit Real2Sim2Real method, combining real-scene reconstruction with physics simulation to provide interactive environments for policy optimization [16] (Wang et al. 2026). GSWorld [10] integrates Gaussian-on-Mesh assets with physics engines to support visually faithful policy training, while RialTo [30] constructs digital twins of real-world environments and uses reinforcement learning in simulation to improve policy robustness. However, constructing environments requires coordinating scene reconstruction, physical asset preparation, and control alignment, making it complex and costly to reproduce. AtomicRL introduces an automated Real2Sim2Real pipeline that uses only a single demonstration to reconstruct a real-world environment and serves as an interactive simulation environment for long-horizon online RL. The trained policies are then transferred to physical robots for evaluation.
Method
Overview
As illustrated in Figure 2, AtomicRL integrates semantic-aware credit assignment and bottleneck-adaptive exploration for long-horizon robotic manipulation. Specifically, SAPO modulates credit propagation across atomic subtasks (§ 3.2), while bottleneck-adaptive targeted exploration adjusts full-task trajectory sampling and local exploration incentives (Section 3.3). Both components are supported by a Progress–Completion Critic (PCC), which estimates subtask progress and identifies completion boundaries (Section 3.4). For real-world tasks, we further introduce a Real2Sim2Real construction workflow to support simulation training and subsequent transfer evaluation (Section 3.5).
Figure 2. Overall pipeline of AtomicRL. (a) Progress–Completion Critic: PCC identifies atomic task boundaries and estimates atomic task progress from interaction histories. (b) Semantic-Aware Policy Optimization: SAPO uses these signals to regulate cross-subtask advantage propagation based on subtask execution outcomes. (c) Bottleneck-Adaptive Targeted Exploration: BATE adapts task sampling and local exploration according to subtask mastery, focusing training on insufficiently mastered bottleneck subtasks. (d) Real2Sim2Real workflow: A complementary Real2Sim2Real pipeline enables parallel simulation-based training and transfer to real-world robots.
Problem Formulation
We consider a language-conditioned partially observable Markov decision process for long-horizon robotic manipulation. At each decision step , the robot receives an observation , which may include third-person images, wrist-camera images, and proprioceptive states, and outputs an action chunk , where denotes the chunk length. A long-horizon instruction is decomposed into an ordered sequence of atomic subtasks, . We denote the active subtask index by . For nonterminal transitions, the semantic-boundary indicator is defined as , indicating whether execution of leads to a subtask transition. Our objective is to maximize full-task success under a fixed environment-interaction budget by improving credit assignment and directing exploration toward bottleneck subtasks.
Semantic-Aware Policy Optimization
SAPO addresses temporal credit assignment in long-horizon tasks. It leverages semantic boundaries to selectively attenuate the contribution of TD residuals from failed subtasks to the advantage estimates of preceding actions, thereby regulating cross-subtask credit propagation.
Let denote the chunk-level reward and indicate environment termination after executing . The one-step TD residual is
where is the discount factor and denotes the observation- and language-conditioned input representation available to the value function, rather than a privileged simulator state. For an episode containing chunk-level transitions indexed by , the standard GAE estimate used by PPO is
where controls trace decay. This expression is evaluated within a single episode and does not span environment resets.
SAPO distinguishes semantic segmentation from environment termination and uses atomic subtask outcomes to regulate cross-boundary credit propagation. Specifically, PCC (Section 3.4) identifies semantic boundaries between atomic subtasks. Keeping the one-step TD residual unchanged, SAPO applies an outcome-aware soft gate to the recursive GAE trace:
Within a subtask, , and the trace follows the standard decay. At a semantic boundary, when the subsequent subtask succeeds, preserving ordinary propagation; when it fails, we set to attenuate propagation across the boundary. Thus, when subtasks succeed but subtask fails, residuals from the failed segment propagate normally within that segment and are additionally attenuated when entering the completed prefix. Residuals within the completed prefix retain their standard trace weights. This design preserves within-task credit propagation and connections between successfully completed subtasks while reducing the direct negative influence of downstream failures on earlier successful behavior. We use the resulting advantages for updates and construct the value targets as before advantage normalization. By making credit propagation sensitive to semantic boundaries and local outcomes, SAPO improves credit assignment and interaction-data efficiency in long-horizon tasks.
Bottleneck-Adaptive Targeted Exploration
During the early stages of RL, broad exploration provides informative samples for most atomic subtasks. As training approaches saturation, failures may concentrate in a few bottleneck subtasks, making persistent aggressive exploration less effective and potentially disruptive to acquired skills. BATE adapts task sampling and local exploration incentives according to atomic subtask mastery.
We estimate the mastery level of each atomic-task category using an exponential moving average of its completion rate:
where indexes statistics updates and is the fraction of successful outcomes among subtask attempts whose outcomes are resolved during the update.
The competence gap, , guides both full-task sampling and local exploration.
where contains the subtasks of full task , controls sensitivity to the competence gap, and limits sampling redistribution. The baseline weight of 1 preserves coverage of mastered tasks to monitor forgetting.
Within each trajectory, mastery-dependent entropy regularization adjusts local exploration:
where is the base entropy coefficient, denotes policy entropy, and is a bounded exploration weight for the active subtask . The exploration strength is thus increased for poorly mastered subtasks and reduced as subtask mastery improves.
Progress–Completion Critic
To identify atomic-task boundaries , we introduce a lightweight Progress–Completion Critic (PCC), denoted by . PCC estimates within-task progress and completion from the recent observation–action pairs, providing semantic signals for SAPO and BATE and support the construction of progress-based shaping rewards (PBSR).
Conditioned on the current atomic task, PCC evaluates its status after executing and outputs ordinal progress logits and a completion logit , where is the number of progress levels. We decode progress as
where denotes the sigmoid function and is the ordinal decoding threshold. An atomic task is considered complete when and for consecutive decision steps. Upon confirming a non-terminal boundary, we advance the atomic-task index as and reset the tracked within-task progress while retaining the physical observation context. PCC is trained offline on demonstration data using a cumulative ordinal loss together with a class-balanced completion loss:
where applies binary cross-entropy to cumulative progress targets, is a class-balanced binary cross-entropy loss for completion prediction. The coefficients and control the relative contributions of the two objectives. The progress estimates can optionally be used to construct progress-based shaping rewards (PBSR), yielding the training reward
where is the sparse task reward, is the progress-based shaping reward.
Real2Sim2Real Pipeline for Reinforcement Learning
We develop an automated Real2Sim2Real reinforcement learning pipeline that unifies interactive environment construction, policy optimization in simulation, and deployment and evaluation on physical robots. Given a single multimodal real-world demonstration , metadata specifying the robot model and supplementary object observations where needed, the pipeline constructs a simulation environment aligned with the physical system, a shared observation–action interface specification, and a set of reproducible randomized initial layouts:
where is the reconstructed interactive simulation environment, specifies the shared observation–action interface, is a set of reproducible randomized initial layouts, and denotes the pipeline workflow. The generated environment is then integrated into the RLinf (Yu et al., 2025) distributed training framework, where SAPO and targeted exploration optimize the initial policy to obtain . Following pre-deployment validation, the optimized policy is deployed on the physical robot through the same interface specification for task evaluation.
Experiment
Experiments Setup
Simulation Benchmarks. We evaluate AtomicRL on two robotic manipulation benchmarks: LIBERO Liu et al. [18] and CALVIN Mees et al. [24]. We use the LIBERO-LONG suite (10 long-horizon tasks), and for CALVIN, we follow the ABC-D protocol, training on environments A, B, and C and evaluating on the held-out environment D. All experiments are implemented within the RLinf training framework Yu et al. (2025). On LIBERO-LONG, we compare AtomicRL against PPO with two policy backbones, Intelligence et al. [9] and OpenVLA-OFT Kim et al. [13]; on CALVIN, we conduct additional evaluations with . Each training run consists of 200 optimization steps.
Real World Robot. We conduct real-world experiments using an AgiBot G1 robot across three long-horizon tabletop manipulation tasks, each consisting of 4 to 6 atomic subtasks. For each task, we collect 100 expert demonstrations for SFT. The three tasks are:
Bread Placement(BP): Place all pieces of bread on the table onto the white plate;
Waste Disposal(WD): Place the tissue and the car into the trash bin;
Tabletop Clean(TC): Place the remote control into the storage box and the snacks into the box.
Evaluation metrics. For LIBERO, we report the mean success rate across all tasks, with 50 trials per task. For CALVIN, we report the success rates for completing one to five consecutive tasks and the average completed sequence length over 1,000 evaluation trials. These metrics measure cumulative sequence completion rather than independent subtask performance. For real-robot evaluation, each task is tested over 25 trials per method, and we report the mean success rate across the three tasks.
Results on Simulation
Results on LIBERO. As shown in Table 1, AtomicRL achieves success rates of 98.8% with OpenVLA-OFT and 99.0% with , exceeding their corresponding PPO baselines by 4.0% and 3.6%. The consistent gains across both backbones support the applicability of AtomicRL to different VLA architectures. Both policies reach near-perfect success rates after 200 optimization iterations, demonstrating further improvements even when PPO already performs well. This efficiency can be attributed to AtomicRL’s effective use of training data: SAPO preserves useful learning signals from completed subtasks in failed trajectories, while BATE concentrates exploration on bottleneck atomic subtasks. Together, these mechanisms enable faster convergence and higher final performance.
| Backbone | Method | SR (%) ↑ | (pp) |
| SFT | 89.0 | – | |
| + PPO | 95.4 | – | |
| + AtomicRL (Ours) | 99.0 | +3.6 | |
| OpenVLA-OFT | SFT | 86.5 | – |
| + PPO | 94.8 | – | |
| + AtomicRL (Ours) | 98.8 | +4.0 |
Table 1. Evaluation on LIBERO-LONG (%). denotes the improvement over PPO.
Results on CALVIN. As shown in Table 2, AtomicRL improves the average completed sequence length from 3.56 with SFT to 4.56, compared with 4.36 for PPO. The success rate for completing all five tasks increases from 70.6% with PPO to 80.1%, a gain of 9.5%. The larger gains on longer sequences indicate more reliable execution across the full task chain, demonstrating the effectiveness of AtomicRL for long-horizon reinforcement learning.
| Method | Task | 1 | 2 | 3 | 4 | 5 | Avg. Len ↑ |
| SFT | ABC→D | 92.8 | 83.5 | 73.2 | 61.5 | 45.3 | 3.56 |
| + PPO | ABC→D | 99.2 | 95.7 | 89.8 | 81.2 | 70.6 | 4.36 |
| + AtomicRL (Ours) | ABC→D | 99.4 | 97.0 | 92.6 | 87.3 | 80.1 | 4.56 |
Table 2. Long-horizon Robotic Manipulation Evaluation on CALVIN Benchmark (%).
Results on Real-world Robot
We conduct online reinforcement learning in the constructed real-to-sim environments and evaluate both the policy improvement after RL and its sim-to-real transfer performance.
Performance on real-world robot. As shown in Table 3, AtomicRL improves over SFT by 21.3% on average across three long-horizon tasks and over PPO by 8.0%. As shown in Figure 4, SAPO preserves useful learning signals from successfully completed subtasks in unsuccessful trajectories, enabling more appropriate credit assignment for suboptimal trajectories. Consequently, AtomicRL
| Method | BP | WD | TC | Avg. SR ↑ |
| SFT | 60 | 52 | 32 | 48.0 |
| + PPO | 68 | 64 | 52 | 61.3 |
| + AtomicRL | 76 | 72 | 60 | 69.3 |
Table 3. Real-world robot results(%).

Figure 4. Training curves on LIBERO-LONG and CALVIN ABC-D and Real-World(BP).
improves faster in early training, especially on the longer-horizon Bread Placement task. These results demonstrate the effectiveness of AtomicRL for long-horizon manipulation.
Simulation–reality correspondence. Figure 3 compares the success rates between simulation and the real robot across nine combinations of three tasks and three methods. The aggregated Pearson correlation coefficient is , indicating a strong positive correlation across the evaluated settings. As shown in Figure 5, the SFT policy fails after completing only part of the task, whereas the AtomicRL-optimized policy successfully completes the illustrated sequence in both simulation and the real world. The qualitative consistency in observations and execution outcomes between the two environments indicates that the reconstructed environment preserves the key visual and interaction structures relevant to the task, supporting effective policy optimization. Combined with the Real2Sim2Real deployment pipeline, AtomicRL shifts the otherwise costly and operationally complex online RL process from physical robots to simulation environments that support fast and parallel interactions. This reduces real-world interaction and reset costs while providing a practical approach for optimizing and deploying long-horizon manipulation policies.

Figure 3. Sim2Real Correlation.

Figure 5. Demonstrations show the execution process of SFT and AtomicRL.
Ablation Study
We conduct policy-learning ablations on LIBERO-LONG using a weaker SFT checkpoint trained with fewer demonstrations than those used in the main evaluations. This few-shot initialization allows us to examine the contribution of each component under a less proficient initial policy.
Reward shaping and SAPO. To investigate the effect of reward granularity, we use the PCC module to construct process-level dense rewards and subtask-level rewards, and compare them under both PPO and SAPO. As shown in Table 4 (a), denser rewards do not always yield better performance: under SAPO, relatively sparse subtask-level rewards outperform process-level dense rewards. When the policy success rate is low, dense rewards provide stronger exploration guidance and accelerate early learning. As the policy improves, however, their benefits diminish, while persistent auxiliary rewards may dilute the terminal success signal and shift optimization toward local gains. Motivated by this observation, we use linear annealing to gradually reduce the PBSR weight, providing stronger dense-reward guidance early in training and emphasizing the sparse terminal reward later. This balances early exploration with late-stage task alignment.
Table 4: Ablations on LIBERO-LONG. (a) Reward shaping with and without SAPO. (b) Component contributions. All results are success rates (%).
| Reward | PPO | SAPO |
| 0/1 | 78.2 | 87.8 |
| SubR | 80.2 | 89.4 |
| PBSR | 81.8 | 89.0 |
| Ours | – | 89.6 |
| SAPO | PBSR | BATE | SR (%) |
| × | × | × | 78.2 |
| ✓ | × | × | 87.8 |
| ✓ | ✓ | × | 89.6 |
| ✓ | ✓ | ✓ | 91.2 |
Table 4. Ablations on LIBERO-LONG. (a) Reward shaping with and without SAPO. (b) Component contributions. All results are success rates (%).
Component analysis. Table 4 (b) presents the incremental effects of successively introducing SAPO, annealed PBSR, and BATE. Starting from a PPO success rate of 78.2%, adding SAPO increases the success rate to 87.8% (+9.6 pp); introducing annealed PBSR further improves it to 89.6% (+1.8 pp); and finally incorporating BATE raises it to 91.2% (+1.6 pp), yielding a cumulative improvement of 13.0 percentage points over PPO. SAPO provides the largest performance gain, while annealed reward shaping and bottleneck-adaptive exploration further improve the policy. Combined with the results in Table 4 (a), where SAPO consistently improves performance across different reward settings, these findings suggest that its gains do not simply arise from denser reward feedback. Together, these results support the complementarity of the proposed components and their ability to further unlock the potential of RL for learning long-horizon tasks.
PCC prediction quality and computational cost. We train PCC on 150 trajectories containing both successful and failed executions and evaluate it on a test set of 500 rollout trajectories. The reference annotations for training and evaluation are automatically generated by a VLM and subsequently refined through manual correction. As shown in Table 5, PCC achieves 95.92% precision and 89.20% recall, with an empirical inference latency of 42.4 ms.
| Category | Metric | Value |
| Prediction quality | Accuracy | 95.92% |
| Prediction quality | False detection rate | 4.08% |
| Prediction quality | Recall | 89.20% |
| Efficiency | Trainable Parameters | 28.8 M |
| Efficiency | Inference latency | 42.4 ms |
Table 5. PCC performance and efficiency.
These results show that PCC can estimate subtask progress and completion status with high precision while maintaining low inference latency. As a core component of AtomicRL, PCC converts the semantic task structure in trajectories into progress and boundary information for policy optimization, supporting SAPO in regulating cross-subtask credit propagation and BATE in estimating subtask mastery. Its prediction accuracy and low inference overhead demonstrate that this lightweight design provides a practical perception and evaluation foundation for RL based on atomic task structure.
Conclusion
In this work, we introduce AtomicRL, a VLA RL post-training framework for long-horizon robotic manipulation centered on atomic task structure. To address temporal credit interference and imbalanced atomic subtask mastery, AtomicRL uses PCC to estimate task progress and completion boundaries, SAPO to regulate the cross-boundary propagation of TD residuals based on subtask execution outcomes, and BATE to shift task sampling and local exploration toward insufficiently mastered bottleneck subtasks. We further develop a Real2Sim2Real pipeline that connects real-task reconstruction, simulation-based RL, and real-robot deployment, enabling policy improvement in simulation environments with fast parallel interactions and reproducible resets. We validate AtomicRL in both simulation and real-world settings, and the results show that it converges faster to a higher performance level, demonstrating its potential for RL-based long-horizon robotic manipulation.
Appendix/supplemental material
Future Work and Limitations
AtomicRL improves temporal credit assignment across atomic tasks through SAPO and concentrates exploration resources on insufficiently mastered bottleneck subtasks through BATE. Although the experimental results demonstrate its effectiveness in long-horizon robotic manipulation, the current approach still has several limitations.
AtomicRL improves the utilization of interaction data through more effective credit assignment and exploration-resource allocation, but does not fundamentally address the overall efficiency of online RL training. Even with open-source RL frameworks such as RLinf (Yu et al., 2025) and SERL [23], online RL still requires substantial environment interaction and training time, particularly in real-robot settings where interaction and exploration costs remain high. Reducing the interaction required for effective policy learning, improving online RL training efficiency, and lowering the cost of real-robot training therefore remain important directions for future work.
Our experiments primarily focus on long-horizon tabletop manipulation tasks and initialize RL from SFT policies. The effectiveness of AtomicRL on more open and dynamic tasks requiring stronger embodiment generalization remains to be further evaluated. Recent Agentic RL approaches, such as HarnessVLA (Zhang et al., 2026b) and AtomBridge [25] have shown potential for open-world tasks and embodiment generalization, providing an alternative direction for long-horizon robotic manipulation. A promising direction is to transfer AtomicRL’s use and optimization of atomic task structure to Agentic RL frameworks, enabling robots to dynamically form and adapt atomic task structures in more open environments and potentially improving the generalization and training efficiency of long-horizon manipulation.
Progress-Based Shaping Rewards
We optionally supplement the sparse task reward with progress-based shaping rewards (PBSR), using the atomic-task index and discrete progress estimates from PCC.
Progress potential. Let denote the number of atomic tasks and denote the decoded progress level. After completion handling, let be the active atomic-task index and the tracked progress level. We set unless atomic-task completion is confirmed, in which case it is reset to zero. The task index advances only if another atomic task remains. The updated progress potential is
The potential combines the atomic-task index with within-task progress, with the latter scaled by .
Shaping reward. For each valid non-terminal PCC update, we compute
where is the discount factor and is the previously stored potential. The stored potential is initialized to zero when the task tracker is reset and replaced by after each valid update. The shaping contribution is zero on terminal transitions, including environment timeouts, and whenever no valid update is available. The training reward is
where is the sparse task reward and controls the shaping contribution at training iteration . The shaping reward can be positive or negative and is not clipped.
Shaping schedule. We linearly anneal the shaping coefficient:
caption: (a) LIBERO-LONG success rate. (b) CALVIN average length. (c) CALVIN task-5 success rate.
fig:6: Training curves on LIBERO-LONG and CALVIN ABC-D.
where is the initial coefficient, specify the annealing interval, is the final retained fraction, and clamps to . Setting removes PBSR after annealing.
PBSR modifies the reward used to compute TD residuals, whereas SAPO regulates their cross-boundary propagation during advantage estimation.
Additional Ablations
Sensitivity to the SAPO Gating Coefficient. We examine the effect of the SAPO gating coefficient on LIBERO-LONG under the same configuration as the component ablations. Here, controls trace propagation across boundaries preceding failed subtasks, while gates preceding successful subtasks remain fixed at one. Setting recovers standard GAE propagation, whereas blocks cross-boundary trace propagation from failed subtasks without removing the one-step value bootstrap. As shown in Table 6, the highest observed success rate is 87.8% at , improving over the unattenuated baseline (78.2%) by 9.6 percentage points. Performance decreases to 85.0% and 81.2% as increases to 0.5 and 0.8, respectively. Hard trace blocking () also performs well, achieving 87.2%, only 0.6 percentage points below . These results support the use of soft attenuation of cross-boundary trace propagation from failed subtasks in this setting.
| Success Rate | |
| 0.0 | 87.2% |
| 0.3 | 87.8% |
| 0.5 | 85.0% |
| 0.8 | 81.2% |
| 1.0 | 78.2% |
Table 6. Effect of the SAPO gating coefficient.
Training Details
Training Setup. For all LIBERO and real-world experiments, we train PCC offline on annotated demonstrations. PCC uses a frozen SigLIP visual encoder with approximately 400M parameters, initialized from the visual backbone, together with lightweight trainable temporal and prediction modules. For data annotation, we adapt the annotation pipeline of AtomicVLA (Zhang et al. (2026a)) to generate fine-grained annotations for demonstration trajectories. We annotate 150 trajectories for LIBERO and 50 trajectories per real-world task, which are subsequently used for offline PCC training. The trained PCC is kept fixed during RL post-training.
All RL post-training experiments are implemented in the RLinf framework. All training is performed on eight NVIDIA H200 GPUs, while inference is conducted on a single NVIDIA GeForce RTX 5090 GPU.
Additional Results
Detail Results on benchmarks. Fig. 6 and Figure 7 show the training dynamics and value-function fitting of AtomicRL. On both LIBERO-LONG and CALVIN ABC-D, AtomicRL improves more rapidly during early training and consistently outperforms PPO throughout subsequent training. Meanwhile, the critic explained variance in the later stages of training is higher than PPO by approximately 12% on LIBERO and 5% on CALVIN, indicating that the value function explains a larger fraction of the variance in the training targets. These trends are consistent with the design of SAPO and BATE:

(a) LIBERO-LONG critic explained variance.

(b) CALVIN critic explained variance.
Figure 7. Critic explained variance.
SAPO attenuates the cross-boundary propagation of TD residuals from failed atomic subtasks to completed prefixes, improving temporal credit assignment and reducing interference from downstream failures with previously successful behaviors; BATE concentrates sampling and exploration resources on insufficiently mastered bottleneck subtasks. The concurrent improvement in task performance and value-function fitting further supports the effectiveness of atomic-task-structured optimization for long-horizon policy learning.
Detail Results on real-world robot. As shown in Figure 8, when online RL is performed in the Real2Sim-reconstructed environments, AtomicRL exhibits better training performance than PPO across all three real-world tasks. The advantage is particularly pronounced during early training on Bread Placement (BP), which consists of six atomic subtasks: PPO remains around 50% success over the first 40 training steps, whereas AtomicRL reaches 60% at around step 30 and continues to improve thereafter. This gap reflects the difficulty of learning long task chains under sparse feedback and is consistent with the design of AtomicRL, which attenuates cross-boundary credit interference from downstream failures and thereby makes better use of successful prefixes in failed trajectories. AtomicRL also achieves higher late-stage training success rates on Waste Disposal (WD) and Tabletop Clean (TC), indicating that its training advantage is not limited to a single task. Combined with the simulation–real-world performance correlation analysis in Figure 3, these training curves provide an indirect reference for the trend of real-world policy improvement.

(a) BP success rate.

(b) WD success rate.

(c) TC success rate.
Figure 8. Training curves on Real-world robot.
Details of automated Real2Sim2Real pipeline
As shown in Figure 9, we develop an automated Real2Sim2Real workflow, with GPT-6 Astra assisting tool orchestration and code implementation. The workflow connects real-task reconstruction, simulation-based online RL in RLinF (Yu et al., 2025), and real-robot deployment, with the key steps and validation requirements summarized in the figure.
Figure 9. Automated Real2Sim2Real workflow with essential tool calls and a real-to-simulation scene example.
Additional Visualizations
We provide additional visualizations of policy execution in simulation and the real world. As shown in Figure 10, SFT completes the initial actions but fails at a later stage, whereas AtomicRL successfully completes the illustrated sequence in both simulation and real-world deployment.

Figure 10. Demonstrations show the execution process of SFT and AtomicRL.
Automated Real2Sim2Real Pipeline
Role: You are a robotics simulation and reinforcement learning engineer. Convert the supplied real-task data into an interactive simulation with aligned camera views and action-space semantics for online RL.
Input: One LeRobot demonstration containing images, states, actions, and task text; object multiview observations and material information; an SFT checkpoint with its configuration and normalization.
Workflow:
Initialize the environment. Build the robot in MuJoCo from its URDF, preserving joint structure, order, and limits. Use COLMAP/PyCOLMAP and Open3D/Trimesh to reconstruct task objects, with explicit shape priors where needed.
Restore scene appearance. Map foreground photos to surface UV atlases; adjust color, specularity, and shininess. Bind deformable textured surfaces to physical nodes. For complex static backgrounds, 3DGS is an optional rendering extension; collision geometry remains separate.
Assign physical properties. Rigid objects use CoACD/primitive proxies with mass, inertia, and friction. Soft objects use MuJoCo flex tetrahedra with mass/density, Young’s modulus, Poisson’s ratio, damping, and friction. Recoverable soft materials additionally use viscoelastic forces, relaxation times, and internal deformation states. Mark measured versus assumed parameters.
Recover and diversify layouts. Combine RGB-D/scale evidence, robot kinematics, and SciPy multiview contour/landmark fitting to align poses, support height, and cameras. Check overlays; sample seeded layouts within feasible regions and retain physically stable initial states.
Validate the environment. Replay recorded actions and run the SFT checkpoint to check action semantics, contact, compression/recovery, and numerical stability. Restore material memory when resetting viscoelastic objects.
Expose the RL interface. Provide observation, action, reward, termination, and reset APIs for parallel RLinf workers. Select PPO, GRPO, or SAPO according to the implemented algorithm adapter; preserve the deployment control interface.
Output: scene preview and required checks
References
- [1]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3b vlm for transfer, 2024. URL https://arxiv.org/abs/2407.07726.
- [2]Kang Chen, Zihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, and Chao Yu. πRL: Online rl fine-tuning for flow-based vision-language-action models, 2026a. URL https://arxiv.org/abs/2510.25889.
- [3]Qianzhong Chen, Justin Yu, Mac Schwager, Pieter Abbeel, Yide Shentu, and Philipp Wu. Sarm: Stage-aware reward modeling for long horizon robot manipulation, 2026b. URL https://arxiv.org/abs/2509.25358.
- [4]Zixuan Chen, Jing Huo, Yangtao Chen, and Yang Gao. Robohorizon: An llm-assisted multi-view world model for long-horizon robotic manipulation, 2025. URL https://arxiv.org/abs/2501.06605.
- [5]Murtaza Dalal, Tarun Chiruvollu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq-learn: Language model guided rl for solving long horizon robotics tasks, 2024. URL https://arxiv.org/abs/2405.01534.
- [6]Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarna, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittliff, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannahnejahishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models, 2024. URL https://arxiv.org/abs/2409.17146.
- [7]Dongchi Huang, Zhirui Fang, Tianle Zhang, Yihang Li, Lin Zhao, and Chunhe Xia. Co-rft: Efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning, 2025. URL https://arxiv.org/abs/2508.02219.
- [8]Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szymon Jakubczak, Rowan Jen, Tim Jones, Ben Katz, Liyiming Ke, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Yao Lu, Vishnu Mano, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Charvi Sharma, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoeckle, Alex Swerdlow, James Tanner, Marcel Torrone, Quan Vuong, Anna Walling, Haohuan Wang, Blake Williams, Sukwon Yoo, Lili Yu, Ury Zhilinsky, and Zhiyuan Zhou. π₀.₆: a vla that learns from experience, 2025a. URL https://arxiv.org/abs/2511.14759.
- [9]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π₀.₅: a vision-language-action model with open-world generalization, 2025b. URL https://arxiv.org/abs/2504.16054.
- [10]Guangqi Jiang, Haoran Chang, Ri-Zhao Qiu, Yutong Liang, Mazeyu Ji, Jiyue Zhu, Zhao Dong, Xueyan Zou, and Xiaolong Wang. Gsworld: Closed-loop photo-realistic simulation suite for robotic manipulation, 2025. URL https://arxiv.org/abs/2510.20813.
- [11]Zhennan Jiang, Kai Liu, Yuxin Qin, Shuai Tian, Yupeng Zheng, Mingcai Zhou, Chao Yu, Haoran Li, and Dongbin Zhao. World4rl: Diffusion world models for policy refinement with reinforcement learning for robotic manipulation, 2026a. URL https://arxiv.org/abs/2509.19080.
- [12]Zhennan Jiang, Shangqing Zhou, Yutong Jiang, Zefang Huang, Mingjie Wei, Yuhui Chen, Tianxing Zhou, Zhen Guo, Hao Lin, Quanlu Zhang, Yu Wang, Haoran Li, Chao Yu, and Dongbin Zhao. Wovr: World models as reliable simulators for post-training vla policies with rl, 2026b. URL https://arxiv.org/abs/2602.13977.
- [13]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success, 2025. URL https://arxiv.org/abs/2502.19645.
- [14]Chenhao Li, Andreas Krause, and Marco Hutter. Robotic world model: A neural network simulator for robust policy optimization in robotics, 2025a. URL https://arxiv.org/abs/2501.10100.
- [15]Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Yang Zhaohui, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al. Simplevla-rl: Scaling vla training via reinforcement learning. In International Conference on Learning Representations, volume 2026, pp. 125846–125869, 2026.
- [16]Wenxuan Li, Hang Zhao, Zhiyuan Yu, Yu Du, Qin Qin, Ruizhen Hu, and Kai Xu. Pin-wm: Learning physics-informed world models for non-prehensile manipulation, 2025b. URL https://arxiv.org/abs/2504.16693.
- [17]Yunfei Li, Xiao Ma, Jiafeng Xu, Yu Cui, Zhongren Cui, Zhigang Han, Liqun Huang, Tao Kong, Yuxiao Liu, Hao Niu, Wanli Peng, Jingchao Qiao, Zeyu Ren, Haixin Shi, Zhi Su, Jiawen Tian, Yuyang Xiao, Shenyu Zhang, Liwei Zheng, Hang Li, and Yonghui Wu. Gr-rl: Going dexterous and precise for long-horizon robotic manipulation, 2025c. URL https://arxiv.org/abs/2512.01801.
- [18]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https://arxiv.org/abs/2306.03310.
- [19]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025a. URL https://arxiv.org/abs/2505.05470.
- [20]Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, and Yu Wang. What can rl bring to vla generalization? an empirical study, 2026. URL https://arxiv.org/abs/2505.19789.
- [21]Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation, 2025b. URL https://arxiv.org/abs/2410.07864.
- [23]Jianlan Luo, Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine. Serl: A software suite for sample-efficient robotic reinforcement learning, 2025. URL https://arxiv.org/abs/2401.16013.
- [24]Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks, 2022. URL https://arxiv.org/abs/2112.03227.
- [25]Yiwen Pang, Bo Zhou, Changjin Li, Xuanhao Wang, Shengxiang Xu, Deng-Bao Wang, Peng Cheng, Shimin Di, Jingkuan Song, and Min-Ling Zhang. Atombrigde: Agentic vla inference plugin for long-horizon tasks in scientific experiments, 2026. URL https://arxiv.org/abs/2602.09430.
- [26]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, 2025. URL https://arxiv.org/abs/2502.02538.
- [27]Zhian Su, Weijie Kong, Haonan Dong, and Huixu Dong. Ig-rft: An interaction-guided rl framework for vla models in long-horizon robotic manipulation, 2026. URL https://arxiv.org/abs/2602.20715.
- [28]Huajie Tan, Sixiang Chen, Yijie Xu, Zixiao Wang, Yuheng Ji, Cheng Chi, Yaoxu Lyu, Zhongxia Zhao, Xiansheng Chen, Peterson Co, Shaoxuan Xie, Guocai Yao, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Robo-dopamine: General process reward modeling for high-precision robotic manipulation, 2025. URL https://arxiv.org/abs/2512.23703.
- [29]GigaWorld Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jiagang Zhu, Kerui Li, Mengyuan Xu, Qiuping Deng, Siting Wang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yankai Wang, Yu Cao, Yifan Chang, Yuan Xu, Yun Ye, Yang Wang, Yukun Zhou, Zhengyuan Zhang, Zhehao Dong, and Zheng Zhu. Gigaworld-0: World models as data engine to empower embodied ai, 2025. URL https://arxiv.org/abs/2511.19861.
- [30]Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation, 2024. URL https://arxiv.org/abs/2403.03949.


