Meituan-Robotics-0: A Vision-Language-Action Foundation Model for Desktop Manipulation

Homepage | GitHub | Model | Data

Introduction

Vision–language–action [1, 2, 3, 4, 5, 6, 7] models provide a general interface for mapping visual observations and language instructions to robot actions. Their increasing data and model scales have substantially broadened the range of manipulation behaviors that can be learned within a single policy. Deploying such policies on desktop manipulation, however, exposes several limitations that cannot be addressed by scaling the action model alone. Robot datasets [8, 9, 10, 11, 12, 13, 14] collected from different sources use inconsistent observation formats, action spaces, embodiments, and task descriptions, which prevents them from providing coherent supervision without careful alignment. Even after data alignment, policies trained primarily on successful demonstrations receive little guidance for recovering from the error states induced by their own closed-loop behavior. Moreover, a reactive policy conditioned only on the current observation cannot preserve task information once it leaves the observation window. A practical desktop-manipulation system must therefore connect aligned heterogeneous data, robust real-world policy learning, and persistent task-level memory.

We present Meituan-Robotics-0 (MR0), a desktop-manipulation framework built around a large-scale data foundation, a generalist VLA policy, and an event-driven embodied agent, as summarized in Figure 1. The data foundation combines 147 million vision–language samples with 15,060 hours of curated robot trajectories spanning 12 embodiments. The vision–language corpus provides general visual-semantic knowledge together with embodied supervision for grounding, scene understanding, affordance prediction, and task reasoning. The robot corpus comprises three complementary components: RoboDesk, a curated collection of open-source real-robot trajectories totaling 3,235 hours; 7,530 hours of licensed real-world data spanning seven robot configurations; and 4,295 hours of open simulation data. The complete corpus is processed through temporal and visual filtering, kinematic-consistency checks, cross-embodiment action canonicalization, and hierarchical task alignment. RoboDesk and the simulation data broaden task, environment, and embodiment coverage, while the licensed real-robot data provide task-focused manipulation experience.

Vector (PDF)

Building on this aligned data foundation, the MR0 policy follows a three-stage training recipe that separates the acquisition of broad action priors from continuous-control learning and real-world adaptation. A shared multimodal backbone is first paired with a discrete action head and pre-trained jointly on vision-language data and tokenized robot actions. Continuous mid-training then introduces an MMDiT [15] action head and optimizes flow matching over a unified end-effector representation, transferring the pre-trained features to fine-grained control across single-arm and bimanual embodiments. Real-world post-training [16, 17, 18] starts with supervised finetuning on successful demonstrations. HG-DAgger [19] adds human corrections from policy-induced error states, while RECAP (RL with Experience and Corrections via Advantage-conditioned Policies) [20] derives advantage labels from outcome-labeled autonomous rollouts. We further introduce Rollback, a collection mechanism that returns to a historical anchor to collect a human replacement while retaining the original policy segment. These paired segments supply direct positive and negative labels to CFG-DAgger and remove the need for value learning and advantage labeling. At deployment, asynchronous action-chunk merging integrates new predictions with commands already being executed, maintaining smooth control without blocking on policy inference.

The VLA policy does not explicitly maintain task state across subtasks. MR0 therefore adds an event-driven embodied agent [21, 22, 23] to retain relevant context, track task progress, and revise plans as conditions change. A Monitor summarizes short visual windows into task-relevant events, and a Memory Manager consolidates these events into a persistent record of task progress and environmental conditions. The Planner updates the active subtask only when a new event changes task progress, reveals a failure, or alters whether the current objective is feasible. Motor execution consequently remains with the VLA policy, while the Planner is invoked only when task-level reassessment is needed.

This separation enables the system to retrieve information acquired in earlier tasks, suspend an objective whose prerequisites are temporarily unavailable, and resume it after the required conditions are restored.

We evaluate the policy and agent under protocols matched to their respective roles. MR0 achieves 93.6% average success on RoboTwin 2.0 [24]. On RoboDojo [25], the history-augmented policy ranks fifth in the VLA track by average success rate. With approximately 13.6k hours of embodied training data, MR0 achieves competitive performance across the evaluated manipulation capabilities. Human-intervention data improve both MR0 and $\pi_{0.5}$ across three real-robot tasks. Under matched HG-DAgger post-training, MR0 achieves higher average success rate and progress score than $\pi_{0.5}$ across the three tasks. In a separate comparison on Ethernet insertion, MR0 trained with CFG-DAgger reaches 70% success and a progress score of 0.90, outperforming HG-DAgger and RECAP on the same Rollback data. Qualitative agent deployments further illustrate cross-task event retrieval and task resumption following temporary interruptions. Together, these evaluations show how aligned data, continuous policy learning, real-world post-training, and event-level memory address complementary requirements of general-purpose desktop manipulation.

Data Curation

To equip MR0 with both broad visual-semantic understanding and diverse cross-embodiment manipulation experience, we independently curate two complementary data resources: 147 million vision-language samples and 15,060 hours of robot trajectories spanning 12 embodiments. The vision-language corpus provides supervision for general perception and embodied reasoning. Constructed through a separate pipeline, the robot corpus combines RoboDesk, licensed real-robot data, and open simulation data for action learning across heterogeneous hardware. We describe the two resources in § 2.1 and § 2.2, respectively.

Vision-Language Data

Visual reasoning for manipulation requires both broad perceptual knowledge and embodied understanding grounded in robot interaction. As shown in Figure 2, we organize the 147M vision-language corpus into two complementary subsets. General-purpose data account for 78% and provide supervision for visual question answering, spatial understanding, visual grounding, and counting. The remaining 22% consist of embodied data for scene captioning, task planning, visual grounding, and affordance grounding. Affordance grounding jointly localizes the manipulated object with a bounding box and identifies a 2D interaction point at which the robot should act. These corpus statistics are

Vector (PDF)

computed before training-time resampling. Table 3 reports the effective pre-training mixture after resampling, with entries organized by training group rather than capability.

Training pool	Data category	Scale	Within-pool ratio
VLA action pool	RoboDesk (open real)	310.7M frames / 3.0k h	21.5%
VLA action pool	Licensed real	746.3M frames / 6.9k h	51.7%
VLA action pool	Synthetic	387.5M frames / 3.6k h	26.8%
VLA action pool	Total	1,444.5M frames / ∼13.6k h	100.0%
VLM co-training pool	Real-robot grounding	66.7M samples	44.2%
VLM co-training pool	General VQA / captioning	50.6M samples	33.6%
VLM co-training pool	Spatial reasoning	11.9M samples	7.9%
VLM co-training pool	Real-robot affordance	8.2M samples	5.5%
VLM co-training pool	Simulation VQA	8.2M samples	5.4%
VLM co-training pool	Task planning / embodied QA	1.8M samples	1.2%
VLM co-training pool	Real-robot captioning	1.7M samples	1.1%
VLM co-training pool	Xingzhi VQA	1.6M samples	1.1%
VLM co-training pool	Total	150.7M effective samples from 64 sub-datasets	100.0%

For the embodied VQA component, we augment the collected open-source datasets with a large-scale in-house corpus derived from real-robot trajectories. To determine which supervision types should be prioritized in this corpus, we conduct controlled CALVIN ABC-to-D ablations over eight candidate tasks and use the results to guide subsequent large-scale data construction, as shown in Table 1. The candidates span three capability families: spatial localization, action-related understanding, and semantic and task reasoning. All variants use the same StarVLA [26] configuration, are trained on environments A, B, and C, and are evaluated on environment D. Table 1 reports success rates for completing sequences of 1 to 5 consecutive subtasks, along with the average completed sequence length. The results support including 2D target-affordance grounding and scene captioning as the action-related and semantic supervision types, respectively. For spatial localization, we choose 2D grounding because it avoids costly geometric annotations and scales more readily than 3D grounding. Taken together, the empirical results and practical annotation considerations lead us to focus our in-house data expansion on three types: 2D target-affordance grounding, scene captioning, and 2D grounding.

Capability family	Training setting	Avg. Len.	Task 1	Task 2	Task 3	Task 4	Task 5
Baseline	–	3.832	91.0%	83.0%	76.2%	70.2%	62.8%
Spatial localization	+ 2D Grounding	3.941	91.3%	83.7%	78.1%	73.2%	67.8%
Spatial localization	+ 2D Pointing	3.806	90.8%	81.8%	74.8%	69.9%	63.3%
Spatial localization	+ 3D Grounding	4.037	92.4%	85.3%	79.6%	75.7%	70.7%
Action-related understanding	+ 2D Target-Affordance Grounding	4.102	93.3%	87.9%	82.5%	76.4%	70.1%
Action-related understanding	+ 3D Target-Affordance Grounding	4.098	93.7%	89.9%	81.2%	73.9%	71.1%
Action-related understanding	+ 2D Trajectory	3.987	92.0%	85.0%	79.2%	74.0%	68.5%
Semantic and task reasoning	+ Subtask Prediction	3.851	90.0%	82.2%	76.2%	71.7%	65.0%
Semantic and task reasoning	+ Scene Captioning	4.082	93.1%	87.1%	81.4%	76.2%	70.4%

To ensure consistent and scalable data construction, we develop task-specific annotation pipelines for the selected supervision types. For 2D grounding, we apply Set-of-Mark prompting [27]: the Segment Anything Model (SAM) [28] proposes and marks candidate regions, and a vision-language model assigns an object identity to each region to form grounding question–answer pairs. For 2D affordance grounding, we obtain the target bounding box and interaction point from the key interaction frame of each action phase and propagate both annotations to earlier frames within that phase. For scene captioning, a vision-language model conditions on the robot observation and task context to describe task-relevant objects, spatial relations, gripper state, and execution progress.

Robotic Data

To provide diverse action supervision for robot control, we assemble 15,060 hours of tabletop trajectories spanning 12 embodiments. We restrict the corpus to stationary-base manipulation and exclude segments containing base motion. As summarized in Figure 3, the corpus consists of RoboDesk (3,235 hours; 21.5%), licensed real-robot data (7,530 hours; 50.0%), and open simulation data (4,295 hours; 28.5%). RoboDesk denotes the six open-source real-world datasets retained after curation, which are listed together with the simulation sources in Table 2.

Data source	Hours	Share
RoboDesk (real-world): 3,235 h (21.5%)
AgiBotWorld-Beta [8]	2,174	14.4%
Galaxea Open-World [9]	333	2.2%
RoboCOIN [12]	311	2.1%
LET [29]	172	1.1%
DROID [10]	164	1.1%
RoboMIND [30]	81	0.5%
Simulation: 4,295 h (28.5%)
InternData-A1 [11]	3,967	26.3%
Genie Sim 3.0 [31]	213	1.4%
RoboTwin 2.0 [24]	115	0.8%
Open-source total: 7,530 h (50%)

Data Curation Pipeline.

Raw trajectories contain idle boundaries, inconsistent kinematic records, visual defects, embodiment-specific action conventions, and uneven language annotations. We address these issues with three filtering stages (S1–S3) followed by two alignment stages (A1–A2). S1 removes static prefixes and suffixes, S2 rejects inconsistent joint–EEF recordings, and S3 removes corrupted visual segments. A1 canonicalizes actions across embodiments, while A2 aligns multi-granularity task annotations with the retained trajectories. Figure 4 summarizes the pipeline.

Vector (PDF)

Static-boundary filtering. Teleoperated recordings often begin before manipulation and continue through task completion or reset, producing long intervals of near-zero motion at episode boundaries. Let $q_t \in\mathbb{R}^{J}$ be the recorded joint vector and $\Delta q_t^{(j)} = q_t^{(j)} - q_{t-1}^{(j)}$. A time index is considered static only when every joint remains below a motion threshold for $L$ consecutive steps:

The persistence window prevents brief low-speed motions from being classified as idle. We trim only static runs connected to an episode

boundary, preserving deliberate pauses during task execution. The corresponding video, state, and action intervals are removed jointly to maintain temporal synchronization.

Kinematic-consistency filtering. For each embodiment, we reconstruct the end-effector position from the recorded joints using the corresponding URDF and compare it with the logged EEF position in a common frame. Samples exceeding an embodiment-specific error threshold are masked, and trajectories with persistent violations are discarded.

Video-defect filtering. We detect decoding failures, blur, frozen segments, and black frames. Corrupted intervals are removed from all synchronized streams, while episodes with persistent defects are discarded. Detection thresholds are calibrated separately by embodiment and data source.

Cross-embodiment action canonicalization. To make action channels physically consistent across embodiments, we define common right-handed body and gripper frames. The body axes point forward, left, and up. At the open-gripper zero pose, $+z$ points from the mounting flange to the gripper tip, $+y$ from the right finger to the left, and $+x$ completes the frame. EEF poses are expressed in a canonical shoulder-centered frame, denoted SC. The frame is shared by both arms of a bimanual robot; for a single-arm embodiment without an explicit shoulder center, the arm-root frame is used instead. For arm $i$, the action contains the absolute EEF position ${}^{\mathrm{SC}}p_{\mathrm{EEF},t}^{(i)} \in\mathbb{R}^{3}$, a continuous 6D orientation representation $r_t^{(i)} \in\mathbb{R}^{6}$ [32], and normalized gripper closure $g_t^{(i)} \in[0,1]$, where 0 and 1 denote fully open and fully closed, respectively:

Single-arm samples use the 10-dimensional form. Let B be the source body frame, ${}^{\mathrm{SC}}T_{\mathrm{B}}$ the embodiment-specific rigid transform from B to SC, and ${}^{\mathrm{B}}T_{\mathrm{EEF},t}^{(i)}$ the source EEF target pose. Canonicalization is

The rotation and translation blocks of ${}^{c}T_{\mathrm{EEF},t}^{(i)}$ provide the canonical orientation and metric absolute position target, respectively; the translation is not a temporal displacement.

Hierarchical task annotation. Each episode receives an overall goal in global_task, while curr_subtask labels the interval in which each execution stage is active. The auxiliary scene field describes task-relevant objects and their spatial arrangement, and skill names the manipulation primitive. The resulting corpus pairs synchronized observations and canonical EEF actions with episode-level goals, time-aligned subtask labels, and scene and skill context, supporting mixed-granularity instruction conditioning during training.

Meituan-Robotics-0

Meituan-Robotics-0 aims to transfer broad priors learned from heterogeneous multimodal data to precise and robust real-robot control. This transition requires the model to preserve its visual-semantic representations while adapting its action interface and supervision to continuous cross-embodiment control and policy-induced error states. We first describe the policy architecture in § 3.1 and then present the three-stage training pipeline in § 3.2: discrete multimodal pre-training, continuous cross-embodiment mid-training, and real-world policy optimization.

Policy Architecture

MR0 formulates low-level desktop manipulation as reactive action-chunk prediction from the current observation. At this level, the policy maintains no temporal history and conditions only on the inputs available at the current decision step. Given multi-view observations $o_t = \{I_t^{(c)}\}_{c=1}^{C}$, language instruction $l$, and proprioceptive state $s_t$ at time $t$, MR0 predicts an $H$-step action chunk

Following the common VLA design [4, 33, 34], MR0 realizes this mapping with a shared vision–language backbone and two stage-specific output heads, as summarized in Figure 1. The shared backbone is initialized from Qwen3.5-4B [35] and remains trainable throughout the pipeline. During pre-training, FAST [36] applies a discrete cosine transform along the temporal axis and compresses the resulting coefficients with byte-pair encoding. The encoded action chunks share the backbone vocabulary and are predicted by the same autoregressive token head used for text.

For mid-training and all downstream stages, MR0 replaces the discrete interface with a continuous MMDiT action head [15, 33]. As illustrated in Figure 5, the head jointly processes an observation stream containing projected backbone features and a proprioceptive-state token, and an action stream containing a noisy action chunk. Joint self-attention exchanges information between the two streams, while the flow timestep modulates block normalization through AdaLN. The state token remains on the observation side, providing condition-side proprioceptive context throughout action generation.

Vector (PDF)

Progressive Training

MR0 is trained progressively to separate three sources of difficulty: heterogeneous multimodal learning, continuous cross-embodiment control, and recovery from policy-induced errors.

Discrete Multimodal Pre-Training

The shared token interface allows robot trajectories and vision–language samples to use the same autoregressive objective without a separate action decoder. FAST action tokens occupy a dedicated vocabulary block whose embeddings are initialized with the mean of the original embedding table. The remaining backbone parameters are inherited and trained end-to-end, and all camera views share the native vision encoder and aspect-preserving smart-resize policy.

Each optimization step draws one batch from the VLA action pool and one from the vision–language pool. The joint objective is

where both terms are next-token cross-entropy over their respective targets. The action loss grounds the backbone in robot behavior, while the VQA loss preserves perceptual and semantic supervision. Table 3 reports the stage-specific mixture. Its 13.6k-hour VLA pool is distinct from the complete 15,060-hour curated inventory in § 2.2. We train the backbone and extended action-token embeddings end to end for 500,000 steps on 256 GPUs using AdamW with cosine learning-rate decay. The resulting checkpoint initializes mid-training. Because FAST tokenization [36] introduces action-reconstruction error, downstream stages discard the token interface in favor of continuous prediction.

Continuous Cross-Embodiment Mid-Training

The second stage retains the token-pre-trained backbone and initializes the MMDiT action head [15] from scratch. This transfers the visual-semantic and behavioral representation learned during pre-training while replacing quantized action generation with continuous control.

Continuous action objective. We train the MMDiT head with the JiT clean-sample parameterization [37]. Following flow matching [38], a clean chunk $a_0$ is perturbed into a noisy sample $a_\tau$, from which the head predicts $\hat{a}_0$ rather than velocity directly. Prediction and target are converted to velocity form using a shared clamped denominator $d_\tau$ with $t_\epsilon= 0.05$, and the model is optimized by mean squared error.

Unified action representation. To apply the continuous objective across heterogeneous embodiments, MR0 maps each native action to the canonical end-effector convention defined in § 2.2. Each action chunk has a fixed width $d_{\max} = 20$, corresponding to two 10-dimensional end-effector actions, while a binary mask $m \in\{0,1\}^{d_{\max}}$ identifies the valid channels for each embodiment:

where $\ell_{h,j}$ is the element-wise JiT loss. Padded channels produce no gradient. For a downstream embodiment with a different action width, only the dimension-dependent projections are reinitialized.

In this stage, we train on 128 GPUs using AdamW with cosine learning-rate decay, applying a lower learning rate to the transferred backbone and a higher rate to the newly initialized action head. The training mixture starts from the data pools summarized in Table 3. We rebalance it by removing a 340-million-frame synthetic sub-corpus, retaining 10% of each of the two largest general-VQA sources, and sampling VLA datasets in proportion to the square root of their frame counts. To expose the model to different levels of linguistic abstraction, we sample the local subtask label or the episode-level goal with equal probability when both are available and otherwise use the available annotation. This language strategy pairs the same actions with both fine-grained and task-level instructions.

Real-World Policy Optimization

We first fine-tune the mid-training checkpoint $\pi_{\mathrm{mid}}$ on successful demonstrations to obtain an initial task policy. To address policy-induced distribution shift [39], we collect human-in-the-loop (HIL) corrections and compare three post-training algorithms. Human-gated DAgger [19] imitates demonstrations and corrections; RECAP [20] uses trajectory outcomes to learn advantage-conditioned behavior; and CFG-DAgger uses positive and negative segments supplied by Rollback. Each collection round adds to the cumulative dataset, and each policy is trained anew from $\pi_{\mathrm{mid}}$. All three methods execute action chunks through the asynchronous controller in § 4.1.

Human-gated DAgger. The policy controls the robot until an operator intervenes to correct a stall, error, or unsafe motion. Only human-controlled segments are retained as corrective supervision. Demonstrations cover the task procedure, while HIL corrections add recovery behavior in states encountered by the policy.

Rollback data collection. We introduce Rollback to pair replaced policy segments with human corrections from a shared anchor. An operator pauses execution near failure, selects an earlier recorded state, returns the robot to it using action history, and supplies a corrective continuation. In Figure 6, each anchor ($t_1$ or $t_2$) pairs a replaced segment $\tau^{-}$ with its human replacement $\tau^{+}$. Alongside demonstrations and human corrections, Rollback retains suboptimal segments to inform value learning and guide policy updates.

RECAP policy optimization. RECAP trains its value model on both successful and failed trajectories reconstructed from Rollback episodes. A trajectory ending with a replaced policy segment $\tau^{-}$ is assigned a failure label. Policy training uses only successful trajectories.² The learned value model provides $N$-step advantages for assigning positive ($I = 1$) or base ($I = \varnothing$) conditions to policy-training chunks. Both branches share the flow-matching loss. At inference, action-space classifier-free guidance (CFG) combines base and positive predictions generated from the same initial noise. Details are in Section C.2.

CFG-DAgger conditional policy training. CFG-DAgger directly trains the policy on demonstrations and $\tau^{+}$ as positive data and $\tau^{-}$ as negative data. A shared policy learns positive ($I = 1$), negative ($I = 0$), and unconditional ($I = \varnothing$) branches with the same flow-matching loss. At inference, CFG combines positive and negative predictions to favor corrective behavior and steer away from replaced actions. Details are in Section C.3.

By obtaining labels directly through Rollback collection, CFG-DAgger avoids value learning and advantage labeling, while outperforming RECAP in our real-robot evaluation (§ 6.1.2).

Embodied Manipulation System

MR0 augments the trained VLA policy with an inference-time system for continuous control and long-horizon task execution. An asynchronous execution layer aligns variable-latency action predictions with robot control, as detailed in § 4.1. At a higher level, an event-driven agent retains task-relevant state beyond the policy’s current observation. As summarized in Figure 8, motor control remains with the VLA policy, and high-level planning is activated only by events that alter task progress or feasibility.

Vector (PDF)

Asynchronous Policy Execution

Action chunks amortize policy inference over multiple control steps, but inference latency requires coordinating chunk generation with ongoing execution [40, 41, 42]. Our EVA-Client based stack [43] keeps control continuous by decoupling the two: a background policy thread predicts a chunk from the latest observation at target rate $f_{\pi}$, while the control loop consumes one buffered command at rate $f_{c}$.

This concurrency creates a temporal offset. By the time a new chunk $Q_{\mathrm{new}}$ arrives, the controller has consumed $k$ commands since the previous successful update. We denote the unexecuted remainder of the current buffer by $Q_{\mathrm{old}}$. Following the temporal chunk-wise smoothing of $\chi_{0}$ [44], we align the two sequences as shown in Figure 7. We first drop $d = \min(k, K_{\mathrm{lat}})$ leading commands and compute the remaining overlap:

Vector (PDF)

Here $K_{\mathrm{lat}}$ is a step-domain cap, set as $K_{\mathrm{lat}} \approx\lceil f_{c}\hat{\tau}_{e2e}\rceil$, where $\hat{\tau}_{e2e}$ is the delay from observation capture to insertion of the processed chunk into the buffer, including communication, policy inference, post-processing, and inverse kinematics. For each position $i = 0,\ldots,L-1$ in the overlapping region, the updated command is

The unused new suffix fills the future buffer, and $k$ is reset after a successful update.

The runtime parameters $f_{c}$, $f_{\pi}$, and $K_{\mathrm{lat}}$ jointly govern buffer consumption and responsiveness. When the inference pipeline meets its target rate, roughly $f_{c}/f_{\pi}$ commands are consumed between updates. If policy inference and sequential inverse kinematics together take longer than $1/f_{\pi}$, the actual update rate falls below $f_{\pi}$, and more buffered commands are consumed before the next update. Updates that arrive too slowly can exhaust the buffer, whereas overly frequent updates repeatedly restart blending from the current buffer and can slow the physical response. The cap $K_{\mathrm{lat}}$ balances removal of stale commands against the remaining executable horizon. The complete procedure and deployment settings are provided in Section B.

Event-Driven Embodied Agent

The chunk merger stabilizes local policy execution but does not preserve information across subtasks or determine when a high-level plan should change. The embodied agent addresses these requirements through explicit task state, persistent event memory, and event-triggered planning.

Task decomposition and execution. Given a user goal $g$, initial scene context $s_{0}$, and skill library $\mathcal{S}$, the Planner first produces an ordered sequence of executable subtasks $Q^{(0)}=\mathcal{P}_{\mathrm{plan}}(g,s_{0},\mathcal{S})$. During execution, the agent maintains the task state $X_{t}=(q_{t},R_{t})$, where $q_{t}$ is the active subtask and $R_{t}$ is the remaining sequence. The active subtask serves as the language instruction to the low-level policy, which generates actions as $\pi_{\mathrm{VLA}}(o_{t},q_{t})$. It remains active until the agent observes sufficient evidence of completion, failure, or a change in task feasibility. This state representation lets the VLA policy execute continuously while the Planner operates only at task-level transition points.

Experiments

Visual monitoring. The Monitor tracks local execution by mapping a short observation window $W_t$ and the active subtask to a structured caption $C_t = \mathcal{M}(W_t, q_t)$. Conditioning on $q_t$ focuses the description on relevant objects, spatial relations, interactions, and execution progress. The temporal window captures changes such as grasping, placement, or object removal that may be ambiguous in a single frame. The Monitor does not decide whether a subtask has finished or modify $X_t$; it only produces visually grounded evidence for event construction.

Event construction and memory. Monitor outputs enter a caption buffer in observation-time order. At each memory update, $B_t$ contains the captions that have not yet been processed, and the persistent history $\mathcal{H}_{<t}$ stores ordered event summaries together with their supporting keyframes. The Memory Manager converts the new captions into an event and determines whether that event warrants high-level reassessment:

When the new evidence continues the same interaction, it is merged into the latest memory entry; a distinct state change creates a new entry. For each event, the keyframe set $K_t$ is obtained by uniformly sampling frames from the observation windows associated with its captions. Memory insertion is independent of planner activation: routine observations can be recorded with $z_t = 0$, while subtask completion, execution failure, and changes in task preconditions set $z_t = 1$. This separation preserves information that may become relevant to a later instruction without repeatedly invoking the Planner over an unchanged state.

Event-triggered planning. The Planner performs two related operations. $\mathcal{P}_{\mathrm{plan}}$ constructs or revises a subtask sequence from the goal, scene context, task state, and available skills. When $z_t = 1$, $\mathcal{P}_{\mathrm{decide}}$ instead evaluates the current event, its keyframes, relevant entries retrieved from $\mathcal{H}_{<t}$, the task state $X_t$, and the user goal $g$. The resulting decision may preserve the active subtask, advance to the next one, wait for a temporally unsatisfied precondition, or invoke $\mathcal{P}_{\mathrm{plan}}$ to regenerate the remaining sequence. The Planner judges progress from the observed task state rather than requiring the complete action trajectory. When the event or its visual evidence is inconclusive, it leaves $X_t$ unchanged.

Asynchronous coordination. The Monitor, Memory Manager, Planner, and VLA policy operate at different rates. Multiple Monitor requests may run concurrently, so their outputs can arrive out of order; captions are therefore committed according to observation time before event construction. Planner inference does not pause monitoring or low-level execution, and captions produced during planning remain buffered for the next memory update. Task-state mutations are serialized after each Planner decision, preventing concurrent decisions from modifying the same execution state. The system consequently preserves continuous perception and control while keeping event memory and task transitions temporally consistent.

Simulation

RoboTwin2.0. RoboTwin2.0 [24] is a bimanual desktop-manipulation benchmark comprising 50 tasks with a 14-dimensional joint-space action interface. Each task provides 50 demonstrations in the clean-scene split (demo_clean) and 500 demonstrations in the randomized split (demo_randomized), which varies the background, table clutter, table height, and lighting. We jointly fine-tune all 50 tasks end-to-end from the stage-2 checkpoint using native-resolution inputs. Evaluation covers 100 episodes per task on each split, resulting in 5,000 episodes per split. We evaluate the VLA policy alone, without the agent system or RL post-training.

As shown in Table 4, Meituan-Robotics-0 achieves the highest overall success rate among the compared methods at 93.6%, with 93.2% on Clean and 93.9% on Randomized, a difference of 0.7 percentage points. The comparison with ABot-M0 is particularly informative: both methods predict clean actions and optimize a velocity-space loss, yet MR0 achieves 8.0 points higher overall success. Since MR0 adapts a shared stage-2 checkpoint learned from heterogeneous data and embodiments, these results demonstrate effective transfer to precise bimanual control and show that action-head parameterization alone does not determine downstream performance.

Method	Clean	Random.	Overall
$\pi_0$ [33]	65.9	58.4	62.2
$\pi_{0.5}$ [45]	82.7	76.8	79.8
X-VLA [7]	72.8	72.8	72.8
ABot-M0 [46]	86.1	85.1	85.6
Qwen-VLA-Instruct [4]	86.1	87.2	86.7
LingBot-VLA [6]	86.5	85.3	85.9
Motus [47]	88.7	87.0	87.8
StarVLA [5]	88.7	87.8	88.3
InternVLA-A1 [1]	89.4	89.6	89.5
Being-H0.7 [48]	90.2	89.6	89.9
Fast-WAM [49]	91.9	91.8	91.8
LingBot-VA [50]	92.9	91.5	92.2
InternVLA-A1.5 [3]	93.3	93.0	93.2
GalaxeaVLA (G0.5) [51]	93.7	92.8	93.3
Meituan-Robotics-0	93.2	93.9	93.6

RoboDojo. RoboDojo [25] evaluates generalist manipulation policies on 42 bimanual simulation tasks spanning Generalization, Precision, Long-Horizon, Memory, and Open-vocabulary instruction following. We extend the base MR0 policy with an initial-scene reference and motion history to provide temporal context. The reference uses the first main-camera image, while a frozen CD-LAM encoder [52] extracts latent actions from historical image pairs. We aggregate these features into one-second intervals over the latest twenty seconds and project them into the backbone with absolute and relative time encodings. Together with the reference image, they complement the current observations and task instruction while preserving the original action-learning objective. The embodied-agent system and RL-based post-training are not used. Table 5 reports Score and Success Rate.

Table 5: RoboDojo simulation results. The top four WAM and top nine VLA entries are selected by SR within the official categories as of October 26, 2026. Results are reported as Score/SR (%). Embodied Data reports disclosed embodied trajectory/video training hours before RoboDojo adaptation, including inherited pre-training where traceable; – denotes an unavailable verified duration. Bold indicates the best Score or SR among the displayed models in each column. Baseline results follow the official leaderboard; MR0 results are the equally weighted mean of two complete evaluations of the same checkpoint. Generalization averages the standard and randomized settings.

Model	Embodied Data	Avg. $\uparrow$	Generalization $\uparrow$	Precision $\uparrow$	Long-Horizon $\uparrow$	Memory $\uparrow$	Open $\uparrow$
WAM
Liber-0 Lite [53]	–	29.24/24.23	25.36/19.11	36.60/31.08	45.50/32.92	35.68/35.44	3.06/2.58
InternW0-Δ [54]	$\sim$23.1k h{sup}†`	30.77/23.91	30.09/22.78	31.98/23.25	46.29/29.33	34.67/34.00	10.84/10.17
Awomo-0.5 [55]	–	35.34/29.64	30.31/23.84	39.65/33.50	40.36/26.75	42.27/41.22	24.09/22.92
VPP2 [56]	–	39.26/32.26	36.38/29.50	43.03/35.17	56.28/37.00	56.01/55.56	4.61/4.08
VLA
SimpleMemVLA [57]	–	12.58/9.27	6.36/3.95	7.42/2.92	14.58/5.50	33.71/33.22	0.85/0.75
KinRT [58]	–	13.02/8.80	14.02/8.61	15.65/9.92	26.40/18.08	4.82/3.56	4.23/3.83
Hy-Embodied-0.5-VLA [13]	$>10$k h	13.07/8.80	11.78/8.39	13.81/8.00	25.74/14.92	13.37/12.11	0.65/0.58
Xiaomi-Robotics-1 [59]	$>100$k h{sup}‡`	20.07/13.93	23.54/17.00	26.69/18.83	38.39/23.67	7.81/6.56	3.94/3.58
Meituan-Robotics-0	$\sim$13.6k h	20.15/14.63	17.43/12.08	23.68/15.00	27.01/14.38	29.03/28.33	3.59/3.38
GalaxeaVLA (G0.5) [51]	–	20.23/14.88	18.46/12.83	28.25/20.42	44.12/32.25	8.61/7.33	1.73/1.58
DM0.5 [60]	–	24.90/19.34	15.77/10.95	24.82/16.75	33.70/19.50	47.74/47.44	2.43/2.08
Rex-M1 Preview [61]	$>100$k h{sup}‡`	33.79/27.44	34.05/26.05	32.82/23.83	46.89/33.00	48.53/48.22	6.67/6.08
Simate-beta [62]	–	33.95/27.96	35.09/27.95	34.35/26.92	57.84/43.42	33.33/33.00	9.12/8.50

With approximately 13.6k hours of embodied training data, the history-augmented MR0 achieves an average Score of 20.15 and an SR of 14.63%, ranking fifth by average success rate among the compared VLA models. These results highlight competitive overall performance at a relatively modest data scale, suggesting effective use of heterogeneous pre-training data. Open-vocabulary instruction following remains challenging, with a Score of 3.59 and an SR of 3.38%. On Align_blocks, the policy sometimes grasps the box rather than aligning it, illustrating a remaining bias toward familiar actions when interpreting unfamiliar instructions.

Real-Robot Evaluation

We evaluate Meituan-Robotics-0 at two complementary levels. The policy evaluation measures manipulation performance and assesses the contributions of HIL data and, for Ethernet insertion, replaced policy segments retained by Rollback. The agent evaluation examines two history-dependent capabilities: retrieving information no longer available in the current observation and resuming an interrupted task when its execution conditions are restored.

Experimental Setup

Tasks and metrics. We evaluate three tasks covering semantic grounding, deformable-object manipulation, and precise bimanual control (Figure 9). We report success rate (SR) and a task-specific progress score (PS):

Vector (PDF)

• Letter-block placement. The robot identifies, reorients, and places lettered blocks into target grooves to spell “MEITUAN”. Success requires all seven letters in the correct order and orientation; $\mathrm{PS}_{\mathrm{block}} = k/7$, where $k$ counts consecutive correct placements before the first error.

• Garment folding. The robot spreads and folds a garment into a target configuration. Success requires completion within 120 seconds. Evaluation continues for up to 300 seconds. Later completion counts as failure but retains

its actual completion time for scoring. Trials unfinished at 300 seconds receive $T = 300$. The progress score is $\mathrm{PS}_{\mathrm{fold}} = \operatorname{clip}((300-T)/210,0,1)$, where $T$ is the completion time in seconds.

• Ethernet insertion. The robot grasps a cable, transfers it between arms, and aligns and inserts the connector into a port. Success requires full insertion; $\mathrm{PS}_{\mathrm{ethernet}} = m/3$, where $m$ counts completed milestones: grasping, handover, and insertion.

Each policy is evaluated autonomously for 30 trials per task, with matched initial conditions and the same rollout protocol within each comparison. We average SR and PS over evaluation trials.

Post-training data. All tasks use successful teleoperated demonstrations. Letter-block placement and garment folding add corrections collected through standard HIL. Table 6 summarizes the data; setup and data preparation are detailed in Section A.

Task	Demo	HIL / Rollback	Total	Training Samples
Letter-block placement	500	200	700	$\sim900$K
Garment folding	400	400	800	$\sim1.1$M
Ethernet insertion	200	450	650	$\sim300$K

Compared policies. We compare MR0 with $\pi_{0.5}$ [45] after SFT and HG-DAgger post-training on all three tasks. Both backbones use the same task data, observation and action representations, preprocessing pipeline, and inference settings. On Ethernet insertion, we compare MR0 trained with HG-DAgger, RECAP, and CFG-DAgger using the same demonstrations and Rollback episodes. All three methods are trained from the same mid-training checkpoint, $\pi_{\mathrm{mid}}$.

Policy Evaluation

Main comparison. Under matched HG-DAgger post-training, MR0 and $\pi_{0.5}$ achieve the same SR on letter-block placement, while MR0 completes more placements before failure (Figure 10). Garment-folding SR is also similar, with slightly higher PS for MR0. On Ethernet insertion, MR0 achieves the largest improvements in both SR and PS over $\pi_{0.5}$.

Vector (PDF)

Effect of HIL data. HIL corrections substantially improve both backbones (Figure 10). Neither SFT policy succeeds on garment folding, highlighting the limited coverage of failure and recovery states in successful demonstrations alone. As HIL data increase (Figure 12), letter-block placement and garment folding improve rapidly before plateauing, while Ethernet insertion continues to improve. The gains from additional corrections therefore depend on the task and existing data coverage.

Vector (PDF)

Learning from Rollback pairs. CFG-DAgger achieves higher SR and PS than both baselines after each collection round (Figure 11). With the full Rollback dataset, CFG-DAgger reaches 0.70 SR and 0.90 PS, compared with 0.60 SR and 0.67 PS for RECAP. CFG-DAgger uses direct segment labels from Rollback and requires neither value learning nor advantage labeling.

Embodied Agent Evaluation

The policy experiments measure execution within a single task, but do not test whether the system can recover information that has disappeared from the current observation or preserve an objective across an interruption. We therefore conduct two controlled qualitative deployments: cross-task event retrieval and condition-aware task resumption.

Agent setup. The Monitor, Memory Manager, and Planner use Qwen3.5-35B-A3B and share an SGLang inference pool with four instances. The Monitor runs at 2 Hz over an eight-frame, two-second observation window. Each persistent event stores a semantic summary and four to eight archived keyframes.

Cross-task event retrieval. In Figure 13, the initial instruction is to place the tabletop objects in the bottom drawer and close it. After the key has been placed and the screwdriver subtask has begun, a person moves the key to the middle drawer. The Monitor captures this state change, and the Memory Manager stores the event with its keyframes. Because the active screwdriver subtask remains feasible, execution continues through the remaining queue and the bottom drawer is closed. This case evaluates event retention and later retrieval, rather than the preservation of all previously satisfied goal conditions under external intervention. When the user later requests the key, neither the closed cabinet nor the empty table reveals its location. The agent retrieves the stored event, identifies the middle drawer as the key’s last observed location, and generates a plan to recover it. This case shows that event summaries and archived visual evidence can support a new task when the relevant state is no longer directly observable.

Vector (PDF)

Interruption and condition recovery. In Figure 14, the agent is instructed to place the key in the bottom drawer and close it. A person removes the key after the drawer is opened but before the grasp is completed. The Monitor detects that the target is unavailable, while the Memory Manager records the condition change. The Planner retains the pending placement subtask and enters a waiting state rather than discarding or replanning the objective. When the key returns, a recovery event triggers reassessment against the latest observation, after which the robot places the key and completes the original instruction. This case demonstrates that persistent task state can preserve intent during temporary infeasibility and resume execution when the missing condition is restored.

Vector (PDF)

These qualitative deployments illustrate two uses of persistent context: retrieving information for a later task and preserving an objective during temporary infeasibility. Event memory supports retrieval when relevant state is no longer observable, while persistent task state allows execution to resume after the required conditions are restored.

Conclusion

We presented Meituan-Robotics-0, a vision-language-action foundation model and embodied system for stationary-base desktop manipulation. MRO combines heterogeneous data curation with a three-stage training recipe that connects multimodal pre-training, continuous cross-embodiment control, and real-world policy optimization. Rollback collection provides human corrections and replaced policy segments as direct supervision for CFG-DAgger, enabling policy improvement without value learning or advantage labeling. At deployment, asynchronous action-chunk merging supports continuous execution, while an event-driven agent maintains persistent context for task-level planning. Simulation and real-robot experiments demonstrate competitive manipulation performance, and qualitative deployments illustrate cross-task event retrieval and task resumption after temporary interruptions. These results support a systems approach in which aligned data, corrective learning, and persistent memory address complementary requirements of desktop manipulation. Future work will focus on agent and harness systems that interact more closely with VLA policies, using execution feedback to guide planning, memory updates, and policy adaptation for reliable long-horizon manipulation.

Contributions

All people are cataloged alphabetically by first name. (†) indicates the project leader and (‡) indicates the sponsors.

Contributors:

• Caicheng Wang

• Chen Zhao

• Chengpeng Jin

• Duo Xu

• Fengjiao Chen

• Guanglu Wan‡

• Heqing Yang

• Jia Wang

• Jiacong Zhou

• Jie Hu†

• Jiyuan Ru

• Lingfeng Zhang

• Linjiang Huang

• Manyuan Zhang

• Ningchu Liu

• Shijie Zhou

• Si Liu

• Shuo Li

• Tianyi Zhang

• Xi Zhang

• Xin Chen

• Xuezhi Cao

• Xunliang Cai‡

• Yanbin Zeng

• Yang Yang

• Yang Yi

• Yanli Tan

• Yifu Xu

• Yimeng Jia

• Yuchen Tang

• Yuchen Xie‡

• Yuxiang Zheng

• Zerwei Chng

• Zilong Huang

• Zipeng Ye

• Ziyuan Qin

• Ziqin Wang

Real-Robot Setup and Data Preparation

Data collection and recording. Our bimanual AgileX PiPER setup contains two human-operated leader arms and two corresponding task-executing follower arms. The operator controls the follower arms through leader-arm teleoperation. Each trajectory records three camera streams, the observed follower-arm configuration, and the commands sent to the follower arms. These commands are supplied by the leader arms during human control and by the policy during autonomous execution.

The demonstration dataset $\mathcal{D}_{\mathrm{demo}}$, collected by teleoperation, contains only successful executions. The tasks and data quantities used in our experiments are specified in § 6.1.

Real-robot observation and action representation. The policy receives synchronized images from three RGB cameras: one head camera and two wrist cameras. Each camera captures images at a native resolution of $640 \times480$. The policy uses 3 camera images at each timestep, without temporal history.

We use Qwen3.5-VL’s Smart Resize during both post-training and inference. It preserves the aspect ratio and bounds the visual token count, resizing images to approximately $352 \times256$ before visual encoding.

The policy state is derived from the observed follower-arm configuration, and the action target from the command sent to the follower arms. Both are converted to the same 20-dimensional end-effector representation. For each arm, this representation contains a 3D Cartesian position, a 6D rotation, and a normalized gripper closure value, with 0 denoting fully open and 1 fully closed:

The policy predicts an action chunk:

Static-interval trimming and training windows. Before constructing post-training samples, we remove sustained intervals in which changes in both observed arm configuration and commanded actions remain below task-specific thresholds. This limits the dominance of hold actions while preserving deliberate slow motion and corrections.

After trimming and source selection, trajectories are split at episode and control-source boundaries. Each training window contains $H = 50$ consecutive recorded actions from one retained segment, without tail padding or crossing a takeover boundary. Each window contributes one training sample in Table 6. SFT uses demonstrations, and HG-DAgger adds human corrections. RECAP and CFG-DAgger share the same demonstrations and collected Rollback episodes. Their sample-selection and labeling procedures are detailed in Sections C.2 and C.3. All retained SFT and HG-DAgger samples receive equal sampling weight.

Post-training implementation details. Small random crops and rotations augment the head-camera view; color jitter augments all three views. State and action dimensions are normalized using their 1st and 99th percentiles, with the same training-set statistics used to unnormalize actions during deployment.

Asynchronous Policy Execution Details

This section gives the complete buffer-update procedure for the asynchronous controller summarized in § 4.1. The control and inference loops share a locked action buffer and run concurrently.

The client retains all $H = 50$ returned actions before latency trimming. Two warm-up queries initialize the inference and IK pipeline. The buffer is then cleared, and the current robot state serves as the initial reference.

Real-World Post-Training Details

Supervised fine-tuning on successful demonstrations initializes data collection. Each post-training policy is trained anew from $\pi_{\mathrm{mid}}$ on the accumulated data. For the Ethernet Rollback comparison (Figure 11), each of the three methods contributes 50 episodes per round to a shared pool of 150 new Rollback episodes. Segment annotations identify replaced branches and human corrections. All methods use the interface in Section A.

Quantity or option	Value	Role
Strategy	Temporal chunk-wise smoothing	Background prediction with latency-aware buffer updates.
$f_c$	30 Hz	Command-consumption and publication rate of the control loop.
$f_\pi$	3 Hz	Target background prediction rate.
$H$	50	Action horizon returned by the policy server.
$K_{\mathrm{lat}}$	4	Maximum number of leading commands removed from an incoming chunk.
$m_{\min}$	8	Minimum old-reference length used for short-buffer padding.
Warm-up queries	2	Synchronous startup queries before the background loop begins.

HG-DAgger Human Intervention

Let $\pi_0$ be the SFT policy trained on demonstrations $\mathcal{D}_{\mathrm{demo}}$. During collection round $k$, the robot executes autonomously under $\pi_{k-1}$ ($g_t=0$) until the operator takes control ($g_t=1$). The operator intervenes when execution stalls, errs, or becomes unsafe, and returns control to the policy after recovery.

Only human-controlled samples from successful episodes are retained, whether collected through standard HIL or Rollback:

After round $k$, the cumulative supervised dataset is

HG-DAgger changes data collection and sample selection but retains the flow-matching action loss.

RECAP Policy Optimization

Data selection and conditioning. Value learning uses successful and failed trajectories. Policy training excludes failed trajectories. RECAP uses positive and base conditions, with no explicit negative condition.

Trajectory reconstruction. For each Rollback episode, we reconstruct paths by joining the recorded prefix to each replaced branch or corrected continuation. Replaced branches terminate at intervention and receive failure labels. Let $\mathcal{T}_{\mathrm{RB}}^{(k)}$ denote the trajectories reconstructed from collection round $k$. The cumulative dataset is

Outcome rewards and value targets. Trajectory outcome labels are converted into per-timestep rewards:

where $T$ is the terminal timestep of the reconstructed trajectory (the intervention point for a replaced branch) and $T_{\max}$ is the maximum episode length. Nonterminal steps incur unit cost, and a failure label adds a terminal penalty of $T_{\max}$. Normalizing by $D=2T_{\max}$ gives the clipped value target

Algorithm 1 Latency-Aware Asynchronous Policy Execution

Require: Action horizon $H$, control rate $f_c$, target inference rate $f_\pi$, latency cap $K_{\mathrm{lat}}$, minimum old-reference length $m_{\min}$

1: Shared state: buffer $Q \leftarrow \varnothing$, counter $k \leftarrow 0$, and $q_{\mathrm{last}} \leftarrow q_{\mathrm{current}}$
2: Each pop and each complete buffer update are protected by the same lock.
3: Run the following control and inference loops concurrently.
4: Control loop (at $f_c$):
5: while the system is running do
6:   if $Q \ne \varnothing$ then
7:     $q \leftarrow \operatorname{POPFRONT}(Q)$; $k \leftarrow k+1$
8:     if $Q = \varnothing$ then
9:       $q_{\mathrm{last}} \leftarrow q$
10:    end if
11:    $\operatorname{PUBLISH}(q)$
12:  end if
13: end while
14: Inference loop (at target $f_\pi$):
15: while the system is running do
16:   $Q_{\mathrm{new}} \leftarrow \operatorname{IK}(\operatorname{PREDICT}(\operatorname{OBSERVE}()))$ ▷ $|Q_{\mathrm{new}}|=H$
17:   if $Q_{\mathrm{new}} = \varnothing$ then
18:     continue
19:   end if
20:   $d \leftarrow \min(k,K_{\mathrm{lat}})$
21:   if $d \ge |Q_{\mathrm{new}}|$ then
22:     continue ▷ $Q$ and $k$ remain unchanged
23:   end if
24:   $\overline{Q}_{\mathrm{new}} \leftarrow Q_{\mathrm{new}}[d:]$
25:   if $Q = \varnothing \land q_{\mathrm{last}} \ne \varnothing$ then
26:     $Q_{\mathrm{old}} \leftarrow [q_{\mathrm{last}}]^{m_{\min}}$; $q_{\mathrm{last}} \leftarrow \varnothing$
27:   else if $Q = \varnothing$ then
28:     $Q \leftarrow \overline{Q}_{\mathrm{new}}$; $k \leftarrow 0$
29:     continue
30:   else
31:     $Q_{\mathrm{old}} \leftarrow Q$
32:     if $|Q_{\mathrm{old}}| < m_{\min}$ then
33:       Extend $Q_{\mathrm{old}}$ to $m_{\min}$ by repeating its terminal command
34:     end if
35:   end if
36:   $L \leftarrow \min(|Q_{\mathrm{old}}|,|\overline{Q}_{\mathrm{new}}|)$
37:   for $i=0$ to $L-1$ do
38:     $\alpha_i \leftarrow i/\max(L-1,1)$
39:     $q_i^{\mathrm{fused}} \leftarrow (1-\alpha_i)q_i^{\mathrm{old}}+\alpha_i\overline{q}_i^{\mathrm{new}}$ ▷ Linear transition
40:   end for
41:   $Q \leftarrow [q_0^{\mathrm{fused}},\ldots,q_{L-1}^{\mathrm{fused}}]\mathbin{\Vert}\overline{Q}_{\mathrm{new}}[L:]$
42:   $k \leftarrow 0$
43: end while

Targets lie in $[-1,0]$, with values closer to zero indicating shorter successful completions and the terminal penalty distinguishing failures.

Distributional value prediction. The value network predicts a categorical distribution over $B=201$ atoms:

Algorithm 2 HG-DAgger real-world post-training

Require: Mid-training checkpoint $\pi_{\mathrm{mid}}$, demonstration dataset $\mathcal{D}_{\mathrm{demo}}$, rounds $K$

1. $\mathcal{D}_{\mathrm{DAgger}}^{(0)} \leftarrow\mathcal{D}_{\mathrm{demo}}$

2. Train $\pi_0$ from $\pi_{\mathrm{mid}}$ on $\mathcal{D}_{\mathrm{DAgger}}^{(0)}$

3. for $k = 1$ to $K$ do

4. &nbsp;&nbsp;&nbsp;&nbsp;Deploy $\pi_{k-1}$ under operator gating; collect $\mathcal{D}_{\mathrm{HIL}}^{(k)}$

5. &nbsp;&nbsp;&nbsp;&nbsp;$\mathcal{D}_{\mathrm{DAgger}}^{(k)} \leftarrow\mathcal{D}_{\mathrm{DAgger}}^{(k-1)} \cup\mathcal{D}_{\mathrm{HIL}}^{(k)}$

6. &nbsp;&nbsp;&nbsp;&nbsp;Train $\pi_k$ from $\pi_{\mathrm{mid}}$ on $\mathcal{D}_{\mathrm{DAgger}}^{(k)}$

7. end for

Ensure: Final task policy $\pi_K$

Algorithm 3 RECAP policy optimization with Rollback trajectories

Require: $\pi_{\mathrm{mid}}$, value initialization $V_{\mathrm{init}}$, demonstrations $\mathcal{D}_{\mathrm{demo}}$, Rollback batches $1,\ldots,K$

1. $\mathcal{D}_{\mathrm{RL}}^{(0)} \leftarrow\mathcal{D}_{\mathrm{demo}}$

2. for $k = 1$ to $K$ do

3. &nbsp;&nbsp;&nbsp;&nbsp;Reconstruct $\mathcal{T}_{\mathrm{RB}}^{(k)}$ from Rollback batch $k$; label replaced branches as failures at intervention

4. &nbsp;&nbsp;&nbsp;&nbsp;$\mathcal{D}_{\mathrm{RL}}^{(k)} \leftarrow\mathcal{D}_{\mathrm{RL}}^{(k-1)} \cup\mathcal{T}_{\mathrm{RB}}^{(k)}$

5. &nbsp;&nbsp;&nbsp;&nbsp;Fit $V_k$ from $V_{\mathrm{init}}$ on $\mathcal{D}_{\mathrm{RL}}^{(k)}$ using outcome-derived targets

6. &nbsp;&nbsp;&nbsp;&nbsp;Assign $I_t = 1$ to HIL chunks; label other training windows using Equation (23)

7. &nbsp;&nbsp;&nbsp;&nbsp;Select policy-training trajectories, excluding failures, to form $\mathcal{D}_{\mathrm{policy}}^{(k)}$

8. &nbsp;&nbsp;&nbsp;&nbsp;Train $\pi_k^{\mathrm{RL}}$ from $\pi_{\mathrm{mid}}$ on $\mathcal{D}_{\mathrm{policy}}^{(k)}$ with advantage conditioning

9. end for

Ensure: Final task policy $\pi_K^{\mathrm{RL}}$

where $V_{\min}=-1$, $V_{\max}=0$, and the atom spacing is $\Delta z=0.005$. We project a scalar target onto its two neighboring atoms using

When $l<u$, the target masses are $q_l=u-b_t$ and $q_u=b_t-l$. When $l=u$, we set $q_l=1$; all other entries are zero. The value loss and continuous prediction are

Value-model architecture. Following RLinf’s RECAP implementation [63], a SigLIP2-So400m/14 encoder maps each of the three RGB views to 256 tokens projected to 640 dimensions. An 18-layer Gemma 3 270M backbone encodes these visual tokens together with the instruction.

A separate 18-layer value expert with hidden dimension 256 updates a learnable [CLS] token. It attends to the vision-language prefix through shared attention keys and values, while the prefix cannot attend back to it. A linear head predicts the value distribution:

The vision encoder and Gemma backbone use pretrained weights. The value expert and head are randomly initialized, while the [CLS] token is initialized to zero. All components are trained jointly.

$N$-step advantage labeling. Transition rewards use the same scale as the normalized value targets:

For a training window starting at frame $t$ with a complete $N$-step horizon, the advantage is

If $t+N>T$, rewards are accumulated through termination and the bootstrap value beyond the episode is set to zero. Before condition dropout, human-controlled (HIL) chunks are always assigned $I_t=1$. For non-HIL chunks, we assign a conditioning state $I_t\in\{1,\varnothing\}$ using the advantage threshold:

Here, $N=H=50$ and $\epsilon$ is the 70th percentile of advantages over the aggregated dataset. Each non-HIL training window inherits the conditioning state of its first frame.

Advantage-conditioned policy training. Before condition dropout, the conditioning state determines the suffix appended to the task instruction:

For chunks with $I_t=1$, the positive suffix is dropped with probability $p_{\mathrm{drop}}=0.3$; chunks with $I_t=\varnothing$ always use the base instruction. All retained chunks use the flow-matching action loss.

Action-space classifier-free guidance. At inference, we denoise a base prediction $a_{\varnothing}$ ($I=\varnothing$) and a positive-conditioned prediction $a_{+}$ ($I=1$) from the same initial noise, then combine the completed action chunks once in action space:

The base branch omits only the advantage condition, retaining the observation and task instruction.

Training configuration. Table 8 summarizes the training configuration.

Parameter	Setting
Value support $[V_{\min}, V_{\max}]$	$[-1, 0]$
Number of value atoms $B$	201
Value learning steps / batch size	6,000 / 256
Value peak learning rate	$5 \times10^{-5}$
Value warm-up steps	500
Value final learning rate	$1 \times10^{-6}$
Advantage horizon $N$ / action horizon $H$	50 / 50
Advantage threshold $\epsilon$	Aggregated-data 70th percentile
Policy conditions	Positive and base; no explicit negative
Condition-dropout rate $p_{\mathrm{drop}}$	0.3
Policy training steps / batch size	30,000 / 128
Policy peak learning rate	$1 \times10^{-5}$
Policy warm-up steps	10,000
Policy minimum learning rate	$5 \times10^{-7}$
Inference guidance scale $\beta$	1.75

CFG-DAgger Conditional Policy Training

Data and labels. In Figure 6, each pair $(\tau^{-},\tau^{+})$ shares an anchor but may diverge afterward; no frame-wise alignment is required. If the scene cannot be restored to an anchor, collection falls back to standard HIL and contributes only positive data.

Replaced segments carry $I=0$, while demonstrations and HIL corrections carry $I=1$:

where $\mathcal{D}_{\mathrm{HIL}}$ includes the human replacements $\tau^{+}$. Labels describe local segments, independent of episode outcomes.

Conditional policy training. A shared policy $\pi_{\theta}(a_{t:t+H-1}\mid o_t,\ell,I)$ learns positive ($I=1$), negative ($I=0$), and unconditional ($I=\varnothing$) branches using the flow-matching loss. Positive samples use $\mathcal{D}^{+}$, with condition dropout of 0.3 to train the unconditional branch while retaining the task instruction. Negative samples from $\mathcal{D}^{-}$ always retain their condition. Other autonomous segments, rollback/reset motions, and segments shorter than $H=50$ are excluded. Valid chunks are sampled without class rebalancing, and each round trains from $\pi_{\mathrm{mid}}$ on the cumulative dataset.

Action-space classifier-free guidance. At inference, we denoise a positive prediction $a_{+}$ ($I = 1$) and a negative prediction $a_{-}$ ($I = 0$) from the same initial noise, then combine the completed action chunks once in action space:

We use $\gamma= 1.25$, extrapolating from the positive prediction away from the negative prediction. The unconditional branch is used only during training.