betaFeedback

WorksComputer Science

Tracking Task Progress Through Errors and Recovery:A Benchmark and Recurrent Distributional Model

Abstract

Operational strategies for long-horizon robotic tasks require reliable evaluation to determine whether execution is progressing, stalling, regressing, or recovering from errors. Yet real- world robot datasets that capture error recovery and provide frame-level error annotations remain scarce. To address this gap, we introduce and open-source ReSTEP, a dataset of 3,156 episodes across two manipulation tasks with semantic stage labels and frame-level er- ror annotations. ReSTEP supports training and benchmarking progress monitors through successful execution, mistakes, and recovery. Existing progress estimators can struggle to distinguish visually similar task stages and maintain temporally consistent predictions while responding to genuine setbacks. We therefore propose REMORA, an online progress esti- mator that combines recurrent visual memory, distributional prediction, and causal peak selection. Memory retains execution history to help resolve visual ambiguity, while distribu- tional prediction represents alternative progress hypotheses and causal peak selection tracks a supported hypothesis over time. Together, these components enable consistent progress tracking that remains responsive to errors and recovery. Experiments on both ReSTEP tasks demonstrate that REMORA outperforms the state-of-the-art SARM baseline in progress ac- curacy and temporal consistency.

Introduction

Recent advances in vision-language-action (VLA) models have enabled language-conditioned robots to perform increasingly diverse manipulation tasks and generalize to unfamiliar objects and environments [1, 2, 3, 4]. However, strong action-generation capabilities do not by themselves provide reliable self-assessment: a policy may produce plausible motions while stalling or undoing earlier work. This limitation becomes especially consequential in long-horizon tasks. A complementary task-progress estimator can provide monitoring, changes in estimated task progress can be used to construct dense rewards for reinforcement learning [5, 6] and to reweight demonstration samples in imitation learning [7, 8, 9]. Progress feedback could also help address the substantial data demands of VLA training by identifying stalled or regressing executions that warrant expert corrections during DAGger-style data collection [10, 11]. These applications require progress estimates that remain temporally consistent under partial observability while remaining sensitive to genuine setbacks.

Existing work on visual progress and reward estimation broadly follows two scoring designs. One derives scores from VLM embedding similarities [12, 13] or judgments generated by pretrained or fine-tuned VLMs [5, 6, 7, 14, 15, 16]. The other trains dedicated prediction heads or networks for progress, relative progress, or task completion, including heads built on pretrained VLM or VLA representations [8, 9, 17, 18, 19]. These approaches demonstrate the utility of visual progress signals, with several reporting improvements in downstream robot learning. Nevertheless, limitations in spatial and temporal reasoning can make VLM-derived scores unreliable for subtle changes in task state [13, 14]. Coarse scoring scales can obscure small improvements [15, 16], while time-derived supervision can conflate elapsed time with actual task advancement [8, 14, 16, 18]. Moreover, some methods rely on full-trajectory context or retrospective progress reconstruction, limiting their applicability to online progress monitoring [7, 9].

Stage-aware progress supervision in a representative bottle-transfer episode

Figure 1. Stage-aware progress supervision in a representative bottle-transfer episode. Head-camera observations are partitioned into four task stages, whose start frames are shown below the trajectory. The white curve denotes the normalized progress target and the cyan curve denotes the corresponding value estimate. Progress is anchored at V=0V = 0 at the episode start and V=1V = 1 upon task completion.

Beyond these challenges, progress estimation requires modeling temporal dependencies in both visual observations and progress predictions. On the input side, successive images reveal motion, state changes, and event order that may be ambiguous or unobservable in an isolated frame. Integrating this information helps distinguish visually similar observations associated with different task stages [17]. However, short or sparsely sampled observation histories can omit earlier events needed to interpret the current state, while processing more frames increases computational cost [8, 19]. On the output side, successive progress estimates should describe a coherent task evolution. Many methods ultimately produce a scalar estimate through direct regression or by taking the expectation of a predicted distribution [8, 18, 19]. When the available visual history leaves multiple task stages plausible, the conditional progress distribution can remain multimodal. Squared-error regression or an expectation-based readout can then average distinct hypotheses into an intermediate value that corresponds to neither stage.

Learning from unsuccessful executions presents a further challenge because dataset with error frame annotation are difficult to obtain. Methods such as ReWiND and SARM therefore augment demonstrations with temporally reversed segments to provide supervision for decreasing progress [8, 18]. Although useful, time reversal can produce physically implausible sequences and does not necessarily capture the dynamics of real mistakes and subsequent recovery [18]. Restricting supervision to successful demonstrations and synthetic reversals can consequently leave informative failure and recovery behavior underused.

We introduce REMORA, a framework for online task-progress estimation that combines temporal visual context with distributional prediction and causal peak selection (Figure 2). We develop and study this framework on two real-world logistics tasks using demonstrations that include successful executions, mistakes, and recoveries. Comparison with SARM on parcel handling shows that REMORA achieves lower mean absolute error and higher Spearman rank correlation, indicating more accurate progress estimates and better preservation of the relative progress ordering across observations.

Figure 2. Overview of REMORA. Recurrent memory enriches multi-camera visual features before the VLA backbone and categorical value head predict a progress distribution. During training, stage and error annotations provide soft progress targets for distributional supervision. During online inference, progress tracking selects a supported peak using the previous estimate. Colors indicate frozen and trainable modules in the frozen-backbone configuration.

Our contributions are threefold:

  • ReSTEP: An error-annotated dataset for progress estimation. We release more than 15 hours of robot demonstrations across 3,156 episodes covering parcel handling and medicine-bottle scanning and storage. Semantic stage annotations and frame-level error labels capture successful executions, mistakes, and recoveries, supporting both model training and benchmarking.

  • A recurrent distributional progress model. We augment a pretrained VLA backbone with categorical value heads and a recurrent visual memory to represent ambiguous progress states in their temporal context. Stage and error annotations supervise both advancing and decreasing progress, retaining observations of mistakes and recoveries during training.

  • Causal peak selection for temporal consistency. We introduce a decoder that selects a supported peak from the current progress distribution based on the previous estimate. This rule favors continuity across successive predictions while allowing decreases supported by the distribution, providing a causal readout of potentially multimodal progress.

Related Work

Inverse reinforcement learning

Inverse reinforcement learning (IRL) is closely related to progress and visual reward modeling: both learn evaluative signals from data to reduce manual reward design. IRL seeks rewards that explain demonstrated behavior and can guide policy learning toward similar behavior [20]. Early feature-based IRL methods share the principle of matching demonstration statistics: apprenticeship learning seeks policies with matching feature expectations [21], while maximum-entropy IRL selects the highest-entropy trajectory distribution consistent with these constraints [22]. Later neural approaches extend this foundation: Guided Cost Learning combines maximum-entropy learning of neural costs with sample-based policy optimization [23], while GAIL connects regularized IRL to adversarial imitation by matching expert and learner state–action visitation distributions [24]. These lines of work overlap, but differ in their primary objective: IRL explains demonstrated behavior through rewards, whereas progress and visual reward models directly score task completion or advancement from observations. Such scores can assess execution without requiring the evaluated policy to reproduce the demonstrator’s behavior.

Progress and visual reward models

Progress and visual reward models convert observations into scores of task advancement, providing feedback for policy learning and execution monitoring. Early progress prediction uses recurrent video models [17]; temporal cycle consistency learns frame alignments [25], which XIRL uses to construct goal-reaching rewards across embodiments [26]. Subsequent work pursues both transfer across tasks and precise estimation within specified tasks.

Multi-task approaches seek reusable reward functions across language-specified goals. LIV and RoboCLIP exploit vision–language representations for reward prediction [12, 27], while VLC and ReWIND incorporate temporal ranking and progress supervision, respectively, to improve transfer [18, 28]. Building on broader VLM capabilities, GVL and ProgressLM infer progress using visual context or reference demonstrations [7, 14]. RoboReward and Large Reward Models scale supervised outcome and progress scoring across diverse tasks [15, 16], while Robometer adds trajectory comparisons to learn from unsuccessful executions [19].

Task-level approaches fit evaluation to a specific task. RL-VLM-F learns a separate reward model from VLM preferences for each task [29], while SARM uses procedure-specific stage annotations to account for variations in execution timing [8]. These directions are complementary: Robo-Dopamine and ProcVLM combine broad pretraining with task-specific adaptation [6, 30].

Our work adopts the task-level setting because reliable temporal estimation remains challenging even for familiar procedures: visually similar states can correspond to different stages, and mistakes can interrupt or reverse progress [6, 8]. Broad task coverage alone does not establish accurate, temporally coherent monitoring. Distributional models already represent progress, remaining steps, or returns, but use expected-value readouts [19, 31, 32]. Task-specific stage and mistake annotations provide a controlled basis for studying these unresolved estimation problems. Within this setting, we combine recurrent visual memory with causal peak selection to track a supported progress hypothesis through mistakes and recovery.

Method

Problem formulation and overview

At time tt, the estimator receives multi-camera observations ot\mathbf{o}_t, robot state st\mathbf{s}_t, and a language instruction ℓ\ell. It estimates normalized task progress using observations available through tt. During training, semantic stage boundaries and annotated error intervals define a reference target y~t∈[0,1]\tilde{y}_t \in[0,1]. We model the corresponding progress distribution as

pt(k)=pθ(Y~t=zk∣o≤t,st,ℓ),(1)p_t(k) = p_\theta\left(\tilde{Y}_t = z_k \mid\mathbf{o}_{\leq t}, \mathbf{s}_t, \ell\right), \tag*{(1)}

where {zk}k=0K−1\{z_k\}_{k=0}^{K-1} are fixed progress atoms spanning [0,1][0,1]. Throughout the paper, “value” refers to annotation-defined task progress.

Fig 2 summarizes the training and online inference pipeline. Multi-camera images are encoded by pretrained vision encoder SigLIP [33] and enriched with execution history through recurrent memory fusion. The VLA backbone combines these visual features with the language instruction and robot state, and a categorical value head reads out the action-expert suffix conditioned on all-zero action input to predict the progress distribution ptp_t. During training, semantic stage boundaries and annotated error intervals define progress targets, which are encoded as soft labels to supervise this distribution. During online inference, progress tracking selects a supported peak according to the previous estimate y^t−1\hat{y}_{t-1}, yielding the current estimate y^t\hat{y}_t. Visual memory thus supplies historical context for distribution prediction, while peak association maintains continuity in the scalar readout. Neither stage nor error annotations are required at inference.

Error-aware progress targets

We construct progress targets from semantic stage annotations and localized execution errors. Each stage is assigned a fixed progress mass, shared across demonstrations of the same task, with the task-specific weights specified in the implementation details. Stages omitted because their requirements are already satisfied contribute initial progress credit. For stage jj spanning frames [bj,dj)[b_j,d_j), let BjB_j and Bj+1B_{j+1} denote its prescribed progress boundaries. In the absence of errors, the target increases linearly between these boundaries. This construction aligns demonstrations at semantic stage endpoints while allowing their execution durations to vary.

To retain supervision from real mistakes, we replace continued growth within annotated error intervals with a bounded decrease. For an error interval of length LL ending at frame cc, let vv be progress immediately before entering the error interval and α\alpha the preceding positive slope. The target decreases by β\beta per frame during the error and then increases with a recomputed slope α+\alpha^+:

β=min⁡(α,vL),\beta= \min\left(\alpha,\frac{v}{L}\right),
α+=Bj+1−(v−βL)dj−c,c<dj.(2)\alpha^+ = \frac{B_{j+1}-(v-\beta L)}{d_j-c}, \qquad c<d_j. \tag*{(2)}

The cap keeps progress nonnegative and limits the descent rate to the preceding rate of advancement. After each error, the positive slope is recomputed toward the stage endpoint and applied until the next error or stage boundary. These targets provide an annotation-based proxy for setbacks and recovery, preserving mistake observations for value learning. The annotations are used only to construct offline supervision and are not required during online inference.

Categorical prediction and online peak association

Categorical progress prediction Visually similar observations can correspond to different task stages. We therefore predict a categorical progress distribution that can retain multiple plausible hypotheses. Target annotated in 3.2 are encoded as soft labels over neighboring progress bins, which means each progress target from 3.2 is assigned to its two neighboring bins using linear interpolation to reduce inaccuracy introduced by discretizations. The prediction heads are trained with cross-entropy.

Online peak association Independently selecting the dominant peak can cause predictions to jump between competing hypotheses. We instead select the supported peak closest to the previous estimate:

k^t=arg⁡min⁡k∈Pt∣zk−y^t−1∣,y^t=zk^t,(3)\hat{k}_t = \arg\min_{k\in\mathcal{P}_t}\left|z_k-\hat{y}_{t-1}\right|,\qquad\hat{y}_t=z_{\hat{k}_t}, \tag*{(3)}

where Pt\mathcal{P}_t indexes the candidate peaks extracted from ptp_t. This association favors temporal continuity while allowing progress to increase or decrease.

Gated recurrent visual memory

To incorporate execution history into progress estimation, we recurrently update a fixed-size visual memory as new frames arrive. At each step, the previous memory and current visual features jointly determine the updated state, which enriches the current frame representation and is carried forward to the next step. This provides temporal context before the progress distribution is predicted, complementing the online peak association that subsequently selects among its modes. As shown in Figure 3, each of the CC cameras maintains eight memory slots, with independent runtime states and shared parameters θmem\theta_{\mathrm{mem}}. For camera cc, the visual encoder produces projected patch tokens Ft(c)\mathbf{F}^{(c)}_t from the current frame. We omit the camera superscript below for clarity.

Figure 3. Gated recurrent visual memory. Each camera maintains eight memory slots, with independent states and shared update parameters. Memory queries current visual tokens to form a candidate state, and a frame-gap-conditioned gate updates the memory. Current visual tokens then attend to the updated memory for residual fusion. The updated state is carried to the next frame with stopped gradients during training and reset at stream boundaries.

In the Read step, cross-attention AwA_w uses the previous memory Mt−1\mathbf{M}_{t-1} as queries and the current patch tokens as keys and values, denoted by QQ, KK, and VV in Figure 3. Its output is added to the previous memory and normalized to form the candidate state Ct\mathbf{C}_t, corresponding to Add & Norm.

In the Update step, the Gate network combines the previous memory, the candidate state, and an encoding of the frame gap Δft\Delta f_t. The sigmoid function σ\sigma converts its output into elementwise update weights Gt∈[0,1]\mathbf{G}_t \in[0,1]:

Mt=LN⁡((1−Gt)⊙Mt−1+Gt⊙Ct).(4)\mathbf{M}_t = \operatorname{LN}((1-\mathbf{G}_t) \odot\mathbf{M}_{t-1} + \mathbf{G}_t \odot\mathbf{C}_t). \tag*{(4)}

Here, LN denotes layer normalization and ⊙\odot denotes elementwise multiplication. Small gate values favor retaining previous features, while large values favor incorporating the candidate state.

In the Readback step, cross-attention Ar\mathbf{A}_r reverses the attention direction: current patch tokens provide the queries, and updated memory provides the keys and values. The resulting readout Rt\mathbf{R}_t passes through a learned output projection and is added to the current patch tokens by Residual Fusion, supplying the VLA predictor with temporal context. Between frames, sg⁡(Mt)\operatorname{sg}(\mathbf{M}_t) carries the updated state forward, where sg denotes stop-gradient during training. At reset boundaries, all camera memories return to the learned initial state M0\mathbf{M}_0, and the decoder’s previous progress estimate is cleared jointly.

We introduce ReSTEP (Real-Robot Stage-Annotated Trajectories for Error-Aware Progress), a dataset covering two practical manipulation scenarios: parcel handling (D1D_1) and medicine-bottle scanning and storage (D2D_2). It contains 3,156 episodes and 1,692,390 synchronized frames, corresponding to approximately 15.7 h of robot operation. The dataset includes expert demonstrations and autonomous executions with human-in-the-loop corrections. We distinguish mistake-free episodes, which contain no execution errors, from mistake episodes, which contain errors followed by recovery. Both types end in successful task completion. Semantic stage annotations identify task phases, while frame-level error annotations localize execution mistakes.

We will publicly release ReSTEP, containing the trajectories and their stage and error annotations to support progress-model training and evaluation. Our dataset also contains full observation and action information, which can also be used to train vision-language-action (VLA) or world-action (WA) models.

Tasks and Data Collection

All trajectories were collected on an AgiBot G02 robot and contain synchronized head and bilateral wrist RGB observations, proprioception, and actions at 30 Hz. Expert demonstrations were acquired through teleoperation. These demonstrations also include episodes with execution mistakes, as errors can occur during teleoperation. The bottle data additionally include human-in-the-loop (HIL) collection, in which a policy acts autonomously and an operator intervenes when correction is needed. With the inclusion of HIL trajectories, D2D_2 contains a higher proportion of episodes with execution mistakes than D1D_1. Table 1 summarizes the statistics for D1D_1 and D2D_2.

SourceEpisodesFramesMistake episodes
D1D1 (expert)1,953793,63218
D2D2 (expert)959657,15966
D2D2 (HIL)244241,599179
Total3,1561,692,390263

Table 1. ReSTEP composition by task and collection source. Mistake episodes contain an available mistake annotation; all episodes end in successful task completion.

D1D_1: Parcel handling The parcel data contain 1,953 expert-demonstration episodes across eight scenarios, combining package type (box or bag), label orientation (up or down), and initial placement (centered or off-center). Using dexterous hands, the robot centers the parcel when necessary, flips it to expose the label, pushes it away, and returns to its initial pose. The corresponding semantic stages are move, flip, push, and return. Centered parcels omit the move stage, and parcels with an upward-facing label omit the flip stage, yielding four stage sequences with two to four stages per episode. These variations allow progress estimation to be studied under different initial conditions and task-stage sequences.

D2D_2: Medicine-bottle scanning and storage The bottle data contain 959 expert-demonstration episodes and 244 HIL episodes. Using suction-equipped grippers, the robot grasps a bottle from a front bin, rotates it to expose its QR code to the head-mounted camera, places it in a neighboring basket, and resets its arms and body. Each episode has four ordered stages: grasp, scan, place, and reset. HIL trajectories include autonomous behavior and corrective actions within the same execution, providing observations of mistakes and subsequent recovery.

Stage and Error Annotations

Each stage annotation consists of a semantic label and a half-open frame interval [sk,ek)[s_k, e_k). Stage annotations cover 584,348 frames in parcel handling and 699,433 frames in bottle scanning. They identify the task phase associated with an observation and provide anchors for constructing progress supervision across episodes of different durations. Stage annotations span the interval from task initiation to completion. The recordings also include frames before task initiation and after completion, so the number of stage-annotated frames is smaller than the total number of recorded frames. Separate Error Frame annotations localize execution errors, including empty grasps and object drops. Parcel handling contains 18 error intervals covering 461 frames, and bottle scanning contains 400 intervals covering 22,002 frames.

The stage labels and error intervals provide annotation-level information for reuse with different progress estimators. Our method converts these annotations into the advance–decline–recovery targets defined in § 3.2. The resulting continuous progress values are constructed supervision; the underlying annotations remain available for alternative target definitions and analyses of within-stage progress, local errors, and corrective behavior.

Experiments

Experiments

We evaluate REMORA on the ReSTEP dataset (Section 4) to address four questions:

Q1: Does our recurrent memory design improve progress estimation?

Q2: How does freezing or fine-tuning the VLM backbone affect performance?

Q3: How does causal peak selection affect prediction accuracy and temporal consistency?

Q4: How does our method compare with SARM, a state-of-the-art progress estimation method?

Implementation Details

Data preparation. All splits are disjoint at the episode level and shared across models. Parcel handling (D1D1) uses 1,757 training and 196 test episodes. For bottle scanning (D2D2), we split the expert and human-in-the-loop sources separately and hold out approximately 10% of each group for testing. This yields 1,081 training, and 120 test episodes. The test sets contain one mistake episode in D1D1 and 25 in D2D2. Stage-duration weights are estimated from mistake-free training episodes. Stages already satisfied at the start of an instruction contribute initial progress. Error frames remain in training and receive the descent targets defined in § 3.2.

Model and training. We implement REMORA on a π0\pi_0-style backbone with a categorical value head predicting 201 uniformly spaced progress bins in [0,1][0,1]. The model receives multi-view RGB observations, proprioception, and a language instruction through the state-only readout described in § 3. All variants within a task start from the same behavior-cloning checkpoint and are fine-tuned using the value-only objective with 30,000 steps. Training is performed on eight NVIDIA H200 GPUs. Recurrent training processes observations in temporal order and uses gradient accumulation to match the effective batch size of the window-based models. All variants share the same progress targets and normalization statistics. 4

Metrics

We evaluate progress accuracy, ordering, and temporal smoothness using four metrics. All metrics use the evaluated predictions without additional smoothing, interpolation, or clipping. Let EE denote the set of test episodes and Fe\mathcal{F}_e the evaluated frame indices within the annotated instruction spans of episode ee, including mistake frames. At frame tt, y^e,t\hat{y}_{e,t} and y~e,t\tilde{y}_{e,t} denote the predicted and reference progress, respectively.

Progress accuracy. Mean absolute error (MAE) AA measures the discrepancy between predicted and reference progress:

A=1∣E∣∑e∈E1∣Fe∣∑t∈Fe∣y^e,t−y~e,t∣.(5)A = \frac{1}{|E|} \sum_{e \in E} \frac{1}{|\mathcal{F}_e|} \sum_{t \in\mathcal{F}_e} \left|\hat{y}_{e,t} - \tilde{y}_{e,t}\right|. \tag*{(5)}

Lower AA indicates more accurate estimates on the normalized [0,1][0,1] scale.

Progress ordering. Value-Order Correlation (VOC), introduced by GVL [7], measures rank agreement between predicted progress and chronological frame order. This reference order assumes that progress increases with time, so VOC can penalize correctly predicted decreases during execution mistakes. We therefore use

Ablation Studies and Comparisons

Spearman’s rank correlation coefficient to compare predicted progress with our stage- and error-aware targets. Let r^e,t\hat{r}_{e,t} and r~e,t\tilde{r}_{e,t} be the ranks of y^e,t\hat{y}_{e,t} and y~e,t\tilde{y}_{e,t} within episode ee, assigning tied values their average rank. Both rank sequences have average me=(∣Fe∣+1)/2m_e = (|F_e| + 1)/2. We compute

ρS,e=∑t∈Fe(r^e,t−me)(r~e,t−me)∑t∈Fe(r^e,t−me)2∑t∈Fe(r~e,t−me)2,ρS=1∣E∣∑e∈EρS,e.(6)\begin{aligned} \rho_{S,e} &= \frac{\sum_{t \in F_e}(\hat{r}_{e,t} - m_e)(\tilde{r}_{e,t} - m_e)}{\sqrt{\sum_{t \in F_e}(\hat{r}_{e,t} - m_e)^2}\sqrt{\sum_{t \in F_e}(\tilde{r}_{e,t} - m_e)^2}}, \\ \rho_S &= \frac{1}{|\mathcal{E}|}\sum_{e \in\mathcal{E}}\rho_{S,e}. \tag*{(6)} \end{aligned}

Higher ρS\rho_S indicates closer agreement with the reference ordering. For strictly increasing targets, this reduces to rank-based VOC under the same tie convention.

Temporal smoothness. We measure local changes in progress increments using the second difference Δ2ye,t=ye,t+1−2ye,t+ye,t−1\Delta^2 y_{e,t} = y_{e,t+1} - 2y_{e,t} + y_{e,t-1}. Let TeT_e contain the center indices of consecutive three-frame windows in FeF_e that do not cross a temporal reset. Prediction curvature SS and curvature error JJ are

S=1∣E∣∑e∈E1∣Te∣∑t∈Te∣Δ2y^e,t∣,S = \frac{1}{|\mathcal{E}|}\sum_{e \in\mathcal{E}}\frac{1}{|T_e|}\sum_{t \in T_e}|\Delta^2\hat{y}_{e,t}|,
J=1∣E∣∑e∈E1∣Te∣∑t∈Te∣Δ2y^e,t−Δ2y~e,t∣.(7)J = \frac{1}{|\mathcal{E}|}\sum_{e \in\mathcal{E}}\frac{1}{|T_e|}\sum_{t \in T_e}|\Delta^2\hat{y}_{e,t} - \Delta^2\tilde{y}_{e,t}|. \tag*{(7)}

Lower SS indicates smoother predictions, while lower JJ indicates closer agreement with reference changes in slope.

Ablation Studies and Comparisons

We compare five ablations (A1–A5) and the SARM baseline to address Q1–Q4. The full REMORA model uses an eight-slot recurrent visual memory, a frozen VLM backbone, and causal peak selection. A1–A4 are evaluated on parcel handling (D1D1). A5 and SARM are evaluated on both D1D1 and medicine-bottle scanning and storage (D2D2).

VLM fine-tuning (Q2). A1 unfreezes the VLM backbone during value-model training. Comparing A1 with the frozen-backbone REMORA configuration evaluates whether adapting the pretrained vision–language representations improves progress estimation.

Memory design (Q1). A2–A4 replace recurrent memory with fixed observation windows (Figure 4). Each variant fuses four multi-view observation sets, sampled at offsets (−3-3, −2-2, −1-1, 0) for A2, (−30-30, −10-10, −3-3, 0) for A3, and (−90-90, −30-30, −10-10, 0) for A4. Offsets are measured in frames, with 0 denoting the current frame. These variants keep the number of sampled observations fixed while extending the temporal span, allowing us to compare recurrent memory with explicit histories of different lengths.

Temporal input configurations

Figure 4. Temporal input configurations. A2 - A4 use four explicit multi-view observations sampled at (t−3,t−2,t−1,t)(t - 3, t - 2, t - 1, t), (t−30,t−10,t−3,t)(t - 30, t - 10, t - 3, t) and (t−90,t−30,t−10,t)(t - 90, t - 30, t - 10, t), respectively. REMORA uses only the current observation and retains temporal context through eight gated recurrent memory tokens.

Peak selection (Q3). A5 disables causal peak selection and instead uses the median of the predicted categorical distribution at each frame. Both readouts use the same checkpoint and categorical predictions, with recurrent memory retained. This comparison isolates the contribution of peak selection to progress accuracy and temporal consistency.

Comparison with SARM (Q4). We compare REMORA with SARM [8], a state-of-the-art progress estimation method. SARM is trained using the same stage- and error-aware progress targets as REMORA. We report AA, ρS\rho_S, SS, and JJ for both the ablations and the baseline comparison.

Results and Discussion

Table 2 reports episode-averaged metrics: AA is shown in units of 10−210^{-2}, SS and JJ in units of 10−310^{-3}, and Spearman’s ρS\rho_S on its native [−1-1, 1] scale.

DatasetMethodA↓A \downarrowρS↑\rho_S \uparrowS↓S \downarrowJ↓J \downarrow
D1REMORA1.1000.9994.1814.188
D1A11.1500.9984.8214.826
D1A21.4900.9866.4786.483
D1A31.2030.9965.4565.460
D1A41.1370.9974.8714.876
D1A51.0870.9993.9153.920
D1SARM1.7670.98812.88212.883
D2REMORA3.3070.9595.6605.665
D2A56.0440.8706.9926.994
D2SARM3.4700.97816.58516.586

Table 2. Ablations and comparison with SARM on D1 and D2. Best values before rounding are bold.

Q1: Recurrent Memory versus Explicit Windows Extending the observation window from A2 to A4 improves all four metrics on D1 (Table 2). Since these variants retain the same number of observations, the trend suggests that earlier task states provide useful context for interpreting the current scene. Recurrent memory further improves on A4, reducing AA by 3.3% and JJ by 14.1%. The larger improvement in curvature error suggests that maintaining a continuously updated history is particularly useful for temporal consistency. Figure 6 illustrates this benefit: REMORA tracks the nonzero initial progress, while A2–A4 initially underestimate it.

Figure 6. Representative progress estimation on a D1 trajectory. The black curve denotes the target progress. The REMORA accurately tracks the nonzero initial target and subsequent progression, whereas A2 - A4 exhibit transient underestimation at the instruction onset. REMORA achieves the lowest episode-level MAE.

Q2: Frozen versus Fine-Tuned VLM Freezing the VLM improves all four metrics on D1 (Table 2) and reduces training time from 66.2 to 40.3 hours on eight H200 GPUs. This suggests that the pretrained representations already contain useful cues for progress estimation, which can be exploited by adapting the temporal memory and value readout. The additional flexibility of backbone fine-tuning does not translate into better test performance under the current training setup, making a frozen VLM both an effective and computationally efficient choice.

Q3: Causal Peak Selection The effect of peak selection differs across the two tasks (Table 2). On D1, where only one of 196 test episodes contains an annotated error, peak selection offers little benefit; the median readout yields slightly lower AA, SS, and JJ. In contrast, on the 120-episode D2D2 test, which contains 25 error episodes, peak selection reduces AA by 45.3% and raises ρS\rho_S from 0.870 to 0.959, while also lowering SS and JJ. This contrast is consistent with peak selection being particularly useful when execution errors make progress ambiguous. A per-frame median can jump as probability mass shifts between competing progress hypotheses; associating supported peaks across time favors continuity while still allowing progress to decrease. The joint improvement in accuracy, ordering, and curvature error on D2D2 supports the value of temporal continuity for tracking progress through execution setbacks.

Q4: Comparison with SARM REMORA achieves lower AA, SS, and JJ than SARM on both tasks, with curvature error reduced by over 65% in each case (Table 2). The reductions in both absolute and curvature error indicate that smoother predictions also capture progress magnitude and changes in its rate more accurately. Progress ordering also improves on D1D1, while SARM achieves higher ρS\rho_S on D2D2. Figure 5 provides a representative example: REMORA follows the target decrease during an error and the subsequent recovery more closely than SARM. These gains under shared progress supervision support the combination of temporal memory, distributional prediction, and causal peak selection for accurate and temporally consistent progress monitoring.

Figure 5. Representative progress estimation on a D2 trajectory. The black curve denotes the stage-and error-aware target, while the shaded region indicates an annotated error interval. REMORA better captures the target decrease and subsequent recovery than SARM, achieving a lower episode-level MAE.

Conclusion and Future Work

We introduced ReSTEP, a real-robot dataset that captures task progression, execution errors, and recovery through detailed stage and error annotations. By making these behaviors available for training and evaluation, ReSTEP provides an important benchmark for future work on progress monitoring with error awareness.

We also proposed REMORA, which combines temporal memory, distributional prediction, and causal peak selection to track task progress through visual ambiguity and execution setbacks. Experiments on both ReSTEP tasks demonstrate more accurate and temporally consistent progress estimates than the state-of-the-art SARM baseline.

Current methods rely on task-specific annotations and have been evaluated on only a single robotic platform across two task types. Future work will focus on expanding the scope of tasks and robotic platforms, reducing annotation requirements, and leveraging progress feedback to optimize control strategies and enable robot self-recovery for self-evolution. The current visual memory approach serves as a foundational prototype for video-based demonstration and in-context learning.

Acknowledgment

OpenAI ChatGPT was used to polish the writing and assist plotting figures.

References

  1. [1]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Chelsea Finn, Karol Hausman, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of the 7th Conference on Robot Learning, 2023.
  2. [2]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P. Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Proceedings of the 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 2679–2713, 2025.arxiv.org/abs/2406.09246
  3. [3]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. In Robotics: Science and Systems, 2025.DOI
  4. [4]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5: A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.arxiv.org/abs/2504.16054
  5. [5]Qi Zhang, Shaopeng Zhai, Shengzhe Zhang, Litao Liu, Tianyi Zhang, Fuxian Huang, and Ming Zhou. A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots. In Proceedings of the International Conference on Machine Learning (ICML), 2026.
  6. [6]Huajie Tan, Sixiang Chen, Yijie Xu, Zixiao Wang, Cheng Chi, Yuheng Ji, Yaoxu Lyu, Zhongxia Zhao, Xiansheng Chen, Peterson Co, Shaoxuan Xie, Guocai Yao, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. General Process Reward Modeling for Robotic Reinforcement Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22412–22422, 2026.
  7. [7]Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia. Vision Language Models are In-Context Value Learners. In International Conference on Learning Representations (ICLR), pages 33984–34009, 2025.arxiv.org/abs/2411.04549
  8. [8]Qianzhong Chen, Justin Yu, Mac Schwager, Pieter Abbeel, Yide Shentu, and Philipp Wu. SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2509.25358
  9. [9]Yiming Mao, Zixi Yu, Weixin Mao, Yinhao Li, Qirui Hu, Zihan Lan, Minzhao Zhu, and Hua Chen. ARM: Advantage Reward Modeling for Long-Horizon Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4468–4477, 2026.arxiv.org/abs/2604.03037
  10. [10]Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 627–635, 2011.arxiv.org/abs/1011.0686
  11. [11]Jonathan Spencer, Sanjiban Choudhury, Matt Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge, and Siddhartha Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Robotics: Science and Systems, 2020.DOI
  12. [12]Sumedh A. Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. RoboCLIP: One Demonstration is Enough to Learn Robot Policies. In Advances in Neural Information Processing Systems, volume 36, pages 55681–55693. Curran Associates, Inc., 2023.arxiv.org/abs/2310.07899
  13. [13]Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.12921
  14. [14]Jianshu Zhang, Chengxuan Qian, Haosen Sun, Haoran Lu, Dingcheng Wang, Letian Xue, and Han Liu. ProgressLM: Towards Progress Reasoning in Vision-Language Models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11243–11271. Association for Computational Linguistics, 2026.
  15. [15]Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. RoboReward: General-Purpose Vision-Language Reward Models for Robotics, 2026.arxiv.org/abs/2601.00675
  16. [16]Yanru Wu, Weiduo Yuan, Ang Qi, Vitor Guizilini, Jiageng Mao, and Yue Wang. Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models, 2026.arxiv.org/abs/2603.16065
  17. [17]Federico Becattini, Tiberio Urucchio, Lorenzo Seidenari, Lamberto Ballan, and Alberto Del Bimbo. Am i done? predicting action progress in videos. ACM Transactions on Multimedia Computing, Communications, and Applications, 16(4):1–24, 2020.arxiv.org/abs/1705.01781
  18. [18]Jiahui Zhang, Yusen Luo, Abrar Anwar, Sumedh Anand Sontakke, Joseph J. Lim, Jesse Thomason, Erdem Bıyık, and Jesse Zhang. ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations. In Proceedings of the Conference on Robot Learning (CoRL), volume 305 of Proceedings of Machine Learning Research, pages 460–488. PMLR, 2025.arxiv.org/abs/2505.10911
  19. [19]Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreaa Bobu, Abhishek Gupta, Stephen Tu, Erdem Bıyık, and Jesse Zhang. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. In Proceedings of Robotics: Science and Systems (RSS), volume 22, 2026.arxiv.org/abs/2603.02115
  20. [20]Andrew Y. Ng and Stuart Russell. Algorithms for Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 663–670. Morgan Kaufmann, 2000.
  21. [21]Pieter Abbeel and Andrew Y. Ng. Apprenticeship Learning via Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), page 1. ACM, 2004.DOI
  22. [22]Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum Entropy Inverse Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 1433–1438. AAAI Press, 2008.
  23. [23]Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. In Proceedings of the International Conference on Machine Learning (ICML), volume 48 of Proceedings of Machine Learning Research, pages 49–58. PMLR, 2016.arxiv.org/abs/1603.00448
  24. [24]Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. In Advances in Neural Information Processing Systems, volume 29, pages 4565–4573. Curran Associates, Inc., 2016.arxiv.org/abs/1606.03476
  25. [25]Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal Cycle-Consistency Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1801–1810, 2019.arxiv.org/abs/1904.07846
  26. [26]Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 537–546, 2022.arxiv.org/abs/2106.03911
  27. [27]Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language-Image Representations and Rewards for Robotic Control. In Proceedings of the International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 23301–23320. PMLR, 2023.arxiv.org/abs/2306.00958
  28. [28]Minttu Alakuijala, Reginald McLean, Isaac Woungang, Nariman Farsad, Samuel Kaski, Pekka Marttinen, and Kai Yuan. Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics. Transactions on Machine Learning Research, 2025.arxiv.org/abs/2405.19988
  29. [29]Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Bıyık, David Held, and Zackory Erickson. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. In Proceedings of the International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 51484–51501. PMLR, 2024.arxiv.org/abs/2402.03681
  30. [30]Youhe Feng, Hansen Shi, Haoyang Li, Xinlei Guo, Yang Wang, Chengyang Zhang, Jinkai Zhang, Xiaohan Zhang, Jie Tang, and Jing Zhang. ProcVLM: Learning procedure-grounded progress rewards for robotic manipulation. arXiv preprint arXiv:2605.08774, 2026.arxiv.org/abs/2605.08774
  31. [31]Seyed Kamyar Seyed Ghasemipour, Ayzaan Wahid, Jonathan Tompson, Pannag Sanketi, and Igor Mordatch. Self-Improving Embodied Foundation Models. In Advances in Neural Information Processing Systems, volume 38, pages 112651–112686. Curran Associates, Inc., 2025.arxiv.org/abs/2509.15155
  32. [32]Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Szymon Jakubczak, Rowan Jen, Tim Jones, Ben Katz, Liyiming Ke, Chandra Kuchi, Marinda Lamb, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Yao Lu, Vishnu Mano, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Charvi Sharma, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, Will Stoekcle, Alex Swerdlow, James Tanner, Marcel Torne, Quan Vuong, Anna Walling, Haohuan Wang, Blake Williams, Sukwon Yoo, Lili Yu, Ury Zhilinsky, and Zhiyuan Zhou. π^0.6: a VLA That Learns From Experience, 2025.
  33. [33]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2303.15343

Paper details

vlnc:W_qjtxwuux
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT

View statement
Looking for related papers…
Contents