In this paper
AbstractIntroductionRelated WorksBackgroundReinforcement LearningFlow MatchingGUARD: Guidance Using Anomaly-infoRmed Dense ValuesAnomaly Signals-Informed Dense RewardsQ-Guided Action DenoisingOffline-to-Online Q-Function AdaptationExperimentsAdroit Simulation BenchmarkReal-World ExperimentsConclusion and Future WorkReferencesWorksComputer Science
GUARD: Anomaly-Informed Q-Steering for Generative Robot Policies in Dexterous Manipulation
Abstract
Generative robot policies based on diffusion and flow matching can model complex, multimodal behaviors, but remain susceptible to covariate shift during closed-loop execution. Existing approaches mitigate such failures through additional demonstrations, policy fine-tuning, or test-time guidance, but often require human intervention, engineered rewards, or other task-specific signals. We introduce GUARD, an anomaly-informed Q-steering framework that leverages the policy's training distribution to construct a dense, failure-anticipatory steering signal. GUARD pairs sparse task rewards with dense anomaly scores learned from policy training data and propagates both through a learned Q-function. Its gradients then steer the denoising process toward actions that improve task return while preventing trajectories from entering poorly supported regions, without directly modifying the pretrained policy. Across four Adroit dexterous manipulation tasks, GUARD achieves average success rates of and under nominal and perturbed dynamics, outperforming the strongest baseline by and percentage points while maintaining the lowest anomaly score. GUARD further demonstrates direct sim-to-real transfer on in-hand screwdriver rotation and reorientation tasks, achieving and percentage-point gains in success rate. These results demonstrate that anomaly signals can serve not only for failure monitoring but also to complement sparse task rewards as effective supervision for robust policy steering.

Figure 1. We propose GUARD, a method for steering pretrained generative policies away from failures. GUARD learns a Q-function from sparse task rewards and dense anomaly signals, thus anticipating the risk of entering anomalous regions and the likelihood of task success. The Q-function gradients provide rich guidance to the policy’s denoising process toward successful, well-supported actions.
Introduction
Generative policies, such as diffusion and flow matching, have emerged as the state-of-the-art for imitation learning in robotics, given their ability to model complex, multimodal action distributions [1, 2, 3]. However, as with other behavior-cloned policies, their performance can degrade under covariate shift: execution errors and environmental variations may drive the system into states poorly represented in the demonstration data, where the learned policy becomes unreliable. This problem is especially pronounced in dexterous manipulation, where complex contact-rich dynamics and long-horizon interactions can amplify small deviations.
Current methods for addressing covariate shift rely primarily on providing additional signal to adjust policy behavior. Human-in-the-loop methods utilize a supervisor to intervene and provide corrective demonstrations that address deviations as they occur [4, 5, 6, 7], but require significant human effort and are difficult to collect for dexterous robot hands. Methods for post-training generative policies with reinforcement learning can introduce more robust behaviors via trial-and-error [8, 9, 10, 11, 12, 13] but rely heavily on the quality of the reward signal. Dense rewards provide rich, informative signals to guide policy improvement, but generally require task-specific shaping, additional failure data, or manual failure annotations [14, 15, 16]. Sparse task-success rewards are often more scalable, but provide delayed signaling, which can increase the data burden and make adjustments challenging to learn [17, 18].
In this work, we explore how we can construct guidance for a generative policy without relying on human input, brittle assumptions, or engineered signals. Our key insight is to explicitly “nudge” the policy back toward its support region rather than relying on laborious task signaling. This prevents our target system from drifting into regions of limited support where our policy is likely to fail. We build on recent work that derives out-of-distribution (OOD) and anomaly scores for robotic behaviors [19, 20, 21] by constructing a model of the policy’s training demonstrations. While previous methods demonstrated the efficacy of anomaly detection for failure monitoring, behavior adjustment is primarily limited to naive retries [22] or state-based interpolation [23, 24], neither of which can be applied to contact-rich dexterous manipulation.
To translate anomaly detection into effective online adjustment, we propose steering the action-generation process at inference time. Inspired by gradient guidance in generative modeling [25], test-time steering preserves the pretrained policy as a behavioral prior while modifying its samples toward desirable actions. To effectively steer, we require an anticipatory signal of covariate shift, i.e., grounding future actions to their likelihood of leading to anomalous states. We propose constructing a Q-function trained on offline policy rollouts and online environment interactions to capture the discounted likelihood of future anomalies. The result is a function we can directly utilize to steer a policy to remain near its support. Further, we compose our proposed signal with a sparse task reward, which specifies what condition should ultimately be achieved, while the anomaly signal indicates where the pretrained policy is likely to remain competent. The resulting value function captures both the task return and the risk of leaving high-support regions, and its action gradient is used to steer each denoising process toward actions that improve the predicted task outcome while remaining close to the trusted region.
Contributions. We introduce GUARD: Guidance Using Anomaly-infoRmed Dense Values, a method that (1) combines sparse task rewards with dense anomaly signals to learn informative Q-functions for steering; and (2) utilizes an offline-to-online Q-function learning mechanism that uses the resulting value gradients to steer frozen flow matching policies toward successful, well-supported actions. We present a comprehensive evaluation on contact-rich dexterous manipulation tasks in the Adroit simulation benchmark and via direct sim-to-real transfer across three real-world tasks.
Related Works
Failure Prediction and Anomaly Detection. Several recent works study failure prediction through anomaly detection, OOD detection, and uncertainty estimation. FAIL-Detect learns a flow model on the observation distribution from policy training data without requiring failure examples [19], while FIPER combines observation OOD and action-distribution uncertainty to predict failures of generative policies [20]. Sentinel monitors temporal action consistency and task progress [26], and other anomaly-based approaches detect deviations from nominal manipulation behavior to trigger failsafe or recovery mechanisms [21, 27, 28]. Policy uncertainty has also been used to improve residual-RL adaptation [29]. In contrast, we use policy-input anomaly scores as dense reward supervision for learning Q-functions, allowing information about distributional support to propagate through the Bellman return and shape the same value gradients that steer actions toward task success.
Improving Generative Robot Policies. The prevalence of generative policy methods has stoked much interest in methods that address failure modes and improve task performance. Applying post-training Reinforcement Learning (RL) is an intuitive choice, and has been demonstrated with success on generative policies [8, 9], however, the generative denoising processes make training unstable and inefficient. Alternative approaches include applying RL in the latent space of the policy [11, 12], adding a residual policy to the generative policy outputs [10, 12, 13], or ranking multiple outputs from the policy with an external value function [30]. Several works build on gradient guidance for generative modeling [25] to inject learned Q-function gradients into the denoising process [31, 32, 33, 34]. We build on this Q-guided sampling machinery but address an orthogonal bottleneck: what signal should the Q-function learn from when task supervision is sparse?
Several other methods have explored moving beyond RL signals for steering. UA-Flow steers flow generation directly using predicted policy uncertainty [35]. DynaGuide uses an action-conditioned dynamics model to steer predicted outcomes toward desired behaviors [36]. Latent Policy Barrier similarly predicts future latent states and guides them toward the expert demonstration distribution [37]. In contrast, our work utilizes a Q-function approach to transform an anomaly signal into a failure anticipatory signal, thereby avoiding dynamics learning. We provide a direct comparison to these methods in our experiments.
Background
Reinforcement Learning
We consider a Markov decision process (MDP) [38] , with state space , action space , transition dynamics , reward , and discount factor . A stationary policy maximizes the expected discounted return . Its action-value function is . Given transitions , a Q-function can be learned through temporal-difference learning by minimizing , where and denotes a target Q-function. Since depends explicitly on the action, its gradient provides a local direction toward actions with a higher predicted return.
Flow Matching
Flow matching [2, 39] policies are parameterized by a state-conditioned, time-dependent velocity field that transports samples from a Gaussian noise distribution to the policy distribution via an ODE: , . Given dataset state-action pairs , we define linear interpolants between noise and data actions and train the velocity field via flow matching: . At inference time, actions are generated by integrating the ODE from to , yielding a sample .
GUARD: Guidance Using Anomaly-infoRmed Dense Values
Our method consists of three components: (i) anomaly signals-informed dense reward construction, (ii) Q-guided action denoising, and (iii) offline-to-online Q-function adaptation. The overall procedure is summarized in Alg. 1.
Anomaly Signals-Informed Dense Rewards
Let denote the demonstration data used to train the frozen flow policy . We fit an anomaly detector on the policy inputs in , where larger values indicate observations that are less supported by the policy’s training distribution. Since is learned solely from the policy training data, it requires neither failure examples nor task-specific failure annotations. Our method naturally admits many available anomaly metrics. In our experiments, we primarily use the logpZO score [19] by fitting a separate flow model to the observations in that transports each observation to a Gaussian latent , and take the squared latent norm as its continuous anomaly score (large when an observation maps far from in-distribution regions).
We assign each state-action pair the anomaly score of , i.e., . Using an offline buffer collected with the frozen flow policy, we compute and freeze reward-normalization statistics . The dense anomaly reward is then defined as
Thus, transitions toward states better supported by the policy’s training distribution receive a larger reward, while transitions toward anomalous states are penalized.
The available task supervision is a sparse binary signal, . To place both rewards on comparable scales, we independently standardize the task reward using the fixed statistics in ,
and define the reward used for Q-learning as their equal-weight sum,
Independent standardization places both signals on comparable scales without introducing a tunable trade-off coefficient.
The two rewards provide complementary supervision: the sparse task reward specifies what the robot should ultimately accomplish, while the dense anomaly reward indicates where the pretrained policy is well supported. Crucially, we do not directly steer actions to minimize instantaneous anomaly. Instead, Eq. (3) enters the Bellman return, allowing the Q-function to propagate both task success and distributional support over future trajectories.
Q-Guided Action Denoising
We learn an ensemble of independently initialized Q-functions, . To reduce exploitation of Q-function extrapolation errors during gradient-based guidance, we follow prior Q-guided methods [34] and construct the pessimistic ensemble estimate
where controls the degree of pessimism. The ensemble disagreement in Eq. (4) is distinct from the anomaly score : measures support of the policy input under the training distribution and is used for reward construction, whereas Q-ensemble disagreement provides conservative value estimates for guidance.
Intermediate flow variables are noisy and generally lie outside the clean-action distribution on which the Q-functions are trained. Following prior works [33, 40], at each denoising step , we estimate the corresponding clean action with a single step over the remaining flow horizon,
query the ensemble at and compute . Following QGF [33], we approximate , avoiding differentiation through the flow velocity. With denoising step size , the guided update is
We denote the action distribution induced by iterating Eq. (6) as . The frozen flow policy remains the behavioral prior, while the Q-gradient biases denoising toward actions with high future anomaly signal-augmented task return. Here is the guidance weight, and we discuss details about the selection of in Sec. V.
Offline-to-Online Q-Function Adaptation
The Q-functions used for guidance should evaluate the policy that is ultimately executed, i.e., the steered policy. We therefore first estimate values under the base policy with offline initialization and subsequently adapt them to the evolving steered policy using online experience.
a) Offline initialization: We collect an offline buffer by rolling out the frozen policy with action noise , recording each transition together with its subsequent behavior action . Each independently initialized ensemble head uses its own exponentially averaged target Q-function . We perform behavior-policy fitted Q evaluation (FQE) using
where is the terminal indicator. Each is regressed toward its corresponding target, while its target parameters are updated as . Using per-head targets during offline initialization preserves ensemble diversity while estimating the anomaly-augmented return in regions covered by the behavior policy.
b) Online adaptation: Once Q-guidance is applied, the executed policy differs from the behavior policy used for offline initialization. A Q-function evaluated only on the original behavior distribution can therefore become mismatched to the policy it is used to guide. We address this by alternating guided interaction with FQE updates under the resulting policy.
At each online iteration, we collect fresh trajectories using and append them to the unified replay buffer . All online transitions are assigned rewards using Eq. (3) with the fixed normalization statistics .
For a sampled transition , we regenerate a fresh bootstrap action using the guided policy induced by the EMA target ensemble, , and construct the pessimistic target
Each ensemble head is regressed toward this shared target, , while its target network is updated by Polyak averaging. Using the slowly evolving target ensemble to generate both and its bootstrap value decouples action generation from instantaneous Q-function updates and stabilizes the guidance-value feedback loop. Repeated interaction and evaluation progressively adapt the Q-functions toward the steered policy, so that estimates the long-horizon return obtained after taking and subsequently following that guided policy. The resulting policy uses the ensemble of online-adapted -functions to guide denoising towards high-return, high-support actions. Refer Alg. 1 for the full training procedure.
Algorithm 1: GUARD Training.
Require: Frozen flow policy ; anomaly detector ; ensemble size ; pessimism ; guidance weight , action noise ; discount ; Polyak rate
Collect by rolling out with action noise , storing
Compute reward statistics from and freeze them
using (3);
Initialize independent and ,
-guided flow policy defined by (6)
Phase-I: Offline Q-function initialization
for do
Sample
for do
Update toward ;
Phase-II: Online Q-function adaptation
for do
Collect fresh transitions using with action noise and append them to
for do
Sample
;
for do
Update toward ;
return
Experiments
We evaluate GUARD on the D4RL Adroit dexterous manipulation benchmark [41] in simulation and further demonstrate its sim-to-real transfer capabilities across three tasks using the Allegro V5 Plus multi-fingered robot hand. Through these experiments, we aim to answer the following research questions (RQs):
RQ1: Can GUARD reliably steer the policy toward successful task completion?
RQ2: Does GUARD keep the system near high-support, in-distribution regions?
RQ3: Is GUARD more robust under distribution shift, including perturbed dynamics and sim-to-real transfer?
Adroit Simulation Benchmark
We run experiments on all four Adroit manipulation tasks: Relocate, Pen, Hammer, and Door [41]. We train base flow-matching policies over 200 episodes, sampled equally from the expert and cloned subsets. The policy uses a four-layer MLP with 512 hidden units, GELU activations, layer normalization, and Euler denoising steps. GUARD uses an ensemble of five -functions with pessimism , trained for 40k offline FQE updates on 800 episodes with batch size 256, learning rate , discount , EMA target rate , followed by 150 iterations of online adaptation collecting 24 episodes each iteration. Each episode includes 200 steps, and the sparse task reward used for Adroit tasks is everywhere and 0 on successful states, aligning with the commonly used choice in offline RL. We calibrate once after offline initialization by matching the average gradient scale to the average flow-velocity scale, , where the expectations are estimated using the offline-initialized ensemble and then hold fixed throughout the online adaptation and deployment.
a) Baselines: We compare GUARD against: (i) BC, the unsteered flow-matching policy; (ii) LPB [37] and (iii) DynaGuide [36], which use a learned dynamics model to steer predicted future states toward in-distribution regions and desired task outcomes respectively; (iv) UA-Flow [35], which guides flow sampling toward actions with lower predicted uncertainty; (v) Sparse-Q, -guided denoising using only the sparse task success reward; and (vi) U-SA, which directly guides denoising using action gradient of joint state-action anomaly scores . Every baseline is trained using its prescribed methodology, with an equal data budget (both offline and online) to GUARD, for fair comparison. UA-Flow and U-SA are trained only with , as they rely only on the training demonstrations.
b) Results: In Table I, we evaluate 100 episodes per task in each of the nominal and perturbed dynamics settings. In the latter setting, we add Gaussian noise to the actions to investigate GUARD under distribution shift.
| Method | Metric | Relocate | Relocate | Pen | Pen | Hammer | Hammer | Door | Door | Average | Average |
| BC | SR | ||||||||||
| AS | |||||||||||
| Sparse-Q | SR | ||||||||||
| AS | |||||||||||
| LPB | SR | ||||||||||
| AS | |||||||||||
| DynaGuide | SR | ||||||||||
| AS | |||||||||||
| UA-Flow | SR | ||||||||||
| AS | |||||||||||
| U-SA | SR | ||||||||||
| AS | |||||||||||
| GUARD | SR | ||||||||||
| AS |
Table I. Adroit simulation results across nominal and perturbed dynamics. Here, denotes nominal, while denotes task-specific dynamics perturbations at the indicated noise level. Results are reported over three seeds using 100 test rollouts. SR denotes success rate (), and AS denotes the RND-based state-anomaly score on successful episodes ().
In the nominal setting, GUARD achieves the strongest overall task performance, reaching an average success rate (SR) of 0.96. This corresponds to an average gain of 36.0 percentage points (pp) over the BC policy and a gain of 21.0 pp over the next-best baseline, DynaGuide. Figure 2 shows how GUARD successfully steers an originally failed trajectory under the BC policy for the Pen task starting from the same initial state. These results establish for RQ1 that GUARD can effectively steer generative policies toward task success and outperform existing steering strategies.

Figure 2. A qualitative visualization of steering with GUARD in the Adroit Pen task. (Left) unsteered BC policy forms a loose grasp of the pen and fails to properly reorient it, as indicated by low Q-values and high state-anomaly scores . (Right) At the moment some fingers lose contact, steering provides a larger action correction folding the fingers back in to maintain a firm grasp and reorient the pen to the target orientation.
GUARD also consistently keeps the system much closer to in-distribution regions, achieving the lowest state-anomaly scores (AS) on successful episodes for each method across all tasks in the nominal setting. Here, the AS is evaluated using random network distillation (RND) [42, 43], a neural metric for fair comparison across all methods, using a fixed random target network and a predictor trained to regress on inputs, calculating the anomaly score as . These results answer RQ2: GUARD realizes substantially stronger support under the policy training distribution.
Finally, we compute both SR and AS in the perturbed-dynamics setting. GUARD achieves an average SR of 0.80 in the shifted environment, a 39.0 and 20.0 pp gain over the BC policy and the next-best baseline, again DynaGuide, and the lowest AS score on average across all tasks. Figure 3(b) shows our method maintains the highest performance with increasing perturbation levels as shown on the Door task. Thus, answering RQ3, GUARD demonstrates more robust performance across both tests under distribution shift.

Figure 3. (a) Failure-detection quality on Door: episode-level AUROC (max-aggregated per-step score) using each method’s steering signal. (b) Robustness on Door to test-time dynamics perturbations of magnitude (action noise). (c) Offline data efficiency on Hammer: success rate vs. offline dataset size; dashed = offline-only (no online adaptation), solid = offline + online training. (d) Online data efficiency on Hammer: success rate vs. online interaction (env steps) at the 800-episode offline budget. Overall, GUARD leads across failure detection, dynamics robustness, and data efficiency.
Our results indicate the efficacy of GUARD for driving task improvement. The performance gains over U-SA (nominal: +24.0 pp; perturbed: +28.0 pp) indicate the importance of grounding the signal in actions with the Q-function. Gains over the Sparse-Q (nominal: +30.0 pp; perturbed: +30.0 pp) show the value of dense anomaly-signaling. While LPB and DynaGuide utilize a failure-anticipation mechanism by rolling out the predicted dynamics, we find that GUARD outperforms both, suggesting that the dynamics of the contact-rich tasks under test may be challenging to capture accurately. Finally, UA-Flow relies only on the policy uncertainty, which provides a weaker steering signal than GUARD.
c) Failure Signal Alignment: We compare each method’s failure-detection performance using the AUROC metric aggregated over the episode. Figure 3(a) shows that GUARD achieves the highest AUROC, demonstrating its ability to reliably identify failures, complementing its steering performance.
d) Training Data Efficiency: GUARD is highly data-efficient in both the offline and online phases. The Q-function reaches near-perfect steered success with only a modest buffer of offline data paired with online training and stays saturated as the buffer grows, whereas the strongest baselines (DynaGuide, LPB) plateau well below it (Figure 3(c)). The gap between the offline-only (dashed) and offline + online (solid) curves shows that offline training alone is insufficient and that online adaptation supplies most of the leverage for steering. Online, GUARD quickly improves its signaling, climbing to within the first environment steps and saturating near 1.0, while the baselines improve more slowly and to a lower ceiling (Figure 3(d)). Thus, GUARD achieves the highest performance with little offline data and some online interactions, a favorable trade-off for a scalable real-world deployment.
e) Ablation on Anomaly Signal: We ablate the choice of state-anomaly score by swapping in alternative OOD signals in GUARD, such as input reconstruction error (Recon-AE) [21], RND [43], and learned dynamics-based metrics such as prediction error (Dyn-Err) and ensemble disagreement (Dyn-Ens) [44], as shown in Table II. The state-density logpZO metric we adopt is both the strongest and most stable, while Recon-AE, RND, and Dyn-Ens also remain competitive. In contrast, Dyn-Err collapses, indicating that prediction error conflates model inaccuracy with genuine anomaly and is a poor proxy for how far a state has drifted OOD. This ablation shows that steering benefits from a state-distributional notion of anomaly, and that logpZO is a robust default choice.
| logpZO | RND | Recon-AE | Dyn-Err | Dyn-Ens |
Table II. Anomaly-Signal Ablation on Door (SR ).
Real-World Experiments
We evaluate on three dexterous tasks with a 16-DoF Allegro V5 Plus hand: (1) screwdriver in-hand rotation, requiring continuous rotations throughout the episode without dropping, and screwdriver in-hand reorientation in (2) palm-down and (3) palm-up configurations, requiring the screwdriver to be reoriented vertically below and above the palm, respectively, from a horizontal initial pose, just using its fingers in the fixed-wrist hand, and holding it there for sec. The BC policies are cloned from 64, 256, and 4096 demonstrations, respectively, for each task, each 40 sec long. These demonstrations are generated for each task using an RL expert policy trained with PPO [45] using self-defined dense rewards in the mjlβab simulation, with domain randomization of dynamics and observations. The BC policy is a conditional 1D U-Net (channels 256/512/1024, kernel 5) taking 41 dim observations containing joint positions, previously commanded joint positions, and object pose in the palm frame for all tasks. It outputs 16 dim joint-position deltas of chunk length 4 with an execution horizon of 2. It is trained for 100 epochs with batch size 256, learning rate , cosine schedule, and EMA decay 0.9999, with Euler steps at inference. GUARD uses an ensemble of five Q-functions with pessimism , trained for 200k offline FQE updates on 2000 policy rollouts with batch size 2048, learning rate , discount , EMA target rate , followed by iterations of online adaptation collecting 128 episodes in each iteration. The sparse task reward is 0 everywhere and 1 on successful states. We keep dynamics and observation domain randomization enabled to train GUARD under the same noise distribution as seen by the BC policy. Note that GUARD is trained entirely in simulation, for both the offline and online stages, and then evaluated directly in the real world to assess its sim-to-real capabilities.
| Task | Success (%) ↑ | Drop (%) ↓ | RND ↓ | |||
| BC | GUARD | BC | GUARD | BC | GUARD | |
| Rotation | 20 | 60 | 80 | 25 | 0.11 ±0.03 | 0.07 ±0.02 |
| Palm-Down | 70 | 95 | 30 | 5 | 0.05 ±0.11 | 0.03 ±0.07 |
| Palm-Up | 30 | 10 | 5 | 20 | 0.01 ±0.00 | 0.02 ±0.01 |
Table 3. Real-robot Allegro V5 Plus screwdriver manipulation results for BC and GUARD, with trials per method. The best result for each metric is shown in bold.
a) Results: On the continuous rotation and palm-down reorientation tasks, GUARD substantially improves real-robot performance over the unsteered BC policy (Table III). For rotation, the success rate improves by 40.0 pp, and the drop rate reduces by 55.0 pp. Figure 4 shows how GUARD successfully steers the policy for the rotation task. On palm-down reorientation, GUARD improves the success rate by 25.0 pp, converting those drops to successful reorientations.

Figure 4. A qualitative visualization of steering with GUARD in the screwdriver in-hand-rotation task. (Left) unsteered BC policy drops the screwdriver, indicated by a large, sudden drop in the Q-value and a jump in state-anomaly score . (Right) Whenever the fingers lose hold of the screwdriver, steering provides a larger action correction , successfully correcting the grasp twice in the episode, and prevents the screwdriver from falling.
The terminal orientation error from the palm normal axis reduces from and the longest stable object hold duration increases from s when evaluated on successful episodes for both methods, indicating tighter and more durable reorientation. GUARD also maintains a low RND score, comparable to BC, on successful episodes, indicating that steered rollouts remain in-distribution and do not encounter OOD states. On the palm-up reorientation task, GUARD hurts performance relative to BC. We attribute this to inaccurate contact modeling in simulation compared to the real world. We observed a pronounced screwdriver rotation mid-execution on the real robot that occurred rarely under the same policy in simulation, resulting in a large sim-to-real gap. Because these transitions were largely absent in the simulation, we hypothesize that the Q-function is unable to anticipate this behavior and thus fails to provide the required correction to recover to high-support regions. We intend to resolve such gaps in future work by training with wider domain randomization and on mixed buffers that incorporate real-world interactions, a commonly used technique in steering methods for real-world deployment (LPB, DynaGuide). This can further close the sim-to-real gap, allowing the Q-function to be well aware of the failure modes actually encountered on hardware.
Conclusion and Future Work
We introduced GUARD, an anomaly-informed Q-guidance framework for robustly steering frozen generative robot policies. Rather than relying on task-specific dense rewards or manual failure annotations, GUARD leverages anomaly signals learned from the base policy’s training distribution and combines them with sparse task rewards to learn Q-functions. By propagating this information through the Bellman return, GUARD enables the value gradients to account for both task completion and the likelihood of entering poorly supported states, rather than greedily minimizing instantaneous uncertainty.
Across Adroit dexterous manipulation tasks, GUARD significantly improves task success while maintaining lower anomaly scores than existing steering methods, including under perturbed dynamics. We further demonstrate direct sim-to-real transfer without any real-robot training, achieving performance gains on two of three evaluated tasks. Together, these results demonstrate that signals typically used for runtime failure monitoring can also provide scalable supervision to actively improve generative robot policies through test-time Q-steering methods.
Our results indicate that our anomaly-informed Q-steering can transfer surprisingly well across shifts in dynamics and even to the real world. We hypothesize that task conditions with overlapping successful state distributions and comparable dynamics can reliably transfer the anomaly metrics and subsequent Q-functions, as demonstrated by successful steering across perturbed simulation dynamics and by rotation and palm-down reorientation results in the real world. However, if the environment shift significantly alters the successful state distribution, turning nominal states anomalous and leaving the Q-function unaware of real failure modes, GUARD may no longer provide the required correction. This is exemplified by the palm-up reorientation results.
Future work will explore resolving these issues through mixed simulation and real-world training to further improve real-world transfer and explore extensions to policies with multimodal inputs and broader manipulation tasks.
References
- [1]C. Chi et al., “Diffusion policy: Visuomotor policy learning via action diffusion,” International Journal of Robotics Research, 2025.
- [2]Y. Lipman et al., “Flow matching for generative modeling,” in International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
- [3]K. Black et al., “π₀: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
- [4]S. Ross et al., “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
- [5]H. Liu et al., “Model-based runtime monitoring with interactive imitation learning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024.arxiv.org/abs/2310.17552
- [6]R. Hoque et al., “Intervengen: Interventional data generation for robust and data-efficient robot imitation learning,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2024.arxiv.org/abs/2405.01472
- [7]R. Hoque et al., “Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning,” in Proceedings of the 5th Conference on Robot Learning, A. Faust et al., Eds., PMLR, 2022.arxiv.org/abs/2109.08273
- [8]A. Ren et al., “Diffusion policy policy optimization,” in International Conference on Learning Representations, 2025.arxiv.org/abs/2409.00588
- [9]T. Zhang et al., “Reinflow: Fine-tuning flow matching policy with online reinforcement learning,” in Advances in Neural Information Processing Systems, D. Belgrave et al., Eds., Curran Associates, Inc., 2025.arxiv.org/abs/2505.22094
- [10]L. L. Ankile et al., “From imitation to refinement–residual rl for precise visual assembly,” in CoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant Data, 2024.arxiv.org/abs/2407.16677
- [11]A. Wagenmaker et al., “Steering your diffusion policy with latent space reinforcement learning,” in Conference on Robot Learning, PMLR, 2025.arxiv.org/abs/2506.15799
- [12]E. Su et al., “Rfs: Reinforcement learning with residual flow steering for dexterous manipulation,” in The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2602.01789
- [13]Z. Sun et al., “From prior to pro: Efficient skill mastery via distribution contractive rl finetuning,” arXiv preprint arXiv:2603.10263, 2026.arxiv.org/abs/2603.10263
- [14]Q. Chen et al., “Sarm2: Multi-task stage aware reward modeling for self improving robotic manipulation,” arXiv preprint arXiv:2606.10305, 2026.arxiv.org/abs/2606.10305
- [15]Y. Fang et al., DenseReward: Dense reward learning via failure synthesis for robotic manipulation, 2026.arxiv.org/abs/2607.13033
- [16]T. W. Ayalew et al., “Progressor: A perceptually guided reward estimator with self-supervised online refinement,” in 2025 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, 2025.arxiv.org/abs/2411.17764
- [17]M. Andrychowicz et al., “Hindsight experience replay,” Advances in neural information processing systems, 2017.arxiv.org/abs/1707.01495
- [18]A. Nair et al., “Overcoming exploration in reinforcement learning with demonstrations,” in 2018 IEEE international conference on robotics and automation (ICRA), IEEE, 2018.arxiv.org/abs/1709.10089
- [19]C. Xu et al., “Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies,” in Proceedings of Robotics: Science and Systems (RSS), 2025.
- [20]R. Römer et al., “Failure prediction at runtime for generative robot policies,” Advances in Neural Information Processing Systems (NeurIPS), 2025.
- [21]N. Grambow et al., “Anomaly detection for generic failure monitoring in robotic assembly, screwing and manipulation,” IEEE Robotics and Automation Letters, 2026.DOI
- [22]S. Zhou et al., “Adaptive online replanning with diffusion models,” in Advances in Neural Information Processing Systems, A. Oh et al., Eds., Curran Associates, Inc., 2023.arxiv.org/abs/2310.09629
- [23]A. Reichlin et al., “Back to the manifold: Recovering from out-of-distribution states,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2022.arxiv.org/abs/2207.08673
- [24]G. J. Gao et al., “Out-of-distribution recovery with object-centric keypoint inverse policy for visuomotor imitation learning,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025.arxiv.org/abs/2411.03294
- [25]P. Dharival et al., “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems, 2021.
- [26]C. Agia et al., “Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress,” in Proceedings of The 8th Conference on Robot Learning, P. Agrawal et al., Eds., PMLR, 2025.arxiv.org/abs/2410.04640
- [27]B. Santhanam et al., “Reliable robotic task execution in the face of anomalies,” IEEE Robotics and Automation Letters, 2025.DOI
- [28]Q. Rolland et al., “Failure identification in imitation learning via statistical and semantic filtering,” in IEEE International Conference on Robotics and Automation (ICRA), 2026.arxiv.org/abs/2604.13788
- [29]L. Dodeja et al., “Accelerating residual reinforcement learning with uncertainty estimation,” IEEE Robotics and Automation Letters, 2026.
- [30]M. Nakamoto et al., “Steering your generalists: Improving robotic foundation models via value guidance,” in Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2410.13816
- [31]Y. Jang et al., “Q-guided flow q-learning,” in CoRL 2025 Workshop RememberRL.
- [32]H. Ye, “Steering diffusion policies with value-guided denoising,” in NeurIPS 2025 Workshop on Embodied World Models for Decision Making, 2025.
- [33]Z. Zhou et al., “Test-time gradient guidance of flow policies in reinforcement learning,” arXiv preprint arXiv:2606.11087, 2026.arxiv.org/abs/2606.11087
- [34]Y. Ruan et al., “Qpilots: Efficient test-time q-steering for flow policies,” arXiv preprint arXiv:2606.14801, 2026.arxiv.org/abs/2606.14801
- [35]J. Han et al., “Flow matching with uncertainty quantification and guidance,” arXiv preprint arXiv:2602.10326, 2026.arxiv.org/abs/2602.10326
- [36]M. Du et al., “Dynaguide: Steering diffusion policies with active dynamic guidance,” in Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2506.13922
- [37]Z. Sun et al., “Latent policy barrier: Learning robust visuomotor policies by staying in-distribution,” in Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2508.05941
- [38]R. S. Sutton et al., Reinforcement learning: An introduction. MIT press Cambridge, 1998.DOI
- [39]X. Liu et al., “Flow straight and fast: Learning to generate and transfer data with rectified flow,” in International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.03003
- [40]B. Kang et al., “Efficient diffusion policies for offline reinforcement learning,” Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.20081
- [41]A. Rajeswaran et al., “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” arXiv preprint arXiv:1709.10087, 2017.arxiv.org/abs/1709.10087
- [42]Y. Burda et al., “Exploration by random network distillation,” arXiv preprint arXiv:1810.12894, 2018.arxiv.org/abs/1810.12894
- [43]N. He et al., “Rediffuser: Reliable decision-making using a diffuser with confidence estimation,” in Forty-first International Conference on Machine Learning, 2024.
- [44]T. Haider et al., “Out-of-distribution detection for reinforcement learning agents with probabilistic dynamics models,” in Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, 2023.
- [45]J. Schulman et al., Proximal policy optimization algorithms, 2017.arxiv.org/abs/1707.06347