betaFeedback

WorksComputer Science

GUARD: Anomaly-Informed Q-Steering for Generative Robot Policies in Dexterous Manipulation

Abstract

Generative robot policies based on diffusion and flow matching can model complex, multimodal behaviors, but remain susceptible to covariate shift during closed-loop execution. Existing approaches mitigate such failures through additional demonstrations, policy fine-tuning, or test-time guidance, but often require human intervention, engineered rewards, or other task-specific signals. We introduce GUARD, an anomaly-informed Q-steering framework that leverages the policy's training distribution to construct a dense, failure-anticipatory steering signal. GUARD pairs sparse task rewards with dense anomaly scores learned from policy training data and propagates both through a learned Q-function. Its gradients then steer the denoising process toward actions that improve task return while preventing trajectories from entering poorly supported regions, without directly modifying the pretrained policy. Across four Adroit dexterous manipulation tasks, GUARD achieves average success rates of 96.0%96.0\% and 80.0%80.0\% under nominal and perturbed dynamics, outperforming the strongest baseline by 21.021.0 and 20.020.0 percentage points while maintaining the lowest anomaly score. GUARD further demonstrates direct sim-to-real transfer on in-hand screwdriver rotation and reorientation tasks, achieving 4040 and 2525 percentage-point gains in success rate. These results demonstrate that anomaly signals can serve not only for failure monitoring but also to complement sparse task rewards as effective supervision for robust policy steering.

GUARD Q-steered flow policy illustration

Figure 1. We propose GUARD, a method for steering pretrained generative policies away from failures. GUARD learns a Q-function from sparse task rewards and dense anomaly signals, thus anticipating the risk of entering anomalous regions and the likelihood of task success. The Q-function gradients provide rich guidance to the policy’s denoising process toward successful, well-supported actions.

Introduction

Generative policies, such as diffusion and flow matching, have emerged as the state-of-the-art for imitation learning in robotics, given their ability to model complex, multimodal action distributions [1, 2, 3]. However, as with other behavior-cloned policies, their performance can degrade under covariate shift: execution errors and environmental variations may drive the system into states poorly represented in the demonstration data, where the learned policy becomes unreliable. This problem is especially pronounced in dexterous manipulation, where complex contact-rich dynamics and long-horizon interactions can amplify small deviations.

Current methods for addressing covariate shift rely primarily on providing additional signal to adjust policy behavior. Human-in-the-loop methods utilize a supervisor to intervene and provide corrective demonstrations that address deviations as they occur [4, 5, 6, 7], but require significant human effort and are difficult to collect for dexterous robot hands. Methods for post-training generative policies with reinforcement learning can introduce more robust behaviors via trial-and-error [8, 9, 10, 11, 12, 13] but rely heavily on the quality of the reward signal. Dense rewards provide rich, informative signals to guide policy improvement, but generally require task-specific shaping, additional failure data, or manual failure annotations [14, 15, 16]. Sparse task-success rewards are often more scalable, but provide delayed signaling, which can increase the data burden and make adjustments challenging to learn [17, 18].

In this work, we explore how we can construct guidance for a generative policy without relying on human input, brittle assumptions, or engineered signals. Our key insight is to explicitly “nudge” the policy back toward its support region rather than relying on laborious task signaling. This prevents our target system from drifting into regions of limited support where our policy is likely to fail. We build on recent work that derives out-of-distribution (OOD) and anomaly scores for robotic behaviors [19, 20, 21] by constructing a model of the policy’s training demonstrations. While previous methods demonstrated the efficacy of anomaly detection for failure monitoring, behavior adjustment is primarily limited to naive retries [22] or state-based interpolation [23, 24], neither of which can be applied to contact-rich dexterous manipulation.

To translate anomaly detection into effective online adjustment, we propose steering the action-generation process at inference time. Inspired by gradient guidance in generative modeling [25], test-time steering preserves the pretrained policy as a behavioral prior while modifying its samples toward desirable actions. To effectively steer, we require an anticipatory signal of covariate shift, i.e., grounding future actions to their likelihood of leading to anomalous states. We propose constructing a Q-function trained on offline policy rollouts and online environment interactions to capture the discounted likelihood of future anomalies. The result is a function we can directly utilize to steer a policy to remain near its support. Further, we compose our proposed signal with a sparse task reward, which specifies what condition should ultimately be achieved, while the anomaly signal indicates where the pretrained policy is likely to remain competent. The resulting value function captures both the task return and the risk of leaving high-support regions, and its action gradient is used to steer each denoising process toward actions that improve the predicted task outcome while remaining close to the trusted region.

Contributions. We introduce GUARD: Guidance Using Anomaly-infoRmed Dense Values, a method that (1) combines sparse task rewards with dense anomaly signals to learn informative Q-functions for steering; and (2) utilizes an offline-to-online Q-function learning mechanism that uses the resulting value gradients to steer frozen flow matching policies toward successful, well-supported actions. We present a comprehensive evaluation on contact-rich dexterous manipulation tasks in the Adroit simulation benchmark and via direct sim-to-real transfer across three real-world tasks.

Failure Prediction and Anomaly Detection. Several recent works study failure prediction through anomaly detection, OOD detection, and uncertainty estimation. FAIL-Detect learns a flow model on the observation distribution from policy training data without requiring failure examples [19], while FIPER combines observation OOD and action-distribution uncertainty to predict failures of generative policies [20]. Sentinel monitors temporal action consistency and task progress [26], and other anomaly-based approaches detect deviations from nominal manipulation behavior to trigger failsafe or recovery mechanisms [21, 27, 28]. Policy uncertainty has also been used to improve residual-RL adaptation [29]. In contrast, we use policy-input anomaly scores as dense reward supervision for learning Q-functions, allowing information about distributional support to propagate through the Bellman return and shape the same value gradients that steer actions toward task success.

Improving Generative Robot Policies. The prevalence of generative policy methods has stoked much interest in methods that address failure modes and improve task performance. Applying post-training Reinforcement Learning (RL) is an intuitive choice, and has been demonstrated with success on generative policies [8, 9], however, the generative denoising processes make training unstable and inefficient. Alternative approaches include applying RL in the latent space of the policy [11, 12], adding a residual policy to the generative policy outputs [10, 12, 13], or ranking multiple outputs from the policy with an external value function [30]. Several works build on gradient guidance for generative modeling [25] to inject learned Q-function gradients into the denoising process [31, 32, 33, 34]. We build on this Q-guided sampling machinery but address an orthogonal bottleneck: what signal should the Q-function learn from when task supervision is sparse?

Several other methods have explored moving beyond RL signals for steering. UA-Flow steers flow generation directly using predicted policy uncertainty [35]. DynaGuide uses an action-conditioned dynamics model to steer predicted outcomes toward desired behaviors [36]. Latent Policy Barrier similarly predicts future latent states and guides them toward the expert demonstration distribution [37]. In contrast, our work utilizes a Q-function approach to transform an anomaly signal into a failure anticipatory signal, thereby avoiding dynamics learning. We provide a direct comparison to these methods in our experiments.

Background

Reinforcement Learning

We consider a Markov decision process (MDP) [38] M=(S,A,P,r,γ)\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, r, \gamma), with state space S\mathcal{S}, action space A\mathcal{A}, transition dynamics P\mathcal{P}, reward rr, and discount factor γ∈[0,1)\gamma\in[0, 1). A stationary policy π(a∣s)\pi(a \mid s) maximizes the expected discounted return J(π)=Eπ,P[∑t=0∞γtr(st,at)]J(\pi) = \mathbb{E}_{\pi,\mathcal{P}}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t)\right]. Its action-value function is Qπ(s,a)=Eπ,P[∑t=0∞γtr(st,at)∣s0=s,a0=a]Q^\pi(s, a) = \mathbb{E}_{\pi,\mathcal{P}}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \mid s_0 = s, a_0 = a\right]. Given transitions (s,a,r,s′)∼D(s, a, r, s') \sim\mathcal{D}, a Q-function QϕQ_\phi can be learned through temporal-difference learning by minimizing LQ(ϕ)=ED[(Qϕ(s,a)−r−γQϕˉ(s′,a′))2]\mathcal{L}_Q(\phi) = \mathbb{E}_{\mathcal{D}}\left[(Q_\phi(s, a) - r - \gamma Q_{\bar{\phi}}(s', a'))^2\right], where a′∼π(⋅∣s′)a' \sim\pi(\cdot\mid s') and QϕˉQ_{\bar{\phi}} denotes a target Q-function. Since Qπ(s,a)Q^\pi(s, a) depends explicitly on the action, its gradient ∇aQπ(s,a)\nabla_a Q^\pi(s, a) provides a local direction toward actions with a higher predicted return.

Flow Matching

Flow matching [2, 39] policies are parameterized by a state-conditioned, time-dependent velocity field vθ(a,k;s)v_\theta(a, k; s) that transports samples from a Gaussian noise distribution p0=N(0,I)p_0 = \mathcal{N}(0, I) to the policy distribution πθ(a∣s)\pi_\theta(a \mid s) via an ODE: dak=vθ(ak,k;s)dkd a_k = v_\theta(a_k, k; s) d k, a0∼p0a_0 \sim p_0. Given dataset state-action pairs (s,a1)∼D(s, a_1) \sim\mathcal{D}, we define linear interpolants ak=(1−k)a0+ka1a_k = (1-k)a_0 + k a_1 between noise and data actions and train the velocity field via flow matching: LFM(θ)=Ek∼U[0,1],a0∼p0,(s,a1)∼D[∥vθ(ak,k;s)−(a1−a0)∥22]\mathcal{L}_{\mathrm{FM}}(\theta) = \mathbb{E}_{k \sim\mathcal{U}[0,1],a_0 \sim p_0,(s,a_1) \sim\mathcal{D}}\left[\left\|v_\theta(a_k,k;s) - (a_1-a_0)\right\|_2^2\right]. At inference time, actions are generated by integrating the ODE from k=0k=0 to 11, yielding a sample a1∼πθ(a∣s)a_1 \sim\pi_\theta(a \mid s).

GUARD: Guidance Using Anomaly-infoRmed Dense Values

Our method consists of three components: (i) anomaly signals-informed dense reward construction, (ii) Q-guided action denoising, and (iii) offline-to-online Q-function adaptation. The overall procedure is summarized in Alg. 1.

Anomaly Signals-Informed Dense Rewards

Let Ddemo\mathcal{D}_{\mathrm{demo}} denote the demonstration data used to train the frozen flow policy πθ\pi_\theta. We fit an anomaly detector U(s)U(s) on the policy inputs in Ddemo\mathcal{D}_{\mathrm{demo}}, where larger values indicate observations that are less supported by the policy’s training distribution. Since UU is learned solely from the policy training data, it requires neither failure examples nor task-specific failure annotations. Our method naturally admits many available anomaly metrics. In our experiments, we primarily use the logpZO score [19] by fitting a separate flow model to the observations in Ddemo\mathcal{D}_{\mathrm{demo}} that transports each observation to a Gaussian latent zz, and take the squared latent norm ∥z∥22\lVert z\rVert_2^2 as its continuous anomaly score (large when an observation maps far from in-distribution regions).

We assign each state-action pair (st,at)(s_t, a_t) the anomaly score of st+1s_{t+1}, i.e., U(st+1)U(s_{t+1}). Using an offline buffer Doff\mathcal{D}_{\mathrm{off}} collected with the frozen flow policy, we compute and freeze reward-normalization statistics Ξoff:={μlog⁡Uoff,σlog⁡Uoff,μtaskoff,σtaskoff}\Xi_{\mathrm{off}} := \{\mu_{\log U}^{\mathrm{off}}, \sigma_{\log U}^{\mathrm{off}}, \mu_{\mathrm{task}}^{\mathrm{off}}, \sigma_{\mathrm{task}}^{\mathrm{off}}\}. The dense anomaly reward is then defined as

rtU=−log⁡U(st+1)−μlog⁡Uoffσlog⁡Uoff(1)r_t^U = -\frac{\log U(s_{t+1})-\mu_{\log U}^{\mathrm{off}}}{\sigma_{\log U}^{\mathrm{off}}} \tag*{(1)}

Thus, transitions toward states better supported by the policy’s training distribution receive a larger reward, while transitions toward anomalous states are penalized.

The available task supervision is a sparse binary signal, rttaskr_t^{\mathrm{task}}. To place both rewards on comparable scales, we independently standardize the task reward using the fixed statistics in Ξoff\Xi_{\mathrm{off}},

rˉttask=rttask−μtaskoffσtaskoff,(2)\bar{r}_t^{\mathrm{task}} = \frac{r_t^{\mathrm{task}}-\mu_{\mathrm{task}}^{\mathrm{off}}}{\sigma_{\mathrm{task}}^{\mathrm{off}}}, \tag*{(2)}

and define the reward used for Q-learning as their equal-weight sum,

r~t=rtU+rˉttask.(3)\tilde{r}_t = r_t^U + \bar{r}_t^{\mathrm{task}}. \tag*{(3)}

Independent standardization places both signals on comparable scales without introducing a tunable trade-off coefficient.

The two rewards provide complementary supervision: the sparse task reward specifies what the robot should ultimately accomplish, while the dense anomaly reward indicates where the pretrained policy is well supported. Crucially, we do not directly steer actions to minimize instantaneous anomaly. Instead, Eq. (3) enters the Bellman return, allowing the Q-function to propagate both task success and distributional support over future trajectories.

Q-Guided Action Denoising

We learn an ensemble of NN independently initialized Q-functions, Qϕ={Qϕi}i=1NQ_\phi= \{Q_{\phi_i}\}_{i=1}^N. To reduce exploitation of Q-function extrapolation errors during gradient-based guidance, we follow prior Q-guided methods [34] and construct the pessimistic ensemble estimate

L(s,a;Qϕ)=1N∑i=1NQϕi(s,a)−ρ std⁡iQϕi(s,a),(4)L(s,a;Q_\phi)=\frac{1}{N}\sum_{i=1}^{N}Q_{\phi_i}(s,a)-\rho\,\operatorname{std}_{i}Q_{\phi_i}(s,a), \tag*{(4)}

where ρ≥0\rho\ge0 controls the degree of pessimism. The ensemble disagreement in Eq. (4) is distinct from the anomaly score U(s)U(s): U(s)U(s) measures support of the policy input under the training distribution and is used for reward construction, whereas Q-ensemble disagreement provides conservative value estimates for guidance.

Intermediate flow variables are noisy and generally lie outside the clean-action distribution on which the Q-functions are trained. Following prior works [33, 40], at each denoising step k∈[0,1]k \in[0,1], we estimate the corresponding clean action with a single step over the remaining flow horizon,

a^k=ak+(1−k) vθ(ak,k;s),(5)\hat{a}_k = a_k + (1-k)\,v_\theta(a_k,k;s), \tag*{(5)}

query the ensemble at a^k\hat{a}_k and compute gk=∇a^kL(s,a^k;Qϕ)g_k=\nabla_{\hat{a}_k}L(s,\hat{a}_k;Q_\phi). Following QGF [33], we approximate ∂a^k/∂ak≈I\partial\hat{a}_k/\partial a_k\approx I, avoiding differentiation through the flow velocity. With denoising step size Δ\Delta, the guided update is

ak+Δ=ak+Δ[vθ(ak,k;s)+λ∇a^kL(s,a^k;Qϕ)].(6)a_{k+\Delta}=a_k+\Delta\left[v_\theta(a_k,k;s)+\lambda\nabla_{\hat{a}_k}L(s,\hat{a}_k;Q_\phi)\right]. \tag*{(6)}

We denote the action distribution induced by iterating Eq. (6) as πQϕ(a∣s)\pi_{Q_\phi}(a\mid s). The frozen flow policy remains the behavioral prior, while the Q-gradient biases denoising toward actions with high future anomaly signal-augmented task return. Here λ\lambda is the guidance weight, and we discuss details about the selection of λ\lambda in Sec. V.

Offline-to-Online Q-Function Adaptation

The Q-functions used for guidance should evaluate the policy that is ultimately executed, i.e., the steered policy. We therefore first estimate values under the base policy with offline initialization and subsequently adapt them to the evolving steered policy using online experience.

a) Offline initialization: We collect an offline buffer Doff\mathcal{D}_{\mathrm{off}} by rolling out the frozen policy πθ\pi_\theta with action noise σa\sigma_a, recording each transition together with its subsequent behavior action aBC′a'_{\mathrm{BC}}. Each independently initialized ensemble head uses its own exponentially averaged target Q-function Qˉi\bar{Q}_i. We perform behavior-policy fitted Q evaluation (FQE) using

yioff=r~+γ(1−d)Qˉi(s′,aBC′),(7)y_i^{\mathrm{off}}=\tilde{r}+\gamma(1-d)\bar{Q}_i(s',a'_{\mathrm{BC}}), \tag*{(7)}

where dd is the terminal indicator. Each QϕiQ_{\phi_i} is regressed toward its corresponding target, while its target parameters are updated as ϕˉi←τϕi+(1−τ)ϕˉi\bar{\phi}_i\leftarrow\tau\phi_i+(1-\tau)\bar{\phi}_i. Using per-head targets during offline initialization preserves ensemble diversity while estimating the anomaly-augmented return in regions covered by the behavior policy.

b) Online adaptation: Once Q-guidance is applied, the executed policy πQϕ\pi_{Q_\phi} differs from the behavior policy used for offline initialization. A Q-function evaluated only on the original behavior distribution can therefore become mismatched to the policy it is used to guide. We address this by alternating guided interaction with FQE updates under the resulting policy.

At each online iteration, we collect fresh trajectories using a∼πQϕ(⋅∣s)a\sim\pi_{Q_\phi}(\cdot\mid s) and append them to the unified replay buffer B=Doff∪Don\mathcal{B}=\mathcal{D}_{\mathrm{off}}\cup\mathcal{D}_{\mathrm{on}}. All online transitions are assigned rewards using Eq. (3) with the fixed normalization statistics Ξoff\Xi_{\mathrm{off}}.

For a sampled transition (s,a,r~,s′,d)(s,a,\tilde{r},s',d), we regenerate a fresh bootstrap action using the guided policy induced by the EMA target ensemble, a′∼πQˉϕ(⋅∣s′)a'\sim\pi_{\bar{Q}_\phi}(\cdot\mid s'), and construct the pessimistic target

y=r~+γ(1−d)L(s′,a′;Qˉ).(8)y=\tilde{r}+\gamma(1-d)L(s',a';\bar{Q}). \tag*{(8)}

Each ensemble head is regressed toward this shared target, EB[(Qϕi(s,a)−y)2]\mathbb{E}_{\mathcal{B}}[(Q_{\phi_i}(s,a)-y)^2], while its target network is updated by Polyak averaging. Using the slowly evolving target ensemble to generate both a′a' and its bootstrap value decouples action generation from instantaneous Q-function updates and stabilizes the guidance-value feedback loop. Repeated interaction and evaluation progressively adapt the Q-functions toward the steered policy, so that Q(s,a)Q(s,a) estimates the long-horizon return obtained after taking aa and subsequently following that guided policy. The resulting policy πQϕ(a∣s)\pi_{Q_\phi}(a \mid s) uses the ensemble of online-adapted QQ-functions to guide denoising towards high-return, high-support actions. Refer Alg. 1 for the full training procedure.

Algorithm 1: GUARD Training.

Require: Frozen flow policy πθ\pi_\theta; anomaly detector UU; ensemble size NN; pessimism ρ\rho; guidance weight λ\lambda, action noise σa\sigma_a; discount γ\gamma; Polyak rate τ\tau

Collect Doff\mathcal{D}_{\mathrm{off}} by rolling out πθ\pi_\theta with action noise σa\sigma_a, storing (s,a,s′,aBC′,rtask,U(s′),d)(s,a,s',a'_{\mathrm{BC}},r^{\mathrm{task}},U(s'),d)

Compute reward statistics Ξoff\Xi_{\mathrm{off}} from Doff\mathcal{D}_{\mathrm{off}} and freeze them

r~:=REWARD⁡(rtask,U(s′);Ξoff)\tilde{r} := \operatorname{REWARD}(r^{\mathrm{task}}, U(s'); \Xi_{\mathrm{off}}) using (3); B←Doff\mathcal{B} \leftarrow\mathcal{D}_{\mathrm{off}}

Initialize independent Qϕ={Qϕi}i=1NQ_\phi= \{Q_{\phi_i}\}_{i=1}^{N} and Qˉi←Qϕi\bar{Q}_i \leftarrow Q_{\phi_i}, ∀i\forall i

L(s,a;Q):=mean⁡iQi(s,a)−ρstd⁡iQi(s,a)L(s,a;Q) := \operatorname{mean}_i Q_i(s,a) - \rho\operatorname{std}_i Q_i(s,a)

πQ(⋅∣s):=Q\pi_Q(\cdot\mid s) := Q-guided flow policy defined by (6)

Phase-I: Offline Q-function initialization

for g=1,…,Goffg=1,\ldots,G_{\mathrm{off}} do

Sample (s,a,s′,aBC′,rtask,U(s′),d)∼Doff(s,a,s',a'_{\mathrm{BC}},r^{\mathrm{task}},U(s'),d)\sim\mathcal{D}_{\mathrm{off}}

r~←REWARD⁡(rtask,U(s′);Ξoff)\tilde{r}\leftarrow\operatorname{REWARD}(r^{\mathrm{task}},U(s');\Xi_{\mathrm{off}})

for i=1,…,Ni=1,\ldots,N do

yϕioff←r~+γ(1−d)Qˉi(s′,aBC′)y_{\phi_i}^{\mathrm{off}}\leftarrow\tilde{r}+\gamma(1-d)\bar{Q}_i(s',a'_{\mathrm{BC}})

Update QϕiQ_{\phi_i} toward yϕioffy_{\phi_i}^{\mathrm{off}}; ϕˉi←τϕi+(1−τ)ϕˉi\bar{\phi}_i\leftarrow\tau\phi_i+(1-\tau)\bar{\phi}_i

Phase-II: Online Q-function adaptation

for n=1,…,Nmax⁡n=1,\ldots,N_{\max} do

Collect fresh transitions using πQϕ\pi_{Q_\phi} with action noise σa\sigma_a and append them to B\mathcal{B}

for g=1,…,Gg=1,\ldots,G do

Sample (s,a,s′,rtask,U(s′),d)∼B(s,a,s',r^{\mathrm{task}},U(s'),d)\sim\mathcal{B}

r~←REWARD⁡(rtask,U(s′);Ξoff)\tilde{r}\leftarrow\operatorname{REWARD}(r^{\mathrm{task}},U(s');\Xi_{\mathrm{off}}); a′∼πQˉ(⋅∣s′)a'\sim\pi_{\bar{Q}}(\cdot\mid s')

y←r~+γ(1−d)L(s′,a′;Qˉ)y\leftarrow\tilde{r}+\gamma(1-d)L(s',a';\bar{Q})

for i=1,…,Ni=1,\ldots,N do

Update QϕiQ_{\phi_i} toward yy; ϕˉi←τϕi+(1−τ)ϕˉi\bar{\phi}_i\leftarrow\tau\phi_i+(1-\tau)\bar{\phi}_i

return QϕQ_\phi

Experiments

We evaluate GUARD on the D4RL Adroit dexterous manipulation benchmark [41] in simulation and further demonstrate its sim-to-real transfer capabilities across three tasks using the Allegro V5 Plus multi-fingered robot hand. Through these experiments, we aim to answer the following research questions (RQs):

  1. RQ1: Can GUARD reliably steer the policy toward successful task completion?

  1. RQ2: Does GUARD keep the system near high-support, in-distribution regions?

  1. RQ3: Is GUARD more robust under distribution shift, including perturbed dynamics and sim-to-real transfer?

Adroit Simulation Benchmark

We run experiments on all four Adroit manipulation tasks: Relocate, Pen, Hammer, and Door [41]. We train base flow-matching policies over 200 episodes, sampled equally from the expert and cloned subsets. The policy uses a four-layer MLP with 512 hidden units, GELU activations, layer normalization, and K=10K=10 Euler denoising steps. GUARD uses an ensemble of five QQ-functions with pessimism ρ=0.5\rho=0.5, trained for 40k offline FQE updates on 800 episodes with batch size 256, learning rate 3×10−43\times10^{-4}, discount γ=0.99\gamma=0.99, EMA target rate τ=0.005\tau=0.005, followed by 150 iterations of online adaptation collecting 24 episodes each iteration. Each episode includes 200 steps, and the sparse task reward used for Adroit tasks is −1-1 everywhere and 0 on successful states, aligning with the commonly used choice in offline RL. We calibrate λ\lambda once after offline initialization by matching the average gradient scale to the average flow-velocity scale, λ=E[∥∇θ∥2]E[∥∇a^kL(s,ak;Qϕ)∥2]\lambda=\frac{\mathbb{E}[\lVert\nabla_\theta\rVert_2]}{\mathbb{E}[\lVert\nabla_{\hat{a}_k}L(s,a_k;Q_\phi)\rVert_2]}, where the expectations are estimated using the offline-initialized ensemble and then hold λ\lambda fixed throughout the online adaptation and deployment.

a) Baselines: We compare GUARD against: (i) BC, the unsteered flow-matching policy; (ii) LPB [37] and (iii) DynaGuide [36], which use a learned dynamics model to steer predicted future states toward in-distribution regions and desired task outcomes respectively; (iv) UA-Flow [35], which guides flow sampling toward actions with lower predicted uncertainty; (v) Sparse-Q, QQ-guided denoising using only the sparse task success reward; and (vi) U-SA, which directly guides denoising using action gradient of joint state-action anomaly scores U(s,a)U(s,a). Every baseline is trained using its prescribed methodology, with an equal data budget (both offline and online) to GUARD, for fair comparison. UA-Flow and U-SA are trained only with Ddemo\mathcal{D}_{\mathrm{demo}}, as they rely only on the training demonstrations.

b) Results: In Table I, we evaluate 100 episodes per task in each of the nominal and perturbed dynamics settings. In the latter setting, we add Gaussian noise to the actions to investigate GUARD under distribution shift.

MethodMetricRelocate σd=0\sigma_d=0Relocate σd=0.1\sigma_d=0.1Pen σd=0\sigma_d=0Pen σd=0.3\sigma_d=0.3Hammer σd=0\sigma_d=0Hammer σd=0.1\sigma_d=0.1Door σd=0\sigma_d=0Door σd=0.05\sigma_d=0.05Average σd=0\sigma_d=0Average −-
BCSR0.74±0.010.74 \pm0.010.45±0.060.45 \pm0.060.75±0.0080.75 \pm0.0080.49±0.040.49 \pm0.040.52±0.030.52 \pm0.030.40±0.050.40 \pm0.050.37±0.080.37 \pm0.080.32±0.050.32 \pm0.050.60±0.030.60 \pm0.030.41±0.050.41 \pm0.05
AS12.14±4.7612.14 \pm4.7625.34±4.5125.34 \pm4.5116.81±1.6516.81 \pm1.6541.37±18.6241.37 \pm18.62105.16±49.48105.16 \pm49.4895.54±38.3695.54 \pm38.3619.94±3.5419.94 \pm3.5419.51±1.1719.51 \pm1.1738.51±14.8638.51 \pm14.8645.44±15.6745.44 \pm15.67
Sparse-QSR0.80±0.110.80 \pm0.110.55±0.130.55 \pm0.130.88±0.040.88 \pm0.040.62±0.020.62 \pm0.020.74±0.090.74 \pm0.090.66±0.110.66 \pm0.110.21±0.070.21 \pm0.070.19±0.050.19 \pm0.050.66±0.080.66 \pm0.080.50±0.080.50 \pm0.08
AS12.67±2.7912.67 \pm2.7988.60±28.7088.60 \pm28.705.71±1.065.71 \pm1.0618.17±1.0618.17 \pm1.0658.58±24.5658.58 \pm24.5683.44±19.4783.44 \pm19.4724.23±4.3624.23 \pm4.3618.98±2.3418.98 \pm2.3425.30±8.1925.30 \pm8.1952.30±12.8952.30 \pm12.89
LPBSR0.78±0.030.78 \pm0.030.53±0.030.53 \pm0.030.74±0.030.74 \pm0.030.50±0.030.50 \pm0.030.57±0.040.57 \pm0.040.47±0.030.47 \pm0.030.49±0.040.49 \pm0.040.36±0.030.36 \pm0.030.64±0.040.64 \pm0.040.47±0.030.47 \pm0.03
AS9.75±3.289.75 \pm3.2842.95±19.1042.95 \pm19.106.46±2.186.46 \pm2.1849.17±30.8649.17 \pm30.8628.03±10.1328.03 \pm10.1339.07±8.8439.07 \pm8.8416.19±2.1316.19 \pm2.1316.93±2.7016.93 \pm2.7015.11±4.4315.11 \pm4.4337.03±15.3837.03 \pm15.38
DynaGuideSR0.87±0.030.87 \pm0.030.55±0.040.55 \pm0.040.79±0.030.79 \pm0.030.54±0.010.54 \pm0.010.78±0.040.78 \pm0.040.79±0.050.79 \pm0.050.55±0.020.55 \pm0.020.51±0.040.51 \pm0.040.75±0.030.75 \pm0.030.60±0.040.60 \pm0.04
AS9.18±3.759.18 \pm3.7535.12±5.3935.12 \pm5.3921.68±7.6021.68 \pm7.6039.29±10.6239.29 \pm10.6222.01±7.1822.01 \pm7.1817.71±7.5817.71 \pm7.5817.37±1.2817.37 \pm1.2819.62±0.2619.62 \pm0.2617.56±4.9517.56 \pm4.9527.94±5.9627.94 \pm5.96
UA-FlowSR0.75±0.010.75 \pm0.010.43±0.020.43 \pm0.020.73±0.0090.73 \pm0.0090.47±0.030.47 \pm0.030.51±0.020.51 \pm0.020.36±0.0090.36 \pm0.0090.39±0.030.39 \pm0.030.34±0.050.34 \pm0.050.60±0.020.60 \pm0.020.40±0.030.40 \pm0.03
AS7.15±1.377.15 \pm1.3746.10±16.2646.10 \pm16.269.15±2.859.15 \pm2.8538.42±10.2338.42 \pm10.2391.51±18.0291.51 \pm18.0284.32±37.3884.32 \pm37.3816.03±0.9416.03 \pm0.9420.30±2.5020.30 \pm2.5030.96±5.8030.96 \pm5.8047.29±16.5947.29 \pm16.59
U-SASR0.84±0.020.84 \pm0.020.48±0.020.48 \pm0.020.75±0.030.75 \pm0.030.53±0.020.53 \pm0.020.77±0.030.77 \pm0.030.68±0.050.68 \pm0.050.52±0.060.52 \pm0.060.39±0.050.39 \pm0.050.72±0.040.72 \pm0.040.52±0.040.52 \pm0.04
AS5.74±1.025.74 \pm1.0245.15±13.5445.15 \pm13.5412.66±6.2112.66 \pm6.2135.84±8.9435.84 \pm8.9414.79±5.3714.79 \pm5.3732.25±16.9432.25 \pm16.9411.17±0.4911.17 \pm0.4917.40±1.5317.40 \pm1.5311.09±3.2711.09 \pm3.2732.66±10.2432.66 \pm10.24
GUARDSR0.98±0.0050.98 \pm0.0050.79±0.030.79 \pm0.030.93±0.010.93 \pm0.010.60±0.030.60 \pm0.030.99±0.0090.99 \pm0.0090.93±0.060.93 \pm0.060.95±0.0050.95 \pm0.0050.89±0.030.89 \pm0.030.96±0.0080.96 \pm0.0080.80±0.040.80 \pm0.04
AS4.21±1.894.21 \pm1.8962.49±31.9462.49 \pm31.944.04±0.534.04 \pm0.5316.83±1.8916.83 \pm1.891.19±0.661.19 \pm0.663.46±1.723.46 \pm1.723.29±0.803.29 \pm0.803.92±0.603.92 \pm0.603.18±0.973.18 \pm0.9721.68±9.0421.68 \pm9.04

Table I. Adroit simulation results across nominal and perturbed dynamics. Here, σd=0\sigma_d = 0 denotes nominal, while σd>0\sigma_d > 0 denotes task-specific dynamics perturbations at the indicated noise level. Results are reported over three seeds using 100 test rollouts. SR denotes success rate (↑\uparrow), and AS denotes the RND-based state-anomaly score on successful episodes (↓\downarrow).

In the nominal setting, GUARD achieves the strongest overall task performance, reaching an average success rate (SR) of 0.96. This corresponds to an average gain of 36.0 percentage points (pp) over the BC policy and a gain of 21.0 pp over the next-best baseline, DynaGuide. Figure 2 shows how GUARD successfully steers an originally failed trajectory under the BC policy for the Pen task starting from the same initial state. These results establish for RQ1 that GUARD can effectively steer generative policies toward task success and outperform existing steering strategies.

Qualitative visualization of steering with GUARD in the Adroit Pen task

Figure 2. A qualitative visualization of steering with GUARD in the Adroit Pen task. (Left) unsteered BC policy forms a loose grasp of the pen and fails to properly reorient it, as indicated by low Q-values L(s,a;Q)L(s,a;Q) and high state-anomaly scores U(s)U(s). (Right) At the moment some fingers lose contact, steering provides a larger action correction ∥Δa∥\lVert\Delta a\rVert folding the fingers back in to maintain a firm grasp and reorient the pen to the target orientation.

GUARD also consistently keeps the system much closer to in-distribution regions, achieving the lowest state-anomaly scores (AS) on successful episodes for each method across all tasks in the nominal setting. Here, the AS is evaluated using random network distillation (RND) [42, 43], a neural metric for fair comparison across all methods, using a fixed random target network gg and a predictor hh trained to regress gg on Ddemo\mathcal{D}_{\mathrm{demo}} inputs, calculating the anomaly score as RND⁡(s)=∥h(s)−g(s)∥22\operatorname{RND}(s)=\lVert h(s)-g(s)\rVert_2^2. These results answer RQ2: GUARD realizes substantially stronger support under the policy training distribution.

Finally, we compute both SR and AS in the perturbed-dynamics setting. GUARD achieves an average SR of 0.80 in the shifted environment, a 39.0 and 20.0 pp gain over the BC policy and the next-best baseline, again DynaGuide, and the lowest AS score on average across all tasks. Figure 3(b) shows our method maintains the highest performance with increasing perturbation levels as shown on the Door task. Thus, answering RQ3, GUARD demonstrates more robust performance across both tests under distribution shift.

Failure detection, dynamics robustness, and data efficiency results

Figure 3. (a) Failure-detection quality on Door: episode-level AUROC (max-aggregated per-step score) using each method’s steering signal. (b) Robustness on Door to test-time dynamics perturbations of magnitude σd\sigma_d (action noise). (c) Offline data efficiency on Hammer: success rate vs. offline dataset size; dashed = offline-only (no online adaptation), solid = offline + online training. (d) Online data efficiency on Hammer: success rate vs. online interaction (env steps) at the 800-episode offline budget. Overall, GUARD leads across failure detection, dynamics robustness, and data efficiency.

Our results indicate the efficacy of GUARD for driving task improvement. The performance gains over U-SA (nominal: +24.0 pp; perturbed: +28.0 pp) indicate the importance of grounding the signal in actions with the Q-function. Gains over the Sparse-Q (nominal: +30.0 pp; perturbed: +30.0 pp) show the value of dense anomaly-signaling. While LPB and DynaGuide utilize a failure-anticipation mechanism by rolling out the predicted dynamics, we find that GUARD outperforms both, suggesting that the dynamics of the contact-rich tasks under test may be challenging to capture accurately. Finally, UA-Flow relies only on the policy uncertainty, which provides a weaker steering signal than GUARD.

c) Failure Signal Alignment: We compare each method’s failure-detection performance using the AUROC metric aggregated over the episode. Figure 3(a) shows that GUARD achieves the highest AUROC, demonstrating its ability to reliably identify failures, complementing its steering performance.

d) Training Data Efficiency: GUARD is highly data-efficient in both the offline and online phases. The Q-function reaches near-perfect steered success with only a modest buffer of offline data paired with online training and stays saturated as the buffer grows, whereas the strongest baselines (DynaGuide, LPB) plateau well below it (Figure 3(c)). The gap between the offline-only (dashed) and offline + online (solid) curves shows that offline training alone is insufficient and that online adaptation supplies most of the leverage for steering. Online, GUARD quickly improves its signaling, climbing to ≈0.9\approx0.9 within the first ∼105\sim10^5 environment steps and saturating near 1.0, while the baselines improve more slowly and to a lower ceiling (Figure 3(d)). Thus, GUARD achieves the highest performance with little offline data and some online interactions, a favorable trade-off for a scalable real-world deployment.

e) Ablation on Anomaly Signal: We ablate the choice of state-anomaly score U(s)U(s) by swapping in alternative OOD signals in GUARD, such as input reconstruction error (Recon-AE) [21], RND [43], and learned dynamics-based metrics such as prediction error (Dyn-Err) and ensemble disagreement (Dyn-Ens) [44], as shown in Table II. The state-density logpZO metric we adopt is both the strongest and most stable, while Recon-AE, RND, and Dyn-Ens also remain competitive. In contrast, Dyn-Err collapses, indicating that prediction error conflates model inaccuracy with genuine anomaly and is a poor proxy for how far a state has drifted OOD. This ablation shows that steering benefits from a state-distributional notion of anomaly, and that logpZO is a robust default choice.

logpZORNDRecon-AEDyn-ErrDyn-Ens
.95±.005.95 \pm.005.89±.03.89 \pm.03.90±.02.90 \pm.02.61±.12.61 \pm.12.91±.008.91 \pm.008

Table II. Anomaly-Signal Ablation on Door (SR ↑\uparrow).

Real-World Experiments

We evaluate on three dexterous tasks with a 16-DoF Allegro V5 Plus hand: (1) screwdriver in-hand rotation, requiring continuous rotations throughout the episode without dropping, and screwdriver in-hand reorientation in (2) palm-down and (3) palm-up configurations, requiring the screwdriver to be reoriented vertically below and above the palm, respectively, from a horizontal initial pose, just using its fingers in the fixed-wrist hand, and holding it there for ≥5\geq5 sec. The BC policies are cloned from 64, 256, and 4096 demonstrations, respectively, for each task, each 40 sec long. These demonstrations are generated for each task using an RL expert policy trained with PPO [45] using self-defined dense rewards in the mjlβab simulation, with domain randomization of dynamics and observations. The BC policy is a conditional 1D U-Net (channels 256/512/1024, kernel 5) taking 41 dim observations containing joint positions, previously commanded joint positions, and object pose in the palm frame for all tasks. It outputs 16 dim joint-position deltas of chunk length 4 with an execution horizon of 2. It is trained for 100 epochs with batch size 256, learning rate 10−410^{-4}, cosine schedule, and EMA decay 0.9999, with K=10K=10 Euler steps at inference. GUARD uses an ensemble of five Q-functions with pessimism ρ=0.5\rho=0.5, trained for 200k offline FQE updates on 2000 policy rollouts with batch size 2048, learning rate 10−410^{-4}, discount γ=0.95\gamma=0.95, EMA target rate τ=0.005\tau=0.005, followed by ∼40\sim40 iterations of online adaptation collecting 128 episodes in each iteration. The sparse task reward is 0 everywhere and 1 on successful states. We keep dynamics and observation domain randomization enabled to train GUARD under the same noise distribution as seen by the BC policy. Note that GUARD is trained entirely in simulation, for both the offline and online stages, and then evaluated directly in the real world to assess its sim-to-real capabilities.

TaskSuccess (%) ↑Drop (%) ↓RND ↓
BCGUARDBCGUARDBCGUARD
Rotation206080250.11 ±0.030.07 ±0.02
Palm-Down70953050.05 ±0.110.03 ±0.07
Palm-Up30105200.01 ±0.000.02 ±0.01

Table 3. Real-robot Allegro V5 Plus screwdriver manipulation results for BC and GUARD, with n=20n=20 trials per method. The best result for each metric is shown in bold.

a) Results: On the continuous rotation and palm-down reorientation tasks, GUARD substantially improves real-robot performance over the unsteered BC policy (Table III). For rotation, the success rate improves by 40.0 pp, and the drop rate reduces by 55.0 pp. Figure 4 shows how GUARD successfully steers the policy for the rotation task. On palm-down reorientation, GUARD improves the success rate by 25.0 pp, converting those drops to successful reorientations.

Qualitative visualization of steering with GUARD in the screwdriver in-hand-rotation task

Figure 4. A qualitative visualization of steering with GUARD in the screwdriver in-hand-rotation task. (Left) unsteered BC policy drops the screwdriver, indicated by a large, sudden drop in the Q-value L(s,a;Q)L(s, a; Q) and a jump in state-anomaly score U(s)U(s). (Right) Whenever the fingers lose hold of the screwdriver, steering provides a larger action correction ∥Δa∥\lVert\Delta a\rVert, successfully correcting the grasp twice in the episode, and prevents the screwdriver from falling.

The terminal orientation error from the palm normal axis reduces from 4.4∘→2.1∘4.4^\circ\to2.1^\circ and the longest stable object hold duration increases from 32.3→37.532.3 \to37.5 s when evaluated on successful episodes for both methods, indicating tighter and more durable reorientation. GUARD also maintains a low RND score, comparable to BC, on successful episodes, indicating that steered rollouts remain in-distribution and do not encounter OOD states. On the palm-up reorientation task, GUARD hurts performance relative to BC. We attribute this to inaccurate contact modeling in simulation compared to the real world. We observed a pronounced screwdriver rotation mid-execution on the real robot that occurred rarely under the same policy in simulation, resulting in a large sim-to-real gap. Because these transitions were largely absent in the simulation, we hypothesize that the Q-function is unable to anticipate this behavior and thus fails to provide the required correction to recover to high-support regions. We intend to resolve such gaps in future work by training with wider domain randomization and on mixed buffers that incorporate real-world interactions, a commonly used technique in steering methods for real-world deployment (LPB, DynaGuide). This can further close the sim-to-real gap, allowing the Q-function to be well aware of the failure modes actually encountered on hardware.

Conclusion and Future Work

We introduced GUARD, an anomaly-informed Q-guidance framework for robustly steering frozen generative robot policies. Rather than relying on task-specific dense rewards or manual failure annotations, GUARD leverages anomaly signals learned from the base policy’s training distribution and combines them with sparse task rewards to learn Q-functions. By propagating this information through the Bellman return, GUARD enables the value gradients to account for both task completion and the likelihood of entering poorly supported states, rather than greedily minimizing instantaneous uncertainty.

Across Adroit dexterous manipulation tasks, GUARD significantly improves task success while maintaining lower anomaly scores than existing steering methods, including under perturbed dynamics. We further demonstrate direct sim-to-real transfer without any real-robot training, achieving performance gains on two of three evaluated tasks. Together, these results demonstrate that signals typically used for runtime failure monitoring can also provide scalable supervision to actively improve generative robot policies through test-time Q-steering methods.

Our results indicate that our anomaly-informed Q-steering can transfer surprisingly well across shifts in dynamics and even to the real world. We hypothesize that task conditions with overlapping successful state distributions and comparable dynamics can reliably transfer the anomaly metrics and subsequent Q-functions, as demonstrated by successful steering across perturbed simulation dynamics and by rotation and palm-down reorientation results in the real world. However, if the environment shift significantly alters the successful state distribution, turning nominal states anomalous and leaving the Q-function unaware of real failure modes, GUARD may no longer provide the required correction. This is exemplified by the palm-up reorientation results.

Future work will explore resolving these issues through mixed simulation and real-world training to further improve real-world transfer and explore extensions to policies with multimodal inputs and broader manipulation tasks.

References

  1. [1]C. Chi et al., “Diffusion policy: Visuomotor policy learning via action diffusion,” International Journal of Robotics Research, 2025.
  2. [2]Y. Lipman et al., “Flow matching for generative modeling,” in International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
  3. [3]K. Black et al., “π₀: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
  4. [4]S. Ross et al., “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
  5. [5]H. Liu et al., “Model-based runtime monitoring with interactive imitation learning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024.arxiv.org/abs/2310.17552
  6. [6]R. Hoque et al., “Intervengen: Interventional data generation for robust and data-efficient robot imitation learning,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2024.arxiv.org/abs/2405.01472
  7. [7]R. Hoque et al., “Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning,” in Proceedings of the 5th Conference on Robot Learning, A. Faust et al., Eds., PMLR, 2022.arxiv.org/abs/2109.08273
  8. [8]A. Ren et al., “Diffusion policy policy optimization,” in International Conference on Learning Representations, 2025.arxiv.org/abs/2409.00588
  9. [9]T. Zhang et al., “Reinflow: Fine-tuning flow matching policy with online reinforcement learning,” in Advances in Neural Information Processing Systems, D. Belgrave et al., Eds., Curran Associates, Inc., 2025.arxiv.org/abs/2505.22094
  10. [10]L. L. Ankile et al., “From imitation to refinement–residual rl for precise visual assembly,” in CoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant Data, 2024.arxiv.org/abs/2407.16677
  11. [11]A. Wagenmaker et al., “Steering your diffusion policy with latent space reinforcement learning,” in Conference on Robot Learning, PMLR, 2025.arxiv.org/abs/2506.15799
  12. [12]E. Su et al., “Rfs: Reinforcement learning with residual flow steering for dexterous manipulation,” in The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2602.01789
  13. [13]Z. Sun et al., “From prior to pro: Efficient skill mastery via distribution contractive rl finetuning,” arXiv preprint arXiv:2603.10263, 2026.arxiv.org/abs/2603.10263
  14. [14]Q. Chen et al., “Sarm2: Multi-task stage aware reward modeling for self improving robotic manipulation,” arXiv preprint arXiv:2606.10305, 2026.arxiv.org/abs/2606.10305
  15. [15]Y. Fang et al., DenseReward: Dense reward learning via failure synthesis for robotic manipulation, 2026.arxiv.org/abs/2607.13033
  16. [16]T. W. Ayalew et al., “Progressor: A perceptually guided reward estimator with self-supervised online refinement,” in 2025 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, 2025.arxiv.org/abs/2411.17764
  17. [17]M. Andrychowicz et al., “Hindsight experience replay,” Advances in neural information processing systems, 2017.arxiv.org/abs/1707.01495
  18. [18]A. Nair et al., “Overcoming exploration in reinforcement learning with demonstrations,” in 2018 IEEE international conference on robotics and automation (ICRA), IEEE, 2018.arxiv.org/abs/1709.10089
  19. [19]C. Xu et al., “Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies,” in Proceedings of Robotics: Science and Systems (RSS), 2025.
  20. [20]R. Römer et al., “Failure prediction at runtime for generative robot policies,” Advances in Neural Information Processing Systems (NeurIPS), 2025.
  21. [21]N. Grambow et al., “Anomaly detection for generic failure monitoring in robotic assembly, screwing and manipulation,” IEEE Robotics and Automation Letters, 2026.DOI
  22. [22]S. Zhou et al., “Adaptive online replanning with diffusion models,” in Advances in Neural Information Processing Systems, A. Oh et al., Eds., Curran Associates, Inc., 2023.arxiv.org/abs/2310.09629
  23. [23]A. Reichlin et al., “Back to the manifold: Recovering from out-of-distribution states,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2022.arxiv.org/abs/2207.08673
  24. [24]G. J. Gao et al., “Out-of-distribution recovery with object-centric keypoint inverse policy for visuomotor imitation learning,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2025.arxiv.org/abs/2411.03294
  25. [25]P. Dharival et al., “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems, 2021.
  26. [26]C. Agia et al., “Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress,” in Proceedings of The 8th Conference on Robot Learning, P. Agrawal et al., Eds., PMLR, 2025.arxiv.org/abs/2410.04640
  27. [27]B. Santhanam et al., “Reliable robotic task execution in the face of anomalies,” IEEE Robotics and Automation Letters, 2025.DOI
  28. [28]Q. Rolland et al., “Failure identification in imitation learning via statistical and semantic filtering,” in IEEE International Conference on Robotics and Automation (ICRA), 2026.arxiv.org/abs/2604.13788
  29. [29]L. Dodeja et al., “Accelerating residual reinforcement learning with uncertainty estimation,” IEEE Robotics and Automation Letters, 2026.
  30. [30]M. Nakamoto et al., “Steering your generalists: Improving robotic foundation models via value guidance,” in Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2410.13816
  31. [31]Y. Jang et al., “Q-guided flow q-learning,” in CoRL 2025 Workshop RememberRL.
  32. [32]H. Ye, “Steering diffusion policies with value-guided denoising,” in NeurIPS 2025 Workshop on Embodied World Models for Decision Making, 2025.
  33. [33]Z. Zhou et al., “Test-time gradient guidance of flow policies in reinforcement learning,” arXiv preprint arXiv:2606.11087, 2026.arxiv.org/abs/2606.11087
  34. [34]Y. Ruan et al., “Qpilots: Efficient test-time q-steering for flow policies,” arXiv preprint arXiv:2606.14801, 2026.arxiv.org/abs/2606.14801
  35. [35]J. Han et al., “Flow matching with uncertainty quantification and guidance,” arXiv preprint arXiv:2602.10326, 2026.arxiv.org/abs/2602.10326
  36. [36]M. Du et al., “Dynaguide: Steering diffusion policies with active dynamic guidance,” in Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2506.13922
  37. [37]Z. Sun et al., “Latent policy barrier: Learning robust visuomotor policies by staying in-distribution,” in Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2508.05941
  38. [38]R. S. Sutton et al., Reinforcement learning: An introduction. MIT press Cambridge, 1998.DOI
  39. [39]X. Liu et al., “Flow straight and fast: Learning to generate and transfer data with rectified flow,” in International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2209.03003
  40. [40]B. Kang et al., “Efficient diffusion policies for offline reinforcement learning,” Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.20081
  41. [41]A. Rajeswaran et al., “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” arXiv preprint arXiv:1709.10087, 2017.arxiv.org/abs/1709.10087
  42. [42]Y. Burda et al., “Exploration by random network distillation,” arXiv preprint arXiv:1810.12894, 2018.arxiv.org/abs/1810.12894
  43. [43]N. He et al., “Rediffuser: Reliable decision-making using a diffuser with confidence estimation,” in Forty-first International Conference on Machine Learning, 2024.
  44. [44]T. Haider et al., “Out-of-distribution detection for reinforcement learning agents with probabilistic dynamics models,” in Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, 2023.
  45. [45]J. Schulman et al., Proximal policy optimization algorithms, 2017.arxiv.org/abs/1707.06347

Paper details

vlnc:W_9h6mend4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents