betaFeedback

Reading v1View latest version

WorksComputer Science

Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences

Abstract

Robots increasingly choose actions by searching against a reward model learned from human preferences, in model-predictive control or best-of-NN selection; we measure what this does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic labels and blind labels from the two authors, harder MPPI search cuts true return by up to 76%: with human labels the drawer opens in 0 of 20 runs against 10 of 10 for the cloned policy the planner starts from, while held-out label accuracy stays at 0.75-0.97. On plate-slide human labels are exploited less than synthetic ones, and on window-open search helps. True return roughly follows how far the planner's weights move from its proposals (KL\sqrt{\mathrm{KL}}) for both weighting rules we test, and best-of-NN with a verifier from synthetic labels falls below a single noisy action. We prove that χ2χ^2-regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, that no positive gain can be certified once that spread exceeds the scorer's own, and that an episode's return splits exactly into per-step terms, of which the guarantee covers the selection terms. Calibrated from 100 proposal pairs ranked by true return (an oracle) on start states the evaluation never uses, this rule is the safest fix we test: with each of five independent sets it succeeds in every arm run and keeps on average 86-94% of the cloned policy's return on plate-slide and drawer-open, though it is cautious where search helps. Calibration from a person's comparisons cannot detect a bias the scorer learned from that person; correcting for the pairs people cannot tell apart, or a second person, limits the damage.

Introduction

A robot that plans against a learned reward is an optimizer pointed at a proxy. Reward models learned from human preferences are now used inside model-predictive control ([17, 9, 28]), and robot foundation models are scaled at test time by sampling many candidate actions and executing the one a learned verifier likes best ([20]). In language-model alignment it is well established that optimizing too hard against a learned reward lowers the quality it was meant to measure ([11, 7, 8]), and a line of theory explains when regularization toward a reference policy does and does not prevent it (Huang et al., 2025b;a; Kwa et al., 2024; Laidlaw et al., 2025). Robot planners differ: the optimizer is a sampling-based search whose strength is set by a temperature and a sample count, the regularizer is implicit in the proposals, the plan is re-optimized at every step, and the labels number hundreds to thousands, not hundreds of thousands. How much such a planner exploits its scorer has not been measured.

We study Model Predictive Path Integral control (MPPI; [41 b]) with proposals around a policy cloned from the labelled data, the planning analogue of RLHF’s reference model, and a Bradley–Terry scorer trained from preference labels. We use the true simulator for planning, so that every failure is attributable to the scorer and not to a learned dynamics model. Our contributions are:

  • Goodhart in the planner (§ 4.1). On HalfCheetah and Hopper with real crowd-sourced labels ([46]), and on Meta-World drawer-open and plate-slide with synthetic labels and blind labels from the two authors (Yu et al., 2020a), harder search lowers true return below that of the cloned policy it starts from, while a perfect scorer gains or, on hopper, loses less. High agreement on decided human comparisons does not make planning safe: on the drawer the pairs a person cannot tell apart carry what the planner needs. On plate-slide human labels are exploited less than synthetic ones, and on window-open search helps.

  • An approximate collapse onto distance (§ 4.2).

Within each task, true return falls near one curve in d=KLd=\sqrt{\mathrm{KL}} between the planner’s weights and its proposals for both weighting rules we test, so χ2\chi^2 weights help mainly by moving less.

  • How hard to search, and when not to (§ 3). Against an error spread σ\sigma on the proposals, χ2\chi^2-regularized MPPI at λχ2=σ\lambda_{\chi^2}=\sigma is worst-case optimal, no positive gain can be certified once σ\sigma exceeds the scorer’s own spread, and the guarantee of hard exponential search falls as −σN-\sigma\sqrt{N}; an exact decomposition of an episode’s return shows which terms this controls. The mathematics is elementary; what is new is bounding the error on the planner’s own proposals, which held-out accuracy does not predict (§ 4.3), and following it through the closed loop.

  • Fixes, measured honestly (§ 4.4). Calibrated from 100 proposal pairs ranked by true return, the worst-case-optimal rule is the safest fix we test, though cautious where search helps; calibrated from the labeller’s own comparisons, it cannot detect a bias the scorer shares with that person; ensembles, χ2\chi^2 weights, automatic temperatures and a guard each win somewhere and lose elsewhere, and a UBP2-inspired CEM planner (Nabail et al., [28]) is the most exploited. Labelling the planner’s own trajectories reduces the exploitation.

  • Best-of-NN is not safe either (§ 4.6). With a verifier from 500 synthetic pairs it falls below one noisy action from N=16N=16 (N=4N=4 on plate-slide); with 2,000 pairs selection helps.

Setting

Planner. At each control step MPPI rolls out NN action sequences of length HH from the current state and executes the first action of their weighted average (Williams et al., [41]). We draw the sequences around the cloned policy’s own closed-loop plan UU: Ai=clip⁡(U+ϵi)A_i=\operatorname{clip}(U+\epsilon_i), ϵi∼N(0,σ2I)\epsilon_i\sim\mathcal{N}(0,\sigma^2I), which plays the role of the reference policy in RLHF. Each plan gets a learned return J^(Ai)=∑h<Hr^(oh,ah)\hat{J}(A_i)=\sum_{h<H}\hat{r}(o_h,a_h) and, for evaluation only, a true return J(Ai)J(A_i). Standard (exponential) MPPI uses wi∝exp⁡(J^i/λ)w_i\propto\exp(\hat{J}_i/\lambda), the solution of max⁡w∑iwiJ^i−λKL⁡(w∥unif)\max_{w}\sum_i w_i\hat{J}_i-\lambda\operatorname{KL}(w\mathbin\Vert\mathrm{unif}), the same KL-regularized objective as RLHF with β=λ\beta=\lambda. The χ2\chi^2 rule of Tsallis-divergence MPPI (Wang et al., [40]) uses wi∝max⁡(0,J^i−c)w_i\propto\max(0,\hat{J}_i-c) with cc set by λ\lambda, the solution with λχ2/2\lambda_{\chi^2}/2 in place of λKL\lambda_{\mathrm{KL}}. Lower λ\lambda means harder search; λ→0\lambda\to0 is best-of-NN over plans. We log the per-step KL⁡(w∥unif)=∑iwilog⁡(Nwi)≤log⁡N\operatorname{KL}(w\mathbin\Vert\mathrm{unif})=\sum_iw_i\log(Nw_i)\le\log N. Uniform weights (λ=106\lambda=10^6) give the planner without search, which we call uniform averaging: it executes the average of the clipped noisy proposals, which is not the cloned policy’s own action, so we also run the cloned policy directly. Unless stated, N=256N=256, H=20H=20, σ=0.5\sigma=0.5, and planning uses the true MuJoCo dynamics (Todorov et al., [38]).

Scorers. A two-layer MLP r^(o,a)\hat{r}(o,a) is trained on segment pairs with the Bradley–Terry likelihood P(σ1≻σ2)=sigmoid⁡(∑tr^(σt1)−∑tr^(σt2))P(\sigma^1\succ\sigma^2)=\operatorname{sigmoid}(\sum_t\hat{r}(\sigma^1_t)-\sum_t\hat{r}(\sigma^2_t)) (Bradley & Terry, [4]; Christiano et al., [5]), with 10% of pairs held out, and standardized per step; a tie (“can’t tell”) is a target of 12\frac{1}{2}, that is, indifference. Five seeds per label set give single scorers and a five-member ensemble (Lakshminarayanan et al., [22]).

Tasks and labels. Locomotion: HalfCheetah-v5 and Hopper-v5 (Towers et al., [39]), D4RL medium-replay data (Fu et al., 2020) and the Uni-RLHF crowd-sourced labels (Yuan et al., [46]): 2,000 pairs of 200-step segments that agree with the true ranking on 73% (cheetah) and 64% (hopper) of pairs, with 10–15% ties. Manipulation: Meta-World v3 drawer-open, plate-slide and window-open (Yu et al., 2020a) with mocap position control. The cloned policy is trained on 10 scripted demonstrations; the labelled data add 100 rollouts of the cloned policy with action noise from 0 to 1, and 2,000 pairs of 50-step segments are labelled by a noisy Bradley–Terry teacher that agrees with the true ranking on 86–88% of pairs. Smaller budgets use the first KK pairs. The authors also labelled pairs blind (two clips side by side in random order, with “can’t tell”): drawer pairs 0–299 and 200–499 (97.6% and 97.7% agreement with the true ranking on decided pairs, 18% and 13% ties; the two agreed on all 73 shared decided pairs) and plate-slide pairs 0–299 (97.2%, 17%). Episodes last 200 steps on the walkers and 150 on the arm. True return is the environment’s reward (Meta-World’s dense reward on the arm), and on the arm we also count successes. Each cell averages 10 runs (5 scorer seeds × 2 start states; 10 start states for ensembles), and 20 for standard MPPI in the five main settings. Appendix B lists all settings; code and results will be released.

Theory: what the planner can and cannot certify

Fix a state. Let qq give each of the NN proposals weight 1/N1/N (the reference policy of RLHF), JJ and J^\hat{J} be the true and learned plan returns, and e=J^−Je=\hat{J}-J the scorer’s error; labels fix J^\hat{J} only up to scale and offset, so JJ is measured in the scorer’s units. A plan distribution p≪qp\ll q is any reweighting of the proposals, such as MPPI’s weights; its density w=dp/dqw=dp/dq gives plan ii the factor wi=Npiw_i=Np_i by which it is up-weighted (wi=2w_i=2 doubles its share, wi=0w_i=0 drops it). Let Δ(p)=Ep[J]−Eq[J]\Delta(p)=\mathbb{E}_p[J]-\mathbb{E}_q[J] be the true gain of selecting with pp, Δ^(p)\widehat{\Delta}(p) the same under J^\hat{J}, and s(p)=χ2(p∥q)=(1N∑i(wi−1)2)1/2s(p)=\sqrt{\chi^2(p\mathbin\Vert q)}=\left(\frac{1}{N}\sum_i(w_i-1)^2\right)^{1/2} the distance moved:

{#eq:1}

s=0s = 0 is no search, and spreading the weight evenly over the best kk plans gives s=N/k−1s = \sqrt{N/k - 1} (1 for the best half, ≈16\approx16 for the single best of 256). What matters is σe=sd⁡q(e)\sigma_e = \operatorname{sd}_q(e), the spread of the scorer’s mistake across the proposals: an error shared by all of them changes no weights and costs nothing; what hurts is overrating some proposals relative to others. Proofs are in Appendix § 7.

Lemma 1 (Tight change of measure). For every p≪qp \ll q, ∣Δ(p)−Δ^(p)∣≤σes(p)|\Delta(p) - \widehat{\Delta}(p)| \le\sigma_e s(p). For each pp, an error of spread σe\sigma_e that overrates every plan in proportion to its extra weight w−1w - 1 attains Δ(p)=Δ^(p)−σes(p)\Delta(p) = \widehat{\Delta}(p) - \sigma_e s(p).

In words, the gain the scorer reports overstates the true gain by at most how far the planner moves times how much the scorer’s mistakes vary across the proposals, and by exactly that much when the scorer overrates precisely the plans the planner up-weights. This sharpens the change-of-measure bound used for language models [15 a]: it vanishes when the planner does not move, and no smaller bound holds. Against every error of spread at most σ\sigma, the planner can therefore guarantee exactly Wσ(p)=Δ^(p)−σs(p)W_\sigma(p) = \widehat{\Delta}(p) - \sigma s(p), and the best it can do is maximize this.

Theorem 2 (Worst-case-optimal search). (i) If σ≥σq(J^)\sigma\ge\sigma_q(\widehat{J}), no plan distribution guarantees a gain: max⁡pWσ(p)=0\max_p W_\sigma(p) = 0, attained by p=qp = q.

(ii) If σ<σq(J^)\sigma< \sigma_q(\widehat{J}), WσW_\sigma is maximized by χ2\chi^2 MPPI, w=max⁡(0,(J^−c)/λ)w = \max(0,(\widehat{J} - c)/\lambda), at a temperature λ⋆\lambda^\star with λ⋆s(pλ⋆)=σ\lambda^\star s(p_{\lambda^\star}) = \sigma (for σ=0\sigma= 0, the hard-search limit λ→0\lambda\to0), and the maximum is positive. Along the χ2\chi^2 path, λs(pλ)\lambda s(p_\lambda) falls from σq(J^)\sigma_q(\widehat{J}) to 0 as search hardens, and WσW_\sigma rises until λ⋆\lambda^\star and falls after it.

(iii) While no proposal’s weight is clipped to zero, Δ^(pλ)=σq(J^)s\widehat{\Delta}(p_\lambda) = \sigma_q(\widehat{J})s and the true gain is exactly Δ(pλ)=ρqσq(J)s\Delta(p_\lambda) = \rho_q \sigma_q(J)s, where ρq=Corr⁡q(J,J^)\rho_q = \operatorname{Corr}_q(J,\widehat{J}).

The theorem turns the error size into a stopping rule the planner can check from its own weights: harden the search until λ/χ2\lambda/\sqrt{\chi^2} falls to the error spread, and do not search at all when the error spread exceeds the scorer’s own spread over the proposals. The χ2\chi^2 geometry comes from the error model, not a choice of divergence: knowing only the error’s spread makes σeχ2\sigma_e\sqrt{\chi^2} the exact worst case (Theorem 1), while sub-Gaussian errors would give a bound in KL\sqrt{\mathrm{KL}} matched by exponential MPPI, an assumption heavy tails break [19].

Scale. Since JJ is known only up to scale, we put it in the scorer’s units by the rescaling that best matches J^\widehat{J} on the proposals (least squares of J^\widehat{J} on JJ), which leaves an error uncorrelated with JJ and gives σe=σq(J^)1−ρq2\sigma_e = \sigma_q(\widehat{J})\sqrt{1-\rho_q^2}. The threshold of (i) then becomes ρq≤0\rho_q \le0: a positive gain can be certified exactly when the scorer’s Pearson correlation with the true return over the proposals is positive. For jointly Gaussian returns this is the same as ordering a random pair correctly more often than chance; in general it is not, since a scorer can order most pairs correctly and still correlate negatively by badly underrating one high-return plan. As ρq→0+\rho_q \to0^+ the rescaling shrinks JJ, so the certified gain is positive but measured in ever smaller units: the theorem then guarantees the sign of the gain, not its size. In scale-free form, χ2\chi^2 MPPI with λs=σ\lambda s = \sigma does not lose for any true return whose correlation with J^\widehat{J} over the proposals is at least 1−σ2/σq(J^)2\sqrt{1-\sigma^2/\sigma_q(\widehat{J})^2}.

MPPI executes the weighted average of the plans, A‾p=Ep[A]\overline{A}_p = \mathbb{E}_p[A], not a plan drawn from pp. The guarantee carries over up to the spread of the plans the weights keep.

Proposition 3 (The executed plan). If JJ is LL-smooth in the plan on the action box, then J(A‾p)≥Ep[J]−L2Ep∥A−A‾p∥2J(\overline{A}_p) \ge\mathbb{E}_p[J] - \frac{L}{2}\mathbb{E}_p\lVert A - \overline{A}_p\rVert^2, so against every error of spread at most σ\sigma, J(A^p)−Eq[J]≥Wσ(p)−L2Ep∥A−A‾p∥2\mathcal{J}(\widehat{A}_p) - \mathbb{E}_q[J] \ge W_\sigma(p) - \frac{L}{2}\mathbb{E}_p\lVert A - \overline{A}_p\rVert^2. The last term vanishes as the weights concentrate on one plan, and it can be dropped when JJ is concave in the plan.

LL is unknown for contact-rich tasks, so we measure the averaging term J(A‾p)−Ep[J]J(\overline{A}_p) - \mathbb{E}_p[J] directly (Section C): at least −0.05-0.05 standard deviations except on window-open (−0.15-0.15).

Corollary 4 (Hard exponential search). On NN samples with a unique top-scored plan, exponential MPPI with λ→0\lambda\to0 selects it, s→N−1s \to\sqrt{N - 1}, and its guarantee is max⁡J^−EqJ^−σN−1\max\widehat{J} - \mathbb{E}_q\widehat{J} - \sigma\sqrt{N - 1}: the worst error overrates the selected plan by σN−1\sigma\sqrt{N - 1}. If the top score’s margin grows more slowly than N\sqrt{N}, as σq(J^)2log⁡N\sigma_q(\widehat{J})\sqrt{2\log N} does for Gaussian scores, this tends to −∞-\infty as NN grows, for every σ>0\sigma> 0; the guarantee of Theorem 2 is never negative.

How large is σe\sigma_e? For a linear reward its bound shrinks with the label count at a rate set by how far the planner’s proposals vary beyond the labelled pairs (Theorem 7, Section A). The next result locates where along the exponential path true return turns down.

Proposition 5 (Where search starts to hurt). Let pη∝qeηJ^p_\eta\propto q e^{\eta\widehat{J}} with η=1/λ\eta= 1/\lambda, and d(η)=KL⁡(pη∥q)d(\eta) = \sqrt{\operatorname{KL}(p_\eta\|q)}. (i) For every η\eta, ddηEpη[J]=Cov⁡pη(J,J^)\frac{d}{d\eta}\mathbb{E}_{p_\eta}[J] = \operatorname{Cov}_{p_\eta}(J,\widehat{J}) and ddηKL⁡(pη∥q)=ηVar⁡pη(J^)\frac{d}{d\eta}\operatorname{KL}(p_\eta\|q) = \eta\operatorname{Var}_{p_\eta}(\widehat{J}). The true return of a plan falls from the planner’s weights rises exactly while learned and true returns are positively correlated under those weights, and it has a local peak only where that covariance changes sign from positive to negative.

(ii) As η→0\eta\to0, Epη[J]−Eq[J]=2ρqσq(J)d+O(d2)\mathbb{E}_{p_\eta}[J] - \mathbb{E}_q[J] = \sqrt{2}\rho_q\sigma_q(J)d + O(d^2), where ρq=Corr⁡q(J,J^)\rho_q = \operatorname{Corr}_q(J,\widehat{J}) and σq(J)\sigma_q(J) is the spread of true return over the proposals.

*(iii) The executed plan A‾=Ep[A]\overline{A} = \mathbb{E}_p[A] satisfies J(A‾)−J(U)=Ep[J]−Eq[J]−(J(U)−Eq[J])+(J(A‾)−Ep[J])J(\overline{A}) - J(U) = \mathbb{E}_p[J] - \mathbb{E}_q[J] - (J(U) - \mathbb{E}_q[J]) + (J(\overline{A}) - \mathbb{E}_p[J]); on NN samples the last term vanishes as λ→0\lambda\to0, so hard search beats the cloned plan only if the selection gain exceeds a price that does not depend on the scorer.

Part (ii) makes KL\sqrt{\mathrm{KL}} the natural axis, as Gao et al. [11] found for language models: to first order true return changes linearly in it, with a slope set by how well the scorer ranks the proposals, not how well it fits its labels (near q,KL≈s/2q,\sqrt{\mathrm{KL}}\approx s/\sqrt{2}, matching Theorem 2(iii)). Part (i) says that exploitation begins where the scorer stops ranking the planner’s own favourites correctly, possibly long before it ranks typical proposals badly. Part (iii) separates exploitation from a cost of sampling that a perfect scorer also pays; drawing one plan from the weights, as inference-time pessimism does (Huang et al., [14]), gives up the averaging term at every temperature, which is why it does worse than averaging on cheetah (167–265) and the drawer.

Over an episode the planner re-plans at every state it visits, and the per-step terms add up exactly.

Proposition 6 (Closed loop). Let the dynamics be deterministic, VtV_t the cloned policy’s true return from step tt to the end of the episode, and Qt(s,a)Q_t(s,a) the return of taking aa at step tt and following the cloned policy after it. If at each visited state sts_t the planner executes the first action ata_t of Aˉt\bar{A}_t, its episode return RR satisfies R−V0(s0)=∑t<T[Δt−ct+αt+ηt]R-V_0(s_0)=\sum_{t<T}\left[\Delta_t-c_t+\alpha_t+\eta_t\right], where Δt\Delta_t, ctc_t and αt\alpha_t are the selection gain, price of noise and averaging term of Theorem 5(iii) at sts_t, and ηt=Qt(st,at)−Vt(st)−(J(Aˉt)−J(Ut))\eta_t=Q_t(s_t,a_t)-V_t(s_t)-\left(J(\bar{A}_t)-J(U_t)\right) is the horizon gap, by which the HH-step comparison with the cloned plan misstates the long-run advantage of the first action. If at every visited state the weights are worst-case optimal for an error spread of at least that state’s σe\sigma_e, then every Δt≥0\Delta_t\ge0 and R≥V0(s0)−∑t(ct−αt−ηt)R\ge V_0(s_0)-\sum_t(c_t-\alpha_t-\eta_t).

The identity does not make the other terms independent of the scorer, which sets the weights and with them the averaging term and the states visited: with J(a)=1−a2J(a)=1-a^2, cloned action 0 and proposals ±1\pm1, hard search on a scorer that favours +1+1 has zero selection gain and loses only the averaging term. Theorem 2 controls the selection terms, given the error spread on the visited states, which can exceed that on the cloned policy’s; we measure the rest (Section D).

Experiments

Harder search lowers true return

Figure 1 shows the main effect. With the crowd scorer, cheetah’s true return falls from 480 at λ=3\lambda=3 to 262 at λ=0.01\lambda=0.01 (uniform averaging scores 233, the cloned policy run directly 412), and hopper’s from 468 to 272–356, while the perfect scorer improves on cheetah (867 to 1,010). On hopper hard search costs return even with the perfect scorer (559 at λ=3\lambda=3, 342–373 at λ≤1\lambda\le1), so part of the hopper loss is the planner’s own (§ 4.2). On the drawer with 500 labels, true return falls from 707 to 504 and the drawer stays shut in 9 of 20 runs, while the scorer’s own estimate of the executed episode rises twelve-fold (56 to 674; uniform averaging and the cloned policy open it every time). Blind human labels on 300 pairs are the starkest case: the gentlest search matches uniform averaging (851, 20 of 20), and every temperature at or below 0.1 opens the drawer in 0 of 20 runs (201–216); both authors’ labels on 500 pairs fare no better (246 at λ=0.01\lambda=0.01, 0/10). On plate-slide, where the cloned policy already succeeds every time, every scorer from synthetic labels loses ground under harder search, with 500 labels (950 to 436) and with 2,000 (977 to 319). Human labels lose less: one author’s 300 keep 1,066 (10/10) at hard search, against 499 (8/10) with synthetic labels on the same pairs and 886 (8/10) with them in true-return order, and both authors’ 500 fall to 681 (6/10). On the drawer, 2,000 labels are enough: harder search then helps (899 to 948, never failing). Search is not always exploited: on window-open it helps at every budget (295 without search, 375–392 at λ=0.01\lambda=0.01; § 4.3), and button-press is a null case (Section F). Hard search still loses with horizon 10 or 40 and proposal noise 0.25 or 1, except on cheetah with noise 0.25 (Table 2).

Figure 1. True return against MPPI temperature (harder search to the right), mean of 10–20 runs with ±1 standard error over runs. Dotted: uniform averaging (no search); dash-dot: the cloned policy run directly. A perfect scorer (the true reward) keeps or improves its return except on hopper (§ 4.2); a learned scorer loses it, except on window-open, where it gains less. Magenta: 300 blind human labels, exploited faster than 500 synthetic ones; grey: 2,000 labels, safe on the drawer and window-open but not on plate-slide.

CheetahDrawer, 500 labels
noneλ=3\lambda= 30.01noneλ=3\lambda= 30.01
σ=0.5\sigma= 0.5, H=20H = 20 (default)233±78233 \pm78480±38480 \pm38262±25262 \pm25852±1852 \pm1 (10/10)707±41707 \pm41 (18/20)504±74504 \pm74 (11/20)
σ=0.25\sigma= 0.25411±113411 \pm113451±100451 \pm100517±39517 \pm39890±1890 \pm1 (10/10)880±4880 \pm4 (10/10)719±71719 \pm71 (8/10)
σ=1\sigma= 13±23 \pm220±1720 \pm173±243 \pm24667±3667 \pm3 (10/10)125±4125 \pm4 (0/10)122±2122 \pm2 (0/10)
H=10H = 10278±91278 \pm91481±54481 \pm54328±36328 \pm36852±1852 \pm1 (10/10)833±8833 \pm8 (10/10)375±80375 \pm80 (4/10)
H=40H = 40157±76157 \pm76443±75443 \pm75173±23173 \pm23851±2851 \pm2 (10/10)508±76508 \pm76 (5/10)369±76369 \pm76 (3/10)

Table 2. Planner settings. True return of uniform averaging (no search) and of standard MPPI with the learned scorer at λ=3\lambda= 3 and 0.01, for proposal noise σ\sigma and horizon HH; mean ±\pm standard error over 10 runs (default row: all runs of the main experiments); successes in parentheses.

The human-label result is not a labelling accident. Those labels agree with the true ranking on 97.6% of decided pairs, against 86% for the synthetic teacher on the same pairs, and give the best held-out accuracy (0.97) and per-step correlation with the true reward (0.91) of any scorer we trained. High agreement on decided pairs does not make a safer planning target. Controls on the same 300 pairs point to the 55 pairs the person could not tell apart. At λ=0.01\lambda=0.01, ordering every pair by true return gives 640 (6/10), and so does dropping 55 random pairs (672, 6/10); but with only the decided pairs in true order the drawer never opens, whether the ties are kept as indifference (268) or dropped (233; 190 with the person’s own answers). What the planner needs is the order of those close pairs, half of the 63 in which neither clip opens the drawer. In recorded episodes (Figure 2) the arm parks in an unvisited corner the scorer rates very highly (synthetic labels) or pulls about 10 cm short of the handle (human labels). A linear fit of each scorer’s per-step output on four state features shows why: the human-label scorer barely penalizes the hand–handle distance (standardized weight −0.16, against −0.40 to −0.49 with synthetic labels) and weights pulling (0.29) and the drawer’s opening (0.63) most.

Figure 2. One drawer start under three planners (true returns). Uniform averaging opens the drawer. Standard MPPI (λ=0.01\lambda= 0.01) on the 500-label scorer lifts the hand into a part of the workspace the data never visited; on the human-label scorer (λ=0.1\lambda= 0.1) it pulls in front of the handle without touching it.

True return follows the distance moved

Figure 3a plots every single-scorer MPPI cell against d=KLd=\sqrt{\mathrm{KL}}. A two-parameter curve ΔJ=ad−bd2\Delta J=ad-bd^2 through uniform averaging (d=0d=0) fits each task well (R2=0.79R^2=0.79–0.99 over 7–11 cells), and the sign of the initial slope aa is clear on every task (90% hierarchical bootstrap intervals, resampling scorer seeds and then runs): cheetah 306 to 664, hopper 163 to 330, drawer −385-385 to −99-99, plate-slide −479-479 to −218-218, window-open 6 to 112). A curve fitted on the exponential cells alone predicts the χ2\chi^2 cells with a mean absolute error of 38% of their spread on cheetah, 57% on hopper, 23% on the drawer (16% with human labels), 22% on plate-slide and 29% on window-open: at equal distance moved the two rules give similar true return, an approximate collapse within each task for the two rules we test, not a law. The advantage of χ2\chi^2 weights at a fixed temperature (§ 4.4) is therefore mostly that they move less. The walkers first gain and then lose (peaks near d=1.2d = 1.2 and 1.0); the drawer and plate-slide, where the cloned policy is near the optimum, lose from the start, and window-open gains throughout.

Figure 3. (a) Return gained over uniform averaging, divided by the perfect scorer’s best, against the per-step distance KL\sqrt{\mathrm{KL}} between MPPI’s weights and its proposals. Circles: exponential weights; squares: χ2\chi^2 weights; each point is one temperature (10–20 runs); curves are least-squares fits of ad−bd2a d-b d^2; faint dashed: perfect scorer, whose highest point is 1−(averaging’s return÷ its best)1-(\text{averaging’s return}\div\text{ its best}), e.g. 0.77 on cheetah and 0.08 on plate-slide. Arm tasks use 500 labels. (b) The same on the drawer for five label budgets and the human labels. (c) Best-of-NN with a learned verifier on the drawer (solid) and plate-slide (dashed), relative to the clean cloned action (dash-dot line); N=1N=1 (open markers) is one noisy action. Every point is 50 runs on the same ten starts (10 for the ensemble).

The dashed curves (the perfect scorer at the same distances) separate exploitation from the planner’s own cost: on cheetah, the drawer and plate-slide the gap opens from the first step and reaches 590–750 points at hard search, and on window-open 175. On hopper the crowd scorer is ahead of the true reward up to d=1.7d = 1.7, possibly because its 200-step segments reward progress the 20-step return misses, and behind it beyond d=1.9d = 1.9 (−17-17 to −101-101).

On the drawer (Figure 3b) the fitted initial slope aa grows steadily with the label count: −507-507, −382-382, −250-250, −2-2 and +99+99 for 200, 300, 500, 1,000 and 2,000 labels, roughly linear in log⁡n\log n: +630+630 per decade (90% hierarchical interval +514+514 to +766+766), crossing zero near 1,200 labels (1,010–1,420). The human labels sit below the synthetic labels on the same 300 pairs on average (a=−460a = -460, interval −642-642 to −236-236, against −382-382), though the intervals overlap. With five budgets, we read the log-linear form as a summary, not a law to extrapolate. On plate-slide, 2,000 labels do not change the sign (a=−342a = -342 at 500, −360-360 at 2,000).

Two ways to be exploited, and why held-out checks do not warn

Figure 4b separates two patterns by the sign of the change in the scorer’s own estimate of the episode the robot actually executes. On the drawer with synthetic labels the estimate rises sharply as true return falls, consistent with the planner finding states the scorer overrates, the extremal case of Manheim & Garrabrant [26]. On plate-slide and with human labels the estimate falls (from 27–43 to between −43-43 and −90-90), consistent with chasing plans that look good at planning time and do not hold up once the planner replans, the optimizer’s curse (Smith & Winkler [35]) in closed loop. In the scorer’s observations hard search leaves the labelled data on all three (51–60% of visited states, none without search), but leaving it is not sufficient: on window-open 60% of states are off it even without search, and search still helps (Section D). The two patterns respond to different fixes (§ 4.4).

Why do the usual checks not warn? Table 3 (Section C) compares them with what Theorem 5 says matters, on the planner’s 256 proposals along the cloned policy’s episodes, scored by the learned scorer and the true reward. The initial slope predicted by Theorem 5(ii) matches the measured one to within 6% in eleven of the thirteen settings (24% and 12%, about one standard error, with 200 drawer labels and with human labels). The correlation ρq\rho_q that sets it is unrelated to the held-out checks: the 500-label drawer scorer has held-out accuracy 0.88 and per-step correlation 0.88 on the labelled data, but ranks the planner’s proposals with ρq=0.10\rho_q=0.10; the hopper scorer has a per-step correlation of only 0.08 on the data, yet ranks proposals with ρq=0.76\rho_q=0.76. Part (iii) explains why hard search loses to the cloned plan even when selection still helps: perturbing the cloned plan costs 0.7–2.2 standard deviations of true plan return, and the scorers recover only a fraction of that by selection. On window-open it costs only 0.33 and selection recovers 0.7–1.0, which is why search helps there. A scorer can be checked only where it will be optimized: on the planner’s own proposals and, after the first step, on states the cloned policy never visited. Whole episodes bear this out (Theorem 6; Section D). On the states hard search visits, the scorer ranks the proposals worse than on the cloned policy’s states (ρ\rho from 0.10 to −0.11-0.11 on the drawer, −0.10-0.10 to −0.31-0.31 with human labels, 0.45 to −0.07-0.07 on plate-slide, 0.76 to 0.51 on hopper; near zero on both on cheetah) and stays positive on window-open (0.35 to 0.22). Summed over a drawer episode, the selection gains are −151-151 with 500 labels and −373-373 with human labels, against +632+632 with the perfect scorer. On plate-slide they stay positive (+113+113) and the loss is in the horizon gap (−497-497; −237-237 for the perfect scorer on the drawer): the executed plans’ 20-step advantage over the cloned plan overstates their long-run effect.

Held-outOn the planner’s proposals
acc.corr.ρq\rho_qnoiseStandard MPPI, λ=0.01\lambda= 0.01 mean Δ\DeltaStandard MPPI, λ=0.01\lambda= 0.01 worst Δ\DeltaTheorem 2 mean Δ\DeltaTheorem 2 κ∗\kappa^*
Cheetah0.750.170.08±0.010.08 \pm0.011.590.18±0.070.18 \pm0.07−3.03±0.41-3.03 \pm0.410.061.4
Hopper0.770.080.66±0.040.66 \pm0.040.670.22±0.190.22 \pm0.19−2.47±0.33-2.47 \pm0.330.531.2
Drawer 2000.840.79−0.00±0.03-0.00 \pm0.030.91−0.64±0.18-0.64 \pm0.18−7.03±0.36-7.03 \pm0.360.051.8
Drawer 3000.840.85−0.01±0.02-0.01 \pm0.020.91−0.21±0.06-0.21 \pm0.06−5.92±0.57-5.92 \pm0.570.071.9
Drawer 5000.880.880.07±0.060.07 \pm0.060.910.16±0.120.16 \pm0.12−5.06±0.51-5.06 \pm0.510.121.4
Drawer 1k0.850.830.06±0.020.06 \pm0.020.910.12±0.080.12 \pm0.08−5.06±0.50-5.06 \pm0.500.111.9
Drawer 2k0.850.810.17±0.010.17 \pm0.010.910.43±0.070.43 \pm0.07−4.12±0.40-4.12 \pm0.400.232.0
Drawer human0.970.91−0.17±0.02-0.17 \pm0.020.91−0.16±0.11-0.16 \pm0.11−6.64±0.90-6.64 \pm0.900.052.8
Drawer 500, half on planner0.840.920.13±0.040.13 \pm0.040.980.29±0.080.29 \pm0.08−5.66±0.45-5.66 \pm0.450.261.1
Plate 5000.840.930.42±0.030.42 \pm0.032.160.85±0.030.85 \pm0.03−4.87±1.21-4.87 \pm1.210.422.0
Plate 2k0.860.930.36±0.020.36 \pm0.022.160.98±0.100.98 \pm0.10−3.65±1.18-3.65 \pm1.180.392.0
Window 5000.860.850.36±0.020.36 \pm0.020.330.98±0.210.98 \pm0.21−8.46±2.61-8.46 \pm2.610.391.0
Window 2k0.850.910.23±0.020.23 \pm0.020.330.72±0.120.72 \pm0.12−5.68±1.07-5.68 \pm1.070.222.0

Table 3. What the scorer knows about the planner’s own proposals. Held-out: label accuracy on held-out pairs and per-step correlation with the true reward on the labelled data. On the proposals, along the cloned policy’s episodes (150–200 states): correlation ρq\rho_q between learned and true plan returns over the 256 proposals; the price of the proposal noise J(U)−EqJJ(U) - \mathbb{E}_qJ; the selection gain Δ\Delta of standard MPPI at λ=0.01\lambda= 0.01, on average and at the worst state; and the average gain of the worst-case-optimal rule of Theorem 2 on the five-scorer mean given the true error spread, whose gain at the worst state is 0.00 in every row. Gains in units of the proposals’ spread σq(J)\sigma_q(J). Mean ±\pm standard error over 5 scorer seeds. κ∗\kappa^*: median ratio of the true error spread to the ensemble’s disagreement.

The same proposals test Theorem 2. Given the true error spread, the worst-case-optimal rule does not lose at any of the 2,042 states and gains 0.05–0.53 standard deviations on average; standard MPPI at hard search gains more on average in ten of thirteen settings but loses 1.6–8.0 at its worst state, which a closed-loop planner has many chances to meet. The unknown is the error spread, which the ensemble’s disagreement underestimates by a factor κ∗\kappa^* of 1.0–2.8, most with human labels: scorers trained on the same labels share their mistakes. A practical version calibrates it from pairs of the planner’s own proposals ranked by true return (an oracle; in deployment, a person’s preferences): σ=σq(J^)1−ρ^+2\sigma= \sigma_q(\hat{J})\sqrt{1-\hat{\rho}_+^2}, with ρ^+=max⁡(0,ρ^)\hat{\rho}_+ = \max(0,\hat{\rho}) and ρ^=sin⁡(π(c−12))\hat{\rho} = \sin\left(\pi\left(c-\frac{1}{2}\right)\right) from the share cc of pairs the scorer orders correctly (Kruskal, [18]), so no search when ρ^≤0\hat{\rho} \le0. With 100 pairs it loses at most 0.22 standard deviations at its worst state in twelve of thirteen settings and keeps 47–93% of the true-spread rule’s gain where that gain exceeds 0.2; a nominal 95% lower bound on cc cuts the loss to 0.05 but keeps only 12–77% of the gain. By Theorem 8 (Section E) the rule cannot lose at a state whose correlation is at least ρ^\hat{\rho}; mistakes the labeller shares with the scorer, sampling noise and pooling (19–59% of states fall below the pooled value) can each put ρ^\hat{\rho} above a state’s own, so the rule is an empirical safeguard, not a per-state certificate.

What helps, and where

Table 1 and Figure 4c compare the fixes at hard search; none wins everywhere. On the walkers, χ2\chi^2 weights on the mean of five scorers do best (cheetah 485, hopper 448, against 262 and 356 for standard MPPI), and the cautious ensemble (mean minus one standard deviation; Coste et al., [7]) helps less (295 and 374). On the drawer with synthetic labels the ranking reverses: the cautious ensemble restores 801 (10/10) while χ2\chi^2 weights do not help (480), consistent with the two patterns above: where the planner reaches states the scorers overrate they disagree and the penalty finds them; where it chases noise they agree, and only gentler search helps. Automatic temperatures (0.5 times the disagreement) are the most even (cheetah 521, hopper 569, drawer 650, plate 662); a disagreement guard and the worst-case-optimal rule with κ=2\kappa= 2 times the disagreement help on the drawer (729, 821) but not on the walkers. With human labels every fix that searches hard fails (at best 382, 2/10).

Figure 4. (a) Drawer, true return against label budget at gentle and hard search; the cautious ensemble rescues 500 and 1,000 labels but not 200 or the human labels. Green: 500 labels, half on the planner’s own trajectories (§ 4.4). (b) Change in the scorer’s own per-step estimate of the executed episode between λ=3\lambda=3 and λ=0.01\lambda=0.01, against the ratio of true returns: the estimate rises (right), consistent with extremal exploitation, or falls (left), consistent with noise chasing; diamonds: human labels. (c) Each fix at λ=0.01\lambda=0.01, relative to standard MPPI at λ=3\lambda=3.

CheetahHopperDrawer 500Drawer humanPlate 500
Cloned policy, run directly412 ± 114502 ± 32898 ± 1 (10/10)898 ± 1 (10/10)1111 ± 5 (10/10)
Uniform averaging (no search)233 ± 78418 ± 38852 ± 1 (10/10)852 ± 1 (10/10)1051 ± 6 (10/10)
Perfect scorer, λ=0.01\lambda= 0.011010 ± 17373 ± 341090 ± 7 (10/10)1090 ± 7 (10/10)1139 ± 7 (10/10)
Learned scorer, gentle (λ=3\lambda= 3)480 ± 38†468 ± 31†707 ± 41† (18/20)851 ± 4† (20/20)950 ± 21† (20/20)
Learned scorer, hard search (λ=0.01\lambda= 0.01)
Standard (exp. weights)262 ± 25†356 ± 39†504 ± 74† (11/20)216 ± 10† (0/20)436 ± 65† (15/20)
χ2\chi^2 weights364 ± 23303 ± 49480 ± 96 (5/10)211 ± 9 (0/10)448 ± 80 (7/10)
Cautious ensemble (mean − sd)295 ± 30374 ± 30801 ± 51 (10/10)261 ± 42 (1/10)406 ± 68 (8/10)
χ2\chi^2 on ensemble mean485 ± 25448 ± 56538 ± 85 (6/10)219 ± 13 (0/10)360 ± 75 (7/10)
Automatic guard409 ± 31361 ± 60729 ± 77 (9/10)222 ± 17 (0/10)335 ± 72 (6/10)
Learned scorer, adaptive temperature or a different optimizer
Automatic λ\lambda (χ2\chi^2, c=0.5c = 0.5)521 ± 20569 ± 1650 ± 60 (7/10)262 ± 23 (0/10)662 ± 61 (10/10)
Worst-case-optimal λ\lambda (Thm. 2, κ=1\kappa= 1)460 ± 52569 ± 2575 ± 76 (8/10)231 ± 13 (0/10)699 ± 58 (10/10)
Worst-case-optimal λ\lambda (Thm. 2, κ=2\kappa= 2)253 ± 84426 ± 32821 ± 8 (10/10)382 ± 75 (2/10)883 ± 47 (10/10)
Worst-case-optimal λ\lambda, 100 oracle pairs357 ± 86‡539 ± 12‡840 ± 16‡ (50/50)852 ± 0‡ (50/50)959 ± 86‡ (50/50)
CEM, cautious472 ± 9337 ± 66425 ± 88 (5/10)136 ± 2 (0/10)79 ± 1 (0/10)
CEM, mean of five460 ± 17∗139 ± 53∗370 ± 90 (3/10)–73 ± 1 (0/10)
CEM, optimistic (UBP2-inspired)423 ± 19195 ± 43260 ± 88 (2/10)129 ± 2 (0/10)68 ± 1 (0/10)

Table 1. True return at hard search, mean ± standard error over 10 runs per cell († 20, ∗ 5); successes over runs in parentheses on the arm. ‡ Mean ± standard deviation across five independent 100-pair calibration sets ranked by true return (an oracle), drawn on start states the evaluation never uses, each calibrating its own controller (Table 6). Bold: best learned-scorer method per column (the calibrated temperature on the arm, the automatic one on the walkers).

Calibrated from 100 pairs of proposals ranked by true return, an oracle check a deployed robot lacks, the rule is the safest fix (Table 1 and Table 6). Each of five independent calibration sets per task, drawn on start states the evaluation never uses, calibrates its own controller: it scores 818–852 on the drawer and 806–1,013 on plate-slide, succeeding in all 50 runs of each, against 898 and 1,111 for the cloned policy run directly and 611 (8/10) and 351 (6/10) for hard search on the same five scorers. With human labels every set declines to search (852, uniform averaging). It gains on hopper with every set (523–550, against 502 for the cloned policy), averages 357 on cheetah (264–445; uniform averaging 233, cloned policy 412) and pays for caution on window-open (309–318, against 392 for standard MPPI and 372).

CheetahHopperDrawer 500Drawer humanPlate 500Window 500
Cloned policy, run directly412±114412 \pm114502±32502 \pm32898±1898 \pm1 (10/10)898±1898 \pm1 (10/10)1111±51111 \pm5 (10/10)372±13372 \pm13 (10/10)
Uniform averaging (no search)233±78233 \pm78418±38418 \pm38852±1852 \pm1 (10/10)852±1852 \pm1 (10/10)1051±61051 \pm6 (10/10)295±9295 \pm9 (10/10)
Standard MPPI, single scorers262±25262 \pm25356±39356 \pm39504±74504 \pm74 (11/20)216±10216 \pm10 (0/20)436±65436 \pm65 (15/20)392±14392 \pm14 (10/10)
Standard MPPI, five-scorer mean296±33296 \pm33339±66339 \pm66611±89611 \pm89 (8/10)206±13206 \pm13 (0/10)351±86351 \pm86 (6/10)425±16425 \pm16 (10/10)
Pooled, true order, point451±54451 \pm54538±30538 \pm30835±6835 \pm6 (10/10)852±1852 \pm1 (10/10)987±18987 \pm18 (10/10)311±9311 \pm9 (10/10)
κ\kappa0.9980.7130.9981.0000.8720.876
Pooled, true order, lower bound264±88264 \pm88519±35519 \pm35852±1852 \pm1 (10/10)852±1852 \pm1 (10/10)1036±61036 \pm6 (10/10)306±9306 \pm9 (10/10)
κ\kappa1.0000.8561.0001.0000.9770.970
Pooled, 85% teacher, point451±54451 \pm54532±31532 \pm31835±6835 \pm6 (10/10)852±1852 \pm1 (10/10)1007±131007 \pm13 (10/10)310±9310 \pm9 (10/10)
κ\kappa0.9980.7710.9981.0000.9050.905
Five independent sets, point307±52307 \pm52, 264551±23551 \pm23, 517838±13838 \pm13, 824 (50/50)851±3851 \pm3, 846 (50/50)1009±61009 \pm6, 999 (50/50)312±5312 \pm5, 308 (50/50)
κ\kappa0.976–1.0000.588–0.7290.969–1.0000.996–1.0000.891–0.9180.771–0.941
Five independent sets, lower bound264±0264 \pm0, 264538±22538 \pm22, 500852±0852 \pm0, 852 (50/50)852±0852 \pm0, 852 (50/50)1039±21039 \pm2, 1036 (50/50)306±2306 \pm2, 303 (50/50)
κ\kappa1.0000.757–0.8731.0001.0000.977–0.9890.903–0.996
Five held-out sets, point357±86357 \pm86, 264539±12539 \pm12, 523840±16840 \pm16, 818 (50/50)852±0852 \pm0, 852 (50/50)959±86959 \pm86, 806 (50/50)313±3313 \pm3, 309 (50/50)
κ\kappa0.960–1.0000.562–0.7710.951–1.0001.0000.750–0.9180.809–0.918
Five held-out sets, lower bound264±0264 \pm0, 264509±30509 \pm30, 472848±8848 \pm8, 835852±0852 \pm0, 852 (50/50)1028±171028 \pm17, 998 (50/50)306±2306 \pm2, 304 (50/50)
κ\kappa1.0000.734–0.9030.998–1.0001.0000.889–0.9890.930–0.989

Table 6. The calibrated rule in closed loop on the same ten starts per column. Top: references, and hard search (λ=0.01\lambda= 0.01) with single scorers (20 runs) and on the five-scorer mean that the calibrated rule uses. Pooled: the median ρ^\hat{\rho} over 50 draws of 100 proposal pairs ranked by true return (point estimate and one-sided Wilson lower bound at nominal 95%) or by a noisy teacher that agrees with the true order on 85% of pairs. Independent sets: five disjoint sets of exactly 100 pairs, each calibrating its own controller; mean ±\pm standard deviation across sets, then the worst set. Held-out sets: the same, with the pairs drawn on start states 10–19, which the evaluation never uses. κ=1−ρ^2\kappa= \sqrt{1-\hat{\rho}^{2}} is the error spread as a share of the scorer’s spread over the proposals, and κ=1\kappa= 1 means no search (uniform averaging). Otherwise mean ±\pm standard error over runs; successes over runs in parentheses.

Calibrating from people. Calibrated from one author’s comparisons of 100 such pairs, the rule trusts the scorer trained on that author’s labels (ρ^=0.63;−0.29\hat{\rho} = 0.63; -0.29 by true return) and opens the drawer in 5/10 runs (574); the other author’s comparisons (ρ^=0.16\hat{\rho} = 0.16) leave it barely searching, at 845 (10/10). Both people called about 70% of pairs ties, which inflates the concordance on the rest; corrected for ties, the first author’s ρ^\hat{\rho} falls to 0.36 and the rule scores 822 (10/10; Table 7). Calibration against the person who labelled the scorer cannot detect a bias they share; an independent evaluator, the tie correction or the lower bound limits the damage.

Calibration pairs ranked bydecided pairsρ^\hat{\rho} (lower bound) as measuredρ^\hat{\rho} (lower bound) ties correctedreturn, as measured pointreturn, as measured lower boundreturn, ties corrected pointreturn, ties corrected lower bound
Oracle (true return), five sets100 of 100−0.45-0.45 to −0.09-0.09(no ties)852 (10/10)852 (10/10)––
Author whose labels trained the scorer32 of 1000.63 (0.23)0.36 (0.12)574 (5/10)839 (10/10)822 (10/10)846 (10/10)
The other author29 of 1000.16 (−0.30-0.30)0.08 (−0.16-0.16)845 (10/10)852 (10/10)846 (10/10)852 (10/10)
Both authors pooled61 of 2000.42 (0.11)0.23 (0.06)812 (10/10)846 (10/10)838 (10/10)846 (10/10)

Table 7. The calibrated rule on the drawer scorer trained on one author’s clip labels (ρq=−0.18\rho_q = -0.18 on the planner’s proposals, Table 5), by who ranks the 100 calibration pairs. ρ^\hat{\rho} with its one-sided Wilson lower bound in parentheses, as measured on the decided pairs and corrected for can’t-tell answers; then the true return (successes over 10 runs) of the rule at each estimate. Uniform averaging scores 852 (10/10) and the cloned policy 898.

A UBP2-inspired optimistic CEM planner (the CEM component of Nabail et al. [28] on the five-scorer mean plus one standard deviation) is the strongest optimizer with the perfect scorer (cheetah 1,244, plate-slide 1,165) and the most exploited on the arm (plate-slide 68–79 with no successes, drawer 260–425, human labels 129–136). The collapse persists without optimism (CEM on the plain mean: drawer 370, plate-slide 73) and at matched budget (MPPI with 768 rollouts: 460, 303), though we do not isolate elite selection.

Labelling where the planner goes.

The scorer is exploited where its labels did not reach, so, as active preference learning for MPC does (El Hasnaouy et al., [9]), we spend half of a 500-label drawer budget on the planner itself: 250 new pairs from MPPI episodes planned with a scorer trained on the first 250 offline pairs (Section B). Hard search then scores 834 (8/10) instead of 504 (11/20), and the cautious ensemble 975 (10/10), level with the cloned policy (898; Figure 4a); the distance curve’s initial slope turns from −250-250 to +33+33.

Best-of-NN with a learned verifier

As supporting evidence beyond MPPI, we test best-of-NN action selection (Kwok et al., [20]) with a state-based verifier trained on 500 or 2,000 synthetic pairs over perturbed actions (Section G). The NN candidates are noisy copies of the cloned action, so the reference is a noisy action: 755 on the drawer (44/50 successes) and 878 on plate-slide (44/50) on the verifiers’ ten starts, against 898 and 1,111 for the clean action, which a perfect verifier keeps for every NN (Figure 3c). With 500 pairs, N=4N = 4 is level with one noisy action on the drawer within noise (728, though 34/50 successes) and below it on plate-slide (723, 26/50); from N=16N = 16 both are well below (530, 375), falling to 388 (14/50) and 143 (0/50) at N=1,024N = 1{,}024. With 2,000 pairs selection beats one noisy action at every NN (drawer 817–895, plate 977–1,061).

Related work

Reward overoptimization. Gao et al. [11] fitted how true quality falls with KL\sqrt{\mathrm{KL}} from the reference policy under best-of-nn and RL against a learned reward model. Ensembles (Coste et al., [7]; Eisenstein et al., [8]; Zhai et al., [47]), weight averaging (Ramé et al., [31]) and constraints

(Moskovitz et al., [27]) mitigate it. Huang et al. [15] showed that χ2\chi^2 regularization gives guarantees KL regularization cannot, and Huang et al. [14] analyzed best-of-NN and proposed inference-time pessimism; our Theorem 1 centers and tightens their change-of-measure bound, and Theorem 2 turns it into a temperature MPPI can check from its own weights and a no-search threshold. Theorem 7 uses the Bradley–Terry estimation bound of Zhu et al. [48]; Aouad et al. [1] derive label-count scaling for language models. Skalse et al. [34]; Pan et al. [29]; Laidlaw et al. [21] define reward hacking. These works study language models or RL policies, not a planner in closed loop.

Preference-based reward learning (Christiano et al., [5]; Sadigh et al., [33]; Lee et al., [23]; Hejna & Sadigh, [13]) is known to pick up spurious features (Tien et al., [37]); Uni-RLHF (Yuan et al., [46]) provides the crowd labels we use. Learned preference rewards are now optimized directly by MPC (Krupa et al., [17]; El Hansaouy et al., 2026) and by CEM-based planning in UBP2 (Nabail et al., [28]), which plans optimistically to collect labels; we study such planners at deployment.

Model-based RL guards against exploiting learned dynamics with ensembles, penalties (Chua et al., [6]; Yu et al., 2020b; Kidambi et al., [16]) or policy constraints (Hansen et al., [12]; Lin et al., [24]); imperfect world models are exploitable (Bhamidipaty et al., [3]); see also conservative MPPI (Yoon & Kim, [43]). MPPI is KL-regularized by construction (Williams et al., [42]); Tsallis-divergence MPPI (Wang et al., [40]) gives the χ2\chi^2 rule. CriticHack (Luo et al., [25]) evaluates visual reward models under policy optimization.

Limitations and conclusion

Our tasks are simulated, our scorers are small MLPs on state, the human labels come from the two authors on two tasks, and most cells have 10 runs. Planning uses the true simulator, which isolates the reward but omits dynamics-model errors. Most calibration pairs are ranked by true return or a simulated labeller, an oracle a deployed robot lacks, apart from two people’s 100 comparisons; the best-of-NN verifier uses synthetic labels. The guarantee of Theorem 6 covers only the selection terms, and the label-count corollary assumes a linear reward. Window-open, button-press and two Adroit hand tasks are null or limit cases (Section F).

For practitioners, a planner’s temperature, sample count and elite fraction are an optimization budget against the scorer, set by its quality on the planner’s own proposals. A calibrated temperature was the safest default tested, given an evaluator independent of the labels; labelling where the planner goes was the most effective fix.

Impact Statement

This work shows that robots which optimize rewards learned from human preferences can behave worse than the policy they start from, without warning from standard validation, and identifies settings and checks that reduce that risk. The intended effect is safer deployment of learned objectives in robotics. The experiments are in simulation and involve no human subjects beyond the two authors’ own labels.

Use of Generative AI

During the preparation of this work the authors used Claude (Anthropic) to search the literature, to write and run the experiment code, and to draft and edit the text. The authors reviewed and edited the content as needed and take full responsibility for it. The code includes a script that recomputes every number in the paper from the stored result files.

Proofs

Proof of Theorem 1. With w=dp/dqw = dp/dq, Δ^(p)−Δ(p)=Ep[e]−Eq[e]=Eq[(w−1)(e−Eqe)]\widehat{\Delta}(p) - \Delta(p) = \mathbb{E}_p[e] - \mathbb{E}_q[e] = \mathbb{E}_q[(w - 1)(e - \mathbb{E}_q e)], which by Cauchy–Schwarz under qq is at most ∥w−1∥L2(q)sdq(e)=σes(p)\lVert w - 1\rVert_{L^2(q)}s_{d_q}(e) = \sigma_e s(p) in absolute value. For p≠qp \ne q, the error e=σe(w−1)/s(p)e = \sigma_e(w - 1)/s(p) has spread σe\sigma_e and attains the bound.

Proof of Theorem 2. (i) By Cauchy–Schwarz, Δ^(p)=Eq[(w−1)(J^−EqJ^)]≤s(p)σq(J^)\widehat{\Delta}(p) = \mathbb{E}_q[(w - 1)(\widehat{J} - \mathbb{E}_q\widehat{J})] \le s(p)\sigma_q(\widehat{J}), so Wσ(p)≤(σq(J^)−σ)s(p)≤0=Wσ(q)W_\sigma(p) \le(\sigma_q(\widehat{J}) - \sigma)s(p) \le0 = W_\sigma(q). (ii) Wσ(p)=Eq[wJ^]−Eq[J^]−σ∥w−1∥L2(q)W_\sigma(p) = \mathbb{E}_q[w\widehat{J}] - \mathbb{E}_q[\widehat{J}] - \sigma\lVert w - 1\rVert_{L^2(q)} is concave in ww on the convex set {w≥0,Eqw=1}\{w \ge0, \mathbb{E}_q w = 1\}, so a point that satisfies the Karush–Kuhn–Tucker conditions is a global maximizer. At w≠1w \ne1 the gradient of ∥w−1∥\lVert w - 1\rVert is (w−1)/s(w - 1)/s; with a multiplier ν\nu for Eqw=1\mathbb{E}_q w = 1 and multipliers κ≥0\kappa\ge0 with κw=0\kappa w = 0 for w≥0w \ge0, stationarity reads J^−σ(w−1)/s−ν+κ=0\widehat{J} - \sigma(w - 1)/s - \nu+ \kappa= 0, whose solution is w=max⁡(0,(J^−c)/λ)w = \max(0,(\widehat{J} - c)/\lambda) with λ=σ/s\lambda= \sigma/s and cc set by normalization: the χ2\chi^2 MPPI weights at a temperature with λs(pλ)=σ\lambda s(p_\lambda) = \sigma. Conversely, the χ2\chi^2 weights at any λ\lambda satisfy these conditions with σ=λs(pλ)\sigma= \lambda s(p_\lambda), so pλp_\lambda maximizes Wλs(pλ)W_{\lambda s(p_\lambda)}. Next, pλp_\lambda maximizes Eq[wJ^]−σ2Eq[(w−1)2]\mathbb{E}_q[w\widehat{J}] - \frac{\sigma}{2}\mathbb{E}_q[(w - 1)^2], so it maximizes Δ^\widehat{\Delta} among all pp with s(p)≤s(pλ)s(p) \le s(p_\lambda); call this maximum g(s)g(s). gg is concave: if w1,w2w_1,w_2 attain g(s1),g(s2)g(s_1),g(s_2), the mixture tw1+(1−t)w2tw_1 + (1-t)w_2 is feasible at ts1+(1−t)s2ts_1 + (1-t)s_2 by the triangle inequality and has Δ^=tg(s1)+(1−t)g(s2)\widehat{\Delta} = tg(s_1) + (1-t)g(s_2). Along the path Wσ=g(s)−σsW_\sigma= g(s) - \sigma s is therefore concave in ss, so it rises to its maximizer and falls after, and the maximizer of g(s)−σsg(s) - \sigma s moves to larger ss as σ\sigma falls, so λs(pλ)\lambda s(p_\lambda) decreases as ss grows. On NN samples, for λ≥EqJ^−min⁡J^\lambda\ge\mathbb{E}_q\widehat{J} - \min\widehat{J} no weight is clipped, w=1+(J^−EqJ^)/λw = 1 + (\widehat{J} - \mathbb{E}_q\widehat{J})/\lambda, s=σq(J^)/λs = \sigma_q(\widehat{J})/\lambda and λs=σq(J^)\lambda s = \sigma_q(\widehat{J}); as λ→0\lambda\to0, λs→0\lambda s \to0 because s≤N−1s \le\sqrt{N-1}. The product is continuous in λ\lambda, so λ⋆\lambda^\star exists for σ>0\sigma> 0; for σ=0\sigma= 0, W0=Δ^W_0 = \widehat{\Delta} is maximized by putting all weight on the top-scored plans, the limit of the path as λ→0\lambda\to0. In both cases and Wσ(pλ⋆)≥Wσ(pλ)=(σq(J^)−σ)s>0W_\sigma(p_{\lambda^\star}) \ge W_\sigma(p_\lambda) = (\sigma_q(\widehat{J}) - \sigma)s > 0 for any unclipped λ\lambda. (iii) Unclipped, w−1=(J^−EqJ^)/λw - 1 = (\widehat{J} - \mathbb{E}_q\widehat{J})/\lambda, so Δ^=Var⁡q(J^)/λ=σq(J^)s\widehat{\Delta} = \operatorname{Var}_q(\widehat{J})/\lambda= \sigma_q(\widehat{J})s and Δ=Cov⁡q(J,J^)/λ=ρqσq(J)s\Delta= \operatorname{Cov}_q(J,\widehat{J})/\lambda= \rho_q\sigma_q(J)s.

Proof of Theorem 3. A‾p\overline{A}_p lies in the action box, which is convex. By LL-smoothness, for every plan AA, J(A)≤J(A‾p)+∇J(A‾p)⊤(A−A‾p)+L2∥A−A‾p∥2J(A) \le J(\overline{A}_p) + \nabla J(\overline{A}_p)^\top(A - \overline{A}_p) + \frac{L}{2}\lVert A - \overline{A}_p\rVert^2. Taking Ep\mathbb{E}_p, the linear term vanishes, which gives the first inequality; the second follows from Theorem 1. For concave JJ, Jensen’s inequality gives J(A‾p)≥Ep[J]J(\overline{A}_p) \ge\mathbb{E}_p[J] directly.

Proof of Theorem 4. As λ→0\lambda\to0 the exponential weights tend to a point mass on the top plan i⋆i^\star: w=N1i⋆w = N\mathbb{1}_{i^\star}, s2=Eq[w2]−1=N−1s^2 = \mathbb{E}_q[w^2] - 1 = N - 1, Δ^=J^(i⋆)−EqJ^\widehat{\Delta} = \widehat{J}(i^\star) - \mathbb{E}_q\widehat{J}. By Theorem 1 the worst error is e−Eqe=σ(w−1)/N−1e - \mathbb{E}_q e = \sigma(w - 1)/\sqrt{N - 1}, which is σN−1\sigma\sqrt{N - 1} on i⋆i^\star. If max⁡J^−EqJ^=o(N)\max\widehat{J} - \mathbb{E}_q\widehat{J} = o(\sqrt{N}), the guarantee Δ^−σN−1→−∞\widehat{\Delta} - \sigma\sqrt{N - 1} \to-\infty. By Theorem 2, max⁡pWσ≥Wσ(q)=0\max_p W_\sigma\ge W_\sigma(q) = 0.

Corollary 7 (Labels needed before search is safe). Let labels follow a correctly specified Bradley–Terry model with r=θ⋆ϕr = \theta^\star\phi, ∥ϕ∥≤L\lVert\phi\rVert\le L summed over each segment and ∥θ⋆∥≤B\lVert\theta^\star\rVert\le B, and let J^\widehat{J} come from the maximum-likelihood estimate constrained to ∥θ∥≤B\lVert\theta\rVert\le B on nn labelled pairs whose feature differences have second-moment matrix ΣD\Sigma_D. With Σq\Sigma_q the covariance of plan features under qq and the coverage Cq=λmax⁡((ΣD+μI)−1/2Σq(ΣD+μI)−1/2)C_q = \lambda_{\max}((\Sigma_D + \mu I)^{-1/2}\Sigma_q(\Sigma_D + \mu I)^{-1/2}), with probability 1−δ1 - \delta, σe2≤C2Cq(d+log⁡(1/δ)γ2n+μB2)\sigma_e^2 \le C^2C_q\left(\frac{d+\log(1/\delta)}{\gamma^2 n} + \mu B^2\right) for any μ>0\mu> 0, with the constants CC and γ\gamma of Zhu et al. [48]. A positive gain can be certified once this is below σq(J^)2\sigma_q(\widehat{J})^2, which holds for n>C2Cq(d+log⁡(1/δ))/(γ2(σq(J^)2−C2CqμB2))n > C^2C_q(d+\log(1/\delta))/(\gamma^2(\sigma_q(\widehat{J})^2 - C^2C_q\mu B^2)) at any μ\mu with C2CqμB2<σq(J^)2C^2C_q\mu B^2 < \sigma_q(\widehat{J})^2.

The labels needed grow linearly with CqC_q, how much further the proposals vary than the labelled pairs did; wider proposal noise and states the labelled data never reached both raise it.

Proof of Theorem 7. The scorer’s error on a plan is e=(θ^−θ⋆)⊤Φe = (\widehat{\theta} - \theta^\star)^\top\Phi, with Φ\Phi the plan’s summed features, so σe2=v⊤Σqv\sigma_e^2 = v^\top\Sigma_qv with v=θ^−θ⋆v = \widehat{\theta} - \theta^\star, which is at most Cq∥v∥ΣD+μI2C_q\lVert v\rVert^2_{\Sigma_D+\mu I}. Lemma 3.1 of Zhu et al. [48] bounds ∥v∥ΣD+μI\lVert v\rVert_{\Sigma_D+\mu I} by C(d+log⁡(1/δ))/(γ2n)+μB2C\sqrt{(d+\log(1/\delta))/(\gamma^2 n) + \mu B^2} with probability 1−δ1-\delta, with γ=1/(2+e−LB+eLB)\gamma= 1/(2 + e^{-LB} + e^{LB}). A positive gain can be certified when σe<σq(J^)\sigma_e < \sigma_q(\widehat{J}), by Theorem 2, and solving the bound for nn gives the condition.

Proof of Theorem 5. (i) log⁡Z(η)=log⁡Eq[eηJ^]\log Z(\eta) = \log\mathbb{E}_q[e^{\eta\widehat{J}}] has ddηlog⁡Z=Epη[J^]\frac{d}{d\eta}\log Z = \mathbb{E}_{p_\eta}[\widehat{J}]. Then ddηEpη[J]=ddηEq[JeηJ^]/Z=Epη[JJ^]−Epη[J]Epη[J^]\frac{d}{d\eta}\mathbb{E}_{p_\eta}[J] = \frac{d}{d\eta}\mathbb{E}_q[Je^{\eta\widehat{J}}]/Z = \mathbb{E}_{p_\eta}[J\widehat{J}] - \mathbb{E}_{p_\eta}[J]\mathbb{E}_{p_\eta}[\widehat{J}]. With KL⁡(pη∥q)=ηEpη[J^]−log⁡Z(η)\operatorname{KL}(p_\eta\mathbin\Vert q) = \eta\mathbb{E}_{p_\eta}[\widehat{J}] - \log Z(\eta), ddηKL⁡=Epη[J^]+ηVar⁡pη(J^)−Epη[J^]=ηVar⁡pη(J^)\frac{d}{d\eta}\operatorname{KL} = \mathbb{E}_{p_\eta}[\widehat{J}] + \eta\operatorname{Var}_{p_\eta}(\widehat{J}) - \mathbb{E}_{p_\eta}[\widehat{J}] = \eta\operatorname{Var}_{p_\eta}(\widehat{J}). Since KL is increasing in η\eta, the sign of ddηEpη[J]\frac{d}{d\eta}\mathbb{E}_{p_\eta}[J] is the sign of the derivative along the distance as well, and a local maximum needs that sign to change from positive to negative; a covariance that touches zero without changing sign gives none. (ii) Expanding at η=0\eta= 0: Epη[J]=Eq[J]+ηCov⁡q(J,J^)+O(η2)\mathbb{E}_{p_\eta}[J] = \mathbb{E}_q[J] + \eta\operatorname{Cov}_q(J,\widehat{J}) + O(\eta^2) and KL⁡(pη∥q)=η22Var⁡q(J^)+O(η3)\operatorname{KL}(p_\eta\mathbin\Vert q) = \frac{\eta^2}{2}\operatorname{Var}_q(\widehat{J}) + O(\eta^3), so η=2d/σq(J^)+O(d)\eta= \sqrt{2d}/\sigma_q(\widehat{J}) + O(d) and Epη[J]−Eq[J]=2Cov⁡q(J,J^)d/σq(J^)+O(d2)=2ρqσq(J)d+O(d2)\mathbb{E}_{p_\eta}[J] - \mathbb{E}_q[J] = \sqrt{2}\operatorname{Cov}_q(J,\widehat{J})d/\sigma_q(\widehat{J}) + O(d^2) = \sqrt{2}\rho_q\sigma_q(J)d + O(d^2). (iii) The decomposition is an identity. On NN samples with a unique maximizer of J^\widehat{J}, pλp_\lambda tends to a point mass at that sample, so A‾\overline{A} tends to it and J(A‾)−Ep[J]→0J(\overline{A}) - \mathbb{E}_{p}[J] \to0.

When the probe pools many states, KL and the gain are averaged over states, and the slope in (ii) becomes 2Cov⁡q(J,J^)/Var⁡q(J^)\sqrt{2}\operatorname{Cov}_q(J,\hat{J})/\sqrt{\operatorname{Var}_q(\hat{J})}, which is the predicted slope reported in Appendix § 9.

Proof of Theorem 6. With deterministic dynamics Qt(st,at)=rt+Vt+1(st+1)Q_t(s_t,a_t) = r_t + V_{t+1}(s_{t+1}), with VT≡0V_T \equiv0 (and V≡0V \equiv0 after termination). Summing Qt(st,at)−Vt(st)Q_t(s_t,a_t) - V_t(s_t) over the episode telescopes to ∑trt−V0(s0)=R−V0(s0)\sum_t r_t - V_0(s_0) = R - V_0(s_0). Adding and subtracting J(Aˉt)−J(Ut)J(\bar{A}_t) - J(U_t) in each term and splitting it as in Theorem 5(iii) gives the identity, along every realization of the proposal noise. If the weights ptp_t maximize WσW_\sigma for some σ≥σe(st)\sigma\ge\sigma_e(s_t), then by Theorem 1 Δt≥Δ^t−σe(st)s(pt)≥Wσ(pt)≥Wσ(q)=0\Delta_t \ge\hat{\Delta}_t - \sigma_e(s_t)s(p_t) \ge W_\sigma(p_t) \ge W_\sigma(q) = 0. □

Setup details

Planner. MPPI with N=256N = 256 sequences, horizon H=20H = 20, noise σ=0.5\sigma= 0.5, warm start from the cloned policy’s closed-loop plan, rolled out with the true MuJoCo dynamics (on the arm, a copy of the Meta-World model with added position sensors whose rollouts match the environment exactly). Temperatures λ∈{3,1,0.3,0.1,0.01}\lambda\in\{3, 1, 0.3, 0.1, 0.01\} on standardized per-step scores. Uniform averaging is MPPI with λ=106\lambda= 10^6: the weights are uniform and it executes the first action of the average of the 256 clipped proposals. Clipping at the action bounds makes this average differ from the clean cloned plan UU (over 20 steps, by 0.4 standard deviations of plan return on the drawer, 0.7 on cheetah, 1.9 on plate-slide and 0.03 on hopper, in the probe), so it is the λ→∞\lambda\to\infty end of the same planner, with the same proposals as every other temperature, and the reference for the distance curves. The cloned policy run directly executes its own action, with no proposals, and scores higher on every task (cheetah 412, hopper 502, drawer 898, plate-slide 1,111, window-open 372 on the same ten starts).

Statistics. The resampling unit is a run, one closed-loop episode. A single-scorer cell pairs scorer seeds 0–4 with two or four start states each; an ensemble cell uses one ensemble on ten starts. Standard errors are over runs, so they include both scorer-seed and start-state variation. Bootstrap intervals resample runs within each cell; the hierarchical version, used for the slopes and the label-budget trend in § 4.2, resamples the five scorer seeds once per draw for all cells of a setting and then runs within each seed. The five calibration sets of Table 6 are disjoint, so their spread includes the sampling of the pairs.

The automatic temperature sets λt=cVar⁡mJ^m(Ai)\lambda_t = c\sqrt{\operatorname{Var}_m \hat{J}_m(A_i)} from the ensemble’s disagreement at each step, c=0.5c = 0.5. The guard compares the ensemble standard deviation of the top 5% of plans by mean score with its average over all plans and uses the cautious ensemble with exponential weights when the ratio exceeds 1, otherwise χ2\chi^2 weights on the mean. CEM: 6 rounds of 128 samples, top 16 kept, minimum standard deviation 0.05, executing the first action of the final mean. Two variants left out of Table 1 do no better at λ=0.01\lambda= 0.01: χ2\chi^2 weights on the cautious ensemble (cheetah 439, hopper 348, drawer 625 with 8/10, human labels 229 with 0/10, plate-slide 414 with 7/10) and MPPI on the optimistic ensemble, mean plus one standard deviation (331 and 221 over 5 runs, 467 with 5/10, 198 with 0/10, 222 with 4/10).

Scorers and cloned policies. Scorer: MLP with two hidden layers of 256, Adam (learning rate 3×10−43 \times10^{-4}, weight decay 10−410^{-4}), 60 epochs, batch 64, 10% of pairs held out; output standardized per step on the labelled data. Cloned policy: MLP with two hidden layers of 256 and tanh output, Adam (10−310^{-3}), 20,000 steps. Locomotion: crowd labels from Uni-RLHF for the D4RL medium-replay datasets (2,000 pairs, segment length 200), loaded with a restricted unpickler. Arm: 10 scripted demonstrations (task variations 40–49) for cloning; the labelled data add 100 rollouts of the cloned policy with action noise 0–1 (variations 10–39); evaluation on variations 0–9.

Labelling where the planner goes. A scorer trained on the first 250 offline drawer pairs drives standard MPPI (λ=0.01\lambda= 0.01 and 0.1) for 20 episodes on training positions; 250 new pairs compare 50-step segments of those episodes with each other or with the offline data, labelled by the same synthetic teacher.

Best-of-N. 20,000 state–action-pair comparisons per task from states of the labelled data; both actions are the cloned action plus N(0,0.52)\mathcal{N}(0, 0.5^2) noise; the Bradley–Terry teacher prefers the action closer to the scripted expert action with temperature set by bisection for 85% agreement. Verifier: MLP on (state, action), two hidden layers of 256, same optimizer and epochs as the scorers, five seeds per budget. Episodes of 150 steps on variations 0–9.

One-step probe

Along 10 episodes of the cloned policy, at every tenth step, we build the planner’s 256 proposals exactly as MPPI does and roll each out with the true dynamics, recording its learned and true 20-step returns; we also roll out the clean cloned plan and the weighted-mean plan at 17 temperatures from 100 to 0.01 for both weighting rules. Table 3 reports the mean correlation ρq\rho_q over states, the price of noise and the selection gain at λ=0.01\lambda= 0.01 in units of the mean proposal spread σq(J)\sigma_q(J), and the code reports the predicted and measured initial slopes and the distance at which the tilted covariance first reaches zero. For Theorem 2 we score the proposals with all five scorers and use their mean J^\hat{J}. The true error spread is that of J^−J\hat{J} - J after the best positive rescaling of JJ into the scorer’s units, σq(J^)1−ρ2\sigma_q(\hat{J})\sqrt{1-\rho^2}, so no search, when ρ≤0\rho\le0; the rule’s temperature solves λχ2=σ\lambda\sqrt{\chi^2} = \sigma by bisection. The temperature-deployable version replaces σq\sigma_q with κ\kappa times the ensemble’s disagreement, the root mean square over proposals of the standard deviation across scorers after centering each scorer over the proposals, and κ∗\kappa^* is the per-state ratio of the true error spread to this disagreement. The calibrated version draws MM pairs of proposals at random probed states, labels each pair by its true plan return, and sets ρ^=sin⁡(π(c−12))\hat{\rho} = \sin(\pi(c - \frac{1}{2})) from the fraction cc of pairs the five-scorer mean orders correctly; the rule then uses σ=σq(J^)1−ρ^2\sigma= \sigma_q(\hat{J})\sqrt{1-\hat{\rho}^2} at every state, so no search when ρ^≤0\hat{\rho} \le0. The probed states lie on cloned-policy episodes from the ten start states that the closed-loop evaluation uses. The five calibration sets of Table 1 come instead from a second probe on start states 10–19, which no closed-loop run uses (on the arm, object positions that the labelled data also cover). The pooled closed-loop runs of Table 6 use the median ρ^\hat{\rho} over 50 draws of 100 pairs from the first probe (cheetah 0.06, hopper 0.70, drawer 0.06, human labels −0.27-0.27, plate-slide 0.49, window-open 0.48), and Section E studies how the estimate varies. The averaging term of Theorem 3, J(Aˉ)−Ep[J]J(\bar{A}) - \mathbb{E}_p[J], is 0.25–0.85 standard deviations at uniform weights and shrinks to about zero at hard exponential search on every task except window-open, where it is −0.10-0.10 at uniform weights and −0.15-0.15 at worst, so there the executed plan can fall short of the guarantee by that much.

Episodes, data support and validation

Closed-loop decomposition. Table 4 measures the terms of Theorem 6 along MPPI episodes. At every visited state the planner’s 256 proposals, its executed average and the clean cloned plan are rolled out with the true reward, and VtV_t is the true return of the cloned policy from that state to the end of the episode; the identity then holds in every run to rounding. Uniform averaging pays a price of noise of 387 over a drawer episode and recovers 225 of it by averaging. Hard search removes the averaging term, so it must recover the price by selection: the perfect scorer does (+632), the learned scorers do not on the drawer (−151-151, and −373-373 with human labels). The horizon gap is large on the walkers (+655 to +1,727), where the cloned policy’s own value from a visited state is a poor guide to what 20-step plans achieve, so the walker rows are best read through ρ\rho. On the states the planner visits the learned scorer’s per-state correlation is lower than along the cloned policy’s own episodes in every exploited setting except cheetah, where it is near zero on both; on the drawer with human labels it is negative at 91% of visited states.

runsRRV0V_0R−V0R-V_0∑Δ\sum\Delta∑c\sum c∑α\sum\alpha∑η\sum\etaρ\rho visitedρ\rho cloned
Cheetah, λ=0.01\lambda= 0.015237140+98±68+98 \pm68+183±153+183 \pm1531818+5+5+1727+1727+0.08+0.08 (34%)+0.05+0.05
Hopper, λ=0.01\lambda= 0.015269426−156±86-156 \pm86−58±278-58 \pm278754+1+1+655+655+0.51+0.51 (13%)+0.76+0.76
Drawer 500, uniform averaging5854898−44±2-44 \pm20387+225+225+118+118+0.09+0.09 (36%)+0.10+0.10
Drawer 500, λ=3\lambda= 35765898−133±59-133 \pm59−31±35-31 \pm35301+119+119+81+81+0.03+0.03 (46%)+0.10+0.10
Drawer 500, λ=0.01\lambda= 0.015472898−426±164-426 \pm164−151±77-151 \pm77750−200-200−0.11-0.11 (62%)+0.10+0.10
Drawer, perfect scorer, λ=0.01\lambda= 0.0151099898+200±5+200 \pm5+632±20+632 \pm201950−237-237+1.00+1.00 (0%)–
Drawer human, λ=0.01\lambda= 0.015218898−680±11-680 \pm11−373±63-373 \pm63690−238-238−0.31-0.31 (91%)−0.10-0.10
Plate 500, λ=0.01\lambda= 0.0155161115−599±108-599 \pm108+113±58+113 \pm58216+1+1−497-497−0.07-0.07 (52%)+0.45+0.45
Window 500, λ=0.01\lambda= 0.015403381+22±17+22 \pm17+145±42+145 \pm42−43-43−1-1−165-165+0.22+0.22 (19%)+0.35+0.35

Table 4. Closed-loop decomposition (Theorem 6) of standard MPPI episodes against the cloned policy: episode return RR, the cloned policy’s return V0V_0 from the same start, and the episode sums of the selection gain Δt\Delta_t, the price of noise ctc_t, the averaging term αt\alpha_t and the horizon gap ηt\eta_t, so that R−V0=∑Δ−∑c+∑α+∑ηR - V_0 = \sum\Delta- \sum c + \sum\alpha+ \sum\eta (exact in every run, to 10−1210^{-12}). ρ\rho: mean per-state correlation of the scorer with the true plan return over the 256 proposals, on the states the planner visits (share of states below zero in parentheses) and on the cloned policy’s own states (one-step probe, same scorer). Mean over runs, with the standard error over runs for R−V0R - V_0 and ∑Δ\sum\Delta. Each run is paired with V0V_0 from its own start, computed in the planner’s simulator copy. On the arm run kk uses scorer seed kk and start kk; the walker runs use starts 0 and 1 only (three from start 0, where the cloned cheetah scores 1), so their V0V_0 (cheetah 140, hopper 426) is not the mean over ten starts of Table 1 (412, 502). The cloned cheetah’s return also depends sharply on small state differences (from start 1: 347 in the copy, 662 in the environment), the source of its large horizon gap.

Data support. For each arm setting, every step of every labelled segment is a point in the scorer’s standardized input (observation and action). A visited state counts as off the data when its distance to the nearest labelled point exceeds the 95th percentile of the labelled points’ own nearest-neighbour distances (to a point of a different segment). In observations alone, hard search (λ=0.01\lambda= 0.01, five recorded episodes) is off the data at 53% of steps on the drawer with 500 labels, 60% with human labels and 51% on plate-slide, and uniform averaging at none; on window-open the shares are 75% and 60%. With actions included the shares are 42%, 47%, 27% and 32% for hard search, and 0–2% without search. The window-open evaluation positions lie at the labelled data in the observation space, and search still helps there, so leaving the data is not by itself what makes a scorer exploitable. In hand position alone, hard search ends on average 6.6 cm from the nearest labelled hand position on the drawer with synthetic labels (0.3 cm without search) but within 0.8 cm on plate-slide.

Held-out accuracy without shared trajectories. The scorers hold out a random 10% of pairs, so a held-out pair can share trajectories with training pairs. Splitting the labelled episodes 70/30 instead, and holding out the pairs whose two segments both come from held-out episodes, gives the same picture (five seeds pooled; a random split with the same number of training pairs in parentheses): drawer 500 labels 180 of 213 correct, 0.85 (0.85); 2,000 labels 0.83 (0.83); human labels 114 of 118, 0.97 (0.99); plate-slide 0.86 (0.81); window-open 0.84 (0.89); hopper 0.70 (0.74); cheetah 0.72 (0.73).

Calibrating from few pairs

Table 5 repeats the calibration of Section C with 200 random draws of exactly MM pairs (pairs tied in true return are redrawn) for M=25M = 25, 50, 100 and 200, and three labellers: the true order, and Bradley–Terry teachers P(i≻j)=1/(1+e−(Ji−Jj)/τ)P(i \succ j) = 1/(1+e^{-(J_i-J_j)/\tau}) with τ\tau set so that they agree with the true order on 85% or 70% of proposal pairs, a stand-in for a person comparing two short plans from the same state. A cautious version replaces cc by its one-sided Wilson lower bound at nominal 95%.

ρq\rho_qρ^\hat{\rho} allbelowM=25M=25M=100M=100M=200M=200Δ\DeltaΔ\DeltaworstΔ\Deltaworstρ^/Δ\hat{\rho}/\Deltaρ^/Δ\hat{\rho}/\Delta
Cheetah0.080.0754%−0.31, 0.43−0.09, 0.28−0.03, 0.220.060.03−0.060.00−0.010.06 / 0.030.06 / 0.03
Hopper0.730.6926%0.43, 0.880.54, 0.810.60, 0.780.530.49−0.020.40−0.010.64 / 0.460.43 / 0.37
Drawer 2000.010.0246%−0.43, 0.43−0.19, 0.25−0.13, 0.160.050.00−0.210.00−0.030.00 / 0.000.00 / 0.00
Drawer 300−0.01−0.0245%−0.54, 0.31−0.22, 0.16−0.16, 0.110.070.01−0.110.00−0.010.00 / 0.010.03 / 0.01
Drawer 5000.080.0747%−0.31, 0.43−0.09, 0.28−0.06, 0.190.120.04−0.220.01−0.040.06 / 0.030.06 / 0.03
Drawer 1k0.070.0748%−0.31, 0.43−0.13, 0.25−0.08, 0.200.110.05−0.150.01−0.020.09 / 0.050.09 / 0.05
Drawer 2k0.190.2046%−0.19, 0.540.03, 0.430.06, 0.320.230.14−0.150.05−0.050.22 / 0.140.13 / 0.11
Drawer human−0.18−0.2153%−0.64, 0.19−0.40, −0.03−0.34, −0.090.050.00−0.020.000.00−0.13 / 0.00−0.06 / −0.01
Drawer, half on planner0.160.1853%−0.31, 0.540.00, 0.370.02, 0.310.260.12−0.170.03−0.040.13 / 0.110.13 / 0.10
Plate 5000.440.4519%0.06, 0.730.25, 0.610.32, 0.580.420.35−0.030.23−0.010.43 / 0.340.28 / 0.27
Plate 2k0.390.4021%0.05, 0.730.22, 0.560.28, 0.540.390.300.000.190.000.37 / 0.290.28 / 0.24
Window 5000.440.4747%0.06, 0.810.28, 0.610.38, 0.590.390.33−0.030.22−0.010.43 / 0.310.28 / 0.24
Window 2k0.240.2859%−0.19, 0.640.06, 0.450.14, 0.380.220.12−0.860.05−0.400.19 / 0.100.09 / 0.07

Table 5. Calibrating the error spread from MM labelled pairs of the planner’s own proposals (one-step probe, 200 random draws of the pairs). ρq\rho_q: mean per-state correlation of the five-scorer mean with the true plan return; ρ^\hat{\rho} all: the Gaussian estimate sin⁡(π(c−12))\sin(\pi(c-\frac{1}{2})) from every pair at every state; below: share of states whose own correlation is under it. Range: 10th and 90th percentile of ρ^\hat{\rho} over draws. Gains Δ\Delta (mean over states) and worst-state Δ\Delta (mean over draws) of the calibrated rule with 100 pairs, in units of σq(J)\sigma_q(J): with each state’s true ρ\rho (worst state 0.00 in every row), with the point estimate, with the one-sided 95% lower bound (LCB), and with labels from a noisy teacher that agrees with the true order on 85% or 70% of proposal pairs (point estimate).

Sampling noise. The 10–90% range of ρ^\hat{\rho} spans 0.45–0.85 with 25 pairs, 0.27–0.44 with 100 and 0.18–0.29 with 200. With 100 pairs the point estimate loses at most 0.22 standard deviations at the worst state on average over draws in twelve of the thirteen settings, but up to 0.47 in the unluckiest tenth of draws (drawer, 200 labels). Lower bound. The lower bound cuts the worst-state loss to at most 0.05 (0.40 on window-open with 2,000 labels) and the unluckiest-tenth loss to 0.25, at a cost in gain: it keeps 77% of the true-ρ\rho rule’s gain on hopper, 48–57% on plate-slide and on window-open with 500 labels, and 12–24% in the other settings where that gain exceeds 0.2. Noisy labels. A noisy teacher shrinks ρ^\hat{\rho} toward zero (hopper 0.71, 0.64 and 0.43 for the true order, 85% and 70%; plate-slide 0.45, 0.43 and 0.28), so where the scorer is informative the rule searches more gently; with 100 pairs the worst-state loss stays at most 0.19 (85%) and 0.18 (70%) in the same twelve settings, and the share of the gain kept falls from 47–93% to 42–87% and 31–71%. Gaussian assumption. Over all pairs, ρ^\hat{\rho} is within 0.05 of the mean Pearson correlation in every setting, and above it by up to 0.04 on plate-slide and window-open, where it is mildly optimistic. The larger approximation is pooling: one ρ^\hat{\rho} serves every state, and 19–59% of states have their own correlation below it. Those states cause the losses, and window-open with 2,000 labels has the most of them (59%).

Closed loop. Table 6 runs the rule with the pooled estimate, its lower bound and a noisy teacher, and with five independent sets of exactly 100 pairs per setting (500 distinct pairs, none tied in true return), each calibrating its own controller on the same ten starts, drawn either from the probe on the evaluation’s start states or from a second probe on start states 10–19 that the evaluation never uses. The noisy teacher changes little (within 20 points of the true-order version). Across independent sets the arm results barely move (evaluation starts: drawer 824–852, plate-slide 999–1,013, window-open 308–320; held-out starts: 818–852, 806–1,013, 309–318; every run successful); the walkers move more (cheetah 264–392 and 264–445, hopper 517–575 and 523–550, since a small change in κ\kappa near 1 changes how hard they search. On plate-slide one held-out set estimates ρ^=0.66\hat{\rho}=0.66, against 0.40–0.51 for the others, so it searches harder and scores 806, still succeeding in every run. With the lower bound every set keeps the arm at or near uniform averaging’s level (evaluation starts: drawer 852 with either label set, plate-slide 1,036–1,041, window-open 303–310; held-out starts: drawer 835–852, plate-slide 998–1,041, window-open 304–308; every run successful), declines to search on cheetah (264 with every set), and keeps most of hopper’s gain (500–560 and 472–551).

What the estimate must get right is stated by the following consequence of Theorem 1 and Theorem 2.

Corollary 8 (Calibrated rule). (i) At a state whose proposals have ρs=Corr⁡q(J,J^)>0\rho_s=\operatorname{Corr}_q(J,\hat{J})>0, the rule with σ=σq(J^)1−ρ^+2\sigma=\sigma_q(\hat{J})\sqrt{1-\hat{\rho}_+^2}, ρ^+=max⁡(0,ρ^)\hat{\rho}_+=\max(0,\hat{\rho}), does not lose (Δ≥0\Delta\ge0) whenever ρ^≤ρs\hat{\rho}\le\rho_s, and it does not search when ρ^≤0\hat{\rho}\le0.

(ii) Let a labeller rank pairs by J+uJ+u, with JJ in the scorer’s units, J^=J+e\hat{J}=J+e, and ee and uu uncorrelated with JJ. If (J^,J^+u)(\hat{J},\hat{J}+u) is bivariate Gaussian under a population concordance cc of the labeller with J^\hat{J} on two independent proposals, satisfies sin⁡(π(c−12))=ρu=Corr⁡q(J^,J+u)\sin(\pi(c-\frac{1}{2}))=\rho_u=\operatorname{Corr}_q(\hat{J},J+u), and ρu>ρq\rho_u>\rho_q exactly when Cov⁡q(e,u)>σq(J)(σq(J)2+σu2−σq(J))\operatorname{Cov}_q(e,u)>\sigma_q(J)(\sqrt{\sigma_q(J)^2+\sigma_u^2}-\sigma_q(J)), about σu2/2\sigma_u^2/2 for a small labelling error; with Cov⁡q(e,u)=0\operatorname{Cov}_q(e,u)=0, ρu<ρq\rho_u<\rho_q whenever σu>0\sigma_u>0.

Proof. (i) By the scale argument σe=σq(J^)1−ρβ2≤σ\sigma_e=\sigma_q(\hat{J})\sqrt{1-\rho_\beta^2}\le\sigma, so the rule’s weights pp maximize Wσ(p)W_\sigma(p) and Δ(p)≥Δ^(p)−σes(p)≥Wσ(p)≥Wσ(q)=0\Delta(p)\ge\hat{\Delta}(p)-\sigma_e s(p)\ge W_\sigma(p)\ge W_\sigma(q)=0 by Theorem 1; if ρ^≤0\hat{\rho}\le0 then ρ^+=0\hat{\rho}_+=0, σ=σq(J^)\sigma=\sigma_q(\hat{J}) and Theorem 2(i) gives p=qp=q. (ii)

For a bivariate Gaussian, two independent draws are concordant with probability 12+arcsin⁡(ρμ)/π\frac{1}{2} + \arcsin(\rho_{\mu})/\pi [18]; and ρμ=(σq(J)2+Cov⁡q(e,u))(σq(J^)σq(J)2+σu2)\rho_{\mu} = \frac{(\sigma_q(J)^2 + \operatorname{Cov}_q(e,u))}{(\sigma_q(\hat{J})\sqrt{\sigma_q(J)^2 + \sigma_u^2})} and ρq=σq(J)/σq(J^)\rho_q = \sigma_q(J)/\sigma_q(\hat{J}). □

In the population, then, a labeller whose mistakes are independent of the scorer’s only lowers the correlation that ρ^\hat{\rho} estimates, as the noisy teachers do; one whose mistakes the scorer has learned, because it was trained on that labeller’s labels, can raise it above the true correlation, and then the rule searches where it should not. A finite, pooled estimate can also exceed a given state’s correlation through sampling noise (the ranges above) and pooling over states. The one-sided Wilson bound at nominal 95% makes ρ^\hat{\rho} at most the pooled correlation with probability close to 0.95 under the Gaussian model (its coverage is approximate: 94.2% for 100 independent pairs at 90% agreement), so (i) then covers every state whose own correlation is at least the pooled one. This is a statement about the pooled correlation, not a simultaneous one over states: the states it leaves uncovered are the 19–59% noted above, so the rule is an empirical safeguard, not a guarantee.

Ties. The true-order and teacher labels above never tie. People often cannot tell two plans apart, and cc is then measured on the decided pairs only. If a person calls a tie when the two plans’ perceived returns differ by less than a threshold, as in threshold models of paired comparison [32], the decided pairs are the clearer ones and agree with the scorer more often than random pairs do, so sin⁡(π(c−12))\sin(\pi(c-\frac{1}{2})) overstates ρu\rho_u. In the Gaussian model of Theorem 8(ii), let the labeller’s difference XX (standardized) and the scorer’s difference have correlation ρu\rho_u, and let a share tt of pairs be tied, those with ∣X∣<zt=Φ−1(1+t2)|X| < z_t = \Phi^{-1}(\frac{1+t}{2}). Given XX, the scorer agrees in sign with probability Φ(ρu∣X∣/1−ρu2)\Phi(\rho_u |X|/\sqrt{1-\rho_u^2}), so the decided pairs’ concordance is

ct(ρu)=E[Φ(ρu∣X∣1−ρu2)∣∣X∣>zt],c_t(\rho_u) = \mathbb{E}\left[\Phi\left(\frac{\rho_u |X|}{\sqrt{1-\rho_u^2}}\right) \mid|X| > z_t\right],

which is increasing in ρu\rho_u, equals 12+arcsin⁡(ρu)/π\frac{1}{2} + \arcsin(\rho_u)/\pi at t=0t = 0, and grows with tt when ρu>0\rho_u > 0. We estimate ρu\rho_u by solving ct(ρ^)=cc_t(\hat{\rho}) = c at the observed share of ties, and its lower bound by applying the same map to the Wilson bound on cc. On the probe’s proposals, a true-order labeller that ties the closest 70% of pairs gives an uncorrected ρ^\hat{\rho} of 0.97 on hopper, 0.77 on plate-slide and 0.73 on window-open, against 0.69, 0.45 and 0.47 over all pairs; the corrected estimates are 0.70, 0.45 and 0.42, and within 0.03 on cheetah and the drawer with 500 labels. With human labels, where the correlation is negative (−0.21 over all pairs), both estimates keep the sign but shrink it (−0.08 and −0.04), so the rule still does not search.

Two people’s comparisons. One author compared 100 random pairs of the drawer proposals blind: the 20 planned steps of two proposals from the same state, as 2-s clips in random order and sides (9 minutes in all). They could not tell 68 apart and agreed with the true return on 16 of the other 32, which is chance. Where one plan opened the drawer more than 1 mm further over the clip (14 decided pairs) they chose it 13 times; the true 20-step return chose it 6 times, and mostly favours the plan that ends with the hand nearer the handle (68% of 31 such pairs, against 42% for the person). The scorers with 500 and 2,000 synthetic labels agree with these choices at chance (ρ^=0.00\hat{\rho} = 0.00, lower bound −0.42-0.42), so the rule does not search with them and keeps 852 (10 of 10). The scorer trained on the same person’s clip labels agrees with them (ρ^=0.63\hat{\rho} = 0.63, lower bound 0.23) although it disagrees with the true return on the same pairs (−0.29-0.29). Calibrated from these choices the rule searches with it (κ=0.77\kappa= 0.77) and scores 574 (5 of 10 opened), against 852 when calibrated by true return; with the lower bound (κ=0.97\kappa= 0.97) it scores 839 (10 of 10). Two thirds of the pairs were ties, so the decided pairs are the clearer ones; corrected for ties, ρ^=0.36\hat{\rho} = 0.36 (lower bound 0.12) and the rule searches gently (κ=0.93\kappa= 0.93), scoring 822 (10 of 10), still below uniform averaging: the correction removes the overconfidence the ties add, not the shared bias. The other author compared the same 100 pairs (38 minutes): they could not tell 71 apart and agreed with the true return on 12 of the other 29, and where both people decided (12 pairs) they agreed 9 times. Their choices do not back the scorer trained on the first author’s clip labels (ρ^=0.16\hat{\rho} = 0.16, lower bound −0.30-0.30, so κ=0.99\kappa= 0.99): the rule barely searches and scores 845 (10 of 10). Pooled over both people, ρ^=0.42\hat{\rho} = 0.42 (lower bound 0.11) gives 812 and 846 (10 of 10 each). On the scorers with 500 and 2,000 synthetic labels the second author’s choices give ρ^=0.16\hat{\rho} = 0.16 and 0.37, and the rule scores 831 and 893 (10 of 10 each). Table 7 collects the three kinds of calibration for this scorer, with and without the tie correction. They are not interchangeable: the oracle sees that the scorer ranks the proposals worse than chance and declines to search; the labeller who trained the scorer shares its bias and endorses it; the other author does not. A rule calibrated from a person’s preferences checks the scorer against that person, so it cannot catch a scorer that learned a bias it shares with the evaluator (Theorem 8(ii)); a second person, the tie correction or the lower bound limits the damage. With two labellers on one task, this is a case study, not an estimate of how often it happens.

Null and limit cases

Button-press. Same protocol as the drawer, 5 runs per setting: not exploited at any budget (standard MPPI 151–157 at λ=3\lambda= 3 and 154–171 at λ=0.01\lambda= 0.01 with 200, 500 or 2,000 labels; nearly every run presses the button). Here the perfect scorer does worse (126–140), because over a 20-step horizon Meta-World’s dense reward pays for hovering near the button rather than pressing it, while the preference-trained scorers learned progress over 50-step clips. The label budget at which exploitation starts is task-specific.

Adroit door and hammer. With crowd labels that compare only successful human demonstrations (Rajeswaran et al., [30]; Yuan et al., [46]), the scorer never opens the door (0 of 12 runs at any temperature) and drives the nail in 2 of 60 runs, while the perfect scorer opens the door in 6 of 6 runs at λ≤0.01\lambda\le0.01 and drives the nail in 4 of 5 at λ=0.1\lambda= 0.1. With no failures in the labelled data the scorer has no signal about failure, and no weighting rule can supply one.

Best-of-NN details

At each step the cloned policy’s action plus N(0,0.52)\mathcal{N}(0, 0.5^2) noise gives NN candidates, and an MLP verifier V(s,a)V(s, a) picks one. It is trained with the Bradley–Terry loss on pairs of perturbed actions, where a synthetic labeller prefers the action closer to Meta-World’s scripted expert, with noise set for 85% agreement. The verifier sees the state, not images, and its labels are synthetic. The perfect verifier is the negative distance to the expert action. Single verifiers pool scorer seeds 0–4 with episode seeds 0–9 (50 runs per NN); the perfect verifier and one randomly perturbed action, with no verifier, use the same ten starts with five noise draws each. In Figure 3c, N=1N = 1 is one noisy action and the clean cloned action, executed without noise, is the reference line. A cautious five-verifier ensemble does not help on plate-slide (0 of 10 at N≥16N \ge16). The verifiers’ held-out pair accuracies are 0.62–0.80 at 500 pairs and 0.69–0.81 at 2,000, again a poor guide to how they behave under selection.

Best-of-NN is itself an exponential tilt of qq, the action noise. For continuous scores the population best-of-NN choice has density dpNdq=NF(J^)N−1\frac{d p_N}{d q} = N F(\hat{J})^{N-1}, with FF the distribution function of J^\hat{J} under qq: this is qq tilted by eηlog⁡F(J^)e^{\eta\log F(\hat{J})} at η=N−1\eta= N - 1, so Theorem 5(i) applies with log⁡F(J^)\log F(\hat{J}) in place of J^\hat{J}, and true return rises with NN exactly while Cov⁡pN(J,log⁡F(J^))>0\operatorname{Cov}_{p_N}(J, \log F(\hat{J})) > 0.

Its KL to qq is log⁡N−(N−1)/N\log N - (N - 1)/N [36, 2], which puts N=4N = 4 to 1,024 at d=0.8d = 0.8 to 2.4, the range of Figure 3a. This is the population value; the selected index on a finite set of NN candidates has KL log⁡N\log N to a uniform pick.

References

  1. [1]Aouad, A., El Gad arri, A., and Farias, V. F. Revisiting scaling laws for reward optimization. arXiv preprint arXiv:2609.38526, 2026.arxiv.org/abs/2609.38526
  2. [2]Beirami, A., Agarwal, A., Berant, J., D’Amour, A., Eisenstein, J., Nagpal, C., and Suresh, A. T. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879, 2024.arxiv.org/abs/2401.01879
  3. [3]Bhamidipaty, L. M., Whittammer, E. S., Abel, D., Kochenderfer, M. J., and Ramamoorthy, S. Imperfect world models are exploitable. arXiv preprint arXiv:2605.15960, 2026.arxiv.org/abs/2605.15960
  4. [4]Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.DOI
  5. [5]Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03741
  6. [6]Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31, 2018.arxiv.org/abs/1805.12114
  7. [7]Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.02743
  8. [8]Eisenstein, J., Nagpal, C., Agarwal, A., Beirami, A., D’Amour, A., Dvijotham, K., Fisch, A., Heller, K., Pfohl, S., Ramachandran, D., Shaw, P., and Berant, J. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. In Conference on Language Modeling, 2024.arxiv.org/abs/2312.09244
  9. [9]El Hasnaouy, H., Krupa, P., Zanon, M., and Bemporad, A. Active learning MPC objective functions from preferences. arXiv preprint arXiv:2605.16071, 2026.arxiv.org/abs/2605.16071
  10. [11]Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866, 2023.arxiv.org/abs/2210.10760
  11. [12]Hansen, N., Su, H., and Wang, X. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.16828
  12. [13]Hejna, J. and Sadigh, D. Few-shot preference learning for human-in-the-loop RL. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, 2023.arxiv.org/abs/2212.03363
  13. [14]Huang, A., Block, A., Liu, Q., Jiang, N., Krishnamurthy, A., and Foster, D. J. Is best-of-N the best of them? Coverage, scaling, and optimality in inference-time alignment. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2025a.
  14. [15]Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the mythos of KL-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In International Conference on Learning Representations, 2025b.arxiv.org/abs/2407.13399
  15. [16]Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. MOReL: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
  16. [17]Krupa, P., El Hasnaouy, H., Zanon, M., and Bemporad, A. Learning the MPC objective function from human preferences. In IEEE Conference on Decision and Control, 2025.arxiv.org/abs/2511.22502
  17. [18]Kruskal, W. H. Ordinal measures of association. Journal of the American Statistical Association, 53(284):814–861, 1958. doi: 10.1080/01621459.1958.10501481.DOI
  18. [19]Kwa, T., Thomas, D., and Garriga-Alonso, A. Catastrophic Goodhart: Regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2407.14503
  19. [20]Kwok, J., Agia, C., Sinha, R., Foutter, M., Li, S., Stoica, I., Mirhoseini, A., and Pavone, M. RoboMonkey: Scaling test-time sampling and verification for vision-language-action models. arXiv preprint arXiv:2506.17811, 2025.arxiv.org/abs/2506.17811
  20. [21]Laidlaw, C., Singhal, S., and Dragan, A. Correlated proxies: A new definition and improved mitigation for reward hacking. In International Conference on Learning Representations, 2025.arxiv.org/abs/2403.03185
  21. [22]Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1612.01474
  22. [23]Lee, K., Smith, L., and Abbeel, P. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 2021.arxiv.org/abs/2106.05091
  23. [24]Lin, H., Wang, P., Schneider, J., and Shi, G. TD-M(PC)²: Improving temporal difference MPC through policy constraint. arXiv preprint arXiv:2502.03550, 2025.arxiv.org/abs/2502.03550
  24. [25]Luo, J., Xu, X., Wang, S., Zhou, Y., and Zhang, Z. CritiHack: Evaluating visual rewards under robot policy optimization. arXiv preprint arXiv:2610.02527, 2026.arxiv.org/abs/2610.02527
  25. [26]Manheim, D. and Garrabrant, S. Categorizing variants of Goodhart’s law. arXiv preprint arXiv:1803.04585, 2018.arxiv.org/abs/1803.04585
  26. [27]Moskovitz, T., Singh, A. K., Strouse, D., Sandholm, T., Salakhutdinov, R., Dragan, A. D., and McAleer, S. Confronting reward model overoptimization with constrained RLHF. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.04373
  27. [28]Nabail, M., Cheng, L., Wang, J., and Rhinehart, N. UBP2: Uncertainty-balanced preference planning for efficient preference-based reinforcement learning. arXiv preprint arXiv:2606.19328, 2026.arxiv.org/abs/2606.19328
  28. [29]Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2201.03544
  29. [30]Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.arxiv.org/abs/1709.10087
  30. [31]Ramé, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. WARM: On the benefits of weight averaged reward models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024.arxiv.org/abs/2401.12187
  31. [32]Rao, P. V. and Kupper, L. L. Ties in paired-comparison experiments: A generalization of the Bradley–Terry model. Journal of the American Statistical Association, 62(317): 194–204, 1967. doi: 10.1080/01621459.1967.10482901.DOI
  32. [33]Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.DOI
  33. [34]Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2209.13085
  34. [35]Smith, J. E. and Winkler, R. L. The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311–322, 2006.DOI
  35. [36]Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2009.01325
  36. [37]Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A., and Brown, D. S. Causal confusion and reward misidentification in preference-based reward learning. In International Conference on Learning Representations, 2023.arxiv.org/abs/2204.06601
  37. [38]Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.DOI
  38. [39]Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2407.17032
  39. [40]Wang, Z., So, O., Gibson, J., Vlahov, B., Gandhi, M., Liu, G.-H., and Theodorou, E. A. Variational inference MPC using Tsallis divergence. In Robotics: Science and Systems, 2021.
  40. [41]Williams, G., Aldrich, A., and Theodorou, E. A. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017a.DOI
  41. [42]Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A. Information theoretic MPC for model-based reinforcement learning. In IEEE International Conference on Robotics and Automation, 2017b.DOI
  42. [43]Yoon, H.-J. and Kim, H. Residual-conservative model predictive path integral control. arXiv preprint arXiv:2607.06950, 2026.arxiv.org/abs/2607.06950
  43. [46]Yuan, Y., Hao, J., Ma, Y., Dong, Z., Liang, H., Liu, J., Feng, Z., Zhao, K., and Zheng, Y. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations, 2024.arxiv.org/abs/2402.02423
  44. [47]Zhai, Y., Lei, Y., Zhang, H., Yu, Y., Xu, K., Feng, D., Ding, B., and Wang, H. Uncertainty-penalized reinforcement learning from human feedback with diversified reward LoRA ensembles. Information Processing & Management, 63(3):104548, 2026. doi: 10.1016/j.ipm.2025.104548.DOI
  45. [48]Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or K-wise comparisons. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 43037–43067, 2023.arxiv.org/abs/2301.11270

Paper details

vlnc:W_7vqt4fdj
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude (Anthropic)

View statement
Looking for related papers…
Contents