WorksComputer Science
Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
Abstract
Robots increasingly plan by searching against a reward model learned from human preferences; we measure what this search does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic labels and blind labels from the two authors, harder MPPI search cuts true return by up to 76% while held-out label accuracy stays at 0.75-0.97: with human labels the drawer opens in 0 of 20 runs, against 10 of 10 for the cloned policy the planner starts from. The effect persists with a learned dynamics model and with an image scorer. True return roughly follows how far the planner's weights move from its proposals (), and best-of- action selection with a verifier trained on synthetic labels falls below a single noisy action. We prove that -regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, that no gain can be certified once that spread exceeds the scorer's own, and that an episode's return splits exactly into per-step terms, only some of which the guarantee covers. Calibrated from 100 proposal pairs ranked by true return (an oracle), this rule is the safest fix we test: it keeps on average 86-94% of the cloned policy's return on drawer-open and plate-slide, though it is cautious on window-open, where search helps. Calibrated from a person's own comparisons, it cannot detect a bias the scorer shares with that person; a second person, or correcting for pairs people cannot tell apart, limits the damage.
Introduction
A robot that plans against a learned reward is an optimizer pointed at a proxy. Reward models learned from human preferences are now used inside model-predictive control ([17, 9, 28]), and robot foundation models are scaled at test time by sampling many candidate actions and executing the one a learned verifier likes best ([20]). In language-model alignment it is well established that optimizing too hard against a learned reward lowers the quality it was meant to measure ([11, 7, 8]), and a line of theory explains when regularization toward a reference policy does and does not prevent it (Huang et al., 2025b;a; Kwa et al., 2024; Laidlaw et al., 2025). Robot planners differ: the optimizer is a sampling-based search whose strength is set by a temperature and a sample count, the regularizer is implicit in the proposals, the plan is re-optimized at every step, and the labels number in the hundreds or thousands. How much such a planner exploits its scorer has not been measured.
We study Model Predictive Path Integral control (MPPI; [41 b]) with proposals around a policy cloned from the labelled data, the planning analogue of RLHF’s reference model, and a Bradley–Terry scorer trained from preference labels. Search pays when the scorer is right: with the true reward it beats running the cloned policy directly on all five tasks of Figure 1 (980 against 412 on HalfCheetah, 1,090 against 898 on the drawer; on hopper only with gentle search). We ask how much of that gain a learned scorer keeps, planning on the true simulator so that every failure is the scorer’s; § 4.7 checks a learned model and an image scorer. Our contributions are:
Figure 1. True return against MPPI temperature (harder search to the right), mean standard error over 10–20 runs. A learned scorer loses return under harder search on every task except window-open; the perfect scorer keeps or improves its return except on hopper; 300 human labels are exploited fastest.
Goodhart in the planner (§ 4.1). On HalfCheetah and Hopper with real crowd-sourced labels ([46]), and on Meta-World drawer-open and plate-slide with synthetic labels and blind labels from the two authors (Yu et al., 2020a), harder search lowers true return below that of the cloned policy it starts from, while a perfect scorer gains or, on hopper, loses less. High agreement on decided human comparisons does not make planning safe: on the drawer the pairs a person cannot tell apart carry what the planner needs.
On plate-slide, human labels are exploited less than synthetic ones; on window-open, search helps.
An approximate collapse onto distance (§ 4.2). Within each task, true return falls near one curve in between the planner’s weights and its proposals for both weighting rules we test, so weights help mainly by moving less.
How hard to search, and when not to (§ 3). Against an error spread on the proposals, -regularized MPPI at is worst-case optimal, no positive gain can be certified once exceeds the scorer’s own spread, and the guarantee of hard exponential search falls as ; an exact decomposition of an episode’s return shows which terms this controls. The mathematics is elementary; what is new is bounding the error on the planner’s own proposals, which held-out accuracy does not predict (§ 4.3), and following it through the closed loop.
Fixes, measured honestly (Section 4.4). Calibrated from 100 proposal pairs ranked by true return, the worst-case-optimal rule is the safest fix we test, though cautious where search helps. Calibrated from the labeller’s own comparisons, it cannot detect a bias the scorer shares with that person. Ensembles, weights, automatic temperatures and a guard each win somewhere and lose elsewhere, and a UBP2-inspired CEM planner (Nabail et al., [28]) is the most exploited. Labelling the planner’s own trajectories helps most.
Best-of- is not safe either (§ 4.6). With a verifier from 500 synthetic pairs it falls below one noisy action from ( on plate-slide); with 2,000 pairs selection helps.
Setting
Planner. At each control step MPPI rolls out action sequences of length from the current state and executes the first action of their weighted average (Williams et al., [41]). We draw the sequences around the cloned policy’s own closed-loop plan : , , which plays the role of the reference policy in RLHF. Each plan gets a learned return and, for evaluation only, a true return . Standard (exponential) MPPI uses , the solution of , the same KL-regularized objective as RLHF with . The rule of Tsallis-divergence MPPI (Wang et al., [40]) uses with set by , the solution with in place of . Lower means harder search; is best-of- over plans. We log the per-step . Uniform weights () give the planner without search, which we call uniform averaging: it executes the average of the clipped noisy proposals, which is not the cloned policy’s own action, so we also run the cloned policy directly. Unless stated, , , , and planning uses the true MuJoCo dynamics (Todorov et al., [38]).
Scorers. A two-layer MLP is trained on segment pairs with the Bradley–Terry likelihood (Bradley & Terry, [4]; Christiano et al., [5]), with 10% of pairs held out, and standardized per step; a tie (“can’t tell”) is a target of , that is, indifference. Five seeds per label set give single scorers and a five-member ensemble (Lakshminarayanan et al., [22]).
Tasks and labels. Locomotion: HalfCheetah-v5 and Hopper-v5 (Towers et al., [39]), D4RL medium-replay data (Fu et al., 2020) and the Uni-RLHF crowd-sourced labels (Yuan et al., [46]): 2,000 pairs of 200-step segments that agree with the true ranking on 73% (cheetah) and 64% (hopper) of pairs, with 10–15% ties. Manipulation: Meta-World v3 drawer-open, plate-slide and window-open (Yu et al., 2020a) with mocap position control. The cloned policy is trained on 10 scripted demonstrations; the labelled data add 100 rollouts of the cloned policy with action noise from 0 to 1, and 2,000 pairs of 50-step segments are labelled by a noisy Bradley–Terry teacher that agrees with the true ranking on 86–88% of pairs. Smaller budgets use the first pairs. The authors also labelled pairs blind (two clips side by side in random order, with “can’t tell”): drawer pairs 0–299 and 200–499 (97.6% and 97.7% agreement with the true ranking on decided pairs, 18% and 13% ties; the two agreed on all 73 shared decided pairs) and plate-slide pairs 0–299 (92.7%, 17%). Episodes last 200 steps on the walkers and 150 on the arm. True return is the environment’s reward (Meta-World’s dense reward on the arm), and on the arm we also count successes. Each cell averages 10 runs (5 scorer seeds 2 start states; 10 start states for ensembles), and 20 for standard MPPI in the five main settings and for the perfect scorer on the walkers. Appendix § 10 lists all settings; code and results will be released.
Terms. The scorer is the learned reward model (the verifier in best-of-); the perfect scorer plans with the true reward instead. Gentle and hard search mean and . The distance moved is between the planner’s weights and its proposals in the experiments and in the theory ( near no search). An oracle ranks pairs by true return, which a deployed robot cannot do; in deployment a person would take its place.
Theory: what the planner can and cannot certify
In plain terms, this section shows three things. The scorer can mislead the planner by at most the spread of its errors across the proposals times the distance the planner moves from them (Theorem 1). The best guarantee therefore comes from searching only until times that distance equals the error spread, and from not searching at all once the error spread exceeds the scorer’s own spread (Theorem 2). Along standard MPPI’s path, true return turns down where the scorer stops ranking the planner’s favourite proposals correctly, and over an episode these per-step effects add up exactly (Theorems 5 and 6).
Fix a state. Let give each of the proposals weight (the reference policy of RLHF), and be the true and learned plan returns, and the scorer’s error; labels fix only up to scale and offset, so is measured in the scorer’s units. A plan distribution is any reweighting of the proposals, such as MPPI’s weights; its density gives plan the factor by which it is up-weighted ( doubles its share, drops it). Let be the true gain of selecting with , the same under , and the distance moved: is no search, and spreading the weight evenly over the best plans gives (1 for the best half, for the single best of 256). What matters is , the spread of the scorer’s mistake across the proposals: an error shared by all of them changes no weights and costs nothing; what hurts is overrating some proposals relative to others. Proofs are in Appendix A.
Lemma 1 (Tight change of measure). For every ,
For each , an error of spread that overrates every plan in proportion to its extra weight attains
The bound is attained when the scorer overrates precisely the plans the planner up-weights. It sharpens the change-of-measure bound used for language models [15 a]: it vanishes when the planner does not move, and no smaller bound holds. Against every error of spread at most , the planner can therefore guarantee exactly , and the best it can do is maximize this.
Theorem 2 (Worst-case-optimal search). (i) If , no plan distribution guarantees a gain: , attained by .
(ii) If , is maximized by MPPI, , at a temperature with (for , the hard-search limit ), and the maximum is positive. Along the path, falls from to as search hardens, and rises until and falls after it.
(iii) While no proposal’s weight is clipped to zero, and the true gain is exactly , where .
The stopping rule can be checked from the planner’s own weights. The geometry comes from the error model, not a choice of divergence: knowing only the error’s spread makes the exact worst case (Theorem 1), while sub-Gaussian errors would give a bound in matched by exponential MPPI, an assumption heavy tails break [19].
Scale. Since is known only up to scale, we put it in the scorer’s units by the rescaling that best matches on the proposals (least squares of on ), which leaves an error uncorrelated with and gives . The threshold of (i) then becomes : a positive gain can be certified exactly when the scorer’s Pearson correlation with the true return over the proposals is positive. For jointly Gaussian returns this is the same as ordering a random pair correctly more often than chance; in general it is not, since a scorer can order most pairs correctly and still correlate negatively by badly underrating one high-return plan. As the rescaling shrinks , so the certified gain is positive but measured in ever smaller units: the theorem then guarantees the sign of the gain, not its size. In scale-free form, MPPI with does not lose for any true return whose correlation with over the proposals is at least .
MPPI executes the weighted average of the plans, , not a plan drawn from . The guarantee carries over up to the spread of the plans the weights keep.
Proposition 3 (The executed plan). If is -smooth in the plan on the action box, then
so against every error of spread at most ,
The last term vanishes as the weights concentrate on one plan, and it can be dropped when is concave in the plan.
is unknown for contact-rich tasks, so we measure the averaging term directly (Section E): at least standard deviations except on window-open ().
Corollary 4 (Hard exponential search). On samples with a unique top-scored plan, exponential MPPI with selects it, , and its guarantee is
the worst error overrates the selected plan by . If the top score’s margin grows more slowly than , as does for Gaussian scores, this tends to as grows, for every ; the guarantee of Theorem 2 is never negative.
How large is ? For a linear reward its bound shrinks with the label count at a rate set by how far the planner’s proposals vary beyond the labelled pairs (Theorem 7, Section A). The next result locates where along the exponential path true return turns down.
Proposition 5 (Where search starts to hurt). Let with , and . (i) For every ,
The true return of a plan drawn from the planner’s weights rises exactly while learned and true returns are positively correlated under those weights, and it has a local peak only where that covariance changes sign from positive to negative.
(ii) As ,
where and is the spread of true return over the proposals.
(iii) The executed plan satisfies
on samples the last term vanishes as , so hard search beats the cloned plan only if the selection gain exceeds a price that does not depend on the scorer.
Part (ii) makes the natural axis, as Gao et al. [11] found for language models: to first order true return changes linearly in it, with a slope set by how well the scorer ranks the proposals, not how well it fits its labels (near , , matching Theorem 2(iii)). Part (i) says that exploitation begins where the scorer stops ranking the planner’s own favourites correctly, possibly long before it ranks typical proposals badly. Part (iii) separates exploitation from a cost of sampling that a perfect scorer also pays; drawing one plan from the weights, as inference-time pessimism does (Huang et al. [14]), gives up the averaging term at every temperature, which is why it does worse than averaging on cheetah (167–265) and the drawer.
Over an episode the planner re-plans at every state it visits, and the per-step terms add up exactly.
Proposition 6 (Closed loop). Let the dynamics be deterministic, the cloned policy’s true return from step to the end of the episode, and the return of taking at step and following the cloned policy after it. If at each visited state the planner executes the first action of , its episode return satisfies
where , and are the selection gain, price of noise and averaging term of Theorem 5(iii) at , and
is the horizon gap, by which the -step comparison with the cloned plan misstates the long-run advantage of the first action. If at every visited state the weights are worst-case optimal for an error spread of at least that state’s , then every and
The identity does not make the other terms independent of the scorer, which sets the weights and with them the averaging term and the states visited: with , cloned action 0 and proposals , hard search on a scorer that favours has zero selection gain and loses only the averaging term. Theorem 2 controls the selection terms, given the error spread on the visited states, which can exceed that on the cloned policy’s; we measure the rest (Section F).
Experiments
Harder search lowers true return
Figure 1 shows the main effect. With the crowd scorer, true return falls under harder search on both walkers (cheetah 480 to 262, hopper 468 to 356), while the perfect scorer improves on cheetah (884 to 980). On hopper even the perfect scorer loses (575 to 396): at the hopper falls before the episode ends in most runs, a horizon gap (Theorem 6) that the 20-step return cannot see, so part of the hopper loss is the planner’s own.
On the arm, where the cloned policy is already near the optimum, harder search hurts with 500 synthetic labels: true return falls from 707 to 504 on the drawer, which stays shut in 9 of 20 runs, and from 950 to 436 on plate-slide, while uniform averaging and the cloned policy succeed every time. With 2,000 labels search becomes safe on the drawer (899 to 948) but not on plate-slide (977 to 319).
Human labels are the starkest case. With one author’s 300 blind drawer labels the gentlest search matches uniform averaging (851, 20 of 20 opened), but at every temperature at or below 0.1 the drawer stays shut in all 20 runs, and both authors’ 500 labels fare no better. On plate-slide human labels are exploited less than synthetic ones on the same pairs (1,066 against 499 at hard search), though both authors’ 500 still fall to 681.
Search is not always exploited: on window-open it helps at every label budget (295 without search, 375–392 at hard search; § 4.3), and button-press is a null case (Section H). Hard search still loses with other horizons and proposal noise, except on cheetah with smaller noise (Table 2). Section C gives the remaining numbers for this section.
| Cheetah | Drawer, 500 labels | |||||
| none | 0.01 | none | 0.01 | |||
| , (default) | (10/10) | (18/20) | (11/20) | |||
| (10/10) | (10/10) | (8/10) | ||||
| (10/10) | (0/10) | (0/10) | ||||
| (10/10) | (10/10) | (4/10) | ||||
| (10/10) | (5/10) | (3/10) |
Table 2. Planner settings. True return of uniform averaging (no search) and of standard MPPI with the learned scorer at and 0.01, for proposal noise and horizon ; mean standard error over 10 runs (default row: all runs of the main experiments); successes in parentheses.
Why accurate human labels are exploited. The human labels agree with the true ranking on 97.6% of decided pairs and give the best held-out accuracy of any scorer we trained (0.97), yet they make the worst planning target. Controls on the same 300 pairs point to the 55 pairs the person could not tell apart. Ordering every pair by true return, or dropping 55 random pairs, lets hard search open the drawer in 6 of 10 runs; with only the decided pairs in true order it never opens. What the planner needs is the order of those close pairs. In recorded episodes (Figure 2) the arm parks in an unvisited corner that the scorer rates highly (synthetic labels) or pulls about 10 cm short of the handle (human labels), and the human-label scorer barely penalizes the hand–handle distance (Section C).
Figure 2. One drawer start under three planners (true returns). Uniform averaging opens the drawer. Standard MPPI () on the 500-label scorer lifts the hand into a part of the workspace the data never visited; on the human-label scorer () it pulls in front of the handle without touching it.
True return follows the distance moved
Figure 3a plots every single-scorer MPPI cell against . Within each task the cells fall near one curve through uniform averaging (–), and the sign of the initial slope is clear on every task. A curve fitted on the exponential cells alone predicts the cells to within 16–57% of their spread: at equal distance moved the two rules give similar true return, so the advantage of weights at a fixed temperature (Section 4.4) is mostly that they move less. The walkers first gain and then lose (peaks near and ); the drawer and plate-slide lose from the start, and window-open gains throughout.
The perfect scorer at the same distances (open diamonds) separates exploitation from the planner’s own cost: on cheetah, the drawer and plate-slide the gap opens from the first step and reaches 590–720 points at hard search. On hopper the crowd scorer stays ahead of the true reward up to , possibly because its 200-step segments reward progress that the 20-step return misses.
On the drawer (Figure 3b) the initial slope grows steadily
with the label count, roughly linearly in , and crosses zero near 1,200 labels. The human labels sit below the synthetic labels on the same 300 pairs, though the intervals overlap. On plate-slide, 2,000 labels do not change the sign. Slopes, intervals and fit errors are in Section C.
Two ways to be exploited, and why held-out checks do not warn
Figure 4b separates two patterns by the sign of the change in the scorer’s own estimate of the episode the robot actually executes. On the drawer with synthetic labels the estimate rises sharply as true return falls, which we read as the planner finding states the scorer overrates, the extremal case of Manheim & Garrabrant [26]. On plate-slide and with human labels the estimate falls, which we read as chasing plans that look good at planning time and do not hold up once the planner re-plans, the optimizer’s curse Smith & Winkler [35] in closed loop. Hard search also leaves the labelled data in all three settings, but that alone does not make a scorer exploitable: on window-open most states are off the data even without search, and search still helps (Section F). The two patterns respond to different fixes (Section 4.4).
Why do the usual checks not warn? Table 4 (Section E) compares them with what Theorem 5 says matters, on the planner’s 256 proposals along the cloned policy’s episodes, scored by the learned scorer and the true reward. The initial slope that Theorem 5(ii) predicts matches the measured one to within 6% in eleven of the thirteen settings, and the correlation that sets it is unrelated to the held-out checks: the 500-label drawer scorer has held-out accuracy 0.88 but ranks the planner’s proposals with , while the hopper scorer, with a per-step correlation of only 0.08 on the labelled data, ranks them with . Part (iii) explains why hard search loses to the cloned plan even when selection helps: perturbing the cloned plan costs 0.7–2.2 standard deviations of true plan return, and the scorers recover only part of that by selection. On window-open the cost is only 0.33 and selection recovers 0.7–1.0, which why search helps there.
| Held-out | On the planner’s proposals | |||||||
| acc. | corr. | noise | mean | worst | mean | |||
| Cheetah | 0.75 | 0.17 | 0.08 ±0.01 | 1.59 | 0.18 ±0.07 | −3.03 ±0.41 | 0.06 | 1.4 |
| Hopper | 0.77 | 0.08 | 0.66 ±0.04 | 0.67 | 0.22 ±0.19 | −2.47 ±0.33 | 0.53 | 1.2 |
| Drawer 200 | 0.84 | 0.79 | −0.00 ±0.03 | 0.91 | −0.64 ±0.18 | −7.03 ±0.36 | 0.05 | 1.8 |
| Drawer 300 | 0.84 | 0.85 | −0.01 ±0.02 | 0.91 | −0.21 ±0.06 | −5.92 ±0.57 | 0.07 | 1.9 |
| Drawer 500 | 0.88 | 0.88 | 0.07 ±0.06 | 0.91 | 0.16 ±0.12 | −5.06 ±0.51 | 0.12 | 1.4 |
| Drawer 1k | 0.85 | 0.83 | 0.06 ±0.02 | 0.91 | 0.12 ±0.08 | −5.06 ±0.50 | 0.11 | 1.9 |
| Drawer 2k | 0.85 | 0.81 | 0.17 ±0.01 | 0.91 | 0.43 ±0.07 | −4.12 ±0.40 | 0.23 | 2.0 |
| Drawer human | 0.97 | 0.91 | −0.17 ±0.02 | 0.91 | −0.16 ±0.11 | −6.64 ±0.90 | 0.05 | 2.8 |
| Drawer 500, half on planner | 0.84 | 0.92 | 0.13 ±0.04 | 0.98 | 0.29 ±0.08 | −5.66 ±0.45 | 0.26 | 1.1 |
| Plate 500 | 0.84 | 0.93 | 0.42 ±0.03 | 2.16 | 0.85 ±0.03 | −4.87 ±1.21 | 0.42 | 2.0 |
| Plate 2k | 0.86 | 0.93 | 0.36 ±0.02 | 2.16 | 0.98 ±0.10 | −3.65 ±1.18 | 0.39 | 2.0 |
| Window 500 | 0.86 | 0.85 | 0.36 ±0.02 | 0.33 | 0.98 ±0.21 | −8.46 ±2.61 | 0.39 | 1.0 |
| Window 2k | 0.85 | 0.91 | 0.23 ±0.02 | 0.33 | 0.72 ±0.12 | −5.68 ±1.07 | 0.22 | 2.0 |
Table 4. Table 4. What the scorer knows about the planner’s own proposals. Held-out: label accuracy on held-out pairs and per-step correlation with the true reward on the labelled data. On the proposals, along the cloned policy’s episodes (150–200 states): correlation between learned and true plan returns over the 256 proposals; the price of the proposal noise ; the selection gain of standard MPPI at , on average and at the worst state; and the average gain of the worst-case-optimal rule of Theorem 2 on the five-scorer mean given the true error spread, whose gain at the worst state is 0.00 in every row. Gains in units of the proposals’ spread . Mean ± standard error over 5 scorer seeds. : median ratio of the true error spread to the ensemble’s disagreement.

Figure 3. (a) True return gained over uniform averaging, as a fraction of the perfect scorer’s best return, against the distance (arm tasks: 500 labels). Circles: exponential weights; squares: weights; one point per temperature; curves: fits of ; open diamonds: perfect scorer ( standard error). (b) The drawer for five label budgets and the human labels. (c) Best-of- on the drawer (solid) and plate-slide (dashed), relative to the clean cloned action (dash-dot); (open markers) is one noisy action.
Figure 4. (a) Drawer: true return against label budget at gentle and hard search; green: 500 labels, half on the planner’s own trajectories (Section 4.4). (b) Change in the scorer’s estimate of the executed episode from to , against the ratio of true returns; diamonds: human labels. (c) Each fix at , relative to standard MPPI at .
A scorer can therefore be checked only where it will be optimized: on the planner’s own proposals and, after the first step, on states the cloned policy never visited. Whole episodes bear this out (Table 5). On the states hard search visits, the scorer ranks the proposals worse than on the cloned policy’s states in every exploited setting but cheetah, where it is near zero on both. Summed over a drawer episode, the selection gains are with 500 labels and with human labels, against with the perfect scorer. On plate-slide they stay positive, and the loss is in the horizon gap: the executed plans’ 20-step advantage over the cloned plan overstates their long-run effect.
| runs | visited | cloned | ||||||||
| Cheetah, | 5 | 237 | 140 | 1818 | (34%) | |||||
| Hopper, | 5 | 269 | 426 | 754 | (13%) | |||||
| Drawer 500, uniform averaging | 5 | 854 | 898 | 0 | 387 | (36%) | ||||
| Drawer 500, | 5 | 765 | 898 | 301 | (46%) | |||||
| Drawer 500, | 5 | 472 | 898 | 75 | 0 | (62%) | ||||
| Drawer, perfect scorer, | 5 | 1099 | 898 | 195 | 0 | (0%) | – | |||
| Drawer human, | 5 | 218 | 898 | 69 | 0 | (91%) | ||||
| Plate 500, | 5 | 516 | 1115 | 216 | (52%) | |||||
| Window 500, | 5 | 403 | 381 | (19%) |
Table 5. Closed-loop decomposition (Theorem 6) of standard MPPI episodes against the cloned policy: episode return , the cloned policy’s return from the same start, and the episode sums of the selection gain , the price of noise , the averaging term and the horizon gap , so that (exact in every run, to ). : mean per-state correlation of the scorer with the true plan return over the 256 proposals, on the states the planner visits (share of states below zero in parentheses) and on the cloned policy’s own states (one-step probe, same scorer). Mean over runs, with the standard error over runs for and . Each run is paired with from its own start, computed in the planner’s simulator copy. On the arm run uses scorer seed and start ; the walker runs use starts 0 and 1 only (three from start 0, where the cloned cheetah scores 1), so their (cheetah 140, hopper 426) is not the mean over ten starts of Table 1 (412, 502). The cloned cheetah’s return also depends sharply on small state differences (from start 1: 347 in the copy, 662 in the environment), the source of its large horizon gap. Best read through . On the states the planner visits the learned scorer’s per-state correlation is lower than along the cloned policy’s own episodes in every exploited setting except cheetah, where it is near zero on both; on the drawer with human labels it is negative at 91% of visited states.
The same proposals test Theorem 2. Given the true error spread, the worst-case-optimal rule does not lose at any of the 2,042 states and gains 0.05–0.53 standard deviations on average; standard MPPI at hard search gains more on average in ten of thirteen settings but loses 1.6–8.0 at its worst state, which a closed-loop planner has many chances to meet. The unknown is the error spread, which the ensemble’s disagreement underestimates by a factor of up to 2.8 (Table 4): scorers trained on the same labels share their mistakes. Algorithm 1 instead calibrates it from the share of 100 pairs of the planner’s own proposals that the scorer orders correctly (Kruskal, [18]). Ranked by an oracle, it loses at most 0.22 standard deviations at its worst state in twelve of thirteen settings and keeps 47–93% of the true-spread rule’s gain where that gain exceeds 0.2; a lower confidence bound on cuts the loss further, at a cost in gain (Section G). The rule cannot lose at a state whose own correlation is at least (Theorem 8), but mistakes the labeller shares with the scorer, sampling noise and pooling over states can each put above a state’s own.
What helps, and where
Table 1 and Figure 4c compare the fixes at hard search; none wins everywhere. Among fixed temperatures, weights on the mean of five scorers do best on the walkers, and the cautious ensemble (mean minus one standard deviation; Coste et al., [7]) helps less. On the drawer with synthetic
@page 6/21 starts_mid=1 ends_mid=1

Figure 3 (PDF p. 6). (a) True return gained over uniform averaging, as a fraction of the perfect scorer’s best return, against the distance (arm tasks: 500 labels). Circles: exponential weights; squares: weights; one point per temperature; curves: fits of ; open diamonds: perfect scorer ( standard error). (b) The drawer for five label budgets and the human labels. (c) Best-of- on the drawer (solid) and plate-slide (dashed), relative to the clean cloned action (dash-dot); (open markers) is one noisy action.
Figure 4 (PDF p. 6). (a) Drawer: true return against label budget at gentle and hard search; green: 500 labels, half on the planner’s own trajectories (Section 4.4). (b) Change in the scorer’s estimate of the executed episode from to , against the ratio of true returns; diamonds: human labels. (c) Each fix at , relative to standard MPPI at .
why search helps there.
A scorer can therefore be checked only where it will be optimized: on the planner’s own proposals and, after the first step, on states the cloned policy never visited. Whole episodes bear this out (Table 5). On the states hard search visits, the scorer ranks the proposals worse than on the cloned policy’s states in every exploited setting but cheetah, where it is near zero on both. Summed over a drawer episode, the selection gains are with 500 labels and with human labels, against with the perfect scorer. On plate-slide they stay positive, and the loss is in the horizon gap: the executed plans’ 20-step advantage over the cloned plan overstates their long-run effect.
The same proposals test Theorem 2. Given the true error spread, the worst-case-optimal rule does not lose at any of the 2,042 states and gains 0.05–0.53 standard deviations on average; standard MPPI at hard search gains more on average in ten of thirteen settings but loses 1.6–8.0 at its worst state, which a closed-loop planner has many chances to meet. The unknown is the error spread, which the ensemble’s disagreement underestimates by a factor of up to 2.8 (Table 4): scorers trained on the same labels share their mistakes. Algorithm 1 instead calibrates it from the share of 100 pairs of the planner’s own proposals that the scorer orders correctly (Kruskal, [18]). Ranked by an oracle, it loses at most 0.22 standard deviations at its worst state in twelve of thirteen settings and keeps 47–93% of the true-spread rule’s gain where that gain exceeds 0.2; a lower confidence bound on cuts the loss further, at a cost in gain (Section G). The rule cannot lose at a state whose own correlation is at least (Theorem 8), but mistakes the labeller shares with the scorer, sampling noise and pooling over states can each put above a state’s own.
What helps, and where
Table 1 and Figure 4c compare the fixes at hard search; none wins everywhere. Among fixed temperatures, weights on the mean of five scorers do best on the walkers, and the cautious ensemble (mean minus one standard deviation; Coste et al., [7]) helps less. On the drawer with synthetic
Algorithm 1 Calibrated MPPI
Once: rank pairs of the planner’s proposals (by an oracle or a person); let be the share the scorer orders the same way; (corrected for ties, Section G);
for each control step do
draw , ; score each by the five-scorer mean
if then
for all (no search)
else
find with by bisection; , such that
end if
execute the first action of $\frac{1}{N}\sum_i w_i A_i
end for
labels the ranking reverses: the cautious ensemble restores 801 (10/10) while weights do not help. This fits the two patterns above: where the planner reaches states the scorers overrate, they disagree and the penalty finds them; where it chases noise, they agree, and only gentler search helps. Automatic temperatures are the most even across tasks, and a disagreement guard and the worst-case-optimal rule with twice the disagreement help on the drawer but not on the walkers. With human labels every hard-search fix fails.
Calibrated from 100 oracle pairs, Algorithm 1 is the safest fix (Table 1 and Table 7). Each of five independent calibration sets per task, drawn on start states the evaluation never uses, calibrates its own controller. On the drawer and plate-slide every controller succeeds in all its runs and keeps on average 86–94% of the cloned policy’s return, against 611 (8/10) and 351 (6/10) for hard search on the same five scorers. With human labels every set declines to search. The rule also gains on hopper with every set, lands between uniform averaging and the cloned policy on cheetah, and pays for caution on window-open, where search helps (Section C).
| Cheetah | Hopper | Drawer 500 | Drawer human | Plate 500 | |
| Cloned policy, run directly | (10/10) | (10/10) | (10/10) | ||
| Uniform averaging (no search) | (10/10) | (10/10) | (10/10) | ||
| Perfect scorer, | † | † | (10/10) | (10/10) | (10/10) |
| Learned scorer, gentle () | † | † | † (18/20) | ‡ (20/20) | † (20/20) |
| Learned scorer, hard search () | |||||
| Standard (exp. weights) | † | † | † (11/20) | † (0/20) | † (15/20) |
| weights | (5/10) | (0/10) | (7/10) | ||
| Cautious ensemble (mean − sd) | (10/10) | (1/10) | (8/10) | ||
| on ensemble mean | (6/10) | (0/10) | (7/10) | ||
| Automatic guard | (9/10) | (0/10) | (6/10) | ||
| Learned scorer, adaptive temperature or a different optimizer | |||||
| Automatic (, ) | (7/10) | (0/10) | (10/10) | ||
| Worst-case-optimal (Thm. 2, ) | (8/10) | (0/10) | (10/10) | ||
| Worst-case-optimal (Thm. 2, ) | (10/10) | (2/10) | (10/10) | ||
| Worst-case-optimal , 100 oracle pairs | ‡ | ‡ | ‡ (50/50) | ‡ (50/50) | ‡ (50/50) |
| CEM, cautious | (5/10) | (0/10) | (0/10) | ||
| CEM, mean of five | ∗ | ∗ | (3/10) | – | (0/10) |
| CEM, optimistic (UBP2-inspired) | (2/10) | (0/10) | (0/10) |
Table 1. Table 1. True return at hard search, mean standard error over 10 runs per cell († 20, ∗ 5); successes over runs in parentheses on the arm. ‡ Mean standard deviation across five independent 100-pair calibration sets ranked by true return (an oracle), drawn on start states the evaluation never uses, each calibrating its own controller (Table 7). Bold: best learned-scorer method per column (the calibrated temperature on the arm, the automatic one on the walkers).
| Cheetah | Hopper | Drawer 500 | Drawer human | Plate 500 | Window 500 | |
| Cloned policy, run directly | (10/10) | (10/10) | (10/10) | (10/10) | ||
| Uniform averaging (no search) | (10/10) | (10/10) | (10/10) | (10/10) | ||
| Standard MPPI, single scorers | (11/20) | (0/20) | (15/20) | (10/10) | ||
| Standard MPPI, five-scorer mean | (8/10) | (0/10) | (6/10) | (10/10) | ||
| Pooled, true order, point | (10/10) | (10/10) | (10/10) | (10/10) | ||
| Pooled, true order, lower bound | (10/10) | (10/10) | (10/10) | (10/10) | ||
| Pooled, 85% teacher, point | (10/10) | (10/10) | (10/10) | (10/10) | ||
| Five independent sets, point | , 264 | , 517 | , 824 (50/50) | , 846 (50/50) | , 999 (50/50) | , 308 (50/50) |
| – | – | – | – | – | – | |
| Five independent sets, lower bound | , 264 | , 500 | , 852 (50/50) | , 852 (50/50) | , 1036 (50/50) | , 303 (50/50) |
| – | – | – | ||||
| Five held-out sets, point | , 264 | , 523 | , 818 (50/50) | , 852 (50/50) | , 806 (50/50) | , 309 (50/50) |
| – | – | – | – | – | ||
| Five held-out sets, lower bound | , 264 | , 472 | , 835 | , 852 (50/50) | , 998 (50/50) | , 304 (50/50) |
| – | – | – | – |
Table 7. The calibrated rule in closed loop on the same ten starts per column. Top: references, and hard search () with single scorers (20 runs) and on the five-scorer mean that the calibrated rule uses. Pooled: the median over 50 draws of 100 proposal pairs ranked by true return (point estimate and one-sided Wilson lower bound at nominal 95%) or by a noisy teacher that agrees with the true order on 85% of pairs. Independent sets: five disjoint sets of exactly 100 pairs, each calibrating its own controller; mean standard deviation across sets, then the worst set. Held-out sets: the same, with the pairs drawn on start states 10–19, which the evaluation never uses. is the error spread as a share of the scorer’s spread over the proposals, and means no search (uniform averaging). Otherwise mean standard error over runs; successes over runs in parentheses.
Calibrating from people. Calibrated instead from one author’s comparisons of 100 such pairs, the rule trusts the scorer trained on that author’s labels and opens the drawer in only 5 of 10 runs (574); the other author’s comparisons leave it barely searching (845, 10/10). Both people called about 70% of pairs ties, which inflates the concordance on the rest; corrected for ties, the first author’s calibration scores 822 (10/10); Table 8. Calibration against the scorer’s own labeller cannot detect a bias they share; an independent evaluator, the tie correction or the lower bound limits the damage.
The calibrated rule on the drawer scorer trained on one author’s clip labels ( on the planner’s proposals, Table 6), by who ranks the 100 calibration pairs. with its one-sided Wilson lower bound in parentheses, as measured on the decided pairs and corrected for can’t-tell answers; then the true return (successes over 10 runs) of the rule at each estimate. Uniform averaging scores 852 (10/10) and the cloned policy 898.
| Calibration pairs ranked by | decided pairs | (lower bound) as measured | (lower bound) ties corrected | return, as measured point | return, as measured lower bound | return, ties corrected point | return, ties corrected lower bound |
| Oracle (true return), five sets | 100 of 100 | −0.45 to −0.09 | (no ties) | 852 (10/10) | 852 (10/10) | – | – |
| Author whose labels trained the scorer | 32 of 100 | 0.63 (0.23) | 0.36 (0.12) | 574 (5/10) | 839 (10/10) | 822 (10/10) | 846 (10/10) |
| The other author | 29 of 100 | 0.16 (−0.30) | 0.08 (−0.16) | 845 (10/10) | 852 (10/10) | 846 (10/10) | 852 (10/10) |
| Both authors pooled | 61 of 200 | 0.42 (0.11) | 0.23 (0.06) | 812 (10/10) | 846 (10/10) | 838 (10/10) | 846 (10/10) |
Table 8. The calibrated rule on the drawer scorer trained on one author’s clip labels ( on the planner’s proposals, Table 6), by who ranks the 100 calibration pairs. with its one-sided Wilson lower bound in parentheses, as measured on the decided pairs and corrected for can’t-tell answers; then the true return (successes over 10 runs) of the rule at each estimate. Uniform averaging scores 852 (10/10) and the cloned policy 898.
A UBP2-inspired optimistic CEM planner (CEM as in Nabail et al. [28] on the five-scorer mean plus one standard deviation) is the strongest optimizer with the perfect scorer and the most exploited on the arm (plate-slide 68–79 with no successes). Its collapse persists without optimism and at matched budget (Section C), though we do not isolate elite selection. Labelling where the planner goes. The scorer is exploited where its labels did not reach, so, as active preference learning for MPC does El Hasnaouy et al. [9], we spend half of a 500-label drawer budget on the planner itself: 250 new pairs from MPPI episodes planned with a scorer trained on the first 250 offline pairs (Section B). Hard search then scores 834 (8/10) instead of 504 (11/20), and the cautious ensemble 975 (10/10), level with 2,000 offline labels and above the cloned policy (898; Figure 4a).
Best-of- with a learned verifier
Beyond MPPI, we test best-of- action selection (Kwok et al. [20]) with a state-based verifier trained on 500 or 2,000 synthetic pairs over perturbed actions (§ 17). The candidates are noisy copies of the cloned action, so the reference is one noisy action (755 on the drawer, 878 on plate-slide), and a perfect verifier recovers the clean action’s return (898 and 1,111) for every (Figure 3c). With 500 pairs selection falls well below one noisy action from on both tasks (530 and 375), and on plate-slide already at . With 2,000 pairs selection beats one noisy action at every .
A learned dynamics model and an image scorer
Neither the simulator nor the state scorer drives the effect (Section D). Planning through a five-member MLP ensemble fit to the logged transitions (Chua et al. [6]), the perfect scorer still gains (cheetah 688, drawer 1,013 at hard search, against 412 and 898 for the cloned policy), while the learned scorer still loses under harder search (cheetah 377 to 255, drawer 842 to 650). A CNN scorer on camera frames, trained on the same 500 drawer labels, ranks held-out pairs as well as the state scorer (0.87 against 0.88), yet hard search against it opens the drawer in 1 of 10 runs (283, against 791 at ; state scorer 504).
Related work
Reward overoptimization. Gao et al. [11] fitted how true quality falls with from the reference policy under best-of- and RL against a learned reward model. Ensembles ([7, 8, 47]), weight averaging (Ramé et al. [31]) and constraints (Moskovitz et al. [27]) mitigate it. Huang et al. [15] showed that regularization gives guarantees KL regularization cannot, and Huang et al. [14] analyzed best-of- and proposed inference-time pessimism; our Theorem 1 centers and tightens their change-of-measure bound, and Theorem 2 turns it into a temperature MPPI can check from its own weights and a no-search threshold. Theorem 7 uses the Bradley–Terry estimation bound of Zhu et al. [48]; Aouad et al. [1] derive label-count scaling laws for language models. Skalse et al. [34]; Pan et al. [29]; Laidlaw et al. [21] define reward hacking. These works study language models or RL policies, not a planner in closed loop.
Preferences for robot control. Preference-based reward learning ([5, 33, 23, 13]) is known to pick up spurious features (Tien et al. [37]); Uni-RLHF (Yuan et al. [46]) provides the crowd labels we use. Learned preference rewards are now optimized directly by MPC (Krupa et al. [17]; ); El Hasnaouy et al. [9]) and by CEM-based planning in UBP2 (Nabail et al. [28]), which plans optimistically to collect labels; we study such planners at deployment.
Planning against learned models. Model-based RL guards against exploiting learned dynamics with ensembles, penalties (Chua et al., [6]; Yu et al. (2020b); Kidambi et al. [16]) or policy constraints (Hansen et al., [12]; Lin et al. [24]); imperfect world models are exploitable (Bhamidipaty et al., [3]); see also conservative MPPI (Yoon & Kim, [43]). MPPI is KL-regularized by construction (Williams et al., [42]); Tsallis-divergence MPPI (Wang et al., [40]) gives the rule. CriticHack (Luo et al. [25]) evaluates visual reward models under policy optimization.
Limitations and conclusion
Our tasks are simulated, our scorers are small networks, the learned-model and image checks cover only cheetah and the drawer, the human labels come from the two authors on two tasks, and most cells have 10 runs. Most calibration pairs are ranked by true return or a simulated labeler, an oracle a deployed robot lacks, apart from two people’s 100 comparisons; the best-of- verifiers use synthetic labels. The guarantee of Theorem 6 covers only the selection terms, and the label-count corollary assumes a linear reward. The calibrated rule is an empirical safeguard, not a per-state certificate (Theorem 8). The distance curves and the label trend summarize the range we tested and are not laws to extrapolate, and the two exploitation patterns of § 4.3 are our reading, not proven mechanisms. Window-open, button-press and two Adroit hand tasks are null or limit cases (Section H).
In practice, temperature, sample count and elite fraction are an optimization budget against the scorer, set by its quality on the planner’s own proposals. A calibrated temperature was the safest default, given an evaluator independent of the labels; labelling where the planner goes helped most.
Impact Statement
This work shows that robots which optimize rewards learned from human preferences can behave worse than the policy they start from, without warning from standard validation, and identifies settings and checks that reduce that risk. The intended effect is safer deployment of learned objectives in robotics. The experiments are in simulation and involve no human subjects beyond the two authors’ own labels.
Use of Generative AI
During the preparation of this work the authors used Claude (Anthropic) to search the literature, to write and run the experiment code, and to draft and edit the text. The authors reviewed and edited the content as needed and take full responsibility for it. The code includes a script that recomputes every number in the paper from the stored result files.
Proofs
Proof of Theorem 1. With , , which by Cauchy–Schwarz under is at most in absolute value. For , the error has spread and attains the bound.
Proof of Theorem 2. (i) By Cauchy–Schwarz, , so . (ii) is concave in on the convex set , so a point that satisfies the Karush–Kuhn–Tucker conditions is a global maximizer. At the gradient of is ; with a multiplier for and multipliers with for , stationarity reads , whose solution is with and set by normalization: the MPPI weights at a temperature with . Conversely, the weights at any satisfy these conditions with , so maximizes . Next, maximizes , so it maximizes among all with ; call this maximum . is concave: if attain , the mixture is feasible at by the triangle inequality and has . Along the path is therefore concave in , so it rises to its maximizer and falls after, and the maximizer of moves to larger as falls, so decreases as grows. On samples, for no weight is clipped, , and ; as , because . The product is continuous in , so exists for ; for , is maximized by putting all weight on the top-scored plans, the limit of the path as . In both cases and for any unclipped . (iii) Unclipped, , so and .
Proof of Theorem 3. lies in the action box, which is convex. By -smoothness, for every plan , . Taking , the linear term vanishes, which gives the first inequality; the second follows from Theorem 1. For concave , Jensen’s inequality gives directly.
Proof of Theorem 4. As the exponential weights tend to a point mass on the top plan : , , . By Theorem 1 the worst error is , which is on . If , the guarantee . By Theorem 2, .
Corollary 7 (Labels needed before search is safe). Let labels follow a correctly specified Bradley–Terry model with , summed over each segment and , and let come from the maximum-likelihood estimate constrained to on labelled pairs whose feature differences have second-moment matrix . With the covariance of plan features under and the coverage , with probability , for any , with the constants and of Zhu et al. [48]. A positive gain can be certified once this is below , which holds for at any with .
The labels needed grow linearly with , how much further the proposals vary than the labelled pairs did; wider proposal noise and states the labelled data never reached both raise it.
Proof of Theorem 7. The scorer’s error on a plan is , with the plan’s summed features, so with , which is at most . Lemma 3.1 of Zhu et al. [48] bounds by with probability , with . A positive gain can be certified when , by Theorem 2, and solving the bound for gives the condition.
Proof of Theorem 5. (i) has . Then . With , . Since KL is increasing in , the sign of is the sign of the derivative along the distance as well, and a local maximum needs that sign to change from positive to negative; a covariance that touches zero without changing sign gives none. (ii) Expanding at : and , so and . (iii) The decomposition is an identity. On samples with a unique maximizer of , tends to a point mass at that sample, so tends to it and
When the probe pools many states, KL and the gain are averaged over states, and the slope in (ii) becomes , which is the predicted slope reported in Appendix § 13.
Proof of Theorem 6. With deterministic dynamics , with (and after termination). Summing over the episode telescopes to . Adding and subtracting in each term and splitting it as in Theorem 5(iii) gives the identity, along every realization of the proposal noise. If the weights maximize for some , then by Theorem 1 . ◻
Setup details
Planner. MPPI with sequences, horizon , noise , warm start from the cloned policy’s closed-loop plan, rolled out with the true MuJoCo dynamics (on the arm, a copy of the Meta-World model with added position sensors whose rollouts match the environment exactly). Temperatures on standardized per-step scores. Uniform averaging is MPPI with : the weights are uniform and it executes the first action of the average of the 256 clipped proposals. Clipping at the action bounds makes this average differ from the clean cloned plan (over 20 steps, by 0.4 standard deviations of plan return on the drawer, 0.7 on cheetah, 1.9 on plate-slide and 0.03 on hopper, in the probe), so it is the end of the same planner, with the same proposals as every other temperature, and the reference for the distance curves. The cloned policy run directly executes its own action, with no proposals, and scores higher on every task (cheetah 412, hopper 502, drawer 898, plate-slide 1,111, window-open 372 on the same ten starts).
Statistics. The resampling unit is a run, one closed-loop episode. A single-scorer cell pairs scorer seeds 0–4 with two or four start states each; an ensemble cell uses one ensemble on ten starts. Standard errors are over runs, so they include both scorer-seed and start-state variation. Bootstrap intervals resample runs within each cell; the hierarchical version, used for the slopes and the label-budget trend in § 4.2, resamples the five scorer seeds once per draw for all cells of a setting and then runs within each seed. The five calibration sets of Table 7 are disjoint, so their spread includes the sampling of the pairs.
The automatic temperature sets from the ensemble’s disagreement at each step, . The guard compares the ensemble standard deviation of the top 5% of plans by mean score with its average over all plans and uses the cautious ensemble with exponential weights when the ratio exceeds 1, otherwise weights on the mean. CEM: 6 rounds of 128 samples, 16 kept, minimum standard deviation 0.05, executing the first action of the final mean. The two variants left out of Table 1 do no better at \chi^2$ weights on the cautious ensemble (cheetah 439, hopper 348, drawer 625 with 8/10, human labels 229 with 0/10, plate-slide 414 with 7/10) and MPPI on the optimistic ensemble, mean plus one standard deviation (331 and 221 over 5 runs, 467 with 5/10, 198 with 0/10, 222 with 4/10).
Scorers and cloned policies. Scorer: MLP with two hidden layers of 256, Adam (learning rate , weight decay ), 60 epochs, batch 64, 10% of pairs held out; output standardized per step on the labelled data. Cloned policy: MLP with two hidden layers of 256 and tanh output, Adam (), 20,000 steps. Locomotion: crowd labels from Uni-RLHF for the D4RL medium-replay datasets (2,000 pairs, segment length 200), loaded with a restricted unpicker. Arm: 10 scripted demonstrations (task variations 40–49) for cloning; the labelled data add 100 rollouts of the cloned policy with action noise 0–1 (variations 10–39); evaluation on variations 0–9.
Labelling where the planner goes. A scorer trained on the first 250 offline drawer pairs drives standard MPPI ( and 0.1) for 20 episodes on training positions; 250 new pairs compare 50-step segments of those episodes with each other or with the offline data, labelled by the same synthetic teacher.
Best-of-. 20,000 state–action-pair comparisons per task from states of the labelled data; both actions are the cloned action plus noise; the Bradley–Terry teacher prefers the action closer to the scripted expert action with temperature set by bisection for 85% agreement. Verifier: MLP on (state, action), two hidden layers of 256, same optimizer and epochs as the scorers, five seeds per budget. Episodes of 150 steps on variations 0–9.
Further results for Section 4
This appendix gives the numbers that Section 4 summarizes and that are not in a table.
Harder search (Section 4.1). With the crowd scorer, hopper’s true return is 310, 272, 320 and 356 at , 0.3, 0.1 and 0.01. With the perfect scorer it is 575 at and 351–396 at ; the hopper falls before step 200 in 15–20 of 20 runs at , against 1 of 20 at . On the drawer with 500 labels, the scorer’s own estimate of the executed episode rises twelve-fold (56 to 674) while true return falls from 707 to 504. With one author’s 300 human labels the drawer scores 201–216 at every (0 of 20 opened); both authors’ labels on 500 pairs give 246 at (0 of 10). On plate-slide at hard search, one author’s 300 labels keep 1,066 (10 of 10 successes), against 499 (8 of 10) with synthetic labels on the same pairs and 886 (8 of 10) with the same pairs in true-return order; both authors’ 500 labels give 681 (6 of 10).
Human-label controls. On the 300 drawer pairs the person’s labels agree with the true ranking on 97.6% of decided pairs, against 86% for the synthetic teacher on the same pairs, and the scorer trained on them correlates per step with the true reward on the labelled data (0.91) better than any drawer scorer trained on offline synthetic labels. At , ordering every pair by true return gives 640 (6 of 10 opened), and dropping 55 random pairs gives 672 (6 of 10). With only the decided pairs in true order the drawer never opens, whether the ties are kept as indifference (268) or dropped (233; 190 with the person’s own answers on the decided pairs). The person tied half of the 63 pairs in which neither clip opens the drawer, against a tenth of the rest. A linear fit of each scorer’s per-step output on four state features gives the human-label scorer a standardized weight of only on the hand–handle distance, against to with synthetic labels, and its largest weights to pulling (0.29) and to the drawer’s opening (0.63).
Distance moved (Section 4.2). The fits of Figure 3a use 7–11 cells per task. The 90% hierarchical bootstrap intervals for the initial slope (Section B) are cheetah 306 to 664, hopper 163 to 330, drawer to , plate-slide to and window-open 6 to 112. A curve fitted on the exponential cells alone predicts the cells with a mean absolute error of 38% of their spread on cheetah, 57% on hopper, 23% on the drawer (16% with human labels), 22% on plate-slide and 29% on window-open. The perfect scorer’s points average 20 runs on the walkers and 10 on the arm; its highest point is (averaging’s return the perfect scorer’s best return), for example 0.76 on cheetah and 0.08 on plate-slide. At hard search the gap between the perfect and the learned scorer is 175 points on window-open, and on hopper the crowd scorer falls behind the true reward beyond ( to ). On the drawer the initial slopes are , , , and for 200, 300, 500, 1,000 and 2,000 labels: per decade of (90% hierarchical interval to ), crossing zero near 1,200 labels (1,010–1,420). The human labels give ( to ), against for synthetic labels on the same 300 pairs. On plate-slide with 500 labels and with 2,000. In Figure 4a the cautious ensemble rescues 500 and 1,000 drawer labels but not 200 or the human labels.
Mechanisms (Section 4.3). On plate-slide and with human labels the scorer’s estimate of the executed episode falls from 27–43 at to between and at . The initial slope predicted by Theorem 5(ii) misses the measured one by 24% and 12% (about one standard error) with 200 drawer labels and with human labels, and by at most 6% in the other eleven settings.
Fixes (Section 4.4). On window-open the calibrated rule scores 309–318 across the five held-out calibration sets, against 392 for standard MPPI at hard search and 372 for the cloned policy run directly. With the perfect scorer the optimistic CEM planner is the strongest optimizer (cheetah 1,244, plate-slide 1,165). CEM on the plain five-scorer mean, without optimism, still collapses on the arm (Table 1), while MPPI with CEM’s budget of 768 rollouts per step scores 460 on the drawer and
| Task | Planning on, scorer | none | 1 | 0.3 | 0.1 | 0.01 | |
| Cheetah | simulator, learned | 233 78 | 480 38† | 466 13† | 349 25† | 264 17† | 262 25† |
| simulator, perfect | 233 78 | 884 10† | 954 48† | 953 45† | 971 45† | 980 21† | |
| learned model, learned | 262 87 | 377 83 | 401 64 | 394 30 | 252 34 | 255 40 | |
| learned model, perfect | 262 87 | 693 53 | 761 14 | 633 54 | 675 18 | 688 18 | |
| Drawer 500 | simulator, learned | 852 1 (10/10) | 707 41† (18/20) | 539 66† (11/20) | – | 497 77† (9/20) | 504 74† (11/20) |
| simulator, perfect | 852 1 (10/10) | – | – | – | 1089 11 (10/10) | 1090 7 (10/10) | |
| learned model, learned | 854 1 (10/10) | 842 8† (20/20) | 769 33† (19/20) | – | 610 57† (13/20) | 650 48† (14/20) | |
| learned model, perfect | 854 1 (10/10) | 899 2 (10/10) | 963 3 (10/10) | – | 1009 7 (10/10) | 1013 8 (10/10) | |
| simulator, image scorer | 852 1 (10/10) | 791 50 (9/10) | 430 86 (3/10) | – | 244 23 (0/10) | 283 45 (1/10) |
Table 3. Planning on a learned dynamics model and with an image scorer (§ 4.7). True return of standard MPPI per temperature and of uniform averaging (none); mean standard error over 10 runs (†20); drawer successes in parentheses. Simulator: planning on the true dynamics, as in the rest of the paper. Learned model: a five-member MLP ensemble fit to the logged transitions; the perfect scorer applies the true reward function to its predicted states. The image scorer is a CNN on 64 64 frames trained on the same 500 drawer labels as the state scorer.
303 on plate-slide. Spending half of the drawer budget on the planner’s own trajectories turns the distance curve’s initial slope from −250 to +33.
Learned dynamics and an image scorer
Learned dynamics. An ensemble of five MLPs (three hidden layers of 256, SiLU) predicts the change of the observation from the observation and action. Each member is trained on its own bootstrap sample of the labelled data’s transitions (drawer 16,390; cheetah the 201,798 D4RL medium-replay transitions) with squared error on standardized targets, Adam (learning rate , batch 256; 40,000 steps on the drawer, 60,000 on cheetah), keeping the checkpoint with the lowest error on 10% of held-out episodes. On the drawer it predicts the hand, the finger opening and the drawer handle; the rest of the observation (the previous frame and constants) is filled in exactly. Over 20 logged actions on held-out episodes the ensemble mean is off by 0.6 cm in the handle position, against 5.5 cm of typical movement, and by 2.0 cm in the hand; on cheetah its forward-velocity error (1.04) is close to the typical change (1.19). In planning, each approach is rolled out by one randomly chosen member, the cloned policy’s plan by the ensemble mean, and the first action is executed in the real environment. The perfect scorer computes Meta-World’s dense reward exactly from the predicted drawer observations, and on cheetah the predicted forward velocity minus the control cost (correlation 0.99 with the logged reward). Scorers, temperatures, starts and episode lengths are those of the main runs (on the drawer 20 runs per temperature with the learned scorer, 10 with the perfect one).
Image scorer. We replay every episode of the drawer data from its seed and logged actions (the observations match the log to ) and render one 64 64 RGB frame per step with MuJoCo’s offscreen renderer from a fixed camera that sees the hand and the drawer, with shadows, reflections and the far arm links switched off. The scorer is a CNN on one frame (convolutions with 32, 64 and 64 channels, then 256 units); a segment’s score is five times the sum over every fifth frame. It is trained on the first 500 drawer pairs with the Bradley–Terry loss, with the state scorer’s split, optimizer and 60 epochs, and random shifts of up to 2 pixels. Over five seeds it ranks 87% of held-out pairs correctly (state scorer 88%), correlates 0.92 per frame with the true reward on the labelled data (state scorer 0.88), and orders 93% of all 2,000 drawer pairs like the true return. In planning, each of the 256 proposals is scored from the frames after its environment steps 5, 10, 15 and 20, rendered from the simulated rollout (1,024 frames per control step at about 2.3 ms each).
Results. Uniform averaging on the learned model is close to the simulator’s (drawer 854 against 852, cheetah 262 against 233). The model costs the perfect scorer about 290 on cheetah and 80 on the drawer at hard search, but search on it still beats the cloned policy run directly at every temperature on cheetah (633–761 against 412) and for on the drawer (963–1,013 against 898). The learned scorer loses to harder search on the model as on the simulator, ending at the same level on cheetah (255 against 262) and higher on the drawer (650, 14 of 20 opened, against 504, 11 of 20). The image scorer is exploited more than the state scorer at every temperature below 3: 430 (3 of 10 opened) at , 244 (0 of 10) at 0.1 and 283 (1 of 10) at 0.01, against 539, 497 and 504, although both are about equally accurate on the labelled data.
One-step probe
Along 10 episodes of the cloned policy, at every tenth step, we build the planner’s 256 proposals exactly as MPPI does and roll each out with the true dynamics, recording its learned and true 20-step returns; we also roll out the clean cloned plan and the weighted-mean plan at 17 temperatures from 100 to 0.01 for both weighting rules. Table 4 reports the mean correlation over states, the price of noise and the selection gain at in units of the mean proposal spread , and the code reports the predicted and measured initial slopes and the distance at which the tilted covariance first reaches zero. For Theorem 2 we score the proposals with all five scorers and use their mean . The true error spread is that of after the best positive rescaling of into the scorer’s units, (and , so no search, when ); the rule’s temperature solves by bisection. The deployable version replaces with times the ensemble’s disagreement, the root mean square over proposals of the standard deviation across scorers after centering each scorer over the proposals, and is the per-state ratio of the true error spread to this disagreement. The calibrated version draws pairs of proposals at random probed states, labels each pair by its true plan return, and sets from the fraction of pairs the five-scorer mean orders correctly; the rule then uses at every state, so no search when . The probed states lie on cloned-policy episodes from the ten start states that the closed-loop evaluation uses. The five calibration sets of Table 1 come instead from a second probe on start states 10–19, which no closed-loop run uses (on the arm, object positions that the labelled data also cover). The pooled closed-loop runs of Table 7 use the median over 50 draws of 100 pairs from the first probe (cheetah 0.06, hopper 0.70, drawer 0.06, human labels −0.27, plate-slide 0.49, window-open 0.48), and Section G studies how the estimate varies. The averaging term of Theorem 3, , is 0.25–0.85 standard deviations at uniform weights and shrinks to about zero at hard exponential search on every task except window-open, where it is −0.10 at uniform weights and −0.15 at worst, so there the executed plan can fall short of the guarantee by that much.
Episodes, data support and validation
Closed-loop decomposition. Table 5 measures the terms of Theorem 6 along MPPI episodes. At every visited state the planner’s 256 proposals, its executed average and the clean cloned plan are rolled out with the true reward, and is the true return of the cloned policy from that state to the end of the episode; the identity then holds in every run to rounding. Uniform averaging pays a price of noise of 387 over a drawer episode and recovers 225 of it by averaging. Hard search removes the averaging term, so it must recover the price by selection: the perfect scorer does (+632), the learned scorers do not on the drawer (−151, and −373 with human labels). The horizon gap is large on the walkers (+655 to +1,727), where the cloned policy’s own value from a visited state is a poor guide to what 20-step plans achieve, so the walker rows are Data support. For each arm setting, every step of every labelled segment is a point in the scorer’s standardized input (observation and action). A visited state counts as off the data when its distance to the nearest labelled point exceeds the 95th percentile of the labelled points’ own nearest-neighbour distances (to a point of a different segment). In observations alone, hard search (, five recorded episodes) is off the data at 53% of steps on the drawer with 500 labels, 60% with human labels and 51% on plate-slide, and uniform averaging at none; on window-open the shares are 75% and 60%. With actions included the shares are 42%, 47%, 27% and 32% for hard search, and 0–2% without search. The window-open evaluation positions lie inside the labelled data in the observation space, and search still helps there, so leaving the data is not by itself what makes a scorer exploitable. In hand position alone, hard search ends on average 6.6 cm from the nearest labelled hand position on the drawer with synthetic labels (0.3 cm without search) but within 0.8 cm on plate-slide.
Held-out accuracy without shared trajectories. The scorers hold out a random 10% of pairs, so a held-out pair can share trajectories with training pairs. Splitting the labelled episodes 70/30 instead, and holding out the pairs whose two segments both come from held-out episodes, gives the same picture (five seeds pooled; a random split with the same number of training pairs in parentheses): drawer 500 labels 180 of 213 correct, 0.85 (0.85); 2,000 labels 0.83 (0.83); human labels 114 of 118, 0.97 (0.99); plate-slide 0.86 (0.81); window-open 0.84 (0.89); hopper 0.70 (0.74); cheetah 0.72 (0.73).
Calibrating from few pairs
Table 6 repeats the calibration of Section E with 200 random draws of exactly pairs (pairs tied in true return are redrawn) for and 200, and three labellers: the true order, and Bradley–Terry teachers with set so that they agree with the true order on 85% or 70% of proposal pairs, a stand-in for a person comparing two short plans from the same state. A cautious version replaces by its one-sided Wilson lower bound at nominal 95%.
| all | below | worst | worst | ||||||||||
| Cheetah | 0.08 | 0.07 | 54% | −0.31, 0.43 | −0.09, 0.28 | −0.03, 0.22 | 0.06 | 0.03 | −0.06 | 0.00 | −0.01 | 0.06 / 0.03 | 0.06 / 0.03 |
| Hopper | 0.73 | 0.69 | 26% | 0.43, 0.88 | 0.54, 0.81 | 0.60, 0.78 | 0.53 | 0.49 | −0.02 | 0.40 | −0.01 | 0.64 / 0.46 | 0.43 / 0.37 |
| Drawer 200 | 0.01 | 0.02 | 46% | −0.43, 0.43 | −0.19, 0.25 | −0.13, 0.16 | 0.05 | 0.00 | −0.21 | 0.00 | −0.03 | 0.00 / 0.00 | 0.00 / 0.00 |
| Drawer 300 | −0.01 | −0.02 | 45% | −0.54, 0.31 | −0.22, 0.16 | −0.16, 0.11 | 0.07 | 0.01 | −0.11 | 0.00 | −0.01 | 0.00 / 0.01 | 0.03 / 0.01 |
| Drawer 500 | 0.08 | 0.07 | 47% | −0.31, 0.43 | −0.09, 0.28 | −0.06, 0.19 | 0.12 | 0.04 | −0.22 | 0.01 | −0.04 | 0.06 / 0.03 | 0.06 / 0.03 |
| Drawer 1k | 0.07 | 0.07 | 48% | −0.31, 0.43 | −0.13, 0.25 | −0.08, 0.20 | 0.11 | 0.05 | −0.15 | 0.01 | −0.02 | 0.09 / 0.05 | 0.09 / 0.05 |
| Drawer 2k | 0.19 | 0.20 | 46% | −0.19, 0.54 | 0.03, 0.43 | 0.06, 0.32 | 0.23 | 0.14 | −0.15 | 0.05 | −0.05 | 0.22 / 0.14 | 0.13 / 0.11 |
| Drawer human | −0.18 | −0.21 | 53% | −0.64, 0.19 | −0.40, −0.03 | −0.34, −0.09 | 0.05 | 0.00 | −0.02 | 0.00 | 0.00 | −0.13 / 0.00 | −0.06 / −0.01 |
| Drawer, half on planner | 0.16 | 0.18 | 53% | −0.31, 0.54 | 0.00, 0.37 | 0.02, 0.31 | 0.26 | 0.12 | −0.17 | 0.03 | −0.04 | 0.13 / 0.11 | 0.13 / 0.10 |
| Plate 500 | 0.44 | 0.45 | 19% | 0.06, 0.73 | 0.25, 0.61 | 0.32, 0.58 | 0.42 | 0.35 | −0.03 | 0.23 | −0.01 | 0.43 / 0.34 | 0.28 / 0.27 |
| Plate 2k | 0.39 | 0.40 | 21% | 0.05, 0.73 | 0.22, 0.56 | 0.28, 0.54 | 0.39 | 0.30 | 0.00 | 0.19 | 0.00 | 0.37 / 0.29 | 0.28 / 0.24 |
| Window 500 | 0.44 | 0.47 | 47% | 0.06, 0.81 | 0.28, 0.61 | 0.38, 0.59 | 0.39 | 0.33 | −0.03 | 0.22 | −0.01 | 0.43 / 0.31 | 0.28 / 0.24 |
| Window 2k | 0.24 | 0.28 | 59% | −0.19, 0.64 | 0.06, 0.45 | 0.14, 0.38 | 0.22 | 0.12 | −0.86 | 0.05 | −0.40 | 0.19 / 0.10 | 0.09 / 0.07 |
Table 6. Calibrating the error spread from labelled pairs of the planner’s own proposals (one-step probe, 200 random draws of the pairs). : mean per-state correlation of the five-scorer mean with the true plan return; all: the Gaussian estimate from every pair at every state; below: share of states whose own correlation is under it. Range: 10th and 90th percentile of over draws. Gains (mean over states) and worst-state (mean over draws) of the calibrated rule with 100 pairs, in units of : with each state’s true (worst state 0.00 in every row), with the point estimate, with the one-sided 95% lower bound (LCB), and with labels from a noisy teacher that agrees with the true order on 85% or 70% of proposal pairs (point estimate).
Sampling noise. The 10–90% range of spans 0.45–0.85 with 25 pairs, 0.27–0.44 with 100 and 0.18–0.29 with 200. With 100 pairs the point estimate loses at most 0.22 standard deviations at the worst state on average over draws in twelve of the thirteen settings, but up to 0.47 in the unluckiest tenth of draws (drawer, 200 labels). Lower bound. The lower bound cuts the worst-state loss to at most 0.05 (0.40 on window-open with 2,000 labels) and the unluckiest-tenth loss to 0.25, at a cost in gain: it keeps 77% of the true- gain on hopper, 48–57% on plate-slide and on window-open with 500 labels, and 12–24% in the other settings where that gain exceeds 0.2. Noisy labels. A noisy teacher shrinks toward zero (hopper 0.71, 0.64 and 0.43 for the true order, 85% and 70%; plate-slide 0.45, 0.43 and 0.28), so where the scorer is informative the rule searches more gently: with 100 pairs the worst-state loss stays at most 0.19 (85%) and 0.18 (70%) in the same twelve settings, and the share of the gain kept falls from 47–93% to 42–87% and 31–71%. Gaussian assumption. Over all pairs, is within 0.05 of the mean Pearson correlation in every setting, and above it by up to 0.04 on plate-slide and window-open, where it is mildly optimistic. The larger approximation is pooling: one serves every state, and 19–59% of states have their own correlation below it. Those states cause the losses, and window-open with 2,000 labels has the most of them (59%).
Closed loop. Table 7 runs the rule with the pooled estimate, its lower bound and a noisy teacher, and with five independent sets of exactly 100 pairs per setting (500 distinct pairs, none tied in true return), each calibrating its own controller on the same ten starts, drawn either from the probe on the evaluation’s start states or from a second probe on start states 10–19 that the evaluation never uses. The noisy teacher changes little (within 20 points of the true-order version). Across independent sets the arm results barely move (evaluation starts: drawer 824–852, plate-slide 999–1,013, window-open 308–320; held-out starts: 818–852, 806–1,013, 309–318; every run successful); the walkers move more (cheetah 264–392 and 264–445, hopper 517–575 and 523–550), since a small change in near 1 changes how hard they search. On plate-slide one held-out set estimates , against 0.40–0.51 for the others, so it searches harder and scores 806, still succeeding in every run. With the lower bound every set keeps the arm at or near uniform averaging’s level (evaluation starts: drawer 852 with either label set, plate-slide 1,036–1,041, window-open 303–310; held-out starts: drawer 835–852, plate-slide 998–1,041, window-open 304–308; every run successful), declines to search on cheetah (264 with every set), and keeps most of hopper’s gain (500–560 and 472–551).
What the estimate must get right is stated by the following consequence of Theorem 1 and Theorem 2.
Corollary 8 (Calibrated rule). (i) At a state whose proposals have , the rule with , , does not lose () whenever , and it does not search when .
(ii) Let a labeller rank pairs by , with in the scorer’s units, , and and uncorrelated with . If is bivariate Gaussian under , the population concordance of the labeller with on two independent proposals satisfies , and exactly when , about for a small labelling error; with , whenever .
Proof. (i) By the scale argument , so the rule’s weights maximize and by Theorem 1; if then , and Theorem 2(i) gives . (ii) For a bivariate Gaussian, two independent draws have concordance exactly [18]; and and . □
In the population, then, a labeller whose mistakes are independent of the scorer’s only lowers the correlation that estimates, as the noisy teachers do; one whose mistakes the scorer has learned, because it was trained on that labeller’s labels, can raise it above the true correlation, and then the rule searches where it should not. A finite, pooled estimate can also exceed a given state’s correlation through sampling noise (the ranges above) and pooling over states. The one-sided Wilson bound at nominal 95% makes at most the pooled correlation with probability close to 0.95 under the Gaussian model (its coverage is approximate: 94.2% for 100 independent pairs at 90% agreement), so (i) then covers every state whose own correlation is at least the pooled one. This is a statement about the pooled correlation, not a simultaneous one over states: the states it leaves uncovered are the 19–59% noted above, so the rule is an empirical safeguard, not a guarantee.
Ties. The true-order and teacher labels above never tie. People often cannot tell two plans apart, and is then measured on the decided pairs only. If a person calls a tie when the two plans’ perceived returns differ by less than a threshold, as in threshold models of paired comparison [32], the decided pairs are the clearer ones and agree with the scorer more often than random pairs do, so overstates . In the Gaussian model of Theorem 8(ii), let the labeller’s difference (standardized) and the scorer’s difference have correlation , and let a share of pairs be tied, those with . Given , the scorer agrees in sign with probability , so the decided pairs’ concordance is
which is increasing in , equals at , and grows with when . We estimate by solving at the observed share of ties, and its lower bound by applying the same map to the Wilson bound on . On the probe’s proposals, a true-order labeller that ties the closest 70% of pairs gives an uncorrected of 0.97 on hopper, 0.77 on plate-slide and 0.73 on window-open, against 0.69, 0.45 and 0.47 over all pairs; the corrected estimates are 0.70, 0.45 and 0.42, and within 0.03 on cheetah and the drawer with 500 labels. With human labels, where the correlation is negative (−0.21 over all pairs), both estimates keep the sign but shrink it (−0.08 and −0.04), so the rule still does not search.
Two people’s comparisons. One author compared 100 random pairs of the drawer proposals blind: the 20 planned steps of two proposals from the same state, as 2-s clips in random order and sides (9 minutes in all). They could not tell 68 apart and agreed with the true return on 16 of the other 32, which is chance. Where one plan opened the drawer more than 1 mm further over the clip (14 decided pairs) they chose it 13 times; the true 20-step return chose it 6 times, and mostly favours the plan that ends with the hand nearer the handle (68% of 31 such pairs, against 42% for the person). The scorers with 500 and 2,000 synthetic labels agree with these choices at chance (, lower bound −0.42), so the rule does not search with them and keeps 852 (10 of 10). The scorer trained on the same person’s clip labels agrees with them (, lower bound 0.23) although it disagrees with the true return on the same pairs (−0.29). Calibrated from these choices the rule searches with it () and scores 574 (5 of 10 opened), against 852 when calibrated by true return; with the lower bound () it scores 839 (10 of 10). Two thirds of the pairs were ties, so the decided pairs are the clearer ones; corrected for ties, (lower bound 0.12) and the rule searches gently (), scoring 822 (10 of 10), still below uniform averaging: the correction removes the overconfidence the ties add, not the shared bias. The other author compared the same 100 pairs (38 minutes): they could not tell 71 apart and agreed with the true return on 12 of the other 29, and where both people decided (12 pairs) they agreed 9 times. Their choices do not back the scorer trained on the first author’s clip labels ( = 0.16, lower bound −0.30, so = 0.99): the rule barely searches and scores 845 (10 of 10). Pooled over both people, = 0.42 (lower bound 0.11) gives 812 and 846 (10 of 10 each). On the scorers with 500 and 2,000 synthetic labels the second author’s choices give = 0.16 and 0.37, and the rule scores 831 and 893 (10 of 10 each). Table 8 collects the three kinds of calibration for this scorer, with and without the tie correction. They are not interchangeable: the oracle sees that the scorer ranks the proposals worse than chance and declines to search; the labeller who trained the scorer shares its bias and endorses it; the other author does not. A rule calibrated from a person’s preferences checks the scorer against that person, so it cannot catch a scorer that learned a bias it shares with the evaluator (Theorem 8(ii)); a second person, the tie correction or the lower bound limits the damage. With two labellers on one task, this is a case study, not an estimate of how often it happens.
Null and limit cases
Button-press. Same protocol as the drawer, 5 runs per setting: not exploited at any budget (standard MPPI 151–157 at and 154–171 at with 200, 500 or 2,000 labels; nearly every run presses the button). Here the perfect scorer does worse (126–140), because over a 20-step horizon Meta-World’s dense reward pays for hovering near the button rather than pressing it, while the preference-trained scorers learned progress over 50-step clips. The label budget at which exploitation starts is task-specific.
Adroit door and hammer. With crowd labels that compare only successful human demonstrations ([46] (Rajasewaran et al. 2018)), the scorer never opens the door (0 of 12 runs at any temperature) and drives the nail in 2 of 60 runs, while the perfect scorer opens the door in 6 of 6 runs at and drives the nail in 4 of 5 at . With no failures in the labelled data the scorer has no signal about failure, and no weighting rule can supply one.
Best-of- details
At each step the cloned policy’s action plus noise gives candidates, and an MLP verifier picks one. It is trained with the Bradley–Terry loss on pairs of perturbed actions, where a synthetic labeller prefers the action closer to Meta-World’s scripted expert, with noise set for 85% agreement. The verifier sees the state, not images, and its labels are synthetic. The perfect verifier is the negative distance to the expert action. Single verifiers pool scorer seeds 0–4 with episode seeds 0–9 (50 runs per ); the perfect verifier and one randomly perturbed action, with no verifier, use the same ten starts with five noise draws each. In Figure 3c, is one noisy action and the clean cloned action, executed without noise, is the reference line. A cautious five-verifier ensemble does not help on plate-slide (0 of 10 at ). The verifiers’ held-out pair accuracies are 0.62–0.80 at 500 pairs and 0.69–0.81 at 2,000, again a poor guide to how they behave under selection. Best-of- itself an exponential tilt of , the action noise. For continuous scores the population best-of- choice has density , with the distribution function of under : this is tilted by at , so Theorem 5(i) applies with in place of , and true return rises with exactly while . Its KL to is ([36, 2]), which puts to 1,024 at to 2.4, the range of Figure 3a. This is the population value; the selected index on a finite set of candidates has KL to a uniform pick.
Results. Every point of Figure 3c is 50 runs on the same ten starts (10 for the ensemble). One noisy action succeeds in 44 of 50 runs on both the drawer and plate-slide. With 500 pairs, is level with it on the drawer within noise (728, though 34 of 50 successes) and below it on plate-slide (723, 26 of 50); at selection falls to 388 (14 of 50) and 143 (0 of 50). With 2{,}000 pairs it scores 817–895 on the drawer and 977–1{,}061 on plate-slide.
References
- [1]Aouad, A., El Gadrari, A., and Farias, V. F. Revisiting scaling laws for reward optimization. arXiv preprint arXiv:2609.38526, 2026.arxiv.org/abs/2609.38526
- [2]Beirami, A., Agarwal, A., Berant, J., D’Amour, A., Eisenstein, J., Nagpal, C., and Suresh, A. T. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879, 2024.arxiv.org/abs/2401.01879
- [3]Bhamidipaty, L. M., Whittamer, E. S., Abel, D., Kochenderfer, M. J., and Ramamoorthy, S. Imperfect world models are exploitable. arXiv preprint arXiv:2605.15960, 2026.arxiv.org/abs/2605.15960
- [4]Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
- [5]Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1706.03741
- [6]Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31, 2018.
- [7]Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.02743
- [8]Eisenstein, J., Nagpal, C., Agarwal, A., Beirami, A., D’Amour, A., Dvijotham, K., Fisch, A., Heller, K., Pfohl, S., Ramachandran, D., Shaw, P., and Berant, J. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. In Conference on Language Modeling, 2024.
- [9]El Hasnaouy, H., Krupa, P., Zanon, M., and Bemporad, A. Active learning MPC objective functions from preferences. arXiv preprint arXiv:2605.16071, 2026.arxiv.org/abs/2605.16071
- [11]Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866, 2023.arxiv.org/abs/2210.10760
- [12]Hansen, N., Su, H., and Wang, X. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, 2024.
- [13]Hejna, J. and Sadigh, D. Few-shot preference learning for human-in-the-loop RL. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, 2023.
- [14]Huang, A., Block, A., Liu, Q., Jiang, N., Krishnamurthy, A., and Foster, D. J. Is best-of-N the best of them? Coverage, scaling, and optimality in inference-time alignment. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 2025a.
- [15]Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the mythos of KL-regularization: Direct alignment without overoptimization via chi-squared preference optimization. In International Conference on Learning Representations, 2025b.
- [16]Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. MOReL: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2005.05951
- [17]Krupa, P., El Hasnaouy, H., Zanon, M., and Bemporad, A. Learning the MPC objective function from human preferences. In IEEE Conference on Decision and Control, 2025.arxiv.org/abs/2511.22502
- [18]Kruskal, W. H. Ordinal measures of association. Journal of the American Statistical Association, 53(284):814–861, 1958. doi: 10.1080/01621459.1958.10501481.DOI
- [19]Kwa, T., Thomas, D., and Garriga-Alonso, A. Catastrophic Goodhart: Regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification. In Advances in Neural Information Processing Systems, volume 37, 2024.
- [20]Kwok, J., Agia, C., Sinha, R., Foutter, M., Li, S., Stoica, I., Mirhoseini, A., and Pavone, M. RoboMonkey: Scaling test-time sampling and verification for vision-language-action models. arXiv preprint arXiv:2506.17811, 2025.arxiv.org/abs/2506.17811
- [22]Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1612.01474
- [23]Lee, K., Smith, L., and Abbeel, P. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, 2021.arxiv.org/abs/2106.05091
- [24]Lin, H., Wang, P., Schneider, J., and Shi, G. TD-M(PC)²: Improving temporal difference MPC through policy constraint. arXiv preprint arXiv:2502.03550, 2025.arxiv.org/abs/2502.03550
- [25]Luo, J., Xu, X., Wang, S., Zhou, Y., and Zhang, Z. CritiHack: Evaluating visual rewards under robot policy optimization. arXiv preprint arXiv:2610.02527, 2026.arxiv.org/abs/2610.02527
- [26]Manheim, D. and Garrabrant, S. Categorizing variants of Goodhart’s law. arXiv preprint arXiv:1803.04585, 2018.arxiv.org/abs/1803.04585
- [27]Moskovitz, T., Singh, A. K., Strouse, D., Sandholm, T., Salakhutdinov, R., Dragan, A. D., and MacAleer, S. Confronting reward model overoptimization with constrained RLHF. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.04373
- [28]Nabail, M., Cheng, L., Wang, J., and Rhinehart, N. UBP2: Uncertainty-balanced preference planning for efficient preference-based reinforcement learning. arXiv preprint arXiv:2606.19328, 2026.arxiv.org/abs/2606.19328
- [29]Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2201.03544
- [31]Ramé, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. WARM: On the benefits of weight averaged reward models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 2024.arxiv.org/abs/2401.12187
- [32]Rao, P. V. and Kupper, L. L. Ties in paired-comparison experiments: A generalization of the Bradley–Terry model. Journal of the American Statistical Association, 62(317): 194–204, 1967. doi: 10.1080/01621459.1967.10482901.DOI
- [33]Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.DOI
- [34]Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems, volume 35, 2022.arxiv.org/abs/2209.13085
- [35]Smith, J. E. and Winkler, R. L. The optimizer’s curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311–322, 2006.DOI
- [36]Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2009.01325
- [37]Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A., and Brown, D. S. Causal confusion and reward misidentification in preference-based reward learning. In International Conference on Learning Representations, 2023.arxiv.org/abs/2204.06601
- [38]Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.DOI
- [39]Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2407.17032
- [40]Wang, Z., So, O., Gibson, J., Vlahov, B., Gandhi, M., Liu, G.-H., and Theodorou, E. A. Variational inference MPC using Tsallis divergence. In Robotics: Science and Systems, 2021.arxiv.org/abs/2104.00241
- [41]Williams, G., Aldrich, A., and Theodorou, E. A. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344–357, 2017a.DOI
- [42]Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A. Information theoretic MPC for model-based reinforcement learning. In IEEE International Conference on Robotics and Automation, 2017b.DOI
- [43]Yoon, H.-J. and Kim, H. Residual-conservative model predictive path integral control. arXiv preprint arXiv:2607.06950, 2026.arxiv.org/abs/2607.06950
- [46]Yuan, Y., Hao, J., Ma, Y., Dong, Z., Liang, H., Liu, J., Feng, Z., Zhao, K., and Zheng, Y. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In International Conference on Learning Representations, 2024.arxiv.org/abs/2402.02423
- [47]Zhai, Y., Lei, Y., Zhang, H., Yu, Y., Xu, K., Feng, D., Ding, B., and Wang, H. Uncertainty-penalized reinforcement learning from human feedback with diversified reward LoRA ensembles. Information Processing & Management, 63(3):104548, 2026. doi: 10.1016/j.ipm.2025.104548.DOI
- [48]Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or K-wise comparisons. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 43037–43067, 2023.arxiv.org/abs/2301.11270


