When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity

to 47% but yields at most a measured 1.15$\times$ end-to-end speedup on the deployed path (1.25$\times$ on a decode-excluded path), short of a 1.5$\times$ target, and our adaptive-allocation experiment is inconclusive under a power gate. We further identify and fix a silent rendering corruption whose rate rises sharply along compute-heavy planning paths while leaving the official direct-policy result unaffected, and release a detector, fix, and scripts. These findings show why test-time scaling should be audited as a chain of premises (decision signal, realized efficiency, evaluation integrity) rather than by success rate alone. Conclusions are scoped to the tested configuration.

Introduction

World-action models (WAMs) such as Cosmos Policy [14] adapt large pretrained video models into robot policies that jointly generate actions, future states, and values within a shared latent diffusion process. A central promise of this design is test-time scaling: by generating candidate actions, predicting their future consequences, and evaluating them with a value function, WAMs can allocate additional inference-time computation to identify promising actions. Related peer-reviewed work has explored complementary aspects of this problem, including model-based planning with action-conditioned video world models [27] and efficient diffusion-based robot policies through one-step distillation [22], training-free adaptive caching [11], and rollout-adaptive pruning and activation reuse [12].

Yet whether extra inference-time computation improves control remains insufficiently examined. More computation helps only if it exposes decision-relevant candidate differences, produces practical end-to-end effects, and is evaluated reliably; success rates alone cannot distinguish a weak selection signal, unrealized compute savings, and simulator artifacts [17]. We therefore audit test-time scaling as a chain of premises rather than a single performance number: is useful signal present, do nominal savings translate into end-to-end gains, and does the evaluation path remain reliable under added computation?

We instantiate this audit on Cosmos Policy and RoboCasa [20]. Rather than proposing a planner, we combine paired candidate comparisons and a deployment-conditioned hindsight oracle, pre-registered decision criteria and a power gate, complete-query latency measurements, and a framebuffer-sentinel integrity check.

The audit yields three findings.

(1) Candidate selection is limited by the available signal, not just by the selector. On render-verified clean data, the deployable value selector performs 1.66 pp worse than random, and even a hindsight oracle (an upper bound) gains only 1.40 pp with an interval including zero, because in 82% of query states all candidates share an outcome. This does not show selection is generally ineffective; it shows selector quality must be read alongside the separability of the candidates it receives.

(2) Nominal compute reduction is not a reliable proxy for practical acceleration. Directly timing the complete query, token pruning yields at most a measured 1.15$\times$ end-to-end speedup on the deployed path (1.25$\times$ with future-image decode excluded), below the pre-specified 1.5$\times$ target, because the gain is bounded by the irreducible autoencoder encode and decode. A separate joint-allocation diagnostic falls below the measurement’s noise resolution and did not clear its power gate, so it is inconclusive rather than evidence of no effect.

(3) Evaluation integrity can depend on the inference path being evaluated. We identify and fix a silent rendering corruption in the RoboCasa/robosuite stack whose frequency is small under the official direct-policy protocol but rises sharply in the compute-intensive planning path. It is more than a generic simulator bug: changing inference-time computation changes exposure to a measurement failure, so the compute-intensive paths that realize test-time scaling are the most exposed.

Together, these findings make the contribution a testable audit methodology and a configuration-specific case study, not a claim that test-time scaling fails for all WAMs.

Related Work

World models for control. World models that predict action-conditioned futures have advanced rapidly, spanning diffusion-based video prediction [1, 4, 7], 3D and persistent representations [19, 26], and manipulation-specific models [13]. The backbone we study, Cosmos Policy [14], follows the joint action-video line in which actions, futures, and values are produced within a single denoising process [7]. We do not propose a new world model; we audit whether the inference-time computation layered on top of such a model improves control.

Efficient world-model inference. Because prediction and planning in these models are costly, a growing body of work reduces their inference cost, through sparse imagination of rollouts [4], disentangled dynamics prediction [23], efficient on-policy rollout in diffusion world models [5], and warm-started prediction that reuses spatiotemporal consistency [15]. These methods report nominal compute reductions. We audit whether such reductions convert to end-to-end latency on a representative stack, and find that token-level savings are capped by an irreducible encoding fraction rather than realized as speedup, and that the joint allocation effect lies below the measurement noise floor. X-WAM [6] audit underscores why architectural avenues can be more fruitful than token pruning alone for practical speedups.

Test-time scaling and candidate selection. A parallel line scales inference by sampling and selecting candidate actions, through verifier-free scoring [9], relative action critics [16], and reward-ranked world-model rollouts [24]; analogous strategies in text and digital-agent settings use self-verification, process reward models, and execution-free critics [2, 8, 9, 21, 25]. These works propose selectors and report positive scaling. We instead audit whether the selection signal exists, and find a learned value selector worse than random on a strong base policy where candidate outcomes rarely differ.

Reliability of policy evaluation. Recent work asks whether simulator- and world-model-based evaluation is trustworthy, training learned evaluators to match real rollouts [18], testing robustness under real-to-sim translation [10], and reducing evaluation cost while preserving rankings [3]. These address the fidelity of a learned evaluator or a synthesized scene. Our reliability finding is at the level of the evaluation stack itself: a silent rendering corruption in the standard RoboCasa and robosuite pipeline whose rate rises with per-query inference compute, for which we release a detector and a fix.

A Diagnostic Methodology for Test-Time Compute

Because extra inference compute can change outputs without improving control, success-rate movements alone are not evidence of scaling gains. Our pre-registered, falsification-first audit tests for usable candidate signal and reliable evaluation before interpreting success (§ 6).

Verify the Premise Before Measuring the Gain

Before evaluating an inference-time mechanism, we test whether its required signal exists. For best-of-$N$ planning, candidate outcomes must differ in ways a selector can recover; we test this directly in § 4, rather than infer it from downstream success. Supplement Figure 5 summarizes the audit gates.

Causal Ablation and an Oracle Headroom Estimate

For each query state, we replay all $N$ candidates under a fixed continuation policy and let a hindsight oracle select a successful one. This estimates outcome-level selection headroom in the current candidate pool, because it uses unavailable outcome information, it is not deployable. We also probe input importance by leave-one-out patch masking, trusting the probe only if irrelevant background ranks below relevant end-effector and contact regions in a pre-registered sanity check.

Pre-Registration and Statistical Power Control

For the selection audit, we fix decision thresholds and the replicate-consistency gate before inspecting the corresponding data and report the gate’s verdict even when it fails. A power gate avoids launching experiments unlikely to yield a verdict: from a smoke sample, we estimate the paired-residual standard deviation $\hat{\sigma}_r$ and project the two-sided confidence-interval half-width at the planned sample size,

launching the full run only if $h(N_{\mathrm{full}}) < \Delta_{\mathrm{prereg}}$, the separation between the pre-registered outcomes; otherwise the run is not launched. Every reported comparison states its bootstrap resampling unit and interval, so that an effect whose significance depends on that choice is presented as such rather than as a settled result. Closed-loop success is measured by snapshot replay, in which we restore a state, execute a candidate action chunk, and continue under a fixed dense base policy, rather than inferred from a single-step action deviation.

Scope

Our empirical findings are limited to Cosmos Policy on RoboCasa. The audit components—premise checks, oracle headroom, a power gate, and render-integrity verification—can also inform audits of other diffusion control policies. We release a framebuffer-sentinel detector, a fix for the identified OpenGL-context bug, and reproduction scripts; the detector principle may transfer across backends, but this fix is stack-specific.

Auditing Best-of-$N$ Planning Selection

We apply the audit to best-of-$N$ planning, comparing the model’s value-based ordering with random selection and estimating outcome-level headroom using a hindsight oracle.

Setup

On render-verified clean data, we compare the value selector, random selection, and the deployment-conditioned hindsight oracle on paired sets of $N=8$ candidates with common continuation seeds. Differences from random are reported with state-within-task and episode-within-task bootstrap intervals.

Continuation protocol. In snapshot replay, each candidate’s first 16 steps of a generated 32-step chunk are executed, stopping on success; the episode then replans every 16 steps under a fixed base policy until success or the 500-step task limit. The base policy uses one sample, all tokens, 10 denoising steps, and no value head; all selectors share the same outcome matrix and continuation seeds ($R=2$). This controls the candidate’s single-switch effect but does not reproduce the full deployment distribution. The selection audit’s 10-step continuation is distinct from the 2-versus-5-step ablation factorial, whose dense reference is $D(5,F)$.

Weak Selection Signal and a Negative Value Selector

The value selector is 1.66 pp worse than random (95% CIs: state bootstrap $[-2.97,-0.31]$; episode bootstrap $[-2.49,-0.71]$), driven largely by TurnOn-Stove ($-3.87$ pp). Excluding that task, the state-weighted estimate remains negative ($-0.77$ pp; episode CI $[-1.27,-0.21]$; state CI $[-1.54,-0.03]$). Thus, the result is not uniform across tasks, but no task reverses into a measured value-selector benefit.

The hindsight oracle’s gain is $+1.40$ pp, with a state-bootstrap CI of $[+0.31,+2.60]$ pp and an episode-bootstrap CI of $[-0.25,+3.40]$ pp. Under the pre-registered rule, the missed replicate-consistency gate and episode-level interval make this weak, inconclusive evidence. With $R=2$, the reliability estimate is only about 0.33 (Supplement Section 10); we therefore do not infer a definitive ceiling from either the cross-fitted estimate or the in-sample $+7.1$ pp maximum. More structurally, outcomes vary across candidates and/or repeats in only 48/263 states (18%; Figure 3b). The within-state ICC is 0.20, which indicates limited repeat agreement under this protocol; the decomposition below distinguishes stable separation from variation not stable across repeats.

How much of this separability is real? Outcome uniformity is not explained simply by near-identical actions: states with no outcome variation have no smaller action-space dispersion (Supplement Section 13), though geometric diversity need not be task-relevant. Of the 48 varying states, 20 show stable separation (one candidate succeeds in both repeats and another fails in both); the other 28 do not meet this criterion. A label-shuffle null expects about 14.8 stable-separable states, so the observed 20 are not 20 confirmed opportunities. The descriptive oracle contribution is +1.85 pp for the first group and −0.44 pp for the second; the pooled oracle gain under the null is +0.01 pp. With only two repeats, reproducible selection headroom remains uncertain.

Interpretation

Under this configuration, the negative value-selector result is clear, whereas reproducible oracle headroom remains uncertain. The oracle measures outcome-level headroom only for the observed candidates and specified single-switch continuation, not general decision relevance. We therefore make no claim that planning is universally ineffective; alternative candidate-generation and continuation policies remain untested.

A Feasibility Audit of Adaptive Compute Allocation

We audit whether reducing denoising steps or processing fewer tokens can deliver practical efficiency on this stack. Because the preregistered criteria are end-to-end latency and a resolvable joint interaction effect, and the power gate was not cleared, this is a feasibility diagnostic rather than a confirmatory test of joint allocation.

Nominal FLOPs Savings Do Not Convert to Wall-Clock Savings

Dropping all future frames plus the value frame removes 47.9% of per-forward FLOPs, but direct complete-query timing shows only 1.152× speedup on the deployed eager path and 1.253× when future-image decode is excluded; neither reaches the preregistered 1.5× target Table 1. This fixed whole-frame mask adds no per-token scoring. The five DiT forwards account for 530.7 ms of the 1241.8 ms dense query, while VAE encode (238.0 ms) and future-image decode (434.4 ms) are not reduced by token pruning. This is a latency-only result (dropped frames receive zero output); full settings and per-level latency percentiles are in Supplement Section 9 and Table 3.

Pruning level	Tokens	TFLOP	Nominal saving	of which SDPA (TFLOP)
Dense	2156	8.535	—	1.319
Drop 1 future image	1960	7.682	10.0%	1.111
Drop 2 future images	1764	6.846	19.8%	0.921
All future images	1568	6.028	29.4%	0.748
All future frames	1372	5.228	38.7%	0.593
All future frames + value	1176	4.446	47.9%	0.455
Pruning level	Deployed path	Decode-excluded	≥ 1.5×?
Dense	1254.9 / —	812.7 / —	—
Drop 1 future image	1232.1 / 1.019	791.2 / 1.027	no
Drop 2 future images	1171.0 / 1.072	728.9 / 1.115	no
All future images	1140.5 / 1.100	699.8 / 1.161	no
All future frames	1096.5 / 1.144	654.0 / 1.243	no
All future frames + value	1088.9 / 1.152	648.6 / 1.253	no
Component (per query)	ms
VAE encode (1×)	238.0
DiT forward (5×, summed)	530.7
Future-image VAE decode	434.4
Full query	1241.8

Pruning level	Tokens	Deployed path: Median (ms)	Deployed path: p10–p90 (ms)	Deployed path: ×	Decode-excluded path: Median (ms)	Decode-excluded path: p10–p90 (ms)	Decode-excluded path: ×
Dense	2156	1254.9	1248.9–1257.1	1.000	812.7	808.6–814.5	1.000
Drop 1 future image	1960	1232.1	1226.7–1233.8	1.019	791.2	786.7–792.5	1.027
Drop 2 future images	1764	1171.0	1168.7–1172.6	1.072	728.9	727.0–730.6	1.115
All future images	1568	1140.5	1136.6–1142.8	1.100	699.8	697.0–703.2	1.161
All future frames	1372	1096.5	1093.9–1099.3	1.144	654.0	652.0–656.5	1.243
All future frames + value	1176	1088.9	1086.4–1092.2	1.152	648.6	647.0–651.9	1.253

The Preliminary Interaction Effect Is Below the Noise Resolution

We then ask whether a joint-allocation benefit could be resolved on this stack even if present. Using the pre-registered factorial diagnostic of § 3, crossing two denoising schedules with full, key-token, and uniform-token ($U$) selection, the joint-allocation effect is the uniform-token interaction

under exact additivity, where $D(k,s)$ is the paired closed-loop success at denoising schedule $k$ under selection $s$, relative to the baseline $D(5,F)=0$. Because seed noise largely cancels in this difference-in-differences structure, with a measured effective correlation of $\rho=0.92$, the residual noise is small but structured: per-state noise magnitude has mean 0.0066 and 90th percentile 0.0173. The systematic component, estimated after noise-bias removal from a 16-state smoke sample, is about 0.0045.

The decisive comparison is independent of any chosen threshold: the preliminary systematic component (0.0045) is smaller than the mean noise floor (0.0066) and comparable to the temporal main effect itself (0.0053) in the same states. A joint effect that is smaller than the per-state noise it must be read against cannot be resolved from this measurement, whatever decision threshold is adopted. Figure 4 places these quantities on the same absolute scale. For completeness, under the currently locked threshold $\theta_B$ (Supplement Eq. 3) the minimum detectable interaction is 0.0265 to 0.0583, that is 5 to 11 times the temporal main effect; we note that this range depends on the threshold calibration still pending sign-off, and it will be recomputed for the camera-ready version, but the threshold-free comparison above does not depend on it.

This is inconclusive, not a null result: the full factorial was not launched because the power gate failed. The estimated sample requirement—hundreds to roughly two thousand states, is an extrapolation from the current detection limit (Supplement Section 10), not a verified power result. Whether joint allocation helps on this stack remains open.

Implications

Efficiency claims should pair FLOPs with complete-query latency, and allocation claims should be judged against a preregistered detection limit. Here, token-pruning savings miss the latency target and the smoke-scale joint effect is unresolved; this does not establish that adaptive allocation is ineffective in general.

Evaluation Reliability: A Silent Rendering Corruption

We identify a third threat to test-time-scaling evaluation: silent, compute-dependent rendering corruption in the RoboCasa/robosuite stack, to which compute-intensive inference paths are more exposed.

Mechanism

The corruption originates in OpenGL context management during environment resets. On a hard reset the stack constructs a new simulation instance while an orphaned instance remains eligible for garbage collection; when the orphan is later collected, its GL context is freed while a different context is active, silently invalidating the framebuffer of the live environment. Subsequent renders return corrupted observations, frozen or noise-like frames, with no error raised. Figure 1 contrasts a corrupted frame with the correct render of the same state. We confirm the mechanism with a framebuffer sentinel that pre-fills each render target with a marker and flags targets left substantially untouched after rendering.¹

Effect on the Official Direct-Policy Benchmark

We first ask whether this corruption materially affects the officially reported RoboCasa numbers. We run the unmodified official evaluation file in two arms, bug-present and render-fixed, paired episode by episode on the four tasks where our baseline run exhibited corruption, over 50 consecutive episodes each (200 paired episodes), with the framebuffer sentinel active throughout.

As Table 2 shows, under the direct-policy protocol the measured corruption rate is 8/1200 (0.67%) per episode in the bug arm, and zero in the fixed arm), and the paired effect of fixing it is +3 successes over 200 episodes. All flips move corrupted episodes from failure to success, so the corruption understates success. Extrapolated to the full 24-task, 50-episode protocol, the effect is about +0.25 pp and never exceeds an upper bound of +0.6 pp. We conclude that the officially reported direct-policy figure of 67.1% is not materially affected: the corruption is real and present in the official stack, but its direct-policy impact is small and in the direction of underreporting success.

Task	Succ (bug→fix)	Corrupt	Flips
PnPCounterToMicrowave	16 → 18	3	+2
PnPMicrowaveToCounter	18 → 18	3	0
PnPStoveToCounter	32 → 33	1	+1
TurnSinkSpout	36 → 36	1	0
Total	102 → 105	8/200	+3

Corruption Scales with Per-Inference Compute

The direct-policy result does not bound the risk for test-time scaling, because corruption frequency depends on how much computation each query allocates. We observe a sharp, monotone dependence:

• Direct policy (one forward pass per query): 0.67% of episodes corrupted, paired-verified in Table 2.

• Direct-mode best-of-$N$ ($N=8$ candidates, simple forward passes): 0/40 episodes corrupted, with identical success (29/40) in both arms. Additional sampling alone does not raise the rate.

• Planning mode (separate future-state and value calls per query, substantially more allocation per query): up to 65% of episodes corrupted.

The 65% figure is observed under a compute-heavy planning configuration and should be read as an upper-bound illustration of the trend, not as the corruption rate of any standard benchmark. RoboCasa provides no native planning-mode evaluation, and this configuration is nonstandard; the official repository exposes best-of-$N$ planning only for a real-robot platform that does not pass through robosuite, so the planning-path corruption touches no number the official benchmark reports. On the LIBERO stack, which on a hard reset explicitly destroys the prior simulation before constructing a new one and so leaves no orphan, the trigger condition is not met; we infer from the source that the reported LIBERO figure is unaffected, though we did not run that environment.

Implications

The official direct-policy result is largely unaffected, but compute-heavy planning is more exposed. Studies using this stack should verify render integrity before attributing gains to inference-time methods.

Discussion

Our three audits are independent, but they point in the same direction: on the tested Cosmos Policy configuration, the computation added at inference time to realize test-time scaling does not buy what it is assumed to buy, and part of the apparent benefit reported in this setting can arise from evaluation artifacts rather than from the method.

What the findings do and do not say. None of the three results claims that planning, adaptive allocation, or test-time scaling is useless in general; each is stated at the scope of the tested configuration. The planning result is that the deployable selector is worse than random and that selection headroom is small, not that no selector can ever help. The allocation result is that nominal FLOPs savings do not convert to end-to-end speedup here, together with a detectability statement from a smoke-scale sample whose preregistered power gate was not cleared, not a confirmed null. The reliability result is that a real, compute-dependent corruption exists in the official stack while leaving the reported direct-policy number essentially intact.

A methodology, not only a set of negative results. The value of a negative audit depends on whether its measurements can be trusted, which is why the diagnostic methodology is our primary contribution. Premise-first verification keeps us from building on a signal that does not exist; the oracle bounds the achievable gain before any selector is tuned; the power gate stops an experiment that cannot yield a verdict before it consumes compute; and render-integrity checking protects every closed-loop number. These components are model-agnostic and transfer to any audit of test-time compute in a diffusion-based control policy, even though the specific render fix is tied to the stack we study. We release the detector, fix, scripts, and preregistered protocols so the audits can be repeated. Our strongest limitation is scope: whether these findings generalize to other models, simulators, or real hardware is an open question the methodology is designed to answer rather than presuppose, and we see the audits not as a verdict against inference-time computation but as checks that should precede any claim that test-time scaling improves control.

Conclusion

We presented an empirical audit of test-time scaling in a representative world-action model, Cosmos Policy on RoboCasa, built around a premise-first, pre-registered methodology with explicit statistical power control and render-integrity verification. The audit yields three independent reasons an apparent gain may not be real: best-of-$N$ candidate selection carries little usable signal and its deployable selector is worse than random; nominal compute savings from token-level allocation do not convert to practical speedup, and the joint allocation effect lies below the system’s noise resolution; and a silent, compute-dependent rendering corruption inflates the reliability risk of exactly the compute-intensive inference paths that test-time scaling relies on, while leaving the official direct-policy benchmark intact. We release the methodology and tools so these checks can be applied before performance gains are attributed to test-time scaling rather than to unverified assumptions or evaluation artifacts.

Timing Measurement Details

This supplement documents the measurement setup behind Table 1 and the single-forward mechanism that explains why per-forward savings do not carry the whole query. All figures are directly measured; none are composed from component times.

Environment. NVIDIA RTX A6000, PyTorch 2.7.0 with CUDA 12.8 and cuDNN 90701, model nvidia/Cosmos-Policy-RoboCasa-Predict2-2B with weights in bf16, latent shape [1, 16, 11, 28, 28] (11 latent frames $\times$ 196 tokens = 2156), batch 1, and 5 denoising steps per action query (confirmed as exactly 5 DiT forwards per call). The only tracked code modification is a logging patch to run_robocasa_eval.py (commit 18a2acc).

End-to-end protocol (Table 1b). For each pruning level and each of the two paths (deployed, and decode-excluded), we time the complete action-inference query, dense against pruned, under the deployed eager backend. We discard warmup and then record $N = 30$ timed queries per cell, reporting the median with the 10th and 90th percentiles; the directory out_e2e-prune/raw.jsonl holds all 420 timed calls. Paired per-state speedups agree with the reported median ratios to within 0.001. The decode-excluded dense baseline is directly measured at 812.7 ms (it is not the decode-subtracted estimate used before this run). The whole-frame prune is applied inside the real query path; because whole-frame pruning is a fixed structural mask, there is no per-token scoring step, and the frame slicing and scatter-back run inside the timed region, so the measured speedups carry full real overhead for this pruning type.

Why the whole query was measured eager. The complete query contains host-side control flow, host-device copies and array post-processing that cannot be CUDA Graph captured without restructuring the code, so no graph-timed stage is mixed into the end-to-end numbers. Eager is in any case the deployed evaluation backend.

Single-forward mechanism (not an end-to-end result). To locate the ceiling of the pruned stage, we separately time the DiT forward in isolation. Under CUDA Graph the single forward speeds up by at most 1.73× (dense 102.75 ms to all-future-frames-plus-value 59.42 ms), with output bit-identical to eager. This is a single-forward, CUDA-Graph measurement and is not directly comparable to the eager end-to-end speedups in Table 1b; we report it only to show that even a forward driven nearly to its floor cannot carry the query, because the five DiT forwards are under half of the end-to-end latency (Table 1c) while VAE encode and future-image decode are not reducible by token pruning.

FLOPs. Per-forward FLOPs in Table 1a are measured with FlopCounterMode (matmul + SDPA).

Reproduction. The summary is regenerated from the raw timings by E2E_MODE= summarize python3 probes/ARC_gate_m1/gm1_e2e_prune.py; out_e2e-prune/ also contains E2E_PRUNE_REPORT.md, summary.json and meta.json.

Statistical Definitions

Decision threshold. The pre-registered interaction threshold is

the larger of a seed-noise floor and a fraction of the temporal main effect, so that a declared interaction must exceed both the noise and a meaningful share of the main effect.

Oracle headroom reliability. With $R$ replicate rollouts per state, the reliability of the mean candidate estimate follows the Spearman–Brown form

With the measured intraclass correlation $\rho_{\mathrm{ICC}} = 0.20$ and $R = 2$, Rel = 0.33, so a single continuation ranks candidates with only about 20% reliability and the cross-fitted oracle estimate is itself noisy.

The Premise-First Audit Procedure

Figure 5 shows the audit as a single procedure: three gates applied in order before any test-time-scaling gain is credited. Each gate corresponds to one of the three ways an apparent gain can fail, and to one of the paper’s three audits.

The first gate verifies the premise that the added computation exposes a usable decision signal, using the causal ablation and oracle-ceiling construction of § 6; when no signal exists the result is reported as negative rather than pursued further, as in the planning audit of § 4. The second gate is statistical: from a smoke sample it projects the confidence-interval width at the full sample size and halts the run when the pre-registered outcome cannot be resolved, so compute is not spent on an experiment that cannot yield a verdict, and the outcome is reported as inconclusive rather than as evidence of absence, as in the allocation audit of § 5. The third gate measures closed-loop success under the framebuffer-sentinel render check and routes the verdict by pre-registered thresholds, the render-integrity guard whose necessity the reliability audit of § 6 establishes.

The value of the procedure is in its two “no” branches: they separate a genuine negative from an underpowered or invalid measurement, a distinction that success rate alone cannot make. The gates are model-agnostic and apply to any audit of test-time compute in a diffusion-based control policy; only the render fix itself is specific to the stack we study.

End-to-End Latency Under Token Pruning

Table 3 reports the full measurement behind Table 1(b): both query paths at all six pruning levels, with the 10th and 90th percentiles over the $N = 30$ timed queries per cell. Three observations follow, all from directly measured values.

First, the end-to-end speedup grows monotonically but sublinearly with the nominal FLOPs reduction. Cutting 47.9% of per-forward FLOPs (all future frames plus value) buys only 15.2% of wall-clock on the deployed path and 25.3% on the decode-excluded path; the remaining time is dominated by the autoencoder encode and future-image decode, which token pruning does not touch.

Second, the decode-excluded path scales better at every level, reaching 1.253× against 1.152× on the deployed path at the most aggressive prune. Removing the fixed future-image decode shrinks the denominator and enlarges the share of the query that pruning can act on, yet even this most-favorable configuration stops well short of the 1.5× target.

Third, the percentile spread is tight: the 10th and 90th percentiles lie within about 1% of the median at every cell, so the medians are stable and the sub-1.5× conclusion is not an artifact of run-to-run variance. No pruning level on either path reaches 1.5×.

Candidate Diversity and Outcome Separability

These diagnostics address whether the limited selection headroom in § 4 is an artifact of near-identical candidates or of replicate noise. They are descriptive and read only saved files (candidate actions, per-candidate closed outcomes with two repeats, and value scores); they do not run inference, and they do not claim that greater diversity would improve success.

Candidates are diverse, not degenerate. For each state we z-score the seven action dimensions across the dataset, flatten each candidate’s executed 16-step chunk to a 112-vector, and take the mean pairwise Euclidean distance among the eight candidates. No state has identical candidates (minimum dispersion 0.18). States without outcome variation show greater absolute dispersion than states with it (pooled median 0.78 vs 0.48; Mann–Whitney $p = 5 \times10^{-5}$; Spearman $\rho= -0.25$ against outcome degree), a pattern driven mainly by TurnOnStove. Scale-free coefficient-of-variation metrics show no significant group difference ($p = 0.07$–$0.10$), so the difference is largely one of absolute action magnitude. Value-score dispersion is uniformly tiny (median standard deviation $\approx3 \times10^{-4}$) and unrelated to outcome variation in the pooled analysis ($p = 0.62$), consistent with a value selector that carries little usable ranking signal; TurnOnStove, where the value selector is worst, is the only task with larger value dispersion on varying states.

Decomposing outcome variation. The 48 of 263 states with any variation (across eight candidates and two repeats) split into those with a stable between-candidate difference and those without one. We call a state stably separable when some candidate succeeds in both repeats while another fails in both.

(p5_diversity.py, p5_noise_decomp.py).

Even the stable states are near chance. A within-state null that independently shuffles the candidate labels in each repeat (preserving each repeat’s success count but breaking candidate identity across repeats, 2000 draws) expects $\approx14.8$ stably separable states by chance, against the observed 20; the excess is about 5 states, mostly PnPCounterToCab. The cross-fitted hindsight oracle (same estimator as the main text) gains +1.40 pp pooled, decomposing as +1.85 pp from the 20 stably separable states and −0.44 pp from the 28 states without stable separation (on which the cross-fitted oracle never beats random). Under this specific label-shuffle null the pooled cross-fitted oracle has a near-zero mean gain (+0.01 pp), providing no indication of upward inflation under the null; with only two repeats per candidate, the amount of reproducible selection headroom remains uncertain. This contribution split is exploratory: the grouping is itself defined from the two repeats and is subject to sampling randomness.

Limitations. Actions cover only the executed first 16 of the 32 generated steps. With two repeats per candidate we cannot adjudicate per state whether a stable separation is real or coincidental; the graded (0.5-valued) states are indistinguishable from noise at this sample size. States within an episode are not independent and the reported $p$-values are not corrected for multiple comparisons, so they are exploratory rather than confirmatory. The diagnostics cover four tasks and three episodes and do not speak to other candidate counts, sampling temperatures, or continuation policies. Full numbers and reproducible scripts are in probes/P5_oracle/

Task	States	Varying	Stably sep.	No stable sep.	No variation
OpenSingleDoor	37	0	0	0	37
PnPCounterToCab	78	10	6	4	68
TurnOnStove	76	33	14	19	43
TurnOffStove	72	5	0	5	67
Pooled	263	48	20	28	215