Introduction
World models that predict how a system will evolve under a sequence of actions are now trained on data from experiments as well as from simulators. In biology the actions are often stimulation programs—light pulses delivered to optogenetically engineered cells, for example—and the hope is that a model trained on such data will answer interventional questions: what would this cell population do under a different program? Before such a model is trusted for that purpose, one wants to know whether its action channel reflects a causal effect of the actions at all. In machine learning this is usually checked with diagnostics that replace the actions by zeros, by another unit’s actions or by a shuffled sequence, or retrain the model without them, and report an effect if predictions get worse. Recent benchmarks for action following in world models instead compare generated rollouts with simulator replays of expert and off-expert actions [9 , 10 ] , or score rollouts under perturbed or commanded actions with trajectory estimators, vision-language judges or human annotation [47 b] ; Oefinger et al. [35 ] propose a ladder of checks that a world-model simulator must pass before its verdicts are trusted. None of them reports a test with finite-sample error control. Neither route is available, or reliable, when the data come from a real experiment that cannot be replayed.
The stand-in trap. Replacement diagnostics fail in a simple way. Experimental designs are often autocorrelated: the next stimulus depends on the stimuli before it. A model that sees only a window of recent history can then use the future actions as a stand-in for earlier history that still drives the outcome. Its predictions change when the future actions change, so every replacement diagnostic reports an effect, even when the actions have none; and a model that predicts better learns the stand-in better (Proposition 6). In development simulations the replacement diagnostics reported effects in up to 249/256 replicates of such a null with twelve units, and in 256/256 for every model with 512 units; in a pixel world in which the logging policy depends on state the model cannot see they do so in 64.1% to 100% of test episodes (§7, Figure 4).
A design-based test. When actions were assigned by a randomized program whose law is known, that law supplies exactly what the diagnostics lack. Drawing alternative action blocks from the design law, given everything the design conditioned on, yields blocks that carry the same stand-in information as the observed one; under the null of no effect within the horizon the observed block is exchangeable with them. Scoring the observed and the resampled blocks with the world model gives a conditional randomization test ([7 ] ) in which the model is only a statistic: it determines power, never validity. Written as an e-value, the test can be multiplied over time and across units and averaged across models without any assumption on the model or on shared noise (§ 3 , Proposition 1). The conditioning matters: in a post hoc analysis, a resampler that draws each action from its marginal frequency rejected the same pixel-world null in 98.4% to 100% of episodes, while the design-law resampler rejected in none for the two-model average (§ 7.2 ).
Pooling amplification. World models raise a second difficulty that does not arise for a single outcome. In an ERK optogenetics experiment, each microscope field receives one light program and records hundreds of cells; in a pixel world, every pixel of a frame shares the frame’s actions. The natural statistic sums the sites’ log predictive densities and bets on the result. If the model errs along the action contrast in a way the sites share, the sum multiplies that error by the number of sites, and the e-value is driven toward zero however strong the true effect is (§ 3.3 ). We met this failure in our own registered analysis: on the ERK positive controls, where light is known to drive the reporter, the pooled statistic gave log E = − 9576 E = -9576 E = − 9576 (§ 8.2 ). In the pixel world, its median log E E E fell from − 17.3 -17.3 − 17.3 to − 6832 -6832 − 6832 as frames grew from 64 to 4096 pixels, while the oracle’s rose from 79.0 to 157 (§ 6 , Figure 1 ).
Figure 1. Pooling amplification in a pixel world model. Median log E E E over test episodes at the larger effect size as frames grow from 64 to 4096 pixels, without shared noise (A), with a frame-wide additive offset (B) and with a multiplicative gain on the blob (C). The pooled, Kish-tempered and contrast-tempered statistics are averages over the two learned models; the oracles use the true dynamics with independence pooling or with the exact joint likelihood. In A and C the Kish factor is close to one and the Kish-tempered curve coincides with the pooled one; the two oracle curves nearly coincide in every panel. Dotted line: log 20, the rejection threshold. Symmetric-log axis, linear for ∣ log E ∣ ≤ 10 |\log E| \le10 ∣ log E ∣ ≤ 10 .
A repair that cannot break validity. Because the design, not the statistic, controls the error rate, the pooled score may be tempered by any factor fixed on training data. We temper by an inverse design effect: the statistic registered for our confirmatory test uses the Kish factor of the residual correlation, and an exploratory variant measures the design effect along the action contrast, which is what governs amplification (§ 3.4 ). The Kish-tempered statistic, selected on synthetic data, gave log E = 349 E = 349 E = 349 on the same ERK test fields in an exploratory reanalysis, and in a confirmatory test registered before the data were obtained it rejected the null on an ERK field that no one involved had opened, with E ≈ 1.1 × 10 10 E \approx1.1 \times10^{10} E ≈ 1.1 × 1 0 10 (§ 8.4 ). In the pixel world, where the Kish factor does not capture the relevant correlation, contrast tempering restored power under every noise condition we simulated, at the larger of the two effect sizes. Estimated factors are fragile, however. The Kish correlation factor and the contrast factor do not change when all per-site precisions are multiplied by one common factor, with predictions, residuals and contrasts held fixed, whereas for unfloored Gaussian scores the tempered log-e e e curve shifts along the log-temperature axis (Proposition 9). A mixture over G G G fixed tempering factors needs no estimated factor and loses at most log G \log G log G in log e-value relative to the best member of that grid chosen in hindsight (Proposition 10); in an exploratory comparison on the saved pixel and ERK-like runs it matched contrast tempering in the pixel world and Kish tempering on ERK-like data (§ 6.4 ). A weighted version that puts most of its weight on the factor chosen on validation data keeps most of validation tuning’s power when that choice transfers, and the mixture’s protection when it does not (§ 3.5 , § 6.5 , § 8.6 ). One property explains why single factors fail and mixtures do not: the log e-value of a unit is concave in the tempering factor and equals zero at zero, so a factor below the best one keeps a proportional share of the evidence, while a factor above it can lose everything (Proposition 13, § 12 ).
Published world models. If pooling amplification is a property of pooled scores and not of our models, it should appear on world models trained by others. We registered three confirmatory tests on released checkpoints that we neither trained nor modified. On DIAMOND, a diffusion world model for Atari games (Alonso et al., [1 ] ), with the mixture as the primary statistic, the pooled score rejected in 0 of 96 units of Boxing, Pong and Breakout in which random actions moved the player, paddle or ball, Kish tempering in 0 and contrast tempering in 1; the mixture rejected in 96 (§ 9 ). The second test covered IRIS, a token-based Transformer world model for 26 Atari games (Micheli et al., [32 ] ), and TD-MPC2, a latent world model for continuous control (Hansen et al., [19 ] ), with a validation-weighted mixture fixed before the data as the primary statistic. Under IRIS’s own token likelihood the pooled score reached the registered power bar, 24 of 32 units, in only 10 of the 26 games; under TD-MPC2’s latent score it did so in 1 of 6 tasks. The weighted mixture reached it in 24 of 26 games and in all 6 tasks (§ 10 ). The third test used recorded video of a real robot, which cannot be replayed: BAIR robot pushing ([12 ] ), scored by iVideoGPT ([46 ] ). Its null arm is exact by construction; its arm with the recorded commands measures power and is conditional on a design law reconstructed from the command records. The weighted mixture rejected in 155 of 159 trajectories. The pooled score did not collapse there: it rejected in 143, and failed badly only in a few trajectories, where the mixture’s loss stayed bounded (§ 11 ). Under exact nulls the statistics of the three tests made 14 rejections in 26,664 tests.
Contributions.
We describe the stand-in trap for action diagnostics of learned world models, give a construction in which replacement diagnostics detect a nonexistent effect with probability tending to one (Proposition 6 ), and show that a design-based test avoids it only if its resampler conditions on the full state the design used; validity also extends to feedback designs under a conditional product law (Assumption 2), for which fixed policies that read observed history with a lag of at least the horizon, use independent randomization and do not read other units’ actions inside the block are a sufficient construction (Proposition 12 ).
We identify pooling amplification, a failure of pooled scores when many sites share an action sequence. It is driven by errors that the sites share along the action contrast—chiefly model misspecification, but also shared noise that lies along the contrast—grows with the number of sites, and is invisible to the usual validation, which ours missed (§ 5.3 ).
We show that within the randomization test, tempering is merely a choice of statistic, fixed on training data and incapable of affecting validity, and we propose measuring the design effect along the action contrast; when the contrast is shared across sites this reduces to a weighted Kish factor (Proposition 7 ). We show that the Kish and contrast factors are invariant under a common rescaling of the per-site precisions that moves the tempered log-e curve, and that a mixture over a fixed grid of factors loses at most the logarithm of the grid size in log e-value relative to the best of them (Propositions 8 –10 ); a version that puts most of its weight on the factor chosen on validation data loses at most log(4/3) relative to that factor (§ 3.5 ).
We give design-only bounds on what any statistic can achieve, computable from the command records before any outcome is opened (Proposition 4 ); they are elementary, but they tell an analyst in advance whether an experiment can yield evidence at all.
We confirm, in three registered tests on four published world-model families—a diffusion model, a token-based Transformer, a decoder-free latent model and a video-token model, on 26 Atari games, 6 control tasks and recorded robot video—that pooling amplification occurs on models we did not build; that tempering by estimated factors did not prevent it; and that mixtures fixed in advance restored or kept power while every statistic kept its error rate under exact nulls (§ 9 –§ 11 ). On the robot video and in three Atari games the pooled score of a model’s own token likelihood did not collapse, and we say where it does and does not (§ 13 ).
We show that the log e-value of a unit is concave in the tempering factor and zero at zero (Proposition 13 ). Hence every negative tempered log e-value is an over-bet, a factor below the best keeps a proportional share of the evidence, and a mixture over a dyadic grid keeps at least half of the best log e-value of any factor in its range, minus the logarithm of the grid size. In 4512 saved units, every observed collapse was an over-bet (§ 12 ).
We report a complete, preregistered validation pipeline for world-model action tests on real experiments, including its failures, with every reported number generated from committed, hashed result files.
Related work. Randomization tests for experiments in which treatments act over time are established: exact tests of the sharp null of no temporal causal effect, with lag-p p p causal estimands, for time-series experiments ([4 ] ); conditional randomization tests of partially sharp nulls indexed by a lag in stepped-wedge trials, and by a distance for spillover ([50 ] ); and tests of the carryover horizon and of non-anticipation in switchback experiments ([26 ] ). The use of machine-learning predictions as randomization-test statistics is also established ([40 , 18 , 17 ] ); we use both and claim neither. E-values and testing by betting ([38 , 44 , 37 , 15 ] ) give the aggregation we rely on; model-X conditional-independence tests with e-values ([16 ] ) and sequential Monte Carlo tests ([13 ] ) are close relatives. Tempering likelihoods under misspecification is established in generalized and safe Bayesian inference ([14 ] ), and learning or weighting a grid of learning rates in online learning ([23 , 42 ] ); we make no priority claim for tempering, for mixtures or for rank calibrators. The tempering formulas are standard; §3.4 and §3.6 give their sources, and the contrast factor resembles the score-based rescalings used in composite likelihood. Criticism of permutation-based feature importance for forcing extrapolation ([21 , 5 ] ) and work on causal confusion, copycat behaviour and policy confounding in imitation and reinforcement learning identify related shortcuts; those works ask whether a policy has learned the wrong cue and repair it with targeted interventions ([11 ] ), adversarial removal of the nuisance information ([45 ] ), or a broader distribution of trajectories ([39 ] ), whereas we ask whether a world model’s action channel reflects an effect and use only the known design, with finite-sample error control. Design-respecting world models ([49 ] ) and advantage-style action channels that contrast a world model’s predictions across actions ([25 ] ) are conceptually adjacent. In biology, randomized optogenetic stimulation has been used for causal inference ([27 ] ); we analyze two published optogenetics datasets ([34 , 29 ] ), released as Nałęcz-Jawecki et al. [33 ] and Lugagne et al. [28 ] .
Organization. Section 2 sets up units, sites, designs and nulls; §3 gives the test and the tempered statistics; §4 states the theory. Sections 5–7 report synthetic validation, pooling amplification and its repair, and the stand-in trap; §8 reports the real-data evidence, and §9–§11 the registered tests on published world models, the last on recorded robot video; §12 gives one account of every collapse; §13 compares all settings and discusses limitations. The appendices give proofs, design laws, all tables and disclosures, the registrations and hashes, the baseline study, the full records of the three published-model tests, and the exploratory studies run after the second version of this paper.
Problem setup
Units, sites and actions
We observe assignment units i = 1 , … , N i=1,\ldots,N i = 1 , … , N at discrete times t t t . At each time unit i i i receives an action A i , t ∈ A A_{i,t} \in\mathcal{A} A i , t ∈ A , and outcomes Y i , c , t Y_{i,c,t} Y i , c , t are recorded at each of its measurement sites c ∈ C i c \in C_i c ∈ C i . All sites of a unit share the unit’s action sequence: actions are assigned to units, not to sites. This distinction matters throughout the paper. In the CcaS/CcaR optogenetic experiment a unit is one cell lineage with a single site; in the ERK experiment a unit is a microscope field, and its sites are the hundreds of cells that receive the same light program; in a pixel world model a unit is an episode and its sites are pixels.
Write a = ( a j , s ) a=(a_{j,s}) a = ( a j , s ) for a complete assignment of actions to all units and times, and Y i , c , s ( a ) Y_{i,c,s}(a) Y i , c , s ( a ) for the corresponding potential outcome. We assume no anticipation: Y i , c , s ( a ) Y_{i,c,s}(a) Y i , c , s ( a ) depends on a a a only through ( a j , r ) r ≤ s (a_{j,r})_{r\leq s} ( a j , r ) r ≤ s . Observed outcomes are Y i , c , s = Y i , c , s ( A ) Y_{i,c,s}=Y_{i,c,s}(A) Y i , c , s = Y i , c , s ( A ) . For an original t t t and a horizon h ≥ 1 h\geq1 h ≥ 1 , the future block of unit i i i is A i , t F = ( A i , t + 1 , … , A i , t + h ) A^F_{i,t}=(A_{i,t+1},\ldots,A_{i,t+h}) A i , t F = ( A i , t + 1 , … , A i , t + h ) and the targets are Y i , c , t + h Y_{i,c,t+h} Y i , c , t + h . Let F t \mathcal{F}_t F t denote the observed joint history up to t t t : all units’ outcomes and actions up to t t t , the clock, and any further covariates that are functions of the past.
Randomized open-loop design
Assumption 1 (Design). For every origin t t t , conditionally on F t \mathcal{F}_t F t and on all potential outcomes,
A ⋅ , t F ∼ ∏ i Q i , t , A^{F}_{\cdot,t} \sim\prod_i Q_{i,t}, A ⋅ , t F ∼ i ∏ Q i , t ,
where Q i , t Q_{i,t} Q i , t is a known law over future blocks that depends only on unit i i i ’s own past actions A i , ≤ t A_{i,\leq t} A i , ≤ t and on a design stratum fixed in advance.
Assumption 1 states that actions were generated by an open-loop randomized program that did not look at outcomes. We call Q i , t Q_{i,t} Q i , t the design law . In the experiments we analyze it is reconstructed from the published protocol and acquisition code (Appendix § 17 ); every validity statement is conditional on that reconstruction, which we call a working law, and approximate laws are treated in § 4 . Two features of the design law matter below. First, Q i , t Q_{i,t} Q i , t may depend on the unit’s entire action history, typically far more of it than a predictive model conditions on. Second, a unit whose design is deterministic at t t t (a zero-entropy law) carries no randomization information, and contributes an e-value of exactly one.
Designs that read past outcomes. Assumption 1 describes open-loop programs. Some experiments, including the trap experiment of § 7 , use a logging policy that reads earlier outcomes or states. The validity results extend to such designs under the following assumption.
Assumption 2 (Design with delayed feedback). For every origin t t t , conditionally on F t \mathcal{F}_t F t , on all potential outcomes and on the frozen training and design information,
A ⋅ , t F ∼ ∏ i Q i , t , A^{F}_{\cdot,t} \sim\prod_i Q_{i,t}, A ⋅ , t F ∼ i ∏ Q i , t ,
where each Q i , t Q_{i,t} Q i , t is known and F t \mathcal{F}_t F t -measurable. For the own-channel null of § 2.3 with a tested independent set I I I , the same product form holds for the blocks of the units in I I I conditionally also on the future blocks of all units outside I I I .
Assumption 1 is the special case in which Q i , t Q_{i,t} Q i , t depends only on the unit’s own past actions and its stratum. The product form is essential: a known marginal law for each unit does not justify multiplying unit-wise factors when one unit’s policy can react to another unit’s actions inside the block. Proposition 12 shows that the validity results hold under Assumption 2, and gives a sufficient condition: each unit’s policy reads observed history only with a lag of at least h h h , never another unit’s actions inside the block, and uses fresh randomization independent of everything else.
Partial sharp nulls
Definition 1 (Global sharp-in-future null). H 0 G ( h ) H_0^{G}(h) H 0 G ( h ) : for every unit i i i , site c c c and origin t t t , Y i , c , t + h ( a ) = Y i , c , t + h ( a ′ ) Y_{i,c,t+h}(a) = Y_{i,c,t+h}(a') Y i , c , t + h ( a ) = Y i , c , t + h ( a ′ ) whenever a a a and a ′ a' a ′ differ only in the future blocks a ⋅ , t F a^{F}_{\cdot,t} a ⋅ , t F of any units.
Definition 2 (Own-channel null). Given a spillover graph N \mathcal{N} N on units, H 0 o w n ( h ; N ) H_0^{\mathrm{own}}(h;\mathcal{N}) H 0 own ( h ; N ) : for every unit i i i , site c c c and origin t t t , Y i , c , t + h ( a ) Y_{i,c,t+h}(a) Y i , c , t + h ( a ) is unchanged when a i , t F a^{F}_{i,t} a i , t F alone is changed. Future actions of N \mathcal{N} N -neighbours may affect Y i , c , t + h Y_{i,c,t+h} Y i , c , t + h ; units that are not neighbours have no effect.
Both nulls are sharp : they constrain every unit’s potential outcomes, not an average effect. A rejection therefore means that some site responds to some future action, not that the response is large or typical. Both are partial : they concern only the effect of the next h h h actions on the outcome at t + h t+h t + h . Actions before the window may have arbitrary effects—this is the dependence that makes action-conditioned prediction possible. At the same target times, the null for horizon h h h therefore implies the null for every shorter horizon, but not for a longer one, whose window includes earlier actions. Shared noise of any form, across sites or across units, is allowed. Rejecting H 0 G H_0^{G} H 0 G means only that global future-action invariance fails; attributing an effect to a unit’s own actions requires H 0 o w n H_0^{\mathrm{own}} H 0 own and its layout assumptions. Conditional randomization tests of partially sharp nulls are not new: Zhang and Zhao [50 ] test nulls indexed by a time lag for lagged treatment effects in stepped-wedge trials, assuming no interference and no anticipation, and nulls indexed by a distance threshold for the range of spillover effects; our contribution is not the null, but the analysis of what goes wrong when it is tested with learned models (§ 2.4 and § 3.3 ).
Admissible statistics. A statistic for unit i i i at origin t t t may use any F t \mathcal{F}_t F t -measurable covariate, unit i i i ’s own targets Y i , c , t + h Y_{i,c,t+h} Y i , c , t + h , and unit i i i ’s candidate future block, and nothing else. In particular, no statistic may use other units’ future actions, and under H 0 G H_0^G H 0 G none may use other units’ targets, which can respond to unit i i i ’s future actions through spillover. This is a requirement to be verified for each dataset, not an assumption: in the CcaS release, for example, the neighbour-exposure covariate is aligned one step ahead of the neighbours’ commands and must be replaced by its lag (Appendix § 17 ). Which sites contribute at an origin is determined by an observation mask (finite own history and target). That this mask does not respond to future actions under the null is a stated assumption about the measurement process; formally, the measurement process, including any randomness in it, is part of the potential outcomes on which Assumption 1 conditions.
Stand-in effects
An action-conditioned world model predicts Y i , c , t + h Y_{i,c,t+h} Y i , c , t + h from a finite window of the past together with the future block. Under an autocorrelated design the future block is correlated with actions that precede the window, and those earlier actions may affect the target through the system’s memory. The future block then carries information about the target that the window lacks, even when H 0 G H_0^G H 0 G holds: future actions act as a stand-in for omitted history. A model trained to predict will use this information, and its predictions will change when the future block is changed.
Common action diagnostics read this change as an action effect. They compare predictions under the observed block with predictions under inputs that break the stand-in dependence—all-zero block, a block taken from another unit or another time, or a model retrained without actions—and report an effect when the prediction loss rises. We call all of them replacement diagnostics (the pixel protocol of § 7 and Appendix § 21 calls them common diagnostics). Nothing in these comparisons controls how often they do so when no effect exists, and a model that predicts better learns the stand-in better. Proposition 6 gives a construction in which the zero-block, other-unit and retraining diagnostics report an “effect” with probability tending to one under H 0 G H_0^G H 0 G ; blocks taken from another time of the same unit are not covered by it.
The design law removes the problem. Blocks drawn from the full history-measurable design law Q i , t Q_{i,t} Q i , t retain the dependence on the recorded design state that the observed block has, so they carry the same stand-in information; under Assumption 1 this state reduces to the unit’s past actions and its stratum. Under H 0 G H_0^G H 0 G and Assumption 1 or 2 2 2 , the observed block is exchangeable with such draws given F t \mathcal{F}_t F t and the targets, and no statistic can separate them except by chance. The conditioning is essential: resampling from a law that ignores part of the state the design used—for example, the marginal frequency of each action—breaks the stand-in dependence again and reproduces the false effect (§ 7 ).
Method
A predictor f f f is trained and frozen, on units whose data do not depend on the test units’ tested actions, before any test outcome is used, and it enters the test only through a statistic. The design law, not the model, controls the error rate; the model determines power.
Scores and per-origin e-values
For unit i i i and origin t t t , let C i , t ⊆ C i C_{i,t} \subseteq C_i C i , t ⊆ C i be the sites whose own history and target are observed, and let n i , t = ∣ C i , t ∣ n_{i,t} = |C_{i,t}| n i , t = ∣ C i , t ∣ . For a candidate block a a a , define the site log score and the pooled unit score
ℓ i , c , t ( a ) = log max { ε , p f ( Y i , c , t + h ∣ H i , c , t , a ) } , S i , t ( a ) = ∑ c ∈ C i , t ℓ i , c , t ( a ) , (1) \ell_{i,c,t}(a) = \log\max\left\{\varepsilon,\ p_f\left(Y_{i,c,t+h} \mid H_{i,c,t},a\right)\right\}, \qquad S_{i,t}(a) = \sum_{c \in C_{i,t}} \ell_{i,c,t}(a), \tag*{(1)} ℓ i , c , t ( a ) = log max { ε , p f ( Y i , c , t + h ∣ H i , c , t , a ) } , S i , t ( a ) = c ∈ C i , t ∑ ℓ i , c , t ( a ) , (1)
where H i , c , t H_{i,c,t} H i , c , t is the site’s admissible history and p f p_f p f is the model’s predictive density. All our reference models use a Gaussian density whose variance, like the floor ε \varepsilon ε , is fixed on training data; the floor is applied per site before summing. The pooled score is the independence log-likelihood of the unit’s sites ([8 ] ). It treats sites as independent, but validity does not rest on that.
For any admissible statistic T i , t ( a ) T_{i,t}(a) T i , t ( a ) —the pooled score, or the tempered scores of § 3.4 —the exact and Monte Carlo (MC) e-values at ( i , t ) (i,t) ( i , t ) are
e i , t e x = exp T i , t ( A i , t F ) ∑ a Q i , t ( a ) exp T i , t ( a ) , (2) e^{\mathrm{ex}}_{i,t}=\frac{\exp T_{i,t}(A^F_{i,t})}{\sum_a Q_{i,t}(a)\exp T_{i,t}(a)}, \tag*{(2)} e i , t ex = ∑ a Q i , t ( a ) exp T i , t ( a ) exp T i , t ( A i , t F ) , (2)
e i , t m c = ( B + 1 ) exp T i , t ( A i , t F ) exp T i , t ( A i , t F ) + ∑ b = 1 B exp T i , t ( A i , t ( b ) ) , A i , t ( 1 ) , … , A i , t ( B ) ∼ i i d Q i , t . (3) e^{\mathrm{mc}}_{i,t}=\frac{(B+1)\exp T_{i,t}(A^F_{i,t})}{\exp T_{i,t}(A^F_{i,t})+\sum_{b=1}^{B}\exp T_{i,t}(A^{(b)}_{i,t})},\qquad A^{(1)}_{i,t},\ldots,A^{(B)}_{i,t}\overset{\mathrm{iid}}{\sim}Q_{i,t}. \tag*{(3)} e i , t mc = exp T i , t ( A i , t F ) + ∑ b = 1 B exp T i , t ( A i , t ( b ) ) ( B + 1 ) exp T i , t ( A i , t F ) , A i , t ( 1 ) , … , A i , t ( B ) ∼ iid Q i , t . (3)
The numerator scores the observed future actions; the denominator averages the score over the design. This is the conditional randomization test [7 ] , with the design law in the role of the known conditional law, written as an e-value. Under H 0 G H_0^G H 0 G and Assumption 1, the observed block is distributed as Q i , t Q_{i,t} Q i , t given F t \mathcal{F}_t F t and the targets (under H 0 o w n H_0^{\mathrm{own}} H 0 own : given F t \mathcal{F}_t F t , the future blocks of units outside the independent set, and the targets of the units in it), and the statistic is a fixed function of the candidate block given these. Both e-values therefore have conditional mean one (Proposition 1). The MC form is valid because it includes the observed score in its denominator, which also caps it at B + 1 B+1 B + 1 . Identical candidate blocks may be merged with their multiplicities, and a zero-entropy design gives e ≡ 1 e\equiv1 e ≡ 1 , as does an origin at which no site is eligible, since the pooled and tempered scores are then identically zero (the tempered score is read as zero there even where its factor would be undefined); statements about the field mean and the site average concern nonempty site sets. We use the exact form when the block space can be enumerated and the MC form otherwise.
Aggregation
Origins are grouped into residue classes r ∈ { 0 , … , h − 1 } r\in\{0,\ldots,h-1\} r ∈ { 0 , … , h − 1 } by t ≡ r ( m o d h ) t\equiv r\pmod h t ≡ r ( mod h ) . Within a class, consecutive origins are h h h apart, so their future blocks do not overlap and each factor is determined by the history at the next origin of the class and the Monte Carlo draws used at its own origin; the factors therefore multiply into an e-value along the class (Proposition 1). Under H 0 G H_0^G H 0 G the joint design also allows multiplication across units:
E r = ∏ i ∏ k : t k ≡ r ( m o d h ) e i , t k , E = 1 h ∑ r = 0 h − 1 E r . (4) E_r=\prod_i\prod_{k:t_k\equiv r\pmod h}e_{i,t_k},\qquad E=\frac{1}{h}\sum_{r=0}^{h-1}E_r. \tag*{(4)} E r = i ∏ k : t k ≡ r ( mod h ) ∏ e i , t k , E = h 1 r = 0 ∑ h − 1 E r . (4)
Averages of e-values are e-values without any independence assumption [44 ] . For H 0 o w n H_0^{\mathrm{own}} H 0 own the product over units is taken within each independent set of the spillover graph, and the set products are averaged; without a known graph, singletons are used.
With K K K reference models we aggregate each model completely and then average the models’ e-values, E = K − 1 ∑ k E ( k ) E=K^{-1}\sum_k E^{(k)} E = K − 1 ∑ k E ( k ) . Because E ≥ max k E ( k ) / K E\geq\max_k E^{(k)}/K E ≥ max k E ( k ) / K , the average loses at most log K \log K log K in log E \log E log E relative to the best model chosen in hindsight, so no model selection is needed. We reject at level α \alpha α iff E ≥ 1 / α E\geq1/\alpha E ≥ 1/ α and report p = min ( 1 , 1 / E ) p=\min(1,1/E) p = min ( 1 , 1/ E ) ; for m m m prespecified hypotheses we use the e-Bonferroni rule E ≥ m / α E\geq m/\alpha E ≥ m / α (Markov’s inequality and a union bound). Where the registration specifies it, we also report a secondary batch randomization p-value at one prespecified common origin, resampling all units jointly under H 0 G H_0^G H 0 G , or the units of one prespecified independent set under H 0 o w n H_0^{\mathrm{own}} H 0 own .
The statistic is a free choice
The validity argument uses only two facts: the observed block is distributed as Q i , t Q_{i,t} Q i , t given F t \mathcal{F}_t F t and the targets (under H 0 o w n H_0^{\mathrm{own}} H 0 own : given F t \mathcal{F}_t F t , the future blocks of units outside the independent set, and the targets of the units in it), and the statistic is fixed once these and the training data are fixed. Every admissible statistic therefore yields a valid test. Choosing the statistic is a question of power alone, and it may be settled by any procedure that uses neither test outcomes nor the test units’ realised future actions—for example, selection on training units or on synthetic data.
Pooling amplification. The pooled score is the natural choice, and it fails in a characteristic way. Since the denominator of (2) is at least any one of its terms, for every block a ′ a' a ′ with Q i , t ( a ′ ) > 0 Q_{i,t}(a')>0 Q i , t ( a ′ ) > 0 ,
log e i , t e x ≤ − [ T i , t ( a ′ ) − T i , t ( A i , t F ) ] − log Q i , t ( a ′ ) , (5) \log e^{\mathrm{ex}}_{i,t}\leq-\left[T_{i,t}(a')-T_{i,t}(A^F_{i,t})\right]-\log Q_{i,t}(a'), \tag*{(5)} log e i , t ex ≤ − [ T i , t ( a ′ ) − T i , t ( A i , t F ) ] − log Q i , t ( a ′ ) , (5)
and the MC form obeys log e i , t m c ≤ log ( B + 1 ) − log K ( a ′ ) − [ T i , t ( a ′ ) − T i , t ( A i , t F ) ] \log e^{\mathrm{mc}}_{i,t} \le\log(B+1) - \log K(a') - [T_{i,t}(a') - T_{i,t}(A^{F}_{i,t})] log e i , t mc ≤ log ( B + 1 ) − log K ( a ′ ) − [ T i , t ( a ′ ) − T i , t ( A i , t F )] for every drawn block a ′ a' a ′ , where K ( a ′ ) ≥ 1 K(a') \ge1 K ( a ′ ) ≥ 1 is the number of times a ′ a' a ′ occurs among the B + 1 B+1 B + 1 blocks. For the pooled score the bracket is a sum over the n i , t n_{i,t} n i , t sites. If the per-site score differences ℓ i , c , t ( a ′ ) − ℓ i , c , t ( A i , t F ) \ell_{i,c,t}(a') - \ell_{i,c,t}(A^{F}_{i,t}) ℓ i , c , t ( a ′ ) − ℓ i , c , t ( A i , t F ) have an average over sites bounded away from zero for some block a ′ a' a ′ whose design probability does not shrink as sites are added—because the model errs along the action contrast in a direction the sites share—the bound decreases at least linearly in n i , t n_{i,t} n i , t , whatever the strength of the true effect. Independent site errors partly cancel in the sum; common errors do not. The pooled e-value thus behaves like an all-in bet on the model’s top-ranked block: valid, but with log E \log E log E driven far below zero as sites are added. We call this failure pooling amplification .
Errors common to the sites along the contrast arise in two ways. A misspecified model can err in its predicted response to the action, which all sites of a unit share; and noise shared by the sites can lie along the contrast, as when field-wide fluctuations move all cells in the direction an action would. Shared noise orthogonal to the contrast does not amplify. In the pixel world of § 6 , a correctly specified model is unaffected by both forms of shared noise we simulate, while the fitted models collapse under every noise condition; the cause there is misspecification.
Tempered scores
We temper the pooled score,
T i , t ( a ) = λ i , t S i , t ( a ) , 0 < λ i , t ≤ 1 , (6) T_{i,t}(a) = \lambda_{i,t} S_{i,t}(a), \qquad0 < \lambda_{i,t} \le1, \tag*{(6)} T i , t ( a ) = λ i , t S i , t ( a ) , 0 < λ i , t ≤ 1 , (6)
with λ i , t \lambda_{i,t} λ i , t a function of training data and of the site count n i , t n_{i,t} n i , t only. The count is determined by the observation mask. Under the stated assumption that the mask does not respond to the tested blocks under the null (§ 2.3 ), λ i , t \lambda_{i,t} λ i , t is fixed given the conditioning field of Proposition 1 , so (6) is admissible and Proposition 1 applies unchanged. (The same assumption is already needed for the pooled score, whose site set is the mask; if the mask responded to the tested blocks, neither score would be covered. The restriction λ ≤ 1 \lambda\le1 λ ≤ 1 is not needed for validity.) Setting λ = 1 \lambda= 1 λ = 1 recovers the pooled score, and λ → 0 \lambda\to0 λ → 0 gives e → 1 e \to1 e → 1 ; 1 / λ i , t 1/\lambda_{i,t} 1/ λ i , t can be read as the number of sites that together carry one site’s worth of independent evidence about the action.
Tempering a log-likelihood in this way has a long history: Kish’s design effect for clustered samples [22 ] , adjustments that rescale independence and composite log-likelihoods so that likelihood-ratio statistics recover their usual reference distribution [8 , 43 , 36 ] , and learning rates or power likelihoods for misspecified models [14 , 20 ] . We claim no novelty for the formulas below.
Kish tempering (registered). With ρ ^ \hat{\rho} ρ ^ an estimate of the correlation between sites’ residuals,
λ i , t K = 1 1 + ( n i , t − 1 ) ρ ^ , (7) \lambda^{\mathrm{K}}_{i,t} = \frac{1}{1 + (n_{i,t}-1)\hat{\rho}}, \tag*{(7)} λ i , t K = 1 + ( n i , t − 1 ) ρ ^ 1 , (7)
the inverse design effect of a cluster of n i , t n_{i,t} n i , t sites with intraclass correlation ρ ^ \hat{\rho} ρ ^ . In the ERK analyses ρ ^ \hat{\rho} ρ ^ is computed separately for each reference model and horizon from training-field validation residuals only: Pearson correlations between the residual series of each pair of sites over their common eligible origins (pairs with fewer than three finite, non-constant matched values are omitted), then the median over pairs within each training field, the median over training fields, and clipping to [ 0 , 1 ] [0,1] [ 0 , 1 ] . The pixel simulation uses the same estimator on standardized validation residuals, with episodes in place of fields. This statistic was selected on synthetic data from three candidates under a rule committed before the evaluation; all three reached full power, the deciding comparison was narrow, and one tie-resolution detail of the rule was fixed only in code committed after the evaluation (§ 6 ). It is the statistic registered for the confirmatory test M5 (§ 8 ).
Contrast tempering (exploratory). Kish tempering measures how strongly residuals are correlated across sites. Amplification, however, depends on how strongly the sites’ contributions to the score difference between candidate blocks are correlated. For a Gaussian score with predicted mean μ c , t ( a ) \mu_{c,t}(a) μ c , t ( a ) and site variance v c v_c v c , replacing the observed block by a design draw a b a^b a b lowers the site log score, to first order in the mean and when the per-site floor is inactive, by
z c , t b = r c , t D c , t b v c , r c , t = Y c , t + h − μ c , t ( A t F ) , D c , t b = μ c , t ( A t F ) − μ c , t ( a b ) , z^b_{c,t}=\frac{r_{c,t}D^b_{c,t}}{v_c},\qquad r_{c,t}=Y_{c,t+h}-\mu_{c,t}(A^F_t),\qquad D^b_{c,t}=\mu_{c,t}(A^F_t)-\mu_{c,t}(a^b), z c , t b = v c r c , t D c , t b , r c , t = Y c , t + h − μ c , t ( A t F ) , D c , t b = μ c , t ( A t F ) − μ c , t ( a b ) ,
where D c , t b D^b_{c,t} D c , t b is the action contrast . The contrast-tempering factor is the ratio of the mean square that the pooled score implicitly assumes for the sum of these contributions (independent sites) to their actual mean square, estimated on validation data with B c B_c B c design draws a b ∼ Q t a^b\sim Q_t a b ∼ Q t per origin:
λ c = ∑ t , b ∑ c ( z c , t b ) 2 ∑ t , b ( ∑ c z c , t b ) 2 , \lambda^c=\frac{\sum_{t,b}\sum_c(z^b_{c,t})^2}{\sum_{t,b}(\sum_c z^b_{c,t})^2}, λ c = ∑ t , b ( ∑ c z c , t b ) 2 ∑ t , b ∑ c ( z c , t b ) 2 ,
clipped to [ 1 / n , 1 ] [1/n,1] [ 1/ n , 1 ] . {#eq:8}
It is computed once per model and setting from validation episodes only, and is written and hashed before any test e-value is computed. The ratio is defined when its denominator is positive. If the contributions have zero cross moments across sites, its expected numerator and denominator agree; the ratio itself need not have expectation one, but it tends to one when the two sums, suitably normalized, satisfy a law of large numbers with a common positive limit. If the contributions are identical across a common number n n n of sites, it equals 1 / n 1/n 1/ n . The two factors can disagree in either direction. When a model misplaces a moving object, its residuals take both signs across pixels and their pairwise correlation averages to nearly zero, so λ K ≈ 1 \lambda^{\mathrm{K}}\approx1 λ K ≈ 1 ; weighted by the contrast, which has the same spatial pattern as the error, the same residuals add coherently, and λ c ≪ 1 \lambda^c\ll1 λ c ≪ 1 . Conversely, a frame-wide offset makes residuals strongly correlated, so λ K \lambda^{\mathrm{K}} λ K is small, but the offset is orthogonal to a contrast that moves intensity without changing its total, so λ c ≈ 1 \lambda^c\approx1 λ c ≈ 1 . Contrast tempering was declared in the simulation protocol, before the pixel experiment was run, as an exploratory generalization. In the registered test on a published world model it was a secondary statistic and did not repair pooling amplification (§ 9 ); we report it but do not recommend it.
When Kish tempering suffices. If the action contrast is the same at every site and the site variances are equal—as for an additive model whose action branch sees only the unit’s shared actions, the form of the ERK reference models—the contrast factor (8), where defined, satisfies the weighted cross-product identity of Proposition 7; for a common site count it has the form of a Kish factor whose correlation is a contrast-weighted ratio of uncentred residual cross-products, which is not in general the median-of-Pearson estimate in (7).
Statistics that need no estimated factor
The factors of § 3.4 are estimated from second moments of validation residuals. Three further statistics avoid that estimate or replace it by a direct search; all keep the design law, the bags and the aggregation, and all are admissible.
Tempering mixture. For a fixed grid λ g = 2 − g \lambda_g=2^{-g} λ g = 2 − g , g = 0 , … , G − 1 g=0,\ldots,G-1 g = 0 , … , G − 1 , let E ( λ g ) E(\lambda_g) E ( λ g ) be the e-value of a whole decision unit computed with the constant factor λ g \lambda_g λ g at every origin, and
E M = 1 G ∑ g = 0 G − 1 E ( λ g ) . (8) E^{\mathrm{M}}=\frac{1}{G}\sum_{g=0}^{G-1}E(\lambda_g). \tag*{(8)} E M = G 1 g = 0 ∑ G − 1 E ( λ g ) . (8)
Each E ( λ g ) E(\lambda_g) E ( λ g ) is an e-value because λ g \lambda_g λ g is fixed, and so is their average; pathwise, log E M ≥ max g log E ( λ g ) − log G \log E^{\mathrm{M}}\geq\max_g\log E(\lambda_g)-\log G log E M ≥ max g log E ( λ g ) − log G . The average is taken over whole units, never over per-origin factors. The grid reaches 1 / n 1/n 1/ n at the largest site count of the application: G = 13 G=13 G = 13 in the pixel world and G = 15 G=15 G = 15 for the 64 × 64 × 3 64\times64\times3 64 × 64 × 3 sites of § 9 , where the 13-value version is also reported for continuity. Learning a learning rate over an exponentially spaced grid, or weighting a grid of learning rates by their performance, is standard in online learning [23 , 42 ] ; Proposition 10 gives the guarantee we use.
Validation-tuned tempering. λ V \lambda^{V} λ V is the grid value with the largest median validation log E \log E log E , computed per model with the same validation episodes and design draws as the contrast factor, ties going to the larger value; the test statistic is (6) with λ = λ V \lambda=\lambda^{V} λ = λ V . It depends on validation data only.
Validation-weighted mixture. Validation tuning stakes everything on one factor chosen on validation data; the mixture spreads its stake evenly. A weighted mixture does both. With g V g_{V} g V the grid index of λ V \lambda_{V} λ V and a weight w w w fixed in advance,
E W ( w ) = w E ( λ g V ) + 1 − w G − 1 ∑ g ≠ g V E ( λ g ) . (9) E^{W}(w)=wE(\lambda_{g_{V}})+\frac{1-w}{G-1}\sum_{g\ne g_{V}}E(\lambda_{g}). \tag*{(9)} E W ( w ) = w E ( λ g V ) + G − 1 1 − w g = g V ∑ E ( λ g ) . (9)
Because g V g_{V} g V depends on validation data only, E W ( w ) E^{W}(w) E W ( w ) is an average of e-values with weights fixed before any test unit is seen, so it is an e-value. Pathwise, log E W ( w ) ≥ log E ( λ g V ) − log ( 1 / w ) \log E^{W}(w)\ge\log E(\lambda_{g_{V}})-\log(1/w) log E W ( w ) ≥ log E ( λ g V ) − log ( 1/ w ) , and log E W ( w ) ≥ log E ( λ g ) − log { ( G − 1 ) / ( 1 − w ) } \log E^{W}(w)\ge\log E(\lambda_{g})-\log\{(G-1)/(1-w)\} log E W ( w ) ≥ log E ( λ g ) − log {( G − 1 ) / ( 1 − w )} for every other g g g . With G = 15 G=15 G = 15 and w = 3 / 4 w=3/4 w = 3/4 the losses are at most log ( 4 / 3 ) ≈ 0.29 \log(4/3)\approx0.29 log ( 4/3 ) ≈ 0.29 against λ V \lambda_{V} λ V and log 56 ≈ 4.03 \log56\approx4.03 log 56 ≈ 4.03 against any other grid member, compared with log 15 ≈ 2.71 \log15\approx2.71 log 15 ≈ 2.71 against every member for the uniform mixture, which is the case w = 1 / G w=1/G w = 1/ G . Weighting a grid by prior information is standard; we claim no priority. W ( 3 / 4 ) W(3/4) W ( 3/4 ) is the primary statistic of § 10 ; W ( 1 / 4 ) W(1/4) W ( 1/4 ) and W ( 1 / 2 ) W(1/2) W ( 1/2 ) are reported as secondary.
Site-count-adaptive factors. When units or origins differ in their number of sites, one factor λ \lambda λ cannot suit all of them. A design-effect form lets the factor follow the site count: λ i , t ( ρ ) = 1 / { 1 + ( n i , t − 1 ) ρ } \lambda_{i,t}(\rho)=1/\{1+(n_{i,t}-1)\rho\} λ i , t ( ρ ) = 1/ { 1 + ( n i , t − 1 ) ρ } , with n i , t n_{i,t} n i , t the number of sites of unit i i i at origin t t t and ρ \rho ρ on the grid 2 − g 2^{-g} 2 − g . Validation tuning, the mixture and the weighted mixture over this grid are written V ρ V_{\rho} V ρ , M ρ M_{\rho} M ρ and W ρ ( w ) W_{\rho}(w) W ρ ( w ) . They are e-values for the same reason as their counterparts. We use them in § 8.7 .
Online rules. A factor can also be chosen within a unit, from that unit’s earlier origins. At origin t t t , the follow-the-leader rule uses the grid factor with the largest cumulative log e-value over the origins before t t t ; at the first origin it uses the smallest grid factor, and ties go to the smaller factor. The choice depends only on earlier origins, so each factor is fixed before its own origin’s label is used, and the product over origins is an e-value under the conditions of Proposition 1. O15 uses the 15 grid values, O57 a finer grid of 57 values 2 − j / 4 2^{-j/4} 2 − j /4 , j = 0 , … , 56 j=0,\ldots,56 j = 0 , … , 56 , and O57h uses half of O57’s choice at every origin. The uniform mixture is itself an online rule (Proposition 13(e)): it weights the grid factors at each origin by the wealth each has earned so far.
Rank e-values. At each origin, let k k k be the position of the observed block’s score among the B + 1 B+1 B + 1 candidate scores, from the highest, with tied positions averaged (identical blocks receive one score). With g ( k ) ∝ ( k / ( B + 1 ) ) κ − 1 g(k)\propto(k/(B+1))^{\kappa-1} g ( k ) ∝ ( k / ( B + 1 ) ) κ − 1 normalized to mean one over the B + 1 B+1 B + 1 positions, the factor is g ( k ) g(k) g ( k ) , averaged over tied positions. Under the exchangeable null a rank with uniform random tie-breaking is uniform, and averaging g g g over the observed tie block preserves mean one; positive rescaling preserves ranks and tie blocks, so the factor is the same for the pooled and both tempered scores. We use κ = 1 / ln ( B + 1 ) \kappa=1/\ln(B+1) κ = 1/ ln ( B + 1 ) (R) and κ = 1 / 2 \kappa=1/2 κ = 1/2 (R1 / 2 _{1/2} 1/2 ). Rank calibrators of this kind are p-to-e calibrators (Vovk and Wang, [44 ] ). For untied scores they pay an expected price per factor under the null (Proposition 11); averaging over tied positions keeps validity but lowers that price, to zero when all candidates tie.
A betting reading
In the exact form, e i , t = Q i , t T ( A i , t F ) / Q i , t ( A i , t F ) e_{i,t}=Q^{T}_{i,t}(A^{F}_{i,t})/Q_{i,t}(A^{F}_{i,t}) e i , t = Q i , t T ( A i , t F ) / Q i , t ( A i , t F ) with Q i , t T ( a ) ∝ Q i , t ( a ) e T i , t ( a ) Q_{i,t}^{T}(a)\propto Q_{i,t}(a)e^{T_{i,t}(a)} Q i , t T ( a ) ∝ Q i , t ( a ) e T i , t ( a ) . The e-value is the likelihood ratio of an alternative design law that up-weights blocks the model scores highly—equivalently, the wealth multiplier of a bet against the null placed according to the model ([38 , 37 , 15 ] ). Tempering replaces this alternative by
Q i , t λ S ∝ Q i , t 1 − λ ( Q i , t S ) λ , Q_{i,t}^{\lambda S}\propto Q_{i,t}^{1-\lambda}(Q_{i,t}^{S})^{\lambda}, Q i , t λ S ∝ Q i , t 1 − λ ( Q i , t S ) λ ,
a geometric interpolation between the null law and the pooled bet. Fractional Kelly betting ([30 , 31 ] ), advocated when one is not confident about the alternative ([37 ] ), instead mixes arithmetically, Q κ = ( 1 − κ ) Q + κ Q S Q_{\kappa}=(1-\kappa)Q+\kappa Q^{S} Q κ = ( 1 − κ ) Q + κ Q S , which gives e = 1 − κ + κ e S e=1-\kappa+\kappa e^{S} e = 1 − κ + κ e S , we include it, with κ = 1 / 2 \kappa= 1/2 κ = 1/2 applied per origin, as a reference arm in § 6 . Both shrink the alternative toward the null as a guard against misspecification, and both keep the pooled bet’s ranking of blocks. They interpolate along different paths: tempering along the exponential family Q S λ Q^\lambda_S Q S λ , fractional Kelly along the mixture family Q κ Q_\kappa Q κ . Geometric shrinkage rescales the score: when per-site score differences are bounded, a factor λ \lambda λ of order 1 / n 1/n 1/ n removes the linear growth of ∣ log e ∣ |\log e| ∣ log e ∣ in the number of sites; arithmetic mixing bounds each factor below by 1 − κ 1-\kappa 1 − κ . When the score takes only two values on the support of Q Q Q (for example, a single binary action) the two families coincide, each λ \lambda λ matching an origin-specific κ \kappa κ . With three or more distinct score values they differ at every interior parameter, 0 < λ < 1 0 < \lambda< 1 0 < λ < 1 , although they share their endpoints; the two are different members of one family of defences.
Statistics used in this paper
Table 1 lists every statistic that appears in the paper, with its label in the registered and protocol documents and its evidential status. Under the conditions of Proposition 1 they give valid tests and differ only in power. The pooled and tempered scores are admissible statistics in (2)–(3); the field-mean and oracle variants of § 6 are too, on their defined domains, when their candidate scores use only the unit’s own history, targets and candidate block; the site average averages, over a nonempty site set fixed in the null conditioning, the per-site e-values that these equations give; and fractional Kelly averages an e-value with the constant one. Averages of e-values are again e-values. The full definitions of the variants are in the frozen protocols and are not restated here.
Table 1. Statistics used in this paper. “Label” is the name in the registered and protocol documents; the paper uses the names in the first column.
Notation in the registered documents. The registered M4 documents write λ \lambda λ for the precision 1 / ( 2 σ 2 ) 1/(2\sigma^2) 1/ ( 2 σ 2 ) of the Gaussian score; in this paper λ \lambda λ denotes only the tempering factor. The labels C1–C5 of the sealed-challenge criteria (§ 5 ) are unrelated to the statistic labels in Table 1 . The specification writes G t G_t G t for the observed history and G G G for the spillover graph; we write F t F_t F t and N N N . Internal labels of our studies (P1–P4, A1, A1-G, A2, ERK-XM) appear in repositories and registrations; the paper uses descriptive names, mapped in Appendix § 25 .
Theory
The statements below restate the obligations of the registered specification, whose labels are given in parentheses, with notation adapted and with the hypotheses that the frozen proofs use stated explicitly. The proofs are in Appendix § 16 . They follow the theory documents that were written before the registration and frozen with it (Appendix § 16 lists them with their hashes), and Appendix § 16 marks each place where we have added a hypothesis or tightened an argument; none of these changes affects the registered method. Proposition 7 and Propositions 8–12 are new in this paper and were not part of any registration. Every guarantee is conditional on the design law used for resampling being the law that generated the actions; § 4.3 treats departures from it.
Validity
Proposition 1 (Validity; P1(a)–(c)). Suppose Assumption 1 holds at every origin, and that
(i) the units, origins and sites that enter each product are fixed by structural events (layout, design allocation) that do not depend on the tested blocks, by own-path eligibility events treated in Proposition 2 (b), or by the observation mask, which under the null in force does not respond to the tested blocks (§ 2.3 );
(ii) the predictor and every score parameter (floor, variance, tempering factor) are functions of the observation mask and of training information that is conditionally independent of the joint vector of tested blocks given F t F_t F t and the potential outcomes;
(iii) Monte Carlo draws are generated from Q i , t Q_{i,t} Q i , t at the realised past, with randomness independent of the potential outcomes, of the assignments, and across units and origins.
Then:
(a) Under H 0 G ( h ) H_0^G(h) H 0 G ( h ) , conditionally on F t F_t F t , the potential outcomes and the targets at t + h t+h t + h of all units, each exact or MC factor has mean one, and so does their product across units. Along each residue grid the running product is a non-negative martingale with initial value one, with respect to the filtration generated by the history at the grid origins, the frozen training and design information, and the Monte Carlo seeds already used.
(b) Under H 0 o w n ( h ; N ) H_0^{\mathrm{own}}(h;N) H 0 own ( h ; N ) , let I I I be an independent set of N N N fixed before testing, and let the statistic of each i ∈ I i\in I i ∈ I use F t F_t F t -measurable covariates and its own candidate block, only unit i i i ’s own targets. Then, conditionally on F t F_t F t , the potential outcomes, the future blocks of the units outside I I I and the targets of the units in I I I , each exact or MC factor of a unit in I I I has mean one, and so does their product over I I I ; along each residue grid the running product over I I I is a martingale as in (a). The conditioning may not include the targets of units outside I I I : through spillover these can depend on the blocks in I I I .
(c) Prechosen arithmetic averages over residue classes, independent sets and models of the quantities in (a)–(b) are e e e -values at the fixed analysis time, so Pr ( E ≥ 1 / α ) ≤ α \Pr(E\ge1/\alpha)\le\alpha Pr ( E ≥ 1/ α ) ≤ α . A batch randomization p p p -value computed at one prespecified common origin, with a statistic that is a fixed function of the joint candidate block given the conditioning field of (a) (respectively (b)), resampling all units jointly under H 0 G H_0^G H 0 G or the units of one independent set under H 0 o w n H_0^{\mathrm{own}} H 0 own , and counting ties as exceedances, is super-uniform.
The argument is short. Under H 0 G H_0^G H 0 G and no anticipation, the targets at t + h t+h t + h are functions of the potential-outcome functions and of the actions up to t t t , so conditioning on them leaves the design law of the tested blocks unchanged. The denominator of (2) is then the conditional expectation of its numerator; in the MC form, the observed block and its draws are exchangeable and the observed share of the total score has mean 1 / ( B + 1 ) 1/(B+1) 1/ ( B + 1 ) . Shared outcome noise is part of the potential outcomes and requires no independence. A factor at a grid origin t k t_k t k is determined by the history at t k + 1 = t k + h t_{k+1}=t_k+h t k + 1 = t k + h and the MC draws used at t k t_k t k , which gives the martingale; averages of e-values are e-values without independence [44 ] . Under H 0 n H_0^n H 0 n the targets of units in an independent set are invariant to the set’s own blocks but those of neighbours outside it need not be, which is why (b) conditions on, and scores with, only the set’s own targets. Two limits are worth stating. Optional stopping is licensed only along a single residue grid (by Ville’s inequality); the average over classes is a fixed-time e-value, not an e-process on a common clock. And nothing in the argument depends on the predictor’s accuracy, on independence between sites, or on the noise distribution: these affect power only.
Proposition 2 (Eligibility; P1(d), P1(e)). (a) Let the structural eligibility events not depend on the tested blocks, and run the structural and the classifier-based pipelines with the same statistics and the same Monte Carlo draws whenever their eligible sets coincide. If the event that any classifier decision differs from its structural event has probability at most δ \delta δ under the law that generated the commands, the classifier-based test has size at most min ( 1 , α + δ ) \min(1,\alpha+\delta) min ( 1 , α + δ ) . The bound is unconditional over repetitions of the experiment and classifier; it is not a guarantee conditional on the selected set.
(b) Suppose Assumption 1 holds at every origin. Let unit i i i be analysed only on an event E i E_i E i that is a fixed function of its own complete action path, chosen before the data and depending on no other unit’s actions and on no outcome (for example, that the path avoids prespecified fixed templates). Suppose that every unit on which E i E_i E i holds follows Q i Q_i Q i (the template list contains every fixed design), and that Pr ( E i ∣ A i , ≤ t ) > 0 \Pr(E_i\mid A_{i,\le t})>0 Pr ( E i ∣ A i , ≤ t ) > 0 at every analysed origin. Then replacing Q i , t Q_{i,t} Q i , t by Q i , t ( ⋅ ∣ E i , A i , ≤ t ) Q_{i,t}(\mathord{\cdot}\mid E_i,A_{i,\le t}) Q i , t ( ⋅ ∣ E i , A i , ≤ t ) keeps Proposition 1 exact, conditionally on the eligibility pattern of all units. For a hidden-Markov design this conditional law is computed exactly by a forward–backward recursion on the hidden state augmented with template-match indicators.
Part (b) is used for the CcaS random-walk lineages, whose resampling law is conditioned on not reproducing either duplicated fixed template; lineages that reproduce a template are excluded (Appendix C). Part (a) enters only the secondary own-channel analyses; with no independent record of the physical layout, the only bound we can establish is δ ≤ 1 \delta\le1 δ ≤ 1 , and we display it as vacuous next to every such result.
Proposition 3 (Distributional nulls do not compose; P1b). Let ( A 1 , k , A 2 , k , U k ) (A_{1,k},A_{2,k},U_k) ( A 1 , k , A 2 , k , U k ) , k = 1 , 2 , 3 k=1,2,3 k = 1 , 2 , 3 , be independent fair bits, drawn afresh at each of three origins; take horizon h = 1 h=1 h = 1 and design law Q = ( 1 2 , 1 2 ) Q=(\frac{1}{2},\frac{1}{2}) Q = ( 2 1 , 2 1 ) . Consider the systems (i) Y 1 , k = U k Y_{1,k}=U_k Y 1 , k = U k , Y 2 , k = U k ⊕ A 1 , k ⊕ A 2 , k Y_{2,k}=U_k\oplus A_{1,k}\oplus A_{2,k} Y 2 , k = U k ⊕ A 1 , k ⊕ A 2 , k , and (ii) Y i , k = A i , k ⊕ U k Y_{i,k}=A_{i,k}\oplus U_k Y i , k = A i , k ⊕ U k , i = 1 , 2 i=1,2 i = 1 , 2 . In both, for every unit i i i and origin k k k , A i , k A_{i,k} A i , k is independent of Y i , k Y_{i,k} Y i , k given both units’ earlier actions and outcomes and the other unit’s actions, so each unit-wise conditional-independence null holds and each unit-wise e-process has mean one. Score each unit by the Gaussian density N ( Y i , k ; A i , k , σ 2 ) \mathcal{N}(Y_{i,k};A_{i,k},\sigma^2) N ( Y i , k ; A i , k , σ 2 ) with σ 2 = 1 / ( 2 log 9 ) \sigma^2=1/(2\log9) σ 2 = 1/ ( 2 log 9 ) , so that the matching bit is nine times as likely as the other and each exact factor equals 9 / 5 9/5 9/5 or 1 / 5 1/5 1/5 . Then the product of the unit-wise e-values across both units and the three origins has mean 1.64 3 ≈ 4.41 1.64^3\approx4.41 1.6 4 3 ≈ 4.41 and exact rejection probability 1 / 8 1/8 1/8 at α = 0.05 \alpha=0.05 α = 0.05 .
The value 1 / 8 1/8 1/8 is the probability that all six factors equal 9 / 5 9/5 9/5 ; among density ratios r ≥ 1 r\ge1 r ≥ 1 it holds exactly when r ≥ 20 1 / 6 / ( 2 − 20 1 / 6 ) ≈ 4.675 r\ge20^{1/6}/(2-20^{1/6})\approx4.675 r ≥ 2 0 1/6 / ( 2 − 2 0 1/6 ) ≈ 4.675 , and the rejection probability is zero for 1 ≤ r < 4.675 1\le r<4.675 1 ≤ r < 4.675 . Fresh noise is part of the construction: if U U U were drawn once and held fixed, earlier outcomes would reveal it, and after the first origin the unit-wise null would fail for unit 2 in system (i) and for both units in system (ii). Both systems violate the sharp nulls, so our four tests they are alternatives. The proposition explains why the product across units in Proposition 1(a) needs a sharp null: unit-wise distributional statements say nothing about how one unit’s action enters another unit’s outcome.
What the design permits
Proposition 4 (Design limits; P2(a)–(e), regrouped).
(a) [P2(a), P2(c)] Pathwise,
e i , t e x ≤ 1 Q i , t ( A i , t F ) , e i , t m c ≤ B + 1 1 + K . e^{\mathrm{ex}}_{i,t} \le\frac{1}{Q_{i,t}(A^{F}_{i,t})}, \qquad e^{\mathrm{mc}}_{i,t} \le\frac{B+1}{1+K}. e i , t ex ≤ Q i , t ( A i , t F ) 1 , e i , t mc ≤ 1 + K B + 1 . where K K K is the number of draws equal to the observed block. A zero-entropy design gives e ≡ 1 e \equiv1 e ≡ 1 .
(b) [P2(b), P2(c)] Under any alternative, the actions still follow the design law, so
Pr Q ( E ≥ 1 / α ) ≤ Pr Q ( max components factors ∑ factors − log Q i , t ( A i , t F ) ≥ log ( 1 / α ) ) . \Pr_{Q}(E \ge1/\alpha) \le\Pr_{Q}\left(\max_{\substack{\text{components}\\\text{factors}}} \sum_{\text{factors}} -\log Q_{i,t}(A^{F}_{i,t}) \ge\log(1/\alpha)\right). Q Pr ( E ≥ 1/ α ) ≤ Q Pr components factors max factors ∑ − log Q i , t ( A i , t F ) ≥ log ( 1/ α ) . the maximum running over residue classes and independent sets, and the probability over coherent full command paths (blocks of different residue classes share actions). The right-hand side is computable from commands alone. With m m m prespecified hypotheses the threshold 1 / α 1/\alpha 1/ α becomes m / α m/\alpha m / α ; under Proposition 2 (b), Q i , t Q_{i,t} Q i , t and Pr Q \Pr_Q Pr Q are replaced by their conditioned versions. The MC form has an analogous ceiling, with log ( B + 1 ) − log ( 1 + K ) \log(B+1)-\log(1+K) log ( B + 1 ) − log ( 1 + K ) in place of − log Q -\log Q − log Q and K ∼ Binomial ( B , Q i , t ( A i , t F ) ) K \sim\operatorname{Binomial}(B,Q_{i,t}(A^{F}_{i,t})) K ∼ Binomial ( B , Q i , t ( A i , t F )) ; it can exceed the exact-form ceiling.
(c) [P2(d)] With T \mathcal{T} T the frozen training information on which the score depends,
E [ log e i , t e x ∣ F t , T ] ≤ I ( Y ⋅ , t + h ; A i , t F ∣ F t , T ) ≤ H Q ( A i , t F ∣ F t ) . \mathbb{E}\left[\log e^{\mathrm{ex}}_{i,t}\mid\mathcal{F}_t,\mathcal{T}\right] \le I\left(Y_{\cdot,t+h};A^{F}_{i,t}\mid\mathcal{F}_t,\mathcal{T}\right) \le H_Q\left(A^{F}_{i,t}\mid\mathcal{F}_t\right). E [ log e i , t ex ∣ F t , T ] ≤ I ( Y ⋅ , t + h ; A i , t F ∣ F t , T ) ≤ H Q ( A i , t F ∣ F t ) . (d) [P2(e)] For the batch test, p ≥ Q j o i n t ( A o b s F ) p \ge Q^{\mathrm{joint}}(A^{F}_{\mathrm{obs}}) p ≥ Q joint ( A obs F ) in the exact form and p ≥ ( 1 + K ) / ( B + 1 ) p \ge(1+K)/(B+1) p ≥ ( 1 + K ) / ( B + 1 ) in the add-one MC form.
Part (a) is one line—the observed term appears in its own denominator—and we do not present it as a deep result. Its value is practical. Parts (a)–(b) tell an analyst, before any outcome is opened, whether any statistic could reach the rejection threshold; a deterministic stimulation protocol yields no evidence whatever the model. Because averaging models adds no design entropy, the same command-only bound applies to the model average. For the ERK analyses (M4, X and M5; Figure 5 ) we report the exact-form bound log E u p p e r \log E_{\mathrm{upper}} log E upper as a command-only comparator. Those analyses use Monte Carlo e e e -values, which the exact-form bound does not bound pathwise (part (b)), so the ratio log E / log E u p p e r \log E/\log E_{\mathrm{upper}} log E / log E upper is descriptive: it could in principle exceed one, and it is not a power percentage. Part (c) is a growth statement only.
Figure 5. The ERK evidence chain. Four-model log E E E of the registered pooled statistic in M4 (nine test fields), and of the Kish-tempered statistic in the exploratory analysis X (the same fields) and in the confirmatory test M5 (one unopened field; thirty minutes descriptive only). Horizontal bars mark the exact-form design bound of each analysis (Proposition 4 ), a command-only comparator for these Monte Carlo e-values; dashed and dotted lines mark the rejection thresholds of M4 (E ≥ 100 E \ge100 E ≥ 100 ) and M5 (E ≥ 20 E \ge20 E ≥ 20 ). Symmetric-log axis.
Approximate design laws
Proposition 5 (Approximate law; P3). Suppose the test resamples from a working law Q ^ \widehat{Q} Q that is fixed independently of the tested blocks, with Q ^ i , t \widehat{Q}_{i,t} Q i , t depending only on A i , ≤ t A_{i,\le t} A i , ≤ t and the stratum, while the actions follow Q Q Q (Assumption 1). Let Δ \Delta Δ be the total-variation distance between the laws, under Q Q Q and under Q ^ \widehat{Q} Q , of the complete command path from the first tested origin on, given the history before it and the frozen training and working-law information; under Assumption 1 these laws do not involve the potential outcomes. Then, under the null,
Pr ( E ≥ 1 / α ) ≤ α + E [ Δ ] . \Pr(E \ge1/\alpha) \le\alpha+ \mathbb{E}[\Delta]. Pr ( E ≥ 1/ α ) ≤ α + E [ Δ ] . If moreover 0 ≤ E ≤ M 0 \le E \le M 0 ≤ E ≤ M , then E [ E ] ≤ 1 + M E [ Δ ] \mathbb{E}[E] \le1 + M\mathbb{E}[\Delta] E [ E ] ≤ 1 + M E [ Δ ] .
The event bound needs neither a coupling construction nor a bound on E E E . The moment bound, with M = ( B + 1 ) m M=(B+1)^m M = ( B + 1 ) m for an MC product of m m m factors, is weaker whenever rejection is possible (M ≥ 1 / α M \ge1/\alpha M ≥ 1/ α ). By a telescoping argument over time steps, Δ \Delta Δ is at most the expected sum, along the true command path, of the one-step total-variation distances between the true and working kernels, and for product kernels each one-step distance is at most the sum of the unit distances. It is unknown for the working laws used here, and those laws were not fitted independently of the released commands, so the proposition does not license our real-data tests; we rely instead on calibration with the real command
records (§ 5 ). The closest published result, Theorem 4 of Berrett et al. [3 ] , bounds the type I error of the conditional randomization test, conditionally on Y Y Y and Z Z Z , by α \alpha α plus the total variation distance between the true and fitted conditional laws of X X X given Z Z Z for the whole sample, for independent and identically distributed data with the statistic and fitted law chosen independently of X X X and Y Y Y ; it covers both the randomization and the permutation forms, and it does not directly cover this longitudinal product. The resampler of § 7 that ignores part of the design state is far from the design law in total variation, and nothing in Proposition 5 protects it.
Why replacement diagnostics fail
Proposition 6 (Failure of replacement diagnostics; P4). Let the units be independent and identically distributed copies of the following experiment, observed at one fixed clock, with all its parameters fixed as the number of units grows. Each unit has a stratum θ i ∈ { 0 , 1 } \theta_i \in\{0,1\} θ i ∈ { 0 , 1 } known to the design, with Pr ( θ i = 1 ) ∈ ( 0 , 1 ) \Pr(\theta_i=1) \in(0,1) Pr ( θ i = 1 ) ∈ ( 0 , 1 ) . Let θ i \theta_i θ i enter only an outcome recorded before the predictor’s window, let the outcomes inside the window be constant, and let the focal outcome be θ i \theta_i θ i plus independent centred noise of finite variance, for every action assignment, so that H 0 G H_0^G H 0 G holds. Let the actions follow the Markov chain Pr ( A s = 1 ∣ A s − 1 = a ) = ρ a + ( 1 − ρ ) p θ i \Pr(A_s=1 \mid A_{s-1}=a) = \rho a + (1-\rho)p_{\theta_i} Pr ( A s = 1 ∣ A s − 1 = a ) = ρ a + ( 1 − ρ ) p θ i , started in its stationary law, with 0 ≤ ρ < 1 0 \le\rho< 1 0 ≤ ρ < 1 , p 0 ≠ p 1 p_0 \ne p_1 p 0 = p 1 and p 0 , p 1 ∈ ( 0 , 1 ) p_0,p_1 \in(0,1) p 0 , p 1 ∈ ( 0 , 1 ) . A predictor sees a window W W W of finitely many recent commands, the recent outcomes and the clock, together with the future block F F F of a fixed length h ≥ 1 h \ge1 h ≥ 1 , but neither θ i \theta_i θ i nor the earlier outcome; its ideal form is μ = E [ Y ∣ W , F ] \mu= \mathbb{E}[Y \mid W,F] μ = E [ Y ∣ W , F ] . Under squared-error loss, the population risk gaps of (i) retraining without actions (μ 0 = E [ Y ∣ W ] \mu_0 = \mathbb{E}[Y \mid W] μ 0 = E [ Y ∣ W ] , or E [ Y ] \mathbb{E}[Y] E [ Y ] if the window’s commands are removed as well), (ii) replacing F F F by zeros and (iii) replacing F F F by the block of an independent unit are strictly positive. For these ideal predictors, as the number of units grows the empirical gaps converge to these constants while unit (for (iii), pair) bootstrap intervals shrink, so each diagnostic reports an effect with probability tending to one. Under the same construction, Proposition 1 bounds by α \alpha α the size of the randomization test that resamples from Q ( ⋅ ∣ A i , t , θ i ) Q(\cdot\mid A_{i,t},\theta_i) Q ( ⋅ ∣ A i , t , θ i ) .
The window must not reveal the stratum: if the recent outcomes did (for instance, if they equalled θ i \theta_i θ i ), the future block would add nothing and all three gaps would vanish. The proposition concerns ideal predictors, as the frozen proof does; fitted predictors inherit it only under further convergence conditions, which we do not pursue. The construction is the stand-in effect of § 2.4 in its simplest form: the future block reveals θ i \theta_i θ i beyond what the window does, so a good predictor uses it, and a replacement that breaks the dependence costs predictive accuracy without any action effect. Design draws preserve the dependence, which is why the randomization test is unaffected. The proposition covers replacement by another unit’s block, not by a block taken from another time of the same unit. The development simulations confirm the construction numerically (§ 5 ).
Pooling and tempering
Tempering needs no separate validity result: the tempered score (6) is admissible (§ 3.4 ), so Proposition 1 applies to it. The bound (5) of § 3.3 describes how the pooled score loses power. The remaining question is when the registered Kish factor approximates the contrast factor.
Proposition 7 (Contrast tempering under a shared contrast; new). If the action contrast is common to the sites, D c , t b = D t b D^b_{c,t}=D^b_t D c , t b = D t b , the site variances are equal, v c = v > 0 v_c=v>0 v c = v > 0 , the same sites enter every candidate at each origin, and the numerator and denominator of (8) are positive, then the unclipped contrast factor equals 1 / ( 1 + X ) 1/(1+X) 1/ ( 1 + X ) with
X = ∑ t w t ∑ c ≠ c ′ r c , t r c ′ , t ∑ t w t ∑ c r c , t 2 , w t = ∑ b ( D t b ) 2 . X=\frac{\sum_t w_t\sum_{c\ne c'}r_{c,t}r_{c',t}}{\sum_t w_t\sum_c r_{c,t}^2},\qquad w_t=\sum_b(D^b_t)^2. X = ∑ t w t ∑ c r c , t 2 ∑ t w t ∑ c = c ′ r c , t r c ′ , t , w t = b ∑ ( D t b ) 2 . For a common site count n ≥ 2 n\ge2 n ≥ 2 this is λ c = 1 / [ 1 + ( n − 1 ) ρ ˉ D ] \lambda^c=1/[1+(n-1)\bar\rho_D] λ c = 1/ [ 1 + ( n − 1 ) ρ ˉ D ] with ρ ˉ D = X / ( n − 1 ) ∈ ( − 1 / ( n − 1 ) , 1 ] \bar\rho_D=X/(n-1)\in(-1/(n-1),1] ρ ˉ D = X / ( n − 1 ) ∈ ( − 1/ ( n − 1 ) , 1 ] .
Proof. Under the hypotheses, z c , t b = r c , t D t b / v z^b_{c,t}=r_{c,t}D^b_t/v z c , t b = r c , t D t b / v , so ∑ c ( z c , t b ) 2 = ( D t b / v ) 2 ∑ c r c , t 2 \sum_c(z^b_{c,t})^2=(D^b_t/v)^2\sum_c r_{c,t}^2 ∑ c ( z c , t b ) 2 = ( D t b / v ) 2 ∑ c r c , t 2 , and ( ∑ c z c , t b ) 2 = ( D t b / v ) 2 [ ∑ c r c , t 2 + ∑ c ≠ c ′ r c , t r c ′ , t ] (\sum_c z^b_{c,t})^2=(D^b_t/v)^2[\sum_c r_{c,t}^2+\sum_{c\ne c'}r_{c,t}r_{c',t}] ( ∑ c z c , t b ) 2 = ( D t b / v ) 2 [ ∑ c r c , t 2 + ∑ c = c ′ r c , t r c ′ , t ] . Summing over b b b gives the weight w t / v 2 w_t/v^2 w t / v 2 for each origin; dividing the two sums over t t t gives the result.
Remark 1. The identity λ c = 1 / ( 1 + X ) \lambda^{c}=1/(1+X) λ c = 1/ ( 1 + X ) holds for any site counts. Its reading as a Kish factor with intraclass correlation ρ ˉ D \bar{\rho}_{D} ρ ˉ D requires a common site count. When the count n t n_t n t varies across origins, X X X is a weighted mean of ( n t − 1 ) ρ t (n_t-1)\rho_t ( n t − 1 ) ρ t , so the single constant λ c \lambda^{c} λ c corresponds to a Kish factor at an effective site count and need not equal 1 / [ 1 + ( n t − 1 ) ρ ^ ] 1/[1+(n_t-1)\hat{\rho}] 1/ [ 1 + ( n t − 1 ) ρ ^ ] at every origin; with at least two distinct counts, equality at every origin requires ρ ^ = 0 \hat{\rho}=0 ρ ^ = 0 and λ c = 1 \lambda^{c}=1 λ c = 1 . For a common n n n , clipping λ c \lambda^{c} λ c to [ 1 / n , 1 ] [1/n,1] [ 1/ n , 1 ] is the same as clipping ρ ˉ D \bar{\rho}_{D} ρ ˉ D to [ 0 , 1 ] [0,1] [ 0 , 1 ] .
When the contrast is shared, contrast tempering is therefore Kish tempering with a particular estimate of the intraclass correlation: a pooled, contrast-weighted mean of uncentred within-origin residual cross-products, instead of the median of pairwise Pearson correlations used in (7) . The two estimate the same correlation only under conditions on the contrast weights as well as on the residuals: for instance, centred, equally variable residuals with an exchangeable and stable correlation, and contrast weights independent of the residuals. Without the condition on the weights, the contrast-weighted correlation can differ from the ordinary one even when the residuals are uncorrelated. The contrast hypothesis holds for additive models whose action branch sees only the unit’s shared inputs; the ERK reference models have this form, so agreement between the two factors on ERK is partly structural. Because ERK site counts vary across fields and origins, the meaningful comparison there is between ρ ^ \hat{\rho} ρ ^ and the pooled correlation implied by λ c \lambda^{c} λ c , not between λ i , t K \lambda^{K}_{i,t} λ i , t K and a single constant. The ERK training diagnostic (§ 8.5 ) finds the implied correlation lower than both the median and the mean of the pairwise correlations, so Kish tempering is somewhat conservative there; it is a description of how these correlation summaries differ, not an independent test of the Kish rule. When the contrast varies across sites, as when an action moves an object across pixels, the hypothesis fails and the two factors can differ in either direction (§ 3.4 , § 6 ).
When tempering helps, and by how much. Write Δ S i , t ( a ) = S i , t ( A i , t F ) − S i , t ( a ) \Delta S_{i,t}(a)=S_{i,t}(A^{F}_{i,t})-S_{i,t}(a) Δ S i , t ( a ) = S i , t ( A i , t F ) − S i , t ( a ) for the observed block’s advantage over a candidate a a a , so that the exact factor with T = λ S T=\lambda S T = λ S is log e i , t e x ( λ ) = − log E a ∼ Q i , t e − λ Δ S i , t ( a ) \log e^{\mathrm{ex}}_{i,t}(\lambda)=-\log\mathbb{E}_{a\sim Q_{i,t}}e^{-\lambda\Delta S_{i,t}(a)} log e i , t ex ( λ ) = − log E a ∼ Q i , t e − λ Δ S i , t ( a ) ; the MC factor obeys the same identity with Q i , t Q_{i,t} Q i , t replaced by the empirical law Q ^ i , t \hat{Q}_{i,t} Q ^ i , t of its observed-inclusive bag, which is not an assignment law (its validity still comes from exchangeability under Q i , t Q_{i,t} Q i , t ). Sums over origins below refer to one product component, one residue class of one unit, unless an averaging formula is given. A factor λ > 1 \lambda>1 λ > 1 appears only in the optimization and is compatible with validity; the tempering procedure of § 3.4 uses λ ≤ 1 \lambda\leq1 λ ≤ 1 .
Proposition 8 (A conditional growth bound; new). Fix the target, the score function and the observed block, and suppose E Q i , t ∣ Δ S i , t ∣ < ∞ \mathbb{E}_{Q_{i,t}}|\Delta S_{i,t}|<\infty E Q i , t ∣Δ S i , t ∣ < ∞ and E Q i , t e − η Δ S i , t < ∞ \mathbb{E}_{Q_{i,t}}e^{-\eta\Delta S_{i,t}}<\infty E Q i , t e − η Δ S i , t < ∞ for some η > 0 \eta>0 η > 0 . On the domain where this exponential moment is finite, λ ↦ log e i , t e x ( λ ) \lambda\mapsto\log e^{\mathrm{ex}}_{i,t}(\lambda) λ ↦ log e i , t ex ( λ ) is concave, with right derivative μ i , t = E Q i , t Δ S i , t \mu_{i,t}=\mathbb{E}_{Q_{i,t}}\Delta S_{i,t} μ i , t = E Q i , t Δ S i , t at zero. If a finite ν i , t ≥ 0 \nu_{i,t}\geq0 ν i , t ≥ 0 satisfies log E Q i , t e − λ ( Δ S i , t − μ i , t ) ≤ λ 2 ν i , t / 2 \log\mathbb{E}_{Q_{i,t}}e^{-\lambda(\Delta S_{i,t}-\mu_{i,t})}\leq\lambda^{2}\nu_{i,t}/2 log E Q i , t e − λ ( Δ S i , t − μ i , t ) ≤ λ 2 ν i , t /2 for every λ ∈ [ 0 , Λ ] \lambda\in[0,\Lambda] λ ∈ [ 0 , Λ ] , then on that interval
log e i , t e x ( λ ) ≥ λ μ i , t − 1 2 λ 2 ν i , t , log E ( λ ) ≥ f ( λ ) : = λ M − 1 2 λ 2 V , \log e^{\mathrm{ex}}_{i,t}(\lambda)\geq\lambda\mu_{i,t}-\frac{1}{2}\lambda^{2}\nu_{i,t},\qquad\log\mathbb{E}(\lambda)\geq f(\lambda):=\lambda\mathcal{M}-\frac{1}{2}\lambda^{2}\mathcal{V}, log e i , t ex ( λ ) ≥ λ μ i , t − 2 1 λ 2 ν i , t , log E ( λ ) ≥ f ( λ ) := λ M − 2 1 λ 2 V , *with M = ∑ t μ i , t \mathcal{M}=\sum_t\mu_{i,t} M = ∑ t μ i , t and V = ∑ t ν i , t \mathcal{V}=\sum_t\nu_{i,t} V = ∑ t ν i , t over one component. If M > 0 \mathcal{M}>0 M > 0 and V > 0 \mathcal{V}>0 V > 0 , the lower bound is maximized at λ q u a d = min ( M / V , 1 ) \lambda_{\mathrm{quad}}=\min(\mathcal{M}/\mathcal{V},1) λ quad = min ( M / V , 1 ) , with value M 2 / ( 2 V ) \mathcal{M}^{2}/(2\mathcal{V}) M 2 / ( 2 V ) when M ≤ V \mathcal{M}\leq\mathcal{V} M ≤ V . For bounded score differences, Hoeffding’s lemma gives ν i , t = ( e s s s u p Δ S i , t − e s s i n f Δ S i , t ) 2 / 4 \nu_{i,t}=(\mathop{\mathrm{ess\,sup}}\Delta S_{i,t}-\mathop{\mathrm{ess\,inf}}\Delta S_{i,t})^{2}/4 ν i , t = ( ess sup Δ S i , t − ess inf Δ S i , t ) 2 /4 and Λ = ∞ \Lambda=\infty Λ = ∞ . The same statements hold for the MC factor with Q ^ i , t \hat{Q}_{i,t} Q ^ i , t in place of Q i , t Q_{i,t} Q i , t . Under the null, for temperatures within a common moment domain and with the stated integrability, E [ μ i , t ∣ conditioning field ] = 0 \mathbb{E}[\mu_{i,t}\mid\text{conditioning field}]=0 E [ μ i , t ∣ conditioning field ] = 0 , so − 1 2 λ 2 E [ ν i , t ] ≤ E [ log e i , t ( λ ) ] ≤ 0 -\frac{1}{2}\lambda^{2}\mathbb{E}[\nu_{i,t}]\leq\mathbb{E}[\log e_{i,t}(\lambda)]\leq0 − 2 1 λ 2 E [ ν i , t ] ≤ E [ log e i , t ( λ )] ≤ 0 .
The null bound is an expected-loss bound. On a single path the first-order term λ μ i , t \lambda\mu_{i,t} λ μ i , t remains and may be negative, and a realized M > 0 \mathcal{M}>0 M > 0 is a pathwise, not an expected, growth condition. The proposition separates what the factor needs from what Kish and contrast tempering estimate: μ \mu μ and ν \nu ν describe score differences between candidate blocks, not correlations between sites.
Proposition 9 (Precision rescaling; new). Let the site scores be unfloored Gaussian log densities, ℓ i , c , t ( a ) = log ϕ ( Y i , c , t + h ; μ c , t ( a ) , ν c ) \ell_{i,c,t}(a)=\log\phi(Y_{i,c,t+h};\mu_{c,t}(a),\nu_c) ℓ i , c , t ( a ) = log ϕ ( Y i , c , t + h ; μ c , t ( a ) , ν c ) , with the scored sites, predicted means, data, validation residuals and contrasts held fixed, and multiply every precision by γ > 0 \gamma>0 γ > 0 (ν c ↦ ν c / γ \nu_c\mapsto\nu_c/\gamma ν c ↦ ν c / γ ). Then:
(a) S i , t ( γ ) ( a ) = γ S i , t ( 1 ) ( a ) + C i , t ( γ ) S_{i,t}^{(\gamma)}(a)=\gamma S_{i,t}^{(1)}(a)+C_{i,t}(\gamma) S i , t ( γ ) ( a ) = γ S i , t ( 1 ) ( a ) + C i , t ( γ ) with C i , t ( γ ) C_{i,t}(\gamma) C i , t ( γ ) independent of a a a , so log E ( γ ) ( λ ) = log E ( 1 ) ( γ λ ) \log E^{(\gamma)}(\lambda)=\log E^{(1)}(\gamma\lambda) log E ( γ ) ( λ ) = log E ( 1 ) ( γ λ ) : on a log \log log -λ \lambda λ axis the tempered curve shifts by − log γ -\log\gamma − log γ , its maximal value is unchanged, and λ q u a d \lambda_{\mathrm{quad}} λ quad of Proposition 8 becomes λ q u a d / γ \lambda_{\mathrm{quad}}/\gamma λ quad / γ under the scaled proxies γ μ \gamma\mu γ μ and γ 2 ν \gamma^2\nu γ 2 ν ;
(b) the contrast factor (8), where defined, does not depend on γ \gamma γ , nor on a rescaling D ↦ κ D D\mapsto\kappa D D ↦ κ D of the contrasts, and the same holds for any factor that uses standardized residuals only through correlation coefficients;
(c) hence, by (5) , for the exact factor at an origin with a competitor of design probability at least q q q and margin m > 0 m>0 m > 0 at γ = 1 \gamma=1 γ = 1 , log e i , t e x , ( γ ) ( λ c ) ≤ log ( 1 / q ) − γ λ c m \log e_{i,t}^{\mathrm{ex},(\gamma)}(\lambda^c)\le\log(1/q)-\gamma\lambda^c m log e i , t ex , ( γ ) ( λ c ) ≤ log ( 1/ q ) − γ λ c m ; for a fixed observed-inclusive MC bag in which the competitor occurs k k k times, log ( 1 / q ) \log(1/q) log ( 1/ q ) is replaced by log ( ( B + 1 ) / k ) \log((B+1)/k) log (( B + 1 ) / k ) .
The proposition concerns factors that are invariant under precision rescaling; it does not exclude an estimated factor that rescales correctly. With the per-site floor used in our applications the identity in (a) holds only where no floor is active at either scale; Appendix § 16 .§ 9 gives the error bound for floored scores and shows that flooring can also stop the collapse in (c).
Proposition 10 (Tempering mixture; new). Let λ g = λ 0 2 − g \lambda_g=\lambda_0 2^{-g} λ g = λ 0 2 − g , g = 0 , … , G − 1 g=0,\ldots,G-1 g = 0 , … , G − 1 , and for one product component let E M = G − 1 ∑ g E ( λ g ) E^M=G^{-1}\sum_g E(\lambda_g) E M = G − 1 ∑ g E ( λ g ) . Then:
(a) under the conditions of Proposition 1 , E M E^M E M is an e-value;
(b) pathwise, log E M ≥ max g log E ( λ g ) − log G \log E^M\ge\max_g\log E(\lambda_g)-\log G log E M ≥ max g log E ( λ g ) − log G ;
(c) if the bound of Proposition 8 holds at the grid points with M > 0 M>0 M > 0 and V > 0 V>0 V > 0 , then log E M ≥ f ( λ q u a d ) − log G \log E^M\ge f(\lambda_{\mathrm{quad}})-\log G log E M ≥ f ( λ quad ) − log G when λ q u a d \lambda_{\mathrm{quad}} λ quad is a grid point, log E M ≥ 8 9 f ( λ q u a d ) − log G \log E^M\ge\frac{8}{9}f(\lambda_{\mathrm{quad}})-\log G log E M ≥ 9 8 f ( λ quad ) − log G when it lies strictly between two grid points, and log E M ≥ λ 0 M / 2 − log G \log E^M\ge\lambda_0 M/2-\log G log E M ≥ λ 0 M /2 − log G when M / V > λ 0 M/V>\lambda_0 M / V > λ 0 ;
(d) with R R R residue classes averaged as in § 3.2 , the mixture of the class average is the average of the class mixtures, and if log E r M ≥ b r − log G \log E_r^M\ge b_r-\log G log E r M ≥ b r − log G for each class, then log E M ≥ log ∑ r e b r − log R − log G \log E^M\ge\log\sum_r e^{b_r}-\log R-\log G log E M ≥ log ∑ r e b r − log R − log G .
In the setting of Proposition 9 , the guarantee in (c) holds uniformly over the finite range of precision multipliers γ \gamma γ that keeps λ q u a d / γ \lambda_{\mathrm{quad}}/\gamma λ quad / γ within the grid and the comparison points within the moment domain; outside that range no such claim is made, and the bound need not be positive. Pathwise, log E M ≥ log E ( λ G − 1 ) − log G \log E^M\ge\log E(\lambda_{G-1})-\log G log E M ≥ log E ( λ G − 1 ) − log G . For one product component under the valid null, if λ G − 1 \lambda_{G-1} λ G − 1 lies in a common deterministic moment domain of Proposition 8 and the required integrability and E [ V ] < ∞ \mathbb{E}[V]<\infty E [ V ] < ∞ hold, the expected log loss is at most log G + 1 2 λ G − 1 2 E [ V ] \log G+\frac{1}{2}\lambda_{G-1}^2\mathbb{E}[V] log G + 2 1 λ G − 1 2 E [ V ] ; for a residue-class average, apply Jensen with the class-averaged proxy, under these conditions for every class.
Proposition 13 (Over-betting; new). Fix one product component of a unit, with pooled scores S t ( b ) S_t(b) S t ( b ) , b = 0 , … , B b=0,\ldots,B b = 0 , … , B , at its origins t t t , and let
L ( λ ) = ∑ t { λ S t ( 0 ) − log 1 B + 1 ∑ b = 0 B e λ S t ( b ) } , λ ≥ 0 , L(\lambda)=\sum_t\left\{\lambda S_t(0)-\log\frac{1}{B+1}\sum_{b=0}^{B}e^{\lambda S_t(b)}\right\},\qquad\lambda\ge0, L ( λ ) = t ∑ { λ S t ( 0 ) − log B + 1 1 b = 0 ∑ B e λ S t ( b ) } , λ ≥ 0 , so that E ( λ ) = e L ( λ ) E(\lambda)=e^{L(\lambda)} E ( λ ) = e L ( λ ) . Then, pathwise:
(a) L L L is concave and L ( 0 ) = 0 L(0)=0 L ( 0 ) = 0 ;
(b) L ( λ ) ≥ ( λ / λ ′ ) L ( λ ′ ) L(\lambda)\ge(\lambda/\lambda')L(\lambda') L ( λ ) ≥ ( λ / λ ′ ) L ( λ ′ ) for 0 ≤ λ ≤ λ ′ 0\le\lambda\le\lambda' 0 ≤ λ ≤ λ ′ ;
(c) if L ( λ ) < 0 L(\lambda)<0 L ( λ ) < 0 for some λ \lambda λ , then L ( λ ′ ′ ) ≤ ( λ ′ ′ / λ ) L ( λ ) < 0 L(\lambda'')\le(\lambda''/\lambda)L(\lambda)<0 L ( λ ′′ ) ≤ ( λ ′′ / λ ) L ( λ ) < 0 for every λ ′ ′ ≥ λ \lambda''\ge\lambda λ ′′ ≥ λ ; and if L ( λ ′ ) ≥ 0 L(\lambda')\ge0 L ( λ ′ ) ≥ 0 , then L ( λ ) ≥ 0 L(\lambda)\ge0 L ( λ ) ≥ 0 for every λ ≤ λ ′ \lambda\le\lambda' λ ≤ λ ′ ;
(d) for the mixture of Proposition 10 with λ g = λ 0 2 − g \lambda_g=\lambda_0 2^{-g} λ g = λ 0 2 − g , and any λ ∗ ∈ [ λ G − 1 , λ 0 ] \lambda^*\in[\lambda_{G-1},\lambda_0] λ ∗ ∈ [ λ G − 1 , λ 0 ] with L ( λ ∗ ) ≥ 0 L(\lambda^*)\ge0 L ( λ ∗ ) ≥ 0 : log E M ≥ 1 2 L ( λ ∗ ) − log G \log E^M\ge\frac{1}{2}L(\lambda^*)-\log G log E M ≥ 2 1 L ( λ ∗ ) − log G .
E M = ∏ t ∑ g w g , t e t ( λ g ) , where e t ( λ ) is the factor of E ( λ ) at origin t and w g , t ∝ ∏ s < t e s ( λ g ) , with w g , 1 = 1 / G . E^{\mathrm{M}} = \prod_t \sum_g w_{g,t} e_t(\lambda_g), \text{ where } e_t(\lambda) \text{ is the factor of } E(\lambda) \text{ at origin } t \text{ and } w_{g,t} \propto\prod_{s<t} e_s(\lambda_g),\\
\text{with } w_{g,1}=1/G. E M = t ∏ g ∑ w g , t e t ( λ g ) , where e t ( λ ) is the factor of E ( λ ) at origin t and w g , t ∝ s < t ∏ e s ( λ g ) , with w g , 1 = 1/ G .
Part (c) says that the factors at which a tempered statistic loses form an upper interval: a negative log e-value means that the factor exceeds the break-even factor, and beyond it the loss grows at least in proportion to the factor. A factor below a profitable one never loses, and by (b) it keeps at least the share λ / λ ′ \lambda/\lambda' λ / λ ′ of the log e-value. Part (d) needs no moment bound, unlike Proposition 10(c); it gives nothing when no factor in the range is profitable. Part (e) is the standard identity between a mixture of products and a product of wealth-weighted averages; we claim no priority for it. The proposition is stated for one product component; for a residue-class average, (a) can fail, and Proposition 10(d) applies instead.
Proposition 11 (Rank calibrators; new). Let g g g be positive and nonincreasing on the N c a n d = B + 1 N_{\mathrm{cand}}=B+1 N cand = B + 1 positions, with mean one; put c g = N c a n d − 1 ∑ k log g ( k ) c_g=N_{\mathrm{cand}}^{-1}\sum_k \log g(k) c g = N cand − 1 ∑ k log g ( k ) and r g = log g ( 1 ) − log g ( N c a n d ) r_g=\log g(1)-\log g(N_{\mathrm{cand}}) r g = log g ( 1 ) − log g ( N cand ) . For e i , t R = g ( R i , t ) e^R_{i,t}=g(R_{i,t}) e i , t R = g ( R i , t ) , with R i , t R_{i,t} R i , t the unique rank of the observed score or a rank drawn by uniform random tie-breaking, let τ i , t \tau_{i,t} τ i , t be the total-variation distance of its conditional law from the uniform one. Then E [ log e i , t R ∣ F t ] ≤ c g + τ i , t r g \mathbb{E}[\log e^R_{i,t}\mid\mathcal{F}_t]\leq c_g+\tau_{i,t}r_g E [ log e i , t R ∣ F t ] ≤ c g + τ i , t r g , and over T T T factors E [ log E R ] ≤ T c g + r g ∑ t E [ τ i , t ] \mathbb{E}[\log E^R]\leq Tc_g+r_g\sum_t\mathbb{E}[\tau_{i,t}] E [ log E R ] ≤ T c g + r g ∑ t E [ τ i , t ] . If g g g is not constant, c g < 0 c_g<0 c g < 0 and the bound is negative whenever the average E [ τ i , t ] \mathbb{E}[\tau_{i,t}] E [ τ i , t ] is below − c g / r g -c_g/r_g − c g / r g .
For the normalized power family of §3.5 with N c a n d = 128 N_{\mathrm{cand}}=128 N cand = 128 , c g = − 0.414 c_g=-0.414 c g = − 0.414 , r g = 3.852 r_g=3.852 r g = 3.852 and the threshold is 0.1075 for κ = 1 / ln 128 \kappa=1/\ln128 κ = 1/ ln 128 (with log g ( 1 ) = 2.665 \log g(1)=2.665 log g ( 1 ) = 2.665 ), and − 0.142 -0.142 − 0.142 , 2.426 and 0.0584 for κ = 1 / 2 \kappa=1/2 κ = 1/2 . The statistic we use averages g g g over tied positions. That keeps the factor’s mean at one, but its null baseline is then a bag-dependent constant between c g c_g c g and zero, equal to zero when all candidates tie (Appendix § 16 .§ 11 ); the untied price is therefore not the price of the implemented statistic when ties are frequent, as in the Atari games of §9.
Designs with delayed feedback
Proposition 12 (Delayed feedback; new). (a) Under Assumption 2 and hypotheses (i)–(iii) of Proposition 1, with (iii) drawing from Q i , t Q_{i,t} Q i , t , the conclusions (a)–(c) of Proposition 1 hold under H 0 G H^G_0 H 0 G , and under H 0 G H^G_0 H 0 G when the product form holds for the tested set conditionally on the outside blocks.
(b) A sufficient condition for Assumption 2, before any eligibility conditioning, is that actions are generated as A i , s = π i , s ( H ⋅ , s − L , A i , < s , U i , s ) A_{i,s}=\pi_{i,s}(H_{\cdot,s-L},A_{i,<s},U_{i,s}) A i , s = π i , s ( H ⋅ , s − L , A i , < s , U i , s ) with L ≥ h L\geq h L ≥ h , where the policies and their parameters are fixed given the permitted information, H ⋅ , s − L H_{\cdot,s-L} H ⋅ , s − L is the history of all units up to time s − L s-L s − L that the design records (outcomes, logged states and actions), no policy reads another unit’s actions inside the block, and the innovations U i , s U_{i,s} U i , s are mutually independent and independent of F t \mathcal{F}_t F t and of all potential outcomes.
Restricting only the tested units’ policies is not enough under spillovers: an outside unit that reacts to a tested unit’s action inside the block can carry that action into the tested target. A known non-product joint design could support a single jointly normalized factor but not a product of unit-wise factors. Eligibility conditioning must be shown separately to preserve the asserted law. The condition L ≥ h L\geq h L ≥ h is sufficient, not necessary. In the trap experiment of §7, each episode is one unit, and its frozen episode-local policy draws each action from a law that depends only on that episode’s own state at least h = 4 h=4 h = 4 steps before the action, with fresh randomization; treating the simulator’s pseudorandom draws as independent innovations, the design-law analysis therefore falls under (b), with no eligibility conditioning and a target mask that does not depend on the actions under the null. This does not cover the deliberately marginal resampler of the negative control (§ 7.2 ).
Synthetic validation
Before any real outcome was reopened, the registered pooled statistic passed through four layers of synthetic checks, each fixed before it was run: development simulations with known truth (M2a), calibration on the real command records with synthetic outcomes (M2b), a power pilot and an end-to-end rehearsal (M2c, M2e), and a sealed challenge built in a separate session after the method was frozen (M3). We report what each layer tested and, equally, what it could not have detected. A fifth check, a single-field rehearsal for M5, is described at the end. Full tables are in Appendix § 18 .
Development simulations
The development grid crossed five design laws (renewal, bounded walk, Markov, clock-locked and shuffle) with linear, saturating and delayed outcome dynamics, for each of the four reference models, with 256 replicates per cell. It contained four null families—the stand-in construction of Proposition 6 , sharp-null mutual spillover (an own-channel null), shared noise under the global null, and zero-entropy designs—and two alternatives built from the masked effects of Proposition 3 .
Validity. Across all designs, dynamics and models, the exact and MC e-value tests never rejected in the stand-in family (0/256 for the exact form and 0/256 for the MC form, in every cell), and rejected at most 5/256 under shared noise and 3/256 under sharp-null spillover. The batch randomization p-value at a single origin is a p-value test at level 0.05 with B = 127 B = 127 B = 127 , so its attainable nominal rate is 6/128, or 12 of 256 replicates. Across the 240 null cells of the grid (60 in each null family) it rejected in at most 20 of 256 replicates, a count reached in 1 cell, under shared noise; 26 cells exceeded 12. With this many cells, some counts above 12 are expected from binomial variation alone. In the largest stand-in setting, with 512 units, the four models rejected at rates from 4.3% to 6.6%, none significantly above the nominal rate (smallest one-sided binomial tail probability 0.096). Validity means a rejection rate at most α \alpha α , not zero rejections; the e-value tests reject far less often than α \alpha α because Markov’s inequality is conservative.
The stand-in construction. In the stand-in family the replacement diagnostics reported an action effect in up to 246/256 (zero block), 249/256 (another unit’s block) and 236/256 (retraining without actions) replicates, although no action had any effect. In the largest setting of the construction every model’s zero-block and swap diagnostics reported an effect in every replicate (minima across models 256/256 for the zero block and 256/256 for the swap), while the e-value tests never rejected (0/256; Figure 4 B). The diagnostics fail just as badly in zero-entropy designs, where the actions follow a deterministic schedule and the randomization test returns e = 1 e = 1 e = 1 by construction: there they reported effects in up to 254/256 replicates. A randomization test that resamples from the marginal frequency of each action instead of the design law (B-marg) reported effects in up to 256/256 stand-in replicates, the same failure that § 7 reproduces in the pixel world.
Figure 4. The stand-in trap. (A) Pixel world whose logging policy depends on the blob’s position six steps earlier, which the models do not see; no action has any effect. For each cell (shared noise and number of training episodes; 64 test episodes), the highest effect-report rate among the three replacement diagnostics for each learned model, and the contrast-tempered randomization test with the design-law resampler, which conditions on the full logged state, and with a resampler that draws actions from their marginal frequencies (post hoc, § 7.2 ). (B) Development stand-in construction with 512 units (256 replicates), for each reference model: the three replacement diagnostics, the exact and Monte Carlo e e e -value tests with the design law, and a randomization test with a marginal resampler. Error bars: 95% Clopper–Pearson intervals. Dotted line: α = 0.05 \alpha= 0.05 α = 0.05 .
Power where the design allows little. The masked alternatives were built to be hard: their effects are invisible to unit-wise marginal comparisons. The e-value tests rejected in at most 9/256 replicates for masked XOR effects and 97/256 for masked shared effects. We report this as a limitation, not a failure of validity; the sealed challenge reproduces it (§ 5.4 ).
Calibration with the real command records
Proposition 5 does not license tests under working laws whose distance from the true law is unknown, so the working laws were calibrated on the released command records themselves. Each replicate drew a fresh random subset of real command paths and fresh synthetic outcomes from a model satisfying the sharp null, so that the estimated size averages over whatever law generated the released commands while the test resamples from the working law. The pre-committed criterion required the one-sided 95% Clopper–Pearson upper bound on size to be at most 0.06 in every primary cell. All 64 of 64 primary cells passed; the largest point false-positive rate was 0.40% and the largest upper bound 1.3%. One generator placed the stand-in construction on the real command records, with a slowly decaying response to past commands and strongly autocorrelated noise; there the test rejected in at most 0.40% of replicates, whereas retraining without actions and zero-block replacement reported effects in up to 92.8% and 91% of them.
Power pilot and end-to-end rehearsal
On the full M4 grid of origins, all 16 of 16 null cells passed the registered validity gate (largest upper bound 3.1%), and the four-model average had minimum power across cells of 100% at 10% signal and 100% at 30%. At 2% signal, power for ERK at thirty minutes ranged from 4.5% to 71%, which the specification recorded as a limitation. An end-to-end rehearsal of the frozen pipeline on a simple synthetic generator rejected strongly (log E = 136 \log E = 136 log E = 136 at ten minutes).
Neither check could have revealed pooling amplification. The pilot gave each synthetic ERK field a single outcome trajectory, not hundreds of cells sharing one light program, and the rehearsal’s generator was simple enough for the fitted models to be nearly correctly specified. The pooled statistic therefore entered M4 without ever having been scored on many sites that share an action sequence under a misspecified model—precisely the setting in which it fails (§ 6 ). We regard this as the main gap in our pre-registration validation.
Sealed challenge
The sealed challenge was generated in a separate session after the method was frozen and registered, from generators the method author never saw, and scored with the frozen implementation; the analysis functions used were verified identical to those of M4. It comprised 11 families: new designs and dynamics, an omitted-history stand-in construction, spillover on a given graph, shared autoregressive outcome noise, masked XOR effects, a known approximate-law error and zero-entropy designs. Its frozen criteria were a size bound, power where the design permits it, and exact neutrality under zero entropy.
The pooled statistic registered for M4 rejected in 0/3400 null replicates (pooled upper bound 0.088%; largest per-family upper bound 1.5%). Zero-entropy cells returned e e e and E E E equal to one exactly. Of the 16 alternative cells whose exact design ceiling was at least 0.95, 15 reached the required power of 0.80. The exception was the masked XOR family, with 24/200 rejections (12%) against a ceiling of 100%. Its effects are sharp but masked marginally, as in the masked alternatives of the development grid (§ 5.1 ), which were also the hard case there. As in M2, the replacement diagnostics reported effects at high rates in null families (Figure 9 ). Two disclosures apply. The challenge’s designer had seen the source of the reference models and one correction to the challenge design. And none of its families was built around units with many sites sharing an action sequence (Appendix § 18 lists them), so, like the pilot, it was not designed to reveal pooling amplification.
Figure 9. Sealed challenge by family. For each of the eleven families, the null rejection rate of the pooled statistic registered for M4 with its one-sided 95% Clopper–Pearson upper bound, and the range of effect-report rates of the replacement diagnostics across the family’s null cells and models. Family names are abbreviated from the challenge report (Table 13 ).
A single-field rehearsal for M5
Before the M5 plan was finalized, the Kish-tempered statistic was rehearsed on synthetic outcomes attached to one real training field’s command path, with the linear-Gaussian model only and the rejection threshold E ≥ 40 E \ge40 E ≥ 40 of the draft plan, which then had two primary horizons. With the simple synthetic response the statistic rejected in every replicate at both horizons (for example 32/32 at ten minutes with field-wide shared noise). With the harder generator and field-wide shared noise it rejected in 27/32 replicates at ten minutes but only 2/32 at thirty minutes. The registered plan therefore made ten minutes the only primary horizon, with threshold E ≥ 20 E \ge20 E ≥ 20 ; because E ≥ 40 E \ge40 E ≥ 40 implies E ≥ 20 E \ge20 E ≥ 20 , these counts are lower bounds on the rehearsal’s rejections at the registered threshold.
Pooling amplification and its repair
The registered M4 analysis failed on its own positive controls (§ 8.2 ). This section reports how we diagnosed the failure and selected a repair, on synthetic outcomes attached to the ERK command records, and a pre-declared demonstration of the same failure in a pixel world model. Everything here is exploratory. The diagnosis and the selection used no test outcome, and the repaired statistic was then confirmed on a field that no one had opened (§ 8.4 ). A later exploratory study (§ 6.4 ) compared the repairs with statistics that estimate no factor, and its outcome fixed the primary statistic of a registered test on a published world model (§ 9 ).
Diagnosis on ERK-like synthetic data
We attached synthetic outcomes to the command records of the M4 test fields, with five hundred cells per field, and fitted the linear-Gaussian reference model exactly as in M4. Three findings located the failure.
First, the pooled score collapsed without any shared noise. With independent cell noise, its median log E E E at ten minutes was −317 at 10% signal and −1680 at 30%: a stronger effect made the evidence more negative, as (5) predicts when a misspecified model’s error along the contrast grows with the effect it misrepresents. These medians come from the diagnostic runs, with thirty-two replicates per cell; the selection grid of § 6.2 , with sixteen replicates per cell, gives similar values (Table 14 ).
Table 14. FIX synthetic selection grid. Sources: FIX_F1_RESULTS.json @ 5a691755b0e8; FIX_SELECTION.json @ 5a691755b0e8; FIX_SETTINGS.json @ 5a691755b0e8.
Second, the oracle predictor rejected in every replicate (16/16 at ten minutes and 16/16 at thirty), and the pipeline remained valid under the null (0/64 rejections with mean e e e 1.00027 on ERK commands, 0/64 with mean e e e 0.99998 on CcaS commands). The oracle’s log E E E of 135 at ten minutes is a ceiling set by ties among resampled blocks—the generator responds only to the last action of the block—and not the design bound of Proposition 4 .
Third, changing the pipeline one component at a time showed that a single step caused the collapse: summing the cells’ log scores and then normalizing once over the field’s design (rejections 8/8 before that step, 0/8 after it; Figure 2 B). Adding field-wide shared noise to the simple rehearsal generator of § 5.3 also collapsed the pooled score (median log E E E −10,494 at ten minutes in the selection grid of § 6.2 ): noise shared by all cells moves them in the direction of the contrast, so it amplifies in the same way as model error.
Figure 2. Diagnosis and repair on ERK-like synthetic data. (A) Median log E E E of the pooled score and of the three repair candidates in the twelve alternative cells of the selection grid; the shaded cell is the one the selection script designated hardest (§ 6.2 ). The Kish-tempered and field-mean curves nearly coincide. The pooled score fails everywhere except without shared noise under the simple generator. (B) Rejection counts before and after the field-pooling step, for the oracle predictor in four cells, and for the pipeline under the null on ERK and CcaS commands. Error bars: 95% Clopper–Pearson intervals. Symmetric-log axis in A.
Selecting a repair
Before evaluating any candidate, we committed a protocol listing three repairs and a selection rule. The candidates were the Kish-tempered score (§ 3.4 ), a field-mean score that scores the field-average outcome under a Gaussian whose variance accounts for the within-field correlation, and a site-average that averages per-cell e-values; each kept the design law, the Monte Carlo bags and the aggregation of M4. The grid crossed two generators (the first at two signal levels), two levels of shared noise and both horizons, giving twelve alternative cells with sixteen replicates each, and two null cells with thirty-two replicates each. The rule was: (1) exclude a candidate with two or more null rejections in any null cell; (2) choose the highest minimum rejection rate across alternative cells; (3) break ties by the highest median log E E E in the hardest cell, the one with the lowest pooled rejection rate; (4) break remaining ties by simplicity, Kish first.
All three candidates rejected in every replicate of every alternative cell (minimum rates 100% for Kish tempering, 100% for the field mean and 100% for the site average) and in no null replicate (Kish tempering: 0/32 and 0/32 in the two null cells; the other candidates likewise, Appendix § 18 ). Step (2) therefore tied, and the selection fell to step (3). Because every cell tied on the pooled rejection rate, the “hardest cell” was not defined by the rule; the selection script took the first cell in grid order. That script was first committed together with the selection result, after the evaluation, so we cannot show that this detail was fixed in advance. In that cell the median log E E E was 131.53 for Kish tempering, 131.39 for the field mean and 83.60 for the site average. Across the twelve possible designations of the hardest cell, the rule would have selected Kish tempering in 9 of them, the field mean in 2 and the site average in 1; the Kish-minus-field-mean difference in median log E E E ranged from −0.04 to 0.96. On this evidence Kish tempering and the field-mean score perform alike, both well ahead of the site average (Figure 2 A). Kish tempering was registered for the confirmatory test; its registration describes the selection as following the pre-committed rule, and we correct that description here (Appendix § 18 ).
A pixel world model
To show the failure where machine-learning practice meets it, we ran a demonstration under a protocol frozen before any learned model was trained, with dated predictions committed before the main run. A Gaussian blob moves on an R × R R \times R R × R frame; each action pushes it in one of four directions or leaves it, and a delayed restoring force adds memory. Frames carry independent pixel noise and one of three kinds of frame-level shared noise: none, an additive offset, or a multiplicative gain on the blob. Actions are drawn uniformly at random, so the design law is known exactly. Two learned models predict the frame four steps ahead from the last two frames and the action window, a convolutional network with action conditioning and a per-pixel ridge regression, and are compared with oracles that know the true dynamics. Each episode is one unit and its n = R 2 n = R^2 n = R 2 pixels are its sites, from 64 to 4096; the effect size was calibrated on the oracle alone. Every state uses Monte Carlo e-values with B = 127 B = 127 B = 127 and the rejection rule E ≥ 20 E \ge20 E ≥ 20 , and reports the average of the two models.
The pooled score collapses as pixels are added, and the cause is misspecification. Figure 1 shows the main grid. With no shared noise and the larger effect, the median log E E E of the pooled score fell from −17.3 at 64 pixels to −6832 at 4096, while the oracle’s rose from 79.0 to 157. The pattern was the same under offset and gain noise (pooled score at 4096 pixels −5996 and −6567; oracle 155 and 157), and from 256 pixels on the pooled score never rejected under any noise (0%). Because the oracle does not collapse under either form of shared noise, the collapse of the fitted models is due to their misspecification. Our protocol had predicted that the gain noise would make even the oracle collapse; it did not, because a multiplicative gain rescales the oracle’s evidence without changing its sign.
Kish tempering does not repair it here. The pixel residuals of a misplaced blob take both signs, so their median pairwise correlation is nearly zero without shared noise or with gain noise (at most 0.000629 across cells), the Kish factor is close to one, and Kish tempering behaves like the pooled score: it rejected in 6.2% of episodes at 64 pixels and in none from 256 on. Under offset noise the residual correlation rose to between 0.159 and 0.182, and Kish tempering did reject, but for the wrong reason: the offset raises the residual correlation while being orthogonal to the action contrast, and at 4096 pixels the resulting over-tempering cut its rejection rate to 43.8%. Kish tempering, the statistic registered for M5, therefore does not repair pooling amplification in the pixel world. The fractional-Kelly reference arm of § 3.6 , which bounds each factor below but keeps the pooled bet, rejected in 0% to 75% of episodes at the larger effect (Appendix § 21 ).
Contrast tempering restores power, not information. Contrast tempering, declared in the protocol as an exploratory generalization, rejected in 81.2% to 100% of episodes across all noise conditions and frame sizes at the larger effect, against 100% for the oracle with the exact joint likelihood. Its median log E E E , however, stayed between 7.06 and 10.3 and did not grow with the number of pixels, far below the oracle: it removes the amplification but does not convert the additional pixels into additional evidence. At the smaller effect every learned statistic was weak (contrast tempering at most 25%). § 6.4 shows that statistics estimating no factor matched these rejection rates, and § 9 that contrast tempering did not carry over to a published world model.
The mechanism of § 3.4 was visible directly (Figure 3 ). For the learned models in the main grid, the contrast design effect exceeded the Kish design effect in every row without shared noise or with gain noise (48/48) and fell below it in every offset row (24/24), as predicted before the main run. For the oracles, whose residuals are independent pixel noise, both design effects are close to one and their ordering carries no information (Appendix § 21 ).
Figure 3. Two design effects. The Kish design effect from the median pairwise residual correlation (x x x ) against the design effect along the action contrast (y y y ), for every learned-model cell of the pixel main grid (colour: shared noise; shape: model), every oracle cell (grey crosses), and the pooled ERK training diagnostic (stars; four models and two horizons, at the diagnostic’s cell counts). Without shared noise or with gain noise, residual correlations are near zero and the Kish factor misses the correlation along the contrast (points above the diagonal); a frame-wide offset correlates the residuals but is orthogonal to the contrast (points below). The ERK points lie slightly below the diagonal, so Kish tempering is somewhat conservative there. Oracle residuals are independent pixel noise, so their contrast design effect is close to one.
Scope. Every statistic was valid in every null cell. The demonstration uses one simulator family, two model classes and one training run per cell; compute limits reduced the episode length and the number of test episodes, so the intervals for alternative cells are wide (Appendix § 21 ). Contrast tempering has only this simulation evidence; on a published world model it did not repair pooling amplification (§ 9 ), and we do not recommend it.
Statistics that estimate no factor
Both repairs above estimate a tempering factor from validation residuals. We asked whether statistics that need no such estimate do as well: the tempering mixture M over a fixed grid of thirteen factors, validation-tuned tempering V, and the rank e e e -values R and R1 / 2 _{1/2} 1/2 (§ 3.5 ). Before any of them was computed we committed a protocol with dated predictions and mechanical decision rules. We then recomputed every existing statistic from the saved models, episodes and seeds of the pixel demonstration and of the selection grid of § 6.2 , on a different machine, and computed the new statistics on the same draws. The recomputation reproduced every rejection decision of the original runs. Its reproduction criterion, as first written, used a relative tolerance that is ill-posed for values near zero; we amended it before any new statistic was computed (Appendix § 22 ). This study is exploratory.
Table 2 gives the results. At the larger effect the mixture and validation tuning matched contrast tempering, and the mixture also rejected where Kish tempering had failed. The pre-registered rule therefore returned outcome 1: a statistic that estimates no factor matched contrast tempering in the pixel world and Kish tempering on ERK-like data. Validation tuning matched contrast tempering, so the contrast formula added nothing over a temperature chosen on validation data. At the smaller effect the mixture was weaker than contrast tempering and validation tuning, though not than Kish tempering. Averaging can reduce log evidence relative to the best grid member by up to log 13 ≈ \approx ≈ 2.56, which can matter near a rejection threshold; the bound alone does not determine these observed differences. The rank e-value R failed, against our dated prediction. On the ERK-like grid its largest count in any cell was 0 of sixteen replicates, consistent with the expected price per factor that a rank calibrator pays when evidence is spread over thousands of weakly informative factors (Proposition 11 , stated for untied scores); R1 / 2 _{1/2} 1/2 , a secondary arm, did well in the pixel world but likewise failed on
Table 2. Exploratory baseline study. Rejections in the pixel world (twelve cells of sixteen episodes at each effect size, two-model average, E ≥ 20 E \ge20 E ≥ 20 ) and the number of ERK-like alternative cells (of twelve) in which every one of sixteen replicates was rejected (E ≥ 100 E \ge100 E ≥ 100 ). A dash: not defined in that setting.
ERK-like data (largest count 2 of sixteen). No new statistic rejected more than 1 of the episodes of any null cell. On the strength of this study the mixture became the primary statistic of the registered test in § 9 .
A regime map (exploratory)
Section 6.4 compared the statistics in two settings. To see where each one works, we ran a simulator in which the relevant quantities can be set directly, under a protocol committed before any statistic was computed (Appendix § 25 ). A unit has 200 origins, and recorded actions are drawn uniformly from four. The executed action equals the recorded one with probability p p p and is otherwise an independent draw. Each of n n n sites responds to the executed action through a fixed pattern, plus shared noise of variance ρ \rho ρ that moves the sites the way an action would, plus independent noise. The model predicts the response to the recorded action, with an error along the action contrast that is a fraction δ \delta δ of the effect and is shared by all sites; it assumes that the recorded action was executed. The grid crossed n ∈ { 16 , 256 , 4096 } n \in\{16, 256, 4096\} n ∈ { 16 , 256 , 4096 } , ρ ∈ { 0 , 0.05 } \rho\in\{0, 0.05\} ρ ∈ { 0 , 0.05 } , δ ∈ { 0 , 0.25 , 0.5 } \delta\in\{0, 0.25, 0.5\} δ ∈ { 0 , 0.25 , 0.5 } , p ∈ { 1 , 0.5 , 0.2 } p \in\{1, 0.5, 0.2\} p ∈ { 1 , 0.5 , 0.2 } and three effect sizes calibrated on the oracle alone, with 32 test units in each of the 162 alternative cells and 64 in each of the 18 null cells.
The pooled score behaved as § 3.3 predicts. With δ = 0 \delta= 0 δ = 0 and p ≤ 0.5 p \le0.5 p ≤ 0.5 it collapsed at 4096 sites in all 12 such cells: when the recorded action is not always executed, a model that assumes it was is misspecified along the contrast even if its response pattern is exact. Kish tempering did no better than the pooled score, with mean rejection rates of 0.222 and 0.221 over the alternative cells; contrast tempering reached 0.496. Validation tuning had the highest mean rate, 0.635, followed by the weighted mixtures (0.625, 0.609 and 0.588 for w = 3 / 4 , 1 / 2 , 1 / 4 w = 3/4, 1/2, 1/4 w = 3/4 , 1/2 , 1/4 ) and the uniform mixture (0.560). In the 98 cells of the simulator and of the saved pixel runs in which some candidate rejected in at least 80
The protocol’s mechanical rule chose the primary statistic of the next registered test among M, W(1/4), W(1/2), W(3/4) and V. It selected V, which tied with W(3/4) within the rule’s tolerance and won on mean rate. The simulator could not test V on ERK-like data, because the saved ERK-like runs lack the validation residuals V needs. Section 8.6 tests V on the real ERK test fields, where it collapsed. For that reason the next registered test used W(3/4) instead of V, a departure from the rule that was recorded in the registered protocol before any of that test’s data existed (§ 10.1 ). A later exploratory analysis showed that the collapse on the ERK fields was specific to tuning one factor for fields with different numbers of cells: with a site-count-adaptive factor, validation tuning no longer collapsed, though it still lost most of the evidence for some models (§ 8.7 ). The choice of W(3/4) was registered before that analysis.
The stand-in trap
Section 2.4 and Proposition 6 describe how a predictor can make future actions look effective when they are not. The development simulations and the calibration on real command records showed the trap on biological designs (§ 5 ). Here we build it into the pixel world, where every quantity is known, and test what the randomization test needs in order to avoid it.
A pixel world with a stand-in
The logging policy now depends on state the model cannot see: with probability 0.6 it chooses the action that points from the blob’s position six steps earlier toward the centre of the frame, and each other action with probability 0.1. The model sees only the last two frames, so the future actions carry information about an earlier position that the restoring force still acts on. The actions themselves have no effect. We used 256-pixel frames, training sets of 16, 64 and 256 episodes, and either no shared noise or gain noise. The randomization test resamples each future action from the logging policy given the full logged state, which the design records.
The replacement diagnostics—retraining without actions, setting the actions to zero, and shuffling the action windows across origins within an episode—reported an action effect in most test episodes (Figure 4 A): for the convolutional model in 90.6% to 100% of them, and for the ridge model in 64.1% to 100%, across diagnostics, training sizes and noise conditions. With the design-law resampler the randomization test never rejected: the Kish-tempered statistic rejected in 0% of episodes in every cell, and the contrast-tempered statistic plotted in Figure 4 A in none for the two-model average (§ 7.2 ).
The experiment does not show the growth of the diagnostics’ confidence with data that Proposition 6 describes: with the smallest training set, the convolutional model’s diagnostics already reported effects in at least 90.6% of episodes, leaving no room to grow. That property rests on the proposition and on the development simulations (§ 5.1 ).
The resampler must condition on the design state
The protocol included a negative control: the Kish-tempered statistic with a deliberately wrong resampler that draws each action from its marginal frequency in the training episodes, ignoring the logged state. It was expected to reject often. It never rejected (0/64 episodes in every cell), and the pre-declared criterion for the negative control was not met. The result is uninformative rather than reassuring: without shared noise or with gain noise, Kish tempering coincides with the pooled score at this frame size, and the pooled score has no power there (§ 6.3 ). A statistic whose log E E E is driven far below zero by amplification cannot reveal that a resampler is invalid.
After seeing this result we added one post hoc analysis, declared in a dated note committed before any of its e-values were computed: the same wrong resampler with the contrast-tempered statistic, which does have power, on the sealed episodes, frozen models and frozen tempering factors of the six trap cells. The wrong resampler rejected in 98.4% to 100% of episodes for the two-model average (92.2% to 100% for the individual models). With the correct resampler, whose results were read from the sealed records rather than recomputed, the same statistic never rejected for the two-model average (0/64 episodes in every cell); one single-model cell had 1/64. The wrong resampler rejected more often in 18/18 comparisons. This analysis is post hoc and exploratory; it does not replace the frozen negative control, whose criterion remains unmet.
Two lessons follow. First, the conditioning in Assumption 1 is not a technicality: a resampler that ignores part of the state the design used is far from the design law in total variation, nothing in Proposition 5 protects it, and it reproduces the false effect that the replacement diagnostics report. Second, a negative control needs a statistic with power; pooling amplification can hide invalidity as easily as it hides an effect.
Summary across settings
The same three-way pattern appears in every setting we examined. Replacement diagnostics report effects at high rates when there is none: in the development simulations, on real command records with synthetic outcomes, in the sealed challenge, and in the pixel world. A randomization test that resamples from the design law, conditioning on the full state the design used, keeps its error rate. A randomization test that resamples from a simpler law loses that guarantee and reproduces the diagnostics’ error. On the real ERK data the diagnostics reported an effect that is biologically established, so we do not count those reports as instances of the trap (§ 8.2 ).
Real data
The real-data evidence has three stages, each with a different evidential status. M4 is the registered analysis of two previously analyzed datasets. X is a single exploratory application of the repaired statistic to the M4 test fields. M5 is a registered confirmatory test of the repaired statistic on a field that no one involved had opened. Appendix § 19 gives registration identifiers, hashes and the time-ordering of every data access; Appendix § 18 lists every deviation and disclosure.
Data, units and design laws
ERK. In the ERK experiment ([34 ] ), with data released as [33 ] ), each microscope field receives one light program, a sequence of pulses whose inter-pulse intervals were randomized by the acquisition protocol, and the field’s hundreds of cells are recorded under that common program. The field is therefore the assignment unit and its cells are sites; the spillover graph between fields is empty, so the primary null is H 0 G H_0^G H 0 G . The working design law, V3, shuffles the inter-pulse intervals uniformly within interval groups, conditionally on the elapsed interval at the origin, and preserves the protocol’s rest segment; two variants, V1 and V2, serve as sensitivity laws. Fields run under a binary policy are deterministic and contribute e = 1 e = 1 e = 1 . The reference models were trained on four minimum-gap fields from the first replicate and tested on four minimum-gap fields from the second replicate and five ordinary-interval fields.
CcaS/CcaR. In the CcaS/CcaR experiment ([29 ] ), with data and trained models released as Lugagne et al. [28 ] , each cell lineage receives its own light program from a random walk, bounded in the released acquisition code, so each lineage is a unit with a single site. The working law, D1, is the generator in the released acquisition code delayed by one step, consistent with a shift at data export; the delay was found after the first design-law checks failed (Appendix § 17 ). Every CcaS rejection below is conditional on D1. Lineages that follow one of two duplicated fixed templates have zero design entropy and contribute e ≡ 1 e \equiv1 e ≡ 1 ; the remaining lineages are tested under the design law conditioned on avoiding both templates (Proposition 2 (b)), which keeps the test exact. In addition to the four reference models, we test the published model of Lugagne et al. [29 ] with its released preprocessing, after replacing its neighbour-exposure input by the lagged, admissible version (§ 2.3 ).
Models and prior use. The four reference models (linear-Gaussian, GRU, small Transformer and gradient boosting) are additive: an own-history branch plus an action branch fitted to its residual. They were trained on training-unit outcomes only and frozen by hash before any test outcome was opened; their Gaussian variances come from held-out validation data. Both datasets were analyzed earlier in our BioWM-Eval study (Appendix § 15 ); M4 is therefore a re-analysis, not a blind test, although every M4 setting was fixed on synthetic data and command records before the outcomes were reopened.
M4: the registered analysis
M4 tests five prespecified hypotheses with the pooled statistic (Table 3 ): ERK at horizons of ten and thirty minutes, registered as positive controls because optogenetic stimulation is known to drive ERK activity; CcaS at one and six five-minute steps with the four-model average; and CcaS at twelve steps with the published model. A slot is rejected iff E ≥ 100 E \ge100 E ≥ 100 , the e-Bonferroni rule for five hypotheses at family level 0.05.
Table 3. M4 primary results (pooled statistic, registered analysis). Natural-log e e e -values; reject iff E ≥ 100 E \ge100 E ≥ 100 .
Neither positive control was rejected, and the failure was not marginal: every reference model gave a strongly negative log E E E for ERK (from −17,989 to −2973), and so did both sensitivity laws. By the interpretation registered with M4, a strongly negative log E E E for a model means that its predictions under the observed actions scored worse than its predictions averaged over the design; it is not evidence against an effect. A dated note written after we had read the M4 report fixes further interpretation rules; it was written post hoc, and its rules only restrict what may be claimed (Appendix § 18 ). § 6 shows that the failure is pooling amplification: the ERK score sums hundreds of cells that share one action sequence.
Both CcaS hypotheses at the longer horizons were rejected. Restricting the published model to lineages whose lagged exposure could be verified from commands gives log E = 16 , 771 E = 16,771 E = 16 , 771 . A second sensitivity analysis that uses the raw neighbour exposure with singleton averaging gives 3.02; it changes both the exposure alignment and the aggregation, is not a valid test, and is reported descriptively only. The secondary own-channel analysis at six steps gives log E E E from 617 to 1338 across models, but its eligibility bound is vacuous (Proposition 2 (a)), so it is descriptive.
Two secondary results bear on the diagnosis. The batch randomization p-value at a single prespecified origin gave per-model values from 1/128 to 2/128 for ERK at thirty minutes and 12/128 for every model at ten minutes (unadjusted, secondary). A rank-based p-value depends only on where the observed score ranks among the design draws, not on the size of the score differences, so it does not suffer from the over-aggressive betting that drives pooling amplification. The contrast with the e-value product is consistent with the diagnosis; but the registered interface covers such p-values only at one common origin, whereas e-values combine evidence across all origins. Second, the replacement diagnostics reported an ERK action effect, with risk differences from 14.6% to 46.3%. The ERK effect is biologically real, so these are most likely true positives and we do not use them as examples of the stand-in trap. For CcaS at one step the same diagnostics reported effects of magnitude 2.9 × 10 − 8 2.9 \times10^{-8} 2.9 × 1 0 − 8 to 2.9 × 10 − 4 2.9 \times10^{-4} 2.9 × 1 0 − 4 while the randomization test did not reject; the truth there is unknown, so this is descriptive only.
X: an exploratory check on the M4 test fields
After diagnosing the failure (§ 6 ), we selected the Kish-tempered statistic on synthetic data and applied it once, with every parameter fixed on training data, to the nine ERK test fields of M4. We call this analysis X. It is exploratory: the fields had been opened in M4, and the analysis was run because the pooled statistic had failed there. No test outcome was used to choose the statistic or its parameters. The four-model log E E E was 349 at ten minutes and 218 at thirty minutes, against −9576 and −2975 for the pooled statistic on the same data; the individual models ranged from 185 to 350. The exact-form design bound of Proposition 4 for these fields is 925 at both horizons, and X reached 37.7% and 23.5% of it on the log scale. Because X used Monte Carlo e-values, this is a descriptive comparison rather than a ceiling; the corresponding Monte Carlo bound of Proposition 4 (a), computed in the M5 registration package from the frozen Monte Carlo bags, is 914 at ten minutes and 848 at thirty. The ratios are not power estimates. These values are reported descriptively: X was prompted by outcomes already opened in M4, and no formal error guarantee is claimed for it.
M5: a confirmatory test on an unopened field
Design. M5 tests the repaired statistic on one ERK field, a minimum-gap field with thirty-minute minimum intervals from the first replicate. It had been excluded from M4 by the scope of the registered sample (minimum gaps of ten to twenty-five minutes, two replicates), not for data quality. None of our projects had opened it, and before registration the first author recorded a statement that he had never opened, previewed or plotted its outcomes (Appendix § 19 ). The analysis plan was committed before X was run and was not changed afterwards (its committed and registered bytes are identical). It was registered before the data were obtained, in a new registration linked to the M4 registration, together with a disclosure that the X results were available when the plan was finalized. The only primary hypothesis was H 0 G H_0^G H 0 G at ten minutes, rejected if E ≥ 20 E \ge20 E ≥ 20 ; thirty minutes is computed and reported descriptively only. The registered plan chose this horizon before any M5 data were obtained, because a single-field rehearsal had shown weak power at thirty minutes under shared noise (§ 5 ). The statistic is the Kish-tempered score with all parameters frozen on the training fields, the working law V3, and Monte Carlo e-values with B = 127 B = 127 B = 127 .
Command-law gate. Before any outcome statistic was computed, a registered gate checked the reconstructed command series alone: that commands were shared by all cells, that every non-rest inter-pulse interval was at least thirty minutes (the minimum was 31 minutes), that the frozen parser found the rest segment (at interval rank 15), and that the exchangeability diagnostics used for the M4 fields did not flag the interval sequence (p = 0.27 p = 0.27 p = 0.27 , 0.37 and 0.56, against the inherited per-statistic threshold 0.01/39). The gate passed.
Result. The primary hypothesis was rejected (Figure 5 ): the four-model log E E E at ten minutes was 23.1, that is E ≈ 1.1 × 10 10 E \approx1.1 \times10^{10} E ≈ 1.1 × 1 0 10 . All four models agreed (23.0, 23.0, 22.5 and 23.7). The field had 500 cells and 1581 origins; 560,409 cell–origin pairs contributed, 230,091 were excluded by the observation mask, and 1083 factors had zero design entropy. The exact-form design bound is 51.7, and the statistic reached 44.7% of it on the log scale, a descriptive comparison as for X. At thirty minutes the descriptive log E E E was 19.1 (models from 17.0 to 19.9); it carries no confirmatory decision.
What M5 shows. Conditional on the registered design, observation-process and frozen-training assumptions, with the working law V3, the rejection is evidence against the sharp null H 0 G H_0^G H 0 G on a new field: M5 detects the known ERK action effect with the repaired statistic. It does not estimate the size of the effect, it says nothing about power for CcaS or other systems, and it does not validate the forecast gains reported in BioWM-Eval, which answer a different question with a with-versus-without-actions comparison of the kind § 7 shows can be inflated. The evidence is narrow: one field, one dataset, one horizon. The field’s thirty-minute intervals lie outside the training range of ten to twenty-five minutes, which may reduce power; the exact acquisition law for this field is unverified, so validity is conditional on V3; and the gate’s threshold, inherited from the multi-field M1 analysis, is lenient for a single field.
Kish versus contrast tempering on the ERK training fields
To ask whether the registered Kish factor is a reasonable choice on ERK, we compared it with the contrast factor of § 3.4 on the training fields’ validation cells. This diagnostic is exploratory and descriptive: it used training outcomes only, computed no e-value, and touched no test field; its reading rule was fixed before the data were accessed. Expressing the contrast factor as an implied correlation ρ c \rho_c ρ c , the pooled ratio ρ c / ρ ^ \rho_c/\hat{\rho} ρ c / ρ ^ ranged from 0.554 to 0.818 across models at ten minutes and from 0.574 to 0.828 at thirty minutes. All values fall inside the pre-declared band from 0.5 to 2, so by the pre-declared reading Kish tempering is approximately adequate on ERK; all are below one, so Kish tempering over-tempers somewhat and is conservative there. Individual fields gave ratios from 0.41 to 1.27; Appendix § 20 lists every field.
Two qualifications follow from Proposition 7 . Because the action branch of the ERK reference models predicts the same response for every cell of a field, the contrast is shared across cells and the contrast factor has the form of Proposition 7 : a Kish factor with a contrast-weighted residual correlation, at an effective cell count since counts vary across origins. The agreement is therefore partly structural, and the diagnostic is not an independent validation of the Kish rule. And the gap is not only a matter of median versus mean: against the mean pairwise residual correlation the ratio at ten minutes was still between 0.598 and 0.854, so contrast weighting itself lowers the effective correlation; we did not decompose the remainder further. The diagnostic used the validation cells of each training field, fewer than the cells of a test field, so comparisons at test-field size assume that correlations do not change with the number of cells. We did not compute contrast-tempered e-values on the M5 field, and we make no claim about what they would have been.
Validation tuning on the ERK test fields (exploratory)
Validation tuning and the mixtures had no ERK entry in § 6.4 , because the saved ERK-like runs lack the validation quantities they need. We therefore computed them on the nine ERK test fields of M4, as a second exploratory analysis after X, under a protocol committed before the fields were reopened (Appendix § 25 ). Everything was as in X—the models, the design law V3 and X’s frozen Monte Carlo bags—except that each model’s factor for V was chosen on the validation cells of the four training fields, with 31 design draws per origin. A reproduction gate first recomputed the four-model log E E E of the pooled statistic in M4 and of the Kish-tempered statistic in X; both matched.
Validation tuning collapsed (Table 4 ). Its four-model log E E E was 210 at ten minutes, with two of the four models strongly negative (−1420 and −979), and −101 at thirty minutes, with all four models negative. The uniform mixture gave 347 and 218, W(3/4) 345 and 217, and Kish tempering 349 and 218 as in X; the mixture was within log 15 of the best fixed grid member at both horizons, as Proposition 10 requires. The factor chosen on the training fields was 1/32 to 1/4, whereas the per-origin Kish factor on the test fields was about 0.008 to 0.04. One reading is that a factor tuned on the training fields’ validation cells, which are fewer than the cells of a test field, is too large for the test fields. § 8.7 tests this reading. The analysis is exploratory for the same reasons as X: the fields had been opened, and no error guarantee is claimed for it.
Table 4. ERK validation-tuning check (exploratory; the nine M4 test fields). Four-model natural-log e e e -values, and the number of the four models with negative log E E E .
These fields cannot serve a confirmatory test of the mixture, and no other field of this dataset can either: an inventory of the released ERK data, using archive metadata and our own records only, found no randomized field that we could certify as never opened by any of our projects (Appendix § 25 ). A registered biological test of the mixture therefore needs new data.
A site-count-adaptive factor on the ERK test fields (exploratory)
If § 8.6 ’s reading is right, a factor that follows the number of cells should not collapse. We repeated the check of § 8.6 with the site-count-adaptive statistics of § 3.5 , under a protocol committed before the fields were reopened. The analysis was gated on reproducing the values of § 8.6 for C0, C1, V and M M M at both horizons. A first attempt failed this gate because of a difference in preprocessing and was discarded; after the preprocessing was matched, all eight comparisons passed (Appendix § 25 .§ 9 ).
The collapse disappeared (Table 5 ). The four-model log E E E of V ρ V_{\rho} V ρ was 350.5 at ten minutes and 206.1 at thirty, against 210 and − 101 -101 − 101 for V V V . But V ρ V_{\rho} V ρ was still not safe for every model: at ten minutes its log E E E was 36.2 for the linear-Gaussian model, 176.9 for the GRU, 300.7 for the Transformer and 351.9 for gradient boosting, while the adaptive mixture M ρ M_{\rho} M ρ gave 293.7, 321.3, 298.0 and 349.2. The four-model value hides this, because it is dominated by the best model. The mixtures and Kish tempering were within a few units of each other at both horizons. So the statement of § 8.6 needs a qualification: on these fields validation tuning collapsed because one factor was tuned for fields of different sizes; tuned in a form that follows the site count, it did not collapse, but it still lost most of the evidence for one model, and the mixtures did not. The analysis is exploratory for the same reasons as § 8.6 .
Table 5. ERK site-count check (exploratory; the nine M4 test fields). Four-model natural-log e-values.
A published world model
Every model in the preceding sections was built by us. If pooling amplification is a property of pooled scores and not of our models, it should appear on a world model trained by others, and the statistic chosen in § 6.4 should repair it there. We tested both claims in a registered confirmatory study on DIAMOND ([1 ] ), a diffusion world model for Atari games whose authors released trained checkpoints. We neither trained nor modified the models.
Design
Models and environment. We used the released checkpoints for Boxing, Pong and Breakout at a fixed revision, with their hashes recorded before registration (Appendix § 23 ). The Arcade Learning Environment ([2 ] ), an interface to the Stella Atari 2600 emulator, was configured exactly as for DIAMOND’s training: four frames per action, the maximum of the last two frames, 64 × 64 64 \times64 64 × 64 RGB frames, no sticky actions and the minimal action sets of 18, 6 and 4 actions. The model predicts the next frame from the last four frames and actions with a three-step sampler; at each origin one noise image was drawn and shared by every candidate action, so the prediction is a fixed function of the candidate. The candidate block is the next action (h = 1 h = 1 h = 1 ), and the sites are the n = 64 ⋅ 64 ⋅ 3 = 12,288 n = 64 \cdot64 \cdot3 = 12{,}288 n = 64 ⋅ 64 ⋅ 3 = 12 , 288 pixel channels. The score is the Gaussian log density of the observed next frame around the predicted one, with per-site variances estimated on validation units and floored at the square of one intensity step.
Two conditions. In the ACTIVE condition the emulator executed the recorded action, drawn uniformly at random, so actions matter. In the NULL condition the recorded labels were drawn the same way but the emulator executed an independent random sequence, so the labels have no effect on any frame and the null holds exactly. A unit is one segment of up to 500 agent steps from a reset. Each game and condition had 16 validation units; there were 32 ACTIVE and 64 NULL test units per game.
Table 6. Registered test on DIAMOND. Rejections (E ≥ 20 E \ge20 E ≥ 20 ) in the ACTIVE (/32) and NULL (/64) test units of each game, and the median log E E E of the ACTIVE units. The mixture over fifteen factors (M) is the primary statistic.
Statistics and criteria. The primary statistic was the tempering mixture over fifteen factors, the grid reaching 1 / n 1/n 1/ n (§ 3.5 ). The pooled score, Kish and contrast tempering, validation-tuned tempering, the thirteen-value mixture and the two rank e-values were secondary. Every statistic used the same Monte Carlo bags (B = 127 B = 127 B = 127 ), rejected at E ≥ 20 E \ge20 E ≥ 20 , and took its factors from the validation units of its own game and condition. The registered criteria, per game, were: validity , each statistic rejecting in at most 2 of the 64 NULL units; collapse , a median pooled log E E E below zero in the ACTIVE units; and repair , the mixture rejecting in at least 24 of the 32 ACTIVE units. Overall confirmation required validity in all three games and both collapse and repair in at least two. Five dated predictions were registered with the criteria.
Registration and seeds. The protocol, the identity of the frozen analysis code and a note on its dates were registered on OSF (ctfxv) before any confirmatory unit existed. The confirmatory seeds were derived from the first beacon round scheduled at least two minutes after the start of the minute in which the registration was recorded, so no unit could have been generated earlier. A pilot on separate seeds had run the full pipeline under NULL only; under ACTIVE it computed predictions and validation quantities but no e-value. Two integrity checks of the frozen code stopped the confirmatory run before any ACTIVE test unit was generated: a field-name collision in the access log, and a storage tolerance tighter than float32 rounding. Each was fixed by a dated deviation published on OSF before the run resumed with the same seeds, and every regenerated unit was identical, array by array, to its earlier copy (Appendix § 23 ).
Results
Table 6 gives the results. Pooling amplification appeared on the published models: although actions visibly move the player, paddle or ball, the pooled score rejected in 0 of the 96 ACTIVE units, with median log E E E between −222,582 and −5408 across games. Tempering by an estimated factor did not prevent it: Kish tempering rejected in 0 and contrast tempering in 1 of the 96 units, with medians from −2698 to −383 and from −4012 to −95.3. The mixture rejected in 96 of the 96 units, with medians from 126 to 455; validation tuning, the thirteen-value mixture and both rank e-values did as well. Under the exact null the eight statistics made 0 rejections in 8 × 192 tests, and the mixture’s median log E E E stayed between −2.79 and −1.35, comparable in magnitude to log 15 ≈ 2.71, the largest log-e loss relative to the best member of the fifteen-factor grid (Proposition 10), although the proposition does not assert a null loss of that size.
All three criteria held in all three games, so the study confirmed its claim. Of the five predictions, 4 held. The one that failed was ours about ranks: we predicted that the rank e-value would reject less often than the mixture in at least two games, and it matched the mixture in all three. The replacement diagnostic reported an action effect in 96 of the 96 ACTIVE units and in 5 of the 192 NULL units; because the NULL labels are independent of everything the model sees, this design produces no stand-in effect, and the trap of § 7 is not tested here.
The estimated factors were an order of magnitude or more above the factor selected on validation data. Across games, λ K \lambda^{K} λ K ranged from 0.0181 to 0.320 and λ c \lambda^{c} λ c from 0.0241 to 0.0773, whereas λ V \lambda^{V} λ V was 0.00195 to 0.00391; the contrast factor exceeded it by a factor of 12.3 to 19.8. With predictions, residuals and contrasts held fixed, both estimated factors are invariant under a common multiplier of all site precisions, whereas for unfloored Gaussian scores the tempered log-e e e curve shifts by the inverse multiplier along the temperature axis (Proposition 9 ). This is consistent with a possible mechanism for the failure; it is a descriptive, post hoc reading and not a test (Appendix § 23 gives the saved score bags it rests on).
What this shows
On world models that we did not build, the pooled design-based test collapsed although actions mattered, tempering by factors estimated from residuals did not prevent the collapse, and a mixture fixed in advance restored power while every statistic kept its error rate under an exact null. The study does not show that DIAMOND predicts the consequences of actions correctly: a rejection is evidence that the recorded actions affect the frames, read through the model’s predictions, not a validation of those predictions. The action effects in these games are strong, so the study says little about power for weak effects, where the mixture was weaker than contrast tempering in the pixel world (§ 6.4 ). It covers one model family, three games and a horizon of one step. The weighted mixture was defined after this study. Computed afterwards from the saved scores of the registered run, as part of the over-betting study (§ 12 , Appendix § 25 ), W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected in all 96 ACTIVE units and in 0 of the 192 NULL units. This computation is post hoc and is not part of the registered test. § 10 extends the test to two further model families.
Score rule and sampling noise (exploratory)
Two objections to § 9.2 can be checked on the saved data: that the collapse is an artifact of the per-pixel Gaussian score, and that a single sampled prediction is too noisy. Both studies below are exploratory, under protocols committed before they were run (Appendix § 25 ).
Score rule. We rescored every saved DIAMOND unit under ten alternative score rules, with predictions, observations and bags unchanged: two higher variance floors, (8/255)2 ^{2} 2 and (32/255)2 ^{2} 2 ; one common variance for all sites; Student-t (ν = 3 \nu= 3 ν = 3 ) and Laplace densities; averages over blocks of 2×2, 4×4 and 8×8 pixels; a random projection to 256 dimensions; and a joint Gaussian whose covariance is a rank-16 principal-component model of the validation residuals plus a diagonal. A reproduction gate first recovered every rejection decision of the registered run exactly. Under every rule, no statistic rejected in any of the 64 NULL units of any game, and the mixture rejected in all 32 ACTIVE units of every game. The pooled score collapsed under every rule in at least two of the three games, including under the joint Gaussian (median log E \log E log E −366,580, −66,379 and −4324). Coarser scores weakened the collapse: with 8×8 blocks, 192 sites instead of 12,288, the median pooled log E \log E log E was −6656 in Boxing, −77 in Pong and 67 in Breakout, against −222,582, −21,805 and −5408 per pixel. Kish tempering reached 24 of 32 units in 1 of the 27 game–rule pairs in which it was defined. Contrast tempering depended on the rule: it reached 24 of 32 units in Pong or Breakout under the higher floors and the coarser blocks, and in Boxing only under the random projection.
Sampling noise. DIAMOND samples its prediction, and the registered test used one sample per origin. We regenerated all 288 units of the three games with predictions averaged over four samples, after a reproduction gate had recovered a registered unit bit for bit. The collapse was unchanged: the median pooled log E \log E log E of the ACTIVE units was −220,241, −22,354 and −5553, against −222,582, −21,805 and −5408 with one sample. Kish and contrast tempering again rejected in 0 of 32 units in every game; validation tuning, the mixture and both rank e-values rejected in all 32; under the null every statistic rejected in 0 of 32 units. The factor chosen on validation data was 0.002 to 0.004, against Kish factors of 0.018 to 0.32.
Two further published world models
§ 9 covers one model family. We registered a second confirmatory test on two families that differ from DIAMOND, and from each other, in how they predict and how they can be scored: IRIS [32 ] , which compresses each frame into 16 discrete tokens and predicts the next frame’s tokens with an autoregressive Transformer, and TD-MPC2 [19 ] , which predicts in a learned latent space and has no decoder. Both were trained and released by their authors; we neither trained nor modified them.
Design
Models and environments. For IRIS we used the released checkpoints of all 26 games of its benchmark, none excluded, with IRIS’s own preprocessing: four frames per action, the maximum of the last two frames, 64 × 64 RGB frames, no sticky actions and minimal action sets. For TD-MPC2 we used released single-task checkpoints for three DeepMind Control Suite tasks [41 ] —cartpole-balance, reacher-easy and walker-walk—and three MyoSuite tasks [6 ] —myo-pose, myo-reach and myo-obj-hold. Code revisions and checkpoint hashes, and one wrapper patch that changes no dynamics, were frozen before registration (Appendix § 24 ).
Scores. For IRIS, the primary score is the model’s own log-likelihood of the 16 tokens of the observed next frame, computed by teacher forcing. It is a joint likelihood over the tokens, not a sum of independent site scores. As a secondary score in Boxing, Pong and Breakout, we decoded a sampled next frame and scored it with the per-pixel Gaussian of § 9 , over 12,288 sites. For TD-MPC2, which has no decoder, the score is a Gaussian on the 512 coordinates of the predicted next latent state, centred on the encoding of the observed next observation, with per-coordinate variances from validation units. The candidate block is the next action (h = 1 h = 1 h = 1 ) throughout.
Conditions. Recorded labels were drawn uniformly at random. In an ACTIVE condition with execution probability p p p , the executed action was the recorded label with probability p p p and otherwise an independent uniform draw; p = 1 p = 1 p = 1 is full execution. In the NULL condition the executed actions were drawn independently of the labels, so the null holds exactly. Every game and task had p = 1 p = 1 p = 1 and NULL; Boxing, Pong, Breakout and the six control tasks also had p = 0.5 p = 0.5 p = 0.5 , 0.2 and 0.05. Partial execution makes the action effect intermittent: the model assumes the recorded action was executed, so it is misspecified along the action contrast even where its dynamics are right. Each game or task and condition had 16 validation units and 32 test units, 64 under NULL. A unit is one segment of up to 500 steps from a reset, and of up to 100 or 75 steps for the MyoSuite tasks.
Primary statistic. The primary statistic was the validation-weighted mixture W(3/4) of § 3.5 . The regime map’s rule had selected V (§6.5), but V collapsed on the ERK test fields (§8.6). Before any data of this test existed, we therefore registered W(3/4), which keeps most of V’s power when V’s factor is right and the mixture’s protection when it is wrong. The later analysis of § 8.7 showed that V’s collapse on the ERK fields can be removed by a site-count-adaptive factor. The registered choice predates it; with that analysis in hand the case against V is weaker than it looked, though Vₚ still lost most of the evidence for one model. The pooled score, Kish and contrast tempering (Gaussian scores only), V, M, M13, W(1/4), W(1/2) and both rank e-values were secondary. All statistics used the same bags (B = 127 B = 127 B = 127 ), took their factors from the validation units of their own game or task and condition, and rejected at E ≥ 20 E \ge20 E ≥ 20 .
Criteria and predictions. For each primary arm—IRIS tokens and TD-MPC2—the registered criteria per game or task were validity, every statistic rejecting in at most 2 of the 64 NULL units, and power, W(3/4) rejecting in at least 24 of the 32 units at p = 1 p = 1 p = 1 . An arm was confirmed if validity held in every game or task and power held in at least 18 of the 26 games, or in at least 4 of the 6 tasks. Eight dated predictions were registered with the criteria.
Registration, seeds and deviations. The protocol and the identity of the frozen code were registered on OSF (j75g2) on 9 November 2026. As in § 9 , the seeds were derived from the first beacon round scheduled at least two minutes after the start of the minute in which the registration was recorded. A pilot on separate seeds had run the full pipeline under NULL only; under ACTIVE it computed predictions and validation quantities but no e-value. The confirmatory run generated all 5392 units once. Its last step, writing the decision file, failed on a data-type error in the JSON writer. A dated deviation changed only that conversion and wrote the decision from the stored, hashed results, and an independent script that does not use the frozen code reproduced every decision value. An earlier attempt had stopped before generating anything, because of an error in our run instruction, not in the protocol (Appendix § 24 ).
Results
Table 7 summarizes the results; Appendix § 24 gives every game, task and condition.
Validity. In the two primary arms the statistics made 4 rejections in 19,200 NULL tests, none of them by the primary statistic. No statistic rejected in more than 1 of the 64 NULL units of any game or task, so validity held everywhere. The secondary pixel score made 0 rejections in 2112 NULL tests.
Power. W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected in at least 24 of 32 units at p = 1 p = 1 p = 1 in 24 of the 26 IRIS games and in all six TD-MPC2 tasks, so both arms were confirmed. It missed in BankHeist (16 of 32) and UpNDown (8 of 32), where no statistic reached the bar; validation tuning had 19 and 9.
The model’s own likelihood. We had predicted, with probability 0.5, that the pooled token score would collapse at p = 1 p = 1 p = 1 in at least two of Boxing, Pong and Breakout. It collapsed in none of the three (median log E E E 144, 428 and 121). It collapsed in all three once execution was partial (at p = 0.5 p = 0.5 p = 0.5 : −3258, −2615 and −617). The protocol had fixed this outcome as meaning that amplification needs misspecification along the action contrast, and that a proper likelihood does not prevent it once the action model is wrong. Across all 26 games, which the prediction did not cover, the pooled token score had a negative median log E E E at p = 1 p = 1 p = 1 in 14 and reached 24 of 32 units in 10. The factor chosen on validation data for tokens was between 1/16 and 1 across the 26 games, with median 1/4; in the three games scored both ways it was 1/4 to 1/2 for tokens and 1/1024 to 1/256 for decoded pixels. The model’s own likelihood needs far less tempering than an independent pixel score, but the factor chosen was below 1 in 25 of the 26 games.
Pixels and latents. On the decoded pixels, the pattern of § 9 recurred on a second model family: the pooled score and Kish tempering rejected in 0 of 96 units, contrast tempering in 20, and V, M and W ( 3 / 4 ) W(3/4) W ( 3/4 ) in 96. On TD-MPC2’s latent score, the pooled score reached the power bar in 1 of 6 tasks, Kish tempering in 2 and contrast tempering in 4. In reacher-easy and walker-walk the Kish factor was 1 in the NULL and partial-execution conditions—the residuals showed no correlation to correct for—while the factor chosen on validation data was between 6 × 10 − 5 6 \times10^{-5} 6 × 1 0 − 5 and 8 × 10 − 3 8 \times10^{-3} 8 × 1 0 − 3 .
Partial execution. As execution became rarer, the pooled score failed first (Figure 7 ). At p = 0.5 p = 0.5 p = 0.5 it rejected in 0 of 32 units in all nine sets, while W ( 3 / 4 ) W(3/4) W ( 3/4 ) reached 24 of 32 in seven. At p = 0.2 p = 0.2 p = 0.2 W ( 3 / 4 ) W(3/4) W ( 3/4 ) reached the bar in four sets, and at p = 0.05 p = 0.05 p = 0.05 no statistic did in any set. W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected in more units than the pooled score at p = 0.2 p = 0.2 p = 0.2 in eight of the nine sets.
Figure 7. Partial execution. Rejections out of 32 test units as the recorded action is executed with probability p p p . The pooled score fails first; at p = 0.05 p = 0.05 p = 0.05 no statistic reaches the power bar.
Of the eight dated predictions, seven held; the one that failed is the prediction about the token likelihood above. The replacement diagnostic reported an effect in 1 to 7 of the 64 NULL units of each game or task, and in most units at p = 1 p = 1 p = 1 ; as in § 9 , the NULL labels are independent of everything the model sees, so this design produces no stand-in effect.
What this shows
On two further model families and three kinds of score, the registered mixture kept its error rate under exact nulls and, at full execution, reached the power bar wherever any statistic did. Pooling amplification was not confined to Gaussian pixel scores: it appeared on a latent score and, in more than half of the games, on a model’s own token likelihood, although there much less tempering was needed. Factors estimated from residuals failed again; on two control tasks the Kish factor saw nothing to correct. The study does not validate the counterfactual predictions of these models: a rejection says that the recorded actions affect the observations, read through the model. It covers simulators with random actions and a horizon of one step, and weak effects only through partial execution, where every statistic had lost its power by p = 0.05 p = 0.05 p = 0.05 .
Table 7. Registered test on IRIS and TD-MPC2. IRIS tokens and TD-MPC2: number of games or tasks in which the statistic rejected in at least 24 of 32 units at p = 1 p = 1 p = 1 (the registered power bar), number with median log E < 0 E < 0 E < 0 at p = 1 p = 1 p = 1 , and NULL rejections. IRIS pixels (secondary, three games): rejected units at p = 1 p = 1 p = 1 and NULL rejections. W(3/4) is the primary statistic. A dash: not defined for that score.
Real robot data with a published model
Sections 9 and 10 used simulators, where the null can be imposed and the data regenerated. We registered a third confirmatory test on recorded robot video, which cannot be replayed, with a published model that we did not train.
Design
Data and model. BAIR robot pushing (Ebert et al., [12 ] ) records a robot arm pushing objects: 30 frames per trajectory at 64 × 64 64 \times64 64 × 64 pixels, with a four-component command at every frame. iVideoGPT (Wu et al., [46 ] ) compresses a frame into tokens and predicts the next frame’s 16 tokens from one context frame and the command. We used its released action-conditioned checkpoint for these data, unchanged.
Units and score. The study set is the official test trajectories 16–255. Their images had not been opened by us before registration; trajectories 0–15 had been read in a feasibility check and served the pilot only. The recorded commands change every two frames. We kept the 239 trajectories in which this holds exactly and excluded one. A unit is one trajectory with 14 origins, the frames at which a new command starts, without the first. The candidate block is the command at the origin (h = 1 h = 1 h = 1 ), and the score is the model’s own log-likelihood of the 16 tokens of the observed next frame, by teacher forcing, as for IRIS in §10. A public beacon split the units at random into 80 validation and 159 test units.
Two arms. In the NULL arm the label of each origin is our own draw, so the null holds by construction and the test is exact. In the ACTIVE arm the label is the recorded command. That the commands move the arm is physically certain, so this arm measures power; it is not evidence that an effect exists.
Two design laws. A design-based test needs the law from which the recorded commands were drawn. The software and configuration of the 2017 collection could not be established, so we worked from the command records: successive commands are uncorrelated (within ± 0.011 \pm0.011 ± 0.011 ) and show no dependence on the arm’s position near the edge of the workspace. Under the primary law, L-emp, the candidates of a unit are recorded commands of training trajectories, drawn without replacement from a fixed pool of 14,294. If commands are independent draws from one law that does not depend on the scene, the recorded command and its candidates are exchangeable whatever the law’s shape. Under the sensitivity law, L-par, candidates are fresh draws from a law fitted to the records. Before the code was frozen, the commands of the study set and of the pool were compared from records alone; the four tests gave p = 0.20 p = 0.20 p = 0.20 , 0.34 0.34 0.34 , 0.16 0.16 0.16 and 0.40 0.40 0.40 . The validity of the ACTIVE arm is conditional on these working assumptions, as M5’s is on its working law (§ 8 ); the NULL arm does not depend on them.
Figure 6. Registered test on IRIS and TD-MPC2: rejections out of 32 test units at full execution. The line marks the registered power bar.
Statistics, criteria and predictions. The primary statistic was again W(3/4). The pooled score, V, M, W(1/4), W(1/2), both rank e-values and two online rules (§ 3.5 ) were secondary. In the NULL arm, V and W(3/4) were also computed at the factor chosen by the ACTIVE arm: a factor chosen on null data sits near zero, where these statistics hardly move and their null check says little. All statistics used bags of 128 and rejected at E ≥ 20 E \ge20 E ≥ 20 . The registered criteria were validity, each of the twelve NULL statistics rejecting in at most 4 of the 159 test units under each law, and power, W(3/4) rejecting in at least 120 of the 159 under L-emp. Six dated predictions were registered with them.
Registration and run. The protocol and the frozen code were registered on OSF (4wj7k) on 10 November 2026, after a pilot that ran the constructed null only, on 48 trajectories outside the study set, and a rehearsal of the complete confirmatory path on synthetic scores. The seeds came from the first beacon round scheduled at least two minutes after the start of the minute in which the registration was recorded. The run was generated and analysed once, with no deviation. An independent script, written after the results were committed, reproduced every unit e-value and decision count.
Results
Validity. Under the constructed null, no statistic rejected in more than 2 of the 159 test units under L-emp, or in more than 1 under L-par (Table 8 ). The primary statistic rejected in none, also at the factor of the ACTIVE arm.
Table 8. Real robot data (BAIR pushing, iVideoGPT; registered, OSF 4wj7k). Test units rejected out of 159, and the median natural-log e-value with the recorded commands. NULL is the constructed null; ACTIVE uses the recorded commands. The validity criterion allowed at most 4 NULL rejections; the power criterion required at least 120 ACTIVE rejections of W(3/4) under L-emp.
Power. With the recorded commands, W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected in 155 of the 159 test units under L-emp, against the required 120, and in 156 under L-par. The test was confirmed. The two laws agreed in 158 of the 159 units.
The pooled score did not collapse. It rejected in 143 units, with a median log E E E of 13.9 against 15.3 for W(3/4). We had given a collapse 40%. The validation factor was 1/2 under both laws. As in Boxing, Pong and Breakout at full execution (§ 10.2 ), a model’s own joint likelihood over 16 tokens, applied to commands that were executed, over-bets only mildly.
What tempering still buys. The pooled score failed in 16 units, and in 13 of them its log E E E was negative, down to − 134.5 -134.5 − 134.5 . In these units the loss was concentrated: the single worst origin accounted for the median unit’s whole net loss (exploratory). The smallest log E E E was − 52.5 -52.5 − 52.5 for V and − 1.8 -1.8 − 1.8 for W(3/4), whose loss against the best grid member is bounded. Each unit’s best factor on a fine grid was below 1 in 150 of the 159 units (median 0.59), and the curve is flat below its maximum and steep above it (Figure 8 b), as Proposition 13 implies: at a factor of 4 the median log E E E was − 41 -41 − 41 . Used on fewer origins, the statistics separate further: with the first four origins of each unit, W(3/4) rejected in 122 units and the pooled score in 104, and the pooled count fell from 130 to 127 between nine and eleven origins while the mixture’s never fell (Figure 8 a). These descriptions were planned in the protocol and are not part of its criteria.
Figure 8. Real robot data. (a) Rejections as origins are added. The pooled count is not monotone; the mixture’s is. (b) The unit log e-value as a function of the tempering factor: flat below its maximum, steep above it. Descriptive; planned in the registered protocol.
Other statistics. M rejected in 154 units, V in 155, the rank e-values in 153 and 154, and the online rules in 149 and 153. The replacement diagnostic was positive in 151 of the 159 units with recorded commands and in 4 under the null.
Predictions. Five of the six held: validity (95%), power (65%), the pooled score rejecting in fewer units than W(3/4) (65%), a validation factor below 1 (75%) and agreement of the two laws within 15 units (80%). The prediction that the pooled score’s median log E E E would be negative (40%) failed.
What this test shows
On real data that cannot be replayed, with a model we did not train, the test kept its error rate under an exact null, and the registered mixture had power, conditional on the working assumptions for the recorded commands. The test does not show a collapse of the pooled score on real data. It places these data in the regime in which the pooled score mostly works, and there the mixture cost nothing in rejections and bounded the loss in the units where the pooled score failed.
Three limits apply to the ACTIVE arm. The design law is reconstructed from the records, not documented. The pool’s commands were part of the model’s training data and the recorded commands of the study units were not; L-par, whose candidates the model has never seen, gave the same result. And the study units belong to the official test set, which the model’s authors may have used to choose the checkpoint. None of these affects the NULL arm.
One account of every collapse (exploratory)
Sections 6 to 11 report collapses of the pooled score, of Kish and contrast tempering, and of validation tuning. Proposition 13 says what they must have in common: a tempered statistic can have a negative log e-value only if its factor exceeds the unit’s break-even factor. We checked this, and how far the failing factors were from the best ones, on the saved scores of the DIAMOND study and of the IRIS and TD-MPC2 study, under a protocol committed before the computation (Appendix § 25 ). The analysis is exploratory: the units had been analysed before.
The asymmetry holds in every unit. In all 4512 units (288 DIAMOND, 288 IRIS decoded-pixel, 2784 IRIS token and 1152 TD-MPC2 test units, including NULL and every stored execution condition), the unit log e-value was concave in the factor on a fine grid, the proportional bound of Proposition 13 (b) held, and every negative value lay above the break-even factor. The mixture’s bound of Proposition 10 (b) held in all 4512 units, and the half bound of Proposition 13 (d) in all 3114 units to which it applies.
How far off the failing factors were. In the twelve cases at full execution in which Kish or contrast tempering rejected in fewer than 24 of the 32 units of a game or task while W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected in at least 24, all on IRIS’s decoded pixels or on TD-MPC2, the estimated factor exceeded the unit’s best factor by a median, over the units of the case, of at least 6.26 times. On DIAMOND the best factor was between 0.0016 and 0.0046 across the three games and the break-even factor between 0.0046 and 0.031, while the Kish and contrast factors were 11 to 82 times the best factor in the median unit (Table 9 gives the medians of each factor). An estimated factor that is too large by an order of magnitude lands beyond break-even; a factor that is too small by the same amount would have kept a tenth of the log e-value.
Table 9. Over-betting study (exploratory). Top: DIAMOND, per game, the unit’s best factor, break-even factor, Kish factor and contrast factor (medians over ACTIVE units). Bottom: units rejected (DIAMOND, of 96; IRIS pixels, of 96) or games and tasks at or above the power bar (IRIS tokens, of 26; TD-MPC2, of 6), with actions executed.
Online rules. Choosing the factor within the unit from earlier origins removes the need for validation data. Under the null these rules made at most 1 rejection in the 64 units of any game or task. With actions executed they were weaker than the mixtures (Table 9 ): the follow-the-leader rules reached the power bar or rejected in fewer units than W ( 3 / 4 ) W(3/4) W ( 3/4 ) on every score, and halving the leader’s factor (O57h) did better on three of the four scores and no worse on the fourth, as the asymmetry suggests. In the robot-data study, where two of them were registered as secondary statistics, they rejected in 149 and 153 of 159 units against 155 for W ( 3 / 4 ) W(3/4) W ( 3/4 ) (§ 11.2 ). A follow-the-leader rule stakes everything on one factor at each origin; the mixture, which is the wealth-weighted rule of Proposition 13 (e), does not, and the identity was confirmed numerically (largest difference 5 × 10 − 12 5 \times10^{-12} 5 × 1 0 − 12 ).
What this explains. The same asymmetry accounts for the three kinds of failure in this paper. The pooled score uses the factor 1, far above break-even when thousands of sites share a model error (§ 6 , § 9 ). Kish and contrast factors estimate a design effect from residuals and came out too large by one to two orders of magnitude on published models (§ 9 , § 10 ). A factor tuned on validation units of a different size transfers badly in the same direction (§ 8.6 , § 8.7 ). It also explains the shape of the result on robot video, where the pooled score’s factor of 1 was above the best factor in 150 of 159 units, but by less than a factor of two in the median unit, so the loss was small in most units and large in a few (§ 11.2 ). For practice it gives a rule that needs no model of the dependence between sites: when unsure, err toward the smaller factor, or do not choose and mix.
One prediction of the study’s protocol was wrong as stated. The protocol claimed a half bound for the mixture without the condition that some factor in the range be profitable; the bound failed in 82 units, all of them units in which no grid factor was profitable. A dated amendment, written after the result, replaced the claim by the two bounds reported above; we record the prediction as contradicted.
Discussion and limitations
Across settings. Table 10 collects every setting in which we compared the tempering statistics. Each statistic that relies on one factor collapsed—its median log e-value fell far below zero while another valid statistic rejected in most units—in at least one setting: the pooled score in most of them; Kish tempering in the pixel world, on DIAMOND, on IRIS’s decoded pixels and on four TD-MPC2 tasks; contrast tempering on DIAMOND, on IRIS’s decoded pixels and on two TD-MPC2 tasks; and validation tuning on the ERK test fields, where a site-count-adaptive form removed the collapse but still lost most of the evidence for one model (§ 8.7 ). The mixtures never collapsed. The uniform mixture paid its log-grid price where effects were weak: it missed the power bar in two IRIS games and one TD-MPC2 task in which validation tuning reached it. In the IRIS and TD-MPC2 study the weighted mixture reached the bar in exactly the games and tasks in which validation tuning did, and it kept the uniform mixture’s protection on the ERK test fields.
Table 10. All settings. Pixel world: rejected episodes at the larger effect (of 192, two-model average). ERK-like: alternative cells with all 16 replicates rejected (of 12, E ≥ 100 E \ge100 E ≥ 100 ). ERK test fields: four-model log E E E at 10 and 30 minutes (exploratory; X and the ERK validation-tuning check). DIAMOND: rejected ACTIVE units (of 96). IRIS tokens and TD-MPC2: games or tasks at or above the power bar (of 26 and 6). IRIS pixels: rejected units at p = 1 p = 1 p = 1 (of 96). Robot video: rejected test units with the recorded commands (of 159). DIAMOND W(3/4): computed post hoc from saved scores (§ 9.3 ). A dash: not defined.
Where the pooled score collapses. It collapsed with per-site Gaussian scores on pixels and latents (the pixel world, DIAMOND, IRIS’s decoded pixels, four of six TD-MPC2 tasks), on the ERK fields, with IRIS’s token likelihood in 14 of 26 games, and in every set once actions were executed only part of the time. It did not collapse with IRIS’s token likelihood in Boxing, Pong and Breakout at full execution, or with VideoGPT’s token likelihood on robot video. A model’s own joint likelihood over a few tokens, applied to actions that were executed, over-bets mildly or not at all; an independence score over thousands of sites, or any score under a wrong action model, over-bets by orders of magnitude (§ 12 ). We cannot tell in advance which case a new model falls in. Where the pooled score worked, the weighted mixture cost nothing in rejections, and it bounded the loss in the units where the pooled score failed.
What to do in practice. For anyone asking whether a world model trained on experimental data responds to actions, our results suggest five habits. First, do not read an action effect off replacement diagnostics; they answer a different question and are fooled by stand-in effects. Second, if actions were randomized, reconstruct the design law and resample from it conditionally on everything the design conditioned on; a resampler that is simpler than the design reintroduces the trap. Third, compute the design-only bound before opening any outcome; a deterministic protocol can never yield evidence, whatever the model. Fourth, when many sites share an action sequence, do not pool their log scores at face value, even when the score is the model’s own likelihood; temper them, and prefer a mixture over a fixed grid of factors to any single factor, estimated or tuned, since every single factor we tried failed somewhere, always by being too large. A factor that is too small costs a proportional share of the evidence; one that is too large can cost all of it (§ 12 ). Weighting the mixture toward the factor chosen on validation data recovers most of that factor’s power at little risk. Fifth, give every negative control a statistic that has power, or an invalid resampler can go unnoticed.
The confirmatory evidence is narrow. On biological data, our confirmatory test of the repaired statistic is one field of one dataset at one horizon. It detects an effect that biology already establishes, so it shows that the repaired test has power on real data, not that it discovers new effects; it says nothing about the size of the effect or about power in other systems. The thirty-minute horizon and the CCaS data were not part of that test. The registered tests on published world models (§ 9 –§11}) cover four model families, 26 Atari games, six control tasks and one robot dataset, at a horizon of one step. Two of them use simulators with random actions. The third uses recorded data, and its arm with the recorded commands is conditional on a design law that we reconstructed from the command records, because the configuration of the original collection could not be established; its null arm is exact. Together they show that pooling amplification and its repair by mixtures carry over to models we did not build, and that the test keeps its error rate and has power on data that cannot be replayed. They do not validate those models’ counterfactual predictions, they do not test the stand-in trap, and they reach weak effects only through partial execution. No registered test of the mixture on biological data exists.
The analysis was not blind, and part of it is post hoc. The datasets had been analyzed before in our own earlier study, so M4 is a registered re-analysis rather than a blind test. The pooled statistic registered for M4 failed its positive controls; the diagnosis, the repair and the exploratory check X all followed that failure, and only M5 is confirmatory for the repair. The repair was selected on synthetic data under a pre-committed rule whose tie-resolution detail was implemented only after the evaluation; on this evidence the Kish-tempered and field-mean statistics are interchangeable. The sealed challenge’s designer had seen the source of the reference models. Every such departure is listed, with dates and hashes, in Appendix § 18 .
Validity rests on assumptions about the design and the measurement. Every guarantee is conditional on the working design law; for ERK it is a reconstruction whose exact acquisition rule is not verified for the confirmatory field, and every CcaS rejection is conditional on a one-step delay, consistent with a shift at data export, that was found after the first design checks failed. We calibrated the working laws on the released command records, but no theorem licenses them (Proposition 5 ). We also assume that the observation mask and segmentation do not respond to future actions. Designs whose logging policy reads past outcomes need the conditional product law of Assumption 2; reading another unit’s actions inside the block can violate that law, so a policy outside the sufficient construction of Proposition 12 (b) needs a separate argument for its conditional law. The own-channel analyses carry a vacuous eligibility bound and are descriptive only.
The repairs have limits of their own. The tests are conservative and have little power for effects that are masked at the level of each unit or that are weak relative to the design’s entropy. Contrast tempering restored power in the pixel world at the larger effect size, but even there without extracting the additional evidence that more pixels carry, so its e-values stayed far below the oracle’s; on published world models it repaired neither DIAMOND nor IRIS’s decoded pixels and failed on two of the six TD-MPC2 tasks, and we do not recommend it. The mixture can lose up to the logarithm of its grid size in log e-value relative to the best grid member, which can matter near the rejection threshold; it was weaker than contrast tempering and validation tuning at the smaller pixel effect, and it missed the power bar in two IRIS games and one TD-MPC2 task in which validation tuning reached it. The weighted mixture closes most of that gap, but trails validation tuning by up to log ( 4 / 3 ) \log(4/3) log ( 4/3 ) when validation tuning is right. Where the pooled score does not collapse, as on robot video, tempering buys little in the typical unit; its value there is the bound on the worst units. The over-betting account (§ 12 ) is exploratory, and the online rules it motivates were weaker than the mixtures in every setting. Its growth guarantee across precision scales also requires grid coverage and the moment-domain conditions of Proposition 8 . Propositions 8 –12 were written after the registrations and checked in several sessions of independent AI review (Appendix § 19 ), not by a human statistician. Proposition 13 was added in this version; its proof is short (Appendix § 16 .§ 13 ) and has had no independent review. The pixel demonstration itself uses one simulator family, two model classes and one training run per cell, at reduced episode counts. Finally, a sharp-null rejection says that some site responds to some future action within the horizon; it does not say which, by how much, or through what mechanism. The authors used Claude (Claude Code and claude.ai) and Codex for code, drafting and review, and they verified all analyses and claims.
Outlook. Three extensions would most strengthen the method: a registered test of the mixture on biological data with weak effects, which the ERK dataset analyzed here cannot provide (§ 8.6 )—randomized intervention experiments with high-dimensional readouts, in which one treatment label is shared by thousands of measured sites, have the same structure; statistics that recover the evidence pooling discards rather than merely discounting it—a joint Gaussian with a low-rank model of the residual covariance did not (§ 9.4 ); and designs that are randomized with these tests in mind, since the design-only bound shows that an experiment’s evidential capacity is fixed before any model is trained. More broadly, wherever world models are evaluated on data that cannot be replayed, the experiment’s own randomization is the only source of counterfactual truth, and it is worth using. The robot-data study also shows what such use requires: for data collected years ago, the design law had to be reconstructed from the command records. Datasets that record how their actions were randomized would make every test of this kind unconditional.
Author Contributions
Bowen Xu: Conceptualization, Methodology, Software, Investigation, Formal analysis, Visualization, Writing – original draft, Writing – review & editing, Project administration.
Boyu Chen: Conceptualization, Methodology, Validation, Writing – review & editing.
Jiaying Li: Writing – review & editing.
BioWM-Eval
Both datasets of § 8 were analyzed earlier in our BioWM-Eval study. This appendix states what that study measured, how its results relate to the tests of this paper, and what its records do and do not establish about its own analysis history. The quotations and figures below are taken from the committed revision package of that study (Appendix E lists the files and hashes).
What it measured. BioWM-Eval defined the utility of future actions as the relative reduction in the best achievable squared prediction error when the future actions are added to the state and the clock, U ∗ ( h ) = [ R 0 ∗ ( h ) − R 1 ∗ ( h ) ] / R 0 ∗ ( h ) U^*(h) = [R_0^*(h) - R_1^*(h)]/R_0^*(h) U ∗ ( h ) = [ R 0 ∗ ( h ) − R 1 ∗ ( h )] / R 0 ∗ ( h ) , and estimated it by comparing held-out risks of matched fitted models with and without the future actions. In its own words, this quantity depends on the information set and does not estimate a causal effect. For ERK it compared matched one-step ridge models with and without the next programmed light pulse, both using the same past values and clock terms, and forecast thirty minutes ahead; for CcaS/CcaR it used seven leave-one-date-out comparisons at the same horizon. The ERK gain on the original minimum-gap fields was 49.6%; the CcaS gain was 0.91%, below the study’s 2% practical cutoff, which was an internal decision rule fixed before the biological analyses and not a public preregistration.
How it relates to this paper. A with-versus-without-actions forecast comparison is the kind of diagnostic whose apparent action effects § 7 shows can arise from stand-in effects. Our rejections therefore neither validate the size of the ERK gain nor say anything about whether the CcaS gain is practically meaningful: the sharp-null tests ask whether any site responds to the future actions, not how much prediction improves. Where the two studies are mentioned together, the juxtaposition is descriptive. The original ERK and CcaS/CcaR studies had already demonstrated that these systems can be predicted or controlled; BioWM-Eval did not claim those biological findings as its own, and neither do we.
What its records establish. BioWM-Eval’s biological analyses were sequential and exploratory; unlike its synthetic challenges, none had sealed inputs or a sealed evaluator. Before the clock-controlled ERK comparison, an earlier comparison without the matched state-and-clock baseline had already been run on the same two held-out experiments, with gains of 56.2% and 57.6%. A rule-freeze record precedes the first clock-controlled result by less than an hour, but it does not name that comparison as primary, and the original project kept no version history; whether the ERK comparison was designated primary before or after its first result therefore cannot be established. The study’s synthetic challenges were sealed and revealed in a recorded order, but the challenge author and the evaluator were operated by the same agent, so its “blind” evaluations reflect an internal sealed-input order, not independent human blinding or public preregistration. These facts are why we treat M4 as a re-analysis (§ 8.1 ) and why no BioWM-Eval figure is used as evidence in this paper.
Proofs
Sources and changes relative to the frozen documents
The obligations P1–P4 of the registered specification are proved in two theory documents that are part of the registered code archive and were therefore fixed before the registration: a core document (P1(a)–(d), P1b, P2(a)–(e), P3, P4) and a calibration supplement (P1(e) and the template-avoidance recursion). A third note in the archive covers averaging across models. Table 11 lists them with their hashes; the files used here are byte-identical to the archived ones.
Table 11. Frozen theory documents. Hashes are SHA-256 of the bytes in the registered code archive.
The proofs below follow these documents. Writing them for the paper, we made the following changes, none of which affects the registered method: [Added] the selection, training-independence and Monte Carlo hypotheses (i)–(iii) of Proposition 1 , which the frozen proofs use implicitly, together with the frozen training and design information in the conditioning fields and the grid filtration; [Tightened] the conditioning field under H 0 o w n H^{\mathrm{own}}_0 H 0 own is written out explicitly, with the targets of units outside the independent set excluded; [Added] the hypotheses of Proposition 2 (b) on the template list and on positive conditioning probability, and the derivation of the product path law from the per-origin design assumption; [Tightened] the approximate-law bound (Proposition 5 ), replacing a coupling argument by a direct total-variation bound conditional on the frozen baseline and defining Δ \Delta Δ on the whole command path; [Tightened] the range of density ratios in Proposition 3 (r ≥ 1 r \ge1 r ≥ 1 , with the reversed case noted) and the conditioning on frozen training information in Proposition 4 (c); [Added] for Proposition 6 , the reading as independent and identically distributed copies at a fixed clock, the restriction of the frozen construction that the predictor’s window contains recent commands and constant outcomes but not the stratum, now stated in the proposition, and a written argument for its probability-one step for ideal predictors. Proposition 7 is new and is proved in § 4.5 .
Propositions 8–12 have no frozen counterpart: they were written after the registrations and checked by Codex, a model family different from the drafting model, in an independent review, a delta review of the revision, a check of the trap experiment’s logging policy, and a final check of the integrated text with a delta check of its revisions; the counterexamples are archived with the reviews (Table 19 ). Their proofs are in §B.8–§B.12.
Table 19. Review records; Rows from the first version are unchanged.
Validity (Proposition 1)
Conditioning fields. Let P \mathcal{P} P be the σ \sigma σ -field generated by all potential-outcome functions, including exogenous noise and the measurement process, T \mathcal{T} T the frozen training information and S \mathcal{S} S the structural design information (strata, layout and fixed allocations); when Proposition 2 (b) is used, let J J J be the eligibility pattern of all units. Put C t G = σ ( F t , P , T , S , J ) \mathcal{C}^{G}_{t} = \sigma(\mathcal{F}_{t}, \mathcal{P}, \mathcal{T}, \mathcal{S}, J) C t G = σ ( F t , P , T , S , J ) and, for an independent set I I I of N \mathcal{N} N , C t I = σ ( F t , P , T , S , J , A j , t F : j ∉ I ) \mathcal{C}^{I}_{t} = \sigma(\mathcal{F}_{t}, \mathcal{P}, \mathcal{T}, \mathcal{S}, J, A^{F}_{j,t}: j \notin I) C t I = σ ( F t , P , T , S , J , A j , t F : j ∈ / I ) , omitting J J J when no eligibility conditioning is used. [Added] By hypothesis (ii), the design law of the joint vector of tested blocks keeps its product form after T \mathcal{T} T and S \mathcal{S} S are adjoined.
Targets are measurable. Under no anticipation and H 0 G ( h ) H^{G}_{0}(h) H 0 G ( h ) , Y i , c , t + h = Y i , c , t + h ( A ) Y_{i,c,t+h}=Y_{i,c,t+h}(A) Y i , c , t + h = Y i , c , t + h ( A ) depends on A A A only through A ≤ t A_{\le t} A ≤ t , so it is measurable with respect to σ ( P , A ≤ t ) = C t G \sigma(\mathcal{P},A_{\le t})=\mathcal{C}^{G}_{t} σ ( P , A ≤ t ) = C t G . Under H 0 G ( h ; N ) H^{G}_{0}(h;N) H 0 G ( h ; N ) and for i ∈ I i\in I i ∈ I : the block has no effect; the blocks of non-neighbours have no effect; and the neighbours of i i i lie outside I I I because I I I is independent. Changing these blocks one at a time, each invariance holding for every value of the other coordinates, shows that Y i , c , t + h Y_{i,c,t+h} Y i , c , t + h is C t I \mathcal{C}^{I}_{t} C t I -measurable. [Tightened] A target of a unit j ∉ I j\notin I j ∈ / I need not be: it may depend on A i , t F A^{F}_{i,t} A i , t F for a neighbour i ∈ I i\in I i ∈ I , which is why the conditioning in Proposition 1 (b) includes only the targets of units in I I I .
The design law is unchanged. Assumption 1 gives Law ( A i , t F ∣ F t , P ) = ∏ i Q i , t \operatorname{Law}(A^{F}_{i,t}\mid\mathcal{F}_{t},\mathcal{P})=\prod_i Q_{i,t} Law ( A i , t F ∣ F t , P ) = ∏ i Q i , t . By hypothesis (ii), adjoining T \mathcal{T} T and S \mathcal{S} S leaves this unchanged; under eligibility conditioning, Q i , t Q_{i,t} Q i , t is replaced by its conditioned version (Proposition 2 (b)). Adjoining the blocks A j , t F A^{F}_{j,t} A j , t F , j ∉ I j\notin I j ∈ / I , leaves ∏ i ∈ I Q i , t \prod_{i\in I}Q_{i,t} ∏ i ∈ I Q i , t as the conditional law of the blocks in I I I , by independence across units. Since the targets in question are measurable with respect to the enlarged field, conditioning on them changes nothing further.
Mean one. By hypotheses (i)–(ii), each T i , t ( ⋅ ) T_{i,t}(\cdot) T i , t ( ⋅ ) is a C t \mathcal{C}_{t} C t -measurable random function of the candidate block, where C t \mathcal{C}_{t} C t is C t G \mathcal{C}^{G}_{t} C t G or C t I \mathcal{C}^{I}_{t} C t I . Hence
E [ e i , t e x ∣ C t ] = ∑ a Q i , t ( a ) e T i , t ( a ) ∑ a Q i , t ( a ) e T i , t ( a ) = 1. \mathbb{E}[e^{\mathrm{ex}}_{i,t}\mid\mathcal{C}_{t}] = \frac{\sum_a Q_{i,t}(a)e^{T_{i,t}(a)}}{\sum_a Q_{i,t}(a)e^{T_{i,t}(a)}} = 1. E [ e i , t ex ∣ C t ] = ∑ a Q i , t ( a ) e T i , t ( a ) ∑ a Q i , t ( a ) e T i , t ( a ) = 1.
For the MC form, by hypothesis (iii) the observed block and the B B B draws are independent and identically distributed with law Q i , t Q_{i,t} Q i , t given C t \mathcal{C}_{t} C t . Their normalised shares e T ( ⋅ ) / ∑ e T ( ⋅ ) e^{T^{(\cdot)}}/\sum e^{T^{(\cdot)}} e T ( ⋅ ) / ∑ e T ( ⋅ ) sum to one and are exchangeable, so the observed share has conditional mean 1 / ( B + 1 ) 1/(B+1) 1/ ( B + 1 ) and e i , t m c e^{\mathrm{mc}}_{i,t} e i , t mc has conditional mean one; ties are immaterial because every score is strictly positive. Conditional independence across units (of blocks, draws) gives the product statements of (a) and (b).
Grid martingale. Fix a residue r r r and its grid t 0 < t 1 < ⋯ t_0 < t_1 < \cdots t 0 < t 1 < ⋯ , t k + 1 = t k + h t_{k+1} = t_k + h t k + 1 = t k + h . Let G k = σ ( F t k , T , S , J , U t 0 , … , U t k − 1 ) \mathcal{G}_k = \sigma(\mathcal{F}_{t_k}, \mathcal{T}, \mathcal{S}, J, U_{t_0}, \ldots, U_{t_{k-1}}) G k = σ ( F t k , T , S , J , U t 0 , … , U t k − 1 ) , where U t j U_{t_j} U t j are the independent auxiliary seeds that generate the Monte Carlo draws at t j t_j t j . The factor X k X_k X k at t k t_k t k is a function of F t k \mathcal{F}_{t_k} F t k -measurable covariates, of the block A ⋅ , t k + 1 : t k + h A_{\cdot,t_{k+1}:t_k+h} A ⋅ , t k + 1 : t k + h and the targets at t k + h t_k+h t k + h (both F t k + 1 \mathcal{F}_{t_{k+1}} F t k + 1 -measurable), and of the draws at t k t_k t k ; so X k X_k X k is G k + 1 \mathcal{G}_{k+1} G k + 1 -measurable. [Tightened] Given the history, the earlier draws are functions of that history and of the earlier seeds, which are independent of the assignments and the potential outcomes; adjoining them leaves the law of the future blocks unchanged, so
E [ X k ∣ G k ] = E [ E [ X k ∣ C t k , earlier seeds ] ∣ G k ] = 1. \mathbb{E}[X_k \mid\mathcal{G}_k] = \mathbb{E}[\mathbb{E}[X_k \mid\mathcal{C}_{t_k},\text{ earlier seeds}] \mid\mathcal{G}_k] = 1. E [ X k ∣ G k ] = E [ E [ X k ∣ C t k , earlier seeds ] ∣ G k ] = 1.
Thus ∏ j < k X j \prod_{j<k} X_j ∏ j < k X j is a non-negative ( G k ) (\mathcal{G}_k) ( G k ) -martingale with initial value one. Ville’s inequality gives Pr ( sup k ∏ j < k X j ≥ 1 / α ) ≤ α \Pr(\sup_k \prod_{j<k} X_j \ge1/\alpha) \le\alpha Pr ( sup k ∏ j < k X j ≥ 1/ α ) ≤ α , and E [ ∏ j < τ X j ] ≤ 1 \mathbb{E}[\prod_{j<\tau} X_j] \le1 E [ ∏ j < τ X j ] ≤ 1 for every ( G k ) (\mathcal{G}_k) ( G k ) -stopping time τ \tau τ , by localization and Fatou’s lemma, with the product on { τ = ∞ } \{\tau= \infty\} { τ = ∞ } read as the almost-sure limit of the non-negative martingale.
Averages and the batch test (part (c)). A prechosen arithmetic average of non-negative random variables with expectation at most one has expectation at most one, without any assumption on their dependence; Markov’s inequality then gives Pr ( E ≥ 1 / α ) ≤ α \Pr(E \ge1/\alpha) \le\alpha Pr ( E ≥ 1/ α ) ≤ α . Averages over independent sets or models of processes adapted to the same filtration are again martingales; processes of different residue classes are adapted to different grids, and on a common clock a class factor is not conditionally mean one once part of its block is revealed, so the class average is a fixed-time e-value only. For the batch test at one common origin, the observed joint block and the joint draws are exchangeable given C t C_t C t (all units under H 0 G H_0^G H 0 G , one independent set under H 0 o w n H_0^{\mathrm{own}} H 0 own ), and the observed-inclusive rank p-value that counts ties as exceedances is super-uniform.
Eligibility (Proposition 2)
Part (a). Let Π ∗ \Pi^* Π ∗ be the pipeline that uses the structural eligibility events and Π ^ \widehat{\Pi} Π the pipeline that uses command-based classifiers, run with the same statistics and Monte Carlo draws. Let D D D be the event that any classifier decision differs from its structural event. On the complement of D D D the two pipelines are identical. Hence, unconditionally,
Pr ( Π ^ rejects ) ≤ Pr ( Π ∗ rejects ) + Pr ( D ) ≤ α + δ , \Pr(\widehat{\Pi}\text{ rejects}) \le\Pr(\Pi^*\text{ rejects}) + \Pr(D) \le\alpha+ \delta, Pr ( Π rejects ) ≤ Pr ( Π ∗ rejects ) + Pr ( D ) ≤ α + δ ,
capped at one; distinct classifier errors may be combined by a union bound. No independence between classifier errors and assignments is needed. The bound does not hold conditionally on an observed selected set.
Part (b): from the per-origin law to the path law. [Added] Let K i , s ( ⋅ ∣ A i , ≤ s ) K_{i,s}(\cdot\mid A_{i,\le s}) K i , s ( ⋅ ∣ A i , ≤ s ) be the one-step kernel of the design, with the fixed stratum suppressed. Assumption 1 at every clock gives the product of these kernels conditionally on the full history, the potential outcomes and the frozen information. Taking conditional expectations onto σ ( F t , P , T , S , A ⋅ , t + 1 : s ) \sigma(\mathcal{F}_t,\mathcal{P},\mathcal{T},\mathcal{S},A_{\cdot,t+1:s}) σ ( F t , P , T , S , A ⋅ , t + 1 : s ) preserves the product, because each kernel is a function only of its own action prefix and of fixed design information, and iterating gives Law ( A ⋅ , t ∣ F t , P , T , S ) = ∏ i Q i p a t h ( ⋅ ∣ A i , ≤ t ) \operatorname{Law}(A_{\cdot,t} \mid\mathcal{F}_t,\mathcal{P},\mathcal{T},\mathcal{S}) = \prod_i Q^{\mathrm{path}}_i(\cdot\mid A_{i,\le t}) Law ( A ⋅ , t ∣ F t , P , T , S ) = ∏ i Q i path ( ⋅ ∣ A i , ≤ t ) : future paths are conditionally independent across units and follow the open-loop design. For an eligibility pattern j = ( j i ) j=(j_i) j = ( j i ) of positive probability, put E i ( 1 ) = E i \mathcal{E}^{(1)}_i=\mathcal{E}_i E i ( 1 ) = E i and E i ( 0 ) = E i c \mathcal{E}^{(0)}_i=\mathcal{E}_i^c E i ( 0 ) = E i c . Conditioning on ⋂ i E i ( j i ) \bigcap_i \mathcal{E}_i^{(j_i)} ⋂ i E i ( j i ) , each event a function of path i i i only, preserves the product; an analysed unit has path law Q i p a t h ( ⋅ ∣ A i , ≤ t , E i ) Q_i^{\mathrm{path}}(\cdot\mid A_{i,\le t},\mathcal{E}_i) Q i path ( ⋅ ∣ A i , ≤ t , E i ) , and an excluded unit contributes the factor one. Its one-block conditionals are
Q i , t ( a ∣ E i , A i , ≤ t ) = Q i , t ( a ) Pr ( E i ∣ A i , ≤ t , a ) Pr ( E i ∣ A i , ≤ t ) , Q_{i,t}(a \mid\mathcal{E}_i,A_{i,\le t}) = \frac{Q_{i,t}(a)\Pr(\mathcal{E}_i \mid A_{i,\le t},a)}{\Pr(\mathcal{E}_i \mid A_{i,\le t})}, Q i , t ( a ∣ E i , A i , ≤ t ) = Pr ( E i ∣ A i , ≤ t ) Q i , t ( a ) Pr ( E i ∣ A i , ≤ t , a ) ,
well defined because the denominator is positive by hypothesis. Under the null, the targets remain measurable as in §B.2, so the argument there applies verbatim under the conditional law given the eligibility pattern ( J (J ( J in C t ) C_t) C t ) . The hypothesis that every unit on which E i \mathcal{E}_i E i holds follows Q i Q_i Q i is what makes the conditioned law the law of the analysed units; a template list that missed a fixed design would violate it.
Part (b): exact computation for a hidden-Markov design. Index the emissions by 0 , … , T − 1 0,\ldots,T-1 0 , … , T − 1 , so that T T T is the terminal clock after the last emission. Let H t H_t H t be the hidden state just before the emission at clock t t t , and let M t M_t M t be the bit mask of which fixed templates still match the emitted prefix (two templates give four mask values). A transition branch j j j from state s s s at clock t t t has destination d t ( s , j ) d_t(s,j) d t ( s , j ) , emission b t ( s , j ) b_t(s,j) b t ( s , j ) and probability w t ( s , j ) w_t(s,j) w t ( s , j ) , and updates the mask to m ∧ match t ( b t ( s , j ) ) m \mathbin{\wedge} \operatorname{match}_t(b_t(s,j)) m ∧ match t ( b t ( s , j )) ; the eligibility event holds iff the final mask is zero. Define β T ( s , m ) = 1 { m = 0 } \beta_T(s,m)=1\{m=0\} β T ( s , m ) = 1 { m = 0 } and
β t ( s , m ) = ∑ j w t ( s , j ) β t + 1 ( d t ( s , j ) , m ∧ match t ( b t ( s , j ) ) ) . \beta_t(s,m)=\sum_j w_t(s,j)\beta_{t+1}\left(d_t(s,j),m\mathbin{\wedge}\operatorname{match}_t(b_t(s,j))\right). β t ( s , m ) = j ∑ w t ( s , j ) β t + 1 ( d t ( s , j ) , m ∧ match t ( b t ( s , j )) ) .
By induction, β t ( s , m ) \beta_t(s,m) β t ( s , m ) is the probability that the remaining path avoids every template, given state s s s and mask m m m . With α t + 1 \alpha_{t+1} α t + 1 the ordinary forward filter conditional on the observed prefix (the current mask m m m is determined by that prefix), the eligibility probability is Z = ∑ s α t + 1 ( s ) β t + 1 ( s , m ) Z=\sum_s\alpha_{t+1}(s)\beta_{t+1}(s,m) Z = ∑ s α t + 1 ( s ) β t + 1 ( s , m ) . For a candidate block a a a , propagate the unnormalised filter through a a a , update the mask, and take the inner product with β t + h + 1 \beta_{t+h+1} β t + h + 1 ; dividing by Z Z Z gives Q i , t ( a ∣ E i , A i , ≤ t ) Q_{i,t}(a\mid\mathcal{E}_i,A_{i,\leq t}) Q i , t ( a ∣ E i , A i , ≤ t ) . Summing over all blocks recovers Z Z Z by the backward recursion, which proves normalisation. For exact sampling, draw the initial state with mass proportional to α β \alpha\beta α β and each branch with mass proportional to w t β t + 1 w_t\beta_{t+1} w t β t + 1 at its updated destination and mask; the β \beta β factors telescope, leaving the full-path mass times 1 { E i } / Z 1\{\mathcal{E}_i\}/Z 1 { E i } / Z . Once the mask is zero, β ≡ 1 \beta\equiv1 β ≡ 1 and the ordinary kernel applies. The sampling transition is used only at states with positive β \beta β ; states with β = 0 \beta=0 β = 0 have probability zero under the conditioned sampler.
Distributional nulls do not compose (Proposition 3)
With design law ( 1 2 , 1 2 ) \left(\frac{1}{2},\frac{1}{2}\right) ( 2 1 , 2 1 ) and a score whose density ratio between the matching and the non-matching bit is r r r , the exact factor is 2 r / ( r + 1 ) 2r/(r+1) 2 r / ( r + 1 ) for a match and 2 / ( r + 1 ) 2/(r+1) 2/ ( r + 1 ) otherwise; r = 9 r=9 r = 9 gives 9 / 5 9/5 9/5 and 1 / 5 1/5 1/5 . In system (ii), unit i i i matches iff U k = 0 U_k=0 U k = 0 , so at each origin both units match or both fail, each with probability 1 2 \frac{1}{2} 2 1 . In system (i), unit 1 matches iff A 1 , k = U k A_{1,k}=U_k A 1 , k = U k and unit 2 matches iff A 2 , k = U k ⊕ A 1 , k ⊕ A 2 , k A_{2,k}=U_k\oplus A_{1,k}\oplus A_{2,k} A 2 , k = U k ⊕ A 1 , k ⊕ A 2 , k , that is iff A 1 , k = U k A_{1,k}=U_k A 1 , k = U k ; again both match or both fail with probability 1 2 \frac{1}{2} 2 1 . The per-origin product therefore has mean 1 2 [ ( 9 / 5 ) 2 + ( 1 / 5 ) 2 ] = 1.64 \frac{1}{2}\left[(9/5)^2+(1/5)^2\right]=1.64 2 1 [ ( 9/5 ) 2 + ( 1/5 ) 2 ] = 1.64 , and the product over three independent origins has mean 1.64 3 ≈ 4.41 1.64^3\approx4.41 1.6 4 3 ≈ 4.41 . Because ( 9 / 5 ) 6 ≥ 20 (9/5)^6\geq20 ( 9/5 ) 6 ≥ 20 while any product containing a factor 1 / 5 1/5 1/5 is below 20, rejection at α = 0.05 \alpha=0.05 α = 0.05 occurs iff both units match at all three origins, with probability 1 / 8 1/8 1/8 . [Tightened] Among ratios r ≥ 1 r\geq1 r ≥ 1 the same holds exactly when ( 2 r / ( r + 1 ) ) 6 ≥ 20 (2r/(r+1))^6\geq20 ( 2 r / ( r + 1 ) ) 6 ≥ 20 , that is r ≥ 20 1 / 6 / ( 2 − 20 1 / 6 ) ≈ 4.675 r\geq20^{1/6}/(2-20^{1/6})\approx4.675 r ≥ 2 0 1/6 / ( 2 − 2 0 1/6 ) ≈ 4.675 , and for 1 ≤ r < 4.675 1\leq r<4.675 1 ≤ r < 4.675 there is rejection only if all three origins are zero: a product that mixes matching and non-matching origins is at most 64 r 4 / ( 1 + r ) 6 ≤ 1024 / 729 < 20 64r^4/(1+r)^6\leq1024/729<20 64 r 4 / ( 1 + r ) 6 ≤ 1024/729 < 20 . For r < 1 r<1 r < 1 the roles reverse, and for r ≤ ( 2 − 20 1 / 6 ) / 20 1 / 6 r\leq(2-20^{1/6})/20^{1/6} r ≤ ( 2 − 2 0 1/6 ) /2 0 1/6 rejection occurs when all three origins mismatch, again with probability 1 / 8 1/8 1/8 . Each unit-wise null holds because U k U_k U k is a fresh fair bit: in (ii), Y i , k = A i , k ⊕ U k Y_{i,k}=A_{i,k}\oplus U_k Y i , k = A i , k ⊕ U k is independent of A i , k A_{i,k} A i , k ; in (i), Y 1 , k = U k Y_{1,k}=U_k Y 1 , k = U k and Y 2 , k Y_{2,k} Y 2 , k is a fair bit independent of A 2 , k A_{2,k} A 2 , k given A 1 , k A_{1,k} A 1 , k . If U U U were held fixed across origins, earlier outcomes would reveal it, and after the first origin the unit-wise null would fail for unit 2 in system (i) and for both units in system (ii). Both systems violate the sharp nulls, because the outcomes depend on the actions within the horizon.
Design limits (Proposition 4)
Parts (a) and (b). The exact denominator includes the term Q i , t ( A i , t F ) e T ( A i , t F ) Q_{i,t}(A^F_{i,t})e^{T(A^F_{i,t})} Q i , t ( A i , t F ) e T ( A i , t F ) , so e i , t e x ≤ 1 / Q i , t ( A i , t F ) e^{\mathrm{ex}}_{i,t}\leq1/Q_{i,t}(A^F_{i,t}) e i , t ex ≤ 1/ Q i , t ( A i , t F ) ; a point-mass law gives e = 1 e=1 e = 1 . If K K K of the B B B draws equal the observed block, the MC denominator contains at least ( K / T ) e T ( A i , t F ) (K/T)e^{T(A^F_{i,t})} ( K / T ) e T ( A i , t F ) , so e i , t m c ≤ ( B + 1 ) / ( 1 + K ) e^{\mathrm{mc}}_{i,t}\leq(B+1)/(1+K) e i , t mc ≤ ( B + 1 ) / ( 1 + K ) , and given the commands K ∼ Binomial ( B , Q i , t ( A i , t F ) ) K\sim\operatorname{Binomial}(B,Q_{i,t}(A^F_{i,t})) K ∼ Binomial ( B , Q i , t ( A i , t F )) . Within a product component, log E \log E log E is therefore at most the sum of the corresponding per-factor log bounds. An arithmetic average of components can reach 1 / α 1/\alpha 1/ α only if some component does. Under any alternative the actions still follow the design law, so integrating this necessary event over coherent full command paths drawn from Q Q Q (and, for MC, over independent draws) bounds the rejection probability. Because averaging models adds no design entropy, the same command-only bound applies to the model average; for the Monte Carlo form this presumes that the models share their candidate bags, and otherwise the maximum runs over models with their own collision counts. The MC ceiling differs from the exact one: for a fair bit with B = 20 B=20 B = 20 and a matching-score ratio of 10 6 10^6 1 0 6 , the MC test rejects only when no draw matches, with probability 2 − 20 2^{-20} 2 − 20 , while the exact-form ceiling at α = 0.05 \alpha= 0.05 α = 0.05 is zero. The exact-form ceiling therefore bounds exact e e e -values only; for a Monte Carlo analysis it is a comparator.
Part (c). [Tightened] Fix F t \mathcal{F}_t F t and the frozen training information T \mathcal{T} T , on which the score s y s_y s y depends; let P y = Law ( A F ∣ Y = y , F t , T ) P_y = \operatorname{Law}(A^F \mid Y = y, \mathcal{F}_t, \mathcal{T}) P y = Law ( A F ∣ Y = y , F t , T ) , Z y = ∑ a Q ( a ) s y ( a ) Z_y = \sum_a Q(a)s_y(a) Z y = ∑ a Q ( a ) s y ( a ) and R y ( a ) = Q ( a ) s y ( a ) / Z y R_y(a) = Q(a)s_y(a)/Z_y R y ( a ) = Q ( a ) s y ( a ) / Z y . Then log e e x = log R Y ( A F ) − log Q ( A F ) \log e^{\mathrm{ex}} = \log R_Y(A^F) - \log Q(A^F) log e ex = log R Y ( A F ) − log Q ( A F ) , and taking expectations over A F ∼ P y A^F \sim P_y A F ∼ P y and then over Y Y Y ,
E [ log e e x ∣ F t , T ] = E Y [ KL ( P Y ∥ Q ) − KL ( P Y ∥ R Y ) ] ≤ E Y KL ( P Y ∥ Q ) , \mathbb{E}[\log e^{\mathrm{ex}} \mid\mathcal{F}_t,\mathcal{T}] = \mathbb{E}_Y[\operatorname{KL}(P_Y\|Q) - \operatorname{KL}(P_Y\|R_Y)] \le\mathbb{E}_Y\operatorname{KL}(P_Y\|Q), E [ log e ex ∣ F t , T ] = E Y [ KL ( P Y ∥ Q ) − KL ( P Y ∥ R Y )] ≤ E Y KL ( P Y ∥ Q ) ,
and E Y KL ( P Y ∥ Q ) = I ( Y ; A F ∣ F t , T ) ≤ H Q ( A F ∣ F t ) \mathbb{E}_Y\operatorname{KL}(P_Y\|Q) = I(Y; A^F \mid\mathcal{F}_t,\mathcal{T}) \le H_Q(A^F \mid\mathcal{F}_t) E Y KL ( P Y ∥ Q ) = I ( Y ; A F ∣ F t , T ) ≤ H Q ( A F ∣ F t ) . The equality uses that A F ∼ Q A^F \sim Q A F ∼ Q given ( F t , T ) (\mathcal{F}_t,\mathcal{T}) ( F t , T ) under Assumption 1 and hypothesis (ii), whatever the alternative. Without T \mathcal{T} T in the conditioning the bound can fail, since a score fitted on training data may carry information about the targets that F t \mathcal{F}_t F t does not. The floor guarantees that R y R_y R y charges the support of Q Q Q . This is a growth statement, not a power threshold.
Part (d). The exact weighted-tail p p p -value includes the observed joint block’s mass, so p ≥ Q j o i n t ( A o b s F ) p \ge Q^{\mathrm{joint}}(A^F_{\mathrm{obs}}) p ≥ Q joint ( A obs F ) ; in the add-one MC form, K K K matching draws give p ≥ ( 1 + K ) / ( B + 1 ) p \ge(1 + K)/(B + 1) p ≥ ( 1 + K ) / ( B + 1 ) .
Approximate design laws (Proposition 5)
[Tightened] Let B \mathcal{B} B contain the pre-test history and the frozen design, training and working-law information. Conditionally on B = b \mathcal{B} = b B = b , let q b q_b q b and q ^ b \hat{q}_b q ^ b be the laws of the command path under the true design Q Q Q and under the working law Q ^ \hat{Q} Q ^ , and let ν b \nu_b ν b be the common conditional law of the potential outcomes and the Monte Carlo seeds, which under Assumption 1 are independent of the path under both laws. Let P 0 P_0 P 0 be the reference experiment in which the path follows Q ^ \hat{Q} Q ^ and everything else is unchanged. Because Q ^ \hat{Q} Q ^ is fixed independently of the tested blocks and satisfies the requirements of Assumption 1, Proposition 1 applies to P 0 P_0 P 0 conditionally on B \mathcal{B} B , so P 0 ( E ≥ 1 / α ∣ b ) ≤ α P_0(E \ge1/\alpha\mid b) \le\alpha P 0 ( E ≥ 1/ α ∣ b ) ≤ α and E 0 [ E ∣ b ] ≤ 1 \mathbb{E}_0[E \mid b] \le1 E 0 [ E ∣ b ] ≤ 1 . The same measurable analysis map produces E E E in both experiments, and d T V ( q b ⊗ ν b , q ^ b ⊗ ν b ) = Δ ( b ) d_{\mathrm{TV}}(q_b \otimes\nu_b, \hat{q}_b \otimes\nu_b) = \Delta(b) d TV ( q b ⊗ ν b , q ^ b ⊗ ν b ) = Δ ( b ) . Hence P ( E ≥ 1 / α ∣ b ) ≤ α + Δ ( b ) P(E \ge1/\alpha\mid b) \le\alpha+ \Delta(b) P ( E ≥ 1/ α ∣ b ) ≤ α + Δ ( b ) and, for 0 ≤ E ≤ M 0 \le E \le M 0 ≤ E ≤ M with a constant M M M , E [ E ∣ b ] ≤ 1 + M Δ ( b ) \mathbb{E}[E \mid b] \le1 + M\Delta(b) E [ E ∣ b ] ≤ 1 + M Δ ( b ) . Integrating over B \mathcal{B} B gives the stated bounds with E [ Δ ] \mathbb{E}[\Delta] E [ Δ ] ; a baseline-dependent bound M ( B ) M(\mathcal{B}) M ( B ) gives 1 + E [ M ( B ) Δ ( B ) ] 1 + \mathbb{E}[M(\mathcal{B})\Delta(\mathcal{B})] 1 + E [ M ( B ) Δ ( B )] instead. For sequential kernels,
Δ ( b ) ≤ min { 1 , E Q [ ∑ s d T V ( K s ( ⋅ ∣ A < s , b ) , K ^ s ( ⋅ ∣ A < s , b ) ) | B = b ] } , \Delta(b) \le\min\left\{1,\mathbb{E}_Q\left[\sum_s d_{\mathrm{TV}}\left(K_s(\cdot\mid A_{<s},b),\hat{K}_s(\cdot\mid A_{<s},b)\right)\middle|\mathcal{B}=b\right]\right\}, Δ ( b ) ≤ min { 1 , E Q [ s ∑ d TV ( K s ( ⋅ ∣ A < s , b ) , K ^ s ( ⋅ ∣ A < s , b ) ) B = b ] } ,
and for product kernels each one-step distance is at most the sum of the unit distances. The frozen document derived the moment bound and, separately, the event bound via a coupling; the direct argument here gives the event bound without a coupling and without a bound on E E E .
Failure of replacement diagnostics (Proposition 6)
Positivity of the gaps. [Added] Write Y = θ + η Y = \theta+ \eta Y = θ + η with η \eta η independent centred noise of finite variance, so μ ( W , F ) = E [ θ ∣ W , F ] \mu(W,F) = \mathbb{E}[\theta\mid W,F] μ ( W , F ) = E [ θ ∣ W , F ] and μ 0 ( W ) = E [ θ ∣ W ] \mu_0(W) = \mathbb{E}[\theta\mid W] μ 0 ( W ) = E [ θ ∣ W ] . As in the frozen construction, W W W consists of finitely many recent commands, the recent outcomes, which are constant, and the clock, and θ \theta θ enters only an earlier outcome outside the window. [Tightened] This restriction is needed: if the recent outcomes revealed θ \theta θ , then μ ( W , F ) = μ 0 ( W ) = θ \mu(W,F) = \mu_0(W) = \theta μ ( W , F ) = μ 0 ( W ) = θ and all three gaps would be zero. Under it, every finite command window has positive probability under both strata, because the chain is stationary with transition probabilities in ( 0 , 1 ) (0,1) ( 0 , 1 ) , so 0 < Pr ( θ = 1 ∣ W ) < 1 0 < \Pr(\theta= 1 \mid W) < 1 0 < Pr ( θ = 1 ∣ W ) < 1 . Given the last command in W W W , the first future command has stratum-specific probabilities that differ by ( 1 − ρ ) ( p 1 − p 0 ) ≠ 0 (1-\rho)(p_1-p_0) \ne0 ( 1 − ρ ) ( p 1 − p 0 ) = 0 , so the likelihood ratio of F F F given W W W between θ = 1 \theta= 1 θ = 1 and θ = 0 \theta= 0 θ = 0 is non-constant and μ ( W , ⋅ ) \mu(W,\cdot) μ ( W , ⋅ ) is non-constant on a set of windows of positive probability. (i) Hence E [ ( μ − μ 0 ) 2 ] > 0 \mathbb{E}[(\mu-\mu_0)^2] > 0 E [( μ − μ 0 ) 2 ] > 0 ; if the window’s commands are removed as well, the retrained predictor is the constant Pr ( θ = 1 ) \Pr(\theta= 1) Pr ( θ = 1 ) and the gap is Var ( μ ) > 0 \operatorname{Var}(\mu) > 0 Var ( μ ) > 0 . (ii) Every future block, including the zero block, has positive conditional probability given W W W , so μ ( W , 0 ) \mu(W,0) μ ( W , 0 ) is defined and E [ ( μ ( W , F ) − μ ( W , 0 ) ) 2 ] > 0 \mathbb{E}[(\mu(W,F)-\mu(W,0))^2] > 0 E [( μ ( W , F ) − μ ( W , 0 ) ) 2 ] > 0 . (iii) With F ′ F' F ′ an independent unit’s block, E [ ( Y − μ ( W , F ′ ) ) 2 ] − E [ ( Y − μ ( W , F ) ) 2 ] = E [ ( μ ( W , F ) − μ ( W , F ′ ) ) 2 ] > 0 \mathbb{E}[(Y-\mu(W,F'))^2]-\mathbb{E}[(Y-\mu(W,F))^2] = \mathbb{E}[(\mu(W,F)-\mu(W,F'))^2] > 0 E [( Y − μ ( W , F ′ ) ) 2 ] − E [( Y − μ ( W , F ) ) 2 ] = E [( μ ( W , F ) − μ ( W , F ′ ) ) 2 ] > 0 ; the cross term vanishes because E [ Y − μ ( W , F ) ∣ W , F , F ′ ] = 0 \mathbb{E}[Y-\mu(W,F)\mid W,F,F'] = 0 E [ Y − μ ( W , F ) ∣ W , F , F ′ ] = 0 when F ′ F' F ′ is independent of ( Y , W , F ) (Y,W,F) ( Y , W , F ) . Under squared-error loss, each population gap equals the corresponding expected squared difference of predictors.
Probability tending to one. [Added] For the ideal predictors the per-unit loss differences D i = ( Y i − g i ) 2 − ( Y i − μ i ) 2 D_i = (Y_i-g_i)^2-(Y_i-\mu_i)^2 D i = ( Y i − g i ) 2 − ( Y i − μ i ) 2 are independent and identically distributed (over fixed pairs for the swap diagnostic), with finite variance because μ , g ∈ [ 0 , 1 ] \mu,g \in[0,1] μ , g ∈ [ 0 , 1 ] and the noise has finite variance, and with mean equal to the population gap γ > 0 \gamma> 0 γ > 0 . By the law of large numbers D ˉ N → γ \bar{D}_N \to\gamma D ˉ N → γ in probability, and by consistency of the nonparametric bootstrap for the mean the lower bootstrap quantile is D ˉ N + O P ( N − 1 / 2 ) \bar{D}_N+O_P(N^{-1/2}) D ˉ N + O P ( N − 1/2 ) . Hence the probability that the lower interval bound exceeds zero tends to one. As in the frozen document, this is a statement about the ideal predictors. It is not claimed for fitted predictors, which would need, in addition, fitting on independent units and convergence in L 2 L^2 L 2 under both the observed and the replacement input laws.
Validity of the randomization test. The design law Q ( ⋅ ∣ A i , t , θ i ) Q(\cdot\mid A_{i,t},\theta_i) Q ( ⋅ ∣ A i , t , θ i ) is known to the design, the outcome does not depend on any action, and the statistic uses only admissible inputs, so Proposition 1 applies: resampling from the design law preserves the dependence between the future block and θ i \theta_i θ i , and the size is at most α \alpha α .
Growth bound (Proposition 8)
Write K ( λ ) = log E Q e − λ Δ S K(\lambda)=\log\mathbb{E}_Q e^{-\lambda\Delta S} K ( λ ) = log E Q e − λ Δ S for one origin, dropping indices, so that log e e x ( λ ) = − K ( λ ) \log e^{\mathrm{ex}}(\lambda)=-K(\lambda) log e ex ( λ ) = − K ( λ ) . K K K is a log Laplace transform, hence convex on the interval where it is finite; integrability of Δ S \Delta S Δ S and a finite right exponential moment give K ′ ( 0 + ) = − μ K'(0+)=-\mu K ′ ( 0 + ) = − μ . Centering, K ( λ ) = − λ μ + log E Q e − λ ( Δ S − μ ) ≤ − λ μ + λ 2 ν / 2 K(\lambda)=-\lambda\mu+\log\mathbb{E}_Qe^{-\lambda(\Delta S-\mu)}\leq-\lambda\mu+\lambda^2\nu/2 K ( λ ) = − λ μ + log E Q e − λ ( Δ S − μ ) ≤ − λ μ + λ 2 ν /2 on [ 0 , Λ ] [0,\Lambda] [ 0 , Λ ] , which is the per-origin bound; summing over the origins of one component gives f f f , and completing the square gives the optimizer when V > 0 V>0 V > 0 (if V = 0 V=0 V = 0 the bound is linear). Hoeffding’s lemma applies to bounded Δ S \Delta S Δ S . For an MC factor, log e m c ( λ ) = − log E ^ Q e − λ Δ S \log e^{\mathrm{mc}}(\lambda)=-\log\widehat{\mathbb{E}}_Qe^{-\lambda\Delta S} log e mc ( λ ) = − log E Q e − λ Δ S with Q ^ \widehat Q Q the empirical law of the observed-inclusive bag, so the same algebra holds with Q ^ \widehat Q Q -moments; a Q Q Q -based bound does not transfer to every sampled denominator. Under the null, Proposition 1 gives E [ e ∣ C t ] = 1 \mathbb{E}[e\mid C_t]=1 E [ e ∣ C t ] = 1 , so Jensen gives E [ log e ∣ C t ] ≤ 0 \mathbb{E}[\log e\mid C_t]\leq0 E [ log e ∣ C t ] ≤ 0 ; exchangeability gives E [ μ ∣ C t ] = 0 \mathbb{E}[\mu\mid C_t]=0 E [ μ ∣ C t ] = 0 when integrable, and taking expectations of the per-origin bound, within a common deterministic moment domain, gives the lower bound − λ 2 E [ V ] / 2 -\lambda^2\mathbb{E}[V]/2 − λ 2 E [ V ] /2 . On a single path the term λ μ \lambda\mu λ μ need not vanish: for a fair binary design with scores ± 1 \pm1 ± 1 , the low-score path has log e = − λ − log cosh λ \log e=-\lambda-\log\cosh\lambda log e = − λ − log cosh λ . A positive expected lower bound needs E [ M ] > 0 \mathbb{E}[M]>0 E [ M ] > 0 , 0 < E [ V ] < ∞ 0<\mathbb{E}[V]<\infty 0 < E [ V ] < ∞ and 0 < λ < 2 E [ M ] / E [ V ] 0<\lambda<2\mathbb{E}[M]/\mathbb{E}[V] 0 < λ < 2 E [ M ] / E [ V ] within the domain; if E [ V ] = 0 \mathbb{E}[V]=0 E [ V ] = 0 the linear bound λ E [ M ] \lambda\mathbb{E}[M] λ E [ M ] applies.
Precision rescaling (Proposition 9)
For an unfloored Gaussian site score, ℓ ( a ) = − 1 2 log ( 2 π ν / γ ) − γ ( Y − μ ( a ) ) 2 / ( 2 ν ) \ell(a)=-\frac{1}{2}\log(2\pi\nu/\gamma)-\gamma(Y-\mu(a))^2/(2\nu) ℓ ( a ) = − 2 1 log ( 2 π ν / γ ) − γ ( Y − μ ( a ) ) 2 / ( 2 ν ) , so the candidate-dependent part scales by γ \gamma γ and the rest, C ( γ ) C(\gamma) C ( γ ) , is common to all candidates and cancels in the exact and the same-bag MC normalizations. Hence λ S ( γ ) = ( λ γ ) S ( 1 ) + λ C ( γ ) \lambda S^{(\gamma)}=(\lambda\gamma)S^{(1)}+\lambda C(\gamma) λ S ( γ ) = ( λγ ) S ( 1 ) + λ C ( γ ) and (a) follows; if an unrestricted finite maximizer of the actual curve exists, the maximizer set scales by 1 / γ 1/\gamma 1/ γ and the maximal value is unchanged. For (b), z = r D / ν z=rD/\nu z = r D / ν becomes γ z \gamma z γ z (and κ z \kappa z κ z under D ↦ κ D D\mapsto\kappa D D ↦ κ D ), which leaves the ratio (8) and fixed clipping endpoints unchanged; correlation coefficients of nonconstant series are scale-free. (c) combines (a) with (5).
Floored scores. With the floor ℓ min = log 10 − 12 \ell_{\min}=\log10^{-12} ℓ m i n = log 1 0 − 12 write L γ , c ( a ) = C γ , c − γ R c ( a ) L_{\gamma,c}(a)=C_{\gamma,c}-\gamma R_c(a) L γ , c ( a ) = C γ , c − γ R c ( a ) , C γ , c = − 1 2 log ( 2 π ν / γ ) C_{\gamma,c}=-\frac{1}{2}\log(2\pi\nu/\gamma) C γ , c = − 2 1 log ( 2 π ν / γ ) , R c ( a ) = ( Y c − μ c ( a ) ) 2 / ( 2 ν c ) R_c(a)=(Y_c-\mu_c(a))^2/(2\nu_c) R c ( a ) = ( Y c − μ c ( a ) ) 2 / ( 2 ν c ) , and d γ ( a ) = ∑ c { [ ℓ min − L γ , c ( a ) ] + } γ d_\gamma(a)=\sum_c\{[\ell_{\min}-L_{\gamma,c}(a)]+\}_\gamma d γ ( a ) = ∑ c {[ ℓ m i n − L γ , c ( a )] + } γ . Then S f l o o r ( a ) = γ S 1 f l o o r ( a ) + C γ + d γ ( a ) S^{\mathrm{floor}}(a)=\gamma S^{\mathrm{floor}}_1(a)+C_\gamma+d_\gamma(a) S floor ( a ) = γ S 1 floor ( a ) + C γ + d γ ( a ) with C γ = ∑ c ( C γ , c − γ C 1 , c ) C_\gamma=\sum_c(C_{\gamma,c}-\gamma C_{1,c}) C γ = ∑ c ( C γ , c − γ C 1 , c ) independent of a a a , and bounding the weighted exponential average of d γ d_\gamma d γ between its extremes gives ∣ log e f l o o r ( λ ) − log e f l o o r ( γ λ ) ∣ ≤ λ osc ( d γ ) |\log e^{\mathrm{floor}}(\lambda)-\log e^{\mathrm{floor}}(\gamma\lambda)|\leq\lambda\operatorname{osc}(d_\gamma) ∣ log e floor ( λ ) − log e floor ( γ λ ) ∣ ≤ λ osc ( d γ ) , the oscillation taken over the relevant support including the observed block (or over the observed-inclusive MC bag); sum along a component. The bound reduces to (a) when no floor is active at either scale. At each site the floored score range is at most max ( 0 , C γ , c − ℓ min ) \max(0,C_{\gamma,c}-\ell_{\min}) max ( 0 , C γ , c − ℓ m i n ) . For a finite candidate set containing the observed block in which every candidate has a nonzero residual at every scored site, all site scores eventually reach the floor together and the factor equals one for all large γ \gamma γ ; this rules out an unconditional collapse linear in γ \gamma γ for floored scores, but it does not give a universal limit e → 1 e\to1 e → 1 (a candidate with a zero residual does not floor).
Tempering mixture (Proposition 10)
(a) is a fixed-weight average of e-values. (b) keeps one term of the sum. For (c), if λ q u a d \lambda_{\mathrm{quad}} λ quad is a grid point use it; otherwise it lies strictly between adjacent grid points λ g < λ q u a d < λ g − 1 \lambda_g < \lambda_{\mathrm{quad}} < \lambda_{g-1} λ g < λ quad < λ g − 1 , and with c = λ g / λ q u a d ∈ ( 1 / 2 , 1 ) c = \lambda_g/\lambda_{\mathrm{quad}} \in(1/2,1) c = λ g / λ quad ∈ ( 1/2 , 1 ) , f ( c λ q u a d ) = ( 2 c − c 2 ) f ( λ q u a d ) f(c\lambda_{\mathrm{quad}}) = (2c-c^2)f(\lambda_{\mathrm{quad}}) f ( c λ quad ) = ( 2 c − c 2 ) f ( λ quad ) and f ( 2 c λ q u a d ) = ( 4 c − 4 c 2 ) f ( λ q u a d ) f(2c\lambda_{\mathrm{quad}}) = (4c-4c^2)f(\lambda_{\mathrm{quad}}) f ( 2 c λ quad ) = ( 4 c − 4 c 2 ) f ( λ quad ) , whose maximum is at least 8 / 9 8/9 8/9 of f ( λ q u a d ) f(\lambda_{\mathrm{quad}}) f ( λ quad ) , with equality at c = 2 / 3 c = 2/3 c = 2/3 . If M / V > λ 0 M/V > \lambda_0 M / V > λ 0 , then λ 0 V < M \lambda_0V < M λ 0 V < M and f ( λ 0 ) ≥ λ 0 M / 2 f(\lambda_0) \ge\lambda_0M/2 f ( λ 0 ) ≥ λ 0 M /2 . For (d), mixing and averaging over classes are both linear, so E M = R − 1 ∑ r E r M E^M = R^{-1}\sum_r E_r^M E M = R − 1 ∑ r E r M , and the logarithm of a sum of positive terms gives the bound; a class of fixed weight w r w_r w r contributes log w r \log w_r log w r . Alternatively, if every classwise quadratic bound holds on the grid, Jensen gives log E ( λ ) ≥ λ M ˉ − λ 2 V ˉ / 2 \log E(\lambda) \ge\lambda\bar{M} - \lambda^2\bar{V}/2 log E ( λ ) ≥ λ M ˉ − λ 2 V ˉ /2 with the class averages, to which (c) applies; summed M \mathcal{M} M and V \mathcal{V} V must not be used as if all origins formed one product. In the setting of Proposition 9 , the scaled proxies give the bound of Proposition 8 on [ 0 , Λ / γ ] [0,\Lambda/\gamma] [ 0 , Λ/ γ ] , so (c) holds for every γ ∈ [ λ q u a d / λ 0 , λ q u a d / λ G − 1 ] \gamma\in[\lambda_{\mathrm{quad}}/\lambda_0,\lambda_{\mathrm{quad}}/\lambda_{G-1}] γ ∈ [ λ quad / λ 0 , λ quad / λ G − 1 ] for which the comparison grid points lie in that domain (for instance when γ λ 0 ≤ Λ \gamma\lambda_0 \le\Lambda γ λ 0 ≤ Λ ). The null statements follow from (b) and Proposition 8 at λ G − 1 \lambda_{G-1} λ G − 1 .
Rank calibrators (Proposition 11)
Jensen gives c g ≤ log ( N c a n d − 1 ∑ k g ( k ) ) = 0 c_g \le\log(N_{\mathrm{cand}}^{-1}\sum_k g(k)) = 0 c g ≤ log ( N cand − 1 ∑ k g ( k )) = 0 , strictly unless g g g is constant. For any function h h h with range r r r , ∣ E p h − E u h ∣ ≤ TV ( p , u ) r |\mathbb{E}_p h-\mathbb{E}_u h| \le\operatorname{TV}(p,u)r ∣ E p h − E u h ∣ ≤ TV ( p , u ) r ; apply it to h = log g h=\log g h = log g and sum expectations over origins. Tie averaging. If the observed score occupies the positions J J J of the sorted bag, the implemented factor is g ˉ J = ∣ J ∣ − 1 ∑ k ∈ J g ( k ) \bar{g}_J=|J|^{-1}\sum_{k\in J}g(k) g ˉ J = ∣ J ∣ − 1 ∑ k ∈ J g ( k ) . For a fixed bag, assign each candidate label the average of g g g over the positions of its tie block; these values sum to ∑ k g ( k ) = N c a n d \sum_k g(k)=N_{\mathrm{cand}} ∑ k g ( k ) = N cand , so conditional exchangeability of the observed label gives mean one regardless of ties, and products follow as in Proposition 1 or Proposition 12 . Conditioning on F t \mathcal{F}_t F t and the unordered bag B \mathcal{B} B , with h B ( k ) = log g ˉ J ( k ) h_{\mathcal{B}}(k)=\log\bar{g}_J(k) h B ( k ) = log g ˉ J ( k ) , c B = N c a n d − 1 ∑ k h B ( k ) c_{\mathcal{B}}=N_{\mathrm{cand}}^{-1}\sum_k h_{\mathcal{B}}(k) c B = N cand − 1 ∑ k h B ( k ) and r B r_{\mathcal{B}} r B the range of h B h_{\mathcal{B}} h B , the same argument gives E [ log e R ∣ F t , B ] ≤ c B + TV ( p B , u ) r B \mathbb{E}[\log e^R\mid\mathcal{F}_t,\mathcal{B}] \le c_{\mathcal{B}}+\operatorname{TV}(p_{\mathcal{B}},u)r_{\mathcal{B}} E [ log e R ∣ F t , B ] ≤ c B + TV ( p B , u ) r B , where p B p_{\mathcal{B}} p B is the law of the observed position with uniform randomization inside its tie block; under the null p B p_{\mathcal{B}} p B is uniform, and Jensen gives c g ≤ c B ≤ 0 c_g \le c_{\mathcal{B}} \le0 c g ≤ c B ≤ 0 , with c B = 0 c_{\mathcal{B}}=0 c B = 0 when all candidates tie.
Delayed feedback (Proposition 12)
(b) Since s − L ≤ t s-L \le t s − L ≤ t for every s ∈ { t + 1 , … , t + h } s\in\{t+1,\ldots,t+h\} s ∈ { t + 1 , … , t + h } , every history input of the policies is fixed given F t \mathcal{F}_t F t , and recursion on s s s makes each A i , t i A^i_{i,t} A i , t i a function of that history and of unit i i i ’s own innovations U i , t + 1 : t + h i U^i_{i,t+1:t+h} U i , t + 1 : t + h i . Independence of these innovation vectors, and their independence of F t \mathcal{F}_t F t and the potential outcomes, gives the product law of Assumption 2 given F t \mathcal{F}_t F t and P \mathcal{P} P ; because no policy reads another unit’s actions inside the block, the same product holds for the tested set given the outside blocks.
(a) Under H 0 G H^G_0 H 0 G the targets are functions of P \mathcal{P} P and the pre-block actions, and under H 0 o w n H^{\mathrm{own}}_0 H 0 own with the conditional factorization the targets of the tested set are fixed given C t l C^l_t C t l , as in §B.2; conditioning on them leaves the law unchanged, and the denominator identity, Monte Carlo exchangeability and grid measurability arguments of §B.2 apply verbatim. Eligibility conditioning must first be shown to preserve the asserted conditional law; the open-loop path factorization of Proposition 2 (b) is not inferred from the delayed-policy construction. Two counterexamples in the review archive show the conditions are needed: when one unit’s policy copies another unit’s earlier action inside the block, the marginal factors each have mean one but their product has mean 1.64; and when an outside policy carries a tested unit’s within-block action into its target through a spillover channel, an autonomous tested policy is rejected with probability one under a true own-unit null.
Over-betting (Proposition 13)
(a) For fixed numbers s 0 , … , s B s_0,\ldots,s_B s 0 , … , s B , the function κ ( λ ) = log { ( B + 1 ) − 1 ∑ b e λ s b } \kappa(\lambda)=\log\{(B+1)^{-1}\sum_b e^{\lambda s_b}\} κ ( λ ) = log {( B + 1 ) − 1 ∑ b e λ s b } is convex by Hölder’s inequality, and κ ( 0 ) = 0 \kappa(0)=0 κ ( 0 ) = 0 . Each summand λ S t ( 0 ) − κ t ( λ ) \lambda S_t(0)-\kappa_t(\lambda) λ S t ( 0 ) − κ t ( λ ) is therefore concave and zero at zero, and so is their sum. (b) By convexity, L ( λ ) = L { ( λ / λ ′ ) λ ′ + ( 1 − λ / λ ′ ) ⋅ 0 } ≥ ( λ / λ ′ ) L ( λ ′ ) + ( 1 − λ / λ ′ ) L ( 0 ) L(\lambda)=L\{(\lambda/\lambda')\lambda'+(1-\lambda/\lambda')\cdot0\}\ge(\lambda/\lambda')L(\lambda')+(1-\lambda/\lambda')L(0) L ( λ ) = L {( λ / λ ′ ) λ ′ + ( 1 − λ / λ ′ ) ⋅ 0 } ≥ ( λ / λ ′ ) L ( λ ′ ) + ( 1 − λ / λ ′ ) L ( 0 ) . (c) Apply (b) to the pair λ ′ = λ ∗ \lambda'=\lambda^* λ ′ = λ ∗ and λ = λ g \lambda=\lambda_g λ = λ g whenever it is (b) with L ( λ ∗ ) ≥ 0 L(\lambda^*)\ge0 L ( λ ∗ ) ≥ 0 . (d) Some grid point satisfies λ ∗ / 2 < λ g ≤ λ ∗ \lambda^*/2<\lambda_g\le\lambda^* λ ∗ /2 < λ g ≤ λ ∗ , so by (b) L ( λ g ) ≥ L ( λ ∗ ) / 2 L(\lambda_g)\ge L(\lambda^*)/2 L ( λ g ) ≥ L ( λ ∗ ) /2 ; then use Proposition 10 (b). (e) Write P g , t = ∏ s ≤ t e s ( λ g ) P_{g,t}=\prod_{s\le t}e_s(\lambda_g) P g , t = ∏ s ≤ t e s ( λ g ) and P g , 0 = 1 P_{g,0}=1 P g , 0 = 1 . Then ∑ g P g , t = ∑ g P g , t − 1 e t ( λ g ) = ( ∑ g P g , t − 1 ) ∑ g w g e t ( λ g ) \sum_g P_{g,t}=\sum_g P_{g,t-1}e_t(\lambda_g)=(\sum_g P_{g,t-1})\sum_g w_g e_t(\lambda_g) ∑ g P g , t = ∑ g P g , t − 1 e t ( λ g ) = ( ∑ g P g , t − 1 ) ∑ g w g e t ( λ g ) , and the product over t t t telescopes to ∑ g P g , T / G \sum_g P_{g,T}/G ∑ g P g , T / G . The scores are fixed numbers once computed, so per-site floors inside the score do not affect the argument.
Numerical checks
Two theory reviews (Appendix § 19 ) recomputed by exact enumeration the exact values stated in Propositions 3 and 4, the regression cases of the specification (exact sizes of the naïve and set-averaged products under mutual spillover, shared noise and a path graph; the design ceilings for Q ( O N ) = 0.01 Q(\mathrm{ON}) = 0.01 Q ( ON ) = 0.01 and for a fair bit with B = 20 B = 20 B = 20 ), and the forward–backward recursion and exact sampler of Proposition 2(b); the values agreed with those of the frozen documents. The first review also recomputed the exposure-clock cases. Both checked the identity of Proposition 7, which has no frozen counterpart, on random arrays. For Propositions 8–12, scripts checked the inequalities on random finite designs, the 8/9 grid constant, the rank constants and a closed-loop toy, and the independent review added ten counterexamples (heavy tails, shrinking competitor mass, floors, residue classes, null paths, ties, cross-unit copying, spillover feedback, grid coverage and sampled denominators), each of which the final statements respect; these checks are not exhaustive. The scripts and their outputs are archived with their hashes (Table 19 ).
Design laws
Every validity statement of the paper is conditional on a working design law. This appendix states each law exactly, reports how it was checked against the released command records, and lists what remains unverified. The defining documents were committed before the M4 registration; their hashes are in Appendix § 19 .
ERK
Laws. In each field the light program is a train of pulses whose inter-pulse intervals were randomized by the acquisition protocol. All working laws hold the first and last pulse and the leading and trailing zero-command periods at each field’s observed values. In fields with a rest of 241 minutes between pulses, the variants differ in how they treat it: V1 fixes the rest’s rank among the intervals; V2 inserts the rest after the first pulse at or beyond a threshold time taken from a stated range, sampling full arrangements uniformly by rejection; and V3, the law used for every primary analysis, shuffles the non-rest intervals before and after the rest separately. Fields without a rest reduce, under every variant, to a uniform permutation of the multiset of intervals. Counts of arrangements are counts of distinct paths, so repeated gap lengths are not double-counted. At an origin, the conditional sampler keeps the completed intervals, conditions on the time elapsed since the last pulse by rejecting the too-short ongoing gap, and returns the future pulse train. Fields run under a binary encoding policy are deterministic and contribute e = 1 e = 1 e = 1 .
Checks. All field-by-variant combinations passed unconditional checks of multiset, rank and segment structure, and simulated combinations passed conditional checks of the future train at several origins. Before any outcome was used, each field’s observed interval sequence was compared with uniform permutations of its own non-rest multiset (10,000 permutations, fixed seed) on three statistics: the Spearman correlation of interval length with position, the number of runs up and down, and the lag-one autocorrelation of intervals. A field was to be flagged if any statistic had two-sided p < 0.01 / 39 p < 0.01/39 p < 0.01/39 , a Bonferroni correction over thirteen fields and three statistics. No field was flagged; the smallest per-field p-value was 0.0288, and the Fisher-combined p-values were 0.2921, 0.5117 and 0.7157 for the three statistics. The same procedure gated the M5 field (§ 8.4 ).
What remains unverified. The acquisition protocol’s finite list of interval values, the uniformity of its shuffle, the rule that triggers the rest and the treatment of the series’ edges and time budget were not established from released code. An exchangeability check cannot certify an exact randomization kernel; it can only fail to find evidence against the working law.
CcaS/CcaR
The source law. The released acquisition code draws, for each lineage, a pair of bounds and a hidden integer that starts at the midpoint and moves up or down by one with equal probability, clipped at the bounds. For the first thirty-six steps the command is forced off while the hidden walk keeps evolving; afterwards the command is on whenever the hidden value is non-negative. The exact forward filter of this law matched an independent brute-force enumeration to rounding error.
The source law fails, and so does a fitted alternative. Under goodness-of-fit criteria committed before any statistic was computed, which compare on- and off-run-length distributions and lag correlations of the released commands with simulations from the law, the source law was rejected: every one of the seven dates failed at least the off-run total-variation criterion. A fitted order-k k k Markov model of the commands (K4) also failed the frozen gate on two dates, so no fitted kernel was admissible.
The working law D1 and when it was found. A diagnostic recorded during that check showed that the released commands match the source law delayed by one step, consistent with a one-step shift introduced at data export. Under the delayed law (D1), the run-length total-variation distances were at most 0.01485 across dates and run types, against a frozen threshold of 0.02. The delay was not one of the candidates evaluated in advance, and its documentation does not overturn the rejection of the source law or establish the actual export transformation; D1 is therefore a working law, and every CcaS result in the paper is conditional on it. Its first numerical record is dated 2026-10-06, and the D1 implementation was committed the same day, both before the M4 registration on 2026-10-08; it was thus found after the first design-law check failed, but before any registered analysis.
Fixed templates and eligibility. On each date, most lineages follow the random walk, but some repeat one of two fixed command templates; across dates, 2735 rows belong to 14 template groups. The templates were identified as the command paths duplicated more than ten times on a date, and frozen by hash before calibration. Template rows have zero design entropy and contribute e ≡ 1 e \equiv1 e ≡ 1 ; the random-walk rows are tested under D1 conditioned on avoiding both templates of their date, computed exactly by the forward–backward recursion of Proposition 2 (b). That conditioning is exact only if the templates are genuinely fixed designs; because they were inferred from released duplicates rather than released by the experimenters, this is an assumption.
Neighbour exposure. The released neighbour-exposure covariate is aligned one step ahead of the neighbours’ commands and would let a score use other lineages’ future actions. Every CcaS score uses its lag instead (§ 2.3 ); the sensitivity analysis restricted to lineages whose lagged exposure could be verified from commands is reported in § 8.2 .
Status of the working laws
Proposition 5 would bound the cost of a wrong working law if the law had been fixed independently of the tested commands and its distance from the true law were known. Neither holds here: the fitted law was estimated from the released commands, and D1 was chosen after the source law failed. We therefore claim no theorem-level guarantee for the working laws and rely on calibration with the released command records (§ 5.2 ), which bounds size empirically under the commands that were actually issued.
Tables and disclosures
Synthetic validation tables
Table 12 gives the development grid of § 5.1 by family, test and baseline; Table 13 the sealed challenge of § 5.4 by family, with the family descriptions of the challenge report; and Table 14 the selection grid of § 6.2 cell by cell. Each table is generated from the committed result files named in its caption.
Table 12. M2a development grid. Sources: DEV_GRID_v3.json @ b0ad5976ff95; DEV_SUMMARY_v3.json @ b0ad5976ff95. Ranges over saved designs, dynamics and four reference models; 12-unit development grid. The separate 512-unit proxy-trap headline is not merged into these cells. Sharp-spillover cells test the own null; the two masked families are alternatives.
Table 13. M3 sealed challenge by family – Descriptions and sharp-null size; Eligible alternative cells. Sources: M3_REPORT_ACCESS.json @ 4c18891d6079; M3_REPORT_9c46394.md @ 4c18891d6079; M3_SCORES_9c46394.json @ 4c18891d6079.
caption: Eligible alternative cells.
Corrections to registered texts
Registered documents cannot be edited. Where a registered text is inaccurate or ambiguous, we correct it here.
Selection of the repaired statistic. The M5 registration states that the statistic was selected on synthetic data under a pre-committed rule. The rule was committed in advance, but it did not
specify how to break a tie between equally hard cells, and the script that resolved the tie by grid order was first committed together with the selection result. Kish tempering would have been selected under most alternative tie resolutions (§ 6.2 , Table 14 ).
Status line of the M5 plan. The registered plan is byte-identical to the file committed before the registration, whose header still reads that it is a draft and not registered. Its registered status is established by the OSF record and by the registration addendum, which states that it supplements the byte-identical plan (Appendix § 19 ).
Name of the M4 operational plan. The frozen specification refers to the operational plan as M4_ANALYSIS_PLAN_FINAL.md; the registered, hash-bound plan is M4_ANALYSIS_PLAN_v6.md, and the registration’s summary field stated this correspondence at registration. An older file named M4_ANALYSIS_PLAN_FINAL.md exists with different content and is a superseded predecessor.
Public versus embargoed registration. The M4 registration text and plan call for a public registration before training; zxar3 and qezbx were created with an embargo, which the governing specification permits. Both embargoes were lifted on 28 October 2026, before this paper was posted.
Development settings. The development simulations used Monte Carlo e-values with B = 63 B = 63 B = 63 and horizon three; the frozen method uses B = 127 B = 127 B = 127 .
Rehearsal threshold. The single-field rehearsal for M5 used the threshold E ≥ 40 E \ge40 E ≥ 40 of an unregistered two-hypothesis draft, not the registered E ≥ 20 E \ge20 E ≥ 20 (§ 5.5 ).
Disclosures by stage
Before M4. M4 is a registered re-analysis of data analyzed earlier in BioWM-Eval (Appendix § 15 ), not a blind test. During the power pilot, a documentation search incidentally displayed earlier aggregate results; no setting changed on that basis. The designer of the sealed challenge had seen
N is null, A alternative; F08 GN is global null, ON own null and OA own alternative. Null counts are the saved C1 family pools; F11 nonzero-TV null cells are not pooled. Replacement-diagnostic ranges use only C1-included null cells and B-abl/B-zero/B-swap/B-marg/B-rule, across models/horizons. Eligible F11 alternatives include approximate-law cells; their recorded delta remains in the full TSV. Family descriptions are verbatim from the final M3 report lines 27-37, accessed only through the paper cache.
log E is the saved median; counts use E ≥ 100 E \ge100 E ≥ 100 . Best all includes the C0 baseline; best fixes compares C1-C3 only. M2e cell-id f0.10 denotes its fixed action coefficient, not a measured variance-explained signal. No selection rule was rerun.
the source of the reference models and one correction to the challenge design. The CcaS working law D1 was adopted after the source law failed its goodness-of-fit check, before the registration (Appendix § 17 ); every CcaS rejection is conditional on it. The own-channel eligibility bound is vacuous. Observation-process assumptions (finite masks, segmentation) are stated, not tested. Power is low for weak or masked effects, and the e-value tests are conservative. The variance floor of the published CcaS model was active. The pilot gave each synthetic ERK field one trajectory, and the end-to-end rehearsal used a simple generator; neither could reveal pooling amplification.
M4 and its aftermath. The M4 registration was submitted, and its content fixed, before any outcome was opened. OSF’s activity log for the associated project, public since the embargoes were lifted, records its approval about two days after submission, after the M4 training and test outcomes had been accessed (Table 16 ); the log does not show whether the approval was given by hand or by OSF’s automatic approval, which falls due 48 hours after submission. Until approval the registration could have been cancelled; it was not, and its content is fixed by the hashes in Table 15 . Neither ERK positive control was rejected in the registered analysis. The diagnosis of pooling amplification, the selection of the repair and the exploratory analysis X all followed that failure; X was run on test fields opened in M4. The dated note that fixes interpretation rules for the positive controls was written after the M4 report had been read, and its rules only restrict claims. Two operational scripts (a launcher and a report formatter) were added during the test stage and are not in the freeze manifest; their hashes are in Appendix § 19 . M4 did not save per-field, per-factor records, so the decomposition of its log e-values by field is not available. The repaired statistic’s correlation parameter was estimated on training fields only.
Table 15. Registrations and attached files. Rows from the first version are unchanged. Attachment hashes for ctfxv, j75g2 and 4wj7k are complete; other displayed hashes retain the 12-hex convention of the first version.
Table 16. M4 execution timeline. Sources: E2_m4_timeline_full.tsv and APPX_TABLE_SOURCES.json.
M5. The M5 registration followed X; the plan had been committed before X and was not changed afterwards, and the registration disclosed the X results. The confirmatory field’s thirty-minute intervals lie outside the ten-to-twenty-five-minute training range. The single-field rehearsal used the linear-Gaussian model only. The command-law gate inherited the multi-field threshold of M1, which is lenient for one field. Permission to retry the data download was added in the execution instruction and implemented in a separate input-output adapter; it was not used. Origins at which every cell was masked were not tallied separately. The acquisition law of the confirmatory field is unverified, so validity is conditional on V3. OSF’s activity log records the approval of the M5 registration minutes after its submission and before the first data access (Table 17 ); a self-reported browser visit shortly after the logged approval is also archived. Five preparation files carry modification times later than X but are byte-identical to their earlier commits.
Table 17. M5, X and ERK diagnostic timeline. Sources: E3_m5_timeline_full.tsv and APPX_TABLE_SOURCES.json.
Pixel demonstration. All results are simulated from one simulator family and two model classes, with one training run per cell and a reduced compute budget. Kish tempering, registered for M5, did not repair pooling amplification there; contrast tempering, which restored power at the larger effect size, is exploratory and has no confirmatory test. Two expectations of the protocol were wrong: the oracle did not collapse under gain noise, and the fitted models also collapsed under offset noise. The trap experiment could not show the diagnostics’ growing confidence with data, and its frozen negative control was uninformative; the informative negative control was added post hoc. Three dated deviation notes record a reclassified pre-freeze audit flag (blocked standard-library probes, no data access), a reconstructed counter in the negative-control code, and a refresh of version-control index metadata in another repository.
ERK training diagnostic. Command records were copied and hashed before the access marker was created; they are design information, not outcomes. The access log merges internal hash reads into logical open events, and the new repository’s version-control identity was read from an existing repository with non-modifying commands. The agreement between the two tempering factors is partly structural (Proposition 7 ).
Registrations and hashes
Every analysis in this paper is bound to committed files by SHA-256 hashes, and every access to real outcome data was preceded by a one-time marker and logged. This appendix lists the registrations, the order of every real-data access, the repositories and their final commits, and the review archives. All times are UTC.
Registrations. Table 15 lists the five OSF registrations: zxar3 (M4), qezbx (M5), ctfxv (the DIAMOND study), j75g2 (the IRIS and TD-MPC2 study) and 4wj7k (the robot-data study). All five are public. The registered component of 4wj7k (2ksxa) also holds the results note, the decision file, the beacon report and the record of the stop before the run. The registered component of j75g2 (7mx6p) also holds the results update, the decision file and the deviation note posted after the run.
Order of real-data access. Table 16 and Table 17 give the time-ordered events of the two registered analyses: registration, creation and consumption of the one-time marker, the first opening of any outcome member, the command-law gate and the first outcome statistic, together with the number of member reads. The exploratory analysis X and the ERK training diagnostic ran under separate authorizations and markers, also listed.
NOT RECORDED means no standalone timestamp in the inspected record. The acquisition timestamp is extraction, not Keras load. The first preflight window bounds the recorded architecture validation. Script hashes are documentary hashes; no operational script is executed.
Repositories. Table 18 lists each stage’s repository, its final commit and the role of the stage; the operational scripts added during the M4 test stage are listed with their hashes in Table 16 . The paper’s own repository records, for every number in the text, the source file, commit, hash and key from which it was generated.
Table 18.
Events are grouped by study; X preceded M5. All times come from saved logs or the dated evidence note. Network addresses and data hashes are quoted metadata only; no archive or outcome is opened. Full event records and hashes are in the TSV.
Table 18 : Repository records; Rows from the first version are unchanged.
Reviews. The theory of § 4 and Appendix § 16 was checked in two AI-assisted review sessions with read-only access to the project, distinct from the drafting sessions and from each other: the first by Claude Code, the same model family as the drafting model, and the second by Codex, a different model family. According to the project records, the frozen theory documents had themselves been written in an earlier Codex session, so neither review is independent of all the authorship involved in the way a human statistician’s would be. Every bibliography entry was checked in a separate Claude Code session, which retrieved each source and checked each attributed claim against its text. Both theory reviews recomputed the stated exact values by enumeration. Neither found an error in the frozen proofs under their stated premises, and the bibliography review found no nonexistent reference. Both theory reviews found statements in our drafts that were stronger than what had been proved; the second also found that the hypotheses added to Proposition 6 after the first review did not exclude a predictor window that reveals the stratum. The revised text narrows these statements; the review archives record each finding and how it was addressed (Table 19 ). For this version, Propositions 8 –12 , which were drafted by Claude, and every sentence that uses them were checked by Codex in five further read-only sessions: an independent review with ten runnable counterexamples, a delta review of the revised draft, a check of the trap experiment’s logging policy against Proposition 12 (b), a final check of the integrated text, and a delta check of its revisions. The final check also found that the duplicate-score rule of the baseline study does not by itself establish the exchangeability that the rank statistics need (Appendix § 22 ). The references added in this version were verified twice, in the drafting session and in a separate Claude Code session. The reviews check mathematics and citations; they do not certify the actual acquisition laws, the implementation of every named statistic, or the chronology of the archives.
Table 37.
Table 38.
Table 39.
ERK per-field diagnostics
Table 20 reports, for each training field, reference model and horizon, the quantities behind § 8.5 : the frozen median-pairwise residual correlation ρ ^ \hat{\rho} ρ ^ , the mean pairwise residual correlation, the correlation ρ c \rho_c ρ c implied by the contrast-tempering factor, the two ratios ρ c / ρ ^ \rho_c/\hat{\rho} ρ c / ρ ^ and ρ c / ρ m e a n \rho_c/\rho_{\mathrm{mean}} ρ c / ρ mean , and the number of validation cells per contributing origin (mean, minimum and maximum). Pooled rows sum the numerators and denominators of the contrast factor across fields, as the pre-declared protocol specifies. The diagnostic used training fields only and computed no e-value. Ratios below the pre-declared band in individual fields are retained; the reading rule concerns the pooled rows.
Pixel demonstration: full grid
The protocol, the dated predictions and every result file of the pixel demonstration are frozen and hashed (Appendix § 19 ). Compute limits fixed the episode length at the reduced level of the protocol’s pre-declared ladder, with sixteen test episodes per alternative cell, thirty-two per null cell of the main grid and sixty-four per trap cell; intervals for alternative cells are correspondingly wide. Every statistic is the average of the two learned models unless marked otherwise.
Table 22 gives the null cells of both grids: rejection counts and mean e-values for every statistic, the basis of the validity criterion. Table 24 gives the alternative cells of the main grid by noise condition, frame size and effect size: median log E E E and rejection rate for the pooled, Kish-tempered, contrast-tempered, field-mean, site-average and fractional-Kelly statistics and for the oracles. Table 27 gives the effect-report rates of the three common diagnostics in every null cell. Table 28 gives the tempering quantities of every model and cell: the median-pairwise correlation, the Kish and contrast factors and the two design effects. For the oracles, whose residuals are independent pixel noise, both design effects are close to one and their ordering is not informative; the rows in which the ordering differs from that of the learned models are all oracle rows and are marked. Table 30 gives the post hoc negative control with contrast tempering, by cell and model, next to the frozen negative control.
Baseline study
This appendix documents the exploratory baseline study of §6.4. It used only simulated data and the saved artifacts of the pixel demonstration and of the selection grid of §6.2; no real data were copied, opened or used. It was not registered on OSF. Its protocol, with the statistics, the decision rules and six dated predictions, was committed to the study’s repository with its hash recorded before any value of a new statistic was computed. Table 31 lists the commits and hashes of the protocol, its amendment and every result record cited below.
Protocol
Statistics. The study added five statistics to those of the original runs: the rank e-values R and R 1 / 2 R_{1/2} R 1/2 , the tempering mixture M over the grid λ g = 2 − g \lambda_g = 2^{-g} λ g = 2 − g , g = 0 , … , 12 g = 0,\ldots,12 g = 0 , … , 12 , validation-tuned tempering V (§3.5), and ORACLE_R, the rank e-value of the oracle’s pooled score, as a reference for what a rank statistic can reach. V was set for each cell and learned model from the ten validation episodes and the validation bags of 31 draws per origin that the contrast factor uses; every selected factor was written Per-field rho_med is the saved, unclipped field median; pooled rho_med is the frozen clipped median used by M5. The field ratios use their saved field median, not the pooled frozen denominator. Ratios to rho_mean are copied from the already-committed paper extraction; no correlation or ratio is recomputed here.
Table 20. ERK training-field diagnostic – Correlations. Sources: FIELD_RESULTS.json @ 0911ef20df6a; POOLED_RESULTS.json @ 0911ef20df6a; ERK_DIAG_RHO_MEAN_EXTRACT.json @ 4c18891d6079.
to a per-cell file whose hash was logged before any test score was computed. V is not defined on the ERK-like grid, which has no contrast design. Every statistic, old and new, was computed on the same test episodes and the same Monte Carlo draws (B = 127 B = 127 B = 127 ).
Ties. In the pixel world the scores of the candidate blocks are computed in batched floating-point arithmetic, so identical action blocks in different columns of a bag can receive scores that differ in the last bits. For the rank statistics each column’s score was therefore replaced, before ranking, by the score of the first column with the same action block; distinct blocks were tied only when those scores were exactly equal, and the factor averaged q q q over tied positions. This makes the scores of duplicate blocks identical within a bag. Exact mean-one validity additionally requires the scored bag to be exchangeable under relabelling of the candidates, and the first-occurrence rule alone does not establish that when numerical errors depend on a candidate’s position; we did not verify it for this run. The registered study of § 9 computed one score per distinct action and does not rely on this rule. M and V use the scores unchanged, as contrast tempering does. On the ERK-like grid identical blocks already share one score.
Table 21. ERK training-field diagnostic (continued) – Saved ratios and finite validation cells. Sources: FIELD_RESULTS.json @ 0911ef20df6a; POOLED_RESULTS.json @ 0911ef20df6a; ERK_DIAG_RHO_MEAN_EXTRACT.json @ 4c18891d6079.
Per-field rho_med is the saved, unclipped field median; pooled rho_med is the frozen clipped median used by M5. The field ratios use their saved field median, not the pooled frozen denominator. Ratios to rho_mean are copied from the already-committed paper extraction; no correlation or ratio is recomputed here.
Scenarios. In the pixel world the study used the main grid of the demonstration: 24 alternative cells with 16 test episodes each and 12 null cells with 32, one episode being one decision unit, with rejection at E ≥ 20 E \ge20 E ≥ 20 for the two-model average, as frozen. The trap grid of § 7 was not part of the study. On the ERK-like grid it used the 12 alternative cells with 16 replicates each and the 2 null cells with 32, one replicate being one decision unit, with the grid’s threshold E ≥ 100 E \ge100 E ≥ 100 .
Decision rules. The primary comparison was the larger effect in the pixel world, 192 decisions. A new statistic matched contrast tempering if its total rejections were at least the contrast-tempering total minus 19 and it was at most 6 of 16 below contrast tempering in every cell; it was clearly weaker if its total was at most the contrast-tempering total minus 48, and intermediate otherwise. At the smaller effect a statistic was clearly stronger at 19 or more rejections above contrast tempering and clearly weaker at 10 or more below. On the ERK-like grid a statistic with two or more alternative cells below 14 of 16 was weaker than Kish tempering . A new statistic with two or more rejections in any null cell would have been stopped as a suspected implementation error. The protocol fixed, in order, four outcomes and what each would mean for this version of the paper: (1) R or M matches contrast tempering and is not weaker than Kish tempering on ERK-like data, in which case the repair is reframed around statistics robust to score magnitude and the better of R and M is recommended for new studies as an exploratory finding, its confirmatory use needing its own preregistration; (2) V matches but R and M are clearly weaker; (3) R, M and V are all clearly weaker; (4) anything else. In every case the role of Kish tempering in the registered analysis M5 was to stay unchanged.
Table 22. Pixel demonstration null cells – G1a: two-model-average counts. Sources: T1_null_validity.csv @ 004fde8f3242. Mean e and all per-model values are retained in G1_null_full.tsv. Counts out of K; O denotes oracle (0, 1, 1c and joint). NC is the frozen marginal-resampler C1 control, available only for G2; – means not run. Mean e and every per-model value are retained in G1_null_full.tsv. G1b selects the larger of the two learned-model counts separately in each cell/statistic; oracle counts appear once.
Table 23. Pixel demonstration null cells (continued) – G1b: maximum per-model count. Sources: T1_null_validity.csv @ 004fde8f3242. Mean e and all per-model values are retained in G1_null_full.tsv. Counts out of K; O denotes oracle (0, 1, 1c and joint). NC is the frozen marginal-resampler C1 control, available only for G2; – means not run. Mean e and every per-model value are retained in G1_null_full.tsv. G1b selects the larger of the two learned-model counts separately in each cell/statistic; oracle counts appear once.
Recomputation and the reproduction gate
The study reused the saved trained models, episodes and frozen parameters and reran only the test stage, and the validation scoring for V, on a different machine; nothing was retrained. Monte Carlo bags were regenerated from the frozen seed keys, and the convolutional model ran on a CPU with the frozen determinism settings. Before any new statistic was computed, a reproduction gate Saved median log E and rejection rates; learned statistics are the two-model average. Each alternative cell has the saved K in the full TSV; no interval is recomputed.
Table 24. Pixel demonstration G1 alternatives by noise – NONE; two-model averages; NONE; oracle rejection rates. Sources: T2_alternative_performance.csv @ 004fde8f3242. NONE; two-model averages.
Table 24 (PDF p. 71). NONE; oracle rejection rates.
Table 25. Pixel demonstration G1 alternatives by noise (continued) – OFFSET; two-model averages; OFFSET; oracle rejection rates. Sources: T2_alternative_performance.csv @ 004fde8f3242. OFFSET; two-model averages.
Table 25 (PDF p. 71). OFFSET; oracle rejection rates.
Saved median log E and rejection rates; learned statistics are the two-model average. Each alternative cell has the saved K in the full TSV; no interval is recomputed.
compared, for every test episode or replicate, the recomputed per-origin log factors and episode log E E E of every existing statistic with the saved records. A cell was EXACT if every difference was at most 10 − 9 max ( 1 , ∣ x ∣ ) 10^{-9}\max(1, |x|) 1 0 − 9 max ( 1 , ∣ x ∣ ) ; CLOSE if it was not exact but every rejection decision was identical and the episode log E E E differences were small; and FAIL otherwise, which would have stopped the study.
The gate compared 51 cells, and in 51 of them every rejection decision was identical to the frozen record. As first written, the CLOSE tier bounded the relative difference of episode log E E E by 10 − 4 10^{-4} 1 0 − 4 .
Table 26. Pixel demonstration G1 alternatives by noise (continued) – GAIN; two-model averages; GAIN; oracle rejection rates. Sources: T2_alternative_performance.csv @ 004fde8f3242.
Table 26 (PDF p. 72). Pixel demonstration G1 alternatives by noise (continued) – GAIN; two-model averages; GAIN; oracle rejection rates. Sources: T2_alternative_performance.csv @ 004fde8f3242.
Saved median log E and rejection rates; learned statistics are the two-model average. Each alternative cell has the saved K in the full TSV; no interval is recomputed.
That is ill-posed for reference values near zero, and 4 pixel cells were classified FAIL only because their references were close to zero. An amendment, written after these gate results and before any new statistic was computed, bounded instead the scaled difference ∣ Δ ∣ / max ( 1 , ∣ log E r e f ∣ ) \lvert\Delta\rvert/\max(1,\lvert\log E_{\mathrm{ref}}\rvert) ∣ Δ ∣ / max ( 1 , ∣ log E ref ∣) by 10 − 4 10^{-4} 1 0 − 4 , the scaling the EXACT tier already used. It changed no statistic, grid, threshold or prediction. Under the amended rule 0 cells failed, and the largest scaled difference in any cell was 8.6 × 10 − 6 8.6 \times10^{-6} 8.6 × 1 0 − 6 . The differences arose only in arms that involve the convolutional model. The amendment also recorded the tie rule above and a mechanical reading of the decision rules, before any rank value was computed; the gate records keep both classifications (Table 31 ).
Table 31. Baseline study records (36 cell-specific validation freezes).
Results
Table 32 gives the mechanical reading of the decision rules and the status of each dated prediction; Table 33 and Table 34 give every cell. The rules returned outcome 1, and §6.4 follows the reading that the protocol attached to it. The prediction that R would match contrast tempering failed; §6.4 discusses a possible mechanism suggested by Proposition 11, whose baseline depends on the tie handling and does not by itself explain the observed counts. The mixture became the primary statistic of the registered study of §9, which is the confirmatory test; this study is not.
Table 32. Baseline study mechanical decisions and separately saved prediction verdicts.
Table 33. Baseline study pixel rejections; every median is retained in the full TSV.
Table 34. Baseline study ERK-like cells (saved medians, display rule (a)).
Table 53.
Table 97.
Registered test on a published world model
This appendix documents the registered study of §9: its registration and the order of events, the design in full, the two deviations and every result. Table 35 gives the identities, hashes and times on which the account rests; all times are UTC.
Table 35. DIAMOND study identities and chronology (UTC); archive metadata authenticated by committed manifests.
Table 27. Pixel demonstration common diagnostics on null cells. Sources: T3_common_diagnostics.csv @ 004fde8f3242. Effect-report rates are copied from T3. Denominators and exact saved counts remain in the full TSV.
Registration and order of events
The protocol, its draft with the dated predictions, the pipeline and a pilot were committed to the study’s repository before registration. The confirmatory code was then frozen in one commit, after unit tests on toy inputs and an exact replay of the pilot, and the protocol, identified by its hash, and the frozen commit were registered on OSF (ctfxv). A note filed with the registration states that the dates written in the protocol text, including the date attached to the predictions, are labels assigned in advance, and that the authoritative times are the commit times and the registration time in Table 35 ; by those, the predictions were fixed in the draft protocol and carried unchanged into the registered version.
The confirmatory seeds could not exist before the registration. The protocol named a public randomness beacon, the drand mainnet chain of the League of Entropy [24 ] , whose rounds are scheduled every 30 seconds, and the round used was the first beacon round scheduled at least two minutes after the start of the minute in which the registration was recorded. Every seed of the confirmatory run was derived by hashing a fixed label together with a master value, itself the hash of the protocol’s hash and the round’s randomness. The runner recorded the round’s signature and checked its format but did not verify it cryptographically; the record can be verified against the chain’s public key. Generation and analysis ran once from the frozen commit, behind a one-time marker with an access log, apart from the two restarts described in §1.3.
Table 28. Pixel demonstration tempering parameters – G4a: learned models. Sources: T4_tempering.csv @ 004fde8f3242; SEC67_T4_PATTERN_EXCEPTIONS.tsv @ 4c18891d6079. K and C subscripts denote Kish and contrast design effects. Lambda and lambda_c, and all original model-level rows, remain in G4_temper_full.tsv. Daggers identify the 28 committed oracle ordering exceptions. d0/dlo/dhi denote the effect levels; G2 N identifies training size.
The pilot used separate seeds and never entered the confirmatory analysis. It ran the full pipeline on null units only (4 validation and 8 test units per game, 200 steps), where the largest number of rejections by any statistic in any game was 0 of 8. On active pilot units it computed predictions, variances and the two estimated factors, but no e-value, rank, mixture, validation-tuned factor, p-value, rejection or replacement diagnostic.
Design
Model and emulator. DIAMOND’s code was used at a fixed commit with its upstream files unchanged, and its released checkpoints for Boxing, Pong and Breakout at a fixed revision, with the hashes in Table 35 . The model card states no licence for the weights; they were used for research inference only and are not redistributed. Only the denoiser was used, with DIAMOND’s three-step Euler sampler conditioned on the last four observations and actions. The emulator and preprocessing were those of DIAMOND’s training: the NoFrameskip-v4 environments of ALE 0.9.0 ([2 ] ), sticky-action probability zero, four frames per action with the maximum of the last two, 64 × 64 RGB frames by area interpolation, 1 to 30 no-op actions at reset and the minimal action sets of 18, 6 and 4 actions. Everything ran on one laptop GPU, in an environment whose exact specification is part of the frozen commit.
Table 29. Pixel demonstration tempering parameters (continued) – G4b: oracle. Sources: T4_tempering.csv @ 004fde8f3242; SEC67_T4_PATTERN_EXCEPTIONS.tsv @ 4c18891d6079. K and C subscripts denote Kish and contrast design effects. Lambda and lambda_c, and all original model-level rows, remain in G4_temper_full.tsv. Daggers identify the 28 committed oracle ordering exceptions. d0/dlo/dhi denote the effect levels; G2 N identifies training size.
Units and eligibility. A unit is one segment: one reset, then up to 500 agent steps, ending earlier at the first termination or truncation. A life loss does not reset the emulator but makes the origins around it ineligible: an origin t t t is eligible if the observations t − 3 , … , t t - 3, \ldots, t t − 3 , … , t and the transition to t + 1 t + 1 t + 1 lie in one life. Each unit is tested on its own. A unit without eligible origins has E = 1 E = 1 E = 1 for every statistic and counts as a non-rejection; Table 36 lists such units.
Table 36. DIAMOND study complete results; factors, replacement diagnostics and empty-unit lists are retained in the full TSV.
Prediction and scores. The candidate at origin t t t is the label a t a_t a t ; the model conditions on the observations t − 3 , … , t t - 3, \ldots, t t − 3 , … , t and on the recorded labels a t − 3 , … , a t − 1 a_{t-3}, \ldots, a_{t-1} a t − 3 , … , a t − 1 , also in the NULL condition, and the target is the real observation Y t + 1 Y_{t+1} Y t + 1 (h = 1 h = 1 h = 1 , one residue class). The bag has 128 columns: the recorded label and 127 independent uniform draws. At each origin one Gaussian noise image was drawn and the model predicted once per distinct action in the bag, the validation contrast draws and the replacement draw, with that image; predictions were expanded back to the columns, so identical actions have identical predictions and scores. Predictions were stored as one of 256 intensity levels, and a storage check stopped the run if a sampler output differed from its stored level by more than a set limit (§I.3). The per-site variance is the mean squared residual of the recorded-label prediction over all eligible origins of the validation units of the same game and condition, floored at ( 2 / 255 ) 2 (2/255)^2 ( 2/255 ) 2 , and each site score is floored at log 10 − 12 \log10^{-12} log 1 0 − 12 .
Table 30. Pixel demonstration negative-control comparison. Sources: T1_null_validity.csv @ 004fde8f3242; H1_SUMMARY.csv @ cd5bb543eb06. The correct-resampler C1c values were sealed originally; they are juxtaposed here as part of the POST HOC comparison and were not recomputed. AVERAGE is the two-model arithmetic average.
Statistics. The pooled score, the mixtures, the rank e-values and the rank tie rule are those of § 3.5 and Appendix § 22 , computed with the study code of the baseline study, copied byte for byte, and h = 1 h = 1 h = 1 . Kish tempering uses the correlation estimate of § 3.4 from the standardized validation residuals of the recorded-label prediction, clipped to [ 0 , 1 ] [0, 1] [ 0 , 1 ] . Contrast tempering uses 31 validation draws per origin and is clipped to [ 1 / n , 1 ] [1/n, 1] [ 1/ n , 1 ] . Validation-tuned tempering maximizes the median validation log E E E over the 16 validation units on the fifteen-value grid, ties going to the larger factor. The replacement diagnostic is, per unit, a one-sided one-sample t t t -test that replacing the recorded label by the replacement draw increases the mean squared prediction error, with an effect reported at p < 0.05 p < 0.05 p < 0.05 ; a unit with fewer than two origins or with all differences zero has no p p p -value and reports no effect.
Criteria, predictions and interpretation. The criteria are those of § 9.1 . For validity the count governs; the accompanying one-sided Clopper–Pearson bound, written in the protocol as 0.095, is 0.0951217 for 2 of 64. The five dated predictions were: validity in all three games (95%); collapse in at least two games (75%); repair in at least two games (60%); Kish tempering rejecting in fewer than 24 of 32 units in at least one game (60%); and the rank e-value R rejecting less often than the mixture in at least two games (70%). The protocol also fixed the interpretation: overall confirmation supports the claim that, on a published world model, pooling collapses the pooled test and the pre-specified mixture restores power while keeping validity; if collapse held but repair failed, the paper would say that tempering prevents collapse without guaranteeing power; and a validity failure would be treated as an implementation error, because validity holds by construction.
Deviations
Two integrity checks of the frozen code stopped the confirmatory run. Each was documented in a dated note, published on OSF before the run resumed (Table 35 ); neither changed generation, prediction, scoring, statistics, decision rules or gates, and both restarts reused the same beacon round and seeds.
Deviation 1: access log. After the first unit, a null validation unit of Boxing, had been generated, stored and its hash verified, the runner’s check of the access log’s hash chain stopped the run. In two calls the unit’s file hash had been passed under the key that the log also uses for each row’s chain digest, so the digest overwrote it and could no longer be reproduced. No score or statistic had been computed for any confirmatory unit and no ACTIVE unit had been generated. The fix renamed the key, made the logging function refuse reserved keys, and added regression tests; the full test suite and the exact pilot replay passed. The aborted run was kept unchanged.
Deviation 2: storage tolerance. The restarted run reproduced the first unit exactly and stopped after 84 completed units, in an ACTIVE validation unit of Boxing, because one sampler output differed from its stored intensity level by 2.66 × 10 − 7 2.66 \times10^{-7} 2.66 × 1 0 − 7 , above the registered limit of two float32 epsilons. The sampler already rounds its output to an intensity level and returns it in float32, so the check measured float32 rounding only, while a level is unambiguous for any difference below 1 / 255 1/255 1/255 . The registered limit left too little margin: the largest difference in the pilot had been 2.05 × 10 − 7 2.05 \times10^{-7} 2.05 × 1 0 − 7 , and this should have been noticed before registration. The limit became 10 − 5 10^{-5} 1 0 − 5 , still far below half the spacing of the levels, so an output off the intensity lattice still stops the run; the change was one line, and no stored value changed. At this stop the runner had already analysed the Boxing NULL group, and ACTIVE validation units of Boxing had been generated; no ACTIVE test unit existed, and no score, e-value, rank, mixture, validation-tuned factor, p-value or rejection had been computed on any ACTIVE unit. The stopped run was kept unchanged.
Reruns. The final run regenerated every unit of the stopped runs from the same seeds. Of 84 units compared array by array, 84 were identical, and the results of the Boxing NULL group were reproduced exactly.
Results
Table 36 gives every statistic in both conditions of each game, with the estimated and selected factors and the replacement diagnostic; Table 38 gives the criteria, the overall decision and the status of each prediction, as written by the analysis code. § 9.2 summarizes them.
A post hoc illustration of the factors
The reading at the end of § 9.2 rests on Table 39, a descriptive and post hoc calculation on a saved subset of the ACTIVE test units, the first four of each game, with no new simulation or sampling. For each unit the tempered log E E E was evaluated over the factors 2 − k / 4 2^{-k/4} 2 − k /4 , k = 0 , … , 71 k=0,\ldots,71 k = 0 , … , 71 , with the saved observed-inclusive bags of 128 scores, and the median of the four unit-wise maximizers is reported as the hindsight peak; this grid is used only here and is not a selection rule of the registered analysis. The margin column is the median across each unit’s median positive maximum competitor margin in its bags. In this subset the validation-selected factor lies near the hindsight peak and the contrast factor is an order of magnitude above it, with larger margin penalties, which is consistent with a possible pooling–amplification mechanism. By the bound (5) , a sampled competitor of multiplicity k k k limits the log factor by log ( 128 / k ) \log(128/k) log ( 128/ k ) minus the tempered margin, so a positive margin alone does not make the factor negative. The calculation does not isolate a cause, does not test the unfloored identity of Proposition 9 , and four units per game do not verify the counts of the full run.
Table 98.
Table 99.
Table 100. Table 38: DIAMOND study saved criteria and registered predictions.
Table 101.
Table 102. Table 39: Post hoc illustration on four saved ACTIVE units per game; no new simulation.
The IRIS and TD-MPC2 study
Registration and order of events
Protocol v0 was dated 8 November 2026, with SHA-256 5912af3dd3991d0190c54bf38cfb155f958adc4c8a9330b44e36ac3ff9cd9bb3. Protocol v1, dated 8 November 2026, has SHA-256 49c3857d329a04c7a0b366cdaa9d2a6f8aed6e0d18d87d8557b12adc25807d77. Version 1 resolves the build pilot’s open issues, freezes the game and task sets, adds the storage policy and the pilot record, and leaves the primary statistic, conditions, sample sizes, score definitions, criteria and predictions unchanged.
Table 40. Order of events. All timestamps are UTC. Commit times identify saved stages, not an inferred execution start.
The beacon randomness was 6fe3f3aa56c76f8d5b18a0ee651b874a0eac91ef5b20acd51730176cfe0698c0. The master seed was aaab1aa8e56ecac4df6b85e6f5fe74f41f06f689abf8ff939661430200a19207. The beacon record verifies saved-response integrity, with no BLS signature verification.
CONF-STOP-01 preceded generation: the run instruction asked for a per-unit disk check that the protocol did not require. The instruction was withdrawn; no unit was generated in that attempt. CONF-STOP-02 occurred at 2026-11-09T13:13:26.240048Z, after all 5392 units and 97 group summaries were stored, because the decision writer could not serialize a NumPy boolean. DEV-P4-01 changed only conversion of NumPy scalar values to built-in types. Three serialization tests and 22 unchanged tests passed; seven tests were skipped, four because they replayed the historical pilot and three because they wrote toy archives or a synthetic audit artifact. Before and after finalization, all 14,015 manifested original inputs and five further frozen metadata inputs matched their hashes. An independent standard-library comparison reproduced all 1279 decision scalar values. No unit or scientific statistic was recomputed.
On 10 November 2026, between 02:58 and 03:04 UTC, the authors posted to component 7mx6p the results update P4_RESULTS_UPDATE_20261109.md (SHA-256 6a2c2127a254964c8dd5509097a8e5094b6557097370cafd244d2ea159025ef6), the decision file, the deviation note and the run summary, and made the component public. The times are those shown in OSF’s file list, as recorded by the authors. The posted results update differs in wording from the draft OSF_UPDATE_P4.md kept with the run records.
Design details
The IRIS code revision was 24326aaaa283c527f42b89b44cfdecf2665a7a16, and its checkpoint revision was c0539e99adf924d44be0f19b12de00639daa5550. The TD-MPC2 code revision was 332f19e4aba4bffe2dc736f45ea7f7bdc5859093a, and its checkpoint revision was 8fb2a82e fb3bae96941da440128fe1332e4394fd. The MyoSuite wrapper patch, SHA-256 b9b09b337b02f2053beff2651c4398501bcdb9b95725e2d046b7f5d2ad598b51, removes the read-only Timeout assignment and passes the seed to the simulator constructor; it changes no dynamics or assets.
Table 41. TD-MPC2 checkpoints from protocol Section 3. SHA-256 values are complete.
Table 42. IRIS checkpoint SHA-256 values from the frozen checkpoint manifest.
IRIS uses four emulator frames per action, 64 by 64 RGB frames resized with bilinear interpolation, and the maximum of the last two resized frames. Sticky actions are disabled, minimal action sets are used, and resets include 1–30 no-op actions. A life loss makes the affected origins ineligible without a pseudo-reset. An origin and its four-frame context must lie in one life. The token score is teacher-forced joint next-frame token likelihood over 16 tokens; the secondary decoded-pixel score uses the Gaussian over 12,288 sites in Boxing, Pong and Breakout. TD-MPC2 uses a Gaussian over 512 latent coordinates, with variances fixed on validation units and action repeat two for DeepMind Control Suite. Each unit ends at the first native termination, truncation or cap, retaining the terminal observation as its last target. The caps are 500 steps for the three DeepMind tasks, 100 for myo-pose and myo-reach, and 75 for myo-obj-hold.
Each condition uses 16 validation units, 32 test units under partial or full execution, and 64 under NULL. Bags have 127 design draws plus the observed action; validation contrast bags have 31 draws plus the observed action. Seeds are the first four big-endian bytes of the SHA-256 namespace consisting of the study identifier, phase, arm, task, condition, split, unit and stream. The confirmatory phase includes the master seed. PCG64 generates external non-noise streams; simulator reset seeds are passed to the frozen native simulator, and CUDA generators supply the pixel-sampling noise. Design draws are stored losslessly as float64 and converted to float32 at required model and simulator boundaries; identical converted inputs share predictions.
Validation units retain targets, observed-action predictions, the 31 contrast-draw predictions and replacement predictions. Test units retain all 128 bag-column scores in float64, the observed-action and replacement predictions, and targets. Other bag predictions are computed and scored but not stored; the token arm retains per-token target log probabilities of every distinct candidate. Decoded pixel means are clipped and rounded to nearest-even uint8, then dequantized by 2 k / 255 − 1 2k/255 - 1 2 k /255 − 1 for scoring. Gaussian validation summaries are fixed before test generation. The protocol stores no model weights in its registration package.
Choice of the primary statistic
The regime map applied its pre-committed rule and selected V. The worst-cell rates were 0.8125 for V and 0.78125 for W(3/4), within the rule’s 1/32 tolerance; V won on mean rate. Amendment 1 left V and the weighted mixtures undefined for the saved ERK-like cells, because validation scores and residuals were missing. The later ERK validation-tuning check gave V a four-model log e-value of -101 at thirty minutes, with all four models negative, and two negative models at ten minutes; M gave 347 and 218 and W(3/4) gave 345 and 217. Protocol Section 2.1 registered W(3/4) before any confirmatory data existed. This departs from the regime map’s protocol, which said that its selection rule would fix the primary statistic.
Full results
The following tables copy every rejection count, median log e-value and factor from the pinned result tables. Rounding affects presentation only.
Table 43. IRIS native: rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 44. IRIS native: rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 45.
Table 46. IRIS decoded pixel (secondary): rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 103.
Table 47. IRIS decoded pixel (secondary): rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 48. IRIS decoded pixel (secondary): rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 49. TD-MPC2: rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 50. TD-MPC2: rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 51. TD-MPC2: rejections and median log E E E for every game/task and condition. Each cell is count; median. Counts are out of 64 under NULL and 32 otherwise.
Table 52. IRIS native: factors fixed on validation units. A dash means not defined.
Table 54. TD-MPC2: factors fixed on validation units. A dash means not defined.
Table 55.
Predictions
Table 55 (PDF p. 95). Eight predictions dated 8 November 2026: probabilities, outcomes and stored numerical evidence.
Replacement diagnostic
Table 56. Replacement diagnostic: effect reports and undefined p-values. Every stored arm, task and condition is shown.
Compute
The synchronized GPU-stage hours were 2.53421079 for IRIS tokens, 2.86836434 for the secondary decoded pixels, and 0.0810389767 for TD-MPC2. The secondary pixel units are shared with IRIS tokens and are not additional distinct units. Overall wall time was 8.27198099 hours. Other processes shared the GPU during the run; this affects timing only. GPU-event hours were not emitted by the frozen code.
Exploratory studies after the second version
Score-rule study
The score-rule study was exploratory, under the protocol dated 7 November 2026, SHA-256 3e772433 628581b0d600a80fdbdce4153b8f0f6b3f6bf36fe459bd2dcfce3c79. It rescored saved DIAMOND predictions, observations and bags without generating new units. The reproduction gate recovered all rejection decisions of the registered run before any alternative score rule. The ten alternatives were the two higher variance floors, one common variance, Student-t and Laplace densities, three block sizes, a 256-dimensional random projection and a rank-16-plus-diagonal joint Gaussian. Kish and contrast factors are undefined for the joint Gaussian. No scientific method deviation was recorded; an initial file-name discovery traversed broader directories, after which access was restricted to the authorized inputs.
Table 57. DIAMOND score-rule study (exploratory). ACTIVE rejections out of 32; median log E E E . The final column is the largest NULL count over the defined statistics.
Table 58. Site-count trend: median ACTIVE pooled log E E E along the block sizes.
Table 59. Score-rule predictions dated 7 November 2026.
Sampling study
The sampling study used the same protocol, with four predictions averaged at each eligible origin, and was exploratory. Its reproduction gate recovered the registered validation unit bit for bit, including all bag, contrast and replacement predictions. The initial disk rule stopped the study before generation. A dated amendment replaced it with a 100 GiB rule; the reproduction and 20-origin timing gates passed before generation. All 288 planned units completed. The saved readout records 12.925915 GPU-event hours and 13.08265 hours elapsed wall time. The two predictions dated 7 November 2026 were that collapse persists (probability 0.65) and that M repairs it where it persists (0.80); both were supported.
Table 60. DIAMOND sampling study (exploratory). ACTIVE rejections out of 32; median log E E E . The final column is the largest NULL count over the defined statistics.
Table 61. Sampling-study factors fixed on validation units.
Regime map
The regime map was exploratory, under the protocol dated 8 November 2026, SHA-256 f9ab5aee d26592704b5a71b0daa6df7df826a00acdb64468daae241f4c078eeb, and Amendment 1, SHA256 9ea8ec9b1d1566da23f7e3e78e55313f96b48f7db94f485bd9d671726847e983. The simulator and crossed grid are described in § 6.5 . Oracle-only calibration fixed the weak, medium and strong effects at 0.04296875, 0.078125 and 0.140625, targeting median log e e e -values of 3, 10 and 30 and attaining 3.23815054017, 10.328241766 and 30.9656204875. Calibration stopped within ten percent of target; its values were committed before the other new statistics were computed.
Amendment 1 followed the initial stop on missing ERK-like validation quantities. It leaves V, contrast tempering and W undefined on those cells; uses the 15-value mixture for selection while reproducing the baseline study’s native 13-value mixture in the gate; restricts decision-relevant cells to common defined candidates; and excludes candidates using only the simulator’s null cells. It fixes 31 validation draws and the calibration stopping rule. The restarted baseline reproduction gate passed on all 51 cells, with 10,824 comparisons, zero log-e e e -value differences and zero rejection mismatches. The baseline study’s pixel-world Kish factors and residual correlations were not saved; their existing C1 results were retained without back-calculating parameters.
Table 62. Regime-map R5: largest null counts.
Table 63. Regime-map R6: mean alternative and worst decision-relevant rates.
Selection step 1 excluded no candidate: every simulator null maximum was zero. Step 2 gave the highest worst-cell rate, 0.8125, to V. Step 3 treated W(3/4), at 0.78125, as tied within 1/32; mean simulator rates 0.634837962963 and 0.625192901235 selected V. Step 4 was not needed. There were 98 common decision-relevant cells. Q2 found a paired median V-minus-M deficit of at least half log 15 in 28 of 54 weak-effect cells, a fraction of 0.518518518519.
Table 64. Regime-map predictions dated 8 November 2026. Percentages are prior confidence, not pass thresholds.
The main driver stopped on an analytic-reference lookup after all other 230 cells were saved. The corrected reference handler used the saved per-replicate residual correlations, reused C0 when the Kish factor equalled one, and completed the reference statistics; all earlier outputs stayed unchanged. Exact total main CPU time was not captured. The original stop records were preserved.
ERK validation-tuning check
Protocol v1, dated 8 November 2026, SHA-256 f48d4ac7460886e104720e052d3b9d3cfb4b6c69 5a8616028082c161bce67e79, stopped because the frozen M4 assets were absent on the new machine. Version 2, SHA-256 baf5090abb2b4c4578056f42ba1efbee558f5d4862296573 35b5c9337985fb2d, ran on the old machine, defined validation bags and added weighted mixtures without changing the four predictions. The reproduction gate recomputed C0 at ten and thirty minutes within 0.01 of M4 and reproduced the rounded C1 values 349 and 218 from X.
Table 65. ERK validation-tuning check: natural-log e-values at 10 minutes (exploratory).
Continued on next page The dated predictions and outcomes were: M reaches E at least 100 at both horizons (0.90), held; V does so (0.75), failed; R stays below 100 at ten minutes (0.65), failed; and M remains within log 15 of the best constant-grid member at both horizons (1.00), held. The gaps were 2.70805020020578 and 2.70802211364344, below log 15 = 2.70805020110221.
Table 104.
Table 66. ERK validation-tuning check: natural-log e-values at 30 minutes (exploratory).
Table 67. ERK factors: the Kish median is descriptive over eligible test origins.
DEVIATION_01 changed a new accumulator from float32 to explicit float64, after the first attempt had passed the reproduction gate. The deviation and fix were committed before reopening. A formatting-only readout step later converted completed-table booleans to native JSON; it reopened no field and recomputed no score or statistic. The cumulative log records 52 ERK file opens: 26 trajectory-member opens and 26 command-snapshot opens, including both attempts. There were zero action-registry opens, zero historical outcome or score-cache reads and no refitting.
ERK field inventory
The ERK field inventory used archive metadata and the projects’ retained identity and access records, without opening a biological archive member. Declared TRAIN, TEST and M5 assignments were preserved. Remaining fields were classified OPENED-OR-UNKNOWN whenever their lifetime opening history was missing or ambiguous; absence of a matching record was not treated as proof of non-access. The result was 67 fields OPENED-OR-UNKNOWN and zero UNOPENED. During the inventory, a source preview showed at least one hard-coded historical ERK summary value; no outcome value was reproduced, and source previews were stopped. This does not affect M5, whose field was opened only under its registration.
Weighted mixture on DIAMOND (post hoc)
The registered run’s saved per-unit records do not contain every unit’s e-values at all 15 grid values, so the third version of this paper reported no value. The over-betting study recomputed them from the saved scores (K.8): W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected in 96 of 96 ACTIVE units and 0 of 192 NULL units. The score bags were recomputed from the saved predictions by the score-rule study’s registered-rule scorer.
Name map
Table 68. Internal labels in records and descriptive names in the paper.
Over-betting study
The protocol is dated 9 November 2026, SHA-256 40aa36ee493192bee2d6e07be42ca955 048265138f80437c3916d89763f273e9. Parts 1–3 computed temperature curves, the validation-weighted mixture on DIAMOND, and online tempering on saved units. The analysis is exploratory: every unit had already been generated and analyzed. IRIS and TD-MPC2 used stored 128-column score bags. DIAMOND score bags were recomputed from saved predictions by the score-rule study’s registered-rule scorer. No new unit was generated. The reproduction gates made 40,730 comparisons on 4224 IRIS and TD-MPC2 test units and 384 DIAMOND validation and test units, with zero differences in log e-values and identical rejection decisions.
The fine grid is 2 − k / 4 2^{-k/4} 2 − k /4 , k = 0 , … , 56 k = 0, \ldots, 56 k = 0 , … , 56 . Best-factor ties go to the smaller factor; break-even is the largest grid factor with positive log e-value, or undefined if none is positive. A collapse is L ( λ f ) < 0 L(\lambda_f) < 0 L ( λ f ) < 0 with L ∗ ≥ log 20 L^* \ge\log20 L ∗ ≥ log 20 ; an over-bet has λ f > λ ∗ \lambda_f > \lambda^* λ f > λ ∗ . Checks (a), (b) and (c) passed in 4512 of 4512 units each. The original check (d) passed in 4430 and failed in 82. AMENDMENT_01, dated 9 November 2026 and committed after these results, retained prediction 1 as contradicted and replaced (d) by (d1), the grid-member bound, and (d2), the half bound conditional on a nonnegative fine-grid maximum. The corrected checks passed in 4512 of 4512 and 3114 of 3114 applicable units. All 82 original failures had negative fine-grid maxima.
Table 69 continued
Table 69. Over-betting study: original checks. Each passed count is out of the group total; the final column retains failures of original check (d).
Table 69 (PDF p. 103).
Table 70. Over-betting study: DIAMOND. Best and break-even factors are medians over all test units of each group; Kish, contrast and validation factors are fixed per group.
Table 71. Over-betting study: IRIS pixels. Best and break-even factors are medians over all test units of each group; Kish, contrast and validation factors are fixed per group.
Table 72. Over-betting study: IRIS tokens. Best and break-even factors are medians over all test units of each group; Kish, contrast and validation factors are fixed per group.
Table 73. Over-betting study: TD-MPC2. Best and break-even factors are medians over all test units of each group; Kish, contrast and validation factors are fixed per group.
The eligible group-factor pairs for prediction 2 were the twelve full-execution IRIS decoded-pixel or TD-MPC2 groups in which C1 or C1c rejected in fewer than 24 of 32 units while W(3/4) rejected in at least 24. The median factor-to-best ratio, over every unit of each eligible group, exceeded two in all twelve; its minimum was 6.2594373676413. The online-mixture identity passed in every unit, with largest absolute difference 5.00222085975 × 10 − 12 5.00222085975 \times10^{-12} 5.00222085975 × 1 0 − 12 .
Table 74. Over-betting study: DIAMOND. Rejections by group, including NULL.
Table 75. Over-betting study: IRIS pixels. Rejections by group, including NULL.
Table 76. Over-betting study: IRIS tokens. Rejections by group, including NULL.
Table 77. Over-betting study: TD-MPC2. Rejections by group, including NULL.
Table 78. Over-betting study: predictions dated 9 November 2026 and mechanical outcomes.
ERK site-count check
Part E of the over-betting protocol used the nine M4 test fields and the four training fields at ten and thirty minutes, with the frozen models, bags, design law, masks, variances and aggregation. The first attempt stopped at its reproduction gate: six of eight comparisons failed, while V passed at both horizons. No adaptive statistic or prediction was evaluated. The cause was conversion of outcomes to float64 before normalization. The recovery removed that conversion and the explicit casts of the normalization constants; the pooled-score accumulator and subsequent log-factor accumulation retained float64. The fix and deviation were committed before reopening. RECOVERY_01 is exploratory; the results are retained at commit 62998d8.
The recovery gate passed all eight comparisons: C0, C1, V and M at both horizons, within max ( 10 − 6 , 10 − 9 ∣ L ∣ ) \max(10^{-6}, 10^{-9}|L|) max ( 1 0 − 6 , 1 0 − 9 ∣ L ∣ ) . The selected ρ \rho ρ maximizes the median validation-field log e-value per model and horizon, with ties to the larger ρ \rho ρ . The grid has 15 values 2 − g 2^{-g} 2 − g , g = 0 , … , 14 g = 0,\ldots,14 g = 0 , … , 14 . M ρ M_{\rho} M ρ averages complete grid-member e-values; W ρ ( 3 / 4 ) W_{\rho}(3/4) W ρ ( 3/4 ) assigns 3/4 to the selected member and 1/56 to each other member. At zero-cell origins the original neutral-factor convention applies.
Table 79. ERK site-count check, 10 minutes (exploratory). Natural-log e-values.
Table 80. ERK site-count check, 30 minutes (exploratory). Natural-log e-values.
Table 81. ERK site-count check: validation-selected site-count factors and their ranges on test fields.
The predictions dated 9 November 2026 both held: prediction 7 (70%) required the four-model V ρ V_{\rho} V ρ e-value to reach 100 at both horizons, and prediction 8 (55%) required positive V ρ V_{\rho} V ρ log e-values for all four models at both horizons. The four-model values were 350.54383590664 and 206.07503526448; every per-model value was positive.
The recovery opened 26 ERK files: 13 outcome members and 13 command snapshots. Cumulative counts over the two attempts were 52 file opens, 26 outcome-member opens and 26 command-snapshot opens, over the same 13 fields. Historical outcome or score-cache reads, refits, network requests, installs, pushes, uploads and contacts were zero. No further recovery was authorized.
The robot-data study
Registration and order of events
The feasibility check (commit 97ec197) opened images of official test trajectories 0–15 only, for a constructed-null smoke test. It established computational feasibility and a working law with unverified parts. No recorded-action/image score, e-value, rank, p-value, likelihood ratio or action-effect result was computed. Protocol v0 had SHA-256 36f0fc6cc92a99e6327187e7a66a1a2d0d22eeb8c7f5bf44155a9c6ebd7b230f; v1 had SHA-256 db523561602736c7f0f3cea782e7773926794d80307a2900345300c07b128dfb. Version 1 fixed the eligible set and count criteria, fixed the pool and disjoint stream draws, added NULL V and W(3/4) at the ACTIVE-selected factor, added the command-record gate, and stated training-pool and checkpoint-selection limits. The primary statistic, laws, origins, power criterion and six dated predictions were unchanged.
The build pilot (fec685d) ran the full NULL pipeline on 48 trajectories outside the study set, with sixteen validation and thirty-two test units. No ACTIVE e-value was computed. The command-record check compared 3346 study rows with 14,294 pool rows. Its four p-values were 0.20207805209385643, 0.3420444752342814, 0.1639206772899076 and 0.40300540275743, all above the gate’s 0.001. Synthetic rehearsals covered confirmation true and false and interrupted resume without image decoding. The exact NULL pilot replay reproduced 294 files byte for byte. The frozen code commit was 68ad9b3775a3d3e1d8e53570b5f0f5aaad2f7ce0.
Table 82. Robot-data study: order of events. Commit timestamps and recorded event times are UTC.
Registration 4wj7k was public with no embargo. The raw registration timestamp has no zone marker and is read as UTC. The beacon rule was the first beacon round scheduled at least two minutes after the start of the minute in which the registration was recorded. Round 6629230 supplied randomness 6a9930ca5dcc2195b3e5ce530bfd144b49f3d0b62577a69747fe52c4b135e962. The master seed was 7901ecac4b253d5465f0db8f107ff7bcdd0d72be1747d1124c66f9a4cd8ac97d, the SHA-256 of the ASCII protocol hash, a vertical bar and the randomness, without a newline.
CONF_STOP_01 occurred before any network request, permit change or study-image access. Our run instruction required an idle GPU and use of the frozen checker on confirmatory archives, while that checker supported synthetic rehearsals only. Nothing had been fetched, changed or opened. The later instruction resolved both requirements. The single run completed with no deviation, interruption or resume in 2.591607 hours. A separately written checker, after the results commit, reproduced 10,516 unit log e-values and all decision counts with zero disagreements. After the reports quoted here were written, the first author posted nine files to the registered component (2ksxa) on 10 November 2026 (OSF file times 22:37 UTC) and made the component public the same day: the results note P5_RESULTS_UPDATE_20261110.md, P5_CONF_REPORT.md, SUMMARY_P5.tsv, DECISION.json, P5_BEACON_REPORT.md, REGISTERED_RUN_PERMIT.json, OUTPUT_MANIFEST.tsv, ACCESS_AUDIT.json and CONF_STOP_01.md. The draft OSF_UPDATE_P5.md was not posted. The posting is reported by the author; no local record of it exists.
Design details
Table 84. Robot-data study: original data-file hashes, as fixed in protocol v1.
iVideoGPT code revision was d601d5cac9e96c6aa0c17cb37ed6a7c7ca1fb210 and checkpoint revision 2863382adad16f49a8a6bc8e61c08a03e98bc5e5. Tokenizer and Transformer weights had SHA-256 f29ac860b2ab8852d504d94e1aac0c39dcee45ea90856a7f44dc1a7e46dd1ec and c7ac81f413f697883a8f770d40d0bb5b109670d0b838d5cef35848778eeca7bb0, respectively. The released action-conditioned checkpoint was unchanged. Eligibility required exactly equal commands at every even/odd frame pair. Of test trajectories 16–255, trajectory 182 was excluded, leaving 239; none was replaced. The split was 80 validation and 159 test trajectories, shared by both laws and arms. Each trajectory used origins t = 2 , 4 , … , 28 t = 2, 4, \ldots, 28 t = 2 , 4 , … , 28 , with one context frame and the next frame as target.
Table 83. Files posted to component 2ksxa, as reported by the author. SHA-256 values of the local copies.
The empirical pool contained 14,294 commands from 1021 eligible training trajectories; training trajectories 371, 29538 and 29627 were excluded. Its SHA-256 was b08500e9b43e786ec852411b59fedd35ae19715fe01486f7ec916da927bbfb9e. The streams bag, tempc and repl each permuted the pool with its own seed, dropped entries used by preceding streams of the unit, and assigned consecutive blocks to origins. No pool entry was used twice within a unit, apart from shared column zero in the contrast bag. Disjointness was by identifier, not command value. L-par drew fresh candidates from the fitted law: independent planar uniforms on (−0.07, 0.07) and two discrete components with the recorded five-category probabilities. The fitted-law SHA-256 was 1fbfbd14256e0dc906c36815d6655c4315af35e418ed74eb11fda3c5aae6705b.
The tokenizer used 256 context tokens, one separator and sixteen target tokens. Inference used bfloat16 weights, eager attention, deterministic algorithms, TF32 off and padded batches of eight. Log-softmax over the full vocabulary of 16,386 was float32; sixteen log-probabilities were cast to float64, floored at log 10 − 12 \log10^{-12} log 1 0 − 12 and summed in float64. Commands identical at model input precision shared a score. Main bags had 128 columns; validation contrast bags had 32. Scores, site log-probabilities, commands, pool identifiers, aliases and token identifiers were stored; no images were copied.
Seeds were the first four big-endian bytes of SHA-256 of the ASCII namespace P5|<PHASE>|<law>|<arm>|<split>|unit<NNNN>|<stream>; the split used P5|<PHASE>|split. PCG64 was used throughout; the confirmatory phase was CONF|<master>. Validation choices were locked before test-image access.
Working assumptions and limits
The recorded-command arm required three conditions: commands at origins are independent draws from one law independent of scene; under L-emp the model treats commands identically whether or not they occurred in training, while L-par instead requires the fitted law to be right; and the model was fixed without the study units’ image-command pairs. The collection software and configuration of 2017 could not be established. The pool commands were from training trajectories, and use of the official test set for checkpoint selection cannot be excluded.
A context frame may already show the start of its command’s motion. Origin zero was excluded because its reset timing was unverified; this did not establish timing at other origins. Under NULL timing does not matter; under ACTIVE it can only change power. The feasibility record supported the working marginals, two-frame repetition and low refresh-lag correlations, but did not verify the exact joint conditional law, exception process, timing or safety-conditioned retention. The constructed NULL was exact under either law and did not need the ACTIVE assumptions.
Full results
Table 85. Robot-data study: NULL rejections out of 159 under each law, for all twelve statistics.
Table 86. Robot-data study: recorded-command results, ten statistics under both laws. Natural logarithms; 159 units per row.
Table 87. Robot-data study: four validation choices locked before test-image access.
Table 88: Robot-data study: replacement diagnostic by law, arm and split. Positive means the frozen one-sided t-test had p < 0.05 p < 0.05 p < 0.05 ; descriptive, outside the criteria.
Table 88.
Predictions
Table 89: Robot-data study: six predictions dated 9 November 2026, unchanged in protocol v1.
Table 89.
Descriptions from the stored scores
D1 and D2 were planned in protocol Section 7; D3–D5 and the scatter below are exploratory. The fine grid had 65 values 2 − j / 4 2^{-j/4} 2 − j /4 , j = − 8 , … , 56 j = -8, \ldots, 56 j = − 8 , … , 56 . Best-factor ties chose the largest factor; break-even was the largest grid factor with nonnegative log e-value, reported above four if profitable at four, or undefined if every grid value was negative. Quartiles used linear interpolation.
Table 90: Robot-data study: planned descriptive best factors. Quartiles use linear interpolation.
Table 90.
Table 91: Robot-data study: planned descriptive break-even factors on the fine grid.
Table 91.
Table 92: Robot-data study: planned rejections by number of origins, L-emp. Each count is out of 159.
Table 92. Table 92 continued
Table 93. Robot-data study: planned rejections by number of origins, L-par. Each count is out of 159.
The pooled score did not reject in sixteen L-emp and seventeen L-par units; thirteen and twelve had negative log e-values. The median worst-origin share of negative net unit loss was 1.0812946 and 1.3009535, and the median share of the worst two origins was 1.6054236 and 1.8323682. Shares can exceed one because positive origins offset negative contributions. Command magnitudes below use recorded units, without an assumed physical-unit conversion.
Table 94. Robot-data study: the sixteen L-emp units in which the pooled score did not reject (exploratory).
Figure 10. Pooled versus weighted-mixture log e-value, ACTIVE L-emp (exploratory). Symmetric-log horizontal axis, equal-values line and log 20 reference lines.
Table 95. Robot-data study: commands at the worst origins and at all other origins (exploratory). Magnitudes use recorded command units.
W ( 3 / 4 ) W(3/4) W ( 3/4 ) rejected under both laws in 155 units, under neither in three, under L-par alone in one and under L-emp alone in none. Pearson and Spearman correlations of pooled unit log e-values were 0.9939136 and 0.97335503; for W ( 3 / 4 ) W(3/4) W ( 3/4 ) they were 0.98242546 and 0.97434022. No correlation test was performed. Sums over units are descriptive: units share the candidate pool under L-emp, so the product is not a registered e-value.
Table 96. Robot-data study: sums of stored trajectory log e-values (exploratory). Units share the candidate pool under L-emp, so the product is a description and not a registered e-value.
Compute
One NVIDIA GeForce RTX 5070 Laptop GPU; elapsed wrapper time 2.591607 hours. Peak sampled device memory was 3223 MiB, from one-second nvidia-smi samples, not an allocator peak. There was one invocation, no interruption and no resume.