betaFeedback

WorksComputer Science

Operator-Mismatch Contrastive Learning for World-Relative Trajectory Coherence

Abstract

World models are increasingly judged by how well they rank plausible futures, yet plausibility alone is the wrong target. The same scene can evolve under many latent rules, and a future that is smooth and physically valid under one rule can be incoherent under the rule actually in force. We formalize the resulting failure, operator-blindness, and show that it is systematically hidden by standard evaluation: a scorer can attain near-perfect generic ranking while separating current-world from other-world futures no better than chance. We introduce Operator-Mismatch Contrastive (OMC) learning, whose semantic hard negatives are valid-but-wrong-operator trajectories, and a matching protocol that measures world-relative coherence directly. On a controlled synthetic benchmark with in- and out-of-distribution operator parameters, across two trajectory-scoring backbones, and on feature-level external validation with CoPhy ballsCF, OMC raises operator-sensitive scoring far above a chance baseline that nonetheless achieves 0.99 generic top-1. We prove that the wrong-operator negative is necessary both for the protocol to detect the failure and for the objective to fix it, and position OMC as an auxiliary coherence signal (a trajectory-coherence scorer, not a generative model, planner, or control agent) that can be attached to existing world-model representations.

Introduction

A world model should anticipate futures, and we usually grade it by whether it ranks a true continuation above corrupted or random alternatives. This target—generic plausibility—is necessary but quietly insufficient. The same multi-object scene can evolve under many latent rules (gravity, damping, attraction, restitution), and a future that is perfectly plausible under one rule can be incoherent under the rule actually governing the present context. A world model must therefore answer not only “is this future plausible?” but “under which world is it coherent?”

The distinction mirrors an asymmetry in human cognition. People reason about what is factually plausible, but they also imagine freely—recombining familiar elements into scenes that need not match the present world, yet remain internally consistent under different rules [1, 2, 24, 25]. A capable world model arguably needs both faculties: the generativity to entertain plausible-elsewhere futures and the discernment to recognize which of them is coherent here. This paper isolates the second, measurable faculty—world-relative coherence—and treats free generativity as motivation rather than claim.

Where should coherence come from? Contemporary world models acquire physical regularities in two broad ways. One line learns them implicitly from data: joint-embedding predictive architectures acquire intuitive physics from natural video without hardwired core knowledge [5, 6, 23], and latent-dynamics models learn dynamics from interaction [3, 4]. A complementary line imposes physical structure explicitly via physics-informed losses or conservation-law architectures [7, 8, 9]. Our contribution is orthogonal to this axis: whether physics is learned implicitly or imposed explicitly, the resulting scorer can still be operator-blind, judging a future plausible without registering that it is coherent only under a different world than the present one.

We make the failure precise (§ 3). A context CC induces a latent operator λC\lambda_C, and a candidate future τ\tau is scored by an energy Eθ(λC,τ)E_\theta(\lambda_C,\tau), lower meaning more coherent. We distinguish coherent-here (τsame\tau_\mathrm{same}, valid under the current operator), plausible-elsewhere (τwrong\tau_\mathrm{wrong}, valid under a different operator), and incoherent (τnoise\tau_\mathrm{noise}). Operator-blindness is the inability to order τsame\tau_\mathrm{same} below τwrong\tau_\mathrm{wrong} even when noise is rejected—and, as we show formally, it is invisible to any generic metric whose candidate set omits τwrong\tau_\mathrm{wrong}.

Our remedy is a change in what counts as a negative. OMC inserts valid futures from a different operator as semantic hard negatives in a contrastive ranking objective (§ 4). Such a negative is a legitimate physical rollout—wrong only here—so contrasting against it forces the scorer to track the inferred operator rather than surface plausibility.

Contributions.

  • A failure mode, formalized. We define operator-blindness and prove that generic top-1 ranking cannot detect it whenever the candidate set excludes wrong-operator futures (§ 3, Prop. 1).

  • An objective and a protocol. OMC uses wrong-operator hard negatives; we pair it with operator-sensitive ranking, coherence stratification, and an operator×\timestrajectory energy matrix, and prove that the wrong-operator negative is what drives the trained ordering (§ 4, Prop. 2).

  • Evidence. On synthetic IID/OOD operator parameters, two backbones, and feature-level CoPhy ballsCF, OMC lifts operator-sensitive metrics well above a baseline that retains ≈0.99\approx0.99 generic top-1 (§ 6–§ 8).

OMC is a coherence signal for trajectory scoring, not a generative architecture or planner; we substantiate exactly the relational claim above and delimit the rest in § 9. In one line: generic top-1 answers “is this future plausible?”, whereas OMC asks “is it coherent here?”—and our central finding is that the first question can be answered almost perfectly while the second is answered at chance.

Related Work

World models. Latent-dynamics and Dreamer-style models learn dynamics and imagine rollouts for control [3, 26, 4, 27]; joint-embedding predictive architectures learn predictive representations and acquire intuitive physics from video [5, 6, 23, 28]. These are representational/generative approaches; OMC is a complementary objective/evaluation signal that can sit atop such representations, and the latent-operator inference we use is standard rather than a contribution.

Explicit physical structure. A parallel tradition encodes physics through physics-informed losses [7] or conservation-law architectures [8, 9]. OMC neither imposes nor presupposes such structure; it asks the scorer to learn a relative coherence signal, and our results show that neither implicit nor explicit physics by itself yields operator-sensitive scoring.

Operator-blindness diagram

Figure 1. Operator-blindness. Under operator AA, both τsame\tau_{\mathrm{same}} and τwrong\tau_{\mathrm{wrong}} (valid under a different operator BB) are physically plausible; only τsame\tau_{\mathrm{same}} is coherent here. A scorer that merely rejects τnoise\tau_{\mathrm{noise}} may still fail to separate the dashed pair—and, by Prop. 1, generic ranking cannot reveal it.

Contrastive and energy-based models. Structured contrastive models such as C-SWM [10] and the InfoNCE family [31, 11, 12, 17, 18, 19, 20, 21, 22] learn representations from contrastive negatives. OMC differs not in using a contrastive energy—that is standard—but in the semantics of the hard negative: a valid trajectory from a mismatched operator. Hard-negative mining [19] sharpens such objectives, but its negatives are defined by embedding similarity rather than operator identity. Ordinary structured negatives (random, state-corrupted, temporal) do not, by themselves, induce operator-sensitive scoring (§ 7). CoPhy [13] (see also CLEVRER [32]) supplies counterfactual mechanics with hidden confounders, a natural external testbed used here for validation.

System identification and context inference. A large literature infers environment parameters or latent task context for prediction and control—system identification, adaptive control, and meta-reinforcement-learning methods that encode task/dynamics context for fast adaptation [30, 14, 15, 16]. These aim to use the inferred context to act; OMC instead asks whether a scorer’s compatibility function is sensitive to that context at all, and supplies a negative-construction and protocol to enforce and measure it. Interactive world models and benchmarks [29] emphasize action-conditioned controllability and long-horizon rollout; OMC is complementary, targeting operator-relative coherence of the scorer rather than control.

Problem Formulation

Generative process. A latent operator λ⋆∼p(λ)\lambda^\star\sim p(\lambda) parameterizes the dynamics; a context CC and its true future are draws from p(⋅∣λ⋆)p(\cdot\mid\lambda^\star). A context encoder infers a posterior over the operator,

qϕ(λ∣C)=N(μϕ(C),diag⁡σϕ2(C)),λC≔μϕ(C),(1)q_\phi(\lambda\mid C) = \mathcal{N}\left(\mu_\phi(C), \operatorname{diag}\sigma_\phi^2(C)\right), \qquad\lambda_C \coloneqq\mu_\phi(C), \tag*{(1)}

and a scorer assigns an energy Eθ(λC,τ)∈RE_\theta(\lambda_C,\tau) \in\mathbb{R} to a candidate future τ=(s0,…,sT)\tau= (s_0,\ldots,s_T), with lower energy denoting greater coherence. Coherence is identified with operator-conditional likelihood: τ\tau is coherent under λ\lambda to the extent that p(τ∣λ)p(\tau\mid\lambda) is high, the ideal scorer tracking Eθ(λ,τ)∝−log⁡p(τ∣λ)E_\theta(\lambda,\tau) \propto-\log p(\tau\mid\lambda).

Method familyLearns dynamicsContrastive negativesWrong-operator futureWorld-relative coherence
Latent-dynamics / Dreamer✓×××
JEPA / I-JEPA∼∼××
C-SWM / structured contrastive✓✓××
CPC / InfoNCE family∼✓××
Physics-informed / HNN / LNN✓×××
Meta-RL context inference∼×××
OMC (ours)×✓✓✓

Table 1. Positioning of OMC against related method families (citations in the text of this section). “Wrong-operator future” marks whether a method ever contrasts against a valid future from a different operator; “World-relative coherence” marks whether evaluation tests coherence under the in-force operator (not merely generic plausibility). ✓ yes, × no, ∼ partial/implicit.

What the operator can be. We keep λ\lambda abstract on purpose. Although our experiments instantiate it as simulator parameters, the same formulation covers any latent that selects which dynamics are in force: a latent dynamics code, a hidden confounder (as in CoPhy), an environmental law or physical regime, a task/parameter context in meta-RL, or an intervention setting in a causal model. Operator-blindness is therefore not a quirk of one simulator but a generic failure of any scorer whose compatibility function is insensitive to the latent that governs the present context.

Candidate classes. From the generative process we define

τsame∼p(⋅∣λ⋆)(coherent-here; the positive),\tau_{\mathrm{same}} \sim p(\cdot\mid\lambda^\star) \qquad\text{(coherent-here; the positive),}
τwrong∼p(⋅∣λ′), λ′≠λ⋆(plausible-elsewhere; operator/law mismatch),\tau_{\mathrm{wrong}} \sim p(\cdot\mid\lambda'),\ \lambda' \ne\lambda^\star\qquad\text{(plausible-elsewhere; operator/law mismatch),}
τnoise∼pcorrupt(incoherent: corrupted, time-shuffled, or impossible).\tau_{\mathrm{noise}} \sim p_{\mathrm{corrupt}} \qquad\text{(incoherent: corrupted, time-shuffled, or impossible).}

A world-relatively coherent scorer should satisfy

Eθ(λC,τsame)<Eθ(λC,τwrong)<Eθ(λC,τnoise).(2)E_\theta(\lambda_C,\tau_{\mathrm{same}}) < E_\theta(\lambda_C,\tau_{\mathrm{wrong}}) < E_\theta(\lambda_C,\tau_{\mathrm{noise}}). \tag*{(2)}

Definition 1 (Operator-blindness). A scorer is operator-blind at CC if

Pr⁡[Eθ(λC,τsame)<Eθ(λC,τnoise)]→1,\Pr[E_\theta(\lambda_C,\tau_{\mathrm{same}}) < E_\theta(\lambda_C,\tau_{\mathrm{noise}})] \to1,
Pr⁡[Eθ(λC,τsame)<Eθ(λC,τwrong)]=12,\Pr[E_\theta(\lambda_C,\tau_{\mathrm{same}}) < E_\theta(\lambda_C,\tau_{\mathrm{wrong}})] = \frac{1}{2},

i.e. it rejects noise but orders same- and wrong-operator futures at chance.

The next statement is why operator-blindness goes unnoticed: it is invisible to any generic ranking metric whose candidate pool excludes τwrong\tau_{\mathrm{wrong}}.

Proposition 1 (Generic ranking cannot detect operator-blindness). Let the generic candidate set be {τsame}∪Neasy\{\tau_{\mathrm{same}}\} \cup\mathcal{N}_{\mathrm{easy}} with Neasy\mathcal{N}_{\mathrm{easy}} containing only corrupted/temporal/random negatives, and suppose Eθ(λC,τsame)<Eθ(λC,e)E_\theta(\lambda_C,\tau_{\mathrm{same}}) < E_\theta(\lambda_C,e) a.s. for every e∈Neasye \in\mathcal{N}_{\mathrm{easy}}. Then top1-acc =1= 1 regardless of the ordering of Eθ(λC,τsame)E_\theta(\lambda_C,\tau_{\mathrm{same}}) and Eθ(λC,τwrong)E_\theta(\lambda_C,\tau_{\mathrm{wrong}}). In particular an operator-blind scorer and a perfectly coherent scorer are indistinguishable under generic top-1.

Proof. Top-1 accuracy is determined solely by whether τsame\tau_{\mathrm{same}} attains the minimum energy within {τsame}∪Neasy\{\tau_{\mathrm{same}}\} \cup\mathcal{N}_{\mathrm{easy}}; since τwrong\tau_{\mathrm{wrong}} is absent from this set, the same/wrong ordering does not enter the event. The hypothesis makes the event certain, so top1-acc =1= 1 for both scorers. □\square

Proposition 1 motivates the protocol of Section 5: to detect operator-blindness one must explicitly insert τwrong\tau_{\mathrm{wrong}} into the comparison, which is exactly what LAW_PAIR and SAME_LT_WRONG do.

Operator-Mismatch Contrastive Learning

Scorer and operator encoder. The context encoder is a permutation-invariant (DeepSets [33]) network over the context transitions, producing μϕ(C)\mu_{\phi}(C), log⁡σϕ2(C)\log\sigma_{\phi}^{2}(C) with latent dimension dλd_{\lambda}. The energy scorer is object-centric, in the spirit of interaction networks [34] and slot-based models [21]: with stis_{t}^{i} the state of object ii at step tt and atia_{t}^{i} its action, a per-object self term scores each transition under the operator code, and an optional pairwise term scores object pairs,

Eθ(λC,τ)=∑t∑ifselfθ(sti,ati,st+1i,λC)+∑t∑i<jfrelθ(sti,stj,st+1i,st+1j,λC)⏟pairwise head (optional)(3)E_{\theta}(\lambda_{C},\tau)=\sum_{t}\sum_{i}f_{\mathrm{self}}^{\theta}(s_{t}^{i},a_{t}^{i},s_{t+1}^{i},\lambda_{C})+\underbrace{\sum_{t}\sum_{i<j}f_{\mathrm{rel}}^{\theta}(s_{t}^{i},s_{t}^{j},s_{t+1}^{i},s_{t+1}^{j},\lambda_{C})}_{\text{pairwise head (optional)}} \tag*{(3)}

with masking over inactive slots and a sum reduction. The pairwise term is an architectural option, disabled in the main objective and studied as a component ablation (§ 6). A direct-context variant keeps the same objective but drops the explicit operator bottleneck, isolating the objective from the backbone.

Candidate set. For each context we form CC of size M=5M=5: the positive plus one negative of each type, C={τsame,τstate,τtemporal,τwrong,τrand}C=\{\tau_{\mathrm{same}},\tau_{\mathrm{state}},\tau_{\mathrm{temporal}},\tau_{\mathrm{wrong}},\tau_{\mathrm{rand}}\}, where τstate\tau_{\mathrm{state}} perturbs future positions/velocities, τtemporal\tau_{\mathrm{temporal}} reverses or permutes the future, τrand\tau_{\mathrm{rand}} is impossible, and τwrong\tau_{\mathrm{wrong}}—the OMC negative—is a valid future from a different operator (distinct identity or parameter distance >δ>\delta). Batches contain distinct operators, so each τwrong\tau_{\mathrm{wrong}} is a genuine wrong-operator rollout.

Objective. We minimize an InfoNCE ranking loss with logits equal to negative energies,

LNCE=−log⁡exp⁡(−Eθ(λC,τsame))∑τ∈Cexp⁡(−Eθ(λC,τ))(4)\mathcal{L}_{\mathrm{NCE}}=-\log\frac{\exp(-E_{\theta}(\lambda_{C},\tau_{\mathrm{same}}))}{\sum_{\tau\in C}\exp(-E_{\theta}(\lambda_{C},\tau))} \tag*{(4)}

regularized by a KL term on the operator posterior [35] and a stability term tying two halves of a context to the same operator, LKL=KL⁡(qϕ(λ∣C)∥N(0,I))\mathcal{L}_{\mathrm{KL}}=\operatorname{KL}(q_{\phi}(\lambda\mid C)\mathbin{\|}\mathcal{N}(0,I)) and Lstable=∥μϕ(C1:k)−μϕ(Ck+1:)∥22\mathcal{L}_{\mathrm{stable}}=\|\mu_{\phi}(C_{1:k})-\mu_{\phi}(C_{k+1:})\|_{2}^{2}. The total objective is

L=LNCE+αocclLOCCL+βKLLKL+αstableLstable,(5)\mathcal{L}=\mathcal{L}_{\mathrm{NCE}}+\alpha_{\mathrm{occl}}\mathcal{L}_{\mathrm{OCCL}}+\beta_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}}+\alpha_{\mathrm{stable}}\mathcal{L}_{\mathrm{stable}}, \tag*{(5)}

where LOCCL\mathcal{L}_{\mathrm{OCCL}} is a symmetric InfoNCE over an operator×\timestrajectory energy matrix (Appendix § 13). In all reported runs αoccl=0\alpha_{\mathrm{occl}}=0: the matrix term is computed for diagnosis but does not enter the gradient, so the effect we report comes entirely from the wrong-operator negative in Eq. (4); the energy matrix is an evaluation instrument, not a trained objective. The only further axis we vary is the architectural pairwise head of Eq. (3).

The wrong-operator negative is not merely helpful but necessary for the objective to induce the target ordering:

Proposition 2 (The wrong-operator negative induces coherence). For a sufficiently expressive scorer and in the population limit, the minimizer of Eq. (4) satisfies −Eθ(λC,τ)=log⁡p(τ∣λC)+c-E_{\theta}(\lambda_{C},\tau)=\log p(\tau\mid\lambda_{C})+c for a candidate-independent cc; hence Eθ(λC,τsame)<Eθ(λC,τwrong)E_{\theta}(\lambda_{C},\tau_{\mathrm{same}})<E_{\theta}(\lambda_{C},\tau_{\mathrm{wrong}}) whenever p(τsame∣λC)>p(τwrong∣λC)p(\tau_{\mathrm{same}}\mid\lambda_{C})>p(\tau_{\mathrm{wrong}}\mid\lambda_{C}). Conversely, if τwrong∉C\tau_{\mathrm{wrong}}\notin C the objective places no constraint on the relative energies of same- and wrong-operator futures, admitting operator-blind minimizers.

Algorithm 1 OMC training step (one minibatch)

  1. sample a minibatch with distinct operators across samples

  2. for each sample ii do

  3. Ci←C_i \leftarrow context; τsame(i)←\tau_{\mathrm{same}}^{(i)} \leftarrow valid future under the current operator

  4. negatives ←τstate,τtemporal,τwrong,τrand\leftarrow\tau_{\mathrm{state}}, \tau_{\mathrm{temporal}}, \tau_{\mathrm{wrong}}, \tau_{\mathrm{rand}} // τwrong\tau_{\mathrm{wrong}}: valid future from a different operator

  5. Ci←[τsame,τstate,τtemporal,τwrong,τrand]C_i \leftarrow[\tau_{\mathrm{same}}, \tau_{\mathrm{state}}, \tau_{\mathrm{temporal}}, \tau_{\mathrm{wrong}}, \tau_{\mathrm{rand}}]

  6. end for

  7. μ,log⁡σ2←Enc⁡(C)\mu, \log\sigma^2 \leftarrow\operatorname{Enc}(C); λ←μ+σ⊙ϵ\lambda\leftarrow\mu+ \sigma\odot\epsilon; Eθ[i,m]←Score⁡(λi,Ci[m])E_\theta[i,m] \leftarrow\operatorname{Score}(\lambda_i,C_i[m])

  8. L←CE⁡(−Eθ,y)+βKLLKL+αstableLstable(αoccl=0)\mathcal{L} \leftarrow\operatorname{CE}(-E_\theta,y) + \beta_{\mathrm{KL}}\mathcal{L}_{\mathrm{KL}} + \alpha_{\mathrm{stable}}\mathcal{L}_{\mathrm{stable}} \quad(\alpha_{\mathrm{occl}}=0)

  9. AdamW update on θ,ϕ\theta,\phi

OMC pipeline

Figure 2. OMC pipeline. A context encoder infers λC\lambda_C; an object-centric energy scorer ranks the candidate set with InfoNCE. The defining choice is the wrong-operator hard negative τwrong\tau_{\mathrm{wrong}}. The operator×\timestrajectory energy matrix / OCCL term is a diagnostic only (αoccl=0\alpha_{\mathrm{occl}}=0).

Proof sketch. Softmax cross-entropy over candidates is minimized when the induced selection distribution matches the true posterior that the positive is the in-context continuation; the noise-contrastive optimality result [12, 11] gives optimal scores s(τ)=log⁡p(τ∣λC)s(\tau)=\log p(\tau\mid\lambda_C) up to an additive constant, and s=−Eθs=-E_\theta yields the ordering. If τwrong\tau_{\mathrm{wrong}} never appears in CC, the loss is invariant to the value of Eθ(λC,τwrong)E_\theta(\lambda_C,\tau_{\mathrm{wrong}}), so minimizers need not satisfy Eq. (2)—the operator-blind regime. Together with Prop. 1, the wrong-operator trajectory is necessary both to train and to detect world-relative coherence.

Training-time requirement and inference-time usage. A natural objection is that constructing τwrong\tau_{\mathrm{wrong}} presupposes access to alternative-operator trajectories. We stress that this is a training-and evaluation-time requirement only. Wrong-operator rollouts are needed to build the contrastive candidate set during learning and to probe operator sensitivity during evaluation; they are not required at inference. Once trained, the scorer consumes a context CC and a single candidate τ\tau and returns an energy Eθ(λC,τ)E_\theta(\lambda_C,\tau) with no alternative-operator data in the loop. OMC is thus an attachable coherence signal: the cost it imposes is the availability of operator-labelled rollouts at train time, exactly the supervision our synthetic generator and CoPhy’s hidden confounder already provide.

Experimental Setup

Synthetic benchmark. LOWM-Synth is a 2D multi-object simulator (horizon T=20T=20) whose hidden operator is drawn from four families (gravity/damping, attraction/repulsion, collision elasticity, boundary behavior) with continuous parameters. test_iid samples operators from the training ranges; test_ood_param uses held-out parameter ranges of the same families, testing whether coherence generalizes beyond memorized parameters. Operators are metadata used to build negatives and to evaluate; they are never given to the model.

MetricCandidate setReads as
top1_acc{τsame}∪\{\tau_{\mathrm{same}}\}\cup easy negsgeneric plausibility (diagnostic)
law_pairfull set incl. τwrong\tau_{\mathrm{wrong}}P(τsame<τwrong)P(\tau_{\mathrm{same}}<\tau_{\mathrm{wrong}}); n/a if τwrong\tau_{\mathrm{wrong}} absent
same_lt_wrong{τsame,τwrong,τnoise}\{\tau_{\mathrm{same}},\tau_{\mathrm{wrong}},\tau_{\mathrm{noise}}\}P(τsame<τwrong)P(\tau_{\mathrm{same}}<\tau_{\mathrm{wrong}}), fixed 3-way set
full_stratification{τsame,τwrong,τnoise}\{\tau_{\mathrm{same}},\tau_{\mathrm{wrong}},\tau_{\mathrm{noise}}\}P(τsame<τwrong<τnoise)P(\tau_{\mathrm{same}}<\tau_{\mathrm{wrong}}<\tau_{\mathrm{noise}})
matrix_mrrB×BB\times B operator↔\leftrightarrowtraj.operator↔\leftrightarrowtrajectory retrieval

Table 2. Metric→\tohypothesis map. Each operator-sensitive metric uses a different candidate set; law_pair is defined only when a wrong-operator negative is present.

External validation and backbones. We additionally validate on CoPhy ballsCF [13] at the feature level (segmentation-derived object features, 64px, 20 frames; 35k/7.5k/7.5k split). Two backbones, LOWM (latent operator) and direct-context (no operator bottleneck), separate the objective from the architecture.

Metrics. We separate primary operator-sensitive metrics from a diagnostic one. Primary: law_pair (Pr⁡[Eθ(τsame)<Eθ(τwrong)]\Pr[E_\theta(\tau_{\mathrm{same}})<E_\theta(\tau_{\mathrm{wrong}})]), same_lt_wrong and full_stratification (the orderings of Eq. (2)), and energy-matrix mrr/top1 (operator×\timestrajectory retrieval). Diagnostic: top1_acc over the generic candidate set. By Prop. 1 the diagnostic alone cannot reveal operator-blindness, so we report it only to expose the trade-off. Formal definitions are in Appendix B. All energy-matrix metrics use matrix_size = 16, max_batches = 8 and are never compared across sizes. We report mean±\pmstd over 5 seeds (synthetic/CoPhy) or 3 (baselines); per-seed values are in Appendix D.

Reading law_pair. It is computed only over samples whose candidate set contains a wrong-operator negative. The no-law baseline has none, so its law_pair is reported as 0/n.a. (vacuous)—an instance of Prop. 1, not a behavior; its genuine chance-level operator sensitivity is read from same_lt_wrong≈\approx 0.5 (Table 7).

Negative settop1law_pairsame<wrmat_mrr
law-mismatch only0.419±0.0030.707±0.0020.711±0.0070.289±0.002
law-mismatch + random0.561±0.0010.708±0.0030.713±0.0030.292±0.007
all-4 (main OMC)0.384±0.1250.694±0.0070.693±0.0030.269±0.004
no-law (τwrong\tau_{\mathrm{wrong}} removed)0.996±0.0020.0000.506±0.0140.211±0.003

Table 7. Sensitivity to negative-set composition (LOWM, αoccl=0\alpha_{\mathrm{occl}}=0, no-pairwise; 3 seeds, test_iid).

Synthetic Diagnostics and Ablations

Reading guide. We organize the empirical sections as answers to five questions a reviewer would ask. R1: Does operator-blindness exist?—Table 3 and Figure 4. R2: Does OMC fix it?—Table 3, on both backbones and both IID/OOD splits. R3: Is the wrong-operator negative the active ingredient, rather than contrastive learning in general? —the negative-type and composition ablations (Table 4, Table 7) and the general world-model baselines (Table 8). R4: Does it generalize beyond a memorized simulator?—the OOD-parameter rows throughout and the external CoPhy validation (Table 9). R5: How should OMC be used?—as an attachable verifier signal, discussed in § 9. Each table below is tagged with the question it answers.

BackboneObj.Splittop1law_pairlaw_onlysame<wrmat_mrr
LOWMOMCIID0.375±0.1390.694±0.0050.411±0.0030.697±0.0060.272±0.012
LOWMOMCOOD0.363±0.1160.650±0.0060.359±0.0110.665±0.0090.271±0.009
LOWMno-lawIID0.996±0.0010.0000.196±0.0130.507±0.0100.213±0.003
LOWMno-lawOOD0.991±0.0010.0000.205±0.0040.514±0.0070.209±0.004
directOMCIID0.574±0.0140.709±0.0080.419±0.0080.723±0.0020.262±0.001
directOMCOOD0.562±0.0230.700±0.0120.409±0.0180.705±0.0110.259±0.005
directno-lawIID0.997±0.0010.0000.207±0.0070.508±0.0020.211±0.000
directno-lawOOD0.996±0.0000.0000.200±0.0070.502±0.0090.211±0.001

Table 3. Main synthetic results (5 seeds, mean±std; matrix_size = 16). LAW_PAIR, SAME_LT_WRONG, MATRIX_MRR are primary; TOP1 is diagnostic.

ObjectiveSplittop1law_pairlaw_gapsame<wrfull str.mat_mrr
all-neg (main OMC)IID0.375±0.1390.694±0.0051.30±0.600.697±0.0060.697±0.0060.272±0.012
all-neg (main OMC)OOD0.363±0.1160.650±0.0061.21±0.610.665±0.0090.665±0.0090.271±0.009
no-law-mismatchIID0.996±0.0010.0000.000.507±0.0100.507±0.0100.213±0.003
no-law-mismatchOOD0.991±0.0010.0000.000.514±0.0070.514±0.0070.209±0.004
law-mismatch onlyIID0.416±0.0070.700±0.0042.17±0.080.712±0.0040.676±0.0110.284±0.012
law-mismatch onlyOOD0.403±0.0030.695±0.0032.11±0.110.692±0.0060.660±0.0130.286±0.013

Table 4. Negative-type ablation (LOWM, 5 seeds, matrix_size = 16).

Figure 4. Coherence stratification (seed 0, IID). OMC separates τsame\tau_{\mathrm{same}}, τwrong\tau_{\mathrm{wrong}}, τnoise\tau_{\mathrm{noise}}; the baseline collapses same vs. wrong while placing noise far above.

ModelVarianttop1law_pairsame<wrmat_mrr
C-SWM-styleOMC-enhanced0.358±0.0420.512±0.0060.491±0.0120.212±0.000
C-SWM-stylerandom-neg0.214±0.0120.483±0.0030.493±0.0060.211±0.000
C-SWM-stylestate-corrupt0.237±0.0110.482±0.0070.492±0.0040.211±0.000
C-SWM-styletemporal-neg0.115±0.0170.513±0.0240.517±0.0260.228±0.015
RSSM-lite(no-law obj.)0.590±0.0050.596±0.0050.609±0.0070.270±0.011

Table 8. General world-model baselines (3 seeds, IID; OOD in Appendix D). Feature-level scorers.

BackboneObj.generic top1same<wrfull str.mat_mrrdiag-gap
LOWMOMC0.733±0.0060.735±0.0060.714±0.0140.344±0.0041.368±0.031
LOWMno-law0.496±0.0020.496±0.0020.449±0.0410.211±0.0010.332±0.215
directOMC0.736±0.0140.736±0.0140.736±0.0140.309±0.0071.044±0.014
directno-law0.504±0.0070.504±0.0070.504±0.0070.210±0.0020.012±0.033

Table 9. CoPhy ballsCF external validation (test, 5 seeds).

(R1) Operator-blindness is real and hidden behind top-1. Table 3 states the central result. The no-law-mismatch baseline reaches ≈0.99\approx0.99 generic top-1 yet law_pair = 0 and same_lt_wrong≈\approx 0.5: it rejects noise almost perfectly while ordering current- vs. other-world futures at chance—an empirical instance of Definition 1 that Prop. 1 predicts the diagnostic will miss. OMC inverts the profile, trading generic top-1 (expected for a harder, operator-targeted objective) for large gains on every operator-sensitive metric. The pattern holds for both backbones, so it is driven by the objective rather than the architecture, and it persists under the OOD operator split, ruling out parameter memorization.

(R3) The wrong-operator negative is the active ingredient. Table 4 ablates the negative set, corroborating Prop. 2. Removing the wrong-operator negative (NO-LAW) collapses operator-sensitive scores to chance while preserving generic ranking; using it exclusively (LAW-ONLY) maximizes the targeted metrics. The balanced all-negative objective trades a little of that peak for broader coverage. This isolates the wrong-operator negative—not contrastive learning per se—as the source of the effect.

The pairwise head trades off, it does not dominate. Table 5 compares the pairwise-on (main_all) and pairwise-off settings under the same all-negative objective. The pairwise head raises generic top-1 and lowers variance and slightly improves SAME_LT_WRONG, while pairwise-off attains slightly higher MATRIX_MRR. We therefore do not attribute the OMC effect to the pairwise component; it shifts the trade-off between generic ranking and matrix retrieval.

ObjectiveSplittop1law_pairsame<wrmat_mrrmat_top1
main_all (pairwise on)IID0.509±0.0100.692±0.0030.703±0.0040.255±0.0070.092±0.008
main_all (pairwise on)OOD0.482±0.0110.656±0.0040.670±0.0030.257±0.0030.096±0.005
no_pairwise (main OMC)IID0.375±0.1390.694±0.0050.697±0.0060.272±0.0120.105±0.017
no_pairwise (main OMC)OOD0.363±0.1160.650±0.0060.665±0.0090.271±0.0090.103±0.011

Table 5. Component ablation (LOWM, 5 seeds, all-negative objective, matrix_size=16).

Visualizing the mechanism. Figure 3 shows a representative operator×\timestrajectory energy matrix: under OMC the matched (diagonal) operator gives the lowest energy within each column, whereas the baseline shows no such structure. Figure 4 shows the corresponding energy distributions: OMC realizes τsame<τwrong<τnoise\tau_{\mathrm{same}} < \tau_{\mathrm{wrong}} < \tau_{\mathrm{noise}}, while the baseline collapses τsame\tau_{\mathrm{same}} and τwrong\tau_{\mathrm{wrong}} yet still separates τnoise\tau_{\mathrm{noise}}—operator-blindness made visible.

Figure 3. Energy matrix (seed 0, IID, 16 × 16), per-operator normalized (0 = most coherent in each column). OMC concentrates low energy on the diagonal; the baseline does not.

The effect is relational, not operator decodability. OMC does not produce a more decodable operator code: linear/MLP probes on λC\lambda_C recover operator identity and binned parameters about equally for OMC and the baseline (seed-0 IID: linear operator-id 0.60 vs. 0.61; binned-parameter 0.89 vs. 0.90; parameter R2R^2 0.35 vs. 0.39). The gain is therefore relational—OMC shapes the compatibility Eθ(λC,τ)E_\theta(\lambda_C,\tau), consistent with Prop. 2, rather than the operator embedding. Full probe numbers are in Appendix D.

Sensitivity analysis

We vary the main design choices around the paper’s operating point (no-pairwise, αoccl=0\alpha_{\mathrm{occl}}=0, all-4 negatives), changing one knob at a time (3 seeds; the αoccl=0\alpha_{\mathrm{occl}}=0 / all-4 row reproduces Table 3). We report test_iid; the OOD split shows the same pattern.

Training the matrix term (αoccl>0\alpha_{\mathrm{occl}}>0). Weighting the operator-coherence matrix loss (Table 6) gives a small additional gain in operator-sensitive metrics (LAW_PAIR 0.694→0.710, MATRIX_MRR 0.269→0.289) at a further cost to generic top-1 (0.38→0.24). Since the wrong-operator negative alone (αoccl=0\alpha_{\mathrm{occl}}=0) already captures most of the effect, we keep αoccl=0\alpha_{\mathrm{occl}}=0 as the simplest sufficient setting; training the matrix term is a viable, marginally stronger alternative when generic top-1 is not a concern.

αoccl\alpha_{\mathrm{occl}}top1law_pairsame<wrmat_mrr
0.00.384±0.1250.694±0.0070.693±0.0030.269±0.004
0.10.219±0.0160.703±0.0060.700±0.0070.290±0.006
0.50.230±0.0330.706±0.0080.703±0.0040.292±0.010
1.00.239±0.0430.710±0.0090.710±0.0060.289±0.005

Table 6. Sensitivity to αoccl\alpha_{\mathrm{occl}} (LOWM, no-pairwise, all-4 negatives; 3 seeds, test_iid). αoccl\alpha_{\mathrm{occl}}=0 is the paper setting.

Negative-set composition. Table 7 varies the candidate negatives. Any set containing the wrong-operator negative reaches LAW_PAIR/SAME_LT_WRONG ≈\approx 0.69–0.71; removing it (NO-LAW) collapses operator sensitivity to chance (SAME_LT_WRONG 0.506, MATRIX_MRR 0.211) while generic top-1 stays near-perfect (0.996). The non-operator negatives mainly trade operator sensitivity for generic top-1 (adding random raises top-1 to 0.56 at no cost to operator metrics), making the OMC top-1 reduction a tunable design choice rather than an inherent penalty. This is the cleanest confirmation of Prop. 2: the wrong-operator negative is necessary and largely sufficient.

Robustness to δ\delta and temperature. Operator-sensitive metrics are essentially flat across the operator-distance threshold δ∈[0.05,0.4]\delta\in[0.05, 0.4] and the InfoNCE temperature T∈[0.5,4]T \in[0.5, 4] (Figure 5): LAW_PAIR stays at ≈\approx 0.69 throughout, so OMC is insensitive to these hyperparameters within the tested ranges.

Robustness of OMC to wrong-operator distance and InfoNCE temperature

Figure 5. Robustness of OMC to wrong-operator distance δ\delta (left) and InfoNCE temperature TT (right); operator-sensitive metrics stay flat (test_iid, 3 seeds).

General World-Model Baselines

Table 8 places OMC against feature-level C-SWM-style [10] and RSSM-lite scorers. With ordinary structured negatives (random, state-corrupted, temporal), the C-SWM-style scorer stays at chance on operator-sensitive metrics—contrastive structure alone does not induce operator sensitivity, exactly as Prop. 2 predicts when τwrong∉C\tau_{\mathrm{wrong}} \notin C. RSSM-lite is stronger but remains below OMC-trained scorers. These are feature-level scorers, not full agents.

Clarification (Table 8). “OMC-enhanced” adds the wrong-operator negative to the C-SWM-style set. That operator sensitivity still stays at chance indicates an objective–architecture interaction: the C-SWM-style encoder does not expose the operator information the negative would exploit.

CoPhy External Validation

The effect transfers to an external counterfactual-physics dataset (Table 9, Figure 6). On CoPhy ballsCF, OMC raises wrong-confounder rejection (same_lt_wrong ≈ 0.74 vs. 0.50) and operator×trajectory retrieval (matrix_mrr ≈ 0.34 vs. 0.21) for both backbones, so operator-blindness and its OMC remedy are not artifacts of the synthetic generator. CoPhy is used as feature-level external validation.

Figure 6. Summary across settings. OMC vs. no-law on operator-coherent ranking (left) and energy-matrix retrieval (right) for synthetic IID/OOD and CoPhy. The baseline stays near chance everywhere; OMC is consistently above it, including on external CoPhy.

Limitations and Future Directions

By design OMC is a minimal and attachable coherence signal rather than a standalone world model: it demonstrates improved discrimination of world-relative coherence—not generation, planning, or control—and its effect is relational rather than a more decodable operator code. This minimality is a feature: the entire gain follows from one drop-in change to what counts as a negative, which makes the signal portable. The pairwise head is not universally beneficial, LAW_GAP is comparable only within a model family (energy scales differ across backbones), and the CoPhy results are feature-level rather than raw-video. Constructing the wrong-operator negative requires access to alternative-operator trajectories at training time, a property of the signal, not of inference.

The coherence signal suggests a generator–critic decomposition: a proposal model free to entertain plausible-elsewhere futures, paired with OMC as a critic that reranks them for coherence here. Whether such generativity helps downstream world models, and whether OMC can serve as an auxiliary verifier inside latent-dynamics or JEPA-style models, is left to future work.

Three analyses would sharpen the present claims and are immediate next steps: (i) a finer operator-distance study—beyond the threshold sweep of § 6.1—quantifying how same-vs-wrong separation degrades as the wrong operator approaches the true one; (ii) a calibration/verifier analysis (reliability or AUROC) testing OMC as a decision rule rather than only a ranking metric; and (iii) a data/operator-budget curve establishing how many alternative-operator trajectories the signal needs. We regard these as confirmatory rather than load-bearing: the necessity of the wrong-operator negative is already established theoretically (Prop. 2) and by the ablations of § 6.

Conclusion

Plausibility is not coherence. A trajectory scorer can rank generic futures almost perfectly while remaining blind to which world is in force—a failure we formalized as operator-blindness and proved invisible to generic ranking. Operator-Mismatch Contrastive learning makes valid-but-wrong-operator trajectories the hard negatives, and we showed both theoretically and empirically—across synthetic IID/OOD, two backbones, and external CoPhy—that this single change to what counts as a negative exposes and repairs the failure.

Notation and Formal Definitions

Context CC; operator prior p(λ)p(\lambda) and conditional p(⋅∣λ)p(\cdot\mid\lambda); posterior qϕ(λ∣C)=N(μϕ(C),diag⁡σϕ2(C))q_{\phi}(\lambda\mid C) = \mathcal{N}(\mu_{\phi}(C), \operatorname{diag}\sigma_{\phi}^{2}(C)); operator code λC=μϕ(C)\lambda_C = \mu_{\phi}(C); energy Eθ(λC,τ)E_{\theta}(\lambda_C, \tau) (lower = more coherent); candidate classes τsame,τwrong,τnoise\tau_{\mathrm{same}}, \tau_{\mathrm{wrong}}, \tau_{\mathrm{noise}} as in § 3; target ordering Eq. (2); operator-blindness Definition 1.

Metric Definitions

For a sample with candidate energies {Em}m=1M\{E_m\}_{m=1}^{M} and positive index yy:

  • top1_acc = Pr⁡[arg⁡min⁡mEm=y]\Pr[\arg\min_m E_m = y] over the M=5M=5 set (diagnostic).

  • law_pair = Pr⁡[Eθ(τwrong)>Eθ(τsame)]\Pr[E_{\theta}(\tau_{\mathrm{wrong}}) > E_{\theta}(\tau_{\mathrm{same}})].

  • law_gap = E[Eθ(τwrong)−Eθ(τsame)]\mathbb{E}[E_{\theta}(\tau_{\mathrm{wrong}}) - E_{\theta}(\tau_{\mathrm{same}})] (within model family only).

  • law_only_top1 = top1_acc on {τsame,τwrong}\{\tau_{\mathrm{same}}, \tau_{\mathrm{wrong}}\}.

  • same_lt_wrong = Pr⁡[Eθ(τsame)<Eθ(τwrong)]\Pr[E_{\theta}(\tau_{\mathrm{same}}) < E_{\theta}(\tau_{\mathrm{wrong}})] on {τsame,τwrong,τnoise}\{\tau_{\mathrm{same}}, \tau_{\mathrm{wrong}}, \tau_{\mathrm{noise}}\}.

  • full_stratification = Pr⁡[Eθ(τsame)<Eθ(τwrong)<Eθ(τnoise)]\Pr[E_{\theta}(\tau_{\mathrm{same}}) < E_{\theta}(\tau_{\mathrm{wrong}}) < E_{\theta}(\tau_{\mathrm{noise}})].

  • Energy matrix Mij=Eθ(τi,λj)M_{ij} = E_{\theta}(\tau_i, \lambda_j) over B=16B=16 distinct-operator samples; with rirow,rjcolr_i^{\mathrm{row}}, r_j^{\mathrm{col}} the diagonal ranks, matrix_mrr = 12(1/rirow+1/rjcol)\frac{1}{2}(1/r_i^{\mathrm{row}} + 1/r_j^{\mathrm{col}}), matrix_top1 = 12([rirow=1]+[rjcol=1])\frac{1}{2}([r_i^{\mathrm{row}}=1] + [r_j^{\mathrm{col}}=1]), diagonal_vs_offdiag_gap = Mi≠j−MiiM_{i\ne j} - M_{ii}.

Definitions match the evaluation code; energy-matrix metrics use matrix_size=16, max_batches=8.

Objective Details and Proofs

LOCCL\mathcal{L}_{\mathrm{OCCL}} is a symmetric InfoNCE over Mij=Eθ(τi,λj)M_{ij} = E_{\theta}(\tau_i, \lambda_j) (matching trajectories to operators in both directions, temperature 1); in all reported runs αoccl=0\alpha_{\mathrm{occl}} = 0, so it never enters the gradient. Proofs of Propositions 1 and 2 are given with their statements in § 3 and § 4; both rely only on (i) the candidate set membership and (ii) the standard noise-contrastive optimality result [12, 11].

Robustness of Proposition 2 in practice. The argument is asymptotic (population limit, expressive scorer, temperature 1). Empirically the induced ordering is stable away from that idealization: it is essentially unchanged across InfoNCE temperature T∈[0.5,4]T \in[0.5, 4] and operator-distance threshold δ∈[0.05,0.4]\delta\in[0.05, 0.4] (§ 6.1, Figure 5), and holds even though the candidate set mixes in noise negatives drawn from outside the data distribution—these are separated trivially and do not interfere with the same-vs-wrong ordering (Figure 4).

Per-Seed Result Tables

All experiments, scenarios, and seeds are reported below (matrix_size=16, max_batches=8). Splits: IID (test_iid), OOD (test_ood_param), and test/val for CoPhy.

Main synthetic (LOWM and direct-context, OMC vs. no-law-mismatch)

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfullstr.mat_mrrmat_t1diag-gap
direct_context_energy_direct_context_energy
0IID0.5860.6993.870.4130.7070.7070.2600.0982.51
1IID0.4020.490-0.010.1990.4990.4990.2110.0620.00
2IID0.5900.7054.100.4090.7030.7030.2550.0902.66
3IID0.6110.7172.570.4010.7160.7160.2570.0901.76
4IID0.5830.7043.940.4140.7160.7160.2570.0982.56
0OOD0.5670.6903.670.3970.7130.7130.2620.1022.47
1OOD0.3910.489-0.040.2080.4900.4900.2110.0620.00
2OOD0.5630.6843.980.3910.7060.7060.2550.0942.64
3OOD0.5830.6872.580.3850.7120.7120.2550.0901.75
4OOD0.5630.6893.690.3760.7040.7040.2600.1022.48
0val0.6080.7444.580.461–––––
1val0.4350.513-0.020.210–––––
2val0.6210.7494.870.461–––––
3val0.6440.7503.090.450–––––
4val0.6100.7504.700.457–––––
lowm_lowm_omcr_no_pairwise
0IID0.3310.6950.990.4090.6960.6960.2730.1130.66
1IID0.5260.6872.070.4090.6920.6920.2680.1021.43
2IID0.2940.7000.830.4090.6900.6900.2650.0940.55
3IID0.5140.6981.980.4130.7050.7050.2610.0861.35
4IID0.2110.6910.650.4160.7020.7020.2920.1290.43
0OOD0.3200.6470.870.3400.6570.6570.2620.0940.59
1OOD0.4970.6472.000.3670.6590.6590.2680.0981.46
2OOD0.2940.6600.730.3620.6770.6770.2860.1170.48
3OOD0.4740.6501.900.3660.6590.6590.2650.0941.38
4OOD0.2320.6470.570.3580.6710.6710.2730.1130.38
0val0.3690.7241.060.399–––––
1val0.5630.7272.410.430–––––
2val0.3340.7340.900.436–––––
3val0.5370.7242.270.429–––––
4val0.2700.7300.700.431–––––
lowm_no_law_mismatch
0IID0.9960.0000.000.1850.5110.5110.2090.0551.62
1IID0.9970.0000.000.2070.5160.5160.2140.0623.41
2IID0.9940.0000.000.1920.4900.4900.2110.0591.53
3IID0.9950.0000.000.1840.5110.5110.2150.0620.57
4IID0.9960.0000.000.2120.5060.5060.2150.0661.45
0OOD0.9920.0000.000.2070.5050.5050.2050.0550.24
1OOD0.9920.0000.000.2040.5240.5240.2070.0620.29
2OOD0.9920.0000.000.2110.5170.5170.2120.0620.36
3OOD0.9890.0000.000.2020.5090.5090.2070.059-0.30
4OOD0.9920.0000.000.2000.5150.5150.2150.0700.21
0val0.9970.0000.000.207–––––
1val0.9950.0000.000.241–––––
2val0.9970.0000.000.208–––––
3val0.9970.0000.000.216–––––
4val0.9950.0000.000.203–––––

Table 10.

Negative-type ablation: law-mismatch-only

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
0IID0.4060.7002.26–0.7160.6880.2990.1331.60
1IID0.4190.6952.18–0.7160.6750.2770.1051.52
2IID0.4170.6992.11–0.7070.6670.2660.0941.47
3IID0.4260.7072.26–0.7080.6610.2940.1251.57
4IID0.4120.6972.04–0.7130.6880.2820.1131.43
0OOD0.3990.6992.20–0.6990.6750.3090.1291.48
1OOD0.4010.6902.11–0.6980.6660.2690.0901.41
2OOD0.4050.6922.01–0.6900.6540.2800.1021.35
3OOD0.4030.6962.25–0.6850.6390.2840.1021.55
4OOD0.4060.6971.95–0.6880.6670.2890.1091.31
0val0.4450.7412.840.445–––––
1val0.4410.7372.730.441–––––
2val0.4520.7382.630.452–––––
3val0.4300.7332.820.430–––––
4val0.4390.7372.530.439–––––

Table 11.

Component ablation: main_all (pairwise energy on)

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
0IID0.5080.6911.61–0.7010.7010.2500.0901.13
1IID0.5200.6961.71–0.6960.6960.2640.1021.20
2IID0.5080.6881.63–0.7050.7050.2520.0861.15
3IID0.4920.6911.72–0.7030.7030.2460.0821.20
4IID0.5180.6921.64–0.7080.7080.2610.1021.17
0OOD0.4770.6511.54–0.6690.6690.2580.0941.18
1OOD0.4950.6591.64–0.6710.6710.2610.1021.22
2OOD0.4890.6611.57–0.6750.6750.2530.0901.18
3OOD0.4630.6531.63–0.6670.6670.2530.0941.21
4OOD0.4870.6561.60–0.6700.6700.2600.1021.18
0val0.5240.7262.090.411–––––
1val0.5270.7192.240.406–––––
2val0.5320.7222.140.413–––––
3val0.5000.7212.240.395–––––
4val0.5320.7302.140.417–––––

Table 12.

Component ablation: no_pairwise (pairwise energy off)

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
0IID0.3310.6950.99–0.6960.6960.2730.1130.66
1IID0.5260.6872.07–0.6920.6920.2680.1021.43
2IID0.2940.7000.83–0.6900.6900.2650.0940.55
3IID0.5140.6981.98–0.7050.7050.2610.0861.35
4IID0.2100.6910.65–0.7020.7020.2920.1290.43
0OOD0.3200.6470.87–0.6570.6570.2620.0940.59
1OOD0.4970.6472.00–0.6590.6590.2680.0981.46
2OOD0.2940.6600.73–0.6770.6770.2860.1170.48
3OOD0.4740.6501.90–0.6590.6590.2650.0941.38
4OOD0.2320.6470.57–0.6710.6710.2730.1130.38
0val0.3690.7241.060.399–––––
1val0.5630.7272.410.430–––––
2val0.3340.7340.900.436–––––
3val0.5370.7242.270.429–––––
4val0.2700.7300.700.431–––––

Table 13.

C-SWM-style baselines (feature-level)

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
0IID0.3480.5050.010.1940.4850.4850.2120.0620.00
1IID0.4040.5170.020.1960.5050.5050.2110.062-0.00
2IID0.3220.5140.010.2190.4840.4840.2110.0620.00
0OOD0.3370.502-0.030.1990.4850.4850.2120.0620.00
1OOD0.3640.5160.010.2050.5000.5000.2110.062-0.00
2OOD0.2990.4960.010.2250.4920.4920.2120.0620.00
cswm_style_energy_cswm_random_neg
0IID0.2040.481-2.590.1880.4880.0780.2110.0620.16
1IID0.2270.486-3.800.1840.4920.0580.2110.0620.08
2IID0.2120.483-2.260.1930.4990.1090.2110.062-0.03
0OOD0.2130.467-6.660.1970.5120.0880.2110.0620.29
1OOD0.2100.458-9.250.1990.5100.0600.2110.0620.63
2OOD0.2250.475-3.960.2040.5010.0870.2110.062-0.08
cswm_style_energy_cswm_state_corrupted_neg
0IID0.2440.490-36.800.1950.4890.3750.2110.0622.51
1IID0.2420.480-99.120.1820.4960.1570.2110.0624.47
2IID0.2240.477-74.540.1890.4920.2160.2110.0624.33
0OOD0.2220.476-100.080.1950.5110.3910.2120.0623.46
1OOD0.2210.462-292.880.2020.5040.1600.2110.06213.84
2OOD0.2040.463-205.580.1990.5130.2250.2110.0627.40
cswm_style_energy_cswm_temporal_neg
0IID0.1030.5141297.240.2380.4990.0020.2340.086610.12
1IID0.1080.4893862.390.2060.5040.0000.2110.0661398.84
2IID0.1340.5376644.520.2450.5470.0010.2400.0826941.41
0OOD0.1290.495-119.400.2150.5140.0020.2060.055169.49
1OOD0.1140.4933586.630.2000.5010.0000.2120.0662645.30
2OOD0.1710.5478494.510.2380.5580.0000.2380.0862536.95

Table 14.

RSSM-lite baseline (feature-level)

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
rssm_lite_scorer_rssm_lite_mse
0IID0.5960.6020.000.2820.6080.6080.2690.1130.00
1IID0.5880.5940.000.2840.6160.6160.2810.1330.00
2IID0.5870.5920.000.2810.6020.6020.2590.1020.00
0OOD0.5480.5570.000.2960.5710.5710.2600.0860.00
1OOD0.5480.5570.000.2930.5670.5670.2620.0900.00
2OOD0.5490.5570.000.2840.5740.5740.2480.0700.00

Table 15.

CoPhy ballsCF external validation (feature-level)

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
direct_context_energy_direct_context_cophy_OMC
0test0.7300.7301.070.406––0.3140.1371.23
1test0.7400.7401.080.424––0.3100.1251.23
2test0.7280.7301.050.402––0.3100.1251.23
3test0.7560.7581.080.460––0.3340.1451.28
4test0.7320.7321.070.386––0.3290.1521.26
0val0.7000.7061.000.424––0.3150.1451.05
1val0.7040.7121.020.410––0.3080.1251.07
2val0.7040.7121.000.412––0.2940.1091.08
3val0.7240.7321.060.444––0.3230.1411.12
4val0.7200.7281.030.428––0.3040.1291.08

Table 16.

seedsplittop1law_pairlaw_gaplaw_onlysame<wrfull str.mat_mrrmat_t1diag-gap
direct_context_energy_direct_context_cophy_no_law_mismatch
0test0.9960.0000.000.188––0.2120.062-0.11
1test0.9960.0000.000.198––0.2120.062-0.14
2test0.9960.0000.000.186––0.2100.059-0.11
3test0.9960.0000.000.188––0.2090.059-0.17
4test0.9960.0000.000.192––0.2080.059-0.13
0val0.9780.0000.000.172––0.2130.0620.14
1val0.9780.0000.000.180––0.2120.0620.10
2val0.9800.0000.000.180––0.2130.0620.11
3val0.9800.0000.000.188––0.2090.0590.14
4val0.9800.0000.000.184––0.2130.0620.10
lowm_lowm_cophy_OMC
0test0.7380.7521.440.426––0.3650.1601.74
1test0.7240.7401.440.420––0.3660.1641.76
2test0.7200.7381.540.418––0.3640.1561.84
3test0.7300.7481.450.430––0.3720.1681.75
4test0.7480.7561.460.438––0.3660.1601.74
0val0.7200.7421.410.430––0.3720.1841.53
1val0.7160.7441.450.426––0.3630.1681.55
2val0.7360.7621.530.436––0.3790.1911.63
3val0.7200.7421.430.430––0.3740.1841.55
4val0.7220.7401.450.446––0.3630.1721.55
lowm_lowm_cophy_no_law_mismatch
0test0.9780.0000.000.202––0.2120.0621.09
1test0.9820.0000.000.198––0.2110.0620.66
2test0.9780.0000.000.190––0.2100.0590.23
3test0.9820.0000.000.194––0.2130.0660.56
4test0.9880.0000.000.190––0.2120.0620.41
0val0.9700.0000.000.204––0.2140.0661.33
1val0.9720.0000.000.208––0.2130.0660.81
2val0.9680.0000.000.216––0.2140.0660.37
3val0.9700.0000.000.210––0.2130.0660.74
4val0.9700.0000.000.212––0.2130.0660.59

Table 19.

Reproducibility

Optimizer AdamW [36] (lr 10−310^{-3}, weight decay 10−410^{-4}); βKL=10−4\beta_{\mathrm{KL}}=10^{-4}, αstable=0.1\alpha_{\mathrm{stable}}=0.1, αoccl=0\alpha_{\mathrm{occl}}=0, OCCL temperature 1; checkpoint by best LAW_PAIR. Candidate set M=5M=5 (one negative per type), operator distance threshold δ=0.15\delta=0.15, distinct operators per batch; dλ=16d_{\lambda}=16, hidden/context dim 128, 2 layers, sum energy reduction. Synthetic: 30 epochs, batch 64, 5000/1000 train/val per epoch, seeds 0–4. CoPhy: 30 epochs, batch 2048, 20000/3000, seeds 0–4. Baselines (C-SWM-style, RSSM-lite): 30 epochs, batch 1024, 20000/3000, seeds 0–2. Energy-matrix eval: matrix_size=16, max_batches=8. Hardware: Colab GPU (T4/L4-class).

CoPhy Preprocessing

CoPhy ballsCF (224px) is reduced to segmentation-derived object features at 64px over 20 frames; the hidden confounder serves as the operator metadata for negatives and evaluation. Episode split: 35,000/7,500/7,500 (train/val/test), total 50,000.

References

  1. [1]B. Lake, T. Ullman, J. Tenenbaum, S. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 2017.arxiv.org/abs/1604.00289
  2. [2]P. Battaglia, J. Hamrick, J. Tenenbaum. Simulation as an engine of physical scene understanding. PNAS, 2013.DOI
  3. [3]D. Ha, J. Schmidhuber. World Models. NeurIPS, 2018.DOI
  4. [4]D. Hafner, T. Lillicrap, J. Ba, M. Norouzi. Dream to Control: Learning Behaviors by Latent Imagina- tion. ICLR, 2020.
  5. [5]Y. LeCun. A Path Towards Autonomous Machine Intelligence. 2022.
  6. [6]Q. Garrido, N. Ballas, M. Assran, A. Bardes, L. Najman, M. Rabbat, E. Dupoux, Y. LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos. arXiv:2502.11831, 2025.arxiv.org/abs/2502.11831
  7. [7]M. Raissi, P. Perdikaris, G. Karniadakis. Physics-informed neural networks. J. Comput. Phys., 2019.
  8. [8]S. Greydanus, M. Dzamba, J. Yosinski. Hamiltonian Neural Networks. NeurIPS, 2019.
  9. [9]M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, S. Ho. Lagrangian Neural Networks. ICLR DeepDiffEq Workshop, 2020.arxiv.org/abs/2003.04630
  10. [10]T. Kipf, E. van der Pol, M. Welling. Contrastive Learning of Structured World Models. ICLR, 2020.
  11. [11]A. van den Oord, Y. Li, O. Vinyals. Representation Learning with Contrastive Predictive Coding. 2018.
  12. [12]M. Gutmann, A. Hyvärinen. Noise-contrastive estimation. AISTATS, 2010.
  13. [13]F. Baradel, N. Neverova, J. Mille, G. Mori, C. Wolf. CoPhy: Counterfactual Learning of Physical Dynamics. ICLR, 2020.arxiv.org/abs/1909.12000
  14. [14]C. Finn, P. Abbeel, S. Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML, 2017.
  15. [15]K. Rakelly, A. Zhou, D. Quillen, C. Finn, S. Levine. Efficient Off-Policy Meta-RL via Probabilistic Context Variables (PEARL). ICML, 2019.DOI
  16. [16]L. Zintgraf, K. Shiarli, M. Igl, S. Schulze, Y. Gal, K. Hofmann, S. Whiteson. VariBAD: Bayes-Adaptive Deep RL via Meta-Learning. ICLR, 2020.
  17. [17]T. Chen, S. Kornblith, M. Norouzi, G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations (SimCLR). ICML, 2020.arxiv.org/abs/2002.05709
  18. [18]K. He, H. Fan, Y. Wu, S. Xie, R. Girshick. Momentum Contrast for Unsupervised Visual Representation Learning (MoCo). CVPR, 2020.arxiv.org/abs/1911.05722
  19. [19]J. Robinson, C.-Y. Chuang, S. Sra, S. Jegelka. Contrastive Learning with Hard Negative Samples. ICLR, 2021.arxiv.org/abs/2010.04592
  20. [20]C.-Y. Chuang, J. Robinson, L. Lin, A. Torralba, S. Jegelka. Debiased Contrastive Learning. NeurIPS, 2020.arxiv.org/abs/2007.00224
  21. [21]F. Locatello, D. Weissenborn, T. Unterthiner, et al. Object-Centric Learning with Slot Attention. NeurIPS, 2020.
  22. [22]Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, F. Huang. A Tutorial on Energy-Based Learning. 2006.
  23. [23]M. Assran, Q. Duval, I. Misra, et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). CVPR, 2023.arxiv.org/abs/2301.08243
  24. [24]R. Riochet, M. Castro, M. Bernard, et al. IntPhys: A Benchmark for Visual Intuitive Physics Reasoning. IEEE TPAMI, 2022.
  25. [25]B. Schölkopf, F. Locatello, S. Bauer, et al. Toward Causal Representation Learning. Proceedings of the IEEE, 2021.DOI
  26. [26]D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, J. Davidson. Learning Latent Dynamics for Planning from Pixels. ICML, 2019.arxiv.org/abs/1811.04551
  27. [27]D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap. Mastering Diverse Domains through World Models. arXiv:2301.04104, 2023.arxiv.org/abs/2301.04104
  28. [28]A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, N. Ballas. Revisiting Feature Prediction for Learning Visual Representations from Video. arXiv:2404.08471, 2024.arxiv.org/abs/2404.08471
  29. [29]J. Bruce, M. Dennis, A. Edwards, et al. Genie: Generative Interactive Environments. ICML, 2024.arxiv.org/abs/2402.15391
  30. [30]L. Ljung. System Identification: Theory for the User. Prentice Hall, 2nd edition, 1999.
  31. [31]R. Hadsell, S. Chopra, Y. LeCun. Dimensionality Reduction by Learning an Invariant Mapping. CVPR, 2006.DOI
  32. [32]K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, J. Tenenbaum. CLEVRER: Collision Events for Video Representation and Reasoning. ICLR, 2020.arxiv.org/abs/1910.01442
  33. [33]M. Zaheer, S. Kottur, S. Ravindraksh, B. Poczos, R. Salakhutdinov, A. Smola. Deep Sets. NeurIPS, 2017.
  34. [34]P. Battaglia, R. Pascanu, M. Lai, D. Rezende, K. Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physics. NeurIPS, 2016.arxiv.org/abs/1612.00222
  35. [35]D. Kingma, M. Welling. Auto-Encoding Variational Bayes. ICLR, 2014.arxiv.org/abs/1312.6114
  36. [36]I. Loshchilov, F. Hutter. Decoupled Weight Decay Regularization. ICLR, 2019.

Paper details

vlnc:W_3uzw3nrq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents