WorksComputer Science
World Models Favor Learned Priors over Visual Evidence
Abstract
World models are increasingly used to simulate physical interactions, but training a separate model for each environment limits scalability. Cross-environment prediction requires identifying the current physical rule, which appearance alone may not reveal. We therefore study visual calibration: using object motion and interactions in a short video as direct evidence to infer the current rule and predict motion under that rule from a different initial state. However, differences in data availability across physical conditions can yield a long-tailed training distribution, biasing models toward common rules even when evidence supports rare ones. To study this challenge, we introduce TailWorld, which evaluates the transfer of demonstrated rules across initial states while independently controlling training imbalance and calibration informativeness. The benchmark compares predictions with two reference futures from the same prediction initial state, one following the demonstrated rule and the other a common rule, to measure prediction bias toward common dynamics. We further propose Rule Inference for Transfer (RIFT), which learns how different rules affect object motion, infers the current rule from calibration interactions, and applies it to prediction from a new initial state. RIFT explicitly controls the influence of training frequency on rule inference, allowing sufficiently informative observational evidence to support rare rules. Experiments on TailWorld show that RIFT improves prediction accuracy and physical consistency across varying training imbalances and distribution shifts, with gains in predicting rare dynamics.

Figure 1. RIFT transfers calibrated physical rules across states to predict rare dynamics.
Introduction
World models are increasingly used to simulate how physical environments evolve [7, 15, 49]. Training a separate model for each environment is costly and difficult to scale, motivating world models that can predict across environments. A key challenge is identifying the physical rule governing the current environment. Visually similar objects may behave differently because their physical properties or surrounding conditions differ, so appearance alone may be insufficient [4, 46]. These properties and conditions shape motion and interactions [5, 36, 41], allowing the dynamics captured in a video to provide evidence for identifying the current rule [6, 39]. We study visual calibration, where a model infers the current rule from a short demonstration and predicts motion under that rule from a different initial state [4]. Changing the initial state makes replaying the demonstrated trajectory insufficient, testing whether the model can understand and transfer the underlying rule.
Recent methods primarily adapt models to different dynamics by updating environment-specific parameters from limited observations [22, 25, 35] or conditioning predictions on representations of observed trajectories [4, 11]. However, real-world objects, materials, and environments differ in prevalence and ease of data collection, leading to uneven coverage of physical conditions and potentially a long-tailed data distribution. Models trained on such data may predict according to common rules even when calibration evidence supports rare ones. The key challenge is whether a model can base its predictions on sufficiently informative observations when they conflict with preferences induced by training frequency.
To study this challenge under controlled conditions, we introduce TailWorld, a benchmark for visual calibration under long-tailed dynamics. Each task requires prediction from a different initial state under the rule demonstrated in a calibration video. TailWorld independently controls rule frequencies during training and calibration informativeness to examine how training imbalance and observational evidence jointly affect prediction. It also provides two reference futures from the same prediction initial state: one follows the demonstrated rule, and the other follows a common rule. Comparing predictions with both quantifies their deviation from the true future and any shift toward the common alternative, directly assessing the model’s preference for common dynamics.
To enable models to infer rules from current evidence and apply them to new trajectories, we propose Rule Inference for Transfer (RIFT). RIFT first learns how different physical rules affect object motion, then uses these relationships to infer the current rule from interactions observed in a calibration video. It applies the inferred rule to prediction from a different initial state and keeps it fixed throughout prediction. Crucially, RIFT explicitly controls the influence of training frequency on rule inference, allowing sufficiently informative observations to support rules rarely seen during training. The model can thus draw on knowledge learned across environments to predict motion under the current rule from a new initial state. Experiments on TailWorld show that RIFT improves prediction accuracy and physical consistency across varying training imbalances and distribution shifts, with gains extending to rare dynamics. Our contributions are threefold:
We study visual calibration under long-tailed dynamics, examining whether models can follow rare rules supported by observational evidence when predicting motion from different initial states.
We introduce TailWorld, which independently controls training imbalance and calibration informativeness and measures prediction bias toward common dynamics using reference futures under the demonstrated and common rules.
We propose RIFT, which infers the current rule from observed interactions, explicitly controls the influence of training frequency on inference, and applies the inferred rule to prediction from different initial states.
Related Work
World models and dynamics adaptation. World models predict environment evolution to support planning and adaptation [1, 7, 15, 16, 17, 32, 49]. For physical systems, interaction-based models capture object interactions and their effects on motion [5, 9, 27, 36, 41]. Adaptation methods update model parameters or condition predictions on limited observations [12, 14, 22, 25, 35, 40], while recent work extracts physical information from trajectories and interaction videos [11, 30]. Under imbalanced training frequencies, however, predictions may favor a common rule even when calibration evidence supports a rare one.
Visual physical inference and evaluation. Visual physical inference recovers hidden properties and interactions from observed motion [24, 28, 44, 45, 46]. Counterfactual prediction tests whether inferred dynamics transfer across initial conditions [4, 19]. Benchmarks evaluate physical plausibility, outcome prediction, and causal reasoning [2, 6, 37, 39, 48], with further work probing hidden properties and challenging interactions [3, 21, 26, 33, 42]. Prediction error alone, however, does not distinguish ambiguous evidence from a preference for common dynamics. TailWorld independently varies training imbalance and calibration informativeness, then compares predictions from a new initial state with futures under the demonstrated and common rules. This reveals whether predictions favor common dynamics despite evidence identifying a rare rule.
Long-tailed learning and prior correction. Long-tailed learning addresses target imbalance through reweighting, margin adjustments, and decoupled training [8, 10, 20]. Prior-aware methods account for label frequencies and distribution shifts [18, 29, 31, 38], with extensions to continuous targets [47]. These approaches motivate frequency correction, but physical prediction requires inferring a latent rule from interactions and transferring it to a new initial state. RIFT addresses this coupled problem by learning how rules shape motion, inferring the governing rule from calibration video while controlling training-frequency effects, and using that rule to predict future dynamics.
TailWorld Benchmark
To examine how calibration evidence and training frequency influence prediction, TailWorld separates observations that establish the query motion state from interactions that reveal the governing rule. Each task pairs a calibration video with a query trajectory under the same rule but different initial positions and velocities. The model observes a collision-free query segment and predicts subsequent motion, using calibration interactions to distinguish rules that the query history alone cannot identify.
We achieve this separation through collision scenes, where restitution and mass ratio affect collision outcomes but leave contact-free motion unchanged. The 2D scenes contain a disc and a movable circular pillar (Figure 2); the 3D scenes contain a sphere and a movable finite cylinder, observed by three calibrated cameras (Figure 3). We vary restitution, mass ratio, or both while keeping appearance and scene geometry fixed. This construction makes calibration informative about the governing rule and subsequent collisions informative about which rule the prediction follows.

Figure 2. Examples of 2D scenes in TailWorld.

Figure 3. Examples of 3D scenes in TailWorld.
Two Futures from the Same Initial State
For each synthetic scene, we generate two future trajectories from exactly the same initial state: one follows the rule demonstrated in the calibration video, and the other follows a reference rule common in training. The model observes the first 0.5 s of query motion and predicts both objects over the next 5 s, corresponding to 15 observed and 150 predicted frames at 30 Hz. The observed segment must be collision-free and identical under both rules. It therefore reveals object positions and velocities without distinguishing the rules, making calibration evidence necessary for rule identification. For samples assessing bias toward common rules, we further require the two futures to differ clearly after subsequent collisions, ensuring that the governing rule materially affects prediction. Both complete trajectories are withheld at inference and serve as references for assessing whether predictions more closely follow the calibration-supported or common rule.
Varying Training Frequency and Available Evidence
We adjust sampling proportions within a fixed training pool to progressively increase common-rule frequency while keeping rare-rule sampling probability unchanged. Rare cases retain 5% of the sampling probability, with a common-to-rare sampling ratio of , where ; other training cases receive the remaining probability. Under a fixed training budget, this preserves the expected number of rare-rule exposures, allowing us to examine the effect of increasing common-rule prevalence. Separately, we provide shorter or longer segments of the same calibration video to test whether additional motion and interaction evidence aids rule identification. The shorter segment ends 0.6 s after the first collision; the longer contains the full 6 s trajectory. For each training distribution, we use
Testing Rule Transfer
both calibration lengths on the same prediction tasks to examine the respective effects of training distribution and current video evidence on prediction.
Beyond assessing prediction under rules infrequent in training, we test whether rules transfer to initial states not paired with them during training. All test tasks combine rules and initial states that have each appeared in training, but whose pairings are excluded from training.
To assess transfer across initial states in real observations, we use pendulum recordings from IRIS [23] and Delys75 [13]. We pair separate recordings under the same physical setting as calibration and query videos, holding out selected settings for testing. These tasks use 0.5 s of query history and a 3 s prediction horizon.
Demonstration Robustness and Control Cases
We complement the transfer tests with two checks on how calibration affects prediction. First, we provide two distinct calibration trajectories per rule, each used separately as input, to test whether predictions remain consistent across demonstrations of the same rule. Second, we retain cases where the demonstrated and common rules produce nearly identical futures, checking for unnecessary prediction changes when rule differences have little effect on the target motion. Full parameters and construction details appear in Appendix B.
Methodology
Problem Formulation and Overview
We study cross-state physical prediction under long-tailed training distributions. Each sample pairs a calibration demonstration with a query sequence governed by the same physical rule but initialized independently. The goal is to infer this rule from observations in and predict the query’s future trajectory from its initial observations .

Figure 4. Overview of RIFT. RIFT infers physical rules from calibration interactions, explicitly controls the influence of training frequency on rule selection, and transfers the inferred rule to trajectory prediction from different initial states.
Rule Inference for Transfer (RIFT) infers a rule from observed motion and applies it to prediction from a new initial state. A learned rule vector represents the environmental rule, shared across trajectories in the same environment. The motion model maps the current state to a base correction and a response matrix, making next-step position predictions linear in this vector. Calibration observations thus yield linear constraints for recovering it. During query prediction, the vector remains fixed while the state-dependent motion response evolves with the query motion.
To let calibration evidence support rare rules, RIFT selects candidates using separate explanatory cost and training frequency terms. Reference rule vectors fitted to training trajectories form a Rule Atlas. For each reference, RIFT balances calibration fit against deviation from that reference, yielding an updated candidate and its explanatory cost. Selection combines this cost with an independently adjustable frequency term. A rare candidate can be selected when its explanatory cost advantage outweighs the frequency preference for common candidates.
Learning Motion Relationships for Rule Identification
This module learns how rules affect motion while keeping predictions linear in the rule vector, enabling its recovery from calibration and reuse for prediction from different initial states. Let concatenate all object coordinates at time , with total dimension . For uniformly spaced frames, constant-velocity extrapolation gives . We learn a correction:
Here, is the predicted position, contains local motion history and scene information, and is the rule vector. Given , the network produces a base correction and a response matrix . The base correction is independent of ; each column of describes the current-state motion response associated with one rule coefficient. Weighted by , these responses form the rule-dependent correction , which combines with to correct the extrapolated position.
Equation (1) allows a shared rule to produce different effects across states. For example, a collision rule induces motion changes that depend on relative positions and velocities. The state-dependent captures this variation, while remains shared within the environment.
This linear form supports both prediction and rule identification with the same motion model. For prediction, the current state and determine the next position. For identification, calibration supplies positions at adjacent time steps, and the network computes and from the observed state. Subtracting the constant-velocity extrapolation and base correction from the observed next position yields a linear constraint on . Because calibration shares one rule vector throughout, constraints across time steps can be combined to solve for .
Recovering the Current Rule from Calibration
This module recovers the rule from observed motion. When calibration cannot determine all coefficients, rule vectors estimated from training environments provide supplementary reference information.
Turning interactions into constraints. Equation (1) yields the calibration constraint
where the superscript denotes calibration. The right-hand side subtracts the constant-velocity extrapolation and base correction from the observed next position; the left-hand side is the response a candidate rule should produce at that state. A single environmental rule governs calibration, so one should explain motion changes throughout the sequence. We therefore solve jointly across time steps, combining rule evidence from different states.
We aggregate constraints over neighboring time steps and weight them by residual reliability to reduce position-difference noise. The resulting system is , where collects the response matrices and the corresponding motion residuals. Thus, measures a candidate’s calibration fitting error, with denoting the Euclidean norm.
Using references to supplement limited observations. Calibration reveals rule information through the motions and interactions present in the sequence. If certain interactions are absent, observations alone may not determine the corresponding coefficients. To supplement this information, we fit a reference vector to independent trajectories from each training environment , minimizing motion fitting error while constraining coefficient magnitude. These vectors form the Rule Atlas.
For each reference , we balance calibration fit against reference deviation to obtain an updated candidate, with minimum objective value
Here, is the explanatory cost associated with reference , denotes transposition, and is a positive-definite constraint matrix shared across references. Its strength is determined by the variability of rule estimates from independent trajectories within the same environment. The first term fits observed motion; the second limits reference deviation. The fitting term drives updates along directions constrained by observations. Where changes in the rule vector have little effect on calibration fit, the reference constraint helps determine its values.
Let denote the minimizer in Eq. (2). Setting the quadratic objective’s gradient to zero yields
This system balances observational and reference constraints by their relative strengths: calibration drives well-constrained directions more strongly, while weakly constrained directions retain more reference information. Each Atlas entry thus yields an updated candidate that can deviate continuously from its reference.
Controlling Frequency Preference and Transferring the Rule
This module combines candidate explanatory costs with an adjustable training-frequency preference to select a rule, then holds it fixed for query prediction from a different initial state. After updating each reference, we select
Here, counts training exposures to environment , controls the frequency preference, indexes the selected reference, and is the rule vector used for prediction. The explanatory cost includes both the updated candidate’s calibration fitting error and its reference deviation. The term favors more frequent references, corresponding to prior weights proportional to .
Equation (4) selects the reference with the lowest total score. A rare reference scores lower than a frequent reference , with , when
Its explanatory cost advantage must therefore outweigh the frequency preference for the frequent reference. Distinguishing interactions alone need not favor the rare candidate; calibration supports it when observed motion yields a sufficient explanatory cost advantage. Reducing lowers this threshold. At , all references have equal prior weights, and selection depends solely on Eq. (2). Since frequency enters only through Eq. (4), its influence can be adjusted without retraining the motion model or refitting the Atlas.
We initialize the motion history with and recursively predict the query using Eq. (1) with . At each step, predicted positions update the state, and and are recomputed while remains fixed. The query’s positions, velocities, and interactions thus evolve jointly under the calibration-inferred rule and the evolving query state.
Training the Framework through Cross-State Prediction
This module learns motion relationships through prediction from a different initial state, enabling a calibration-inferred rule to transfer to a new trajectory. During training, we sample calibration–query pairs from the same environment with independently sampled initial states. We first estimate the rule directly from calibration:
Here, is the training-time rule estimate, and is the regularization strength. The observation term fits calibration, while regularization limits coefficient magnitudes along weakly constrained directions, stabilizing estimation under limited observations. The same procedure estimates Atlas references from independent trajectories in training environments.
We apply to the query, predict recursively from the end of its observation segment, and minimize future trajectory error:
The cross-state prediction loss averages errors over a horizon of steps following the query observation segment. At step , denotes the prediction under the calibration-inferred rule, and denotes the ground-truth position. The expectation is over training environments and independently initialized calibration–query pairs within each environment.
Training with (5) encourages calibration-based transfer without relying on Atlas references. Gradients through the solver and query prediction train and to make inferred coefficients transferable across states within the same environment. Inference retains the learned representation, observation constraints, and prediction mechanism, but replaces zero-centered regularization with regularization around Atlas references to guide estimation along directions weakly constrained by calibration observations. Theoretical analysis appears in Appendix A, hyperparameter and efficiency analyses appear in Appendix D.
Experiment
Experimental Setup
Data and protocols. We evaluate RIFT on the 2D and 3D collision scenes in TailWorld and assess its ability to predict real-world motion using pendulum videos from IRIS [23] and Delfys75 [13]. Each task requires predicting future motion from a different initial state under the same physical rule, given an independent calibration demonstration. The 2D and 3D models are trained and evaluated separately. In synthetic experiments, all trajectory methods share a frozen visual state readout , trained separately for 2D and 3D, to extract states from both query and calibration videos within their permitted observation windows (Appendix B.§ 2). Real pendulum experiments use position trajectories extracted from video and calibrated geometrically.
We conduct two complementary comparisons. First, we compare RIFT with CoPhyNet [4], GEPS [22], CoDA [25], NCF [35], and MaNGO [11] to assess how effectively each method uses demonstrations to adapt to the current dynamics and transfer to new initial states. All methods use the same task splits, state observations, and demonstration durations. To compare adaptation mechanisms, RIFT and the shared-backbone variants of GEPS, CoDA, and NCF use the same pretrained, frozen object-interaction Transformer, with their respective prediction and adaptation modules trained separately. CoPhyNet and MaNGO retain their own dynamics architectures to compare the predictive performance of the complete methods.
Second, we include strong pretrained world models, Cosmos-Predict2.5-2B [34] and Wan2.2-TI2V-5B [43], to assess their physical prediction capabilities on these tasks. Both models are evaluated without fine-tuning: on synthetic tasks, we compare their native inputs without demonstrations against inputs augmented with visual demonstrations; on real-video tasks, we use their native inputs. We extract trajectories from generated videos and score them using the same metrics, reporting trajectory recovery rates and the corresponding evaluation subsets.
Metrics. We use five metrics to quantify prediction errors in position, velocity, and physical quantities. Average Displacement Error (ADE) averages the Euclidean distance between predicted and ground-truth positions over all objects and forecast steps. Final Displacement Error (FDE) measures the average position error across objects at the final forecast step. Both ADE and FDE are measured in meters. Velocity RMSE is the square root of the mean squared Euclidean norm of the velocity error over all objects and forecast steps. Predicted and ground-truth velocities are computed from their respective position trajectories using the same finite-difference scheme.
For synthetic scenes, we additionally report Kinetic Energy MAE and Momentum MAE. The total kinetic energy and the total linear momentum used in this work are defined as
where and denote object mass and velocity, respectively. Both metrics sum quantities over objects, then compute errors between predicted and ground-truth totals using absolute differences for energy and Euclidean distances for momentum, averaging across forecast steps. Lower values indicate better performance for all five metrics.
Trajectory Accuracy and Physical Consistency
As shown in Table 1, RIFT achieves the best trajectory and velocity predictions in both 2D and 3D physical rule transfer tasks, demonstrating consistent advantages across settings. Joint reductions in trajectory and velocity errors show that these advantages extend from object positions to motion evolution. RIFT also leads in 2D kinetic energy and 3D momentum prediction, providing complementary evidence of physical consistency. Together, these results demonstrate RIFT’s ability to combine accurate trajectory prediction with faithful physical quantity estimates under physical rule transfer. Appendix C.§ 1 shows that RIFT maintains competitive accuracy across different demonstrations of the same rule.
| Method | ADE | FDE | Velocity RMSE | Kinetic Energy MAE | Momentum MAE |
| 2D Physical Rule Transfer | |||||
| CoPhyNet [4] | 0.1900 | 0.3980 | 0.1126 | 0.0146 | 0.1168 |
| GEPS [22] | 0.2201 | 0.4992 | 0.0933 | 0.0200 | 0.0879 |
| CoDA [25] | 0.2612 | 0.5235 | 0.1571 | 0.0271 | 0.1550 |
| NCF [35] | 0.2993 | 0.5554 | 0.1195 | 0.0376 | 0.1376 |
| MaNGO [11] | 0.1767 | 0.3485 | 0.1655 | 0.0123 | 0.1696 |
| Cosmos-Predict2.5-2B [34] | 0.4375 | 0.6398 | 0.2356 | 0.0673 | 0.4850 |
| Wan2.2-TI2V-5B [43] | 0.4693 | 0.6331 | 0.2631 | 0.1044 | 0.4735 |
| RIFT | 0.0895 | 0.1388 | 0.0722 | 0.0115 | 0.1033 |
| 3D Physical Rule Transfer | |||||
| CoPhyNet [4] | 0.2023 | 0.4167 | 0.1483 | 0.0137 | 0.1390 |
| GEPS [22] | 0.1916 | 0.3261 | 0.1148 | 0.0057 | 0.1242 |
| CoDA [25] | 0.1927 | 0.3908 | 0.1409 | 0.0234 | 0.0848 |
| NCF [35] | 0.1772 | 0.3918 | 0.1335 | 0.0188 | 0.1635 |
| MaNGO [11] | 0.2081 | 0.3767 | 0.1667 | 0.0270 | 0.1491 |
| Cosmos-Predict2.5-2B [34] | 0.5874 | 0.7565 | 0.3622 | 0.1332 | 0.6641 |
| Wan2.2-TI2V-5B [43] | 0.6859 | 1.1701 | 0.3925 | 0.1058 | 0.6513 |
| RIFT | 0.0989 | 0.1205 | 0.0982 | 0.0059 | 0.0689 |
Table 1. Prediction accuracy and physical consistency on TailWorld.
Physical Rule Transfer under Training Imbalance
Figure 5 compares predicted and ground-truth trajectories, with the common-rule trajectories from the same initial state serving as a reference. We select MaNGO as a competitive baseline and also include Cosmos. MaNGO’s postcollision predictions more closely follow the common-rule reference, indicating a bias toward the frequent rule, while Cosmos also fails to accurately reproduce the true collision response. RIFT closely follows the ground truth throughout the rollout, accurately capturing the ball’s deflection and the cylinder’s limited displacement. These results show that RIFT uses demonstrations to predict motion consistent with the current rule, even when it differs from the pattern prevalent in training. Appendix C.2 shows that full calibration reduces RIFT’s sensitivity to explicit preference for frequent rules.

Figure 5. Trajectory prediction under long-tailed dynamics, where the ground truth trajectories exhibit rare dynamics..
Rule Switching Guided by Demonstrations
We fix the query initial state and vary only the physical rule in the calibration demonstration to test whether predictions change accordingly. In Figure 6, switching gain measures the reduction in trajectory error achieved by correct rather than incorrect demonstrations; direction accuracy measures the fraction of samples whose predicted trajectories shift in the correct direction after changing the demonstration. RIFT leads on both metrics, achieving the largest error reduction and correct directional shifts across all valid samples. Baselines do not attain comparable performance on both metrics simultaneously. By combining correct directional shifts with substantial accuracy gains, RIFT demonstrates its ability to use rule information from demonstrations to adjust future predictions. Appendix C.3 confirms that RIFT also leads in switching gain and direction accuracy in 3D.

Figure 6. Rule switching guided by calibration demonstrations.
Effect of Calibration Evidence on Rule Transfer
We evaluate RIFT with no, short, and full demonstrations under different training frequency ratios to examine how it uses demonstrations to improve prediction. As shown in Figure 7, short demonstrations substantially reduce trajectory errors across all training settings, while full demonstrations yield low errors across training ratios. The inset further shows that, although RIFT has higher errors without demonstrations, the gains from short demonstrations enable it to outperform the baselines shown. These results demonstrate that RIFT effectively uses limited calibration evidence to achieve accurate predictions under imbalanced training distributions. Appendix C.§ 4 confirms RIFT’s lowest 3D ADE with short and full demonstrations across evaluated training ratios.

Figure 7. Effect of calibration evidence on trajectory prediction under varying training imbalance.
Trajectory Prediction in Real Videos
We evaluate RIFT on the IRIS and Delfys75 real video datasets to assess its predictive capability under real observation conditions. As shown in Figure 9, RIFT achieves the lowest ADE on both datasets, outperforming all baselines. This consistent advantage demonstrates that RIFT’s trajectory prediction capability extends beyond synthetic scenes to real videos, providing further evidence of its effectiveness in real settings. Appendix C.§ 5 shows that RIFT also

Figure 9. Trajectory prediction on real videos.
achieves the lowest ADE under parameter shift and unseen rule–state combinations.
Ablation of Key Components
Figure 8 isolates the contributions of calibration updates, the Rule Atlas, and temporal aggregation. Removing calibration updates causes the largest performance drop, highlighting the importance of demonstration-driven rule refinement. The Rule Atlas supports prediction accuracy and physical consistency, while temporal aggregation improves selected physical metrics. Together, these components yield balanced performance across metrics. Appendix C.§ 6 further supports the value of rule updates and Atlas references under short calibration.

Figure 8. Ablation study of the key components in RIFT.
Conclusion
Predicting rare physical dynamics requires reliable rule inference that responds to calibration evidence despite imbalanced training. We introduced TailWorld to independently control training imbalance and calibration informativeness, enabling systematic evaluation of common-rule bias. RIFT infers governing rules from calibration observations and applies them to predict physical dynamics from new initial states. Its key advantage is explicit control over training frequency’s influence on rule selection, enabling calibration evidence to guide the identification of rules rarely observed during training. Experiments demonstrate improved rare-dynamics prediction and lower errors on real videos. These results demonstrate that grounding rule inference in calibration evidence while controlling training-frequency influence can effectively mitigate common-rule bias and improve the prediction of rare physical dynamics.
LLM Usage Statement
OpenAI’s ChatGPT/GPT was used at an assistive level to review and refine author-produced text, related work and citations, code, experimental design and analysis, figures, data preparation, and research ideas, hypotheses, claims, and conclusions. It also provided reviewer-style feedback on the manuscript. The authors take full responsibility for all submitted content.
References
- [1]Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for World Modeling: Visual Details Matter in Atari. In Advances in Neural Information Processing Systems, pages 58757–58791, 2024. 2arxiv.org/abs/2405.12399
- [2]Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. PHYRE: A New Benchmark for Physical Reasoning. In Advances in Neural Information Processing Systems, 2019. 2arxiv.org/abs/1908.05656
- [3]Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. VideoPhy: Evaluating Physical Commonsense for Video Generation. In International Conference on Learning Representations, 2025. 2arxiv.org/abs/2406.03520
- [4]Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. CoPhy: Counterfactual Learning of Physical Dynamics. In International Conference on Learning Representations, 2020. 1, 2, 6, 7
- [5]Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physics. In Advances in Neural Information Processing Systems, 2016. 1, 2arxiv.org/abs/1612.00222
- [6]Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Fei-Fei Li, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating Physical Prediction from Vision in Humans and Machines. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. 1, 2arxiv.org/abs/2106.08261
- [7]Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalanakar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Maria Elisabeth Bechtle, Feryal Behbahani, Stephanie C.Y. Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando De Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative Interactive Environments. In Proceedings of the 41st International Conference on Machine Learning, pages 4603–4623, 2024. 1, 2arxiv.org/abs/2402.15391
- [8]Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. In Advances in Neural Information Processing Systems, 2019. 2arxiv.org/abs/1906.07413
- [9]Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A Compositional Object-Based Approach to Learning Physical Dynamics. In International Conference on Learning Representations, 2017. 2arxiv.org/abs/1612.00341
- [10]Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-Balanced Loss Based on Effective Number of Samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019. 2arxiv.org/abs/1901.05555
- [11]Philipp Dahlinger, Tai Hoang, Denis Blessing, Niklas Freymuth, and Gerhard Neumann. MaGNO: Adaptive Graph Network Simulators via Meta-Learning. In Advances in Neural Information Processing Systems, pages 59419–59445, 2025. 1, 2, 6, 7arxiv.org/abs/2510.05874
- [12]Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1126–1135, 2017. 2arxiv.org/abs/1703.03400
- [13]Alejandro Castañeda Garcia, Jan Warchocki, Jan van Gemert, Daan Brinks, and Nergis Tomen. Learning physics from video: Unsupervised physical parameter estimation for continuous dynamical systems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27924–27933, 2025. 3, 6arxiv.org/abs/2410.01376
- [14]Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Proceedings of the 35th International Conference on Machine Learning, pages 1704–1713, 2018. 2arxiv.org/abs/1807.01613
- [15]David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In Advances in Neural Information Processing Systems, 2018. 1, 2arxiv.org/abs/1809.01999
- [16]Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. In Proceedings of the 36th International Conference on Machine Learning, pages 2555–2565, 2019.arxiv.org/abs/1811.04551
- [17]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning Behaviors by Latent Imagination. In International Conference on Learning Representations, 2020. 2arxiv.org/abs/1912.01603
- [18]Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling Label Distribution for Long-Tailed Visual Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6626–6636, 2021. 2arxiv.org/abs/2012.00321
- [19]Steeven Janny, Fabien Baradel, Natalia Neverova, Madiha Nadri, Greg Mori, and Christian Wolf. Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space. In International Conference on Learning Representations, 2022. 2arxiv.org/abs/2202.00368
- [20]Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long-Tailed Recognition. In International Conference on Learning Representations, 2020. 2arxiv.org/abs/1910.09217
- [21]Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How Far Is Video Generation from World Model: A Physical Law Perspective. In Proceedings of the 42nd International Conference on Machine Learning, pages 28991–29017, 2025. 2arxiv.org/abs/2411.02385
- [22]Armand Kassaï Koupai, Jorge Mifsut Benet, Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari. Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning. In Advances in Neural Information Processing Systems, pages 70659–70692, 2024. 1, 2, 6, 7
- [23]Rasul Khanbayov, Mohamed Rayan Barhdadi, Erchin Serpedin, and Hasan Kurban. IRIS: A real-world benchmark for inverse recovery and identification of physical dynamic systems from monocular video. In Proceedings of the 19th European Conference on Computer Vision, pages 366–383. Springer Nature Switzerland, 2026. 3, 6arxiv.org/abs/2603.16432
- [24]Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural Relational Inference for Interacting Systems. In Proceedings of the 35th International Conference on Machine Learning, pages 2688–2697, 2018. 2
- [25]Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to New Physical Systems via Context-Informed Dynamics Model. In Proceedings of the 39th International Conference on Machine Learning, pages 11283–11301, 2022. 1, 2, 6, 7arxiv.org/abs/2202.01889
- [26]Bingxuan Li, Yining Hong, Cheng Qian, Hyeonjeong Ha, Jiateg Liau, Zhenhaiwang Wang, Yue Guo, Yunzhu Li, and Heng Ji. Trimming the Long-Tail of Visual World Modeling Evaluation. arXiv preprint arXiv:2606.24256, 2026. 2arxiv.org/abs/2606.24256
- [27]Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids. In International Conference on Learning Representations, 2019. 2arxiv.org/abs/1810.01566
- [28]Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal Discovery in Physical Systems from Videos. In Advances in Neural Information Processing Systems, pages 9180–9192, 2020. 2arxiv.org/abs/2007.00631
- [29]Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In Proceedings of the 35th International Conference on Machine Learning, pages 3122–3130, 2018. 2arxiv.org/abs/1802.03916
- [30]Yunhao Luo, Wentao He, Yinpei Dai, Hyelin Nam, Hongze Fu, Joyce Chai, Nima Fazeli, and Jeong Joon Park. PhysiCalWorld: In-Context Physics Calibration for World Models. arXiv preprint arXiv:2610.30125, 2026. 2arxiv.org/abs/2610.30125
- [31]Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021. 2arxiv.org/abs/2007.07314
- [32]Vincent Micheli, Eloi Alonso, and Francois Fleuret. Transformers are Sample-Efficient World Models. In International Conference on Learning Representations, 2023. 2arxiv.org/abs/2209.00588
- [33]Saman Motamed, Laura Culp, Kevin Sversky, Priyank Jain, and Robert Geirhos. Do Generative Video Models Understand Physical Principles? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 948–958, 2026. 2arxiv.org/abs/2501.09038
- [34]NVIDIA, Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, Prithvijit Chattopadhyay, Mike Chen, Yongxin Chen, Yu Chen, Shuai Cheng, Yin Cui, Jenna Diamond, Yifan Ding, Jiaojiao Fan, Linxi Fan, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Ruiyuan Gao, Yun- hao Ge, Jinwei Gu, Aryaman Gupta, Siddharth Gururani, Imad El Hanafi, Ali Hassani, Zekun Hao, Jacob Huffman, Joel Jang, Pooya Jannaty, Jan Kautz, Grace Lam, Xuan Li, Zhaoshou Li, Maosheng Liao, Chen-Hsuan Lin, Tsung-Yi Lin, Yen-Chen Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Seungjun Nah, Yashraj Narang, Abhijeet Panaskar, Lindsey Pavao, Trung Pham, Morteza Ramezani, Fitsum Reda, Scott Reed, Xuanchi Ren, Haonan Shao, Yue Shen, Stella Shi, Shuran Song, Bartosz StefaniaK, Shangkun Sun, Shitao Tang, Sameena Tasmeen, Lyne Tchapmi, Wei-Cheng Tseng, Jibin Varghese, Andrew Z. Wang, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Jiashu Xu, Dinghao Yang, Xiaodong Yang, Haotian Ye, Seonghyeon Ye, Xiaohui Zeng, Jing Zhang, Qinsheng Zhang, Kaiwen Zheng, Andrew Zhu, and Yuke Zhu. World simulation with video foundation models for physical AI. arXiv preprint arXiv:2511.00062, 2026. 6, 7
- [35]Rousseau Desmond Nzoyem, David A. W. Barton, and Tom Deakin. Neural CotFlows for Meta-Learning of Dynamical Systems. In International Conference on Learning Representations, 2025. 1, 2, 6, 7
- [36]Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning Mesh-Based Simulation with Graph Networks. In International Conference on Learning Representations, 2021. 1, 2arxiv.org/abs/2010.03409
- [37]Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature Human Behaviour, 6:1257–1267, 2022. 2DOI
- [38]Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced Meta-Softmax for Long-Tailed Visual Recognition. In Advances in Neural Information Processing Systems, pages 4175–4186, 2020. 2arxiv.org/abs/2007.10740
- [39]Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Ler er, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning. arXiv preprint arXiv:1803.07616, 2018. 1, 2arxiv.org/abs/1803.07616
- [40]Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Had sell, and Peter Battaglia. Graph Networks as Learnable Physics Engines for Inference and Control. In Proceedings of the 35th International Conference on Machine Learning, pages 4470–4479, 2018. 2
- [41]Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to Simulate Complex Physics with Graph Networks. In Proceedings of the 37th International Conference on Machine Learning, pages 8459–8468, 2020. 1, 2
- [42]Hsiao-Yu Tung, Mingyu Ding, Zhenfang Chen, Daniel M. Bear, Chuang Gan, Joshua B. Tenenbaum, Daniel L. K. Yamins, Judith Fan, and Kevin A. Smith. Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties. In Advances in Neural Information Processing Systems, pages 67048–67068, 2023. 2arxiv.org/abs/2306.15668
- [43]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyu an Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. 2025. 6, 7arxiv.org/abs/2503.20314
- [44]Jiajun Wu, Ilker Yildirim, Joseph J. Lim, William T. Freeman, and Joshua B. Tenenbaum. Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning. In Advances in Neural Information Processing Systems, 2015. 2
- [45]Jiajun Wu, Joseph J. Lim, Hongyi Zhang, Joshua B. Tenenbaum, and William T. Freeman. Physics 101: Learning Physical Object Properties from Unlabeled Videos. In Proceedings of the British Machine Vision Conference, pages 39.1–39.12, 2016.DOI
- [46]Jiajun Wu, Erika Lu, Pushmeet Kohli, William T. Freeman, and Joshua B. Tenenbaum. Learning to See Physics via Visual De-animation. In Advances in Neural Information Processing Systems, 2017. 1, 2
- [47]Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into Deep Imbalanced Regression. In Proceedings of the 38th International Conference on Machine Learning, pages 11842–11851, 2021. 2arxiv.org/abs/2102.09554
- [48]Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: CoLlision Events for Video Representation and Reasoning. In International Conference on Learning Representations, 2020. 2arxiv.org/abs/1910.01442
- [49]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning. In Proceedings of the 42nd International Conference on Machine Learning, pages 79115–79135, 2025. 1, 2