WorksComputer Science
SciWorldBench: Do Video World Models Really Understand Scientific Experiments?
Abstract
While the ability of video generation models to accurately depict scientific phenomena has received growing attention, determining whether these models genuinely understand the underlying principles remains a critical challenge. Existing benchmarks typically evaluate video samples in isolation, making it difficult to expose models that simply fit superficial visual priors. In scientific experiments, a small change in one condition produces a markedly different outcome even when the rest of the setup remains fixed. Capitalizing on this sensitivity, we introduce SciWorldBench, a benchmark comprising paired scientific experiments. Each pair varies a single condition in an otherwise fixed setup, yielding two mutually exclusive expected outcomes. Each generation prompt specifies the condition but omits the expected outcome. We evaluate ten text-and-image-to-video (TI2V) models using an independent vision-language model (VLM) that assesses each video without access to the condition text or the label identifying the correct outcome. A pair is scored as correct only when both videos depict their respective scientifically correct outcomes. Across all ten models, we find that the two videos in an experimental pair frequently depict the same outcome even though the conditions require different results. This recurring failure suggests that current video world models struggle to apply the underlying scientific principles reliably during generation. We will publicly release the benchmark dataset and open-source the evaluation code.
Introduction
Text-and-image-to-video (TI2V) generation has advanced rapidly [1, 6, 10, 13, 18, 26, 28, 34, 38]. Recent models, including Seedance 2.5 [6], Kling 3.0 [18], Wan 3.0 [1], and Vidu Q3 Pro [34], generate high-resolution videos with realistic appearance and smooth motion from a first frame and a text prompt. These models are increasingly used as video world models to simulate scene evolution. In scientific experiments, changing a single condition can lead to a markedly different outcome while the rest of the setup remains unchanged. Consider an orange released into water: air in the intact peel lowers the fruit’s average density, allowing it to float, whereas a peeled orange sinks. The two experiments otherwise share the same setup. In the example shown in Figure 1, Seedance 2.5 generates videos in which the orange floats under both conditions, although the peeled orange should sink. Generating the correct outcome under one condition therefore does not establish that the model understands how the experimental condition affects the outcome. This failure motivates our central question: do video world models really understand scientific experiments?
Figure 1. Overview of SciWorldBench. Top: benchmark construction. Bottom: VLM evaluation. The prompts, five VLM scoring questions, and answers shown in the figure are shortened for illustration.
Existing benchmarks broaden task coverage and refine evaluation criteria [2, 11] to assess whether models can generate physically correct videos from prompts specifying scenes, events, and outcomes [3, 12]. However, generating the correct outcome does not establish that the model derived it from the given conditions through causal reasoning. A complementary line of work examines responses to controlled changes in autonomous driving and robotic manipulation [7] and simulated physical scenes [4, 19, 20]. Their focus on embodied interactions and mechanical events provides limited evidence of whether models understand condition–outcome relationships across a broader range of scientific experiments.
We introduce SciWorldBench, a benchmark of 222 experiment pairs across eight categories. Each pair holds the setup fixed and varies one condition, with mutually exclusive expected outcomes. Generation prompts specify the condition but omit the outcome. Half of the pairs use an identical first frame, so the condition is specified only in text. In the other half, the condition is visible in the first frame. The two videos are generated in separate requests, and a pair passes only when both show their respective scientifically correct outcomes. A VLM uses visual evidence to check experimental prerequisites and separately assess whether each candidate outcome occurs. We score a video as correct only when all prerequisite checks pass and the judgments confirm the expected outcome while ruling out the alternative.
Across ten TI2V models, Paired Accuracy (PA) ranges from 1.4% to 24.8%, and the highest Scientific Outcome Accuracy (SOA) is 53.8%. A recurring failure is that models generate the same outcome despite the contrasting conditions: among pairs in which both videos pass the prerequisite checks and each shows exactly one outcome, the Collapse Rate (CR) is 55.7%–91.7%. Providing the correct outcome in the prompt improves generation for all six tested open-weight models, but most still tend to generate the same outcome under both conditions. The highest PA is 33.3%, and CR exceeds 50% for five of the six models. Both Cosmos models combine understanding and generation, yet their ability to infer experimental outcomes in text does not reliably carry over to video generation. Cosmos3-Super, for example, predicts both outcomes correctly in text for 63.1% of pairs but generates both correctly in only 9.0%.
Our contributions are as follows.
We introduce SciWorldBench, a benchmark of 222 experiment pairs across eight categories for evaluating whether video world models understand scientific experiments. Each pair specifies contrasting experimental conditions and their distinct, visually verifiable outcomes.
We establish an evaluation protocol that distinguishes unmet experimental prerequisites from incorrect outcomes and assesses correctness under each condition. The VLM identifies the outcome shown in a video without receiving the condition text or the correct-outcome label, and we validate its judgments against human consensus.
We evaluated 10 mainstream TI2V models on our benchmark. Across all the models, the highest PA was 24.8%, and paired videos frequently showed the same outcome even though the two conditions required different results. These findings show that current video world models lack reliable scientific understanding in generation.
Related Work
Physical fidelity in video generation. General video generation benchmarks evaluate visual quality and text–video alignment [16, 17, 24], while other benchmarks examine temporal change and compositionality [9, 35, 40]. VBench-2.0 [42] includes physics and common sense in its evaluation of intrinsic faithfulness. Benchmarks dedicated to physical fidelity assess whether generated actions and material interactions obey physical laws [2, 25, 33, 37]. Some tasks specify both the event and its outcome in the prompt, testing how accurately a model depicts the prescribed motion [3, 11, 12]. Other benchmarks evaluate physical prediction by comparing generated continuations with recordings of real experiments [27, 32] or measuring whether generated dynamics satisfy conservation laws [36]. Physically correct motion alone, however, does not establish that the model derived the outcome from the given conditions through causal reasoning. When the prompt explicitly states the outcome, the model can depict it without making that inference, leaving its understanding of the causal relationship untested.
Scientific understanding and reasoning in video generation. Scientific reasoning has been studied both in the interpretation of recorded experiments [8, 22, 39] and in the generation of scientific phenomena [23]. VideoScience-Bench [15] evaluates whether models generate the phenomena predicted by interacting concepts in physics and chemistry. Sci-VBench [41] uses expert criteria to assess whether generated processes are scientifically and causally correct across disciplines. A correct result for one experimental setup does not establish whether the model can also generate the appropriate alternative when one condition changes.
Condition dependence of generated outcomes. Prior work examines how changes in physical parameters affect generated motion [21] and whether a single set of physical parameters can explain the trajectories predicted under different actions from the same initial state [14]. CRONOS [4] studies physical consistency as viewpoint or appearance varies while the event type remains fixed. Other work examines how interventions alter driving and robotic manipulation outcomes [7], how initial geometry determines mechanical outcomes [20], and how changes propagate through a sequence of physical interactions [19]. These benchmarks focus primarily on embodied interactions and mechanical events, leaving open whether models understand how conditions determine outcomes across a broader range of scientific experiments. SciWorldBench examines this question across eight experimental categories by pairing two conditions of the same experiment that require mutually exclusive outcomes.
SciWorldBench
SciWorldBench tests whether video world models understand scientific experiments by evaluating whether they generate the scientifically correct outcome under each condition. The benchmark contains 222 pairs of experiments. Within each pair, the setup is fixed except for one condition, and the two conditions have different expected outcomes. We present the task formulation in § 3.1, benchmark construction in § 3.2, the evaluation protocol in § 3.3, and the metrics in § 3.4.
Task Formulation
We represent a pair as . The shared setup contains the scene, procedure, and camera requirements shared by the two sides. Conditions and distinguish the two sides and are expressed as short clauses in the prompts. A condition may specify a property, such as a material, temperature, or liquid, or an action, such as whether a bottle is squeezed. and denote the first frames of the two sides. and denote the corresponding expected outcomes, which are mutually exclusive and visually assessable from the frames.
For side , a TI2V model receives the first frame and the prompt and generates a video . The prompt specifies the condition but omits the outcome, requiring the model to generate the result without explicit outcome guidance. The two videos are generated in separate requests. Each includes only the first frame and prompt for its own side and provides no access to the other condition. A video is correct if it passes all three prerequisite checks, shows , and does not show the other outcome. A pair passes only when both and are correct.
Benchmark Construction
We use GPT-5.6-sol [29] to generate 600 candidate pairs of 8-second video experiments, following the pipeline in Figure 1. Each candidate defines a shared experimental setup and two alternative conditions, which are combined to form a pair of video generation prompts. Each candidate also specifies the expected outcome under each condition and a brief explanation of the causal relationship between the condition and its expected outcome. We then conduct expert review to assess the candidates along three dimensions. (1) For clarity of the experimental description, reviewers check that each candidate specifies the shared setup and how the two conditions differ. (2) For scientific validity, they verify that each condition leads to its stated outcome under that setup. (3) To assess suitability for video evaluation, they check whether each outcome is observable within eight seconds and distinguishable from the alternative using only the video frames. After review and refinement, 222 pairs are retained. We then use GPT-6 Astra [30] to reformulate the video generation prompts into a structured format with four labeled fields: scene, condition, procedure, and camera requirements. The two video generation prompts in each pair differ only in the condition field, and neither explicitly specifies the expected outcome or its explanation.
In the TI2V setting, each experimental condition is associated with a first frame. We generate and edit these reference frames using Qwen-Image 2.1 [31]. The side-A frame is generated from the shared experimental setup and its corresponding condition. When the condition is visually distinguishable, the side-B frame is edited from the side-A frame while preserving the scene layout, camera viewpoint, and unrelated objects. When the condition is not visually distinguishable in a static image, as in the ice cube experiments with water and alcohol (Figure 2), the same first frame is used for both sides, with the difference specified only in the generation prompts. All reference frames are initially created at pixels and resized to match each model’s output resolution before video generation.

Figure 2. Example pairs from SciWorldBench. Each pair varies one condition while holding the setup fixed; the conditions are listed below the images. Top: both sides use the same first frame, and the condition is specified only in the prompt. Bottom: the condition is visible in the first frame.
Finally, we organize the 222 pairs into eight categories based on the underlying physical mechanisms (Figure 3). We also divide them into two equally sized subsets according to whether the paired videos share the same first frame: 111 same-first-frame pairs and 111 different-first-frame pairs.
Figure 3. Categories of SciWorldBench. The left panel illustrates the eight categories with representative objects and phenomena. The bars on the right show how the 222 experimental pairs are distributed across them.
Evaluation Protocol
We use doubao-seed-2-1-pro [5] at temperature 0 to evaluate the two videos in each experimental pair separately (Figure 1). For each video, the VLM receives 49 full-resolution frames sampled uniformly over time, their timestamps, and a description of the shared experimental setup. We withhold the condition-specific text, the name of the generating model, the A/B side label, and the label identifying the expected outcome.
We use the same five questions to score both videos in a pair. We shuffle the question order for each video using a fixed seed. Three questions check whether the target’s behavior is visible, whether it remains the same object throughout the video, and whether the experimental preconditions are met. The other two questions each ask whether the frames show a particular candidate outcome. We describe each outcome in terms of observable behav… ior, removing condition-specific attributes and referring to the target in neutral terms. For example, we rewrite “The peeled orange sinks to the bottom” as “The round fruit sinks through the water and settles on the bottom of the tank.”
For each question, the VLM returns an answer, a brief observation, and a rationale in a fixed JSON format. Both yes and no require the indices of at least two supporting frames and the time interval they span. When the visual evidence is insufficient, the VLM answers unknown. We score a video as correct only if all three prerequisite checks pass and the VLM answers yes for the outcome expected under that condition and no for the alternative.
Metrics
We report four metrics for each model. Let be the set of pairs. For a pair , and are the videos generated for its two sides. For a video , let if visibility, identity, and execution are all answered yes and otherwise, and let be the answers to the questions on whether shows and .
Prerequisite Pass Rate (PPR) is the share of the videos that pass all three prerequisite checks:
This metric reports the proportion of videos that satisfy the prerequisites for outcome evaluation.
Scientific Outcome Accuracy (SOA) is the share of the videos that are correct. For side , with the other side, the video is correct () when it passes the prerequisite checks, shows its own outcome, and does not show the other one:
SOA is
A video is scored as incorrect if any required answer is unknown or if both outcomes are present. SOA measures success for individual conditions, as in evaluations that score each prompt separately.
Paired Accuracy (PA) is our main metric:
Evaluating the two sides independently can overestimate performance across conditions: a commonly generated outcome may be correct for only one side. PA counts a pair only when both outcomes are correct, requiring successful generation under both experimental conditions. A model that generates the same outcome under both conditions receives a pair score of 0.
Figure 5.
Collapse Rate (CR) quantifies how often a model generates the same outcome under both conditions. If any prerequisite check fails, is undefined. For videos that pass all three checks (), we use the following rule.
A pair is decided when both videos pass the prerequisite checks and and are both defined, and we write for the decided pairs. The CR is
PA measures whether a model generates the correct outcomes under both experimental conditions. CR characterizes a specific failure among decided pairs: both generated videos satisfy the prerequisite checks and show the same clearly identifiable outcome, so the observed outcome is unchanged across the two conditions. A lower CR indicates a smaller proportion of same-outcome failures among decided pairs. It does not necessarily imply higher PA, since the two outcomes can also be reversed.
Experiments
Setup
We evaluate ten TI2V models. Four are commercial: Seedance 2.5 [6], Kling 3.0 [18], Wan 3.0 [1], and Vidu Q3 Pro [34]. Six are open-weight: MiniMax-H3 (33B) [26], Cosmos3-Super (64B) and Cosmos3-Nano (16B) [28], LTX-2.3 (22B) [13], and Wan2.2-I2V-A14B (mixture of experts, 14B active) and Wan2.2-TI2V-5B (5B) [38]. For each condition, all models use the same prompt and source image (§ 3.2). For each pair, we generate one video per condition in separate requests using the same random seed (42) to reduce sampling variability when comparing outcomes across conditions. We set the target video duration to 8 seconds for all ten models. To make full use of each model’s generation capabilities, we follow its official recommendations for both output resolution and frame rate.
Main Results
Across the ten TI2V models, PA ranges from 1.4% to 24.8% (Table 1). The four commercial models occupy the top four positions, with PA ranging from 12.6% to 24.8%, compared with 1.4% to 9.0% for the six open-weight models. Seedance 2.5 ranks first, followed by Kling 3.0 and Vidu Q3 Pro. The four commercial models, Cosmos3-Super, and MiniMax-H3 pass all three prerequisite checks in 88.7%–94.6% of their videos, yet their SOA ranges from only 37.8% to 53.8%. Even Seedance 2.5, which generates the scientifically correct outcome in 53.8% of individual videos, does so under both conditions in only 24.8% of all experimental pairs.
| Model | Type | Params | PPR | SOA | PA | CR |
| Seedance 2.5 | Commercial | – | 94.6 | 53.8 | 24.8 | 55.7 |
| Kling 3.0 | Commercial | – | 92.6 | 45.0 | 15.8 | 66.9 |
| Vidu Q3 Pro | Commercial | – | 88.7 | 44.6 | 15.3 | 72.2 |
| Wan 3.0 | Commercial | – | 92.1 | 43.2 | 12.6 | 74.0 |
| Cosmos3-Super | Open-weight | 64B | 90.1 | 37.8 | 9.0 | 79.3 |
| MiniMax-H3 | Open-weight | 33B | 91.7 | 39.9 | 7.2 | 84.3 |
| LTX-2.3 | Open-weight | 22B | 78.4 | 32.0 | 7.2 | 75.6 |
| Cosmos3-Nano | Open-weight | 16B | 77.0 | 30.4 | 5.4 | 82.1 |
| Wan2.2-I2V-A14B | Open-weight | 14B active | 75.0 | 28.8 | 3.6 | 87.1 |
| Wan2.2-TI2V-5B | Open-weight | 5B | 48.0 | 16.2 | 1.4 | 91.7 |
Table 1. Results on SciWorldBench (%). PPR and SOA are computed over the 444 videos of each model, and PA over the 222 pairs. CR is computed over the decided pairs, in which both videos pass the prerequisite checks and each shows exactly one outcome. We assess model performance across four complementary metrics and rank models separately on each. In each column, the best and second-best results are shaded light blue and light yellow.
Among decided pairs, all ten models more often repeat the same outcome than generate different outcomes across conditions (Figure 4). CR ranges from 55.7% for Seedance 2.5 to 91.7% for Wan2.2-TI2V-5B. When a model repeats an outcome, it is correct under one condition but fails to produce the different outcome required by the other.
Figure 4. Outcome distribution among decided pairs for each model. A pair is decided when both videos pass all prerequisite checks and each shows exactly one candidate outcome. The bars show the percentages of these pairs with both outcomes correct (light blue), the two expected outcomes reversed (light gray), or the same outcome under both conditions (dark blue). The last percentage is the Collapse Rate (CR).
Outcome Stated in the Prompt
To test whether stating the expected outcome reduces this repetition, we append the correct outcome to each condition’s original prompt and regenerate the videos with the six open-weight models. We score the new videos using the same protocol.
With the outcome given, all six models achieve higher SOA and PA, and all six show lower CR (Table 2). Averaged across models, PA rises from 5.6% to 19.5%. MiniMax-H3 shows the largest improvement. Its PA rises from 7.2% to 33.3%, the highest among the six models, and its CR falls from 84.3% to 40.2%. Wan2.2-TI2V-5B shows the smallest reduction in CR, from 91.7% to 81.1%. Even with explicit outcome guidance, five of the six models still generate the same outcome under both conditions in more цә than half of their decided pairs.
| Model | PA | SOA | CR |
| MiniMax-H3 | 7.2 | 39.9 | 84.3 |
| + outcome stated | 33.3 (+26.1) | 59.5 (+19.6) | 40.2 (−44.1) |
| Cosmos3-Super | 9.0 | 37.8 | 79.3 |
| + outcome stated | 27.5 (+18.5) | 56.5 (+18.7) | 50.4 (−28.9) |
| Cosmos3-Nano | 5.4 | 30.4 | 82.1 |
| + outcome stated | 19.4 (+14.0) | 47.5 (+17.1) | 58.6 (−23.5) |
| LTX-2.3 | 7.2 | 32.0 | 75.6 |
| + outcome stated | 19.4 (+12.2) | 45.9 (+13.9) | 57.5 (−18.1) |
| Wan2.2-I2V-A14B | 3.6 | 28.8 | 87.1 |
| + outcome stated | 14.0 (+10.4) | 40.1 (+11.3) | 69.1 (−18.0) |
| Wan2.2-TI2V-5B | 1.4 | 16.2 | 91.7 |
| + outcome stated | 3.6 (+2.20) | 20.7 (+4.50) | 81.1 (−10.6) |
Table 2. Results (%) without and with explicit outcome guidance. For each model, the first row reports the original setting and the second includes the outcome in the prompt. Colored values in parentheses give percentage-point changes from the first row (green for increases and red for decreases). CR is computed over decided pairs separately for each setting.
Figure 5 (PDF p. 7). Textual outcome prediction and video generation for the same models across 222 pairs. Light blue bars show the percentage of pairs with correct text predictions under both conditions. Dark blue bars show video PA, the percentage of pairs in which both videos depict their respective correct outcomes.
Predicting Experimental Outcomes in Text
We return to the original prompts, which omit the expected outcome, and compare how often Cosmos3-Super and Cosmos3-Nano predict both outcomes correctly in text with how often they depict both correctly in video. For each condition, we provide the full original generation prompt, which specifies the scene, condition, procedure, and camera requirements. We omit the first frame and ask what happens. The question is identical for both conditions of a pair. We manually assess each answer against the expected outcome for its condition. Both models predict the two outcomes correctly in text far more often than they generate them correctly in video (Fig. 5). Cosmos3-Super gives correct text answers for both conditions in 63.1% of pairs, compared with a video PA of 9.0%. For Cosmos3-Nano, the corresponding rates are 44.6% and 5.4%. Even when these models correctly infer both outcomes in text, they often fail to depict those outcomes in their respective videos.
Results by First-Frame Group
Table 3 compares the two first-frame groups in the original generation setting, with 111 experimental pairs in each group. When the condition is visible in the first frame, PA ranges from 2.7% to 30.6%. When the two conditions share a first frame and differ only in text, PA ranges from 0.0% to 18.9%. Nine models have lower PA in the shared-frame group, while Cosmos3-Super achieves 9.0% in both groups. All ten models have higher CR in the shared-frame group. The largest difference occurs for Seedance 2.5, whose CR is 43.8% with different first frames and 67.2% with a shared first frame. This pattern suggests that models are less responsive to a change in experimental conditions when the prompt provides the only cue to that change.
| Model | PA different | PA same | CR different | CR same |
| Seedance 2.5 | 30.6 | 18.9 | 43.8 | 67.2 |
| Kling 3.0 | 16.2 | 15.3 | 65.5 | 68.1 |
| Vidu Q3 Pro | 16.2 | 14.4 | 68.4 | 75.4 |
| Wan 3.0 | 14.4 | 10.8 | 67.2 | 79.7 |
| Cosmos3-Super | 9.0 | 9.0 | 75.5 | 82.4 |
| MiniMax-H3 | 8.1 | 6.3 | 79.6 | 88.1 |
| LTX-2.3 | 8.1 | 6.3 | 65.8 | 82.7 |
| Cosmos3-Nano | 8.1 | 2.7 | 71.4 | 90.6 |
| Wan2.2-I2V-A14B | 6.3 | 0.9 | 80.6 | 91.8 |
| Wan2.2-TI2V-5B | 2.7 | 0.0 | 83.3 | 100.0 |
Table 3. Results by first-frame group (%), 111 pairs per group. In each column, the best and second-best results are shaded light blue and light yellow.
Results by Scientific Category
Each of the eight scientific categories contains 16 to 34 experimental pairs. We pool results across the ten models
within each category (Table 4). Optics has the highest PA (18.1%) and the lowest CR (65.8%), while chemistry has the lowest PA (6.2%) and the highest CR (83.6%). For individual videos, SOA ranges from 30.2% in mechanics to 45.6% in optics. Even in optics, where models are most accurate on both individual videos and pairs, nearly two thirds of decided pairs show the same outcome under both conditions.
| Category | Pairs | SOA | PA | CR |
| Mechanics | 27 | 30.2 | 8.1 | 71.3 |
| Fluids | 33 | 37.4 | 10.6 | 73.9 |
| Thermodynamics | 32 | 40.8 | 10.0 | 76.5 |
| Optics | 26 | 45.6 | 18.1 | 65.8 |
| Electromagnetism | 29 | 35.2 | 7.6 | 80.6 |
| Chemistry | 34 | 32.4 | 6.2 | 83.6 |
| Materials | 25 | 36.8 | 10.0 | 75.2 |
| Biology | 16 | 42.2 | 14.4 | 72.6 |
Table 4. Results by category (%), with metrics pooled across all ten models. The best and second-best results in each column are shaded light blue and light yellow, respectively.
Discussion
Our empirical findings expose a sharp divide between recognizing what should happen and synthesizing how it unfolds. In this section, we trace this failure to two underlying bottlenecks—outcome-repetition bias stemming from generation without reasoning, and a modality disconnect between reasoning and generation.
Outcome-repetition bias stems from generation without reasoning. Because this bias is systematic, persists under explicit instruction, and occurs across independent generation requests, it reflects a shared prior rather than cross-run copying. We attribute this bias to visual associations that text-and-image-to-video (TI2V) models learn from observational videos. During generation, the models rely more heavily on the association between a scene’s initial appearance and a familiar outcome than on the experimental condition specified in the prompt. As a result, they favor the same outcome in both videos of a pair even when the two conditions require different results. Current models behave more like renderers than world simulators, producing visually realistic videos that often fail to show how changing an experimental condition alters the outcome. Even when the expected outcome is explicitly stated in the prompt, they merely translate that label into surface rendering, accounting for their partial compliance without true reasoning.
The text–video gap reflects a modality disconnect. Both unified Cosmos models predict the correct outcomes for the two experimental conditions far more often in text than they depict them in video (Fig. 5 and § 4.4). This gap may partly reflect an intrinsic difference in task demands. In the text task, the model makes a low-dimensional judgment about which outcome follows from the stated experimental condition. Video generation poses a higher-dimensional reasoning problem because the model must map that same condition to a visual trajectory that captures how the scene evolves across space and time to produce the corresponding outcome. Even when the prompt supplies the correct outcome, removing the need to infer it, the best video PA among the six tested open-weight models is 33.3% (Table 2). In the original setting, all ten models show higher CR for pairs distinguished only by text than for pairs in which the condition difference is also visible in the first frame (Table 3). Together, these findings suggest that textual information sufficient to identify an outcome does not reliably guide its realization in video. Whether this difficulty arises from a mismatch between the representations used for inference and generation remains open.
Conclusion
We introduced SciWorldBench to test whether video world models understand scientific experiments. In each of its 222 pairs, two versions of the same experiment differ in one condition and have mutually exclusive expected outcomes. A VLM assesses each video without access to the condition text or the expected-outcome label. A pair passes only if both videos depict the scientifically correct outcomes for their respective conditions. Across ten TI2V models, paired accuracy (PA) ranges from 1.4% to 24.8%. Models often repeat the same outcome under both conditions, particularly when the first frame is shared and only the text distinguishes the two inputs. Stating the correct outcome improves generation for all six tested open-weight models, yet the highest PA is 33.3%. The two Cosmos models predict both outcomes correctly in text far more often than they depict them correctly in video. Together, these results suggest that current video world models have a limited understanding of how experimental conditions determine outcomes.
Future work could use paired training videos to help models learn how a change in one experimental condition affects the outcome. When both videos start from the same frame, the model must use the condition stated in the prompt to infer which outcome should occur. A latent or symbolic representation of that prediction could guide video synthesis and provide a reference for checking the generated result. Beyond checking whether an individual video matches that prediction and appears plausible, benchmarks should test whether each video in a controlled pair depicts the scientifically correct outcome for its condition.
Limitations. Two limitations warrant consideration. First, our mechanistic interpretation is derived from behavioral outputs and text predictions rather than from probing internal representations; the deeper mechanisms through which video generation models produce these phenomena remain to be investigated. Second, the accuracy of our evaluation depends heavily on the VLM judge: even with a state-of-the-art model, its agreement with human annotators remains imperfect, which can introduce scoring errors.
References
- [1]Alibaba Cloud. Wan 3.0 video generation, 2026.
- [2]Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. VideoPhy: Evaluating physical commonsense for video generation. In International Conference on Learning Representations, pages 102075–102121, 2025.arxiv.org/abs/2406.03520
- [3]Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, and Kai-Wei Chang. VideoPhy-2: A challenging action-centric physical commonsense evaluation in video generation. In International Conference on Learning Representations, pages 118456–118470, 2026. 2arxiv.org/abs/2503.06800
- [4]León Beigristain, Olaf Dünkel, and Adam Kortylewski. CRONOS: Benchmarking counterfactual physical consistency in video models. arXiv preprint arXiv:2605.23699, 2026. 2, 3
- [5]ByteDance Seed. Seed2.1 model card, 2026. 4
- [6]ByteDance Seed. Seedance 2.5, 2026. 2, 5
- [7]Kunlin Cai, Rui Song, Jinghuai Zhang, Kaiyuan Zhang, Pranav Bodapati, Alicia Yu, Fnu Suya, Mohammad Rostami, Jiaqi Ma, and Yuan Tian. What-If World: A causal benchmark for general world models in embodied scenarios. arXiv preprint arXiv:2605.27589, 2026. 2, 3arxiv.org/abs/2605.27589
- [8]Andong Deng, Taojiannan Yang, Shoubin Yu, Lincoln Spencer, Mohit Bansal, Chen Chen, Serena Yeung-Levy, and Xiaohan Wang. SciVideoBench: Benchmarking scientific video reasoning in large multimodal models. In Proceedings of the 43rd International Conference on Machine Learning, pages 24373–24406. PMLR, 2026. 3arxiv.org/abs/2510.08559
- [9]Weixi Feng, Jiachen Li, Michael Saxon, Tsu-Jui Fu, Wenhu Chen, and William Yang Wang. TC-Bench: Benchmarking temporal compositionality in conditional video generation. In Findings of ACL, pages 4638–4662, 2025. 2DOI
- [10]Google DeepMind. Veo 3.1, 2025. 2
- [11]Jing Gu, Xian Liu, Yu Zeng, Ashwin Nagarajan, Fangrui Zhu, Daniel Hong, Yue Fan, Qianqi Yan, Kaiwen Zhou, Ming-Yu Liu, et al. PhyWorldBench: A comprehensive evaluation of physical realism in text-to-video models. In International Conference on Learning Representations, pages 75130–75164, 2026. 2arxiv.org/abs/2507.13428
- [12]Xuyang Guo, Jiajun Huo, Zhenmei Shi, Zhao Song, Jiahao Zhang, and Jiale Zhao. T2VPhysBench: A first-principles benchmark for physical consistency in text-to-video generation. arXiv preprint arXiv:2505.00337, 2025. 2arxiv.org/abs/2505.00337
- [13]Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifshitz, Dudu Moshe, Eitan Porat, et al. LTX-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2601.03233, 2026. 2, 5arxiv.org/abs/2601.03233
- [14]Ke He, Yichen Ding, and Bin Yang. OneWorld: Learning consistent physics across actions in world models. arXiv preprint arXiv:2609.30946, 2026. 3arxiv.org/abs/2609.30946
- [15]Lanxiang Hu, Abhilash Shankarampeta, Yixin Huang, Zilin Dai, Haoyang Yu, Yujie Zhao, Haoqiang Kang, Daniel Zhao, Tajana Rosing, and Hao Zhang. Benchmarking scientific understanding and reasoning for video generation using VideoScience-Bench. In European Conference on Computer Vision, pages 151–168, 2026. 3arxiv.org/abs/2512.02942
- [16]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Tian, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024. 2arxiv.org/abs/2311.17982
- [17]Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, et al. VBench++: Comprehensive and versatile benchmark suite for video generative models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3):3268–3285, 2026. 2DOI
- [18]Kling AI. Kling Video 3.0, 2026. 2, 5
- [19]Jialiang Li, Zhengning Xu, Zongyang Ma, and Shaobo Cui. A plausible future is not a causal future: Can video models generate how the world unfolds? arXiv preprint arXiv:2610.15562, 2026. 2, 3arxiv.org/abs/2610.15562
- [20]Yuqi Li, Junming Liu, Bowen Hu, Ruize Niu, Huiran Duan, Ming-Hsuan Yang, Yingli Tian, and Hao Wu. GeoFlip preview: Benchmarking geometry-sensitive physics in video generation. arXiv preprint arXiv:2610.42002, 2026. 2, 3arxiv.org/abs/2610.42002
- [21]Yanxun Li, Hao Wen, Bingze Song, Jiashu Zhu, Aiming Hao, Chubin Chen, Jintao Chen, Jiahong Wu, Xiangxiang Chu, and Miao Wang. Learning explicit physical parameter control and benchmarking for video generation. arXiv preprint arXiv:2607.18924, 2026. 3arxiv.org/abs/2607.18924
- [22]Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai, Dong Li, Yining Jiang, et al. BEAKER: An expert-curated benchmark for embodied brains in self-driving chemical laboratories. In Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track, 2026. Accepted for publication. 3arxiv.org/abs/2611.06911
- [23]Mingxin Liu, Shuran Ma, Shibei Meng, Xiangyu Zhao, Zicheng Zhang, Shaofeng Zhang, Zhihang Zhong, Peixian Chen, Haoyu Cao, Xing Sun, et al. RISE-Video: Can video generators decode implicit world rules? In Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track, 2026. Accepted for publication. 3arxiv.org/abs/2602.05986
- [24]Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. EvalCrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22139–22149, 2024. 2arxiv.org/abs/2310.11440
- [25]Fanqing Meng, Jiaqi Liao, Xinyu Tan, Quanfeng Lu, Wenqi Shao, Kaipeng Zhang, Yu Cheng, Dianqi Li, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation. In Proceedings of the 42nd International Conference on Machine Learning, pages 43781–43806. PMLR, 2025. 2arxiv.org/abs/2410.05363
- [26]MiniMax. MiniMax H3, 2026. 2, 5
- [27]Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jain, and Robert Geirhos. Do generative video models understand physical principles? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 948–958, 2026. 2arxiv.org/abs/2501.09038
- [28]NVIDIA. Cosmos 3: Omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800, 2026. 2, 5
- [29]OpenAI. GPT-5.6 system card, 2026. 2, 4
- [30]OpenAI. GPT-6 Astra system card, 2026. 4
- [31]Qwen Team. Qwen-Image-2.1, 2026. 4
- [32]Tim Rä dsch, Yüki M Asano, Hilde Kuehne, Stefan Bauer, Priyank Jaini, Robert Geirhos, and Carsten T. Lüth. Physics-IQ verified. arXiv preprint arXiv:2606.18943, 2026. 2arxiv.org/abs/2606.18943
- [33]Enes Sanli, Baris Sarper Tezcan, Erkut Erdem, and Aykut Erdem. PhysVidBench: Language-grounded evaluation of physical commonsense in text-to-video models. In Findings of EMNLP, pages 45311–45359, 2026. 2arxiv.org/abs/2507.15824
- [34]ShengShu Technology. Vidu Q3, 2026. 2, 5
- [35]Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2V-CompBench: A comprehensive benchmark for compositional text-to-video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8406–8416, 2025. 2arxiv.org/abs/2407.14505
- [36]Antonios Tragoudaras, Chenyu Zhang, Daniil Cherniavskii, Antonios Vozikis, Thijmen Nijdam, Derck W. E. Prinzhorn, Mark Bodracska, Nicu Sebe, Andrii Zadaianchuk, and Stratis Gavves. Evaluating Newtonian mechanics in video generative models with real physical systems. In Proceedings of the 43rd International Conference on Machine Learning, pages 121912–121951. PMLR, 2026. 2arxiv.org/abs/2504.02918
- [37]Rishi Upadhyay, Howard Zhang, Jim Solomon, Ayush Agrawal, Yunhao Ba, Alex Wong, Celso M de Melo, and Achuta Kadambi. WorldBench: Benchmarking physical understanding of world models by isolating physics concepts. arXiv preprint arXiv:2601.21282, 2026. 2arxiv.org/abs/2601.21282
- [38]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. 2, 5arxiv.org/abs/2503.20314
- [39]Yicheng Xu, Yue Wu, Jiashuo Yu, Xiang Yan, Tianxiang Jiang, Jinan He, Qingsong Zhao, Kai Chen, Yu Qiao, Limin Wang, et al. ExpVid: A benchmark for experiment video understanding & reasoning. In International Conference on Learning Representations, pages 83310–83348, 2026. 3arxiv.org/abs/2510.11606
- [40]Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. ChronoMagic-Bench: A benchmark for metamorphic evaluation of text-to-time-lapse video generation. In Advances in Neural Information Processing Systems, pages 21236–21270. Curran Associates, Inc., 2024. 2arxiv.org/abs/2406.18522
- [41]Diandian Zhang, Tingyu Song, Lin Fu, Zhe yuan Yang, and Yilun Zhao. Sci-VBench: Evaluating knowledge- and reasoning-intensive video generation in science domains. In Third Annual Conference on Language Modeling (COLM), 2026. 3arxiv.org/abs/2608.09873
- [42]Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, et al. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. International Journal of Computer Vision, 2026. Accepted for publication. 2arxiv.org/abs/2503.21755



