WorksComputer Science
AgentDreamer: Agentic 3D Motion Reasoning for Text-Controllable Physics Simulation
Abstract
We propose AgentDreamer, an agent-guided optimization framework that aligns text instructions with simulated motion through explicit 3D motion planning. Given a text instruction and scene geometry, an off-the-shelf agent translates the instruction into a structured semantic motion prior, specifying what should move, in which direction, and in what temporal order. The agent represents this prior using parametric functions over a series of temporal chunks, yielding dense 3D trajectories. We then optimize the parameters of a differentiable MPM simulator such that its rollout best matches the prior while remaining constrained by the scene dynamics. This allows the agent to provide semantic guidance from language, while the simulator grounds it in physically feasible motion. We further introduce Motion-IG, an information-theoretic metric for text-motion alignment based on masked text recovery. By comparing the likelihood of recovering masked motion-related content with and without the simulated motion, Motion-IG measures the information contributed by the motion beyond that available from the remaining text and scene context. Experiments show that AgentDreamer improves text alignment over video-based guidance, producing simulations that more closely follow the requested physical behavior.
Introduction
Controlling physical motion in simulated scenes is important for creating dynamic environments for embodied AI and interactive content generation. Without an intuitive control interface, users must manually tune simulation parameters through repeated simulation and adjustment to obtain the desired motion. Text-controllable physics simulation offers a promising alternative by allowing users to directly describe the intended behavior in natural language. Recent approaches, including [17] and [26], leverage video generative priors to guide the optimization of simulation parameters. In this way, they connect high-level user specifications to the dynamics of a 3D scene.
Despite this progress, the resulting simulation may not reproduce the physical behavior described by the text. A central difficulty lies in the motion reference itself: visually plausible video generation does not ensure that the requested dynamics are represented correctly. For example, a request for a rolling ball may produce sliding, omitting the rotation that distinguishes the two behaviors. Optimizing a simulator to follow such a reference can therefore yield physically plausible motion that remains inconsistent with the text. This discrepancy motivates explicit 3D motion planning that describes how the object should move to reproduce the requested behavior.
We propose AgentDreamer, a framework for text-controllable physics simulation that bridges text and simulated motion through agent-based 3D motion planning. Our key idea is to formulate motion planning as a physical reasoning problem. Given a text instruction and scene geometry, an off-the-shelf multimodal agent draws on its language understanding and physical knowledge to infer how the object should move, as shown in Figure 1. The agent thus takes a direct role in constructing the motion plan, extending its use beyond estimating material parameters [17, 26]. This plan provides explicit 3D guidance for translating the text into simulated motion.
Figure 1. 3D motion planning as physical reasoning. Given a text instruction and scene geometry, the agent reasons about the physical behavior required by the instruction and constructs a corresponding parametric motion plan. The plan produces dense 3D trajectories that guide physics optimization toward the requested motion.
To support motion reasoning, we introduce automatic semantic injection that associates sampled points with visible object parts. We provide a scene image with colored point markers corresponding to the 3D point input, together with coordinate-system information. This correspondence helps the agent interpret the role of each point in the requested motion. The agent then divides the time horizon into temporal chunks and infers a parametric motion function for each. These functions describe distinct motion stages and provide dense trajectories without predicting each future position independently. The resulting trajectories serve as pseudo supervision for a differentiable Material Point Method (MPM) simulator. We optimize simulation parameters to match these trajectories, guiding the simulator toward physically plausible motion aligned with the text.
We further introduce Motion-IG, an information-gain-based metric for measuring text-motion alignment through masked text recovery. An MLLM-as-a-judge score [35, 4], read as the model’s verbalized confidence, considers the complete text and simulated motion together, without isolating the information provided by the motion. We instead mask motion-related content in the text and compare its recovery likelihoods with and without simulated motion under a fixed evaluation model. The difference measures how much the motion contributes to recovering the masked content beyond the remaining text.
Experiments demonstrate that AgentDreamer improves text alignment over video-based guidance, producing simulations that more closely follow the requested physical behavior. Our contributions are summarized as follows:
We propose AgentDreamer, a framework for text-controllable physics simulation that bridges text and simulated motion through agent-based 3D motion planning.
We formulate motion planning as physical reasoning through parametric functions over temporal chunks, providing dense trajectories as pseudo supervision for differentiable physics optimization.
We introduce Motion-IG, an information-gain-based metric for text-motion alignment that compares masked text recovery likelihoods with and without simulated motion.
Related Works
Dynamic 3D representations Recent methods extend 3D Gaussian Splatting (3DGS) [9] to reconstruct dynamic scenes from video sequences. These methods [18, 28, 33, 13, 8, 6, 10] represent temporal changes through persistent Gaussian trajectories, deformation fields, or space-time primitives. The particle-based structure of Gaussian primitives further enables these representations to be extended to physics-based dynamics. PhysGaussian [31] couples Gaussian primitives with Material Point Method (MPM) simulation, using a shared representation for physical dynamics and rendering. Subsequent works develop elastic deformation models [36], estimate material properties [34, 2], and support diverse material behaviors and interactions [14, 7].
Text-conditional physics simulation Text provides an intuitive interface for specifying object dynamics and interactions. Recent approaches [26, 17, 16, 14] translate these instructions into simulation guidance through video diffusion priors. PhysFlow [17] uses optical-flow guidance to optimize physical parameters, while MotionPhysics [26] employs learnable motion distillation to guide simulation toward text-specified dynamics. These approaches primarily derive motion guidance in 2D space from generated videos. In contrast, our method constructs an explicit 3D motion prior through physical reasoning with an off-the-shelf multimodal agent, directly specifying the desired scene dynamics. Prior works have also used multimodal models to construct explicit motion priors, including bounding-box plans in VLIPP [32] and 2D point trajectories in C-Drag [12]. However, these representations remain sparse and confined to image space. Our approach instead produces dense 3D trajectories over the scene geometry, which serve as pseudo supervision for differentiable physics simulation.
Agent-guided physics-aware motion evaluation Vision-language models provide semantic and physical feedback for evaluating and improving generated motion [15, 1, 19]. PhyGDPO [3] uses VLM-based scores of semantic adherence and physical commonsense to guide groupwise preference optimization. PhyCo [20] evaluates generated videos through targeted physical questions and converts response logits into differentiable rewards for refining video generation. These approaches assess motion through semantic or physical judgments [35, 4], but do not explicitly separate the evidence supplied by motion from information available in the textual context alone. We instead formulate text-motion alignment as conditional information gain (Wyner, 1978), measuring how much a simulated trajectory improves the recovery of masked motion semantics over the remaining text.
Method
We present AgentDreamer, a framework for text-controllable physics simulation through agent-generated 3D motion planning, as shown in Figure 2. The agent translates the requested behavior into parametric motion functions, whose trajectories provide pseudo supervision for physics optimization. We further introduce Motion-IG to measure text–motion alignment through masked text recovery.

Figure 2. Overview of AgentDreamer. Given a text instruction and scene geometry, an off-the-shelf agent constructs a structured semantic motion prior using parametric functions over temporal chunks. The resulting dense 3D trajectories guide optimization of a differentiable MPM simulator under the scene dynamics.
Agent-Guided Physics Simulation
Given a text instruction and initial scene geometry , we construct a 3D motion plan and optimize simulation parameters to follow its trajectories.
Agent-Based Motion Planning
Automatic semantic injection Point coordinates alone provide limited semantic information about the object parts relevant to the instruction . Let be the initial position of MPM particle , and let be the set of particle indices selected for guidance. The set can contain any selected MPM particles; in our experiments, it consists of sampled particles. For each initial view , we project the selected points with camera projection and mark their indices as follows:
Let denote the annotated views, where is the number of views. Each marker displays the particle index consistently across views, allowing the agent to associate its 3D position with a visible object part. We also specify the coordinate origin, axis directions, and spatial units. Examples of annotated inputs are given in the Appendix.
Parametric motion planning Direct prediction of future point positions can distort the point configuration over time, as shown in Figure 3. We instead ask the agent to infer motion functions and their parameters. Since a single function may not capture distinct motion stages, the agent divides the time horizon into temporal chunks and specifies a function for each. The agent groups points with similar motion and divides the time horizon into chunks with boundaries . Given the instruction, selected point coordinates , and annotated views , the agent produces the plan as follows:

Figure 3. Effect of parametric motion planning. Direct prediction of future point positions produces irregular trajectories and distorts the object structure (top). Parametric functions yield coherent trajectories that better preserve spatial relationships between points (bottom). Colors identify corresponding points across time.
For chunk , combines a center trajectory with the local motion of each point group. The function gives the target 3D position for point from its position at the chunk start after elapsed time . Starting from , the target trajectory is given by
The condition ensures continuity between chunks. Evaluating these functions at the desired times yields dense 3D trajectories.
MPM Optimization
MPM simulation The planned trajectories provide pseudo supervision for a differentiable Material Point Method (MPM) simulator. We optimize the simulation parameters , including object-level material parameters and per-particle initial velocities, to reproduce the planned motion under the scene dynamics. Let denote the MPM state after simulation step , including particle positions, velocities, and deformation-related quantities, and let be the step size. The state evolves as
Let be the simulated position of particle at output time . The particle index establishes correspondence with the target position .
Motion reconstruction loss We match the simulated and target positions of the selected particles at output times :
Here . Although the loss is evaluated only for particles in , MPM evolves the full scene. The motion functions provide target positions for the selected particles at each output time. We keep the plan fixed and update using gradients through the MPM rollout.
Motion-IG
We introduce Motion-Conditioned Information Gain (Motion-IG) to measure text–motion alignment through masked text recovery. It measures how much observing the simulated motion improves recovery of motion-related content beyond the remaining text.
Notation Let be the instruction word sequence and the set of masked positions, where . Replacing these words with [MASK] yields the masked sequence ; the removed words form the ordered subsequence . We mask motion words and attributes while retaining object and part names (Table 1). Let denote the simulated motion, represented by rendered frames and serialized 3D trajectories. The masked text and annotated initial views remain identical in both conditions. For brevity, we omit the shared conditioning on in the equations below.
| Sequence | Instruction |
| The soft fox sways from side to side while remaining attached at the rear. | |
| The [MASK] fox [MASK] from [MASK] to side while [MASK] [MASK] at the rear. | |
| The basketball rolls forward in a straight line, rotating naturally without slipping. | |
| The basketball [MASK] [MASK] in a [MASK] [MASK], [MASK] [MASK] [MASK] [MASK]. | |
| The rigid playdoh ball falls straight down and bounces several times without deforming. | |
| The [MASK] playdoh ball [MASK] [MASK] [MASK] and [MASK] [MASK] [MASK] without [MASK]. |
Table 1. Examples of word-level masking for Motion-IG. Bold words in the original instruction are replaced by [MASK] in the masked instruction .
Information gain The masked content may be partially inferred from the remaining text and annotated views. We therefore measure the additional information provided by the motion through conditional mutual information as follows:
Since these probabilities are unknown, we use a fixed vision-language model to compute a pointwise score as follows:
The denominator gives the recovery likelihood without the simulated motion. A positive score indicates that observing the motion increases the likelihood of the removed words; a negative score indicates that it decreases this likelihood.
Score computation We provide the masked text as the query and teacher-force the original instruction as the response, with and without simulated motion. Let be the number of evaluated text–motion pairs. For pair , let denote the response tokens. The token log-probability gain is given by
The preceding response tokens are identical in both conditions. Let contain the masked positions selected for scoring, excluding words such as “is” and “the” but retaining negations such as “not”. For each selected word at position , we score its first subword token at position . Averaging first over words and then over instructions gives Motion-IG as follows:
This first-subword score is a proxy for Eq. (7). We compute Motion-IG for each method using its simulated motions while keeping the instructions, masks, annotated views, and evaluation models identical across methods.
Experiments
We evaluate whether agent-generated 3D motion plans improve text-controllable physics simulation across diverse object motions. We use GPT-5.6 Sol as the off-the-shelf agent for 3D motion planning throughout our experiments. We compare the simulations with video-guided methods using Motion-IG and blinded human rankings of text–motion alignment.
Experimental Setup
Scenes and instructions. We conduct experiments on eight scenes with 20 text instructions. The dataset comprises four real-world scenes used by PhysFlow [17] (Alocasia, Carnation, Telephone and Fox) and four synthetic scenes, one from PAC-NeRF [11] (Playdoh) and three from GaussianVerse [30] (Basketball, Soccer Ball and Can). The instructions cover falling and bouncing, rolling, slowing rotation, oscillation, and bending or deformation, with 10 applied to the real-world scenes and 10 to the synthetic scenes.
Baselines and evaluation. We compare AgentDreamer with PhysFlow [17] and MotionPhysics [26], which use text-conditioned video diffusion to guide physics simulation. We use four open-weight evaluators: InternVL3.5-8B [27], Qwen3-VL8B [23], Qwen3.5-9B [25], and Gemma-4-12B [25]. We additionally report results with five API-based evaluators: GPT-5.6 Sol and GPT-5.6 Terra [21], GPT-6 Astra [22], Gemini 3.1 Pro [5], and Gemini 3.8 Flash [5]. To compute Motion-IG, we use teacher-forced token likelihoods for open-weight evaluators and directly requested probabilities for API-based evaluators. For human evaluation, participants rank three anonymized simulations for each instruction by how well the motion follows the text.
Comparison with Existing Methods
Qualitative Results
Figure 4 compares the three methods on a bouncing playdoh ball and a fox instructed to rock while its rear remains attached. For the ball, PhysFlow and MotionPhysics deform on impact but remain near the ground, whereas AgentDreamer rebounds after compression as requested. For the fox, the shown frames of AgentDreamer display the requested rocking motion while the rear remains attached; the baseline sequences show less of this motion. Both examples depend on the motion sequence specified in the instruction, which the agent represents through a 3D trajectory prior before simulation.

Figure 4. Qualitative comparison of text-controllable physics simulation. AgentDreamer more closely follows the requested motion than PhysFlow and MotionPhysics in the shown examples. The shown examples contrast the requested motion with the behaviors produced by video-guided optimization and explicit 3D planning.
Quantitative Results
Motion-IG with open-weight evaluators Table 2a compares text–motion alignment using four open-weight evaluators. On the 20 evaluated instructions, AgentDreamer achieves the highest score under all four evaluators. Its positive scores indicate that providing the simulated motion increases the likelihood of the correct masked words, on average, compared with the condition without motion. These results suggest that motion generated by AgentDreamer provides more useful information for recovering the requested behavior than the compared methods.
| Method | Gemma-4 | Qwen3.5 | Qwen3-VL | InternVL3.5 | Average | Best in | |
| PhysFlow ([17]) | 0.36 | 0.62 | 0.58 | 0.52 | 0.52 | 15/80 | |
| MotionPhysics ([26]) | 0.13 | 0.58 | 0.81 | 0.56 | 0.52 | 22/80 | |
| AgentDreamer (ours) | 0.63 | 0.67 | 0.92 | 0.64 | 0.71 | 43/80 | |
| Method | GPT 6 Astra | GPT 5.6 Sol | GPT 5.6 Terra | Gemini 3.8 Flash | Gemini 3.1 Pro | Average | Best in |
| PhysFlow ([17]) | −0.13 | −0.45 | +0.04 | −0.69 | +0.14 | −0.22 | 23/100 |
| MotionPhysics ([26]) | −0.49 | −0.96 | −0.44 | −0.93 | −0.32 | −0.63 | 15/100 |
| AgentDreamer (ours) | +0.56 | +0.25 | +0.48 | +0.12 | +1.00 | +0.48 | 62/100 |
Table 2. Comparison of text–motion alignment using Motion-IG with (a) token likelihoods from open-weight evaluators and (b) verbalized probabilities from black-box API evaluators (higher is better). Best in counts the evaluator–instruction pairs in which each method ranks first.
Motion-IG with API-based evaluators tbl:2b reports results using verbalized probabilities from five API-based evaluators. The lower average scores may partly reflect stronger recovery from the masked text and annotated views alone: if an evaluator already predicts the masked words with high probability, observing motion provides less additional gain. Nevertheless, AgentDreamer achieves the highest score under all five evaluators and ranks first in 62/100 evaluator–instruction pairs. This consistent ranking suggests that its simulated motions provide more useful evidence for recovering the requested behavior across both evaluation approaches. We compare rankings within each evaluator group because verbalized probabilities and teacher-forced token likelihoods have different scales.
Human evaluation We conduct a human evaluation of text–motion alignment with 25 participants. For each instruction, participants view the results of all three methods and rank them by how well they match the requested behavior. AgentDreamer is ranked first most often (73.6%), compared with PhysFlow (14.0%) and MotionPhysics (12.4%). Figure 5 further shows that AgentDreamer is preferred over each baseline in pairwise comparisons. This agrees with the Motion-IG ranking, providing further evidence that AgentDreamer follows the text instructions more closely than the baselines.
Figure 5. Pairwise human preference for text–motion alignment. Win rate is the percentage of trials in which one method is ranked above the other. AgentDreamer is preferred over PhysFlow in 82.6% of trials and over MotionPhysics in 85.6% of trials.
Analysis
Correspondence to classical mechanics. We examine whether the agent-generated functions follow motion equations derived from classical mechanics. Table 3 reports how often the plans match the expected function family. The generated forms resemble the classical relations for rolling, falling, spin decay, and damped oscillation, including coupled translation and rotation, quadratic trajectories, and damped sinusoids. For deformation, the agent uses smoothstep interpolation to describe the temporal change rather than a classical mechanics equation. These similarities suggest that the agent draws on familiar physical motion patterns when constructing its 3D prior.
| Motion cue | Function family | Representative expression | Matched / Total |
| Roll | Coupled translation and rotation | 2/2 | |
| Spin, slow down, stop | Quadratic | 2/2 | |
| Fall, bounce, jump | Quadratic | 4/4 | |
| Sway, oscillate, rock | Sinusoid | 8/8 | |
| Pull, bend, squash, deform | Smoothstep interpolation | 4/4 |
Table 3. Correspondence between agent-generated motion functions and expected forms across 20 instructions. The expected forms follow classical mechanics for rolling, spin decay, falling, and oscillation, while deformation uses smoothstep interpolation. Expressions show representative components, and Matched / Total counts plans that use the expected form in each category. : elapsed time within a chunk; : normalized time. : travel distance; : rotation angle; : ball radius.
Benefits of 3D motion planning across motion categories. To examine the benefit of 3D planning over 2D video-based guidance, we compare Motion-IG across five motion categories in Figure 6. AgentDreamer achieves the highest score in every category, with the largest improvements for rolling, spin decay, and falling and bouncing. Notably, both baselines obtain negative scores for rolling, indicating that their simulated motions make the masked behavior harder to recover than from the shared context without motion. Rolling and sliding can appear similar in video, although rolling requires coupled translation and rotation in 3D. The larger gain for rolling suggests that explicit 3D planning is especially useful when the requested motion is difficult to distinguish from video appearance alone. Gains are smaller for oscillation and deformation, where the baselines already achieve relatively high scores.
Figure 6. Text–motion alignment across motion categories measured by Motion-IG (higher is better). AgentDreamer achieves the highest score in every category; for rolling, both baselines have negative information gain.
Consistency with human evaluation. We examine whether Motion-IG reflects human preferences by comparing how often AgentDreamer ranks first among the evaluated methods in each motion category. As shown in Figure 7, both human evaluation and Motion-IG select our method most frequently for rolling and spinning. The first-rank rates decrease for oscillation and deformation, where the baselines are selected more often. Although the absolute rates differ, this shared trend suggests that Motion-IG captures category-dependent differences in text–motion alignment observed by human evaluators. These category-level trends do not measure agreement on individual instruction rankings.
Figure 7. Consistency of Motion-IG with human evaluation across motion categories. Both show higher first-rank rates for rolling and spinning and lower rates for oscillation and deformation.
Temporal visualization of motion planning Figure 8 shows how the agent divides the requested motion into temporal chunks and constructs a parametric function for each. The planned sequence captures the ball motion, strong compression on impact, and subsequent partial shape recovery. The chunks correspond to these behavioral changes, suggesting that the agent identifies distinct motion stages within the instruction. By describing each stage separately, the plan captures changes in physical motion that are difficult to represent with a single parametric function.

Figure 8. Temporal decomposition of an agent-generated motion plan. The agent uses separate parametric functions to describe the ball movement, compression on impact, and partial shape recovery. Dashed lines mark temporal chunks, and colored points show target positions over time.
Effect of planning components Table 4 shows that both components improve text–motion alignment, with parametric planning providing the larger gain. The small gain from semantic injection alone suggests that point annotations are most useful with parametric planning, where they help associate selected 3D points with visible object parts. Without parametric planning, independently predicted point positions lose coherence over time (Figure 3). Motion functions instead provide coherent, dense target trajectories for physics optimization.
| Semantic injection | Parametric planning | Motion-IG ↑ |
| 0.37 | ||
| ✓ | 0.38 | |
| ✓ | 0.63 | |
| ✓ | ✓ | 0.72 |
Table 4. Effect of planning components including semantic injection and parametric motion predictions. Parametric planning provides the largest individual gain, while combining both components achieves the highest score.
Comparison with MLLM judge Figure 9 compares Motion-IG with MLLM judgments ([35, 4]) on the same 60 text–motion pairs after scaling each score type to its own range.1 The direct judge scores the instruction and motion together, whereas Motion-IG measures the additional evidence that the motion provides for recovering masked motion words. Although asked for a real-valued score, the MLLM judge assigns only 11 distinct values to the 60 pairs, giving the same score to multiple motions. In contrast, Motion-IG assigns a distinct score to every pair in this comparison. The reduced number of ties gives finer resolution when ranking these pairs; Figure 7 separately compares category-level first-rank rates with human evaluation.
Figure 9. Score distributions on identical text–motion pairs, scored by gemma-4. Motion-IG gives finer score resolution than an MLLM judge on the same text–motion pairs.
Conclusion
We presented AgentDreamer, a framework that uses an agent to generate 3D motion priors for text-controllable physics simulation. Given a text instruction and scene geometry, the agent reasons about the underlying physics and constructs parametric functions to describe motion consistent with the instruction. The resulting trajectories guide differentiable physics optimization to reproduce the requested motion. We have also introduced Motion-IG to measure how much simulated motion helps recover masked motion words beyond the remaining text. Experiments show improved text alignment over video-guided baselines, demonstrating the effectiveness of agentic motion priors.
AI Use Statement
In this work, we used generative AI tools to help develop theoretical models or conceptual frameworks, design or provide feedback on research methodology or experiments, implement methods, and assist with translation. We have not used generative AI tools to generate synthetic datasets, formulate mathematical claims, clean or reformat datasets, or interpret results, and tasks related to providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, proposing or refining hypotheses, and supporting qualitative or thematic data analysis are not applicable to this work.
Additionally, we used generative AI tools to create or modify scientific figures or images, suggest experimental parameters, create or edit software code and research artifacts, draft parts of the manuscript, summarize or analyze existing literature, discover research topics or identify gaps, brainstorm research ideas, search for information, improve manuscript readability, identify relevant literature, format references, and suggest manuscript structure.
We have reviewed all AI-assisted work. AI-assisted code was verified through execution and testing, and all methodological suggestions and translations were manually reviewed by the authors before use. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.
Ethics Statement
This work uses existing 3D scenes to study text-controlled physical simulation. The human evaluation involved 25 participants who ranked simulated motions by their alignment with text instructions. Method identities were hidden as A, B, and C, their presentation positions were balanced across instructions, and results are reported in aggregate. Because the motion plans and automated evaluations rely on pretrained multimodal models, they may reflect model errors or biases. We therefore report human judgments alongside model-based scores and do not treat the latter as a sole measure of physical correctness.
Reproducibility Statement
§ 3.1 and § 3.2 describe the motion-planning and physics-optimization procedure and define the Motion-IG metric. § 4.1 identifies the scene sources, text instructions, baselines, and evaluation models. The appendix provides additional results (Appendices § 9 and § 10), examples of semantic injection (Appendix § 11), the prompts used for motion planning, code generation, and evaluation (Appendix § 12), and the human evaluation procedure (Appendix § 13).
Additional Qualitative Results
Figure 10 and Figure 11 present additional comparisons of PhysFlow, MotionPhysics and AgentDreamer under the same text instructions. In the shown examples, AgentDreamer produces motion that more closely follows the instructions than the baselines. These results complement the main-paper comparisons and further support the use of agent-generated 3D motion priors for text-controllable physics simulation.

Figure 10. Additional qualitative comparisons with PhysFlow [17] and MotionPhysics [26] under the same text instructions. The left panels show frames from the simulated videos, while the right panels show the corresponding sampled-point trajectories. In the shown examples, AgentDreamer produces simulated motion that more closely follows the requested behavior.

Figure 11. Additional qualitative comparisons with PhysFlow [17] and MotionPhysics [26] under the same text instructions. The left panels show frames from the simulated videos, while the right panels show the corresponding sampled-point trajectories.
Additional Quantitative Results
Text–motion alignment across motion categories Table 5 breaks the Motion-IG scores down by the motion categories of the main paper. AgentDreamer ranks first in every category under both open-weight and black-box evaluators. The gap is clear for simple rigid-body motions governed by a single physical law, namely rolling, falling and bouncing, and spin decay: on rolling, AgentDreamer reaches +0.46 with open-weight evaluators while both baselines stay negative (−0.22 and −0.24). This is where the benefit of planning shows most, since the plan can state the motion by its governing law. On more complex motions that repeat or change shape, oscillation and bending and deformation, AgentDreamer still scores highest but by a narrower margin; with black-box evaluators it reaches +0.35 on oscillation, close to PhysFlow at +0.33.
| Category | Open-weight (avg.) | Black-box (avg.) | |||||
| PhysFlow | MotionPhysics | Ours | PhysFlow | MotionPhysics | Ours | ||
| Fall and bounce | 4 | +0.18 | +0.36 | +0.63 | −1.46 | −0.87 | +0.77 |
| Roll | 3 | −0.22 | −0.24 | +0.46 | −1.23 | −2.01 | +0.92 |
| Spin decay | 2 | +0.29 | +0.20 | +0.55 | −0.15 | −0.23 | +0.62 |
| Oscillation | 9 | +0.77 | +0.75 | +0.83 | +0.33 | −0.50 | +0.35 |
| Bend and deform | 5 | +0.67 | +0.70 | +0.74 | +0.34 | −0.12 | +0.39 |
Table 5. Text–motion alignment per motion category measured by Motion-IG (higher is better), averaged over four open-weight and five black-box evaluators.
Human evaluation details We collect rankings from 25 participants on 20 instructions, for 500 trials in total. In each trial, participants read the instruction and watch the results of the three methods, stacked vertically and labeled A, B, and C without method names. The position of each method is balanced across instructions. Participants rank the three results from first to third by how well the motion matches the instruction. Table 6 reports how often each method receives each rank. AgentDreamer is ranked first in 73.6% of trials and last in only 5.4%, whereas PhysFlow and MotionPhysics are ranked last in 40.2% and 54.4% of trials, respectively.
| Method | 1st (%) ↑ | 2nd (%) | 3rd (%) ↓ |
| PhysFlow | 14.0 | 45.8 | 40.2 |
| MotionPhysics | 12.4 | 33.2 | 54.4 |
| Ours | 73.6 | 21.0 | 5.4 |
Table 6. Rank distribution in the human evaluation (25 participants 20 instructions = 500 trials). Each entry is the percentage of trials in which the method receives the rank.
Semantic Injection Examples
Figure 12 presents examples of annotated images for two objects. The numerical identifiers show where the sampled points appear in each initial scene image, helping the agent associate the points with visible object parts relevant to the text instruction.

Figure 12. Examples of annotated images used for automatic semantic injection. Control points sampled from the initial scene geometry are marked with numerical identifiers in two views of a fox (top) and a soccer ball (bottom).
Details of Agent Prompts
Prompts for Motion planning
Figure 13 presents the prompt used to construct the motion plan. The agent groups the sampled points, divides the time horizon into temporal chunks, and specifies a center function and a local function for each group in each chunk. Figure 14 presents the prompt used to translate the resulting motion plan into deterministic Python code. The code computes target point positions from the specified functions without modifying the motion plan.
Figure 13. Prompt for agent-based motion planning. The agent divides the time horizon into temporal chunks and specifies parametric motion functions for groups of sampled points.
Create an explicit grouped parametric 3D motion plan from the supplied object data.
Inputs:
Clean multi-view images: {{IMAGES}}
Matching point-ID overlay images: {{OVERLAY_IMAGES}}
Point and camera data: {{POINTS_JSON}}
Motion instruction: {{MOTION_PROMPT}}
Maximum frame: {{MAX_FRAMES}}
Treat the clean images and overlay images as multimodal image inputs. Match each clean image to its overlay using the camera entries and filenames in points.json. Use them to understand the scene geometry, semantic parts, coordinate system, and requested motion.
Point-group rules:
Partition every supplied point ID into exactly one point group.
Do not omit or duplicate any point ID.
Keep the same point groups throughout the complete timeline.
Group points that share one coherent center-motion and local-motion function.
Prefer a small number of meaningful groups instead of one group per point.
Use the clean and indexed multi-view images to distinguish moving material from static support or background.
A static group must use motion_state static, function_type none, function none, empty parameter arrays, and composition p_i(tau)=p_i,start.
Timeline rules:
Interpret {{MAX_FRAMES}} as the final frame index; frame 0 is the input state.
Treat frame 0 through frame 40 as exactly 2.0 seconds of motion: 40 equal frame intervals and 41 trajectory states.
Use dt_seconds = 0.05 seconds per frame interval, so frame f corresponds to physical time t_seconds = f * dt_seconds.
Create a new segment whenever any point group's center-motion or local-motion function changes.
Adjacent segments must share a boundary and together cover frame 0 through MAX_FRAMES.
Set state_reference to initial_state for the first segment and previous_segment_end for every later segment.
Every segment must contain exactly one group_motion for every declared group.
Function rules:
Define center motion, local motion, and composition independently per group.
Use local frame offset tau = f - start_frame; tau is an integer frame offset, not a time in seconds.
For every time-dependent center or local function, declare dt_seconds = 0.05 as a numerical parameter and use the local physical time dt_seconds * tau explicitly in the function.
Express linear velocity in normalized-coordinate units per second, angular velocity in radians per second, and acceleration in normalized-coordinate units per second squared.
Let p_i,start be the group member's position at segment start.
The same group function is evaluated consistently with each member's p_i,start.
Every moving group function must be explicit, plain ASCII, single-line, and use only finite numerical parameters declared as name/value records.
A moving center function returns absolute center c(tau)=c_start+vtaudt_seconds.
A moving local function returns relative offset r_i(tau)=R(tau)r_i,start.
Every group function at tau=0 must reproduce p_i,start exactly.
Preserve continuity between adjacent segments.
Do not output frame-by-frame coordinates.
Return valid JSON only, with no Markdown or commentary, in this shape:
{ "point_groups": [ { "group_id": "stable_group_id", "semantic_part": "Brief semantic description.", "point_ids": ["p000", "p001"] } ], "segments": [ { "start_frame": 0, "end_frame": 40, "state_reference": "initial_state", "group_motions": [ { "group_id": "stable_group_id", "motion_state": "moving", "motion_description": "Brief motion explanation.", "center_motion": { "function_type": "smoothstep", "function": "c(tau)=...", "parameters": [{"name": "parameter_name", "value": 0.0}] }, "local_motion": { "function_type": "smoothstep", "function": "q_i(tau)=...", "parameters": [{"name": "parameter_name", "value": 0.0}] }, "composition": "p_i(tau)=c(tau)+q_i(tau)" } ] } ] }
Figure 14.
Figure 14 (PDF p. 18). Prompt for translating the motion plan into deterministic Python code. The code implements the specified motion functions to compute target point positions.
Prompts for Evaluation
Figure 15 presents the prompt given to the open-weight evaluators. The evaluator receives the annotated initial views, the generated motion as a trajectory and as rendered frames, and the description with its motion words masked; the masked words are scored by teacher forcing rather than by sampling the completion. Figure 16 presents the prompt given to the black-box evaluators. Since current API models expose no teacher-forced log-probabilities, the evaluator states a probability for each masked word, and these ratings replace the log-probabilities in the same gain.
Figure 15. Prompt for the open-weight evaluators. The masked words are scored by teacher forcing, and the same prompt without the motion gives the reference term.
Figure 16. Prompt for the black-box evaluators. Each masked word receives a stated probability, collected in one JSON response per query.
Details of User Study
We conducted a user study to assess how closely the simulated motions follow their text instructions. For each of 20 instructions, participants viewed three videos of the same object, corresponding to PhysFlow, MotionPhysics, and AgentDreamer, and ranked them from first to third according to their text–motion alignment. At the beginning of the survey, we present the following instructions:
In each of the 20 questions, you will see three short videos (A, B, C) of the same object and a text description below them. The videos loop automatically. For each question, please rank the three videos 1st, 2nd, and 3rd: 1st is the one whose motion best matches the text, and 3rd is the one that matches it the least. Each rank can be used only once.
Please judge only how well the motion follows the description: what moves, in which way, and how strongly. Do not judge image quality, lighting, or rendering artifacts. The key words in each description are highlighted in orange. Please watch all three videos at least once before answering.
Figure 17 shows an example question presented to participants in the user study.

Figure 17. Example question in the human evaluation of text–motion alignment. Given a text instruction and three simulation results for the same object, participants rank the motions by how closely they follow the requested behavior.
References
- [1]Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical common-sense for video generation. In ICLR, 2025.
- [2]Junhao Cai, Yuji Yang, Weihao Yuan, Yisheng He, Zilong Dong, Liefeng Bo, Hui Cheng, and Qifeng Chen. Gic: Gaussian-informed continuum for physical property identification and simulation. Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.14927
- [3]Yuanhao Cai, Kunpeng Li, Menglin Jia, Jialiang Wang, Junzhe Sun, Feng Liang, Weifeng Chen, Felix Juefei-Xu, Chu Wang, Ali Thabet, et al. Phygdpo: Physics-aware groupwise direct preference optimization for physically consistent text-to-video generation. In ECCV, 2026.
- [4]Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. MLLM-as-a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. In ICML, 2024.arxiv.org/abs/2402.04788
- [5]Google DeepMind. Gemini 3.1 Pro: A smarter model for your most complex tasks, 2026a. URL https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/. Google DeepMind. Gemini 3.8 Flash. 2026b. URL https://ai.google.dev/gemini-api/docs/latest-model.
- [6]Dadong Jiang, Zhi Hou, Zhihui Ke, Xianghui Yang, Xiaobo Zhou, and Tie Qiu. Timeformer: Capturing temporal relationships of deformable 3d gaussians for robust reconstruction. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8721–8732. IEEE, 2025a.arxiv.org/abs/2411.11941
- [7]Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. In ICCV, 2025b.arxiv.org/abs/2503.17973
- [8]Kai Katsumata, Duc Minh Vo, and Hideki Nakayama. A compact dynamic 3d gaussian representation for real-time dynamic view synthesis. In ECCV, 2024.arxiv.org/abs/2311.12897
- [9]Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, et al. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 2023.arxiv.org/abs/2308.04079
- [10]Junoh Lee, ChangYeon Won, Hyunjun Jung, Inhwan Bae, and Hae-Gon Jeon. Fully explicit dynamic gaussian splatting. NeurIPS, 2024.arxiv.org/abs/2410.15629
- [11]Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In ICLR, 2023.arxiv.org/abs/2303.05512
- [12]Yuhao Li, Mirana Claire Angel, Salman Khan, Yu Zhu, Jinqiu Sun, Yanning Zhang, and Fahad Shahbaz Khan. C-drag: chain-of-thought driven motion controller for video generation. arXiv preprint arXiv:2502.19868, 2025.arxiv.org/abs/2502.19868
- [13]Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In CVPR, 2024a.arxiv.org/abs/2312.03431
- [14]Yuchen Lin, Chenguo Lin, Jianjin Xu, and Yadong Mu. Omniphysgs: 3d constitutive gaussians for general physics-based dynamics generation. In ICLR, volume 2025, 2025.arxiv.org/abs/2501.18982
- [15]Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In ECCV, 2024b.arxiv.org/abs/2404.01291
- [16]Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprint arXiv:2406.04338, 2024.arxiv.org/abs/2406.04338
- [17]Zhuoman Liu, Weicai Ye, Yan Luximon, Pengfei Wan, and Di Zhang. Unleashing the potential of multi-modal foundation models and video diffusion for 4d dynamic physical scene simulation. In CVPR, 2025.
- [18]Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.arxiv.org/abs/2308.09713
- [19]Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quan feng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation. arXiv preprint arXiv:2410.05363, 2024.arxiv.org/abs/2410.05363
- [20]Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan, and Manmohan Chandraker. Phyco: Learning controllable physical priors for generative motion. In CVPR, 2026.
- [21]OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition, 2026a. URL https://openai.com/index/gpt-5-6/.
- [22]OpenAI. GPT-6 Astra: A new generation of intelligence, 2026b. URL https://openai.com/index/gpt-6-astra/.
- [23]Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.
- [25]Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Alqayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
- [26]Miaowei Wang, Jakub Zadrożny, Oisin Mac Aodha, and Amir Vaxman. Motionphysics: Learnable motion distillation for text-guided simulation. In AAAI, 2026.
- [27]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025.arxiv.org/abs/2508.18265
- [28]Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024.arxiv.org/abs/2310.08528
- [30]Tiange Xiang, Kai Li, Chengjiang Long, Christian Häne, Peihong Guo, Scott Delp, Ehsan Adeli, and Li Fei-Fei. Repurposing 2d diffusion models with gaussian atlas for 3d generation. In ICCV, 2025.arxiv.org/abs/2503.15877
- [31]Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Phys-gaussian: Physics-integrated 3d gaussians for generative dynamics. In CVPR, 2024.DOI
- [32]Xindi Yang, Baolu Li, Yiming Zhang, Zhenfei Yin, Lei Bai, Liqian Ma, Zhiyong Wang, Jianfei Cai, Tien-Tsin Wong, Huchuan Lu, et al. Vlipp: Towards physically plausible video generation with vision and language informed physical prior. In ICCV, 2025.arxiv.org/abs/2503.23368
- [33]Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR, 2024.arxiv.org/abs/2309.13101
- [34]Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. In ECCV, 2024.arxiv.org/abs/2404.13026
- [35]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NeurIPS Datasets and Benchmarks Track, 2023.arxiv.org/abs/2306.05685
- [36]Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In ECCV, 2024.arxiv.org/abs/2403.09434








