betaFeedback

WorksComputer Science

Coda-DaVinci: Reimagining Video Generation with Code2Video

Abstract

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Homepage

Code

VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Introduction

Recent video generation models can synthesize realistic and visually appealing videos from text and reference media [7, 11, 22, 24, 27]. Yet following detailed instructions, preserving scene state over time, and producing physically plausible interactions remain challenging. Code2Video approaches, including VideoCoCo [13] and Code Video Model [14], address these challenges through executable scene programs. In this paradigm, a coding agent constructs and renders a 3D proxy whose structure and motion guide a video model in synthesizing the final video. This explicit scene representation helps maintain scene state across camera revisits, as shown in Figure 2(a).

Selective 3D modeling

Figure 2. Selective 3D modeling. Each method pairs 3D proxy frames with generated video frames at 3, 5, and 7 seconds. Codex + Blender and VideoCoCo explicitly model hands and arms. Coda-DaVinci models the objects involved in cutting and unwrapping, while leaving detailed hand articulation to the video generator.

The benefits of Code2Video depend on the task and on how the proxy is constructed and used. Scene programs provide precise control over layout and trajectories, but detailed modeling can introduce geometric or motion errors that may carry over to the final video. When motion naturalness is central and explicit scene control offers limited benefit, an imperfect proxy may yield less natural results than direct generation. Proxies for the same task can differ substantially in their level of detail. Physical interactions also require choosing between scripted motion and physics simulation. The synthesis prompt must then align the intended appearance and finer actions with the rendered proxy. These considerations raise two central questions: How can we achieve the best video generation performance within the Code2Video paradigm, and how far can Code2Video extend the capabilities of video generation?

We study these questions through controlled comparisons with blinded human evaluation on tasks involving scene revisitation, complex motion, and physical processes. The benefits of proxy guidance vary by task: improved consistency and instruction following can come at the cost of motion naturalness. Under matched settings, coarse proxies preserve more natural motion than detailed alternatives on motion-intensive tasks, while medium-grained proxies provide stronger consistency when physics simulation is required. Prompts written by the coding agent outperform the tested external enhancement strategy in pooled scores. Among the tested configurations, medium-grained proxies combined with physics simulation and agent-authored prompts achieve the highest pooled performance.

Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation as shown in Figure 1. Coda-DaVinci selects 3D modeling when precise scene or motion control is needed and favors direct generation for natural body movement when explicit control is unnecessary and proxy animation risks stiffness. For modeled scenes, it uses medium-grained proxies for environments and structured objects, coarse proxies for human subjects, and Blender simulation for supported physical interactions, as shown in Figure 2(b). The agent inspects the proxy before writing a synthesis prompt that links appearance references to proxy entities and specifies finer human actions. A multi-agent harness preserves the scene workspace across revisions and successive requests, supports visual inspection, and uses an independent acceptance agent to identify unmet requirements for repair.

Figure 1. Comparison of video generation workflows. (a) Direct video generation. (b) Basic Code2Video workflows model the inputs as 3D scenes and render them into the final video. (c) Coda-DaVinci combines task-level routing with multi-granularity 3D modeling and generates prompts for subject binding.

Experiments show that Coda-DaVinci leads the compared methods in instruction following on VTR [9] and scene consistency on R2M [6]. It also improves physical plausibility over direct generation on World Models’ Last Exam in Physics [5] and outperforms Code2Video baselines in visual realism on the T2AV-Compass subset [2], approaching direct generation. We also examine agent–generator pairings and rendering engines; Blender offers higher quality than Three.js at greater token cost.

Long-video and multi-turn evaluations show gains in long-form structure and scene consistency over direct continuation, but lower transition stability and setting adherence.

Our main contributions are threefold:

  • We systematically analyze proxy granularity, physics simulation, and prompting, deriving task-dependent guidelines for proxy construction and video synthesis.

  • We introduce Coda-DaVinci, a task-adaptive agentic framework that coordinates selective 3D modeling and proxy-grounded synthesis, supported by a persistent scene workspace and visual feedback.

  • We empirically characterize Code2Video’s capabilities and limitations across tasks, agent–generator pairings, rendering engines, and long-video settings, revealing trade-offs in visual quality, consistency, and token cost.

Video Generation. Recent advances in video generation have substantially improved visual fidelity, motion quality, and text–video alignment [7, 10, 11, 17, 22, 23, 25, 26, 27, 30, 31]. Building on this progress, recent work increasingly emphasizes reliable instruction following, spatiotemporal consistency, and physical plausibility. TempAct [28] improves adherence to temporally complex instructions through planner–executor reinforcement learning. To maintain visual consistency over longer sequences, Context Forcing [3] combines long-context supervision with slow–fast memory, while LongLive-RAG [8] retrieves previously generated video latents to mitigate identity drift. MMPhysVideo [15] improves physical plausibility by jointly modeling RGB video with semantic, geometric, and trajectory cues. Despite these advances, reliably following complex instructions, maintaining spatiotemporal consistency, and generating physically plausible dynamics remain challenging.

Code2Video. Code2Video generation uses executable programs to control visual content and motion, enabling targeted editing. The Code2Video framework [4] targets educational videos, and MoVer [19] guides motion graphics refinement through spatiotemporal verification. StoryBlender [12] produces consistent 3D storyboards using persistent scene memory and engine-based verification, while SimWorlds [16] constructs dynamic Blender scenes through planning, coding, and verification. For proxy-guided synthesis, GPT4Motion [18] uses Blender-simulated motion cues to guide diffusion models. VideoCoCo [13] guides video generation with Blender-rendered spatiotemporal drafts, while Code Video Model [14] combines Three.js proxies with early-step conditioning to preserve structure and reduce synthetic-appearance leakage.

Empirical Analysis

Code2Video Pipeline Formulation

Given an input prompt xx with optional image or video references rr, a coding agent A\mathcal{A} constructs a proxy video v~\tilde{v}. A video-generation model G\mathcal{G} then produces the final video vv conditioned on the proxy and a generation prompt qq:

v~=A(x,r),v=G(v~,q,r),(1)\tilde{v} = \mathcal{A}(x, r), \qquad v = \mathcal{G}(\tilde{v}, q, r), \tag*{(1)}

where qq is either the original prompt xx or an enhanced version produced by the coding agent A\mathcal{A} or an external model.

To address the central questions raised in § 1, we examine how different proxy construction and prompting strategies affect video quality across tasks.

Evaluation Setup

Analyzed Components and Baselines. We analyze three design factors and combine their labels to denote experimental configurations (e.g., G1/P1/PE2). Implementation details are provided in the supplementary material.

  1. Proxy granularity (G): G0, G1, and G2 denote coarse, medium, and fine geometry and motion, respectively.

  2. Physics requirements (P): P1 requires Blender’s built-in physics simulation; P0 does not.

  3. Prompt enhancement and proxy alignment (PE): PE0 applies no additional enhancement; PE1 uses MiniMax H3-Context-IR [20] to enrich the prompt; and PE2 requires the coding agent itself to specify subject–color bindings in the prompt and add complementary appearance details based on the constructed proxy.

We use Seedance 2.5 as the direct generation baseline, conditioned on the original prompt and optional references.

Visualization of different implementations of the analyzed components. Figure 3 presents qualitative comparisons of proxy videos and their corresponding generated outputs. The G0–G2 rows illustrate changes in geometric structure and articulation as proxy granularity increases. The P0–P1 rows compare two configurations, highlighting differences in the represented physical motion and its realization in the generated video. The PE0–PE2 rows compare the original prompt, independent text-only enhancement by MiniMax H3-Context-IR, and agent-guided proxy-aware enhancement with explicit subject–color bindings. For each configuration, we uniformly sample eight timestamps over the shared video interval and align the proxy and generated frames at identical timestamps. These paired sequences allow direct inspection of how subject placement, motion, interactions, and appearance are transferred from the proxy to the final output.

Figure 3. Qualitative visualization of component configurations. Within each labeled group, the upper sequence shows the proxy and the lower sequence shows the corresponding generated video; columns follow the same eight sampled timestamps. The groups labeled P0 and P1 correspond to G1_P1_PE1 and G0_P1_PE0, respectively. Both enable explicit physics simulation and differ in geometry and prompt enhancement.

Tasks. Our evaluation covers three task categories:

  1. WorldMemBench: Scene revisitation, evaluating consistency on return visits.

  2. VGIF-Bench: Complex instruction following involving human or object motion and interactions.

  3. Principia: Physical process simulation, which evaluates videos depicting physical phenomena.

Evaluation metrics and protocol. Experts score each video on four dimensions under a blinded protocol using {0,1,2,3}\{0, 1, 2, 3\} (higher is better).

  1. Consistency: Temporal consistency within the generated video and consistency with the input reference.

  2. Instruction following: The extent to which the generated video satisfies the input instruction.

  3. Physical plausibility: The physical plausibility of the depicted motion, interactions, and processes.

  4. Naturalness and visual realism: The perceived naturalness of motion and overall visual realism.

The implementation details of the empirical analysis are in Appendix § 9.§ 1.

Evaluation Results

Table 1 summarizes the results across configurations and benchmarks. We focus on trends that recur across settings and evaluation dimensions. Figure 4 shows how the four-metric mean changes within each benchmark when one component varies and the other two are held fixed.

Figure 4. Effects of proxy granularity, physics simulation, and prompt enhancement on three benchmarks. Each bar shows the mean of the four human evaluation scores (C/I/P/N) reported for one configuration on one benchmark in Table 1. Higher scores are better. From left to right, the groups vary granularity at fixed P1/PE0, physics simulation at fixed G1/PE0, and prompt enhancement at fixed G1/P1. All three groups share the reference configuration G1/P1/PE0.

ConfigurationWorldMemBench C↑I↑P↑N↑VGIF-Bench C↑I↑P↑N↑Principia C↑I↑P↑N↑Average C↑I↑P↑N↑
Direct generation2.1821.2422.4852.3332.4831.8282.2072.0002.5622.4062.0002.1562.4041.8192.2342.170
G0/P0/PE02.7583.0002.6062.4552.4292.2761.8971.4142.9692.9382.5622.5942.7312.7552.3722.181
G1/P0/PE02.6063.0002.5762.4552.6902.2071.4481.0342.9693.0002.6562.7192.7552.7552.2552.106
G2/P0/PE02.7883.0002.6062.4552.5522.0691.4481.0002.9062.8752.5312.5622.7552.6702.2232.043
G0/P1/PE02.6062.9702.7582.6972.4482.2761.7591.3452.9692.9692.5312.6252.6812.7552.3722.255
G1/P1/PE02.8183.0002.7882.6062.5862.2761.6901.2072.9692.9382.6882.7812.7982.7552.4152.234
G2/P1/PE02.7273.0002.7272.5452.4142.2411.3790.9312.9692.9382.7502.5622.7132.7452.3192.053
G1/P0/PE12.4552.9702.6362.6062.4142.1721.3450.8972.0622.4692.4382.2812.3092.5532.1701.968
G1/P0/PE22.7883.0002.5762.5452.5862.3101.4481.0342.9382.9692.5312.6562.7772.7772.2132.117
G1/P1/PE12.1822.6972.5152.2732.7242.8971.8281.3793.0003.0002.1112.2782.5622.8372.1751.950
G1/P1/PE22.8182.9702.7272.5152.9662.9312.0341.4483.0003.0002.5002.5942.9262.9682.4362.213

Table 1. Human evaluation on three benchmarks and per-metric averages over their pooled samples. C/I/P/N denote consistency, instruction following, physical plausibility, and naturalness and visual realism, respectively.

Observation 1: Favorable and Challenging Scenarios of Code2Video

Code2Video most consistently improves consistency and instruction following over direct generation. Its weaknesses concern physical plausibility, motion fluency, and realism on VGIF-Bench, with extensive human and object motion.

Under PE0, all six configurations outperform Seedance on all four dimensions on WorldMemBench and Principia (Table 1). On VGIF-Bench, all improve instruction following but score lower in physical plausibility, naturalness, and realism, with mixed consistency results. All six also outperform Seedance in consistency and instruction following averaged over samples from the three benchmarks.

Observation 2: Granularity Trade-offs. With PE0 fixed, higher modeling and motion detail does not necessarily improve video quality. Coarse proxies (G0) favor naturalness and physical plausibility on VGIF, while medium-grained proxies (G1) achieve the highest or tied-highest consistency across all three benchmarks under P1.

Table 1 shows that, with PE0 fixed, G2 never exceeds G1 in naturalness and realism across benchmarks and physics settings. On VGIF-Bench, G0 leads the three granularities in physical plausibility and naturalness under both physics settings. Manual inspection attributes this advantage primarily to leaving limb motion and fine details to the video generator, reducing stiffness inherited from detailed proxy animation. Under P1, G1 improves consistency over G0 on WorldMemBench and VGIF while tying on Principia.

Observation 3: Benefits of Physics Simulation. Explicit physics requirements (P1) improve the four-metric mean across the three benchmarks under the tested proxy granularities and conditioning strategies. Their benefits are more consistent

in physical plausibility and visual quality than in consistency.

Table 2 shows improvements in both the four-metric mean and the naturalness and realism average across the three benchmarks under every tested P0/P1 comparison. Physical plausibility and instruction following either improve or remain unchanged, whereas consistency changes in both directions. The largest gains in the four-metric mean and physical plausibility occur under G1/PE2.

ComparisonConfig.CIPNMean
P1−P0G0/PE0-0.0540.0000.000+0.075+0.005
P1−P0G1/PE0+0.0430.000+0.160+0.128+0.082
P1−P0G2/PE0-0.043+0.074+0.096+0.011+0.035
P1−P0G1/PE1+0.200+0.250+0.062+0.012+0.131
P1−P0G1/PE2+0.149+0.191+0.223+0.096+0.165

Table 2. Changes in per-metric averages over pooled samples from three benchmarks (P1 minus P0); Mean: C/I/P/N average.

Observation 4: Benefits of Agent-guided PE With G1 fixed, PE2 achieves higher averages across the three benchmarks in all four metrics than PE1 under both physics settings. Under P1, it also improves the four-metric mean over PE0, with gains in consistency, instruction following, and physical plausibility.

With G1 fixed, PE2 outperforms PE1 in all four metrics averaged over samples from the three benchmarks under both physics settings (Table 3). Its advantage over PE0 in the four-metric mean is larger under P1, where consistency, instruction following, and physical plausibility improve despite a slight decline in naturalness and realism. Figure 5 shows how the effect of prompt enhancement varies across benchmarks at fixed G1. On VGIF-Bench, switching from PE0 to PE2 raises the four-metric mean by 0.405 points under P1, while the change under P0 is less than 0.001 points in magnitude. On WorldMemBench and Principia, the same switch slightly lowers the mean under P1.

ComparisonConfig.CIPNMean
PE2−PE1G1/P0+0.468+0.224+0.043+0.149+0.221
PE2−PE1G1/P1+0.364+0.131+0.261+0.263+0.255
PE2−PE0G1/P0+0.022+0.022-0.042+0.011+0.003
PE2−PE0G1/P1+0.128+0.213+0.021-0.021+0.085

Table 3. Changes in per-metric averages over pooled samples from three benchmarks using PE2 at fixed G1; Mean: C/I/P/N average.

Figure 5. Prompt enhancement under two physics settings at fixed proxy granularity (G1). Each point is the unweighted mean of the four human evaluation scores (C/I/P/N) reported in Table 1 for one benchmark and configuration. Dashed lines with open circles denote optional physics simulation (P0), and solid lines with diamonds denote required physics simulation (P1).

Observation 5: Best Overall Configuration Among the tested configurations, G1/P1/PE2 achieves the highest four-metric mean across the three benchmarks, together with the highest corresponding averages in consistency, instruction following, and physical plausibility.

Table 1 ranks G1/P1/PE2 first in the four-metric mean across the three benchmarks and in the corresponding consistency, instruction-following, and physical-plausibility averages. Its naturalness and realism average across the three benchmarks exceeds Seedance but trails G0/P1/PE0, which ranks first on this metric.

Empirical Answers and Workflow Insights

The above empirical observations provide the following answers to the two central questions.

Answers to the Central Questions

  1. How far can Code2Video extend video generation? Code2Video most consistently improves consistency and instruction following over direct generation, with broader gains on scene revisitation and physical-process simulation. Its benefits remain task-dependent: complex human and object motion on VGIF-Bench still presents weaknesses in physical plausibility, naturalness, and realism.

  1. How should the two models’ strengths be combined? Coding agents should specify essential scene structure and dynamics, while video-generation models realize appearance and fine motion. The strongest tested configuration, G1/P1/PE2, combines medium-grained proxies, explicit physics simulation, and agent-guided prompt enhancement. The advantages of coarse proxies on motion-intensive tasks further motivate adapting the allocation of control to the task.

Workflow Design Insights. These findings motivate three coordinated workflow choices: adapt modeling and motion-control granularity to each entity, favoring coarse proxies for human motion and medium-grained proxies where structural consistency matters; selectively invoke Blender’s built-in physics engine for interactions the proxy can meaningfully simulate; and coordinate proxy construction with prompt enhancement by establishing subject–color bindings during modeling and explicitly describing these bindings alongside the proxy’s geometry, motion, and simulated behavior in the generation prompt.

Method

We introduce Coda-DaVinci, a task-adaptive agentic framework for code2video generation. Coda-DaVinci selects a generation path based on task requirements and applies the corresponding modeling and prompting skills. An execution harness coordinates scene construction and video synthesis, using visual feedback to guide revisions, as shown in Figure 6.

Figure 6. Overview of Coda-DaVinci. Coda-DaVinci combines task-adaptive proxy construction with proxy-grounded video synthesis. The harness maintains the working scene and supports visual feedback throughout generation.

Task-Adaptive Proxy Construction

Route selection. Based on the input instruction and reference media, Coda-DaVinci selects 3D modeling for tasks requiring precise control over scene structure or motion. For tasks centered on natural body movement, it selects direct video generation when explicit control is unnecessary and simplified proxy animation could introduce stiffness. Both routes support video synthesis, with Blender tools additionally enabled for 3D modeling.

Modeling granularity. We encode the entity-wise policy from the preceding analysis in modeling skills: G1 for environments and structured objects, and G0 for human subjects. The scene code specifies overall human movement, while the synthesis prompt describes finer actions. The agent simulates requested physical interactions when they can be meaningfully represented in Blender.

Proxy-Grounded Video Synthesis

From proxy to synthesis prompt. After constructing the proxy, the agent inspects the rendered sequence and revises the scene code when the depicted events deviate from the request. Guided by the synthesis skill, it then uses the inspected proxy to write a prompt describing the intended final video. The prompt links subject descriptions to the corresponding proxy entities and appearance references, follows the actions depicted in the proxy, and specifies the visual details and subtle movements to be synthesized.

Video synthesis. Before synthesis, the harness checks that the proxy video and its shot description correspond to the same render and resolves subject references. The generation tool then passes the prompt, proxy video, and associated reference media to the video model, which produces a candidate video. The direct-generation path uses the same synthesis stage without a proxy, supplying a task-derived action and camera prompt with the available references.

Harness Design and Visual Feedback

Workflow execution. The harness provides a shared Blender workspace and tool interfaces. A lead generation agent coordinates execution and can delegate separable modeling tasks to coding subagents. Each assignment specifies how a component should fit into the scene, and returned assets are then integrated and inspected in context. The workspace preserves this work across revisions, allowing subagents to continue refining their components and supporting asset reuse in later video extensions.

Visual feedback. Visual inspection covers both the proxy and the generated video. For code2video outputs, the harness displays generated frames alongside the corresponding proxy views to help identify the source of discrepancies. Proxy errors lead to scene revisions, while discrepancies introduced during synthesis guide changes to the prompt or reference inputs. An independent acceptance agent reviews each candidate against the original task and its registered criteria. The harness links the review to the submitted video and returns unmet requirements for repair.

Experiments

Experimental Setup

We evaluate Coda-DaVinci on World Models’ Last Exam in Physics [5], R2M [6], and VTR [9] for physical behavior, visual consistency, and detailed instruction following, respectively. We further assess video realism and visual instruction following on 200 randomly sampled T2AV-Compass tasks [2]. All methods use Seedance 2.5 [22] for video synthesis, with benchmark images supplied through the reference-media interface to ensure consistent input.

Baselines. We compare three groups: (1) direct generation using task instructions and reference media; (2) agentic generation with Codex or Claude equipped with video-generation tools and skills, with or without VideoWeaver [29]; and (3) Code2Video methods, including Codex + Blender, Claude + Blender, VideoCoCo [13], CodeVideoModel [14], and Coda-DaVinci. Code2Video methods render proxy videos to guide Seedance synthesis.

Models. All Codex-based methods, VideoCoCo, CodeVideoModel, and Coda-DaVinci use GPT-6-Astra [21], while Claude-based methods use Claude Opus 5.5 [1]. Coda-DaVinci and CodeVideoModel additionally uses GPT-Image-2 for reference-image generation.

Main Results

Coda-DaVinci leads on both instruction following and scene consistency. Coda-DaVinci achieves the highest scores across the reported VTR and R2M metrics and improves physical plausibility over direct generation. On VTR, it surpasses the strongest agentic baseline in checklist satisfaction and reduces WER from 59.21% for direct generation to 11.08%. These gains also extend to preserving scene structure and object states across views.

Explicit scene modeling strengthens physical fidelity. Each code2video method outperforms direct generation on World Models’ Last Exam in Physics. Coda-DaVinci ranks second to Claude + Blender, supporting the use of executable scenes for physical interactions. While claude also outperforms Codex with the same Blender tools, suggesting that the coding model’s ability to implement physical behavior contributes to the remaining performance gap.

Adaptive routing improves visual quality. On T2AV-Compass, Coda-DaVinci leads the code2video methods in realism and instruction following, with realism approaching direct generation and conventional agentic methods. This suggests that direct video generation of complex body motion reduces stiffness from proxy animation.

Ablation Experiments

We conduct the ablations on a 33-task diagnostic subset. Gemini 3.8 Flash scores each video from 1 to 5 on four dimensions: physical plausibility, instruction following, visual consistency, and joint aesthetics and fluency. All configurations use the same task-specific checklists, derived from the original instructions and references. We average the four scores to obtain the overall score.

MethodLast Exam Score↑R2M Scene ID↑R2M Local Geo.↑R2M State↑R2M Object↑VTR Checklist↑VTR WER↓T2AV-Compass VR↑T2AV-Compass IF↑
Direct generation47.160.3470.3510.1590.34385.1559.2171.3865.11
Agentic Video Generation
Codex53.180.5750.5830.5200.58789.7225.7372.2168.05
Claude51.950.5940.5900.5230.52287.0753.6271.5068.22
Codex + VideoWeaver55.110.7390.7050.6460.66590.8823.5871.8367.60
Claude + VideoWeaver48.450.6580.6520.6590.56089.8755.0671.5468.19
Code2Video
Codex + Blender59.490.9360.8400.7430.86088.5222.3067.2964.51
Claude + Blender65.520.9500.8470.8230.80588.2350.8866.5464.28
VideoCoCo59.370.9310.8530.6960.82983.6727.1166.2861.61
CodeVideoModel53.860.9320.8780.7080.95384.7520.7464.8861.50
Coda-DaVinci (Ours)60.771.0110.9470.8750.96691.4211.0870.2667.99

Table 4. Table 4 | Main results across four benchmarks. R2M reports normalized memory ratios (NMR). WER: word error rate; VR: video realism; IF: visual instruction following. Green cells mark the best result in each column.

Coda-DaVinci is robust across coding agents. We evaluate Coda-DaVinci with four coding agents and three video generators in Table 5. Overall scores remain similar across coding agents, suggesting that our task-adaptive design works with different models. The video generator also affects the result: Supplementary Material shows that even with correct proxy actions or camera movements, the video may deviate if the model does not follow the reference.

AgentPICA&FMean
Seedance 2.5
Direct generation3.2423.3643.7583.6673.508
GPT-6-Astra3.4553.7583.8793.7583.712
GPT-6.1-Sol3.3943.5453.8183.6673.606
Claude Opus 5.53.5483.7423.8393.6453.694
Gemini 4 Argon3.5623.7193.6563.6563.648
MiniMax H3
Direct generation3.3333.5453.6363.6063.530
GPT-6-Astra3.3943.4853.6973.6673.561
GPT-6.1-Sol3.4193.7103.7103.6773.629
Claude Opus 5.53.2263.7103.8393.7743.637
Gemini 4 Argon3.1563.4383.7503.6563.500
Kling 4.0 Omni
Direct generation3.3943.4243.7583.6063.545
GPT-6-Astra3.3553.7423.8393.6133.637
GPT-6.1-Sol3.3233.7423.7423.6133.605
Claude Opus 5.53.2413.7933.7243.6213.595
Gemini 4 Argon3.4763.7623.8103.7143.690

Table 5. Table 5 | Ablations on the test subset. P: physics; I: instruction following; C: consistency; A&F: aesthetics and fluency. Green marks column maxima (including ties) per video model.

Task adaptation and independent review improve video quality. We ablate three components in Coda-DaVinci with GPT-6-Astra and Seedance 2.5: independent acceptance, adaptive routing, and entity-wise modeling granularity. Each ablation lowers scores on all four dimensions as shown in Figure 7(b). Independent acceptance is particularly useful for consistency, which drops most when this review is removed. Fixing routing to 3D modeling in all tasks or applying G1 to all subjects has its largest effect on instruction following and aesthetics. These results suggest that fulfilling the task depends on both when to use explicit modeling and how much detail to represent.

Figure 7. (a) Normalized benchmark results. (b) Ablations of adaptive routing, independent review, and modeling granularity. (c) Video quality and token cost across rendering engines. (d) On LongAV-Compass, Coda-DaVinci improves event fulfillment and long-form structure over direct generation, with lower transition stability. (e) On WBench, it improves interaction adherence and consistency, with lower setting adherence.

Blender improves video quality at a higher token cost. We replace Blender with Three.js while keeping the coding agent and other settings fixed. As shown in Figure 7(c), both modeling engines improve overall quality over direct generation. Blender scores higher on all four dimensions, suggesting that it provides a better basis for representing these tasks. It also consumes roughly twice as many agent tokens per case. Three.js thus offers a less costly option, with a modest loss in video quality on this subset.

Long-Video Consistency Study

A reusable 3D scene preserves objects and spatial relationships across clips, providing persistent memory for long-video generation.

We compare Coda-DaVinci with direct Seedance 2.5 on 29 LongAV-Compass and 50 WBench cases. LongAV-Compass sequences span 25–90 seconds and are generated event by event, using the preceding tail frame (T2AV/I2AV) or clip (V2AV) as reference. WBench evaluates successive interactions within a scene, with each turn continuing from the previous tail frame.

As shown in Figure 7(d,Figure 7e), Coda-DaVinci improves long-form structure on LongAV-Compass, with its largest WBench gains in interaction adherence and consistency. These findings support scene reuse for maintaining state across successive instructions. However, direct generation performs better in transition stability on LongAV-Compass and setting adherence on WBench, revealing remaining limitations in cross-clip continuity and scene fidelity.

Qualitative Analysis

Scene modeling supports instruction following and state persistence. In Figure 8, the camera is instructed to turn away from a person clearing leaves and then return to the steps. Seedance keeps the steps largely in view, with many leaves still present near the end. Both Codex + Blender and Coda-DaVinci complete the camera departure and return to cleared steps. The ongoing action therefore advances across the unseen interval and leaves a lasting change in the scene. This example illustrates how explicit modeling helps coordinate camera instructions with event progression while preserving the resulting state across a revisit.

Figure 8. Camera instruction following and state persistence. The camera turns away while leaves are being cleared, then returns to the steps. Each method occupies two rows, read consecutively from left to right. Frames are sampled at matching timestamps, with denser sampling during the return. Both code2video methods show cleared steps on revisitation, while Seedance retains substantial leaf coverage.

Selective modeling preserves natural hand motion. Figure 9 compares a close-up of hands flattening dough. Codex + Blender keeps the hands in nearly the same pose throughout the sequence, giving the interaction a static appearance. In Coda-DaVinci, the wrists turn and the fingers adjust through successive presses as the dough changes shape. These coordinated changes make the action more fluid and the contact more convincing, retaining the lively motion visible in native Seedance generation. Here, Coda-DaVinci uses the Blender route: the result illustrates the value of leaving fine articulation to the video model while using the proxy to organize the interaction.

Figure 9. Hand articulation and visual fluency. Consecutive frames from a dough-flattening task, sampled at 4 fps over 4.50–8.25 seconds. Each method occupies two rows, with the second continuing the first. Compared with the nearly fixed pose of Codex + Blender, Coda-DaVinci shows changing wrist and finger poses through repeated contact with the dough. Seedance provides the direct-generation comparison.

Conclusion

We investigated how coding agents and video-generation models can be coordinated within the Code2Video paradigm. Our analysis shows that effective proxy guidance requires matching modeling granularity, physics simulation, and prompting to task requirements. These findings inform Coda-DaVinci, which assigns essential structure and dynamics to coding agents while leaving appearance and fine motion to video models, supported by persistent scene assets and visual feedback. Experiments demonstrate leading performance on VTR and R2M, improved physical plausibility over direct generation, and further gains in long-form structure and consistency across successive interactions. Remaining gaps in motion naturalness, transition stability, and adherence to scene settings highlight the need to better preserve explicit control during synthesis as video generation extends to longer and more complex scenarios.

Case Study

We compare the generated videos with the proxies supplied to the video model. Each example shows the same task across the evaluated configurations, using shared timestamps to follow the scene through the sequence.

Scene and State Consistency

The transit-plaza example in Figure 10 shows how the circular portal and overhead structure persist through camera motion. State changes also need to survive later instructions: in Figure 11, Coda-DaVinci keeps the monitor black as the player reacts, whereas direct generation restores the game image. Figure 12 illustrates the same issue across views, with the modeled bench retaining its structure as the camera moves through the park.

Comparison of Seedance and Coda-DaVinci generated videos and 3D proxy at 0.5s, 4.5s, 9.5s, and 14.5s

Figure 11. Preserving an edited scene state. WBench case 248. The monitor goes black before the player stands up and turns around. Direct generation brings back the game display in later turns, while Coda-DaVinci preserves the black-screen state in both the proxy and the generated video.

Scene consistency across successive views in an autumn park, comparing Seedance and Coda-DaVinci generated videos with 3D proxy frames at 0.5s, 9.5s, and 19.5s

Figure 12. Scene consistency across successive views. WBench case 92. The camera moves through an autumn park. Coda-DaVinci retains the wooden bench across views; the direct-generation sequence introduces a bench with a different armrest structure. Corresponding proxy frames show the shared scene used for continuation.

Figure 10. Scene geometry during navigation. R2M-Bench case WSLRRLL_0046. The portal, curved roof, and surrounding station provide landmarks for comparing scene structure through the camera movement. Proxy renders and generated frames reveal how each modeled scene carries through synthesis.

Scene geometry during navigation: Task: Navigate the transit plaza while preserving the portal, roof, and station layout. Claude + Blender, VideoCoCo, CodeVideoModel, and Coda-DaVinci (Ours), each showing 3D proxy and Generated video at 0.5s, 10s, and 20s.

Figure 10 (PDF p. 20). Scene geometry during navigation (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Task: The game screen goes black; the gamer slams the desk, stands up, and turns around.

Task: Navigate the autumn park while retaining the bench and surrounding scene.

Instruction Following

We next examine tasks that combine several objects and actions. The cassette and keyboard examples in Figures 13 and 14 show how explicit object layout supports the requested interactions while leaving room for natural human appearance. The pet-care example in Figure 15 adds a camera constraint: Coda-DaVinci presents the handover and subsequent actions within a continuous wide view.

Task: Two friends compare written notes and a tracklist, then play a cassette.

Figure 13. Coordinating objects and interactions. VTR-Bench case CULT-0034. Two friends compare a set of written notes and a tracklist before playing a cassette. Coda-DaVinci represents the tabletop objects and subject identities in the proxy, while the final video supplies natural hand movements and human appearance.

Comparison of Claude + Blender, VideoCoCo, CodeVideoModel, and Coda-DaVinci (Ours), showing 3D proxy and Generated video frames at 1.5s, 5s, and 8.5s

Figure 13 (PDF p. 24). Coordinating objects and interactions (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Task: Adjust a split keyboard, show its layout diagram, and display the engraved badge.

Figure 14. Preserving task-critical object structure. VTR-Bench case AD-0059. The task involves adjusting a split keyboard, inspecting its layout diagram, and showing its engraved badge. The proxy specifies the keyboard components and camera views, which guide the corresponding actions in the generated video.

Task: Adjust a split keyboard, show its layout diagram, and display the engraved badge.

Comparison of Claude + Blender, VideoCoCo, CodeVideoModel, and Coda-DaVinci (Ours), each showing 3D proxy and Generated video frames at 0.5s, 3.5s, and 6.5s

Figure 14 (PDF p. 26). Preserving task-critical object structure (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Task: In one wide shot, hand over the leash, explain the care instructions, and hide the key.

Following actions within a continuous shot. Panels labeled Seedance, Codex, Claude, Codex + VideoWeaver, Claude + VideoWeaver, and Codex + Blender; each shows Generated video, with Codex + Blender also showing 3D proxy, at 1s, 5s, and 9.5s.

Figure 15. Following actions within a continuous shot. VTR-Bench case LIFE-0008. The owner hands over a leash and explains the care instructions before the neighbor hides a key under the frog-shaped planter. Coda-DaVinci keeps these actions within the requested wide view, preserving the room layout throughout.

Task: In one wide shot, hand over the leash, explain the care instructions, and hide the key.

Following actions within a continuous shot: Claude + Blender, VideoCoCo, CodeVideoModel, and Coda-DaVinci (Ours), each showing 3D proxy and Generated video at 1s, 5s, and 9.5s.

Figure 15 (PDF p. 28). Following actions within a continuous shot (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Selective Modeling and Visual Quality

The following examples show how simplified proxies can guide the scene while allowing the video model to supply finer appearance and motion. In Figure 16, the proxy describes the flattening dough and surrounding objects, and the final video adds the hands performing the action. The wheelchair interaction in Figure 17 and the alternating cockpit views in Figure 19 extend this division of work to human movement and expression. Figure 18 provides an object-focused example, tracing the transition from modeled food geometry to the final surface detail.

Figure 16. Deformation with natural hand motion. T2AV-Compass case 0348. The proxy controls the dough flattening beneath the paper doilies. Coda-DaVinci leaves the hands to the video model, producing the requested manipulation around the changing dough shape.

Figure 16 (PDF p. 30). Deformation with natural hand motion (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Figure 17. Natural interaction from a simplified scene. T2AV-Compass case 0333. An elderly wheelchair user stops to greet a small dog. The proxy establishes their relative positions and motion, while the generated video supplies the finer posture and appearance of the interaction.

Task: An elderly wheelchair user stops and leans down to greet a small dog.

Natural interaction from a simplified scene: Claude + Blender; VideoCoCo; CodeVideoModel; Coda-DaVinci (Ours). Each configuration shows 3D proxy and Generated video at 1s, 5s, and 9s.

Figure 17 (PDF p. 32). Natural interaction from a simplified scene (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Task: Move from a palace courtyard toward a bowl of steaming, finely pleated dumplings. Seedance — Generated video; Codex — Generated video; Claude — Generated video; Codex + VideoWeaver — Generated video; Claude + VideoWeaver — Generated video; Codex + Blender — 3D proxy, Generated video.

Figure 18. From modeled objects to fine visual detail. T2AV-Compass case 0064. The camera approaches a bowl of steaming dumplings in a palace courtyard. Paired proxy and generated frames show how the modeled arrangement is rendered into food texture, pleats, and steam.

Task: Move from a palace courtyard toward a bowl of steaming, finely pleated dumplings.

Comparison of Claude + Blender, VideoCoCo, CodeVideoModel, and Coda-DaVinci (Ours), showing 3D proxy and generated video frames at 1s, 5s, and 9s

Figure 18 (PDF p. 34). From modeled objects to fine visual detail (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Task: Alternate between a stressed pilot and his focused co-pilot in a spacecraft cockpit.

Figure 19. Subject identity across alternating close-ups. T2AV-Compass case 0424. The sequence alternates between a stressed pilot and his focused co-pilot. Coda-DaVinci uses simple subject proxies to establish identity and framing, allowing the video model to generate facial detail and expression.

Subject identity across alternating close-ups

Figure 19 (PDF p. 36). Subject identity across alternating close-ups (continued). The remaining configurations are shown at the same timestamps, with their corresponding proxy renders.

Implementation Details

Implementation Details for Empirical Analysis

We implement the three experimental components through explicit instructions to the coding agent and controlled selection of the final generation prompt. The agent constructs and renders Blender proxies under the selected configuration. Across configurations, we preserve task-required entity counts, identities, spatial layout, event order, endpoint states, and camera requirements. Background coverage, lighting, shadows, and rendering settings follow the same policy.

Proxy granularity (G). The three levels control the geometry and motion of subjects and interaction-relevant objects:

  • G0 (coarse). Each human is represented by a cylindrical body and a spherical head, without limbs or articulated joints. Animals use simple head and body volumes, and objects primarily use boxes, spheres, or cylinders. Animation represents whole-entity translation and rotation. Invisible logical anchors express possession and object transfer, while task-critical openings or passages may use a few additional primitives.

  • G1 (medium). Humans have a simple torso, head, and four unsegmented limbs with a fixed relative pose and recognizable front/back orientation. Root motion and interaction objects are animated, but local anatomical articulation is excluded. Objects use small assemblies of primitives that retain their coarse silhouettes, major openings, and necessary supports. Contacts are represented through coarse endpoints or logical anchors.

  • G2 (fine). Subjects use proportionate segmented bodies and task-relevant articulated joints. Animation explicitly represents local poses, contact, release, support, and visible state changes. Objects include recognizable components, thickness, cavities, openings, and moving parts at the rendered viewing scale. Facial, finger, and clothing geometry is added only when required by the task; refinement focuses on structure and motion rather than surface textures.

Task-required material changes, including cloth deformation, liquids, particles, and smoke, are permitted at every granularity level using sufficient simulation geometry. These exceptions do not relax the anatomical articulation limits of G0 and G1. Revisions remain within the selected granularity.

Physics requirements (P).

  • P0 (no mandatory simulation). The agent is instructed to produce physically reasonable motion without a mandatory explicit solver. It may use authored trajectories or keyframes. Physics simulation remains permitted when independently selected by the agent.

  • P1 (explicit simulation). The agent is required to execute task-appropriate physics simulation in Blender. Interacting bodies and boundaries use collision geometry with actual contact, non-penetration, support, and collision response. Cloth motion uses cloth simulation; liquid flow, pouring, and splashing use fluid simulation; discrete grains use particle systems; and airborne dust or smoke uses gas simulation. Only processes relevant to the original task are introduced.

Required simulations are executed or baked over the relevant interval, and their evaluated states, deformed meshes, particles, or volumes drive the rendered proxy. Merely configuring a collider or simulation domain is insufficient. The agent records initial conditions, physical parameters, solver settings, and cache paths, together with failed or unsupported simulation attempts.

Prompt enhancement and proxy alignment (PE).

  • PE0 (no enhancement). The original task instruction is retained verbatim as the base generation prompt. Subjects and objects use neutral gray/white proxy materials and are identified through geometry, position, and semantic roles. The controller appends the proxy-interpretation instructions and reference mappings, indicating that final appearance should follow the task and authorized references.

  • PE1 (independent enhancement). MiniMax H3-Context-IR [20] receives only the unchanged original dataset prompt, without the proxy, reference media, agent history, or benchmark-added camera and duration instructions. Its complete response becomes the base generation prompt and is reused across revisions of the same run. The proxy retains the neutral gray/white appearance used in PE0.

  • PE2 (agent-guided enhancement). Before construction, the coding agent assigns each tracked subject a stable entity identifier, an unambiguous semantic description, and a distinct identity color. Colors remain consistent across frames and changes of possession. After inspecting the proxy, the same agent explicitly states the observed color–subject bindings in the generation prompt. These colors serve as identity markers; final colors and materials follow the original task and references. The agent additionally describes compatible appearance details absent from the proxy, including materials, surface properties, lighting, and natural anatomy or motion. Shot descriptions follow the constructed proxy’s actions, camera behavior, timing, and persistent states. Incorrect represented geometry or motion must be corrected in the proxy before generation.

Thus, PE2 jointly changes prompt construction and proxy identity encoding, whereas PE0 and PE1 share the neutral-clay representation.

Experimental configurations. We evaluate all six G×PG \times P combinations under PE0, and all three PE levels at fixed G1 under both physics settings, yielding ten unique Code2Video configurations. The direct-generation baseline uses Seedance 2.5 with the original prompt and optional original references.

Human evaluation. We recruit 10 human evaluators to assess the generated videos under a blinded protocol, with model and configuration identities hidden. Each evaluator scores consistency, instruction following, physical plausibility, and naturalness and visual realism on a four-point scale {0,1,2,3}\{0, 1, 2, 3\}, where higher scores indicate better quality. For each video and dimension, we average the scores from the 10 evaluators. Benchmark-level results average these video-level scores over the evaluated samples, and the overall results pool samples from the three benchmarks.

Benchmarks. We evaluate on three complementary benchmarks. WorldMemBench assesses world consistency during scene revisitation, including the preservation of identities, object appearances, spatial geometry, and evolving states. VGIF-Bench evaluates complex spatio-temporal instruction following, including entity attributes, actions, interactions, and event ordering. Principia assesses physical consistency through motion relationships between paired objects across eight Newtonian phenomena. For this analysis, we retain only samples that produce valid outputs under every evaluated configuration, ensuring that all configurations are compared on the same inputs. This filtering is necessary because Seedance’s privacy policy prevents some samples from returning valid results. The resulting evaluation set contains 33 WorldMemBench samples, 29 VGIF-Bench samples, and 32 Principia samples, totaling 94 samples. Figure 20 shows their distribution.

Figure 20. Composition of the 94-sample evaluation set retained across all evaluated configurations. Each sector shows the benchmark name, percentage, and sample count.

Metrics. We assess generated videos along four dimensions. Consistency measures temporal and cross-shot coherence, including whether scenes, spatial layouts, character identities, and object attributes remain consistent except where changes are explicitly required. It also evaluates fidelity to input reference images when applicable, such as preserving a specified first frame or faithfully reproducing the appearance of a reference subject. Instruction following measures whether the video completely and correctly fulfills the original input requirements, including events, character actions, interactions, environmental layouts, lighting, and camera movements, with the specified temporal order and spatial relationships. Physical plausibility evaluates whether depicted physical processes visually conform to physical laws, including plausible trajectories, gravity, collisions, contact, support, and material responses. Naturalness and visual realism assesses the perceptual quality of appearance and motion, including plausible anatomy, fluid movement, and convincing rendering, while penalizing malformed limbs, stiff or jerky actions, and other visible artifacts. Ten expert evaluators independently rate the videos along these dimensions in a blind evaluation, without access to method identities or configuration labels. Evaluators are provided with the original instructions and any applicable reference images to assess adherence and reference fidelity. We report the mean score across the ten evaluators for each dimension.

Harness Configuration

ToolFunction
Skills and source files
read_skillLoad the instructions for a selected skill.
read_fileRead scene code, manifests, or other workspace files.
write_fileCreate or overwrite a workspace file.
edit_fileApply targeted edits to an existing file.
Scene execution and rendering
run_blenderExecute a Blender scene program.
render_imageRender a still image from the scene.
ToolFunction
render_sequenceRender a sequence of proxy frames.
encode_videoEncode rendered frames into a video.
Camera inspection
inspect_scene_atInspect the scene at a specified time.
preview_camera_variantsRender previews of alternative camera plans.
apply_camera_variantApply a selected camera plan to the scene.
Visual inspection
view_imageDisplay an image for agent inspection.
view_video_frameExtract and display a selected video frame.
view_video_clipPresent a video segment for visual inspection.
inspect_videoInspect a video through sampled frames.
Subject and appearance references
register_subjectRegister a subject identity in the workspace.
bind_subject_referenceAssociate a reference image with a subject.
generate_subject_referenceGenerate an appearance reference for a subject.
generate_scene_referenceGenerate an appearance reference for the scene.
Video synthesis and continuation
video_modelGenerate a video from a prompt and reference media.
video_tail_frameExtract a tail frame for continued generation.
Task delegation
delegate_tasksAssign scoped construction tasks to subagents.
resume_taskResume a subagent session for further work.
send_messageSend feedback or instructions to a subagent.
submit_taskReturn subagent results and assets to the lead agent.
Review and acceptance
set_acceptance_criteriaRegister task-specific acceptance criteria.
record_video_reviewRecord the generation agent’s review of a candidate.
request_video_acceptanceRequest an independent review of the candidate.
submit_video_acceptanceSubmit the acceptance agent’s verdict and feedback.
Execution control and delivery
wait_toolWait for a background tool job to complete.
interrupt_toolInterrupt a running tool job.
finishSubmit the selected output and end the task.

Table 6. Harness tools and their functions.

Additional Analyses of Component and Model Effects

Where proxy guidance helps and hurts. Figure 21 compares G1/P1/PE2, the configuration with the highest pooled four-metric mean in Table 1, with direct generation. On WorldMemBench and Principia, all four metric differences are positive. VGIF-Bench shows a different pattern. Instruction following improves by 1.103 points and consistency by 0.483 points, while physical plausibility falls by 0.173 points and naturalness and visual realism by 0.552 points. For this benchmark, the four-metric mean therefore combines a substantial gain in instruction following with a loss in perceived motion and visual quality.

Figure 21. Per-metric differences between proxy-guided and direct generation. Each cell shows the human evaluation score for G1/P1/PE2 minus the direct-generation score on the same benchmark and metric, using the 0–3 scale in Table 1. Positive values favor the proxy-guided configuration.

SettingConfiguration
Agent coordination and review
Lead generation agentPlans the task, integrates scene assets, and reviews the generated video.
Modeling subagentsReceive scoped construction tasks and return editable assets; their sessions can be resumed for revisions.
Acceptance agentIndependently checks the submitted video against task-specific criteria using read-only inspection tools.
Acceptance repairsUp to three repairs per interaction. The generation agent then inspects and submits the candidate, recording remaining issues.
Persistent workspaceRetains scene code, reusable assets, subject references, and scene state across revisions and continuation requests.
Execution settings
Proxy renderingBlender 5.2.1 with Cycles and CUDA.
Video format24 FPS and 720p by default; duration and aspect ratio follow the task. Audio generation is disabled.
Reference inputsBenchmark images and rendered proxies are supplied through the video model’s reference-media interface.

Table 7. Harness settings for task execution and video review.

Trade-offs between coarse and medium proxy granularity. Figure 22 isolates the change from coarse (G0) to medium (G1) granularity while holding physics and prompting fixed at P1/PE0. On VGIF-Bench, consistency rises by 0.138 points, naturalness and visual realism fall by the same amount, and instruction following is unchanged. On Principia, medium granularity improves both physical plausibility (+0.157) and naturalness and visual realism (+0.156). These benchmark-specific differences motivate adapting the level of proxy detail to the task. This comparison varies granularity for the whole proxy; the entity-wise policy is evaluated separately in § 6.3.

Figure 22. Per-metric changes from coarse to medium proxy granularity. Each point shows the score for G1/P1/PE0 minus the score for G0/P1/PE0, computed from the 0–3 human evaluation scores in Table 1. Points to the right of zero favor medium granularity.

Gains across coding agents and video generators. Figure 23 compares each coding-agent configuration with direct generation using the same video generator. On the 33-task diagnostic subset, 11 of the 12 combinations have a higher reported mean than their corresponding baseline. The largest gain occurs with GPT-6-Astra and Seedance 2.5 (+0.204). The only negative difference occurs with Gemini 4 Argon and MiniMax H3 (−0.030). The size of the gain also varies for a given coding agent: for GPT-6-Astra, it ranges from +0.031 with MiniMax H3 to +0.204 with Seedance 2.5.

Figure 23. Mean score gains over direct generation for each coding-agent and video-generator pair. Each cell subtracts the direct-generation mean for its column from the mean for the indicated pair in Table 5. Scores use the 1–5 diagnostic scale and average physical plausibility, instruction following, visual consistency, and aesthetics and fluency over the 33-task subset.

Prompt Template

The following templates collect the core instructions from the system prompt and workflow skills. Braced fields are filled with task-specific inputs. Task routing

Inputs: {task}, {references}
Read the original instruction and inspect the supplied references. Assess whether
    natural articulated motion is the main challenge and whether a simplified geometric
    proxy would constrain its quality. Consider expressive body movement, animal gait,
    and complex articulated motion when natural appearance is central to the task.
Identify requirements for explicit control, including exact trajectories, physical
    interactions, scene revisits, persistent object states, or precise spatial
    relationships. When these coexist with natural motion, explicit control takes
    priority. Mark uncertainty explicitly.
Return one JSON object with boolean fields natural_motion_primary,
    proxy_rigidity_risk, requires_explicit_control, and uncertain; include evidence as
    exact quotes from the task and a short reason. The controller uses this assessment
    to select the route.

Acceptance criteria

Inputs: {task}, {references}
Before construction, translate the original task into observable requirements for the
    finished video. Cover the subjects, relationships, event order, resulting states,
    and visual qualities that determine success. Select relevant criteria for
    appearance, action continuity, environment, and physical behavior.
For each criterion, record its basis in the task or reference, the expected visible
    result, the event or interval to inspect, and acceptable variation. Merge repeated
    checks of the same requirement.
Call set_acceptance_criteria with id, category, requirement, basis, check, and
    tolerance for every criterion. Keep these criteria fixed during repairs. For a new
    interaction round, register the new requirements together with the states that must
    persist.

Scene appearance reference

Inputs: {task}, {scene_brief}

For a text-only task, call generate_scene_reference to establish the intended scene appearance. Describe the visual medium, major structures, object proportions, occupied space, materials, and lighting. Include required people or animals in their intended locations.

View the returned image. Check object counts and spatial relationships as well as appearance, and revise concrete conflicts with the task. Use the inspected reference when constructing the scene, retaining the geometry needed for movement and occlusion.

Pass the reference to video_model with an explicit appearance role. Describe time-dependent actions and state changes through the proxy and shot prompts. Reuse the established scene design on later visits.

Modeling granularity

Inputs: {scene_brief}, {references}

Use G1 for environments and objects: build recognizable structures from a small number of geometric parts. Preserve major silhouettes, openings, supports, required counts, and spatial relationships. Allocate geometry according to the views and interactions that the video must show.

Use G0 for each human: a cylindrical body and spherical head, animated through whole-body translation and changes in facing direction. Keep this representation consistent through close views and revisions. Use logical anchors to record holding, transfer, and ownership when needed.

Preserve the requested event order and resulting states in the proxy. Describe detailed human appearance and fine articulation in the video-generation prompt, supported by the appearance references.

Physics simulation

Inputs: {scene}, {required_physical_events}

Identify the physical interactions that determine the requested outcome. Choose appropriate simulation mechanisms and construct the geometry needed for collision, contact, support, or containment. Configure initial conditions and the simulation interval from the task.

Execute or bake the simulation over that interval. Use its evaluated geometry, particles, or volumes in the proxy video so the rendered sequence shows the simulated evolution.

Inspect the interaction from the actual camera views, including the approach, contact, and resulting motion or state. Correct visible failures in the scene or simulation and render the affected interval again before submitting the proxy for video synthesis.

Modeling delegation

Inputs: {scene_plan}, {component_scope}

Establish the coarse scene layout before delegating components. Split work where independent construction and review are useful. For each child task, provide the component’s scope, dimensions, origin, connection points, required detail, and views used for inspection.

Submit independent components together through delegate_tasks. Reuse shared meshes or collection instances for repeated assets. Ask children to return editable scene assets and their construction sources.

Load the returned assets into the parent scene and inspect their scale, placement, and connections. When a correction is needed, use resume_task with concrete feedback and updated views so the responsible child can continue from its existing assets and conversation.

Scene assembly

Inputs: {scene_plan}, {component_assets}

Assemble the assets according to the planned layout. Verify their bounds and origins before placement, and preserve required instance counts, spacing, and occupied areas. Inspect overview and camera previews to find missing structures or obstructed action paths.

Animate persistent entities on a continuous scene timeline. Record each object’s location, holder or support, and visible state before and after the relevant events. Carry completed changes through camera cuts and periods outside the frame.

For revisits, evaluate the same scene at the later time. Preview important contacts, releases, and later outcomes from the actual cameras, then correct missing or ambiguous events before rendering the complete proxy.

Camera planning and inspection

Inputs: {assembled_scene}, {shot_requirements}

Plan the camera together with geometry and action timing. Decide what each shot should reveal and preserve requested fixed views, movement directions, cuts, and continuous takes. Choose travel speed according to scene scale and the intended pace.

Specify each move’s duration, travel distance or orbit angle, framing target, and speed profile. Carry motion through adjacent shots when appropriate, and retain the endpoint camera state for continuation.

Render previews along the actual path. Inspect framing and clearance throughout the move, with denser checks around contacts, occlusions, and cuts. For a return to an earlier view, compare visible landmarks and framing with the saved accepted frame while retaining the current scene state.

Subject identity and binding

Inputs: {subjects}, {appearance_references}

Register each recurring subject with a stable subject_id and identity description. Record clothing or appearance variants separately from temporary actions and states. Have delegated tasks retrieve and reuse the registered subject.

Inspect and bind the selected appearance references. Use the same subject_id in the scene and shot manifest, including when the subject leaves and returns. Specify the applicable appearance version and temporary state in each shot when they change.

In the video prompt, explicitly connect each subject’s proxy identity color to its reference appearance. Explain that the color identifies the subject and that the intended final appearance follows the task and reference. Reuse these bindings throughout the video.

Proxy-grounded video prompting

Inputs: {task}, {inspected_proxy}, {references}

Inspect the proxy and reference media before writing a self-contained prompt for the complete video. Begin with the intended finished appearance, explain the role of each reference, and describe the shots in temporal order.

Bind every recurring subject to its proxy identity color and appearance reference. Use the proxy to specify layout, trajectories, camera motion, contact, and event timing. Describe the natural anatomy, local articulation, and surface appearance that the video model should synthesize.

Preserve state changes across shots and specify what should remain visible after an event. Correct wrong geometry, motion, or outcomes in the proxy before synthesis. Inspect the generated video at corresponding moments to locate any failures requiring revision.

Direct video generation

Inputs: {task}, {references}

For a task routed to direct generation, register acceptance criteria and prepare a compact action plan. Specify who acts, how the movement develops, the important contacts, the outcome, and camera framing. Preserve the roles of supplied reference media.

Write a self-contained generation prompt and call video_model with the requested duration, references, proxy_video=null, and shot_manifest_path=null. Retain the returned job and revision identifiers; resume the same submission while its content is unchanged.

Inspect the downloaded video and record concrete findings with record_video_review. Request independent acceptance, then revise the action plan, prompt, or references as required. Deliver the inspected result through finish using its render_uuid.

Long-video continuation

Inputs: {next_instruction}, {scene_checkpoint}, {previous_video}

Load the established scene, scripts, and subject identities. Continue from the latest accumulated object and camera states. Preserve completed changes, including events that occurred outside the camera view.

Inspect the accepted video’s endpoint and use it to guide the next opening. Select the previous video or tail frame and the updated proxy as required by the generation interface, declaring each reference’s role. Keep the established appearance references across segments.

Generate the current segment and inspect the transition from the preceding video. Check the camera view, subject identity, and ongoing action around the boundary. Save the updated scene checkpoint and event states for the next continuation.

Independent video acceptance

Inputs: {original_task}, {acceptance_criteria}, {candidate_video}

Independently inspect the actual candidate against every registered criterion. Cover the sequence and examine key actions, transitions, and resulting states. View short continuous intervals for motion quality and collect additional views when evidence is unclear.

Assign pass when a requirement is satisfied, minor for an acceptable localized defect, rework for a failure requiring correction, or unverified when the available evidence is insufficient. Ground each judgment in timestamped visual observations.

For rework, describe the visible failure and the required correction. Call submit_video_acceptance with the criterion checks and a summary so the generating agent can revise the relevant content.

Revision and delivery

Inputs: {acceptance_feedback}, {proxy}, {candidate_video}

Review the reported failures against the original task and the frozen acceptance criteria. Compare corresponding proxy and generated frames to locate the cause. Revise the responsible scene component, simulation, reference, or video prompt; resume the relevant modeling subagent when its asset needs repair.

Generate the revised candidate and inspect the affected events together with the surrounding sequence. Keep job and revision records so unchanged submissions can be resumed.

Allow at most three repair rounds. After the third rejection, complete the final repair and have the generating agent inspect the result. Deliver the best available video through finish, recording any remaining issues.

References

  1. [1]Anthropic. Claude opus 5.5, 2026. URL https://www.anthropic.com/claude-opus-5-5.
  2. [2]Zhe Cao, Tao Wang, Jiaming Wang, Yanghai Wang, Yuanxing Zhang, Jiahao Wang, Jialu Chen, Miao Deng, Yubin Guo, Chenxi Liao, Yize Zhang, Zhaoxiang Zhang, and Jiaheng Liu. T2AV-Compass: Towards unified evaluation for text-to-audio-video generation. In ICML, 2026. URL https://arxiv.org/abs/2512.21094.
  3. [3]Shuo Chen, Cong Wei, Sun Sun, Ping Nie, Kai Zhou, Ge Zhang, Ming-Hsuan Yang, and Wenhu Chen. Context forcing: Consistent autoregressive video generation with long context. arXiv preprint arXiv:2602.06028, 2026.arxiv.org/abs/2602.06028
  4. [4]Yanzhe Chen, Kevin Qinghong Lin, and Mike Zheng Shou. Code2Video: A code-centric paradigm for educational video creation. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 15370–15396. PMLR, 2026. URL https://proceedings.mlr.press/v306/chen26bx.html.
  5. [5]Mingjiu Gao, Qingle Liu, Yuzhao Peng, Xinjie Lin, Ziming Qin, Zheng Jiang, Wenyi Li, Calvin Xiao, Youjie Zheng, Kaisen Yang, and Qinhuai Na. World models’ last exam in physics. arXiv preprint arXiv:2610.08791, 2026. URL https://arxiv.org/abs/2610.08791.
  6. [6]Qiwen Gu, Bingjie Gao, Rui Chen, Geng Li, Jifan Li, Qishuai Wen, Li Niu, Jing Tang, Xiangxiang Chu, and Junqiao Zhao. R2M-Bench: Evaluating revisit memory via relative consistency in interactive video world models. arXiv preprint arXiv:2608.27328, 2026. URL https://arxiv.org/abs/2608.27328.
  7. [7]Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, Eitan Richardson, Guy Shiran, Itay Chachy, Jonathan Chetboun, Michael Finkelson, Michael Kupchick, Nir Zabari, Nitzan Guetta, Noa Kotler, Ofir Bibi, Ori Gordon, Poriya Panet, Roi Benita, Shahar Armon, Victor Kulikov, Yaron Inger, Yonatan Shiftan, Zeev Melulman, and Zeev Farbman. Ltx-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2601.03233, 2025.arxiv.org/abs/2601.03233
  8. [8]Qixin Hu, Shuai Yang, Wei Huang, Song Han, and Yukang Chen. Longlive-rag: A general retrieval-augmented framework for long video generation. arXiv preprint arXiv:2606.02553, 2026.arxiv.org/abs/2606.02553
  9. [9]Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei, Song Dai, Jiayu Yang, Deyuan Liu, Xiang Zheng, Xiaoshuang Shi, Hao Cheng, and Kaidi Xu. VTR-Bench: A systematic benchmark for evaluating visual text rendering in video generation. arXiv preprint arXiv:2610.01499, 2026. URL https://arxiv.org/abs/2610.01499.
  10. [10]Subin Kim, Sangwoo Mo, Mamshad Nayeem Rizve, Yiran Xu, Difan Liu, Jinwoo Shin, and Tobias Hinz. Rethinking prompt design for inference-time scaling in text-to-visual generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22090–22099, 2026.arxiv.org/abs/2512.03534
  11. [11]Kuaishou AI Team. Kling ai: Video generation model. https://klingai.com/, 2024.
  12. [12]Bingliang Li, Zhenhong Sun, Jiaming Bian, Yuehao Wu, Yifu Wang, Hongdong Li, Yatao Bian, Huadong Mo, and Daoyi Dong. Storyblender: Inter-shot consistent and editable 3d storyboard with spatial-temporal dynamics. In ECCV, 2026.arxiv.org/abs/2604.03315
  13. [13]Haodong Li, Tianfei Ren, Xiaoxiao Ma, Chunmei Qing, Zhen Fang, Sipeng He, Ziyu Guo, Haoyu Wu, Juanxi Tian, Yihang Zou, et al. Videococo: Code-as-cot for physically-consistent video generation via an agentic dual-engine system. arXiv preprint arXiv:2607.27380, 2026.arxiv.org/abs/2607.27380
  14. [14]Zhiyang Liang, Jiawei Zhang, Guocheng Qian, Dongdong Chen, and Ziyu Wan. Code video model, 2026. URL https://code-video-model.github.io/tech_report.pdf.DOI
  15. [15]Shubo Lin, Xuanyang Zhang, Wei Cheng, Weiming Hu, Gang Yu, and Jin Gao. Mmphysvideo: Scaling physical plausibility in video generation via joint multimodal modeling. arXiv preprint arXiv:2604.02817, 2026.
  16. [16]Chunjiang Liu, Xiaoyuan Wang, Haoyu Chen, Yizhou Zhao, Ming-Hsuan Yang, and László A. Jeni. Simworlds: A multi-agent system for dynamic 3d scene creation. arXiv preprint arXiv:2607.01766, 2026.arxiv.org/abs/2607.01766
  17. [17]Tengfei Liu, Yang Shi, Xuanyu Zhu, Jiafu Tang, Liu Yang, Qixun Wang, Zhuoran Zhang, Yuqi Tang, Fengxiang Wang, Yuhao Dong, et al. Longav-compass: Towards unified evaluation of minute-scale audio-visual generation across t2av, i2av, and v2av. arXiv preprint arXiv:2605.26244, 2026.arxiv.org/abs/2605.26244
  18. [18]Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafei Wen, Xiaoxin Chen, and Shifeng Chen. Gpt4motion: Scripting physical motions in text-to-video generation via blender-oriented gpt planning. In CVPR Workshop. IEEE, 2024.arxiv.org/abs/2311.12631
  19. [19]Jiaju Ma and Maneesh Agrawala. MoVer: Motion verification for motion graphics animations. ACM Transactions on Graphics, 44(4), 2025. doi: 10.1145/3731209. URL https://doi.org/10.1145/3731209.
  20. [20]MiniMax. MiniMax H3-Context-IR, 2026. URL https://github.com/MiniMax-AI/MiniMax-H3.
  21. [21]OpenAI. GPT-6 Astra, 2026. URL https://openai.com/index/gpt-6-astra/.
  22. [22]Team Seedance, De Chen, Liyang Chen, Xin Chen, Ying Chen, Zhuo Chen, Zhuowei Chen, Feng Cheng, Tianheng Cheng, Yufeng Cheng, et al. Seedance 2.0: Advancing video generation for world complexity. arXiv preprint arXiv:2604.14148, 2026.arxiv.org/abs/2604.14148
  23. [23]Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi-granularity video representation for multimodal large language model. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 10994–11003, 2025.arxiv.org/abs/2504.10068
  24. [24]Jiafu Tang, Yifan Xu, Eric Tang, Junjie Zhao, Shipeng Luo, Te Sun, Haoxuan Hu, Yiyan Ji, Xinlong Chen, Zihao Wang, et al. Codecap-bench: Can code serve as a good video caption? arXiv preprint arXiv:2610.37341, 2026.arxiv.org/abs/2610.37341
  25. [25]Yuqi Tang, Tengfei Liu, Yizheng Lai, Yuran Wang, Yang Shi, Wanshun Su, Zhuoran Zhang, Qixun Wang, Xiaohan Zhang, Xinlei Yu, et al. Keyframe-compass: Towards comprehensive evaluation of keyframe-conditioned video generation. arXiv preprint arXiv:2607.14202, 2026.arxiv.org/abs/2607.14202
  26. [26]Kling Team, Dong Chen, Jialu Chen, Xinlong Chen, Yifan Dai, Haotian Dong, Binzong Geng, Yushuo Guan, Minghui Guo, Guanlin He, et al. Behind video generation: Observations and insights on multimodal understanding from kling. arXiv preprint arXiv:2611.04916, 2026.arxiv.org/abs/2611.04916
  27. [27]Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuanyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
  28. [28]Jing Wang, Xiangxin Zhou, Jiajun Liang, Kaiqi Liu, Wanyuan Pang, Zhenyu Xie, Tianyu Pang, and Xiaodan Liang. Tem­pact: Advancing temporal plausibility in autoregressive video generation via planner-executor rl. arXiv preprint arXiv:2606.28016, 2026.arxiv.org/abs/2606.28016
  29. [29]Jianhui Wei, Yan Zhang, Jie Tan, Hengchuan Zhu, Xiaotian Zhang, Ziyi Chen, Daoan Zhang, Wei Xu, Yeying Jin, and Zuozhuo Liu. Videoeawer: Evaluating and evolving skills for agentic long video generation. arXiv preprint arXiv:2606.08091, 2026.arxiv.org/abs/2606.08091
  30. [30]Mingrui Wu, Lu Wang, Pu Zhao, Fangkai Yang, Jianjin Zhang, Jianfeng Liu, Yuefeng Zhan, Weihao Han, Hao Sun, Jiayi Ji, et al. Reprompt: Reasoning-augmented reprompting for text-to-image generation via reinforcement learning. In International Conference on Learning Representations, volume 2026, pages 14030–14057, 2026.arxiv.org/abs/2505.17540
  31. [31]Xiaohan Zhang, Yuqing Wen, Junlin Chen, Yuqi Tang, Yiting He, Lizhuo Shao, Weiming Zhu, Tengfei Liu, Yang Shi, Jialu Chen, et al. Multiref-compass: Towards comprehensive evaluation of multi-reference-to-audio-video generation. arXiv preprint arXiv:2607.14189, 2026.arxiv.org/abs/2607.14189

Paper details

vlnc:W_j8p3ffrk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents