VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation

Introduction

Video generation has advanced rapidly through diffusion transformers [7, 21, 32, 45, 51, 56], and benchmarks such as VBench [27, 59] and EvalCrafter [36] measure how plausible and prompt-aligned the generated clips look. Yet many of the videos people actually make—classroom lessons, paper explainers, data stories, product walkthroughs, assembly sequences, rhythm games—demand properties that pixel generators do not guarantee: exact text and numbers, rule-consistent state changes, multi-minute structure, sound aligned to events, and an artifact that can be revised. These are natural properties of programs. Humans already make such videos with Manim or Remotion, and coding agents such as Claude Code and Codex [1, 41] can now work autonomously for hours, driving a full toolchain—a renderer, a headless browser, a TTS engine, ffmpeg—on their own. Users have begun to ask agents for finished videos this way, which requires a multimodal agent that reads input assets, produces picture and sound through code, and inspects its own renders to correct them.

How good are agents at making videos? Existing evaluations do not answer this. Video-generation benchmarks score short, silent clips from generative models [24, 27, 36]. Code-to-visual benchmarks score single programs or static artifacts such as charts, web pages and SVGs [11, 48, 49, 58]. Code-based lesson and paper-to-video systems [12, 33, 63] are language-model pipelines: a workflow designed by hand for one domain and one renderer, in which a model is called at preset steps; they do not test general agents. Agentic benchmarks [30, 38, 54] verify textual or functional outcomes with unit tests, whereas a video is a perceptual, temporal and audio-visual outcome for which no unit test exists. Video therefore tests what text-only agentic benchmarks cannot: whether an agent can check its own output across picture, sound and time.

We present VibeVideoBench (Figs. 1 and Figure 2), a benchmark that evaluates multimodal agents end to end on code-driven video generation: given a specification and optional input assets, an agent must deliver a finished MP4, built entirely with code, inside a sandbox within one hour, with narration, music and sound effects as each task’s audio policy specifies. Every frame is rendered by code: no image or video generation model is available, and the only generative tool is TTS. Its 239 tasks span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers, in English and Chinese. Three design choices shape it.

Tasks must be realistic yet checkable. Real requests are open-ended, vague and often missing the content they refer to, so they cannot be checked as written. We build tasks from four sources—real user requests to code-video tools (rewritten), real videos (seeded from their dense captions), real asset packs such as 3D scans and game levels, and lecture notes and research papers—and rewrite each into a self-contained specification that fixes facts, identities, stages, timing and the audio policy while preserving creative freedom.

Evaluation must rest on evidence in the video. A video has no unit test, a holistic score cannot be traced to evidence, a judge may credit a result the video merely claims, and many VLMs cannot hear. Evaluation therefore has three layers with different executors. Program checks ($L1$) automatically verify the delivery spec—resolution, frame rate, duration, audible sound, source project. Rubric checks ($L2$; about 13k in total) are each traced to the requirement they enforce, separate process shown from result claimed, and are tagged by modality. Quality scores ($L3$) rate five dimensions—visual quality, aesthetics, content logic, timing and audio—each anchored at 1, 3 and 5 by descriptions written for the task. $L2$ and $L3$ are judged by VLM judges in three independent panels: two watch the full video with sound, and the third routes visual checks to a vision model and audio checks to an audio-capable model, so audio checks never reach a judge that cannot hear.

Agents are evaluated as they are deployed. Users adopt a model together with its harness, yet a fair comparison needs identical conditions. Each model therefore runs fully autonomously in its vendor’s native harness at the highest effort its provider accepts, with no follow-up turns and all tool permissions pre-approved, in identical containers with the same one-hour budget; we evaluate complete agent systems rather than separating model from harness. Nine frontier systems from six labs yield 2,151 scored rollouts.

Findings. (1) The benchmark is far from solved, and two systems lead. GPT-6 Astra [43] and Claude Opus 5.5 [2] rank first and second with both the judges and human annotators, yet even the best system satisfies every core requirement on only 76.3% of tasks. (2) Delivery is a second axis. Counting missing videos as zero changes the ranking markedly: Gemini 4 Argon [20] delivers only 77% of its videos and falls from 4th to 7th. (3) Videos are lost to how agents use their time. Of the 224 runs without a playable video, 169 are final renders that started too late, ran too slowly to finish before the deadline (e.g., frame-by-frame browser capture) or never started; Opus, the only system without a failure, finishes a median of 24 minutes before the deadline. (4) Agents build their own renderers, and self-inspection is universal but not decisive. Each system keeps a house rendering stack (cairo, PIL/numpy, or browser canvases and three.js); 95% of runs view their own frames, but how much they look does not predict quality ($\rho=-0.07$), and final reports often claim verifications that do not hold. (5) Judges are reliable for ranking systems, not single videos. The mean of the three panels ranks systems as humans do ($\rho=0.98$), but agrees with humans on only 73% of individual pairs (§5).

Our contributions are:

• VibeVideoBench, to our knowledge the first benchmark of general multimodal agents on open-domain code-driven video generation with audio: 239 tasks in 12 categories (English and Chinese), about 13k task-specific rubric checks, each traced to the requirement it enforces, input assets and a reproducible sandbox (§ 3, Tab. 1).

• A hierarchical, evidence-based evaluation protocol: program checks of the delivery spec, rubric checks that separate process shown from result claimed, and quality scores anchored per task, judged by three independent panels with modality routing so that audio checks reach only judges that can hear; missing videos are handled explicitly by reporting both delivered-only and no-video-as-zero results (§ 4).

• A large-scale empirical study of nine frontier agent systems from six labs (2,151 scored rollouts, 6.5k video judgments), with a run-by-run analysis of all 224 failed deliveries, a trajectory-level analysis of how agents make videos (renderers, self-inspection, delivery timing), and a blind pairwise human preference study on all 239 tasks (§ 5).

Related Work

Video generation: the pixel paradigm. Diffusion-transformer generators [7, 21, 34, 51] synthesize pixels directly from text or images and are evaluated by dimensionwise suites [27, 36, 59], learned scorers [24], physics probes [5, 37] and arenas [28]; VideoArgus [57], closest in spirit to our rubric checks, builds sample-specific rubrics before seeing outputs. Story-video systems [35, 53] add an LLM planner, but a generative model still paints the pixels. We study a different paradigm, in which an agent produces a video by writing programs that drive renderers, TTS and ffmpeg, so that content is exact, editable, and controllable in length and sound. Evaluation that asks whether a few seconds of mostly silent footage look plausible does not transfer to it.

Code as a visual medium: the program paradigm. Programs are an editable, verifiable visual representation of charts, web pages, SVG and 3D scenes [11, 22, 48, 49]; ArtifactsBench [58] grades interactive artifacts from screenshots with checklist-guided MLLM judges. For video, TheoremExplainAgent [33], Code2Video [12], VisualEDU [10] and ManimAgent [29] generate educational Manim videos, Paper2Video [63] turns research papers into narrated slide talks, and an industry benchmark [25] scores short silent clips written in one framework. Code2Video-Compass [13] evaluates language models rather than agents: each video is code written in a single model call, without execution feedback or repair, and has no sound. None of these is agentic in our sense: each video comes from a language-model pipeline built around one renderer or genre, or from a single model call, rather than from an agent that chooses its own tools and renders, inspects and revises its own video.

Benchmark	Output	Pixels from	General agents	Open domain	Audio judged	Input assets	Task-specific rubric
VBench [27]	T2V clip	diffusion	✗	✓	✗	✗	✗
EvalCraft [36]	T2V clip	diffusion	✗	✓	✗	✗	✗
VideoArgus [57]	T2V clip	diffusion	✗	✓	✗	✗	✓
ArtifactsBench [58]	web page	LLM code	✗	✓	✗	✗	✓
TheoremExplain [33]	Manim video	pipeline	✗	✓	✗	✗	✓
Paper2Video [63]	talk video	pipeline	✗	✓	◦	✓	◦
C2V-Compass [13]	code video	LLM code	✗	✓	✗	✗	✗
AgenticVBench [8]	edited video	footage	✓	✓	✓	✓	✓
⭐ VibeVideoBench	code video	agent code	✓	✓	✓	✓	✓

Agent benchmarks. Software and computer-use agents are measured by SWE-bench [30], Terminal-Bench [38], OS-World [54] and PaperBench [50], and GDPval [44] grades occupational deliverables by expert comparison. These verify text, files or functional behavior, whereas a video is a perceptual, temporal and audio-visual outcome for which no unit test exists. AgenticVBench [8] also runs general agents in vendor harnesses, but on post-production of given footage; in our setting every frame and sound must be constructed by code. Tab. 1 contrasts VibeVideoBench with prior work on who makes the video, what is asked, what is delivered and how it is graded.

Judges, rubrics and human preference. LLM and multi-modal judges [9, 31, 60] exhibit position and self-preference biases [52] (we test self-preference in App. § 10.§ 4); rubric decomposition improves reliability in non-verifiable domains [4, 23], and pairwise human preference with Bradley–Terry aggregation [6, 14] is the standard validation. Omni models can see and hear [55], but audio-visual benchmarks show they remain weak at fine temporal alignment [17, 26]. We therefore route checks by modality, require timestamped evidence, use three independent panels, and measure where they fail.

VibeVideoBench

Task formulation

A task $\tau= (p, \mathcal{A}, d, a, c)$ consists of a natural-language specification $p$, an optional read-only asset package $\mathcal{A}$ (data, fonts, sprites, 3D scans, audio), a machine-readable delivery spec $d$ (resolution, frame rate, duration range $[t_{\min}, t_{\max}]$, optional exact frame count), an audio policy $a$ that marks each of four channels—narration, music, sound effects, ambience—as required, optional or not used, and an explicit creative-freedom statement $c$. An agent receives $(p, \mathcal{A}, d, a, c)$ in a fresh container and must deliver a video $v$, an editable source project and a timing sheet before the deadline. There is no ground-truth video: a deliverable is judged only against the frozen specification, and any rendering stack is allowed. No image- or video-generation model is provided; the only generative service is a TTS command-line tool backed by Qwen3-TTS-Flash [46].

Task construction

Figure 2a shows the pipeline. Sources. Tasks start from four kinds of real-world sources: requests people wrote for code-based video tools (what people ask for), dense captions of real videos (what people actually make), real asset packs such as 3D scans, archival records and game sprites, and lecture notes and research papers. Specification contract. Real requests are rarely self-contained, so a source only seeds a task: LLM drafters with cross-review expand and rewrite it under a shared contract that fixes every fact, number and object identity the video depends on, divides it into stages with dwell times and states a four-channel audio policy, while protecting creative freedom; missing assets are searched for or synthesized. What the contract fixes can be checked, and what it leaves open measures the agent’s own design. Human curation. Humans select checkable tasks over open briefs, rewrite each into a self-contained prompt without authoring traces, verify its facts and remove duplicates across sources. The result is 239 tasks in 12 categories and 43 subcategories,

from 8 s to 15 min, with narration, music, effects and ambience, with or without input assets (Figure 3).

Statistics

Figure 3 summarizes the benchmark (App. § 7). Each task has one of 12 categories (content and purpose) and one or more of 43 subcategories, which cut across categories: 34 presentation forms (e.g., mechanism and causal chain, worked examples, map journeys) and 9 knowledge domains (e.g., mathematics, physics, AI). 209 tasks are in English and 30 in Chinese; 201 ship input assets (median 7 files; 2.7 GB); 70% require at least one audio channel and 49% require speech; most require 1920 × 1080 at 30 fps, 54 an exact frame count. One pass asks for 11.1 hours of finished video.

Rubrics

Figure 2b shows how rubrics are built. Every check traces to the final prompt (its requirements, delivery spec $d$ and audio policy $a$) and to the input assets. Each task is evaluated at three levels: $L1$ program checks, derived from $d$ and $a$ and decided by code; $L2$ task checks, atomic pass/fail checks that each quote the requirement they enforce as their basis, are marked core or detail, require the process to be shown rather than a result to be claimed, and carry a tag for the modality needed to decide them; and L3 quality, five 1–5 dimensions—visual quality (VQ), visual aesthetics (VA), content logic (CL), timing and motion (TM), and audio quality and A/V sync (AQ)—with anchors written per task. Rubrics go through rounds of model review, which flags over-strict, missing or undecidable checks, and human verification, until no flag holds; since judges never see the input assets, each check states the correct content itself. The final set has 1,449 L1 checks, 12,985 L2 checks (about 54 per task) and 1,195 L3 rubrics.

Evaluation Protocol

Rollout. Figure 2c summarizes the protocol. Every rollout runs in a fresh Docker sandbox with a common rendering toolchain (e.g., Python, Node, a headless browser, ffmpeg) and the TTS tool of § 3, under a one-hour wall-clock budget. All model traffic is logged, so every trajectory, token and tool call is recorded, and each (system, task) has exactly one scored run; runs lost to an infrastructure fault are re-run.

Details are in App. § 9.

L1: delivery and program checks. L1 is decided by programs alone; no model is involved. A run delivers if it leaves an MP4 that decodes and can be judged; the delivery rate is the share of tasks with such a video. Each delivery is then measured (ffprobe, a full decode, audio levels, VAD and Whisper language ID) and checked by code against the task’s machine-readable delivery spec $d$ and audio policy $a$: every task checks container and decodability, resolution, frame rate ($\pm0.01$) and duration within $[t_{\min}-0.5,t_{\max}+0.5]$ s. The 205 core tasks add a delivered source project, an exact frame count where $d$ requires it (54 tasks) and an audible audio stream (peak $>-50$ dBFS) where $a$ labels a channel required. Lessons and papers, which all require narration, add an audible audio stream, narration coverage (VAD), narration language (ASR) and the absence of long black or frozen stretches. This gives 1,449 checks per system (1,177 core, 272 lesson and paper). Spec compliance is the share of tasks whose run passes all applicable checks; a non-delivery fails them all. (The rollout collector also runs a lighter container-level check at collection time, App. § 9; we use it only in the behavior analysis of § 5.3.)

Model	Harness	Videos	Spec (%)	L2 pass (%)	L3 overall	Human Elo	L2 strict (%)	L3 overall (rank)
GPT-6 Astra	Codex CLI	238	93.7	97.1	4.126	1242	74.5	4.109 (1)
Claude Opus 5.5	Claude Code	239	99.6	97.3	4.074	1203	76.3	4.074 (2)
Claude Sonnet 5.5	Claude Code	232	94.6	93.9	3.999	1121	67.0	3.880 (4)
Gemini 4 Argon	Antigravity CLI	184	76.6	96.5	3.989	1081	55.2	3.069 (7)
DeepSeek-V4.1-Flash	DSH	236	94.6	93.1	3.961	1022	57.6	3.911 (3)
Kimi K3	Kimi Code	213	85.8	91.1	3.864	997	46.3	3.440 (5)
GLM-5.3-Flash	ZCode	179	66.9	85.3	3.736	905	33.5	2.790 (9)
GPT-5.6 Sol	Codex CLI	213	84.5	89.5	3.735	705	39.0	3.317 (6)
Gemini 3.8 Flash	Antigravity CLI	193	77.4	80.5	3.470	723	26.6	2.798 (8)

L2: rubric checks. For each check a judge returns pass, fail, unknown or n/a, with timestamped evidence. Judges score the 12,985 L2 checks (§ 3.4), and we report two metrics. L2 pass pools checks: it is #pass/(#pass+#fail), with unknown and n/a excluded. L2 strict (strict task success) is decided per run: a run succeeds if none of its core checks fails and at least one passes; unknown and n/a do not count against it, and a non-delivery fails. For example, the Super Mario level run (CG01) has 59 checks; a video in which Mario does not grow after touching the mushroom fails the core check “growth is discernible after contact”, so the run fails L2 strict even if the other 58 pass (L2 pass 98%); strict success is thus far more discriminative.

L3: quality. The judge scores each of the five dimensions from 1 to 5 against the task’s own anchors, with reasons, strengths and defects; a dimension the judge cannot assess is left unknown and excluded. For CG01, timing and motion (TM) asks whether continuous motion, landings and state changes can be followed with enough time to see key dangers and successes; it scores 1 if key processes are hard to track, 3 if they can be tracked but transitions or holds are awkward, and 5 if key changes are clear and motion serves the task. The overall score is the mean of the five dimensions, not a weighted total of L1–L3.

Delivered-only and no video = 0. Our primary display reports the delivery rate next to delivered-only L2 and L3 scores, which isolate the quality of what was delivered. We also report every metric with no video = 0: a non-delivery scores zero on every dimension and fails every check, so reliability and quality combine into one number.

Judge panels. Each delivered video is judged independently by three panels, each given the exact task prompt and the rubric but never the system identity. Judges are told that on-screen or narrated self-declarations are not evidence, and that unsampled instants cannot be asserted. J1 (Gemini 4 Argon [20]) and J2 (Gemini 3.8 Flash [19]) watch the full MP4 with sound (4 fps up to 150 s, adaptively lower for longer videos) and judge everything. J3 splits by modality following the check tags: GPT-5.6 Sol [42] judges visual checks and VQ/VA/CL/TM from 24–300 sampled frames (up to 1280 px), and Qwen3.8-Omni-Flash [47] judges audio and audio-visual checks and AQ from a full-length proxy with the original audio. Audio observations are passed to the vision judge as reference. All three panels contain evaluated models (J1, J2 and J3’s vision judge); J3 shares no family with J1/J2. Each panel’s metrics are computed separately and then averaged; scores are never mixed across panels within a metric (3 × 2,151 run results). Tab. 2 reports the three-panel mean (per panel: App. § 11).

Human preference. To validate the judges we ran a blind pairwise study on all 239 tasks. Annotators saw the full prompt and two synchronized players with videos of two systems on the same task, both playable, and chose A, B, tie or both bad by overall quality and adherence. Human preference therefore measures the quality of delivered videos, not delivery. We fit Bradley–Terry strengths [6] on decisive votes (Elo scale, geometric mean 1000) with 2,000-sample bootstrap CIs and compare system ranks and pair outcomes with each panel.

Experiments

Systems. We evaluate nine agent systems from six labs, each in its native harness at the highest available reasoning effort and run fully autonomously, with no follow-up turns and all tool permissions pre-approved (Tab. 2): GPT-6 Astra [43] and GPT-5.6 Sol [42] (Codex CLI [41]), Claude Opus 5.5 [2] and Sonnet 5.5 [3] (Claude Code [1]), Gemini 4 Argon [20] and Gemini 3.8 Flash [19] (Antigravity CLI [18]), DeepSeek-V4.1-Flash, version 1017 [16] (DSH, DeepSeek’s own harness [15]), Kimi K3 [40] (Kimi Code [39]) and GLM-5.3-Flash [61] (ZCode [62]). Agents receive one fixed instruction (“Read ENVIRONMENT.md and prompt.txt. Complete only that task autonomously using your native harness before the deadline…” ). Every multimodal agent system ran all 239 tasks (2,151 rollouts).

Main results

The left half of Tab. 2 describes what systems deliver. L1: Opus is the only system to deliver all 239 videos and passes every program check on 99.6% of tasks; at the other end GLM-5.3-Flash delivers 179 videos and complies with the specification on 66.9%. L2: on delivered videos, item-level pass rates are close to the ceiling (80.5–97.3%), so single checks rarely fail; what fails is getting all of them right on one task (below). L3: Astra (4.13) and Opus (4.07) lead across the board: they hold the top two overall scores on delivered videos and the top two places on every quality dimension except audio (AQ), where Gemini 4 Argon is second; Gemini 3.8 Flash trails (3.47). Human: blind pairwise preferences between two playable videos confirm this order: Astra (Elo 1242) and Opus (1203) lead, and the human ranking matches delivered L3 almost exactly (Spearman $\rho=0.98$; only the last two places swap). Yet L2 strict success (no core check fails and at least one passes; a missing video fails) spreads systems from 26.6% to 76.3%: even the best system, Opus, fails a core requirement on about a quarter of tasks.

The right half of Tab. 2 counts a missing video as a failure, and this largely changes both scores and ranks. Astra and Opus deliver almost every video and stay on top, whereas Gemini 4 Argon, which delivers only 77% of its videos, falls from 4th to 7th. Reliability also has quieter forms: 48 of the 213 playable videos of GPT-5.6 Sol have no audio track, which gives it the lowest AQ on delivered videos (3.71); see App. § 11 for details.

Why videos are not delivered

We read the records of all 224 runs that ended without a playable video and assigned each one main cause (Figure 4a).

Three quarters of the failures (169) are final renders that started too late, ran too slowly or never started; the rest are stalled renders (45, 20%), outputs the agent destroyed itself (7, 3%) and other causes (3). Time management. Agents polish components and previews and launch the full render last: GPT-5.6 Sol’s final full renders in failed runs start at a median of 9 minutes before the deadline, often with frame-by-frame browser capture that needs 5–35 minutes,

and across systems 65 runs (31 of them GLM-5.3-Flash’s) never start a final render at all. Stalled renders occur only in the two Gemini systems: 33 renders deadlock because the agent’s script pipes frames into ffmpeg while never reading its error stream (one stopped at frame 7,744 of 10,102, about 90 s from completion), and in 12 runs the agent ended its turn while a render ran in the background, which was then cut off. Self-destruction is the most avoidable: DeepSeek verified its video twice, then re-encoded it in place one minute before the deadline and left a truncated 0.26 MB file. The successful systems leave margin. Opus, the only system without a failure, finishes its video a median of 24 minutes before the deadline, against 10–11 minutes for Sol and GLM (Figure 4b); it first produces a complete deliverable, then improves only copies and verifies the encoded MP4 itself. Given a longer budget, slower systems recover some of these videos (App. § 15.§ 3).

How agents make videos

We analyze the 2,151 trajectories (tool calls, written files, commands) to characterize the process.

The pipeline is always “code → frames → ffmpeg”: runs call ffmpeg in 96–100% of cases, while Manim appears in 8 of 2,151 runs and MoviePy in 13. Each system has a preferred rendering stack (Figure 4c): Claude agents and Gemini 4 Argon draw vectors with cairo (42–54%), GPT-5.6 Sol, Kimi and GLM rasterize frames with PIL/numpy (43–55%), and Astra, DeepSeek and Gemini 3.8 Flash capture browser canvases and three.js scenes (52–60%). Soundtracks are programmed: 57–96% of runs synthesize music or effects in code. Cairo and browser-2D runs pass the container-level check most often (88% and 85%) and score highest (3.79 and 3.73), three.js scenes least (78%, 3.00); since renderer and system are coupled, this is an association, consistent with slow frame capture being a main cause of late renders (§5.2).

95% of runs view their own rendered frames, yet the amount of looking does not predict delivered quality ($\rho=-0.07$, $n=9$; Figure 4d): Astra looks least (median 3 images per run) and scores best, while DeepSeek looks most (median 35). Reports often claim verifications that do not hold: among playable videos that fail script compliance, the final report still claims verification (“verified”, “ffprobe”, “confirmed”) in 10 of 14 for Astra and 3 of 11 for Sol.

Judges and humans

	J1–J2	J1–J3	J2–J3
System Spearman $\rho$ (overall)	0.950	0.967	0.983
Check-level agreement / Cohen’s $\kappa$	94.6 / 0.54	91.3 / 0.51	91.6 / 0.42
Video-level Pearson $r$ (overall)	0.44	0.49	0.50
Same-task pair ordering accuracy	69.1	69.7	65.6
	J1	J2	J3	Mean
Human vs. judge, system $\rho$	0.98	0.80	0.90	0.98
Human vs. judge, pair agreement (%)	70.5	73.3	69.6	73.3
Judge ties on human-decisive pairs (%)	18	46	14	6

The three panels rank systems almost identically (system-level Spearman 0.95–0.98), and the three-panel mean ranks them like humans do ($\rho=0.98$ with human Elo; Table 3). On single pairs the mean agrees with humans 73.3% of the time, but the agreement depends on the margin: 63% when the panels’ score gap is below 0.2 (about a third of pairs) and 86% when it is 0.5 or more. Judges also miss what humans flag: on pairs that annotators label both bad, the panels give the same scores (3.81) as on other pairs, so judge scores order videos but do not certify them as acceptable. The disagreements concentrate on two systems, in opposite directions: humans prefer GPT-5.6 Sol’s videos 16 points less often than the judges do, as the judges penalize it mainly on audio (AQ), and Gemini 3.8 Flash 11 points more often. Per video, check verdicts reach only $\kappa=0.42$–$0.54$ despite 91–95% raw agreement, overall scores correlate at $r=0.44$–$0.50$, and same-task orderings agree only 66–70% of the time. Panels also differ in strictness: on quality scores J1 is stricter and J2 more lenient, especially on weaker systems, while on core checks J3 is the strictest; strict success, which fails a task on a single failed core check, is sensitive to this (83.7%, 93.3% and 51.9% under J1–J3), so only the three-panel mean is comparable. Implication: current judges can certify system-level rankings over hundreds of tasks, but per-video verdicts—and per-video rewards for training—need margins, multi-panel consensus or human checks before they can be trusted.

Cost

Mean list-price API cost per task ranges from $0.32 (GLM-5.3-Flash) to $15.38 (Claude Opus 5.5), a $49 \times$ spread, and its composition differs by system (Figure 5a): prompt-cache writes account for 76% of Opus’s cost and 48% of Sonnet’s, and fresh, uncached input for 55% of GPT-5.6 Sol’s. Across the nine systems, the rank correlation between mean cost and the overall score is $\rho=0.55$ (permutation $p=0.13$). Three systems lie on the cost–score Pareto frontier (Figure 5b): GLM-5.3-Flash ($0.32, 2.79), DeepSeek-V4.1-Flash ($0.47, 3.91) and GPT-6 Astra ($8.19, 4.11). Output tokens per task range from 61k (Astra) to 289k (Sonnet), and within each system run duration and score are negatively correlated ($\rho$ from $-0.22$ to $-0.69$). Costs are list-price equivalents of logged token usage (billing-verified for Claude); per-system details are in App. § 13.

Conclusion

We presented VibeVideoBench, 239 realistic, checkable tasks on which multimodal agents make complete audiovisual videos with code, evaluated by program checks, task-specific rubric checks and quality scores from three judge panels. Across nine frontier agent systems, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead, yet the best system satisfies every core requirement on only 76.3% of tasks. Delivery is a second axis next to quality: counting missing videos as zero changes the ranking, and most lost videos are final renders that started too late, ran too slowly or never started. The judges’ system rankings agree with human preferences, while on individual pairs of videos they agree with humans only 73% of the time.

Dataset Construction and Statistics

Source streams and funnel

Tab. A.1 traces the 239 tasks back to their sources.

Stage	Description	Count
Stream (i): web prompts
Collected	prompts written for code-based video tools (452 social posts, 25 Remotion gallery, 7 other)	484
Kept after strict review	every prompt read in full; nine removal codes	152 (31%)
Stream (ii): dense captions of real videos
Scanned	caption records (HD-VILA, Molmo2, LLaVA-Video, ActivityNet sources)	508,993
Lexical recall	26 content directions	6,711
LLM screen and close reading	graded screen, then full-caption review (keep 200, hold 109, reject 13)	956 → 322 → 200
Human decision	two annotators on the 200 kept; “≥1 keep and no discard”	84 (42%)
Authoring and selection
Drafted	drafts from the streams above, plus gap-filling drafts	447
Human selection	two selectors	184 (41%)
Final rewrite	removal of leaked authoring traces	182
Extension
Added tasks	asset-grounded tasks (stream iii), classic-game tasks and tasks kept from a pilot	+29
Removed on review	duplicates and an infeasible task	−6
Core tasks	199 English, 6 Chinese	205
Stream (iv): lecture notes and papers
Lessons (ZH)	self-written lecture notes, seven subjects; 2 politics lessons removed	24
Papers (PA)	classic papers supplied as PDF (Transformer, ResNet, GAN, DDPM, NeRF, VAE, DQN, ViT, CLIP, LIGO)	10
Released and scored	209 English, 30 Chinese; every task judged by all three panels	239

What was removed. Most web prompts failed because they presupposed material that was not provided (125 of 332 removals) or were content-free style templates (99); the rest were intros and logo stings (43), ambient loops without development (25), tasks that do not ask for a finished video (19) and duplicates (15); three rarer codes cover the remaining 6.

Selection. Human selectors preferred checkable tasks: they kept 88% of the music and sound-driven drafts and about half of the explainers, but only 17% of the story and comedy drafts and 15% of the game drafts.

Authoring. LLM drafters with cross-review recorded for every task its source, the selected time range and everything added at authoring time (facts, lines, durations, assets, structure). A final rewrite removed every authoring trace (reviewer notes, review status, scoring hints) from the prompts.

Taxonomy

Each task has exactly one of 12 categories (content and purpose; Tab. A.2) and one or more of 43 subcategories that cut across categories (Tab. A.3): 34 presentation forms, the first of which is the task’s primary form, and 9 knowledge domains, tagged on tasks that convey subject knowledge (C04, C05, C11, C12). Tasks carry 2.0 subcategories on average; the per-task mapping is released with the benchmark.

Task format and statistics

An agent sees only the prompt and the task’s input files; source records and audit notes stay internal, and the rubric goes only to the judges. A typical prompt gives the audience and purpose, a structure line, the content inputs (facts, numbers, real or fictional assets), the progression with minimum dwell times, the four-channel audio policy, the creative freedom and the delivery list (MP4, editable project, a timing sheet and production notes). Tab. A.4 summarizes the tasks.

Category	Tasks	Most frequent subcategories (tasks; multi-label)
C01 Brand & marketing	9	Contextual selling point & call to action (7), Brand & product showcase (3), Musical structure & AV choreography (2), Visual metaphor (1)
C02 Product & feature demos	6	Software & device tutorials (4), Making, assembly & modification (1), Musical structure & AV choreography (1), System simulation & causal modelling (1)
C03 Events & personal expression	7	Event invitation (4), Personal expression & tributes (3), Statistical charts & comparison (1), Musical structure & AV choreography (1)
C04 Concept & pipeline explainers	57	Mechanism & causal chain (26), Engineering & machines (14), Physics & astronomy (13), Mathematical derivation & counting (12)
C05 Data & fact stories	24	History & society (12), Statistical charts & comparison (11), History & chronology (11), Geography & earth science (8)
C06 Visual art & form	26	Spatial tour & scene experience (9), Character performance & choreography (5), Visual metaphor (5), Abstract form & scale (3)
C07 Games & sports	19	Real-time follow-along guidance (7), Tactics & strategy breakdown (6), Classic game reconstruction (4), Observe–guess–reveal (2)
C08 Story & visual comedy	15	Escalating missions & crises (8), Musical structure & AV choreography (2), Character performance & choreography (2), Dialogue & punchline (2)
C09 Building & process	23	Making, assembly & modification (23), Mechanism & causal chain (5), Spatial tour & scene experience (4), Interactive model, structure & scale exploration (1)
C10 Music & sound-driven	19	Musical structure & AV choreography (18), Character performance & choreography (7), Real-time follow-along guidance (2), Abstract form & scale (2)
C11 Classroom lessons	24	Mechanism & causal chain (12), Mathematical derivation & counting (8), Mathematics (8), Interactive model, structure & scale exploration (6)
C12 Paper explainers	10	Artificial intelligence (9), System architecture & information flow (5), Mechanism & causal chain (4), System simulation & causal modelling (2)
Total	239	205 core tasks + 24 classroom lessons (C11) + 10 paper explainers (C12)

ID	Presentation form	$n$	Cat.	ID	Presentation form	$n$	Cat.
S09	Mechanism & causal chain	51	6	S27	Abstract form & scale	6	3
S29	Musical structure & AV choreography	28	7	S28	Visual metaphor	6	2
S07	Making, assembly & modification	25	3	S38	Worked examples & problem solving	6	2
S08	Mathematical derivation & counting	23	4	S34	Conceptual frameworks & decision paths	5	1
S14	Statistical charts & comparison	18	5	S01	Software & device tutorials	4	1
S10	System architecture & information flow	17	3	S25	Time-span, identity & emotional change	4	2
S16	Interactive model, structure & scale exploration	17	4	S32	Event invitation	4	1
E04	Character performance & choreography	15	3	CG	Classic game reconstruction	4	1
E01	Spatial tour & scene experience	14	3	E05	Poetry & emotion	3	2
S05	System simulation & causal modelling	14	6	S23	Journey & arrival	3	3
S13	History & chronology	13	2	S26	Kinetic typography & channel packaging	3	3
E02	Real-time follow-along guidance	11	3	S31	Brand & product showcase	3	1
S03	Map journeys & route tours	10	4	S37	Personal expression & tributes	3	1
S22	Escalating missions & crises	10	3	E07	Miniature & fictional worlds	2	1
S04	Tactics & strategy breakdown	8	2	S06	Observe–guess–reveal	2	1
S12	Experiments & evidence reading	8	4	S24	Dialogue & punchline	2	1
S33	Contextual selling point & call to action	7	1	S18	Picture-book reading & early concepts	1	1
ID	Knowledge domain	$n$	Cat.	ID	Knowledge domain	$n$	Cat.
D01	Mathematics	17	2	D06	History & society	18	3
D02	Physics & astronomy	18	4	D07	Computing & information	12	2
D03	Chemistry & materials	6	2	D08	Artificial intelligence	13	3
D04	Biology & medicine	11	3	D09	Engineering & machines	15	2
D05	Geography & earth science	12	3

Language	209 English / 30 Chinese [core 199 / 6; lessons 24 Chinese; papers 10 English]
Prompt length	core English prompts: median 1,638 words, IQR 967–2,179, range 503–5,581; lesson and paper prompts are short (median 262 words/characters) and carry their content in the supplied note or PDF
Resolution / fps	1920×1080: 227; other: 12; 30 fps: 228, 60 fps: 11; exact frame count: 54
Target duration	midpoint median 118 s, IQR 55–270 s, range 8–894 s; lessons 180–480 s; 38 tasks exceed 5 minutes; 11.1 hours per pass
Audio policy	≥1 required channel: 168 (70%) [core 134 (65%)]; speech required: 117 (49%); every lesson and paper requires narration
Input assets	201 tasks [core 167, all 34 lessons and papers]; 2.69 GB
Rubric	12,985 L2 checks [core 10,813; lessons and papers 2,172]; mean 54.3, median 54, range 13–110

Rubric Design

Structure of a rubric

Each rubric groups the task checks (L2) under a few core goals, maps every requirement of the prompt to the checks that enforce it, and adds the task’s quality rubric (L3), whose five dimensions are anchored at 1, 3 and 5 by descriptions written for that task. A check states what passes and what fails, where in the video to look and which prompt sentence it enforces, and carries a criticality and a modality tag (Tab. B.1). Over the 239 tasks there are 12,985 checks (54.3 per task, 13–110); 85.8% are core, the rest detail (supporting for lessons and papers). Unknown is reserved for insufficient evidence: “confirmed omissions or violations in the complete work are a fail and must not be concealed with unknown”.

ID / tag	Title	Pass	Fail	Observation scope
V05 core, visual	Initial small form	Mario is small at departure and does not possess large-form abilities without cause during the initial stage.	Starts in the large form and skips the first growth, or has already grown before acquiring the mushroom.	From departure until before the first mushroom contact
V12 core, visual	Large form breaks a brick from below	After becoming large, Mario hits BRICK1 or BRICK2 from below and causes at least one to shatter.	Breaking the brick early in small form, without contact, or hitting only the question block.	After the transformation until the first brick breaks
V30 core, visual	Complete jumps	Required jumps have traceable takeoff, airborne movement and landing/contact.	Pose changes impersonate displacement; teleporting to the landing point.	All required jumps
V41 core, visual	Order of key events	Q1 mushroom → brick → G1 stomp → shell kicked into G2 → hit by G3 → left over pit → flagpole and castle.	Reordering events, or montage that hides the actual sequence.	Whole video
V50 core, audio-visual	Jump SFX synchronization	A jump cue corresponding to the action is present at each required takeoff.	No cue, or the effect occurs during an unrelated action.	Every required takeoff and the audio track

Quality rubric. The five L3 dimensions share 20 observation points:

1	2	3	4
VQ key-content legibility	stability	occlusion/cropping/layering	object consistency
VA hierarchy	composition	unity of visual language	audience fit
CL organization	key relations	information load	channel coordination
TM action	state change and transitions	dwell on key content	overall rhythm
AQ intelligibility and pronunciation	prosody	mixing	sound–event alignment

Each dimension has task-specific anchors at 1, 3 and 5 (e.g., VQ for CG01: 1 = “key content remains unclear or is disrupted by visual defects”, 5 = “key content clear, image stable, details support viewing”; a 2 or 4 must say which anchor it is closer to, and insufficient evidence yields unknown, never a default 3. Intentional pixel-art style, grain or blur is not a defect.

Authoring principles

• Traceability. Every check quotes the prompt sentence it enforces, and optional language (“may”, “e.g.”) never becomes mandatory.

• Process over claims. A correct final number or a caption reading “synced” cannot pass a check that requires a visible process: in a marble adder the sum must be produced by actual contacts.

• Criticality fixed in advance. Core and detail are fixed before any output is seen, so cosmetic passes cannot outvote a failed core goal.

• Timing under sampling. “At least $N$ seconds” passes only if the lower bound of the sampled interval is $\ge N$, fails if the upper bound is $<N$, and is otherwise unknown.

• No prescribed method. 40–50 checks is a coverage aim, not a quota, and no check may prescribe a production method.

Rubrics for lessons and papers

The 34 lessons and papers (C11, C12) use the same levels and judge interface, but their checks come from the supplied material. L1 adds narration coverage (VAD speech share $\ge50\%$, no gap over 20 s), narration language (ASR, $\ge90\%$ in the target language) and no long black or frozen stretch. L2 has 48–83 checks per task (2,172 in total; 1,566 core): content checks enumerate the knowledge points (each concept, derivation step, worked answer, common misconception and relation to be drawn; for papers the motivation, key idea, method components, key equations, results, evidence and limitations), and narration checks require correct reading of terms, formulas and numbers, explanation rather than reading on-screen text aloud, and synchronization at 4–6 key moments. Because judges never see the note or the paper, 330 checks that referred to them (“consistent with the note”) were rewritten to state the required content, and six audit agents re-derived every number and formula against the source. Each task also has a grounded quiz (404 questions), reported separately (App. § 15.§ 2).

Quality control

Rubrics were refined over many rounds in a closed loop. In each round an independent model reviewed every rubric against its prompt, each finding was re-adjudicated in a fresh context and verified by a human reviewer, confirmed findings were fixed, and the revised rubrics re-entered review until no confirmed finding remained. The most frequent defect was over-specification (a rubric demanding more than the prompt), ahead of coverage gaps; the review also surfaced bugs in the tasks themselves, such as an infeasible inventory or contradictory timing tables, which were fixed in the prompts.

Modality tags

Each check is tagged with the evidence needed to decide it: visual 9,006 (69.4%), audio-visual 3,012 (23.2%) and audio 967 (7.4%). The tags route checks in J3 (App. § 10).

Sandbox, Harnesses and Rollout Protocol

System	Lab	Harness	Effort
GPT-6 Astra	OpenAI	Codex CLI 0.162	max
GPT-5.6 Sol	OpenAI	Codex CLI 0.162	max
Claude Opus 5.5	Anthropic	Claude Code 2.1.301	max
Claude Sonnet 5.5	Anthropic	Claude Code 2.1.301	max
Gemini 4 Argon	Google	Antigravity CLI	high
Gemini 3.8 Flash	Google	Antigravity CLI	high
DeepSeek-V4.1-Flash	DeepSeek	DSH 0.2.1	max
Kimi K3	Moonshot	Kimi Code 2.2.0	max
GLM-5.3-Flash	Zhipu	ZCode 0.17.0	max

🖥️ Container. Each run uses a fresh container from one image (Ubuntu 24.04) with 8 CPUs, 32 GB of memory and no GPU; the task’s input files are read-only.

⚙️ Preinstalled tooling. Python and Node with common imaging and browser-automation libraries, headless Chromium with software WebGL, ffmpeg, ImageMagick, SoX, three.js, GSAP and a large CJK and Latin font set. Internet access goes through a proxy, and package installation is allowed. A text-to-speech tool (Qwen3-TTS-Flash) is provided; no other generative APIs are available.

📋 Instruction. The harness prompt is fixed: “Read ENVIRONMENT.md and prompt.txt. Complete only that task autonomously using your native harness before the deadline. . . .” ENVIRONMENT.md states the 3,600 s budget, which includes model waits, tools, TTS, previews and the final render. It imposes no token, tool-call or preview limits and lists the deliverables (final MP4, editable project, timing sheet and production notes). It asks the agent to “finish with honest verification results”. At the deadline the container is killed.

Judge Protocol and Judge Analysis

Protocol

Each judge receives the prompt the agent received, the task checks and the quality rubric, but never the system identity, the source code or the agent’s notes. For every check it returns pass, fail, unknown or n/a with the time range and the channel (picture or sound) that support the verdict, and it scores the five quality dimensions against the task’s anchors with reasons. On-screen or narrated claims (“✓ requirement met”) are not evidence, properties of the delivered file are left to the program checks, a judge that sees sampled frames may not assert what happens between them, and insufficient evidence yields unknown.

J1 (Gemini 4 Argon) and J2 (Gemini 3.8 Flash) watch the whole video with sound and judge everything. J3 splits the work by modality tag: GPT-5.6 Sol judges the visual checks and VQ, VA, CL and TM from sampled frames, and Qwen3.8-Omni-Flash judges the audio and audio-visual checks and AQ from the full video with its original sound. Each of the 1,927 playable videos was judged by all three panels. A run without a playable video scores zero on every dimension and fails every check (§4).

Example judge output (J1, CG01, Claude Opus 5.5):

CG01-V41 pass. “The events occur in exact order: Q1 hit & mushroom power-up (9.0–15.75 s) → brick break (21.0 s) → G1 stomp (22.75 s) → K1 stomp, kick & G2 defeat (30.25–33.0 s) → G3 side hit (41.0 s) → MOVE1 crossing (47.5–54.0 s) → flagpole & castle (61.75–68.5 s).”

CG01-V50 pass (audio). “A rising pitch jump sound effect plays synchronously at each of Mario’s takeoffs (e.g., 6.0 s, 8.75 s, 17.5 s, 20.5 s, . . . , 61.75 s).”

VQ = 4. “Closer to 5 than 3 because all characters, states, HUD counters, and contact points are crisp and stable, with only minor visual overlap during the side-hit protection pass.”

Agreement between panels

The three panels rank the systems almost identically (Spearman $\rho$ 0.95–0.98 per pair, Kendall’s $W$=0.98; Fig. D.1a), but agree much less on single videos: the per-video overall correlates at $r$=0.44–0.50, and two panels order two videos of the same task alike only 66–70% of the time. On checks both panels decide, raw agreement is high (91–95%) only because most checks pass; Cohen’s $\kappa$ is 0.42–0.54.

Leniency and scale use

Panels differ in strictness, especially on weak systems. J1 is the harshest on L3 ($-0.41$ against the other two panels) and the most discriminative (spread 0.89 on delivered videos). J2 is lenient and compressed: 70% of its dimension scores are a 4 and its delivered-only spread is 0.46. J3 is the harshest on checks (L2 88.2%; it also leaves 6.3% unknown) but the most generous on VA and AQ (+0.46 and +0.61 against the other two). Leniency is not uniform (Fig. D.1b): J1 is harsher on the weaker systems (GPT-5.6 Sol −0.67, Gemini 3.8 Flash −0.59, against −0.24 for Astra and Opus), and J2 and J3 are more lenient on

Table D.1. Judge scale use (239 tasks). L3, L2 and unknown: averages over the nine systems (delivered runs). Leniency: mean of the panel’s five-dimension score minus the mean of the other two panels on the same run (1,927 runs). Spread: best minus worst system overall. Right: distribution of all dimension scores (%; unk.: unknown or n/a).

Panel	L3 (deliv.)	L2 pass	Unknown	Leniency	Spread nv=0	Spread deliv.	1	2	3	4	5	unk.
J1	3.61	90.9%	0.9%	−0.41	1.47	0.89	0.8	6.0	28.6	57.4	6.0	1.3
J2	4.01	95.7%	0.4%	+0.19	1.19	0.46	0.2	2.5	9.2	69.9	17.3	0.8
J3	4.03	88.2%	6.3%	+0.22	1.39	0.65	2.1	2.2	16.3	46.8	31.4	1.2

them (J2: Sol +0.35, Flash +0.31, Astra −0.02). This is why panels are never mixed within a metric and only the three-panel mean is compared across systems.

Self-preference

Three panels are run by models that are also evaluated as agents: J1 is Gemini 4 Argon, J2 is Gemini 3.8 Flash, and J3’s vision judge is GPT-5.6 Sol. Tab. D.2 therefore measures, through a relative score defined in the caption, how much more each panel likes a system than the other two panels do.

Table D.2. Self-preference on the 1,886 videos that all panels scored on the four visual dimensions (VQ, VA, CL, TM; for J3 these come from GPT-5.6 Sol alone). Relative score: the panel’s score, centered on its mean over all videos, minus the same centered score averaged over the other two panels; positive = the panel rates the system higher than the other two panels do. Shaded: the panel’s own model. 95% CIs from a 2,000-sample bootstrap over videos.

System	J1 (Gemini 4 Argon)	J2 (Gemini 3.8 Flash)	J3 vision (GPT-5.6 Sol)
GPT-6 Astra	+0.17 [0.11, 0.23]	−0.23 [−0.29, −0.17]	+0.06 [−0.02, 0.13]
Claude Opus 5.5	+0.21 [0.15, 0.27]	−0.11 [−0.17, −0.04]	−0.10 [−0.18, −0.03]
DeepSeek-V4.1-Flash	−0.03 [−0.10, 0.05]	+0.05 [−0.02, 0.13]	−0.03 [−0.09, 0.04]
Claude Sonnet 5.5	+0.18 [0.11, 0.25]	−0.08 [−0.15, −0.01]	−0.10 [−0.17, −0.03]
Kimi K3	+0.02 [−0.04, 0.08]	+0.03 [−0.04, 0.10]	−0.05 [−0.12, 0.03]
GPT-5.6 Sol	−0.32 [−0.40, −0.24]	+0.12 [0.05, 0.20]	+0.20 [0.12, 0.27]
Gemini 4 Argon	+0.03 [−0.05, 0.10]	+0.01 [−0.08, 0.09]	−0.03 [−0.11, 0.04]
Gemini 3.8 Flash	−0.22 [−0.28, −0.15]	+0.16 [0.09, 0.23]	+0.05 [−0.02, 0.12]
GLM-5.3-Flash	−0.16 [−0.25, −0.07]	+0.13 [0.05, 0.22]	+0.03 [−0.05, 0.11]
Own model vs. others	+0.03 [−0.05, 0.11]	+0.18 [0.10, 0.26]	+0.22 [0.14, 0.29]
Own family vs. others	−0.12 [−0.18, −0.06]	+0.11 [0.05, 0.17]	+0.16 [0.10, 0.22]

Two of three judges favor their own model, by about a sixth of a point. J2 rates the Gemini 3.8 Flash agent +0.16 and J3’s vision judge rates the GPT-5.6 Sol agent +0.20 higher than the other panels do; both intervals exclude zero. J1 shows no preference for the Gemini 4 Argon agent (+0.03) and rates the Gemini family lower than the other panel’s (−0.12). Panels also have tastes of similar size that have nothing to do with themselves: J1 favors the Claude agents and Astra (+0.17 to +0.21) and disfavors Sol (−0.32), and J2 disfavors Astra (−0.23) while being lenient on the weaker systems (GLM +0.13, Sol +0.12), so part of its +0.16 for Flash is its general leniency toward weaker systems.

The ranking does not depend on it. Scoring each system without the panel that is its own model (J1 for Argon, J2 for Flash, J3 for Sol) changes only the last two places: Flash and GLM, 0.008 apart on the pooled leaderboard (Tab. E.1) and statistically tied, swap (Kendall $\tau$=0.94 with the headline ranking). Scoring each system only with panels outside its family (the Gemini agents by J3 alone, the GPT agents by J1 and J2), after shifting every panel by its leniency offset on the commonly judged videos (−0.27, +0.13 and +0.14 against the three-panel mean; the leniency of Tab. D.1, which is measured against the mean of the other two panels, rescaled by 2/3), leaves the order unchanged ($\tau$=1): the Astra-Opus gap narrows from 0.034 to 0.018 (paired bootstrap $P$(Astra>Opus)=0.74), and Sol stays above Argon ($P$=0.95). Without the shift Opus overtakes Astra, but only because the panel dropped for the GPT agents (J3) is a lenient one.

Full Results

All numbers in this section are on the 239 tasks, with the three panels computed separately and then averaged unless a panel is named.

Full leaderboard

Tab. E.1 repeats the main table with every column: the five L3 dimensions, spec compliance, the L2 pass rate with no video = 0, the human Bradley–Terry rating and the API cost.

#	System	Harness	$/task | Deliv. | Spec | Pass | VQ | VA | CL | TM | AQ | Overall | Pass | Strict | Overall (rank) | Human Elo
1	GPT-6 Astra	Codex CLI	8.19	99.6	93.7	97.1	4.08	4.30	4.20	3.95	4.10	4.126	96.9	74.5	4.109 (1)	1242
2	Claude Opus 5.5	Claude Code	15.38	100.0	99.6	97.3	3.99	4.19	4.18	3.95	4.07	4.074	97.3	76.3	4.074 (2)	1203
3	Claude Sonnet 5.5	Claude Code	8.26	97.1	94.6	93.9	3.93	4.12	4.10	3.86	3.98	3.999	91.2	66.9	3.880 (4)	1121
4	Gemini 4 Argon	Antigravity CLI	2.94	77.0	76.6	96.5	3.80	4.05	4.14	3.86	4.09	3.989	73.5	55.2	3.069 (7)	1081
5	DeepSeek-V4.1-Flash	DSH	0.47	98.7	94.6	93.1	3.86	4.13	4.04	3.75	4.03	3.961	92.0	57.6	3.911 (3)	1022
6	Kimi K3	Kimi Code	3.53	89.1	85.8	91.1	3.73	3.98	3.97	3.70	3.95	3.864	79.3	46.3	3.440 (5)	997
7	GLM-5.3-Flash	ZCode	0.32	74.9	66.9	85.3	3.66	3.87	3.77	3.57	3.81	3.736	61.7	33.5	2.790 (9)	905
8	GPT-5.6 Sol	Codex CLI	11.84	89.1	84.5	89.5	3.72	3.94	3.86	3.45	3.71	3.735	78.0	39.1	3.317 (6)	705
9	Gemini 3.8 Flash	Antigravity CLI	2.44	80.8	77.4	80.4	3.33	3.57	3.59	3.12	3.74	3.470	61.3	26.6	2.798 (8)	723

Strict metrics and core vs. detail. Core checks are harder than detail checks for every system (Astra 97.0 vs. 97.8%; Gemini 3.8 Flash 79.4 vs. 86.7%). The share of tasks with no failed check of any kind (no video counts as failure) ranges from 73.6% (Opus) and 72.1% (Astra) to 23.8% (Gemini 3.8 Flash), close to strict success (76.3, 74.5 and 26.6%): most tasks that fail strict success fail a core check, not only detail checks. What keeps a task from strict success differs by system: for Opus, Astra, Sonnet and DeepSeek it is almost always a failed core check, whereas 19–25% of the runs of Gemini 4 Argon, GLM and Gemini 3.8 Flash fail it by delivering no playable video.

Per-panel leaderboards

Tab. E.2 reports each panel separately. The panels differ in leniency but largely agree on the order: the top four are the same in every panel (Astra, Opus, DeepSeek, Sonnet), followed by Kimi and GPT-5.6 Sol. Their disagreements are between near-ties: J1 and J3 put Astra first (3.946 vs. 3.915; 4.285 vs. 4.172) and J2 puts Opus first (4.137 vs. 4.095); at the bottom J1 ranks GLM above Gemini 3.8 Flash (2.529 vs. 2.478), while J2 and J3 rank Flash above GLM. J3, whose vision judge seems sampled frames, leaves far more checks unknown (5.4–7.0% of delivered checks, against 0.6–1.7% for J1 and 0.2–0.6% for J2), which is why the “all core” rate of Tab. E.2, where an unknown counts against the task, is much lower under J3 and why strict success counts only failures. Even so, the panels differ widely in strictness (Opus: 83.7, 93.3 and 51.9% strict success under J1–J3), so only the three-panel mean is comparable across systems (App. § 10).

Uncertainty

Tab. E.3 gives bootstrap intervals for the headline score and compares each system with the next one. Adjacent pairs show how delivery and quality interact. Astra leads Opus ($P=0.91$; 127 vs. 89 head-to-head wins), and both are clearly ahead of DeepSeek ($P=1.00$). DeepSeek is ahead of Sonnet on the mean only through delivery (Sonnet wins 121 of the head-to-heads, DeepSeek 89; $P=0.70$), and Gemini 4 Argon wins 142 of 239 head-to-heads against GPT-5.6 Sol yet ranks below it, because its 23% non-delivery contributes zeros. Gemini 3.8 Flash and GLM are statistically tied ($P=0.50$). Fig. E.1 sets the intervals of the headline score beside those of the delivered-only score: in the same row order, DeepSeek drops below Sonnet and Gemini 4 Argon rises to fourth once missing videos no longer count.

Table E.2. Per-panel results (239 tasks; rows in the panel’s no-video = 0 order). nv=0: no video = 0. Task success: % of tasks whose core checks all pass or are n/a, with at least one pass (“all core”, unknown counts against), and strict success (no core check fails, at least one passes); no video counts as failure. Unk.: unknown share of delivered checks.

#	System	L2 pass (%) deliv.	L2 pass (%) nv=0	Task success (%) all core	Task success (%) strict	VQ	VA	CL	TM	AQ	Overall deliv.	Overall nv=0	Unk. (%)
1	GPT-6 Astra	98.4	98.2	76.6	83.7	4.00	4.06	4.07	3.87	3.80	3.963	3.946	0.6
2	Claude Opus 5.5	98.4	98.4	76.6	83.7	3.92	4.01	4.04	3.92	3.68	3.915	3.915	0.7
3	Claude Sonnet 5.5	95.2	92.5	67.8	73.6	3.79	3.92	3.97	3.83	3.62	3.826	3.712	0.7
4	DeepSeek-V4.1-Flash	91.6	90.5	49.0	54.4	3.64	3.82	3.78	3.50	3.60	3.667	3.620	0.9
5	Kimi K3	91.4	79.8	40.2	43.1	3.44	3.67	3.78	3.54	3.57	3.600	3.205	0.9
6	GPT-5.6 Sol	85.3	74.4	24.3	28.9	3.25	3.42	3.50	2.97	3.28	3.285	2.913	0.9
7	Gemini 4 Argon	96.8	74.0	50.2	54.8	3.55	3.76	3.96	3.68	3.72	3.734	2.874	1.0
8	GLM-5.3-Flash	83.8	60.9	25.1	29.7	3.26	3.46	3.50	3.30	3.41	3.385	2.529	1.7
9	Gemini 3.8 Flash	77.0	58.8	14.6	17.2	2.87	3.09	3.32	2.79	3.29	3.072	2.478	1.0
1	Claude Opus 5.5	99.4	99.4	90.4	93.3	4.16	4.15	4.14	4.14	4.10	4.137	4.137	0.2
2	GPT-6 Astra	99.0	98.8	91.2	92.5	4.14	4.14	4.11	4.11	4.07	4.112	4.095	0.4
3	DeepSeek-V4.1-Flash	97.6	96.5	77.4	80.3	4.16	4.14	4.10	4.10	4.08	4.115	4.063	0.4
4	Claude Sonnet 5.5	96.2	93.5	82.0	84.5	4.12	4.09	4.07	4.07	4.07	4.084	3.963	0.4
5	Kimi K3	95.1	83.1	64.4	66.1	4.05	4.01	3.98	3.96	4.00	3.999	3.562	0.4
6	GPT-5.6 Sol	96.2	84.1	58.6	60.7	4.04	4.01	3.94	3.87	3.94	3.963	3.523	0.4
7	Gemini 4 Argon	99.3	76.1	69.9	71.1	4.12	4.08	4.15	4.10	4.07	4.107	3.160	0.3
8	Gemini 3.8 Flash	87.9	67.3	45.2	46.0	3.75	3.70	3.70	3.46	3.80	3.681	2.968	0.6
9	GLM-5.3-Flash	90.2	65.8	47.7	49.0	4.02	3.97	3.87	3.92	3.90	3.935	2.943	0.5
1	GPT-6 Astra	93.9	93.7	27.6	47.3	4.08	4.71	4.43	3.88	4.42	4.303	4.285	5.9
2	Claude Opus 5.5	94.2	94.2	35.1	51.9	3.88	4.41	4.36	3.79	4.42	4.172	4.172	5.4
3	DeepSeek-V4.1-Flash	90.1	89.0	22.6	38.1	3.79	4.43	4.23	3.65	4.40	4.101	4.049	6.8
4	Claude Sonnet 5.5	90.4	87.7	26.4	42.7	3.87	4.35	4.27	3.68	4.26	4.087	3.965	6.7
5	Kimi K3	86.6	75.1	20.1	29.7	3.70	4.25	4.13	3.59	4.29	3.992	3.554	5.6
6	GPT-5.6 Sol	87.1	75.3	18.8	27.6	3.85	4.38	4.14	3.51	3.90	3.956	3.514	6.9
7	Gemini 4 Argon	93.4	70.3	22.2	39.7	3.73	4.29	4.33	3.80	4.47	4.125	3.171	6.6
8	Gemini 3.8 Flash	76.5	57.7	8.8	16.7	3.38	3.92	3.75	3.12	4.12	3.658	2.948	5.8
9	GLM-5.3-Flash	81.8	58.4	13.0	21.8	3.70	4.19	3.95	3.50	4.11	3.889	2.898	7.0

System	Mean	95% CI	Comparison	$P$	W/L/T
GPT-6 Astra	4.109	[4.052, 4.156]	Astra > Opus	0.911	127/89/23
Claude Opus 5.5	4.074	[4.037, 4.109]	Opus > DeepSeek	1.000	133/74/32
DeepSeek-V4.1-Flash	3.907	[3.826, 3.978]	DeepSeek > Sonnet	0.697	89/121/29
Claude Sonnet 5.5	3.878	[3.769, 3.977]	Sonnet > Kimi	1.000	152/68/19
Kimi K3	3.441	[3.267, 3.593]	Kimi > GPT-5.6 Sol	0.860	143/86/10
GPT-5.6 Sol	3.328	[3.174, 3.479]	GPT-5.6 Sol > Argon	0.980	82/142/15
Gemini 4 Argon	3.070	[2.854, 3.269]	Argon > Flash	0.979	159/55/25
Gemini 3.8 Flash	2.799	[2.609, 2.979]	Flash > GLM	0.498	79/139/21
GLM-5.3-Flash	2.796	[2.582, 3.007]

Quality profiles by dimension and category

Fig. E.2(a) shows that the systems keep their tiers on every dimension; only near-tied neighbors swap (e.g., Opus edges out Astra on TM). VA is the highest dimension for most systems and TM the lowest for all but Gemini 4 Argon. By category (Fig. E.2b; mean over the nine systems, no video = 0), Building & process is hardest (3.07) and spreads the systems most (best minus worst 2.66), because it demands long, stateful assembly; Visual art & form is next (3.15). Classroom lessons score highest (4.19) and spread least (0.88); every system averages at least 3.59 on them. Astra or Opus is best in 10 of the 12 categories, Sonnet on lessons and papers.

Program checks (L1)

System	Playable	Resolution	FPS	Duration	Frame count†	Audio‡	Project		All pass
GPT-6 Astra	99.5	98.5	96.6	98.0	98.1	100.0	96.1	190/205	92.7
Claude Opus 5.5	100.0	100.0	99.5	100.0	100.0	100.0	100.0	204/205	99.5
Claude Sonnet 5.5	96.6	96.6	96.6	94.1	92.6	98.0	95.1	192/205	93.7
Gemini 4 Argon	75.6	75.6	75.6	75.6	79.6	74.5	75.6	155/205	75.6
DeepSeek-V4.1-Flash	98.5	98.5	97.6	97.1	96.3	99.0	96.6	192/205	93.7
Kimi K3	87.8	87.8	87.8	84.9	87.0	86.7	87.3	172/205	83.9
GLM-5.3-Flash	70.7	70.7	70.2	64.9	64.8	67.3	69.3	127/205	62.0
GPT-5.6 Sol	87.3	85.4	84.9	86.8	88.9	88.8	86.8	169/205	82.4
Gemini 3.8 Flash	78.0	78.0	78.0	76.1	85.2	80.6	77.6	156/205	76.1

Among the 1,628 playable deliveries on the core tasks the most frequent program-check failure is duration (34), followed by a missing source project (20), frame rate (15; e.g., 24 instead of 30 fps, or drift such as 30.01), exact frame count (10), silent or absent audio where audio is required (10) and resolution (6). On the 34 lessons and papers, the rule-based checks (decodable MP4, resolution, frame rate, duration, audible audio, narration coverage by VAD, narration language by Whisper language ID, black or frozen stretches) fail in only 14 of 306 runs, 6 of them Gemini 4 Argon’s; 7 of the 14 are the runs without a playable video, the rest delivered videos failing a format or narration check. Program checks thus certify the container reliably.

Agent Behavior

This section covers all 2,151 runs (239 tasks × 9 systems).

Outcomes and effort

System	Self-ended	Timed out	Error	Video	Playable	Median s	Calls	Tool calls	Out. tok (k)	Reas.
GPT-6 Astra	229	10	0	238	238	2,464	51	49	60.9	49%
Claude Opus 5.5	238	1	0	239	239	2,283	51	49	130.7	56%
DeepSeek-V4.1-Flash	217	22	0	239	236	3,070	117	130	166.8	59%
Claude Sonnet 5.5	218	20	1	235	232	3,065	56	63	254.7	75%
Kimi K3	180	58	1	218	213	2,874	63	80	75.6	51%
GPT-5.6 Sol	147	90	2	224	213	3,370	109	101	129.7	57%
Gemini 4 Argon	177	60	2	209	184	2,994	46	68	125.5	28%
Gemini 3.8 Flash	184	55	0	224	193	2,935	122	118	106.9	41%
GLM-5.3-Flash	160	79	0	193	179	3,234	64	83	103.3	56%

Strong systems finish with time to spare. Opus and Astra usually end well before the hour (Fig. F.1b), while GPT-5.6 Sol and GLM often run into the deadline; Sol has usually saved a playable video by then, GLM often has not. Within every system longer runs score lower. The harnesses shape how agents work (Fig. F.1a): Claude Code and Codex agents act almost entirely through the shell, Gemini, Kimi and GLM rely more on dedicated file tools, and Gemini 3.8 Flash spends much of its time waiting on background tasks. Sub-agents and web search are rarely used; installing packages is common.

Rendering stacks

System	cairo	PIL/numpy	Browser 2D	three.js	Blender	Other	ffmpeg	Synth. audio
GPT-6 Astra	25	16	23	29	7	0	100	93
Claude Opus 5.5	54	5	15	20	4	2	100	96
DeepSeek-V4.1-Flash	24	12	36	24	4	0	100	95
Claude Sonnet 5.5	46	10	22	19	2	1	100	89
Kimi K3	3	45	26	23	3	0	97	85
GPT-5.6 Sol	4	55	13	13	14	2	99	57
Gemini 4 Argon	42	20	17	18	3	1	97	93
Gemini 3.8 Flash	31	13	21	32	2	1	100	84
GLM-5.3-Flash	11	43	21	21	3	1	96	86
Pooled over the nine systems, by primary renderer
Runs	576	524	463	473	95	20
Playable (%)	94	90	94	82	83	70
Timed out (%)	9	22	14	28	31	15
Overall, no video = 0	3.79	3.49	3.73	3.00	3.12	2.88
Overall, delivered	4.02	3.88	3.98	3.67	3.75	4.11

The pipeline is always “code → frames → ffmpeg”, and each system has a house style (Tab. F.2): the Claude agents and Gemini 4 Argon draw vectors with cairo, GPT-5.6 Sol, Kimi and GLM rasterize frames with PIL/numpy, and Astra, DeepSeek and Gemini 3.8 Flash capture browser canvases and three.js scenes. Off-the-shelf video frameworks such as Manim or MoviePy are almost never chosen. Vector and 2D-canvas pipelines deliver most often and score highest; 3D scenes, rendered with software WebGL, are the slowest and score lowest. Since renderer and system are coupled, this is an association.

Self-inspection

Table F.3. Self-inspection. An image view is a tool call that returns an image to the model. Extract/run: shell commands pulling stills or contact sheets from a rendered video. Last 25%: share of views in the final quarter of the budget. Right: mean delivered score of playable runs by number of image views (“–”: no runs; parentheses: fewer than 5 runs).

System	≥1 view	Median	Mean	p90	Extract/run	ffprobe/run	Runs ffprobe	Last 25%	0	1–4	5–14	15–39	40+
GPT-6 Astra	229/239	3	3.4	6	3.1	2.1	202	22%	4.27	4.13	4.08	–	–
Claude Opus 5.5	238/239	7	7.9	12	4.6	3.9	237	11%	(3.47)	3.97	4.09	4.24	(3.80)
DeepSeek-V4.1-Flash	239/239	35	36.0	56	6.9	8.0	238	10%	–	(4.27)	3.53	3.91	4.07
Claude Sonnet 5.5	229/239	10	14.3	28	4.7	4.7	232	24%	3.27	3.77	3.94	4.20	4.24
Kimi K3	239/239	17	18.8	31	1.9	2.7	214	38%	–	3.29	3.84	3.90	4.05
GPT-5.6 Sol	231/239	6	6.8	13	2.8	4.6	222	20%	3.63	3.77	3.72	3.76	–
Gemini 4 Argon	235/239	18	18.3	29	0.3	1.3	182	27%	–	(4.04)	3.97	4.00	(3.85)
Gemini 3.8 Flash	182/239	6	6.5	14	0.4	3.4	218	19%	3.23	3.42	3.57	3.57	–
GLM-5.3-Flash	230/239	14	14.4	26	1.5	2.4	196	15%	(2.17)	3.07	3.69	3.89	–

Almost every run looks at its own rendered frames, but how much it looks does not predict delivered quality across systems: Astra looks least and scores best. Within systems, looking more goes with better videos for some (Sonnet, Kimi, GLM, Gemini 3.8 Flash) and not for others. Final reports often claim verification that does not hold: for videos that fail spec compliance, the report still says the video was “verified” in most of Astra’s cases and in a minority of the others’.

Failure taxonomy

Table F.4. Why videos are not delivered. Top: main cause of each of the 224 runs without a playable video, assigned by reading each run’s records. Bottom: state at the end of these runs.

Cause or failure mode (runs)	GPT-6 Astra	Opus 5.5	DeepSeek	Sonnet 5.5	Kimi K3	GPT-5.6 Sol	Gem. 4 Argon	Gem. 3.8 Flash	GLM-5.3-Flash	Total
Time management	1	0	2	7	25	25	31	22	56	169
Final render started late or ran too slowly	1	0	2	5	14	18	20	19	25	104
Final render never started	0	0	0	2	11	7	11	3	31	65
Stalled or cut-off render	0	0	0	0	0	0	21	24	0	45
ffmpeg pipe deadlock (stderr never drained)	0	0	0	0	0	0	19	14	0	33
Turn ended, background render cut off	0	0	0	0	0	0	2	10	0	12
Own output destroyed or lost	0	0	1	0	1	1	1	0	3	7
Other	0	0	0	0	0	0	2	0	1	3
Provider content filter	0	0	0	0	0	0	0	0	1	1
Request over the 20 MB size limit	0	0	0	0	0	0	2	0	0	2
No playable video	1	0	3	7	26	26	55	46	60	224
State at the end of these runs
Truncated MP4 (no moov atom)	0	0	3	3	5	11	25	30	13	90
MP4 without a video stream	0	0	0	0	0	0	0	1	1	2
No MP4, timed out	1	0	0	3	21	15	26	9	38	113
No MP4, agent ended its turn	0	0	0	1	0	0	2	6	8	17
No MP4, run ended on an error	0	0	0	0	0	0	2	0	0	2

Tab. F.4 extends § 5.2. Writing renders to a temporary file would not have saved most truncated MP4s, since they mostly come from stalled or unfinished renders rather than from overwriting a finished video.

Cost

The mean list-price API cost per task ranges from $0.32 (GLM-5.3-Flash) to $15.38 (Claude Opus 5.5), a 49× spread. Across the nine systems its rank correlation with the headline score is $\rho=0.55$ (permutation $p=0.13$; 0.43 with the delivered-only score). The cost–score Pareto frontier is GLM → DeepSeek → Astra (Figure 5b). Mean output tokens per task range from 61k (Astra) to 289k (Sonnet); Tab. F.1 reports the lower per-run medians. The total is about $12.8k. 76% of Opus’s cost is cache writes (2.3M tokens written per task at a 65% cache-hit rate), and 55% of Sol’s is uncached input. Some prices are promotional: at the announced regular prices Gemini 4 Argon would cost about $5.9 and Gemini 3.8 Flash about $4.9 per task.

System	Mean	Median	p90	Max	Total (239)	Calls/task	Cache hit (%)	$/playable video | Overall | Overall per $
GPT-6 Astra	8.19	7.88	11.88	15.61	1,957	46	97	8.22	4.109	0.50
Claude Opus 5.5	15.38	10.19	31.22	47.08	3,676	51	65	15.38	4.074	0.26
DeepSeek-V4.1-Flash	0.47	0.45	0.67	1.30	113	117	97	0.48	3.911	8.25
Claude Sonnet 5.5	8.26	7.09	13.07	34.99	1,974	63	81	8.51	3.880	0.47
Kimi K3	3.53	3.53	4.94	7.25	843	61	97	3.96	3.440	0.98
GPT-5.6 Sol	11.84	11.49	19.59	28.39	2,831	106	81	13.29	3.317	0.28
Gemini 4 Argon	2.94	2.84	4.32	7.13	703	48	89	3.82	3.069	1.04
Gemini 3.8 Flash	2.44	2.09	4.16	6.66	584	126	91	3.03	2.798	1.14
GLM-5.3-Flash	0.32	0.30	0.45	0.68	76	65	93	0.42	2.790	8.83

Human Preference Study

Protocol. Annotators compare two videos made by two systems for the same task. Both videos are playable and judged by all three panels, so the study measures the quality of delivered videos, not delivery. System pairs are balanced across tasks and the sides are randomized; annotators are blind to system identity and judge scores. The interface shows the full prompt and two synchronized players with a choice of audio track, and offers A better, B better, tie and both bad; the instruction is “which better matches the prompt and is better overall”. We fit Bradley–Terry strengths on decisive votes (MM algorithm with a weak prior; Elo scale with geometric mean 1000), with 2,000-sample bootstrap CIs over decisive pairs and Wilson intervals for win rates.

System	Win rate [Wilson 95%]	BT-Elo [95% CI]	Rank CI	Overall, delivered (rank)
GPT-6 Astra	0.783 [0.644, 0.877]	1242 [1134, 1387]	1–3	4.126 (1)
Claude Opus 5.5	0.741 [0.616, 0.837]	1203 [1111, 1327]	1–3	4.074 (2)
Claude Sonnet 5.5	0.676 [0.508, 0.809]	1121 [1016, 1257]	1–5	3.999 (3)
Gemini 4 Argon	0.607 [0.476, 0.724]	1081 [1001, 1176]	2–6	3.989 (4)
DeepSeek-V4.1-Flash	0.511 [0.370, 0.650]	1022 [900, 1154]	3–7	3.961 (5)
Kimi K3	0.449 [0.319, 0.587]	997 [873, 1107]	4–7	3.864 (6)
GLM-5.3-Flash	0.375 [0.252, 0.516]	905 [792, 1012]	5–7	3.736 (7)
Gemini 3.8 Flash	0.170 [0.089, 0.301]	723 [548, 833]	8–9	3.470 (9)
GPT-5.6 Sol	0.156 [0.077, 0.288]	705 [516, 814]	8–9	3.735 (8)

Three tiers. Astra and Opus lead and cannot be separated (+39 Elo, $P=0.69$); both are significantly above DeepSeek, Kimi and GLM, and Astra also above Argon. Sonnet, Argon, DeepSeek, Kimi and GLM form a middle group whose neighbors cannot be separated. Gemini 3.8 Flash and GPT-5.6 Sol share the last place (+18 Elo, $P=0.58$) and are both significantly below the other seven.

Humans validate quality, not delivery. The human ranking is almost the delivered-only judge ranking (Spearman $\rho=0.98$, Kendall $\tau=0.94$; only Flash and Sol swap the last two places), and it agrees less with metrics that include delivery (Fig. H.1, Tab. H.2). With no video = 0, Argon falls from 4th to 7th and Sol rises from 9th to 6th, because humans never see a missing video. The study thus validates the quality part of the score; delivery is reported separately as the delivery rate.

Judges do not flag “both bad”. On pairs that humans labeled both bad, the panels’ mean score (3.81) equals that on decisive pairs (3.83), and only 4% of such pairs have both videos below 3.5. Judge scores order videos but do not certify them as acceptable.

Metric	Spearman	Kendall
Overall, delivered only	0.98	0.94
L2 pass rate (delivered)	0.92	0.78
L2 strict success	0.92	0.78
Overall, no video = 0	0.77	0.61
Playable-delivery rate	0.66	0.47

Ablations

Four ablations, each run once per cell after the main evaluation, ask whether the scores can be trusted and whether the setting limited the results:

• 🔎 Can the judges see defects? (§I.1)

• 🎓 Do the scores reflect whether a video actually teaches? (§I.2)

• ⏱️ Was one hour enough? (§I.3)

• 🖥️ Did the agents need more CPU cores? (§I.4)

Judge stress test

We took 30 delivered videos, re-encoded each one cleanly and built ten defective versions with ffmpeg: five picture defects in the window from 35% to 65% of the runtime (black frames, a grey box over the center, heavy blur, a frozen picture) or over the whole video (left–right mirror), two sound defects (a silent window, another task’s soundtrack), audio delayed by 0.5 or 1.5 s, and the two halves swapped. Apart from the black window, every defective version passes all program checks, so only a judge can notice it. J1 judged each video in its clean version and in five of the defects (15 videos per defect, 180 judgments) with the unchanged prompt and rubric, and was not told that anything was injected.

Judges see broken pictures but not late sound (Fig. I.1). Every picture defect, the foreign soundtrack and the swapped halves are named in all 15 reports and cost 1.5–1.9 points on the related dimension; a silent window is named in 7 of 15 (−0.80 on AQ). Neither audio delay is mentioned in any report, and reports often praise the synchronization; the five-dimension mean drops only slightly (−0.11 and −0.16; Wilcoxon $p=0.07$ and 0.05), spread over several dimensions rather than AQ. Detection does not always reach the checks: mirrored and reordered videos are named in every report, yet most of their target checks still pass (flip rate +4.5 and +14.9 points over clean), and the penalty lands on VQ and CL instead. Finally, the clean re-encode scores +0.12 [+0.03, +0.22] above the original judgment of the same video, so gaps of 0.1–0.2 between single videos are within judge noise. Audio–visual timing is thus a blind spot of both the judges and the program checks, so our rubrics set no strict offset limit: a synchronization check asks only that the right sound accompanies the right event (App. § 8), not that audio and picture align within a fixed tolerance.

Do the videos teach? A grounded quiz

Each of the 34 lessons and papers has a multiple-choice quiz written from its source material (404 questions). A plain multiple-choice quiz is not enough: a strong model can answer many of these questions from prior knowledge without watching the video, so every answer must point to the moment in the video that supports it. A grader (Gemini 4 Argon) watches only the finished video and answers A–D or E (“the video does not cover it”). Every A–D answer must give a timestamp and the video’s own words; narration quotes are matched against a Whisper transcript and on-screen quotes against the frames, and an answer whose evidence does not check out counts as E. A task’s score is its share of correct answers; a run without a video scores 0. The quiz does not enter the leaderboard.

The quiz agrees with the judges. Although it uses neither the rubric nor the quality anchors, its system ranking correlates with the three-panel overall at Spearman $\rho=0.93$ (Fig. I.Figure 2; also 0.93 on playable videos only). The top four systems convey 93–96% of the tested points; Gemini 4 Argon’s 77.4% comes from its five tasks without a playable video (90.8% on the other 29). Almost no point is lost to a wrong answer (2 of 3,425 A–D answers); the losses are questions the video does not cover or whose evidence does not check out, so the quiz measures coverage of the material more than correctness.

Time budget

We reran DeepSeek-V4.1-Flash, Gemini 3.8 Flash and GLM-5.3-Flash on 20 tasks with budgets of 15, 30, 60 and 120 minutes (one run per cell, otherwise as in the main evaluation). Runs are judged by J2 with no video = 0 and compared with the same system’s 60-minute arm by a paired Wilcoxon test.

System	Budget	Score	$\Delta$ ($p$)
DeepSeek-V4.1-Flash	15 min	2.89 (16)	−1.03 (0.006)
DeepSeek-V4.1-Flash	30 min	3.80 (20)	−0.13 (0.11)
DeepSeek-V4.1-Flash	60 min	3.92 (20)	–
DeepSeek-V4.1-Flash	120 min	3.94 (20)	+0.01 (0.78)
Gemini 3.8 Flash	15 min	1.84 (10)	−1.67 (0.001)
Gemini 3.8 Flash	30 min	2.72 (15)	−0.79 (0.13)
Gemini 3.8 Flash	60 min	3.51 (19)	–
Gemini 3.8 Flash	120 min	3.61 (19)	+0.10 (0.32)
GLM-5.3-Flash	15 min	0.00 (0)	−2.44 (<0.001)
GLM-5.3-Flash	30 min	1.16 (6)	−1.28 (0.017)
GLM-5.3-Flash	60 min	2.44 (13)	–
GLM-5.3-Flash	120 min	2.98 (16)	+0.54 (0.25)

The budget decides whether a video is delivered, not how good it is (Fig. I.Figure 3, Tab. I.1). From 15 to 120 minutes, playable videos rise from 16, 10 and 0 of 20 (DeepSeek, Flash, GLM) to 20, 19 and 16. Except for DeepSeek at 15 minutes (3.61 against 3.92 at 60), the mean over playable videos moves by at most 0.17 within a system (DeepSeek 3.80–3.94, Flash 3.63–3.80, GLM 3.73–3.87), with overlapping CIs.

How much time is enough differs between systems. DeepSeek and Flash gain little once the budget reaches that of the main run, while GLM delivers more videos at every larger budget (0, 6, 13 and 16 of 20; Tab. I.1). Flash tends to end its runs on its own well before the deadline, whereas DeepSeek and GLM use the extra time they are given; some GLM runs are still cut at the deadline even with the longest budget, and a single GLM call can spend a long time reasoning before issuing any tool call.

CPU budget

We reran six systems (Gemini 4 Argon, Gemini 3.8 Flash, DeepSeek-V4.1-Flash, GLM-5.3-Flash, Kimi K3, GPT-5.6 Sol) on the same 20 render-heavy tasks with 2, 8 and 16 CPU cores; the main evaluation uses 8. Everything else is as in the main run; scores are the three-panel mean with no video = 0 (360 runs), compared over the 120 task–system pairs.

More cores do not measurably help (Fig. I.Figure 4). None of the pooled differences is significant (+0.09 for 8 vs. 2 cores, +0.13 for 16 vs. 8, +0.22 for 16 vs. 2; $p$=0.57, 0.44, 0.20; a bootstrap over tasks gives the same conclusion), and playable videos go from 99 to 101 and 103 of 120 (sign tests $p\geq0.56$). The systems disagree in direction (Kimi and Flash do best with eight cores, Sol with two, DeepSeek is flat), and delivered quality hardly moves (3.81, 3.84, 3.92). Argon’s rise from 9 to 12 videos comes mostly from fewer ffmpeg pipe deadlocks in the 16-core arm (2 of its 8 failures there, against 9 of 21 in the other two), which barely use the CPU; render-heavy tasks still run out of time with sixteen cores. The CPU budget therefore does not measurably limit the main results.

Limitations and Broader Impact

Limitations.

Fixed budget. Every run gets one hour. Longer budgets, which may let agents make better videos, are left to future work.

Adherence over creativity. The rubrics mainly measure whether a video follows its specification. Creativity and originality enter only partly, through the aesthetics dimension (VA), and are not evaluated on their own.

Judge dependence. All panels are proprietary models, and the judges are insensitive to audio-visual timing (App. § 15.§ 1). Rankings are stable across the three panels, but absolute values are panel-specific and are never mixed.

Coverage. The task set is mostly English (209 of 239) and limited to 1080p-class outputs. Tasks with very long runtimes are few.

Broader impact. Code-driven video makes precise, editable and accessible content (lessons, data stories, instructions) cheaper to produce, and our findings point to concrete reliability improvements. Risks include persuasive marketing content and misinformation rendered with an appearance of precision. Tasks with factual content ship their facts as inputs and forbid fabricated claims (e.g., “do not claim a real interception probability”). No task depicts real private individuals. Classic-game tasks use newly designed levels and assets in a recognizable style for research evaluation; we will not redistribute original game assets.

Selected Cases

One case per category, each on one page: a frame from one delivered video (the system is named), the opening of the task specification, the supplied assets, the core goals with the number of L2 checks under each, one L2 check, and the five L3 dimensions with the anchor for a score of 5. Fail conditions are shown where the rubric names specific failures; the L3 scale and the shared observation points are in App. § 8.

Brand & marketing (C01)

Ember Atlas: An Old TV Gateway to Three Adventure Worlds

Goal	Core goal	L2	core
G1	Old TV interior/exterior relationship and opening	3	3
G2	Actions and outcomes of the three gameplay modes	8	5
G3	Television array and its initial takeoff	4	4
G4	Fixed release card and original (fictional) identity	4	4
G5	Music and action sounds	8	6
G6	Complete delivery	0	0
All L2 checks		27	22

Task specification — opening of 612 words

Ember Atlas: An Old TV Gateway to Three Adventure Worlds

For viewers who enjoy fantasy adventure games, create a fictional game trailer in which the old television visual motif connects three distinct gameplay types, culminating in a complete product launch.

Task type: structural-constraint task; structure: single-screen suspense — three gameplay previews — television array — cosmic product reveal.

Content inputs

The game “EMBER ATLAS” is a fictional work for this task; do not use characters or battle footage from the original film “Tales of Destiny”, or the PlayStation logo. The protagonist is a short-haired explorer wearing a red scarf and always carries a blue lantern. All footage must be self-drawn; pixel-art environments with simple 3D television shells may be used.

Three preview worlds and required actions: ① Twilight Ruins: the protagonist uses the lantern to illuminate three symbols on a door, and the stone door opens; ② Floating Islands: the protagonist jumps across a broken bridge, and the blue lantern lights up at the landing point; […]

Supplied assets. None; everything is made from the specification.

Example L2 check — G2 · core · one of 27 checks

Pass In Dark Arena, a circular red attack warning first appears and shrinks; the protagonist dodges outside the circle and then hits the glowing point on the statue’s chest.

Prompt “③ Dark Arena: a circular red attack warning contracts, the protagonist dodges outside the circle, then strikes the bright point on the stone statue’s chest.”

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

caption:

:::

Product & feature demos (C02)

Branchboard: See One Color Change Clearly

Task specification

Create an exactly 15-second promotional video in an app/SaaS style for the fictional app Branchboard, a visual tool that helps designers understand Git version changes. Use a light visual style with a #F7F8FA interface, black and gray text, blue and purple accents, dramatic cuts, kinetic typography, and a dynamic Apple-style product presentation. All interface elements must be self-drawn.

The audience is designers who want to understand Git version changes. Use one clear design change to show a verifiable sequence of branch creation, modification, commit, comparison, and merge. The source data consists only of the accent field in the supplied file and one webpage Hero preview whose title remains unchanged.

Begin with the dynamic text “Design it. Version it.” and a blue preview. The initial state is main pointing to c101, accent=#246BFD, a blue preview button, and a clean worktree. Show the user clicking the “New branch” control, creating a branch named palette-test from c101, and selecting it. In the design panel, change accent to #7C3AED; the button becomes purple and the status becomes “1 change”. Show the user clicking the “Commit” control to commit the change, producing c102 with the description “Update accent”; palette-test points to c102 while main still points to c101.

Show a Compare panel containing only the accent change from blue to purple and the corresponding preview comparison; do not add unrelated files. Then show the user clicking the “Merge to main” control to merge palette-test to main. Because main has no other changes, the merge must fast-forward to c102; do not create a third merge commit. [. . . ]

Supplied assets. None; everything is made from the specification.

Example L2 check

Pass The same button or the same Hero card serves as the persistent visual anchor throughout the video; the preview button is blue in its initial state and purple after the color change, corresponding to accent in the field and Compare; no particular button instance is mandatory

Prompt “Use the same button or the same Hero card as a visual anchor throughout the video so that the color change can be found in the preview, the field, and the comparison.”

L3 quality rubric

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

Events & personal expression (C03)

🎉 ARI LIN: A 15-Second Motion Resume That Proves Each Item Before Checking It

🎉 Task specification                                                                 opening of 1,774 words

Create a 15-second motion-resume video for viewers browsing a code/motion-design portfolio. Use one resume to show actual visual proof of five capabilities, checking each item only after its proof is complete, and finally return to the complete identity page with the portrait and all checks. The resume identity is the original fictional character ARI LIN, with the role line MOTION SYSTEMS; do not add fictional employers, awards, or career achievements. The five rows, in order, are SKILLS, TYPE, SYSTEMS, DEPTH, and DATA. Use the original Bit Finch pixel character in the final photo frame; create and provide its 16x16 matrix and reference assets with the editable project, and use the same character for the later grid and photo frame. Build the whole piece from a dark page, a paper resume, and orange accent elements passed through the transitions; do not use corner labels or fictional HUD numbers.

[…]

Clock and overall rhythm

The entire film is 128 BPM, 4/4 time, 8 bars, 32 quarter beats, and 15 seconds long. Each quarter beat is 0.46875 seconds, and each bar is 1.875 seconds.

[…]

📦 Supplied assets (43 files, 2.0 MB), read-only in the sandbox:

✅ Example L2 check                                                                 G_STORY · core · one of 46 checks

Pass     From b2.5 to b4, continuously push in from the original paper orange dot, until orange fills the screen at b4 and connects to the meter fill; there must be no switch to an unrelated background during the entire push-in, and correct endpoints alone cannot prove a continuous handoff.

Prompt     “At b2.5, push the camera into the orange dot; by b4, orange fills the frame and connects into the skill-meter fill. Do not cut to unrelated background during the push.”

🏆 L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

Concept & pipeline explainers (C04)

Goal	Core goal	L2	core
G1	One complete machine unit across the four segments	4	3
G2	Eight-legged closed chain, drive and leg phases	23	22
G3	Support and displacement, elevated and on the ground	16	15
G4	Material, identity and expression boundaries respected	16	12
G_DELIVERY	Complete delivery: specified media parameters, editable project and task-specified attachments	0	0
	All L2 checks	59	52

💡 Task specification — opening of 1,261 words

How Wood Walks: Transmission and Foot Placement of an Eight-Legged Walker

For general viewers interested in mechanical motion who have no background in linkage analysis, use a complete machine unit to answer how a continuously rotating motor makes a wooden machine take alternating steps, with the transmission, joints, and ground support visibly corresponding on the same machine.

Format: structural-constraint task; structure: observe walking on the floor—observe the leg cycle while elevated—trace the drive along the motor and gears—return to the floor to verify.

Content input

Use only the complete 01:13—01:50 walker unit retained from the original compilation, and independently produce a 37—45-second 3D short; do not mix in other electronic products, software tutorials, or the original channel intro. Preserve the complete structure of floor walking, elevated-base observation, motor/gear/bearing/yellow-motor close-ups, and return to the floor; the original footage is not a production input.

Use an unbranded eight-legged wooden walker; do not claim to reproduce the original clip’s model or exact CAD. Use the supplied input data for all rod lengths, node connections, circle-intersection branches, gear phases, and leg phases. Model all scenes and mechanical parts independently; plywood texture may be procedural, with no external photos or models needed.

The machine has four jointed legs on each side, a single through-frame chassis, four transverse crankshafts, a small yellow motor, a visible rear cable, plywood gears, and metal pins/bearings. Keep stable leg IDs L1—L4 and R1—R4. [. . . ]

📦 Supplied assets (2 files, 9 KB), read-only in the sandbox:

🧮 Data 2 the walker’s linkage (coordinates, bar lengths, fixed pivots, moving joints, rigid triangles) and a manifest

✅ Example L2 check                                                                                                 G2 · core · one of 59 checks

Pass Each connecting rod changes its angle around the actual pin shaft, with no rubber tube bending or stretching substituting for joint motion.

Prompt “Use the input’s Jansen-style closed chain: two rigid triangular pieces and connecting rods change relative angles around pin joints; do not deform them like rubber tubes or let rod ends leave their joints.”

🏆 L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

VQ Visual quality	The same transmission chain and foot ends remain discernible throughout.
VA Visual aesthetics	The plywood-layer hole edges and metal bearings form a clear mechanical hierarchy throughout.
CL Content logic	The distinction between walking on the ground and being airborne, and the relationship from drive to foot end, remain clear throughout.
TM Timing & motion	Rigid rods and pins move compatibly, and the feet’s high return and low backsweep remain trackable throughout.
AQ Audio & A/V sync	The sound actually used never interferes with mechanical understanding.

Data & fact stories (C05)

Fifteen-Second Boomerang: LLM History and Imagined RSI

Goal	Core goal	L2	core
G1	Three historical stages defined by the content clock $u$: $u=0$–1.5 seconds is 2003 Neural LM, $u=1.5$–3 seconds is 2017 Transformer, and $u=3$–4.5 seconds is 2020 GPT-3 [. . .]	9	9
G2	At $u=4.5$, when entering the future, a clear dashed boundary, FICTION, and RSI? appear; retain the question mark. RSI here is hypothetical feedback for recursive self-improvement [. . .]	7	7
G3	The complete forward progression runs from 0–7.5 seconds to the climax [. . .]	8	7
	All L2 checks	24	23

Task specification	opening of 830 words
Pixel-art boomerang render of the history of LLMs. Then RSI. Then it goes fucking crazy. Create a 15-second pixel-art boomerang video for viewers with a basic interest in AI development. It should sequentially present milestones in LLM history, a fictional recursive-self-improvement feedback escalation, and a complete rewind back to the opening state.
Content and visuals
Select three representative milestones to give an overview of LLM prehistory, a key architecture, and scaling; do not claim to exhaust the history: 2003 Neural LM, 2017 Transformer, 2020 GPT-3. The 2003 segment depicts the prehistory of early neural language models, not the appearance of today’s large models; 2017 depicts the Transformer architecture; 2020 depicts GPT-3 and its scaling phase. Do not treat the originally titled Opus5.5 as a production version for this task or as a historical milestone. The word blocks from each preceding milestone must continue transforming in the next; do not make three unrelated logos.
In the future segment, define RSI as Recursive Self-Improvement: an imagined feedback loop in which a system improves its own capabilities. The model proposing a plan, testing it, and forming the next version is only an artistic hypothesis; do not claim that autonomous unlimited intelligence growth has been achieved. When the forward content clock $u$ reaches 4.5 seconds, FICTION and RSI? must clearly appear, with the question mark retained. During the rewind, as $u$ falls below 4.5, FICTION and RSI? must exit with the original transition; do not leave them permanently visible based on finished-video time $t \ge4.5$. [. . .]

Supplied assets (2 files, 11 KB), read-only in the sandbox:

Example L2 check	G1 · core · one of 24 checks
Pass	The 2020 same-grid board expands, and after the example card enters, a new text block continues to emerge, expressing in-context few-shot; parameters are not changed during the demonstration
Fail	The animation trains the prompt example into the weights; only the name GPT-3 appears with no continuation process; or unprovided capability rankings are added
Prompt	“Expand the same grid; after the example card enters, continue with new text blocks; do not animate parameter changes to represent few-shot.”

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

VQ Visual quality	The key graphics from characters to city/nebula are clear and stable; the fictional and replay labels are discernible in the specified time windows.
VA Visual aesthetics	The cyan, orange, and purple visual language runs through the technical nodes, city, and nebula, with changes in form and layering serving a clear expressive purpose.
CL Content logic	Three real technology anchors lead to clearly labeled future speculation, with feedback and replay forming an understandable chain of imagery.
TM Timing & motion	The 7.5-second forward motion and 7.5-second reverse motion return precisely along the same clock; states, labels, and the final fade-out form a clear rhythm.
AQ Audio & A/V sync	The music/sound effects actually used support the forward/reverse imagery and do not interfere with the loop; when silence is permitted and actually used, AQ is N/A; pronunciation is not evaluated when there is no speech.

Visual art & form (C06)

Painting the Night: A 50-Second Image in Paper, Graphite, and Moonlight

Goal	Core goal	L2	core
G_MATERIAL	First establish visible warm-colored blank paper and fine-grained fibrous paper texture / Fix the paper texture to the same sheet, transforming consistently with the entire sheet [. . . ]	9	9
G_MEANING	The same positions on the given paper surface as T1 through T7 become 7 main stars / The 6 connections follow T1→T2→T3→T4→T5→T6→T7 / The main stars and background stars have hierarchy [. . . ]	11	11
G_AUDIO	There is complete 50-second original creative music, with soft percussion onset, piano-key pitch, and low/high dynamics [. . . ]	5	5
	All L2 checks	25	25

Task specification

Create a 50-second generative-art short for a general audience interested in generative art and short films, titled Painting the Night. On a single sheet of paper, continuously show a quiet sense of wonder developing from tentative marks into a complete night sky: materials accumulate, graphite imagery becomes a constellation, moonlight completes the sky, and the film settles into a quiet afterglow. Use zero image assets; paper grain, graphite hatching, the appearance of pigment, stars, the constellation, moonlight, and felt-piano music must all be generated at runtime. Implement the primary visuals with native JavaScript and WebGL.

Create one night sky on one sheet of paper: first show warm blank paper, then let tentative graphite hatching and a few dots accumulate on that same sheet; indigo pigment advances along irregular wet edges while the original paper grain and some graphite remain visible; the early dots become stars, and their existing connections become an original constellation; the early reserved circle gradually becomes a luminous moon, whose light affects the surrounding pigment and paper; finally show the complete painting and settle quietly. The key is the transformation of materials and meaning within the same motif. Do not present the paper grain, particles, and glow as three separate images pasted in one after another. No characters, conflict, competition, or scientific instruction are required.

[. . . ]

Geometry and identity: The paper design rectangle is [160,90,1600,900], with UV increasing rightward and downward; measure circular distances in units normalized to paper height, and account for the horizontal 1600/900 ratio; [. . . ]

Supplied assets (14 files, 41 KB), read-only in the sandbox:

• Data 9 — art direction (stages, palette, text), paper and constellation geometry, music, measurement and runtime contracts, and sources

• Scripts 4 — optional checkers and an offline renderer

• Notes 1 — run notes

Example L2 check

L3 quality rubric

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

Games & sports (C07)

Plants vs. Zombies: One Lawn Battle from 2D into 3D

Frame at 12.9 s of the video delivered by Kimi K3.

Goal	Core goal	L2	core
G1	Establish the classic five-row, nine-column daytime lawn with the specified initial formation, enemy identities, and gaps.	6	6
G2	Complete P5 and CB1 through genuinely visible card selection, collection, and planting, strictly following the single sun ledger.	8	8
G3	Make same-lane attacks, zombie movement, contact damage, and Wall-nut blocking form a traceable battle.	7	7
G4	Transition from 2D to actual 3D within the same ongoing battle, and continue the identities, states, and clock.	9	9
G5	Present the Wall-nut being chewed, three-lane gathering, effective Cherry rescue, and attacks clearing enemies under 3D pressure.	8	8
G6	Conclude with a complete victory in this wave, with all 8 enemies eliminated and the residence unbreached.	3	3
G7	Support gameplay with the event sounds required by the prompt, while preserving freedom to include optional music and narration.	9	8
G8	Deliver an MP4 with sound in the specified specifications, an editable code project, the local assets used, a timeline, and rendering instructions.	0	0
All L2 checks		50	49

Task specification

Plants vs. Zombies: One Lawn Battle from 2D into 3D

Creative brief

Use code to create a 90–120 second Plants vs. Zombies gameplay expansion video. Build around the classic daytime five-row, nine-column lawn, Sunflowers, Peashooters, Wall-nuts and Cherry Bombs, with regular, Conehead and Buckethead Zombies. Establish an active multi-lane defense in a clear 2D view, then transform that same ongoing battle into a 3D battlefield with visible volume and spatial relationships. Continue the defense to its conclusion so viewers recognize the classic gameplay and understand that the battle did not restart. [. . . ]

Supplied assets (141 files, 3.8 MB), read-only in the sandbox:

Example L2 check

Pass First use 100 of 125 to plant P5, leaving 25; the five specified sun collection events in this round occur after that charge.

Fail After planting, 100 was not deducted, the required sunlight was collected before payment, the cherry was planted first, or the balance cannot be matched to the actual operation.

Look at From before P5 is selected until the first specified sunlight collection.

Prompt “First spend 100 to add the Peashooter, collect five sun tokens to reach 150, then plant the Cherry Bomb and finish at 0.”

L3 quality rubric

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

Story & visual comedy (C08)

Goal	Core goal	L2	core
G1	Independent dual storylines and the causality of failure	12	11
G2	Identity, injuries, and prop states	31	31
G3	Three-dimensional building-block action and space	20	19
G4	Ambush, disarmament, and everyone’s capture	23	23
G5	Actual complete delivery	0	0
All L2 checks		86	84

Supplied assets (7 files, 43 KB), read-only in the sandbox:

• Data 4 characters, hostiles, props, scene and events; a ledger of the survivors’ states; font provenance; a manifest

• Font 1 a monospaced display font

• Notes 2 font licence and usage notes

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

VQ Visual quality	The block identity and left/right injuries remain discernible and effectively support this task’s purpose.
VA Visual aesthetics	The plastic-stage atmosphere in the snow and camp consistently and effectively supports the purpose of this task.
CL Content logic	The causality behind the convergence of the two storylines and the outcome of failure consistently and effectively supports the purpose of this task.
TM Timing & motion	Joints/climbing/bone-staff weight-bearing and handoff consistently and effectively support the purpose of this task.
AQ Audio & A/V sync	The sound actually used is clear, with natural loudness and timing; there is continuous positive evidence that optional sound supports threat and brief relief; do not gain points by adding more sound layers.

Building & process

Embed Voxel Explosions into a Real Street: From 15 TNT Blocks to Two Road Craters

Task specification (opening of 1,486 words)

Create a Minecraft-style voxel animation that integrates block construction and destruction into a real street scene. Use the street photograph in the input package as the full-frame background. Stack 15 TNT blocks one by one on the foreground road, build a tower from gold and diamond blocks, then have a giant TNT block fall from the sky and trigger an explosion that creates two craters in the actual road surface. The output must be a complete, coherent, directly playable video.

[…]

First let the viewer clearly see the photographed street, including the real buildings on both sides, the empty foreground road, and the yellow road centerline. People and vehicles visible in the distance remain part of the background; […]

Supplied assets (9 files, 0.4 MB), read-only in the sandbox:

Example L2 check                                                                                                                                                 G2 · core · one of 60 checks

Pass 15 ordinary TNT blocks can be individually tracked from actual entry through placement, without using a counter or a screenshot of the final pile in place of 15 entries

Look at Track each of T01–T15 entering, contacting support and settling, without requiring every block to be visible in the final frame or relying on a counting caption.

Prompt “For the 15-block TNT stack, the viewer must still see every block enter and settle, followed by a brief view of the complete stack. A numeric counter is optional, but do not merely draw a pile and label it “15.””

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

Music & sound-driven (C10)

A 15-Second Code Film About Its Own Prompt: Score, Pure Time-Based Drawing, Real FFT, and Beat-by-Beat Self-Audit

Goal	Core goal	L2	core
G1	Complete 15-second four-step fixed time window, original code score, and genuinely audible music.	25	19
G2	Purely time-deterministic seek, fixed runtime verification, and authentic FFT of the actual mix.	25	21
G3	Freeze V0, use the lowest score from all 30 real scores, make a real repair attempt at the same t, and report the result honestly	29	24
All L2 checks		79	64

Task specification (opening of 2,215 words)

Create a complete 15-second code film for general viewers interested in generative audiovisual work and reproducible creative processes. The film must help viewers understand how audiovisuals are generated from the film’s own prompt and must show real screenshots of checking and repair. Present the four steps in this order: score first, pure seek-based drawing, the film’s own mix spectrum, and beat-by-beat scoring of candidate versions followed by actual lowest-score-frame repair. The viewer must see the actual object or state change at every step; do not merely replace four title cards.

[. . .]

All four steps must be executed. The 15 seconds must be a complete short film with a beginning, production logic, checking, and a repair resolution, not an excerpt from a longer work. Do not add model, API, or paid-tool requirements. The C10 sound-driven component is central: the score, drawing, spectrum, and self-audit mechanisms must be understandable from the actual picture and sound.

Implement 0 samples as using no prerecorded audio clips, sample libraries, or recorded voice; digital PCM still contains discrete samples and must not be described as having no sample rate. Implement 0 keyframes as using no manual animation-keyframe system; note events, beat data, and analytic time functions are not animation keyframes. Do not use random numbers or a seeded PRNG, and do not drive drawing through a wall clock or timer.

Deliver a 1920×1080, 30fps, exactly 450-frame, 15-second complete MP4. Frame $n$ has time $n/30$, $n=0\ldots449$; do not output a terminal frame at $t=15$. 120 BPM equals 0.5 seconds per beat, or 30 beats; 4/4 time is 7 complete bars plus 2 beats, not 8 complete bars. [. . .]

Supplied assets (18 files, 0.1 MB), read-only in the sandbox:

caption:

:::

Example L2 check (G1 · core · one of 79 checks)

Pass Even-beat low kick and bass, odd-beat composite hits, and the melody/tick relationship on every beat are audible.

Prompt “Every beat must have an audible rhythmic anchor.”

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

VQ Visual quality	The same-track spectrum, version labels, and actual review images are clear and stable.
VA Visual aesthetics	The abstract geometry is unified, with precise hierarchy for the overview and enlarged minimum-issue view.
CL Content logic	The score-first approach, true spectrum, frozen scoring, and organisation of same-t actual attempts/results are clear.
TM Timing & motion	The current responses of the 12 specified frequency bands are discernible and present corresponding changes with the actual mix (not every band is required to change, and full-spectrum coverage is not claimed).
AQ Audio & A/V sync	The coded melody, bass, and beat are distinct, and the mix itself is coordinated with the spectral changes.

Classroom lessons (C11)

Digitizing images

Goal	Core goal	L2	core
G1	The two key stages of digitization, sampling and quantization, and why and with what result a continuous image becomes a digital one.	9	8
G2	Pixel, resolution, total pixel count, bit depth, number of colors, the typical meaning of 1/8/24 bits, and how the three RGB channels relate to true color.	14	11
G3	Derive and apply the storage formula of an uncompressed bitmap, with B/KB/MB conversion and the 1024×768 worked example.	8	6
G4	How file size relates to resolution, bit depth and compression, correcting common misconceptions.	9	7
G5	Diagrams or animation for the relations of Section 6: pixel grid, sampling density, quantization levels, binary mapping, RGB mixing and the storage calculation.	6	6
G6	Chinese narration carries the explanation, synchronized with the picture.	13	10
All L2 checks		59	48

Task specification

opening of 186 words · translated from Chinese

Digitizing images

Make an explainer video for 10th-grade students that makes "digitizing images" clear, based on the lecture note in the supplied file.

The note’s core concepts, principles and derivation steps, worked examples and common misconceptions must all be covered, and covered correctly; the relationships listed in Section 6 of the note should be made clear with diagrams or animation, so that students can see the spatial, process or causal relations in them.

The whole video must have Chinese narration that carries the main explanation and is synchronized with the picture: whatever step the narration is on, the picture shows that step.

The order of explanation, visual style, camera, layout and color scheme are up to you. You need not read the note word for word and may add intuitive examples or analogies beyond it, but you must not contradict the note or make scientific errors. [. . . ]

Supplied assets (1 file, 8 KB), read-only in the sandbox:

• Lecture note 1 the lecture note the lesson is based on (Markdown, in Chinese)

Example L2 check

G3 · core · one of 59 checks · translated from Chinese

Pass	Narration or picture gives the 24-bit data size as 2,359,296 bytes (from 786,432×24÷8 or 786,432×3), converted to 2,304 KB, about 2.25 MB; if the total of 18,874,368 bits is shown, its value must be correct.
Fail	The data size is miscalculated; or the division by 8 is forgotten and 18,874,368 is taken as the byte count; or the MB result is not about 2.25; or the 24-bit size and the unit conversion are not given.
Look at	The worked example’s 24-bit size and unit conversion.
Lecture note	“Total bits = 786432×24 = 18874368 bits; total bytes = 18874368÷8 = 2359296 bytes (or 786432×3 = 2359296 bytes); 2359296÷1024 = 2304 KB; 2304÷1024 = 2.25 MB.”

L3 quality rubric

all five dimensions with the anchor for a score of 5 on this task (translated from Chinese):

VQ Visual quality	At 1920×1080 the picture is sharp and stable; text, formulas, arrows, grids, RGB mixing and the storage flow chart are clearly legible and still identifiable in sampled frames; no occlusion or unintended flicker.
VA Visual aesthetics	A unified, simple teaching design whose visual metaphors (grids, gray-level strips, binary mapping, RGB sliders, the storage flow) explain image digitization and engage 10th-graders without distracting from the task.
CL Content logic	A clear path from analog versus digital through sampling, quantization, pixels, bit depth, RGB, the storage formula, the worked example and the misconceptions; narration and picture advance together, with a summary.
TM Timing & motion	Motion and pacing match the Chinese narration; the sampling grid, gray-level steps, RGB mixing and unit conversion unfold step by step with the explanation; key points pause, and the whole is paced but not rushed within 120–300 s.
AQ Audio & A/V sync	Clear, natural Chinese narration throughout reads terms, numbers and formulas correctly at a pace that allows following the derivation, keeps step with the picture, and is never covered by music or effects.

Paper explainers (C12)

Explaining "Auto-Encoding Variational Bayes"

Goal	Core goal	L2	core
G1	Explain the problem and motivation: efficient inference and learning in directed probabilistic models with continuous latent [. . . ]	6	4
G2	Explain the key idea: reparameterizing the variational lower bound yields the differentiable SGVB estimator, enabling AEVB and the variational auto-encoder.	4	3
G3	Explain how the method works: generative model, recognition model, variational bound, reparameterization trick, SGVB estimators [. . . ]	24	11
G4	Report the paper’s main results, supporting evidence, and stated limitations or open questions accurately.	12	4
G5	Use diagrams and animations to make mechanisms, data flow, equations, and result comparisons clear and consistent with the paper.	6	3
G6	English narration carries the explanation throughout and stays in sync with the visuals.	12	9
	All L2 checks	64	34

Supplied assets (1 file, 3.7 MB), read-only in the sandbox:

• Paper 1 the paper itself (PDF)

L3 quality rubric, all five dimensions with the anchor for a score of 5 on this task:

VQ Visual quality	All equations, labels, axes, and diagram text are clearly legible at 1920×1080, with sufficient size, contrast, and on-screen duration for first-year graduate viewers to read key symbols and numbers.
VA Visual aesthetics	Visuals are purpose-built for this paper: the graphical model, encoder-decoder flow, reparameterization, lower-bound decomposition, training loop, and results charts are represented with clear, task-specific diagrams or animations that directly support understanding. [. . . ]
CL Content logic	Content is organized as a clear explanatory arc: problem and why it matters, key idea, step-by-step method, evidence-based results, then limitations or open questions, with smooth transitions that make the logical dependencies easy to follow.
TM Timing & motion	Motion and pacing are carefully synced with narration: each mechanism is revealed as it is described, dense equations get enough dwell time, and transitions are smooth without distracting motion.
AQ Audio & A/V sync	English narration is clear and intelligible throughout, technical terms, equations, and numbers are read correctly, music or sound effects never mask the voice, and narration and visuals are tightly synchronized so every described step appears when spoken.