WorksComputer Science
WeVisCut: The Whole Editing Room in One Agent From Narration to Long-Form Video
Abstract
Producing long-form videos from narration requires interdependent decisions that remain coherent across shots. Existing agentic editors often use multi-stage workflows driven by general models, leaving the quality-cost trade-off of compact, task-adapted policies unclear. We address this gap with WeVisCut, a long-form script-to-video system that connects tool execution with policy learning. Its agent harness maintains shared production state so that tool feedback informs subsequent decisions and repairs. Because finished videos do not expose these decisions, our four-stage data pipeline combines editing inversion with verified replay to turn recovered editing intent into executable supervision. Applied to 3,750 finished videos, this pipeline yields over 130K training examples. We use this supervision to train a single 9B policy that coordinates production and recovers from failures within the same harness. Moreover, we introduce a 100-task benchmark and five metrics assessing editing quality and delivery reliability. Under a shared tool environment, WeVisCut substantially outperforms its base model and achieves comparable performance to Claude Opus 5 while using nearly 33x fewer generated tokens per successful output minute. It also outperforms task-matched native editors on our proposed metrics.
Project Page: https://tencent.github.io/WeVisCut
Code: https://github.com/Tencent/WeVisCut
Weights: https://huggingface.co/tencent/WeVisCut

Figure 1. Overview of WeVisCut. Top: Script-driven stateful video production (planning, retrieval, composition, inspection, repair) with multi-round edits. Middle: Baseline comparisons in time, cost, preference, and reliability. Bottom: Examples in science, history, geography, tech, and health.
Introduction
“The whole eloquence of cinema is achieved in the editing room.”
— Orson Welles
Creating a narrated long-form video involves turning a script into a sequence of coherent shots. Planning, visual retrieval, narration synthesis, and timeline assembly are interdependent. Changing a shot’s duration affects later timestamps, while unsuitable visual material may require both retrieval and planning to be revisited across the affected shot sequence.
Recent agentic editors coordinate production through specialized agents and tool workflows [21, 39, 44, 47, 48, 49]. Several systems use general models with feedback or graph refinement. Yet orchestration alone does not specify how to train a compact policy that maintains video quality at low token cost. Crayotter trains a 9B editing model using final-video preferences on manually constructed tasks [38]. Such a policy needs a stateful environment for execution and recovery, together with supervision grounded in that environment. Finished videos, however, omit the tool calls, intermediate artifacts, and failures needed for such supervision. The challenge is therefore to connect stateful execution, executable supervision, and policy learning.
To address these challenges, we present WeVisCut, which integrates a script-to-video agent harness, executable data construction, and a task-adapted policy. The harness (Figure 2) maintains shared production state and validates tool actions. It exposes execution feedback for inspection, bounded repair, and replanning while preserving unaffected artifacts from earlier successful production steps.

Figure 2. The WeVisCut agent harness with stateful tool selection. A shared policy chooses legal tools from the evolving production state. Validated returns update artifacts and inform inspection, local repair, and replanning before final stitching and delivery.
Within this harness, we construct executable supervision through the four-stage pipeline in Figure 3. We first recover script–shot intent through editing inversion and execute it through verified replay. When replay reveals defects, we apply observe–reflect–repair for bounded corrections that preserve successful artifacts. We then use trajectory materialization to organize the recorded executions into training examples with tool feedback and repair outcomes. From 3,750 video projects, this process yields over 130K examples.
Using these examples, we train a single 9B policy across production roles in three phases (Figure 4). In Phase 1, we establish tool use and recovery through unified supervised fine-tuning [46] and runtime-verified rejection-sampled fine-tuning [45]. In Phase 2, we apply Guided-OPD [12] to learn recovery on policy-reached states while phasing out teacher intervention. In Phase 3, we use RCF-DAPO, our role- and context-factored DAPO [43], to optimize decisions by comparing runtime outcomes within matched roles and contexts.
Figure 4. Three training phases of WeVisCut. Phase 1 learns tool use and recovery through unified SFT and runtime-verified RFT. Phase 2 applies Guided-OPD with decreasing teacher intervention. Phase 3 uses RCF-DAPO to compare alternatives within matched roles and decision contexts.
Moreover, we introduce a benchmark of 100 script-to-video tasks and five complementary metrics to evaluate editing quality and delivery reliability. These metrics relate rendered videos to their scripts and production plans, so evaluation goes beyond successful rendering alone. In a shared tool environment, our policy improves over its base model by 16.0 points in Overall Score and achieves comparable performance to Claude Opus 5 while using nearly 33× fewer generated tokens per successful output minute. Our complete system also outperforms task-matched native editors.
We summarize our contributions as follows.
We design a script-to-video agent harness that maintains shared production state, validates tool actions, and supports inspection, bounded repair, and final delivery.
We build a four-stage data pipeline based on editing inversion and verified replay, converting finished videos into executable training examples.
We train a unified 9B policy for tool use and recovery through sequential SFT, runtime-verified RFT, Guided-OPD, and RCF-DAPO within a shared production harness.
We introduce a 100-task benchmark and five metrics. WeVisCut improves its base policy, outperforms task-matched systems, and achieves comparable quality to Claude Opus 5 at markedly lower cost.
Related Work
Tool agents and long-horizon post-training. Producing long-form videos requires agents to choose tools and revise plans from execution feedback. Earlier methods range from interleaved reasoning and action [41] to learned API calls [22, 23, 29] and runnable tool dialogues [13]. To train such behavior, researchers curate interactions and synthesize instructions or tool calls [2, 16, 20, 46]. Reflection and reusable skills also support recovery [32, 35]. Closer to our data pipeline, AgentTrek builds verified web trajectories from tutorials [37]. Finished videos, however, do not record editing actions. We therefore infer script–shot intent and replay it with production tools to obtain executable trajectories. Long-horizon training also needs to handle policy-generated states. DAgger [28] collects expert labels for learner states, while generalized knowledge distillation trains on student-generated sequences [1]. Guided-OPD mixes teacher and student turns, while TurnOPD and Step-Level OPD refine supervision at turn and step boundaries [12, 33, 51]. We apply Guided-OPD with truncated teacher logits. For reward optimization, GRPO and DAPO build on PPO-style updates with group-relative comparisons [30, 31, 43]. We adapt DAPO for our multi-role policy as RCF-DAPO, comparing rewards only within matched roles and decision contexts.
Video editing. Video editing systems range from user-assisted tools to autonomous agents. LAVE keeps users in the editing loop [34]. In more automated settings, CutClaw and CineAgents handle music-synchronized editing and cinematic compilation [47, 48]. The original Crayotter uses a traceable multi-agent workflow, while VideoAgent assembles specialists dynamically [39, 49]. Beyond workflow design, task-specific training has also been explored. “The Script is All You Need” trains a ScripterAgent but uses a separate DirectorAgent [21]. Aurora trains a planner for local edits [44]. Building on the Crayotter environment, subsequent work trains a 9B policy with final-video preferences on manually constructed tasks [38]. In contrast, WeVisCut learns one role-routed policy for fixed-narration production from executable trajectories built through editing inversion, verified replay, observe–reflect–repair, and trajectory materialization.
Agent and video evaluation. Evaluation of narration-driven video production needs to consider both the rendered video and the production process. VBench, VideoScore, and VMBench assess generated-video quality and motion [6, 9, 15], while Goku-Bench tests instruction-based edits [14]. Script–video timing has also been evaluated [21]. Complementary agent evaluation ranges from recorded web and mobile trajectories [3, 27] to interactive shopping, browser, and desktop environments that validate action outcomes [4, 11, 36, 40, 50]. Software-repair benchmarks use repository tests [10], while multi-turn and tool-oriented suites measure progress, valid transitions, and task constraints [17, 18, 19, 42]. We combine these perspectives in a benchmark of 100 script-to-video tasks, linking scripts, shot plans, tool trajectories, and rendered videos. Five metrics are introduced to assess script fidelity, visual–narration alignment, timeline plan realization, long-horizon consistency, and end-to-end delivery success.
WeVisCut Agent Framework
Figure Figure 2 presents the WeVisCut agent framework for long-form script-to-video production. A shared policy makes editing decisions, while a stateful runtime executes tools and maintains the resulting production artifacts. Their interaction connects the narration script to a rendered video through an inspectable process that supports revision when execution reveals problems. We first define the task and policy–runtime interface, then describe the harness that implements this process.
Script-to-Video Task
A task input consists of a fixed narration script , target aspect ratio , feasible duration interval , optional source assets, and a versioned shared tool space . The goal is to render a video satisfying the delivery constraints without rewriting the narration. The outputs also include a typed shot plan , specifying narration coverage, visual content, and timing, and an execution trajectory recording calls and returns throughout the production process (Appendix § 10).
Production uses tools that accept typed arguments and return artifacts or execution feedback. At call , a single policy with shared parameters serves a role . The orchestrator (O) selects tools and arguments, the generator (G) supplies structured content, and the critic (C) inspects artifacts and requests or verifies repairs based on recorded execution feedback.
At each call, the role context combines the task input, role, a view of the production state (shot plan, artifacts, and progress), relevant call-and-return history , and tool schemas. The policy produces a complete role output , which the runtime processes under a tool contract :
The contract specifies the tool, required fields, and value types. The runtime returns observation , reporting results or blockers, and the updated production state . For orchestrator calls, the role output specifies a tool action containing the selected tool and arguments (Figure 2).
Harness Design
The harness in Figure 2 implements the interaction in Equation (1) through a shared tool space, persistent production state, and runtime checks. Each orchestrator turn selects one tool, which may process multiple shots and invoke generator or critic calls. Tool selection depends on the current production state, including available artifacts and blockers, rather than a predetermined sequence.
Production flow and tool system. Starting from the narration script , segmentation produces production segments, and shot planning expands them into the shot plan . Shot records connect narration spans to visual queries, target durations, and spotlight assignments. Visual retrieval and text-to-speech (TTS) tools use these records to obtain shot assets and narration audio. For spotlight shots, decoration tools select verified HyperFrames templates [7] and fill typed slots, allowing the policy to control presentation through structured content rather than raw layout code. Music search supplies background music (BGM) for the final mix. Timeline composition combines shot assets, narration audio, and decorations, while final rendering produces the shot videos required for stitching. Quality judging and repair tools allow the policy to revisit these intermediate products before delivery. This progression describes artifact dependencies, not an obligatory ordering of every tool call.
Persistent state and selective context. These tools communicate through the production state . Global records track narration coverage, shot order, and timing, while per-shot records associate assets, audio, templates, renders, and unresolved issues with shot IDs. Planned content remains distinct from artifacts that have actually been produced. Episode memory retains these artifacts, and the event trace preserves calls, returns, and failures. The initial observation exposes the full production state. Subsequent role contexts contain the latest runtime observation, state changes, a progress summary, and relevant local records. Separating persistent storage from model context avoids repeatedly serializing the entire project while preserving the information needed for later inspection and repair.
Validated execution and actionable feedback. Given a tool action , the runtime identifies the tool contract and checks argument schemas, artifact references, and execution prerequisites. When structured content or inspection is required, it supplies this contract in the generator’s or critic’s role context. Only the runtime commits validated results to the production state, whereas rejected calls return typed blockers that identify what must be corrected or produced next. For example, a stitching request cannot substitute for missing shot renders or background music. The runtime reports these missing prerequisites, allowing the orchestrator to select the necessary production tools before retrying. This separation keeps model decisions grounded in execution evidence rather than treating a proposed plan as completed work.
Artifact-grounded inspection and bounded repair. Execution feedback also determines when revision is needed. Quality inspection combines structured checks of shot records, timing, and render status with a VLM-Judge over spotlight renders. The critic identifies affected shots and the defects that a repair should address. Repair tools then revise the relevant content and recheck affected downstream artifacts while retaining still-valid results. For example, correcting a decoration need not regenerate unchanged narration audio. Repair limits bound repeated correction attempts, but exhausting this budget does not remove the artifact prerequisites for delivery. Quality warnings and missing production artifacts therefore remain distinct conditions. The representative trajectory in Appendix § 11 illustrates how inspection, blocked calls, and targeted follow-up actions interact within this loop.
Verified delivery and trace retention. When the required artifacts are available, the policy requests final stitching. Reaching the call limit also triggers a delivery attempt from the available production state. Before stitching, the runtime checks the required shot renders, narration audio, and background music. After stitching, it validates the video and delivery manifest. Missing prerequisites are returned as blockers while further calls are available. A run that terminates without a validated final video is recorded as incomplete, with its execution trace retained for diagnosis. The resulting production records support executable data construction in Section 4 and policy training in Section 5. This production harness is separate from the read-only evaluation harness in Appendix § 9.2.
Data Construction
The harness records execution, but finished videos expose only narration and shot structure, not the tool calls, feedback, and repairs that produced them. To train the shared policy, we recover editing intent and replay it under the same state and action contract, constructing role-specific examples grounded in real tool returns from executing the recovered intent.
Our source data consists of finished narration-driven videos collected from Bilibili, spanning technology and digital products, business and finance, natural science, and social sciences and humanities. The admitted set contains 3,750 video projects.
Executable Trajectory Construction
To convert these videos into executable supervision, we use a four-stage pipeline comprising editing inversion, verified replay, observe–reflect–repair, and trajectory materialization (Figure 3).
Editing Inversion. We first recover script–shot intent from the source videos (Figure 3A), including timestamped narration, shot structure, and complete sentence–shot alignment for every admitted project. Narration provides temporal anchors, while aligned shots indicate how the content is organized visually. Together, this evidence forms a structured storyboard describing what each shot should communicate, how long it should last, and which production constraints it should satisfy. This process recovers the intended video structure, not the original editor’s tool actions.
Verified Replay. The recovered storyboard specifies what to produce, but does not provide execution feedback. Verified replay (Figure 3B) therefore executes this intent through a production toolchain aligned with the deployment state and action contract. Schema-checked tools maintain a shared production state across planning (article segmentation, shot planning, and self-reflection), asset production (visual retrieval with TTS, spotlight decoration, music search, and severity ranking), and timeline execution (composition with judging, final rendering, and stitching). Each real tool return updates this state before the next decision. The runtime verifies asset references, audio durations, template inputs, timeline positions, and rendered outputs against the artifacts actually produced. Spotlight shots use verified HYPERFRAMES templates [7] that separate layout, motion, and visual style. The policy selects a template and fills typed slots instead of generating raw HTML, CSS, or JavaScript.
Observe–Reflect–Repair. During replay, quality inspection combines structured shot, timing, decoration-slot, and render-status checks with a VLM-Judge over spotlight renders. Using these inspection results, we locate production problems, apply bounded local repairs supported by the toolchain, and verify these changes while preserving successful artifacts (Figure 3C).
Trajectory Materialization. These executions produce trajectories that retain pre-action states, tool names and arguments, real returns, created artifacts, quality verdicts, and failure–repair links. Missing arguments, unresolved references, invalid stage order, and failed delivery checks remain visible rather than being rewritten as successes. We then organize these trajectories into role-specific examples and rollout records (Figure 3D).
Training Examples and Data Split
These examples provide role-specific supervision under the framework in Figure 2. Orchestrator samples supervise tool selection and structured arguments, while generator samples cover script segments, shot plans, retrieval queries, and template content. Critic samples provide failure localization, repair requests, and verification decisions. Runtime observations remain context, with loss applied only to model-authored tokens.
We partition by source project, keeping all its examples and repair records together. Held-out projects supply the benchmark in § 6.1 and are excluded from training. The 134,771 examples supervise a single Qwen3.5-9B policy with shared production state and tool vocabulary across all three roles.
Training WeVisCut
The executable examples provide initial supervision, but WeVisCut also needs to handle the states produced by its own decisions. Figure 4 summarizes three training phases, comprising unified SFT and runtime-verified RFT, recovery learning with Guided-OPD, and decision optimization with RCF-DAPO. The policy interaction and agent harness are defined in Section 3.
Phase 1: Unified SFT and RFT
We first train Qwen3.5-9B with LoRA [8] on the 134,771 examples constructed above. Following AgentTuning and Agent-FLAN [2, 46], we supervise model responses while retaining tool feedback as input.
Let contain training examples. We index examples by and tokens within an example by . The conversation has tokens, and is the prefix before position . Before each response, this prefix encodes the role context , including instructions and available tool feedback. The loss mask is one for tokens in the target model response and zero for instructions, tool-return messages, and padding tokens added to equalize sequence lengths for batching. Thus only response tokens contribute to the supervised fine-tuning loss
The denominator counts the supervised tokens, so the loss averages their negative log probabilities across all supervised model responses in the corpus.
SFT teaches the recorded production steps, but the student can encounter new failures when executing tasks on its own. We collect additional supervision through runtime-verified rejection-sampled fine-tuning (RFT). At the first recoverable failure, we restore the last replayable state before the error and keep the valid earlier steps. The teacher regenerates the complete remaining sequence from that state. Both student-only successes and teacher-assisted repairs are replayed and admitted only if they pass the runtime checks and a quality threshold selected on development tasks. We track and balance these two sources of supervision. Failures that cannot be restored enter a bounded pool for later rollout sampling. Appendix § 8.1 gives the admission rules for both sources of supervision.
Phase 2: Recovery Learning with Guided-OPD
We next apply Guided-OPD [12] to learn decisions while interacting with the production environment. At training update , the teacher generates a complete role output with probability , and the student generates it with probability . At call , we draw and use
Here is the frozen teacher’s parameter set. Both policies receive the same role context . As in Equation (1), the complete output is executed before the next call, when control is sampled again.
Both branches train the student. Teacher outputs use forward KL against the teacher distribution renormalized over its most likely tokens, reducing the cost of distillation. Student outputs use reverse KL on their own generated prefixes. We gradually reduce from a value below one to zero, so training ends with fully student-controlled executions. Appendix § 8.1 specifies the token masks, KL losses, and intervention schedule used throughout this training phase.
Phase 3: Decision Optimization with RCF-DAPO
Finally, we compare alternative outputs for the same production decision. Our Role- and Context-Factored DAPO (RCF-DAPO) adapts DAPO [43] by restricting reward comparisons to the same role and production context. At a fixed role call, we sample candidate outputs , where indexes alternatives in the comparison group. For example, generator candidates may be different retrieval queries for the same shot. For orchestrator calls, we restore the same production state, sample alternative next actions, and execute each with a frozen continuation policy, the same tool registry, budget, and paired random seeds. Generator and critic outputs are evaluated under the same local request, artifact, and tool contract.
Each candidate receives a role-specific reward . Orchestrator rewards combine successful delivery and final quality with partial credit for persistent progress on failed executions. Generator rewards assess schema validity, requirement coverage, and content usability. Critic rewards assess diagnosis, repair utility, and verification, with a penalty for unnecessary repairs. The scoring rules, rubrics, and weights are fixed. Appendix § 8.2 defines the complete reward formulas.
Within a group with matching conditions , we convert rewards into relative advantages
where and are the group’s mean reward and standard deviation, prevents division by zero, and stops gradients through the advantages. Above-mean rewards yield positive advantages, and below-mean rewards yield negative ones. Groups with insufficient reward variation are resampled, with quotas and a replay queue maintaining coverage. Contexts are collected from the current policy with gradients stopped, and role losses are balanced after token-level normalization. We retain DAPO’s asymmetric Clip-Higher and soft overlong shaping. Appendix § 8.2 details the sampling and updates.
Experiments
We evaluate whether task-specific training improves script-to-video production and whether these gains extend to complete editing systems. We first compare policies in a shared tool environment, then compare WeVisCut with editors using their native workflows. Separating these comparisons distinguishes policy performance from differences in tools and orchestration. We further examine the cumulative training stages and assess the rendered videos through blinded human evaluation.
Experimental Setup
Benchmark and implementation. We construct a fixed benchmark of 100 script-to-video tasks from the held-out source projects in Section 4. Each task specifies a narration script, source pool, target duration, and aspect ratio. All systems receive identical task inputs and delivery constraints. Our trainable variants share the Qwen3.5-9B backbone, training split, decoding configuration, tool registry, and rendering budget. The Guided-OPD and RCF-DAPO training configurations are given in Appendix Table 5.
Comparison methods. For the policy comparison, all models use the same instructions, tool schemas, runtime, and rendering budget. We evaluate Qwen3.5-9B, Qwen3.6-27B, and Qwen3.6-35B-A3B [24, 25, 26], together with Qwen3.5-397B-A17B, Gemini 3.1 Pro [5], and a fixed Claude Opus 5 snapshot. These reference models receive no task-specific post-training. Our variants differ only in the training stages applied to the shared 9B backbone within the same production environment.
We compare WeVisCut with Crayotter [38] and the commentary-capable VideoAgent workflow [49], each using its released tools and orchestration. Adapters may translate file formats and export evaluation artifacts, but may not retrieve assets, write narration, create plans, or repair outputs. CutClaw, CineAgents, and DirectorAgent [21, 47, 48] are excluded from this quantitative comparison because their native workflows omit or replace parts of the fixed-narration, fixed-source task contract. Appendix Table 6 shows whether each system supports the required inputs, shot plans, and execution.
Evaluation metrics. We propose five complementary metrics for script-to-video production that assess the relationship between the source script, editing plan, execution trace, and rendered video. All scores lie in , with higher values indicating better performance. Script Content Fidelity (SCF) measures the coverage, faithfulness, and ordering of narration claims relative to the source script. Visual–Narration Semantic Match (VNSM) assesses whether rendered visuals support the concurrent narration. Timeline Plan Realization (TPR) compares planned and realized shots in presence, duration, and order. Long-Horizon Consistency (LHC) measures late-stage shot-health retention and asset diversity. End-to-End Rendering Yield (ERY) is the fraction of held-out tasks producing a decodable, sufficiently complete video with the required non-silent narration. Overall Score is the equally weighted mean of the five reported metric means, used only as a descriptive summary for comparing aggregate task performance.
Production Cost Efficiency (PCE) is reported separately as tool calls and generated tokens per successful output minute. For native systems, we instead report wall-clock minutes and estimated USD per successful output minute, since the underlying tools and services differ. Both measures include failed attempts in the cost numerator, and lower values are better. Definitions, eligibility rules, and missing-data treatment are given in Appendix § 9.
Evaluation protocol. Each system is evaluated in five independent runs over the 100 tasks. We report across-run means and standard deviations for the five component metrics, with no uncertainty estimate for Overall Score. Permissions and retry limits are fixed within the shared environment, and thresholds, adapters, and evaluator settings are selected on development tasks. Semantic evaluation uses timestamped contact sheets in a read-only, network-free environment with access restricted to metric-specific evidence. The held-out evaluator, its prompts, and the five reported metrics are not used as post-SFT training signals. The metric-validation protocol is described in Appendix § 9.1, and evidence collection and isolation within the read-only evaluation environment are detailed in Appendix § 9.2.
Policy Comparison
We first compare policies within the shared production environment to examine the effect of task-specific training without changing the tool pipeline. We report task performance and policy-generation cost for each evaluated policy variant in Table 1.
Table 1. Policy comparison on 100 held-out tasks in the shared tool environment. Component scores are mean±standard deviation across five runs over the full test set. Overall is the equal-weight mean of the five component means. Calls and generated tokens include all attempts and are normalized by successful output minutes. Arrows indicate preferred directions, and the best results are bold.
| Method | SCF | VNSM | TPR | LHC | ERY | Overall | Calls / min | Tokens / min |
| General-purpose baselines | ||||||||
| Qwen3.5-9B | 74.5 | 3.1 | 11,883 | |||||
| Qwen3.6-27B | 75.9 | 3.9 | 6,742 | |||||
| Qwen3.6-35B-A3B | 77.7 | 4.7 | 27,554 | |||||
| Reference models | ||||||||
| Qwen3.5-397B-A17B | 81.5 | 6.0 | 15,223 | |||||
| Gemini 3.1 Pro | 84.6 | 4.5 | 13,814 | |||||
| Claude Opus 5 | 89.9 | 11.6 | 78,763 | |||||
| WeVisCut (Ours) | 90.5 | 3.1 | 2,377 |
caption:
Task-specific performance. WeVisCut improves the Overall Score from 74.5 to 90.5 over its Qwen3.5-9B initialization, a gain of 16.0 points, with improvements in all five metrics. The gains extend beyond successful delivery to script fidelity and visual grounding, as SCF and VNSM increase by 18.6 and 15.4 points, respectively. By comparison, the unadapted models show less uniform improvements over the 9B base policy. Qwen3.6-35B-A3B gains 9.0 points on ERY but loses 2.5 points on VNSM, illustrating that delivery success does not necessarily imply better visual–narration alignment. WeVisCut also obtains higher means on all five metrics than Qwen3.5-397B-A17B and Gemini 3.1 Pro.
Comparison with Claude Opus 5. WeVisCut achieves an Overall Score of 90.5, compared with 89.9 for Claude Opus 5. Its mean scores are higher on four metrics and slightly lower on VNSM, with no component differing by more than 1.3 points. Yet it uses 2,377 generated tokens per successful output minute, compared with 78,763 for Claude Opus 5, or approximately one thirty-third as many. Relative to its base policy, WeVisCut also reduces token usage from 11,883 while maintaining 3.1 tool calls per successful output minute. These results show that the task-adapted 9B WeVisCut policy achieves mean task scores close to those of Claude Opus 5 at substantially lower generation cost. We next examine quality and cost in complete editing workflows with each system’s native tools.
End-to-End System Comparison
We next evaluate whether the gains persist when each editor uses its own tools and orchestration. In Table 2, we compare WeVisCut with Crayotter and VideoAgent using identical task inputs.
Table 2. End-to-end comparison using each system’s native workflow on the same 100 tasks. Component scores are mean±standard deviation across five runs, and Overall is the equal-weight mean of the component means. Native costs include all attempted runs and report wall-clock minutes and estimated USD per successful output minute. The best results are bold.
| System | SCF | VNSM | TPR | LHC | ERY | Overall | Time (min) / out min | Cost (USD) / out min |
| Crayotter | 84.9 | 8.6 | 0.97 | |||||
| VideoAgent | 82.5 | 10.4 | 0.58 | |||||
| WeVisCut (Ours) | 90.5 | 2.1 | 0.09 |
caption:
WeVisCut achieves the highest mean score on all five metrics, with an Overall score of 90.5 compared with 84.9 for Crayotter and 82.5 for VideoAgent. The improvements cover both semantic fidelity and execution quality. This pattern indicates consistent gains across content selection, temporal planning, execution stability, and final delivery. For example, relative to Crayotter, SCF and VNSM each improve by 6.8 points, alongside higher plan realization, long-horizon consistency, and rendering yield. The advantage observed in the shared environment therefore also holds for these complete editing workflows.
These gains are accompanied by lower end-to-end costs. For each successful output minute, WeVisCut requires 2.1 wall-clock minutes, compared with 8.6 for Crayotter and 10.4 for VideoAgent. The corresponding estimated monetary costs are $0.09, $0.97, and $0.58, respectively. Both measures include unsuccessful attempts. Note that these costs reflect the complete native workflows, rather than the efficiency of an isolated model or tool.
Human Evaluation
Training Stage Analysis
To examine how the training phases in Section 5 contribute to the final policy, we evaluate successive checkpoints in Table 3. The backbone, data split, tools, decoding configuration, and rendering budget remain fixed. Each checkpoint adds the next training stage to the preceding one, allowing us to track the cumulative gains across the three training phases.
| Phase | Variant | SCF | VNSM | TPR | LHC | ERY | Overall |
| – | Base | 74.5 | |||||
| 1 | SFT | 79.5 | |||||
| 1 | SFTRFT | 83.5 | |||||
| 2 | SFTRFT Guided-OPD | 84.9 | |||||
| 3 | WeVisCut (Ours) | 90.5 |
Table 3. Cumulative training-stage analysis with a fixed Qwen3.5-9B backbone and evaluation environment. Phase 1 includes SFT and RFT, Phase 2 adds Guided-OPD, and Phase 3 adds RCF-DAPO. Component scores are mean±standard deviation across five runs on the 100 held-out tasks. Overall is the equal-weight mean of the component means. The best results are bold.
In Phase 1, SFT raises ERY from 70.4 to 87.6, providing the largest stage-wise improvement in delivery reliability. This is consistent with its role in establishing valid tool use from executable demonstrations. RFT then adds supervision from successful student trajectories and teacher-repaired failures. Relative to SFT alone, it improves SCF by 6.2 points and TPR by 5.7 points, indicating gains in script fidelity and plan realization beyond the initial improvement in delivery.
Building on this initialization, Phase 2 applies Guided-OPD to mixed teacher–student rollouts. All five metrics improve at this checkpoint, including VNSM from 65.1 to 67.4 and ERY from 89.8 to 91.5. This pattern suggests that Guided-OPD complements trajectory-level RFT by refining decisions at intermediate states, where the student must continue from its own partially correct actions rather than replaying teacher-generated contexts. These results show further gains after distillation on states reached during policy execution, although the aggregate metrics do not isolate failure recovery.
Finally, Phase 3 optimizes production decisions with RCF-DAPO. SCF and VNSM improve by another 7.4 and 7.2 points, respectively, while Overall increases from 84.9 to 90.5. Figure 5 shows the cumulative improvements in script fidelity, visual–narration alignment, and long-horizon consistency against the reference policies. These comparisons measure the benefit of adding training stages in this order, rather than the independent effects of turn guidance, snapshot matching, or individual role rewards.

Figure 5. Cumulative training progression for (a) SCF, (b) VNSM, and (c) LHC. OPD and RL denote Guided-OPD and RCF-DAPO, respectively. Error bars show standard deviations across five evaluation runs on the 100 held-out tasks. Horizontal lines show Qwen3.5-9B, Qwen3.5-397B-A17B, and Claude Opus 5 under the shared tool environment. Results at these checkpoints on all five metrics are shown in Table 3.
Human Evaluation
To test whether the automatic gains carry over to human judgments, three qualified annotators independently rate anonymized videos from the three systems in Table 2 on the same 100 held-out tasks. Videos are shown individually in random order, with scripts, source pools, and target durations fixed. The shared tasks allow paired comparisons. Each video is rated from 1 to 5 on the six dimensions in Table 4.
Table 4. Blinded human evaluation on 100 held-out tasks with three raters. Entries are mean 1–5 ratings (higher is better). Overall preference is a human rating, distinct from the Overall Score.
As shown in Table 4, WeVisCut obtains the highest mean rating on all six dimensions, with an overall preference of 4.56 versus 3.48 for Crayotter and 3.46 for VideoAgent. The largest gains occur in narrative coherence (1.23 and 1.28 points), while higher editing smoothness and visual quality indicate that the preference is not confined to narrative structure, consistent with the automatic comparisons. Ratings are averaged across annotators within each task. These descriptive means do not establish statistical significance or validate individual automatic metrics (see Appendix § 9.1).
Conclusion
We presented WeVisCut, a system for long-form script-to-video production built around a single 9B policy shared by the orchestrator, generator, and critic. Our four-stage data pipeline combines editing inversion and verified replay to recover executable supervision from finished videos. Training progresses from executable imitation to recovery learning and reward-based decision optimization. Our 100-task benchmark connects source scripts, editing plans, execution traces, and final videos through five complementary metrics. The trained policy improves all five metrics over its base model and achieves mean scores close to those of Claude Opus 5 while using approximately one thirty-third as many generated tokens per successful output minute. Native-system comparisons and blinded human ratings also favor WeVisCut. Broader template coverage and more robust retrieval are needed to extend the system to a wider range of editing tasks.
Optimization and Training Details
We use the notation of Sections 3 and 5 throughout: indexes role calls, indexes tokens, and indexes training updates. The task input is , and is the shared policy conditioned on role .
Executable RFT and Guided-OPD
Replay-safe RFT. Let index the call at which the first runtime failure occurs. A failure is recoverable when the environment can restore the latest transaction snapshot before . External side effects are executed in a sandbox or through idempotent calls. We retain the valid prefix, provide the teacher with the failure record and restored state, and regenerate the complete suffix. The repaired trajectory is retained only after replay from the snapshot passes schema, prerequisite, artifact-reference, render, duration, narration, lint, and black-frame checks. Its quality score also needs to meet the development-selected minimum : . Complete student successes pass the same admission checks. Student-only and teacher-assisted records have separate provenance tags and fixed source–task quotas. Unrecoverable cases are kept in a class-balanced, capped hard-case pool for later rollout sampling, with fixed mixing probability .
Turn-level intervention and loss routing. Following Guided-OPD [12], call consists of a complete role output followed by its runtime observation . At update , selects the teacher when and the student otherwise. The selected output is executed before control is sampled again. Both branches contribute distillation loss. We denote the mixed rollout distribution by , where and are the student and frozen teacher parameters.
The models share a tokenizer. For teacher outputs, the teacher distribution is renormalized over its most likely tokens, giving . Student outputs use reverse KL, while teacher outputs use this truncated forward KL:
Here contains the role context and all tokens preceding position in call . The policies in each KL term are conditioned on this token prefix. The mask is one for assistant tokens and zero for input, padding, and runtime observations, and counts supervised tokens. Thus teacher-only trajectories also train the student. We log retained top- probability mass to audit truncation. This is an efficiency variant of Guided-OPD. Step-Level OPD instead keeps trajectories student-controlled and queries teacher targets afterward [33].
Teacher intervention follows
where is the number of updates over which intervention decays, set to 70% of training. The remaining 30% uses student-only rollouts. Only checkpoints evaluated without teacher intervention proceed to RL. Guided-OPD uses a 50-call horizon with early termination, while RL uses the 16-call deployment budget. Token limits are per-call context/response caps, and compact state updates limit history growth.
Role- and Context-Factored DAPO
Optimization scope. Each trainable call records its role , context , output , observation , and execution contract . Main rollouts populate a snapshot buffer. For an orchestrator context , we restore the pre-decision state, sample alternative outputs, and execute each with the same frozen continuation policy , budget, and seed set. Here is the policy parameter snapshot frozen for the current optimization iteration. Nested generator and critic tokens are masked from the orchestrator loss. Generator and critic contexts are also collected from current-policy rollouts and resampled locally. Let be the context distribution for role and its outer loss weight. The optimized proxy is
Here is the expected execution reward for output in context , with runtime observations averaged out. This is a conditional call-level objective. Frozen continuations and the sampled context distribution remain part of the proxy, so it does not provide an unbiased gradient of the terminal production return.
Matched comparison groups. For orchestrator call , we fix : the task input, context, restored state, tool space, remaining budget , and continuation policy . Only the next output varies. Generator groups fix the local request, artifact, tool, schema/version, budget, and initial state. Critic groups also fix the defect and verification contract. Candidate executions share paired seeds. Let be the group of outputs sampled under matching conditions .
Features are mapped to their registered ranges: , except repair utility in . Fixed weights produce one reward per output before group normalization. Contexts are sampled through role–task–tool quotas. A group is accepted only if , where is the minimum reward-variance threshold. Otherwise it is resampled up to a fixed cap. Low-variance contexts remain in a coverage queue and are revisited through executable imitation. For an accepted group,
Here is the group reward standard deviation, stabilizes normalization, and stops gradients through group statistics. No normalization mixes roles or unmatched contexts. Resampling emphasizes informative reward differences and changes the sampled context distribution.
Role-specific rewards. For candidate , let be its role output and its execution feedback. An orchestrator output is scored by continuing from snapshot . We measure progress through validated milestones:
Here indicates whether unique milestone holds in state , is its fixed weight, is the discount factor, and indexes subsequent calls until branch terminates. Repetition adds no milestone credit, and later invalidation removes it. The positive margin keeps failed-branch progress below the delivery tier. The operator bounds its argument to .
The rewards are
The delivery indicator is one only if termination, schema, prerequisite, artifact, and loop checks pass and final quality , where is the minimum delivery quality. Otherwise it is zero. A legal stop below this threshold is premature, recorded by the indicator . The parameter weights delivery and progress credit relative to final quality, and weights the premature-stop penalty.
Final quality combines a rule score and a sampled render audit: , where and is the fixed audit weight. The six rule features are plan coverage, spotlight-decoration coverage, inspection coverage, plan hygiene, completion, and efficiency, weighted 0.353, 0.186, 0.179, 0.144, 0.081, and 0.057. The render audit scores visual impact and clarity. Thus these six weights apply to rule features, with the audit blended separately.
Generator features measure schema validity, grounding, requirement coverage, and downstream usability using deterministic checks and a frozen tool-specific rubric. All features are set to zero for an invalid schema. Critic features and score diagnosis and verification, and penalizes unnecessary repair. The coefficients are fixed feature weights. Here is the diagnosis provided by critic output . For a diagnosis , controlled repair utility is
where is a frozen repair executor with identical tool version and budget, is an empty diagnosis, is a paired seed set, and maps repair results to a fixed quality score in . Thus utility measures the improvement over the empty-diagnosis baseline under controlled execution. The training judge, rubric, prompt, and version hash are frozen and independent of the held-out evaluator. Rule and judge scores are logged separately.
DAPO update. Let be the candidate batch for role , and the model-authored token positions in output . For token , the policy probability ratio is , where is the candidate’s input context. We use DAPO’s asymmetric Clip-Higher interval [43], with and :
Token losses are aggregated within each role before applying the outer weights . The old policy is synchronized once per optimization iteration. We use neither dual clipping nor a loss-side KL term.
For generated token count , let denote the no-penalty threshold and generation hard cap. For non-truncated responses, soft overlong shaping is
We set and the length-penalty weight . Responses reaching the hard cap are replaced before an eight-candidate group is formed. If the resampling cap is reached first, the group is dropped. Truncated responses enter neither rewards, group statistics, nor policy loss. Role-wise gradient norms and pairwise cosine similarities are logged, since outer loss weights control scale but do not eliminate shared-parameter gradient conflict.
The registered post-training configuration is shown in Table 5.
| Stage | Key configuration |
| Guided-OPD | Qwen3.5-9B student with a Qwen3.5-397B-A17B-FP8 teacher and . Per-call prompt/response caps: 12,288/4,096 tokens. Batch 16, 90 steps, decaying over 70 steps, and learning rate . At most 50 assistant turns with early termination. |
| RCF-DAPO | Per-call prompt/response caps: 24,576/12,288 tokens. Train/generation/mini-batch: 48/72/24, matched call group size 8, and separate O/G/C batches. Stratified dynamic sampling with coverage replay. Clip [0.8, 1.28], temperature 0.9, 16-turn deployment horizon, learning rate , LoRA rank/alpha 32/64, and no KL penalty. |
Table 5. Registered post-training configuration.
Metrics and Evidence Protocol
The five component metrics lie in and are reported individually, alongside the descriptive Overall Score defined below. Component claims, shot IDs, matching rules, thresholds, and denominators are frozen before test outputs are inspected.
Script Content Fidelity (SCF). Let be the claims in source script , and the claims in output narration . The entailment indicator tests whether text supports claim , and is the Kendall order correlation of matched source and output claims. We compute
SCF is valid for at least five source claims and one narration unit.
Visual–Narration Semantic Match (VNSM). Let contain the registered shots with a valid evaluator response. For each shot , four uniformly sampled frames form montage , paired with concurrent narration . The held-out Claude Opus 5 evaluator assigns an anchored grade (Poor, Fair, Good, Excellent) using the rubric below. We rescale grades to and aggregate
Timeline Plan Realization (TPR). Planned and realized shots are joined by frozen shot ID:
We set when .
Long-Horizon Consistency (LHC). Shot health averages render success, audio–clip duration agreement, and the rescaled VNSM grade:
Unavailable is omitted and the observable terms are renormalized. For the first and last thirds , and canonical asset digests ,
LHC requires and .
End-to-End Rendering Yield (ERY).
hasNarration requires non-silent speech covering at least half of the planned narration duration. Background music does not qualify. Timeouts, crashes, and unstarted tasks remain in the denominator.
Overall Score (descriptive summary). For each model or system within a comparison track, let denote its reported mean for component . We add a post-hoc descriptive summary of these means:
Each component receives weight 0.2. These weights are not fitted to model rankings or human preferences. The shared [0, 100] scale makes this a transparent macro-average, but does not make the components independent or equally calibrated: LHC already incorporates VNSM in shot health. Overall therefore summarizes task performance rather than serving as a separately validated measure of perceptual video quality. It is distinct from the human-rated overall preference in Table 4.
The displayed scores are computed from the reported, rounded component means and rounded to one decimal place. Each component retains its own eligibility rules and denominator. Overall does not imply a common per-task population. If any component mean is unavailable, Overall is N/A rather than an average over the remaining metrics. Costs are excluded. We report no Overall standard deviation, confidence interval, or significance test: these require aligned run-level results or the cross-metric covariance, which cannot be recovered from the component standard deviations alone. All five components remain visible so that the summary does not conceal trade-offs.
Production Cost Efficiency (PCE). For tool calls or generated tokens ,
PCE is lower-is-better and is never combined with quality. Eligibility counts and validity conditions are recorded alongside metric reporting. Conditional means use valid tasks, whereas unconditional means assign zero to system failures. Harness or provider outages remain missing rather than being converted into low-quality scores.
Diagnostic Validation
We apply three strengths of metric-targeted corruption: claim edits (SCF), time-local visual substitution (VNSM), shot removal/duration/order changes (TPR), tail degradation or asset repetition (LHC), and delivery failure (ERY). For each metric–corruption pair we report the paired drop, bootstrap 95% interval, severity monotonicity, and the 5 × 5 sensitivity matrix. Human pairwise validation is metric-specific and blinded. VNSM additionally reports cross-family judge agreement and human correlation. Agreement below Krippendorff’s prevents a standalone VNSM claim.
Evidence-Seeking Harness
At most three contact sheets are generated per video with sampling step
Tiles carry source-frame and timestamp labels, while a sidecar stores FPS, duration, sampling step, and sheet count. Representative inputs are shown in Appendix § 12. Read-only mounts isolate the evidence available to each metric. SCF sees script and narration, while VNSM sees aligned narration and rendered frames/clips. TPR sees plan and realized timeline. LHC sees ordered health inputs and asset digests. ERY/PCE see delivery probes and accounting. The network-free harness may call only bounded FFmpeg (editing and probing), PySceneDetect (scene detection), Whisper (ASR), PaddleOCR (OCR), frozen detectors, and manifest/diff readers. Tools expose evidence but never scores, and every call is archived.
Held-Out VNSM Evaluator Prompt
The evaluator is called independently for each registered shot. The complete version tuple—prompt, rubric, model snapshot, decoding parameters, sampler, and JSON schema—is frozen after development-only calibration and before any test output is rendered. Figure 6 visualizes the exact evidence path. The four image tiles are frames sampled by the evaluation harness rather than illustrative replacements.

Figure 6. Auditable VNSM evaluator path. A shot packet binds the narration window, timestamps, video and manifest digests, sampler/prompt versions, judge snapshot, and decoding configuration. The frozen sampler exposes four timestamped frames and, only when motion is necessary, one bounded clip. The held-out Claude Opus 5 evaluator must first record visible evidence, then choose one anchored grade and emit schema-validated JSON. Aggregation excludes unavailable calls rather than converting infrastructure failure into a zero-quality judgment.
Held-Out VNSM Evaluator: System Contract
Role. You are an evidence-seeking evaluator for one registered metric. Evaluate only visual–narration semantic correspondence; do not grade overall video quality, aesthetics, editing smoothness, source fidelity, tool use, or hidden production intent.
Immutable inputs. A shot packet (JSON) with criterion=VNSM, case and shot IDs, the narration text active over the registered shot interval, video and manifest SHA-256 digests, sampler and prompt versions, and model snapshot and decoding configuration; and sampled evidence comprising four chronologically ordered frames – with source timestamps (hh:mm:ss) and, on request, one bounded clip. No plan, retrieval query, similarity score, reference video, method name, or training reward is supplied.
Procedure.
Verify that all four frames, their timestamps, and the narration window are present and internally consistent. Otherwise set status to one registered failure code.
Decompose the narration into its central visually testable units: referent, setting, attribute, action, and event. Do not require support for abstract discourse that cannot be depicted.
Inspect every frame in chronological order and record two to four time-local visible observations before assigning a score. Each observation must be paired with one supplied frame timestamp.
For each central unit, decide whether it is directly supported, merely generic/topical, absent, or contradicted. One matching object cannot override a contradictory action or event.
Request the single bounded clip only when motion or temporal order is necessary to distinguish an action. The clip may cover only the registered shot interval and cannot introduce outside context.
Select exactly one anchored score whose full anchor is satisfied. When evidence falls between anchors, choose the lower anchor; never interpolate or invent a new score.
Evidence rules. A generic presenter, publisher card, or decorative overlay alone cannot score above 2 (Fair). OCR text counts as visual support only when the narration is explicitly about displayed text. A correct setting with the wrong central action is at most 2 (Fair); a direct contradiction scores 1 (Poor). Evidence outside the registered narration interval is inadmissible. Never invent a timestamp or infer content absent from the supplied frames or bounded clip. Missing input, decoding failure, or harness/provider failure sets status to a failure code; it is missing evidence, not a Poor score.
Output. Return strict JSON only, with no markdown or free-form text.
Held-Out VNSM Evaluator: Anchors and Output
Anchored rubric.
4 – Excellent (direct and complete): every central visually testable referent/action/event is directly visible in the matching interval, with no material contradiction.
3 – Good (specific but partial): the correct subject and scene are visible and the central referent or action is substantially supported, but one important attribute, phase, or consequence is not verifiable.
2 – Fair (topical or generic): the imagery is in the correct broad topic or setting, but the central action, relation, or specific referent is not visually verified. Generic B-roll belongs here even when aesthetically apt.
1 – Poor (atmospheric, absent, or contradictory): only color, mood, or a weak association relates to the narration, or the imagery is irrelevant, shows the wrong subject/action/event, or materially contradicts the narration.
Required JSON fields.
{
"criterion": "VNSM",
"score": 1|2|3|4,
"evidence": ["<visible observation>", ...],
"timestamps": ["<hh:mm:ss>", ...],
"status": "ok"
}
For status="ok", score is an allowed anchor, and evidence contains 2–4 distinct visible observations, each aligned with one entry of timestamps drawn from the supplied frames. Otherwise, score is null, evidence and timestamps are empty, and status is exactly one of missing_input, digest_-mismatch, sampler_failure, provider_failure, decode_failure, or schema_failure. Case and shot IDs, digests, and version IDs are bound by the shot packet rather than repeated in the output.
Deterministic validation and retry. The evaluator runs at temperature 0 with the frozen decoding parameters shown in the packet. A deterministic validator checks the criterion and score values, the status-dependent fields, timestamp membership, evidence count, and the match between each response and its shot packet. An invalid response receives one schema-only repair request containing the validation errors but no new visual evidence. A second invalid response is recorded as unavailable with schema_failure. No human edits the score or evidence.
Aggregation and claim gate. The held-out Claude Opus 5 evaluator judges each registered shot once. Video-level VNSM is the unweighted mean of rescaled grades over shots with status="ok" and is reported only when . Unavailable calls are excluded rather than scored. A second held-out VLM family independently re-judges the same shots solely to audit agreement, and its grades do not enter VNSM. We report coverage, eligible-shot count, the two family-specific means, rank correlation, and Krippendorff’s . Agreement below blocks a standalone VNSM claim. Before the prompt is frozen, a disjoint 50-shot development set is used for blinded human correlation and controlled corruption tests (wrong subject, wrong action, temporal shift, and generic B-roll). Test tasks never participate in calibration.
Training/evaluation separation. The training-side VLM-Judge is a sampled spotlight audit, used only for reward features and limited to visual_impact and clarity: the former checks whether the animation is visually engaging in layout and aesthetics, while the latter checks whether motion and text convey information without distraction or layout errors. The held-out VNSM evaluator and audit family instead use final rendered evidence, separately versioned models, isolated credentials, and prompts unavailable to the training jobs.
Experimental Protocol and Reporting
External Baseline Contract
Task-matched native systems receive the same frozen script, source pool, duration, aspect ratio, and budget while retaining their released orchestration and tools. Adapters may translate serialization and export native artifacts, but may not retrieve assets, rewrite narration, invent plans, add manual edits, or perform unavailable repairs. We record the plan, call log, artifacts, final video, failure code, time, and cost. A missing native artifact makes its trace-dependent metric N/A. Systems requiring semantic adapter work enter the task-adjacent track and are excluded from the headline matched ranking. The five component-metric entries average five independent runs over the same 100 held-out tasks, with denoting the standard deviation across runs. Overall Score instead averages the five reported component means and is shown without an uncertainty estimate. It is N/A when any component mean is unavailable. Missing components are neither imputed nor omitted from the denominator.
| System | Paradigm | Fixed narration | Shot plan | Trace / repair | Fixed sources | Matched native track |
| Crayotter | Artifact-centric multi-agent | ✓ | ✓ | ✓ | ✓ | ✓ |
| VideoAgent | Large agent library | ✓ | ✓ | ✓ | ✓ | ✓ |
| CutClaw | Specialized multi-agent | ✗ | ✓ | ✓ | ✓ | ✗ |
| CineAgents | Design-and-compose agents | ✗ | ✓ | ✓ | ✓ | ✗ |
| DirectorAgent | Script-to-video generation | ✗ | ✓ | ✗ | ✗ | ✗ |
Table 6. Baseline task-contract audit. Systems satisfying every required contract element enter the task-matched native-system comparison.
Plan–Render Visualization Protocol
Qualitative examples are selected from registered length, domain, and failure strata before method identities are revealed. The native-system panel fixes rows to Crayotter and WeVisCut and uses one common narration and source pool. Each row shows five rendered keyframes at identical absolute timestamps and a compact evidence strip restricted to visible properties. Missing frames remain visibly missing and an invalid delivery retains its terminal failure stage. Additional systems, when available under the same contract, appear in separate panels rather than changing the frozen comparison. Visualizations explain the registered metrics but do not re-score or reorder examples.
Depth and Real-Tool Diagnostics
For a subset of tasks, the evaluation protocol injects a registered late-stage fault and records the repair branch . The shared case ID links the script, shot plan, original execution, delivered video, and changes made during repair, allowing recovery behavior to be inspected alongside normal production.
For horizon and first failure turn ,
Completion and repair at depth use the same eligible denominator. The proxy–real gap is a paired mean between frozen-fixture replay and production execution. Unmatched or failed runs are retained in reporting. These depth and proxy–real diagnostics complete the registered protocol. The experiment tables report the five registered metrics and the descriptive Overall Score.
A Representative Autonomous Tool-Use Flow
This appendix illustrates the inference-time behavior of WeVisCut with one complete production trace. The purpose is not to expose the full generated screenplay or every JSON payload, but to show the closed-loop decision pattern used by the policy: it first builds a long-horizon production plan, then executes real tools, observes typed runtime returns, repairs bounded failures, satisfies missing prerequisites, and terminates only after the video has actually been delivered.
The raw trace is stored as agent_trace.jsonl and the delivery record as stitch_manifest.json. The transcript below is a faithful, translated rendering of that trace, organized as
Thinking → Observation → Tool Call → Runtime Return → Next Decision.
The Thinking field reports the deliberation the policy emits before committing to a tool action. It is part of the recorded action interface rather than hidden reasoning: every claim it makes about the world is either grounded in a previous typed return or is falsified by the next one, which is precisely what makes the trace usable as training supervision.
Representative Autonomous Trace: Plan–Act–Repair–Deliver
Run card. The run identifier is aicut_20260824_152401_9f7f9d. The input is a fixed long-form narration about aircraft science and engineering, tokenized by the runtime into 51 semantic units (u01–u51). The finished project contains 6 screenplay segments, 30 realized shots, 16 spotlight shots, and 3 background-music tracks. The policy runs for 16 tool turns over 31 minutes of wall-clock time and delivers a 395.0-second video:
final_video_with_bgm_20260824_155554.mp4
Build the high-level production plan
Thinking. I hold nothing but the narration. Every downstream tool in this runtime is gated on structure that does not exist yet, so the only admissible first action is the one that turns free text into a typed screenplay. Two choices matter here. First, granularity: the narration runs several hundred seconds, and a segment count near six keeps each act long enough to carry an argument while still giving the shot planner meaningful boundaries to respect. Second, I should request the global music style now rather than at delivery time, because the stitcher will need a music query long after the narration’s tone is still in context, and recovering that tone later from rendered shots would be lossy. I also pin narration order and full-script coverage, since any dropped span becomes a semantic hole that no later repair tool is able to detect.
Observation. Only the raw narration is available. No shot plan, assets, audio, templates, rendered paths, or delivery artifacts exist.
Call.
Tool_SegmentArticle(
num_segments_target=6,
segment_granularity="medium",
preserve_narration_order=True,
cover_full_script=True,
need_music_style=True
)
Return. status=success. The runtime returns a six-segment outline — a prologue (“Looking Up in Wonder”), four core acts (“A Precision Airborne Ecosystem”, “Extreme Lightweighting and Flexible Wisdom”, “Eyes of Technology and Invisible Guardians”, “From Adventure to Everyday”), and a coda (“A Love Letter to the Sky”) — together with a per-segment narration script, a planner strategy, and the global music style: “epic orchestral, piano lead, strings underneath, tempo moving from slow to fast, grand and warm”.
Next decision. Convert the outline into executable shot-level production cards.
Expand the plan into shot-level production cards
Thinking. The segment outline is an argument, not a shot list. Nothing downstream can consume it: retrieval needs a visual query per shot, synthesis needs a vocal script per shot, and the renderer needs durations and identifiers. I set the shot target well above what I expect to keep, because the planner is allowed to merge and drop but not to invent coverage, and an over-request costs nothing at this stage while an under-request silently truncates the tail of the script. I forbid repeated visual queries, since duplicate queries collapse to the same retrieved asset and produce visible repetition that no per-shot quality check will flag — the failure only becomes obvious in the stitched cut, which is far too late to fix cheaply. I also want spotlight marks assigned here rather than later, because the decoration and rendering budget is the scarcest resource in the run and I would rather allocate it from a global view of the script than greedily, shot by shot.
Observation. A segment-level plan exists, but the runtime still lacks shot IDs, visual queries, durations, spotlight marks, and coverage records.
Call.
Tool_PlanShots(
mode="plan_shots_all",
num_shots_target=50,
planning_policy={
"preserve_narration_order": True,
"avoid_repeated_visual_query": True
}
)
Return. status=success. The planner realizes 30 shots spanning 397.4 planned seconds, distributed 2/2/2/7/10/7 across the six segments, and marks 16 of them as spotlight shots. The union of covered_unit_ids is u01–u51, i.e. semantic coverage is complete with no uncovered narration unit.
Next decision. Materialize the planned shots with real assets and narration audio.
Execute batched visual retrieval and narration synthesis
Thinking. Retrieval and text-to-speech are the two slowest operations in the pipeline and they are independent per shot, so issuing them one shot at a time would waste most of the run’s wall-clock budget on serialization. I dispatch all 30 shots in a single batch. This is also the first turn where the plan meets the real asset library, which means it is the first turn where my assumptions can be falsified: a visual query that reads well as prose may still retrieve nothing usable. I therefore care less about the average match score in the return than about whether any shot comes back with an empty candidate set, because a partial batch would force me to choose between re-querying and accepting a hole in the cut.
Observation. The shot plan is complete, but no media artifacts exist.
Call.
Tool_BatchVisualAudio(
shot_ids=["1", "2", ..., "30"]
)
Return. status=success for all 30 shots. Every shot receives a clip path, a source duration, a match score, and a non-empty candidate set — for example shot 2 retrieves at match_score = 0.592 from 12 candidates and shot 5 at 0.611 from 12 candidates, while the weakest, shot 1, still returns 0.452 from 5 candidates. All shots report playlist_ready=true. The batch takes roughly 14 minutes, the single most expensive turn in the run.
Next decision. Spend the decoration and rendering-template budget only on spotlight shots.
Decorate the spotlight shots
Thinking. Decoration is what separates a slideshow from an edited video, but it is also the most expensive thing I can ask for, both in template rendering and in the downstream quality checks it triggers. The 16 spotlight marks from step 2 are exactly the allocation decision I already made, so I honor them and set spotlight_only rather than re-deciding now with less context than the planner had. I ask for slot verification in the same call, because a template whose slots are filled but semantically wrong is worse than no template at all: it renders cleanly, passes structural checks, and puts an incorrect caption on screen.
Observation. Shot assets and narration audio exist for all 30 shots, and the 16 spotlight IDs are known from the plan.
Call.
Tool_BatchDecorateSpotlights(
shot_ids=["1","2","5","6","7","10","11","13",
"14","16","19","21","22","23","26","28"],
template_policy={
"select_template": True,
"fill_slots": True,
"verify_slots": True,
"spotlight_only": True
}
)
Return. status=success, decorated_count=16, failed_count=0, spotlight_count=16. Templates are selected per rhetorical role: hook_title_intro for the opening shot, definition_card for the Bernoulli explanation, key_takeaway for the materials act, analogy_card for the winglet–rowing-blade comparison, and historical_moment for the aviation-history shot. Critically, verified_count=0: slot verification was requested but produced no verdicts, and every verify_result is an empty object.
Next decision. The decoration is structurally complete but semantically unverified. Inspect and improve the plan before paying for the final render.
Encounter a quality-warning state
Thinking. The zero verification count from the previous return is the loose thread in this run. I have 16 decorated shots that nothing has actually judged, and the render stage will happily bake whatever text is in those slots into the frames. Rendering first and inspecting afterwards would be the expensive ordering, so I inspect now, over all 30 shots rather than only the decorated ones, since a visual mismatch on a plain shot is just as damaging in the final cut. I scope the repair to the two dimensions that decoration can actually move — decoration quality and visual match — and request minimal repair with an immediate re-judge, so that whatever the tool changes is validated in the same call instead of drifting unchecked into the render.
Observation. The decorated plan carries no verification verdicts.
Call.
Tool_Stage4DimensionalRepair(
failed_shot_ids=["1", "2", ..., "30"],
target_dimensions=["decoration_quality", "visual_match"],
minimal_repair=True,
rejudge_after_repair=True,
repair_round=0
)
RETURN. status=completed_with_warnings after 189.9 seconds: passed_count=0, failed_count=30. Every shot remains below the strict quality threshold.
NEXT DECISION. A blanket failure over all 30 shots indicates missing evidence rather than providing a per-shot defect report. Produce explicit self-reflection findings before attempting another patch.
## Hit a typed precondition error
THINKING. I want to patch the plan against the reflection findings, and reward shaping is the augmentation I would like applied so that both passing and failing shots receive graded per-dimension scores rather than a binary verdict. What I am about to do, however, is issue synthetic issue identifiers of my own invention. This is worth recording as an error rather than hiding, because it is the exact mistake the runtime’s typed preconditions exist to catch: I am attempting to repair against evidence I have not yet collected.
OBSERVATION. No `self_reflect_issues` object exists in the state.
CALL.
```python
Tool_SelfReflect_Patch(
augmentation_strategy={
"id": "aug09_reward_shaping",
"name": "Reward Shaping",
"focus": "Emit finer-grained reward signal: score every dimension for
both passing and failing shots, and state the preference
basis used for SFT/RL."
},
issue_ids=["sr_001", "sr_002", "sr_003", "sr_004", "sr_005"],
target_shot_ids=["1", "2", "3", "4", "5"],
minimal_repair=True,
requires_downstream_propagation_hint=True
)
RETURN. status=blocked_precondition, missing=["self_reflect_issues"]. The runtime rejects the call with “insufficient preconditions for Tool_SelfReflect_Patch”.
NEXT DECISION. Treat the rejection as an instruction rather than a failure: run reflection first and patch against the identifiers it actually returns.
Generate the missing self-reflection evidence
THINKING. The precondition told me exactly which object is missing, so this turn is fully determined—the only real choice left is what to inspect for. I widen the focus beyond decoration to the five dimensions where an error would be unrecoverable after rendering: script coverage, shot rhythm, visual-query alignment, spotlight ratio, and duplication. I deliberately switch the augmentation strategy from reward shaping to strict semantic coverage. Reward shaping produces a smoother signal, but at this moment I do not need a smoother signal. I need a judge that would rather fail a shot than let a semantic omission through, because a missed narration unit cannot be repaired once the timeline is composed.
OBSERVATION. The patch tool is blocked, and no reflection object exists.
CALL.
Tool_SelfReflect(
scope="full_repair_with_downstream_propagation",
num_segments=6,
num_shots=30,
focus_dimensions=["script_coverage", "shot_rhythm",
"visual_query_alignment", "spotlight_ratio",
"duplication"],
mode="gemini_augmentation",
augmentation_strategy={
"id": "aug01_semantic_coverage_strict",
"name": "Semantic Coverage Strict",
"focus": "Re-audit whether the narration covers the core information
of the source; prefer failing a shot over letting a semantic
omission pass."
}
)
RETURN. status=needs_fix, passed=false, overall_score = 0.70. The judge emits 13 concrete issues: 11 decoration_quality findings (shots 5, 6, 7, 10, 11, 14, 16, 21, 23, 26, 28) and 2 visual_match findings (shots 14 and 28), all at medium severity, with suggested_fix="redecorate". The summary reads “13 decoration-quality problems found”, and the state now records repair_history=[{round: 0, passed: false, issue_count: 13}].
NEXT DECISION. Patch exactly these 13 issues. The blanket 30-shot failure of step 5 has been resolved into a bounded, addressable defect list.
Patch the plan against the returned issue identifiers
THINKING. Now I can issue the call I attempted two turns ago, but with the identifiers the judge actually produced instead of ones I made up. I keep the strict coverage strategy so that the patch is evaluated by the same standard that produced the findings. Switching strategies mid-repair would mean patching against one rubric and being graded by another. I set minimal repair, because the failures are concentrated in decoration slots and visual queries, and any edit that ripples into narration text would invalidate the audio I already synthesized in step 3 at a cost of 14 minutes. I request downstream propagation hints so the runtime marks which artifacts the patch has invalidated rather than leaving me to infer it.
OBSERVATION. Thirteen typed issues exist over 13 distinct shots.
CALL.
Tool_SelfReflect_Patch(
issue_ids=["qj_5_decoration_quality", "qj_6_decoration_quality",
"qj_7_decoration_quality", "qj_10_decoration_quality",
"qj_11_decoration_quality", "qj_14_visual_match",
"qj_14_decoration_quality", "qj_16_decoration_quality",
"qj_21_decoration_quality", "qj_23_decoration_quality",
"qj_26_decoration_quality", "qj_28_visual_match",
"qj_28_decoration_quality"],
target_shot_ids=["1","2","5","6","7","10","11",
"14","16","21","23","26","28"],
minimal_repair=True,
requires_downstream_propagation_hint=True,
augmentation_strategy={"id": "aug01_semantic_coverage_strict", ...}
)
RETURN. status=patched, repair_applied=true. The patch is indeed localized: visual queries are sharpened (shot 3’s query gains the explicit “conveying precision and complexity” qualifier), and shot 2’s duration is re-estimated from 61.11 s to 60.70 s. Narration scripts, segment boundaries, and spotlight assignments are left untouched, so the synthesized audio remains valid.
Next decision. Re-run decoration so the repaired shots pick up new templates and slots.
Re-decorate under the repaired plan
Thinking. The patch changed the plan, not the rendered overlays, so the decorations currently in state were selected against text that no longer exists. I re-run decoration over the full spotlight set rather than only the 13 patched shots: template selection is a global assignment, and if a patched shot now claims a template that an unpatched neighbour is already using, restricting the call to the patched subset would leave that collision in place. The cost is the same either way, since decoration is cheap relative to rendering.
Observation. The plan has been patched but the spotlight decorations still reflect the pre-patch text.
Call.
Tool_BatchDecorateSpotlights(
shot_ids=["1","2","5","6","7","10","11","13",
"14","16","19","21","22","23","26","28"],
template_policy={
"select_template": True,
"fill_slots": True,
"verify_slots": True,
"spotlight_only": True
}
)
Return. status=success, decorated_count=16, failed_count=0. The re-decoration is visibly better targeted: shot 2’s definition now states both surfaces (“upper surface faster and lower pressure, lower surface slower and higher pressure”) instead of only the upper one. Shot 5’s takeaways are compressed from full clauses into three scannable phrases. Shot 6 moves from a generic overlay to a concept_explanation type, and shot 10 switches to the more specific historical_moment_card template.
Next decision. Re-judge the repaired, re-decorated state.
Re-judge all shots against the full dimension set
Thinking. This is the verification half of the repair loop, and I want it against all five delivery dimensions rather than the two I repaired, because a decoration edit can break timing alignment even when it improves decoration quality. I hold the threshold at the strict 0.75 and focus the judge on decoration slots, which is where the patch actually landed.
Observation. Thirteen shots have been patched and all 16 spotlight shots re-decorated.
Call.
Tool_Stage4DimensionJudge(
shot_ids=["1", "2", ..., "30"],
dimensions=["visual_match", "audio_sync", "decoration_quality",
"timing_alignment", "render_integrity"],
threshold=0.75,
focus_decoration_slots=True
)
Return. status=completed_with_warnings, passed_count=0, failed_count=30, _elapsed_ sec=0.0.
Next decision. The zero elapsed time is the informative part: the judge did not re-evaluate, it replayed a cached verdict. Two of the five requested dimensions (render_integrity and timing_alignment) cannot be scored before a render exists, so this verdict is structurally unable to improve at this stage.
Probe whether the failure is real or structural
Thinking. Before I spend the rest of the repair budget, I need to distinguish a genuine quality failure from a stage-ordering artifact. I re-issue the step 5 repair verbatim. If the state has actually changed, the tool should now do work and take time. If it returns instantly with the same counts, the verdict is not tracking my edits and further repair at this stage is pure waste.
Observation. The re-judge reports the same blanket failure it reported before any repair.
Call.
Tool_Stage4DimensionalRepair(
failed_shot_ids=["1", "2", ..., "30"],
target_dimensions=["decoration_quality", "visual_match"],
minimal_repair=True,
rejudge_after_repair=True,
repair_round=0
)
Return. status=completed_with_warnings, passed_count=0, failed_count=30, _elapsed_sec=0.0 — identical counts, identical zero cost. The hypothesis is confirmed: the warning is structural.
Next decision. Make one final, explicitly budgeted repair attempt so that the runtime, not the policy, decides when to stop.
Exhaust the repair budget deliberately
Thinking. I could loop here indefinitely, and that is precisely the failure mode I must avoid: a repair loop that never converges consumes the entire run and delivers nothing. So I escalate rather than repeat — I declare this repair_round=2, the last one the runtime permits, and widen the target to all five dimensions so that if any residual defect is genuinely fixable at plan level, this call has the broadest possible chance to fix it. Setting the round explicitly hands the stopping decision to the runtime’s budget accounting instead of to my own judgment, which is the safer place for it.
Observation. Two repair calls have produced no state change at zero cost.
Call.
Tool_Stage4DimensionalRepair(
failed_shot_ids=["1", "2", ..., "30"],
target_dimensions=["visual_match", "audio_sync", "decoration_quality",
"timing_alignment", "render_integrity"],
minimal_repair=True,
rejudge_after_repair=True,
repair_round=2
)
Return. status=budget_exhausted: “the repair limit of 2 has been reached; remaining warnings will not be repaired and current quality is treated as passing (rejection 1 of 2)”.
Next decision. The quality gate is closed by budget rather than by score. Stop repairing and move to delivery.
Attempt delivery and learn the real blocker
Thinking. With the gate closed I go for the final artifact. I set final_gate_required so the runtime enforces its own delivery contract rather than trusting my belief that 30 shots are approved, and I pre-declare the audio mix — background music at half volume, ducked under the voice track — because the narration is the payload and music that competes with it would ruin an otherwise correct cut. I expect this call to succeed, and if it does not, its rejection will tell me exactly what the state is still missing.
Observation. All 30 shots are treated as approved.
Call.
Tool_StitchVideo(
shot_count=30,
approved_count=30,
final_gate_required=True,
bgm_volume=0.5,
duck_under_voice=True
)
Return. status=blocked_precondition, missing=["timeline_render", "bgm", "rendered_path_ready:0/30"].
Next decision. This is the decisive observation of the run, and it retroactively explains steps 10–12: rendered_path_ready:0/30 means no shot has ever been rendered, which is why render_integrity could never pass and why the repair loop kept failing on a defect that plan-level edits could not reach. The correct action was never more repair — it was to satisfy the two unmet prerequisites, timeline rendering and background music, in that order.
Satisfy the first prerequisite: compose and render timelines
Thinking. This is the expensive, irreversible turn, so I want the runtime to refuse it rather than produce a subtly broken render. I require both that every shot have its retrieved visual and synthesized audio present, and that every spotlight shot carry a decoration — the second condition is what makes the re-decoration work of step 9 actually reach the frames instead of sitting unused in state. I compose all 30 in one batch so that timing offsets are computed against a single consistent timeline rather than accumulated shot by shot.
Observation. Assets, audio, and decorations exist in state, but no rendered paths do.
Call.
Tool_BatchTimelineCompose(
shot_ids=["1", "2", ..., "30"],
require_tts_visual_audio=True,
require_spotlight_decoration=True
)
Return. status=success after 378.6 seconds: composed=30, render_success_count=30, failed_count=0, concat_count=16. The renderer routes by decoration need — the 16 spotlight shots go through the HyperFrames backend for template compositing, the remaining 14 through the direct ffmpeg path.
Next decision. One prerequisite remains: background music.
Satisfy the second prerequisite: retrieve background music
Thinking. The music style I requested back in step 1 is still in state, which is why this turn is a lookup rather than a re-derivation. I query with the full style description instead of a keyword, since the retrieval index is caption-based and the tonal arc — slow building to fast — is what distinguishes a usable track from a merely orchestral one. I request more duration than the video needs so the mixer has slack for crossfades and never runs out of music before the last shot ends. Setting must_finish_before_stitch makes the ordering constraint explicit rather than implicit in my turn sequence.
Observation. Thirty rendered shots exist, but the bgm slot is still empty.
Call.
Tool_MusicSearch(
query="epic orchestral, piano lead, strings underneath, tempo from slow
to fast, grand and warm",
music_style="epic orchestral, piano lead, strings underneath, tempo from
slow to fast, grand and warm",
target_duration_sec=713.71,
must_finish_before_stitch=True
)
Return. status=success, selected_count=3. The runtime returns a sequenced playlist rather than three independent tracks, each with a placed offset: a 60.0 s cinematic orchestral opener at [0, 60.0], a 99.9 s lyrical orchestral piece at [60.0, 159.9], and a 326.6 s track whose caption reads “tempo moving from slow to urgent, emotion building” at [159.9, 486.5]. Total coverage of 486.5 s comfortably exceeds the cut.
Next decision. Both prerequisites are now satisfied. Retry the final gate.
Complete the autonomous final stitch
Thinking. I re-issue the exact call the runtime rejected in step 13, unchanged. That it is unchanged is the point: the call was never wrong, the state was incomplete, and the two intervening turns closed precisely the gap the rejection named. I keep final_gate_required set so that delivery is validated rather than asserted, and I will only consider this run finished if the return contains a real file path and a ready count of 30.
Observation. The state now contains 30 rendered shots with timeline paths, narration audio, spotlight decorations, and a three-track music playlist.
Call.
Tool_StitchVideo(
shot_count=30,
approved_count=30,
final_gate_required=True,
bgm_volume=0.5,
duck_under_voice=True
)
Return. status=success after 50.1 seconds. The manifest reports ready_shot_count=30, missing_shots=[], validation_errors=[], and duration_ok=true on every shot, with 394.98 s of stitched video carrying 390.48 s of narration. concat_status=success and bgm_status=bgm_mixed across 3 tracks with ducking_fallback_used=false, yielding a final 395.0-second delivery:
final_video_with_bgm_20260824_155554.mp4
Takeaway. This trajectory exhibits the target inference pattern end to end. The policy constructs a long-horizon production plan before acting, executes real tools, and treats every runtime failure as a structured observation rather than a terminal condition. The invented issue identifiers of step 6 are corrected by running reflection first. The blanket quality failure of step 5 is resolved into 13 addressable defects by step 7. The non-converging repair loop of steps 10–12 is diagnosed only when the delivery gate at step 13 names rendered_path_ready:0/30 as the true blocker. The policy then satisfies both missing prerequisites and terminates only after the delivered artifact exists on disk. Notably, the run consumes its full repair budget without ever entering an unbounded loop, and the final recovery suffix is a three-call sequence that repeats no earlier work:
Tool_BatchTimelineCompose
-> Tool_MusicSearch
-> Tool_StitchVideo (identical arguments to the rejected step 13)
Visual Inputs to the Evaluation Harness
Figure 7 and Figure 8 show two representative contact-sheet inputs provided to the evaluation harness. Each contact sheet contains temporally sampled video frames annotated with source-frame indices and timestamps, allowing the judges to inspect local visual consistency, timeline progression, long-range coherence, and rendering quality. These inputs are exposed through read-only, network-free tools and archived together with their metadata, ensuring that metric decisions are grounded in traceable visual evidence rather than evaluator-side assumptions.

Figure 7. Visual Input to the Evaluation Harness: Case 1. A contact sheet of temporally sampled and timestamped video frames used for evidence-grounded diagnostic evaluation.

Figure 8. Visual Input to the Evaluation Harness: Case 2.
Limitations
WeVisCut is limited by the current HyperFrames template library, so uncommon visual languages may be approximated using the closest available structure, motion, and style. Expanding the library through automatic template generation could broaden the design space. The dataset and project-disjoint split cannot cover all domains, editing conventions, runtimes, or asset distributions, while verified replay guarantees executability only within the registered runtime. The quality of long-form videos also depends on the stability of the video retriever, particularly for instruction-based retrieval. Integrating a trained video generation model could reduce this dependency and improve robustness. Finally, VNSM relies on a VLM. Although evidence isolation, frozen anchors, agreement reporting, and human calibration improve auditability, they cannot eliminate judge bias or service drift, motivating broader blinded studies and cross-family audits.
References
- [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024.arxiv.org/abs/2306.13649
- [2]Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, 2024.arxiv.org/abs/2403.12881
- [3]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023.arxiv.org/abs/2306.06070
- [4]Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024.arxiv.org/abs/2403.07718
- [5]Google DeepMind. Gemini 3.1 pro model card. Model card, 2026. URL https://deepmind.google/models/model-cards/gemini-3-1-pro. Published 2026-02-19.
- [6]Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videocore: Building automatic metrics to simulate fine-grained human feedback for video generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2105–2123, 2024.arxiv.org/abs/2406.15252
- [7]HeyGen. Hyperframes: Write html. render video. built for agents. https://github.com/heygen-com/hyperframes, 2026. Accessed: Sep. 30, 2026.
- [8]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. CodeLora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
- [9]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818. IEEE, 2024.arxiv.org/abs/2311.17982
- [10]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024.arxiv.org/abs/2310.06770
- [11]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuya Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881–905, 2024.
- [12]Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, et al. On-policy distillation with curriculum turn-level guidance for multi-turn agents. arXiv preprint arXiv:2606.15912, 2026.arxiv.org/abs/2606.15912
- [13]Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 3102–3116, 2023.arxiv.org/abs/2304.08244
- [14]Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu, Youliang Zhang, Yuan Zhou, Xin Li, Qinglin Lu, and Zhibo Chen. Goku: A million-scale universal dataset and benchmark for instruction-based video editing. arXiv preprint arXiv:2606.30599, 2026. URL https://arxiv.org/abs/2606.30599.
- [15]Xinran Ling, Chen Zhu, Meiqi Wu, Hangyu Li, Xiaokun Feng, Cundian Yang, Aiming Hao, Jiashu Zhu, Jiahong Wu, and Xiangxiang Chu. Vmbench: A benchmark for perception-aligned video motion generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13087–13098. IEEE, 2025.
- [16]Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, et al. Toolace: Winning the points of llm function calling. In International conference on learning representations, volume 2025, pages 41359–41381, 2025.arxiv.org/abs/2409.00920
- [17]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pages 52989–53046, 2024.arxiv.org/abs/2308.03688
- [18]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, 2025.
- [19]Chang Ma, Junlei Zhang, Zihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325–74362, 2024.arxiv.org/abs/2401.13178
- [20]Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502, 2024.arxiv.org/abs/2407.03502
- [21]Chenyu Mu, Xin He, Qu Yang, Wanshun Chen, Jiadi Yao, Huang Liu, Zihao Yi, Bo Zhao, Xingyu Chen, Ruotian Ma, et al. The script is all you need: An agentic framework for long-horizon dialogue-to-cinematic video generation. arXiv preprint arXiv:2601.17737, 2026.arxiv.org/abs/2601.17737
- [22]Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544–126565, 2024.arxiv.org/abs/2305.15334
- [23]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations, volume 2024, pages 9695–9717, 2024.arxiv.org/abs/2307.16789
- [24]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [25]Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026. URL https://qwen.ai/blog?id=qwen3.6-27b.
- [26]Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026. URL https://qwen.ai/blog?id=qwen3.6-35b-a3b.
- [27]Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708–59728, 2023.DOI
- [28]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
- [29]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Răileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023.arxiv.org/abs/2302.04761
- [30]John Schulman, Filip Wolski, Prafulla Dhar̄iwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
- [31]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [32]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
- [33]Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, et al. Step-level on-policy distillation: Interpolating between on-policy distillation and supervised fine-tuning. arXiv preprint arXiv:2608.16333, 2026.arxiv.org/abs/2608.16333
- [34]Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, and Raj Sodhi. Lave: Llm-powered agent assistance and language augmentation for video editing. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 699–714, 2024.arxiv.org/abs/2402.10294
- [35]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.arxiv.org/abs/2305.16291
- [36]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024.arxiv.org/abs/2404.07972
- [37]Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In International Conference on Learning Representations, volume 2025, pages 79822–79843, 2025.arxiv.org/abs/2412.09605
- [38]Lecheng Yan, Jianze Lin, Yichong Zhang, Ben Pan, Wenxi Li, Chenyang Lyu, Liting Zhou, and Cathal Gurrin. Crayotter: Learning long-horizon video editing agents via group-relative preference backpropagation. arXiv preprint arXiv:2608.02694, 2026.arxiv.org/abs/2608.02694
- [39]Lecheng Yan, Yichong Zhang, Xiantao Xu, Jianze Lin, Ben Pan, Xiaoyu Zheng, Jiawei Qian, Anqi Wu, Jiahui Geng, Ruizhe Li, et al. Crayotter: Traceable multi-agent workflows for long-form video editing. arXiv preprint arXiv:2606.07636, 2026.arxiv.org/abs/2606.07636
- [40]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022.arxiv.org/abs/2207.01206
- [41]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
- [42]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.arxiv.org/abs/2406.12045
- [43]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476
- [44]Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, and Jiebo Luo. Aurora: Unified video editing with a tool-using agent. arXiv preprint arXiv:2605.18748, 2026.arxiv.org/abs/2605.18748
- [45]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023.arxiv.org/abs/2308.01825
- [46]Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, 2024.arxiv.org/abs/2310.12823
- [47]Peixuan Zhang, Chang Zhou, Ziyuan Zhang, Hualuo Liu, Chunjie Zhang, Jingqi Liu, Xiaohui Zhou, Xi Chen, Shuchen Weng, Si Li, et al. A benchmark and multi-agent system for instruction-driven cinematic video compilation. arXiv preprint arXiv:2604.10456, 2026.arxiv.org/abs/2604.10456
- [48]Shifang Zhao, Yihan Hu, Ying Shan, Yunchao Wei, and Xiaodong Cun. Cutclaw: Agentic hours-long video editing via music synchronization. arXiv preprint arXiv:2603.29664, 2026.arxiv.org/abs/2603.29664
- [49]Hengji Zhou, Lingxuan Huang, Jian Wang, Bing Zhou, Si Wu, Lianghao Xia, and Chao Huang. Videoagent: All-in-one framework for video understanding and editing. arXiv preprint arXiv:2606.23327, 2026.arxiv.org/abs/2606.23327
- [50]Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abhishek Sridhar, Xianyi Cheng, Tianyue Qu, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606, 2024.arxiv.org/abs/2307.13854
- [51]Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, and Jingjing Chen. Turnopd: Making on-policy distillation turn-aware for efficient long-horizon agent training. arXiv preprint arXiv:2607.05804, 2026.arxiv.org/abs/2607.05804
