betaFeedback

WorksComputer Science

WeVisCut: The Whole Editing Room in One Agent From Narration to Long-Form Video

Abstract

Producing long-form videos from narration requires interdependent decisions that remain coherent across shots. Existing agentic editors often use multi-stage workflows driven by general models, leaving the quality-cost trade-off of compact, task-adapted policies unclear. We address this gap with WeVisCut, a long-form script-to-video system that connects tool execution with policy learning. Its agent harness maintains shared production state so that tool feedback informs subsequent decisions and repairs. Because finished videos do not expose these decisions, our four-stage data pipeline combines editing inversion with verified replay to turn recovered editing intent into executable supervision. Applied to 3,750 finished videos, this pipeline yields over 130K training examples. We use this supervision to train a single 9B policy that coordinates production and recovers from failures within the same harness. Moreover, we introduce a 100-task benchmark and five metrics assessing editing quality and delivery reliability. Under a shared tool environment, WeVisCut substantially outperforms its base model and achieves comparable performance to Claude Opus 5 while using nearly 33x fewer generated tokens per successful output minute. It also outperforms task-matched native editors on our proposed metrics.

Project Page: https://tencent.github.io/WeVisCut

Code: https://github.com/Tencent/WeVisCut

Weights: https://huggingface.co/tencent/WeVisCut

Overview of WeVisCut: script-driven stateful video production, baseline comparisons, and examples across educational domains

Figure 1. Overview of WeVisCut. Top: Script-driven stateful video production (planning, retrieval, composition, inspection, repair) with multi-round edits. Middle: Baseline comparisons in time, cost, preference, and reliability. Bottom: Examples in science, history, geography, tech, and health.

Introduction

“The whole eloquence of cinema is achieved in the editing room.”

— Orson Welles

Creating a narrated long-form video involves turning a script into a sequence of coherent shots. Planning, visual retrieval, narration synthesis, and timeline assembly are interdependent. Changing a shot’s duration affects later timestamps, while unsuitable visual material may require both retrieval and planning to be revisited across the affected shot sequence.

Recent agentic editors coordinate production through specialized agents and tool workflows [21, 39, 44, 47, 48, 49]. Several systems use general models with feedback or graph refinement. Yet orchestration alone does not specify how to train a compact policy that maintains video quality at low token cost. Crayotter trains a 9B editing model using final-video preferences on manually constructed tasks [38]. Such a policy needs a stateful environment for execution and recovery, together with supervision grounded in that environment. Finished videos, however, omit the tool calls, intermediate artifacts, and failures needed for such supervision. The challenge is therefore to connect stateful execution, executable supervision, and policy learning.

To address these challenges, we present WeVisCut, which integrates a script-to-video agent harness, executable data construction, and a task-adapted policy. The harness (Figure 2) maintains shared production state and validates tool actions. It exposes execution feedback for inspection, bounded repair, and replanning while preserving unaffected artifacts from earlier successful production steps.

The WeVisCut agent harness with stateful tool selection

Figure 2. The WeVisCut agent harness with stateful tool selection. A shared policy chooses legal tools from the evolving production state. Validated returns update artifacts and inform inspection, local repair, and replanning before final stitching and delivery.

Within this harness, we construct executable supervision through the four-stage pipeline in Figure 3. We first recover script–shot intent through editing inversion and execute it through verified replay. When replay reveals defects, we apply observe–reflect–repair for bounded corrections that preserve successful artifacts. We then use trajectory materialization to organize the recorded executions into training examples with tool feedback and repair outcomes. From 3,750 video projects, this process yields over 130K examples.

Using these examples, we train a single 9B policy across production roles in three phases (Figure 4). In Phase 1, we establish tool use and recovery through unified supervised fine-tuning [46] and runtime-verified rejection-sampled fine-tuning [45]. In Phase 2, we apply Guided-OPD [12] to learn recovery on policy-reached states while phasing out teacher intervention. In Phase 3, we use RCF-DAPO, our role- and context-factored DAPO [43], to optimize decisions by comparing runtime outcomes within matched roles and contexts.

Figure 4. Three training phases of WeVisCut. Phase 1 learns tool use and recovery through unified SFT and runtime-verified RFT. Phase 2 applies Guided-OPD with decreasing teacher intervention. Phase 3 uses RCF-DAPO to compare alternatives within matched roles and decision contexts.

Moreover, we introduce a benchmark of 100 script-to-video tasks and five complementary metrics to evaluate editing quality and delivery reliability. These metrics relate rendered videos to their scripts and production plans, so evaluation goes beyond successful rendering alone. In a shared tool environment, our policy improves over its base model by 16.0 points in Overall Score and achieves comparable performance to Claude Opus 5 while using nearly 33× fewer generated tokens per successful output minute. Our complete system also outperforms task-matched native editors.

We summarize our contributions as follows.

  1. We design a script-to-video agent harness that maintains shared production state, validates tool actions, and supports inspection, bounded repair, and final delivery.

  1. We build a four-stage data pipeline based on editing inversion and verified replay, converting finished videos into executable training examples.

  1. We train a unified 9B policy for tool use and recovery through sequential SFT, runtime-verified RFT, Guided-OPD, and RCF-DAPO within a shared production harness.

  1. We introduce a 100-task benchmark and five metrics. WeVisCut improves its base policy, outperforms task-matched systems, and achieves comparable quality to Claude Opus 5 at markedly lower cost.

Tool agents and long-horizon post-training. Producing long-form videos requires agents to choose tools and revise plans from execution feedback. Earlier methods range from interleaved reasoning and action [41] to learned API calls [22, 23, 29] and runnable tool dialogues [13]. To train such behavior, researchers curate interactions and synthesize instructions or tool calls [2, 16, 20, 46]. Reflection and reusable skills also support recovery [32, 35]. Closer to our data pipeline, AgentTrek builds verified web trajectories from tutorials [37]. Finished videos, however, do not record editing actions. We therefore infer script–shot intent and replay it with production tools to obtain executable trajectories. Long-horizon training also needs to handle policy-generated states. DAgger [28] collects expert labels for learner states, while generalized knowledge distillation trains on student-generated sequences [1]. Guided-OPD mixes teacher and student turns, while TurnOPD and Step-Level OPD refine supervision at turn and step boundaries [12, 33, 51]. We apply Guided-OPD with truncated teacher logits. For reward optimization, GRPO and DAPO build on PPO-style updates with group-relative comparisons [30, 31, 43]. We adapt DAPO for our multi-role policy as RCF-DAPO, comparing rewards only within matched roles and decision contexts.

Video editing. Video editing systems range from user-assisted tools to autonomous agents. LAVE keeps users in the editing loop [34]. In more automated settings, CutClaw and CineAgents handle music-synchronized editing and cinematic compilation [47, 48]. The original Crayotter uses a traceable multi-agent workflow, while VideoAgent assembles specialists dynamically [39, 49]. Beyond workflow design, task-specific training has also been explored. “The Script is All You Need” trains a ScripterAgent but uses a separate DirectorAgent [21]. Aurora trains a planner for local edits [44]. Building on the Crayotter environment, subsequent work trains a 9B policy with final-video preferences on manually constructed tasks [38]. In contrast, WeVisCut learns one role-routed policy for fixed-narration production from executable trajectories built through editing inversion, verified replay, observe–reflect–repair, and trajectory materialization.

Agent and video evaluation. Evaluation of narration-driven video production needs to consider both the rendered video and the production process. VBench, VideoScore, and VMBench assess generated-video quality and motion [6, 9, 15], while Goku-Bench tests instruction-based edits [14]. Script–video timing has also been evaluated [21]. Complementary agent evaluation ranges from recorded web and mobile trajectories [3, 27] to interactive shopping, browser, and desktop environments that validate action outcomes [4, 11, 36, 40, 50]. Software-repair benchmarks use repository tests [10], while multi-turn and tool-oriented suites measure progress, valid transitions, and task constraints [17, 18, 19, 42]. We combine these perspectives in a benchmark of 100 script-to-video tasks, linking scripts, shot plans, tool trajectories, and rendered videos. Five metrics are introduced to assess script fidelity, visual–narration alignment, timeline plan realization, long-horizon consistency, and end-to-end delivery success.

WeVisCut Agent Framework

Figure Figure 2 presents the WeVisCut agent framework for long-form script-to-video production. A shared policy makes editing decisions, while a stateful runtime executes tools and maintains the resulting production artifacts. Their interaction connects the narration script to a rendered video through an inspectable process that supports revision when execution reveals problems. We first define the task and policy–runtime interface, then describe the harness that implements this process.

Script-to-Video Task

A task input xx consists of a fixed narration script SS, target aspect ratio a∗a^{*}, feasible duration interval D∗D^{*}, optional source assets, and a versioned shared tool space TT. The goal is to render a video VV satisfying the delivery constraints without rewriting the narration. The outputs also include a typed shot plan PP, specifying narration coverage, visual content, and timing, and an execution trajectory τ\tau recording calls and returns throughout the production process (Appendix § 10).

Production uses tools that accept typed arguments and return artifacts or execution feedback. At call tt, a single policy πθ\pi_{\theta} with shared parameters θ\theta serves a role rt∈{O,G,C}r_t \in\{\mathrm{O}, \mathrm{G}, \mathrm{C}\}. The orchestrator (O) selects tools and arguments, the generator (G) supplies structured content, and the critic (C) inspects artifacts and requests or verifies repairs based on recorded execution feedback.

At each call, the role context ctc_t combines the task input, role, a view of the production state sts_t (shot plan, artifacts, and progress), relevant call-and-return history hth_t, and tool schemas. The policy produces a complete role output ztz_t, which the runtime E\mathcal{E} processes under a tool contract utu_t:

zt∼πθ(rt)(⋅∣ct),(ot,st+1)∼Ert,ut(⋅∣st,zt).(1)z_t \sim\pi_{\theta}^{(r_t)}(\cdot\mid c_t), \qquad(o_t, s_{t+1}) \sim\mathcal{E}_{r_t,u_t}(\cdot\mid s_t, z_t). \tag*{(1)}

The contract specifies the tool, required fields, and value types. The runtime returns observation oto_t, reporting results or blockers, and the updated production state st+1s_{t+1}. For orchestrator calls, the role output ztz_t specifies a tool action ata_t containing the selected tool and arguments (Figure 2).

Harness Design

The harness in Figure 2 implements the interaction in Equation (1) through a shared tool space, persistent production state, and runtime checks. Each orchestrator turn selects one tool, which may process multiple shots and invoke generator or critic calls. Tool selection depends on the current production state, including available artifacts and blockers, rather than a predetermined sequence.

Production flow and tool system. Starting from the narration script SS, segmentation produces production segments, and shot planning expands them into the shot plan PP. Shot records connect narration spans to visual queries, target durations, and spotlight assignments. Visual retrieval and text-to-speech (TTS) tools use these records to obtain shot assets and narration audio. For spotlight shots, decoration tools select verified HyperFrames templates [7] and fill typed slots, allowing the policy to control presentation through structured content rather than raw layout code. Music search supplies background music (BGM) for the final mix. Timeline composition combines shot assets, narration audio, and decorations, while final rendering produces the shot videos required for stitching. Quality judging and repair tools allow the policy to revisit these intermediate products before delivery. This progression describes artifact dependencies, not an obligatory ordering of every tool call.

Persistent state and selective context. These tools communicate through the production state sts_t. Global records track narration coverage, shot order, and timing, while per-shot records associate assets, audio, templates, renders, and unresolved issues with shot IDs. Planned content remains distinct from artifacts that have actually been produced. Episode memory retains these artifacts, and the event trace preserves calls, returns, and failures. The initial observation exposes the full production state. Subsequent role contexts ctc_t contain the latest runtime observation, state changes, a progress summary, and relevant local records. Separating persistent storage from model context avoids repeatedly serializing the entire project while preserving the information needed for later inspection and repair.

Validated execution and actionable feedback. Given a tool action ata_t, the runtime identifies the tool contract utu_t and checks argument schemas, artifact references, and execution prerequisites. When structured content or inspection is required, it supplies this contract in the generator’s or critic’s role context. Only the runtime commits validated results to the production state, whereas rejected calls return typed blockers that identify what must be corrected or produced next. For example, a stitching request cannot substitute for missing shot renders or background music. The runtime reports these missing prerequisites, allowing the orchestrator to select the necessary production tools before retrying. This separation keeps model decisions grounded in execution evidence rather than treating a proposed plan as completed work.

Artifact-grounded inspection and bounded repair. Execution feedback also determines when revision is needed. Quality inspection combines structured checks of shot records, timing, and render status with a VLM-Judge over spotlight renders. The critic identifies affected shots and the defects that a repair should address. Repair tools then revise the relevant content and recheck affected downstream artifacts while retaining still-valid results. For example, correcting a decoration need not regenerate unchanged narration audio. Repair limits bound repeated correction attempts, but exhausting this budget does not remove the artifact prerequisites for delivery. Quality warnings and missing production artifacts therefore remain distinct conditions. The representative trajectory in Appendix § 11 illustrates how inspection, blocked calls, and targeted follow-up actions interact within this loop.

Verified delivery and trace retention. When the required artifacts are available, the policy requests final stitching. Reaching the call limit also triggers a delivery attempt from the available production state. Before stitching, the runtime checks the required shot renders, narration audio, and background music. After stitching, it validates the video and delivery manifest. Missing prerequisites are returned as blockers while further calls are available. A run that terminates without a validated final video is recorded as incomplete, with its execution trace retained for diagnosis. The resulting production records support executable data construction in Section 4 and policy training in Section 5. This production harness is separate from the read-only evaluation harness in Appendix § 9.2.

Data Construction

The harness records execution, but finished videos expose only narration and shot structure, not the tool calls, feedback, and repairs that produced them. To train the shared policy, we recover editing intent and replay it under the same state and action contract, constructing role-specific examples grounded in real tool returns from executing the recovered intent.

Our source data consists of finished narration-driven videos collected from Bilibili, spanning technology and digital products, business and finance, natural science, and social sciences and humanities. The admitted set contains 3,750 video projects.

Executable Trajectory Construction

To convert these videos into executable supervision, we use a four-stage pipeline comprising editing inversion, verified replay, observe–reflect–repair, and trajectory materialization (Figure 3).

Editing Inversion. We first recover script–shot intent from the source videos (Figure 3A), including timestamped narration, shot structure, and complete sentence–shot alignment for every admitted project. Narration provides temporal anchors, while aligned shots indicate how the content is organized visually. Together, this evidence forms a structured storyboard describing what each shot should communicate, how long it should last, and which production constraints it should satisfy. This process recovers the intended video structure, not the original editor’s tool actions.

Verified Replay. The recovered storyboard specifies what to produce, but does not provide execution feedback. Verified replay (Figure 3B) therefore executes this intent through a production toolchain aligned with the deployment state and action contract. Schema-checked tools maintain a shared production state across planning (article segmentation, shot planning, and self-reflection), asset production (visual retrieval with TTS, spotlight decoration, music search, and severity ranking), and timeline execution (composition with judging, final rendering, and stitching). Each real tool return updates this state before the next decision. The runtime verifies asset references, audio durations, template inputs, timeline positions, and rendered outputs against the artifacts actually produced. Spotlight shots use verified HYPERFRAMES templates [7] that separate layout, motion, and visual style. The policy selects a template and fills typed slots instead of generating raw HTML, CSS, or JavaScript.

Observe–Reflect–Repair. During replay, quality inspection combines structured shot, timing, decoration-slot, and render-status checks with a VLM-Judge over spotlight renders. Using these inspection results, we locate production problems, apply bounded local repairs supported by the toolchain, and verify these changes while preserving successful artifacts (Figure 3C).

Trajectory Materialization. These executions produce trajectories that retain pre-action states, tool names and arguments, real returns, created artifacts, quality verdicts, and failure–repair links. Missing arguments, unresolved references, invalid stage order, and failed delivery checks remain visible rather than being rewritten as successes. We then organize these trajectories into role-specific examples and rollout records (Figure 3D).

Training Examples and Data Split

These examples provide role-specific supervision under the framework in Figure 2. Orchestrator samples supervise tool selection and structured arguments, while generator samples cover script segments, shot plans, retrieval queries, and template content. Critic samples provide failure localization, repair requests, and verification decisions. Runtime observations remain context, with loss applied only to model-authored tokens.

We partition by source project, keeping all its examples and repair records together. Held-out projects supply the benchmark in § 6.1 and are excluded from training. The 134,771 examples supervise a single Qwen3.5-9B policy with shared production state and tool vocabulary across all three roles.

Training WeVisCut

The executable examples provide initial supervision, but WeVisCut also needs to handle the states produced by its own decisions. Figure 4 summarizes three training phases, comprising unified SFT and runtime-verified RFT, recovery learning with Guided-OPD, and decision optimization with RCF-DAPO. The policy interaction and agent harness are defined in Section 3.

Phase 1: Unified SFT and RFT

We first train Qwen3.5-9B with LoRA [8] on the 134,771 examples constructed above. Following AgentTuning and Agent-FLAN [2, 46], we supervise model responses while retaining tool feedback as input.

Let DSFT\mathcal{D}_{\mathrm{SFT}} contain NSFTN_{\mathrm{SFT}} training examples. We index examples by ee and tokens within an example by jj. The conversation ye=(ye,1,…,ye,Le)y_e = (y_{e,1}, \ldots, y_{e,L_e}) has LeL_e tokens, and ye,<jy_{e,<j} is the prefix before position jj. Before each response, this prefix encodes the role context ctc_t, including instructions and available tool feedback. The loss mask me,jm_{e,j} is one for tokens in the target model response and zero for instructions, tool-return messages, and padding tokens added to equalize sequence lengths for batching. Thus only response tokens contribute to the supervised fine-tuning loss

LSFT=−∑e=1NSFT∑j=1Leme,jlog⁡πθ(ye,j∣ye,<j)∑e=1NSFT∑j=1Leme,j.(2)\mathcal{L}_{\mathrm{SFT}} = -\frac{\sum_{e=1}^{N_{\mathrm{SFT}}}\sum_{j=1}^{L_e}m_{e,j}\log\pi_\theta(y_{e,j}\mid y_{e,<j})}{\sum_{e=1}^{N_{\mathrm{SFT}}}\sum_{j=1}^{L_e}m_{e,j}}. \tag*{(2)}

The denominator counts the supervised tokens, so the loss averages their negative log probabilities across all supervised model responses in the corpus.

SFT teaches the recorded production steps, but the student can encounter new failures when executing tasks on its own. We collect additional supervision through runtime-verified rejection-sampled fine-tuning (RFT). At the first recoverable failure, we restore the last replayable state before the error and keep the valid earlier steps. The teacher regenerates the complete remaining sequence from that state. Both student-only successes and teacher-assisted repairs are replayed and admitted only if they pass the runtime checks and a quality threshold selected on development tasks. We track and balance these two sources of supervision. Failures that cannot be restored enter a bounded pool for later rollout sampling. Appendix § 8.1 gives the admission rules for both sources of supervision.

Phase 2: Recovery Learning with Guided-OPD

We next apply Guided-OPD [12] to learn decisions while interacting with the production environment. At training update nn, the teacher generates a complete role output with probability βn\beta_n, and the student generates it with probability 1−βn1-\beta_n. At call tt, we draw qt∼Bernoulli⁡(βn)q_t \sim\operatorname{Bernoulli}(\beta_n) and use

zt∼{πϕ(rt)(⋅∣ct),qt=1,πθ(rt)(⋅∣ct),qt=0.(3)z_t \sim\begin{cases} \pi_{\phi}^{(r_t)}(\cdot\mid c_t), & q_t=1, \\ \pi_{\theta}^{(r_t)}(\cdot\mid c_t), & q_t=0. \end{cases} \tag*{(3)}

Here ϕ\phi is the frozen teacher’s parameter set. Both policies receive the same role context ctc_t. As in Equation (1), the complete output ztz_t is executed before the next call, when control is sampled again.

Both branches train the student. Teacher outputs use forward KL against the teacher distribution renormalized over its KvocabK_{\mathrm{vocab}} most likely tokens, reducing the cost of distillation. Student outputs use reverse KL on their own generated prefixes. We gradually reduce βn\beta_n from a value below one to zero, so training ends with fully student-controlled executions. Appendix § 8.1 specifies the token masks, KL losses, and intervention schedule used throughout this training phase.

Phase 3: Decision Optimization with RCF-DAPO

Finally, we compare alternative outputs for the same production decision. Our Role- and Context-Factored DAPO (RCF-DAPO) adapts DAPO [43] by restricting reward comparisons to the same role and production context. At a fixed role call, we sample candidate outputs ziz_i, where ii indexes alternatives in the comparison group. For example, generator candidates may be different retrieval queries for the same shot. For orchestrator calls, we restore the same production state, sample alternative next actions, and execute each with a frozen continuation policy, the same tool registry, budget, and paired random seeds. Generator and critic outputs are evaluated under the same local request, artifact, and tool contract.

Each candidate receives a role-specific reward Ri(r)R_i^{(r)}. Orchestrator rewards combine successful delivery and final quality with partial credit for persistent progress on failed executions. Generator rewards assess schema validity, requirement coverage, and content usability. Critic rewards assess diagnosis, repair utility, and verification, with a penalty for unnecessary repairs. The scoring rules, rubrics, and weights are fixed. Appendix § 8.2 defines the complete reward formulas.

Within a group with matching conditions κ\kappa, we convert rewards into relative advantages

Ai(r)=sg⁡[Ri(r)−μκ(r)σκ(r)+ϵ],(4)A_i^{(r)} = \operatorname{sg}\left[\frac{R_i^{(r)}-\mu_\kappa^{(r)}}{\sigma_\kappa^{(r)}+\epsilon}\right], \tag*{(4)}

where μκ(r)\mu_\kappa^{(r)} and σκ(r)\sigma_\kappa^{(r)} are the group’s mean reward and standard deviation, ϵ>0\epsilon>0 prevents division by zero, and sg⁡\operatorname{sg} stops gradients through the advantages. Above-mean rewards yield positive advantages, and below-mean rewards yield negative ones. Groups with insufficient reward variation are resampled, with quotas and a replay queue maintaining coverage. Contexts are collected from the current policy with gradients stopped, and role losses are balanced after token-level normalization. We retain DAPO’s asymmetric Clip-Higher and soft overlong shaping. Appendix § 8.2 details the sampling and updates.

Experiments

We evaluate whether task-specific training improves script-to-video production and whether these gains extend to complete editing systems. We first compare policies in a shared tool environment, then compare WeVisCut with editors using their native workflows. Separating these comparisons distinguishes policy performance from differences in tools and orchestration. We further examine the cumulative training stages and assess the rendered videos through blinded human evaluation.

Experimental Setup

Benchmark and implementation. We construct a fixed benchmark of 100 script-to-video tasks from the held-out source projects in Section 4. Each task specifies a narration script, source pool, target duration, and aspect ratio. All systems receive identical task inputs and delivery constraints. Our trainable variants share the Qwen3.5-9B backbone, training split, decoding configuration, tool registry, and rendering budget. The Guided-OPD and RCF-DAPO training configurations are given in Appendix Table 5.

Comparison methods. For the policy comparison, all models use the same instructions, tool schemas, runtime, and rendering budget. We evaluate Qwen3.5-9B, Qwen3.6-27B, and Qwen3.6-35B-A3B [24, 25, 26], together with Qwen3.5-397B-A17B, Gemini 3.1 Pro [5], and a fixed Claude Opus 5 snapshot. These reference models receive no task-specific post-training. Our variants differ only in the training stages applied to the shared 9B backbone within the same production environment.

We compare WeVisCut with Crayotter [38] and the commentary-capable VideoAgent workflow [49], each using its released tools and orchestration. Adapters may translate file formats and export evaluation artifacts, but may not retrieve assets, write narration, create plans, or repair outputs. CutClaw, CineAgents, and DirectorAgent [21, 47, 48] are excluded from this quantitative comparison because their native workflows omit or replace parts of the fixed-narration, fixed-source task contract. Appendix Table 6 shows whether each system supports the required inputs, shot plans, and execution.

Evaluation metrics. We propose five complementary metrics for script-to-video production that assess the relationship between the source script, editing plan, execution trace, and rendered video. All scores lie in [0,100][0, 100], with higher values indicating better performance. Script Content Fidelity (SCF) measures the coverage, faithfulness, and ordering of narration claims relative to the source script. Visual–Narration Semantic Match (VNSM) assesses whether rendered visuals support the concurrent narration. Timeline Plan Realization (TPR) compares planned and realized shots in presence, duration, and order. Long-Horizon Consistency (LHC) measures late-stage shot-health retention and asset diversity. End-to-End Rendering Yield (ERY) is the fraction of held-out tasks producing a decodable, sufficiently complete video with the required non-silent narration. Overall Score is the equally weighted mean of the five reported metric means, used only as a descriptive summary for comparing aggregate task performance.

Production Cost Efficiency (PCE) is reported separately as tool calls and generated tokens per successful output minute. For native systems, we instead report wall-clock minutes and estimated USD per successful output minute, since the underlying tools and services differ. Both measures include failed attempts in the cost numerator, and lower values are better. Definitions, eligibility rules, and missing-data treatment are given in Appendix § 9.

Evaluation protocol. Each system is evaluated in five independent runs over the 100 tasks. We report across-run means and standard deviations for the five component metrics, with no uncertainty estimate for Overall Score. Permissions and retry limits are fixed within the shared environment, and thresholds, adapters, and evaluator settings are selected on development tasks. Semantic evaluation uses timestamped contact sheets in a read-only, network-free environment with access restricted to metric-specific evidence. The held-out evaluator, its prompts, and the five reported metrics are not used as post-SFT training signals. The metric-validation protocol is described in Appendix § 9.1, and evidence collection and isolation within the read-only evaluation environment are detailed in Appendix § 9.2.

Policy Comparison

We first compare policies within the shared production environment to examine the effect of task-specific training without changing the tool pipeline. We report task performance and policy-generation cost for each evaluated policy variant in Table 1.

Table 1. Policy comparison on 100 held-out tasks in the shared tool environment. Component scores are mean±standard deviation across five runs over the full test set. Overall is the equal-weight mean of the five component means. Calls and generated tokens include all attempts and are normalized by successful output minutes. Arrows indicate preferred directions, and the best results are bold.

MethodSCFVNSMTPRLHCERYOverallCalls / minTokens / min
General-purpose baselines
Qwen3.5-9B69.8±1.669.8 \pm1.659.2±1.859.2 \pm1.879.5±1.779.5 \pm1.793.7±0.893.7 \pm0.870.4±2.170.4 \pm2.174.53.111,883
Qwen3.6-27B72.6±1.472.6 \pm1.459.7±1.659.7 \pm1.680.1±1.580.1 \pm1.595.1±0.795.1 \pm0.772.1±1.972.1 \pm1.975.93.96,742
Qwen3.6-35B-A3B71.9±1.971.9 \pm1.956.7±2.056.7 \pm2.083.4±1.883.4 \pm1.897.3±0.697.3 \pm0.679.4±2.379.4 \pm2.377.74.727,554
Reference models
Qwen3.5-397B-A17B75.8±1.575.8 \pm1.564.1±1.764.1 \pm1.784.6±1.684.6 \pm1.696.2±0.896.2 \pm0.886.9±1.886.9 \pm1.881.56.015,223
Gemini 3.1 Pro80.2±1.280.2 \pm1.266.5±1.366.5 \pm1.388.2±1.288.2 \pm1.298.0±0.598.0 \pm0.590.3±1.590.3 \pm1.584.64.513,814
Claude Opus 587.1±1.087.1 \pm1.074.8±1.174.8 \pm1.193.1±1.093.1 \pm1.098.6±0.498.6 \pm0.495.8±1.295.8 \pm1.289.911.678,763
WeVisCut (Ours)88.4±2.188.4 \pm2.174.6±2.374.6 \pm2.394.0±2.094.0 \pm2.098.8±0.598.8 \pm0.596.7±1.896.7 \pm1.890.53.12,377

caption:

Task-specific performance. WeVisCut improves the Overall Score from 74.5 to 90.5 over its Qwen3.5-9B initialization, a gain of 16.0 points, with improvements in all five metrics. The gains extend beyond successful delivery to script fidelity and visual grounding, as SCF and VNSM increase by 18.6 and 15.4 points, respectively. By comparison, the unadapted models show less uniform improvements over the 9B base policy. Qwen3.6-35B-A3B gains 9.0 points on ERY but loses 2.5 points on VNSM, illustrating that delivery success does not necessarily imply better visual–narration alignment. WeVisCut also obtains higher means on all five metrics than Qwen3.5-397B-A17B and Gemini 3.1 Pro.

Comparison with Claude Opus 5. WeVisCut achieves an Overall Score of 90.5, compared with 89.9 for Claude Opus 5. Its mean scores are higher on four metrics and slightly lower on VNSM, with no component differing by more than 1.3 points. Yet it uses 2,377 generated tokens per successful output minute, compared with 78,763 for Claude Opus 5, or approximately one thirty-third as many. Relative to its base policy, WeVisCut also reduces token usage from 11,883 while maintaining 3.1 tool calls per successful output minute. These results show that the task-adapted 9B WeVisCut policy achieves mean task scores close to those of Claude Opus 5 at substantially lower generation cost. We next examine quality and cost in complete editing workflows with each system’s native tools.

End-to-End System Comparison

We next evaluate whether the gains persist when each editor uses its own tools and orchestration. In Table 2, we compare WeVisCut with Crayotter and VideoAgent using identical task inputs.

Table 2. End-to-end comparison using each system’s native workflow on the same 100 tasks. Component scores are mean±standard deviation across five runs, and Overall is the equal-weight mean of the component means. Native costs include all attempted runs and report wall-clock minutes and estimated USD per successful output minute. The best results are bold.

SystemSCFVNSMTPRLHCERYOverallTime (min) / out minCost (USD) / out min
Crayotter81.6±1.381.6 \pm1.367.8±1.467.8 \pm1.489.1±1.389.1 \pm1.394.2±0.994.2 \pm0.991.6±1.491.6 \pm1.484.98.60.97
VideoAgent78.4±1.578.4 \pm1.565.2±1.665.2 \pm1.686.7±1.586.7 \pm1.593.5±0.693.5 \pm0.688.7±1.788.7 \pm1.782.510.40.58
WeVisCut (Ours)88.4±2.188.4 \pm2.174.6±2.374.6 \pm2.394.0±2.094.0 \pm2.098.8±0.598.8 \pm0.596.7±1.896.7 \pm1.890.52.10.09

caption:

WeVisCut achieves the highest mean score on all five metrics, with an Overall score of 90.5 compared with 84.9 for Crayotter and 82.5 for VideoAgent. The improvements cover both semantic fidelity and execution quality. This pattern indicates consistent gains across content selection, temporal planning, execution stability, and final delivery. For example, relative to Crayotter, SCF and VNSM each improve by 6.8 points, alongside higher plan realization, long-horizon consistency, and rendering yield. The advantage observed in the shared environment therefore also holds for these complete editing workflows.

These gains are accompanied by lower end-to-end costs. For each successful output minute, WeVisCut requires 2.1 wall-clock minutes, compared with 8.6 for Crayotter and 10.4 for VideoAgent. The corresponding estimated monetary costs are $0.09, $0.97, and $0.58, respectively. Both measures include unsuccessful attempts. Note that these costs reflect the complete native workflows, rather than the efficiency of an isolated model or tool.

Human Evaluation

Training Stage Analysis

To examine how the training phases in Section 5 contribute to the final policy, we evaluate successive checkpoints in Table 3. The backbone, data split, tools, decoding configuration, and rendering budget remain fixed. Each checkpoint adds the next training stage to the preceding one, allowing us to track the cumulative gains across the three training phases.

PhaseVariantSCFVNSMTPRLHCERYOverall
–Base69.8±1.669.8\pm1.659.2±1.859.2\pm1.879.5±1.779.5\pm1.793.7±0.893.7\pm0.870.4±2.170.4\pm2.174.5
1SFT73.6±2.073.6\pm2.060.5±2.260.5\pm2.281.2±2.181.2\pm2.194.6±1.094.6\pm1.087.6±2.487.6\pm2.479.5
1SFT→\toRFT79.8±1.879.8\pm1.865.1±2.065.1\pm2.086.9±1.986.9\pm1.995.8±0.995.8\pm0.989.8±2.289.8\pm2.283.5
2SFT→\toRFT →\to Guided-OPD81.0±1.681.0\pm1.667.4±1.867.4\pm1.887.8±1.787.8\pm1.796.6±0.896.6\pm0.891.5±1.991.5\pm1.984.9
3WeVisCut (Ours)88.4±2.188.4\pm2.174.6±2.374.6\pm2.394.0±2.094.0\pm2.098.8±0.598.8\pm0.596.7±1.896.7\pm1.890.5

Table 3. Cumulative training-stage analysis with a fixed Qwen3.5-9B backbone and evaluation environment. Phase 1 includes SFT and RFT, Phase 2 adds Guided-OPD, and Phase 3 adds RCF-DAPO. Component scores are mean±standard deviation across five runs on the 100 held-out tasks. Overall is the equal-weight mean of the component means. The best results are bold.

In Phase 1, SFT raises ERY from 70.4 to 87.6, providing the largest stage-wise improvement in delivery reliability. This is consistent with its role in establishing valid tool use from executable demonstrations. RFT then adds supervision from successful student trajectories and teacher-repaired failures. Relative to SFT alone, it improves SCF by 6.2 points and TPR by 5.7 points, indicating gains in script fidelity and plan realization beyond the initial improvement in delivery.

Building on this initialization, Phase 2 applies Guided-OPD to mixed teacher–student rollouts. All five metrics improve at this checkpoint, including VNSM from 65.1 to 67.4 and ERY from 89.8 to 91.5. This pattern suggests that Guided-OPD complements trajectory-level RFT by refining decisions at intermediate states, where the student must continue from its own partially correct actions rather than replaying teacher-generated contexts. These results show further gains after distillation on states reached during policy execution, although the aggregate metrics do not isolate failure recovery.

Finally, Phase 3 optimizes production decisions with RCF-DAPO. SCF and VNSM improve by another 7.4 and 7.2 points, respectively, while Overall increases from 84.9 to 90.5. Figure 5 shows the cumulative improvements in script fidelity, visual–narration alignment, and long-horizon consistency against the reference policies. These comparisons measure the benefit of adding training stages in this order, rather than the independent effects of turn guidance, snapshot matching, or individual role rewards.

Cumulative training progression for SCF, VNSM, and LHC

Figure 5. Cumulative training progression for (a) SCF, (b) VNSM, and (c) LHC. OPD and RL denote Guided-OPD and RCF-DAPO, respectively. Error bars show standard deviations across five evaluation runs on the 100 held-out tasks. Horizontal lines show Qwen3.5-9B, Qwen3.5-397B-A17B, and Claude Opus 5 under the shared tool environment. Results at these checkpoints on all five metrics are shown in Table 3.

Human Evaluation

To test whether the automatic gains carry over to human judgments, three qualified annotators independently rate anonymized videos from the three systems in Table 2 on the same 100 held-out tasks. Videos are shown individually in random order, with scripts, source pools, and target durations fixed. The shared tasks allow paired comparisons. Each video is rated from 1 to 5 on the six dimensions in Table 4.

MethodTheme fidelityContent richnessNarrative coherenceEditing smoothnessVisual qualityOverall preference
Crayotter [38]3.583.423.353.483.553.48
VideoAgent [49]3.463.603.303.423.523.46
WeVisCut (Ours)4.454.324.584.524.384.56

Table 4. Blinded human evaluation on 100 held-out tasks with three raters. Entries are mean 1–5 ratings (higher is better). Overall preference is a human rating, distinct from the Overall Score.

As shown in Table 4, WeVisCut obtains the highest mean rating on all six dimensions, with an overall preference of 4.56 versus 3.48 for Crayotter and 3.46 for VideoAgent. The largest gains occur in narrative coherence (1.23 and 1.28 points), while higher editing smoothness and visual quality indicate that the preference is not confined to narrative structure, consistent with the automatic comparisons. Ratings are averaged across annotators within each task. These descriptive means do not establish statistical significance or validate individual automatic metrics (see Appendix § 9.1).

Conclusion

We presented WeVisCut, a system for long-form script-to-video production built around a single 9B policy shared by the orchestrator, generator, and critic. Our four-stage data pipeline combines editing inversion and verified replay to recover executable supervision from finished videos. Training progresses from executable imitation to recovery learning and reward-based decision optimization. Our 100-task benchmark connects source scripts, editing plans, execution traces, and final videos through five complementary metrics. The trained policy improves all five metrics over its base model and achieves mean scores close to those of Claude Opus 5 while using approximately one thirty-third as many generated tokens per successful output minute. Native-system comparisons and blinded human ratings also favor WeVisCut. Broader template coverage and more robust retrieval are needed to extend the system to a wider range of editing tasks.

Optimization and Training Details

We use the notation of Sections 3 and 5 throughout: tt indexes role calls, jj indexes tokens, and nn indexes training updates. The task input is xx, and πθ(r)\pi_{\theta}^{(r)} is the shared policy conditioned on role rr.

Executable RFT and Guided-OPD

Replay-safe RFT. Let t⋆t^{\star} index the call at which the first runtime failure occurs. A failure is recoverable when the environment can restore the latest transaction snapshot before t⋆t^{\star}. External side effects are executed in a sandbox or through idempotent calls. We retain the valid prefix, provide the teacher with the failure record and restored state, and regenerate the complete suffix. The repaired trajectory is retained only after replay from the snapshot passes schema, prerequisite, artifact-reference, render, duration, narration, lint, and black-frame checks. Its quality score QRFTQ_{\mathrm{RFT}} also needs to meet the development-selected minimum qRFTq_{\mathrm{RFT}}: QRFT≥qRFTQ_{\mathrm{RFT}} \ge q_{\mathrm{RFT}}. Complete student successes pass the same admission checks. Student-only and teacher-assisted records have separate provenance tags and fixed source–task quotas. Unrecoverable cases are kept in a class-balanced, capped hard-case pool for later rollout sampling, with fixed mixing probability η\eta.

Turn-level intervention and loss routing. Following Guided-OPD [12], call tt consists of a complete role output ztz_t followed by its runtime observation oto_t. At update nn, qt∼Bernoulli⁡(βn)q_t \sim\operatorname{Bernoulli}(\beta_n) selects the teacher when qt=1q_t = 1 and the student otherwise. The selected output is executed before control is sampled again. Both branches contribute distillation loss. We denote the mixed rollout distribution by μn=πθ,ϕβn\mu_n = \pi_{\theta,\phi}^{\beta_n}, where θ\theta and ϕ\phi are the student and frozen teacher parameters.

The models share a tokenizer. For teacher outputs, the teacher distribution is renormalized over its KvocabK_{\mathrm{vocab}} most likely tokens, giving π~ϕKvocab\widetilde{\pi}_{\phi}^{K_{\mathrm{vocab}}}. Student outputs use reverse KL, while teacher outputs use this truncated forward KL:

LOPD=Eτ∼μn[1Nτ∑t,jmt,j((1−qt)DKL(πθ∥πϕ)+qtDKL(π~ϕKvocab∥πθ))]ct,j.(5)\mathcal{L}_{\mathrm{OPD}} = \mathbb{E}_{\tau\sim\mu_n} \left[ \frac{1}{N_{\tau}} \sum_{t,j} m_{t,j} \left( (1-q_t)D_{\mathrm{KL}}(\pi_{\theta} \Vert\pi_{\phi}) + q_t D_{\mathrm{KL}}\left(\widetilde{\pi}_{\phi}^{K_{\mathrm{vocab}}} \Vert\pi_{\theta}\right) \right) \right]_{c_{t,j}} . \tag*{(5)}

Here ct,jc_{t,j} contains the role context ctc_t and all tokens preceding position jj in call tt. The policies in each KL term are conditioned on this token prefix. The mask mt,jm_{t,j} is one for assistant tokens and zero for input, padding, and runtime observations, and Nτ=∑t,jmt,j>0N_{\tau} = \sum_{t,j} m_{t,j} > 0 counts supervised tokens. Thus teacher-only trajectories also train the student. We log retained top-KvocabK_{\mathrm{vocab}} probability mass to audit truncation. This is an efficiency variant of Guided-OPD. Step-Level OPD instead keeps trajectories student-controlled and queries teacher targets afterward [33].

Teacher intervention follows

βn=βstart1+cos⁡(πmin⁡(n/Tdecay,1))2,βstart=0.8,(6)\beta_n = \beta_{\mathrm{start}} \frac{1 + \cos\left(\pi\min\left(n/T_{\mathrm{decay}},1\right)\right)}{2}, \qquad\beta_{\mathrm{start}} = 0.8, \tag*{(6)}

where TdecayT_{\mathrm{decay}} is the number of updates over which intervention decays, set to 70% of training. The remaining 30% uses student-only rollouts. Only checkpoints evaluated without teacher intervention proceed to RL. Guided-OPD uses a 50-call horizon with early termination, while RL uses the 16-call deployment budget. Token limits are per-call context/response caps, and compact state updates limit history growth.

Role- and Context-Factored DAPO

Optimization scope. Each trainable call records its role rr, context cc, output zz, observation oo, and execution contract uu. Main rollouts populate a snapshot buffer. For an orchestrator context ctc_t, we restore the pre-decision state, sample alternative outputs, and execute each with the same frozen continuation policy πθold\pi_{\theta_{\mathrm{old}}}, budget, and seed set. Here θold\theta_{\mathrm{old}} is the policy parameter snapshot frozen for the current optimization iteration. Nested generator and critic tokens are masked from the orchestrator loss. Generator and critic contexts are also collected from current-policy rollouts and resampled locally. Let drd_r be the context distribution for role rr and λr\lambda_r its outer loss weight. The optimized proxy is

Jproxy=∑r∈{O,G,C}λrEc∼sg⁡(dr), z∼πθ(r)(⋅∣c)[R(r)(c,z)].(7)J_{\mathrm{proxy}}=\sum_{r\in\{\mathrm{O},\mathrm{G},\mathrm{C}\}}\lambda_r\mathbb{E}_{c\sim\operatorname{sg}(d_r),\,z\sim\pi_{\theta}^{(r)}(\cdot\mid c)}\left[R^{(r)}(c,z)\right]. \tag*{(7)}

Here R(r)(c,z)R^{(r)}(c,z) is the expected execution reward for output zz in context cc, with runtime observations averaged out. This is a conditional call-level objective. Frozen continuations and the sampled context distribution remain part of the proxy, so it does not provide an unbiased gradient of the terminal production return.

Matched comparison groups. For orchestrator call tt, we fix κO,t=(x,ct,st,T,Bt,πcont)\kappa_{\mathrm{O},t}=(x,c_t,s_t,T,B_t,\pi_{\mathrm{cont}}): the task input, context, restored state, tool space, remaining budget BtB_t, and continuation policy πcont=πθold\pi_{\mathrm{cont}}=\pi_{\theta_{\mathrm{old}}}. Only the next output varies. Generator groups fix the local request, artifact, tool, schema/version, budget, and initial state. Critic groups also fix the defect and verification contract. Candidate executions share paired seeds. Let Gκ\mathcal{G}_{\kappa} be the group of outputs sampled under matching conditions κ\kappa.

Features are mapped to their registered ranges: [0,1][0,1], except repair utility in [−1,1][-1,1]. Fixed weights produce one reward per output before group normalization. Contexts are sampled through role–task–tool quotas. A group is accepted only if Var⁡i∈Gκ(Ri(r))>δ\operatorname{Var}_{i\in\mathcal{G}_{\kappa}}(R_i^{(r)})>\delta, where δ\delta is the minimum reward-variance threshold. Otherwise it is resampled up to a fixed cap. Low-variance contexts remain in a coverage queue and are revisited through executable imitation. For an accepted group,

Ai(r)=sg⁡[Ri(r)−μκ(r)σκ(r)+ϵ],μκ(r)=1∣Gκ∣∑i∈GκRi(r).(8)A_i^{(r)}=\operatorname{sg}\left[\frac{R_i^{(r)}-\mu_{\kappa}^{(r)}}{\sigma_{\kappa}^{(r)}+\epsilon}\right], \qquad \mu_{\kappa}^{(r)}=\frac{1}{\lvert\mathcal{G}_{\kappa}\rvert}\sum_{i\in\mathcal{G}_{\kappa}}R_i^{(r)}. \tag*{(8)}

Here σκ(r)\sigma_{\kappa}^{(r)} is the group reward standard deviation, ϵ>0\epsilon>0 stabilizes normalization, and sg⁡\operatorname{sg} stops gradients through group statistics. No normalization mixes roles or unmatched contexts. Resampling emphasizes informative reward differences and changes the sampled context distribution.

Role-specific rewards. For candidate ii, let ziz_i be its role output and oio_i its execution feedback. An orchestrator output is scored by continuing from snapshot tt. We measure progress through validated milestones:

Φ(s)=∑kωkIk(s)∑kωk,rℓprog=γΦ(sℓ+1)−Φ(sℓ),\Phi(s)=\frac{\sum_k\omega_k I_k(s)}{\sum_k\omega_k}, \qquad r_{\ell}^{\mathrm{prog}}=\gamma\Phi(s_{\ell+1})-\Phi(s_\ell),
Fi,t→=(1−ϵp)clip⁡(∑ℓ≥tγℓ−trℓprog,0,1).(9)F_{i,t}^{\rightarrow}=(1-\epsilon_p)\operatorname{clip}\left(\sum_{\ell\ge t}\gamma^{\ell-t}r_{\ell}^{\mathrm{prog}},0,1\right). \tag*{(9)}

Here Ik(s)∈{0,1}I_k(s)\in\{0,1\} indicates whether unique milestone kk holds in state ss, ωk>0\omega_k>0 is its fixed weight, 0<γ≤10<\gamma\le1 is the discount factor, and ℓ\ell indexes subsequent calls until branch ii terminates. Repetition adds no milestone credit, and later invalidation removes it. The positive margin 0<ϵp<10<\epsilon_p<1 keeps failed-branch progress below the delivery tier. The operator clip⁡(⋅,0,1)\operatorname{clip}(\cdot,0,1) bounds its argument to [0,1][0,1].

The rewards are

RiO=gdel,i[α+(1−α)Qend,i]+(1−gdel,i)αFi,t→−λppi,R_i^{\mathrm{O}}=g_{\mathrm{del},i}\left[\alpha+(1-\alpha)Q_{\mathrm{end},i}\right]+(1-g_{\mathrm{del},i})\alpha F_{i,t}^{\rightarrow}-\lambda_p p_i,
RiG=∑kwkGfkG(c,zi,oi),R_i^{\mathrm{G}}=\sum_k w_k^{\mathrm{G}}f_k^{\mathrm{G}}(c,z_i,o_i),
RiC=wdfdiag,i+wrUrepair(di)+wvfverify,i−wufunnecessary,i.(10)R_i^{\mathrm{C}}=w_d f_{\mathrm{diag},i}+w_r U_{\mathrm{repair}}(d_i)+w_v f_{\mathrm{verify},i}-w_u f_{\mathrm{unnecessary},i}. \tag*{(10)}

The delivery indicator gdel,ig_{\mathrm{del},i} is one only if termination, schema, prerequisite, artifact, and loop checks pass and final quality Qend,i≥qmin⁡Q_{\mathrm{end},i}\ge q_{\min}, where qmin⁡q_{\min} is the minimum delivery quality. Otherwise it is zero. A legal stop below this threshold is premature, recorded by the indicator pi∈{0,1}p_i\in\{0,1\}. The parameter α∈[0,1]\alpha\in[0,1] weights delivery and progress credit relative to final quality, and λp≥0\lambda_p\ge0 weights the premature-stop penalty.

Final quality combines a rule score and a sampled render audit: Qend=(1−wa)Qrule+waQauditQ_{\mathrm{end}}=(1-w_a)Q_{\mathrm{rule}}+w_aQ_{\mathrm{audit}}, where Qrule,Qaudit∈[0,1]Q_{\mathrm{rule}},Q_{\mathrm{audit}}\in[0,1] and wa∈[0,1]w_a\in[0,1] is the fixed audit weight. The six rule features are plan coverage, spotlight-decoration coverage, inspection coverage, plan hygiene, completion, and efficiency, weighted 0.353, 0.186, 0.179, 0.144, 0.081, and 0.057. The render audit scores visual impact and clarity. Thus these six weights apply to rule features, with the audit blended separately.

Generator features fkGf^{\mathrm{G}}_k measure schema validity, grounding, requirement coverage, and downstream usability using deterministic checks and a frozen tool-specific rubric. All features are set to zero for an invalid schema. Critic features fdiag,if_{\mathrm{diag},i} and fverify,if_{\mathrm{verify},i} score diagnosis and verification, and funnecessary,if_{\mathrm{unnecessary},i} penalizes unnecessary repair. The coefficients wkG,wd,wr,wv,wuw_k^{\mathrm{G}}, w_d, w_r, w_v, w_u are fixed feature weights. Here did_i is the diagnosis provided by critic output ziz_i. For a diagnosis dd, controlled repair utility is

Urepair(d)=1∣Ξ∣∑ξ∈Ξ[Q(Afix(d;ξ))−Q(Afix(d0;ξ))],(11)U_{\mathrm{repair}}(d) = \frac{1}{|\Xi|} \sum_{\xi\in\Xi} \left[ Q\left(A_{\mathrm{fix}}(d;\xi)\right) - Q\left(A_{\mathrm{fix}}(d_0;\xi)\right) \right], \tag*{(11)}

where AfixA_{\mathrm{fix}} is a frozen repair executor with identical tool version and budget, d0d_0 is an empty diagnosis, Ξ\Xi is a paired seed set, and QQ maps repair results to a fixed quality score in [0,1][0,1]. Thus utility measures the improvement over the empty-diagnosis baseline under controlled execution. The training judge, rubric, prompt, and version hash are frozen and independent of the held-out evaluator. Rule and judge scores are logged separately.

DAPO update. Let BrB_r be the candidate batch for role rr, and Mi(r)\mathcal{M}^{(r)}_i the model-authored token positions in output ziz_i. For token jj, the policy probability ratio is ρi,j=πθ(zi,j∣ci,zi,<j)/πθold(zi,j∣ci,zi,<j)\rho_{i,j} = \pi_\theta(z_{i,j} \mid c_i, z_{i,<j})/\pi_{\theta_{\mathrm{old}}}(z_{i,j} \mid c_i, z_{i,<j}), where cic_i is the candidate’s input context. We use DAPO’s asymmetric Clip-Higher interval [43], with ϵlow=0.20\epsilon_{\mathrm{low}} = 0.20 and ϵhigh=0.28\epsilon_{\mathrm{high}} = 0.28:

ℓi,j(r)=min⁡(ρi,jAi(r), clip⁡(ρi,j,0.8,1.28)Ai(r)),\ell^{(r)}_{i,j} = \min\left(\rho_{i,j} A^{(r)}_i,\ \operatorname{clip}(\rho_{i,j},0.8,1.28) A^{(r)}_i\right),
JRCF-DAPO=∑rλrE[∑i∈Br∑j∈Mi(r)ℓi,j(r)∑i∈Br∣Mi(r)∣].(12)\mathcal{J}_{\mathrm{RCF\text{-}DAPO}} = \sum_r \lambda_r \mathbb{E}\left[\frac{\sum_{i \in B_r}\sum_{j \in\mathcal{M}^{(r)}_i} \ell^{(r)}_{i,j}}{\sum_{i \in B_r} |\mathcal{M}^{(r)}_i|}\right]. \tag*{(12)}

Token losses are aggregated within each role before applying the outer weights λO=λG=λC=1/3\lambda_{\mathrm{O}} = \lambda_{\mathrm{G}} = \lambda_{\mathrm{C}} = 1/3. The old policy is synchronized once per optimization iteration. We use neither dual clipping nor a loss-side KL term.

For generated token count LL, let Lsoft<LcapL_{\mathrm{soft}} < L_{\mathrm{cap}} denote the no-penalty threshold and generation hard cap. For non-truncated responses, soft overlong shaping is

plen(L)={0,L≤Lsoft,−λlenL−LsoftLcap−Lsoft,Lsoft<L<Lcap.(13)p_{\mathrm{len}}(L) = \begin{cases} 0, & L \le L_{\mathrm{soft}}, \\ -\lambda_{\mathrm{len}} \dfrac{L-L_{\mathrm{soft}}}{L_{\mathrm{cap}}-L_{\mathrm{soft}}}, & L_{\mathrm{soft}} < L < L_{\mathrm{cap}}. \end{cases} \tag*{(13)}

We set Lsoft=Lcap−2,048L_{\mathrm{soft}} = L_{\mathrm{cap}} - 2{,}048 and the length-penalty weight λlen=0.1\lambda_{\mathrm{len}} = 0.1. Responses reaching the hard cap are replaced before an eight-candidate group is formed. If the resampling cap is reached first, the group is dropped. Truncated responses enter neither rewards, group statistics, nor policy loss. Role-wise gradient norms and pairwise cosine similarities are logged, since outer loss weights control scale but do not eliminate shared-parameter gradient conflict.

The registered post-training configuration is shown in Table 5.

StageKey configuration
Guided-OPDQwen3.5-9B student with a Qwen3.5-397B-A17B-FP8 teacher and Kvocab=50K_{\mathrm{vocab}} = 50. Per-call prompt/response caps: 12,288/4,096 tokens. Batch 16, 90 steps, βstart=0.8\beta_{\mathrm{start}} = 0.8 decaying over 70 steps, and learning rate 10−610^{-6}. At most 50 assistant turns with early termination.
RCF-DAPOPer-call prompt/response caps: 24,576/12,288 tokens. Train/generation/mini-batch: 48/72/24, matched call group size 8, and separate O/G/C batches. Stratified dynamic sampling with coverage replay. Clip [0.8, 1.28], temperature 0.9, 16-turn deployment horizon, learning rate 10−610^{-6}, LoRA rank/alpha 32/64, and no KL penalty.

Table 5. Registered post-training configuration.

Metrics and Evidence Protocol

The five component metrics lie in [0,100][0,100] and are reported individually, alongside the descriptive Overall Score defined below. Component claims, shot IDs, matching rules, thresholds, and denominators are frozen before test outputs are inspected.

Script Content Fidelity (SCF). Let c1,…,cmc_1,\ldots,c_m be the claims in source script SS, and s1,…,sks_1,\ldots,s_k the claims in output narration VV. The entailment indicator E(a,B)∈{0,1}E(a,B) \in\{0,1\} tests whether text BB supports claim aa, and τK\tau_K is the Kendall order correlation of matched source and output claims. We compute

Cov⁡=m−1∑i=1mE(ci,V),Faith⁡=k−1∑j=1kE(sj,S),\operatorname{Cov}=m^{-1}\sum_{i=1}^{m}E(c_i,V), \qquad\operatorname{Faith}=k^{-1}\sum_{j=1}^{k}E(s_j,S),
Ord⁡=(1+τK)/2,SCF⁡=100(0.40Cov⁡+0.35Faith⁡+0.25Ord⁡).(14)\operatorname{Ord}=(1+\tau_K)/2, \qquad\operatorname{SCF}=100(0.40\operatorname{Cov}+0.35\operatorname{Faith}+0.25\operatorname{Ord}). \tag*{(14)}

SCF is valid for at least five source claims and one narration unit.

Visual–Narration Semantic Match (VNSM). Let SS contain the registered shots with a valid evaluator response. For each shot ss, four uniformly sampled frames form montage FsF_s, paired with concurrent narration TsT_s. The held-out Claude Opus 5 evaluator assigns an anchored grade gs=g(Fs,Ts)∈{1,2,3,4}g_s=g(F_s,T_s) \in\{1,2,3,4\} (Poor, Fair, Good, Excellent) using the rubric below. We rescale grades to [0,100][0,100] and aggregate

VNSM⁡=∣S∣−1∑s∈S100(gs−1)3,∣S∣≥5.(15)\operatorname{VNSM}=|S|^{-1}\sum_{s\in S}\frac{100(g_s-1)}{3}, \qquad|S|\geq5. \tag*{(15)}

Timeline Plan Realization (TPR). Planned and realized shots are joined by frozen shot ID:

Rpres=∣P∩A∣∣P∣,R_{\mathrm{pres}}=\frac{|P\cap A|}{|P|},
Rdur=∣P∩A∣−1∑s∈P∩Aexp⁡[−∣Dsplan−Dsact∣max⁡(0.5,0.3Dsplan)],R_{\mathrm{dur}}=|P\cap A|^{-1}\sum_{s\in P\cap A}\exp\left[-\frac{|D_s^{\mathrm{plan}}-D_s^{\mathrm{act}}|}{\max(0.5,0.3D_s^{\mathrm{plan}})}\right],
Rord=LCS⁡(P,A)max⁡(∣P∣,∣A∣),TPR⁡=100(0.30Rpres+0.40Rdur+0.30Rord).(16)R_{\mathrm{ord}}=\frac{\operatorname{LCS}(P,A)}{\max(|P|,|A|)}, \qquad \operatorname{TPR}=100(0.30R_{\mathrm{pres}}+0.40R_{\mathrm{dur}}+0.30R_{\mathrm{ord}}). \tag*{(16)}

We set Rdur=0R_{\mathrm{dur}}=0 when P∩A=∅P\cap A=\varnothing.

Long-Horizon Consistency (LHC). Shot health averages render success, audio–clip duration agreement, and the rescaled VNSM grade:

qs=13[1(s rendered)+exp⁡(−∣Dsaudio−Dsclip∣max⁡(0.5,0.3Dsaudio))+gs−13].(17)q_s=\frac{1}{3}\left[\mathbf{1}(s\text{ rendered})+\exp\left(-\frac{|D_s^{\mathrm{audio}}-D_s^{\mathrm{clip}}|}{\max(0.5,0.3D_s^{\mathrm{audio}})}\right)+\frac{g_s-1}{3}\right]. \tag*{(17)}

Unavailable gsg_s is omitted and the observable terms are renormalized. For the first and last thirds HH, TT and canonical asset digests AA,

Rret=clip⁡(qˉT/qˉH,0,1),Rnov=∣unique⁡(A)∣∣A∣,R_{\mathrm{ret}}=\operatorname{clip}(\bar q_T/\bar q_H,0,1), \qquad R_{\mathrm{nov}}=\frac{|\operatorname{unique}(A)|}{|A|},
LHC⁡=100(0.60Rret+0.40Rnov).(18)\operatorname{LHC}=100(0.60R_{\mathrm{ret}}+0.40R_{\mathrm{nov}}). \tag*{(18)}

LHC requires N≥12N\geq12 and qˉH≥0.5\bar q_H\geq0.5.

End-to-End Rendering Yield (ERY).

ERY⁡=100Ntotal∑c1[ok⁡(c)],ok⁡=exists⁡∧decodable⁡∧[Dact≥0.5Dplan]∧hasNarration⁡.(19)\operatorname{ERY}=\frac{100}{N_{\mathrm{total}}}\sum_c\mathbf{1}[\operatorname{ok}(c)], \qquad\operatorname{ok}=\operatorname{exists}\land\operatorname{decodable}\land[D^{\mathrm{act}}\geq0.5D^{\mathrm{plan}}]\land\operatorname{hasNarration}. \tag*{(19)}

hasNarration requires non-silent speech covering at least half of the planned narration duration. Background music does not qualify. Timeouts, crashes, and unstarted tasks remain in the denominator.

Overall Score (descriptive summary). For each model or system within a comparison track, let M‾k\overline{M}_k denote its reported mean for component kk. We add a post-hoc descriptive summary of these means:

Overall⁡=15∑k∈{SCF,VNSM,TPR,LHC,ERY}M‾k(20)\operatorname{Overall} = \frac{1}{5}\sum_{k\in\{\mathrm{SCF},\mathrm{VNSM},\mathrm{TPR},\mathrm{LHC},\mathrm{ERY}\}} \overline{M}_k \tag*{(20)}

Each component receives weight 0.2. These weights are not fitted to model rankings or human preferences. The shared [0, 100] scale makes this a transparent macro-average, but does not make the components independent or equally calibrated: LHC already incorporates VNSM in shot health. Overall therefore summarizes task performance rather than serving as a separately validated measure of perceptual video quality. It is distinct from the human-rated overall preference in Table 4.

The displayed scores are computed from the reported, rounded component means and rounded to one decimal place. Each component retains its own eligibility rules and denominator. Overall does not imply a common per-task population. If any component mean is unavailable, Overall is N/A rather than an average over the remaining metrics. Costs are excluded. We report no Overall standard deviation, confidence interval, or significance test: these require aligned run-level results or the cross-metric covariance, which cannot be recovered from the component standard deviations alone. All five components remain visible so that the summary does not conceal trade-offs.

Production Cost Efficiency (PCE). For tool calls or generated tokens XX,

PCEX=∑c∈CallX(c)∑c∈CokDcact/60(21)\mathrm{PCE}_X = \frac{\sum_{c\in\mathcal{C}_{\mathrm{all}}} X(c)}{\sum_{c\in\mathcal{C}_{\mathrm{ok}}} D^{\mathrm{act}}_c/60} \tag*{(21)}

PCE is lower-is-better and is never combined with quality. Eligibility counts and validity conditions are recorded alongside metric reporting. Conditional means use valid tasks, whereas unconditional means assign zero to system failures. Harness or provider outages remain missing rather than being converted into low-quality scores.

Diagnostic Validation

We apply three strengths of metric-targeted corruption: claim edits (SCF), time-local visual substitution (VNSM), shot removal/duration/order changes (TPR), tail degradation or asset repetition (LHC), and delivery failure (ERY). For each metric–corruption pair we report the paired drop, bootstrap 95% interval, severity monotonicity, and the 5 × 5 sensitivity matrix. Human pairwise validation is metric-specific and blinded. VNSM additionally reports cross-family judge agreement and human correlation. Agreement below Krippendorff’s α=0.6\alpha= 0.6 prevents a standalone VNSM claim.

Evidence-Seeking Harness

At most three 4×44 \times4 contact sheets are generated per video with sampling step

Δ=max⁡(1,Δuser,∣N/(16Smax⁡)∣),Smax⁡=3.(22)\Delta= \max(1,\Delta_{\mathrm{user}},\lvert N/(16S_{\max})\rvert), \qquad S_{\max}=3. \tag*{(22)}

Tiles carry source-frame and timestamp labels, while a sidecar stores FPS, duration, sampling step, and sheet count. Representative inputs are shown in Appendix § 12. Read-only mounts isolate the evidence available to each metric. SCF sees script and narration, while VNSM sees aligned narration and rendered frames/clips. TPR sees plan and realized timeline. LHC sees ordered health inputs and asset digests. ERY/PCE see delivery probes and accounting. The network-free harness may call only bounded FFmpeg (editing and probing), PySceneDetect (scene detection), Whisper (ASR), PaddleOCR (OCR), frozen detectors, and manifest/diff readers. Tools expose evidence but never scores, and every call is archived.

Held-Out VNSM Evaluator Prompt

The evaluator is called independently for each registered shot. The complete version tuple—prompt, rubric, model snapshot, decoding parameters, sampler, and JSON schema—is frozen after development-only calibration and before any test output is rendered. Figure 6 visualizes the exact evidence path. The four image tiles are frames sampled by the evaluation harness rather than illustrative replacements.

Auditable VNSM evaluator path

Figure 6. Auditable VNSM evaluator path. A shot packet binds the narration window, timestamps, video and manifest digests, sampler/prompt versions, judge snapshot, and decoding configuration. The frozen sampler exposes four timestamped frames and, only when motion is necessary, one bounded clip. The held-out Claude Opus 5 evaluator must first record visible evidence, then choose one anchored grade and emit schema-validated JSON. Aggregation excludes unavailable calls rather than converting infrastructure failure into a zero-quality judgment.

Held-Out VNSM Evaluator: System Contract

Role. You are an evidence-seeking evaluator for one registered metric. Evaluate only visual–narration semantic correspondence; do not grade overall video quality, aesthetics, editing smoothness, source fidelity, tool use, or hidden production intent.

Immutable inputs. A shot packet (JSON) with criterion=VNSM, case and shot IDs, the narration text active over the registered shot interval, video and manifest SHA-256 digests, sampler and prompt versions, and model snapshot and decoding configuration; and sampled evidence comprising four chronologically ordered frames t1t_1–t4t_4 with source timestamps (hh:mm:ss) and, on request, one bounded clip. No plan, retrieval query, similarity score, reference video, method name, or training reward is supplied.

Procedure.

  1. Verify that all four frames, their timestamps, and the narration window are present and internally consistent. Otherwise set status to one registered failure code.

  2. Decompose the narration into its central visually testable units: referent, setting, attribute, action, and event. Do not require support for abstract discourse that cannot be depicted.

  3. Inspect every frame in chronological order and record two to four time-local visible observations before assigning a score. Each observation must be paired with one supplied frame timestamp.

  4. For each central unit, decide whether it is directly supported, merely generic/topical, absent, or contradicted. One matching object cannot override a contradictory action or event.

  5. Request the single bounded clip only when motion or temporal order is necessary to distinguish an action. The clip may cover only the registered shot interval and cannot introduce outside context.

  6. Select exactly one anchored score whose full anchor is satisfied. When evidence falls between anchors, choose the lower anchor; never interpolate or invent a new score.

Evidence rules. A generic presenter, publisher card, or decorative overlay alone cannot score above 2 (Fair). OCR text counts as visual support only when the narration is explicitly about displayed text. A correct setting with the wrong central action is at most 2 (Fair); a direct contradiction scores 1 (Poor). Evidence outside the registered narration interval is inadmissible. Never invent a timestamp or infer content absent from the supplied frames or bounded clip. Missing input, decoding failure, or harness/provider failure sets status to a failure code; it is missing evidence, not a Poor score.

Output. Return strict JSON only, with no markdown or free-form text.

Held-Out VNSM Evaluator: Anchors and Output

Anchored rubric.

  • 4 – Excellent (direct and complete): every central visually testable referent/action/event is directly visible in the matching interval, with no material contradiction.

  • 3 – Good (specific but partial): the correct subject and scene are visible and the central referent or action is substantially supported, but one important attribute, phase, or consequence is not verifiable.

  • 2 – Fair (topical or generic): the imagery is in the correct broad topic or setting, but the central action, relation, or specific referent is not visually verified. Generic B-roll belongs here even when aesthetically apt.

  • 1 – Poor (atmospheric, absent, or contradictory): only color, mood, or a weak association relates to the narration, or the imagery is irrelevant, shows the wrong subject/action/event, or materially contradicts the narration.

Required JSON fields.

{
  "criterion": "VNSM",
  "score": 1|2|3|4,
  "evidence": ["<visible observation>", ...],
  "timestamps": ["<hh:mm:ss>", ...],
  "status": "ok"
}

For status="ok", score is an allowed anchor, and evidence contains 2–4 distinct visible observations, each aligned with one entry of timestamps drawn from the supplied frames. Otherwise, score is null, evidence and timestamps are empty, and status is exactly one of missing_input, digest_-mismatch, sampler_failure, provider_failure, decode_failure, or schema_failure. Case and shot IDs, digests, and version IDs are bound by the shot packet rather than repeated in the output.

Deterministic validation and retry. The evaluator runs at temperature 0 with the frozen decoding parameters shown in the packet. A deterministic validator checks the criterion and score values, the status-dependent fields, timestamp membership, evidence count, and the match between each response and its shot packet. An invalid response receives one schema-only repair request containing the validation errors but no new visual evidence. A second invalid response is recorded as unavailable with schema_failure. No human edits the score or evidence.

Aggregation and claim gate. The held-out Claude Opus 5 evaluator judges each registered shot once. Video-level VNSM is the unweighted mean of rescaled grades over shots with status="ok" and is reported only when ∣S∣≥5|S| \ge5. Unavailable calls are excluded rather than scored. A second held-out VLM family independently re-judges the same shots solely to audit agreement, and its grades do not enter VNSM. We report coverage, eligible-shot count, the two family-specific means, rank correlation, and Krippendorff’s α\alpha. Agreement below α=0.6\alpha= 0.6 blocks a standalone VNSM claim. Before the prompt is frozen, a disjoint 50-shot development set is used for blinded human correlation and controlled corruption tests (wrong subject, wrong action, temporal shift, and generic B-roll). Test tasks never participate in calibration.

Training/evaluation separation. The training-side VLM-Judge is a sampled spotlight audit, used only for reward features and limited to visual_impact and clarity: the former checks whether the animation is visually engaging in layout and aesthetics, while the latter checks whether motion and text convey information without distraction or layout errors. The held-out VNSM evaluator and audit family instead use final rendered evidence, separately versioned models, isolated credentials, and prompts unavailable to the training jobs.

Experimental Protocol and Reporting

External Baseline Contract

Task-matched native systems receive the same frozen script, source pool, duration, aspect ratio, and budget while retaining their released orchestration and tools. Adapters may translate serialization and export native artifacts, but may not retrieve assets, rewrite narration, invent plans, add manual edits, or perform unavailable repairs. We record the plan, call log, artifacts, final video, failure code, time, and cost. A missing native artifact makes its trace-dependent metric N/A. Systems requiring semantic adapter work enter the task-adjacent track and are excluded from the headline matched ranking. The five component-metric entries average five independent runs over the same 100 held-out tasks, with ±\pm denoting the standard deviation across runs. Overall Score instead averages the five reported component means and is shown without an uncertainty estimate. It is N/A when any component mean is unavailable. Missing components are neither imputed nor omitted from the denominator.

SystemParadigmFixed narrationShot planTrace / repairFixed sourcesMatched native track
CrayotterArtifact-centric multi-agent✓✓✓✓✓
VideoAgentLarge agent library✓✓✓✓✓
CutClawSpecialized multi-agent✗✓✓✓✗
CineAgentsDesign-and-compose agents✗✓✓✓✗
DirectorAgentScript-to-video generation✗✓✗✗✗

Table 6. Baseline task-contract audit. Systems satisfying every required contract element enter the task-matched native-system comparison.

Plan–Render Visualization Protocol

Qualitative examples are selected from registered length, domain, and failure strata before method identities are revealed. The native-system panel fixes rows to Crayotter and WeVisCut and uses one common narration and source pool. Each row shows five rendered keyframes at identical absolute timestamps and a compact evidence strip restricted to visible properties. Missing frames remain visibly missing and an invalid delivery retains its terminal failure stage. Additional systems, when available under the same contract, appear in separate panels rather than changing the frozen comparison. Visualizations explain the registered metrics but do not re-score or reorder examples.

Depth and Real-Tool Diagnostics

For a subset of tasks, the evaluation protocol injects a registered late-stage fault and records the repair branch τ+\tau^{+}. The shared case ID links the script, shot plan, original execution, delivered video, and changes made during repair, allowing recovery behavior to be inspected alongside normal production.

For horizon HiH_i and first failure turn tifailt_i^{\mathrm{fail}},

FailRate⁡@k=∑i1[Hi≥k]1[tifail=k]∑i1[Hi≥k].(23)\operatorname{FailRate}@k = \frac{\sum_i \mathbf{1}[H_i \ge k]\mathbf{1}[t_i^{\mathrm{fail}} = k]}{\sum_i \mathbf{1}[H_i \ge k]}. \tag*{(23)}

Completion and repair at depth kk use the same eligible denominator. The proxy–real gap is a paired mean between frozen-fixture replay and production execution. Unmatched or failed runs are retained in reporting. These depth and proxy–real diagnostics complete the registered protocol. The experiment tables report the five registered metrics and the descriptive Overall Score.

A Representative Autonomous Tool-Use Flow

This appendix illustrates the inference-time behavior of WeVisCut with one complete production trace. The purpose is not to expose the full generated screenplay or every JSON payload, but to show the closed-loop decision pattern used by the policy: it first builds a long-horizon production plan, then executes real tools, observes typed runtime returns, repairs bounded failures, satisfies missing prerequisites, and terminates only after the video has actually been delivered.

The raw trace is stored as agent_trace.jsonl and the delivery record as stitch_manifest.json. The transcript below is a faithful, translated rendering of that trace, organized as

Thinking → Observation → Tool Call → Runtime Return → Next Decision.

The Thinking field reports the deliberation the policy emits before committing to a tool action. It is part of the recorded action interface rather than hidden reasoning: every claim it makes about the world is either grounded in a previous typed return or is falsified by the next one, which is precisely what makes the trace usable as training supervision.

Representative Autonomous Trace: Plan–Act–Repair–Deliver

Run card. The run identifier is aicut_20260824_152401_9f7f9d. The input is a fixed long-form narration about aircraft science and engineering, tokenized by the runtime into 51 semantic units (u01–u51). The finished project contains 6 screenplay segments, 30 realized shots, 16 spotlight shots, and 3 background-music tracks. The policy runs for 16 tool turns over 31 minutes of wall-clock time and delivers a 395.0-second video:

final_video_with_bgm_20260824_155554.mp4

Build the high-level production plan

Thinking. I hold nothing but the narration. Every downstream tool in this runtime is gated on structure that does not exist yet, so the only admissible first action is the one that turns free text into a typed screenplay. Two choices matter here. First, granularity: the narration runs several hundred seconds, and a segment count near six keeps each act long enough to carry an argument while still giving the shot planner meaningful boundaries to respect. Second, I should request the global music style now rather than at delivery time, because the stitcher will need a music query long after the narration’s tone is still in context, and recovering that tone later from rendered shots would be lossy. I also pin narration order and full-script coverage, since any dropped span becomes a semantic hole that no later repair tool is able to detect.

Observation. Only the raw narration is available. No shot plan, assets, audio, templates, rendered paths, or delivery artifacts exist.

Call.

Tool_SegmentArticle(
    num_segments_target=6,
    segment_granularity="medium",
    preserve_narration_order=True,
    cover_full_script=True,
    need_music_style=True
)

Return. status=success. The runtime returns a six-segment outline — a prologue (“Looking Up in Wonder”), four core acts (“A Precision Airborne Ecosystem”, “Extreme Lightweighting and Flexible Wisdom”, “Eyes of Technology and Invisible Guardians”, “From Adventure to Everyday”), and a coda (“A Love Letter to the Sky”) — together with a per-segment narration script, a planner strategy, and the global music style: “epic orchestral, piano lead, strings underneath, tempo moving from slow to fast, grand and warm”.

Next decision. Convert the outline into executable shot-level production cards.

Expand the plan into shot-level production cards

Thinking. The segment outline is an argument, not a shot list. Nothing downstream can consume it: retrieval needs a visual query per shot, synthesis needs a vocal script per shot, and the renderer needs durations and identifiers. I set the shot target well above what I expect to keep, because the planner is allowed to merge and drop but not to invent coverage, and an over-request costs nothing at this stage while an under-request silently truncates the tail of the script. I forbid repeated visual queries, since duplicate queries collapse to the same retrieved asset and produce visible repetition that no per-shot quality check will flag — the failure only becomes obvious in the stitched cut, which is far too late to fix cheaply. I also want spotlight marks assigned here rather than later, because the decoration and rendering budget is the scarcest resource in the run and I would rather allocate it from a global view of the script than greedily, shot by shot.

Observation. A segment-level plan exists, but the runtime still lacks shot IDs, visual queries, durations, spotlight marks, and coverage records.

Call.

Tool_PlanShots(
    mode="plan_shots_all",
    num_shots_target=50,
    planning_policy={
        "preserve_narration_order": True,
        "avoid_repeated_visual_query": True
    }
)

Return. status=success. The planner realizes 30 shots spanning 397.4 planned seconds, distributed 2/2/2/7/10/7 across the six segments, and marks 16 of them as spotlight shots. The union of covered_unit_ids is u01–u51, i.e. semantic coverage is complete with no uncovered narration unit.

Next decision. Materialize the planned shots with real assets and narration audio.

Execute batched visual retrieval and narration synthesis

Thinking. Retrieval and text-to-speech are the two slowest operations in the pipeline and they are independent per shot, so issuing them one shot at a time would waste most of the run’s wall-clock budget on serialization. I dispatch all 30 shots in a single batch. This is also the first turn where the plan meets the real asset library, which means it is the first turn where my assumptions can be falsified: a visual query that reads well as prose may still retrieve nothing usable. I therefore care less about the average match score in the return than about whether any shot comes back with an empty candidate set, because a partial batch would force me to choose between re-querying and accepting a hole in the cut.

Observation. The shot plan is complete, but no media artifacts exist.

Call.

Tool_BatchVisualAudio(
    shot_ids=["1", "2", ..., "30"]
)

Return. status=success for all 30 shots. Every shot receives a clip path, a source duration, a match score, and a non-empty candidate set — for example shot 2 retrieves at match_score = 0.592 from 12 candidates and shot 5 at 0.611 from 12 candidates, while the weakest, shot 1, still returns 0.452 from 5 candidates. All shots report playlist_ready=true. The batch takes roughly 14 minutes, the single most expensive turn in the run.

Next decision. Spend the decoration and rendering-template budget only on spotlight shots.

Decorate the spotlight shots

Thinking. Decoration is what separates a slideshow from an edited video, but it is also the most expensive thing I can ask for, both in template rendering and in the downstream quality checks it triggers. The 16 spotlight marks from step 2 are exactly the allocation decision I already made, so I honor them and set spotlight_only rather than re-deciding now with less context than the planner had. I ask for slot verification in the same call, because a template whose slots are filled but semantically wrong is worse than no template at all: it renders cleanly, passes structural checks, and puts an incorrect caption on screen.

Observation. Shot assets and narration audio exist for all 30 shots, and the 16 spotlight IDs are known from the plan.

Call.

Tool_BatchDecorateSpotlights(
    shot_ids=["1","2","5","6","7","10","11","13",
              "14","16","19","21","22","23","26","28"],
    template_policy={
        "select_template": True,
        "fill_slots": True,
        "verify_slots": True,
        "spotlight_only": True
    }
)

Return. status=success, decorated_count=16, failed_count=0, spotlight_count=16. Templates are selected per rhetorical role: hook_title_intro for the opening shot, definition_card for the Bernoulli explanation, key_takeaway for the materials act, analogy_card for the winglet–rowing-blade comparison, and historical_moment for the aviation-history shot. Critically, verified_count=0: slot verification was requested but produced no verdicts, and every verify_result is an empty object.

Next decision. The decoration is structurally complete but semantically unverified. Inspect and improve the plan before paying for the final render.

Encounter a quality-warning state

Thinking. The zero verification count from the previous return is the loose thread in this run. I have 16 decorated shots that nothing has actually judged, and the render stage will happily bake whatever text is in those slots into the frames. Rendering first and inspecting afterwards would be the expensive ordering, so I inspect now, over all 30 shots rather than only the decorated ones, since a visual mismatch on a plain shot is just as damaging in the final cut. I scope the repair to the two dimensions that decoration can actually move — decoration quality and visual match — and request minimal repair with an immediate re-judge, so that whatever the tool changes is validated in the same call instead of drifting unchecked into the render.

Observation. The decorated plan carries no verification verdicts.

Call.

Tool_Stage4DimensionalRepair(
    failed_shot_ids=["1", "2", ..., "30"],
target_dimensions=["decoration_quality", "visual_match"],
minimal_repair=True,
rejudge_after_repair=True,
repair_round=0
)

RETURN. status=completed_with_warnings after 189.9 seconds: passed_count=0, failed_count=30. Every shot remains below the strict quality threshold.

NEXT DECISION. A blanket failure over all 30 shots indicates missing evidence rather than providing a per-shot defect report. Produce explicit self-reflection findings before attempting another patch.

## Hit a typed precondition error

THINKING. I want to patch the plan against the reflection findings, and reward shaping is the augmentation I would like applied so that both passing and failing shots receive graded per-dimension scores rather than a binary verdict. What I am about to do, however, is issue synthetic issue identifiers of my own invention. This is worth recording as an error rather than hiding, because it is the exact mistake the runtime’s typed preconditions exist to catch: I am attempting to repair against evidence I have not yet collected.

OBSERVATION. No `self_reflect_issues` object exists in the state.

CALL.

```python
Tool_SelfReflect_Patch(
    augmentation_strategy={
        "id": "aug09_reward_shaping",
        "name": "Reward Shaping",
        "focus": "Emit finer-grained reward signal: score every dimension for
                  both passing and failing shots, and state the preference
                  basis used for SFT/RL."
    },
    issue_ids=["sr_001", "sr_002", "sr_003", "sr_004", "sr_005"],
    target_shot_ids=["1", "2", "3", "4", "5"],
    minimal_repair=True,
    requires_downstream_propagation_hint=True
)

RETURN. status=blocked_precondition, missing=["self_reflect_issues"]. The runtime rejects the call with “insufficient preconditions for Tool_SelfReflect_Patch”.

NEXT DECISION. Treat the rejection as an instruction rather than a failure: run reflection first and patch against the identifiers it actually returns.

Generate the missing self-reflection evidence

THINKING. The precondition told me exactly which object is missing, so this turn is fully determined—the only real choice left is what to inspect for. I widen the focus beyond decoration to the five dimensions where an error would be unrecoverable after rendering: script coverage, shot rhythm, visual-query alignment, spotlight ratio, and duplication. I deliberately switch the augmentation strategy from reward shaping to strict semantic coverage. Reward shaping produces a smoother signal, but at this moment I do not need a smoother signal. I need a judge that would rather fail a shot than let a semantic omission through, because a missed narration unit cannot be repaired once the timeline is composed.

OBSERVATION. The patch tool is blocked, and no reflection object exists.

CALL.

Tool_SelfReflect(
    scope="full_repair_with_downstream_propagation",
    num_segments=6,
    num_shots=30,
    focus_dimensions=["script_coverage", "shot_rhythm",
                      "visual_query_alignment", "spotlight_ratio",
                      "duplication"],
    mode="gemini_augmentation",
    augmentation_strategy={
        "id": "aug01_semantic_coverage_strict",
        "name": "Semantic Coverage Strict",
        "focus": "Re-audit whether the narration covers the core information
                  of the source; prefer failing a shot over letting a semantic
                  omission pass."
    }
)

RETURN. status=needs_fix, passed=false, overall_score = 0.70. The judge emits 13 concrete issues: 11 decoration_quality findings (shots 5, 6, 7, 10, 11, 14, 16, 21, 23, 26, 28) and 2 visual_match findings (shots 14 and 28), all at medium severity, with suggested_fix="redecorate". The summary reads “13 decoration-quality problems found”, and the state now records repair_history=[{round: 0, passed: false, issue_count: 13}].

NEXT DECISION. Patch exactly these 13 issues. The blanket 30-shot failure of step 5 has been resolved into a bounded, addressable defect list.

Patch the plan against the returned issue identifiers

THINKING. Now I can issue the call I attempted two turns ago, but with the identifiers the judge actually produced instead of ones I made up. I keep the strict coverage strategy so that the patch is evaluated by the same standard that produced the findings. Switching strategies mid-repair would mean patching against one rubric and being graded by another. I set minimal repair, because the failures are concentrated in decoration slots and visual queries, and any edit that ripples into narration text would invalidate the audio I already synthesized in step 3 at a cost of 14 minutes. I request downstream propagation hints so the runtime marks which artifacts the patch has invalidated rather than leaving me to infer it.

OBSERVATION. Thirteen typed issues exist over 13 distinct shots.

CALL.

Tool_SelfReflect_Patch(
    issue_ids=["qj_5_decoration_quality", "qj_6_decoration_quality",
               "qj_7_decoration_quality", "qj_10_decoration_quality",
               "qj_11_decoration_quality", "qj_14_visual_match",
               "qj_14_decoration_quality", "qj_16_decoration_quality",
               "qj_21_decoration_quality", "qj_23_decoration_quality",
               "qj_26_decoration_quality", "qj_28_visual_match",
               "qj_28_decoration_quality"],
    target_shot_ids=["1","2","5","6","7","10","11",
                     "14","16","21","23","26","28"],
    minimal_repair=True,
    requires_downstream_propagation_hint=True,
    augmentation_strategy={"id": "aug01_semantic_coverage_strict", ...}
)

RETURN. status=patched, repair_applied=true. The patch is indeed localized: visual queries are sharpened (shot 3’s query gains the explicit “conveying precision and complexity” qualifier), and shot 2’s duration is re-estimated from 61.11 s to 60.70 s. Narration scripts, segment boundaries, and spotlight assignments are left untouched, so the synthesized audio remains valid.

Next decision. Re-run decoration so the repaired shots pick up new templates and slots.

Re-decorate under the repaired plan

Thinking. The patch changed the plan, not the rendered overlays, so the decorations currently in state were selected against text that no longer exists. I re-run decoration over the full spotlight set rather than only the 13 patched shots: template selection is a global assignment, and if a patched shot now claims a template that an unpatched neighbour is already using, restricting the call to the patched subset would leave that collision in place. The cost is the same either way, since decoration is cheap relative to rendering.

Observation. The plan has been patched but the spotlight decorations still reflect the pre-patch text.

Call.

Tool_BatchDecorateSpotlights(
    shot_ids=["1","2","5","6","7","10","11","13",
              "14","16","19","21","22","23","26","28"],
    template_policy={
        "select_template": True,
        "fill_slots": True,
        "verify_slots": True,
        "spotlight_only": True
    }
)

Return. status=success, decorated_count=16, failed_count=0. The re-decoration is visibly better targeted: shot 2’s definition now states both surfaces (“upper surface faster and lower pressure, lower surface slower and higher pressure”) instead of only the upper one. Shot 5’s takeaways are compressed from full clauses into three scannable phrases. Shot 6 moves from a generic overlay to a concept_explanation type, and shot 10 switches to the more specific historical_moment_card template.

Next decision. Re-judge the repaired, re-decorated state.

Re-judge all shots against the full dimension set

Thinking. This is the verification half of the repair loop, and I want it against all five delivery dimensions rather than the two I repaired, because a decoration edit can break timing alignment even when it improves decoration quality. I hold the threshold at the strict 0.75 and focus the judge on decoration slots, which is where the patch actually landed.

Observation. Thirteen shots have been patched and all 16 spotlight shots re-decorated.

Call.

Tool_Stage4DimensionJudge(
    shot_ids=["1", "2", ..., "30"],
    dimensions=["visual_match", "audio_sync", "decoration_quality",
                "timing_alignment", "render_integrity"],
    threshold=0.75,
    focus_decoration_slots=True
)

Return. status=completed_with_warnings, passed_count=0, failed_count=30, _elapsed_ sec=0.0.

Next decision. The zero elapsed time is the informative part: the judge did not re-evaluate, it replayed a cached verdict. Two of the five requested dimensions (render_integrity and timing_alignment) cannot be scored before a render exists, so this verdict is structurally unable to improve at this stage.

Probe whether the failure is real or structural

Thinking. Before I spend the rest of the repair budget, I need to distinguish a genuine quality failure from a stage-ordering artifact. I re-issue the step 5 repair verbatim. If the state has actually changed, the tool should now do work and take time. If it returns instantly with the same counts, the verdict is not tracking my edits and further repair at this stage is pure waste.

Observation. The re-judge reports the same blanket failure it reported before any repair.

Call.

Tool_Stage4DimensionalRepair(
    failed_shot_ids=["1", "2", ..., "30"],
    target_dimensions=["decoration_quality", "visual_match"],
    minimal_repair=True,
    rejudge_after_repair=True,
    repair_round=0
)

Return. status=completed_with_warnings, passed_count=0, failed_count=30, _elapsed_sec=0.0 — identical counts, identical zero cost. The hypothesis is confirmed: the warning is structural.

Next decision. Make one final, explicitly budgeted repair attempt so that the runtime, not the policy, decides when to stop.

Exhaust the repair budget deliberately

Thinking. I could loop here indefinitely, and that is precisely the failure mode I must avoid: a repair loop that never converges consumes the entire run and delivers nothing. So I escalate rather than repeat — I declare this repair_round=2, the last one the runtime permits, and widen the target to all five dimensions so that if any residual defect is genuinely fixable at plan level, this call has the broadest possible chance to fix it. Setting the round explicitly hands the stopping decision to the runtime’s budget accounting instead of to my own judgment, which is the safer place for it.

Observation. Two repair calls have produced no state change at zero cost.

Call.

Tool_Stage4DimensionalRepair(
    failed_shot_ids=["1", "2", ..., "30"],
    target_dimensions=["visual_match", "audio_sync", "decoration_quality",
                       "timing_alignment", "render_integrity"],
    minimal_repair=True,
    rejudge_after_repair=True,
    repair_round=2
)

Return. status=budget_exhausted: “the repair limit of 2 has been reached; remaining warnings will not be repaired and current quality is treated as passing (rejection 1 of 2)”.

Next decision. The quality gate is closed by budget rather than by score. Stop repairing and move to delivery.

Attempt delivery and learn the real blocker

Thinking. With the gate closed I go for the final artifact. I set final_gate_required so the runtime enforces its own delivery contract rather than trusting my belief that 30 shots are approved, and I pre-declare the audio mix — background music at half volume, ducked under the voice track — because the narration is the payload and music that competes with it would ruin an otherwise correct cut. I expect this call to succeed, and if it does not, its rejection will tell me exactly what the state is still missing.

Observation. All 30 shots are treated as approved.

Call.

Tool_StitchVideo(
    shot_count=30,
    approved_count=30,
    final_gate_required=True,
    bgm_volume=0.5,
    duck_under_voice=True
)

Return. status=blocked_precondition, missing=["timeline_render", "bgm", "rendered_path_ready:0/30"].

Next decision. This is the decisive observation of the run, and it retroactively explains steps 10–12: rendered_path_ready:0/30 means no shot has ever been rendered, which is why render_integrity could never pass and why the repair loop kept failing on a defect that plan-level edits could not reach. The correct action was never more repair — it was to satisfy the two unmet prerequisites, timeline rendering and background music, in that order.

Satisfy the first prerequisite: compose and render timelines

Thinking. This is the expensive, irreversible turn, so I want the runtime to refuse it rather than produce a subtly broken render. I require both that every shot have its retrieved visual and synthesized audio present, and that every spotlight shot carry a decoration — the second condition is what makes the re-decoration work of step 9 actually reach the frames instead of sitting unused in state. I compose all 30 in one batch so that timing offsets are computed against a single consistent timeline rather than accumulated shot by shot.

Observation. Assets, audio, and decorations exist in state, but no rendered paths do.

Call.

Tool_BatchTimelineCompose(
    shot_ids=["1", "2", ..., "30"],
    require_tts_visual_audio=True,
    require_spotlight_decoration=True
)

Return. status=success after 378.6 seconds: composed=30, render_success_count=30, failed_count=0, concat_count=16. The renderer routes by decoration need — the 16 spotlight shots go through the HyperFrames backend for template compositing, the remaining 14 through the direct ffmpeg path.

Next decision. One prerequisite remains: background music.

Satisfy the second prerequisite: retrieve background music

Thinking. The music style I requested back in step 1 is still in state, which is why this turn is a lookup rather than a re-derivation. I query with the full style description instead of a keyword, since the retrieval index is caption-based and the tonal arc — slow building to fast — is what distinguishes a usable track from a merely orchestral one. I request more duration than the video needs so the mixer has slack for crossfades and never runs out of music before the last shot ends. Setting must_finish_before_stitch makes the ordering constraint explicit rather than implicit in my turn sequence.

Observation. Thirty rendered shots exist, but the bgm slot is still empty.

Call.

Tool_MusicSearch(
    query="epic orchestral, piano lead, strings underneath, tempo from slow
        to fast, grand and warm",
    music_style="epic orchestral, piano lead, strings underneath, tempo from
                 slow to fast, grand and warm",
    target_duration_sec=713.71,
    must_finish_before_stitch=True
)

Return. status=success, selected_count=3. The runtime returns a sequenced playlist rather than three independent tracks, each with a placed offset: a 60.0 s cinematic orchestral opener at [0, 60.0], a 99.9 s lyrical orchestral piece at [60.0, 159.9], and a 326.6 s track whose caption reads “tempo moving from slow to urgent, emotion building” at [159.9, 486.5]. Total coverage of 486.5 s comfortably exceeds the cut.

Next decision. Both prerequisites are now satisfied. Retry the final gate.

Complete the autonomous final stitch

Thinking. I re-issue the exact call the runtime rejected in step 13, unchanged. That it is unchanged is the point: the call was never wrong, the state was incomplete, and the two intervening turns closed precisely the gap the rejection named. I keep final_gate_required set so that delivery is validated rather than asserted, and I will only consider this run finished if the return contains a real file path and a ready count of 30.

Observation. The state now contains 30 rendered shots with timeline paths, narration audio, spotlight decorations, and a three-track music playlist.

Call.

Tool_StitchVideo(
    shot_count=30,
    approved_count=30,
    final_gate_required=True,
    bgm_volume=0.5,
    duck_under_voice=True
)

Return. status=success after 50.1 seconds. The manifest reports ready_shot_count=30, missing_shots=[], validation_errors=[], and duration_ok=true on every shot, with 394.98 s of stitched video carrying 390.48 s of narration. concat_status=success and bgm_status=bgm_mixed across 3 tracks with ducking_fallback_used=false, yielding a final 395.0-second delivery:

final_video_with_bgm_20260824_155554.mp4

Takeaway. This trajectory exhibits the target inference pattern end to end. The policy constructs a long-horizon production plan before acting, executes real tools, and treats every runtime failure as a structured observation rather than a terminal condition. The invented issue identifiers of step 6 are corrected by running reflection first. The blanket quality failure of step 5 is resolved into 13 addressable defects by step 7. The non-converging repair loop of steps 10–12 is diagnosed only when the delivery gate at step 13 names rendered_path_ready:0/30 as the true blocker. The policy then satisfies both missing prerequisites and terminates only after the delivered artifact exists on disk. Notably, the run consumes its full repair budget without ever entering an unbounded loop, and the final recovery suffix is a three-call sequence that repeats no earlier work:

Tool_BatchTimelineCompose
    -> Tool_MusicSearch
    -> Tool_StitchVideo    (identical arguments to the rejected step 13)

Visual Inputs to the Evaluation Harness

Figure 7 and Figure 8 show two representative contact-sheet inputs provided to the evaluation harness. Each contact sheet contains temporally sampled video frames annotated with source-frame indices and timestamps, allowing the judges to inspect local visual consistency, timeline progression, long-range coherence, and rendering quality. These inputs are exposed through read-only, network-free tools and archived together with their metadata, ensuring that metric decisions are grounded in traceable visual evidence rather than evaluator-side assumptions.

Visual Input to the Evaluation Harness: Case 1

Figure 7. Visual Input to the Evaluation Harness: Case 1. A contact sheet of temporally sampled and timestamped video frames used for evidence-grounded diagnostic evaluation.

Visual Input to the Evaluation Harness: Case 2

Figure 8. Visual Input to the Evaluation Harness: Case 2.

Limitations

WeVisCut is limited by the current HyperFrames template library, so uncommon visual languages may be approximated using the closest available structure, motion, and style. Expanding the library through automatic template generation could broaden the design space. The dataset and project-disjoint split cannot cover all domains, editing conventions, runtimes, or asset distributions, while verified replay guarantees executability only within the registered runtime. The quality of long-form videos also depends on the stability of the video retriever, particularly for instruction-based retrieval. Integrating a trained video generation model could reduce this dependency and improve robustness. Finally, VNSM relies on a VLM. Although evidence isolation, frozen anchors, agreement reporting, and human calibration improve auditability, they cannot eliminate judge bias or service drift, motivating broader blinded studies and cross-family audits.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024.arxiv.org/abs/2306.13649
  2. [2]Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, 2024.arxiv.org/abs/2403.12881
  3. [3]Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023.arxiv.org/abs/2306.06070
  4. [4]Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024.arxiv.org/abs/2403.07718
  5. [5]Google DeepMind. Gemini 3.1 pro model card. Model card, 2026. URL https://deepmind.google/models/model-cards/gemini-3-1-pro. Published 2026-02-19.
  6. [6]Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videocore: Building automatic metrics to simulate fine-grained human feedback for video generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2105–2123, 2024.arxiv.org/abs/2406.15252
  7. [7]HeyGen. Hyperframes: Write html. render video. built for agents. https://github.com/heygen-com/hyperframes, 2026. Accessed: Sep. 30, 2026.
  8. [8]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. CodeLora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.arxiv.org/abs/2106.09685
  9. [9]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807–21818. IEEE, 2024.arxiv.org/abs/2311.17982
  10. [10]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024.arxiv.org/abs/2310.06770
  11. [11]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuya Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881–905, 2024.
  12. [12]Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, et al. On-policy distillation with curriculum turn-level guidance for multi-turn agents. arXiv preprint arXiv:2606.15912, 2026.arxiv.org/abs/2606.15912
  13. [13]Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 3102–3116, 2023.arxiv.org/abs/2304.08244
  14. [14]Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu, Youliang Zhang, Yuan Zhou, Xin Li, Qinglin Lu, and Zhibo Chen. Goku: A million-scale universal dataset and benchmark for instruction-based video editing. arXiv preprint arXiv:2606.30599, 2026. URL https://arxiv.org/abs/2606.30599.
  15. [15]Xinran Ling, Chen Zhu, Meiqi Wu, Hangyu Li, Xiaokun Feng, Cundian Yang, Aiming Hao, Jiashu Zhu, Jiahong Wu, and Xiangxiang Chu. Vmbench: A benchmark for perception-aligned video motion generation. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 13087–13098. IEEE, 2025.
  16. [16]Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, et al. Toolace: Winning the points of llm function calling. In International conference on learning representations, volume 2025, pages 41359–41381, 2025.arxiv.org/abs/2409.00920
  17. [17]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pages 52989–53046, 2024.arxiv.org/abs/2308.03688
  18. [18]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, 2025.
  19. [19]Chang Ma, Junlei Zhang, Zihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325–74362, 2024.arxiv.org/abs/2401.13178
  20. [20]Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgos, Corby Rosset, et al. Agentinstruct: Toward generative teaching with agentic flows. arXiv preprint arXiv:2407.03502, 2024.arxiv.org/abs/2407.03502
  21. [21]Chenyu Mu, Xin He, Qu Yang, Wanshun Chen, Jiadi Yao, Huang Liu, Zihao Yi, Bo Zhao, Xingyu Chen, Ruotian Ma, et al. The script is all you need: An agentic framework for long-horizon dialogue-to-cinematic video generation. arXiv preprint arXiv:2601.17737, 2026.arxiv.org/abs/2601.17737
  22. [22]Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544–126565, 2024.arxiv.org/abs/2305.15334
  23. [23]Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations, volume 2024, pages 9695–9717, 2024.arxiv.org/abs/2307.16789
  24. [24]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
  25. [25]Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026. URL https://qwen.ai/blog?id=qwen3.6-27b.
  26. [26]Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all, April 2026. URL https://qwen.ai/blog?id=qwen3.6-35b-a3b.
  27. [27]Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708–59728, 2023.DOI
  28. [28]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
  29. [29]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Răileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023.arxiv.org/abs/2302.04761
  30. [30]John Schulman, Filip Wolski, Prafulla Dhar̄iwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  31. [31]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  32. [32]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
  33. [33]Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, et al. Step-level on-policy distillation: Interpolating between on-policy distillation and supervised fine-tuning. arXiv preprint arXiv:2608.16333, 2026.arxiv.org/abs/2608.16333
  34. [34]Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, and Raj Sodhi. Lave: Llm-powered agent assistance and language augmentation for video editing. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 699–714, 2024.arxiv.org/abs/2402.10294
  35. [35]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.arxiv.org/abs/2305.16291
  36. [36]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024.arxiv.org/abs/2404.07972
  37. [37]Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In International Conference on Learning Representations, volume 2025, pages 79822–79843, 2025.arxiv.org/abs/2412.09605
  38. [38]Lecheng Yan, Jianze Lin, Yichong Zhang, Ben Pan, Wenxi Li, Chenyang Lyu, Liting Zhou, and Cathal Gurrin. Crayotter: Learning long-horizon video editing agents via group-relative preference backpropagation. arXiv preprint arXiv:2608.02694, 2026.arxiv.org/abs/2608.02694
  39. [39]Lecheng Yan, Yichong Zhang, Xiantao Xu, Jianze Lin, Ben Pan, Xiaoyu Zheng, Jiawei Qian, Anqi Wu, Jiahui Geng, Ruizhe Li, et al. Crayotter: Traceable multi-agent workflows for long-form video editing. arXiv preprint arXiv:2606.07636, 2026.arxiv.org/abs/2606.07636
  40. [40]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022.arxiv.org/abs/2207.01206
  41. [41]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
  42. [42]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.arxiv.org/abs/2406.12045
  43. [43]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026.arxiv.org/abs/2503.14476
  44. [44]Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, and Jiebo Luo. Aurora: Unified video editing with a tool-using agent. arXiv preprint arXiv:2605.18748, 2026.arxiv.org/abs/2605.18748
  45. [45]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023.arxiv.org/abs/2308.01825
  46. [46]Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077, 2024.arxiv.org/abs/2310.12823
  47. [47]Peixuan Zhang, Chang Zhou, Ziyuan Zhang, Hualuo Liu, Chunjie Zhang, Jingqi Liu, Xiaohui Zhou, Xi Chen, Shuchen Weng, Si Li, et al. A benchmark and multi-agent system for instruction-driven cinematic video compilation. arXiv preprint arXiv:2604.10456, 2026.arxiv.org/abs/2604.10456
  48. [48]Shifang Zhao, Yihan Hu, Ying Shan, Yunchao Wei, and Xiaodong Cun. Cutclaw: Agentic hours-long video editing via music synchronization. arXiv preprint arXiv:2603.29664, 2026.arxiv.org/abs/2603.29664
  49. [49]Hengji Zhou, Lingxuan Huang, Jian Wang, Bing Zhou, Si Wu, Lianghao Xia, and Chao Huang. Videoagent: All-in-one framework for video understanding and editing. arXiv preprint arXiv:2606.23327, 2026.arxiv.org/abs/2606.23327
  50. [50]Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abhishek Sridhar, Xianyi Cheng, Tianyue Qu, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606, 2024.arxiv.org/abs/2307.13854
  51. [51]Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, and Jingjing Chen. Turnopd: Making on-policy distillation turn-aware for efficient long-horizon agent training. arXiv preprint arXiv:2607.05804, 2026.arxiv.org/abs/2607.05804

Paper details

vlnc:W_8race8sq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 13

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents