betaFeedback

WorksComputer Science

Recognizing Events Is Not Enough: Why Streaming VLMs Struggle in Compositional Instruction Triggering

Abstract

Recognizing individual events does not guarantee correct execution of compositional instructions in streaming video. For example, given the request "Notify me when the pot boils after someone turns on the stove and leaves the kitchen", existing models may recognize the individual events yet notify too early or miss the intended moment, failing to track how their temporal conditions are satisfied over time. To evaluate this gap, we introduce CoTriggerBench, which composes annotated video events into temporally constrained instructions and verifies notification timing and silence through executable event plans and source-event replay. To address this challenge, we propose a real-time VLM agent, TEMPI. TEMPI decomposes instructions into atomic event queries and temporal constraints, and uses Scenario Memory to continuously maintain task progress as event evidence arrives, allowing earlier observations to support subsequent decisions beyond the detector's visual window. Even when no notification is required, relevant events continue to update the scenario state. On CoTriggerBench, TEMPI achieves 44.4% instruction-level success, outperforming the strongest closed-source and fine-tuned open-source VLM baselines by 15.6 (Qwen3.8-Omni-Flash-Realtime) and 18.1 (MOSS-VL-Realtime) percentage points, respectively. To support efficient on-device streaming inference, we further optimize the streaming inference pipeline by overlapping partial visual encoding and LLM prefix prefill with frame arrival. On an 8 GB RTX 5060, our optimized detector implementation achieves a 4.90×\times speedup.

Introduction

Streaming video assistants continuously observe incoming video and respond to users based on their queries. They can support everyday tasks by monitoring relevant situations and providing timely assistance, reducing the need for users to watch continuously. In practice, users express their needs through natural-language instructions that often include prerequisites. For example, a user may ask, “Notify me when the pot boils after someone turns on the stove and leaves the kitchen” (Figure 1a). Detecting the pot boiling alone is insufficient. The assistant must also verify that the stove was lit and a person left the kitchen before the pot boils. Premature or missed notifications undermine the purpose of delegation. Motivated by this need, we study compositional event triggering: determining when user-specified combinations of events and temporal constraints are satisfied over a video stream, and issuing notifications at the appropriate moment using only observations available so far.

Figure 1. Atomic and compositional triggering. Figure (a) shows that existing models succeed with atomic queries and explicit execution but fail on the compositional instruction. Figure (b) reports request-level accuracy rates in atomic and composite triggering events. Most models achieve much higher trigger accuracy in atomic events than compositional events. † denotes closed models.

A key challenge in compositional event triggering is maintaining instruction progress across video observations, even when no notification is required. In the stove example, we refer to turning on the stove and leaving the kitchen as prerequisite events, and the pot boiling as the target event. As prerequisite events are observed, the assistant must update its record of satisfied conditions without issuing a notification. When the target event occurs, it must determine whether the accumulated evidence satisfies the instruction’s prerequisites and temporal constraints. Existing approaches learn response readiness [2, 23] or retain historical evidence and structured task information [8, 31, 34, 36], but these capabilities do not necessarily ensure correct execution of compositional instructions. As shown in Figure 1b, fine-tuning on task-specific event data [18] improves atomic-event accuracy, while gains in compositional triggering remain limited. Several fine-tuned VLMs [22, 29, 35] achieve over 50% atomic accuracy but remain below 25% compositional accuracy. This gap indicates that recognizing individual events is insufficient for executing compositional instructions.

To evaluate the gap between atomic event recognition and compositional instruction execution, we introduce CoTriggerBench. Using event annotations from Perception Test [18], we define five task types: conditional, sequential, counting, timing, and adversarial, covering prerequisites, event ordering, occurrence counts, delays, and silence when the specified conditions are not satisfied. We construct the tasks through event composition, natural-language instruction generation, and consistency verification, with expected outcomes validated through deterministic event replay (Figure 2). The test set contains 448 instructions across 96 videos, including 82 adversarial cases, evaluating notification timing and correct silence.

CoTriggerBench construction pipeline

Figure 2. CoTriggerBench construction pipeline. The full construction pipeline comprises event normalization, task composition, query generation, verification, and dataset formation.

We introduce TEMPI (Temporal Execution and Monitoring of Planned Instructions), a streaming video agent for compositional event triggering that separates event detection from instruction execution. TEMPI maintains instruction progress in Scenario Memory, consisting of a Scenario Plan and a Scenario State. Specifically, a 4B planner decomposes each instruction into atomic event queries and their compositional constraints, while a 2B detector extracts timestamped event evidence from the video stream. During execution, TEMPI updates the scenario state based on the event evidence and compositional constraints, tracking satisfied conditions, occurrence counts, and timing information to determine when to issue notifications. Even when no notification is required, relevant events continue to update the scenario state, allowing earlier observations to inform subsequent decisions after they leave the visual window.

On CoTriggerBench, TEMPI achieves the SOTA instruction-level success rate in compositional instruction execution (44.4%), outperforming existing real-time VLMs such as Qwen3.8-Omni-Flash-Realtime (28.8%) and Gemini 3.8 Live (20.5%), as well as MOSS-VL-Realtime (11B, 26.3%) and JoyAI-VL-Interaction (8B, 26.1%), which are fine-tuned on the same dataset as TEMPI. Further experiments demonstrate that these improvements stem not only from event perception but also from explicit instruction decomposition and stateful execution. The TEMPI framework also improves compositional instruction execution across different perception backends. For example, integrating TEMPI improves instruction-level success for fine-tuned JoyAI from 26.1 % to 39.1% and Intern-Decision from 18.3% to 43.1%, demonstrating its applicability across diverse perception models. On a single H800, TEMPI achieves a median GPU execution time of only 61.0 ms per query/window, compared with 240.3 ms for JoyAI, representing 3.94× faster GPU execution.

To further support efficient on-device streaming inference, we propose a streaming early-execution strategy. Instead of waiting for all four frames in a decision window, we initiate partial ViT encoding and LLM prefix prefill after the first two frames arrive. Once the remaining frames become available, inference continues with suffix prefill by reusing the cached prefix KV states, thereby reducing decision latency after the final frame arrives. On an 8 GB RTX 5060, the optimized NVFP4 implementation achieves up to a 4.90× speedup in post-window decision latency, while nearly preserving instruction-level success (43.8% vs. 44.4%), demonstrating TEMPI’s potential for efficient streaming inference under edge deployment.

Our contributions are as follows:

  • We formulate compositional instruction triggering in streaming video and introduce CoTriggerBench. CoTriggerBench evaluates instructions involving prerequisites, event ordering, occurrence counts, and delays. By assessing both timely notifications and correct silence, CoTriggerBench reveals the gap between event recognition and compositional instruction execution in streaming VLMs.

  • We propose TEMPI, a compositional-triggering agentic framework that integrates instruction planning, event detection, and stateful execution to track task progress and determine when to issue notifications. Specifically, TEMPI achieves the highest success rate on CoTriggerBench (44.4%), outperforming the strongest baseline (28.8%).

  • We develop an efficient streaming-aware inference pipeline for edge deployment. Instead of waiting for the complete video window, our pipeline overlaps early ViT encoding and LLM prefix prefill with streaming frame arrival, reusing cached prefix KV states to minimize post-window decision latency. On an 8 GB RTX 5060, our optimized implementation achieves a 4.90× speedup over native inference for three queries, while preserving effectiveness.

Related Work

Benchmarks for proactive interaction. Existing benchmarks evaluate diverse aspects of proactive video interaction. StreamingBench [16] and OVO-Bench [14] cover online video understanding; OmniMMI [25], ProactiveVideoQA [24], LiveProBench [7], and OmniPro [33] assess alerting, response timing, silence, counting, and monitoring, while IPIBench [13] focuses on dynamic requests. In contrast, CoTriggerBench evaluates single-turn instructions with explicit temporal constraints, including prerequisites, event ordering, occurrence counts, and event-relative delays, requiring timely notifications when conditions are satisfied and silence otherwise.

Proactive streaming models. Executing temporally constrained instructions requires maintaining task progress beyond local visual observations. Existing streaming models retain visual context through compact or hierarchical memory [12, 26, 31], regulate responses through readiness estimation or inference gating [2, 6, 21, 23], and support continuous interaction or lightweight decisions [22, 27, 29, 35]. Other approaches match event cues [34], anticipate relevant observations [28], or leverage task graphs for proactive actions [36], while structured programs compose perception and execution [11, 15]. In contrast, TEMPI explicitly represents prerequisites, ordering, counts, and delays through Scenario Memory, maintaining persistent execution state from interchangeable event detectors.

Benchmark

We introduce CoTriggerBench, a benchmark for compositional event triggering across five notification task types grounded in annotated video events.

Task Definition

Given a user instruction qq and a streaming video VV, a streaming VLM fθ(⋅)f_{\theta}(\cdot) produces an output at time tt based on the current and past video observations V≤tV_{\leq t}:

ot=fθ(q,V≤t).(1)o_t = f_{\theta}(q, V_{\leq t}). \tag*{(1)}

Here, qq is a natural language instruction that specifies logical and temporal constraints, and oto_t represents either a generated notification or silence.

Task Types. Let AitA_i^t denote that event AiA_i occurs at time tit_i, where tt is the current decision time. We define five task types to evaluate whether streaming VLMs can follow complex instructions involving event conditions, temporal order, counting, and timing Figure 3.

CoTriggerBench task types

Figure 3. CoTriggerBench task types. This figure shows event timelines and notification outcomes for the five task types.

  • Conditional. A notification is triggered when the target event AnA_n occurs after all required conditions have been satisfied, regardless of their relative order, defined as q=(⋀i=1n−1Aiti)≺Antq = \left(\bigwedge_{i=1}^{n-1} A_i^{t_i}\right) \prec A_n^t.

  • Sequential. A notification is triggered when events occur in the specified order, with the current event completing the sequence, defined as q=A1t1≺A2t2≺⋯≺Antq = A_1^{t_1} \prec A_2^{t_2} \prec\cdots\prec A_n^t.

  • Counting. A notification is triggered at the kk-th distinct occurrence of event AA, defined as q=At∧[NA(t)=k]q = A^t \land[N_A(t) = k], where NA(t)N_A(t) denotes the number of distinct object occurrences of AA up to time tt.

  • Timing. A notification is triggered when a specified time interval Δ\Delta has elapsed since a reference event AA, defined as q=AtA∧[t−tA=Δ]q = A^{t_A} \land[t - t_A = \Delta], where tAt_A denotes the time of the reference event. A small timing tolerance is allowed during evaluation.

  • Adversarial. Adversarial tasks pair valid instructions with videos that do not satisfy their constraints, requiring the model to remain silent even when some relevant events occur. For example, a sequential instruction q=A1t1≺A2t2q = A_1^{t_1} \prec A_2^{t_2} may be paired with a video satisfying ¬(A1t1≺A2t2)\neg(A_1^{t_1} \prec A_2^{t_2}). Sim-

ilar negative cases can be constructed for the other tasks.

Benchmark Construction

Event normalization. Perception Test [18] provides temporally localized action segments linked to object tracks, with action labels, object identities, and start/end timestamps. Following Figure 2, we associate each action with its annotated objects, normalize action–object categories, and convert timestamps to seconds while preserving source boundaries and identifiers. We define each resulting record, comprising an action, its associated objects, and a temporal interval, as a primitive event.

Task composition. Within each video, type-specific rules enumerate task candidates from primitive events. Distinct, nonoverlapping event pairs and ordered triples form conditional and sequential candidates. Repeated occurrences of the same action–object category form counting candidates. Timing candidates attach a duration to an anchor, with timer expiry constrained to the video duration. Adversarial candidates pair an earlier target with a later prerequisite and require no annotated target recurrence afterwards. Temporal-gap and duration checks constrain event selection. Each candidate records event roles, dependencies, any count or delay, and the expected outcome.

Query generation. We use DeepSeek-V4.1-Flash [5] to express structured task candidates as natural-language notification requests. The prompt specifies the task type, primitive-event descriptions, dependency order, and any count or delay. The prompt is provided in the supplementary material. The resulting instruction states the requested trigger condition with explicit temporal requirements and object roles. We also derive canonical, subject-free atomic queries for the constituent primitive events.

Verification. GLM-5.3-Flash [30] and Gemini 3.8 Flash [10] check the visual grounding of event descriptions and object roles against video evidence. Human annotators additionally inspect all test samples for visual alignment, the naturalness of compositional queries, and consistency between each query and its constituent subgoals. To verify executability, we deterministically compile candidates into gold plans and validate event references, dependency graphs, and count/delay parameters. We check instruction–plan consistency and replay source event intervals using the task-state update rules to verify that the resulting notification or silence matches the expected outcome.

Dataset formation. Table 1 summarizes the video-disjoint splits. Training retains the inherited source instruction pool and supplies atomic-event supervision and compiler examples. Validation selects thresholds and policies, and each frozen configuration is evaluated once on test. We exclude evaluation tasks with ambiguous event/object references, unresolved event roles, or invalid plan expansion because they lack an unambiguous, executable ground truth. The retained evaluation sets contain 451 validation tasks and 448 test tasks. Test comprises 366 positive and 82 adversarial tasks over 96 videos.

Task typeTrainVal.Test
Conditional9,2709595
Sequential4,4109096
Counting3,9128784
Timing3,8069191
Adversarial7,4578882
Total Instructions28,855451448
Unique Videos7,1039696

Table 1. Benchmark split statistics. Task rows count instructions.

Evaluation Protocol

Trigger timing. For a positive task with source target interval [s,e][s,e], the first notification is accepted within

W(q)=[s−δ,e+δ],δ=1.6 s(2)\mathcal{W}(q) = [s-\delta,e+\delta], \qquad\delta= 1.6\ \mathrm{s} \tag*{(2)}

For timing tasks, we shift the anchor interval by the requested delay before applying Eq. (2). These intervals are independent of model sampling rates and discretized stream labels.

Metrics. Let P\mathcal{P} and F\mathcal{F} denote positive and adversarial tasks. Let t^(q)\hat{t}(q) be the first notification time, with t^(q)=+∞\hat{t}(q)=+\infty if none is issued. Define H(q)=1[t^(q)∈W(q)]H(q)=\mathbf{1}[\hat{t}(q)\in\mathcal{W}(q)], and let Z(q)Z(q) indicate valid output with no notification. Trigger success (TS), adversarial silence (AS), and strict success are

TS=∑q∈PH(q)∣P∣(3)\mathrm{TS} = \frac{\sum_{q\in\mathcal{P}} H(q)}{|\mathcal{P}|} \tag*{(3)}
AS=∑q∈FZ(q)∣F∣(4)\mathrm{AS} = \frac{\sum_{q\in\mathcal{F}} Z(q)}{|\mathcal{F}|} \tag*{(4)}
Strict=∑q∈PH(q)+∑q∈FZ(q)∣P∣+∣F∣(5)\mathrm{Strict} = \frac{\sum_{q\in\mathcal{P}} H(q)+\sum_{q\in\mathcal{F}} Z(q)}{|\mathcal{P}|+|\mathcal{F}|} \tag*{(5)}

TS is measured on conditional, sequential, counting, and timing tasks that require a notification. It measures whether the first notification falls within the acceptance window for a compositional instruction. AS is measured only on adversarial tasks, which require silence because their constraints are unmet. Strict success is the primary metric and reports the overall instruction-level success rate across both groups according to the TS and AS criteria above. Furthermore, we evaluate single-event detections by measuring all preceding events and triggering events under compositional instructions with the same tolerance and report event-level precision, recall, and F1.

Methodology

We propose TEMPI, a streaming video agent that separates event perception from compositional instruction execution through Scenario Memory (Figure 4). A planner decomposes each instruction into atomic event queries and temporal constraints, forming a Scenario Plan that guides event detection. As event evidence arrives, TEMPI updates the Scenario State and issues a notification once all constraints are satisfied.

Figure 4. TEMPI architecture and streaming execution. A skill-guided TEMPI planner decomposes the instruction into atomic queries and compositional constraints to form the scenario plan. The detector supplies event/silence evidence from recent frames, and the TEMPI agent writes scenario state updates into scenario memory, activates dependent steps, and issues a notification once all constraints are satisfied. External backends can supply the same event interface.

Task-Conditioned Scenario Memory

Unlike streaming models such as JoyAI [29] and MOSS [22], which retain both current and historical visual information within a limited visual-token budget, TEMPI maintains compositional constraints and task progress in Scenario Memory outside the detector’s visual context, allowing the detector to focus on local event perception. Specifically, the detector processes non-overlapping windows of four consecutive frames, while Scenario Memory preserves task progress across windows.

Scenario Memory consists of a fixed Scenario Plan and an evolving Scenario State. Guided by a fixed decomposition skill, a planner using Qwen3.5-4B [19] constructs the plan once per instruction, specifying atomic event queries and compositional constraints, including task types, step dependencies, event boundaries, occurrence counts, and delays. The skill specifies the output schema, task-type selection rules, and examples for extracting atomic queries while preserving action and object semantics. The full planner skill is provided in the supplementary material. The Scenario State records step activation and completion, event counts, and timestamps, and is continuously updated using event evidence extracted from the current video window.

Query-Conditioned Event Detection

Given a four-frame window vv and an atomic event query qq, the Event Detector predicts whether the queried event is present. We use Qwen3-VL-2B-Instruct [3] with a two-class linear head applied to the final hidden state. The head produces event and silence logits, where silence denotes absence of the queried event even when other actions occur. We train the language backbone and decision head while freezing the vision encoder, visual merger, and shared embeddings. We define the event margin as m(v,q)=zevent−zsilencem(v,q)=z_{\mathrm{event}}-z_{\mathrm{silence}} and threshold it at inference.

To distinguish the queried event from other visible actions, we combine background windows and negative windows reliably away from annotated events with query replacements on the same visual window. Replacement queries are selected for semantic overlap from the same or other videos and supervised only with valid negative evidence. For densely overlapping annotations, we disable same-video replacements and background negatives.

We train the detector with binary state supervision and query contrast. The state loss applies weighted cross-entropy to known event/silence labels, while query contrast compares a positive query with valid replacement negatives on the same window. For a positive query q+q^{+} and known-negative queries Q−\mathcal{Q}^{-}, define Q={q+}∪Q−\mathcal{Q}=\{q^{+}\}\cup\mathcal{Q}^{-} and sq=m(v,q)/τs_q=m(v,q)/\tau. The objective is

Lqc=−log⁡exp⁡(sq+)∑q∈Qexp⁡(sq)(6)\mathcal{L}_{\mathrm{qc}}=-\log\frac{\exp(s_{q^{+}})}{\sum_{q\in\mathcal{Q}}\exp(s_q)} \tag*{(6)}
L=Lstate+λLqc,(7)\mathcal{L}=\mathcal{L}_{\mathrm{state}}+\lambda\mathcal{L}_{\mathrm{qc}}, \tag*{(7)}

where τ=1\tau=1 and λ=0.25\lambda=0.25. Query contrast applies only when the positive query is labeled as an event and at least one negative query has valid supervision. It penalizes assigning a higher event margin to an irrelevant query than to the supported query, making query discrimination an explicit training objective.

Memory-Guided Scenario Execution

As shown in Figure 4, upon receiving an instruction, TEMPI invokes its planner once and stores the resulting Scenario Plan of atomic queries and constraints in Scenario Memory. For each four-frame window, the detector evaluates the atomic queries and returns timestamped event/silence predictions. TEMPI groups consecutive positive decisions into an event run and transitions between event and silence identify its onset and offset, allowing TEMPI to determine event completion. Completed events update Scenario State, which is maintained in Scenario Memory to track task progress and evaluate constraints against the fixed Scenario Plan. Once all prerequisites are satisfied, TEMPI submits the final target query to the detector. A valid event prediction for the target triggers a single notification and completes the instruction. Otherwise, TEMPI continues monitoring and updating the state without notifying the user.

Experiments

Experimental Setup

All methods are evaluated on the held-out test split of CoTriggerBench, comprising 448 task queries across 96 videos (§ 3). We retain each baseline’s official native inference pipeline and adapt its outputs to the notification protocol described below.

Baseline settings. The main comparison evaluates baselines through their full-instruction Direct interface and our complete TEMPI system. Controlled Direct/TEMPI comparisons are reported in the ablation studies. JoyAI-VL-Interaction [29] and MOSS-VL-Realtime [22] retain native streaming memory at 1 fps, with non-silence responses counted as notifications. Intern-Decision [35], OmniJev [27], and our detector use four-frame windows at 5 fps. StreamingVLM [26] and LiveCC [4] use native incremental inference at 2 fps with an ALERT prompt. Em-Garde [34] uses native proposal–matching inference at 2 fps, with surge detections counted as notifications.

Gemini 3.8 Live [9] and Qwen3.5/3.8-Omni-Flash-Realtime [1] use their native APIs with 1 fps video and ALERT readouts. Em-Garde is Direct-only; all other maintainable baselines support paired Direct and TEMPI evaluation with backend weights and native sampling held fixed.

Training settings. Our detector and all SFT baselines are fine-tuned on the CoTriggerBench training split to reduce mismatch with the evaluation domain. Baseline SFT follows the supervised fine-tuning formulations and output interfaces described in the official releases. Our detector uses the objective in § 4.2. Checkpoints, thresholds are selected on validation data and fixed for testing.

Metrics. All tasks are scored, including failed plans and missing outputs. As defined in § 3, TS measures whether the first notification falls within the accepted window on positive tasks, and AS measures whole-stream silence on adversarial tasks. Strict success, our primary metric, combines correct outcomes across both groups. MAE measures first-notification timing error over emitted positive tasks; silent tasks are excluded from MAE but remain scored in TS and Strict. The single-event columns report precision, recall, F1, and false event runs per minute (FP/min).

Baseline Comparison

Overall performance. As shown in Table 2, TEMPI achieves the highest strict success in the main comparison, 44.4%, versus 28.8% for Qwen3.8-Omni, the strongest baseline. Em-Garde attains a strict success of 8.9% with no adversarial silence. For atomic event detection, our detector achieves the highest F1 (61.2%), exceeding Intern-Decision (56.0%) while producing fewer false events (1.29 versus 2.45/min). Across the four open-source baselines, fine-tuning improves precision and reduces false events.

ModelParamsStrict (%) ↑TS (%) ↑AS (%) ↑MAE (s) ↓F1 (%) ↑P (%) ↑R (%) ↑FP/min ↓
Zero-shot
JoyAI-VL-Interaction [29]8B19.221.68.56.8421.315.832.85.62
MOSS-VL-Realtime [22]11B18.316.426.85.1033.325.149.74.76
StreamingVLM [26]7B15.63.868.36.393.027.61.60.14
LiveCC-7B-Instruct [4]7B4.94.18.511.3810.210.59.92.69
Em-Garde [34]2B+7B8.910.90.09.5228.519.850.76.59
Intern-Decision [35]2B18.57.169.56.3543.833.363.84.09
OmniJev [27]2B17.43.679.36.5840.531.457.14.00
Gemini 3.8 Live [9]–20.510.963.44.1916.929.511.90.91
Qwen3.5-Omni-Flash-Realtime [1]–16.719.44.97.4036.533.939.42.46
Qwen3.8-Omni-Flash-Realtime [1]–28.834.72.44.6840.439.541.42.04
Fine-tuned
JoyAI-VL-Interaction [29]8B26.123.537.84.3550.357.944.51.04
MOSS-VL-Realtime [22]11B26.320.552.44.6552.369.242.10.60
Intern-Decision [35]2B18.37.964.66.4156.047.368.62.45
OmniJev [27]2B15.83.073.27.1344.137.653.32.84
TEMPI2B+4B44.440.462.21.9761.260.661.81.29

Table 2. Compositional triggering and atomic event detection. Strict, TS, and AS denote overall instruction success, positive-task trigger success, and adversarial silence. These rates, precision (P), recall (R), and their harmonic mean (F1) are reported as percentages. MAE is the mean absolute distance from the first notification to the unextended target interval, in seconds; FP/min counts false event runs per video minute. Params are in billions (B). For Em-Garde, 2B+7B denotes its embedding and proposal/response models, respectively. For TEMPI, 2B+4B denotes our detector and planner, respectively. The best result of each metric is shown in bold.

Recognition–execution gap. Fine-tuning improves atomic-event F1 for all four open-source baselines, yet these gains do not consistently translate into Direct task success (Table 2). Intern-Decision’s F1 rises from 43.8% to 56.0% while strict success remains nearly unchanged (18.5% to 18.3%); OmniJev’s F1 rises from 40.5% to 44.1% while strict success declines from 17.4% to 15.8%. With Direct thresholds selected on all 451 validation tasks, these classifiers achieve AS silence of 64.6%–79.3% but TS of only 3.0%–7.9%: withholding notifications reduces false alarms while missing most positive triggers. Stronger local recognition alone does not ensure correct execution of a compositional instruction. TEMPI maintains task progress and enforces the corresponding state transitions.

Ablation Studies

Effect of the TEMPI execution. We compare Direct and TEMPI execution while holding backend weights fixed. As shown in Table 3, our workflow improves strict success for four of the five zero-shot backends shown and all four fine-tuned baselines. Explicit task decomposition and stateful execution improve overall success while exposing a trigger–silence tradeoff. TEMPI-D denotes our 2B detector (D for detector). Direct execution supplies the full compositional instruction to TEMPI-D, whereas TEMPI execution uses its atomic-event predictions within our planning and scenario-state framework. Table 3 shows that compiled execution more than doubles strict success from 18.3% to 44.4%; TS increases from 10.9% to 40.4%; AS rises from 51.2% to 62.2%. The large gains highlight the value of decomposing compositional instructions and enforcing task-state constraints beyond event detection alone.

BackendTS (%) ↑ DTS (%) ↑ TAS (%) ↑ DAS (%) ↑ TStrict (%) ↑ DStrict (%) ↑ TMAE (s) ↓ DMAE (s) ↓ T
Zero-shot
StreamingVLM3.80.868.396.315.618.36.394.51
LiveCC4.19.38.547.64.916.311.386.99
Gemini Live10.94.163.487.820.519.44.195.21
Qwen3.5-Omni19.426.84.928.016.727.07.404.67
Qwen3.8-Omni34.727.92.443.928.830.84.684.55
Fine-tuned
JoyAI23.530.337.878.026.139.14.352.88
MOSS20.526.852.478.026.336.24.653.17
Intern-D7.944.564.636.618.343.16.412.79
OmniJev3.023.073.240.215.826.17.134.03
TEMPI-D10.940.451.262.218.344.45.251.97

Table 3. Effect of TEMPI across perception backends. D/T: Direct/TEMPI execution. D supplies the full instruction directly to the backend; T uses it as an atomic-event detector in our framework. TEMPI-D denotes our 2B detector. TS, AS, and Strict are percentages; MAE is in seconds. Better values within each pair are bold.

Frame-rate alignment. We examine whether aligning baseline input frame rates with our detector improves performance. Fine-tuned JoyAI and MOSS receive 5 fps input instead of their native 1 fps, with consistent 1 s decision intervals, memory policies, and decoding settings fixed. As shown in Table 4, denser input reduces both TEMPI strict success and atomic-event F1 for both backends compared with their native settings.

BackendFPSDirect (%) TS↑Direct (%) AS↑Direct (%) Strict↑TEMPI (%) TS↑TEMPI (%) AS↑TEMPI (%) Strict↑Atomic events (%) P↑Atomic events (%) R↑Atomic events (%) F1↑
JoyAI123.537.826.130.378.039.157.944.550.3
JoyAI521.667.129.924.076.833.753.037.443.9
MOSS120.552.426.326.878.036.269.242.152.3
MOSS57.478.020.39.691.524.666.918.729.2

Table 4. Effect of input frame rate. Better scores within each 1/5 fps pair are bold. Increasing the frame rate does not guarantee better performance; native settings yield higher TEMPI strict success and atomic-event F1 for both backends.

Planner Capability

As shown in Table 5, as the parameter quantity of the scenario planning agent increases, the quality of the scenario task memory construction improves accordingly, especially in task-type prediction and the strict match of sub-task decomposition and semantic consistency. However, the most fundamental factor for the successful execution of a task is the consistency of the number of sub-tasks. Beyond a parameter quantity of 4B, there is no significant improvement in this value. The subtle differences in semantics may not lead to detection errors, and the classification error of task type may not affect the execution sequence of the execution graph. To balance inference speed and effectiveness, we select the 4B agent as the final scenario planning agent.

PlannerQty.Sem.MatchTypeP50P95
Qwen3.5-2B97.391.285.788.686.7107.5
Qwen3.5-4B99.392.787.790.9129.0160.6
Qwen3.5-9B99.393.589.194.4179.2226.6
Qwen3.6-35B-A3B98.793.488.698.2159.6244.2
Qwen3.8-27B99.194.790.498.9709.8878.2

Table 5. Scenario Planning Agent size. Planning quality is reported as percentages. Qty.: correct subquery count rate. Sem.: position-wise semantic equivalence judged by Qwen3.8-Flash-Next-FP8 [20]. Match: correct count and equivalent queries. Type: task-type accuracy. Latency: complete-plan request time on one H800.

Computational Efficiency. TEMPI requires only one planning call per instruction, leaving repeated inference of the event detector. The planner incurs a latency of 129 ms on an H800 (Table 5). For subsequent detection, our detector takes 61.0 ms per query/window, compared with 240.3 ms for JoyAI, using one-quarter of the parameters with a 3.94× speedup in GPU execution (Table 6). Under the same TEMPI framework, it also achieves higher strict success than JoyAI (44.4% vs. 39.1%; Table 3), demonstrating efficient continuous detection with strong task performance.

ModelParamsP50 ↓P90 ↓P99 ↓
TEMPI-D2B61.062.767.9
Intern-Decision-2B [35]2B75.282.085.7
OmniJev-2B [27]2B165.0170.6174.4
JoyAI-VL-Interaction [29]8B240.3469.6573.1
MOSS-VL-Realtime [22]11B91.695.5219.9
StreamingVLM [26]7B93.698.5105.8
LiveCC-7B-Instruct [4]7B67.884.4116.9
Em-Garde [34]2B53.355.056.6

Table 6. Detector GPU execution latency (ms). Measurements are obtained on one H800 with a batch size of 1. TEMPI-D denotes our 2B detector.

Edge Deployment

Streaming Prefix Overlap. We exploit idle intervals in streaming frame collection to overlap computation with frame arrival. In the baseline pipeline, inference starts only after all four frames in a decision window arrive. The wait for frames 3–4 therefore creates a computation bubble, where frames 1–2 are already available, but their processing is deferred. In Figure 5a, visual encoding of frames 1–2 is independent of frames 3–4, allowing us to fill this bubble with early ViT encoding and LLM prefix prefill.

Figure 5. Local efficiency and early execution on an RTX 5060 (8 GB). Figure (a) shows how early ViT and prefix prefill overlap frame arrival; cached KV features suffix prefill after frame 4. Figure (b) reports decision latency P50 for one/three queries and independently evaluated strict success. †: VRAM measured per precision with three queries.

Once frame 4 arrives, inference continues with the remaining frames and suffix prefill, reusing the prefix KV caches. This remaining computation fits within the otherwise idle interval in which the next window collects frames 1–2, completing before its prefix computation begins. The schedule thus reduces decision latency after the final frame arrives while avoiding contention between consecutive windows.

Result Analysis. We deploy the detector on a single RTX 5060 with 8 GB of VRAM. In addition to streaming prefix overlap, we apply quantization to reduce memory consumption. The FP8 implementation uses native Deep-GEMM [32] kernels, while the NVFP4 implementation uses native cuBLASLt [17] FP4 GEMMs with fused scaling and output operations. We further reduce execution overhead through compilation, kernel fusion, CUDA graphs, and shared activation quantization. As shown in Figure 5b, the FP8 configuration with early execution achieves 2.95× and 4.23× speedups over native BF16 in median decision latency for one and three queries per batch, respectively. The NVFP4 configuration with early execution increases these speedups to 3.33× and 4.90×. Separate precision measurements show VRAM reductions of 29.3% and 40.7% for FP8 and NVFP4, respectively.

Conclusion

We introduce CoTriggerBench to evaluate compositional event triggering in streaming video and TEMPI to bridge the gap between event recognition and instruction execution. By maintaining persistent task progress through Scenario Memory, TEMPI achieves state-of-the-art instruction-level success (44.4%), while its compact 2B detector achieves the highest atomic-event F1 (61.2%) among evaluated models. Furthermore, we combine low-bit quantization with early ViT and LLM prefix execution to optimize streaming inference, achieving up to 4.90×4.90 \times speedup in post-window decision latency on an 8 GB RTX 5060.

LLM Usage Statement

DeepSeek-V4.1-Flash was used to generate natural-language benchmark instructions. GLM-5.3-Flash and Gemini 3.8 Flash assisted with visual-grounding checks during benchmark construction. Qwen3.8-Flash-Next-FP8 was used to judge the semantic equivalence of planner-generated atomic queries. Qwen3.5-4B and Qwen3-VL-2B-Instruct provide the planning and perception components described in § 4; other evaluated LLM-based systems are identified in § 5. OpenAI Codex assisted with manuscript editing, language polishing, LaTeX formatting, and submission checks. The authors retain responsibility for the scientific content, experimental results, citations, and conclusions.

References

  1. [1]Alibaba Cloud. Qwen-Omni-Realtime real-time audio video chat model, 2026. Accessed October 8, 2026. 6, 7
  2. [2]Shehreen Azad, Vibhav Vineet, and Yogesh Singh Rawat. StreamReady: Learning what to answer and when in long streaming videos. arXiv preprint arXiv:2603.08620, 2026. 2, 3arxiv.org/abs/2603.08620
  3. [3]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. 5arxiv.org/abs/2511.21631
  4. [4]Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. LiveCC: Learning video LLM with streaming speech transcription at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 6, 7, 8arxiv.org/abs/2504.16030
  5. [5]DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026. 4
  6. [6]Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Donglin Bai, Zhibo Chen, and Ting Cao. StreamMind: Unlocking full frame rate streaming video dialogue through event-gated cognition. arXiv preprint arXiv:2503.06220, 2025. 3arxiv.org/abs/2503.06220
  7. [7]Kaixuan Du, Xin Wan, Hang Zhang, Meng Cao, Dai Guan, Ming Chen, and YuKun Wang. LiveProBench: Can streaming video models really interact like humans? arXiv preprint arXiv:2609.12658, 2026. 3
  8. [8]Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, and Yifei Huang. Vinci2: Providing proactive assistance in continuous egocentric videos. arXiv preprint arXiv:2607.11523, 2026. 2arxiv.org/abs/2607.11523
  9. [9]Google. Live API capabilities guide, 2026. Accessed October 8, 2026. 6, 7
  10. [10]Google DeepMind. Gemini 3.8 Flash model card, 2026. Accessed October 12, 2026. 4
  11. [11]Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.11559, 2022. 3arxiv.org/abs/2211.11559
  12. [12]Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: OVBench and VideoChat-Online. arXiv preprint arXiv:2501.00584, 2025. 3arxiv.org/abs/2501.00584
  13. [13]Jinzhao Li, Yinuo Chen, Wenxuan Song, Yijia Lei, Yichi Zhang, Honglei Yan, Panwang Pan, and Miao Liu. IPIBench: Evaluating interactive proactive intelligence of MLLMs under continuous streams. arXiv preprint arXiv:2605.27074, 2026. 3arxiv.org/abs/2605.27074
  14. [14]Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihai He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. OVO-Bench: How far is your Video-LLMs from real-world online video understanding? arXiv preprint arXiv:2501.05510, 2025. 3arxiv.org/abs/2501.05510
  15. [15]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022. 3arxiv.org/abs/2209.07753
  16. [16]Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streaming-Bench: Assessing the gap for MLLMs to achieve streaming video understanding. arXiv preprint arXiv:2411.03628, 2024. 3arxiv.org/abs/2411.03628
  17. [17]NVIDIA Corporation. cuBLAS library: cuBLASLt API. API documentation, 2026. Accessed October 26, 2026. 2
  18. [18]Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Suslyk, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. arXiv preprint arXiv:2305.13786, 2023. 2, 4arxiv.org/abs/2305.13786
  19. [19]Qwen Team. Qwen3.5-4B. Model card. https://huggingface.co/Qwen/Qwen3.5-4B, 2026. 5DOI
  20. [20]Qwen Team. Qwen3.8-Flash-Next: A new architecture, towards ultimate cost-efficiency, 2026. 8
  21. [21]Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. StreamBridge: Turning your offline video large language model into a proactive streaming assistant. arXiv preprint arXiv:2505.05467, 2025. 3arxiv.org/abs/2505.05467
  22. [22]Pengyu Wang, Chenkun Tan, Shaojun Zhou, Qirui Zhou, Yanxin Chen, Xingyang He, Huazheng Zeng, Jijun Cheng, Chenghao Wang, Xiaomeng Qian, Pengfei Wang, Zhan Huang, Shanqing Gao, Wei Huang, Longjun Cao, Wu Ran, Jie Liu, Changtai Zhu, Hongkai Wang, Yixian Tian, Chenghao Liu, Zhen Ye, Xinghao Wang, Botian Jiang, Guoguo Feng, Zhaoye Fei, Ruixiao Li, Mingshu Chen, Yang Gao, Qinyuan Cheng, Shimin Li, and Xipeng Qiu. MOSS-VL technical report. arXiv preprint arXiv:2608.15045, 2026. 2, 3, 5, 6, 7, 8arxiv.org/abs/2608.15045
  23. [23]Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. VideoLLM knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction format. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6338–6359. Association for Computational Linguistics, 2025. 2, 3
  24. [24]Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, and Dongyan Zhao. ProactiveVideoQA: A comprehensive benchmark evaluating proactive interactions in video large language models. arXiv preprint arXiv:2507.09313, 2025. 3arxiv.org/abs/2507.09313
  25. [25]Yuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu, Dongyan Zhao, and Zilong Zheng. OmniMMI: A comprehensive multi-modal interaction benchmark in streaming video contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 3arxiv.org/abs/2503.22952
  26. [26]Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. StreamingVLM: Real-time understanding for infinite video streams. In International Conference on Learning Representations, pages 61463–61475, 2026. 3, 6, 7, 8arxiv.org/abs/2510.09608
  27. [27]Tianrun Xu, Hongbang Fan, Jiahao Lin, Zilin Zhu, Zhenxin Diao, Longteng Guo, and Jing Liu. OmniJev: An omni-modal system one decision model. Software repository, https://github.com/tinell123666888/OmniJev, 2026. 3, 6, 7, 8
  28. [28]Haolin Yang, Feilong Tang, Lingxiao Zhao, Xinlin Zhuang, Yifan Lu, Xiang An, Ming Hu, Xiaofeng Zhang, Abdalla Swikir, Junjun He, Zongyuan Ge, Muhammad Haris Khan, and Imran Razzak. StreamAgent: Towards anticipatory agents for streaming video understanding. arXiv preprint arXiv:2508.01875, 2025. 3arxiv.org/abs/2508.01875
  29. [29]Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Xiangyu Zeng, Yifei Li, Haowen Hou, Zheming Liang, Congcong Wang, Kaiwen Tuo, Jun Zhang, Yuhan Zhu, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, and Jiaqi Wang. JoyAI-VL-Interaction: Real-time vision-language interaction intelligence. arXiv preprint arXiv:2606.14777, 2026. 2, 3, 5, 6, 7, 8arxiv.org/abs/2606.14777
  30. [30]Z.ai. GLM-5.3-Flash. Model card, 2026. Accessed October 12, 2026. 4DOI
  31. [31]Xiangyu Zeng, Yuandong Yang, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Qingyi Si, Dingyu Yao, Changlian Ma, Haoran Chen, Xinyu Chen, Yansong Shi, Junhao Zhou, Yifei Li, Jun Zhang, Chuanyu Qin, Chenxu Yang, Xinlei Yu, Kun Ouyang, Yuchen Shao, Qianshan Wei, Changhai Zhou, Jun Gao, Jiaqi Wang, and Limin Wang. OneStreamer: Unifying perception, memory, and proactive response in streaming video interaction. arXiv preprint arXiv:2610.01762, 2026. 2, 3arxiv.org/abs/2610.01762
  32. [32]Chenggang Zhao, Zhean Xu, Liang Zhao, Jiashi Li, Chenhao Xu, Anyi Xu, Shengyu Liu, Kexing Zhou, and Kuai Yu. DeepGEMM: Clean and efficient BLAS kernel library on GPU. https://github.com/deepseek-ai/DeepGEMM, 2025. 8
  33. [33]Ruixiang Zhao, Jie Yang, Zijie Xin, Tianyi Wang, Fengyun Rao, Jing Lyu, and Xirong Li. OmniPro: A comprehensive benchmark for omni-proactive streaming video understanding. arXiv preprint arXiv:2605.18577, 2026. 3arxiv.org/abs/2605.18577
  34. [34]Yikai Zheng, Xin Ding, Yifan Yang, Shiqi Jiang, Hao Wu, Qianxi Zhang, Weijun Wang, Ting Cao, and Yunxin Liu. Em-Garde: A propose-match framework for proactive streaming video understanding. arXiv preprint arXiv:2603.19054, 2026. 2, 3, 6, 7, 8arxiv.org/abs/2603.19054
  35. [35]Tong Zhu, Zhe Xu, Zefeng He, Yunkai Chai, Xuyang Hu, Dehong Zheng, Yun Luo, Yafu Li, Qi Zhang, and Yu Cheng. Intern-Decision: Efficient multimodal decision making with autoregressive masked language modeling. arXiv preprint arXiv:2610.22131, 2026. 2, 3, 6, 7, 8arxiv.org/abs/2610.22131
  36. [36]Xiaomeng Zhu, Fengming Zhu, Weijie Zhou, Ye Tian, Zhenlin Hu, Yufei Huang, Yuchun Guo, Xinyu Wu, Zhengyou Zhang, Fangzhen Lin, and Xuantang Xiong. ProAct: A benchmark and multimodal framework for structure-aware proactive response. arXiv preprint arXiv:2602.03430, 2026. 2, 3arxiv.org/abs/2602.03430

Paper details

vlnc:W_kcmnfhve
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

DeepSeek-V4.1-Flash, GLM-5.3 (bare), Gemini 3.8 Flash (bare), Qwen3.8-Flash-Next-FP8, Codex

View statement
Looking for related papers…
Contents