betaFeedback

WorksComputer Science

BLIP-Live: Balancing Timeliness & Firing Precision in Proactive Streaming Video Agents

Abstract

Streaming video understanding is becoming increasingly important for real-world interactive systems. Such systems must be proactive for natural and useful human interaction: they should respond as soon as sufficient evidence becomes available while avoiding premature, redundant, or incorrect responses. We present BLIP-Live, a framework unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence. First, we construct BLIP-Live-780K dataset containing human annotations of proactive response content and timing, which defines this desired behavior. We then propose PAUC-F, a metric to effectively measure our desired proactive behavior. PAUC-F combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing. With our desired behavior defined and measurable, we finally investigate model-agnostic inference harnesses that draw out proactive streaming capabilities of offline vision language models (VLMs), together with training pipelines that learn to further improve their proactive responsiveness. Across both settings, we find that combinations of sliding visual windows, online interaction memory, and structured response outputs consistently yield strongest proactive performance. Our experiments result in two models, with our BLIP-Live-Qwen model achieving state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models. Our code and data will be released publicly.

bliplive.salesforceresearch.ai

Introduction

Proactive streaming video agents must learn not only what to say, but also when to say it, and when to remain silent (Wang et al., 2026; 2025b). This differs from offline video question-answering, where queries are immediately followed by responses. Streaming agents observe videos incrementally (causal), and at each frame decide whether to respond or remain silent for a query provided at an earlier time (Chen et al., 2024a; Yu et al., 2025). This causal firing decision presents a key challenge: a correct response is unhelpful if it arrives too late or is repeated too often (Wang et al., 2026). Effective proactivity requires balancing response timeliness with firing precision.

In order to achieve this desired behavior in streaming video agents, we propose aligning the end-to-end pipeline of data, metrics, harness, and training under a shared formulation. While existing works separately address subsets of these, we present BLIP-Live, Bootstrapping Live Interactive Perception, a single framework aligning data, metrics, harness, and training around a shared formulation of proactive responding as effective causal response decisions (illustrated in Figure 1).

Figure 1. Overview of proposed BLIP-Live framework. We unify data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence in the proactive streaming task.

We first construct BLIP-Live-780K, a large-scale dataset of proactive responses built through automated proactive question-answer generation followed by comprehensive human verification. In contrast to existing proactive streaming datasets (Table 1), BLIP-Live-780K provides human verified response content and intervals, providing explicit human supervision on when to remain silent. Next, we propose PAUC-F, an evaluation metric that captures both timeliness and firing precision in proactive streaming model responses. Prior metrics such as PAUC (Wang et al., 2025b) and ESTP-F1 (Yu et al., 2025) separately measure individual aspects of the proactive streaming task, leaving room for models to game these metrics with undesirable traits (see Figure 2 and Table 4). Our proposed PAUC-F, retains PAUC’s quality-weighted timeliness and balances it with response precision.

DatasetVideosQuestionsResponsesHuman-verified ContentHuman-verified Intervals
MMDuet2 (Wang et al., 2026)52,771105,800350,800NoNo
VideoChat3-OL (Li et al., 2026)191,000617,183617,183NoNo
ESTP-IT (Yu et al., 2025)1,16680,00080,000NoNo
PARSE-Ego4D [1]5,51518,36018,360YesNo
BLIP-Live-780K (Ours)177,789177,789784,944YesYes

Table 1. Proposed Streaming Proactive Dataset: We construct a training dataset for proactive streaming task with human-annotated content and intervals.

Figure 2. PAUC-F vs Prior Metrics: We simulate excessive firing behaviour (undesired) across WEB & EGO splits of ProactiveVideoQA (Wang et al., 2025b) and Explicit Proactive subset of ESTP-Bench (Yu et al., 2025) benchmarks, reporting our proposed PAUC-F metric alongside PAUC & ESTP-F1 metrics. Along the x-axes, we repeat ground-truth responses (Oracle) 5 additional times within golden windows (Oracle + 5x) and every 0.5s throughout the video (Spam 0.5). PAUC-F correctly identified these undesired response patterns while PAUC and ESTP-F1 incorrectly rewards all or some of these undesired test cases.

MetricWEB v1WEB v2EGO v1EGO v2
PAUC (ω=0.5\omega= 0.5) ↑64.8257.3066.5737.57
PAUC-F (ω=0.5\omega= 0.5) ↑31.9243.393.5835.94
Firing Precision ↑21.1734.911.8434.45
Fires : Gold ↓4.50×2.29×48.18×0.81×
Repeat Responses ↓14.302.20185.901.32
Human Preference ↑6%94%3%97%

Table 4. PAUC-F vs PAUC for model selection. We build two harness variants optimizing PAUC (v1) vs PAUC-F (v2). PAUC-F optimized v2 shows better firing precision, closer firing counts as ground-truth (Fires:Gold), less consecutive repeated responses, and higher human preference.

With desired behavior defined and measurable, next we investigate harness designs and training pipelines that improve proactive behavior in VLMs. Existing works explore proactivity with learned activation models (Wang et al., 2025a), hand-crafted harness designs (Yao et al., 2026), learned response decision tokens (Chen et al., 2024a; Li et al., 2026), and response policy optimization (Wang et al., 2025c; 2026). We introduce a model-agnostic harness tuning strategy to elicit proactivity in VLMs. Our selected harness combines a sliding visual window, interaction memory, and structured decision states. The same interface then supports efficient teacher-forced training with our datasets, further improving proactive responsiveness in VLMs, and resulting in two models, BLIP-Live-Gemma and BLIP-Live-Qwen. Our BLIP-Live-Qwen model achieves state-of-the-art performance across four proactive streaming benchmarks, outperforming multiple closed source models as well.

We summarize our key contributions as follows:

  1. BLIP-Live-780K Dataset: A high-quality proactive streaming dataset for training built through automated generation of proactive question–response pairs, followed by comprehensive human review and filtering to ensure accurate response content and timing annotations.

  1. PAUC-F Metric: Evaluation metric designed to measure proactive understanding in VLMs that captures both response timeliness and firing precision.

  1. Inference Harness: Model-agnostic harness tuning and calibration that adapts offline VLMs for proactive responding under streaming style inference.

  1. Training Framework: Efficient training pipeline resulting in BLIP-Live-Gemma and BLIP-Live-Qwen models that yield relative improvements up to 44% over prior baselines across benchmarks.

Related Work

Proactive Video Understanding. Proactive responsiveness is tackled under two key directions: inference harness and training pipeline. Harness design involves response gates, event detection, and long-context memory management (Wang et al., 2025a; Qian et al., 2025; Ding et al., 2025; Azad et al., 2026; Zhang et al., 2026a; Tian et al., 2026; Yao et al., 2025; Zheng et al., 2026; Yao et al., 2026), but its role is mainly streaming setting adaptation, overlooking direct optimization for proactivity. Proactive behavior is handled mostly in model through response decision states (Chen et al., 2024a; Xia et al., 2026; Li et al., 2026), implicit signals in interleaved video-text data (Wang et al., 2025c; Yang et al., 2025), and reinforcement learning (Wang et al., 2026). In contrast, our proposed harness is model agnostic and tuned specifically for proactivity, while the same interface extends to supervision when training data is available and transfers across model families rather than committing to fixed architectures.

Proactive Streaming Datasets. Prior works convert offline annotations from existing datasets to streaming style, often with LLM assistance. Chen et al. (2024a) reformats COIN and Ego4D narrations directly into streaming dialogues, while Wang et al. (2025c;a) use dense captioning task data to determine response timing using annotated temporal segments. Wang et al. (2026); Yao et al. (2025) generate dialogue from scene captions with text-only LLMs. Xia et al. (2026); Li et al. (2026) derive labels from a VLM at scale. However, existing datasets suffer from poor response timing (Wang et al., 2026), affecting quality of proactive responsiveness. In contrast, BLIP-Live-780K ensures accurate response content and timing through comprehensive human verification (see Table 1).

Evaluation Metrics for Proactive Understanding. Early benchmarks evaluate proactivity through targeted tasks. OVO-Bench (Li et al., 2025) introduces CRR (Clue Reveal Responding) task where the same query requires different responses at different timesteps. StreamingBench proposes Proactive Output (PO) task that evaluates the first response occurring within ±2 seconds of an annotated timestamp (Lin et al., 2024). In contrast, later ESTP-F1 (Yu et al., 2025) and PAUC (Wang et al., 2025b) metrics evaluate continuous interaction under true streaming settings, but both overlook firing-precision to certain extents (Figure 2), motivating our proposed PAUC-F metric.

Automated Harness Design. Evolving inference harnesses is explored for general agents and offline video understanding (Zhang et al., 2026c; Xu & Chen, 2026; Zhang et al., 2026b), but ours is a first to demonstrate successful integration to proactive or streaming video tasks.

The BLIP-Live-780K Dataset

We construct a dataset for learning proactive response behavior in streaming video tasks, containing 784,944 proactive responses across 177,789 videos. Each video includes a user query followed by a sequence of temporally grounded natural-language responses. Our dataset uses the same source videos as LLaVA-Video-178K (Zhang et al., 2024). We employ a three-stage annotation pipeline:

Stage 1: Automated Generation. Creates candidate proactive query and response sequences.

Stage 2: Human Verification and Correction. Human annotators inspect generated samples and correct errors related to both response timing and content.

Stage 3: Expert Quality Review. Independent PhD-level reviewers evaluate corrected annotations in batches. Batches below acceptance threshold are returned to Stage 2 for further revision.

Note that each query can require a single or multiple responses across the video, with each response corresponding to a single frame or frame interval. Our annotations capture both response and silent timings to encourage firing precision. Table 1 shows how compared to earlier work, our dataset is larger in scale and provides human-verified annotations for both response content and intervals.

Stage 1: Automated Proactive Query-Response Generation. We generate proactive interactions for a given video using a VLM in two successive stages. First, for query generation, we provide the VLM with a single video and obtain a single video-specific user query relevant to measuring proactive responsiveness. Next, for response generation, we provide the VLM both the video and query to obtain an ordered sequence of response or silence states. This is generated at 1 FPS, i.e. one state per second. Response states also include the response content. The generator receives the full video but is prompted to use only evidence available up to each response timestamp; humans then verify and correct timing and content.

Stage 2: Human Verification and Correction. Human annotators first filter videos based on query validity, and then evaluate each response or silent state. Incorrect states are fixed, including correction of response content when necessary. Query validity is defined as requiring only visually verifiable conditions and unambiguous answers. Response and silent states are evaluated based on alignment with evidence present in video until the relevant timestep, and corrected accordingly. Note that response states capture not only instantaneous events but also continuous events which involve consecutive response states over multiple timesteps. Of the responses generated, human review modified 11% and removed 40%, while adding 28% new responses. We provide full editing and filtering statistics in Table 2 and Table 3.

RecordsGeneratedFinalNet reductionReduction (%)
Queries186,751177,7898,9624.80
Responses893,562784,944108,61812.16

Table 2. Dataset Filtering. Generated and human edited (final) record counts; net reductions include deduplication.

OutcomeCount% of generated
Unchanged427,15047.80
Modified104,46811.69
Deleted361,94440.51
Added253,32628.35

Table 3. Annotation outcomes. Percentages relative to the 893,562 response records from automated generation stage.

Stage 3: Quality Control. All annotators are recruited and trained by a professional video data annotation team. We provide detailed instructions and visual examples describing our annotation requirement. The annotated samples undergo quality control at batch level, using PhD-level expert reviewers (who designed the annotation process). After each batch, annotation proceeds to next only if quality exceeds our required threshold. Failing batches are returned for further review and correction. While initial batches required one repeat cycle, later batches passed required quality directly, ascertaining that annotation process was well aligned with our specified requirement.

PAUC-F: Proactive Streaming Evaluation Metric

We propose PAUC-F that jointly captures answer quality, timeliness, and firing precision. In Table 4, humans prefer a harness optimized with this PAUC-F metric over 90% of the time.

Limitations of Prior Metrics

PAUC (Wang et al., 2025b) and ESTP-F1 (Yu et al., 2025) are two key metrics in recent literature relevant to proactivity in streaming video agents. PAUC measures quality weighted timeliness in responses. However, PAUC only measures recall within the gold-window. This results in responses outside of the gold-window window being discarded, and repeated responses within the window not affecting the score (see Figure 2). Similar observations are noted in MMDuet2 (Wang et al., 2026), where removing the repetition reward increases PAUC scores although duplicate replies increase by 4x. ESTP-F1 addresses out-of-window firing with the addition of a precision term, but ignores redundant in-window firing and requires task-dependent timing term for recall calculation. These two key metrics provide complementary but incomplete views, which we demonstrate in Figure 2.

PAUC-F Definition

We propose PAUC-F, which maintains PAUC as its recall and introduces a precision term:

R=PAUC,P=NvalueNfires,PAUC-F=2PRP+R(1)R = \mathrm{PAUC}, \qquad P = \frac{N_{\mathrm{value}}}{N_{\mathrm{fires}}}, \qquad\mathrm{PAUC\text{-}F} = \frac{2PR}{P + R} \tag*{(1)}

Here NfiresN_{\mathrm{fires}} counts all emitted responses, including out-of-window fires. Each gold turn contributes at most one value-adding response to NvalueN_{\mathrm{value}}: its earliest in-window fire with judge score 1 or 2 (maximum S=2S = 2). Additional fires increase only the denominator. PAUC-F inherits PAUC’s quality–timeliness weight ω\omega and penalizes unnecessary firing through precision. Aggregation, zero-fire handling, and PAUC’s [0,1][0, 1] range (with no-reply score 0.25) are detailed in Appendix § 11.

Metric Properties

We investigate properties of our proposed PAUC-F metric in Figure 2 and Table 4. First, we construct a toy experiment simulating undesired excessive firing behavior. We use ground-truth response content and intervals to mimic models exhibiting within golden window repetitions and excessive out-of-window firing. As observed in Figure 2, both PAUC and ESTP-F1 fail to capture at least one of these undesired repetitive response behaviors, while PAUC-F correctly penalizes both. This excessive firing simulation was motivated by findings in prior work where models optimized for PAUC alone exhibit such repetitive behavior in their responses (Wang et al., 2026).

We also observe similar findings in our experiments, which we summarize in Table 4. We optimize two inference harness variants (details in § 5.1) to independently maximize PAUC (v1) and PAUC-F (v2) on a held-out validation set of our BLIP-Live-780K dataset. PAUC-F optimized v2 leads to clearly more desired behavior on downstream tasks compared to the PAUC optimized v1 agent. We also conduct a human preference study involving 10 participants rating 20 samples from WEB and EGO splits of (Wang et al., 2025b). Each sample contains response predictions from v1 and v2 agents that are rated for better proactive response quality. Human raters show clear preference for the PAUC-F optimized v2 agent (more details in Appendix § 11).

We further highlight how PAUC-F better captures the range of outcomes worse than silence (incorrect responses that reduce precision). Table 5 illustrates this, where additional out-of-window fires leave PAUC unchanged while reducing PAUC-F.

PolicyPAUCPrecisionPAUC-F
One perfect response; otherwise silent0.25751.00000.4095
Same response + 1 out-of-window fire0.25750.50000.3399
Same response + 5 out-of-window fires0.25750.16670.2024
Same response + 10 out-of-window fires0.25750.09090.1344
Same response + 100 out-of-window fires0.25750.00990.0191

Table 5. Sparse-response example over 100 gold turns. A single perfect response at a window’s start raises mean PAUC to (1+99×0.25)/100=0.2575(1 + 99 \times0.25)/100 = 0.2575. Additional out-of-window fires leave PAUC unchanged but reduce PAUC-F. Scores are on [0, 1].

Method

In this section, we introduce our inference harness and training pipeline for streaming video tasks.

Streaming Interaction Protocol. We first establish our definition of streaming inference, which is aligned with prior work (Chen et al., 2024a; Wang et al., 2026; Li et al., 2026). At time tt, a model receives a window of past video frames and a textual state capturing prior queries and responses. In turn, the model must produce either silence or a response at the current timestamp. The model cannot access future frames, revise an earlier response, or assign a new answer to an older timestamp. This protocol aligns with the same prior work and allows models to understand streaming videos extending beyond model visual context and to proactively respond at any time after a user query.

Streaming Video Model. We also define a streaming video model as one trained specifically to consume inputs and generate outputs in this format, as opposed to offline VLMs trained for offline video question-answering that can be adapted to this task through suitable harness design.

Inference Harness

Prior works often measure proactive behaviour of offline VLMs using non-streaming inference settings (Wang et al., 2025b; Chen et al., 2024a; Yu et al., 2025), making direct comparison to streaming models difficult. For example, Wang et al. (2025b) evaluates offline VLMs on 5s frame chunks with responses of frame tt conditioned on frames up to t+5t + 5, deviating from the streaming task setting. More recent works integrate offline VLMs with an inference harness (Li et al., 2026; Wang et al., 2025a; Yao et al., 2026) for evaluation under true streaming settings. While these works hand-craft harness designs, we focus on tuning a harness, motivated by recent work on automated harness evolution (Zhang et al., 2026c), to elicit stronger proactive responsiveness from an offline model.

Design Overview. As illustrated in Figure 3, our harness contains four key components: system prompt defining task and output schema, past frame window, memory holding prior interactions, and structured output parsing. The underlying model is called at the desired frame rate, with dynamic inputs at each timestep conditioned on the past model responses.

Figure 3. Inference Harness. At each streaming timestep tt, the harness constructs an input from a system prompt (task and output schema), a rolling past-frame window, and memory of user queries and prior responses. A frozen offline VLM is queried at the target frame rate; structured output is parsed into either silence or a response at tt, and accepted responses are updated into memory.

Harness Tuning. We next optimize the design of its four key components. Instead of hand-crafting these design choices, we utilize agent-assisted search to modify the harness, keeping underlying VLM model weights frozen (i.e. no model training). The agent first proposes hypotheses for modifying one of these four components, implements relevant code changes, and evaluates on BLIP-Live-780K validation set using our PAUC-F metric. These results guide new hypothesis creation over several rounds and the final design selection. Each round contains ten candidate variants and we promote variants that improve our objective (PAUC-F on validation set). Combination of complementary promoted changes is treated as a novel hypothesis for next round. We run 10 rounds of propose-evaluate-promote cycles with a total of 100 candidate evaluations each for two different models: Gemma4-26B-A4B (Gemma Team, 2026) and Qwen3.5-35B-A3B (Qwen Team, 2026). Both variants converge to a common design with minimal variations in system prompt and output schema.

Harness Calibration. Given our fixed harness, we perform further model-specific calibration of system prompt and output schema with a more modest compute budget, using the same validation set and PAUC-F metric. We evaluate five candidates on a fixed 20 sample subset, with each candidate modifying only the system prompt and output schema (joint changes allowed). This corresponds to 5 rounds (one candidate per round) and a total budget of evaluating 100 video samples. We perform this limited calibration on all other offline models prior to evaluation.

Interactivity: The user query is included in our memory which can be updated midway during a video. This design choice allows user interaction with the framework mid-video: a human user can pose new queries as the framework continues processing an existing video, further aligning with true streaming setting. We investigate and evaluate this behavior further in Table 9.

Table 9: StreamingBench Proactive Split (PO) time accuracy (follow official leaderboard) & our interactive protocol on same split (PO-I). Best and second-best marked in each column.

ModelPOPO-I
Gemini 1.5 Pro45.10–
GPT-4o39.22–
Claude 3.5 Sonnet35.29–
VideoLLM-Online1.96–
VITA-1.525.49–
Dispider25.34–
InternVL220.45–
MiniCPM-o 2.629.4130.27
MMDuet234.695.95
VideoChat39.8018.38
Gemma4-A4B36.8038.9
Qwen3.5-A3B25.2015.1
BLIP-Live-Gemma44.0048.6
BLIP-Live-Qwen47.6054.6

Table 9.

See Appendix § 12 for further details of our training-free harness design and its prompts.

Streaming Model Training

We next investigate how supervised finetuning (SFT) can further improve proactivity. We use streaming data derived from BLIP-Live-780K and construct a training pipeline that retains the efficiency of standard teacher-forced SFT. An overview of our training formulation is illustrated in Figure 4.

Figure 4. Streaming Model Training. Each training sequence concatenates a system prompt, memory of user queries / past responses, and an interleaved frame-target stream. Prompt, memory, and visual tokens receive no loss; the VLM is supervised on state targets (silence, response, and hold) and the accompanying response text. One video is split into multiple chunks so that long streams remain within the training context.

Streaming Episodes and Targets

Causal episode construction. We construct dense sequences of frame-level decisions from our annotated videos. First we build full video timelines followed by division into bounded chunks. A video timeline is a sequence of timestamped frame inputs and model turns, where model turns contain one of three state tokens, </Silence>, </Response>, or </Hold>. The response token is followed by the annotated natural-language response. These timelines are split into chunks that contain the memory, absolute timestamps, and its corresponding source-frame range. Within chunks, all response turns are fully supervised, while long runs of non-response states are subsampled to avoid silence dominating the token budget. A chunk corresponds to L=32L = 32 frames, which is the past frame window that is provided to our model.

State targets. Human annotations define three states: irrelevant frames are Silence; the first frame with sufficient evidence is Response; subsequent frames of the same ongoing event are Hold, suppressing repetition. Only 18% of responses are followed by Hold. Post-response silence-only chunks teach sustained silence. Finetuned models predict all three states at inference; Hold and Silence emit no response and leave memory unchanged. The frozen-model harness uses binary silence/response outputs (Table A.11). See Appendix § 13 for further details on state targets and our training data distribution.

Streaming Supervision

We perform teacher-forcing SFT by using individual chunks as samples. Each chunk contains a system prompt, memory, and an interleaved sequence of frame-target pairs. The targets are the states </Silence>, </Response>, or </Hold>.

Memory Module. Each training chunk includes the user query and last m=1m = 1 annotated response; inference uses the model’s last generated response. Memory retains responses outside the visual window to discourage repetition, and new queries can update it mid-video. Both finetuned models use L=32L = 32 frames and m=1m = 1 during training and inference. The frozen-model harness uses three frames and m=1m = 1; both settings run at 1 Hz.

Loss Masking. While our training employs a standard next token prediction loss, within each chunk we mask the cross-entropy loss term at token level. This is motivated by the large portion of silence timestamps inherent to the task and similar strategies in prior work ((Li et al., 2026)). Our token-level loss masking retains all Response targets while subsampling repetitive Silence and Hold continuations. Chunks containing only Silent responses also undergo masking. The train-time sequence also appends the natural language response after each response token, where the language tokens undergo no loss masking.

Overall, our inference harness and SFT pipeline share a single streaming interface, and demonstrate how offline VLMs can be adapted to streaming inference eliciting their proactive responsiveness.

Experiments

Experimental Setup. We train on the BLIP-Live-780K train split and select harnesses and checkpoints by validation PAUC-F; the four test benchmarks are held out. Overlap filtering is described in

Appendix Appendix § 10. We report PAUC-F (GPT-4.1 judge) on ProactiveVideoQA and ESTP-Bench Explicit Proactive. Configurations are in Appendices § 12 and § 13. Both training and inference use 8×8 \times H200 GPU nodes.

Main Results

ProactiveVideoQA. We present PAUC-F evaluations across the four splits from Wang et al. (2025b) in Table 6. All offline proprietary and open-weight models are evaluated using our harness with per-model harness calibration. The Gemini-Live model is evaluated with its default harness as well as an integration of our harness (ours-v) without calibration. For streaming video model baselines, we evaluate the official model checkpoints with their default streaming inference settings.

ModelHarness[WEB] 0.0[WEB] 0.5[WEB] 1.0[EGO] 0.0[EGO] 0.5[EGO] 1.0[TV] 0.0[TV] 0.5[TV] 1.0[VAD] 0.0[VAD] 0.5[VAD] 1.0[Mean] 0.0[Mean] 0.5[Mean] 1.0
Proprietary Models
GPT-4.1ours57.9260.1962.3133.1534.3335.3631.3032.5633.5828.0028.8029.4937.5938.9740.19
GPT-4.1-miniours54.3657.1959.8121.6322.0722.4424.0724.8225.4219.9220.3620.7430.0031.1132.10
GPT-5.6-Solours57.6260.3562.8731.0132.0332.9238.7540.9442.7433.5934.8135.9040.2442.0343.61
Gemini-2.5-Proours55.1357.6760.0132.7033.9735.0636.1437.9839.4823.8424.5325.1136.9538.5439.92
Gemini-2.5-Flashours40.0241.3142.459.279.339.3817.9218.3318.654.314.334.3517.8818.3318.71
Gemini-3.1-Proours50.2152.7655.0019.8320.1320.3837.6239.4941.020.660.660.6627.0828.2629.27
Gemini-3.5-Flash-Liteours48.7851.2053.3311.1411.2311.3120.4420.9621.380.440.440.4420.2020.9621.62
Gemini-3.8-Flashours51.2253.8456.1519.1219.4419.7130.5631.7832.760.330.330.3325.3126.3527.24
Gemini-Live-2.5default27.5627.5627.560.000.000.0015.4815.4815.480.000.000.0010.7610.7610.76
Gemini-Live-2.5ours-v42.0446.5950.299.8910.0010.096.856.947.037.607.657.6916.6017.8018.78
Open-Weight Models (No Streaming Video Training)
Molmo2 8Bours36.1738.2240.0710.7010.8110.909.799.889.956.616.646.6615.8216.3916.90
MiniCPM-V 4.5ours37.1639.4341.4011.8812.0412.184.384.404.414.384.394.4014.4515.0715.60
Inkling-Smallours48.3250.4252.3924.6225.3926.0615.1115.3115.499.629.669.6924.4225.2025.91
Gemma4-A4Bours41.9243.3944.7134.6335.9437.1526.1726.6027.0127.8228.5029.1332.6433.6134.50
Qwen3.5-A3Bours51.8554.2456.5437.6239.2940.8233.4234.4735.4641.9043.7545.5141.2042.9444.58
Streaming Video Models (Trained for Streaming Responses)
VideoLLM-Onlinedefault32.1932.2032.206.646.646.642.452.452.460.000.000.0010.3210.3210.33
EyeWOdefault38.9740.4241.814.764.774.783.243.233.230.000.000.0011.7412.1112.46
MMDuetdefault48.4350.7652.927.487.527.552.702.712.722.412.422.4215.2615.8516.40
MMDuet2default50.6956.5161.4631.6432.2832.8930.3432.2933.9114.7014.9715.2231.8434.0135.87
VideoChat3default33.6834.7035.6828.8629.7130.5136.9039.5341.8922.8423.2623.6630.5731.8032.94
BLIP-Live-Gemma (ours)ours52.1956.4660.3841.8143.8045.6131.6533.2734.7030.8531.9833.0139.1341.3843.43
BLIP-Live-Qwen (ours)ours53.1357.4861.4143.7746.0548.1340.7143.1145.2546.5649.6752.5546.0449.0851.84

Table 6. Results on ProactiveVideoQA Benchmark. We report PAUC-F with varying ω\omega using GPT-4.1 judge. Harness is ours for offline models and default from streaming native models. Our harness is modified to integrate with Gemini-Live (ours-v) due to closed API nature. Best and second-best marked in each column.

Our two trained models, BLIP-Live-Gemma and BLIP-Live-Qwen, use our harness for inference. Our trained BLIP-Live-Qwen model achieves state-of-the-art performance overall, demonstrating the usefulness of our BLIP-Live framework. BLIP-Live-Qwen (49.08) also outperforms prior best MMDuet2 (34.01) by 44%. We also report the raw PAUC numbers (official metric of benchmark) in Appendix § 8 for reference.

ESTP-Bench. We next report PAUC-F on the Explicit Proactive subset of tasks from Yu et al. (2025), which measures proactive responsiveness. Our results in Table 7 indicate state-of-the-art performance of proposed BLIP-Live-Gemma and BLIP-Live-Qwen. All baselines use the same official model checkpoints with our harness, with streaming native models using their default inference setting.

ModelHarnessORAPTRUOLOSCEOLEOSCARAll
Proprietary Models
GPT-4.1ours20.1226.8823.5723.6017.8919.7119.5124.0121.91
GPT-4.1-miniours21.8027.4127.0431.3919.4826.5118.1718.1724.20
Gemini-2.5-Proours25.3627.1434.5330.1421.0119.3618.9422.8924.92
Gemini-2.5-Flashours9.4415.6719.029.409.228.047.6212.9811.42
Open-Weight Models (No Streaming Video Training)
Molmo2 8Bours8.8411.1813.014.933.985.573.258.047.35
MiniCPM-V 4.5ours6.759.353.208.104.116.323.986.386.02
Inkling-Smallours10.2814.1110.4513.279.0810.918.1212.2511.06
Gemma4-A4Bours11.0516.9015.5914.077.5211.166.4110.3311.63
Qwen3.5-A3Bours23.8827.1723.0729.7829.2423.1029.8723.4726.20
Streaming Video Models (Trained for Streaming Responses)
VideoLLM-Onlinedefault0.730.080.050.331.190.450.452.530.73
EyeWOdefault10.867.083.4310.727.1610.204.5611.238.16
MMDuetdefault1.912.960.723.243.833.062.436.223.05
MMDuet2default15.4720.8914.1514.7411.3211.5010.4310.1713.58
VideoChat3default12.6316.7113.0719.8312.0817.1912.7611.6514.49
BLIP-Live-Gemma (ours)ours20.2325.9730.0629.8320.8924.4117.7321.2823.80
BLIP-Live-Qwen (ours)ours24.6231.7730.4729.0830.8227.8431.6424.2428.81

Table 7. Results on ESTP-Bench. We report PAUC-F (ω=0.5\omega= 0.5, GPT-4.1 judge) on Explicit Proactive subset of ESTP-Bench. Our harness used for offline models and default harness for streaming models. The task categories are Object Recognition (OR), Attribute Perception (AP), Text-Rich Understanding (TRU), Object Localization (OL), Object State Change Recognition (OSC), Ego Object Localization (EOL), Ego Object State Change Recognition (EOSC), and Action Recognition (AR). Best and second-best marked in each column.

Official ESTP-Bench, StreamingBench (PO), and OVOBench (CRR). We next report results on existing benchmarks using their default metrics, allowing direct comparison to baseline results originally reported in Yu et al. (2025); Lin et al. (2024); Li et al. (2025). We report results for Gemma4 and Qwen3.5 offline models converted to online streaming setting with our harness, as well as BLIP-Live-Gemma and BLIP-Live-Qwen which are our two trained models.

In Table 8, we report ESTP-F1 metric on full ESTP-Bench benchmark. Baseline results are identical to Yu et al. (2025). Our BLIP-Live-Qwen model achieves best overall performance on Explicit (Exp.) split. Implicit (Imp.) and Contextual (Ctx.) splits contain longer gold intervals and because ESTP-F1 overlooks in-window repetition, models with repetitive behavior can score higher on these (more in Table A.7).

Table 8: ESTP-F1 on ESTP-Bench with baseline results identical to Yu et al. (2025). Our methods (both harness and trained) are highlighted in gray. Best and second-best marked in each column.

ModelExp.Imp.Ctx.All
Offline MLLMs
LLaVA-OV13.631.88.218.0
Qwen2-VL15.439.310.421.3
MiniCPM-V15.236.526.622.9
LLaVA-NeXT16.534.613.821.3
InternVL-V25.66.75.75.9
Online MLLMs
LIVE0.87.09.311.89.1
LIVE0.99.525.618.915.5
MMDuet9.134.220.317.8
EyeWO23.652.543.634.7
Gemma4-A4B18.231.227.123.2
Qwen3.5-A3B16.622.614.118.0
BLIP-Live-Gemma31.630.440.532.5
BLIP-Live-Qwen35.232.140.435.1

Table 8.

In Table 9, we report time accuracy for the Proactive (PO) split of StreamingBench with baseline results obtained from official leaderboard. Our BLIP-Live-Qwen model similarly achieves overall best performance. We additionally introduce an interactive split (PO-I) where new queries are introduced mid-video in a true streaming fashion (more details in Appendix § 9). Baselines are evaluated using official checkpoints under identical settings as our models.

In Table 10, we report accuracy for the CRR (Clues Reveal Responding) task of OVOBench. In this task, a query is provided before the video contains sufficient information and models are expected to wait until evidence appears and then proactively respond. This is the key task in OVOBench that measures proactive responsiveness; hence we use it in our evaluations. In this task too, our BLIP-Live-Qwen model achieves overall best performance.

Table 10: OVO-Bench Forward CRR Task. Baseline results follow Li et al. (2025). Our methods highlighted gray. Best and second-best marked.

ModelCRR
Gemini 1.5 Pro61.67
GPT-4o59.40
LLaVA-Video-7B60.42
LLaVA-OV-7B58.75
Qwen2-VL-72B45.00
Qwen2-VL-7B48.75
Gemma4-A4B59.58
Qwen3.5-A3B56.67
BLIP-Live-Gemma60.83
BLIP-Live-Qwen62.50

Table 10.

Ablative Studies

Dataset. We establish the contribution of our human review and filtering during dataset construction in Table 11. Results also highlight how performance scales going from 10% to 100% of data in BLIP-Live-780K for model training.

VersionFrac.WEB ↑EGO ↑
Raw Gemini-generated100%54.842.4
Intermediate, pre-cleaning100%55.442.9
Final human-verified100%56.543.8
Final human-verified10%53.842.3

Table 11. Annotation & Data Scale Ablation. PAUC-F (ω=0.5\omega= 0.5) on EGO & WEB for BLIP-Live-Gemma. Each stage of our dataset construction (human review and filtering) improves performance. Results also scale when training with increasing data.

Training Data. Table 14(a) fixes the Qwen3.5-A3B backbone, our training recipe, and inference harness, varying only the training dataset. BLIP-Live-780K yields the best performance across all four splits. Finetuning on BLIP-Live-780K improves mean PAUC-F from 42.94 to 49.08, a 6.14-point gain over the same pretrained model with our harness.

Dataset / MethodWEBEGOTVVADMean
(a) Training-data ablation: fixed backbone, recipe, and harness
Base model (no finetuning)54.2439.2934.4743.7542.94
MMDuet2 data27.3424.3916.497.4618.92
VideoChat3 data55.1539.7140.5345.7245.28
BLIP-Live-780K data57.4846.0543.1149.6749.08
(b) Training-method comparison: fixed backbone, original data and recipes
MMDuet232.7629.5722.2013.0324.39
VideoChat335.8630.3640.7524.6732.91
BLIP-Live-Qwen57.4846.0543.1149.6749.08

Table 14. Controlled training comparisons on ProactiveVideoQA. PAUC-F at ω=0.5\omega= 0.5 with a common Qwen3.5-A3B backbone. Panel (a) fixes the BLIP-Live training recipe and inference harness while varying training data; panel (b) compares each method’s own dataset and training recipe. Mean averages the four splits.

Training Method. Table 14(b) holds the backbone fixed while MMDuet2 and VideoChat3 retain their original datasets and training recipes. BLIP-Live-Qwen outperforms both, separating the combined contribution of our data and training pipeline from backbone differences. Full results at all three ω\omega values and target-conversion details are in Appendix § 8.

Inference Harness. In Table 12, we first ablate the key components of our proposed inference harness to verify their usefulness. Next we compare against an existing generic harness design from Li et al. (2026) to demonstrate our gains in proactive responsiveness.

MethodWEBEGO
Ours (Default)43.436.0
Ours w/o sliding window33.67.0
Ours w/o online memory36.814.1
Ours w/o structured output42.234.6
VideoChat3 Harness26.32.4

Table 12. Inference Harness Ablation. PAUC-F (ω=0.5\omega= 0.5) on WEB and EGO for Gemma4-A4B. Baseline from VideoChat3 for comparison purposes.

Training Pipeline. We ablate our training pipeline in Table 13. Memory and Hold discourage repetitive firing. On WEB, removing Hold increases fires from 978 to 2,840 and lowers firing precision from 75.1% to 25.8%, while PAUC remains approximately unchanged (45.26% vs. 45.39%). This supports improved firing selectivity as the mechanism; full WEB/EGO diagnostics are in Table 15. Loss masking is crucial to prevent models from collapsing to always remaining silent, as query-relevant events are sparse and the decision-token distribution is heavily skewed toward silence.

MemoryHold StateLoss MaskingWEB ↑EGO ↑
✓✓✓56.543.8
✗✓✓54.840.5
✓✗✓32.924.4
✓✓✗0.00.0

Table 13. Ablation on Training Pipeline. PAUC-F (ω=0.5\omega= 0.5) on WEB & EGO for BLIP-Live-Gemma. Memory and Hold-State help avoid repetitive firing. Loss-Masking is crucial to avoid collapsing to silence.

SplitHoldPAUC-FFiresPrecision (%)PAUC (%)Repeated (%)
WEBWith56.4697875.145.2615
WEBWithout32.902,84025.845.3965
EGOWith43.801,95542.245.5240
EGOWithout24.404,95016.745.2875

Table 15. Hold-state diagnostics for BLIP-Live-Gemma on ProactiveVideoQA (ω=0.5\omega= 0.5). Repeated fires are responses repeating previously reported content, expressed as a percentage of all fires. Removing Hold increases firing and repetition while PAUC remains approximately unchanged.

Conclusion

Proactive streaming video agents must decide between responding and remaining silent repeatedly for each timestep. We formulate this task as a series of causal response decisions at each timestep. Our BLIP-Live framework unifies data, metric, harness, and training around this formulation and demonstrates state-of-the-art performance across four proactive streaming benchmarks.

AI Use Statement

In this work, we used generative AI tools for following tasks: generate synthetic data sets, implement methods, clean and reformat dataset, propose or refine hypotheses, design or provide feedback on research methodology or experiments. To reiterate, as explained in the paper, generative AI tools were used for automated preliminary dataset creation (part of our contribution) and automated inference harness tuning (part of our contribution). We have not used generative AI tools for the following tasks: help develop theoretical models or conceptual frameworks, formulate mathematical claims, provide critical ingredients for proving mathematical claims, assist in the writing of proofs, assist with translation, support qualitative and thematic data analysis, interpret results. The rest of the required disclosure tasks are not applicable to this work. Additionally, we used generative AI tools for the following: formulate questions for surveys or interviews, create or modify scientific figures or images, create or edit software code, summarize or analyse existing literature, sourcing/searching for information, identify relevant literature, format references, propose a title. We have reviewed all AI-assisted work. Particularly, all AI generated code and AI assisted literature survey have been verified by authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Reproducibility Statement

Our data, code, and trained model checkpoints will be released publicly. For closed source models, all generation outputs will be released. For open models, we use publicly available checkpoints in all our evaluations.

Author Contributions

Kanchana Ranasinghe led the project, developing the inference harness, PAUC-F metric, and training pipelines, and running most of the training, inference, and evaluation experiments.

Honglu Zhou set the research and experimental directions, led dataset annotation, conducted initial training and metric-comparison experiments, and contributed substantially to the project.

Dominick Reilly helped run baselines, surveyed the literature, and gave feedback on the method and the manuscript.

Le Xue and Shusheng Yang built the Gemini question-generation pipeline used in dataset creation; Le Xue also contributed to discussions of the dataset design.

Juan Carlos Niebles, Silvio Savarese, and Ran Xu provided high-level direction and discussed many aspects of the project.

Acknowledgments

We thank Artemis Panagopoulou for reviewing this paper and providing extensive feedback. We also thank the broader team at Salesforce AI Research for their support.

Additional Experiments

We report additional results proactive streaming video benchmarks, ProactiveVideoQA and ESTP-Bench as well as more splits of OVO-Bench. We also provide additional ablations comparing our method to existing harness designs and experiments involving our harness tuning.

Baselines. We use official checkpoints of Molmo2 8B (Clark et al., 2026), MiniCPM-V 4.5 (Yu et al., 2026), Inkling-Small (Thinking Machines Lab, 2026), MiniCPM-o 2.6 (OpenBMB, 2025), VideoLLM-Online (Chen et al., 2024a), MMDuet (Wang et al., 2025c), MMDuet2 (Wang et al., 2026), EyeWO (Yu et al., 2025), and VideoChat3 (Li et al., 2026) for the baselines we report in our evaluations. We also borrow numbers directly from prior evaluations in published papers for LLaVA-OneVision (Li et al., 2024a), LLaVA-NeXT-Video (Li et al., 2024b), LLaVA-Video (Zhang et al., 2024), Qwen2-VL (Wang et al., 2024), MiniCPM-V (Yao et al., 2024), InternVL2 (Chen et al., 2024b), CLIP (Radford et al., 2021), LaViLa (Zhao et al., 2023), EgoVLP (Lin et al., 2022), VITA-1.5 (Fu et al., 2025), and Dispider (Qian et al., 2025).

ProactiveVideoQA PAUC. In Table A.1, we report the original PAUC metric (Wang et al., 2025b) based evaluations for the same methods in Table 6. We highlight that the zero-point of PAUC is 25.0 (i.e. a model with no responses) and models scoring below 25 are worse than fully silent models (according to PAUC). Also note how models like Gemini-2.5-Flash (64.51) outperform the Gemini-2.5-Pro model (55.07) at ω=0.5\omega= 0.5 in the Mean PAUC column, misaligning with our priors about these models’ strengths. We reiterate our observations (see Figure 2) how models can game this metric with excessive firing, sacrificing precision for increased recall.

Nevertheless, it is interesting to note that our models (BLIP-Live-Gemma and BLIP-Live-Qwen) optimized for PAUC-F also achieve strong PAUC numbers while reducing excessive firing (see Table 4), with BLIP-Live-Qwen achieving state-of-the-art performance among non-proprietary models.

ESTP-Bench. We also report the full per-task breakdown of ESTP-Bench ESTP-F1 (official metric) scores in Table A.4. These numbers correspond to the results summary in Table 8 of main paper. Our BLIP-Live-Qwen showcases state-of-the-art overall performance on this benchmark as well.

We note how on Implicit and Contextual splits several models outperform both our models by large margins. We argue this is due to weaknesses of the ESTP-F1 evaluation metric. Across the three splits, source videos are of similar length, but Implicit and Contextual gold windows are much longer (Table A.7). Models that repeat fires get penalized much less with the longer golden windows under ESTP-F1 metric. We highlight how our models outperform prior best EyeWO model across all 3 splits when evaluating with the more balanced PAUC-F metric in Table A.5. Table A.6 shows why: EyeWO baseline emits an order of magnitude more fires per item than the ground-truth, especially on Implicit and Contextual splits, while BLIP-Live-Gemma and BLIP-Live-Qwen stay near the gold rate. Measuring repeats in this way to affirm prediction quality has been used in the prior work MMDuet2 (Wang et al., 2026) as well.

Extended Training Comparisons. Tables A.2 and A.3 extend the controlled comparisons in Table 14 to ω=0.0\omega= 0.0, 0.5, and 1.0. For the fixed-recipe data ablation, both external datasets are converted into our Silence, Response, and Hold targets: each original response is kept, the next two seconds are labeled Hold, and the source labels then resume. For VideoChat3, its standby state is additionally mapped to Silence. Mean averages the four splits.

Table A.1: PAUC on ProactiveVideoQA Benchmark. We report PAUC with varying ω\omega using GPT-4.1 judge, for the same runs, predictions, and judge configuration as Table 6. PAUC is the recall component of PAUC-F and does not penalize the number of responses emitted, so these values should be read together with Table 6. Models with excessive responsiveness (over-firing) can score higher on this metric than conservative models with balance precision and recall. Harness is ours for offline models and default for streaming native models. Our harness is modified to integrate with Gemini-Live (ours-v) due to closed API nature. Best and second-best are marked separately within proprietary models and within the remaining models.

ModelHarness[WEB] ω=0.0\omega=0.0[WEB] ω=0.5\omega=0.5[WEB] ω=1.0\omega=1.0[EGO] ω=0.0\omega=0.0[EGO] ω=0.5\omega=0.5[EGO] ω=1.0\omega=1.0[TV] ω=0.0\omega=0.0[TV] ω=0.5\omega=0.5[TV] ω=1.0\omega=1.0[VAD] ω=0.0\omega=0.0[VAD] ω=0.5\omega=0.5[VAD] ω=1.0\omega=1.0[Mean] ω=0.0\omega=0.0[Mean] ω=0.5\omega=0.5[Mean] ω=1.0\omega=1.0
Proprietary Models
GPT-4.1ours56.0160.4264.8348.6254.0659.5156.4465.5774.7050.0055.4960.9852.7758.8965.01
GPT-4.1-miniours47.2251.6756.1254.4460.4966.5455.3164.2573.2045.2750.2055.1450.5656.6562.75
GPT-5.6-Solours52.5457.2661.9749.4455.0460.6352.1560.9069.6543.2847.5851.8749.3555.1961.03
Gemini-2.5-Proours51.9356.6361.3346.3251.8057.2954.0563.2072.3543.7948.6553.5049.0255.0761.12
Gemini-2.5-Flashours60.6267.0073.3862.4068.7875.1657.6167.2576.9048.3155.0061.6857.2364.5171.78
Gemini-3.1-Proours56.2563.0769.9061.4467.7274.0056.8266.3175.8025.5226.1126.7050.0055.8061.60
Gemini-3.5-Flash-Liteours55.8362.5969.3564.6771.5778.4853.2961.1569.0023.4223.2823.1349.3054.6559.99
Gemini-3.8-Flashours56.8063.6670.5260.3767.3274.2758.2968.3278.3525.1325.3525.5750.1556.1662.18
Gemini-Live-2.5default24.1024.1024.1025.0025.0025.0020.3120.3120.3125.0025.0025.0023.6023.6023.60
Gemini-Live-2.5ours-v41.0650.7460.4246.6652.1557.6416.4217.5818.7547.9152.0856.2538.0143.1448.27
Open-Weight Models (No Streaming Video Training)
Molmo2 8Bours34.8038.8042.8135.3937.8140.2428.7030.2331.7526.9927.8628.7431.4733.6835.88
MiniCPM-V 4.5ours40.4346.2252.0139.0742.8746.6723.1123.8324.5519.4019.7520.0930.5033.1735.83
Inkling-Smallours40.8743.9647.0443.3148.5453.7632.8434.7736.7030.5931.3032.0136.9039.6442.38
Gemma4-A4Bours52.4557.3062.1434.8137.5740.3228.7029.7530.8034.2636.4138.5537.5640.2642.95
Qwen3.5-A3Bours37.0439.5342.0238.4342.1045.7633.3235.4837.6533.2835.6838.0835.5238.2040.88
Streaming Video Models (Trained for Streaming Video)
VideoLLM-Onlinedefault26.2026.2126.2225.0825.1025.1316.9217.4618.0025.0025.0025.0023.3023.4423.59
EyeWOdefault30.2532.0333.8139.8441.3242.7920.9120.4820.0522.4921.4120.3328.3728.8129.25
MMDuetdefault43.1346.9750.8141.5143.6245.7322.1323.1424.1524.1824.3624.5332.7434.5236.31
MMDuet2default43.2652.4861.7127.9828.9930.0037.5044.1050.7028.4130.5632.7134.2939.0343.78
VideoChat3default28.7430.2631.7829.0030.7732.5432.9837.4441.9027.0328.2329.4429.4431.6833.91
BLIP-Live-Gemma (ours)ours40.0045.2650.5141.4245.5249.6235.3639.6844.0134.1637.0639.9537.7441.8846.02
BLIP-Live-Qwen (ours)ours42.5748.4354.2941.6345.9650.2940.4345.4750.5036.4740.4344.3940.2845.0749.87

Table A.1.

Table A.2: Training-data comparison on ProactiveVideoQA. We report PAUC-F for the pretrained Qwen3.5-A3B checkpoint (Base Model) and 3 checkpoints post-trained on datasets from MMDuet2 (Wang et al., 2026), VideoChat3 Li et al. (2026), and our proposed BLIP-Live-780K dataset. Our BLIP-Live-780K dataset demonstrates better suitability as a post-training dataset for proactive streaming usecases. Best and second-best marked in each column.

Dataset[WEB] ω=0.0\omega=0.0[WEB] ω=0.5\omega=0.5[WEB] ω=1.0\omega=1.0[EGO] ω=0.0\omega=0.0[EGO] ω=0.5\omega=0.5[EGO] ω=1.0\omega=1.0[TV] ω=0.0\omega=0.0[TV] ω=0.5\omega=0.5[TV] ω=1.0\omega=1.0[VAD] ω=0.0\omega=0.0[VAD] ω=0.5\omega=0.5[VAD] ω=1.0\omega=1.0[Mean] ω=0.0\omega=0.0[Mean] ω=0.5\omega=0.5[Mean] ω=1.0\omega=1.0
Base Model51.8554.2456.5437.6239.2940.8233.4234.4735.4641.9043.7545.5141.2042.9444.58
MMDuet226.7227.3427.8923.6424.3925.2116.3116.4916.727.427.467.8218.5218.9219.41
VideoChat352.3055.1556.9238.2439.7141.5239.5240.5343.8243.2445.7247.4543.3345.2847.43
BLIP-Live-780K53.1357.4861.4143.7746.0548.1340.7143.1145.2546.5649.6752.5546.0449.0851.84

Table A.2.

Harness Ablations

We next provide additional ablations on our harness design. We first compare against existing harness designs from Wang et al. (2025a); Yao et al. (2026) in Table A.8. Then we compare against harness tuning with a general strategy built following Zhang et al. (2026c) without our explicit separation of harness components used in our tuning pipeline. This comparison is also included in Table A.8. These results demonstrate how our gains through automated harness tuning are non-trivial and result in a harness outperforming prior work on proactive streaming tasks.

We also report the full numbers for harness tuning on the BLIP-Live-780K validation set for our Gemma4-A4B model in Figure A.1.

Both backbone-specific searches converge to the same high-level design despite different pretraining, and the final search iteration changes performance only slightly (Figure A.1). This supports convergence across these two backbones, rather than establishing stability across random validation subsets. Longer visual windows did not improve the frozen-model harness; finetuned models benefited from context up to 32 frames. We fix the decision rate at 1 Hz to match prior work; higher rates could improve temporal resolution but were not evaluated.

Methodω\omega = 0.0ω\omega = 0.5ω\omega = 1.0ω\omega = 0.0ω\omega = 0.5ω\omega = 1.0ω\omega = 0.0ω\omega = 0.5ω\omega = 1.0ω\omega = 0.0ω\omega = 0.5ω\omega = 1.0ω\omega = 0.0ω\omega = 0.5ω\omega = 1.0
[WEB][EGO][TV][VAD][Mean]
MMDuet231.8232.7633.5428.9529.5730.7021.3622.2022.5412.8813.0313.4223.7524.3925.05
VideoChat334.6735.8636.8830.1230.3631.3437.9740.7542.8424.2824.6724.8831.7632.9133.99
BLIP-Live-Qwen53.1357.4861.4143.7746.0548.1340.7143.1145.2546.5649.6752.5546.0449.0851.84

Table A.3. Training Backbone Ablation. We report PAUC-F for MMDuet2 and VideoChat3 with a common Qwen3.5-A3B backbone as our BLIP-Live-Qwen. The backbone is held fixed, so the comparison reflects our training method and dataset. Best and second-best marked in each column.

ModelORAPTRUOLOSCEOLEOSCARAllOFRIFRNARTUAllORCTRCAllOverall
Explicit Proactive Task
Implicit Proactive Task
Contextual Q
Offline MLLMs Response-in-Last
LLaVA-OneVision7.211.54.910.04.96.95.63.26.83.86.311.629.812.910.85.78.28.7
Qwen2-VL11.78.114.910.51.78.910.66.09.010.24.426.549.522.613.39.411.313.3
MiniCPM-V12.312.610.713.78.67.511.95.510.411.89.236.055.328.132.625.429.018.1
LLaVA-NeXT-Video8.39.47.410.27.87.410.35.68.36.46.721.145.920.010.19.89.911.9
InternVL-V29.314.69.510.61.76.33.03.67.33.39.215.528.214.016.915.616.210.5
VLMs for Streaming Detection
CLIP7.39.57.48.51.84.72.22.75.52.85.251.329.322.24.63.84.210.1
LaViLa8.410.79.09.13.15.43.64.36.77.810.056.234.427.19.428.919.214.3
EgoVLP10.511.08.78.55.55.65.34.47.46.210.758.448.330.98.025.316.615.5
Offline MLLMs Polling Strategy
LLaVA-OneVision8.38.822.825.413.59.89.610.313.620.320.935.949.931.814.61.98.218.0
Qwen2-VL13.713.515.429.58.015.416.610.915.417.819.856.463.139.313.07.710.421.3
MiniCPM-V14.916.817.126.87.712.912.513.115.215.921.046.862.236.524.328.926.622.9
LLaVA-NeXT-Video15.614.621.926.812.814.213.512.316.518.623.244.951.634.619.97.713.821.3
InternVL-V211.35.97.010.10.72.75.22.25.68.32.94.311.26.76.15.35.75.9
Online MLLMs
LIVE(threshold=0.8)9.711.07.410.81.96.03.65.67.04.27.412.912.89.319.613.811.89.1
LIVE(threshold=0.9)11.213.97.913.25.69.46.08.99.55.88.941.046.725.611.326.518.915.5
MMDuet7.210.317.610.24.26.18.88.59.110.07.750.169.134.217.423.120.317.8
EyeWO26.626.625.126.819.822.320.820.723.624.831.075.378.752.539.547.843.634.7
Ours
Gemma4-A4B19.423.524.020.910.616.813.117.418.212.616.144.152.031.221.033.327.123.2
Qwen3.5-A3B14.415.522.919.716.912.220.011.116.619.914.414.541.822.616.112.214.118.0
BLIP-Live-Gemma30.324.738.933.528.331.532.832.931.626.430.528.436.330.441.439.640.532.5
BLIP-Live-Qwen32.831.836.935.632.234.538.639.135.227.630.425.345.332.145.834.940.435.1

Table A.4. ESTP-F1 score on Full ESTP-Bench. We report complete per-task ESTP-Bench results corresponding to the category-level summary in Table 8 of our main paper. Baselines are from Yu et al. (2025) and our 4 models evaluated following identical settings. Best and second-best are marked in each column.

Interactive PO Task

Existing proactive evaluation tasks contain a single query asked at start of the video and their established inference strategy does not involve injecting new queries mid video. We introduce our PO-I task using the same videos in the PO split of StreamingBench (Lin et al., 2024), where each video has up to 5 queries injected during the video. This requires agentic models to handle multiple queries at different times of the videos, introducing several new challenges.

We build each PO-I episode from one video in the StreamingBench PO split. The video is streamed once, and its five annotated queries are injected at their original timestamps. We score every query with the same time-accuracy rule as the official PO split. The key difference lies in the inference setting, which injects queries interleaved with video frames, making the task more difficult. In these videos, each next query arrives only after the preceding gold response, so PO-I evaluates completed-request updates. The model must switch to the newly injected query while earlier frames and responses remain in context, and must keep later responses specific to that query.

We note that our training videos do not contain this style of queries mid video. However, our memory design allows model to condition its behavior on the current memory state. This is enforced in the harness and training as well. This allows the model to generalize zero-shot to multi-query interactions during inference, which aligns with positive results we obtain for this task.

Additional Dataset Details

For all experiments, BLIP-Live-780K train and validation splits were filtered for overlap with ProactiveVideoQA, ESTP-Bench, StreamingBench (PO), and OVOBench forward-active-responding tasks, by matching normalized source-video IDs (including parent-video IDs) and file checksums.

Table A.5: PAUC-F on Full ESTP-Bench. We report per-task PAUC-F (ω=0.5\omega= 0.5, GPT-4.1 judge) on the full ESTP-Bench. Best and second-best are marked in each column. EyeWO (Yu et al., 2025) official checkpoint evaluated under identical settings.

ModelExplicit Proactive TaskImplicit Proactive TaskContextual QOverall
ORAPTRUOLOSCEOLEOSCARAllOFRIFRNARTUAllORCTRCAll
EyeWO10.867.083.4310.727.1610.204.5611.238.1617.4818.355.766.8012.1016.226.6411.439.75
Gemma4-A4B11.0516.9015.5914.077.5211.166.4110.3311.639.6610.709.9213.4310.9311.5114.0212.7711.59
Qwen3.5-A3B23.8827.1723.0729.7829.2423.1029.8723.4726.2029.4923.2831.6535.1529.9035.6728.4632.0728.09
BLIP-Live-Gemma20.2325.9730.0629.8320.8924.4117.7321.2823.8023.4330.2726.5135.9529.0432.3625.6128.9926.04
BLIP-Live-Qwen24.6231.7730.4729.0830.8227.8431.6424.2428.8128.5636.5627.0736.4232.1538.8729.5034.1930.53

Table A.5.

Table A.6: Golden Fires per query on Full ESTP-Bench. We report mean assistant fires per query, so repeated and rephrased responses both count. A query denotes one question on Explicit and Implicit splits, and one multi-question session on Contextual split. The first row calculates Gold Rate by assigning a single response to each gold window, which is how an ideal assistant is expected to respond.

ModelExplicit Proactive TaskImplicit Proactive TaskContextual QOverall
ORAPTRUOLOSCEOLEOSCARAllOFRIFRNARTUAllORCTRCAll
Gold Rate3.984.632.222.674.773.452.106.294.072.732.735.346.344.452.8615.108.875.41
EyeWO15.4317.088.8311.2325.4517.3611.8223.7317.1213.8110.4196.02109.9362.6119.01234.29124.6555.28
Gemma4-A4B19.7212.9614.2517.7933.2925.5733.4539.4825.6526.1519.6049.4132.9233.3323.0348.9035.7529.98
Qwen3.5-A3B1.020.920.961.020.861.241.101.061.041.341.210.843.811.860.582.191.371.31
BLIP-Live-Gemma4.975.012.364.517.006.675.077.055.533.683.194.214.253.904.299.726.965.55
BLIP-Live-Qwen4.573.581.894.143.093.582.875.733.942.291.753.835.203.433.178.295.694.29

Table A.6.

Table A.7: Gold-window lengths across ESTP-Bench splits. Explicit, Implicit, and Contextual items use source videos of similar length, but their gold windows differ sharply. Compared to explicit, implicit gold windows are 3.85× longer and contextual gold windows are 3.14× longer.

SplitQueriesAverage Video LengthAverage Gold WindowTotal Gold-Window Time / Query
Explicit85129:094.81 s19.99 s
Implicit36130:2918.53 s84.07 s
Contextual43033:3415.08 s133.70 s

Table A.7.

Table A.8: Additional Ablation on Harness Design. PAUC-F (ω=0.5\omega= 0.5) on WEB and EGO for Gemma4-A4B. We compare our harness against a harness tuned by general harness evolution (Zhang et al., 2026c), and harnesses from StreamBridge (Wang et al., 2025a) and Yao et al. (2026). All baselines are implemented by us and evaluated under identical settings.

MethodWEBEGO
Ours (Default)43.436.0
StreamBridge Harness32.56.2
Streaming Harness28.512.7
General Harness Evolution40.231.4

Table A.8.

Dataset Splits. We report statistics on the data splits of BLIP-Live-780K dataset used for our training and validation in Table A.9.

Table A.9: BLIP-Live-780K Additional Details. Training and validation split details of our BLIP-Live-780K dataset.

SplitVideosQuestionsResponses
Train176,021176,021777,278
Validation1,7681,7687,666
Total177,789177,789784,944

Table A.9.

Automated Harness Tuning

Figure A.1. Automated Harness Tuning. We plot all 100 evaluations on our BLIP-Live-780K validation with various harness design candidates for Gemma4-A4B model. The selected models for each round are highlights (blue). The full tuning process achieves a gain of 12.85 PAUC-F points on the validation data.

Dataset Generation Details

Automated Proactive Query-Response Generation: For a given video VV, we automatically generate a proactive interaction in two successive model calls.

First, an instruction-generation model observes the complete video and synthesizes a single video-specific user query qq. The query requests notifications about a concrete and visually verifiable event, action, or state change (e.g., an object being picked up or a person entering the scene). We require each query to define a specific, visually observable trigger, rather than requesting general video narration or relying on subjective criteria such as “interesting” events.

Next, a response-generation model receives VV and qq and produces a temporally ordered sequence R={(ti,ri)}i=1N\mathcal{R} = \{(t_i,r_i)\}_{i=1}^{N}, where tit_i is the trigger time and rir_i is a concise natural-language response. To approximate real-time deployment, we instruct the model to emulate an online observer. Although the complete video is provided to the model, we prompt the model and require it to emulate causal streaming observation: each response must rely only on evidence visible up to tit_i, must not anticipate future events, and must be emitted at the first frame at which the query condition is satisfied. This is a prompt-level constraint rather than a restriction on visual access; human reviewers correct responses that precede sufficient visual evidence. The model remains silent between triggers and returns an empty sequence when no valid trigger is detected throughout the video.

To balance computational cost with the need for fine-grained temporal reasoning, we use different models for the two generation steps. Specifically, we use gemini-3-flash-preview for query generation and gemini-3-pro-preview for response generation. We remove the audio track and sample each video at 1 FPS. Consequently, trigger times are annotated at one-second resolution. The generated queries, responses, timestamps, and associated metadata are stored in structured JSON format and subsequently verified and refined during the human-review stage.

Human Annotation Repair: Human annotators first assess the validity of each generated query. A valid query must specify a concrete and visually recognizable condition that can be detected causally from a streaming video and can therefore support a well-defined response decision. We reject queries whose triggers are ambiguous, subjective, not visually observable, or dependent on future context. Only samples with valid queries proceed to response-level annotation repair.

Dataset review interface

Figure A.2. Dataset review interface. Reviewers inspect each sampled video frame against the instruction and existing annotation, navigate temporally, and mark only frames requiring correction. The interface records missing responses, hallucinations, temporal boundary cases, and deletion decisions while automatically treating unmarked frames as correct.

For each valid query, annotators inspect the timing and content of every generated response and repair the annotations when necessary. A valid response must be relevant to the query, visually grounded at its assigned timestamp, free of hallucinated content, and consistent with causal observation. In particular, the response should be unambiguously supported by the frames observed up to that time, while it may use preceding context but must not reveal information from future frames. Annotators correct inaccurate text, shift temporally misaligned responses, remove spurious responses, and add missed responses as needed.

For instantaneous events, the response is generated at the moment when the event becomes visually identifiable. For continuous events that persist over a period of time, annotators additionally enforce semantic continuity across consecutive responses. For example, later responses use expressions such as “continues” or “still” to indicate that the previously observed event remains ongoing.

Quality Control: A professional video data-annotation team recruits and trains the annotators, who receive detailed instructions and examples before performing annotation repair. We then conduct quality control at the batch level using PhD-level expert reviewers who are independent of the annotation repair process.

At the beginning of the human annotation process, we partition the automatically generated samples into ten disjoint batches and process them sequentially for human annotation repair. After the annotators complete a batch, independent expert reviewers audit a randomly selected subset of videos and their annotations using a purpose-built web interface. The reviewers examine the complete video timeline. They evaluate the annotation at each one-second time step, treating both the presence and absence of a response as a decision to assess. A response is labeled a true positive if the reviewer confirms that it is warranted and its content is correct, whereas a no-response decision is labeled a true negative if the reviewer agrees that no response is warranted. An unwarranted or incorrect response is labeled a false positive, and a missing response is labeled a false negative.

Human rating interface for PAUC-F

Figure A.3. Human rating interface for PAUC-F. Raters inspect the video, ground-truth response regions, and timestamped responses from two anonymized system variants, then select the preferred response stream. The aligned timeline and side-by-side response lists expose differences in event coverage, response timing, and repetitive firing behavior.

We quantify annotation quality using the F1 metric. A batch is accepted, and annotation proceeds to the next batch, only if F1>0.95F1 > 0.95. Batches that do not meet this threshold undergo further correction and review. During the initial batches, we have repeated the cycle of annotator training, annotation, and independent evaluation multiple times to calibrate the annotation criteria and establish consistent quality before processing the remaining batches.

Dataset Review Tool We use the purpose-built interface shown in Figure A.2 for frame-level review of the generated annotations. Reviewers step through the video at one-second intervals and flag problematic frames as missing, hallucinated, boundary cases, or incorrect deletion decisions. Unmarked frames are recorded as correct when the reviewer completes the video, enabling efficient and consistent auditing of both responses and silence decisions.

PAUC-F Metric Details

We discuss the behavior of proposed PAUC-F compared to the parent PAUC metric and then present details on our human study.

Behavior Compared to PAUC. PAUC Wang et al., 2025b scores each gold turn on its reply window (tstart,tend)(t^{\mathrm{start}}, t^{\mathrm{end}}). In-window replies at τ1<⋯<τP\tau_1 < \cdots< \tau_P get judge scores sps_p, and PAUC is the area under that time–score polyline divided by the maximum area (tend−tstart)S(t^{\mathrm{end}} - t^{\mathrm{start}})S, with S=2S = 2:

PAUC=(τ1−tstart)⋅0.5+∑p=1P−1(τp+1−τp)sp+(tend−τP)sP(tend−tstart)S.(2)\mathrm{PAUC} = \frac{(\tau_1-t^{\mathrm{start}})\cdot0.5+\sum_{p=1}^{P-1}(\tau_{p+1}-\tau_p)s_p+(t^{\mathrm{end}}-\tau_P)s_P}{(t^{\mathrm{end}}-t^{\mathrm{start}})S}. \tag*{(2)}

With no in-window reply this ratio is fixed at 0.5/S=0.250.5/S = 0.25. This is the no-reply score, not a global minimum: wrong in-window replies can lower PAUC toward zero. Before the area is taken, timestamps are shifted by ω∈[0,1]\omega\in[0,1],

τp′=tstart+(1−ω)(τp−tstart),(3)\tau'_p=t^{\mathrm{start}}+(1-\omega)(\tau_p-t^{\mathrm{start}}), \tag*{(3)}
SplitChoice12345678910Total%
WEBv10010043022126.0
v22020192016171820181818894.0
EGOv1010022001063.0
v22019202018182020192019497.0

Table A.10. Human Preference Study. We present the exact results from each of the 10 human participants involved in our study. A clear preference for the PAUC-F optimized v2 model was observed.

and τp′\tau_p^{\prime} replaces τp\tau_p in Equation (2). ω=0\omega= 0 leaves time unchanged, so a late reply adds little area. ω=1\omega= 1 moves every τp\tau_p to tstartt^{\mathrm{start}}, so PAUC reduces to sP/Ss_P/S and timing drops out. Our PAUC-F metric inherits the ω\omega term from PAUC, which allows adjusting the weight between quality and timeliness.

Repeating the definition from § 4,

R=PAUC,P=NvalueNfires,PAUC-F=2PRP+R.(4)R = \mathrm{PAUC}, \qquad P = \frac{N_{\mathrm{value}}}{N_{\mathrm{fires}},} \qquad\mathrm{PAUC\text{-}F} = \frac{2PR}{P+R}. \tag*{(4)}

A model that never fires has P=0P = 0. PAUC still returns its no-reply score 0.25, but the harmonic mean is zero, so PAUC-F = 0.

The judge scores responses on {0,1,2}\{0,1,2\}, with maximum S=2S = 2 as in PAUC. Both scores 1 and 2 count as value-adding for precision, while PAUC retains their quality difference. Within each benchmark split or task, we pool firing counts across videos to compute PP and follow official PAUC aggregation by averaging over gold response turns to compute RR, then take their harmonic mean. Reported cross-split Means average the split scores. We set P=0P = 0 when there are no fires and PAUC-F = 0 when P+R=0P + R = 0.

Rescaling PAUC by (PAUC − 0.25)/0.75 can yield negative inputs to the harmonic mean because incorrect in-window replies can score below 0.25 (Table A.1). The sparse-policy example in Table 5 shows how additional fires reduce PAUC-F while leaving PAUC unchanged.

Human Study. This two-harness study tests whether excessive-firing traits are undesirable to humans; it does not establish metric rank agreement across a broad set of model pairs.

We conduct a human preference study with ten raters using the interface shown in Figure A.3. Humans rate response streams from two distinct models across 20 videos from WEB and EGO splits of ProactiveVideoQA dataset. Raters are instructed to select the response that seems more helpful from a real-time video assistant and are shown the video, query, and ground-truth responses (content and intervals).

Human raters overwhelmingly select responses generated from agentic model v2. This model uses a harness tuned to maximize PAUC-F metric. The other v1 model whose responses are less preferred by humans uses a harness tuned to maximize PAUC metric, often resulting in excessive repetition within windows and firing outside windows. The exact ratings from each human participant are provided in Table A.10.

Theoretical Motivation. Our PAUC-F metric is simply the harmonic mean between two established components, the PAUC metric from Wang et al. (2025b) and firing precision. We justify our use of the harmonic mean with two key reasons.

First, the harmonic mean is the strictly correct operator for calculating the true average of rates or ratios (Yule & Kendall, 1947; Ferger, 1931). In our case, both PAUC and Firing Precision represent rates of success. PAUC is the rate of accumulated quality-score over a bounded temporal window, while Firing Precision is the rate of valid fires over total fires.

Second, from an optimization perspective, the harmonic mean severely penalizes high variance between its inputs (Van Rijsbergen, 1979). For streaming video agents, timeliness (PAUC) and firing restraint (Precision) are competing objectives; a model can easily maximize PAUC by spamming responses, driving Precision near zero. An arithmetic or geometric mean allows linear or proportional substitutability, allowing the model to hide terrible precision behind a perfect PAUC. The harmonic mean enforces that the marginal utility of maximizing PAUC drops to zero if Precision is sacrificed, mathematically guaranteeing that a high PAUC strictly requires Pareto-optimal balance between the two. This also mirrors the theoretical foundation of the F1 score in information retrieval.

ComponentConfiguration
Decision rate1 Hz
FramesThree most recent, [t−2,t][t - 2, t], truncated at the video start
TimestampsAbsolute MM:SS immediately before each frame
QuestionStanding query; a later query enters only at its reveal time
MemoryLast accepted response and its emission time. Acceptance replaces it; silence leaves it unchanged
DecodeGreedy, at most 192 new tokens
Output[] for silence, or one {time, content} object
ParserStrip optional Markdown fences. Decode a JSON array, otherwise the first array-shaped span. Keep objects with a numeric time and non-empty content. Drop times outside ±0.5\pm0.5 s, snap the rest to the current step, and keep at most one. Anything else is silence
Gemma inputNative system, query, and observation turns. Timestamps sit beside image embeddings. Generation starts after an empty model turn
Qwen inputOfficial chat template with thinking disabled. Query, memory, and timestamp–image pairs share one user message

Table A.11. Training-free harness shared by Gemma4-A4B and Qwen3.5-A3B. Model weights stay frozen.

ComponentFull harness searchLimited calibration
ModelsGemma and QwenOther offline VLMs
ObjectiveMaximize validation PAUC-FMaximize validation PAUC-F
Editable surfaceWindow, memory, prompt, and output protocolPrompt and output schema only
Proposal schedule10 candidates per round1 candidate per round
Budget10 rounds; 100 candidate evaluations/model5 candidates ×\times 20 examples/model
Samples per round50020
FeedbackPAUC-F MetricPAUC-F Metric

Table A.12. Search and calibration budgets. A model-example evaluation is one candidate on one video.

TaskPrompt Description
Harness tuningTune a frozen VLM harness so validation PAUC-F rises. Each round provides the current harness, one hypothesis, the editable surface, and prior scores and diagnostics. Propose the budgeted edits on that surface, and state the expected effect of each. Edits may only change the window, memory, prompt, or output protocol. Keep an edit only when validation PAUC-F improves.
Harness calibrationCalibrate the prompt and output schema of a frozen harness so validation PAUC-F rises. Each round provides the fixed harness, the previous candidate, and its score and diagnostics. Propose one edit, and state its expected effect. Only the prompt and output schema may change. Keep an edit only when validation PAUC-F improves.
Inference (Final)Streaming video assistant. Each second, the input is the last accepted answer and its time (empty if none) plus the latest frames, each marked with absolute MM:SS. The newest stamp is now. Speak only when those frames show a useful change; otherwise return an empty list and leave the last answer as it stands. A spoken turn is one short factual sentence tagged with the current time in seconds.

Table A.13. Training-free harness prompts for search, calibration, and inference. The frozen VLM uses a binary silence/response interface.

StepRule
Timeline1 FPS frames with absolute MM:SS timestamps. Consecutive frames that share a caption are one event
StatesFirst frame that supports the caption is Response; later frames of that event are Hold; every other frame is Silence
ChunksAt most 32 frames. A cut that would split a response from its hold is moved so they stay together. Original indices and timestamps are kept
ContextSystem instruction, standing query, and the last response that precedes the visual window
Extra chunksResponse-bearing chunks are kept, together with trailing and post-response chunks so sustained silence is seen
Response tokensEvery Response position, including its answer text
Non-response runIts first position, plus up to four further Silence positions and two further Hold positions
No-response chunkTwo Silence positions

Table A.14 (PDF p. 24). Episode construction and token supervision. A non-response run is a maximal stretch of Silence and Hold.

SettingValue
OptimizerAdamW
Learning rate2×10−42 \times10^{-4}
ScheduleCosine decay; 5% warmup
Training durationOne epoch
Per-device batch size1
Gradient accumulation1
Effective global batch size8
Precisionbfloat16 with TF32 enabled
AttentionSDPA
Memory optimizationNon-reentrant gradient checkpointing; ZeRO-2
Hardware8×8 \times NVIDIA H200, one data-parallel process per GPU
Random and data seeds42
Frames384×384384 \times384, bicubic, then the native image processor
LoRARank 64, scale 128, dropout 0.05
Gemma modulesAttention, dense MLP, and router
Qwen modulesAttention, shared-expert MLP, and linear-recurrence projections

Table A.15 (PDF p. 24). Shared optimization configuration. Adapters sit in the language tower; the visual encoder stays frozen.

Inference Harness Details

This section describes the training-free harness for frozen offline VLMs. Table A.11 is the harness used for every reported offline model. Table A.12 is the validation-only search that selected it, and Table A.13 presents only the harness-search, calibration, and training-free inference prompts.

The frozen VLM returns silence or a response using the structured output format; it does not predict a Hold token. Prompts and state tokens for the finetuned models are described separately in Appendix § 13. In proprietary-model evaluation, malformed outputs trigger up to three retries (four attempts in total) before applying the parser’s silence fallback. No unresolved parsing failures remained after retries across all proprietary models tested.

During harness tuning, each round tests ten edits of one hypothesis. An edit is promoted when validation PAUC-F improves on its parent. Combined promotions are kept only after that combination itself improves, and those confirmation runs count toward the budget in Table A.12.

TaskPrompt Description
Training and inferenceStreaming assistant over frames labeled with absolute MM:SS times. Emit </Silence> when the frame is irrelevant, </Hold> when the event continues but no new answer is due, and </Response> plus one complete answer when the frames suffice. A prior answer may be attached; repeat it only when the new frames add evidence. Stay within what the frames show.

Table A.16. System prompt shared by supervised finetuning and finetuned inference. The three-state output interface is used by BLIP-Live-Gemma and BLIP-Live-Qwen.

Training Pipeline Details

Table A.16 presents the shared system instruction used for supervised finetuning and inference with BLIP-Live-Gemma and BLIP-Live-Qwen. These models predict </Silence>, </Response>, or </Hold>. Only </Response> produces a user-facing answer and updates response memory; Silence and Hold produce no answer and leave memory unchanged. Hold distinguishes an ongoing, already-reported event from irrelevant frames labeled Silence. Both training and inference use up to 32 frames and the last response as memory, at a 1 Hz decision rate. Table A.14 describes how we build the target episodes for supervision. The loss is next-token prediction on the sampled state tokens and full answer text for response text tokens, all at equal weight. System, query, memory, visual, and unsampled target tokens stay in context only.

Examples of similar length are grouped into batches. Intermediate exposure checkpoints are saved, and the reported checkpoint is the one selected by validation PAUC-F (§ 6).

References

  1. [1]Steven Abreu, Tiffany D. Do, Karan Ahuja, Eric J. Gonzalez, Lee Payne, Daniel McDuff, and Mar Gonzalez-Franco. PARSE-Ego4D: Toward bidirectionally aligned action recommendations for egocentric videos. In ICLR Workshop on Bidirectional Human-AI Alignment, 2025.

Paper details

vlnc:W_h6quckwz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents