betaFeedback

WorksComputer Science

FLUID: Flexible, Evidence-Grounded Situated Task Guidance

Abstract

Real-time task guidance is inherently flexible: when an agent guides a user through a task, multiple responses and timings can be appropriate, and sometimes the best action is to remain silent. Existing benchmarks often miss this flexibility by treating synthesized dialogues as fixed targets for what to say and when to say it. We introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for modeling, evaluating, and enabling flexible interaction in multimodal task guidance. Using naturally occurring human-human guidance from WTaG and HoloAssist, FLUID grounds interventions in their supporting evidence, necessity, and timing flexibility, and uses this representation both to evaluate flexible guidance and to build a real-time agent for task guidance. Our agent achieves the strongest overall balance of response correctness, intervention precision, timely guidance, and response latency among evaluated systems, while still leaving room for improvement in avoiding speech overlap. Together, FLUID moves situated task guidance beyond rigid reference matching toward more natural and adaptive human-AI interaction. Project page: https://fluid-guidance.github.io.

Introduction

With the growing adoption of smart glasses and other wearable devices, multimodal AI assistants are becoming increasingly accessible for providing real-time, situated assistance in everyday activities [7, 17, 11]. An important emerging application is perceptual task guidance, where an AI assistant observes the user’s environment and ongoing actions through an egocentric camera and provides context-aware guidance through natural language as shown in Figure 1. This kind of situated perceptual task guidance requires continuous coordination between visual perception, task understanding, and communication. The AI assistant must interpret the user’s

Figure 1. Illustration of perceptual task guidance.

actions and task progress, respond to questions, provide step-by-step instructions, detect mistakes, and proactively offer assistance when appropriate.

To evaluate these capabilities, existing work commonly relies on human-human interaction data or synthetically generated dialogues as references for expected guidance behaviors, measuring how closely the AI instructor’s responses align with the reference responses in both semantic content and temporal placement [3, 32, 25]. Although a good starting point, these approaches treat reference interactions as fixed targets, overlooking the inherent flexibility of situated guidance. In practice, human-human guidance is fluid and adaptive. Certain situations, such as incorrect actions or imminent safety risks, require immediate intervention, while others allow a wider range of appropriate timing for proactive assistance. Similarly, backchannels and acknowledgments help maintain shared understanding, but their timing and frequency can vary considerably. This creates a fundamental mismatch between rigid reference-based evaluation and the flexible nature of real-world task guidance, calling for evaluation criteria that can take into account both the correctness of guidance and the flexibility of communication.

Addressing this challenge requires a principled framework for characterizing the flexibility of situated guidance. Importantly, flexibility does not imply unconstrained behavior. Effective guidance must remain grounded in perceptual and interaction evidence, with different communicative behaviors subject to different requirements. When AI assistants decide to intervene or provide guidance, some interventions (e.g., correcting a user mistake or giving a warning about a potential hazard) are necessary within narrow temporal windows, while others (e.g., backchannel) are optional and allow greater flexibility in content and timing. Human references remain valuable for identifying effective guidance behaviors, but should not prescribe a single correct response at a fixed moment. Instead, the key is to characterize the underlying evidence that warrants communication and the conditions under which different responses are appropriate. This evidence-grounded perspective provides a foundation for both evaluating and generating flexible guidance behaviors.

In this paper, we introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for characterizing, evaluating, and enabling flexible interaction dynamics in multimodal task guidance. First, we develop an evidence-grounded representation that characterizes communicative behaviors in terms of their necessity, supporting evidence, and temporal flexibility (FLUID-Modeling). Second, we introduce an evaluation framework that leverages human references while moving beyond strict response matching to assess the appropriateness of guidance content and timing (FLUID-Evaluation). Third, we propose an agent architecture that explicitly models and adapts to evolving interaction dynamics, enabling flexible, evidence-grounded decisions about when and how to communicate, including reactive responses, proactive guidance, and appropriate silence (FLUID-Agent). Together, these contributions provide a systematic approach to understanding, evaluating, and generating flexible, evidence-grounded guidance, advancing multimodal AI assistants toward more natural and adaptive human–AI interaction.

Related Work

Real-Time Interaction. Recent advances in multimodal systems have rapidly expanded real-time human-AI interaction, from streaming audio-visual perception [17, 21] to low-latency, full-duplex spoken interaction [7, 26, 13, 4, 11]. Scaling such interactive capabilities has increasingly relied on synthetic or automatically constructed interaction data, often by repurposing existing egocentric videos [12, 19, 8, 14] and generating intervention traces at scale [25, 32, 15]. In contrast, naturally occurring human-human interactions in which one person continuously observes and guides another through a physical task remain scarce. Existing efforts include WTaG [1], HoloAssist [28], EgoInstruct [24], Ego-EXTRA [23], and Ego-MC-Bench [2], but some of them remain limited in released modalities. In this work, we primarily build on WTaG and HoloAssist, both of which provide audio alongside egocentric video, to study how interactions naturally unfold during human guidance.

Proactive Agent Modeling. Existing proactive video agents span query-conditioned event response, continuous commentary, and situated procedural guidance [30, 16, 5, 9, 20, 29, 6, 32]. Representative systems typically produce textual responses or discrete decisions about whether to intervene, with timing supervision derived from annotated event boundaries, task transitions, mistakes, or synthesized utterances [18, 34, 33, 3, 15]. Recent methods move beyond purely event-triggered policies: Pro²Assist additionally annotates assistance need, while Vinci2 generates visually grounded service opportunities and reasons about whether intervention is warranted [31, 10]. They provide increasingly rich supervision; however, their assistance targets are constructed from task states or annotated events rather than naturally occurring intervention decisions. In contrast, we study human-human guidance. This preserves a broader range of communicative behaviors and enables intervention-specific annotations of necessity and valid timing, characterizing not only when assistance may occur but how much flexibility each interaction permits.

FLUID-Modeling: Grounded Evidence in Task Guidance

Problem Definition

In real-time task guidance, an agent continuously observes a user performing a task and must decide how to respond appropriately as the task and interaction evolve. We characterize each intervention decision along four dimensions. WHY: what evidence warrants the intervention, namely the user utterance, action, or state that the guide reacts to; we refer to it as the grounded evidence of the intervention. WHETHER: whether the intervention is necessary, optional, or not needed. WHEN: the period during which the intervention is useful. WHAT: the interaction act of the intervention and its content. Interventions triggered by a user utterance are reactive; all others are proactive.

Source Data

We study WTaG ([1]) and HoloAssist ([28]), two egocentric datasets of naturally occurring human-human task guidance in which a remote instructor observes the user’s view and provides verbal guidance. The two datasets exhibit complementary interaction regimes. WTaG is relatively self-paced: users receive a high-level task recipe in advance and can often proceed independently, which produces mixed-initiative interaction. HoloAssist has a stronger knowledge asymmetry and is predominantly instructor-led, with users relying on ongoing guidance for each step. In total, we select 43 WTaG sessions and 72 HoloAssist sessions with substantive interactions between users and instructors.

Multi-Layer Annotation

To reveal the interaction semantics underlying naturally occurring human interventions, we augment WTaG and HoloAssist with four linked annotation layers, which together cover the four dimensions of § 3.1 (see Figure 2 for an example).

Four-layer annotation of a WTaG coffee session

Figure 2. Four-layer annotation of a WTaG coffee session, covering utterance taxonomy, action and state, trigger links, and intervention timing. Each instructor intervention is grounded in its triggering evidence, labeled by necessity, and assigned an evidence-derived intervention window.

Layer 1: Utterance Taxonomy. We categorize every user and instructor utterance using an utterance-level taxonomy with 11 user categories and 13 instructor categories.

Layer 2: Action and State. We annotate user actions and states that are closely related to instructor interventions, including hand actions, visual loss, stalling, and searching. Each action is further labeled by whether it is valid or deviating and by its effect on task execution, such as task progress, step completion, or recovery.

Layer 3: Trigger Link. Using the annotated utterances and actions, we link each instructor intervention to the preceding utterance, action, or state that triggered it, forming a grounded evidence link that explains why the human instructor decided to intervene. Each link receives a trigger type derived from the linked events, such as Step Done, Error, or User Query. We further categorize each intervention as necessary if, without it, the user would make a mistake, get stuck, or lack needed information; optional if the user would proceed correctly on their own, so that speaking and remaining silent are both acceptable; and excluded if it is irrelevant to the task, such as self-repairs, or relies on knowledge that cannot be obtained from the shared egocentric view, the dialogue, or the task knowledge.

Layer 4: Evidence Onset and Intervention Window. We annotate temporal anchor points for interventions, which define the time window within which an intervention should be initiated. For reactive interventions, we annotate (1) the time at which the user finishes their most recent utterance, tusrt_{\mathrm{usr}}, and (2) the latest time at which the intervention remains useful, tlatestt_{\mathrm{latest}}. For proactive interventions, there is no corresponding tusrt_{\mathrm{usr}}; instead, we annotate (1) the earliest time at which initial signs suggest that an intervention may be necessary, tearlyt_{\mathrm{early}}, and (2) the time at which enough evidence becomes available to determine that the intervention is necessary, tevidt_{\mathrm{evid}}, together with tlatestt_{\mathrm{latest}}. Together, these anchors define an evidence-derived window for evaluating intervention timing.

Annotation Workflow. For WTaG, we first use majority voting across multiple language-model annotators to produce initial annotations, followed by careful manual review of all instances. Based on the errors and practical observations identified during this process, we refine the machine annotation pipeline and apply it to the selected HoloAssist sessions, and manually review about 10% of the generated annotations to ensure the synthetic data quality.

Interventions Are Context-Dependent

Our annotations show that natural human guidance does not follow a fixed trigger-to-intervention mapping: whether an intervention is needed, and what form it should take, depends on the surrounding interaction context.

Similar situations may or may not require intervention. Among step completions on the self-paced WTaG, only 12.9% are followed by a necessary intervention (e.g., providing the next-step instruction), while 17.5% are optional (e.g., “Nice” or “uh-huh”) and 69.6% receive no intervention (n=418n = 418). In the more instructor-led HoloAssist setting, the corresponding proportions are 65.9%, 12.2%, and 21.9% (n=649n = 649). Thus, whether a step completion warrants guidance depends strongly on the surrounding interaction regime and context, rather than on the event type alone. In contrast, existing approaches often use a more rigid event-to-response mapping, such as triggering a next-step instruction whenever a step is completed.

These observations motivate evaluating guidance against the grounded evidence that warrants an intervention, whether intervention is necessary, and when it remains useful.

FLUID-Evaluation on Task Guidance

Existing reference-based evaluations assess guidance primarily through content similarity and temporal proximity to human reference utterances, often using a fixed timing window [32, 3, 2]. Such criteria overlook the flexibility of natural interaction, where appropriate guidance depends on task evidence and may vary in necessity and timing. We introduce an evidence-derived evaluation framework that leverages our annotations to evaluate intervention appropriateness based on supporting evidence, necessity, and valid timing windows. Appendix § 9.3 specifies the protocol and metrics in full.

Intervention Matching

Real-time interaction does not follow fixed turn boundaries: within any time interval, a system may produce zero, one, or multiple utterances, whereas evaluation requires a one-to-one correspondence between system utterances and annotated human interventions. We therefore first match them based on content and supporting evidence. For each necessary or optional intervention, we consider candidate system utterances within a matching window that covers its annotated evidence window and the surrounding intervals of adjacent reference utterances. An LLM judge then assesses content equivalence based on the dialogue history and triggering evidence. Since a system may generate multiple equivalent utterances within the same time interval, we select the earliest qualifying utterance as the match.

Timing Appropriateness

Evaluating the timing of matched interventions is essential, yet common approaches [32, 3, 2] have two limitations: (1) fixed timing windows ignore when an intervention becomes possible and when it stops being useful, admitting invalid onsets and excluding valid ones; and (2) binary in-window scoring treats all onsets within a window as equally appropriate, although their usefulness varies with timing. For the first limitation, we derive intervention-specific valid windows from the temporal anchors annotated in § 3.3: [tusr,tlatest)[t_{\mathrm{usr}}, t_{\mathrm{latest}}) for reactive guidance and [tearly,tlatest)[t_{\mathrm{early}}, t_{\mathrm{latest}}) for proactive guidance. Unlike a fixed timing window around the reference utterance, each window opens after the user’s turn ends or at the first sign of need, and closes when the intervention is no longer useful. An onset is valid if it falls anywhere within the window of the corresponding intervention.

We validate these windows with a human study that isolates timing from content. For each of 79 WTaG instructor utterances, we keep the utterance fixed and sample six candidate onsets around its annotated anchors, spanning early, in-window, and late times and including the instructor’s actual onset. Three researchers rate how appropriate it would be to deliver the same utterance at each onset on a 0-10 scale (See Appendix § 10.4 for more details). We treat an onset as acceptable when its mean human rating is at least 6, and compare this label with whether the onset falls within the proposed intervention window. We report Macro-F1 to measure how well the window classification agrees with human judgments across acceptable and unacceptable timings. Overall, our evidence-derived windows outperform the fixed [−6,+3][-6, +3] s window around reference utterance onsets adopted from ProAssist [32] (Macro-F1: 0.788 vs. 0.713). The improvement is more pronounced for reactive guidance, where our windows exclude inappropriate onsets that would interrupt the user, while for proactive guidance, our windows achieve comparable performance while admitting earlier evidence-supported onsets missed by the fixed window.

For the second limitation, we introduce a continuous timing score that captures varying degrees of timing appropriateness. The score peaks at a preferred response time tpreft_{\mathrm{pref}} and decreases for earlier or later onsets. For reactive guidance, tpref=tusrt_{\mathrm{pref}} = t_{\mathrm{usr}}, since responding immediately after the user finishes speaking is preferred; for proactive guidance, tpref=tevidt_{\mathrm{pref}} = t_{\mathrm{evid}}, when sufficient evidence becomes available. An earlier utterance lacks supporting evidence, and a later one becomes less useful as the task progresses.

Next, we test whether a continuous timing score better reflects human preferences than binary in-window scoring. We start with a simple linear score that decreases from tpreft_{\mathrm{pref}} toward the window boundaries. For each utterance, we compute Spearman’s correlation between the scores assigned to the six candidate onsets and their human ratings. Averaged across utterances, the linear score achieves a correlation of 0.83, compared with 0.65 for binary in-window scoring.

We further refine the shape of the timing penalty. Let tt denote the onset of the system utterance matched to an intervention. For a proactive intervention with t∈WE=[tearly,tevid)t \in W_E = [t_{\mathrm{early}}, t_{\mathrm{evid}}), we use a quadratic penalty that grows as tt moves further before tevidt_{\mathrm{evid}}. For tt at or after the preferred time, in WD=[tpref,tlatest)W_D = [t_{\mathrm{pref}}, t_{\mathrm{latest}}) for both reactive and proactive interventions, we use a cosine-based decay that penalizes short delays mildly and long delays strongly. We compare this score and the linear score with human ratings rescaled to [0,1][0, 1] using mean absolute error (MAE). The cosine-based score achieves lower MAE than the linear score (0.182 vs. 0.225) and is adopted as the final scoring function:

s(t)={1−λE(tevid−ttevid−tearly)2,t∈WE,cos⁡pL(π(t−tpref)2(tlatest−tpref)),t∈WD,0,otherwise.(1)s(t) = \begin{cases} 1 - \lambda_E \left(\frac{t_{\mathrm{evid}} - t}{t_{\mathrm{evid}} - t_{\mathrm{early}}}\right)^2, & t \in W_E, \\ \cos^{p_L}\left(\frac{\pi(t-t_{\mathrm{pref}})}{2(t_{\mathrm{latest}}-t_{\mathrm{pref}})}\right), & t \in W_D, \\ 0, & \text{otherwise.} \end{cases} \tag*{(1)}

The parameters λE\lambda_E and pLp_L control the early penalty and the decay rate, respectively. We set (λE,pL)=(0.7,0.5)(\lambda_E, p_L) = (0.7, 0.5) by cross-validated calibration against the same human ratings, as detailed in Appendix § 10.4.

Based on these, we define Timely Correctness as the average timing score over all necessary interventions. An intervention contributes zero if it has no content-matched system utterance or if that utterance’s onset falls outside its window. Optional interventions are excluded, as remaining silent is also acceptable. We additionally report Response Latency, the median delay from tusrt_{\mathrm{usr}} to the start of system speech tt for content-matched reactive responses.

Complementary Evaluation Metrics

Reference Fidelity. We measure agreement with human references using two metrics following Zhang et al. [32]. Similarity F1 measures embedding-based similarity, while LLM-Judged F1 assesses content equivalence using the criterion described above. Both use a fixed timing window around reference utterance onsets for intervention matching.

Intervention Precision and Turn-Taking. Timely Correctness measures whether necessary guidance is delivered correctly and on time, but frequent interventions may improve coverage at the cost of unnecessary or disruptive speech. We therefore introduce Intervention Precision, the proportion of all played system utterances that correctly match a necessary or optional intervention within its valid window. Optional interventions are counted as correct when produced, while remaining silent for an optional intervention is not penalized. Unmatched, repeated, incorrect, or mistimed utterances lower this score. Additionally, we report Speech Overlap Ratio, the proportion of user-speaking time overlapping with system speech, to measure potential disruptions to natural turn-taking.

FLUID-Agent: From Interaction Dynamics to Real-Time Guidance

Motivated by the interaction dynamics characterized in § 3, FLUID-Agent is a real-time system for flexible, evidence-grounded task guidance. As shown in Figure 3, it uses an asynchronous agentic architecture built around a single Qwen3.6-35B-A3B vision-language model (Qwen Team, [22]), which serves perception, utterance understanding, and intervention reasoning through task-specific prompting. The components communicate through a shared task memory that tracks task-relevant evidence and interaction context over time (see Appendix § 9.4 for implementation details).

Figure 3. Overview of the FLUID-Agent framework. Perception, task memory, and user utterance understanding jointly support evidence-grounded intervention decisions as task and interaction context evolve.

Perception and Task Memory. The perception module continuously analyzes keyframes sampled from short temporal windows of the live video stream. Guided by task knowledge and the current task state, it tracks relevant user actions, states, and task progress. These observations, together with user and system utterances and step-level completion status, are maintained in a continuously updated shared task memory, providing grounded task and interaction context for subsequent intervention decisions.

User Utterance Understanding. Not every user utterance calls for a response. The user utterance understanding module uses the utterance taxonomy and task context to distinguish utterances that require a response from those that do not, such as task narration, progress reports, and acknowledgments. This enables FLUID-Agent to respond when needed while avoiding unnecessary interruptions.

Evidence-Conditioned Intervention Decision. Each potential intervention is grounded in evidence from the ongoing task and interaction, such as an observed error, action progress, or user request. The intervention module uses this evidence and the interaction history to reason about whether, when, and how to provide guidance. As the task evolves, FLUID-Agent reassesses pending interventions, providing evidence-grounded guidance when appropriate or remaining silent when it is no longer warranted.

Asynchronous Execution. FLUID-Agent coordinates perception, utterance understanding, and intervention reasoning asynchronously, allowing each component to process new information as it becomes available. The system continues monitoring the task while speaking, yields to new user requests, and cancels pending guidance when it is no longer relevant.

Experiments & Results

Experimental Setup

Systems. We evaluate four representative baselines with FLUID-Agent: ProAssist 8B (Zhang et al., [32]), VideoLLM-online 8B (Chen et al., [5]), JoyAI-VL-Interaction (Video Understanding Team of JoyAI-VL @ Joy Future Academy, JD, 2026, and Gemini 3.8 Live (Google DeepMind, [11]. These baselines are selected based on four criteria: (1) streaming input and output, (2) proactive response generation, (3) training on task-guidance data, and (4) compatibility with the teacher-forced inference setting.

Protocol. To isolate the quality of individual responses in offline evaluation and avoid error propagation across the interaction, we use teacher-forced inference, conditioning each system on the ground-truth interaction history rather than its own previous outputs. All systems also receive the same structured task knowledge extracted from demonstration videos. They operate in real time with streaming video and audio inputs and spoken audio output, allowing us to measure response latency and speech timing. See Appendix § 10 for implementation details.

Main Results

We evaluate all models alongside the original human instructor guidance from both datasets under the FLUID-Evaluation framework. As shown in Table 1, human instructors substantially outperform all evaluated models, establishing a human performance baseline. By incorporating interaction dynamics, FLUID-Agent generally achieves more timely and precise interventions than other models. However, Gemini exhibits less speech overlap on both datasets, suggesting that effective turn-taking remains a challenge for FLUID-Agent. Substantial gaps remain between FLUID-Agent and human instructors, particularly in proactive Timely Correctness (below 0.07 vs. above 0.88) and Intervention Precision (below 0.17 vs. above 0.83). These results highlight the remaining challenges in providing timely and appropriate guidance.

MetricWTaG ProAssistWTaG VideoLLMWTaG JoyAIWTaG GeminiWTaG FLUID-AgentWTaG HumanHoloAssist ProAssistHoloAssist VideoLLMHoloAssist JoyAIHoloAssist GeminiHoloAssist FLUID-AgentHoloAssist Human
Reference Fidelity
Similarity F1 ↑0.101±0.0180.101 \pm0.0180.091±0.0160.091 \pm0.0160.099±0.0140.099 \pm0.0140.116±0.0210.116 \pm0.0210.146±0.0210.146 \pm0.0211.0001.0000.045±0.0100.045 \pm0.0100.048±0.0120.048 \pm0.0120.076±0.0120.076 \pm0.0120.028±0.0080.028 \pm0.0080.063±0.0120.063 \pm0.0121.0001.000
LLM-Judged F1 ↑0.127±0.0190.127 \pm0.0190.138±0.0190.138 \pm0.0190.138±0.0200.138 \pm0.0200.127±0.0190.127 \pm0.0190.225±0.0220.225 \pm0.0221.0001.0000.051±0.0100.051 \pm0.0100.075±0.0150.075 \pm0.0150.098±0.0150.098 \pm0.0150.037±0.0130.037 \pm0.0130.099±0.0150.099 \pm0.0151.0001.000
Reactive Guidance
Timely Correctness ↑0.088±0.0170.088 \pm0.0170.095±0.0250.095 \pm0.0250.107±0.0200.107 \pm0.0200.126±0.0280.126 \pm0.0280.209±0.0300.209 \pm0.0300.862±0.0270.862 \pm0.0270.020±0.0130.020 \pm0.0130.029±0.0130.029 \pm0.0130.048±0.0170.048 \pm0.0170.016±0.0120.016 \pm0.0120.055±0.0200.055 \pm0.0200.615±0.0540.615 \pm0.054
Response Latency (s) ↓2.00±0.202.00 \pm0.201.70±0.131.70 \pm0.131.52±0.111.52 \pm0.112.16±0.102.16 \pm0.101.15±0.041.15 \pm0.040.20±0.050.20 \pm0.051.35±0.401.35 \pm0.401.44±0.191.44 \pm0.191.66±0.381.66 \pm0.381.28±1.431.28 \pm1.431.17±0.131.17 \pm0.130.47±0.110.47 \pm0.11
Proactive Guidance
Timely Correctness ↑0.046±0.0220.046 \pm0.0220.015±0.0120.015 \pm0.0120.023±0.0120.023 \pm0.0120.063±0.0260.063 \pm0.0260.046±0.0150.046 \pm0.0150.921±0.0180.921 \pm0.0180.025±0.0090.025 \pm0.0090.018±0.0080.018 \pm0.0080.029±0.0090.029 \pm0.0090.020±0.0090.020 \pm0.0090.062±0.0130.062 \pm0.0130.881±0.0110.881 \pm0.011
Intervention Precision and Turn-Taking
Intervention Precision ↑0.088±0.0160.088 \pm0.0160.059±0.0120.059 \pm0.0120.080±0.0140.080 \pm0.0140.145±0.0230.145 \pm0.0230.164±0.0220.164 \pm0.0220.845±0.0270.845 \pm0.0270.052±0.0130.052 \pm0.0130.073±0.0180.073 \pm0.0180.050±0.0080.050 \pm0.0080.128±0.0430.128 \pm0.0430.166±0.0240.166 \pm0.0240.833±0.0210.833 \pm0.021
Speech Overlap Ratio ↓0.213±0.0370.213 \pm0.0370.442±0.0460.442 \pm0.0460.224±0.0250.224 \pm0.0250.059±0.0080.059 \pm0.0080.207±0.0140.207 \pm0.0140.038±0.0090.038 \pm0.0090.206±0.0320.206 \pm0.0320.218±0.0390.218 \pm0.0390.480±0.0710.480 \pm0.0710.063±0.0160.063 \pm0.0160.153±0.0270.153 \pm0.0270.116±0.0260.116 \pm0.026

Table 1. Evidence-grounded evaluation on WTaG and HoloAssist. Values are reported with half-widths of 95% session-cluster bootstrap confidence intervals.

Evidence-grounded evaluation captures intervention quality beyond reference matching. Systems with similar LLM-Judged F1 can exhibit substantially different interaction behavior. On WTaG, Gemini and VideoLLM achieve comparable F1 scores (0.127 vs. 0.138), yet Gemini has much higher Intervention Precision and substantially less speech overlap. Similarly, on HoloAssist, FLUID-Agent and JoyAI achieve nearly identical F1 (0.099 vs. 0.098), while FLUID-Agent achieves higher proactive Timely Correctness and Intervention Precision. By accounting for intervention necessity and evidence-derived timing windows, our evaluation reveals differences in intervention quality that reference-matching metrics cannot capture.

Fine-grained evaluation reveals interaction-specific strengths and failures. Figure 4 reports correctness by trigger type and instructor utterance type. FLUID-Agent leads across multiple categories on both datasets. Across systems, correctness is generally higher for explicit user queries, but remains low for proactive interventions requiring an understanding of task progress and user behavior. Correctness remains low for Error and Stuck triggers, as well as Question to User utterances that require seeking missing information. These results highlight the challenge of grounding proactive interventions in task evidence and deciding when and what to say, motivating our diagnostic analysis in Section 6.3.

Figure 4. Fine-grained intervention correctness (%). Percentage of necessary interventions correctly addressed by each system, grouped by trigger type and instructor taxonomy. Gray cells indicate fewer than 10 interventions.

Diagnostic Analysis with Evidence Ladder

While the preceding results reveal difficulties in deciding when and what to say, it remains unclear how intervention decisions and evidence grounding contribute to these difficulties. To help better understand what types of evidence is difficult to capture and affects agent’s performance, we construct an evidence ladder on WTaG that progressively controls intervention decisions and introduces ground-truth task evidence. L0 retains native behavior; L1 enforces one response at each annotated necessary-intervention onset while suppressing other speech; L2 adds the evidence type; and L3 further provides specific observed facts for action- and state-triggered events (see Appendix § 10.3 for details). None of these levels provides the reference guidance.

Deciding whether and when to intervene is a major bottleneck. The largest gains occur at L1 (Table 2), where reactive and proactive Timely Correctness increase by 0.269 and 0.105 for JoyAI, and by 0.050 and 0.195 for Gemini, respectively. Intervention Precision also improves, reflecting fewer unnecessary interventions. Together, these gains show the importance of deciding whether and when to intervene in real-time task guidance.

MetricL0L1L2L3Δonset\Delta_{\mathrm{onset}}Δtype\Delta_{\mathrm{type}}Δfact\Delta_{\mathrm{fact}}
JoyAI
LLM-Judged F1 ↑\uparrow0.138 ±\pm0.0200.314 ±\pm0.0340.330 ±\pm0.0350.346 ±\pm0.038+0.175 ±\pm0.029+0.016 ±\pm0.013+0.016 ±\pm0.013
Reactive Timely Correctness ↑\uparrow0.107 ±\pm0.0200.375 ±\pm0.0350.409 ±\pm0.0370.411 ±\pm0.037+0.269 ±\pm0.035+0.034 ±\pm0.020+0.002 ±\pm0.008
Proactive Timely Correctness ↑\uparrow0.023 ±\pm0.0120.127 ±\pm0.0330.119 ±\pm0.0340.178 ±\pm0.035+0.105 ±\pm0.032−0.008 ±\pm0.026+0.059 ±\pm0.037
Intervention Precision ↑\uparrow0.080 ±\pm0.0140.347 ±\pm0.0370.378 ±\pm0.0400.398 ±\pm0.037+0.267 ±\pm0.035+0.031 ±\pm0.017+0.021 ±\pm0.015
Gemini
LLM-Judged F1 ↑\uparrow0.127 ±\pm0.0190.301 ±\pm0.0290.313 ±\pm0.0300.362 ±\pm0.027+0.174 ±\pm0.024+0.012 ±\pm0.023+0.049 ±\pm0.021
Reactive Timely Correctness ↑\uparrow0.126 ±\pm0.0280.176 ±\pm0.0270.195 ±\pm0.0280.217 ±\pm0.031+0.050 ±\pm0.030+0.019 ±\pm0.020+0.022 ±\pm0.026
Proactive Timely Correctness ↑\uparrow0.063 ±\pm0.0260.259 ±\pm0.0440.273 ±\pm0.0440.354 ±\pm0.052+0.195 ±\pm0.045+0.015 ±\pm0.036+0.080 ±\pm0.043
Intervention Precision ↑\uparrow0.145 ±\pm0.0230.291 ±\pm0.0310.310 ±\pm0.0350.348 ±\pm0.038+0.145 ±\pm0.028+0.019 ±\pm0.026+0.038 ±\pm0.030

Table 2. Evidence ladder on WTaG. L0 retains native behavior; L1 enforces interventions at evidence onsets; L2 adds trigger types; and L3 adds specific evidence. Δ\Delta columns show differences (L1−L0, L2−L1, and L3−L2). Blue and orange shading highlight gains from onset control and specific evidence for proactive guidance, respectively.

Specific evidence improves proactive guidance. Coarse evidence at L2 provides limited gains, as it lacks concrete details about the ongoing task. In contrast, providing specific observations at L3 improves proactive Timely Correctness by 0.059 for JoyAI and 0.080 for Gemini. Gains in reactive guidance are smaller, as user utterances already provide relevant evidence. These results suggest that strong grounding in the ongoing task is key to effective proactive guidance.

FLUID-Agent Ablation

We ablate the two decision modules in Figure 3 to assess taxonomy-guided reactive and evidence-grounded proactive decisions. Direct Reactive replaces User Utterance Understanding with an LLM that directly decides whether and how to respond to each user utterance. Direct Proactive replaces Intervention Decision with an LLM that directly decides whether to speak. Both retain the same backbone, perception pipeline, and task memory (see Appendix § 9.4.§ 3 for details).

Taxonomy-guided understanding reduces unnecessary reactive responses. Compared with Direct Reactive, FLUID-Agent achieves comparable Reactive Timely Correctness on WTaG and higher correctness on HoloAssist, with higher Intervention Precision and fewer reactive utterances on both datasets. Direct Reactive responds more often to turns requiring no answer (see Response Rate on Unanswered Turns), supporting the role of taxonomy guidance in deciding when to respond and when to remain silent.

Evidence-conditioned intervention decisions enable more selective proactive guidance. Compared with Direct Proactive, FLUID-Agent achieves higher Proactive Intervention Precision with roughly 11 times fewer proactive utterances and substantially less speech overlap on both datasets. Although Direct Proactive achieves higher Proactive Timely Correctness, over 60% of its utterances repeat earlier responses (see Repetition Rate). These results show that structured intervention decisions enable more appropriate proactive guidance while reducing redundant and unnecessary speech.

Discussion and Conclusion

Recent advances in vision and language foundation models as well as agentic architecture have made it feasible to develop perceptual agents that can provide real-time task guidance to help people acquire new skills and complete physical tasks. However, evaluating perceptual guidance agents remains a significant challenge due to the inherent flexibility of instructional dialogue, where appropriate guidance can vary in content and timing.

To address this issue, we developed FLUID, a framework that evaluates not only the accuracy of generated guidance but also the flexibility of AI agents in providing different types of guidance based on users’ needs and task contexts. FLUID grounds its evaluation metrics in human instructor guidance, focusing on the evidence supporting the guidance, its necessity, and the appropriateness of its timing. Our current FLUID framework

contains instructional interaction data, fine-grained annotation and evaluation protocols. It can be used to support systematic and comparative evaluation of AI agents’ guidance capabilities. Our experimental results show that, while human instructors achieve high performance under FLUID metrics, state-of-the-art AI models still lag significantly behind. These findings highlight the substantial gap between perceptual agents’ capabilities and proficient human instruction.

For tasks that require specialized procedures, such as special daily tasks relevant to different people, FLUID provides a pipeline to establish evaluation metrics based on a few examples of natural interaction sessions between an instructor and a follower. FLUID can then provide an initial assessment of agent behavior and reliability before deploying agents in real-world user tasks.

Evaluating real-time guidance systems is an extremely challenging problem. Ultimately, these agents must be evaluated through real-time interactions with actual users. However, such evaluations are prohibitively expensive and difficult to scale. FLUID provides a way to bootstrap the development and evaluation of agents’ guidance capabilities before moving to real-world user studies. We hope FLUID will offer new perspectives on how to evaluate real-time guidance systems and motivate the development of perceptual guidance agents capable of providing flexible, contextually appropriate, and timely guidance comparable to that of skilled human instructors.

Limitations

Our annotations primarily recover the grounded evidence underlying observed instructor interventions, which provides a strong reference for evaluation but does not exhaust all potentially valid interaction behaviors. In particular, the absence of an intervention in the demonstration does not imply that no intervention would have been appropriate at that moment. We also do not explicitly model silent moments, partly because WTaG and HoloAssist contain few cases where speaking is clearly prohibited; many additional utterances could be non-harmful as long as they do not disrupt task progress. Identifying such unobserved but potentially valid intervention opportunities, as well as moments where silence is truly required, involves more subjective judgments and is left for future work.

Our framework supports a broad range of proactive and reactive assistance behaviors in complex task interactions. However, more fluid and dynamic interaction scenarios, such as full-duplex perception and generation, remain challenging, as overlapping perception and response generation are difficult to support naturally within a modular system. In the datasets, we do observe several dynamic interactive behaviors, including backchanneling, overlapping speech, and visual barge-ins, and some of these are partially reflected in our evaluation. However, a comprehensive characterization and evaluation of richer full-duplex interaction is beyond the scope of this work. Future work could study these behaviors more systematically, supported by interaction data with denser and more diverse back-and-forth dynamics.

Acknowledgments

This work was supported in part by the Johns Hopkins Artificial Intelligence and Technology Collaboratory (JH AITC), funded by the National Institute on Aging of the National Institutes of Health under grant P30AG073104. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Appendix

Dataset Construction and Preprocessing

WTaG and HoloAssist record task guidance in which a remote instructor observes the user’s view and provides verbal assistance. This appendix describes how sessions were selected from the two datasets, how their video, audio, and transcripts were prepared, and how the resulting sessions differ in their interaction.

Source Datasets and Session Selection

From WTaG, we exclude sessions with incomplete task executions. From HoloAssist, we select sessions with substantive user–instructor interaction, excluding those dominated by repetitive next-step instructions and acknowledgments, and then manually check the selected recordings for audio quality. Table 4 reports the final dataset sizes.

WTaGHoloAssist
Video hours7.576.29
Test sessions4372
User utterances2,3241,358
Instructor utterances1,4822,736

Table 4. Statistics of the two source datasets. HoloAssist instructor utterances are counted as native transcript rows; merging fragments of one contribution gives 2,731 units.

Multimodal Preprocessing and Synchronization

Video, audio, and annotation timestamps are measured in seconds from the start of each recording. Video is sampled at 8 Hz using presentation timestamps, taking the latest frame available at each sampling time.

For the transcripts, we manually revise the WTaG transcripts to correct speech boundaries and restore missing utterances, and keep HoloAssist’s released transcripts and timing. These speaker-labeled transcripts are used for annotation and evaluation; at runtime, user text comes from streaming automatic speech recognition (ASR) instead.

For the audio, instructor speech is muted, except where it overlaps user speech. There it is left in place to preserve the user’s signal, so the instructor may remain audible in overlap regions. Recordings are replayed continuously. Streaming ASR produces partial transcripts during an utterance and a final transcript once the utterance ends. These outputs become available only after the corresponding audio has been received and processed.

Dataset Statistics and Interaction Characteristics

HoloAssist sessions are shorter than WTaG sessions but contain denser instructor guidance. The median session duration is 265.8 s in HoloAssist and 594.0 s in WTaG, while instructors speak 8.25 and 3.21 turns per minute on average per session, with a median interval between adjacent instructor utterances of 1.99 s and 7.19 s, respectively.

The two datasets also differ in the type of guidance (Figure 5). HoloAssist has a higher proportion of instructions, while WTaG has higher proportions of answers, approvals, and corrections. This pattern is consistent with denser ongoing instruction in HoloAssist and more frequent answers and corrections in WTaG. Correction labels describe the instructor’s guidance, not the frequency of actual task errors.

Figure 5. Utterance taxonomy distributions in WTaG and HoloAssist.

The necessary interventions in the released annotations show a corresponding difference in how guidance is initiated. In WTaG, 66.1% are reactive and 33.9% proactive; in HoloAssist, 31.4% and 68.6%. WTaG therefore puts more weight on responding to users, while HoloAssist more often requires guidance that is not prompted by a user request.

Evidence-Grounded Annotation Protocol

The annotations connect each instructor intervention to the user speech, visible actions, and task states that support it. This appendix defines the four annotation layers, including the rules for intervention necessity and timing, then describes the annotation workflow and the review procedures for WTaG and HoloAssist, and closes with boundary cases and a worked example.

Annotation Overview and Label Schema

We annotate 43 WTaG and 72 HoloAssist test sessions with the four linked layers of § 3.3. Layer 1: Utterance Taxonomy assigns each turn a communicative-function label and at most one attribute. Layer 2: Action and State locates visible task evidence in time and records its validity and task effect. Layer 3: Trigger Link connects each instructor intervention to its supporting evidence and records its trigger type and intervention status. Layer 4: Evidence Onset and Intervention Window records the temporal anchors that define when the intervention should begin. Three additional WTaG sessions provide task knowledge only and are not annotated or evaluated.

The unit of annotation for speech is the utterance unit, one speaker contribution in the cleaned transcript. Fragments of one contribution that were split by speech recognition or transcript cleaning are merged before labeling. A contribution is split into several units only when its functions are clearly separable, for example There is already water in it. (U7) and Should I pour it out? (U2); otherwise it receives the label of its main communicative function.

Layer 1: Utterance Taxonomy

This layer labels the communicative function of each utterance, that is, what the speaker is asking, reporting, or responding to. Users and instructors have separate label sets, and labels are assigned from the transcript and the conversational context. For brief responses, the position of the turn in the interaction also matters, since it separates acknowledgments from backchannels (U5 vs. U10; I10 vs. I11).

User Utterance Taxonomy

Eleven labels describe user turns: requests, responses, task reports, and speech with little or no task-related content (Table 5). When more than one label could apply, annotators check the labeling rules in the order U10, U9, U11, U8, U3, U2, U1, U4, U5, U7, U6 and use the first one that fits. An incomplete utterance that clearly asks a question is not labeled U9 but receives one of the three question labels: U3 asks about something the instructor recently said, U2 proposes an answer about the task or the world and asks whether it is correct, and U1 asks for information without proposing an answer. Appendix § 9.2.7.1 explains how these labels are separated when the interpretation depends on context.

IDLabel, definition, and example
U1Information request. An open request for task content the user lacks; no candidate answer is proposed. Umm, how much oil?
U2Verification question. Proposes a candidate object, amount, action, plan, interpretation, state, or result and requests validation, as a question or as a declarative awaiting confirmation. It is a teaspoon, right?
U3Clarification request. Requests repetition, rephrasing, or confirmation of the user’s hearing or understanding of an instructor utterance. What do you mean by pinch, sorry?
U4Answer. Supplies information requested by an instructor question, state check, or check instruction. Five.
U5Acknowledgment/uptake. Signals receipt, understanding, or acceptance of a completed instructor turn without new task content, including an echo the user closes themselves. Twelve ounces? Okay.
U6Task narration. Propositional verbalization of the user’s own action, plan, reasoning, need, decision, or progress, including reading instructions aloud. I already added the sugar.
U7Trouble report. Reports that something is wrong or has deviated from the expected course. Wait, that wasn’t a tablespoon.
U8Affective expression/social talk. Emotion, evaluation, politeness, apology, laughter, or conversation not required for task completion. Oh my God.
U9Filler/hesitation. Non-propositional or self-directed speech: fillers, false starts, fragments, muttered reading, self-directed pacing tokens, or unintelligible speech. And, uh. . .
U10Backchanneling. A brief continuer produced while the instructor is still speaking. Mm-hm.
U11Meta-communication. Discusses or manages the interaction or session itself, such as protocol, recording, apparatus, or audio and camera tests. Can you still hear me?

Table 5. User utterance taxonomy. Examples illustrate a label but do not replace the context-dependent rules in Appendix § 9.2.7.1.

Instructor Utterance Taxonomy

Thirteen labels describe instructor turns (Table 6). They record what the instructor is doing in the conversation, regardless of whether the content is correct. When more than one label could apply, annotators check the labeling rules in the order I12, I13, I8, I5, I9, I6, I2, I1, I4, I3, I7, I11, I10 and use the first one that fits. A correction is therefore labeled as a correction even if it also expresses approval or answers a question, and information given in response to a user request is labeled as an answer rather than as an instruction offered without a request. For turns with little substantive content, the label depends on timing: I11 while the user is still speaking or performing an action, and I10 after the user has finished.

IDLabel, definition, and example
I1Answer. Supplies task content the user requested explicitly or implicitly, through a question, clarification request, stated need, trouble report, or audible search. The filter cone is the metal thing.
I2Approval. Validates a concrete user candidate, action, state, or result as correct or acceptable. Yep, like that.
I3Instruction. Introduces or elaborates a required task step on the instructor’s own initiative, without presupposing an error. Grind it for about twenty seconds.
I4Advisory. Gives optional, cautionary, conditional, or alternative guidance that does not repair an existing error. You can use the glass bowl if that is easier.
I5Question-to-user. Requests information about the task state, the user’s intent, the user’s speech, or the user’s understanding. How many toothpicks do you have?
I6Reassurance. Mitigates user worry, trouble, or a mishap by marking it as tolerable or recoverable. That is okay, no worries.
I7Pacing/closure. Regulates starting, continuing, waiting, transitioning, or ending without substantive procedural content. Keep going.
I8Environmental repair. Asks the user to improve the instructor’s incoming visual or audio evidence. Could you look down more?
I9Correction. Repairs an actual, proposed, imminent, or completed user error or an omitted step. No, use the jelly instead of the marmalade.
I10Acknowledgment. Signals receipt of a completed user contribution without information, evaluation, reassurance, or flow direction. Oh, okay.
I11Backchanneling. A brief, low-content signal of attention while the user is still speaking or performing an extended action. Mm-hm.
I12Filler/self-repair. Hesitation, an abandoned start, or a repair of the instructor’s own wording or instruction. Sorry, I meant one teaspoon.
I13Other. Meta-communication, session or recording management, connection checks, social talk, or rare residual content. You can stop the recording.

Table 6. Instructor utterance taxonomy.

Utterance Attributes

An attribute records what a turn is about, while the function label records what the turn does. Each turn takes at most one category-specific attribute (Table 7); U4, U5, U9–U11, I6, I10, and I13 take none. U2, I1–I4, and I9 share content_target, whose values are object_tool, action_method, quantity_parameter, sequence_step, state_result, and other_unclear, so that a verification and its response can be compared on the same content axis. Figure 2 shows the label and attribute together, such as Verification—quantity.

Layer 2: Action and State

Annotators identify actions and task states from visible evidence in the video, using the dialogue as additional context when needed, and mark the start and end time of each. Each interval is written as [tstart,tend)[t_{\mathrm{start}}, t_{\mathrm{end}}), which includes the start time and excludes the end time. The interval of an action is called an action span, and each span also receives a validity label and a task-effect label, defined below. Table 8 lists the fields recorded for each WTaG annotation.

FieldValues or boundary rule
Unit kindaction (motion directed toward a task goal) or state_event (a task condition or a change in that condition).
TimesStart: the first frame that clearly shows the action or state. End: the first frame after it ends or changes identity.
Verb/nounA short description supported by the video, e.g., grab–fork.
ValidityVALID, DEVIATION, AMBIGUOUS.
Task effectNO_PROGRESS, PROGRESS, COMPLETION, DEVIATION_ONSET, RECOVERY, UNKNOWN.
State labelFor state_event units: STALL, SEARCHING, VISIBILITY_LOSS, VISIBILITY_RECOVERY, SALIENT_STATE_CHANGE.

Table 8. Fields used to annotate actions and task states in WTaG.

An action span covers one continuous attempt to achieve a task goal. It may include several hand movements as long as the goal, object, and validity stay the same; a new span starts when any of these changes, or when the user begins another attempt after disengaging their hand or resetting the task state. If an action is briefly occluded, the span continues when continuous motion lets us identify it across the occlusion. If the action cannot be identified reliably, the uncertain interval is left unannotated.

Validity indicates whether the observed behavior is appropriate given the task instructions and the current state: VALID when it follows these requirements, DEVIATION when it conflicts with a task goal or constraint, and AMBIGUOUS when the available evidence is insufficient to decide.

LabelsAttribute and focus
U1request_type: information requested.
U2, I1–I4, I9content_target: task content.
U3repair_target: hearing, meaning, or task detail.
U6narration_type: action, progress, reasoning, or reading aloud.
U7trouble_type: problem reported.
U8expression_type: affective, task-related, or social talk.
I5question_purpose: information sought.
I7flow_function: starting, continuing, waiting, transition, or closure.
I8repair_target: visual or audio access.
I11backchannel_target: speech, action, progress, or affect/trouble.
I12repair_form: filler or self-repair.

Table 7. Utterance attributes. The two uses of repair_target have different value sets.

Task effect records how the action changes the task state. PROGRESS marks an action that advances a step or subgoal, and COMPLETION one that completes a step or subgoal or reaches a meaningful task state. DEVIATION_ONSET identifies the action that first creates an erroneous or undesirable state, and RECOVERY an action that begins or completes the correction of that state. NO_PROGRESS marks an action with no substantive advancement, and UNKNOWN one whose effect cannot be determined reliably. A span can carry several task-effect labels if the task state changes during the same continuous action, for example when an action first makes progress and then completes a step.

Action Vocabulary

WTaG action descriptions use task-specific vocabularies developed from the recipes and checked against the videos. Qwen3-VL-30B-A3B-Instruct generated initial verb, noun, and adjective lists from the task steps, their execution variants, and the 48 HoloAssist verbs as a reference, and then checked sampled frames from 506 video segments for missing words. Annotators reviewed the words supported by at least two segments and merged synonyms. Compound objects are represented by a noun and an adjective, such as jar with nut-butter; locations and hand use are recorded separately. A final video check with Qwen3-VL-32B-Instruct added remove and shake and found no missing nouns. The resulting vocabulary contains 83 verbs, 120 nouns, and 107 adjectives. Observations outside the vocabulary are retained as out-of-vocabulary descriptions rather than forced into an unsuitable category.

Action-Candidate Generation and Human Review

WTaG action annotations combine model proposals with a human scan of the full task timeline. Four models independently propose intervals and descriptions: Qwen3.5-27B, Qwen3.6-35B-A3B, Qwen3.8-27B, and Qwen3-VL-32B-Instruct. Temporally clustered proposals supported by at least two models enter review, including incompatible proposals that meet this threshold; the votes decide only which candidates are reviewed, not their validity or boundaries. Reviewers resolve the candidates using the video and task context and add manual pins for missed events, which follow the same review procedure. Any candidate may be accepted, edited, merged, split, or rejected: of 2,611 model proposals, 665 were kept unchanged, 1,423 were kept after editing, and 523 were rejected. After this review, a separate model-based pass assigns validity and task effect to the reviewed actions using the task context and the annotation guidelines.

For every instructor utterance, the annotator selects the smallest set of preceding evidence that warrants the intervention. A link records the instructor utterance ID, the evidence source, one or more evidence IDs, the trigger type, a confidence, and optional ambiguity flags; Table 9 gives the controlled values.

FieldControlled values
SourceUSER_UTTERANCE, USER_ACTION_STATE, MIXED, INSTRUCTOR_SELF, NONE_FOUND.
Trigger typeUSER_QUERY, USER_REPORT, ERROR, PROGRESS_OK, STEP_DONE, STUCK, RECOVERY, VISIBILITY, UNCLEAR_STATE, DIALOG_OTHER, SELF, NONE.
Confidencelow, medium, high.
Audit flagsunclear_source, competing_evidence, missing_evidence, boundary_uncertain.

Table 9. Trigger-link schema. Confidence concerns the evidence identity and relation, not transcription quality.

The source is MIXED only when neither modality alone is sufficient. For example, Does that look good? asked while the user is rolling the tortilla links the instructor’s Yeah to both the U2 turn and the visible action. An intervention is reactive if

its trigger evidence includes a completed user utterance (USER_UTTERANCE or MIXED) and proactive otherwise, so this intervention is reactive. Whether an intervention is reactive is determined by its trigger link, not its utterance label: an I1 Answer is proactive when it responds to a visible search without a user turn, such as searching for a half-teaspoon.

Intervention Necessity

An intervention is necessary when the user needs it to avoid a mistake, get unstuck, or obtain needed information; optional when it can be omitted while the user proceeds correctly; and excluded when it is task-irrelevant or requires knowledge outside the shared view, the dialogue, or the task knowledge. Table 10 assigns these statuses from the trigger type, the linked user label, and the instructor label. When a link has several trigger types, the highest status applies, in the order necessary > optional > excluded. Only necessary interventions are scored by Correctness and Timely Correctness; optional interventions take part in matching (§ 4.1) but are not scored.

Trigger typeConditionStatus
AnyRelies on knowledge that cannot be obtained from the shared egocentric view, the dialogue, or the task knowledgeExcluded
USER_QUERY, ERROR, UNCLEAR_STATE, VISIBILITY, STUCKNoneNecessary
AnyLinked user utterance is U1, U2, U3, or U7Necessary
STEP_DONE, PROGRESS_OK, RECOVERY, USER_REPORTInstructor label is I1, I3, I8, or I9Necessary
STEP_DONE, PROGRESS_OK, RECOVERY, USER_REPORTAny other instructor labelOptional
DIALOG_OTHERNoneOptional
NONE, SELFNoneExcluded

Table 10. Rules for intervention status.

For example, Yep in response to Twelve ounce right? (U2) is necessary, while Great after a correct pour (PROGRESS_OK, I2) is optional (Figure 2). On WTaG, the rules assign 1,158 of 1,482 statuses (78.1%), 121 model proposals are retained after review (8.2%), and annotators set 203 (13.7%); every status is reviewed. On HoloAssist, Qwen3.8-27B assigns 2,492 of 2,736 statuses (91.1%), the rules assign 235 (8.6%), and annotators set 9 (0.3%).

Layer 4: Evidence Onset and Intervention Window

Temporal anchors define when the target instructor intervention can begin. They are annotated against a streaming prefix: evidence after time tt may confirm that a condition continues and may identify a later deadline, but may not be used to claim that enough evidence was already available at tt. In the main text, ES, EO, LU, and R are written tearlyt_{\mathrm{early}}, tevidt_{\mathrm{evid}}, tlatestt_{\mathrm{latest}}, and tusrt_{\mathrm{usr}}.

Action-Triggered Interventions. Proactive interventions use three anchors. Evidence Start (ES) is the earliest frame showing a specific sign that the intervention may be needed, even if the condition is not yet certain; entering a task step alone does not establish ES. Evidence Onset (EO) is the earliest frame that makes the condition unambiguous and complete enough to justify the intervention; the whole object need not be visible, but its identity and task relation must be supported. Latest Useful Time (LU) is the exclusive boundary after which the target intervention is no longer useful on the recorded trajectory. The valid window is [ES,LU)[\mathrm{ES}, \mathrm{LU}), with the timing score peaking at EO (§ 4.2).

Utterance-Triggered Interventions. For a reactive intervention, including one with MIXED evidence, the first eligible time is the Response-Ready Time (R), the end of the completed user turn in the trigger evidence; ES and EO are not separately annotated for utterance-triggered interventions. Partial speech does not set R unless the utterance is already semantically complete and only a delayed transcription endpoint remains. The reactive window is [R,LU)[\mathrm{R}, \mathrm{LU}).

Operational Definition of LU. LU is the earlier of task expiry and dialogue cutoff on the recorded trajectory:

LU=min⁡(task-expiry,dialogue-cutoff).\mathrm{LU} = \min(\text{task-expiry}, \text{dialogue-cutoff}).

Task expiry occurs when the user completes the relevant action, starts self-correction or recovery, or reaches a state in which the intervention would have to be reformulated. Dialogue cutoff is the start of the first acknowledgment, answer, or downstream action that presupposes the target intervention, since inserting the intervention later would reverse the recorded conversational order. Unrelated speech does not set LU. For self-correction, LU marks its start rather than its completion.

Frame Selection and Precision. Annotators scrub coarsely, then step frame by frame around each boundary and select the earliest frame that satisfies the definition. Stored timestamps are in seconds at the source video’s frame resolution and are not rounded to whole seconds.

Annotation Workflow and Quality Control

Annotation proceeds layer by layer. Annotators first review segmentation, transcripts, and utterance labels, then action candidates and manual pins, and finally trigger links, intervention statuses, and temporal anchors. Completed upstream layers stay visible during review, and errors found in them are resolved with the annotator responsible. The three annotators are authors and received no additional compensation. They developed the rubrics together, annotated a shared session, and discussed disagreements before annotation began, and guideline changes led to re-review of the affected annotations. Review coverage differs between the two datasets (Table 11).

Before release, schema, ordering, and reference checks are run and the flagged cases are resolved. The automatic checks reject unknown controlled values, missing referenced evidence, non-positive intervals, and impossible timing orders: proactive cases require ES ≤\leq EO and ES << LU, and reactive cases require R << LU. Human-audit flags cover competing triggers, missing evidence, and uncertain boundaries.

LayerWTaG (43 sessions)HoloAssist (72 sessions)
1. Utterance TaxonomyQwen3.6-35B-A3B; all reviewedConsensus of four modelsa; audited
2. Action and StateFour modelsa, ≥\ge2 votes, plus manual pins; all reviewedOriginal dataset actions and model-proposed state events; not reviewed
3. Trigger LinkLinks: Qwen3.8-27B. Statuses: rules (Table 10) and annotators. All reviewed.Links: merged proposals of four modelsa. Statuses: Qwen3.8-27B and rules. Audited.
4. Evidence Onset and Intervention WindowModel-proposed; all reviewedModel-proposed, as for WTaG; audited

Table 11. Annotation provenance for the four layers. aQwen3.5-27B, Qwen3.6-35B-A3B, Qwen3.8-27B, and Qwen3-VL-32B-Instruct. “Audited”: reviewed in eight sessions after release (Appendix § 9.2.§ 6.1).

Review Coverage and Audit

All layers of all 43 WTaG test sessions were manually reviewed. Figure 6 summarizes the changes to the WTaG drafts and to action/state boundaries: annotators adjusted 68.1% of the proposed ES anchors, 78.6% of the EO anchors, and 56.1% of the LU anchors, with median absolute shifts of 2.95 s, 1.46 s, and 0.95 s.

Figure 6. WTaG manual review. Top: review outcome of the AI proposals in the 43 test sessions; nn is the number of proposals per layer. Bottom: shift of the edited start and end times of the retained action and state proposals (10th, 25th, 50th, 75th, and 90th percentiles).

For HoloAssist, the released annotations were frozen before the experiments, and the audit is a quality check of that frozen release rather than a revision of it. After the freeze, we manually checked 399 utterances across eight test sessions, 9.75% of the released utterances, on all annotation layers except actions. Figure 7 reports the share of audited items that the auditors changed on each layer. All reported experiments use the released annotations as frozen; the corrections identified in this audit are not applied to them, so every system is scored against the same annotations.

Figure 7. HoloAssist audit. Annotation changes in the eight-session audit. nn is 399 utterances for labels and attributes and 301 trigger links for the other fields.

Release and Reproducibility Notes

Every utterance, action/state, trigger link, and timing record has a stable ID, and records refer to one another by these IDs rather than by matching transcript text. To reproduce or extend the annotations, follow the controlled values and boundary rules above, and leave uncertain cases explicitly marked, for example with AMBIGUOUS validity or an audit flag, rather than assigning a confident label.

Annotation Examples and Boundary Cases

This section first lists the label distinctions that depend on conversational context and then follows one intervention through all four layers.

Utterance-Label Boundary Cases

A turn’s role in the exchange determines its label, even when its wording is the same. The rules below separate requests, uptake, and task feedback.

U1 versus U2. Use U2 only when the user proposes a concrete candidate that can be affirmed or rejected; otherwise use U1. Thus Which spoon should I use? is U1, whereas Should I use this spoon? is U2. A declarative such as That should be enough is U2 when it functions as a bid for validation. U2 versus U3. Use U3 when the object of the request is the instructor’s recent words themselves; a question about the task world is U2 even if it follows an instruction. What do you mean by pinch? is U3 with repair_target=meaning, and Thirty seconds, you mean? is U3 with task_detail, whereas a delayed check of remembered content, I think you said it was a teaspoon earlier, right?, is U2.

U4, U5, U9, and U10. Okay or yeah is U4 when it answers an instructor question, U5 when it takes up a completed instructor turn, U10 when it occurs while the instructor is still speaking, and U9 when it is self-directed. An echo is U5 when the user closes it (Twelve ounces? Okay.) and U3 when the user waits for confirmation (Twelve ounces?).

I1, I2, I3, and I9. I1 supplies content requested explicitly or implicitly by the user; the same content is I3 when volunteered as a required step. For a proposed candidate, I2 accepts it and I9 rejects or replaces it. Thus The teaspoon answers Which spoon? as I1, while Yes after This teaspoon? is I2 and No, use a teaspoon after Should I use a spoon? is I9.

I3 versus I4 versus I9. I3 is required guidance that presupposes no error; I4 is optional or cautionary guidance with no existing error (Be careful not to add too much salt); I9 repairs an actual, proposed, imminent, or omitted error (You added too much salt; remove some). Modal wording such as should, can, or might want to does not determine the label.

I10 and I11 versus Task Feedback. A receipt token such as okay is I10 after a completed user contribution and I11 during ongoing speech or action; if it evaluates correctness, it is I2. For example, Okay after the user answers Five is I10, while Yes after Is that correct? is I2.

End-to-End Four-Layer Example

Table 12 follows a wrong-utensil correction in session T2 through the four layers. The first sign of the error appears at 81.20 s (ES), and the fork becomes identifiable at 82.84 s (EO). The instructor starts the correction at 83.35 s. At 85.38 s, the user starts a compliant reach toward the knife, which sets the dialogue cutoff and therefore LU while the instructor is still speaking. The knife-grab span itself starts only at 85.54 s, when the grasped object becomes identifiable, but LU marks the start of the reach. LU can precede the end of the instructor turn because it concerns the usefulness of the intervention’s speech start time.

LayerTime (s)Annotation
1. Utterance Taxonomy83.35–87.85You should use the knife instead of the fork; I9 Correction; content_target=object_tool.
2. Action and State81.20–83.80; 85.54–87.83grab–fork; validity DEVIATION; reviewed action a:356. The subsequent grab–knife action follows the correction.
3. Trigger Linkaction to utteranceUSER_ACTION_STATE; ERROR; high confidence; no ambiguity flag; necessary.
4. Evidence Onset and Intervention WindowES 81.20; EO 82.84; LU 85.38First sign of a wrong-utensil choice; earliest unambiguous fork evidence; start of the compliant reach toward the knife (dialogue cutoff).

Table 12. Complete four-layer example from session T2 (intervention T2_R0005).

Evaluation Protocol and Metrics

This appendix specifies the evaluation of § 4: the teacher-forced protocol and its real-time execution, intervention matching, the metrics, the LLM judges, and the statistical analysis. The annotations it reads are defined in Appendix § 9.2.

Evaluation Regime and Information Availability

Every system is evaluated under a teacher-forced setting: the interaction history it conditions on contains the human instructor’s utterances, released by the fixed rule of Appendix § 9.3.§ 2, while its own utterances are recorded for scoring and do not re-enter its context. This isolates each decision from earlier system errors and gives every system the same instructor history. The user side of the history is what the system itself perceives: the shared streaming-ASR transcript, or the user audio for Gemini (Appendix § 10.§ 1).

Teacher-Forced History Release Protocol

The nn-th instructor utterance, spoken over [tn0,tn1][t_n^0,t_n^1], enters the history at

release⁡n=max⁡(tn1,tnlatest),(2)\operatorname{release}_n = \max\left(t_n^1, t_n^{\mathrm{latest}}\right), \tag*{(2)}

where tnlatestt_n^{\mathrm{latest}} is the annotated latest useful time of the intervention it realizes (Appendix § 9.2.§ 5). An utterance therefore becomes visible only after it has been fully spoken and the opportunity it addresses has closed. An utterance without an annotated window, such as an excluded intervention, enters at tn1t_n^1, and release times are made non-decreasing in transcript order, so the history always follows the transcript order. A decision made at time tt sees utterance nn if and only if release⁡n≤t\operatorname{release}_n \le t. How each baseline inserts released utterances into its history is described in Appendix § 10.§ 2.

Real-Time Execution and Spoken Response Timing

Each system processes each session in real time. The locally served systems (ProAssist, VideoLLM, JoyAI, and FLUID-Agent) run on one accelerator each with batch size one: a model call starts when it is issued or when the system’s previous call has finished, whichever is later, and occupies the accelerator for its measured duration, so queueing is part of every latency. Perception, decision, generation, speech synthesis, and sequential playback run on this clock; frames become available at their capture time and final ASR transcripts at their recorded availability time. Gemini is accessed through the vendor-hosted Live API: its audio and video are streamed at wall-clock speed, a response becomes audible when its first audio arrives, and the time spent rebuilding a session (Appendix § 10.§ 2) does not advance the replayed session timeline.

The evaluation reads every played system utterance pp with its audible onset t(p)t(p), its playback end e(p)e(p), and the last instructor utterance in the system’s history when pp was decided; utterances that were queued but never played are not candidates. Timing is read at the audible onset, as the user experiences it.

Intervention Event Construction and Matching

Events and Windows. An intervention event is an instructor utterance of the test set whose intervention status is necessary or optional; its side, reactive or proactive, is defined in Appendix § 9.2.§ 4. For event nn the annotation gives the start unu_n and end tnusrt_n^{\mathrm{usr}} of the triggering user turn (reactive), the earliest sign tnearlyt_n^{\mathrm{early}} and the evidence onset tnevidt_n^{\mathrm{evid}} (proactive; tnearly=tnevidt_n^{\mathrm{early}} = t_n^{\mathrm{evid}} when no earlier sign is annotated), and the latest useful time tnlatestt_n^{\mathrm{latest}}. The intervention window is

Wn={[tnusr,tnlatest)reactive,[tnearly,tnlatest)proactive.(3)W_n = \begin{cases} [t_n^{\mathrm{usr}}, t_n^{\mathrm{latest}}) & \text{reactive},\\ [t_n^{\mathrm{early}}, t_n^{\mathrm{latest}}) & \text{proactive}. \end{cases} \tag*{(3)}

Candidates for event nn are the played utterances whose audible onset falls in the matching window Mn=[Ln,Un]M_n = [L_n, U_n], which extends WnW_n to the surrounding interval of adjacent reference utterances:

rounding interval of adjacent reference utterances:

L_n = \begin{cases} u_n & \text{reactive},\\ \min(t_n^{1-}, t_n^{\mathrm{early}}_n) & \text{proactive}, \end{cases} \tag*{(4)}
Un=max⁡(tnlatest,min⁡(tn0+,on+)),(5)U_n = \max(t_n^{\mathrm{latest}}, \min(t_n^{0+}, o_n^{+})), \tag*{(5)}

where tn1−t_n^{1-} is the end of the previous instructor utterance, tn0+t_n^{0+} the start of the next one, and on+o_n^{+} the opening time of the next necessary event of the session, of either side (uu for a reactive and tearlyt^{\mathrm{early}} for a proactive event); the session boundaries stand in when no such utterance or event exists. A candidate is admissible for nn if it was decided before the reference entered the system’s history, that is, if the last instructor utterance in its history precedes utterance nn.

Grading and Claiming. Each admissible candidate–event pair is graded once by the event judge of the event’s side (Appendix § 9.3.8) on the scale of Table 13. A pair is a content match at grade ≥4\ge4; a grade-3 pair neither matches nor blocks a later match. Events are processed in the order of the instructor utterances, optional events included, and each claims the earliest still-unclaimed matching candidate by audible onset, regardless of timing. An utterance can satisfy at most one event. The metrics are read from the necessary events.

GradeMeaningUse
5The same guidance effect: the user would do and understand what the reference leads them to.Match; F1 credit 1
4The same effect with a non-critical difference, such as an extra supported detail or a supported alternative.Match; F1 credit 0.75
3Partly meets the need: something still needed is missing, or an unsupported claim could change what the user does.No match; partial
2Does not address this need: another turn, step, or problem, or narration where the reference guides.No match; off-target
1Works against this need: the user would do the opposite of what the reference directs.No match; contradiction

Table 13. Grading scale shared by the three judges.

Reference Fidelity Metrics

Similarity F1 and LLM-Judged F1 follow Zhang et al. [32] and are computed over all instructor utterances of the test set. A played utterance pp may pair with a reference rr if t(p)−t0(r)∈[−6,+3]t(p)-t^{0}(r) \in[-6,+3] s and pp is admissible for rr. For Similarity F1, texts are encoded with the all-mpnet-base-v2 sentence encoder, pairs are assigned one-to-one by a cost that combines the time offset with the cosine distance, and an assigned pair is correct when the cosine similarity exceeds 0.5. For LLM-Judged F1, pairs are pre-filtered to each reference’s and each candidate’s four most similar partners, graded by the content judge (Appendix § 9.3.8), and assigned one-to-one at grade ≥4\ge4 with credit 1 (grade 5) or 0.75 (grade 4) so as to maximize total credit. Precision divides the correct pairs (or credit) by the number of played utterances and recall by the number of references.

Timely Correctness and Response Latency

Timely Correctness. Timely Correctness assigns a temporal score to each necessary intervention and averages over all of them,

1N∑n=1Nsn,(6)\frac{1}{N}\sum_{n=1}^{N}s_n, \tag*{(6)}
sn={Tn(tn),if intervention n is correct,0,otherwise,s_n = \begin{cases} T_n(t_n), & \text{if intervention } n \text{ is correct},\\ 0, & \text{otherwise,} \end{cases}

where NN is the number of necessary interventions of the side (reactive or proactive) and tnt_n is the audible onset of the claimed utterance. Intervention nn is correct when it claims an utterance (Appendix § 9.3.§ 4) whose audible onset falls in its intervention window WnW_n; necessary interventions that are unmatched, incorrectly matched, or delivered outside their valid timing windows receive zero. The preferred response time is tusrt_{\mathrm{usr}} for a reactive intervention and tevidt_{\mathrm{evid}} for a proactive intervention. With the normalized positions

uE=tevid−ttevid−tearly,uL=t−tevidtlatest−tevid,u_E = \frac{t_{\mathrm{evid}}-t}{t_{\mathrm{evid}}-t_{\mathrm{early}}}, \qquad u_L = \frac{t-t_{\mathrm{evid}}}{t_{\mathrm{latest}}-t_{\mathrm{evid}}},
uR=t−tusrtlatest−tusr,(7)u_R = \frac{t-t_{\mathrm{usr}}}{t_{\mathrm{latest}}-t_{\mathrm{usr}},} \tag*{(7)}

the temporal function of a proactive intervention is

Tnpro(t)={1−λEuE2,t∈[tearly,tevid),cos⁡pL(π2uL),t∈[tevid,tlatest),0,otherwise,(8)T_n^{\mathrm{pro}}(t) = \begin{cases} 1-\lambda_E u_E^2, & t\in[t_{\mathrm{early}},t_{\mathrm{evid}}),\\ \cos^{p_L}\left(\frac{\pi}{2}u_L\right), & t\in[t_{\mathrm{evid}},t_{\mathrm{latest}}),\\ 0, & \text{otherwise,} \end{cases} \tag*{(8)}

and that of a reactive intervention is

Tnrea(t)={cos⁡pL(π2uR),t∈[tusr,tlatest),0,otherwise.(9)T_n^{\mathrm{rea}}(t) = \begin{cases} \cos^{p_L}\left(\frac{\pi}{2}u_R\right), & t\in[t_{\mathrm{usr}},t_{\mathrm{latest}}),\\ 0, & \text{otherwise.} \end{cases} \tag*{(9)}

We use (λE,pL)=(0.7,0.5)(\lambda_E,p_L)=(0.7,0.5), calibrated in Appendix § 10.4. With these values, the proactive score starts at 0.3 at the earliest sign, peaks at the evidence onset, and is zero at the latest useful time, and the exponent pLp_L allows moderately delayed responses to retain partial credit. If tearly=tevidt_{\mathrm{early}}=t_{\mathrm{evid}}, the early branch is empty.

Correctness, reported in Figure 4, is the share of necessary interventions that are correct, that is, Eq. (6) with Tn≡1T_n\equiv1.

Response Latency. Response Latency is the median of tn−tusrt_n-t_{\mathrm{usr}} over the reactive necessary interventions whose claimed utterance begins at or after the end of the user’s turn. It is conditional on answering, so a system that answers few turns can still have a low latency, and its interval can be wide: Gemini’s HoloAssist latency in Table 1 is a median over 11 content-matched reactive responses from eight sessions.

Intervention Precision and Turn-Taking

Intervention Precision. Let PP be the played non-empty utterances of a run and P⋆⊆PP^\star\subseteq P the utterances that are the claimed response of some necessary or optional event nn and whose audible onset lies in WnW_n. Intervention Precision is ∣P⋆∣/∣P∣|P^\star|/|P|. Utterances that match no event, repeat a matched one, contradict the reference, or start outside the window lower the score. Optional events count because answering one constitutes a valid intervention, while leaving one unanswered costs nothing.

Speech Overlap Ratio. With Uˉ\bar{U} the union of the session’s user speech intervals, the same for every system, and S=⋃p∈P[t(p),e(p))S=\bigcup_{p\in P}[t(p),e(p)), the ratio is ∣S∩Uˉ∣/∣Uˉ∣|S\cap\bar{U}|/|\bar{U}|. The user audio is replayed and does not react to the system, so every overlap is attributed to the system.

LLM Judges and Evaluation Context

Configuration. Content agreement is the one part of the evaluation decided by a model; admissibility, claiming, timing, overlap, and aggregation are deterministic. One self-hosted Qwen3.8-27B model (bf16, temperature 0) serves every judge call for every system and run. Each call grades one candidate–reference pair from 1 to 5 with a short reason; only the grade enters a metric, and an unparsable output counts as no match. The complete judge prompts are provided in the released code repository.

Three Judges, One Scale. The reactive event judge, the proactive event judge, and the *content judge share the grading scale of Table 13, the context rules, and the output format. Each asks whether, after hearing the candidate, the user would do and understand the same thing as after the reference, where the user’s need at that moment is read from the trigger, the preceding dialogue, and the task knowledge rather than from the reference’s wording. For the reactive event judge the user’s turn sets the need; for the proactive event judge the observed action or state sets it, so that, for example, approving an action the instructor corrects is a contradiction. The content judge, used for LLM-Judged F1, receives no trigger and grades every reference utterance by its communicative function first, so a reference with no task content can be missed but not contradicted. A candidate that differs from the reference is credited when the task knowledge supports it and it leads to the same next action or understanding.

Judge Context. The judges read the benchmark’s ground-truth transcript, not the streaming-ASR transcript or audio that the evaluated system perceived. Every call contains the task knowledge text that every system received (Appendix § 10.§ 1), the event context (the benchmark dialogue before the reference, with the event’s own turn or observation marked), the event, the reference, the candidate context, and the candidate; the content judge receives the same message without the event block. The candidate context is the benchmark transcript cut at the candidate’s input cutoff, the start of the model call that produced it: the user turns that had started by then, the instructor utterances released into the system’s history by then, and the annotated user action in progress at that moment together with the last completed one. A user turn that was still in progress at the cutoff is given in full, so the candidate context can include the end of a turn that the system had not yet heard; it approximates what the system had available rather than reproducing it. It is kept separate from the event context, which may extend past the cutoff, so that the candidate is interpreted in the situation in which it was produced: a reply such as “yes” means what it answered at that time, and a candidate produced before the event occurred is off-target rather than a contradiction. A judge sees no timestamps, trigger types, necessity labels, windows, or system identity.

Aggregation and Statistical Analysis

Every metric is pooled over the test sessions of a dataset: event metrics over events, Intervention Precision over played utterances, the Speech Overlap Ratio over summed user speaking time, the F1 scores from pooled precision and recall, and Response Latency as the pooled median. Every reported value carries a 95% session-cluster bootstrap interval (10,000 resamples of sessions; tables print the half-width of the 2.5–97.5 percentile interval).

ProAssist, VideoLLM, JoyAI, and FLUID-Agent were each run twice under the same configuration and the reported values are the mean of the two runs; Gemini and the two ablation variants of Table 3 were run once. A difference between two systems or variants is computed on identical session sets as the pooled difference with a paired session-cluster bootstrap interval.

WTaG FLUID-AgentWTaG Direct ReactiveHoloAssist FLUID-AgentHoloAssist Direct Reactive
(a) Utterance Understanding
Reactive Timely Correctness ↑\uparrow0.2090.2100.0550.038
Reactive Intervention Precision ↑\uparrow0.2240.1580.1580.141
Reactive Utterances/min1.942.990.860.96
Response Rate on Unanswered Turns ↓\downarrow0.2360.4260.3160.405
Speech Overlap Ratio ↓\downarrow0.2070.2940.1530.168
(b) Intervention Decision
Proactive Timely Correctness ↑\uparrow0.0460.1190.0620.171
Proactive Intervention Precision ↑\uparrow0.0720.0200.1710.049
Proactive Utterances/min1.3214.831.5516.88
Repetition Rate ↓\downarrow0.0950.6190.0890.684
Speech Overlap Ratio ↓\downarrow0.2070.7520.1530.799

Table 3. Ablation of FLUID-Agent decision modules.

FLUID-Agent Implementation

This appendix describes FLUID-Agent (§ 5, Figure 3) as run in every experiment, with one configuration on both datasets. A single frozen Qwen3.6-35B-A3B vision-language model (FP8 weights, vLLM, temperature 0) serves every online role through task-specific prompting: it perceives the scene, judges the task state, interprets and answers user turns, and generates spoken intervention utterances. Deterministic code around the model maintains the shared task memory, schedules the calls, evaluates the evidence conditions under which an intervention is opened, and manages turn-taking. The model therefore supplies the evidence, and the agentic structure decides what that evidence licenses.

Session Loop. When a visual change is detected, the perception module analyzes the video and adds an observation report to the task memory. The Situation Tracker (the task-state tracking part of the Task Memory block in Figure 3) reassesses the task state when the observed activity or the dialogue has changed, and the intervention conditions are re-evaluated after each judgment. In parallel, each completed user turn is handled by one call that interprets it and, when a reply is due, answers in the same model call. Under the teacher-forced protocol (Appendix § 9.3.§ 2), released instructor utterances are visible to perception, to the Tracker as testimony of what the user was told, and to utterance understanding; the system’s own outputs do not enter its prompts.

Perception and Task-State Tracking

Task Knowledge Compilation. An offline compiler (Qwen3.8-27B, distinct from the online backbone) reads the same task knowledge that every baseline receives (Appendix § 10.§ 1) and produces an action-level plan: each step becomes a few imperative actions with the tools and destinations the step names, its stated quantities, an observable completion check where the knowledge states one, and the step’s prerequisite relations. Task-specific procedural knowledge reaches the online model only through this plan, which supplies the checks the perception module answers, the reference for assessing the user’s progress through the task plan and detecting deviations, and the content of intervention utterances. The dialogue and the visual observations supply the task context of the moment.

Perception Calls. A novelty signal computed from DINOv2 features of the incoming frames triggers a perception call at a novelty peak, during sustained activity, or after a fixed interval. The perception call receives recent keyframes together with the dialogue associated with the current step and its completion checks, and reports the user’s hand activity, the current action, visible state changes, the check values, and per-step progress, marking whether each value was seen or inferred from the dialogue.

Task Memory. Perception reports are deterministically integrated into a belief state that tracks task progress, committed completion checks, and recent evidence for each step. A check value is committed when consecutive perception calls agree, a committed completion is retained while the check is out of view, and the memory records per step how often the action associated with that step was observed. The Tracker, the intervention conditions, and utterance understanding read this belief state rather than individual reports.

Situation Tracker. The Tracker is a text-only call. It runs after a perception update has been integrated into the belief state when the observed activity has changed, when a user turn or a released instructor utterance has arrived since its last judgment, when a pending intervention candidate awaits confirmation, or otherwise after a bounded interval, with a minimum spacing between calls. From the plan, its previous judgment, the step evidence of the belief state, and the recent reports interleaved with the dialogue, it returns the current step and action, each step’s state with its basis (observed, moved past, or the user’s testimony), at most one deviation with whether it matters, and wait and stuck states. It weighs evidence conservatively: the user’s behavior rather than the plan order defines the user’s position in the task plan, a deviation requires agreeing consecutive reports or one report plus the user’s own words, and instructor utterances count as testimony of what the user was told, not as evidence of what was done. A consequential deviation is accepted only when consecutive judgments agree, and a stuck state only when it persists; the resulting position in the task plan sets the checks of the next perception call.

Proactive and Reactive Intervention Decisions

Evidence-Conditioned Proactive Decision. After each accepted judgment, a fixed set of intervention conditions is checked against the current belief state. When a condition is satisfied, the agent creates a typed intervention candidate (typed by the trigger types of § 3.3) grounded in the current evidence, which may subsequently be spoken, held, or discarded as that evidence develops. Correction opens on an accepted consequential deviation and states the contrast between what the user is doing and what the action calls for. Progress coaching opens at the onset of an action with a stated amount, manner, or critical parameter while the user is manipulating task objects and no deviation is open. Completion and hand-off opens when a step is reliably done: the final step receives a closing confirmation, and a completed non-final step is followed by the next step’s instruction unless that step has already been given, by the system or by a released instructor utterance, or has already started or passed. Stall help opens on a persisting stuck judgment. A ledger makes sure that the same evidence is voiced only once per visit to a step, and a pending candidate is discarded when its evidence is withdrawn or its window closes. The content of an intervention utterance is assembled from the plan (and, for corrections, from the Tracker’s deviation description, which states what the user is doing and what the action calls for), and one short model call rephrases it as a spoken sentence.

Taxonomy-Guided Reactive Decision. Each final ASR transcript with speech content is handled by one model call; final transcripts without words and very short acknowledgment or filler transcripts are filtered without invoking the model. The call labels the turn with one of the user-utterance categories of Table 5 and, for labels that call for a reply, continues in the same model call with the answer. The label places the turn in one of three response tiers: requests for information, verification, or clarification and trouble reports are answered. Narration and answers to a pending question are voiced only when the user’s turn states something that can be checked against the task context and the model produces a brief confirmation or correction. Plain narration receives no reply, and the remaining labels are silent. A reply is one or two short spoken sentences grounded in the plan, the latest situation judgment, and the recent dialogue, and a failed call produces no response.

Turn-Taking and Speech. While the system speaks, incoming partial ASR transcripts are checked: a filler-only partial transcript lets it continue, and a question cue or a label requiring a reply causes the system to defer to the user. Deferring stops the current sentence, holds queued proactive responses until the user has finished (dropping those whose step is no longer current), and cancels unstarted answers. Sentences are synthesized by the shared speech server (Appendix § 10.§ 1) and played one after another while perception continues.

Ablation Implementation

The two variants of § 6.4 run the final system with one switch each. Each replaces one decision mechanism, together with the control logic attached to it, by a direct decision of the same model on the evidence that mechanism used, while the backbone, perception, task memory, and Situation Tracker stay those of the final system. The contrasts therefore show what the taxonomy-guided reactive decision and the evidence-conditioned proactive decision contribute to the system’s selectivity. They do not isolate the contribution of any single rule, since each variant also differs from the final system in the control logic noted below.

Direct Reactive. This variant replaces the taxonomy-guided utterance understanding (the category label, the response tiers, and the filler guard) by one direct decision of the model on the same turn context. The prompt keeps the reply rules without reference to the taxonomy, states no preference for silence, and asks the model to decide whether the turn needs a spoken reply now and, if so, to write it in the same call. Turn-taking on partial hypotheses and the proactive side are unchanged.

Direct Proactive. This variant replaces the evidence-conditioned proactive decision (the intervention conditions, the candidate ledger, and the templated content with its phrasing step) by one direct decision of the model every 2 s of session time, skipped while a previous decision is still being computed. The decision reads the same evidence products (the plan, the latest Tracker judgment, the step evidence, and the recent reports with the dialogue) together with the utterances the user has recently heard, and returns either a response text, spoken as written, or nothing. Proactive utterances are thus generated on a fixed schedule rather than on evidence-triggered candidates, and a newer utterance supersedes an older one that has not started playing; replies to user turns and speech already playing are unaffected. The reactive side is unchanged.

Ablation Diagnostics. Table 3 classifies each played utterance as reactive (a reply to a user turn) or proactive (an utterance the system initiated), as the system labels its own output. Reactive and proactive Intervention Precision restrict Intervention Precision (Appendix § 9.3.§ 7) to the utterances of that side, and utterances per minute count them per minute of session time. The response rate on unanswered turns measures the proportion of user turns that did not require an instructor response but nevertheless received a system response. A user turn is considered not to require a response if it is not linked as the trigger of any necessary or optional reactive intervention. Each turn is paired with its most-overlapping final ASR transcript; turns without a corresponding final transcript are excluded. The repetition rate is the share of proactive utterances whose text, after case and punctuation normalization, repeats an earlier proactive utterance of the same session verbatim.

Experimental Setup and Baselines

This appendix gives the conditions shared by the systems of § 6.1 and the adaptation of each baseline. FLUID-Agent is described in Appendix § 9.4 and the evidence-ladder conditions in Appendix § 10.3.

Common Experimental Setup

Systems. We evaluate FLUID-Agent and the four baselines of § 6.1, each with its released checkpoint or hosted model. To be compatible with the teacher-forced protocol, a system must allow instructor utterances to be inserted into its history at a specified time without its own earlier outputs; Appendix § 10.§ 2 states how each baseline realizes the release rule of Appendix § 9.3.§ 2. All systems run each session in real time on the clock of Appendix § 9.3.§ 3.

Shared Inputs. User audio and transcript: the three open-weight baselines and FLUID-Agent hear the user through one shared causal streaming-ASR stream (NVIDIA Nemotron streaming speech recognition, 0.6B parameters), produced once per session from the user-only audio track (Appendix § 9.1.§ 2) and replayed for every system with each final transcript’s recorded availability time, so recognition errors are shared; Gemini hears the same audio natively. Video: frames are decoded once per session by source timestamp; ProAssist and VideoLLM read the latest frame at 2 fps, JoyAI samples at its native 1 s ticks, Gemini receives 1 fps (the Live API maximum), and the perception trigger of FLUID-Agent runs at 8 fps. Task knowledge: every system receives the same structured task knowledge of a session, extracted from egocentric demonstration video by a multi-round vision-language pipeline: one text per task for WTaG, whose sessions follow a common recipe, and one text per session for HoloAssist, whose procedures vary across sessions. The text lists the steps with an instruction sentence and prerequisite relations and, for WTaG, the objects of the task and a completion criterion per step. It is rendered once and provided to each system in the input format its model

expects; the plan compiler of FLUID-Agent (Appendix § 9.4.§ 1) and the LLM judges (Appendix § 9.3.8) read the same text. Speech output: the open-weight baselines and FLUID-Agent speak through the same Kokoro text-to-speech server, sentence by sentence; Gemini produces speech natively, and the API’s output transcription is the scored text.

Models and Hardware. Table 14 lists the backbone, hardware, and decoding of each system. Accelerators differ across systems and Gemini runs on vendor infrastructure, so Response Latency reflects end-to-end performance under each system’s actual deployment configuration and should not be interpreted as a controlled comparison of model inference speed. Gemini was accessed through its API in September 2026. Besides the streaming-ASR and text-to-speech servers above, FLUID-Agent uses DINOv2 perception-trigger features computed on the CPU of the machine running FLUID-Agent, as well as offline plan compilation with Qwen3.8-27B (Appendix § 9.4.§ 1). For evaluation, one Qwen3.8-27B judge is shared across all systems (Appendix § 9.3.8). The number of runs per system is given in Appendix § 9.3.9.

SystemBackboneHardwareDecoding
ProAssistLlama-3.1-8B + SigLIPA40greedy; speak gate θ=0.2\theta= 0.2
VideoLLMLlama-3-8B-Instruct + SigLIPA40greedy; upstream keep-watching threshold
JoyAIJoyAI-VL-InteractionRTX PRO 6000official sampling, fixed seed
GeminiGemini 3.8 Live (Live API)hostedvendor defaults
FLUID-AgentQwen3.6-35B-A3B (FP8), one instanceRTX PRO 6000temperature 0

Table 14. Models, hardware, and decoding.

Baseline Configurations and Adaptations

Shared Guidance Instructions. Every baseline receives the same task-guidance objectives in the slot its training format uses for instructions: a role description (the camera is the user’s first-person view; speak when it helps, by confirming a completed step and giving the next one, correcting a visible mistake, helping when the user is stuck, or answering a question; otherwise stay silent), the task-knowledge text, delivery rules (one short spoken sentence, no preamble, no repetition, no scene description), and the instruction to speak only when supported by concrete visual or auditory evidence. The wording follows each model’s native prompt format, and the prompts are provided in the released code repository.

ProAssist. ProAssist receives frames at its native 2 fps with the official preprocessing. At every frame the speak gate compares the probability of the end-of-turn (not-talk) token with the threshold θ=0.2\theta= 0.2, which lies within the range of the model’s own ablation; the model stays silent when the probability exceeds the threshold and otherwise generates with the official greedy decoding loop. The dialogue opens in the checkpoint’s training form with the shared instructions and the knowledge text; final ASR transcripts enter as user turns and released instructor utterances as assistant turns.

VideoLLM. VideoLLM also receives frames at 2 fps. Its streaming loop is re-implemented in-process: at each frame the probability of the keep-watching token is compared with the upstream default threshold, and a response is decoded greedily when that probability falls below the threshold. The system prompt is the checkpoint’s own, and the shared instructions with the knowledge text form the first user turn, the trained instruction slot. Final ASR transcripts enter as user turns and released instructor utterances as assistant turns, and the context is rebuilt without the system’s own outputs.

JoyAI. The released checkpoint is served by a bare vLLM server, and the live adapter (ticks, action tokens, frame window) is re-implemented in our evaluation code, so the evaluated system is the released checkpoint with our real-time reconstruction of its adapter. At every 1 s tick the model receives the last 140 ticks in native format (frames, final ASR transcripts, and released instructor utterances at their ticks), with earlier ticks kept only as their text turns and the history rebuilt without its own outputs, and emits either a silence token or a response token followed by text. Decoding uses the official adapter’s sampling with a fixed per-request seed.

Gemini. The vendor-hosted Live model hears the user audio natively, watches the video at 1 fps, and decides itself when to speak. Its system instruction carries the shared instructions adapted to native audio and a note that the history it starts from is the real record of the session, with the instructor’s utterances in its own role. The teacher-forced history is realized by rebuilding: whenever an instructor utterance is released, the current Live session is closed and a new one is opened whose initial content is the recorded interaction so far (the user’s audio, frames at a reduced rate for the earlier part of the session, and the released instructor utterances as model turns), after which real-time streaming resumes. The model’s own earlier outputs are not part of a rebuilt history, and output produced while a history is being seeded is discarded.

Evidence Ladder Experimental Design

The evidence ladder (§ 6.3, Table 2) was run on the 43 WTaG test sessions with JoyAI and Gemini. Its levels change the conditions under which a system intervenes at the annotated necessary interventions and the information it receives about them, while what to say is left to the system at every level; no level gives the reference response. This appendix defines the levels and then describes how they are implemented for each system.

Ladder Conditions

Triggers. The triggers are the necessary WTaG interventions at their annotated onset: the end of the user’s turn for interventions whose trigger includes a user utterance, and the evidence onset for the others. Annotated relations from the same source within 1 s of each other are merged into one intervention. Interventions whose trigger times fall in the same one-second decision step form one trigger, timed at the latest of their trigger times; the same decision-step rule is used for JoyAI and Gemini. Table 15 defines the four levels.

LevelIntervention decisionInformation at the trigger
L0Native: the system decides itself whether and when to speak.None.
L1Enforced: at each trigger the system is made to attempt one response; its speech elsewhere is suppressed.That an intervention is required now.
L2As L1.L1 plus the trigger type of each intervention, with a one-line gloss (§ 3.3).
L3As L1.L2 plus a short observed-fact sentence for the interventions whose trigger includes annotated action or state evidence and for which such a sentence exists; the other interventions keep the L2 note.

Table 15. Conditions of the evidence ladder.

At a trigger, the note is one sentence in the request (“Intervene now. Evidence requiring an instructor intervention is available. Type: not provided | type (gloss).”), to which L3 appends “Observed (video): fact.”; it is not kept in later history. The observed-fact sentences are short descriptions derived from the action and state annotations (Appendix § 9.2). An enforced response attempt can fail to produce a playable utterance (an empty generation, or a response not received within the API’s time limit), and such triggers count as no intervention. For JoyAI, a trigger that includes a user utterance also receives the annotated transcript of that utterance in place of the final ASR transcript, which often arrives after the annotated end of the turn; Gemini hears the audio itself.

Reading the Ladder. L1 changes the intervention regime as a whole, since the system is told the annotated onset, made to respond there, silenced elsewhere, and, for JoyAI, given the annotated transcript of the triggering turn and the ladder prompt. Δonset\Delta_{\mathrm{onset}} therefore measures the combined effect of these changes, not the contribution of timing information alone. Δtype\Delta_{\mathrm{type}} and Δfact\Delta_{\mathrm{fact}} add information to the note under the same regime; because the levels are separate runs, their differences also contain run-to-run variation. L3 adds observed facts only for interventions with action or state evidence, so its effect concentrates on proactive guidance (§ 6.3). The values at L1–L3 describe the systems under enforced intervention and are not estimates of their native capability.

Implementation

JoyAI. At a trigger tick the note is appended to the request and the decoder is made to speak by prefilling the assistant turn with the response action token, so that the model writes the intervention; at other ticks the request is made as usual, and a decision to speak is recorded but not played. The decision time of an enforced intervention is the start of the inference at the trigger tick. L1–L3 share one system prompt, which describes the note and differs in wording from the main-experiment prompt (Table 1); L0 is the main-experiment run (Table 1).

Gemini. The Live API generates at activity boundaries, so each trigger is realized as one fresh Live session seeded with the recorded interaction up to the last released instructor utterance (the re-build of Appendix § 10.§ 2). The audio and video up to the trigger are then streamed in real time inside one manual activity with automatic activity detection disabled, and the note is sent as text followed by the end of the activity, which marks the session’s only turn boundary and triggers generation. Output produced before the note is discarded. The system instruction is the same across L1–L3, and the main-experiment run (Table 1) serves as L0.

Human Timing Study

This study tests the annotated timing windows and calibrates the graded credit used by Timely Correctness. Three raters judged the timing of WTaG interventions whose content was held fixed while only the speech start time was varied, so that the ratings reflect timing rather than wording. We first describe the study events, the candidate moments, and the rating procedure, then compare the annotated windows with the human judgments of acceptable timing, and finally compare temporal scoring functions and select their parameters with leave-one-session-out (LOSO) cross-validation. Throughout, the anchors are written as in Appendix § 9.2.§ 5: ES, EO, LU, and R correspond to tearlyt_{\mathrm{early}}, tevidt_{\mathrm{evid}}, tlatestt_{\mathrm{latest}}, and tusrt_{\mathrm{usr}} of the main text and Appendix § 9.3.§ 6.

Study Events

The formal study covers 79 WTaG events from 31 sessions, and a separate set of 12 events was used for calibration. Formal events required complete source media, a reference instructor utterance, and the anchors needed for the reactive or proactive window. We included the 12 non-Other instructor categories (Table 6) and excluded reference utterances that overlap user speech, since these would confound the speech start time with simultaneous speech.

Events were sampled with category quotas to cover the intervention types, with at most three events per session and none from the calibration sessions; the quotas are not intended to reflect corpus prevalence. The formal set contains 26 coffee, 28 mug-cake, and 25 pinwheel events: 43 reactive and 36 proactive, 46 necessary and 33 optional.

The calibration set contains one event from each of 12 distinct sessions, four per task. It is balanced between reactive and proactive, necessary and optional, and three anticipated difficulty levels, covers 11 of the 12 non-Other categories, and uses the same candidate-moment construction as the formal study.

Candidate-Moment Construction

Each event has six candidate moments spanning early, valid, and late responses. Two are fixed: the preferred time, which is the end of the triggering user turn (R) for reactive events and the evidence onset (EO) for proactive events, and the speech start time of the original instructor utterance. The other four are sampled relative to the event’s anchors. The early moment precedes EO for proactive events and falls within the triggering user turn for reactive events. Two moments cover the first and second halves of the interval from the preferred time to the latest useful time (LU). The last slot is sampled after LU with probability 0.5, from a heavy-tailed distribution, and otherwise uniformly between the earliest candidate position (ES for proactive events, the start of the triggering user turn for reactive events) and LU. Moments must be at least 0.1 s apart; near-duplicate fixed moments are merged and the open slot is resampled. Sampling therefore covers positions relative to the anchors rather than uniformly across the video.

Rating Protocol and Calibration

Raters viewed the source context up to each candidate moment and then judged the timing of the same synthesized reference utterance on an integer scale from 0 (definitely should not intervene now) to 10 (exactly the desired moment); they were asked to judge timing rather than wording. The rubric marked 5 as borderline and 6 as acceptable but improvable, and Cannot judge responses were treated as missing. All three raters were authors. The roles of candidate moments (early, preferred, or late), annotated timing boundaries, and original reference start-time labels were hidden from raters.

All raters completed calibration before the formal study. The 12 calibration events produced 216 records, including one Cannot judge. Calibration familiarized the raters with the interface and the rubric; its ratings were excluded from the primary reference, MAE, and parameter selection, and were not used to exclude or reweight raters.

The formal phase produced 1,422 rating rows over 474 candidate moments, of which 1,415 are valid ratings and seven are Cannot judge. The human reference for each moment is the mean available rating divided by ten.

Validation of the Annotated Timing Windows

Before calibrating graded credit, we tested whether the annotated valid windows agree with binary human judgments of acceptable timing. For candidate jj of event ii, let rˉij\bar{r}_{ij} be the mean of the valid rater scores on the 0–10 scale. Following the rubric’s threshold for acceptable timing, human acceptability is

Aij=1[rˉij≥6].(10)A_{ij} = 1[\bar{r}_{ij} \ge6]. \tag*{(10)}

We compare two binary timing rules:

Bijwin=1[tij∈Wi],(11)B_{ij}^{\mathrm{win}} = 1[t_{ij} \in W_i], \tag*{(11)}
Bij[−6,+3]=1[−6≤tij−tref,i≤3],(12)B_{ij}^{[-6,+3]} = 1[-6 \le t_{ij} - t_{\mathrm{ref},i} \le3], \tag*{(12)}

where WiW_i is the annotated valid window of event ii ([R, LU) for reactive and [ES, LU) for proactive events) and tref,it_{\mathrm{ref},i} is the speech start time of the original instructor utterance, in seconds. Each rule is treated as a classifier of AijA_{ij}, and its agreement with the human judgments is measured by Macro-F1, the unweighted mean of the F1 scores of the acceptable and the unacceptable class:

Macro-F1⁡=12(F1++F1−),\operatorname{Macro\text{-}F1} = \frac{1}{2}(F_1^+ + F_1^-),
F1c=2TPc2TPc+FPc+FNc,(13)F_1^c = \frac{2\mathrm{TP}_c}{2\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c}, \tag*{(13)}

where F1+F_1^+ takes the acceptable moments (Aij=1A_{ij}=1) as the positive class and F1−F_1^- the unacceptable moments (Aij=0A_{ij}=0), with TPc\mathrm{TP}_c, FPc\mathrm{FP}_c, and FNc\mathrm{FN}_c counted for class cc. Unacceptable moments are the minority (146 of the 474 candidate moments), so the macro average weights the two classes equally. Confidence intervals and paired differences use 5,000 session-cluster bootstrap draws, each resampling the 31 formal sessions with replacement and keeping every candidate moment of a sampled session. Figure 8 reports the results. Over all 474 moments, the evidence-derived window reaches a Macro-F1 of 0.788 (95% CI [0.741, 0.832]) against 0.713 [0.663, 0.762] for the fixed window, a paired difference of +0.075 ([0.002, 0.149]). The gap comes from reactive events, 0.787 against 0.659, a paired difference of +0.127 ([0.040, 0.220]): the fixed window accepts 61 unacceptable reactive moments, 39 of them while the user is still speaking, whereas the evidence window opens only after the user’s turn ends. On proactive events the two windows perform comparably, 0.781 ([0.686, 0.861]) against 0.773 ([0.701, 0.836]), a difference of +0.008 ([−0.099, 0.116]); the fixed window rejects 16 acceptable moments that lie inside the evidence window, 5 of them earlier than 6 s before the reference onset and 11 later than 3 s after it.

Figure 8. Agreement with human acceptability. Macro-F1 of the evidence-derived windows (blue) and the fixed [−6, +3]-second windows (orange) against human acceptability (mean rating of at least 6), per guidance side and pooled. Whiskers show 95% session-cluster bootstrap confidence intervals; Δ\Delta is the paired difference (evidence minus fixed), starred when its 95% interval excludes zero; nn counts candidate moments.

This comparison tests the window boundaries. The following analyses test how much credit Timely Correctness should assign within a valid window.

Temporal Scoring Hypotheses

We compared three families of graded timing scores, each assigning zero outside the valid window. They are written in terms of the anchors ES, EO, R, and LU of Appendix § 9.2.§ 5 and the normalized positions

uE=EO−tEO−ES,uL=t−EOLU−EO,uR=t−RLU−R(14)u_E = \frac{\mathrm{EO}-t}{\mathrm{EO}-\mathrm{ES}}, \qquad u_L = \frac{t-\mathrm{EO}}{\mathrm{LU}-\mathrm{EO}}, \qquad u_R = \frac{t-R}{\mathrm{LU}-R} \tag*{(14)}

where uEu_E falls from 1 to 0 over [ES, EO), while uLu_L and uRu_R rise from 0 to 1 over [EO, LU) and [R, LU), respectively.

Hypothesis 1: Normalized Linear Position. Under this hypothesis, credit changes at a constant rate with the normalized position in the event-specific window:

Tlinpro(t)={1−uE,t∈[ES,EO),1−uL,t∈[EO,LU),(15)T_{\mathrm{lin}}^{\mathrm{pro}}(t) = \begin{cases} 1-u_E, & t \in[\mathrm{ES},\mathrm{EO}),\\ 1-u_L, & t \in[\mathrm{EO},\mathrm{LU}), \end{cases} \tag*{(15)}
Tlinrea(t)=1−uR,t∈[R,LU).(16)T_{\mathrm{lin}}^{\mathrm{rea}}(t)=1-u_R, \qquad t \in[\mathrm{R},\mathrm{LU}). \tag*{(16)}

Hypothesis 2: Base-2 Squared Exponential. This hypothesis uses absolute elapsed seconds rather than the normalized window position:

Texppro(t)={2−((t−EO)/τE)2,t∈[ES,EO),2−((t−EO)/τL)2,t∈[EO,LU),(17)T_{\mathrm{exp}}^{\mathrm{pro}}(t) = \begin{cases} 2^{-((t-\mathrm{EO})/\tau_E)^2}, & t \in[\mathrm{ES},\mathrm{EO}),\\ 2^{-((t-\mathrm{EO})/\tau_L)^2}, & t \in[\mathrm{EO},\mathrm{LU}), \end{cases} \tag*{(17)}
Texprea(t)=2−((t−R)/τR)2,t∈[R,LU),(18)T_{\mathrm{exp}}^{\mathrm{rea}}(t)=2^{-((t-R)/\tau_R)^2}, \qquad t \in[\mathrm{R},\mathrm{LU}), \tag*{(18)}

with the WTaG corpus-level scales τE=1.10\tau_E = 1.10 s, τL=2.80\tau_L = 2.80 s, and τR=1.22\tau_R = 1.22 s.

Hypothesis 3: Quadratic Onset and Cosine-Power Decay. This family controls the proactive early discount and the shape of the late decline separately:

Tcospro(t)={1−λEuE2,t∈[ES,EO),cos⁡pL(π2uL),t∈[EO,LU),(19)T_{\mathrm{cos}}^{\mathrm{pro}}(t) = \begin{cases} 1-\lambda_E u_E^2, & t \in[\mathrm{ES},\mathrm{EO}),\\ \cos^{p_L}\left(\frac{\pi}{2}u_L\right), & t \in[\mathrm{EO},\mathrm{LU}), \end{cases} \tag*{(19)}
Tcosrea(t)=cos⁡pL(π2uR),t∈[R,LU).(20)T_{\mathrm{cos}}^{\mathrm{rea}}(t)=\cos^{p_L}\left(\frac{\pi}{2}u_R\right), \qquad t \in[\mathrm{R},\mathrm{LU}). \tag*{(20)}

Here λE\lambda_E sets the proactive discount at ES, where the credit is 1−λE1-\lambda_E, and pLp_L sets the decline after the preferred time. The settings (λE,pL)=(0.2,4)(\lambda_E,p_L)=(0.2,4) and (0.4,1)(0.4,1) are reported as reference points within this family, not as separate models.

As diagnostic baselines, we also score the two binary rules of Eqs. (11) and (12) as step functions, which compares graded credit with binary timing acceptability.

Scoring functions are selected by event-level mean absolute error (MAE) against the normalized human ratings. For a temporal function TT,

MAE⁡(T)=1∣E∣∑i∈E1∣Mi∣∑j∈Mi∣T(tij)−yˉij∣,(21)\operatorname{MAE}(T)=\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{E}}\frac{1}{|\mathcal{M}_i|}\sum_{j\in\mathcal{M}_i}\left|T(t_{ij})-\bar{y}_{ij}\right|, \tag*{(21)}

where yˉij\bar{y}_{ij} is the mean valid human rating divided by ten and Mi\mathcal{M}_i contains the valid candidate moments of event ii. Errors are thus averaged within each event and then across events, giving events equal weight. Mean within-event Kendall τb\tau_b serves as a secondary measure of ordering; MAE is primary. because Timely Correctness uses the amount of credit. Uncertainty intervals come from a session-cluster bootstrap in which each draw resamples sessions and keeps every event and candidate moment of each sampled session, and differences between functions are paired within the same draw.

For the cosine family, we search

λE∈{0,0.1,…,1},pL∈{0.1,0.25,0.5,0.75,1,1.5,2,3,4}.(22)\lambda_E \in\{0, 0.1, \ldots, 1\}, \\ p_L \in\{0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 4\}. \tag*{(22)}

The parameters are evaluated by leave-one-session-out (LOSO) cross-validation over the 31 formal sessions: in each fold, the pair minimizing event-level MAE on the other 30 sessions is applied, without refitting, to the held-out session, so that session’s ratings are not used to select the parameters applied to it.

Results

The LOSO-selected cosine has the lowest MAE among the compared functions (Figure 9). Thirty of the 31 folds selected (λE,pL)=(0.7,0.5)(\lambda_E, p_L) = (0.7, 0.5) and one selected (0.6,0.5)(0.6, 0.5). The parameters used by Timely Correctness, (0.7,0.5)(0.7, 0.5), are those selected on all 31 formal sessions; the MAE reported for the selected cosine is its out-of-fold value.

Figure 9. Agreement of temporal scores with human ratings. Event-level MAE with 95% session-cluster bootstrap confidence intervals; lower is better. Cosine settings are (λE,pL)(\lambda_E, p_L). The selected cosine uses LOSO out-of-fold predictions.

The selected cosine achieves MAE 0.182, compared with 0.225 for linear scoring and 0.262 for constant valid-window credit. Its paired difference from linear is −0.043-0.043 with a 95% confidence interval (CI) of [−0.064,−0.023][-0.064, -0.023], its improvement over constant credit is 0.080 ([0.069,0.091])([0.069, 0.091]), and it also improves over the (0.4,1)(0.4, 1) cosine setting by 0.015 ([0.005,0.025])([0.005, 0.025]). These comparisons support graded credit within the window. The selected pL=0.5p_L = 0.5 lies inside the search grid rather than at its lower boundary, and it gives a shallower decline through moderate delay than pL=4p_L = 4 while still assigning zero at LU. Linear and selected cosine scores show similar within-event ranking agreement (τb=0.740\tau_b = 0.740 and 0.738), but the selected cosine achieves better calibration to the human rating scale, as reflected by its lower MAE.

Calibration and Rater Sensitivity Analyses

The comparison with binary scoring in § 4.2 uses Spearman’s rank correlation computed within each event between a function’s scores and the mean human ratings of its six candidate moments, averaged over events. Linear and selected cosine scores rank the six moments identically in 78 of the 79 events and reach mean correlations of 0.833 and 0.832. Constant valid-window credit is identical across all six moments of 12 events, where the correlation is undefined; over the remaining 67 events it averages 0.645 (0.547 when undefined events count as zero), and the fixed [−6,+3][-6, +3] s step averages 0.616 over its 50 defined events. Pooling all 474 moments instead gives 0.787 for both graded scores and 0.586 for constant valid-window credit.

Selection using only the 46 necessary events from 27 sessions gives the same parameters in all 27 LOSO folds, with out-of-fold MAE 0.178 versus 0.219 for linear scoring, so including the optional events does not change the selected setting. We use one shared function rather than separate scoring functions for individual instructor categories, whose small groups are partly confounded with the reactive/proactive composition.

The selected cosine keeps its advantage after adjusting for rater differences and when any one rater is omitted. For the calibration-informed analysis, we jointly modeled the valid calibration and formal ratings with a random intercept for candidate moment and fixed effects for study phase, rater, timing region (before the preferred time, within the window, or after LU), intervention side, and the corresponding rater interactions; the calibration ratings serve here only to estimate rater severity and timing tolerance. The adjusted formal reference correlates 0.993 with the raw rater mean. Against it, cosine MAE is 0.190 and linear MAE is 0.236, a paired difference of −0.046-0.046 (95% CI [−0.063,−0.029][-0.063, -0.029]). This agrees with the primary analysis. Given the small number of raters and calibration events, we treat this adjustment as an exploratory sensitivity analysis rather than independent validation.

The two-way mixed-effects absolute-agreement intraclass correlation coefficient (ICC) is 0.663 for one rater (ICC(A,1)) and 0.855 for the three-rater mean (ICC(A,3)). Omitting each rater in turn preserves the cosine advantage, with MAE improvements of 0.031–0.058. Raters differed in strictness and preferred timing, which supports using their mean rather than a single rater’s curve.

References

  1. [1]Yuwei Bao, Keunwoo Peter Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yosef, Alex de la Iglesia, Megan Su, Xiao Lin Zheng, and Joyce Chai. 2023. Can foundation models watch, talk and guide you step by step to make a cake? In Findings of the Association for Computational Linguistics: EMNLP 2023.arxiv.org/abs/2311.00738
  2. [2]Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh, Rajeev Yasarla, Reza Pourreza, Litian Liu, Risheek Garrepalli, and Roland Memisevic. 2026. Streaming interventions: Can video large language models correct mistakes as they occur? arXiv preprint arXiv:2606.09547.arxiv.org/abs/2606.09547
  3. [3]Apratim Bhattacharyya, Bicheng Xu, Sanjay Haresh, Reza Pourreza, Litian Liu, Sunny Panchal, Leonid Sigal, and Roland Memisevic. 2025. Can multi-modal LLMs provide live step-by-step task guidance? In The Thirty-ninth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2511.21998
  4. [4]ByteDance Seed. 2026. SeedRealtime: An Audio-Visual Full-Duplex LLM. Technical blog, published August 5, 2026.
  5. [5]Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024. Videollm-online: Online video large language model for streaming video. In CVPR.arxiv.org/abs/2406.11816
  6. [6]Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. 2025. Livecc: Learning video llm with streaming speech transcription at scale. In CVPR.arxiv.org/abs/2504.16030
  7. [7]Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, Jiancheng Gui, Luoyuan Zhang, Xian Sun, Fuwei Huang, Moye Chen, Zhuo Lin, Hanyu Liu, Qingxin Gui, Qingzhe Han, and 17 others. 2026. Minimcp-0 4.5: Towards real-time full-duplex omni-modal interaction. Preprint, arXiv:2604.27393.
  8. [8]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2018. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV).arxiv.org/abs/1804.02748
  9. [9]Shenghao Fu, Qize Yang, Yuan-Ming Li, Yi-Xing Peng, Kun-Yu Lin, Xihan Wei, Jian-Fang Hu, Xiaohua Xie, and Wei-Shi Zheng. 2025. Vispeak: Visual instruction feedback in streaming videos. arXiv preprint arXiv:2503.12769.arxiv.org/abs/2503.12769
  10. [10]Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xi-ang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, and Yifei Huang. 2026. Vinci2: Providing proactive assistance in continuous egocentric videos. In European Conference on Computer Vision (ECCV).arxiv.org/abs/2607.11523
  11. [11]Google DeepMind. 2026. Gemini 3.8 Audio (Live, Live Extended Thinking, Flash TTS, Flash-Lite TTS) Model Card. Google DeepMind Model Card. Published September 15, 2026.
  12. [12]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Javant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, and 66 others. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. Preprint, arXiv:2110.07058.arxiv.org/abs/2110.07058
  13. [13]Ant Group. 2026. Realtime-venus: A full-duplex interaction system with asynchronous delegation. Preprint, arXiv:2609.13814.arxiv.org/abs/2609.13814
  14. [14]Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Dong Lu, Yali Wang, Limin Wang, and Yu Qiao. 2024. Egoex-olearn: A dataset for bridging asynchronous ego- and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  15. [15]Kaustav Kundu, Ritvik Shrivastava, Maxim Arap, Nanshu Wang, Xianhui Zhu, Quintin Fettes, Gautam Tiwari, Parth Suresh, Théo Moutakanni, Alejandro Castillejo Munoz, Allen Bolourchi, Pascale Fung, Priyan Donmez, Babak Damavandi, Anuj Kumar, and Seungwhan Moon. 2026. Plan, watch, recover: A benchmark and architectures for proactive procedural assistance. Preprint, arXiv:2606.04970.arxiv.org/abs/2606.04970
  16. [16]Xudong Lu, Xueying Li, Annan Wang, Yang Bo, Jinping Chen, Zengliang Li, Nianzu Yang, Rui Liu, Xue Yang, Jingwen Hou, and Hongsheng Li. 2026. OmniInteract: Benchmarking real-world streaming interaction for real-time omnmodal assistants. arXiv preprint arXiv:2605.26485.arxiv.org/abs/2605.26485
  17. [17]OpenAI. 2026. GPT Realtime 2. Model: gpt-realtime-2. Accessed Oct 2026.
  18. [18]Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Böhm, Florian Diettriecht, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovic, Ingo Bax, and Roland Memisevic. 2024. What to say and when to say it: Live fitness coaching as a testbed for situated interaction. In NeurIPS (D&B Track).arxiv.org/abs/2407.08101
  19. [19]Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Qifan Zhang, Jikai Wang, Vasundhara Komaragiri, Eric Ragan, Nicholas Ruozzi, Yu Xiang, and Vibhav Gogate. 2024. Captaincook4d: A dataset for understanding errors in procedural activities. In Advances in Neural Information Processing Systems, volume 37, pages 135626–135679. Curran Associates, Inc.arxiv.org/abs/2312.14556
  20. [20]Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling video LLMs with active real-time interaction via disentangled perception, decision, and reaction. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 24045–24055.arxiv.org/abs/2501.03218
  21. [21]Qwen Team. 2026a. Qwen3.5-omni technical report. Preprint, arXiv:2604.15804.arxiv.org/abs/2604.15804
  22. [22]Qwen Team. 2026b. Qwen3.6-35B-A3B: Agentic coding power, now open to all.
  23. [23]Francesco Ragusa, Michele Mazzamuto, Rosario Forte, Irene D’Ambra, James Fort, Jakob Engel, Antonino Furnari, and Giovanni Maria Farinella. 2026. Ego-extra: Video-language egocentric dataset for expert-trainee assistance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).arxiv.org/abs/2512.13238
  24. [24]Yuki Sakai, Ryosuke Furuta, Juichun Yen, and Yoichi Sato. 2025. Egoinstruct: An egocentric video dataset of face-to-face instructional interactions with multimodal llm benchmarking. In Proceedings Workshop on Interactive Human-centric Foundation Models (I-HFM 2025) (in conjunction with ICCV 2025).arxiv.org/abs/2509.22019
  25. [25]Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, and Limin Wang. 2026. River: A real-time interaction benchmark for video llms. Preprint, arXiv:2603.03985.arxiv.org/abs/2603.03985
  26. [26]Thinking Machines Lab. 2026. Interaction models: A scalable approach to human-ai collaboration. Thinking Machines Lab: Connectionism. https://thinkingmachines.ai/blog/interaction-models/.DOI
  27. [28]Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. 2023. Holoassist: an ego-centric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20270–20281.DOI
  28. [29]Yueqian Wang, Songxiang Liu, Disong Wang, Nuo Xu, Wan Guanglu, Huishuai Zhang, and Dongyan Zhao. 2026. MMDuet2: Enhancing proactive interaction of video MLLMs with multi-turn reinforcement learning. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2512.06810
  29. [30]Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, and Dongyan Zhao. 2025. Proactivevideoqa: A comprehensive benchmark evaluating proactive interactions in video large language models. Preprint, arXiv:2507.09313.arxiv.org/abs/2507.09313
  30. [31]Lilin Xu, Bufang Yang, Siyang Jiang, Kaiwei Liu, Kaiyuan Hou, Yuang Fan, Hongkai Chen, Zhenyu Yan, and Xiaofan Jiang. 2026. Pro²assist: Continuous step-aware proactive assistance with multi-modal egocentric perception for long-horizon procedural tasks. Preprint, arXiv:2605.04227.arxiv.org/abs/2605.04227
  31. [32]Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, and Seungwhan Moon. 2025a. Proactive assistant dialogue generation from streaming egocentric videos. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12055–12079.arxiv.org/abs/2506.05904
  32. [33]Yichi Zhang, Run Peng, Yinpei Dai, Lingyun Wu, Xuweiyi Chen, Qiaozi Gao, and Joyce Chai. 2025b. Bootstrapping visual assistant modeling with situated interaction simulation. In Second Conference on Language Modeling.
  33. [34]Yichi Zhang, Jianing Yang, Keunwoo Yu, Yinpei Dai, Shane Storks, Yuwei Bao, Jiayi Pan, Nikhil Devraj, Ziqiao Ma, and Joyce Chai. 2023. Seagull: an embodied agent for instruction following through situated dialog. Alexa Prize SimBot Challenge Proceedings.

Paper details

vlnc:W_fmct527s
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents