WorksComputer Science
FLUID: Flexible, Evidence-Grounded Situated Task Guidance
Abstract
Real-time task guidance is inherently flexible: when an agent guides a user through a task, multiple responses and timings can be appropriate, and sometimes the best action is to remain silent. Existing benchmarks often miss this flexibility by treating synthesized dialogues as fixed targets for what to say and when to say it. We introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for modeling, evaluating, and enabling flexible interaction in multimodal task guidance. Using naturally occurring human-human guidance from WTaG and HoloAssist, FLUID grounds interventions in their supporting evidence, necessity, and timing flexibility, and uses this representation both to evaluate flexible guidance and to build a real-time agent for task guidance. Our agent achieves the strongest overall balance of response correctness, intervention precision, timely guidance, and response latency among evaluated systems, while still leaving room for improvement in avoiding speech overlap. Together, FLUID moves situated task guidance beyond rigid reference matching toward more natural and adaptive human-AI interaction. Project page: https://fluid-guidance.github.io.
Introduction
With the growing adoption of smart glasses and other wearable devices, multimodal AI assistants are becoming increasingly accessible for providing real-time, situated assistance in everyday activities [7, 17, 11]. An important emerging application is perceptual task guidance, where an AI assistant observes the user’s environment and ongoing actions through an egocentric camera and provides context-aware guidance through natural language as shown in Figure 1. This kind of situated perceptual task guidance requires continuous coordination between visual perception, task understanding, and communication. The AI assistant must interpret the user’s
Figure 1. Illustration of perceptual task guidance.
actions and task progress, respond to questions, provide step-by-step instructions, detect mistakes, and proactively offer assistance when appropriate.
To evaluate these capabilities, existing work commonly relies on human-human interaction data or synthetically generated dialogues as references for expected guidance behaviors, measuring how closely the AI instructor’s responses align with the reference responses in both semantic content and temporal placement [3, 32, 25]. Although a good starting point, these approaches treat reference interactions as fixed targets, overlooking the inherent flexibility of situated guidance. In practice, human-human guidance is fluid and adaptive. Certain situations, such as incorrect actions or imminent safety risks, require immediate intervention, while others allow a wider range of appropriate timing for proactive assistance. Similarly, backchannels and acknowledgments help maintain shared understanding, but their timing and frequency can vary considerably. This creates a fundamental mismatch between rigid reference-based evaluation and the flexible nature of real-world task guidance, calling for evaluation criteria that can take into account both the correctness of guidance and the flexibility of communication.
Addressing this challenge requires a principled framework for characterizing the flexibility of situated guidance. Importantly, flexibility does not imply unconstrained behavior. Effective guidance must remain grounded in perceptual and interaction evidence, with different communicative behaviors subject to different requirements. When AI assistants decide to intervene or provide guidance, some interventions (e.g., correcting a user mistake or giving a warning about a potential hazard) are necessary within narrow temporal windows, while others (e.g., backchannel) are optional and allow greater flexibility in content and timing. Human references remain valuable for identifying effective guidance behaviors, but should not prescribe a single correct response at a fixed moment. Instead, the key is to characterize the underlying evidence that warrants communication and the conditions under which different responses are appropriate. This evidence-grounded perspective provides a foundation for both evaluating and generating flexible guidance behaviors.
In this paper, we introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for characterizing, evaluating, and enabling flexible interaction dynamics in multimodal task guidance. First, we develop an evidence-grounded representation that characterizes communicative behaviors in terms of their necessity, supporting evidence, and temporal flexibility (FLUID-Modeling). Second, we introduce an evaluation framework that leverages human references while moving beyond strict response matching to assess the appropriateness of guidance content and timing (FLUID-Evaluation). Third, we propose an agent architecture that explicitly models and adapts to evolving interaction dynamics, enabling flexible, evidence-grounded decisions about when and how to communicate, including reactive responses, proactive guidance, and appropriate silence (FLUID-Agent). Together, these contributions provide a systematic approach to understanding, evaluating, and generating flexible, evidence-grounded guidance, advancing multimodal AI assistants toward more natural and adaptive human–AI interaction.
Related Work
Real-Time Interaction. Recent advances in multimodal systems have rapidly expanded real-time human-AI interaction, from streaming audio-visual perception [17, 21] to low-latency, full-duplex spoken interaction [7, 26, 13, 4, 11]. Scaling such interactive capabilities has increasingly relied on synthetic or automatically constructed interaction data, often by repurposing existing egocentric videos [12, 19, 8, 14] and generating intervention traces at scale [25, 32, 15]. In contrast, naturally occurring human-human interactions in which one person continuously observes and guides another through a physical task remain scarce. Existing efforts include WTaG [1], HoloAssist [28], EgoInstruct [24], Ego-EXTRA [23], and Ego-MC-Bench [2], but some of them remain limited in released modalities. In this work, we primarily build on WTaG and HoloAssist, both of which provide audio alongside egocentric video, to study how interactions naturally unfold during human guidance.
Proactive Agent Modeling. Existing proactive video agents span query-conditioned event response, continuous commentary, and situated procedural guidance [30, 16, 5, 9, 20, 29, 6, 32]. Representative systems typically produce textual responses or discrete decisions about whether to intervene, with timing supervision derived from annotated event boundaries, task transitions, mistakes, or synthesized utterances [18, 34, 33, 3, 15]. Recent methods move beyond purely event-triggered policies: Pro²Assist additionally annotates assistance need, while Vinci2 generates visually grounded service opportunities and reasons about whether intervention is warranted [31, 10]. They provide increasingly rich supervision; however, their assistance targets are constructed from task states or annotated events rather than naturally occurring intervention decisions. In contrast, we study human-human guidance. This preserves a broader range of communicative behaviors and enables intervention-specific annotations of necessity and valid timing, characterizing not only when assistance may occur but how much flexibility each interaction permits.
FLUID-Modeling: Grounded Evidence in Task Guidance
Problem Definition
In real-time task guidance, an agent continuously observes a user performing a task and must decide how to respond appropriately as the task and interaction evolve. We characterize each intervention decision along four dimensions. WHY: what evidence warrants the intervention, namely the user utterance, action, or state that the guide reacts to; we refer to it as the grounded evidence of the intervention. WHETHER: whether the intervention is necessary, optional, or not needed. WHEN: the period during which the intervention is useful. WHAT: the interaction act of the intervention and its content. Interventions triggered by a user utterance are reactive; all others are proactive.
Source Data
We study WTaG ([1]) and HoloAssist ([28]), two egocentric datasets of naturally occurring human-human task guidance in which a remote instructor observes the user’s view and provides verbal guidance. The two datasets exhibit complementary interaction regimes. WTaG is relatively self-paced: users receive a high-level task recipe in advance and can often proceed independently, which produces mixed-initiative interaction. HoloAssist has a stronger knowledge asymmetry and is predominantly instructor-led, with users relying on ongoing guidance for each step. In total, we select 43 WTaG sessions and 72 HoloAssist sessions with substantive interactions between users and instructors.
Multi-Layer Annotation
To reveal the interaction semantics underlying naturally occurring human interventions, we augment WTaG and HoloAssist with four linked annotation layers, which together cover the four dimensions of § 3.1 (see Figure 2 for an example).

Figure 2. Four-layer annotation of a WTaG coffee session, covering utterance taxonomy, action and state, trigger links, and intervention timing. Each instructor intervention is grounded in its triggering evidence, labeled by necessity, and assigned an evidence-derived intervention window.
Layer 1: Utterance Taxonomy. We categorize every user and instructor utterance using an utterance-level taxonomy with 11 user categories and 13 instructor categories.
Layer 2: Action and State. We annotate user actions and states that are closely related to instructor interventions, including hand actions, visual loss, stalling, and searching. Each action is further labeled by whether it is valid or deviating and by its effect on task execution, such as task progress, step completion, or recovery.
Layer 3: Trigger Link. Using the annotated utterances and actions, we link each instructor intervention to the preceding utterance, action, or state that triggered it, forming a grounded evidence link that explains why the human instructor decided to intervene. Each link receives a trigger type derived from the linked events, such as Step Done, Error, or User Query. We further categorize each intervention as necessary if, without it, the user would make a mistake, get stuck, or lack needed information; optional if the user would proceed correctly on their own, so that speaking and remaining silent are both acceptable; and excluded if it is irrelevant to the task, such as self-repairs, or relies on knowledge that cannot be obtained from the shared egocentric view, the dialogue, or the task knowledge.
Layer 4: Evidence Onset and Intervention Window. We annotate temporal anchor points for interventions, which define the time window within which an intervention should be initiated. For reactive interventions, we annotate (1) the time at which the user finishes their most recent utterance, , and (2) the latest time at which the intervention remains useful, . For proactive interventions, there is no corresponding ; instead, we annotate (1) the earliest time at which initial signs suggest that an intervention may be necessary, , and (2) the time at which enough evidence becomes available to determine that the intervention is necessary, , together with . Together, these anchors define an evidence-derived window for evaluating intervention timing.
Annotation Workflow. For WTaG, we first use majority voting across multiple language-model annotators to produce initial annotations, followed by careful manual review of all instances. Based on the errors and practical observations identified during this process, we refine the machine annotation pipeline and apply it to the selected HoloAssist sessions, and manually review about 10% of the generated annotations to ensure the synthetic data quality.
Interventions Are Context-Dependent
Our annotations show that natural human guidance does not follow a fixed trigger-to-intervention mapping: whether an intervention is needed, and what form it should take, depends on the surrounding interaction context.
Similar situations may or may not require intervention. Among step completions on the self-paced WTaG, only 12.9% are followed by a necessary intervention (e.g., providing the next-step instruction), while 17.5% are optional (e.g., “Nice” or “uh-huh”) and 69.6% receive no intervention (). In the more instructor-led HoloAssist setting, the corresponding proportions are 65.9%, 12.2%, and 21.9% (). Thus, whether a step completion warrants guidance depends strongly on the surrounding interaction regime and context, rather than on the event type alone. In contrast, existing approaches often use a more rigid event-to-response mapping, such as triggering a next-step instruction whenever a step is completed.
These observations motivate evaluating guidance against the grounded evidence that warrants an intervention, whether intervention is necessary, and when it remains useful.
FLUID-Evaluation on Task Guidance
Existing reference-based evaluations assess guidance primarily through content similarity and temporal proximity to human reference utterances, often using a fixed timing window [32, 3, 2]. Such criteria overlook the flexibility of natural interaction, where appropriate guidance depends on task evidence and may vary in necessity and timing. We introduce an evidence-derived evaluation framework that leverages our annotations to evaluate intervention appropriateness based on supporting evidence, necessity, and valid timing windows. Appendix § 9.3 specifies the protocol and metrics in full.
Intervention Matching
Real-time interaction does not follow fixed turn boundaries: within any time interval, a system may produce zero, one, or multiple utterances, whereas evaluation requires a one-to-one correspondence between system utterances and annotated human interventions. We therefore first match them based on content and supporting evidence. For each necessary or optional intervention, we consider candidate system utterances within a matching window that covers its annotated evidence window and the surrounding intervals of adjacent reference utterances. An LLM judge then assesses content equivalence based on the dialogue history and triggering evidence. Since a system may generate multiple equivalent utterances within the same time interval, we select the earliest qualifying utterance as the match.
Timing Appropriateness
Evaluating the timing of matched interventions is essential, yet common approaches [32, 3, 2] have two limitations: (1) fixed timing windows ignore when an intervention becomes possible and when it stops being useful, admitting invalid onsets and excluding valid ones; and (2) binary in-window scoring treats all onsets within a window as equally appropriate, although their usefulness varies with timing. For the first limitation, we derive intervention-specific valid windows from the temporal anchors annotated in § 3.3: for reactive guidance and for proactive guidance. Unlike a fixed timing window around the reference utterance, each window opens after the user’s turn ends or at the first sign of need, and closes when the intervention is no longer useful. An onset is valid if it falls anywhere within the window of the corresponding intervention.
We validate these windows with a human study that isolates timing from content. For each of 79 WTaG instructor utterances, we keep the utterance fixed and sample six candidate onsets around its annotated anchors, spanning early, in-window, and late times and including the instructor’s actual onset. Three researchers rate how appropriate it would be to deliver the same utterance at each onset on a 0-10 scale (See Appendix § 10.4 for more details). We treat an onset as acceptable when its mean human rating is at least 6, and compare this label with whether the onset falls within the proposed intervention window. We report Macro-F1 to measure how well the window classification agrees with human judgments across acceptable and unacceptable timings. Overall, our evidence-derived windows outperform the fixed s window around reference utterance onsets adopted from ProAssist [32] (Macro-F1: 0.788 vs. 0.713). The improvement is more pronounced for reactive guidance, where our windows exclude inappropriate onsets that would interrupt the user, while for proactive guidance, our windows achieve comparable performance while admitting earlier evidence-supported onsets missed by the fixed window.
For the second limitation, we introduce a continuous timing score that captures varying degrees of timing appropriateness. The score peaks at a preferred response time and decreases for earlier or later onsets. For reactive guidance, , since responding immediately after the user finishes speaking is preferred; for proactive guidance, , when sufficient evidence becomes available. An earlier utterance lacks supporting evidence, and a later one becomes less useful as the task progresses.
Next, we test whether a continuous timing score better reflects human preferences than binary in-window scoring. We start with a simple linear score that decreases from toward the window boundaries. For each utterance, we compute Spearman’s correlation between the scores assigned to the six candidate onsets and their human ratings. Averaged across utterances, the linear score achieves a correlation of 0.83, compared with 0.65 for binary in-window scoring.
We further refine the shape of the timing penalty. Let denote the onset of the system utterance matched to an intervention. For a proactive intervention with , we use a quadratic penalty that grows as moves further before . For at or after the preferred time, in for both reactive and proactive interventions, we use a cosine-based decay that penalizes short delays mildly and long delays strongly. We compare this score and the linear score with human ratings rescaled to using mean absolute error (MAE). The cosine-based score achieves lower MAE than the linear score (0.182 vs. 0.225) and is adopted as the final scoring function:
The parameters and control the early penalty and the decay rate, respectively. We set by cross-validated calibration against the same human ratings, as detailed in Appendix § 10.4.
Based on these, we define Timely Correctness as the average timing score over all necessary interventions. An intervention contributes zero if it has no content-matched system utterance or if that utterance’s onset falls outside its window. Optional interventions are excluded, as remaining silent is also acceptable. We additionally report Response Latency, the median delay from to the start of system speech for content-matched reactive responses.
Complementary Evaluation Metrics
Reference Fidelity. We measure agreement with human references using two metrics following Zhang et al. [32]. Similarity F1 measures embedding-based similarity, while LLM-Judged F1 assesses content equivalence using the criterion described above. Both use a fixed timing window around reference utterance onsets for intervention matching.
Intervention Precision and Turn-Taking. Timely Correctness measures whether necessary guidance is delivered correctly and on time, but frequent interventions may improve coverage at the cost of unnecessary or disruptive speech. We therefore introduce Intervention Precision, the proportion of all played system utterances that correctly match a necessary or optional intervention within its valid window. Optional interventions are counted as correct when produced, while remaining silent for an optional intervention is not penalized. Unmatched, repeated, incorrect, or mistimed utterances lower this score. Additionally, we report Speech Overlap Ratio, the proportion of user-speaking time overlapping with system speech, to measure potential disruptions to natural turn-taking.
FLUID-Agent: From Interaction Dynamics to Real-Time Guidance
Motivated by the interaction dynamics characterized in § 3, FLUID-Agent is a real-time system for flexible, evidence-grounded task guidance. As shown in Figure 3, it uses an asynchronous agentic architecture built around a single Qwen3.6-35B-A3B vision-language model (Qwen Team, [22]), which serves perception, utterance understanding, and intervention reasoning through task-specific prompting. The components communicate through a shared task memory that tracks task-relevant evidence and interaction context over time (see Appendix § 9.4 for implementation details).
Figure 3. Overview of the FLUID-Agent framework. Perception, task memory, and user utterance understanding jointly support evidence-grounded intervention decisions as task and interaction context evolve.
Perception and Task Memory. The perception module continuously analyzes keyframes sampled from short temporal windows of the live video stream. Guided by task knowledge and the current task state, it tracks relevant user actions, states, and task progress. These observations, together with user and system utterances and step-level completion status, are maintained in a continuously updated shared task memory, providing grounded task and interaction context for subsequent intervention decisions.
User Utterance Understanding. Not every user utterance calls for a response. The user utterance understanding module uses the utterance taxonomy and task context to distinguish utterances that require a response from those that do not, such as task narration, progress reports, and acknowledgments. This enables FLUID-Agent to respond when needed while avoiding unnecessary interruptions.
Evidence-Conditioned Intervention Decision. Each potential intervention is grounded in evidence from the ongoing task and interaction, such as an observed error, action progress, or user request. The intervention module uses this evidence and the interaction history to reason about whether, when, and how to provide guidance. As the task evolves, FLUID-Agent reassesses pending interventions, providing evidence-grounded guidance when appropriate or remaining silent when it is no longer warranted.
Asynchronous Execution. FLUID-Agent coordinates perception, utterance understanding, and intervention reasoning asynchronously, allowing each component to process new information as it becomes available. The system continues monitoring the task while speaking, yields to new user requests, and cancels pending guidance when it is no longer relevant.
Experiments & Results
Experimental Setup
Systems. We evaluate four representative baselines with FLUID-Agent: ProAssist 8B (Zhang et al., [32]), VideoLLM-online 8B (Chen et al., [5]), JoyAI-VL-Interaction (Video Understanding Team of JoyAI-VL @ Joy Future Academy, JD, 2026, and Gemini 3.8 Live (Google DeepMind, [11]. These baselines are selected based on four criteria: (1) streaming input and output, (2) proactive response generation, (3) training on task-guidance data, and (4) compatibility with the teacher-forced inference setting.
Protocol. To isolate the quality of individual responses in offline evaluation and avoid error propagation across the interaction, we use teacher-forced inference, conditioning each system on the ground-truth interaction history rather than its own previous outputs. All systems also receive the same structured task knowledge extracted from demonstration videos. They operate in real time with streaming video and audio inputs and spoken audio output, allowing us to measure response latency and speech timing. See Appendix § 10 for implementation details.
Main Results
We evaluate all models alongside the original human instructor guidance from both datasets under the FLUID-Evaluation framework. As shown in Table 1, human instructors substantially outperform all evaluated models, establishing a human performance baseline. By incorporating interaction dynamics, FLUID-Agent generally achieves more timely and precise interventions than other models. However, Gemini exhibits less speech overlap on both datasets, suggesting that effective turn-taking remains a challenge for FLUID-Agent. Substantial gaps remain between FLUID-Agent and human instructors, particularly in proactive Timely Correctness (below 0.07 vs. above 0.88) and Intervention Precision (below 0.17 vs. above 0.83). These results highlight the remaining challenges in providing timely and appropriate guidance.
| Metric | WTaG ProAssist | WTaG VideoLLM | WTaG JoyAI | WTaG Gemini | WTaG FLUID-Agent | WTaG Human | HoloAssist ProAssist | HoloAssist VideoLLM | HoloAssist JoyAI | HoloAssist Gemini | HoloAssist FLUID-Agent | HoloAssist Human |
| Reference Fidelity | ||||||||||||
| Similarity F1 ↑ | ||||||||||||
| LLM-Judged F1 ↑ | ||||||||||||
| Reactive Guidance | ||||||||||||
| Timely Correctness ↑ | ||||||||||||
| Response Latency (s) ↓ | ||||||||||||
| Proactive Guidance | ||||||||||||
| Timely Correctness ↑ | ||||||||||||
| Intervention Precision and Turn-Taking | ||||||||||||
| Intervention Precision ↑ | ||||||||||||
| Speech Overlap Ratio ↓ |
Table 1. Evidence-grounded evaluation on WTaG and HoloAssist. Values are reported with half-widths of 95% session-cluster bootstrap confidence intervals.
Evidence-grounded evaluation captures intervention quality beyond reference matching. Systems with similar LLM-Judged F1 can exhibit substantially different interaction behavior. On WTaG, Gemini and VideoLLM achieve comparable F1 scores (0.127 vs. 0.138), yet Gemini has much higher Intervention Precision and substantially less speech overlap. Similarly, on HoloAssist, FLUID-Agent and JoyAI achieve nearly identical F1 (0.099 vs. 0.098), while FLUID-Agent achieves higher proactive Timely Correctness and Intervention Precision. By accounting for intervention necessity and evidence-derived timing windows, our evaluation reveals differences in intervention quality that reference-matching metrics cannot capture.
Fine-grained evaluation reveals interaction-specific strengths and failures. Figure 4 reports correctness by trigger type and instructor utterance type. FLUID-Agent leads across multiple categories on both datasets. Across systems, correctness is generally higher for explicit user queries, but remains low for proactive interventions requiring an understanding of task progress and user behavior. Correctness remains low for Error and Stuck triggers, as well as Question to User utterances that require seeking missing information. These results highlight the challenge of grounding proactive interventions in task evidence and deciding when and what to say, motivating our diagnostic analysis in Section 6.3.
Figure 4. Fine-grained intervention correctness (%). Percentage of necessary interventions correctly addressed by each system, grouped by trigger type and instructor taxonomy. Gray cells indicate fewer than 10 interventions.
Diagnostic Analysis with Evidence Ladder
While the preceding results reveal difficulties in deciding when and what to say, it remains unclear how intervention decisions and evidence grounding contribute to these difficulties. To help better understand what types of evidence is difficult to capture and affects agent’s performance, we construct an evidence ladder on WTaG that progressively controls intervention decisions and introduces ground-truth task evidence. L0 retains native behavior; L1 enforces one response at each annotated necessary-intervention onset while suppressing other speech; L2 adds the evidence type; and L3 further provides specific observed facts for action- and state-triggered events (see Appendix § 10.3 for details). None of these levels provides the reference guidance.
Deciding whether and when to intervene is a major bottleneck. The largest gains occur at L1 (Table 2), where reactive and proactive Timely Correctness increase by 0.269 and 0.105 for JoyAI, and by 0.050 and 0.195 for Gemini, respectively. Intervention Precision also improves, reflecting fewer unnecessary interventions. Together, these gains show the importance of deciding whether and when to intervene in real-time task guidance.
| Metric | L0 | L1 | L2 | L3 | |||
| JoyAI | |||||||
| LLM-Judged F1 | 0.138 0.020 | 0.314 0.034 | 0.330 0.035 | 0.346 0.038 | +0.175 0.029 | +0.016 0.013 | +0.016 0.013 |
| Reactive Timely Correctness | 0.107 0.020 | 0.375 0.035 | 0.409 0.037 | 0.411 0.037 | +0.269 0.035 | +0.034 0.020 | +0.002 0.008 |
| Proactive Timely Correctness | 0.023 0.012 | 0.127 0.033 | 0.119 0.034 | 0.178 0.035 | +0.105 0.032 | −0.008 0.026 | +0.059 0.037 |
| Intervention Precision | 0.080 0.014 | 0.347 0.037 | 0.378 0.040 | 0.398 0.037 | +0.267 0.035 | +0.031 0.017 | +0.021 0.015 |
| Gemini | |||||||
| LLM-Judged F1 | 0.127 0.019 | 0.301 0.029 | 0.313 0.030 | 0.362 0.027 | +0.174 0.024 | +0.012 0.023 | +0.049 0.021 |
| Reactive Timely Correctness | 0.126 0.028 | 0.176 0.027 | 0.195 0.028 | 0.217 0.031 | +0.050 0.030 | +0.019 0.020 | +0.022 0.026 |
| Proactive Timely Correctness | 0.063 0.026 | 0.259 0.044 | 0.273 0.044 | 0.354 0.052 | +0.195 0.045 | +0.015 0.036 | +0.080 0.043 |
| Intervention Precision | 0.145 0.023 | 0.291 0.031 | 0.310 0.035 | 0.348 0.038 | +0.145 0.028 | +0.019 0.026 | +0.038 0.030 |
Table 2. Evidence ladder on WTaG. L0 retains native behavior; L1 enforces interventions at evidence onsets; L2 adds trigger types; and L3 adds specific evidence. columns show differences (L1−L0, L2−L1, and L3−L2). Blue and orange shading highlight gains from onset control and specific evidence for proactive guidance, respectively.
Specific evidence improves proactive guidance. Coarse evidence at L2 provides limited gains, as it lacks concrete details about the ongoing task. In contrast, providing specific observations at L3 improves proactive Timely Correctness by 0.059 for JoyAI and 0.080 for Gemini. Gains in reactive guidance are smaller, as user utterances already provide relevant evidence. These results suggest that strong grounding in the ongoing task is key to effective proactive guidance.
FLUID-Agent Ablation
We ablate the two decision modules in Figure 3 to assess taxonomy-guided reactive and evidence-grounded proactive decisions. Direct Reactive replaces User Utterance Understanding with an LLM that directly decides whether and how to respond to each user utterance. Direct Proactive replaces Intervention Decision with an LLM that directly decides whether to speak. Both retain the same backbone, perception pipeline, and task memory (see Appendix § 9.4.§ 3 for details).
Taxonomy-guided understanding reduces unnecessary reactive responses. Compared with Direct Reactive, FLUID-Agent achieves comparable Reactive Timely Correctness on WTaG and higher correctness on HoloAssist, with higher Intervention Precision and fewer reactive utterances on both datasets. Direct Reactive responds more often to turns requiring no answer (see Response Rate on Unanswered Turns), supporting the role of taxonomy guidance in deciding when to respond and when to remain silent.
Evidence-conditioned intervention decisions enable more selective proactive guidance. Compared with Direct Proactive, FLUID-Agent achieves higher Proactive Intervention Precision with roughly 11 times fewer proactive utterances and substantially less speech overlap on both datasets. Although Direct Proactive achieves higher Proactive Timely Correctness, over 60% of its utterances repeat earlier responses (see Repetition Rate). These results show that structured intervention decisions enable more appropriate proactive guidance while reducing redundant and unnecessary speech.
Discussion and Conclusion
Recent advances in vision and language foundation models as well as agentic architecture have made it feasible to develop perceptual agents that can provide real-time task guidance to help people acquire new skills and complete physical tasks. However, evaluating perceptual guidance agents remains a significant challenge due to the inherent flexibility of instructional dialogue, where appropriate guidance can vary in content and timing.
To address this issue, we developed FLUID, a framework that evaluates not only the accuracy of generated guidance but also the flexibility of AI agents in providing different types of guidance based on users’ needs and task contexts. FLUID grounds its evaluation metrics in human instructor guidance, focusing on the evidence supporting the guidance, its necessity, and the appropriateness of its timing. Our current FLUID framework
contains instructional interaction data, fine-grained annotation and evaluation protocols. It can be used to support systematic and comparative evaluation of AI agents’ guidance capabilities. Our experimental results show that, while human instructors achieve high performance under FLUID metrics, state-of-the-art AI models still lag significantly behind. These findings highlight the substantial gap between perceptual agents’ capabilities and proficient human instruction.
For tasks that require specialized procedures, such as special daily tasks relevant to different people, FLUID provides a pipeline to establish evaluation metrics based on a few examples of natural interaction sessions between an instructor and a follower. FLUID can then provide an initial assessment of agent behavior and reliability before deploying agents in real-world user tasks.
Evaluating real-time guidance systems is an extremely challenging problem. Ultimately, these agents must be evaluated through real-time interactions with actual users. However, such evaluations are prohibitively expensive and difficult to scale. FLUID provides a way to bootstrap the development and evaluation of agents’ guidance capabilities before moving to real-world user studies. We hope FLUID will offer new perspectives on how to evaluate real-time guidance systems and motivate the development of perceptual guidance agents capable of providing flexible, contextually appropriate, and timely guidance comparable to that of skilled human instructors.
Limitations
Our annotations primarily recover the grounded evidence underlying observed instructor interventions, which provides a strong reference for evaluation but does not exhaust all potentially valid interaction behaviors. In particular, the absence of an intervention in the demonstration does not imply that no intervention would have been appropriate at that moment. We also do not explicitly model silent moments, partly because WTaG and HoloAssist contain few cases where speaking is clearly prohibited; many additional utterances could be non-harmful as long as they do not disrupt task progress. Identifying such unobserved but potentially valid intervention opportunities, as well as moments where silence is truly required, involves more subjective judgments and is left for future work.
Our framework supports a broad range of proactive and reactive assistance behaviors in complex task interactions. However, more fluid and dynamic interaction scenarios, such as full-duplex perception and generation, remain challenging, as overlapping perception and response generation are difficult to support naturally within a modular system. In the datasets, we do observe several dynamic interactive behaviors, including backchanneling, overlapping speech, and visual barge-ins, and some of these are partially reflected in our evaluation. However, a comprehensive characterization and evaluation of richer full-duplex interaction is beyond the scope of this work. Future work could study these behaviors more systematically, supported by interaction data with denser and more diverse back-and-forth dynamics.
Acknowledgments
This work was supported in part by the Johns Hopkins Artificial Intelligence and Technology Collaboratory (JH AITC), funded by the National Institute on Aging of the National Institutes of Health under grant P30AG073104. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Appendix
Dataset Construction and Preprocessing
WTaG and HoloAssist record task guidance in which a remote instructor observes the user’s view and provides verbal assistance. This appendix describes how sessions were selected from the two datasets, how their video, audio, and transcripts were prepared, and how the resulting sessions differ in their interaction.
Source Datasets and Session Selection
From WTaG, we exclude sessions with incomplete task executions. From HoloAssist, we select sessions with substantive user–instructor interaction, excluding those dominated by repetitive next-step instructions and acknowledgments, and then manually check the selected recordings for audio quality. Table 4 reports the final dataset sizes.
| WTaG | HoloAssist | |
| Video hours | 7.57 | 6.29 |
| Test sessions | 43 | 72 |
| User utterances | 2,324 | 1,358 |
| Instructor utterances | 1,482 | 2,736 |
Table 4. Statistics of the two source datasets. HoloAssist instructor utterances are counted as native transcript rows; merging fragments of one contribution gives 2,731 units.
Multimodal Preprocessing and Synchronization
Video, audio, and annotation timestamps are measured in seconds from the start of each recording. Video is sampled at 8 Hz using presentation timestamps, taking the latest frame available at each sampling time.
For the transcripts, we manually revise the WTaG transcripts to correct speech boundaries and restore missing utterances, and keep HoloAssist’s released transcripts and timing. These speaker-labeled transcripts are used for annotation and evaluation; at runtime, user text comes from streaming automatic speech recognition (ASR) instead.
For the audio, instructor speech is muted, except where it overlaps user speech. There it is left in place to preserve the user’s signal, so the instructor may remain audible in overlap regions. Recordings are replayed continuously. Streaming ASR produces partial transcripts during an utterance and a final transcript once the utterance ends. These outputs become available only after the corresponding audio has been received and processed.
Dataset Statistics and Interaction Characteristics
HoloAssist sessions are shorter than WTaG sessions but contain denser instructor guidance. The median session duration is 265.8 s in HoloAssist and 594.0 s in WTaG, while instructors speak 8.25 and 3.21 turns per minute on average per session, with a median interval between adjacent instructor utterances of 1.99 s and 7.19 s, respectively.
The two datasets also differ in the type of guidance (Figure 5). HoloAssist has a higher proportion of instructions, while WTaG has higher proportions of answers, approvals, and corrections. This pattern is consistent with denser ongoing instruction in HoloAssist and more frequent answers and corrections in WTaG. Correction labels describe the instructor’s guidance, not the frequency of actual task errors.
Figure 5. Utterance taxonomy distributions in WTaG and HoloAssist.
The necessary interventions in the released annotations show a corresponding difference in how guidance is initiated. In WTaG, 66.1% are reactive and 33.9% proactive; in HoloAssist, 31.4% and 68.6%. WTaG therefore puts more weight on responding to users, while HoloAssist more often requires guidance that is not prompted by a user request.
Evidence-Grounded Annotation Protocol
The annotations connect each instructor intervention to the user speech, visible actions, and task states that support it. This appendix defines the four annotation layers, including the rules for intervention necessity and timing, then describes the annotation workflow and the review procedures for WTaG and HoloAssist, and closes with boundary cases and a worked example.
Annotation Overview and Label Schema
We annotate 43 WTaG and 72 HoloAssist test sessions with the four linked layers of § 3.3. Layer 1: Utterance Taxonomy assigns each turn a communicative-function label and at most one attribute. Layer 2: Action and State locates visible task evidence in time and records its validity and task effect. Layer 3: Trigger Link connects each instructor intervention to its supporting evidence and records its trigger type and intervention status. Layer 4: Evidence Onset and Intervention Window records the temporal anchors that define when the intervention should begin. Three additional WTaG sessions provide task knowledge only and are not annotated or evaluated.
The unit of annotation for speech is the utterance unit, one speaker contribution in the cleaned transcript. Fragments of one contribution that were split by speech recognition or transcript cleaning are merged before labeling. A contribution is split into several units only when its functions are clearly separable, for example There is already water in it. (U7) and Should I pour it out? (U2); otherwise it receives the label of its main communicative function.
Layer 1: Utterance Taxonomy
This layer labels the communicative function of each utterance, that is, what the speaker is asking, reporting, or responding to. Users and instructors have separate label sets, and labels are assigned from the transcript and the conversational context. For brief responses, the position of the turn in the interaction also matters, since it separates acknowledgments from backchannels (U5 vs. U10; I10 vs. I11).
User Utterance Taxonomy
Eleven labels describe user turns: requests, responses, task reports, and speech with little or no task-related content (Table 5). When more than one label could apply, annotators check the labeling rules in the order U10, U9, U11, U8, U3, U2, U1, U4, U5, U7, U6 and use the first one that fits. An incomplete utterance that clearly asks a question is not labeled U9 but receives one of the three question labels: U3 asks about something the instructor recently said, U2 proposes an answer about the task or the world and asks whether it is correct, and U1 asks for information without proposing an answer. Appendix § 9.2.7.1 explains how these labels are separated when the interpretation depends on context.
| ID | Label, definition, and example |
| U1 | Information request. An open request for task content the user lacks; no candidate answer is proposed. Umm, how much oil? |
| U2 | Verification question. Proposes a candidate object, amount, action, plan, interpretation, state, or result and requests validation, as a question or as a declarative awaiting confirmation. It is a teaspoon, right? |
| U3 | Clarification request. Requests repetition, rephrasing, or confirmation of the user’s hearing or understanding of an instructor utterance. What do you mean by pinch, sorry? |
| U4 | Answer. Supplies information requested by an instructor question, state check, or check instruction. Five. |
| U5 | Acknowledgment/uptake. Signals receipt, understanding, or acceptance of a completed instructor turn without new task content, including an echo the user closes themselves. Twelve ounces? Okay. |
| U6 | Task narration. Propositional verbalization of the user’s own action, plan, reasoning, need, decision, or progress, including reading instructions aloud. I already added the sugar. |
| U7 | Trouble report. Reports that something is wrong or has deviated from the expected course. Wait, that wasn’t a tablespoon. |
| U8 | Affective expression/social talk. Emotion, evaluation, politeness, apology, laughter, or conversation not required for task completion. Oh my God. |
| U9 | Filler/hesitation. Non-propositional or self-directed speech: fillers, false starts, fragments, muttered reading, self-directed pacing tokens, or unintelligible speech. And, uh. . . |
| U10 | Backchanneling. A brief continuer produced while the instructor is still speaking. Mm-hm. |
| U11 | Meta-communication. Discusses or manages the interaction or session itself, such as protocol, recording, apparatus, or audio and camera tests. Can you still hear me? |
Table 5. User utterance taxonomy. Examples illustrate a label but do not replace the context-dependent rules in Appendix § 9.2.7.1.
Instructor Utterance Taxonomy
Thirteen labels describe instructor turns (Table 6). They record what the instructor is doing in the conversation, regardless of whether the content is correct. When more than one label could apply, annotators check the labeling rules in the order I12, I13, I8, I5, I9, I6, I2, I1, I4, I3, I7, I11, I10 and use the first one that fits. A correction is therefore labeled as a correction even if it also expresses approval or answers a question, and information given in response to a user request is labeled as an answer rather than as an instruction offered without a request. For turns with little substantive content, the label depends on timing: I11 while the user is still speaking or performing an action, and I10 after the user has finished.
| ID | Label, definition, and example |
| I1 | Answer. Supplies task content the user requested explicitly or implicitly, through a question, clarification request, stated need, trouble report, or audible search. The filter cone is the metal thing. |
| I2 | Approval. Validates a concrete user candidate, action, state, or result as correct or acceptable. Yep, like that. |
| I3 | Instruction. Introduces or elaborates a required task step on the instructor’s own initiative, without presupposing an error. Grind it for about twenty seconds. |
| I4 | Advisory. Gives optional, cautionary, conditional, or alternative guidance that does not repair an existing error. You can use the glass bowl if that is easier. |
| I5 | Question-to-user. Requests information about the task state, the user’s intent, the user’s speech, or the user’s understanding. How many toothpicks do you have? |
| I6 | Reassurance. Mitigates user worry, trouble, or a mishap by marking it as tolerable or recoverable. That is okay, no worries. |
| I7 | Pacing/closure. Regulates starting, continuing, waiting, transitioning, or ending without substantive procedural content. Keep going. |
| I8 | Environmental repair. Asks the user to improve the instructor’s incoming visual or audio evidence. Could you look down more? |
| I9 | Correction. Repairs an actual, proposed, imminent, or completed user error or an omitted step. No, use the jelly instead of the marmalade. |
| I10 | Acknowledgment. Signals receipt of a completed user contribution without information, evaluation, reassurance, or flow direction. Oh, okay. |
| I11 | Backchanneling. A brief, low-content signal of attention while the user is still speaking or performing an extended action. Mm-hm. |
| I12 | Filler/self-repair. Hesitation, an abandoned start, or a repair of the instructor’s own wording or instruction. Sorry, I meant one teaspoon. |
| I13 | Other. Meta-communication, session or recording management, connection checks, social talk, or rare residual content. You can stop the recording. |
Table 6. Instructor utterance taxonomy.
Utterance Attributes
An attribute records what a turn is about, while the function label records what the turn does. Each turn takes at most one category-specific attribute (Table 7); U4, U5, U9–U11, I6, I10, and I13 take none. U2, I1–I4, and I9 share content_target, whose values are object_tool, action_method, quantity_parameter, sequence_step, state_result, and other_unclear, so that a verification and its response can be compared on the same content axis. Figure 2 shows the label and attribute together, such as Verification—quantity.
Layer 2: Action and State
Annotators identify actions and task states from visible evidence in the video, using the dialogue as additional context when needed, and mark the start and end time of each. Each interval is written as , which includes the start time and excludes the end time. The interval of an action is called an action span, and each span also receives a validity label and a task-effect label, defined below. Table 8 lists the fields recorded for each WTaG annotation.
| Field | Values or boundary rule |
| Unit kind | action (motion directed toward a task goal) or state_event (a task condition or a change in that condition). |
| Times | Start: the first frame that clearly shows the action or state. End: the first frame after it ends or changes identity. |
| Verb/noun | A short description supported by the video, e.g., grab–fork. |
| Validity | VALID, DEVIATION, AMBIGUOUS. |
| Task effect | NO_PROGRESS, PROGRESS, COMPLETION, DEVIATION_ONSET, RECOVERY, UNKNOWN. |
| State label | For state_event units: STALL, SEARCHING, VISIBILITY_LOSS, VISIBILITY_RECOVERY, SALIENT_STATE_CHANGE. |
Table 8. Fields used to annotate actions and task states in WTaG.
An action span covers one continuous attempt to achieve a task goal. It may include several hand movements as long as the goal, object, and validity stay the same; a new span starts when any of these changes, or when the user begins another attempt after disengaging their hand or resetting the task state. If an action is briefly occluded, the span continues when continuous motion lets us identify it across the occlusion. If the action cannot be identified reliably, the uncertain interval is left unannotated.
Validity indicates whether the observed behavior is appropriate given the task instructions and the current state: VALID when it follows these requirements, DEVIATION when it conflicts with a task goal or constraint, and AMBIGUOUS when the available evidence is insufficient to decide.
| Labels | Attribute and focus |
| U1 | request_type: information requested. |
| U2, I1–I4, I9 | content_target: task content. |
| U3 | repair_target: hearing, meaning, or task detail. |
| U6 | narration_type: action, progress, reasoning, or reading aloud. |
| U7 | trouble_type: problem reported. |
| U8 | expression_type: affective, task-related, or social talk. |
| I5 | question_purpose: information sought. |
| I7 | flow_function: starting, continuing, waiting, transition, or closure. |
| I8 | repair_target: visual or audio access. |
| I11 | backchannel_target: speech, action, progress, or affect/trouble. |
| I12 | repair_form: filler or self-repair. |
Table 7. Utterance attributes. The two uses of repair_target have different value sets.
Task effect records how the action changes the task state. PROGRESS marks an action that advances a step or subgoal, and COMPLETION one that completes a step or subgoal or reaches a meaningful task state. DEVIATION_ONSET identifies the action that first creates an erroneous or undesirable state, and RECOVERY an action that begins or completes the correction of that state. NO_PROGRESS marks an action with no substantive advancement, and UNKNOWN one whose effect cannot be determined reliably. A span can carry several task-effect labels if the task state changes during the same continuous action, for example when an action first makes progress and then completes a step.
Action Vocabulary
WTaG action descriptions use task-specific vocabularies developed from the recipes and checked against the videos. Qwen3-VL-30B-A3B-Instruct generated initial verb, noun, and adjective lists from the task steps, their execution variants, and the 48 HoloAssist verbs as a reference, and then checked sampled frames from 506 video segments for missing words. Annotators reviewed the words supported by at least two segments and merged synonyms. Compound objects are represented by a noun and an adjective, such as jar with nut-butter; locations and hand use are recorded separately. A final video check with Qwen3-VL-32B-Instruct added remove and shake and found no missing nouns. The resulting vocabulary contains 83 verbs, 120 nouns, and 107 adjectives. Observations outside the vocabulary are retained as out-of-vocabulary descriptions rather than forced into an unsuitable category.
Action-Candidate Generation and Human Review
WTaG action annotations combine model proposals with a human scan of the full task timeline. Four models independently propose intervals and descriptions: Qwen3.5-27B, Qwen3.6-35B-A3B, Qwen3.8-27B, and Qwen3-VL-32B-Instruct. Temporally clustered proposals supported by at least two models enter review, including incompatible proposals that meet this threshold; the votes decide only which candidates are reviewed, not their validity or boundaries. Reviewers resolve the candidates using the video and task context and add manual pins for missed events, which follow the same review procedure. Any candidate may be accepted, edited, merged, split, or rejected: of 2,611 model proposals, 665 were kept unchanged, 1,423 were kept after editing, and 523 were rejected. After this review, a separate model-based pass assigns validity and task effect to the reviewed actions using the task context and the annotation guidelines.
Layer 3: Trigger Link
For every instructor utterance, the annotator selects the smallest set of preceding evidence that warrants the intervention. A link records the instructor utterance ID, the evidence source, one or more evidence IDs, the trigger type, a confidence, and optional ambiguity flags; Table 9 gives the controlled values.
| Field | Controlled values |
| Source | USER_UTTERANCE, USER_ACTION_STATE, MIXED, INSTRUCTOR_SELF, NONE_FOUND. |
| Trigger type | USER_QUERY, USER_REPORT, ERROR, PROGRESS_OK, STEP_DONE, STUCK, RECOVERY, VISIBILITY, UNCLEAR_STATE, DIALOG_OTHER, SELF, NONE. |
| Confidence | low, medium, high. |
| Audit flags | unclear_source, competing_evidence, missing_evidence, boundary_uncertain. |
Table 9. Trigger-link schema. Confidence concerns the evidence identity and relation, not transcription quality.
The source is MIXED only when neither modality alone is sufficient. For example, Does that look good? asked while the user is rolling the tortilla links the instructor’s Yeah to both the U2 turn and the visible action. An intervention is reactive if
its trigger evidence includes a completed user utterance (USER_UTTERANCE or MIXED) and proactive otherwise, so this intervention is reactive. Whether an intervention is reactive is determined by its trigger link, not its utterance label: an I1 Answer is proactive when it responds to a visible search without a user turn, such as searching for a half-teaspoon.
Intervention Necessity
An intervention is necessary when the user needs it to avoid a mistake, get unstuck, or obtain needed information; optional when it can be omitted while the user proceeds correctly; and excluded when it is task-irrelevant or requires knowledge outside the shared view, the dialogue, or the task knowledge. Table 10 assigns these statuses from the trigger type, the linked user label, and the instructor label. When a link has several trigger types, the highest status applies, in the order necessary > optional > excluded. Only necessary interventions are scored by Correctness and Timely Correctness; optional interventions take part in matching (§ 4.1) but are not scored.
| Trigger type | Condition | Status |
| Any | Relies on knowledge that cannot be obtained from the shared egocentric view, the dialogue, or the task knowledge | Excluded |
| USER_QUERY, ERROR, UNCLEAR_STATE, VISIBILITY, STUCK | None | Necessary |
| Any | Linked user utterance is U1, U2, U3, or U7 | Necessary |
| STEP_DONE, PROGRESS_OK, RECOVERY, USER_REPORT | Instructor label is I1, I3, I8, or I9 | Necessary |
| STEP_DONE, PROGRESS_OK, RECOVERY, USER_REPORT | Any other instructor label | Optional |
| DIALOG_OTHER | None | Optional |
| NONE, SELF | None | Excluded |
Table 10. Rules for intervention status.
For example, Yep in response to Twelve ounce right? (U2) is necessary, while Great after a correct pour (PROGRESS_OK, I2) is optional (Figure 2). On WTaG, the rules assign 1,158 of 1,482 statuses (78.1%), 121 model proposals are retained after review (8.2%), and annotators set 203 (13.7%); every status is reviewed. On HoloAssist, Qwen3.8-27B assigns 2,492 of 2,736 statuses (91.1%), the rules assign 235 (8.6%), and annotators set 9 (0.3%).
Layer 4: Evidence Onset and Intervention Window
Temporal anchors define when the target instructor intervention can begin. They are annotated against a streaming prefix: evidence after time may confirm that a condition continues and may identify a later deadline, but may not be used to claim that enough evidence was already available at . In the main text, ES, EO, LU, and R are written , , , and .
Action-Triggered Interventions. Proactive interventions use three anchors. Evidence Start (ES) is the earliest frame showing a specific sign that the intervention may be needed, even if the condition is not yet certain; entering a task step alone does not establish ES. Evidence Onset (EO) is the earliest frame that makes the condition unambiguous and complete enough to justify the intervention; the whole object need not be visible, but its identity and task relation must be supported. Latest Useful Time (LU) is the exclusive boundary after which the target intervention is no longer useful on the recorded trajectory. The valid window is , with the timing score peaking at EO (§ 4.2).
Utterance-Triggered Interventions. For a reactive intervention, including one with MIXED evidence, the first eligible time is the Response-Ready Time (R), the end of the completed user turn in the trigger evidence; ES and EO are not separately annotated for utterance-triggered interventions. Partial speech does not set R unless the utterance is already semantically complete and only a delayed transcription endpoint remains. The reactive window is .
Operational Definition of LU. LU is the earlier of task expiry and dialogue cutoff on the recorded trajectory:
Task expiry occurs when the user completes the relevant action, starts self-correction or recovery, or reaches a state in which the intervention would have to be reformulated. Dialogue cutoff is the start of the first acknowledgment, answer, or downstream action that presupposes the target intervention, since inserting the intervention later would reverse the recorded conversational order. Unrelated speech does not set LU. For self-correction, LU marks its start rather than its completion.
Frame Selection and Precision. Annotators scrub coarsely, then step frame by frame around each boundary and select the earliest frame that satisfies the definition. Stored timestamps are in seconds at the source video’s frame resolution and are not rounded to whole seconds.
Annotation Workflow and Quality Control
Annotation proceeds layer by layer. Annotators first review segmentation, transcripts, and utterance labels, then action candidates and manual pins, and finally trigger links, intervention statuses, and temporal anchors. Completed upstream layers stay visible during review, and errors found in them are resolved with the annotator responsible. The three annotators are authors and received no additional compensation. They developed the rubrics together, annotated a shared session, and discussed disagreements before annotation began, and guideline changes led to re-review of the affected annotations. Review coverage differs between the two datasets (Table 11).
Before release, schema, ordering, and reference checks are run and the flagged cases are resolved. The automatic checks reject unknown controlled values, missing referenced evidence, non-positive intervals, and impossible timing orders: proactive cases require ES EO and ES LU, and reactive cases require R LU. Human-audit flags cover competing triggers, missing evidence, and uncertain boundaries.
| Layer | WTaG (43 sessions) | HoloAssist (72 sessions) |
| 1. Utterance Taxonomy | Qwen3.6-35B-A3B; all reviewed | Consensus of four modelsa; audited |
| 2. Action and State | Four modelsa, 2 votes, plus manual pins; all reviewed | Original dataset actions and model-proposed state events; not reviewed |
| 3. Trigger Link | Links: Qwen3.8-27B. Statuses: rules (Table 10) and annotators. All reviewed. | Links: merged proposals of four modelsa. Statuses: Qwen3.8-27B and rules. Audited. |
| 4. Evidence Onset and Intervention Window | Model-proposed; all reviewed | Model-proposed, as for WTaG; audited |
Table 11. Annotation provenance for the four layers. aQwen3.5-27B, Qwen3.6-35B-A3B, Qwen3.8-27B, and Qwen3-VL-32B-Instruct. “Audited”: reviewed in eight sessions after release (Appendix § 9.2.§ 6.1).
Review Coverage and Audit
All layers of all 43 WTaG test sessions were manually reviewed. Figure 6 summarizes the changes to the WTaG drafts and to action/state boundaries: annotators adjusted 68.1% of the proposed ES anchors, 78.6% of the EO anchors, and 56.1% of the LU anchors, with median absolute shifts of 2.95 s, 1.46 s, and 0.95 s.
Figure 6. WTaG manual review. Top: review outcome of the AI proposals in the 43 test sessions; is the number of proposals per layer. Bottom: shift of the edited start and end times of the retained action and state proposals (10th, 25th, 50th, 75th, and 90th percentiles).
For HoloAssist, the released annotations were frozen before the experiments, and the audit is a quality check of that frozen release rather than a revision of it. After the freeze, we manually checked 399 utterances across eight test sessions, 9.75% of the released utterances, on all annotation layers except actions. Figure 7 reports the share of audited items that the auditors changed on each layer. All reported experiments use the released annotations as frozen; the corrections identified in this audit are not applied to them, so every system is scored against the same annotations.
Figure 7. HoloAssist audit. Annotation changes in the eight-session audit. is 399 utterances for labels and attributes and 301 trigger links for the other fields.
Release and Reproducibility Notes
Every utterance, action/state, trigger link, and timing record has a stable ID, and records refer to one another by these IDs rather than by matching transcript text. To reproduce or extend the annotations, follow the controlled values and boundary rules above, and leave uncertain cases explicitly marked, for example with AMBIGUOUS validity or an audit flag, rather than assigning a confident label.
Annotation Examples and Boundary Cases
This section first lists the label distinctions that depend on conversational context and then follows one intervention through all four layers.
Utterance-Label Boundary Cases
A turn’s role in the exchange determines its label, even when its wording is the same. The rules below separate requests, uptake, and task feedback.
U1 versus U2. Use U2 only when the user proposes a concrete candidate that can be affirmed or rejected; otherwise use U1. Thus Which spoon should I use? is U1, whereas Should I use this spoon? is U2. A declarative such as That should be enough is U2 when it functions as a bid for validation. U2 versus U3. Use U3 when the object of the request is the instructor’s recent words themselves; a question about the task world is U2 even if it follows an instruction. What do you mean by pinch? is U3 with repair_target=meaning, and Thirty seconds, you mean? is U3 with task_detail, whereas a delayed check of remembered content, I think you said it was a teaspoon earlier, right?, is U2.
U4, U5, U9, and U10. Okay or yeah is U4 when it answers an instructor question, U5 when it takes up a completed instructor turn, U10 when it occurs while the instructor is still speaking, and U9 when it is self-directed. An echo is U5 when the user closes it (Twelve ounces? Okay.) and U3 when the user waits for confirmation (Twelve ounces?).
I1, I2, I3, and I9. I1 supplies content requested explicitly or implicitly by the user; the same content is I3 when volunteered as a required step. For a proposed candidate, I2 accepts it and I9 rejects or replaces it. Thus The teaspoon answers Which spoon? as I1, while Yes after This teaspoon? is I2 and No, use a teaspoon after Should I use a spoon? is I9.
I3 versus I4 versus I9. I3 is required guidance that presupposes no error; I4 is optional or cautionary guidance with no existing error (Be careful not to add too much salt); I9 repairs an actual, proposed, imminent, or omitted error (You added too much salt; remove some). Modal wording such as should, can, or might want to does not determine the label.
I10 and I11 versus Task Feedback. A receipt token such as okay is I10 after a completed user contribution and I11 during ongoing speech or action; if it evaluates correctness, it is I2. For example, Okay after the user answers Five is I10, while Yes after Is that correct? is I2.
End-to-End Four-Layer Example
Table 12 follows a wrong-utensil correction in session T2 through the four layers. The first sign of the error appears at 81.20 s (ES), and the fork becomes identifiable at 82.84 s (EO). The instructor starts the correction at 83.35 s. At 85.38 s, the user starts a compliant reach toward the knife, which sets the dialogue cutoff and therefore LU while the instructor is still speaking. The knife-grab span itself starts only at 85.54 s, when the grasped object becomes identifiable, but LU marks the start of the reach. LU can precede the end of the instructor turn because it concerns the usefulness of the intervention’s speech start time.
| Layer | Time (s) | Annotation |
| 1. Utterance Taxonomy | 83.35–87.85 | You should use the knife instead of the fork; I9 Correction; content_target=object_tool. |
| 2. Action and State | 81.20–83.80; 85.54–87.83 | grab–fork; validity DEVIATION; reviewed action a:356. The subsequent grab–knife action follows the correction. |
| 3. Trigger Link | action to utterance | USER_ACTION_STATE; ERROR; high confidence; no ambiguity flag; necessary. |
| 4. Evidence Onset and Intervention Window | ES 81.20; EO 82.84; LU 85.38 | First sign of a wrong-utensil choice; earliest unambiguous fork evidence; start of the compliant reach toward the knife (dialogue cutoff). |
Table 12. Complete four-layer example from session T2 (intervention T2_R0005).
Evaluation Protocol and Metrics
This appendix specifies the evaluation of § 4: the teacher-forced protocol and its real-time execution, intervention matching, the metrics, the LLM judges, and the statistical analysis. The annotations it reads are defined in Appendix § 9.2.
Evaluation Regime and Information Availability
Every system is evaluated under a teacher-forced setting: the interaction history it conditions on contains the human instructor’s utterances, released by the fixed rule of Appendix § 9.3.§ 2, while its own utterances are recorded for scoring and do not re-enter its context. This isolates each decision from earlier system errors and gives every system the same instructor history. The user side of the history is what the system itself perceives: the shared streaming-ASR transcript, or the user audio for Gemini (Appendix § 10.§ 1).
Teacher-Forced History Release Protocol
The -th instructor utterance, spoken over , enters the history at
where is the annotated latest useful time of the intervention it realizes (Appendix § 9.2.§ 5). An utterance therefore becomes visible only after it has been fully spoken and the opportunity it addresses has closed. An utterance without an annotated window, such as an excluded intervention, enters at , and release times are made non-decreasing in transcript order, so the history always follows the transcript order. A decision made at time sees utterance if and only if . How each baseline inserts released utterances into its history is described in Appendix § 10.§ 2.
Real-Time Execution and Spoken Response Timing
Each system processes each session in real time. The locally served systems (ProAssist, VideoLLM, JoyAI, and FLUID-Agent) run on one accelerator each with batch size one: a model call starts when it is issued or when the system’s previous call has finished, whichever is later, and occupies the accelerator for its measured duration, so queueing is part of every latency. Perception, decision, generation, speech synthesis, and sequential playback run on this clock; frames become available at their capture time and final ASR transcripts at their recorded availability time. Gemini is accessed through the vendor-hosted Live API: its audio and video are streamed at wall-clock speed, a response becomes audible when its first audio arrives, and the time spent rebuilding a session (Appendix § 10.§ 2) does not advance the replayed session timeline.
The evaluation reads every played system utterance with its audible onset , its playback end , and the last instructor utterance in the system’s history when was decided; utterances that were queued but never played are not candidates. Timing is read at the audible onset, as the user experiences it.
Intervention Event Construction and Matching
Events and Windows. An intervention event is an instructor utterance of the test set whose intervention status is necessary or optional; its side, reactive or proactive, is defined in Appendix § 9.2.§ 4. For event the annotation gives the start and end of the triggering user turn (reactive), the earliest sign and the evidence onset (proactive; when no earlier sign is annotated), and the latest useful time . The intervention window is
Candidates for event are the played utterances whose audible onset falls in the matching window , which extends to the surrounding interval of adjacent reference utterances:
rounding interval of adjacent reference utterances:
where is the end of the previous instructor utterance, the start of the next one, and the opening time of the next necessary event of the session, of either side ( for a reactive and for a proactive event); the session boundaries stand in when no such utterance or event exists. A candidate is admissible for if it was decided before the reference entered the system’s history, that is, if the last instructor utterance in its history precedes utterance .
Grading and Claiming. Each admissible candidate–event pair is graded once by the event judge of the event’s side (Appendix § 9.3.8) on the scale of Table 13. A pair is a content match at grade ; a grade-3 pair neither matches nor blocks a later match. Events are processed in the order of the instructor utterances, optional events included, and each claims the earliest still-unclaimed matching candidate by audible onset, regardless of timing. An utterance can satisfy at most one event. The metrics are read from the necessary events.
| Grade | Meaning | Use |
| 5 | The same guidance effect: the user would do and understand what the reference leads them to. | Match; F1 credit 1 |
| 4 | The same effect with a non-critical difference, such as an extra supported detail or a supported alternative. | Match; F1 credit 0.75 |
| 3 | Partly meets the need: something still needed is missing, or an unsupported claim could change what the user does. | No match; partial |
| 2 | Does not address this need: another turn, step, or problem, or narration where the reference guides. | No match; off-target |
| 1 | Works against this need: the user would do the opposite of what the reference directs. | No match; contradiction |
Table 13. Grading scale shared by the three judges.
Reference Fidelity Metrics
Similarity F1 and LLM-Judged F1 follow Zhang et al. [32] and are computed over all instructor utterances of the test set. A played utterance may pair with a reference if s and is admissible for . For Similarity F1, texts are encoded with the all-mpnet-base-v2 sentence encoder, pairs are assigned one-to-one by a cost that combines the time offset with the cosine distance, and an assigned pair is correct when the cosine similarity exceeds 0.5. For LLM-Judged F1, pairs are pre-filtered to each reference’s and each candidate’s four most similar partners, graded by the content judge (Appendix § 9.3.8), and assigned one-to-one at grade with credit 1 (grade 5) or 0.75 (grade 4) so as to maximize total credit. Precision divides the correct pairs (or credit) by the number of played utterances and recall by the number of references.
Timely Correctness and Response Latency
Timely Correctness. Timely Correctness assigns a temporal score to each necessary intervention and averages over all of them,
where is the number of necessary interventions of the side (reactive or proactive) and is the audible onset of the claimed utterance. Intervention is correct when it claims an utterance (Appendix § 9.3.§ 4) whose audible onset falls in its intervention window ; necessary interventions that are unmatched, incorrectly matched, or delivered outside their valid timing windows receive zero. The preferred response time is for a reactive intervention and for a proactive intervention. With the normalized positions
the temporal function of a proactive intervention is
and that of a reactive intervention is
We use , calibrated in Appendix § 10.4. With these values, the proactive score starts at 0.3 at the earliest sign, peaks at the evidence onset, and is zero at the latest useful time, and the exponent allows moderately delayed responses to retain partial credit. If , the early branch is empty.
Correctness, reported in Figure 4, is the share of necessary interventions that are correct, that is, Eq. (6) with .
Response Latency. Response Latency is the median of over the reactive necessary interventions whose claimed utterance begins at or after the end of the user’s turn. It is conditional on answering, so a system that answers few turns can still have a low latency, and its interval can be wide: Gemini’s HoloAssist latency in Table 1 is a median over 11 content-matched reactive responses from eight sessions.
Intervention Precision and Turn-Taking
Intervention Precision. Let be the played non-empty utterances of a run and the utterances that are the claimed response of some necessary or optional event and whose audible onset lies in . Intervention Precision is . Utterances that match no event, repeat a matched one, contradict the reference, or start outside the window lower the score. Optional events count because answering one constitutes a valid intervention, while leaving one unanswered costs nothing.
Speech Overlap Ratio. With the union of the session’s user speech intervals, the same for every system, and , the ratio is . The user audio is replayed and does not react to the system, so every overlap is attributed to the system.
LLM Judges and Evaluation Context
Configuration. Content agreement is the one part of the evaluation decided by a model; admissibility, claiming, timing, overlap, and aggregation are deterministic. One self-hosted Qwen3.8-27B model (bf16, temperature 0) serves every judge call for every system and run. Each call grades one candidate–reference pair from 1 to 5 with a short reason; only the grade enters a metric, and an unparsable output counts as no match. The complete judge prompts are provided in the released code repository.
Three Judges, One Scale. The reactive event judge, the proactive event judge, and the *content judge share the grading scale of Table 13, the context rules, and the output format. Each asks whether, after hearing the candidate, the user would do and understand the same thing as after the reference, where the user’s need at that moment is read from the trigger, the preceding dialogue, and the task knowledge rather than from the reference’s wording. For the reactive event judge the user’s turn sets the need; for the proactive event judge the observed action or state sets it, so that, for example, approving an action the instructor corrects is a contradiction. The content judge, used for LLM-Judged F1, receives no trigger and grades every reference utterance by its communicative function first, so a reference with no task content can be missed but not contradicted. A candidate that differs from the reference is credited when the task knowledge supports it and it leads to the same next action or understanding.
Judge Context. The judges read the benchmark’s ground-truth transcript, not the streaming-ASR transcript or audio that the evaluated system perceived. Every call contains the task knowledge text that every system received (Appendix § 10.§ 1), the event context (the benchmark dialogue before the reference, with the event’s own turn or observation marked), the event, the reference, the candidate context, and the candidate; the content judge receives the same message without the event block. The candidate context is the benchmark transcript cut at the candidate’s input cutoff, the start of the model call that produced it: the user turns that had started by then, the instructor utterances released into the system’s history by then, and the annotated user action in progress at that moment together with the last completed one. A user turn that was still in progress at the cutoff is given in full, so the candidate context can include the end of a turn that the system had not yet heard; it approximates what the system had available rather than reproducing it. It is kept separate from the event context, which may extend past the cutoff, so that the candidate is interpreted in the situation in which it was produced: a reply such as “yes” means what it answered at that time, and a candidate produced before the event occurred is off-target rather than a contradiction. A judge sees no timestamps, trigger types, necessity labels, windows, or system identity.
Aggregation and Statistical Analysis
Every metric is pooled over the test sessions of a dataset: event metrics over events, Intervention Precision over played utterances, the Speech Overlap Ratio over summed user speaking time, the F1 scores from pooled precision and recall, and Response Latency as the pooled median. Every reported value carries a 95% session-cluster bootstrap interval (10,000 resamples of sessions; tables print the half-width of the 2.5–97.5 percentile interval).
ProAssist, VideoLLM, JoyAI, and FLUID-Agent were each run twice under the same configuration and the reported values are the mean of the two runs; Gemini and the two ablation variants of Table 3 were run once. A difference between two systems or variants is computed on identical session sets as the pooled difference with a paired session-cluster bootstrap interval.
| WTaG FLUID-Agent | WTaG Direct Reactive | HoloAssist FLUID-Agent | HoloAssist Direct Reactive | |
| (a) Utterance Understanding | ||||
| Reactive Timely Correctness | 0.209 | 0.210 | 0.055 | 0.038 |
| Reactive Intervention Precision | 0.224 | 0.158 | 0.158 | 0.141 |
| Reactive Utterances/min | 1.94 | 2.99 | 0.86 | 0.96 |
| Response Rate on Unanswered Turns | 0.236 | 0.426 | 0.316 | 0.405 |
| Speech Overlap Ratio | 0.207 | 0.294 | 0.153 | 0.168 |
| (b) Intervention Decision | ||||
| Proactive Timely Correctness | 0.046 | 0.119 | 0.062 | 0.171 |
| Proactive Intervention Precision | 0.072 | 0.020 | 0.171 | 0.049 |
| Proactive Utterances/min | 1.32 | 14.83 | 1.55 | 16.88 |
| Repetition Rate | 0.095 | 0.619 | 0.089 | 0.684 |
| Speech Overlap Ratio | 0.207 | 0.752 | 0.153 | 0.799 |
Table 3. Ablation of FLUID-Agent decision modules.
FLUID-Agent Implementation
This appendix describes FLUID-Agent (§ 5, Figure 3) as run in every experiment, with one configuration on both datasets. A single frozen Qwen3.6-35B-A3B vision-language model (FP8 weights, vLLM, temperature 0) serves every online role through task-specific prompting: it perceives the scene, judges the task state, interprets and answers user turns, and generates spoken intervention utterances. Deterministic code around the model maintains the shared task memory, schedules the calls, evaluates the evidence conditions under which an intervention is opened, and manages turn-taking. The model therefore supplies the evidence, and the agentic structure decides what that evidence licenses.
Session Loop. When a visual change is detected, the perception module analyzes the video and adds an observation report to the task memory. The Situation Tracker (the task-state tracking part of the Task Memory block in Figure 3) reassesses the task state when the observed activity or the dialogue has changed, and the intervention conditions are re-evaluated after each judgment. In parallel, each completed user turn is handled by one call that interprets it and, when a reply is due, answers in the same model call. Under the teacher-forced protocol (Appendix § 9.3.§ 2), released instructor utterances are visible to perception, to the Tracker as testimony of what the user was told, and to utterance understanding; the system’s own outputs do not enter its prompts.
Perception and Task-State Tracking
Task Knowledge Compilation. An offline compiler (Qwen3.8-27B, distinct from the online backbone) reads the same task knowledge that every baseline receives (Appendix § 10.§ 1) and produces an action-level plan: each step becomes a few imperative actions with the tools and destinations the step names, its stated quantities, an observable completion check where the knowledge states one, and the step’s prerequisite relations. Task-specific procedural knowledge reaches the online model only through this plan, which supplies the checks the perception module answers, the reference for assessing the user’s progress through the task plan and detecting deviations, and the content of intervention utterances. The dialogue and the visual observations supply the task context of the moment.
Perception Calls. A novelty signal computed from DINOv2 features of the incoming frames triggers a perception call at a novelty peak, during sustained activity, or after a fixed interval. The perception call receives recent keyframes together with the dialogue associated with the current step and its completion checks, and reports the user’s hand activity, the current action, visible state changes, the check values, and per-step progress, marking whether each value was seen or inferred from the dialogue.
Task Memory. Perception reports are deterministically integrated into a belief state that tracks task progress, committed completion checks, and recent evidence for each step. A check value is committed when consecutive perception calls agree, a committed completion is retained while the check is out of view, and the memory records per step how often the action associated with that step was observed. The Tracker, the intervention conditions, and utterance understanding read this belief state rather than individual reports.
Situation Tracker. The Tracker is a text-only call. It runs after a perception update has been integrated into the belief state when the observed activity has changed, when a user turn or a released instructor utterance has arrived since its last judgment, when a pending intervention candidate awaits confirmation, or otherwise after a bounded interval, with a minimum spacing between calls. From the plan, its previous judgment, the step evidence of the belief state, and the recent reports interleaved with the dialogue, it returns the current step and action, each step’s state with its basis (observed, moved past, or the user’s testimony), at most one deviation with whether it matters, and wait and stuck states. It weighs evidence conservatively: the user’s behavior rather than the plan order defines the user’s position in the task plan, a deviation requires agreeing consecutive reports or one report plus the user’s own words, and instructor utterances count as testimony of what the user was told, not as evidence of what was done. A consequential deviation is accepted only when consecutive judgments agree, and a stuck state only when it persists; the resulting position in the task plan sets the checks of the next perception call.
Proactive and Reactive Intervention Decisions
Evidence-Conditioned Proactive Decision. After each accepted judgment, a fixed set of intervention conditions is checked against the current belief state. When a condition is satisfied, the agent creates a typed intervention candidate (typed by the trigger types of § 3.3) grounded in the current evidence, which may subsequently be spoken, held, or discarded as that evidence develops. Correction opens on an accepted consequential deviation and states the contrast between what the user is doing and what the action calls for. Progress coaching opens at the onset of an action with a stated amount, manner, or critical parameter while the user is manipulating task objects and no deviation is open. Completion and hand-off opens when a step is reliably done: the final step receives a closing confirmation, and a completed non-final step is followed by the next step’s instruction unless that step has already been given, by the system or by a released instructor utterance, or has already started or passed. Stall help opens on a persisting stuck judgment. A ledger makes sure that the same evidence is voiced only once per visit to a step, and a pending candidate is discarded when its evidence is withdrawn or its window closes. The content of an intervention utterance is assembled from the plan (and, for corrections, from the Tracker’s deviation description, which states what the user is doing and what the action calls for), and one short model call rephrases it as a spoken sentence.
Taxonomy-Guided Reactive Decision. Each final ASR transcript with speech content is handled by one model call; final transcripts without words and very short acknowledgment or filler transcripts are filtered without invoking the model. The call labels the turn with one of the user-utterance categories of Table 5 and, for labels that call for a reply, continues in the same model call with the answer. The label places the turn in one of three response tiers: requests for information, verification, or clarification and trouble reports are answered. Narration and answers to a pending question are voiced only when the user’s turn states something that can be checked against the task context and the model produces a brief confirmation or correction. Plain narration receives no reply, and the remaining labels are silent. A reply is one or two short spoken sentences grounded in the plan, the latest situation judgment, and the recent dialogue, and a failed call produces no response.
Turn-Taking and Speech. While the system speaks, incoming partial ASR transcripts are checked: a filler-only partial transcript lets it continue, and a question cue or a label requiring a reply causes the system to defer to the user. Deferring stops the current sentence, holds queued proactive responses until the user has finished (dropping those whose step is no longer current), and cancels unstarted answers. Sentences are synthesized by the shared speech server (Appendix § 10.§ 1) and played one after another while perception continues.
Ablation Implementation
The two variants of § 6.4 run the final system with one switch each. Each replaces one decision mechanism, together with the control logic attached to it, by a direct decision of the same model on the evidence that mechanism used, while the backbone, perception, task memory, and Situation Tracker stay those of the final system. The contrasts therefore show what the taxonomy-guided reactive decision and the evidence-conditioned proactive decision contribute to the system’s selectivity. They do not isolate the contribution of any single rule, since each variant also differs from the final system in the control logic noted below.
Direct Reactive. This variant replaces the taxonomy-guided utterance understanding (the category label, the response tiers, and the filler guard) by one direct decision of the model on the same turn context. The prompt keeps the reply rules without reference to the taxonomy, states no preference for silence, and asks the model to decide whether the turn needs a spoken reply now and, if so, to write it in the same call. Turn-taking on partial hypotheses and the proactive side are unchanged.
Direct Proactive. This variant replaces the evidence-conditioned proactive decision (the intervention conditions, the candidate ledger, and the templated content with its phrasing step) by one direct decision of the model every 2 s of session time, skipped while a previous decision is still being computed. The decision reads the same evidence products (the plan, the latest Tracker judgment, the step evidence, and the recent reports with the dialogue) together with the utterances the user has recently heard, and returns either a response text, spoken as written, or nothing. Proactive utterances are thus generated on a fixed schedule rather than on evidence-triggered candidates, and a newer utterance supersedes an older one that has not started playing; replies to user turns and speech already playing are unaffected. The reactive side is unchanged.
Ablation Diagnostics. Table 3 classifies each played utterance as reactive (a reply to a user turn) or proactive (an utterance the system initiated), as the system labels its own output. Reactive and proactive Intervention Precision restrict Intervention Precision (Appendix § 9.3.§ 7) to the utterances of that side, and utterances per minute count them per minute of session time. The response rate on unanswered turns measures the proportion of user turns that did not require an instructor response but nevertheless received a system response. A user turn is considered not to require a response if it is not linked as the trigger of any necessary or optional reactive intervention. Each turn is paired with its most-overlapping final ASR transcript; turns without a corresponding final transcript are excluded. The repetition rate is the share of proactive utterances whose text, after case and punctuation normalization, repeats an earlier proactive utterance of the same session verbatim.
Experimental Setup and Baselines
This appendix gives the conditions shared by the systems of § 6.1 and the adaptation of each baseline. FLUID-Agent is described in Appendix § 9.4 and the evidence-ladder conditions in Appendix § 10.3.
Common Experimental Setup
Systems. We evaluate FLUID-Agent and the four baselines of § 6.1, each with its released checkpoint or hosted model. To be compatible with the teacher-forced protocol, a system must allow instructor utterances to be inserted into its history at a specified time without its own earlier outputs; Appendix § 10.§ 2 states how each baseline realizes the release rule of Appendix § 9.3.§ 2. All systems run each session in real time on the clock of Appendix § 9.3.§ 3.
Shared Inputs. User audio and transcript: the three open-weight baselines and FLUID-Agent hear the user through one shared causal streaming-ASR stream (NVIDIA Nemotron streaming speech recognition, 0.6B parameters), produced once per session from the user-only audio track (Appendix § 9.1.§ 2) and replayed for every system with each final transcript’s recorded availability time, so recognition errors are shared; Gemini hears the same audio natively. Video: frames are decoded once per session by source timestamp; ProAssist and VideoLLM read the latest frame at 2 fps, JoyAI samples at its native 1 s ticks, Gemini receives 1 fps (the Live API maximum), and the perception trigger of FLUID-Agent runs at 8 fps. Task knowledge: every system receives the same structured task knowledge of a session, extracted from egocentric demonstration video by a multi-round vision-language pipeline: one text per task for WTaG, whose sessions follow a common recipe, and one text per session for HoloAssist, whose procedures vary across sessions. The text lists the steps with an instruction sentence and prerequisite relations and, for WTaG, the objects of the task and a completion criterion per step. It is rendered once and provided to each system in the input format its model
expects; the plan compiler of FLUID-Agent (Appendix § 9.4.§ 1) and the LLM judges (Appendix § 9.3.8) read the same text. Speech output: the open-weight baselines and FLUID-Agent speak through the same Kokoro text-to-speech server, sentence by sentence; Gemini produces speech natively, and the API’s output transcription is the scored text.
Models and Hardware. Table 14 lists the backbone, hardware, and decoding of each system. Accelerators differ across systems and Gemini runs on vendor infrastructure, so Response Latency reflects end-to-end performance under each system’s actual deployment configuration and should not be interpreted as a controlled comparison of model inference speed. Gemini was accessed through its API in September 2026. Besides the streaming-ASR and text-to-speech servers above, FLUID-Agent uses DINOv2 perception-trigger features computed on the CPU of the machine running FLUID-Agent, as well as offline plan compilation with Qwen3.8-27B (Appendix § 9.4.§ 1). For evaluation, one Qwen3.8-27B judge is shared across all systems (Appendix § 9.3.8). The number of runs per system is given in Appendix § 9.3.9.
| System | Backbone | Hardware | Decoding |
| ProAssist | Llama-3.1-8B + SigLIP | A40 | greedy; speak gate |
| VideoLLM | Llama-3-8B-Instruct + SigLIP | A40 | greedy; upstream keep-watching threshold |
| JoyAI | JoyAI-VL-Interaction | RTX PRO 6000 | official sampling, fixed seed |
| Gemini | Gemini 3.8 Live (Live API) | hosted | vendor defaults |
| FLUID-Agent | Qwen3.6-35B-A3B (FP8), one instance | RTX PRO 6000 | temperature 0 |
Table 14. Models, hardware, and decoding.
Baseline Configurations and Adaptations
Shared Guidance Instructions. Every baseline receives the same task-guidance objectives in the slot its training format uses for instructions: a role description (the camera is the user’s first-person view; speak when it helps, by confirming a completed step and giving the next one, correcting a visible mistake, helping when the user is stuck, or answering a question; otherwise stay silent), the task-knowledge text, delivery rules (one short spoken sentence, no preamble, no repetition, no scene description), and the instruction to speak only when supported by concrete visual or auditory evidence. The wording follows each model’s native prompt format, and the prompts are provided in the released code repository.
ProAssist. ProAssist receives frames at its native 2 fps with the official preprocessing. At every frame the speak gate compares the probability of the end-of-turn (not-talk) token with the threshold , which lies within the range of the model’s own ablation; the model stays silent when the probability exceeds the threshold and otherwise generates with the official greedy decoding loop. The dialogue opens in the checkpoint’s training form with the shared instructions and the knowledge text; final ASR transcripts enter as user turns and released instructor utterances as assistant turns.
VideoLLM. VideoLLM also receives frames at 2 fps. Its streaming loop is re-implemented in-process: at each frame the probability of the keep-watching token is compared with the upstream default threshold, and a response is decoded greedily when that probability falls below the threshold. The system prompt is the checkpoint’s own, and the shared instructions with the knowledge text form the first user turn, the trained instruction slot. Final ASR transcripts enter as user turns and released instructor utterances as assistant turns, and the context is rebuilt without the system’s own outputs.
JoyAI. The released checkpoint is served by a bare vLLM server, and the live adapter (ticks, action tokens, frame window) is re-implemented in our evaluation code, so the evaluated system is the released checkpoint with our real-time reconstruction of its adapter. At every 1 s tick the model receives the last 140 ticks in native format (frames, final ASR transcripts, and released instructor utterances at their ticks), with earlier ticks kept only as their text turns and the history rebuilt without its own outputs, and emits either a silence token or a response token followed by text. Decoding uses the official adapter’s sampling with a fixed per-request seed.
Gemini. The vendor-hosted Live model hears the user audio natively, watches the video at 1 fps, and decides itself when to speak. Its system instruction carries the shared instructions adapted to native audio and a note that the history it starts from is the real record of the session, with the instructor’s utterances in its own role. The teacher-forced history is realized by rebuilding: whenever an instructor utterance is released, the current Live session is closed and a new one is opened whose initial content is the recorded interaction so far (the user’s audio, frames at a reduced rate for the earlier part of the session, and the released instructor utterances as model turns), after which real-time streaming resumes. The model’s own earlier outputs are not part of a rebuilt history, and output produced while a history is being seeded is discarded.
Evidence Ladder Experimental Design
The evidence ladder (§ 6.3, Table 2) was run on the 43 WTaG test sessions with JoyAI and Gemini. Its levels change the conditions under which a system intervenes at the annotated necessary interventions and the information it receives about them, while what to say is left to the system at every level; no level gives the reference response. This appendix defines the levels and then describes how they are implemented for each system.
Ladder Conditions
Triggers. The triggers are the necessary WTaG interventions at their annotated onset: the end of the user’s turn for interventions whose trigger includes a user utterance, and the evidence onset for the others. Annotated relations from the same source within 1 s of each other are merged into one intervention. Interventions whose trigger times fall in the same one-second decision step form one trigger, timed at the latest of their trigger times; the same decision-step rule is used for JoyAI and Gemini. Table 15 defines the four levels.
| Level | Intervention decision | Information at the trigger |
| L0 | Native: the system decides itself whether and when to speak. | None. |
| L1 | Enforced: at each trigger the system is made to attempt one response; its speech elsewhere is suppressed. | That an intervention is required now. |
| L2 | As L1. | L1 plus the trigger type of each intervention, with a one-line gloss (§ 3.3). |
| L3 | As L1. | L2 plus a short observed-fact sentence for the interventions whose trigger includes annotated action or state evidence and for which such a sentence exists; the other interventions keep the L2 note. |
Table 15. Conditions of the evidence ladder.
At a trigger, the note is one sentence in the request (“Intervene now. Evidence requiring an instructor intervention is available. Type: not provided | type (gloss).”), to which L3 appends “Observed (video): fact.”; it is not kept in later history. The observed-fact sentences are short descriptions derived from the action and state annotations (Appendix § 9.2). An enforced response attempt can fail to produce a playable utterance (an empty generation, or a response not received within the API’s time limit), and such triggers count as no intervention. For JoyAI, a trigger that includes a user utterance also receives the annotated transcript of that utterance in place of the final ASR transcript, which often arrives after the annotated end of the turn; Gemini hears the audio itself.
Reading the Ladder. L1 changes the intervention regime as a whole, since the system is told the annotated onset, made to respond there, silenced elsewhere, and, for JoyAI, given the annotated transcript of the triggering turn and the ladder prompt. therefore measures the combined effect of these changes, not the contribution of timing information alone. and add information to the note under the same regime; because the levels are separate runs, their differences also contain run-to-run variation. L3 adds observed facts only for interventions with action or state evidence, so its effect concentrates on proactive guidance (§ 6.3). The values at L1–L3 describe the systems under enforced intervention and are not estimates of their native capability.
Implementation
JoyAI. At a trigger tick the note is appended to the request and the decoder is made to speak by prefilling the assistant turn with the response action token, so that the model writes the intervention; at other ticks the request is made as usual, and a decision to speak is recorded but not played. The decision time of an enforced intervention is the start of the inference at the trigger tick. L1–L3 share one system prompt, which describes the note and differs in wording from the main-experiment prompt (Table 1); L0 is the main-experiment run (Table 1).
Gemini. The Live API generates at activity boundaries, so each trigger is realized as one fresh Live session seeded with the recorded interaction up to the last released instructor utterance (the re-build of Appendix § 10.§ 2). The audio and video up to the trigger are then streamed in real time inside one manual activity with automatic activity detection disabled, and the note is sent as text followed by the end of the activity, which marks the session’s only turn boundary and triggers generation. Output produced before the note is discarded. The system instruction is the same across L1–L3, and the main-experiment run (Table 1) serves as L0.
Human Timing Study
This study tests the annotated timing windows and calibrates the graded credit used by Timely Correctness. Three raters judged the timing of WTaG interventions whose content was held fixed while only the speech start time was varied, so that the ratings reflect timing rather than wording. We first describe the study events, the candidate moments, and the rating procedure, then compare the annotated windows with the human judgments of acceptable timing, and finally compare temporal scoring functions and select their parameters with leave-one-session-out (LOSO) cross-validation. Throughout, the anchors are written as in Appendix § 9.2.§ 5: ES, EO, LU, and R correspond to , , , and of the main text and Appendix § 9.3.§ 6.
Study Events
The formal study covers 79 WTaG events from 31 sessions, and a separate set of 12 events was used for calibration. Formal events required complete source media, a reference instructor utterance, and the anchors needed for the reactive or proactive window. We included the 12 non-Other instructor categories (Table 6) and excluded reference utterances that overlap user speech, since these would confound the speech start time with simultaneous speech.
Events were sampled with category quotas to cover the intervention types, with at most three events per session and none from the calibration sessions; the quotas are not intended to reflect corpus prevalence. The formal set contains 26 coffee, 28 mug-cake, and 25 pinwheel events: 43 reactive and 36 proactive, 46 necessary and 33 optional.
The calibration set contains one event from each of 12 distinct sessions, four per task. It is balanced between reactive and proactive, necessary and optional, and three anticipated difficulty levels, covers 11 of the 12 non-Other categories, and uses the same candidate-moment construction as the formal study.
Candidate-Moment Construction
Each event has six candidate moments spanning early, valid, and late responses. Two are fixed: the preferred time, which is the end of the triggering user turn (R) for reactive events and the evidence onset (EO) for proactive events, and the speech start time of the original instructor utterance. The other four are sampled relative to the event’s anchors. The early moment precedes EO for proactive events and falls within the triggering user turn for reactive events. Two moments cover the first and second halves of the interval from the preferred time to the latest useful time (LU). The last slot is sampled after LU with probability 0.5, from a heavy-tailed distribution, and otherwise uniformly between the earliest candidate position (ES for proactive events, the start of the triggering user turn for reactive events) and LU. Moments must be at least 0.1 s apart; near-duplicate fixed moments are merged and the open slot is resampled. Sampling therefore covers positions relative to the anchors rather than uniformly across the video.
Rating Protocol and Calibration
Raters viewed the source context up to each candidate moment and then judged the timing of the same synthesized reference utterance on an integer scale from 0 (definitely should not intervene now) to 10 (exactly the desired moment); they were asked to judge timing rather than wording. The rubric marked 5 as borderline and 6 as acceptable but improvable, and Cannot judge responses were treated as missing. All three raters were authors. The roles of candidate moments (early, preferred, or late), annotated timing boundaries, and original reference start-time labels were hidden from raters.
All raters completed calibration before the formal study. The 12 calibration events produced 216 records, including one Cannot judge. Calibration familiarized the raters with the interface and the rubric; its ratings were excluded from the primary reference, MAE, and parameter selection, and were not used to exclude or reweight raters.
The formal phase produced 1,422 rating rows over 474 candidate moments, of which 1,415 are valid ratings and seven are Cannot judge. The human reference for each moment is the mean available rating divided by ten.
Validation of the Annotated Timing Windows
Before calibrating graded credit, we tested whether the annotated valid windows agree with binary human judgments of acceptable timing. For candidate of event , let be the mean of the valid rater scores on the 0–10 scale. Following the rubric’s threshold for acceptable timing, human acceptability is
We compare two binary timing rules:
where is the annotated valid window of event ([R, LU) for reactive and [ES, LU) for proactive events) and is the speech start time of the original instructor utterance, in seconds. Each rule is treated as a classifier of , and its agreement with the human judgments is measured by Macro-F1, the unweighted mean of the F1 scores of the acceptable and the unacceptable class:
where takes the acceptable moments () as the positive class and the unacceptable moments (), with , , and counted for class . Unacceptable moments are the minority (146 of the 474 candidate moments), so the macro average weights the two classes equally. Confidence intervals and paired differences use 5,000 session-cluster bootstrap draws, each resampling the 31 formal sessions with replacement and keeping every candidate moment of a sampled session. Figure 8 reports the results. Over all 474 moments, the evidence-derived window reaches a Macro-F1 of 0.788 (95% CI [0.741, 0.832]) against 0.713 [0.663, 0.762] for the fixed window, a paired difference of +0.075 ([0.002, 0.149]). The gap comes from reactive events, 0.787 against 0.659, a paired difference of +0.127 ([0.040, 0.220]): the fixed window accepts 61 unacceptable reactive moments, 39 of them while the user is still speaking, whereas the evidence window opens only after the user’s turn ends. On proactive events the two windows perform comparably, 0.781 ([0.686, 0.861]) against 0.773 ([0.701, 0.836]), a difference of +0.008 ([−0.099, 0.116]); the fixed window rejects 16 acceptable moments that lie inside the evidence window, 5 of them earlier than 6 s before the reference onset and 11 later than 3 s after it.
Figure 8. Agreement with human acceptability. Macro-F1 of the evidence-derived windows (blue) and the fixed [−6, +3]-second windows (orange) against human acceptability (mean rating of at least 6), per guidance side and pooled. Whiskers show 95% session-cluster bootstrap confidence intervals; is the paired difference (evidence minus fixed), starred when its 95% interval excludes zero; counts candidate moments.
This comparison tests the window boundaries. The following analyses test how much credit Timely Correctness should assign within a valid window.
Temporal Scoring Hypotheses
We compared three families of graded timing scores, each assigning zero outside the valid window. They are written in terms of the anchors ES, EO, R, and LU of Appendix § 9.2.§ 5 and the normalized positions
where falls from 1 to 0 over [ES, EO), while and rise from 0 to 1 over [EO, LU) and [R, LU), respectively.
Hypothesis 1: Normalized Linear Position. Under this hypothesis, credit changes at a constant rate with the normalized position in the event-specific window:
Hypothesis 2: Base-2 Squared Exponential. This hypothesis uses absolute elapsed seconds rather than the normalized window position:
with the WTaG corpus-level scales s, s, and s.
Hypothesis 3: Quadratic Onset and Cosine-Power Decay. This family controls the proactive early discount and the shape of the late decline separately:
Here sets the proactive discount at ES, where the credit is , and sets the decline after the preferred time. The settings and are reported as reference points within this family, not as separate models.
As diagnostic baselines, we also score the two binary rules of Eqs. (11) and (12) as step functions, which compares graded credit with binary timing acceptability.
Selection Criterion and Hyperparameter Search
Scoring functions are selected by event-level mean absolute error (MAE) against the normalized human ratings. For a temporal function ,
where is the mean valid human rating divided by ten and contains the valid candidate moments of event . Errors are thus averaged within each event and then across events, giving events equal weight. Mean within-event Kendall serves as a secondary measure of ordering; MAE is primary. because Timely Correctness uses the amount of credit. Uncertainty intervals come from a session-cluster bootstrap in which each draw resamples sessions and keeps every event and candidate moment of each sampled session, and differences between functions are paired within the same draw.
For the cosine family, we search
The parameters are evaluated by leave-one-session-out (LOSO) cross-validation over the 31 formal sessions: in each fold, the pair minimizing event-level MAE on the other 30 sessions is applied, without refitting, to the held-out session, so that session’s ratings are not used to select the parameters applied to it.
Results
The LOSO-selected cosine has the lowest MAE among the compared functions (Figure 9). Thirty of the 31 folds selected and one selected . The parameters used by Timely Correctness, , are those selected on all 31 formal sessions; the MAE reported for the selected cosine is its out-of-fold value.
Figure 9. Agreement of temporal scores with human ratings. Event-level MAE with 95% session-cluster bootstrap confidence intervals; lower is better. Cosine settings are . The selected cosine uses LOSO out-of-fold predictions.
The selected cosine achieves MAE 0.182, compared with 0.225 for linear scoring and 0.262 for constant valid-window credit. Its paired difference from linear is with a 95% confidence interval (CI) of , its improvement over constant credit is 0.080 , and it also improves over the cosine setting by 0.015 . These comparisons support graded credit within the window. The selected lies inside the search grid rather than at its lower boundary, and it gives a shallower decline through moderate delay than while still assigning zero at LU. Linear and selected cosine scores show similar within-event ranking agreement ( and 0.738), but the selected cosine achieves better calibration to the human rating scale, as reflected by its lower MAE.
Calibration and Rater Sensitivity Analyses
The comparison with binary scoring in § 4.2 uses Spearman’s rank correlation computed within each event between a function’s scores and the mean human ratings of its six candidate moments, averaged over events. Linear and selected cosine scores rank the six moments identically in 78 of the 79 events and reach mean correlations of 0.833 and 0.832. Constant valid-window credit is identical across all six moments of 12 events, where the correlation is undefined; over the remaining 67 events it averages 0.645 (0.547 when undefined events count as zero), and the fixed s step averages 0.616 over its 50 defined events. Pooling all 474 moments instead gives 0.787 for both graded scores and 0.586 for constant valid-window credit.
Selection using only the 46 necessary events from 27 sessions gives the same parameters in all 27 LOSO folds, with out-of-fold MAE 0.178 versus 0.219 for linear scoring, so including the optional events does not change the selected setting. We use one shared function rather than separate scoring functions for individual instructor categories, whose small groups are partly confounded with the reactive/proactive composition.
The selected cosine keeps its advantage after adjusting for rater differences and when any one rater is omitted. For the calibration-informed analysis, we jointly modeled the valid calibration and formal ratings with a random intercept for candidate moment and fixed effects for study phase, rater, timing region (before the preferred time, within the window, or after LU), intervention side, and the corresponding rater interactions; the calibration ratings serve here only to estimate rater severity and timing tolerance. The adjusted formal reference correlates 0.993 with the raw rater mean. Against it, cosine MAE is 0.190 and linear MAE is 0.236, a paired difference of (95% CI ). This agrees with the primary analysis. Given the small number of raters and calibration events, we treat this adjustment as an exploratory sensitivity analysis rather than independent validation.
The two-way mixed-effects absolute-agreement intraclass correlation coefficient (ICC) is 0.663 for one rater (ICC(A,1)) and 0.855 for the three-rater mean (ICC(A,3)). Omitting each rater in turn preserves the cosine advantage, with MAE improvements of 0.031–0.058. Raters differed in strictness and preferred timing, which supports using their mean rather than a single rater’s curve.
References
- [1]Yuwei Bao, Keunwoo Peter Yu, Yichi Zhang, Shane Storks, Itamar Bar-Yosef, Alex de la Iglesia, Megan Su, Xiao Lin Zheng, and Joyce Chai. 2023. Can foundation models watch, talk and guide you step by step to make a cake? In Findings of the Association for Computational Linguistics: EMNLP 2023.arxiv.org/abs/2311.00738
- [2]Apratim Bhattacharyya, Shweta Mahajan, Sanjay Haresh, Rajeev Yasarla, Reza Pourreza, Litian Liu, Risheek Garrepalli, and Roland Memisevic. 2026. Streaming interventions: Can video large language models correct mistakes as they occur? arXiv preprint arXiv:2606.09547.arxiv.org/abs/2606.09547
- [3]Apratim Bhattacharyya, Bicheng Xu, Sanjay Haresh, Reza Pourreza, Litian Liu, Sunny Panchal, Leonid Sigal, and Roland Memisevic. 2025. Can multi-modal LLMs provide live step-by-step task guidance? In The Thirty-ninth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2511.21998
- [4]ByteDance Seed. 2026. SeedRealtime: An Audio-Visual Full-Duplex LLM. Technical blog, published August 5, 2026.
- [5]Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. 2024. Videollm-online: Online video large language model for streaming video. In CVPR.arxiv.org/abs/2406.11816
- [6]Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. 2025. Livecc: Learning video llm with streaming speech transcription at scale. In CVPR.arxiv.org/abs/2504.16030
- [7]Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, Jiancheng Gui, Luoyuan Zhang, Xian Sun, Fuwei Huang, Moye Chen, Zhuo Lin, Hanyu Liu, Qingxin Gui, Qingzhe Han, and 17 others. 2026. Minimcp-0 4.5: Towards real-time full-duplex omni-modal interaction. Preprint, arXiv:2604.27393.
- [8]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2018. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV).arxiv.org/abs/1804.02748
- [9]Shenghao Fu, Qize Yang, Yuan-Ming Li, Yi-Xing Peng, Kun-Yu Lin, Xihan Wei, Jian-Fang Hu, Xiaohua Xie, and Wei-Shi Zheng. 2025. Vispeak: Visual instruction feedback in streaming videos. arXiv preprint arXiv:2503.12769.arxiv.org/abs/2503.12769
- [10]Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xi-ang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, and Yifei Huang. 2026. Vinci2: Providing proactive assistance in continuous egocentric videos. In European Conference on Computer Vision (ECCV).arxiv.org/abs/2607.11523
- [11]Google DeepMind. 2026. Gemini 3.8 Audio (Live, Live Extended Thinking, Flash TTS, Flash-Lite TTS) Model Card. Google DeepMind Model Card. Published September 15, 2026.
- [12]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Javant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, and 66 others. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. Preprint, arXiv:2110.07058.arxiv.org/abs/2110.07058
- [13]Ant Group. 2026. Realtime-venus: A full-duplex interaction system with asynchronous delegation. Preprint, arXiv:2609.13814.arxiv.org/abs/2609.13814
- [14]Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Dong Lu, Yali Wang, Limin Wang, and Yu Qiao. 2024. Egoex-olearn: A dataset for bridging asynchronous ego- and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
- [15]Kaustav Kundu, Ritvik Shrivastava, Maxim Arap, Nanshu Wang, Xianhui Zhu, Quintin Fettes, Gautam Tiwari, Parth Suresh, Théo Moutakanni, Alejandro Castillejo Munoz, Allen Bolourchi, Pascale Fung, Priyan Donmez, Babak Damavandi, Anuj Kumar, and Seungwhan Moon. 2026. Plan, watch, recover: A benchmark and architectures for proactive procedural assistance. Preprint, arXiv:2606.04970.arxiv.org/abs/2606.04970
- [16]Xudong Lu, Xueying Li, Annan Wang, Yang Bo, Jinping Chen, Zengliang Li, Nianzu Yang, Rui Liu, Xue Yang, Jingwen Hou, and Hongsheng Li. 2026. OmniInteract: Benchmarking real-world streaming interaction for real-time omnmodal assistants. arXiv preprint arXiv:2605.26485.arxiv.org/abs/2605.26485
- [17]OpenAI. 2026. GPT Realtime 2. Model: gpt-realtime-2. Accessed Oct 2026.
- [18]Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Böhm, Florian Diettriecht, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovic, Ingo Bax, and Roland Memisevic. 2024. What to say and when to say it: Live fitness coaching as a testbed for situated interaction. In NeurIPS (D&B Track).arxiv.org/abs/2407.08101
- [19]Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Qifan Zhang, Jikai Wang, Vasundhara Komaragiri, Eric Ragan, Nicholas Ruozzi, Yu Xiang, and Vibhav Gogate. 2024. Captaincook4d: A dataset for understanding errors in procedural activities. In Advances in Neural Information Processing Systems, volume 37, pages 135626–135679. Curran Associates, Inc.arxiv.org/abs/2312.14556
- [20]Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. Dispider: Enabling video LLMs with active real-time interaction via disentangled perception, decision, and reaction. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 24045–24055.arxiv.org/abs/2501.03218
- [21]Qwen Team. 2026a. Qwen3.5-omni technical report. Preprint, arXiv:2604.15804.arxiv.org/abs/2604.15804
- [22]Qwen Team. 2026b. Qwen3.6-35B-A3B: Agentic coding power, now open to all.
- [23]Francesco Ragusa, Michele Mazzamuto, Rosario Forte, Irene D’Ambra, James Fort, Jakob Engel, Antonino Furnari, and Giovanni Maria Farinella. 2026. Ego-extra: Video-language egocentric dataset for expert-trainee assistance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).arxiv.org/abs/2512.13238
- [24]Yuki Sakai, Ryosuke Furuta, Juichun Yen, and Yoichi Sato. 2025. Egoinstruct: An egocentric video dataset of face-to-face instructional interactions with multimodal llm benchmarking. In Proceedings Workshop on Interactive Human-centric Foundation Models (I-HFM 2025) (in conjunction with ICCV 2025).arxiv.org/abs/2509.22019
- [25]Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, and Limin Wang. 2026. River: A real-time interaction benchmark for video llms. Preprint, arXiv:2603.03985.arxiv.org/abs/2603.03985
- [26]Thinking Machines Lab. 2026. Interaction models: A scalable approach to human-ai collaboration. Thinking Machines Lab: Connectionism. https://thinkingmachines.ai/blog/interaction-models/.DOI
- [28]Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys. 2023. Holoassist: an ego-centric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20270–20281.DOI
- [29]Yueqian Wang, Songxiang Liu, Disong Wang, Nuo Xu, Wan Guanglu, Huishuai Zhang, and Dongyan Zhao. 2026. MMDuet2: Enhancing proactive interaction of video MLLMs with multi-turn reinforcement learning. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2512.06810
- [30]Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, and Dongyan Zhao. 2025. Proactivevideoqa: A comprehensive benchmark evaluating proactive interactions in video large language models. Preprint, arXiv:2507.09313.arxiv.org/abs/2507.09313
- [31]Lilin Xu, Bufang Yang, Siyang Jiang, Kaiwei Liu, Kaiyuan Hou, Yuang Fan, Hongkai Chen, Zhenyu Yan, and Xiaofan Jiang. 2026. Pro²assist: Continuous step-aware proactive assistance with multi-modal egocentric perception for long-horizon procedural tasks. Preprint, arXiv:2605.04227.arxiv.org/abs/2605.04227
- [32]Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, and Seungwhan Moon. 2025a. Proactive assistant dialogue generation from streaming egocentric videos. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12055–12079.arxiv.org/abs/2506.05904
- [33]Yichi Zhang, Run Peng, Yinpei Dai, Lingyun Wu, Xuweiyi Chen, Qiaozi Gao, and Joyce Chai. 2025b. Bootstrapping visual assistant modeling with situated interaction simulation. In Second Conference on Language Modeling.
- [34]Yichi Zhang, Jianing Yang, Keunwoo Yu, Yinpei Dai, Shane Storks, Yuwei Bao, Jiayi Pan, Nikhil Devraj, Ziqiao Ma, and Joyce Chai. 2023. Seagull: an embodied agent for instruction following through situated dialog. Alexa Prize SimBot Challenge Proceedings.







