WorksComputer Science
EgoWitness: Benchmarking Instance-Level Spatial Memory and Reasoning in Egocentric Video Streams
Abstract
Recalling where and when an object was last seen and who saw it requires recognizing it across days and people. Existing long-video benchmarks focus on events, object categories, or relative locations, leaving instance-level spatial memory largely untested in multimodal models. To address this gap, we present EgoWitness, a benchmark for instance-level spatial memory and reasoning in egocentric video. Its 8,453 human-verified questions span approximately 324 hours of week-long and short-horizon recordings and test seven abilities, from recognizing an object or room to recalling who saw it and reasoning about room visits. Over one million reviewed masks of 34 registered landmarks and 21 movable-object labels are linked through room timelines. Among twelve direct vision-language models (VLMs), open-weight models reach at most 46.97% accuracy, and none exceeds 35% on identifying who saw an object last. We therefore propose EgoClue, a spatial-memory agent that retrieves candidate intervals, verifies instance identity, and computes spatial and temporal relations with five tools. Two-stage training with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) raises its 4B controller from 23.64% to 72.65% accuracy on 1,320 held-out questions. Evidence-trace synthesis matches the best direct VLM at 74.85% on week-long questions and achieves 80.56% in environments excluded from training.
Introduction
Smart glasses equipped with cameras and microphones hold the promise of becoming personal AI assistants that understand and remember our everyday experiences. Such assistants must be capable of more than simply recognizing visible objects. They must also remember where and when those objects appeared. Imagine a user rushing to an important meeting and asking, “Did I last see my work laptop in the upstairs living room or the downstairs one?” Answering it requires more than recognizing a laptop or a living room. The assistant must distinguish the user’s laptop from lookalikes, recognize the specific room, and connect the sighting to its time and location [17, 57, 69]. In a shared environment, answering “Who saw it last?” also requires linking observations across people. We refer to this capability as instance-level spatial memory and reasoning. It becomes particularly challenging in multi-day, multi-wearer egocentric streams, where the same objects and rooms reappear across changing viewpoints, repeated visits, and different people.
Despite progress in long-video understanding [8, 19, 38], this capability is rarely evaluated directly. Object-centric benchmarks [31, 33, 57] track or localize targets within one recording, while memory-centric QA benchmarks [29, 62, 68] ask about past events and relative locations across days or wearers. Both rarely test whether today’s object is the physical instance seen days earlier or who saw it last. Asking where scissors were last used differs from identifying the pair a housemate saw two days ago. Few annotations link each sighting to an instance, a room, and its wearer.
To address this gap, we introduce EgoWitness for instance-level spatial memory across week-long, multi-wearer recordings and short-horizon environments. Its human-reviewed annotations link questions to evidence and record each wearer’s room over time to distinguish lookalike instances across visits. In Fig- ure 1A, detergent-bottle views are matched to references, while Figure 1B links three wearers’ sightings across three days to times and rooms. The seven abilities in Figure 1C form two groups. Object identity and memory questions test which instance appeared, when, and who saw it, while room identity and visit reasoning questions test room recognition, paths, and visits. Ego4D-derived recordings with aligned annotations [25, 31, 33, 41, 42, 57] extend evaluation beyond one home. As shown in Table 1, EgoWitness alone combines object and room re-identification, cross-wearer questions, scored evidence, masks, and room instances with intervals.
Figure 1. Instance-level spatial memory across egocentric streams. (A) Instance-reference comparison. (B) Historical observations with wearer, time, and room context. (C) Seven QA abilities in two groups: object identity and memory, and room identity and visit reasoning.
| (a) Memory and spatial question answering | ||||||
| Benchmark | Scale | Span | Object identity | Place identity | Cross-wearer | QA evidence |
| EgoLifeQA [68] | 300 h | 7 days | Object recall | Location recall | ◦ | Evidence time |
| MA-EgoQA [29] | 266 h | 7 days | Object usage | Event context | • | Context windows |
| HD-EPIC [45] | 41 h | 3 days | Object itineraries | Fixture identity | ◦ | ◦ |
| AMB / AMEGO [21] | 22.7 h | 14 min avg. | Interaction links | Activity zones | ◦ | ◦ |
| UCS-Bench [62] | 170+ h | Seconds to hours | Trajectory recall | Relative places | ◦ | Evidence segments |
| 🕶️ EgoWitness | ∼324 h | Minutes to week | Object Re-ID | Room Re-ID | • | Scored evidence |
| (b) Object tracking, grounding, and environment resources | ||||||
| Resource | Scale | Span | Boxes | Masks | Room annotations | |
| EgoTracks [57] | 602.9 h | 6.13 min avg. | • | ◦ | ◦ | |
| RefEgo [31] | 41 h | 5 to 20 s | • | ◦ | ◦ | |
| EgoMask [33] | 315 videos | 5 s to 16 min | • | • | ◦ | |
| LongEgoRefer [28] | 433 videos | 45 min avg. | • | ◦ | ◦ | |
| EgoEnv [42] | 1,259 scenes | Source-specific | ◦ | ◦ | Types + local IDs | |
| 🕶️ EgoWitness | ∼324 h | Minutes to week | • | • | Instances + intervals |
Table 1. Table 1 Related benchmarks and annotation resources. (a) Identity-related QA. (b) Object and environment annotations. •: included in the reported protocol; ◦: not specified. QA evidence: reference evidence per question; EgoWitness also scores it. Days denote collection span; avg. denotes mean video length. Cross-wearer requires multi-wearer evidence. EgoMask reports its test set; EgoEnv reports the Ego4D subset. EgoEnv provides room-type intervals and local instance metadata.
We evaluate twelve vision-language models (VLMs) that answer week-long questions directly from uniformly sampled frames. The strongest reaches 73.56% accuracy, the best open-weight model reaches 46.97%, and no model exceeds 35% on identifying who saw an object last. They cannot request another view or search for later sightings.
These results motivate an agent that revisits clips, verifies identity, and compares sightings across time. Tool-using video agents [18, 58, 61] mainly retrieve relevant clips. Building on them, we propose EgoClue, which also verifies sightings against references and computes room paths and durations with five tools. Its 4B controller learns tool use through supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) [53].
Our contributions are summarized as follows:
An instance-level spatial-memory benchmark. EgoWitness provides 8,453 questions across seven abilities over approximately 324 hours. Twelve direct VLMs struggle most when identifying who saw an object last.
Room-guided, human-reviewed annotations. Over one million masks cover 34 registered landmark instances at 1 Hz and 21 movable-object labels at 5 Hz, linked through room timelines and reviewed identities.
A spatial-memory agent framework. EgoClue retrieves candidate intervals, verifies instance identity, and computes room paths and visit durations with five tools. Two-stage training raises its 4B controller from 23.6% to 72.7% on 1,320 held-out questions. Trace-based synthesis reaches 74.8%, matching the best direct VLM.
Related work
Egocentric data and representations. Large egocentric datasets supply the recordings on which spatial memory builds. EPIC-KITCHENS [14, 15] annotates kitchen actions and objects, Ego4D [25] spans diverse daily environments, and Goal-Step [56] adds procedural labels. Ego-Exo4D [26] synchronizes first- and thirdperson views, while EgoExoLearn [27] connects asynchronous demonstrations. EgoVLP, EgoVLPv2, and LaViLa [34, 46, 74] learn video-language representations from narrations. EgoLife [68] records shared week-long experience, and HD-EPIC [45] links object movements to fixtures and connects them into itineraries. However, these resources mainly organize supervision around activities, narrations, or object movements. In comparison, EgoWitness records which room each wearer occupies and which physical instance each mask shows across days and wearers.
Long-video and embodied question answering. EgoSchema and LongVideoBench [38, 64] pose questions over long clips, while Video-MME, MLVU, LVBench, and HourVideo [8, 19, 60, 76] expand duration and task coverage. SuperMemory-VQA [1] adds unanswerable memory queries, and EgoMonth [9] extends personal memory across 20 to 120 days. Embodied QA and OpenEQA [16, 37] test navigation and episodic scene memory, and R2R, REVERIE, and SOON [2, 47, 78] ground navigation in instructions and target descriptions. The memory QA benchmarks in Table 1a [29, 62, 68] ask about object recall, usage, and trajectories. Beyond3D [36] tests spatiotemporal reasoning about relocated objects after they leave view. In contrast, EgoWitness links room and object identities across days and wearers to recover where, when, and by whom instances were observed, scoring both answers and evidence.
Instance-level objects, places, and memory. Table 1b summarizes instance-level resources. EgoObjects [77] separates categories from physical instances, and MyEgo [65] personalizes object queries. EgoTracks, RefEgo, and EgoMask [31, 33, 57] localize targets, and IT3DEgo [75] tracks registered instances in 3D. Whareformer [7] combines appearance and 3D location to associate observations with object tracks and maintain locations out of view. AMEGO [21] links interactions and locations in memory; EgoEnv [42] learns environment-aware representations. ESOM [39] builds online object memory from detections and tracks. Its oracle experiments identify detection and tracking bottlenecks, motivating identity and evidence verification. For cross-day, cross-wearer questions, EgoWitness links physical instances to local tracks, room timelines, and the wearers who saw them.
Memory and tool-using video agents. Video agents extend memory through retrieval and tools. MovieChat and LLoVi [55, 72] reason through compressed memory and clip captions. Wang et al. [61] select evidence iteratively, while Fan et al. [18] maintain event and object memory. VideoTree [63] adapts a video hierarchy to the query. Video-RAG [35] retrieves auxiliary visual information, and EgoButler [68] retrieves relevant life experience. Self-RAG, Reflexion, and RAFT [5, 54, 73] refine generation through critique, feedback, or retrieval-conditioned training. Ego-R1 [58] learns dynamic tool selection through SFT and reinforcement learning. However, retrieval mainly identifies candidate content, whereas finding who saw an object last also requires verifying instance identity and checking later sightings. Building on these agents, EgoClue pairs visual verification with deterministic path and duration tools. Its controllers share perception, retrieval, and execution budgets across training stages, which isolates how training changes evidence acquisition.
The EgoWitness benchmark
We build EgoWitness by selecting recordings, annotating rooms and objects with human review, and converting the annotations into verified questions.
Source recordings and scope
EgoWitness spans approximately 324 hours of long- and short-horizon recordings, summarized in Table 2.
| Attribute | Week-long | Short-horizon |
| Recordings and spatial coverage | ||
| Video source | EgoLife | Ego4D-derived |
| Source hours | ∼300 | 23.46 |
| Recording units | 42 wearer-days | 43 videos |
| Room references | 17 zones | 400 local entries |
| Reviewed annotations and questions | ||
| Label space | 34 L + 21 M | 123 raw names |
| Sampling, Hz | 1 L / 5 M | 1 / 2 |
| Visible observations | 1,034,075 masks | 14,270 boxes |
| Local tracks | 92,669 | 170 |
| Reviewed questions | 7,666 | 787; eval. only |
Table 2. Source collection and annotation coverage. L/M denote landmark instances/movable labels. Room references retain their source-local scope.
Long-horizon recordings. EgoLife [68] provides approximately 300 hours over 42 wearer-days from six wearers in 32,001 source MP4 files. We add 17,269 room intervals over 17 named rooms and zones, covering all recorded wearer-hours, and 1,034,075 masks of 34 registered landmark instances and 21 movable-object labels in Figure 2b.

Figure 2. Room references, object annotations, and question coverage. a. The external room catalog contains 400 source-local reference entries across 81 scopes and 18 room types, separate from the 17 named zones in the shared home. b. The week-long component contains 34 registered landmark instances and 21 movable-object labels. c. Questions cover both components; Q07 is split into five subtypes.
Short-horizon recordings. The 43 selected Ego4D-derived full videos span 23.46 hours. We align EgoEnv room annotations with EgoMask and RefEgo object annotations [31, 33, 42] by source-video ID and time interval. The 400 source-local room references span 81 scopes and 18 room types in Figure 2a. EgoTracks supplies tracking lineage and ego_plan provides full-duration copies of 15 videos [41]. Supp. Sec. A details selection and alignment.
Layered annotation and review
Room annotation comes first in Figure 3a because shared room identities constrain where instances appear. SAM 3.1 [40] then turns reliable anchors into local tracks, which annotators link into physical instances before accepted observations fix question facts. Supp. Sec. B details each step.

Figure 3. From reviewed observations to evidence-linked questions. a. Room timelines and landmark locations supply spatial context. b. SAM 3.1 tracks Gemini- and human-checked keyframes. c. Annotators resolve instance identity using reference views and room context. d. Accepted observations fix question facts before wording. Full review interfaces appear in Figure S1.
Room timelines. The room layer records each wearer’s room over time. Annotators edit intervals in the Online Room Curator System. Reviewed intervals cover 4,257 segments with gaps of at most 0.5 s. Reviewers corrected 3,383 intervals. Room IDs are shared across wearers and days, and unknown and outdoor states stay explicit.
Registered landmarks. Landmarks are registered visual references, such as fixtures and designated reference objects, each with a reference room, floor, and zone. Because landmarks rarely move in the videos, 1 Hz sampling suffices to capture their spatial relation to the wearer. SAM 3.1 [40], which builds on promptable and memory-based segmentation [6, 11, 12, 13, 30, 51], propagates accepted landmark detections at this rate. All 34 landmark labels link to reviewed instance records observed by multiple wearers across days.
Movable objects. Movable objects are portable everyday items, including bottles, scissors, and bowls. Because people may move these objects at any time, we sample them at 5 Hz for finer-grained tracks. A YOLO detector proposes candidate boxes, and confidence, persistence, duplicate, and room filters retain 19,921 anchors. These anchors are clustered by appearance similarity within each room segment. Gemini 3.8 Flash [23] scores whether a cluster shows the same object from its appearance, function, and scene context, and annotators check every selected keyframe. SAM 3.1 tracks these keyframes forward and backward at 5 Hz into local mask tracks in Figure 3b. A track stays within one processing unit, so annotators link tracks to physical instances in Figure 3c. Room context separates look-alikes, such as two bins seen at once on two floors.
Context and logical checks. Room states, conflicts with registered floors, duplicate detections, and simultaneous placements of one instance in disconnected rooms trigger further review. Annotators inspect the video to remove false boxes and correct identity links or room boundaries. Accepted masks then fix the question facts in Figure 3d. Supp. Sec. B.3 details the checks and corrections.
A unified representation
Questions in § 3.4 query identity, place, time, and observer across both sources, so every accepted annotation becomes one observation record. A record links a target instance to its wearer, time, box, mask, local track, category, attributes, and provenance. Questions ask about both observers and targets, so the schema separates three locations. Observer location is the wearer’s room from the room intervals, *registered landmark location comes from the landmark registry, and object location records the established position of a target. The grill in Figure 3a, for example, lies in the courtyard but is seen from the kitchen. Supp. Sec. B.4.3 maps both sources to common source time and geometry.
Question design and verification
Task definition. Each question specifies an object or room, public references, and a wearer and time scope. Answers use observations within that scope and, for historical questions, before the query time. Models select an answer option, and agents also cite supporting observations. For the detergent bottle in Figure 1A and B, a model must decide whether a new crop shows the same bottle, which wearer saw it, or who saw it last. The seven abilities in Figure 2c form two groups: object identity and memory, and room identity and visit reasoning. In the object group, Q01 compares reference images to distinguish the same instance, different instances of one category, and different categories. Q02 asks which of two wearers saw the target in a shared 6-s window, Q03 who saw it last, and Q04 which time windows contain it. Q02 to Q04 require matching sightings across separate recordings. In 103 of 120 evaluated Q04 questions, the object appears in a window ending over 24 hours before the query time, and in 78 it is seen by multiple wearers. In the room group, Q05 finds a shortest path between rooms identified from images, and Q06 applies the Q01 distinctions to room references, using room type as the category. Q07 orders, times, or counts visits and recalls the preceding room.
Five room-visit subtypes. Q07A orders the visited rooms, including returns, and Q07D identifies the room before the current stay. Q07B times the current stay up to the query time, Q07C times a specified completed visit among repeated visits, and Q07E counts visits overlapping the observation interval. Supp. Sec. C.1 lists the templates.
Type-specific generation and verification. Reviewed identities and categories determine Q01/Q06 reference pairs. Scoped target sightings determine Q02/Q03/Q04 answers. Q05 uses shortest paths on the supplied room graph, and Q07 uses reviewed room intervals. Answers, options, and evidence are fixed before language-model rewriting. Historical checks cover all queried wearers and candidate windows before the query time. After model-assisted review, experts verify every question, answer, and supporting evidence against source videos and room timelines in the QA Audit Workbench. Supp. Sec. C.5 details this review.
Splits, access conditions, and metrics
Splits. Week-long questions are grouped by core task and split into 5,750 training, 383 validation, and 1,533 test questions in a 75/5/20% ratio. Core task groups are disjoint. Cross-environment evaluation reserves all 787 short-horizon questions and their environments for testing, with SFT and GRPO confined to the week-long data.
Access conditions. Beyond the query, direct VLMs inspect preselected frames, while agents select video intervals and retrieve coarse text memory summaries. Q05 supplies a room graph for path computation after visual endpoint identification. Private room timelines, instance sightings, and answers serve only question construction and scoring.
Metrics. Micro accuracy weights questions equally. Week-long Macro averages the seven abilities so that the five Q07 subtypes do not dominate the overall score,
where contains queries of ability and . Here and are the predicted and reference answers. Supp. Sec. F.2 reports subtype scores. A shared parser extracts choices using only model responses and public options. Outputs without a resolved choice count as incorrect. Evidence scoring checks instance identity, witness, and time. Joint scores multiply answer correctness by evidence credit. Assigning unresolved evidence scores zero or one gives lower and upper bounds. Bootstrap intervals group related questions by core task or source video.
EgoClue: Spatial-memory agent
Retrieval, visual inspection, and reasoning
Answering EgoWitness questions involves finding candidate sightings, verifying their identity, and relating them in space and time. Following tool-augmented reasoning [48, 52, 70], EgoClue uses Qwen3.5-4B [49] as its controller to decide where to look and how to combine observations from five tools in Figure 4. Memory retrieves intervals from text summaries, Video inspects intervals for targets, rooms, and events, and Image compares references with individual frames. Topology computes shortest paths between identified rooms, and Timeline computes durations from observed boundaries. Image and Video share a fixed visual backend. Memory queries a text model, while Topology and Timeline compute relations deterministically. Each returns text for the controller’s next decision.
Figure 4. EgoClue: finding, verifying, and linking evidence. The trained 4B controller selects tools and video intervals, then refines its search from returned observations. The optional EgoClue + Sol stage synthesizes an answer from the trace without new visual calls.
Memory uses a text model to select candidate intervals from prebuilt captions, transcripts, and coarse annotation categories. Video inspection starts at 1 FPS, with local refinement guided by returned observations. For visit duration, Video identifies entry and exit events and Timeline computes their separation. For witness queries, the controller verifies each candidate sighting and compares their times.
EgoClue uses SFT+GRPO by default. In the last stage of Figure 4, optional GPT-6.1 Sol synthesis reads the public question and trace preceding the controller’s final answer, producing a choice and evidence references without additional visual calls. Supp. Sec. G specifies its inputs and cost.
Two-stage training
To learn when to call which tool and when to answer, the controller is trained in two stages. During SFT, the controller learns to select tools and answer from teacher trajectories. GRPO then improves it with answer and evidence rewards on its own rollouts. Supp. Sec. D.1 details both stages.
GRPO samples four rollouts per question and masks tool observations from the policy loss. For a trajectory , the reward combines answer correctness , task-specific evidence credit , and format validity :
Q02 and Q03 compute evidence credit from verified sighting anchors. Submission validation checks format
Experiments
and references, while the training evaluator supplies private answer and evidence scores. Supp. Sec. D.2 defines the normalized group advantage and clipped policy objective.
Experiments
Experimental setup
Benchmark protocol and direct VLMs. All systems answer the same 1,320 questions, a type-balanced subset of the 1,533 test questions with 120 per detailed type. Metrics follow § 3.5. We evaluate nine open-weight and three proprietary VLMs, listed in Table S11. These direct VLMs answer from public references and up to 32 timestamped frames shared across models. Uniform guessing and a public-text ridge classifier serve as baselines.
| Open-weight models | ||
Qwen3.5-4B [49]<br>Qwen/Qwen3.5-4B | Gemma 4 12B [20]<br>google/gemma-4-12B-it | Qwen3.8-27B [50]<br>Qwen/Qwen3.8-27B |
InternVL3.5-8B [59]<br>OpenGVLab/InternVL3_5-8B | MiMo-VL-7B-RL [66]<br>XiaomiMiMo/MiMo-VL-7B-RL | VideoLLaMA3-7B [71]<br>DAMO-NLP-SG/VideoLLaMA3-7B |
InternVideo3-8B [67]<br>yanziang/InternVideo3-8B-Instruct | VideoChat-Flash-7B [32]<br>OpenGVLab/VideoChat-Flash-Qwen2-7B_res448 | LongVILA-R1-7B [10]<br>Efficient-Large-Model/LongVILA-R1-7B |
| Closed-source models | ||
GPT-6.1 Sol [44]<br>gpt-6.1-sol | Gemini 4 Argon [24]<br>gemini-4-argon | Claude Opus 5.5 [4]<br>claude-opus-5-5 |
| Prompted API controllers | ||
GPT-5.3 Chat [43]<br>azure/gpt-5.3-chat | Gemini 3.5 Flash-Lite [22]<br>gemini-3.5-flash-lite | Claude Haiku 5.5 [3]<br>claude-haiku-5-5 |
Table S11. Table S11 Evaluated direct models and API controllers. Each entry gives the model and its checkpoint or API identifier. The shared-tool Base, SFT, and GRPO conditions use Qwen3.5-4B.
EgoClue controllers. To isolate controller learning, Base, SFT, and SFT+GRPO share Gemini 3.6 Flash perception, GPT-5.3 Chat retrieval, limits of 12 decisions and 11 tool calls, and greedy decoding. SFT trains for 0.1 epoch on 1,100 teacher dialogues with a learning rate of . GRPO then trains on 4,650 other questions with four rollouts each and a learning rate of for 259 steps. GPT-5.3 Chat, Gemini 4 Argon, Gemini 3.5 Flash-Lite, and Claude Haiku 5.5 run as zero-shot controllers in the same setting.
Main results
Instance-level memory remains challenging. Table 3 groups direct VLMs, prompted shared-tool controllers, trained EgoClue controllers, and evidence synthesis, and reports accuracy per ability, the Macro and Micro aggregates, and cross-environment Micro. Direct VLMs show a large gap between model families. Proprietary VLMs reach 58.79% to 73.56% Micro accuracy, whereas open-weight VLMs reach 28.11% to 46.97%. The public-text ridge classifier reaches 31.59%, only 5.07 points above uniform guessing, so most answers require visual evidence. Accuracy also varies by ability. Room path Q05 is one of the two easiest abilities for all three proprietary models, reaching up to 91.67%. Latest-witness Q03 is the hardest type for eleven of the twelve direct models, and even Gemini reaches only 34.17% from uniformly sampled frames. In Table S28, the proprietary models omit positive windows in 56% of their Q04 errors and undercount visits in 77% of Q07E errors, while 89% of their Q02 errors credit a wearer who did not see the target. These patterns suggest that current VLMs struggle mainly to retrieve every relevant sighting from a long multi-wearer history and verify its identity.
| Model or system | Q01 | Q02 | Q03 | Q04 | Q05 | Q06 | Q07 | Macro | Week-long Micro | Cross-env. Micro | |
| Uniform random, expected | 33.33 | 25.00 | 25.00 | 25.00 | 25.00 | 33.33 | 25.00 | 27.38 | 26.52 | – | 27.12 |
| Public-text ridge | 42.50 | 28.33 | 27.50 | 27.50 | 25.00 | 43.33 | 30.67 | 32.12 | 31.59 | – | 27.06 |
| Direct open-weight VLMs | |||||||||||
| Qwen3.8-27B [50] | 38.33 | 50.00 | 27.50 | 57.50 | 52.50 | 62.50 | 45.67 | 47.71 | 46.97 | 0.00 | 58.07 |
| VideoChat-Flash-7B [32] | 62.50 | 26.67 | 25.00 | 32.50 | 63.33 | 30.83 | 43.83 | 40.67 | 41.82 | -5.15 | 38.37 |
| MiMo-VL-7B-RL [66] | 35.00 | 43.33 | 17.50 | 37.50 | 47.50 | 36.67 | 46.00 | 37.64 | 40.68 | -6.29 | 46.25 |
| VideoLLaMA3-7B [71] | 49.17 | 45.83 | 20.83 | 38.33 | 52.50 | 48.33 | 37.50 | 41.79 | 40.23 | -6.74 | 47.78 |
| InternVideo3-8B [67] | 37.50 | 57.50 | 23.33 | 40.83 | 49.17 | 33.33 | 39.33 | 40.14 | 39.85 | -7.12 | 44.09 |
| InternVL3.5-8B [59] | 43.33 | 44.17 | 22.50 | 43.33 | 59.17 | 35.00 | 37.83 | 40.76 | 39.70 | -7.27 | 47.90 |
| LongVILA-R1-7B [10] | 47.50 | 32.50 | 15.00 | 31.67 | 31.67 | 39.17 | 24.17 | 31.67 | 28.94 | -18.03 | 28.46 |
| Gemma 4 12B [20] | 25.83 | 50.00 | 5.83 | 29.17 | 14.17 | 36.67 | 30.83 | 27.50 | 28.71 | -18.26 | 49.81 |
| Qwen3.5-4B [49] | 29.17 | 35.83 | 25.83 | 30.83 | 10.00 | 34.17 | 28.67 | 27.79 | 28.11 | -18.86 | 41.93 |
| Direct proprietary VLMs | |||||||||||
| Gemini 4 Argon [24] | 74.17 | 83.33 | 34.17 | 77.50 | 85.83 | 84.17 | 74.00 | 73.31 | 73.56 | 0.00 | 86.28 |
| Claude Opus 5.5 [4] | 70.00 | 80.00 | 29.17 | 73.33 | 91.67 | 80.00 | 69.17 | 70.48 | 70.00 | -3.56 | 80.81 |
| GPT-6.1 Sol [44] | 62.50 | 77.50 | 28.33 | 61.67 | 74.17 | 65.00 | 55.50 | 60.67 | 58.79 | -14.77 | 67.60 |
| Prompted shared-tool controllers: relative to Qwen3.5-4B base | |||||||||||
| Qwen3.5-4B [49] (base) | 43.33 | 49.17 | 3.33 | 20.00 | 18.33 | 6.67 | 23.83 | 23.52 | 23.64 | 0.00 | 22.24 |
| GPT-5.3 Chat [43] | 69.17 | 66.67 | 27.50 | 52.50 | 62.50 | 80.83 | 72.83 | 61.71 | 65.76 | +42.12 | 67.85 |
| Gemini 4 Argon [24] | 63.33 | 60.00 | 7.50 | 35.00 | 54.17 | 52.50 | 47.00 | 45.64 | 46.14 | +22.50 | 71.54 |
| Gemini 3.5 Flash-Lite [22] | 37.50 | 31.67 | 16.67 | 36.67 | 68.33 | 57.50 | 33.00 | 40.19 | 37.58 | +13.94 | 52.60 |
| Claude Haiku 5.5 [3] | 40.83 | 62.50 | 5.00 | 26.67 | 34.17 | 44.17 | 28.17 | 34.50 | 32.20 | +8.56 | 49.56 |
| Trained EgoClue controllers: relative to Qwen3.5-4B base | |||||||||||
| EgoClue-SFT | 64.17 | 61.67 | 15.83 | 32.50 | 55.00 | 80.00 | 67.00 | 53.74 | 58.56 | +34.92 | 68.11 |
| EgoClue-SFT+GRPO | 71.67 | 68.33 | 38.33 | 53.33 | 73.33 | 87.50 | 81.33 | 67.69 | 72.65 | +49.02 | 78.40 |
| Evidence-synthesis system: no additional visual calls | |||||||||||
| EgoClue + Sol [44] | 72.50 | 67.50 | 44.17 | 55.83 | 77.50 | 87.50 | 83.67 | 69.81 | 74.85 | +2.20 | 80.56 |
Table 3. Spatial memory and reasoning across model and system families. Accuracy in % on 1,320 week-long and 787 cross-environment questions. Direct VLMs use public references and up to 32 frames. Agents share the budget in Table S12. Macro/Micro average abilities/questions. Bold/underline mark group-best/runner-up aggregates. reports Micro differences in percentage points relative to direct-group leaders, 4B base for controllers, or SFT+GRPO for synthesis.
| Type | Err. (%) | Dominant error direction |
| Object-instance memory | ||
| Q01 | 31.1 | different categories 44%; merge 36% |
| Q02 | 19.7 | over-report 89%; under-report 11% |
| Q03 | 69.4 | – |
| Q04 | 29.3 | omit a positive window 56% |
| Room and visit memory | ||
| Q05 | 16.2 | shorter 48%; longer 45% |
| Q06 | 23.6 | type confusion 49%; split 28% |
| Q07A | 16.1 | extra rooms 52%; missing 36% |
| Q07B | 17.2 | overestimate 68% |
| Q07C | 41.5 | underestimate 55% |
| Q07D | 38.6 | – |
| Q07E | 55.3 | undercount 77% |
Table S28. Frontier-VLM error patterns by question type. Error pools the stored direct predictions of Gemini 4 Argon, Claude Opus 5.5, and GPT-6.1 Sol (358–360 answered per type). Shares are fractions of errors.
Training drives the gains. In Table 3, the prompted Base controller reaches 23.64%. With the same tools, the strongest prompted API controller, GPT-5.3 Chat, reaches 65.76%, 6.89 points below EgoClue-SFT+GRPO. SFT raises accuracy to 58.56%, and GRPO reaches 72.65%. GRPO improves all seven abilities over SFT. The largest subtype gains are 25.00 points for completed-visit duration, 22.50 for last-observer attribution, and 20.83 for historical-window verification. These subtypes require Timeline computations or repeated Video checks of candidate sightings. Figure 5 tracks reward, policy entropy, response tokens, and dialogue turns. SFT+GRPO keeps a higher reward with about seven turns, whereas RL without SFT reaches a lower reward with about 25 turns. Its policy entropy also drops to about 0.1 within 100 steps, while SFT+GRPO stays near 0.6, so SFT keeps exploration alive for GRPO. On 1,021 applicable questions, GRPO raises joint answer-evidence correctness, and its lower bound exceeds the SFT upper bound in Table S24.
Figure 5. EgoClue training dynamics. Thin traces show raw scalars and thick traces show EMA with coefficient 0.2. Supp. Sec. D.1 details both training schedules.
| Type | Applicable | Continuous joint : SFT | Continuous joint : GRPO | Continuous joint : + Sol | Strict semantic success : SFT | Strict semantic success : GRPO | Strict semantic success : + Sol |
| Q02 | 95 | 34.74–49.47 | 33.68–47.37 | 43.16–61.05 | 34.74–49.47 | 33.68–47.37 | 43.16–61.05 |
| Q03 | 120 | 0.00–8.33 | 0.00–17.50 | 0.00–20.00 | 0.00–8.33 | 0.00–17.50 | 0.00–20.00 |
| Q05 | 120 | 40.00 | 52.08 | 54.17 | 29.17 | 37.50 | 39.17 |
| Q06 | 120 | 56.67 | 62.92 | 60.00 | 45.00 | 52.50 | 51.67 |
| Q07A | 120 | 61.66 | 69.79 | 68.95 | 55.83 | 61.67 | 62.50 |
| Q07B | 120 | 44.84 | 47.86 | 47.82 | 56.67 | 60.00 | 60.00 |
| Q07C | 120 | 33.45 | 47.05 | 47.49 | 33.33 | 46.67 | 47.50 |
| Q07D | 120 | 59.89 | 69.67 | 68.74 | 65.83 | 75.00 | 72.50 |
| Q07E | 86 | 15.36 | 30.95 | 35.44 | 10.47 | 24.42 | 27.91 |
| All | 1,021 | 39.37–41.72 | 46.80–50.13 | 47.80–51.82 | 37.71–40.06 | 44.37–47.70 | 45.54–49.56 |
Table S24. Table S24 Semantic joint correctness by week-long type. Values are percentages on the fixed applicable subset. Ranges bound unresolved scores. Q01 and Q04 are answer-only under the frozen object contract. Q02 excludes 25 evidence-inapplicable cases and Q07E excludes 34, identically across runs. Scores include Q03 exact-citation normalization, applied by the same rule to SFT, GRPO, and synthesis with the frozen scorer.
Evidence synthesis improves accuracy. Sol synthesizes answers from the observations the controller retains. This raises EgoClue’s accuracy from 72.65% to 74.85%, the highest week-long Micro in Table 3, with 95% intervals in Table S14. The same Sol model scores 58.79% under direct access, so the gain comes from the evidence the controller gathers. Cross-environment evaluation tests the week-long-trained system on 787 questions from other environments. The cross-environment setting in Supp. Sec. G uses only Image, Video, and Timeline. With the same budget as all controllers, EgoClue reaches 78.40%, above every prompted controller, and synthesis raises it to 80.56%.
| System | Micro | Macro |
| Public-text ridge | 31.59 [29.19, 34.08] | 32.12 [29.19, 35.08] |
| Direct open-weight VLMs | ||
| Qwen3.8-27B | 46.97 [43.98, 50.00] | 47.71 [44.44, 50.99] |
| VideoChat-Flash-7B | 41.82 [38.95, 44.76] | 40.67 [37.72, 43.70] |
| MiMo-VL-7B-RL | 40.68 [37.72, 43.68] | 37.64 [34.60, 40.77] |
| VideoLLaMA3-7B | 40.23 [37.16, 43.28] | 41.79 [38.42, 45.21] |
| InternVideo3-8B | 39.85 [37.00, 42.84] | 40.14 [36.91, 43.55] |
| InternVL3.5-8B | 39.70 [36.77, 42.64] | 40.76 [37.38, 44.23] |
| LongVILA-R1-7B | 28.94 [26.42, 31.70] | 31.67 [28.77, 34.92] |
| Gemma 4 12B | 28.71 [25.90, 31.57] | 27.50 [24.74, 30.22] |
| Qwen3.5-4B | 28.11 [25.76, 30.52] | 27.79 [25.17, 30.43] |
| Direct proprietary VLMs | ||
| Gemini 4 Argon | 73.56 [70.15, 76.98] | 73.31 [70.35, 76.55] |
| Claude Opus 5.5 | 70.00 [66.40, 73.49] | 70.48 [67.41, 73.71] |
| GPT-6.1 Sol | 58.79 [55.54, 62.07] | 60.67 [57.59, 63.82] |
| Prompted shared-tool controllers | ||
| Qwen3.5-4B, base | 23.64 [20.99, 26.34] | 23.52 [20.97, 26.12] |
| GPT-5.3 Chat | 65.76 [62.25, 69.24] | 61.71 [58.31, 65.29] |
| Gemini 4 Argon | 46.14 [42.77, 49.50] | 45.64 [42.73, 48.48] |
| Gemini 3.5 Flash-Lite | 37.58 [34.48, 40.58] | 40.19 [36.94, 43.80] |
| Claude Haiku 5.5 | 32.20 [29.42, 35.19] | 34.50 [31.64, 37.39] |
| Trained controllers and synthesis | ||
| EgoClue-SFT | 58.56 [54.89, 62.29] | 53.74 [50.87, 56.83] |
| EgoClue-SFT+GRPO | 72.65 [69.38, 75.79] | 67.69 [64.48, 70.92] |
| EgoClue + Sol | 74.85 [71.89, 77.77] | 69.81 [66.66, 72.91] |
Table S14. Table S14 Week-long accuracy with 95% intervals. Micro and Macro in percent with percentile intervals over 5,000 core-group resamples.
Ablation and cost analysis
To identify which tools matter, we remove each of the five tools individually. Questions, controller, backends, and budget remain fixed. Removing Video lowers accuracy by 25.68 points, and the other removals cost 3.94 to 8.64 points in Table 4. The right half of Table 4 keeps questions where the full agent invokes the removed tool, and there the Topology, Timeline, and Video drops reach 25.23, 26.97, and 32.85 points. Paired core-group bootstrap intervals exclude zero. Without Video, Q07 visit reasoning falls from 81.33% to 37.17% and latest-witness Q03 from 38.33% to 21.67%, while object identity Q01 changes by only 1.67 points because Image still compares the two references. Removing Topology lowers room path Q05 from 73.33% to 54.17%, and removing Timeline lowers the duration questions Q07B and Q07C from 82.08% to 58.33%. SFT initialization also matters. Without SFT, RL from the base controller learns the output format, with 99.24% validity, but reaches 40.68%, 31.97 points below SFT+GRPO. GRPO also reduces mean tool calls from SFT’s 2.67 to 2.31 per question. Against the prompted shared-tool controllers, EgoClue needs the fewest turns, 3.30 versus 4.34 to 8.07, and the fewest proprietary-model tokens, 38.1k versus 44.3k to 97.4k per question, since its controller runs locally. At list prices, it is also the cheapest system above 65% accuracy in Supp. Sec. F.4.
| Configuration | Correct | Accuracy, % | Questions | Full | Ablated | ||
| Full | 959 | 72.65 | 0.00 | ||||
| Without Video | 620 | 46.97 | -25.68 | 971 | 688 | 369 | -32.85 |
| Without Timeline | 845 | 64.02 | -8.64 | 178 | 151 | 103 | -26.97 |
| Without Topology | 881 | 66.74 | -5.91 | 107 | 83 | 56 | -25.23 |
| Without Memory | 900 | 68.18 | -4.47 | 111 | 44 | 31 | -11.71 |
| Without Image | 907 | 68.71 | -3.94 | 458 | 322 | 312 | -2.18 |
Table 4. Individual tool removal from EgoClue. Left: all 1,320 test questions. Right: questions on which Full invokes the removed tool, with Full and Ablated counting correct answers. Signed changes are percentage points relative to the Full evaluation results. Supp. Sec. F.3 details the paired comparison.
Conclusion
EgoWitness evaluates instance-level spatial memory through 8,453 questions that link object and room identity to past sightings across days and wearers. Over one million reviewed masks and room timelines ground each answer in verifiable observations. Twelve direct VLMs struggle most on identifying who saw an object last. As a strong baseline, EgoClue searches for and verifies evidence with five tools. Removing Video lowers accuracy most, by 25.68 points, highlighting the importance of revisiting recorded intervals. Two-stage training raises its 4B controller from 23.64% to 72.65%. Evidence synthesis reaches 74.85% on week-long questions and 80.56% in unseen environments.
LLM Usage Statement
OpenAI Codex assisted with manuscript drafting and editing, source inspection, code and LaTeX preparation, and figure layout. GPT-6.1 Sol assisted with first-pass QA review and evidence-only answer synthesis. The research pipeline also uses language and vision-language models for candidate annotation review, question wording, teacher tool-use trajectories, and memory construction. Qwen3.5-4B is the trained controller, with Gemini 3.6 Flash and GPT-5.3 Chat providing visual and retrieval services. The method and supplementary material specify model roles and settings. Trained experts verify the QA items after model-assisted review. The authors retain responsibility for the scientific claims, annotations, code, figures, and results.
Supplementary Material
This supplement documents how EgoWitness is built and how models are evaluated on it. Appendices A–C cover the data sources, the room, object, and landmark annotations, and question construction and verification. Appendices D–G cover EgoClue training, the evaluation setup, additional results, and answer synthesis, and Appendix H presents case studies and error analysis.
Data sources and annotation integration
Table S1 distinguishes original recordings from the annotation resources applied to them. EgoLife provides the week-long recordings and house assets. The short-horizon resources share Ego4D source-video identifiers, which we use to align their annotations and count each original video once.
| Resource | Layer | Scope / sampling | Integration in EgoWitness |
| Week-long recordings | |||
| EgoLife [68] | Videos, house assets | Six wearers, seven days | New room intervals, instance links, and masks |
| Short-horizon recordings and annotations | |||
| Ego4D [25] | Videos | 43 original videos | Shared video IDs and time axes |
| EgoMask / EgoTracks [33, 57] | Masks, track lineage | 1 Hz retained | Boxes and local tracks aligned to rooms |
| RefEgo [31] | Boxes, expressions | 2 Hz, width 640 | Target deduplication and coordinate scaling |
| EgoEnv [42] | Room intervals, references | 400 source-local entries | Observer-room context for object sightings |
Table S1. Source resources and their integration. Rows distinguish original media from inherited annotations. Sampling rates describe the annotations retained in EgoWitness. The shared Ego4D videos are counted once across resources.
EgoLife recordings and new annotations
We add room intervals, instance associations, and mask observations to EgoLife [68]. Room IDs are shared across six wearers and seven days, allowing an observation to be linked to both its target and its observing wearer. Appendix § 11 specifies the annotation operations, acceptance rules, and exported records. Question construction is described separately in Appendix § 12.
Ego4D-derived annotations and alignment
Matching annotation sources. We associate object annotations from EgoMask and RefEgo with observer-room intervals from EgoEnv using source-video identifiers and aligned time intervals [31, 33, 42]. For each object observation, the join finds the compatible room interval on the same video timeline:
Here denotes this temporal association, and the union combines the two aligned object sources while retaining their provenance. The selected subset contains 43 original videos, 83 source clips, 170 source-local tracks, and 172 cross-room observation candidates. Table S2 lists the per-source counts. Four original videos occur in both object sources. We preserve each source’s track IDs and record cross-room observation support as a separate attribute. EgoTracks [57] provides EgoMask’s upstream tracking annotations over Ego4D media [25].
| Quantity | EgoMask | RefEgo | Total |
| Source media | |||
| Original videos | 37 | 10 | 43 |
| Source clips | 70 | 13 | 83 |
| Tracks and object observations | |||
| Source-local tracks | 157 | 13 | 170 |
| Observation candidates | 159 | 13 | 172 |
| Visible boxes | 13,988 | 282 | 14,270 |
| Matched visible boxes | 13,402 | 253 | 13,655 |
| Retained annotation rate, Hz | 1 | 2 | mixed |
Table S2. Selected short-horizon data. Counts from EgoMask-Train and RefEgo. Totals deduplicate videos. Tracks retain source-local scope. “Matched” denotes an unambiguous association with the camera wearer’s occupied room.
Media completion uses 15 full-duration 224p versions from the MickeyLLG/ego_plan repository [41]. These copies belong to the same 43-video inventory.
Media duration. The 43 full videos span 23.46 hours, equivalent to 84,441.037 seconds, and occupy 17.94 GB. The manifest records each video’s resolution. EgoLife reports approximately 300 hours of recordings [68], and this collection adds approximately 24 hours, giving the approximately 324-hour source scale.
Visible-box and room-matched-box counts in Table S2 specify the usable object observations.
Track selection. Within these 43 videos, we select 170 tracks with unambiguous EgoEnv room matches. Cross-room support describes the observer’s room contexts, and a stationary target can be seen from different rooms.
Time and geometry alignment. EgoMask retains 1 Hz observations although its frame filenames use a 5 Hz index: the original-video time is the clip offset plus the filename index divided by five. RefEgo retains its nominal 2 Hz convention, with time seconds, where and are its image ID and image-clip ID. RefEgo boxes refer to annotation images of width 640, so coordinates are normalized using the source annotation dimensions and rescaled to the actual media dimensions. Duplicate expressions for the same RefEgo target are deduplicated. The original aspect ratio is preserved.
Room semantics and identity scope. The room-reference catalog contains 400 source-local entries across 81 scopes and 18 room types. Each room instance is scoped by source video, EgoEnv clip, room label, and instance index. The temporal join assigns the observer’s occupied room, which can differ from the visible target’s room. Cross-room candidates require observations in at least two room contexts. The join excludes overlapping intervals and a 0.5-second band around room boundaries, and connectivity records the room transitions observed in the video.
Short-horizon questions. The short-horizon component contributes 787 reviewed questions from 40 videos to the 8,453-question benchmark. EgoWitness reserves all 787 for evaluation and excludes their environments from SFT and GRPO. They cover 120 object-identity questions in Q01, 169 object-history questions in Q04, 80 room-identity questions in Q06, and 418 visit-reasoning questions across Q07B to Q07E. The 418 visit-reasoning questions carry 676 scoring-evidence references over 316 unique intervals of 2 to 4 seconds. Identity and object-history questions are scored from public references and candidate windows. Each question retains the source-video timeline and source-local room identity used to derive its answer. Video, participant-group, and core-task identifiers determine split membership. Deterministic checks verify answer consistency against the selected room intervals.
Room, object, and landmark annotation
Ten annotators produced the room timelines and the movable-object and landmark annotations. Figure S1 shows their two linked workspaces: room-interval editing and instance association. We describe the editing operations and export rules first, followed by identity checks, annotation fields, and cross-wearer coverage.
Figure S1. Annotation workspaces and review operations. A. The Online Room Curator System supports boundary editing, room-instance assignment, landmark registration, and visual verification. The DAY1/A1 example contains 14 reviewed intervals. Shared room IDs and aligned intervals supply context to B, where annotators compare landmark and movable-object instances, inspect logical conflicts, and correct associations before export. A5, A6, and A4 show one shared instance. Room-boundary conflicts return to A for interval correction.
Room segment annotation
In Figure S1A, annotators replay segments, adjust transition boundaries, and assign room instances in the Online Room Curator System. The interface organizes work by wearer, day, and review status. Annotators inspect frames around a transition and split, merge, or adjust its intervals. They assign room type and room instance separately and record the decision. The room label describes the camera wearer’s occupied space, including when the camera looks through a doorway into another room.
Spatial references and room identities
Figure S2A locates the courtyard and bedroom assignments within the shared home. Panels B and C pair the upper- and ground-floor plans with video observations of the corresponding rooms. Together, the panels identify all 17 room and zone IDs used in annotation. The exterior model and floor plans use the official EgoLife house assets [68]. GF denotes the ground floor and 1F the upper floor. The courtyard belongs to the home and remains a named zone. Annotators use these references to distinguish adjacent rooms at doorways, corridors, and stairs, then assign the occupied room from the video.
Figure S2. Spatial reference for the week-long environment. A. The official EgoLife house model, a courtyard observation, and bedroom ownership. B and C. Upper- and ground-floor plans paired with real video observations of all 16 indoor room instances; the courtyard is the seventeenth named zone. Numbered pins match the photo cards, which retain the room IDs and colors used in Figure S3. Geometry and textures come from the official EgoLife assets [68]. The pins are schematic labels.
Room timeline scope
Figure S3A places the six wearers’ room intervals on a shared seven-day timeline. Panel B expands one hour on DAY6 to show their different room observations at the same time. Shared colors identify rooms and zones, while white gaps mark missing recordings. The media manifest groups 32,001 source videos into these 42 wearer-day streams. We compute recording coverage within each stream and sum durations over wearers to obtain camera-hours.
Figure S3. Room observations across wearers and days. A. Each row represents one wearer and each column one day, with a common 09:00 to 24:00 day-clock axis. Colors identify 17 named rooms and zones, with separate colors for outdoor, transition, and unresolved-room states. White gaps indicate missing recordings. B. The DAY6 window from 15:00 to 16:00, outlined in A, expands the room intervals for all six wearers. Plotted intervals are clipped to the media manifest, preserving their boundaries and recording gaps.
Interval export and completion. The annotation store contains 4,408 source parent segments, 695 parent-review records, and 17,076 saved subintervals. Saved subintervals replace their parent annotation to avoid overlapping representations. Otherwise, a completed parent review contributes one whole-parent interval. This yields 4,257 retained parents and 17,269 intervals: 17,076 subintervals plus 193 whole-parent reviews. Parent segments without recorded video are excluded. The completion rule requires saved, human-reviewed coverage, rejects overlapping child intervals, and tolerates uncovered boundary gaps of at most 0.5 seconds. The resulting intervals record 13,886 accepted and 3,383 corrected decisions.
Recording coverage. The retained intervals cover every recorded stream. The export records the room-database snapshot separately from the object-layer versions.
Room IDs and occupancy states. The 17 room and zone IDs in Figure S2 are shared across recordings. Each ID identifies the same physical space. The courtyard is an in-property zone with no assigned floor. Occupancy states are stored separately: 16,964 intervals are inside a room, 295 are outdoors, nine are transitions, and one is unknown. The occupancy state distinguishes outdoor observations, transitions, and unknown states.
Movable-object and landmark annotation
In Figure S1B, accepted room intervals supply the context for comparing object views with registered references. Annotators resolve identity associations before exporting the reviewed records. The following counts describe the reviewed prompts and their propagated mask observations, and Table S3 summarizes the accepted masks by layer.
| Landmark | Movable | |
| Instance/target labels | 34 | 21 |
| Sampling rate, Hz | 1 | 5 |
| Mask observations | 795,319 | 238,756 |
| Annotated frames | 453,813 | 224,123 |
| Videos with accepted masks | 22,335 | 2,164 |
| Local SAM tracks | 90,261 | 2,408 |
| Total accepted masks | 1,034,075 | |
| Auxiliary detector boxes | 890,360 | |
| Original anchor boxes | — | 19,921 |
Table S3. EgoLife annotation inventory. Primary counts are accepted mask observations. Auxiliary layers are retained separately.
Movable objects and identity. Filtering the 406,644 proposals from the 960-pixel detector retains 19,921 anchors in 2,403 class clips from 2,164 videos.
The filters use class-specific confidence thresholds of 0.90 to 0.97 and persistence of three to five frames, followed by duplicate and room checks in Table S4. Appearance clustering and Gemini 3.8 Flash [23] organize candidates for review. Annotators check several keyframes per track against the reference views and surrounding video. SAM 3.1 propagates these checked prompts forward and backward at 5 Hz. Visual acceptance of the propagated results yields 238,756 nonempty masks. Each mask retains its reviewed target label, local track, and quality flags.
| Rule | Action | Boxes |
| Movable candidates | 406,644 boxes | |
| Filtering | ||
| Below class confidence | Reject | 357,116 |
| Below minimum persistence | Reject | 5,729 |
| Duplicate box | Suppress | 7,342 |
| Wearer outdoors | Exclude | 16,001 |
| Review holds | ||
| Unknown room or transition | Hold | 368 |
| Box quality | Hold | 148 |
| Uncertain detection | Hold | 19 |
Table S4. Context and quality rules by action. Counts describe boxes affected by each rule among the 406,644 movable candidates. Held candidates proceed to visual review.
Registered landmarks. A detector proposes landmark boxes, a verifier filters them, and SAM 3.1 propagates accepted prompts at 1 Hz within local processing units, yielding 795,319 landmark masks. Each detector label indexes a registered instance whose reference frames, room, floor, and zone support cross-segment comparison. Annotators inspect conflicts with the registered floor in the source video.
Cross-view instance association. The A5, A6, and A4 views in Figure S1B show the same registered instance from different wearers. The workspace groups candidate views by room and time so annotators can compare their appearance and scene context. Annotators compare each view with the reference frames and surrounding video, then attach it to an existing instance, create an instance, or merge entries. Exported associations retain the instance ID and source views.
All 34 released landmark names match source instance records with both cross-wearer and cross-day observations, covering 70,286 source observations. These counts describe the source association registry. The propagated masks are counted separately in the released observation layer.
Context checks and visual acceptance
The review loop in Figure S1 links instance checks in panel B to room-interval corrections in panel A. Annotators inspect a conflicting association and correct either its identity or its room context.
Physical-instance consistency. We review identity assignments against two logical constraints. First, each unique physical instance receives at most one accepted detection per frame. Annotators remove erroneous or duplicate boxes after inspecting the frame. Second, one physical instance cannot occupy spatially disconnected rooms at the same time. We compare synchronized observations and inspect conflicting locations in the source video. Confirmed errors lead to box removal, identity reassignment, or room-interval correction. Annotators then recheck the affected associations using the corrected identity and room context.
These constraints apply to individually annotated movable instances and registered landmarks. Cross-room views through doorways or windows are checked against the object’s location, with each wearer’s occupied room recorded separately.
Context rules and acceptance. Table S4 groups the movable-candidate rules into filtering and review holds. A hold routes a candidate to visual review, while rejection or suppression removes it. Simultaneous sightings assigned to different floors trigger a joint review of object identity and room context. Correcting room intervals resolves 29 one-second conflicts and updates the room assignment of 338 boxes. Each accepted record stores the curator’s decision and its source observations. Model proposals retain separate provenance fields.
Unified observation records
Every accepted annotation from either source becomes one observation record, linked to its source frame by a unique observation ID. Its other fields form four groups: identifiers, locations, time and coordinates, and visibility and review status.
Category, label, instance, and track IDs
Four identifiers describe the target at different levels:
Category ID: one of 24 normalized semantic types, to which the registry maps the 34 landmark and 21 movable labels. Color and material are separate attributes.
Release label: one of the 55 original labels, kept through the release class ID and a legacy identity-group ID.
Physical-instance ID: an optional key for one physical object, linking its observations through reviewed labels and landmark associations. The two bowl labels keep group-level identity.
Track ID: one SAM prompt trajectory, local to one run and processing unit.
A shared category does not imply a shared object. Identity across videos, wearers, and days comes from the reviewed associations in Appendix § 11.§ 2.
Observer, landmark, and object locations
Three fields separate wearer and target locations:
Observer location: the room or zone the camera wearer occupies, from the reviewed room intervals.
Registered landmark location: the reference room, floor, and zone from the landmark registry.
@page 21/43 starts_mid=1 ends_mid=1
Object location: the target’s location in this observation, established independently of the observer location.
A wearer looking through a doorway or into the courtyard can occupy a different space from the target, so a change in observer location alone does not show that an object moved. Short-horizon room IDs stay local to their source.
Sampling, timestamps, and coordinates
Landmark and movable masks keep nominal rates of 1 Hz and 5 Hz. Each record stores its source-frame index, sampled-frame index, and video-relative timestamp separately. A day-clock timestamp is meaningful only with its day and wearer.
Temporal alignment maps the source-frame index and video-relative time through the clip origin in the room manifest, and inherited day-clock values are checked against this mapping before subsecond joins. The export keeps the original timestamps and frame indices for boundary-level alignment.
EgoLife boxes give corner coordinates in a central crop of the source frame; adding 216 to each coordinate restores source coordinates. Masks keep their original compressed COCO RLE in the same crop region. Appendix § 10.§ 2 gives the EgoMask and RefEgo conversions.
Visibility and review status
An accepted mask shows that the target is visible at its timestamp. Frames without an accepted mask are not treated as evidence of absence, and confirming absence requires interval review. “Last seen,” visit-count, duration, and absence questions therefore pair accepted observations with review of the whole search interval, as Appendix § 12 describes.
The primary object layers contain 795,319 landmark and 238,756 movable masks. Human acceptance, the mask-generation model, and the filtering history are stored as separate fields. Original anchors and historical detections are auxiliary records linked to the 1,034,075 primary observations. Export checks verify RLE decoding, foreground area, source-record identity, and record consistency.
Recurrence across days, wearers, and rooms
Figure S4 and b show representative movable and landmark instances recurring across days and wearers. Their shared time axes expose repeated sightings, while the wearer summaries show who observes each target. Table S5 reports coverage for all 21 movable labels and 34 registered landmarks. To compare layers sampled at 5 Hz and 1 Hz, we count distinct wearer-seconds with an accepted mask and join each second to the wearer’s room in the Online Room Curator System.
Table S5 Target recurrence across days, wearers, and rooms. Rows marked are group-level labels. Landmark rows are registered instances, and registered rooms follow each name. Numbers separate distinct instances sharing a name. For landmarks, the last column names a room only when it differs from the registered room. Seen time counts distinct wearer-seconds with an accepted mask, which removes the 5 Hz/1 Hz sampling difference. Rooms are occupied wearer rooms from the Online Room Curator System. Same minute gives the wall-clock minutes in which at least two wearers see the target label, over all minutes with a sighting, and the largest number of wearers in one minute. Cell shading follows each wearer’s share. Summary rows give totals, medians for days, wearers, and rooms, and pooled shares. Object icons are schematic.
| Target | Masks | Tracks | Days | Wear. | Rooms | ≥2/seen | Max | A1 | A2 | A3 | A4 | A5 | A6 | Top wearer, room (%) |
| Movable targets (21 labels) | ||||||||||||||
| Yellow detergent bottle | 32,410 | 342 | 6 | 6 | 5 | 38/206 | 4 | 26 | 30 | 24 | 12 | 5 | 4 | A2, kitchen_gf_0 (26) |
| Stainless-steel pot / bowl (no side handles) | 23,456 | 242 | 6 | 6 | 5 | 33/140 | 6 | 18 | 15 | 21 | 38 | 6 | 3 | A4, living_gf_0 (35) |
| Umbrella | 22,376 | 243 | 5 | 6 | 2 | 30/141 | 4 | 19 | 4 | 21 | 16 | 6 | 34 | A6, courtyard_0 (34) |
| Seasoning / soap bottle | 16,786 | 172 | 5 | 6 | 2 | 11/121 | 2 | 11 | 15 | 26 | 15 | 32 | 1 | A5, kitchen_gf_0 (32) |
| Trash bin | 15,185 | 181 | 4 | 6 | 5 | 17/128 | 3 | 11 | 13 | 7 | 15 | 36 | 17 | A5, living_1f_0 (31) |
| Stainless-steel strainer | 15,012 | 131 | 7 | 6 | 3 | 19/77 | 3 | 13 | 18 | 11 | 27 | 29 | 2 | A5, living_gf_0 (29) |
| Small baking bowl / cup | 15,146 | 120 | 4 | 5 | 2 | 23/47 | 4 | 29 | 18 | 15 | 22 | 16 | 0 | A1, living_gf_0 (29) |
| Cooking-wine bottle | 14,049 | 141 | 4 | 6 | 2 | 4/108 | 2 | 28 | 15 | 40 | 2 | 11 | 3 | A3, kitchen_gf_0 (39) |
| Moutai bottle | 13,819 | 112 | 4 | 6 | 2 | 0/84 | 1 | 2 | 22 | 6 | 57 | 3 | 10 | A4, living_gf_0 (54) |
| Seasoning bottle | 11,082 | 100 | 4 | 4 | 2 | 8/56 | 2 | 0 | 18 | 45 | 1 | 0 | 36 | A3, kitchen_gf_0 (45) |
| Stainless-steel pot (two side handles) | 10,031 | 81 | 2 | 5 | 1 | 6/52 | 2 | 43 | 32 | 10 | 11 | 4 | 0 | A1, living_gf_0 (43) |
| Red / maroon scissors | 8,657 | 84 | 3 | 6 | 2 | 12/45 | 3 | 9 | 69 | 11 | 4 | 5 | 1 | A2, living_gf_0 (69) |
| Blue scissors | 8,278 | 64 | 3 | 6 | 2 | 1/40 | 2 | 3 | 84 | 1 | 1 | 2 | 10 | A2, living_gf_0 (84) |
| Detergent bottle | 7,415 | 81 | 3 | 5 | 2 | 5/57 | 2 | 18 | 5 | 0 | 5 | 25 | 47 | A6, kitchen_1f_0 (44) |
| Vase | 6,343 | 112 | 7 | 6 | 4 | 6/95 | 2 | 14 | 24 | 7 | 19 | 8 | 28 | A6, kitchen_gf_0 (26) |
| Apple Pencil | 5,346 | 41 | 3 | 2 | 1 | 0/30 | 1 | 0 | 2 | 98 | 0 | 0 | 0 | A3, living_gf_0 (98) |
| Laptop | 5,327 | 40 | 4 | 1 | 5 | 0/31 | 1 | 0 | 0 | 0 | 0 | 0 | 100 | A6, courtyard_0 (80) |
| Flower / plant | 2,367 | 37 | 7 | 6 | 3 | 1/32 | 2 | 6 | 39 | 6 | 14 | 14 | 21 | A2, kitchen_gf_0 (34) |
| Pot with lid | 2,052 | 36 | 5 | 6 | 2 | 1/32 | 2 | 14 | 47 | 13 | 3 | 8 | 15 | A2, kitchen_gf_0 (41) |
| Whiteboard marker | 1,977 | 20 | 2 | 1 | 1 | 0/15 | 1 | 0 | 0 | 0 | 0 | 0 | 100 | A6, living_gf_0 (100) |
| Gray bowl | 1,642 | 28 | 6 | 6 | 2 | 1/25 | 4 | 18 | 26 | 20 | 20 | 13 | 3 | A3, kitchen_gf_0 (19) |
| All 21 targets | 238,756 | 2,408 | 4 | 6 | 2 | 216/1,562 | 6 | 16 | 22 | 19 | 17 | 12 | 14 | A2, living_gf_0 (13) |
| Registered landmark instances (34) | ||||||||||||||
| Dining table 1 (living_gf_0) | 173,760 | 10,323 | 7 | 6 | 8 | 1,588/2,261 | 6 | 18 | 20 | 19 | 20 | 18 | 6 | A4 (20) |
| Projection screen 1 (living_1f_0) | 74,669 | 6,194 | 7 | 6 | 7 | 866/1,401 | 6 | 19 | 15 | 8 | 14 | 19 | 25 | A6 (24) |
| Dining table 2 (living_1f_0) | 60,558 | 3,881 | 6 | 6 | 7 | 600/1,207 | 6 | 14 | 15 | 15 | 16 | 16 | 24 | A6 (24) |
| Projection screen 2 (living_gf_0) | 60,418 | 7,514 | 7 | 6 | 9 | 1,369/2,167 | 6 | 16 | 18 | 14 | 22 | 18 | 12 | A4 (22) |
| Blackboard / whiteboard 1 (living_gf_0) | 44,399 | 4,392 | 5 | 6 | 7 | 776/1,181 | 6 | 19 | 11 | 16 | 20 | 19 | 15 | A4 (20) |
| Photographic softbox 1 (living_1f_0) | 40,394 | 4,002 | 7 | 6 | 6 | 643/1,001 | 6 | 19 | 19 | 14 | 15 | 12 | 22 | A6 (22) |
| Cabinet shelf 1 (living_gf_0) | 34,536 | 4,305 | 7 | 6 | 8 | 778/1,675 | 6 | 25 | 21 | 20 | 13 | 13 | 9 | A1 (24) |
| Long table (courtyard_0) | 34,381 | 1,644 | 7 | 6 | 2 | 258/512 | 6 | 15 | 13 | 8 | 15 | 22 | 26 | A6 (26) |
| Cabinet shelf 2 (living_gf_0) | 30,266 | 4,484 | 7 | 6 | 9 | 857/1,786 | 6 | 15 | 17 | 12 | 25 | 18 | 14 | A4 (24) |
| Red fridge 1 (living_gf_0) | 23,449 | 3,793 | 7 | 6 | 6 | 763/1,567 | 6 | 15 | 37 | 17 | 13 | 11 | 8 | A2 (35) |
| Sink counter cabinet 1 (kitchen_gf_0) | 23,375 | 4,670 | 7 | 6 | 8 | 939/1,782 | 6 | 23 | 28 | 11 | 11 | 11 | 15 | A2, living_gf_0 (22) |
| Potted plant 1 (living_gf_0) | 22,061 | 3,464 | 7 | 6 | 8 | 671/1,515 | 6 | 12 | 19 | 16 | 22 | 18 | 12 | A4 (22) |
| White microwave GF-0 (kitchen_gf_0) | 20,587 | 3,474 | 7 | 6 | 7 | 658/1,533 | 6 | 22 | 20 | 25 | 11 | 16 | 6 | A1, living_gf_0 (18) |
| Photographic softbox 2 (living_gf_0) | 19,164 | 2,513 | 7 | 6 | 6 | 485/976 | 5 | 19 | 13 | 14 | 21 | 20 | 12 | A4 (20) |
| Lamp (living_gf_0) | 18,173 | 4,482 | 7 | 6 | 9 | 944/1,770 | 6 | 20 | 15 | 10 | 22 | 13 | 19 | A4 (20) |
| Potted plant 2 (living_1f_0) | 14,956 | 1,910 | 7 | 6 | 6 | 341/781 | 5 | 17 | 17 | 4 | 22 | 19 | 20 | A4 (22) |
| Blackboard / whiteboard 2 (living_gf_0) | 14,436 | 1,831 | 7 | 6 | 7 | 305/871 | 6 | 24 | 6 | 14 | 23 | 28 | 6 | A6, courtyard_0 (25) |
| Black BBQ grill (courtyard_0) | 14,169 | 2,185 | 7 | 6 | 6 | 396/961 | 6 | 27 | 29 | 11 | 6 | 12 | 16 | A2 (25) |
| White microwave GF-1 (kitchen_gf_0) | 12,928 | 2,287 | 7 | 6 | 7 | 396/1,139 | 6 | 23 | 21 | 23 | 12 | 13 | 8 | A1, living_gf_0 (21) |
| Laptop rack (bedroom_gf_0) | 8,501 | 1,189 | 7 | 6 | 6 | 142/726 | 6 | 68 | 3 | 5 | 6 | 4 | 14 | A1 (56) |
| White microwave 1F-0 (kitchen_1f_0) | 8,130 | 1,710 | 7 | 6 | 5 | 298/840 | 6 | 28 | 9 | 15 | 18 | 8 | 22 | A1, living_1f_0 (27) |
| Yellow bistro table (courtyard_0) | 7,575 | 1,319 | 7 | 6 | 8 | 213/686 | 5 | 18 | 17 | 14 | 7 | 16 | 27 | A5, kitchen_gf_0 (14) |
| Red laptop 5 (bedroom_gf_0) | 5,726 | 1,267 | 7 | 6 | 5 | 153/814 | 4 | 65 | 5 | 4 | 5 | 9 | 12 | A1 (45) |
| Wooden console bench (living_gf_0) | 5,310 | 1,213 | 7 | 6 | 5 | 209/640 | 5 | 35 | 6 | 6 | 12 | 16 | 25 | A1 (35) |
| Fixed shelf laptop (living_gf_0) | 3,956 | 879 | 7 | 6 | 6 | 84/597 | 4 | 70 | 6 | 4 | 6 | 6 | 8 | A1, bedroom_gf_0 (54) |
| Track light (kitchen_gf_0) | 3,471 | 833 | 7 | 6 | 2 | 128/536 | 6 | 18 | 33 | 17 | 16 | 9 | 8 | A2, living_gf_0 (31) |
| Rack laptop 3 (bedroom_gf_0) | 3,003 | 735 | 7 | 6 | 4 | 72/532 | 4 | 57 | 8 | 7 | 8 | 7 | 13 | A1 (37) |
| Red fridge 2 (living_1f_0) | 2,791 | 679 | 7 | 6 | 3 | 101/447 | 4 | 22 | 9 | 8 | 14 | 19 | 28 | A6 (25) |
| Potted plant 3 (entryway_gf_0) | 2,665 | 843 | 7 | 6 | 5 | 123/607 | 4 | 31 | 11 | 11 | 18 | 14 | 15 | A1, corridor_gf_0 (16) |
| Black console laptop (living_gf_0) | 1,955 | 305 | 4 | 6 | 2 | 59/141 | 4 | 16 | 22 | 16 | 41 | 2 | 3 | A4 (41) |
| Sink counter cabinet 2 (kitchen_1f_0) | 1,884 | 806 | 7 | 6 | 4 | 115/546 | 2 | 24 | 20 | 3 | 10 | 10 | 32 | A6, living_1f_0 (23) |
| Rack laptop 6 (bedroom_gf_0) | 1,578 | 471 | 7 | 6 | 5 | 47/352 | 3 | 55 | 6 | 9 | 5 | 7 | 19 | A1 (28) |
| Washing machine (entryway_gf_0) | 1,067 | 371 | 7 | 6 | 4 | 42/288 | 5 | 27 | 14 | 11 | 16 | 15 | 16 | A1 (16) |
| Rack laptop 2 (bedroom_gf_0) | 1,028 | 293 | 7 | 6 | 4 | 6/245 | 2 | 76 | 5 | 5 | 9 | 3 | 1 | A1 (47) |
| All 34 instances | 795,319 | 90,261 | 7 | 6 | 6 | 15,425/34,103 | 6 | 20 | 18 | 14 | 17 | 16 | 15 | A4, living_gf_0 (11) |
Table S5.
Figure S4. Recurrence of representative targets across the shared week. a. Four human-verified movable instances. b. Registered landmark instances. Thumbnails are unedited source-frame crops around one accepted mask of each target, labeled with its wearer and day. Each day spans 09:30 to 23:30 in 30-minute bins. A bin’s shade gives the largest number of wearers who see the target label within one wall-clock minute of that bin. Dots give each wearer’s share of seen seconds, and the note under each timeline gives the most frequent wearer and wearer room. Room labels indicate wearer occupancy.
All 21 movable targets recur on at least two days. Nineteen are seen by more than one wearer, 14 by all six wearers, and 18 from more than one wearer room. All 34 landmark instances are seen by all six wearers, and 31 on all seven days.
Question construction and verification
Table S7 shows the question forms, and Figure S6 shows how experts check their answers against video evidence. This section details answer construction, visit-boundary conventions, temporal coverage, and the review operations.
| Q01 Object identity | What is the relationship between the main object in [reference A] and the main object in [reference B]? Answer: Same physical instance / same category but different instances / different categories. |
| Q04 Object history | Which option lists all and only the windows in [candidate windows] that contain the same physical object as in [object reference]? Answer: The complete subset of target-containing candidate windows. |
| Q02 Witness presence | During [observation interval], which of [wearer A] and [wearer B] recorded the same physical object as in [object reference]? Answer: Wearer A only / wearer B only / both / neither. |
| Q05 Spatial path | According to [supplied room graph], what is the shortest path from the room in [reference A] to the room in [reference B]? Answer: A room sequence that minimizes the number of graph edges between the identified endpoints. |
| Q03 Latest witness | Among [candidate wearers], who most recently recorded the same physical object as in [object reference], using the available recordings from [search start] up to [query time]? Answer: The wearer with the latest supported observation within the permitted scope. |
| Q06 Room identity | What is the relationship between the room in [reference A] and the room in [reference B]? Answer: Same physical room / same room type but different physical rooms / different room types. |
| Q07 Room-visit reasoning | Five subtypes |
| Q07A Ordered visits | What room-ID sequence was recorded for [wearer] during [observation interval]? Answer: The occupied room at the interval start followed by subsequent room changes, including returns to earlier rooms. |
| Q07B Current visit duration | At [query time], how long has [wearer] continuously occupied [target room] during the current stay? Answer: The duration option containing the elapsed time from actual entry to the query time. |
| Q07C Completed visit duration | During [observation interval], how long did [wearer] stay in [target room] on the [visit ordinal] visit? This visit has both its entry and exit within the interval. Answer: The duration option containing exit time minus entry time for the specified visit. |
| Q07D Preceding room | At [query time], [wearer] is in the current continuous stay in [target room]. Which room did the wearer occupy immediately before entering this stay? Answer: The physical room ID immediately preceding entry into the current stay. |
| Q07E Visit count | How many separate visits by [wearer] to [target room] are recorded during [observation interval]? Answer: The count option containing the number of distinct visits overlapping the interval. |
Table S7. Question templates for the seven spatial-memory abilities. Colored slots vary with the query’s public references, wearers, and time scope. Q07 expands into five visit-reasoning subtypes.

Figure S6. Two-pass QA review. The top row links model-assisted review, expert verification, and versioned decisions. a. Cropped question and answer view. b. Cropped multi-frame references and source video for identity verification. c. Redrawn room context and textual cues aligned with the selected interval. Timecodes use HH:MM:SS.FF at 20 FPS. The example compares kitchens observed by A4 on DAY2 and A3 on DAY4.
Question inventory and answer construction
The collection contains 8,453 questions across seven abilities, Q01 to Q07. Q07 comprises five visit-reasoning subtypes. Table S6 lists the training and evaluation partitions. Each question stores its source component, wearer scope, target, time interval, public references, answer, and required supporting evidence. Split assignment groups questions by their core-task identifier. We derive answers from reviewed identities, target observations, room intervals, and the supplied environment graph using four procedures.
| Component | Train | Val. | Test | Total |
| Week-long | 5,750 | 383 | 1,533 | 7,666 |
| Short-horizon | 0 | 0 | 787 | 787 |
| All sources | 5,750 | 383 | 2,320 | 8,453 |
Table S6. Question inventory. Unique questions by partition. All short-horizon questions are evaluation-only.
Identity comparisons: Q01 and Q06. We pair object or room references using reviewed instance identities and category labels. The paired identities determine whether the answer is the same physical instance, different instances of the same category, or different categories. Room comparisons use room type as the category and room ID as the identity.
Historical observations: Q02, Q03, and Q04. We gather target sightings within each question’s wearer scope and time interval. Q02 determines which of the two queried wearers recorded the target. Q03 compares sighting times across all permitted wearers before the query time to select the latest witness. Q04 labels every candidate window and selects the complete subset containing the target. An observation supports a positive label only when its reviewed identity matches the reference target. Missing recordings retain unknown visibility.
Spatial paths: Q05. Reference views identify the start and destination rooms. We compute shortest paths between their room IDs on the supplied environment graph to establish the answer. Thus the reference views determine the endpoints, and the graph determines the path between them.
Room visits: Q07A to Q07E. We derive ordered stays from the reviewed room intervals. Q07A reads the room sequence and retains returns. Q07B subtracts the current stay’s entry time from the query time. Q07C selects the requested visit ordinal and subtracts entry from exit, requiring both boundaries within the observation interval. Q07D retrieves the room immediately before the current stay. Q07E counts separate visits overlapping the interval. The visit-boundary conventions below handle stays crossing clip or query boundaries.
From annotations to question text. Each procedure fixes the answer, options, references, and evidence intervals before language-model rewriting. Semantic checks compare the rewritten question with these fixed fields. Experts then verify the wording, answer, and evidence in the QA Audit Workbench, as detailed in Appendix § 12.§ 6. Public question inputs and private answer records are exported separately.
Evaluation composition. The evaluated week-long subset contains 120 questions for each of 11 detailed types. Q07 comprises five types and contributes 600 questions to micro accuracy. The seven-ability macro average assigns Q07 the same weight as each other ability. This evaluation subset is drawn from the test partition in Table S6.
Question templates and answer forms
Table S7 gives the question and answer form for each subtype. Colored brackets identify the reference, wearer, room, and time parameters filled for each question. The response selects one answer option and supplies the evidence required by that question. For Q04, one option can encode several target-containing windows.
Visit boundaries and ordinals. A visit is uninterrupted occupancy in one physical room and continues across adjacent clip boundaries. For Q07C and Q07E, the visit list includes stays with positive-duration overlap with the observation interval. A stay already ongoing at the interval start counts once. Entry exactly at the interval end does not count. Q07C numbers this list before selecting the requested ordinal and requires the selected visit’s entry and exit to lie within the interval. Duration options use the bounds specified by each question.
Identifier mapping. Released identifiers use the internal codes Q1, Q2, Q3A, Q3B, and Q4 for Q01, Q02, Q03, Q04, and Q06. Q05 and Q07A through Q07E retain their identifiers.
Temporal and cross-wearer coverage
Table S8 measures how far the 120 evaluated Q04 questions reach across dates and wearers. Each question provides an object reference and four candidate windows. We select the target-containing windows using the correct option and read their wearers, dates, and intervals from the public window metadata. This yields 300 positive windows among 480 candidates. Window lengths range from 6 to 600 seconds, with a median of 120 seconds.
| Property | Questions |
| Candidate scope | |
| Candidates on multiple dates | 118 / 120 |
| Candidates from multiple wearers | 117 / 120 |
| Target-positive support | |
| At least one positive window older than 24 h | 103 / 120 |
| Positive windows on multiple dates | 78 / 120 |
| Positive windows from multiple wearers | 78 / 120 |
| Positive-window span longer than 24 h | 70 / 120 |
Table S8. Historical support in the 120 evaluated Q04 questions. Positive windows are those selected by the frozen correct answer. Cross-day and cross-wearer rows are separate counts.
For query time and window end , we measure window age as . Every sighting inside the window is therefore at least this old. As shown in Table S8, 103 questions contain at least one positive window lying entirely more than 24 hours before the query time. Across these questions, 198 positive windows satisfy this condition. The oldest positive-window age per question has a median of 78.9 hours and a maximum of 150.7 hours. Positive windows span multiple dates in 78 questions and multiple wearers in 78 questions, counted separately. The positive-window span runs from the earliest positive start to the latest positive end.
A cross-day example. Table S9 lists all four candidates for query EW010M_Q3B_e3b45c4031c16d89, issued at DAY7 18:00:06. The question asks which windows contain the referenced physical object. The correct option B selects W1, W2, and W3, linking A4 on DAY3, A3 on DAY5, and A6 on DAY1. W4 on DAY7 is a distractor. All three positive windows precede the query by more than 24 hours. Their temporal span exceeds four days.
Table S9. One Q04 query links three historical wearers and dates. The query time is DAY7 18:00:06.000. Each candidate spans ten minutes. Positive labels come from correct option B, which selects W1 + W2 + W3. Age is measured from each window’s end to the query time.
| Window | Wearer | Day | Time interval | Age, h | Same target |
| W1 | A4 | DAY3 | 20:04:31.297 to 20:14:31.297 | 93.76 | • |
| W2 | A3 | DAY5 | 19:11:33.909 to 19:21:33.909 | 46.64 | • |
| W3 | A6 | DAY1 | 13:55:56.799 to 14:05:56.799 | 147.90 | • |
| W4 | A6 | DAY7 | 14:23:37.783 to 14:33:37.783 | 3.44 | ◦ |
Question and answer vocabulary
Figure S5 summarizes the English vocabulary of all 8,453 questions and their correct answers across both benchmark components. Room, object, wearer, same, and physical dominate the questions, and room names
Figure S5. Question and answer vocabulary in EgoWitness. The panels show questions, correct answers, and their combined vocabulary for 7,666 week-long and 787 short-horizon questions. Font sizes reflect relative token frequencies within each panel. The corpus uses question text and correct-answer text only.
Scope. Candidate windows specify wearer and time. Q02 checks both wearers, Q03 compares all permitted wearers before the query time, and Q04 checks every candidate window. Missing recordings retain unknown visibility. Q05 computes paths on the supplied environment graph.
with same and different dominate the answers, matching the focus on instance identity, observers, and places. We select one question formulation per identifier and remove repeated paraphrases, capture metadata, and response-format instructions. Correct-answer text comes from the gold option, using fixed English translations for Chinese options.
We lowercase words, normalize common inflections, and remove stopwords, timestamps, and numeric identifiers. Room identifiers contribute their category words without floor or instance suffixes. This preprocessing yields lexical tokens from 6,884 answers. The remaining 1,569 answers contain only wearer or candidate-window codes and contribute no tokens to the answer panel. The combined panel sums the question and answer counts without source reweighting.
QA refinement and expert verification
Figure S6 shows two review passes: a model-assisted examination followed by expert verification in the QA Audit Workbench. All questions across the seven task types and both benchmark components have completed both passes. Experts establish the final wording, answer labels, and supporting evidence.
The first step below describes the model pass. Steps 2 to 5 describe the expert pass through the workbench’s question, evidence, and context views.
Step 1: Model-assisted preliminary review. GPT-6.1 Sol compares each question, its answer choices, the proposed answer, and the associated annotations. It flags unclear references, ambiguous wording, and annotation mismatches for expert inspection. These flags accompany each item into the expert pass.
Step 2: Question and answer refinement. In Figure S6, experts check the target, query time, permitted observation scope, and meaning of each answer choice. The bilingual display lets them compare Chinese and English formulations. They correct wording or labels when the question and intended answer express different spatial or temporal relations.
Step 3: Visual identity verification. Figure S6 pairs reference images with source-video playback. Experts compare physical identity across views, using enlarged references and frame-level navigation to inspect occlusion and similar instances. For object questions, they distinguish individual objects within a category. For room questions, they distinguish physical rooms within a room type. The illustrated example compares similar kitchens observed by A4 on DAY2 and A3 on DAY4.
Step 4: Temporal and witness verification. The context view in Figure S6 aligns room intervals with captions and transcripts to locate relevant video segments. Experts then inspect those segments in panel b, switching wearers and extending playback around event boundaries. For visit questions, they verify entry, exit, and visit order. For witness questions, they check the permitted wearers, dates, query time, and evidence intervals. Latest-witness verification includes checking for later sightings before the query time. Unobserved periods retain unknown visibility.
Step 5: Final labeling and revision tracking. Experts record approval, a revision request, or rejection in the question view of Figure S6. They save corrections to wording, answer choices, the correct answer, explanations, and evidence windows. The workbench stores these decisions and edits in a versioned history linked to the original records.
EgoClue training
Training setup
Table S10 summarizes the SFT and GRPO configurations. SFT uses 1,100 teacher dialogues expanded into 3,835 assistant-step samples. A GPT-5.1 Chat controller generated the dialogues with the same tools and backends, 20 decisions, and 19 tool calls. The 1,100 dialogues take 100 per detailed type without using model outputs. They are selected by format validity alone, without filtering by answer correctness. We update the language parameters while freezing the vision tower and aligner. Training uses ms-swift, BF16, and DeepSpeed ZeRO-3 on six GPUs, without packing, with a cosine learning-rate schedule and 5% warmup. The step-32 checkpoint at 0.1 epoch of a 64-step SFT run initializes GRPO, which uses the remaining 4,650 training questions and four rollouts per question. The resulting 18,600 trajectories produce 259 updates.
Table S10 Training configuration. The visual and retrieval services remain fixed across the Base, SFT, and GRPO controller comparison.
| Item | Value |
| Controller | Qwen3.5-4B |
| RL algorithm / rollout group | GRPO / 4 |
| Prompt batch | 18 |
| RL learning rate | |
| KL coefficient / clip | 0.001 / 0.2 |
| Training temperature / top- | 1.0 / 1.0 |
| SFT dialogues / assistant-step samples | 1,100 / 3,835 |
| SFT learning rate / global batch | / 12 |
| SFT checkpoint / | step 32 / epoch 0.1 |
| RL checkpoint / total updates | step 259 / 259 |
Table S10.
Training schedules. The main GRPO run comprises 259 updates from the 0.1-epoch SFT checkpoint, which was fixed before RL training began.
SFT/2 is the step-16 checkpoint of the same SFT run, at 0.05 epoch, and SFT/4 is a separate 8-step run of 0.025 epoch. Their RL continuations both use the 4,650 GRPO questions. The Direct-RL checkpoint is step 414.
Training curves. Main-paper Figure 5 plots the SFT+GRPO and Direct-RL updates. Each curve overlays raw scalars with an exponential moving average of coefficient 0.2. The KL curve measures divergence from the fixed reference policy.
GRPO objective
The outcome reward in main-paper Eq. (2) combines answer correctness, task-specific evidence credit, and episode-format validity. The private evaluator scores answers and supporting observations. Submission validation checks structure, citations, and the permitted evidence scope.
Group-relative advantage. We use Group Relative Policy Optimization [53] for the multi-turn controller. Tool observations condition subsequent decisions, as in tool-augmented egocentric reasoning [58]. For each question , the rollout policy generates trajectories. Let be the outcome reward of trajectory defined above. Its group mean , sample standard deviation , and advantage are
The same advantage weights every generated token in a trajectory. A group with identical rewards has zero advantage and remains in the batch.
Masked policy objective. Let be an optimization batch of trajectories. We omit the question index below. At position of trajectory , is the token and contains its prompt and preceding tokens, including tool observations. The binary mask is one for sampled assistant tokens and zero for tool observations, injected context, and padding. The token count is . For policy , write . Here denotes the trainable policy, old its rollout version, and ref the fixed SFT checkpoint at epoch 0.1. With clipping width , the probability ratio and clipped surrogate are
The implementation applies dual clipping with to negative advantages, giving when and otherwise. For the sampled KL penalty, define and . With coefficient , we minimize
This is a token-level mean over all generated assistant tokens in the batch. The KL penalty enters the loss, and the outcome reward excludes it. No entropy bonus is used.
Evaluation setup
Models and visual inputs
Table S11 identifies the twelve direct VLMs and the API controllers. The shared-tool comparison uses the public Qwen3.5-4B controller, its SFT checkpoint, its SFT+GRPO checkpoint, and four zero-shot API controllers. All seven conditions share the visual and retrieval services, public references, and execution limits.
Direct VLMs receive the question, public reference images, and 32 timestamped video frames selected before inference from public task metadata. Week-long object questions, Q01 to Q04, are posed in Chinese, and week-long room questions in English. Each question’s public time windows are intersected with the recordings of its listed wearers, the available media are concatenated in time order, and one frame is decoded at the midpoint of each of 32 equal segments. The windows are ±15 s around each reference capture for Q01, the stated 6-s interval of both wearers for Q02, the listed candidate windows for Q04, the stated observation interval for Q07A, Q07C, and Q07E, the 600 s before the query time for Q07B and Q07D, and the full permitted scope up to the query time for Q03, Q05, and Q06. Wearers share frames in proportion to their recorded footage. For Q03, uniform frames over a scope with a median of 112 h never come within five minutes of the latest sighting, so answering requires search, which motivates EgoClue. The selected timestamps and references are shared across models. Model-specific preprocessing records decoded timestamps, resolution, and encoded frame count. Gemini uses static video processing in this setting. Direct outputs contain answers, while agents choose additional observation intervals through their tools. Audio and transcript access follow the same condition across direct models.
Agent execution
EgoClue’s Memory, Video, Image, Topology, and Timeline interfaces map to search_memory, video_qa_updated, image_qa, query_topology, and reduce_timeline. The runtime also exposes validate_evidence for submission checks. The controller receives textual tool observations and no task-specific demonstrations. Native function calling and JSON adapters use the same argument schema and returned observations. Malformed calls receive an error, and retries consume the episode budget.
Memory retrieves coarse category information from captions, transcripts, and annotation categories. Its public interface omits instance identifiers, precise sightings, discriminative attributes, private answers, and room timelines. Image and Video compare the selected media with public references. Table S13 specifies the inputs, outputs, and implementation of each tool, and Table S12 gives the fixed perception and retrieval backends and per-episode limits.
| Attribute | Shared setting |
| Public inputs and shared services | |
| Wearers / query time | Question-defined |
| Visual references | Public instance and room catalogs |
| Private GT / demonstrations | Excluded / 0 |
| Controller observations | Textual tool outputs |
| Visual backend | Gemini 3.6 Flash |
| Retrieval backend | GPT-5.3 Chat |
| Per-episode execution limits | |
| Decisions / tool actions | 12 / 11 |
| Deadline / final reserve | 400 s / 60 s |
| Context / response tokens | 16,384 / 12,288 |
| Tokens per turn | 4,096 |
| Temperature / top-p | 0 / 1 |
| Request-attempt cap | 48 |
Table S12. Agent access and execution limits. Shared settings for Base, SFT, and SFT+GRPO on the week-long questions.
| Tool | Inputs | Returns | Implementation and access limits |
| Retrieval and visual inspection with fixed external models | |||
Memory search_memory | Text query, level of day, hour, or 10-min segment, optional wearer and time scope | Relevance note, selected records, and video locators for segment hits | No vector index is built. The text model GPT-5.3 Chat selects record IDs from batches of at most 30 prebuilt records, newest first, and IDs outside the batch are rejected. Records come from dense captions, transcripts, and coarse annotations. Instance IDs, attributes, boxes, exact sighting times, and answers are removed. |
Video video_qa_updated | Prompt and sources such as public references, wearer intervals, or subclips | Text observation, evidence IDs, and sampled time points | Gemini 3.6 Flash receives one silent MP4 per request at 1 FPS with source timestamps. Subclips of at most 40 s use 5 FPS, and object checks of at most 2 s use 20 FPS. Room IDs appear as permuted labels, and the room graph is withheld. Media outside the question scope or after the query time are rejected. |
Image image_qa | Prompt, reference images, and frames at given times | Text observation and evidence IDs | The same Gemini model compares target references and frames in one request of at most 50 images. Access limits match Video. |
| Deterministic spatial and temporal computation | |||
Topology query_topology | Start and end room IDs | All tied shortest paths, or none when unreachable | Breadth-first search on the unweighted room adjacency graph. It receives no visual input, so Video or Image must first identify the endpoint rooms. |
Timeline reduce_timeline | Start and end timestamps | Duration in seconds | Subtraction at millisecond resolution. Both times must lie within the question scope and before the query time. Video or Image must first locate the visit boundaries. |
Table S13. EgoClue tool implementations. Each tool returns text to the controller. Shaded bands separate tools that call fixed external models from tools that compute their results deterministically.
Response tokens include tool observations. Retries share the episode budget and remain subject to its tool-action limit.
Questions and score aggregation
The week-long comparison uses 1,320 questions in 1,154 core groups from the 1,533-question test partition. Each condition contributes one output per question. Q01 to Q06 each contain 120 questions, and Q07 contains 600. Micro accuracy weights questions equally. Macro accuracy first pools Q07A to Q07E, then averages the seven ability scores. Failed submissions and exhausted budgets remain in the denominator.
All direct-VLM and controller comparisons use this subset, and only the text baseline also reports the full 1,533-question partition.
The within-environment split holds out core tasks within the week-long residences. The cross-environment setting reserves all 787 questions and their environments for evaluation and uses the same execution limits, as Appendix § 16.§ 2 specifies. Appendix § 15.§ 6 reports source-identifier overlap and results on non-overlapping subsets.
Answer extraction. A shared deterministic normalizer reads only the model response and public options. It accepts an explicit choice letter, a leading choice label with answer text, a JSON choice field, or an exact unique option-text match. It removes closed reasoning blocks and accepts one complete answer block, including a final answer following discussion of other options. Responses without a unique final choice count as incorrect. The same rules apply to all 24 model-setting groups and 25,284 outputs, and open-weight models decode greedily. Table S20 reports extraction rates.
| Model | Week-long Parsed | Week-long | Cross-env. Parsed | Cross-env. |
| Qwen3.8-27B [50] | 92.80 | 46.97 | 99.75 | 58.07 |
| VideoChat-Flash-7B [32] | 100.00 | 41.82 | 100.00 | 38.37 |
| MiMo-VL-7B-RL [66] | 99.39 | 40.68 | 99.75 | 46.25 |
| VideoLLaMA3-7B [71] | 100.00 | 40.23 | 100.00 | 47.78 |
| InternVideo3-8B [67] | 100.00 | 39.85 | 100.00 | 44.09 |
| InternVL3.5-8B [59] | 100.00 | 39.70 | 100.00 | 47.90 |
| LongVILA-R1-7B [10] | 90.30 | 28.94 | 87.04 | 28.46 |
| Gemma 4 12B [20] | 69.47 | 28.71 | 88.56 | 49.81 |
| Qwen3.5-4B [49] | 82.05 | 28.11 | 96.06 | 41.93 |
| Gemini 4 Argon [24] | 99.92 | 73.56 | 100.00 | 86.28 |
| Claude Opus 5.5 [4] | 99.92 | 70.00 | 99.75 | 80.81 |
| GPT-6.1 Sol [44] | 100.00 | 58.79 | 100.00 | 67.60 |
Table S20. Direct-output normalization. Parsed is the percentage assigned a choice by the common normalizer. is normalized accuracy on all 1,320 week-long and 787 cross-environment questions, as reported in the main model comparison and Table S25.
Evidence scores. Semantic measures agreement with the fixed task-specific evidence matcher. Continuous joint correctness is , where is answer correctness. Strict success requires a correct answer and the task’s strict semantic match. This criterion is defined per task. All conditions use the same exact-citation normalization and semantic scorer. Unresolved evidence lies in , while unresolved and lie in . The reported bounds retain all applicable questions in the denominator.
The execution score additionally applies the runtime provenance gate to the original submissions. Its resolved mean uses the available judgments, and its bounds include every applicable question. Format validity checks the complete episode and is reported separately from semantic evidence scores.
Data access and privacy. The release contains annotations and their provenance. Source-video access follows the original providers’ terms. Wearer IDs identify camera streams, and witness answers describe supported camera observations. Because linked locations and wearers can reveal daily routines, public examples and media permissions require privacy review separately from annotation acceptance. Run records associate results with question IDs, checkpoints, service versions, and execution settings.
Additional results
Uncertainty of the main comparison
Table S14 gives 95% intervals for the week-long accuracies in main-paper Table 3, from 5,000 resamples of the 1,154 core groups. Paired over the same resamples, EgoClue + Sol exceeds Claude Opus 5.5 by 4.85 points, with interval [1.55, 8.08], and differs from Gemini 4 Argon by 1.29 points, with interval [−1.88, 4.46].
Post-training results
Table S15 compares Base, SFT, and SFT+GRPO on the same questions. GRPO improves answer accuracy over SFT on every detailed type. The largest gains occur in completed-visit duration Q07C, last-observer attribution Q03, and historical-window verification Q04. Table S16 and Figure S7 compare additional SFT initializations under the training schedules in Appendix § 13.§ 1. RL raises every SFT initialization, by 12.80, 6.81, and 14.09 points for SFT/4, SFT/2, and SFT, and all three end above the 40.68% of Direct RL. The 0.1-epoch initialization leads with 72.65%. Figure S8 compares format validity with answer accuracy. Direct RL reaches 99.24% validity but 40.68% accuracy, so valid output format explains little of the SFT+GRPO gain.
| Type | Base | SFT | +GRPO | Gain Δ pp |
| Q01 | 43.33 | 64.17 | 71.67 | +7.50 |
| Q02 | 49.17 | 61.67 | 68.33 | +6.67 |
| Q03 | 3.33 | 15.83 | 38.33 | +22.50 |
| Q04 | 20.00 | 32.50 | 53.33 | +20.83 |
| Q05 | 18.33 | 55.00 | 73.33 | +18.33 |
| Q06 | 6.67 | 80.00 | 87.50 | +7.50 |
| Q07 visit-reasoning subtypes | ||||
| Q07A | 38.33 | 78.33 | 85.83 | +7.50 |
| Q07B | 12.50 | 85.00 | 90.00 | +5.00 |
| Q07C | 21.67 | 49.17 | 74.17 | +25.00 |
| Q07D | 30.83 | 76.67 | 91.67 | +15.00 |
| Q07E | 15.83 | 45.83 | 65.00 | +19.17 |
| Micro | 23.64 | 58.56 | 72.65 | +14.09 |
| Macro7 | 23.52 | 53.74 | 67.69 | +13.95 |
Table S15. Table S15 EgoClue by ability and visit subtype. Answer accuracy on the same 1,320 questions, 120 per detailed type. Base is the public 4B controller. GRPO starts from SFT and uses step 259. Δ is GRPO minus SFT, in percentage points. Macro7 pools Q07 subtypes before averaging. Answer accuracy (%) ↑ Gain
| Controller | Correct | Micro | Macro | Format | |
| Without SFT | |||||
| Base Qwen3.5-4B [49] | 312 | 23.64 | 23.52 | 25.76 | 0.099 |
| Direct RL, step 414 | 537 | 40.68 | 39.55 | 99.24 | 0.123 |
| SFT initializations, before and after RL | |||||
| SFT/4 | 686 | 51.97 | 52.14 | 81.52 | 0.414 |
| SFT/4 + RL | 855 | 64.77 | 62.74 | 99.09 | 0.456 |
| SFT/2 | 800 | 60.61 | 56.95 | 92.35 | 0.461 |
| SFT/2 + RL | 890 | 67.42 | 65.10 | 99.85 | 0.465 |
| SFT | 773 | 58.56 | 53.74 | 93.71 | 0.457 |
| SFT + RL, step 259 | 959 | 72.65 | 67.69 | 97.88 | 0.482 |
Table S16. Table S16 Post-training configurations on 1,320 questions. Micro, Macro, and Format are percentages. is the resolved runtime-grounded evidence mean. SFT/2 and SFT/4 stop SFT at 0.05 and 0.025 epoch. Shading marks EgoClue-SFT+GRPO (step 259), and bold marks the best value in each column. Metric definitions appear in Appendix § 14.§ 3.
Figure S8. Figure S8 Execution quality and answers. Format validity and answer accuracy on 1,320 questions.
Figure S7. Post-training across initializations and abilities. Left compares micro accuracy before and after RL at three SFT initializations under their recorded training schedules. Right gives the SFT and step-259 SFT+RL comparison by ability.
Best accuracy in each row is bold. Appendix § 15.§ 7 reports per-type semantic joint scores.
Tool ablations
Figure S9 and Table S17 compare individual removals of Image, Memory, Topology, Timeline, and Video with the full agent. The controller, visual backend, question IDs, and execution budget remain fixed. Without Video, the controller cannot call the Video tool, while Image keeps reference comparison and frame extraction at specified times. Without Image, independent image queries, stand-alone frame extraction, and cached independent frames are disabled. Video retains public references and can inspect them for reference-only tasks. Without the Topology query, the supplied graph and transition checks remain available.
Paired tool-removal outcomes. Full-invocation subsets are defined from full-agent execution traces. Task-applicable subsets use question types independently of observed calls. Full and Removed give correct-answer counts. The final column is Removed minus Full accuracy in percentage points.
| Removed tool | Subset | Questions | Full | Removed | Δ pp |
| Image | All questions | 1,320 | 959 | 907 | −3.94 |
| Memory | All questions | 1,320 | 959 | 900 | −4.47 |
| Topology | All questions | 1,320 | 959 | 881 | −5.91 |
| Timeline | All questions | 1,320 | 959 | 845 | −8.64 |
| Video | All questions | 1,320 | 959 | 620 | −25.68 |
| Topology | Q05 | 120 | 88 | 65 | −19.17 |
| Timeline | Q07B / Q07C | 240 | 197 | 140 | −23.75 |
| Image | Full-invocation | 458 | 322 | 312 | −2.18 |
| Memory | Full-invocation | 111 | 44 | 31 | −11.71 |
| Topology | Full-invocation | 107 | 83 | 56 | −25.23 |
| Timeline | Full-invocation | 178 | 151 | 103 | −26.97 |
| Video | Full-invocation | 971 | 688 | 369 | −32.85 |
Table S17.
Figure S9. Tool removal on all questions and invocation-selected subsets. The left panel uses the same 1,320 questions for each removal. The right panel uses a different Full-invocation subset for each tool. Subsets can overlap and include unsuccessful calls.
We report all questions, task-defined subsets, and subsets
selected by Full’s tool calls. Task-defined subsets are Q05 for Topology and Q07B/Q07C for Timeline. Invocation subsets include failed calls and can overlap.
On the task-defined subsets, removing Topology changes accuracy from 73.33% to 54.17%, and removing Timeline changes it from 82.08% to 58.33%. Across all questions, 5,000 resamples of the 1,154 core groups give 95% percentile intervals of [−6.33, −1.49], [−7.13, −1.83], [−8.49, −3.29], [−11.09, −6.08], and [−28.86, −22.34] percentage points for Image, Memory, Topology, Timeline, and Video removal, respectively. All five intervals exclude zero.
Inference cost
Table S19 shows that GRPO lowers tool calls, external-model calls, policy tokens, and episode latency relative to SFT. Mean tool calls fall from 2.67 to 2.31, and mean latency from 175.71 to 151.59 seconds.
| Metric | SFT | +GRPO | Δ |
| Calls and tokens per QA: relative change, % | |||
| Tool calls | 2.67 | 2.31 | −13.37 |
| External model calls | 1.90 | 1.59 | −16.19 |
| Policy tokens | 635.5 | 496.2 | −21.92 |
| Observation tokens | 3,524.5 | 3,459.5 | −1.84 |
| Episode latency, seconds: relative change, % | |||
| Mean | 175.71 | 151.59 | −13.73 |
| Median | 140.44 | 118.78 | −15.43 |
| 95th percentile | 360.36 | 352.63 | −2.15 |
Table S19. Recorded inference resources. SFT and GRPO use the same 1,320 questions. Changes are relative to SFT. Latency includes service queueing, and controller tokens exclude visual-backend inputs.
Table S18 compares EgoClue with the prompted API controllers of main-paper Table 3. EgoClue needs the fewest turns and proprietary-model tokens. Its tool backends use about as many tokens as those of GPT-5.3 Chat, so the saving comes from running the controller locally. Figure S10 compares API fees at comparable accuracy. EgoClue + Sol reaches the highest accuracy, 74.85%, for $90, below the $110 and $115 of direct Gemini 4 Argon and Claude Opus 5.5, and the 4B controller alone costs $67 at 72.65%.

Figure S10. API cost at comparable accuracy. Fees for one pass over the 1,320 questions for all systems above 65% in main-paper Table 3, from recorded tokens at November 2026 list prices. Gemini thinking is billed as output, and GPT-5.3 Chat uses the gpt-5.3 price. EgoClue fees exclude local 4B inference.
| Controller | Micro, % | Turns | Tool calls | Controller tokens | Tool-backend tokens | Proprietary total | Proprietary per correct |
| EgoClue-SFT+GRPO, 4B | 72.65 | 3.30 | 2.31 | 18.3k{sup}†` | 38.1k | 38.1k | 52.4k |
| GPT-5.3 Chat [43] | 65.76 | 4.34 | 2.55 | 25.0k | 38.3k | 63.3k | 96.3k |
| Gemini 4 Argon [24] | 46.14 | 4.83 | 2.78 | 32.1k | 28.9k | 61.0k | 132.2k |
| Gemini 3.5 Flash-Lite [22] | 37.58 | 7.06 | 1.15 | 32.6k | 11.7k | 44.3k | 117.8k |
| Claude Haiku 5.5 [3] | 32.20 | 8.07 | 2.22 | 69.4k | 27.9k | 97.4k | 302.4k |
Table S18. Per-question cost of shared-tool controllers. All controllers answer the same 1,320 week-long questions with the same tools, backends, and budget. Turns count controller generations. Token counts include Gemini thinking. †Local 4B controller, excluded from proprietary totals. Bold marks the best accuracy, turns, controller tokens, and totals.
Per-question inference totals exclude one-time memory construction and optional synthesis.
Nonvisual baselines and training overlap
Uniform guessing has expected correctness for a question with options. The week-long set contains 240 three-option and 1,080 four-option questions, yielding 26.52% Micro and 27.38% Macro accuracy. The cross-environment set contains 200 three-option and 587 four-option questions, yielding 27.12% expected Micro accuracy.
Answer extraction. Table S20 separates response formatting from answer accuracy using the normalizer in Appendix § 14.§ 3. Parsed records successful choice extraction, and all questions contribute to accuracy.
Public-text prediction. We fit one TF-IDF ridge classifier per detailed task type using the public English question, option text, and type. Word unigrams and bigrams use training-only vocabulary and IDF statistics, a minimum document frequency of two, and at most 30,000 features per type. Features use sublinear term frequency, smoothed IDF, and row-wise normalization. One-versus-rest ridge regression uses targets in and a regularized intercept.
We fit on 5,750 training questions and select a regularization coefficient from by seven-ability macro accuracy on 383 validation questions. Ties favor validation micro accuracy, then the larger coefficient. The selected value is 1, giving validation micro and macro accuracy of 30.03% and 29.03%. The training-only fit is retained for testing. Table S21 compares it with the most frequent training answer for each detailed type.
| Ability | Questions | Majority | Ridge |
| Q01 | 120 | 30.83 | 42.50 |
| Q02 | 120 | 18.33 | 28.33 |
| Q03 | 120 | 22.50 | 27.50 |
| Q04 | 120 | 25.00 | 27.50 |
| Q05 | 120 | 25.00 | 25.00 |
| Q06 | 120 | 35.00 | 43.33 |
| Q07 | 600 | 25.00 | 30.67 |
| Main set, Micro | 1,320 | 25.61 | 31.59 |
| Main set, Macro | 1,320 | 25.95 | 32.12 |
| Full test, Micro | 1,533 | 32.35 | |
| Full test, Macro | 1,533 | 32.82 |
Table S21. Nonvisual baselines on the week-long setting. Majority selects the most frequent training answer per detailed type. Ridge uses public text and validation-selected regularization. Q07 pools five subtypes. These scores use the same question sets as the main model comparison.
The classifier answers 417/1,320 questions correctly, compared with 338/1,320 for majority prediction. On the 787 cross-environment questions, the same classifier scores 27.06%, close to the 27.12% expected from uniform guessing.
Permuting the options of each test question 20 times lowers the week-long Micro accuracy of the fitted classifier to 27.87 1.74%, close to uniform guessing. Q02, whose windows all last 6 s and whose options appear in all 24 orders, stays at 27.54 3.98%. Q01 drops from 42.50% to 34.75 4.22% because some option orders co-occur with the answer position, while Gemini 4 Argon and Claude Opus 5.5 differ by at most 2.3 points between questions with and without these orders.
Training-overlap sensitivity. Table S22 counts shared source identifiers between training and test data. Core groups and exact public inputs are disjoint, while references and source events can recur. These identifiers describe source-record reuse. An answer fact also depends on the target, queried relation, wearer scope, and time interval.
| Identifier type | Shared atoms | Affected questions |
| Core group | 0 | 0 |
| Exact public input | 0 | 0 |
| Object episode | 503 | 364 |
| Object event | 209 | 445 |
| Reference | 566 | 562 |
| Room anchor | 229 | 227 |
| Room boundary | 2 | 2 |
| Room dwell | 2,335 | 576 |
| Room transition | 1,487 | 499 |
Table S22. Table S22 Train/test identifier overlap. Recomputed from assignment atoms. Shared atoms are distinct identifiers. Question counts refer to the evaluated 1,320 and may overlap across rows.
Table S23 retains questions without training-shared identifiers and selects these subsets from identifier manifests alone. Reference filtering applies to questions with reference identifiers, object-event filtering to object tasks, and room-event filtering to room tasks. The checkpoints and question inputs stay fixed.
| Unshared identifiers | SFT | GRPO | ||
| Reference | 114 | 35.09 | 56.14 | +21.05 |
| Object event or episode | 32 | 31.25 | 65.63 | +34.38 |
| Room event | 81 | 76.54 | 88.89 | +12.35 |
| All recorded atoms | 113 | 63.72 | 82.30 | +18.58 |
Table S23. Table S23 Training-overlap sensitivity. Each row retains questions with no training-shared atom of the indicated kind, within its applicable domain. Checkpoints and inputs are unchanged. Values are micro accuracy and the GRPO gain in percentage points. Rows can overlap.
The final row checks all recorded identifier types except the already disjoint exact-public-input hash. It contains 113 questions in 93 core groups. SFT answers 72 correctly and GRPO answers 93. Two thousand paired core-group resamples give a 95% gain interval of [6.14, 30.83] percentage points. Public-text ridge, direct Gemini, and GRPO with synthesis score 36.28%, 74.34%, and 87.61% on the same subset.
Q01 to Q06 contribute 4, 4, 18, 6, 4, and 9 questions. Q07A to Q07E contribute 14, 21, 1, 31, and 1.
Answer and evidence agreement
Table S24 evaluates SFT, GRPO, and synthesis on the same 1,021 evidence-applicable questions using the semantic scores defined in Appendix § 14.§ 3. The aggregate bounds are [46.73, 49.18]%, [51.56, 54.89]%, and [51.61, 55.63]%. GRPO’s aggregate lower bound exceeds the SFT upper bound.
Latest-witness evidence. Q03 requires the observing wearer and the latest supporting sighting. The common citation normalizer restores a submission only when its role, wearer, date, and exact time window uniquely match evidence returned by a successful tool call in that trajectory. It preserves the predicted answer, window, and observation. Ambiguous matches remain unresolved.
Visual evidence must establish both target identity and temporal recency. A visible earlier sighting establishes presence but does not establish the latest sighting. Unresolved visibility or recency judgments retain the bounds in Table S24. Missing recordings remain unknown, and coverage checks range over the permitted wearers up to the query time.
Answer synthesis
Synthesis inputs
EgoClue + Sol passes the controller’s tool observations and preceding decisions to GPT-6.1 Sol after removing the terminal answer turn. Sol produces the final choice and evidence references without new visual calls or additional tools. It receives no private answers or scores. Decoding uses a 4,096-token output limit. The instruction requests chronological event consolidation, consistent visit counting, and evidence-role alignment, with unknown intervals distinguished from absence.
For episodes without a submitted answer, synthesis reads the recorded trajectory in the same way.
Cross-environment setup
All 787 cross-environment questions and their environments are excluded from SFT and GRPO. All tool systems use the week-long execution budget in Table S12. Gemini 3.6 Flash remains the visual backend. Image, Video, Timeline, and submission validation are available. There is no prebuilt Memory or supplied Topology graph.
Table S25 compares the twelve direct VLMs and the prompted shared-tool controllers with this week-long-trained system before and after synthesis. Without training in these environments, EgoClue-SFT+GRPO exceeds every prompted controller and every open-weight direct VLM, 78.40% against at most 71.54% and 58.07%. GRPO raises EgoClue from 68.11% after SFT to 78.40%, with Q04 rising from 43.20% to 85.21%.
| Model or system | Q01 | Q04 | Q06 | Q07 | Macro | Micro |
| Direct visual reasoning, ordered by Micro | ||||||
| Gemini 4 Argon [24] | 90.00 | 95.27 | 83.75 | 82.06 | 87.77 | 86.28 |
| Claude Opus 5.5 [4] | 77.50 | 94.08 | 80.00 | 76.56 | 82.03 | 80.81 |
| GPT-6.1 Sol [44] | 70.83 | 73.96 | 56.25 | 66.27 | 66.83 | 67.60 |
| Qwen3.8-27B [50] | 53.33 | 76.33 | 62.50 | 51.20 | 60.84 | 58.07 |
| Gemma 4 12B [20] | 49.17 | 55.62 | 53.75 | 46.89 | 51.36 | 49.81 |
| InternVL3.5-8B [59] | 57.50 | 71.01 | 52.50 | 34.93 | 53.98 | 47.90 |
| VideoLLaMA3-7B [71] | 65.00 | 43.79 | 48.75 | 44.26 | 50.45 | 47.78 |
| MiMo-VL-7B-RL [66] | 30.00 | 86.39 | 42.50 | 35.41 | 48.57 | 46.25 |
| InternVideo3-8B [67] | 49.17 | 65.68 | 37.50 | 35.17 | 46.88 | 44.09 |
| Qwen3.5-4B [49] | 25.83 | 57.40 | 53.75 | 38.04 | 43.75 | 41.93 |
| VideoChat-Flash-7B [32] | 50.00 | 50.89 | 21.25 | 33.25 | 38.85 | 38.37 |
| LongVILA-R1-7B [10] | 25.83 | 38.46 | 42.50 | 22.49 | 32.32 | 28.46 |
| Prompted shared-tool controllers, ordered by Micro | ||||||
| Gemini 4 Argon [24] | 69.17 | 72.19 | 86.25 | 69.14 | 74.19 | 71.54 |
| GPT-5.3 Chat [43] | 70.00 | 76.92 | 65.00 | 64.11 | 69.01 | 67.85 |
| Gemini 3.5 Flash-Lite [22] | 60.00 | 59.17 | 61.25 | 46.17 | 56.65 | 52.60 |
| Claude Haiku 5.5 [3] | 57.50 | 52.07 | 48.75 | 46.41 | 51.18 | 49.56 |
| Qwen3.5-4B [49] (base) | 22.50 | 15.38 | 21.25 | 25.12 | 21.06 | 22.24 |
| Week-long-trained systems, target environments excluded from post-training | ||||||
| EgoClue-SFT | 86.67 | 43.20 | 81.25 | 70.33 | 70.36 | 68.11 |
| EgoClue-SFT+GRPO | 87.50 | 85.21 | 83.75 | 72.01 | 82.12 | 78.40 |
| EgoClue + Sol [44] | 84.17 | 88.17 | 81.25 | 76.32 | 82.47 | 80.56 |
Table S25. Table S25 Cross-environment model and system results. All 787 questions are evaluation-only. Q01, Q04, Q06, and Q07 contain 120, 169, 80, and 418 questions. Macro weights these four abilities equally. Micro weights all questions equally. Direct models answer from their frame inputs, and all tool systems use the week-long execution budget. All direct rows use the same normalizer. Bold marks the best Micro within each group.
Answer and evidence gains
Answer and evidence gains
Table S26 and Figure S11 compare answer accuracy before and after synthesis. On the week-long set, 5,000 paired resamples of 1,154 core groups give a 95% gain interval of [0.59, 3.82] percentage points.
| Evaluation scope | Questions | 4B, % | + Sol, % | Δ, pp | Corrected | Regressed |
| Week-long | 1,320 | 72.65 | 74.85 | +2.20 | 75 | 46 |
| Cross-environment | 787 | 78.40 | 80.56 | +2.16 | 41 | 24 |
Table S26. Answer synthesis. Corrected and regressed count paired answer changes.
Figure S11. Evidence-only synthesis. (a) Answer accuracy. (b) Incorrect-to-correct and correct-to-incorrect answer changes. (c) Accuracy change by week-long ability. (d) Semantic evidence , strict success , and evidence-submission validity on the same 498 cross-environment questions. Accuracy comparisons use 1,320 week-long and 787 cross-environment questions under their respective controller settings. Orange denotes the controller and red denotes synthesis. Gains in (a), (c), and (d) are percentage points.
For the 1,021 week-long evidence-applicable questions, semantic bounds are [51.56, 54.89]% before and [51.61, 55.63]% after synthesis. Table S24 gives the per-type joint results.
On the same 498 cross-environment evidence-applicable questions, semantic stays at 35.6%, while joint rises from 31.65% to 32.74% and strict success from 20.08% to 20.88%. Evidence-submission validity increases from 325/498 to 371/498, or 65.26% to 74.50%. This validity check concerns submission structure and required fields, while the semantic scores assess the evidence match.
We pair question IDs and bootstrap source videos 3,000 times, retaining all questions from each sampled video. Answer evaluation spans 40 videos, and evidence evaluation spans 37. Table S27 reports 95% percentile intervals. The answer and joint evidence gains exclude zero.
| Metric | Questions | Δ | 95% CI |
| Answer | 787 | +2.16 | [0.11, 4.65] |
| Semantic | 498 | −0.04 | [−0.93, 0.86] |
| Joint | 498 | +1.09 | [0.42, 1.81] |
| Strict | 498 | +0.80 | [0.00, 1.80] |
Table S27. Video-cluster uncertainty for synthesis. Changes are percentage points relative to the 4B controller. CI denotes the 95% paired video-bootstrap interval. Evidence metrics use their applicable subset, with each metric’s denominator listed below.
Synthesis cost
The 1,320 week-long synthesis requests use 10,239,769 prompt tokens and 252,425 completion tokens. The 787 cross-environment requests use 8,413,064 and 90,231, respectively. Cross-environment synthesis adds a mean 4.95 seconds per question, with median 4.30 and 95th percentile 7.54 seconds. These measurements cover the additional synthesis call.
Case studies and error analysis
This section examines how models fail on the seven abilities. The eleven detailed types form two families: object-instance memory (Q01–Q04) and room and visit memory (Q05, Q06, and Q07A–Q07E). Every example reports the same models: the three proprietary frontier VLMs of the main results table under direct access, and EgoClue-SFT+GRPO with its 4B controller. Object questions were posed in Chinese, and the figures show English renderings.
Representative failure cases
Figure S12 gives one example for each of six abilities, grouped by the three error types of the question design: identity recognition, historical retrieval, and spatial-temporal reasoning.
Figure S12. Representative failure cases. Six abilities grouped by error type. Chips report the three frontier VLMs and EgoClue (green when correct), and the red line repeats Gemini 4 Argon’s option. Green and red frame outlines mark gold evidence and the frame behind a wrong answer; yellow boxes are released masks at the same timestamp. All timelines share the legend below: room bands come from reviewed room intervals, and Q03 ticks are released screen masks in 1-min bins. Q06 references show one of five views.
Type I: Identity recognition. These errors confuse physical identity with appearance.
Q01: Are the main objects in Image A and Image B the same physical instance? All three frontier VLMs answer “same physical instance”; the ground truth is “same category, different instances”. Both images show a plain white projection screen, but one hangs in the ground-floor living room and the other in the first-floor living room.
Q06: Do Image A and Image B show the same physical room? All three answer “same type, different rooms”; both views show
living_1f_0. The views share no visible surface, so the match must come from the room’s fixed layout.
Type II: Historical retrieval. These errors stop at an earlier sighting or skip a short one.
Q03: As of DAY7 11:16:01, who most recently recorded the projection screen? Gemini 4 Argon and GPT-6.1 Sol answer A4, who last recorded it on DAY6 evening; Claude Opus 5.5 and the ground truth give A6, who recorded it again at DAY7 11:03:33.
Q04: Which of four windows contain the lamp? All three omit W3. That window is mostly kitchen and living-room activity, and the wearer enters the lamp’s bedroom only once, for 89 s.
Type III: Spatial-temporal reasoning. These errors lose room boundaries or graph structure.
Q05: What is the shortest path between the rooms at two references? All three route through
living_gf_0, which has no edge to either first-floor endpoint; the endpoints are adjacent.Q07E: How many separate visits did A2 make to the bedroom? All three answer one visit; the reviewed timeline contains three, separated by trips of 11.5 s and 26.4 s.
Insights and implications
Table S28 counts error directions of the three frontier VLMs on all 120 questions per type. Three gaps follow.
Identity without place. Q01 errors split among category confusions (44%), merging two instances (36%), and splitting one instance (21%). These errors are consistent with appearance matching that ignores instance location.
Retrieval stops early. Q03 has the highest frontier error, 69.4%. In Q04, 56% of errors omit a positive window.
Short exits split visits. In Q07E, 77% of errors undercount visits, and in Q07B, 68% overestimate the current stay. The exits separating Q07E visits last a median of 92.5 s, and 31% last at most 30 s.
Directions. The gaps suggest three requirements: verifying instance identity together with location, retrieving over every candidate window and the interval before the query time, and computing durations and counts from explicit room boundaries. EgoClue addresses these with its Image, Memory and Video, and Timeline tools. Its accuracy is highest on boundary types, 90.0% on Q07B and 91.7% on Q07D. On the hardest type, Q03, it reaches 38.3%, compared with 34.2% for the best frontier VLM.
Qualitative analysis
Figure S13 and Figure S15 annotate one question per type with references, evidence frames, answers, and model chips.

Figure S13. Evidence-annotated examples: object-instance memory. Evidence windows come from the gold records. Yellow boxes are released masks at the shown timestamp. Frame labels give wearer, day, time, and the wearer’s reviewed room. In Q04, W1 and W2 frames illustrate the negative windows. Chips report the three frontier VLMs and EgoClue (green when correct).
Figure S15. Evidence-annotated examples: room and visit memory. Timelines show reviewed room intervals over each question’s interval, with the shared legend; the outlined segments are the queried room. Q05 and Q06 references show one of five views, and the Q05 graph highlights the shortest path on the supplied room graph. Q07E enlarges the first 30 s of its interval, where the first visit lasts 7.8 s. Chips report the three frontier VLMs and EgoClue.
Figure S14 contrasts the frontier VLMs with EgoClue on two questions. In Q02, one Video call matches the board’s handwriting to the reference and separates the wearer who writes on the board from the one facing the dining table. In Q07B, Video finds a 6-s kitchen visit and Timeline measures the stay from re-entry.

Figure S14. Frontier VLMs versus EgoClue. Both questions are wrong for all three frontier VLMs and correct for EgoClue-SFT+GRPO. Over all questions, the three frontier VLMs are all wrong on 229 answered questions, and EgoClue answers 116 of them. Tool calls and results are condensed, and Chinese controller text is translated.
References
- [1]Samiul Alam, Shakhrul Iman Siam, Michael J. Proulx, James Fort, Richard Newcombe, Hyo Jin Kim, and Mi Zhang. SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory. arXiv preprint arXiv:2606.00825, 2026.arxiv.org/abs/2606.00825
- [2]Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.arxiv.org/abs/1711.07280
- [3]Anthropic. Claude Haiku 5.5. Claude Platform documentation, 2026. Accessed November 8, 2026.
- [4]Anthropic. Claude Opus 5.5. Claude Platform documentation, 2026. Accessed October 25, 2026.
- [5]Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In International Conference on Learning Representations, pages 9112–9141, 2024.arxiv.org/abs/2310.11511
- [6]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Trifantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhilia Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. SAM 3: Segment Anything with Concepts. In International Conference on Learning Representations, 2026.arxiv.org/abs/2511.16719
- [7]Jacob Chalk, Saptarshi Sinha, Dima Damen, Yannis Kalantidis, and Diane Larlus. Whereformer: Learning to Track What is Where in Long Egocentric Videos. In European Conference on Computer Vision, 2026.arxiv.org/abs/2607.08537
- [8]Keshigeyan Chandrasengaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. HourVideo: 1-Hour Video-Language Understanding. In Advances in Neural Information Processing Systems, 2024.
- [9]Weitao Chen, Hu Jiaxin, Xie Tianyidan, Yang Li, Yuyi Qian, Banghao Xu, Ziheng Tang, Sheny i Wang, Mingyue Yu, Duo Li, Jiacheng Shi, Gao Wang, Zhan Xu, Zhicheng Qiu, Xuanfu Li, Jian Yang, Lanjun Wang, and Zili Yi. EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory. arXiv preprint arXiv:2603.13113, 2026.
- [10]Yuchen Chen, Wei Xue, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojun Qi, et al. Scaling RL to Long Videos. In Advances in Neural Information Processing Systems, 2025.
- [11]Ho Kei Cheng and Alexander G. Schwing. XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model. In European Conference on Computer Vision, 2022.arxiv.org/abs/2207.07115
- [12]Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking Anything with Decoupled Video Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2309.03903
- [13]Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the Object Back into Video Object Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3151–3161, 2024.arxiv.org/abs/2310.12982
- [14]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720–736, 2018.arxiv.org/abs/1804.02748
- [15]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130:33–55, 2022.DOI
- [16]Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–10, 2018.
- [17]Pablo Diaz-Pereda, Alejandro Rodriguez-Ramos, David Perez-Saura, and Pascual Campoy. REMIND: RE-Identification with Memory for INDoor Navigation. arXiv preprint arXiv:2607.09267, 2026.
- [18]Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding. In Proceedings of the European Conference on Computer Vision, 2024.arxiv.org/abs/2403.11481
- [19]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24108–24118, 2025.arxiv.org/abs/2405.21075
- [20]Gemma Team. Gemma 4 Technical Report. arXiv preprint arXiv:2607.02770, 2026.
- [21]Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. AMEGO: Active Memory from long EGocentric videos. In European Conference on Computer Vision, 2024.
- [22]Google DeepMind. Gemini 3.5 Flash-Lite. Gemini API documentation, 2026. Accessed November 8, 2026.
- [23]Google DeepMind. Gemini 3.8 Flash Model Card, 2026. Accessed October 24, 2026.
- [24]Google DeepMind. Gemini 4 Argon Model Card, 2026. Accessed October 24, 2026.
- [25]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jacky Hambuger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abraham Gebreselassie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolář, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Rujie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyı Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.arxiv.org/abs/2110.07058
- [26]Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abraham Gebreselassie, Sanjay Haresh, Jing Huang, Md Mohaminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatuminwı Oguntola, Xiaoqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Southerland, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Gedas Bertasius, Dima Damen, Jakob Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C.V. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray. Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024.
- [27]Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang, Limin Wang, and Yu Qiao. EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exocentric View of Procedural Activities in Real World. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22072–22086, 2024.arxiv.org/abs/2403.16182
- [28]Shunya Kato, Taiki Miyanishi, Shuhei Kurita, Mahiro Ukai, Nakamasa Inoue, and Chenhui Chu. LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension. In European Conference on Computer Vision, 2026.arxiv.org/abs/2607.02096
- [29]Kangsan Kim, Yanlai Yang, Suji Kim, Woongyeong Yeo, Youngwan Lee, Mengye Ren, and Sung Ju Hwang. MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents. arXiv preprint arXiv:2603.09827, 2026.arxiv.org/abs/2603.09827
- [30]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2304.02643
- [31]Shuhei Kurita, Naoki Katsura, and Eri Onami. RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.arxiv.org/abs/2308.12035
- [32]Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling. In International Conference on Learning Representations, 2026.arxiv.org/abs/2501.00574
- [33]Shuo Liang, Yiwu Zhong, Zi-Yuan Hu, Yeyao Tao, and Liwei Wang. Fine-Grained Spatiotemporal Grounding on Egocentric Videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9385–9395, 2025.arxiv.org/abs/2508.00518
- [34]Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z. Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric Video-Language Pretraining. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2206.01670
- [35]Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2411.13093
- [36]Fangzhou Ma, Ivo Alexander Ban, Eren Homburg, Gabriele Goletto, Rémi Pautrat, Mahdi Rad, Chiara Plizzari, and Marc Pollefeys. Long Time No See: Benchmarking VLMs for Out-of-Sight Spatiotemporal Reasoning in Egocentric Videos. arXiv preprint arXiv:2609.34630, 2026.
- [37]Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. OpenEQA: Embodied Question Answering in the Era of Foundation Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16488–16498, 2024.DOI
- [38]Karttikeya Mangalampalli, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. In Advances in Neural Information Processing Systems, 2023.
- [39]Zaira Manigrasso, Matteo Dunnhöfer, Antonino Furnari, Moritz Nottebaum, Antonio Fioniacchiro, Davide Marana, Rosario Forte, Giovanni Maria Farinella, and Christian Micheloni. Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2026.DOI
- [40]Meta. SAM 3.1 Release Notes. Official SAM 3 repository, March 27 release, 2026.
- [41]MickeyLLG. ego_plan: Media Repository. Hugging Face Datasets, n.d. Accessed October 24, 2026.
- [42]Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai, James Hillis, and Kristen Grauman. EgoEnv: Human-centric environment representations from egocentric video. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2207.11365
- [43]OpenAI. GPT-5.3 Chat. Azure OpenAI deployment gpt-5.3-chat, 2026. Accessed November 8, 2026.
- [44]OpenAI. GPT-6.1 Sol Model. OpenAI API documentation, 2026. Accessed October 25, 2026.
- [45]Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, Jacob Chalk, Zhifan Zhu, Rhodri Guerrier, Fahd Abdelazim, Bin Zhu, Davide Moltisanti, Michael Wray, Hazel Doughty, and Dima Damen. HD-EPIC: A Highly-Detailed Egocentric Video Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23901–23913, 2025.arxiv.org/abs/2502.04144
- [46]Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
- [47]Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. REVERIE: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
- [48]Yujia Qin, Shiao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. In International Conference on Learning Representations, 2024.arxiv.org/abs/2307.16789
- [49]Qwen Team. Qwen3.5-4B Model Card. Official model repository, 2026.
- [50]Qwen Team. Qwen3.8-27B Model Card. Official model repository, 2026. Accessed October 24, 2026.
- [51]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations, 2025.arxiv.org/abs/2408.00714
- [52]Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, pages 68539–68551, 2023.arxiv.org/abs/2302.04761
- [53]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [54]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, pages 8634–8652, 2023.arxiv.org/abs/2303.11366
- [55]Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. MovieChat: From Dense Token to Sparse Memory for Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221–18232, 2024.arxiv.org/abs/2307.16449
- [56]Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4D Goal-Step: Toward Hierarchical Understanding of Procedural Activities. In Advances in Neural Information Processing Systems, 2023.DOI
- [57]Hao Tang, Kevin J. Liang, Kristen Grauman, Matt Feiszli, and Weiyao Wang. EgoTracks: A Long-term Ego-centric Visual Object Tracking Dataset. In Advances in Neural Information Processing Systems, 2023.DOI
- [58]Shulin Tian, Ruiqi Wang, Hongming Guo, Penghao Wu, Yuhao Dong, Xiuying Wang, Jingkang Yang, Hao Zhang, Hongyuan Zhu, and Ziwei Liu. Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning. arXiv preprint arXiv:2506.13654, 2025.arxiv.org/abs/2506.13654
- [59]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Yingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv preprint arXiv:2508.18265, 2025.
- [60]Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohao Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, and Jie Tang. LVBench: An Extreme Long Video Understanding Benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967, 2025.arxiv.org/abs/2406.08035
- [61]Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. In Proceedings of the European Conference on Computer Vision, 2024.arxiv.org/abs/2403.10517
- [62]Yun Wang, Junbin Xiao, Han Lyu, Yifan Wang, Jing Zuo, Zhanjie Zhang, Hong Huang, Dapeng Wu, and Angela Yao. Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams. arXiv preprint arXiv:2606.15200, 2026. ICML 2026.arxiv.org/abs/2606.15200
- [63]Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3272–3283, 2025.arxiv.org/abs/2405.19209
- [64]Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2407.15754
- [65]Junbin Xiao, Shenglang Zhang, Pengxiang Zhu, and Angela Yao. Ego-Grounding for Personalized Question-Answering in Egocentric Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 40537–40547, 2026.arxiv.org/abs/2604.01966
- [66]Xiaomi LLM-Core Team. MiMo-VL Technical Report. arXiv preprint arXiv:2506.03569, 2025.
- [67]Ziang Yan, Sheng Xia, Jiashuo Yu, Yue Wu, Tianxiang Jiang, Songze Li, Kanghui Tian, Yicheng Xu, Yinan He, Kai Chen, Limin Wang, Yu Qiao, and Yi Wang. InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning. arXiv preprint arXiv:2606.12195, 2026.arxiv.org/abs/2606.12195
- [68]Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cornia, Zhongang Cai, Bo Li, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, and Ziwei Liu. EgoLife: Towards Egocentric Life Assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28885–28900, 2025.arxiv.org/abs/2503.03803
- [69]Runmao Yao, Yi Du, Zhuoqun Chen, Haoze Zheng, and Chen Wang. AirRoom: Objects Matter in Room Reidentification. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1385–1394, 2025.
- [70]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.03629
- [71]Boqiang Zhang, Kehan Li, Zezen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Video-LLama 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv preprint arXiv:2501.13106, 2025.arxiv.org/abs/2501.13106
- [72]Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A Simple LLM Framework for Long-Range Video Question-Answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21715–21737, 2024.arxiv.org/abs/2312.17235
- [73]Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. RAFT: Adapting Language Model to Domain Specific RAG. In First Conference on Language Modeling, 2024.arxiv.org/abs/2403.10131
- [74]Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Giridhar. Learning Video Representations From Large Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6586–6597, 2023.arxiv.org/abs/2212.04501
- [75]Yunhan Zhao, Haoyu Ma, Shu Kong, and Charless Fowlkes. Instance Tracking in 3D Scenes from Egocentric Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21933–21944, 2024.arxiv.org/abs/2312.04117
- [76]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: Benchmarking Multi-task Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13691–13701, 2025.arxiv.org/abs/2406.04264
- [77]Chenchen Zhu, Fanyi Xiao, Andres Alvarado, Yasmine Babaei, Jiabo Hu, Hichem El-Mohri, Sean Culatana, Roshan Sumbaly, and Zhicheng Yan. EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20110–20120, 2023.arxiv.org/abs/2309.08816
- [78]Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. SOON: Scenario oriented object navigation with graph-based exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12689–12699, 2021.













