Holi-World: Building Holistic World Intelligence from Human-Ego Experience

Code: github.com/Ropedia/holi-world Model: huggingface.co/ropedia-ai/holiworld-vl Dataset: huggingface.co/datasets/ropedia-ai/holiworld-data

Introduction

Learning how the physical world is organized and how it changes through interaction is a fundamental goal of artificial intelligence and an essential step toward artificial general intelligence (AGI) [39, 18, 19]. Although vision–language learning and predictive video modeling enable machines to recognize objects, describe scenes, interpret activities, and anticipate visual changes [35, 20, 23, 3, 1], building a coherent understanding of the world across viewpoints and time remains an open challenge [44, 41, 22]. Everyday human experience naturally brings these dimensions together: moving through an environment reveals its enduring spatial structure, while interacting with objects reveals how they are used and how their states and relationships change. Therefore, we place human experience at the center of learning world intelligence, asking how it should be captured, organized, and learned from to support an understanding of environments, objects, and interactions as parts of a shared, evolving world.

Egocentric data offers a direct route to learning about the world through the breadth of human life. It records environments as people experience them, bringing together the places they inhabit, the objects they use, and the activities through which their surroundings change [36, 9, 40]. Large-scale efforts [13, 14, 26, 45] have expanded this record across diverse people, places, and activities. However, turning these recordings into detailed learning supervision often requires substantial human input, tying further expansion to additional annotation and curation costs [10]. The challenge is therefore to scale high-quality supervision with incoming experience while retaining the spatial and temporal information needed to understand environments, objects, and interactions as parts of an evolving world.

Recent advances provide complementary capabilities for automating this process: reconstruction and semantic annotation of predominantly static scenes for spatial supervision [12], and egocentric tracking of object identities and 3D locations over time [48, 31]. Turning these capabilities into coherent learning resources requires connecting scene geometry and object semantics with interaction histories, then deriving supervision that remains traceable to the underlying observations. To this end, we introduce HOLIWORLD, an integrated ecosystem for automatically constructing high-quality learning resources at scale from egocentric recordings that we collect in the wild using our capture device. At its core, HOLIWORLD-Foundry first reconstructs scene geometry and camera poses, then uses VLM-derived motion and interaction cues to separate static objects from dynamic tracking targets. For static objects, it lifts image detections into 3D and consolidates them with scene point-cloud detections to recover object geometry and semantics. For dynamic objects, it associates identities across interaction segments, tracks object regions in image space, and recovers 3D states and trajectories in shared world coordinates. Built on these multimodal records, HOLIWORLD-Data organizes approximately 100 scenes and 444 egocentric videos across four levels: raw observations, reconstructed geometry, object and interaction semantics, and spatial intelligence supervision. By further parsing geometric relations, object trajectories, and interaction states, we construct approximately 80K evidence-driven QA pairs covering spatial relations, temporal changes, and interactions, with each answer linked to its supporting observations and annotations. We train HOLIWORLD-VL on this supervision for visual grounding and spatial reasoning, providing a downstream test of the generated data’s learning value. Finally, we establish HOLIWORLD-Bench through independent human annotation of held-out recordings, providing reference annotations for evaluating both Foundry’s world-record accuracy and HOLIWORLD-VL’s spatial understanding.

Experiments evaluate HOLIWORLD through both annotation quality and downstream spatial reasoning. For static perception, HOLIWORLD-Foundry achieves an $AP_{25}$ of 0.68 on ScanNet [8], compared with 0.40 for the strongest baseline in our evaluation. For dynamic tracking, it achieves a $HOTA_{0.1}$ of 54.31% on 20 Aria Digital Twin sequences [29], compared with 36.00% for IT3DEgo [48] with ground-truth-guided initialization. The resulting supervision also supports substantial improvements in downstream reasoning: compared with the original Qwen3.5-4B, HOLIWORLD-VL-4B improves Token F1 from 16.35% to 78.02% on HOLIWORLD-Bench and from 18.29% to 66.39% on EgoDynamic4D [15].

Our contributions are threefold:

• We introduce HOLIWORLD, an integrated ecosystem that provides a practical blueprint for learning world intelligence from human egocentric experience. It connects experience capture, world representation, automated data construction, model learning, and independent evaluation around a shared representation of the evolving world.

• We develop HOLIWORLD-Foundry to automatically construct world records that preserve persistent scene structure and evolving object states. Complementary static and dynamic perception links geometry, semantics, and interaction histories through persistent identities and supporting observations, enabling scalable construction of grounded supervision.

• We construct HOLIWORLD-Data to support grounding and spatial and temporal reasoning, and use these resources to train HOLIWORLD-VL. We further establish HOLIWORLD-Bench through independent human annotation, providing reference data for evaluating both the quality of automatically constructed world records and the capabilities of learned models.

Related Work

Egocentric Datasets and Annotations

Large-scale egocentric datasets cover diverse human activities and environments [9, 13, 14]. EPIC Fields and Aria Digital Twin add camera and scene geometry [40, 29], while HD-EPIC [30] connects actions, object movements, and gaze to 3D digital twins through detailed manual annotation. However, extending such detailed supervision to new recordings entails further annotation effort [10, 30]. HOLIWORLD addresses this scaling bottleneck by automatically recovering scene structure, object states, and interaction annotations from newly captured recordings and converting them into learning supervision, reducing dependence on manual labeling as new experience is collected.

3D Scene Reconstruction and Object Tracking

Holi-Spatial [12] automates static scene reconstruction, semantic annotation, and spatial QA generation at scale, while Boxer [11] consolidates multi-view detections into static 3D boxes. However, static scene consolidation does not retain the successive object states needed to describe motion and interactions. Egocentric tracking provides complementary capabilities: IT3DEgo and EgoSeg3D [48, 4] maintain object identities and locations over time, and LMK, introduced for the Out of Sight, Not Out of Mind (OSNOM) task [31], maintains spatial memory beyond the current view. HOLIWORLD-Foundry connects these static and dynamic perception objectives within an automated data construction pipeline. Shared world coordinates and persistent identities link scene geometry, evolving object states, and interaction evidence, enabling spatial and temporal supervision to refer to the same objects and supporting observations.

Spatial and Temporal Reasoning in Vision–Language Models

Dedicated spatial supervision [6, 27] and geometry-aware visual representations [50, 49, 42] improve spatial reasoning, while video benchmarks [44, 22] test these capabilities across observations. ST-Think and EgoReasoner [43, 51] further study reasoning over egocentric spatial and temporal information. EgoDynamic4D [15] constructs dynamic QA from existing annotated 4D resources and introduces a model integrating static and dynamic scene information. We follow its QA patterns, while addressing the upstream challenge of automatically constructing the underlying annotations from new recordings. HOLIWORLD connects experience capture, automated annotation, supervision generation, and model training, allowing new recordings to expand the learning resources. Independent human-annotated evaluation then assesses both the accuracy of the recovered records and the capabilities learned from them.

Method

HoliWorld Overview

HoliWorld converts egocentric recordings into spatially and temporally aligned supervision (Figure 2). The Eagle Capture Device records synchronized stereo RGB and IMU measurements; HoliWorld-Foundry reconstructs scenes and annotates static objects and dynamic tracks. HoliWorld-Data organizes these outputs into grounding, captioning, and spatial QA supervision for HoliWorld-VL. Independent human annotations of held-out recordings form HoliWorld-Bench for evaluating Foundry’s annotation quality and HoliWorld-VL’s spatial understanding.

HoliWorld Foundry

Geometric Preparation and Shared Semantic Cues

As illustrated in Figure 3, we first recover camera poses and sparse points from stereo RGB and IMU using visual-inertial SLAM, then estimate depth and dense points from RGB keyframes and these poses with Depth Anything 3 (DA3) [21]. Using RGB observations and this geometry, we optimize a 3D Gaussian scene representation $\mathcal{G}$, render refined depth at the observed poses, and complete dynamic regions with per-frame DA3 depth estimation. We retain $\mathcal{G}$ and the reconstructed scene point cloud $\mathcal{P}$, and organize the resulting observations as

where $I_t$ is an RGB frame, $D_t$ is a depth map, $K_t$ is the camera intrinsic matrix, and $T_t$ maps camera coordinates to world coordinates. To route static and dynamic objects to their respective perception pipelines, a vision–language model (VLM) extracts categories, attributes, motion cues, and interaction descriptions from RGB keyframes in $O$; categories guide static image-based proposals, while motion and interaction cues identify dynamic tracking targets.

Static Object Perception

Dual-path 3D detection. To capture both object details and scene-level geometry, we combine image-based and point-cloud-based 3D detection (Figure 4A). In the image-based path, we run Boxer [11] separately on the left and right RGB streams to capture complementary object details. Open-vocabulary 2D detections are lifted into metric 3D boxes using the corresponding depth and camera poses, then associated across frames. In parallel, TR3D [38] detects objects in the reconstructed scene point cloud $\mathcal{P}$, capturing large-object extents beyond individual views.

Box merging and geometric filtering. Across these complementary sources, multiple proposals may refer to the same object, while others have little geometric support. We therefore use low-cost geometric checks to merge spatially compatible boxes and discard those with insufficient point support in $\mathcal{P}$ before VLM validation (Figure 4B). For validation, we also preserve one fallback per selected box when available: the rank-1 alternative in its candidate cluster, or, if unavailable, a directly suppressed candidate linked by IoU suppression. Each box retains its own supporting evidence.

VLM validation. The remaining boxes may still have inaccurate extents or incorrect category labels. A VLM therefore checks each selected box against source observations and supplementary views rendered from $\mathcal{G}$ (Figure 4C). Based on this evidence, it can keep the box, refine its geometry, relabel its category, or reject it. If the selected box is rejected, we evaluate its preselected fallback once using the fallback’s own evidence. If no unused fallback is available or the retry fails, no box is retained. Accepted boxes form $\mathcal{B}^{\mathrm{sta}}$ with links to their supporting observations.

Dynamic Object Perception

Interaction grounding and identity association. Manipulated objects may disappear from view and reappear in later interaction segments. To recognize the same object across these interruptions, we use hand regions and VLM interaction descriptions to identify the target in each segment, then compare descriptions and keyframe evidence across segments. This temporal label continuity establishes consistent object identities that guide target mask selection (Figure 5).

Identity-guided mask tracking. The associated identities specify which object to follow, while frame-level masks are needed to localize it over time. Guided by these identities, the Segment Anything Model (SAM) [17, 37] selects and initializes the target mask, then propagates it within each interaction segment to obtain $\{m_t\}$. We assume at most one moving object at each time, with $m_t$ denoting its foreground pixels.

3D recovery and motion cues. Image-space motion also reflects camera movement. To recover object states in world coordinates, we back-project the pixels of each valid tracked mask using the DA3-completed dynamic depths in $D_t$ and the camera pose:

where $\bar{\mathbf{u}}$ is the homogeneous image coordinate, $\mathbf{T}_t(\cdot)$ applies the camera-to-world rigid transformation, and $\mathcal{X}_t$ contains the object-supported world points. After filtering unsupported depth samples, we fit an OBB to each point set and link boxes with the same identity across time to form a dynamic 4D track.

Unified World Representation

Object records connect instance identities and categories to image evidence, 3D geometry, and interaction cues. We express the Foundry output at time $t$ as

where $O_t = (I_t,D_t,K_t,T_t)$ is the input observation, $\mathcal{U}(t)$ contains the detected 2D boxes and tracked masks, and $\mathcal{Z}(t)$ records the associated labels, identities, confidence scores, interaction semantics, and evidence links. Static instances persist across observations; dynamic instances retain time-indexed states. This representation lets supervision refer to specific objects, times, and supporting observations.

HoliWorld-Data

HoliWorld-Data organizes the world records constructed by Foundry into grounding, captioning, and spatial, temporal, and interaction QA supervision. The available evidence determines the scope of these tasks: static objects and their geometry support spatial understanding; dynamic tracks with consistent object identities support motion and temporal understanding; and tracks paired with interaction semantics support descriptions of object manipulation and state changes. Each example retains links to source observations and annotations, allowing its target output to be traced to specific objects, times, and supporting evidence.

QA pairs construction. We construct QA pairs from the spatial, temporal, and interaction annotations in Foundry’s world records, following the QA patterns of EgoDynamic4D [15]. Grounding tasks associate referring expressions with the corresponding 2D observations, static 3D boxes, or dynamic tracks. Spatial QA uses object geometry and camera poses to derive direction, distance, containment, support, and viewpoint-dependent relations. Temporal QA compares the states of the same object over time to describe motion, event ordering, and duration. Interaction QA additionally uses interaction semantics and source observations to describe manipulated objects and state changes. Scene captions and inventory questions aggregate object records across frames to describe the observed environment. Dynamic supervision follows Foundry’s current assumption of at most one moving object at each time.

Dataset composition. HoliWorld-Data contains roughly 100 scenes and 444 egocentric videos, with approximately 100 3D Gaussian splatting (3DGS) reconstructions, 1,322K RGB frames with corresponding depth maps and camera poses, 90K 2D boxes, 2,466 static 3D boxes, and 2,095 dynamic tracks (Figure 6). These records support about 4K 3D grounding annotations and 80K captions, together with 13,647 static QA pairs and 64,409 dynamic QA pairs, totaling about 80K QA pairs.

Experiments

We evaluate the pipeline’s ability to generate object-centric 3D annotations for both static scenes and moving objects. § 4.1 and § 4.2 present the results for static object perception and dynamic object tracking. § 4.4 examines the contributions of key pipeline components through controlled ablations. Detailed experimental settings are provided in Appendix Sections A.1 and A.2.

Static 3D Object Perception

As shown in Table 1, HOLIWORLD-FOUNDRY achieves the highest $\mathrm{AP}_{25}$ and $\mathrm{AP}_{50}$ on all three datasets. On ScanNet [8] and ScanNet++ [47], $\mathrm{AP}_{50}$ reaches 0.55 and 0.13, respectively, compared with 0.12 and 0.05 for the strongest baselines, indicating more accurate detection under stricter geometric matching. The method also leads on the more complex real-world scenes in HoliWorld-Bench. Figure 7 shows more complete object coverage and better-aligned box extents across representative scenes, consistent with these quantitative gains.

Method	ScanNet			ScanNet++			HoliWorld-Bench
	$\mathrm{AP}_{25}\uparrow$	$\mathrm{AP}_{50}\uparrow$	$\mathrm{mIoU}\uparrow$	$\mathrm{AP}_{25}\uparrow$	$\mathrm{AP}_{50}\uparrow$	$\mathrm{mIoU}\uparrow$	$\mathrm{AP}_{25}\uparrow$	$\mathrm{AP}_{50}\uparrow$	$\mathrm{mIoU}\uparrow$
Model-based methods
SpatialLM	0.18	0.09	0.52	0.03	0.01	0.50	0.04	0.00	0.23
LLaVA-3D	0.02	0.00	0.34	0.00	0.00	0.34	0.00	0.00	0.00
SceneScript	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Mapter-3D	0.40	0.12	0.48	0.18	0.05	0.48	0.12	0.01	0.37
3DGS-based methods
M3-Spatial	0.00	0.00	0.37	0.00	0.00	0.60	0.06	0.00	0.28
LangSplat	0.02	0.00	0.45	0.01	0.00	0.33	0.01	0.00	0.10
Spatial pipeline frameworks
Boxer	0.32	0.09	0.47	0.23	0.05	0.46	0.16	0.07	0.52
Holi-Spatial	0.33	0.10	0.47	0.17	0.05	0.45	0.07	0.01	0.37
Ours	0.68	0.55	0.72	0.30	0.13	0.56	0.24	0.12	0.53

Dynamic 3D Object Perception and Tracking

In Table 2, HOLIWORLD-FOUNDRY attains HOTA scores of 54.31% on ADT and 43.14% on HoliWorld-Bench, compared with 36.00% and 15.13% for the strongest baselines, respectively. Higher AssA and PCL indicate more consistent identities and broader trajectory coverage across both datasets. Box metrics provide a complementary comparison of spatial extents on matched objects. Figure 8 illustrates trajectories that more closely follow the reference paths across the shown interaction stages.

Method	ADT20						HoliWorld-Bench
	Bounding box			Trajectory			Bounding box			Trajectory
	IoU↑	GTC↑	PP↑	HOTA↑	AssA↑	PCL↑	IoU↑	GTC↑	PP↑	HOTA↑	AssA↑	PCL↑
Instance segmentation and tracking pipeline
EgoSeg3D{sup}†`	–	–	–	21.89	37.11	36.47	–	–	–	5.76	21.78	36.84
IT3DEgo{sup}*`	–	–	–	36.00	53.60	34.64	–	–	–	2.44	4.36	7.89
SAM+Whareformer{sup}*`	–	–	–	30.76	40.99	31.26	–	–	–	14.63	20.97	36.84
3D object perception and tracking pipelines
SAM+LMK{sup}*`	7.96	21.27	14.53	26.10	30.76	21.45	20.28	50.84	21.30	15.13	14.35	23.68
Ours	30.88	43.03	56.25	54.31	71.94	47.95	41.79	53.18	76.49	43.14	56.54	57.89

Downstream Spatial and Temporal Understanding

Fine-tuning Qwen3.5-4B [33] on HoliWorld-Data raises Token F1 from 16.35% to 78.02% on HoliWorld-Bench and from 18.29% to 66.39% on EgoDynamic4D (Table 3), supporting the value of the generated supervision for spatial and temporal reasoning. The resulting HoliWorld-VL-4B also surpasses zero-shot Qwen3.8-27B [34] on all four metrics across both benchmarks. The gains in yes/no, multiple-choice, and numerical accuracy complement the Token F1 improvements, indicating that the generated supervision benefits multiple question formats across the two benchmarks.

Model	\multicolumn{4}{c}{HoliWorld-Bench}	\multicolumn{4}{c}{EgoDynamic4D}
	F1↑	Y/N↑	MCQ↑	Num.↑	F1↑	Y/N↑	MCQ↑	Num.↑
Qwen3.5-4B	16.35	34.86	62.00	10.80	18.29	59.47	79.16	0.02
Qwen3.8-27B	43.67	82.85	75.00	66.13	42.93	61.07	57.54	0.16
HoliWorld-VL-4B	78.02	93.52	94.00	81.84	66.39	99.67	94.92	16.33

Ablation Studies

Static components. Figure 9(a–b) illustrates complementary geometric coverage: image-based lifting method captures more small objects, point-cloud detection recovers larger extents, and stereo fusion combines observations from both views. Caption guidance recovers missed entities (c). VLM verification shows modest changes in geometric metrics, while the qualitative examples illustrate corrections to object labels and unsuitable candidates (d), suggesting a complementary role in semantic annotation. Detailed $AP_{25}$ and $m\mathrm{IoU}$ comparisons are reported in Table 5 (Appendix § 6.§ 3).

Table 5 | Component ablations. Static AP$_{25}$/mIoU: [0, 1]; dynamic HOTA/PCL: % at 0.1 m. Checkmarks/crosses: enabled/disabled; dashes: unavailable or inapplicable. Shaded rows (S6, D3) enable all components.

Components

Static data

Dynamic data

Static

Dynamic

Boxer

TR3D

StereoGeo

Caption

VLM-Ver.

Label-Cont.

SAM-Assoc.

ScanNet

HoliWorld-Bench

ADT20

HoliWorld-Bench

ID

AP$_{25}$ ↑

mIoU↑

AP$_{25}$ ↑

mIoU↑

HOTA↑

PCL↑

HOTA↑

PCL↑

S1

✗

✓

✓

✓

✓

–

–

0.58

0.80

0.10

0.47

–

–

S2

✓

✗

✓

✓

✓

–

–

0.32

0.30

0.20

0.52

–

–

S3

✓

✓

✗

✓

✓

–

–

–

–

0.20

0.53

–

–

S4

✓

✓

✓

✗

✓

–

–

0.67

0.70

0.21

0.52

–

–

S5

✓

✓

✓

✓

✗

–

–

0.68

0.72

0.24

0.52

–

–

S6

✓

✓

✓

✓

✓

–

–

0.68

0.72

0.24

0.53

–

–

D1

–

–

–

–

–

✗

✓

–

–

–

–

44.09

42.62

D2

–

–

–

–

–

✓

✗

–

–

–

–

38.38

34.08

D3

–

–

–

–

–

✓

✓

–

–

–

–

54.31

47.95

Dynamic components. Figure 9(e–f) highlights two complementary requirements for reliable trajectories. Cross-interaction label continuity reduces identity fragmentation across repeated observations, while SAM association supplies temporally consistent masks for 3D lifting, reducing the trajectory deviations seen with per-frame detection. Together, they preserve both object correspondence and spatial continuity. Corresponding HOTA and PCL gains on both datasets are reported in Table 5 (Appendix § 6.§ 3).

Conclusion

In this paper, we presented HOLIWORLD, an integrated ecosystem that places human egocentric experience at the center of learning world intelligence. Through complementary static and dynamic perception, HOLIWORLD-Foundry automatically constructs world records linking scene geometry, object semantics, and interaction histories to supporting observations. These records form HOLIWORLD-Data and support HOLIWORLD-VL training, while independently annotated HOLIWORLD-Bench provides reference data for evaluating annotation quality and learned capabilities. Experiments demonstrate improved static perception and dynamic tracking, alongside gains in spatial and temporal reasoning on HOLIWORLD-Bench and EgoDynamic4D. HOLIWORLD provides a practical foundation for expanding these grounded learning resources with new recordings, enabling machines to learn how the world is organized and how it changes through interaction.

Experimental Details

Static Perception Evaluation Details

Data and metrics. We evaluate on 20 sequences from each of ScanNet and ScanNet++, using their official GT 3D annotations, and 20 self-collected in-house sequences with manually annotated GT. AP$_{25}$ and AP$_{50}$ use 3D IoU thresholds of 0.25 and 0.50, respectively. Matched-box mIoU is averaged over box pairs matched at IoU 0.25. All static metrics use a [0, 1] scale; dashes indicate unavailable results.

Baselines. The comparison covers model-based methods, including the generalist scene models SpatialLM [28], LLaVA-3D [50], and SceneScript [2], and the specialist detector Map-Det3D [46]; 3DGS-based methods M3-Spatial [52] and LangSplat [32], which encode semantics in Gaussian representations; and spatial pipelines Boxer [11] and Holi-Spatial [12], which integrate multiple perception and geometric processing stages.

Additional analysis. At IoU 0.50, HOLIWORLD-FOUNDRY reaches AP$_{50}$ of 0.55 on ScanNet and 0.13 on ScanNet++, compared with the strongest baseline scores of 0.12 and 0.05 (Table 1). These gains persist under stricter geometric matching. On the in-house data, the improvement over Boxer is more pronounced in AP than in matched-box mIoU, suggesting that improved detection contributes beyond the extent accuracy of successfully matched boxes. Because mIoU is conditional on matching, it should be read jointly with AP when assessing overall perception quality. Figure 10 compares all evaluated methods qualitatively.

Full Dynamic Tracking Results

Table 4 and Figure 11 report the full quantitative and qualitative comparisons.

	Bounding box			Trajectory
Method	IoU↑	GTC↑	PP↑	HOTA$_{0.1}$ ↑	DetA$_{0.1}$ ↑	AssA$_{0.1}$ ↑	PCL$_{0.1}$ ↑	R$_{0.1}$ ↑	P$_{0.1}$ ↑	TP-L2↓
ADT20
2D instance tracking pipelines
EgoSeg3D†	–	–	–	21.89	13.78	37.11	36.47	46.96	16.67	0.0432
IT3DEgo*	–	–	–	36.00	24.63	53.60	34.64	35.69	40.93	0.0490
SAM+Whareformer*	–	–	–	30.76	23.60	40.99	31.26	33.35	41.33	0.0497
3D object perception and tracking pipelines
SAM+LMK*	7.96	21.27	14.53	26.10	22.84	30.76	21.45	27.87	52.17	0.0485
Ours	30.88	43.03	56.25	54.31	42.13	71.94	47.95	49.56	76.68	0.0333
HoliWorld-Bench (in-house)
2D instance tracking pipelines
EgoSeg3D†	–	–	–	5.76	1.52	21.78	36.84	42.11	1.55	0.0094
IT3DEgo*	–	–	–	2.44	1.36	4.36	7.89	7.89	1.62	0.0479
SAM+Whareformer*	–	–	–	14.63	10.20	20.97	36.84	39.47	12.10	0.0113
3D object perception and tracking pipelines
SAM+LMK*	20.28	50.84	21.30	15.13	15.96	14.35	23.68	39.47	21.13	0.0113
Ours	41.79	53.18	76.49	43.14	32.91	56.54	57.89	68.42	38.81	0.0280

Metrics. HOTA$_{0.1}$, DetA$_{0.1}$, and AssA$_{0.1}$ follow Luiten et al. [25], using a 0.1 m world-coordinate center-distance tolerance. They measure joint detection and association quality, detection accuracy, and identity association consistency, respectively. PCL$_{0.1}$ measures correctly localized GT instance-frames under a fixed, sequence-global one-to-one GT–prediction assignment that maximizes matches within 0.1 m; unassigned trajectories, missing predictions, and localization failures count as incorrect. Recall R$_{0.1}$, precision P$_{0.1}$, and TP-L2 use HOTA’s frame-level matches to measure GT coverage, prediction reliability, and mean center error among successful matches. Box geometry is assessed separately using IoU, GT coverage (GTC), and prediction precision (PP). All metrics are percentages except TP-L2, which is in meters.

Baselines and inputs. EgoSeg3D† [4] receives sequence-level GT category queries. Its GroundingDINO-SAM-v1–DEVA frontend [24, 17, 7] generates and propagates masks before 3D lifting and native association. For IT3DEgo* [48], we provide a predefined target set and convert each target’s first-visible GT mask to a bounding box for its native SVOE enrollment interface; subsequent mask/proposal inference and memory updates use the original pipeline.

SAM+Whareformer* [5] and SAM+LMK* [31] share a SAM2 propagation frontend [37], initialized with each target’s GT mask at its first visible frame in the video, irrespective of when it starts moving. Propagation continues through the remaining video to supply mask observations for native association. An adapter derives Whareformer’s world centers from masks, depth, and camera poses; LMK retains its native 3D center/scale outputs. Ours uses no annotation-derived test-time inputs. GT initialization defines baseline inputs only, not evaluation correspondences.

Evaluation and binding rules. We evaluate 20 sequences each from ADT and HoliWorld-

Bench over the full RGB timeline, including out-of-view periods, wherever a valid GT pose is available within 20 ms. Missing predictions remain evaluation failures, and sequence scores are averaged equally. We group pipelines by whether they output 3D boxes. Output-dependent GT–prediction binding uses nearest world-coordinate centers; for pipelines with 3D boxes, candidate tracks must additionally have maximum 3D IoU $> 0.5$. IoU, GTC, and PP are computed on the resulting bound-object subset. This binding does not filter trajectory evaluation: all methods use the common distance-based scoring described above, with native predicted identities and PCL’s own sequence-global assignment.

Additional Ablation Results

Table 5 reports component-wise results; Figure 12 compares the two static perception branches.

Holi-World Static and Dynamic QA

30 QA types. We design 30 fine-grained QA types, broadly divided into static and dynamic reasoning. The static category contains 11 types that evaluate camera motion and object-level spatial relationships, including distance, depth, height, and relative position under single- and multi-view settings. The dynamic category comprises seven trajectory-based types for motion status, timing, direction, sequence, and path characteristics, together with 12 caption-based types covering action recognition, object interaction, temporal grounding, interaction phases, event ordering, state transitions, and contrastive hard negatives. As illustrated in Figure 13, these questions combine visual observations with 3D bounding boxes, object trajectories, and temporally aligned captions, while their answers span categorical labels, numerical values, multiple-choice options, time intervals, and free-form descriptions.

Reproducibility and Responsible Use

Reproducibility. To ensure reproducibility, § 3 details the HoliWorld pipeline, covering geometric reconstruction, static object perception, dynamic tracking, and supervision construction, alongside dataset composition and QA construction. Appendix § 6 further specifies evaluation datasets, metric definitions, baseline adaptations, matching protocols, and ablation configurations.

Responsible use. HoliWorld is designed for research on scene geometry and spatial-temporal understanding. Responsible data collection and release therefore require appropriate consent, review and redaction of identifying content, and compliance with dataset and model licenses. The pipeline should not be used for non-consensual surveillance or high-stakes decisions without additional safeguards and validation.