betaFeedback

WorksComputer Science

HumanCOIN: A 12,500-Hour Dataset for Human Intent in Egocentric Manipulation

Abstract

First-person manipulation datasets treat the observation-action pair as the atomic unit and compress the generation and execution of an action into a policy network's weights, rendering the intent state behind the action unobservable. Human motor control research shows that intent mediates between observation and action and leaves observable traces along the intent pathway from decision to effect: gaze reaches the target before the hand, muscle load builds before contact, and cortical activity precedes the motor command. To record this pathway at scale, we present HumanCOIN, a first-person manipulation dataset built around human intent: 12,500 hours across 38 scene classes with three tiers of proxies. At the output end, the L1 attention tier (10,000 h) infers gaze from headband camera video, exposing the operator's next object of attention and reach before the hand moves. Midway, the L2 exertion tier (2,000 h) records surface EMG from armbands, making load expectations readable before execution is complete. At the origin, the L3 cognition tier (500 h) records scalp EEG, giving early readouts of task decisions and monitoring. Each tier inherits all proxies from the tiers below. The dataset further provides multi-view RGB images, depth maps and point clouds, 6-DoF head and wrist pose, and step-level action and object annotations. The controlled L3 setting additionally measures gaze, hand pose and finger kinematics directly. Clock and calibration data are released with the dataset so that any read rate can be re-aligned. The three tiers cover three stations, from visual orienting to cortical preparation, making the process behind visual-action pairs available for modeling at dataset scale. Project page: https://flagopen.github.io/HumanCOIN.

HumanCOIN dataset overview

Figure 1.

Corresponding authors: Zhongyuan Wang and Guocai Yao ([email protected]); contact: [email protected]

Introduction

Egocentric data has become the primary route from human video to embodied learning. A head-mounted camera records the scene from the operator’s viewpoint, hand pose recovers the motion of the hands, and together they form a visual–action trajectory: what was seen and how the hand moved. This paradigm has scaled from passive everyday video [1, 2] to instrumented manipulation corpora [3, 4, 5] reaching tens of thousands of hours [6, 7], and the resulting data has been used directly to train robot policies [8, 9, 10]. Across all variations in scale and form, the basic unit remains the observation–action pair (o,a)(o, a).

Two broad approaches have organised work around this unit. The first directly models π(a∣o)\pi(a \mid o): all available information—visual observations, depth, pose, and language—is subsumed into oo, and the mapping to aa is learned from data at scale [11, 12, 13, 14, 15, 16]; the operator’s intent state is compressed wholesale into network weights, rendering it unobservable—when a policy fails, there is no way to localise whether the target was wrong, the force incorrect, or an error signal missed. The second introduces an explicit intermediate and chooses the future: world models and video–action co-models predict a latent representation o^′\hat{o}^{\prime} of the next observation, then generate the action, π:o→o^′→a\pi: o \to\hat{o}^{\prime} \to a [17, 18, 19, 20]; however, a predicted future records only the visible outcome of an action, not the operator state that produced it, and therefore diverges from intent precisely where supervision is needed most.

Routes from observation to action through implicit, predicted, and proxy-conditioned intermediates

Figure 1 (PDF p. 3). The middle of o→ao \to a, and what each route puts there. An action policy models π(a∣o)\pi(a \mid o) directly, leaving the middle implicit in its weights; a world model inserts a predicted future o^′\hat{o}^{\prime} and factors through it. The third row is not another method: HumanCOIN supplies the middle as proxies m=(mgaze,memg,meeg)m = (m_{\mathrm{gaze}}, m_{\mathrm{emg}}, m_{\mathrm{eeg}}) for the intent zz, on a common time base with the action.

Figure 1 sets the two routes against the alternative: both place something in the middle of o→ao \to a, and what they place there is either folded away or inferred, never read off the operator. What is missing is not another estimate of that middle but a record of it, and that is what HumanCOIN provides.

Intent cannot be observed directly, but leaves measurable physical traces along the chain from decision to effect—a chain we term the intent pathway—and we denote the operator’s state along this pathway as the latent variable zz. The standard policy implicitly integrates over zz, p(a∣o)=∫p(a∣o,z) p(z∣o) dzp(a \mid o) = \int p(a \mid o, z)\,p(z \mid o)\,\mathrm{d}z, folding it into weights and rendering it unobservable; recording proxies for zz would allow both factors—p(m∣o)p(m \mid o), a proxy model, and p(a∣o,m)p(a \mid o, m), the proxy-conditioned policy—to be trained independently. Consider lifting a cup: gaze settles on the rim before the hand departs, externalising intent as a visual orienting signal (mgazem_{\mathrm{gaze}} [21, 22, 23]); forearm muscles then load grip force to the anticipated weight before the cup leaves the surface, encoding a prior over object properties as a motor command (memgm_{\mathrm{emg}} [24, 25]); and cortical activity has already been building a Bereitschaftspotential for hundreds of milliseconds, with an error-related negativity ready to fire within milliseconds of a mismatch (meegm_{\mathrm{eeg}} [26, 27, 28]). Each proxy precedes aa and has mature wearable sensors with decades of signal-processing precedent [29, 30]; synchronised recordings exist [31], yet all remain confined to laboratory settings with few participants and short scripted trials. Large-scale synchronised recording of all three proxies in naturalistic manipulation scenes has not been achieved.

We present HumanCOIN, a large-scale egocentric manipulation dataset that records a proxy vector m=(mgaze,memg,meeg)m = (m_{\mathrm{gaze}}, m_{\mathrm{emg}}, m_{\mathrm{eeg}}) for zz alongside every visual–action pair, making both p(m∣o)p(m \mid o) and p(a∣o,m)p(a \mid o, m) supervisable from data. The corpus comprises 12,500 hours across 38 scene classes, collected at 6 sites. It is structured into three tiers (Figure 2a), one per proxy:

Figure 2. HumanCOIN in one view. (a) The three tiers and how their hours split: 12,500 h in total, at 10,000 / 2,000 / 500 h (80 / 16 / 4%) for L1 / L2 / L3, drawn on a log axis; solid segments are unscripted (open) recording, hatched segments scripted or bench-controlled. (b) Episode length and (c) task depth: stacked kernel densities of episode duration and of subtasks per episode, with the peak and the mean marked. (d) All 38 scene classes, grouped into the 5 zones and a residual class, with the tier split of each class’s hours. (e) The task verbs: the upper panel ranks each tier’s occurrences against vocabulary rank, and the lower panel gives the thirty most frequent verbs on a log occurrence axis. (f) The 635 object classes, ranked by frequency, with the thirty most frequent below, again on a log axis. Every stacked bar is split by tier.

  • L1, Attention tier (10,000 h). Proxy mgazem_{\mathrm{gaze}}: multiview egocentric RGB with inferred gaze, recording the output of visual orienting decisions.

  • L2, Exertion tier (2,000 h). Proxy memgm_{\mathrm{emg}}: bilateral forearm surface EMG, recording the peripheral projection of motor commands; the signal envelope reflects muscle activity from which grip force and load may be inferred before object motion.

  • L3, Cognitive tier (500 h). Proxy meegm_{\mathrm{eeg}}: scalp EEG, recording movement preparation potentials and error-related potentials that precede observable motion.

The tiers are nested in instrumentation: each inherits every proxy of the tier below and takes on one more, each proxy being a suite of instruments rather than a single device. Depth is a design axis, not a separate collection: the families that survive the same canonical filters at every tier recur at all three measurement depths on a shared time base. Scene, objects, posture, and protocol nonetheless move with depth, so a cross-tier comparison is task-matched rather than condition-matched; § 3.1 states what the shared families hold fixed and what they leave open. The three proxies are recorded by the three tiers, and the action aa itself is captured by the egocentric video every tier carries: zz remains latent, but mm is recorded rather than marginalised. Beyond the tier-specific modalities, the dataset provides depth maps and 3D point clouds reconstructed from egocentric views, 6-DoF head and wrist pose with finger kinematics, and step-level action–object annotations, all registered to a single coordinate frame and time axis. L3 serves as a multimodal reference in its controlled setting: there, mgazem_{\mathrm{gaze}}, head and wrist pose, and finger kinematics are measured directly with dedicated instruments, while their counterparts at L1, at L2, and in the open L3 subset are predicted from multiview video, aligning all three tiers in semantics, coordinate frame, and time base so that the controlled measurements provide per-frame reference targets for shallow-tier predictors.

Relative to existing corpora, HumanCOIN differs not in scale but in coverage: three proxy depths of the intent pathway under one task set and one time base, making the generative process behind each visual–action pair available for modelling rather than only for imitation. The dataset is released under a research licence (CC BY-NC 4.0) together with acquisition software, a data browser and downloader, and a ready-to-use client library. Our contributions are threefold:

  • A proxy-grounded acquisition paradigm. Three wearable proxies of the intent pathway are recorded alongside first-person manipulation and aligned to a common time base, making measurement depth a design axis (§ 4).

  • A large-scale intent-pathway dataset. The corpus comprises 12,500 hours over 38 scene classes, organised into three tiers, each contributing one proxy of the pathway, sharing a common task ontology and time base (§ 3).

  • An open ecosystem for the community. Acquisition software, a data browser and downloader, and an out-of-the-box client library are released with the corpus (§ 4.3).

Related Work

Egocentric Vision Corpora

Egocentric video has been scaled along two axes, hours and annotation density. Ego4D [1] collected 3,670 hours from 931 participants in nine countries and defined benchmarks for episodic memory, forecasting, and social interaction. EPIC-KITCHENS [32] traded breadth for density, annotating cooking with verb–noun segments; its hundred-hour extension, EPIC-KITCHENS-100 [2], added 89K action segments. EGTEA

Gaze+ [33] recorded eye movements over 28 hours of cooking and turned attention into a label. Ego-Exo4D [34] added a synchronized third-person view: 1,286 hours from 740 participants, annotated with skills, keysteps, and proficiency.

Assembly101 [3] instrumented tabletop assembly with 4,321 multi-view videos; ARCTIC [4] captured bimanual manipulation of articulated objects with 2.1M frames and 3D hand–object annotations; HOI4D [35] recorded 2.4M RGB-D frames of category-level hand–object interaction; and HoloAssist [36] annotated 166 hours of human–assistant collaboration, including mistakes and their corrections.

A newer generation raises the bar on geometry and duration: Nymeria [37] captures multimodal daily motion with full-body and hand tracking, HD-EPIC [38] adds 3D object and gaze grounding to kitchen annotation, EgoDex [5] records 829 hours of manipulation with on-device 3D hand tracking, and EgoLife [39] follows six participants for a week. All remain external observations: none records muscle or cortex during the act. EgoSuite-Open100K [6] has opened the first 10,000 hours of a planned 100,000-hour programme of egocentric manipulation, with 3D hand and body pose over 10,000 tasks and 500 environments, and Xperience-10M [7] releases 10,000 hours and 2.88B RGB frames of in-the-wild human experience with depth, camera pose, and captions. Both record the outside of the action at a scale no research corpus reaches. Neither records gaze, muscle, or cortex, and neither pairs the same task with a shallower version of itself.

Physiological and Neural Sensing

Surface EMG has long served as the interface between motor intent and machine. Farina et al. [29] review how motor-unit activity is extracted from the surface signal and used to decode intended movement and force, and Ninapro [40] standardised forearm recordings across subjects. Grip-force studies show that forearm muscles load to the anticipated object weight before the object leaves the surface [24, 25], encoding the operator’s prior over object properties as an observable EMG signal before any visual feedback is available. EgoEMG [41] pairs bilateral forearm EMG with egocentric video for hand pose estimation.

EEG-based interfaces have decoded motor preparation, error monitoring, and cognitive load for decades [30]. Kornhuber and Deecke identified the Bereitschaftspotential, a slow cortical build-up that precedes voluntary movement by hundreds of milliseconds [26], and Shibasaki and Hallett provide a systematic account of its neural basis [27]. Gehring et al. established the error-related negativity (ERN): cortex produces a discriminable response within milliseconds of an incorrect outcome, before any external feedback [28]. Taking EEG outside the laboratory is complicated by motion artefacts; the mobile brain/body imaging paradigm [42] and active-electrode hardware have made wearable acquisition feasible, though signal quality remains below that of seated recordings.

Gaze provides the earliest behavioural signature of intent: it reaches the target of a pointing movement before the hand does [43], and appearance-based trackers [44] made the signal recordable outside the laboratory; in everyday tasks fixations run ahead of the hand and mark the objects about to be acted on [21, 22]; and in object manipulation gaze reaches the contact point before the fingers close [23]. Pupil diameter adds a physiological readout of processing load [45]. When gaze and EEG are recorded together, fixation-related potentials lock cortical activity to the currently fixated object [46], showing that the three signals are mutually constraining rather than independent.

The neuroergonomics programme [47] argued that these measurements belong inside real tasks. Most studies nevertheless remain laboratory experiments: the task is designed to elicit a signal, scenes are controlled, participants few and trials short, so behaviour, muscle, and cortex are rarely recorded on one timeline. EgoBrain [31] pairs a 32-channel EEG headset with egocentric video over 61 hours from 40 participants, the first corpus to put cortex on the same timeline as first-person video; it is seated and scripted, however, covering 29 daily actions in one setting, with no gaze, no muscle, and no scene geometry. Large-scale synchronised recording of all three signals in naturalistic manipulation scenes has not been achieved.

Human Data for Robot Learning

Human video has long been used to teach robots, but the methods differ in what they inherit from the human. TCN [48] learned a self-supervised video representation for imitation, DexMV [49] brought human video and estimated hand pose to dexterous manipulation, R3M [50] pretrained a visual representation on Ego4D and transferred it to manipulation, VRB [51] extracted affordances from human videos as a representation for robot policies, and EgoMimic [8] scaled imitation learning from egocentric human video captured with a portable rig. These methods inherit appearance and affordance from the human, but not the signals that produced the action.

A second line introduces an explicit intermediate between oo and aa by choosing the future. UniPi [17] casts the policy as a video diffusion model and reads actions from the generated frames; GR-1 [18] jointly predicts future frames and actions in a transformer; VPP [19] and UVA [20] condition the policy on latent representations of predicted future observations. These models make the gap between oo and aa explicit, but the intermediate they supervise is a predicted future observation rather than the operator’s current state: the two diverge when an attempt fails or is abandoned, and neither grip force nor cortical preparation is recoverable from a predicted frame.

A third line adds modalities to the policy itself. Any2Policy [52] trains a modality-agnostic encoder that accepts text, audio, images, or point clouds in any combination and connects them to a policy head. Chain-of-Modality [53] prompts a vision-language model to chain modalities from human demonstration videos—images, armband EMG, and audio—into task plans with explicit control parameters. Play to the Score [54] fuses multiple sensory streams with stage-guided dynamic weighting during manipulation. These works show that extra modalities help a policy, and some already incorporate EMG at the policy end; none, however, is grounded in a corpus that systematically records the operator’s intent proxies across multiple depths under a shared task set and time base.

Data Distribution

HumanCOIN holds 12,500 hours of egocentric manipulation data, organised along three measurement depths: each tier corresponds to one observable proxy mgazem_{\mathrm{gaze}}, memgm_{\mathrm{emg}}, or meegm_{\mathrm{eeg}}, from the visual orienting signal at the pathway’s distal end, through the peripheral motor command, to cortical preparatory activity. The corpus spans 38 scene classes, 99 task verbs, and 635 object classes. The three tiers share a common task ontology and time base; scene and behavioural diversity unfold within each tier.

The Depth Axis

Existing egocentric corpora record the two ends of an operation: the observation oo gives the visual environment the operator inhabited, and the action aa gives the motion and effect of the hands—together, what happened. These two ends form the baseline modality set M0={o,a}\mathcal{M}_0 = \{o, a\} that all existing corpora share. What M0\mathcal{M}_0 does not answer is why a particular action occurred in a particular way at a particular moment.

We use intent as a working term for the operator’s task-relevant state, denoted zz. zz is neither a synonym for oo nor an early version of aa; it lies between perception and execution and shapes how an action unfolds. For any o,ao, a,

p(a∣o)=∫p(a∣o,z) p(z∣o) dz.(1)p(a \mid o) = \int p(a \mid o, z)\,p(z \mid o)\,\mathrm{d}z. \tag*{(1)}

Equation [1] writes p(a∣o)p(a \mid o) as an integral over zz: given oo, the distribution of aa is determined jointly by the posterior p(z∣o)p(z \mid o) and the action model p(a∣o,z)p(a \mid o, z). It supplies no directly observable quantity: zz itself is not observed. The question is not whether zz exists, but whether zz carries information about aa beyond oo, i.e., whether p(a∣o,z)p(a \mid o, z) differs from p(a∣o)p(a \mid o). We do not assume a⊥o∣za \perp o \mid z; the observation remains in the policy.

Because zz is latent, Equation (1) cannot be used directly for training or evaluation. The corpus records proxies for zz at three stages along the intent pathway, m=(mgaze,memg,meeg)m = (m_{\mathrm{gaze}}, m_{\mathrm{emg}}, m_{\mathrm{eeg}}), not zz itself. A proxy is a lossy measurement of zz:

mj=fj(zj,o)+εj,j∈{gaze,emg,eeg}.(2)m_j = f_j(z_j, o) + \varepsilon_j, \qquad j \in\{\mathrm{gaze}, \mathrm{emg}, \mathrm{eeg}\}. \tag*{(2)}

fjf_j reads out only the part of zjz_j that the sensor can reach, and εj\varepsilon_j is noise; fjf_j is not required to be invertible, nor mjm_j to contain all of zjz_j. The corpus therefore supports a proxy-conditioned policy p(a∣o,m)p(a \mid o,m), not p(a∣o,z)p(a \mid o,z). This distinction runs through the paper: zz is the explanatory frame, mm is what the data contain.

mm is not a sufficient statistic for zz. The three proxies differ in noise, latency, and coverage; they are not interchangeable, nor are they nested estimates of a single scalar. Together they cover three observable stations, from visual orienting to cortical preparation and monitoring, but they do not exhaust zz. We do not claim that these proxies identify zz; the limitation is discussed in § 6.

HumanCOIN asks one measurement question at each station:

  1. Q1. Where does gaze land before the hand departs, and how early does it carry information about the upcoming action?

  1. Q2. How does the effector respond to the motor command before the action occurs, and what does the EMG envelope carry about grasp type and load?

  1. Q3. What cortical activity precedes the motor command, and what does it carry about preparation and error monitoring?

The three tiers are not independent datasets but a ladder nested in instrumentation: each tier inherits every proxy of the tier below and takes on one more—gaze, EMG, EEG—and every proxy is a suite of instruments, whose members the controlled setting can upgrade from estimation to measurement. Their hours—10,000, 2,000, and 500—decrease with depth, reflecting the higher acquisition cost of deeper proxies.

  • L1, Attention tier (10,000 h). Fourteen heterogeneous headbands from thirteen vendors record multiview egocentric RGB; gaze is estimated from video and depth is reconstructed from the front stereo pair. L1 is collected in open scenes, where operators move freely and tasks are not scripted. The tier has neither EMG nor EEG.

  • L2, Exertion tier (2,000 h). The headband is fixed to a single model, and bilateral forearm surface EMG is added. Gaze is still estimated from video; scenes and objects become scripted, and tasks follow a task sheet. The tier has no EEG.

  • L3, Cognition tier (500 h). The cortical proxy is added, together with the L3 kit: two wrist views and a third-person view, and, in the controlled setting, the eye tracker, the trackers, and the gloves that measure gaze, hand pose, and finger kinematics directly. L3 has two settings: open uses a lightweight EEG configuration in natural scenes; controlled uses a high-density cap at a fixed desk, with imagery before execution. By the split used in Figure 2a, about 350 h of L3 are controlled and 150 h open; only the controlled part supplies direct measurements.

Thus the gaze, hand pose, and finger kinematics of L1, L2, and open L3 are estimates; the same quantities in the controlled L3 subset are measurements. That subset therefore provides the reference measurements yy against which the estimates mm can be compared frame by frame on the same episode. “L3 as reference” means that yy is a reference measurement for mm, not ground truth for zz.

The tiers are nested in instrumentation but not in distribution: scene, objects, posture, and protocol move from open to controlled with depth. A cross-tier comparison is therefore task-matched, not condition-matched. What the tiers share is a core of task families: those that survive both canonical filters, C1C_1 and C2C_2, and across them the task is the same at every depth while the staging is not—L1 instances are unguided, and the object set, step order, posture, and protocol are fixed only from L2 onward. The probe sets E2\mathcal{E}_2 and E3\mathcal{E}_3 are the increments and need not be shared. What they do not share is a condition: the corpus does not pair a family with one scene, posture, and protocol across tiers. A depth-only comparison—same scene, objects, posture, and task, only the instruments changed—is thus a construction the corpus admits, not a split it releases: one fixes a family and its staging, takes the episodes each tier recorded for it, and reports the residual differences rather than assuming them away.

Beyond modalities, the tiers follow an open-to-controlled gradient in task, scene, and acquisition paradigm: the deeper the tier, the more controlled and the narrower its coverage. Open means the task happens in a natural scene in the operator’s own way—objects occur as they occur, posture is unconstrained, recording runs free. Controlled means the scene, object set, posture, and acquisition protocol are fixed one by one, trading behavioural freedom for reproducibility. The design keeps one task comparable across depths while the corpus spans natural behaviour to controlled conditions.

In task, a family—a goal, an object set, and a dominant action sequence, coarse enough to recur at every tier—enters a tier only after canonical filtering, and each tier adds a probe set targeting the proxy it introduces:

T1=U,T2=C1(T1)∪E2,T3=C2(T2)∪E3,(3)\mathcal{T}_1 = \mathcal{U}, \quad\mathcal{T}_2 = \mathcal{C}_1(\mathcal{T}_1) \cup\mathcal{E}_2, \quad\mathcal{T}_3 = \mathcal{C}_2(\mathcal{T}_2) \cup\mathcal{E}_3, \tag*{(3)}

where Tk\mathcal{T}_k is the family set at tier kk, U\mathcal{U} the natural repertoire, and C1\mathcal{C}_1, C2\mathcal{C}_2 the two canonical filters: C1\mathcal{C}_1 fixes the object set, placement, and step order; C2\mathcal{C}_2 additionally fixes posture and phases (imagery before execution in the controlled setting). The probe sets are the tier’s increment: muscle-load tasks at L2—heavy objects, balancing, and tactile discrimination (E2\mathcal{E}_2)—place the effector at a loaded working point; high-cognition tasks at L3—instruction-following, long-horizon stacking, and error-prone insertion (E3\mathcal{E}_3)—impose complex multi-stage demands on the cortex.

Four dimensions move together from open to controlled: scene from fully open to staged to a fixed desk; objects from natural occurrence to a specified set with each item measured for colour, shape, size, material, and weight, so that object weight becomes a known quantity; posture from unconstrained to as still as possible, since electrode contact and eye tracking degrade with movement; and acquisition from free recording to phased protocols with an imagery stage, so that motor preparation is not buried by the movement that follows. Table 1 sets the four dimensions side by side. These four axes co-vary with depth, so they describe a design gradient rather than a controlled experiment; isolating depth requires the depth-only comparison described above.

L1L2L3
Sceneopenstagedstaged + desk
Objectsnaturalspecifiedspecified
Posturefreefreestill
Protocolfreephasedphased + imagery

Table 1. The four axes that separate the three tiers. Each tier inherits every modality class of the tier below; the four axes co-vary with depth, so the tiers form a design gradient rather than a controlled experiment.

Scope. HumanCOIN records proxies m=(mgaze,memg,meeg)m = (m_{\mathrm{gaze}}, m_{\mathrm{emg}}, m_{\mathrm{eeg}}) for the latent state zz, not zz itself. It provides time-aligned multimodal streams, L3 controlled reference measurements yy, and per-episode calibration and alignment.

Episode Duration and Task Stages

Episode duration determines how much context and how many steps a single sample carries, and it largely constrains the kind of training and evaluation the sample can support. The duration of HumanCOIN is not set uniformly; it varies systematically with tier, task type, and difficulty.

Duration distribution (Figure 2b). The distribution is unimodal with a long tail: the density peaks at 52 s and the mean sits at 88 s, with the shortest episodes near 10 s and an open tail beyond 5 min. The tiers differ systematically. L1 has the widest range and the heaviest tail, at a mean of 101 s; L2 and L3 are more concentrated, at 65 and 60 s. That gap belongs to the recording protocol rather than to the tasks: L1 is segmented after the fact from long continuous recordings, whereas L2 and L3 are scripted episodes with bounded sessions; electrode stability further shortens the deepest sessions. The gap is therefore not evidence about task difficulty, nor about depth as such.

Task stages (Figure 2c). Subtasks per episode count the stages a task is composed of rather than the temporal granularity of annotation. The count is highest at L1, at 11.0 per episode, where natural activity is split into many short steps; it falls to 8.3 at L2 and 7.9 at L3, where a fixed protocol groups the work into fewer, longer stages.

Overall, HumanCOIN shows clear tier differences along both duration and task structure: the shallow tiers hold long, loosely segmented, and diverse everyday activity, while the deep tiers concentrate on scripted, standardized tasks in which the same work is organized into fewer, longer stages. Duration characterizes the recording window, stage count characterizes task structure, and together they form the corpus’s spectrum from natural behaviour to complex controlled tasks.

Scene Distribution

The value of egocentric data depends largely on how far it generalizes in the real world, so the scene distribution should not be trimmed for convenience of collection. The HumanCOIN scene vocabulary holds 38 classes, grouped into 5 zones—residential (9), commercial (10), professional (5), industrial (6), and public (7)—plus 1 residual class. Figure 2d arranges them by zone and by tier, with the hours of each class stacked to show how the three tiers split the total; Appendix § 10 lists all 38 with their tier coverage.

A residential centre of gravity. Residential holds 4,521 hours; commercial, 2,588; professional, 1,982; industrial, 1,957; public, 1,211; and the residual, 242. The ordering is not incidental: manipulation is densest where people live and work, and the corpus follows that density rather than flattening it. Within each zone a few classes carry most of the hours—residential runs from the kitchen’s 1,175 hours down to the balcony’s 13, while the industrial zone is led by the factory workshop at 1,168 and the professional zone by the office at 1,129. Across the whole corpus the five largest classes take 39% of the hours, a concentration that mirrors how often those spaces are actually used.

Selective coverage by tier. L1 holds all 38 classes, L2 holds 21, and L3 holds 9, with the remaining 29 absent from L3; the 9 L3 classes account for 33.2% of L1 hours and 50.9% of L2. The scene set therefore contracts at each step—38 →\to 21 →\to 9, or 100% →\to 55 →\to 24% of the vocabulary—which is why the deep tier reads as a deliberate instrumented subsample rather than a miniature of the whole. Its nine classes are weighted toward the office, the dining room, the workroom, and the kitchen—spaces that admit a fixed desk, stable lighting, and a bounded session—while the controlled subset is narrower still, concentrated on the desk.

This is not a sampling bias but a direct consequence of the controlled setting: a high-density cap, an eye tracker, and a fixed desk require a fixed station, stable lighting, and a bounded session, and scenes such as the street, the shop floor, and the farm cannot physically accommodate them. The nine L3 classes are the classes with any L3 hours; the controlled subset is narrower still, concentrated on the desk. L1 is collected in open scenes, L2 in staged ones, and L3 spans both settings. The three tiers therefore differ in what they can record, and the scene distribution is a direct reflection of that difference.

Taken together, the scene distribution expresses a trade-off between diversity and record quality. From L1 to L3, coverage contracts from all 38 indoor and outdoor classes to a concentration on the 9 classes that can be instrumented. The result is a corpus that spans from natural behaviour to complex controlled tasks, with task families that recur across tiers observed at several measurement depths.

Task Distribution

A task entry is the unit of reuse: one captured segment, its ordered step sentences, and the objects it touches. The entries span cooking, assembly, cleaning, errands, and teaching, and differ in step count, object set, and manipulation difficulty; that variation is what allows the corpus to be split by domain, by horizon, or by difficulty instead of being used as one monolithic training set. Each step carries a task verb, the hand that performed it, and the objects it touches, so an entry can be read both as a plan and as a record of execution (Figure 2e). The upper tiers are, with few exceptions, subsets of the lower tiers: scenes are strictly nested (L3 ⊂\subset L2 ⊂\subset L1); objects and verbs are nested except for a small probe increment—five objects first added at L2 and six verbs first added at L3. Figure 2d–Figure 2f shows the resulting vocabulary split.

Action: 99 task verbs. A task verb names what a step accomplishes as a goal—wipe, open, fold—and is the unit a plan is written in. The vocabulary is open-ended in principle but compact in practice: 99 classes cover the step record, a median of 81 occurrences per class, and the ten most frequent take 60.0% of it. The head is everyday manipulation—wipe, open, fold, close, arrange—above a long tail of rarer verbs, so the same set serves both as a small label vocabulary and as a description fine-grained enough to distinguish what is being done. Behind each step sits a second, finer record: the motion verbs that carry it out—pick, place, move, grasp—46 classes over the motion record, 2.5 per step.

The tiers treat the vocabulary differently. L3 keeps 40 task verbs (40%), and those verbs cover 77.3% of L1 task occurrences. A protocol fixes the task set, so the verb list contracts to the tasks it scripts, while the steps that are recorded stay labelled in full: the deep tier is selective in vocabulary but not in the record.

Object: 635 classes. The object side is longer-tailed still: the ten most frequent classes take about 42.5% of the object occurrences, while the median class occurs only about 130 times in total, and the 46 most frequent classes already carry 80% of the record (Figure 2f). Coverage again separates the tiers: L1 holds 630 classes and L2 386, while L3 keeps 95 and leaves the other 540 absent from L3; those 95 still account for 82.0% of L1 occurrences and 97.3% of L2. L3 is a subset of L2; L2 adds a small probe set of objects absent from L1. The deep tier thus keeps 15% of the vocabulary and more than four fifths of the record—the same contraction seen in the scenes, and for the same reason. Appendix § 12 describes the measured object library—its size, weight, material and colour distributions—and how it is counted against this annotation vocabulary.

Taken together, all three vocabularies are long-tailed, and what the tiers trade is breadth against depth rather than vocabulary size: the shallow tiers spread the full vocabulary over everyday activity, while the deep tier contracts to the classes its protocol needs and concentrates the record there.

Inside an Episode

Figure 3 opens one L3 episode in full: the metric scene in Figure 3a, the camera views and depth map in Figure 3b, the annotation of the selected step in Figure 3c, and the task stages with the intent proxies of the same interval in Figure 3d. It is shown in the controlled L3 setting because that is where the body is instrumented: the gaze, the hands, and the wrists that the other tiers infer from video are measured here.

One controlled L3 episode in full

Figure 3. One controlled L3 episode in full, at t=27.6t = 27.6 s (frame 829/1126). (a) The metric point cloud, coloured by the head-camera RGB, with the head pose, both hand skeletons and their 6-DoF poses overlaid in one shared frame. (b) Eight synchronized camera views arranged two per row: the four head-mounted views, the two wrist views, the metric depth map, and the third-person view. (c) The annotation of the selected instance: the step sentence stack the tomato on the bread, the measured object attributes (name, colour, shape, material, size, weight), the atomic action grasp, and the working hand left. (d) The task stages and the three intent proxies of the same interval: gaze (xx, yy), eight-plus-eight-channel EMG, and sixteen of the 128 EEG channels, on a shared time axis with the current instant marked; the eye reaches the object before the hand and the muscle load rises before contact, so each trace leads the action it announces. The stages are imagery, place bread, add tomato, add lettuce, add patty, and top with bread.

Geometry. The point cloud is reconstructed from the two front views; the head, both wrists, and the finger joints are placed in one metric frame, so a grasp is located on the object rather than only in the image (Figure 3a). In the controlled L3 subset these quantities are measured: three trackers give the head and both wrists, and the gloves give the joints. At L1, at L2, and in the open L3 subset the same quantities are estimated from RGB, with each wrist root anchored in the scene frame through the depth map and the finger joints following in three dimensions. The frame is shared; the evidence behind it is not.

Views. Four egocentric views pair the two front lenses (left-front and right-front) with the left and right views, and at L3 two wrist views add the hands at work (Figure 3b). A depth map is reconstructed on demand from the front pair, and in the controlled setting a third-person view records the operator and the desk from outside. The frames are stored as recorded, with no annotation drawn over the pixels.

Annotation. The selected instance is shown in Figure 3c: each step carries a short description, a task verb, the hand that executed it, and the objects it touched, and at L2 and L3 the object record adds a photo, a hex colour, a shape, a material, a weight, and a size given in the parameters its shape calls for (radius and height for a cylinder, for instance). For the step in the panel the atomic action is grasp and the working hand left.

Intent proxies. Each episode is cut into stages, shown as the strip at the top of Figure 3d. The three traces beneath the strip are the proxies for the state zz that the action is about to express, and each is read as a lead or a lag against the action it announces. Gaze is the eye’s (x,y)(x,y) position, measured by the eye tracker in the pixels of its scene camera in the controlled L3 subset and estimated from the rectified head view at L1, at L2, and in the open L3 subset: the eye reaches the object before the hand does, so the trace externalises the target of the next reach—a fixation is a flat stretch, a saccade a near-vertical step, and the moment the eye settles marks what the hand is about to act on (mgazem_{\mathrm{gaze}}). EMG is eight envelope traces per forearm, flat at rest and rising before contact: the forearm loads grip force to the weight the hand expects, so a burst records the force the operator intends to apply, not only the force applied when the fingers close (memgm_{\mathrm{emg}}). EEG is a stack of channel traces, three orders of magnitude below EMG and drawn on its own scale: the slow build before the movement and the sharp response after an error are the cortical preparation and the outcome check, read as deviations from the ongoing rhythm rather than as spikes (meegm_{\mathrm{eeg}}). On one clock, the three answer what the instant means—where the operator is about to reach, how hard they expect to grip, and what was decided before the hand moved.

An episode is therefore one record aligned to a common time base and in a single frame: the views and the point cloud give the scene, the poses, joint angles, and gaze give the body’s place in it, EMG and EEG give the exertion and the cortical preparation behind the movement, and the annotation gives each step its place in the task.

Comparison with Existing Datasets

Table 2 compares a representative set of these corpora on what they record rather than on scale: it is a comparison, not a census, and the corpora that share an axis with a row here are discussed above rather than tabulated. All of them observe the task from the head, and neither a third-person view nor a wrist view (Ego-Exo4D, Nymeria, EgoSuite) adds more than the visible outcome of manipulation. Only EgoBrain carries a neural signal, and it carries just one (EEG); gaze is the only human signal shared by several corpora—Ego4D on a subset, Ego-Exo4D in 99% of takes, EGTEA and Nymeria throughout—and always in isolation, so that where the eye went and how hard the hand pulled have never been observed at the same moment. HumanCOIN records the core families at three instrumentation depths, which is what lets “measured in the controlled L3 setting, predicted from video elsewhere” hold for those families, and a shallow-tier estimator be scored against the dataset’s own reference measurements rather than an external benchmark.

Table 2: Representative prior egocentric datasets against HumanCOIN. ✓ measured, P predicted from video, R reconstructed from video, ✓* subset only, ✓/P measured where instrumented and predicted elsewhere, a cross not provided; 3D is a reconstructed depth or point cloud, Fingers joint-level hand tracking. Cameras counts per recording: head a head-mounted first-person camera, wrist a wrist-mounted one, exo a third-person camera. Frames at 30 fps where only a duration is reported, and the HumanCOIN count is master frames, one per 30 Hz tick. †First 10,000 h open; 2–6 head means the number of head-camera lenses varies between batches, always with a front stereo pair. Shaded rows are HumanCOIN and its three tiers.

DatasetHoursFramesCameras3DFingersGazeEMGEEGSubtask anno.Object anno.
Ego4D [1]3,670≈396M1 head (+224 h multi-cam)✓*✗✓*✗✗✓*✗
Ego-Exo4D [34]1,286≈139M1 head, 4–5 exoR✓*✓✗✗✓✗
EPIC-KITCHENS [2]10020M1 head✗✗✗✗✗✓✗
EGTEA Gaze+ [33]282.5M1 head✗✗✓✗✗✓✗
Nymeria [37]300201.2M1 head, 2 wrist, exoR✗✓✗✗✓✗
EgoDex [5]82990M1 head✗✓✗✗✗✓✗
EgoSuite-Open100K [6]10,000†1.08B1 head, 1 wrist*R✓✗✗✗✓✗
Xperience-10M [7]10,0002.88B6 headR✓✗✗✗✓✗
EgoBrain [31]616.6M1 head✗✗✗✗✓✓✗
HumanCOIN12,5001.35B2–6 head, 2 wrist*, exo*R✓ / P✓ / P✓*✓*✓✓
L1 · Attention10,0001.08B2–6 headRPP✗✗✓✗
L2 · Exertion2,000216M4 headRPP✓✗✓✓
L3 · Cognition50054M4 head, 2 wrist, exo*R✓*✓*✓✓✓✓

Table 2.

EgoBrain is a complement rather than a replacement. It has already shown that cortex can be recorded on the first-person timeline, but does so in a controlled, seated, scripted setting, trading breadth for signal density: 61 hours, 40 participants, 32 channels, 29 daily actions. HumanCOIN takes the same premise into naturalistic scenes and trades action variety for volume, holding 500 hours at L3. What it adds is two things EgoBrain does not attempt: gaze and surface EMG on a common time base, so that three proxies can be compared at one instant, and scene geometry with hand kinematics, so that an action lands on an object rather than staying inside the image.

Subtask annotation is already common under other names, but object annotation is not: elsewhere an object is a name, while at L2 and L3 every instance is measured for shape, material, size, and weight. That is a difference in capability, not in labelling style—it lets an EMG envelope be read against a measured object weight rather than an estimated one, so that the anticipation of grip force before contact can be inferred from muscle activity rather than only assumed; grip force itself is not recorded.

Hardware Configuration

The intent proxies summarized above do not appear in a recording by themselves. Each becomes a record only through an instrument that converts a physical quantity into a measurable signal, so the kit is a set of measurement proxies: one class of instrument per component of zz, each chosen for what it can physically reach. Every instrument keeps its own clock, its own native sampling rate, and its own coordinate frame. This section describes what is worn at each tier, the two settings in which the kit is deployed, the two-phase protocol that organizes a controlled session, and how the resulting streams are stored.

The Instrument Set

This subsection is organised by proxy rather than by device: the instruments a tier adds are the reading apparatus for the intent proxy that tier measures, and the criterion for choosing one is single—whether it can physically reach the quantity without altering the behaviour being measured. The three proxies sit differently against that criterion, so the hardware differs, and the order of depth follows from it.

L1, Attention—the mgazem_{\mathrm{gaze}} proxy. Visual orienting projects onto the external scene as light, so a headband that never touches the operator can record the scene from which gaze is inferred. L1 is not a single device but a collection layer assembled from fourteen third-party headbands supplied by thirteen vendors. All fourteen provide a forward stereo pair (two of them as a side-by-side image split into left and right, and one through a pair of its positioning cameras rather than a dedicated front pair). Seven add fisheye or side views for peripheral coverage. Native resolutions range from 512×512512 \times512 (rectified) to 3840×12003840 \times1200, and video rates from 20 to 60 Hz; Table 6 lists the units.

L2, Exertion—the memgm_{\mathrm{emg}} proxy. Action potentials raised at the muscle fibres reach the skin surface by volume conduction, so eight electrodes per forearm over the flexor and extensor bellies suffice, sampled at 2000 Hz. The cost is contact: impedance degrades with sweat and movement. From L2 onward the headband is fixed at four lenses at 1920×12001920 \times1200, which distinguishes it from the heterogeneous L1 units.

L3, Cognition—the meegm_{\mathrm{eeg}} proxy. Preparatory cortical activity crosses the skull only as a field potential, small in amplitude and sensitive to movement. The headband and the EMG are identical to L2, and two 640×480640 \times480 wrist cameras at 30 fps add the near-hand view the head cannot see. EEG comes in two configurations: an 8-channel cap connected through the headband computer, for the open setting in which the operator moves freely; and a 32- or 128-channel cap connected through the fixed bench host, for the controlled setting in which it is joined by the binocular eye tracker, the three 6-DoF trackers, the 20-node gloves, and the third-person RGB-D view.

The correspondence between channel count and connection is not incidental: the denser the cap, the higher the data rate and the tighter the electrode stability required, and the more the record depends on a fixed station and a still posture. Hence the distinction this subsection turns on. The same proxy is obtained differently under different conditions: gaze, head and wrist pose, and finger kinematics are measured directly, with dedicated instruments, only within the controlled L3 subset; at L1, at L2, and within the open L3 subset, the same quantities are inferred from headband video. “L3 as reference” refers to that controlled record. What the tiers add, instrument by instrument, is the upgrade of an estimate into a measurement, and the preconditions of that upgrade tighten with depth—which is why the deepest tier is also the narrowest.

Two invariants hold across the kit. First, no instrument is daisy-chained through another: a single host receives every stream and stamps arrival offsets, so each device keeps its own clock and native rate, and a failure or a dropped frame in one stream cannot contaminate the others; that host is the headband computer in the open setting and the bench host in the controlled one. Second, phase and run transitions are driven through a parallel-port box into the EEG amplifier as well, so protocol events are stamped on the amplifier’s clock and not only on the host’s. Every camera is stored twice, raw and rectified—the depth view rectified only—and the calibration ships with every episode. Poses share one frame throughout: xx points left, yy up, and zz forward, with millimetres for translation and degrees for joint angles; the alignment and calibration pipeline is described in § 5.1. Figure 4(a)–(Figure 4c) show the three kits in tier order: the headband at L1, one armband on each forearm at L2, and at L3 the same kit with the EEG cap and a wrist camera on each forearm. Only the devices a tier adds are numbered on its operator; the eye tracker, the gloves and the head and wrist trackers are controlled-setting only, which the dashed numbers mark. The full modality matrix is in Appendix § 13, and the fourteen L1 units are audited in Appendix § 14.

Figure 4. Instrumentation by tier. (a) The L1 kit: a four-camera headband. (b) The L2 kit adds two surface-EMG armbands worn on the forearm. (c) The L3 kit adds the two wrist cameras and the EEG cap, and, in the controlled setting only, the eye tracker, the motion-capture gloves, and three 6-DoF trackers (head and both wrists). The devices each tier introduces stand in a row beneath its operator, numbered as on the body, and only the devices a tier adds are marked on that operator; solid numbers are hardware worn in both settings, dashed numbers the hardware used in the controlled setting only. Faces are blurred for anonymity.

Settings and Protocol

A tier’s collection is governed by two requirements: how much of the everyday scene it must preserve, and how much signal fidelity its instrumentation demands. The two pull in opposite directions, and the corpus resolves them tier by tier.

Two settings. L1 and L2 privilege mobility over signal depth. Every instrument they add is wearable and connects to a headband-mounted computer, so the operator moves through everyday environments—standing, turning, walking, and gesturing—without a fixed bench or cable constraints. This is the open setting: L1 and L2 are collected in it throughout, and L3 also uses it, with a lighter configuration, trading EEG density for scene coverage. Open here names the acquisition setting, not the task: an open-setting session can still use a scripted task sheet and a staged scene.

L3 is where the trade-off reverses. Dense electrophysiology requires a stable posture: electrode contact and impedance degrade with movement, and the denser the recording, the less it tolerates motion. eye tracker, the gloves, and the wrist trackers load the same platform, so acquisition calls for a bench that does not move. Sessions run in short blocks, with a re-baseline between blocks to limit drift. L3 is collected under both settings: in the controlled setting the amplifier and the host remain on a dedicated desk and the operator holds still, and the session runs imagery then execution; in the open setting the kit runs on the headband computer in everyday scenes, and the session runs execution only.

Imagery, then execution. The controlled setting also carries the paradigm that makes the EEG informative, built around the high-density record. Imagery is confined to this setting for two reasons: its boundaries are delimited by phase markers that the experimenter records during the session, and the preparatory signal is low in amplitude and spatially structured, so only the dense array separates it from noise. Preparation and motor imagery are brief and are easily masked by the movement that follows, so the session separates imagery from execution (Figure 5). In the imagery phase (Figure 5a) the screen turns blue: the operator faces it, sits still, and rehearses the task sequence without moving, so that motor imagery and preparatory EEG are recorded free of movement artefact; gaze is recorded as well, so that the scanpath of the plan can be read off. In the execution phase (Figure 5b) the screen turns green: the same operator manipulates the objects on the desk, and every modality is recorded, so that the EEG and the other streams can be read against the actual movement. Every transition is also written to the record as a marker—run start, imagine start, execute start, and run end—so that the phases can be cut out of any stream on a common time base rather than by hand.

Two cued phases of a controlled session

Figure 5. The two cued phases of a controlled session (e.g., pick up the cup). (a) Imagery (blue screen): the operator sits still, hands on the desk, and rehearses the task with only EEG and gaze recorded. (b) Execution (green screen): the same operator manipulates the objects, head held as still as the grasp allows. The eyes move in both phases, the hands only in the second.

Each controlled episode therefore comprises three segments, not two cues alone: preparation while the operator settles, imagery of the task sequence, and execution of that sequence. Because the first two are acquired while the body is still, the EEG records the task as it is planned rather than as it is performed: the step sequence and the action each step will require, with no movement to mask them. The high-density record exists for that pairing: the same task as a plan, acquired while the body is still, and as a performance seconds later. Open settings (L1, L2, and L3 with the lighter configuration) record the execution segment alone.

Collection procedure. L1 is recorded without guidance: operators are asked only to carry out the activity as they normally would. L2 and L3 use task sheets: the operator arranges the objects as drawn and follows the step order marked on the sheet, so that the same task is staged identically across sessions. At L3, tasks are drawn in random order rather than blocked, which sustains engagement and limits drift of the EEG towards drowsiness. Operators also rotate between scenes at a fixed interval, so that no operator works a single scene for an entire shift and no scene is shaped by an individual’s habits.

All operators are trained to a single standard before collection: how to wear each device, how to run a session, and how to record it. L3 adds two requirements: operators are trained to keep the head still during acquisition and are responsible for seating and maintaining the electrodes so that contact quality stays within range; participants assigned to L3 are also screened so that the EEG is clear and stable. Each hour of collection is followed by fifteen minutes of rest, to sustain attention and energy.

Data Storage

Streams are stored as recorded and linked by reference, never merged into a single file. Each device keeps its own clock, its own native sampling rate, and its own coordinate frame; forcing them into one stream would discard the timing the corpus exists to keep. Synchronization is therefore a property the data declares rather than one it enforces: no resampling and no alignment happen at write time, and the corpus is written as many asynchronous files, each at its own rate.

At write time, every episode keeps one Parquet table per modality (Figure 6a), one file per observation stream: the camera views, EEG, gaze, hand pose, glove skeleton, three device poses (head and both wrists), head and wrist IMUs, and wrist EMG. videos/ holds one encoded video per camera view, calibrations/ holds the rectification parameters and the manifest, annotation/ the structured annotation and the QC verdict, and meta/ the episode metadata with its alignment and statistics reports. Every sample is stamped relative to the start of the recording, and that difference is stored as given, not snapped to a master frame, so non-uniform intervals and device jitter survive into the file. The complete directory layout is in Appendix § 15.

Figure 6. One episode on disk, and how it is read. (a) One Parquet file per stream, each a timestamp column plus data columns, beside a master index. (b) Timestamps are relative to the start of the recording, so no feature sits on a fixed grid; a query fixes a master rate and resolves each feature from its own timestamps. Every camera appears twice under videos/, once raw and once rectified, except the depth view, which is rectified only.

Quality is enforced on the recording, not on the downstream analysis. Every session is checked stream by stream once it is written—dropped or out-of-order samples, black or frozen video, dead channels, impossible value jumps, a device missing from the set—and a session that fails is recaptured rather than repaired, so a released episode never mixes a good half with a patched one. The verdicts travel with the episode in its metadata; Appendix § 16 lists the checks, their thresholds, and what each level costs.

Reading is fully decoupled from writing. A separate master-index file is sampled at a fixed rate, and the reader chooses that rate (30 fps by default); a master frame then indexes every modality through its own timestamps, resolved by nearest sample, interpolation, or all samples in a window such as (−1/FPS,0](-1/\mathrm{FPS}, 0] (Figure 6b). The format is a modification of LeRobot [55] and remains compatible with it: installing pip install humancoin is enough to preview, download, and use the data, and the acquisition software and the documentation are released alongside it.1

Data Processing

Processing runs per stream, because the noise and the failure modes of each modality are set by its own measurement physics; the streams meet only at the end (Figure 7). Each stream is reduced to a timestamp, a unit, a declared frame, and an alignment report, and the five groups follow the dependency order of the data—synchronization and calibration, geometry and motion, gaze, electrophysiology—with annotation last, since it consumes the others. Chains are fixed, their parameters are logged per episode, and a failing step is flagged rather than filled, so any run can be replayed.

Figure 7. Processing pipeline. Each stream enters the group that handles its physics: synchronization and calibration (clock offsets, camera intrinsics, eye calibration); geometry and motion (stereo depth from FoundationStereo, point cloud, 6-DoF poses with finger joints, hand reconstruction from the rectified front view); gaze (acquisition at L3, prediction at L1 and L2); and electrophysiology (band-pass filtering, notch, bad-channel interpolation). The groups then align to a common time base and pass through annotation before an episode is written.

Synchronization and Calibration

We use soft synchronization. The recorder stamps every sample relative to the start of the recording, and the sync host records arrival offsets for the streams that share a transport; the two together place each sample on a common time base without resampling. A per-stream audit of released L3 episodes is reported in Appendix § 17: the high-rate streams align below one sample period, while the 30 Hz cameras are frame-quantized. Calibration (intrinsics and distortion for every camera, the tracker-to-frame transform, and the eye-tracker calibration) ships with the episode in calibrations/, alongside both encodings of every camera, so a reader can re-derive the geometry rather than accept the undistortion applied at write time. Because neither offsets nor calibrations are applied at write time, a reader can align at any custom rate and choose nearest sample, interpolation, or a window. Residuals between the chosen rate and the samples actually read are written to the per-episode alignment report, so a downstream analysis can audit the timing it inherited.

Geometry and Motion

Depth is built from RGB. The left-front and right-front lenses are hardware-timestamp synchronized, so every stereo pair comes from the same instant; the pair, together with the camera parameters, is passed to FoundationStereo [56], which returns a depth map stored with the intrinsics that produced it. Point clouds are formed per frame by projecting depth through those intrinsics and are subsampled to a fixed budget.

Motion follows the same route. In the controlled L3 subset three 6-DoF trackers give the head and both wrists directly (~800 Hz, millimetres, on one shared frame), and the gloves give 20 joint angles per hand (~800 Hz, degrees). At L1, at L2, and in the open L3 subset both hands are reconstructed from video over each continuous segment: the model builds on ACE-Ego-Hand [57] and takes clean-latent features from Wan2.2-Fun-5B-Control [58, 59], and a bidirectional spatiotemporal decoder returns, for every frame, the two wrists and the finger joints in the camera frame, MANO shape and articulation, wrist translation, the intrinsics it assumed, and two confidence fields per hand—whether it is present, and how much of it is visible.2

Before reconstruction, the main view of every unit is rectified to a common pinhole domain, the intrinsics are updated after cropping and resizing, and the analytic rays are generated from the updated intrinsics; a unit’s declared calibration is used where it exists, and the pinhole parameters are predicted from the features where it does not. The intrinsics released with the track are those of this domain, not those of the raw lens. Overlapping windows are fused at their source timestamps: rotations combine in rotation space, one shape per hand and one camera per segment are fixed, and the geometry is regenerated, so that shape and scale do not drift as the visible part of the hand changes—what a user reads is a hand that keeps its size through a grasp. The output is already three-dimensional in the camera frame, so only the wrist root has to be anchored: its pixel is looked up in the depth map and back-projected, and the finger joints, three-dimensional relative to that root, ride along. The depth map remains the metric reference for the scene and the objects. In the controlled L3 subset the same fields are measured rather than inferred, which is what makes that record the reference against which the reconstruction is scored. All poses (head, both hands, hand kinematics) and the environment point cloud are mapped into one unified coordinate frame.

Gaze

Gaze is obtained by acquisition and by estimation. Both routes write the same fields—a timestamp and a pixel coordinate—in the pixels of the image each route reads: the scene camera of the eye tracker in the controlled L3 subset, the rectified head view at L1, at L2, and in the open L3 subset.

In the controlled L3 subset the eye tracker is calibrated per participant and per session and logs at 200 Hz; its coordinates are pixels of the scene-camera image, whose video runs at 30 fps. The calibration that produced each record—the intrinsics and distortion of that camera, with the transform that carries a ray into the head frame—ships with the episode, so a pixel can be back-projected and placed in the head frame. Because L3 records a reference measurement, it also serves as the test bed for the estimator used at L1 and L2.

Outside that subset there is no eye tracker—at L1, at L2, and in the open L3 subset—and the gaze point is predicted by a pipeline built on EgoM2P [61] that runs from video to coordinates. It reads the unit’s rectified left-front view, where the unit provides one—left not because it is more accurate, but because its rectification, depth, hands, and annotation all use that view, so cross-modal alignment needs no extra transform—under the model’s fixed input specification: a 2 s window, 16 frames, and a 256×256256 \times256 input. The gaze points of a window are predicted and then linearly resampled to the window’s frame count, so that every frame carries exactly one row; a tail shorter than a window is covered by an additional window aligned to the end of the video, so that the prediction is as long as the video. The output is fixed at 30 Hz and is not interpolated to L3’s 200 Hz: the model’s positional encoding fixes 30 tokens per second, and interpolation would only manufacture precision. The difference ships with the column, and a user who wants another rate can resample. Coordinates are pixels of the video: the model works inside a fixed 256×256256 \times256 centre crop, and the pipeline maps its output back, so that a released coordinate indexes the frame a user has rather than a position inside the model’s input. The prediction is written as a table isomorphic to L3’s but registered under a separate feature name, so that measurement and estimate cannot be confused.

Electrophysiology

Both streams are cleaned with a fixed chain. EMG is filtered in zero phase and reduced to an envelope. The trace is mirror-padded at both ends, band-passed between 20 and 450 Hz with second- and fourth-order Butterworth sections applied forward and backward, and comb-notched at 50 Hz with a 2 Hz bandwidth, with periodic extension and period alignment at the edges so that the notch does not ring there. The cleaned trace is rectified and reduced to one envelope per channel, and the raw trace is kept beside it as *_emg_raw. Appendix § 18 lists the orders, cut-offs, and padding steps.

EEG passes through seven. The channel is de-meaned, band-passed between 0.5 and 100 Hz with the high cut held below Nyquist, and notched at 50 Hz, which keeps the delta-to-gamma range in which motorimagery and error-related potentials live. Bad channels are flagged by five joint rules—flat, electrode contact, bridging between neighbours, decorrelation from the rest of the montage, and high-frequency noise—and a session passes only when fewer than 30% of its channels are flagged. An infomax ICA fitted once on the clean segments of a session removes ocular components, selected by correlation with the frontal channels, and muscle components, selected by their power above 30 Hz. Flagged channels are then interpolated by spherical spline over the retained ones, and the cleaned trace is re-referenced to their average. The stages are released as separate tables—eeg_raw as acquired, eeg filtered, and eeg_repair after interpolation and component removal. Appendix § 18 gives the rule thresholds, the component criteria, and the electrode positions that lack a conventional coordinate.

Motion cannot be removed from a recording in which the operator moves, and even the deepest tier is stationary only by protocol (§ 4.2). The cleaned EEG is therefore signal plus residual motion noise: we release the metrics that bound it—impedance pass rate, share of interpolated channels, and share of samples marked artefactual—and leave the choice of a stricter rejection threshold to the user. EMG is recorded at 2000 Hz in millivolts, eight channels per arm; EEG at 1000 Hz over 128 saline channels or at 250 Hz over 32 or 8 gel channels, in microvolts, with every channel named by brain region and its impedance logged in kilohms as a quality input.

Annotation and Segmentation

Every episode is annotated at the level of task steps (Figure 3c): temporally bounded segments carrying the step sentence, the action it names, the hand in use, and every object the sentence reaches. Annotation is produced with Argus [62], an open-source harness for dense annotation of robot-learning data, running on top of Qwen3.8-27B [63]. Argus reasons jointly over video, state, and sensor evidence to produce dense labels: action phases, subgoals, key events, outcomes, operator mistakes and recoveries, and object-state changes, each with an execution and outcome score and with the evidence and confidence behind it. It exports the labels as JSON and as LeRobot language columns, and a human reviewer checks every export before it enters the release. Objects are not mere name strings: for L2 and L3 every object recorded in an episode’s object list was measured for size and weight, and the measurement is linked to the object instances appearing in each trajectory. Collective references (“the cups”) are expanded to their member objects, and hand-in-use is read from wrist-tracker displacement inside the step window when no explicit label exists.

L1 needs a different route. Because passive collection does not intervene, a raw L1 recording is one continuous stream that lasts hours, and episodes are cut after the fact: frames and the available annotation are passed to a VLM, which proposes episode boundaries and step segments, and a human reviewer confirms or adjusts every boundary before release. This is why L1 episodes vary widely in length while controlled L3 sessions are scripted, and why an L1 boundary carries the uncertainty of a segmentation rather than the certainty of a cue.

Discussion

An egocentric corpus pairs an observation with an action and so answers what happened; why the action took that form belongs to the intent that preceded it, which is often settled before execution. HumanCOIN collects that intent, recording proxies for it at three signal depths. Whether it holds up depends on work we have not done; we state its limitations first, its openings second.

Limitations

The premise is untested. That intent sits between observation and action and carries information the observation does not is untested here: the corpus only makes it testable, and policy training, lead-time baselines, and cross-site quality gates lie outside its scope. Nor can the opposite be excluded: intent may be redundant with observation, since gaze is predictable from the scene and preparatory EEG may follow the kinematics. What fails then is not the concept but the proxies; the negative result turns “is intent redundant?” from a position into a measurement.

Coverage and representativeness. L3 accounts for 4% of the hours (500 h); its controlled subset is about 350 h (2.8%) and is the part concentrated on the desk, and electrode stability bounds how long the densest episodes can run. The scene and task mix is not a random sample of manipulation. L1 is assembled from fourteen heterogeneous units whose camera counts, resolutions, calibration, and declared frames differ (Appendix § 14), so how much of an L1 statistic reflects behaviour rather than the device mix cannot yet be separated.

Annotation and comparability. Step boundaries and object attributes are human judgements—L1 boundaries from automatic segmentation with human review, attributes measured only at L2 and L3. Hand motion is estimated at L1, at L2, and in the open L3 subset and measured in the controlled L3 subset, so the estimated part carries a reconstruction error. Comparability holds only in part: timestamps, units, frames, and calibration are released as each source declares them and the declarations are uneven, while quality metrics such as the impedance pass rate exist only at L3.

Long tails. The vocabularies are long-tailed: rare classes have few samples within a tier, cross-tier comparisons are unevenly powered, and the L3 share limits the deep tier’s rare classes, so “the deep tier is more concentrated” holds only qualitatively.

Research Directions

None of these questions can be answered on (o,a)(o, a) data alone.

Testing the mediation. One test compares p(a∣o,m)p(a \mid o, m) with p(a∣o)p(a \mid o): if the proxies add nothing, the latter is no worse. Another is a cross-tier ablation; L1 and L3 share the task ontology and differ in depth, so an L1-trained model applied to L3 mixes distribution shift with the proxy difference. The imagery phase supplies a third: it records a plan without its execution, so a model can be asked to read the plan without action evidence.

Evaluating before the fact. With only oo and aa a corpus scores after the fact; with intent one can ask before: how the action would change under a different intent, whether the decision stays readable when no action occurs, and whether a model predicts the action or the intent behind it. Such answers do not depend on success, so they retain signal where failures are rare.

Transport. Three depths, several sites, many operators, and fourteen units make transport measurable: which proxies stay calibrated across tiers, which labels stay stable across annotators, and whether any component is independent of the sensor.

Applications. Intent-conditioned policies, earlier failure prediction, corrections with less supervision, and an intent interface for human–robot collaboration follow once zz enters the model: the pay-off of the chain, not the test of whether it holds.

The corpus succeeds if it makes these questions falsifiable; a negative answer counts as much as a positive one.

Conclusion

An egocentric manipulation corpus usually records two things: the observation oo and the action aa. It records what happened, not what was decided—and the decision is what the action is answerable to. HumanCOIN takes that middle term, the intent zz, as the object of collection: one task ontology and one time base, with proxies for zz recorded at three signal depths—gaze for attention, EMG for execution, EEG for decision—and three tiers nested in instrumentation but not in distribution: passive and open at L1, active and open at L2, active and controlled at L3. What results is an instrument for testing intent rather than a larger behavioural dataset: task families that recur across tiers observed at several depths, which makes three questions measurable—where intent becomes readable, whether it mediates the action beyond the observation, and whether it transports across tiers.

We do not claim that these questions are answered; the value of the corpus is that they can be falsified, including by the negative result that intent is redundant with observation. The data, the acquisition software, and the documentation are released openly for research use.

Data Statement

All participants gave written informed consent before any recording and could withdraw at any time without penalty. Episodes are released only in de-identified form: no personally identifying information is included, faces are blurred in every released view, and all tiers—L1, L2, and L3—share the same consent and de-identification protocol. Task protocols and instrumentation differ (§ 4.2): L1 is unguided, L2 and L3 use task sheets; L2 adds EMG, and L3 adds EEG and, in the controlled setting, eye tracking and motion capture, all de-identified the same way. Code is released under Apache-2.0 and data under CC BY-NC 4.0.

References

Scene Classes

The corpus spans 38 scene classes in 5 zones—residential (9 classes), commercial (10), professional (5), industrial (6), and public (7)—plus one residual class. Table 3 lists all 38 with their hours in each tier, their share of the corpus, and the tiers in which they appear; Figure 8 shows a recorded frame for each.

#ClassL1 (h)L2 (h)L3 (h)Total (h)Share (%)Tier
Residential — 9 classes, 4,520.7 h, 36.2% of the corpus, 4 in L3
1kitchen868.2271.535.11,174.89.4L1–L3
2living room759.043.30.0802.26.4L1–L2
3dining room400.5174.175.2649.95.2L1–L3
4storage room502.898.40.0601.24.8L1–L2
5bathroom338.9169.20.0508.14.1L1–L2
6bedroom431.351.20.0482.53.9L1–L2
7study95.434.034.5163.91.3L1–L3
8laundry area69.425.030.7125.11.0L1–L3
9balcony13.00.00.013.00.1L1
Commercial — 10 classes, 2,588.1 h, 20.7% of the corpus, 2 in L3
10retail store355.0157.432.7545.14.4L1–L3
11logistics facility401.0127.90.0528.94.2L1–L2
12entertainment venue472.80.00.0472.83.8L1
13restaurant197.728.00.0225.71.8L1–L2
14beverage shop141.232.029.7202.91.6L1–L3
15pet shop202.60.00.0202.61.6L1
16hotel113.038.40.0151.31.2L1–L2
17performance venue123.80.00.0123.81.0L1
18hair salon78.40.00.078.40.6L1
19beauty salon56.50.00.056.50.5L1
Professional — 5 classes, 1,982.3 h, 15.9% of the corpus, 2 in L3
20office789.2177.1162.41,128.69.0L1–L3
21classroom338.912.50.0351.42.8L1–L2
22laboratory215.737.534.1287.32.3L1–L3
23healthcare facility200.30.00.0200.31.6L1
24conference centre14.70.00.014.70.1L1
Industrial — 6 classes, 1,956.7 h, 15.7% of the corpus, 1 in L3
25factory workshop857.1311.10.01,168.29.3L1–L2
26workroom387.6110.065.5563.24.5L1–L3
27industrial warehouse126.60.00.0126.61.0L1
28construction site74.50.00.074.50.6L1
29equipment room15.60.00.015.60.1L1
30testing laboratory8.70.00.08.70.1L1
Public — 7 classes, 1,210.6 h, 9.7% of the corpus, 0 in L3
31park & garden247.617.70.0265.32.1L1–L2
32public facility215.30.00.0215.31.7L1
33street193.70.00.0193.71.5L1
34farm150.719.70.0170.41.4L1–L2
35indoor sports venue157.60.00.0157.61.3L1
36sports area118.40.00.0118.40.9L1
37transportation90.00.00.090.00.7L1
Other — 1 class, 241.6 h, 1.9% of the corpus, 0 in L3
38other177.564.10.0241.61.9L1–L2

Table 3. Table 3: Scene classes in full. Hours are recorded hours by tier; total is the class total over all tiers; share is the percentage of the 12,500 h corpus; tier gives the tiers in which the class appears. Zone headings give the class count, the zone’s hours and share, and the number of classes in L3. Values are rounded to 0.1 h, so rows and columns may not sum exactly.

Head-camera frames from the scene vocabulary

Figure 8. Head-camera frames from the scene vocabulary: one frame for each of the 38 classes, labelled with its class name and ordered by zone (square colour) and by recorded hours within a zone. Per-class hours and tier coverage are in Table 3.

What a class is. A class names the physical setting in which an episode was recorded, not the activity performed there: the same task may occur in several classes, and a class hosts many tasks. Every episode carries a single scene label, so the class hours partition the corpus and sum to 12,500. The five zones are functional strata for reporting and stratified use, not a hierarchy—they group settings by primary function (living, commerce, work, production, and public life)—and two classes in one zone need not be closer than two in different zones. The residual class is a label like the others, not missing data: it holds 241.6 h, 1.9% of the corpus, and appears at L1 and L2 but not L3.

Boundaries. The label follows the setting in which the hands act for the greater part of the episode; crossing a doorway does not change it. The pairs that carry most of the ambiguity are resolved by function, not by furniture: kitchen against dining room separates preparation and cleanup from eating and serving; office against study separates a workplace from a home workspace, so the same desk task is labelled by the setting; storage room against industrial warehouse separates domestic from industrial storage, and hotel against bedroom a commercial guest room from a home bedroom. In the public zone, street is outdoor circulation, park and garden managed green space, and public facility an indoor public building; restaurant is a sit-down meal and a beverage shop a drinks counter.

The nine L3 classes. The deep tier does not spread evenly over the vocabulary. Office (162.4 h), dining room (75.2), workroom (65.5), and kitchen (35.1) carry 67.6% of its 500 h; the other five are study (34.5), laboratory (34.1), retail store (32.7), laundry area (30.7), and beverage shop (29.7). The nine span four zones—residential, commercial, professional, and industrial—and no public-zone class reaches L3. The set is therefore concentrated but not confined to one room type: two of its nine classes are shops.

Verb Lexicon

Every step carries two labels rather than one. A task verb names the goal the step accomplishes and is the unit a plan is written in; the motion verbs recorded inside the step name what the hand does to reach that goal, 2.5 of them per step. The two vocabularies are disjoint—no verb appears in both layers—and they behave in opposite ways: the task layer is open and long-tailed, the motion layer closed and concentrated. Table 4 lists both in full, with a boundary statement for each verb, the tiers in which it is observed, and its total occurrences; per-tier counts ship with the release.

#VerbGlossTiersShare (%)
Task layer — 99 verbs; share of all task annotations
1wipeclean a surface with a cloth kept in contactL1–L312.39
2openmove a closure so the container or space becomes accessibleL1–L39.93
3foldbend a flexible object back onto itselfL1–L37.75
4closereturn a closure to its shut positionL1–L35.75
5arrangeset several objects into an intended spatial orderL1–L35.47
6removetake an object away from its current placeL1–L35.26
7throwrelease an object so it travels through the airL1, L24.87
#VerbGlossTiersShare (%)
8organizegroup objects into an intended arrangement or containerL1–L33.26
9pourtip a vessel so its contents fall into a targetL1–L32.75
10unfoldopen out something previously foldedL1, L22.56
11collectgather scattered objects into one placeL1–L32.55
12hangsuspend an object from a supportL1–L32.41
13insertplace a part into a receptacle and seat itL1–L32.13
14stackplace objects one on top of anotherL1–L32.08
15cleanremove dirt from a surface or objectL1–L32.02
16coverplace something over an object to hide or protect itL1, L21.82
17wrapenclose an object in a flexible sheetL1–L31.34
18cutdivide an object with a bladeL1–L31.32
19securefix an object so it cannot moveL1–L31.31
20teardivide a flexible object by pulling it apartL11.20
21installmount a unit in its working position and fix it thereL1–L31.17
22sprayrelease a liquid from a nozzle over a targetL1, L21.13
23rinserun water over an object to carry residue awayL1, L21.11
24sortdistribute objects among classes by an attributeL1–L31.02
25tiefasten with a cord drawn into a knotL1–L30.97
26tightenturn a fastener until the joint stops yieldingL1–L30.94
27sweeppush loose material along a surface with a brushL1–L30.91
28attachjoin a part to a host so that the two move togetherL1–L30.90
29sealclose an opening so that it no longer passes air or liquidL1, L20.88
30unscrewturn a threaded fastener out of its seatL1–L30.84
31stickfix a flat item to a surface by adhesionL1, L20.75
32coilwind a flexible line into concentric loopsL1, L20.67
33screwturn a threaded fastener into its seatL1–L30.58
34disassembleseparate an assembly into its partsL1–L30.56
35washclean an object with water and agitationL1, L20.53
36checkinspect an object against an expected stateL1–L30.52
37assemblejoin parts into a working wholeL1–L30.51
38connectjoin two ends so that a link passes between themL1, L20.47
39replaceput a new part where the old one was taken outL1–L30.36
40dryremove moisture from a surface by wiping or by airL1, L20.36
41threadpass a line through an eye or an openingL1, L20.32
42writedeposit marks on a surface with a held implementL1–L30.32
43untieundo a knot and free the cordL1, L20.27
44peelstrip an outer layer from an object by handL1, L20.25
45rearrangemove objects already placed into a new orderL1–L30.25
46buttonfasten a garment through its buttonholeL1, L20.24
47separatepart two joined objects so that they no longer touchL1–L30.23
48applyspread a substance over a surfaceL1–L30.22
49scrapedraw a hard edge along a surface to lift material offL1, L20.22
50capfit a cover onto an openingL1–L30.22
51untangleseparate a snarl of flexible material into free endsL1, L30.21
52reassemblerejoin the parts of an assembly taken apart earlierL1, L20.21
53scrubclean a surface with a brush under pressureL1, L20.18
54scooplift loose material with a cupped toolL10.17
55bindfasten several objects together as one bundleL1, L20.17
56windturn a flexible line around an axisL10.17
57unzipopen a zip fastenerL1, L20.17
58slicedivide food into thin flat piecesL1, L20.16
59crumplecompress a sheet into an irregular ballL10.16

Table 4.

The task layer. 99 verbs over the annotated steps. The ten most frequent take 60.0% of the record and the median verb occurs 81 times, but the tail is long: 28 verbs occur 25 times or fewer, down to uncap at 4. Coverage varies by tier—98 verbs are observed at L1, 79 at L2 and 40 at L3; 34 appear in all three and 15 in only one. What a protocol removes is informative. Fifty-nine verbs are absent at L3, and the ones that matter for the protocol fall into three groups: four are cleaning (spray, rinse, wash, dry), six are cord or closure work (seal, stick, thread, untie, coil, connect), and four are whole-object actions (throw, unfold, cover, peel); tear, which never reaches L2, is absent for a simpler reason. A scripted desk task does not throw, unfold or wash.

The motion layer. 46 verbs over the motion annotations, and effectively a closed set: the four commonest—place, grasp, pick, move—take 82.2% of all motion annotations and the ten most frequent take 92.5%, against 35.8% and 60.0% for the same ranks in the task layer. Every one of the 46 verbs occurs at L1. Twenty-four occur in all three tiers, one (tilt) at L1 and L3 only, and 21 are absent at L3, falling into three groups: all six fine-contact verbs (pinch, poke, feel, rub, tap, squeeze), all five transport verbs (carry, drag, slide, transfer, bring), and ten posture or alignment verbs (push, lay, spread, roll, stand, dip, lean, bend, extend, retract). What drops out is contact without motion, and transport over distance.

The two layers together. The task verb is the sentence a plan is written in and the motion verbs are the record of how the hand executed it, and the tiers treat the two differently: L3 keeps 40 of 99 task verbs (40%) but 25 of 46 motion verbs (54%), and the motion verbs it keeps carry 95.2% of L1’s motion annotations against 77.3% for the task verbs. The decomposition is also finer where the record is deepest—2.50 motion verbs per step at L1, 3.00 at L3—so the deep tier says less about what was done and more about how.

Table 4: The verb lexicon in full. A task verb names the goal a step accomplishes; a motion verb names what the hand does inside the step. Glosses state the boundary used by the annotators, tiers give where the verb is observed, and share is each verb’s percentage of the annotations in its layer; percentages may not sum to 100 because of rounding.

#VerbGlossTiersShare (%)
60staplefix sheets together with a metal stapleL1, L20.15
61zipclose a zip fastenerL1, L20.15
62drawtrace a line on a surface with a held implementL1, L30.15
63fillput material into a container up to a levelL1, L20.13
64eraseremove marks from a surface by rubbingL1, L30.11
65unplugwithdraw a plug from its socketL1, L20.11
66testoperate an object to see whether it worksL1–L30.10
67stamppress an inked face onto a surfaceL10.09
68wringtwist a wet cloth to press the water outL1, L20.08
69loosenturn a fastener until it no longer holdsL10.08
70stirmove a tool through a mixture to blend itL1, L30.08
71drapelay a flexible sheet over a form so that it hangsL10.07
72loadput an object into a carrier for transportL1, L20.07
73discardput an object into wasteL1, L30.06
74tuckpush an edge of flexible material under or intoL10.06
75unwrapremove a sheet that encloses an objectL10.06
76detachtake a part off its hostL1, L20.06
77trimcut a small excess from an edgeL1, L20.06
78dividesplit one quantity into partsL1, L20.06
79unrollunroll a wound sheet or lineL1, L20.06
80punchmake a hole by pressing a tool throughL10.06
81reattachattach a part again after it was detachedL1, L20.06
82scanpass a reader over a code to register itL10.05
83pluginsert a plug into its socketL1, L20.05
84tapefix something with adhesive tapeL1, L20.05
85disconnectundo a link between two endsL1, L20.04
86switchchange the state of a controlL1, L20.04
87wetapply water to an objectL1, L20.04
88prylever an object open with a rigid toolL1, L20.04
89reinsertinsert a part again after removalL1, L20.03
90clamphold an object with a device that grips itL1, L20.03
91dispenserelease a measured amount from a containerL1, L20.03
92packput objects into a container for storageL1, L20.03
93brushclean a surface with a brushL1, L20.03
94pairbring two items together as a matched setL10.02
95polishrub a surface until it shinesL1, L20.02
96soakleave an object in liquid for a periodL1, L20.02
97dicecut food into small cubesL10.02
98sewjoin material with needle and threadL10.02
99uncaptake a cap off its openingL30.01

Table 11.

Motion layer — 46 verbs; share of all motion annotations

#VerbGlossTiersShare (%)
1placeput an object down at a target poseL1–L326.75
2graspclose the hand on an object so that it can be heldL1–L320.75
3picklift an object off its supportL1–L319.99
4movechange the position of a held object without releasing itL1–L314.67
5reachcarry the empty hand toward a targetL1–L33.26
6pushmove an object away by force applied through contactL1, L21.88
7adjustnudge an object to correct its poseL1–L31.83
8transportcarry a held object between placesL1–L31.21
9turnrotate a held object about a vertical axisL1–L31.12
10pulldraw an object toward the body by forceL1–L31.02
11liftraise an object against gravityL1–L30.80
#VerbGlossTiersShare (%)
12holdkeep an object in the hand without moving itL1–L30.70
13releaseopen the hand so that the object leaves itL1–L30.69
14spreadlay something out over an areaL1, L20.67
15rollturn an object over on its sideL1, L20.54
16layset an object down flat on a surfaceL1, L20.53
17alignbring an object into line with a referenceL1–L30.49
18flipturn an object over to its other faceL1–L30.47
19straightenremove a bend from a flexible objectL1–L30.38
20presspush down on a surface through a small areaL1–L30.22
21squeezecompress a soft object in the handL1, L20.22
22positionbring an object to a specified place without releasing itL1–L30.21
23repositionmove an object already placed to a new placeL1–L30.20
24carrymove an object while supporting its weightL1, L20.18
25standset an object upright on its baseL10.18
26diplower an object into a liquid and take it out againL1, L20.16
27rotateturn an object about its own axisL1–L30.15
28shakemove an object to and fro rapidlyL1–L30.12
29droplet an object fall from the handL1–L30.11
30bringmove an object toward a person or targetL1, L20.06
31dragmove an object along a surface without lifting itL1, L20.05
32lowermove an object downwardL1–L30.04
33tiltincline an object from its upright poseL1, L30.04
34slidemove an object along a surface in contact with itL1, L20.04
35retractdraw the hand back from an objectL1, L20.03
36tapstrike a surface briefly with the fingersL1, L20.03
37transfermove an object from one hand or place to anotherL1, L20.03
38orientturn an object so that a chosen face points at a targetL1–L30.03
39leanrest an object against a support at an angleL10.03
40adjust_gripchange the placement of the fingers on a held objectL1–L30.03
41rubmove the hand back and forth over a surfaceL1, L20.02
42benddeform an object along its lengthL10.02
43pinchgrip a small object between two fingersL1, L20.02
44pokepush a finger into an object or an openingL1, L20.02
45extendstraighten the arm or the fingers outwardL10.01
46feeltouch an object to judge its stateL10.01

Table 12.

Object Library

Every object an episode touches is an entry of a measured library: 839 instances under 442 distinct names, of which 829 are active and 10 retired, the retired ones shipped with a flag. An entry records a name, a shape together with the parameters that shape calls for, a primary and an optional secondary colour in hexadecimal, a material, a weight, a photograph, and a UUID. The UUID is the only key that joins an entry to the object list written with an episode, so the per-instance table ships with the release and is not printed here. The library grew with collection—35 entry dates between September and November 2026—rather than being assembled in advance.

Shape. Twelve primitives are in use (Figure 9a). Cuboid and cylinder alone account for 715 instances, or 85%; the other ten cover 124, the largest being the frustum (54, the open vessels: cups, bowls, pots, bins) and the torus (18, objects with a through-hole: tape rolls, washers, a bracelet). Size is stored as the parameters the shape calls for—length, width, and thickness for a cuboid, diameter and height for a cylinder or a frustum, three axes for an ellipsoid—so one schema serves a mug, a box, and a sheet. Lengths are read with a caliper and weights on a scale, and both are written to one decimal place, 0.1 cm and 0.1 g, which is the resolution the table carries.

Figure 9. The object library. (a) The twelve shape primitives and their percentage share of the library; the axis is logarithmic, and the solid beside each bar is the primitive. (b) Colour of every instance in CIELAB, lightness against chroma; the dotted line at C∗=12C^{*} = 12 separates the achromatic entries. (c) Longest dimension. (d) Weight, with quartiles dotted and the median dashed. (e) Weight of the ten commonest materials, ordered by median, with each material’s percentage share in brackets. n=839n = 839.

Size. By longest dimension the median instance is 15 cm, with quartiles at 8.6 and 23 cm. 86 instances are shorter than 5 cm—screws, washers, nuts, the parts a hand manipulates at the fingertips—and 54 reach 40 cm or more, the furniture and fabric at the other end. Between them, 84% of the library falls in 5–40 cm, the range in which a seated operator reaches and grasps (Figure 9c).

Weight and material. Weight spans six orders of magnitude, from 0.02 g (a plastic shaving) to 20 kg (a shelving unit), with a median of 42 g and quartiles at 7 and 172 g. 29% of the instances are under 10 g and only 38, some 4.5%, exceed 1 kg: the library is one of small, light objects (Figure 9d). Plastic accounts for 307 entries (37%), paper for 148, metal for 141, together about seven in ten. Weight separates by material (Figure 9e): glass and ceramic are highest, at medians of 470 and 326 g, paper lowest at 3.6 g, and plastic and metal both near 40–50 g but with long tails. At a given size it is the material that decides whether an object can be lifted in one hand, which is why it is recorded. The three commonest shape–material pairs are cuboid–plastic (168), cuboid–paper (139), and cylinder–metal (50).

Colour. Colour is the dominant value sampled from the photograph, in hexadecimal, with an optional second value. Mapped to names the library is greyscale-dominated—grey 25%, white 15%, black 13%, brown 11%, transparent 11%—and chromatic objects are only 35% of the total. In CIELAB (Figure 9b) most instances sit in the low-chroma band (C∗<20C^{*} < 20), and saturated reds and yellows are rare. Colour is therefore a weak discriminator between objects in one scene, which is why shape and size are recorded beside it.

Counting. The class count of § 3.4 counts the object classes named by annotators in a step; this appendix counts the physical instances that were measured. Each measured instance carries a UUID that also appears in the object list written with every episode that used it; a class reaches the library only through the instances its steps name. The two counts are therefore not the same number: a class may be realised by several measured instances, and one instance may be named by more than one class.

Hardware Bill of Materials

Table 5: Stream specifications of the in-house kit, which records L2 and L3. Every camera records RGB and runs at 30 fps; every view is released raw and rectified, the depth view rectified only; the third-person view exists only in the controlled setting. L1 records on third-party headbands, audited unit by unit in Appendix § 14. The pose and hand streams are released as fused outputs at about 800 Hz; the trackers and the glove that feed them run at their own device rates.

Table 5 lists the in-house instruments that record L2 and L3, with the resolution, channel count, rate, and unit of every stream; the L1 headbands are third-party and are audited in Appendix § 14. The headband is in-house; the remaining devices are commercial and are used as delivered.

StreamDevice (model)Resolution / channelsRateUnit
Head RGB ×4four-camera headband, Ego Headband V1.5 (in-house)1920×1200 each; left-front, right-front, left, right; clock hardware-synced30 fpsRGB
Wrist RGB ×2JIERUWEITONG FH868640×48030 fpsRGB
Scene cameraPupil Labs Neon1600×120030 fpsRGB
Third-person viewIntel RealSense D455f640×48030 fpsRGB
GazePupil Labs Neonx,yx,y pixel in the scene-camera image (L3), in the rectified head view (L1–L2)200 Hzpx
Head / wrist pose ×3HTC VIVE Tracker 3.0one shared frame: +xx left, +yy up, +zz forward~800 Hzmm
Finger kinematicsMANUS Metaglove Pro Haptics20 joint angles per hand~800 Hzdeg
Surface EMGWLEMGUV2A armbands ×28 channels per arm2000 HzmV
EEG (saline)Neuroscan SynAmps2 (Curry)128 channels, brain-region labels, impedance logged1000 Hzμ\muV
EEG (gel)BrainCo BCIGo32 or 8 channels, brain-region labels, impedance logged250 Hzμ\muV

Table 5. Stream specifications of the in-house kit, which records L2 and L3.

Headband. Four synchronised lenses sit on a rigid frame with a fixed baseline, so the stereo pair used for depth keeps its geometry across sessions; an inertial unit and the acquisition computer are mounted behind them, and the battery is carried at the back of the band. The band is fitted to the head with a replaceable inner liner, which also keeps the lenses at a repeatable height and angle.

Wrist kit. Each wrist carries an armband with eight dry electrodes on the palm side. At L3 the armband is paired with a camera on a strap aimed along the hand, and the two are cabled together so the forearm does not pull on the camera; the pair is worn over the glove, and at L2 only the armband is worn, directly on the forearm.

Cognition kit. The L3 kit adds a binocular eye tracker with its scene camera, three 6-DoF trackers (head and both wrists), a pair of 20-node gloves, and an EEG cap. Two cap types are used: a 128-channel saline cap for the controlled setting and a 32- or 8-channel gel cap for the open setting; both are seated and checked for impedance before each block.

Synchronisation hardware. A UDP marker is emitted by the recorder, and phase and run transitions are additionally driven through a parallel-port box into the EEG amplifier, so protocol events are stamped on the amplifier clock as well as on the host clock. Cameras, headband, and eye tracker are calibrated before each session against a printed target; the resulting intrinsics, distortion, and tracker transforms ship with the episode.

Consumables and spares. Saline, gel, syringes, electrode paste, spare straps, liners, and batteries are restocked per shift; a failed cap or camera is swapped and the session restarted rather than repaired in place, so that a released episode never mixes two calibrations.

L1 Source Audit

Why an audit. L1 is the only tier whose hardware is not ours, and it is not one device: it is a collection layer of fourteen headbands from thirteen vendors. Everything the corpus reports for L1 is conditioned on what those units physically provide—the view gaze is inferred from, the pair that depth is reconstructed from, the rate the hands are estimated at—so the layer is only interpretable if that condition is written down unit by unit. The audit records it, so that a result can be read against the unit it came from and the layer can be reproduced.

What is recorded. Four properties are taken from each unit rather than from its documentation. The lens set and the direction of every view are read off the device. Native resolution and frame rate are measured on the recorded stream, not on the specification sheet. Whether the main view is a genuine stereo pair is checked on synchronized frames: twelve of the fourteen provide one, two of them as a single side-by-side image that the pipeline splits into left and right, and one through a pair of its positioning cameras (up, down, left, and right) rather than a dedicated front pair; units 2 and 7 are fisheye arrays without a stereo pair. The stream is then classified as raw or pre-processed, because the domain matters downstream: one unit delivers only rectified 512×512512 \times512 frames, one delivers the pair already split, and one adds an extension view beside the pair.

What the variation implies. Pairing decides how a depth estimate is formed, and the pairs differ—a dedicated front rig, a side-by-side image split after the fact, or two positioning cameras—so the audit records the configuration each episode rests on, and a result that depends on the pair is stated with the unit it came from. Optics trade coverage against geometry: the seven units with fisheye or side views see further around the operator, but every unit is rectified to a common pinhole domain before estimation, so the peripheral views buy coverage rather than accuracy, and their corners are lost at rectification (§ 5). Rate and resolution set the domain in which every L1 estimate lives: from 512×512512 \times512 at 20 Hz to 3840×12003840 \times1200 at 60 Hz. Nothing is resampled to a common grid at write time (§ 4.3), so an estimator trained on one unit is trained in that unit’s domain, and the audit is what allows the domain to be declared. Table 6 lists the units, numbered by collection volume, with two packaged sets from one vendor taking consecutive numbers; vendor, model, and toolchain names are withheld, so the table documents capability rather than procurement.

Episode File Layout

Every episode is self-contained and follows a chunked hierarchy compatible with a modified LeRobot v3.1 release. The directory is named after the session (collector, date, clock). Every camera is released twice, raw and rectified, and the electrophysiology is released both raw and cleaned. The example in Figure 10 holds 1,444 master frames (48.1 s), 17 videos, one audio track, and 42 Parquet tables.

Signals. Each stream is one Parquet file on its own time base, with a timestamp column in seconds from episode start and an episode_index; columns are named after channels (FP1...), joints (node00_x...), or dim_0... for EMG, so no stream is snapped to the master grid. The pose and hand streams are fusion outputs and carry the timestamps of the fused track rather than of a single device. Pixels are not stored in Parquet: observation.images.*.parquet lists the timestamp of every frame and the frames themselves live in the matching mp4 (H.264, 30 fps), one for the raw view and one for its rectified counterpart; depth ships rectified only. Audio is 16 kHz mono PCM in wav, with its Parquet file recording the sample count per master frame. The names are literal: eeg_raw is the record as acquired, eeg eeg_repair the trace after bad-channel interpolation and component removal, while each wrist EMG ships as a raw trace beside its envelope. Table 7 lists the shape and the fields of every signal table. @page 35/41 starts_mid=0 ends_mid=0

StreamShapeFields
eeg_raw[?]132 electrodes with conventional scalp labels (Fp1. . . ), as acquired
eeg[?]the same electrodes after a 0.5–100 Hz band-pass and a 50 Hz notch
eeg_repair[?]the same electrodes after bad-channel interpolation and component removal
eeg_impedance[?]per-electrode impedance in kΩ plus Trigger, pass_rate, gate_pass, n_snapshots
eye_gaze[2]x, y in scene-camera pixels
eye_imu[6]ax, ay, az, gx, gy, gz
head_pose[6]x, y, z, roll, pitch, yaw in the shared frame
hand_pose[40]20 joint angles per hand (left_ThumbMCPSpread, . . . )
hand_skeleton[?]50 nodes, each x, y, z, roll, pitch, yaw
hand_left_points[63]wrist and 20 joints, each x, y, z
hand_right_points[63]the same fields for the right hand
headband_imu0[6]ax, ay, az, gx, gy, gz
headband_microphone[?]16 kHz audio, 320 samples per master frame
left_wrist_emg_raw[8]eight forearm channels, as acquired
left_wrist_emg[8]the same channels, rectified and enveloped
left_wrist_imu[6]ax, ay, az, gx, gy, gz
left_wrist_pose[6]x, y, z, roll, pitch, yaw
right_wrist_emg_raw[8]channels of the right forearm, as acquired
right_wrist_emg[8]the same channels, rectified and enveloped
right_wrist_imu[6]ax, ay, az, gx, gy, gz
right_wrist_pose[6]x, y, z, roll, pitch, yaw
marker[1]protocol code: run start, imagine, execute, run end
state[58]both wrist poses and the head pose (18) plus the 40 joint angles

Table 7. The 23 signal tables: per-frame shape and the fields each one carries.

Table 6: L1 source audit. Vendor, model, and toolchain names are withheld; units are numbered by collection volume, and two packaged sets from the same vendor take consecutive numbers. Views give the direction of each lens, with fisheye marked explicitly. Cameras counts the physical lenses of the unit; Views lists the released views. The two differ where one lens contributes more than one view (a side-by-side pair split into left and right, or an extension view) or where a lens is not released as a separate view.

UnitCamerasMain resolutionViewsfps
161920×12001920 \times1200 stereoleft-front, right-front; four fisheye around60
261920×15361920 \times1536six fisheye: front and side views; no stereo pair30
323840×12003840 \times1200 side-by-side stereoleft-front, right-front; extension view30
421920×12001920 \times1200 stereoleft-front, right-front30
56512×512512 \times512 rectified stereoleft-front, right-front; four fisheye around20
662328×17482328 \times1748 single RGBfront; positioning cameras up, down, left, right30
761920×12001920 \times1200six equidistant fisheye; no stereo pair30
833840×12003840 \times1200 front; 2560×9602560 \times960 down, sideleft-front, right-front, down, side30
921920×12001920 \times1200left-front, right-front30
1022160×21602160 \times2160left-front, right-front30
1123840×12003840 \times1200 side-by-side, split after processingleft-front, right-front30
1221920×14561920 \times1456left-front, right-front30
1361600×13001600 \times1300 main stereoleft-front, right-front; left outer, right outer30
1422048×15362048 \times1536left-front, right-front30

Table 6.

Figure 10. Episode file layout. One episode directory, L3-example/, with its tables, videos, audio, calibration, annotation, and metadata. <signal> expands to the 23 stream tables, <view> to the 17 camera views (each with a timestamp table and one mp4 per encoding, raw and rectified, except depth, which is rectified only), and <stream> to the microphone stream.

Table 6: L1 source audit. Vendor, model, and toolchain names are withheld; units are numbered by collection volume, and two packaged sets from the same vendor take consecutive numbers. Views give the direction of each lens, with fisheye marked explicitly. Cameras counts the physical lenses of the unit; Views lists the released views. The two differ where one lens contributes more than one view (a side-by-side pair split into left and right, or an extension view) or where a lens is not released as a separate view.

UnitCamerasMain resolutionViewsfps
161920×12001920 \times1200 stereoleft-front, right-front; four fisheye around60
261920×15361920 \times1536six fisheye: front and side views; no stereo pair30
323840×12003840 \times1200 side-by-side stereoleft-front, right-front; extension view30
421920×12001920 \times1200 stereoleft-front, right-front30
56512×512512 \times512 rectified stereoleft-front, right-front; four fisheye around20
662328×17482328 \times1748 single RGBfront; positioning cameras up, down, left, right30
761920×12001920 \times1200six equidistant fisheye; no stereo pair30
833840×12003840 \times1200 front; 2560×9602560 \times960 down, sideleft-front, right-front, down, side30
921920×12001920 \times1200left-front, right-front30
1022160×21602160 \times2160left-front, right-front30
1123840×12003840 \times1200 side-by-side, split after processingleft-front, right-front30
1221920×14561920 \times1456left-front, right-front30
1361600×13001600 \times1300 main stereoleft-front, right-front; left outer, right outer30
1422048×15362048 \times1536left-front, right-front30

Table 6 (PDF p. 35).

Figure 10 (PDF p. 35). Episode file layout. One episode directory, L3-example/, with its tables, videos, audio, calibration, annotation, and metadata. <signal> expands to the 23 stream tables, <view> to the 17 camera views (each with a timestamp table and one mp4 per encoding, raw and rectified, except depth, which is rectified only), and <stream> to the microphone stream.

eeg_repair the trace after bad-channel interpolation and component removal, while each wrist EMG ships as a raw trace beside its envelope. Table 7 lists the shape and the fields of every signal table.

Metadata. info.json records the codebase version, master feature, fps, path templates, chunk size, and the shape and channel names of every feature. episodes.jsonl and tasks.jsonl map indices to the task string and episode length. collect_info.jsonl stores collector, hardware, scene, the object list (colour, shape, dimensions, material, pose), and free-text notes. alignment_check.jsonl reports, per feature, the tolerance, number of readings, violations, and maximum offset, and episodes_stats.jsonl stores per-feature min, max, mean, and standard deviation.

Calibration and annotation. calibrations/ holds what the tables reference: the reference cameras with per-view rate, frame count, size, and intrinsics, and the P1/P2 projections the rectification applied. annotation/ holds the structured annotation—scene, objects, timed subtask segments, tasks, key events, state changes—and the QC gate with its checks and warnings.

Reading. The reader fixes a master rate (30 fps by default) and walks master_index; each row gives a time, and every other stream is resolved from its own timestamps by nearest sample, linear interpolation, or all samples in a window such as (−1/FPS, 0]. Nothing is stored on the master grid, so a stream can be re-read at any rate without degradation: a query that needs raw EMG reads the 2 kHz table directly, while a query that needs one frame per 33 ms reads the index and never touches the other tables.

Quality Control

Every session is checked after it is written and before it enters the release. The checker walks each recorder’s output directory—the NPZ arrays, the MP4 containers, and the logs—and returns, per stream, a verdict with its evidence attached; no stream is re-recorded and nothing is repaired. A stream that fails is not patched: the whole session is recaptured, so that a released episode never mixes a good half with a repaired one.

Alignment comes first. All checks run inside the analysis window bounded by the RUN_START and RUN_END markers. The pair is a hard precondition: if it is missing, empty, or unpaired, the session is reported as an error rather than checked against the whole recording, because without the window every per-stream crop and comparison is undefined. The same rule applies to the clock: the EEG stream is accepted only when the amplifier clock was fitted to the host clock, and a failed fit is an error, not a warning.

Two verdicts reach the release. Each check yields a finding at one of three levels, of which two matter downstream. An error means the data cannot be used—a corrupt container, a frame count that disagrees with the timestamps, a missing marker pair, a tracker that was absent when the recorder opened—and the session is recaptured. A warning means the data is usable but carries a documented defect: a 2 s frozen stretch, a dead channel, a tracker that lost sight for part of the window, a gap shorter than the threshold for an error. Informational findings, such as the phone-to-host clock correction, never gate anything. Thresholds are the checks’ own defaults and are reported with the observed value.

What is checked. The checks run across the nine stream families the kit produces, and each is an independent, thresholded verdict: a modality declares the checks that apply to it, and a shared context loads the NPZ once and decodes each video once, so the checks compose without repeating work. Table 8 lists the families and the default thresholds. Timing checks ask whether a series is usable at all (present, finite, ordered, distinct), whether the cadence is steady, whether an interval jumped far above the median, whether samples are missing, and whether the rate held inside a one-second window—the last of these catches a drop that a mean would hide. Signal checks ask whether the values are plausible: NaN fractions, channels that never move, robust outliers, physical bounds, and motions no body could make. Modality checks are specific to one instrument’s failure modes: the EEG amplifier’s clock fit, fixed-size blocks and header sample count, the EMG armband’s shared 8-bit serial counter, the glove skeleton dropping out window by window, the tracker count and per-tracker validity, and the camera set the headband promised at stream start. Video checks decode every file once and compare the container’s frame count and pixel statistics against the timestamps, which is the only way to see a writer that fell behind silently; the container test uses the same fprobe criterion as the packaging gate. Session checks close the window: the RUN_START/RUN_END pair and the recorder logs.

CheckApplies toRule (default)Level
Timing
Presence and orderevery seriesmissing series, no samples, or NaN = ERROR; backward = WARN; duplicate = WARN (ERROR for EMG)ERROR / WARN
Gapevery seriesinterval > max(0.1 s, 1.5× median), including head and tail against the window edgesERROR
Jumpevery seriesinterval > 3× medianWARN
Jitterevery seriesinterval CV > 0.3WARN
Windowed rateevery seriesany 1 s window outside ±50% of the median sample countWARN
Expected rateseries with a nominal ratemean rate < 50% of nominalWARN
Sample countEEGdeviation from window × nominal rate > 1%WARN
Signal values
NaN and dead channelsEEG, EMG, handNaN > 0.1%; zero-variance channelWARN
Robust outliersEEG, EMG, hand, headband IMU> 0.1% beyond median ±20 MADWARN
Value boundseye gazegaze outside [0, 2000] pxWARN
CheckApplies toRule (default)Level
Implausible jumpshand skeleton, VIVEa glove node moving >0.05> 0.05 m or a tracker >0.2> 0.2 m between samplesWARN
Modality-specific
EEG clock and blocksEEGclock fit failure = ERROR; block discontinuity = WARN; header count mismatch = WARNERROR / WARN
EMG sequenceEMGlost frames >0.1%> 0.1\% = WARN; duplicate timestamps = ERRORWARN / ERROR
Hand dropouthand pose>50%> 50\% skeleton NaN in a 1 s windowWARN
VIVE validityVIVE trackersvalid fraction <50%< 50\% = WARN; any invalid sample = ERROR; tracking gap >0.1> 0.1 s = ERROR; teleport >0.2> 0.2 m = WARN; device count ≠3\ne3 = ERRORERROR / WARN
Headband camera setego headbandmissing camera = ERROR; extra camera = WARNERROR / WARN
Video
Container and countevery MP4ffprobe failure = ERROR; container frame count differs from timestamp count = ERRORERROR
Black and freezeevery MP4mean luminance <8< 8 for >2> 2 s = WARN; >50%> 50\% black = ERROR; frame difference <1.0< 1.0 for >2> 2 s = WARNERROR / WARN
Session
Marker windowsessionmissing RUN_START/RUN_END pair = ERROR; out-of-order markers = WARNERROR / WARN
Recorder logsevery streamlines matching ERROR, FAILED, Traceback, or crashWARN

Table 8. Quality-control families and default thresholds. A row is a family of checks; where its sub-checks differ in level, both are listed. A session’s outcome is the worst finding across its streams and its session-level checks, and ERROR blocks packing. Thresholds are code defaults, overridable in configs/checker.yaml; not every check runs on every stream.

The verdict travels with the episode. The checker writes one report per session and its summary is copied into the episode metadata; an optional self-contained page renders the same findings on one time axis for review. Each finding names the check that produced it, the field or file it concerns, the sub-stream it belongs to where there is one (a tracker serial, a channel index), the threshold and the observed value, and the time span of the problem; a warning is therefore a pointer to the affected seconds, not a blanket statement about the recording.

Recapture and what it costs. Because an error retires the whole session, the pass rate is a property of the kit rather than of the analysis: more than 99% of L1 recordings, 96% of L2 and 78% of L3 pass. The gradient is the instrumentation. L1 has a single headband; L2 adds two armbands whose electrodes must hold contact through sweat and movement; L3 adds the EEG cap, which must reach impedance on enough channels, the gloves and three trackers, each able to be absent at open or lose sight mid-window, and the eye tracker, whose clock must be fitted. The deepest tier, which is also the smallest, is therefore the tier most often repeated.

Before recording. On the floor, four checks—camera labels against the live tiled view, wrist-camera seating, EMG contact, and tracker visibility—must pass before a session starts; any re-plugged or swapped device triggers all four again, and the passing screen is photographed and filed, so a session without evidence counts as unchecked. A disk-space gate refuses a recording below 50 GB and warns below 100 GB.

What the gate does not cover. The automated layer tests data integrity, not semantics. It does not verify annotation boundaries, task labels, object attributes, or calibration accuracy beyond the presence of a fitted clock; those remain with the annotation review. It is also per session, so cross-session consistency—one operator, scene, or unit behaving differently across sessions—is outside its scope. The checks are meant to separate a usable recording from an unusable one, not to rank two usable recordings against each other.

Synchronization Audit

Measurement. Synchronization is audited against the protocol markers rather than inferred from the recorder. For each audited L3 episode we take the six phase cues (INSTR_ON, INSTR_OFF, IMG_START, IMG_END, EXEC_START, EXEC_END) and, for every stream, the sample nearest each cue; the event-locked residual is the nearest sample time minus the cue time. EEG is split by amplifier: Neuroscan 128-channel at 1 kHz and BrainCo 32/8-channel at 250 Hz.

Reading the table. For regular sampling at period TT, the nearest-sample residual lies within ±T/2\pm T/2, and a uniform residual has an interquartile range (IQR) of about T/2T/2. An IQR close to the half-period therefore indicates that the residual is set by sampling quantization rather than by a clock offset; the normalized column makes this comparison explicit. A value near one is the quantization floor, and a value well above one indicates additional timing irregularity.

Results. Table 9 shows that the high-rate streams reach the sampling floor. The 1 kHz EEG has a median residual of +0.10 ms and an IQR of 0.47 ms, against a half-period of 0.50 ms; the 2 kHz EMG bands have IQRs of 0.27 and 0.22 ms against 0.25 ms; and the fused 480–940 Hz pose stream has an IQR of 1.04 ms against about 0.80 ms, its half-period at the median rate of 628 Hz. Their IQR/half-period ratios are 0.93, 1.08, 0.87, and 1.30, with no systematic offset beyond sampling quantization. The 250 Hz EEG (IQR 2.22 ms) and the 200 Hz gaze (IQR 2.29 ms) are quantized at their 4 ms and 5 ms sample periods; their ratios of 1.11 and 0.91 place them at the same floor, and they should not be read as less well synchronized.

StreamRate (Hz)T/2T/2 (ms)Offset (ms)IQR (ms)IQR/(T/2)\left(T/2\right)P95 (ms)Max (ms)Drift (ms/min)
EEG, Neuroscan 128-ch1000.00.50+0.100.470.930.460.48+0.35
EEG, BrainCo 32/8-ch250.22.00−0.472.221.112.383.19n.r.
EMG, left2000.00.250.000.271.080.350.45+0.09
EMG, right2000.00.25−0.050.220.870.340.44+0.15
Pose (VIVE + Manus)480–9400.800.001.041.301.041.66n.r.
Gaze (Neon)199.82.50−0.032.290.912.473.93n.r.
Head RGB30.016.67−1.6720.361.2225.4227.14n.r.

Table 9. Event-locked synchronization residuals on released L3 episodes. Each episode contributes six protocol cues; EEG is split by amplifier (Neuroscan, 1 kHz; BrainCo, 250 Hz). Residual is the nearest released sample minus the cue timestamp; IQR is the interquartile range; P95 and Max are absolute. The normalized column divides the IQR by half the sampling period, so values near one are at the quantization floor. n.r. marks drift not resolvable over a 40 s episode.

Drift. The residual trend is +0.35 ms/min for the 1 kHz EEG (median; episode range 0.03–0.92) and +0.09/+0.15 ms/min for the two EMG bands (episode range −0.60 to +0.34); the EEG and EMG residuals stay within ±1 ms/min (±17 ppm) over the episode. For the remaining streams (250 Hz EEG, 200 Hz gaze, 30 Hz camera, and the fused pose output) the sample period exceeds the drift resolvable in 40 s, and the table marks drift as not resolvable. The audit supports the synchronization claim of § 5.1: each device keeps its own clock, and the released streams are aligned to a common time base through markers and arrival offsets, not through a shared hardware clock.

Timestamp integrity. The high-rate streams are regular: EEG at both configurations has no duplicate timestamps and no interval beyond five median intervals, and EMG has no duplicate timestamps. The fused pose stream is less regular: across the audited episodes it contains 39 duplicate timestamps and six intervals beyond five median intervals, and its rate ranges from 480 to 940 Hz (median 628 Hz) rather than the nominal 800 Hz. The 30 Hz camera streams are the least regular: the head RGB stream in these episodes contains twelve duplicate timestamps, and every camera container holds 24–72 frames more than its timestamp table (28–67 for the head RGB stream). The excess is consistent with pre- and post-roll frames outside the episode window; those frames are not represented in the Parquet table, so a reader should index the episode through the timestamp table rather than through the container frame number.

Electrophysiology Processing

Every stage below runs on the raw stream, and its parameters are logged with the episode; the chain is fixed across sessions so that the same code can be replayed on any release.

EMG. (1) Mirror padding: the trace is reflected about its first and last sample, filtered, and cropped back to its original length, which suppresses the transient a filter otherwise leaves at the boundary. (2) High-pass, 20 Hz, second-order Butterworth: removes baseline drift and part of the motion artefact, and a Butterworth response is maximally flat in the passband. (3) Low-pass, 450 Hz, fourth-order Butterworth: bounds the analysis band; a digital filter cannot undo aliasing already introduced at sampling. (4) Zero-phase filtering: the sections are applied forward and then backward, so the interior response has zero phase and the magnitude is the square of the one-way response, an order doubled and a steeper roll-off at the cut-off. (5) Comb notch, 50 Hz with a 2 Hz bandwidth (Q=50/2=25Q = 50/2 = 25): periodic nulls at the mains and its harmonics; the backward pass deepens and widens each null, and also attenuates genuine EMG in the same bands. (6) Periodic extension: the edge periods are repeated so that the comb has boundary data, which holds only as far as the edges represent the interference. (7) Period alignment: anomalous edge periods are detected and replaced by interior ones, which rewrites boundary samples and can touch real transients. (8) Endpoint damping: decaying edge energy is detected and cross-faded toward the interior, which removes residual ringing at the cost of possibly attenuating a genuine activation. The cleaned trace is then rectified and low-pass filtered into one envelope per channel.

EEG. The chain is DC removal by the per-channel median, a 0.5–100 Hz band-pass with the high cut clamped below Nyquist, and a 50 Hz notch, followed by four stages. Bad channels are flagged by five joint rules: flat (σ<0.2 μV\sigma< 0.2\,\mu\mathrm{V} with more than 70% of sample differences below 0.02 μV0.02\,\mu\mathrm{V}); contact (robust zz of σ\sigma above 3.5, or above 3.0 with σ>25 μV\sigma> 25\,\mu\mathrm{V}); bridging (a neighbouring pair whose electrical distance stays below 16 μV216\,\mu\mathrm{V}^{2} for more than half of the 2 s epochs and below half the reference of their other neighbours); decorrelation (maximum ∣r∣\lvert r\rvert against any other channel below 0.4 in more than 20% of 1 s windows); and high-frequency noise (robust zz above 5 of the 30–100 Hz to 1–30 Hz MAD ratio). Frontal channels are excused a contact flag when no other rule fires. A session passes when fewer than 30% of its channels are flagged. ICA is infomax with up to 32 components, capped at the number of retained channels, fitted once on the clean segments of a session concatenated in time, with flagged channels held out; ocular components are those correlated with the frontal proxy channels, at most two, and muscle components those whose 30–100 Hz power ratio is a robust z>3z > 3 above the other components, capped at 20% of the retained channels. Interpolation is spherical spline over the retained channels, and the cleaned trace is finally re-referenced to their average.

Montage and regions. Every channel of the release carries a conventional scalp label and the region group of its row in Table 10. The 128-channel saline cap covers the full conventional set plus the intermediate positions, including the mastoids and the two cerebellar electrodes; the 32-channel gel cap is a subset of it with two auxiliary channels; and the 8-channel gel cap keeps two frontal-central, two central, one midline parietal, and three occipital sites. Where a position lacks a conventional coordinate, Cb1, Cb2, and Cbz take the coordinates of I1, I2, and Iz on the colin27 template, matching the CURRY labels, and F11, F12, FT11, and FT12 take the coordinates supplied with the cap. Both sets ship with the release and are used only for neighbour finding and interpolation, never for source modelling.

IndexGroupnnChannels
128-channel saline cap (Neuroscan)
IndexGroupnnChannels
1–64core electrodes, incl. M1/M2 and Cb1/Cb264Fp1, Fpz, Fp2, AF3, AF4, F11, F7, F5, F3, F1, Fz, F2, F4, F6, F8, F12, FT11, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT12, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, M1, M2, P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO3, POz, PO4, PO8, O1, Oz, O2, Cb1, Cb2
65–77Prefrontal and frontal extension13AFp1, AFp2, AF7, AF5, AFz, AF6, AF8, AFF5h, AFF3h, AFF1h, AFF2h, AFF4h, AFF6h
78–79Lateral frontal2F9, F10
80–87Fronto-central extension8FFT7h, FFC5h, FFC3h, FFC1h, FFC2h, FFC4h, FFC6h, FFT8h
88–91Lateral fronto-temporal4FT9, FT7, FT8, FT10
92–99Fronto-temporo-central extension8FTT7h, FCC5h, FCC3h, FCC1h, FCC2h, FCC4h, FCC6h, FTT8h
100–107Centro-temporo-parietal extension8TTP7h, CCP5h, CCP3h, CCP1h, CCP2h, CCP4h, CCP6h, TTP8h
108–115Parietal extension8TPP7h, CPP5h, CPP3h, CPP1h, CPP2h, CPP4h, CPP6h, TPP8h
116–117Lateral parietal2P9, P10
118–121Parieto-occipital extension4PPO3h, PPO1h, PPO2h, PPO4h
122–127Occipital extension6PO9, PO5, PO1, PO2, PO6, PO10
128Midline cerebellar1Cbz
129–130Ocular2VEOG (vertical), HEOG (horizontal)
131Cardiac1EKG
132Muscular1EMG
32-channel gel cap (BrainCo)
1–7Frontal and prefrontal7FP1, FP2, F3, F4, F7, F8, Fz
8–10Central3C3, C4, Cz
11–15Parietal5P3, P4, P7, P8, Pz
16–17Occipital2O1, O2
18–19Temporal2T7, T8
20–23Fronto-central4FC1, FC2, FC5, FC6
24–27Centro-parietal4CP1, CP2, CP5, CP6
28–29Fronto-temporal2FT9, FT10
30–31Temporo-parietal2TP9, TP10
32–33Non-EEG (ocular, trigger)2IO, TRIG
8-channel gel cap
1–2Fronto-central2FC3, FC4
3–4Central2C3, C4
5Midline parietal1Pz
6–8Occipital3O1, Oz, O2

Table 10. Channel groups by brain region. Rows give the connector index of the channels, the region label stored with them, their count, and their names.

References

  1. [1]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2110.07058
  2. [2]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangellos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130:33–55, 2022.DOI
  3. [3]Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2203.14712
  4. [4]Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. doi: 10.1109/CVPR52729.2023.01244.DOI
  5. [5]Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2505.11709
  6. [6]Lightwheel AI. EgoSuite-Open100K: A fully annotated open egocentric human dataset. https://huggingface.co/collections/LightwheelAI/egosuite-open100k, 2026. Dataset release.
  7. [7]Ropedia. Xperience-10M: A large-scale egocentric multimodal dataset with structured 3D/4D annotations. https://huggingface.co/datasets/ropedia-ai/xperience-10m, 2026. Dataset release.
  8. [8]Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. EgoMimic: Scaling imitation learning via egocentric video. In IEEE International Conference on Robotics and Automation (ICRA), 2025. doi: 10.1109/ICRA55743.2025.11127989.DOI
  9. [9]Ruihan Yang, Q inxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng, Ri-Zhao Qiu, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, and Xiaolong Wang. EgoVLA: Learning vision-language-action models from egocentric human videos. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.arxiv.org/abs/2507.12440
  10. [10]Hao Luo et al. Being-H0: Vision-language-action pretraining from large-scale human videos. arXiv preprint arXiv:2507.15597, 2025.arxiv.org/abs/2507.15597
  11. [11]Anthony Brohan et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2307.15818
  12. [12]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2406.09246
  13. [13]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025.DOI
  14. [14]Abby O’Neill et al. Open X-Embodiment: Robotic learning datasets and RTX-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.arxiv.org/abs/2310.08864
  15. [15]Alexander Khazatsky et al. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2403.12945
  16. [16]Shihan Wu et al. RoboCOIN: An open-sourced bimanual robotic data collection for integrated manipulation. arXiv preprint arXiv:2511.17441; https://github.com/FlagOpen/RoboCOIN, 2025.arxiv.org/abs/2511.17441
  17. [17]Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2302.00111
  18. [18]Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2312.13139
  19. [19]Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267, pages 24328–24346, 2025.arxiv.org/abs/2412.14803
  20. [20]Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. In Robotics: Science and Systems (RSS), 2025. doi: 10.15607/RSS.2025.XXI.074.
  21. [21]Michael F. Land and Mary Hayhoe. In what ways do eye movements contribute to everyday activities? Vision Research, 41(25–26):3559–3565, 2001.pubmed.ncbi.nlm.nih.gov/11718795
  22. [22]Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4):188–194, 2005.pubmed.ncbi.nlm.nih.gov/15808501
  23. [23]Roland S. Johansson, Göran Westling, Anders Bäckström, and J. Randall Flanagan. Eye–hand coordination in object manipulation. Journal of Neuroscience, 21(17):6917–6932, 2001.pubmed.ncbi.nlm.nih.gov/11517279
  24. [24]Roland S. Johansson and Göran Westling. Programmed and triggered actions to rapid load changes during precision grip. Experimental Brain Research, 71(1):72–86, 1988.pubmed.ncbi.nlm.nih.gov/3416959
  25. [25]Roland S. Johansson and J. Randall Flanagan. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience, 10(5):345–359, 2009.DOI
  26. [26]Hans Helmut Kornhuber and Lüder Deecke. Hirnpotentialänderungen bei Willkürbewegungen und passiven Bewegungen des Menschen: Bereitschaftspotential und reafferente Potentiale. Pflügers Archiv für die gesamte Physiologie des Menschen und der Tiere, 284(1):1–17, 1965. doi: 10.1007/BF00412364.DOI
  27. [27]Hiroshi Shibasaki and Mark Hallett. What is the Bereitschaftspotential? Clinical Neurophysiology, 117 (11):2341–2356, 2006.pubmed.ncbi.nlm.nih.gov/16876476
  28. [28]William J. Gehring, Brian Goss, Michael G. H. Coles, David E. Meyer, and Emanuel Donchin. A neural system for error detection and compensation. Psychological Science, 4(6):385–390, 1993.DOI
  29. [29]Dario Farina, Ning Jiang, Hubertus Rehbaum, Aleš Holobar, Bernhard Graimann, Heinz Dietl, and Oskar C. Aszmann. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: Emerging avenues and challenges. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(4):797–809, 2014.DOI
  30. [30]Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. Brain–computer interfaces for communication and control. Clinical Neurophysiology, 113(6):767–791, 2002.
  31. [31]Nie Lin, Yansen Wang, Dongqi Han, Weibang Jiang, Jingyuan Li, Ryosuke Furuta, Yoichi Sato, and Dongsheng Li. EgoBrain: Synergizing minds and eyes for human action understanding. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2506.01353
  32. [32]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The EPIC-KITCHENS dataset. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.arxiv.org/abs/1804.02748
  33. [33]Yin Li et al. In the eye of the beholder: Gaze and actions in first person video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):6731–6747, 2023.
  34. [34]Kristen Grauman et al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  35. [35]Yunze Liu et al. HOI4D: A 4D egocentric dataset for category-level human–object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2203.01577
  36. [36]Xin Wang et al. HoloAssist: An egocentric human interaction dataset for interactive AI assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2309.17024
  37. [37]Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2406.09905
  38. [38]Toby Perrett, Ahmad Darkhalil, et al. HD-EPIC: A highly-detailed egocentric video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2502.04144
  39. [39]Jingkang Yang, Shuai Liu, et al. EgoLife: Towards egocentric life assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2503.03803
  40. [40]Manfredo Atzori, Arjan Gijsberts, Claudio Castellini, Barbara Caputo, Anne-Gabrielle Mittaz Hager, Simone Elsig, Giorgio Giatsidis, Franco Bassetto, and Henning Müller. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Scientific Data, 1:140053, 2014.
  41. [41]EgoEMG: A multimodal egocentric dataset with bilateral EMG and vision for hand pose estimation. GitHub repository, 2026. URL https://github.com/zhengqis123/EgoEMG.
  42. [42]Klaus Gramann, Joseph T. Gwinn, Daniel P. Ferris, Kelvin Oie, Tzyy-Ping Jung, Chin-Teng Lin, Lun-De Liao, and Scott Makeig. Cognition in action: Imaging brain/body dynamics in mobile humans. Reviews in the Neurosciences, 22(6):593–608, 2011.DOI
  43. [43]Sebastiaan F. W. Neggers and Harold Bekkering. Ocular gaze is anchored to the target of an ongoing pointing movement. Journal of Neurophysiology, 83(2):639–651, 2000.pubmed.ncbi.nlm.nih.gov/10669480
  44. [44]Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. Eye tracking for everyone. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  45. [45]Jackson Beatty. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2):276–292, 1982.pubmed.ncbi.nlm.nih.gov/7071262
  46. [46]Olaf Dimigen, Werner Sommer, Annette Hohlfeld, Arthur M. Jacobs, and Reinhold Kliegl. Coregistration of eye movements and EEG in natural reading: Analyses and review. Journal of Experimental Psychology: General, 140(4):552–572, 2011.DOI
  47. [47]Raja Parasuraman and Matthew Rizzo, editors. Neuroergonomics: The Brain at Work. Oxford University Press, 2007.DOI
  48. [48]Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), 2018.arxiv.org/abs/1704.06888
  49. [49]Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision (ECCV), 2022.arxiv.org/abs/2108.05877
  50. [50]Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.arxiv.org/abs/2203.12601
  51. [51]Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. VRB: Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2304.08488
  52. [52]Yichen Zhu, Zhicai Ou, Feifei Feng, and Jian Tang. Any2Policy: Learning visuomotor policy with any-modality. In Advances in Neural Information Processing Systems (NeurIPS), 2024.DOI
  53. [53]Chen Wang, Fei Xia, Wenhao Yu, Tingnan Zhang, Ruohan Zhang, C. Karen Liu, Li Fei-Fei, Jie Tan, and Jacky Liang. Chain-of-Modality: Learning manipulation programs from multimodal human videos with vision-language-models. In IEEE International Conference on Robotics and Automation (ICRA), 2025. doi: 10.1109/ICRA55743.2025.11128270.DOI
  54. [54]Ruoxuan Feng, Di Hu, Wenke Ma, and Xuelong Li. Play to the score: Stage-guided dynamic multisensory fusion for robotic manipulation. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2408.01366
  55. [55]Remi Cadene, Simon Alibert, Alexander Sochorn, et al. Lerobot: State-of-the-art machine learning for real-world robotics in PyTorch. https://github.com/huggingface/lerobot, 2024.
  56. [56]Bowen Wen et al. FoundationStereo: Zero-shot stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2501.09898
  57. [57]Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao, Kaitong Cai, Chengkai Jin, Chunxiao Liu, Jianbo Liu, Siyuan Huang, Xingang Pan, et al. ACE-Ego-Hand: Repurposing video diffusion models for occlusion-robust egocentric 3D hand motion recovery. arXiv preprint arXiv:2608.20308, 2026.arxiv.org/abs/2608.20308
  58. [58]aigc apps. VideoX-Fun: A video generation pipeline for diffusion transformer. https://github.com/aigc-apps/VideoX-Fun, 2026.
  59. [59]Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
  60. [60]Innata Team. Innata-0: A native embodied world system. Technical report, Beijing Academy of Artificial Intelligence, 2026. To appear; hand reconstruction appendix, Section D.
  61. [61]Gen Li et al. EgoM2P: Egocentric multimodal multitask pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
  62. [62]Pantheon Industries. Argus: An open-source annotator for robotics data. https://github.com/Pantheon-Industries-Inc/argus, 2026.
  63. [63]Qwen Team. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026.

Paper details

vlnc:W_qxeqdpgu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents