WorksComputer Science
HumanCOIN: A 12,500-Hour Dataset for Human Intent in Egocentric Manipulation
Abstract
First-person manipulation datasets treat the observation-action pair as the atomic unit and compress the generation and execution of an action into a policy network's weights, rendering the intent state behind the action unobservable. Human motor control research shows that intent mediates between observation and action and leaves observable traces along the intent pathway from decision to effect: gaze reaches the target before the hand, muscle load builds before contact, and cortical activity precedes the motor command. To record this pathway at scale, we present HumanCOIN, a first-person manipulation dataset built around human intent: 12,500 hours across 38 scene classes with three tiers of proxies. At the output end, the L1 attention tier (10,000 h) infers gaze from headband camera video, exposing the operator's next object of attention and reach before the hand moves. Midway, the L2 exertion tier (2,000 h) records surface EMG from armbands, making load expectations readable before execution is complete. At the origin, the L3 cognition tier (500 h) records scalp EEG, giving early readouts of task decisions and monitoring. Each tier inherits all proxies from the tiers below. The dataset further provides multi-view RGB images, depth maps and point clouds, 6-DoF head and wrist pose, and step-level action and object annotations. The controlled L3 setting additionally measures gaze, hand pose and finger kinematics directly. Clock and calibration data are released with the dataset so that any read rate can be re-aligned. The three tiers cover three stations, from visual orienting to cortical preparation, making the process behind visual-action pairs available for modeling at dataset scale. Project page: https://flagopen.github.io/HumanCOIN.

Figure 1.
Corresponding authors: Zhongyuan Wang and Guocai Yao ([email protected]); contact: [email protected]
Introduction
Egocentric data has become the primary route from human video to embodied learning. A head-mounted camera records the scene from the operator’s viewpoint, hand pose recovers the motion of the hands, and together they form a visual–action trajectory: what was seen and how the hand moved. This paradigm has scaled from passive everyday video [1, 2] to instrumented manipulation corpora [3, 4, 5] reaching tens of thousands of hours [6, 7], and the resulting data has been used directly to train robot policies [8, 9, 10]. Across all variations in scale and form, the basic unit remains the observation–action pair .
Two broad approaches have organised work around this unit. The first directly models : all available information—visual observations, depth, pose, and language—is subsumed into , and the mapping to is learned from data at scale [11, 12, 13, 14, 15, 16]; the operator’s intent state is compressed wholesale into network weights, rendering it unobservable—when a policy fails, there is no way to localise whether the target was wrong, the force incorrect, or an error signal missed. The second introduces an explicit intermediate and chooses the future: world models and video–action co-models predict a latent representation of the next observation, then generate the action, [17, 18, 19, 20]; however, a predicted future records only the visible outcome of an action, not the operator state that produced it, and therefore diverges from intent precisely where supervision is needed most.

Figure 1 (PDF p. 3). The middle of , and what each route puts there. An action policy models directly, leaving the middle implicit in its weights; a world model inserts a predicted future and factors through it. The third row is not another method: HumanCOIN supplies the middle as proxies for the intent , on a common time base with the action.
Figure 1 sets the two routes against the alternative: both place something in the middle of , and what they place there is either folded away or inferred, never read off the operator. What is missing is not another estimate of that middle but a record of it, and that is what HumanCOIN provides.
Intent cannot be observed directly, but leaves measurable physical traces along the chain from decision to effect—a chain we term the intent pathway—and we denote the operator’s state along this pathway as the latent variable . The standard policy implicitly integrates over , , folding it into weights and rendering it unobservable; recording proxies for would allow both factors—, a proxy model, and , the proxy-conditioned policy—to be trained independently. Consider lifting a cup: gaze settles on the rim before the hand departs, externalising intent as a visual orienting signal ( [21, 22, 23]); forearm muscles then load grip force to the anticipated weight before the cup leaves the surface, encoding a prior over object properties as a motor command ( [24, 25]); and cortical activity has already been building a Bereitschaftspotential for hundreds of milliseconds, with an error-related negativity ready to fire within milliseconds of a mismatch ( [26, 27, 28]). Each proxy precedes and has mature wearable sensors with decades of signal-processing precedent [29, 30]; synchronised recordings exist [31], yet all remain confined to laboratory settings with few participants and short scripted trials. Large-scale synchronised recording of all three proxies in naturalistic manipulation scenes has not been achieved.
We present HumanCOIN, a large-scale egocentric manipulation dataset that records a proxy vector for alongside every visual–action pair, making both and supervisable from data. The corpus comprises 12,500 hours across 38 scene classes, collected at 6 sites. It is structured into three tiers (Figure 2a), one per proxy:
Figure 2. HumanCOIN in one view. (a) The three tiers and how their hours split: 12,500 h in total, at 10,000 / 2,000 / 500 h (80 / 16 / 4%) for L1 / L2 / L3, drawn on a log axis; solid segments are unscripted (open) recording, hatched segments scripted or bench-controlled. (b) Episode length and (c) task depth: stacked kernel densities of episode duration and of subtasks per episode, with the peak and the mean marked. (d) All 38 scene classes, grouped into the 5 zones and a residual class, with the tier split of each class’s hours. (e) The task verbs: the upper panel ranks each tier’s occurrences against vocabulary rank, and the lower panel gives the thirty most frequent verbs on a log occurrence axis. (f) The 635 object classes, ranked by frequency, with the thirty most frequent below, again on a log axis. Every stacked bar is split by tier.
L1, Attention tier (10,000 h). Proxy : multiview egocentric RGB with inferred gaze, recording the output of visual orienting decisions.
L2, Exertion tier (2,000 h). Proxy : bilateral forearm surface EMG, recording the peripheral projection of motor commands; the signal envelope reflects muscle activity from which grip force and load may be inferred before object motion.
L3, Cognitive tier (500 h). Proxy : scalp EEG, recording movement preparation potentials and error-related potentials that precede observable motion.
The tiers are nested in instrumentation: each inherits every proxy of the tier below and takes on one more, each proxy being a suite of instruments rather than a single device. Depth is a design axis, not a separate collection: the families that survive the same canonical filters at every tier recur at all three measurement depths on a shared time base. Scene, objects, posture, and protocol nonetheless move with depth, so a cross-tier comparison is task-matched rather than condition-matched; § 3.1 states what the shared families hold fixed and what they leave open. The three proxies are recorded by the three tiers, and the action itself is captured by the egocentric video every tier carries: remains latent, but is recorded rather than marginalised. Beyond the tier-specific modalities, the dataset provides depth maps and 3D point clouds reconstructed from egocentric views, 6-DoF head and wrist pose with finger kinematics, and step-level action–object annotations, all registered to a single coordinate frame and time axis. L3 serves as a multimodal reference in its controlled setting: there, , head and wrist pose, and finger kinematics are measured directly with dedicated instruments, while their counterparts at L1, at L2, and in the open L3 subset are predicted from multiview video, aligning all three tiers in semantics, coordinate frame, and time base so that the controlled measurements provide per-frame reference targets for shallow-tier predictors.
Relative to existing corpora, HumanCOIN differs not in scale but in coverage: three proxy depths of the intent pathway under one task set and one time base, making the generative process behind each visual–action pair available for modelling rather than only for imitation. The dataset is released under a research licence (CC BY-NC 4.0) together with acquisition software, a data browser and downloader, and a ready-to-use client library. Our contributions are threefold:
A proxy-grounded acquisition paradigm. Three wearable proxies of the intent pathway are recorded alongside first-person manipulation and aligned to a common time base, making measurement depth a design axis (§ 4).
A large-scale intent-pathway dataset. The corpus comprises 12,500 hours over 38 scene classes, organised into three tiers, each contributing one proxy of the pathway, sharing a common task ontology and time base (§ 3).
An open ecosystem for the community. Acquisition software, a data browser and downloader, and an out-of-the-box client library are released with the corpus (§ 4.3).
Related Work
Egocentric Vision Corpora
Egocentric video has been scaled along two axes, hours and annotation density. Ego4D [1] collected 3,670 hours from 931 participants in nine countries and defined benchmarks for episodic memory, forecasting, and social interaction. EPIC-KITCHENS [32] traded breadth for density, annotating cooking with verb–noun segments; its hundred-hour extension, EPIC-KITCHENS-100 [2], added 89K action segments. EGTEA
Gaze+ [33] recorded eye movements over 28 hours of cooking and turned attention into a label. Ego-Exo4D [34] added a synchronized third-person view: 1,286 hours from 740 participants, annotated with skills, keysteps, and proficiency.
Assembly101 [3] instrumented tabletop assembly with 4,321 multi-view videos; ARCTIC [4] captured bimanual manipulation of articulated objects with 2.1M frames and 3D hand–object annotations; HOI4D [35] recorded 2.4M RGB-D frames of category-level hand–object interaction; and HoloAssist [36] annotated 166 hours of human–assistant collaboration, including mistakes and their corrections.
A newer generation raises the bar on geometry and duration: Nymeria [37] captures multimodal daily motion with full-body and hand tracking, HD-EPIC [38] adds 3D object and gaze grounding to kitchen annotation, EgoDex [5] records 829 hours of manipulation with on-device 3D hand tracking, and EgoLife [39] follows six participants for a week. All remain external observations: none records muscle or cortex during the act. EgoSuite-Open100K [6] has opened the first 10,000 hours of a planned 100,000-hour programme of egocentric manipulation, with 3D hand and body pose over 10,000 tasks and 500 environments, and Xperience-10M [7] releases 10,000 hours and 2.88B RGB frames of in-the-wild human experience with depth, camera pose, and captions. Both record the outside of the action at a scale no research corpus reaches. Neither records gaze, muscle, or cortex, and neither pairs the same task with a shallower version of itself.
Physiological and Neural Sensing
Surface EMG has long served as the interface between motor intent and machine. Farina et al. [29] review how motor-unit activity is extracted from the surface signal and used to decode intended movement and force, and Ninapro [40] standardised forearm recordings across subjects. Grip-force studies show that forearm muscles load to the anticipated object weight before the object leaves the surface [24, 25], encoding the operator’s prior over object properties as an observable EMG signal before any visual feedback is available. EgoEMG [41] pairs bilateral forearm EMG with egocentric video for hand pose estimation.
EEG-based interfaces have decoded motor preparation, error monitoring, and cognitive load for decades [30]. Kornhuber and Deecke identified the Bereitschaftspotential, a slow cortical build-up that precedes voluntary movement by hundreds of milliseconds [26], and Shibasaki and Hallett provide a systematic account of its neural basis [27]. Gehring et al. established the error-related negativity (ERN): cortex produces a discriminable response within milliseconds of an incorrect outcome, before any external feedback [28]. Taking EEG outside the laboratory is complicated by motion artefacts; the mobile brain/body imaging paradigm [42] and active-electrode hardware have made wearable acquisition feasible, though signal quality remains below that of seated recordings.
Gaze provides the earliest behavioural signature of intent: it reaches the target of a pointing movement before the hand does [43], and appearance-based trackers [44] made the signal recordable outside the laboratory; in everyday tasks fixations run ahead of the hand and mark the objects about to be acted on [21, 22]; and in object manipulation gaze reaches the contact point before the fingers close [23]. Pupil diameter adds a physiological readout of processing load [45]. When gaze and EEG are recorded together, fixation-related potentials lock cortical activity to the currently fixated object [46], showing that the three signals are mutually constraining rather than independent.
The neuroergonomics programme [47] argued that these measurements belong inside real tasks. Most studies nevertheless remain laboratory experiments: the task is designed to elicit a signal, scenes are controlled, participants few and trials short, so behaviour, muscle, and cortex are rarely recorded on one timeline. EgoBrain [31] pairs a 32-channel EEG headset with egocentric video over 61 hours from 40 participants, the first corpus to put cortex on the same timeline as first-person video; it is seated and scripted, however, covering 29 daily actions in one setting, with no gaze, no muscle, and no scene geometry. Large-scale synchronised recording of all three signals in naturalistic manipulation scenes has not been achieved.
Human Data for Robot Learning
Human video has long been used to teach robots, but the methods differ in what they inherit from the human. TCN [48] learned a self-supervised video representation for imitation, DexMV [49] brought human video and estimated hand pose to dexterous manipulation, R3M [50] pretrained a visual representation on Ego4D and transferred it to manipulation, VRB [51] extracted affordances from human videos as a representation for robot policies, and EgoMimic [8] scaled imitation learning from egocentric human video captured with a portable rig. These methods inherit appearance and affordance from the human, but not the signals that produced the action.
A second line introduces an explicit intermediate between and by choosing the future. UniPi [17] casts the policy as a video diffusion model and reads actions from the generated frames; GR-1 [18] jointly predicts future frames and actions in a transformer; VPP [19] and UVA [20] condition the policy on latent representations of predicted future observations. These models make the gap between and explicit, but the intermediate they supervise is a predicted future observation rather than the operator’s current state: the two diverge when an attempt fails or is abandoned, and neither grip force nor cortical preparation is recoverable from a predicted frame.
A third line adds modalities to the policy itself. Any2Policy [52] trains a modality-agnostic encoder that accepts text, audio, images, or point clouds in any combination and connects them to a policy head. Chain-of-Modality [53] prompts a vision-language model to chain modalities from human demonstration videos—images, armband EMG, and audio—into task plans with explicit control parameters. Play to the Score [54] fuses multiple sensory streams with stage-guided dynamic weighting during manipulation. These works show that extra modalities help a policy, and some already incorporate EMG at the policy end; none, however, is grounded in a corpus that systematically records the operator’s intent proxies across multiple depths under a shared task set and time base.
Data Distribution
HumanCOIN holds 12,500 hours of egocentric manipulation data, organised along three measurement depths: each tier corresponds to one observable proxy , , or , from the visual orienting signal at the pathway’s distal end, through the peripheral motor command, to cortical preparatory activity. The corpus spans 38 scene classes, 99 task verbs, and 635 object classes. The three tiers share a common task ontology and time base; scene and behavioural diversity unfold within each tier.
The Depth Axis
Existing egocentric corpora record the two ends of an operation: the observation gives the visual environment the operator inhabited, and the action gives the motion and effect of the hands—together, what happened. These two ends form the baseline modality set that all existing corpora share. What does not answer is why a particular action occurred in a particular way at a particular moment.
We use intent as a working term for the operator’s task-relevant state, denoted . is neither a synonym for nor an early version of ; it lies between perception and execution and shapes how an action unfolds. For any ,
Equation [1] writes as an integral over : given , the distribution of is determined jointly by the posterior and the action model . It supplies no directly observable quantity: itself is not observed. The question is not whether exists, but whether carries information about beyond , i.e., whether differs from . We do not assume ; the observation remains in the policy.
Because is latent, Equation (1) cannot be used directly for training or evaluation. The corpus records proxies for at three stages along the intent pathway, , not itself. A proxy is a lossy measurement of :
reads out only the part of that the sensor can reach, and is noise; is not required to be invertible, nor to contain all of . The corpus therefore supports a proxy-conditioned policy , not . This distinction runs through the paper: is the explanatory frame, is what the data contain.
is not a sufficient statistic for . The three proxies differ in noise, latency, and coverage; they are not interchangeable, nor are they nested estimates of a single scalar. Together they cover three observable stations, from visual orienting to cortical preparation and monitoring, but they do not exhaust . We do not claim that these proxies identify ; the limitation is discussed in § 6.
HumanCOIN asks one measurement question at each station:
Q1. Where does gaze land before the hand departs, and how early does it carry information about the upcoming action?
Q2. How does the effector respond to the motor command before the action occurs, and what does the EMG envelope carry about grasp type and load?
Q3. What cortical activity precedes the motor command, and what does it carry about preparation and error monitoring?
The three tiers are not independent datasets but a ladder nested in instrumentation: each tier inherits every proxy of the tier below and takes on one more—gaze, EMG, EEG—and every proxy is a suite of instruments, whose members the controlled setting can upgrade from estimation to measurement. Their hours—10,000, 2,000, and 500—decrease with depth, reflecting the higher acquisition cost of deeper proxies.
L1, Attention tier (10,000 h). Fourteen heterogeneous headbands from thirteen vendors record multiview egocentric RGB; gaze is estimated from video and depth is reconstructed from the front stereo pair. L1 is collected in open scenes, where operators move freely and tasks are not scripted. The tier has neither EMG nor EEG.
L2, Exertion tier (2,000 h). The headband is fixed to a single model, and bilateral forearm surface EMG is added. Gaze is still estimated from video; scenes and objects become scripted, and tasks follow a task sheet. The tier has no EEG.
L3, Cognition tier (500 h). The cortical proxy is added, together with the L3 kit: two wrist views and a third-person view, and, in the controlled setting, the eye tracker, the trackers, and the gloves that measure gaze, hand pose, and finger kinematics directly. L3 has two settings: open uses a lightweight EEG configuration in natural scenes; controlled uses a high-density cap at a fixed desk, with imagery before execution. By the split used in Figure 2a, about 350 h of L3 are controlled and 150 h open; only the controlled part supplies direct measurements.
Thus the gaze, hand pose, and finger kinematics of L1, L2, and open L3 are estimates; the same quantities in the controlled L3 subset are measurements. That subset therefore provides the reference measurements against which the estimates can be compared frame by frame on the same episode. “L3 as reference” means that is a reference measurement for , not ground truth for .
The tiers are nested in instrumentation but not in distribution: scene, objects, posture, and protocol move from open to controlled with depth. A cross-tier comparison is therefore task-matched, not condition-matched. What the tiers share is a core of task families: those that survive both canonical filters, and , and across them the task is the same at every depth while the staging is not—L1 instances are unguided, and the object set, step order, posture, and protocol are fixed only from L2 onward. The probe sets and are the increments and need not be shared. What they do not share is a condition: the corpus does not pair a family with one scene, posture, and protocol across tiers. A depth-only comparison—same scene, objects, posture, and task, only the instruments changed—is thus a construction the corpus admits, not a split it releases: one fixes a family and its staging, takes the episodes each tier recorded for it, and reports the residual differences rather than assuming them away.
Beyond modalities, the tiers follow an open-to-controlled gradient in task, scene, and acquisition paradigm: the deeper the tier, the more controlled and the narrower its coverage. Open means the task happens in a natural scene in the operator’s own way—objects occur as they occur, posture is unconstrained, recording runs free. Controlled means the scene, object set, posture, and acquisition protocol are fixed one by one, trading behavioural freedom for reproducibility. The design keeps one task comparable across depths while the corpus spans natural behaviour to controlled conditions.
In task, a family—a goal, an object set, and a dominant action sequence, coarse enough to recur at every tier—enters a tier only after canonical filtering, and each tier adds a probe set targeting the proxy it introduces:
where is the family set at tier , the natural repertoire, and , the two canonical filters: fixes the object set, placement, and step order; additionally fixes posture and phases (imagery before execution in the controlled setting). The probe sets are the tier’s increment: muscle-load tasks at L2—heavy objects, balancing, and tactile discrimination ()—place the effector at a loaded working point; high-cognition tasks at L3—instruction-following, long-horizon stacking, and error-prone insertion ()—impose complex multi-stage demands on the cortex.
Four dimensions move together from open to controlled: scene from fully open to staged to a fixed desk; objects from natural occurrence to a specified set with each item measured for colour, shape, size, material, and weight, so that object weight becomes a known quantity; posture from unconstrained to as still as possible, since electrode contact and eye tracking degrade with movement; and acquisition from free recording to phased protocols with an imagery stage, so that motor preparation is not buried by the movement that follows. Table 1 sets the four dimensions side by side. These four axes co-vary with depth, so they describe a design gradient rather than a controlled experiment; isolating depth requires the depth-only comparison described above.
| L1 | L2 | L3 | |
| Scene | open | staged | staged + desk |
| Objects | natural | specified | specified |
| Posture | free | free | still |
| Protocol | free | phased | phased + imagery |
Table 1. The four axes that separate the three tiers. Each tier inherits every modality class of the tier below; the four axes co-vary with depth, so the tiers form a design gradient rather than a controlled experiment.
Scope. HumanCOIN records proxies for the latent state , not itself. It provides time-aligned multimodal streams, L3 controlled reference measurements , and per-episode calibration and alignment.
Episode Duration and Task Stages
Episode duration determines how much context and how many steps a single sample carries, and it largely constrains the kind of training and evaluation the sample can support. The duration of HumanCOIN is not set uniformly; it varies systematically with tier, task type, and difficulty.
Duration distribution (Figure 2b). The distribution is unimodal with a long tail: the density peaks at 52 s and the mean sits at 88 s, with the shortest episodes near 10 s and an open tail beyond 5 min. The tiers differ systematically. L1 has the widest range and the heaviest tail, at a mean of 101 s; L2 and L3 are more concentrated, at 65 and 60 s. That gap belongs to the recording protocol rather than to the tasks: L1 is segmented after the fact from long continuous recordings, whereas L2 and L3 are scripted episodes with bounded sessions; electrode stability further shortens the deepest sessions. The gap is therefore not evidence about task difficulty, nor about depth as such.
Task stages (Figure 2c). Subtasks per episode count the stages a task is composed of rather than the temporal granularity of annotation. The count is highest at L1, at 11.0 per episode, where natural activity is split into many short steps; it falls to 8.3 at L2 and 7.9 at L3, where a fixed protocol groups the work into fewer, longer stages.
Overall, HumanCOIN shows clear tier differences along both duration and task structure: the shallow tiers hold long, loosely segmented, and diverse everyday activity, while the deep tiers concentrate on scripted, standardized tasks in which the same work is organized into fewer, longer stages. Duration characterizes the recording window, stage count characterizes task structure, and together they form the corpus’s spectrum from natural behaviour to complex controlled tasks.
Scene Distribution
The value of egocentric data depends largely on how far it generalizes in the real world, so the scene distribution should not be trimmed for convenience of collection. The HumanCOIN scene vocabulary holds 38 classes, grouped into 5 zones—residential (9), commercial (10), professional (5), industrial (6), and public (7)—plus 1 residual class. Figure 2d arranges them by zone and by tier, with the hours of each class stacked to show how the three tiers split the total; Appendix § 10 lists all 38 with their tier coverage.
A residential centre of gravity. Residential holds 4,521 hours; commercial, 2,588; professional, 1,982; industrial, 1,957; public, 1,211; and the residual, 242. The ordering is not incidental: manipulation is densest where people live and work, and the corpus follows that density rather than flattening it. Within each zone a few classes carry most of the hours—residential runs from the kitchen’s 1,175 hours down to the balcony’s 13, while the industrial zone is led by the factory workshop at 1,168 and the professional zone by the office at 1,129. Across the whole corpus the five largest classes take 39% of the hours, a concentration that mirrors how often those spaces are actually used.
Selective coverage by tier. L1 holds all 38 classes, L2 holds 21, and L3 holds 9, with the remaining 29 absent from L3; the 9 L3 classes account for 33.2% of L1 hours and 50.9% of L2. The scene set therefore contracts at each step—38 21 9, or 100% 55 24% of the vocabulary—which is why the deep tier reads as a deliberate instrumented subsample rather than a miniature of the whole. Its nine classes are weighted toward the office, the dining room, the workroom, and the kitchen—spaces that admit a fixed desk, stable lighting, and a bounded session—while the controlled subset is narrower still, concentrated on the desk.
This is not a sampling bias but a direct consequence of the controlled setting: a high-density cap, an eye tracker, and a fixed desk require a fixed station, stable lighting, and a bounded session, and scenes such as the street, the shop floor, and the farm cannot physically accommodate them. The nine L3 classes are the classes with any L3 hours; the controlled subset is narrower still, concentrated on the desk. L1 is collected in open scenes, L2 in staged ones, and L3 spans both settings. The three tiers therefore differ in what they can record, and the scene distribution is a direct reflection of that difference.
Taken together, the scene distribution expresses a trade-off between diversity and record quality. From L1 to L3, coverage contracts from all 38 indoor and outdoor classes to a concentration on the 9 classes that can be instrumented. The result is a corpus that spans from natural behaviour to complex controlled tasks, with task families that recur across tiers observed at several measurement depths.
Task Distribution
A task entry is the unit of reuse: one captured segment, its ordered step sentences, and the objects it touches. The entries span cooking, assembly, cleaning, errands, and teaching, and differ in step count, object set, and manipulation difficulty; that variation is what allows the corpus to be split by domain, by horizon, or by difficulty instead of being used as one monolithic training set. Each step carries a task verb, the hand that performed it, and the objects it touches, so an entry can be read both as a plan and as a record of execution (Figure 2e). The upper tiers are, with few exceptions, subsets of the lower tiers: scenes are strictly nested (L3 L2 L1); objects and verbs are nested except for a small probe increment—five objects first added at L2 and six verbs first added at L3. Figure 2d–Figure 2f shows the resulting vocabulary split.
Action: 99 task verbs. A task verb names what a step accomplishes as a goal—wipe, open, fold—and is the unit a plan is written in. The vocabulary is open-ended in principle but compact in practice: 99 classes cover the step record, a median of 81 occurrences per class, and the ten most frequent take 60.0% of it. The head is everyday manipulation—wipe, open, fold, close, arrange—above a long tail of rarer verbs, so the same set serves both as a small label vocabulary and as a description fine-grained enough to distinguish what is being done. Behind each step sits a second, finer record: the motion verbs that carry it out—pick, place, move, grasp—46 classes over the motion record, 2.5 per step.
The tiers treat the vocabulary differently. L3 keeps 40 task verbs (40%), and those verbs cover 77.3% of L1 task occurrences. A protocol fixes the task set, so the verb list contracts to the tasks it scripts, while the steps that are recorded stay labelled in full: the deep tier is selective in vocabulary but not in the record.
Object: 635 classes. The object side is longer-tailed still: the ten most frequent classes take about 42.5% of the object occurrences, while the median class occurs only about 130 times in total, and the 46 most frequent classes already carry 80% of the record (Figure 2f). Coverage again separates the tiers: L1 holds 630 classes and L2 386, while L3 keeps 95 and leaves the other 540 absent from L3; those 95 still account for 82.0% of L1 occurrences and 97.3% of L2. L3 is a subset of L2; L2 adds a small probe set of objects absent from L1. The deep tier thus keeps 15% of the vocabulary and more than four fifths of the record—the same contraction seen in the scenes, and for the same reason. Appendix § 12 describes the measured object library—its size, weight, material and colour distributions—and how it is counted against this annotation vocabulary.
Taken together, all three vocabularies are long-tailed, and what the tiers trade is breadth against depth rather than vocabulary size: the shallow tiers spread the full vocabulary over everyday activity, while the deep tier contracts to the classes its protocol needs and concentrates the record there.
Inside an Episode
Figure 3 opens one L3 episode in full: the metric scene in Figure 3a, the camera views and depth map in Figure 3b, the annotation of the selected step in Figure 3c, and the task stages with the intent proxies of the same interval in Figure 3d. It is shown in the controlled L3 setting because that is where the body is instrumented: the gaze, the hands, and the wrists that the other tiers infer from video are measured here.

Figure 3. One controlled L3 episode in full, at s (frame 829/1126). (a) The metric point cloud, coloured by the head-camera RGB, with the head pose, both hand skeletons and their 6-DoF poses overlaid in one shared frame. (b) Eight synchronized camera views arranged two per row: the four head-mounted views, the two wrist views, the metric depth map, and the third-person view. (c) The annotation of the selected instance: the step sentence stack the tomato on the bread, the measured object attributes (name, colour, shape, material, size, weight), the atomic action grasp, and the working hand left. (d) The task stages and the three intent proxies of the same interval: gaze (, ), eight-plus-eight-channel EMG, and sixteen of the 128 EEG channels, on a shared time axis with the current instant marked; the eye reaches the object before the hand and the muscle load rises before contact, so each trace leads the action it announces. The stages are imagery, place bread, add tomato, add lettuce, add patty, and top with bread.
Geometry. The point cloud is reconstructed from the two front views; the head, both wrists, and the finger joints are placed in one metric frame, so a grasp is located on the object rather than only in the image (Figure 3a). In the controlled L3 subset these quantities are measured: three trackers give the head and both wrists, and the gloves give the joints. At L1, at L2, and in the open L3 subset the same quantities are estimated from RGB, with each wrist root anchored in the scene frame through the depth map and the finger joints following in three dimensions. The frame is shared; the evidence behind it is not.
Views. Four egocentric views pair the two front lenses (left-front and right-front) with the left and right views, and at L3 two wrist views add the hands at work (Figure 3b). A depth map is reconstructed on demand from the front pair, and in the controlled setting a third-person view records the operator and the desk from outside. The frames are stored as recorded, with no annotation drawn over the pixels.
Annotation. The selected instance is shown in Figure 3c: each step carries a short description, a task verb, the hand that executed it, and the objects it touched, and at L2 and L3 the object record adds a photo, a hex colour, a shape, a material, a weight, and a size given in the parameters its shape calls for (radius and height for a cylinder, for instance). For the step in the panel the atomic action is grasp and the working hand left.
Intent proxies. Each episode is cut into stages, shown as the strip at the top of Figure 3d. The three traces beneath the strip are the proxies for the state that the action is about to express, and each is read as a lead or a lag against the action it announces. Gaze is the eye’s position, measured by the eye tracker in the pixels of its scene camera in the controlled L3 subset and estimated from the rectified head view at L1, at L2, and in the open L3 subset: the eye reaches the object before the hand does, so the trace externalises the target of the next reach—a fixation is a flat stretch, a saccade a near-vertical step, and the moment the eye settles marks what the hand is about to act on (). EMG is eight envelope traces per forearm, flat at rest and rising before contact: the forearm loads grip force to the weight the hand expects, so a burst records the force the operator intends to apply, not only the force applied when the fingers close (). EEG is a stack of channel traces, three orders of magnitude below EMG and drawn on its own scale: the slow build before the movement and the sharp response after an error are the cortical preparation and the outcome check, read as deviations from the ongoing rhythm rather than as spikes (). On one clock, the three answer what the instant means—where the operator is about to reach, how hard they expect to grip, and what was decided before the hand moved.
An episode is therefore one record aligned to a common time base and in a single frame: the views and the point cloud give the scene, the poses, joint angles, and gaze give the body’s place in it, EMG and EEG give the exertion and the cortical preparation behind the movement, and the annotation gives each step its place in the task.
Comparison with Existing Datasets
Table 2 compares a representative set of these corpora on what they record rather than on scale: it is a comparison, not a census, and the corpora that share an axis with a row here are discussed above rather than tabulated. All of them observe the task from the head, and neither a third-person view nor a wrist view (Ego-Exo4D, Nymeria, EgoSuite) adds more than the visible outcome of manipulation. Only EgoBrain carries a neural signal, and it carries just one (EEG); gaze is the only human signal shared by several corpora—Ego4D on a subset, Ego-Exo4D in 99% of takes, EGTEA and Nymeria throughout—and always in isolation, so that where the eye went and how hard the hand pulled have never been observed at the same moment. HumanCOIN records the core families at three instrumentation depths, which is what lets “measured in the controlled L3 setting, predicted from video elsewhere” hold for those families, and a shallow-tier estimator be scored against the dataset’s own reference measurements rather than an external benchmark.
Table 2: Representative prior egocentric datasets against HumanCOIN. ✓ measured, P predicted from video, R reconstructed from video, ✓* subset only, ✓/P measured where instrumented and predicted elsewhere, a cross not provided; 3D is a reconstructed depth or point cloud, Fingers joint-level hand tracking. Cameras counts per recording: head a head-mounted first-person camera, wrist a wrist-mounted one, exo a third-person camera. Frames at 30 fps where only a duration is reported, and the HumanCOIN count is master frames, one per 30 Hz tick. †First 10,000 h open; 2–6 head means the number of head-camera lenses varies between batches, always with a front stereo pair. Shaded rows are HumanCOIN and its three tiers.
| Dataset | Hours | Frames | Cameras | 3D | Fingers | Gaze | EMG | EEG | Subtask anno. | Object anno. |
| Ego4D [1] | 3,670 | ≈396M | 1 head (+224 h multi-cam) | ✓* | ✗ | ✓* | ✗ | ✗ | ✓* | ✗ |
| Ego-Exo4D [34] | 1,286 | ≈139M | 1 head, 4–5 exo | R | ✓* | ✓ | ✗ | ✗ | ✓ | ✗ |
| EPIC-KITCHENS [2] | 100 | 20M | 1 head | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| EGTEA Gaze+ [33] | 28 | 2.5M | 1 head | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ |
| Nymeria [37] | 300 | 201.2M | 1 head, 2 wrist, exo | R | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ |
| EgoDex [5] | 829 | 90M | 1 head | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| EgoSuite-Open100K [6] | 10,000† | 1.08B | 1 head, 1 wrist* | R | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Xperience-10M [7] | 10,000 | 2.88B | 6 head | R | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| EgoBrain [31] | 61 | 6.6M | 1 head | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ |
| HumanCOIN | 12,500 | 1.35B | 2–6 head, 2 wrist*, exo* | R | ✓ / P | ✓ / P | ✓* | ✓* | ✓ | ✓ |
| L1 · Attention | 10,000 | 1.08B | 2–6 head | R | P | P | ✗ | ✗ | ✓ | ✗ |
| L2 · Exertion | 2,000 | 216M | 4 head | R | P | P | ✓ | ✗ | ✓ | ✓ |
| L3 · Cognition | 500 | 54M | 4 head, 2 wrist, exo* | R | ✓* | ✓* | ✓ | ✓ | ✓ | ✓ |
Table 2.
EgoBrain is a complement rather than a replacement. It has already shown that cortex can be recorded on the first-person timeline, but does so in a controlled, seated, scripted setting, trading breadth for signal density: 61 hours, 40 participants, 32 channels, 29 daily actions. HumanCOIN takes the same premise into naturalistic scenes and trades action variety for volume, holding 500 hours at L3. What it adds is two things EgoBrain does not attempt: gaze and surface EMG on a common time base, so that three proxies can be compared at one instant, and scene geometry with hand kinematics, so that an action lands on an object rather than staying inside the image.
Subtask annotation is already common under other names, but object annotation is not: elsewhere an object is a name, while at L2 and L3 every instance is measured for shape, material, size, and weight. That is a difference in capability, not in labelling style—it lets an EMG envelope be read against a measured object weight rather than an estimated one, so that the anticipation of grip force before contact can be inferred from muscle activity rather than only assumed; grip force itself is not recorded.
Hardware Configuration
The intent proxies summarized above do not appear in a recording by themselves. Each becomes a record only through an instrument that converts a physical quantity into a measurable signal, so the kit is a set of measurement proxies: one class of instrument per component of , each chosen for what it can physically reach. Every instrument keeps its own clock, its own native sampling rate, and its own coordinate frame. This section describes what is worn at each tier, the two settings in which the kit is deployed, the two-phase protocol that organizes a controlled session, and how the resulting streams are stored.
The Instrument Set
This subsection is organised by proxy rather than by device: the instruments a tier adds are the reading apparatus for the intent proxy that tier measures, and the criterion for choosing one is single—whether it can physically reach the quantity without altering the behaviour being measured. The three proxies sit differently against that criterion, so the hardware differs, and the order of depth follows from it.
L1, Attention—the proxy. Visual orienting projects onto the external scene as light, so a headband that never touches the operator can record the scene from which gaze is inferred. L1 is not a single device but a collection layer assembled from fourteen third-party headbands supplied by thirteen vendors. All fourteen provide a forward stereo pair (two of them as a side-by-side image split into left and right, and one through a pair of its positioning cameras rather than a dedicated front pair). Seven add fisheye or side views for peripheral coverage. Native resolutions range from (rectified) to , and video rates from 20 to 60 Hz; Table 6 lists the units.
L2, Exertion—the proxy. Action potentials raised at the muscle fibres reach the skin surface by volume conduction, so eight electrodes per forearm over the flexor and extensor bellies suffice, sampled at 2000 Hz. The cost is contact: impedance degrades with sweat and movement. From L2 onward the headband is fixed at four lenses at , which distinguishes it from the heterogeneous L1 units.
L3, Cognition—the proxy. Preparatory cortical activity crosses the skull only as a field potential, small in amplitude and sensitive to movement. The headband and the EMG are identical to L2, and two wrist cameras at 30 fps add the near-hand view the head cannot see. EEG comes in two configurations: an 8-channel cap connected through the headband computer, for the open setting in which the operator moves freely; and a 32- or 128-channel cap connected through the fixed bench host, for the controlled setting in which it is joined by the binocular eye tracker, the three 6-DoF trackers, the 20-node gloves, and the third-person RGB-D view.
The correspondence between channel count and connection is not incidental: the denser the cap, the higher the data rate and the tighter the electrode stability required, and the more the record depends on a fixed station and a still posture. Hence the distinction this subsection turns on. The same proxy is obtained differently under different conditions: gaze, head and wrist pose, and finger kinematics are measured directly, with dedicated instruments, only within the controlled L3 subset; at L1, at L2, and within the open L3 subset, the same quantities are inferred from headband video. “L3 as reference” refers to that controlled record. What the tiers add, instrument by instrument, is the upgrade of an estimate into a measurement, and the preconditions of that upgrade tighten with depth—which is why the deepest tier is also the narrowest.
Two invariants hold across the kit. First, no instrument is daisy-chained through another: a single host receives every stream and stamps arrival offsets, so each device keeps its own clock and native rate, and a failure or a dropped frame in one stream cannot contaminate the others; that host is the headband computer in the open setting and the bench host in the controlled one. Second, phase and run transitions are driven through a parallel-port box into the EEG amplifier as well, so protocol events are stamped on the amplifier’s clock and not only on the host’s. Every camera is stored twice, raw and rectified—the depth view rectified only—and the calibration ships with every episode. Poses share one frame throughout: points left, up, and forward, with millimetres for translation and degrees for joint angles; the alignment and calibration pipeline is described in § 5.1. Figure 4(a)–(Figure 4c) show the three kits in tier order: the headband at L1, one armband on each forearm at L2, and at L3 the same kit with the EEG cap and a wrist camera on each forearm. Only the devices a tier adds are numbered on its operator; the eye tracker, the gloves and the head and wrist trackers are controlled-setting only, which the dashed numbers mark. The full modality matrix is in Appendix § 13, and the fourteen L1 units are audited in Appendix § 14.
Figure 4. Instrumentation by tier. (a) The L1 kit: a four-camera headband. (b) The L2 kit adds two surface-EMG armbands worn on the forearm. (c) The L3 kit adds the two wrist cameras and the EEG cap, and, in the controlled setting only, the eye tracker, the motion-capture gloves, and three 6-DoF trackers (head and both wrists). The devices each tier introduces stand in a row beneath its operator, numbered as on the body, and only the devices a tier adds are marked on that operator; solid numbers are hardware worn in both settings, dashed numbers the hardware used in the controlled setting only. Faces are blurred for anonymity.
Settings and Protocol
A tier’s collection is governed by two requirements: how much of the everyday scene it must preserve, and how much signal fidelity its instrumentation demands. The two pull in opposite directions, and the corpus resolves them tier by tier.
Two settings. L1 and L2 privilege mobility over signal depth. Every instrument they add is wearable and connects to a headband-mounted computer, so the operator moves through everyday environments—standing, turning, walking, and gesturing—without a fixed bench or cable constraints. This is the open setting: L1 and L2 are collected in it throughout, and L3 also uses it, with a lighter configuration, trading EEG density for scene coverage. Open here names the acquisition setting, not the task: an open-setting session can still use a scripted task sheet and a staged scene.
L3 is where the trade-off reverses. Dense electrophysiology requires a stable posture: electrode contact and impedance degrade with movement, and the denser the recording, the less it tolerates motion. eye tracker, the gloves, and the wrist trackers load the same platform, so acquisition calls for a bench that does not move. Sessions run in short blocks, with a re-baseline between blocks to limit drift. L3 is collected under both settings: in the controlled setting the amplifier and the host remain on a dedicated desk and the operator holds still, and the session runs imagery then execution; in the open setting the kit runs on the headband computer in everyday scenes, and the session runs execution only.
Imagery, then execution. The controlled setting also carries the paradigm that makes the EEG informative, built around the high-density record. Imagery is confined to this setting for two reasons: its boundaries are delimited by phase markers that the experimenter records during the session, and the preparatory signal is low in amplitude and spatially structured, so only the dense array separates it from noise. Preparation and motor imagery are brief and are easily masked by the movement that follows, so the session separates imagery from execution (Figure 5). In the imagery phase (Figure 5a) the screen turns blue: the operator faces it, sits still, and rehearses the task sequence without moving, so that motor imagery and preparatory EEG are recorded free of movement artefact; gaze is recorded as well, so that the scanpath of the plan can be read off. In the execution phase (Figure 5b) the screen turns green: the same operator manipulates the objects on the desk, and every modality is recorded, so that the EEG and the other streams can be read against the actual movement. Every transition is also written to the record as a marker—run start, imagine start, execute start, and run end—so that the phases can be cut out of any stream on a common time base rather than by hand.

Figure 5. The two cued phases of a controlled session (e.g., pick up the cup). (a) Imagery (blue screen): the operator sits still, hands on the desk, and rehearses the task with only EEG and gaze recorded. (b) Execution (green screen): the same operator manipulates the objects, head held as still as the grasp allows. The eyes move in both phases, the hands only in the second.
Each controlled episode therefore comprises three segments, not two cues alone: preparation while the operator settles, imagery of the task sequence, and execution of that sequence. Because the first two are acquired while the body is still, the EEG records the task as it is planned rather than as it is performed: the step sequence and the action each step will require, with no movement to mask them. The high-density record exists for that pairing: the same task as a plan, acquired while the body is still, and as a performance seconds later. Open settings (L1, L2, and L3 with the lighter configuration) record the execution segment alone.
Collection procedure. L1 is recorded without guidance: operators are asked only to carry out the activity as they normally would. L2 and L3 use task sheets: the operator arranges the objects as drawn and follows the step order marked on the sheet, so that the same task is staged identically across sessions. At L3, tasks are drawn in random order rather than blocked, which sustains engagement and limits drift of the EEG towards drowsiness. Operators also rotate between scenes at a fixed interval, so that no operator works a single scene for an entire shift and no scene is shaped by an individual’s habits.
All operators are trained to a single standard before collection: how to wear each device, how to run a session, and how to record it. L3 adds two requirements: operators are trained to keep the head still during acquisition and are responsible for seating and maintaining the electrodes so that contact quality stays within range; participants assigned to L3 are also screened so that the EEG is clear and stable. Each hour of collection is followed by fifteen minutes of rest, to sustain attention and energy.
Data Storage
Streams are stored as recorded and linked by reference, never merged into a single file. Each device keeps its own clock, its own native sampling rate, and its own coordinate frame; forcing them into one stream would discard the timing the corpus exists to keep. Synchronization is therefore a property the data declares rather than one it enforces: no resampling and no alignment happen at write time, and the corpus is written as many asynchronous files, each at its own rate.
At write time, every episode keeps one Parquet table per modality (Figure 6a), one file per observation stream: the camera views, EEG, gaze, hand pose, glove skeleton, three device poses (head and both wrists), head and wrist IMUs, and wrist EMG. videos/ holds one encoded video per camera view, calibrations/ holds the rectification parameters and the manifest, annotation/ the structured annotation and the QC verdict, and meta/ the episode metadata with its alignment and statistics reports. Every sample is stamped relative to the start of the recording, and that difference is stored as given, not snapped to a master frame, so non-uniform intervals and device jitter survive into the file. The complete directory layout is in Appendix § 15.
Figure 6. One episode on disk, and how it is read. (a) One Parquet file per stream, each a timestamp column plus data columns, beside a master index. (b) Timestamps are relative to the start of the recording, so no feature sits on a fixed grid; a query fixes a master rate and resolves each feature from its own timestamps. Every camera appears twice under videos/, once raw and once rectified, except the depth view, which is rectified only.
Quality is enforced on the recording, not on the downstream analysis. Every session is checked stream by stream once it is written—dropped or out-of-order samples, black or frozen video, dead channels, impossible value jumps, a device missing from the set—and a session that fails is recaptured rather than repaired, so a released episode never mixes a good half with a patched one. The verdicts travel with the episode in its metadata; Appendix § 16 lists the checks, their thresholds, and what each level costs.
Reading is fully decoupled from writing. A separate master-index file is sampled at a fixed rate, and the reader chooses that rate (30 fps by default); a master frame then indexes every modality through its own timestamps, resolved by nearest sample, interpolation, or all samples in a window such as (Figure 6b). The format is a modification of LeRobot [55] and remains compatible with it: installing pip install humancoin is enough to preview, download, and use the data, and the acquisition software and the documentation are released alongside it.1
Data Processing
Processing runs per stream, because the noise and the failure modes of each modality are set by its own measurement physics; the streams meet only at the end (Figure 7). Each stream is reduced to a timestamp, a unit, a declared frame, and an alignment report, and the five groups follow the dependency order of the data—synchronization and calibration, geometry and motion, gaze, electrophysiology—with annotation last, since it consumes the others. Chains are fixed, their parameters are logged per episode, and a failing step is flagged rather than filled, so any run can be replayed.
Figure 7. Processing pipeline. Each stream enters the group that handles its physics: synchronization and calibration (clock offsets, camera intrinsics, eye calibration); geometry and motion (stereo depth from FoundationStereo, point cloud, 6-DoF poses with finger joints, hand reconstruction from the rectified front view); gaze (acquisition at L3, prediction at L1 and L2); and electrophysiology (band-pass filtering, notch, bad-channel interpolation). The groups then align to a common time base and pass through annotation before an episode is written.
Synchronization and Calibration
We use soft synchronization. The recorder stamps every sample relative to the start of the recording, and the sync host records arrival offsets for the streams that share a transport; the two together place each sample on a common time base without resampling. A per-stream audit of released L3 episodes is reported in Appendix § 17: the high-rate streams align below one sample period, while the 30 Hz cameras are frame-quantized. Calibration (intrinsics and distortion for every camera, the tracker-to-frame transform, and the eye-tracker calibration) ships with the episode in calibrations/, alongside both encodings of every camera, so a reader can re-derive the geometry rather than accept the undistortion applied at write time. Because neither offsets nor calibrations are applied at write time, a reader can align at any custom rate and choose nearest sample, interpolation, or a window. Residuals between the chosen rate and the samples actually read are written to the per-episode alignment report, so a downstream analysis can audit the timing it inherited.
Geometry and Motion
Depth is built from RGB. The left-front and right-front lenses are hardware-timestamp synchronized, so every stereo pair comes from the same instant; the pair, together with the camera parameters, is passed to FoundationStereo [56], which returns a depth map stored with the intrinsics that produced it. Point clouds are formed per frame by projecting depth through those intrinsics and are subsampled to a fixed budget.
Motion follows the same route. In the controlled L3 subset three 6-DoF trackers give the head and both wrists directly (~800 Hz, millimetres, on one shared frame), and the gloves give 20 joint angles per hand (~800 Hz, degrees). At L1, at L2, and in the open L3 subset both hands are reconstructed from video over each continuous segment: the model builds on ACE-Ego-Hand [57] and takes clean-latent features from Wan2.2-Fun-5B-Control [58, 59], and a bidirectional spatiotemporal decoder returns, for every frame, the two wrists and the finger joints in the camera frame, MANO shape and articulation, wrist translation, the intrinsics it assumed, and two confidence fields per hand—whether it is present, and how much of it is visible.2
Before reconstruction, the main view of every unit is rectified to a common pinhole domain, the intrinsics are updated after cropping and resizing, and the analytic rays are generated from the updated intrinsics; a unit’s declared calibration is used where it exists, and the pinhole parameters are predicted from the features where it does not. The intrinsics released with the track are those of this domain, not those of the raw lens. Overlapping windows are fused at their source timestamps: rotations combine in rotation space, one shape per hand and one camera per segment are fixed, and the geometry is regenerated, so that shape and scale do not drift as the visible part of the hand changes—what a user reads is a hand that keeps its size through a grasp. The output is already three-dimensional in the camera frame, so only the wrist root has to be anchored: its pixel is looked up in the depth map and back-projected, and the finger joints, three-dimensional relative to that root, ride along. The depth map remains the metric reference for the scene and the objects. In the controlled L3 subset the same fields are measured rather than inferred, which is what makes that record the reference against which the reconstruction is scored. All poses (head, both hands, hand kinematics) and the environment point cloud are mapped into one unified coordinate frame.
Gaze
Gaze is obtained by acquisition and by estimation. Both routes write the same fields—a timestamp and a pixel coordinate—in the pixels of the image each route reads: the scene camera of the eye tracker in the controlled L3 subset, the rectified head view at L1, at L2, and in the open L3 subset.
In the controlled L3 subset the eye tracker is calibrated per participant and per session and logs at 200 Hz; its coordinates are pixels of the scene-camera image, whose video runs at 30 fps. The calibration that produced each record—the intrinsics and distortion of that camera, with the transform that carries a ray into the head frame—ships with the episode, so a pixel can be back-projected and placed in the head frame. Because L3 records a reference measurement, it also serves as the test bed for the estimator used at L1 and L2.
Outside that subset there is no eye tracker—at L1, at L2, and in the open L3 subset—and the gaze point is predicted by a pipeline built on EgoM2P [61] that runs from video to coordinates. It reads the unit’s rectified left-front view, where the unit provides one—left not because it is more accurate, but because its rectification, depth, hands, and annotation all use that view, so cross-modal alignment needs no extra transform—under the model’s fixed input specification: a 2 s window, 16 frames, and a input. The gaze points of a window are predicted and then linearly resampled to the window’s frame count, so that every frame carries exactly one row; a tail shorter than a window is covered by an additional window aligned to the end of the video, so that the prediction is as long as the video. The output is fixed at 30 Hz and is not interpolated to L3’s 200 Hz: the model’s positional encoding fixes 30 tokens per second, and interpolation would only manufacture precision. The difference ships with the column, and a user who wants another rate can resample. Coordinates are pixels of the video: the model works inside a fixed centre crop, and the pipeline maps its output back, so that a released coordinate indexes the frame a user has rather than a position inside the model’s input. The prediction is written as a table isomorphic to L3’s but registered under a separate feature name, so that measurement and estimate cannot be confused.
Electrophysiology
Both streams are cleaned with a fixed chain. EMG is filtered in zero phase and reduced to an envelope. The trace is mirror-padded at both ends, band-passed between 20 and 450 Hz with second- and fourth-order Butterworth sections applied forward and backward, and comb-notched at 50 Hz with a 2 Hz bandwidth, with periodic extension and period alignment at the edges so that the notch does not ring there. The cleaned trace is rectified and reduced to one envelope per channel, and the raw trace is kept beside it as *_emg_raw. Appendix § 18 lists the orders, cut-offs, and padding steps.
EEG passes through seven. The channel is de-meaned, band-passed between 0.5 and 100 Hz with the high cut held below Nyquist, and notched at 50 Hz, which keeps the delta-to-gamma range in which motorimagery and error-related potentials live. Bad channels are flagged by five joint rules—flat, electrode contact, bridging between neighbours, decorrelation from the rest of the montage, and high-frequency noise—and a session passes only when fewer than 30% of its channels are flagged. An infomax ICA fitted once on the clean segments of a session removes ocular components, selected by correlation with the frontal channels, and muscle components, selected by their power above 30 Hz. Flagged channels are then interpolated by spherical spline over the retained ones, and the cleaned trace is re-referenced to their average. The stages are released as separate tables—eeg_raw as acquired, eeg filtered, and eeg_repair after interpolation and component removal. Appendix § 18 gives the rule thresholds, the component criteria, and the electrode positions that lack a conventional coordinate.
Motion cannot be removed from a recording in which the operator moves, and even the deepest tier is stationary only by protocol (§ 4.2). The cleaned EEG is therefore signal plus residual motion noise: we release the metrics that bound it—impedance pass rate, share of interpolated channels, and share of samples marked artefactual—and leave the choice of a stricter rejection threshold to the user. EMG is recorded at 2000 Hz in millivolts, eight channels per arm; EEG at 1000 Hz over 128 saline channels or at 250 Hz over 32 or 8 gel channels, in microvolts, with every channel named by brain region and its impedance logged in kilohms as a quality input.
Annotation and Segmentation
Every episode is annotated at the level of task steps (Figure 3c): temporally bounded segments carrying the step sentence, the action it names, the hand in use, and every object the sentence reaches. Annotation is produced with Argus [62], an open-source harness for dense annotation of robot-learning data, running on top of Qwen3.8-27B [63]. Argus reasons jointly over video, state, and sensor evidence to produce dense labels: action phases, subgoals, key events, outcomes, operator mistakes and recoveries, and object-state changes, each with an execution and outcome score and with the evidence and confidence behind it. It exports the labels as JSON and as LeRobot language columns, and a human reviewer checks every export before it enters the release. Objects are not mere name strings: for L2 and L3 every object recorded in an episode’s object list was measured for size and weight, and the measurement is linked to the object instances appearing in each trajectory. Collective references (“the cups”) are expanded to their member objects, and hand-in-use is read from wrist-tracker displacement inside the step window when no explicit label exists.
L1 needs a different route. Because passive collection does not intervene, a raw L1 recording is one continuous stream that lasts hours, and episodes are cut after the fact: frames and the available annotation are passed to a VLM, which proposes episode boundaries and step segments, and a human reviewer confirms or adjusts every boundary before release. This is why L1 episodes vary widely in length while controlled L3 sessions are scripted, and why an L1 boundary carries the uncertainty of a segmentation rather than the certainty of a cue.
Discussion
An egocentric corpus pairs an observation with an action and so answers what happened; why the action took that form belongs to the intent that preceded it, which is often settled before execution. HumanCOIN collects that intent, recording proxies for it at three signal depths. Whether it holds up depends on work we have not done; we state its limitations first, its openings second.
Limitations
The premise is untested. That intent sits between observation and action and carries information the observation does not is untested here: the corpus only makes it testable, and policy training, lead-time baselines, and cross-site quality gates lie outside its scope. Nor can the opposite be excluded: intent may be redundant with observation, since gaze is predictable from the scene and preparatory EEG may follow the kinematics. What fails then is not the concept but the proxies; the negative result turns “is intent redundant?” from a position into a measurement.
Coverage and representativeness. L3 accounts for 4% of the hours (500 h); its controlled subset is about 350 h (2.8%) and is the part concentrated on the desk, and electrode stability bounds how long the densest episodes can run. The scene and task mix is not a random sample of manipulation. L1 is assembled from fourteen heterogeneous units whose camera counts, resolutions, calibration, and declared frames differ (Appendix § 14), so how much of an L1 statistic reflects behaviour rather than the device mix cannot yet be separated.
Annotation and comparability. Step boundaries and object attributes are human judgements—L1 boundaries from automatic segmentation with human review, attributes measured only at L2 and L3. Hand motion is estimated at L1, at L2, and in the open L3 subset and measured in the controlled L3 subset, so the estimated part carries a reconstruction error. Comparability holds only in part: timestamps, units, frames, and calibration are released as each source declares them and the declarations are uneven, while quality metrics such as the impedance pass rate exist only at L3.
Long tails. The vocabularies are long-tailed: rare classes have few samples within a tier, cross-tier comparisons are unevenly powered, and the L3 share limits the deep tier’s rare classes, so “the deep tier is more concentrated” holds only qualitatively.
Research Directions
None of these questions can be answered on data alone.
Testing the mediation. One test compares with : if the proxies add nothing, the latter is no worse. Another is a cross-tier ablation; L1 and L3 share the task ontology and differ in depth, so an L1-trained model applied to L3 mixes distribution shift with the proxy difference. The imagery phase supplies a third: it records a plan without its execution, so a model can be asked to read the plan without action evidence.
Evaluating before the fact. With only and a corpus scores after the fact; with intent one can ask before: how the action would change under a different intent, whether the decision stays readable when no action occurs, and whether a model predicts the action or the intent behind it. Such answers do not depend on success, so they retain signal where failures are rare.
Transport. Three depths, several sites, many operators, and fourteen units make transport measurable: which proxies stay calibrated across tiers, which labels stay stable across annotators, and whether any component is independent of the sensor.
Applications. Intent-conditioned policies, earlier failure prediction, corrections with less supervision, and an intent interface for human–robot collaboration follow once enters the model: the pay-off of the chain, not the test of whether it holds.
The corpus succeeds if it makes these questions falsifiable; a negative answer counts as much as a positive one.
Conclusion
An egocentric manipulation corpus usually records two things: the observation and the action . It records what happened, not what was decided—and the decision is what the action is answerable to. HumanCOIN takes that middle term, the intent , as the object of collection: one task ontology and one time base, with proxies for recorded at three signal depths—gaze for attention, EMG for execution, EEG for decision—and three tiers nested in instrumentation but not in distribution: passive and open at L1, active and open at L2, active and controlled at L3. What results is an instrument for testing intent rather than a larger behavioural dataset: task families that recur across tiers observed at several depths, which makes three questions measurable—where intent becomes readable, whether it mediates the action beyond the observation, and whether it transports across tiers.
We do not claim that these questions are answered; the value of the corpus is that they can be falsified, including by the negative result that intent is redundant with observation. The data, the acquisition software, and the documentation are released openly for research use.
Data Statement
All participants gave written informed consent before any recording and could withdraw at any time without penalty. Episodes are released only in de-identified form: no personally identifying information is included, faces are blurred in every released view, and all tiers—L1, L2, and L3—share the same consent and de-identification protocol. Task protocols and instrumentation differ (§ 4.2): L1 is unguided, L2 and L3 use task sheets; L2 adds EMG, and L3 adds EEG and, in the controlled setting, eye tracking and motion capture, all de-identified the same way. Code is released under Apache-2.0 and data under CC BY-NC 4.0.
References
Scene Classes
The corpus spans 38 scene classes in 5 zones—residential (9 classes), commercial (10), professional (5), industrial (6), and public (7)—plus one residual class. Table 3 lists all 38 with their hours in each tier, their share of the corpus, and the tiers in which they appear; Figure 8 shows a recorded frame for each.
| # | Class | L1 (h) | L2 (h) | L3 (h) | Total (h) | Share (%) | Tier |
| Residential — 9 classes, 4,520.7 h, 36.2% of the corpus, 4 in L3 | |||||||
| 1 | kitchen | 868.2 | 271.5 | 35.1 | 1,174.8 | 9.4 | L1–L3 |
| 2 | living room | 759.0 | 43.3 | 0.0 | 802.2 | 6.4 | L1–L2 |
| 3 | dining room | 400.5 | 174.1 | 75.2 | 649.9 | 5.2 | L1–L3 |
| 4 | storage room | 502.8 | 98.4 | 0.0 | 601.2 | 4.8 | L1–L2 |
| 5 | bathroom | 338.9 | 169.2 | 0.0 | 508.1 | 4.1 | L1–L2 |
| 6 | bedroom | 431.3 | 51.2 | 0.0 | 482.5 | 3.9 | L1–L2 |
| 7 | study | 95.4 | 34.0 | 34.5 | 163.9 | 1.3 | L1–L3 |
| 8 | laundry area | 69.4 | 25.0 | 30.7 | 125.1 | 1.0 | L1–L3 |
| 9 | balcony | 13.0 | 0.0 | 0.0 | 13.0 | 0.1 | L1 |
| Commercial — 10 classes, 2,588.1 h, 20.7% of the corpus, 2 in L3 | |||||||
| 10 | retail store | 355.0 | 157.4 | 32.7 | 545.1 | 4.4 | L1–L3 |
| 11 | logistics facility | 401.0 | 127.9 | 0.0 | 528.9 | 4.2 | L1–L2 |
| 12 | entertainment venue | 472.8 | 0.0 | 0.0 | 472.8 | 3.8 | L1 |
| 13 | restaurant | 197.7 | 28.0 | 0.0 | 225.7 | 1.8 | L1–L2 |
| 14 | beverage shop | 141.2 | 32.0 | 29.7 | 202.9 | 1.6 | L1–L3 |
| 15 | pet shop | 202.6 | 0.0 | 0.0 | 202.6 | 1.6 | L1 |
| 16 | hotel | 113.0 | 38.4 | 0.0 | 151.3 | 1.2 | L1–L2 |
| 17 | performance venue | 123.8 | 0.0 | 0.0 | 123.8 | 1.0 | L1 |
| 18 | hair salon | 78.4 | 0.0 | 0.0 | 78.4 | 0.6 | L1 |
| 19 | beauty salon | 56.5 | 0.0 | 0.0 | 56.5 | 0.5 | L1 |
| Professional — 5 classes, 1,982.3 h, 15.9% of the corpus, 2 in L3 | |||||||
| 20 | office | 789.2 | 177.1 | 162.4 | 1,128.6 | 9.0 | L1–L3 |
| 21 | classroom | 338.9 | 12.5 | 0.0 | 351.4 | 2.8 | L1–L2 |
| 22 | laboratory | 215.7 | 37.5 | 34.1 | 287.3 | 2.3 | L1–L3 |
| 23 | healthcare facility | 200.3 | 0.0 | 0.0 | 200.3 | 1.6 | L1 |
| 24 | conference centre | 14.7 | 0.0 | 0.0 | 14.7 | 0.1 | L1 |
| Industrial — 6 classes, 1,956.7 h, 15.7% of the corpus, 1 in L3 | |||||||
| 25 | factory workshop | 857.1 | 311.1 | 0.0 | 1,168.2 | 9.3 | L1–L2 |
| 26 | workroom | 387.6 | 110.0 | 65.5 | 563.2 | 4.5 | L1–L3 |
| 27 | industrial warehouse | 126.6 | 0.0 | 0.0 | 126.6 | 1.0 | L1 |
| 28 | construction site | 74.5 | 0.0 | 0.0 | 74.5 | 0.6 | L1 |
| 29 | equipment room | 15.6 | 0.0 | 0.0 | 15.6 | 0.1 | L1 |
| 30 | testing laboratory | 8.7 | 0.0 | 0.0 | 8.7 | 0.1 | L1 |
| Public — 7 classes, 1,210.6 h, 9.7% of the corpus, 0 in L3 | |||||||
| 31 | park & garden | 247.6 | 17.7 | 0.0 | 265.3 | 2.1 | L1–L2 |
| 32 | public facility | 215.3 | 0.0 | 0.0 | 215.3 | 1.7 | L1 |
| 33 | street | 193.7 | 0.0 | 0.0 | 193.7 | 1.5 | L1 |
| 34 | farm | 150.7 | 19.7 | 0.0 | 170.4 | 1.4 | L1–L2 |
| 35 | indoor sports venue | 157.6 | 0.0 | 0.0 | 157.6 | 1.3 | L1 |
| 36 | sports area | 118.4 | 0.0 | 0.0 | 118.4 | 0.9 | L1 |
| 37 | transportation | 90.0 | 0.0 | 0.0 | 90.0 | 0.7 | L1 |
| Other — 1 class, 241.6 h, 1.9% of the corpus, 0 in L3 | |||||||
| 38 | other | 177.5 | 64.1 | 0.0 | 241.6 | 1.9 | L1–L2 |
Table 3. Table 3: Scene classes in full. Hours are recorded hours by tier; total is the class total over all tiers; share is the percentage of the 12,500 h corpus; tier gives the tiers in which the class appears. Zone headings give the class count, the zone’s hours and share, and the number of classes in L3. Values are rounded to 0.1 h, so rows and columns may not sum exactly.

Figure 8. Head-camera frames from the scene vocabulary: one frame for each of the 38 classes, labelled with its class name and ordered by zone (square colour) and by recorded hours within a zone. Per-class hours and tier coverage are in Table 3.
What a class is. A class names the physical setting in which an episode was recorded, not the activity performed there: the same task may occur in several classes, and a class hosts many tasks. Every episode carries a single scene label, so the class hours partition the corpus and sum to 12,500. The five zones are functional strata for reporting and stratified use, not a hierarchy—they group settings by primary function (living, commerce, work, production, and public life)—and two classes in one zone need not be closer than two in different zones. The residual class is a label like the others, not missing data: it holds 241.6 h, 1.9% of the corpus, and appears at L1 and L2 but not L3.
Boundaries. The label follows the setting in which the hands act for the greater part of the episode; crossing a doorway does not change it. The pairs that carry most of the ambiguity are resolved by function, not by furniture: kitchen against dining room separates preparation and cleanup from eating and serving; office against study separates a workplace from a home workspace, so the same desk task is labelled by the setting; storage room against industrial warehouse separates domestic from industrial storage, and hotel against bedroom a commercial guest room from a home bedroom. In the public zone, street is outdoor circulation, park and garden managed green space, and public facility an indoor public building; restaurant is a sit-down meal and a beverage shop a drinks counter.
The nine L3 classes. The deep tier does not spread evenly over the vocabulary. Office (162.4 h), dining room (75.2), workroom (65.5), and kitchen (35.1) carry 67.6% of its 500 h; the other five are study (34.5), laboratory (34.1), retail store (32.7), laundry area (30.7), and beverage shop (29.7). The nine span four zones—residential, commercial, professional, and industrial—and no public-zone class reaches L3. The set is therefore concentrated but not confined to one room type: two of its nine classes are shops.
Verb Lexicon
Every step carries two labels rather than one. A task verb names the goal the step accomplishes and is the unit a plan is written in; the motion verbs recorded inside the step name what the hand does to reach that goal, 2.5 of them per step. The two vocabularies are disjoint—no verb appears in both layers—and they behave in opposite ways: the task layer is open and long-tailed, the motion layer closed and concentrated. Table 4 lists both in full, with a boundary statement for each verb, the tiers in which it is observed, and its total occurrences; per-tier counts ship with the release.
| # | Verb | Gloss | Tiers | Share (%) |
| Task layer — 99 verbs; share of all task annotations | ||||
| 1 | wipe | clean a surface with a cloth kept in contact | L1–L3 | 12.39 |
| 2 | open | move a closure so the container or space becomes accessible | L1–L3 | 9.93 |
| 3 | fold | bend a flexible object back onto itself | L1–L3 | 7.75 |
| 4 | close | return a closure to its shut position | L1–L3 | 5.75 |
| 5 | arrange | set several objects into an intended spatial order | L1–L3 | 5.47 |
| 6 | remove | take an object away from its current place | L1–L3 | 5.26 |
| 7 | throw | release an object so it travels through the air | L1, L2 | 4.87 |
| # | Verb | Gloss | Tiers | Share (%) |
| 8 | organize | group objects into an intended arrangement or container | L1–L3 | 3.26 |
| 9 | pour | tip a vessel so its contents fall into a target | L1–L3 | 2.75 |
| 10 | unfold | open out something previously folded | L1, L2 | 2.56 |
| 11 | collect | gather scattered objects into one place | L1–L3 | 2.55 |
| 12 | hang | suspend an object from a support | L1–L3 | 2.41 |
| 13 | insert | place a part into a receptacle and seat it | L1–L3 | 2.13 |
| 14 | stack | place objects one on top of another | L1–L3 | 2.08 |
| 15 | clean | remove dirt from a surface or object | L1–L3 | 2.02 |
| 16 | cover | place something over an object to hide or protect it | L1, L2 | 1.82 |
| 17 | wrap | enclose an object in a flexible sheet | L1–L3 | 1.34 |
| 18 | cut | divide an object with a blade | L1–L3 | 1.32 |
| 19 | secure | fix an object so it cannot move | L1–L3 | 1.31 |
| 20 | tear | divide a flexible object by pulling it apart | L1 | 1.20 |
| 21 | install | mount a unit in its working position and fix it there | L1–L3 | 1.17 |
| 22 | spray | release a liquid from a nozzle over a target | L1, L2 | 1.13 |
| 23 | rinse | run water over an object to carry residue away | L1, L2 | 1.11 |
| 24 | sort | distribute objects among classes by an attribute | L1–L3 | 1.02 |
| 25 | tie | fasten with a cord drawn into a knot | L1–L3 | 0.97 |
| 26 | tighten | turn a fastener until the joint stops yielding | L1–L3 | 0.94 |
| 27 | sweep | push loose material along a surface with a brush | L1–L3 | 0.91 |
| 28 | attach | join a part to a host so that the two move together | L1–L3 | 0.90 |
| 29 | seal | close an opening so that it no longer passes air or liquid | L1, L2 | 0.88 |
| 30 | unscrew | turn a threaded fastener out of its seat | L1–L3 | 0.84 |
| 31 | stick | fix a flat item to a surface by adhesion | L1, L2 | 0.75 |
| 32 | coil | wind a flexible line into concentric loops | L1, L2 | 0.67 |
| 33 | screw | turn a threaded fastener into its seat | L1–L3 | 0.58 |
| 34 | disassemble | separate an assembly into its parts | L1–L3 | 0.56 |
| 35 | wash | clean an object with water and agitation | L1, L2 | 0.53 |
| 36 | check | inspect an object against an expected state | L1–L3 | 0.52 |
| 37 | assemble | join parts into a working whole | L1–L3 | 0.51 |
| 38 | connect | join two ends so that a link passes between them | L1, L2 | 0.47 |
| 39 | replace | put a new part where the old one was taken out | L1–L3 | 0.36 |
| 40 | dry | remove moisture from a surface by wiping or by air | L1, L2 | 0.36 |
| 41 | thread | pass a line through an eye or an opening | L1, L2 | 0.32 |
| 42 | write | deposit marks on a surface with a held implement | L1–L3 | 0.32 |
| 43 | untie | undo a knot and free the cord | L1, L2 | 0.27 |
| 44 | peel | strip an outer layer from an object by hand | L1, L2 | 0.25 |
| 45 | rearrange | move objects already placed into a new order | L1–L3 | 0.25 |
| 46 | button | fasten a garment through its buttonhole | L1, L2 | 0.24 |
| 47 | separate | part two joined objects so that they no longer touch | L1–L3 | 0.23 |
| 48 | apply | spread a substance over a surface | L1–L3 | 0.22 |
| 49 | scrape | draw a hard edge along a surface to lift material off | L1, L2 | 0.22 |
| 50 | cap | fit a cover onto an opening | L1–L3 | 0.22 |
| 51 | untangle | separate a snarl of flexible material into free ends | L1, L3 | 0.21 |
| 52 | reassemble | rejoin the parts of an assembly taken apart earlier | L1, L2 | 0.21 |
| 53 | scrub | clean a surface with a brush under pressure | L1, L2 | 0.18 |
| 54 | scoop | lift loose material with a cupped tool | L1 | 0.17 |
| 55 | bind | fasten several objects together as one bundle | L1, L2 | 0.17 |
| 56 | wind | turn a flexible line around an axis | L1 | 0.17 |
| 57 | unzip | open a zip fastener | L1, L2 | 0.17 |
| 58 | slice | divide food into thin flat pieces | L1, L2 | 0.16 |
| 59 | crumple | compress a sheet into an irregular ball | L1 | 0.16 |
Table 4.
The task layer. 99 verbs over the annotated steps. The ten most frequent take 60.0% of the record and the median verb occurs 81 times, but the tail is long: 28 verbs occur 25 times or fewer, down to uncap at 4. Coverage varies by tier—98 verbs are observed at L1, 79 at L2 and 40 at L3; 34 appear in all three and 15 in only one. What a protocol removes is informative. Fifty-nine verbs are absent at L3, and the ones that matter for the protocol fall into three groups: four are cleaning (spray, rinse, wash, dry), six are cord or closure work (seal, stick, thread, untie, coil, connect), and four are whole-object actions (throw, unfold, cover, peel); tear, which never reaches L2, is absent for a simpler reason. A scripted desk task does not throw, unfold or wash.
The motion layer. 46 verbs over the motion annotations, and effectively a closed set: the four commonest—place, grasp, pick, move—take 82.2% of all motion annotations and the ten most frequent take 92.5%, against 35.8% and 60.0% for the same ranks in the task layer. Every one of the 46 verbs occurs at L1. Twenty-four occur in all three tiers, one (tilt) at L1 and L3 only, and 21 are absent at L3, falling into three groups: all six fine-contact verbs (pinch, poke, feel, rub, tap, squeeze), all five transport verbs (carry, drag, slide, transfer, bring), and ten posture or alignment verbs (push, lay, spread, roll, stand, dip, lean, bend, extend, retract). What drops out is contact without motion, and transport over distance.
The two layers together. The task verb is the sentence a plan is written in and the motion verbs are the record of how the hand executed it, and the tiers treat the two differently: L3 keeps 40 of 99 task verbs (40%) but 25 of 46 motion verbs (54%), and the motion verbs it keeps carry 95.2% of L1’s motion annotations against 77.3% for the task verbs. The decomposition is also finer where the record is deepest—2.50 motion verbs per step at L1, 3.00 at L3—so the deep tier says less about what was done and more about how.
Table 4: The verb lexicon in full. A task verb names the goal a step accomplishes; a motion verb names what the hand does inside the step. Glosses state the boundary used by the annotators, tiers give where the verb is observed, and share is each verb’s percentage of the annotations in its layer; percentages may not sum to 100 because of rounding.
| # | Verb | Gloss | Tiers | Share (%) |
| 60 | staple | fix sheets together with a metal staple | L1, L2 | 0.15 |
| 61 | zip | close a zip fastener | L1, L2 | 0.15 |
| 62 | draw | trace a line on a surface with a held implement | L1, L3 | 0.15 |
| 63 | fill | put material into a container up to a level | L1, L2 | 0.13 |
| 64 | erase | remove marks from a surface by rubbing | L1, L3 | 0.11 |
| 65 | unplug | withdraw a plug from its socket | L1, L2 | 0.11 |
| 66 | test | operate an object to see whether it works | L1–L3 | 0.10 |
| 67 | stamp | press an inked face onto a surface | L1 | 0.09 |
| 68 | wring | twist a wet cloth to press the water out | L1, L2 | 0.08 |
| 69 | loosen | turn a fastener until it no longer holds | L1 | 0.08 |
| 70 | stir | move a tool through a mixture to blend it | L1, L3 | 0.08 |
| 71 | drape | lay a flexible sheet over a form so that it hangs | L1 | 0.07 |
| 72 | load | put an object into a carrier for transport | L1, L2 | 0.07 |
| 73 | discard | put an object into waste | L1, L3 | 0.06 |
| 74 | tuck | push an edge of flexible material under or into | L1 | 0.06 |
| 75 | unwrap | remove a sheet that encloses an object | L1 | 0.06 |
| 76 | detach | take a part off its host | L1, L2 | 0.06 |
| 77 | trim | cut a small excess from an edge | L1, L2 | 0.06 |
| 78 | divide | split one quantity into parts | L1, L2 | 0.06 |
| 79 | unroll | unroll a wound sheet or line | L1, L2 | 0.06 |
| 80 | punch | make a hole by pressing a tool through | L1 | 0.06 |
| 81 | reattach | attach a part again after it was detached | L1, L2 | 0.06 |
| 82 | scan | pass a reader over a code to register it | L1 | 0.05 |
| 83 | plug | insert a plug into its socket | L1, L2 | 0.05 |
| 84 | tape | fix something with adhesive tape | L1, L2 | 0.05 |
| 85 | disconnect | undo a link between two ends | L1, L2 | 0.04 |
| 86 | switch | change the state of a control | L1, L2 | 0.04 |
| 87 | wet | apply water to an object | L1, L2 | 0.04 |
| 88 | pry | lever an object open with a rigid tool | L1, L2 | 0.04 |
| 89 | reinsert | insert a part again after removal | L1, L2 | 0.03 |
| 90 | clamp | hold an object with a device that grips it | L1, L2 | 0.03 |
| 91 | dispense | release a measured amount from a container | L1, L2 | 0.03 |
| 92 | pack | put objects into a container for storage | L1, L2 | 0.03 |
| 93 | brush | clean a surface with a brush | L1, L2 | 0.03 |
| 94 | pair | bring two items together as a matched set | L1 | 0.02 |
| 95 | polish | rub a surface until it shines | L1, L2 | 0.02 |
| 96 | soak | leave an object in liquid for a period | L1, L2 | 0.02 |
| 97 | dice | cut food into small cubes | L1 | 0.02 |
| 98 | sew | join material with needle and thread | L1 | 0.02 |
| 99 | uncap | take a cap off its opening | L3 | 0.01 |
Table 11.
Motion layer — 46 verbs; share of all motion annotations
| # | Verb | Gloss | Tiers | Share (%) |
| 1 | place | put an object down at a target pose | L1–L3 | 26.75 |
| 2 | grasp | close the hand on an object so that it can be held | L1–L3 | 20.75 |
| 3 | pick | lift an object off its support | L1–L3 | 19.99 |
| 4 | move | change the position of a held object without releasing it | L1–L3 | 14.67 |
| 5 | reach | carry the empty hand toward a target | L1–L3 | 3.26 |
| 6 | push | move an object away by force applied through contact | L1, L2 | 1.88 |
| 7 | adjust | nudge an object to correct its pose | L1–L3 | 1.83 |
| 8 | transport | carry a held object between places | L1–L3 | 1.21 |
| 9 | turn | rotate a held object about a vertical axis | L1–L3 | 1.12 |
| 10 | pull | draw an object toward the body by force | L1–L3 | 1.02 |
| 11 | lift | raise an object against gravity | L1–L3 | 0.80 |
| # | Verb | Gloss | Tiers | Share (%) |
| 12 | hold | keep an object in the hand without moving it | L1–L3 | 0.70 |
| 13 | release | open the hand so that the object leaves it | L1–L3 | 0.69 |
| 14 | spread | lay something out over an area | L1, L2 | 0.67 |
| 15 | roll | turn an object over on its side | L1, L2 | 0.54 |
| 16 | lay | set an object down flat on a surface | L1, L2 | 0.53 |
| 17 | align | bring an object into line with a reference | L1–L3 | 0.49 |
| 18 | flip | turn an object over to its other face | L1–L3 | 0.47 |
| 19 | straighten | remove a bend from a flexible object | L1–L3 | 0.38 |
| 20 | press | push down on a surface through a small area | L1–L3 | 0.22 |
| 21 | squeeze | compress a soft object in the hand | L1, L2 | 0.22 |
| 22 | position | bring an object to a specified place without releasing it | L1–L3 | 0.21 |
| 23 | reposition | move an object already placed to a new place | L1–L3 | 0.20 |
| 24 | carry | move an object while supporting its weight | L1, L2 | 0.18 |
| 25 | stand | set an object upright on its base | L1 | 0.18 |
| 26 | dip | lower an object into a liquid and take it out again | L1, L2 | 0.16 |
| 27 | rotate | turn an object about its own axis | L1–L3 | 0.15 |
| 28 | shake | move an object to and fro rapidly | L1–L3 | 0.12 |
| 29 | drop | let an object fall from the hand | L1–L3 | 0.11 |
| 30 | bring | move an object toward a person or target | L1, L2 | 0.06 |
| 31 | drag | move an object along a surface without lifting it | L1, L2 | 0.05 |
| 32 | lower | move an object downward | L1–L3 | 0.04 |
| 33 | tilt | incline an object from its upright pose | L1, L3 | 0.04 |
| 34 | slide | move an object along a surface in contact with it | L1, L2 | 0.04 |
| 35 | retract | draw the hand back from an object | L1, L2 | 0.03 |
| 36 | tap | strike a surface briefly with the fingers | L1, L2 | 0.03 |
| 37 | transfer | move an object from one hand or place to another | L1, L2 | 0.03 |
| 38 | orient | turn an object so that a chosen face points at a target | L1–L3 | 0.03 |
| 39 | lean | rest an object against a support at an angle | L1 | 0.03 |
| 40 | adjust_grip | change the placement of the fingers on a held object | L1–L3 | 0.03 |
| 41 | rub | move the hand back and forth over a surface | L1, L2 | 0.02 |
| 42 | bend | deform an object along its length | L1 | 0.02 |
| 43 | pinch | grip a small object between two fingers | L1, L2 | 0.02 |
| 44 | poke | push a finger into an object or an opening | L1, L2 | 0.02 |
| 45 | extend | straighten the arm or the fingers outward | L1 | 0.01 |
| 46 | feel | touch an object to judge its state | L1 | 0.01 |
Table 12.
Object Library
Every object an episode touches is an entry of a measured library: 839 instances under 442 distinct names, of which 829 are active and 10 retired, the retired ones shipped with a flag. An entry records a name, a shape together with the parameters that shape calls for, a primary and an optional secondary colour in hexadecimal, a material, a weight, a photograph, and a UUID. The UUID is the only key that joins an entry to the object list written with an episode, so the per-instance table ships with the release and is not printed here. The library grew with collection—35 entry dates between September and November 2026—rather than being assembled in advance.
Shape. Twelve primitives are in use (Figure 9a). Cuboid and cylinder alone account for 715 instances, or 85%; the other ten cover 124, the largest being the frustum (54, the open vessels: cups, bowls, pots, bins) and the torus (18, objects with a through-hole: tape rolls, washers, a bracelet). Size is stored as the parameters the shape calls for—length, width, and thickness for a cuboid, diameter and height for a cylinder or a frustum, three axes for an ellipsoid—so one schema serves a mug, a box, and a sheet. Lengths are read with a caliper and weights on a scale, and both are written to one decimal place, 0.1 cm and 0.1 g, which is the resolution the table carries.
Figure 9. The object library. (a) The twelve shape primitives and their percentage share of the library; the axis is logarithmic, and the solid beside each bar is the primitive. (b) Colour of every instance in CIELAB, lightness against chroma; the dotted line at separates the achromatic entries. (c) Longest dimension. (d) Weight, with quartiles dotted and the median dashed. (e) Weight of the ten commonest materials, ordered by median, with each material’s percentage share in brackets. .
Size. By longest dimension the median instance is 15 cm, with quartiles at 8.6 and 23 cm. 86 instances are shorter than 5 cm—screws, washers, nuts, the parts a hand manipulates at the fingertips—and 54 reach 40 cm or more, the furniture and fabric at the other end. Between them, 84% of the library falls in 5–40 cm, the range in which a seated operator reaches and grasps (Figure 9c).
Weight and material. Weight spans six orders of magnitude, from 0.02 g (a plastic shaving) to 20 kg (a shelving unit), with a median of 42 g and quartiles at 7 and 172 g. 29% of the instances are under 10 g and only 38, some 4.5%, exceed 1 kg: the library is one of small, light objects (Figure 9d). Plastic accounts for 307 entries (37%), paper for 148, metal for 141, together about seven in ten. Weight separates by material (Figure 9e): glass and ceramic are highest, at medians of 470 and 326 g, paper lowest at 3.6 g, and plastic and metal both near 40–50 g but with long tails. At a given size it is the material that decides whether an object can be lifted in one hand, which is why it is recorded. The three commonest shape–material pairs are cuboid–plastic (168), cuboid–paper (139), and cylinder–metal (50).
Colour. Colour is the dominant value sampled from the photograph, in hexadecimal, with an optional second value. Mapped to names the library is greyscale-dominated—grey 25%, white 15%, black 13%, brown 11%, transparent 11%—and chromatic objects are only 35% of the total. In CIELAB (Figure 9b) most instances sit in the low-chroma band (), and saturated reds and yellows are rare. Colour is therefore a weak discriminator between objects in one scene, which is why shape and size are recorded beside it.
Counting. The class count of § 3.4 counts the object classes named by annotators in a step; this appendix counts the physical instances that were measured. Each measured instance carries a UUID that also appears in the object list written with every episode that used it; a class reaches the library only through the instances its steps name. The two counts are therefore not the same number: a class may be realised by several measured instances, and one instance may be named by more than one class.
Hardware Bill of Materials
Table 5: Stream specifications of the in-house kit, which records L2 and L3. Every camera records RGB and runs at 30 fps; every view is released raw and rectified, the depth view rectified only; the third-person view exists only in the controlled setting. L1 records on third-party headbands, audited unit by unit in Appendix § 14. The pose and hand streams are released as fused outputs at about 800 Hz; the trackers and the glove that feed them run at their own device rates.
Table 5 lists the in-house instruments that record L2 and L3, with the resolution, channel count, rate, and unit of every stream; the L1 headbands are third-party and are audited in Appendix § 14. The headband is in-house; the remaining devices are commercial and are used as delivered.
| Stream | Device (model) | Resolution / channels | Rate | Unit |
| Head RGB ×4 | four-camera headband, Ego Headband V1.5 (in-house) | 1920×1200 each; left-front, right-front, left, right; clock hardware-synced | 30 fps | RGB |
| Wrist RGB ×2 | JIERUWEITONG FH868 | 640×480 | 30 fps | RGB |
| Scene camera | Pupil Labs Neon | 1600×1200 | 30 fps | RGB |
| Third-person view | Intel RealSense D455f | 640×480 | 30 fps | RGB |
| Gaze | Pupil Labs Neon | pixel in the scene-camera image (L3), in the rectified head view (L1–L2) | 200 Hz | px |
| Head / wrist pose ×3 | HTC VIVE Tracker 3.0 | one shared frame: + left, + up, + forward | ~800 Hz | mm |
| Finger kinematics | MANUS Metaglove Pro Haptics | 20 joint angles per hand | ~800 Hz | deg |
| Surface EMG | WLEMGUV2A armbands ×2 | 8 channels per arm | 2000 Hz | mV |
| EEG (saline) | Neuroscan SynAmps2 (Curry) | 128 channels, brain-region labels, impedance logged | 1000 Hz | V |
| EEG (gel) | BrainCo BCIGo | 32 or 8 channels, brain-region labels, impedance logged | 250 Hz | V |
Table 5. Stream specifications of the in-house kit, which records L2 and L3.
Headband. Four synchronised lenses sit on a rigid frame with a fixed baseline, so the stereo pair used for depth keeps its geometry across sessions; an inertial unit and the acquisition computer are mounted behind them, and the battery is carried at the back of the band. The band is fitted to the head with a replaceable inner liner, which also keeps the lenses at a repeatable height and angle.
Wrist kit. Each wrist carries an armband with eight dry electrodes on the palm side. At L3 the armband is paired with a camera on a strap aimed along the hand, and the two are cabled together so the forearm does not pull on the camera; the pair is worn over the glove, and at L2 only the armband is worn, directly on the forearm.
Cognition kit. The L3 kit adds a binocular eye tracker with its scene camera, three 6-DoF trackers (head and both wrists), a pair of 20-node gloves, and an EEG cap. Two cap types are used: a 128-channel saline cap for the controlled setting and a 32- or 8-channel gel cap for the open setting; both are seated and checked for impedance before each block.
Synchronisation hardware. A UDP marker is emitted by the recorder, and phase and run transitions are additionally driven through a parallel-port box into the EEG amplifier, so protocol events are stamped on the amplifier clock as well as on the host clock. Cameras, headband, and eye tracker are calibrated before each session against a printed target; the resulting intrinsics, distortion, and tracker transforms ship with the episode.
Consumables and spares. Saline, gel, syringes, electrode paste, spare straps, liners, and batteries are restocked per shift; a failed cap or camera is swapped and the session restarted rather than repaired in place, so that a released episode never mixes two calibrations.
L1 Source Audit
Why an audit. L1 is the only tier whose hardware is not ours, and it is not one device: it is a collection layer of fourteen headbands from thirteen vendors. Everything the corpus reports for L1 is conditioned on what those units physically provide—the view gaze is inferred from, the pair that depth is reconstructed from, the rate the hands are estimated at—so the layer is only interpretable if that condition is written down unit by unit. The audit records it, so that a result can be read against the unit it came from and the layer can be reproduced.
What is recorded. Four properties are taken from each unit rather than from its documentation. The lens set and the direction of every view are read off the device. Native resolution and frame rate are measured on the recorded stream, not on the specification sheet. Whether the main view is a genuine stereo pair is checked on synchronized frames: twelve of the fourteen provide one, two of them as a single side-by-side image that the pipeline splits into left and right, and one through a pair of its positioning cameras (up, down, left, and right) rather than a dedicated front pair; units 2 and 7 are fisheye arrays without a stereo pair. The stream is then classified as raw or pre-processed, because the domain matters downstream: one unit delivers only rectified frames, one delivers the pair already split, and one adds an extension view beside the pair.
What the variation implies. Pairing decides how a depth estimate is formed, and the pairs differ—a dedicated front rig, a side-by-side image split after the fact, or two positioning cameras—so the audit records the configuration each episode rests on, and a result that depends on the pair is stated with the unit it came from. Optics trade coverage against geometry: the seven units with fisheye or side views see further around the operator, but every unit is rectified to a common pinhole domain before estimation, so the peripheral views buy coverage rather than accuracy, and their corners are lost at rectification (§ 5). Rate and resolution set the domain in which every L1 estimate lives: from at 20 Hz to at 60 Hz. Nothing is resampled to a common grid at write time (§ 4.3), so an estimator trained on one unit is trained in that unit’s domain, and the audit is what allows the domain to be declared. Table 6 lists the units, numbered by collection volume, with two packaged sets from one vendor taking consecutive numbers; vendor, model, and toolchain names are withheld, so the table documents capability rather than procurement.
Episode File Layout
Every episode is self-contained and follows a chunked hierarchy compatible with a modified LeRobot v3.1 release. The directory is named after the session (collector, date, clock). Every camera is released twice, raw and rectified, and the electrophysiology is released both raw and cleaned. The example in Figure 10 holds 1,444 master frames (48.1 s), 17 videos, one audio track, and 42 Parquet tables.
Signals. Each stream is one Parquet file on its own time base, with a timestamp column in seconds from episode start and an episode_index; columns are named after channels (FP1...), joints (node00_x...), or dim_0... for EMG, so no stream is snapped to the master grid. The pose and hand streams are fusion outputs and carry the timestamps of the fused track rather than of a single device. Pixels are not stored in Parquet: observation.images.*.parquet lists the timestamp of every frame and the frames themselves live in the matching mp4 (H.264, 30 fps), one for the raw view and one for its rectified counterpart; depth ships rectified only. Audio is 16 kHz mono PCM in wav, with its Parquet file recording the sample count per master frame. The names are literal: eeg_raw is the record as acquired, eeg eeg_repair the trace after bad-channel interpolation and component removal, while each wrist EMG ships as a raw trace beside its envelope. Table 7 lists the shape and the fields of every signal table. @page 35/41 starts_mid=0 ends_mid=0
| Stream | Shape | Fields |
eeg_raw | [?] | 132 electrodes with conventional scalp labels (Fp1. . . ), as acquired |
eeg | [?] | the same electrodes after a 0.5–100 Hz band-pass and a 50 Hz notch |
eeg_repair | [?] | the same electrodes after bad-channel interpolation and component removal |
eeg_impedance | [?] | per-electrode impedance in kΩ plus Trigger, pass_rate, gate_pass, n_snapshots |
eye_gaze | [2] | x, y in scene-camera pixels |
eye_imu | [6] | ax, ay, az, gx, gy, gz |
head_pose | [6] | x, y, z, roll, pitch, yaw in the shared frame |
hand_pose | [40] | 20 joint angles per hand (left_ThumbMCPSpread, . . . ) |
hand_skeleton | [?] | 50 nodes, each x, y, z, roll, pitch, yaw |
hand_left_points | [63] | wrist and 20 joints, each x, y, z |
hand_right_points | [63] | the same fields for the right hand |
headband_imu0 | [6] | ax, ay, az, gx, gy, gz |
headband_microphone | [?] | 16 kHz audio, 320 samples per master frame |
left_wrist_emg_raw | [8] | eight forearm channels, as acquired |
left_wrist_emg | [8] | the same channels, rectified and enveloped |
left_wrist_imu | [6] | ax, ay, az, gx, gy, gz |
left_wrist_pose | [6] | x, y, z, roll, pitch, yaw |
right_wrist_emg_raw | [8] | channels of the right forearm, as acquired |
right_wrist_emg | [8] | the same channels, rectified and enveloped |
right_wrist_imu | [6] | ax, ay, az, gx, gy, gz |
right_wrist_pose | [6] | x, y, z, roll, pitch, yaw |
marker | [1] | protocol code: run start, imagine, execute, run end |
state | [58] | both wrist poses and the head pose (18) plus the 40 joint angles |
Table 7. The 23 signal tables: per-frame shape and the fields each one carries.
Table 6: L1 source audit. Vendor, model, and toolchain names are withheld; units are numbered by collection volume, and two packaged sets from the same vendor take consecutive numbers. Views give the direction of each lens, with fisheye marked explicitly. Cameras counts the physical lenses of the unit; Views lists the released views. The two differ where one lens contributes more than one view (a side-by-side pair split into left and right, or an extension view) or where a lens is not released as a separate view.
| Unit | Cameras | Main resolution | Views | fps |
| 1 | 6 | stereo | left-front, right-front; four fisheye around | 60 |
| 2 | 6 | six fisheye: front and side views; no stereo pair | 30 | |
| 3 | 2 | side-by-side stereo | left-front, right-front; extension view | 30 |
| 4 | 2 | stereo | left-front, right-front | 30 |
| 5 | 6 | rectified stereo | left-front, right-front; four fisheye around | 20 |
| 6 | 6 | single RGB | front; positioning cameras up, down, left, right | 30 |
| 7 | 6 | six equidistant fisheye; no stereo pair | 30 | |
| 8 | 3 | front; down, side | left-front, right-front, down, side | 30 |
| 9 | 2 | left-front, right-front | 30 | |
| 10 | 2 | left-front, right-front | 30 | |
| 11 | 2 | side-by-side, split after processing | left-front, right-front | 30 |
| 12 | 2 | left-front, right-front | 30 | |
| 13 | 6 | main stereo | left-front, right-front; left outer, right outer | 30 |
| 14 | 2 | left-front, right-front | 30 |
Table 6.
Figure 10. Episode file layout. One episode directory, L3-example/, with its tables, videos, audio, calibration, annotation, and metadata. <signal> expands to the 23 stream tables, <view> to the 17 camera views (each with a timestamp table and one mp4 per encoding, raw and rectified, except depth, which is rectified only), and <stream> to the microphone stream.
Table 6: L1 source audit. Vendor, model, and toolchain names are withheld; units are numbered by collection volume, and two packaged sets from the same vendor take consecutive numbers. Views give the direction of each lens, with fisheye marked explicitly. Cameras counts the physical lenses of the unit; Views lists the released views. The two differ where one lens contributes more than one view (a side-by-side pair split into left and right, or an extension view) or where a lens is not released as a separate view.
| Unit | Cameras | Main resolution | Views | fps |
| 1 | 6 | stereo | left-front, right-front; four fisheye around | 60 |
| 2 | 6 | six fisheye: front and side views; no stereo pair | 30 | |
| 3 | 2 | side-by-side stereo | left-front, right-front; extension view | 30 |
| 4 | 2 | stereo | left-front, right-front | 30 |
| 5 | 6 | rectified stereo | left-front, right-front; four fisheye around | 20 |
| 6 | 6 | single RGB | front; positioning cameras up, down, left, right | 30 |
| 7 | 6 | six equidistant fisheye; no stereo pair | 30 | |
| 8 | 3 | front; down, side | left-front, right-front, down, side | 30 |
| 9 | 2 | left-front, right-front | 30 | |
| 10 | 2 | left-front, right-front | 30 | |
| 11 | 2 | side-by-side, split after processing | left-front, right-front | 30 |
| 12 | 2 | left-front, right-front | 30 | |
| 13 | 6 | main stereo | left-front, right-front; left outer, right outer | 30 |
| 14 | 2 | left-front, right-front | 30 |
Table 6 (PDF p. 35).
Figure 10 (PDF p. 35). Episode file layout. One episode directory, L3-example/, with its tables, videos, audio, calibration, annotation, and metadata. <signal> expands to the 23 stream tables, <view> to the 17 camera views (each with a timestamp table and one mp4 per encoding, raw and rectified, except depth, which is rectified only), and <stream> to the microphone stream.
eeg_repair the trace after bad-channel interpolation and component removal, while each wrist EMG ships as a raw trace beside its envelope. Table 7 lists the shape and the fields of every signal table.
Metadata. info.json records the codebase version, master feature, fps, path templates, chunk size, and the shape and channel names of every feature. episodes.jsonl and tasks.jsonl map indices to the task string and episode length. collect_info.jsonl stores collector, hardware, scene, the object list (colour, shape, dimensions, material, pose), and free-text notes. alignment_check.jsonl reports, per feature, the tolerance, number of readings, violations, and maximum offset, and episodes_stats.jsonl stores per-feature min, max, mean, and standard deviation.
Calibration and annotation. calibrations/ holds what the tables reference: the reference cameras with per-view rate, frame count, size, and intrinsics, and the P1/P2 projections the rectification applied. annotation/ holds the structured annotation—scene, objects, timed subtask segments, tasks, key events, state changes—and the QC gate with its checks and warnings.
Reading. The reader fixes a master rate (30 fps by default) and walks master_index; each row gives a time, and every other stream is resolved from its own timestamps by nearest sample, linear interpolation, or all samples in a window such as (−1/FPS, 0]. Nothing is stored on the master grid, so a stream can be re-read at any rate without degradation: a query that needs raw EMG reads the 2 kHz table directly, while a query that needs one frame per 33 ms reads the index and never touches the other tables.
Quality Control
Every session is checked after it is written and before it enters the release. The checker walks each recorder’s output directory—the NPZ arrays, the MP4 containers, and the logs—and returns, per stream, a verdict with its evidence attached; no stream is re-recorded and nothing is repaired. A stream that fails is not patched: the whole session is recaptured, so that a released episode never mixes a good half with a repaired one.
Alignment comes first. All checks run inside the analysis window bounded by the RUN_START and RUN_END markers. The pair is a hard precondition: if it is missing, empty, or unpaired, the session is reported as an error rather than checked against the whole recording, because without the window every per-stream crop and comparison is undefined. The same rule applies to the clock: the EEG stream is accepted only when the amplifier clock was fitted to the host clock, and a failed fit is an error, not a warning.
Two verdicts reach the release. Each check yields a finding at one of three levels, of which two matter downstream. An error means the data cannot be used—a corrupt container, a frame count that disagrees with the timestamps, a missing marker pair, a tracker that was absent when the recorder opened—and the session is recaptured. A warning means the data is usable but carries a documented defect: a 2 s frozen stretch, a dead channel, a tracker that lost sight for part of the window, a gap shorter than the threshold for an error. Informational findings, such as the phone-to-host clock correction, never gate anything. Thresholds are the checks’ own defaults and are reported with the observed value.
What is checked. The checks run across the nine stream families the kit produces, and each is an independent, thresholded verdict: a modality declares the checks that apply to it, and a shared context loads the NPZ once and decodes each video once, so the checks compose without repeating work. Table 8 lists the families and the default thresholds. Timing checks ask whether a series is usable at all (present, finite, ordered, distinct), whether the cadence is steady, whether an interval jumped far above the median, whether samples are missing, and whether the rate held inside a one-second window—the last of these catches a drop that a mean would hide. Signal checks ask whether the values are plausible: NaN fractions, channels that never move, robust outliers, physical bounds, and motions no body could make. Modality checks are specific to one instrument’s failure modes: the EEG amplifier’s clock fit, fixed-size blocks and header sample count, the EMG armband’s shared 8-bit serial counter, the glove skeleton dropping out window by window, the tracker count and per-tracker validity, and the camera set the headband promised at stream start. Video checks decode every file once and compare the container’s frame count and pixel statistics against the timestamps, which is the only way to see a writer that fell behind silently; the container test uses the same fprobe criterion as the packaging gate. Session checks close the window: the RUN_START/RUN_END pair and the recorder logs.
| Check | Applies to | Rule (default) | Level |
| Timing | |||
| Presence and order | every series | missing series, no samples, or NaN = ERROR; backward = WARN; duplicate = WARN (ERROR for EMG) | ERROR / WARN |
| Gap | every series | interval > max(0.1 s, 1.5× median), including head and tail against the window edges | ERROR |
| Jump | every series | interval > 3× median | WARN |
| Jitter | every series | interval CV > 0.3 | WARN |
| Windowed rate | every series | any 1 s window outside ±50% of the median sample count | WARN |
| Expected rate | series with a nominal rate | mean rate < 50% of nominal | WARN |
| Sample count | EEG | deviation from window × nominal rate > 1% | WARN |
| Signal values | |||
| NaN and dead channels | EEG, EMG, hand | NaN > 0.1%; zero-variance channel | WARN |
| Robust outliers | EEG, EMG, hand, headband IMU | > 0.1% beyond median ±20 MAD | WARN |
| Value bounds | eye gaze | gaze outside [0, 2000] px | WARN |
| Check | Applies to | Rule (default) | Level |
| Implausible jumps | hand skeleton, VIVE | a glove node moving m or a tracker m between samples | WARN |
| Modality-specific | |||
| EEG clock and blocks | EEG | clock fit failure = ERROR; block discontinuity = WARN; header count mismatch = WARN | ERROR / WARN |
| EMG sequence | EMG | lost frames = WARN; duplicate timestamps = ERROR | WARN / ERROR |
| Hand dropout | hand pose | skeleton NaN in a 1 s window | WARN |
| VIVE validity | VIVE trackers | valid fraction = WARN; any invalid sample = ERROR; tracking gap s = ERROR; teleport m = WARN; device count = ERROR | ERROR / WARN |
| Headband camera set | ego headband | missing camera = ERROR; extra camera = WARN | ERROR / WARN |
| Video | |||
| Container and count | every MP4 | ffprobe failure = ERROR; container frame count differs from timestamp count = ERROR | ERROR |
| Black and freeze | every MP4 | mean luminance for s = WARN; black = ERROR; frame difference for s = WARN | ERROR / WARN |
| Session | |||
| Marker window | session | missing RUN_START/RUN_END pair = ERROR; out-of-order markers = WARN | ERROR / WARN |
| Recorder logs | every stream | lines matching ERROR, FAILED, Traceback, or crash | WARN |
Table 8. Quality-control families and default thresholds. A row is a family of checks; where its sub-checks differ in level, both are listed. A session’s outcome is the worst finding across its streams and its session-level checks, and ERROR blocks packing. Thresholds are code defaults, overridable in configs/checker.yaml; not every check runs on every stream.
The verdict travels with the episode. The checker writes one report per session and its summary is copied into the episode metadata; an optional self-contained page renders the same findings on one time axis for review. Each finding names the check that produced it, the field or file it concerns, the sub-stream it belongs to where there is one (a tracker serial, a channel index), the threshold and the observed value, and the time span of the problem; a warning is therefore a pointer to the affected seconds, not a blanket statement about the recording.
Recapture and what it costs. Because an error retires the whole session, the pass rate is a property of the kit rather than of the analysis: more than 99% of L1 recordings, 96% of L2 and 78% of L3 pass. The gradient is the instrumentation. L1 has a single headband; L2 adds two armbands whose electrodes must hold contact through sweat and movement; L3 adds the EEG cap, which must reach impedance on enough channels, the gloves and three trackers, each able to be absent at open or lose sight mid-window, and the eye tracker, whose clock must be fitted. The deepest tier, which is also the smallest, is therefore the tier most often repeated.
Before recording. On the floor, four checks—camera labels against the live tiled view, wrist-camera seating, EMG contact, and tracker visibility—must pass before a session starts; any re-plugged or swapped device triggers all four again, and the passing screen is photographed and filed, so a session without evidence counts as unchecked. A disk-space gate refuses a recording below 50 GB and warns below 100 GB.
What the gate does not cover. The automated layer tests data integrity, not semantics. It does not verify annotation boundaries, task labels, object attributes, or calibration accuracy beyond the presence of a fitted clock; those remain with the annotation review. It is also per session, so cross-session consistency—one operator, scene, or unit behaving differently across sessions—is outside its scope. The checks are meant to separate a usable recording from an unusable one, not to rank two usable recordings against each other.
Synchronization Audit
Measurement. Synchronization is audited against the protocol markers rather than inferred from the recorder. For each audited L3 episode we take the six phase cues (INSTR_ON, INSTR_OFF, IMG_START, IMG_END, EXEC_START, EXEC_END) and, for every stream, the sample nearest each cue; the event-locked residual is the nearest sample time minus the cue time. EEG is split by amplifier: Neuroscan 128-channel at 1 kHz and BrainCo 32/8-channel at 250 Hz.
Reading the table. For regular sampling at period , the nearest-sample residual lies within , and a uniform residual has an interquartile range (IQR) of about . An IQR close to the half-period therefore indicates that the residual is set by sampling quantization rather than by a clock offset; the normalized column makes this comparison explicit. A value near one is the quantization floor, and a value well above one indicates additional timing irregularity.
Results. Table 9 shows that the high-rate streams reach the sampling floor. The 1 kHz EEG has a median residual of +0.10 ms and an IQR of 0.47 ms, against a half-period of 0.50 ms; the 2 kHz EMG bands have IQRs of 0.27 and 0.22 ms against 0.25 ms; and the fused 480–940 Hz pose stream has an IQR of 1.04 ms against about 0.80 ms, its half-period at the median rate of 628 Hz. Their IQR/half-period ratios are 0.93, 1.08, 0.87, and 1.30, with no systematic offset beyond sampling quantization. The 250 Hz EEG (IQR 2.22 ms) and the 200 Hz gaze (IQR 2.29 ms) are quantized at their 4 ms and 5 ms sample periods; their ratios of 1.11 and 0.91 place them at the same floor, and they should not be read as less well synchronized.
| Stream | Rate (Hz) | (ms) | Offset (ms) | IQR (ms) | IQR/ | P95 (ms) | Max (ms) | Drift (ms/min) |
| EEG, Neuroscan 128-ch | 1000.0 | 0.50 | +0.10 | 0.47 | 0.93 | 0.46 | 0.48 | +0.35 |
| EEG, BrainCo 32/8-ch | 250.2 | 2.00 | −0.47 | 2.22 | 1.11 | 2.38 | 3.19 | n.r. |
| EMG, left | 2000.0 | 0.25 | 0.00 | 0.27 | 1.08 | 0.35 | 0.45 | +0.09 |
| EMG, right | 2000.0 | 0.25 | −0.05 | 0.22 | 0.87 | 0.34 | 0.44 | +0.15 |
| Pose (VIVE + Manus) | 480–940 | 0.80 | 0.00 | 1.04 | 1.30 | 1.04 | 1.66 | n.r. |
| Gaze (Neon) | 199.8 | 2.50 | −0.03 | 2.29 | 0.91 | 2.47 | 3.93 | n.r. |
| Head RGB | 30.0 | 16.67 | −1.67 | 20.36 | 1.22 | 25.42 | 27.14 | n.r. |
Table 9. Event-locked synchronization residuals on released L3 episodes. Each episode contributes six protocol cues; EEG is split by amplifier (Neuroscan, 1 kHz; BrainCo, 250 Hz). Residual is the nearest released sample minus the cue timestamp; IQR is the interquartile range; P95 and Max are absolute. The normalized column divides the IQR by half the sampling period, so values near one are at the quantization floor. n.r. marks drift not resolvable over a 40 s episode.
Drift. The residual trend is +0.35 ms/min for the 1 kHz EEG (median; episode range 0.03–0.92) and +0.09/+0.15 ms/min for the two EMG bands (episode range −0.60 to +0.34); the EEG and EMG residuals stay within ±1 ms/min (±17 ppm) over the episode. For the remaining streams (250 Hz EEG, 200 Hz gaze, 30 Hz camera, and the fused pose output) the sample period exceeds the drift resolvable in 40 s, and the table marks drift as not resolvable. The audit supports the synchronization claim of § 5.1: each device keeps its own clock, and the released streams are aligned to a common time base through markers and arrival offsets, not through a shared hardware clock.
Timestamp integrity. The high-rate streams are regular: EEG at both configurations has no duplicate timestamps and no interval beyond five median intervals, and EMG has no duplicate timestamps. The fused pose stream is less regular: across the audited episodes it contains 39 duplicate timestamps and six intervals beyond five median intervals, and its rate ranges from 480 to 940 Hz (median 628 Hz) rather than the nominal 800 Hz. The 30 Hz camera streams are the least regular: the head RGB stream in these episodes contains twelve duplicate timestamps, and every camera container holds 24–72 frames more than its timestamp table (28–67 for the head RGB stream). The excess is consistent with pre- and post-roll frames outside the episode window; those frames are not represented in the Parquet table, so a reader should index the episode through the timestamp table rather than through the container frame number.
Electrophysiology Processing
Every stage below runs on the raw stream, and its parameters are logged with the episode; the chain is fixed across sessions so that the same code can be replayed on any release.
EMG. (1) Mirror padding: the trace is reflected about its first and last sample, filtered, and cropped back to its original length, which suppresses the transient a filter otherwise leaves at the boundary. (2) High-pass, 20 Hz, second-order Butterworth: removes baseline drift and part of the motion artefact, and a Butterworth response is maximally flat in the passband. (3) Low-pass, 450 Hz, fourth-order Butterworth: bounds the analysis band; a digital filter cannot undo aliasing already introduced at sampling. (4) Zero-phase filtering: the sections are applied forward and then backward, so the interior response has zero phase and the magnitude is the square of the one-way response, an order doubled and a steeper roll-off at the cut-off. (5) Comb notch, 50 Hz with a 2 Hz bandwidth (): periodic nulls at the mains and its harmonics; the backward pass deepens and widens each null, and also attenuates genuine EMG in the same bands. (6) Periodic extension: the edge periods are repeated so that the comb has boundary data, which holds only as far as the edges represent the interference. (7) Period alignment: anomalous edge periods are detected and replaced by interior ones, which rewrites boundary samples and can touch real transients. (8) Endpoint damping: decaying edge energy is detected and cross-faded toward the interior, which removes residual ringing at the cost of possibly attenuating a genuine activation. The cleaned trace is then rectified and low-pass filtered into one envelope per channel.
EEG. The chain is DC removal by the per-channel median, a 0.5–100 Hz band-pass with the high cut clamped below Nyquist, and a 50 Hz notch, followed by four stages. Bad channels are flagged by five joint rules: flat ( with more than 70% of sample differences below ); contact (robust of above 3.5, or above 3.0 with ); bridging (a neighbouring pair whose electrical distance stays below for more than half of the 2 s epochs and below half the reference of their other neighbours); decorrelation (maximum against any other channel below 0.4 in more than 20% of 1 s windows); and high-frequency noise (robust above 5 of the 30–100 Hz to 1–30 Hz MAD ratio). Frontal channels are excused a contact flag when no other rule fires. A session passes when fewer than 30% of its channels are flagged. ICA is infomax with up to 32 components, capped at the number of retained channels, fitted once on the clean segments of a session concatenated in time, with flagged channels held out; ocular components are those correlated with the frontal proxy channels, at most two, and muscle components those whose 30–100 Hz power ratio is a robust above the other components, capped at 20% of the retained channels. Interpolation is spherical spline over the retained channels, and the cleaned trace is finally re-referenced to their average.
Montage and regions. Every channel of the release carries a conventional scalp label and the region group of its row in Table 10. The 128-channel saline cap covers the full conventional set plus the intermediate positions, including the mastoids and the two cerebellar electrodes; the 32-channel gel cap is a subset of it with two auxiliary channels; and the 8-channel gel cap keeps two frontal-central, two central, one midline parietal, and three occipital sites. Where a position lacks a conventional coordinate, Cb1, Cb2, and Cbz take the coordinates of I1, I2, and Iz on the colin27 template, matching the CURRY labels, and F11, F12, FT11, and FT12 take the coordinates supplied with the cap. Both sets ship with the release and are used only for neighbour finding and interpolation, never for source modelling.
| Index | Group | Channels | |
| 128-channel saline cap (Neuroscan) | |||
| Index | Group | Channels | |
| 1–64 | core electrodes, incl. M1/M2 and Cb1/Cb2 | 64 | Fp1, Fpz, Fp2, AF3, AF4, F11, F7, F5, F3, F1, Fz, F2, F4, F6, F8, F12, FT11, FC5, FC3, FC1, FCz, FC2, FC4, FC6, FT12, T7, C5, C3, C1, Cz, C2, C4, C6, T8, TP7, CP5, CP3, CP1, CPz, CP2, CP4, CP6, TP8, M1, M2, P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO3, POz, PO4, PO8, O1, Oz, O2, Cb1, Cb2 |
| 65–77 | Prefrontal and frontal extension | 13 | AFp1, AFp2, AF7, AF5, AFz, AF6, AF8, AFF5h, AFF3h, AFF1h, AFF2h, AFF4h, AFF6h |
| 78–79 | Lateral frontal | 2 | F9, F10 |
| 80–87 | Fronto-central extension | 8 | FFT7h, FFC5h, FFC3h, FFC1h, FFC2h, FFC4h, FFC6h, FFT8h |
| 88–91 | Lateral fronto-temporal | 4 | FT9, FT7, FT8, FT10 |
| 92–99 | Fronto-temporo-central extension | 8 | FTT7h, FCC5h, FCC3h, FCC1h, FCC2h, FCC4h, FCC6h, FTT8h |
| 100–107 | Centro-temporo-parietal extension | 8 | TTP7h, CCP5h, CCP3h, CCP1h, CCP2h, CCP4h, CCP6h, TTP8h |
| 108–115 | Parietal extension | 8 | TPP7h, CPP5h, CPP3h, CPP1h, CPP2h, CPP4h, CPP6h, TPP8h |
| 116–117 | Lateral parietal | 2 | P9, P10 |
| 118–121 | Parieto-occipital extension | 4 | PPO3h, PPO1h, PPO2h, PPO4h |
| 122–127 | Occipital extension | 6 | PO9, PO5, PO1, PO2, PO6, PO10 |
| 128 | Midline cerebellar | 1 | Cbz |
| 129–130 | Ocular | 2 | VEOG (vertical), HEOG (horizontal) |
| 131 | Cardiac | 1 | EKG |
| 132 | Muscular | 1 | EMG |
| 32-channel gel cap (BrainCo) | |||
| 1–7 | Frontal and prefrontal | 7 | FP1, FP2, F3, F4, F7, F8, Fz |
| 8–10 | Central | 3 | C3, C4, Cz |
| 11–15 | Parietal | 5 | P3, P4, P7, P8, Pz |
| 16–17 | Occipital | 2 | O1, O2 |
| 18–19 | Temporal | 2 | T7, T8 |
| 20–23 | Fronto-central | 4 | FC1, FC2, FC5, FC6 |
| 24–27 | Centro-parietal | 4 | CP1, CP2, CP5, CP6 |
| 28–29 | Fronto-temporal | 2 | FT9, FT10 |
| 30–31 | Temporo-parietal | 2 | TP9, TP10 |
| 32–33 | Non-EEG (ocular, trigger) | 2 | IO, TRIG |
| 8-channel gel cap | |||
| 1–2 | Fronto-central | 2 | FC3, FC4 |
| 3–4 | Central | 2 | C3, C4 |
| 5 | Midline parietal | 1 | Pz |
| 6–8 | Occipital | 3 | O1, Oz, O2 |
Table 10. Channel groups by brain region. Rows give the connector index of the channels, the region label stored with them, their count, and their names.
References
- [1]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2110.07058
- [2]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangellos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision, 130:33–55, 2022.DOI
- [3]Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2203.14712
- [4]Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. doi: 10.1109/CVPR52729.2023.01244.DOI
- [5]Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2505.11709
- [6]Lightwheel AI. EgoSuite-Open100K: A fully annotated open egocentric human dataset. https://huggingface.co/collections/LightwheelAI/egosuite-open100k, 2026. Dataset release.
- [7]Ropedia. Xperience-10M: A large-scale egocentric multimodal dataset with structured 3D/4D annotations. https://huggingface.co/datasets/ropedia-ai/xperience-10m, 2026. Dataset release.
- [8]Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. EgoMimic: Scaling imitation learning via egocentric video. In IEEE International Conference on Robotics and Automation (ICRA), 2025. doi: 10.1109/ICRA55743.2025.11127989.DOI
- [9]Ruihan Yang, Q inxi Yu, Yecheng Wu, Rui Yan, Borui Li, An-Chieh Cheng, Xueyan Zou, Yunhao Fang, Xuxin Cheng, Ri-Zhao Qiu, Hongxu Yin, Sifei Liu, Song Han, Yao Lu, and Xiaolong Wang. EgoVLA: Learning vision-language-action models from egocentric human videos. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2026.arxiv.org/abs/2507.12440
- [10]Hao Luo et al. Being-H0: Vision-language-action pretraining from large-scale human videos. arXiv preprint arXiv:2507.15597, 2025.arxiv.org/abs/2507.15597
- [11]Anthony Brohan et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023.arxiv.org/abs/2307.15818
- [12]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2406.09246
- [13]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025.DOI
- [14]Abby O’Neill et al. Open X-Embodiment: Robotic learning datasets and RTX-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.arxiv.org/abs/2310.08864
- [15]Alexander Khazatsky et al. DROID: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems (RSS), 2024.arxiv.org/abs/2403.12945
- [16]Shihan Wu et al. RoboCOIN: An open-sourced bimanual robotic data collection for integrated manipulation. arXiv preprint arXiv:2511.17441; https://github.com/FlagOpen/RoboCOIN, 2025.arxiv.org/abs/2511.17441
- [17]Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2302.00111
- [18]Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2312.13139
- [19]Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267, pages 24328–24346, 2025.arxiv.org/abs/2412.14803
- [20]Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. In Robotics: Science and Systems (RSS), 2025. doi: 10.15607/RSS.2025.XXI.074.
- [21]Michael F. Land and Mary Hayhoe. In what ways do eye movements contribute to everyday activities? Vision Research, 41(25–26):3559–3565, 2001.pubmed.ncbi.nlm.nih.gov/11718795
- [22]Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4):188–194, 2005.pubmed.ncbi.nlm.nih.gov/15808501
- [23]Roland S. Johansson, Göran Westling, Anders Bäckström, and J. Randall Flanagan. Eye–hand coordination in object manipulation. Journal of Neuroscience, 21(17):6917–6932, 2001.pubmed.ncbi.nlm.nih.gov/11517279
- [24]Roland S. Johansson and Göran Westling. Programmed and triggered actions to rapid load changes during precision grip. Experimental Brain Research, 71(1):72–86, 1988.pubmed.ncbi.nlm.nih.gov/3416959
- [25]Roland S. Johansson and J. Randall Flanagan. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience, 10(5):345–359, 2009.DOI
- [26]Hans Helmut Kornhuber and Lüder Deecke. Hirnpotentialänderungen bei Willkürbewegungen und passiven Bewegungen des Menschen: Bereitschaftspotential und reafferente Potentiale. Pflügers Archiv für die gesamte Physiologie des Menschen und der Tiere, 284(1):1–17, 1965. doi: 10.1007/BF00412364.DOI
- [27]Hiroshi Shibasaki and Mark Hallett. What is the Bereitschaftspotential? Clinical Neurophysiology, 117 (11):2341–2356, 2006.pubmed.ncbi.nlm.nih.gov/16876476
- [28]William J. Gehring, Brian Goss, Michael G. H. Coles, David E. Meyer, and Emanuel Donchin. A neural system for error detection and compensation. Psychological Science, 4(6):385–390, 1993.DOI
- [29]Dario Farina, Ning Jiang, Hubertus Rehbaum, Aleš Holobar, Bernhard Graimann, Heinz Dietl, and Oskar C. Aszmann. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: Emerging avenues and challenges. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(4):797–809, 2014.DOI
- [30]Jonathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert Pfurtscheller, and Theresa M. Vaughan. Brain–computer interfaces for communication and control. Clinical Neurophysiology, 113(6):767–791, 2002.
- [31]Nie Lin, Yansen Wang, Dongqi Han, Weibang Jiang, Jingyuan Li, Ryosuke Furuta, Yoichi Sato, and Dongsheng Li. EgoBrain: Synergizing minds and eyes for human action understanding. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2506.01353
- [32]Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The EPIC-KITCHENS dataset. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.arxiv.org/abs/1804.02748
- [33]Yin Li et al. In the eye of the beholder: Gaze and actions in first person video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):6731–6747, 2023.
- [34]Kristen Grauman et al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
- [35]Yunze Liu et al. HOI4D: A 4D egocentric dataset for category-level human–object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2203.01577
- [36]Xin Wang et al. HoloAssist: An egocentric human interaction dataset for interactive AI assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2309.17024
- [37]Lingni Ma, Yuting Ye, Fangzhou Hong, Vladimir Guzov, Yifeng Jiang, et al. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2406.09905
- [38]Toby Perrett, Ahmad Darkhalil, et al. HD-EPIC: A highly-detailed egocentric video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2502.04144
- [39]Jingkang Yang, Shuai Liu, et al. EgoLife: Towards egocentric life assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2503.03803
- [40]Manfredo Atzori, Arjan Gijsberts, Claudio Castellini, Barbara Caputo, Anne-Gabrielle Mittaz Hager, Simone Elsig, Giorgio Giatsidis, Franco Bassetto, and Henning Müller. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Scientific Data, 1:140053, 2014.
- [41]EgoEMG: A multimodal egocentric dataset with bilateral EMG and vision for hand pose estimation. GitHub repository, 2026. URL https://github.com/zhengqis123/EgoEMG.
- [42]Klaus Gramann, Joseph T. Gwinn, Daniel P. Ferris, Kelvin Oie, Tzyy-Ping Jung, Chin-Teng Lin, Lun-De Liao, and Scott Makeig. Cognition in action: Imaging brain/body dynamics in mobile humans. Reviews in the Neurosciences, 22(6):593–608, 2011.DOI
- [43]Sebastiaan F. W. Neggers and Harold Bekkering. Ocular gaze is anchored to the target of an ongoing pointing movement. Journal of Neurophysiology, 83(2):639–651, 2000.pubmed.ncbi.nlm.nih.gov/10669480
- [44]Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. Eye tracking for everyone. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- [45]Jackson Beatty. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2):276–292, 1982.pubmed.ncbi.nlm.nih.gov/7071262
- [46]Olaf Dimigen, Werner Sommer, Annette Hohlfeld, Arthur M. Jacobs, and Reinhold Kliegl. Coregistration of eye movements and EEG in natural reading: Analyses and review. Journal of Experimental Psychology: General, 140(4):552–572, 2011.DOI
- [47]Raja Parasuraman and Matthew Rizzo, editors. Neuroergonomics: The Brain at Work. Oxford University Press, 2007.DOI
- [48]Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), 2018.arxiv.org/abs/1704.06888
- [49]Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. DexMV: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision (ECCV), 2022.arxiv.org/abs/2108.05877
- [50]Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2022.arxiv.org/abs/2203.12601
- [51]Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. VRB: Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2304.08488
- [52]Yichen Zhu, Zhicai Ou, Feifei Feng, and Jian Tang. Any2Policy: Learning visuomotor policy with any-modality. In Advances in Neural Information Processing Systems (NeurIPS), 2024.DOI
- [53]Chen Wang, Fei Xia, Wenhao Yu, Tingnan Zhang, Ruohan Zhang, C. Karen Liu, Li Fei-Fei, Jie Tan, and Jacky Liang. Chain-of-Modality: Learning manipulation programs from multimodal human videos with vision-language-models. In IEEE International Conference on Robotics and Automation (ICRA), 2025. doi: 10.1109/ICRA55743.2025.11128270.DOI
- [54]Ruoxuan Feng, Di Hu, Wenke Ma, and Xuelong Li. Play to the score: Stage-guided dynamic multisensory fusion for robotic manipulation. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2408.01366
- [55]Remi Cadene, Simon Alibert, Alexander Sochorn, et al. Lerobot: State-of-the-art machine learning for real-world robotics in PyTorch. https://github.com/huggingface/lerobot, 2024.
- [56]Bowen Wen et al. FoundationStereo: Zero-shot stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2501.09898
- [57]Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao, Kaitong Cai, Chengkai Jin, Chunxiao Liu, Jianbo Liu, Siyuan Huang, Xingang Pan, et al. ACE-Ego-Hand: Repurposing video diffusion models for occlusion-robust egocentric 3D hand motion recovery. arXiv preprint arXiv:2608.20308, 2026.arxiv.org/abs/2608.20308
- [58]aigc apps. VideoX-Fun: A video generation pipeline for diffusion transformer. https://github.com/aigc-apps/VideoX-Fun, 2026.
- [59]Wan Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025.arxiv.org/abs/2503.20314
- [60]Innata Team. Innata-0: A native embodied world system. Technical report, Beijing Academy of Artificial Intelligence, 2026. To appear; hand reconstruction appendix, Section D.
- [61]Gen Li et al. EgoM2P: Egocentric multimodal multitask pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
- [62]Pantheon Industries. Argus: An open-source annotator for robotics data. https://github.com/Pantheon-Industries-Inc/argus, 2026.
- [63]Qwen Team. On the design of Qwen3.8-Next architecture: Evaluation, efficiency, and training stability. arXiv preprint arXiv:2608.30320, 2026.





