betaFeedback

WorksComputer Science

Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos

Abstract

To interact with an object, we need to anticipate both how it will move and how our hands should move with it. Human videos capture these relationships across everyday activities, offering a natural source for learning to predict future interactions. Learning from them requires recovering motion from completed interactions and using it to anticipate what could happen next. We present Grasping-the-Future\textit{Grasping-the-Future}, a unified generative model of joint 3D hand-object motion, including wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, we train a single model for several prediction tasks: forecasting future interactions, predicting hand motion given object motion or vice versa, generating both trajectories toward a desired final object pose, and completing partially observed interactions. We explicitly model hand-object relations and incorporate contact constraints to help preserve grasp relationships during motion. To obtain 3D supervision from internet videos, we develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct completed interactions. We combine these recovered trajectories with annotated interaction datasets as training data for all tasks. We demonstrate strong performance across all tasks, outperforming released forecasters on datasets with 3D annotations. Moreover, we show geometric and physical plausibility of the predicted trajectories by retargeting them to a robotic embodiment. Videos are available at https://grasping-the-future.github.io.

Introduction

“Life can only be understood backwards; but it must be lived forwards.” — Søren Kierkegaard

To interact with an object, we need to anticipate how our actions will affect it. Placing a mug on a shelf is a simple everyday action, but predicting how it will unfold requires understanding how the hand and mug move together. The fingers must hold the mug as it is lifted and release it only once it rests on the shelf. The way the mug is grasped affects how it can be carried, and the position of the shelf affects where the hand needs to move. From the eyes of the human manipulating, parts of the hand or the object may be occluded, and several plausible future motions may be possible. Anticipating the interaction therefore requires predicting the motion of both the hand and the object, while preserving firm hand-object contact as the object moves through the 3D world.

Human videos are a natural source for learning these predictions, showing how people handle objects across many everyday situations. Learning from them involves two steps: first, extracting hand and object motion from completed interactions, and then using these examples to learn how an interaction might continue from a partial observation. These steps echo Kierkegaard’s distinction: a recorded video allows us to look back and understand what happened, while acting requires anticipating what comes next. Image-based predictions, such as generated videos, can leave relative depth and contact ambiguous. Predicting in 3D makes the spatial relationships between the hands, objects, and scene explicit: where the fingers touch the mug, how it is oriented as it is carried, and where it comes to rest on the shelf. This geometric description also provides an unambiguous reference for later retargeting the predicted motion to a different embodiment like a robot.

Our key insight is that a single model can learn these different forms of prediction: predicting an object’s trajectory given the hand’s motion, predicting the hand’s trajectory given the object’s motion, generating both trajectories given a desired final object pose, and forecasting or completing an interaction from partial 3D observations. People reason this way intuitively: watching a hand move suggests where an object will go, while imagining where an object should end up suggests how the hand needs to move. We present Grasping-the-Future, a unified generative model of joint 3D hand-object motion. The same model forecasts an interaction from a short observed context, predicts hand trajectories given object motion or object trajectories given hand motion, generates motion toward a desired object pose, and fills gaps between observed poses (Figure 1). Across these tasks, we must handle observations of different entities and times. We bring these tasks together through masked spatiotemporal completion: we reveal different parts of a hand-object trajectory to the prediction model, and a flow-matching objective learns to generate the missing pieces of the trajectory. The generated trajectories include wrist motion, finger articulation, and object position and orientation. Since plausible hand and object trajectories can still drift apart, we use explicit hand-object relations and contact constraints to keep them moving together during a grasp.

Unified model for different types of hand-object prediction

Figure 1. Unified model for different types of hand-object prediction. Grasping-the-Future forecasts future motion, fills temporal gaps, predicts either entity from the other’s motion, and generates motion toward a given goal. Neutral poses show observations; teal and amber arrows indicate hand and object prediction, respectively. Fading poses show successive times.

Learning this predictive model requires 3D hand-object supervision that large-scale internet videos do not directly provide. We therefore develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to extract hand-object trajectories from complete monocular videos. The pipeline separates camera motion from object motion, refines the alignment between the hands and object using contact and temporal constraints, and filters unreliable reconstructions. We extract over 62,000 trajectories from internet videos and combine them with existing annotated hand-object datasets in a common 3D representation, accounting for differences in annotation quality during training. On egocentric video, our reconstructions are on par with or slightly better than those of HOPFormer ([3]), a recent model for egocentric images. From third-person viewpoints, our hand error is about nine times lower and our object pose error about a quarter lower (Appendix § 8.§ 3). The reconstructed trajectories serve as pseudo-ground truth for all tasks.

We evaluate Grasping-the-Future across forecasting and conditional generation tasks on datasets with 3D annotations, including OakInk2 ([63]) and HOT3D ([2]), and on held-out internet videos using our extracted trajectories as pseudo-ground truth. On the annotated datasets, a single forecast from Grasping-the-Future has 17% lower final wrist error and 10% lower final object error than the best released methods. The same model also predicts object motion from hand motion, which none of these methods supports, with 63% lower final error than keeping the object rigidly attached to the hand. We also demonstrate that these predictions can guide robot manipulation: by retargeting the predicted human motion to simulated robot hands with SPIDER ([42]), we enable the hands to manipulate objects along the forecast trajectories for different types of manipulation tasks.

In summary, our contributions are threefold. We develop a unified generative model of 3D hand-object motion that supports forecasting, cross-entity prediction, goal-conditioned generation, and temporal completion through a common masking formulation. We propose a scalable pipeline for reconstructing temporally aligned hand-object trajectories from internet videos and using them as supervision for learning future interactions. We incorporate explicit hand-object relations and geometric contact constraints into trajectory generation, producing structured 3D predictions that we evaluate through robot retargeting and object manipulation.

Related Work

Reconstructing hand-object interactions from video. Recent methods recover 3D hand-object geometry and motion from monocular videos by combining visual observations with geometric and learned priors [26]. Joint priors over hand configurations and object shape support reconstruction and grasp synthesis [58], while temporal attention helps recover coherent hand trajectories [59]. Other methods reconstruct bimanual interactions using Gaussian splatting [40] or use novel-view synthesis priors to recover object geometry that is not visible in short clips [56]. Hand priors also support joint hand-object pose estimation from a single egocentric image [3]. Recent work recovers world-space hand and object motion from egocentric videos with known object templates and supports hand-guided interaction synthesis [60]. We build on these advances to extract supervision from completed human interactions. Our focus is on using the recovered trajectories to learn how an interaction unfolds over time from a partial observation.

Interaction forecasting and robot manipulation. Prior work predicts hand trajectories and interaction hotspots from egocentric video [36]. Recent methods account for uncertainty in future 3D hand motion [5], use language queries to guide trajectory prediction [4], and combine interaction reasoning with 3D hand motion prediction [14]. Forecasting object poses [51] and 3D point trajectories [6, 64] further describes how objects move during an interaction. Some methods model the relationship between human and object motion by conditioning object forecasts on hand motion [52] or combining diffusion with physics-informed correction for full-body human-object forecasting [57]. Other work forecasts motion in unconstrained videos through dense 2D point trajectories [53]. These predictions can also guide robot manipulation. HOPMan translates predicted human interaction plans into robot actions [7], while Track2Act combines predicted point tracks with geometric transfer and a residual policy [8]. Gen2Act learns policies conditioned on generated human videos [9]. We learn to predict articulated hand motion together with object pose, capturing how the fingers, wrist, and object move through an interaction. We then retarget these forecasts to simulated robot hands to test their use for manipulation.

Controllable motion synthesis. A complementary line of work, often motivated by graphics and animation, generates plausible motion from text, object geometry, reference grasps, or goal poses. These methods typically emphasize generation plausibility without necessarily requiring grounding the predictions in a real scene. Prior work synthesizes dynamic grasps from a grasp reference and a desired object pose using physics-based control [17]. Bimanual manipulation datasets also provide models for completing task primitives [63]. Text-conditioned methods generate hand-object motion through contact-guided diffusion [11] or masked latent flow matching [16], while other work separates global motion from local hand articulation using latent diffusion and a grasp VAE [33]. Some methods ground generated motion through initial human-object poses and sparse object waypoints [32]. Masks over frames and joints also support sparse or partial motion constraints [19]. We ground our predictions in partial observations of a hand-object interaction and the surrounding scene geometry. In Grasping-the-Future, one model forecasts its continuation, predicts either entity from the other’s motion, generates both trajectories toward a desired object pose, and completes missing portions of an interaction while modeling hand-object contact.

Method

We develop Grasping-the-Future to predict how hands and objects interact with each other in 3D. We represent an interaction as a trajectory of hand and object poses. We observe a part of this trajectory, and our goal is to generate the rest. By changing which parts we observe, we use the same model to forecast the future, to generate the hand trajectory from a given object trajectory and vice versa, to generate the motion that reaches a given final pose, and to fill a gap in the middle of a trajectory. We first describe this representation and the prediction model and then introduce the losses that help maintain hand-object contact. Finally, we describe how we obtain 3D hand-object reconstructions from internet videos to create a dataset for training the model.

Problem Formulation

Representation. We segment each recorded interaction into windows of TT consecutive frames. A window contains up to two hands and one rigid object, which are the three entities we model. At frame kk, each hand e∈{L,R}e \in\{L, R\} has a state sek∈R99s_e^k \in\mathbb{R}^{99} made of the wrist position and rotation and the 15 joint rotations of the MANO hand model Romero et al. [48]. The object state, sok∈R9s_o^k \in\mathbb{R}^{9}, is made of its position and rotation. We use the continuous 6-D form Zhou et al. [66] for the rotations. The states of a window form X\mathbf{X}. We also give the model a static context c\mathbf{c} with the object mesh and its scale, the MANO shape of each hand, the gravity direction when available, and a point cloud of the scene.

Input and output. A mask m∈{0,1}3×T\mathbf{m} \in\{0,1\}^{3 \times T} is formed with one row per entity and one column per frame, so each cell represents one entity at one frame. If an entity’s state is visible at that frame, a cell is 1; therefore, its complement mˉ\bar{\mathbf{m}} indicates the cells the model needs to generate. The model learns a distribution over the generated states, Xmˉ∼pθ(⋅∣Xm,c)\mathbf{X}_{\bar{\mathbf{m}}} \sim p_\theta(\cdot\mid\mathbf{X}_{\mathbf{m}}, \mathbf{c}), and samples from this distribution. This formulation unifies the tasks by using randomized masks to train a single model on all of them (Table 3). At evaluation, the masks are:

  • Forecasting. The first 0.5 s of the window are observed and the remaining 4.5 s are generated for all entities.

  • Hands →\to object and object →\to hands. One entity is observed throughout the window and the other only at the first frame.

  • Goal-conditioned generation. Only the first frame is observed, together with the final state of the object, the hands, or both, and the model generates the motion toward that final state.

  • Temporal infilling. The middle third of the window is hidden for every entity and generated from the frames before and after it.

Window coordinates. We call the last frame with an observed cell the present frame. For forecasting, it is the last frame of the observed 0.5 s. For hands →\to object and object →\to hands, where one entity is always observed, it is the last frame of the window. We re-express all poses in the camera coordinates of the present frame, which removes camera motion from the window. We also subtract the 3D point halfway between the observed wrists and object at that frame from every position so that each window is centered on the interaction.

Generation as Masked Spatiotemporal Completion

We generate the masked cells with a flow-matching transformer over a grid of tokens (Figure 2).

Figure 2. Model overview. An interaction is represented as a grid of tokens, with rows indicating states ss and relations rr, and one column per frame. Observed cells are given, and unobserved cells are initialized to noise. A transformer attends to rows, columns and across the grid to predict the flow velocity of the noise cells (with the context tokens (blue) as additional keys/values). The noise is decoded into a sample by ten Euler steps, producing hand and object trajectories. Multiple samples for the same input vs. ground truth (dashed). The bottom row is used for training. An observation mask is sampled per window during training so the same model trains on all tasks (but with a different set of observed cells). Flow matching is used for velocity supervision and auxiliary losses are applied to decoded trajectories.

Relation tokens. We also include a relation token for each hand-object pair, which encourages the model to directly output the hand-object relation rather than depending on the two independently generated poses to somehow result in the hand-object correlation. Each frame then has five tokens, the three states and the two relation tokens. The relation token holds the wrist pose expressed in the object’s coordinates and the distance from each of the five fingertips to the object surface. Let (Rko,pko)(R_k^o, p_k^o) denote the object’s rotation and position at frame kk, (Rk,pk)(R_k, p_k) the wrist’s rotation and position, uf,keu_{f,k}^e the position of fingertip ff, and SkS_k the object surface. In the object’s coordinates, the wrist has rotation Qke=RkoTRkeQ_k^e = R_k^{o\mathsf{T}}R_k^e and position qke=RkoT(pke−pko)q_k^e = R_k^{o\mathsf{T}}(p_k^e - p_k^o). We define the token as

rke=[qke, Qke, {dist⁡(uf,ke,Sk)}f=15]∈R14.(1)\mathbf{r}_k^e = \left[q_k^e,\ Q_k^e,\ \left\{\operatorname{dist}\left(u_{f,k}^e,S_k\right)\right\}_{f=1}^{5}\right] \in\mathbb{R}^{14}. \tag*{(1)}

If the hand or the object is not observed in a frame, we do not give the model that frame’s relation token either, since it would reveal a pose the model has to generate. The model then generates the relation token along with the pose.

Δ\Delta-codec and flow matching. We represent motion using a temporal-difference encoding, which we call the Δ\Delta-codec. The initial state is presented in standardized absolute form and subsequent states as changes per unit time for each continuous interval in which an entity is present. Writing sks_k for the state of any entity at time τk\tau_k, we encode these changes as

zk=sk−sk−1(τk−τk−1)σv.(2)z_k = \frac{s_k-s_{k-1}}{(\tau_k-\tau_{k-1})\sigma_v}. \tag*{(2)}

Here, σv\sigma_v contains per-channel velocity scales estimated from training data. Stationary states have zero difference tokens, and dividing by elapsed time accommodates different frame rates of different windows. A difference token is observed only when both adjacent states are observed. We also add the absolute observed states as additional inputs so the model can attend to observations on both sides of a gap. We reconstruct the absolute states by summing the changes from the interval’s start state. During inference, we correct the generated gaps by linear interpolation (in time) of translation corrections and geodesics on SO(3)\mathrm{SO}(3) for rotation corrections of observed entity state past the gap. We preserve all observed states exactly.

We train the masked tokens with flow matching ([35]). With x1\mathbf{x}_1 their clean encoded values, ϵ\epsilon Gaussian noise and xt=(1−t)ϵ+tx1\mathbf{x}_t=(1-t)\epsilon+t\mathbf{x}_1 at a flow time t∼U(0,1)t\sim\mathcal{U}(0,1), the network vθ(xt,t;Xm,c)v_\theta(\mathbf{x}_t,t;\mathbf{X}_m,c) regresses the velocity x1−ϵ\mathbf{x}_1-\epsilon on the masked tokens, while the observed tokens enter without noise. We call this loss LFM\mathcal{L}_{\mathrm{FM}}.

Architecture and sampling. Each hand, object, or relation token is embedded into the feature space using a layer for its type. Both hands share a layer, as do both relation tokens, so a learned embedding identifies each of the five tokens. We also add a learned embedding for whether the token is observed or masked, a sinusoidal encoding of elapsed time, and a learned projection of its confidence. The confidence distinguishes annotated data from trajectories extracted from video (§ 3.4), so the model can rely less on noisy labels. The transformer repeats a group of four blocks. The first three attend to the five tokens over time and to each of the five tokens for each frame. The fourth attends over all tokens of all frames at once. Every attention layer also reads the context cc as extra key-value tokens, with the object mesh encoded by Uni3D ([65]) and the scene point cloud, carrying DINOv3 features ([50]), by LitePT ([62]). At inference, the masked tokens start as Gaussian noise and are denoised by integrating the predicted velocity over 10 Euler steps.

Losses on decoded trajectories

Small errors under framewise flow matching can accumulate in decoded trajectories, causing the hand to drift from the object. We therefore supervise hand-object attachment and score complete sampled trajectories.

Attachment loss. We estimate and decode a clean trajectory from a single flow-velocity prediction. For each hand ee, we use the object-frame wrist pose (Qke,qke)(Q_k^e, q_k^e) and the object-frame fingertip positions wf,ke=RkO⊤(uf,ke−pkO)w_{f,k}^e = R_k^{O\top}(u_{f,k}^e - p_k^O). We define reference contact frames CeC_e as those with at least one fingertip within 5 mm of the object surface. For each hand, we minimize

Latte=∑k∈Ce[∥q^ke−qke∥+15∑f∥w^f,ke−wf,ke∥+λRdc(Q^ke,Qke)+λv∥q˙^ke−q˙ke∥].(3)\mathcal{L}_{\mathrm{att}}^e = \sum_{k \in C_e} \left[ \left\lVert\widehat{q}_k^e - q_k^e \right\rVert+ \frac{1}{5}\sum_f \left\lVert\widehat{w}_{f,k}^e - w_{f,k}^e \right\rVert+ \lambda_R d_c\left(\widehat{Q}_k^e,Q_k^e\right) + \lambda_v \left\lVert\widehat{\dot{q}}_k^e - \dot{q}_k^e \right\rVert\right]. \tag*{(3)}

where hats denote predictions, dcd_c is the chordal rotation distance, and q˙ke\dot{q}_k^e is the object-relative wrist velocity. We match the reference relative motion, allowing changes such as regrasping. We also supervise fingertip-to-surface distances and penalize acceleration, jerk, and finger-joint violations beyond their reference values (Appendix § 7).

Scoring sets of samples. To capture multiple plausible futures, we generate M=3M = 3 trajectories X^i\widehat{X}_i from the same observations on a quarter of optimizer steps. We use six Euler steps per sample and backpropagate through all six. We compare these samples with the annotated or reconstructed trajectory XX using the energy score ([24, 41, 21]), which balances agreement with the reference against diversity among samples:

LES=ES⁡({ϕℓ(X^i)}i=1M,ϕℓ(X))+λdES⁡({ϕd(X^i)}i=1M,ϕd(X)).(4)\mathcal{L}_{\mathrm{ES}} = \operatorname{ES}\left(\left\{\phi_\ell(\widehat{X}_i)\right\}_{i=1}^{M},\phi_\ell(X)\right) + \lambda_d \operatorname{ES}\left(\left\{\phi_d(\widehat{X}_i)\right\}_{i=1}^{M},\phi_d(X)\right). \tag*{(4)}

The landmark summary ϕℓ\phi_\ell records how far each entity, and each wrist relative to the object, has moved after one-third, two-thirds, and all of the generated frames, along with each entity’s final rotation change. The dense summary ϕd\phi_d records positional changes at every generated frame, with λd=0.5\lambda_d = 0.5. To maintain focus on final positions and rotations, we score these summaries independently. The expected score is minimized when the predicted and reference summary distributions match ([44]).

Extracting trajectories from video

Existing annotated 3D datasets cover a limited range of objects, environments, and interactions, so we also extract hand and object trajectories from monocular internet videos and train on both (§ 4.1). Figure 3 summarizes the pipeline; we provide implementation details in Appendix § 8 and compare the extracted trajectories with HOPformer on EPIC-Contact, HOT3D, and DexYCB in Appendix § 8.§ 3.

Figure 3. Extraction pipeline. Using four refinement steps, we combine pretrained estimates of hands, objects, depth, and camera motion into a canonical trajectory (green). We only keep clips that pass all consistency checks. The bottom row displays examples rejected by each check, and the bar displays the number of clips we discard across stages.

Per-frame estimates. We use Molmo 2 (Clark et al., [18]) to identify manipulation clips from Action100M (Chen et al., [13]) and locate the manipulated object, then track its mask with SAM 3 (Carion et al., [10]). We estimate MANO hand poses with WiLoR (Potamias et al., [45]), object meshes and poses with SAM 3D (Chen et al., [15]), metric depth with MoGe (Wang et al., [55]), and camera motion with MegaSaM (Li et al., [34]).

Trajectory refinement. Each per-frame estimate is in its own frame’s camera coordinates, so we use the MegaSaM camera poses to bring them into one world coordinate system. We lift CoTracker3 (Karaev et al., [29]) object tracks into 3D with depth and fit a rigid object trajectory by robust bundle adjustment (Huber, [28]). We then jointly refine hand and object poses to match the 2D masks and keep contact, penalizing interpenetration, abrupt motion, and implausible finger poses, with hand scale fixed. When a held object is hidden for a long stretch, its rotation follows the hand’s. A transformer trained on corrupted motion-capture trajectories estimates each clip’s depth bias, which we correct by shifting the object and the grasping hand along the viewing direction. Finally, we smooth the remaining hand jitter (Appendix § 8.§ 2).

Verification. Finally, we check image agreement, contact, interpenetration, temporal consistency, and object size. To make sure we tracked the correct object, we also ask a VLM to point to the object in the hand through the clip. Of the total candidate clips, about 12% yield valid trajectories after final verification. For each verified trajectory, we also compute an interaction-fidelity score. A higher score means the reconstructed hand(s)-object contact fits the video better, the object motion is smoother, and the hands and object align better with their 2D detections. This score goes into the model as part of the confidence input. If a supporting surface is visible, we also estimate the direction of gravity from it.

Experiments

Our experiments evaluate prediction across seven tasks and its transfer to robotic motion. We compare against non-learned baselines and released forecasting models from prior work, and evaluate the benefit of training on trajectories extracted from internet videos (§ 4.2). how well forecast trajectories transfer to robots (§ 4.3). Additional ablations on the model’s losses and inputs are provided in Appendix § 10.

Setup

Training. We train on eight annotated hand-object datasets ([12, 2, 37, 63, 47, 25, 30, 54]) and trajectories extracted from internet videos. Together, they provide 74,312 trajectories and 134,661 five-second windows, including 62,562 trajectories and 109,346 windows from internet videos. Each training window receives a task observation pattern sampled uniformly across the five task families, or a random block mask. Training details are provided in Appendix § 7.

Benchmark. We evaluate every task on the same 331 five-second windows from diverse scenarios unseen during training, 256 from the annotated datasets and 75 from internet videos, with one window per video. For forecasting, the model observes the first 0.5 seconds and predicts the remaining 4.5 seconds. For internet videos, the references are our extracted trajectories, which are less accurate than the annotations (Appendix § 9). We will release the code and data publicly.

Metrics. We compare each prediction with the recorded trajectory of its window, which we call the reference, and measure position error in millimeters at the wrists and the object center. For a predicted trajectory, the average displacement error (ADE) is this error averaged over its generated frames, while the final displacement error (FDE) is the error at its last frame. When the final pose is observed, the final error is zero by definition, so we report the mid-gap error, calculated at the midpoint of the generated frames. Since a single observation can lead to multiple plausible futures, we generate 20 samples per input and evaluate the results in three different ways..

  • Single sample. We score the first of the 20 samples on its own, as when the model is asked to make a single prediction.

  • Energy score. The energy score ([24, 49]) of the 20 samples is their average error minus half their average distance from each other. The second term rewards spread, while the first penalizes samples far from the reference. If we view the reference as one draw from all the futures that could follow the observation, the expected score is lowest when the samples come from the same distribution as those futures. A method that makes only one prediction has no spread, so its score equals its error and can be compared directly.

  • Best of 20. The error of the sample with the lowest ADE shows how close the nearest of the 20 samples comes to the reference. This sample is selected using the ground-truth reference, which is unavailable to the model at inference time.

The energy score rewards both accuracy and spread, so we also check the spread on its own. Let x1,…,x20x_1,\ldots,x_{20} be the sample positions and x21=yx_{21}=y the reference position at the frame scored by FDE or mid-gap error, and let did_i be the distance from xix_i to the mean of the other 20 positions. The reference lies inside the samples if d21d_{21} is not the largest of d1,…,d21d_1,\ldots,d_{21}. If the samples and the reference come from the same distribution, each did_i is equally likely to be the largest, so the reference lies inside the samples in 20/21≈95%20/21 \approx95\% of windows. A rate below 95% means that the samples are too concentrated or centered away from the reference, and a rate above 95% means that they are too dispersed. For all of these metrics, each annotated dataset counts equally, and annotated data and internet videos each count for half. Appendix § 9 gives full definitions, eligibility rules, and additional interaction metrics.

Baselines. Stay-put holds the last observed pose, while constant velocity extrapolates observed motion. Interpolation connects observed endpoint poses. Rigid carry predicts one entity from the other’s motion. It keeps their relative pose from the first frame fixed, as if the hand and object were rigidly attached, so a predicted object moves with the nearest wrist and predicted hands move with the object. We also compare with the prior released models for hand and object trajectory forecasting in Table 2. Each released model uses its required inputs, and all predictions are evaluated with our scoring code.

MethodRGBDepthHandsObjectTextNNADE L wristADE R wristADE ObjectFDE L wristFDE R wristFDE Object
USST (Bao et al., [5])✓×✓××1134.6135.2–196.3182.3–
ForeHand4D (Prakash et al., [46])✓××××2092.4103.1–122.8133.6–
EgoH4 (Hatano et al., [27])✓×2D××1223.4266.9–292.5327.8–
Uni-Hand (Ma et al., [39])✓✓✓××20360.8344.1–359.5348.7–
ObjectForesight (Soraki et al., [51])✓✓×✓×20––77.0––130.4
MotionForesight (Bharadhwaj & Jangir, [6])✓✓×mask×1––85.1––142.9
General Flow (Yuan et al., [61])✓✓×3D pts✓20––121.0––198.9
MolmoMotion (Zhang et al., [64])✓××3D pts✓10––98.9––182.3
Grasping-the-Future, one sample✓✓✓✓×1113.1118.095.9170.3156.1150.9
Grasping-the-Future✓✓✓✓×2067.565.055.5103.492.491.1

Table 2. Comparison with different forecasting approaches. ADE and FDE in millimeters with the listed inputs. Methods with NN samples are scored with the energy score of all samples, which equals the error when N=1N=1. Our model uses estimated depth here. Bold and underline mark the best and second-best results.

Prediction Across Tasks

Table 1 compares our model with non-learned baselines across all seven tasks. When predicting one entity from the other’s motion, a single sample has 46% lower object FDE and 37% lower hand FDE than rigid carry. A single sample is less accurate than stay-put for forecasting (2–8% higher error) and than interpolation at the middle of gaps whose endpoints are both given (3–28%). These baselines stay close to the observed poses, which is a safe guess when the future is uncertain. A sample instead commits to one plausible motion, and when this motion differs from the reference, its error can exceed that of not moving at all. The energy score accounts for this by scoring the 20 samples together, and under it our model scores 30–68% lower than the best baseline in every task column. Since the energy score subtracts the distance between samples and a single prediction has none, part of this difference could come from spread alone. To test this, we give stay-put and interpolation the spread of our samples, turning each of their predictions into 20 samples with Gaussian noise that has the covariance of our samples in that window. With about the same spread, our score is still 9–44% lower (Appendix § 11.§ 1), so our samples also lie closer to the reference. Figure 4 shows examples across five tasks.

Figure 4. Qualitative predictions across five tasks. Each row shows one clip, its observation mask, the input frame with and without the observed motion, and three snapshots of the predictions. Mask rows correspond to the two hands and the object. Orange denotes observations and purple predictions; darker orange indicates later observations, while faded purple shows earlier predictions. Blue denotes the reference object pose in the hands-to-object row and the given goal pose.

EntityFuture predictionHands→obj.Obj.→handsTemporal infillingGoal, all givenGoal, hands givenGoal, object given
objecthandsobjecthandsobjecthandsobjecthandsobjecthandsobjecthandsobjecthands
Error atendendendendmiddlemiddlemiddlemiddleendmiddlemiddleend
Stay-put142.6145.6149.0156.651.769.398.8127.7149.0127.798.8156.6
Constant velocity627.1819.3––––––––––
Interpolation––––38.249.076.092.3–92.376.0–
Rigid carry––108.6156.4––––––––
Motion capture only159.0185.671.6132.939.050.091.2101.9146.1110.396.4166.9
Motion capture only, ES115.3136.752.297.825.034.061.767.199.869.062.8117.9
Ours145.8157.459.198.739.552.189.8101.6126.1110.297.4129.3
Ours, ES89.094.435.159.622.229.851.059.371.162.053.570.5
Best of the 20 samples, chosen using the reference
Motion capture only121.8153.950.0103.726.635.062.674.1107.478.268.0137.1
Ours102.8113.236.675.123.432.856.570.291.173.861.691.7
Reference inside the 20 samples (%)
Motion capture only594854467270726564697551
Ours787677818685867983828788

Table 1. Prediction across seven tasks. Position errors are in millimeters, and hand errors average the left and right wrists. The row “Error at” shows whether a column scores the last frame or, when the final pose is given as input, the middle of the generated frames. “Motion capture only” is our model trained without the trajectories extracted from internet videos. For both models, we report the error of a single sample, the energy score of the 20 samples, and the error of the best of the 20 samples (§ 4.1). Because the energy score subtracts the distance between samples, it is not the error of any one trajectory and can be lower than the best-of-20 error. The best of 20 is chosen using the reference, so it has its own block and is left out when marking the best (bold) and second-best (underlined) results. The last block shows the percentage of windows in which the reference lies within the 20 samples, ideally 95%.

We also train the same model without the trajectories extracted from internet videos (“Motion capture only” in Table 1) to see how much they help. Compared with this model, a single sample of our model has 8% lower forecasting FDE for objects and 15% for hands, and 18–26% lower FDE when predicting one entity from the other’s motion. When only the final pose of the hands or of the object is given, the FDE of the other entity is 14–23% lower. On internet videos, the forecasting gain comes from motion across the image, where extracted references are more accurate than in depth (Appendix § 9). Across the image, the final error falls from 112 to 73 mm for objects and from 155 to 83 mm for hands, while the error in depth barely changes. The reference also lies inside our samples more often, in 76–88% of windows in every column (ideally 95%), against 46–75% for the motion-capture-only model, whose samples are too close together.

Table 2 compares our model with released forecasting methods. Under the energy score, our ADE and FDE are 16–37% lower than those of the best released method, ForeHand4D for the wrists and ObjectForesight for the object. When each method is scored on one prediction, our wrist ADE and FDE are 13–16% lower than those of USST, the most accurate released method in this setting. For the object, MotionForesight has 11% lower ADE and 5% lower FDE than our single sample, but it barely moves the object. Its median predicted displacement at the end of the window is 18 mm, against 81 mm for the reference, and its FDE is within 1 mm of stay-put. The references lie inside our samples more often than inside those of ObjectForesight, General Flow, and Uni-Hand (77–78% versus 0–69%). ForeHand4D and MolmoMotion contain it more often (82% and 95% versus 76% and 71%), but their samples are much farther apart and farther from the reference. They are on average 19 and 48 cm apart with errors of 22 and 42 cm, against 13 and 11 cm apart with errors of

16 and 15 cm for ours (Table 12). A set this wide contains the reference easily but provides poor motion predictions, which the energy score penalizes.

MethodNNInsideApartError
Hand forecasting
ForeHand4D ([46])2082%191.8224.1
EgoH4 ([27])200%0.2310.2
Uni-Hand ([39])200%11.2359.8
Ours2076%125.2160.5
Object forecasting
ObjectForesight ([51])2069%107.4184.0
General Flow ([61])204%32.4215.1
MolmoMotion ([64])1095%480.4422.5
Ours2078%110.3146.2
Ours1071%110.0146.7

Table 12. Reference inside the samples. Forecasting at the final position on the windows of Table 2, with both wrists averaged for hand methods. Ours is scored on the same windows and with the same number of samples as the methods in its group. Inside is the percentage of windows with the reference inside the samples, ideally 95% for N=20N=20 and 91% for N=10N=10. Apart is the mean distance between two samples and Error the mean distance from a sample to the reference, both in millimeters.

Testing Physical Plausibility of Forecasting through Retargeting

We assess the physical plausibility of our forecasts by retargeting them to simulated XHands. We test whether the robot hands can manipulate the object and reproduce its predicted motion under the simulator’s dynamics.

Kinematic retargeting matches the robot’s wrist and fingertips to the predicted hand poses. Dynamic retargeting with SPIDER (Pan et al., [42]) further optimizes robot controls in simulation to follow the predicted object motion. We also test kinematic retargeting with SPIDER’s grasp preparation but without trajectory optimization. Using only the observed frames, grasp preparation adjusts the robot hand’s starting pose and grip in simulation until the hand holds the object steadily. While the forecast hand holds the object, it then moves the robot wrist together with the predicted object and keeps the grip unchanged. All three methods use the same forecast for each observation.

We evaluate 103 examples whose reference hand-object trajectories retarget successfully with SPIDER, and we run SPIDER with three optimizer seeds per example. We consider a retarget to be successful if the robot hand moves the object through the whole prediction interval and the object’s position and rotation errors relative to the forecast, averaged over the predicted frames, are below 100 mm and 0.5 rad.

We see that kinematic retargeting succeeds in only 14.6% of examples, showing that these manipulations are difficult to carry out under physics by matching wrist and fingertip poses alone (Figure 5). Grasp preparation raises success to 43.7%, so many of these failures come from a grasp that does not hold the object. When the robot controls are also optimized with SPIDER, 73.8% of our forecasts are reproduced in simulation. Once adapted to the robot hand, most of our forecast hand-object motions can therefore be carried out through physical interaction, which indicates that they are physically plausible.

Testing physical plausibility through retargeting

Figure 5. Testing physical plausibility through retargeting. We test whether simulated robot hands can manipulate the object along the predicted trajectory from Grasping-the-Future. The left panel shows the observed frame, our 4.5 s forecast, and simulated replay after SPIDER retargeting (Pan et al., [42]). The table reports success across 103 trajectories in different scenes.

Conclusion

We presented Grasping-the-Future, a unified model for forecasting and completing joint 3D hand-object motion. We combined masked spatiotemporal completion with explicit hand-object relations and contact supervision to predict coordinated motion under different observations and goals. We reconstructed over 62,000 trajectories from internet videos and showed that this supervision improved forecasting beyond training on annotated datasets alone. Under the energy score, our forecasts outperformed all released hand and object forecasting methods by 16–37%. We also demonstrated that our forecasts could guide object manipulation after retargeting to simulated robot hands, connecting prediction from human videos to robot execution.

Supplementary Material

Skip printed table of contents.

Model and Training Details

In this section we provide additional details on the model, losses and training from § 3.2, § 3.3, and § 4.1. Also, Table 4 and Table 5 provide information on the settings and loss weights.

Architecture
Transformer12 blocks; width 512; 8 heads; feed-forward width 2048; GELU; dropout 0.1
Parameters103.3M trainable; 12.7M frozen
Training data
Dataset sizesAction100M 62,562/109,346; TACO 4,089/6,267; DexYCB 2,280/2,280; OakInk2 2,213/9,938; SHOW3D 1,638/3,552; HOT3D 1,016/1,016; H2O 191/930; HO3D 173/867; HO-Cap 150/465; total 74,312/134,661
Data splitsGrouped by recording
Source weightingSampling weight per dataset ∝Ns1/4\propto N_s^{1/4} at the start of training, where NsN_s is its number of training windows
Label-quality weightsHOT3D, OakInk2 3; DexYCB, TACO, H2O, HO3D, HO-Cap 2; SHOW3D 1.5; Action100M 1
Optimization
OptimizerAdamW, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay 0.01
Learning rateLinear warmup to 10−410^{-4} over 1,000 steps, then cosine decay to 10−510^{-5} at step 40,000
Gradient clippingGlobal norm 1.0
Batch size512 windows; 32 per GPU
Compute16 H100 GPUs; about two days
Loss settings
Finger-joint limitsTwist up to 25∘25^\circ at finger bases and 15∘15^\circ elsewhere; swing up to 135∘135^\circ
Landmark rotation changeFlattened 0.3(RrefTRend−I)/20.3(R_{\mathrm{ref}}^{\mathsf{T}}R_{\mathrm{end}} - I)/\sqrt{2}, where RrefR_{\mathrm{ref}} is the reference rotation at the present frame and RendR_{\mathrm{end}} the rotation at the last predicted frame
Dense-summary scalingEach position-change sequence is scaled by 3/n\sqrt{3/n}, where nn is its number of predicted frames

Table 4. Model and training settings. Dataset sizes give training trajectories / five-second windows.

Flow matchingDecoded trajectoryContact and samples
Hand translation 0.884Wrist and object position 1.0Fingertip distance 0.335
Wrist rotation 0.738Wrist and object rotation 0.368Attachment position 0.178
Finger articulation 1.112Finger-joint rotation 0.422Attachment rotation 0.041
Object translation 0.730Smoothness 3.2×10−43.2 \times10^{-4}Attachment velocity 0.121
Object rotation 0.306Joint limits 3.35×10−33.35 \times10^{-3}Contact classifier 0.1
Hand-object relations 0.185MANO pose prior 5.0×10−45.0 \times10^{-4}Energy score 1.0
Support penalty 1.0

Table 5. Training loss weights. The flow-matching weights combine the six components of LFM\mathcal{L}_{\mathrm{FM}}, and the other weights scale the auxiliary losses.

Representation. We divide each trajectory into five-second windows with a 2.5-second stride, with the last window ending at the trajectory’s endpoint. Frames keep their timestamps and cover 5 seconds for each window, regardless of frame rate. So, a window can hold at most 150 frames for 30 fps recordings or about 50 frames for 10 fps extracted videos. We pad shorter windows (and windows from trajectories less than 5 seconds) and exclude padding and absent entities from attention and losses.

The coordinates are as described in § 3.1, with two additional details. If the gravity direction is known, we also rotate them so that this direction points downward. When the object is unobserved at the present frame, centering uses the mean wrist position alone, and when no hand is observed there, the object alone.

Δ\Delta-codec. We compute rotation changes by subtracting consecutive 6-D rotation vectors, including those for finger joints. Relation tokens skip this encoding and describe each frame directly. The absolute observed states of § 3.2 are standardized and paired with a flag, 1 when the state is observed and 0 otherwise. Both are zero at unobserved frames. They are mapped to a vector added to the state embedding by a linear layer initialized to zero, so they have no effect at the start of training.

Flow-matching loss. The flow-matching loss LFM\mathcal{L}_{\mathrm{FM}} is split into six components, for hand translation, wrist rotation, finger articulation, object translation, object rotation, and hand-object relations. Each component is the squared velocity error averaged over its channels and masked frames. Hand and relation components are computed for each side and averaged over the sides with masked frames. The weights in Table 5 combine the components into a weighted mean for each window, and the loss averages these means across windows. A component with no masked frames in a window is left out of that window’s mean, and a window with no masked frames is left out entirely. This averaging prevents over-weighting windows with many masked frames and components with many channels. For losses on decoded trajectories, we estimate clean tokens as x^1=xt+(1−t)vθ\hat{x}_1 = x_t + (1-t)v_\theta and decode them into poses.

Architecture. We use a 12-block transformer with 512-dimensional embeddings and eight attention heads, with the attention pattern and token embeddings of § 3.2. Each token type has a two-layer MLP that embeds it and a linear layer that predicts its flow velocity. Both hands use the same layers, and so do both relation tokens. AdaLN-Zero (Peebles & Xie [43]) gives every attention and feed-forward layer the flow time and the average time between frames. Another MLP takes the three state tokens and predicts fingertip contact, defined as a fingertip-to-object distance below 5 mm, and is trained with binary cross-entropy. Other settings are given in Table 4.

Each entity’s confidence features describe motion smoothness, stability of fingertip-to-object distances, presence in adjacent frames, and source quality. The relation token receives the confidence features of both its hand and the object. Smoothness and fingertip-distance stability are computed from neighboring frames, so close to a masked frame they can reveal the hidden pose. We therefore set them to 0.5 within 0.1 seconds of the entity’s masked frames. Fingertip distances depend on both the hand and the object, so for this feature we use the masked frames of both.

Conditioning. The scene point cloud is derived from the final frame in which all entities are observed. This is because using partially observed frames might reveal a masked entity’s pose. We then remove the hand and object pixels using their masks. We then prune background points farther than 0.5 m from the hand-object midpoint, widening this to 1 or 2 m when too few points remain. We then downsample the point cloud to 4,096 points with farthest-point sampling and express it in the present frame’s centered coordinates.

Each scene point has 12 channels to pack the point’s position, surface normal, distances to the hands and object, height above the support plane, normal tilt from vertical, curvature, and log density. It also has 32 channels of PCA-reduced DINOv3 image features (Siméoni et al. [50]). We then use LitePT-S (Yue et al. [62]) (pretrained on ScanNet (Dai et al. [20])) to encode the point cloud. Finally, eight learned queries pool the scene encoder’s output into eight scene tokens through two attention blocks. We train only the input layer, the output projection, and the pooling blocks while the rest of the scene encoder remains frozen during training. In 20% of training windows, we zero the image features.

We also uniformly sample 10,000 points from the object mesh’s surface and use a frozen Uni3D-g encoder (Zhou et al., [65]) to encode them. A trainable projection layer then maps the encoded vector to four object-shape tokens. We also add one context token for the object’s log scale and one for each hand’s MANO shape.

Training masks. Table 3 lists the observed frames of each task in training and evaluation and how often each task is drawn during training. In temporal infilling, the gap length and start are drawn uniformly, and the first and last frames are always observed. For random blocks, each entity is selected independently with probability 0.5 and gets one masked segment of random length, from one frame to all but one valid frame, at a random start. The first frame stays observed, and at least one hand or object pose is masked after it.

TaskObserved in trainingObserved at evaluationTraining probability
Forecastingfirst KK frames, K∼Unif⁡{1,…,Tv−1}K \sim\operatorname{Unif}\{1,\ldots,T_v-1\}first 0.5 s0.19
Hands →\to objecthands throughout, object in the first framesame0.19
Object →\to handsobject throughout, hands in the first framesame0.19
Goal-conditioned{sup}†`first 1–5 and last 1–3 framesfirst and last frames0.19
Temporal infillingall except one gap of 20–50% of the windowall except the middle third0.19
Random blocksall except one block per selected entitytraining only0.05

Table 3. Observation patterns of each task. TvT_v is the number of valid frames. †Goal-conditioned training gives goals for all entities, only the hands, or only the object, with probabilities 0.5, 0.25, and 0.25. Entities without a goal are observed only in the first frame.

Trajectory losses. We supervise decoded wrist and object positions with squared error and decoded rotations of the wrists, object, and 15 finger joints of each hand with θ2/2\theta^2/2, where θ\theta represents the angle to the reference rotation. These losses also include observed frames decoded by the Δ\Delta-codec based on predicted differences, in addition to masked frames.

We also regularize smoothness and finger articulation. To improve smoothness, we penalize increases in the squared acceleration and jerk of the wrists and object over the reference within masked segments, with jerk weighted 1/30 relative to acceleration. For finger articulation, we penalize joint-limit violations and the MANO pose prior when they exceed the reference’s. Table 4 lists the joint limits.

The fingertip-distance loss is the absolute error in fingertip-to-object distance, counted only for fingertips within 10 cm of the object in the reference. For each hand, we evaluate it on the last frame where the hand or the object is predicted and on up to 15 other such frames chosen at random. The attachment loss of Equation (3) applies to periods of at least two consecutive frames in which the hand touches the object in the reference. For each hand, we take up to 48 frames spread across these periods, longest first. Its velocity term also covers steps between observed and predicted frames within a contact period.

Each energy-score term in Equation (4) is

ES⁡({y^i}i=1M,y)=1M∑i∥y^i−y∥2−12M(M−1)∑i≠j∥y^i−y^j∥2,(5)\operatorname{ES}(\{\hat{y}_i\}_{i=1}^{M},y)=\frac{1}{M}\sum_i\lVert\hat{y}_i-y\rVert_2-\frac{1}{2M(M-1)}\sum_{i\ne j}\lVert\hat{y}_i-\hat{y}_j\rVert_2, \tag*{(5)}

where y^i=ϕ(X^i)\hat{y}_i=\phi(\hat{X}_i) and y=ϕ(X)y=\phi(X) summarize a sample and the reference. Both summaries measure changes from the reference pose at the present frame, and when the hand or the object is predicted, they also include each wrist’s change relative to the object. We scale the dense summary so that its size does not depend on the number of predicted frames. Table 4 gives this scaling and how the landmark summary represents the final rotation change. We also penalize predicted objects that extend more than 10 mm below nearby supporting surfaces, such as tabletops.

Training. We train for 40,000 steps with AdamW (Loschihlov & Hutter, 2019) and evaluate the final checkpoint (see Table 4 for the remaining settings). We sample windows with source and label-quality weights, up-weighting windows with a left hand by up to 10× to balance supervision across hands. We also use a curriculum in which extracted videos make up about 23.5% of sampled windows for the first 20,000 steps, then decrease linearly to 2% by step 32,000 and stay there, to focus the end of training on the more accurate annotated data.

For augmentation, with probability 0.5, we rotate the hands, object, and scene collectively around the vertical axis by a Gaussian angle with standard deviation 0.35 rad. Independently, with probability 0.5, we also resample a 30 fps window at 30, 15, or 10 fps while maintaining the frame timestamps.

Loss weights. Since our losses differ greatly in scale, we use gradient norms to set their weights instead of setting them manually. Before training, we set each loss weight so it contributes a target fraction of the flow-matching gradient by measuring its gradient norm on batches from the annotated datasets. Wrist and object position, the energy score, and the support penalty keep a weight of 1, and the contact-classifier weight rises linearly from about zero to 0.1 over the first 5,000 steps. Table 5 lists the resulting weights.

Extraction Pipeline Details

Pipeline Implementation

We describe the exported trajectories, processing steps, and quality checks for the video extraction pipeline introduced in § 3.4.

Exported trajectories. Each exported trajectory contains frames sampled at approximately 10 Hz, with their original video timestamps. It includes MANO parameters [48] for up to two hands, per-frame validity flags for each hand, and the rotation and translation of a reconstructed object mesh. The export also includes an estimated gravity direction when available.

Clip selection and validation. We take candidate clips from Action100M’s timestamped action captions [13]. Molmo 2 [18] names the manipulated object from each caption, with no predefined object list, and clips where it names none are discarded. The same model is used again, this time on the video, to determine whether a hand is interacting with any objects. In clips that pass, it checks one frame per second to find when the interaction occurs, and these times decide which frames we use to locate the object. Clips are split at detected shot boundaries, and each shot of at least 4 seconds is processed independently.

Hand and object estimation. WiLoR [45] gives MANO hand estimates in each frame. We discard low-confidence detections and short-lived tracks, and reject the clip if no hand track remains. To find the object, we ask Molmo 2 [18] to point to it in a frame using the name given in the clip’s caption. We remove outlier points and require the remaining points to lie near a hand whenever one is tracked in that frame. If we can’t find the object with enough points in up to four frames, we discard the candidate clip. SAM 3 [10] then tracks the object mask forward and backward through the clip, starting from the accepted points.

SAM 3D [15] then reconstructs the object mesh from one frame, chosen to have a large object mask. Masks much larger than the adjacent frames or multi-part or oddly shaped are given lower priority. At other sampled frames, SAM 3D reuses this mesh to estimate the object’s pose and scale. We run the full reconstruction only when this fails for a frame.

Object motion estimation. CoTracker3 [29] tracks points in the object mask, and depth from MoGe [55] lifts them into 3D. RANSAC PnP [23, 31] gives initial object poses, which we refine before removing tracks with high reprojection error. Bundle adjustment then refines the poses and the points’ positions within the object together, with fixed camera poses. The Huber loss [28] assesses agreement with image tracks, while additional penalties promote smooth rotations and maintain the estimated 3D shape. Rotations come from this fit and object centers from SAM 3D’s estimates. If no fit is found or if the median per-frame reprojection error is > 12 pixels, we fall back on the rotation estimates from SAM 3D.

Joint optimization. We then fix the hand shape and size and jointly refine the hand and object poses. Although the object shape is fixed, we refine the object size estimated from MoGe depth with a single scale factor per clip. The objective penalizes interpenetration, fits the reprojected object to its mask, and maintains hand contact with the object. Other terms discourage abrupt motion, unnatural finger poses, and large deviations from the initial estimates. We also penalize changes in the hand’s position relative to the object and, during grasps, changes in its relative rotation.

We optimize with Adam and a cosine learning-rate schedule in stages. In the initial frames, we align the wrists and fix those poses. We then refine the wrist and object motions, followed by the finger poses, and finally all of them together. A final stage again refines the wrists and fingers in frames with low tracking confidence and poor hand-object alignment by fixing the object and pulling each finger toward its position relative to the palm in nearby reliable frames. After optimization, we correct object rotation during occlusions, remove a learned depth bias, and smooth the remaining hand jitter (Appendix § 8.§ 2).

Object scale and depth correction. An object’s reconstruction can change size and depth across frames and still match the image mask, since a larger object farther away casts the same silhouette. We correct this when the 95th-percentile object size is at least 1.35 times the 5th-percentile size. Each frame’s mesh is first rescaled to the clip’s median object size. Comparing its projected area with the observed mask area then gives a smooth depth correction, leaving out frames where hands cover at least half the projected object. We reject a correction with too few reliable observations or increased depth variation. Any hand in contact with the object shifts along with it, so the contact is maintained.

Interaction intervals. We call a frame an interaction frame when the object is visible and a tracked hand is within 5 cm of its surface. Consecutive interaction frames form intervals, and we merge intervals separated by at most 2 seconds. We then extend each interval by up to 0.75 seconds on both sides to include the frames where the hand approaches the object or releases it. Frames where the estimated object pose jumps abruptly are likely reconstruction errors. We leave them out, and no interval is merged or extended across them. We export each interval as a trajectory if it lasts at least 2.5 seconds and includes at least eight frames. Trajectories from the same clip do not overlap.

Final pose refinement. We adjust each trajectory’s object position so that the projected silhouette more closely resembles the observed mask. Each modification is restricted to 0.30 m of translation and 35° of rotation. We consider a hand to be grasping an object if it remains within 4 cm of it for multiple consecutive frames. To maintain their relative stance throughout a grip, we adjust each frame and move the hand with the object. Next, without altering finger articulation, we smooth the hand’s rotation in relation to the object during grasps. Lastly, we relocate each hand by up to 8 mm to minimize residual hand-object penetration.

Gravity estimation. We estimate gravity from a supporting surface, such as a table or the floor, in the MoGe depth maps of several trajectory frames. The surface must appear consistently across these frames and lie below the hands, the object, and most of the scene. We favor large surfaces near the interaction, especially one the object rests on. A wall can meet these conditions too, since the hands and the object are in front of it. But for a wall to be the floor, the camera would usually have to be rolled far to the side or pointed sharply upward. People rarely film their hands like that, so we reject these surfaces. There are two more cases that we skip, because in both of them we cannot tell which way is up. A surface the camera sees at a grazing angle could face up or down. A surface facing the camera squarely could be a wall in front of the camera or a table filmed from straight above. Gravity is the normal of the selected surface, pointing away from the interaction; if no surface qualifies, we leave it undetermined.

Quality filtering. We keep only trajectories that agree with the video. The projected object must match the outline and size of its tracked mask, and we reject trajectories in which the object changes shape or size or moves abruptly. A hand must also stay within 5 cm of the object in at least 25% of the frames. Further checks catch shot boundaries, excessive hand-object penetration, and object masks that follow the hand rather than the object. We also drop trajectories where the hand touches the object in 3D but, in the image, almost never touches its mask. Lastly, we ask Molmo 2 to point to “the object held in the hand” in several frames, and we reject the trajectory if it often returns no point or points far from our tracked mask (i.e., we are likely tracking the wrong object).

Refinement Details

Object rotation. When the object is occluded for at least three consecutive frames, the grasping hand’s motion guides its rotation only when contact is stable and the camera rotates slowly. During the occlusion, the object rotates more closely with the hand, with the correction growing gradually as the occlusion begins and shrinking as it ends. We reject corrections that introduce large rotation jumps.

Bias denoiser. A separate denoiser learns to recover clean trajectories from simulated extraction errors. It trains on 11,568 windows of 50 frames from DexYCB and HO3D, sampled at 10 Hz, excluding the 129 segments from the sequences we use to evaluate refinement. The simulated errors are designed to reproduce the observed errors in the extraction pipeline. For instance, hands and objects are positioned too near or too far from the camera, their positions are disturbed over several frames, and the object’s orientation changes gradually.

The denoiser receives the hand and object poses at each frame, including finger articulation, and predicts corrections that we add to the standardized input states. It is an eight-layer bidirectional transformer with width 384, six attention heads, and feedforward width 1024. We train it for 15,000 steps with batch size 256, learning rate 2×10−42 \times10^{-4}, and 500 warmup steps. The loss is squared reconstruction error with weights of 10 for translation, 3 for rotation, and 1 for finger articulation, plus a velocity term with weight 0.5 that encourages the recovered motion to match the clean trajectory.

Applying the correction. On extracted trajectories, we use only part of this output: how far each hand and the object should move toward or away from the camera. We average these corrections over the trajectory and apply the same shift at each frame. To maintain their relative positions, the object and any hands in contact combine their shifts and move together. Next, we smooth each wrist’s position and rotation over three consecutive sampled frames. We do the same for the finger rotations, which we also keep within anatomical limits.

Evaluation. Table 6 evaluates refinement on 30 DexYCB and 28 HOT3D sequences. Full refinement follows the procedure described above, which keeps only the denoiser’s constant depth shift. We compare the final recipe with the raw, unrefined poses and two other refinement variants: one that applies every denoiser correction and one that takes a five-frame moving average of the raw poses. Full refinement lowers the median wrist error from 106.4 to 88.6 mm on DexYCB and from 191.2 to 158.7 mm on HOT3D. Although applying every denoiser correction reduces depth bias about as much as full refinement, it increases hand surface error on both datasets and object rotation drift on HOT3D. Moving-average smoothing slightly lowers the error on DexYCB but raises it on HOT3D.

VariantDexYCB wristDexYCB depthDexYCB surfaceDexYCB obj. depthHOT3D wristHOT3D depthHOT3D surfaceHOT3D obj. depthHOT3D drift
Raw extraction106.4−93.526.1−56.2191.2153.367.469.730.1
All denoiser corrections97.6−4.341.9−12.3162.572.593.619.643.5
Moving-average smoothing99.2−62.429.1−48.6205.7173.272.175.732.2
Full refinement88.6−4.727.3−23.8158.778.670.219.430.1

Table 6. Refinement results. Median errors in millimeters, except object rotation drift in degrees. Hand metrics use the right hand, and surface is the hand vertex error after aligning the wrists. Depth errors are signed and best at zero.

Extraction Quality

We evaluate the extraction pipeline of § 3.4 on the EPIC-Contact test set [3] with the authors’ released evaluator and fitted reference poses (Table 7). Unlike HOPformer, our pipeline is not trained on EPIC-Contact. It produces trajectories for 215 of the 237 test clips, and we score both methods on these clips. Since our trajectories are sampled at about 10 Hz, we sample HOPformer’s predictions at the same frames for metrics that depend on frame rate. Our hand joint error is lower than HOPformer’s (16.0 vs. 19.7 mm), and so is our motion deviation (10.6 vs. 13.8 mm). Our contact deviation (34.6 vs. 20.8 mm) and hand-object relative position error (71.8 vs. 64.7 mm) are higher, reflecting the main limitation of the extracted trajectories, namely absolute depth and hand placement relative to the object.

Table 7: EPIC-Contact reconstruction. Published results cover all 237 test clips. We score the other rows on the 215 clips our pipeline completes. Motion deviation and acceleration depend on frame rate, so we compute them for both methods at our frames, sampled at about 10 Hz. Acceleration is given for the hand and the object. SR is the percentage of frames whose object pose error is below 5% or 10% of the object diameter. HOPformer is trained on EPIC-Contact and, like the other published methods, receives prepared hand crops and each clip’s ground-truth object mesh. In the last block, HOPformer instead receives only the video frames, hand crops from the WiLoR detector, and the SAM 3D mesh of the object, aligned to the template of the class it predicts, and our pipeline reconstructs the mesh from video given the object’s name. Bold marks the better result in the last block.

MethodContact deviation ↓\downarrow (mm)Relative position ↓\downarrow (mm)Motion deviation ↓\downarrow (mm)Acceleration ↓\downarrow (m/s2m/s^2)Hand joint error ↓\downarrow (mm)[email protected] ↑\uparrow (%)[email protected] ↑\uparrow (%)
Published results
ArcticNet-SF ([22])94.2166.770.13.9 / 5.344.816.555.9
JointTransformer ([1])30.178.620.03.1 / 9.522.917.656.9
HOPformer ([3])20.765.811.42.5 / 4.119.929.869.7
Same 215 clips, prepared inputs and ground-truth mesh
HOPformer20.864.713.83.9 / 7.919.729.668.9
Same 215 clips, video frames only
HOPformer27.268.514.24.5 / 8.123.915.156.1
Ours34.671.810.63.7 / 3.816.028.574.9

Table 7.

Table 8 repeats the comparison when every input comes from the video, as for an internet clip. HOPformer then receives hand crops from the WiLoR detector and the SAM 3D reconstruction of the object, aligned to the template of the class it predicts. On EPIC-Contact, its hand error rises from 19.7 to 23.9 mm and its object error from 20.6 to 24.0 mm, both above ours (16.0 and 18.3 mm). Its object head was trained to pose its own templates, and it predicts the right class for only 60% of clips and labels almost every DexYCB object a bottle. HOPformer also depends on the egocentric viewpoint. On egocentric HOT3D, its hand error is close to ours (52.2 vs. 55.0 mm), but on third-person DexYCB it rises to 96.6 mm, against 10.3 mm for ours. The failure depends on the camera. For the camera that views the table from the side, its hand error is 117.8 mm, compared with 41.4 mm for a camera that looks down at about 50°, while ours stays between 9 and 16 mm. Our object error is also lower on both datasets (25.5 vs. 38.5 mm on HOT3D and 20.7 vs. 28.0 mm on DexYCB).

Table 8: HOPformer and our pipeline when every input comes from the video. HOPformer receives the video frames, hand crops from the WiLoR detector, and the SAM 3D mesh of the object, aligned to the template of the class it predicts. Hand error is the 21-joint error after aligning the wrist. Object error is the ADD-S error after aligning the object centers, as in the EPIC-Contact evaluator, which averages the distance from each point of the predicted mesh to the nearest point of the reference mesh. Errors are in millimeters, and both methods are scored on the same frames. EPIC-Contact is HOPformer’s training domain, HOT3D is also egocentric, and DexYCB is filmed from third-person views.

MethodEPIC-Contact Hand ↓\downarrowEPIC-Contact Object ↓\downarrowHOT3D Hand ↓\downarrowHOT3D Object ↓\downarrowDexYCB Hand ↓\downarrowDexYCB Object ↓\downarrow
HOPformer23.924.052.238.596.628.0
Ours15.818.355.025.510.320.7

Table 8.

HOPformer is cheaper to run. On one H100, it needs about 39 ms of GPU time per frame including hand detection, or about 9 GPU-seconds per candidate clip, whereas our pipeline needs about 2.8 GPU-minutes per candidate clip, or about 28,700 H100-hours for the 610,080 candidate clips we processed. HOPformer, however, can only pose objects from its nine EPIC-Contact classes, has no check that rejects failed reconstructions, and is accurate only near its training viewpoint, so it cannot replace our pipeline for building a corpus from arbitrary internet videos.

Prediction Benchmark Details

Benchmark construction. The 256 annotated windows come from 168 recordings: 36 from DexYCB; 35 each from HO3D, HOT3D, OakInk2, SHOW3D, and TACO; 24 from HO-Cap; and 21 from H2O. The 75 internet windows come from 75 different Action100M videos. We further refine the extracted poses to better match the video and add a new AI-assisted review using six sampled frames to increase reference accuracy. The selection process never uses model predictions or errors, and no window is hand-selected.

How extraction error affects internet-video scores. We evaluate predictions on internet videos against our own extracted trajectories, so extraction errors can affect the results. Much of what Appendix § 8.§ 2 reports, however, is a constant offset per clip, mostly in depth (45–62% of the squared position error on DexYCB, 77% on HOT3D). An offset like this largely cancels. If a clip sits 10 cm too far from the camera, the observed frames the model receives sit 10 cm too far as well, and so does the reference. What remains drifts slowly over the clip, is also mostly in depth, and this part does affect the scores. On clips that pass our quality checks, a true motion of 50–150 mm comes out 10–15% too short, and the error in that motion is smaller across the image than in depth, with medians of 28 versus 49 mm for objects and 41 versus 55 mm for wrists.

Trajectory scoring. ADE uses only generated frames in which the hand or object is present, and FDE uses only windows in which the reference moves at least 50 mm from its last observed position. Best of 20 selects the sample with the lowest ADE summed over the predicted hands and object, and reports every metric for that sample. For samples x1,…,xNx_1,\ldots,x_N and a reference yy, the energy score is 1N∑i∥xi−y∥−12N(N−1)∑i≠j∥xi−xj∥\frac{1}{N}\sum_i\lVert x_i-y\rVert-\frac{1}{2N(N-1)}\sum_{i\ne j}\lVert x_i-x_j\rVert, an unbiased estimate of E∥X−y∥−12E∥X−X′∥\mathbb{E}\lVert X-y\rVert-\frac{1}{2}\mathbb{E}\lVert X-X'\rVert ([24]). It is computed on the same frames and windows as the matching ADE, FDE, or mid-gap error.

Physical metrics. Finger error (Table 9) is the angle between predicted and reference rotations, averaged over the 15 finger joints of each hand. For closed object meshes, we measure how deeply hand vertices penetrate the object and the percentage of vertices inside it. Joint limits follow Table 4 and are strict, since 14.8% of joint poses in the annotated references exceed them. A violation therefore counts only when the predicted joint exceeds a limit and the reference joint does not, and we report violations as a percentage of all evaluated joint poses.

ModelHands ADE ↓\downarrowObject ADE ↓\downarrowHands FDE ↓\downarrowObject FDE ↓\downarrowInside (%)
N=1N=1N=20N=20ESN=1N=1N=20N=20ESN=1N=1N=20N=20ESN=1N=1N=20N=20ESHandsObject
Main model86.760.249.664.942.037.5128.593.374.8110.376.965.18279
Losses and relation tokens
No physical-loss group90.262.251.364.943.038.5135.4101.380.1111.980.568.88176
No relation tokens91.262.451.167.943.739.3132.798.876.4113.478.867.78176
Scene conditioning and augmentation
No scene encoder86.760.750.566.542.738.9129.695.475.9116.180.169.38479
No pose or stride augmentation90.262.251.267.443.339.0134.5102.778.7115.981.269.08077
ModelFinger error (°) ↓\downarrowPenetration depth (mm) ↓\downarrowPenetrating vertices (%) ↓\downarrowJoint-limit violations (%) ↓\downarrow
Main model12.93.014.209.9
Losses and relation tokens
No physical-loss group12.92.983.9410.2
No relation tokens13.13.033.9410.1
Scene conditioning and augmentation
No scene encoder12.82.934.149.9
No pose or stride augmentation13.33.033.9410.8

Table 9. Model ablations. (a) Trajectory errors in millimeters for one sample (N=1N=1), the best of 20 (N=20N=20), and the energy score of all 20 (ES). ADE averages the six tasks that generate each entity, and FDE the three where its final pose is not given. Inside is the percentage of windows with the reference inside the samples at the final position (§ 4.1). (b) Physical metrics of one sample.

Interaction metrics. The hand-object metrics of Table 10 use frames where the hand and object are both present and at least one of them is generated. A fingertip touches the object when it is within 10 mm of its surface. Tip error is the fingertip position error relative to the object on frames where the reference hand touches the object, and contact F1 accepts predicted contact up to two frames earlier or later than the reference. A reference grasp is a run of at least three frames in which at least two reference fingertips touch the object. On each grasp frame, the movement error is the difference between how the predicted and reference fingertips have moved relative to the object since the grasp began, averaged over the five fingertips. The frame is held when at least two predicted fingertips touch the object and this error is at most 10 mm, slipping when a predicted fingertip touches the object but the frame is not held, and detached when every predicted fingertip is at least 20 mm away. Other frames are unclassified, and we report the percentage of grasp frames in each class.

ModelTip error (mm) ↓\downarrowContact F1 (%) ↑\uparrowHeld (%) ↑\uparrowSlipping (%) ↓\downarrowDetached (%) ↓\downarrow
Main model65.768.215.547.524.9
Losses and relation tokens
No physical-loss group72.265.314.344.529.1
No relation tokens70.466.314.245.528.1
Scene conditioning and augmentation
No scene encoder67.667.315.646.626.0
No pose or stride augmentation70.066.514.445.827.8

Table 10. Interaction quality of the ablations. Means over the 20 samples of each window and over all seven tasks. Tip error is the fingertip error during contact, and held, slipping, and detached are percentages of reference-grasp frames. For every ablation, the 95% confidence interval of its difference from the main model excludes zero for tip error, contact F1, and detached frames.

Each window’s interaction score averages the 20 samples, and the two hands when both are present. We then average windows within each task, separately for annotated data and internet videos, and unlike for the other metrics, every annotated window counts equally regardless of its dataset. Tasks count equally, and so do annotated data and internet videos. For differences between models, we estimate 95% confidence intervals by resampling video segments 5,000 times, using the same re-sampled segments for both models.

Learned baselines. For Table 2, we use the USST ResNet-18, ObjectForesight EPIC-DiT, and Uni-Hand SAM-3D-Body/DINOv2 checkpoints, the variant of each that scores best on this benchmark. ObjectForesight takes one step per video frame, which scores better than one step per 1/55 s, the frame interval of its training videos. The ForeHand4D checkpoint was trained on DexYCB, HOT3D, and H2O, and the Uni-Hand checkpoints on robot demonstrations. Methods that use depth get monocular depth from Depth Anything 3 for MotionForesight and from MoGe-2 for the others, including our model. General Flow and MolmoMotion get action text for each clip, from dataset labels where available and a generic description otherwise. All predictions start from the reference at the last observed frame. To ensure a fair comparison with hand forecasting methods, we shift their predictions to start at the observed wrist, removing a constant placement offset that would otherwise dominate their error. For methods that predict 3D point trajectories, we fit a rotation and translation per frame without changing object size, and we interpolate all predictions to the reference timestamps.

Model Ablations

We ablate our main model with configurations that exclude specific losses, inputs, or augmentations (Table 9 and Table 10). Each configuration is trained for the same number of steps and with the same optimizer settings as the main model, and we evaluate its final checkpoint as described in Appendix § 9.

Accuracy and spread of the samples. By energy score, the main model has the lowest ADE and FDE, yielding 1–7% lower values than the other configurations for hands and 3–6% lower for objects. At the final position, the main model’s samples are on average 2–6% closer to the reference for hands and 3–5% closer for objects, while the distance between samples is about the same or smaller. Across both data sources, the reference lies inside the samples about equally often for all configurations: 80–84% of windows for hands and 76–79% for objects, versus an ideal of 95%. On annotated data, the samples are too close together, and the reference lies inside the main model’s samples more often than for the ablations without physical losses, relation tokens, or augmentation (69% versus 64–66% for hands and 69% versus 60–63% for objects). On internet video, where every configuration comes close to the ideal, the main model’s samples are closer together and closer to the reference.

Losses and relation tokens. Two parts of the model are there for hand-object contact, the physical losses and the relation tokens (§ 3.2). The physical losses are the attachment, fingertip-distance, joint-limit, MANO pose prior, and support penalty terms, which supervise contact and plausibility on decoded trajectories (§ 3.3). We train one ablation without them and one without the relation tokens. The latter still has the attachment and fingertip-distance losses, so its contact is supervised. Removing either part weakens the grasp. In the main model, fingertip error during contact is 7–9% lower, contact F1 is 68.2% (versus 65.3% and 66.3%), and 24.9% of grasp frames are detached (versus 29.1% and 28.1%; Table 10). The closer grasp costs a little. Slipping, i.e., frames where the hand touches the object without following the reference grasp, rises to 47.5% (from 44.5% and 45.5%), and the share of penetrating vertices to 4.20% (from 3.94%), at about the same depth. The main model’s hand ADE and FDE are also 2–8% lower in every column of Table 9.

Scene conditioning. Nearby surfaces and obstacles limit where an object can go, and the scene encoder tells the model about them. We also train without it, i.e., without scene geometry or DINOv3 features (the object shape is still given). The hands are barely affected, within 2% on ADE and FDE, and detach about as often (26.0% versus 24.9% of grasp frames). The objects are not. The main model’s object FDE is 4–6% lower in every column of Table 9, and 7% lower for a single sample on annotated data, where the references are most accurate.

Augmentation. We rotate the hands, object, and scene together around the vertical axis and resample windows at lower frame rates (Appendix § 7) so that the model does not depend on the usual orientation and frame rate of each training dataset. Without this augmentation, every ADE and FDE in Table 9 gets worse. The main model’s are 3–9% lower, and the gap is largest for best-of-20 hand FDE. The ablation’s hand also detaches from the object more often (27.8% versus 24.9% of grasp frames). It has the most joint-limit violations in the table (10.8% versus 9.9%) and the largest finger error (13.3° versus 12.9°).

Interpreting the interaction scores. Each sample is scored against a single reference, frame by frame, so a sample that follows a plausible but different hand path is penalized. The scores therefore depend on how much of the hand’s path the model has to generate. Fingertip error during contact is 26 mm in temporal infilling and 46 mm when the hand motion is given, but 60–93 mm in the five tasks where the model generates most of the hand’s path. Table 10 averages all seven tasks. For scale, stay-put on the same frames has a fingertip error of 92 mm and a contact F1 of 55%. The main model beats it on both in every task, and its overall fingertip error is 28% lower.

Additional Model Results

This section adds energy-score checks and task-specialist comparisons to the evaluation in § 4, following the protocol in Appendix § 9.

Energy Score

Table 11 gives the numbers behind the matched-spread test of § 4.2. With the spread of our samples, the best baseline scores lower than before, but ours is still 9–44% lower in every column where the test applies. Paired 95% confidence intervals, from resampling windows within each dataset 5,000 times, exclude zero for every comparison with the best baseline in Table 1. Figure 6 shows examples.

Entity Error atFuture prediction object endFuture prediction hands endHands→obj. object endObj.→hands hands endTemporal infilling object middleTemporal infilling hands middleGoal, all given object middleGoal, all given hands middleGoal, hands given object endGoal, hands given hands middleGoal, object given object middleGoal, object given hands end
Best baseline142.6145.6108.6156.438.249.076.092.3149.092.376.0156.6
Best baseline with our spread112.2117.2––29.937.858.673.1120.273.258.6126.5
Ours, energy score of 20 samples89.094.435.159.622.229.851.059.371.162.053.570.5

Table 11. Energy score with matched spread. The best baseline of Table 1, alone and with Gaussian noise matching our samples’ spread in each window, against our energy score. Rigid carry is not tested.

Figure 6. What the energy score rewards. Examples for Table 11. Forecast object paths of our 20 samples (top) and of stay-put with noise matching their spread (bottom), in the plane of largest motion. The windows are clear cases where our score is lower, which holds in 65% of forecasting windows with at least 50 mm of object motion. Numbers are in millimeters.

Table 12 applies the inside test of § 4.1 to the released methods of Table 2, at their final predicted positions. USST and MotionForesight make one prediction, so they cannot be tested. The 20 samples of EgoH4 coincide, which is why Table 2 lists it with N=1N=1, and those of Uni-Hand and General Flow almost never contain the reference (0–4% of windows). ObjectForesight contains it in 69% of windows, against 78% for our model. ForeHand4D and MolmoMotion contain it more often than ours, but their samples are much wider and farther from the reference (§ 4.2).

Task Specialists

We train five specialists with the shared model’s architecture, data, source-sampling schedule, objective, and training budget. Four cover forecasting, hands-to-object prediction, object-to-hands prediction, and temporal infilling, and the fifth covers all three goal patterns. A specialist trains only on its assigned task family and has its own normalization statistics. Table 13 compares each one with the shared model on the tasks it trained for.

Future predictionHands→obj.Obj.→handsTemporal infillingGoal, all givenGoal, hands givenGoal, object given
Entityobjecthandsobjecthandsobjecthandsobjecthandsobjecthandsobjecthands
Error atendendendendmiddlemiddlemiddlemiddleendmiddlemiddleend
Energy score of the 20 samples
Specialist95.2101.237.562.021.728.753.660.053.061.354.761.0
Ours89.094.435.159.622.229.851.059.371.162.053.570.5
Reference inside the 20 samples (%)
Specialist737358597878817780778283
Ours787677818685867983828788

Table 13. Comparison with task specialists. In each column, the specialist is the model trained only for that task, and one goal specialist covers all three goal patterns. Bold marks the better of the two models in each block. Notation follows Table 1.

Under the energy score, the shared model is lower than the specialists in 7 of the 12 columns of Table 13, including every forecasting and cross-entity column (4–7%). The temporal-infilling specialist is only 2–4% lower. The goal specialist gains most for the entity whose final pose is not given, with a 13–25% lower energy score. These partial-goal patterns are each about 5% of the shared model’s training windows and 25% of the specialist’s (Table 3). The reference lies inside the shared model’s samples more often in every column, by 2–22 points.

Limitations

We extract the internet-video trajectories with pretrained models, and their mistakes end up in our training data. MoGe estimates depth from a single image, where a large object far away casts the same silhouette as a small one nearby, so the hands and object can sit at the wrong distance from the camera. Refinement fixes much of this, though not all, and extracted motions come out a bit short. WiLoR loses the fingers when the object covers them and occasionally takes a hand-shaped object for a hand. SAM 3D can flip an object’s orientation between frames, and we often fail to pose thin objects such as spoons or cables. On tripod shots, MegaSaM sometimes reports a small camera motion, which then shows up as motion of an object resting on the table. We also reject most candidate clips, many only after all these models and the per-clip optimization have run, so extracting more videos takes a lot of compute.

References

  1. [1]Karim Abou Zeid. JointTransformer: Winning submission to the HANDS’2023 ARCTIC challenge. https://github.com/kalabouzeid/JointTransformer, 2023.
  2. [2]Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Lingjuang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. In CVPR, 2025.arxiv.org/abs/2411.19167
  3. [3]Siddhant Bansal, Zhifan Zhu, Shashank Tripathi, Jiahe Zhao, Michael J. Black, and Dima Damen. Towards in-the-wild egocentric 3d hand-object pose estimation. In ECCV, 2026.arxiv.org/abs/2606.30598
  4. [4]Chen Bao, Jiarui Xu, Xiaolong Wang, Abhinav Gupta, and Homanga Bharadhwaj. HandsOnVLM: Vision-language models for hand-object interaction prediction. TMLR, 2025.arxiv.org/abs/2412.13187
  5. [5]Wentao Bao, Lele Chen, Libing Zeng, Zhong Li, Yi Xu, Junsong Yuan, and Yu Kong. Uncertainty-aware state space transformer for egocentric 3D hand trajectory forecasting. In ICCV, 2023.arxiv.org/abs/2307.08243
  6. [6]Homanga Bharadhwaj and Yash Jangir. MotionForesight: Re-purposing video models for future 3D scene-flow prediction. arXiv preprint arXiv:2607.16192, 2026.arxiv.org/abs/2607.16192
  7. [7]Homanga Bharadhwaj, Abhinav Gupta, Vikash Kumar, and Shubham Tulsiani. Towards generalizable zero-shot manipulation via translating human interaction plans. In ICRA, 2024a.arxiv.org/abs/2312.00775
  8. [8]Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2Act: Predicting point tracks from internet videos enables generalizable robot manipulation. In ECCV, 2024b.arxiv.org/abs/2405.01527
  9. [9]Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation. In CoRL, 2025.arxiv.org/abs/2409.16283
  10. [10]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025.arxiv.org/abs/2511.16719
  11. [11]Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2HOI: Text-guided 3D motion generation for hand-object interaction. In CVPR, 2024.arxiv.org/abs/2404.00562
  12. [12]Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. In CVPR, 2021.arxiv.org/abs/2104.04631
  13. [13]Delong Chen, Tejaswi Kasarla, Yejin Bang, Mustafa Shukor, Willy Chung, Jade Yu, Allen Bolourchi, Theo Moutakanni, and Pascale Fung. Action100M: A large-scale video action dataset. arXiv preprint arXiv:2601.10592, 2026a.arxiv.org/abs/2601.10592
  14. [14]Mingfei Chen, Yifan Wang, Zhengqin Li, Homanga Bharadhwaj, Yujin Chen, Chuan Qin, Ziyi Kou, Yuan Tian, Eric Whitmire, Rajinder Sodhi, Hrvoje Benko, Eli Shlizerman, and Yue Liu. Ego-MAN: Interaction-structured reasoning for egocentric 3D hand trajectory prediction. In ECCV, 2026b.DOI
  15. [15]Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, et al. SAM 3D: 3Dfy anything in images. arXiv preprint arXiv:2511.16624, 2025.arxiv.org/abs/2511.16624
  16. [16]Zerui Chen, Rolando Alexandros Potamias, Shizhe Chen, Jiankang Deng, Cordelia Schmid, and Stefanos Zafeiriou. HO-Flow: Generalizable hand-object interaction generation with latent flow matching. In ECCV, 2026c.arxiv.org/abs/2604.10836
  17. [17]Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-Grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In CVPR, 2022.
  18. [18]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611, 2026. 7, 20arxiv.org/abs/2601.10611
  19. [19]Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in-betweening with diffusion models. In ACM SIGGRAPH, 2024. 3arxiv.org/abs/2405.11126
  20. [20]Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, 2017. 17arxiv.org/abs/1702.04405
  21. [21]Valentin De Bortoli, Alexandre Galashov, J. Swaroop Guntupalli, Guangyao Zhou, Kevin Patrick Murphy, Arthur Gretton, and Arnaud Doucet. Distributional diffusion models with scoring rules. In ICML, 2025. 6arxiv.org/abs/2502.02483
  22. [22]Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In CVPR, 2023. 23arxiv.org/abs/2204.13662
  23. [23]Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981. 20
  24. [24]Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007. 6, 8, 24
  25. [25]Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnotate: A method for 3D annotation of hand and object poses. In CVPR, 2020. 8arxiv.org/abs/1907.01481
  26. [26]Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from RGB videos. In 3DV, 2021. 3arxiv.org/abs/2108.07044
  27. [27]Masashi Hatano, Zhifan Zhu, Hideo Saito, and Dima Damen. The invisible EgoHand: 3D hand forecasting through EgoBody pose estimation. arXiv preprint arXiv:2504.08654, 2025. 10, 29arxiv.org/abs/2504.08654
  28. [28]Peter J. Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 1964. 7, 20
  29. [29]Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. CoTracker3: Simpler and better point tracking by pseudo-labelling real videos. In ICCV, 2025. 7, 20arxiv.org/abs/2410.11831
  30. [30]Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2O: Two hands manipulating objects for first person interaction recognition. In ICCV, 2021. 8arxiv.org/abs/2104.11181
  31. [31]Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. EPnP: An accurate O(n) solution to the PnP problem. IJCV, 2009. 20DOI
  32. [32]Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C. Karen Liu. Controllable human-object interaction synthesis. In ECCV, 2024. 3arxiv.org/abs/2312.03913
  33. [33]Muchen Li, Sammy Christen, Chengde Wan, Yujun Cai, Renjie Liao, Leonid Sigal, and Shugao Ma. LatentHOI: On the generalizable hand object motion generation with latent hand diffusion. In CVPR, 2025a. 3DOI
  34. [34]Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos. In CVPR, 2025b. 7arxiv.org/abs/2412.04463
  35. [35]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023. 5arxiv.org/abs/2210.02747
  36. [36]Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In CVPR, 2022. 3arxiv.org/abs/2204.01696
  37. [37]Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang, Yebin Liu, and Li Yi. TACO: Benchmarking generalizable bimanual tool-action-object understanding. In CVPR, 2024. 8arxiv.org/abs/2401.08399
  38. [39]Junyi Ma, Wentao Bao, Jingyi Xu, Guanzhong Sun, Yu Zheng, Erhang Zhang, Xieyuanli Chen, and Hesheng Wang. Uni-Hand: Universal hand motion forecasting in egocentric views. PAMI, 2026. 10, 29DOI
  39. [40]Jeongwan On, Kyeonghwan Gwak, Gunyoung Kang, Junuk Cha, Soohyun Hwang, Hyein Hwang, and Seungyul Baek. BIGS: Bimanual category-agnostic interaction reconstruction from monocular videos via 3D gaussian splatting. In CVPR, 2025. 3arxiv.org/abs/2504.09097
  40. [41]Lorenzo Pacchiardi, Rilwan Adewoyin, Peter Dueben, and Ritabrata Dutta. Probabilistic forecasting with generative networks via scoring rule minimization. JMLR, 2024. 6
  41. [42]Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi, Jitendra Malik, and Francois Hogan. SPIDER: Scalable physics-informed dexterous retargeting. In IROS, 2026. 2, 10arxiv.org/abs/2511.09484
  42. [43]William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 17arxiv.org/abs/2212.09748
  43. [44]Romain Pic, Clément Dombry, Philippe Naveau, and Maxime Taillardat. Proper scoring rules for multivariate probabilistic forecasts based on aggregation and transformation. Advances in Statistical Climatology, Meteorology and Oceanography, 2025. 6DOI
  44. [45]Rolandos Alexandros Potamias, Jinglei Zhang, Jiankang Deng, and Stefanos Zafeiriou. WiLor: End-to-end 3D hand localization and reconstruction in-the-wild. In CVPR, 2025. 7, 20arxiv.org/abs/2409.12259
  45. [46]Aditya Prakash, Richard Li, David Forsyth, and Saurabh Gupta. Bimanual 3D hand motion and articulation forecasting in everyday images. arXiv preprint arXiv:2510.06145, 2025. 10, 29arxiv.org/abs/2510.06145
  46. [47]Patrick Rim, Kevin Harris, Braden Copple, Shangchen Han, Xu Xie, Ivan Shugurov, Siziňhe An, He Wen, Alex Wong, Tomas Hodan, and Kun He. SHOW3D: Capturing scenes of 3D hands and objects in the wild. In CVPR, 2026. 8arxiv.org/abs/2603.28760
  47. [48]Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 2017. 4, 20
  48. [49]Novin Shahroudi, Mihkel Lepson, and Meelis Kull. Evaluation of trajectory distribution predictions with energy score. In ICML, 2024. 8
  49. [50]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. DINOv3. arXiv preprint arXiv:2508.10104, 2025. 5, 17arxiv.org/abs/2508.10104
  50. [51]Rustin Soraki, Homanga Bharadhwaj, Ali Farhadi, and Roozbeh Mottaghi. ObjectForesight: Predicting future 3D object trajectories from human videos. In ECCV, 2026. 3, 10, 29arxiv.org/abs/2601.05237
  51. [52]Neerja Thakkar, Tara Sadjadpour, Jathushan Rajasegeran, Shiry Ginosar, and Jitendra Malik. Poly-autoregressive prediction for modeling interactions. In CVPR, 2025. 3arxiv.org/abs/2502.08646
  52. [53]Neerja Thakkar, Shiry Ginosar, Jacob C Walker, Jitendra Malik, João Carreira, and Carl Doersch. Forecasting motion in the wild. In ECCV, 2026. 3arxiv.org/abs/2604.01015
  53. [54]Jikai Wang, Qifan Zhang, Yu-Wei Chao, Bowen Wen, Xiaohu Guo, and Yu Xiang. HO-Cap: A capture system and dataset for 3D reconstruction and pose tracking of hand-object interaction. arXiv preprint arXiv:2406.06843, 2024. 8arxiv.org/abs/2406.06843
  54. [55]Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jialong Yang. MoGe: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In CVPR, 2025a. 7, 20arxiv.org/abs/2410.19115
  55. [56]Shibo Wang, Haonan He, Maria Parelli, Christoph Gebhardt, Zicong Fan, and Jie Song. MagicHOI: Leveraging 3D priors for accurate hand-object reconstruction from short monocular video clips. In ICCV, 2025b.arxiv.org/abs/2508.05506
  56. [57]Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. InterDiff: Generating 3D human-object interactions with physics-informed diffusion. In ICCV, 2023.arxiv.org/abs/2308.16905
  57. [58]Yufei Ye, Abhinav Gupta, Kris Kitani, and Shubham Tulsiani. G-HOP: Generative hand-object prior for interaction reconstruction and grasp synthesis. In CVPR, 2024.arxiv.org/abs/2404.12383
  58. [59]Yufei Ye, Yao Feng, Omid Taheri, Haiwen Feng, Shubham Tulsiani, and Michael J. Black. Predicting 4D hand trajectory from monocular videos. In 3DV, 2026a.arxiv.org/abs/2501.08329
  59. [60]Yufei Ye, Jiaman Li, Ryan Rong, and C. Karen Liu. WHOLE: World-grounded hand-object lifted from egocentric videos. In CVPR Findings, 2026b.arxiv.org/abs/2602.22209
  60. [61]Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. In CoRL, 2024.arxiv.org/abs/2401.11439
  61. [62]Yuanwen Yue, Damien Robert, Jianyuan Wang, Sunghwan Hong, Jan Dirk Wegner, Christian Rup-precht, and Konrad Schindler. LitePT: Lighter yet stronger point transformer. In CVPR, 2026.arxiv.org/abs/2512.13689
  62. [63]Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. OAKINK2: A dataset of bimanual hands-object manipulation in complex task completion. In CVPR, 2024.arxiv.org/abs/2403.19417
  63. [64]Jianing Zhang, Chenhao Zheng, Yajun Yang, Max Argus, Rustin Soraki, Winson Han, Taira Anderson, Chun-Liang Li, Shuo Liu, Jiafei Duan, Zhongzheng Ren, Jieyu Zhang, and Ranjay Krishna. MolmoMotion: Forecasting point trajectories in 3D with language instruction. arXiv preprint arXiv:2606.18558, 2026.arxiv.org/abs/2606.18558
  64. [65]Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3D: Exploring unified 3D representation at scale. In ICLR, 2024.arxiv.org/abs/2310.06773
  65. [66]Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In CVPR, 2019.arxiv.org/abs/1812.07035

Paper details

vlnc:W_23c72m34
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents