ToolDex: An Ego-Exo Human Demonstration Dataset for Creative Physical Tool Use

Introduction

Tool use is a fundamental human capability and a key step toward general-purpose embodied intelligence [23, 53, 13] (Goodall 1964). In everyday environments, humans do not rely solely on a fixed set of predefined tools; they also flexibly repurpose available objects based on their physical properties and functional affordances. For example, when a knife is unavailable, a person may use a spoon, chopsticks, or another nearby object to divide a cake. This form of creative tool use requires more than recognizing object categories. It demands physical reasoning about contact, geometry and task constraints. As such, physical tool use provides a natural testbed for evaluating whether embodied agents can generalize beyond memorized object-action associations and adapt to novel physical scenarios.

Despite recent progress in robotic manipulation [5, 6, 26], learning generalizable tool-use behavior remains difficult [51, 56]. Task and motion planning methods can solve structured tool-use problems when symbolic task descriptions are carefully specified, but often struggle in open-ended real-world settings [16]. Recent large language models and vision-language models have shown promising reasoning ability for selecting or describing tools, yet they remain limited in grounding such reasoning into physically executable manipulation [12, 61, 31, 15]. This gap is especially pronounced in tasks involving deformable target objects (such as sand or cake), non-rigid contacts, and long-horizon interactions, where success depends on fine-grained hand-object-tool dynamics rather than high-level semantic reasoning alone.

A major obstacle is the lack of datasets and benchmarks that capture human physical tool-use capabilities at sufficient scale, fidelity, and diversity. Existing robot tool-use benchmarks are often simulation-based or focus on simplified object interactions, making it difficult to evaluate the complexity of real-world manipulation [61, 15, 2, 67]. Meanwhile, recent egocentric human manipulation datasets have greatly expanded the coverage of hand-object interactions [22, 44, 30, 32]. Among them, TACO explicitly captures tool-action-object compositions for hand-object manipulation [32]. However, it does not explicitly model the target objects manipulated by the tools, such as cakes to be cut or water to be stirred. As a result, existing datasets either contain limited tool-use instances or lack the full hand–tool–target-object interactions and annotations needed to study physical tool use.

In this paper, we introduce TOOLDEX, a large-scale ego-exo human demonstration dataset for learning and benchmarking creative tool-based manipulation (Figure 1). Our dataset is designed around everyday tool-use scenarios in which humans creatively solve manipulation tasks by selecting, adapting, and repurposing available objects. A key feature of our dataset is its explicit coverage of deformable target objects, such as dough, fluids, and sand, where success depends on fine-grained contact and material interactions. To capture these behaviors with high fidelity, we use a synchronized multi-view capture system consisting of two egocentric cameras and eight exocentric cameras. Egocentric views capture local interaction context, while exocentric views enable accurate multi-view tracking and reconstruction. The resulting dataset provides rich annotations for studying both high-level tool-use reasoning and low-level physical manipulation.

Vector (PDF)

Beyond data collection, we use TOOLDEX to establish a benchmark suite for evaluating tool-use capabilities across three levels. First, we evaluate physical tool-use reasoning, testing whether vision-language models can identify appropriate tools, infer object positions, and provide plausible manipulation strategies. Second, we evaluate 3D and 4D reconstruction, focusing on the recovering hand, tool, and deformable scene states from multi-view observations. Third, we evaluate robot policy learning, where human demonstrations are used to train robot policies for tool-based manipulation. Together, these benchmarks characterize semantic reasoning, spatial perception, and embodied control, offering a comprehensive evaluation protocol for physical tool-use.

Our contributions are threefold. First, we present a real-world ego-exo dataset centered on creative physical tool use spanning diverse tools, deformable target objects, tasks, and interaction strategies. Second, we provide rich multi-view annotations that support hand tracking, tool tracking, object interaction analysis, and scene reconstruction. Third, we introduce a benchmark suite that evaluates embodied agents from reasoning to reconstruction to policy execution. We hope this dataset will serve as a foundation for studying human tool-use behavior and for developing robots that can creatively use tools in complex, open-world environments.

Related Work

Human manipulation datasets. Large-scale human video datasets have substantially advanced research on egocentric perception, activity understanding, and human-object interaction (Damen et al., 2022; Grauman et al., [20, 21]). A related line of work focuses on hand-object interaction and manipulation (Chao et al., [7]; Liu et al., [33]; Yang et al., [62]; Zhan et al., [65]; Banerjee et al., [3]; Fu et al., [14]; Hoque et al., [22]; Punamiya et al., [44]; Li et al., [30]), shown in Table 1. HOI4D provides RGB-D observations with 3D hand and object annotations (Liu et al., [33]), OakInk focuses on affordance-aware hand-object interactions (Yang et al., [62]; Zhan et al., [65]), HOT3D enables accurate multi-view hand-object tracking (Banerjee et al., [3]), and GigaHands scales bimanual manipulation data (Fu et al., [14]). However, they are not designed to systematically capture creative physical tool use in everyday scenarios. They focus on general activities, rigid-object interaction, or affordance modeling and lack detailed hand-tool-target-object interactions, and coverage of deformable objects. TOOLDEX explicitly targets creative physical tool use and provides data and benchmarks for both high-level reasoning and low-level physical interaction.

Tool-use and physical reasoning benchmarks. Recent work has increasingly explored tool use and physical reasoning as core aspects of embodied intelligence. TACO is closely related to tool-based manipulation, providing a bimanual hand-object interaction dataset with diverse tool–action–object compositions (Liu et al., [32]). However, it does not model the target objects being acted upon, limiting its ability to capture full tool-use dynamics. Beyond datasets, several benchmarks study physical reasoning and tool use from complementary perspectives. PHYRE evaluates agents on physics-based puzzle solving, emphasizing intuitive reasoning (Bakhtin et al., [2]), while PhysToolBench focuses on tool-use understanding in vision-language models, such as tool selection and strategy explanation (Zhang et al., [67]). In robotics, prior work explores creative tool use and planning under physical constraints (Xu et al., [61]), as well as leveraging vision-language models for tool design and manipulation (Gao et al., [15]). While these efforts highlight the importance of tool use, they either focus on reasoning without real-world interaction data or rely on simplified or simulated settings. In contrast, TOOLDEX provides real-world ego-exo demonstrations and a unified benchmark that connects tool-use reasoning, physical perception, and policy learning.

The ToolDex Dataset

Problem Formulation

Target-centric tool manipulation. Tool manipulation, following work on tool use in human and animal cognition Beck [4]; St Amant & Horton [54]; Shumaker et al. [52], is the indirect alteration of a target object’s physical state by an agent acting through a tool object, in a scene that may also contain support objects, such as containers or work surfaces, that constrain the interaction without being directly actuated. The agent does not act on the target object (e.g., a cake on a plate) directly: it actuates the tool, and the tool’s contact with the target produces the state change. We adopt this three-role decomposition—hand, tool, and target object—because it is the interaction between tool and target, shaped by tool geometry, target physical properties, and environmental support, that determines whether a manipulation succeeds. TOOLDEX is designed to capture precisely these interactions.

Tool manipulation as a goal-conditioned MDP. We formalize a tool-manipulation trial as a goal-conditioned Markov decision process $(S, A, T, \gamma, \mathcal{G})$. The state $s_t \in S$ is the configuration of all participants of the scene—tool, target, support, and agent—at time $t$. The action $a_t \in A$ is decomposed into two levels: a tool action that determines how the tool moves and, through contact, how the target’s state changes; and an agent action that determines how the agent’s manipulator moves and thereby controls the tool. The transition $T$ captures the physics of the interaction. The goal $g \in\mathcal{G}$ is the desired terminal state of the target object, specified per trial in two complementary ways: a natural-language description $l$, and a terminal-observation $s_T$ from a successful demonstration. A trial is a trajectory $\tau= (s_0, a_0, s_1, a_1, \ldots, a_{T-1}, s_T)$ paired with language goal $l$, and TOOLDEX is a collection of $(\tau, l)$ pairs.

Dataset Properties and Statistics

Diversity of tools, target objects, and actions. TOOLDEX contains 10,853 trials spanning 60 high-level (tool, action, target) task combinations, and 168 total variants that capture changes in tool instances and task attributes (e.g., different tool materials or shapes, and variations in target configurations). In total, the dataset includes 45 tool objects, 23 support objects, 16 deformable object types, and 16 action types. As shown in Figure 3, the tools span a range of geometries, including flat blades (knife, spatula), volumetric scoopers (spoon, ladle, ice-cream scooper, open scooper, measuring scooper), elongated rods (rolling pin, rolling rod, chopstick, straw), and bulkier creative-repurposing items (book, cup, pan), each affording a different contact profile with the target. Target objects span a wide range of physical properties: rigid (toy, box, banana), granular (nuts, sand, kinetic sand, coffee), deformable (dough, naan, towel, tomato sauce). For each (tool, action, target) combination, we collect trials for 60 minutes in which we vary the initial pose of the tool, target, and support, and additionally place 4–5 distractor tools on the work surface to produce realistic clutter and to require visual grounding of the active tool when conditioned on language.

Zero-shot and compositional generalization. The combinatorial structure of our dataset enables the study of both zero-shot and compositional generalization. As shown in Figure 2, each tool participates in 2.3 actions on average, and each action is realized by 3.6 tools on average across targets. From these (tool, action, target) triples, we construct two families of evaluation splits. Zero-shot generalization encompasses three cases, testing the model on a withheld tool, target, or action. Compositional generalization holds out a co-occurrence rather than an element, specifically a (tool, action, target) combination in which each individual component has been seen but their joint has not. For example, a held-out triple may evaluate a known tool performing a known action on a target it has not been paired with at training, even though that target has been manipulated by other tools.

Vector (PDF)

Multi-view capture of 3D interaction. We capture each interaction with high fidelity to recover the 3D state of the tool, target, and hand, which together provide the inputs required to learn the interaction function T defined in § 3.1. Each trial records eight calibrated exocentric Intel RealSense RGB-D streams at 1280 × 720 resolution and 30 Hz, together with a stereo egocentric stream from a ZED Mini stereo camera mounted on an Apple Vision Pro headset worn by the demonstrator. Depth observations from each exocentric viewpoint enable 3D reconstruction of state changes, while synchronized multi-view videos capture non-rigid phenomena such as dough deformation and granular flow. We provide per-frame multi-view segmentation masks for the target object, and tool and hand. Each trial lasts an average of 18.92 seconds and contains 567.58 frames.

Embodiment-agnostic data representation and collection. We collect human data rather than teleoperated robot data because we aim to record interactions instead of embodiment-specific motor commands, and because humans achieve substantially higher throughput on creative tool-substitution trials than teleoperation systems. To make the human data useful for robot learning, we record demonstrations at two levels. The tool action, represented by the 6-DoF tool pose, captures the physics of how the tool affects the target. This representation is embodiment-invariant and directly transferable to any manipulator capable of reproducing the trajectory. The agent action, represented by the MANO hand pose and wrist trajectory, provides the human-side supervision required by retargeting pipelines such as AnyTeleop Qin et al. [45] to map demonstrations onto a robot dexterous hand. In total, TOOLDEX contains 60 hours of recorded demonstrations across 10,853 trials and 6,158,809 annotated frames, collected at a throughput of approximately 60 trials per hour.

Data Collection

Following the task taxonomy introduced in § 3, we collect each demonstration in a single calibrated multi-camera studio rig and run an automated annotation pipeline offline after each capture session. The capture rig and annotation pipeline are designed jointly so that each recorded trial yields synchronized RGB-D video, per-frame 6-DoF tool pose, MANO hand poses for both hands, and multi-view segmentation masks for the target object.

Task and Object Curation

Each task combination is a (tool, action, target) triplet specifying a set of demonstrations. We curated the task combinations in two stages. We first enumerated 16 everyday tabletop tool-use scenarios drawn from cooking, food preparation, child play, and household activities, each defined by an action verb and a default tool–target pair. We then expanded each scenario along the $action \times tool$ axis. For each action, we explicitly enumerated the alternative tools a human might creatively substitute when the canonical tool is unavailable. We select tool, target, and support objects to span diverse geometries, material properties, covering rigid, granular, and deformable objects. Within each task combination, we vary the initial pose of the tool and target object on the work surface, and additionally place 4-5 tool objects as distractor objects in the scene to match the cluttered tabletop conditions of real deployments and require visual grounding of the correct tool when conditioned on language. Each task combination is paired with a natural-language instruction that specifies both the tool motion (e.g., “swing the spatula laterally under the egg, then lift”) and the intended effect on the target (“flip the egg so its underside faces up”).

Hardware Setup

Our capture rig (Figure 3) consists of 8 Intel RealSense 435 RGB-D cameras and an Apple Vision Pro headset fitted with a Stereolabs ZED Mini stereo camera. The 8 RGB-D cameras are mounted on a Vention extruded-aluminum frame surrounding the work surface, in positions designed to maximize joint coverage of the work volume while preserving sufficient view overlap for multi-view triangulation, 6-DoF object pose estimation, and 4D scene reconstruction. RGB and depth streams are recorded at 1280 × 720 resolution and 30 Hz; streams across the 8 cameras are synchronized in software. We calibrate the cameras using an April Tag followed by multi-view bundle adjustment to obtain rig-relative camera parameters. The ZED Mini, mounted on the Vision Pro, provides two additional 1280 × 720 stereo RGB egocentric streams. To support 6-DoF tool pose estimation, we scan each tool with a Shining3D scanner to obtain a high-resolution textured mesh. The sparse 8 camera setup overcomes occlusion and enables photo-realistic 3D and 4D scene reconstruction.

Data Collection and Annotation

Data is collected under a scripted protocol with a predetermined execution procedure. We collect one hour of data per task. We also vary the positions of the 4–5 distractor tools across recordings to introduce greater spatial diversity. All 8 exocentric RGB-D streams and the ZED Mini stereo stream are recorded synchronously and stored together. We then run an annotation pipeline that produces three synchronized outputs for each trial: 6-DoF tool pose, MANO hand pose for both hands, and per-frame multi-view segmentation masks for the target object. All annotations are manually verified for each trajectory. When automatic segmentation or tracking fails, annotators provide additional initialization prompts to correct the annotations.

Target-object segmentation. We segment target objects across 8 exocentric views using SAM2 (Ravi et al., [47]) with DINOV2 (Oquab et al., [42]) seed initialization using the cosine similarity of the scanned tool meshes; SAM2’s built-in video tracker propagates the mask across the rest of the trial.

Tool pose. We obtain per-frame 6-DoF tool pose by running FoundationPose (Wen et al., [58]) independently on each exocentric stream, conditioned on the scanned textured mesh of the tool used in that trial and seeded with a SAM2 (Ravi et al., [47]) first-frame mask. To consolidate per-view estimates into a single trajectory in the rig frame, we perform a confidence-weighted multi-view fusion using FoundationPose’s per-view score, followed by a sliding-window bundle adjustment that minimizes per-view reprojection error together with a temporal-smoothness term on the 6-DoF pose.

Hand pose. We estimate per-view MANO (Romero et al., [48]) parameters with WiLoR (Potamias et al., [43]), which produces hand pose at unknown scale from a single RGB image. We recover metric MANO predictions by solving for a per-trial global scale and rigid alignment of the WiLoR-predicted 3D hand keypoints to the multi-view RGB-D point cloud and the multi-view-triangulated 2D hand keypoints. We then apply the same multi-view fusion and temporal smoothing as for tool pose to obtain a single MANO trajectory per hand in the rig frame.

Joint hand–tool refinement. Following HO-Cap (Wang et al., [55]), we perform a joint hand–tool pose optimization that minimizes a weighted sum of (i) multi-view 2D hand-keypoint reprojection error, (ii) hand-keypoint-to-RGB-D-depth distance, (iii) tool 6-DoF reprojection error against the multi-view masks, and (iv) a signed-distance-field-based contact prior between the MANO mesh and the tool mesh that penalizes interpenetration while encouraging contact during active manipulation phases. Full pipeline details, hyperparameters, and per-component ablations are in the appendix.

Creative Physical Tool-Use Benchmarks

We establish a suite of creative tool-use benchmarks spanning physical reconstruction, reasoning, and robot learning. We evaluate sparse-view static and dynamic reconstruction with Gaussian Splatting (§ 5.1), benchmark vision-language models on physical tool-use reasoning (§ 5.2), and study robot policy learning from human tool-use demonstrations (§ 5.3).

Novel-View Gaussian Splatting for Sparse-View Tooluse: Static and Dynamic

Experiment setup. We benchmark sparse-view novel-view synthesis on 8-camera calibrated capture in two regimes: (i) static 3DGS using InstantSplat (Fan et al., [11]) (IS) as a backbone (ii) dynamic 4DGS training InstantSplat on $T$-frame sequences applied per-timestep, denoted IS-4D. Each scene is reconstructed twice (once with 8 cameras and once with $\mathrm{cam}_6$ dropped for ground truth comparison). We add a second stage single-step diffusion to the pipeline utilizing GSFix3D (Yin et al., [64]) and Difix3D (Wu et al., [59]) to enhance reconstruction and decrease noise.

Method evaluation. We also evaluate two-stage recipes: IS + Difix3D for 3DGS and IS-4D+Difix3D+GSFix3D for 4DGS. Stage 1 (reconstruction): a Mast3R (Leroy et al., [28]) crossview dense-point prior seeds the joint pose-and-Gaussian optimizer of InstantSplat, trained for 1k iterations under a clean photometric loss with no depth/distortion/normal regularizers. For 4DGS, Stage 1 is run independently per timestep. Stage 2 (rendered-view repair): a single-step Difix3D (Wu et al., [59]) pass, with GSFix3D (Yin et al., [64]) chained after Difix3D in the 4DGS variant (More benchmarks in Appendix § 8.§ 1).

Vector (PDF)

Reconstruction results. Table 2 summarizes both regimes. PSNR (in dB; higher = closer pixel-by-pixel to the held-out cam6 ground truth) measures photometric fidelity, while MUSIQ (on a 0–100 scale; higher = better learned-perceptual appeal) judges visual quality without any reference. Static: IS + Difix3D reaches 20.45 PSNR / 55.11 MUSIQ, beating InstantSplat alone by +0.5 dB and SplatFormer (Chen et al., [9]) by ∼7 dB. Dynamic: IS-4D + Difix3D + GSFix3D reaches 69.92 MUSIQ (+18.92 over IS-4D) while lowering PSNR by 3.6 dB – the photometric-vs-perceptual trade-off (Wu et al., [59]) is now measurable on held-out 4D. Both findings are made quantifiable only by our dataset’s calibrated 8-camera capture, which uniquely supports a held-out camera view test on tool-use scenes that competing tabletop datasets cannot reproduce. Without it, neither the static held-out gap nor the 4D photometric-vs-perceptual dissociation can be measured.

Benchmarking VLMs’ Physical Tool-use Capabilities

Experiment setup. We evaluate three levels of reasoning capabilities. For Tool Selection, models are asked to identify all tools that could plausibly complete the task, and a prediction is counted as correct if any predicted tool matches the demonstrated ground-truth tool. For Spatial Grounding, models localize the selected tool using bounding boxes, evaluated using IoU only on semantically matched tool predictions. For Behavioral Planning, models are provided with a natural-language instruction, e.g., “stir the coffee,” and must generate a high-level manipulation trajectory based on the first frame from a top-view camera. The evaluation subset contains 180 manually verified trajectories spanning 32 task variants. We evaluate a tool-center projection ground-truth trajectory and a correspondence-aware transported-point trajectory, using trajectory distance metrics including Mean L2 and Chamfer-L2. Details are in Appendix § 8.§ 2.

Tool selection and spatial grounding results. As shown in Table 3, modern VLMs demonstrate strong capability in selecting plausible tools. Claude-Sonnet-4.6, Claude-Opus-4.7 and GPT-6-Astra achieve the highest tool-selection performance of 80%, while GPT-6-Astra substantially outperforms all other models on spatial grounding with a bounding box IoU of 0.891. Notably, Gemini and Qwen models frequently correctly identify plausible tools but struggle to obtain correct bounding boxes suggesting that these VLMs possess substantially stronger semantic reasoning than fine-grained spatial understanding.

Table 3: Benchmark results on tool selection accuracy and bounding box IoU. Models: GPT-5-mini [37], GPT-6-Astra [38], Claude-Sonnet-4.6 (Anthropic, 2026), Claude-Opus-4.7 (Anthropic, 2026), Gemini-3.1-Pro [19], Qwen-3.6-27B [41].

Behavioral planning results. We additionally compare several general-purpose reasoning/VLMs with Gemini Robotics ER on zero-shot behavioral planning. As shown in Table 4, GPT-6-Astra achieves the highest evaluation-success rate (89.4%). Among successfully evaluated trajectories Gemini Robotics ER 1.6 (Gemini Robotics Team, 2025) achieves the lowest trajectory errors under both trajectory constructions. These results highlight a tradeoff between output reliability and model spatial awareness.

Table 4: Behavioral-planning results across all 160 trajectories for GPT-6-Astra [38], Gemma 4 31B [39], Gemini Robotics ER (Gemini Robotics Team, 2025), Claude Fable 5 [40], and Qwen 3.6 27B [41]. All distance metrics are reported in pixels and computed over successfully evaluated outputs.

Qualitative results. Qualitatively, many VLMs demonstrate creative physical reasoning beyond reproducing the canonical ground-truth tool. As shown in Figure 5, several models identify the rolling pin as a plausible alternative to the mortar and pestle for crushing nuts, while GPT-6-Astra produces near ground-truth spatial grounding. Additional examples are provided in Appendix § 8.§ 3. Predicted trajectories generally capture the intended manipulation behavior for short-horizon tasks such as pushing and cutting, but are less consistent for longer-horizon actions requiring repeated interactions such as sweeping or shaping. In Figure 5, Gemini Robotics ER correctly approaches the mortar and pestle but fails to capture the repeated grinding motion needed to crush the nuts. Additional qualitative examples are provided in Appendix § 8.§ 4. These results highlight the capabilities and limitations of modern VLMs, demonstrating that TOOLDEX provides a challenging benchmark for evaluating semantic understanding, spatial grounding, and task planning in real-world settings.

Vector (PDF)Vector (PDF)

Robot Physical Tool-Use from Human Demonstrations

We benchmark representative paradigms for transferring human tool-use demonstrations to dexterous robot policies, ranging from direct motion transfer to interaction-aware reinforcement learning. Kinematic Retarget directly executes retargeted human motion, while BC (Retarget) learns from the retargeted trajectories. ManipTrans Li et al. [29] uses residual reinforcement learning to track demonstrated interactions, and Dex-X (RL) Chen et al. [68] jointly adapts hand and arm motion through physical interaction.

Results. Table 5 summarizes the task success rates. Direct motion transfer performs poorly under physical interaction: Kinematic Retarget achieves 14.0% average success, while BC reaches 4.6%. In contrast, reinforcement-learning-based methods perform substantially better, with ManipTrans achieving 40.5% and Dex-X RL 55.1%. The gap is especially pronounced on contact-intensive tasks: pestle crushing improves from 16.8% to 46.7%, and squeegee collecting from 12.6% to 57.5%. Performance also varies considerably across tools for the same action, indicating that tool geometry and contact structure strongly affect transfer difficulty.

Findings. The benchmark highlights three main challenges in human-to-robot tool-use transfer: contact-rich interactions remain difficult, tool geometry can substantially affect performance even within the same action category, and embodiment mismatch limits direct transfer across human and robot morphologies. These results motivate further study of cross-tool generalization, embodiment-aware policy learning, and robust transfer from human demonstrations to physical robot interaction.

Conclusion

We introduced TOOLDEX, a large-scale ego-exo human demonstration dataset and benchmark for creative physical tool use. TOOLDEX explicitly captures the full hand–tool–target interaction in real-world manipulation scenarios, including challenging deformable target objects. With multi-view observations, the dataset provides rich annotations for studying embodied physical tool-use. Beyond the dataset itself, we establish a unified benchmark spanning physical tool-use reasoning, 3D and 4D reconstruction, and robot policy learning. Meanwhile, the dataset may raise concerns related to the misuse of human behavior data or unsafe downstream robotic deployment. To mitigate these, all data are collected with consent and released for research purposes under a non-commercial license. We hope TOOLDEX will serve as a foundation for future research on developing robots capable of adaptive tool use in open-world environments. Limitations are discussed in Appendix § 8.§ 6.

AI Use Statement

Large language models were used throughout this work under author direction: for literature collection, experimental statistics, code development, and manuscript drafting, with additional assistance in experimental design. All core ideas, the experimental program, and the conclusions originate from the authors. The authors verified the paper’s numerical claims against the underlying configurations, logs, evaluation records, and analysis artifacts, reviewed all AI-assisted text, code, and figures, and take full responsibility for the paper’s content.

Appendix

Novel-View Gaussian Splatting for Sparse-View Tooluse: Static and Dynamic

Experiment Setup. We benchmark sparse-view novel-view synthesis in two regimes that share a single calibrated capture: static 3DGS on a per-scene image, and dynamic 4DGS on a per-scene $T$ sequence. In both cases we use the same 8 calibrated RealSense cameras and the same held-out $\mathrm{cam}_6$ Protocol B – each scene is reconstructed twice: an 8-cam run that uses all eight cameras for training and a $\mathrm{cam}_6$-dropped run that uses $\mathrm{cam}_0,\ldots,5,7$ for training and reserves $\mathrm{cam}_6$ for held-out evaluation. From the $\mathrm{cam}_6$-dropped run we score the predicted $\mathrm{cam}_6$ image (or render across time at $\mathrm{cam}_6$’s pose, in 4D) against the captured ground truth with full-reference PSNR, SSIM [57], and LPIPS [66]. From the 8-cam run we score the rendered novel-view tour without a reference using the appeal-composite metrics MUSIQ [25], MANIQA [63], TOPIQ [8], BRISQUE [34], NIQE [35], and PIQE (N et al., 2015). Baselines use their native initialization: SparseGS reads the COLMAP [50] (Schönberger & Frahm 2016) sparse reconstruction, while Sparfels, SplatFormer (applied to our InstantSplat model), and InstantSplat are seeded from MASt3R [28]. We use InstantSplat [11] as our main backbone; in the dynamic regime we apply InstantSplat across the $T$ timesteps, which we refer to as InstantSplat-4D.

Method. We propose a two-stage sparse-view recipe specifically engineered for the failure modes of unposed tabletop tool-use reconstruction, taking the seven (or eight) training views of our calibrated capture as input – and, in the dynamic case, applied across $T$ timesteps. Stage 1 – reconstruction. We obtain a geometry-aware initialization from a MASt3R [28] cross-view dense point prior and train the joint pose-and-Gaussian optimizer of InstantSplat [11] for 1k Table 6: 3DGS novel-view synthesis with 8 sparse cameras, benchmarked on three tabletop tool-use scenes ($pestle\_crush\_nuts$, $ladle\_scoop\_nuts$, $knife\_slice\_banana$). Each entry is the mean over the three scenes. Methods: SplatFormer ([9]), Sparfels ([24]), SparseGS ([60]), InstantSplat ([11]).

iterations under a clean photometric loss; we deliberately omit every reconstruction-time regularizer (depth, distortion, normals) that we found to leave held-out PSNR flat or harm SSIM/LPIPS in our setting. For 4DGS we instantiate an independent Gaussian field $\mathcal{G}_t$ at each timestep $t$ using the same Stage-1 design, with the MASt3R prior recomputed per timestep. Stage 2 – rendered-view repair. Rather than expending optimizer budget on 3D losses that do not address the dominant sparse-cam failure mode, we move the heavy lifting to the rendered-view level where ghosting and missing high-frequency detail actually live. For 3DGS we apply a single-step diffusion utilizing Difix3D ([59]) repair tuned to the rendering distribution of tool-use scenes; for 4DGS we additionally chain GSFix3D ([64]) after Difix3D.

Findings (Static, 3DGS). The defining property of our dataset is a calibrated 8-camera capture per scene, which uniquely supports a held-out-cam$_6$ full-reference evaluation that competing tabletop tool-use datasets cannot reproduce – and Table 6 shows what this enables. Across three tool-use scenes our recipe reaches 20.45 PSNR / 0.779 SSIM / 0.364 LPIPS on cam$_6$, beating InstantSplat by +0.5 dB and Sparfels / SplatFormer / SparseGS by 5–8 dB, while also winning BRISQUE, TOPIQ, and PIQE on the same predicted image. Sparfels takes the top MUSIQ (61.16) because its surfel rasterization produces smooth, locally pleasing texture, but its low full-reference numbers show that clean appearance need not imply faithful geometry. The headline result is that high-quality sparse-view 3DGS for tool-use scenes emerges only when both sides are in place: our calibrated capture (dataset) and our two-stage pipeline (method). Without the dataset, neither the held-out gap nor its closure would be measurable; without the method, the gap remains.

Findings (Dynamic, 4DGS). The same dataset property extends naturally to the dynamic regime, supporting a held-out-cam$_6$ full-reference evaluation at every timestep that competing tabletop dynamic-tool-use datasets cannot reproduce. Table 7 shows across three tool-use scenes our final pipeline reaches 69.92 MUSIQ / 0.39 MANIQA / 0.54 TOPIQ on the 8-cam novel-view tour, and 4.35 NIQE on the same render – a +18.92 MUSIQ jump over the raw InstantSplat-4D backbone. The full-reference columns reveal that the photometric/perceptual dissociation we observed in the static case, now sharpened and localized to the second repair stage: the single-step Difix3D pass leaves held-out fidelity intact (19.60 → 19.63 PSNR, with the best SSIM and LPIPS of any stage), whereas chaining GSFix3D lowers PSNR by 3.6 dB and SSIM by 0.08 relative to the raw backbone (19.60 → 15.97 PSNR, 0.715 → 0.631 SSIM) while raising every no-reference appeal metric by 19–36%. This is the well-known photometric-vs-perceptual trade-off [59] now measured directly on a held-out 4D novel-view, Our method utilizing Difix3D and GSFix3D rewire noisy per-pixel detail into smoother, plausible-looking content that does not pixel-match the GT but scores higher on learned-perceptual judges. The drop6 protocol is what makes this trade-off measurable in 4D at all; without our calibrated capture, neither the held-out 4D gap nor the photometric-vs-perceptual dissociation would be quantifiable. The headline result mirrors the static case: high-quality sparse-view dynamic novel-view synthesis emerges only when both sides are in place: the 4D-extended calibrated capture (dataset) and the two-stage pipeline (method).

Expanded observed-view evaluation (Dynamic, 4DGS). Complementing the three-scene held-out protocol above, we additionally ran the full 4D pipeline on 20 dynamic tool-use scenes with all 8 cameras used for training. In this setting the full-reference metrics score each of the 8 observed camera views against its own captured ground-truth frame, and the no-reference metrics again score the rendered novel-view tour (Table 8). The raw InstantSplat-4D backbone reconstructs the observed views at 36.97 PSNR / 0.975 SSIM / 0.044 LPIPS, and the Difix3D → GSFix3D repair stack lifts every no-reference metric (MUSIQ 48.79 → 64.69, MANIQA 0.209 → 0.388, TOPIQ 0.354 → 0.528, with the best BRISQUE, NIQE, and PIQE) while lowering observed-view fidelity (36.97 → 21.11 PSNR) – the same photometric-vs-perceptual dissociation as above, now measured over 20 scenes.

VLM Benchmarking Details

Prompt Used We use the prompts in Table 9 for tool selection and spatial grounding, while we used the prompts Table 10 for behavioral trajectory generation.

System Prompt:
You are a robot tool-use reasoning assistant. Select which visible tools could realistically
↪ complete the task. Do not default to the most typical tool.

User Prompt:
Given the scene with multiple laid-out tools, identify which of them could be used to
↪ complete the task: "<TASK>".

Only select tools that are both visible and plausible for completing the task. Unconventional
↪ tools are allowed, but do not include tools that clearly would not work.

# only added for Spatial Grounding Tasks
Return one bounding box for each tool in the same order as the tools list. Use pixel
↪ coordinates [x1, y1, x2, y2] around the visible tool, with x2 and y2 as the
↪ bottom-right exclusive coordinates.

Return only the canonical base tool name.
Do NOT include adjectives, materials, colors, or sizes.

Return ONLY the requested JSON object using this schema:
{
  "tools": ["<exact_tool_name>", "..."],
  # only added for Spatial Grounding Tasks
  "bboxes": [[0, 0, 0, 0]]
}

System Prompt:
You are a robot trajectory planning assistant. Reply with JSON only.

User Prompt:
Plan a 2D image-space trajectory for this manipulation task:
"<TASK>"

Place point 0 on the <TOOL_NAME>. Then return <NUM_POINTS>
additional points that describe a plausible trajectory for
using the tool to complete the task.

Use normalized image coordinates in [y, x] format, where each
coordinate is an integer in the range 0–1000.

Return ONLY a JSON array in the following format:
[
  {"point": [y, x], "label": "0"},
  {"point": [y, x], "label": "1"}
]

Requirements:
- Labels must be strings from "0" through "<NUM_POINTS>".
- Include exactly one point for each label.
- Do not include code fences or explanatory text.

VLM-source We use the OpenAI API, Anthropic Claude API, and Google Gemini API to evaluate proprietary models. Open-weight Qwen models are served using the vLLM Kwon et al. [27] inference framework. For the evaluated Claude models, we enable extended thinking with a maximum budget of 4096 thinking tokens.

Additional VLM outputs

Qualitatively, the different VLM families exhibit distinct behaviors. Qwen generally predicts fewer bounding boxes and often defaults to the most common tool associated with a task. While it demonstrates some understanding of spatial relationships, its localization accuracy is weaker; for example, in (a) it correctly reasons that the fork is left of the skewer, but the predicted bounding box is slightly offset.

GPT models consistently produce the highest-quality spatial grounding results, with bounding boxes frequently overlapping closely with ground-truth annotations. Though occasionaly it doesn’t recognize or find smaller objects, such as in (e). Gemini models are notably creative and often generate plausible alternative tool choices, such as suggesting a scoop for stirring coffee. However, Gemini occasionally exhibits weaker spatial understanding, sometimes predicting long horizontal boxes for vertically oriented objects. Claude models are generally the most consistent, frequently producing the correct outputs with relatively few hallucinations, although occasional semantic mistakes still occur, such as identifying a rolling pin as tongs.

Gemini Robotics Evaluation and Result Details

Trajectory Representation For behavioral planning evaluation, models predict a sequence of normalized 2D image-space trajectory points conditioned on the task instruction and the initial scene image. The first predicted point is constrained to lie on the selected tool, while the remaining points describe the predicted manipulation trajectory.

To simplify evaluation across models with different output styles, trajectories are represented as sparse waypoint sequences rather than dense per-frame motion predictions. Predicted trajectories are converted back to pixel coordinates using the image resolution before evaluation. We prompted the model to generate 32 points, enough to create a dense table-top action task trajectory.

Ground-Truth Trajectory Construction Ground-truth trajectories are derived from tracked tool pose sequences recovered from the object pose solver. For each frame, the tool translation component is projected into image space using calibrated camera intrinsics, producing a dense 2D tool trajectory. The projected trajectory is then arc-length resampled to match the number of predicted points generated by the model.

The first ground truth trajectory is generated by taking the center point of the tool mesh and applying transforms to that throughout. We additionally evaluate a correspondence-aware trajectory variant that anchors evaluation to the predicted interaction point. Specifically, we identify the tool mesh vertex whose first-frame projection is nearest to the predicted starting point, and transport this vertex through the full pose sequence before projection into image space. If the predicted initial point does not lie directly on the tool surface, we associate it with the nearest projected mesh vertex in the first frame before transporting that vertex through the pose sequence. This produces a more interaction-centric trajectory target with more fair comparison than using only the tool-center translation, as we observed during preliminary experiments that Gemini Robotics ER does not always generate a consistent starting point, sometimes initializing trajectories near the tool center, grasp region, or intended contact point depending on the task and tool geometry.

Qualitative Results It can be seen in Figure 11 that tasks with relatively well-defined motions—typically those involving short actions, clear end states, or trajectories that start directly on the target object—tend to achieve stronger results. For example, the banana slicing trajectory(b) closely matches the ground-truth cutting motion, despite only requiring the model to infer the cup position and cutting direction from the scene context.

However, for more complex tasks, such as shaping kinetic sand into a circular structure (e), the model often oversimplifies the interaction dynamics and fails to fully capture the physical difficulty of the task. In these scenarios, successful manipulation requires more than simply tracing a circular trajectory; the tool must repeatedly reposition, push material forward, and gradually accumulate shape over multiple passes. The generated trajectories instead tend to approximate only the high-level geometric pattern of the motion.

Nevertheless, these results still demonstrate the strong capability of the model to capture the overall structure and intent of tool-use trajectories across a wide range of diverse manipulation settings.

Policy Learning Pipeline Details

This section provides additional implementation details for the policy learning baselines described in the main text, including retargeting, training, and evaluation.

Retargeting

The evaluated policies use retargeted robot trajectories derived from the human hand and tool motion annotations. Different retargeting procedures are used according to the robot embodiment and policy formulation.

ManipTrans + Inspire. For ManipTrans Li et al. [29], we follow its hand retargeting procedure and optimize Inspire hand joint angles to match the MANO reference, with emphasis on fingertip alignment. Since this setting controls only the dexterous hand, the resulting joint-space trajectory is used as the reference for residual policy learning.

Dex-X. For Dex-X Chen et al. [68], the human demonstration is retargeted to the Franka FR3 + Sharpa hand embodiment. The retargeting jointly maps wrist motion and hand articulation into the robot arm–hand configuration space while respecting robot joint limits and reachable configurations. We follow the retargeting and simulation pipeline described in Dex-X and use the resulting trajectories as references for interaction-aware reinforcement learning.

Evaluation Protocol

We evaluate six task signatures spanning four action categories: stir, scoop, crush, and collect. Demonstrations contain 254–600 frames ($\approx4$–10 s at 60 Hz). Each method is evaluated over 1,000 episodes per task with randomized initial states and reference start frames.

Following the reference-tracking evaluation protocol, an episode is considered successful if the policy reaches the end of the demonstrated interaction without violating the task-specific tracking and stability constraints. The same success definition is used consistently across methods whenever applicable.

Training Configuration

For ManipTrans(RL), we use the training configuration of ManipTrans Li et al. [29]. A residual policy is trained with PPO in IsaacGym for each task using the same hyperparameter configuration across the evaluated task set.

For Dex-X (RL), we follow the released training pipeline of Dex-X Chen et al. [68]. Policies are trained with reinforcement learning in IsaacLab using the retargeted human demonstrations as interaction references. We use the same training pipeline across all evaluated tasks without task-specific policy design.

Limitations and future directions.

Despite its scale and diversity, TOOLDEX has several limitations. First, the dataset focuses primarily on tabletop setting in indoor environments, and therefore does not cover the full range of human tool-use behaviors found in large-scale outdoor or mobile settings. Second, while we include rigid, granular, and deformable target objects, the physical coverage of materials and interaction dynamics remains incomplete. Third, although the ego-exo capture setup enables high-fidelity reconstruction and tracking, some annotations may still contain noise due to occlusion or challenging contact configurations. In addition, the VLM benchmarking pipeline is intentionally permissive to account for multiple plausible tool choices. While this better reflects real-world tool use, future work could incorporate human-validated sets of acceptable tools.