WorksComputer Science
AuRA: From Atoms to Bits through Autonomous Robot Interaction
Abstract
Simulation enables robot-learning experiments at scale, but acquiring physics-informed digital twins often requires human manipulation and manual annotation of recorded interactions. To reduce this reliance on human effort, we present AuRA, a system that automates robot interaction and data preparation for digital-twin reconstruction. AuRA combines execution-aware grasp selection, gripper-motion measurements derived from robot states, and registered pre-interaction geometry and appearance priors. These choices enable autonomous collection, avoid reliance on visual gripper tracking, and recover object information obscured by the robot. Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Relative to the human-collected PhysTwin baseline, AuRA improves seven of nine metrics for future prediction under known gripper motion, including a 28.1% reduction in one-way Chamfer distance. It also achieves similar mean silhouette IoU and Chamfer distance under the evaluated unseen interactions. These results highlight the potential of autonomous robot interaction to support more efficient digital-twin acquisition with comparable reconstruction quality.
Introduction
Simulation enables repeated robot-learning experiments in digital environments, supporting scalable data generation and policy training [11, 12, 13]. To reflect interactions with particular real-world objects, these environments need digital assets that represent the objects’ geometry, appearance, and physical behavior. Real-to-sim reconstruction offers a way to build such assets from observations of real objects, with manipulation providing information about their dynamics. The resulting assets can be reused across simulated experiments, supporting a shift from an “atoms loop” of physical manipulation to a “bits loop” of simulation.
A key bottleneck, however, lies in acquiring the real-world observations required for reconstruction. Existing workflows often rely on human manipulation or manual annotation of recorded data [14, 17]. This per-object human involvement limits the rate at which asset collections can be expanded. Automating interaction-data acquisition offers a way to reduce this effort: a robot can select and execute interactions while recording the observations needed for reconstruction. In practice, a reliable autonomous pipeline must address three coupled challenges.
First, selected interactions must satisfy robot and workspace constraints. A visually plausible grasp may violate joint limits or bring the gripper too close to the table or cameras, creating collision risks. Second, reconstruction requires reliable gripper-motion inputs. Segmentation, tracking, and depth errors can corrupt visually estimated gripper trajectories, potentially biasing the physical parameters inferred from the object’s response. Third, robot-induced occlusion can leave object geometry and appearance incomplete. The gripper and arm may obscure portions of the object throughout the trajectory. Missing surface observations can leave gaps in both the physical model and its rendered appearance.
To address these challenges, we introduce AuRA (Autonomous Real-to-sim Asset Reconstruction), a system for autonomously collecting object-interaction data and reconstructing simulation-ready assets. First, AuRA autonomously identifies and executes feasible object interactions. After object placement and system initialization, AuRA generates grasp candidates from pre-interaction RGB-D observations and checks their feasibility against robot and workspace constraints. The robot then executes the selected interaction while the system records RGB-D videos and robot states.
Second, AuRA separates measurement of the robot’s action from observation of the object’s response. Recorded robot states and calibrated gripper geometry are used to derive controller trajectories, while visual tracking captures the object’s motion and deformation. This separation provides reliable motion inputs without requiring visual gripper segmentation, tracking, or depth lifting.
Third, AuRA reuses pre-interaction observations to mitigate robot-induced occlusion. It constructs a static point cloud and a Gaussian appearance model, then aligns both with the first interaction frame to account for small pose changes caused by grasping. These registered priors supplement geometry and appearance missing from the interaction observations. The same pre-interaction captures therefore support both physical acquisition and digital reconstruction, without requiring a prebuilt object asset or a separate manual capture stage.
We evaluate data collection efficiency, reconstruction reliability, and asset quality across 13 deformable objects. Compared with human-operated collection overall, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Across all acquisition attempts in the timed collection study, AuRA achieves an end-to-end reconstruction success rate of 74.3%, compared with 26.9% for human-operated collection overall. Moreover, AuRA improves seven of nine metrics compared to the human-collected PhysTwin baseline for future prediction, reducing one-way Chamfer distance by 28.1% and increasing depth coverage by 9.9 percentage points. The two methods also achieve similar mean silhouette IoU and Chamfer distance on the evaluated unseen interactions. In summary, our main contributions are:
Autonomous data acquisition for digital-twin reconstruction. We introduce AuRA, a system that integrates execution-aware grasp selection, autonomous object interaction, and synchronized recording of RGB-D observations and robot states. After setup and initialization, collection proceeds without manual manipulation or annotation.
Reliable controller trajectories from robot states. We derive controller-point trajectories from recorded robot joint states and calibrated gripper geometry. Together with visually tracked object motion, these trajectories support dynamics reconstruction without relying on visual gripper segmentation, tracking, or depth lifting.
Reusing pre-interaction observations for asset reconstruction. We align pre-interaction geometry priors with the recorded interaction to fill gaps caused by robot occlusion, providing a more complete geometric initialization for dynamics optimization. Together with 3D Gaussians trained from the same pre-interaction captures, AuRA maintains reconstruction quality while significantly reducing human labor through autonomous data collection.
Related Work
Reconstruction and Simulation of Deformable Objects
Geometric and visual reconstruction methods recover object shape and appearance using representations such as point clouds, neural radiance fields, and 3D Gaussians [15, 23, 16]. Digital twins for deformable-object interaction additionally require dynamics that predict how shape changes under applied motion or forces.
One line of work uses neural networks to learn predictive models of object dynamics from data. Graph-based simulators represent objects as particles or nodes and learn to predict their motion [19, 20, 21]. Recent systems further couple learned dynamics with Gaussian appearance representations to support both motion prediction and rendering [4, 3].
An alternative fits explicit physical models to observed behavior, using differentiable mesh simulation [18], spring–mass systems [17, 22, 7], or the material point method [2, 5]. Within this family, PhysTwin [17] reconstructs geometry, appearance, and dynamics from sparse-view RGB-D videos; PhysHanDI [7] emphasizes coupled hand–object reconstruction; and EMPM [5] combines offline identification with online material adaptation. Their demonstrated acquisition pipelines rely on human manipulation or teleoperation. AuRA builds on this reconstruction paradigm and addresses the complementary challenge of autonomously acquiring the interaction data needed for reconstruction.
Robotic Interaction for Object Modeling
Beyond reconstruction from recorded observations, another line of work uses robotic interaction to actively acquire information for object modeling. Autonomous soft-object manipulation [25], viewpoint and object-reorientation selection [26], and active visuotactile cable exploration [27] improve observations of object shape or appearance. These methods focus on geometry and/or appearance reconstruction, without jointly modeling deformation dynamics.
Other systems use interaction to identify physical properties or adapt predictive models. DPSI [28] estimates material and environment parameters through differentiable simulation, while AdaptiGraph adapts learned dynamics through active interaction [29]; neither jointly reconstructs a renderable appearance model. Sim2Real [30] and SAGCI-System [31] construct or refine articulated-object models for planning and control, using rigid-link representations rather than continuous deformation within links. AuRA brings autonomous data acquisition to digital-twin reconstruction encompassing geometry, appearance, and deformation dynamics.
AuRA: Autonomous Data Preparation and Reconstruction
Figure 1. AuRA’s autonomous physical-to-digital acquisition pipeline. Multi-view RGB-D capture supports grasp generation and execution-feasibility filtering. The robot executes the selected grasp and records the interaction with synchronized robot states. During processing, visual tracking provides object trajectories, while robot poses and calibrated gripper geometry provide controller trajectories. AuRA also constructs a static point cloud and trains a Gaussian appearance model from its pre-interaction captures, then aligns both with the first interaction frame to supplement robot-occluded regions. These inputs support spring–mass optimization and Gaussian rendering through the PhysTwin reconstruction backend.
To address the challenges of autonomous acquisition, we propose AuRA, a pipeline with three stages: (1) autonomous data acquisition, (2) data post-processing, and (3) physics-informed asset reconstruction, as shown in Figure 1. The first stage uses pre-interaction RGB-D observations to generate and filter grasp candidates, then executes a selected grasp-and-lift interaction while recording synchronized RGB-D videos and robot joint states. The second stage extracts object and controller trajectories and registers the pre-interaction static point cloud to supplement missing object geometry, preparing the inputs for physical reconstruction. The third stage fits a spring–mass model using these inputs and trains a Gaussian appearance model using the pre-interaction observations. The Gaussians are then aligned and bound to the physical model, yielding a digital twin that represents geometry, appearance, and dynamics applicable to both rigid and deformable objects.
Data Acquisition
After object placement and system initialization, AuRA acquires interaction data without manual object manipulation or recording-boundary annotation. Before manipulation, calibrated RGB-D cameras capture the stationary object from multiple viewpoints. AuRA segments the object in each view using a text prompt derived from a user-provided case name, then back-projects valid depth measurements within the masks and fuses them into a point cloud in the common world frame. The cloud is filtered to suppress spurious points caused by depth noise and RGB–depth misalignment, using the parameters in Table 1.
| Parameter | Default value |
| Valid depth measurement range | m |
| Reference table height | m |
| Below-table tolerance | m |
| Downsampling voxel size | m |
| Neighbors for statistical filtering | |
| Statistical rejection threshold | standard deviations |
| Local neighborhood radius | m |
| Required neighbors within radius | |
| Principal-axis cropping percentiles | st–th |
| Cropping margin along each axis | m |
Table 1. Point-cloud preprocessing parameters for grasp pose generation.
We then use AnyGrasp [24] to generate candidate grasp poses and gripper opening widths from the filtered point cloud. AuRA checks tabletop clearance using the end-effector center and calibrated fingertip positions. It also uses PyRoki [1] to solve inverse kinematics and check joint-limit margins at sampled poses along the planned motion path. The robot then performs the highest-confidence candidate that passes these checks while AuRA records synchronized RGB-D videos and robot joint states. Recording stops automatically when the lift is complete.
Figure 2. Data collection efficiency and reconstruction reliability. Human (Overall) pools the three groups below the dotted line. (a) End-to-end reconstruction rate: AuRA fails during acquisition, whereas human recordings fail during data processing. (b) Wall-clock time () and human labor () per reconstructed asset (log scale): the two are equal for human collection. (c) Mean failure detection time: AuRA detects failures after 88.3 s on average, compared with 133.0 s for human collection overall and 89.2 s for experts. Collection costs in (b) exclude data processing, reconstruction training, and untimed backup trials.
Data Post-processing
After data acquisition, AuRA prepares object trajectories, controller trajectories, and a supplemented object point cloud for reconstruction. It segments the object in the interaction videos using Grounded-SAM2 [10] and initializes tracking points from the filtered first-frame point cloud. MV-TAP [6] jointly tracks points initially visible in multiple cameras, while CoTracker3 [8] handles those initially visible in only one camera. The resulting 2D tracks are lifted into 3D using the corresponding depth maps and camera calibration, then filtered for visibility, valid depth, and local motion consistency.
Controller trajectories are derived deterministically from recorded robot joint states. The resulting end-effector poses, calibrated fingertip geometry, and robot-to-world transformation determine the paths of gripper-attached controller points that drive the simulated object. This avoids visual gripper segmentation, tracking, and depth lifting, providing controller-motion inputs even when the gripper is occluded.
To supplement geometry obscured by the robot, AuRA registers the pre-interaction static point cloud to the visible object geometry in the first interaction frame. Assuming that grasping causes only small shape changes before recording begins, registration uses a rigid transformation estimated through PCA initialization and ICP refinement. An alignment is accepted only if it passes geometric checks and sufficiently reduces the trimmed symmetric Chamfer distance; otherwise, the identity transformation is retained. The registered cloud adds surface points only in voxels unoccupied by tracked object points, preserving the tracked observations while filling missing regions. This supplemented geometry provides a more complete geometric initialization for dynamics optimization.
Asset Reconstruction
Using the supplemented object geometry and the object and controller trajectories, AuRA fits a physical model through PhysTwin’s spring–mass optimization [17]. Gripper-attached controller points drive the simulated object through springs connecting controller and object nodes, while the tracked object trajectories supervise the optimization.
For appearance, AuRA trains a static 3D Gaussian model from the pre-interaction RGB-D observations, initialized with the static point cloud. These observations provide appearance information for regions later obscured by the robot. The rigid transformation obtained during point-cloud registration is applied to the Gaussian positions and orientations to align the appearance model with the interaction sequence.
The aligned Gaussians are then bound to the fitted physical model through weighted blending of nearby mass-node motion. This binding transfers simulated deformation to the Gaussians for rendering, yielding a digital twin that represents object geometry, appearance, and dynamics.
Experiment
Our setup consists of a gripper-equipped Franka robot arm and three OAK-D RGB-D cameras positioned around a tabletop workspace. The cameras and robot are calibrated to a common world frame.
We evaluate data collection efficiency, reliability, and reconstruction quality on 13 deformable objects. Each trial begins with the object placed within the workspace and consists of a single grasp-and-lift interaction for either human-operated or robot-driven collection. We use the human-collected PhysTwin pipeline [17] as our baseline because it reconstructs geometry, appearance, and dynamics from limited interaction data. The baseline uses CoTracker3 [8] for object tracking. The default AuRA pipeline uses MV-TAP [6] for object tracking with a CoTracker3 fallback for points initially visible in only one camera, the registered pre-interaction static point cloud as a geometry prior, and static 3D Gaussians for appearance. Data collection and PhysTwin baseline training run on a desktop equipped with an NVIDIA RTX 5090 GPU, while AuRA reconstruction runs on a server equipped with an NVIDIA RTX A5000 GPU.
| Metric | Resimulation | Future Prediction | ||
| PhysTwin | AuRA | PhysTwin | AuRA | |
| PSNR (dB) | 12.289 | 11.549 | 10.325 | 10.590 |
| SSIM | 0.9856 | 0.9852 | 0.9805 | 0.9797 |
| LPIPS | 0.0126 | 0.0148 | 0.0233 | 0.0230 |
| IoU | 0.6675 | 0.6090 | 0.4607 | 0.5001 |
| Precision | 0.7521 | 0.7021 | 0.5899 | 0.6132 |
| Recall | 0.7972 | 0.7791 | 0.6260 | 0.7085 |
| Depth MAE (mm) | 21.797 | 24.093 | 27.784 | 34.923 |
| Depth Coverage | 0.8152 | 0.8050 | 0.6392 | 0.7386 |
| One-WayChamfer (mm) | 11.225 | 10.104 | 23.335 | 16.783 |
Table 2. Resimulation and future-prediction quality. Trajectory-level metrics are averaged within each object and then equally across 13 objects. PhysTwin uses 29 human trajectories, with shape priors when available; AuRA uses 26 robot trajectories processed by its standard pipeline. PhysTwin (CoTracker3) and AuRA (MV-TAP) retain an average of 1,493.76 and 615.96 tracking points per trajectory, respectively. All available views with usable object masks are used in evaluation.
| Metric | PhysTwin | AuRA | |
| Static GS | First-Frame GS | ||
| First-Frame Alignment | |||
| Mask IoU | 0.5096 | 0.4939 | 0.4095 |
| Mask Precision | 0.6228 | 0.5817 | 0.5285 |
| Mask Recall | 0.7146 | 0.7650 | 0.6752 |
| Unseen Interaction | |||
| PSNR (dB) | 10.309 | 10.379 | 9.622 |
| SSIM | 0.9842 | 0.9815 | 0.9798 |
| LPIPS | 0.0233 | 0.0256 | 0.0288 |
| Mask IoU | 0.3849 | 0.3861 | 0.3294 |
| Mask Precision | 0.4673 | 0.4444 | 0.4020 |
| Mask Recall | 0.5991 | 0.6522 | 0.5821 |
| Depth MAE (mm) | 18.936 | 19.522 | 18.478 |
| Depth Coverage | 0.5991 | 0.6522 | 0.5821 |
| One-WayChamfer (mm) | 43.430 | 43.554 | 43.554 |
Table 3. First-frame alignment and unseen-interaction quality. The upper block reports silhouette metrics at the aligned first frame; the lower block evaluates the subsequent interaction. Bold indicates the best value in each row.
Data Collection Efficiency and Reliability
We compare the collection cost and reliability of AuRA with human-operated acquisition across three experience groups, namely amateur, trained, and expert. Each human group contributes two trajectories per object. Eight additional untimed trials supplement the reconstruction-quality evaluation to ensure at least two successfully reconstructed assets per object; they are excluded from the efficiency comparison.
As shown in Figure 2, AuRA achieves both a shorter mean failure-detection time and a higher end-to-end yield than all human groups. AuRA detects failed runs after an average of 88.3s and achieves a yield of 74.3%, compared with 126.4s and 7.7% for the amateur group, 158.1s and 15.4% for the trained group, and 89.2s and 57.7% for the expert group. Human collection overall requires 133.0s to detect failures and achieves a yield of 26.9%. Specifically, AuRA produces 26 reconstructed assets from 35 acquisition attempts, whereas human-operated collection produces 21 from 78 attempts overall. We attribute AuRA’s higher reliability to two design choices: gripper-pose feasibility filtering rejects infeasible interaction candidates before execution, while controller trajectories derived deterministically from robot states avoid downstream reconstruction failures caused by visual gripper tracking, as further examined in the ablation studies.
For efficiency, we measure wall-clock collection time and active human labor per successfully reconstructed asset. Costs are computed over all timed acquisition attempts, including failed attempts, while excluding downstream data processing and reconstruction, including PhysTwin’s failure-detection processing. Again, as demonstrated in Figure 2, AuRA requires 126.7s of wall-clock collection time and 32.7s of active human labor per asset, corresponding to reductions of 55.4% and 88.5%, respectively, relative to human collection overall, which requires 284.4s for both measures. AuRA also incurs substantially less human labor than each individual human group, compared with 1,108.5s for amateurs, 561.8s for trained users, and 100.6s for experts. Its wall-clock collection time is lower than that of the amateur and trained groups, though higher than the 100.6s required by experts. Because AuRA requires human involvement only for object placement and execution initialization, it eliminates the manual effort associated with performing interactions and annotating recording boundaries. Combined with its high reconstruction success rate, this reduction in manual effort enables more efficient acquisition of reconstructed assets.
| Data collection | ||||
| frame 0 (GT) & Rendering aligned to | ||||
| unseen interaction frame 0 & Rendering overlay & 1cUnseen interaction | ||||
| frame 0 (GT) | ||||
| AuRA | ||||
Static GS & & & &![]() | ||||
| AuRA | ||||
First-frame GS & & & &![]() | ||||
| PhysTwin | ![]() | ![]() | ![]() | ![]() |
Figure 3. Object appearance at the first frame of an unseen interaction. From left to right: the original data-collection frame 0, the rendered object at unseen interaction frame 0 after manual asset alignment, its overlay with the unseen-interaction ground truth (GT), and that GT image.
Reconstruction Quality
Resimulation and Future Prediction
Following PhysTwin [17], we use the first 70% of the frames in each interaction sequence for training and the remaining 30% for testing. The learned spring–mass model is driven by recorded controller trajectories, and its motion animates the associated 3D Gaussians for rendering. Resimulation measures agreement with observations over the training frames, while future prediction evaluates the held-out frames under known controller motion. The mean sequence lengths are 34.62 frames for the human baseline and 33.31 frames for AuRA.
As shown in Table 2, AuRA outperforms PhysTwin on seven of nine future-prediction metrics, whereas PhysTwin performs better on eight of nine resimulation metrics. In future prediction, AuRA shows particularly clear gains in silhouette and geometry, improving mask IoU from 0.4607 to 0.5001, recall from 0.6260 to 0.7085, and depth coverage from 0.6392 to 0.7386, while reducing the one-way Chamfer distance from 23.335 mm to 16.783 mm. AuRA also achieves slightly better PSNR and LPIPS, although PhysTwin retains higher SSIM and lower depth MAE. In contrast, PhysTwin performs consistently better during resimulation, with AuRA improving only the one-way Chamfer distance. One possible explanation is that PhysTwin retains substantially more tracking points, providing denser supervision for fitting the observed motion. We further examine this effect through tracker and initialization ablations in Table 4 and discuss the retained tracking-point counts in ablation studies.
Unseen Interactions
For each of the 13 objects, we select one asset from each method and collect one new interaction per asset, resulting in 26 unseen-interaction trajectories. Assets are selected based on their original future-prediction performance; for AuRA, we prioritize cases with both static and first-frame Gaussian models, while shape-prior-enabled PhysTwin assets are used when available. Before each interaction, the physical object is manually aligned with the selected asset, and the Franka robot performs a teleoperated drag-and-lift interaction. Recorded robot states drive the reconstructed dynamics, and the resulting sequence is evaluated against the recorded observations using the same metrics as above.
As shown in Table 3, AuRA with static Gaussians and PhysTwin achieve similar mean performance on the unseen interactions. AuRA attains a mask IoU of 0.3861, compared with 0.3849 for PhysTwin, while their one-way Chamfer distances are 43.554 mm and 43.43 mm, respectively. Appearance quality is also similar: AuRA achieves 10.379 dB PSNR, 0.9815 SSIM, and 0.0256 LPIPS, compared with 10.309 dB, 0.9842, and 0.0233 for PhysTwin.
The comparison between AuRA’s static and first-frame Gaussian models further highlights the benefit of pre-interaction priors. Static Gaussians improve all reported appearance and silhouette metrics over first-frame Gaussians, preserving visual information from regions that become occluded by the robot during interaction, as illustrated in Figure 3. The registered static point cloud provides a complementary geometry prior by supplying surface observations missing from the interaction sequence. Together, these priors reduce the impact of robot-induced occlusion on appearance and dynamics reconstruction.
Ablation Studies
| Metric | CoTracker3-m | CoTracker3-p | MVTracker | MV-TAP |
| Resimulation | ||||
| PSNR (dB) | 11.341 / 11.629 | 11.245 / 11.520 | 11.445 / 11.543 | 11.468 / 11.549 |
| SSIM | 0.9843 / 0.9852 | 0.9843 / 0.9851 | 0.9847 / 0.9851 | 0.9848 / 0.9852 |
| LPIPS | 0.0173 / 0.0148 | 0.0172 / 0.0149 | 0.0162 / 0.0149 | 0.0159 / 0.0148 |
| Mask IoU | 0.5651 / 0.6131 | 0.5564 / 0.6093 | 0.5826 / 0.6067 | 0.5875 / 0.6090 |
| Mask Precision | 0.6577 / 0.7040 | 0.6539 / 0.7015 | 0.6726 / 0.6966 | 0.6825 / 0.7021 |
| Mask Recall | 0.7397 / 0.7810 | 0.7263 / 0.7771 | 0.7536 / 0.7753 | 0.7578 / 0.7791 |
| Depth MAE (mm) | 24.197 / 24.181 | 23.909 / 23.740 | 24.227 / 24.366 | 24.091 / 24.093 |
| Depth Coverage | 0.7714 / 0.8060 | 0.7574 / 0.8030 | 0.7811 / 0.7997 | 0.7864 / 0.8050 |
| One-WayChamfer (mm) | 14.894 / 9.599 | 16.794 / 10.150 | 12.947 / 8.317 | 14.793 / 10.104 |
| Future Prediction | ||||
| PSNR (dB) | 10.049 / 10.604 | 10.027 / 10.551 | 10.218 / 10.463 | 10.308 / 10.590 |
| SSIM | 0.9778 / 0.9794 | 0.9780 / 0.9796 | 0.9784 / 0.9793 | 0.9788 / 0.9797 |
| LPIPS | 0.0282 / 0.0235 | 0.0279 / 0.0230 | 0.0260 / 0.0237 | 0.0252 / 0.0230 |
| Mask IoU | 0.4099 / 0.4887 | 0.3993 / 0.4975 | 0.4461 / 0.4794 | 0.4502 / 0.5001 |
| Mask Precision | 0.5366 / 0.6044 | 0.5280 / 0.6115 | 0.5624 / 0.5948 | 0.5690 / 0.6132 |
| Mask Recall | 0.6185 / 0.6977 | 0.5978 / 0.7028 | 0.6562 / 0.6914 | 0.6569 / 0.7085 |
| Depth MAE (mm) | 30.920 / 33.765 | 31.757 / 32.972 | 32.988 / 34.494 | 33.371 / 34.923 |
| Depth Coverage | 0.6651 / 0.7331 | 0.6330 / 0.7360 | 0.6935 / 0.7205 | 0.6852 / 0.7386 |
| One-WayChamfer (mm) | 25.457 / 17.429 | 29.662 / 17.354 | 25.383 / 18.758 | 25.270 / 16.783 |
Table 4. Object-tracker and pre-interaction static geometry-prior ablations. All configurations use the same 26 robot-collected trajectories and static Gaussians. Each cell reports results without / with the geometry prior. CoTracker3-m refers to mask-based initialization and CoTracker3-p refers to point-cloud-based initialization, respectively. Bold indicates the best score across all methods within each prior setting, and underlining marks the better CoTracker3 initialization. Both use unrounded values.
Controller trajectory source
We compare controller-point trajectories deterministically derived from robot joint states with those obtained through MV-TAP [6] visual tracking, depth lifting, and PhysTwin-style controller filtering. On the same 26 robot-collected trajectories, robot-state-derived controllers enable successful reconstruction in all 26 cases, compared with only two cases for visually tracked controllers, corresponding to success rates of 100.0% and 7.7%, respectively. This comparison demonstrates the reliability benefit of robot-state-derived controllers under the tested conditions.
Static geometry prior and object tracking methods
We ablate the object tracking method and the pre-interaction geometry prior by evaluating CoTracker3 [8], MVTracker [9], and MV-TAP [6] with and without the prior, as shown in Table 4. CoTracker3 uses either mask-based or point-cloud-based initialization. The mask variant samples 5,000 points per view and follows PhysTwin’s track-filtering procedure [17]; the point-cloud variant, MVTracker, and MV-TAP each sample 1,024 points from the filtered first-frame object point cloud. CoTracker3 tracks projected queries independently per camera, while MV-TAP exploits multi-view visibility. MVTracker directly tracks 3D queries without depth lifting.
The geometry prior improves most metrics across all tracking configurations. Among methods using point-cloud initialization, MV-TAP generally performs best in both resimulation and future prediction. With the geometry prior, CoTracker3-m leads on most resimulation metrics, whereas MV-TAP leads on most future-prediction metrics.
On the robot-collected data, MV-TAP and MVTracker retain similar average numbers of tracking points per trajectory, at 615.96 and 625.46 respectively, while CoTracker3-m and CoTracker3-p retain 2,853.46 and 1,313.92. With the geometry prior, CoTracker3-m generally achieves better resimulation than CoTracker3-p, but worse future prediction. This trade-off mirrors the PhysTwin–AuRA comparison in Table 2: PhysTwin retains an average of 1,493.76 points per trajectory, compared with AuRA’s 615.96, and performs better on most resimulation metrics but worse on most future-prediction metrics. We hypothesize that denser tracking supervision helps fit observed motion but does not necessarily improve future prediction.
Static versus first-frame Gaussians
We compare static and first-frame Gaussian models on the same 13 unseen interactions using AuRA assets, keeping the simulated motion fixed. As shown in Table 3, static Gaussians improve mask IoU from 0.3294 to 0.3861 and PSNR from 9.622 to 10.379 dB, slightly exceeding PhysTwin’s 0.3849 and 10.309 dB, respectively. Mask recall and depth coverage both increase from 0.5821 to 0.6522, although depth MAE rises from 18.478 to 19.522 mm. These results show improved appearance and surface coverage with pre-interaction captures, accompanied by a modest increase in depth error.
Conclusion
In this work, we present AuRA, an autonomous pipeline for interaction-data acquisition and physics-informed digital-twin reconstruction. AuRA combines execution-aware grasp selection, robot-state-derived controller trajectories, and pre-interaction geometry and appearance priors to reduce manual effort, avoid visual gripper tracking, and mitigate robot-induced occlusion. Across 13 deformable objects, AuRA reduces both human labor and collection time while achieving a higher end-to-end reconstruction yield than human-operated acquisition. Furthermore, compared with the human-collected PhysTwin baseline, AuRA achieves comparable reconstruction quality. Together, these results show that autonomous robot interaction can make real-to-sim asset acquisition more efficient and reliable.
ACKNOWLEDGMENT
Jung, Chen, Lee and Huang are supported by DARPA HR001124S0029-AIQ-FP-019, National Science Foundation TRAILS Institute (2229885). Private support was provided by Open Philanthropy and Apple. The authors acknowledge the National Artificial Intelligence Research Resource (NAIRR) Pilot for contributing to this research result. We thank Wenqi Marcus Wu for helping validate the human-operated data collection pipeline and Shengming Qi for annotating human involvement using videos recorded during data collection with a separate external camera.
References
- [1]Kim, C. M., Yi, B., Choi, H., Ma, Y., Goldberg, K., & Kanazawa, A. (2025). PyRoki: A Modular Toolkit for Robot Kinematic Optimization. IROS.
- [2]Jiang, C., Schroeder, C., Teran, J., Stomakhin, A., & Selle, A. (2016). The Material Point Method for Simulating Continuum Materials. ACM SIGGRAPH 2016 Courses.
- [3]Huang, M., Wang, H., Ren, K., Xu, L., Zhou, Y., Yu, M., Dai, B., & Pang, J. (2026). SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-Body Manipulation. ICML.
- [4]Zhang, K., Li, B., Hauser, K., & Li, Y. (2025). Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos. RSS.
- [5]Chen, Y., Hu, Y., Sun, L., Kusnur, T., Herlant, L., & Jiang, C. (2026). EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects. IEEE Robotics and Automation Letters, 11(4), 4179–4186.
- [6]Koo, J., Kim, I. H., Kim, M., Park, J., Park, S., Kim, J., Yi, J., Cho, S., & Kim, S. (2026). MV-TAP: Tracking Any Point in Multi-View Videos. CVPR.
- [7]Lee, J., Lee, C., Kim, D., & Kim, T.-K. (2026). PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions. ICML.
- [8]Karaev, N., Makarov, I., Wang, J., Neverova, N., Vedaldi, A., & Rupprecht, C. (2025). CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. ICCV.
- [9]Rajič, F., Xu, H., Mihajlovic, M., Li, S., Demir, I., Gündoğdu, E., Ke, L., Prokudin, S., Pollefeys, M., & Tang, S. (2025). Multi-View 3D Point Tracking. ICCV.
- [10]Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., & Zhang, L. (2024). Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. https://arxiv.org/abs/2401.14159
- [11]Gong, R., Huang, J., Zhao, Y., Geng, H., Gao, X., Wu, Q., Ai, W., Zhou, Z., Terzopoulos, D., Zhu, S.-C., et al. (2023). ARNOLD: A Benchmark for Language-Grounded Task Learning with Continuous States in Realistic 3D Scenes. ICCV.
- [12]Margolis, G., Yang, G., Paigwar, K., Chen, T., & Agrawal, P. (2022). Rapid Locomotion via Reinforcement Learning. RSS.
- [13]Acar, C., Binici, K., Tekirdağ, A., & Wu, Y. (2024). Visual-Policy Learning Through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Tasks. IEEE Robotics and Automation Letters, 9(1), 691–698.
- [14]Torne, M., Simeonov, A., Li, Z., Chan, A., Chen, T., Gupta, A., & Agrawal, P. (2024). Reconciling Reality Through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation. RSS.
- [15]Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., & Novotny, D. (2025). VGGT: Visual Geometry Grounded Transformer. CVPR.
- [16]Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4), 139:1–139:14.
- [17]Jiang, H., Hsu, H.-Y., Zhang, K., Yu, H.-N., Wang, S., & Li, Y. (2025). PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos. ICCV.
- [18]Qiao, Y.-L., Liang, J., Koltun, V., & Lin, M. C. (2021). Differentiable Simulation of Soft Multi-body Systems. NeurIPS.
- [19]Shi, H., Xu, H., Huang, Z., Li, Y., & Wu, J. (2022). RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks. RSS.
- [20]Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., & Battaglia, P. W. (2020). Learning to Simulate Complex Physics with Graph Networks. ICML.
- [21]Lin, X., Wang, Y., Huang, Z., & Held, D. (2021). Learning Visible Connectivity Dynamics for Cloth Smoothing. CoRL.
- [22]Zhong, L., Yu, H.-X., Wu, J., & Li, Y. (2024). Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians. ECCV.
- [23]Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV.
- [24]Fang, H.-S., Wang, C., Fang, H., Gou, M., Liu, J., Yan, H., Liu, W., Xie, Y., & Lu, C. (2023). AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains. IEEE Transactions on Robotics, 39(5), 3929–3945.
- [25]Hu, J., Jones, D., Loza Galindo, G., Huang, S., & Valdastri, P. (2026). Autonomous Robotic Exploration of Unknown Soft Objects. The International Journal of Robotics Research.
- [26]Dasgupta, S., Gupta, A., Tuli, S., & Paul, R. (2024). ActNeRF: Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions. IROS.
- [27]Mazza, R., Natale, C., & Falco, P. (2026). Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects. https://arxiv.org/abs/2601.13979
- [28]Yang, X., Ji, Z., & Lai, Y.-K. (2025). Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materials. The International Journal of Robotics Research, 44(13), 2126–2155.
- [29]Zhang, K., Li, B., Hauser, K., & Li, Y. (2024). AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation. RSS.
- [30]Ma, L., Meng, J., Liu, S., Chen, W., Xu, J., & Chen, R. (2023). Sim2Real$^2$: Actively Building Explicit Physics Model for Precise Articulated Object Manipulation. ICRA.
- [31]Lv, J., Yu, Q., Shao, L., Liu, W., Xu, W., & Lu, C. (2022). SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning. ICRA.


&
&
&
&
&


