WorksComputer Science
SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization
Abstract
Markerless hand-eye calibration under continuous motion requires jointly resolving camera geometry and the time offset between images and joint states. We present SyncHeC (Synchronized Hand-Eye Calibration), a dense differentiable-rendering framework that estimates both from a single robot trajectory. Continuous-time forward kinematics makes latency differentiable, metric depth complements mask contours, and an offline planner combines spatial sensitivity with temporal excitation. Alternating updates support Eye-to-Hand, Eye-in-Hand, and multi-camera calibration. On a 7-DoF arm, one ,s motion yields ,cm translation error against a marker-board reference, ,cm touch-test error, and ,ms latency error relative to an ,ms reference with uncertainty of ,ms. Against CM-Calib on matched recordings, translation and nominal latency errors improve by and . Experiments span seven arms and four depth sensors. Compensating the recovered offset during policy training and deployment increases success by percentage points across six families and six tasks ( matched pairs; paired confidence interval points).
Code: github.com/hongzhengdong/SyncHeC
Introduction
Visual robot control depends on two correspondences (Figure 1). Spatial calibration maps camera observations into the robot’s kinematic frame ([41, 40, 5, 17]). Temporal calibration associates each observation with the joint state it depicts ((Wu et al., 2020)). Exposure, transport, processing, and clock offsets separate a frame’s recorded timestamp from its physical observation time. During motion, this mismatch changes the robot geometry associated with an image and couples a timing error to the estimated extrinsics.
Figure 1. Robotic hand-eye coordination has two coupled facets. Left: spatial pose calibration recovers the camera pose in the robot frame, in both Eye-to-Hand and Eye-in-Hand setups. Right: temporal calibration recovers the sensor-to-control latency : a frame time-stamped at actually observes the joint state at . We solve both jointly.
The same correspondence matters for learned control. Vision-language-action models ([1, 25, 33]) and 3D policies such as R3D ([11]) consume visual and proprioceptive information to predict actions. A calibration layer that aligns these streams provides geometrically and temporally consistent training tuples and deployment observations across policy architectures.
Marker-based methods and active-vision extensions estimate extrinsics from target observations ([52]).
Markerless methods use robot keypoints or direct pose regression ([29, 42]) or align a rendered robot model with image masks ([3, 13]). These approaches enable automatic spatial calibration from static poses. CM-Calib ((Wu et al., 2020)) jointly estimates pose and latency under continuous motion through target-derived correspondences. SyncHeC combines continuous-motion calibration with dense, markerless supervision (Table 1).
Table 1. Capability comparison of representative hand-eye calibration paradigms. Marker-free, EiH (Eye-in-Hand support), Latency (estimates ), Hetero. (heterogeneous arms), Dyn. (calibrates during continuous motion).
SyncHeC integrates three components. First, interpolating the high-rate joint log creates a continuous-time kinematic model through which gradients reach the unknown latency. Second, metric RGB-D depth supplements mask contours with direct surface-distance constraints, strengthening optical-axis pose estimation. Third, a planned trajectory supplies spatially diverse views and sustained active motion for joint estimation. An alternating optimizer updates the temporal and spatial variables in turn.
SyncHeC supports both Eye-to-Hand (EtH) and Eye-in-Hand (EiH) cameras, and extends to multiple cameras through a common kinematic model. One short acquisition replaces repeated stop-and-go motion. We evaluate spatial accuracy, timing accuracy, conditioning, depth processing, and hardware transfer, then test the value of the recovered offset through matched downstream manipulation trials.
Related Work
Spatial hand-eye calibration. Classical solvers estimate or from calibration targets and forward kinematics ([41, 40, 5, 17]). Active approaches select informative viewpoints ([52, 48]). Markerless networks predict keypoints or camera poses ([29, 42]). EasyHeC and EasyHeC++ optimize rendered robot silhouettes, with pretrained image models supporting automatic initialization ([3, 13]). Hydra registers RGB-D observations to the robot model ([16]). Our dense spatial objective adds a differentiable timing variable to robot-model alignment.
Differentiable rendering and dynamic reconstruction. Differentiable rasterization propagates image residuals to scene geometry [34, 22, 28], supporting inverse rendering [23, 24] and object pose estimation [44]. DRRobot differentiates a kinematics-aware robot appearance model with respect to joint controls [32]; RobotArena∞ uses differentiable rendering for camera-pose estimation in real-to-sim reconstruction [18]. Dynamic free-viewpoint reconstruction with a flying camera also couples camera calibration and articulated motion estimation [9]. Here, recorded robot kinematics provide a measured trajectory, and shifting its time argument makes sensor latency directly observable through rendered masks and depth.
Temporal calibration and informative motion. Camera–IMU timing is estimated jointly with motion in visual-inertial systems [36] (Li and Mourikis 2014), including continuous-time batch formulations [6]. Related methods address targetless LiDAR–IMU and LiDAR–camera calibration (Lv et al., 2020; Li et al., 2024). For arms, CM-Calib uses correspondence matching (Wu et al., 2020), while laser-vision methods exploit structured features [50]. SyncHeC estimates latency through dense mask and depth observations of the robot. Our planner combines spatial sensitivity with temporal excitation; this extends informative-view selection [19, 20] to a timed robot trajectory.
Geometry and dynamics for manipulation. Several complementary approaches organize manipulation around geometric or motion representations. R3D studies 3D policy encoders and training [11]; LAMP lifts image-editing cues into 3D inter-object transformations [43]. AdaDexGrasp uses single human demonstrations to guide grasp-skill learning [39], and object-centric motion priors transfer predicted human–object states to robot learning [12]. Particle-based world models learn interaction dynamics from human data and generate reference trajectories through model predictive control [10]. Our downstream study examines the observation-to-joint alignment used by image policies, point-cloud policies, and a particle-based dynamics controller.
Method
SyncHeC couples continuous-time robot kinematics with dense rendering losses to estimate an extrinsic transform and an image–joint offset. Figure 2 summarizes spatial initialization, trajectory acquisition, and alternating optimization.
Figure 2. Overview. Stage 1 imports the URDF into a simulator, sizes an adaptive sampling sphere from the first real depth frame, selects an initial pose by a 2D–3D fitness, and refines it by differentiable rendering to obtain . Stage 2 executes one offline-planned continuous trajectory, parameterizes the high-frequency joint stream as continuous-time forward kinematics, and jointly optimizes the extrinsics and latency from a 2D–3D differentiable-rendering loss.
Observation Model
Let map coordinates from frame to frame . For EtH we estimate the base-to-camera transform ; for EiH we estimate the flange-to-camera transform . Their inverses map camera coordinates into the corresponding robot frame. RGB-D frames have timestamps , and joint measurements form a denser log . The frame stamped observes the joint state at
The constant offset includes sensing-to-availability delay and the difference between clock origins. It specifies image–joint pairing during acquisition, training, and deployment.
For link , maps link coordinates into the robot base, and denotes flange kinematics [2]. The two camera models are
Given link meshes, camera intrinsics, and a differentiable renderer, these transforms produce a mask and depth image for each observation. We parameterize the static extrinsic by and estimate jointly.
Metric Depth and Spatial Initialization
Axial conditioning. Known-scale perspective geometry carries an optical-axis constraint: for , . Its sensitivity becomes small for distant or narrow projections, and mask contours provide limited support along this direction. Metric depth adds direct surface-distance residuals over the visible robot, strengthening axial conditioning (Figure 3).

Figure 3. Complementary geometric constraints. Solid and dashed shapes show reference and axially shifted geometry. (a) Mask contours constrain axial translation through perspective, with weak sensitivity in distant or narrow projections. (b) Metric depth adds a direct surface-distance residual.
Sensor-aligned depth. We use the metric Hypersim checkpoint of Depth Anything V2 (ViT-L) [49]. Let be sensor depth and the predicted depth, both registered to the RGB grid. On pixels where both are valid, their median ratio sets a scale . We retain valid sensor measurements and fill holes with ; nonpositive and nonfinite values are excluded. The resulting metric map and validity mask are shared by initialization and dynamic calibration. Appendix B specifies the fusion and supervision region.
Stage 1: spatial anchoring. A single static RGB-D frame initializes the dynamic pipeline. We load the robot URDF into SAPIEN [46], segment its base link with SAM2 [37], and take the median valid depth as a surface-distance anchor . A side-profile correction estimates the base-center distance from this surface anchor (Appendix B). The camera-sampling sphere has radius , with margin m.
For each candidate extrinsic , we render its mask and depth and evaluate
where contains valid overlapping robot pixels. Mask-based rendering refinement of the best candidate produces . The five-view static benchmark additionally refines the extrinsic using the remaining static views.
Continuous-Time Joint Calibration
Differentiable latency. For , the synchronized joint state and its temporal derivative are
The active interval is recomputed as changes. With a 250 Hz log, adjacent samples are 4 ms apart; bounded joint acceleration gives an joint-value interpolation error. Substitution into Equations (2)–(3) propagates time gradients through forward kinematics and rendering.
Dense 2D–3D objective. SAM2 / Grounded-SAM provide observation masks [27, 37, 38]. For frame , the rasterizer outputs a soft mask and metric depth . We minimize
Here intersects the observed robot mask, rendered mask , and valid observed depth. The depth term is zero for an empty intersection. Depth is measured in meters and in seconds.
Joint observability. Stack the rendered observations across frames and differentiate with respect to the six pose parameters and time:
where the expression for is stacked framewise and includes the appropriate EtH or EiH kinematic chain. Local first-order identifiability requires [14]. A temporal change becomes indistinguishable from a pose change when its stacked column lies in the span of the six pose columns. Exciting multiple configurations and velocities helps separate these effects. This criterion depends on the full projected motion across the sequence.
Alternating updates. Starting from and , we alternate a temporal block with a spatial block. The temporal block fixes pose and updates using frames with rad/s. The spatial block resynchronizes all selected frames and updates pose. Both use Adam with persistent optimizer moments. Each block performs 15 passes over mini-batches of four frames. In the dynamic benchmark, all 60 selected frames pass the temporal gate in every cycle, giving 225 actual Adam updates per variable per cycle. Convergence takes 3–5 cycles, or 675–1,125 temporal updates. Appendix A gives the algorithm and a representative trace.
Excitation-Based Trajectory Planning
We optimize a short timed trajectory using spatial and temporal rendering sensitivities. For a candidate camera set , spatial excitation is
The rendered image uses the appropriate observation model in § 3.1. Temporal excitation is
These terms measure pose sensitivity and the magnitude of the rendered observation’s temporal derivative. For Gaussian observation noise, squared sensitivity determines a Fisher-information block; our unsquared temporal score rewards motion over the sampled trajectory. Appendix § 8 derives this relationship and describes the joint conditioning diagnostic.
RRT* ([21]) supplies a collision-free geometric seed, followed by optimization of a timed -knot B-spline:
The refinement enforces joint limits, collision avoidance, and base-link visibility. Planning uses the Stage-1 extrinsic. We normalize the two scores by fixed initial-reference scales and , giving and . The relative weight is selected on the simulation split and held fixed for real experiments.
Multi-Camera Calibration
For several cameras, we sum their normalized rendering objectives, assign each camera its own extrinsic, and evaluate their observations through the shared joint log. The model admits a separate per camera. When paired timing measurements establish equivalence within a specified tolerance, a shared-offset variant imposes . The temporal block then accumulates evidence from both views, while each spatial block updates its own extrinsic. Our dual-camera experiment uses a ms equivalence criterion verified by an independent paired timing reference.
Experiments
Setup and Evaluation
The main platform is a 7-DoF xArm7 with environment-mounted EtH and wrist-mounted EiH RealSense D435i cameras. Both stream 848 480 RGB-D at 30 Hz, and the controller records joints at 250 Hz. Rendering and optimization use Nvdiffrast (Laine et al., [28]) on an RTX 4090 with an i9-13900K. The planned 25-knot trajectory lasts approximately 6.5 s. We retain the 60 highest-flow frames from approximately 195 recorded frames.
Optimization settings. We set , , and . Adam (Kingma and Ba, [26]) uses learning rates for rotation, for translation, and for time, with moments and numerical epsilon . Each temporal and spatial block uses 15 passes, batch size four, and persistent moments. At most 15 cycles give 3,375 temporal updates. We stop when cycle-to-cycle changes are below 0.05 mm in translation, 0.01° in rotation, and 0.5 ms in time. Loss weights are selected on the simulation split from and , then held fixed across real experiments. Varying within changes translation error by less than 0.03 cm.
Independent references. A signal-generator, LED, and oscilloscope protocol relates image exposure to the logged physical onset of an arm micro-motion. Exposure-phase, clock-mapping, and physical-onset measurements bound residual biases by 0.25, 0.10, and 0.20 ms, respectively; the phase budget includes RGB–depth timestamp skew. Twenty repetitions give an 87 ms nominal reference with approximately 2 ms single-repeat spread and a 0.95 ms confidence half-width for the mean. Combining that interval with the three residual bounds gives 95% reference uncertainty ms (Appendix § 9). Spatial references are obtained independently from repeated marker-board measurements and a workspace touch test.
Metrics and comparisons. We report translation error in cm, geodesic rotation error in degrees, and nominal latency error in ms. Marker-board pose error measures extrinsic agreement; the touch test additionally includes probe-offset and robot-positioning errors. Spatial reference agreement is approximately 1 mm.
Static baseline results in Table 2–Table 3 follow the simulation and touch protocols reported by EasyHeC++ (Hong, Zheng, and Chen, [13]); SynchHeC reruns those protocols with . We also run Hydra under the matched static touch protocol. On dynamic data, CM-Calib and SL-Calib use the same planned trajectory, recordings, and joint logs as SynchHeC, with initialization matched where applicable. CM-Calib receives target-derived poses and forward kinematics. SL-Calib is our straight-line-motion implementation of the correspondence-based approach. Appendix § 9.§ 3 records comparison provenance and controls.
| Method | #v | (°)↓ | (cm)↓ |
| Eye-to-Hand | |||
| Marker-based (Tsai and Lenz, [41]) | 20 | 0.870 | 2.000 |
| DREAM (Lee et al., [29]) | 5 | 0.704 | 0.303 |
| EasyHeC (Chen et al., [3]) | 5 | 0.081 | 0.206 |
| EasyHeC++ (Hong, Zheng, and Chen, [13]) | 5 | 0.045 | 0.135 |
| SynchHeC () | 5 | 0.038 | 0.105 |
| Eye-in-Hand | |||
| Zhang et al. [52] | 10 | 0.148 | 0.315 |
| Valasskis, Dreczykowski, and Johns (2022) | 1 | 4.400 | 1.340 |
| EasyHeC++ (Hong, Zheng, and Chen, [13]) | 5 | 0.117 | 0.112 |
| SynchHeC () | 5 | 0.095 | 0.086 |
Table 2. Simulation calibration accuracy. Top: Eye-to-Hand. Bottom: Eye-in-Hand. #v: number of views. Baseline values are reproduced from the EasyHeC++ evaluation (Hong, Zheng, and Chen, [13]).
Table 3. Real static touch-test error (cm), averaged over ten poses. Hydra and SynchHeC are evaluated here; the other results are reproduced from Hong, Zheng, and Chen [13]. Protocols are specified in Appendix § 9.§ 3.
Static Spatial Accuracy
In simulation (Table 2), our translation errors are 0.105 cm for EtH and 0.086 cm for EiH, compared with 0.135 and 0.112 cm for EasyHeC++. On the real touch test (Table 3), EtH performance is comparable across SynchHeC, EasyHeC++, and Hydra at the reference resolution. EiH error decreases from 0.31 cm for EasyHeC++ and 0.28 cm for Hydra to 0.16 cm. Metric depth and the surface-to-base correction are particularly useful in the constrained wrist-camera view.
Dynamic Spatio-Temporal Accuracy
Table 4 evaluates continuous-motion calibration. Across five matched recordings, SyncHeC reaches 0.21 ± 0.04 cm / 0.15 ± 0.05° / 2.7 ± 0.6 ms for EtH, and 0.24 ± 0.05 cm / 0.18 ± 0.07° / 5.4 ± 0.9 ms for EiH. CM-Calib gives 0.78 ± 0.08 and 0.95 ± 0.10 cm translation error, with latency errors 6.2 ± 0.8 and 8.5 ± 1.1 ms. The EtH ratios are 3.7× for translation and 2.3× for nominal latency error.
| Method | (cm)↓ | (°)↓ | (ms)↓ |
| Eye-to-Hand | |||
| SL-Calib‡ | 1.15 | 0.92 | 11.5 |
| CM-Calib (Wu et al., 2020) | 0.78 ± 0.08 | 0.55 | 6.2 ± 0.8 |
| EasyHeC† [3] | 4.85 | 3.42 | — |
| EasyHeC++† [13] | 3.92 | 2.75 | — |
| SyncHeC | 0.21 ± 0.04 | 0.15 ± 0.05 | 2.7 ± 0.6 |
| Eye-in-Hand | |||
| SL-Calib‡ | 1.34 | 1.15 | 14.8 |
| CM-Calib (Wu et al., 2020) | 0.95 ± 0.10 | 0.72 | 8.5 ± 1.1 |
| EasyHeC† [3] | 5.68 | 4.15 | — |
| EasyHeC++† [13] | 4.55 | 3.38 | — |
| SyncHeC | 0.24 ± 0.05 | 0.18 ± 0.07 | 5.4 ± 0.9 |
Table 4. Continuous-motion calibration against the 87 ms reference ( ms). CM-Calib and ours use five matched recordings; SDs are shown for CM translation/latency and all our metrics.
The corresponding mean offset estimates are 84.3/81.6 ms for SyncHeC and 80.8/78.5 ms for CM-Calib (EtH/EiH). Our latency error remains smaller throughout the timing reference’s ±1.5 ms uncertainty interval. For EtH, the CM-Calib-to-SyncHeC error ratio over this interval is [1.83, 3.92] (Appendix § 9).
EasyHeC and EasyHeC++ applied directly to the moving sequence associate each frame with the unshifted joint state, giving EtH errors of 4.85 and 3.92 cm. A fixed-reference timing control synchronizes the inputs at 87 ms before spatial optimization: EasyHeC++ then reaches 0.43/0.48 cm and the SyncHeC spatial objective reaches 0.20/0.23 cm (EtH/EiH). Joint estimation reaches essentially the same spatial accuracy as the fixed-reference version of SyncHeC. This control separates the benefit of temporal alignment from that of dense metric-depth supervision.
Optimization behavior. A representative EtH run reaches 84.3 ms from zero in 900 temporal updates (Appendix § 6). Its data and regularizer gradients are approximately −0.0843 and +0.0843 s⁻¹. Continuing alternation for 900 additional temporal updates with stopping disabled changes the final offset by less than 0.2 ms, characterizing the local stationary solution of the regularized objective.
Efficiency. The dynamic pipeline uses one 6.5 s acquisition and approximately eight minutes of total wall-clock time. On the same EtH task, EasyHeC takes approximately 32 minutes, EasyHeC++ 12 minutes, and our static mode 13 minutes. Dynamic touch-test error is 0.34 cm, compared with 0.28 cm for our static mode and 0.30 cm for EasyHeC++. The dynamic board-reference translation error is 0.21 cm.
Multi-Camera and Hardware Transfer
For the dual-camera rig (Figure 4), independent paired timing measurements give an EtH–EiH offset difference of 0.2 ms with a 95% interval of [−0.4, 0.8] ms, contained within the ±1 ms sharing criterion. This supports the shared-offset variant of Section 3.5. On four matched dual-view recordings, drawn from the five-run benchmark, it obtains 0.45 ± 0.27 cm / 0.63 ± 0.50° for EtH and 0.59 ± 0.22 cm / 1.24 ± 0.73° for EiH. All four runs converge. The shared fit uses both views in one calibration, while the separate fits attain the single-camera accuracy in Table 4.

Figure 4. Heterogeneous multi-camera co-calibration. Left: the rig, with the tripod-mounted Eye-to-Hand and wrist-mounted Eye-in-Hand cameras labelled. Right: frames from each camera over one continuous trajectory.
Table 5 shows representative runs across seven arms and four depth sensors. The xArm7/D435i configuration is the shared reference, yielding ten distinct hardware configurations. Five independent motions per configuration give 50 calibration runs. The representative arm runs span 0.24–0.68 cm translation error and 1.2–4.8 ms latency error. Across the four sensors, reference offsets range from 66 to 115 ms and estimated offsets lie within 3 ms of their references. The more difficult Z1 Pro and Azure Kinect configurations give five-run translation errors of 0.67 ± 0.09 and 0.74 ± 0.08 cm, respectively. The table’s xArm7 entry is a separate EtH run from the five-run mean in Table 4.
| Platform | (cm) | (°) | ||
| xArm7 | 0.25 | 0.18 | 85.6 | 87.0 |
| Franka Panda | 0.31 | 0.22 | 86.8 | 88.5 |
| RM75-B | 0.28 | 0.19 | 86.1 | 87.5 |
| Z1 Pro | 0.68 | 0.45 | 84.2 | 89.0 |
| EC66 | 0.36 | 0.25 | 86.5 | 88.0 |
| ER7 | 0.24 | 0.16 | 87.3 | 88.5 |
| CR5 | 0.42 | 0.31 | 85.2 | 87.0 |
| D435i (Stereo) | 0.25 | 0.18 | 85.6 | 87.0 |
| L515 (LiDAR) | 0.18 | 0.13 | 112.4 | 115.0 |
| Femto Bolt (ToF) | 0.21 | 0.15 | 64.5 | 66.0 |
| Azure Kinect (ToF) | 0.75 | 0.52 | 92.1 | 94.0 |
Table 5. Representative runs across arms (top) and depth sensors (bottom). xArm7/D435i is repeated as the shared reference, giving ten distinct configurations. Each configuration has five motions; repeat summaries appear in the text. Offsets are in ms.
Controlled Ablations
Metric depth and completion. We hold frames, masks, Stage-1 anchor, and optimizer fixed across three Stage-2 variants. Raw sensor depth gives translation errors of 0.38/0.44 cm; sensor-aligned completion gives 0.21/0.24 cm; and disabling the Stage-2 depth term gives 1.18/1.35 cm (EtH/EiH), averaged over five matched runs. Thus raw metric depth already supplies a useful constraint, and completing missing surface observations further improves pose estimation. Table 6 also shows the associated latency improvement.
| Variant | \multicolumn{3}{c | }{EtH} | \multicolumn{3}{c | }{EiH} | ||
| Stage-2 depth off | 1.18 | 0.85 | 3.5 | 1.35 | 0.96 | 8.2 |
| Spatial planning only | 0.86 | 0.65 | 16.4 | 0.95 | 0.72 | 18.2 |
| Full (Ours) | 0.21 | 0.15 | 2.7 | 0.24 | 0.18 | 5.4 |
Table 6. Dynamic ablations with the Stage-1 anchor fixed. is in cm, in degrees, and in ms. The full row repeats the five-run means in Table 4.
Temporal excitation and joint conditioning. With , latency errors increase to 16.4/18.2 ms. For the stacked seven-column observation Jacobian, after consistent channel scaling and unit-column normalization, the minimum singular values are 0.21/0.18 for the full planner and 0.03/0.02 for spatial-only planning. Both trajectories yield full-rank Jacobians; temporal excitation improves the conditioning of joint pose–time estimation.
Alternation and interpolation. Joint Adam with the same per-variable learning rates, batch size, and actual total number of gradient evaluations reaches approximately 0.5 cm translation and 6 ms latency error over five runs. Alternation reaches the accuracy in Table 4 in 3–5 cycles. On the same 250 Hz joint logs, replacing the linear interpolant with a cubic interpolant changes the estimated translation vector by less than 0.01 cm and latency by less than 0.2 ms.
Downstream Visuomotor Policies
We evaluate six policy families on six real tasks (Table 7): ACT (Zhao et al., [53]), Diffusion Policy (DP) (Chi et al., [4]), (Black et al., [1]), DP3 (Ze et al., 2024), ManiFlow (Yan et al., [47]), and a particle-based world model with sampling-based MPC (WM). Particle dynamics and learned-model control provide the basis for the latter family (Hong et al., [10]; Hafner et al., [7]; 2025; Zhu et al., [54]; Huang et al., [15]).
Table 7. Success rate (%) without → with recovered-offset compensation. Each cell uses 50 matched pairs on the six tasks: stack three blocks; place cup, banana, or kettle; open drawer; fold towel. Across 1800 pairs, the gain is 8.11 points (paired 95% CI [6.3,9.9]).
Each task uses the same 50 demonstrations for both conditions. We fix extrinsics, use one matched training seed per policy–task cell, and replay the same 50 initial configurations in each pair of evaluations. Without compensation, an image is paired with the nearest joint timestamp. With compensation, it is paired with , using our recovered offset at both training and deployment. Action and chunk timestamps remain fixed.
Across 36 cells, 1,800 matched pairs comprise 3,600 physical executions. Successful trials increase from 681 to 827. Compensation uniquely succeeds in 210 pairs and the uncompensated policy uniquely succeeds in 64, giving 146 net additional successes and an 8.11-percentage-point gain. A cell-stratified paired Wald interval is [6.3, 9.9] percentage points at 95% confidence, conditional on the trained policies and calibration. Appendix § 10 gives task definitions, pairing, and the variance calculation.
Mean gains are 4.3 points for ACT, 7.7 for DP, 11.7 for , 6.0 for DP3, 7.7 for ManiFlow, and 11.3 for WM. Every policy–task cell has a positive observed difference. The aggregate improvement spans image-based, point-cloud, and dynamics-based control, demonstrating the practical value of aligning the observation and joint streams before policy learning and execution.
Operating Conditions
The pipeline uses a coarse spatial anchor within roughly 15 cm, visible robot geometry, and sufficiently excited motion. Strong blur becomes prominent above approximately 0.5 m/s in the evaluated imaging setup; acquisition speed should preserve reliable mask boundaries. Pixels without valid depth after completion contribute through the mask channel. A constant offset describes a stable sensing and control stack; changes in the bus, driver, or host state that produce drifts exceeding 20 ms call for recalibration. These conditions guide acquisition and deployment.
Conclusion
SyncHeC combines continuous-time kinematics, metric depth, and planned excitation to jointly estimate hand-eye pose and latency from a single continuous robot motion. It supports EtH, EiH, and heterogeneous hardware. Applying the recovered offset during policy training and deployment raises real manipulation success by 8.1 points across six families.
Optimization and Implementation
Update Accounting and Stopping
Algorithm 1 specifies the alternating optimizer. A pass visits every mini-batch in the active frame set once. With , batch size , and passes, each block executes
Adam updates. There is one optimizer step per batch and no gradient accumulation. The temporal gate is evaluated at the current synchronized times at the start of each cycle; all 60 selected benchmark frames pass it. Each cycle therefore uses 225 temporal and 225 spatial gradient evaluations. Three to five cycles correspond to 675–1,125 temporal updates and 1,350–2,250 total gradient evaluations; the maximum of 15 cycles allows 3,375 temporal updates.
For a mini-batch , the optimized objective is the mean frame data loss plus one regularization term:
This keeps the penalty’s scale consistent with Equation (9). Time is stored in seconds, translation in meters, and rotation updates use the Lie-algebra representation. Separate optimizers retain their moments across passes and alternating cycles.
Representative Trace and Regularization
Table 8 records one EtH run from the dynamic benchmark. The pose is initialized by Stage 1 and the temporal variable starts at zero. Its final offset, 84.3 ms, is also close to the five-run mean.
| Temporal updates | 0 | 225 | 450 | 675 | 900 | |
| Offset (ms) | 0 | 43.8 | 78.1 | 84.0 | 84.3 |
Table 8. Representative EtH optimization trace. The count is the cumulative number of temporal optimizer updates.
Writing the mean data objective as , the temporal stationarity condition is
At and , the penalty gradient is , balancing the observed data gradient of approximately . Retaining optimizer states and continuing the original alternating sequence for 900 further temporal updates, together with the associated spatial blocks, changes the final offset by less than 0.2 ms with stopping disabled. The penalty’s influence is governed by the local data gradient and curvature.
Algorithm 1: Alternating pose and latency optimization
Input: selected frames, joint log, mesh, intrinsics, , , , .
Initialize , , and the spatial and temporal Adam states.
For each cycle : (a) Save the current extrinsic and offset for the stopping test. Select . (b) Fix . For each of passes over , form batches of frames, recompute active interpolation intervals, evaluate , and take one Adam update of per batch. (c) Fix . Resynchronize all frames. For each of passes, form batches of frames and take one Adam update of per batch. (d) Stop if all cycle-to-cycle changes satisfy: translation , rotation , and time .
Return and .
Software Implementation
SyncHeC offers Nvdiffrast rendering on CUDA and soft rendering in PyTorch. Both use the same continuous-time kinematics, per-frame normalized losses, and mini-batch alternating optimizer. Configuration files specify batch size, passes per block, loss weights, and learning rates; each run records the selected frames, recovered transform, offset, and actual update counts.
Recordings supply camera intrinsics, time-stamped joints, robot masks, metric depth, and URDF geometry. The depth-fusion function preserves valid sensor measurements and completes holes using sensor-scaled predictions. Spatial initialization uses a separate base-link mask and the surface-to-center correction. EtH and EiH use the transforms in Equations (2)–(3); multi-camera calibration supports camera-specific or shared offsets.
The acquisition pipeline uses rendered-pixel trajectory scoring with RRT* and collision checks. The software also provides a lightweight B-spline refinement interface based on projected URDF landmarks, with collision and visibility callbacks. Table 9 maps the calibration components to their implementation.
| Component | Function |
| Calibration objective | optimize.CalibrationProblem |
| Joint interpolation | kinematics.q_sync |
| Temporal derivative | kinematics.temporal_jacobian |
| EtH / EiH transforms | kinematics.LinkPoseModel |
| Mask / depth losses | render.losses |
| Initialization fitness | stage1.joint_fitness |
| Alternating optimizer | optimize.calibrate |
| Joint optimizer | optimize.calibrate_joint |
Table 9. Core mathematical components in SyncHeC.
The executable synthetic example generates a four-link robot recording with a known 87 ms offset, initializes pose by perturbing the reference extrinsic, and starts the temporal variable at zero. Its default configuration uses 24 frames at 64 × 48 resolution, , temporal learning rate 0.001, and five passes over batches of four frames. Each block therefore performs 30 updates per cycle. The output reports pose and timing errors against the generating parameters together with the complete optimization history.
Metric Depth and Anchoring
Metric Fusion and Validity
Let contain valid raw sensor pixels and contain finite, positive Depth Anything V2 predictions. Using the ViT-L metric Hypersim checkpoint, we compute
With overlap available for this scale estimate, the observation depth is
Pixels without a valid retained measurement or scaled prediction remain invalid. The per-frame depth-supervision region is
Mask residuals are evaluated across the full image. The scale is anchored to sensor measurements, and valid sensor depth is preserved throughout the fusion.
The raw-depth control uses for its Stage-2 depth residual. The completed-depth control uses . The control switches off this Stage-2 residual. All three use the same completed-depth Stage-1 anchor, frames, masks, and optimizer, isolating the contribution of Stage-2 depth supervision and completion.
Surface-to-Base Correction
The median depth over valid segmented base-link pixels yields . Under the base-centered viewing approximation used for candidate sampling, this gives a surface-distance estimate. The side-profile model relates this estimate, camera height relative to the base origin, and base cylinder radius to center distance :
It describes the visible cylindrical side in the regime and . A first-order correction evaluates the right-hand side at , then sets the sampling radius to $L + .
Excitation and Joint Conditioning
Sensitivity Interpretation
For a differentiable observation vector with independent Gaussian noise of variance , the Fisher matrix is
For small camera-pose perturbations with covariance , linearization gives
For latency, the corresponding per-frame information is
The planner uses alongside the spatial variance surrogate. The un squared temporal form reduces the relative reward for isolated high-amplitude peaks compared with . Fixed reference scales make the relative weight dimensionless and comparable across planning iterations. The joint Jacobian diagnostic below evaluates pose–time coupling after planning.
Seven-Parameter Diagnostic
For each EtH or EiH sequence, we stack the six pose derivatives and one temporal derivative of the mask and valid-depth observation channels. The diagnostic uses consistent channel weights, pixel-count normalization, and sampling rules across variants. It differentiates the data observations, excluding the latency penalty. Let be the scaled matrix and
Unit-column normalization makes the singular-value comparison independent of the units used for translation, rotation, and time.
All four matrices in Table 10 are full rank. Because seven columns have unit norm, and . The full planner therefore has condition number at most 12.6/14.7 (EtH/EiH); spatial-only planning has condition number at least 33.3/50. These bounds quantify improved conditioning for the joint estimation problem.
| Trajectory | EtH | EiH |
| Spatial excitation only () | 0.03 | 0.02 |
| Spatial and temporal excitation | 0.21 | 0.18 |
Table 10. Minimum singular value of the normalized, stacked data Jacobian.
Structural first-order ambiguity occurs when one fixed pose perturbation reproduces the effect of a time shift across all stacked observations. A constant rigid-body twist can produce such a case. Constant joint velocity alone does not determine the stacked column relationship, since articulated kinematics, perspective, and visibility also change along a trajectory.
Timing Reference and Comparison Protocols
Reference Error Budget
The reference maps image exposure and physical motion onset into a common timing measurement. The recorded event is the robot’s actual motion onset in the joint stream. Exposure-phase measurements include the RGB–depth timestamp relation, and clock mapping accounts for the two stream origins. Table 11 lists the residual error budget.
| Component | ms |
| Exposure phase, including RGB–depth skew | 0.25 |
| Clock mapping | 0.10 |
| Physical motion-onset localization | 0.20 |
| Repeated-mean 95% half-width | 0.95 |
| Combined 95% uncertainty | 1.50 |
Table 11. Uncertainty budget for the nominal 87 ms reference. The first three entries are residual bias bounds; the fourth is a confidence half-width for the repeated mean.
The 20 repetitions characterize repeatability; the exposure-phase and onset measurements supply sub-frame timing. Adding the three residual bias bounds to the repeated-mean interval gives ms.
For a reference adjustment ms, the EtH mean offsets give
Thus the ordering is maintained across the reference interval. The EiH means, 78.5 and 81.6 ms, likewise retain their ordering. Spatial evaluation uses separate marker-board and touch references.
Paired Multi-Camera Timing
The independent timing reference gives an EtH–EiH difference of 0.2 ms and a 95% interval ms. Its containment in ms motivates the shared-offset model for the four paired dual-view recordings. Common reference errors cancel in this difference, giving a tighter differential interval than the absolute reference budget. Camera-specific regularized calibration estimates (84.3 and 81.6 ms) reflect each view’s data objective as well as the penalty; the sharing criterion is evaluated using the independent timing reference.
Matched Baselines and Controls
Static protocol. The marker-based, DREAM, EasyHeC, EasyHeC++, Zhang et al., and Valassakis et al. entries in Table 2–Table 3 are taken from the corresponding EasyHeC++ comparison [13]. Our static reruns use the same simulation/touch evaluation definitions, and the Hydra comparison uses the matched real static protocol. The touch evaluation averages ten poses.
Dynamic protocol. CM-Calib, SL-Calib, and SyncHeC share recordings, selected trajectory, and joint logs; initialization is matched where applicable. CM-Calib uses target-derived poses and forward kinematics. The fixed-reference control sets ms for EasyHeC++ and the SyncHeC spatial objective while preserving their remaining inputs, initialization, and pose-update rules. This yields 0.43/0.48 cm and 0.20/0.23 cm, respectively.
Optimization control. Joint Adam and alternating Adam use the same per-variable learning rates and batch size, and are compared at the same actual total count of data-gradient evaluations. A joint update evaluates both variable gradients in one forward/backward computation; an alternating update evaluates the active block. The evaluation budget counts executed computations, including the treatment of early stopping.
Hardware repetitions. Seven arm configurations and four sensor configurations share the xArm7/D435i reference, giving ten unique configurations. Five motions per configuration give 50 runs in total, including the ten representative runs in Table 5. The additional repetitions supply the Z1 Pro and Azure Kinect means and standard deviations reported in § 4.4.
Downstream Study: Protocol and Statistics
Controlled Pairing
The study varies image–joint temporal pairing while fixing robot hardware, cameras, extrinsics, demonstrations, architecture, training hyperparameters, and the matched training seed within each policy–task cell. Each task provides 50 successful teleoperated demonstrations. The same recordings train both conditions:
Uncompensated: pair each image with the joint state carrying the nearest recorded timestamp.
Compensated: pair each image at with the interpolated joint state at .
The same rule applies during deployment. Compensation uses the recovered calibration estimate. Action timestamps and action-chunk scheduling are fixed across conditions.
Observations and Policy Families
The robot uses the two D435i cameras at 30 Hz and joint logs at 250 Hz. The constant offset includes the stream-clock difference and sensing-to-availability delay. For point-cloud policies, offline demonstration processing enhances depth, transforms both views into the robot base frame, merges and crops them to the workspace, and applies farthest-point sampling to 8192 points. At deployment, point clouds are produced online at approximately 30 Hz with offline depth enhancement disabled. Both temporal conditions use this same observation pipeline.
ACT uses RGB observations and an action-chunking transformer (Zhao et al., [53]); DP uses RGB-conditioned action diffusion (Chi et al., [4]); is fine-tuned from its released vision-language-action checkpoint (Black et al., [1]). DP3 (Ze et al., 2024) and ManiFlow (Yan et al., [47]) use the point clouds above. The WM family models particle-based 3D dynamics and uses MPPI-style sampling-based MPC with 1000 action samples per iteration and 15–30 refinement iterations. Particle-based dynamics learned from human interaction and model-based control motivate this representation (Hong et al., [10]; Zhu et al., [54]; Hafner et al., [8]).
Tasks and Execution
Table 12 defines the six tasks. For each of the 36 policy–task cells, 50 independently randomized initial configurations are replayed in both conditions. Initial-state randomization is independent across cells. A trial succeeds if its task criterion is met within 60 s; timeouts, dropped objects, and unrecoverable states count as failures. Each cell therefore contains 50 matched pairs and 100 physical executions.
| Task | Regime | Randomization of the initial configuration | Success criterion |
| Stack three blocks | long-horizon | three 20×40 cm cells, ±5 cm jitter | all three stacked, tower stable at timeout |
| Place cup | pose-constrained | 60×40 cm board, 360° yaw | cup upright inside the target region |
| Place banana on plate | pose-constrained | plate on midline +10 cm; banana 10–15 cm left, 360° yaw | banana resting on the plate |
| Place kettle on stove | pose-constrained | 20×20 cm region, ±60° | kettle upright on the stove plate |
| Open drawer | articulated | L-shaped region (70×20 + 20×20 cm), rotated 90° | drawer pulled to ≥ 50% extension |
| Fold towel | deformable | 70×50 cm region, ±30° | towel folded, free edge within tolerance |
Table 12. The six real-robot tasks: regime, randomization of the initial object configuration, and success criterion. Yaw ranges are about the vertical axis.
Paired Aggregate Effect
Let denote the difference in binary success between conditions for pair in cell . With and , the aggregate estimate is
For each cell, let and count pairs won only with and only without compensation. Then is fixed by the cell’s marginal success counts, and the sample variance is
The variance of the stratified aggregate, conditional on the trained policies and calibration, is estimated by
With equal cell sizes, this reduces to
The counts satisfy , , and . The estimated gain is and its standard error is 0.00904. The paired Wald interval is [0.0634, 0.0988], or [6.3, 9.9] percentage points.
This interval measures execution variability for the fixed trained policies and calibration. The 36 cell differences and six family averages describe how the aggregate gain is distributed across tasks and architectures. The total counts in Table 13 agree with the 681 and 827 successes obtained from Table 7.
| Uncompensated | Compensated | ||
| Success | Failure | ||
| Success | 617 | 64 | |
| Failure | 210 | 909 |
Table 13. Aggregate paired outcomes over 1800 initial configurations.
References
- [1]Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; Jakubczak, S.; Jones, T.; Ke, L.; Levine, S.; Li-Bell, A.; Mothukuri, M.; Nair, S.; Pertsch, K.; Shi, L. X.; Tanner, J.; Vuong, Q.; Walling, A.; Wang, H.; and Zhilinsky, U. 2025. π₀: A Vision-Language-Action Flow Model for General Robot Control. In Robotics: Science and Systems.DOI
- [2]Carpentier, J.; Saurel, G.; Buondonno, G.; Mirabel, J.; Lamiraux, F.; Stasse, O.; and Mansard, N. 2019. The Pinocchio C++ Library: A Fast and Flexible Implementation of Rigid Body Dynamics Algorithms and Their Analytical Derivatives. In IEEE/ICSE International Symposium on System Integration, 614–619.DOI
- [3]Chen, L.; Qin, Y.; Zhou, X.; and Su, H. 2023. Easyhec: Accurate and Automatic Hand-Eye Calibration via Differential Rendering and Space Exploration. IEEE Robotics and Automation Letters, 8(11): 7234–7241.DOI
- [4]Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burchfiel, B.; Tedrake, R.; and Song, S. 2025. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. The International Journal of Robotics Research, 44(10-11): 1684–1704.
- [5]Daniilidis, K. 1999. Hand-Eye Calibration Using Dual Quaternions. The International Journal of Robotics Research, 18(3): 286–298.DOI
- [6]Furgale, P.; Barfoot, T. D.; and Sibley, G. 2012. Continuous-Time Batch Estimation Using Temporal Basis Functions. In IEEE International Conference on Robotics and Automation, 2088–2095. IEEE.DOI
- [7]Hafner, D.; Lillicrap, T.; Ba, J.; and Norouzi, M. 2020. Dream to Control: Learning Behaviors by Latent Imagination. In International Conference on Learning Representations.arxiv.org/abs/1912.01603
- [8]Hafner, D.; Pasukonis, J.; Ba, J.; and Lillicrap, T. 2025. Mastering Diverse Control Tasks through World Models. Nature, 640(8059): 647–653.DOI
- [9]Hong, Z. 2025. Free-Viewpoint Video of Outdoor Sports Using a Flying Camera. In Computer Vision – ECCV 2024, 178–195. Springer.DOI
- [10]Hong, Z.; Liu, Y.; Hou, H.; Ai, B.; Wang, J.; Mu, T.; Qin, Y.; Gu, J.; and Su, H. 2025. Learning Particle-Based World Model from Human for Robot Dexterous Manipulation. In RSS Workshop on Dexterous Manipulation: Learning and Control with Diverse Data.
- [11]Hong, Z.; Wu, S.; Cui, H.; Zhao, B.; Ji, R.; He, Y.; Zhang, H.; Ke, Z.; Wang, J.; Zhang, G.; and Gu, J. 2026. R3D: Revisiting 3D Policy Learning. arXiv:2604.15281.arxiv.org/abs/2604.15281
- [12]Hong, Z.; and Zhang, G. 2026. Learning Object-Centric Motion Priors from Human for Robotic Dexterous Manipulation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(22): 18288–18296.
- [13]Hong, Z.; Zheng, K.; and Chen, L. 2024. EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 816–823.arxiv.org/abs/2410.09293
- [14]Horn, R. A.; and Johnson, C. R. 2012. Matrix Analysis. Cambridge, UK: Cambridge University Press, 2nd edition. ISBN 9781139020411.
- [15]Huang, W.; Chao, Y.-W.; Mousavian, A.; Liu, M.-Y.; Fox, D.; Mo, K.; and Fei-Fei, L. 2026. PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. arXiv:2601.03782.arxiv.org/abs/2601.03782
- [16]Huber, M.; Tian, H.; Mower, C. E.; Müller, L.-R.; Ourselin, S.; Bergeles, C.; and Vercauteren, T. 2025. Hydra: Marker-Free RGB-D Hand-Eye Calibration. arXiv:2504.20584.arxiv.org/abs/2504.20584
- [17]Ilonen, J.; and Kyrki, V. 2011. Robust Robot-Camera Calibration. In International Conference on Advanced Robotics, 67–74.DOI
- [18]Jangir, Y.; Zhang, Y.; Yamazaki, K.; Zhang, C.; Tu, K.-H.; Ke, T.-W.; Ke, L.; Bisk, Y.; and Fragkiadaki, K. 2026. RobotArena: Scalable Robot Benchmarking via Real-to-Sim Translation. In International Conference on Learning Representations.arxiv.org/abs/2510.23571
- [19]Jiang, W.; Lei, B.; and Daniilidis, K. 2024. FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields Using Fisher Information. In European Conference on Computer Vision.arxiv.org/abs/2311.17874
- [20]Jin, L.; Chen, X.; Rückin, J.; and Popović, M. 2023. NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering. In IEEE/RSJ International Conference on Intelligent Robots and Systems.arxiv.org/abs/2303.01284
- [21]Karaman, S.; and Frazzoli, E. 2011. Sampling-Based Algorithms for Optimal Motion Planning. The International Journal of Robotics Research, 30(7): 846–894.arxiv.org/abs/1105.1186
- [22]Kato, H.; Beker, D.; Morariu, M.; Ando, T.; Matsuoka, T.; Kehl, W.; and Gaidon, A. 2020. Differentiable Rendering: A Survey. arXiv:2006.12057.arxiv.org/abs/2006.12057
- [23]Kaya, B.; Kumar, S.; Oliveira, C.; Ferrari, V.; and Van Gool, L. 2021. Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3804–3814.arxiv.org/abs/2012.06777
- [24]Kerbl, B.; Kopanas, G.; Leimkühler, T.; and Drettakis, G. 2023. 3d Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph., 42(4): 139–1.arxiv.org/abs/2308.04079
- [25]Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; Vuong, Q.; Kollar, T.; Burchfiel, B.; Tedrake, R.; Sadigh, D.; Levine, S.; Liang, P.; and Finn, C. 2024. OpenVLA: An Open-Source Vision-Language-Action Model. In Conference on Robot Learning.arxiv.org/abs/2406.09246
- [26]Kingma, D. P.; and Ba, J. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.arxiv.org/abs/1412.6980
- [27]Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Dollar, P.; and Girshick, R. 2023. Segment Anything. In IEEE/CVF International Conference on Computer Vision, 4015–4026.arxiv.org/abs/2304.02643
- [28]Laine, S.; Hellsten, J.; Karras, T.; Seol, Y.; Lehtinen, J.; and Aila, T. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics, 39(6): 1–14.arxiv.org/abs/2011.03277
- [29]Lee, T. E.; Tremblay, J.; To, T.; Cheng, J.; Mosier, T.; Kroemer, O.; Fox, D.; and Birchfield, S. 2020. Camera-to-Robot Pose Estimation from a Single Image. In IEEE International Conference on Robotics and Automation, 9426–9432. IEEE.arxiv.org/abs/1911.09231
- [32]Liu, R.; Canberk, A.; Song, S.; and Vondrick, C. 2025a. Differentiable Robot Rendering. In Proceedings of the 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, 117–129.arxiv.org/abs/2410.13851
- [33]Liu, S.; Wu, L.; Li, B.; Tan, H.; Chen, H.; Wang, Z.; Xu, K.; Su, H.; and Zhu, J. 2025b. Rtd-1b: A Diffusion Foundation Model for Bimanual Manipulation. In International Conference on Learning Representations, 29982–30009.arxiv.org/abs/2410.07864
- [34]Loper, M. M.; and Black, M. J. 2014. OpenDR: An Approximate Differentiable Renderer. In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., European Conference on Computer Vision, volume 8695, 154–169. Cham: Springer International Publishing. ISBN 978-3-319-10583-3.DOI
- [36]Qin, T.; Li, P.; and Shen, S. 2018. Vins-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator. IEEE transactions on robotics, 34(4): 1004–1020.DOI
- [37]Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; Mintun, E.; Pan, J.; Alwala, K. V.; Carion, N.; Wu, C.-Y.; Girshick, R.; Dollar, P.; and Feichtenhofer, C. 2025. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations, volume 2025, 28085–28128.arxiv.org/abs/2408.00714
- [38]Ren, T.; Liu, S.; Zeng, A.; Lin, J.; Li, K.; Cao, H.; Chen, J.; Huang, X.; Chen, Y.; Yan, F.; Zeng, Z.; Zhang, H.; Li, F.; Yang, J.; Li, H.; Jiang, Q.; and Zhang, L. 2024. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. arXiv:2401.14159.arxiv.org/abs/2401.14159
- [39]Shi, L.; Liu, Y.; Zeng, L.; Ai, B.; Hong, Z.; and Su, H. 2025. Learning Adaptive Dexterous Grasping from Single Demonstrations. In IEEE/RSJ International Conference on Intelligent Robots and Systems.arxiv.org/abs/2503.20208
- [40]Shiu, Y. C.; and Ahmad, S. 1989. Calibration of Wrist-Mounted Robotic Sensors by Solving Homogeneous Transform Equations of the Form AX=XB. IEEE Transactions on Robotics and Automation, 5(1): 16–29.DOI
- [41]Tsai, R. Y.; and Lenz, R. K. 1989. A New Technique for Fully Autonomous and Efficient 3 d Robotics Hand/Eye Calibration. IEEE Transactions on robotics and automation, 5(3): 345–358.DOI
- [42]Valassakis, E.; Drezckowski, K.; and Johns, E. 2022. Learning Eye-in-Hand Camera Calibration from a Single Image. In Conference on Robot Learning, 1336–1346. PMLR.arxiv.org/abs/2111.01245
- [43]Wang, J.; Hong, Z.; Bao, C.; Zhu, Y.; Sun, J.; and Zhang, G. 2026. LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation. arXiv:2604.08475.arxiv.org/abs/2604.08475
- [44]Wen, B.; Yang, W.; Kautz, J.; and Birchfield, S. 2024. Foundation-Pose: Unified 6D Pose Estimation and Tracking of Novel Objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17868–17879.arxiv.org/abs/2312.08344
- [46]Xiang, F.; Qin, Y.; Mo, K.; Xia, Y.; Zhu, H.; Liu, F.; Liu, M.; Jiang, H.; Yuan, Y.; Wang, H.; Yi, L.; Chang, A. X.; Guibas, L. J.; and Su, H. 2020. SAPIEN: A Simulated Part-Based Interactive ENvironment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11097–11107.arxiv.org/abs/2003.08515
- [47]Yan, G.; Zhu, J.; Deng, Y.; Yang, S.; Qiu, R.-Z.; Cheng, X.; Memmel, M.; Krishna, R.; Goyal, A.; Wang, X.; and Fox, D. 2025. ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training. arXiv:2509.01819.arxiv.org/abs/2509.01819
- [48]Yang, J.; Rebello, J.; and Waslander, S. L. 2023. Next-Best-View Selection for Robot Eye-in-Hand Calibration. In Conference on Robots and Vision, 161–168. IEEE.arxiv.org/abs/2303.06766
- [49]Yang, L.; Kang, B.; Huang, Z.; Zhao, Z.; Xu, X.; Feng, J.; and Zhao, H. 2024. Depth Anything V2. In Advances in Neural Information Processing Systems, volume 37, 21875–21911. Curran Associates, Inc.arxiv.org/abs/2406.09414
- [50]Yang, P.; Jiang, M.; Shen, X.; and Zhang, H. 2025. Spatiotemporal Calibration for Laser Vision Sensor in Hand-Eye System Based on Straight-Line Constraint. IEEE Robotics and Automation Letters, 10(12): 13320–13327.arxiv.org/abs/2509.12928
- [52]Zhang, X.; Xi, Y.; Huang, Z.; Zheng, L.; Huang, H.; Xiong, Y.; and Xu, K. 2023. Active Hand-Eye Calibration via Online Accuracy-Driven next-Best-View Selection. The Visual Computer, 39(1): 381–391.DOI
- [53]Zhao, T. Z.; Kumar, V.; Levine, S.; and Finn, C. 2023. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Robotics: Science and Systems.arxiv.org/abs/2304.13705
- [54]Zhu, C.; Yu, R.; Feng, S.; Burchfiel, B.; Shah, P.; and Gupta, A. 2025. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. arXiv:2504.02792.arxiv.org/abs/2504.02792

