betaFeedback

WorksComputer Science

SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Abstract

Markerless hand-eye calibration under continuous motion requires jointly resolving camera geometry and the time offset between images and joint states. We present SyncHeC (Synchronized Hand-Eye Calibration), a dense differentiable-rendering framework that estimates both from a single robot trajectory. Continuous-time forward kinematics makes latency differentiable, metric depth complements mask contours, and an offline planner combines spatial sensitivity with temporal excitation. Alternating updates support Eye-to-Hand, Eye-in-Hand, and multi-camera calibration. On a 7-DoF arm, one 6.56.5,s motion yields 0.210.21,cm translation error against a marker-board reference, 0.340.34,cm touch-test error, and 2.72.7,ms latency error relative to an 8787,ms reference with 95%95\% uncertainty of ±1.5\pm1.5,ms. Against CM-Calib on matched recordings, translation and nominal latency errors improve by 3.7×3.7\times and 2.3×2.3\times. Experiments span seven arms and four depth sensors. Compensating the recovered offset during policy training and deployment increases success by 8.18.1 percentage points across six families and six tasks (1,8001{,}800 matched pairs; paired 95%95\% confidence interval [6.3,9.9][6.3,9.9] points).

Code: github.com/hongzhengdong/SyncHeC

Introduction

Visual robot control depends on two correspondences (Figure 1). Spatial calibration maps camera observations into the robot’s kinematic frame ([41, 40, 5, 17]). Temporal calibration associates each observation with the joint state it depicts ((Wu et al., 2020)). Exposure, transport, processing, and clock offsets separate a frame’s recorded timestamp from its physical observation time. During motion, this mismatch changes the robot geometry associated with an image and couples a timing error to the estimated extrinsics.

Figure 1. Robotic hand-eye coordination has two coupled facets. Left: spatial pose calibration recovers the camera pose in the robot frame, in both Eye-to-Hand and Eye-in-Hand setups. Right: temporal calibration recovers the sensor-to-control latency Δt\Delta t: a frame time-stamped at t0+Δtt_0+\Delta t actually observes the joint state at t0t_0. We solve both jointly.

The same correspondence matters for learned control. Vision-language-action models ([1, 25, 33]) and 3D policies such as R3D ([11]) consume visual and proprioceptive information to predict actions. A calibration layer that aligns these streams provides geometrically and temporally consistent training tuples and deployment observations across policy architectures.

Marker-based methods and active-vision extensions estimate extrinsics from target observations ([52]).

Markerless methods use robot keypoints or direct pose regression ([29, 42]) or align a rendered robot model with image masks ([3, 13]). These approaches enable automatic spatial calibration from static poses. CM-Calib ((Wu et al., 2020)) jointly estimates pose and latency under continuous motion through target-derived correspondences. SyncHeC combines continuous-motion calibration with dense, markerless supervision (Table 1).

MethodMarker-freeEiHLatencyHetero.Dyn.
Marker-based [41]✗✓✗✓✗
Pose networks [29, 42]✓✓✗✗✗
EasyHeC [3]✓✗✗✓✗
EasyHeC++ [13]✓✓✗✓✗
CM-Calib (Wu et al., 2020)✗✓✓✓✓
SyncHeC✓✓✓✓✓

Table 1. Capability comparison of representative hand-eye calibration paradigms. Marker-free, EiH (Eye-in-Hand support), Latency (estimates Δt\Delta t), Hetero. (heterogeneous arms), Dyn. (calibrates during continuous motion).

SyncHeC integrates three components. First, interpolating the high-rate joint log creates a continuous-time kinematic model through which gradients reach the unknown latency. Second, metric RGB-D depth supplements mask contours with direct surface-distance constraints, strengthening optical-axis pose estimation. Third, a planned trajectory supplies spatially diverse views and sustained active motion for joint estimation. An alternating optimizer updates the temporal and spatial variables in turn.

SyncHeC supports both Eye-to-Hand (EtH) and Eye-in-Hand (EiH) cameras, and extends to multiple cameras through a common kinematic model. One short acquisition replaces repeated stop-and-go motion. We evaluate spatial accuracy, timing accuracy, conditioning, depth processing, and hardware transfer, then test the value of the recovered offset through matched downstream manipulation trials.

Related Work

Spatial hand-eye calibration. Classical solvers estimate AX=XBAX = XB or AX=YBAX = YB from calibration targets and forward kinematics ([41, 40, 5, 17]). Active approaches select informative viewpoints ([52, 48]). Markerless networks predict keypoints or camera poses ([29, 42]). EasyHeC and EasyHeC++ optimize rendered robot silhouettes, with pretrained image models supporting automatic initialization ([3, 13]). Hydra registers RGB-D observations to the robot model ([16]). Our dense spatial objective adds a differentiable timing variable to robot-model alignment.

Differentiable rendering and dynamic reconstruction. Differentiable rasterization propagates image residuals to scene geometry [34, 22, 28], supporting inverse rendering [23, 24] and object pose estimation [44]. DRRobot differentiates a kinematics-aware robot appearance model with respect to joint controls [32]; RobotArena∞ uses differentiable rendering for camera-pose estimation in real-to-sim reconstruction [18]. Dynamic free-viewpoint reconstruction with a flying camera also couples camera calibration and articulated motion estimation [9]. Here, recorded robot kinematics provide a measured trajectory, and shifting its time argument makes sensor latency directly observable through rendered masks and depth.

Temporal calibration and informative motion. Camera–IMU timing is estimated jointly with motion in visual-inertial systems [36] (Li and Mourikis 2014), including continuous-time batch formulations [6]. Related methods address targetless LiDAR–IMU and LiDAR–camera calibration (Lv et al., 2020; Li et al., 2024). For arms, CM-Calib uses correspondence matching (Wu et al., 2020), while laser-vision methods exploit structured features [50]. SyncHeC estimates latency through dense mask and depth observations of the robot. Our planner combines spatial sensitivity with temporal excitation; this extends informative-view selection [19, 20] to a timed robot trajectory.

Geometry and dynamics for manipulation. Several complementary approaches organize manipulation around geometric or motion representations. R3D studies 3D policy encoders and training [11]; LAMP lifts image-editing cues into 3D inter-object transformations [43]. AdaDexGrasp uses single human demonstrations to guide grasp-skill learning [39], and object-centric motion priors transfer predicted human–object states to robot learning [12]. Particle-based world models learn interaction dynamics from human data and generate reference trajectories through model predictive control [10]. Our downstream study examines the observation-to-joint alignment used by image policies, point-cloud policies, and a particle-based dynamics controller.

Method

SyncHeC couples continuous-time robot kinematics with dense rendering losses to estimate an extrinsic transform and an image–joint offset. Figure 2 summarizes spatial initialization, trajectory acquisition, and alternating optimization.

Figure 2. Overview. Stage 1 imports the URDF into a simulator, sizes an adaptive sampling sphere from the first real depth frame, selects an initial pose by a 2D–3D fitness, and refines it by differentiable rendering to obtain T0T^{0}. Stage 2 executes one offline-planned continuous trajectory, parameterizes the high-frequency joint stream as continuous-time forward kinematics, and jointly optimizes the extrinsics and latency Δt\Delta t from a 2D–3D differentiable-rendering loss.

Observation Model

Let TabT_{ab} map coordinates from frame bb to frame aa. For EtH we estimate the base-to-camera transform Tcb∈SE(3)T_{cb} \in\mathrm{SE}(3); for EiH we estimate the flange-to-camera transform Tce∈SE(3)T_{ce} \in\mathrm{SE}(3). Their inverses map camera coordinates into the corresponding robot frame. RGB-D frames have timestamps {tic}i=1N\{t_i^c\}_{i=1}^{N}, and joint measurements form a denser log Q={(tjr,qj)}j=1M\mathcal{Q}=\{(t_j^r,\mathbf{q}_j)\}_{j=1}^{M}. The frame stamped tict_i^c observes the joint state at

τi=tic−Δt.(1)\tau_i=t_i^c-\Delta t. \tag*{(1)}

The constant offset Δt\Delta t includes sensing-to-availability delay and the difference between clock origins. It specifies image–joint pairing during acquisition, training, and deployment.

For link ll, FK(l)(q)\mathrm{FK}^{(l)}(\mathbf{q}) maps link coordinates into the robot base, and FK(e)\mathrm{FK}^{(e)} denotes flange kinematics [2]. The two camera models are

TclEtH(τi)=TcbFK(l)(q(τi)),(2)\mathbf{T}_{cl}^{\mathrm{EtH}}(\tau_i)=\mathbf{T}_{cb}\mathrm{FK}^{(l)}(\mathbf{q}(\tau_i)), \tag*{(2)}
TclEiH(τi)=Tce[FK(e)(q(τi))]−1FK(l)(q(τi)).(3)\mathbf{T}_{cl}^{\mathrm{EiH}}(\tau_i)=\mathbf{T}_{ce}\left[\mathrm{FK}^{(e)}(\mathbf{q}(\tau_i))\right]^{-1}\mathrm{FK}^{(l)}(\mathbf{q}(\tau_i)). \tag*{(3)}

Given link meshes, camera intrinsics, and a differentiable renderer, these transforms produce a mask and depth image for each observation. We parameterize the static extrinsic by ξ∈se(3)\boldsymbol{\xi}\in\mathfrak{se}(3) and estimate (ξ,Δt)(\boldsymbol{\xi},\Delta t) jointly.

Metric Depth and Spatial Initialization

Axial conditioning. Known-scale perspective geometry carries an optical-axis constraint: for u=fX/Zu=fX/Z, ∂u/∂Z=−fX/Z2\partial u/\partial Z=-fX/Z^2. Its sensitivity becomes small for distant or narrow projections, and mask contours provide limited support along this direction. Metric depth adds direct surface-distance residuals over the visible robot, strengthening axial conditioning (Figure 3).

Complementary geometric constraints

Figure 3. Complementary geometric constraints. Solid and dashed shapes show reference and axially shifted geometry. (a) Mask contours constrain axial translation through perspective, with weak sensitivity in distant or narrow projections. (b) Metric depth adds a direct surface-distance residual.

Sensor-aligned depth. We use the metric Hypersim checkpoint of Depth Anything V2 (ViT-L) [49]. Let DrawD_{\mathrm{raw}} be sensor depth and DpredD_{\mathrm{pred}} the predicted depth, both registered to the RGB grid. On pixels where both are valid, their median ratio sets a scale ss. We retain valid sensor measurements and fill holes with sDpredsD_{\mathrm{pred}}; nonpositive and nonfinite values are excluded. The resulting metric map DobsD_{\mathrm{obs}} and validity mask are shared by initialization and dynamic calibration. Appendix B specifies the fusion and supervision region.

Stage 1: spatial anchoring. A single static RGB-D frame initializes the dynamic pipeline. We load the robot URDF into SAPIEN [46], segment its base link with SAM2 [37], and take the median valid depth as a surface-distance anchor dsurfd_{\mathrm{surf}}. A side-profile correction estimates the base-center distance LL from this surface anchor (Appendix B). The camera-sampling sphere has radius Radaptive=L+ΔrR_{\mathrm{adaptive}}=L+\Delta r, with margin Δr=0.10\Delta r=0.10 m.

For each candidate extrinsic Ts\mathbf{T}_s, we render its mask and depth and evaluate

S(Ts)=IoU⁡(Ts)−λinit∣Ωs∣∑p∈Ωs∣Drender(p)(Ts)−Dobs(p)∣.(4)S(\mathbf{T}_s)=\operatorname{IoU}(\mathbf{T}_s)-\frac{\lambda_{\mathrm{init}}}{|\Omega_s|}\sum_{p\in\Omega_s}\left|D_{\mathrm{render}}^{(p)}(\mathbf{T}_s)-D_{\mathrm{obs}}^{(p)}\right|. \tag*{(4)}

where Ωs\Omega_s contains valid overlapping robot pixels. Mask-based rendering refinement of the best candidate produces T0\mathbf{T}^0. The five-view static benchmark additionally refines the extrinsic using the remaining static views.

Continuous-Time Joint Calibration

Differentiable latency. For τi∈[tkr,tk+1r)\tau_i\in[t_k^r,t_{k+1}^r), the synchronized joint state and its temporal derivative are

q(τi)=qk+tic−Δt−tkrtk+1r−tkr(qk+1−qk),(5)\mathbf{q}(\tau_i)=\mathbf{q}_k+\frac{t_i^c-\Delta t-t_k^r}{t_{k+1}^r-t_k^r}(\mathbf{q}_{k+1}-\mathbf{q}_k), \tag*{(5)}
∂q(τi)∂Δt=−qk+1−qktk+1r−tkr.(6)\frac{\partial\mathbf{q}(\tau_i)}{\partial\Delta t}=-\frac{\mathbf{q}_{k+1}-\mathbf{q}_k}{t_{k+1}^r-t_k^r}. \tag*{(6)}

The active interval is recomputed as Δt\Delta t changes. With a 250 Hz log, adjacent samples are 4 ms apart; bounded joint acceleration gives an O(h2)O(h^2) joint-value interpolation error. Substitution into Equations (2)–(3) propagates time gradients through forward kinematics and rendering.

Dense 2D–3D objective. SAM2 / Grounded-SAM provide observation masks [27, 37, 38]. For frame ii, the rasterizer outputs a soft mask MirendM_i^{\mathrm{rend}} and metric depth DirendD_i^{\mathrm{rend}}. We minimize

Lm,i=1HW∑p(Mirend,(p)−Miobs,(p))2,(7)\mathcal{L}_{m,i}=\frac{1}{HW}\sum_p\left(M_i^{\mathrm{rend},(p)}-M_i^{\mathrm{obs},(p)}\right)^2, \tag*{(7)}
Ld,i=1∣Ωi∣∑p∈Ωi∣Dirend,(p)−Diobs,(p)∣,(8)\mathcal{L}_{d,i}=\frac{1}{|\Omega_i|}\sum_{p\in\Omega_i}\left|D_i^{\mathrm{rend},(p)}-D_i^{\mathrm{obs},(p)}\right|, \tag*{(8)}
L(ξ,Δt)=1N∑i=1N(Lm,i+λDLd,i)+λregΔt2.(9)\mathcal{L}(\boldsymbol{\xi},\Delta t)=\frac{1}{N}\sum_{i=1}^{N}\left(\mathcal{L}_{m,i}+\lambda_D\mathcal{L}_{d,i}\right)+\lambda_{\mathrm{reg}}\Delta t^2. \tag*{(9)}

Here Ωi\Omega_i intersects the observed robot mask, rendered mask >0.5>0.5, and valid observed depth. The depth term is zero for an empty intersection. Depth is measured in meters and Δt\Delta t in seconds.

Joint observability. Stack the rendered observations across frames and differentiate with respect to the six pose parameters and time:

J=[Jξ Jt],Jt=−Jπq˙,(10)\mathbf{J}=[\mathbf{J}_\xi\ \mathbf{J}_t],\qquad\mathbf{J}_t=-\mathbf{J}_\pi\dot{\mathbf{q}}, \tag*{(10)}

where the expression for Jt\mathbf{J}_t is stacked framewise and Jπ\mathbf{J}_\pi includes the appropriate EtH or EiH kinematic chain. Local first-order identifiability requires rank⁡(J)=7\operatorname{rank}(\mathbf{J})=7 [14]. A temporal change becomes indistinguishable from a pose change when its stacked column lies in the span of the six pose columns. Exciting multiple configurations and velocities helps separate these effects. This criterion depends on the full projected motion across the sequence.

Alternating updates. Starting from T0\mathbf{T}^0 and Δt=0\Delta t=0, we alternate a temporal block with a spatial block. The temporal block fixes pose and updates Δt\Delta t using frames with ∥q˙(tic−Δt)∥2>0.05\lVert\dot{\mathbf{q}}(t_i^c-\Delta t)\rVert_2>0.05 rad/s. The spatial block resynchronizes all selected frames and updates pose. Both use Adam with persistent optimizer moments. Each block performs 15 passes over mini-batches of four frames. In the dynamic benchmark, all 60 selected frames pass the temporal gate in every cycle, giving 225 actual Adam updates per variable per cycle. Convergence takes 3–5 cycles, or 675–1,125 temporal updates. Appendix A gives the algorithm and a representative trace.

Excitation-Based Trajectory Planning

We optimize a short timed trajectory using spatial and temporal rendering sensitivities. For a candidate camera set Tcand\mathbb{T}_{\mathrm{cand}}, spatial excitation is

Vspace(q)=∑T∈TcandVar⁡p(π(p)(T,q)).(11)V_{\mathrm{space}}(\mathbf{q}) = \sum_{\mathbf{T}\in\mathbb{T}_{\mathrm{cand}}} \operatorname{Var}_{p}\left(\pi^{(p)}(\mathbf{T},\mathbf{q})\right). \tag*{(11)}

The rendered image π\pi uses the appropriate observation model in § 3.1. Temporal excitation is

Vtime(q,q˙)=∥Jπ(q)q˙∥F.(12)V_{\mathrm{time}}(\mathbf{q},\dot{\mathbf{q}}) = \lVert J_{\pi}(\mathbf{q})\dot{\mathbf{q}}\rVert_{F}. \tag*{(12)}

These terms measure pose sensitivity and the magnitude of the rendered observation’s temporal derivative. For Gaussian observation noise, squared sensitivity determines a Fisher-information block; our unsquared temporal score rewards motion over the sampled trajectory. Appendix § 8 derives this relationship and describes the joint conditioning diagnostic.

RRT* ([21]) supplies a collision-free geometric seed, followed by optimization of a timed KK-knot B-spline:

Q∗=arg⁡max⁡Q∑k=1K[αVspace(qk)+βVtime(qk,q˙k)].(13)\mathbf{Q}^{*} = \underset{\mathbf{Q}}{\arg\max}\sum_{k=1}^{K}\left[\alpha V_{\mathrm{space}}(\mathbf{q}_{k})+\beta V_{\mathrm{time}}(\mathbf{q}_{k},\dot{\mathbf{q}}_{k})\right]. \tag*{(13)}

The refinement enforces joint limits, collision avoidance, and base-link visibility. Planning uses the Stage-1 extrinsic. We normalize the two scores by fixed initial-reference scales sspace=2000s_{\mathrm{space}}=2000 and stime=400s_{\mathrm{time}}=400, giving α=1/sspace=5×10−4\alpha=1/s_{\mathrm{space}}=5\times10^{-4} and β=γ/stime=2.5×10−3\beta=\gamma/s_{\mathrm{time}}=2.5\times10^{-3}. The relative weight γ=1\gamma=1 is selected on the simulation split and held fixed for real experiments.

Multi-Camera Calibration

For several cameras, we sum their normalized rendering objectives, assign each camera its own extrinsic, and evaluate their observations through the shared joint log. The model admits a separate Δtj\Delta t_j per camera. When paired timing measurements establish equivalence within a specified tolerance, a shared-offset variant imposes Δt1=Δt2\Delta t_1 = \Delta t_2. The temporal block then accumulates evidence from both views, while each spatial block updates its own extrinsic. Our dual-camera experiment uses a ±1\pm1 ms equivalence criterion verified by an independent paired timing reference.

Experiments

Setup and Evaluation

The main platform is a 7-DoF xArm7 with environment-mounted EtH and wrist-mounted EiH RealSense D435i cameras. Both stream 848 ×\times 480 RGB-D at 30 Hz, and the controller records joints at 250 Hz. Rendering and optimization use Nvdiffrast (Laine et al., [28]) on an RTX 4090 with an i9-13900K. The planned 25-knot trajectory lasts approximately 6.5 s. We retain the 60 highest-flow frames from approximately 195 recorded frames.

Optimization settings. We set λD=0.1\lambda_D = 0.1, λreg=0.5\lambda_{\mathrm{reg}} = 0.5, and λinit=1.0\lambda_{\mathrm{init}} = 1.0. Adam (Kingma and Ba, [26]) uses learning rates 10−310^{-3} for rotation, 5×10−45 \times10^{-4} for translation, and 2×10−42 \times10^{-4} for time, with moments (0.9,0.999)(0.9, 0.999) and numerical epsilon 10−810^{-8}. Each temporal and spatial block uses 15 passes, batch size four, and persistent moments. At most 15 cycles give 3,375 temporal updates. We stop when cycle-to-cycle changes are below 0.05 mm in translation, 0.01° in rotation, and 0.5 ms in time. Loss weights are selected on the simulation split from λD∈{0.05,0.1,0.2,0.5}\lambda_D \in\{0.05, 0.1, 0.2, 0.5\} and λreg∈{0.1,0.5,1}\lambda_{\mathrm{reg}} \in\{0.1, 0.5, 1\}, then held fixed across real experiments. Varying λD\lambda_D within [0.05,0.2][0.05, 0.2] changes translation error by less than 0.03 cm.

Independent references. A signal-generator, LED, and oscilloscope protocol relates image exposure to the logged physical onset of an arm micro-motion. Exposure-phase, clock-mapping, and physical-onset measurements bound residual biases by 0.25, 0.10, and 0.20 ms, respectively; the phase budget includes RGB–depth timestamp skew. Twenty repetitions give an 87 ms nominal reference with approximately 2 ms single-repeat spread and a 0.95 ms confidence half-width for the mean. Combining that interval with the three residual bounds gives 95% reference uncertainty U95=1.5U_{95} = 1.5 ms (Appendix § 9). Spatial references are obtained independently from repeated marker-board measurements and a workspace touch test.

Metrics and comparisons. We report translation error EtE_t in cm, geodesic rotation error ErE_r in degrees, and nominal latency error Elat=∣Δt∗−Δtref∣E_{\mathrm{lat}} = |\Delta t^* - \Delta t_{\mathrm{ref}}| in ms. Marker-board pose error measures extrinsic agreement; the touch test additionally includes probe-offset and robot-positioning errors. Spatial reference agreement is approximately 1 mm.

Static baseline results in Table 2–Table 3 follow the simulation and touch protocols reported by EasyHeC++ (Hong, Zheng, and Chen, [13]); SynchHeC reruns those protocols with Δt=0\Delta t = 0. We also run Hydra under the matched static touch protocol. On dynamic data, CM-Calib and SL-Calib use the same planned trajectory, recordings, and joint logs as SynchHeC, with initialization matched where applicable. CM-Calib receives target-derived poses and forward kinematics. SL-Calib is our straight-line-motion implementation of the correspondence-based approach. Appendix § 9.§ 3 records comparison provenance and controls.

Method#vErE_r (°)↓EtE_t (cm)↓
Eye-to-Hand
Marker-based (Tsai and Lenz, [41])200.8702.000
DREAM (Lee et al., [29])50.7040.303
EasyHeC (Chen et al., [3])50.0810.206
EasyHeC++ (Hong, Zheng, and Chen, [13])50.0450.135
SynchHeC (Δt=0\Delta t=0)50.0380.105
Eye-in-Hand
Zhang et al. [52]100.1480.315
Valasskis, Dreczykowski, and Johns (2022)14.4001.340
EasyHeC++ (Hong, Zheng, and Chen, [13])50.1170.112
SynchHeC (Δt=0\Delta t=0)50.0950.086

Table 2. Simulation calibration accuracy. Top: Eye-to-Hand. Bottom: Eye-in-Hand. #v: number of views. Baseline values are reproduced from the EasyHeC++ evaluation (Hong, Zheng, and Chen, [13]).

MethodEtHEiH
DREAM (Lee et al., [29])1.50—
Zhang et al. [52]—1.35
Valasskis, Dreczykowski, and Johns (2022)—4.30
EasyHeC (Chen et al., [3])0.40—
EasyHeC++ (Hong, Zheng, and Chen, [13])0.300.31
Hydra (Huber et al., [16])0.350.28
SynchHeC (Δt=0\Delta t = 0)0.280.16

Table 3. Real static touch-test error (cm), averaged over ten poses. Hydra and SynchHeC are evaluated here; the other results are reproduced from Hong, Zheng, and Chen [13]. Protocols are specified in Appendix § 9.§ 3.

Static Spatial Accuracy

In simulation (Table 2), our translation errors are 0.105 cm for EtH and 0.086 cm for EiH, compared with 0.135 and 0.112 cm for EasyHeC++. On the real touch test (Table 3), EtH performance is comparable across SynchHeC, EasyHeC++, and Hydra at the reference resolution. EiH error decreases from 0.31 cm for EasyHeC++ and 0.28 cm for Hydra to 0.16 cm. Metric depth and the surface-to-base correction are particularly useful in the constrained wrist-camera view.

Dynamic Spatio-Temporal Accuracy

Table 4 evaluates continuous-motion calibration. Across five matched recordings, SyncHeC reaches 0.21 ± 0.04 cm / 0.15 ± 0.05° / 2.7 ± 0.6 ms for EtH, and 0.24 ± 0.05 cm / 0.18 ± 0.07° / 5.4 ± 0.9 ms for EiH. CM-Calib gives 0.78 ± 0.08 and 0.95 ± 0.10 cm translation error, with latency errors 6.2 ± 0.8 and 8.5 ± 1.1 ms. The EtH ratios are 3.7× for translation and 2.3× for nominal latency error.

MethodEtE_t (cm)↓ErE_r (°)↓ElatE_{\mathrm{lat}} (ms)↓
Eye-to-Hand
SL-Calib‡1.150.9211.5
CM-Calib (Wu et al., 2020)0.78 ± 0.080.556.2 ± 0.8
EasyHeC† [3]4.853.42—
EasyHeC++† [13]3.922.75—
SyncHeC0.21 ± 0.040.15 ± 0.052.7 ± 0.6
Eye-in-Hand
SL-Calib‡1.341.1514.8
CM-Calib (Wu et al., 2020)0.95 ± 0.100.728.5 ± 1.1
EasyHeC† [3]5.684.15—
EasyHeC++† [13]4.553.38—
SyncHeC0.24 ± 0.050.18 ± 0.075.4 ± 0.9

Table 4. Continuous-motion calibration against the 87 ms reference (U95=1.5U_{95} = 1.5 ms). CM-Calib and ours use five matched recordings; SDs are shown for CM translation/latency and all our metrics.

The corresponding mean offset estimates are 84.3/81.6 ms for SyncHeC and 80.8/78.5 ms for CM-Calib (EtH/EiH). Our latency error remains smaller throughout the timing reference’s ±1.5 ms uncertainty interval. For EtH, the CM-Calib-to-SyncHeC error ratio over this interval is [1.83, 3.92] (Appendix § 9).

EasyHeC and EasyHeC++ applied directly to the moving sequence associate each frame with the unshifted joint state, giving EtH errors of 4.85 and 3.92 cm. A fixed-reference timing control synchronizes the inputs at 87 ms before spatial optimization: EasyHeC++ then reaches 0.43/0.48 cm and the SyncHeC spatial objective reaches 0.20/0.23 cm (EtH/EiH). Joint estimation reaches essentially the same spatial accuracy as the fixed-reference version of SyncHeC. This control separates the benefit of temporal alignment from that of dense metric-depth supervision.

Optimization behavior. A representative EtH run reaches 84.3 ms from zero in 900 temporal updates (Appendix § 6). Its data and regularizer gradients are approximately −0.0843 and +0.0843 s⁻¹. Continuing alternation for 900 additional temporal updates with stopping disabled changes the final offset by less than 0.2 ms, characterizing the local stationary solution of the regularized objective.

Efficiency. The dynamic pipeline uses one 6.5 s acquisition and approximately eight minutes of total wall-clock time. On the same EtH task, EasyHeC takes approximately 32 minutes, EasyHeC++ 12 minutes, and our static mode 13 minutes. Dynamic touch-test error is 0.34 cm, compared with 0.28 cm for our static mode and 0.30 cm for EasyHeC++. The dynamic board-reference translation error is 0.21 cm.

Multi-Camera and Hardware Transfer

For the dual-camera rig (Figure 4), independent paired timing measurements give an EtH–EiH offset difference of 0.2 ms with a 95% interval of [−0.4, 0.8] ms, contained within the ±1 ms sharing criterion. This supports the shared-offset variant of Section 3.5. On four matched dual-view recordings, drawn from the five-run benchmark, it obtains 0.45 ± 0.27 cm / 0.63 ± 0.50° for EtH and 0.59 ± 0.22 cm / 1.24 ± 0.73° for EiH. All four runs converge. The shared fit uses both views in one calibration, while the separate fits attain the single-camera accuracy in Table 4.

Heterogeneous multi-camera co-calibration

Figure 4. Heterogeneous multi-camera co-calibration. Left: the rig, with the tripod-mounted Eye-to-Hand and wrist-mounted Eye-in-Hand cameras labelled. Right: frames from each camera over one continuous trajectory.

Table 5 shows representative runs across seven arms and four depth sensors. The xArm7/D435i configuration is the shared reference, yielding ten distinct hardware configurations. Five independent motions per configuration give 50 calibration runs. The representative arm runs span 0.24–0.68 cm translation error and 1.2–4.8 ms latency error. Across the four sensors, reference offsets range from 66 to 115 ms and estimated offsets lie within 3 ms of their references. The more difficult Z1 Pro and Azure Kinect configurations give five-run translation errors of 0.67 ± 0.09 and 0.74 ± 0.08 cm, respectively. The table’s xArm7 entry is a separate EtH run from the five-run mean in Table 4.

PlatformEtE_t (cm)ErE_r (°)Δt∗\Delta t^{*}Δtref\Delta t_{\mathrm{ref}}
xArm70.250.1885.687.0
Franka Panda0.310.2286.888.5
RM75-B0.280.1986.187.5
Z1 Pro0.680.4584.289.0
EC660.360.2586.588.0
ER70.240.1687.388.5
CR50.420.3185.287.0
D435i (Stereo)0.250.1885.687.0
L515 (LiDAR)0.180.13112.4115.0
Femto Bolt (ToF)0.210.1564.566.0
Azure Kinect (ToF)0.750.5292.194.0

Table 5. Representative runs across arms (top) and depth sensors (bottom). xArm7/D435i is repeated as the shared reference, giving ten distinct configurations. Each configuration has five motions; repeat summaries appear in the text. Offsets are in ms.

Controlled Ablations

Metric depth and completion. We hold frames, masks, Stage-1 anchor, and optimizer fixed across three Stage-2 variants. Raw sensor depth gives translation errors of 0.38/0.44 cm; sensor-aligned completion gives 0.21/0.24 cm; and disabling the Stage-2 depth term gives 1.18/1.35 cm (EtH/EiH), averaged over five matched runs. Thus raw metric depth already supplies a useful constraint, and completing missing surface observations further improves pose estimation. Table 6 also shows the associated latency improvement.

Variant\multicolumn{3}{c}{EtH}\multicolumn{3}{c}{EiH}
EtE_tErE_rElatE_{\mathrm{lat}}EtE_tErE_rElatE_{\mathrm{lat}}
Stage-2 depth off1.180.853.51.350.968.2
Spatial planning only0.860.6516.40.950.7218.2
Full (Ours)0.210.152.70.240.185.4

Table 6. Dynamic ablations with the Stage-1 anchor fixed. EtE_t is in cm, ErE_r in degrees, and ElatE_{\mathrm{lat}} in ms. The full row repeats the five-run means in Table 4.

Temporal excitation and joint conditioning. With β=0\beta= 0, latency errors increase to 16.4/18.2 ms. For the stacked seven-column observation Jacobian, after consistent channel scaling and unit-column normalization, the minimum singular values are 0.21/0.18 for the full planner and 0.03/0.02 for spatial-only planning. Both trajectories yield full-rank Jacobians; temporal excitation improves the conditioning of joint pose–time estimation.

Alternation and interpolation. Joint Adam with the same per-variable learning rates, batch size, and actual total number of gradient evaluations reaches approximately 0.5 cm translation and 6 ms latency error over five runs. Alternation reaches the accuracy in Table 4 in 3–5 cycles. On the same 250 Hz joint logs, replacing the linear interpolant with a cubic interpolant changes the estimated translation vector by less than 0.01 cm and latency by less than 0.2 ms.

Downstream Visuomotor Policies

We evaluate six policy families on six real tasks (Table 7): ACT (Zhao et al., [53]), Diffusion Policy (DP) (Chi et al., [4]), π0\pi_0 (Black et al., [1]), DP3 (Ze et al., 2024), ManiFlow (Yan et al., [47]), and a particle-based world model with sampling-based MPC (WM). Particle dynamics and learned-model control provide the basis for the latter family (Hong et al., [10]; Hafner et al., [7]; 2025; Zhu et al., [54]; Huang et al., [15]).

PolicyStackCupBan.KettleDrawerTowel
ACT (Zhao et al., [53])6→86\to844→5044\to5040→4440\to4442→4642\to4638→4238\to4234→4034\to40
DP (Chi et al., [4])12→1612\to1658→6458\to6448→5648\to5664→6864\to6846→5446\to5436→5236\to52
π0\pi_0 (Black et al., [1])16→2216\to2254→6654\to6650→6250\to6246→6246\to6248→6048\to6062→7462\to74
DP3 (Ze et al., 2024)2→42\to412→1612\to1642→4842\to4840→4840\to4828→3228\to3254→6654\to66
ManiFlow (Yan et al., [47])10→1410\to1452→6052\to6046→5446\to5436→5036\to5046→5446\to5460→6460\to64
WM†8→148\to1440→5440\to5438→5038\to5034→4834\to4840→5240\to5230→4030\to40

Table 7. Success rate (%) without → with recovered-offset compensation. Each cell uses 50 matched pairs on the six tasks: stack three blocks; place cup, banana, or kettle; open drawer; fold towel. Across 1800 pairs, the gain is 8.11 points (paired 95% CI [6.3,9.9]).

Each task uses the same 50 demonstrations for both conditions. We fix extrinsics, use one matched training seed per policy–task cell, and replay the same 50 initial configurations in each pair of evaluations. Without compensation, an image is paired with the nearest joint timestamp. With compensation, it is paired with q(tc−Δt∗)\mathbf{q}(t_c-\Delta t^{*}), using our recovered offset at both training and deployment. Action and chunk timestamps remain fixed.

Across 36 cells, 1,800 matched pairs comprise 3,600 physical executions. Successful trials increase from 681 to 827. Compensation uniquely succeeds in 210 pairs and the uncompensated policy uniquely succeeds in 64, giving 146 net additional successes and an 8.11-percentage-point gain. A cell-stratified paired Wald interval is [6.3, 9.9] percentage points at 95% confidence, conditional on the trained policies and calibration. Appendix § 10 gives task definitions, pairing, and the variance calculation.

Mean gains are 4.3 points for ACT, 7.7 for DP, 11.7 for π0\pi_0, 6.0 for DP3, 7.7 for ManiFlow, and 11.3 for WM. Every policy–task cell has a positive observed difference. The aggregate improvement spans image-based, point-cloud, and dynamics-based control, demonstrating the practical value of aligning the observation and joint streams before policy learning and execution.

Operating Conditions

The pipeline uses a coarse spatial anchor within roughly 15 cm, visible robot geometry, and sufficiently excited motion. Strong blur becomes prominent above approximately 0.5 m/s in the evaluated imaging setup; acquisition speed should preserve reliable mask boundaries. Pixels without valid depth after completion contribute through the mask channel. A constant offset describes a stable sensing and control stack; changes in the bus, driver, or host state that produce drifts exceeding 20 ms call for recalibration. These conditions guide acquisition and deployment.

Conclusion

SyncHeC combines continuous-time kinematics, metric depth, and planned excitation to jointly estimate hand-eye pose and latency from a single continuous robot motion. It supports EtH, EiH, and heterogeneous hardware. Applying the recovered offset during policy training and deployment raises real manipulation success by 8.1 points across six families.

Optimization and Implementation

Update Accounting and Stopping

Algorithm 1 specifies the alternating optimizer. A pass visits every mini-batch in the active frame set once. With N=60N = 60, batch size B=4B = 4, and P=15P = 15 passes, each block executes

P⌈N/B⌉=15×15=225(14)P\lceil N/B\rceil= 15 \times15 = 225 \tag*{(14)}

Adam updates. There is one optimizer step per batch and no gradient accumulation. The temporal gate is evaluated at the current synchronized times at the start of each cycle; all 60 selected benchmark frames pass it. Each cycle therefore uses 225 temporal and 225 spatial gradient evaluations. Three to five cycles correspond to 675–1,125 temporal updates and 1,350–2,250 total gradient evaluations; the maximum of 15 cycles allows 3,375 temporal updates.

For a mini-batch B\mathcal{B}, the optimized objective is the mean frame data loss plus one regularization term:

LB=1∣B∣∑i∈B(Lm,i+λDLd,i)+λregΔt2.(15)\mathcal{L}_{\mathcal{B}} = \frac{1}{|\mathcal{B}|} \sum_{i\in\mathcal{B}} \left(\mathcal{L}_{m,i} + \lambda_D\mathcal{L}_{d,i}\right) + \lambda_{\mathrm{reg}}\Delta t^2. \tag*{(15)}

This keeps the penalty’s scale consistent with Equation (9). Time is stored in seconds, translation in meters, and rotation updates use the Lie-algebra representation. Separate optimizers retain their moments across passes and alternating cycles.

Representative Trace and Regularization

Table 8 records one EtH run from the dynamic benchmark. The pose is initialized by Stage 1 and the temporal variable starts at zero. Its final offset, 84.3 ms, is also close to the five-run mean.

Temporal updates0225450675900
Offset (ms)043.878.184.084.3

Table 8. Representative EtH optimization trace. The count is the cumulative number of temporal optimizer updates.

Writing the mean data objective as Ldata\mathcal{L}_{\mathrm{data}}, the temporal stationarity condition is

∂Ldata∂Δt+2λregΔt=0.(16)\frac{\partial\mathcal{L}_{\mathrm{data}}}{\partial\Delta t} + 2\lambda_{\mathrm{reg}}\Delta t = 0. \tag*{(16)}

At Δt=0.0843 s\Delta t = 0.0843\ \mathrm{s} and λreg=0.5\lambda_{\mathrm{reg}} = 0.5, the penalty gradient is +0.0843 s−1+0.0843\ \mathrm{s}^{-1}, balancing the observed data gradient of approximately −0.0843 s−1-0.0843\ \mathrm{s}^{-1}. Retaining optimizer states and continuing the original alternating sequence for 900 further temporal updates, together with the associated spatial blocks, changes the final offset by less than 0.2 ms with stopping disabled. The penalty’s influence is governed by the local data gradient and curvature.

Algorithm 1: Alternating pose and latency optimization

Input: selected frames, joint log, mesh, intrinsics, T0\mathbf{T}^{0}, P=15P = 15, B=4B = 4, Kmax⁡=15K_{\max} = 15.

  1. Initialize ξ=log⁡T0\xi= \log\mathbf{T}^{0}, Δt=0 s\Delta t = 0\ \mathrm{s}, and the spatial and temporal Adam states.

  2. For each cycle k=1,…,Kmax⁡k = 1,\ldots,K_{\max}: (a) Save the current extrinsic and offset for the stopping test. Select S={i:∥q˙(tic−Δt)∥2>0.05 rad/s}\mathcal{S} = \{i : \lVert\dot{\mathbf{q}}(t_i^c - \Delta t)\rVert_2 > 0.05\ \mathrm{rad/s}\}. (b) Fix ξ\xi. For each of PP passes over S\mathcal{S}, form batches of BB frames, recompute active interpolation intervals, evaluate LB\mathcal{L}_{\mathcal{B}}, and take one Adam update of Δt\Delta t per batch. (c) Fix Δt\Delta t. Resynchronize all NN frames. For each of PP passes, form batches of BB frames and take one Adam update of ξ\xi per batch. (d) Stop if all cycle-to-cycle changes satisfy: translation <0.05 mm< 0.05\ \mathrm{mm}, rotation <0.01∘< 0.01^\circ, and time <0.5 ms< 0.5\ \mathrm{ms}.

  3. Return T∗=exp⁡(ξ)\mathbf{T}^{*} = \exp(\xi) and Δt∗\Delta t^{*}.

Software Implementation

SyncHeC offers Nvdiffrast rendering on CUDA and soft rendering in PyTorch. Both use the same continuous-time kinematics, per-frame normalized losses, and mini-batch alternating optimizer. Configuration files specify batch size, passes per block, loss weights, and learning rates; each run records the selected frames, recovered transform, offset, and actual update counts.

Recordings supply camera intrinsics, time-stamped joints, robot masks, metric depth, and URDF geometry. The depth-fusion function preserves valid sensor measurements and completes holes using sensor-scaled predictions. Spatial initialization uses a separate base-link mask and the surface-to-center correction. EtH and EiH use the transforms in Equations (2)–(3); multi-camera calibration supports camera-specific or shared offsets.

The acquisition pipeline uses rendered-pixel trajectory scoring with RRT* and collision checks. The software also provides a lightweight B-spline refinement interface based on projected URDF landmarks, with collision and visibility callbacks. Table 9 maps the calibration components to their implementation.

ComponentFunction
Calibration objectiveoptimize.CalibrationProblem
Joint interpolationkinematics.q_sync
Temporal derivativekinematics.temporal_jacobian
EtH / EiH transformskinematics.LinkPoseModel
Mask / depth lossesrender.losses
Initialization fitnessstage1.joint_fitness
Alternating optimizeroptimize.calibrate
Joint optimizeroptimize.calibrate_joint

Table 9. Core mathematical components in SyncHeC.

The executable synthetic example generates a four-link robot recording with a known 87 ms offset, initializes pose by perturbing the reference extrinsic, and starts the temporal variable at zero. Its default configuration uses 24 frames at 64 × 48 resolution, λreg=0\lambda_{\mathrm{reg}} = 0, temporal learning rate 0.001, and five passes over batches of four frames. Each block therefore performs 30 updates per cycle. The output reports pose and timing errors against the generating parameters together with the complete optimization history.

Metric Depth and Anchoring

Metric Fusion and Validity

Let Vs\mathcal{V}_s contain valid raw sensor pixels and Vp\mathcal{V}_p contain finite, positive Depth Anything V2 predictions. Using the ViT-L metric Hypersim checkpoint, we compute

s=median⁡p∈Vs∩VpDraw(p)Dpred(p)(17)s = \operatorname{median}_{p \in\mathcal{V}_s \cap\mathcal{V}_p} \frac{D_{\mathrm{raw}}^{(p)}}{D_{\mathrm{pred}}^{(p)}} \tag*{(17)}

With overlap available for this scale estimate, the observation depth is

Dobs(p)={Draw(p),p∈Vs,sDpred(p),p∈Vp∖Vs.(18)D_{\mathrm{obs}}^{(p)} = \begin{cases} D_{\mathrm{raw}}^{(p)}, & p \in\mathcal{V}_s, \\ sD_{\mathrm{pred}}^{(p)}, & p \in\mathcal{V}_p \setminus\mathcal{V}_s. \end{cases} \tag*{(18)}

Pixels without a valid retained measurement or scaled prediction remain invalid. The per-frame depth-supervision region is

Ωi={p:Mirend,(p)>0.5, Miobs,(p)=1, p∈Vobs,i}.(19)\Omega_i = \{p : M_i^{\mathrm{rend},(p)} > 0.5,\ M_i^{\mathrm{obs},(p)} = 1,\ p \in\mathcal{V}_{\mathrm{obs},i}\}. \tag*{(19)}

Mask residuals are evaluated across the full image. The scale is anchored to sensor measurements, and valid sensor depth is preserved throughout the fusion.

The raw-depth control uses Vs\mathcal{V}_s for its Stage-2 depth residual. The completed-depth control uses Vobs\mathcal{V}_{\mathrm{obs}}. The λD=0\lambda_D = 0 control switches off this Stage-2 residual. All three use the same completed-depth Stage-1 anchor, frames, masks, and optimizer, isolating the contribution of Stage-2 depth supervision and completion.

Surface-to-Base Correction

The median depth over valid segmented base-link pixels yields dsurfd_{\mathrm{surf}}. Under the base-centered viewing approximation used for candidate sampling, this gives a surface-distance estimate. The side-profile model relates this estimate, camera height HH relative to the base origin, and base cylinder radius rbaser_{\mathrm{base}} to center distance LL:

L=dsurf+rbaseLL2−H2.(20)L = d_{\mathrm{surf}} + \frac{r_{\mathrm{base}}L}{\sqrt{L^2-H^2}}. \tag*{(20)}

It describes the visible cylindrical side in the regime dsurf>∣H∣d_{\mathrm{surf}} > \lvert H\rvert and L≫rbaseL \gg r_{\mathrm{base}}. A first-order correction evaluates the right-hand side at L0=dsurfL_0 = d_{\mathrm{surf}}, then sets the sampling radius to $L + Δr\Delta r.

Excitation and Joint Conditioning

Sensitivity Interpretation

For a differentiable observation vector y(θ)\mathbf{y}(\theta) with independent Gaussian noise of variance σ2\sigma^2, the Fisher matrix is

F=1σ2JTJ,J=∂y∂θ.(21)\mathbf{F} = \frac{1}{\sigma^2}\mathbf{J}^{\mathsf{T}}\mathbf{J}, \qquad\mathbf{J} = \frac{\partial\mathbf{y}}{\partial\theta}. \tag*{(21)}

For small camera-pose perturbations with covariance Cξ\mathbf{C}_{\xi}, linearization gives

Vspace≈tr⁡(JξCξJξT).(22)V_{\mathrm{space}} \approx\operatorname{tr}(\mathbf{J}_{\xi}\mathbf{C}_{\xi}\mathbf{J}_{\xi}^{\mathsf{T}}). \tag*{(22)}

For latency, the corresponding per-frame information is

Ftt=∥Jτq˙∥F2σ2=Vtime2σ2.(23)F_{tt} = \frac{\lVert\mathbf{J}_{\tau}\dot{\mathbf{q}}\rVert_F^2}{\sigma^2} = \frac{V_{\mathrm{time}}^2}{\sigma^2}. \tag*{(23)}

The planner uses ∑kVtime\sum_k V_{\mathrm{time}} alongside the spatial variance surrogate. The un squared temporal form reduces the relative reward for isolated high-amplitude peaks compared with ∑kVtime2\sum_k V_{\mathrm{time}}^2. Fixed reference scales make the relative weight γ\gamma dimensionless and comparable across planning iterations. The joint Jacobian diagnostic below evaluates pose–time coupling after planning.

Seven-Parameter Diagnostic

For each EtH or EiH sequence, we stack the six pose derivatives and one temporal derivative of the mask and valid-depth observation channels. The diagnostic uses consistent channel weights, pixel-count normalization, and sampling rules across variants. It differentiates the data observations, excluding the latency penalty. Let Jw\mathbf{J}_w be the scaled matrix and

J~=Jwdiag⁡(∥Jw,:,1∥2−1,…,∥Jw,:,7∥2−1).(24)\widetilde{\mathbf{J}} = \mathbf{J}_w \operatorname{diag}\left(\lVert\mathbf{J}_{w,:,1}\rVert_2^{-1}, \ldots, \lVert\mathbf{J}_{w,:,7}\rVert_2^{-1}\right). \tag*{(24)}

Unit-column normalization makes the singular-value comparison independent of the units used for translation, rotation, and time.

All four matrices in Table 10 are full rank. Because seven columns have unit norm, ∥J~∥F2=7\lVert\widetilde{\mathbf{J}}\rVert_F^2 = 7 and 1≤σmax⁡≤71 \leq\sigma_{\max} \leq\sqrt{7}. The full planner therefore has condition number at most 12.6/14.7 (EtH/EiH); spatial-only planning has condition number at least 33.3/50. These bounds quantify improved conditioning for the joint estimation problem.

TrajectoryEtHEiH
Spatial excitation only (β=0\beta= 0)0.030.02
Spatial and temporal excitation0.210.18

Table 10. Minimum singular value of the normalized, stacked data Jacobian.

Structural first-order ambiguity occurs when one fixed pose perturbation reproduces the effect of a time shift across all stacked observations. A constant rigid-body twist can produce such a case. Constant joint velocity alone does not determine the stacked column relationship, since articulated kinematics, perspective, and visibility also change along a trajectory.

Timing Reference and Comparison Protocols

Reference Error Budget

The reference maps image exposure and physical motion onset into a common timing measurement. The recorded event is the robot’s actual motion onset in the joint stream. Exposure-phase measurements include the RGB–depth timestamp relation, and clock mapping accounts for the two stream origins. Table 11 lists the residual error budget.

Componentms
Exposure phase, including RGB–depth skew0.25
Clock mapping0.10
Physical motion-onset localization0.20
Repeated-mean 95% half-width0.95
Combined 95% uncertainty U95U_{95}1.50

Table 11. Uncertainty budget for the nominal 87 ms reference. The first three entries are residual bias bounds; the fourth is a confidence half-width for the repeated mean.

The 20 repetitions characterize repeatability; the exposure-phase and onset measurements supply sub-frame timing. Adding the three residual bias bounds to the repeated-mean interval gives U95=0.25+0.10+0.20+0.95=1.50U_{95} = 0.25 + 0.10 + 0.20 + 0.95 = 1.50 ms.

For a reference adjustment b∈[−1.5,1.5]b \in[-1.5, 1.5] ms, the EtH mean offsets give

R(b)=∣87+b−80.8∣∣87+b−84.3∣=6.2+b2.7+b∈[1.83,3.92].(25)R(b) = \frac{\left|87+b-80.8\right|}{\left|87+b-84.3\right|} = \frac{6.2+b}{2.7+b} \in[1.83,3.92]. \tag*{(25)}

Thus the ordering is maintained across the reference interval. The EiH means, 78.5 and 81.6 ms, likewise retain their ordering. Spatial evaluation uses separate marker-board and touch references.

Paired Multi-Camera Timing

The independent timing reference gives an EtH–EiH difference of 0.2 ms and a 95% interval [−0.4,0.8][-0.4,0.8] ms. Its containment in [−1,1][-1,1] ms motivates the shared-offset model for the four paired dual-view recordings. Common reference errors cancel in this difference, giving a tighter differential interval than the absolute reference budget. Camera-specific regularized calibration estimates (84.3 and 81.6 ms) reflect each view’s data objective as well as the penalty; the sharing criterion is evaluated using the independent timing reference.

Matched Baselines and Controls

Static protocol. The marker-based, DREAM, EasyHeC, EasyHeC++, Zhang et al., and Valassakis et al. entries in Table 2–Table 3 are taken from the corresponding EasyHeC++ comparison [13]. Our static reruns use the same simulation/touch evaluation definitions, and the Hydra comparison uses the matched real static protocol. The touch evaluation averages ten poses.

Dynamic protocol. CM-Calib, SL-Calib, and SyncHeC share recordings, selected trajectory, and joint logs; initialization is matched where applicable. CM-Calib uses target-derived poses and forward kinematics. The fixed-reference control sets Δt=87\Delta t = 87 ms for EasyHeC++ and the SyncHeC spatial objective while preserving their remaining inputs, initialization, and pose-update rules. This yields 0.43/0.48 cm and 0.20/0.23 cm, respectively.

Optimization control. Joint Adam and alternating Adam use the same per-variable learning rates and batch size, and are compared at the same actual total count of data-gradient evaluations. A joint update evaluates both variable gradients in one forward/backward computation; an alternating update evaluates the active block. The evaluation budget counts executed computations, including the treatment of early stopping.

Hardware repetitions. Seven arm configurations and four sensor configurations share the xArm7/D435i reference, giving ten unique configurations. Five motions per configuration give 50 runs in total, including the ten representative runs in Table 5. The additional repetitions supply the Z1 Pro and Azure Kinect means and standard deviations reported in § 4.4.

Downstream Study: Protocol and Statistics

Controlled Pairing

The study varies image–joint temporal pairing while fixing robot hardware, cameras, extrinsics, demonstrations, architecture, training hyperparameters, and the matched training seed within each policy–task cell. Each task provides 50 successful teleoperated demonstrations. The same recordings train both conditions:

  • Uncompensated: pair each image with the joint state carrying the nearest recorded timestamp.

  • Compensated: pair each image at tct_c with the interpolated joint state at tc−Δt∗t_c-\Delta t^*.

The same rule applies during deployment. Compensation uses the recovered calibration estimate. Action timestamps and action-chunk scheduling are fixed across conditions.

Observations and Policy Families

The robot uses the two D435i cameras at 30 Hz and joint logs at 250 Hz. The constant offset includes the stream-clock difference and sensing-to-availability delay. For point-cloud policies, offline demonstration processing enhances depth, transforms both views into the robot base frame, merges and crops them to the workspace, and applies farthest-point sampling to 8192 points. At deployment, point clouds are produced online at approximately 30 Hz with offline depth enhancement disabled. Both temporal conditions use this same observation pipeline.

ACT uses RGB observations and an action-chunking transformer (Zhao et al., [53]); DP uses RGB-conditioned action diffusion (Chi et al., [4]); π0\pi_0 is fine-tuned from its released vision-language-action checkpoint (Black et al., [1]). DP3 (Ze et al., 2024) and ManiFlow (Yan et al., [47]) use the point clouds above. The WM family models particle-based 3D dynamics and uses MPPI-style sampling-based MPC with 1000 action samples per iteration and 15–30 refinement iterations. Particle-based dynamics learned from human interaction and model-based control motivate this representation (Hong et al., [10]; Zhu et al., [54]; Hafner et al., [8]).

Tasks and Execution

Table 12 defines the six tasks. For each of the 36 policy–task cells, 50 independently randomized initial configurations are replayed in both conditions. Initial-state randomization is independent across cells. A trial succeeds if its task criterion is met within 60 s; timeouts, dropped objects, and unrecoverable states count as failures. Each cell therefore contains 50 matched pairs and 100 physical executions.

TaskRegimeRandomization of the initial configurationSuccess criterion
Stack three blockslong-horizonthree 20×40 cm cells, ±5 cm jitterall three stacked, tower stable at timeout
Place cuppose-constrained60×40 cm board, 360° yawcup upright inside the target region
Place banana on platepose-constrainedplate on midline +10 cm; banana 10–15 cm left, 360° yawbanana resting on the plate
Place kettle on stovepose-constrained20×20 cm region, ±60°kettle upright on the stove plate
Open drawerarticulatedL-shaped region (70×20 + 20×20 cm), rotated 90°drawer pulled to ≥ 50% extension
Fold toweldeformable70×50 cm region, ±30°towel folded, free edge within tolerance

Table 12. The six real-robot tasks: regime, randomization of the initial object configuration, and success criterion. Yaw ranges are about the vertical axis.

Paired Aggregate Effect

Let Xji∈{−1,0,1}X_{ji} \in\{-1, 0, 1\} denote the difference in binary success between conditions for pair ii in cell jj. With nj=50n_j = 50 and N=∑jnj=1800N = \sum_j n_j = 1800, the aggregate estimate is

δ^=1N∑j=136∑i=1njXji.(26)\hat{\delta} = \frac{1}{N}\sum_{j=1}^{36}\sum_{i=1}^{n_j} X_{ji}. \tag*{(26)}

For each cell, let bjb_j and cjc_j count pairs won only with and only without compensation. Then dj=bj−cjd_j = b_j - c_j is fixed by the cell’s marginal success counts, and the sample variance is

sj2=bj+cj−dj2/njnj−1.(27)s_j^2 = \frac{b_j + c_j - d_j^2/n_j}{n_j - 1}. \tag*{(27)}

The variance of the stratified aggregate, conditional on the trained policies and calibration, is estimated by

Var⁡^(δ^)=1N2∑j=136njsj2.(28)\widehat{\operatorname{Var}}(\hat{\delta}) = \frac{1}{N^2}\sum_{j=1}^{36} n_j s_j^2. \tag*{(28)}

With equal cell sizes, this reduces to

Var⁡^(δ^)=50(b+c)−∑jdj249N2.(29)\widehat{\operatorname{Var}}(\hat{\delta}) = \frac{50(b+c)-\sum_j d_j^2}{49N^2}. \tag*{(29)}

The counts satisfy b=210b = 210, c=64c = 64, and ∑jdj2=740\sum_j d_j^2 = 740. The estimated gain is δ^=146/1800=0.08111\hat{\delta} = 146/1800 = 0.08111 and its standard error is 0.00904. The paired Wald interval δ^±1.96SE\hat{\delta} \pm1.96\mathrm{SE} is [0.0634, 0.0988], or [6.3, 9.9] percentage points.

This interval measures execution variability for the fixed trained policies and calibration. The 36 cell differences and six family averages describe how the aggregate gain is distributed across tasks and architectures. The total counts in Table 13 agree with the 681 and 827 successes obtained from Table 7.

UncompensatedCompensated
SuccessFailure
Success61764
Failure210909

Table 13. Aggregate paired outcomes over 1800 initial configurations.

References

  1. [1]Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; Jakubczak, S.; Jones, T.; Ke, L.; Levine, S.; Li-Bell, A.; Mothukuri, M.; Nair, S.; Pertsch, K.; Shi, L. X.; Tanner, J.; Vuong, Q.; Walling, A.; Wang, H.; and Zhilinsky, U. 2025. π₀: A Vision-Language-Action Flow Model for General Robot Control. In Robotics: Science and Systems.DOI
  2. [2]Carpentier, J.; Saurel, G.; Buondonno, G.; Mirabel, J.; Lamiraux, F.; Stasse, O.; and Mansard, N. 2019. The Pinocchio C++ Library: A Fast and Flexible Implementation of Rigid Body Dynamics Algorithms and Their Analytical Derivatives. In IEEE/ICSE International Symposium on System Integration, 614–619.DOI
  3. [3]Chen, L.; Qin, Y.; Zhou, X.; and Su, H. 2023. Easyhec: Accurate and Automatic Hand-Eye Calibration via Differential Rendering and Space Exploration. IEEE Robotics and Automation Letters, 8(11): 7234–7241.DOI
  4. [4]Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burchfiel, B.; Tedrake, R.; and Song, S. 2025. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. The International Journal of Robotics Research, 44(10-11): 1684–1704.
  5. [5]Daniilidis, K. 1999. Hand-Eye Calibration Using Dual Quaternions. The International Journal of Robotics Research, 18(3): 286–298.DOI
  6. [6]Furgale, P.; Barfoot, T. D.; and Sibley, G. 2012. Continuous-Time Batch Estimation Using Temporal Basis Functions. In IEEE International Conference on Robotics and Automation, 2088–2095. IEEE.DOI
  7. [7]Hafner, D.; Lillicrap, T.; Ba, J.; and Norouzi, M. 2020. Dream to Control: Learning Behaviors by Latent Imagination. In International Conference on Learning Representations.arxiv.org/abs/1912.01603
  8. [8]Hafner, D.; Pasukonis, J.; Ba, J.; and Lillicrap, T. 2025. Mastering Diverse Control Tasks through World Models. Nature, 640(8059): 647–653.DOI
  9. [9]Hong, Z. 2025. Free-Viewpoint Video of Outdoor Sports Using a Flying Camera. In Computer Vision – ECCV 2024, 178–195. Springer.DOI
  10. [10]Hong, Z.; Liu, Y.; Hou, H.; Ai, B.; Wang, J.; Mu, T.; Qin, Y.; Gu, J.; and Su, H. 2025. Learning Particle-Based World Model from Human for Robot Dexterous Manipulation. In RSS Workshop on Dexterous Manipulation: Learning and Control with Diverse Data.
  11. [11]Hong, Z.; Wu, S.; Cui, H.; Zhao, B.; Ji, R.; He, Y.; Zhang, H.; Ke, Z.; Wang, J.; Zhang, G.; and Gu, J. 2026. R3D: Revisiting 3D Policy Learning. arXiv:2604.15281.arxiv.org/abs/2604.15281
  12. [12]Hong, Z.; and Zhang, G. 2026. Learning Object-Centric Motion Priors from Human for Robotic Dexterous Manipulation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(22): 18288–18296.
  13. [13]Hong, Z.; Zheng, K.; and Chen, L. 2024. EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 816–823.arxiv.org/abs/2410.09293
  14. [14]Horn, R. A.; and Johnson, C. R. 2012. Matrix Analysis. Cambridge, UK: Cambridge University Press, 2nd edition. ISBN 9781139020411.
  15. [15]Huang, W.; Chao, Y.-W.; Mousavian, A.; Liu, M.-Y.; Fox, D.; Mo, K.; and Fei-Fei, L. 2026. PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. arXiv:2601.03782.arxiv.org/abs/2601.03782
  16. [16]Huber, M.; Tian, H.; Mower, C. E.; Müller, L.-R.; Ourselin, S.; Bergeles, C.; and Vercauteren, T. 2025. Hydra: Marker-Free RGB-D Hand-Eye Calibration. arXiv:2504.20584.arxiv.org/abs/2504.20584
  17. [17]Ilonen, J.; and Kyrki, V. 2011. Robust Robot-Camera Calibration. In International Conference on Advanced Robotics, 67–74.DOI
  18. [18]Jangir, Y.; Zhang, Y.; Yamazaki, K.; Zhang, C.; Tu, K.-H.; Ke, T.-W.; Ke, L.; Bisk, Y.; and Fragkiadaki, K. 2026. RobotArena: Scalable Robot Benchmarking via Real-to-Sim Translation. In International Conference on Learning Representations.arxiv.org/abs/2510.23571
  19. [19]Jiang, W.; Lei, B.; and Daniilidis, K. 2024. FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields Using Fisher Information. In European Conference on Computer Vision.arxiv.org/abs/2311.17874
  20. [20]Jin, L.; Chen, X.; Rückin, J.; and Popović, M. 2023. NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering. In IEEE/RSJ International Conference on Intelligent Robots and Systems.arxiv.org/abs/2303.01284
  21. [21]Karaman, S.; and Frazzoli, E. 2011. Sampling-Based Algorithms for Optimal Motion Planning. The International Journal of Robotics Research, 30(7): 846–894.arxiv.org/abs/1105.1186
  22. [22]Kato, H.; Beker, D.; Morariu, M.; Ando, T.; Matsuoka, T.; Kehl, W.; and Gaidon, A. 2020. Differentiable Rendering: A Survey. arXiv:2006.12057.arxiv.org/abs/2006.12057
  23. [23]Kaya, B.; Kumar, S.; Oliveira, C.; Ferrari, V.; and Van Gool, L. 2021. Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3804–3814.arxiv.org/abs/2012.06777
  24. [24]Kerbl, B.; Kopanas, G.; Leimkühler, T.; and Drettakis, G. 2023. 3d Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph., 42(4): 139–1.arxiv.org/abs/2308.04079
  25. [25]Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; Vuong, Q.; Kollar, T.; Burchfiel, B.; Tedrake, R.; Sadigh, D.; Levine, S.; Liang, P.; and Finn, C. 2024. OpenVLA: An Open-Source Vision-Language-Action Model. In Conference on Robot Learning.arxiv.org/abs/2406.09246
  26. [26]Kingma, D. P.; and Ba, J. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.arxiv.org/abs/1412.6980
  27. [27]Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; Dollar, P.; and Girshick, R. 2023. Segment Anything. In IEEE/CVF International Conference on Computer Vision, 4015–4026.arxiv.org/abs/2304.02643
  28. [28]Laine, S.; Hellsten, J.; Karras, T.; Seol, Y.; Lehtinen, J.; and Aila, T. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics, 39(6): 1–14.arxiv.org/abs/2011.03277
  29. [29]Lee, T. E.; Tremblay, J.; To, T.; Cheng, J.; Mosier, T.; Kroemer, O.; Fox, D.; and Birchfield, S. 2020. Camera-to-Robot Pose Estimation from a Single Image. In IEEE International Conference on Robotics and Automation, 9426–9432. IEEE.arxiv.org/abs/1911.09231
  30. [32]Liu, R.; Canberk, A.; Song, S.; and Vondrick, C. 2025a. Differentiable Robot Rendering. In Proceedings of the 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, 117–129.arxiv.org/abs/2410.13851
  31. [33]Liu, S.; Wu, L.; Li, B.; Tan, H.; Chen, H.; Wang, Z.; Xu, K.; Su, H.; and Zhu, J. 2025b. Rtd-1b: A Diffusion Foundation Model for Bimanual Manipulation. In International Conference on Learning Representations, 29982–30009.arxiv.org/abs/2410.07864
  32. [34]Loper, M. M.; and Black, M. J. 2014. OpenDR: An Approximate Differentiable Renderer. In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., European Conference on Computer Vision, volume 8695, 154–169. Cham: Springer International Publishing. ISBN 978-3-319-10583-3.DOI
  33. [36]Qin, T.; Li, P.; and Shen, S. 2018. Vins-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator. IEEE transactions on robotics, 34(4): 1004–1020.DOI
  34. [37]Ravi, N.; Gabeur, V.; Hu, Y.-T.; Hu, R.; Ryali, C.; Ma, T.; Khedr, H.; Rädle, R.; Rolland, C.; Gustafson, L.; Mintun, E.; Pan, J.; Alwala, K. V.; Carion, N.; Wu, C.-Y.; Girshick, R.; Dollar, P.; and Feichtenhofer, C. 2025. SAM 2: Segment Anything in Images and Videos. In International Conference on Learning Representations, volume 2025, 28085–28128.arxiv.org/abs/2408.00714
  35. [38]Ren, T.; Liu, S.; Zeng, A.; Lin, J.; Li, K.; Cao, H.; Chen, J.; Huang, X.; Chen, Y.; Yan, F.; Zeng, Z.; Zhang, H.; Li, F.; Yang, J.; Li, H.; Jiang, Q.; and Zhang, L. 2024. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. arXiv:2401.14159.arxiv.org/abs/2401.14159
  36. [39]Shi, L.; Liu, Y.; Zeng, L.; Ai, B.; Hong, Z.; and Su, H. 2025. Learning Adaptive Dexterous Grasping from Single Demonstrations. In IEEE/RSJ International Conference on Intelligent Robots and Systems.arxiv.org/abs/2503.20208
  37. [40]Shiu, Y. C.; and Ahmad, S. 1989. Calibration of Wrist-Mounted Robotic Sensors by Solving Homogeneous Transform Equations of the Form AX=XB. IEEE Transactions on Robotics and Automation, 5(1): 16–29.DOI
  38. [41]Tsai, R. Y.; and Lenz, R. K. 1989. A New Technique for Fully Autonomous and Efficient 3 d Robotics Hand/Eye Calibration. IEEE Transactions on robotics and automation, 5(3): 345–358.DOI
  39. [42]Valassakis, E.; Drezckowski, K.; and Johns, E. 2022. Learning Eye-in-Hand Camera Calibration from a Single Image. In Conference on Robot Learning, 1336–1346. PMLR.arxiv.org/abs/2111.01245
  40. [43]Wang, J.; Hong, Z.; Bao, C.; Zhu, Y.; Sun, J.; and Zhang, G. 2026. LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation. arXiv:2604.08475.arxiv.org/abs/2604.08475
  41. [44]Wen, B.; Yang, W.; Kautz, J.; and Birchfield, S. 2024. Foundation-Pose: Unified 6D Pose Estimation and Tracking of Novel Objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17868–17879.arxiv.org/abs/2312.08344
  42. [46]Xiang, F.; Qin, Y.; Mo, K.; Xia, Y.; Zhu, H.; Liu, F.; Liu, M.; Jiang, H.; Yuan, Y.; Wang, H.; Yi, L.; Chang, A. X.; Guibas, L. J.; and Su, H. 2020. SAPIEN: A Simulated Part-Based Interactive ENvironment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11097–11107.arxiv.org/abs/2003.08515
  43. [47]Yan, G.; Zhu, J.; Deng, Y.; Yang, S.; Qiu, R.-Z.; Cheng, X.; Memmel, M.; Krishna, R.; Goyal, A.; Wang, X.; and Fox, D. 2025. ManiFlow: A General Robot Manipulation Policy via Consistency Flow Training. arXiv:2509.01819.arxiv.org/abs/2509.01819
  44. [48]Yang, J.; Rebello, J.; and Waslander, S. L. 2023. Next-Best-View Selection for Robot Eye-in-Hand Calibration. In Conference on Robots and Vision, 161–168. IEEE.arxiv.org/abs/2303.06766
  45. [49]Yang, L.; Kang, B.; Huang, Z.; Zhao, Z.; Xu, X.; Feng, J.; and Zhao, H. 2024. Depth Anything V2. In Advances in Neural Information Processing Systems, volume 37, 21875–21911. Curran Associates, Inc.arxiv.org/abs/2406.09414
  46. [50]Yang, P.; Jiang, M.; Shen, X.; and Zhang, H. 2025. Spatiotemporal Calibration for Laser Vision Sensor in Hand-Eye System Based on Straight-Line Constraint. IEEE Robotics and Automation Letters, 10(12): 13320–13327.arxiv.org/abs/2509.12928
  47. [52]Zhang, X.; Xi, Y.; Huang, Z.; Zheng, L.; Huang, H.; Xiong, Y.; and Xu, K. 2023. Active Hand-Eye Calibration via Online Accuracy-Driven next-Best-View Selection. The Visual Computer, 39(1): 381–391.DOI
  48. [53]Zhao, T. Z.; Kumar, V.; Levine, S.; and Finn, C. 2023. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Robotics: Science and Systems.arxiv.org/abs/2304.13705
  49. [54]Zhu, C.; Yu, R.; Feng, S.; Burchfiel, B.; Shah, P.; and Gupta, A. 2025. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. arXiv:2504.02792.arxiv.org/abs/2504.02792

Paper details

vlnc:W_axs9s8yy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents