betaFeedback

WorksComputer Science

Domain-Aware Conformal Prediction for Trustworthy Spacecraft Pose Estimation under Sim-to-Real Shift

Abstract

Autonomous rendezvous and proximity operations require not only an accurate estimate of the pose of the target spacecraft, but also a reliable indication of when this estimate can be trusted. Conformal prediction provides such an indication with a formal guarantee, by producing pose uncertainty sets that contain the true pose with a specified probability. Such a guarantee, however, assumes that calibration and test data follow the same distribution, an assumption that is violated in orbital perception, where labelled calibration images are available only from synthetic renderings while operational images are real. In this work, we show on the SPEED+ dataset that conformal pose estimation calibrated on synthetic images loses its coverage guarantee on real images, and that many of the resulting failures are silent: the predicted pose set appears valid but excludes the true pose. To address this, we calibrate on synthetic images translated to the appearance of real images. Since translation alters appearance while preserving geometry, the synthetic pose labels remain valid and no labelled real images are required. Experiments with two translation models show that this calibration restores coverage above the specified level for miscoverage levels of 0.1 and above, and reduces silent failures from up to 43.5% to at most 3.3% of the test images, bringing conformal pose estimation a step closer to trustworthy deployment in space.

Introduction

Estimating the relative position and orientation (6-DoF6\text{-DoF} pose) of a target spacecraft with respect to a servicer is a key enabling technology for autonomous rendezvous and proximity operations, such as on-orbit servicing, inspection and active debris removal [1, 2]. When the target is non-cooperative, it provides no information about its own state, and its pose must therefore be estimated from onboard sensors. Monocular cameras are an attractive choice owing to their low mass, power consumption and cost [1, 2], and methods based on deep neural networks have demonstrated high accuracy on monocular spacecraft pose estimation benchmarks [3, 4]. Training these networks requires large datasets of images annotated with accurate pose labels, which are difficult and costly to acquire in orbit. Networks are therefore typically trained on synthetic images rendered from a 3D model of the target. However, these images do not fully capture the visual characteristics of real imagery, such as surface textures and complex illumination effects. As a result, networks trained exclusively on synthetic data suffer a significant drop in accuracy when evaluated on real images [5]. This synthetic-to-real (sim2real) gap has become a major focus of the space community, and several datasets have been proposed to study it, including SPEED+ [6], SPARK [7] and SPADES [8]. Numerous domain adaptation methods have since been proposed to reduce this gap, with encouraging results [4, 9, 10]. Nevertheless, even the best-performing methods still produce erroneous estimates on a subset of real images, and aggregate metrics such as the mean error may conceal these failures [10].

An important question therefore remains open: how can the system determine whether an estimated pose can be trusted? This question is critical in practice, since an inaccurate pose that goes undetected may lead to unsafe manoeuvres of the servicer. Each estimate should therefore be accompanied by a reliable measure of its uncertainty, so that unreliable estimates can be identified and handled accordingly. The confidence scores produced by the network itself are not suitable for this purpose, as they are not calibrated and may be particularly misleading on real images, where errors are most likely.

Several approaches have been proposed to quantify the uncertainty of deep neural networks, including Monte Carlo dropout [11], deep ensembles [12] and the direct prediction of error variances, which has also been applied to spacecraft keypoint detection [13]. While informative, these estimates come with no formal guarantee, and their quality has been shown to deteriorate under distribution shift [14]. Conformal prediction [15] provides an alternative with formal guarantees: using a held-out labelled calibration set, it converts the output of any pre-trained model into a prediction set that contains the ground truth with a specified probability. Because it requires neither retraining nor assumptions on the model, it can be applied to an already validated network, which makes it particularly well suited to space applications.

Yang and Pavone [16] extended conformal prediction, previously applied mainly to classification and low-dimensional regression, to object pose estimation. Their method calibrates prediction regions around detected keypoints and propagates them to a set of poses that contains the ground-truth pose with the same probability, together with worst-case bounds on the estimation error. Wang et al. [17] subsequently adapted this approach to satellite pose estimation on synthetic images. The validity of these guarantees, however, relies on the assumption that calibration and test data are exchangeable, which in practice requires them to follow the same distribution. Both works satisfy this assumption by calibrating and testing within the same domain. In spacecraft pose estimation, this is not possible: labelled calibration data are available only for synthetic images, whereas operational images are real. Since the network is less accurate on real images, prediction regions calibrated on synthetic data can be expected to be too narrow, and the true pose may fall outside the predicted set more often than specified. To the best of our knowledge, this effect has not yet been studied.

In this work, we first quantify this effect on SPEED+ and show that the coverage obtained on real images falls well below the specified level. We then address it by observing that recovering the specified coverage requires a calibration set that is both labelled and representative of real images. Since the pose label of an image is determined by its geometry rather than its appearance, a synthetic image whose appearance is modified while its geometry is preserved retains a valid label. Image-to-image translation models perform precisely this transformation, and have been used successfully to reduce the sim2real gap on SPEED+, for instance by the winning team of the Sunlamp category of the 2021 Satellite Pose Estimation Competition [4]. We therefore use translation not to train the network, but to construct a calibration set that approximates the real domain. The conformal method of [16] is otherwise left unchanged, and no labelled real images are required.

The contributions of this paper are as follows:

  • We apply the conformal keypoint detection and pose uncertainty propagation of [16] to spacecraft pose estimation on SPEED+, and analyse why its coverage guarantee no longer holds when calibration images are synthetic and test images are real.

  • We quantify this loss of coverage on the real Lightbox and Sunlamp domains, and show that a large share of the uncovered images are silent failures, in which a non-empty pose set excludes the true pose.

  • We propose to calibrate on synthetic images translated to the appearance of real images, and show that this restores coverage above the specified level for miscoverage levels of 0.1 and above and reduces silent failures from up to 43.5% to at most 3.3% of the test images, without requiring labelled real images.

The remainder of this paper is organized as follows. § 2 reviews related work on spacecraft pose estimation under the sim2real gap, uncertainty quantification and image translation. § 3 presents the conformal pose estimation method and analyses why its guarantee is lost under the sim2real gap. § 4 introduces calibration with translated images. § 5 reports the experimental results on SPEED+, § 6 discusses the limitations of the proposed approach, and § 7 provides summary and concludes the paper.

Related Works

Spacecraft pose estimation under the sim2real gap. Deep learning methods for monocular spacecraft pose estimation fall into two families [2]: Direct approaches that regress the pose directly from the image, and Hybrid approaches that first predict the 2D locations of predefined keypoints and then recover the pose with a Perspective-n-Point (PnP) solver [18]. The latter have generally achieved higher accuracy, as shown by the results of the SPEED challenge [3, 19], and form the basis of the present work. As these methods matured on synthetic data, attention shifted to their performance on real images. The 2021 Satellite Pose Estimation Competition, built on SPEED+ [4, 6], made the sim2real gap its central challenge, and the top-performing entries addressed it through strategies such as extensive data augmentation, image translation and self-training on unlabelled real images. Building on these results, later works further reduced the gap through multi-task learning with online refinement [9], unsupervised domain adaptation [10, 20] and, when a small number of labelled real images is available, supervised adaptation [5]. Throughout this line of work, progress has been measured by average accuracy, while the reliability of individual estimates has received comparatively little attention.

Uncertainty quantification and conformal prediction for pose. Assessing the reliability of an estimate requires quantifying its uncertainty. For deep networks, this is commonly done with Bayesian approximations such as Monte Carlo dropout [11], with deep ensembles [12], or by training the network to predict its own error variance. The latter approach has been applied to spacecraft pose estimation by Wang et al. [13], who used the predicted keypoint uncertainty to improve pose accuracy. Such estimates are informative, but they offer no guarantee that the true value lies within the predicted range. Conformal prediction [15, 21] provides such a guarantee: given a held-out labelled calibration set, it constructs prediction sets that contain the ground truth with a specified probability, provided that calibration and test data are exchangeable. Yang and Pavone [16] extended this framework to object pose estimation by calibrating prediction regions around keypoints detected from heatmaps and propagating them to a pose uncertainty set with worst-case error bounds. Wang et al. [17] later adapted this approach to satellite pose estimation, replacing the heatmaps with a regression network that predicts keypoint uncertainty and calibrating each keypoint separately. In both works, however, calibration and test images are drawn from the same domain, and Wang et al. identify the extension to real imagery as future work. Applying these methods to spacecraft therefore requires a labelled calibration set that is representative of real images, which is precisely what is unavailable in practice. § 3.6 analyses the consequences of this mismatch.

Image translation for the sim2real gap. A natural way to obtain such a calibration set is to change the appearance of labelled synthetic images while preserving their content. Unpaired image-to-image translation models, introduced by CycleGAN [22], learn this mapping between two domains without requiring paired examples. In domain adaptation, they are commonly used to render labelled source images in the style of the target domain so that the source labels can be reused for training [23], and later models such as DUNIT [24] better preserve object content by translating object instances separately from the background. In spacecraft pose estimation, synthetic-to-real translation has likewise been used to reduce the sim2real gap on SPEED+ [25]. In our work, translation serves a different purpose: rather than improving the network, it is used to construct a calibration set that is representative of real images, as described in § 4. We consider both CycleGAN and DUNIT to assess how the choice of translation model affects the recovered coverage.

Conformal Pose Estimation

This section reviews the method of Yang and Pavone [16], which we apply to spacecraft pose estimation. The method proceeds in two stages. Conformal keypoint detection first converts the output of a keypoint network into prediction regions that contain the true keypoints with a specified probability. Geometric uncertainty propagation then maps these regions to a set of poses that contains the true pose with the same probability, from which a pose estimate is derived.

Problem setup

The target spacecraft is described by KK keypoints with known 3D coordinates Y1,…,YKY_1,\ldots,Y_K in its body frame (K=11K = 11 for SPEED+). For an image xx acquired under the true pose (R,t)∈SE(3)(R,t) \in\mathrm{SE}(3), the true 2D keypoints are given by

yk=π(RYk+t),k=1,…,K,(1)y_k = \pi(RY_k + t), \qquad k = 1,\ldots,K, \tag*{(1)}

where π\pi denotes the perspective projection of the calibrated camera. A keypoint network ff predicts one heatmap per keypoint. Following [16], each heatmap is normalized to a probability distribution over pixels, from which we read the most likely location qkq_k and its probability pkp_k. In a conventional pipeline, the pose is then obtained by solving the Perspective-n-Point (PnP) problem on the correspondences {qk↔Yk}\{q_k \leftrightarrow Y_k\} [18], which gives a single estimate with no indication of its reliability.

Inductive conformal prediction

Inductive conformal prediction [15, 21] turns the output of any trained model into a prediction set with a coverage guarantee. Its central ingredient is a non-conformity score s(x,y)s(x,y), which measures how poorly a label yy agrees with the prediction of the model on input xx. Given nn labelled calibration samples (xi,yi)(x_i,y_i), disjoint from the training data, the scores si=s(xi,yi)s_i=s(x_i,y_i) are computed and, for a miscoverage level α∈[0,1]\alpha\in[0,1], the threshold τ\tau is set to the ⌈(n+1)(1−α)⌉\lceil(n+1)(1-\alpha)\rceil-th smallest score. The prediction set for a new input xx then contains every label whose score does not exceed this threshold:

C(x)={y:s(x,y)≤τ}.(2)C(x)=\{y:s(x,y)\le\tau\}. \tag*{(2)}

If the calibration samples and the test sample (x,y)(x,y) are exchangeable, that is, if their joint distribution is invariant to permutations, then

1−α≤P(y∈C(x))≤1−α+1n+1,(3)1-\alpha\le\mathbb{P}(y\in C(x)) \le1-\alpha+\frac{1}{n+1}, \tag*{(3)}

where the upper bound holds when scores have no ties [21]. The argument rests on ranks: under exchangeability, the score of the test sample is equally likely to occupy any of the n+1n+1 positions among all scores, and therefore falls below τ\tau with probability at least 1−α1-\alpha. The guarantee is marginal, i.e., it holds on average over the draw of calibration and test samples, and it makes no assumption on the model or on the distribution of its errors.

Conformal keypoint detection

To apply this framework to keypoints, Yang and Pavone define the score of an image as

s(x,y)=max⁡k=1,…,Kpk∥yk−qk∥.(4)s(x,y)=\max_{k=1,\ldots,K}p_k\lVert y_k-q_k\rVert. \tag*{(4)}

Each term weights the localisation error of a keypoint by the confidence of the network, so that the score is largest when the network is confidently wrong. Taking the maximum over keypoints makes the score depend on the worst keypoint in the image. Consequently, the condition s(x,y)≤τs(x,y)\le\tau in Eq. (2) holds if and only if every keypoint satisfies pk∥yk−qk∥≤τp_k\lVert y_k-q_k\rVert\le\tau, and the prediction set decomposes into one disk per keypoint:

Ck(x)={y∈R2:∥y−qk∥≤τpk},k=1,…,K.(5)C_k(x)=\left\{y\in\mathbb{R}^2:\lVert y-q_k\rVert\le\frac{\tau}{p_k}\right\},\qquad k=1,\ldots,K. \tag*{(5)}

Each disk is centred at the predicted keypoint, and its radius is inversely proportional to the confidence of the network and proportional to the threshold τ\tau, which reflects the errors observed on the calibration set. By Eq. (3), the KK true keypoints lie simultaneously inside their disks with probability at least 1−α1-\alpha. This joint coverage, rather than a per-keypoint one, is what allows the guarantee to be transferred to the pose.

From keypoint regions to a pose uncertainty set

The keypoint disks constrain the pose through the projection in Eq. (1). Yang and Pavone define the pose uncertainty set as the set of all poses whose projected keypoints fall inside their respective disks:

S(x)={(R,t)∈SE⁡(3):π(RYk+t)∈Ck(x), k=1,…,K}.(6)S(x)=\{(R,t)\in\operatorname{SE}(3):\pi(RY_k+t)\in C_k(x),\ k=1,\ldots,K\}. \tag*{(6)}

Since the true keypoints are the projections of the 3D keypoints under the true pose, they lie in all disks if and only if the true pose belongs to S(x)S(x). The pose set therefore inherits the guarantee of the keypoint disks,

P((R,t)∈S(x))≥1−α,(7)\mathbb{P}((R,t)\in S(x))\ge1-\alpha, \tag*{(7)}

without any assumption on the pose solver. This equivalence also provides a direct way to evaluate pose coverage on labelled data, by checking whether the ground-truth keypoints lie in their disks.

The set S(x)S(x) admits an explicit algebraic description. Writing the pose as the vector s=[vec⁡(R)T,tT]T∈R12s=[\operatorname{vec}(R)^\mathsf{T},t^\mathsf{T}]^\mathsf{T}\in\mathbb{R}^{12}, the camera-frame keypoint RYk+tRY_k+t is linear in ss. Multiplying each disk constraint by the squared depth of the keypoint, which is positive for points in front of the camera, turns it into a quadratic inequality sTAks≤0s^\mathsf{T}A_ks\le0, while the requirement of positive depth becomes a linear inequality bkTs>0b_k^\mathsf{T}s>0. Here AkA_k and bkb_k depend only on the disk, the 3D keypoint and the camera intrinsics [16]. S(x)S(x) is thus the intersection of SE⁡(3)\operatorname{SE}(3) with KK quadratic and KK linear constraints. It is in general nonconvex, and it may be empty when the disks are mutually inconsistent. Since an empty set cannot contain the true pose, this event occurs with probability at most α\alpha and itself indicates an unreliable prediction.

Average pose

The pose set characterizes the uncertainty, but does not directly provide a point estimate. Sampling S(x)S(x) directly is difficult because of its nonconvex constraints, whereas sampling the keypoint disks is straightforward. Yang and Pavone therefore propose random sample averaging (RANSAG). At each of TT iterations, three distinct keypoints are selected at random, a point is sampled within each of their disks, and the resulting minimal problem is solved with P3P. The candidate poses that satisfy all constraints of S(x)S(x) are retained. If no candidate is retained, poses are instead obtained by solving PnP on points sampled in all KK disks. The average pose (Rˉ,tˉ)(\bar{R}, \bar{t}) is computed from the retained samples, by projecting the arithmetic mean of the rotations onto SO(3) and averaging the translations. Yang and Pavone [16] further derive worst-case bounds on the error of the average pose through semidefinite relaxations, which are not considered in this work. The pose set thus accompanies the average pose with an uncertainty statement that holds with a specified probability, and an empty pose set indicates that the estimate should not be trusted.

Exchangeability under the sim2real gap

In spacecraft pose estimation, reliable pose labels are available only for synthetic images. The calibration set is therefore necessarily synthetic, whereas the images acquired during operations are real. Calibration and test scores then follow two different distributions, and the rank argument of § 3.2 no longer applies.

The direction of the resulting error can be anticipated. Owing to the sim2real gap, the keypoint network makes larger errors on real images than on synthetic ones. If its confidence pkp_k does not decrease accordingly, the network becomes confidently wrong on real images, which is precisely the situation penalized by the score in Eq. (4). Real scores are therefore expected to be shifted towards larger values than synthetic ones. The threshold τsyn\tau_{\mathrm{syn}} estimated on synthetic data is then too small for real images, the disks in Eq. (5) are too narrow, and the true keypoints, and hence the true pose, fall outside the predicted sets more often than specified. The coverage actually obtained on real images is the fraction of real scores below the synthetic threshold,

Px∼real(s(x,y)≤τsyn),(8)\mathbb{P}_{x \sim\mathrm{real}}\left(s(x,y) \leq\tau_{\mathrm{syn}}\right), \tag*{(8)}

which may fall well below the specified level 1−α1-\alpha. This failure is silent: the prediction sets retain a plausible size and shape, and nothing in the output indicates that the guarantee no longer holds. Section 5 quantifies this effect on SPEED+.

Recalibrating on labelled real images would restore exchangeability, but such images are not available in orbit. What is required instead is a calibration set that carries reliable labels while being representative of real images. Section 4 describes how such a set can be constructed from synthetic data.

Domain-Aware Calibration with Translated Images

Section 3.6 showed that the coverage guarantee of conformal pose estimation depends on the calibration set being representative of the test images, a condition that a synthetic calibration set does not satisfy for real images. We address this mismatch by modifying only the calibration set: each labelled synthetic calibration image is rendered in the appearance of the real domain by an image-to-image translation model GG, and the nonconformity scores are computed on the translated images. The keypoint network, the score, the keypoint disks and the propagation to the pose described in Section 3 remain unchanged. Figure 1 illustrates the resulting pipeline.

Figure 1. Overview of the proposed pipeline. Labelled synthetic calibration images are translated to the appearance of the real domain, and the nonconformity scores computed on the translated images by the pretrained keypoint network determine the threshold τ\tau. For each real test image, τ\tau defines the keypoint disks, which are propagated to a pose set that contains the true pose with probability at least 1−α1-\alpha.

This construction relies on the fact that the pose label of an image is determined by the geometry of the spacecraft in the image, namely its position, orientation and silhouette, and not by its appearance. If translation modifies textures and illumination while preserving this geometry, the translated image G(xi)G(x_i) depicts the spacecraft under the same pose as xix_i, and the pair (G(xi),yi)(G(x_i), y_i) is correctly labelled without any annotation of real images. The translation model should therefore alter appearance as much as needed to resemble the real domain, while retaining the structure of the spacecraft. We consider two unpaired translation models that pursue this objective in different ways. CycleGAN [22] learns a mapping between the two domains at the image level, constrained by a cycle-consistency loss that encourages the translated image to retain the content of its input. DUNIT [24] additionally detects object instances and translates them separately from the background, which is expected to better preserve the structure of the spacecraft. Since the two real domains of SPEED+ differ substantially in illumination, a separate translation model is trained for each of them, from synthetic to Lightbox and from synthetic to Sunlamp, using synthetic images and unlabelled real images of the corresponding domain. The keypoint network is not retrained.

Algorithm 1 summarises the procedure. During calibration, the synthetic calibration images are translated, and the score of Eq. (4) is computed on each translated image using the original synthetic label. The threshold τ\tau is then obtained as in § 3.2. At test time, each real image is processed exactly as in § 3, with this threshold in place of the one estimated on synthetic images.

If translated images followed exactly the distribution of real images, calibration and test samples would be exchangeable, and the guarantee of Eq. (3) would hold again. In practice, translation is only approximate, and the remaining mismatch can be related to the loss of coverage. Let PP and QQ denote the distributions of the scores on translated and real images, respectively. The coverage obtained on real images then satisfies [26]

P(s(x,y)≤τ)≥1−α−dTV(P,Q),(9)\mathbb{P}\left(s(x,y) \le\tau\right) \ge1-\alpha-d_{\mathrm{TV}}(P,Q), \tag*{(9)}

where dTVd_{\mathrm{TV}} denotes the total variation distance. The loss of coverage is thus controlled by the discrepancy between score distributions, rather than by the visual discrepancy between images. Translated images therefore need not be indistinguishable from real ones; it suffices that the keypoint network errs on them in a similar way. Since QQ is unknown in practice, Eq. (9) does not provide a guarantee at test time, but it clarifies the condition under which calibration on translated images can be expected to recover the specified coverage.

Experiments

The experiments address three questions. First, does conformal pose estimation calibrated on synthetic images retain its coverage on real images? Second, does calibration on translated images recover the specified coverage? Third, how does the choice of calibration set affect the accuracy of the average pose?

Experimental setup

Dataset. All experiments are conducted on SPEED+ [6]. The synthetic domain contains 59,960 rendered images of the Tango spacecraft, split into 47,966 training and 11,994 validation images. The two real domains, Lightbox (6,740 images) and Sunlamp (2,791 images), were acquired with a physical mockup of Tango under two illumination conditions: Lightbox reproduces diffuse Earth albedo, whereas Sunlamp uses a direct light source that produces strong specular reflections and lens flare. All real images are used as test images.

Algorithm 1 Domain-aware conformal pose estimation

Require: keypoint network ff; labelled synthetic calibration set {(xi,yi)}i=1n\{(x_i,y_i)\}_{i=1}^{n}; unlabelled real images U\mathcal{U} of the target domain; miscoverage level α∈[0,1]\alpha\in[0,1]

Calibration

1: Train the translation model GG on synthetic images and U\mathcal{U} 2: for i=1,…,ni=1,\ldots,n do 3: x~i←G(xi)\tilde{x}_i \leftarrow G(x_i) ▹\triangleright appearance translated, label yiy_i kept 4: Obtain (qk,pk)k=1K(q_k,p_k)_{k=1}^{K} from the heatmaps of f(x~i)f(\tilde{x}_i) 5: si←max⁡kpk∥yi,k−qk∥s_i \leftarrow\max_k p_k \lVert y_{i,k}-q_k\rVert ▹\triangleright Eq. (4) 6: end for 7: τ←⌈(n+1)(1−α)⌉\tau\leftarrow\lceil(n+1)(1-\alpha)\rceil-th smallest value of {si}i=1n\{s_i\}_{i=1}^{n}

Prediction for a real test image xx

8: Obtain (qk,pk)k=1K(q_k,p_k)_{k=1}^{K} from the heatmaps of f(x)f(x) 9: Ck(x)←{y∈R2:∥y−qk∥≤τ/pk}C_k(x) \leftarrow\{y \in\mathbb{R}^{2}:\lVert y-q_k\rVert\le\tau/p_k\} for k=1,…,Kk=1,\ldots,K ▹\triangleright Eq. (5) 10: Define the pose set S(x)S(x) from {Ck(x)}k=1K\{C_k(x)\}_{k=1}^{K} ▹\triangleright Eq. (6) 11: Compute the average pose (Rˉ,tˉ)(\bar{R},\bar{t}) with RANSAG 12: return S(x),(Rˉ,tˉ)S(x),(\bar{R},\bar{t})

Implementation. The keypoint network is a heatmap-based model with an EfficientNet-B5 backbone [27], trained on the synthetic training images only. The calibration set is drawn from the synthetic validation images. For each real domain, a CycleGAN and a DUNIT model are trained at a resolution of 640×400640 \times400 on the synthetic images and the real images of that domain, the latter being used without their pose labels. The synthetic images used to train the translation models include the calibration images.

Calibration protocols. All protocols use the same keypoint network and the same test images, and differ only in the calibration set. In Synthetic, the scores are computed on the synthetic calibration images, which is the setting available in practice without translation. In CycleGAN and DUNIT, the calibration images are first translated to the corresponding real domain.

Metrics. The coverage is the fraction of test images whose ground-truth pose lies in the pose set S(x)S(x); a valid calibration achieves a coverage of at least 1−α1-\alpha. Each test image that is not covered falls into one of two cases. Either the pose set is empty, in which case the estimate is explicitly flagged as unreliable, or the pose set is non-empty but does not contain the ground truth, in which case the failure is missed. The latter case is the most critical for operations, since nothing in the output indicates that the estimate is wrong. The accuracy of the average pose is evaluated by the median rotation error, the median translation error and the pose score EE used in the Satellite Pose Estimation Competition [4], which sums the mean rotation error in radians and the mean normalised translation error.

Coverage on real images

Table 1 reports the coverage obtained under each protocol for α∈{0.1,0.2,0.3}\alpha\in\{0.1,0.2,0.3\}, and Figure 2 extends the comparison to α∈[0.05,0.4]\alpha\in[0.05,0.4]. With synthetic calibration, the coverage on real images falls well below the specified level and degrades rapidly as α\alpha increases. At α=0.1\alpha=0.1, it reaches 80.4% on Lightbox and 66.0% on Sunlamp instead of the specified 90%, and at α=0.3\alpha=0.3 it drops to 10.4% and 1.5%, respectively. The loss is larger on Sunlamp, consistent with its stronger departure from the synthetic illumination. At α=0.05\alpha=0.05, synthetic calibration does reach the specified coverage (96.6% and 96.8%), which indicates that the synthetic threshold exceeds the scores of most real images only for very small values of α\alpha.

DomainCalibrationα=0.1\alpha= 0.1α=0.2\alpha= 0.2α=0.3\alpha= 0.3
Cov.↑EmptyMissed↓Cov.↑EmptyMissed↓Cov.↑EmptyMissed↓
LightboxSynthetic80.411.68.031.227.541.310.446.143.5
CycleGAN93.63.52.992.85.02.292.16.51.3
DUNIT93.53.62.992.85.02.292.26.51.4
SunlampSynthetic66.015.918.110.648.540.91.572.426.0
CycleGAN92.94.22.991.36.12.687.99.03.1
DUNIT92.44.63.090.86.52.887.59.23.3

Table 1. Coverage, empty-set rate and missed-failure rate (%) on real test images. A missed failure is a non-empty pose set that does not contain the ground-truth pose. The specified coverage is 1−α1 - \alpha.

Figure 2. Empirical versus specified coverage on real test images for each calibration set. Points below the dashed diagonal indicate under-coverage.

Calibration on translated images restores valid coverage for all α≥0.1\alpha\ge0.1 on both domains. At α=0.1\alpha=0.1, CycleGAN and DUNIT reach 93.6% and 93.5% on Lightbox, and 92.9% and 92.4% on Sunlamp. At α=0.05\alpha=0.05, the coverage remains slightly below the specified 95% (94.5% and 93.7% with CycleGAN). The coverage obtained with translated calibration, however, varies little with α\alpha: on Lightbox, it decreases only from 93.6% to 92.1% when α\alpha increases from 0.1 to 0.3. The calibration is therefore conservative for larger values of α\alpha. This behaviour suggests that the scores of most real images lie well below the thresholds obtained from translated images, while a small fraction of real images, on which the network fails severely, have scores that exceed all thresholds considered.

The decomposition of the uncovered images in Table 1 and Figure 3 shows that the consequences of synthetic calibration are more severe than the coverage alone suggests. With synthetic calibration, a large part of the uncovered images are missed failures: at α=0.1\alpha= 0.1, 8.0% of the Lightbox images and 18.1% of the Sunlamp images receive a non-empty pose set that excludes the true pose, and this proportion exceeds 40% on both domains at α=0.2\alpha= 0.2. Empty sets also become frequent, reaching 72.4% of the Sunlamp images at α=0.3\alpha= 0.3, so that most estimates would be rejected. With translated calibration, the missed-failure rate remains between 1.3% and 3.3% on both domains and for all values of α\alpha, and most uncovered images are instead flagged by an empty pose set.

Figure 3. Outcome of conformal pose estimation on real test images for synthetic (S), CycleGAN (C) and DUNIT (D) calibration: pose set containing the ground truth (covered), empty pose set, and non-empty pose set excluding the ground truth (missed).

Accuracy of the average pose

Since the average pose is computed from samples drawn within the keypoint disks, its accuracy depends on the size of the disks, and thus on the calibration set. Table 2 reports the accuracy of the average pose under each protocol. Synthetic calibration yields the most accurate average pose for α=0.1\alpha= 0.1 and α=0.2\alpha= 0.2, since its threshold, and therefore its disks, are smaller. This accuracy, however, comes with the loss of coverage reported above. With translated calibration, the larger disks increase the error of the average pose at α=0.1\alpha= 0.1, with rotation errors between 2.3 and 2.6 degrees on both domains. This error decreases as α\alpha increases, and at α=0.3\alpha= 0.3 translated calibration reaches an accuracy comparable to synthetic calibration on Lightbox (pose score 0.082 versus 0.081) and slightly better on Sunlamp (0.100 versus 0.103), while maintaining a coverage of approximately 92% and 88%, respectively. These results indicate a trade-off between coverage and the accuracy of the average pose, which, given the conservative behaviour of translated calibration, favours moderate values of α\alpha.

DomainCalibrationα=0.1\alpha= 0.1 Rot.α=0.1\alpha= 0.1 Trans.α=0.1\alpha= 0.1 EEα=0.2\alpha= 0.2 Rot.α=0.2\alpha= 0.2 Trans.α=0.2\alpha= 0.2 EEα=0.3\alpha= 0.3 Rot.α=0.3\alpha= 0.3 Trans.α=0.3\alpha= 0.3 EE
LightboxSynthetic0.850.0260.0780.890.0270.0790.950.0280.081
LightboxCycleGAN2.640.1180.1521.300.0470.0931.010.0330.082
LightboxDUNIT2.500.1110.1461.290.0470.0931.020.0340.083
SunlampSynthetic1.350.0420.0981.440.0450.1011.540.0480.103
SunlampCycleGAN2.610.1180.1461.640.0640.1071.410.0490.100
SunlampDUNIT2.290.1050.1341.560.0620.1051.390.0480.100

Table 2. Accuracy of the average pose on real test images: median rotation error (deg), median translation error (m) and pose score EE.

Comparison of translation models

CycleGAN and DUNIT lead to nearly identical coverage, with differences of at most 0.5 percentage points in Table 1. DUNIT yields a slightly more accurate average pose at α=0.1\alpha= 0.1, in particular on Sunlamp, where the median rotation error decreases from 2.61 to 2.29 degrees and the pose score from 0.146 to 0.134. These differences remain small compared with the gap between synthetic and translated calibration, which indicates that the recovery of coverage stems primarily from translating the calibration images rather than from the specific choice of translation model.

Qualitative results

Figure 4 shows two synthetic calibration images translated to each real domain. In all cases, the silhouette of the spacecraft and the position of its antennas are preserved, so that the synthetic keypoint labels remain valid for the translated images. The Lightbox translations introduce sensor noise, darker surfaces and localised specular highlights. The Sunlamp translations reproduce the strong overexposure and glare characteristic of this domain. CycleGAN tends to saturate large parts of the spacecraft body and to add diffuse glare to the background, whereas DUNIT confines the saturation to smaller regions and better preserves surface structures such as the pattern of the solar panel. This difference is consistent with the instance-level translation performed by DUNIT and with its slightly more accurate average pose on Sunlamp. The saturation introduced by the Sunlamp translations also makes some keypoints harder to localise on the calibration images, which may contribute to the conservative behaviour of translated calibration observed in Figure 2. Figure 5 shows keypoint disks predicted on real test images of both domains. At α=0.3\alpha= 0.3, synthetic calibration yields a median disk radius below 4 pixels, and several ground-truth keypoints fall outside their disks, whereas CycleGAN calibration increases the radius to between 26 and 29 pixels and covers all 11 keypoints. At α=0.1\alpha= 0.1, both calibrations cover all keypoints in the examples shown, but the disks obtained with CycleGAN calibration are considerably larger, with a median radius between 86 and 111 pixels against about 11 pixels, which is consistent with the lower accuracy of the average pose reported in § 5.3. The disks also vary in size within an image, being larger for keypoints on which the network is less confident.

Two synthetic calibration images and their translations to the Lightbox and Sunlamp domains by CycleGAN and DUNIT

Figure 4. Two synthetic calibration images (left) and their translations to the Lightbox and Sunlamp domains by CycleGAN and DUNIT.

Keypoint disks on a real Lightbox and Sunlamp test image at two values of alpha, obtained with synthetic calibration and CycleGAN calibration

Figure 5. Keypoint disks on a real Lightbox (top) and Sunlamp (bottom) test image at α=0.1\alpha= 0.1 (left) and α=0.3\alpha= 0.3 (right), obtained with synthetic calibration and with CycleGAN calibration. Yellow crosses indicate the ground-truth keypoints; green and red disks indicate covered and missed keypoints, respectively. The boxes in the bottom-left corner of each panel show a zoomed view of the region marked in the image; at α=0.3\alpha= 0.3, they show a keypoint missed under synthetic calibration and the same region under translated calibration.

Discussion and Limitations

Silent failures under synthetic calibration. The experiments confirm that the coverage guarantee of conformal pose estimation does not transfer from synthetic to real images. The most concerning aspect of this failure is not the loss of coverage itself, but the fact that a large share of the uncovered images receive a pose set that appears valid while excluding the true pose. Such failures cannot be detected from the output, and they are precisely the failures that uncertainty quantification is meant to prevent. Calibration on translated images reduces them to a few percent of the test images, while most of the remaining failures are flagged by an empty pose set.

Conservative coverage. Translated calibration restores valid coverage for α≥0.1\alpha\ge0.1, but the coverage it achieves varies little with α\alpha and exceeds the specified level by a wide margin for larger values of α\alpha. The guarantee is therefore recovered in a conservative rather than an exact form. This conservativeness has a cost: the keypoint disks are larger than necessary, and the average pose computed from them is less accurate at small values of α\alpha. In practice, this suggests selecting a moderate α\alpha, for which translated calibration still provides high coverage while the accuracy of the average pose remains comparable to that obtained with synthetic calibration.

Absence of a formal guarantee. Calibration on translated images restores exchangeability only approximately. As discussed in § 4, the remaining loss of coverage is bounded by the discrepancy between the score distributions on translated and real images, which cannot be measured without labelled real images. The coverage obtained on SPEED+ is therefore an empirical result, and it does not constitute a guarantee for other targets, illumination conditions or translation models.

Availability of real images. The translation models require unlabelled real images of the target domain. In our experiments, these images were drawn from the same domain as the test images, which represents a favourable setting. During a mission, representative images may be only partially available before operations, for instance from ground test facilities, and conditions that are absent from these images may again lead to a loss of coverage.

Scope of the evaluation. The evaluation is limited to SPEED+, whose real images were acquired with a mockup in a laboratory rather than in orbit, and to a single keypoint network. In addition, the non-conformity score of [16] is driven by the worst keypoint of each image, which contributes to the size of the disks.

Conclusion

This paper studied conformal pose estimation for spacecraft under the sim2real gap. We applied the conformal keypoint detection and geometric uncertainty propagation to spacecraft pose estimation, and showed on SPEED+ that calibration on synthetic images leads to a substantial loss of coverage on real images, together with a large proportion of silent failures in which a non-empty pose set excludes the true pose. To address this issue, we calibrated on synthetic images translated to the appearance of real images, which preserves the synthetic pose labels and requires no labelled real images. With both CycleGAN and DUNIT, this calibration restored a coverage above the specified level for α≥0.1\alpha\ge0.1 on both real domains and reduced silent failures to a few percent of the test images, at the cost of larger prediction sets.

Future work will aim to reduce the conservativeness of translated calibration, for instance through nonconformity scores that account for the differences between keypoints, and to combine it with domain adaptation in order to obtain a more accurate keypoint network and therefore tighter prediction sets. Further directions include the evaluation on other datasets such as SPARK and SPADES, the extension to image sequences, in which temporal correlation challenges the exchangeability assumption, and the evaluation of the worst-case error bounds of [16] under the sim2real gap.

Acknowledgment

The work was funded by Luxembourg National Research Fund under the project reference BRIDGES/2025/IS/19855391/U-ADAPT and Infinite Orbits. Experiments were performed on the Luxembourg national supercomputer MeluXina at LuxProvide.

References

  1. [1]Sumant Sharma, Connor Beierle, and Simone D’Amico. Pose estimation for non-cooperative spacecraft rendezvous using convolutional neural networks. In 2018 IEEE Aerospace Conference, pages 1–12. IEEE, 2018.arxiv.org/abs/1809.07238
  2. [2]Leo Pauly, Wassim Rharbaoui, Carl Shneider, Arunkumar Rathinam, Vincent Gaudilliere, and Djamila Aouada. A survey on deep learning-based monocular spacecraft pose estimation: Current state, limitations and prospects. Acta Astronautica, 212:339–360, 2023.DOI
  3. [3]Bo Chen, Jiewei Cao, Alvaro Parra, and Tat-Jun Chin. Satellite pose estimation with deep landmark regression and nonlinear pose refinement. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 2816–2824. IEEE, 2019.arxiv.org/abs/1908.11542
  4. [4]Tae Ha Park, Marcus Märtens, Mohsi Jawaid, Zi Wang, Bo Chen, Tat-Jun Chin, Dario Izzo, and Simone D’Amico. Satellite pose estimation competition 2021: Results and analyses, 2023.DOI
  5. [5]Inder Pal Singh, Nidhal Eddine Chenni, Abd El Rahman Shabayek, Arunkumar Rathinam, and Djamila Aouada. Bridging the synthetic-real gap: Supervised domain adaptation for robust spacecraft 6-dof pose estimation. arXiv preprint arXiv:2509.13792, 2025.arxiv.org/abs/2509.13792
  6. [6]Tae Ha Park, Marcus Märtens, Gurvan Lecuyer, Dario Izzo, and Simone D’Amico. Speed+: Next-generation dataset for spacecraft pose estimation across domain gap. In 2022 IEEE aerospace conference (AERO), pages 1–15. IEEE, 2022.DOI
  7. [7]Mohamed Adel Musallam, Vincent Gaudilliere, Enjie Ghorbel, Kassem Al Ismaeil, Marcos Damian Perez, Michel Poucet, and Djamila Aouada. Spacecraft recognition leveraging knowledge of space environment: Simulator, dataset, competition design and analysis. In 2021 IEEE International Conference on Image Processing Challenges (ICIPC), pages 11–15. IEEE, 2021.DOI
  8. [8]Arunkumar Rathinam, Haytam Qadadri, and Djamila Aouada. Spades: A realistic spacecraft pose estimation dataset using event sensing. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 11760–11766. IEEE, 2024.
  9. [9]Tae Ha Park and Simone D’Amico. Robust multi-task learning and online refinement for spacecraft pose estimation across domain gap. Advances in Space Research, 73(11):5726–5740, 2024.DOI
  10. [10]Nidhal Eddine Chenni, Arunkumar Rathinam, and Djamila Aouada. Unsupervised domain adaptation for sim-to-real object pose estimation with contrastive alignment and pseudo-label refinement. arXiv preprint arXiv:2606.21287, 2026.arxiv.org/abs/2606.21287
  11. [11]Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.arxiv.org/abs/1506.02142
  12. [12]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1612.01474
  13. [13]Jinghao Wang, Yang Li, Zhang Li, Zi Wang, and Qifeng Yu. Monocular satellite pose estimation based on uncertainty estimation and self-assessment. IEEE Transactions on Aerospace and Electronic Systems, 60(6):9163–9178, 2024.DOI
  14. [14]Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1906.02530
  15. [15]Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023.
  16. [16]Heng Yang and Marco Pavone. Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8947–8958. IEEE, 2023.arxiv.org/abs/2303.12246
  17. [17]Jinghao Wang, Zhang Li, Cong Sun, Yulan Guo, Zi Wang, and Qifeng Yu. Satellite pose set estimation by uncertainty-guided conformal keypoint detection. IEEE Transactions on Neural Networks and Learning Systems, 2025.DOI
  18. [18]Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Ep n p: An accurate o (n) solution to the p n p problem. International journal of computer vision, 81(2):155–166, 2009.
  19. [19]Mate Kisantal, Sumant Sharma, Tae Ha Park, Dario Izzo, Marcus Märtens, and Simone D’Amico. Satellite pose estimation challenge: Dataset, competition design, and results. IEEE Transactions on Aerospace and Electronic Systems, 56(5):4083–4098, 2020.DOI
  20. [20]Juan Ignacio Bravo Pérez-Villar, Álvaro García-Martín, and Jesús Bescós. Spacecraft pose estimation based on unsupervised domain adaptation and on a 3d-guided loss combination. In European Conference on Computer Vision, pages 37–52. Springer, 2022.
  21. [21]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer, 2005.
  22. [22]Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2017.arxiv.org/abs/1703.10593
  23. [23]Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989–1998. Pmlr, 2018.arxiv.org/abs/1711.03213
  24. [24]Deblina Bhattacharjee, Seungryong Kim, Guillaume Vizier, and Mathieu Salzmann. Dunit: Detection-based unsupervised image-to-image translation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4786–4795. IEEE, 2020.DOI
  25. [25]Zi Wang, Minglin Chen, Yulan Guo, Zhang Li, and Qifeng Yu. Bridging the domain gap in satellite pose estimation: A self-training approach based on geometrical constraints. IEEE transactions on aerospace and electronic systems, 60(3):2500–2514, 2023.arxiv.org/abs/2212.12103
  26. [26]Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816–845, 2023.arxiv.org/abs/2202.13415
  27. [27]Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PmLR, 2019.arxiv.org/abs/1905.11946

Paper details

vlnc:W_5qkgy43t
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents