WorksComputer Science
Unreasonable Confidence from Simulator Misspecification: Directional Robustness on the SPEED+ Sim-to-Real Benchmark
Abstract
Vision-based navigation systems for orbit must be trained almost entirely on synthetic imagery, because real labelled images of a given spacecraft in flight rarely exist. Benchmarks like SPEED+ are built for that situation: the training set comes from a simulator, and the test sets are real images of a satellite mockup under two kinds of lighting. That simulator approximates how light reflects off mylar and aluminium, and like every scientific simulator it is misspecified. Its error is radiometric, since it misses the specular highlights, hard shadows and sensor saturation common in real images. We probe the resulting synthetic-to-real gap, the central risk for such systems, with a deliberately simple monocular pose estimator. In a 2x2 study of backbone adaptation and photometric augmentation, trained on synthetic images and tested on both real hardware-in-the-loop domains, we find three things. (1) The model does not know it is failing: regions conformally calibrated to 90% on synthetic data cover only 9-15% of rotation errors on sunlamp and 21-38% on lightbox, so a wrong pose carries the confidence of a correct one. (2) Transfer is asymmetric: scaling data and features improved synthetic rotation error from 1.65 to 1.03 rad but moved the harshly lit sunlamp domain by only 0.05 rad, under twice its seed s.d., whereas adapting the backbone's last stage carried 63-83% of its extra synthetic gain to both real domains. (3) Once the backbone adapts, augmentation helps only in the direction it models: our photometric stack improved the diffusely lit lightbox domain from 1.17 to 0.99 rad on every seed but moved sunlamp no more than seed-to-seed variation, while with the backbone frozen it helped sunlamp more. Every post-review experiment was pre-registered, and the code, per-image predictions and pre-registration record are at https://github.com/lokindoo/spacecraft-pose-estimation.
Introduction
Machine learning models for spacecraft navigation are trained almost entirely on the output of a simulator, then deployed against a physical system that the simulator only approximates. Real images of a target spacecraft in orbit, labelled with its pose, essentially do not exist. Estimators therefore learn from rendered images and meet real ones for the first time during the mission. The SPEED+ benchmark [20] was built to make that discrepancy measurable. It pairs ~60,000 labelled synthetic training images of the Tango spacecraft with ~9,500 real images of a physical mockup, lit two ways: lightbox, diffuse light like Earth albedo, and sunlamp, a collimated solar simulator that gives harsh directional light. On its predecessor SPEED, whose real test images show the same mockup, the top ten entries of the 2019 challenge degraded 2.7–40× in pose score from synthetic to real images (Table 1: winner 40×, runner-up 5.3×, median 8.5×) [12].
Preprint.
SPEED+’s synthetic images do not come from a physics simulation. They are rendered by an OpenGL rasteriser, the Optical Stimulator camera emulator, which shades sunlit foil and metal with an empirical formula chosen for speed rather than fidelity [20]. That formula stands in for the physics of light that the renderer never computes, much as a climate model uses simple rules for the clouds it cannot resolve, and it is where the simulator’s error lives. The error is therefore radiometric rather than geometric: ranges and pose distributions match across domains, and only appearance differs. Real mylar and machined metal reflect specularly, and a collimated lamp turns those reflections into hard shadow edges and saturated pixels. Saturation is not a shading error at all but happens in the camera’s detector, which means the simulator gets the camera wrong as well as the scene. In these terms a photometric augmentation stack is a discrepancy model, an unvalidated guess at how simulator and reality differ. It does not correct the simulator but spreads the training images across the differences it guesses at. This paper is about what happens when that guess is wrong: the model loses accuracy, and it also loses its calibration without any outward sign.
We use SPEED+ as a controlled testbed and ask what actually transfers from simulation to reality. Our estimator is a direct-regression baseline, an ImageNet-pretrained ResNet-50 with a small head. We train it on synthetic images alone, in a design of {backbone frozen, last stage trained} {no augmentation, photometric augmentation}, and evaluate it on the full lightbox and sunlamp test sets. Every arm runs on three seeds (for the frozen un-augmented arm, two of them retrain only the head), and checkpoints are selected on synthetic validation images only. Our contributions:
The model does not know it is failing. Split-conformal regions calibrated to 90% on held-out synthetic data keep that coverage in-domain. On sunlamp they cover only 9–15% of rotation errors, and test-time BatchNorm adaptation lifts that only to 11–20%. On the weakest arms the attitude cone spans roughly half of all orientations, which keeps it valid but makes it meaningless.
Transfer is asymmetric. With the backbone frozen, scaling data and features improved synthetic rotation error from 1.65 to 1.03 rad, but sunlamp only from 2.04 to 1.99 rad, less than twice its seed s.d., and the same holds for either factor alone. Training layer 4 took synthetic error further, from 1.03 to 0.32 rad, and 63–83% of that extra gain carried across: lightbox went from 1.76 to 1.17 rad and sunlamp from 1.99 to 1.53.
Augmentation is directional. We built our photometric stack in good faith to anticipate harsh real illumination, yet it models neither real domain well, and the diffuse lightbox markedly less badly (Figure 2). With layer 4 adapting, it improved lightbox by ∼0.18 rad on every seed while sunlamp moved no further than seed-to-seed variation, with the backbone frozen the order reverses. Removing blur, noise and the glare blob does not resolve sunlamp either.
The revision is pre-registered and auditable. Every experiment added after review had its prediction and decision rule committed before its result was opened, and five of those rules softened or qualified a claim of the workshop submission (App. § 18). The per-image predictions are public, and so is a script that checks every number in this paper against the released results.

Figure 2. What the augmentation actually models. Columns: augmented synthetic training samples, and real test samples from each domain. Below each is the pixel-intensity density (log counts), pooled over every image in that split. The augmented density is overlaid on both real columns (dashed). It is far from both, and closer to lightbox: total variation 0.63 against 0.80. Gradient statistics give the same ordering (Table 8).
This extended version adds what the workshop’s five-page limit left out. § 2 places the study among prior work, including the 2019 challenge’s per-entry results, and § 4 reports what it took to get a baseline worth testing. The post-review experiments move from the appendix into the body, and every arm now has three seeds. § 7 shows what the real-domain failures look like. § 8 brings together the results on normalisation statistics, and § 9 adds coverage by range. Appendices E–H give the per-checkpoint analyses, the reproducibility details and the full pre-registration record.
Related work
Prior work on spacecraft pose has measured how far methods fall across the synthetic-to-real gap. We hold one simple method fixed and ask a different question: which of its gains cross the gap, and does its uncertainty survive it?
Spacecraft pose benchmarks and the gap. SPEED and its 2019 challenge [12] set up the problem: synthetic training images, and real test images of a Tango mockup. The top ten entries degraded 2.7–40× from synthetic to real (Table 1). The runner-up, EPFL_cv1ab, had the best real score, and the best entry without PnP, pedro_fairspace, finished third. SPEED+ [20] made the gap itself the object of study, with two real domains, one lit diffusely and one by a collimated sun simulator, and SPEC2021 [21] reports the competition held on them. SPNv2 [18], the reference method on SPEED+, combines multi-task representations with heavy style augmentation, the texture randomisation that Park et al. [19] first used with keypoints and PnP.
| # | Team | Ratio | PnP | ||
| 1 | UniAdelaide | 0.0094 | 0.3752 | 39.9× | yes |
| 2 | EPFL_cvlab | 0.0215 | 0.1140 | 5.3× | yes |
| 3 | pedro_fairspace | 0.0571 | 0.1555 | 2.7× | no |
| 4 | Team Platypus | 0.0703 | 1.7201 | 24.5× | no |
| 5 | motokimura1 | 0.0758 | 0.6011 | 7.9× | no |
| 6 | Magpies | 0.1393 | 1.2659 | 9.1× | no |
| 7 | GabrielA | 0.2423 | 2.6209 | 10.8× | no |
| 8 | stainsby | 0.3711 | 5.0004 | 13.5× | no |
| 9 | VSI_Feeney | 0.4658 | 1.5993 | 3.4× | no |
| 10 | jblumenkamp | 0.8859 | 4.2418 | 4.8× | yes |
Table 1. SPEC2019 top ten on SPEED [12]: SPEED score on the synthetic and real test sets, and their ratio. Median ratio 8.5×.
Direct regression and keypoints with PnP. Direct methods map an image straight to a pose, as PoseNet [11] did for camera relocalisation and SPN [24] and URSO [22] did for spacecraft. Keypoint pipelines instead regress 2D landmarks and solve for the pose geometrically. The 2019 winner [2] regressed landmark heatmaps and refined the pose by nonlinear optimisation, and keypoint methods took the top two places, but the ranking was on synthetic data. On real data the 2019 results do not separate the two classes: PnP entries in the top ten degraded 4.8–40×, the others 2.7–24×. For regression the rotation representation matters too, since quaternions cover twice and every rotation representation in four or fewer dimensions is discontinuous [32]. We keep the quaternion and remove the sign ambiguity instead § 3.
Sim-to-real for vision. Domain randomisation [27, 28] varies rendering so widely that the real world looks like one more variation, and style augmentation [10] randomises texture and colour by style transfer. Our photometric stack is the cheap end of the same idea, and its blur and noise are standard corruptions [9]. Test-time adaptation of BatchNorm statistics [14, 23, 16] uses unlabelled target images, as does adversarial domain adaptation [6], and we test the first § 8 but not the second. Across many models and shifts, out-of-distribution accuracy is often close to a linear function of in-distribution accuracy [15], and robustness beyond that line is rare [25], a question Figure 1 asks of our configurations.

Figure 1. Synthetic against real rotation error for every configuration: frozen-backbone heads (open circles) and layer-4 arms (squares), mean ± s.d. over seeds. The grey line marks perfect transfer and the dashed line chance.
Uncertainty under shift. Split conformal prediction [30, 13, 1] guarantees coverage in finite samples, on average over test points, as long as calibration and test data are exchangeable, which is exactly what a synthetic-to-real shift breaks. Two extensions relax the assumption, but neither fits here. Weighted conformal prediction [26] restores validity under covariate shift given the likelihood ratio between domains, which needs domains that overlap. Adaptive conformal inference [7] tracks coverage online from feedback on its own errors, and a navigator in orbit has no labels to give that feedback. Coverage conditional on a feature such as range is attainable only in weaker forms [29, 5], so we measure it § 9 rather than guarantee it. Predictive uncertainty is known to degrade under dataset shift [17], and conformal prediction has been carried to object pose through keypoints [31]. We measure what a synthetic calibration is worth on real images.
Setup
Data. From SPEED+’s synthetic domain we sample 28,000 training images from the official training labels and 2,000 validation images from the official validation set, and downscale them to 320×512 grayscale to fit the GPU available. The conformal analysis § 9 draws two more disjoint sets of 1,000 from the official validation set, excluding those 2,000: one to calibrate on and one to test in-domain. Both real domains serve in full as test sets, lightbox (6,740 images) and sunlamp (2,791), and no real image influences training, tuning or model selection. Range distributions match across domains, so any extra difficulty on the real images comes from appearance, not from harder poses.
Model and loss. A ResNet-50 [8] pretrained on ImageNet [4] feeds its global-average-pooled 2048- features to a two-layer MLP head. The head regresses translation (targets -scored per component) and a unit quaternion : direct pose regression [11, 24, 22]. The loss treats the two separately,
with and -normalised. Because and encode the same rotation, naive quaternion MSE gives the network contradictory targets [32]. Two choices remove that ambiguity: the squared inner product ignores sign, and the labels are canonicalised to one hemisphere. § 4 shows what it cost us to find this out.
The 2×2 arms. In the two frozen arms the backbone runs in inference mode and keeps its ImageNet BatchNorm statistics, so only the head is trained. In the two layer-4 arms the final ResNet stage is trained as well, so its BatchNorm statistics follow the (augmented) synthetic training images. All arms share one configuration: Adam, learning rate , batch 16 and 20 epochs, with a checkpoint every second epoch. Each arm keeps the checkpoint with the best synthetic validation rotation error, and each runs on seeds 42, 43 and 44. A frozen backbone without augmentation sees the same inputs in every epoch, so its pooled features can be computed once and cached, and seeds 43 and 44 of that arm retrain the head on the cache. As a check, the published seed-42 head applied to the cache reproduces its errors to within 0.0005 rad (App. § 14).
Augmentation. The photometric factor is a good-faith attempt to anticipate harsh real illumination. It jitters brightness (0.65–1.5) and contrast (±0.5), applies a log-uniform gamma in [0.5, 2], and adds a soft, clamped Gaussian “sun glare” blob ( px), Gaussian blur and additive Gaussian noise, each stochastic component with , followed by random erasing. Table 2 gives every parameter, and § 6 examines what the stack actually prepares the model for.
| Component | Parameters | Arm A | Arm B |
| Brightness / contrast jitter | factor [0.65, 1.5] / ±0.5 | ✓ | ✓ |
| Gamma | log-uniform in [0.5, 2] | ✓ | ✓ |
| Sun-glare blob () | clamped Gaussian, px, peak [0.8, 2] | ✓ | – |
| Gaussian blur () | kernel 7, | – | – |
| Additive Gaussian noise () | – | – | |
| Random erasing () | area [0.02, 0.06] | ✓ | ✓ |
Table 2. The photometric stack, in order. Stochastic components apply with . Arm A drops blur and noise, and arm B also drops the glare blob.
Metrics. We track normalised translation error and geodesic rotation error . Their sum is the official SPEED+ score [12], which we report as a plain sum, whereas SPEC2021 [21] also zeroes errors below a calibration threshold. Rotation errors also need a reference point, which uniform guessing provides: if orientations are uniformly random, the rotation angle has density , so random guessing scores rad, which we call the chance level.1 We define skill as 2.207 − , how far an error sits below chance, because it separates errors that look close. As raw errors, 1.67 and 2.07 rad are neighbours, but as skill they are 0.54 and 0.13 rad, one meaningfully better than chance and the other barely.
Hardware. The eight published runs were trained on a free-tier Colab GPU. The post-review runs (the ablation arms, the control, and frozen augmented seeds 43 and 44) were trained with the released code on an Apple M5 GPU, and the cached-feature heads on its CPU, as App. § 17 lists run by run.
Code and data. SPEED+ comes from its official hosts. The code, configs, per-seed results, per-image predictions and conformal analysis behind every number here are at https://github.com/lokindoo/spacecraft-pose-estimation.
Getting an honest baseline
Before the gap can be measured, the in-domain error has to mean what it says. Two problems stood between our first model and that, and neither was a sim-to-real effect (Table 3).
| Step | Synthetic (val) | Lightbox | Sunlamp |
| 0. 4k images, logits head, MSE on raw quaternions | – | – | – |
| 1. + labels canonicalised to one hemisphere | 1.666 | 2.002 | 2.073 |
| 2. + sign-invariant rotation loss | – | – | – |
| 3. + pooled 2048-d features | – | – | – |
| 4. + 28k images (published frozen arm, seed 42) | 1.053 | 1.752 | 2.048 |
Table 3. The baseline in sequence: (rad), on the 2,000-image synthetic validation split used everywhere else. Step 0 recorded only its loss, and steps 2 and 3 have no saved score on this split.
The double cover. Our first model regressed all seven pose targets, z-scored, with plain MSE. A quaternion and its negative encode the same rotation, so near-identical images can carry targets on opposite sides of the sphere, and MSE pulls the prediction towards their average: the dataset mean [32]. After 50 epochs on 4,000 images the training loss was still 0.892, barely below the 1.0 that predicting the mean of z-scored targets scores. Canonicalising the labels to one hemisphere, with nothing else changed, brought the training loss to 0.285 and the validation rotation error to 1.666 rad. This bug would appear even if no real image existed, because it belongs to the labels rather than to the shift.
Memorisation. The next step scaled to 28,000 images and pooled 2048-d features. The first such run (seed 42) scores 0.654 rad on its own training images, 0.40 rad better than on validation. Those 0.40 rad are a generalisation gap, and the rest of its validation error is what a head on frozen ImageNet features can fit at all. More data attacks the first part, and adapting the backbone the second.
One lesson carries beyond this study: take a bundled change apart before crediting any piece of it, as § 5 does for data and features.
Synthetic gains transfer asymmetrically
| Configuration | Synthetic (val) | Lightbox | Sunlamp |
| 4k images, 1000-d logits | 1.651 ± 0.024 | 1.962 ± 0.012 | 2.040 ± 0.051 |
| 4k images, 2048-d pooled | 1.395 ± 0.008 | 1.872 ± 0.016 | 2.062 ± 0.033 |
| 28k images, 1000-d logits | 1.353 ± 0.026 | 1.878 ± 0.015 | 2.016 ± 0.027 |
| 28k images, 2048-d pooled (published config.) | 1.030 ± 0.017 | 1.761 ± 0.026 | 1.986 ± 0.031 |
| 28k, pooled, tuned (lr , 60 ep.) | 0.929 ± 0.014 | 1.751 ± 0.035 | 2.001 ± 0.025 |
| 28k, pooled, synthetic-BN backbone | 1.394 ± 0.002 | 2.064 ± 0.040 | 2.107 ± 0.010 |
Table 5. Frozen-backbone heads retrained on cached features: (rad), mean ± s.d. over three seeds, every one a head retrain. The published configuration is Table 4’s frozen arm, whose seed 42 was trained end to end instead.
Table 4 gives the results of all four arms, but we start further back, from our first frozen-backbone configuration: 4,000 training images, with the head reading the backbone’s 1,000-d ImageNet logits rather than its pooled features. Over three seeds, each head retrained on cached features, it reaches rad on the 2,000-image synthetic validation split, 1.962 on lightbox and 2.040 on sunlamp (Table 5). Scaling to 28,000 images and 2048-d pooled features improves synthetic rotation error by 0.620 rad (1.651 → 1.030), but sunlamp by only 0.054 rad (2.040 → 1.986). That is less than twice the scaled head’s seed s.d. on sunlamp (0.031 rad), the yardstick our pre-registered rule uses (App. § 18), while lightbox absorbs roughly a third of the gain (1.962 → 1.761). In terms of skill, the scaled head is 1.18 rad better than chance on synthetic images but keeps only 19% of that on sunlamp (38% on lightbox), so under harsh illumination the improvement all but vanishes.
| Backbone | Aug. | Synthetic (val) | Synthetic (val) | Lightbox | Lightbox | Sunlamp | Sunlamp |
| frozen1 | – | 0.066±0.001 | 1.044±0.009 | 0.150±0.001 | 1.763±0.024 | 0.254±0.008 | 2.004±0.049 |
| frozen2 | ✓ | 0.073±0.002 | 1.185±0.019 | 0.121±0.005 | 1.708±0.031 | 0.304±0.024 | 1.876±0.010 |
| layer 4 | – | 0.036±0.002 | 0.317±0.010 | 0.169±0.027 | 1.166±0.018 | 0.311±0.036 | 1.534±0.075 |
| layer 4 | ✓ | 0.042±0.002 | 0.322±0.027 | 0.150±0.017 | 0.987±0.010 | 0.355±0.081 | 1.468±0.119 |
Table 4. Main results: mean ± s.d. over seeds {42, 43, 44}. range-normalised, in rad (chance level 2.207). Checkpoints are picked by best synthetic validation .
The same cells separate the two factors, as pre-registered (App. § 14). Data and features split the synthetic gain roughly evenly (0.33 and 0.29 rad), and neither factor alone moves sunlamp by more than twice its seed s.d. Two checks rule out the easy explanations: a tuned head gains another 0.10 rad on synthetic but nothing on the real domains, and re-estimating every BatchNorm statistic on synthetic images, with the weights fixed, makes the frozen head worse everywhere (§ 8). The layer-4 gain below is therefore not a matter of normalisation statistics alone.
Training layer 4 behaves differently: against the same frozen head, synthetic drops 1.03 → 0.32 (Table 5 and Table 4), and this time the real domains follow, sunlamp 1.99 → 1.53 and lightbox 1.76 → 1.17. Between 63% (sunlamp) and 83% (lightbox) of the extra synthetic gain carries across. What produced a gain therefore matters as much as its size, since more data for a head on frozen ImageNet features helped little while adapting the representation helped a lot. Even so, the layer-4 arm without augmentation keeps only 36% of its skill on sunlamp (55% on lightbox), and its SPEED+ score degrades 3.8–5.2× (=0.353 synthetic, 1.335 lightbox, 1.845 sunlamp). That sits at the low end of the 2019 challenge’s range, but it is not evidence of a smaller gap (§ 10).
Figure 1 puts the asymmetry in one picture, using the points of Table 4 and Table 5. Across the six frozen configurations that keep ImageNet’s BatchNorm statistics, synthetic error spans 0.93–1.65 rad. Yet a least-squares line through them moves sunlamp error by only 0.11 rad per radian of synthetic error (lightbox 0.32). The layer-4 arms sit well off that line: at a synthetic error of 0.30–0.32 rad, every one of them beats every frozen configuration, by at least 0.34 rad on sunlamp and 0.54 on lightbox. In terms of accuracy on the line [15], the frozen configurations lie on a shallow line, and adapting the backbone moves a configuration off it rather than along it.
Augmentation is directional once the backbone adapts
Which way augmentation helps depends on the backbone it trains. With layer 4 adapting, the stack improves the diffuse domain it models and leaves sunlamp unresolved, but with the backbone frozen the order reverses.
Seeds are paired across arms (42, 43 and 44 in both), so we read per-seed differences rather than comparing spreads. On lightbox, augmentation lowers the error on every seed: −0.183, −0.185 and −0.170 rad, about 15% each time, and the mean moves from 1.166 ± 0.018 to 0.987 ± 0.010 rad. On sunlamp the paired differences change sign: +0.116, −0.265 and −0.047 rad (1.534 ± 0.075 1.468 ± 0.119), a spread wider than the effect it would have to establish. Translation is a weaker indicator, but it points the same way (Table 4). Lightbox improves within noise, while on sunlamp two of the three augmented seeds (0.405, 0.399) fail to beat the fitted constant’s 0.372, so augmentation pushes sunlamp translation to the no-skill line.
With layer 4 adapting, the lightbox gain does not come at the cost of the synthetic reference, whose error is essentially unchanged (0.317 0.322 rad). We report absolute error rather than the synthetic-to-real “gap”, because a gap can narrow merely because the reference gets worse. With the backbone frozen, that is what happens: augmentation worsens synthetic error on every seed (by 0.134–0.169 rad, paired against the head-only retrains in Table 6). There the direction also reverses: sunlamp improves on every seed (−0.069 to −0.132 rad), and by more than lightbox (−0.012 to −0.117), so augmentation favours lightbox only once layer 4 adapts.
| vs. no augmentation | ||||||
| Seed | Synth. | Lightbox | Sunlamp | Synth. | Lightbox | Sunlamp |
| 42 | 1.180 | 1.718 | 1.864 | +0.169 | −0.029 | −0.128 |
| 43 | 1.205 | 1.734 | 1.883 | +0.161 | −0.012 | −0.069 |
| 44 | 1.169 | 1.673 | 1.881 | +0.134 | −0.117 | −0.132 |
Table 6. Frozen backbone with augmentation, (rad) per seed and paired difference from the head-only retrain without augmentation (same seed).
Ablation (Table 7). Dropping blur and noise lowers sunlamp on two of three seeds (−0.184, +0.006, −0.117 rad) and worsens lightbox on all three. Dropping the glare blob as well recovers lightbox, but not sunlamp. Neither sunlamp effect is resolved: a control rerun on the same hardware shifts seed 42’s sunlamp error by −0.142 rad, as much as the effects themselves.
| Arm | Seed | vs. full stack | Sunlamp | |||||
| Synth. | Lightbox | Sunlamp | Synth. | Lightbox | Sunlamp | |||
| full stack | 42 | 0.352 | 0.986 | 1.592 | − | − | − | 0.405 |
| 43 | 0.302 | 0.997 | 1.354 | − | − | − | 0.399 | |
| 44 | 0.311 | 0.977 | 1.459 | − | − | − | 0.261 | |
| A: no blur/noise | 42 | 0.320 | 1.070 | 1.408 | −0.032 | +0.084 | −0.184 | 0.311 |
| 43 | 0.328 | 1.001 | 1.360 | +0.026 | +0.004 | +0.006 | 0.325 | |
| 44 | 0.320 | 1.081 | 1.342 | +0.009 | +0.104 | −0.117 | 0.280 | |
| B: jitter, gamma, erasing | 42 | 0.297 | 0.983 | 1.390 | −0.055 | −0.003 | −0.202 | 0.286 |
| 43 | 0.310 | 0.947 | 1.437 | +0.008 | −0.050 | +0.083 | 0.309 | |
| 44 | 0.291 | 1.068 | 1.414 | −0.020 | +0.091 | −0.045 | 0.330 |
Table 7. Augmentation ablation, (rad) per seed and paired difference from the published full stack (layer 4, same seed). Last column: sunlamp .
Mechanism. The stack models diffuse illumination change: jitter and a soft, clamped blob simulate gentle, spatially smooth variation in lighting, which is what lightbox is. Sunlamp’s collimated beam instead produces highlights off metal and foil, cast shadows, and clipped regions where the sensor saturates. Because photometric jitter only brightens smoothly, the augmented images end up brighter overall than either real domain yet clip less often than sunlamp (2.0% of pixels against 2.8%, Table 8). We expected blur, noise and the glare blob to work against sunlamp’s sharp, bright structure. The ablation does not confirm this, and per-pixel gradients do not even show sunlamp as sharper, since most of its frame is dark (edge density 2.0%, against 6.6% for lightbox and 10.7% for the augmented stack). What gradients do show is direction: un-augmented synthetic images are closer to lightbox than to sunlamp (gradient total variation 0.07 against 0.26), and the stack, mostly through blur and noise, pushes them away from both (0.29 and 0.51). Layer 4’s BatchNorm statistics follow these augmented images, which fit neither real domain well and lightbox less badly (Figure 2), so augmentation pushed training towards the domain it was already least wrong about. Domain randomisation [27] helps where it brackets the target domain, but ours brackets neither and misses sunlamp by more.
| Pool | Edge density (%) | Clipped (%) | TV intensity lightbox | TV intensity sunlamp | TV gradient lightbox | TV gradient sunlamp |
| lightbox | 6.6 | 0.08 | 0.00 | 0.28 | 0.00 | 0.22 |
| sunlamp | 2.0 | 2.78 | 0.28 | 0.00 | 0.22 | 0.00 |
| synthetic, no aug. | 10.9 | 0.07 | 0.68 | 0.83 | 0.07 | 0.26 |
| synthetic, full stack | 10.7 | 1.96 | 0.63 | 0.80 | 0.29 | 0.51 |
| synthetic, arm A | 9.6 | 2.62 | 0.65 | 0.81 | 0.10 | 0.33 |
| synthetic, arm B | 9.7 | 1.67 | 0.66 | 0.82 | 0.09 | 0.31 |
Table 8. Pooled image statistics and total-variation distance to each real domain.
What the failures look like
The real-domain failures are broad rather than discrete confusions. A pose estimator can fail in two quite different ways: it can mistake an object for a symmetric copy of itself, which piles up errors near , or it can spread its errors widely, and we pre-registered tests for both (App. § 18).
Neither real domain is bimodal (Figure 3): on every layer-4 checkpoint the share of errors within 0.3 rad of is 0.03–0.05 on lightbox and 0.07–0.10 on sunlamp, below the 0.19 that uniform guessing puts there. Nor do the near- errors share an axis. A symmetry flip would concentrate their axes in the body frame, but the share within 20° of the principal axis reaches the pre-registered 0.18 on only two of six checkpoints per domain, and an in-plane confusion would concentrate them in the camera frame, where the share reaches 0.18 on none (Table 16). Lightbox errors peak near (medians 0.69–0.92 rad). Sunlamp errors spread from there towards (medians 1.11–1.54), with a long tail that stays below the uniform density beyond about . Under harsh light, then, the model is not mistaking Tango for a flipped copy of itself: its errors are simply broad, even though its stated confidence says otherwise (§ 9).

Figure 3. Rotation-error densities on each domain for the layer-4 arms, pooled over three seeds, against the density uniform guessing gives, (dotted).
| Run | Domain | Median | Near- | Cone, body | Cone, camera | |
| layer 4, no aug., 42 | lightbox | 0.92 | 0.051 | 344 | 0.105 | 0.122 |
| layer 4, no aug., 42 | sunlamp | 1.31 | 0.086 | 240 | 0.083 | 0.071 |
| layer 4, no aug., 43 | lightbox | 0.92 | 0.050 | 339 | 0.198 | 0.112 |
| layer 4, no aug., 43 | sunlamp | 1.54 | 0.095 | 265 | 0.132 | 0.068 |
| layer 4, no aug., 44 | lightbox | 0.90 | 0.044 | 299 | 0.100 | 0.094 |
| layer 4, no aug., 44 | sunlamp | 1.37 | 0.082 | 229 | 0.140 | 0.118 |
| layer 4, full stack, 42 | lightbox | 0.74 | 0.027 | 183 | 0.191 | 0.131 |
| layer 4, full stack, 42 | sunlamp | 1.50 | 0.090 | 250 | 0.128 | 0.092 |
| layer 4, full stack, 43 | lightbox | 0.71 | 0.040 | 270 | 0.163 | 0.144 |
| layer 4, full stack, 43 | sunlamp | 1.11 | 0.071 | 198 | 0.192 | 0.096 |
| layer 4, full stack, 44 | lightbox | 0.69 | 0.039 | 263 | 0.171 | 0.122 |
| layer 4, full stack, 44 | sunlamp | 1.31 | 0.071 | 197 | 0.315 | 0.071 |
Table 16. Shape of the rotation errors on the real domains, layer-4 checkpoints. Near-: share of errors within 0.3 rad of (uniform guessing: 0.19). Cone: share of the near- errors’ axes within 20° of their principal axis, in the body and camera frames. A symmetry flip would concentrate the first and an in-plane confusion the second.
Normalisation statistics
The layer-4 arms differ from the frozen ones in two ways at once: layer 4’s weights adapt, and so do its BatchNorm statistics, which then follow the augmented synthetic images. The three results below pull these two changes apart.
First, statistics alone do not explain the layer-4 gain, since re-estimating every BatchNorm layer of the frozen backbone on synthetic images, weights untouched, makes its head worse on every domain: 1.394 rad on synthetic, 2.064 on lightbox and 2.107 on sunlamp, against 1.030, 1.761 and 1.986 with ImageNet statistics (Table 5). Second, the direction augmentation pushes in depends on whether layer 4, and with it its statistics, adapts. With the backbone frozen, augmentation helps sunlamp more than lightbox on every seed, but once layer 4 adapts it helps lightbox on every seed and leaves sunlamp unresolved (Table 6). Third, swapping in real-domain statistics at test time helps accuracy but not calibration. AdaBN [14] re-estimates layer 4’s statistics on unlabelled images from one half of each real domain, and we test on the other half. It lowers rotation error on every layer-4 checkpoint, by 0.09 rad on lightbox and 0.16 on sunlamp on average, yet rotation coverage reaches only 24–40% and 11–20% (Table 9). The cone was calibrated on synthetic errors, and adaptation leaves it where it was.
| Aug. | Seed | Domain | Rotation coverage (%) | ||
| – | 42 | lightbox | 1.171 → 1.074 | 0.202 → 0.149 | 21.1 → 23.9 |
| sunlamp | 1.519 → 1.393 | 0.298 → 0.196 | 8.5 → 11.2 | ||
| 43 | lightbox | 1.199 → 1.110 | 0.153 → 0.130 | 26.0 → 28.0 | |
| sunlamp | 1.644 → 1.422 | 0.351 → 0.199 | 8.8 → 14.5 | ||
| 44 | lightbox | 1.160 → 1.057 | 0.156 → 0.132 | 25.6 → 29.3 | |
| sunlamp | 1.540 → 1.393 | 0.283 → 0.181 | 11.7 → 11.6 | ||
| ✓ | 42 | lightbox | 1.012 → 0.933 | 0.153 → 0.136 | 36.4 → 40.4 |
| sunlamp | 1.594 → 1.400 | 0.406 → 0.210 | 12.3 → 18.6 | ||
| 43 | lightbox | 1.013 → 0.908 | 0.165 → 0.170 | 32.2 → 35.5 | |
| sunlamp | 1.348 → 1.239 | 0.397 → 0.210 | 14.8 → 19.1 | ||
| 44 | lightbox | 0.978 → 0.914 | 0.133 → 0.129 | 37.5 → 40.0 | |
| sunlamp | 1.483 → 1.342 | 0.262 → 0.192 | 14.6 → 19.6 |
Table 9. AdaBN on the held-out half of each real domain: unadapted → adapted. Coverage is of the 90% synthetic-calibrated rotation cone.
In the discrepancy-model reading of § 1, the augmentation stack is a guess at the simulator’s error, and these results say where that guess enters the model. A frozen backbone sees the guess only through what its head learns. An adapted one also absorbs it into layer 4, normalisation included, and there the guess’s bias towards diffuse light becomes the model’s own. Test-time adaptation replaces the guessed statistics with measured ones and lowers the error, but it cannot repair a confidence that was calibrated under the guess.
AdaBN also lowers sunlamp translation error on every layer-4 checkpoint, by 0.07–0.20, to 0.18–0.21, against the fitted constant’s 0.372. Without adaptation, predicted range on sunlamp barely follows the true range (slope 0.11–0.33, against 0.98–1.01 on synthetic), which places far targets much too close and is also why translation coverage collapses with range (§ 9). No fixed position scores below 0.33 there, even one fitted to sunlamp’s own labels, so after adaptation the models read range from the image again.
The model does not know it is failing
Does the estimator’s uncertainty survive the shift? For each arm we calibrate split-conformal regions [30, 13, 1] on 1,000 fresh synthetic images, used neither for training nor for checkpoint selection. For translation the score is the error in metres, which gives a keep-out sphere, and for rotation it is the geodesic angle, which gives a cone, both at a nominal 90% with the finite-sample-corrected quantile.
In-domain, the regions work: test coverage is close to nominal for every arm and metric (87.7–91.4%), and its Clopper–Pearson [3] interval includes 90% in every case but one, the frozen un-augmented arm’s translation (87.7%, 85.5–89.7%), which is slightly anti-conservative.
On the real domains coverage collapses (Table 10): for the layer-4 arm without augmentation, over its three seeds, rotation coverage is 9–13% on sunlamp and 21–27% on lightbox (translation 14–18% and 33–35%), and App. B gives every arm, domain and seed, with intervals and inflation factors. To restore 90% rotation coverage on sunlamp, that arm’s attitude cone would have to grow ∼5–6×, from ∼0.5 to ∼2.8 rad, into a cone holding roughly 78% of all orientations.2 Coverage follows the directional finding of § 6: augmentation raises lightbox rotation coverage to 33–38%, but sunlamp only to 13–15%. The frozen arms fail the opposite way, because their 90% attitude cone already spans 2.16–2.42 rad, or 42–56% of all possible orientations, which keeps it valid in-domain but makes it too wide to navigate by. A navigation filter fed these poses in deployment would receive wrong inputs that look exactly as confident as correct ones (Figure 4). Test-time adaptation does not rescue this (§ 8): AdaBN lowers the error, but lifts rotation coverage only to 24–40% on lightbox and 11–20% on sunlamp.
| Arm | (rad) | In-domain | Lightbox | Sunlamp | Lightbox | Sunlamp |
| frozen, no aug. | 2.16–2.30 | 88.7–90.0 | 63.6–71.0 | 53.4–59.1 | – | – |
| frozen, full stack | 2.37–2.42 | 88.7–91.1 | 75.1–76.3 | 68.9–71.0 | – | – |
| layer 4, no aug. | 0.46–0.54 | 88.8–89.7 | 21.5–26.7 | 9.0–12.9 | 23.9–29.3 | 11.2–14.5 |
| layer 4, full stack | 0.48–0.58 | 91.1–91.4 | 32.6–38.0 | 12.6–15.4 | 35.5–40.4 | 18.6–19.6 |
| layer 4, arm A | 0.54–0.56 | 89.2–90.5 | 31.8–33.9 | 16.0–18.7 | – | – |
| layer 4, arm B | 0.48–0.54 | 89.7–91.4 | 28.7–36.1 | 12.6–14.5 | – | – |
Table 10. Rotation coverage (%) of the 90% synthetic-calibrated cone, range over three seeds. AdaBN columns: after re-estimating layer 4’s BatchNorm statistics on the other half of each real domain.

Figure 4. Conformal reliability of the attitude cone for the layer-4 arm without augmentation: mean over seeds 42, 43 and 44, with their range shaded. In-domain coverage tracks the diagonal, while both real domains collapse far below it. The guarantee itself is intact, and what broke is exchangeability.
Coverage by range. Marginal coverage averages over poses, and in-domain it hides a slope. For the layer-4 arm without augmentation, over three seeds, the metre-radius sphere covers 92.9% of the nearest fifth of in-domain targets but only 80.9% of the farthest. The range-normalised score reverses the slope: 73.9% near and 96.4% far (Figure 5). The rotation cone under-covers at both ends (86.0% and 81.8%) and over-covers in between. A fixed metre radius is too tight far away and range-normalising overcorrects close in, so neither score is conditionally calibrated, and a marginal guarantee cannot make it so [5]. The real domains are worse still, and on sunlamp translation coverage collapses with range, from 58.4% of the nearest fifth to 0.1% of the farthest for the metre sphere, and from 35.0% to 0.2% when range-normalised, while rotation coverage stays at 7.1–11.6% throughout, and App. F gives every arm.

Figure 5. Coverage of the marginal 90% region within each range bin (quintiles of the calibration split’s range) for the layer-4 arm without augmentation: mean over three seeds, range shaded. Dashed: nominal.
Discussion and limitations
Ratios flatter weak models. A degradation ratio divides by the in-domain error. On sunlamp, even the skill-free constant pose (App. C) scores only our synthetic , so our modest ratio does not mean a small gap. Nor do the 2019 results separate method classes: PnP entries in the top ten degraded 4.8–40×, the others 2.7–24× (Table 1). SPNv2 [18], the reference sim-to-real method on SPEED+, closes far more of the gap with multi-task representations and heavy style augmentation. We do not compete with it but characterise how gains transfer under a simple estimator.
Limitations. In the frozen un-augmented arm, seed 42 was trained end to end and seeds 43 and 44 retrain only the head on cached features. With the backbone frozen the two compute the same thing, but they draw different random numbers and ran on different hardware: when retrained head-only for comparison, seed 42 moves by 0.04 rad on synthetic and 0.06 on sunlamp, about as far as one seed from another. The difference is negligible, but we state it because the frozen cell of Table 4 combines the two. The ablation arms and two frozen augmented seeds ran on different hardware from the published runs. Sunlamp is also the smaller test set (2,791 images), and three seeds do not resolve sunlamp effects of 0.1 rad. We study one estimator on one benchmark at , and with real-domain errors near 1 rad it is a diagnostic baseline rather than a deployable navigator. Our directionality claim is about the match between an augmentation and the shift it faces, not about photometric augmentation in general. Our image statistics pool whole frames, which dark background dominates, so object-masked statistics would test the mechanism more directly. Finally, our conformal coverage is marginal, and § 9 shows how far it is from conditional.
Future work. The mechanism suggests its own tests: augmentation that models sunlamp’s actual structure (specular highlights, hard shadows, saturation clipping), full fine-tuning, and weighted conformal prediction [26]. The last needs density ratios between domains that barely overlap, which is exactly what sim-to-real lacks.
For fields that must rely on synthetic data, the question to ask of a robustness intervention is not “how much does it help?” but “which way does it push the training distribution?”, and uncertainty has to be validated under the shift, or a wrong pose arrives wrapped in a confident interval.
Per-seed results
Table 11 lists every run behind Table 4: all 36 arm seed domain evaluations, with the hardware each run was trained on. Seed 42 of the frozen un-augmented arm was trained end to end, and seeds 43 and 44 retrain its head on cached features. With the backbone frozen and the inputs un-augmented, the head sees the same features either way, and the published seed-42 head reproduces its errors on the cache to within 0.0005 (App. D). Table 5 instead retrains the head for all three seeds, so its row for the same configuration differs slightly from this arm’s mean, and the body quotes Table 5 because its comparisons are made there. Each run keeps the checkpoint with the best synthetic validation rotation error, chosen without reference to either real domain.
| Backbone | Aug. | Seed | Trained on | Domain | ||
| frozen | – | 42 | Colab | synthetic (val) | 0.066 | 1.053 |
| frozen | – | 42 | Colab | lightbox | 0.151 | 1.752 |
| frozen | – | 42 | Colab | sunlamp | 0.253 | 2.048 |
| frozen | – | 43 | head only | synthetic (val) | 0.064 | 1.045 |
| frozen | – | 43 | head only | lightbox | 0.150 | 1.746 |
| frozen | – | 43 | head only | sunlamp | 0.263 | 1.952 |
| frozen | – | 44 | head only | synthetic (val) | 0.066 | 1.036 |
| frozen | – | 44 | head only | lightbox | 0.148 | 1.791 |
| frozen | – | 44 | head only | sunlamp | 0.246 | 2.013 |
| frozen | ✓ | 42 | Colab | synthetic (val) | 0.071 | 1.180 |
| frozen | ✓ | 42 | Colab | lightbox | 0.118 | 1.718 |
| frozen | ✓ | 42 | Colab | sunlamp | 0.306 | 1.864 |
| frozen | ✓ | 43 | M5 | synthetic (val) | 0.073 | 1.205 |
| frozen | ✓ | 43 | M5 | lightbox | 0.126 | 1.734 |
| frozen | ✓ | 43 | M5 | sunlamp | 0.328 | 1.883 |
| frozen | ✓ | 44 | M5 | synthetic (val) | 0.074 | 1.169 |
| frozen | ✓ | 44 | M5 | lightbox | 0.118 | 1.673 |
| frozen | ✓ | 44 | M5 | sunlamp | 0.279 | 1.881 |
| layer 4 | – | 42 | Colab | synthetic (val) | 0.039 | 0.306 |
| layer 4 | – | 42 | Colab | lightbox | 0.200 | 1.169 |
| layer 4 | – | 42 | Colab | sunlamp | 0.299 | 1.476 |
| layer 4 | – | 43 | Colab | synthetic (val) | 0.035 | 0.326 |
| layer 4 | – | 43 | Colab | lightbox | 0.152 | 1.182 |
| layer 4 | – | 43 | Colab | sunlamp | 0.352 | 1.619 |
| layer 4 | – | 44 | Colab | synthetic (val) | 0.035 | 0.318 |
| layer 4 | – | 44 | Colab | lightbox | 0.154 | 1.147 |
| layer 4 | – | 44 | Colab | sunlamp | 0.283 | 1.506 |
| layer 4 | ✓ | 42 | Colab | synthetic (val) | 0.041 | 0.352 |
| layer 4 | ✓ | 42 | Colab | lightbox | 0.152 | 0.986 |
| layer 4 | ✓ | 42 | Colab | sunlamp | 0.405 | 1.592 |
| layer 4 | ✓ | 43 | Colab | synthetic (val) | 0.044 | 0.302 |
| layer 4 | ✓ | 43 | Colab | lightbox | 0.165 | 0.997 |
| layer 4 | ✓ | 43 | Colab | sunlamp | 0.399 | 1.354 |
| layer 4 | ✓ | 44 | Colab | synthetic (val) | 0.042 | 0.311 |
| layer 4 | ✓ | 44 | Colab | lightbox | 0.132 | 0.977 |
| layer 4 | ✓ | 44 | Colab | sunlamp | 0.261 | 1.459 |
Table 11. Per-seed results behind Table 4. range-normalised, in rad (chance level 2.207). Head only: the head retrained on cached frozen features.
Conformal coverage, all arms and domains
Split-conformal regions are calibrated per arm on 1,000 held-out synthetic images, used neither for training nor for checkpoint selection, at nominal 90% coverage with the finite-sample-corrected quantile. Coverage carries Clopper–Pearson 95% intervals [3]. is the calibrated region size: a keep-out sphere radius for translation, a cone half-angle for rotation. Inflation is the factor by which would have to grow to restore 90% coverage on a domain, so it measures directly how wrong the calibration is. It sits near 1 in-domain and reaches 1.2–13 on the real domains, worst on the sharper layer-4 arms. A low factor on the frozen arms is not good news: their cones are already too wide in-domain to be informative. There are two kinds of in-domain row: in-domain test rows are a further 1,000 fresh synthetic images, and Synthetic (val) rows are the 2,000-image validation split used for checkpoint selection, kept as a second in-domain control that is not fully independent.3
Table 12 uses absolute scores: Euclidean metres for translation, geodesic radians for rotation. Table 13 repeats the translation analysis under the range-normalised score. Table 14 gives rotation coverage for the ablation arms and for every frozen-backbone head.
| Backbone | Aug. | Seed | Domain | Translation (sphere) Cov. | Infl. | Rotation (cone) Cov. | Infl. | ||
| frozen | – | 42 | in-domain test | 87.7 (85.5–89.7) | 0.712 | 1.11 | 90.0 (88.0–91.8) | 2.299 | 1.01 |
| frozen | – | 42 | synthetic (val) | 88.0 (86.5–89.4) | 0.712 | 1.08 | 90.5 (89.1–91.7) | 2.299 | 0.99 |
| frozen | – | 42 | lightbox | 52.1 (50.9–53.3) | 0.712 | 2.48 | 71.0 (69.9–72.1) | 2.299 | 1.24 |
| frozen | – | 42 | sunlamp | 29.2 (27.5–30.9) | 0.712 | 4.76 | 58.2 (56.3–60.0) | 2.299 | 1.28 |
| frozen | – | 43 | in-domain test | 90.7 (88.7–92.4) | 0.751 | 0.96 | 88.9 (86.8–90.8) | 2.215 | 1.04 |
| frozen | – | 43 | synthetic (val) | 90.5 (89.1–91.7) | 0.751 | 0.98 | 89.1 (87.7–90.5) | 2.215 | 1.03 |
| frozen | – | 43 | lightbox | 55.4 (54.2–56.6) | 0.751 | 2.32 | 67.0 (65.9–68.1) | 2.215 | 1.30 |
| frozen | – | 43 | sunlamp | 32.0 (30.3–33.8) | 0.751 | 4.95 | 59.1 (57.3–61.0) | 2.215 | 1.31 |
| frozen | – | 44 | in-domain test | 88.3 (86.1–90.2) | 0.704 | 1.05 | 88.7 (86.6–90.6) | 2.159 | 1.03 |
| frozen | – | 44 | synthetic (val) | 88.6 (87.1–90.0) | 0.704 | 1.06 | 88.4 (87.0–89.8) | 2.159 | 1.04 |
| frozen | – | 44 | lightbox | 53.4 (52.2–54.6) | 0.704 | 2.58 | 63.6 (62.4–64.7) | 2.159 | 1.33 |
| frozen | – | 44 | sunlamp | 30.6 (28.9–32.3) | 0.704 | 4.72 | 53.4 (51.6–55.3) | 2.159 | 1.36 |
| frozen | ✓ | 42 | in-domain test | 89.3 (87.2–91.1) | 0.793 | 1.01 | 88.7 (86.6–90.6) | 2.403 | 1.02 |
| frozen | ✓ | 42 | synthetic (val) | 89.9 (88.5–91.2) | 0.793 | 1.01 | 89.5 (88.0–90.8) | 2.403 | 1.02 |
| frozen | ✓ | 42 | lightbox | 68.3 (67.2–69.4) | 0.793 | 1.84 | 75.1 (74.1–76.2) | 2.403 | 1.19 |
| frozen | ✓ | 42 | sunlamp | 26.3 (24.6–27.9) | 0.793 | 5.36 | 71.0 (69.3–72.7) | 2.403 | 1.20 |
| frozen | ✓ | 43 | in-domain test | 89.7 (87.6–91.5) | 0.828 | 1.01 | 90.9 (88.9–92.6) | 2.415 | 0.98 |
| frozen | ✓ | 43 | synthetic (val) | 89.6 (88.2–91.0) | 0.828 | 1.01 | 88.9 (87.4–90.2) | 2.415 | 1.03 |
| frozen | ✓ | 43 | lightbox | 66.9 (65.7–68.0) | 0.828 | 1.91 | 75.5 (74.4–76.5) | 2.415 | 1.18 |
| frozen | ✓ | 43 | sunlamp | 23.4 (21.8–25.0) | 0.828 | 5.16 | 70.0 (68.3–71.7) | 2.415 | 1.20 |
| frozen | ✓ | 44 | in-domain test | 89.8 (87.8–91.6) | 0.839 | 1.00 | 91.1 (89.2–92.8) | 2.372 | 0.97 |
| frozen | ✓ | 44 | synthetic (val) | 89.8 (88.4–91.1) | 0.839 | 1.01 | 89.5 (88.1–90.8) | 2.372 | 1.01 |
| frozen | ✓ | 44 | lightbox | 71.1 (70.0–72.2) | 0.839 | 1.72 | 76.3 (75.2–77.3) | 2.372 | 1.18 |
| frozen | ✓ | 44 | sunlamp | 30.1 (28.4–31.9) | 0.839 | 4.69 | 69.8 (68.1–71.5) | 2.372 | 1.21 |
| layer 4 | – | 42 | in-domain test | 89.3 (87.2–91.1) | 0.410 | 1.04 | 88.8 (86.7–90.7) | 0.460 | 1.06 |
| layer 4 | – | 42 | synthetic (val) | 89.6 (88.2–90.9) | 0.410 | 1.01 | 87.2 (85.7–88.7) | 0.460 | 1.10 |
| layer 4 | – | 42 | lightbox | 34.1 (32.9–35.2) | 0.410 | 7.92 | 21.5 (20.5–22.5) | 0.460 | 5.52 |
| layer 4 | – | 42 | sunlamp | 18.2 (16.8–19.6) | 0.410 | 10.92 | 9.1 (8.1–10.3) | 0.460 | 6.07 |
| layer 4 | – | 43 | in-domain test | 90.2 (88.2–92.0) | 0.378 | 0.99 | 89.1 (87.0–91.0) | 0.540 | 1.04 |
| layer 4 | – | 43 | synthetic (val) | 89.2 (87.8–90.5) | 0.378 | 1.02 | 89.2 (87.8–90.5) | 0.540 | 1.05 |
| layer 4 | – | 43 | lightbox | 35.4 (34.3–36.6) | 0.378 | 6.41 | 26.7 (25.6–27.7) | 0.540 | 4.76 |
| layer 4 | – | 43 | sunlamp | 13.8 (12.6–15.2) | 0.378 | 13.42 | 9.0 (8.0–10.1) | 0.540 | 5.23 |
| layer 4 | – | 44 | in-domain test | 89.9 (87.9–91.7) | 0.368 | 1.01 | 89.7 (86.9–91.5) | 0.526 | 1.02 |
| layer 4 | – | 44 | synthetic (val) | 88.4 (87.0–89.8) | 0.368 | 1.05 | 90.5 (89.1–91.7) | 0.526 | 0.99 |
| layer 4 | – | 44 | lightbox | 32.9 (31.8–34.1) | 0.368 | 6.52 | 26.5 (25.4–27.5) | 0.526 | 4.71 |
| layer 4 | – | 44 | sunlamp | 18.0 (16.6–19.5) | 0.368 | 11.73 | 12.9 (11.7–14.2) | 0.526 | 5.30 |
| layer 4 | ✓ | 42 | in-domain test | 89.8 (87.8–91.6) | 0.425 | 1.01 | 91.2 (89.3–92.9) | 0.576 | 0.97 |
| layer 4 | ✓ | 42 | synthetic (val) | 88.8 (88.3–91.0) | 0.425 | 1.01 | 90.0 (88.7–91.3) | 0.576 | 1.00 |
| layer 4 | ✓ | 42 | lightbox | 44.3 (43.1–45.5) | 0.425 | 6.45 | 37.2 (36.1–38.4) | 0.576 | 3.82 |
| layer 4 | ✓ | 42 | sunlamp | 10.0 (8.9–11.2) | 0.425 | 12.79 | 12.6 (11.4–13.9) | 0.576 | 4.88 |
| layer 4 | ✓ | 43 | in-domain test | 90.4 (88.4–92.2) | 0.494 | 0.98 | 91.1 (89.2–92.8) | 0.483 | 0.96 |
| layer 4 | ✓ | 43 | synthetic (val) | 91.3 (90.0–92.5) | 0.494 | 0.95 | 89.8 (88.3–91.0) | 0.483 | 1.01 |
| layer 4 | ✓ | 43 | lightbox | 47.8 (46.6–49.0) | 0.494 | 6.15 | 32.6 (31.5–33.7) | 0.483 | 4.85 |
| layer 4 | ✓ | 43 | sunlamp | 10.7 (9.6–11.9) | 0.494 | 10.80 | 15.3 (14.0–16.7) | 0.483 | 5.59 |
| layer 4 | ✓ | 44 | in-domain test | 91.0 (89.1–92.7) | 0.460 | 0.98 | 91.4 (89.5–93.1) | 0.544 | 0.92 |
| layer 4 | ✓ | 44 | synthetic (val) | 91.0 (89.6–92.2) | 0.460 | 0.97 | 90.8 (89.4–92.0) | 0.544 | 0.96 |
| layer 4 | ✓ | 44 | lightbox | 51.0 (49.8–52.2) | 0.460 | 5.10 | 38.0 (36.9–39.2) | 0.544 | 4.19 |
| layer 4 | ✓ | 44 | sunlamp | 24.4 (22.9–26.1) | 0.460 | 9.21 | 15.4 (14.1–16.8) | 0.544 | 5.03 |
Table 12. Conformal coverage at nominal 90%, absolute scores, for every run in Table 4. Coverage in % with Clopper–Pearson 95% intervals, in metres and in radians.
| Backbone | Aug. | Seed | Domain | Coverage | (relative) | Infl. |
| frozen | – | 42 | in-domain test | 89.4 (87.3–91.2) | 0.1189 | 1.01 |
| frozen | – | 42 | synthetic (val) | 89.3 (87.9–90.7) | 0.1189 | 1.03 |
| frozen | – | 42 | lightbox | 49.6 (48.4–50.8) | 0.1189 | 2.42 |
| frozen | – | 42 | sunlamp | 21.7 (20.2–23.3) | 0.1189 | 3.85 |
| frozen | – | 43 | in-domain test | 90.3 (88.3–92.1) | 0.1217 | 0.99 |
| frozen | – | 43 | synthetic (val) | 91.2 (89.9–92.4) | 0.1217 | 0.94 |
| frozen | – | 43 | lightbox | 51.7 (50.5–52.9) | 0.1217 | 2.38 |
| frozen | – | 43 | sunlamp | 23.0 (21.5–24.6) | 0.1217 | 3.94 |
| frozen | – | 44 | in-domain test | 89.9 (87.9–91.7) | 0.1176 | 1.01 |
| frozen | – | 44 | synthetic (val) | 89.4 (88.0–90.7) | 0.1176 | 1.03 |
| frozen | – | 44 | lightbox | 50.6 (49.4–51.8) | 0.1176 | 2.50 |
| frozen | – | 44 | sunlamp | 22.1 (20.5–23.7) | 0.1176 | 3.74 |
| frozen | ✓ | 42 | in-domain test | 88.9 (86.8–90.8) | 0.1253 | 1.05 |
| frozen | ✓ | 42 | synthetic (val) | 89.6 (88.2–90.9) | 0.1253 | 1.01 |
| frozen | ✓ | 42 | lightbox | 66.2 (65.0–67.3) | 0.1253 | 1.78 |
| frozen | ✓ | 42 | sunlamp | 15.7 (14.3–17.1) | 0.1253 | 4.20 |
| frozen | ✓ | 43 | in-domain test | 86.2 (83.9–88.3) | 0.1259 | 1.10 |
| frozen | ✓ | 43 | synthetic (val) | 88.5 (87.0–89.9) | 0.1259 | 1.04 |
| frozen | ✓ | 43 | lightbox | 61.7 (60.5–62.9) | 0.1259 | 1.94 |
| frozen | ✓ | 43 | sunlamp | 13.4 (12.1–14.7) | 0.1259 | 4.46 |
| frozen | ✓ | 44 | in-domain test | 88.6 (86.5–90.5) | 0.1306 | 1.04 |
| frozen | ✓ | 44 | synthetic (val) | 89.5 (88.1–90.8) | 0.1306 | 1.02 |
| frozen | ✓ | 44 | lightbox | 67.8 (66.7–68.9) | 0.1306 | 1.69 |
| frozen | ✓ | 44 | sunlamp | 19.1 (17.7–20.6) | 0.1306 | 3.82 |
| layer 4 | – | 42 | in-domain test | 91.0 (89.1–92.7) | 0.0787 | 0.96 |
| layer 4 | – | 42 | synthetic (val) | 90.2 (88.8–91.5) | 0.0787 | 0.99 |
| layer 4 | – | 42 | lightbox | 34.1 (32.9–35.2) | 0.0787 | 6.07 |
| layer 4 | – | 42 | sunlamp | 13.5 (12.2–14.8) | 0.0787 | 6.93 |
| layer 4 | – | 43 | in-domain test | 92.0 (90.1–93.6) | 0.0739 | 0.93 |
| layer 4 | – | 43 | synthetic (val) | 91.3 (90.0–92.5) | 0.0739 | 0.95 |
| layer 4 | – | 43 | lightbox | 35.5 (34.4–36.7) | 0.0739 | 4.70 |
| layer 4 | – | 43 | sunlamp | 9.0 (8.0–10.2) | 0.0739 | 8.07 |
| layer 4 | – | 44 | in-domain test | 89.7 (87.6–91.5) | 0.0678 | 1.01 |
| layer 4 | – | 44 | synthetic (val) | 89.2 (87.8–90.6) | 0.0678 | 1.03 |
| layer 4 | – | 44 | lightbox | 32.2 (31.1–33.3) | 0.0678 | 5.11 |
| layer 4 | – | 44 | sunlamp | 12.0 (10.9–13.3) | 0.0678 | 7.68 |
| layer 4 | ✓ | 42 | in-domain test | 88.8 (86.7–90.7) | 0.0814 | 1.07 |
| layer 4 | ✓ | 42 | synthetic (val) | 90.5 (89.1–91.7) | 0.0814 | 0.99 |
| layer 4 | ✓ | 42 | lightbox | 43.7 (42.5–44.9) | 0.0814 | 4.58 |
| layer 4 | ✓ | 42 | sunlamp | 5.2 (4.4–6.1) | 0.0814 | 7.99 |
| layer 4 | ✓ | 43 | in-domain test | 90.3 (88.3–92.1) | 0.0911 | 1.00 |
| layer 4 | ✓ | 43 | synthetic (val) | 90.8 (89.5–92.1) | 0.0911 | 0.98 |
| layer 4 | ✓ | 43 | lightbox | 46.3 (45.1–47.5) | 0.0911 | 4.48 |
| layer 4 | ✓ | 43 | sunlamp | 5.7 (4.9–6.6) | 0.0911 | 6.96 |
| layer 4 | ✓ | 44 | in-domain test | 90.5 (88.5–92.2) | 0.0894 | 0.99 |
| layer 4 | ✓ | 44 | synthetic (val) | 91.0 (89.7–92.2) | 0.0894 | 0.97 |
| layer 4 | ✓ | 44 | lightbox | 51.4 (50.2–52.6) | 0.0894 | 3.60 |
| layer 4 | ✓ | 44 | sunlamp | 18.7 (17.2–20.2) | 0.0894 | 5.74 |
Table 13. Conformal coverage at nominal 90%, translation scored with the range-normalised error .
| Run | Seed | In-domain test | Synthetic (val) | Lightbox | Sunlamp | |
| A: no blur/noise | 42 | 90.5 | 91.1 | 33.2 | 16.0 | 0.558 |
| A: no blur/noise | 43 | 89.2 | 88.8 | 33.9 | 16.8 | 0.538 |
| A: no blur/noise | 44 | 90.4 | 90.5 | 31.8 | 18.7 | 0.540 |
| B: jitter, gamma, erasing | 42 | 89.7 | 89.0 | 30.7 | 14.5 | 0.478 |
| B: jitter, gamma, erasing | 43 | 91.4 | 90.0 | 36.1 | 14.4 | 0.539 |
| B: jitter, gamma, erasing | 44 | 90.5 | 90.2 | 28.7 | 12.6 | 0.481 |
| heads: 4k, logits | 42–44 | 89.1–92.4 | 89.5–92.2 | 85.3–87.1 | 83.0–85.3 | 2.81–2.86 |
| heads: 4k, pooled | 42–44 | 88.5–91.9 | 88.5–92.5 | 77.0–83.2 | 71.0–77.7 | 2.57–2.71 |
| heads: 28k, logits | 42–44 | 87.7–91.4 | 89.8–92.0 | 74.3–80.4 | 67.5–76.9 | 2.50–2.67 |
| heads: 28k, pooled | 42–44 | 88.7–91.8 | 88.4–92.0 | 63.6–72.2 | 53.4–64.5 | 2.16–2.35 |
| heads: tuned | 42–44 | 88.6–90.4 | 89.5–92.0 | 64.1–72.7 | 52.0–61.8 | 2.15–2.31 |
| heads: synthetic BN | 42–44 | 89.4–90.1 | 89.3–90.5 | 73.6–77.1 | 72.2–74.2 | 2.64–2.67 |
Table 14. Rotation coverage (%) at nominal 90% for the ablation arms (per seed) and the frozen-backbone heads (range over seeds 42–44), each calibrated on its own predictions. in rad.
Constant-predictor baseline
The chance level of rad assumes orientations drawn uniformly from , which SPEED+ poses are not. This appendix gives the empirical counterpart quoted in § 3: the score of a single fixed pose. It anchors “near chance” to a predictor one could actually build, rather than to an assumption about the distribution.
The constant is fitted on the 28,000 training labels only. Fitting it on a test domain would make it an oracle rather than a baseline, so neither real domain enters the estimate. For rotation we take the principal eigenvector of , which minimises chordal distance and is unaffected by the quaternion double cover. We then refine it with five Karcher iterations on the sphere: sign-align each to the current mean, log-map into the tangent space, average, and exp-map back. The geodesic distance is twice the one, so iterating on the sphere targets the same minimiser as our reported metric. For translation we take the arithmetic mean of the training translations. This gives up to sign and m.
Two caveats belong with the numbers. First, the Karcher steps do not contract sharply here, because the training orientations cover most of the sphere, which leaves the Fréchet functional close to flat and the mean weakly identified. This does not affect the result: the chordal and Karcher estimates lie 0.022 rad apart and agree to 0.001 on every score in Table 15. Second, both estimators minimise squared error, whereas we report mean unsquared error, whose minimisers are the corresponding geometric medians. The scores below are therefore achievable rather than optimal, and they are an upper bound on the error of the best constant predictor. We do not claim optimality, and no conclusion in the paper needs it.
| Domain | Skill | |||
| synthetic (val) | 2,000 | 0.415 | 2.176 | 0.031 |
| lightbox | 6,740 | 0.367 | 2.203 | 0.005 |
| sunlamp | 2,791 | 0.372 | 2.199 | 0.009 |
Table 15. A constant pose fitted on the training split, evaluated on each test domain. range-normalised, in radians. Skill is 2.207 − . Reported for the Karcher-refined estimate.
The empirical no-skill rotation level therefore sits at most 0.031 rad from the uniform one on all three domains, and we use 2.207 in the body. Every trained run behind Table 4 keeps some skill over it, even the weakest, seed 42 of the frozen un-augmented arm on sunlamp, which scores 2.048 rad and keeps 0.15 rad.
Translation is less comfortable, and it is where the second caveat matters most. On sunlamp the constant scores 0.372, which two of the three augmented layer-4 seeds (0.405, 0.399) do not beat, while a constant fitted directly to the unsquared metric does better on translation, scoring 0.330 on sunlamp, and matches this one on rotation to within 0.001. We quote the Karcher-refined constant throughout, as the workshop version did, because no conclusion changes with the tighter one: the same two seeds fail to beat it, and every other comparison the paper draws with the constant still holds.
Revision experiments
All runs in this appendix were added after review and pre-registered: predictions and decision rules were written down and committed before any result was opened (App. § 18), and selection uses synthetic validation only, as in the body.
Frozen-backbone block. The frozen backbone runs in inference mode, so without augmentation its pooled features (and the ImageNet logits computed from them) can be cached once, and heads retrained on them in minutes. Applying the published seed-42 head to the cache reproduces its published errors to within 0.0005. Table 5 gives three seeds (42, 43, 44) for each configuration, all with the published head, loss and selection rule. The 4,000-image cells use one fixed draw from the 28,000, not the original 4,000-image set. The tuned head is the best of learning rate epochs on seed 42, chosen on synthetic validation. The synthetic-BN head re-estimates every BatchNorm layer on the un-augmented synthetic training images, with the weights untouched, before caching.
Frozen backbone with augmentation. Augmented inputs cannot use the feature cache, so seeds 43 and 44 of the frozen augmented arm were trained end to end with the released code on an Apple M5 GPU, while seed 42 is the published Colab run. Table 6 pairs each seed with the head-only retrain of the same seed in the published configuration (Table 5), and their conformal coverage is in App. § 12.
Augmentation ablation. Arm A is the full stack without blur and noise. Arm B keeps only brightness and contrast jitter, gamma and random erasing (no glare blob, blur or noise). Both train layer 4 on seeds 42, 43 and 44 with the published configuration, and we read them as per-seed differences. They were trained with the released code on an Apple M5 GPU, whereas the published arms were trained on Colab. A control rerun of the full stack (seed 42) on the M5 differs from its published counterpart by , and rad in (synthetic, lightbox, sunlamp). That is within seed variation, but on sunlamp it is as large as the ablation effects themselves.
Test-time BatchNorm adaptation (AdaBN). For each layer-4 checkpoint we re-estimate only layer 4’s BatchNorm statistics, the ones fitted on synthetic data. We use the unlabelled images of one half of each real domain (a fixed random split by filename), leave every weight untouched, and evaluate on the other half (3,370 lightbox and 1,396 sunlamp images), with each arm’s published synthetic-calibrated for coverage. This is test-time adaptation: it uses target images, though not their labels. The frozen arms are excluded, as all their statistics are ImageNet’s.
Image statistics. Figure 2 compares pooled intensity densities, which cannot tell a soft bright blob from a hard edge. Table 8 adds spatial statistics, pooled over every image of each split after the same resize and grayscale conversion (augmentation seed 7, as in Figure 2):
the Sobel gradient magnitude, scaled so a full black-to-white step reads 1
edge density, the share of pixels whose gradient exceeds 0.1
the clipped fraction, the share of pixels at or above 254/255
the total variation (TV) between pooled densities (48 intensity bins, 64 gradient bins)
The intensity TV of the full stack reproduces Figure 2 (0.63, 0.80).
Rotation-error shape
For each layer-4 checkpoint and real domain we measure two things, both pre-registered (App. § 18). The first is the share of rotation errors within 0.3 rad of . Under uniform guessing the angle density, , rises towards , so chance alone puts 0.19 of the mass there. The second is how concentrated the axes of those near- errors are, in the body frame of and in the camera frame . We measure it as the share of axes within of their principal axis, the leading eigenvector of the axes’ second-moment matrix. A flip about a near-symmetry axis would concentrate the body-frame axes, whereas an in-plane confusion about the boresight would concentrate the camera-frame ones. Because the principal axis is estimated from the same sample, the share’s chance level is not the 0.06 of a fixed axis. Simulation with uniform random axes gives 0.074–0.079 at these sample sizes (95th percentile about 0.10).
Coverage by range bin
For each run we calibrate on the calibration split exactly as in § 9, hold it fixed, and measure coverage within five range bins. The bin edges are quintiles of the calibration split’s range, so no test image shapes them. In-domain bins hold 173–234 images each, and the per-bin counts sum to the marginal coverage on every run. Tables 17, 18 and 19 give each arm’s coverage per bin, averaged over its seeds, for the metre sphere, the range-normalised sphere and the attitude cone.
| Arm | Domain | < 3.8 | 3.8–5.3 | 5.3–6.7 | 6.7–8.3 | ≥ 8.3 |
| frozen, no aug. | in-domain | 97.5 | 95.7 | 90.2 | 82.9 | 75.2 |
| frozen, no aug. | lightbox | 73.3 | 70.7 | 50.5 | 37.4 | 28.6 |
| frozen, no aug. | sunlamp | 63.6 | 53.0 | 22.2 | 6.6 | 1.4 |
| frozen, full stack | in-domain | 97.4 | 95.0 | 92.1 | 90.9 | 75.6 |
| frozen, full stack | lightbox | 91.3 | 84.1 | 64.4 | 58.2 | 45.6 |
| frozen, full stack | sunlamp | 81.7 | 39.4 | 11.6 | 3.0 | 0.1 |
| layer 4, no aug. | in-domain | 92.9 | 95.0 | 92.7 | 89.8 | 80.9 |
| layer 4, no aug. | lightbox | 58.2 | 39.1 | 27.8 | 22.8 | 24.9 |
| layer 4, no aug. | sunlamp | 58.4 | 23.1 | 3.5 | 0.6 | 0.1 |
| layer 4, full stack | in-domain | 92.0 | 94.1 | 93.6 | 89.3 | 84.8 |
| layer 4, full stack | lightbox | 81.0 | 53.6 | 39.2 | 33.4 | 33.9 |
| layer 4, full stack | sunlamp | 57.7 | 16.0 | 3.4 | 0.8 | 0.0 |
Table 17. Coverage (%) of the marginal 90% region by range bin (m), translation, metres, as the mean over the seeds of each arm. Bins are quintiles of the calibration split’s range.
| Arm | Domain | < 3.8 | 3.8–5.3 | 5.3–6.7 | 6.7–8.3 | ≥ 8.3 |
| frozen, no aug. | in-domain | 77.3 | 93.5 | 89.0 | 90.7 | 95.7 |
| frozen, no aug. | lightbox | 38.7 | 56.2 | 50.1 | 48.5 | 54.0 |
| frozen, no aug. | sunlamp | 31.8 | 39.6 | 21.9 | 9.7 | 5.3 |
| frozen, full stack | in-domain | 74.5 | 88.5 | 88.6 | 94.3 | 92.9 |
| frozen, full stack | lightbox | 66.3 | 69.6 | 60.1 | 66.6 | 63.1 |
| frozen, full stack | sunlamp | 45.0 | 22.2 | 9.8 | 4.0 | 0.6 |
| layer 4, no aug. | in-domain | 73.9 | 91.9 | 95.8 | 96.4 | 96.4 |
| layer 4, no aug. | lightbox | 35.1 | 33.7 | 31.1 | 32.3 | 38.3 |
| layer 4, no aug. | sunlamp | 35.0 | 18.0 | 4.6 | 1.0 | 0.2 |
| layer 4, full stack | in-domain | 67.7 | 89.5 | 96.0 | 97.4 | 98.3 |
| layer 4, full stack | lightbox | 57.0 | 48.3 | 42.8 | 41.9 | 47.0 |
| layer 4, full stack | sunlamp | 33.7 | 12.3 | 3.6 | 1.0 | 0.1 |
Table 18. Coverage (%) of the marginal 90% region by range bin (m), translation, range-normalised, as the mean over the seeds of each arm. Bins are quintiles of the calibration split’s range.
| Arm | Domain | < 3.8 | 3.8–5.3 | 5.3–6.7 | 6.7–8.3 | ≥ 8.3 |
| frozen, no aug. | in-domain | 96.1 | 91.4 | 93.6 | 87.3 | 83.3 |
| frozen, no aug. | lightbox | 75.9 | 74.2 | 71.1 | 69.3 | 64.2 |
| frozen, no aug. | sunlamp | 64.2 | 56.5 | 61.4 | 56.2 | 52.5 |
| frozen, full stack | in-domain | 92.0 | 92.8 | 92.9 | 90.1 | 84.9 |
| frozen, full stack | lightbox | 78.9 | 78.2 | 76.0 | 73.5 | 71.4 |
| frozen, full stack | sunlamp | 74.2 | 70.3 | 70.0 | 70.0 | 67.1 |
| layer 4, no aug. | in-domain | 86.0 | 93.9 | 95.0 | 91.7 | 81.8 |
| layer 4, no aug. | lightbox | 22.8 | 29.0 | 27.5 | 24.5 | 19.7 |
| layer 4, no aug. | sunlamp | 11.1 | 11.6 | 11.3 | 10.3 | 7.1 |
| layer 4, full stack | in-domain | 86.2 | 95.5 | 95.4 | 92.8 | 87.7 |
| layer 4, full stack | lightbox | 35.6 | 40.9 | 38.3 | 34.5 | 29.5 |
| layer 4, full stack | sunlamp | 17.3 | 17.7 | 16.8 | 11.0 | 9.3 |
Table 19. Coverage (%) of the marginal 90% region by range bin (m), rotation, as the mean over the seeds of each arm. Bins are quintiles of the calibration split’s range.
Reproducibility
Code, results and predictions. Everything is in the public repository, https://github.com/lokindoo/spacecraft-pose-estimation. The tag arxiv-v1 reproduces this version, and the tag sim2science-2026-camera-ready the workshop version. Zenodo archives every tag under doi:10.5281/zenodo.24011140, which lists each version, and the workshop version alone under doi:10.5281/zenodo.24011141. Each tag holds its paper’s source, with a script that checks every number in the paper against the repository’s results/. Per-image predictions for all 20 runs behind the tables are released as CSV, so a new metric needs neither the data pipeline nor a GPU.
Runs. Configurations, seeds and the selection rule are as in § 3. The eight published runs (seed 42 of both frozen arms, and all six layer-4 runs) were trained on a free-tier Colab GPU. Frozen augmented seeds 43 and 44, ablation arms A and B (three seeds each) and the control were trained with the released package on an Apple M5 GPU. The frozen-backbone heads were trained on cached features on the same machine’s CPU, in minutes each. On Colab a 20-epoch run took about 1.8 h for the frozen un-augmented arm and 2.0–2.3 h for the layer-4 un-augmented arm. With augmentation, epochs took about 790 s, some 4.4 h per run, whether or not the backbone was frozen. On the M5 most runs took 3.1–4.2 h, and two arm-A runs took about 14 h of wall-clock time. A head on cached features trains in about 2.5 minutes.
Checkpoints. results/checkpoints.json records, for each of the 17 trained checkpoints, its configuration, seed, selected epoch, hardware and sha256. The weights are on Zenodo with a model card, under CC BY 4.0: doi:10.5281/zenodo.24140385.
Software. requirements.txt pins the versions the pipeline was written against (PyTorch 2.13.0, torchvision 0.28.0). The M5 runs used Python 3.14.8, PyTorch 2.14.1 and torchvision 0.29.1, frozen in requirements-m5.txt. The Colab runs used the Colab image of July 2026, whose versions were not recorded. The control rerun (§ 6) bounds what hardware and software together moved seed 42, and none of the 81 unit tests needs data.
Pre-registration record
Every experiment added after review had its prediction and decision rule committed to version control before its result was opened. The two files are published verbatim in the repository’s preregistration/, with every later edit and its time. The revision predictions were committed on 28 Oct 2026 at 17:42 UTC. The extension’s decision rules followed on 30 Oct at 11:31 and its predictions at 11:54, and its results were opened at 13:36 the same day. These are commit times in the author’s own repository, not a third-party registration.
Deviations and corrections. Frozen augmented seeds 43 and 44 were pre-registered for this version but finished after the camera-ready’s results freeze. Once their outcome was known they were added to the camera-ready anyway, the only result to cross the freeze. The range-bin rule also left “monotone” undefined, as Table 20 notes.
| Experiment | Rule | Outcome |
| Revision (locked 28 Oct) | ||
| Arm A | Sunlamp, A − full stack, 3/3 negative: keep “blur and noise work against sunlamp outright” | 2/3 (−0.184, +0.006, −0.117): softened to a hypothesis |
| Arm B | Sunlamp, B − A, 3/3 negative: the glare sentence gains evidence | 1/3 (−0.018, +0.077, +0.072): softened |
| Control | Within 2 s.d. of published seed 42 on both real domains | Lightbox +0.006 (±0.020), sunlamp −0.142 (±0.239): inside; pairing holds |
| Frozen seeds | Sunlamp s.d. ≥ 0.025: “only 0.03 rad” becomes “no detectable change” | s.d. 0.031: reworded |
| Decomposition | One factor ≥ 2/3 of the synthetic gain: name it; either factor moves sunlamp by > 2 frozen s.d.: split the claim | Data 0.33, features 0.29 (53%); largest sunlamp move 0.024 against 0.062: neither |
| Head sweep | Beats published seed 42 by > 2 frozen s.d.: “under-trained”; recovers ≥ 50% of the layer-4 gain on a real domain: restate | 0.913 against 1.053: tuned head reported; real domains unmoved: finding stands |
| Synthetic-BN head | Share of the layer-4 gain recovered: ≥ 50% / 0–50% / ≤ 0 | −0.51 (lightbox), −0.27 (sunlamp): “not explained by normalisation statistics alone” |
| AdaBN | Improved if 6/6 checkpoints share the sign; coverage < 80%: “coverage stays low” | 6/6 on both domains; 5/6 lightbox, 6/6 sunlamp; coverage at most 40.4% |
| Image statistics | Sunlamp clipped more than the full stack, and sharper than lightbox and the full stack; else two passages become hypotheses | Clipping holds (2.78% vs 1.96%); edge density fails (2.0% vs 6.6%, 10.7%): passages reworded |
| Extension (locked 30 Oct) | ||
| Frozen + aug | Synthetic aug − no aug, 3/3 positive: keep “the reference degrades”; sunlamp 3/3 negative and larger than lightbox: qualify the directional claim | +0.169, +0.161, +0.134: kept; sunlamp −0.128, −0.069, −0.132, larger than lightbox on every seed: qualified |
| Error shape | Near-π share ≥ 0.28 on all six: bimodal; body cone ≥ 0.18 on all six and camera not: flips; reverse: in-plane | Near-π 0.03–0.10: not bimodal; body cone ≥ 0.18 on 2/6 per domain: neither |
| Range bins | Both in-domain translation patterns hold and the extreme bin excludes 90%: keep; monotone but intervals include 90%: soften; not monotone: cut | Ambiguous: “monotone” undefined. Near-to-far, both hold and the extreme bin excludes 90% on 7/8 and 8/8; strictly over five bins, 3/8 and 2/8. Reported as near and far numbers |
| Ablation coverage | Coverage opposite to error on ≥ 5/6 per domain: “coverage mirrors error” | 4/6 on each domain: not claimed |
Table 20. Pre-registered decision rules and their outcomes. Differences are paired by seed and given in rad, and “3/3” means all three seeds share the sign.
Not pre-registered. Three results in the body were found or computed outside the pre-registration: the transfer plot (Figure 1), AdaBN’s effect on translation (§ 8) and the collapse of sunlamp translation coverage with range (§ 9). Every point in the transfer plot had already been published.
References
- [1]Anastasios N. Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023.
- [2]Bo Chen, Jiewen Cao, Alvaro Parra, and Tat-Jun Chin. Satellite pose estimation with deep landmark regression and nonlinear pose refinement. In IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 2816–2824, 2019.arxiv.org/abs/1908.11542
- [3]Charles J. Clopper and Egon S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934.DOI
- [4]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
- [5]Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 2021.arxiv.org/abs/1903.04684
- [6]Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016.arxiv.org/abs/1505.07818
- [7]Isaac Gibbs and Emmanuel Candès. Adaptive conformal inference under distribution shift. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2106.00170
- [8]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.arxiv.org/abs/1512.03385
- [9]Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
- [10]Philip T. Jackson, Amir Atapour-Abarghouei, Stephen Bonner, Toby P. Breckon, and Boguslaw Obara. Style augmentation: Data augmentation via style randomization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
- [11]Alex Kendall, Matthew Grimes, and Roberto Cipolla. PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In IEEE International Conference on Computer Vision (ICCV), pages 2938–2946, 2015.arxiv.org/abs/1505.07427
- [12]Mate Kisantal, Sumant Sharma, Tae Ha Park, Dario Izzo, Marcus Märtens, and Simone D’Amico. Satellite pose estimation challenge: Dataset, competition design, and results. IEEE Transactions on Aerospace and Electronic Systems, 56(5):4083–4098, 2020.DOI
- [13]Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018.arxiv.org/abs/1604.04173
- [14]Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80:109–117, 2018.DOI
- [15]John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2107.04649
- [16]Zachary Nado, Shreyas Padhy, D. Sculley, Alexander D’Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift, 2020. arXiv:2006.10963.arxiv.org/abs/2006.10963
- [17]Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- [18]Tae Ha Park and Simone D’Amico. Robust multi-task learning and online refinement for spacecraft pose estimation across domain gap. Advances in Space Research, 73(11):5726–5740, 2024.DOI
- [19]Tae Ha Park, Sumant Sharma, and Simone D’Amico. Towards robust learning-based pose estimation of noncooperative spacecraft. In AAS/AIAA Astrodynamics Specialist Conference, 2019. arXiv:1909.00392.arxiv.org/abs/1909.00392
- [20]Tae Ha Park, Marcus Märtens, Gurvan Lecuyer, Dario Izzo, and Simone D’Amico. SPEED+: Next-generation dataset for spacecraft pose estimation across domain gap. In IEEE Aerospace Conference, 2022. Dataset: Zenodo record 5588480.DOI
- [21]Tae Ha Park, Marcus Märtens, Mohsi Jawaid, Zi Wang, Bo Zhao, Tat-Jun Chin, Dario Izzo, and Simone D’Amico. Satellite pose estimation competition 2021: Results and analyses. Acta Astronautica, 204:640–665, 2023.DOI
- [22]Pedro F. Proença and Yang Gao. Deep learning for spacecraft pose estimation from photorealistic rendering. In IEEE International Conference on Robotics and Automation (ICRA), pages 6007–6013, 2020.arxiv.org/abs/1907.04298
- [23]Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.16971
- [24]Sumant Sharma and Simone D’Amico. Neural network-based pose estimation for noncooperative spacecraft rendezvous. IEEE Transactions on Aerospace and Electronic Systems, 56(6):4638–4658, 2020.DOI
- [25]Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2007.00644
- [26]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1904.06019
- [27]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 23–30, 2017.arxiv.org/abs/1703.06907
- [28]Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.arxiv.org/abs/1804.06516
- [29]Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conference on Machine Learning (ACML), 2012.arxiv.org/abs/1209.2673
- [30]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005.
- [31]Heng Yang and Marco Pavone. Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2303.12246
- [32]Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5745–5753, 2019.arxiv.org/abs/1812.07035