WorksComputer Science
Weather Adaptive Autonomous Driving
Abstract
Camera object detectors lose recall in rain, fog, snow and sand, and the confidence threshold at which a detector is deployed decides how many objects it misses and how many false alarms it raises. We study how a frozen detector should make this decision in adverse weather. We first reproduce SeqCRC, a conformal object detector whose first step calibrates the confidence threshold by conformal risk control (CRC), and use that step, calibrated separately in each weather condition, as our baseline. With YOLOv8x it keeps the held-out miss fraction within a 20% budget in every DAWN weather bin, where a single global threshold fails in fog, but the false alarms it needs are an output rather than a choice: from 1.6 to 117 per image across DAWN and BDD100K conditions. We then propose a calibrated trade-off dial. One setting, , the number of false alarms accepted to avoid one missed object, selects the expected-cost-optimal decision rule: either one score threshold, or a per-detection rule that keeps a box when its calibrated probability of being a real object, estimated by gradient-boosted trees from the detector's own per-box outputs, is at least . F-score preferences map onto the same dial, and CRC applied to itself is planned to restore a per-condition miss guarantee. The dial is fitted once on BDD100K train and graded only on BDD100K val and DAWN.
Introduction
Autonomous vehicles must detect objects reliably when rain, fog, snow, sand, glare, and low light alter the camera image. A detector can lose recall under these conditions even when its architecture and confidence scores remain unchanged. Lowering a fixed confidence threshold may recover detections, but it can also produce an unacceptable number of false alarms. The two errors do not cost the same: a false alarm may cost an unnecessary slowdown, whereas a missed pedestrian or vehicle can cost a collision, and a miss leaves no trace at run time.
Most adverse-weather detection research retrains the detector and reports AP or mAP, which integrate over score thresholds and leave the deployed operating point unstated [1, 2, 3, 4]. Conformal methods bound what a detector misses at its operating point [5], [6], but they assume that calibration and test images are exchangeable, which persistent weather breaks, and outside medicine [7] they do not report the false positives that a recall guarantee costs.
This work asks whether a frozen camera detector can be wrapped with a decision rule that adapts that trade-off while providing a finite-sample miss-rate control. We first use a per-weather conformal risk control (CRC) threshold, the first step of SeqCRC [6] calibrated separately in each weather condition, as a baseline. It meets its miss budget in every weather bin, but its price ranges from 1.6 to 117 false alarms per image, and nothing lets a user choose that price. We therefore propose a calibrated per-detection trade-off dial, in which the price is the input. So far, this work makes the following contributions:
Reproduction. We reproduce SeqCRC with its official code on COCO with YOLOv8x and trace its main discrepancy, localization sets smaller than reported, most likely to how the official YOLO wrapper rescales boxes when images are batched (Section IV).
Baseline and its price. Calibrated per weather bin, the CRC threshold holds a 20% miss budget in every DAWN weather bin, where one global threshold fails in fog, and restores the budget in every BDD100K night bin, at 1.6 to 117 false alarms per image (Section III-A).
A calibrated trade-off dial. A single setting, , the false alarms accepted per avoided miss, selects the expected-cost-optimal rule: one threshold, or a per-detection cut on a calibrated probability estimated from the frozen YOLOv8x’s own per-box outputs. weights and a CRC miss budget map onto the same dial (Sections III-B–III-E).
An evaluation plan on BDD100K val and DAWN, which are never used for fitting, with baselines, ablations and a compute budget (Section III-F).
Section II reviews related work and states the research gap. Section III describes the baseline, the proposed dial, its architecture, algorithms and design choices, and the experiment plan. Section IV gives the experimental setup and the reproduction of the baseline, and Appendix A lists the reproduction issues and their fixes.
Related Work
Our project asks whether a confidence threshold chosen per weather condition can recover the recall that a fixed camera detector loses in adverse weather, with a finite-sample bound on the expected fraction of missed objects and a stated false-positive cost. Four lines of work bear on this question: methods and benchmarks that make detection robust to weather, studies of how detectors’ confidences and operating thresholds behave under shift, distribution-free guarantees for detection, and risk control under shift or within groups. Table I lists the method, input, data, headline result and main limitation of each paper we discuss.
Weather-Robust Detection and How It Is Measured
Most adverse-weather detection research changes the detector or its input [1, 2, 3, 4]. Sakaridis et al. [1] render homogeneous fog onto clear Cityscapes images with the Koschmieder model, creating Foggy Cityscapes. Domain-adaptive detectors use such synthetic fog without target labels: DA Faster R-CNN [2] aligns image- and instance-level features adversarially, while Li et al. [3] add hardness weighting and a triplet loss for synthetic rain. Image-adaptive YOLO [4] instead predicts differentiable defogging, colour, tone, and sharpening filters jointly with YOLOv3. These methods improve detection on their target benchmarks, but their results depend on retraining and are usually reported as AP or mAP rather than at a specified deployment threshold. Real adverse-weather test sets also remain relatively small: Foggy Driving has 101 real images [1], and DAWN [8] contains real traffic images in fog, rain, snow and sand. The paper that introduces DAWN also proposes a visibility-enhancement front end, a multi-scale CNN detector and a GM-PHD tracker, compared with 21 vehicle detectors; DAWN’s four subsets define our weather conditions.
The common limitation is that these methods retrain the network for a particular shift [1, 2, 3, 4]. Because AP and mAP integrate over score thresholds, they do not directly state the miss rate or false-positive cost at a deployed operating point. Each method therefore yields a detector that still needs an operating point, which makes this line complementary to ours.
Confidence, Calibration and Operating Thresholds
A second line studies whether detector scores can be trusted and where to cut them. Oksuz et al. [9] argue that a deployed detector must be judged at its operating threshold and build a self-aware detector whose class-wise thresholds are tuned on in-distribution validation data. Trained on nuImages and tested on BDD45K, their Faster R-CNN variant errs mostly by missing objects at the selected operating point. Kuzucu et al. [10] show that common protocols score calibration and accuracy on different detection sets, and evaluate both on the thresholded detections a system would deploy. On Foggy Cityscapes, isotonic regression improves localization-aware calibration, yet fog still degrades localization performance. Read together, the two studies show that shift hurts a detector at its operating point, mainly through misses [9], and that recalibration can repair confidence while accuracy still falls [10]. Yet both tune class-wise thresholds once on validation data from the training distribution and carry them to shifted test data, so neither guarantees a miss rate in the shifted condition.
Distribution-Free Guarantees for Detection
Conformal risk control (CRC) [5] chooses the least conservative value of a scalar parameter whose corrected calibration risk is at most a budget ; for exchangeable data and a bounded loss that is non-increasing in the parameter and within budget at its most permissive value, the expected test loss is then at most . Conformal methods for detection differ mainly in which quantity they calibrate, and that choice decides whether objects the detector never finds are covered. Andéol et al. [11] calibrate box margins for railway-signal detection at a fixed objectness threshold of 0.3, with image-wise recall losses that need far smaller margins than a Hausdorff-coverage baseline, and state that they do not consider false positives. Timans et al. [12] first build conformal class sets and then box intervals; their intervals approach the target coverage on COCO but only for objects detected above a fixed score. AR-CP [13] is the one conformal method we found aimed at adverse driving conditions: it builds class prediction sets for agents detected on the ROAD dataset and replaces large sets by their common parent class for an augmented-reality display, with coverage reported over all conditions together. SeqCRC [6] calibrates the confidence threshold itself and then the box and class sets, so that undetected objects enter the guaranteed quantity, but it does not bound a false-positive quantity. Outside driving, Hulsman et al. [7] apply single-parameter CRC to the score threshold of a frozen 3-D lung-nodule detector with a per-scan false-negative-rate loss and report the corresponding false-positive price, which varies substantially across annotation-consensus subsets.
Across this line, a guarantee counts missed objects only when the confidence threshold itself is calibrated with a recall-type loss [6], [7], or when boxes at a fixed threshold are enlarged until they cover them, which fails on images without any detection: the railway budget is unattainable for two of its three pretrained detectors [11]. Coverage of matched detections [12], [13] ignores missed objects altogether. All of these studies assume that calibration and test images are exchangeable, splitting the pool at most by object class [12]. The closest to per-condition calibration is the separate calibration of each annotation-consensus subset in [7], but these subsets are nested label definitions, not deployment conditions. That study is also the only one to report false positives at its guaranteed recall; SeqCRC reports only how many boxes it keeps.
Risk Control Under Shift and Within Groups
Weather persists for minutes and may be rare in a calibration mix, so the test condition is not a random draw from the pool. Under arbitrary shift, the CRC bound loosens by the loss bound times the summed total-variation distance between calibration and test distributions [5]. Non-exchangeable CRC [14] down-weights calibration points believed to be far from the test point and improves control under synthetic changepoints, but the total-variation penalty in its bound is unknown in practice, so the weights are heuristics. Blot et al. [15] keep exchangeability but recast CRC as an optimization over a class of functions of the input, which yields risk control within each group when the class is spanned by group indicators; with a learned embedding they meet a segmentation recall target at higher precision than marginal CRC. The two routes differ in what they need: weighting leaves an unknown total-variation term in its bound, whereas group conditioning needs only a group label for each image, which DAWN’s weather subsets supply. Neither has been applied to the confidence threshold of an object detector under weather.
| Paper | Method | Sensor / input | Dataset | Key result | Limitation |
| Weather-robust detection and benchmarks | |||||
| Sakaridis [1], IJCV’18 | Physics-based synthetic fog; fog fine-tuning | RGB camera | Foggy Cityscapes; Foggy Driving (101 real) | Car AP 30.5→35.3 on 33 finely annotated real images | Homogeneous daytime fog; tiny real test set |
| Hassaballah [8], T-ITS’21 | Visibility enhancement + multi-scale CNN + GM-PHD tracker; introduces DAWN | RGB camera | DAWN (fog, rain, snow, sand), KITTI, COCO | Reported to outperform 21 compared vehicle detectors | Small real-image set (1,027 released images); no official splits |
| Chen [2], CVPR’18 | Adversarial image + instance alignment (DA Faster R-CNN) | RGB camera | Cityscapes→Foggy Cityscapes; SIM10k; KITTI | Foggy [email protected] 18.8→27.6 | Retrains per shift; synthetic fog; AP/mAP only, no operating point |
| Li [3], WACV’23 | Hardness-weighted GRL + rainy-domain triplet loss (DA-Detect) | RGB camera | Cityscapes→Foggy Cityscapes; KITTI | Foggy [email protected] 23.41→42.34 (oracle 45.6) | Synthetic fog only; 3.5-pt hyperparameter swing; no operating point |
| Liu [4], AAAI’22 | CNN-predicted differentiable filters before YOLOv3 | RGB camera | VOC-Foggy/Dark; RTTS; ExDark | RTTS mAP 28.82→37.08 (30.80 for same-data YOLOv3) | Retrains detector; mAP only |
| Confidence calibration and operating thresholds | |||||
| Oksuz [9], CVPR’23 | Self-aware detector: image rejection, calibration, LRP-optimal thresholds | RGB camera | nuImages→BDD45K, BDD45K-C | BDD45K (F-RCNN, IoU 0.1): 1−recall 58.1%, 1−precision 13.2% | Val-tuned thresholds, no guarantee; weather pooled |
| Kuzucu [10], ECCV’24 | Calibration and accuracy on the thresholded set; Platt/isotonic baselines | RGB camera | COCO, Cityscapes, LVIS; corrupted, Foggy Cityscapes, Obj45K | Foggy Cityscapes LaECE 18.5→10.0 (isotonic) | In-distribution thresholds reused under shift; no guarantee |
| Distribution-free guarantees for detection | |||||
| Angelopoulos [5], ICLR’24 | CRC: scalar parameter bounding expected monotone loss | Any model’s scores | Polyps, COCO, ImageNet, NQ | COCO mean FNR 0.0996 at (1,000 splits) | Marginal only; shift needs weights or TV slack |
| Andeól [11], COPA’23 | Conformal box margins; image-wise recall CRC | RGB cab-view camera | Railway signals, 3,414 images | DiffusionDet stretch 1.56 (pixel CRC) vs. 9.83 (Hausdorff CP), risk 0.09 | Fixed 0.3 threshold; ignores false positives |
| Timans [12], ECCV’24 | Two-step CP: class sets, then box intervals | Detector outputs (RGB) | COCO, Cityscapes, BDD100K | COCO box coverage 0.88 (target 0.9) vs. 0.21 deep ensemble | Detected objects only; misses uncovered |
| Doula [13], CVPRW’24 | Conformal class sets for detected agents; taxonomy parent for AR display | RGB camera video | ROAD; user study (ROAD, Seeing Through Fog, Gated2Gated) | APS set size 6.16→1.00 at coverage 0.90 () | Classes of detected agents only; coverage pooled over conditions |
| Andeól [6], arXiv’25 | Sequential CRC: confidence threshold, then box/class sets | Detector outputs (RGB) | COCO val (2,500 cal./2,500 test) | YOLOv8x box-count-recall risk 0.019 (=0.02), 11.7 boxes/img | False positives not bounded; i.i.d. COCO only |
| Hulsman [7], COPA’25 | CRC on a frozen detector’s score threshold; per-scan miss-rate loss | 3-D thoracic CT | LIDC-IDRI, four reader-consensus subsets | ≥3 readers: 91.35% sensitivity at 2.25 FP/scan () | Exchangeability assumed; 118 FP/scan on low-consensus set |
| Risk control under shift and within groups | |||||
| Farinhas [14], ICLR’24 | Weighted CRC with total-variation penalty | Any model’s scores | Synthetic time series; ELEC2; NQ | FNR 0.246→0.196 at (changepoints) | TV penalty unknown; no detection |
| Blot [15], AISTATS’25 | CRC conditional on a function class (incl. group indicators) | Any model’s scores (images) | Polyp and fire segmentation; simulated regression | Polyp recall 0.906 (target 0.9); precision 0.395→0.457 | Assumes exchangeability; no detection or weather shift |
Table I. Related work at a glance (one row per paper; AP and mAP as defined in each paper).
Research Gap
Prior work leaves three gaps. First, weather-robust detectors [1]–[4] improve AP or mAP but leave the operating point and its miss rate unstated, and threshold-selection studies [9], [10] tune class-wise thresholds on in-distribution data without a guarantee. Second, conformal detection methods [6], [11]–[13] calibrate under an exchangeability assumption, at most split by object class, and, except in medicine [7], do not report how many false positives a recall guarantee costs; the one study in adverse driving conditions covers only the classes of objects already detected [13]. Third, group-conditional risk control has been developed [15], but to our knowledge it has not been used to calibrate a frozen camera detector’s confidence threshold per weather condition.
Our work addresses all three. We keep a COCO-pretrained YOLOv8 detector fixed. As a baseline, we apply CRC separately in each weather condition of DAWN with an IoU- and class-matched per-image miss loss, compare these thresholds with one pooled threshold, report the false positives per image at every budget, and flag budgets that no threshold can certify because some objects are never detected. We then make the false-positive price an input rather than an output: one setting , the number of false alarms accepted per avoided miss, selects the expected-cost-optimal rule, either one threshold or a per-detection cut on a calibrated probability that each box is real, and conformal risk control applied to itself (planned) restores a per-condition miss guarantee. Unlike perception-side methods, our approach needs no retraining and can wrap any of them; unlike prior conformal detection, its guarantee is stated per weather condition and comes with its false-positive price.
Methodology
Baseline: per-weather conformal threshold (Phase 1)
Our baseline is the first step of SeqCRC [6]: conformal risk control (CRC) [5] on the confidence threshold of a frozen detector, with an image-level loss , the fraction of ground-truth objects in calibration image with no matched detection scoring at least . Calibrated separately in each weather bin with calibration images and mean calibration loss , the threshold is
which bounds the expected miss fraction of an exchangeable test image from the same bin by . In Phase 1 we reproduced SeqCRC with its official code on COCO, where the global test risk stayed within its budget in every configuration we ran (see Section IV for our Phase 1 report), and ran it on DAWN [16]. Our harness implements Eq. (1) per weather bin on YOLOv8x [17]. With on DAWN, the per-bin threshold keeps the held-out miss fraction at 0.188–0.194 in every bin, at 1.6–6.2 false positives per image (FPPI), while one global threshold misses 0.251 in fog (above in 98% of 200 splits). On BDD100K [18], a threshold calibrated on clear days misses 0.28–0.36 at night; per-bin thresholds restore the budget at 43–117 FPPI.
a) Limitation (the gap): The baseline fixes the miss budget and lets the false alarms be whatever results: at night it buys with up to 117 false alarms per image, and there is no way to ask for a different balance. It applies one score cut to every box, it needs the weather bin at run time (our camera-only weather estimators did not isolate fog), and its guarantee assumes exchangeable data within each bin.
Proposed method: a calibrated trade-off dial
Core idea. Instead of one confidence threshold per weather bin, every detection receives a calibrated probability that it is a real object, computed from the frozen detector’s own outputs, and is kept if and only if . The single user-facing parameter , the number of false alarms accepted to avoid one missed object, sets the trade-off for the use case. An F-score weight (and, in Phase 4, a CRC miss budget ) maps onto the same dial.
Table II lists what changes relative to the baseline. The detector, data, matching rule and evaluation protocol stay the same; only the decision of which boxes to keep is replaced.
| Baseline (SeqCRC step 1 per weather bin) | Ours (trade-off dial) | |
| Detector | YOLOv8x, frozen COCO weights, 1280 px, score floor 0.001 | unchanged |
| Decision rule | one score threshold per weather bin, | per detection, ; simple option: one threshold |
| Trade-off | fixed miss budget ; false alarms unpriced | explicit input (or ); planned as a floor via CRC over |
| Run-time inputs | score and the weather bin | score, box context and the detector’s per-box outputs (no weather label) |
| Fitting | calibration half of each bin | once, on 21,831 labelled BDD100K train frames per-box feature tap; calibrated model; map; label-free estimate of false alarms |
| Added | — |
Table II. What our method keeps, replaces and adds relative to the Phase-1 baseline.
Architecture
Figure 1 shows the full pipeline. Online, YOLOv8x runs once per camera frame and returns candidate boxes with class and score . We also keep the detector’s per-box outputs at the anchor each box came from: the 80 class probabilities, the four distribution-focal-loss (DFL) distributions over the box sides [19], the neck feature vector, and the pyramid level. A feature builder forms : 20 context features (score, class, size and position, rank, overlap with stronger boxes, crowding) plus 48 per-box features (six class probabilities, strongest other class, top-two margin, sum and entropy of the probabilities, DFL entropy per side and mean, level, and 32 principal components of the neck feature). A gradient-boosted model maps to , and the dial keeps or drops each box. Offline, the same frozen detector is run once over BDD100K train, boxes are labelled by matching, and the dial is fitted, checked and then graded on data never used for fitting.
Figure 1. Architecture, following one BDD100K val frame (rain at night) through the online path. New components are orange, unchanged ones grey; the dashed grey lane is the Phase-1 baseline, kept for comparison, and the dashed blue box is planned for Phase 4. Fitting (green band) uses only BDD100K train; grading uses BDD100K val and DAWN.
Algorithm
a) Cost: For a keep rule over the detections of image , let be the number of ground-truth objects left without a kept matched box and the number of kept boxes that match no object. The dial setting defines the cost
b) Option A: one threshold: with
Greedy non-maximum suppression and greedy matching are both score-ordered, so filtering one detector run at equals running the detector at . (3) is therefore solved exactly from cumulative counts: the minimisers for all are the corners of the lower convex hull of (real kept, false kept), stored as 258 switch points.
c) Option B: per detection: Keeping a box with probability of being real costs an expected false alarms; dropping it costs an expected in misses. The cost-optimal rule is therefore [20]
which is optimal when is calibrated. Because the kept set grows with , the miss count is non-increasing in . We estimate with gradient-boosted trees [21, 22, 23]. We train on all train detections by minimising the log-loss , where comes from greedy IoU-0.5 matching with grouped classes (vehicle, two-wheeler, person). We check calibration with the expected calibration error [24] and with the share of real boxes among detections near each cut .
d) F-score units: With calibrated probabilities, the -optimal cut is , where is the best achievable [25, 26]. Setting this cut equal to gives the equivalent dial setting
with measured on train for each option. For example, corresponds to and to with the per-detection option.
e) Planned: a miss budget on top of the dial (CRC over ): Because the image-level miss fraction is non-increasing in , the baseline’s rule (1) applies to itself. Per condition with calibration images,
and deploying keeps the CRC guarantee under exchangeability within the condition, while never being less cautious than the user asked. The fit is infeasible when is below the miss fraction at the score floor (11–18% of objects are never detected).
Design choices
tbl:3 gives each design choice, the reason for it, and the alternatives we tried or considered. Most of the alternatives were run in the harness, several under pre-registered criteria.
Experiment plan (Phase 4)
a) Data and splits: The dial is fitted on 21,831 BDD100K train frames (3.34 M detections), with 5-fold out-of-fold estimates grouped by near-duplicate group. It is graded on BDD100K val (8,090 frames in seven weather-by-time bins) and on DAWN (889 frames with clean labels: fog 256, rain 187, sand 243, snow 203). Neither set is used for any choice. For CRC over we use 200 random calibration/test halves per condition, keeping near-duplicate groups together, as in the baseline runs.
b) Metrics: The main metric is the cost relative to one threshold, with 95% image-bootstrap intervals [27]. We also report recall, precision and false alarms per image at ; at against no tuning and the best in-sample constant; and calibration (expected calibration error, share real near each cut). For CRC over $\lambda
Algorithm 1 Fitting the dial (offline, BDD100K train only)
Require: labelled train frames ; frozen detector
for all do
{floor 0.001, NMS 0.5, boxes; keep per-box outputs}
greedy IoU match of to labels {grouped classes}
end for
Option A: sort scores; keep ends of equal-score blocks; cumulative real and false counts
corners of the lower convex hull {all at once}
Option B: [context, per-box] {PCA fitted on train}
gradient boosting on {log-loss}
5-fold out-of-fold probabilities, folds by near-duplicate group
for all do
{F-score map, on train}
end for
check calibration near every cut and fold stability; adopt only if the pre-registered criteria hold 14.
return , (with PCA),
Algorithm 2 Applying the dial (online, every camera frame)
Require: frame ; setting (or ); option
if is given then
{F-score map}
end if
if a budget is certified for condition then
{planned: CRC over }
end if
{one detector pass}
for all do
if option A then
keep iff
else
; keep iff {cost-optimal cut}
end if
end for
{expected false alarms, no labels needed} 16.
return kept detections,
we report the held-out miss fraction against (share of splits above), its price in false alarms per image, and the smallest certifiable .
Baselines: (i) No tuning (, the Ultralytics default). (ii) One global CRC threshold. (iii) The per-weather CRC threshold, (1) (Phase-1 baseline). (iv) Option A, the one-threshold dial. (v) The context-only per-detection model. As reference bounds that no deployable rule can reach, we also report the best constant threshold chosen in-sample per condition and the per-frame hindsight best.
d) Ablations: (a) Retraining without each feature group: class probabilities, neck components, DFL entropy, level, context. (b) Per-condition recalibration, which targets the under-stated probabilities in DAWN fog (17% real at the 9.1% cut). (c) Training-set size: 25, 50 and 100% of frames. (d) Detector size and input resolution: YOLOv8x at 640 versus 1280, and YOLOv8s. (e) CRC over versus CRC over at equal . Already completed: model 1 versus model 2, per-weather thresholds, and dedicated models.
e) Compute: Measured on our laptop GPU (RTX 3050 Ti, 4 GB):
YOLOv8x at 1280 px takes 0.15–0.17 s per frame;
extracting per-box outputs for all 30.9k frames takes about 85 minutes;
fitting the per-detection model takes about 11 minutes on the CPU;
the CRC splits take minutes on the CPU.
We will use an NRP A100 for the new detector passes in ablation (d) and for the repeated fits in (a)–(c). At an estimated the laptop’s throughput, these take a few GPU-hours in total. The laptop remains a fallback for every step.
Experimental Setup and Baseline Reproduction
Before building the dial we reproduced its baseline, SeqCRC [6], on its own benchmark: a COCO-pretrained YOLOv8x [17] calibrated on MS-COCO [28], with the authors’ official code (cods, https://github.com/leoandeol/cods, commit 269ff11). SeqCRC trains nothing: its “training” is the calibration of three scalars on 2,500 images, and evaluation is on the other 2,500. Our repository, with the run scripts, logs, result files and a patch to cods, is https://github.com/Nicholas55555/seqcrc-baseline.
Setup
a) Hardware: A local laptop instead of a hosted GPU: an NVIDIA GeForce RTX 3050 Ti Laptop GPU (4 GB, driver 566.07, CUDA 12.7), an AMD Ryzen 7 4800H (8 cores), 32 GB RAM and Windows 11. Detection ran on the GPU (a one-off 390 s pass); calibration ran on the CPU. The weather experiments of Section III use the same laptop (Section III-F).
b) Software: Python 3.11.15 (uv), PyTorch 2.14.1 and torchvision 0.29.1 (CUDA 12.6 wheels), Ultralytics 8.4.177, NumPy 2.4.6, numba 0.68.0, pycocotools 2.0.11 and scikit-learn 1.9.1; cods 0.4 at commit 269ff11 plus our patch. The repository pins every version in requirements.txt.
c) Data: The full COCO val2017 set (5,000 images, SHA-256-checked): the first 2,500 images calibrate and the last 2,500 test, which is the official code’s unshuffled 50/50 split. A 300/300 subset (first 300 for calibration, next 300 for test) served for fast checks.
d) Protocol: As in the paper (its Sec. V-A): the official yolov8x.pt at Ultralytics’ default 640 px (the paper states neither), NMS IoU 0.5, confidence floor , total budget split as , and mix-matching weight 0.25. The paper’s appendix (Table VII) reports 92 YOLOv8x configurations at this budget. We ran four of them in full (F1–F4), chosen to cover both confidence losses, the Hausdorff, LAC and Mix matchings, pixel-wise and box-wise localization losses, additive and multiplicative boxes, and APS and LAC class sets.
Reproduction Results
Table IV compares our runs with the paper. On the full 2,500/2,500 split the confidence and classification steps reproduce closely: the box-count-recall step keeps 11.53 boxes per image in F1 and F3 (paper 11.71) and the box-count-threshold step 18.02 in F2 and F4 (paper 18.86), and the class sets hold 40.6, 0.75, 66.8 and 38.3 classes in F1–F4 (paper 41.7, 0.72, 67.6 and 40.8). Test risks sit at or near their budgets, as in the paper: global risks are 0.085, 0.083, 0.0997 and 0.088 against the 0.1 budget (paper 0.082, 0.086, 0.098 and 0.078), and a few single-split task risks are slightly above theirs (at most 0.055 against 0.05), as is the paper’s own F3 classification risk (0.053). Localization is the exception: our stretches are smaller in every configuration (F1: 1.30 against 3.87; F3: 1.24 against 4.54; F4: 2.56 against 6.12), which Section IV-C traces to the batch size.
| Matching / conf. loss / loc. loss-set / class set | Source | Confidence size | Confidence risk | Localization size | Localization risk | Classification size | Classification risk | Global risk |
| F1 Hausdorff / bc-recall / pixel-add / APS | Paper | 11.71 | 0.019 | 3.87 | 0.049 | 41.73 | 0.043 | 0.082 |
| Ours, full | 11.53 | 0.020 | 1.30 | 0.053 | 40.64 | 0.042 | 0.085 | |
| Ours, 300/300 | 11.11 | 0.025 | 1.30 | 0.040 | 42.67 | 0.033 | 0.066 | |
| F2 LAC / bc-thresh / pixel-add / LAC | Paper | 18.86 | 0.012 | 20.64 | 0.047 | 0.72 | 0.049 | 0.086 |
| Ours, full | 18.02 | 0.015 | 17.69 | 0.047 | 0.75 | 0.046 | 0.083 | |
| Ours, 300/300 | 14.29 | 0.047 | 13.56 | 0.046 | 1.02 | 0.045 | 0.080 | |
| F3 Mix / bc-recall / pixel-mult / LAC | Paper | 11.71 | 0.019 | 4.54 | 0.049 | 67.58 | 0.053 | 0.098 |
| Ours, full | 11.53 | 0.020 | 1.24 | 0.052 | 66.82 | 0.055 | 0.100 | |
| Ours, 300/300 | 11.11 | 0.025 | 1.24 | 0.038 | 66.06 | 0.061 | 0.093 | |
| F4 Mix / bc-thresh / box-add / APS | Paper | 18.86 | 0.012 | 6.12 | 0.043 | 40.75 | 0.042 | 0.078 |
| Ours, full | 18.02 | 0.015 | 2.56 | 0.053 | 38.28 | 0.043 | 0.088 | |
| Ours, 300/300 | 14.29 | 0.047 | 2.21 | 0.055 | 40.66 | 0.038 | 0.085 |
Table IV. SeqCRC with YOLOv8x on COCO val2017, : set size and test risk per task (confidence: boxes per image, budget 0.02; localization: box stretch, budget 0.05; classification: classes per set, budget 0.05) and global risk (budget 0.1). Paper: its Appendix Table VII. Full: the same 2,500/2,500 split as the paper’s code. 300/300: first 300 images for calibration, next 300 for test.
Discrepancies
a) Localization sets: Our localization sets are much smaller than reported for YOLOv8x, while the class sets and box-count-recall confidence sets are close. The likely cause is in the official YOLO wrapper (cods/od/models/yolo.py): it rescales the raw head output by the ratio of original to letterboxed image size and never removes the letterbox padding. With batch size 1 (our runs) Ultralytics pads each image only to a multiple of 32, so the error is small; with a batch of differently sized images it pads every image to 640 × 640 and the boxes land far from the objects. Table V shows the effect directly: the same wrapper scores 45.1 AP at batch 1 but 10.7 AP (AP75 1.4) at batch 8. Re-running F3 on the 300/300 subset with batch 8 raises the stretch from 1.24 to 4.70, close to the paper’s 4.54, while the confidence and class sets barely move (11.11 vs. 11.46 boxes, 66.1 vs. 65.7 classes). The paper’s best YOLOv8x stretches (3.87–5.45, its Table III) are also 3–4 times those of DETR-101 (1.04–1.55), whose wrapper rescales normalized boxes and is unaffected. The paper does not state its batch size, but the official tutorial notebook calls build_predictions with batch_size=12, so its YOLOv8x localization numbers were most likely produced with a batch size above 1. This rests on one configuration on 300/300 images; a full-size batch-8 run did not finish in time.
| Choice | Why | Alternatives considered |
| Keep the detector frozen; decide after it | calibration stays valid; one detector run serves every setting; no retraining per condition | domain adaptation (DA-Detect: 20.41 vs 21.44 AP@50 on real DAWN fog in Phase 1); fine-tuning (invalidates every calibration) |
| Trade-off as , false alarms per avoided miss | use case sets the balance; the optimal rule is explicit (Eq. 4); and map onto it | fixed only (baseline, false alarms unpriced); maximise (one fixed balance); FPPI cap |
| Decide per detection | 87–91% of a per-image oracle’s advantage on BDD100K is chance; the learnable signal is in individual boxes | per-image thresholds from pixels, from the confidence profile, or from YOLOv8x image features (all failed pre-registered tests); box-combination rules; summed-confidence maps |
| One threshold kept as the simple option | one auditable number per ; per-weather thresholds give no gain (0.98–1.03× the cost) | per-weather thresholds (the baseline’s form) |
| Gradient-boosted trees on detector outputs | tabular inputs; trains in minutes on CPU; calibrated without post-hoc scaling (expected calibration error 0.11 points on BDD100K val) | recalibrating the score alone (this is Option A); a neural head; retraining YOLOv8x’s head |
| YOLOv8x’s own per-box outputs as features | pre-registered gain: cost 0.972× (0.969–0.975) of the context-only model on BDD100K, not worse on DAWN (0.995) | context features only (model 1); a separate CNN on box crops (more compute) |
| One model for every | theory: only the cut moves with ; dedicated -weighted models tie (0.997–1.001×) | one cost-weighted model per |
| Fit on BDD100K train; grade on BDD100K val and DAWN | nothing is chosen on graded data; DAWN tests transfer to real fog, rain, sand and snow | calibrating on halves of the test pool (the baseline protocol) |
| Grouped classes in matching | car/truck confusions are not safety misses | class-aware matching (run 1) |
Table III. Design choices, the reasons for them, and the alternatives considered (with the evidence).
| Boxes scored | AP | AP50 | AP75 |
| Ultralytics model table (paper’s detector) | 53.9 | – | – |
| Ours: Ultralytics predict, NMS IoU 0.7 per class | 53.2 | 69.8 | 58.0 |
| Ours: Ultralytics predict, class-agnostic NMS 0.5 | 51.6 | 68.8 | 55.6 |
| Ours: cods wrapper, batch 1 (SeqCRC input) | 45.1 | 65.9 | 48.2 |
| Ours: cods wrapper, batch 8 | 10.7 | 32.1 | 1.4 |
Table V. Detector check: COCO box AP of YOLOv8x on all 5,000 val2017 images (pycocotools).
b) The detector itself: Through Ultralytics we measure 53.2 AP against the published 53.9 (Table V; we use predict() with per-image letterboxing and pycocotools, not val()). The boxes SeqCRC actually receives score 45.1: class-agnostic NMS at IoU 0.5 (the paper’s IoU; the cods NMS is class-agnostic) costs 1.6 AP, and the wrapper’s box rescaling (padding not removed, separate and scales, no clipping) costs the remaining 6.5. Its scores and classes are the same maximum sigmoid score and argmax that Ultralytics uses. We kept the official wrapper because the paper’s numbers come from it.
c) One split: The paper reports a single calibration/test draw, as we do. The guarantee holds in expectation over draws, so single-split risks may sit slightly above or below the budget; on 300 test images the fluctuations are larger (e.g., box-count-threshold risk 0.047 against 0.02).
d) Software drift: We ran the code at commit 269ff11 (May 22, 2026; library code last changed Oct. 22, 2025) with Ultralytics 8.4.177; the paper’s runs used earlier versions, which it does not state. Our two speed patches changed no output: the mask selection is exact, and the float64 running sum moved calibration risks by about , leaving every output of a 300/300 check identical (Appendix A).
Reproduction Issues and Fixes
These are the problems we met while reproducing SeqCRC (Section IV) and how we solved them. The patch, scripts and logs are in our repository.
Hard-coded path and logging. The official evaluation script,
cods/od/evaluate.py, reads COCO from/datasets/shared_datasets/coco/and requires Weights & Biases. Fix: ourscripts/run_seqcrc.pyrepeats itsrun_experimentstep for step with a--cocoargument and JSON-lines output, and adds the paper’s confidence floor, whichevaluate.pynever applies (the tutorial notebook uses ).
Calibration crashes at the current commit with a
TypeError(“expected Tensor ... but got list”). Lint commite70d773(Oct. 20, 2025) turnedx = list([...])intox = ([...],)inbuild_predictions, wrapping every per-image list in a 1-tuple; the tutorial notebook only works from older cached predictions. Fix: unwrap the tuples whenODPredictionsis built (no upstream edit).
4 GB GPU and device mismatch. Unfiltered YOLOv8x outputs (hundreds of boxes per image, each with a 91-way class vector) do not fit, and ground truth and boxes ended up on different devices. Fix: apply the paper’s floor right after NMS (the same box set), keep boxes on the CPU, then move all predictions to one device; this leaves 50.8 boxes per test image on average (median 30, maximum 741).
Calibration too slow for a laptop. A 50/50-image smoke test was still calibrating after about 12 min on the GPU and was stopped. Profiling with
py-spypointed to (a) a Python loop over every box for every candidate threshold and (b) a full re-summation of all per-image losses after each single update, overall (100 images: 294 s; 300: 1,151 s; extrapolated to 2,500: about 8 h per configuration). Fix (patches/cods-reproduction.patch): boolean-mask indexing, which selects the same rows in the same order, and a running sum of the losses. Unit tests show identical selections and sums (within ), and a 300/300 run with and without the running sum gives the same step-1 thresholds to every printed digit. Calibration then runs on the CPU, one thread per process. The four full runs
ran in parallel, with diagnostic runs alongside, so their wall-clock times (143–253 min) were measured under CPU contention.
Smaller snags. With 10 calibration images cods reports “No good lambda found”, correctly: SeqCRC needs , so smoke tests use . On Windows,
pythonwas the Microsoft Store stub and console output failed oncp1252; we used a uv-managed CPython andPYTHONIOENCODING=utf-8.
Figure 2 shows excerpts of the saved logs for one full-size SeqCRC run and for the detector check.

Figure 2. Evidence: excerpts of the saved logs (lines selected and wrapped, text unchanged; the first line names the command). Top: a full-size SeqCRC run, F3 (logs/10_full_F3.log). Bottom: COCO evaluation of YOLOv8x through Ultralytics (logs/11_detector_sanity_native.log).
References
- [1]C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” Int. J. Comput. Vis., vol. 126, no. 9, pp. 973–992, 2018.DOI
- [2]Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, “Domain adaptive Faster R-CNN for object detection in the wild,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 3339–3348.
- [3]J. Li, R. Xu, J. Ma, Q. Zou, J. Ma, and H. Yu, “Domain adaptive object detection for autonomous driving under foggy weather,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), 2023, pp. 612–622.arxiv.org/abs/2210.15176
- [4]W. Liu, G. Ren, R. Yu, S. Guo, J. Zhu, and L. Zhang, “Image-adaptive YOLO for object detection in adverse weather conditions,” in Proc. AAAI Conf. Artif. Intell. (AAAI), 2022, pp. 1792–1800.arxiv.org/abs/2112.08088
- [5]A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster, “Conformal risk control,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2024, pp. 559 198–55 218.
- [6]L. Andéol, L. Mossina, A. Mazoyer, and S. Gerchinovitz, “Conformal object detection by sequential risk control,” arXiv:2505.24038, 2025, arXiv preprint.arxiv.org/abs/2505.24038
- [7]R. Hulsman, V. Comte, L. Bertolini, T. Wiesenthal, A. Puertas Gallardo, and M. Ceresa, “Conformal risk control for pulmonary nodule detection,” in Proc. Symp. Conformal Probabilistic Predict. Appl., ser. Proc. Mach. Learn. Res., vol. 266, Sep. 2025, pp. 445–463.arxiv.org/abs/2412.20167
- [8]M. Hassaballah, M. A. Kenk, K. Muhammad, and S. Minaee, “Vehicle detection and tracking in adverse weather using a deep learning framework,” IEEE Trans. Intell. Transp. Syst., vol. 22, no. 7, pp. 4230–4242, 2021.DOI
- [9]K. Oksuz, T. Joy, and P. K. Dokania, “Towards building self-aware object detectors via reliable uncertainty quantification and calibration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 9263–9274.arxiv.org/abs/2307.00934
- [10]S. Kuzucu, K. Oksuz, J. Sadeghi, and P. K. Dokania, “On calibration of object detectors: Pitfalls, evaluation and baselines,” in Proc. Eur. Conf. Comput. Vis. (ECCV), ser. Lect. Notes Comput. Sci., vol. 15079, 2024, pp. 185–204.arxiv.org/abs/2405.20459
- [11]L. Andéol, T. Fel, F. de Grancey, and L. Mossina, “Confident object detection via conformal prediction and conformal risk control: an application to railway signaling,” in Proc. Symp. Conformal Probabilistic Predict. Appl., ser. Proc. Mach. Learn. Res., vol. 204, Sep. 2023, pp. 36–55.arxiv.org/abs/2304.06052
- [12]A. Timans, C.-N. Straehle, K. Sakmann, and E. Nalisnick, “Adaptive bounding box uncertainties via two-step conformal prediction,” in Proc. Eur. Conf. Comput. Vis. (ECCV), ser. Lect. Notes Comput. Sci., vol. 15146, 2024, pp. 363–398.arxiv.org/abs/2403.07263
- [13]A. Doula, M. Mühlhäuser, and A. Sanchez Guinea, “AR-CP: Uncertainty-aware perception in adverse conditions with conformal prediction and augmented reality for assisted driving,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), 2024, pp. 216–226.DOI
- [14]A. Farinhas, C. Zerva, D. Ulmer, and A. F. T. Martins, “Non-exchangeable conformal risk control,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2024.arxiv.org/abs/2310.01262
- [15]V. Blot, A. N. Angelopoulos, M. I. Jordan, and N. J.-B. Brunel, “Automatically adaptive conformal risk control,” in Proc. Int. Conf. Artif. Intell. Statist. (AISTATS), ser. Proc. Mach. Learn. Res., vol. 258, 2025, pp. 19–27.arxiv.org/abs/2406.17819
- [16]M. A. Kenk and M. Hassaballah, “DAWN: Vehicle detection in adverse weather nature dataset,” arXiv preprint arXiv:2008.05402, 2020.DOI
- [17]G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” https://github.com/ultralytics/ultralytics, 2023.DOI
- [18]F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “BDD100K: A diverse driving dataset for heterogeneous multitask learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 2633–2642.arxiv.org/abs/1805.04687
- [19]X. Li, W. Wang, L. Wu, S. Chen, X. Hu, J. Li, J. Tang, and J. Yang, “Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 33, 2020, pp. 21002–21012.arxiv.org/abs/2006.04388
- [20]C. Elkan, “The foundations of cost-sensitive learning,” in Proc. Int. Joint Conf. Artif. Intell. (IJCAI), 2001, pp. 973–978.
- [21]J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” Ann. Statist., vol. 29, no. 5, pp. 1189–1232, 2001.DOI
- [22]G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “LightGBM: A highly efficient gradient boosting decision tree,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017, pp. 3146–3154.
- [23]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay, “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.arxiv.org/abs/1201.0490
- [24]C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res., vol. 70, 2017, pp. 1321–1330.arxiv.org/abs/1706.04599
- [25]M.-J. Zhao, N. Edakunni, A. Pocock, and G. Brown, “Beyond Fano’s inequality: Bounds on the optimal F-score, BER, and cost-sensitive risk and their implications,” J. Mach. Learn. Res., vol. 14, pp. 1033–1090, 2013.
- [26]Z. C. Lipton, C. Elkan, and B. Naryanaswamy, “Optimal thresholding of classifiers to maximize F1 measure,” in Proc. Eur. Conf. Mach. Learn. Knowl. Discov. Databases (ECML PKDD), ser. Lect. Notes Comput. Sci., vol. 8725. Springer, 2014, pp. 225–239.pubmed.ncbi.nlm.nih.gov/26023687
- [27]B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993.
- [28]T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in Proc. Eur. Conf. Comput. Vis. (ECCV), ser. Lect. Notes Comput. Sci., vol. 8693, 2014, pp. 740–755.arxiv.org/abs/1405.0312
