WorksComputer Science
Region-aware error triage for medical image segmentation with groupwise conformal risk control
Abstract
Segmentation masks are increasingly reviewed rather than redrawn, which requires knowing where a mask is likely wrong. Learned error maps rank pixels by risk, but the number of pixels to warn is chosen ad hoc, and calibrating one warning threshold over all error pixels may leave errors far from the predicted contour under-flagged. We propose RACE-Triage, a post-hoc error triage framework that leaves the segmentation mask unchanged. A risk network is trained on out-of-fold predictions from backbone-independent inputs. The predicted contour partitions each image into a boundary band and a non-boundary region, and conformal risk control is applied separately in each region to bound the expected fraction of missed errors among cases with errors there. On ISIC 2018 dermoscopy and BUSI breast ultrasound (where patient identifiers are unavailable, so independence across images is assumed), pooled control met its overall target but missed on average 31.8% and 36.4% of non-boundary errors. Regional control reduced these fractions to 19.3% and 20.9% and raised the recall of size-filtered false-negative components from 0.722 to 0.818 and from 0.611 to 0.763. The mean warned area grew by 1.6 and 4.9 percentage points, and its 95th percentile reached 23.5% and 38.5%. The effect held across three seeds and a second backbone, and area-matched heuristics did not reproduce it. On the external PH set, frozen thresholds exceeded both targets, whereas recalibration with 100 local images brought the mean over repeated splits below them. Regional calibration improves non-boundary error coverage at a measured increase in warned area.
Introduction
Deep segmentation networks are now routine components of medical image analysis pipelines, from convolutional encoder–decoders [1, 2] to transformer-based and foundation models [3, 4]. In dermoscopy and breast ultrasound, recent architectures place particular emphasis on lesion boundaries and low-contrast margins [5, 6, 7], and boundary-oriented training objectives target contour accuracy directly [8, 9]. Even accurate models, however, produce masks that contain errors, and in practice a clinician or annotator still has to decide whether a mask can be used as it is. Inspecting every pixel of every image defeats the purpose of automation. A more useful system would indicate where the mask is likely to be wrong, so that review effort is spent where errors are concentrated. We refer to this task as post-segmentation error triage.
Uncertainty estimation is the usual starting point. Predictive entropy, Monte Carlo dropout, deep ensembles and test-time augmentation provide pixel-wise uncertainty maps [10, 11, 12, 13], and uncertainty-aware training has been proposed to align confidence with correctness [14]. Uncertainty and error are nevertheless different targets. Uncertainty maps of segmentation networks concentrate along predicted contours [15, 16], while confidently mislabelled regions, such as a missed lesion lobe or a spurious detection, can receive low uncertainty. Learned error and quality predictors address this gap more directly [17, 18, 19, 20]. A score map alone, however, does not specify how many pixels to warn. Thresholds are typically tuned by hand, and their consequences for the fraction of errors that a reviewer will miss are rarely quantified.
Conformal risk control (CRC) offers a principled way to set such thresholds [21, 22]. Given a nested family of warning sets and a bounded loss, such as the fraction of erroneous pixels that are not warned, CRC selects the smallest family member whose expected loss is guaranteed not to exceed a target, using only exchangeable calibration data. Related ideas have recently been applied to segmentation [23, 24, 25]. A single pooled target, however, averages per-image missed-error fractions whose denominators contain all errors of an image. Because most segmentation errors lie in thin bands along the predicted contour, contour errors dominate these denominators, and a pooled target can be met largely by covering those bands, while errors far from the contour remain under-flagged. These distant errors include whole missed regions and confident false positives. They are the errors that a reviewer who checks the contour will not find, and they often matter more than small boundary disagreements. The general phenomenon, that marginal guarantees can hide poor coverage of minority subgroups, is well documented in the conformal literature [26, 27, 28]. Group-wise risk control has recently been applied to anatomical structures in multi-organ segmentation [29]. To our knowledge, however, it has not been examined for spatial groups of segmentation errors defined by the predicted contour.
In this work we study whether allocating warnings separately near and away from the predicted contour provides useful protection against distant segmentation failures at an acceptable review cost. We propose RACE-Triage, a post-hoc framework that leaves the segmentation mask unchanged. A risk network is trained on out-of-fold predictions using only inputs whose meaning does not depend on the segmenter, namely the image, foreground probability, predicted mask and contour distance. As a result, it can be deployed with a final segmenter that it has never seen. The predicted contour partitions each image into a boundary band and a non-boundary region, and CRC is applied separately in each region to control the expected missed-error fraction among cases that contain errors there. We also examine an error-load-adaptive variant that scales the warning budget of each image by its predicted number of errors. The approach is evaluated on dermoscopy (ISIC 2018) and breast ultrasound (BUSI), with an external dermoscopy set (PH²) processed both with frozen thresholds and after local recalibration. Pooled and regional policies are compared on the same frozen risk map, and the evaluation reports missed-error risk together with the warned area, fragmentation and detection of size-filtered false-positive and false-negative components.
RACE-Triage differs from uncertainty estimation, including boundary-aware uncertainty training [14], in what it produces. Those methods produce or improve a score map; RACE-Triage takes a score map as given and decides how much of each region to warn so that a stated group-wise missed-error property holds. It also differs from conformal methods for segmentation [23, 24, 25], which calibrate pixel-wise prediction sets or case-level quality estimates, because it leaves the mask unchanged and controls the errors that a reviewer would miss in regions defined by the prediction. The calibration rule itself is standard conformal risk control.
The main contributions of this work are summarized as follows:
We formulate segmentation error triage as groupwise control of the expected missed-error fraction over a partition defined by the predicted contour, and we show that restricting calibration to error-positive cases preserves the conformal guarantee. The calibration rule itself is standard; the resulting nested warning families admit both fixed thresholds and error-load-adaptive budgets.
We describe an out-of-fold protocol for learning pixel error maps from backbone-independent inputs. This avoids feature-space mismatch between fold models and the deployed segmenter while keeping the error target honest.
We report a controlled study on two modalities and an external dataset. It shows that pooled control can meet its overall target while missing about one third of non-boundary errors, and it quantifies the review area, fragmentation and component-level detection associated with regional control. Comparisons with uncalibrated allocation rules, per-image area-matched warnings, boundary-aware uncertainty training, three training seeds, a second backbone and target-site recalibration delimit what the calibration step contributes and when the guarantee applies.
The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the error-map learning protocol, the spatial groups and the calibration procedure. Section 4 describes datasets, models and evaluation, Section 5 reports the results, Section 6 discusses implications and limitations, and Section 7 concludes the paper.
Related work
Uncertainty estimation in medical image segmentation
Bayesian approximations and their practical surrogates are the most common route to pixel-wise uncertainty. Monte Carlo dropout [10], deep ensembles [11] and test-time augmentation [13] estimate predictive variability through repeated stochastic passes, and the distinction between aleatoric and epistemic components has been made explicit for vision tasks [12]. Generative models such as the probabilistic U-Net and PHiSeg capture annotation ambiguity by sampling plausible masks [30, 31]. Calibration studies have shown that segmentation networks are often over-confident and that ensembling and post-hoc scaling improve calibration [32, 16]. In the same journal, uncertainty-aware training has been used to align single-pass entropy with pixel-wise error, with additional emphasis on boundaries [14]. Several evaluations, however, caution that good calibration does not imply good error detection. Uncertainty concentrates at object boundaries, and its quality depends strongly on how it is validated and aggregated [15, 33, 34, 35]. Our work uses a learned error map rather than an uncertainty proxy, and it asks how such a map should be converted into warnings with stated properties.
Error prediction and segmentation quality control
A second line of work predicts segmentation quality directly. Image-level approaches estimate Dice or flag failed cases. Examples include reverse classification accuracy [36], its use for automated quality control at population scale [37], and quality estimation from small datasets [19]. Pixel-level methods predict where errors occur, for example by comparing the prediction with self-generated references [18] or by learning the true-class probability (TCP) of the segmenter as a confidence target [17, 38]. Disagreement-guided quality control has also been combined with reliability-aware breast tumor segmentation in DCE-MRI [39]. A recent benchmark of failure detection in medical segmentation showed that the way pixel confidence is aggregated strongly affects case-level failure detection [20], in line with broader concerns about how failure detection is evaluated [40]. These methods produce scores or rankings. They do not specify how much of an image should be shown to a reviewer, or what fraction of errors would then remain unseen; selective prediction addresses this question for classification [41]. We adopt a multiscale error-map network and a TCP-style regressor as learned baselines and focus on the warning policy built on top of a fixed score.
Conformal prediction and risk control in imaging
Conformal prediction provides finite-sample, distribution-free guarantees under exchangeability [42, 43]. Risk-controlling prediction sets, learn-then-test and conformal risk control extend this idea from coverage to general monotone losses [21, 44, 22], and they have been used to calibrate image-to-image regression [46] [45]. In segmentation, Kandinsky conformal prediction calibrates pixel-wise nonconformity scores jointly across images and spatial regions, which improves efficiency when few calibration images are available [23]. Conformal semantic segmentation produces post-hoc multi-label masks [24], and conformal performance range prediction targets output-level quality control [25]. Closer to clinical end-points, conformal calibration has been used to bound the false-negative rate of 3D lesion segmentation [47] and the false discovery rate of medical instance segmentation [48]. A study on PET/CT tumour segmentation found that a global threshold can certify a voxel-level missed-tumour loss but not a binary missed-lesion loss, because thresholding cannot recover lesions that the network never supported [49]. Conformal methods have also been applied to medical classification, including skin lesions, breast cancer diagnosis and ECG analysis [50, 51, 52], and to hierarchical histopathology analysis [53].
A known limitation of marginal guarantees is that they may hold on average while failing for identifiable subgroups. Equalized-coverage and Mondrian constructions calibrate within groups [26, 27, 28], and conditional guarantees over richer function classes have been developed [54], although exact conditional validity is impossible without restrictions [55]. Under distribution shift, guarantees require reweighting or new calibration data [56, 57]. Group-wise recall control has recently been applied to anatomical structures, with per-organ thresholds for a frozen multi-organ CT segmenter; in that study, guarantees calibrated at one site failed for most organs at another, and the number of local cases needed for recertification depended on whether expectation-level or high-probability bounds were used [29]. The present work applies group-wise risk control to a spatial partition that is specific to segmentation errors, namely pixels near and away from the predicted contour. Its calibration step is therefore an instance of group-conditional risk control rather than a new conformal procedure. The contribution lies in groups that are derived from the prediction and available at inference, in restricting each group to its error-positive cases, and in an empirical comparison with allocation rules that do not use conformal calibration. Table 1 summarizes how the method families above differ from this formulation. We use CRC, which bounds the expected loss, rather than risk-controlling prediction sets, which bound it with high probability over the calibration draw [21], because high-probability bounds typically require more eligible cases per group than the non-boundary groups provide [29]; exceedance in individual cases is addressed separately by the tail analysis in § 3.6.
| Method family | Produces or controls | RACE-Triage instead |
| Uncertainty estimation [10, 13, 14] | Pixel-wise uncertainty scores | Calibrates warning regions for a given score |
| Error and quality prediction [17, 18, 20] | Error maps or case-level quality scores | Converts an error map into warnings with a stated risk |
| Conformal segmentation [23, 24, 25] | Prediction sets or quality ranges for the segmentation | Leaves the mask unchanged |
| Conformal false-negative or false-discovery control [47, 48, 49] | Missed or spurious parts of the prediction | Controls the share of all segmentation errors left unwarned |
| Group-wise risk control [26, 29] | Risk per predefined group, e.g., per organ | Spatial groups derived from the predicted contour |
Table 1. Positioning of RACE-Triage relative to related method families.
Unlike prediction-set methods, we keep the segmentation unchanged and control the fraction of errors that a reviewer would miss, which can be stated directly in terms of review workload.
Methodology
Fig. 1 summarizes the proposed framework, which we refer to as RACE-Triage (Region-Aware Calibrated Error Triage). A frozen segmenter produces a mask that is never modified. A separately trained error-map network estimates where this mask is wrong, the predicted contour partitions every image into a boundary band and a non-boundary region, and one calibrated warning rule per region converts the error map into a review overlay. The three stages of the figure correspond to how the components are fitted: the error-map network is trained on out-of-fold predictions (§ 3.3), the warning rules are calibrated on independent cases (§ 3.5), and deployment uses only the image.
Problem formulation
Let be an image with pixels indexed by . A segmenter returns a foreground probability and a predicted label . Given a reference mask , the error target is
which covers false positives and false negatives alike. The reference mask is available only for training, calibration and evaluation. At deployment, the system outputs the unchanged mask together with a set of warned pixels , which a reviewer inspects. A useful overlay should leave few errors outside while keeping small.
A separately trained network produces a pixel risk that estimates . Its permitted inputs are the normalized image, , and, optionally, deterministic geometry derived from . These channels have the same meaning for every segmenter trained with the same recipe. We deliberately exclude hidden activations of the segmenter, raw logits with unverified cross-model scale and any quantity derived from . The reason is that is trained on predictions of several fold-specific segmenters but deployed with a different, final segmenter (§ 3.3); hidden channels of independently trained networks are not aligned, whereas probability and mask channels are.
Figure 1. Overview of RACE-Triage. Top: out-of-fold error learning with backbone-independent inputs. Middle: deployment, in which the predicted contour splits each image into and and each group is warned by its own calibrated rule. Bottom: per-group conformal risk control on error-positive calibration cases. Map panels are schematic.
Contour-defined spatial groups
Let denote the Euclidean distance from pixel to the interface between predicted foreground and background. For a band half-width we define
is a band around the predicted contour, and contains the remaining far foreground and background. We call the non-boundary group rather than the interior, because it also contains distant background. When is empty or covers the whole image, no interface exists and every pixel is assigned to ; no contour is constructed from the image frame. The distance channel of the risk network then takes its clipping value at every pixel (§ 4.6), so the network sees the whole image as far from any contour, and the empty or full mask identifies the degenerate case without a separate indicator. The groups are disjoint and exhaustive, depend only on the prediction, and are therefore available at inference. Reference boundaries never define groups.
This partition reflects how segmentation errors arise. Contour disagreement produces thin error bands inside that are numerous, visually obvious and often of limited consequence. Missed lesion lobes, under-segmented low-contrast margins and spurious regions of sufficient size place errors in . These errors are fewer, but a reviewer who follows the predicted contour does not encounter them. A pooled objective divides the missed errors of a case by all of its errors. Because contour errors dominate this denominator in most images, the pooled target can be met mainly by covering them, and it gives no assurance for .
For a case and group , let be the number of errors in the group. The case is error-positive in group if . Error positivity is used only to define offline calibration and evaluation populations; it is never an input at inference.
Error-map learning from out-of-fold predictions
Training on the predictions of a segmenter that has seen the same images would expose it to optimistic, near-perfect masks. We therefore build the training set out of fold. The development pool is split into group-preserving folds. For each , a backbone is trained without fold , using a fixed epoch budget so that no checkpoint is selected on the held-out fold. It then predicts and on fold , and the reference masks of that fold give through Eq. (1). The records from all folds are pooled to train a single network ; no fold-specific activations are stored. A final segmenter is then trained on all of with the same recipe and deployed with . The network is trained with unweighted binary cross-entropy averaged first over the pixels of a case and then over cases,
so that images with many pixels do not dominate. We do not use focal or class-weighted losses, which would change the probabilistic interpretation of that the load-adaptive family below relies on. As the default architecture we use a multiscale encoder–decoder that receives , , and the clipped distance , which gives it full-field context for errors far from the contour. § 4.2 lists the local, capacity-matched and target-modified variants used as controls.
Out-of-fold training removes the feature-alignment problem but leaves a residual shift, because each sees 80% of whereas sees all of it. The final pipeline may therefore make somewhat fewer errors than the fold models. Calibration on independent data (§ 3.5) accounts for this shift in the reported risk, but it cannot improve the ranking quality of ; we audit both on development data before calibration.
Nested warning families
A warning policy is a family of warning sets indexed by a scalar , such that larger never removes a warning. Nesting is the property that the calibration step requires. We consider two families.
Fixed threshold. For ,
which warns every pixel of the group at . Equality is warned.
Error-load-adaptive budget. A fixed threshold warns similar areas on easy and difficult images. As an alternative, we allocate a pixel budget proportional to the predicted number of errors. With and predicted error load , the family warns the
highest-risk pixels of , with ties broken by pixel index. Because , a grid that ends at the image size warns every pixel at its last point. This family is nested for every image even when is poorly calibrated. Its efficiency, however, depends on how well tracks the true error count, so we treat it as a candidate that must improve on both the fixed regional family and its pooled counterpart before it is adopted.
Groupwise conformal risk control
For a warning set , the missed-error fraction in group is
and is the corresponding calibration loss. Each lies in , is non-increasing in for a nested family, and equals zero at the all-warning endpoint. Let be the number of calibration cases that are error-positive in group . Following conformal risk control (CRC) [22], we choose
on a finite grid fixed on development data and containing the all-warning endpoint. If the set is empty or , the entire group is warned. The deployed overlay is (Algorithm 1).
Suppose that calibration and test cases are drawn independently from the same distribution, and that the segmenter, risk network, groups, family and grid are fixed independently of the calibration data. Selecting the cases that are error-positive in group then yields a sample of random size that, conditionally on , is again independent and identically distributed, now from the distribution of cases with . Applying CRC [22] conditionally on and averaging over (Proposition 1, Appendix A) shows that the bounded, monotone losses satisfy
where the expectation is over calibration and test draws. This is an average-case statement for each group. It is not a high-probability bound for a particular calibration set and not a per-image guarantee; the latter is unattainable without further assumptions [55]. For BUSI, where several images may come from one patient, independence across cases is an assumption rather than a property of the data (§ 4.1). Because only error-positive cases enter Eq. (7), zero-error images do not dilute the target. The term is the price of finite calibration data; with , at least four eligible cases are needed for any non-trivial threshold, and many more for useful precision.
The calibration rule in Eq. (7) is standard CRC applied within each group, in the spirit of Mondrian conformal methods [42, 26]; what is specific to segmentation is the choice of groups and the restriction to error-positive cases. The pooled baseline applies Eq. (7) once, with a single group that contains all pixels. It controls the mean missed fraction over all errors but makes no statement about either group. Regional control imposes two constraints instead of one, so it is not expected to reduce the total warned area; the relevant question is what the additional protection for costs. When several objectives must hold for one overlay, for example regional pixel targets and a component-level target, we take the union of the separately calibrated warning sets. Adding warnings can only lower each monotone loss, so every expected-risk statement is preserved, and the extra area is reported. For the component-level target used in § 5.9, the loss of a case is the fraction of its size-filtered false-positive and false-negative components (§ 4.4) with , computed over cases with at least one such component. With the fixed-threshold family applied to all pixels, this loss lies in and is non-increasing in , so Eq. (7) applies with a single group.
Deployment protocol and secondary tail control
The segmenter, risk network, normalization, resolution, band width, family, grid, tie rule and targets are frozen before cali Algorithm 1 Regional calibration and deployment of error warnings
Require: Frozen segmenter and risk network ; calibration cases ; band half-width ; targets ; nested family on a finite grid that contains the all-warning endpoint
Calibration (offline, uses reference masks)
for do
; ;
distance to the interface of ; ,
for with do
store for all ▷ Eq. (6)
end for
end for
for do
number of eligible cases;
if the set is empty or then all-warning endpoint
end for
Deployment (no reference mask)
For a new image : compute as above
return unchanged mask and overlay
ibration data are accessed. Calibration selects only and . Changing the segmenter changes the error target. Recalibration on cases processed by the new segmenter is then required for Eq. (8) to hold. Refitting on the new segmenter’s out-of-fold errors is not needed for validity, but it improves the ranking and hence the warned area (§ 5.6). Images from a different site can be processed by the frozen pipeline, but Eq. (8) then holds only if exchangeability is re-established, for example by recalibrating on target-site data [56, 57]. Recalibration repeats Algorithm 1 with labelled target-site cases and changes only and ; § 5.8 evaluates it on PH.
Eq. (8) controls an average, so individual images may still exceed . As a secondary analysis we therefore consider tail control for fixed thresholds. For each eligible calibration case we record the critical score , i.e., the -th smallest risk among its erroneous pixels in group , and set to the -th smallest with ( when ). Warning every pixel of with , a case violates the target exactly when its own critical score lies below , so exchangeability of ranks gives (Proposition 2, Appendix § 8), and a union bound gives an unconditional per-case violation rate of at most . This construction needs more calibration cases and, as shown in § 5.9, considerably more warned area.
Experimental setup
Datasets and partitions
ISIC 2018. We use the 2594 dermoscopic images of the ISIC 2018 lesion segmentation training release together with their public lesion masks [58, 59]. Reliable lesion identifiers are not available for this task, so we ran an automated near-duplicate audit that compares 64-bit perceptual hashes and links two images when their Hamming distance is at most 8; distances of 4 and 12 serve as sensitivity settings (Appendix § 9). The primary audit found 21 groups that contain 52 images. Each group was kept within one partition, and all groups were assigned to the fitting pool, so that every calibration and test record is a single image.
BUSI. The breast ultrasound dataset contains 780 images from 600 female patients, labelled as normal (133), benign (437) or malignant (210) [60]. Images with several lesion masks were merged by a union, and normal images keep their empty masks, so that false-positive predictions on normal scans count as errors. Patient identifiers are not released, and duplicated or problematic images have been reported for this dataset [61]. The same audit grouped 17 images into eight near-duplicate groups, which were placed in the fitting pool. Partitions were stratified by category. A hash audit detects repeated or re-cropped acquisitions of one view, but it cannot link different views of the same patient, and no patient-level metadata are available to do so. The groupwise statement of Eq. (8) therefore applies to BUSI only under an assumed independence across images. We report BUSI results as empirical evidence and examine their sensitivity to a more inclusive grouping (Appendix § 9).
PH. The PH database contains 200 dermoscopic images (80 common nevi, 80 atypical nevi and 40 melanomas) with manual lesion masks [62]. It serves as an external test set for the frozen ISIC pipeline, without retraining. To evaluate target-site recalibration, we additionally drew 100 random splits of PH into 100 calibration and 100 test images, stratified by diagnosis, and recalibrated only the two thresholds on each calibration half. Before these analyses, the hash audit was also run between PH and the ISIC 2018 training release to rule out shared images.
Each in-domain dataset is split into four disjoint partitions, as summarized in Table 2. (65%) provides the out-of-fold records and trains the final segmenter. (10%) is divided into , used to select risk-network architectures and checkpoints, and , used for the development policy pilot (§ 4.5). (12.5%) is used only to compute and , and (12.5%) is used for the final evaluation and for retrospective area matching of diagnostic baselines. All prospective policy design choices, including the band width, warning families, calibration grids and display rules, were fixed before and were accessed. Retrospective area-matched diagnostic baselines are reported separately and are not considered prospectively specified deployment policies. All images were resized to 256 256 pixels with bilinear interpolation and all masks with nearest-neighbour interpolation. Intensities were scaled to [0, 1] and standardized per channel with the mean and standard deviation of ; grayscale ultrasound images were replicated to three channels.
| Dataset | Modality | Images | Partition sizes | Test Dice | Error pixels (%) | Cal. | Test | share (%) |
| ISIC 2018 | Dermoscopy | 2594 | 1686 / 259 / 324 / 325 | 0.883 | 3.2 | 322 / 128 | 323 / 131 | 21.6 |
| BUSI | Breast ultrasound | 780 | 507 / 78 / 97 / 98 | 0.751 | 5.6 | 82 / 49 | 83 / 52 | 37.9 |
| PH² | Dermoscopy (external) | 200 | – / – / – / 200 | 0.862 | 5.4 | – | 200 / 118 | 33.4 |
Table 2. Datasets, partitions and error audit of the frozen segmenters. Partitions are given as . counts error-positive cases in the boundary and non-boundary groups. Dice is the mean over test images with a non-empty reference mask; the last column gives the share of test error pixels that lie in .
Segmentation and error-map models
Segmenter. We use a standard 2D U-Net [1] with four resolution levels, 32 base channels, batch normalization and 7.76 M parameters. It is trained with the sum of binary cross-entropy and soft Dice loss [63]. The same recipe (120 epochs, no checkpoint selection) is used for the five fold backbones and for , and one pipeline is trained independently for each modality. To measure variability due to retraining, the complete pipeline, comprising the five fold backbones, and , was trained with three random seeds (1, 2 and 3), which set weight initialization, data order and augmentation; seed 1 provides the main results. The data partitions are the same for all seeds and were drawn once with a separate seed (2026). As a second backbone of a different design, we used DeepLabV3+ [64] with an ImageNet-pretrained ResNet-50 encoder [65] (26.7 M parameters), trained with the same loss, schedule and out-of-fold protocol and paired with its own risk network. To test the transfer of a risk network across architectures, we also deployed the U-Net risk network with the DeepLabV3+ segmenter and recalibrated only the thresholds. Finally, as a baseline that improves uncertainty through training rather than through a separate network, we re-implemented Boundary-Guided US-Loss [14]. It adds to the segmentation loss a boundary-weighted squared error between the predictive entropy and the pixel-wise prediction error, and we used the settings reported by its authors (, , ). The fold backbones and the final U-Net were trained with this loss, and its single-pass entropy was compared with a risk network refitted on its out-of-fold errors under the same policies.
Error scores. All scores are computed from the same frozen segmentations. Three uncertainty baselines leave the segmenter unchanged: the predictive entropy of (P0); test-time augmentation, i.e., the entropy of the mean foreground probability over eight views drawn from the training augmentations [13]; and the disagreement of the five fold backbones , measured as the variance of their foreground probabilities, which forms a deep ensemble at no additional training cost [11]. Monte Carlo dropout [10] was not included because the segmenter has no dropout layers, and adding them would change the segmentation that is being triaged. The learned variants share Eq. (3) and the out-of-fold protocol, so they differ only in inputs and architecture:
P1, a local head with , and stride-one convolutions on , whose receptive field on raw inputs is pixels;
P2, which adds the clipped contour distance to P1;
P3, which further adds the absolute cosine alignment between and the intensity gradient, with a validity indicator that is zero outside the band and where gradients vanish;
P4, a capacity-matched control with the same number of input channels and parameters as P3 but without engineered evidence;
a multiscale error-map U-Net (16 base channels, 1.94 M parameters) on , which supplies full-field context;
the same multiscale network with the additional channel, which is the default risk network of RACE-Triage.
In addition, a TCP-style regressor uses the multiscale architecture and permitted inputs to predict the true-class probability with a squared loss, and it reports as its failure score [17]. The selection among P1–P4 and the multiscale variants was made on before any policy experiment.
Warning policies
All policies operate on the default risk map and target , a research operating point rather than a clinical tolerance. We compare pooled and regional CRC with the fixed-threshold family (Eq. (4)), and pooled and regional CRC with the error-load-adaptive family (Eq. (5)). The band half-width is pixels, with as a predeclared sensitivity setting. The fixed family uses 201 equally spaced values of in . The adaptive family uses a geometric grid of 120 values between 0.25 and 64, chosen on development data, followed by the all-warning endpoint. As described references without any guarantee, we include top-ranked policies that warn exactly 10% of each image, or the mean area of the regional policy, according to the risk ranking.
To separate the contribution of conformal calibration from that of spatial allocation, we compare regional CRC with four rules that use the same risk map. (i) A regional plug-in rule selects, in each group, the smallest whose empirical mean loss is at most , i.e., Eq. (7) without the finite-sample term. (ii) Fixed group quotas warn a constant fraction of pixels within each of and in every image. The ratio between the two regional quotas was selected on to balance the empirical groupwise missed-error risks. The overall quota level was then retrospectively selected to reproduce the regional CRC policy’s mean warned area on the held-out test set (11.4% for ISIC 2018 and 20.0% for BUSI). This area matching uses information from the test-set evaluation to compare spatial warning allocation at the same mean warned area across test images; it does not match the warning budget of each image. As evaluated here, the fixed-group-quota baseline is a diagnostic comparison rather than a prospectively specified deployment policy, and it provides no conformal risk guarantee. (iii) A distance-boosted ranking adds a constant to the risk of every pixel in and then applies pooled CRC, with tuned on to equalize the group risks (0.10 for ISIC and 0.14 for BUSI). (iv) A per-image matched reference warns, in each test image, exactly as many top-ranked pixels as the regional policy, so that the two overlays differ only in where the pixels are placed within the image. Because it requires the regional policy’s area for each image, this reference is a diagnostic comparison rather than a deployable policy.
Finally, we vary the target and the band width over and pixels. Each setting is recalibrated on without retraining , because the distance channel of is clipped at 32 pixels and does not depend on .
Evaluation metrics
Missed-error risk. The primary risk metrics are the mean missed-error fractions and over test cases that are error-positive in the respective group, together with the pooled mean over all error pixels. These are the quantities controlled by Eq. (8).
Review burden. We report the warned share of the image, the warned share of correctly segmented pixels and the number of 8-connected warning regions per image, which reflects fragmentation. Because a mean can hide heavy tails, we also give the median and the 90th and 95th percentiles of the per-image warned area.
Error type and subgroups. The primary loss treats false-positive and false-negative pixels alike, although a missed lesion region and a spurious region have different clinical consequences. We therefore also report the non-boundary missed fraction separately for false-positive and false-negative pixels, and the BUSI results by diagnostic category. These subgroup summaries are empirical and carry no separate guarantee. In particular, the pixel-level guarantee treats false-positive and false-negative pixels alike, and neither it nor the component-level results constitute a guarantee against missed lesions.
Size-filtered components. False-positive and false-negative masks are decomposed into 8-connected components, and components smaller than 25 pixels are discarded. A component counts as detected when with . These thresholds are operational definitions rather than clinical severity labels [66, 67], so we repeat the component analysis with and with a minimum component size of 100 pixels (Appendix B). Component recall at exactly 10% warned area (CR@10) is the co-primary endpoint for score quality, because it compares rankings at a fixed budget. For the calibrated policies, component recall is instead reported at each policy’s own warned area (Table 4), so it reflects both ranking and allocation. Pixel-level CRC certifies neither quantity, so both are empirical outcomes.
Score quality. For the error scores we report AUPRC for raw errors and for -distant errors (error pixels more than pixels from the reference boundary), the Brier score, CR@10 and the recall of non-boundary error pixels at 10% area (N@10). We also report the Spearman correlation between the predicted error load and the observed number of errors.
Development pilot and statistical analysis
Before final calibration, we ran a policy pilot on of ISIC with the frozen default risk map. Fifty random 50:50 splits into pseudo-calibration and pseudo-test halves were drawn, and pooled and regional CRC were fitted on each pseudo-calibration half and evaluated on the other. The repeated splits reuse the same images, so they describe variability and are not treated as independent evidence. Final results use one calibration on and one evaluation on . Uncertainty is quantified with 95% percentile intervals from 2000 paired bootstrap resamples of test cases [68], and tail violation rates use exact binomial intervals. These intervals resample test cases for the selected calibration set. They therefore reflect test-case sampling, but neither the randomness of the calibration draw nor retraining. Variability due to retraining is reported separately as the standard deviation over the three seeds, and the calibration draw is probed by the re-partitions below. Eq. (8) concerns an expectation over calibration and test draws, which a single split cannot show. After the primary analysis, we therefore re-partitioned , 500 at random into halves of the original sizes, calibrated on one half and evaluated on the other. The re-partitions reuse the same images, so we report them as a sensitivity analysis of the calibration rules rather than as independent evidence. Because patient identifiers are unavailable for BUSI, the bootstrap and the re-partitions resample images rather than patients and may understate uncertainty if images of one patient are correlated.
Implementation details
All models were implemented in PyTorch [69] and trained on a single NVIDIA A100 GPU (40 GB). Segmenters were optimized with Adam [70] (learning rate , cosine decay, batch size 16) with random flips, rotations of up to and, for dermoscopy, mild colour jitter. DeepLabV3+ used output stride 16 and the same optimizer, schedule and augmentation. Risk networks were trained for 80 epochs with Adam (learning rate , batch size 16), and the checkpoint with the highest AUPRC on was retained. The distance channel is computed with an exact Euclidean distance transform: for a foreground pixel of it is the distance to the nearest background pixel, and vice versa, so that pixels adjacent to the interface have . It is clipped at 32 pixels and scaled to ; when has no interface, it equals 1 at every pixel. Since is also an input, the side of the contour is available to the network. Distance transforms and connected components were computed on the CPU, and calibration was implemented with vectorized loss curves over the grid. Runtime is measured per 256 256 image with batch size one.
Results
Segmentation quality and error audit
The frozen segmenters reached a mean test Dice of 0.883 (95% CI 0.871–0.894) and IoU of 0.807 on ISIC 2018, and 0.751 (0.713–0.786) and 0.652 on BUSI lesion images. On BUSI, five of the 17 normal test images received a non-empty prediction, and three lesion images received an empty one. Transferred to PH2 without retraining, the ISIC segmenter obtained a Dice of 0.862 (0.848–0.875).
The error audit in Table 2 establishes that non-boundary errors are common enough to matter. On ISIC, 131 of 325 test images (40.3%) contained at least one error pixel in , although these pixels made up only 21.6% of all errors. On BUSI the corresponding figures were 53.1% of images and 37.9% of error pixels. Most BUSI non-boundary errors came from spurious detections in hypoechoic regions and from lesions that were missed entirely; in the latter case the prediction is empty and the whole image belongs to . PH2 lesions are larger than typical ISIC lesions, and the transferred segmenter tended to under-segment their margins. As a result, 59.0% of PH2 images had non-boundary errors. The hash audit found no PH2 image within the duplicate threshold of any ISIC 2018 training image; the smallest Hamming distance between the two collections was 17.
Error-score quality
Table 3 compares the error scores under identical segmentations. Predictive entropy was the weakest score on every metric. Its recall of non-boundary error pixels at 10% warned area was 0.214 on ISIC and 0.152 on BUSI, which confirms that it mainly highlights the contour. Test-time augmentation and fold-ensemble disagreement improved on entropy (ISIC AUPRC 0.336 and 0.352, 0.236 and 0.268) but remained clearly below the learned error maps. Both measure variability of the prediction, which is again concentrated along the contour, and confident misses vary little across views or fold models. The local head P1 improved AUPRC from 0.318 to 0.394 on ISIC. Adding the contour distance (P2) gave a small further gain, whereas gradient alignment (P3) did not separate from the capacity-matched control P4 on either dataset, so it was not used in the subsequent experiments. The TCP-style regressor improved on the local variants. The largest gain, however, came from full-field context: the multiscale error-map U-Net raised CR@10 to 0.731 on ISIC and 0.622 on BUSI.
| Error score | Params (M) | ms | ISIC 2018 AUPRC↑ | ISIC 2018 ↑ | ISIC 2018 Brier↓ | ISIC 2018 CR@10↑ | ISIC 2018 N@10↑ | BUSI AUPRC↑ | BUSI ↑ | BUSI Brier↓ | BUSI CR@10↑ | BUSI N@10↑ |
| Predictive entropy | 0 | 0.1 | 0.318 | 0.192 | – | 0.512 | 0.214 | 0.276 | 0.163 | – | 0.384 | 0.152 |
| Test-time augmentation | 0 | 31.4 | 0.336 | 0.205 | – | 0.538 | 0.236 | 0.291 | 0.172 | – | 0.405 | 0.169 |
| Fold-ensemble disagreement | 38.8 | 19.7 | 0.352 | 0.221 | – | 0.561 | 0.268 | 0.309 | 0.188 | – | 0.431 | 0.204 |
| Local head (P1) | 0.01 | 0.4 | 0.394 | 0.251 | 2.48 | 0.603 | 0.331 | 0.341 | 0.207 | 4.71 | 0.466 | 0.248 |
| P1 + distance (P2) | 0.01 | 1.1 | 0.405 | 0.262 | 2.44 | 0.618 | 0.352 | 0.350 | 0.214 | 4.66 | 0.479 | 0.262 |
| P2 + alignment (P3) | 0.01 | 1.6 | 0.409 | 0.263 | 2.43 | 0.621 | 0.349 | 0.347 | 0.211 | 4.68 | 0.472 | 0.255 |
| Capacity-matched P1 (P4) | 0.01 | 0.5 | 0.401 | 0.258 | 2.45 | 0.615 | 0.343 | 0.345 | 0.210 | 4.69 | 0.470 | 0.251 |
| TCP-style regressor | 1.94 | 2.8 | 0.421 | 0.287 | 2.37 | 0.668 | 0.429 | 0.379 | 0.246 | 4.52 | 0.548 | 0.347 |
| Multiscale error-map U-Net | 1.94 | 2.8 | 0.449 | 0.317 | 2.26 | 0.731 | 0.512 | 0.423 | 0.291 | 4.31 | 0.622 | 0.437 |
| Multiscale U-Net + (ours) | 1.94 | 3.5 | 0.458 | 0.324 | 2.23 | 0.748 | 0.526 | 0.419 | 0.295 | 4.33 | 0.615 | 0.429 |
Table 3. Error-score quality on the held-out test sets with identical frozen segmentations. The first block contains uncertainty baselines that leave the segmenter unchanged: predictive entropy, test-time augmentation (entropy of the mean over 8 augmented views) and the disagreement of the five fold backbones (parameters of the additional segmenters). AUPRC is computed for raw errors and, in the columns headed , for -distant errors ( px); Brier scores are and are not defined for unnormalized uncertainty scores. CR@10 is the size-filtered component recall and N@10 the recall of non-boundary error pixels when exactly 10% of each image is warned. Latency includes geometry computation (ms per image).
Adding the distance channel to the multiscale network gave the best ISIC scores (AUPRC 0.458, CR@10 0.748), but on BUSI it was marginally below the network without (AUPRC 0.419 vs. 0.423; CR@10 0.615 vs. 0.622). The differences between the two multiscale variants are small relative to their bootstrap variability on both datasets. A plausible explanation is that BUSI non-boundary errors are dominated by isolated false detections and missed lesions, for which distance to the predicted contour carries little information. We kept a single default network for both modalities, as fixed before testing. Figure 2 shows that the ordering of the learned scores is stable across warned areas from 5% to 30%.
Figure 2. Size-filtered component recall versus warned area for top-ranked warnings of each error score. Markers show the calibrated pooled and regional policies on the default risk map.
The predicted error load correlated with the observed error count (Spearman on ISIC and 0.61 on BUSI), but it was strongly underestimated for the most difficult images. In the top decile of observed errors, the median ratio of predicted to observed load was 0.58 on ISIC and 0.44 on BUSI. This limits the load-adaptive family, as shown in § 5.7.
Development policy pilot
The pilot on tested the premise of the method before the final calibration (Figure 3). Each pseudo-calibration half contained between 63 and 65 cases that were error-positive in , but only 19 to 34 (median 27) that were error-positive in . Pooled CRC met its own target, yet the non-boundary missed fraction had a median of 0.315 (interquartile range 0.277–0.355) across splits and exceeded in 48 of 50 splits. Boundary errors were over-protected, with a median of 0.140. Regional CRC moved both group means to the target, with medians of 0.187 for and 0.198 for . The non-boundary mean exceeded in 24 of 50 splits, which is the behavior expected from an expectation-level guarantee rather than a per-split one. The median warned area increased from 9.7% to 11.6% of the image. With so few non-boundary cases, the secondary tail rule used the smallest critical value () in every split, which illustrates why we treat tail control as secondary. These results satisfied the predeclared condition for continuing to the final experiments, namely a material non-boundary deficit under pooled control together with a regional remedy at a moderate area cost.
Figure 3. ISIC development pilot over 50 pseudo-calibration/pseudo-test splits. (a) Mean missed-error fraction per group. (b) Mean warned area per split; lines connect the two policies on the same split.
Regional versus pooled control on held-out data
Table 4 and Figure 4 report the final comparison on the test sets. On ISIC, pooled CRC with fixed thresholds () satisfied its own objective (). It also over-protected the boundary band () but missed 0.318 (0.271–0.366) of non-boundary errors on average. Regional CRC selected a higher boundary threshold and a much lower non-boundary threshold (, ). This reduced the non-boundary missed fraction to 0.193 (0.154–0.234), a paired difference of −0.125 (−0.168 to −0.083), and raised the boundary missed fraction to 0.188, close to its target. The cost was 1.6 percentage points of additional warned area (1.2–2.0), most of which fell on correctly segmented patches (8.9% vs. 7.3% of the image), and more fragmented overlays (4.7 vs. 3.4 regions per image). Size-filtered false-negative components benefited most, with recall increasing from 0.722 to 0.818 (difference 0.096, 0.067–0.126).
Table 4: Warning policies on one frozen risk map (). , and are mean missed-error fractions over error-positive test cases (pooled, boundary and non-boundary). Area is the warned share of the image and Corr. the share of the image that is correctly segmented but warned; Reg. is the number of connected warning regions per image; FP/FN are size-filtered component recalls () at each policy’s own warned area, unlike CR@10 in Table 3, which fixes the area at 10%. Top-ranked rows use exact per-image budgets and carry no calibration guarantee.
| Policy | Area (%) | Corr. (%) | Reg. | FP rec. | FN rec. | |||
| ISIC 2018 (in-domain, ) | ||||||||
| Pooled CRC, fixed | 0.181 | 0.137 | 0.318 | 9.8 | 7.3 | 3.4 | 0.802 | 0.722 |
| Regional CRC, fixed (ours) | 0.190 | 0.188 | 0.193 | 11.4 | 8.9 | 4.7 | 0.829 | 0.818 |
| Pooled CRC, adaptive | 0.183 | 0.142 | 0.301 | 9.3 | 6.7 | 3.1 | 0.795 | 0.731 |
| Regional CRC, adaptive | 0.192 | 0.190 | 0.197 | 10.6 | 8.1 | 4.2 | 0.824 | 0.812 |
| Top-ranked, 10% area | 0.174 | 0.121 | 0.352 | 10.0 | 7.5 | 3.6 | 0.789 | 0.716 |
| Top-ranked, matched area (11.4%) | 0.158 | 0.104 | 0.331 | 11.4 | 8.8 | 3.9 | 0.811 | 0.733 |
| BUSI (in-domain, ) | ||||||||
| Pooled CRC, fixed | 0.179 | 0.118 | 0.364 | 15.1 | 10.9 | 3.0 | 0.744 | 0.611 |
| Regional CRC, fixed (ours) | 0.188 | 0.176 | 0.209 | 20.0 | 15.7 | 5.4 | 0.802 | 0.763 |
| Pooled CRC, adaptive | 0.181 | 0.124 | 0.342 | 14.6 | 10.3 | 2.8 | 0.737 | 0.626 |
| Regional CRC, adaptive | 0.199 | 0.182 | 0.227 | 20.6 | 16.3 | 5.1 | 0.796 | 0.748 |
| Top-ranked, 10% area | 0.236 | 0.142 | 0.418 | 10.0 | 6.2 | 2.6 | 0.681 | 0.531 |
| Top-ranked, matched area (20.0%) | 0.141 | 0.071 | 0.297 | 20.0 | 15.5 | 3.3 | 0.788 | 0.702 |
| PH2 (external, frozen ISIC pipeline, ) | ||||||||
| Pooled CRC, fixed | 0.214 | 0.161 | 0.371 | 12.4 | 8.5 | 3.6 | 0.771 | 0.648 |
| Regional CRC, fixed (ours) | 0.236 | 0.214 | 0.262 | 13.9 | 9.9 | 4.9 | 0.804 | 0.752 |
| Top-ranked, 10% area | 0.243 | 0.177 | 0.433 | 10.0 | 6.3 | 3.1 | 0.752 | 0.593 |
Table 4.
Figure 4. Held-out comparison on one frozen risk map. (a,b) Mean missed-error fraction per group with 95% bootstrap intervals; the dashed line is . (c) Distribution of per-image warned area for fixed-threshold pooled and regional CRC; long bars mark the means, white dots the medians and short bars the 95th percentiles.
The matched-area references indicate that the gain is not a consequence of the larger budget alone. Warning the top-ranked 11.4% of each image, the same mean area as the regional policy, still missed 0.331 of non-boundary errors, because the highest risks concentrate along the contour. A fixed per-image budget, however, also distributes area across images differently from the regional policy. § 5.5 therefore matches the warned area image by image. Figure 2 shows the same effect for components: both calibrated policies lie near or above the top-ranked curve, and the regional policy lies above it on both datasets.
Per-image warned area was right-skewed (Figure 4c). On ISIC, the median area was 9.0% under pooled and 9.5% under regional control, while the 90th percentiles were 16.3% and 21.2% and the 95th percentiles 19.6% and 23.5%. On BUSI, the medians were 13.7% and 18.1%, the 90th percentiles 24.2% and 34.9%, and the 95th percentiles 27.2% and 38.5%. The additional area of regional control therefore falls disproportionately on a minority of images, and at least one BUSI image in ten would require reviewing more than a third of the image.
On BUSI, the deficit of pooled control was larger, with a non-boundary missed fraction of 0.364 (0.287–0.442). Regional CRC reduced it by 0.155 (−0.231 to −0.083) to 0.209. False-negative component recall rose from 0.611 to 0.763. The area cost was also larger, at 4.9 percentage points (3.7–6.2), bringing the mean warned area to 20.0%. The realized non-boundary mean slightly exceeded . With 52 eligible test cases its interval (0.142–0.281) comfortably contains the target, and Eq. (8) bounds the expectation rather than a single test average. The larger BUSI cost has two causes. Non-boundary errors are a larger share of all errors, and with only 49 eligible calibration cases the finite-sample correction is about 2.5 times larger than on ISIC. Because BUSI images cannot be grouped by patient, the BUSI comparison is empirical evidence under an assumed independence across images, not a verified instance of Eq. (8).
Comparison with spatial allocation rules
Table 5 compares regional CRC with allocation rules that use the same risk map. Matching the regional area image by image reduced the non-boundary missed fraction of the top-ranked reference from 0.331 to 0.284 on ISIC and from 0.297 to 0.262 on BUSI, but it remained well above the regional values of 0.193 and 0.209. Distributing area across images accounts for part of the gap; most of it comes from where the pixels are placed within each image.
| Policy | Guarantee | Area (%) | Reg. | FN rec. | Mean | (%) | |||
| ISIC 2018 | |||||||||
| Regional CRC, fixed (ours) | group | 0.190 | 0.188 | 0.193 | 11.4 | 4.7 | 0.818 | 0.194 | 41 |
| Regional plug-in (no term) | – | 0.194 | 0.191 | 0.199 | 11.2 | 4.6 | 0.812 | 0.201 | 52 |
| Fixed group quotas | – | 0.219 | 0.214 | 0.236 | 11.4 | 4.1 | 0.792 | 0.231 | 89 |
| Distance-boosted ranking + pooled CRC | pooled | 0.186 | 0.171 | 0.238 | 11.0 | 4.3 | 0.781 | 0.234 | 91 |
| Top-ranked, per-image matched area | – | 0.152 | 0.112 | 0.284 | 11.4 | 4.0 | 0.748 | 0.281 | 100 |
| BUSI | |||||||||
| Regional CRC, fixed (ours) | group | 0.188 | 0.176 | 0.209 | 20.0 | 5.4 | 0.763 | 0.193 | 43 |
| Regional plug-in (no term) | – | 0.197 | 0.185 | 0.226 | 19.3 | 5.2 | 0.749 | 0.214 | 64 |
| Fixed group quotas | – | 0.221 | 0.205 | 0.268 | 20.0 | 4.6 | 0.731 | 0.262 | 94 |
| Distance-boosted ranking + pooled CRC | pooled | 0.184 | 0.152 | 0.271 | 18.2 | 4.5 | 0.712 | 0.266 | 95 |
| Top-ranked, per-image matched area | – | 0.135 | 0.078 | 0.262 | 20.0 | 3.6 | 0.715 | 0.259 | 93 |
Table 5. Regional CRC compared with spatial allocation rules on the same risk map (). Guarantee indicates which expected missed-error fraction is controlled. For fixed group quotas, the ratio of boundary to non-boundary warning shares was tuned on , while the overall quota level was retrospectively matched to the regional CRC policy’s mean test warned area; as evaluated here, this baseline is diagnostic rather than a prospectively specified deployment policy. The distance-boosted ranking adds a constant to the risk scores in before pooled CRC. The per-image matched diagnostic baseline warns, in each test image, exactly as many top-ranked pixels as the regional policy. The last two columns summarize 500 random re-partitions of into halves of the original sizes: the mean of and the percentage of re-partitions in which exceeded .
The two heuristics that move warnings away from the contour without calibration helped, but less than regional CRC. The retrospectively area-matched fixed-group-quota diagnostic baseline missed 0.236 and 0.268 of non-boundary errors, because the same share of is warned in an easy and in a difficult image. The distance-boosted ranking with pooled CRC used slightly less area but missed 0.238 and 0.271; its offset, tuned on development data, did not equalize the group risks on test data, and pooled calibration offers no protection when it fails to do so.
The regional plug-in rule, which omits the finite-sample term, behaved almost like regional CRC on ISIC ( 0.199 vs. 0.193, area 11.2% vs. 11.4%). The difference matters where calibration sets are small. Across 500 re-partitions of the calibration and test cases, the mean non-boundary missed fraction of regional CRC was 0.194 on ISIC and 0.193 on BUSI, consistent with Eq. (8), whereas the plug-in rule averaged 0.201 and 0.214. Under CRC, the non-boundary test mean exceeded in 41% and 43% of re-partitions, which is expected for an expectation-level guarantee; under the plug-in rule, the corresponding percentages were 52% and 64%. The contribution of conformal calibration is therefore not a lower warned area, but a stated property that holds on average even with fewer than 50 eligible cases per group.
Robustness to retraining and to the segmentation backbone
Table 6 repeats the main comparison over three training seeds and with a second backbone. Across seeds, the calibrated non-boundary threshold varied little ( on ISIC and on BUSI), and the pooled deficit was stable: pooled control missed and of non-boundary errors, and regional control and . The mean area cost was 1.7 and 4.6 percentage points, close to the single-seed estimates.
DeepLabV3+ produced more accurate masks (test Dice 0.897 on ISIC and 0.788 on BUSI) and therefore fewer to warn, which lowered the area of both policies. The pattern was unchanged: pooled control missed 0.296 and 0.341 of non-boundary errors, and regional control 0.191 and 0.202, at 10.3% and 17.4% warned area. When the U-Net risk network was paired with DeepLabV3+ and only the thresholds were recalibrated, both group means stayed close to the target, as the calibration argument requires, but the warned area rose to 12.6% and 19.9%. A risk network can thus be moved to a different architecture without losing calibration validity, but a network trained on the deployed architecture’s own out-of-fold errors ranks those errors better.
Table 7 compares RACE-Triage with boundary-aware uncertainty training. The US-Loss segmenter was slightly more accurate than the standard U-Net (test Dice 0.886 on ISIC and 0.758 on BUSI), and its entropy ranked errors better than plain entropy (AUPRC 0.361 vs. 0.318 on ISIC, 0.318 vs. 0.276 on BUSI). Its uncertainty is trained with boundary emphasis, however, and it remained concentrated along the contour: pooled CRC on the US-Loss entropy missed 0.402 and 0.441 of non-boundary errors. Regional CRC brought the non-boundary mean close to the target with either score, as the calibration argument predicts, but with the entropy it required 16.9% and 27.8% of the image, compared with 11.2% and 19.6% with the refitted risk network. The two approaches therefore act at different steps. US-Loss improves the score that a segmenter provides, whereas RACE-Triage converts a given score into warnings with a stated group-wise property, at an area that depends on how well the score ranks distant errors.
| Risk score | Pooled CRC AUPRC | Pooled CRC | Pooled CRC Area | Regional CRC | Regional CRC Area |
| ISIC 2018 (US-Loss segmenter, test Dice 0.886) | |||||
| US-Loss entropy | 0.361 | 0.402 | 10.6 | 0.198 | 16.9 |
| Multiscale (refitted) | 0.452 | 0.309 | 9.6 | 0.194 | 11.2 |
| BUSI (US-Loss segmenter, test Dice 0.758) | |||||
| US-Loss entropy | 0.318 | 0.441 | 16.3 | 0.211 | 27.8 |
| Multiscale (refitted) | 0.426 | 0.355 | 14.8 | 0.207 | 19.6 |
Table 7. Boundary-Guided US-Loss [14] as an alternative segmenter and risk score. Both scores are computed on the US-Loss segmentation, so the policies differ only in the ranking they calibrate (, fixed thresholds). AUPRC refers to raw errors.
Error-load-adaptive allocation
The adaptive family addresses a weakness of fixed thresholds: they warn similar areas on easy and difficult images. On ISIC, regional adaptive CRC reduced the mean warned area from 11.4% to 10.6% (difference −0.8, −1.2 to −0.4) at essentially unchanged group risk ( 0.197 vs. 0.193). Relative to pooled adaptive CRC it showed the same trade-off as the fixed variants, with lower non-boundary risk ( 0.197 vs. 0.301) at a larger area. Figure 5 shows that the saving came from images with low predicted load (6.9% to 5.2%), and that the non-boundary missed fraction became more uniform across load tertiles, from 0.131–0.229 to 0.168–0.214.
Figure 5. Warned area of regional CRC with fixed and load-adaptive families by tertile of predicted error load (tertile boundaries fixed on development data). Numbers above the bars give .
On BUSI, adaptation did not help. The mean area changed by +0.6 percentage points (−0.7 to 1.9), and the non-boundary missed fraction increased slightly to 0.227. The high-load tertile illustrates the mechanism. Confidently missed lesions and false detections have severely underestimated loads, so the calibrated multiplier must be large enough to cover them. This inflates the budget of every image whose load is estimated correctly. The ISIC area reduction was also smaller than the three percentage points that we had predeclared as a material improvement. We therefore report the adaptive family as an analysis and keep fixed thresholds as the default policy.
External transfer to PH2
The frozen ISIC pipeline and thresholds were applied to PH2 without modification, so no guarantee applies and the results are empirical. Both group means of the regional policy exceeded the target (, ). The likely cause is the larger lesions and the under-segmented margins, which produced more distant errors than the ISIC calibration cases. The relative pattern was nevertheless preserved. Regional control reduced the non-boundary missed fraction by 0.109 (−0.151 to −0.068) relative to pooled control, at 1.5 additional percentage points of area (1.1–1.9), and false-negative component recall rose from 0.648 to 0.752.
Table 8 evaluates target-site recalibration through repeated splits of PH2. The splits reuse the same 200 images, so they form a sensitivity analysis of recalibration within PH2 rather than 100 independent external validations or a prospective deployment. Each calibration half contained a median of 59 cases with non-boundary errors (range 51–66). With frozen ISIC thresholds, the regional non-boundary mean exceeded in 94% of the test halves. After recalibration, the regional policy missed on average 0.191 of boundary and 0.193 of non-boundary errors (interquartile range of over splits 0.171–0.216), and false-negative component recall rose to 0.792. The cost was 2.3 additional percentage points of area relative to the frozen thresholds (16.2% vs. 13.9%), and the regional area varied across splits with an interquartile range of 14.7%–
| Policy | Area | FN rec. | (%) | ||
| Pooled, frozen ISIC | 0.162 | 0.372 | 12.4 | 0.649 | 100 |
| IQR | 0.146–0.179 | 0.341–0.401 | 11.3–13.5 | ||
| Regional, frozen ISIC | 0.215 | 0.263 | 13.9 | 0.751 | 94 |
| IQR | 0.196–0.233 | 0.241–0.287 | 12.6–15.1 | ||
| Pooled, recalibrated | 0.144 | 0.349 | 13.4 | 0.668 | 100 |
| IQR | 0.129–0.160 | 0.318–0.379 | 12.0–14.9 | ||
| Regional, recalibrated | 0.191 | 0.193 | 16.2 | 0.792 | 41 |
| IQR | 0.176–0.205 | 0.171–0.216 | 14.7–17.8 |
Table 8. Target-site recalibration on . Values are means over 100 stratified random splits into 100 calibration and 100 test images; frozen rows apply the ISIC thresholds to the same test halves. IQR rows give the interquartile range over splits. The last column is the percentage of splits in which exceeded ; the guarantee concerns the mean over splits, not each split.
| Setting | Test Dice | Pooled | Regional | Regional | Area (%) Pooled | Area (%) Regional | ||
| ISIC 2018 | ||||||||
| U-Net, seed 1 (main results) | 0.883 | 0.260 | 0.120 | 0.318 | 0.188 | 0.193 | 9.8 | 11.4 |
| U-Net, 3 seeds (mean ± SD) | 0.881±0.003 | 0.258±0.008 | 0.122±0.003 | 0.309±0.012 | 0.189±0.003 | 0.197±0.007 | 9.9±0.3 | 11.6±0.4 |
| DeepLabV3+, own | 0.897 | 0.270 | 0.110 | 0.296 | 0.186 | 0.191 | 8.9 | 10.3 |
| DeepLabV3+, U-Net , recalibrated | 0.897 | 0.245 | 0.095 | 0.327 | 0.190 | 0.198 | 10.6 | 12.6 |
| BUSI | ||||||||
| U-Net, seed 1 (main results) | 0.751 | 0.230 | 0.095 | 0.364 | 0.176 | 0.209 | 15.1 | 20.0 |
| U-Net, 3 seeds (mean ± SD) | 0.746±0.011 | 0.225±0.005 | 0.097±0.008 | 0.352±0.019 | 0.181±0.006 | 0.204±0.012 | 15.0±0.7 | 19.6±0.9 |
| DeepLabV3+, own | 0.788 | 0.240 | 0.100 | 0.341 | 0.179 | 0.202 | 13.2 | 17.4 |
| DeepLabV3+, U-Net , recalibrated | 0.788 | 0.215 | 0.085 | 0.372 | 0.184 | 0.211 | 15.6 | 19.9 |
Table 6. Robustness of the pooled-versus-regional comparison to retraining and to the segmentation backbone (, fixed thresholds). Each seed retrains the five fold backbones, the final segmenter and , followed by a new calibration on the same . DeepLabV3+ is paired either with its own risk network, trained on its own out-of-fold predictions, or with the U-Net risk network, in which case only the thresholds are recalibrated. are the calibrated regional thresholds.
17.8% Recalibrated pooled control met its overall target but still missed 0.349 of non-boundary errors. In this dataset, recalibration with 100 labelled images thus brought the mean group-wise missed fractions over splits back below the targets, although individual test halves still exceeded in 41% of splits, as an expectation-level guarantee. The number of labelled cases needed at another site depends on the number of error-positive cases within each region and the selected risk targets; a general sample-size requirement is not established by this experiment. Recalibration also moves only the thresholds: errors to which the risk map assigns low risk, such as the confident failures in Figure 7, remain unwarned unless the warned area is increased substantially.

Figure 7. Test cases from ISIC 2018 and BUSI. (a,b) Cases with the median paired reduction in ; (c,d) cases with the largest regional (failure cases). From left: image with reference (solid) and predicted (dashed) contours; errors over the groups and ; risk map; warnings of pooled and regional CRC, with unwarned errors in red. In (d) the prediction is empty.
Ablation and sensitivity analyses
Table 9 varies one component of the default regional policy at a time on ISIC. A narrower band () shifted some contour errors into and slightly increased area and fragmentation. Removing the distance channel from the risk network increased the area by 0.7 percentage points at equal risk. Replacing the risk network with the local head P1 still satisfied both targets, because calibration validity does not depend on score quality, but it required 17.8% of the image and produced 7.6 regions per image. This illustrates that a valid calibration does not make a weak score useful.
| Variant | Area | Reg. | FN rec. | ||
| Default ( px) | 0.188 | 0.193 | 11.4 | 4.7 | 0.818 |
| Narrower band ( px) | 0.191 | 0.196 | 11.9 | 5.2 | 0.821 |
| No distance channel in | 0.187 | 0.195 | 12.1 | 4.9 | 0.806 |
| Local head P1 as | 0.190 | 0.198 | 17.8 | 7.6 | 0.734 |
| Overlay dilated by 2 px | 0.151 | 0.172 | 15.3 | 2.9 | 0.861 |
| Union with component CRC | 0.176 | 0.181 | 12.9 | 5.0 | 0.897 |
Table 9. Ablation of the regional fixed-threshold policy on ISIC 2018 test data. Each variant is recalibrated on except the two superset constructions, which enlarge the calibrated overlay.
Two superset constructions trade area for additional protection. Dilating the calibrated overlay by two pixels merged fragments (2.9 regions per image) and lowered both missed fractions, but increased the area to 15.3%. Adding a separately calibrated component-level policy (§ 3.5; , 301 eligible calibration cases) raised false-negative component recall to 0.897, with a test component-miss fraction of 0.168, for 1.5 additional percentage points of area. Because both constructions only add warnings, the regional pixel-level statements remain valid.
Figure 6 varies the target and the band width. Regional CRC kept the non-boundary mean close to each target for every band width, whereas pooled control exceeded the non-boundary target at every , even at (0.188 on ISIC and 0.231 on BUSI). The target set the warned area: tightening from 0.20 to 0.10 raised the regional area from 11.4% to 18.9% on ISIC and from 20.0% to 30.9% on BUSI, and relaxing it to 0.30 lowered the area to 7.3% and 13.1%. The band width mattered much less. A wider band () moved contour-adjacent errors into and reduced the area slightly, to 11.1% and 19.2% at . The choice of is therefore the main lever for trading protection against review effort, and it should be set from the cost of a missed error in the intended application.
Figure 6. Non-boundary missed fraction versus warned area for and pixels. Colours encode (dashed lines) and marker shapes the band width; arrows connect pooled CRC to regional CRC with at the same . Every setting is recalibrated on with the same risk network.
The regional gain was not tied to the detection criterion or to one error type (Appendix § 9). With a stricter overlap requirement (), false-negative component recall fell to 0.412 under pooled and 0.537 under regional control on ISIC, and to 0.318 and 0.469 on BUSI, so the absolute advantage ranged from 0.096 to 0.173 across the criteria tested (Table B.11). Regional control reduced the non-boundary missed fraction for both error types. Missed lesion area remained harder to cover than spurious regions: on ISIC, regional control missed 0.204 of non-boundary false-negative pixels and 0.171 of false-positive pixels, compared with 0.341 and 0.274 under pooled control (Table B.12). On BUSI, malignant cases had the highest regional non-boundary missed fraction (0.224) and the largest warned area (26.6%). Only four normal test images contained non-boundary errors, which is too few for a separate conclusion.
The guarantee concerns raw disagreement . When the default policy was scored with -distant errors only, the missed fractions were 0.236 in and 0.204 in . Errors far from the reference boundary are harder to rank than boundary slivers, so the raw-error guarantee does not transfer to this stricter definition; a dedicated loss would have to be calibrated if -distant errors were the primary target.
Secondary tail control (Table 10) kept the per-group violation rates near or below on ISIC (0.046 and 0.053), and the union rate was 0.058. This required 23.7% warned area, roughly twice the area of expectation control. On BUSI the same construction warned 38.6% of the image, which is unlikely to be acceptable for review. These results support CRC as the primary policy and tail control as an option only where calibration sets are large.
| Dataset | Viol. | Viol. | Either | Area (%) | |
| ISIC 2018 | 16 / 6 | 0.046 (0.026–0.075) | 0.053 (0.022–0.107) | 0.058 (0.036–0.090) | 23.7 |
| BUSI | 4 / 2 | 0.060 (0.020–0.135) | 0.077 (0.021–0.185) | 0.082 (0.036–0.155) | 38.6 |
Table 10. Secondary tail control (, fixed thresholds). Violation rates are the shares of error-positive test cases with , with exact 95% intervals below; “Either” is the per-case union rate, bounded by 0.10.
Qualitative examples and computational cost
Figure 7 shows four test cases. Cases (a) and (b) are, for each dataset, the image with non-boundary errors whose paired reduction in was closest to the median; cases (c) and (d) are the images with the largest regional . In (a), the prediction misses the low-contrast lesion periphery. The resulting false negatives lie partly beyond the band, and the pooled policy leaves this periphery unwarned while the regional policy covers it. In (b), a confident false detection in a hypoechoic region receives only moderate risk in its interior. That risk falls below the pooled threshold but above the non-boundary threshold, so only the regional policy warns it.
The failure cases show the limits of the approach. In (c), a compact false positive far from the lesion is predicted with high confidence. Its border lies in the band and is warned, but its interior receives a risk below both thresholds and remains unwarned under either policy. In (d), a small lesion is missed entirely, so the prediction is empty and the whole image belongs to . The risk map responds only weakly at the lesion; the regional policy warns part of it and the pooled policy none. In both cases the deficit lies in the risk map rather than in the calibration, and no threshold short of warning a large part of the image would repair it.
The additional inference cost is small. The distance transform, risk network and policy step add 0.7, 2.8 and 0.6 ms to the 3.9 ms segmenter pass, giving 8.0 ms per image on an A100 GPU. The training cost is dominated by the out-of-fold protocol, which requires six segmenter runs (about 34 minutes each on ISIC) and one risk-network run (41 minutes) per seed. Calibration itself takes seconds once the loss curves are cached.
Discussion
What regional control provides and what it costs
The central observation of this study is that a correctly calibrated pooled policy can be adequate overall and still inadequate where it matters most. On both modalities, pooled CRC met its own target by over-protecting the boundary band, which contains most error pixels. It left roughly one third of non-boundary errors unwarned in an average error-positive case. This is not a failure of the pooled guarantee, which never promised protection for a subgroup. It is a consequence of a pooled objective in which the denominator is dominated by contour disagreement. Separating the two groups changes what is controlled rather than how well the score ranks errors. The same risk map, recalibrated with two constraints, brought both group means close to the target and increased the recall of size-filtered false-negative components on both modalities. This pattern held across three training seeds, a second segmentation backbone and stricter definitions of a detected component, so it does not appear to depend on one trained model or one evaluation choice.
Regional control is not free, and we do not present it as an efficiency gain. On ISIC it required about one sixth more warned area than pooled control, and on BUSI about one third more. The cost is also unevenly distributed. The 95th percentile of per-image warned area rose to 23.5% on ISIC and 38.5% on BUSI, so on ultrasound a minority of images would require reviewing more than a third of the image. Overlays also became more fragmented, because the non-boundary threshold admits scattered low-risk pixels. Whether this trade-off is worthwhile depends on the relative cost of a missed distant error and of additional review. Distant errors include missed lesion lobes and false detections, which change lesion extent or count, whereas boundary slivers mostly change area by a few pixels. In many settings the former are the more consequential. The matched-area comparisons show that the benefit cannot be obtained simply by warning more of the top-ranked pixels, even when the area is matched image by image, because the ranking itself favours the contour.
The calibration rule itself is standard CRC applied within groups, and we do not claim a new conformal procedure. The comparison with simpler allocation rules clarifies what it adds in this setting. Heuristics that move warnings away from the contour, such as fixed group quotas or a boosted ranking, improved on pooled control but left more non-boundary errors unwarned at similar area. The quota ratio and ranking offset were tuned on development data, but this tuning offers no groupwise risk guarantee at deployment. The overall quota level was retrospectively matched to the regional policy’s mean test warned area, so the fixed-group-quota comparison is diagnostic rather than a prospectively specified deployment policy. An uncorrected groupwise threshold produced nearly the same overlay as regional CRC when calibration sets were large, but its average risk exceeded the target on BUSI, where fewer than 50 eligible cases were available. The practical value of the conformal step is therefore a stated property at small calibration sizes, not a lower warned area.
Regional calibration reallocates warnings according to the available risk ranking. It does not guarantee that every clinically important error receives a high score or is warned in an individual image. The confident failures in Figure 7 remain unwarned under both policies, and a better risk map, not a different threshold, would be needed to change that.
The adaptive family gave a mixed result that we consider informative. Scaling budgets by predicted error load saved area on easy dermoscopy images and equalized risk across difficulty strata. On ultrasound it failed precisely on the cases that motivate adaptation, namely confident misses with severely underestimated load. Its usefulness therefore depends on error-load estimation more than on the calibration mechanism. Improving load estimates for confident failures, for example with case-level failure detectors [20], is a more promising direction than finer budget rules.
Implications for review workflows
The proposed overlay is designed to be attached to an existing segmentation tool without modifying it. The segmentation network and its mask are unchanged, the risk network uses only outputs that any segmenter provides, and the calibration step requires labelled cases from the target site. The sample size needed for useful thresholds depends on the number of error-positive cases within each region and the selected risk targets; there is no universal requirement of a few hundred labelled cases. The output has a direct interpretation at the level of a case population: averaged over cases that contain errors in a given region, and over the draw of the calibration set, the fraction of those errors left outside the overlay is at most . It does not promise that any particular image, or any particular calibration, meets this level; individual images can exceed it, which is why the per-image tail and the workload percentiles are reported separately. This statement can be communicated to reviewers, and can be set per group if missed distant errors are considered more costly than missed boundary errors. The sensitivity analysis shows that , rather than the band width, is the main lever for review effort. On , recalibration with one hundred labelled images brought the mean missed fractions over repeated splits back below both targets at a moderate area cost; whether a comparable number suffices at other sites remains to be shown. Warned area and component recall are proxies, so these results do not show that review becomes faster or that more errors are corrected. The fragmentation results suggest that display post-processing is worthwhile. Because the guarantee survives any superset, it can be applied without invalidating the calibration, provided that the added area is reported.
Limitations
Several limitations qualify these findings. First, the guarantee is an expectation over exchangeable error-positive cases. It does not bound any particular image, and on small calibration sets the realized test mean can exceed the target, as observed for BUSI, where only 49 calibration cases contained non-boundary errors. Second, BUSI does not provide patient identifiers. Our duplicate audit detects repeated acquisitions but cannot establish patient-level independence, so the BUSI guarantee rests on an assumption, and the BUSI results should be read as empirical evidence. A replication on an ultrasound dataset with patient identifiers and a patient-disjoint split would test this assumption directly; we did not have access to such a dataset. Third, the external results show that guarantees do not transfer across sites. Recalibration restored the targets, but it requires labelled local data, and our recalibration experiment reused one public dataset rather than a prospective deployment. Fourth, the band width, the component size threshold and the detection overlap are operational choices; the conclusions were stable across the settings we tested, but these settings were not derived from clinical requirements. Missed-error fraction and warned area are proxies for review effort that were not validated in a reader study, and the clinical benefit of the overlay, including review time and the rate at which warned errors are actually corrected, remains to be demonstrated. Fifth, three seeds describe retraining variability only coarsely, and the two backbones are convolutional; transformer-based and foundation segmenters were not evaluated. Finally, the study considers binary 2D segmentation; multi-class and volumetric settings require group definitions and losses that we have not evaluated.
Conclusion
This paper studied how to turn a learned pixel error map into review warnings with stated properties when the segmentation itself is left unchanged. We showed that conformal risk control applied to all error pixels at once can meet its overall target while leaving roughly one third of errors far from the predicted contour unwarned on dermoscopy and breast ultrasound. Calibrating warnings separately in a boundary band and in the remaining image brought the missed-error fraction of both regions close to the target. It also increased the recall of size-filtered false-negative components, at the cost of 1.6 and 4.9 percentage points of additional warned area on average and considerably more on the most difficult images (95th percentile 23.5% and 38.5%). The effect persisted across training seeds, a second backbone and stricter detection criteria, and it was not reproduced by heuristic allocation rules or by top-ranked warnings matched in area image by image. On an external dermoscopy set, recalibration with one hundred labelled images brought the mean group-wise missed fractions over repeated splits back below the targets. An error-load-adaptive budget reduced area on dermoscopy but not on ultrasound, where confident failures have underestimated loads. The approach requires only the outputs of an existing segmenter, a risk network trained on out-of-fold predictions and a modest calibration set from the deployment site.
Future work will evaluate the overlay in a reader study with measured review time and correction rates, extend the group definition to multi-class and volumetric segmentation, and investigate better error-load estimates for confident failures, which currently limit both adaptive allocation and the coverage of confidently missed lesions.
CRediT authorship contribution statement
Nishi Kanta Paul: Conceptualization, Methodology, Software, Formal analysis, Visualization, Writing – original draft. Md Shihabul Islam Shovo: Methodology, Software, Validation, Data curation, Writing – review & editing. Hoileong Lee: Supervision, Writing – review & editing.
Data availability
All datasets used in this study are publicly available from their providers. The ISIC 2018 lesion segmentation data can be obtained from the ISIC challenge archive (https://challenge.isic-archive.com/data/). The BUSI dataset is distributed by its authors as described in [60]. The PH² database is available for research use after registration (https://www.fc.up.pt/addi/ph2%20database.html). The code will be made publicly available upon publication.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Declaration of generative AI and AI-assisted technologies in the manuscript preparation process
During the preparation of this work the authors used Claude (Anthropic) in order to draft and edit the manuscript text and to write the code that generates the figures and tables. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.
Proofs
Throughout, denote calibration cases and a test case, each consisting of an image and its reference mask. The segmenter, risk network, band width, warning family and grid are fixed independently of these cases, so that every quantity defined in § 3 is a deterministic function of the case it belongs to. Let denote the all-warning endpoint.
Proposition 1 (Calibration on error-positive cases). Let be independent and identically distributed, fix with , and let and . Let be given by Eq. (7) computed on the cases in , with if or if the set in Eq. (7) is empty. Then
Proof. Let and fix an index set with . Eligibility of a case is a function of that case alone. Because the cases are independent, conditionally on the cases and are independent and each follows the conditional distribution of a case given . They are therefore exchangeable. If , then and . If , the losses , , take values in , are non-increasing in because the family is nested, and vanish at . Eq. (7) is the CRC rule with loss bound , and the proof of [22 Theorem 1] applies without change when is restricted to a finite grid that contains . Hence for every , and averaging over the conditional distribution of given proves the claim. Remarks. (i) The number of eligible cases is random; the statement holds for every realization and therefore on average. (ii) If the cases are exchangeable rather than independent, the conclusion still holds: any permutation of the indices in leaves the event that exactly these cases are eligible unchanged, so the eligible cases remain exchangeable conditionally on that event. (iii) The guarantee is an expectation over both the calibration set and the test case. It does not imply that a particular calibrated meets the target, nor that every image does.
Proposition 2 (Secondary tail control). For an error-positive case in group with erroneous pixels, let be the risks of these pixels and , which exists for . Warn every pixel of with .
(a) The case violates the target, , if and only if .
(b) Under the conditions of Proposition 1, let be the -th smallest critical score of the eligible calibration cases, with , and if . Then .
Proof. (a) A pixel is missed exactly when it is erroneous and , so the number of missed errors is . Since this number is an integer, holds if and only if at least erroneous pixels have risk below , which is equivalent to . (b) If , then , no pixel has , and . Otherwise, by (a), a violation occurs exactly when . In that case is strictly smaller than at least calibration scores, so its rank among the exchangeable scores is at most under any tie-breaking. Conditionally on and eligibility, this rank is uniformly distributed when ties are broken at random, so the probability is at most . Averaging over completes the proof. The per-case union bound follows from .
Additional analyses
Duplicate-audit sensitivity. At Hamming thresholds of 4, 8 and 12, the perceptual-hash audit grouped 29, 52 and 91 ISIC images into 13, 21 and 34 groups, and 11, 17 and 36 BUSI images into 5, 8 and 15 groups. We re-partitioned both datasets with the most inclusive grouping, again assigning every grouped image to the fitting pool, and repeated training and calibration. The regional policy changed little: , and warned area were 0.187, 0.195 and 11.5% on ISIC and 0.179, 0.214 and 20.4% on BUSI, compared with 0.188, 0.193 and 11.4%, and 0.176, 0.209 and 20.0%, in the primary analysis. This is a sensitivity analysis of the grouping procedure. Because a hash audit cannot detect different views of the same patient, it addresses only one source of dependence in BUSI and does not establish patient-level independence.
Component criteria. Table B.11 reports false-negative component recall under stricter overlap and size requirements. Recall decreases for both policies as the criterion becomes stricter, while the advantage of regional control persists.
| ISIC 2018 Pooled | ISIC 2018 Regional | ISIC 2018 | BUSI Pooled | BUSI Regional | BUSI | ||
| 0.10 | 25 | 0.722 | 0.818 | +0.096 | 0.611 | 0.763 | +0.152 |
| 0.25 | 25 | 0.583 | 0.709 | +0.126 | 0.472 | 0.642 | +0.170 |
| 0.50 | 25 | 0.412 | 0.537 | +0.125 | 0.318 | 0.469 | +0.151 |
| 0.10 | 100 | 0.701 | 0.815 | +0.114 | 0.598 | 0.771 | +0.173 |
| 0.50 | 100 | 0.368 | 0.511 | +0.143 | 0.291 | 0.447 | +0.156 |
Table B.11. Size-filtered false-negative component recall of pooled and regional CR under stricter detection criteria (fixed thresholds, ). is the required overlap and the minimum component size in pixels; the first row is the default criterion of Table 4.
| Subset | Pooled | Regional | Area (%) | |
| Error type in | ||||
| ISIC 2018, FP pixels | 74 | 0.274 | 0.171 | – |
| ISIC 2018, FN pixels | 96 | 0.341 | 0.204 | – |
| BUSI, FP pixels | 38 | 0.318 | 0.188 | – |
| BUSI, FN pixels | 33 | 0.402 | 0.226 | – |
| BUSI diagnostic category | ||||
| Normal (17 images) | 4 | 0.412 | 0.236 | 7.9 / 10.6 |
| Benign (55 images) | 27 | 0.338 | 0.193 | 14.6 / 19.8 |
| Malignant (26 images) | 21 | 0.389 | 0.224 | 20.8 / 26.6 |
Table B.12. Non-boundary missed-error fraction by error type and by BUSI category (fixed thresholds, ). is the number of test cases with errors of the given type, or of any type, in ; areas are averaged over all test images of a category. These are empirical subgroup summaries without a separate guarantee.
Error type and diagnostic category. Table B.12 splits the non-boundary missed fraction by error type and the BUSI results by category. The regional reduction applies to both error types, and false-negative pixels, which correspond to missed lesion area, remain harder to cover than false-positive pixels.
References
- [1]O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Vol. 9351 of Lecture Notes in Computer Science, Springer, 2015, pp. 234–241. doi: 10.1007/978-3-319-24574-4_28.DOI
- [2]F. Isensee, P. F. Jaeger, S. A. A. K. Kohl, J. Petersen, K. H. Maier-Hein, nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nature Methods 18 (2) (2021) 203–211. doi:10.1038/s41592-020-01008-z.DOI
- [3]J. Chen, J. Mei, X. Li, Y. Lu, O. Yu, Q. Wei, X. Luo, Y. Xie, E. Adeli, Y. Wang, M. P. Lungren, S. Zhang, L. Xing, L. Lu, A. Yuille, Y. Zhou, TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers, Medical Image Analysis 97 (2024) 103280. doi:10.1016/j.media.2024.103280.DOI
- [4]J. Ma, Y. He, F. Li, L. Han, C. You, B. Wang, Segment anything in medical images, Nature Communications 15 (2024) 654. doi:10.1038/s41467-024-44824-z.DOI
- [5]X. Wang, G. Jiao, EAGNet: Edge aware guided network for skin lesion image segmentation, Biomedical Signal Processing and Control 113 (2026) 109157. doi:10.1016/j.bspc.2025.109157.DOI
- [6]X. Bai, Y. Ma, W. Wang, X. Gao, Z. Xue, Edge-guided adaptive multi-feature learning for breast ultrasound image segmentation, Biomedical Signal Processing and Control 91 (2024) 106076. doi:10.1016/j.bspc.2024.106076.DOI
- [7]Y. Zheng, Y. Dong, Z. Yao, J. Yu, G. Wu, Lightweight medical SAM with self-prompting for ultrasound breast tumor segmentation, Biomedical Signal Processing and Control 127 (2026) 111184. doi:10.1016/j.bspc.2026.111184.DOI
- [8]H. Kervadec, J. Bouchtiba, C. Desrosiers, E. Granger, J. Dolz, I. Ben Ayed, Boundary loss for highly unbalanced segmentation, Medical Image Analysis 67 (2021) 101851. doi:10.1016/j.media.2020.101851.DOI
- [9]D. Karimi, S. E. Salcudean, Reducing the Hausdorff distance in medical image segmentation with convolutional neural networks, IEEE Transactions on Medical Imaging 39 (2) (2020) 499–513. doi:10.1109/TMI.2019.2930068.DOI
- [10]Y. Gal, Z. Ghahramani, Dropout as a Bayesian approximation: Representing model uncertainty in deep learning, in: Proceedings of the 33rd International Conference on Machine Learning, Vol. 48 of Proceedings of Machine Learning Research, 2016, pp. 1050–1059.arxiv.org/abs/1506.02142
- [11]B. Lakshminarayanan, A. Pritzel, C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles, in: Advances in Neural Information Processing Systems, Vol. 30, 2017, pp. 6402–6413.arxiv.org/abs/1612.01474
- [12]A. Kendall, Y. Gal, What uncertainties do we need in Bayesian deep learning for computer vision?, in: Advances in Neural Information Processing Systems, Vol. 30, 2017, pp. 5574–5584.arxiv.org/abs/1703.04977
- [13]G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, T. Vercauteren, Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks, Neurocomputing 338 (2019) 34–45. doi:10.1016/j.neucom.2019.01.103.DOI
- [14]A. A. Munia, T. Broti, M. Hasan, A. Khosravi, I. Hossain, Z. Sharmin, A. Rahman, Boundary-guided US-Loss: Uncertainty-aware training for reliable image segmentation, Biomedical Signal Processing and Control 129 (2027) 111413. doi:10.1016/j.bspc.2026.111413.DOI
- [15]A. Jungo, M. Reyes, Assessing reliability and challenges of uncertainty estimations for medical image segmentation, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Vol. 11765 of Lecture Notes in Computer Science, Springer, 2019, pp. 48–56. doi:10.1007/978-3-030-32245-8_6.DOI
- [16]A. Mehrtash, W. M. Wells, C. M. Tempany, P. Abolmaesumi, T. Kapur, Confidence calibration and predictive uncertainty estimation for deep medical image segmentation, IEEE Transactions on Medical Imaging 39 (12) (2020) 3868–3878. doi:10.1109/TMI.2020.3006437.DOI
- [17]C. Corbière, N. Thome, A. Bar-Hen, M. Cord, P. Pérez, Addressing failure prediction by learning model confidence, in: Advances in Neural Information Processing Systems, Vol. 32, 2019, pp. 2902–2913.arxiv.org/abs/1910.04851
- [18]K. Li, L. Yu, P.-A. Heng, Towards reliable cardiac image segmentation: Assessing image-level and pixel-level segmentation quality via self-reflective references, Medical Image Analysis 78 (2022) 102426. doi:10.1016/j.media.2022.102426.DOI
- [19]F. Uslu, M. Varela, A robust quality estimation method for medical image segmentation with small datasets, Biomedical Signal Processing and Control 95 (2024) 106300. doi:10.1016/j.bspc.2024.106300.DOI
- [20]M. Zenk, D. Zimmerer, F. Isensee, J. Traub, T. Norajitra, P. F. Jäger, K. Maier-Hein, Comparative benchmarking of failure detection methods in medical image segmentation: Unveiling the role of confidence aggregation, Medical Image Analysis 101 (2025) 103392. doi:10.1016/j.media.2024.103392.DOI
- [21]S. Bates, A. Angelopoulos, L. Lei, J. Malik, M. Jordan, Distribution-free, risk-controlling prediction sets, Journal of the ACM 68 (6) (2021) 1–34. doi:10.1145/3478535.DOI
- [22]A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, T. Schuster, Conformal risk control, in: The Twelfth International Conference on Learning Representations (ICLR), 2024.
- [23]J. Brunekreef, E. Marcus, R. Sheombarsing, J.-J. Sonke, J. Teuwen, Kandinsky conformal prediction: Efficient calibration of image segmentation algorithms, in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 4135–4143. doi:10.1109/CVPR52733.2024.00396.DOI
- [24]L. Mossina, J. Dalmau, L. Andéol, Conformal semantic image segmentation: Post-hoc quantification of predictive uncertainty, in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024, pp. 3574–3584. doi:10.1109/CVPRW63382.2024.00361.DOI
- [25]A. M. Wundram, P. Fischer, M. Mühlebach, L. M. Koch, C. F. Baumgartner, Conformal performance range prediction for segmentation output quality control, in: Uncertainty for Safe Utilization of Machine Learning in Medical Imaging (UNSURE 2024), Lecture Notes in Computer Science, Springer, 2024, pp. 81–91. doi:10.1007/978-3-031-73158-7_8.DOI
- [26]Y. Romano, R. F. Barber, C. Sabatti, E. Candès, With malice toward none: Assessing uncertainty via equalized coverage, Harvard Data Science Review 2 (2) (2020). doi:10.1162/99608092.03f00592.DOI
- [27]C. Lu, A. Lemay, K. Chang, K. Höbel, J. Kalpathy-Cramer, Fair conformal predictors for applications in medical imaging, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, 2022, pp. 12008–12016. doi:10.1609/aaai.v36i11.21459.DOI
- [28]T. Ding, A. Angelopoulos, S. Bates, M. Jordan, R. Tibshirani, Class-conditional conformal prediction with many classes, in: Advances in Neural Information Processing Systems, Vol. 36, 2023, pp. 64555–64576.arxiv.org/abs/2306.09335
- [29]S. Adhikary, N. Chabi, A. Mastmeyer, Bound-aware per-organ recall risk control for multi-organ CT segmentation under clinical domain shift, arXiv:2608.18193 (MICCAI UNSURE Workshop 2026) (2026). arXiv:2608.18193.arxiv.org/abs/2608.18193
- [30]S. A. A. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. H. Maier-Hein, S. M. A. Eslami, D. J. Rezende, O. Ronneberger, A probabilistic U-Net for segmentation of ambiguous images, in: Advances in Neural Information Processing Systems, Vol. 31, 2018, pp. 6965–6975.arxiv.org/abs/1806.05034
- [31]C. F. Baumgartner, K. C. Tezcan, K. Chaitanya, A. M. Hötker, U. J. Muehlematter, K. Schawkat, A. S. Becker, O. Donati, E. Konukoglu, PHiSeg: Capturing uncertainty in medical image segmentation, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Vol. 11765 of Lecture Notes in Computer Science, Springer, 2019, pp. 119–127. doi:10.1007/978-3-030-32245-8_14.DOI
- [32]C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On calibration of modern neural networks, in: Proceedings of the 34th International Conference on Machine Learning, Vol. 70 of Proceedings of Machine Learning Research, 2017, pp. 1321–1330.arxiv.org/abs/1706.04599
- [33]K.-C. Kahl, C. T. Lüth, M. Zenk, K. Maier-Hein, P. F. Jaeger, ValUES: A framework for systematic validation of uncertainty estimation in semantic segmentation, in: The Twelfth International Conference on Learning Representations (ICLR), 2024.
- [34]B. Lambert, F. Forbes, S. Doyle, H. Dehaene, M. Dojat, Trustworthy clinical AI solutions: A unified review of uncertainty quantification in deep learning models for medical image analysis, Artificial Intelligence in MedicineDOI
- [35]K. Zou, Z. Chen, X. Yuan, X. Shen, M. Wang, H. Fu, A review of uncertainty estimation and its application in medical imaging, Meta-Radiology 1 (1) (2023) 100003. doi:10.1016/j.metrad.2023.100003.
- [36]V. V. Valindria, I. Lavdas, W. Bai, K. Kamnitsas, E. O. Aboagye, A. G. Rockall, D. Rueckert, B. Glocker, Reverse classification accuracy: Predicting segmentation performance in the absence of ground truth, IEEE Transactions on Medical Imaging 36 (8) (2017) 1597–1606. doi:10.1109/TMI.2017.2665165.
- [37]R. Robinson, V. V. Valindria, W. Bai, O. Oktay, B. Kainz, H. Suzuki, M. M. Sanghvi, N. Aung, J. M. Paiva, F. Zemrak, K. F. Fung, E. Lukaschuk, A. M. Lee, V. Carapella, Y. J. Kim, S. K. Piechnik, S. Neubauer, S. E. Petersen, C. Page, P. M. Matthews, D. Rueckert, B. Glocker, Automated quality control in image segmentation: application to the UK Biobank cardiovascular magnetic resonance imaging study, Journal of Cardiovascular Magnetic Resonance 21 (2019) 18. doi:10.1186/s12968-019-0523-x.DOI
- [38]C. Corbière, N. Thome, A. Saporta, T.-H. Vu, M. Cord, P. Pérez, Confidence estimation via auxiliary models, IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (10) (2022) 6043–6055. doi:10.1109/TPAMI.2021.3085983.DOI
- [39]C. M. Tsai, J.-D. Lee, Reliability-aware dual-representation DCE-MRI segmentation with disagreement-guided quality control for breast tumor delineation, Biomedical Signal Processing and Control 130 (2027) 111588. doi:10.1016/j.bspc.2026.111588.
- [40]P. F. Jaeger, C. T. Lüth, L. Klein, T. J. Bungert, A call to reflect on evaluation practices for failure detection in image classification, in: The Eleventh International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2211.15259
- [41]Y. Geifman, R. El-Yaniv, Selective classification for deep neural networks, in: Advances in Neural Information Processing Systems, Vol. 30, 2017, pp. 4878–4887.
- [42]V. Vovk, A. Gammerman, G. Shafer, Algorithmic Learning in a Random World, Springer, New York, 2005.
- [43]A. N. Angelopoulos, S. Bates, Conformal prediction: A gentle introduction, Foundations and Trends in Machine Learning 16 (4) (2023) 494–591. doi:10.1561/2200000101.DOI
- [44]A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, L. Lei, Learn then test: Calibrating predictive algorithms to achieve risk control, The Annals of Applied Statistics 19 (2) (2025) 1641–1662. doi:10.1214/24-AOAS1998.DOI
- [46]J. Teneggi, M. Tinvan, J. W. Stayman, J. Sulam, How to trust your diffusion model: A convex optimization approach to conformal risk control, in: Proceedings of the 40th International Conference on Machine Learning, Vol. 202 of Proceedings of Machine Learning Research, 2023, pp. 33940–33960.
- [47]B. Tan, Z. Wang, J. Duan, K. Xu, H. T. Shen, X. Shi, F. Shen, Conformal lesion segmentation for 3D medical images, arXiv:2510.17897 (2025). arXiv:2510.17897.arxiv.org/abs/2510.17897
- [48]M. Dai, W. Luo, T. Li, Statistical management of the false discovery rate in medical instance segmentation based on conformal risk control, in: 2025 IEEE 3rd International Conference on Image Processing and Computer Applications (ICIPCA), 2025, arXiv:2504.04482.arxiv.org/abs/2504.04482
- [49]W. Zheng, S. Ray, What can conformal risk control certify for PET/CT tumor segmentation?, in: Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications, Vol. 329 of Proceedings of Machine Learning Research, 2026, pp. 1063–1065.
- [50]J. Fayyad, S. Alijani, H. Najjaran, Empirical validation of conformal prediction for trustworthy skin lesions classification, Computer Methods and Programs in Biomedicine 253 (2024) 108231. doi:10.1016/j.cmpb.2024.108231.DOI
- [51]B. Almaslukh, A reliable breast cancer diagnosis approach using an optimized deep learning and conformal prediction, Biomedical Signal Processing and Control 98 (2024) 106743. doi:10.1016/j.bspc.2024.106743.
- [52]O. El Allam, M. Hamlich, Quantization-aware Mondrian conformal prediction for embedded ECG classification, Biomedical Signal Processing and Control 127 (2026) 111217. doi:10.1016/j.bspc.2026.111217.DOI
- [53]H. Wislander, P. J. Harrison, G. Skogberg, S. Jackson, M. Fridén, J. Karlsson, O. Spjuth, C. Wählby, Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images, IEEE Journal of Biomedical and Health Informatics 25 (2) (2021) 371–380. doi:10.1109/JBHI.2020.2996300.
- [54]I. Gibbs, J. J. Cherian, E. J. Candès, Conformal prediction with conditional guarantees, Journal of the Royal Statistical Society Series B: Statistical Methodology 87 (4) (2025) 1100–1126. doi:10.1093/jrsssb/qkaf008.DOI
- [55]R. Foygel Barber, E. J. Candès, A. Ramdas, R. J. Tibshirani, The limits of distribution-free conditional predictive inference, Information and Inference: A Journal of the IMA 10 (2) (2021) 455–482. doi:10.1093/imaia/iaaa017.arxiv.org/abs/1903.04684
- [56]R. J. Tibshirani, R. F. Barber, E. J. Candès, A. Ramdas, Conformal prediction under covariate shift, in: Advances in Neural Information Processing Systems, Vol. 32, 2019, pp. 2530–2540.arxiv.org/abs/1904.06019
- [57]R. F. Barber, E. J. Candès, A. Ramdas, R. J. Tibshirani, Conformal prediction beyond exchangeability, The Annals of Statistics 51 (2) (2023) 816–845. doi:10.1214/23-AOS2276.DOI
- [58]N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, H. Kittler, A. Halpern, Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC), arXiv:1902.03368 (2019). arXiv:1902.03368.arxiv.org/abs/1902.03368
- [59]P. Tschandl, C. Rosendahl, H. Kittler, The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions, Scientific Data 5 (2018) 180161. doi:10.1038/sdata.2018.161.DOI
- [60]W. Al-Dhabyani, M. Gomaa, H. Khaled, A. Fahmy, Dataset of breast ultrasound images, Data in Brief 28 (2020) 104863. doi:10.1016/j.dib.2019.104863.DOI
- [61]A. Pawłowska, P. Karwat, N. Zołek, Letter to the editor. re: “[dataset of breast ultrasound images by W. Al-Dhabyani, M. Gomaa, H. Khaled & A. Fahmy, data in brief, 2020, 28, 104863]”, Data in Brief 48 (2023) 109247. doi:10.1016/j.dib.2023.109247.DOI
- [62]T. Mendonça, P. M. Ferreira, J. S. Marques, A. R. S. Marcal, J. Rozeira, PH² – a dermoscopic image database for research and benchmarking, in: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2013, pp. 5437–5440. doi:10.1109/EMBC.2013.6610779.DOI
- [63]F. Milletari, N. Navab, S.-A. Ahmadi, V-Net: Fully convolutional neural networks for volumetric medical image segmentation, in: 2016 Fourth International Conference on 3D Vision (3DV), 2016, pp. 565–571. doi:10.1109/3DV.2016.79.
- [64]L.-C. Chen, Y. Zhu, G. Papandreou, F. Schro ff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Computer Vision – ECCV 2018, Vol. 11211 of Lecture Notes in Computer Science, Springer, 2018, pp. 833–851. doi:10.1007/978-3-030-01234-2_49.DOI
- [65]K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2016, pp. 770–778. doi:10.1109/CVPR. 2016.90.arxiv.org/abs/1512.03385
- [66]L. Maier-Hein, A. Reinke, P. Godau, M. D. Tizabi, F. Buettner, E. Christodoulou, B. Glocker, F. Isensee, et al., Metrics reloaded: recommendations for image analysis validation, Nature Methods 21 (2) (2024) 195–212. doi:10.1038/s41592-023-02151-z.DOI
- [67]A. Reinke, M. D. Tizabi, M. Baumgartner, M. Eisenmann, D. Heckmann-Nötzel, A. E. Kayur, T. Rätsch, C. H. Sudre, et al., Understanding metric-related pitfalls in image analysis validation, Nature Methods 21 (2) (2024) 182–194. doi:10.1038/s41592-023-02150-0.DOI
- [68]B. Efron, R. J. Tibshirani, An Introduction to the Bootstrap, Chapman & Hall, New York, 1993.
- [69]A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köp̈f, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, PyTorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems, Vol. 32, 2019, pp. 8024–8035.arxiv.org/abs/1912.01703
- [70]D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: 3rd International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980





