WorksComputer Science
Calibrate Once, Grow the Vocabulary: Candidate-Independent Scores for Conformal Risk Control in Open-Vocabulary Segmentation
Abstract
Open-vocabulary segmenters accept class names at inference. Conformal risk control can bound how often their prediction sets miss the true label, but we show that this bound breaks when names are added after calibration. On ADE20K, a threshold calibrated for 10% risk at 150 names reaches up to 15.4% risk at 845 names in released CAT-Seg models. The risk exceeds the target on every calibration split, yet the prediction sets grow far less than the target requires. In SED they instead grow larger than needed. The cause is that pruning, cross-label attention and vocabulary normalisation make each label's score depend on the other requested labels. Scores can move either way, so a threshold recalibrated at a larger vocabulary need not hold at smaller ones. We remove this dependence without training by scoring each label against a fixed reference bank of names. A label's score then stays the same when names are added, so one calibration keeps its guarantee as the vocabulary grows. The guarantee holds whatever names are added, even ones chosen per image after seeing it. On three benchmarks and five segmenters, the converted models meet the target at every vocabulary size we evaluate. For CAT-Seg and SED at alpha=10%, they use at most one more label per pixel than the released models recalibrated at the grown vocabulary. Treating names instead of images as calibration samples also bounds the miss rate of added names. If the annotated names are a random sample of the names in the calibration images, an added name's expected miss rate on those images is at most the target. For harder added names, the excess is at most their gap in mean miss rate. Applied to Grounding DINO, which splits large vocabularies into chunks, the conversion also makes inference about six times faster.
Introduction
Open-vocabulary segmenters assign pixels to classes named in text [17, 26, 42, 84, 89]. The list of requested names, the vocabulary, often changes after deployment. Robots are asked for objects outside the list their map was built with [33, 44, 88], species catalogues grow between releases [70, 75, 76], and taggers, captioners or language models choose names per image [18, 29, 31, 45, 57, 60, 62, 90] (§ 20). (Lettered sections and S-numbered tables and figures are in the supplementary material.)
A prediction set offers several plausible labels for a pixel rather than a single guess. In an annotation tool, for example, it shows the annotator the plausible labels and bounds how often the correct one is missing. Conformal prediction [2, 78] and conformal risk control (CRC) [4], both applied to segmentation [12, 54], calibrate a score threshold on labelled images so that the expected fraction of missed pixels, the risk, stays below a target . The guarantee requires calibration and test images to be exchangeable.
Calibration uses a source vocabulary, and the guarantee counts pixels whose true label belongs to it. Adding names produces a grown vocabulary and raises two questions: does the guarantee for the source labels survive, and what can be guaranteed for the unannotated added labels? Recalibrating on the labelled images restores the source-label guarantee when the new vocabulary, or a fixed rule for choosing it, is known in advance [39]. This needs little compute but is costly to operate. Every change of vocabulary needs another recalibration, images processed before it finishes are not covered, and the outputs barely show when the guarantee has broken. We study how to keep the guarantee without recalibrating (Sec. H.5).
In released segmenters, adding names can push the risk above the target or make the sets over-conservative, larger than the target requires. The cause is that the score of a requested label, or candidate, depends on the other candidates. We call this candidate dependence. CAT-Seg’s code [17] keeps only the 256 labels with the highest initial scores in each image crop, SED [84] lets the labels attend to one another, and FC-CLIP [89] normalises its scores with a softmax over the vocabulary. Adding labels therefore changes the scores of the labels already present, and with them the risk of a threshold calibrated at the source vocabulary (Figure 1). When ADE20K [92] grows from its 150 benchmark classes to 845 classes, a threshold calibrated for 10% risk reaches 13.8–15.4% risk with CAT-Seg and FC-CLIP. It exceeds the target on every one of 500 random calibration splits.
Figure 1. Calibration breaks when names are added. Each released segmenter is calibrated by CRC for a risk of (dashed) on the source vocabulary, and the same threshold is used after the vocabulary grows (means over 500 random calibration splits, set size in labels per pixel). After growth, the risk of CAT-Seg and FC-CLIP exceeds the target, and the prediction sets of SED become several times larger.
We make the scores candidate-independent by scoring each label against a fixed reference bank of label names instead of the requested vocabulary. The released network processes the bank without retraining (Figure 3). Each candidate attends to the bank and to itself but not to other candidates, so its score depends only on the image, the label and the bank. We call the converted model anchored and the released model native. The score of a source label then stays the same when names are added. A threshold calibrated once therefore keeps its guarantee for every larger vocabulary, including vocabularies chosen per image. The anchored sets remain far smaller than those of models that are candidate-independent by design, such as CLIPSeg [49] (§ 6.1).

Figure 3. Class attention at one pixel. (a) The sums (2) run over the requested labels, so every score depends on . (b) They run over a fixed bank, computed once per image. Each candidate attends to the bank and to itself but not to other candidates.
Candidate independence also permits a separate guarantee for added labels. On fixed images, a label’s miss rate, the fraction of its pixels left out at a given threshold, then does not depend on the vocabulary. CRC can therefore treat labels instead of images as its exchangeable samples. If the annotated source labels are a random sample of the names that may be requested, CRC bounds the expected miss rate of a randomly added label on the calibration images. Annotating a random subset of the names gives such a sample. When added labels are harder, such as rare benchmark classes, we bound the excess risk by the difference in mean miss rate between added and source labels. We also characterise the worst case, when nothing is assumed about the added labels.
We evaluate five released models (CAT-Seg with ViT-B, ViT-L and DeCLIP [81] backbones, SED and FC-CLIP) on ADE20K 150 845, Pascal Context [55] 59 458 and COCO-Stuff [13] 171 1,283, grown with LVIS [30] categories. The anchored models meet the target at every evaluated vocabulary size on every benchmark. At , their prediction sets have at most one more label per pixel than native recalibration for CAT-Seg and SED, and about three more for FC-CLIP on ADE20K. For CAT-Seg and SED, their mIoU is at most 1.7 points below the native models. Our contributions are:
We show that CRC thresholds calibrated at a source vocabulary fail after growth in five released segmenters. For CAT-Seg and FC-CLIP the risk exceeds the target on every split while the sets grow far less than needed. SED’s sets become over-conservative. We trace both failures to pruning, attention across labels and softmax normalisation, and show that recalibrating at a larger vocabulary does not cover per-image vocabularies (§ 4).
We convert released CAT-Seg, SED and FC-CLIP segmenters and Grounding DINO detectors to candidate-independent scores without additional training, so that one threshold calibrated at the source vocabulary remains valid for every grown vocabulary (§ 5.1, § 6.1 and § 6.2).
We give a guarantee for added labels that are exchangeable with the source labels, bound the excess risk when added labels are harder, and characterise the worst case with no assumption on the added labels (§ 5.3 and § 6.3).
Related work
Conformal prediction in dense and zero-shot vision. Conformal prediction and risk control [4] have been applied to semantic segmentation [54] through region-pooled calibration [12], image-level guarantees [77] and improved image-conditional coverage [50], and to object detection [1, 73]. Extensions support input-adaptive and non-monotone losses [5, 9]. These methods calibrate for a fixed label set. For zero-shot CLIP classifiers, Con-فت [69] transports calibration and test scores jointly. A recent study [67] adds up to 750 distractor names to CLIP’s candidate list, measures the loss of coverage and restores it by recalibrating. Work on temperature scaling across label sets [74] and calibration for novel names after fine-tuning [82] addresses calibration error rather than a risk bound. We show that calibrating at the largest vocabulary does not cover its subsets (§ 6.2) and construct scores that do not change when names are added.
Changing distributions and label sets. Conformal methods for non-exchangeable or drifting data [24, 36, 56] and for label shift [58, 68] reweight calibration data to account for changes in images or class proportions. Vocabulary growth instead changes the score function itself. A selection rule keeps the conformal guarantee if it is applied in the same way to calibration and test points [7, 39]. Calibrating for the worst member of a fixed family covers every member [4, 8], but choosing per point among separately calibrated sets can break coverage [32]. With candidate-independent scores, the worst case over vocabularies is the loss at the source vocabulary. Concurrent work [83] studies a growing hazard vocabulary with vision–language embedding distances, which are candidate-independent. We convert released segmenters whose scores depend on the vocabulary. Xie et al. [85, 86] consider classes unseen at calibration that appear in test data. Here the deployer adds the labels. Our added-label guarantee treats labels as exchangeable units, like the tasks in conformal few-shot learning [25] and class pooling in clustered conformal prediction [20]. We also test weighted conformal prediction [72] when added labels differ.
Candidate dependence in open-vocabulary models. In vision–language models, a class score depends on the other classes queried. Accuracy falls as the vocabulary grows [51, 61], and tuning a model for openness reduces the drop [16]. In segmentation, results change with the evaluation vocabulary and the class names [34, 35]. Several methods prune large vocabularies to improve accuracy [15, 59], as CAT-Seg does within each crop, and Sarić et al. [66] analyse FC-CLIP’s classification, including its learned void class.
Scoring against a fixed reference. Normalising a score against a fixed reference set is standard in speaker verification [6, 63]. It has also been applied to cross-modal retrieval [11] and used with CLIP for out-of-distribution detection [38], score calibration [80] and 3D segmentation [22]. Queries that attend to a shared context but not to other classes’ queries appear in SegPrompt [93] and Perceiver IO [37], and CLIPSeg [49] segments each query independently. In CAT-Seg and SED, attention across labels is linear, so a fixed bank reduces to one cached sum per layer. We use the resulting scores for calibration, not to improve accuracy.
Risk control under a growing vocabulary
Prediction sets and conformal risk control
A segmenter deployed with vocabulary assigns to each pixel of an image and each label a score . The score is candidate-independent if for every vocabulary that contains , that is, if it does not depend on the other requested labels. Given a threshold , the prediction set at a pixel contains the labels that score at least , . We write when the vocabulary matters. All labels share the threshold [65]. Set constructions that normalise over the vocabulary [3, 64] make sets depend on it even when the scores do not (Sec. K.5).
Let be the annotated label of pixel , the source vocabulary at which the threshold is calibrated, and the annotated pixels of whose label lies in (images with none are excluded). The loss of at threshold is the fraction of these pixels whose true label is missing from the set,
It is non-decreasing in , since a higher threshold yields smaller sets. Given calibration images, CRC [4] selects the largest threshold on a fixed grid with . If no grid point qualifies, it selects the lowest one, at which every label is included (the all-label endpoint). If the test image is exchangeable with the calibration images (equally likely in any order), then . We call the risk. The guarantee holds on average over calibration and test images, and counts only pixels of labels in . We evaluate the loss on a random sample of pixels per image, drawn independently of the model. CRC on these samples still bounds the full-image risk over images with a sampled source pixel (Sec. A).
Reusing a threshold after vocabulary growth
The threshold is calibrated with the scores . At deployment the vocabulary grows to , the grown vocabulary, and the scores become . The labels of are the added labels. Growth affects the loss (1) in two ways. First, added labels enter the prediction sets. This only enlarges the sets, so it cannot increase the loss on source labels. Second, the model now sees more candidates, so the score of a source label can change and its true label can cross the threshold in either direction. All models share the first effect. The second is the failure we study, and candidate-independent scores remove it:
Proposition 1 (Transfer of the guarantee). Let be candidate-independent and let be calibrated by CRC at on images exchangeable with the test image. For any , including one chosen from the test image, at every pixel. Hence the loss (1) is identical under and , and holds under .
Whether a source label enters the set depends only on , which does not involve , so the guarantee follows directly from CRC. Candidate-dependent scores can also support per-image vocabularies when the selection rules are fixed before calibration:
Proposition 2 (Per-image vocabularies). Let be rules fixed before calibration. Rule maps an image and randomness , drawn independently for each image, to a vocabulary . Let be the loss (1) under . CRC applied to yields a threshold satisfying for every choice of a rule per image, provided calibration and test examples are exchangeable.
The loss is bounded, non-decreasing and at least as large as every selected loss, so CRC applies [4 Sec. 4.4] (see also [8, Sec. 6.3]). This requires running every rule on the calibration images. Alternatively, a single fixed rule can be rerun on the calibration images [39]`, but changing the rule later requires recalibration. With candidate-independent scores, the maximum over supersets equals , so calibration at the source vocabulary covers rules chosen later.
If adding names can only lower source-label scores, calibration at a lexicon (a large fixed name list) also covers every vocabulary between and . A single softmax over fixed CLIP logits has this property [67]. The segmenters studied here do not, since cross-label attention and FC-CLIP’s void term can also raise scores (§ 4.2).
Why calibration fails
Measured failure
On the five released models and three benchmarks of § 6, we calibrate a CRC threshold at the source vocabulary for and reuse it at the grown one instead of recalibrating. For the three CAT-Seg models and FC-CLIP, the risk exceeds the target on each of 500 splits on all benchmarks (12.8–17.1%; Figure 1). SED fails in the other direction. Its reused threshold is over-conservative, with 8.80 labels per pixel on ADE20K where recalibration needs 5.28.
The outputs give little sign of the failure. Without labels at deployment, the prediction sets are the only signal. For CAT-Seg ViT-L on ADE20K they grow from 1.91 to 2.16 labels per pixel while the risk rises to 15.2%. Meeting the target would need 3.47 (Figure 2). In the 12 combinations that exceed the target, the sets gain only 14–40% of the labels that meeting the target would need. SED’s oversized sets give no sign that they are too large. Detecting either failure requires labelled images at the grown vocabulary, the same data that recalibration needs.
Figure 2. Same threshold before and after growth (CAT-Seg ViT-L, , median split, selected ADE20K image). Blue: labels per pixel. Orange: missed source-labelled pixels. From left to right, the maps miss 2.5, 29.2 and 5.5% of them in this image and 9.8, 15.6 and 9.8% over 305 of the split’s test images.
Sources of candidate dependence
Cost aggregation: CAT-Seg and SED. CAT-Seg [17] computes a cost volume between CLIP image patches and the text embedding of every label. It refines this volume by alternating two steps: spatial aggregation, a Swin block [47] applied to each label separately, and class aggregation, a transformer layer over the labels at each position. Class aggregation uses linear attention [40]: with the feature map , the output for label at a position is
where , and are the query, key and value vectors of label tokens and (spatial and layer indices omitted), and is a set of learned padding tokens that fill the vocabulary up to 256 labels. The sums run over the requested vocabulary , so the output for each label depends on every other label. The decoder then processes each label’s map separately and outputs a per-label logit, trained with a sigmoid loss. CAT-Seg’s code adds a second source of dependence. When , each image crop keeps only the 256 labels with the highest initial score and assigns the others a logit of . SED [84] follows the same design, with a ConvNeXt backbone [48] and linear class attention ((2)) in each of three decoder stages, but without padding or pruning. Its optional fast mode adds category early rejection: each stage keeps only the labels in the top at some pixel ().
Mask classification: FC-CLIP. FC-CLIP [89] predicts mask proposals , each with two logit vectors over the labels: one from its trained classifier and one from mask-pooled CLIP features. The score of label at pixel is , where depends on the vocabulary in three places. Both logit vectors are normalised by a softmax over , their weighted geometric mean is normalised again, and the result is multiplied by one minus a void probability, from a softmax over [66]. The masks and per-query logits do not depend on other labels.
Direction of the effect. Attention over more labels can move source-label scores in either direction. Discarding a label lowers its score, and a single softmax over fixed logits lowers existing probabilities when names are added. FC-CLIP’s void probability also falls, which raises every score, and the combined effect need not be monotone. The mechanisms therefore predict that pruning and a single softmax tend to raise the risk, while cross-label attention can also make a reused threshold over-conservative. Two controls on ADE20K support this. Removing the pruning from CAT-Seg ViT-B lowers the risk of the reused threshold from 15.9% to 7.7% (400 images). SED’s fast mode, whose early rejection discards labels, raises it from 7.1% to 12.8%. Because scores can also rise, recalibrating at a larger vocabulary need not cover the vocabularies within it (§ 3.2). With per-image vocabularies, SED’s threshold recalibrated at 845 names reaches 18.8% risk, and 25.4% when recalibrated at a 17, 001-name lexicon (§ 6.2).
Candidate-independent scores from released models
Reference-anchored conversion
The conversion keeps the released weights and changes only the set over which the sums in (2) run. Fix a reference bank of label names. For each image, we first run the bank through the network with pruning disabled. At every class-attention layer we record the key–value sum and the key sum over the bank and padding tokens ( is empty for SED and for banks of 256 or more names). Each candidate is then run through the same layers with the attention output
The candidate attends to the bank, the padding and itself. The bank does not attend to the candidate, and candidates do not attend to one another. (A candidate that is also a bank name attends to both its bank token and itself.) The other components (spatial aggregation, decoder and per-label logit) already treat each label separately. We do not apply the pruning to 256 labels. The anchored score therefore depends only on the image, the candidate and . It is candidate-independent, so Proposition 1 applies. When a tagger or language model produces the names, adding the source names to its output ensures .
FC-CLIP. The same idea applies to the softmax normalisations. For a label , every softmax over becomes a softmax over (with removed from the bank if it is a bank entry), and the void probability becomes a softmax over . Masks and per-query logits are unchanged. For a label that is a bank entry with a single name, this reproduces the released pipeline run on the vocabulary .
Implementation and cost. Our reimplementation of the native forward pass matches the released code (maximum absolute logit difference for CAT-Seg and for SED). Anchored scores change by no more than this when the other candidates or their order change. Letting the bank also attend to the candidate needs one native pass on the bank plus each candidate. It costs 130 times as much as anchored scoring and changes the logits by only 0.015 on average (ViT-B, COCO-171 bank; banks in § 5.2). On a GH200 with the default banks, the conversion takes 4.0× the time of one native pass at 845 labels for CAT-Seg ViT-L, 1.7× for SED and at least 1.3× for FC-CLIP. Stopping the bank pass once the key and value sums are recorded gives the same scores and, on an RTX 5070 Ti, lowers the ViT-L ratio from 4.3 to 3.7× (Sec. H.3).
The reference bank
The bank sets the context in which each candidate is scored. In our experiments, banks work best when they contain the training vocabulary plus enough other names that a candidate does not dominate its own attention (§ 6.5). For CAT-Seg and SED we use the 171 COCO-Stuff names on which both models were trained (the COCO-171 bank). From the Open Images class list [41] we remove every name that matches an ADE20K or Pascal Context label and add 512 names drawn at random from the rest (the 683-name bank). For FC-CLIP the default bank is its training vocabulary (the 133 COCO panoptic classes) merged with the 683-name bank, giving 701 names. Some bank names coincide with requested labels: 47 of the 150 ADE20K source names are in the COCO-171 part, and 12 of the Open Images names are COCO-Stuff added labels. The guarantees hold for any bank chosen without the calibration and test images. We chose ours on 400 ADE20K validation images. The other 1,600, Pascal Context, COCO-Stuff and the reserved images (§ 6.6) test it independently.
Guarantees for the added labels
Propositions 1 and 2 concern the source labels. To obtain a guarantee for unannotated added labels, we need an assumption linking their losses to those of the source labels. Fix the calibration images and define the label loss as the fraction of the pixels with true label whose score for is below . For a candidate-independent score, depends on the label and the images but not on the vocabulary in which is requested. Labels can then take the place of images as the exchangeable units of calibration:
Proposition 3 (Added labels). Let be candidate-independent, and let the source labels and an added label , all present in the fixed calibration images, be exchangeable, for instance because are drawn uniformly without replacement from the requested names present in these images. The label-level threshold , the largest grid threshold with (the all-label endpoint if none), satisfies .
This is CRC applied to the label losses (Sec. B). The expectation is over the label draw with the images fixed, so the bound is for the calibration images, not new ones. A label absent from the calibration images is not covered. If each label comes with its own sample of images, drawn the same way for every label, the same construction also bounds an added label’s expected miss rate on a new image of it (Proposition S1 and Sec. C). The label-level threshold needs annotations of the source labels only. Deploying , with the image-level threshold of § 3.1, keeps both guarantees. For candidate-dependent scores, the source-label losses are computed at and the added label’s at , so exchangeable names do not give exchangeable losses. When source and added labels come from different distributions, their gap in difficulty bounds the excess risk:
Proposition 4 (Harder added labels). Let be candidate-independent, let the source labels be drawn i.i.d. from a distribution and the added label independently from , both over names present in the calibration images, and let , where and . Then .
Since the added label is independent of , its risk is , and the first term is the risk of a further draw from , which is at most by Proposition 3. If a fraction of the added labels comes from an arbitrary distribution and the rest from , the excess is at most . If unannotated added labels can be adversarial, any valid method, recalibration included, selects the all-label endpoint with probability at least (Proposition S2).
Obtaining exchangeable labels
Dense annotation takes more than 1.5 hours per image for the 30 classes of Cityscapes [19], so large vocabularies are annotated only in part. When the names that may be requested are known (a taxonomy, a product catalogue), annotating a uniformly random subset makes the remaining names exchangeable with the annotated ones. The guarantee is then an average over the remaining names. LVIS and Open Images let annotators or a classifier choose which names to annotate [30, 41]. For such labels, the gap bound of Proposition 4 applies (Sec. G.2).
Experiments
Benchmarks. On ADE20K [92] the source vocabulary is the 150-class benchmark, and the grown one adds the 695 classes of ADE20K-847 not among them, in a fixed nested order. On Pascal Context [55] the vocabulary grows from the 59-class benchmark to the 458 named classes of its 459-class version. On COCO-Stuff [13] the source vocabulary is the 171 classes on which CAT-Seg and SED were trained, and the grown one adds the 1,112 LVIS [30] categories that are not COCO-Stuff concepts (Sec. E.1). We use 2,000 validation images per benchmark and the official class names.
Models. We evaluate five released models and their anchored conversions: CAT-Seg with ViT-B and ViT-L [21] CLIP backbones, CAT-Seg with the DeCLIP [81] EVA02-L [23] backbone, SED ConvNeXt-L, and FC-CLIP ConvNeXt-L. Inference follows the released pipelines (Sec. H.2). Unless stated otherwise, the anchored models use the 683-name bank (701 names for FC-CLIP), chosen on ADE20K and kept fixed for Pascal Context and COCO-Stuff.
Protocol and metrics. We store scores at 4,096 uniformly sampled pixels per image. Each of 500 random splits divides the images into equal calibration and test halves. CRC chooses the threshold on the calibration half at , and we evaluate it on the test half. At the grown vocabulary we report the risk (the loss (1) averaged over splits) and the set size (the mean number of labels per pixel). We compare three thresholds: the reused native one, the recalibrated native one, which is the reference for set size (§ 4.1), and our conversion calibrated once at the source vocabulary (anchored). The overhead of the conversion is the difference in set size between the anchored and recalibrated native models. Thresholds are image-level (§ 3.1) except in § 6.3. We developed the method on the validation images. § 6.6 confirms the results on images not used in development. Code, manifests and results are available at https://github.com/muninml10/open-vocab-crc (Sec. E.2).
Anchored models after growth
With the source-vocabulary threshold, the anchored models meet the target for every model and benchmark (Figure 4; images in Figure S2 and Figure S3). In each of the 12 combinations where the native threshold fails, the anchored sets grow by – with the vocabulary and the native sets by –, so growth of the sets does not show that the target is met. The overhead is at most 0.99 labels per pixel for CAT-Seg and SED, and on COCO-Stuff SED’s anchored sets are smaller than the recalibrated ones. FC-CLIP’s overhead is 2.99 labels on ADE20K (Table S28), and on COCO-Stuff its sets are large with or without the conversion. Models that are candidate-independent by design need 46.2 (CLIPSeg) and 99.3 (SCLIP [79]) labels per pixel at 845 names (Table S3). Claude Opus 5.5, asked to name marked pixels, omits the true label so often that CRC includes all 845 names (Sec. F).
Figure 4. The anchored models meet the target in every case. Risk and set size after growth, as in Figure 1. Blue: our anchored models, with the threshold from the source vocabulary. Grey: native recalibration at the grown vocabulary. The values, in labels per pixel, are in Table S4.
Figure S2. Prediction sets at 845 names (CAT-Seg ViT-L, , median split, selected ADE20K image). Blue: labels per pixel. Orange: missed source-labelled pixels. From left to right, the maps miss 29.2, 5.5 and 2.7% of them in this image and 15.6, 9.8 and 9.7% over the 305 test images of this split with stored full-image outputs. Random images are in Figure S3.
Figure S3. Prediction sets at 845 names on random images (CAT-Seg ViT-L, ; the thresholds of the median split as in Figure S2; 6 images drawn at random with a fixed seed from the same pool of 305 test images). Blue: labels per pixel; orange: missed source-labelled pixels; faded: unlabelled pixels. Above each panel: missed source-labelled pixels and mean labels per source-labelled pixel on that image.
| Model | risk | set size |
| SCLIP (CLIP ViT-B) | 9.8 | 25.0 → 99.3 |
| CLIPSeg [49] | 9.4 | 10.2 → 46.2 |
| anchored CAT-Seg ViT-B | 9.9 | 2.76 → 5.98 |
Table S3. Scores that are candidate-independent by construction (ADE20K , , threshold calibrated at 150 names): risk at 845 names (%) and set size at 150 and 845 names. SCLIP: CLIP ViT-B patch–text similarities with the correlative self-attention and sliding-window inference of SCLIP [79]. The CLIPSeg row uses 400 development images and 200 random splits into halves; the other rows use the 500 splits of Table S6.
Table S28. FC-CLIP: bank size and overhead (ADE20K, 845 labels, ): set sizes of the anchored and the recalibrated native model. “Added risk”: risk on the 695 added labels for the anchored threshold / for the native threshold moved to the anchored set size (Sec. G.3).
| bank names | anchored | native | recal. | overhead | added risk |
| 133 | 13.56 | 8.18 | 5.37 | 31.0 / 26.8 | |
| 701 | 11.18 | 8.18 | 2.99 | 28.6 / 29.4 | |
| 1,366 | 10.83 | 8.18 | 2.65 | 29.2 / 29.9 | |
| 2,731 | 10.71 | 8.18 | 2.53 | 29.2 / 30.1 |
Table S28.
Vocabularies chosen per image
Proposition 1 covers names added to for each test image by any process, even one that cannot be rerun on the calibration images. On ADE20K, we add to the 150 source names of each image one of: the names annotated in that image (a perfect tagger), 50, 200 or 500 random names, the 50 names the native model scores highest (a tagger sharing the model’s confusions), or the worst of these for that image. A language model listing what it sees, mostly names outside the 845, is a further rule (last block of Table 1; Sec. F.1). For every rule and every evaluated model, the anchored threshold has 9.9% risk on ADE20K and 9.8% on Pascal Context (Table 1; Pascal rules in Sec. F.1). Under the worst rule the reused native threshold exceeds the target for every model, including SED (over-conservative in § 4.1). Native calibration also meets the target when it uses the deployed rule, or the worst case over the rules (Proposition 2). Without knowing the rule, every native calibration we tested exceeds the target for at least one model (§ 4.2 and F.1; by deployment in Table S36). With the language model’s names, recalibration at all 845 names exceeds it for every model (12.6–30.3%).
| anchored | native | ||||||
| calibration needs | once | reused | recal. all | random all | recal. 17k | rules all rules | |
| ADE20K, 150 → 845 | |||||||
| CAT-Seg ViT-L | 9.9/4.2 | 14.9/2.2 | 9.5/3.6 | 10.4/3.3 | 4.1/8.9 | 9.9/3.4 | |
| SED | 9.9/5.9 | 12.8/3.4 | 18.8/2.0 | 18.4/2.1 | 25.4/1.4 | 9.9/5.0 | |
| FC-CLIP | 9.9/9.7 | 14.1/4.8 | 10.3/7.4 | 10.8/6.9 | 5.6/17.4 | 9.9/7.7 | |
| Pascal Context, 59 → 458 | |||||||
| CAT-Seg ViT-L | 9.8/2.3 | 15.2/1.5 | 10.0/2.0 | 10.9/1.9 | – | 9.8/2.1 | |
| SED | 9.8/2.8 | 12.3/1.9 | 21.1/1.1 | 19.8/1.2 | – | 9.8/2.4 | |
| ADE20K, names from a language model | |||||||
| CAT-Seg ViT-L | 9.9/4.6 | 20.6/2.7 | 12.6/4.1 | – | – | 9.9/4.8 | |
| SED | 9.9/5.7 | 20.5/3.4 | 30.3/2.3 | – | 41.2/1.5 | 9.9/6.4 | |
| FC-CLIP | 9.9/8.0 | 21.5/4.4 | 15.6/5.9 | – | 7.8/11.0 | 9.9/8.8 |
Table 1. Per-image vocabularies (): risk (%) / labels per pixel under each image’s worst (highest-loss) rule (last block: a language model’s names). Calibration needs: names known in advance (all: every grown name; 17k: a 17, 001-name lexicon). random: calibrated on source names plus 500 (ADE20K) or 300 (Pascal) random grown names. rules: calibrated under the worst rule or on those names. –: not run. All rules: Table S13 to Table S16.
Table S13. Per-image vocabularies, CAT-Seg ViT-L (ADE20K, ): source-label risk (%) / set size on source-labelled pixels. Native thresholds: reused, recalibrated at 845 names or at a 17,001-name lexicon, calibrated on vocabularies of the source names plus 500 random added names (random), or calibrated under the row’s own rule (matched; Proposition 2 for the worst case).
| Added names | anchored once | anchored reused | native recal. 845 | native random | native recal. 17k | native matched |
| present in image | 9.9/2.3 | 10.3/1.9 | 6.2/2.9 | 6.8/2.6 | 2.3/6.0 | 9.9/2.0 |
| 50 random | 9.9/2.4 | 10.3/2.0 | 6.2/2.9 | 6.8/2.7 | 2.2/6.5 | 9.9/2.0 |
| 200 random | 9.9/2.8 | 11.8/2.0 | 7.3/3.1 | 8.0/2.9 | 2.9/7.3 | 9.9/2.4 |
| 500 random | 9.9/3.7 | 14.1/2.1 | 9.1/3.4 | 9.9/3.1 | 3.9/8.3 | 9.9/3.1 |
| tagger, top 50 | 9.9/4.2 | 14.8/2.2 | 9.4/3.7 | 10.3/3.3 | 4.0/9.2 | 9.9/3.5 |
| worst, per image | 9.9/4.2 | 14.9/2.2 | 9.5/3.6 | 10.4/3.3 | 4.1/8.9 | 9.9/3.4 |
Table S13.
| Deployment | Native threshold | Anchored, calibrated once |
| One vocabulary, fixed before deployment | Recalibrate at it: valid, usually smaller sets (Table S4); cheaper above the break-even | Valid; cheaper below the break-even |
| Vocabulary fixed per project before its images are seen | Recalibrate for each project: valid, one pass over the calibration images each time | Valid, no pass |
| Vocabularies chosen per image by a rule replayable on the calibration images, e.g. a fixed image tagger | Calibrate under the rule, or the worst case over a fixed set of rules: valid, smaller sets; repeated whenever a rule changes | Valid |
| Names from a known lexicon, rule not known | Recalibrate at the lexicon: meets the target for CAT-Seg and FC-CLIP, with sets about twice the anchored size at a large lexicon, not for SED, which needs more memory than a 16 GB card from 3,380 names (Table S35). Calibrate on random vocabularies from the lexicon: exceeds the target under the tagger rule for all three models | Valid |
| Names listed per image by a language model, not limited to a list (Table S17) | Calibrate under the model’s names on the calibration images: valid, larger sets than anchored; repeated whenever the model or prompt changes. Recalibrating at the 845 names exceeds the target for all three models; a 17,001-name lexicon meets it for FC-CLIP at 1.4× the anchored size, not for SED | Valid |
| Names written per image or per query by a user, or added during annotation | Not replayable; guaranteed if every vocabulary the process can produce is scored on every calibration image (Proposition 2) | Valid, with appended to the names |
| Vocabulary grown by a party without the labelled calibration images | Not possible | Valid |
Table S36. Calibrating once or recalibrating, by deployment. All guarantees in the table assume exchangeable calibration and test images; per-image vocabularies are evaluated on ADE20K and Pascal Context (Table 1).
A guarantee for the added labels
Table 2 tests Proposition 3. The added-label risk is the label loss averaged over added labels, whose benchmark annotations are used for evaluation only. The source vocabulary is a random subset of the grown one (exchangeable labels) or the benchmark’s own classes.
| Source vocabulary | anchored image-level | anchored label-level (ours) | native, label-level recal. at grown | native, label-level reused from source | |
| random draw | |||||
| ADE20K | .10 | 23.7 / 6.8 | 9.1 / 28.9 | 9.2 / 35.8 | 20.8 / 9.7 |
| ADE20K | .05 | 14.3 / 15.0 | 4.5 / 74.5 | 4.3 / 228.1 | 11.8 / 24.3 |
| Pascal Context | .10 | 16.7 / 3.7 | 7.6 / 11.5 | 7.6 / 14.9 | – |
| Pascal Context | .05 | 9.8 / 7.1 | 2.8 / 68.4 | 2.7 / 92.8 | – |
| COCO | .10 | 17.7 / 5.7 | 8.9 / 11.9 | 9.0 / 13.6 | – |
| COCO | .05 | 10.3 / 10.4 | 4.0 / 30.2 | 4.0 / 42.1 | – |
| benchmark’s own | |||||
| ADE20K | .10 | 34.2 / 4.5 | 22.2 / 9.0 | 22.6 / 10.1 | 33.8 / 5.1 |
| Pascal Context | .10 | 24.7 / 2.5 | 22.3 / 2.7 | 26.5 / 2.6 | 39.9 / 1.6 |
| COCO | .10 | 32.1 / 3.3 | 17.6 / 6.3 | 21.2 / 5.6 | 29.2 / 3.9 |
Table 2. Added-label risk on the calibration images (%) / set size at the grown vocabulary (CAT-Seg ViT-L, 500 trials). A dash means not run. Each trial draws a calibration half and, for random draw, a source vocabulary (on ADE20K one of 10 fixed draws; Table S18 redraws it).
Random source vocabulary. The image-level threshold of § 6.1 misses 23.7% of an added label’s pixels on ADE20K, with 6.8 labels per pixel where the label-level threshold needs 28.9. When the source vocabulary is redrawn in every trial (Table S18), the label-level threshold on the anchored scores keeps the added-label risk on the calibration images below the target for all five models on all three benchmarks (7.5–9.6%; and held-out images in Sec. G.1). The native model, calibrated the same way and deployed at 845 names, misses 20.8%, since its label losses change with the vocabulary. Native label-level CRC at the grown vocabulary is valid but needs that vocabulary in advance and gives larger sets (35.8 labels per pixel).
| Model | benchmark | random source vocabulary, anchored | random source vocabulary, native recal. | random source vocabulary, anchored | random source vocabulary, native recal. | benchmark’s own, anchored | |
| CAT-Seg ViT-B | ADE20K | 9.5 / 46.4 | 9.6 / 59.5 | 4.5 / 102.6 | 3.4 / 430.1 | 22.6 | 13.2 |
| CAT-Seg ViT-B | Pascal Context | 7.5 / 23.9 | 7.6 / 32.1 | 2.6 / 98.4 | 2.5 / 160.7 | 27.4 | 19.0 |
| CAT-Seg ViT-B | COCO-Stuff | 9.1 / 21.7 | 9.1 / 26.6 | 4.1 / 58.8 | 4.0 / 177.3 | 22.5 | 13.1 |
| CAT-Seg ViT-L | ADE20K | 9.5 / 27.6 | 9.5 / 34.2 | 4.5 / 76.4 | 4.0 / 260.1 | 22.2 | 12.8 |
| CAT-Seg ViT-L | Pascal Context | 7.6 / 11.5 | 7.6 / 14.9 | 2.8 / 68.4 | 2.7 / 92.8 | 22.3 | 13.9 |
| CAT-Seg ViT-L | COCO-Stuff | 8.9 / 11.9 | 9.0 / 13.6 | 4.0 / 30.2 | 4.0 / 42.1 | 17.6 | 8.2 |
| CAT-Seg + DeCLIP | ADE20K | 9.5 / 19.0 | 9.4 / 22.2 | 4.5 / 52.3 | 4.2 / 194.3 | 20.7 | 11.3 |
| CAT-Seg + DeCLIP | Pascal Context | 7.5 / 10.4 | 7.6 / 17.6 | 2.8 / 62.4 | 2.8 / 132.0 | 18.1 | 9.6 |
| CAT-Seg + DeCLIP | COCO-Stuff | 9.0 / 9.5 | 9.1 / 10.1 | 4.1 / 22.5 | 4.1 / 46.8 | 14.6 | 5.2 |
| SED | ADE20K | 9.5 / 45.1 | 9.5 / 51.4 | 4.6 / 118.5 | 4.6 / 139.8 | 22.2 | 12.8 |
| SED | Pascal Context | 7.5 / 17.5 | 7.6 / 20.1 | 2.8 / 80.4 | 2.8 / 86.9 | 26.2 | 17.8 |
| SED | COCO-Stuff | 9.0 / 21.2 | 9.1 / 23.9 | 4.0 / 60.1 | 4.1 / 66.6 | 24.3 | 14.9 |
| FC-CLIP | ADE20K | 9.6 / 61.0 | 9.6 / 65.6 | 4.5 / 142.2 | 4.6 / 152.4 | 16.4 | 7.0 |
| FC-CLIP | Pascal Context | 7.5 / 43.4 | 7.5 / 49.7 | 2.5 / 132.3 | 2.5 / 158.7 | 22.9 | 14.5 |
| FC-CLIP | COCO-Stuff | 9.0 / 82.9 | 9.1 / 66.1 | 4.0 / 227.6 | 4.1 / 186.4 | 16.6 | 7.2 |
Table S18. Added-label risk for every model (%) / set size at the grown vocabulary, label-level threshold, 500 trials. Random source vocabulary: anchored model calibrated at the source vocabulary, and native model calibrated at the grown vocabulary. Benchmark’s own: anchored model, with of Proposition 4 in points.
The benchmark’s own source classes. These are the frequent classes, and label-level CRC leaves an added-label risk of 17.6–22.3%. The decomposition of Proposition 4 explains the excess. Source labels stay below the target, while added labels have a loss higher by –13.9 points, against with a random source vocabulary. Annotating 100 random added names on the calibration images restores exchangeability and keeps the ADE20K added-label risk at 9.3% for ViT-L and 9.3% for SED (Sec. G.2).
Open-vocabulary detection
We also convert three Grounding DINO detectors [46, 91] on COCO [43] 80 1,204 (COCO LVIS v1). OWLv2 [53], candidate-independent by design, is a control. In the anchored detector, the image branch sees only an 80-name COCO bank, and each candidate is scored against the object queries computed from the bank. At LVIS scale the reused native threshold meets the target for every detector (Table S38). At anchored sets are no larger than those of native recalibration, and inference is 6.0–6.3 faster, as the image branch runs twice instead of once per chunk of names (Table S33). Anchoring leaves more added-class objects with no overlapping box, often setting the label-level threshold to the all-label endpoint (§ 17), and cuts their top-1 accuracy from 23.8–34.2% to 14.7–16.5%.
Table S33. Runtime and memory (ratios from unrounded times): seconds per image and peak memory, fp32, on one GH200. Segmentation, 845 labels: “anchored” is the bank pass plus all 845 candidates, and ratio = (features + anchored) / (features + native); the rows were timed consecutively on an idle device over the same 100 development images, except the one-label-at-a-time row (local RTX 5070 Ti). ViT-B and ViT-L share the aggregator, so only the CLIP features differ. Detection, 1,204 names: ratio = anchored / native, timed consecutively on an idle device over the first 100 benchmark images. Peak memory is the maximum over all passes of a run; Table S35 gives each arm separately.
| Model | features | native | anchored | ratio | peak memory |
| CAT-Seg ViT-B, COCO-171 bank, sliding window 640 | 0.03 | 0.27 | 0.88 | 3.0× | 8.3 GB |
| CAT-Seg ViT-B, 683-name bank | 0.03 | 0.27 | 1.29 | 4.3× | 8.4 GB |
| CAT-Seg ViT-L, COCO-171 bank | 0.07 | 0.28 | 0.88 | 2.8× | 9.3 GB |
| CAT-Seg ViT-L, 683-name bank | 0.07 | 0.28 | 1.29 | 4.0× | 9.4 GB |
| SED ConvNeXt-L, COCO-171 bank, whole image 768 | 0.03 | 0.34 | 0.40 | 1.2× | 8.0 GB |
| SED ConvNeXt-L, fast mode (top-8), native only | 0.03 | 0.07 | – | – | 5.5 GB |
| SED ConvNeXt-L, 683-name bank | 0.03 | 0.34 | 0.60 | 1.7× | 8.2 GB |
| FC-CLIP ConvNeXt-L, 133-name bank, shortest edge 800 | 0.31† | 0.0003 | 0.0018 | – | 3.7 GB |
| FC-CLIP ConvNeXt-L, 701-name bank | 0.40† | 0.0003 | 0.0076 | ≥1.3× | 4.8 GB |
| native, one label at a time against the bank (ViT-B, RTX 5070 Ti) | – | 350 | – | – | – |
| Detection, 1,204 names | |||||
| Grounding DINO T, 80-name bank | – | 2.97 | 0.50 | 0.17× (6.0× faster) | 9.3 GB |
| MM-Grounding DINO T, 80-name bank | – | 3.04 | 0.51 | 0.17× (6.0× faster) | 9.3 GB |
| Grounding DINO B, 80-name bank | – | 3.31 | 0.52 | 0.17× (6.3× faster) | 9.5 GB |
| OWLv2 B (control) | – | 0.09 | – | – | 1.4 GB |
Table S33.
| Detector | native reused | native recal. | anchored once |
| Grounding DINO T | 9.6/94.7 | 9.9/88.3 | 9.9/34.2 |
| MM-Grounding DINO T | 9.0/23.0 | 9.9/20.2 | 9.9/3.1 |
| Grounding DINO B | 9.8/3.7 | 9.9/3.7 | 9.9/3.6 |
| OWLv2 B (control) | 9.9/8.3 | – | – |
Table S38. Detection label-loss surrogate (COCO 80 → 1,204, COCO ∪ LVIS v1, chunks of 40 names, ): risk on COCO boxes (%) / labels per ground-truth box. The all-label endpoint covers an unmatched box without localizing it.
Choice of reference bank
Every bank we tried meets the target and changes only the set size (Table S27). For anchored CAT-Seg ViT-B on ADE20K (400 images), the sets contain 82.34 labels per pixel with the candidate alone, 7.92 with the COCO-171 bank and 6.44 with the 683-name bank, and grow slowly with larger banks. Five redraws of the Open Images names differ by at most 0.28 labels (Table S29). On COCO-Stuff, a smaller bank gives smaller sets. Banks without the training-vocabulary names (WordNet [52] nouns, non-words) give larger sets.
Table S27. Reference-bank sweep (anchored CAT-Seg, ): set size, risk (%) and top-10 accuracy (%). ADE20K 150845 on a 400-image subset (sizes differ from Table S4). COCO-Stuff: 171 classes plus 695 Open Images (OI) names that are in no bank (500 images). Abstract nouns: WordNet nouns with only abstract senses; non-words: pronounceable non-words; –: not run.
| Reference bank | names | ADE20K, ViT-B size | risk | top-10 | COCO-Stuff ViT-B size | ViT-B risk | ViT-L size |
| native, recalibrated | – | – | – | – | 2.87 | 9.7 | 2.33 |
| none (candidate only) | 0 | 82.34 | 9.4 | 49.3 | 108.50 | 9.7 | 139.25 |
| COCO-20 | 20 | 33.68 | 9.5 | 88.4 | 32.74 | 9.6 | 28.05 |
| COCO-85 | 85 | 11.18 | 9.5 | 90.9 | 6.18 | 9.6 | 4.81 |
| COCO-171 | 171 | 7.92 | 9.5 | 90.8 | 3.15 | 9.7 | 2.86 |
| Pascal-171 | 171 | 8.85 | 9.5 | 91.0 | – | – | – |
| COCO + OI 255 | 255 | 7.65 | 9.5 | 90.9 | 2.99 | 9.7 | 2.71 |
| COCO + Pascal 255 | 255 | 7.39 | 9.6 | 90.9 | – | – | – |
| COCO + OI 342 | 342 | 7.07 | 9.6 | 91.1 | 3.04 | 9.7 | 2.60 |
| COCO + Pascal 342 | 342 | 6.91 | 9.6 | 91.2 | – | – | – |
| COCO + OI 683 | 683 | 6.44 | 9.5 | 91.5 | 3.32 | 9.7 | 2.81 |
| OI 683, no COCO names | 683 | 8.78 | 9.5 | 91.0 | 4.58 | 9.7 | 3.95 |
| COCO + OI 1,366 | 1,366 | 6.87 | 9.5 | 91.5 | 3.72 | 9.7 | 3.19 |
| COCO + OI 2,731 | 2,731 | 7.20 | 9.5 | 91.3 | 3.89 | 9.6 | 3.39 |
| COCO + OI 5,462 | 5,462 | 7.62 | 9.5 | 91.2 | 4.10 | 9.6 | 3.57 |
| abstract nouns 683 | 683 | – | – | – | 4.56 | 9.7 | – |
| non-words 683 | 683 | – | – | – | 5.06 | 9.7 | – |
| COCO + abstract 683 | 683 | – | – | – | 3.92 | 9.7 | – |
| COCO + non-words 683 | 683 | – | – | – | 4.01 | 9.7 | – |
Table S27.
Table S29. Redraws of the bank’s Open Images names (ADE20K 150845, the 400 images of Table S27, 200 half/half splits): risk (%) and set size at , and set size at . Every bank holds the COCO-171 names and 512 Open Images names.
| Open Images names | SED risk | SED size | SED size, 5% | CAT-Seg ViT-L risk | CAT-Seg ViT-L size | CAT-Seg ViT-L size, 5% |
| native, recalibrated | 9.51 | 5.65 | 17.44 | 9.41 | 3.80 | 7.86 |
| draw of the paper | 9.46 | 6.50 | 16.42 | 9.48 | 4.48 | 8.72 |
| redraw 1 | 9.49 | 6.23 | 15.98 | 9.53 | 4.34 | 8.65 |
| redraw 2 | 9.49 | 6.44 | 16.02 | 9.50 | 4.52 | 8.62 |
| redraw 3 | 9.44 | 6.42 | 16.75 | 9.46 | 4.62 | 8.94 |
| redraw 4 | 9.46 | 6.42 | 16.31 | 9.50 | 4.38 | 8.89 |
| redraw 5 | 9.46 | 6.50 | 16.27 | 9.51 | 4.44 | 9.03 |
| mean of the redraws | 9.47 | 6.40 | 16.27 | 9.50 | 4.46 | 8.82 |
| s.d. of the redraws | 0.02 | 0.11 | 0.31 | 0.03 | 0.11 | 0.18 |
Table S29.
Segmentation accuracy. On full images of ADE20K (A-150, A-847) and Pascal Context (PC-59, PC-459), anchored CAT-Seg and SED with the 683-name bank are at most 1.7 mIoU points below the native models. FC-CLIP loses 4.0 on A-150, 2.9 on A-847 and none on Pascal Context (Sec. H.2). FC-CLIP, whose class softmax runs on the bank, loses added-class accuracy. CAT-Seg and SED do not (Table S32).
Table S32. Pixel accuracy (%) on source and added classes of the A-847 and PC-459 runs in Table S30, from their confusion matrices. Source: the classes of A-150 or PC-59; added: the other classes (“unknown” belongs to neither). Source and added: pixel accuracy over the pixels of each group; class mean and mIoU: over the added classes with ground-truth pixels. The arg-max is over the whole benchmark vocabulary.
| Model | scoring | A-847 (150 source, 697 added) | PC-459 (59 source, 399 added) | ||||||
| source | added | class mean | mIoU | source | added | class mean | mIoU | ||
| CAT-Seg ViT-B | native (released) | 64.8 | 25.7 | 19.2 | 8.9 | 73.7 | 35.4 | 29.0 | 12.6 |
| anchored, COCO-171 bank | 61.4 | 27.3 | 21.1 | 8.4 | 71.7 | 37.7 | 33.8 | 12.2 | |
| anchored, 683-name bank | 63.4 | 25.9 | 18.7 | 8.7 | 73.1 | 36.4 | 29.3 | 12.4 | |
| CAT-Seg ViT-L | native (released) | 66.6 | 32.9 | 25.8 | 12.7 | 76.3 | 40.2 | 37.3 | 17.2 |
| anchored, COCO-171 bank | 62.8 | 35.2 | 29.7 | 12.3 | 74.5 | 42.2 | 42.7 | 16.7 | |
| anchored, 683-name bank | 65.3 | 31.6 | 24.9 | 12.0 | 75.6 | 40.4 | 37.5 | 16.7 | |
| CAT-Seg + DeCLIP EVA-L | native (released) | 68.6 | 35.6 | 27.9 | 14.2 | 76.3 | 43.5 | 43.1 | 19.6 |
| anchored, COCO-171 bank | 65.2 | 37.3 | 30.2 | 13.2 | 74.5 | 45.3 | 47.5 | 19.0 | |
| anchored, 683-name bank | 67.7 | 34.1 | 26.7 | 13.2 | 75.6 | 44.5 | 43.1 | 19.0 | |
| SED ConvNeXt-L | native (released) | 64.0 | 31.4 | 24.8 | 10.7 | 74.3 | 37.0 | 33.8 | 15.9 |
| anchored, COCO-171 bank | 60.9 | 34.9 | 28.7 | 10.8 | 73.2 | 38.3 | 38.9 | 15.8 | |
| anchored, 683-name bank | 62.9 | 32.5 | 26.0 | 10.8 | 74.0 | 36.9 | 34.0 | 15.5 | |
| FC-CLIP ConvNeXt-L | native (released) | 52.7 | 37.5 | 29.0 | 10.3 | 65.2 | 35.5 | 36.7 | 10.1 |
| anchored, 133-name bank | 66.4 | 11.6 | 8.3 | 4.0 | 73.6 | 26.6 | 18.9 | 7.5 | |
| anchored, 701-name bank | 61.7 | 23.2 | 17.0 | 7.7 | 69.8 | 31.4 | 31.5 | 9.8 |
Table S32.
Confirmation on reserved images
We scored 2,000 ADE20K training images not used in development, once, with CAT-Seg ViT-L, fixing the 683-name bank and the 845-name split in advance. The reused native threshold has 13.8% risk and the anchored one 8.4%, with 0.85 more labels per pixel than native recalibration. The added-label risk with random source labels is 9.3% (Sec. J).
Conclusion
A risk threshold calibrated for a released open-vocabulary segmenter does not survive adding names, and the failure shows little in its prediction sets. Scoring each label against a fixed reference bank makes its score independent of the other names. One calibration then keeps the source-label guarantee as names are added, including names chosen per image, without retraining. Across our benchmarks, the cost at is at most one extra label per pixel for CAT-Seg and SED. Anchored CAT-Seg ViT-L takes 3.7 the native inference time on an RTX 5070 Ti. On fixed images, calibrating over labels gives an added-label guarantee when annotated names are sampled at random. With about 20 labelled images per name, a threshold per name, capped at the image-level threshold, bounds its miss rate after names are added, with sets 1.8–3.6 larger (Sec. K.2).
The source-label guarantee assumes exchangeable calibration and test images. It holds on average over images, not for each scene or each per-image vocabulary [27]. The conversion covers linear attention across labels, softmaxes over the vocabulary and Grounding DINO’s text–image fusion. Softmax attention across labels is untested. Native recalibration is valid when the vocabulary is fixed in advance and labelled images exist. Anchoring is for vocabularies that arrive during use, such as the names a tagger supplies for each image [29, 60, 62].
LLM Usage Statement
The research question and the experimental design are the authors’. An AI coding assistant, Claude Code (Anthropic), helped write and debug the experiment and figure code, which the authors ran after review. It also helped search for and summarise related work and edited the text. A multimodal language model, Claude Opus 5.5, is also evaluated in the experiments (Secs. F and F.1). The authors reviewed all AI-assisted work, and every number in the text is generated from stored results by the released code. We take responsibility for the final content.
Supplementary Material
This supplement provides proofs, implementation details and complete experimental results. Sections A to D complete the theory of § 3 and § 5.3. Section E completes the experimental setup of § 6. Section F extends § 6.1 and § 6.2, and Secs. G to J extend § 6.3 to § 6.6; Sec. K contains the further analyses summarised at the end of § 6, and § 20 describes deployments in which the vocabulary grows.
Sampled pixels and the guarantee
This section shows that calibrating on a sample of pixels bounds the full-image risk, as stated in § 3.1. The stored scores cover pixels per image, drawn uniformly without replacement. The seed of each image is its index, which does not depend on the model or the labels, and we model it as independent of the image content. Let be the full-image set of annotated source-label pixels, as in (1), the sample and . Images with no source-labelled pixel (), for which (1) is undefined, are excluded. The sampled loss is
Unbiasedness. Given and , the set is uniform among the -subsets of , so for every fixed .
Exchangeability. depends only on the image size and the seed, which we model as independent of the image content. The pairs are therefore exchangeable whenever the images are. They remain exchangeable after images with are excluded, since the exclusion is a fixed rule applied to each pair. is bounded in and non-decreasing in , so CRC on the calibration pairs gives .
Full-image loss. is a function of the calibration pairs, which are independent of given . Conditioning on the calibration pairs, and and applying unbiasedness gives . The bound therefore holds for the full-image miss rate, although calibration reads only the sampled pixels. The expectation is over images with , i.e. the image distribution reweighted by . With pixels no development image has ; one of the reserved images does and is excluded, leaving . Sampling adds variance to the loss of each image, which the marginal bound absorbs. The argument uses linearity in the pixels; the stricter losses of Sec. K.4 are not linear, so for them the guarantee holds for the losses as evaluated on the sample.
Proof of the added-label guarantee
This section proves Proposition 3. The calibration images are fixed, and the only randomness is the draw of the labels. For a label present in these images, let be the pixels annotated with and
Exchangeable losses. Because is candidate-independent, is the same function of whether is requested in or in . The loss curves , , are therefore one fixed map applied to each coordinate of an exchangeable sequence, and they are exchangeable. This is the only step that uses candidate independence. For a candidate-dependent score, the source losses are computed with and the added label’s with , which are two different maps, so exchangeable names do not give exchangeable losses. Each takes values in , is non-decreasing in , and is zero at the lowest grid point, where every label is included.
Risk bound. Let be the largest grid threshold with
it exists because every loss is zero at the lowest grid point. Since , satisfies this inequality, so ; this also holds when is the all-label endpoint. Since is non-decreasing, . The threshold is a symmetric function of , so exchangeability gives for every , and
Random source vocabularies. Let be the names that will be requested and are present in the calibration images, let be a uniformly random -subset of , and let be any added label drawn uniformly from . Listing in uniformly random order makes a uniformly random ordered draw without replacement from , which is exchangeable. The order does not matter, since depends on only as a set. A uniformly chosen added label is such a draw, so by linearity the bound also holds for the risk averaged over all added labels, which is the quantity in Table 2.
Scope of the assumption. The argument uses exchangeability of the loss curves , which is weaker than exchangeability of the names. Exchangeability is also not necessary for the bound. By Proposition 4, any added-label distribution with , i.e. one whose labels are no harder at the calibrated threshold, satisfies it as well. Without a restriction on added-label losses, uniform validity requires the all-label threshold with probability at least (Proposition S2).
Added labels on new images
Proposition S1 (Added labels on new images). In Proposition 3, let each label come with images annotated with it, such that these units are exchangeable and, conditional on the labels, the joint distribution is invariant to permutations of images within each unit. Replace by , the mean over these images of the fraction of ’s pixels scored below . Then an image of the added label, drawn as its images are, satisfies , where is the fraction of ’s pixels in scored below .
This section proves Proposition S1. For a label and an image annotated with it, let be the fraction of ’s pixels in whose score for is below ; it is non-decreasing in and, for a candidate-independent score, does not depend on the vocabulary. Label comes with images , and . The units , , are exchangeable, and their joint distribution is invariant to permuting the images within while leaving the other units fixed; the test image is , and the other images of enter the argument only.
Let be the largest grid threshold with ; it exists because every loss is zero at the lowest grid point. Since , every that is feasible for , i.e. , is feasible for , so and . The threshold is a symmetric function of the units and, within , of the images. Exchangeability of the images gives , and exchangeability of the units gives . Hence .
The number of images may differ between labels if it is part of the unit (e.g. drawn independently for each label). Sharing images does not itself preclude exchangeability. However, a random split of benchmark images does not
implement the per-label sampling design assumed here. Table S1 therefore repeats the label-level analysis of Table S19 with the image-averaged loss, and reports on held-out images, where it estimates the expected miss rate of an added label on a new image. The trials are those of Table S19; with the pooled loss the replay reproduces that table. On held-out images the added-label risk is 7.3–9.6% at and 2.4–4.5% at , within 0.15 points of its value on the calibration images, for every model and benchmark. As for Proposition 3, the bound holds in expectation over the label draw, and 23–41% of single draws exceed .
| Model | benchmark | cal. | held-out | draws | cal. | held-out | draws |
| CAT-Seg ViT-B | ADE20K | 9.5 | 9.6 | 40 | 4.4 | 4.4 | 34 |
| CAT-Seg ViT-B | Pascal Context | 7.3 | 7.3 | 24 | 2.4 | 2.4 | 10 |
| CAT-Seg ViT-B | COCO-Stuff | 9.0 | 9.1 | 35 | 4.0 | 4.1 | 27 |
| CAT-Seg ViT-L | ADE20K | 9.4 | 9.5 | 39 | 4.5 | 4.5 | 35 |
| CAT-Seg ViT-L | Pascal Context | 7.5 | 7.4 | 23 | 2.5 | 2.5 | 9 |
| CAT-Seg ViT-L | COCO-Stuff | 8.9 | 8.9 | 34 | 3.9 | 4.0 | 27 |
| CAT-Seg + DeCLIP | ADE20K | 9.5 | 9.4 | 40 | 4.4 | 4.4 | 31 |
| CAT-Seg + DeCLIP | Pascal Context | 7.4 | 7.4 | 24 | 2.6 | 2.6 | 13 |
| CAT-Seg + DeCLIP | COCO-Stuff | 9.0 | 9.0 | 32 | 4.0 | 4.1 | 26 |
| SED | ADE20K | 9.5 | 9.5 | 40 | 4.5 | 4.5 | 37 |
| SED | Pascal Context | 7.5 | 7.5 | 25 | 2.6 | 2.6 | 12 |
| SED | COCO-Stuff | 9.0 | 9.0 | 35 | 4.0 | 4.0 | 27 |
| FC-CLIP | ADE20K | 9.5 | 9.5 | 41 | 4.5 | 4.5 | 35 |
| FC-CLIP | Pascal Context | 7.4 | 7.5 | 24 | 2.4 | 2.4 | 11 |
| FC-CLIP | COCO-Stuff | 9.0 | 9.0 | 31 | 4.0 | 3.9 | 25 |
Table S1. Added labels on new images (%): anchored model, label-level threshold with the image-averaged label loss of Proposition S1, random source vocabulary, 500 trials. cal. and held-out: mean added-label risk on the calibration and held-out images; draws : percentage of draws whose held-out risk exceeds .
| Model | benchmark | risk cal. | risk held-out | draws cal. | draws held-out | risk cal. | risk held-out | draws cal. | draws held-out |
| CAT-Seg ViT-B | ADE20K | 9.5 | 9.6 | 39 | 39 | 4.5 | 4.5 | 34 | 35 |
| CAT-Seg ViT-B | Pascal Context | 7.5 | 7.5 | 25 | 26 | 2.6 | 2.6 | 14 | 15 |
| CAT-Seg ViT-B | COCO-Stuff | 9.1 | 9.2 | 36 | 37 | 4.1 | 4.1 | 29 | 28 |
| CAT-Seg ViT-L | ADE20K | 9.5 | 9.5 | 40 | 40 | 4.5 | 4.5 | 36 | 37 |
| CAT-Seg ViT-L | Pascal Context | 7.6 | 7.5 | 23 | 25 | 2.8 | 2.7 | 16 | 16 |
| CAT-Seg ViT-L | COCO-Stuff | 8.9 | 9.0 | 33 | 34 | 4.0 | 4.1 | 27 | 28 |
| CAT-Seg + DeCLIP | ADE20K | 9.5 | 9.4 | 39 | 38 | 4.5 | 4.5 | 34 | 34 |
| CAT-Seg + DeCLIP | Pascal Context | 7.5 | 7.5 | 25 | 25 | 2.8 | 2.8 | 19 | 19 |
| CAT-Seg + DeCLIP | COCO-Stuff | 9.0 | 9.1 | 35 | 33 | 4.1 | 4.1 | 27 | 29 |
| SED | ADE20K | 9.5 | 9.5 | 42 | 42 | 4.6 | 4.6 | 36 | 37 |
| SED | Pascal Context | 7.5 | 7.5 | 25 | 27 | 2.8 | 2.8 | 17 | 19 |
| SED | COCO-Stuff | 9.0 | 9.1 | 34 | 36 | 4.0 | 4.1 | 27 | 29 |
| FC-CLIP | ADE20K | 9.6 | 9.5 | 40 | 40 | 4.5 | 4.5 | 36 | 36 |
| FC-CLIP | Pascal Context | 7.5 | 7.5 | 23 | 22 | 2.5 | 2.6 | 12 | 13 |
| FC-CLIP | COCO-Stuff | 9.0 | 9.0 | 32 | 33 | 4.0 | 4.0 | 26 | 28 |
Table S19. Added-label risk on calibration and held-out images (%), anchored model, label-level threshold, random source vocabulary, 500 trials: mean risk, and the percentage of draws whose risk exceeds . cal.: the calibration images, on which Proposition 3 holds; held-out: the other half of the images.
Worst-case added-label losses
Proposition S2 (No assumption-free guarantee). Fix a threshold rule that uses no annotations of the added label, with on a grid of minimum . Uniformly over all added-label loss functions that are non-decreasing and satisfy , including loss functions chosen after seeing , the bound holds if and only if .
The admissible loss , fixed in advance, proves necessity. Every admissible loss, however it is chosen, is bounded by this indicator at every , which proves sufficiency. An adversary who selects the added label after seeing the threshold is therefore no stronger than one who selects it in advance. This result applies to any threshold rule in the stated class, including native recalibration and anchored scoring. Randomly annotating added names (Sec. G.2) supplies an exchangeable sample for the remaining names, but does not protect against adversarial selection from them. The result concerns worst-case loss functions; a particular model need not contain a label attaining that worst case.
Experimental setup
This section describes the parts of the setup of § 6 that are specific to this paper: the COCO-Stuff benchmark and the fixed choices that make the experiments reproducible.
COCO-Stuff grown with LVIS
The third benchmark starts from the training vocabulary of CAT-Seg and SED. All choices below were fixed before any score was computed.
Source vocabulary: the 171 COCO-Stuff classes [13], named as in CAT-Seg’s release.
Added labels: the 1,112 LVIS v1 categories [30] that are not COCO-Stuff concepts, for 1,283 labels in total. An LVIS category counts as a COCO-Stuff concept when its synset is the one LVIS assigns to a COCO thing class, or when a synonym without a parenthetical qualifier, with underscores read as spaces, equals a COCO-Stuff name after its suffix -other, -stuff or -merged is dropped.
Images: the first 2,000 COCO val2017 images that LVIS annotates. None of the five models was trained on them.
Added-label annotation: a sampled pixel takes an added label when the smallest LVIS instance covering it belongs to an added category and its COCO-Stuff label is a stuff class or none. COCO thing masks are never overridden, since LVIS also annotates parts and worn items.
Reproducibility
All thresholds use a score grid of step on with the all-label endpoint; the CRC rule selects the largest grid threshold whose corrected calibration loss is at most . Random splits use a fixed seed. The reserved images are listed in a manifest fixed before any experiment, and label names, nested addition orders and prompt templates are fixed in hashed manifests. Scores are stored in float16, clamped to . They are logits, except for FC-CLIP, whose stored score is the logarithm of its mask-weighted class score. Every number in the paper and this supplement is generated from the stored results by a script. Code, manifests and results are available at https://github.com/munim110/open-vocab-crc. The stored scores are too large to host; the code regenerates them from the datasets and the released checkpoints.
Calibration reuse: complete results
This section gives the complete results behind Figure 4. Table S4 lists its values, and Table S5 repeats the comparison at . Table S6 to Table S8 give every main configuration on each benchmark, including the COCO-171 bank and SED’s fast mode, with the fraction of splits on which each threshold exceeds . Table S9 and Figure S1 give the set size at each step of the nested ADE20K growth. The section ends with the distribution of the set size over pixels (Table S10), native references (Table S2) and a general multimodal model.
Figure S1. Set size against the number of added labels on ADE20K for every configuration.
| Model | native | anch. | risk | risk | size | size | size | outside | ||
| CAT-Seg ViT-L | 0.10 | 1.91 | 2.25 | 15.2 | 6.00 | 9.9 | 3.48 | 3.47 | 4.27 | 0.34 |
| 0.05 | 3.28 | 3.66 | 8.4 | 2.50 | 4.8 | 7.95 | 7.81 | 8.63 | 0.01 | |
| SED | 0.10 | 2.22 | 2.76 | 7.1 | 15.25 | 10.0 | 5.20 | 5.27 | 6.04 | 0.20 |
| 0.05 | 4.70 | 5.30 | 3.8 | 6.50 | 4.8 | 16.41 | 16.14 | 15.58 | 0.00 |
Table S2. Native references (ADE20K , 500 splits). Size at 150: labels per pixel when calibrated and deployed at the source names. Reused: risk (%) at 845 names of the native threshold calibrated at 150. Reused at : the native model calibrated at 150 names with the level (%) that meets at 845 names on the evaluation images (an oracle), with its risk (%) and set size at 845 names. Recal., anchored: set size at 845 names, as in Table S6. Top-1 outside: percentage of source-labelled pixels whose native top-1 label at 845 names is not in the anchored set.
| Model | ADE20K 150845 risk (%) reused | ADE20K 150845 risk (%) ours | ADE20K 150845 set size recal. | ADE20K 150845 set size ours | Pascal Context 59458 risk (%) reused | Pascal Context 59458 risk (%) ours | Pascal Context 59458 set size recal. | Pascal Context 59458 set size ours | COCO-Stuff 1711,283 risk (%) reused | COCO-Stuff 1711,283 risk (%) ours | COCO-Stuff 1711,283 set size recal. | COCO-Stuff 1711,283 set size ours |
| CAT-Seg ViT-B | 15.4 | 9.9 | 4.99 | 5.98 | 15.2 | 9.8 | 2.52 | 2.82 | 12.9 | 9.9 | 3.11 | 3.84 |
| CAT-Seg ViT-L | 15.2 | 9.9 | 3.47 | 4.27 | 15.0 | 9.8 | 2.07 | 2.34 | 12.8 | 9.8 | 2.66 | 3.22 |
| CAT-Seg + DeCLIP | 14.7 | 9.9 | 3.39 | 4.38 | 14.7 | 9.8 | 2.09 | 2.42 | 12.9 | 9.9 | 2.64 | 3.19 |
| SED | 7.1 | 9.9 | 5.28 | 6.05 | 4.6 | 9.8 | 2.68 | 2.80 | 3.6 | 9.9 | 3.74 | 3.55 |
| FC-CLIP | 13.8 | 9.9 | 8.18 | 11.18 | 16.2 | 9.9 | 5.30 | 5.80 | 17.1 | 9.9 | 26.50 | 26.82 |
Table S4. Risk and set size after vocabulary growth (values of Figure 4; , mean over 500 splits). Risk: native threshold reused from the source vocabulary, anchored calibrated there (ours). Set size (labels per pixel at the grown vocabulary): native recalibrated there, anchored.
| Model | ADE20K 150845 risk (%) reused | ADE20K 150845 risk (%) ours | ADE20K 150845 set size recal. | ADE20K 150845 set size ours | Pascal Context 59458 risk (%) reused | Pascal Context 59458 risk (%) ours | Pascal Context 59458 set size recal. | Pascal Context 59458 set size ours | COCO-Stuff 1711,283 risk (%) reused | COCO-Stuff 1711,283 risk (%) ours | COCO-Stuff 1711,283 set size recal. | COCO-Stuff 1711,283 set size ours |
| CAT-Seg ViT-B | 8.1 | 4.9 | 13.29 | 14.25 | 8.4 | 4.9 | 4.66 | 4.95 | 6.9 | 4.9 | 5.96 | 7.12 |
| CAT-Seg ViT-L | 8.4 | 4.9 | 7.81 | 8.63 | 8.5 | 4.9 | 3.38 | 3.66 | 6.9 | 4.9 | 4.89 | 5.75 |
| CAT-Seg + DeCLIP | 7.9 | 4.9 | 7.64 | 8.74 | 8.4 | 4.9 | 3.58 | 3.85 | 6.8 | 4.9 | 4.84 | 5.67 |
| SED | 3.8 | 4.9 | 16.19 | 15.62 | 2.1 | 4.9 | 4.63 | 4.77 | 1.6 | 4.9 | 6.47 | 6.17 |
| FC-CLIP | 6.8 | 4.9 | 25.65 | 31.19 | 8.6 | 4.9 | 12.77 | 13.99 | 9.8 | 4.9 | 82.70 | 77.21 |
Table S5. Risk and set size after vocabulary growth at (columns as in Table S4).
| Model | bank | native, reused risk | native, reused splits | native, recalibrated risk | native, recalibrated splits | native, recalibrated size | anchored (ours) risk | anchored (ours) splits | anchored (ours) size | overhead |
| CAT-Seg ViT-B | COCO-171 bank | 15.4 | 100% | 9.9 | 41% | 4.99 | 9.9 | 39% | 3.107.32 | +2.32 |
| CAT-Seg ViT-B | 683-name bank | 15.4 | 100% | 9.9 | 41% | 4.99 | 9.9 | 41% | 2.765.98 | +0.98 |
| CAT-Seg ViT-L | COCO-171 bank | 15.2 | 100% | 9.9 | 43% | 3.47 | 9.9 | 40% | 2.485.08 | +1.61 |
| CAT-Seg ViT-L | 683-name bank | 15.2 | 100% | 9.9 | 43% | 3.47 | 9.9 | 42% | 2.254.27 | +0.79 |
| CAT-Seg + DeCLIP EVA-L | 683-name bank | 14.7 | 100% | 9.9 | 41% | 3.39 | 9.9 | 42% | 2.184.38 | +0.99 |
| SED ConvNeXt-L | COCO-171 bank | 7.1 | 0% | 9.9 | 43% | 5.28 | 9.9 | 42% | 2.946.96 | +1.69 |
| SED ConvNeXt-L | fast mode (top-8 rejection) | 12.8 | 100% | 9.9 | 41% | 5.20 | 9.9 | 42% | 2.946.96 | +1.76 |
| SED ConvNeXt-L | 683-name bank | 7.1 | 0% | 9.9 | 43% | 5.28 | 9.9 | 43% | 2.766.05 | +0.77 |
| FC-CLIP ConvNeXt-L | 133-name bank | 13.8 | 100% | 9.9 | 41% | 8.18 | 9.9 | 41% | 4.1113.56 | +5.37 |
| FC-CLIP ConvNeXt-L | 701-name bank | 13.8 | 100% | 9.9 | 41% | 8.18 | 9.9 | 38% | 3.8011.18 | +2.99 |
Table S6. ADE20K 150845, every main configuration (, 500 splits). Risk in %; set size in labels per pixel, anchored at the source grown vocabulary; overhead = anchored minus recalibrated native size, computed before rounding. “Splits ”: fraction of splits on which the risk of that arm exceeds the target. In the SED fast-mode row only the native arms use fast mode; the anchored arm is SED in its default mode with the COCO-171 bank, identical to the row above.
| Model | bank | native, reused risk | native, reused splits > | native, recalibrated risk | native, recalibrated splits > | native, recalibrated size | anchored (ours) risk | anchored (ours) splits > | anchored (ours) size | anchored (ours) overhead |
| CAT-Seg ViT-B | 683-name bank, COCO 1711,283 | 12.9 | 100% | 9.9 | 38% | 3.11 | 9.9 | 41% | 2.333.84 | +0.73 |
| CAT-Seg ViT-L | 683-name bank, COCO 1711,283 | 12.8 | 100% | 9.8 | 40% | 2.66 | 9.8 | 40% | 2.063.22 | +0.56 |
| CAT-Seg + DeCLIP EVA-L | 683-name bank, COCO 1711,283 | 12.9 | 100% | 9.9 | 41% | 2.64 | 9.9 | 42% | 1.943.19 | +0.55 |
| SED ConvNeXt-L | 683-name bank, COCO 1711,283 | 3.6 | 0% | 9.9 | 43% | 3.74 | 9.9 | 43% | 2.193.55 | −0.20 |
| FC-CLIP ConvNeXt-L | 701-name bank, COCO 1711,283 | 17.1 | 100% | 9.9 | 44% | 26.50 | 9.9 | 43% | 5.8626.82 | +0.32 |
Table S8. COCO-Stuff 1711,283, every configuration (columns as in Table S6).
| Configuration | +0 | +50 | +150 | +500 | +695 |
| CAT-Seg ViT-B, COCO-171 bank | 3.10 | 3.65 | 4.44 | 6.78 | 7.32 |
| CAT-Seg ViT-B, 683-name bank | 2.76 | 3.21 | 3.82 | 5.62 | 5.98 |
| CAT-Seg ViT-L, COCO-171 bank | 2.48 | 2.84 | 3.36 | 4.76 | 5.08 |
| CAT-Seg ViT-L, 683-name bank | 2.25 | 2.55 | 2.97 | 4.06 | 4.27 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank | 2.18 | 2.53 | 2.94 | 4.19 | 4.38 |
| SED ConvNeXt-L, COCO-171 bank | 2.94 | 3.50 | 4.35 | 6.45 | 6.96 |
| SED ConvNeXt-L, 683-name bank | 2.76 | 3.25 | 3.94 | 5.66 | 6.05 |
| FC-CLIP ConvNeXt-L, 133-name bank | 4.11 | 5.36 | 7.13 | 11.96 | 13.56 |
| FC-CLIP ConvNeXt-L, 701-name bank | 3.80 | 4.84 | 6.21 | 10.01 | 11.18 |
Table S9. Anchored set size at each nested addition on ADE20K (labels per pixel, ).
Table S10. Distribution of the per-pixel set size (, 500 splits as in Table S4): percentage of source-labelled test pixels whose prediction set is empty, has one label, at most three, at most ten or more than ten; median of the split-averaged distribution; mean as in Table S4. native at source: calibrated and deployed at the source names.
| Configuration | threshold | names | pixels with (%) = 0 | = 1 | labels per pixel median | mean | |||
| ADE20K, CAT-Seg ViT-L | native at source | 150 | 0.2 | 46.6 | 90.5 | 100.0 | 0.0 | 2 | 1.91 |
| native, reused | 845 | 0.3 | 38.0 | 85.5 | 100.0 | 0.0 | 2 | 2.16 | |
| native, recalibrated | 845 | 0.0 | 22.1 | 58.0 | 99.4 | 0.6 | 3 | 3.47 | |
| anchored | 845 | 0.3 | 23.5 | 51.8 | 94.5 | 5.5 | 3 | 4.27 | |
| ADE20K, SED | native at source | 150 | 0.1 | 38.9 | 83.3 | 100.0 | 0.0 | 2 | 2.22 |
| native, reused | 845 | 0.0 | 5.9 | 15.2 | 67.0 | 33.0 | 8 | 8.80 | |
| native, recalibrated | 845 | 0.0 | 10.5 | 29.8 | 95.2 | 4.8 | 5 | 5.28 | |
| anchored | 845 | 0.1 | 19.4 | 39.7 | 82.9 | 17.1 | 5 | 6.05 | |
| Pascal Context, CAT-Seg ViT-L | native at source | 59 | 0.2 | 70.7 | 99.2 | 100.0 | 0.0 | 1 | 1.37 |
| native, reused | 458 | 0.5 | 63.6 | 97.8 | 100.0 | 0.0 | 1 | 1.49 | |
| native, recalibrated | 458 | 0.0 | 47.4 | 85.1 | 100.0 | 0.0 | 2 | 2.07 | |
| anchored | 458 | 0.4 | 44.6 | 79.3 | 99.8 | 0.2 | 2 | 2.34 | |
| Pascal Context, SED | native at source | 59 | 0.1 | 62.7 | 97.2 | 100.0 | 0.0 | 1 | 1.53 |
| native, reused | 458 | 0.0 | 7.5 | 36.0 | 95.8 | 4.2 | 4 | 4.80 | |
| native, recalibrated | 458 | 0.2 | 17.6 | 77.2 | 100.0 | 0.0 | 3 | 2.68 | |
| anchored | 458 | 0.8 | 27.0 | 72.2 | 99.6 | 0.4 | 2 | 2.80 | |
| COCO-Stuff, SED | native at source | 171 | 0.1 | 41.2 | 89.6 | 100.0 | 0.0 | 2 | 2.01 |
| native, reused | 1,283 | 0.0 | 1.1 | 14.8 | 72.7 | 27.3 | 8 | 8.21 | |
| native, recalibrated | 1,283 | 0.3 | 7.2 | 49.4 | 100.0 | 0.0 | 4 | 3.74 | |
| anchored | 1,283 | 0.2 | 17.9 | 61.5 | 98.2 | 1.8 | 3 | 3.55 |
Table S10.
The conclusions of § 6.1 hold at . The reused native threshold exceeds the target for CAT-Seg and FC-CLIP on every benchmark and stays below that target for SED. The anchored threshold meets the target in every case, with an overhead of at most 1.16 labels per pixel for CAT-Seg and SED. The guarantee holds in expectation over the calibration draw: at single splits exceed the target in 38–43% of cases for the anchored threshold and in 38–46% for native recalibration. With the smaller COCO-171 bank the anchored threshold remains valid, but at the overhead roughly doubles for CAT-Seg and SED (1.61 instead of 0.79 labels per pixel for ViT-L on ADE20K). Models whose scores are candidate-independent by construction are valid but far less efficient (Table S3).
Native references. Table S2 adds two native references on ADE20K, from the stored scores and splits of Table S6. Calibrated and deployed at the 150 source names, the native model gives smaller sets than the anchored model (1.91 against 2.25 labels per pixel for CAT-Seg ViT-L at , and 2.22 against 2.76 for SED), so part of the overhead of anchoring is present before the vocabulary grows. The reused native threshold can also be corrected by calibrating at the source vocabulary with a level in place of . We take the largest , on a grid of step 0.25%, whose reused threshold has mean risk at most at 845 names on the evaluation images. The corrected sets are as small as those of native recalibration (3.48 against 3.47 for ViT-L). Selecting requires labelled evaluation images at the grown vocabulary, and the direction of correction depends on the model: for ViT-L, whose reused threshold exceeds the target, and 15.25% for SED, whose reused threshold is over-conservative. This is therefore an oracle comparison: a level chosen without those labels has no corresponding guarantee.
Agreement with native predictions. The native top-1 label over the 845 names lies outside the anchored set at 0.34% of the source-labelled evaluation pixels for ViT-L and 0.20% for SED at , and at 0.01% and 0.00% at .
A general multimodal model. A general multimodal model can name image regions from a list without a segmentation head. We asked Claude Opus 5.5 (claude-opus-5-5, Anthropic API, November 2026) about 1,000 ADE20K images drawn at random. Each image carries 15 numbered markers, placed on stored sample pixels with a source label (Set-of-Mark prompting [87]). For each marker the model returns up to 10 names from the given list, each with a probability, and these probabilities are its scores; unlisted names score 0, since the API gives no token probabilities. Each image is asked twice, with the 150 source names and with all 845 names in the same relative order. The top name is correct at 67.5% of the markers with 150 names and at 62.8% with 845. The scores are candidate-dependent: adding the names changes the probability of the true label at 57% of the markers and the top name at 18%. The true label is missing from the answer at 11.6% of the markers with 150 names and at 17.1% with 845. Both exceed , so CRC, reused or recalibrated, falls to the all-label endpoint and includes all 845 labels. On the same markers and 500 random halves of the images, CAT-Seg ViT-L with the 683-name bank has 15.6% risk with the reused native threshold; the anchored threshold has 9.8% with 4.56 labels per marker, against 3.69 for native recalibration. The queries cost about $0.034 per image for both vocabularies. Under the API terms, inputs and outputs are kept for up to 30 days and are not used for training; the ADE20K terms permit non-commercial research use. The prompts and every response are released with the code (Sec. E.2).
Distribution of the set size. The set sizes in Table S4 are means over source-labelled pixels. Table S10 gives the distribution behind them on the same splits and thresholds for 5 of the configurations of Table S4; Table S11 repeats it at . Compared with the recalibrated native sets, the anchored sets have the same or a smaller median in every configuration. They also hold a single label at more pixels in every configuration but one (on ADE20K, 23.5% of pixels against 22.1% for CAT-Seg ViT-L and 19.4% against 10.5% for SED); the exception is CAT-Seg ViT-L on Pascal Context (44.6% against 47.4%). Their larger mean comes from a tail of pixels with more than ten labels: 5.5% against 0.6% for CAT-Seg ViT-L and 17.1% against 4.8% for SED on
Table S11. Distribution of the per-pixel set size at (columns as in Table S10).
| Configuration | threshold | names | pixels with (%) = 0 | = 1 | labels per pixel median | mean | |||
| ADE20K, CAT-Seg ViT-L | native at source | 150 | 0.0 | 23.9 | 62.4 | 99.4 | 0.6 | 3 | 3.28 |
| native, reused | 845 | 0.0 | 17.0 | 48.4 | 97.0 | 3.0 | 4 | 4.21 | |
| native, recalibrated | 845 | 0.0 | 6.5 | 23.5 | 74.2 | 25.8 | 7 | 7.81 | |
| anchored | 845 | 0.0 | 11.0 | 27.9 | 69.1 | 30.9 | 7 | 8.63 | |
| ADE20K, SED | native at source | 150 | 0.0 | 14.8 | 43.6 | 93.9 | 6.1 | 4 | 4.71 |
| native, reused | 845 | 0.0 | 0.2 | 1.9 | 15.7 | 84.3 | 22 | 24.84 | |
| native, recalibrated | 845 | 0.0 | 1.9 | 5.9 | 32.0 | 68.0 | 15 | 16.19 | |
| anchored | 845 | 0.0 | 7.2 | 16.5 | 44.4 | 55.6 | 12 | 15.62 | |
| Pascal Context, CAT-Seg ViT-L | native at source | 59 | 0.0 | 52.6 | 89.7 | 100.0 | 0.0 | 1 | 1.87 |
| native, reused | 458 | 0.0 | 43.3 | 79.8 | 100.0 | 0.0 | 2 | 2.30 | |
| native, recalibrated | 458 | 0.0 | 31.1 | 62.4 | 98.1 | 1.9 | 3 | 3.38 | |
| anchored | 458 | 0.1 | 30.2 | 61.1 | 96.0 | 4.0 | 3 | 3.66 | |
| Pascal Context, SED | native at source | 59 | 0.0 | 42.4 | 82.9 | 100.0 | 0.0 | 2 | 2.20 |
| native, reused | 458 | 0.0 | 16.3 | 36.0 | 72.4 | 27.6 | 7 | 8.45 | |
| native, recalibrated | 458 | 0.0 | 7.9 | 37.9 | 96.7 | 3.3 | 4 | 4.63 | |
| anchored | 458 | 0.3 | 12.2 | 44.1 | 93.3 | 6.7 | 4 | 4.77 | |
| COCO-Stuff, SED | native at source | 171 | 0.0 | 25.0 | 64.7 | 99.3 | 0.7 | 3 | 3.21 |
| native, reused | 1,283 | 0.0 | 0.2 | 5.4 | 38.3 | 61.7 | 13 | 14.67 | |
| native, recalibrated | 1,283 | 0.0 | 2.0 | 20.9 | 88.1 | 11.9 | 6 | 6.47 | |
| anchored | 1,283 | 0.0 | 7.1 | 34.3 | 84.8 | 15.2 | 5 | 6.17 |
Table S11.
ADE20K. The reused native SED threshold, which is over-conservative after growth, places 33.0% of the ADE20K pixels above ten labels. Empty sets occur at under 1% of pixels under every threshold. Figure S2 adds the anchored threshold to the image of Figure 2. Figure S3 shows the sets on 6 images drawn at random, with the same thresholds. Over these images, the reused native threshold misses 26.0% of the source-labelled pixels, recalibration 20.1% and the anchored threshold 22.1%, at 2.9, 4.6 and 4.8 labels per pixel. Two of the six images are hard for every threshold; the guarantee is an average over images (15.6, 9.8 and 9.7% over the 305 images of the pool).
Under the label-level threshold. Table S12 gives the same distribution for the thresholds of Table 2 and Table S18, on their trials and, as there, over the labelled held-out pixels, source or added. Here the sets are larger at most pixels, not only in a tail. For CAT-Seg ViT-L on ADE20K at the median pixel has 20 labels and 75.3% of pixels have more than ten, against 5 and 20.3% under the image-level threshold, which gives no guarantee for the added labels. On Pascal Context, where fewer names are added, the median is 6 against 2. Each label counts equally in the label-level loss, so rare added labels with low scores at their pixels
Table S12. Distribution of the per-pixel set size under the label-level threshold (random source vocabulary, the trials of Tabs. 2 and S18; for CAT-Seg ViT-L on ADE20K one of 10 draws, as in Table 2): percentage of labelled held-out pixels whose prediction set has one label, at most three, at most ten or more than ten; median of the trial-averaged distribution; mean as in those tables. anchored, image-level: the threshold of Table S4, which gives no guarantee for the added labels.
| Configuration | threshold | pixels with (%) = 1 | labels per pixel median | mean | |||
| ADE20K, CAT-Seg ViT-L | anchored, image-level | 15.3 | 35.8 | 79.7 | 20.3 | 5 | 6.79 |
| anchored, label-level | 1.1 | 4.8 | 24.7 | 75.3 | 20 | 28.85 | |
| native recal., label-level | 0.1 | 0.8 | 12.6 | 87.4 | 27 | 35.80 | |
| ADE20K, SED | anchored, image-level | 10.8 | 22.9 | 55.4 | 44.6 | 9 | 16.79 |
| anchored, label-level | 0.6 | 2.9 | 12.9 | 87.1 | 34 | 45.10 | |
| native recal., label-level | 0.0 | 0.2 | 3.5 | 96.5 | 44 | 51.38 | |
| Pascal Context, CAT-Seg ViT-L | anchored, image-level | 34.5 | 65.2 | 94.5 | 5.5 | 2 | 3.73 |
| anchored, label-level | 14.1 | 35.0 | 70.1 | 29.9 | 6 | 11.54 | |
| native recal., label-level | 9.8 | 25.4 | 59.5 | 40.5 | 8 | 14.92 | |
| Pascal Context, SED | anchored, image-level | 15.3 | 44.8 | 87.0 | 13.0 | 4 | 5.87 |
| anchored, label-level | 3.3 | 14.1 | 51.2 | 48.8 | 10 | 17.48 | |
| native recal., label-level | 1.3 | 7.4 | 42.0 | 58.0 | 13 | 20.15 | |
| COCO-Stuff, SED | anchored, image-level | 6.7 | 30.4 | 76.2 | 23.8 | 6 | 7.68 |
| anchored, label-level | 0.5 | 7.0 | 34.7 | 65.3 | 15 | 21.24 | |
| native recal., label-level | 0.1 | 2.7 | 23.5 | 76.5 | 19 | 23.91 | |
| ADE20K, CAT-Seg ViT-L | anchored, image-level | 5.2 | 16.2 | 47.9 | 52.1 | 11 | 14.98 |
| anchored, label-level | 0.0 | 0.3 | 4.7 | 95.3 | 53 | 74.52 | |
| native recal., label-level | 0.0 | 0.0 | 0.7 | 99.3 | 113 | 228.11 | |
| ADE20K, SED | anchored, image-level | 3.6 | 9.1 | 26.5 | 73.5 | 22 | 39.10 |
| anchored, label-level | 0.0 | 0.1 | 1.5 | 98.5 | 95 | 118.47 | |
| native recal., label-level | 0.0 | 0.0 | 0.0 | 100.0 | 121 | 139.80 | |
| Pascal Context, CAT-Seg ViT-L | anchored, image-level | 19.9 | 45.1 | 81.2 | 18.8 | 4 | 7.14 |
| anchored, label-level | 2.4 | 9.0 | 27.8 | 72.2 | 25 | 68.36 | |
| native recal., label-level | 1.3 | 5.0 | 18.1 | 81.9 | 40 | 92.77 | |
| Pascal Context, SED | anchored, image-level | 6.0 | 22.6 | 65.1 | 34.9 | 7 | 11.78 |
| anchored, label-level | 0.3 | 2.0 | 15.0 | 85.0 | 40 | 80.37 | |
| native recal., label-level | 0.1 | 0.7 | 8.9 | 91.1 | 48 | 86.88 | |
| COCO-Stuff, SED | anchored, image-level | 1.9 | 13.3 | 49.5 | 50.5 | 11 | 14.77 |
| anchored, label-level | 0.0 | 1.0 | 9.7 | 90.3 | 42 | 60.08 | |
| native recal., label-level | 0.0 | 0.2 | 4.4 | 95.6 | 52 | 66.58 |
Table S12.
pull the threshold down, and the lower threshold applies at every pixel. In each of the 5 configurations of Table S12 the anchored sets are smaller than those of native recalibration with the label-level threshold (ADE20K, CAT-Seg ViT-L: median 27, 87.4% above ten). The one exception in Table S18, FC-CLIP on COCO-Stuff, is not among these configurations. With the benchmark’s own source classes the sets are smaller (median 7, 32.9% above ten on ADE20K) because the rare labels are not among the calibrated ones, which is also why the guarantee fails there (Table 2).
Per-image vocabularies
Table S13 gives the per-image results of § 6.2 for CAT-Seg ViT-L. The worst case in these tables is over the five fixed rules, not over all vocabularies, and which rules a native threshold fails depends on the architecture (Table S14 and Table S15). These rules vary both the number of added names and their relation to the image. Anchored source-label scores remain identical across rules, while native scores can change. An external tagger such as RAM [90] would leave the anchored risk unchanged; only the anchored set sizes and the native results would change. Of the native thresholds in Table 1, the one reused from the source names exceeds the target under the worst rule for every model, as does the one calibrated on the source names plus 500 random added names (column random). Recalibrating at all grown names meets the target for CAT-Seg ViT-L but exceeds it for FC-CLIP (10.3%) and SED (18.8%). Recalibrating CAT-Seg ViT-L at a 17,001-name lexicon gives 4.1% risk at 8.92 labels per pixel, about twice the anchored set size. At the lexicon, the FC-CLIP threshold meets the target (5.6%) at 17.35 labels per pixel against 9.71 for the anchored threshold, and the SED threshold exceeds it further (25.4%). On Pascal Context the threshold recalibrated at all 458 names is at the target under the worst rule, with 51% of the splits above it; for FC-CLIP on ADE20K 67% of the splits lie above it. Table S14 repeats the experiment for SED in its default mode, on the same 2,000 ADE20K images and with the same rules; the added names of the tagger rule are those that SED scores highest. The anchored threshold again has 9.9% risk under every rule. The native threshold reused from 150 names stays below the target when random names are added (9.0% with 50, 7.3% with 500), as in Table S4: at a fixed threshold, SED misses fewer source pixels as random names are added. It exceeds the target when the added names are those annotated in the image (10.4%) or those SED scores highest (12.7%). For CAT-Seg, the native threshold recalibrated at the 845 grown names meets the target under every tested rule. For SED it exceeds the target under every rule (10.1–18.8%): it is calibrated where SED misses least, and each per-image vocabulary is a subset of the grown one. At a lexicon the threshold moves further in the same direction: under the worst rule the risk rises from 21.8% at 1,690 names to 25.4% at 17,001 names, with 1.38 labels per pixel (column recal. 17k). Recalibrating at a lexicon therefore relies on how the model’s scores move as names are added, which differs between architectures. The native thresholds that meet the target under every rule are those calibrated under the deployed rule, or under the worst case over all rules (Proposition 2), which gives 5.00 labels per pixel against 5.87 for the anchored threshold. Table S15 repeats the experiment for FC-CLIP, whose tagger rule adds the names FC-CLIP scores highest. The anchored threshold has 9.9% risk under every rule. As for CAT-Seg, the reused native threshold exceeds the target under every rule (10.3% with 50 random names, 14.1% in the worst case). The native threshold recalibrated at the 845 grown names lies between
Table S14. Per-image vocabularies, SED (ADE20K, ): source-label risk (%) / set size on source-labelled pixels, as in Table S13. matched: native threshold calibrated under the row’s own rule; for the last row, under the worst case over all rules.
| Added names | anchored once | anchored reused | native recal. 845 | native random | native recal. 17k | native matched |
| present in image | 9.9/2.9 | 10.4/2.3 | 15.5/1.6 | 15.1/1.6 | 21.7/1.2 | 9.9/2.4 |
| 50 random | 9.9/3.0 | 9.0/2.6 | 13.6/1.8 | 13.1/1.8 | 19.4/1.4 | 9.9/2.4 |
| 200 random | 9.9/3.7 | 8.0/4.0 | 11.4/2.6 | 11.1/2.7 | 15.9/1.8 | 9.9/3.0 |
| 500 random | 9.9/5.1 | 7.3/6.8 | 10.1/4.2 | 9.9/4.3 | 13.8/2.8 | 9.9/4.3 |
| tagger, top 50 | 9.9/5.9 | 12.7/3.4 | 18.5/2.1 | 18.2/2.1 | 25.0/1.4 | 9.9/4.8 |
| worst, per image | 9.9/5.0 | 12.8/3.4 | 18.8/2.0 | 18.4/2.1 | 25.4/1.4 | 9.9/5.0 |
Table S14.
| Added names | anchored once | reused | native recal. 845 | native random | native recal. 17k | native matched |
| present in image | 9.9/4.0 | 10.5/3.2 | 7.6/4.7 | 8.0/4.4 | 4.3/10.4 | 9.9/3.4 |
| 50 random | 9.9/4.3 | 10.3/3.4 | 7.3/5.1 | 7.8/4.8 | 4.1/11.9 | 9.9/3.6 |
| 200 random | 9.9/5.9 | 11.6/3.9 | 8.2/6.0 | 8.7/5.6 | 4.6/14.7 | 9.9/4.8 |
| 500 random | 9.9/9.0 | 13.0/4.7 | 9.4/7.4 | 9.9/6.9 | 5.1/19.3 | 9.9/6.9 |
| tagger, top 50 | 9.9/9.2 | 14.0/4.9 | 10.2/7.4 | 10.7/6.9 | 5.6/15.9 | 9.9/7.6 |
| worst, per image | 9.9/9.7 | 14.1/4.8 | 10.3/7.4 | 10.8/6.9 | 5.6/17.4 | 9.9/7.7 |
Table S15. Per-image vocabularies, FC-CLIP (ADE20K, ): source-label risk (%) / set size on source-labelled pixels, as in Table S14.
| Model | bank | native, reused risk | native, reused splits | native, recalibrated risk | native, recalibrated splits | native, recalibrated size | anchored (ours) risk | anchored (ours) splits | anchored (ours) size | overhead |
| CAT-Seg ViT-B | COCO-171 bank, Pascal 59458 | 15.2 | 100% | 9.9 | 46% | 2.52 | 9.8 | 42% | 1.783.35 | +0.83 |
| CAT-Seg ViT-B | 683-name bank, Pascal 59458 | 15.2 | 100% | 9.9 | 46% | 2.52 | 9.8 | 42% | 1.672.82 | +0.30 |
| CAT-Seg ViT-L | 683-name bank, Pascal 59458 | 15.0 | 100% | 9.9 | 43% | 2.07 | 9.8 | 40% | 1.512.34 | +0.27 |
| CAT-Seg + DeCLIP EVA-L | 683-name bank, Pascal 59458 | 14.7 | 100% | 9.9 | 39% | 2.09 | 9.8 | 40% | 1.512.42 | +0.33 |
| SED ConvNeXt-L | 683-name bank, Pascal 59458 | 4.6 | 0% | 9.8 | 43% | 2.68 | 9.8 | 41% | 1.672.80 | +0.13 |
| FC-CLIP ConvNeXt-L | 701-name bank, Pascal 59458 | 16.2 | 100% | 9.9 | 40% | 5.30 | 9.9 | 41% | 2.015.80 | +0.50 |
Table S7. Pascal Context 59458, every configuration (columns as in Table S6).
CAT-Seg and SED. It is below the target for random names (7.3–9.4%) and for the names present in the image (7.6%). It is slightly above the target for the tagger rule (10.2%, 62% of the splits above it) and in the worst case (10.3%, 67% of the splits). Recalibrated at a lexicon it meets the target under every rule: the worst-rule risk falls from 9.1% at 1,690 names to 5.6% at 17,001 names, while the set size grows from 8.71 to 17.35 labels per pixel, against 9.71 for the anchored threshold (column recal. 17k). FC-CLIP’s source-label scores mostly fall as names are added, so a threshold set at the lexicon admits more labels on every per-image vocabulary; SED’s rise, and the same recalibration fails. As for SED, the native threshold calibrated under the worst case over all rules gives smaller sets than the anchored one (7.74 against 9.71 labels per pixel), but it requires knowing the rules.
Pascal Context. Table S16 repeats the experiment for CAT-Seg ViT-L and SED on the 2,000 Pascal Context validation images, whose 59 source names grow to 458, under a protocol fixed before the per-image scores were computed. Each rule adds one of: the names annotated in the image (1.1 on average); 50, 100 or 300 of the 399 added names at random (300 is about the same share of the added names as 500 on ADE20K); the 50 added names the model scores highest; or the worst of these per image. The anchored threshold has 9.8% risk under every rule for both models. For CAT-Seg, the reused threshold exceeds the target under every rule (10.1–15.2%). The threshold recalibrated at all 458 names is below the target for random names and the names present in the image (6.0–8.9%) and at the target under the tagger rule and in the worst case (10.0%, 51% of the splits above it). For SED, the reused threshold stays below the target for random names (8.2% with 50, 5.0% with 300) and for the names present in the image (9.9%), and exceeds it under the tagger rule (11.8%) and in the worst case (12.3%). Recalibrating at 458 names exceeds the target under every rule (10.6–21.1%), as on ADE20K. The native threshold calibrated under the worst case over all rules gives 2.06 and 2.42 labels per pixel, against 2.33 and 2.80 for the anchored threshold. At , in the worst case, the anchored threshold has 4.9 and 4.8% risk, and recalibration at 458 names 5.1 and 12.7%.
| Added names | anchored once | native reused | native recal. 458 | native random | native matched |
| \mathit{CAT-Seg ViT-L} | |||||
| present in image | 9.8/1.5 | 10.1/1.4 | 6.0/1.7 | 6.6/1.7 | 9.8/1.4 |
| 50 random | 9.8/1.6 | 10.9/1.4 | 6.4/1.8 | 7.1/1.7 | 9.8/1.5 |
| 100 random | 9.8/1.7 | 11.5/1.4 | 7.0/1.8 | 7.7/1.7 | 9.8/1.5 |
| 300 random | 9.8/2.1 | 14.0/1.5 | 8.9/2.0 | 9.8/1.9 | 9.8/1.9 |
| tagger, top 50 | 9.8/2.3 | 15.2/1.5 | 10.0/2.0 | 10.9/1.9 | 9.8/2.1 |
| worst, per image | 9.8/2.3 | 15.2/1.5 | 10.0/2.0 | 10.9/1.9 | 9.8/2.1 |
| \mathit{SED} | |||||
| present in image | 9.8/1.7 | 9.9/1.5 | 18.9/1.1 | 17.6/1.1 | 9.8/1.6 |
| 50 random | 9.8/1.8 | 8.2/1.8 | 15.2/1.2 | 14.2/1.3 | 9.8/1.6 |
| 100 random | 9.8/2.0 | 7.3/2.1 | 13.7/1.4 | 12.8/1.4 | 9.8/1.7 |
| 300 random | 9.8/2.5 | 5.0/3.9 | 10.6/2.2 | 9.8/2.3 | 9.8/2.3 |
| tagger, top 50 | 9.8/2.8 | 11.8/2.1 | 19.4/1.3 | 18.4/1.3 | 9.8/2.4 |
| worst, per image | 9.8/2.8 | 12.3/1.9 | 21.1/1.1 | 19.8/1.2 | 9.8/2.4 |
Table S16. Per-image vocabularies on Pascal Context (59 → 458, ): source-label risk (%) / set size on source-labelled pixels, as in Table S14. random: calibrated on vocabularies of the source names plus 300 random added names.
Names from a language model. The rules above draw the added names from the 845 evaluated names. A tagger or language model in deployment is not limited to a list. Under a protocol fixed before any score was computed, we asked Claude Opus 5.5 to list what it sees in each of the 2,000 ADE20K images, without showing it any list of names. It gave 25.1 names per image. After removing the source names, their ADE20K synonyms and repeats, 16.1 added names remain per image, 5,797 distinct names in all; 77% of the added names are not among the ADE20K-847 names and 51% are not in the 17,001-name lexicon. Each image deploys the 150 source names followed by its own added names. Table S17 gives the source-label risk. The anchored threshold, calibrated once at the source names, has 9.9% risk for every model (Proposition 1). The native threshold reused from the source names has 20.5–21.5% risk, and the one recalibrated at all 845 names 12.6–30.3%; both exceed the target for every model in every split. For SED, recalibrating at a lexicon moves the risk further from the target (41.2% at 17,001 names), as under the fixed rules. For FC-CLIP, the threshold recalibrated at the 17,001-name lexicon meets the target with 10.99 labels per pixel, 1.4 times the anchored set size, although 51% of the added names are outside the lexicon, where no guarantee applies. Calibrating the native model under the language model’s names on the calibration images is valid by exchangeability, but needs the language model at calibration time and again whenever the model or the prompt changes; its sets are larger than the anchored ones for every model (4.79, 6.39 and 8.80 against 4.59, 5.69 and 8.00 labels per pixel). Names outside ADE20K-847 have no
| Model | anchored once | native reused | native recal. 845 | native recal. 1.7k | native recal. 17k | native matched |
| CAT-Seg ViT-L | 9.9/4.6 | 20.6/2.7 | 12.6/4.1 | – | – | 9.9/4.8 |
| SED | 9.9/5.7 | 20.5/3.4 | 30.3/2.3 | 35.4/1.9 | 41.2/1.5 | 9.9/6.4 |
| FC-CLIP | 9.9/8.0 | 21.5/4.4 | 15.6/5.9 | 13.6/6.6 | 7.8/11.0 | 9.9/8.8 |
| CAT-Seg ViT-L | 4.9/7.0 | 10.2/4.7 | 5.1/7.3 | – | – | 4.9/7.5 |
| SED | 4.9/9.8 | 8.9/7.0 | 12.7/5.2 | 15.5/4.4 | 18.9/3.7 | 4.9/10.9 |
| FC-CLIP | 4.9/15.1 | 10.4/8.5 | 7.0/12.2 | 6.2/13.6 | 4.1/22.0 | 4.9/18.0 |
Table S17. Names from a language model (ADE20K, 2,000 images; the 150 source names plus 16.1 names per image on average from Claude Opus 5.5, most outside ADE20K-847): source-label risk (%) / set size on source-labelled pixels, 500 random half/half splits. Native thresholds: reused from the source names, recalibrated at the 845 names or at a lexicon of 1,690 or 17,001 names (SED and FC-CLIP), or calibrated under the language model’s names on the calibration images (matched).
annotations, so only the source-label risk is evaluated; one language model and one prompt were used.
Added labels
This section extends § 6.3. Section G.1 repeats the label-level analysis for every model. Section G.2 analyses the benchmark’s own source classes, including calibration on randomly drawn added names. Section G.3 evaluates the added labels under the image-level thresholds of § 6.1, which carry no guarantee for them. Per-name thresholds (Sec. K.2) bound the risk of each added name with enough labelled images, without exchangeable labels.
Every model
Table S18 applies the label-level threshold of Proposition 3 to every model with a random source vocabulary, drawn afresh in each trial. The added-label risk is 7.5–9.6% at and 2.5–4.6% at . The label-level threshold on the native model at the grown vocabulary is also valid, and the anchored sets are smaller in every comparison except FC-CLIP on COCO-Stuff (Table S18). On the benchmark’s own split every model shows an excess of the kind analysed in Sec. G.2. Proposition 3 bounds the added-label risk on the calibration images. Table S19 compares it with the risk of the same added labels on the held-out half of the images, which a deployment would see: the two differ by at most 0.09 points for every model and benchmark. Since the bound holds in expectation over the label draw, a single draw can exceed : 23–42% of the draws do at (22–42% on held-out images) and 12–36% at .
The benchmark’s own source classes
Decomposition. On the calibration images, the added-label risk equals the mean loss of the source labels at the calibrated threshold plus , the difference in Proposition 4. At the source labels are at or below the target on every benchmark, and the gap accounts for the added-label excess (§ 6.3). Native recalibration at the grown vocabulary shows the same pattern, with an added-label risk of 21.2–26.5% across the benchmarks ( points on ADE20K). For anchored CAT-Seg ViT-L on ADE20K, the worst-case bound is 1.7 times the realised excess (22.0 against 12.8 points). For the native model with the threshold reused from the source vocabulary, also contains candidate dependence: it is 11.6 points on ADE20K even under a random source vocabulary, and 24.4 points on the benchmark’s own split, roughly the sum of the two effects.
Added names selected by difficulty. Table S20 measures the excess of Proposition 4 when the added names are selected by how hard they are. Each of 500 trials follows the exchangeable design of Sec. G.1: a random calibration half of the ADE20K images and 150 uniformly drawn calibration names. The remaining names present in both halves are ranked by their loss on the held-out half, at the label-level threshold calibrated on that half, and split into fifths; is uniform over one fifth. The ranking uses the held-out images and the drawn names, which also determine , so is not independent of as Proposition 4 assumes. The reported bounds do not need this independence. Let be uniform over all names present in the calibration images, source names included. For any , in every realisation. The CRC constraint bounds the mean loss of the source names by , and Proposition 3, applied to a uniformly chosen remaining name, bounds the expected mean loss of the remaining names by , so . The columns and therefore bound the expected risk, and so does below. All risks are measured on the calibration images.
Uniform over all remaining names, the added-label risk for CAT-Seg ViT-L at is 8.7%. The four easier fifths have mean risk below the target, whereas the hardest fifth reaches 24.9% (25.6% for SED), within the worst-case bound . The guarantee thus does not extend to names chosen because they are difficult. When is contaminated with a fraction of the hardest fifth, the risk grows linearly in from 9.4% at , with slope equal to the gap between the hardest fifth and . It stays well inside the bound : at it is 11.0% against 20%. At the hardest fifth reaches 12.5% for ViT-L and 13.2% for SED.
Prevalence weights. If source and added labels differ only through a label feature that can be computed without annotations, weighted conformal prediction [72] restores a guarantee. As the feature we take the predicted prevalence of a label: the number of sampled pixels, over all images, at which its anchored score exceeds a logit of 0. We divide it into bins, at quantiles over all names or with equal numbers of source names per bin, and weight each source label in bin by ({share of added names in } b)/({share of source names in } b)$. For an added label in bin the threshold is the largest with ; if no such exists and the label receives the all-label threshold. As a diagnostic we also use the true prevalence, which is not available at deployment.
Table S22 and Table S23 give the results for anchored CAT-Seg ViT-L. With predicted prevalence, three bins over all names reach the target on ADE20K at moderate set size (9.6% at 27.4 labels per pixel). The same rule, however, assigns the all-label threshold to many added labels at and on Pascal Context, whose 59 source labels are too few per bin, and it is over-conservative on COCO-Stuff. Weights from true prevalence do not improve the results either, suggesting that prevalence alone does not capture the difference in difficulty between source and added classes. The binning rules were explored on ADE20K and then run once on Pascal Context and COCO-Stuff.
| Prevalence bins | weights from | ADE20K 150→845, | ADE20K 150→845, | Pascal Context 59→458, | Pascal Context 59→458, |
| unweighted (label-level CRC) | – | 22.2 / 9.0 | 13.1 / 20.6 | 22.3 / 2.7 | 12.6 / 5.0 |
| all names, 2 bins | predicted | 10.1 / 26.8 | 5.1 / 58.9 | 8.1 / 232.6 (41) | 5.9 / 234.2 (41) |
| all names, 2 bins | true | 11.5 / 426.6 (56) | 7.4 / 428.8 (56) | 18.1 / 231.2 (14) | 10.5 / 232.8 (14) |
| all names, 3 bins | predicted | 9.6 / 27.4 | 1.9 / 322.4 (38) | 4.2 / 309.3 (65) | 2.9 / 310.5 (65) |
| all names, 3 bins | true | 4.3 / 569.5 (78) | 2.2 / 575.5 (78) | 17.5 / 307.5 (47) | 10.5 / 308.5 (47) |
| all names, 5 bins | predicted | 10.1 / 25.6 | 4.4 / 62.5 | 4.8 / 278.9 (55) | 2.1 / 283.1 (55) |
| all names, 5 bins | true | 4.5 / 511.9 (69) | 2.2 / 516.9 (69) | 7.8 / 368.3 (80) | 5.0 / 369.0 (80) |
| source names, 2 bins | predicted | 16.1 / 14.5 | 9.2 / 32.7 | 16.8 / 3.5 | 7.5 / 9.2 |
| source names, 2 bins | true | 20.3 / 10.1 | 12.7 / 21.0 | 19.5 / 3.0 | 8.5 / 7.9 |
| source names, 3 bins | predicted | 13.5 / 18.8 | 7.4 / 42.7 | 13.3 / 4.3 | 1.8 / 55.5 |
| source names, 3 bins | true | 19.2 / 11.0 | 11.0 / 25.2 | 15.9 / 3.6 | 1.1 / 85.3 |
| source names, 5 bins | predicted | 11.6 / 22.9 | 6.1 / 51.6 | 10.4 / 5.5 | 0.8 / 379.2 (88) |
| source names, 5 bins | true | 15.1 / 15.6 | 7.6 / 41.0 | 9.7 / 6.4 | 1.1 / 406.1 (96) |
Table S22. Prevalence-weighted label-level CRC on the benchmark’s own split (anchored CAT-Seg ViT-L, 500 trials): added-label risk (%) / labels per pixel at the grown vocabulary; in parentheses, the share (%) of added labels that receive the all-label threshold, where it is not zero. Predicted: prevalence from the anchored scores; true: from the annotations (a diagnostic).
| Prevalence bins | weights from | ||
| unweighted (label-level CRC) | — | 17.6 / 6.3 | 8.9 / 13.2 |
| all names, 2 bins | predicted | 2.7 / 54.9 | 0.6 / 689.8 (37) |
| all names, 2 bins | true | 16.6 / 647.1 (3) | 8.4 / 652.0 (3) |
| all names, 3 bins | predicted | 3.9 / 34.0 | 1.2 / 484.9 (17) |
| all names, 3 bins | true | 9.6 / 860.2 (39) | 6.8 / 862.5 (39) |
| all names, 5 bins | predicted | 3.0 / 277.5 (19) | 1.1 / 365.4 (20) |
| all names, 5 bins | true | 2.2 / 1031.1 (78) | 1.5 / 1032.2 (78) |
| source names, 2 bins | predicted | 11.7 / 9.7 | 5.3 / 21.8 |
| source names, 2 bins | true | 11.9 / 9.5 | 5.3 / 21.8 |
| source names, 3 bins | predicted | 8.9 / 12.7 | 3.8 / 29.5 |
| source names, 3 bins | true | 11.4 / 9.7 | 4.5 / 25.9 |
| source names, 5 bins | predicted | 6.4 / 17.5 | 2.3 / 53.5 |
| source names, 5 bins | true | 6.6 / 17.5 | 1.9 / 71.9 |
Table S23. Prevalence-weighted label-level CRC on COCO-Stuff 171→1,283, benchmark’s own split (as in Table S22).
Annotating randomly drawn added names. A deployment can instead restore exchangeability by construction. It draws names uniformly at random from the names that may be added, annotates the calibration images for them, and calibrates the label-level threshold of Proposition 3 on the drawn names that occur in these images. Given , these form a uniformly random subset of the added labels present, so a remaining added label is exchangeable with them and Proposition 3 applies. The source classes keep the image-level threshold, whose guarantee is unaffected, since with candidate-independent scores the loss at a pixel depends only on the threshold of its true label. Table S21 gives results for CAT-Seg ViT-L and SED on ADE20K, Pascal Context and COCO-Stuff, with a fresh image split and draw in each of 500 trials and the drawn names nested across . Mean added-label risk on the calibration images stays below the target for every tested in every run. At and , ViT-L gives 9.3% risk on calibration images and 9.4% on held-out images, with 29.5 labels per pixel against 43.4 for native recalibration. The held-out result is empirical; the fixed-image guarantee concerns the calibration images.
Set sizes at this budget approach the random-source references: 29.5 versus 27.6 for ViT-L and 48.0 versus 45.1 for SED. This is budget-dependent: at , sizes are 102.1 and 157.7, respectively. At and , the finite-sample correction forces the all-label threshold in 86% of ViT-L draws and 86% of SED draws.
On Pascal Context and COCO-Stuff, fewer of the drawn names occur in the calibration images ( and 33 at ), so small budgets reach the all-label threshold more often: at , in 54% of Pascal and 84% of COCO-Stuff draws for ViT-L. At , ViT-L gives 8.9 labels per pixel on Pascal Context and 13.7 on COCO-Stuff, against 11.5 and 20.2 for native recalibration, and SED gives 14.1 on Pascal Context and 29.4 on COCO-Stuff, against 15.7 and 32.7. The cost is annotating names on the calibration images, only of which are present.
Added labels under image-level thresholds
The image-level thresholds of § 6.1 are calibrated on the source labels only and give no guarantee for the added labels. We compare their empirical miss rates at equal set size to measure how each model distributes its predictions between source and added labels. We compare the anchored threshold with the native threshold moved on the calibration images until its set size matches the anchored one (Table S24 to Table S26, Figure S4). An oracle image-level calibration using annotations for all 845 labels leaves an added-label risk of 26.7–45.7% on ADE20K across the configurations of Table S24, anchored or native, so these labels are hard for every method.
Figure S4. Added-label risk on ADE20K at 845 labels, per configuration (mean ± s.d. over splits). Numbers: set size at 845 labels.
| Configuration | anchored | native, recalibrated | native at anchored size | native, reused |
| CAT-Seg ViT-B, COCO-171 bank | 36.0 / 7.8 | 49.9 / 5.1 | 40.8 / 7.8 | 63.2 / 2.8 |
| CAT-Seg ViT-B, 683-name bank | 40.3 / 6.2 | 49.9 / 5.1 | 45.4 / 6.2 | 63.2 / 2.8 |
| CAT-Seg ViT-L, COCO-171 bank | 27.6 / 5.4 | 42.0 / 3.6 | 31.7 / 5.4 | 55.4 / 2.2 |
| CAT-Seg ViT-L, 683-name bank | 32.4 / 4.5 | 42.0 / 3.6 | 36.1 / 4.5 | 55.4 / 2.2 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank | 27.1 / 4.6 | 36.9 / 3.5 | 30.0 / 4.6 | 49.6 / 2.2 |
| SED ConvNeXt-L, COCO-171 bank | 30.8 / 7.4 | 42.5 / 5.3 | 35.7 / 7.4 | 31.9 / 8.9 |
| SED ConvNeXt-L, fast mode (top-8 rejection) | 30.8 / 7.4 | 46.2 / 5.3 | 41.1 / 7.4 | 52.8 / 3.6 |
| SED ConvNeXt-L, 683-name bank | 36.2 / 6.3 | 42.5 / 5.3 | 39.0 / 6.3 | 31.9 / 8.9 |
| FC-CLIP ConvNeXt-L, 133-name bank | 31.0 / 14.1 | 34.5 / 8.2 | 26.8 / 14.1 | 43.0 / 5.2 |
| FC-CLIP ConvNeXt-L, 701-name bank | 28.6 / 11.5 | 34.5 / 8.2 | 29.4 / 11.5 | 43.0 / 5.2 |
| FC-CLIP ConvNeXt-L, 1,366-name bank | 29.2 / 11.2 | 34.5 / 8.2 | 29.9 / 11.2 | 43.0 / 5.2 |
| FC-CLIP ConvNeXt-L, 2,731-name bank | 29.2 / 11.0 | 34.5 / 8.2 | 30.1 / 11.0 | 43.0 / 5.2 |
| ViT-B, aggregator fine-tuned 10k | 37.7 / 7.4 | 46.6 / 5.4 | 39.4 / 7.4 | 49.7 / 4.7 |
| ViT-B, fine-tuned with distractors | 44.9 / 6.5 | 46.5 / 5.6 | 43.3 / 6.5 | 53.5 / 4.1 |
Table S24. Added-label risk under image-level thresholds on ADE20K (, 500 splits, risk % / set size at 845 labels). Thresholds are calibrated on source labels only; the 695 added labels are evaluated with the ADE20K-847 annotations at the same pixels (6.6% of the sampled pixels). “Native at anchored size”: the native threshold moved on the calibration images until its set size matches that of the anchored model. In the SED fast-mode row only the native arms use fast mode; the anchored arm is SED in its default mode with the COCO-171 bank.
| Configuration | anchored | native, recalibrated | native at anchored size | native, reused |
| CAT-Seg ViT-B, 683-name bank, COCO 171→1,283 | 38.0 / 3.9 | 45.9 / 3.1 | 38.9 / 3.9 | 54.1 / 2.4 |
| CAT-Seg ViT-L, 683-name bank, COCO 171→1,283 | 30.2 / 3.3 | 38.9 / 2.7 | 33.2 / 3.3 | 46.4 / 2.1 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, COCO 171→1,283 | 25.2 / 3.2 | 30.8 / 2.6 | 24.7 / 3.2 | 38.6 / 2.1 |
| SED ConvNeXt-L, 683-name bank, COCO 171→1,283 | 41.2 / 3.6 | 42.0 / 3.8 | 43.5 / 3.6 | 22.2 / 8.2 |
| FC-CLIP ConvNeXt-L, 701-name bank, COCO 171→1,283 | 23.1 / 27.2 | 18.3 / 26.5 | 18.1 / 27.2 | 29.3 / 9.8 |
Table S26. Added-label risk on COCO-Stuff (columns as in Table S24; risk on the 1,112 added labels / set size at 1,283 labels; 3.1% of the sampled pixels, in 1,300 images, carry an added label).
For CAT-Seg ViT-L on ADE20K, the anchored threshold misses 32.4% of the added-label pixels and the size-matched native threshold 36.1%; the latter lowers the source-label risk to 8.1%, i.e. the native model spends the additional set size on the source labels. With the released weights, the anchored threshold misses fewer added-label pixels for every CAT-Seg and SED configuration on ADE20K and Pascal Context, and on COCO-Stuff for ViT-B, ViT-L and SED; for DeCLIP on COCO-Stuff the two are about equal (25.2 against 24.7%). The ViT-B aggregator fine-tuned with distractor names is an exception (44.9 against 43.3%; Sec. H.6). For FC-CLIP the comparison depends on the bank and the benchmark (28.6 against 29.4% on ADE20K, 36.5 against 34.9% on Pascal Context, 23.1 against 18.1% on COCO-Stuff). Lower added-label risk at equal set size is therefore a property of CAT-Seg and SED rather than of the conversion in general.
Reference bank and cost
This section completes § 6.5: the full bank sweep for CAT-Seg and the bank size for FC-CLIP (Sec. H.1), segmentation accuracy (Sec. H.2), runtime and memory (Sec. H.3), and fine-tuning the aggregator for anchored scoring (Sec. H.6).
| Added names | CAT-Seg ViT-L held-out | CAT-Seg ViT-L risk | CAT-Seg ViT-L | CAT-Seg ViT-L | SED held-out | SED risk | SED | SED |
| fifth 1 (easiest) | 0.0 | 1.7 | 2.3 | 10.0 | 0.0 | 1.3 | 1.8 | 10.0 |
| fifth 2 | 0.0 | 6.1 | 6.7 | 13.5 | 0.0 | 6.3 | 6.9 | 12.9 |
| fifth 3 | 0.3 | 4.4 | 5.0 | 10.7 | 0.2 | 5.2 | 5.7 | 11.4 |
| fifth 4 | 3.3 | 6.2 | 6.8 | 12.7 | 3.3 | 6.1 | 6.7 | 15.1 |
| fifth 5 (hardest) | 39.6 | 24.9 | 25.4 | 36.1 | 41.9 | 25.6 | 26.2 | 38.5 |
| all remaining | 8.6 | 8.7 | 9.2 | 10.3 | 9.0 | 8.9 | 9.4 | 10.6 |
| 10.2 | 15 | 10.3 | 15 | |||||
| 11.0 | 20 | 11.1 | 20 | |||||
| 13.3 | 35 | 13.5 | 35 | |||||
| 17.2 | 60 | 17.5 | 60 |
Table S20. Added names selected by difficulty (ADE20K 150 845, , 500 trials of the exchangeable design, label-level threshold from 150 uniformly drawn names). Added names: uniform over one fifth of the remaining names, ranked by their loss on the held-out images (the first three fifths have held-out loss of at most 0.32%, so their order is largely arbitrary), or over all of them. Held-out: that loss (%). Risk: added-label risk on the calibration images (%), next to the bound of Proposition 4 at the realised threshold and its worst case. Bottom: the uniform draw over all names present in the calibration images, source names included, contaminated with a fraction of the hardest fifth. uniform draw hardest fifth: risk, and the bound .
| Calibration labels | risk | draws | size | native | risk | size | |
| CAT-Seg ViT-L, ADE20K benchmark source classes | 149 | 22.2 | 100 | 7.6 | – | 13.1 | 17.5 |
| random source vocabulary | 113 | 9.5 | 40 | 27.6 | – | 4.5 | 76.4 |
| random added names | 7 | 0.3 | 0 | 627.7 | 666.8 | 0.0 | 698.7 |
| random added names | 14 | 5.8 | 17 | 102.1 | 335.6 | 0.0 | 695.2 |
| random added names | 21 | 7.0 | 22 | 60.3 | 232.3 | 1.9 | 297.7 |
| random added names | 35 | 8.3 | 29 | 42.3 | 134.4 | 3.2 | 142.9 |
| random added names | 52 | 8.8 | 34 | 33.9 | 79.0 | 3.9 | 102.1 |
| random added names | 70 | 9.3 | 38 | 29.5 | 43.4 | 4.2 | 85.3 |
| random added names | 105 | 9.4 | 41 | 27.6 | 34.7 | 4.4 | 77.3 |
| random added names | 140 | 9.6 | 42 | 26.2 | 32.2 | 4.6 | 71.7 |
Table S21. Added-label guarantee from randomly drawn added names (ADE20K 150 845, Pascal Context 59 458, COCO-Stuff 171 1,283; benchmark’s own source classes, 500 trials). : drawn names present in the calibration images. Risk: added-label risk (%) on the calibration images, and percentage of draws above . Size: labels per pixel at the grown vocabulary, with the source classes at the image-level threshold and the added names at the label-level threshold; native: the same calibration on the native model at the grown vocabulary. Reference rows: label-level threshold from the benchmark’s source classes, and from a random source vocabulary (one threshold for all names, as in Table S18).
| Calibration labels | risk | draws | size | native | risk | size | |
| CAT-Seg ViT-L, Pascal Context benchmark source classes | 59 | 22.3 | 100 | 2.6 | – | 12.6 | 4.7 |
| random source vocabulary | 29 | 7.6 | 23 | 11.5 | – | 2.8 | 68.4 |
| random added names | 4 | 0.0 | 0 | 399.7 | 399.7 | 0.0 | 401.0 |
| random added names | 8 | 1.6 | 2 | 251.2 | 261.0 | 0.0 | 401.0 |
| random added names | 12 | 4.4 | 8 | 74.8 | 91.4 | 0.0 | 399.5 |
| random added names | 21 | 6.8 | 24 | 20.0 | 25.1 | 1.5 | 179.7 |
| random added names | 31 | 7.8 | 28 | 11.4 | 14.7 | 2.8 | 67.2 |
| random added names | 42 | 8.5 | 32 | 8.9 | 11.5 | 3.3 | 43.7 |
| random added names | 62 | 8.9 | 37 | 7.3 | 9.6 | 3.8 | 27.3 |
| random added names | 83 | 9.2 | 37 | 6.7 | 8.7 | 4.1 | 23.1 |
Table S21.
| Calibration labels | risk | draws | size | native | risk | size | |
| CAT-Seg ViT-L, COCO-Stuff benchmark source classes | 170 | 17.6 | 100 | 4.9 | – | 8.9 | 10.3 |
| random source vocabulary | 72 | 8.9 | 33 | 11.9 | – | 4.0 | 30.2 |
| random added names | 3 | 0.0 | 0 | 1114.1 | 1114.1 | 0.0 | 1115.3 |
| random added names | 7 | 0.6 | 1 | 946.9 | 970.7 | 0.0 | 1115.3 |
| random added names | 10 | 3.0 | 7 | 387.8 | 493.4 | 0.0 | 1115.3 |
| random added names | 17 | 6.0 | 17 | 39.9 | 124.7 | 0.5 | 825.5 |
| random added names | 25 | 7.4 | 23 | 18.8 | 47.4 | 2.4 | 147.2 |
| random added names | 33 | 7.9 | 26 | 13.7 | 20.2 | 3.0 | 59.3 |
| random added names | 49 | 8.7 | 32 | 10.7 | 13.0 | 3.7 | 35.8 |
| random added names | 66 | 9.0 | 35 | 9.9 | 11.6 | 4.0 | 28.6 |
Table S21.
| Calibration labels | risk | draws | size | native | risk | size | |
| SED, ADE20K benchmark source classes | 149 | 22.2 | 100 | 12.0 | – | 12.7 | 30.7 |
| random source vocabulary | 113 | 9.5 | 42 | 45.1 | – | 4.6 | 118.5 |
| random added names | 7 | 0.3 | 0 | 637.7 | 641.3 | 0.0 | 700.4 |
| random added names | 14 | 5.8 | 16 | 157.7 | 181.3 | 0.0 | 696.4 |
| random added names | 21 | 7.0 | 24 | 102.5 | 122.9 | 1.8 | 370.8 |
| random added names | 35 | 8.3 | 30 | 69.0 | 80.7 | 3.1 | 221.6 |
| random added names | 52 | 8.9 | 33 | 54.4 | 62.7 | 3.8 | 165.8 |
| random added names | 70 | 9.3 | 37 | 48.0 | 54.9 | 4.2 | 143.8 |
| random added names | 105 | 9.6 | 40 | 44.9 | 51.0 | 4.5 | 122.1 |
| random added names | 140 | 9.7 | 42 | 43.3 | 48.8 | 4.6 | 115.3 |
Table S21.
| Calibration labels | risk | draws | size | native | risk | size | |
| SED, Pascal Context benchmark source classes | 59 | 26.2 | 100 | 3.3 | – | 15.5 | 6.2 |
| random source vocabulary | 29 | 7.5 | 25 | 17.5 | – | 2.8 | 80.4 |
| random added names | 4 | 0.0 | 0 | 400.4 | 400.1 | 0.0 | 401.3 |
| random added names | 8 | 1.5 | 2 | 262.8 | 264.4 | 0.0 | 401.3 |
| random added names | 12 | 4.5 | 11 | 85.7 | 87.9 | 0.0 | 401.3 |
| random added names | 21 | 6.6 | 20 | 27.9 | 30.7 | 1.4 | 193.5 |
| random added names | 31 | 7.9 | 26 | 16.2 | 18.4 | 2.9 | 72.9 |
| random added names | 42 | 8.5 | 32 | 14.1 | 15.7 | 3.4 | 52.8 |
| random added names | 62 | 9.0 | 39 | 12.0 | 13.6 | 3.9 | 36.3 |
| random added names | 83 | 9.4 | 44 | 11.1 | 12.8 | 4.3 | 31.4 |
Table S21.
| Calibration labels | risk | draws | size | native | risk | size | |
| SED, COCO-Stuff benchmark source classes | 170 | 24.3 | 100 | 5.6 | – | 14.2 | 11.5 |
| random source vocabulary | 72 | 9.0 | 34 | 21.2 | – | 4.0 | 60.1 |
| random added names | 3 | 0.0 | 0 | 1114.2 | 1114.3 | 0.0 | 1115.4 |
| random added names | 7 | 0.7 | 2 | 956.8 | 959.2 | 0.0 | 1115.4 |
| random added names | 10 | 3.3 | 9 | 427.7 | 437.4 | 0.0 | 1115.4 |
| random added names | 17 | 6.3 | 22 | 73.2 | 81.3 | 0.5 | 858.1 |
| random added names | 25 | 7.5 | 26 | 39.7 | 45.2 | 2.6 | 216.7 |
| random added names | 33 | 8.1 | 29 | 29.4 | 32.7 | 3.0 | 117.7 |
| random added names | 49 | 8.8 | 33 | 22.9 | 25.6 | 3.8 | 71.2 |
| random added names | 66 | 9.0 | 36 | 20.9 | 23.5 | 4.0 | 60.5 |
Table S21.
Bank sweep
Table S27 gives the sweep summarised in § 6.5, with intermediate sizes, banks built from Pascal Context names (evaluated on ADE20K only) and banks that combine the COCO-Stuff names with abstract WordNet [52] nouns or non-words (evaluated on COCO-Stuff only). Top-10 accuracy barely changes beyond 20 names, suggesting that the bank acts mainly on the scale of the scores, not on their ranking. For FC-CLIP (Table S28) the overhead decreases with the bank size up to about 1,400 names and then levels off.
Images that chose the bank. The sweep ran on the first 400 of the 2,000 ADE20K validation images used in Table S4. Repeating the half/half splits on the other 1,600 images alone, with the 683-name bank, gives an anchored risk of at most 9.8% at for CAT-Seg ViT-L, SED and FC-CLIP, with set sizes within 0.3 labels per pixel of those in Table S4.
Redrawing the Open Images names. The 683-name bank holds the COCO-171 names and 512 names drawn at random from Open Images. We redrew the 512 names five times, keeping the COCO-171 names, and scored the 400 sweep images with each bank for SED and CAT-Seg ViT-L (Table S29); a redraw shares 11–20 names with the draw used in the paper. Every draw meets the target at both levels. At , the standard deviation of the set size over the redraws is 0.11 labels per pixel for SED and 0.11 for ViT-L, less than its standard deviation over the calibration splits of one bank (0.49 and 0.27). The overhead stays between 0.54 and 0.85 labels, and top-1 accuracy at 845 names changes by at most 0.4 points.
Segmentation accuracy
Tables S30 and S31 give the accuracy summarised in § 6.5. CAT-Seg runs the released sliding-window inference (640 square, kernel 384, overlap 0.333) with one global view, obtained by resizing the 640 square to 384 (the release resizes the input image instead). SED runs the released whole-image inference at 768 × 768. FC-CLIP runs the released ConvNeXt-L COCO-panoptic weights with its A-847 configuration (shortest edge 800, 250 queries) on all four benchmarks. Class names are those shipped with each model; for CAT-Seg we average the synonym embeddings, where the release uses the first synonym (on PC-59 this changes CAT-Seg ViT-B by at most 0.02 points). Scores are resampled to the annotation grid, the arg-max is taken over the benchmark vocabulary, and mIoU follows the detectron2 convention; PC-459 keeps “unknown” as a class, as in the released preparation. All validation images are used (2,000 for ADE20K, 5,104 for Pascal Context).
The native rows are our runs of the released code. They are within 0.4 points of the published values on A-847, PC-459 and A-150 and 1.0–1.8 points below on PC-59 for CAT-Seg, SED and DeCLIP; FC-CLIP is 1.3–3.4 points below its published values on all four benchmarks, including A-847, whose released configuration we use throughout; we did not identify the cause. The anchored–native gaps use these same runs as the baseline; the FC-CLIP shortfall limits comparison with its published performance. For FC-CLIP the conversion raises pixel accuracy on every benchmark while lowering mIoU on ADE20K (Table S31): the softmax over the bank favours frequent classes.
| Model | scoring | A-150 | PC-459 | A-847 | PC-59 |
| CAT-Seg ViT-B | native (released) | 72.9 | 69.4 | 62.1 | 79.4 |
| anchored, COCO-171 bank | 70.4 | 67.6 | 59.0 | 77.8 | |
| anchored, 683-name bank | 72.1 | 68.8 | 60.8 | 78.6 | |
| CAT-Seg ViT-L | native (released) | 76.1 | 72.1 | 64.3 | 81.8 |
| anchored, COCO-171 bank | 73.4 | 70.6 | 60.9 | 80.4 | |
| anchored, 683-name bank | 75.3 | 71.4 | 63.0 | 80.9 | |
| CAT-Seg + DeCLIP EVA-L | native (released) | 77.0 | 72.2 | 66.3 | 81.9 |
| anchored, COCO-171 bank | 74.4 | 70.6 | 63.3 | 80.4 | |
| anchored, 683-name bank | 76.3 | 71.6 | 65.4 | 81.1 | |
| SED ConvNeXt-L | native (released) | 73.8 | 70.0 | 61.7 | 80.1 |
| anchored, COCO-171 bank | 71.6 | 69.1 | 59.1 | 79.0 | |
| anchored, 683-name bank | 73.1 | 69.7 | 60.8 | 79.3 | |
| FC-CLIP ConvNeXt-L | native (released) | 67.7 | 61.5 | 51.6 | 76.0 |
| anchored, 133-name bank | 70.7 | 68.8 | 62.6 | 78.4 | |
| anchored, 701-name bank | 69.9 | 65.5 | 59.0 | 76.7 |
Table S31. Pixel accuracy (%) of the runs in Table S30.
| Model | scoring | A-150 | PC-459 | A-847 | PC-59 |
| CAT-Seg ViT-B | native (released) | 31.7 | 19.0 | 12.2 | 56.4 |
| anchored, COCO-171 bank | 30.4 | 18.4 | 11.6 | 54.3 | |
| anchored, 683-name bank | 31.0 | 18.7 | 11.9 | 55.0 | |
| CAT-Seg ViT-L | native (released) | 37.8 | 23.7 | 15.9 | 62.0 |
| anchored, COCO-171 bank | 36.1 | 23.1 | 15.4 | 60.1 | |
| anchored, 683-name bank | 36.8 | 23.2 | 15.2 | 60.4 | |
| CAT-Seg + DeCLIP EVA-L | native (released) | 40.6 | 25.8 | 17.9 | 62.8 |
| anchored, COCO-171 bank | 38.5 | 25.1 | 16.9 | 60.1 | |
| anchored, 683-name bank | 39.7 | 25.2 | 16.9 | 61.1 | |
| SED ConvNeXt-L | native (released) | 34.9 | 22.2 | 13.7 | 59.1 |
| anchored, COCO-171 bank | 33.5 | 22.1 | 13.6 | 57.5 | |
| anchored, 683-name bank | 34.0 | 21.9 | 13.7 | 57.8 | |
| FC-CLIP ConvNeXt-L | native (released) | 32.0 | 16.6 | 13.5 | 55.0 |
| anchored, 133-name bank | 24.3 | 15.1 | 7.0 | 57.2 | |
| anchored, 701-name bank | 28.0 | 16.8 | 10.7 | 55.7 |
Table S30. Full-resolution mIoU of the released models and their anchored conversions, with the official inference of each model. Gaps quoted in the text are computed before rounding.
Source and added classes. Table S32 splits the pixel accuracy on A-847 and PC-459 into the classes of A-150 or PC-59 (source) and the others (added: 697 classes and 6.9% of the annotated pixels on A-847; 399 classes, 285 of them present, and 5.0% of the pixels on PC-459, where “unknown” belongs to neither group). The prediction is still the arg-max over the whole benchmark vocabulary. For CAT-Seg and SED with the 683-name bank, added-class pixel accuracy changes by at most 1.4 points, and with the COCO-171 bank it rises by 1.3–3.5 points; the small loss in overall pixel accuracy falls on the source classes. In these models the bank enters only the class attention, and each label keeps its own logit. For FC-CLIP with the 701-name bank, added-class pixel accuracy falls from 37.5 to 23.2% on A-847 and from 35.5 to 31.4% on PC-459, while source-class accuracy rises. Here the bank is the normaliser of each mask’s class probabilities: an added candidate competes with every bank name, but no bank name competes with it. Grounding DINO loses added-class accuracy in the same way, with the bank fixing its boxes (Table S40). Anchoring thus costs added-class accuracy where the bank fixes the output on which a candidate is scored, not where it only provides context.
Table S40. Detection accuracy at 1,204 names (COCO 801,204, COCO LVIS v1; from the stored box-level scores). Top-1: the true label has the largest score among all names at the ground-truth box (%); class mean: top-1 averaged over the labels present; localised: some predicted box overlaps the ground-truth box with IoU ; top-1 | loc.: top-1 among localised boxes. Unlocalised boxes and ties count as wrong.
| Detector | scoring | COCO boxes top-1 | COCO boxes class mean | COCO boxes localised | LVIS (added) boxes top-1 | LVIS (added) boxes top-1 | loc. | LVIS (added) boxes class mean |
| Grounding DINO T | native, chunks of 40 | 69.7 | 61.3 | 83.0 | 23.8 | 28.6 | 15.5 |
| anchored, bank A | 79.7 | 73.8 | 76.4 | 15.5 | 20.3 | 9.1 | |
| anchored, bank B | 48.5 | 35.6 | 67.6 | 13.3 | 19.7 | 10.6 | |
| MM-Grounding DINO T | native, chunks of 40 | 65.3 | 53.6 | 88.4 | 34.2 | 38.7 | 24.5 |
| anchored, bank A | 85.0 | 79.9 | 70.1 | 14.7 | 21.0 | 8.8 | |
| anchored, bank B | 43.0 | 24.0 | 72.8 | 17.6 | 24.2 | 12.0 | |
| Grounding DINO B | native, chunks of 40 | 89.7 | 86.3 | 85.1 | 24.7 | 29.0 | 16.6 |
| anchored, bank A | 88.0 | 85.3 | 74.1 | 16.5 | 22.3 | 12.0 | |
| anchored, bank B | 51.3 | 43.6 | 71.5 | 17.3 | 24.2 | 17.4 | |
| OWLv2 B (control) | native | 62.6 | 67.6 | 89.3 | 55.4 | 62.0 | 49.5 |
Table S40.
Runtime and memory
Table S33 gives the cost quoted in § 5.1. The native model is run once at the grown vocabulary, as at deployment; the pass at the source vocabulary is needed only once, for calibration. The segmentation block also includes a separate native pass over the bank and each candidate, to measure the cost of recomputing their joint context. The detection rows compare the native model run on all 1,204 names in chunks of 40 with the anchored model (§ 1); here the ratio is below one, because the anchored model runs the image side twice, not once per chunk.
A leaner bank pass. The bank pass timed in Table S33 also computes outputs that no candidate reads: for CAT-Seg the bank’s own logits and the output of the last attention layer, for SED the stage-3 attention output, the third fusion block and the prediction head. Stopping the pass once the key and value sums of every layer are recorded leaves each candidate score unchanged bit for bit (checked on 20 images per model), so the calibration carries over as it is. Table S34 compares the two passes on an RTX 5070 Ti (Tab. S33 was timed on a GH200). With the 683-name bank, the lean pass removes 15% of the anchored time for CAT-Seg ViT-L and 16% for SED; the ratio falls from 4.3 to 3.7× for ViT-L and from 1.6 to 1.4× for SED. CAT-Seg with the DeCLIP backbone, which has the ViT-L aggregator, runs at 4.2 and 3.6× the native time. Much of the remaining work comes from the number of labels processed: the anchored pass
| Model | features | full bank pass | full bank pass | lean bank pass | lean bank pass | peak memory | |
| Model | features | native | anchored | ratio | anchored | ratio | peak memory |
| CAT-Seg ViT-B, COCO-171 bank, sliding window 640 | 0.07 | 0.75 | 2.69 | 3.3× | 2.56 | 3.2× | 8.2 GB |
| CAT-Seg ViT-B, 683-name bank | 0.07 | 0.77 | 3.98 | 4.8× | 3.38 | 4.1× | 8.2 GB |
| CAT-Seg ViT-L, COCO-171 bank | 0.22 | 0.75 | 2.70 | 3.0× | 2.55 | 2.8× | 9.3 GB |
| CAT-Seg ViT-L, 683-name bank | 0.23 | 0.75 | 3.98 | 4.3× | 3.38 | 3.7× | 9.3 GB |
| SED ConvNeXt-L, COCO-171 bank, whole image 768 | 0.10 | 0.64 | 0.75 | 1.2× | 0.71 | 1.1× | 8.0 GB |
| SED ConvNeXt-L, 683-name bank | 0.15 | 0.63 | 1.13 | 1.6× | 0.95 | 1.4× | 8.1 GB |
| CAT-Seg + DeCLIP EVA-L, 683-name bank | 0.24 | 0.77 | 4.02 | 4.2× | 3.39 | 3.6× | 9.3 GB |
Table S34. Full against lean bank pass (ratios from unrounded times): seconds per image, fp32, on one RTX 5070 Ti over the same 100 development images as Table S33, each pair timed back to back on an idle device. The lean pass stops once the key/value sums are recorded; candidate scores are identical. Features and the native pass are shared by both runs; peak memory is the maximum over all passes of the lean run.
| Configuration | anchored | native, recalibrated | native at anchored size | native, reused |
| CAT-Seg ViT-B, COCO-171 bank, Pascal 59→458 | 28.3 / 3.7 | 44.3 / 2.6 | 35.1 / 3.7 | 58.7 / 1.7 |
| CAT-Seg ViT-B, 683-name bank, Pascal 59→458 | 36.5 / 3.0 | 44.3 / 2.6 | 40.5 / 3.0 | 58.7 / 1.7 |
| CAT-Seg ViT-L, 683-name bank, Pascal 59→458 | 30.7 / 2.5 | 40.6 / 2.1 | 34.9 / 2.5 | 55.2 / 1.5 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, Pascal 59→458 | 22.6 / 2.6 | 32.7 / 2.1 | 27.2 / 2.6 | 47.2 / 1.5 |
| SED ConvNeXt-L, 683-name bank, Pascal 59→458 | 43.6 / 2.9 | 49.8 / 2.7 | 47.3 / 2.9 | 31.2 / 4.9 |
| FC-CLIP ConvNeXt-L, 701-name bank, Pascal 59→458 | 36.5 / 6.1 | 37.0 / 5.4 | 34.9 / 6.1 | 49.9 / 2.9 |
Table S25. Added-label risk on Pascal Context (columns as in Table S24; risk on the 399 added labels / set size at 458 labels; 5.5% of the sampled pixels carry an added label).
scores the bank names and all 845 candidates, where native CAT-Seg scores 256 per crop.
Memory. Table S35 gives the peak GPU memory of each arm on its own, as the vocabulary grows from the 150 source names to 6,760 (the 845 evaluated names, then Open Images names). For CAT-Seg ViT-L, the anchored model with the lean bank pass needs 2.4 GB more than the native model at the source vocabulary and 0.7 GB less from 845 names on. Both stay flat beyond 845 names, the native model because it keeps 256 labels per crop. Native SED grows with every name: 8.01 GB at 845 names, 27.4 GB at 3,380, more than the card holds, and out of memory at 6,760. Each of its stages attends over all labels, so the feature maps of every label are kept until the stage is complete. An anchored candidate needs only the bank’s key and value sums, so candidates are processed in chunks, and the anchored peak stays between 4.65 and 5.91 GB. FC-CLIP classifies a fixed set of mask queries, and both of its arms stay below 2.7 GB. CAT-Seg with the DeCLIP backbone follows ViT-L: the anchored model needs 8.44 GB at every size, the native model 9.10 GB from 845 names on. Besides the weights, the anchored model keeps the bank’s text embeddings (2.0 MB for CAT-Seg ViT-L, 160 MB for SED). A recalibration needs the labelled calibration images (42 MB for 1,000 ADE20K images) and the memory of a native pass at the new vocabulary.
| Model | arm | 150 | 845 | 1,690 | 3,380 | 6,760 |
| CAT-Seg ViT-B | native | 4.99 | 8.05 | 8.05 | 8.05 | 8.06 |
| CAT-Seg ViT-B | anchored | 7.44 | 7.44 | 7.44 | 7.44 | 7.44 |
| CAT-Seg ViT-B | anchored, lean | 7.39 | 7.39 | 7.39 | 7.39 | 7.39 |
| CAT-Seg ViT-L | native | 6.04 | 9.10 | 9.10 | 9.11 | 9.12 |
| CAT-Seg ViT-L | anchored | 8.49 | 8.49 | 8.49 | 8.49 | 8.49 |
| CAT-Seg ViT-L | anchored, lean | 8.44 | 8.44 | 8.44 | 8.44 | 8.44 |
| SED | native | 3.55 | 8.01 | 14.47 | 27.39‡ | OOM |
| SED | anchored | 6.62 | 6.62 | 6.62 | 6.62 | 6.62 |
| SED | anchored, lean | 4.65 | 4.65 | 4.65 | 4.68 | 5.91 |
| FC-CLIP | native | 2.59 | 2.60 | 2.61 | 2.63 | 2.67 |
| FC-CLIP | anchored | 2.60 | 2.61 | 2.62 | 2.64 | 2.67 |
| CAT-Seg + DeCLIP | native | 6.04 | 9.10 | 9.10 | 9.11 | 9.12 |
| CAT-Seg + DeCLIP | anchored | 8.49 | 8.49 | 8.49 | 8.49 | 8.49 |
| CAT-Seg + DeCLIP | anchored, lean | 8.44 | 8.44 | 8.44 | 8.44 | 8.44 |
Table S35. Peak GPU memory per arm (GB), fp32, on one RTX 5070 Ti, maximum over 3 ADE20K development images, with the peak reset before each arm. Weights, text embeddings and image features are included. Vocabularies: the 845 evaluated names, then Open Images names. lean: lean bank pass. Native CAT-Seg matches the released code to within 0.02 GB. ‡More than the 15.9 GB of the card, allocated only through the driver’s fallback to system memory. OOM: out of memory.
When recalibration suffices
Recalibrating the native model at a new vocabulary needs no new annotation, since the calibration images are labelled for the source vocabulary and the loss (1) counts only source-labelled pixels. It requires access to those images and a known vocabulary or selection rule. A fixed per-image rule can be replayed on the calibration images [39]; a fixed family of rules can be calibrated under its worst case (Proposition 2). Each vocabulary or rule requires a native pass over the calibration images.
With 1,000 calibration images, this pass takes 5.7 minutes on one GH200 for CAT-Seg ViT-L, 6.1 for SED and at most 5.1 for FC-CLIP (Table S33). The conversion instead adds (ratio ) native passes to every test image. Anchored inference therefore costs less total compute whenever fewer than 1, 000/() test images are served per vocabulary: 340 for CAT-Seg ViT-L, 300 for ViT-B and 1, 400 for SED, with the 683-name bank. If the vocabulary-independent features of the calibration images are cached, the break-even falls to 270, 270 and 1, 300 images.
For FC-CLIP the conversion adds the per-query logits of the 701 bank names to every test image. No native pass without these columns was timed, but the pass with the 133-name bank bounds it from above, so the ratio is at least 1.3 and the break-even is at most 3, 000 images. FC-CLIP’s backbone and mask proposals do not depend on the vocabulary, so with these cached, recalibration reduces to the logits of the new names for the cached queries. This step was not timed separately. Table S36 summarises which deployments each option serves.
Recalibration in deployment
Table S36 lists when recalibration is available. This section describes what it requires when the vocabulary changes during use.
Detecting the failure. Recalibration starts only once the failure is noticed. Without labels at deployment, the prediction sets are the only signal, and in the 12 of 15 combinations where the reused threshold exceeds the target, they grow by only 14–40% of what the target needs (§ 4.1). The valid anchored sets grow more than the failing native ones in each of these combinations (§ 6.1), so set growth does not separate a valid threshold from a failing one. Detecting the rise in risk requires labelled images at the grown vocabulary.
No FC-CLIP pass without bank columns was timed; the 133-name-bank pass bounds it from above, so the 701-name-bank ratio is a lower bound.
Until recalibration ends. Recalibration takes one native pass over the 1,000 calibration images, as long as segmenting them: 5.7 minutes for CAT-Seg ViT-L on one GH200 and 16.4 on one RTX 5070 Ti (Sec. H.4). On a single device the system either stops for this time or keeps running under the invalid threshold, and recalibrating afterwards does not cover the images already processed. When a tagger chooses the names for each frame, the vocabulary changes before any recalibration ends.
Data on the device. Each recalibration needs the labelled calibration images where the model runs. Their licence or privacy may prevent copying them to the device, e.g. medical images that cannot leave a site, and a party that receives a calibrated model without them cannot recalibrate at all. Segmentation also runs on board satellites, which send only selected images to the ground [28], and is proposed for driving planetary rovers [71]; there, recalibration needs the labelled images and a native pass over them on the device, with its memory at the new vocabulary (Table S35), or a round trip to the ground. The anchored model needs the bank names, a list of text. Its cost is inference time (3.7× native for CAT-Seg ViT-L with the lean bank pass, Table S34), which is the scope condition on constrained hardware.
Recalibrating many times. The CRC guarantee holds on average over calibration sets, and for each recalibrated vocabulary separately. Vocabularies that arrive one after another, and are not known in advance, need a guarantee that holds for all of them at once with probability over the calibration set. By the union bound, this requires level for each. The anchored source-label scores do not depend on the vocabulary, so a single calibration at level covers every vocabulary. We compare the two with RCPS [8] (Hoeffding–Bentkus bound) at on the splits of § 6.1, half of the images calibrating and half evaluating. The recalibrated native sets become larger than the anchored ones from for CAT-Seg ViT-L on ADE20K and Pascal Context; at on ADE20K they hold 5.37 against 5.11 labels per pixel. The crossover is at for SED on ADE20K and Pascal Context, for SED on COCO-Stuff and for FC-CLIP on ADE20K. When the vocabularies are known in advance, a single calibration under their worst case (Proposition 2) avoids the union bound. If the next vocabulary is chosen after inspecting results on the calibration images, for example by adding the names that fail there, the calibration images are no longer exchangeable with the test images, and recalibrating on them carries no guarantee.
Fine-tuning the aggregator
The conversion requires no training, but the aggregator can be fine-tuned for anchored scoring (bank plus candidates, per-label sigmoid loss on COCO-Stuff, 10k iterations). The plain recipe reduces the ViT-B set size at 845 labels from 7.32 to 7.01. Adding hard-negative distractor names reduces it to 6.15, but lowers the top-10 accuracy at 845 labels on a fixed group of 30 ADE20K source labels from 82.7 to 80.1% and loses the advantage on added labels at equal set size (Table S37). We retain the released weights with a large bank in the main experiments.
| Aggregator | anchored size at +695 | native recal. size | held-out top-10 | added-label risk at matched size (anchored / native) |
| CAT-Seg ViT-B, COCO-171 bank | 7.32 | 4.99 | 82.7 | 36.0 / 40.8 |
| ViT-B, aggregator fine-tuned 10k | 7.01 | 5.31 | 82.8 | 37.7 / 39.4 |
| ViT-B, fine-tuned with distractors | 6.15 | 5.47 | 80.1 | 44.9 / 43.3 |
Table S37. Fine-tuning the CAT-Seg ViT-B aggregator for anchored scoring (COCO-171 bank). Held-out top-10: accuracy (%) of the true label among the ten highest anchored scores at 845 labels, on a fixed group of 30 ADE20K source labels held out in an earlier pilot, over 400 development images. Some of these labels are also COCO-Stuff classes, so the group is not outside the training vocabulary.
Open-vocabulary detection
This section gives the protocol and the full results of § 6.4. The protocol was fixed before any score was computed; its deviations are dated in the released protocol document.
Models. Grounding DINO T and B [46] and MM-Grounding DINO T [91], a later training of the same architecture, all with their released weights and processor defaults (fp32). The Hugging Face Transformers implementation we use (version 5.17) does not load the decoder heads of two checkpoints as released: MM-Grounding DINO T receives the box and class heads of the first decoder layer in every layer, and Grounding DINO B a box head that differs from the checkpoint’s. We copy the heads from the checkpoint files after loading; for Grounding DINO T, whose heads load correctly, this leaves the scores unchanged. OWLv2 B [53] is a control whose scores are candidate-independent by construction, since its queries do not see the text and each class is scored by its own sigmoid.
Growth. The source vocabulary is the 80 COCO classes. The 1,124 added labels are the LVIS categories that are not COCO classes, in a fixed seeded order. Two deployments are evaluated. In the first, the prompt is extended with added names until it reaches 256 tokens (100 names in total). In the second, all 1,204 names are split into consecutive chunks of 40, with the source names first, and a box is scored for a label by the chunk that holds it.
Scores and loss. The score of a query for a label is the largest token logit over the label’s phrase tokens. The score of a ground-truth box for a label is the largest score over the predicted boxes that overlap it with IoU , and , the lowest point of the threshold grid, if there is none. The loss of an image is the fraction of its COCO boxes whose label is missing from the set. Images without a COCO box are excluded (1, 989 images with 14, 855 boxes remain). The set size is the mean number of labels in the set of a COCO box. This loss is evaluated at annotated boxes and stands in for a detection loss: the minimum threshold includes all labels even for an unmatched true box. That endpoint does not produce a localized detection, so the guarantee is not one of object recall; we report unmatched-box rates separately. Labels per ground-truth box are therefore an evaluation metric, not the number of predictions emitted by the detector.
Conversion. The image side (feature fusion, query selection and decoder) sees only the bank tokens. Each candidate phrase passes through the text branch together with the bank but is hidden from the image side, and it is scored against the final queries by the model’s contrastive head. The queries and boxes are those of the model with the bank as the whole prompt. Before any analysis, we checked on five images that the anchored scores do not change with the order of the candidates or with the other candidates (largest logit difference for Grounding DINO T, for Grounding DINO B and for MM-Grounding DINO T). The bank is either the 80 COCO names (A, the main configuration) or 80 Open Images names that are neither COCO nor LVIS concepts (B).
Results. Table S39 reports all evaluated configurations. The native source scores move when names are added to the prompt. At LVIS scale the source names fill their own chunks, and the reused threshold stays below the target for every detector at both values of . With one prompt it does so for Grounding DINO T and MM-Grounding DINO T, but not for Grounding DINO B (18.3% at , 9.9% at ). The anchored model with bank A equals the native model at the source vocabulary, is valid for every detector, and gives the smallest sets at LVIS scale, except for Grounding DINO B at (12.2 against 9.7 labels per box after recalibration). With bank B it is also valid, but its sets hold 4.1–11.9 labels per box already at the source vocabulary; as for the segmenters, the bank changes only the set sizes. OWLv2, calibrated once at the 80 names, has 9.9% risk with 8.3 labels per box at LVIS scale. At 1,204 names the anchored Grounding DINO models take 0.50–0.52 s per image against 2.97–3.31 s for the chunked native models (Table S33), because the image side runs twice (bank pass and joint pass) rather than once per chunk.
Table S39. Detection, every arm (COCO 801,204, COCO LVIS v1; mean over 500 splits). Risk (%) on COCO boxes; for native reused and recalibrated, the one-prompt and LVIS-scale deployments. Size: labels per COCO box at the 100-name prompt and at 1,204 names.
| Detector | arm | risk | size, one prompt | size, LVIS | risk | size, one prompt | size, LVIS |
| Grounding DINO T | native, source | 9.9 | – | – | 4.9 | – | – |
| native reused | 9.6, 9.6 | 1.7 | 94.7 | 4.6, 4.8 | 3.8 | 291.1 | |
| native recal. | 9.9, 9.9 | 1.6 | 88.3 | 4.9, 4.9 | 3.5 | 280.5 | |
| anchored, bank A | 9.9 | 1.7 | 34.2 | 4.9 | 4.0 | 121.2 | |
| anchored, bank B | 9.9 | 16.0 | 295.4 | 1.7 | 93.9 | 1158.5 | |
| MM-Grounding DINO T | native, source | 9.9 | – | – | 4.9 | – | – |
| native reused | 9.3, 9.0 | 1.7 | 23.0 | 4.3, 4.5 | 4.1 | 63.8 | |
| native recal. | 9.9, 9.9 | 1.6 | 20.2 | 4.9, 4.9 | 3.4 | 55.8 | |
| anchored, bank A | 9.9 | 1.2 | 3.1 | 4.9 | 2.0 | 9.5 | |
| anchored, bank B | 9.9 | 8.5 | 84.1 | 4.9 | 42.6 | 474.1 | |
| Grounding DINO B | native, source | 9.9 | – | – | 4.9 | – | – |
| native reused | 18.3, 9.8 | 0.9 | 3.7 | 9.9, 4.9 | 1.2 | 9.7 | |
| native recal. | 9.9, 9.9 | 1.2 | 3.7 | 4.9, 4.9 | 2.2 | 9.7 | |
| anchored, bank A | 9.9 | 1.0 | 3.6 | 4.9 | 1.4 | 12.2 | |
| anchored, bank B | 9.9 | 7.7 | 243.0 | 4.3 | 71.5 | 1026.7 | |
| OWLv2 B (control) | reused | 9.9 | – | 8.3 | 4.9 | – | 28.5 |
Table S39.
Added labels. We apply label-level CRC with a random source vocabulary of 80 of the 1,204 names (Proposition 3) and the image-averaged label loss (Proposition S1). For the anchored Grounding DINO models, the added-label risk is 3.2–4.8% on the calibration images and 3.2–4.8% on held-out images; for the native models at LVIS scale it is 7.3–7.9% on held-out images. A box that produced box overlaps is covered only at the lowest grid point. With the anchored queries, 23.6–29.9% of the added-label boxes are unmatched in this way, against 11.6–17.0% for the native models (source boxes: 2.1–2.5%). Often these boxes alone exceed for the drawn labels, and the calibrated threshold then includes every label. At this happens in 57–74% of the draws for the anchored models and 5–17% for the native models, and at in 77–100% for both. For OWLv2 the guarantee is informative: 10.7% of the added-label boxes are unmatched, the threshold is trivial in 3% of the draws at , and the added-label risk on held-out images is 7.9%. With the benchmark’s own split (source = COCO), the added-label risk is 38.1–66.1% for the anchored models and 19.0–63.6% for the native models, as Proposition 4 allows.
Accuracy. Table S40 gives the detection counterpart of the pixel accuracy of Table S31, computed from the same stored scores: at each ground-truth box, the name with the largest box-level score among all 1,204 is compared with the true label. A box that no predicted box overlaps counts as wrong, and ties count against the true label. On COCO boxes, top-1 is 79.7–88.0% for the anchored models with bank A and 65.3–89.7% for the chunked native models. On the 12,952 boxes of added labels it falls from 23.8–34.2% (native) to 14.7–16.5% (anchored). The drop is not only one of localisation: among localised boxes, top-1 falls from 28.6–38.7% to 20.3–22.3%, since the queries are selected and decoded for the bank names alone. Over all boxes, COCO and added together, top-1 moves from 48.3–59.4% to 49.8–54.7% and rises for two of the three detectors. Bank B lowers COCO top-1 to 43.0–51.3%. OWLv2, whose queries do not see the text, reaches 55.4% on the added boxes, so the cost comes from conditioning the image side on a fixed bank, not from candidate independence itself. LVIS labels are not exhaustive, so a box whose top name is correct but unannotated counts as wrong for every model alike.
Reserved images
This section reports the confirmation of § 6.6 in full. 2,000 ADE20K training images were drawn with a recorded seed, listed in a manifest and never used in development; one has no sampled source-label pixel and is excluded (Sec. A). They were passed through CAT-Seg ViT-L with the 683-name bank once, and the stored scores were analysed twice.
Source labels. The first analysis used a design fixed before the reserved images were evaluated: calibration on the first 999 images and evaluation on the rest (Table S41). The risks quoted in § 6.6 are those of this single split, so the anchored 8.4% varies around the expected risk. Over 500 random splits of the reserved images, the anchored risk is 9.7% and the reused native risk 15.0%.
Table S41. Reserved images (1,999 ADE20K training images not used in development; CAT-Seg ViT-L, 683-name bank; 150845). The pre-declared split calibrates on the first 999 images and evaluates on the rest.
| Design | risk, native reused | risk, anchored | size, anchored | size, native recal. | |
| pre-declared split | .10 | 13.8 | 8.4 | 2.424.55 | 3.71 |
| pre-declared split | .05 | 7.0 | 4.5 | 3.808.82 | 8.07 |
| 500 random splits | .10 | 15.0 | 9.7 | 2.334.43 | 3.49 |
| 500 random splits | .05 | 8.3 | 4.7 | 3.839.11 | 7.56 |
Table S41.
Added labels. The second analysis tested the added-label guarantee under a protocol written after the development results of § 6.3 and before any added-label annotation of a reserved image was decoded; the protocol is released with the code. The ADE20K-847 labels at the stored pixels come from the ADE20K-full export and agree with the ADE20K-150 annotation on 99.5% of the pixels labelled in both. Only designs that use the stored scores were pre-specified. The primary test was whether the anchored label-level threshold meets the target under a random source vocabulary. Its added-label risk is 9.3% at and 4.4% at 5%, with 26.2 and 71.3 labels per pixel. Native recalibration at 845 labels is also valid (9.4 and 4.2%) with larger sets (32.8 and 200.3), and the image-level threshold is not (23.1 and 13.9%). On the benchmark’s own split the label-level threshold misses 18.4% at , with an added–source gap of points (Proposition 4).
Beyond the marginal risk
The guarantee of Proposition 1 concerns the marginal image-level risk. This section examines how the risk is distributed over classes and scenes (Sec. K.1), how per-name thresholds bound it for each name (Sec. K.2), whether the anchored threshold remains valid under stricter losses (Sec. K.4) and other set constructions (Sec. K.5), and how many calibration images it needs (Sec. K.3).
Risk by class and scene
Averaged over the source classes, the anchored threshold has a lower risk than native recalibration in every ADE20K configuration (16.6 against 20.6% for ViT-L) and for CAT-Seg and SED on Pascal Context and COCO-Stuff, but not for FC-CLIP on those benchmarks (Table S42 to Table S44). By scene it is worse: in the worst ADE20K scene category (bathroom) its risk exceeds that of recalibration for every model except SED with the 683-name bank, most for ViT-B with the COCO-171 bank (20.7 against 11.1%; no other stratum in any configuration exceeds ). The 683-name bank narrows this gap for each model run with both banks (CAT-Seg ViT-L, SED and FC-CLIP).
| Configuration | class-averaged risk | worst ADE scene | ||||
| native reused | native recal. | anchored | native reused | native recal. | anchored | |
| CAT-Seg ViT-B, COCO-171 bank | 33.0 | 23.3 | 17.2 | 18.7 | 11.1 | 20.7 |
| CAT-Seg ViT-L, COCO-171 bank | 30.2 | 20.6 | 14.3 | 15.4 | 9.7 | 17.9 |
| CAT-Seg ViT-L, 683-name bank | 30.2 | 20.6 | 16.6 | 15.4 | 9.7 | 12.2 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank | 25.2 | 17.6 | 13.8 | 15.3 | 9.9 | 13.5 |
| SED ConvNeXt-L, COCO-171 bank | 14.2 | 20.5 | 14.9 | 7.5 | 11.3 | 16.9 |
| SED ConvNeXt-L, fast mode (top-8 rejection) | 25.8 | 20.7 | 14.9 | 13.0 | 9.9 | 16.9 |
| SED ConvNeXt-L, 683-name bank | 14.2 | 20.5 | 17.3 | 7.5 | 11.3 | 10.7 |
| FC-CLIP ConvNeXt-L, 133-name bank | 24.1 | 18.7 | 17.4 | 15.8 | 10.9 | 15.0 |
| FC-CLIP ConvNeXt-L, 701-name bank | 24.1 | 18.7 | 16.6 | 15.8 | 10.9 | 13.5 |
Table S42. Risk by stratum on ADE20K at 845 labels (): class-averaged risk over the source classes, and the worst ADE20K scene category (scenes with at least 20 evaluation images). The marginal risk of the anchored model is at the target in every row.
| Configuration | class-averaged risk | ||
| native reused | native recal. | anchored | |
| CAT-Seg ViT-B, 683-name bank, COCO | 24.9 | 20.2 | 19.0 |
| CAT-Seg ViT-L, 683-name bank, COCO | 23.2 | 18.9 | 18.0 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, COCO | 22.3 | 17.7 | 17.1 |
| SED ConvNeXt-L, 683-name bank, COCO | 8.6 | 18.8 | 18.6 |
| FC-CLIP ConvNeXt-L, 701-name bank, COCO | 16.0 | 9.4 | 10.0 |
Table S44. Class-averaged risk on COCO-Stuff (columns as in Table S42; no scene labels).
Per-name thresholds
The image-level threshold bounds the risk averaged over images, and single names can lie far above it. A deployer who needs a bound for each name can calibrate each name on its own images instead, as in Mondrian, or class-conditional, conformal prediction [20, 78]. For a name , let be the fraction of ’s pixels in calibration image scored below , over the calibration images annotated with , and let be the largest grid threshold with . If the images containing are exchangeable, a test image containing misses in expectation at most a fraction of ’s pixels. Conditioning on preserves exchangeability, and a threshold exists once (9 images at , 19 at 5%). The bound holds for each name separately, not for all names at once.
What anchoring adds. On anchored scores, does not depend on the requested vocabulary. Each therefore stays valid as names are added, and a new name is calibrated from its own images, each annotated only with that name’s mask. The images must be drawn because the name is present, not curated, so that they are exchangeable with the test images containing it; a name of frequency needs about screened images. On native scores, per-name thresholds calibrated at the source vocabulary leave 89 of 131 source names above for CAT-Seg ViT-L on ADE20K and 65 of 131 for FC-CLIP once the vocabulary grows (Table S45). Native per-name thresholds calibrated at the grown vocabulary are valid, but every added name then requires recalibrating every name.
Table S45. Per-name thresholds on anchored scores at the grown vocabulary (500 half/half splits, set sizes on the first 20). Calibrated: names with a per-name threshold per split, of the names present. Over : calibrated names, in at least 10 splits, whose held-out risk averaged over splits exceeds by more than three standard errors (for the image-level threshold, among the same names). Per-name sets hold the calibrated names only. Lower of both: calibrated names take the lower of the two thresholds, the others the image-level one; image risk is the loss (1). Native, reused: native per-name thresholds calibrated at the source vocabulary, of the source names calibrated in at least 10 splits.
| Benchmark | Model | calibrated | image-level over | image-level size | per-name over | per-name size | lower of both image risk | lower of both over | lower of both size | native, reused over | |
| 10% | ADE20K | CAT-Seg ViT-L | 160 of 836 | 154 | 4.47 | 0 | 10.94 | 4.8 | 0 | 13.30 | 89 of 131 |
| ADE20K | SED | 160 of 836 | 145 | 6.29 | 0 | 14.61 | 4.1 | 0 | 18.46 | 0 of 131 | |
| ADE20K | FC-CLIP | 160 of 836 | 135 | 11.56 | 0 | 25.29 | 4.7 | 0 | 33.12 | 65 of 131 | |
| Pascal Context | CAT-Seg ViT-L | 87 of 270 | 59 | 2.45 | 0 | 3.58 | 4.9 | 0 | 4.48 | 41 of 59 | |
| Pascal Context | SED | 87 of 270 | 70 | 2.86 | 0 | 5.52 | 4.3 | 0 | 6.71 | 0 of 59 | |
| COCO-Stuff | SED | 198 of 660 | 176 | 3.61 | 0 | 11.68 | 3.4 | 0 | 13.04 | 0 of 161 | |
| 5% | ADE20K | CAT-Seg ViT-L | 96 of 836 | 82 | 8.95 | 0 | 9.04 | 2.5 | 0 | 15.26 | 65 of 104 |
| ADE20K | SED | 96 of 836 | 76 | 15.96 | 0 | 13.12 | 1.9 | 0 | 25.67 | 0 of 104 | |
| ADE20K | FC-CLIP | 96 of 836 | 67 | 31.37 | 0 | 21.06 | 1.9 | 0 | 46.70 | 37 of 104 | |
| Pascal Context | CAT-Seg ViT-L | 61 of 270 | 41 | 3.86 | 0 | 4.12 | 2.1 | 0 | 6.04 | 35 of 54 | |
| Pascal Context | SED | 61 of 270 | 44 | 4.90 | 0 | 6.88 | 1.9 | 0 | 9.55 | 0 of 54 | |
| COCO-Stuff | SED | 129 of 660 | 125 | 6.25 | 0 | 12.03 | 2.2 | 0 | 15.20 | 0 of 147 |
Table S45.
Two options and their hybrid. The image-level threshold needs one calibration, includes every name and gives the smallest sets. Per-name thresholds bound each calibrated name, at the price of labelled images per name and larger sets, and names with fewer than calibration images get no threshold. The two combine: calibrated names take the lower of and the image-level threshold, and the other names take the image-level threshold. The loss is non-increasing as a threshold falls, so each image’s loss is at most its loss under the image-level threshold, and each name’s loss at most its loss under ; both guarantees hold at once.
Results. We repeat 500 random half/half splits of the images on six score stores (Table S45). Per-name thresholds
| Configuration | class-averaged risk | ||
| native reused | native recal. | anchored | |
| CAT-Seg ViT-B, COCO-171 bank, Pascal | 19.4 | 12.2 | 9.5 |
| CAT-Seg ViT-B, 683-name bank, Pascal | 19.4 | 12.2 | 11.0 |
| CAT-Seg ViT-L, 683-name bank, Pascal | 17.4 | 11.1 | 9.8 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, Pascal | 16.0 | 10.5 | 9.2 |
| SED ConvNeXt-L, 683-name bank, Pascal | 5.7 | 12.1 | 10.9 |
| FC-CLIP ConvNeXt-L, 701-name bank, Pascal | 15.9 | 9.3 | 10.4 |
Table S43. Class-averaged risk on Pascal Context (columns as in Table S42; no scene labels).
leave no calibrated name above in any store at either level, while the image-level threshold leaves 154 of the 160 calibrated ADE20K names above for CAT-Seg ViT-L. Their sets are 1.5–3.2 those of the image-level threshold at . The hybrid keeps the image risk at or below 4.9%, leaves no calibrated name above , and its sets are 1.8–3.6 those of the image-level threshold. At , 160 of the 836 names present in the ADE20K images have enough calibration images. With at most 20 calibration images per name, chosen at random, no calibrated name is above either.
Calibration-set size
We calibrate on the first images of each split, for from 10 to 1,000. At every , the anchored and the recalibrated native thresholds have similar mean risk, spread across splits and fractions of splits above (Table S46). At small both are conservative because of CRC’s finite-sample correction. On ADE20K the overhead settles within a fifth of a label of its final value from for ViT-L and for SED with the 683-name bank, and from for ViT-B with the COCO-171 bank, while for FC-CLIP both set sizes keep shrinking up to . At very small the anchored sets can be smaller than the recalibrated native ones, because the native score at 845 labels has the heavier tail.
Table S46. Risk (%) / set size at the grown vocabulary against the number of calibration images (). Evaluation on the second half of each split; calibration on the first images of the first half.
| Configuration / arm | |||||||
| CAT-Seg ViT-B, COCO-171 bank, native recal. | 2.1 / 158.5 | 6.0 / 16.6 | 8.3 / 7.0 | 9.1 / 5.8 | 9.5 / 5.3 | 9.8 / 5.1 | 9.9 / 5.0 |
| CAT-Seg ViT-B, COCO-171 bank, anchored | 2.0 / 62.8 | 6.0 / 15.2 | 8.2 / 9.5 | 9.0 / 8.3 | 9.5 / 7.7 | 9.8 / 7.4 | 9.9 / 7.3 |
| CAT-Seg ViT-L, 683-name bank, native recal. | 2.1 / 69.6 | 6.0 / 8.1 | 8.3 / 4.6 | 9.1 / 3.9 | 9.5 / 3.7 | 9.8 / 3.5 | 9.9 / 3.5 |
| CAT-Seg ViT-L, 683-name bank, anchored | 2.1 / 34.3 | 6.1 / 8.7 | 8.3 / 5.4 | 9.1 / 4.7 | 9.5 / 4.5 | 9.8 / 4.3 | 9.9 / 4.3 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, native recal. | 2.2 / 119.1 | 6.0 / 7.9 | 8.3 / 4.4 | 9.2 / 3.8 | 9.5 / 3.6 | 9.8 / 3.4 | 9.9 / 3.4 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, anchored | 2.2 / 37.1 | 6.2 / 8.9 | 8.3 / 5.5 | 9.1 / 4.8 | 9.5 / 4.6 | 9.8 / 4.4 | 9.9 / 4.4 |
| SED ConvNeXt-L, 683-name bank, native recal. | 2.4 / 87.1 | 6.1 / 18.6 | 8.3 / 7.9 | 9.1 / 6.3 | 9.5 / 5.7 | 9.8 / 5.4 | 9.9 / 5.3 |
| SED ConvNeXt-L, 683-name bank, anchored | 2.5 / 91.1 | 6.1 / 19.1 | 8.3 / 8.4 | 9.1 / 6.9 | 9.5 / 6.4 | 9.8 / 6.1 | 9.9 / 6.0 |
| FC-CLIP ConvNeXt-L, 701-name bank, native recal. | 2.3 / 143.7 | 6.0 / 28.7 | 8.3 / 11.9 | 9.1 / 9.8 | 9.5 / 8.9 | 9.8 / 8.4 | 9.9 / 8.2 |
| FC-CLIP ConvNeXt-L, 701-name bank, anchored | 2.5 / 155.2 | 6.0 / 36.0 | 8.3 / 16.0 | 9.0 / 13.2 | 9.5 / 12.1 | 9.8 / 11.4 | 9.9 / 11.1 |
| CAT-Seg ViT-B, COCO-171 bank, Pascal 59→458, native recal. | 2.2 / 17.5 | 6.1 / 4.7 | 8.4 / 3.1 | 9.2 / 2.8 | 9.6 / 2.6 | 9.8 / 2.6 | 9.9 / 2.5 |
| CAT-Seg ViT-B, COCO-171 bank, Pascal 59→458, anchored | 2.3 / 18.1 | 6.1 / 6.0 | 8.4 / 4.0 | 9.2 / 3.6 | 9.6 / 3.5 | 9.8 / 3.4 | 9.9 / 3.3 |
| CAT-Seg ViT-B, 683-name bank, Pascal 59→458, native recal. | 2.2 / 17.5 | 6.1 / 4.7 | 8.4 / 3.1 | 9.2 / 2.8 | 9.6 / 2.6 | 9.8 / 2.6 | 9.9 / 2.5 |
| CAT-Seg ViT-B, 683-name bank, Pascal 59→458, anchored | 2.3 / 16.4 | 6.1 / 5.0 | 8.3 / 3.4 | 9.2 / 3.0 | 9.6 / 2.9 | 9.8 / 2.8 | 9.9 / 2.8 |
| CAT-Seg ViT-L, 683-name bank, Pascal 59→458, native recal. | 2.3 / 10.4 | 6.2 / 3.3 | 8.4 / 2.4 | 9.2 / 2.2 | 9.6 / 2.1 | 9.8 / 2.1 | 9.9 / 2.1 |
| CAT-Seg ViT-L, 683-name bank, Pascal 59→458, anchored | 2.3 / 9.6 | 6.2 / 3.6 | 8.4 / 2.7 | 9.2 / 2.5 | 9.7 / 2.4 | 9.8 / 2.3 | 9.9 / 2.3 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, Pascal 59→458, native recal. | 2.3 / 12.9 | 6.2 / 3.6 | 8.4 / 2.5 | 9.2 / 2.3 | 9.6 / 2.2 | 9.8 / 2.1 | 9.9 / 2.1 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, Pascal 59→458, anchored | 2.4 / 10.7 | 6.1 / 3.9 | 8.4 / 2.8 | 9.2 / 2.6 | 9.6 / 2.5 | 9.8 / 2.4 | 9.9 / 2.4 |
| SED ConvNeXt-L, 683-name bank, Pascal 59→458, native recal. | 2.3 / 12.1 | 6.1 / 4.6 | 8.4 / 3.2 | 9.2 / 2.9 | 9.6 / 2.8 | 9.8 / 2.7 | 9.9 / 2.7 |
| SED ConvNeXt-L, 683-name bank, Pascal 59→458, anchored | 2.3 / 11.8 | 6.2 / 4.7 | 8.5 / 3.3 | 9.2 / 3.0 | 9.6 / 2.9 | 9.8 / 2.8 | 9.9 / 2.8 |
| FC-CLIP ConvNeXt-L, 701-name bank, Pascal 59→458, native recal. | 2.2 / 49.8 | 6.1 / 13.0 | 8.4 / 7.1 | 9.2 / 6.1 | 9.6 / 5.6 | 9.8 / 5.4 | 9.9 / 5.3 |
| FC-CLIP ConvNeXt-L, 701-name bank, Pascal 59→458, anchored | 2.1 / 48.7 | 6.0 / 14.2 | 8.5 / 7.8 | 9.2 / 6.7 | 9.6 / 6.1 | 9.8 / 5.9 | 9.9 / 5.8 |
| CAT-Seg ViT-B, 683-name bank, COCO 171→1,283, native recal. | 1.9 / 88.0 | 5.9 / 6.0 | 8.5 / 3.8 | 9.2 / 3.4 | 9.7 / 3.2 | 9.9 / 3.1 | 9.9 / 3.1 |
| CAT-Seg ViT-B, 683-name bank, COCO 171→1,283, anchored | 1.9 / 37.3 | 5.9 / 7.1 | 8.4 / 4.6 | 9.2 / 4.1 | 9.7 / 3.9 | 9.9 / 3.9 | 9.9 / 3.8 |
| CAT-Seg ViT-L, 683-name bank, COCO 171→1,283, native recal. | 2.0 / 44.6 | 5.9 / 4.8 | 8.5 / 3.2 | 9.2 / 2.9 | 9.7 / 2.7 | 9.9 / 2.7 | 9.9 / 2.6 |
| CAT-Seg ViT-L, 683-name bank, COCO 171→1,283, anchored | 2.0 / 31.6 | 5.9 / 5.6 | 8.5 / 3.8 | 9.2 / 3.5 | 9.7 / 3.3 | 9.9 / 3.2 | 9.9 / 3.2 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, COCO 171→1,283, native recal. | 2.0 / 55.2 | 5.9 / 4.8 | 8.4 / 3.1 | 9.2 / 2.8 | 9.7 / 2.7 | 9.9 / 2.6 | 9.9 / 2.6 |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, COCO 171→1,283, anchored | 2.0 / 28.8 | 5.8 / 5.6 | 8.4 / 3.8 | 9.2 / 3.4 | 9.6 / 3.3 | 9.8 / 3.2 | 9.9 / 3.2 |
| SED ConvNeXt-L, 683-name bank, COCO 171→1,283, native recal. | 1.9 / 24.1 | 5.9 / 6.4 | 8.4 / 4.4 | 9.2 / 4.0 | 9.7 / 3.8 | 9.8 / 3.8 | 9.9 / 3.7 |
| SED ConvNeXt-L, 683-name bank, COCO 171→1,283, anchored | 1.9 / 28.8 | 5.9 / 6.2 | 8.5 / 4.2 | 9.2 / 3.8 | 9.7 / 3.6 | 9.8 / 3.6 | 9.9 / 3.5 |
| FC-CLIP ConvNeXt-L, 701-name bank, COCO 171→1,283, native recal. | 2.0 / 305.0 | 6.0 / 80.1 | 8.5 / 39.3 | 9.2 / 32.3 | 9.6 / 28.7 | 9.8 / 27.2 | 9.9 / 26.5 |
| FC-CLIP ConvNeXt-L, 701-name bank, COCO 171→1,283, anchored | 1.9 / 251.6 | 6.0 / 72.4 | 8.4 / 39.4 | 9.2 / 32.7 | 9.6 / 29.3 | 9.8 / 27.6 | 9.9 / 26.8 |
Table S46.
Stricter losses
We replace the loss (1) by four stricter image losses, 1[miss 25%], 1[miss 10%], a class-balanced miss rate and the worst class in the image, and recalibrate by CRC on each. The anchored threshold is at the target in every configuration (9.84–10.06%), and the reused native threshold fails or is over-conservative under each loss, as under (1). Under the worst-class loss the anchored sets are smaller than the recalibrated native ones for every CAT-Seg configuration, but not for FC-CLIP on ADE20K (Table S47). Like (1), each of these losses counts a source-labelled pixel as covered only if its annotated name is in the set; a set that holds only a finer
| Configuration | loss | native risk | reused native recal. size | anchored risk | anchored size |
| CAT-Seg ViT-B, COCO-171 bank | pixel miss (headline) | 15.4 | 5.0 | 9.9 | 7.3 |
| 18.6 | 6.3 | 10.0 | 8.7 | ||
| 17.2 | 19.2 | 9.9 | 21.8 | ||
| class-balanced | 15.3 | 14.1 | 9.9 | 14.1 | |
| worst class in image | 25.0 | 615.2 | 9.9 | 102.9 | |
| CAT-Seg ViT-L, COCO-171 bank | pixel miss (headline) | 15.2 | 3.5 | 9.9 | 5.1 |
| 18.0 | 4.1 | 9.9 | 5.8 | ||
| 18.8 | 10.4 | 9.9 | 11.9 | ||
| class-balanced | 15.3 | 7.4 | 9.9 | 7.8 | |
| worst class in image | 22.4 | 77.0 | 9.9 | 57.3 | |
| CAT-Seg ViT-L, 683-name bank | pixel miss (headline) | 15.2 | 3.5 | 9.9 | 4.3 |
| 18.0 | 4.1 | 10.0 | 5.0 | ||
| 18.8 | 10.4 | 9.9 | 11.1 | ||
| class-balanced | 15.3 | 7.4 | 9.9 | 7.8 | |
| worst class in image | 22.4 | 77.0 | 9.9 | 62.7 | |
| CAT-Seg + DeCLIP EVA-L, 683-name bank | pixel miss (headline) | 14.7 | 3.4 | 9.9 | 4.4 |
| 17.1 | 4.1 | 10.0 | 5.4 | ||
| 17.7 | 10.4 | 9.9 | 12.1 | ||
| class-balanced | 14.3 | 6.2 | 9.9 | 7.0 | |
| worst class in image | 19.1 | 60.1 | 9.9 | 51.3 | |
| SED ConvNeXt-L, COCO-171 bank | pixel miss (headline) | 7.1 | 5.3 | 9.9 | 6.9 |
| 8.3 | 6.6 | 10.0 | 8.6 | ||
| 8.1 | 25.9 | 9.9 | 22.9 | ||
| class-balanced | 7.2 | 15.8 | 9.9 | 13.3 | |
| worst class in image | 4.9 | 134.6 | 9.9 | 116.7 | |
| SED ConvNeXt-L, fast mode (top-8 rejection) | pixel miss (headline) | 12.8 | 5.2 | 9.9 | 6.9 |
| 14.4 | 7.3 | 10.0 | 8.6 | ||
| 14.1 | 648.3 | 9.9 | 22.9 | ||
| class-balanced | 14.7 | 845.0 | 9.9 | 13.3 | |
| worst class in image | 0.0 | 845.0 | 9.9 | 116.7 | |
| SED ConvNeXt-L, 683-name bank | pixel miss (headline) | 7.1 | 5.3 | 9.9 | 6.0 |
| 8.3 | 6.6 | 10.0 | 7.4 | ||
| 8.1 | 25.9 | 9.9 | 23.4 | ||
| class-balanced | 7.2 | 15.8 | 9.9 | 15.0 | |
| worst class in image | 4.9 | 134.6 | 9.9 | 138.3 | |
| FC-CLIP ConvNeXt-L, 133-name bank | pixel miss (headline) | 13.8 | 8.2 | 9.9 | 13.5 |
| 16.1 | 9.4 | 10.0 | 16.5 | ||
| 13.0 | 42.9 | 10.0 | 53.4 | ||
| class-balanced | 13.4 | 25.0 | 9.9 | 35.0 | |
| worst class in image | 15.9 | 211.2 | 9.9 | 243.4 | |
| FC-CLIP ConvNeXt-L, 701-name bank | pixel miss (headline) | 13.8 | 8.2 | 9.9 | 11.1 |
| 16.1 | 9.4 | 10.0 | 13.4 | ||
| 13.0 | 42.9 | 10.0 | 48.5 | ||
| class-balanced | 13.4 | 25.0 | 9.9 | 29.2 | |
| worst class in image | 15.9 | 211.2 | 9.9 | 222.9 | |
| CAT-Seg ViT-B, COCO-171 bank, Pascal 59458 | pixel miss (headline) | 15.2 | 2.5 | 9.9 | 3.3 |
| 17.4 | 3.2 | 9.9 | 4.4 | ||
| 19.7 | 5.6 | 9.9 | 7.2 | ||
| class-balanced | 15.7 | 3.6 | 9.9 | 4.0 | |
| worst class in image | 20.1 | 16.1 | 9.9 | 15.4 |
Table S47. Risk (%) and set size under five image losses (); thresholds are recalibrated by CRC on each loss. Risk 0.0 with 845.0 labels: CRC returned the all-label endpoint, as no threshold on the grid meets the target.
added name for the same object misses it. Pixels of added classes enter only the added-label risk (§ 6.3).
Set constructions
Proposition 1 requires the set construction, and not only the score, to be candidate-independent. With the per-label threshold, the anchored thresholds calibrated at the source and at the grown vocabulary coincide in every configuration, as the proposition predicts. Constructions that normalise over the requested vocabulary remain candidate-dependent and fail even on anchored scores, which is why the main text uses the per-label threshold [65]: the softmax cutoff has 12.2–17.4% risk, RAPS [3] up to 12.2% and APS [64] 6.0–11.4% (Table S49).
| Configuration | construction | native reused risk | native recal. size | anchored reused risk | anchored reused size | anchored recal. size |
| CAT-Seg ViT-B, COCO-171 bank | per-label threshold (ours) | 15.4 | 4.98 | 9.9 | 7.30 | 7.30 |
| softmax cutoff | 14.6 | 5.56 | 17.0 | 3.02 | 6.39 | |
| APS | 11.8 | 8.43 | 8.4 | 10.73 | 8.56 | |
| RAPS | 12.8 | 5.77 | 10.0 | 7.17 | 7.26 | |
| CAT-Seg ViT-L, COCO-171 bank | per-label threshold (ours) | 15.2 | 3.47 | 9.9 | 5.08 | 5.08 |
| softmax cutoff | 14.4 | 3.76 | 17.0 | 2.36 | 4.40 | |
| APS | 12.3 | 5.04 | 9.2 | 6.15 | 5.57 | |
| RAPS | 12.8 | 4.03 | 10.0 | 4.93 | 4.99 | |
| CAT-Seg ViT-L, 683-name bank | per-label threshold (ours) | 15.2 | 3.47 | 9.9 | 4.27 | 4.27 |
| softmax cutoff | 14.4 | 3.76 | 14.8 | 2.50 | 4.22 | |
| APS | 12.3 | 5.04 | 10.2 | 6.72 | 7.00 | |
| RAPS | 12.8 | 4.03 | 11.2 | 4.52 | 5.14 | |
| CAT-Seg + DeCLIP EVA-L, 683-name bank | per-label threshold (ours) | 14.7 | 3.38 | 9.9 | 4.37 | 4.37 |
| softmax cutoff | 14.0 | 3.64 | 14.7 | 2.48 | 4.15 | |
| APS | 11.6 | 4.97 | 10.1 | 6.61 | 6.81 | |
| RAPS | 12.2 | 3.94 | 11.1 | 4.39 | 4.93 | |
| SED ConvNeXt-L, COCO-171 bank | per-label threshold (ours) | 7.1 | 5.27 | 9.9 | 6.93 | 6.93 |
| softmax cutoff | 14.0 | 6.06 | 16.0 | 3.16 | 6.20 | |
| APS | 11.0 | 9.49 | 9.7 | 9.85 | 9.52 | |
| RAPS | 12.3 | 5.71 | 11.0 | 6.22 | 7.18 | |
| SED ConvNeXt-L, fast mode (top-8 rejection) | per-label threshold (ours) | 12.8 | 5.18 | 9.9 | 6.93 | 6.93 |
| softmax cutoff | 14.8 | 5.05 | 16.0 | 3.16 | 6.20 | |
| APS | 13.2 | 5.44 | 9.7 | 9.85 | 9.52 | |
| RAPS | 13.7 | 5.18 | 11.0 | 6.22 | 7.18 | |
| SED ConvNeXt-L, 683-name bank | per-label threshold (ours) | 7.1 | 5.27 | 9.9 | 6.04 | 6.04 |
| softmax cutoff | 14.0 | 6.06 | 14.1 | 3.54 | 6.34 | |
| APS | 11.0 | 9.49 | 10.5 | 9.51 | 10.58 | |
| RAPS | 12.3 | 5.71 | 12.0 | 5.21 | 6.41 | |
| FC-CLIP ConvNeXt-L, 133-name bank | per-label threshold (ours) | 13.8 | 8.17 | 9.9 | 13.49 | 13.49 |
| softmax cutoff | 17.2 | 9.10 | 16.3 | 3.31 | 8.15 | |
| APS | 10.9 | 23.77 | 7.7 | 25.84 | 16.86 | |
| RAPS | 12.6 | 11.36 | 9.8 | 11.01 | 10.81 | |
| FC-CLIP ConvNeXt-L, 701-name bank | per-label threshold (ours) | 13.8 | 8.17 | 9.9 | 11.14 | 11.14 |
| softmax cutoff | 17.2 | 9.10 | 16.0 | 4.03 | 8.80 | |
| APS | 10.9 | 23.77 | 8.4 | 23.39 | 17.48 | |
| RAPS | 12.6 | 11.36 | 10.6 | 10.01 | 10.90 | |
| CAT-Seg ViT-B, COCO-171 bank, Pascal 59458 | per-label threshold (ours) | 15.2 | 2.52 | 9.9 | 3.35 | 3.35 |
| softmax cutoff | 15.2 | 2.71 | 16.0 | 1.75 | 2.90 | |
| APS | 12.7 | 3.71 | 9.3 | 4.21 | 3.90 | |
| RAPS | 13.1 | 3.03 | 9.9 | 3.48 | 3.49 |
Table S49. Four set constructions on the same scores (; RAPS with , ). The anchored per-label threshold is the only construction whose reused and recalibrated sizes coincide.
Deployments with growing vocabularies
This section describes deployments in which the vocabulary changes after calibration and the guarantee that applies to each. Table S36 compares calibrating once with recalibrating in each case.
Robots asked for new objects. OK-Robot builds its memory of a home with an open-vocabulary detector run on a fixed list of ScanNet200 names and later answers requests for arbitrary objects [44]. HomeRobot sets the detector’s class list for each skill and episode, and its goal objects include unseen categories [88]. VLMaps receives its list of obstacle categories at run time [33]. The anchored threshold keeps the source-label guarantee (Proposition 1) for the classes calibrated before deployment, whatever names are requested later.
Names chosen per image. Image taggers, captioners and language models choose the names from the test image [29, 57, 60, 62, 90]. Such a rule cannot be replayed on the calibration images unless it is fixed and known; the anchored threshold keeps the source-label guarantee under any rule (Proposition 1 and Sec. 6.2).
Catalogues that grow. The iNaturalist species benchmark grew from about 5,000 species [75] to 10,000 [76], and taxonomic names “are added, removed, and changed as biologists discover and classify new and existing species” [70]. Anomaly-segmentation benchmarks for driving state that networks trained on a fixed set of classes are “ill-equipped to handle previously-unseen objects” [14]. When the annotated classes are a random sample of the catalogue, label-level calibration bounds the miss rate of the others (Proposition 3); when the added classes are harder, the gap bound applies (Proposition 4).
Scope. Open-vocabulary models are less accurate than supervised ones in these domains: on the MESS benchmark, zero-shot CAT-Seg reaches about half of the supervised mIoU in earth monitoring and medicine, and less in engineering and agriculture [10]. The guarantee bounds the miss rate at any accuracy, so lower accuracy appears as larger sets rather than more misses. The risk is pooled over the pixels and images of the calibration distribution. Classes that need a bound of their own, such as hazards, require class-conditional calibration (Sec. K.1), and the images seen in deployment must be exchangeable with the calibration images.
| Configuration | loss | native reused risk | native recal. size | anchored risk | anchored size |
| CAT-Seg ViT-B, 683-name bank, Pascal 59→458 | pixel miss (headline) | 15.2 | 2.5 | 9.9 | 2.8 |
| 17.4 | 3.2 | 10.0 | 3.4 | ||
| 19.7 | 5.6 | 10.0 | 5.9 | ||
| class-balanced | 15.7 | 3.6 | 9.9 | 3.9 | |
| worst class in image | 20.1 | 16.1 | 9.9 | 15.6 | |
| CAT-Seg ViT-L, 683-name bank, Pascal 59→458 | pixel miss (headline) | 15.0 | 2.1 | 9.9 | 2.3 |
| 17.2 | 2.5 | 9.9 | 2.7 | ||
| 19.3 | 4.0 | 10.1 | 4.3 | ||
| class-balanced | 15.5 | 2.6 | 9.9 | 2.8 | |
| worst class in image | 18.9 | 10.4 | 9.9 | 9.8 | |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, Pascal 59→458 | pixel miss (headline) | 14.7 | 2.1 | 9.9 | 2.4 |
| 16.8 | 2.7 | 9.9 | 3.0 | ||
| 17.9 | 4.4 | 10.0 | 4.7 | ||
| class-balanced | 15.0 | 2.6 | 9.9 | 2.8 | |
| worst class in image | 18.3 | 9.2 | 9.9 | 9.0 | |
| SED ConvNeXt-L, 683-name bank, Pascal 59→458 | pixel miss (headline) | 4.7 | 2.7 | 9.9 | 2.8 |
| 3.9 | 3.3 | 10.1 | 3.5 | ||
| 3.9 | 5.5 | 10.0 | 5.7 | ||
| class-balanced | 5.0 | 3.9 | 9.9 | 4.0 | |
| worst class in image | 4.5 | 14.9 | 9.9 | 14.8 | |
| FC-CLIP ConvNeXt-L, 701-name bank, Pascal 59→458 | pixel miss (headline) | 16.2 | 5.3 | 9.9 | 5.8 |
| 18.3 | 7.2 | 9.9 | 8.2 | ||
| 18.0 | 16.7 | 10.0 | 17.4 | ||
| class-balanced | 16.5 | 9.2 | 9.9 | 10.3 | |
| worst class in image | 19.5 | 61.3 | 9.9 | 59.7 | |
| CAT-Seg ViT-B, 683-name bank, COCO 171→1,283 | pixel miss (headline) | 13.0 | 3.1 | 9.9 | 3.8 |
| 14.1 | 3.3 | 10.0 | 3.9 | ||
| 15.0 | 7.1 | 10.0 | 8.0 | ||
| class-balanced | 15.4 | 7.4 | 9.9 | 8.1 | |
| worst class in image | 21.9 | 54.2 | 10.0 | 45.5 | |
| CAT-Seg ViT-L, 683-name bank, COCO 171→1,283 | pixel miss (headline) | 12.9 | 2.6 | 9.9 | 3.2 |
| 13.9 | 2.9 | 10.0 | 3.4 | ||
| 15.6 | 5.8 | 10.0 | 6.6 | ||
| class-balanced | 15.3 | 5.7 | 9.9 | 5.9 | |
| worst class in image | 21.4 | 35.6 | 9.9 | 31.8 | |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, COCO 171→1,283 | pixel miss (headline) | 12.9 | 2.6 | 9.9 | 3.2 |
| 13.9 | 2.9 | 10.0 | 3.5 | ||
| 14.6 | 6.0 | 10.0 | 6.8 | ||
| class-balanced | 14.8 | 5.2 | 9.9 | 5.7 | |
| worst class in image | 19.9 | 31.2 | 9.9 | 28.5 | |
| SED ConvNeXt-L, 683-name bank, COCO 171→1,283 | pixel miss (headline) | 3.6 | 3.7 | 9.9 | 3.5 |
| 2.8 | 3.9 | 10.0 | 3.7 | ||
| 2.8 | 7.7 | 10.0 | 7.2 | ||
| class-balanced | 4.3 | 8.2 | 9.9 | 7.5 | |
| worst class in image | 4.8 | 47.9 | 10.0 | 45.3 | |
| FC-CLIP ConvNeXt-L, 701-name bank, COCO 171→1,283 | pixel miss (headline) | 17.1 | 26.5 | 9.9 | 26.8 |
| 22.6 | 36.8 | 9.9 | 38.8 | ||
| 24.4 | 120.7 | 9.9 | 112.0 | ||
| class-balanced | 17.4 | 81.8 | 9.9 | 72.9 | |
| worst class in image | 23.0 | 518.4 | 9.8 | 463.9 |
Table S48. Table S48. Table S47, continued (part 2 of 2).
| Configuration | construction | native | anchored | |||
| reused risk | recal. size | reused risk | reused size | recal. size | ||
| CAT-Seg ViT-B, 683-name bank, Pascal 59→458 | per-label threshold (ours) | 15.2 | 2.52 | 9.9 | 2.81 | 2.81 |
| softmax cutoff | 15.2 | 2.71 | 15.0 | 1.78 | 2.85 | |
| APS | 12.7 | 3.71 | 10.4 | 3.84 | 4.14 | |
| RAPS | 13.1 | 3.03 | 11.1 | 3.03 | 3.41 | |
| CAT-Seg ViT-L, 683-name bank, Pascal 59→458 | per-label threshold (ours) | 15.0 | 2.07 | 9.9 | 2.33 | 2.33 |
| softmax cutoff | 14.8 | 2.18 | 14.9 | 1.58 | 2.28 | |
| APS | 13.0 | 2.67 | 11.4 | 2.52 | 3.05 | |
| RAPS | 13.2 | 2.38 | 11.8 | 2.24 | 2.64 | |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, Pascal 59→458 | per-label threshold (ours) | 14.7 | 2.09 | 9.9 | 2.41 | 2.41 |
| softmax cutoff | 14.8 | 2.20 | 14.5 | 1.62 | 2.36 | |
| APS | 13.2 | 2.73 | 11.3 | 2.63 | 3.20 | |
| RAPS | 13.4 | 2.37 | 11.7 | 2.27 | 2.69 | |
| SED ConvNeXt-L, 683-name bank, Pascal 59→458 | per-label threshold (ours) | 4.7 | 2.67 | 9.9 | 2.79 | 2.79 |
| softmax cutoff | 16.5 | 2.88 | 14.0 | 1.88 | 2.77 | |
| APS | 14.7 | 3.91 | 11.4 | 3.19 | 3.94 | |
| RAPS | 14.9 | 3.01 | 11.9 | 2.57 | 3.06 | |
| FC-CLIP ConvNeXt-L, 701-name bank, Pascal 59→458 | per-label threshold (ours) | 16.2 | 5.31 | 9.9 | 5.77 | 5.77 |
| softmax cutoff | 18.6 | 5.76 | 17.4 | 2.06 | 4.57 | |
| APS | 9.4 | 15.10 | 6.9 | 17.68 | 10.62 | |
| RAPS | 11.1 | 8.41 | 9.0 | 7.55 | 6.80 | |
| CAT-Seg ViT-B, 683-name bank, COCO 171→1,283 | per-label threshold (ours) | 13.0 | 3.09 | 9.9 | 3.83 | 3.83 |
| softmax cutoff | 12.6 | 3.35 | 12.7 | 2.69 | 3.48 | |
| APS | 10.6 | 4.97 | 8.7 | 6.99 | 5.85 | |
| RAPS | 11.2 | 3.85 | 9.6 | 4.53 | 4.39 | |
| CAT-Seg ViT-L, 683-name bank, COCO 171→1,283 | per-label threshold (ours) | 12.9 | 2.65 | 9.9 | 3.21 | 3.21 |
| softmax cutoff | 12.4 | 2.88 | 12.2 | 2.46 | 3.01 | |
| APS | 11.3 | 3.78 | 9.4 | 4.87 | 4.52 | |
| RAPS | 11.6 | 3.21 | 10.0 | 3.62 | 3.64 | |
| CAT-Seg + DeCLIP EVA-L, 683-name bank, COCO 171→1,283 | per-label threshold (ours) | 12.9 | 2.63 | 9.9 | 3.19 | 3.19 |
| softmax cutoff | 12.8 | 2.79 | 12.5 | 2.40 | 2.96 | |
| APS | 11.2 | 3.81 | 9.6 | 4.79 | 4.56 | |
| RAPS | 11.6 | 3.16 | 10.2 | 3.52 | 3.58 | |
| SED ConvNeXt-L, 683-name bank, COCO 171→1,283 | per-label threshold (ours) | 3.6 | 3.74 | 9.9 | 3.55 | 3.55 |
| softmax cutoff | 10.5 | 3.39 | 12.5 | 2.67 | 3.30 | |
| APS | 8.1 | 4.41 | 10.0 | 4.54 | 4.53 | |
| RAPS | 8.9 | 3.70 | 10.6 | 3.52 | 3.70 | |
| FC-CLIP ConvNeXt-L, 701-name bank, COCO 171→1,283 | per-label threshold (ours) | 17.1 | 26.53 | 9.9 | 26.85 | 26.85 |
| softmax cutoff | 18.0 | 32.08 | 15.5 | 9.47 | 25.73 | |
| APS | 9.8 | 74.90 | 6.0 | 92.28 | 49.83 | |
| RAPS | 14.7 | 27.27 | 12.2 | 17.67 | 25.40 |
Table S50. Table S49, continued (part 2 of 2).
References
- [1]Léo Andréol, Thomas Fel, Florence de Grancey, and Luca Mossina. Confident object detection via conformal prediction and conformal risk control: an application to railway signaling. In COPA, pages 36–55, 2023. 2arxiv.org/abs/2304.06052
- [2]Anastasios N. Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023. 1
- [3]Anastasios N. Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I. Jordan. Uncertainty sets for image classifiers using conformal prediction. In ICLR, 2021. 3, 40arxiv.org/abs/2009.14193
- [4]Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In ICLR, 2024. 1, 2, 3, 4
- [5]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. Annals of Applied Statistics, 2025. 2arxiv.org/abs/2110.01052
- [6]Roland Auckenthaler, Michael Carey, and Harvey Lloyd-Thomas. Score normalization for text-independent speaker verification systems. Digital Signal Processing, 10(1–3):42–54, 2000. 3
- [7]Yajie Bao, Yuyang Huo, Haojie Ren, and Changliang Zou. Selective conformal inference with false coverage-statement rate control. Biometrika, 111(3):727–742, 2024. 3DOI
- [8]Stephen Bates, Anastasios N. Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6), 2021. 3, 4, 33arxiv.org/abs/2101.02703
- [9]Vincent Blot, Anastasios N. Angelopoulos, Michael I. Jordan, and Nicolas J-B. Brunel. Automatically adaptive conformal risk control. In AISTATS, 2025. 2arxiv.org/abs/2406.17819
- [10]Benedikt Blumenstiel, Johannes Jakubik, Hilde Kühne, and Michael Vössing. What a MFSS: Multi-domain evaluation of zero-shot semantic segmentation. In NeurIPS Datasets and Benchmarks, 2023. 40
- [11]Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. In CVPR, 2022. 3arxiv.org/abs/2112.12777
- [12]Joren Brunekreef, Eric Marcus, Ray Sheombarsing, Jan-Jakob Sonke, and Jonas Teuwen. Kandinsky conformal prediction: Efficient calibration of image segmentation algorithms. In CVPR, 2024. 1, 2arxiv.org/abs/2311.11837
- [13]Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018. 2, 6, 14arxiv.org/abs/1612.03716
- [14]Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, and Matthias Rottmann. SegmentMeIfYouCan: A benchmark for anomaly segmentation. In NeurIPS Datasets and Benchmarks, 2021. 40arxiv.org/abs/2104.14812
- [15]Qi Chen, Lingxiao Yang, Yun Chen, Nailong Zhao, Jianhuang Lai, Jie Shao, and Xiaohua Xie. Training-free class purification for open-vocabulary semantic segmentation. In ICCV, 2025. 3arxiv.org/abs/2508.00557
- [16]Ziliang Chen, Yulu Li, Liangda Fang, Jusheng Zhang, Yongsen Zheng, Quanlong Guan, and Xipeng Chen. Vocabulary scaling law: Tuning open-vocabulary predictors for their openness. In CVPR, 2026. 3
- [17]Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuk Seo, and Seunggryong Kim. CAT-Seg: Cost aggregation for open-vocabulary semantic segmentation. In CVPR, 2024. 1, 4arxiv.org/abs/2303.11797
- [18]Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. Vocabulary-free image classification. In NeurIPS, 2023. 1arxiv.org/abs/2306.00917
- [19]Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 6arxiv.org/abs/1604.01685
- [20]Tiffany Ding, Anastasios N. Angelopoulos, Stephen Bates, Michael I. Jordan, and Ryan J. Tibshirani. Class-conditional conformal prediction with many classes. In NeurIPS, 2023. 3, 36arxiv.org/abs/2306.09335
- [21]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 6
- [22]Dominik Engel, Yousef Rajeh, Leon Sick, and Ivan Viola. Query-time calibration of CLIP features for open-vocabulary 3D segmentation. arXiv preprint arXiv:2610.30908, 2026. 3arxiv.org/abs/2610.30908
- [23]Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. Image and Vision Computing, 149, 2024. 6DOI
- [24]António Farinhas, Chrysoula Zerva, Dennis Ulmer, and André F. T. Martins. Non-exchangeable conformal risk control. In ICLR, 2024. 3arxiv.org/abs/2310.01262
- [25]Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Few-shot conformal prediction with auxiliary tasks. In ICML, 2021. 3arxiv.org/abs/2102.08898
- [26]Golnaz Ghiasi, Xiuyue Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, 2022. 1arxiv.org/abs/2112.12143
- [27]Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B, 87(4):1100–1126, 2025. 8arxiv.org/abs/2305.12616
- [28]Gianluca Giuffrida, Luca Fanucci, Gabriele Meoni, Matej Batič, Léonie Buckley, Aubrey Dunne, Chris van Dijk, Marco Esposito, John Hefele, Nathan Vercuyssen, Gianluca Furano, Massimiliano Pastena, and Josef Aschbacher. The Φ-Sat-1 mission: The first on-board deep neural network demonstrator for satellite earth observation. IEEE Transactions on Geoscience and Remote Sensing, 60:1–14, 2022. 33DOI
- [29]Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull. ConceptGraphs: Open-vocabulary 3D scene graphs for perception and planning. In ICRA, 2024. 1, 9, 40arxiv.org/abs/2309.16650
- [30]Agrim Gupta, Piotr Dollár, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 2, 6, 14arxiv.org/abs/1908.03195
- [31]Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023. 1arxiv.org/abs/2211.11559
- [32]Mahmoud Hegazy, Líviu Aolaritei, Michael I. Jordan, and Aymeric Dieuleveut. Valid selection among conformal sets. In NeurIPS, 2025. 3arxiv.org/abs/2506.20173
- [33]Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In ICRA, 2023. 1, 40arxiv.org/abs/2210.05714
- [34]Haiwen Huang, Songyou Peng, Dan Zhang, and Andreas Geiger. Renovating names in open-vocabulary segmentation benchmarks. In NeurIPS, 2024. 3arxiv.org/abs/2403.09593
- [35]Qiming Huang, Han Hu, and Jianbo Jiao. Revisit the open nature of open vocabulary semantic segmentation. In ICLR, 2025. 3
- [36]Bron Hultberg, Dave Zachariah, and António H. Ribeiro. Anytime-valid conformal risk control. In NeurIPS, 2026. 3arxiv.org/abs/2602.04364
- [37]Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In ICLR, 2022. 3
- [38]Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided OOD detection with pretrained vision-language models. In ICLR, 2024. 3
- [39]Ying Jin and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. Journal of the Royal Statistical Society Series B, 87(4):1239–1259, 2025. 1, 3, 4, 30arxiv.org/abs/2403.03868
- [40]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In ICML, 2020. 4arxiv.org/abs/2006.16236
- [41]Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Mallocci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 128:1956–1981, 2020. 5, 6DOI
- [42]Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. In ICLR, 2022. 1arxiv.org/abs/2201.03546
- [43]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 8arxiv.org/abs/1405.0312
- [44]Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Demonstrating OK-Robot: What really matters in integrating open-knowledge models for robotics. In RSS, 2024. 1, 40
- [45]Shilong Liu, Hao Cheng, Haotian Lian, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. LLaVA-Plus: Learning to use tools for creating multimodal agents. In ECCV, 2024. 1arxiv.org/abs/2311.05437
- [46]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In ECCV, 2024. 8, 34
- [47]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 4
- [48]Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 4
- [49]Timo Lüdd(e)cke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, 2022. 2, 3, 18arxiv.org/abs/2112.10003
- [50]Rui Luo, Jie Bao, Xiaoyi Su, Wen Jung Li, and Suqun Cao. Enhancing image-conditional coverage in segmentation: Adaptive thresholding via differentiable miscoverage loss. In ICLR, 2026. 2
- [51]Dimity Milbich, Nikó Sünderhauf, Alex Kenna, and Keita Mason. Open-set recognition in the age of vision-language models. In ECCV, 2024. 3
- [52]George A. Miller. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41, 1995. 8, 27
- [53]Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In NeurIPS, 2023. 8, 34arxiv.org/abs/2306.09683
- [54]Luca Mossina, Joseba Dalmau, and Léo Andéol. Conformal semantic image segmentation: Post-hoc quantification of predictive uncertainty. In CVPR Workshops, 2024. 1, 2arxiv.org/abs/2405.05145
- [55]Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014. 2, 6
- [56]Jeffery Opoku and David Banahene. PromptShift-CRC: What drift-aware weighting adds to online conformal risk control for foundation models. arXiv preprint arXiv:2606.15964, 2026.
- [57]Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Leiwei Yao, Jianhua Han, Hang Xu, Linpeng Kong, and Tong Zhang. DetGPT: Detect what you need via reasoning. In EMNLP, 2023. 1, 40arxiv.org/abs/2305.14167
- [58]Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In UAI, 2021. 3arxiv.org/abs/2103.03323
- [59]Yuhao Qing, Yueying Wang, Chaoyang Chen, Weidong Zhang, Jie Wen, and Xin Xu. S2C2Seg: Semantic-spatial consistency and category optimization for open-vocabulary segmentation. In CVPR, 2026. 3
- [60]Klara Reichardt, Giulia Rizzoli, Stefano Gasperini, Lukas Hoyer, Pietro Zanuttigh, Nassir Navab, and Federico Tombari. From open-vocabulary to vocabulary-free semantic segmentation. Pattern Recognition Letters, 198:14–21, 2025. 1, 9, 40
- [61]Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. Delving into the openness of CLIP. In Findings of ACL, 2023. 3arxiv.org/abs/2206.01986
- [62]Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded SAM: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024. 1, 9, 40arxiv.org/abs/2401.14159
- [63]Douglas A. Reynolds, Thomas F. Quatieri, and Robert B. Dunn. Speaker verification using adapted Gaussian mixture models. Digital Signal Processing, 10(1–3):19–41, 2000. 3DOI
- [64]Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In NeurIPS, 2020. 3, 40arxiv.org/abs/2006.02544
- [65]Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223–234, 2019. 3, 40
- [66]Josip Šarić, Ivan Martinović, Matej Kristan, and Sinisa Šegvić. What holds back open-vocabulary segmentation? In ICCV Workshops, 2023. 3, 4
- [67]Jai Kumar Sharma and Amartya Dutta. Does marginal coverage guarantee class-conditional safety for zero-shot VLMs under shift? In ECCV Workshops (UNCV), 2026. 2, 4arxiv.org/abs/2608.19376
- [68]Wenwen Si, Sangdon Park, Insup Lee, Edgar Dobriban, and Osbert Bastani. PAC prediction sets under label shift. In ICLR, 2024. 3arxiv.org/abs/2310.12964
- [69]Julio Silva-Rodriguez, Ismail Ben Ayed, and Jose Dolz. Conformal prediction for zero-shot models. In CVPR, 2025. 2
- [70]Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su. BioCLIP: A vision foundation model for the tree of life. In CVPR, 2024. 1, 40arxiv.org/abs/2311.18803
- [71]R. Michael Swan, Deegan Atha, Henry A. Leopold, Matthew Gildner, Stephanie Oji, Cindy Chiu, and Masahiro Ono. AI4MARS: A dataset for terrain-aware autonomous driving on Mars. In CVPR Workshops, pages 1982–1991, 2021. 33DOI
- [72]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In NeurIPS, 2019. 3, 25arxiv.org/abs/1904.06019
- [73]Alexander Timans, Christoph-Nikolas Straehle, Kaspar Sakmann, and Eric Nalisnick. Adaptive bounding box uncertainties via two-step conformal prediction. In ECCV, 2024. 2arxiv.org/abs/2403.07263
- [74]Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating vision-language models. In ICML, 2024. 2arxiv.org/abs/2402.07417
- [75]Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018. 1, 40arxiv.org/abs/1707.06642
- [76]Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In CVPR, 2021. 1, 40arxiv.org/abs/2103.16483
- [77]Bruno Viti, Elias Karabelas, and Martin Holler. CONSIGN: Conformal segmentation informed by spatial groupings via decomposition. In ICLR, 2026. 2arxiv.org/abs/2505.14113
- [78]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005. 1, 36
- [79]Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking self-attention for dense vision-language inference. In ECCV, 2024. 7, 18arxiv.org/abs/2312.01597
- [80]Hanyao Wang, Yibing Zhan, Liu Liu, Liang Ding, Yan Yang, and Jun Yu. Towards alleviating hallucination in text-to-image retrieval for CLIP in zero-shot learning. Neurocomputing, 671:132640, 2026. 3
- [81]Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. DeCLIP: Decoupled learning for open-vocabulary dense perception. In CVPR, 2025. 2, 6arxiv.org/abs/2505.04410
- [82]Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for fine-tuned CLIP. In ICML, 2024. 2arxiv.org/abs/2402.04655
- [83]Shuqi Wang, Yiwei Zheng, Pingrui Zhang, Yue Gao, and Xiang Yin. CoSMO: Conformal semantic mapping for open-world safe robot navigation. In CoRL, 2026. 3arxiv.org/abs/2610.18321
- [84]Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. SED: A simple encoder-decoder for open-vocabulary semantic segmentation. In CVPR, 2024. 1, 4
- [85]Tianmin Xie, Ziyi Liang, Stefano Favaro, and Matteo Sesia. Conformal classification with new labels. In COPA, pages 765–767, 2025. 3
- [86]Tianmin Xie, Yanfei Zhou, Ziyi Liang, Stefano Favaro, and Matteo Sesia. Conformal inference for open-set and imbalanced classification. arXiv preprint arXiv:2510.13037, 2025. 3arxiv.org/abs/2510.13037
- [87]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuany Li, and Jiangfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441, 2023. 16arxiv.org/abs/2310.11441
- [88]Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, and Chris Paxton. HomeRobot: Open-vocabulary mobile manipulation. In CoRL, 2023. 1, 40arxiv.org/abs/2306.11565
- [89]Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional CLIP. In NeurIPS, 2023. 1, 4arxiv.org/abs/2308.02487
- [90]Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, Yandong Guo, and Lei Zhang. Recognize anything: A strong image tagging model. In CVPR Workshops, 2024. 1, 18, 40arxiv.org/abs/2306.03514
- [91]Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361, 2024. 8, 34arxiv.org/abs/2401.02361
- [92]Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017. 2, 6
- [93]Muzhi Zhu, Hengtao Li, Hao Chen, Chengxiang Fan, Weian Mao, Chenchen Jing, Yifan Liu, and Chunhua Shen. Seg-Prompt: Boosting open-world segmentation via category-level prompt learning. In ICCV, 2023. 3DOI






