betaFeedback

WorksComputer Science

Adapting Frozen Promptable Segmenters to User-Defined Taxonomies via Few-Shot Proposal Learning

Abstract

Promptable segmenters of the SAM family turn a click or a phrase into an instance mask, so the costly part of a label is now its class. However, each annotation project draws its own class boundary, often through objects these models already know: people from cars, sedans from trucks, one sedan model from another. A generic model's boundary is fixed by its training vocabulary. We present SNAP, an annotation pipeline that learns the project's boundary from the labels the annotator draws anyway. On one frozen SAM2.1 pass per image, we fit a per-project head, SnapHead, that classifies every proposal and refits in seconds after each labeled image. Where SAM 3 is available, the same labels decide class by class whether to keep its boundary or ours. Given the true masks of eight look-alike cell lines, SAM 3 assigns classes at Cohen's kappa = 0.038 and SnapHead at 0.754 (60 labeled images), and when only the class names change, SAM 3 falls to chance and SnapHead's score is unchanged. End to end, with 20 labeled images, SNAP beats a SAM 3 oracle tuned on the evaluation labels on five microscopy projects (0.484 against 0.441 F1), and run unchanged on five held-out datasets, it beats SAM 3 as shipped on four. Our results suggest that frozen foundation features already encode the class boundaries a generic vocabulary misses, and that a project's own labels are enough to read them out.

Introduction

Instance masks with class labels are the slowest part of building a vision dataset. Promptable segmenters such as SAM, SAM2 and SAM 3 [13, 33, 52] made the mask itself cheap: a click, a box or a short phrase returns a mask with no training on the project. A label, however, is a mask plus a class, and the class is now the costly half. Annotation tools

put one public checkpoint behind every prompt [18, 56, 69], so the class comes either from the annotator, one object at a time, or from the model’s own training vocabulary. In the first case, cost still grows with the number of objects; in the second, the classes are limited to the model’s training vocabulary.

Where a project draws its class line depends on the project. Separating people from cars is a cut SAM 3 already makes. Sedans versus trucks and buses is finer, one sedan model against another is finer still, and one brand against another is finer again. The annotator knows exactly which cut they want; a generic model was never trained to make most of them. Our key observation is that an annotation project rarely brings new objects; it brings a new boundary between objects a model has already seen (Figure 1). The gap persists when we supply the correct masks and ask for the class alone: SAM 3 tells common animal species apart almost perfectly, yet on eight look-alike cell lines it does no better than always naming the most common line. Better words do not close the gap: SAM 3 finds nuclei from “bright spots” but almost none from “nucleus”, and without labels only the answer key tells which phrase works (§ 4.4). A failed prompt gives the model nothing to learn from, while every label the annotator draws is a training example.

Figure 1. A generic segmenter has a fixed class boundary, but every project needs its own. Two annotation projects over the same scene, colored by class. SAM 3’s boundary (orange) is fixed by its training vocabulary and stays at the coarse person/car cut. A boundary learned from a project’s own labels (blue) moves to the finer sedan/truck/bus cut that project needs.

A project can already get its own boundary, at a price. Fine-tuning a detector on the project’s labels [29, 42], or retraining a network each round as labels arrive [20], fits any cut, but every project, and often every round, pays for a full training run. Pixel classifiers on frozen features [10, 59, 62] train in seconds, but they label pixels, so the annotator is still left to split and check each instance. What is missing is a way to learn an instance-level boundary from the few images the annotator has already labeled, fast enough to refit after every image and with nothing set by hand for each project.

We therefore keep the segmenter frozen and learn only the part that changes between projects, the class boundary. SNAP (§ 3) runs one procedure, unchanged, on every project. Class-agnostic proposers find candidate masks on the features of one frozen SAM2.1 pass [52], and each candidate is described by a vector read from the same encoder pass that produced its mask. A small per-project head, SnapHead, fit to the annotator’s labels in seconds, gives each candidate a class or rejects it as background. SAM2’s automatic mask generator supplies proposals from the first image; once labels exist, a proposer trained on the cached features joins it where the generator’s point grid misses densely packed objects, and the labels decide when. Proposals from other sources enter the same path, and when SAM 3 is one of them, the labels also decide, class by class, whether its generic boundary or the learned one is kept, so a stronger generic model adds to the learned boundary. Each component builds on established tools; together they give a class boundary set by each project’s own labels, refit in seconds after every labeled image, unchanged when only the class names change (last two rows of Table 1), and weighed class by class against SAM 3’s own boundary on the same labels.

ProjectKKSAM 3ours
LIVECell, 8 cell lines80.0380.754
COCO animals100.9400.392
LVIS rare200.9270.605
A172, small/medium/large30.1250.722
A172, type A/B/C3−0.0070.722

Table 1. Class assignment on shared ground-truth masks. Cohen’s κ\kappa, zero at chance. SAM 3: zero-shot prompt oracle; ours is SnapHead at N=60N=60. The last two rows split LIVECell-A172 into area terciles under two class-name sets. Bold: better of the two.

We judge the tool by the work it leaves the annotator. Our main measure is the drawn fraction, which needs no cost model; for total effort, priced from published timing studies, we report it over the whole cited range. On every development project, the annotator draws fewer objects by hand with SNAP than without it. Run once, frozen, on held-out datasets that played no part in its design, it beats SAM 3 as shipped on most of them. On cuts outside SAM 3’s vocabulary, such as look-alike cell lines, it beats SAM 3 even with prompt and threshold chosen on the evaluation labels (§ 4). Where the cut already sits in SAM 3’s vocabulary, the labels hand most classes to SAM 3, and once the cut moves past it they hand them to SnapHead; on microscopy this combination beats SAM 3’s oracle.

In summary, we make three contributions:

  1. Learned Boundary: SnapHead, fit in seconds to the annotator’s labels, gives every proposal a class or background from a descriptor read from the same SAM2.1 pass that produces its mask (§ 3.3).

  1. Plug-in Proposals: Any class-agnostic source, from SAM2’s mask generator to a few-shot proposer on cached features or Cellpose, feeds the same SnapHead; the project’s labels decide which sources run and, when SAM 3 is available, whose class boundary each class uses (§ 3.2 to § 3.3).

  1. Effort Accounting: An optional per-class threshold never costs more than drawing by hand on its tuning evidence, and the bound holds on 221/231 held-out splits; it charges each prediction the annotator action it triggers, priced from published timing studies (§ 3.4).

Related work

Promptable and open-vocabulary segmentation. SAM [33] and SAM2 [52] turn a click or a box into a mask; their automatic mask generator sweeps a point grid over the image, but neither assigns a class. SAM 3 [13] adds a concept path (a noun phrase or an image exemplar) that names what to segment, and SAM 3.1 is a checkpoint release without a paper [47]. Grounding DINO [39], YOLOE [74] and other text- and exemplar-prompted detectors [15, 31, 54] reach the same interface from detection. Fine-grained benchmarks such as Stanford Cars [34] and FGVC-Aircraft [43] separate objects that differ only in make, model or variant, and open-vocabulary detectors drift as the label moves down such a hierarchy [38]; SAM 3’s authors note that fine-grained concepts remain hard for their model [13]. In these models the class boundary is set by the training vocabulary; we learn it from each project’s own labels.

Few-shot and per-project adaptation. A project boundary can also come from fine-tuning. Few-shot object detectors fine-tune a detector’s last layer on a handful of boxes per class [76] or fine-tune a vision-language detector on a few multimodal examples per target class [42], and each new class needs a retrain. FeatureForest [59] fits a random forest on frozen SAM2 and DINOv2 features from a few user strokes, in the line of pixel classifiers inside bioimage tools [6, 7, 10, 30, 62]; we run this recipe as a baseline. Teuber et al. [68] probe SAM-family embeddings for classification with instance masks given as input, and linear probes on SAM tokens appear in analyses on episodic splits [17, 32]. M2C [86] fits a small concept embedding to a frozen SAM 3 from a few labeled masks for medical semantic segmentation. Episodic few-shot methods [40, 66, 82] and a spatial-canvas recipe for SAM 3 [70] transfer one concept from a support image, #+#+#+#+ and MultiverSeg [77] grows an in-context set [11] to segment one structure at a time. In microscopy, CellSAM [45], micro-SAM [5] and specialists such as StarDist and Cellpose [50, 58, 64] train backbones or decoders on large curated corpora of annotated cells. SAMCell [73] fine-tunes SAM to predict a distance map that watershed [8] decodes into instances. These methods adapt a model to one concept, label pixels, fine-tune on many boxes per class, or retrain on thousands of masks; we fit SnapHead, a multi-class classifier, together with a proposal stage to the few images the annotator has labeled.

Annotation tools with a model in the loop. CVAT [18], Label Studio [69] and Roboflow [56] embed SAM-family interactors, as do desktop and bioimage tools [19, 21, 48, 49, 75]. In these tools one public checkpoint answers every prompt in every project: the annotator still prompts each object, and cost grows with the object count. Pre-labeling followed by human correction is equally established [2, 57, 72], and Fluid Annotation [2] edits a full-image proposal set with a few actions per object. Where a project model exists, as in MONAI Label [20] and related tools [44, 67, 83], it is a full network retrained each round, with a confidence slider as the main cost control. We fit SnapHead in seconds on a frozen encoder, with an explicit background class that lets a proposal be rejected and an operating point that minimizes the annotator’s expected effort (§ 3.3 and § 3.4).

Proposals and effort evaluation. R-CNN and its successors [26, 29, 53] established propose-then-classify. Annotation cost is commonly counted in annotator actions: vertex corrections [1, 14], clicks to a quality target in interactive segmentation [61, 79], and actions per object in large annotation campaigns [9, 35]. Timed studies supply the relative cost of each action, from drawing and refining a mask to verifying a proposal [2, 49, 51, 71]. RClicks [3] shows that click-position oracles overstate interactive performance, a bias our simulated annotator avoids, since it places no clicks. For choosing which image to label next, standard active-learning strategies [25, 60] rarely beat random selection at low budgets [28, 80]. We build on both lines. Our learned proposer follows CenterNet’s formulation [36, 87] on cached encoder features, and our effort model counts every action of a simulated annotator (accept, refine, relabel, delete, draw) and prices each one with weights drawn from these timed studies.

Method

Given a frozen promptable segmenter and the few images an annotator has labeled, our goal is to segment and classify the objects in the rest of the project. We therefore keep the segmenter frozen and learn two small parts from the labels: a proposer for the objects its point grid misses (§ 3.2) and SnapHead, which draws the project’s class boundary (§ 3.3). When SAM 3 is available, the same labels also decide, per class, whether its boundary or SnapHead’s is kept (§ 3.3).

Problem setup

A project defines a set of classes CC over a pool of images. Once the annotator has finished NN images, the labeled set is L={(Ij,{(gk,yk)}k=1Nj)}j=1N\mathcal{L} = \{(I_j, \{(g_k, y_k)\}_{k=1}^{N_j})\}_{j=1}^{N}, where gkg_k is an object mask and yk∈Cy_k \in C its class. For every other image II we predict masks with classes, {(mi,y^i)}\{(m_i, \hat{y}_i)\} with y^i∈C\hat{y}_i \in C. We build on SAM2.1 [52]: its encoder Φ\Phi runs once per image, and its decoder Ψ\Psi turns a point or box prompt into a mask mm, a mask-quality estimate qq and an object pointer oo; both stay frozen. We fit three parts, all from L\mathcal{L}: the proposer weights θ\theta, the classifier weights WW and the per-class choice of boundary, and refit all three as L\mathcal{L} grows.

Plug-in proposals

SnapHead can only classify an object that some proposer has found. SAM2’s automatic mask generator (AMG) prompts Ψ\Psi with a regular point grid and keeps the masks that pass a quality and overlap filter, which gives a proposal set PAMG(I)P_{\mathrm{AMG}}(I). The grid needs no labels, so it works from the first image, but its fixed spacing misses objects that are small and densely packed.

Learning where the grid fails. To recover these objects, we fit a second proposer gθg_{\theta} on the annotator’s own masks. It follows CenterNet [87]: from the stride-4, -8 and -16 features that Φ\Phi already computes, normalized per image, it predicts a centre heatmap H^\hat{H} and a log-size map S^\hat{S}, trained with

Lθ=Lfocal(H^,H)+1∣Ω∣∑u∈Ω∥S^u−log⁡(wu,hu)∥1(1)\mathcal{L}_{\theta} = \mathcal{L}_{\mathrm{focal}}(\hat{H}, H) + \frac{1}{|\Omega|} \sum_{u \in\Omega} \left\lVert\hat{S}_u - \log(w_u, h_u) \right\rVert_1 \tag*{(1)}

where HH places a Gaussian at every object centre, Ω\Omega is the set of centre cells, (wu,hu)(w_u, h_u) is the box size of the object centred at uu, and Lfocal\mathcal{L}_{\mathrm{focal}} is the focal loss of CornerNet [36]. The top-KK heatmap peaks become boxes, and each box prompts Ψ\Psi as an annotator’s box would, which gives Pθ(I)P_{\theta}(I). We set KK to κ\kappa times the median object count of a labeled image, clipped to [Kmin⁡,Kmax⁡][K_{\min}, K_{\max}], so a crowded project gets a larger budget than a sparse one. Since gθg_{\theta} reads cached features, it adds no encoder pass.

Choosing the pool from the labels. The learned proposer pays off only where the grid fails, and the labels already show where that is. Let G\mathcal{G} pool the object masks of all images in L\mathcal{L}. We measure the grid’s recall on them and switch the pool on it:

Figure 2. Overview of SNAP. Gray blocks are frozen, blue blocks are fit to the labeled images L\mathcal{L}, and the dashed block is an optional proposal source. Every proposal passes through one frozen SAM2.1 decoder, and each newly labeled image refits SnapHead and the learned proposer.

r=1∣G∣∑g∈G[max⁡m∈PAMGIoU⁡(m,g)≥0.5],(2)r = \frac{1}{|G|} \sum_{g \in G} \left[ \max_{m \in P_{\mathrm{AMG}}} \operatorname{IoU}(m,g) \ge0.5 \right], \tag*{(2)}
P={PAMG∪Pθif r<τr,PAMGotherwise.P = \begin{cases} P_{\mathrm{AMG}} \cup P_{\theta} & \text{if } r < \tau_r, \\ P_{\mathrm{AMG}} & \text{otherwise.} \end{cases}

The switch reads only labels the annotator has already drawn, and the grid’s proposals stay in the pool either way.

Any proposal source plugs in. Every proposal reaches SnapHead the same way: as a box prompt to the frozen decoder Ψ\Psi, followed by its descriptor xx (Eq. (3)). The pool is therefore open to any class-agnostic source. Boxes from a zero-shot detector such as SAM 3 [13], Grounding DINO [39] or YOLOE [74] join PP alongside PAMGP_{\mathrm{AMG}} and PθP_{\theta}, with no change to any other part of the pipeline. Unless stated otherwise, results use PAMGP_{\mathrm{AMG}} and PθP_{\theta} only; Table 2 also adds Cellpose [64] boxes to PAMGP_{\mathrm{AMG}}.

Table 2. Five microscopy projects at N=20N=20, mean over projects (tt: median). F1clF1_{\mathrm{cl}}, F1nuF1_{\mathrm{nu}}: F1 on LIVECell’s eight cell lines and on BBBC038 nuclei. DD: fraction drawn by hand. EE: effort at wcheck=0.1w_{\mathrm{check}}=0.1, wrefine=0.5w_{\mathrm{refine}}=0.5; [E][E]: min–max of EE over the cited box, wcheck∈[0.063,0.473]w_{\mathrm{check}} \in[0.063, 0.473] [35, 51], wrefine∈[0.268,0.5]w_{\mathrm{refine}} \in[0.268, 0.5] [49]. tt: seconds to adapt, ccCPU, ggGPU; bold: fastest in group. Drawing every object by hand gives D=E=1D=E=1. †: oracle, phrasing and threshold tuned on evaluation labels. ‡: proposals from Cellpose, whose training data include LIVECell and BBBC038 (Sec. D.7). Bold: best without † or ‡.

MethodF1↑F1 \uparrowF1cl↑F1_{\mathrm{cl}} \uparrowF1nu↑F1_{\mathrm{nu}} \uparrowD↓D \downarrowE↓E \downarrow[E]↓[E] \downarrowt↓t \downarrow
SAM 3, name0.2190.0000.0000.8580.8930.878–0.9360
SAM 3, blind0.1060.0160.1700.911——0
SAM 3†0.4410.0780.7420.6090.7190.672–0.9040
G-DINO†0.0900.0000.3590.9400.9560.950–0.9910
pixel forest0.2490.0100.8060.8060.8470.830–0.9150.30ᶜ
PerSAM0.2530.0970.7040.7410.8150.785–0.9670.07ᶜ
ours, AMG0.2850.1290.8420.7620.8140.792–0.9010.03ᶜ
Mask R-CNN0.4190.0010.8160.6500.7410.702–0.8577.39ᵍ
ours, gated0.4330.2160.8420.5780.7360.668–0.9493.83ᵍ
SAM 3, L\mathcal{L}-phrase0.4200.0000.7260.6780.764——
ours, router0.4840.2160.8420.5720.694——
ours + Cellpose‡0.5440.3380.8450.4950.655——

Table 2.

Learning the project boundary

The class boundary is the part that changes between projects, so we read it out of the frozen model instead of retraining it. For every proposal, Ψ\Psi already computes an embedding of the region and an object pointer, so we describe a proposal with mask mm by their concatenation:

x=[ϕˉ(m); o(b(m))]∈R512,(3)x = \left[ \bar{\phi}(m);\, o(b(m)) \right] \in\mathbb{R}^{512}, \tag*{(3)}

where ϕˉ(m)\bar{\phi}(m) averages the stride-16 embedding of Φ\Phi over mm, o(b(m))o(b(m)) is the object pointer returned for the tight box b(m)b(m), and each half is ℓ2\ell_2-normalized. The first half describes how the region looks; the second is the decoder’s own summary of the object it segmented, and from two labeled images on, the pair beats either half alone (Sec. D.8). Both come from the pass that produced the mask, so the descriptor needs no second model.

Fitting SnapHead. The labels supply examples of every class and of background at once. On every labeled image, a proposal that overlaps an object of class cc at IoU 0.5 or more becomes an example of cc, and one that overlaps every object below 0.1 becomes an example of background, ∅\varnothing; the ambiguous rest are left out. On standardized descriptors x~\tilde{x} we fit a multinomial logistic regression,

min⁡W∑iωyi ℓCE(softmax⁡(Wx~i),yi)+λ∥W∥22,(4)\min_W \sum_i \omega_{y_i}\,\ell_{\mathrm{CE}}\left(\operatorname{softmax}(W\tilde{x}_i), y_i\right) + \lambda\lVert W \rVert_2^2, \tag*{(4)}

where yi∈C∪{∅}y_i \in\mathcal{C} \cup\{\varnothing\} and ωy\omega_y is inversely proportional to the number of examples of yy, so a rare class weighs as much as a common one. Linear on a fixed 512-d input, SnapHead fits in seconds, which is what lets it refit after every labeled image.

From probabilities to masks. A proposal is kept only if its most likely label is a class rather than ∅\varnothing. It then takes its best class c^\hat{c} and the score

s=p(c^∣x)∑c∈Cp(c∣x) q,(5)s = \frac{p(\hat{c} \mid x)}{\sum_{c \in\mathcal{C}} p(c \mid x)}\,q, \tag*{(5)}

the confidence among the named classes times the decoder’s own mask quality, so a well-classified but poorly segmented proposal still ranks low. We keep proposals with s≥τs \ge\tau, one threshold for every class, and remove duplicates by greedy suppression in score order at mask IoU τnms\tau_{\mathrm{nms}}; Sec. D.9 studies a containment-aware variant of this step.

Choosing whose boundary each class uses. Where a cut already lies in SAM 3’s vocabulary, SAM 3’s own class is hard to beat § 4.2. When SAM 3 is available, we therefore let the labels choose, one class at a time. On L\mathcal{L} we pick SAM 3’s phrasing and confidence threshold by macro-F1 from the same five phrasings and seven thresholds its oracle searches § 4.1, and we score SnapHead on L\mathcal{L} by five-fold cross-validation grouped by image. Each class cc keeps the source with the higher F1 on its labeled instances, and a class with no labeled instance keeps SAM 3. On a new image, each class takes its detections from its source, and greedy suppression in score order at mask IoU τnms\tau_{\mathrm{nms}} removes duplicates across sources; no choice reads an evaluation label.

Pricing predictions by annotator action

F1 scores a set of predictions but not the work it leaves: a wrong class costs a click to fix, while a missed object costs a full drawing. We therefore charge each prediction the action it triggers in a simulated annotator, in the spirit of Fluid Annotation [2].

Effort accounting. Predictions are matched to ground truth greedily in score order at mask IoU 0.5. A matched prediction of the right class is an accept if its IoU reaches τacc\tau_{\mathrm{acc}} and a refine otherwise; one of the wrong class is a relabel; an unmatched prediction is a delete; and every object left unmatched is a draw. Each action aa costs cac_a in units of one draw: cdraw=1c_{\mathrm{draw}} = 1, crefine=wrefinec_{\mathrm{refine}} = w_{\mathrm{refine}}, and the three check actions, accept, relabel and delete, each a look and at most one click, cost wcheckw_{\mathrm{check}}, with 0<wcheck≤wrefine<10 < w_{\mathrm{check}} \le w_{\mathrm{refine}} < 1. Effort per object is then

E(wcheck,wrefine)=D+wrefineR+wcheckC,(6)E(w_{\mathrm{check}}, w_{\mathrm{refine}}) = D + w_{\mathrm{refine}} R + w_{\mathrm{check}} C, \tag*{(6)}

where DD, RR and CC are the draw, refine and check shares of the object count. The drawn fraction DD needs no weight, so we lead with it; the two weights come from published timing studies, and we report EE over their whole cited range § 4.1.

Effort-optimal thresholds. A single τ\tau treats an easy and a hard class alike, and the same accounting can give each class its own operating point. We decompose the total cost of all actions over classes as

A(t)=ngtcdraw+∑cδc(tc),(7)A(t) = n_{\mathrm{gt}} c_{\mathrm{draw}} + \sum_c \delta_c(t_c), \tag*{(7)}

where δc(t)\delta_c(t) sums, over the detections of class cc kept at threshold tt, the cost cac_a of each triggered action minus one cdrawc_{\mathrm{draw}} per ground-truth object claimed. The optional per-class dial sets tc=arg⁡min⁡tδc(t)t_c = \arg\min_t \delta_c(t) on cross-validated curves over L\mathcal{L}, grouped by image. Each curve includes the choice of dropping class cc entirely, so δc≤0\delta_c \le0 at the optimum and A(t)≤ngtcdrawA(t) \le n_{\mathrm{gt}} c_{\mathrm{draw}} for any choice of costs: on its tuning evidence, the dial never costs more than drawing every object by hand. The flat τ\tau is one point on the same curves, and Sec. E.1 measures how far the bound carries to unseen images.

Experiments

Setup

Projects and splits. Thirteen public projects form our development set: six natural, two aerial and five microscopy. The natural set is COCO [37] vehicles, kitchen and animals, LVIS-rare20 [27], VOC-20 [23, 24] and ADE20K-indoor [85]; the aerial set is iSAID [81] vehicles and buildings on DOTA [78] tiles; the microscopy set is LIVECell [22] A172, SH-SY5Y, the eight-cell-line mix, the LIVECell split defined by micro-SAM [5] and BBBC038 [12, 41] nuclei. Each project has 180 images: a pool of 60 from which the annotator labels N∈{1,2,5,10,20,50}N \in\{1, 2, 5, 10, 20, 50\} images, and 120 held out for evaluation. We report mean ±\pm sd over three seeds. Five further datasets are held out for transfer § 4.5. After fixing the configuration of Table A1, we drew them from RF100-VL [55], a benchmark new to this work: we sorted the 74 datasets with at least two classes and 180 images by name and used the first five of a random permutation as drawn.

ComponentSetting
EncoderSAM2.1-Hiera-L, frozen, one forward pass per image
Proposer poolSAM2’s automatic mask generator (AMG), fixed knobs; plus a 97,603-parameter proposer, architecture fixed, weights fit per project (auto)
Pool switchauto: the union of AMG and the learned proposer once AMG’s recall on L\mathcal{L} falls below 0.5, AMG alone otherwise (§ 3.2)
Descriptorxx: the mask-pooled SAM2 image embedding (256-d) joined with the box-prompt object pointer (256-d)
SnapHeadmultinomial logistic regression, balanced class weights, standardized descriptors, explicit background class
Decision ruleone flat threshold, 0.5, on SnapHead’s class probability times SAM’s mask-quality score, the same for every class and every project
Suppressioncontainment-aware: candidates linked at IoU ≥0.4\ge0.4 or intersection-over-smaller-mask ≥0.8\ge0.8 collapse to the one ranked highest by that same score
External boxesSAM 3, Grounding DINO or YOLOE; optional, off by default (§ 3.2)

Table A1.

Baselines. All methods in Table 2 use the same split; our variants also share the encoder and SnapHead and differ only in the proposal source or, for the router, in whether a class keeps SAM 3’s own label. Our strongest baseline is SAM 3 [13] with an oracle choice of phrasing and threshold. We prompt it with five distinct phrasings of every class: the class name, a synonym, a hypernym, the name with a true modifier (“cell” → “phase contrast cell”), and a short description of what the object looks like (“nucleus” → “bright spots”). Each phrasing is scored at the best of seven confidence thresholds, chosen on the evaluation labels. The best of the five is therefore an oracle unavailable in practice, and the strongest prompt-only baseline. We also report the expected F1 of a blind pick among the five. Grounding DINO [39] and YOLOE [74], given the same five phrasings and each its own oracle, trail SAM 3 on 12 of the 13 projects; on the eight cell lines all three score near zero Table A3. The PerSAM baseline uses no text: it matches a prototype from one support example against the same frozen proposals, which scores above PerSAM, PerSAM-F and Matcher at their published settings Sec. D.4. The pixel classifier is a random forest on the same frozen SAM2 features followed by connected components, the ilastik and FeatureForest recipe [10, 59], trained on the same labeled images. Few-shot Mask R-CNN [29] is trained on the same labels at all six budgets, including N=1N=1. The manual baseline draws every object from scratch.

ProjectSAM 3Grounding DINOYOLOE
coco-animals0.9140.8060.780
voc-200.8290.7770.735
coco-vehicles0.7660.3790.652
coco-kitchen0.6360.2790.476
lvis-rare200.4390.1160.207
ade20k-indoor0.3790.2180.305
bbbc038-nuclei0.7420.3590.000
livecell-A1720.5860.0570.000
livecell-microsam0.4240.0220.000
livecell-SHSY5Y0.3750.0100.000
isaid-vehicles0.3040.0670.002
isaid-buildings0.1720.0380.001
livecell-8type0.0000.0000.000

Table A3. Oracle F1 for SAM 3, Grounding DINO and YOLOE under one protocol: 120 evaluation images, the same five phrasings per class, each model’s own oracle over seven thresholds. Bold: best per row. SAM 3 uses this shared protocol without the matched-KK prompt set of Sec. C.5, so livecell-8type reads 0.000 here against 0.078 in Table 2. That row is unbolded: all three round to 0.000 (Grounding DINO 0.0002).

Metrics. Our main metric is instance macro-F1 at mask IoU 0.5, with greedy matching in score order; we also report AP50_{50}. For the proposer we report class-agnostic recall at IoU 0.5 and 0.75, averaged over images. For annotation cost we report the drawn fraction DD, the share of objects the annotator still draws from scratch, together with corrections per object and the effort E(wcheck,wrefine)E(w_{\mathrm{check}}, w_{\mathrm{refine}}) of § 3.4 (Eq. (6)). Timed studies set the range of both weights: verifying a box takes 1.6 s against 26–42 s to draw it [51] and 3.5 s against 7.4 s [35], and assisted refinement takes 2.6 s against 9.7 s [49] and 4.20 s against 9.92 s [71]. We mark defaults wcheck=0.1w_{\mathrm{check}}=0.1 and wrefine=0.5w_{\mathrm{refine}}=0.5, the latter above both refinement ratios, a setting that favours the baselines, and report EE over the whole cited range. One configuration (Table A1) serves every project, and code and configurations will be released on GitHub.

Fine-grained class assignment

To separate class assignment from localization, we give every method the same ground-truth instance masks and score only the assigned class. SAM 3 uses its prompt oracle: the best of five phrasings and four decision rules, chosen on the evaluation labels, at confidence threshold 0. SnapHead is fitted to NN labeled images on frozen features, while SAM 3 runs zero-shot, so the comparison measures what a few labels add. We report Cohen’s κ\kappa, which is zero at chance whatever the class balance.

Table 1 gives the result. On eight look-alike LIVECell cell lines, SAM 3 reaches κ=0.038\kappa=0.038, and its accuracy of 0.135 is below the 0.299 of always guessing the commonest line. The same protocol reaches κ=0.940\kappa=0.940 on ten COCO animal species and 0.927 on twenty rare LVIS classes. SnapHead reaches κ=0.754\kappa=0.754 on the cell lines at N=60N=60, and 0.429 from eight images, one per line. On these two controls, where the cut lies in its vocabulary, SAM 3 stays ahead: it separates species but not cell lines. SAM 3 scores each phrase independently, so its classes never compete for a mask. Where its vocabulary already separates the classes, each phrase finds its own objects; on the cell lines, every phrase returns largely the same cells. SnapHead fits one softmax over all classes (Eq. (4)), so the line between them is what it learns. SAM 3’s released image model accepts visual exemplars only from the image being segmented, so a user cannot demonstrate the cut from other images (Sec. C); Figure 3 shows an example.

Figure 3. Class assignment on one LIVECell image of SH-SY5Y cells (densest third, 112 of 190 cells). Every panel draws the same ground-truth masks; color shows the assigned line. Titles give the count right in this crop and the accuracy over all 35,458 evaluation cells.

Words versus labels. To put look-alike classes in one image, we split LIVECell-A172 cells into area terciles and give the same split two sets of names (last two rows of Table 1). SAM 3 reaches κ=0.125\kappa=0.125 with “small”, “medium” and “large”, and chance (−0.007-0.007) with “type A”, “type B”, and “type C”. SnapHead learns from labels alone, so both namings give it the same score, and it rises with each labeled image instead: 0.068 at N=1N=1, 0.554 at N=5N=5 and 0.722 at N=60N=60.

Effect of label granularity

If the class boundary is what changes between projects (§ 1), then on fixed objects a finer cut should hurt a generic model and cost SnapHead only more labels. We test this on three datasets whose labels form a ladder: Stanford Cars by body type, make and model [34], FGVC-Aircraft by manufacturer, family and variant [43], and ShipRSImageNet by ship versus dock, category and hull type [84]. Every method classifies the same mask, which SAM2 decodes from the dataset’s box, on the same 546, 300 and 120 evaluation images at every cut; SAM 3 again uses its prompt oracle, and SnapHead sees twenty labeled images per finest class.

Figure 4 places every cut against SAM 3. On cars, SAM 3 falls from κ=0.481\kappa=0.481 at body type to 0.063 at model, while SnapHead leads at every cut, from 0.608 to 0.269. On aircraft, SAM 3 sits near chance at every cut (κ≤0.045\kappa\leq0.045), below always answering Boeing at manufacturer (accuracy 0.100 against 0.220); SnapHead reaches 0.582 there and 0.425 at variant. On ships, we match SAM 3 where the cut is ship versus dock (0.761 against 0.753) and lead once it moves to category (0.607 against 0.375) and hull type (0.245 against 0.152). On PanNuke, whose five nucleus types are in SAM 3’s vocabulary, SAM 3 leads at the finest cut, as on COCO and LVIS (Sec. B.1).

Figure 4. Comparison with SAM 3 on the label ladders and class-assignment projects. Each point is one cut; above the diagonal, the learned boundary wins. Colored markers follow the three ladders, coarse to fine; SnapHead sees 20 labeled images per finest class. Gray diamonds are the single-cut projects of Table 1 at N=60N=60. The dashed arrow renames the A172 size classes: SAM 3 moves, ours stays.

Joint localization and class assignment

End to end, a method can miss objects before class assignment matters. On LIVECell-A172, SAM2’s automatic mask generator (AMG) recalls 0.167 of the cells at IoU 0.5, while a ground-truth box through the same frozen decoder recalls 0.880, so the cells are lost at the proposal stage, not in the decoder (Sec. D.2). Our learned proposer (§ 3.2) lifts recall to 0.406 from one labeled image, 0.587 from five and 0.644 from twenty. Instance F1 rises from 0.124±0.004 with AMG to 0.432 ± 0.046 at N=5N=5 and 0.481 ± 0.003 at N=20N=20, and the drawn fraction DD falls from 0.932 to 0.585 and 0.504.

The automatic switch (§ 3.2) reads AMG’s recall on the labeled images and adds the proposer only where that recall is low. Over the 13 projects at N=20N=20 it reaches mean F1 0.239, against 0.184 for AMG alone and 0.234 for always taking the union, within 0.003 of an oracle that picks the better source per project on evaluation labels (0.241).

Comparison with baselines. We group projects by how much of SAM 3’s oracle F1 a blind choice of phrasing keeps: 0.623–0.750 on the six natural projects, which we call robust, and 0.200–0.483 on the other seven, which we call fragile (Mann-Whitney pp = 0.0012, Sec. C.5). Only the evaluation labels reveal which phrase works: on BBBC038 the class name “nucleus” scores 0.003 and “bright spots” 0.742, and a blind pick among the five expects 0.170. An independent microscopy benchmark reports the same pattern: SAM 3 does not recognize “nucleus”, and words for the shape often beat the biological term [4]. Table 2 compares every method on the five microscopy projects at N=20N=20, on the same images and seeds, with all three metrics of a method taken from one set of predictions. The other two fragile projects are aerial, whose large tiles Mask R-CNN cannot run on (recall in Table A7). With 20 labeled images, the router (§ 3.3) picks SAM 3’s phrasing from them and reaches F1 0.484, above SAM 3’s oracle (0.441; +0.043, 95% CI [0.031, 0.056], p<10−4p < 10^{-4}, paired bootstrap over evaluation images, Table A11), although the oracle tunes phrasing and threshold on the evaluation labels. The labels send 0.467 of the microscopy classes to SAM 3 and the rest to SnapHead, and each source alone scores lower: 0.420 for SAM 3 at the phrasing chosen on L\mathcal{L} ([0.051, 0.077] below the router) and 0.433 for SnapHead ([0.044, 0.059] below). SnapHead alone, with no SAM 3 at all, comes within 0.008 F1 of the oracle (95% CI [−0.024, 0.007]) and leads SAM 3 with its class name by 0.214 ([0.195, 0.231], p<10−4p < 10^{-4}). It also outperforms SAM 3 with a blind phrasing, the pixel forest, PerSAM and Grounding DINO on all five projects. An annotator without the answer key or labels gets the blind row, 0.106, and more exemplars do not lift it: through the published few-shot recipe, SAM 3 gains at most 0.013 F1 from 1 to 20 examples on any of six projects (Table A4). SnapHead improves with every label instead, from 0.290 at N=1N=1 to 0.433 at N=20N=20 and 0.449 at N=50N=50, and it matches few-shot Mask R-CNN trained on the same labels (+0.014, p=0.50p = 0.50). Its drawn fraction, 0.578, is 0.280 below SAM 3 with its class name ([0.262, 0.298]), lower on all five projects, and 0.031 below even SAM 3’s oracle ([0.012, 0.048], p=0.0004p = 0.0004). The router’s effort, which charges every deletion, is 0.694, the lowest in Table 2 without ‡ and below SAM 3’s oracle (0.719). Compared like for like, SnapHead is the fastest in each trained group of Table 2 (Sec. F). On the six natural projects, where the cut already lies in SAM 3’s vocabulary, SnapHead alone trails SAM 3 with class names (0.114 against 0.629); there the labels send 0.911 of the classes to SAM 3, and the router reaches 0.615, close to SAM 3’s 0.629. Over all 13 projects the router reaches 0.503, above SAM 3 with class names (0.404), SAM 3 at the phrasing chosen on L\mathcal{L} (0.495, p=0.003p = 0.003) and SnapHead (0.239), and 0.008 below the oracle (0.511) (Table A10).

ProjectN=1N=1N=5N=5N=20N=20text
coco-animals0.8840.8960.8930.916
coco-vehicles0.7260.7150.7110.765
lvis-rare200.5200.5330.5330.439
bbbc038-nuclei0.7560.6760.6340.742
livecell-A1720.2560.2190.2190.586
livecell-8type0.0920.0970.0830.078

Table A4. SAM 3 few-shot by the spatial-canvas recipe: six projects, N∈{1,5,20}N \in\{1, 5, 20\}, three seeds, a 60-image evaluation pre-fix; full coverage in Sec. D.3. text: the matched-KK zero-shot oracle of Table A5. Bold: best per project.

ProjectnGTn_{\mathrm{GT}}AMGboxpointgapSnapHead
coco-animals4520.9120.9900.9020.0780.433
bbbc038-nuclei69510.8510.9720.8890.1210.954
voc-203960.7600.9370.7400.1770.357
coco-kitchen6670.7480.8830.7090.1350.181
coco-vehicles6870.7370.9580.7100.2200.523
lvis-rare2010190.6570.9480.6890.2910.360
ade20k-indoor6810.5650.7150.4480.1490.429
livecell-3type538200.1810.8160.2610.6350.725
livecell-microsam681930.1600.8090.2630.6490.932
livecell-A172385000.0900.8800.2100.7840.944
livecell-SHSY5Y669070.0580.5810.1280.5220.911
isaid-buildings42320.3800.5940.4400.2140.351
isaid-vehicles537130.2920.5340.4270.2420.695

Table A7. Recall and SnapHead accuracy per project at IoU 0.5, image-macro; aerial rows cover 179 images (iSAID-vehicles box oracle: subsample, below). box/point: oracle recall from a ground-truth box or centroid. gap: box oracle minus AMG. SnapHead: accuracy on instances AMG hits.

ProjectSnapHeadnamelabeledr. namerouterSAM 3†SnapHeadnamelabeledrouterSnapHeadnamelabeledrouter
ADE20K indoor0.0770.3130.3620.3140.3440.3790.7630.4560.4670.4930.8990.9470.8370.835
COCO animals0.1770.8810.8980.8810.8980.9140.4110.0730.1060.1060.5060.2540.2670.267
COCO kitchen0.0810.6200.6290.5870.5960.6360.8360.3250.4040.4700.9130.5140.5570.604
COCO vehicles0.2160.7660.7450.7660.7450.7660.4820.1980.2650.2650.6210.3710.4150.415
LVIS rare0.0450.4080.4060.3270.3350.4390.9040.6820.5980.6690.9760.7390.6840.754
VOC-200.0890.7850.8190.7130.7710.8290.5720.1340.1520.2200.6570.3340.3370.378
mean, natural0.1140.6290.6430.5980.6150.6610.6610.3110.3320.3700.7620.5270.5160.542
iSAID buildings0.0920.0920.1690.0950.1260.1720.9090.9310.7920.8620.9650.9550.8960.932
iSAID vehicles*0.1640.2930.3040.2930.3040.3040.9390.9290.9180.9180.9720.9550.9500.950
mean, aerial0.1280.1920.2370.1940.2150.2380.9240.9300.8550.8900.9690.9550.9230.941
BBBC038 nuclei0.8420.0000.7260.8420.8420.7420.1971.0000.4090.1970.3431.0000.5220.343
LIVECell 8 lines0.2160.0000.0000.2160.2160.0780.6961.0001.0000.6960.8061.0001.0000.806
LIVECell A1720.4810.5030.5780.5030.5780.5860.5040.6470.5450.5450.7710.7300.6730.673
LIVECell SH-SY5Y0.2360.2570.3750.2520.3750.3750.8070.8490.7290.7290.9370.9010.8450.845
LIVECell micro-SAM split0.3900.3360.4200.3830.4100.4240.6860.7940.7060.6910.8220.8340.7810.804
mean, microscopy0.4330.2190.4200.4390.4840.4410.5780.8580.6780.5720.7360.8930.7640.694
mean, all 130.2390.4040.4950.4750.5030.5110.6700.6170.5450.5280.7840.7330.6740.662

Table A10. Router per project, N=20N=20, mean over three seeds. SnapHead: our head with the gated proposer. name: SAM 3 with the class name at threshold 0.5. labeled: SAM 3 with phrasing and threshold chosen on the labeled images. r. name: per-class choice between SnapHead and SAM 3 with its class name. router: per-class choice that also picks SAM 3’s phrasing on the labeled images (Sec. 3.3). †: oracle, phrasing and threshold chosen on evaluation labels. DD, EE as in Table 2. Bold: best F1 without †. *Two seeds.

Projectours\multicolumn{3}{c}{vs. SAM 3, name}\multicolumn{3}{c}{vs. SAM 3†}
Δ\DeltaCIppΔ\DeltaCIpp
Microscopy, instance F1 at mask IoU 0.5
BBBC038 nuclei0.842+0.842[0.797, 0.882]<10−4<10^{-4}+0.100[0.061, 0.147]<10−4<10^{-4}
LIVECell 8 lines0.216+0.216[0.192, 0.237]<10−4<10^{-4}+0.138[0.109, 0.162]<10−4<10^{-4}
LIVECell A1720.481−0.022[−0.065, 0.019]0.85−0.104[−0.128, −0.082]1.00
LIVECell SH-SY5Y0.236−0.021[−0.059, 0.013]0.88−0.139[−0.162, −0.117]1.00
LIVECell micro-SAM split0.390+0.054[−0.004, 0.103]0.03−0.034[−0.085, 0.010]0.93
mean of 50.433+0.214[0.195, 0.231]<10−4<10^{-4}−0.008[−0.024, 0.007]0.86
Microscopy, drawn fraction DD
BBBC038 nuclei0.197+0.803[0.741, 0.858]<10−4<10^{-4}+0.172[0.108, 0.240]<10−4<10^{-4}
LIVECell 8 lines0.696+0.304[0.261, 0.355]<10−4<10^{-4}+0.016[−0.015, 0.043]0.14
LIVECell A1720.504+0.144[0.109, 0.173]<10−4<10^{-4}+0.027[0.004, 0.047]0.01
LIVECell SH-SY5Y0.807+0.042[0.019, 0.061]0.0001−0.078[−0.103, −0.057]1.00
LIVECell micro-SAM split0.686+0.108[0.068, 0.148]<10−4<10^{-4}+0.015[−0.030, 0.054]0.25
mean of 50.578+0.280[0.262, 0.298]<10−4<10^{-4}+0.031[0.012, 0.048]0.0004
RF100-VL, instance F1 at box IoU 0.5
everdaynew0.294+0.187[0.136, 0.237]<10−4<10^{-4}+0.149[0.104, 0.192]<10−4<10^{-4}
stomata0.260+0.260[0.220, 0.291]<10−4<10^{-4}+0.106[0.048, 0.151]0.0004
floating waste0.196−0.041[−0.119, 0.041]0.84−0.058[−0.143, 0.030]0.90
Router (§ 3.3), bin means of instance F1
microscopy (5)0.484+0.265[0.250, 0.278]<10−4<10^{-4}+0.043[0.031, 0.056]<10−4<10^{-4}
natural (6)0.615−0.014[−0.026, −0.004]1.00−0.046[−0.054, −0.034]1.00
all 130.503+0.099[0.090, 0.106]<10−4<10^{-4}−0.008[−0.014, −0.000]0.98
\multicolumn{3}{c}{vs. SnapHead}\multicolumn{3}{c}{vs. SAM 3, labeled}
Router against its two inputs, bin means of instance F1
microscopy (5)0.484+0.051[0.044, 0.059]<10−4<10^{-4}+0.064[0.051, 0.077]<10−4<10^{-4}
natural (6)0.615+0.501[0.477, 0.522]<10−4<10^{-4}−0.029[−0.034, −0.023]1.00
all 130.503+0.264[0.253, 0.275]<10−4<10^{-4}+0.008[0.002, 0.014]0.0027

Table A11. Paired bootstrap over evaluation images, N=20N=20, 10,000 resamples. Δ\Delta: ours minus SAM 3 for F1; SAM 3 minus ours for DD, so a positive Δ\Delta always favors ours. CI: 95% percentile interval. pp: one-sided. name: SAM 3 with its class name at threshold 0.5 (RF100-VL: as shipped). †: oracle, phrasing and threshold chosen on evaluation labels; on RF100-VL it is chosen on box IoU, the reported metric.

Annotation effort. SNAP needs less effort than drawing by hand on all 13 development projects: at the default weights of Eq. (6), pooled E=0.807E = 0.807 at N=5N=5 and 0.784 at N=20N=20. Inside the cited box, the minimum effort of SnapHead (0.668) is below even that of SAM 3’s oracle (0.672), in the cheap-check region. The effort dial (§ 3.4) bounds cost at manual: it holds on 231/231 tuning curves and 221/231 held-out splits (95.7%); with it on, effort stays below manual over the whole cited box on all five microscopy projects (Sec. E.1). Replaying the loop, 36 rounds of five images on four projects, effort falls below manual from the second round on, every project and seed, staying below it over 0.988–1.000 of the cited range (Table A15). Most of what remains is drawing (pooled D=0.670D = 0.670 of E=0.784E = 0.784, Figure A3) …

Figure A3. Effort per project at N=20N=20, split into drawn (DD), refined (wrefineRw_{\mathrm{refine}}R) and checked (wcheckCw_{\mathrm{check}}C) shares. Whiskers: cited ranges, wcheckw_{\mathrm{check}} 0.063–0.473 [35, 51] and wrefinew_{\mathrm{refine}} 0.268–0.423 [49, 71], extended to our 0.5 default. Dashed: manual.

ProjectnGTn_{\mathrm{GT}}D↓D \downarrowE↓E \downarrowrange ↑\uparrow
bbbc038-nuclei69510.213 ± 0.0060.3571.000
coco-animals4520.324 ± 0.0140.4531.000
coco-vehicles6870.455 ± 0.0020.6020.988
livecell-A172385000.504 ± 0.0030.7670.994

Table A15. Endpoint of the 36-round loop, SNAP configuration, mean over three seeds. DD: hand-drawn share, ± sd. EE: effort at default weights. range: share of the cited range below manual.

Generalization to held-out datasets

We ran the frozen configuration once, unchanged, at N=20N=20 on the five RF100-VL datasets of § 4.1, scoring at box IoU 0.5 as RF100-VL has only boxes. We count a win when our mean minus one standard deviation over seeds exceeds SAM 3’s score. By that rule we outperform SAM 3 with class names on four of the five datasets and its phrasing-and-threshold oracle on two (Table 3); a paired bootstrap over evaluation images confirms both oracle wins (p≤0.0004p \le0.0004, Table A11). With SAM 3 as a second source, the frozen router reaches the best mean F1, 0.328, against 0.308 for SnapHead alone (p=0.0005p = 0.0005) and 0.306 for SAM 3’s oracle. The clearest win is the finest cut in the draw, everydayn (one product in different package sizes): we reach 0.294 against 0.107 for SAM 3 with class names and 0.145 for its oracle. Where SAM 3’s oracle leads, the classes are everyday words (people and hats; bottle and can; humans), where a generic vocabulary should win.

Table 3. Transfer to five randomly drawn RF100-VL datasets at N=20N=20. Instance F1 at box IoU 0.5, mean ±\pm std over three seeds; SAM 3 is deterministic. ours: SnapHead. SAM 3: class names, default threshold. oracle: best phrasing and threshold on evaluation labels. router: § 3.3. SW: the automatic switch (§ 3.2) is within seed noise of the better fixed proposer. Last row: mean; sd over seeds.

dataset (classes)oursrouterSAM 3oracleSW
everydayn (pack sizes)0.294±0.0300.250±0.0420.1070.145y
aerial-pool (hats, people)0.188±0.0090.317±0.0100.0520.354y
stomata (open, closed)0.260±0.0120.239±0.0380.0000.154n
floating-waste (litter)0.196±0.0270.233±0.0030.2370.253y
floods (human)0.602±0.0110.601±0.0110.5780.622y
mean0.308±0.0140.328±0.0170.1950.306—

Table 3.

Conclusion

A generic segmenter’s class boundary is fixed by its vocabulary; SNAP learns each project’s own boundary on the fly from its labels, on one frozen SAM2.1 pass per image. Handed identical masks of eight look-alike cell lines, SnapHead reaches κ=0.754\kappa=0.754 against SAM 3’s 0.038, and holds 0.722 when only the class names change, while SAM 3 moves from 0.125 to −0.007. With twenty labeled images SnapHead alone beats SAM 3’s oracle on those cell lines (0.216 against 0.078 instance F1), on nuclei (0.842 against 0.742) and on two of five held-out RF100-VL projects. Where the cut already lies in SAM 3’s vocabulary, SAM 3’s own class is better, so the two complement each other: keeping, per class, the boundary the labels favor beats SAM 3’s oracle on microscopy (0.484 against 0.441) and trails it by 0.008 over all 13 projects. We build on SAM2.1 for its Apache 2.0 license; SAM 3 (SAM License [46]) stays optional. A foundation model supplies where the objects are; the project’s own labels decide which boundary to trust.

Supplementary Material

In this supplementary material, we provide implementation details (Sec. A), dataset and protocol details (Sec. B), the full comparison with SAM 3 (Sec. C), additional results and ablations (Sec. D), annotation effort (Sec. E), and runtime and memory cost (Sec. F).

Implementation Details

Table A1. Default SNAP configuration, used for every project. auto entries are read from the labeled images by the rule shown; the rest are fixed.

Proposals and descriptors. We run SAM2.1-Hiera-L frozen at production AMG density (64 points per side, one crop layer). The descriptor xx is the 256-d mean image embedding over the mask, ℓ2\ell_2-normalized and joined with the 256-d box-prompt object pointer. We drop proposals whose best ground-truth IoU falls in an ambiguous band from training.

Learned proposer. AMG’s point grid bottlenecks resolution on dense projects (Sec. D.2). LIVECell’s stride-16 embedding spans a 30 px cell; we reuse the stride-4 decoder features instead, for a stride of 2.75 px on a 704 px frame. Input: two high-resolution feature maps (32×\times256×\times256 at stride 4, 64×\times128×\times128 at stride 8) and the image embedding (256×\times64×\times64), upsampled to 352×\times256×\times256; cell (gx,gy)(g_x,g_y) maps to ((gx+0.5)W/256,(gy+0.5)H/256)((g_x+0.5)W/256,(g_y+0.5)H/256). Trunk: group normalization (11 groups, 352 channels), 1×\times1 conv to 64, two [3×\times3 conv 64, group normalization (8 groups), ReLU] blocks, two 1×\times1 heads.

Center head: CornerNet’s penalty-reduced focal loss (α=2\alpha=2, β=4\beta=4) on Gaussian-splatted centers (overlap 0.7), plus ℓ1\ell_1 on log size. Inference: sigmoid, 3×\times3 max-pool peak test, top-KK cut (K=3×K=3\times median seed-image count, cap 512); SAM2 decodes, keeps the best mask, ranks by peak score times predicted IoU (NMS 0.7). Training: 300 Adam steps, learning rate 3e−3, cosine decay, batch min(4,N)(4,N).

Distance-map head: binary cross-entropy on foreground, ℓ1\ell_1 on a per-instance distance map normalized to 1 at the deepest point; decoding seeds at d≥0.5d \ge0.5 and grows by Euclidean influence zones. Sec. D.9 scores it against the center head.

The proposer has 97,603 parameters and fits in under four seconds on cached features (Table A16 gives per-image cost against AMG). Micro-SAM’s decoder beats AMG on microscopy but needs a large annotated corpus [5]; our proposer fits from the images already labeled for the project.

Stagemeasured
encoder pass, once per image18.6–19.7 ms
encoder peak VRAM1.64 GB
production AMG proposals9.0–9.9 s/img
AMG at half grid density0.53–0.75 s/img
learned proposer, stage only0.02–0.49 s/img
learned proposer, parameters97,603
learned proposer, training1.5–3.9 s
descriptor pooling over proposals0.14–0.88 s/img
SnapHead fit8 ms–2.6 s
grouped CV, background (feeds the dial)0.08–304 s
SnapHead artifact, 1 class / 10 class33 / 143 KB
proposer fit in SNAP (3 images, 300 steps)2.98 s
learned rows, deployed path1.68 s/img
peak CUDA, union mode3.06 GB
supplied zero-shot boxes: decode + descriptor0.81 s/img
browser decode per prompt, median147–156 ms
browser decode per prompt, p95171–182 ms
decoder ONNX, session create15.8 MiB, 466–507 ms
embedding frame, parse8.00 MiB, 42–56 ms

Table A16. Measured per-image cost, median over 8 images; AMG at production density. stage only: proposer vs. AMG alone. deployed path: adds decode, NMS, re-prompt and descriptor. Bold: deployed learned proposer. Accuracy: Sec. 4.4.

Budgets and seeds. We use budgets N∈{1,2,5,10,20,50}N \in\{1, 2, 5, 10, 20, 50\}, three seeds each, disjoint subsets nested in NN. We read KK and the AMG-recall gate from labeled images. Below seed-set AMG recall 0.5 the pool unions AMG and the learned proposer; at or above it, AMG runs alone. Sec. D.3 gives the band SNAP uses by default.

Constants. We fix the proposer at κ=3\kappa= 3, [Kmin⁡,Kmax⁡]=[16,512][K_{\min}, K_{\max}] = [16, 512]; the switch uses τr=0.5\tau_r = 0.5. SnapHead uses scikit-learn’s default class weight (C=1C=1), τ=0.5\tau= 0.5, τnms=0.4\tau_{\mathrm{nms}} = 0.4; accept needs IoU τacc=0.75\tau_{\mathrm{acc}} = 0.75, fixed across projects (Sec. F).

Metric choices. Our F1 at IoU 0.5 follows CellSAM [45], stricter than mSAM/COCO mask AP (IoU up to 0.95, omitted here). F1 pools all instances across the 13 projects rather than averaging over images. Averaging over images instead raises F1 on 11 of 13, most on LIVECell-A172/micro-SAM (+0.075 each, 230.2/396.0 instances per image). It falls for COCO vehicles (−0.008) and the eight-line mix (−0.001). We report pooled F1 throughout.

DatasetCutKKSAM 3\multicolumn{2}{c}{ours, fixed}
Stanford Carsbody type90.4810.4480.608
Stanford Carsmake480.1150.1930.380
Stanford Carsmodel1820.0630.0880.269
FGVC-Aircraftmanufacturer300.0450.3850.582
FGVC-Aircraftfamily700.0360.3290.577
FGVC-Aircraftvariant1000.0300.1850.425

Table A2. Cohen’s kappa per cut, Stanford Cars and FGVC-Aircraft. SAM 3: zero-shot prompt oracle; SnapHead, three seeds.

Datasets and Evaluation Protocol

Coarse-to-fine ladders

SAM 3 still finds the object: in all 72 settings on cars and aircraft, every phrase localizes it at IoU 0.5, and only the confidence score changes. On a fourth ladder, PanNuke, SAM 3 leads at the finest cut, already in its vocabulary.

We give § 4.3’s per-cut numbers, two more ladders, and their construction and protocol.

Tab. A2 gives the per-cut numbers, labeled images fixed at NN per finest class and relabeled per cut.

Best-kappa rule. For SAM 3’s cell in Tab. A2 we pick the phrasing and decision rule by kappa instead of by macro-F1 as in § 4.2, which can only raise its score.

ShipRSImageNet. We cut ShipRSImageNet’s ships three ways over 120 images: ship/dock (K=2K=2, chance 0.386), four categories (K=4K=4, chance 0.224), fifty hull types (K=50K=50, chance 0.014). SAM 3’s oracle reaches 0.753/0.375/0.152. At N≥20N \ge20 SnapHead leads every cut: 0.761/0.931 (N=20/60N=20/60) for ship/dock; 0.607/0.591 for category; 0.245 for hull type.

PanNuke. PanNuke’s finest cut splits nuclei into five pathology types (K=5K=5, chance 0.176, 120 images), a vocabulary SAM 3 already has. Its oracle reaches kappa 0.724 from the bare class names. SnapHead reaches 0.173/0.309/0.362/0.363 at N=1/5/20/60N=1/5/20/60, well short of SAM 3. Here the cut sits inside SAM 3’s vocabulary, and it wins.

Dataset construction. Stanford Cars pairs each Hugging Face mirror image2 to the devkit’s box by assumed filename order. Shifting it by one image puts 44.9% of boxes outside their image, a random pairing 46.2%, versus none assumed. We drop classes with no body word (14 of 196), leaving 182 models and 48 makes over 546 images. FGVC-Aircraft’s

box file is 1-based inclusive; 130 of its 10,000 boxes start at pixel 0 and are clipped, 65 here. We score every arm on SAM2’s decoded mask, at median IoU 0.954/0.961 (cars/aircraft).

The iSAID-vehicles subsample. iSAID-vehicles has an oracle estimated on a subsample. Its tiles carry up to 7,374 objects, so we cap ground truth at 1000 per image, capping 12 of its 179 images and leaving 31,515 of 53,713 instances prompted, seeded and unbiased.

Aerial projects. We append both iSAID projects to the grid under SNAP’s gated system. With our descriptor both sit at a resolution floor: most iSAID-vehicles instances are under 8 px after SAM2’s 1024-square resize, below any method’s workable size (Sec. D.2 prices the same floor on iSAID-buildings). We skipped few-shot Mask R-CNN on both: torchvision caps the long side at 1333 px, so a 13,383 px tile loses its sub-8 px instances.

SAM 3 oracle’s two-collection rule. For the SAM 3 oracle row in Tab. A8, we take the better of two prompt collections per project, both fixed in advance. It switches to the later matched-KK set of Sec. C.5 only past run-to-run noise (0.005 F1). Only livecell-8type switches: correct cell-line names score 0.000, an appearance description 0.078.

Comparison with SAM 3

Class assignment

Fig. A1 compares class assignment on the eight LIVECell cell lines. SAM 3 runs zero-shot with the phrasing that scores best on the evaluation labels; SnapHead is fitted to N=60N=60 labeled images per line. We score instance classification on fixed masks, rather than the segmentation F1 of the main paper.

Reading the matrices. SAM 3’s matrix on the eight cell lines is diffuse, mean diagonal 0.141, its largest error Huh7 read as A172 at 0.84. Per-line recognition ranges from SkBr3 at 0.33 to Huh7 at 0.01. SnapHead’s matrix is a diagonal, mean 0.770, peaking at BV2 with 0.96. The same code and prompts on ten COCO animal species (n=246n=246) give SAM 3 a near-perfect diagonal, mean 0.957, with largest errors of 0.08 and 0.05.

Qualitative image. We show one SH-SY5Y image with 190 instances in Figure 3, picked by the lower median of per-image instance count over the 120-image evaluation split; its panels crop the image to its densest third, 112 of the 190 instances, at about the same SAM 3 error rate (∼0.95\sim0.95) as the full image.

Figure A1. Class assignment on eight LIVECell cell lines, SAM 3 against SnapHead. Masks fixed; n=35,458n=35{,}458 instances, row-normalized, cells below 0.005 blank.

Tercile split. The words-versus-labels test of § 4.2 splits the 27,623 LIVECell-A172 instances into area terciles. All three classes are found in 118 of the 120 evaluation images. At N=60N=60 SnapHead scores macro-F1 0.816 under both namings.

Text-encoder check. The text encoder keeps “A172 cell” and “SHSY5Y cell” apart. Raw phrase cosines are about 0.99 for every prompt set, yet the eight line names are mutually distinguishable after mean-centering. We therefore attribute the failure to vision-language alignment.

Architectural limit and composition. At most 200 masks are emitted per prompt by SAM 3, one per object query. On a project averaging 295 instances per image, recall caps at 0.520 regardless of phrasing. Restricted to the instances a phrase actually returns, SAM 3 scores 0.150 against a 0.121 chance level. The per-instance best-IoU spread across the eight phrases is 0.08 to 0.13. Combining models beats either alone: a generic “cell” prompt with SnapHead assigning the class reaches 0.766 on the eight lines, against 0.752 for it on ground-truth boxes, both at N=20N=20.

Pairwise separation. We score each of the 28 cell-line pairs as a two-way decision, so 0.5 is chance. SnapHead’s accuracy ranges from 0.544 to 0.998 (median 0.971).

SAM 3’s ranges from 0.497 to 0.657 (median 0.547), with 25 of the 28 pairs below 0.6; it leads on all 28.

Prompts and exemplars

Prompts. We add the description phrasing because SAM 3’s own prompt study reports 0.085 mSA for “nucleus” and 0.509 for “bright spots” on the Data Science Bowl. We fix all 530 distinct strings in advance. The same five phrasings per class are used on every project (Sec. C.5). Grounding DINO [39] and YOLOE [74] run the same phrasings; SAM 3 beats both on 12 of the 13 projects (Table A3), and on the eight cell lines all three score near zero, so we report SAM 3.

Exemplars. The released image API has no image-level exemplar slot. Table A4 gives the full few-shot grid. Under the published spatial-canvas workaround [70], no project improves by more than 0.001 from N=5N=5 to N=20N=20, and five of six fall.

Few-shot grid

Few-shot recipe. Table A4 uses the published spatial-canvas recipe [70]: up to three support panels and four canvases per class, one exemplar box per panel. Detections are unioned over canvases and NMS’d at IoU 0.7. The N=1N=1 canvas holds two panels against four for N≥5N\ge5, so part of the 1→51\to5 drop is panel resolution.

Fine-tuning

SAM 3’s own 10-shot fine-tuning raises its RF100-VL box AP from 15.2 to 36.5 [13]. That setting fine-tunes the whole model per dataset. SnapHead instead fits on a frozen encoder inside the annotation loop.

Phrasing sensitivity per project

We prompt SAM 3 with five phrasings per class on all 13 projects. An oracle picking the best phrasing and threshold on evaluation labels beats a blind pick on every project, by 0.062 to 0.573 F1. SAM 3’s worst of five phrasings never falls below 0.079 on a natural project and never rises above 0.009 on a scientific one. Six of seven scientific projects have a phrasing scoring exactly zero; no natural project does. The retained fraction, exp/oracle, is 0.62 to 0.75 on natural projects and 0.20 to 0.48 on scientific ones (Mann-Whitney p=0.0012p=0.0012, six against seven projects). On BBBC038, “nucleus” scores 0.003 against 0.742 for “bright spots.” On LIVECell-A172, “cell” scores 0.586, and one

true adjective drops it to 0.000. Table A5 gives the per-project detail, all of it SAM 3.

ProjectclsoracleexpΔ\Deltaworstret.
coco-animals100.9160.6280.2880.3530.69
voc-20200.8220.6160.2050.2740.75
coco-vehicles50.7650.4770.2880.0820.62
coco-kitchen110.6360.4210.2150.1940.66
lvis-rare20200.4390.3260.1130.1990.74
ade20k-indoor220.3750.2410.1330.0790.64
bbbc038-nuclei10.7420.1700.5730.0000.23
livecell-A17210.5860.1170.4690.0000.20
livecell-microsam10.4240.1250.2990.0000.30
livecell-SHSY5Y10.3750.1040.2710.0000.28
isaid-vehicles20.3040.1470.1570.0000.48
isaid-buildings40.1720.0690.1030.0090.40
livecell-8type80.0780.0160.0620.0000.20

Table A5. Fragility axis, all 13 projects. SAM 3 under five phrasings per class (K=5K=5), each at its best threshold; 120 evaluation images, 119 for the two aerial projects. oracle: best phrasing on evaluation labels, in bold. exp: expected F1 of a blind pick. Δ\Delta: their difference. worst: weakest phrasing. ret.: exp/oracle, kept by a blind pick.

Domain dependence. On scientific projects, the per-slot means show where phrasing fails. Only two slots fire there: the name at 0.266 and the appearance phrase at 0.177; synonym (0.054), hypernym (0.006) and qualified (0.031) score near zero. The appearance slot is weakest on natural images and the rescue on nuclei and the eight cell lines, so the two domains want opposite phrasing. The class name is the best of the five phrasings on nine of the 13 projects. It fails hardest on the rest.

Additional Results and Ablations

Learned proposer

Table A6 gives proposal recall at N=5N=5, and Figure A2 shows the proposals on one image.

ProjectAMGLAP ∪ AMGours ∪ AMGoracle
livecell-A1720.1670.2950.5870.880
livecell-8type0.2590.3090.4890.816
livecell-microsam0.2480.2910.4860.809
livecell-SHSY5Y0.1080.1550.3200.581
bbbc038-nuclei0.8760.8910.8930.972
ade20k-indoor0.5060.5660.5290.715

Table A6. Proposal recall at IoU 0.5, N=5N=5, mean over 3 seeds and 120 images. LAP: AMG knobs plus a stride-16 box scorer. oracle: decoder ceiling. Bold: best below the oracle.

Proposals on one LIVECell-A172 image

Figure A2. Proposals on one LIVECell-A172 image: production AMG, the learned proposer (N=5N=5, top-K=512K=512 boxes), and ground truth, chosen by Figure 3’s rule (Sec. C). Panels give frame recall.

Proposer image. We pick the image with Figure 3’s rule (Sec. C): seed 2, the median learned-proposer recall across seeds (0.548, 0.710, 0.688). The learned panel draws 470 boxes against 186 ground-truth cells, reaching recall 0.688 against a split mean of 0.575. AMG reaches 0.075 against a split mean of 0.167.

Fine-tuned detector, per project. Across the 11 core projects, we beat few-shot Mask R-CNN on instance F1 on 9, 10 and 9 of 11 at N=1N=1, 5 and 20. We also beat it on drawn fraction, 11, 11 and 9 of the same budgets. Pooled over three seeds, F1 is 0.141/0.202/0.259 against 0.054/0.162/0.215 at N=1N=1, 5, 20. On LIVECell-A172 (230 instances per image), the detector leads F1 at N=2N=2, 5, 20, 50 (0.370/0.510/0.620/0.600 against 0.307/0.432/0.481/0.484); we lead at N=1N=1, 10 (0.249/0.479 against 0.059/0.365). On BBBC038 we lead on F1 at every budget: 0.819, 0.823, 0.825, 0.825, 0.842 and 0.843 against 0.413, 0.614, 0.725, 0.777, 0.816 and 0.834 at N=1N=1, 2, 5, 10, 20, 50. A score of exactly 0.000 is recorded for the detector on 3 of the 11 core projects at N=1N=1 (ade20k-indoor, lvis-rare20, voc-20). Our proposer (Sec. A) uses a fraction of its 44M parameters and 40 steps.

AMG-strong projects. Adding the proposer where AMG already recalls well is mostly neutral. At N=5N=5 on BBBC038, F1 is 0.825±0.003/0.780±0.035/0.573±0.058 for AMG alone/the union/proposals alone. Drawn fraction is 0.199±0.013/0.203±0.013/0.517±0.106. On ADE20K-indoor, proposals alone reach only 0.005±0.002 and AMG wins outright. AMG’s recall on the seed images is measurable from labels alone, so the switch adds no cost.

Oracle decomposition

Frozen-decoder ceiling. Table A7 aggregates recall per image, unlike Table 2’s instance-pooled F1, so the two scales differ. The gap matters where object sizes are skewed.

iSAID-vehicles’ box oracle is 0.534 image-macro but, once instances are pooled, only 0.228, since 82% of its instances sit under 8 px at model input and crowd a few dense tiles.

Error splits into “the proposer never found it” and “SnapHead got it wrong.” Natural projects and nuclei are classification-bound. AMG recalls 0.57 to 0.91 of ground truth, and it is the limit. LIVECell is the reverse. AMG recalls only 0.058 to 0.181, and it is correct on 0.72 to 0.94 of what it sees. A ground-truth box recovers 0.581 to 0.880 there. Centroid points do not help either (0.13 to 0.26). An independent 2026 benchmark measures unmodified SAM’s AMG at 0.075 mSA on LIVECell and 0.331 on the Data Science Bowl [4].

All methods on one split

Table A8 is Table 2 with the robust bin restored. § 4.4 states the robust-bin result: SAM 3 reaches 0.661 with its oracle phrasing and 0.629 with the class name, against our 0.114.

MethodF1drawncorrF1drawn
manual—1.0001.000—1.000
SAM 3, class name0.2190.8580.9240.6290.311
SAM 3, blind phrasing0.1060.9110.9980.4520.517
SAM 3, oracle0.4410.6091.0020.6610.338
G-DINO, oracle0.0900.9401.0010.4290.448
Mask R-CNN, few-shot0.4190.6500.8500.0450.900
pixel forest0.2490.8060.8850.0170.720
pixel forest, argmax0.2670.7730.8930.0010.999
PerSAM reduction0.2530.7411.0410.0580.273
ours0.2850.7620.8840.1140.661
ours, ∪\cup zero-shot0.4190.6231.0130.1280.582
ours, effort dial0.2460.7960.8730.0580.846
ours, gated0.4330.5781.1530.1140.661

Table A8. Every method, both bins, N=20N=20. 11 core projects; the two aerial projects are analyzed separately, by recall, in Table A7. drawn: fraction of objects still drawn by hand. corr: corrective actions per object. manual has no F1 by construction. Every ours row uses the same descriptor xx. Bold: best F1 and drawn fraction per bin.

Significance. Every significant result in Table A8 lands at exactly p=0.03125p = 0.03125, the smallest value a one-sided Wilcoxon signed-rank test returns over five projects. A paired bootstrap over evaluation images has no such floor (Sec. D.6).

Effort columns. The PerSAM reduction reaches the best drawn fraction in Table A8 (0.273, robust bin), but scores only 0.058 F1 and 21.0 corrective actions per object. Almost everything is predicted and the annotator deletes the rest, so we report drawn fraction and corrections per object together.

Exemplar baselines at their published settings

We also run PerSAM, PerSAM-F [82] and Matcher [40] from the authors’ code at their published operating points, on the five microscopy projects with the same 20 labeled images, seeds and scorer as Table 2 (Table A9). Matcher uses up to 20 support images per class. Neither method handles several classes, so each class runs on its own and overlapping masks across classes are merged at IoU 0.5. At these settings no method passes 0.045 mean F1: the published loops stop after a few objects, while a LIVECell image holds hundreds of cells. Lifting the object caps raises PerSAM to 0.193, still below the reduction we report in Table 2 (0.253) and well below SnapHead (0.433).

method8 linesA172SH-SY5Ymicro-SAMnucleimeanDD
Matcher0.0030.0010.0010.0020.0630.0140.990
PerSAM0.0080.0080.0020.0060.2030.0450.969
PerSAM-F0.0080.0090.0030.0050.1710.0390.972
Matcher*0.0070.0070.0020.0060.2900.0620.948
PerSAM*0.0370.0630.0140.0870.7660.1930.804
PerSAM-F*0.0290.0650.0150.0620.5850.1510.837
PerSAM reduction0.0970.1280.1050.2310.7040.2530.741
ours, gated0.2160.4810.2360.3900.8420.4330.578

Table A9. Exemplar baselines at their published settings, five microscopy projects, N=20N=20, instance F1 (mean over three seeds). *: object cap lifted.

Pixel forest. A random forest on the same frozen SAM 2 features ties us on BBBC038 (argmax variant, 0.834 against 0.842 at N=20N=20). It loses across the bin (0.267 against 0.433), close only where the proposer does no work.

Mask R-CNN across budgets. We compare against few-shot Mask R-CNN project by project in Sec. D.1; it leads on two dense single-class projects, LIVECell-A172 and the LIVECell split defined by micro-SAM.

Gate threshold. Our system switches below a seed-set AMG recall of 0.35. Table 2 and Table A8 use 0.5 instead, disagreeing on four of 198 seed-level cells. Seed-set recall is close to bimodal, 0.024–0.422 on the four LIVECell projects, 0.389–1.000 on five of six natural ones, except BBBC038 (0.787–1.000) and LVIS-rare, spanning both modes (0.153–1.000). Recomputed at 0.35, no bin mean moves more than 0.005 F1 or 0.004 drawn, unchanged at N≥5N \ge5. Any threshold from 0.259 to 0.506 matches 0.5’s switch choice on the 13 projects. Helped projects read at or below 0.259, unhelped ones at or above 0.506; our default, 0.35, sits inside.

Switch versus oracle choice. Two fixed policies bracket the switch on the five microscopy projects at N=20N=20: AMG alone reaches 0.285 F1, the learned-proposer union 0.429, and the oracle choice 0.433. Using only seed-set AMG recall, our switch matches the oracle choice on all five at N=20N=20 and four of five at N=1N=1. Livecell-8type costs it 0.001 F1 there.

Router per project

Table A10 lists every project at N=20N=20 for the router of § 3.3 and its inputs. SAM 3, labeled picks its phrasing set and threshold on the 20 labeled images, the evidence the router also uses; it is the operating point an annotator with labels reaches without the answer key. The router sends 0.911 of natural classes to SAM 3, 0.792 of aerial ones and 0.467 of microscopy ones (0.895, 0.667 and 0.333 when SAM 3 keeps its class name). On microscopy it reaches 0.484 F1, above both of its inputs (SnapHead 0.433, SAM 3 labeled 0.420) and above SAM 3’s oracle (0.441); over all 13 projects it reaches 0.503, against 0.511 for the oracle. With 20 labeled images, a class with few instances can be routed on noise: on three many-class natural projects, LVIS rare, VOC-20 and COCO kitchen, the router scores below SAM 3 with its class name, and each further labeled image adds instances to that per-class estimate.

Paired bootstrap

A test over five projects cannot go below p=0.03125p = 0.03125, so we also resample evaluation images. For each project we draw 10,000 resamples of its 120 evaluation images, apply the same draw to every method, and recompute pooled instance macro-F1 and the drawn fraction DD from per-image counts; our score is the mean over the three seeds inside each resample. The five-project mean resamples each project independently. pp is one-sided, the share of resamples in which ours is not better.

Table A11 gives the result. On the five microscopy projects, SnapHead leads SAM 3 with its class name by 0.214 F1 (95% CI [0.195, 0.231]) and draws 0.280 fewer objects by hand ([0.262, 0.298]), with a lower DD on all five projects. Against SAM 3’s oracle, the F1 difference is −0.008-0.008 ([−0.024,0.007-0.024, 0.007]), while our drawn fraction is lower by 0.031 ([0.012,0.0480.012, 0.048], p=0.0004p = 0.0004). On RF100-

VL, SnapHead beats both SAM 3 arms on everydaynew and on stomata (p≤0.0004p \le0.0004); on floating waste the interval contains zero. The bootstrap covers every method with stored per-image predictions; aerial-pool and floods (where the switch picks the learned proposer) and few-shot Mask R-CNN keep the Wilcoxon test. The router (Sec. 3.3) is tested the same way, with each bin resampling its projects independently. On microscopy it leads SAM 3’s oracle by 0.043 F1 ([0.031, 0.056], p<10−4p < 10^{-4}) and leads both of its inputs, SnapHead by 0.051 and SAM 3 at its labeled operating point by 0.064. Over all 13 projects it comes within 0.008 F1 of the oracle and still leads both inputs.

Microscopy specialists

We run Cellpose cyto3 [63] zero-shot, with its own size model and no per-project tuning, on all five microscopy projects, and StarDist (2D-{:xref?}versatile_fluo) [58] on

BBBC038, the one fluorescence project. Neither model outputs a class; on the eight-line project every Cellpose mask takes the line most common in the labeled images. We also feed Cellpose’s boxes to SnapHead as one more proposal source (§ 3.2): each box prompts the frozen decoder, and SnapHead classifies the result, at N=20N=20 over three seeds.

Table A12 gives the result. Cellpose alone cannot separate the eight cell lines (0.053 F1); SnapHead on its proposals reaches 0.338. Cellpose proposals raise SnapHead on all five projects, from a mean F1 of 0.433 to 0.544, above SAM 3’s oracle in Table 2. Its masks enter SnapHead as boxes re-decoded by SAM2, and the re-decoded masks match fewer cells than Cellpose’s own; of those that match a cell, SnapHead keeps 80–95% and rejects 1–14% as background, most on SH-SY5Y and the micro-SAM split. This is why the combination still draws more by hand than Cellpose alone (D=0.495D = 0.495 against 0.310). A variant that exempts Cellpose proposals from the background reject reaches 0.560 F1 and D=0.464D = 0.464; the default treats every source alike.

Training overlap. Our LIVECell images come from

Project\multicolumn{3}{c}{Cellpose‡}SD‡\multicolumn{3}{c}{CP + head}head
F1DDEEF1F1DDEEF1
BBBC038 nuclei0.8170.2200.3660.7560.8450.1800.3340.842
8 lines0.0530.2570.365—0.3380.5420.6680.216
A1720.7510.3350.514—0.6260.4530.6800.481
SH-SY5Y0.6680.4440.631—0.3670.7190.8680.236
micro-SAM split0.7990.2930.446—0.5450.5810.7270.390
mean of 50.6180.3100.464—0.5440.4950.6550.433

Table A12. Microscopy specialists, instance F1 at mask IoU 0.5, with DD and EE as in Table 2. Cellpose: cyto3, zero-shot. SD: StarDist, BBBC038 only. CP + head: Cellpose boxes as proposals for SnapHead, N=20N=20, mean over three seeds. head: SnapHead with its own proposals (Table 2). ‡: likely trained on these images.

LIVECell’s own training annotation file, and Cellpose cyto3 was trained on a mix that includes LIVECell; BBBC038 is in its nuclei training mix as well. We therefore report the Cellpose and StarDist numbers here, as an upper bound for unseen cells.

Projectimagesxxmask-pooledpointer
8 lines80.4290.4170.344
160.5580.5530.421
400.7130.7080.479
600.7540.7470.501
A172, size split10.0680.1100.084
20.2060.1630.193
50.5540.4550.515
200.6740.5970.571
600.7220.6410.595

Table A13. Descriptor halves, given ground-truth masks as in Table 1. Cohen’s κ\kappa, mean over three seeds. images: labeled images; on the eight lines the budget is set per line, and 20 per line takes the whole 60-image pool. Bold: best per row.

Descriptor halves

The descriptor xx joins two 256-d halves (Eq. (3)}). We refit SnapHead on each half alone under the given-masks protocol of Tab. 1 ({xref}Tab. A13`). The joint descriptor has the highest κ\kappa at every budget on the eight lines, and on the A172 size split from two images on; at one image all three are near chance there. The halves carry different information. On the eight lines the mask-pooled half carries most of the signal (0.747 against 0.754 at 60 images) and the pointer half alone reaches 0.501. On the size split the pointer half leads the mask-pooled half at two and five images, and at 60 images each half alone trails xx (0.641 and 0.595 against 0.722).

Per-dataset alternatives

Table A14 lists alternatives to our fixed configuration and what each changes. Recall is per-image recall at IoU 0.5 on the evaluation images; F1, mean instance F1 over the 13 projects at N=20N=20.

alternativeresulttrade-off
AMG knobs tuned from seed imagesOn LIVECell-A172 the costliest grid reaches recall 0.358 at 84.9 s per image; the learned proposer reaches 0.575 at 0.41 s.More compute buys less recall than our proposer.
AMG filters switched offThe raw pool covers 0.694 of A172 cells; SAM2’s predicted-IoU >0.7> 0.7 filter cuts that to 0.146, and NMS removes almost nothing.About 10410^4 masks per image is too many to score.
AMG filter learned from seed imagesOn single-scale grids it keeps 78–99% of the raw pool on cells (A172 0.355 of 0.369), but fails on crops and COCO vehicles.Still below the learned proposer (0.575 on A172) at higher cost.
Training-free top-hat blob proposerJoined with AMG it lifts A172 recall 0.167→0.282 and 8type 0.259→0.330; alone it finds almost nothing large (0.014 on VOC).The learned proposer is better wherever top-hat helps.
Native-resolution tiling, aerial tiles83% of iSAID vehicles are under 8 px after resize, where both methods find about 1%; above it we match SAM 3. Tiled AMG costs about 110 s.It lifts SAM 3 and us alike, so we treat it as pre-processing.
Proposer choice by cross-validationAn oracle picking the best proposer set per project beats the one-line switch by only 0.005 (N=5N=5) and 0.002 (N=20N=20) mean F1.The one-line switch already captures the gain.
Fragments trained as backgroundMean F1 falls 0.239→0.233, and the share left to draw rises.A fragment’s descriptor looks like the whole object, so SnapHead rejects whole objects.
Learned ranker over same-location versionsMean F1 is indistinguishable from the plain rule at three decimals (0.239 either way), but loses more than seed noise on livecell-A172.The plain rule reaches the same mean without that loss; its benefit is in effort (Sec. E.1).
Distance-map proposer, a plug-in source for dense cellsAt N=5N=5 on LIVECell-A172 it reaches 0.463 ± 0.013 F1, above the center head’s 0.433 ± 0.046, with a third of the spread.Its gain is project-specific: on BBBC038 two of three seeds give no usable head (0.222 ± 0.313), and it trails in recall on every other project measured. As a pool source (Sec. 3.2) it can be switched on from the labels, as (2) switches the learned proposer.

Table A14. Alternatives to the fixed configuration and their trade-offs.

We use containment-aware suppression instead: keeping one object per location, using containment as well as IoU. Section E.1 reports its effect on F1 and duplicate detections.

Annotation Effort

Effort-optimal dial

Minimizing cdeleteFP+cdrawFNc_{\mathrm{delete}}\mathrm{FP} + c_{\mathrm{draw}}\mathrm{FN} weights errors by cdraw/cdeletec_{\mathrm{draw}}/c_{\mathrm{delete}}; maximizing FβF_\beta weights them by β2\beta^2. They agree when

β=cdrawcdelete.(8)\beta= \sqrt{\frac{c_{\mathrm{draw}}}{c_{\mathrm{delete}}}}. \tag*{(8)}

Our default F1F_1 dial assumes deleting a false positive costs as much as drawing a missed object.

Eq. (7) bounds the dial at manual cost on its tuning evidence. We retuned it for § 3.4’s check/refine/draw vector: the bound holds on 231/231 tuning curves, 231/231 cross-validated replays, and 221/231 held-out splits (95.7%). The ten violations are small (0.02%–8.0% over manual) and confined to N≤10N \le10 on three projects (coco-kitchen, coco-vehicles, lvis-rare20).

Over 0<wcheck≤wrefine<10 < w_{\mathrm{check}} \le w_{\mathrm{refine}} < 1, SNAP beats manual at the default weights on all 13 projects. The winning region ranges from 0.26 on lvis-rare20 to 0.99 on bbbc038-nuclei.

Containment. Sec. D.9 studies a one-object-per-location suppression rule. With our descriptor, F1 is barely changed (0.239 without it, 0.237 with it, over 13 projects at N=20N=20). We remove about 600 duplicate detections per project. The effort gain grows with the check weight: break-even sits at wcheck≈0.259w_{\mathrm{check}} \approx0.259 (wrefine=0w_{\mathrm{refine}} = 0); past that, we win over 64% of the cited region.

Sensitivity. We recounted stored action counts across all 15 methods under three alternative policies: a looser accept/refine line, a loose match charged as a full redraw, and a required mask fix with the class fix. Order stability is 0.857, top-1 stability 0.889 (Kendall τ\tau mean 0.892, 7560 cells). The harshest policy, charging both corrections as a full redraw, moves the ranking most.

Effort across the cost plane

We split the effort of § 4.4 by project at N=20N=20 in Figure A3.

The cited cost ratios bound wcheckw_{\mathrm{check}} at 0.063–0.473 [35, 51, 65] and wrefinew_{\mathrm{refine}} at 0.268–0.423 [49, 71].

Annotation loop

We replay SNAP’s loop: 36 rounds of five images over a 180-image pool. Each round, the annotator corrects SNAP’s predictions and we refit SnapHead. The configuration matches Tab. A1; the proposer switch fires on LIVECell-A172 from the first refit and on none of the other three projects.

Tab. A15 gives the endpoint. On A172, AMG alone would leave 0.915 to draw; the learned proposer drives the drop. Cumulative effort falls below manual from the second round on, staying below it over 0.988–1.000 of the cited range.

Runtime and Memory Cost

We measured every number here on SNAP’s code paths: one H100 for GPU stages, an Intel Xeon Platinum 8480+ for CPU work, and headless Chrome 153 (onnxruntime-web, WASM, one thread) for the browser.

Per-image cost. In Tab. A16, the encoder stays flat across a 20× range of input pixels, since every image resizes to one square before the trunk. Production AMG, by contrast, costs roughly 480× more and scales with proposal count, not input size. Halving the AMG grid cuts that cost 13–17× while the encoder stays unchanged to 0.2 ms. Cost scales with the square of the AMG point-grid density, so we state that density for every AMG figure.

Figure A4. Time on the annotator’s clock and in the background. (a) Seconds per object to draw or check [49, 71], plus the browser’s decode wait (one CPU thread). (b) GPU stages per image or update, one H100. Bars: median; whiskers: range; log axis.

SnapHead size. Its artifact is 33–143 KB at D=512D=512 (one to ten classes). It is bounded by the ceiling (C+1)(D+1)+2D(C+1)(D+1)+2D for CC classes; the weights make up 95–98% of the serialized bytes.

Adaptation time. Among methods that learn only the class, SnapHead fits in 0.03 s (CPU) against 0.07 s for PerSAM and 0.30 s for the pixel forest, at a higher F1 (0.285 vs. 0.253/0.249). Among methods that also learn object location, ours, gated adapts in 3.83 s against 7.39 s for Mask R-CNN, again at a higher F1 (0.433 vs. 0.419).

Our proposer trains a fixed 300 steps, so cost stays flat as labels grow: 3.70–3.81 s from N=5N=5 to 50 on A172, against 1.41–15.88 s for Mask R-CNN. Figure A5 pairs this cost with § 4.4’s accuracy: SnapHead fits under 0.1 s (median 0.004 s at N=1N=1, 0.066 s at N=50N=50), and ours, gated stays under 4 s.

Figure A5. Mean instance F1 over the five microscopy projects against time to adapt, per budget NN (log x-axis). no training: prompt-only methods. ours, gated: SnapHead fit (CPU) plus proposer training (GPU, only where the gate fires), as in Table 2.

An update lands right after SnapHead’s fit: on the five microscopy projects at N=20N=20, 0.04–0.15 s (median 0.04 s). The optional dial’s cross-validation runs afterward in the background, without blocking predictions: 0.8–54.7 s later (median 23.6 s) on the same five cells.

Browser decoder. A prompt decodes locally in 147–156 ms (median) and 171–182 ms (p95), single-threaded on the CPU above. Box and point prompts cost the same. Over 50 box prompts per image, the displayed mask agrees with the server’s torch decoder at mean IoU 0.993–0.997, never below 0.972.

References

  1. [1]David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Efficient interactive annotation of segmentation datasets with Polygon-RNN++. In CVPR, pages 859–868, 2018. 3
  2. [2]Mykhaylo Andriluka, Jasper R. R. Uijlings, and Vittorio Ferrari. Fluid annotation: A human-machine collaboration interface for full image annotation. In ACM MM, pages 1957–1966, 2018. 3, 5
  3. [3]Anton Antonov, Andrey Moskalenko, Denis Shepelev, Alexander Krapukhin, Konstantin Soshin, Anton Konushin, and Vlad Shakhurov. RClicks: Realistic click simulation for benchmarking interactive segmentation. In NeurIPS, 2024. Datasets and Benchmarks Track; arXiv:2410.11722. 3arxiv.org/abs/2410.11722
  4. [4]Anwai Archit and Constantin Pape. Revisiting foundation models for cell instance segmentation. In Med. Imaging with Deep Learning (MIDL), 2026. 7, 17arxiv.org/abs/2603.17845
  5. [5]Anwai Archit, Luca Freckmann, Sushmita Nair, Nabeel Khalid, Paul Hilt, Vikas Rajshekar, Marei Freitag, Carolin Teuber, Melanie Spitzner, Konstantan Tapia Contreras, Genevieve Buckley, Sebastian von Haaren, Sagnik Gupta, Marian Grade, Matthias Wirth, Günter Schneider, Andreas Dengel, Sheraz Ahmed, and Constantin Pape. Segment anything for microscopy. Nature Methods, 22(3):579–591, 2025. 3, 5, 13
  6. [6]Ignacio Arganda-Carreras, Verena Kaynig, Curtis Rueden, Kevin W. Eliceiri, Johannes Schindelin, Albert Cardona, and H. Sebastian Seung. Trainable Weka segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics, 33(15):2424–2426, 2017. 2
  7. [7]Matthias Arzt, Joran Deschamps, Christopher Schmied, Tobias Pietzsch, Deborah Schmidt, Pavel Tomancak, Robert Haase, and Florian Jug. LABKIT: Labeling and segmentation toolkit for big image data. Frontiers in Computer Science, 4:777728, 2022. 2
  8. [8]Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. In CVPR, 2017. 3arxiv.org/abs/1611.08303
  9. [9]Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR, pages 11700–11709, 2019. 3arxiv.org/abs/1903.10830
  10. [10]Stuart Berg, Dominik Kutra, Thorben Kroeger, Christoph N. Straehle, Bernhard X. Kausler, Carsten Haubold, Martin Schiegg, Janez Ales, Thorsten Beier, Markus Rudy, Kemal Eren, Jaime I. Cervantes, Buote Xu, Fynn Beuttenmueller, Adrian Wolny, Chong Zhang, Ullrich Koethe, Fred A. Hamprecht, and Anna Kreshuk. ilastik: Interactive machine learning for (bio)image analysis. Nature Methods, 16(12):1226–1232, 2019. 2, 5DOI
  11. [11]Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R. Sabuncu, John Guttag, and Adrian V. Dalca. UniVerSeg: Universal medical image segmentation. In ICCV, pages 21438–21451, 2023. 3arxiv.org/abs/2304.06131
  12. [12]Juan C. Caicedo, Allen Goodman, Kyle W. Karhohs, Beth A. Cimini, Jeanelle Ackerman, Marzieh Haghighi, CherKeng Heng, Tim Becker, Minh Doan, Claire McQuin, Mohammad Rohban, Shantanu Singh, and Anne E. Carpenter. Nucleus segmentation across imaging experiments: The 2018 Data Science Bowl. Nature Methods, 16(12):1247–1253, 2019. 5DOI
  13. [13]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Piotr Dollar, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. SAM 3: Segment anything with concepts. In ICLR, 2026. arXiv:2511.16719. 1, 2, 4, 5, 10, 16
  14. [14]Lluís Castrejón, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Annotating object instances with a Polygon-RNN. In CVPR, pages 5230–5238, 2017. 3arxiv.org/abs/1704.05548
  15. [15]Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. YOLO-World: Real-time open-vocabulary object detection. In CVPR, 2024. 2arxiv.org/abs/2401.17270
  16. [17]Claudia Cuttano et al. SANSA: Unleashing the hidden semantics in SAM2 for few-shot segmentation. In NeurIPS, 2025. 2arxiv.org/abs/2505.21795
  17. [18]CVAT.ai Corporation. Computer vision annotation tool (CVAT). Software, https://github.com/cvat-ai/cvat, 2023. Version 2.25.0; DOI and authorship taken from the project’s CITATION.cff. Zenodo all-versions record: https://zenodo.org/doi/10.5281/zenodo.3497105. 1, 3
  18. [19]Carlos García-López de Haro, Caterina Fuster-Barceló, Curtis R. Rueden, Jonathan Heras, Vladimír Ulman, Daniel Franco-Barranco, Adrián Inés, Kevin W. Eliceiri, Jean-Christophe Olivo-Marin, Jean-Yves Tinevez, Daniel Sage, and Arrate Muñoz-Barrutia. SAMJ: fast image annotation on ImageJ/Fiji via segment anything model. Nature Communications, 17:5402, 2026. Preprint: arXiv:2506.02783. 3
  19. [20]Andres Diaz-Pinto, Sachidanand Alle, Vishwesh Nath, Yucheng Tang, Alvin Isani, Muhammad Asad, Fernando Pérez-García, Pritesh Mehta, Wenqi Li, Mona Flores, Holger R. Roth, Tom Vercauteren, Daguang Xu, Prerna Dogra, Sebastian Ourselin, Andrew Feng, and M. Jorge Cardoso. MONAI Label: A framework for AI-assisted interactive labeling of 3D medical images. Medical Image Analysis, 95:103207, 2024. 2, 3
  20. [21]Gergely Dinya, András Gelencsér, Krisztina Kupán, Clemens Küpper, Kristóf Karacs, and Anna Gelencsér-Horváth. SAMannot: A memory-efficient, local, open-source framework for interactive video instance segmentation based on SAM2. Journal of Open Research Software, 2026. Preprint: arXiv:2601.11301. 3DOI
  21. [22]Christoffer Edlund, Timothy R. Jackson, Nabeel Khalid, Nicola Bevan, Timothy Dale, Andreas Dengel, Sheraz Ahmed, Johan Trygg, and Rickard Sjögren. LIVECell—a large-scale dataset for label-free live cell segmentation. Nature Methods, 18(9):1038–1045, 2021. 5DOI
  22. [23]Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (VOC) challenge. IJCV, 88(2):303–338, 2010. 5DOI
  23. [24]Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes challenge: A retrospective. IJCV, 111(1):98–136, 2015. 5
  24. [25]Yarin Gal, Riasht Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In ICML, pages 1183–1192, 2017. 3
  25. [26]Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014. 3arxiv.org/abs/1311.2524
  26. [27]Agrim Gupta, Piotr Dollár, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, pages 5356–5364, 2019. 5arxiv.org/abs/1908.03195
  27. [28]Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. In ICML, 2022. 3
  28. [29]Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2961–2969, 2017. 2, 3, 5
  29. [30]Lucien Hinderling et al. Convpaint. bioRxiv, 2024. 2
  30. [31]Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-Rex2: Towards generic object detection via text-visual prompt synergy. In ECCV, pages 38–57, 2024. 2arxiv.org/abs/2403.14610
  31. [32]Kim et al. SOLA: Referring video object segmentation. arXiv:2412.01136, 2024. 2
  32. [33]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, pages 4015–4026, 2023. 1, 2arxiv.org/abs/2304.02643
  33. [34]Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops (ICCVW), pages 554–561, 2013. 2, 6DOI
  34. [35]Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 128(7):1956–1981, 2020. 3, 6, 8, 21
  35. [36]Hei Law and Jia Deng. CornerNet: Detecting objects as paired keypoints. In ECCV, 2018. 3
  36. [37]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740–755, 2014. 5arxiv.org/abs/1405.0312
  37. [38]Mingxuan Liu, Tyler L. Hayes, Elisa Ricci, Gabriela Csurka, and Riccardo Volpi. SHINe: Semantic hierarchy nexus for open-vocabulary object detection. In CVPR, pages 16634–16644, 2024. arXiv:2405.10053. 2arxiv.org/abs/2405.10053
  38. [39]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In ECCV, 2024. arXiv:2303.05499. 2, 4, 5, 15arxiv.org/abs/2303.05499
  39. [40]Yang Liu et al. Matcher: Segment anything with one shot using all-purpose feature matching. In ICLR, 2024. 2, 18arxiv.org/abs/2305.13310
  40. [41]Vebjorn Ljøsa, Katherine L. Sokolnicki, and Anne E. Carpenter. Annotated high-throughput microscopy image sets for validation. Nature Methods, 9(7):637, 2012. 5DOI
  41. [42]Anish Madan, Neehar Peri, Shu Kong, and Deva Ramanan. Revisiting few-shot object detection with vision-language models. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. arXiv:2312.14494. 2arxiv.org/abs/2312.14494
  42. [43]Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 2, 6arxiv.org/abs/1306.5151
  43. [44]Franco Marchesoni-Acland and Gabriele Facciolo. IAdet: Simplest human-in-the-loop object detection. arXiv preprint arXiv:2307.01582, 2023. 3
  44. [45]Markus Marks, Uriah Israel, Rohit Dilip, Qilin Li, Changhua Yu, Emily Laubscher, Ahamed Iqbal, Elora Pradhan, Ada Ates, Martin Abt, Caitlin Brown, Edward Pao, Shenyi Li, Alexander Pearson-Goulart, Pietro Perona, Georgia Gkioxari, Ross Barnowski, Yisong Yue, and David Van Valen. CellSAM: A foundation model for cell segmentation. Nature Methods, 22(12):2585–2593, 2025. 3, 13DOI
  45. [46]Meta AI. SAM 3 code and model license. GitHub repository facebookresearch/sam3, 2025. License file, last updated 2025-11-19. 8
  46. [47]Meta AI. SAM 3.1 checkpoint release. Release note, https://github.com/facebookresearch/sam3/blob/main/RELEASE_SAM3p1.md, 2026. Checkpoint and blog post only; no accompanying paper. Cite together with [13]. 2
  47. [48]napari contributors. napari: a multi-dimensional image viewer for Python. Software, https://napari.org, 2019. Concept (all-versions) DOI; software, no journal paper. 3
  48. [49]Leon H. Oehme, Jonas Boysen, Zhangkai Wu, Anthony Stein, and Joachim Müller. Orchestrating segment anything models to accelerate segmentation annotation on agricultural image datasets. Frontiers in Artificial Intelligence, 8:1748468, 2026. 3, 6, 8, 21, 23DOI
  49. [50]Marius Pachitariu, Michael Raiden, and Carsen Stringer. Cellpose-SAM: Superhuman generalization for cellular segmentation. bioRxiv, 2025. Preprint, not peer reviewed. 3DOI
  50. [51]Dim P. Papadopoulos, Jasper R. R. Uijlings, Frank Keller, and Vittorio Ferrari. We don’t need no bounding-boxes: Training object class detectors using only human verification. In CVPR, pages 854–863, 2016. 3, 6, 8, 21arxiv.org/abs/1602.08405
  51. [52]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In ICLR, 2025. OpenReview Ha6RTeWMd0; arXiv:2408.00714. 1, 2, 3arxiv.org/abs/2408.00714
  52. [53]Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS, 2015. 3arxiv.org/abs/1506.01497
  53. [54]Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhaoyang Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, and Lei Zhang. DINO-X: A unified vision model for open-world object detection and understanding. arXiv preprint arXiv:2411.14347, 2024. 2arxiv.org/abs/2411.14347
  54. [55]Peter Robichaeu, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow100-VL: A multi-domain object detection benchmark for vision-language models. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2025. 5
  55. [56]Roboflow, Inc. Roboflow: Annotate, train and deploy computer vision models. Commercial platform, https://roboflow.com, 2026. Software, no paper. The company’s citable research artifact is the RF100 benchmark [16]. 1, 3
  56. [57]Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. LabelMe: A database and web-based tool for image annotation. IJCV, 77(1–3):157–173, 2008. 3DOI
  57. [58]Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers. Cell detection with star-convex polygons. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 265–273, 2018. 3, 19DOI
  58. [59]Mehdi Seifi, Damian Dalle Nogare, Juan Manuel Battagliotti, Vera Galinova, Ananya Kedige Rao, Pierre-Henri Jouneau, Anwai Archit, Constantin Pape, Johan Decelle, Florian Jug, and Joran Deschamps. FeatureForest: The power of foundation models, the usability of random forests. npj Imaging, 3:32, 2025. 2, 5DOI
  59. [60]Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018. 3
  60. [61]Konstantin Sofiuk, Ilia A. Petrov, and Anton Konushin. Reviving iterative training with mask guidance for interactive segmentation. In ICIP, pages 3141–3145, 2022. 3
  61. [62]Christoph Sommer, Christoph Straehle, Ullrich Köthe, and Fred A. Hamprecht. ilastik: Interactive learning and segmentation toolkit. In IEEE Int. Symp. Biomed. Imaging (ISBI), pages 230–233, 2011. 2DOI
  62. [63]Carsen Stringer and Marius Pachitariu. Cellpose3: One-click image restoration for improved cellular segmentation. Nature Methods, 22(3):592–599, 2025. 19DOI
  63. [64]Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu. Cellpose: A generalist algorithm for cellular segmentation. Nature Methods, 18(1):100–106, 2021. 3, 4DOI
  64. [65]Hao Su, Jia Deng, and Li Fei-Fei. Crowdsourcing annotations for visual object detection. In AAAI Human Computation Workshop, 2012. 21
  65. [66]Yangpeng Sun et al. VRP-SAM: SAM with visual reference prompt. In CVPR, 2024. 2arxiv.org/abs/2402.17726
  66. [67]Sönke Tenckhoff, Mario Koddenbrock, and Erik Rodner. Feedback-driven object detection and iterative model improvement for accurate annotations. arXiv preprint arXiv:2411.19835, 2024. 3arxiv.org/abs/2411.19835
  67. [68]Carolin Teuber, Anwai Archit, Tobias Boothe, Peter Ditte, Jochen Rink, and Constantin Pape. Evaluating vision foundation models for pixel and object classification in microscopy. arXiv preprint arXiv:2603.19802, 2026. 2arxiv.org/abs/2603.19802
  68. [69]Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software. Software, https://github.com/HumanSignal/label-studio, 2020–2025. Open-source software from HumanSignal (formerly Heartex); citation as given in the project README. 1, 3
  69. [70]Yi-Jen Tsai, Yen-Yu Lin, and Chien-Yao Wang. Few-shot semantic segmentation meets SAM3. arXiv preprint arXiv:2604.05433, 2026. 2, 15arxiv.org/abs/2604.05433
  70. [71]Aayush Kumar Tyagi, Vaibhav Mishra, Prathosh A. P., and Mausam. BoxCell: Leveraging SAM for cell segmentation with box supervision. arXiv preprint arXiv:2311.17960, 2023. 3, 6, 21, 23arxiv.org/abs/2311.17960
  71. [72]Jasper R. R. Uijlings, Mykhaylo Andrіluka, and Vittorio Ferrari. Panoptic image annotation with a collaborative assistant. In ACM MM, 2020. 3
  72. [73]Alexandra Dunnum VandeLoo, Nathan J. Malta, Saahil Sanganiyera, Emilio Aponte, Caitlin van Zyl, Danfei Xu, and Craig Forest. SAMCell: Generalized label-free biological cell segmentation with segment anything. PLOS ONE, 20 (9):e0319532, 2025. 3
  73. [74]Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. YOLOE: Real-time seeing anything. In ICCV, 2025. arXiv:2503.07465. 2, 4, 5, 15arxiv.org/abs/2503.07465
  74. [75]Wei Wang. X-AnyLabeling: A unified desktop platform for AI-assisted data annotation. Software, https://github.com/CVHub520/X-AnyLabeling, 2023. Citation as given in the project README and CITATION.cff; GPL-3.0. 3
  75. [76]Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E. Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In ICML, pages 9919–9928, 2020. 2arxiv.org/abs/2003.06957
  76. [77]Hallee E. Wong, Jose Javier Gonzalez Ortiz, John Guttag, and Adrian V. Dalca. MultiverSeg: Scalable interactive segmentation of biomedical imaging datasets with in-context guidance. In ICCV, 2025. arXiv:2412.15058. 3arxiv.org/abs/2412.15058
  77. [78]Gui-Song Xia, Xiang-Gen Wang, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. DOTA: A large-scale dataset for object detection in aerial images. In CVPR, 2018. 5arxiv.org/abs/1711.10398
  78. [79]Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S. Huang. Deep interactive object selection. In CVPR, pages 373–381, 2016. 3arxiv.org/abs/1603.04042
  79. [80]Ke Yu, Stephen Albro, Giulia DeSalvo, and Suraj Kothawade. Two-step active learning for instance segmentation with uncertainty and diversity sampling. In ICCV Workshop on Uncertainty Quantification for Computer Vision (UNCV), 2023. 3arxiv.org/abs/2309.16139
  80. [81]Syed Waqas Zaman, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. iSAID: A large-scale dataset for instance segmentation in aerial images. In CVPRW, pages 28–37, 2019. 5
  81. [82]Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Personalize segment any­thing model with one shot. In ICLR, 2024. 2, 18arxiv.org/abs/2305.03048
  82. [83]Yu Zhang, Yun Wang, Haidong Zhang, Bin Zhu, Siming Chen, and Dongmei Zhang. OneLabeler: A flexible system for building data labeling tools. In ACM Conference on Human Factors in Computing Systems (CHI), 2022. 3DOI
  83. [84]Zhengning Zhang, Lin Zhang, Yue Wang, Pengming Feng, and Ran He. ShipRSImageNet: A large-scale fine-grained dataset for ship detection in high-resolution optical remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:8458–8472, 2021. 6DOI
  84. [85]Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. IJCV, 127(3):302–321, 2019. 5arxiv.org/abs/1608.05442
  85. [86]Quan Zhou, Shaoqing Zhai, Qiang Hu, Jia Chen, Qiang Li, and Zhiwei Wang. Mask to concept: Auto-promptable SAM3 via efficient test-time concept embedding search for few-shot annotation. In Medical Image Computing and Computer Assisted Intervention (MICCAI), 2026. arXiv:2606.26711. 2arxiv.org/abs/2606.26711
  86. [87]Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv:1904.07850, 2019. Preprint, not peer reviewed. 3arxiv.org/abs/1904.07850

Paper details

vlnc:W_6yf5xk7k
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents