WorksComputer Science
Scoring, Not Learning: Decomposing Catalog-Consolidation Gains in Web-Scraped Retail Product Recognition
Abstract
Retail product catalogs assembled from web image search are cheap to build and pervasively noisy: queries return packaging shots instead of products, and entries proliferate under names that cannot be distinguished visually. Consolidating such a catalog can look like an alternative to scaling the model. We show this is not a comparison between two ways of improving a recognizer, but between improving a recognizer and changing what counts as correct. We study a public archive of 165,769 cheese images across 1,824 query-named entries. A packaging filter and near-duplicate removal reduce it to a 998-class recognition set on one fixed, group-aware split. Canonicalizing entries to their base type (844 classes) raises top-1 accuracy by 4.0 points. Scoring the unchanged model against the merged label space, by summing its predicted probability within each canonical group, recovers +4.27 points with no retraining (+4.25 over six seeds). Retraining on the consolidated catalog then adds -0.100.53, +0.080.65 and -0.100.65 points at 28, 50 and 88 M parameters, indistinguishable from zero at every capacity ( is the across-seed standard deviation). That component is specific, not mechanical: random merges with the identical size profile yield only +0.68 points, and the three backbones agree to within 0.1 points. But it is overwhelmingly category coarsening, not alias correction: across the 37 merged groups, type coarsenings supply 94% of the gain and alias collapses only 7%. We also detect no interaction between consolidation and backbone size: tripling backbone parameters is worth +1.500.57 points on the raw catalog and +1.4200.44 on the canonical one, a within-seed interaction of -0.08 [-0.79,+0.63]. On this archive, the reported consolidation gain primarily reflects the evaluation label space, and separating that out costs one evaluation
Introduction
Recognizing individual products is a core capability for physical retail. Shelf auditing, self-checkout, planogram compliance and catalog maintenance all reduce to deciding which of many visually similar items an image depicts. Curated retail benchmarks exist [3, 15, 19, 25, 33], but each covers a small slice of a real assortment, and extending coverage is expensive. Issuing product names as web-search queries is the standard shortcut.
This approach is cheap, but it inherits two problems specific to retail imagery. A query such as “brie cheese” returns many photographs of supermarket packaging rather than the product, so branding becomes a confound across unrelated items. The label space is also a product catalog rather than a designed taxonomy: “cheddar”, “mature cheddar” and “orkney extra mature cheddar” appear as three query-named classes despite being difficult to separate at web-image resolution. Collapsing entries of this kind to a base type is routine work for anyone maintaining a product database.
We study a public archive of 165,769 readable cheese images in 1,824 query-named folders. It is a deep fine-grained slice of one grocery category, and it exhibits both problems acutely. A third compounds them: a severely long-tailed distribution over those already redundant entries. The images are web photographs rather than shelf or checkout captures, so what transfers to a store deployment is the catalog-side finding, where the label space rather than the imaging condition is the variable.
Data-centric work often reports that improving the data can rival scaling the model [14, 24, 27], which would make a catalog-hygiene project an alternative to a larger GPU budget for a retail team. That work, however, keeps the evaluation label space fixed § 2, whereas catalog consolidation changes it, and that difference hides an asymmetry. Canonicalizing entries to their base type does not merely recover accuracy a redundant label space was suppressing. It also changes the task, replacing a 998-way problem with an easier 844-way one, whereas scaling the backbone leaves the task fixed. Comparing the two head-to-head therefore mixes a definitional effect with whatever retraining adds. We separate them by scoring a single frozen model under both label spaces, a control that costs one evaluation pass. We therefore treat consolidation throughout as the adoption of a coarser, predefined label space, and ask what accuracy gain that space provides. We do not treat it as a demonstration that the merged entries name one product, which our name-based map does not establish.
Our contributions are:
A pipeline for turning a noisy web-scraped product archive into a usable recognition set: a packaging filter and a duplicate-group split that removes the pixel-identical train–test overlap a filename split leaves behind.
A decomposition of the consolidation gain into a definitional component and a retraining residual, using hierarchical marginalization as the rescoring control it requires. The definitional component accounts for the entire gain within our resolution, the residual is indistinguishable from zero at all three capacities, and random-merge nulls show the component is specific to the merged classes rather than to class-count reduction (Table 3, § 6.3).
No detectable interaction between curation and capacity over a parameter range, with comparisons paired within each of six seeds.
| Model | raw | marg. | retr. | defin. | resid. |
| Tiny (28 M) | 25.30 | 29.60 | 29.51 | +4.30 ± .24 | −0.10 ± .53 |
| Small (50 M) | 26.11 | 30.36 | 30.44 | +4.25 ± .09 | −0.08 ± .65 |
| Base (88 M) | 26.80 | 31.03 | 30.93 | +4.23 ± .20 | −0.10 ± .65 |
| CLIP probe, frozen | 20.44 | 24.21 | 23.34 | +3.76 | −0.87 |
| Small, filter 0.7 | 25.66 | 30.75 | 30.70 | +5.09 | −0.05 |
Table 3. Decomposing the merge gain: six-seed mean top-1 for the three ConvNeXt backbones. Accuracy values are percentages; components are percentage points with six-seed standard deviations. Each row’s own raw model is rescored under the same canonical map by the marginal rule. The last two rows are single runs, and the 0.7-filter row rebuilds the split, so only its decomposition is comparable.
Related Work
Vision-language pretraining. CLIP [26] learns aligned image-text representations that transfer well downstream. We use a frozen ViT-L/14 encoder [12] via OpenCLIP [16], with the original openai weights, for feature extraction and filtering. Zero-shot prompt matching performs poorly on the rare proper-noun names in our archive, and § 6.2 shows that this poor performance persists after both prompt engineering and the removal of label redundancy, leaving limited text grounding as an untested candidate explanation.
Fine-grained visual categorization. FGVC benchmarks [6, 30, 32] define explicit category spaces, although supervision can remain noisy: Food-101 deliberately retains noise in its training set. High-capacity backbones such as ConvNeXt [22] provide a basis for fine-tuning. Chang et al. [7] question which label granularity users need, and Bertinetto et al. [4] weigh hierarchy-aware error severity. Reading a fine-grained classifier out at a coarser level by summing the predicted probabilities of its child classes is the standard marginalization readout in hierarchical classification [10, 17]; we do not propose this readout but apply it as a control, to separate a change in the evaluation label space from a change in the model. Our label space is itself an artifact of scraping, never designed to be mutually exclusive.
Learning from web supervision and label noise. WebVision [21] studies learning from noisy web labels, and Krause et al. [20] show that for fine-grained recognition the volume of noisy web data can outweigh its noisiness. WebFG-496 [28] formalizes weakly supervised fine-grained recognition. This line of work largely treats noise as instance-level (wrong image for a correct class). The dominant problem in our archive is class-level. Distinct folders are not separable at web-image resolution, whether because they name one product or because they name products of one type, and no per-image relabeling can fix either.
Data-centric learning and dataset curation. A growing body of work argues that dataset quality, not model scale, limits progress. Label errors reorder model rankings [24], and much of ImageNet’s residual error stems from label definitions rather than model failure [5, 31]. Relabeling improves fixed architectures [35], pruning can improve on power-law data scaling [27], and semantic deduplication reduces training cost with little accuracy loss [1]. DataComp and MetaCLIP demonstrate gains from data curation under controlled training procedures [14, 34]. Most of these operations leave the evaluation label space fixed. Ours does not, and that is the case we study: when curation changes the label space, part of the measured gain is definitional.
Retail product recognition. Retail benchmarks cover shelf-object detection [15], checkout imagery [33], SKU-level recognition [3, 25], grocery hierarchies [19], instance segmentation [13] and grocery-category classification [18]. These tasks have deliberately defined annotations and category spaces. Our archive inherits its labels from search queries, so we treat catalog structure as the primary variable.
Long-tailed recognition. Reweighting the loss toward minority classes raises their accuracy but can reduce overall accuracy under naturally imbalanced test distributions, as we observe. We use inverse-frequency weights , for training images over classes with in class , not the effective-number scheme of Cui et al. [9].
The Archive
The dataset derives from the public Cheese Pics collection on Kaggle [2], which contains web-search images in folders named by cheese. We discard 8 unreadable files. Table 1 gives its statistics and those of every subset below. We contribute no new images: the contribution is the filtering, reorganization and evaluation of an existing archive.
| Quantity | Value |
| Folders (cheese names) | 1,824 |
| Total readable images | 165,769 |
| Images kept after packaging filter | 62,174 (37.5%) |
| Distinct photographs after dedup | 33,583 (54.0% of kept) |
| Cheeses with distinct images | 998 |
| Distinct images/class, median / mean | 35 / 36.1 |
| Recognition subset, images (cap 80) | 35,986 |
| train / val / test | 25,325 / 5,344 / 5,317 |
| distinct photograph groups, test | 5,086 |
| Canonical classes after merge (Stage 3) | 844 |
Table 1. Archive and label-set statistics. The recognition subset exceeds the distinct-photograph count because a duplicate group spanning several folders contributes one image to each (§ 4.2).
Both defects of § 1 are prevalent (Figure 1), and folder sizes are highly uneven. The categories retaining at least twenty distinct clean images after Stage 1 filtering and deduplication form the recognition set, which remains both long-tailed and redundant (§ 4.4).
Figure 1. The two defects. Top: the Stage-1 probe scores packaging shots near zero and product shots near one. The leftmost image shows a packaged snack rather than unpackaged cheese, returned by “cheddar cheese”. Bottom: four distinct photographs from folders the scrape treats as four classes and Stage 3 maps to cheddar. This is why no per-image relabeling fixes a defect of this kind.
Canonicalization aligns cheese names to a tabular catalog scraped from cheese.com and redistributed by the TidyTuesday project [29], used as a fixed external reference. We redistribute no catalog text.
Method
Figure 2 summarizes the pipeline: one data-filtering stage and two treatments of the recognition labels. Stage 1 and deduplication determine which images enter the recognition set. Stage 2 recognizes the raw folder name. Stage 3 canonicalizes it and is evaluated both by rescoring Stage 2’s frozen model and by retraining.
Figure 2. The pipeline. Near-duplicate grouping reduces the filtered images to 33,583 distinct photographs before any split is drawn. Stage 3 changes the label space, not the weights, so Stage 2’s frozen model can be read out over it without retraining. The two branches illustrate this comparison, decomposed in Figure 3. Six-seed ConvNeXt-Small means Table 3.
All CLIP features are 768-dimensional -normalized embeddings from ViT-L/14.
Stage 1: Packaging Filter
The filter is a binary classifier: does an image depict unpackaged cheese (“good”) or commercial packaging (“not good”)? We hand-label 208 images (105 good, 103 packaging) and fit a logistic regression on their CLIP features, with standardized features and balanced class weights. 5-fold stratified cross-validation selects . We then run the probe over all 165,769 images and keep those scoring . Discarding packaging fixes the task as recognition of the unpackaged product, not packaged-SKU recognition, where the label is on the wrapper and packaging is the signal.
Deduplication and Leakage Control
Query-based scraping returns the same photograph more than once: the same JPEG under two filenames in one folder, and the same image at two resolutions under two cheese names. Splitting on filenames therefore leaks. Shuffling each folder’s filenames into a 70/15/15 train/validation/test split leaves 60.5% of test images with a pixel-identical twin in training. The fine-tuned ConvNeXt-Small model of § 4.3 scores 43.8% on these test images, compared with 28.7% on the remainder: such a split measures memorization as much as recognition.
We therefore group the pool by image identity before splitting. Each image carries an MD5 hash and two 64-bit perceptual hashes, a dHash over a luminance gradient and an aHash against the image mean. Two images join a group when their MD5s match, or when both hashes agree to within two bits. Candidate pairs come from banded lookup, so the 1.9 B pair space is never enumerated. The threshold is not tuned for effect: at two bits the largest group stays at its exact-duplicate value of 131, whereas transitive chaining inflates it to 617 at six bits. Across 400 sampled merges missed by MD5 matching, the median pixel correlation is 0.9996, consistent with re-encodings of the same photograph. The pairs of four-bit threshold would add have a median correlation of 0.63. The sampled pairs give an estimated false-merge rate of 0.8%, and a false merge discards a photograph rather than creating leakage.
Two rules then build the split: each group contributes at most one image per folder, and every surviving member lands in the same split. No identified duplicate group therefore straddles train and test, even under two cheese names. Auditing candidate pairs added at four bits finds 49 near-identical train–test pairs, so at most 49 test images (0.9%) carry an audited near-duplicate in training, against the 60.5% a filename split leaves. The audit does not cover all possible pairs. The first rule deduplicates within folders but not across them, so one photograph can still appear in the test split under several names. The 5,317 test images occupy 5,086 groups, of which 103 carry more than one canonical label (304 images, 5.7%). Conflicting labels on identical inputs impose unavoidable errors, though not one per image, since near-duplicate members may still receive different predictions. These repeated photographs also make image-level outcomes dependent (§ 6.4).
The redundancy is severe (Table 1): 6,164 duplicate groups span more than one catalog entry, and 5,566 still span distinct labels under the recognition merge map, with unmapped entries kept separate. Holding the protocol constants fixed ( unique images per class, cap 80, 70/15/15), only 998 of the 1,238 entries that qualify based on filtered file counts still qualify based on distinct photographs.
Stage 2: Fine-Grained Type Recognition
On the 998-class deduplicated clean set, we compare three approaches in increasing order of cost.
Zero-shot CLIP. Each image is matched to the textual prompt “a photo of {name}” for every cheese, and the argmax is the prediction. A single hand-written prompt confounds text grounding with prompt design. We therefore also report an eight-template ensemble, a variant using names stripped of the trailing “cheese”, and a variant mapped to the canonical label space of § 4.4.
Linear probe. We fit logistic regression () on standardized frozen CLIP features from the training split and evaluate it on the test split.
Fine-tuned backbone. We fine-tune ConvNeXt [22] from torchvision’s IMAGENET1K_V1 weights end-to-end with AdamW [23] (, weight decay 0.05), a cosine schedule, linear smoothing 0.1, mixed precision, and RandAugment [8] over random-resized crops. We report ConvNeXt-Tiny, -Small and -Base (28, 50 and 88 M parameters), all trained for the full 25 epochs on the identical deduplicated split (§ 4.2), a 3.1 capacity range at a fixed epoch budget.
Stage 3: Base-Type Canonicalization
This stage canonicalizes each entry to its base type by name alone. Whether a given group collects aliases of one product or distinct products of one type is not determined by the procedure, and § 6.3 shows the distinction matters. Using cheese.com names as anchors, the shortest catalog name that is a suffix of the folder name becomes the canonical label, so “orkney extra mature cheddar” becomes cheddar. A stop-list prevents marketing descriptors such as gold or reserve from becoming a base. For a folder whose nonempty, normalized token sequence, after removing trailing cheese tokens, is , with anchor set and trailing-token frequencies ,
where collects the suffixes of of at most three tokens that lie in , excluding single-token suffixes in the modifier stop-list , and is its shortest element. The anchors are normalized catalog names, synonyms and alternative spellings of at least three characters from records with recognized type/rind metadata. Here counts trailing tokens over the 1,238 eligible folders, , and maps goats to goat and leaves other names unchanged. Restricted to the 998-entry recognition set, Eq. (1) collapses 191 entries into 37 groups, reducing the label count to 844. The largest groups are blue (43 folders), cheddar (28), brie (11), gouda (10) and goat (9). The stop-list must also block trailing tokens that are not cheese types at all (designations of origin, milk animals, nationalities), which otherwise clear and pull unrelated products together, putting murazzano dop and pecorino toscano dop onto dop. We then both (a) rescore the existing 998-class model under the merged labels and (b) retrain on the 844 classes, keeping the split identical.
What the unchanged model predicts in the merged space is a modeling choice, not a formality. Let map raw class to its canonical label, and let be the frozen model’s softmax probability. We marginalize:
The five highest-scoring groups give top-5. Thus is the model’s posterior over canonical labels, without retraining. A weaker alternative, argmax-then-map, takes the 998-way argmax and translates that one label, so a group whose mass is split across several raw entries can lose to a single entry outside the group. We report both, because argmax-then-map is what a curation report implicitly computes when it relabels predictions. Marginalizing is not merely the more generous rule: it is not monotone, losing 32 test images the raw argmax got right while gaining 259. The merge uses only string normalization and external catalog matching, with no visual features, model predictions or test-set information.
Experimental Setup
ConvNeXt test accuracies and paired comparisons use saved per-image predictions, not separate training-log evaluations. Differences are computed before rounding; accuracy differences are in percentage points. Unless otherwise stated, recognition experiments use the fixed deduplicated split of § 4.2 (25,325 / 5,344 / 5,317 images over 998 categories, capped at 80 per type, a cap no deduplicated class reaches) on one NVIDIA RTX 4060 (8 GB). Stage-1 numbers are over the full archive. Frozen-feature baselines use the same splits as the fine-tuned models, so Table 2 carries no protocol difference between rows.
| Method | top-1 | top-5 |
| Uniform random baseline | 0.10% | 0.50% |
| Zero-shot CLIP (text-name match) | 5.4% | 11.1% |
| CLIP linear probe (frozen) | 20.4% | 37.7% |
| ConvNeXt-Tiny fine-tuned, 25 ep | 25.6% | 44.1% |
| ConvNeXt-Small fine-tuned, 25 ep | 26.3% | 44.7% |
| ConvNeXt-Base fine-tuned, 25 ep | 26.6% | 45.4% |
Table 2. Stage 2: cheese-type recognition (998 classes). Every row is scored on the same held-out test split () of the deduplicated 80-capped subset. The probe is fit on that split’s 25,325 training images. ConvNeXt-Small remains the reference model for § 6.3 to § 6.4, with six-seed results in Table 3. ConvNeXt rows here use seed 42. No row is bolded: no single capacity step separates from its neighbor (Table 4).
The backbone, merged and class-balanced runs share the fine-tuning configuration of § 4.3, so only the studied variable changes. Batch size is the one exception (64 for Tiny, 48 for Small, 32 for Base), set by the 8 GB memory ceiling rather than tuned. It is therefore not fixed along the capacity axis, and confounds every capacity step.
Results
Stage 1: Packaging Filter
Over the full archive, the probe labels 62,174 images (37.5%) as real cheese and 103,595 as packaging or otherwise unusable. The score distribution is bimodal but not cleanly separated, with 9.4% of images in the ambiguous band between 0.4 and 0.6.
The filter gates every downstream result, so we evaluate it out-of-fold. On the 208 hand-labeled images, 5-fold cross-validation gives a precision of 90.4% (95% Wilson interval [83.2, 94.7%]) and a recall of 89.5% (82.2, 94.0%) for the “good” class. Two consequences matter downstream. The filter discards about one genuine image in ten, and precision on a near-balanced label set overstates deployment precision: assuming the cross-validation error rates transfer to the full archive, the observed 37.5% retention implies a true prevalence near 34.8% and precision near 83% among retained images. Contamination after recognition-set selection is unmeasured.
Stage 2: Recognition
Table 2 compares methods on the same held-out test split. Zero-shot CLIP reaches 5.4%, some 50× the uniform random baseline (998-way, 0.10%) but far from usable, consistent with, but not establishing, limited text grounding of rare category names. A linear probe on the same frozen features reaches 20.4%: discriminative structure is present in the frozen image representation even where the text encoder cannot address it. End-to-end fine-tuning adds a further 6 points.
Tested prompt variants do not close the gap. An eight-template ensemble reaches 4.8%, slightly below the single prompt. A cleaned-name prompt (“a photo of {name}, a type of cheese”) gives 5.5%. Mapping ensemble predictions to canonical labels reaches 7.5%, using the maximum similarity within each group rather than the sum of probabilities. The spread across four protocols is under three points, so prompt design and label redundancy do not close the gap to the probe.
Capacity buys little at this budget. Top-1 moves on this split, and neither adjacent step separates over six seeds (Table 4).
| Comparison | McNemar | ||
| Tiny → Small (capacity) | +0.8 | 322 / 363 | 0.13 |
| Small → Base (capacity) | +0.2 | 317 / 328 | 0.69 |
| Tiny → Base (capacity) | +1.0 | 302 / 354 | 0.046 |
| Small → Base, merged labels | +1.0 | 279 / 333 | 0.03 |
| Merged → merged + class-bal. | −4.1 | 429 / 213 | |
| Raw → rescored, marginal | +4.3 | 32 / 259 | |
| argmax-then-map → marginal | +1.2 | 32 / 98 | |
| Rescored → retrained (resid.) | −0.2 | 319 / 307 | 0.66 |
Table 4. Exact McNemar tests on paired per-image outcomes, shared test split (), with the discordant counts used in each test: correct before and wrong after, wrong before and correct after.
Validation top-1 peaks at 24.8% for Tiny, 25.8% for Small and 26.5% for Base, in each case before epoch 25, and none has collapsed (Figure 4). We therefore compare at a fixed 25-epoch budget, which equalizes epochs but not compute or optimizer steps.
Figure 4. Validation top-1 per epoch, seed 42. The three raw-label backbones each peak before epoch 25 and their final accuracies differ by under two points, so Table 2 compares them at a fixed epoch budget. The two 844-class runs are plotted for reference. Inset: epochs 15 to 25.
Stage 3: Decomposing the Merge Gain
Canonicalizing categories to their base type raises ConvNeXt-Small top-1 by 4.3 points over six seeds, from 26.1% to 30.4% (Table 3). Writing , and for raw, marginally rescored (Eq. (2)) and retrained accuracies, we define and , so . We report six-seed means throughout and condition on the reference seed-42 run only for the paired per-image tests.
Definitional (+4.3 points). Scoring the unchanged 998-class model against the merged label space gives 30.36% top-1 over six seeds, against 26.11% raw, a +4.25-point gain (Table 3). Only the label space used to interpret the posterior changes, so predictions previously marked wrong for naming “mature cheddar” when the label said “cheddar” now count. This already matches the 30.44% reached by retraining.
Retraining residual (+0.1 points). Over six seeds, retraining on the 844 merged classes reaches 30.44%, indistinguishable from the rescored 30.36% (+0.08 ± 0.65; Table 3). On the reference seed-42 run the retrained model reaches 30.47%, slightly below its rescored figure, and a paired McNemar test does not separate them (, ). The residual is indistinguishable from zero at all three capacities (§ 6.4), so retraining on the consolidated catalog buys nothing we can resolve.
The rescoring rule carries this conclusion. Under argmax-then-map the same model reaches only 29.4%, leaving a +1.0-point residual, positive in all six seeds (+1.01±0.58, ), that reads as a genuine gain from retraining. A curation report that relabels predictions rather than re-reading the posterior will therefore credit about a point to retraining that the frozen model’s own posterior already delivers.
Is the definitional gain merely a smaller label space? Collapsing 998 classes to 844 makes the task easier by construction, so we compare it against nulls drawing 10,000 random merges with the identical 191 37 size profile, holding the model, predictions and split fixed: this class-set-matched null merges exactly the 191 entries Stage 3 merges and shuffles only their grouping. Under marginal rescoring it gains +0.68 against the observed +4.27, which exceeds all 10,000 draws (permutation ).
The remaining comparisons are computed under argmax-then-map, where the corresponding observed gain is +3.03, ensuring a like-for-like comparison. A permute-all null randomizes which entries merge as well as how they group, and gains +0.22 against the class-set-matched +0.42. That splits the component three ways: class-count reduction +0.22, high-frequency entries +0.19, and grouping the specifically confusable ones +2.61. A comparator merge built greedily from the model’s own test confusion reaches +3.71, so exploiting the model’s confusions does not do dramatically better. Being test-informed and greedy, it bounds nothing.
What canonicalization merges, and why it matters. The merge does two different things: it collapses several names for one product (an alias collapse) and it coerces distinct products into a broader type (a type coarsening). We hand-label each of the 37 merged groups as one or the other and split the gain accordingly. It is overwhelmingly the second kind: the 29 type-coarsening groups supply +4.01 points (94% of the definitional gain), while the 8 alias collapses supply only +0.28 (7%). Leave-one-out sensitivities localize it further: blue, which gathers 43 distinct blue cheeses into one type, accounts for 1.82 points (43%) on its own, cheddar (28 entries) for 0.87, and the five largest groups for 74% (Figure 3b). These compete under marginalization rather than partition the gain, so they need not sum to the total (4.29 against 4.27). This reading is also the paper’s practical diagnostic. The alias-versus-coarsening split separates the part of a curation gain that is duplicate-label correction from the part that is merely an easier task, and on this archive the gain is almost all the latter: a curation step that merged genuine aliases would show the opposite profile.
Figure 3. (a) Raw-label evaluation, two canonical rescoring rules, and canonical-label retraining, across three backbone sizes (six-seed means). The filled marker shows the accuracy obtained from the frozen model’s posterior over canonical labels, without retraining. It matches the retrained model at every size. (b) Leave-one-out sensitivity: the gain is concentrated in blue, which coarsens 43 entries into a broad type rather than collapsing aliases.
The definitional component is stable across recognizers. The three backbones agree on this component to within 0.1 points across a 3.1× parameter range, and a frozen CLIP probe gives +3.76 (Table 3); a stricter 0.7 filter raises it to +5.09 on a non-comparable split. Two cautions bound this. The probe’s lower value shows the component still depends on feature quality when features are weak enough. And it is a net change under a new decision rule, not a count of recovered errors: marginalizing can also lose images the raw argmax got right, when a merged group’s pooled mass over-takes the correct singleton. The gains under both rules are bounded above by the raw error rate, so we cannot extrapolate beyond the 20 to 27% band measured here.
No detectable interaction with capacity. If a redundant label space suppressed the return on capacity, the capacity step would be larger once the catalog is canonical. Over the full 3.1× range it is not: paired within six seeds, Tiny-to-Base is worth +1.50 ± 0.57 raw () and +1.42 ± 0.44 canonical (). Comparing two such estimates is not itself a test. We therefore form the interaction within each seed, . It is −0.08, with a 95% interval of [−0.79, +0.63]. That interval still admits effects large relative to the 1.50-point capacity benefit itself. A single seed suggests otherwise. At seed 42 the Small-to-Base step is +0.21 raw and +1.02 canonical, the pattern one would report from one run.
Class-balanced loss. Inverse-frequency weighted cross-entropy on the merged task (one seed) reduces micro top-1 from 30.4% to 26.3% (Table 4) and raises macro-recall from 25.3% to 25.8%. It buys a tail gain (17.7% → 20.4% in the rarest tertile by training count) with a larger head loss (37.4% → 28.8%). Micro accuracy follows the head classes that dominate the test distribution.
Statistical Reliability
We assess uncertainty in two ways: paired comparisons on the reference run, and repeated training with the split held fixed.
A top-1 accuracy near 26% on this test split has an image-level 95% Wilson half-width of about 1.2 points. Individual intervals do not give the interval of a paired difference. Because every model is scored on the same images, we compare their paired outcomes with exact McNemar tests [11] (Table 4). Both the Wilson intervals and McNemar tests treat test images as independent, which the residual cross-folder duplication of § 4.2 violates: pairing is appropriate, since all runs are scored on the identical 5,317 images, but those images are not independent draws. We therefore rest substantive conclusions on group-aware intervals rather than on these -values. Percentile intervals from 5,000 bootstrap resamples of the 5,086 photograph groups give +4.27 [+3.61, +4.91] definitional and −0.23 [−1.09, +0.68] residual, so the dependence widens intervals without changing either conclusion. Restricting to the 5,013 images whose group carries one canonical label gives +4.11 and −0.24.
Seed replication. The paired tests condition on one run, so we retrain every backbone in both label spaces under six seeds (42 and 1 to 5) with the split held fixed. We test within-seed differences with two-sided one-sample tests with 5 degrees of freedom. Table 3 gives the resulting means and the decomposition repeated within each seed. The 0.7-threshold and class-balanced runs remain single-seed. The residual is indistinguishable from zero at all three capacities (, 0.78 and 0.71, positive in three of six seeds in each case). Paired by seed, Small exceeds Tiny by only +0.81 points, less than the 0.88 spread of the per-seed differences themselves (, ). Neither the split nor the seeds resolve that step, while Tiny-to-Base does. Under argmax-then-map the same runs give residuals of +0.91 ± 0.62, +1.01 ± 0.58 and +0.90 ± 0.65. That component exists only relative to the weaker scoring rule, and it does not grow with capacity either.
Discussion
The gain is real, and it is bookkeeping. Sometimes the base type is the required output, as in catalog normalization or product-type retrieval over unpackaged goods. There the +4.3 definitional points are delivered accuracy. Where the entries are genuinely different purchasable products, they are not, since predicting cheddar does not identify mature cheddar. Which applies is a per-group question, and the gain is concentrated in exactly the groups where it is least clear. But the gain is available as soon as the merge map is defined, from a model already trained. The consolidation changed what the system is asked to say, and parity in accuracy is not evidence that the learned features are unchanged. Absolute top-1 also stays near 30% even after consolidation, an 844-way fine-grained problem over long-tailed web photographs; this rules out the checkout and shelf-audit uses that nominally motivate retail recognition, which must name a packaged SKU and whose signal our packaging filter (§ 4.1) deliberately removes. What the pipeline supports is the catalog-side work above, where the decomposition, not the absolute number, is the contribution.
Grounding, hierarchy and deployment. Neither prompt design nor label redundancy closes the zero-shot gap (§ 6.2), and we did not vary the text tower, so its ability to represent rare proper nouns remains untested; that is a reason not to assume open-vocabulary recognition will absorb catalog maintenance. A hierarchy adopted as evaluation ground truth deserves the same near-duplicate audit and rescoring control as this flat label space. Top-5 (50.1%) favors a ranked shortlist over one guess, but the correct label is missing from about half the shortlists, motivating an assisted-identification study rather than a checkout deployment.
Limitations
One split, one budget. Six seeds characterize training stochasticity, but all runs share one partition, so split-to-split variation is unmeasured. Capacity is explored only at for 25 epochs, so every capacity statement is at this budget.
The Stage-1 filter is thinly supervised, and it selects. Out-of-fold precision and recall are near 90%. But they rest on 208 labels cross-validated over images rather than folders, and the regularization strength was chosen on those same folds rather than in a nested loop. Both effects make them optimistic. The archive-level estimate of § 6.1 implies substantial residual packaging. It also thins the catalog unevenly (mean per-folder retention 37.3%, standard deviation 24.1 percentage points), only weakly associated with folder size (, ), though the 191 merged entries retain more than the 807 untouched ones (56.8% vs 53.2%, ).
A null, not a proof of absence. For ConvNeXt-Small, the retraining residual is +0.08 ± 0.65 over six seeds, with a 95% confidence interval of [−0.60, +0.76] points, so an effect of a few tenths would not be detected. We claim only that retraining buys nothing at the scale of the reported curation gain, and we set no equivalence margin. The quantity also mixes representation with head fitting, calibration and optimization, hence retraining residual rather than a claim about features. Separately, class sizes are limited per raw entry, before the merge, so canonical blue holds 1,140 training images against a per-class maximum of 50 in the raw model. Capping the training split at 50 per canonical class costs 4.4 points (retrained canonical mean 30.6 → 26.2 over the same three seeds). But that control alters class composition as well as training volume, since the 31 classes it touches carry 19% of the test set. It therefore neither isolates the volume advantage nor bounds its effect on the residual.
The merge map is name-based, not image-verified. We hand-classify the 37 merged groups as alias collapses or type coarsenings (§ 6.3), but the merges themselves are formed by name rule and are not inspected image by image, so a group we label a coarsening could still contain near-identical products, or the reverse. The definitional component should therefore be read as the gain from scoring in this canonical space, with the alias-versus-coarsening split as its interpretation, not as an exact measure of catalog duplication. Coverage is the sharper limit. All 1,824 folders clear 20 raw images, but only 998 do after filtering and deduplication. That leaves 45% of the archive’s named categories outside the label space, most lost to the filter. The tail measured here is not the tail a deployment would face.
Conclusion
We separate a catalog-curation gain into the part reflecting a changed task and the part that could reflect a better model. On this archive the second yields no statistically resolved improvement: scoring the unchanged model against the merged labels already recovers the full +4.3-point gain, and retraining adds nothing we can resolve over six seeds at three capacities spanning in parameters. That gain is also overwhelmingly a category coarsening (94%) rather than alias correction, so the same decomposition both prices the retraining residual and tells a curator which kind of merge produced the gain. A curation gain that changes the label space should be reported relative to a frozen-model baseline rescored in that same space; this control costs one evaluation pass.
References
- [1]Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. SemDeDup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023. 2arxiv.org/abs/2303.09540
- [2]Mathurin Ache. Cheese pics. Kaggle dataset, 2021. https://www.kaggle.com/datasets/mathurinache/cheese-pics. 2
- [3]Yalong Bai, Yuxiang Chen, Wei Yu, Linfang Wang, and Wei Zhang. Products-10K: A large-scale product recognition dataset. arXiv preprint arXiv:2008.10545, 2020. 1, 2arxiv.org/abs/2008.10545
- [4]Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A. Lord. Making better mistakes: Leveraging class hierarchies with deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12506–12515, 2020. 2arxiv.org/abs/1912.09393
- [5]Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Åaron van den Oord. Are we done with ImageNet? arXiv preprint arXiv:2006.07159, 2020. 2arxiv.org/abs/2006.07159
- [6]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101: Mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), pages 446–461, 2014. 2DOI
- [7]Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Your “flamingo” is my “bird”: Fine-grained, or not. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11476–11485, 2021. 2arxiv.org/abs/2011.09040
- [8]Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with a reduced search space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3008–3017, 2020. 4arxiv.org/abs/1909.13719
- [9]Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9268–9277, 2019. 2arxiv.org/abs/1901.05555
- [10]Jia Deng, Jonathan Krause, Alexander C. Berg, and Li Fei-Fei. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3450–3457, 2012. 2DOI
- [11]Thomas G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998. 7
- [12]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 2
- [13]Patrick Follmann, Tobias Böttger, Philipp Härtinger, Rebecca König, and Markus Ulrich. MVTec D2S: Densely segmented supermarket dataset. In European Conference on Computer Vision (ECCV), pages 569–585, 2018. 2arxiv.org/abs/1804.08292
- [14]Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat-aComp: In search of the next generation of multimodal datasets. In NeurIPS Datasets and Benchmarks Track, 2023. 1, 2arxiv.org/abs/2304.14108
- [15]Eran Goldman, Roei Herzig, Aviv Eisenschtat, Jacob Goldberger, and Tal Hassner. Precise detection in densely packed scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5227–5236, 2019. 1, 2arxiv.org/abs/1904.00853
- [16]Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannah Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP. Zenodo, 2021. 2
- [17]Kanishk Jain, Shyamgopal Karthik, and Vineet Gandhi. Test-time amendment with a coarse classifier for fine-grained classification. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 2arxiv.org/abs/2302.00368
- [18]Philipp Jund, Nichola Abdo, Andreas Eitel, and Wolfram Burgard. The freiburg groceries dataset. arXiv preprint arXiv:1611.05799, 2016. 2arxiv.org/abs/1611.05799
- [19]Marcus Klasson, Cheng Zhang, and Hedvig Kjellström. A hierarchical grocery store image dataset with visual and semantic labels. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 491–500, 2019. 1, 2arxiv.org/abs/1901.00711
- [20]Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei. The unreasonable effectiveness of noisy data for fine-grained recognition. In European Conference on Computer Vision (ECCV), 2016. 2arxiv.org/abs/1511.06789
- [21]Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. WebVision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862, 2017. 2arxiv.org/abs/1708.02862
- [22]Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976–11986, 2022. 2, 4
- [23]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. 4
- [24]Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In NeurIPS Datasets and Benchmarks Track, 2021. 1, 2arxiv.org/abs/2103.14749
- [25]Jingtian Peng, Chang Xiao, and Yifan Li. RP2K: A large-scale retail product dataset for fine-grained image classification. arXiv preprint arXiv:2006.12634, 2020. 1, 2arxiv.org/abs/2006.12634
- [26]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748–8763, 2021. 2arxiv.org/abs/2103.00020
- [27]Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: Beating power law scaling via data pruning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 1, 2arxiv.org/abs/2206.14486
- [28]Zeren Sun, Yazhou Yao, Xiu-Shen Wei, Yongshun Zhang, Fumin Shen, Jianxin Wu, Jian Zhang, and Heng Tao Shen. Webly supervised fine-grained recognition: Benchmark datasets and an approach. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 10602–10611, 2021. 2arxiv.org/abs/2108.02399
- [29]TidyTuesday. Cheese. R for Data Science Online Learning Community, weekly data project, 2024-06-04, 2024. Tabular catalog scraped from https://www.cheese.com; distributed via https://github.com/rfordatascience/tidytuesday/tree/main/data/2024/2024-06-04. 2
- [30]Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8769–8778, 2018. 2arxiv.org/abs/1707.06642
- [31]Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on ImageNet. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2
- [32]Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 2
- [33]Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, and Lingqiao Liu. RPC: A large-scale retail product checkout dataset. arXiv preprint arXiv:1901.07249, 2019. 1, 2arxiv.org/abs/1901.07249
- [34]Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gari Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In International Conference on Learning Representations (ICLR), 2024. 2arxiv.org/abs/2309.16671
- [35]Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling ImageNet: From single to multi-labels, from global to localized labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2340–2350, 2021. 2



