When Cell Crops Share a Parent: Measuring and Isolating Acquisition Leakage in Cervical Cytology Benchmarks

Introduction

Public cervical cytology benchmarks are mostly released as cell crops cut from far fewer microscope fields or slide images [1, 2, 3]. Sibling crops share their field’s stain, illumination, focus and background, and sometimes pixels. A crop-level split, the default for files that look like independent images, puts siblings on both sides of the split Figure 1a.

That sibling crops inflate scores is expected: placing tiles of one subject in training and validation inflated digital pathology scores by up to 41% [4], and non-independent splits and shortcut cues misled medical image evaluation more broadly [5, 6, 7]. Benchmark artifacts have been audited in chest radiographs [8], shortcut mechanisms dissected in segmentation [9], and image-level splitting shown to inflate CT accuracy [10]. Contamination does not imply consequence,

however: in brain-tumor MRI, removing heavily contaminated test images, without retraining, left balanced accuracy essentially unchanged [11]. We ask the complementary question of when acquisition siblings do change the score, which requires retraining under each protocol rather than filtering the test set.

Three questions follow. How much does crop-level splitting change public cytology benchmarks? Is the change leakage? A grouped split removes the test parents’ siblings but, at a fixed crop budget, also leaves fewer distinct training parents (crop-level training sets hold 18% more on SIPaKMeD), so a lower score could reflect lost diversity, a confound that split comparisons leave in place. Can common diagnostics certify an evaluation? Decodable source identity is increasingly read as a sign of shortcut use [12], and saliency maps as evidence of where a model looks. A preregistered study and a registered follow-up (2,281 trained models) contribute:

• Structure: closed forms that predict a split’s exposure from its crop-to-parent table (§ 2).

• Consequence: one simulation-powered test per benchmark across twelve architectures (§ 4.1).

• Causal isolation: a SIPaKMeD factorial that varies sibling exposure at fixed crop and parent counts (§ 4.2).

• Diagnostic caution: interventions showing that the linearly decodable parent code and saliency do not track the leak (§ 4.3).

Data and acquisition hierarchy

Table 1 lists four public datasets. SIPaKMeD crops come from cluster slides; filename parsing and template matching independently assign all 4,049 crops to the same field. Cx22 regions come from CEED source images, CRIC cells from 400 slide images with nucleus coordinates, and Herlev, with one cell per image, is a negative control. A parent is the deepest acquisition unit it releases identities; it is not a patient.

Dataset	Parent	Crops	Parents	Sib. %	Pix. %	Models
SIPaKMeD [1]	field	4,049	966	92.6	52.6	✓
Cx22 [2]	image	1,320	514	85.2	34.0	–
CRIC [3]	slide	11,534	400	99.97	71.5	✓
Herlev [13]	cell	917	917	0.0	0.0	–

If a parent’s $k$ crops are allocated independently to a test fraction $q$, a test crop has a training sibling with probability $1-q^{k-1}, and at equal size a crop-level training set holds \sum_j(1 -

$q_j^k/\{(1-q)^m\}$ times as many distinct parents as a parent-level one over $m$ parents (1.18 for SIPaKMeD, 1.25 for CRIC). These closed forms reproduce Table 1. The parent is also learnable: a ResNet-18 trained to identify 424 SIPaKMeD fields recognised held-out siblings at 25.7% (chance 0.24%), and at 18.7% from cell-free backgrounds. Each SIPaKMeD field holds one class; in CRIC, slide identity carries 68% of the class entropy.

Study design

Everything reported comes from the preregistered Study 2 and its registered follow-up; an earlier pilot (Study 1, six architectures, five seeds) served only the power analysis. Each phase was registered before its own runs: Study 2’s splits, caches, design tables and analysis scripts were hashed and committed before its first model was trained, with smoke tests on crop-level splits only, and the follow-up after the Study 2 analysis but before any follow-up model or analysis.

Protocols and training. P1 is a stratified crop-level 80:20 split, P2 a parent-disjoint split of equal test size, and P3 additionally merges parents linked by near-duplicates. Twelve ImageNet-pretrained backbones (ResNet-18/50, DenseNet-121, MobileNet-V2, EfficientNet-B0/V2-S, RegNetY-1.6GF, ConvNeXt-T, Swin-T, MaxViT-T, ViT-B/16, DINOv2 ViT-S/14) are fine-tuned with one deterministic recipe (AdamW, $10^{-4}$, batch 64, 20 epochs, resized-crop and flip augmentation) and scored at the final epoch without model selection; per-crop logits and weights are kept.

One test per benchmark (H1, H2). Instead of one test per architecture, the protocol effect is tested once,

where $x = \pm\frac{1}{2}$ codes P1 vs. P2 and split, shared by all architectures within a seed, stops twelve evaluations of one test set from counting as twelve [14, 15]. A registered residual check selected a two-way cluster bootstrap as the primary inference, with Satterthwaite, Kenward–Roger, binomial-mixed-model and sign-flip tests as sensitivity analyses. Twelve architectures and twenty seeds gave a simulated power of 0.977 for a 0.8-pp effect on SIPaKMeD; CRIC uses six architectures and ten seeds.

Leakage vs. diversity (H3). The factorial (Figure 1b) fixes one test set per seed, half the crops of a random 30% of multi-crop SIPaKMeD fields (about 527 crops), and varies only the training set, always 220 crops per class. Clean cells draw from other fields; leak cells swap the last fields of the same permutation for the test fields’ remaining siblings. Within a diversity level, leak and clean cells thus share crop count, field count and allocation; 720 vs. 504 fields give a 1.43-fold diversity contrast that brackets the natural 1.18. H3 is the leak effect in a mixed model (four architectures, ten seeds, 160 runs); diversity is estimated within clean cells.

Diagnostics. Study 2 registered background transplants (H4) and a gradient-reversal parent adversary (H5) [16]. The follow-up tests eight hypotheses (F1–F8), 940 new models; Figure 1c). The H3 models are re-scored after background removal, illumination flattening, greyscale, low-pass filtering, or Reinhard transfer [17] of each cell’s color statistics to an unseen field of its own class (F2; the target is chosen by label, so this is a controlled diagnostic); the factorial is retrained with masked backgrounds (F3) or strong HED color jitter [18] (F4). Parent recoverability $R$, the within-class accuracy of a logistic field probe on frozen features, is lowered by rank-limited LEACE erasure with a refitted head [19] (F5), stain normalisation (F6) and a probe-matched linear adversary (F7), each required to remove a quarter of the above-floor $R$ while costing at most 1 pp of P2 accuracy; F8 pools the dose–response, and F1 tests a slide-level label prior in CRIC. Grad-CAM [20] attribution outside the cell mask is compared descriptively, with a randomised-head check [21]. H1 stands alone; H2–H5 and F1–F8 are separate Holm families [22].

Results

All 1,341 study runs and 1,580 follow-up tasks completed, none diverged, and every result matched its registered split hash. Table 2 lists all registered tests.

Hypothesis	Est.	95% CI	$p$
H1 SIPaKMeD crop vs. field	0.97	0.59, 1.39	<0.001 ✓
H2 CRIC crop vs. slide	12.08	11.16, 12.99	<0.001 ✓
H3 Leakage, matched size	1.73	0.95, 2.51	0.02 ✓
H4 Own-background route	0.42	−0.35, 1.18	0.23 ✗
H5 Gap tracks $R$ (GRL)	0.06	−0.01, 0.12	0.16 ✗
F1 CRIC slide-majority pull	−4.29	−8.60, −1.37	0.001 ✓†
F2 Color transfer cuts leak	1.27	0.38, 2.16	0.03 ✓
F3 Leak, no background	1.86	1.42, 2.30	<0.001 ✓
F4 HED jitter cuts leak	−0.23	−0.81, 0.35	1.00 ✗
F5 LEACE erasure cuts gap	−0.01	−0.10, 0.07	1.00 ✗
F6 Normalisation cuts gap	0.02	−0.41, 0.44	1.00 ✗
F7 Gap tracks $R$ (linear)	0.000	−0.038, 0.039	1.00 ✗
F8 Gap tracks $R$ (pooled)	0.000	−0.005, 0.005	1.00 ✗

Crop-level splitting inflates both benchmarks

On SIPaKMeD, accuracy was 97.14% under P1 and 96.18% under P2 (H1 = 0.97 pp, 0.59–1.39), with all four sensitivity analyses at $p < 0.001$. Every architecture scored higher under P1, by 0.66–1.44 pp (Figure 2a): crop-level splitting hides a quarter of the errors. Merging near-duplicate-linked parents changed nothing (P2 − P3: 0.03 pp, −0.49 to 0.56), so shared parents, not stray duplicates, carry the effect, and the architecture ranking was largely preserved (Kendall $\tau= 0.82$). On CRIC, with about 29 mixed-class cells per slide, accuracy fell from 90.48% to 78.40% (H2 = 12.08 pp; 11.6–12.8 pp per architecture; Figure 2b) and balanced accuracy from 78.53% to 58.89%; a size-matched crop-level control gave the same gap (12.11 pp). That control fixes training size, not slide composition, and the factorial below was run on SIPaKMeD only, so the CRIC gap is consistent with acquisition leakage but not causally decomposed. Our CRIC inputs are 100 × 100 windows around the published nucleus coordinates, so absolute accuracies are not comparable with other CRIC results.

On SIPaKMeD, the inflation is leakage, not diversity

At fixed crop count, field count and allocation, adding the test fields’ siblings raised accuracy on the common test set by 1.73 pp (0.95–2.51) in every architecture at both diversity levels, while 43% more fields without leakage did nothing (0.00 pp, −0.65 to 0.64; interaction −0.20 pp, $p = 0.60$; Figure 2c). The leak replicated on fresh seeds (1.47 pp) and persisted with every background masked in training and testing (F3: 1.86 pp). In an exploratory split it was as large on test crops sharing no pixel with any training crop (1.68 pp) as on overlapping ones (1.78 pp): what helps is a seen field, not memorised pixels.

Linear decodability and saliency do not track the leak

Neither registered Study 2 mechanism test located the leaked signal: background transplants (H4) and gradient reversal (H5), which ConvNeXt-T evaded by making parents more decodable ($R$ 29.5% to 55.1%), were not supported. Reducing the linearly decodable parent code did not move the gap either (Figure 3a). Rank-256 LEACE cut $R$ from 25.1%

to 5.4%, removing 82% of its above-floor value (class-only floor 1.2%), in all twelve architectures at a 0.17-pp cost in P2 accuracy, and the gap stayed at 0.93–0.95 pp (F5); the linear adversary passed its check in all three architectures with no change (F7); pooled over 15 cells the slope was 0.000 (−0.005 to 0.005; F8), so a 20-pp loss of $R$ moves the gap by at most 0.1 pp. Grad-CAM put 21.8% of attribution outside the cell in leak and clean models alike (difference −0.09 pp, −0.57 to 0.39; Figure 3c), although its maps passed the randomised-head check (median Spearman $\rho= 0.08$). On CRIC, seen slides pulled minority-class cells away from the slide majority (F1: −4.29 pp), which argues against a simple slide-majority prior. Only color transfer moved the leak. Matching each test cell’s color statistics to an unseen field of its own class cut the leak from 1.72 to 0.46 pp (F2: 1.27 pp, 0.38–2.16), whereas flattening illumination (1.30 pp) or removing the background (1.17 pp), at comparable accuracy cost, did not (Figure 3b); strong HED jitter in training left it intact (1.71 vs. 1.47 pp; F4). Because the transfer is label-conditioned and shifts the test distribution, it implicates field-specific appearance without identifying what the networks learned.

Discussion and conclusion

Crop-level splitting inflates both public cervical cytology benchmarks we trained on. On SIPaKMeD the inflation is leakage: it appears when a test cell’s acquisition siblings are in training, not when training merely gains parent diversity. On CRIC, with 29 mixed-class cells per slide instead of four single-class crops per field, the gap is twelve times larger; without a factorial there, it is consistent with, not attributed to, sibling leakage. A benchmark’s exposure can be computed from the closed forms of § 2 before any model is trained. Rankings largely survived on SIPaKMeD, but accuracies under crop- and parent-level splits are not comparable: on CRIC the protocol gap exceeds the spread between architectures.

The mechanism is narrowed, not identified. The advantage survives background masking, does not depend on shared pixels, does not follow a simple slide-majority prior and is unaffected by reducing the linear parent code, while label-conditioned color transfer implicates field-specific appearance; nonlinear or spatially encoded parent information may still carry it. The methodological lesson holds regardless: models whose measured parent code was largely erased, and whose saliency matched that of clean models, still benefited from leakage. Probes establish availability, not use [23, 24], neither probes, saliency nor augmentation can stand in for a grouped split.

Limitations. Parents are the deepest released units, not patients: parent-disjoint evaluation removes the leakage measured here, but patient-level grouping could reveal further dependence that these experiments cannot bound. The factorial trains on 1,100 crops rather than 3,239, establishing the leak’s existence and direction more firmly than an exact decomposition of H1. One training recipe was used, erasure acts on linearly readable pooled features, and the follow-up, registered after the Study 2 results, is confirmatory only for its own questions.

Recommendations. Releases should label every observation with its parent, and its patient where permitted; evaluations should split at the deepest released unit, publish hashed fold manifests and report the fraction of test observations with a training sibling. Code, provenance maps, split manifests, run records and a risk calculator are at https://github.com/optimisticexquisite/cytofair.

Compliance with ethical standards

This research was conducted retrospectively using de-identified human cytology images made available in open access by the creators of SIPaKMeD, Cx22, CRIC and Herlev. No new data were collected, and ethical approval was not required for this secondary analysis of public data.

Acknowledgments

No funding was received for conducting this study. The authors have no relevant financial or non-financial interests to disclose. Claude (Anthropic), used through Claude Code, assisted in writing and debugging the experiment and analysis code, orchestrating the training runs, producing the figures, and drafting and editing the text of all sections. The authors reviewed all code, results, references and text, and take full responsibility for the content.

References