betaFeedback

WorksComputer Science

When Do Histology Foundation Models Need Geometric Test-Time Augmentation?

Abstract

Test-time augmentation (TTA) aggregates model predictions across multiple transformed views of an input during inference. Because histology images lack a canonical orientation, D4D_4 TTA—pooling across the 8 rotations and reflections of a square—is a standard default in computational pathology. However, the rise of self-supervised histology foundation models (FMs) with inherent rotational invariance calls its continued utility into question. We benchmark 9 histology FMs and 6 general-purpose models across four patch-level and two whole-slide image (WSI) datasets. We find that D4D_4 TTA remains a reliable post hoc improvement, but benefits diminish from +1.98+1.98 percentage points (pp) for general-purpose backbones to +0.50+0.50 pp for histology FMs on TCGA-UT, and shrink further for the largest models. To eliminate computational overhead, our experiments demonstrate that entropy routing—applying TTA selectively to uncertain predictions—recovers most of the gain while augmenting only ∼13%{\sim}13\% of samples. Finally, we show that patch-level TTA improvements do not transfer to slide-level multiple instance learning (MIL). Code: https://github.com/lars-chen/histology-tta

Introduction

Histopathology image analysis increasingly relies on foundation models (FMs): large vision transformers pretrained on millions of hematoxylin and eosin (H&E) tissue patches by self-supervised learning [5, 30, 10, 35, 26]. The strong downstream performance of FMs has made them the standard feature extractor in computational pathology.

Because tissue patches have no canonical orientation, evaluating all eight dihedral-group views at inference and aggregating predictions is a principled inference-time strategy for enforcing rotational and reflection invariance without retraining. Given recent advances in pretraining scale and objectives, however, whether test-time augmentation (TTA) still yields meaningful gains for FM-based classifiers remains an open question. We provide a systematic evaluation of this strategy across modern FMs commonly deployed in computational pathology pipelines [22, 13]. Figure 1 provides a graphical overview of our approach, and Table 2 summarizes the datasets used in our experiments.

D4 test-time augmentation with selective routing diagram and performance plots

Figure 1. D4D_4 test-time augmentation with selective routing improves histology foundation model classification. (Left) A lightweight classifier is trained using FM-extracted patch embeddings zpz_p. At inference, patch prediction entropy HH gates augmentation: low-entropy patches receive a single-view prediction y^p\hat{y}_p; high-entropy patches are passed through the D4D_4 symmetry group and predictions are averaged across views. (Top right) D4D_4 TTA improves balanced accuracy across both histology FMs and general-purpose models. Dots show per-model gains, on TCGA-UT, with the mean bar indicating +0.5 pp and +1.98 pp respectively. Applying TTA selectively with entropy thresholds (curves, averaged across models) recovers most of this gain while augmenting only a fraction of patches. (Bottom right) Applied to all patches in an exemplar Camelyon17 slide [1], full TTA corrects predictions from UNI embeddings. Green patches indicate corrected and blue line shows tumor boundary. Zoomed image shows the overall increase in tumor probability within boundary.

DatasetClassesTrainTestDomain
TCGA-UT31190k40.9kMulti-organ
NCT-CRC-HE-100K9100k7.2kColorectal
NCT-CRC-HE-NoNorm9100k7.2kColorectal
MHIST22,175977Colorectal polyp

Table 2. Datasets used for evaluation. Train and test sizes are tile counts.

Tissue patches are approximately symmetric under the dihedral group D4D_4, i.e., the eight isometries of a square, comprising four rotations (0∘0^\circ, 90∘90^\circ, 180∘180^\circ, 270∘270^\circ) and their respective reflections. An ideal classifier should therefore be invariant to these transforms. This can be achieved in two ways: training-time augmentation applies random D4D_4 transforms during training, encouraging the model to learn invariant representations; instead, TTA averages predictions across all eight views at inference, approximating invariance post hoc without modifying the trained model.

Group-averaged TTA corresponds to applying the Reynolds operator [8]. For a group of transformations GG acting on input space X\mathcal{X} and a function f:X→RCf:\mathcal{X}\to\mathbb{R}^{C},

f^(x)=1∣G∣∑g∈Gf(g⋅x)(1)\hat{f}(x)=\frac{1}{\lvert G\rvert}\sum_{g\in G}f(g\cdot x) \tag*{(1)}

produces the unique GG-invariant function closest to ff in L2L^2 norm when the data are GG-invariant, as histology patches are, justifying averaging as the optimal strategy to enforce invariance. In practice, however, averaging can correct some predictions while corrupting others. When ff is already GG-invariant, the Reynolds operator reduces to the identity and TTA provides no benefit.

Hence, we investigate to what extent modern histology FMs are D4D_4 invariant, and under what conditions TTA provides meaningful gains. We address this through a large-scale TTA evaluation in computational pathology, spanning:

  • 16 models from three families: 9 histology-specialized FMs, 6 general-purpose vision models, and 1 D4D_4-equivariant reference model (Table 1)

  • 4 patch datasets of varying difficulty: TCGA-UT (31 classes) [18], NCT-CRC-HE-100K and NCT-CRC-HE-Norm (9 classes each) [16], and MHIST (2 classes) [32]

  • 2 whole-slide image (WSI) datasets for slide-level TTA analysis: Camelyon17 (lymph node sections) [1], and PANDA (prostate biopsy samples) [2], covering both patch-level and weakly supervised multiple instance learning (MIL) classification

ModelTypeParamsPatchesArch.
CTransPath†Histo2815Swin-T
Phikon†Histo8640ViT-B/16
Phikon-v2†Histo300456ViT-L/16
UNIHisto300100ViT-L/16
UNI-v2Histo681200ViT-H/14
VirchowHisto6322000ViT-H/14
Virchow-v2Histo6322000ViT-H/14
GigaPathHisto11001400ViT-g/16
H-Optimus-1Histo1100>1000ViT-g/14
ResNet-18General12–ResNet
ResNet-50General26–ResNet
ConvNeXtV2-TGeneral28–ConvNeXt
ConvNeXtV2-BGeneral89–ConvNeXt
DINOv2-SGeneral22–ViT-S/14
DINOv2-BGeneral86–ViT-B/14
D4WRNEquivariant12–Wide ResNet

Table 1. Models evaluated in this study. Histology FMs are pretrained on histopathology data via self-supervised learning. General-purpose models are pretrained on ImageNet. The equivariant model (D4WRN) has architectural D4D_4 symmetry guarantees. Number of parameters and training patches are reported in millions (M). †TCGA-UT slides may appear in self-supervised pretraining, however without labels (§ 4.1).

Our key finding is that D4D_4 TTA is a reliable, modest post hoc improvement for patch classification, even with strong histology FMs. On TCGA-UT, histology FMs gain +0.5 pp in balanced accuracy on average, while general-purpose backbones benefit +2 pp. TTA improves predictions overall for all four patch benchmarks, but the average gains are smaller in higher baseline tasks (see Figure 2 and Table 5).

Figure 2. D4D_4 TTA consistently improves histopathology classifiers, with stronger gains for weaker baselines. (a) Distribution of ΔTTA\Delta_{\mathrm{TTA}} in balanced accuracy (pp) across models for each dataset, split by model family (histology FMs vs. general-purpose backbones). Points show individual models; boxplots summarize the distribution; D4WRN (equivariant reference) exhibits near-zero gain by construction. (b) Relationship between baseline balanced accuracy and ΔTTA\Delta_{\mathrm{TTA}} for each dataset: weaker baselines tend to benefit more from TTA.

TypeModelTCGA-UT BaseTCGA-UT V4V_4TCGA-UT D4D_4NCT-100K BaseNCT-100K V4V_4NCT-100K D4D_4NCT-NoNorm BaseNCT-NoNorm V4V_4NCT-NoNorm D4D_4MHIST BaseMHIST V4V_4MHIST D4D_4
General-purposeResNet-1837.5+2.48+3.23{sup}***`86.9+1.06+1.11{sup}***`69.5+0.72+0.66{sup}*`71.8+0.94+1.84{sup}***`
ResNet-5042.5+1.92+2.34{sup}***`87.4+1.10+1.52{sup}***`80.4+0.85+0.82{sup}***`72.8+1.27+2.01{sup}**`
ConvNeXtV2-Tiny48.3+1.31+1.92{sup}***`88.8+0.43+0.49{sup}**`70.4+0.47+1.05{sup}***`76.2+1.43+2.48{sup}**`
ConvNeXtV2-Base49.6+1.01+1.32{sup}***`90.1+0.83+1.22{sup}***`69.3+0.20−0.24{sup}**`77.8+0.43+0.94
DINOv2-S49.8+1.27+1.88{sup}***`90.7+0.78+0.75{sup}***`75.5+0.40+0.94{sup}**`77.3+1.28+1.98{sup}***`
DINOv2-B50.7+0.90+1.23{sup}***`90.7+0.68+0.51{sup}**`80.8+0.64+0.58{sup}***`80.4+0.11+0.12
Histology foundation modelsCTransPath57.9+0.67+0.90{sup}***`94.2+0.48+0.35{sup}**`80.2+0.27+0.0059.0+0.09−0.58
Phikon74.0+0.45+0.65{sup}***`92.2+0.48+0.65{sup}***`76.4+2.34+1.66{sup}***`73.8+0.28+0.72
Phikon-v276.8+0.31+0.44{sup}***`91.6+0.89+1.60{sup}***`89.5−0.14−0.1771.2−0.05−0.16
UNI72.9+0.41+0.55{sup}***`93.2−0.26−0.24{sup}*`85.8−0.35−0.38{sup}*`81.2−0.50−0.33
UNI-v278.5+0.51+0.62{sup}***`95.5−0.04+0.0093.0+0.13+0.0181.1+0.97+0.71
Virchow73.3+0.46+0.50{sup}***`94.5+0.01+0.0181.6−0.17+0.2581.6+0.42+0.48{sup}*`
Virchow-v277.9+0.20+0.28{sup}***`94.8+0.13+0.0591.6+0.14+0.1486.7+0.35+0.20
GigaPath73.7+0.24+0.34{sup}***`95.3+0.05+0.0989.4+0.38+1.18{sup}***`80.4−0.37−0.02
H-Optimus-179.4+0.13+0.25{sup}**`95.0+0.12+0.0894.4+0.13+0.0780.8+0.05+0.00
Equivariant referenceD4WRN†69.6+0.17+0.1794.3+0.08+0.0872.6−0.19−0.1872.3−0.10−0.06

Table 5. Per-model TTA results across all four datasets (frozen backbone / SGD linear probe, probability-mean aggregation, 5 seeds). Base: no-TTA balanced accuracy (%). V4V_4 and D4D_4: change in percentage points from 4-view (flips) and 8-view (full dihedral) TTA respectively. Significance of ΔTTA\Delta_{\mathrm{TTA}} (D4D_4) vs. zero (two-sided paired tt-test, n=5n = 5 seeds, uncorrected): *p<0.05p < 0.05, **p<0.01p < 0.01, ***p<0.001p < 0.001. Best ΔTTA\Delta_{\mathrm{TTA}} (D4D_4) per dataset in bold. Class counts: TCGA-UT 31, NCT-100K / NCT-NoNorm 9, MHIST 2. D4WRN is finetuned (seeds vary by dataset; see †) and shown as an uncontrolled equivariant reference (see § 4.1).

At inference time, the full G=D4G = D_4 TTA increases the computational cost by a factor of ∣G∣=8|G| = 8. For histology FMs, where the average gain is small, this cost could be hard to justify in routine clinical or research use. By contrast, general-purpose backbones exhibit greater view-to-view disagreement and weaker class separability, so TTA yields larger average improvements and can be worthwhile. We therefore evaluate entropy-based selective TTA [28] to determine how much overhead can be avoided while retaining most of the performance benefit across models with different baseline degrees of rotational invariance.

Finally, we extend the analysis beyond independent tiles to WSIs. Because these gigapixel-scale clinical slides require aggregating thousands of patches via MIL, we test whether localized tile-level corrections survive this process to improve overall slide-level performance.

Related Works

Test-Time Augmentation

TTA has a long history in computer vision, from early ensemble methods ([19]) to systematic studies of augmentation policies and aggregation ([27, 17]). In biomedical imaging, it is routinely applied in histopathology ([21]), microscopy cell segmentation ([23]), and dermatology ([25]). Most prior work studies whether TTA helps a particular model on a particular task, without systematically evaluating how its benefit varies across modern backbone families (notably transformer-based models) or across different classifier heads and pretraining strategies.

Elesedy and Zaidi [8] prove that averaging a predictor over a symmetry group strictly improves generalization when the target is invariant. Beyond computer vision, Foit et al. [12] recently demonstrated the efficacy of group-averaged TTA for autoregressive prediction in physics.

Shanmugam et al. [27] find that TTA, while improving accuracy overall, also turns some correct predictions into incorrect ones, and that learned aggregation outperforms simple averaging. TTA also multiplies inference cost, so Son and Kang [28] propose selective TTA that routes uncertain samples to additional augmentations. However, it remains unclear how D4D_4 TTA and selective routing behave across modern histology FMs versus general-purpose backbones.

Foundation Models for Histopathology

The past two years have seen an explosion of histology FMs pretrained by self-supervised learning on large histopathology corpora (Table 1): CTransPath ([31]), Phikon ([10]) and Phikon-v2 ([11]), UNI ([5]) and UNI-v2, Virchow ([30]) and Virchow-v2 ([38]), GigaPath ([35]) and H-Optimus-1 ([26]). They range from 28M to 1.1B parameters and from tens of millions to billions of training tiles, and most are pretrained with DINOv2 ([24]); Phikon uses iBOT ([37]) and CTransPath a contrastive objective.

A common feature of these models’ pretraining is strong stochastic augmentation, including geometric ones such as crops, flips and rotations, used within self-supervised learning objectives such as DINO and iBOT ([3, 24, 37]). By explicitly matching model outputs across differently augmented views of the same patch ([3, 37]), these objectives encourage invariance to the augmentation set, promoting D4D_4-stable representations when D4D_4 transforms are included. Along with the orientation diversity of histopathology data, these objectives would be expected to yield approximately D4D_4-invariant representations. We find instead that histology FM embeddings are about as D4D_4-invariant as those of general-purpose models (§ 4.6); their prediction stability arises from classification margin.

In addition to histology-specialized FMs, computational pathology pipelines frequently rely on general-purpose FMs (e.g., ImageNet-pretrained CNNs and self-supervised transformers such as DINOv2) as tile encoders and baselines due to their accessibility and strong performance ([22, 13, 24]). Because these models are not trained on histology with the same orientation diversity as histology FMs, they provide a useful contrast case in our study.

Equivariant Neural Networks

Group-equivariant convolutional networks ([7, 33]) provide architectural guarantees of equivariance to specified symmetry groups. For the D4D_4 group, architectures like E2-CNNs ([33]) and p4mp4m-CNNs ([7]) ensure that feature maps transform predictably under rotations and reflections. Rotation-equivariant architectures have shown benefits in microscopy and histopathology ([6, 29, 20]). We construct D4WRN, a Wide ResNet with a D4D_4-equivariant backbone and an invariant readout, as a reference point: its predictions are D4D_4-invariant by construction, so its TTA benefit should be near zero.

Prior work establishes TTA as a generally useful ensembling technique and motivates D4D_4 symmetry for histology. Elphick et al. [9] found that FMs pretrained with rotation augmentation have tighter rotation orbits in embedding space, but did not examine downstream predictions or TTA. We show that orbit size barely predicts whether a prediction changes under D4D_4; what matters is the relative distance to the decision boundary (§ 4.6).

Methods

Models

We evaluate 16 models spanning three categories (Table 1).

Histology foundation models (9 models) are used with frozen backbones and a trained linear classification head, following the standard linear probing protocol. This isolates the quality of the pretrained representations without confounding from downstream finetuning.

General-purpose models (6 models) pretrained on ImageNet are evaluated in both frozen (linear probe) and finetuned configurations to assess how downstream adaptation affects TTA benefit.

Equivariant reference (1 model): a D4D_4-equivariant Wide ResNet, D4WRN [33] (Zagoruyko and Komodakis 2016) trained from scratch, providing an architecture with exact D4D_4-equivariance guarantees.

Datasets

We evaluate on four tile-level classification datasets using the standard splits, summarized in Table 2: TCGA-UT is split by patient, MHIST uses the provided patient-disjoint split, and both NCT-CRC-HE variants use the standard split with a fixed test set across runs. For the frozen-backbone heads, hyperparameters are selected by 3-fold stratified cross-validation on the training set, with folds fixed across seeds; three folds keep selection tractable at this scale, and all datasets except MHIST leave tens of thousands of tiles per validation fold. Early stopping uses a separate 10% validation split carved from the training data; the test split is never used during training.

TCGA-UT [18] is the most challenging benchmark with 31 fine-grained tissue classes across multiple organs. NCT-CRC-HE-100K [16] and its non-normalized variant isolate the impact of stain normalization versus geometric TTA. MHIST [32] is a clinically challenging binary task requiring the differentiation of benign Hyperplastic Polyps (HP) from precancerous Sessile Serrated Adenomas/Polyps (SSA).

TTA Strategies

We evaluate three TTA strategies of increasing group size:

  1. None: standard single-view inference (baseline)

  2. V4V_4 (4 views): horizontal flip, vertical flip, and both, forming the Klein four-group V4⊂D4V_4 \subset D_4

  3. D4D_4 (8 views): all eight elements of the dihedral group, comprising four rotations (0°, 90°, 180°, 270°) composed with identity and horizontal flip

For each strategy, we evaluate four aggregation methods. Let GG denote the set of test-time views, with per-view logits ℓg∈RC\ell_g \in\mathbb{R}^{C} and probabilities pg=softmax⁡(ℓg)p_g = \operatorname{softmax}(\ell_g). All sums below are over the group elements g∈Gg \in G

a=softmax⁡(1∣G∣∑gℓg)(logit-mean)a = \operatorname{softmax}\left(\frac{1}{|G|}\sum_g \ell_g\right) \qquad\text{(logit-mean)}
a=1∣G∣∑gpg(prob-mean)a = \frac{1}{|G|}\sum_g p_g \qquad\text{(prob-mean)}
a=∑gwgpg∑gwg,wg=max⁡c(pg)c(conf.-weighted)a = \frac{\sum_g w_g p_g}{\sum_g w_g}, \qquad w_g = \max_c (p_g)_c \qquad\text{(conf.-weighted)}
y^=arg⁡max⁡cac(prediction)\hat{y} = \arg\max_c a_c \qquad\text{(prediction)}

Majority vote predicts y^=mode⁡g(arg⁡max⁡c(pg)c)\hat{y} = \operatorname{mode}_g(\arg\max_c (p_g)_c). Unless stated otherwise, we use logit-mean aggregation, which applies Equation (1) directly to the logits, as the default; probability-mean, gives the same average gain (within 0.01 pp). Results for all aggregation variants are reported in § 4.

Finally, we evaluate selective TTA, which adaptively applies full D4D_4 only to uncertain samples based on the entropy of the base-view prediction, trading off accuracy and inference cost (§ 4.7).

Metrics

We report balanced accuracy (macro-averaged recall) as our primary metric, since it weights all classes equally regardless of prevalence, which is critical for the imbalanced TCGA-UT dataset. For a CC-class problem, balanced accuracy is

BalAcc⁡(f)=1C∑c=1CTPcTPc+FNc.\operatorname{BalAcc}(f) = \frac{1}{C}\sum_{c=1}^{C}\frac{\mathrm{TP}_c}{\mathrm{TP}_c+\mathrm{FN}_c}.

The TTA benefit is quantified as the difference in balanced accuracy between TTA and baseline, ΔTTA=BalAcc⁡(fTTA)−BalAcc⁡(fbase)\Delta_{\mathrm{TTA}} = \operatorname{BalAcc}(f_{\mathrm{TTA}})-\operatorname{BalAcc}(f_{\mathrm{base}}), reported in percentage points (pp).

For each TTA evaluation, we count corrections: samples the baseline gets wrong but TTA gets right, and corruptions: samples the baseline gets right but TTA gets wrong. The correction ratio ncorrected/ncorrupted>1n_{\mathrm{corrected}}/n_{\mathrm{corrupted}} > 1 indicates net benefit; it measures the quality of TTA’s prediction flips independently of their total count.

Prediction entropy. As an uncertainty proxy for selective TTA, we use the Shannon entropy of the single-view softmax p(y∣x)∈[0,1]Cp(y \mid x) \in[0,1]^C:

H(x)=−∑c=1Cp(y=c∣x)log⁡p(y=c∣x),H(x) = -\sum_{c=1}^{C} p(y=c \mid x)\log p(y=c \mid x),

where CC is the number of classes and log⁡\log is the natural logarithm. When comparing across datasets with different CC, we report the normalized entropy H~(x)=H(x)/log⁡C∈[0,1]\widetilde{H}(x)=H(x)/\log C \in[0,1].

Orbit tightness (MPCS). To quantify representation-level D4D_4 invariance (§ 4.6), we ask how close a patch’s eight D4D_4-view embeddings zgz_g are to one another. Let zgz_g be the frozen-backbone feature vector of view gg. The mean pairwise cosine similarity (MPCS) averages, over all (82)\binom{8}{2} pairs of views, their cosine similarity,

MPCS⁡=1(82)∑g<g′⟨zg,zg′⟩∥zg∥∥zg′∥.(2)\operatorname{MPCS} = \frac{1}{\binom{8}{2}}\sum_{g<g'}\frac{\langle z_g,z_{g'}\rangle}{\lVert z_g\rVert\lVert z_{g'}\rVert}. \tag*{(2)}

which equals 1 when all eight views point in the same direction (a perfectly invariant orbit).

Boundary geometry (margin and straddle ratio). Whether the orbit’s movement changes the prediction depends not on its size but on its position relative to the decision boundary. Let W∈RC×DW \in\mathbb{R}^{C\times D} and β∈RC\beta\in\mathbb{R}^{C} be the linear probe’s weights and biases, and let i,ji,j index the top-two classes of the base (identity) view zidz_{\mathrm{id}}. The i∣ji\mid j boundary is the hyperplane with normal wi−wjw_i-w_j, and the geometric margin is the signed distance of the base view from it,

m=⟨wi−wj,zid⟩+(βi−βj)∥wi−wj∥.(3)m = \frac{\langle w_i-w_j,z_{\mathrm{id}}\rangle+(\beta_i-\beta_j)}{\lVert w_i-w_j\rVert}. \tag*{(3)}

Let n^=(wi−wj)/∥wi−wj∥\hat{n}=(w_i-w_j)/\lVert w_i-w_j\rVert be the unit normal to that boundary, the only direction along which motion can change which class wins. The orbit’s spread along it, and the straddle ratio, are

s=std⁡g∈G⟨zg,n^⟩,r=m/s.(4)s=\operatorname{std}_{g\in G}\langle z_g,\hat{n}\rangle,\qquad r=m/s. \tag*{(4)}

The orbit straddles the boundary when ∣r∣<1\lvert r\rvert<1. To predict flips we score patches by −∣m∣-\lvert m\rvert and −∣r∣-\lvert r\rvert, so larger scores indicate more likely flips.

Calibration. Let p^i=max⁡cp(y=c∣xi)\hat{p}_i=\max_c p(y=c\mid x_i) be the confidence of prediction y^i\hat{y}_i on sample ii. Partitioning the NN test samples into ten equal-width confidence bins B1,…,B10B_1,\ldots,B_{10}, the expected calibration error is

ECE⁡=∑b=110∣Bb∣N∣acc⁡(Bb)−conf⁡(Bb)∣,\operatorname{ECE}=\sum_{b=1}^{10}\frac{\lvert B_b\rvert}{N}\left\lvert\operatorname{acc}(B_b)-\operatorname{conf}(B_b)\right\rvert,

the sample-weighted gap between accuracy and mean confidence within each bin. Alongside ECE, which is computed from the same top-label confidences, we report the top-label Brier score

Brier⁡=1N∑i=1N(p^i−1[y^i=yi])2.\operatorname{Brier}=\frac{1}{N}\sum_{i=1}^{N}\left(\hat{p}_i-\mathbb{1}[\hat{y}_i=y_i]\right)^2.

Training Protocol

Frozen backbones and trained heads. We pre-extract patch embeddings once from each frozen backbone and train lightweight heads on top of the fixed features. Our primary head is a linear probe: a single fully-connected layer trained by stochastic gradient descent (Adam) with cross-entropy loss, following the standard self-supervised linear-probe protocol ([24]). We additionally report one- and two-hidden-layer MLP probes and kk-nearest-neighbor baselines (k∈{5,10}k\in\{5,10\}). For the linear and MLP heads, the learning rate ({10−3, 3×10−4}\{10^{-3},\ 3\times10^{-4}\}) and weight decay ({10−4,10−3}\{10^{-4},10^{-3}\}) are selected via 3-fold cross-validation on the training set; the head is then trained on the full training set for up to 30 epochs with early stopping (patience 3) on a held-out 10% validation split, with batch size 512. Hyperparameters are selected once per (model, dataset, head) and shared across seeds.

Finetuned general-purpose models. The full network is optimized end-to-end with standard augmentation (random D4D_4 transforms; color jitter [brightness, contrast, saturation ±0.25\pm0.25; hue ±0.05\pm0.05]; random resized crops [scale 0.6–1.0]; Gaussian blur), differential learning rates (10−410^{-4} backbone, 10−310^{-3} head), cosine annealing over up to 30 epochs, batch size 64, early stopping patience 5, and label smoothing 0.1.

Seeds. We run 5 random seeds ({0, 1, 2, 3, 42}) and report mean ±\pm SEM (standard error of the mean); all patch-level results are averaged over the five seeds; the kkNN classifier head and all TTA evaluation are deterministic. The seed controls the initialization of the trained head, the mini-batch order, and which 10% of the training data is held out for early stopping, and, for finetuned models, augmentation sampling. Hyperparameter selection, the frozen embeddings, the test split and TTA evaluation are identical across seeds, so the reported variability reflects training stochasticity alone.

Implementation

Repository identifiers, pinned revisions, access conditions and references for all pretrained backbones are listed in Table 7. D4WRN was constructed using the escnn library (Cesa et al., [4]; Weiler and Cesa, [33]) as a Wide ResNet-28-6 with steerable convolutions over the dihedral group D4D_{4}, yielding 12M parameters, comparable to ResNet-18—the weakest model. Feature maps use regular field types throughout, aggregating in the final stage to produce D4D_{4}-invariant embeddings. For the frozen-backbone experiments we pre-extract embeddings for all eight D4D_{4} views of every test patch (and a single view of each training patch) in one batched, mixed-precision (FP16) pass per backbone, then train and evaluate all heads on the cached embeddings; per-strategy TTA results are obtained by aggregating the cached per-view embeddings/logits, avoiding redundant forward passes.

ModelRepositoryRevisionAccessReference
Histology foundation models
CTransPathlaurent/swin_tiny_patch4_window7_224.CTransPath59c740eopenWang et al. [31]
Phikonowkin/phikon057cc02openFiliot et al. [10]
Phikon-v2owkin/phikon-v22ae989aopenFiliot et al. [11]
UNIMahmoodLab/UNIb55a5ecagreementChen et al. [5]
UNI-v2MahmoodLab/UNI2-hd517a8dagreementChen et al. [5]
Virchowpaige-ai/Virchow19eebc8agreementVorontsov et al. [30]
Virchow-v2paige-ai/Virchow23158645agreementZimmermann et al. [38]
GigaPathprov-gigapath/prov-gigapatheba85ddagreementXu et al. [35]
H-Optimus-1bioptimus/H-optimus-1910efa9agreementScalbert et al. [26]
General-purpose models
ConvNeXtV2-Ttimm/convnextv2_tiny.fcmae_ft_in22k_in1kb1dd462openWoo et al. [34]
ConvNeXtV2-Btimm/convnextv2_base.fcmae_ft_in22k_in1k9e250edopenWoo et al. [34]
DINOv2-S/Btorch.hub: facebookresearch/dinov2vits14/vitb14openOquab et al. [24]
ResNet-18/50torchvisionV1/V2openHe et al. [14]

Table 7. Pretrained backbones. Revision: commit id, model name, or torchvision weights. Agreement: model requires accepting a license agreement.

Results

TTA Consistently Improves Predictions

Across the 15 frozen-backbone models and 4 datasets, D4D_{4} TTA improves balanced accuracy in 50 of 60 model–dataset configurations, indicating that group-averaging is a reliable, but modest, post hoc gain (binomial test vs. 50%: p<10−6p < 10^{-6}). Improvements are larger for general-purpose backbones than for histology FMs (Figure 2). We also observe a clear trend in D4D_{4} prediction stability: models with smaller ΔTTA\Delta_{\mathrm{TTA}} tend to show lower view-induced flip rates (Figure 3(a)).

Figure 3. Prediction entropy explains D4D_4 TTA outcomes. (a) Per model (TCGA-UT; other datasets as offset segments), predictions flipped by D4D_4 TTA split into corrections and corruptions, sorted by ΔTTA\Delta_{\mathrm{TTA}}. The thick segment is the net effect; the right margin gives the correction/corruption ratio. Models with small ΔTTA\Delta_{\mathrm{TTA}} flip few predictions. (b) On TCGA-UT, binned by baseline normalized entropy: flips and net gains concentrate in high-entropy predictions, where corrections outpace corruptions. (c) Model-level association (MHIST excluded). Mean entropy is moderately associated with ΔTTA\Delta_{\mathrm{TTA}} (Spearman ρ=0.54\rho= 0.54, p=10−4p = 10^{-4}); within datasets ρ=0.89\rho= 0.89 (TCGA-UT), 0.710.71 (NCT-CRC-HE-100K) and 0.250.25 (NCT-CRC-HE-NoNorm, where stain variation dominates; § 4.8). This trend is descriptive; routing rests on the per-patch results in (b) and Table 4, and is evaluated in § 4.7.

Averaged across all four datasets and 5 seeds, histology FMs gain +0.30 pp (95% CI [+0.13, +0.47] across the 36 model–dataset cells; Wilcoxon signed-rank p<10−3p < 10^{-3}), 4.3×\times smaller than general-purpose models (+1.28 pp; 95% CI [+0.94, +1.62], p<10−6p < 10^{-6}). The full per-model, per-dataset breakdown is reported in Table 5. Decomposing each ΔTTA\Delta_{\mathrm{TTA}} into the patches TTA corrects versus corrupts (Figure 3(a)), histology FMs show both fewer corrections and a lower net correction rate than general-purpose backbones.

General-purpose backbones gain more than histology FMs on every dataset individually, significantly on all but NCT-CRC-HE-NoNorm (Table 3). The performance gap between model families is significant by a rank-based test that makes no linearity assumption (general > FM, Mann–Whitney U/Wilcoxon rank-sum test p<10−3p < 10^{-3} on TCGA-UT), and the association between ΔTTA\Delta_{\mathrm{TTA}} and baseline accuracy is negative in all four datasets, strongly on TCGA-UT and NCT-CRC-HE-100K (Spearman rank correlation ρ=−0.90\rho= -0.90, −0.75-0.75) and weakly on the other two (Fisher’s combined p<10−5p < 10^{-5}).

DatasetFamilyBase (%)V4V_{4}D4D_{4}95% CI (D4D_{4})Corr./Corrupt.Agree. (%)
TCGA-UTHistology FM73.8+0.37+0.50[+0.34, +0.66]1.4295.6
General46.4+1.48+1.98[+1.21, +2.76]1.7483.0
NCT-CRC-HE-100KHistology FM94.0+0.21+0.29[−0.14, +0.71]2.1598.7
General89.1+0.81+0.93[+0.50, +1.37]1.6696.2
NCT-CRC-HE-NoNormHistology FM86.9+0.30+0.30[−0.21, +0.82]1.3897.2
General74.3+0.55+0.64[+0.15, +1.12]1.3593.1
MHISTHistology FM77.3+0.14+0.11[−0.23, +0.46]1.0696.5
General76.1+0.91+1.56[+0.65, +2.47]1.4892.7

Table 3. Summary of D4D_{4} TTA effects by dataset and model family (frozen linear probes, probability-mean aggregation). Base is single-view balanced accuracy; ΔTTA\Delta_{\mathrm{TTA}} is the mean gain in percentage points from V4V_{4} and D4D_{4} TTA, with the 95% confidence interval over models for D4D_{4}. Corr./Corrupt. is the ratio of predictions TTA fixes to those it breaks, and Agree. the base-vs-D4D_{4} prediction agreement rate. General-purpose backbones start lower, agree less across views, and gain more; histology FMs gain little beyond flips.

Where the losses occur. Histology FMs gain on average on every dataset (significantly on TCGA-UT and pooled across datasets) and improve in 27 of their 36 model–dataset combinations (9 FMs ×\times 4 datasets). The nine losses are small, averaging −0.21 pp against +0.47 pp for the gains, and concentrate on MHIST, the smallest test set; none occurs on TCGA-UT. Across all models and datasets, TTA corrects 1.6 predictions for every one it corrupts, and every model corrects more than it corrupts (Figure 3(a)), even where its net gain is small.

Robustness checks. These conclusions are robust to the choice of lightweight classifier head on top of frozen features (linear vs. MLP vs. kNN), with head-family averages summarized in Table 6. They are likewise robust to pretraining-corpus overlap: excluding the three histology FMs whose pretraining corpora include TCGA slides (Table 1) widens the gap on TCGA-UT from 3.9× to 4.7×, and the gap replicates on the three patch datasets with no TCGA content.

FamilyHeadTCGA-UT (31 cls) BaseTCGA-UT (31 cls) ΔTTA\Delta_{\mathrm{TTA}}NCT-CRC-100K (9 cls) BaseNCT-CRC-100K (9 cls) ΔTTA\Delta_{\mathrm{TTA}}NCT-CRC-NoNorm (9 cls) BaseNCT-CRC-NoNorm (9 cls) ΔTTA\Delta_{\mathrm{TTA}}MHIST (2 cls) BaseMHIST (2 cls) ΔTTA\Delta_{\mathrm{TTA}}
General-purpose modelskNN (kk=10)27.6+0.8683.9+0.9980.3+1.1971.9+0.95
Linear46.4+1.9889.1+0.9374.3+0.6476.1+1.56
MLP-1h48.4+2.3389.8+1.1773.8+1.0676.8+1.73
MLP-2h48.3+2.3889.6+1.1373.5+1.3777.3+1.67
Histology FMskNN (kk=10)63.0+0.3294.2+0.2389.9−0.0173.8−0.14
Linear73.8+0.5094.0+0.2986.9+0.3077.3+0.11
MLP-1h74.5+0.3994.0+0.3086.4+0.3980.2+0.33
MLP-2h74.5+0.4093.9+0.3685.5+0.4880.0+0.45

Table 6. Mean D4D_4 TTA benefit (ΔTTA\Delta_{\mathrm{TTA}} in balanced accuracy, pp) by classifier head type and model family (frozen backbone, probability-mean aggregation). Values averaged over all models within each family and 5 seeds (kNN: 1 seed, deterministic). Base: mean no-TTA balanced accuracy (%). ΔTTA\Delta_{\mathrm{TTA}}: pp gain from D4D_4 TTA.

Headroom. We test whether the decline in TTA benefit for stronger models is simply due to reduced error (less “headroom”) by measuring the net correction rate per dataset. On TCGA-UT, the corrected-to-corrupted ratio becomes less favorable as model quality increases (roughly from ∼2:1 for the weakest backbones to ∼1.3:1 for the strongest FMs), suggesting that the remaining mistakes of stronger models are less often fixable by simple geometric averaging. The remaining gains concentrate among high-entropy predictions (Figure 3(b–Figure 3c)), a pattern we explain in § 4.6. Table 5 reports the per-model results for all datasets; on TCGA-UT, general-purpose backbones tend to benefit the most, whereas histology FMs, especially the largest ones, show consistently smaller improvements.

D4WRN as a near-zero reference. The D4WRN, exactly D4D_4-equivariant by construction, shows no TTA benefit on average (+0.00 pp over all four datasets, within ±0.2 pp on each; Table 5), despite being much smaller (∼12M parameters) and having no large-scale pretraining. We treat it as an informative but uncontrolled reference, since it differs from histology FMs in architecture (Wide ResNet vs. ViT), scale (86M–1.1B parameters), and training protocol.

Frozen Backbones Benefit More Than Finetuned

We evaluate the six general-purpose backbones in both frozen (linear-probe) and finetuned (end-to-end) configurations across all four datasets. Frozen backbones derive a significantly larger benefit from D4D_4 TTA, gaining +1.28 pp versus +0.80 pp when finetuned—a 1.6× reduction in TTA utility. Importantly, finetuning simultaneously raises the mean baseline balanced accuracy from 71.5% to 79.9% (+8.4 pp), meaning this shrinking TTA benefit accompanies a much stronger base model overall.

The reduction varies by dataset: it is largest on the two large, stain-normalized training sets (TCGA-UT 2.5×, NCT-CRC-HE-100K 2.3×), weaker on NCT-CRC-HE-NoNorm (1.3×), where stain rather than geometric variation dominates the error (§ 4.8), and absent on MHIST (1.0×), whose 2,175 training patches are too few to instill invariance. Finetuning with augmentation thus absorbs part of the TTA benefit only when training data are sufficient and geometric variation limits accuracy. We conjecture that the same mechanism underlies the small gains of the histology FMs, pretrained with geometric augmentation at far larger scale, but do not test this directly (§ 6.3).

Strategy Progression: Diminishing Returns Beyond Flips

Doubling the views from V4V_4 to D4D_4 adds little: most of the gain comes from flips alone, especially for histology FMs, for which the proper rotations add almost nothing (Table 3 and Table 5). The aggregation method matters little as well: logit-mean and probability-mean averaging are equivalent and confidence weighting is slightly worse, whereas majority voting forfeits about 30% of the gain.

TTA Lowers Predicted Confidence

Histology FMs are not better calibrated than general-purpose backbones; on TCGA-UT they are consistently overconfident (ECE 0.058 vs. 0.011; Figure 5). D4D_4 averaging lowers predicted confidence on average, so its effect depends on the starting point: it reduces overconfidence in histology FMs (ECE improves in 26 of 36 model–dataset pairs, Wilcoxon p=0.02p = 0.02), but pushes already well-calibrated general-purpose backbones into underconfidence about as often as it improves them. Pooled over all models, ECE is unchanged, while the top-label Brier score improves (p<10−8p < 10^{-8}), partly reflecting the accuracy gain. TTA thus reduces overconfidence where it is present but does not replace explicit recalibration.

Figure 5. Reliability on TCGA-UT, single-view versus D4D_4 TTA. Points below the diagonal indicate overconfidence; grey bars show the share of predictions in each confidence bin. Curves and ECE are computed per model and seed and averaged within each family; bins holding fewer than 0.5% of a model’s predictions are omitted from the curve but retained in the ECE. Histology FMs are overconfident, and D4D_4 TTA moves them toward the diagonal (ECE 0.058 → 0.049); general-purpose backbones are near-calibrated, and TTA pushes them into underconfidence (0.011 → 0.036). Histology FMs place 65% of their predictions in the top confidence bin, against 21% for general-purpose backbones.

Agreement Rate Quantifies Prediction-Level Invariance

Histology FMs achieve a mean base-vs-D4D_4 prediction agreement rate of 97.0% (range 91.8%–99.7%) compared to 91.2% (range 74.3%–97.5%) for general-purpose models; per-dataset values are given in Table 3. This gap quantifies D4D_4 stability at the prediction level, which, as § 4.6 shows, is a property of the classifier’s decision geometry and is not explained by embedding-space (representation) invariance. Within the FM family the ranking tracks model scale: CTransPath shows the lowest agreement (91.8% on TCGA-UT) while the largest models (Virchow-v2, H-Optimus-1) exceed 98% averaged over datasets.

Baseline softmax entropy also predicts prediction correctness, not just TTA flip events. Across model × dataset combinations, entropy achieves an Area Under the Receiver Operating Characteristic (AUROC) of 0.849 ± 0.058 for discriminating incorrect from correct predictions in histology FMs (general-purpose: 0.816 ± 0.049). Incorrect predictions carry 4.3× higher normalized entropy than correct ones for FMs (2.6× for general-purpose models): when the model is uncertain, it is substantially more likely to be wrong.

A Flip Occurs When the Orbit Straddles the Decision Boundary

We first test the natural hypothesis that representation-level D4D_4 invariance explains TTA benefit, measuring orbit tightness by MPCS (2). It does not: histology FMs are about as orbit-invariant as general-purpose models (mean MPCS 0.953 vs. 0.942), and per patch, orbit tightness barely separates flipped from unflipped predictions (AUROC 0.57, near chance). The architecturally D4D_4-equivariant D4WRN is the one exception, collapsing each orbit almost to a point (MPCS ≈0.999\approx0.999). This confirms that MPCS detects invariance when it is present, and that FMs’ high prediction agreement (§ 4.5) is not embedding-level invariance.

A flip happens only when the orbit straddles the decision boundary between the base-view top-two classes. The two boundary-geometry quantities, the geometric margin (3) and the straddle ratio (4), separate flipped from unflipped patches almost perfectly and match or exceed softmax entropy, whereas the two orbit-size quantities barely do (Table 4). Flip risk therefore varies with margin rather than orbit size, and the separation is wider for histology FMs than for general-purpose backbones.

PredictorAllHisto FMGeneral
Orbit position relative to the boundary
Straddle ratio ∣r∣|r|0.954±0.0350.954 \pm0.0350.9740.9740.9240.924
Geometric margin ∣m∣|m|0.947±0.0370.947 \pm0.0370.9690.9690.9140.914
Softmax entropy (proxy)0.925±0.0540.925 \pm0.0540.9500.9500.8870.887
Orbit size
Orbit spread ss0.586±0.0560.586 \pm0.0560.6000.6000.5660.566
Orbit tightness (MPCS)0.568±0.0770.568 \pm0.0770.5700.5700.5660.566

Table 4. AUROC for predicting whether D4D_4 TTA flips a patch prediction. Computed per patch for the 15 frozen linear probes on all four patch datasets (mean ±\pm SD over the 60 model–dataset combinations). Where the orbit sits relative to the decision boundary predicts flips; how large the orbit is does not.

This recasts the FM/general gap as a margin effect, not an invariance one. Since orbit size is essentially equal across families, FMs flip less only because their more separable representations place the same-sized orbit farther from decision boundaries, crossing one 3.4× less often than general models on TCGA-UT patches with ∣r∣<1|r| < 1: 9.9% vs. 33.9%). Because softmax entropy is a cheap proxy for this margin, it provides an effective signal for deciding which patches to augment, which we exploit next.

Selective TTA Preserves Benefits at Reduced Inference Cost

Histology FM predictions are highly confident (low entropy) on the large majority of patches, so full D4D_4 TTA adds redundant computation for most inputs. We exploit this by applying selective TTA: single-view inference for samples whose base-view softmax entropy falls below a threshold, and full D4D_4 for uncertain samples:

f^sel(x)={f(x)if H[σ(f(x))]≤t⋅ln⁡C1∣G∣∑g∈Gf(g⋅x)otherwise(5)\hat{f}_{\mathrm{sel}}(x) = \begin{cases} f(x) & \text{if } H[\sigma(f(x))] \le t \cdot\ln C \\ \frac{1}{\lvert G\rvert}\sum_{g\in G} f(g\cdot x) & \text{otherwise} \end{cases} \tag*{(5)}

where normalizing by ln⁡C\ln C makes the threshold t∈[0,1]t \in[0,1] comparable across datasets with different numbers of classes CC. Figure 4 shows the efficiency–accuracy tradeoff curve (Pareto frontier) for histology FMs and general-purpose models on TCGA-UT. Averaged over the histology FMs on TCGA-UT:

Figure 4. Selective TTA accuracy–efficiency trade-off on TCGA-UT. Each point corresponds to an entropy threshold tt for routing samples to full D4D_4 TTA; lower tt applies TTA to more samples (higher cost) and vice versa. Histology FMs recover nearly the full D4D_4 benefit while saving a substantial fraction of forward passes. Thin lines indicate individual model curves, the thick line indicates their mean, and the shaded area indicates the confidence interval.

  • At t=0.25t = 0.25: only 21% of samples receive TTA (saving 79% of forward passes), recovering +0.43 pp, about 85% of the full benefit as in Figure 4.

  • Moreover, at t=0.05t = 0.05: 46% of samples receive TTA, recovering 100.0% of full TTA gain at +0.50 pp.

For general-purpose models, savings are more modest as their higher disagreement means more samples genuinely benefit. Base-view entropy is thus an effective routing signal, concentrating TTA cost on the few unstable predictions. Routing can concentrate the flips but not improve their quality: among patches that flip, no base-view signal (entropy or margin) separates beneficial corrections from harmful corruptions (AUROC ≈0.56\approx0.56, near chance), so the net benefit of selective TTA is bounded by the intrinsic correction-to-corruption ratio of the flipped set.

Stain Normalization Matters More Than Geometric TTA

Comparing NCT-CRC-HE-100K (stain-normalized) with NCT-CRC-HE-NoNorm reveals that stain normalization has a far larger effect than geometric TTA. For histology FMs, the mean baseline balanced accuracy drops by 7.2 pp without stain normalization (94.0% → 86.9%), while TTA recovers an average of +0.3 pp on both datasets. This suggests that, on these benchmarks, geometric D4D_4 invariance is not the dominant source of error for modern histology FMs, whereas stain and scanner robustness remains a significant open challenge.

While the preceding results treat tiles as independent samples, clinical deployment and many research pipelines operate on WSIs. We therefore ask whether the (modest) tile-level corrections induced by D4D_4 averaging translate into improvements for slide-level prediction.

WSI-Level Analysis

Datasets and Setup

Camelyon17 (Bandi et al. [1]) provides 500 H&E-stained lymph-node WSIs from 5 acquisition centers (100 slides, 20 patients per center); 50 slides carry pixel-level lesion polygon annotations. Tissue patches are 256×256256 \times256 at 20× magnification, taken from CLAM segmentation. The frozen UNI patch probe (binary tumor/normal) is evaluated leave-one-patient-out over the 50 annotated slides to avoid leakage. For slide-level MIL we use the standard domain-generalization split: train on centers 0–3 and test on the unseen-scanner center 4 (100 slides).

PANDA (Bulten et al. [2]) provides prostate biopsy WSIs with pixel-level Gleason-grade annotations. We binarize annotations into cancer / non-cancer and extract 256×256256 \times256 patches at 20× magnification, yielding 10,353 slides with at least one patch (3,721 with ≥1\ge1 cancer patch).

Slide-level TTA and MIL training. Because patch embeddings are cached for all eight D4D_4 views, slide-level TTA can be applied at two points, which are not equivalent because the aggregator is non-linear: feature-level averages a patch’s eight view embeddings before aggregation, while logit-level runs the trained MIL head once per view and averages the slide logits. We report feature-level results; logit-level gives the same null. The MIL head is gated-attention Attention-based Deep Multiple Instance Learning (ABMIL) (Ilse et al. [15]) (attention dimension 256, classifier hidden dimension 512, dropout 0.25), trained under cross-entropy with Adam (learning rate 10−410^{-4}, weight decay 10−510^{-5}), cosine annealing over 50 epochs and early stopping with patience 15. Heads are trained on single-view features and TTA is applied only at inference, so each comparison isolates test-time averaging on a fixed model.

Patch Corrections Do Not Transfer to Slide-Level MIL

Adding D4D_4 TTA to a trained ABMIL model (Ilse et al. [15]) does not measurably change bag-level performance. On Camelyon17 this null holds across six backbones spanning weak general-purpose models to strong FMs (ResNet-50, ConvNeXtV2, Phikon-v2, GigaPath, UNI and Virchow-v2). Because TTA is toggled at inference on a fixed trained head, each run forms a paired comparison, which bounds the slide-level effect to within ±0.003 in test AUROC (95% CI, pooled over backbones and seeds). The design detects changes several times smaller than the gains TTA produces at the patch level, so this bounds the effect rather than merely failing to find one. No backbone shows a change surviving correction for six comparisons. On PANDA, ABMIL quadratic-weighted κ\kappa is 0.901 with TTA versus 0.905 without, when using the UNI backbone evaluated across 3 cross-validation folds.

The null is structural rather than a property of any one backbone. It holds even for ResNet-50 and ConvNeXtV2, whose patch features are far from D4D_4-invariant and which gain the most from patch-level TTA (§ 4). ABMIL pools hundreds to thousands of patches per slide, and this aggregation already averages out the small, largely canceling per-patch perturbations that D4D_4 introduces, leaving the bag representation essentially unchanged. The effect is not that attention ignores the affected patches: using the patch probe as a diagnostic, the ∼0.1%\sim0.1\% of patches whose prediction D4D_4 would flip fall in the top decile of UNI ABMIL attention. They are simply too few to move the pooled logit.

At the patch level, the corrections are modest and headroom-gated, as on the tile benchmarks. Leave-one-patient-out per-slide Dice improves on 59% of PANDA cancer slides (mean +0.66 pp; n=3,721n = 3{,}721), with the gain concentrated on poorly-segmented slides (+1.0 pp in the lowest baseline-Dice tertile versus +0.25 pp in the highest). The frozen UNI probe already segments tumor well, leaving little headroom for D4D_4 averaging to recover.

Discussion

Prediction Stability Reflects Joint Backbone–Classifier Geometry

Embedding invariance is neither necessary nor sufficient for stable predictions. Histology FMs reach 97% D4D_4 agreement with orbits as large as those of general-purpose models, because larger margins keep each orbit on one side of the decision boundary; D4WRN reaches stability the other way, by collapsing its orbits. Two implications follow. First, embedding-level invariance measures such as orbit tightness are a poor guide to whether TTA will help; the margin, or its inexpensive proxy, prediction entropy, is the relevant quantity. Second, because stability depends on the classifier as well as the backbone, TTA benefit is a property of the whole pipeline rather than of the FM alone, consistent with its reduction under finetuning (§ 4.2).

Practical Recommendations

  1. Histology FMs: the +0.30 pp average gain rarely justifies the 8×8\times inference cost, so skip TTA by default; stain normalization (§ 4.8) is a higher-impact priority.

  1. General-purpose backbones: TTA is more worthwhile (+1.28 pp), especially for frozen linear probes, whose linear head cannot learn invariance from training data (+0.80 pp when finetuned; § 4.2).

  1. High-stakes applications: where small gains matter at scale, TTA may justify its cost, since it corrects more predictions than it corrupts.

  1. If TTA is used: average probabilities or logits rather than voting, and apply it selectively (§ 4.7); routing patches whose normalized entropy exceeds t≈0.25t \approx0.25 augments about one in five FM patches while recovering most of the benefit, and tt can be lowered if the full gain is required.

  1. Calibration: for histology FMs, TTA modestly reduces overconfidence; where calibrated probabilities matter, explicit recalibration is cheaper.

Limitations

Our study has several limitations. First, training protocol is a confound when comparing histology FMs (frozen linear probes only) to general-purpose models (frozen and finetuned). Finetuning reduces mean TTA gain from +1.28 pp to +0.80 pp, so part of the FM vs. general-purpose gap may reflect downstream adaptation rather than pretraining alone. This reduction is itself dataset-dependent (§ 4.2), and we do not test it on histology FMs directly. Second, the D4WRN comparison is uncontrolled (architecture, scale, and training protocol all differ; § 4.1); a matched equivariant control is left to future work. Third, our WSI analysis (§ 5) is limited to two datasets with patch-level annotations (Camelyon17, PANDA); generalization to other cancer types and scanners remains to be evaluated. Fourth, our statistical tests (binomial test, Wilcoxon signed-rank, Mann–Whitney/Wilcoxon rank-sum, Spearman correlation with Fisher’s combined p-value) are each reported without correction for the multiple comparisons made across datasets and models; future work could apply multiple-comparison corrections. Fifth, datasets are limited to H&E staining; immunohistochemistry or other stain types may have different symmetry properties.

Future Directions

Our results point to several promising research directions:

  • Stain-equivariant architectures: Analogous to D4D_4-equivariant networks for geometric transforms, architectures with built-in stain invariance could address the larger remaining gap.

  • Adaptive thresholding for selective TTA: We demonstrate entropy-based routing at fixed thresholds (§ 4.7); future work could learn per-dataset or per-class thresholds, or jointly optimize the routing policy with the classifier head.

  • Invariance probing: Developing systematic benchmarks for measuring the degree of learned invariance across different transformation groups (geometric, photometric, stain) would help characterize FM capabilities beyond standard accuracy metrics.

Conclusion

We have presented the first large-scale evaluation of TTA across 16 vision models and 4 histopathology datasets. Our results show that modern histology FMs exhibit high prediction stability under D4D_4 transforms and gain only +0.30 pp from D4D_4 TTA, a 4.3× smaller benefit than general-purpose models. While TTA remains a safe and reliable technique, improving most configurations and correcting more predictions than it corrupts, its marginal benefit for histology FMs suggests that geometric D4D_4 robustness is often not the primary performance bottleneck for tile-level classification with modern FMs.

We further show that entropy-based selective TTA, which routes only uncertain samples to full D4D_4, recovers over half of the TTA benefit for histology FMs while augmenting only ∼13% of patches, providing a practical path to reduced-cost TTA when inference budget is constrained. By contrast, stain normalization has a far larger effect on these models, pointing to stain and scanner robustness as the more pressing open challenge. We release our complete evaluation framework to facilitate further research into invariance properties of histopathology foundation models.

Acknowledgments

We utilized generative artificial intelligence (AI) to assist in creating and refining code segments based on specified tasks, including debugging, editing, and autocompleting code. Additionally, generative AI tools were used to help structure sentences and perform grammar checks. The conceptualization and ideation, as well as all instructions given to the AI, were exclusively the result of the authors’ creative and intellectual input. We take full responsibility for reviewing and ensuring the accuracy and quality of all AI-generated content included in this work.

Data Availability

All datasets are available from their original providers: TCGA-UT [18] (Hugging Face dakomura/tcga-ut); NCT-CRC-HE-100K, its non-normalized variant and CRC-VAL-HE-7K [16] (Zenodo; we use the Hugging Face mirror 1aurent/NCT-CRC-HE); MHIST [32] (from the dataset authors upon completion of a data-use form); Camelyon17 [1] (Camelyon17 challenge); and PANDA [2] (Kaggle, under the competition terms). Repositories, revisions and access conditions of all pretrained backbones are listed in Table 7. Code for embedding extraction, training, TTA evaluation and all analyses, together with configuration files, the seeded procedure that regenerates the PANDA cross-validation folds, and the compiled results underlying all tables and figures, is available at https://github.com/lars-chen/histology-tta. Cached embeddings are not redistributed given the license terms of several source models; they can be regenerated with the released code.

Additional Tables and Figures

References

  1. [1]Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550–560, 2018.DOI
  2. [2]Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester Van Boven, Robert Vink, et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine, 28(1):154–163, 2022.DOI
  3. [3]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF international conference on computer vision (ICCV), pages 9630–9640. IEEE, 2021.arxiv.org/abs/2104.14294
  4. [4]Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build e (n)-equivariant steerable cnns. In International conference on learning representations, 2022.
  5. [5]Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature medicine, 30(3):850–862, 2024.DOI
  6. [6]Benjamin Chidester, Tianming Zhou, Minh N Do, and Jian Ma. Rotation equivariant and invariant neural networks for microscopy image analysis. Bioinformatics, 35(14):i530–i537, 2019.DOI
  7. [7]Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990–2999. PMLR, 2016.arxiv.org/abs/1602.07576
  8. [8]Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In International conference on machine learning, pages 2959–2969. PMLR, 2021.arxiv.org/abs/2102.10333
  9. [9]Matouš Elphick, Guang Yang, and Samra Turajlic. Are the latent representations of foundation models for pathology invariant to rotation? In Medical Imaging with Deep Learning - Short Papers, 2025. URL https://openreview.net/forum?id=7402G6pGEj.arxiv.org/abs/2412.11938
  10. [10]Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Axel Camara, Alice Mac Kain, Charlie Saillard, and Jean-Baptiste Schiratti. Scaling self-supervised learning for histopathology with masked image modeling. MedRxiv, pages 2023–07, 2023.DOI
  11. [11]Alexandre Filiot, Paul Jacob, Alice Mac Kain, and Charlie Saillard. Phikon-v2, a large and public feature extractor for biomarker prediction. arXiv preprint arXiv:2409.09173, 2024.arxiv.org/abs/2409.09173
  12. [12]Valentino F Foit, David W Hogg, and Soledad Villar. Group averaging for physics applications: Accuracy improvements at zero training cost. arXiv preprint arXiv:2511.09573, 2025.arxiv.org/abs/2511.09573
  13. [13]Zeyu Gao, Anyu Mao, Yuxing Dong, Hannah Clayton, Jialun Wu, Jiashuai Liu, ChunBao Wang, Kai He, Tieliang Gong, Chen Li, et al. Smmile enables accurate spatial quantification in digital pathology using multiple-instance learning. Nature cancer, 6(12):2025–2041, 2025.DOI
  14. [14]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.arxiv.org/abs/1512.03385
  15. [15]Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. pages 2127–2136, 2018.
  16. [16]Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine, 16(1):e1002730, 2019.
  17. [17]Ildoo Kim, Younghoon Kim, and Sungwoong Kim. Learning loss for test-time augmentation. Advances in neural information processing systems, 33:4163–4174, 2020.arxiv.org/abs/2010.11422
  18. [18]Daisuke Komura, Akihiro Kawabe, Keisuke Fukuta, Kyohei Sano, Toshikazu Umezaki, Hirotomo Koda, Ryohei Suzuki, Ken Tominaga, Mieko Ochi, Hiroki Konishi, et al. Universal encoding of pan-cancer histology by deep texture representations. Cell Reports, 38(9), 2022.DOI
  19. [19]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. volume 25, 2012.
  20. [20]Maxime W Lafarge, Erik J Bekkers, Josien PW Pluim, Remco Duits, and Mitko Veta. Roto-translation equivariant convolutional networks: Application to histopathology image analysis. Medical Image Analysis, 68:101849, 2021.arxiv.org/abs/2002.08725
  21. [21]Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q Nelson, Greg S Corrado, et al. Detecting cancer metastases on gigapixel pathology images. arXiv preprint arXiv:1703.02442, 2017.arxiv.org/abs/1703.02442
  22. [22]Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555–570, 2021.DOI
  23. [23]Nikita Moshkov, Botond Mathe, Attila Kertesz-Farkas, Reka Hollandi, and Peter Horvath. Test-time augmentation for deep learning-based cell segmentation on microscopy images. Scientific reports, 10(1):5068, 2020.DOI
  24. [24]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.arxiv.org/abs/2304.07193
  25. [25]Fábio Perez, Cristina Vasconcelos, Sandra Avila, and Eduardo Valle. Data augmentation for skin lesion analysis. pages 303–311, 2018.DOI
  26. [26]Marin Scalbert, Charlie Saillard, Thomas Peeters, Liam Gonzalez, Dasha Valter, Felipe Llinares-López, Zelda E. Mariet, and Rodolphe Jenatton. H-optimus-1: A foundation model for computational histopathology. Cancer Research, 86 (8_Suppl):LB174, 2026. doi: 10.1158/1538-7445.AM2026-LB174.
  27. [27]Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In 2021 IEEE/CVF international conference on computer vision (ICCV), pages 1194–1203. IEEE, 2021.arxiv.org/abs/2011.11156
  28. [28]Jongwook Son and Seokho Kang. Efficient improvement of classification accuracy via selective test-time augmentation. Information Sciences, 642:119148, 2023.DOI
  29. [29]Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In International Conference on Medical image computing and computer-assisted intervention, pages 210–218. Springer, 2018.arxiv.org/abs/1806.03962
  30. [30]Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, et al. Virchow: A million-slide digital pathology foundation model. arXiv preprint arXiv:2309.07778, 2023.arxiv.org/abs/2309.07778
  31. [31]Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81:102559, 2022.DOI
  32. [32]Jerry Wei, Arief Suriawinata, Bing Ren, Xiaoying Liu, Mikhail Lisovsky, Louis Vaickus, Charles Brown, Michael Baker, Naofumi Tomita, Lorenzo Torresani, et al. A petri dish for histopathology image analysis. pages 11–24, 2021.arxiv.org/abs/2101.12355
  33. [33]Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Conference on Neural Information Processing Systems (NeurIPS), 2019. URL https://arxiv.org/abs/1911.08251.
  34. [34]Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16133–16142. Ieee, 2023.arxiv.org/abs/2301.00808
  35. [35]Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 630(8015):181–188, 2024.DOI
  36. [37]Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT Pre-Training with Online Tokenizer. In ICLR, 2022.arxiv.org/abs/2111.07832
  37. [38]Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, et al. Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv preprint arXiv:2408.00738, 2024.arxiv.org/abs/2408.00738

Paper details

vlnc:W_reru6ajr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative artificial intelligence (AI)

View statement
Looking for related papers…
Contents