betaFeedback

WorksEngineering

When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis

Abstract

Histopathology and computed tomography (CT) provide complementary views of renal cancer. We connect these modalities through bidirectional cross-attention between 16 histology prototypes and up to 32 kidney-centered CT tokens from frozen foundation models. In 210 TCGA-KIRC patients with 46 disease-specific survival (DSS) events, 5-fold evaluation yielded a C-index of 0.760±0.0810.760\pm0.081 and 3-year AUROC of 0.773 (95% CI 0.678–0.860), the highest averages among evaluated unimodal and fusion baselines. External testing in 180 CPTAC-CCRCC patients yielded DSS C-index of 0.629 with CT withheld. Attribution analysis linked predictions to image regions while exposing acquisition artifacts. These findings support further study of cross-scale fusion and independent multimodal validation.

Introduction

Renal cell carcinoma exhibits heterogeneity across microscopic tissue architecture and macroscopic tumor anatomy, motivating prognostic models that integrate information across imaging scales [3]. Histopathology captures cellular morphology and tissue organization, while computed tomography provides a volumetric view of the tumor and surrounding anatomy. Combining these complementary perspectives offers an opportunity to characterize disease aggressiveness more comprehensively than either modality alone.

Foundation models provide transferable pathology and CT representations that make such integration increasingly feasible [7, 8]. However, effective multimodal learning requires more than combining pretrained features. Global pooling can obscure tissue heterogeneity and regional anatomical information, while retaining thousands of image features complicates learning in small survival cohorts. Compact representations that preserve meaningful within-modality variation could provide a practical balance between expressivity and data efficiency. PANTHER addresses this challenge in histopathology by summarizing whole-slide images through morphological prototypes that encode tissue prevalence, appearance, and variability [1]. Multimodal prototyping extends this principle to histology and transcriptomics for survival prediction [2]. Renal imaging studies have also explored probabilistic graph integration of radiology and pathology and feature fusion for recurrence prediction [3, 4]. These approaches motivate further investigation of how compact tissue representations can interact with spatially organized CT information.

We investigate bidirectional cross-attention between histology prototypes and kidney-centered CT tokens for disease-specific survival prediction as illustrated in Figure 1. The model retains prototype-specific tissue statistics and regional CT features, allowing each modality to condition the other before patient-level aggregation. We compare this formulation with unimodal, early-fusion, and late-fusion models on a shared TCGA-KIRC cohort and examine external histology-pathway transfer on CPTAC-CCRCC with CT withheld. Attribution analysis further connects predictions to modality contributions and image regions, supporting examination of complementary information and potential acquisition artifacts.

Overall architecture diagram

Figure 1. Overall architecture diagram

Methods

Cohorts and endpoint

The primary cohort included 210 TCGA-KIRC patients with histopathology images, CT scans, and disease-specific survival (DSS) labels from TCGA-CDR [5]. The cohort included 46 DSS events with median follow-up of 3.7 years. Patients with missing or invalid outcomes or an empty mask were excluded, as were frozen sections. Imaging modes used no clinical covariates. For external evaluation, we used PANTHER’s CPTAC-CCRCC test list comprising 180 patients and 341 tumor slides, with 15 DSS events, 26 overall-survival (OS) events, and median follow-up of 2.5 years [6]. Since only one DSS event occurred among the 52 patients with CT, external testing withheld CT. Adding 291 histology-only TCGA patients expanded training for this experiment to 501 patients with 109 DSS events. Slides were pooled per patient for external training and testing.

Histology prototypes and anatomical CT tokens

Frozen UNI-2h [7] encoded 256 × 256-pixel tissue patches at 20× into 1,536-dimensional embeddings, yielding approximately 6,300 patches per slide. Following PANTHER [1], we initialized 16 shared prototypes using kk-means on 100,000 TCGA patches and fitted a patient-specific diagonal Gaussian mixture with one expectation–maximization iteration. Each prototype produced

hk=[log⁡(πk+10−4),μk,σk2]∈R3073,(1)\mathbf{h}_k = [\log(\pi_k + 10^{-4}), \mu_k, \sigma_k^2] \in\mathbb{R}^{3073}, \tag*{(1)}

encoding prevalence, mean appearance, and variability. Shared initialization established corresponding prototype slots across patients. The label-free dictionary was shared across TCGA folds and fixed for CPTAC.

For each patient, we selected one axial CT series from the earliest eligible study, prioritizing nephrographic or portal venous contrast phases. Eligible series contained ≥20\ge20 slices with spacing ≤5\le5 mm and coverage ≥100\ge100 mm. Volumes were oriented superior–posterior–left, resampled to 3×1×13 \times1 \times1 mm, and scaled from [−1024,2048][-1024,2048] HU to [0,1][0,1]. BAMF masks defined a kidney-region box with a 30 mm margin around retained components of ≥10\ge10 ml within a 250 mm superior–inferior span. Tumor labels were not supplied as predictive features. Frozen CT-FM [8] encoded the padded box in one pass. Adaptive pooling of its final 512-channel feature map onto a 2×4×42 \times4 \times4 grid produced M≤32M \le32 tokens cj∈R512c_j \in\mathbb{R}^{512}, with centers expressed relative to the kidney-mask centroid in 100 mm units. All feature standardization used training-fold statistics.

Bidirectional fusion and survival learning

We projected both token sets to d=64d = 64, adding learned prototype embeddings for histology and linearly embedded three-dimensional positions for CT. Each stream passed through one pre-normalized self-attention layer with four heads, feed-forward width 128, GELU activation, and dropout 0.25. Bidirectional cross-attention then updated the streams according to

Xm←Xm+MHA⁡m←n(Xˉm,Xˉn,Xˉn),(2)X_m \leftarrow X_m + \operatorname{MHA}_{m\leftarrow n}(\bar{X}_m,\bar{X}_n,\bar{X}_n), \tag*{(2)}

where Xˉ=LN⁡(X)\bar{X} = \operatorname{LN}(X), LN denotes layer normalization, and MHA denotes multi-head attention. The pairs (m,n)∈{(h,c),(c,h)}(m,n) \in\{(h,c),(c,h)\} represent the two directions between histology (hh) and CT (cc), with separate directional weights and CT padding masks. Following residual feed-forward updates, mean-pooled histology and gated-attention-pooled CT [9] were concatenated into a 128-dimensional representation and linearly mapped to risk rir_i.

Training minimized the event-averaged negative Cox partial log-likelihood with Breslow handling of ties,

L=−1Ne∑ℓ[∑i∈Dℓri−∣Dℓ∣log⁡∑j∣Tj≥tℓerj],(3)\mathcal{L} = -\frac{1}{N_e}\sum_{\ell}\left[\sum_{i\in D_\ell}r_i-\lvert D_\ell\rvert\log\sum_{j\mid T_j\ge t_\ell}e^{r_j}\right], \tag*{(3)}

where DℓD_\ell contains patients with events at distinct time tℓt_\ell, TjT_j is the observed follow-up time, and NeN_e is the total event count. Training-only modality dropout withheld CT or histology with probability 0.15, disabling that modality’s contribution to the other stream and zeroing its pooled representation. CT was withheld when unavailable, and both image encoders remained frozen.

Comparators and evaluation

We compared fusion against unimodal token models with linear heads and a histology MLP using log mixture weights and their weighted mean. Early fusion concatenated these histology features with pooled CT (2,064-D), followed by ridge Cox regression or a 32-unit MLP. Ridge penalties were tuned over {10−3,10−2,10−1,1,10}\{10^{-3},10^{-2},10^{-1},1,10\} using inner five-fold CV. Late fusion averaged standardized risks from independently trained token models.

All models used identical event-stratified five-fold splits. Neural models trained for 40 full-batch epochs across three seeds with AdamW and gradient clipping at 1. Learning rates and weight decay were (3×10−4,10−2)(3 \times10^{-4},10^{-2}) for token models and (10−4,0.1)(10^{-4},0.1) for MLPs. Per-seed risks were standardized against a separate five-repeat, five-fold protocol.

We reported fold-wise Harrell’s concordance (C-index) and three-year cumulative/dynamic AUROC and sensitivities at 80% and 90% specificity from pooled out-of-fold risks with inverse-probability-of-censoring weighting. Three-year events were cases, follow-up beyond three years defined controls, and earlier censorings were excluded (31 cases, 126 controls). Intervals used 1,000 patient-level bootstrap resamples. Thresholds derived from evaluation controls yielded descriptive ROC operating points.

Attribution analysis

Integrated gradients (IG) [10] explained the standardized three-seed risk from a model fitted to all 210 patients. The baseline replaced token content with cohort averages while retaining patient-specific CT positions and token count. A 64-step right-Riemann approximation yielded modality-, prototype-, and CT-token attributions with median absolute completeness error of 0.0026. To visualize histology contributions, prototype scores were distributed over patches using Gaussian-mixture posteriors and spatially aggregated. The resulting maps represent

prototype projections rather than patch-level IG. Attributions were in-sample, whereas risk percentiles were out-of-fold.

Results

Fusion performance on a common cohort

On the common cohort, cross-attention fusion achieved the highest mean C-index (0.760±0.0810.760 \pm0.081), three-year AUROC (0.7730.773), and sensitivity at 80% specificity (0.6030.603), alongside the lowest C-index SD (Table 2). At 90% specificity, sensitivities were similar across nonlinear fusion designs (0.4400.440–0.4480.448). Although fusion increased mean concordance over histology MLP, histology tokens, and CT by 0.0300.030, 0.0540.054, and approximately 0.0900.090, the corresponding paired intervals of [−0.151,0.211-0.151,0.211], [−0.104,0.212-0.104,0.212], and [−0.086,0.265-0.086,0.265] included zero and did not establish superiority.

ModelC-index3-year AUROC [95% CI]Sens., 80% spec.Sens., 90% spec.
CT tokens + positions0.671±0.1300.671 \pm0.1300.6750.675 [0.5640.564, 0.7820.782]0.4160.4160.1660.166
Histology tokens0.706±0.1110.706 \pm0.1110.7230.723 [0.6200.620, 0.8250.825]0.5700.5700.4430.443
Histology summary + MLP0.730±0.0830.730 \pm0.0830.7180.718 [0.6220.622, 0.8140.814]0.4480.4480.4180.418
Early fusion, ridge Cox0.688±0.1150.688 \pm0.1150.7110.711 [0.6100.610, 0.8140.814]0.4470.4470.1900.190
Early fusion, MLP0.744±0.1100.744 \pm0.1100.7380.738 [0.6390.639, 0.8360.836]0.5690.5690.4480.448
Late fusion, equal risks0.736±0.1160.736 \pm0.1160.7430.743 [0.6400.640, 0.8450.845]0.5340.5340.4400.440
Cross-attention fusion (F2)0.760±0.0810.760 \pm0.0810.7730.773 [0.6780.678, 0.8600.860]0.6030.6030.4430.443

Table 2. TCGA-KIRC DSS prediction (n=210n = 210). C-index is the 5-fold mean ±\pm SD. 3-year metrics use censoring-adjusted, pooled out-of-fold predictions. Brackets indicate 95% CIs and bold indicates highest means.

External histology-pathway evaluation

With CT withheld, the fusion model yielded external DSS C-index of 0.6290.629 [0.434,0.8160.434,0.816], compared with 0.5520.552 [0.339,0.7510.339,0.751] for histology tokens and 0.5540.554 [0.403,0.7180.403,0.718] for histology MLP. This result followed expansion of TCGA training from 210 to 501 patients, which increased external DSS C-index from 0.5410.541 to 0.6290.629. Exploratory OS C-index was 0.6740.674 [0.541,0.7890.541,0.789].

From patient ranking to image evidence

Fusion re-ranked two representative cases (Figure 2A): a grade-4 patient who died at 0.3 years rose from the 49th (histology-only) and 82nd (CT-only) percentiles to the 86th, and a grade-3 patient event-free at 5.6 years fell from the 27th and 38th to the 15th. Fused risk rose from −0.47-0.47 to +1.92+1.92 through histology (+1.58+1.58) and CT (+0.81+0.81) in the high-risk case, while low-risk contributions were near zero (−0.05-0.05, −0.01-0.01; Figure 2B). Unimodal models assigned larger CT (+2.33+2.33) and low-risk histology (−0.93-0.93) contributions (Figure 3), suggesting fusion redistributes evidence across modalities. Histology provided a median 77% (IQR 62–87%) of absolute cohort IG; CT contributions concentrated on the tumor side, and histology maps exposed potential pen/scanner shortcuts.

Out-of-fold histology/CT risk percentiles, in-sample IG decomposition, and prototype scores

Figure 2. (A) Out-of-fold histology/CT risk percentiles, colored by fusion percentile; triangles denote DSS deaths. (B) In-sample IG decomposition into reference and modality contributions. (C) Prototype scores: red raises risk and blue lowers it.

CT token integrated gradients and histology prototype projections and patches

Figure 3. (A) CT token IG at the largest tumor cross-section, with shared scaling and kidney/tumor/cyst contours. (B) Prototype projections and patches with slide-specific scales. The blue ring is ink. Attributions are in-sample and percentiles out-of-fold. Small residuals reflect rounding and numerical integration.

Discussion

Preserving within-modality structure while enabling cross-modal interaction yielded the highest mean DSS performance among evaluated designs. Although weaker alone, CT improved fusion performance beyond histology, highlighting its complementary value. Our approach integrates CT spatial information with histology prototype appearance and variability, extending cross-scale fusion [1, 2, 3, 4]. PANTHER Top + linear achieved 0.741 on TCGA, while All + MLP achieved 0.691 on CPTAC, exceeding our CT-hidden external mean of 0.629 as seen from Table 1. However, differences in cohorts, encoders, splits, and uncertainty estimates prevent direct superiority claims. Limited DSS events (46 internally and 15 externally) and wide intervals constrain inference. Differences in representations and training warrant matched cross-attention ablations. Screening-informed selection and a shared unlabeled dictionary may introduce optimism, while strong hospital signals in histology embeddings motivate site-heldout evaluation. Pen and scanner artifacts remained accessible to the model despite display filtering. Automatic masks and in-sample, baseline-dependent attributions further limit biological interpretation. External validation lacked CT, while calibration and incremental value beyond stage-inclusive clinical models remain untested. Future work should prioritize independent paired-modality validation, artifact removal, fold-specific prototypes, and matched ablations.

MethodTCGA-KIRCCPTAC-CCRCC
ABMIL0.684±0.060.684 \pm0.060.613±0.060.613 \pm0.06
TransMIL0.671±0.100.671 \pm0.100.639±0.040.639 \pm0.04
DSMIL0.500±0.000.500 \pm0.000.500±0.000.500 \pm0.00
AttnMISL0.649±0.090.649 \pm0.090.608±0.060.608 \pm0.06
ILRA0.637±0.140.637 \pm0.140.611±0.030.611 \pm0.03
DeepSets0.715±0.080.715 \pm0.080.634±0.010.634 \pm0.01
ProtoCounts0.461±0.060.461 \pm0.060.503±0.090.503 \pm0.09
H2T0.703±0.110.703 \pm0.110.631±0.040.631 \pm0.04
OT0.695±0.090.695 \pm0.090.650±0.020.650 \pm0.02
PANTHER Top + lin.0.741±0.110.741 \pm0.110.608±0.010.608 \pm0.01
PANTHER Bot. + lin.0.524±0.140.524 \pm0.140.532±0.080.532 \pm0.08
PANTHER WA + lin.0.730±0.070.730 \pm0.070.623±0.010.623 \pm0.01
PANTHER All + lin.0.703±0.130.703 \pm0.130.649±0.040.649 \pm0.04
PANTHER All + MLP0.716±0.100.716 \pm0.100.691±0.030.691 \pm0.03
Ours (fusion)0.760±0.0810.760 \pm0.0810.6290.629{sup}†`

Table 1. DSS concordance comparison. Published histology-only results are reproduced from PANTHER [1] (mean ±\pm SD). Bold numbers indicate the highest reported mean in each column. Published results use UNI and different TCGA cohorts and splits, making comparisons contextual.

Conclusion

Cross-attention fusion achieved the highest average DSS performance among evaluated designs. Attribution revealed complementary image contributions and artifacts, motivating independent multimodal validation.

Compliance with Ethical Standards

This study used public, de-identified data, requiring no additional ethical approval.

References

  1. [1]A. H. Song et al., “Morphological prototyping for unsupervised slide representation learning in computational pathology,” in CVPR, 2024, pp. 11566–11578.arxiv.org/abs/2405.11643
  2. [2]A. H. Song et al., “Multimodal prototyping for cancer survival prediction,” in ICML, PMLR 235, 2024, pp. 46050–46073.arxiv.org/abs/2407.00224
  3. [3]S. Gao, et al., “Probabilistic integration of renal cancer radiology and pathology using graph neural networks,” in MICCAI, 2025, pp. 564–573.DOI
  4. [4]D. Boeke et al., “Integrating pathology and CT imaging for personalized recurrence risk prediction in renal cancer,” arXiv:2508.21581, 2025.arxiv.org/abs/2508.21581
  5. [5]J. Liu et al., “An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics,” Cell, vol. 173, no. 2, pp. 400–416.e11, 2018.
  6. [6]Clark, David J., et al. ”Integrated proteogenomic characterization of clear cell renal cell carcinoma.” Cell 179.4 (2019): 964-983.
  7. [7]Chen, Richard J., et al. ”Towards a general-purpose foundation model for computational pathology.” Nature medicine 30.3 (2024): 850-862.DOI
  8. [8]S. Pai et al., “Vision foundation models for computed tomography,” arXiv:2501.09001, 2025.arxiv.org/abs/2501.09001
  9. [9]M. Ilse, et al., “Attention-based deep multiple instance learning,” in ICML, 2018, pp. 2127–2136.
  10. [10]M. Sundararajan, et al., “Axiomatic attribution for deep networks,” in ICML, 2017, pp. 3319–3328.arxiv.org/abs/1703.01365

Paper details

vlnc:W_s3m3yy3u
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents