WorksEngineering
When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis
Abstract
Histopathology and computed tomography (CT) provide complementary views of renal cancer. We connect these modalities through bidirectional cross-attention between 16 histology prototypes and up to 32 kidney-centered CT tokens from frozen foundation models. In 210 TCGA-KIRC patients with 46 disease-specific survival (DSS) events, 5-fold evaluation yielded a C-index of and 3-year AUROC of 0.773 (95% CI 0.678–0.860), the highest averages among evaluated unimodal and fusion baselines. External testing in 180 CPTAC-CCRCC patients yielded DSS C-index of 0.629 with CT withheld. Attribution analysis linked predictions to image regions while exposing acquisition artifacts. These findings support further study of cross-scale fusion and independent multimodal validation.
Introduction
Renal cell carcinoma exhibits heterogeneity across microscopic tissue architecture and macroscopic tumor anatomy, motivating prognostic models that integrate information across imaging scales [3]. Histopathology captures cellular morphology and tissue organization, while computed tomography provides a volumetric view of the tumor and surrounding anatomy. Combining these complementary perspectives offers an opportunity to characterize disease aggressiveness more comprehensively than either modality alone.
Foundation models provide transferable pathology and CT representations that make such integration increasingly feasible [7, 8]. However, effective multimodal learning requires more than combining pretrained features. Global pooling can obscure tissue heterogeneity and regional anatomical information, while retaining thousands of image features complicates learning in small survival cohorts. Compact representations that preserve meaningful within-modality variation could provide a practical balance between expressivity and data efficiency. PANTHER addresses this challenge in histopathology by summarizing whole-slide images through morphological prototypes that encode tissue prevalence, appearance, and variability [1]. Multimodal prototyping extends this principle to histology and transcriptomics for survival prediction [2]. Renal imaging studies have also explored probabilistic graph integration of radiology and pathology and feature fusion for recurrence prediction [3, 4]. These approaches motivate further investigation of how compact tissue representations can interact with spatially organized CT information.
We investigate bidirectional cross-attention between histology prototypes and kidney-centered CT tokens for disease-specific survival prediction as illustrated in Figure 1. The model retains prototype-specific tissue statistics and regional CT features, allowing each modality to condition the other before patient-level aggregation. We compare this formulation with unimodal, early-fusion, and late-fusion models on a shared TCGA-KIRC cohort and examine external histology-pathway transfer on CPTAC-CCRCC with CT withheld. Attribution analysis further connects predictions to modality contributions and image regions, supporting examination of complementary information and potential acquisition artifacts.

Figure 1. Overall architecture diagram
Methods
Cohorts and endpoint
The primary cohort included 210 TCGA-KIRC patients with histopathology images, CT scans, and disease-specific survival (DSS) labels from TCGA-CDR [5]. The cohort included 46 DSS events with median follow-up of 3.7 years. Patients with missing or invalid outcomes or an empty mask were excluded, as were frozen sections. Imaging modes used no clinical covariates. For external evaluation, we used PANTHER’s CPTAC-CCRCC test list comprising 180 patients and 341 tumor slides, with 15 DSS events, 26 overall-survival (OS) events, and median follow-up of 2.5 years [6]. Since only one DSS event occurred among the 52 patients with CT, external testing withheld CT. Adding 291 histology-only TCGA patients expanded training for this experiment to 501 patients with 109 DSS events. Slides were pooled per patient for external training and testing.
Histology prototypes and anatomical CT tokens
Frozen UNI-2h [7] encoded 256 × 256-pixel tissue patches at 20× into 1,536-dimensional embeddings, yielding approximately 6,300 patches per slide. Following PANTHER [1], we initialized 16 shared prototypes using -means on 100,000 TCGA patches and fitted a patient-specific diagonal Gaussian mixture with one expectation–maximization iteration. Each prototype produced
encoding prevalence, mean appearance, and variability. Shared initialization established corresponding prototype slots across patients. The label-free dictionary was shared across TCGA folds and fixed for CPTAC.
For each patient, we selected one axial CT series from the earliest eligible study, prioritizing nephrographic or portal venous contrast phases. Eligible series contained slices with spacing mm and coverage mm. Volumes were oriented superior–posterior–left, resampled to mm, and scaled from HU to . BAMF masks defined a kidney-region box with a 30 mm margin around retained components of ml within a 250 mm superior–inferior span. Tumor labels were not supplied as predictive features. Frozen CT-FM [8] encoded the padded box in one pass. Adaptive pooling of its final 512-channel feature map onto a grid produced tokens , with centers expressed relative to the kidney-mask centroid in 100 mm units. All feature standardization used training-fold statistics.
Bidirectional fusion and survival learning
We projected both token sets to , adding learned prototype embeddings for histology and linearly embedded three-dimensional positions for CT. Each stream passed through one pre-normalized self-attention layer with four heads, feed-forward width 128, GELU activation, and dropout 0.25. Bidirectional cross-attention then updated the streams according to
where , LN denotes layer normalization, and MHA denotes multi-head attention. The pairs represent the two directions between histology () and CT (), with separate directional weights and CT padding masks. Following residual feed-forward updates, mean-pooled histology and gated-attention-pooled CT [9] were concatenated into a 128-dimensional representation and linearly mapped to risk .
Training minimized the event-averaged negative Cox partial log-likelihood with Breslow handling of ties,
where contains patients with events at distinct time , is the observed follow-up time, and is the total event count. Training-only modality dropout withheld CT or histology with probability 0.15, disabling that modality’s contribution to the other stream and zeroing its pooled representation. CT was withheld when unavailable, and both image encoders remained frozen.
Comparators and evaluation
We compared fusion against unimodal token models with linear heads and a histology MLP using log mixture weights and their weighted mean. Early fusion concatenated these histology features with pooled CT (2,064-D), followed by ridge Cox regression or a 32-unit MLP. Ridge penalties were tuned over using inner five-fold CV. Late fusion averaged standardized risks from independently trained token models.
All models used identical event-stratified five-fold splits. Neural models trained for 40 full-batch epochs across three seeds with AdamW and gradient clipping at 1. Learning rates and weight decay were for token models and for MLPs. Per-seed risks were standardized against a separate five-repeat, five-fold protocol.
We reported fold-wise Harrell’s concordance (C-index) and three-year cumulative/dynamic AUROC and sensitivities at 80% and 90% specificity from pooled out-of-fold risks with inverse-probability-of-censoring weighting. Three-year events were cases, follow-up beyond three years defined controls, and earlier censorings were excluded (31 cases, 126 controls). Intervals used 1,000 patient-level bootstrap resamples. Thresholds derived from evaluation controls yielded descriptive ROC operating points.
Attribution analysis
Integrated gradients (IG) [10] explained the standardized three-seed risk from a model fitted to all 210 patients. The baseline replaced token content with cohort averages while retaining patient-specific CT positions and token count. A 64-step right-Riemann approximation yielded modality-, prototype-, and CT-token attributions with median absolute completeness error of 0.0026. To visualize histology contributions, prototype scores were distributed over patches using Gaussian-mixture posteriors and spatially aggregated. The resulting maps represent
prototype projections rather than patch-level IG. Attributions were in-sample, whereas risk percentiles were out-of-fold.
Results
Fusion performance on a common cohort
On the common cohort, cross-attention fusion achieved the highest mean C-index (), three-year AUROC (), and sensitivity at 80% specificity (), alongside the lowest C-index SD (Table 2). At 90% specificity, sensitivities were similar across nonlinear fusion designs (–). Although fusion increased mean concordance over histology MLP, histology tokens, and CT by , , and approximately , the corresponding paired intervals of [], [], and [] included zero and did not establish superiority.
| Model | C-index | 3-year AUROC [95% CI] | Sens., 80% spec. | Sens., 90% spec. |
| CT tokens + positions | [, ] | |||
| Histology tokens | [, ] | |||
| Histology summary + MLP | [, ] | |||
| Early fusion, ridge Cox | [, ] | |||
| Early fusion, MLP | [, ] | |||
| Late fusion, equal risks | [, ] | |||
| Cross-attention fusion (F2) | [, ] |
Table 2. TCGA-KIRC DSS prediction (). C-index is the 5-fold mean SD. 3-year metrics use censoring-adjusted, pooled out-of-fold predictions. Brackets indicate 95% CIs and bold indicates highest means.
External histology-pathway evaluation
With CT withheld, the fusion model yielded external DSS C-index of [], compared with [] for histology tokens and [] for histology MLP. This result followed expansion of TCGA training from 210 to 501 patients, which increased external DSS C-index from to . Exploratory OS C-index was [].
From patient ranking to image evidence
Fusion re-ranked two representative cases (Figure 2A): a grade-4 patient who died at 0.3 years rose from the 49th (histology-only) and 82nd (CT-only) percentiles to the 86th, and a grade-3 patient event-free at 5.6 years fell from the 27th and 38th to the 15th. Fused risk rose from to through histology () and CT () in the high-risk case, while low-risk contributions were near zero (, ; Figure 2B). Unimodal models assigned larger CT () and low-risk histology () contributions (Figure 3), suggesting fusion redistributes evidence across modalities. Histology provided a median 77% (IQR 62–87%) of absolute cohort IG; CT contributions concentrated on the tumor side, and histology maps exposed potential pen/scanner shortcuts.

Figure 2. (A) Out-of-fold histology/CT risk percentiles, colored by fusion percentile; triangles denote DSS deaths. (B) In-sample IG decomposition into reference and modality contributions. (C) Prototype scores: red raises risk and blue lowers it.

Figure 3. (A) CT token IG at the largest tumor cross-section, with shared scaling and kidney/tumor/cyst contours. (B) Prototype projections and patches with slide-specific scales. The blue ring is ink. Attributions are in-sample and percentiles out-of-fold. Small residuals reflect rounding and numerical integration.
Discussion
Preserving within-modality structure while enabling cross-modal interaction yielded the highest mean DSS performance among evaluated designs. Although weaker alone, CT improved fusion performance beyond histology, highlighting its complementary value. Our approach integrates CT spatial information with histology prototype appearance and variability, extending cross-scale fusion [1, 2, 3, 4]. PANTHER Top + linear achieved 0.741 on TCGA, while All + MLP achieved 0.691 on CPTAC, exceeding our CT-hidden external mean of 0.629 as seen from Table 1. However, differences in cohorts, encoders, splits, and uncertainty estimates prevent direct superiority claims. Limited DSS events (46 internally and 15 externally) and wide intervals constrain inference. Differences in representations and training warrant matched cross-attention ablations. Screening-informed selection and a shared unlabeled dictionary may introduce optimism, while strong hospital signals in histology embeddings motivate site-heldout evaluation. Pen and scanner artifacts remained accessible to the model despite display filtering. Automatic masks and in-sample, baseline-dependent attributions further limit biological interpretation. External validation lacked CT, while calibration and incremental value beyond stage-inclusive clinical models remain untested. Future work should prioritize independent paired-modality validation, artifact removal, fold-specific prototypes, and matched ablations.
| Method | TCGA-KIRC | CPTAC-CCRCC |
| ABMIL | ||
| TransMIL | ||
| DSMIL | ||
| AttnMISL | ||
| ILRA | ||
| DeepSets | ||
| ProtoCounts | ||
| H2T | ||
| OT | ||
| PANTHER Top + lin. | ||
| PANTHER Bot. + lin. | ||
| PANTHER WA + lin. | ||
| PANTHER All + lin. | ||
| PANTHER All + MLP | ||
| Ours (fusion) | {sup}†` |
Table 1. DSS concordance comparison. Published histology-only results are reproduced from PANTHER [1] (mean SD). Bold numbers indicate the highest reported mean in each column. Published results use UNI and different TCGA cohorts and splits, making comparisons contextual.
Conclusion
Cross-attention fusion achieved the highest average DSS performance among evaluated designs. Attribution revealed complementary image contributions and artifacts, motivating independent multimodal validation.
Compliance with Ethical Standards
This study used public, de-identified data, requiring no additional ethical approval.
References
- [1]A. H. Song et al., “Morphological prototyping for unsupervised slide representation learning in computational pathology,” in CVPR, 2024, pp. 11566–11578.arxiv.org/abs/2405.11643
- [2]A. H. Song et al., “Multimodal prototyping for cancer survival prediction,” in ICML, PMLR 235, 2024, pp. 46050–46073.arxiv.org/abs/2407.00224
- [3]S. Gao, et al., “Probabilistic integration of renal cancer radiology and pathology using graph neural networks,” in MICCAI, 2025, pp. 564–573.DOI
- [4]D. Boeke et al., “Integrating pathology and CT imaging for personalized recurrence risk prediction in renal cancer,” arXiv:2508.21581, 2025.arxiv.org/abs/2508.21581
- [5]J. Liu et al., “An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics,” Cell, vol. 173, no. 2, pp. 400–416.e11, 2018.
- [6]Clark, David J., et al. ”Integrated proteogenomic characterization of clear cell renal cell carcinoma.” Cell 179.4 (2019): 964-983.
- [7]Chen, Richard J., et al. ”Towards a general-purpose foundation model for computational pathology.” Nature medicine 30.3 (2024): 850-862.DOI
- [8]S. Pai et al., “Vision foundation models for computed tomography,” arXiv:2501.09001, 2025.arxiv.org/abs/2501.09001
- [9]M. Ilse, et al., “Attention-based deep multiple instance learning,” in ICML, 2018, pp. 2127–2136.
- [10]M. Sundararajan, et al., “Axiomatic attribution for deep networks,” in ICML, 2017, pp. 3319–3328.arxiv.org/abs/1703.01365