betaFeedback

WorksComputer Science

XGLASS: X-Ray, Gaze and Language Assisted Slice Sequencing for CT Volumes

Abstract

Vision-language models (VLMs) achieve remarkable performance on 2D medical images, such as X-rays or pathology images, while 3D VLMs that directly process CT/MRI volumes lag behind due to limited 3D training data and the predominance of 2D foundation architectures. To leverage powerful 2D VLMs for analyzing medical volumes, which contain hundreds of slices, it is therefore necessary to focus only on those few slices, which are diagnostically relevant. To enable such a prioritization, we propose XGLASS, a method that transfers medical expertise from 2D X-rays to 3D CT for task-aware slice prioritization. XGLASS uses disease-conditioned importance maps derived from language-aligned radiologist gaze on X-rays and spatially aligns them with coronal CT projections, bringing gaze priors and CT content into a shared 2D representation. The fused gaze-prior/CT projection is then used to predict a disease-specific ordering of axial slices, prioritizing diagnostically relevant slices. XGLASS significantly improves budgeted retrieval over 2D and 3D-native baselines, achieving a top-10 hit rate up to 75.9% for the best-performing disease. In practice, this means that after inspecting only the top 10 slices (4.17% of a 240-slice scan), XGLASS identifies a diagnostically relevant slice in more than 75% of cases. For downstream slice-wise finding classification, an XGLASS-selected 30-slice candidate set reduces model calls by 87.5% while improving F1 by 11.3 percentage points over all-slice inference, 4.2 points over the strongest 2D selector, and remains competitive with 3D-native selectors while requiring up to 130.8 times less time and 10.9 times less GPU memory per finding, supporting its use as a lightweight preselector for volumetric CT analysis. The code and pretrained checkpoint are available on https://github.com/michaelgloeckler/XGLASS.

Introduction

Vision-language models (VLMs) have recently enabled remarkable progress in computer vision, in tasks such as zero-/few-shot recognition, retrieval, and visual question answering (VQA), largely driven by the abundance of real-world image-text pairs [1, 2, 8, 33, 37, 38]. Analogously, the release of large paired medical datasets [7, 17, 18] has catalyzed medical adaptations, leading to CLIP-style medical contrastive models [25, 42, 46, 48] and medical VLMs [24, 31, 35] capable of report generation, diagnosis, and medical VQA. Alongside these models, a growing ecosystem of 2D medical datasets has emerged, such as closed- and open-ended VQA benchmarks targeting medical questions [15, 23, 26, 47]. However, the majority of these resources contains 2D medical images, such as x-ray or pathology slides. When CT or MRI appear in those datasets, they are typically represented by single, diagnostically relevant slices that were manually extracted by experts, rather than complete 3D volumes [15, 23, 26]. This results in a mismatch between dataset availability and diagnostic requirements, since core diagnostic modalities, such as CT or MRI, are inherently 3D. Applying 2D VLMs directly to complete CT volumes would require processing every axial slice independently, which is computationally expensive and often inefficient because most slices are redundant or irrelevant to a queried finding. In practice, downstream applications such as slice-wise VLM screening, computer-aided detection (CAD), and 2D dataset curation benefit from compact disease-conditioned candidate sets rather than exhaustive all-slice inference [41]. Motivated by this gap between predominantly 2D VLMs and the 3D nature of clinical imaging, we propose X-Ray, Gaze and Language Assisted Slice Sequencing (XGLASS), a lightweight method that transfers radiologist gaze-language priors learned on chest X-rays (CXRs) to rank axial CT slices by diagnostic relevance in a disease-conditioned manner. XGLASS effectively regresses a continuous, 240-dimensional relevance profile over axial slices, enabling relevance ranking, top-kk slice extraction, and importance-based traversal for retrieval or VLM conditioning. Empirically, XGLASS improves budgeted slice retrieval over 2D and 3D-native baselines, reaching 75.9% Hit @10 for the best-performing disease, while downstream VLM screening shows that XGLASS-selected candidate sets reduce model calls by 87.5% and improve F1 by up to 11.3 points over all-slice inference. To this end, we make the following contributions:

  • We introduce disease-conditioned importance maps from medical expert knowledge collected through x-ray eye-tracking data and synchronized speech recordings.

  • We propose XGLASS, a lightweight 2D-to-3D cross-modal model that transfers these priors to CT projections and predicts disease-specific axial slice relevance scores.

  • We evaluate XGLASS against 2D and 3D-native baselines, showing significantly better budgeted retrieval, substantially lower inference cost, and downstream F1 gains while reducing model calls by 87.5%.

Related Work

Eye-tracking in medical imaging. Radiologist eye-tracking provides implicit supervision of diagnostic search behavior and has been used to study visual expertise, diagnostic performance, and attention allocation [12, 20, 22, 39]. Recent CXR datasets such as REFLACX and MIMIC-Eye link gaze with report transcripts [5, 16, 19], enabling gaze-guided localization, attention steering, and multimodal alignment [4, 21, 28]. In contrast, eye-tracking data for 3D CT remain scarce: CT-Scangaze [32] provides 3D scanpaths with dictation transcripts, but its limited number of only 57 distinct scanpath–report pairs makes it unsuitable for deriving stable disease-specific importance maps.

Slice selection for 3D medical imaging. CT and MRI interpretation requires scrolling through volumetric slices, making slice selection useful for reducing redundancy and enabling 2D models to process 3D data. Vote-MI [41] selects representative slices from 2D features, but remains unsupervised, not disease-conditioned, and requires encoding all slices before selection, increasing computational cost. Although CT reading differs from CXR interpretation through slice drilling and z-axis-ordered search [12], our CXR–CT gaze-prior comparison in Supplementary Sec. B.4 shows correlated superior–inferior disease-attention profiles across modalities (r=0.65r=0.65–0.960.96), supporting XGLASS’s transfer of 2D gaze priors for CT axial slice sequencing.

Method

Based on x-ray, gaze, and language input, XGLASS learns f(P,G)→y^f(P, G) \to\hat{y}, which processes a coronal CT intensity projection PP, enriched with a disease-specific gaze prior GG, in the form of an importance map. For clarity, the overall pipeline is illustrated in Figure 1. Based on PP and GG, ff predicts a continuous importance vector y^\hat{y} of length ZtargetZ_{\mathrm{target}}. Each entry of this vector represents the diagnostic relevance of a single axial slice with respect to the disease conditioned through the importance map. By learning f(P,G)f(P, G) with a ViT-based regression model, we enable effective disease-specific sequencing of the CT volume, from which PP is obtained. In the following subsections, we first describe how to generate GG, then how to project a CT volume to obtain PP, and finally how we learn ff.

Figure 1. Overview of the XGLASS Pipeline. Given a chest CT and chest X-ray eye-tracking data with matching report transcripts as input, we first normalize CT volumes and compute coronal CT intensity projections, while also matching gaze and disease mentions to derive disease-conditioned importance maps. These importance maps are spatially aligned to the CT projection and fused via alpha blending. The gaze-enriched intensity projection is input to the ViT-based regression head trained with MSE to predict a slice-importance vector. For inference, the model uses the precomputed importance map for the desired disease fused with the CT intensity projection and outputs a disease-specific relevance ordering of axial slices, ranking diagnostically important slices at the top.

Disease-Conditioned Importance Maps

A central component of XGLASS is the construction of disease-specific 2D importance maps GG that capture how expert radiologists navigate through chest X-rays. To obtain these priors, we leverage the eye-tracking data paired with dictated report texts, available in the REFLACX dataset [5]. For disease filtering for each report transcript, we use custom regular expressions similar to SARLE [11] to identify disease-specific mentions in the spoken text. The timestamps associated with these textual matches define a temporal window during which the radiologist was verbally describing the relevant findings. Prior work in psycholinguistics shows that speakers’ gaze reliably anticipates verbal articulation by approximately one second, motivating the inclusion of a short pre-speech buffer on the order of 1–1.5 s [14]. Given the cognitive load associated with CT interpretation, we included a slightly longer pre-speech buffer of 2 s to ensure we capture all relevant fixation points. Formally, the gaze pooling process is described as follows, for a spoken mention with start time tstartt_{\mathrm{start}} and end time tendt_{\mathrm{end}}, we define the effective gaze window as [tstart−2 s,tend][t_{\mathrm{start}} - 2\ \mathrm{s}, t_{\mathrm{end}}]. All gaze fixations that fall into this window are then extracted, ensuring that only gaze from the relevant diagnostic moment contributes to the disease-conditioned importance map. To obtain a consistent spatial representation, we normalize each fixation with respect to the REFLACX chest bounding box Bchest=(x0,y0,w,h)B_{\mathrm{chest}} = (x_0, y_0, w, h) and discard fixations falling outside BchestB_{\mathrm{chest}} (potentially resulting from tracker drift or blinks). The box-normalized coordinates are

x~=x−x0w,y~=y−y0h\tilde{x} = \frac{x-x_0}{w}, \qquad\tilde{y} = \frac{y-y_0}{h}

so that (x~,y~)∈[0,1]2(\tilde{x},\tilde{y}) \in[0,1]^2. We then aggregate the normalized fixations across studies for the same disease category and smooth them by placing a Gaussian at each fixation to obtain a continuous density map [43]. Since REFLACX contains frontal CXRs acquired in both AP and PA views, we additionally analyze whether the resulting gaze priors are sensitive to view-specific acquisition geometry. In Sec. B.3 of the supplementary material, AP- and PA-specific gaze maps show only minor differences in their superior–inferior profiles; therefore, we aggregate both views to maximize the number of gaze samples available per disease. The resulting disease-conditioned importance maps capture the aggregated spatial attention pattern for each finding and serve as weak expert annotation priors that are subsequently fused with coronal CT intensity projections to create the final input for XGLASS.

CT Volume Projection

Our method takes two complementary sources of supervision: pre-computed, disease-specific 2D gaze importance maps derived from radiologist eye-tracking on chest X-rays using the REFLACX dataset [5], and a volumetric CT dataset that shares matching disease findings and therefore ReXGroundingCT [3] is to our knowledge the largest CT dataset that fulfills our requirements. To project a given CT volume into the disease-conditioned importance map domain, we first need to perform an intensity projection followed by a projection window normalization.

Intensity projection: CT volume projections enable us to project a given 3D volume into a 2D image, which in case of a coronal projection, is aligned with our disease-conditioned importance maps. Formally, let f(x)f(\mathbf{x}) denote the volumetric attenuation at location x∈R3\mathbf{x} \in\mathbb{R}^{3}. A projection along an imaging direction d\mathbf{d} samples parallel-beams x(u,v,t)=o(u,v)+td\mathbf{x}(u,v,t) = \mathbf{o}(u,v) + t\mathbf{d} and applies an aggregation operator A\mathcal{A} to the line signal f(x(u,v,t))f(\mathbf{x}(u,v,t)):

PA(u,v)=A({f(o(u,v)+td)}t∈[0,L]).(1)\mathcal{P}_{\mathcal{A}}(u,v) = \mathcal{A}\left(\left\{f\left(\mathbf{o}(u,v)+t\mathbf{d}\right)\right\}_{t\in[0,L]}\right). \tag*{(1)}

Because the coronal projection preserves the superior–inferior coordinate indexing axial slices, aggregating anterior–posterior depth retains the positional information required for axial slice selection. Accordingly, d\mathbf{d} follows the anterior–posterior axis, while different choices of A\mathcal{A} yield standard projection types. To simulate real-world x-ray projection ABL\mathcal{A}_{\mathrm{BL}} would resemble the Beer-Lambert law [36], which assumes that x-rays are only attenuated but not reflected at x:

ABL(u,v)=exp⁡(−κ∫0Lf(o(u,v)+td) dt),(2)\mathcal{A}_{\mathrm{BL}}(u,v) = \exp\left(-\kappa\int_{0}^{L}f\left(\mathbf{o}(u,v)+t\mathbf{d}\right)\,\mathrm{d}t\right), \tag*{(2)}

whereby κ\kappa is the constant extinction coefficient, which models absorption and out-scattering. An average-intensity projection (AIP) in contrast uses the line integral normalized by its length:

AAIP(u,v)=1L∫0Lf(o(u,v)+td) dt.(3)\mathcal{A}_{\mathrm{AIP}}(u,v) = \frac{1}{L}\int_{0}^{L}f\left(\mathbf{o}(u,v)+t\mathbf{d}\right)\,\mathrm{d}t. \tag*{(3)}

In the field of radiology, additionally, the maximum-intensity projection (MIP), defined as AMIP(u,v)=max⁡t∈[0,L]f(o(u,v)+td)\mathcal{A}_{\mathrm{MIP}}(u,v) = \max_{t\in[0,L]} f(\mathbf{o}(u,v)+t\mathbf{d}), plays an important role. However, MIPs emphasize very high-density structures (e.g. bones) and de-emphasize soft-tissue pathology, which conflicts with the intensity ranges in which our target findings occur. Therefore, throughout this paper, we solely focus on the Beer-Lambert projection and the AIP. We have ablated both (see § 4.4.2), whereby we compare three standard CT windowings using AIP, i.e., lung, soft-tissue, and a wide window.

Projection window normalization. To ensure the described intensity projections are aligned with the obtained disease-conditioned importance maps, anatomy normalization needs to be applied. During this normalization, our training set of 3,142 CT volumes was resampled to a uniform isotropic spacing of 1.25×1.25×1.251.25 \times1.25 \times1.25 mm. While this standardizes resolution, it does not reconcile variability in scan coverage along the ZZ-axis, which is critical since XGLASS 2D-to-3D matching expects a fixed amount of axial slices. To avoid the anatomical distortion introduced by global resizing (i.e., stretching or squashing), we adopt a fixed-size cropping strategy and define a target axial block size ZtargetZ_{\mathrm{target}} to be extracted from each volume. To further minimize exclusions, we sweep ZtargetZ_{\mathrm{target}} to find an optimal setting. A CT volume ViV_i is included only if two constraints hold: (1) its total axial slice count Ztotal,iZ_{\mathrm{total},i} (at 1.25 mm) is large enough to contain the block, and (2) its pre-computed chest bounding-box height Zbbox,iZ_{\mathrm{bbox},i} (at 1.25 mm) fits within the block.

Vi∈Cohortfinal⟺(Ztotal,i≥Ztarget)∧(Zbbox,i≤Ztarget)V_i \in\mathrm{Cohort}_{\mathrm{final}} \Longleftrightarrow(Z_{\mathrm{total},i} \ge Z_{\mathrm{target}}) \land(Z_{\mathrm{bbox},i} \le Z_{\mathrm{target}})

We tested integer values Ztarget∈[128,512]Z_{\mathrm{target}} \in[128,512] and found an optimal value of Ztarget=240Z_{\mathrm{target}} = 240 slices. This choice minimized exclusions by balancing chest-bbox inclusion (median Zbbox,i=205Z_{\mathrm{bbox},i} = 205) against total volume length (median Ztotal,i=269Z_{\mathrm{total},i} = 269). As a result, the final cohort comprises 2,078 unique CTs, with 1,064 excluded for not meeting the constraints. For all included CTs, we then extract a fixed axial block of 240 slices (1.25 mm spacing) centered on the chest bbox, yielding a standardized ReXGroundingCT subset with a consistent 240-slice axial size across scans. Further dataset statistics and category-selection details are provided in Sec. C of the supplementary material.

Ground-truth slice-area vector extraction. A key advantage of XGLASS is that it does not necessarily require full 3D segmentation masks: it can also be trained using only the start and end slices of a finding, as demonstrated by the SE-Binary and SE-Smooth ablations in Sec. 4.4.1. When segmentation masks are available, as in ReXGroundingCT, we use the more detailed Slice-Area target by deriving per-slice relevance scores from the 3D ground-truth masks. Therefore each mask is converted into a 240-dimensional disease-area vector over the fixed axial block. Let AiA_i denote the disease area on slice ii, with Ai=0A_i = 0 when the finding is absent. We compute

M=max⁡j=1,…,240Aj,M = \max_{j=1,\ldots,240} A_j,

and define the target vector y=(y1,…,y240)\mathbf{y} = (y_1,\ldots,y_{240}) by max-normalization:

yi={Ai/M,if M>0,0,if M=0,i=1,…,240.y_i = \begin{cases} A_i/M, & \text{if } M > 0,\\ 0, & \text{if } M = 0, \end{cases} \qquad i = 1,\ldots,240.

Thus, the slice with the largest disease extent receives yi=1y_i = 1, absent slices remain at 0, and intermediate slices receive values proportional to their disease area.

2D-3D Cross-Modal Slice Sequencing

Our model builds on a ViT-B/16 backbone with AugReg ImageNet-21k pretraining [10], which processes the fused representation of the disease-conditioned importance map GG and projected CT volume PP. To adapt the network from classification to slice-importance regression, we replace the original classification head with a lightweight two-layer MLP that outputs a fixed-length vector y^∈RZtarget\hat{\mathbf{y}} \in\mathbb{R}^{Z_{\mathrm{target}}} (Ztarget=240Z_{\mathrm{target}} = 240). A final sigmoid activation constrains all predictions to the range [0,1][0,1], matching the normalized ground-truth slice–area profiles. The model receives a 3-channel 224×224224 \times224 image initialized with a coronal CT intensity projection PP, replicated across all three input channels. The corresponding disease-specific importance map GG is then spatially scaled to the chest bounding-box region in the projection and blended into the red channel with an opacity coefficient α\alpha, while the green and blue channels remain unchanged. This channel-wise injection allows the ViT to process the anatomical projection and gaze-derived prior jointly in a single forward pass, enabling the model to estimate slice relevance directly from the fused representation. During training, the model obtains as input the fused GG and PP image, and learns to regress the ground-truth slice–area vector derived in Sec. 3.2. For each input image, the network outputs y^∈[0,1]240\hat{\mathbf{y}} \in[0,1]^{240} and is optimized using MSE loss between y^\hat{\mathbf{y}} and the corresponding normalized ground-truth vector y\mathbf{y}. We train the full model end-to-end using the AdamW optimizer with a differential learning rate schedule, assigning a higher learning rate to the regression head (1×10−41 \times10^{-4}) and a lower rate to the ViT backbone (1×10−51 \times10^{-5}). Training is performed with a batch size of 16. This setup preserves the pretrained backbone’s representational capacity while allowing the head to adapt to the slice-importance regression task.

Results

We evaluate XGLASS on a held-out test set (split details in Supp. Sec. C.2) to answer the following questions: (i) How does XGLASS perform compared to existing 2D and 3D slice selection baselines such as Vote-MI [41], BioMed-CLIP [46], Merlin [6], 3D U-Net [9] and heuristic approaches (ii) does injecting 2D gaze–language priors improve 3D slice sequencing, and (iii) does this translate into practically useful retrieval of diagnostically relevant slices?

Evaluation Metrics

We used different metrics to assess how well XGLASS orders all axial slices and how useful this ordering is for retrieval in potential downstream tasks. Let Z=240Z=240 be the number of axial slices, y∈[0,1]Z\mathbf{y} \in[0,1]^Z the ground-truth slice-area vector, and y^∈[0,1]Z\hat{\mathbf{y}} \in[0,1]^Z the predicted importance scores and a slice is positive if yi>0y_i > 0. All metrics are macro-averaged per study; AUC is computed over pooled slices. For overall ordering quality, we use ROC-AUC and Spearman’s rank correlation ρ\rho [34] between y\mathbf{y} and y^\hat{\mathbf{y}} over all ZZ slices, which directly evaluates agreement of the complete 240-slice ordering from most to least relevant. For practical analysis of performance on downstream tasks, we measure the hit rate at KK (Hit@KK) by sorting slices in descending y^\hat{\mathbf{y}} and checking whether any positive slice appears among the first KK predicted slices. Further, we also calculate the average recall at KK (AR@KK), indicating the fraction of all GT positive slices that are represented within the KK predictions.

In summary, higher ρ\rho and ROC-AUC indicate better global ranking alignment, higher Hit@KK shows that a GT slice is in the KK predictions, and AR@KK reflects the percentage amount of all GT slices inside KK predictions.

Baselines

We benchmark XGLASS against the state-of-the-art slice selection method Vote-MI [41], a BioMedCLIP-based slice-wise retrieval baseline [46], heuristic and prior-only baselines, two 3D-native baselines [6, 9], and a gaze-free variant of our model. Only the gaze-prior baseline and gaze-guided XGLASS use REFLACX gaze information; BioMedCLIP, 3D Merlin + MLP, 3D U-Net, and XGLASS are disease-conditioned, while XGLASS (w/out gaze), Vote-MI, Gaussian Centered, and Random are disease-agnostic. Gaze prior only ranks slices using only the disease-specific gaze-derived superior–inferior profile, without patient-specific CT input, serving as a control for static anatomical prior effects. Vote-MI was originally proposed for brain MRI; we follow its preprocessing and two-stage slice-selection procedure, but adapt it to thoracic CT by replacing the original feature extractor with MedVAE [40], a medical foundation model, and convert the selected representatives into continuous slice scores for fair metric-based comparison. BioMedCLIP is used as a slice-wise image–text retrieval baseline. For each CT-condition pair, we encode all axial slices with the BioMedCLIP image encoder and the queried disease text with the text encoder, and then rank slices by image–text cosine similarity to obtain a disease-conditioned 240-dimensional relevance score vector. Full prompt and implementation details are provided in Supplementary Sec. G.1. For the 3D-native model comparisons, 3D Merlin + MLP uses a frozen Merlin volumetric encoder [6] followed by a disease-conditioned MLP trained to regress the same 240-dimensional slice-relevance target as XGLASS, and 3D U-Net [9] uses a 3D U-Net encoder trained from scratch on ReXGroundingCT with a slice-regression head for the same targets. Further implementation details for both 3D baselines are provided in Supplementary Sec. G.2. XGLASS (w/o gaze cues) uses the identical backbone, projections, loss, and training as XGLASS but disables the gaze blend (α=0\alpha=0), isolating the architecture without disease-specific importance priors. Gaussian Centered places a Gaussian bump over slice indices with fractional width σfrac\sigma_{\mathrm{frac}}, yielding higher scores near the midthorax, and Random draws an independent score for each slice from Uniform[0, 1] to reflect chance performance. All methods output a 240-dimensional score vector per study and are evaluated with the same metrics (ROC-AUC, Spearman’s ρ\rho, Hit@K, and AR@K), as described in § 4.1.

Overall Performance

Table 1 shows that gaze-guided XGLASS outperforms the 2D baselines across all reported metrics, with paired bootstrap intervals indicating significant gains in Hit@5, Hit@10, AUC, and Spearman’s ρ\rho. The native 3D baselines are competitive on global ranking metrics, with 3D U-Net achieving the highest AUC and Spearman’s ρ\rho point estimates and 3D Merlin achieving the highest Hit@1 and AR@10 point estimates. Despite operating on the full CT volume, the 3D baselines do not translate their small non-significant advantages in Hit@1 and global ranking into reliable slice retrieval and XGLASS remains significantly better on Hit@5 and Hit@10, the most relevant metrics for prioritizing a compact inspection set. Figure 3 further shows that XGLASS reaches the first diagnostically relevant slice earlier and more consistently than all baselines. In the coverage analysis (Figure 2), the 3D baselines become competitive, but together with their later first-hit distributions in Figure 3, it suggests that the 3D models recover dense relevant regions once the correct location is found, but are less reliable at finding that location in every case. A case-level Miss@10 analysis supports this interpretation: Merlin and 3D U-Net miss the early inspection budget in 190/316 and 202/316 cases, respectively, compared with only 138/316 missed cases for XGLASS, indicating that their slightly higher Hit@1, AR@10, and global-ranking point estimates do not translate into more reliable compact-budget retrieval. For clinical use cases, reliably surfacing at least one diagnostic slice within the inspected budget is more valuable than dense coverage only after successful localization, making XGLASS more suitable for practical slice selection, especially since it also requires 34.8× less inference time and 6.5× less GPU memory than Merlin, and 130.8× less time and 10.9× less memory than U-Net.

MethodHit@1Hit@3Hit@5Hit@10AR@10AUCSpear. (ρ\rho)
Random0.101 [0.070, 0.136]0.250 [0.203, 0.298]0.342 [0.291, 0.396]0.491 [0.437, 0.547]0.043 [0.035, 0.051]0.504 [0.497, 0.511]0.003 [-0.004, 0.010]
Gaussian Centered0.190 [0.149, 0.234]0.237 [0.190, 0.285]0.278 [0.231, 0.329]0.323 [0.272, 0.377]0.082 [0.064, 0.101]0.718 [0.698, 0.738]0.208 [0.185, 0.233]
Gaze prior only0.158 [0.120, 0.199]0.177 [0.136, 0.222]0.209 [0.165, 0.253]0.263 [0.215, 0.313]0.068 [0.051, 0.088]0.638 [0.615, 0.661]0.163 [0.141, 0.186]
Vote-MI [41]0.044 [0.022, 0.070]0.190 [0.146, 0.234]0.339 [0.288, 0.392]0.465 [0.411, 0.519]0.033 [0.027, 0.038]0.472 [0.454, 0.491]-0.025 [-0.036, -0.015]
BioMedCLIP [46]0.212 [0.168, 0.256]0.282 [0.234, 0.332]0.339 [0.288, 0.392]0.443 [0.389, 0.500]0.070 [0.058, 0.083]0.658 [0.627, 0.688]0.164 [0.142, 0.187]
3D U-Net [9]0.231 [0.187, 0.278]0.275 [0.225, 0.326]0.301 [0.250, 0.351]0.361 [0.307, 0.415]0.089 [0.071, 0.107]0.785 [0.763, 0.806]0.228 [0.204, 0.253]
3D Merlin + MLP [6]0.247 [0.199, 0.294]0.297 [0.247, 0.348]0.320 [0.269, 0.370]0.399 [0.345, 0.453]0.094 [0.075, 0.114]0.779 [0.755, 0.800]0.224 [0.201, 0.249]
Ours (w/o gaze cues)0.199 [0.155, 0.244]0.288 [0.237, 0.339]0.342 [0.288, 0.396]0.468 [0.415, 0.525]0.077 [0.065, 0.091]0.768 [0.745, 0.791]0.212 [0.189, 0.235]
Ours (w/ gaze cues)0.228 [0.180, 0.275]0.339 [0.288, 0.392]0.421 [0.367, 0.475]0.563 [0.509, 0.617]0.091 [0.077, 0.106]0.781 [0.760, 0.801]0.224 [0.202, 0.246]
Δ\Delta vs. best 2D baseline+1.6%+5.7%+7.9%+7.3%+0.9%+6.3%+1.5%
best 3D baseline-1.9%+4.1%+10.1%+16.5%-0.3%-0.5%-0.5%

Table 1. Table 1. Main Results. Metrics are reported as point estimate with 95% confidence interval below. Intervals are computed using the nonparametric percentile bootstrap [13] over the test set with 10,000 resamples. The last rows report paired absolute differences in percentage points between XGLASS and the strongest 2D or 3D baseline for each metric; bold indicates statistical significance.

Figure 2. Average Ground-Truth (GT) Coverage. For each CT, we compute the fraction of all GT slices recovered within the top-KK predictions. XGLASS outperforms the 2D baselines and compete with the 3D baselines despite substantially less compute.

Figure 3. Slices until first Ground-Truth (GT). Boxplots show the distribution of slices required to reach the first GT slice (median, mean, 25–75% IQR, and 5–95% whiskers). XGLASS reaches a relevant slice earlier and with lower variability than the baselines.

Disease-specific Performance

The per-category trends in Table 2 align with typical anatomical distributions. Consolidation and Pleural abnormalities are often gravity-dependent and cluster in posterior lower lobes, producing a relatively narrow and predictable band along the zz-axis. This yields the highest AUC/ρ\rho and strong Hit@K performance. Ground-glass opacities are frequently diffuse and extend across adjacent slices, which lifts mid- to higher-KK retrieval (e.g., Hit@10) but softens global ranking metrics. In contrast, lung nodules/masses are small, sparse, and spatially unpredictable along the zz-axis, which reduces the chance that the very top-ranked slices contain a hit and depresses global rank agreement. Thus, lung nodules are relatively less suited to our approach due to their small size and variable location, characteristics that also make them challenging for slice-selection methods in general. When evaluated only on the lung-nodule subset, XGLASS achieves the strongest Hit@KK and AUC, showing that despite performing worse than on other diseases, it still provides the best slice prioritization (Supplementary Sec. D). Taken together, these findings indicate that XGLASS learns disease-specific zz-axis priors similar to radiologist behaviour, prioritizes anatomically informative regions, and therefore performs best for conditions with consistent spatial distributions such as consolidations. baselines, making local retraining on commodity GPUs feasible for institutions with private clinical data.

Category (N)Hit@1Hit@3Hit@5Hit@10AR@10AUCSpear. (ρ\rho)
Atelectasis (39)0.2050.3850.4360.5640.0750.7730.265
Consolidation (29)0.4830.5860.6550.7590.1140.8100.386
Groundglass opacity (95)0.2840.4000.5160.6740.1140.7280.272
High lung volume (29)0.1720.2410.2760.4140.0550.6570.161
Lung nodule or mass (112)0.1070.2050.2860.4460.0740.7220.129
Pleural abnormality (12)0.5000.5830.6670.6670.1390.7790.345
Overall (316)0.2280.3390.4210.5630.0910.7810.224

Table 2. Disease-Specific Results. The “Overall” row corresponds to the “Ours (w/ gaze cues)” model from Table 1. Counts (N) are shown in parentheses. We report hit-rate metrics Hit@K, the averaged recall at 10 (AR@10), AUC and rank correlation (Spearman’s ρ\rho).

Computational Efficiency

XGLASS contains approximately 86M parameters and is trained on a single 2D projection per CT study. On an NVIDIA RTX 4080, one training epoch over 2,512 projections with batch size 16 takes 62.2 s, corresponding to roughly 40 samples/s, with approximately 3 GB peak GPU memory. Early stopping on validation loss consistently selects checkpoints after 6–8 epochs, yielding a complete training run in under 10 minutes. At inference time, XGLASS processes a full CT study, including intensity projection, in approximately 0.025 s per finding (0.150 s for all findings) with an estimated cost of 33.7 GFLOPs and 0.34 GB memory. In comparison, Vote-MI requires 7.51 s and approximately 116.33 TFLOPs per CT, while slice-wise BioMedCLIP requires 1.40 s and approximately 8.09 TFLOPs per CT. The 3D Merlin and U-Net baselines require 0.87 s and 3.27 s per finding, respectively, with 2.21 GB and 3.72 GB peak inference memory, making them 34.8×/130.8×34.8 \times/ 130.8 \times slower and 6.5×/10.9×6.5 \times/ 10.9 \times more memory-intensive than XGLASS. Overall, XGLASS is orders of magnitude cheaper than slice-wise 2D baselines and substantially faster and more memory-efficient than the 3D

Downstream Slice-wise VLM Screening

We evaluate XGLASS as a disease-conditioned pre-filter for slice-wise VLM screening with 2D VLMs. For each CT-condition pair, XGLASS ranks all axial slices by relevance to the queried finding, and the downstream model is evaluated only on the Top-KK selected slices. Metrics are computed over the inspected slices and compared against applying the same model to all 240 slices. Prompts, detailed inference settings and a matched-budget comparison to other slice selection methods are provided in Supplementary Sec. G. As shown in Figure 4, smaller XGLASS budgets consistently improve precision across MedGemma 4B [35], Lingshu 7B [44], and BioMedCLIP [46], indicating that XGLASS determines disease-enriched slice subsets. This also improves F1 over inspected slices at small budgets: Top-30 improves F1 from 0.288 to 0.414 for MedGemma, from 0.327 to 0.441 for Lingshu, and from 0.235 to 0.316 for BioMedCLIP. Compared with other slice-selected subsets, XGLASS improves average Top-30 F1 over the strongest 2D selector from 0.338 to 0.390, while remaining competitive to the strongest 3D selector. Evaluating fewer slices also reduces downstream runtime, with Top-30 requiring only 12.5% of the slice-wise model calls compared with all-slice inference. This supports XGLASS as a fast pre-selection step that enriches disease-relevant slices before costly slice-wise VLM review, 2D dataset curation or similar applications.

Figure 4. Downstream slice-wise VLM screening under XGLASS budgets. For each CT-condition pair, XGLASS selects the Top-KK axial slices and the downstream model is evaluated only on this subset. Precision, recall, F1, and runtime are reported for MedGemma 4B, Lingshu 7B, and BioMedCLIP under the same single-GPU setting. Smaller XGLASS budgets consistently improve precision and F1, while reducing runtime by decreasing the number of inspected slices. Comparison to other slice selectors are in Supplementary Sec. G.4.

Ablation Studies

We perform targeted ablations to isolate the effect of the main XGLASS design choices. Hyperparameter choices were made using the validation split resulting in ViT-B/16 backbone, AIP-wide CT projection, gaze blending strength of α=0.75\alpha=0.75, and training using MSE. The corresponding validation results and selection criteria are reported in Supplementary Sec. E. All ablations tables modify only one factor at a time, and are reported on the held-out test set. Additional backbone, multi-channel-fusion, gaze-prior perturbations, loss ablation, and gaze prior-specificity ablations are provided in Supplementary Sec. F.

Weak Supervision.

Our best training target uses segmentation-derived slice-area scores, but XGLASS can also be trained from substantially weaker annotations. We evaluate three reduced-supervision targets: SE-Binary, which marks all slices between annotated start and end positions as positive; SE-Smooth, which additionally applies Gaussian smoothing to this interval; and Rel-Rank, which uses relative slice ranking instead of dense area scores. As shown in Table 3, reduced-supervision targets remain competitive, highlighting XGLASS’s ability to be trained from weak annotations.

TargetAUCSpear.Hit@1Hit@5Hit@10
SE-Binary0.7910.2230.1680.3640.490
SE-Smooth0.7920.2270.2200.4130.524
Rel-Rank0.7930.2120.1870.4170.524
Slice-Area0.7810.2240.2280.4210.563

Table 3. Supervision Target Ablation. Comparison of dense Slice-Area supervision with only start/end slice annotations.

Volume Projection Ablation

The results in Table 4 show a trade-off between global ranking and early retrieval utility. Beer–Lambert achieves the highest AUC (0.795) and Spearman’s ρ\rho (0.232), and it edges out others at Hit@5 (0.427), indicating strong overall separability and mid-KK retrieval. In contrast, AIP (wide) delivers the best utility-centric performance with the top Hit@1 (0.228), Hit@10 (0.563), and near-best Hit@5 (0.421), suggesting that preserving a broad intensity range helps rank diagnostically relevant slices.

ProjectionAUCSpear.Hit@1Hit@5Hit@10
AIP (lung)0.7840.2220.2060.3610.462
Beer-Lambert0.7950.2320.1960.4270.535
AIP (soft tissue)0.7830.2230.2280.3830.519
AIP (wide)0.7810.2240.2280.4210.563

Table 4. CT Projection Ablation. Comparison of Beer–Lambert and AIP projections with lung, soft-tissue, and wide CT window.

Gaze Fusion Ablation

As described in § 3.1, XGLASS fuses the disease-specific gaze prior with the coronal CT projection by alpha-blending the importance map into one RGB input channel. Table 5 evaluates both the blending strength and the injection channel. Adding gaze guidance consistently improves over the gaze-free setting (α=0\alpha=0), confirming that the disease-conditioned prior provides useful localization information. While α=1.0\alpha=1.0 achieves the highest global AUC and Spearman’s ρ\rho, but it partially suppresses CT intensity cues lowering retrieval metrics. In contrast, α=0.75\alpha=0.75 gives the best early retrieval performance, reaching Hit@10 of 0.563. Among single-channel injection variants, red-channel fusion performs best, outperforming green and blue injection on all metrics. Additional multi-RGB-channel fusion results are reported in the Supplementary Material in Sec. F.1.

Channelα\alphaAUCSpear.Hit@1Hit@5Hit@10
0.000.7680.2120.1990.3420.468
0.250.7870.2210.1870.3920.506
Red0.500.7850.2200.1960.4180.509
1.000.7970.2270.1990.4150.528
0.750.7810.2240.2280.4210.563
Green0.750.7540.2120.1930.3450.459
Blue0.750.7610.2090.1900.4180.519

Table 5. Gaze fusion ablation. Effect of blending opacity α\alpha and RGB injection channel for the disease-specific importance map.

Conclusion and Limitations

We presented XGLASS, a lightweight disease-conditioned model that transfers aggregated CXR gaze-language priors to CT for relevance-ordered axial slice retrieval. XGLASS significantly improves Hit@5 and Hit@10 over the strongest evaluated 2D and 3D baselines while remaining computationally efficient. Downstream experiments further show that XGLASS enriches compact disease-relevant candidate sets, improves downstream classification performance, and reduces slice-wise model calls. Beyond slice selection, XGLASS illustrates how heterogeneous 2D supervision can help bridge the gap between 2D foundation models and 3D medical image analysis. Nevertheless, several limitations remain. Despite surveying publicly available CT datasets, ReXGroundingCT is the only dataset that overlaps with the 14 REFLACX-derived disease priors, yielding only to one dataset and six shared thoracic findings for evaluation. Our downstream screening experiments provide an initial demonstration of XGLASS utility and not direct use cases; broader applications such as VQA, report generation, retrieval, and localization remain future work.

Acknowledgments

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – GRK 3012 – 520750254 (https://kemai.uni-ulm.de/).

Qualitative Results

We complement the quantitative metrics with a qualitative analysis of slice selection across six chest CT findings. Figure 6 visualizes the axial slice indices (zz) selected by a Random baseline, Vote-MI, and XGLASS. Across categories, XGLASS not only hits a ground-truth slice earlier, but also concentrates its selections within the annotated region, yielding denser and more contiguous coverage than competing methods. These examples illustrate that our model prioritizes clinically meaningful slices rather than scattered or isolated hits, aligning with the intended notion of region-level localization.

Figure 6. Qualitative Results. Slice retrieval across six chest CT findings: Atelectasis, Consolidation, Ground-glass Opacity, Lung Nodule/Mass, High Lung Volume, and Pleural Abnormality. For each category, we show the axial slice indices (zz) predicted by a Random baseline, Vote-MI, and our XGLASS selector. XGLASS not only identifies a ground-truth slice (green border) earlier, but also provides broader coverage of the ground-truth region by retrieving multiple slices that fall within it, rather than a single correct slice.

Disease-Conditioned Importance Maps

Text–Gaze Alignment

To construct disease-conditioned 2D importance maps from REFLACX [5], we jointly process the word-level transcripts and synchronized gaze recordings. For each target disease, we first detect positive (non-negated) mentions in the transcript using label-specific regular expressions together with clause-level negation handling, as described in § 3.1. Each detected mention defines a speech window [tstart−τpre,tend][t_{\mathrm{start}}-\tau_{\mathrm{pre}},t_{\mathrm{end}}] with a short pre-speech buffer τpre\tau_{\mathrm{pre}} of 2 seconds to account for anticipatory gaze. We then collect all gaze samples whose timestamps fall inside this window and project their screen coordinates onto the corresponding chest X-ray. Following the REFLACX protocol, we normalize coordinates to the chest bounding box and discard samples that fall outside this region (e.g., blink artefacts or fixations off the image). Figure 5 shows qualitative examples of the resulting text–gaze alignment for the six disease categories used in this work.

Six-panel chest X-rays showing text–gaze alignment for disease phrases

Figure 5. Text–gaze alignment for disease phrases. Examples of text–gaze alignment used to construct disease-conditioned importance maps for the six target findings: Atelectasis, Consolidation, Groundglass opacity, Lung nodule or mass, High lung volume, and Pleural abnormality. Each panel shows a REFLACX chest X-ray with the chest bounding box (green dashed line), disease-specific annotation ellipses (black), and gaze trajectories for the words in the corresponding disease phrase (color-coded by word with the pre-speech buffer in orange). Fixations that fall outside the chest bounding box, visible for the High lung volume and Groundglass opacity examples, are removed during preprocessing as described in § 3.1.

Extracted Gaze Importance Maps

Using label-specific regular expressions and clause-level negation on the REFLACX reports, we defined 14 non–device pathology categories and, for each of them, extracted a disease-specific importance map by aggregating gaze samples associated with positive mentions across all matching sessions. Figure 7 summarizes how many gaze–report pairs contribute to each category.

Figure 7. REFLACX sessions per disease category. Number of REFLACX sessions contributing to each disease-conditioned gaze heatmap. Bars show the number of matching gaze–report pairs for each category defined in REFLACX. Categories highlighted in blue correspond to the six findings used in XGLASS, whereas orange bars denote additional categories for which we obtained gaze heatmaps but did not use them due to the lack of corresponding findings in the CT dataset.

AP–PA Robustness of CXR Gaze Priors

Because REFLACX is derived from frontal chest radiographs, one potential concern is that the resulting gaze priors may depend on acquisition view. AP and PA radiographs can differ in apparent anatomy due to projection geometry, magnification, and patient positioning. We therefore stratify REFLACX sessions by MIMIC-CXR view position and construct separate AP-only and PA-only disease-conditioned gaze maps. Figure 12 and Figure 13 compare the resulting maps for all six target findings. Despite view-dependent differences in the 2D heatmaps, the superior–inferior profiles remain similar across AP and PA views, with correlations ranging from r=0.53r=0.53 to 0.960.96. Since this axis is the one used for axial slice ordering, these results support combining AP and PA sessions to obtain more stable disease-conditioned gaze priors for XGLASS.

Figure 12. AP–PA comparison of disease-conditioned gaze priors, part 1. For each finding, we construct separate AP-only and PA-only REFLACX gaze heatmaps and compare their superior–inferior profiles. The superior–inferior axis encodes the axial slices and therefore it is most relevant for XGLASS.

Figure 13. AP–PA comparison of disease-conditioned gaze priors, part 2. Continuation of the AP–PA comparison for the remaining findings. Similar superior–inferior profiles across views support combining AP and PA sessions into more stable disease-conditioned gaze priors.

CXR–CT Gaze Alignment

To further assess whether CXR-derived gaze priors capture transferable anatomical search patterns, we compare them with CT-ScanGaze [32], a CT eye-tracking dataset with timestamped dictation. CT-ScanGaze contains 57 unique CT cases, but several scans have inconsistent orientation metadata or cover non-chest anatomy. We therefore performed visual quality control and excluded cases with incorrect projection geometry or non-thoracic coverage. For the remaining cases, we constructed disease-conditioned CT gaze maps analogously to the REFLACX priors: report sentences were matched to gaze by timestamp, using the same 2 s pre-speech buffer, and fixations were aggregated for the corresponding finding. These CT-ScanGaze maps provide insight into CT gaze patterns but are substantially sparser than REFLACX, with 6–23 CT cases and 244–1,215 fixations per finding (mean: 12.8 cases and 731 fixations). Figure 14–Figure 15 compare the resulting CT-ScanGaze maps with the REFLACX CXR priors. The 2D heatmaps differ visibly, reflecting different reading behavior in CT: radiologists often fixate a local region and scroll through adjacent axial slices, rather than scanning broadly across the frontal image as in CXR. However, the superior–inferior gaze-density profiles remain aligned across modalities, with correlations ranging from 0.65 to 0.96 for the analyzed findings. This indicates that although CT and CXR differ in traversal order and fixation dynamics, radiologists still attend to similar disease-relevant anatomical regions. Thus, the CXR-derived gaze priors encode non-arbitrary, disease-specific search patterns that can be transferred to CT and used as additional input for axial slice selection.

Figure 14. CXR–CT comparison of disease-conditioned gaze priors, part 1. We compare REFLACX CXR-derived gaze priors with sentence-matched CT-ScanGaze fixation heatmaps. Although CT reading follows different fixation dynamics, the superior–inferior profiles show similar coarse disease-specific attention patterns, supporting the idea to transfer CXR gaze priors to CTs.

Figure 15. Figure 15. CXR–CT comparison of disease-conditioned gaze priors, part 2. Continuation of the CXR–CT comparison for the remaining findings. The aligned superior–inferior profiles support transferring CXR-derived disease-specific search priors to CT slice selection.

CT Dataset and Volume Projections

Dataset Selection

To evaluate how effectively our disease-specific eye-tracking priors transfer from CXR to CT, we require a CT dataset that (i) provides report text and findings aligned with individual volumes, and (ii) contains findings that can be reliably matched to our REFLACX-derived disease-specific importance maps. As discussed in § 3.2, ReXGroundingCT [3] is, to our knowledge, the only large-scale CT dataset that satisfies both criteria. Because this dataset will also be used to train XGLASS, each disease category must be supported by a sufficient number of CT studies. We therefore exclude any finding category with fewer than 100 matching CTs. As shown in Figure 7 and Figure 8, only six of the 14 REFLACX-defined categories reach this threshold in our standardized 240-slice

Figure 8. CT disease category distribution. Counts of CT–finding cases for the six ReXGroundingCT categories selected for XGLASS, stacked by train/validation/test split. These categories were retained because they overlap with the REFLACX-derived gaze priors and provide sufficient CT cases for training and evaluation.

ReXGroundingCT subset introduced in § 3.2, and we consequently restrict our experiments to these six categories.

Training Dataset and Splits

We use an 80/10/10 CT-volume-level train–validation–test split of the standardized 240-slice ReXGroundingCT subset, with no volume overlap between partitions and all findings from each CT retained in the same split. This yields 2,514/306/316 CT–finding cases for training/validation/testing, respectively, totaling 3,136 cases across six categories: Lung nodule or mass (1,096), Groundglass opacity (925), Atelectasis (412), High lung volume (301), Consolidation (284), and Pleural abnormality (118), as shown in Figure 8. The disease distribution in our ReXGroundingCT subset is markedly imbalanced across our six categories. However the best performing model did not apply class-balancing or reweighting (e.g.,

oversampling, inverse-frequency sampling, focal/weighted losses), because the most prevalent class, lung nodule or mass, is also the hardest to localize due to its small size, variable location, and sparse slice extent. This choice is supported by the disease-specific results in Table 2 of the main paper, where XGLASS improves across all disease categories, and by the loss-function ablation in Sec. F.5, where reweighted objectives do not improve small-finding and overall retrieval performance.

CT Volume Projections

For XGLASS, each 3D chest CT volume is converted into a 2D coronal intensity projection that lives in the same domain as our disease-conditioned importance maps, as explained in § 3.2. Formally, we project along the anterior–posterior axis and aggregate attenuation values along each ray using an operator AA (Beer–Lambert or average-intensity), yielding a coronal image PA(u,v)P_A(u,v) for each volume. In the main experiments we focus on two projection families: (i) a physically motivated Beer–Lambert projection that mimics x-ray transmission, and (ii) average-intensity projections (AIP) combined with standard CT windowings. For AIP we consider three clinical window settings: a lung window, a soft-tissue window, and a wide window spanning a broad Hounsfield range. As discussed in § 3.2 and ablated in § 4.4.2, Beer–Lambert provides slightly better global ranking (AUC, Spearman), whereas AIP with a wide window offers the best early retrieval utility (Hit@K). Figure 9 illustrates how a single CT scan appears under these different projection choices, highlighting how windowing and projection type emphasize complementary structures (e.g., parenchyma in the lung window vs. mediastinum and pleural surfaces under broader windows).

Figure 9. CT volume projections in XGLASS. Example coronal slice projections of the same chest CT under different projection methods and windowing settings. AIP (lung) emphasizes parenchymal findings and saturates mediastinal structures, AIP (soft) suppresses low-attenuation lung parenchyma, so bony structures dominate the appearance, AIP (wide) exposes a broader range of intensities, and the Beer–Lambert projection yields a more radiograph-like appearance with stronger contrast in high-attenuation structures.

Gaze-Prior Blending Strength

In XGLASS, the 2D CT projection and the disease-specific REFLACX gaze importance map are combined by injecting the normalized gaze heatmap Hd(u,v)H_d(u,v) into the red channel inside the chest bounding box using a blending coefficient α∈[0,1]\alpha\in[0,1]. Here, α=0\alpha=0 corresponds to the CT-only input, whereas larger values increase the contribution of the disease-conditioned gaze prior in the fused representation. Figure 11 illustrates the qualitative effect of varying α\alpha for representative CT studies. The quantitative test-set ablation of α\alpha is reported in § 4.4.3 of the main paper. The final value α=0.75\alpha=0.75 was selected on the validation split, as described in Sec. E.

A six-row by five-column grid of chest CT projections showing the effect of gaze–heatmap blending strength $\alpha$; rows are labeled Pleural Abnormality, Lung Nodules, High Lung Volume, Groundglass Opacity, Consolidation, and Atelectasis, and columns range from $\alpha = 0.00$ to $\alpha = 1.00$.

Figure 11. Effect of the gaze–heatmap blending strength α\alpha on the AIP-wide CT projections. Rows correspond to the six disease categories used in XGLASS (one representative CT per category), and columns sweep the blending coefficient from left to right: α∈{0.0,0.25,0.5,0.75,1.0}\alpha\in\{0.0, 0.25, 0.5, 0.75, 1.0\}. The leftmost column (α=0\alpha= 0) shows the raw CT projection without gaze information, while increasing α\alpha progressively injects the REFLACX-derived gaze importance map into the red channel within the chest bounding box.

Lung Nodule Baseline Comparison

Lung nodules/masses are the hardest evaluated category because they are small, sparse, and spatially variable. However, this relative weakness does not indicate near-random performance. Across all 112 nodule cases, XGLASS achieves the best Hit@K and AUC among all 2D and 3D baselines, showing that it more reliably retrieves at least one relevant slice early. The 3D U-Net obtains slightly higher AR@10 and Spearman’s ρ\rho, but performs substantially worse on Hit@5, Hit@10, and Hit@20 (Table 6).

MethodHit@1Hit@3Hit@5Hit@10Hit@20AR@10AUCSpearman (ρ\rho)
Random0.0800.1250.1790.2860.4460.0380.491-0.007
Vote-MI [41]0.0180.1070.1790.3130.5450.0350.478-0.017
BioMedCLIP [46]0.0800.1430.2140.2860.4550.0620.5850.093
3D Merlin + MLP [6]0.0800.1070.1430.2590.4200.0570.6880.125
3D U-Net [9]0.0800.1430.1610.2320.3840.0790.6980.132
XGLASS0.1070.2050.2860.4460.6070.0740.7220.129

Table 6. Lung-nodule-specific baseline comparison. Results on all 112 lung nodule/mass test cases, the most spatially sparse and variable finding category. XGLASS achieves the strongest Hit@KK and AUC compared to the other 2D and 3D baselines, indicating more reliable early retrieval despite the difficulty of this disease.

Hyperparameter Selection

We selected the model’s hyperparameters such as the visual backbone, CT projection, gaze-blending coefficient α\alpha, loss function, and gaze-injection channel using the validation split. All optimization hyperparameters were kept fixed across configurations: AdamW with learning rates of 1×10−41 \times10^{-4} for the regression head and 1×10−51 \times10^{-5} for the ViT backbone, and a batch size of 16. Since XGLASS targets slice retrieval across different inspection budgets, selection was based on the mean of Hit@5, Hit@10, and Hit@20, denoted as Hit‾\overline{\mathrm{Hit}} in Table 7. We exclude Hit@1 and Hit@3 from the selection criterion because these very small retrieval budgets are more sensitive to single-slice rank changes. When Hit‾\overline{\mathrm{Hit}} was tied, AUC and Spearman’s ρ\rho were considered as secondary ranking criteria. ViT-B/16, AIP-wide, and α=0.75\alpha=0.75 achieved the highest Hit‾\overline{\mathrm{Hit}} within their respective ablations. For the gaze-injection channel, red and blue achieve the same Hit‾\overline{\mathrm{Hit}}. Red yields a slightly higher Spearman correlation (0.250 vs. 0.243) while differing only marginally in AUC (0.796 vs. 0.801). Given the similar slice retrieval performance and the ranking objective of XGLASS, we therefore selected red-channel injection for the final model. In the loss-function ablation, unweighted MSE achieves a higher Hit‾\overline{\mathrm{Hit}} than binary focal loss and lesion-balanced MSE, while preserving competitive AUC and Spearman correlation; we therefore selected the unweighted MSE as the final training objective. The resulting configuration, ViT-B/16, AIP-wide, α=0.75\alpha=0.75, red-channel gaze injection, and unweighted MSE, was fixed and used for the test-set ablations in § 4.4 and Sec. F.

Table 7. Validation-based hyperparameter selection. All values are computed on the validation split. Hit denotes the mean of Hit@5, Hit@10, and Hit@20. Gray rows indicate the settings selected for the final model; bold values indicate the best performance within each ablation.

SettingHit@5Hit@10Hit@20HitAUCρ\rho
Backbone
ViT-B/160.4410.5590.7090.5700.7960.250
RadioDino-s80.4350.5390.6440.5390.8040.256
Swin-Tiny0.4050.4640.5780.4820.7980.244
CT projection
AIP-wide0.4410.5590.7090.5700.7960.250
AIP-soft0.3950.5720.6540.5400.7980.245
AIP-lung0.4310.5100.6180.5200.7940.242
Beer-Lambert0.4280.5490.6470.5410.7960.246
Gaze blending coefficient α\alpha
0.000.3950.5000.5780.4910.7710.236
0.250.4220.5490.6410.5370.8020.246
0.500.4120.5390.6930.5480.7970.248
0.750.4410.5590.7090.5700.7960.250
1.000.4280.5130.6570.5330.8030.248
Gaze-injection channel
R0.4410.5590.7090.5700.7960.250
B0.4510.5750.6830.5700.8010.243
R+B0.4120.5390.6830.5450.8040.248
G0.4150.5360.6500.5340.8060.243
G+B0.4220.5100.6500.5270.7980.241
R+G0.4350.5200.6240.5260.8000.248
Loss function
Area MSE0.4410.5590.7090.5700.7960.249
Binary focal0.4180.5820.6830.5610.8040.250
Slice-bal. MSE0.3890.5360.6900.5380.7260.229
Size-wt. MSE0.4050.5200.6700.5320.7900.235

Table 7.

Extended Ablations

Gaze-Injection Channel Combinations

We additionally evaluate every RGB-channel combination for injecting the gaze prior at the selected blending strength α=0.75\alpha=0.75 on the held-out test set. Table 8 shows that the two strongest configurations both include the red channel: magenta (R+BR+B) performs best for AUC, Spearman correlation, Hit@1, and Hit@5, while red-only achieves similar overall performance and the highest Hit@10 and Hit@20. Red-only was selected on the validation split, where it tied with blue for Hit and achieved the higher Spearman correlation. The test-set results further show that red-only remains particularly strong at the larger retrieval budgets.

Table 8. Channel Injection Strategy Ablation. Gaze-prior injection into individual RGB channels and their combinations.

ChannelsAUCSpearmanHit@1Hit@5Hit@10Hit@20
R0.7810.2240.2280.4210.5630.693
G0.7540.2120.1930.3450.4590.582
B0.7610.2090.1900.4180.5190.655
R+G0.7740.2200.1840.3830.5030.639
R+B0.7840.2260.2410.4430.5470.661
G+B0.7490.2120.2220.3670.4680.617
RGB0.7610.1960.1920.3640.4650.594

Table 8.

XGLASS Backbone

Different backbones encode different inductive biases (e.g., local vs. global receptive fields, domain pretraining). We test a ViT-B/16 backbone (patch size 16, input 224×224) with AugReg ImageNet-21k pretraining [10], a Swin transformer with shifted windows (patch size 4, window 7) at 224×224, pretrained on ImageNet-22k [27], and RadioDino-s8 [45], a DINO-style self-supervised ViT pretrained on RadImageNet [29] as a domain-adapted medical backbone. Holding inputs fixed, this ablation isolates how architectural choices and pretraining affect the model’s ability to learn disease-aware slice-importance scores from our importance map enhanced CT intensity projections. The results in Table 9 show that while Swin-Tiny attains the highest AUC and RadioDino-s8 slightly leads on Spearman’s ρ\rho, the validation-selected ViT-B/16 backbone delivers the best diagnostically relevant slice retrieval performance, with higher Hit@KK across K=1,5,10K=1, 5, 10.

Table 9. Backbone Ablation. Under AIP with wide window projection and α=0.75\alpha=0.75.

BackboneAUCSpear.Hit@1Hit@5Hit@10
RadioDino-s80.7770.2270.1800.3610.487
Swin-Tiny0.7840.2220.2030.3580.472
ViT-B/160.7810.2240.2280.4210.563

Table 9.

Gaze-Only and CT-Derived Prior Ablations

We further analyze the role of disease-conditioned priors in XGLASS by comparing transferred CXR gaze priors with empirical CT abnormality-location priors. First, we evaluate the CXR gaze prior alone by averaging each disease-specific gaze map along the left–right axis, interpolating the resulting superior–inferior profile to 240 axial positions, and ranking slices by profile intensity. Second, we construct an empirical CT location prior using only train+val labels by averaging slice-presence vectors for each disease category and applying the resulting fixed category-specific prior to all corresponding test cases. Third, we replace the gaze map in XGLASS with this CT-derived prior converted into a synthetic heatmap and fuse it with the CT projection in the same way as the gaze heatmap. Since this CT prior is computed from train+val ground-truth slice-presence targets from the same CT dataset distribution, it represents an empirical average lesion-location prior for each abnormality. In contrast, the gaze prior is transferred from CXR eye-tracking and does not use CT lesion-location statistics; instead, it captures radiologist visual search patterns that include both observed abnormality regions and plausible candidate regions inspected for each finding. In contrast, the CT-derived prior is less reliable when abnormalities occur in rare locations, whereas the broader radiologist search patterns encoded by the gaze prior can still overlap these regions, supporting the use of gaze over CT derived location priors. Table 10 shows that CT location statistics are informative: the CT location prior alone achieves the best Hit@1 and Spearman correlation. However, the full gaze-guided XGLASS model achieves the best AUC and performs substantially better for utility-centric top-KK retrieval metrics, improving Hit@10 from 0.453 to 0.563 and Hit@20 from 0.522 to 0.693 compared with the CT location prior alone.

Table 10. Prior-Only and Prior-Injection Ablations. Comparison of static CXR gaze priors, empirical CT location priors, CT-only XGLASS, CT-prior injection, and full gaze-guided XGLASS.

MethodAUCSpear.Hit@1Hit@5Hit@10Hit@20
Gaze prior only0.6380.1630.1580.2090.2630.358
CT location prior only0.7340.2290.2500.3290.4530.522
XGLASS w/o gaze cues0.7680.2120.1990.3420.4680.619
XGLASS + CT loc. prior0.7220.1990.1990.4240.5350.658
XGLASS w/ gaze cues0.7810.2240.2280.4210.5630.693

Table 10.

Injecting the CT-derived prior into XGLASS also improves over CT-only input at Hit@10 (0.535 vs. 0.468). Nevertheless, gaze-guided XGLASS remains stronger at Hit@10 and Hit@20, indicating that the transferred radiologist gaze prior provides complementary information beyond average CT lesion-frequency statistics, which improves the slice selection performance especially for higher KK. We additionally stratify test samples by their typicality under the train+val CT location prior, defined as the average CT prior score over the ground-truth positive slices of each test sample. In low-typicality cases, where the empirical CT prior assigns low probability to the true lesion region, the CT location prior drops sharply to Hit@10 of 0.151. In contrast, the CXR gaze prior alone remains stronger at Hit@10 of 0.330, while full XGLASS reaches Hit@10 of 0.358 and Hit@20 of 0.528. This suggests that empirical CT priors primarily capture common dataset-specific lesion locations, whereas radiologist gaze priors cover broader disease-specific search regions. Combined with patient-specific CT projection evidence, this leads to more robust disease-conditioned slice prioritization than static location priors alone.

Robustness to Gaze-Prior Perturbations

To test whether XGLASS depends on exact pixel-level alignment of the transferred gaze prior, we first measured coarse anatomical alignment between CXR-derived priors and CT projections using lung and heart layouts, obtaining a lungs+heart centroid error of 16.2 px and an average symmetric surface distance of 11.6 px; for lungs alone, the corresponding errors were 8.8 px and 8.0 px. We then perturb the disease-conditioned heatmap at test time while keeping the CT projection unchanged. Table 11 shows that random spatial shifts, additive noise, and left-right heatmap flips only mildly affect performance, supporting that XGLASS uses the gaze map as a broad anatomical search prior, that is robust to minor perturbations, rather than an exact lesion-location mask.

ConditionAUCSpear.Hit@1Hit@5Hit@10
Baseline0.781‾\underline{0.781}0.2240.2280.421‾\underline{0.421}0.563
Random shift ±16\pm16 px0.7780.221‾\underline{0.221}0.2060.4050.528
Noise σ=0.1\sigma=0.10.7750.2200.215‾\underline{0.215}0.4270.557‾\underline{0.557}
Left–right flip0.7860.2240.2030.3960.563

Table 11. Robustness to Gaze-Prior Perturbations. Test-time perturbations of the disease-conditioned gaze map.

Loss Function Ablation

Abnormalities vary substantially in spatial extent, e.g., consolidations often cover larger regions than lung nodules. We therefore compare four training objectives while keeping all other settings fixed: MSE regresses the normalized slice-area target, Focal treats each slice as positive or negative and emphasizes difficult cases, Slice-bal. MSE gives equal aggregate weight to positive and negative slices within each CT, preventing scans with many positive slices from dominating the loss, and Size-wt. MSE inversely weights positive-slice errors by the total lesion size in each CT, reducing the contribution of spatially extensive lesions. Table 12 shows that MSE gives the strongest overall retrieval-ranking trade-off, with the best Hit@1 and Hit@10 and tied-best Hit@5. We further stratify the same ablation by finding extent. Test cases are split into small, medium, and large tertiles by total ground-truth slice-area mass, corresponding to average extents of 5.6, 14.7, and 52.4 positive slices, respectively.

LossAUCSpear.Hit@1Hit@5Hit@10
MSE0.7810.2240.2280.4210.563
Focal0.7900.2180.1800.3920.516
Slice-bal. MSE0.7090.2020.2060.4210.519‾\underline{0.519}
Size-wt. MSE0.785‾\underline{0.785}0.221‾\underline{0.221}0.215‾\underline{0.215}0.411‾\underline{0.411}0.519‾\underline{0.519}

Table 12. Loss-function ablation. We compare Area MSE with binary and lesion-size-aware alternatives.

Table 13 shows that MSE is strongest for small sparse findings across retrieval metrics, whereas Focal and Slice-bal. MSE mainly improve medium or large findings. Although the reweighted objectives were intended to improve small-lesion retrieval, they appear to amplify the instability of very sparse positive targets: for small findings, only a few slices are labeled relevant and the corresponding projection evidence is often weak, so upweighting these slices can emphasize case-specific noise. In contrast, MSE preserves the continuous slice-area target and provides a smoother ranking signal, making it the most stable default objective.

LossSizeHit@1Hit@3Hit@5Hit@10AR@10
MSESmall0.0570.1230.1890.3400.103
MSEMedium0.114‾\underline{0.114}0.2760.3520.543‾\underline{0.543}0.079
MSELarge0.5140.6190.724‾\underline{0.724}0.810‾\underline{0.810}0.090‾\underline{0.090}
FocalSmall0.0190.0750.1040.2260.065
FocalMedium0.1620.3520.4760.5810.108
FocalLarge0.4380.6000.6670.7520.090
Slice-bal. MSESmall0.047‾\underline{0.047}0.113‾\underline{0.113}0.1420.2080.073
Slice-bal. MSEMedium0.1050.2570.3520.4860.077
Slice-bal. MSELarge0.4670.6570.7710.8670.087
Size-wt. MSESmall0.0280.1040.151‾\underline{0.151}0.264‾\underline{0.264}0.085‾\underline{0.085}
Size-wt. MSEMedium0.114‾\underline{0.114}0.257‾\underline{0.257}0.362‾\underline{0.362}0.5050.087
Size-wt. MSELarge0.505‾\underline{0.505}0.648‾\underline{0.648}0.7240.7900.089

Table 13. Loss retrieval metrics by finding size. Test cases are grouped into tertiles by total ground-truth slice-area mass. We report retrieval metrics within each size group.

Additional Implementation Details

BioMedCLIP Slice Retrieval

We use BioMedCLIP [46] as a zero-shot slice-wise image–text retrieval baseline. We follow its official evaluation pipeline by rendering each axial CT slice with a wide window (center 0 HU, width 2000 HU), converted to RGB, and encoded independently. Finding names are normalized where appropriate (e.g., “lung nodule or mass” to “pulmonary nodule” and “groundglass opacity” to “ground-glass opacity”). To reduce sensitivity to a single prompt formulation, we average the cosine similarities obtained from the following five templates, where {finding} denotes the normalized finding name:

  1. The radiological finding is {finding}.

  2. This axial CT image shows {finding}.

  3. The CT slice is positive for {finding}.

  4. There is {finding} in this CT slice.

  5. A finding of {finding} is present on this CT image.

Slices are ranked in descending order of the ensemble-averaged similarity. For the downstream binary-screening analysis, one global similarity threshold (0.3751) is selected on the all-slice BioMedCLIP baseline by maximizing F1 and then applied unchanged to every XGLASS Top-KK subset.

3D Baseline Implementation Details

For the 3D Merlin + MLP baseline, we follow the CT preparation used by Merlin [6]: full CT volumes are resampled to 1.5×1.5×3.01.5 \times1.5 \times3.0 mm spacing and resized to 160×224×224160 \times224 \times224 before encoding. The Merlin encoder is kept frozen and used to precompute one 2048-dimensional volume embedding per CT. We then train a disease-conditioned MLP on ReXGroundingCT to regress the same 240-dimensional slice-relevance target as XGLASS. The MLP uses a 32-dimensional disease embedding concatenated with the Merlin feature, followed by two 512-unit hidden layers with ReLU activations and dropout 0.1, and a final linear layer producing 240 slice scores. The MLP is trained with AdamW, learning rate 10−310^{-3}, weight decay 10−410^{-4}, batch size 64, and the best checkpoint is selected on the validation loss. Including embedding precomputation, the full Merlin baseline required approximately 6 hours on an A100 GPU. For the 3D U-Net baseline, we train a compact 3D U-Net from scratch on ReXGroundingCT. Full CT volumes are resampled to 1.25 mm isotropic spacing and resized to 160×224×224160 \times224 \times224. The encoder uses 3D convolutional blocks with base width 16 and instance normalization. To obtain slice scores, the final feature maps are averaged over the in-plane dimensions, interpolated to 240 axial positions, concatenated with a 32-dimensional disease embedding, and passed through a slice-regression head. The model is trained with AdamW, learning rate 10−410^{-4}, weight decay 10−410^{-4}, batch size 16, dropout 0.1, and the best checkpoint is selected on the validation loss and Training to the selected checkpoint required approximately 22 hours on an A100 GPU. Both 3D baselines are therefore substantially more compute-intensive to train than XGLASS, which trains in under 10 minutes on a single RTX 4080.

Slice-wise Medical VLM Screening

We evaluate google/medgemma-4b-it [35] and lingshu-medical-mllm/Lingshu-7B [44] using the same disease-conditioned binary screening task. Motivated by CT vision-language transfer work that maps complementary HU windows to RGB channels to expose different tissue contrasts [30], the main-paper experiments use a pseudo-RGB representation with wide/lung, soft-tissue, and mediastinal/fluid windows assigned to separate channels. As an input-rendering sensitivity analysis, we additionally evaluate a standard wide-window representation (−1024 to 1024 HU) and replicated across all RGB channels. This alternative setting shows the same trend: XGLASS Top-KK preselection improves F1 over all-slice screening, with the strongest gains at small budgets (e.g., Top-30 F1 of 0.414 for MedGemma and 0.441 for Lingshu). Thus, the downstream benefit of XGLASS is consistent across CT slice renderings. Both models receive the same system and user instructions:

System: You are an expert thoracic radiologist. Evaluate only the queried finding on one axial chest CT slice.

User: Are there clear visual signs of {finding} in this axial CT slice? Ignore other abnormalities. If the queried finding is not visible in this slice, answer no. Return only one short line, with no explanation: Answer: yes or Answer: no.

We use deterministic decoding and parse the response as yes, no, or unknown; no unknown responses occurred in the reported MedGemma and Lingshu evaluation. MedGemma is evaluated in bfloat16 without quantization, while Lingshu is loaded with 4-bit NF4 quantization and bfloat16 computation due to memory requirements. Predictions are generated independently for every slice and cached, allowing identical predictions to be evaluated on either all 240 slices or XGLASS-selected Top-KK subsets. Precision, recall, and F1 are computed only over the inspected slices.

Downstream Slice Selection Comparison

We further analyze how slice selection affects downstream screening when the number of inspected slices is fixed. For each CT–finding pair, we vary only the ranking used to choose the Top-KK inspected slices, while keeping the downstream slice-wise predictions unchanged. This isolates the effect of slice selection from stochasticity or repeated downstream model calls. We compare XGLASS with Random selection, Vote-MI, BioMedCLIP retrieval, and the two 3D-native selectors, 3D Merlin + MLP and 3D U-Net. BioMedCLIP retrieval is evaluated as a selector for MedGemma and Lingshu, but omitted from the BioMedCLIP downstream panel to avoid selecting and classifying slices with the same image–text similarity model. Figure 10 shows that selector choice strongly affects low-budget screening, especially at K=30K=30 and K=60K=60. XGLASS consistently outperforms the other 2D selectors under matched slice budgets, while the 3D selectors are competitive and sometimes achieve the highest downstream F1 for specific model–budget combinations. Notably, XGLASS remains close to these volumetric selectors despite using only a lightweight 2D projection, and outperforms them for Lingshu at K=30K=30, 60, and 180. This suggests that the disease-conditioned XGLASS ranking transfers effectively to downstream screening, narrowing much of the gap to more expensive 3D-native selection.

Figure 10. Matched-budget downstream screening across slice selectors. For each CT–finding pair, different selectors choose the Top-KK inspected axial slices under the same slice budget. XGLASS is compared with BioMedCLIP retrieval, Vote-MI, 3D U-Net, 3D Merlin and Random selection across MedGemma 4B, Lingshu 7B, and BioMedCLIP downstream screening.

Downstream Computational Cost

Applying the downstream models to all 240 slices of one CT requires approximately 1.40 s for BioMedCLIP, 154.91 s for MedGemma, and 267.83 s for Lingshu on a single NVIDIA RTX 4080. XGLASS requires only 0.025 s per finding (0.150 s per CT) to rank the complete volume and allows the downstream model to inspect a smaller, disease-enriched subset. At Top-30, including XGLASS preselection, the corresponding per-CT runtimes decrease to approximately 0.20 s, 19.39 s, and 33.51 s, respectively. Thus, XGLASS reduces downstream runtime by 85.7–87.5% while simultaneously improving F1 at the smaller slice budget, showing that preselection improves both computational efficiency and downstream screening performance.

References

  1. [1]Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt-4 technical report. arXiv, 2023. 1arxiv.org/abs/2303.08774
  2. [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: a visual language model for few-shot learning. Adv. Neural Inf. Process. Syst., 35:23716–23736, 2022. 1arxiv.org/abs/2204.14198
  3. [3]Mohammed Baharoon, Luyang Luo, Michael Moritz, et al. Rexgroundingct: A 3d chest ct dataset for segmentation of findings from free-text reports. arXiv, 2025. 3, 12arxiv.org/abs/2507.22030
  4. [4]Ricardo Bigolin Lanfredi, Joyce D Schroeder, and Tolga Tasdizen. Localization supervision of chest x-ray classifiers using label-specific eye-tracking annotation. Frontiers in radiology, 3:1088068, 2023. 2DOI
  5. [5]Ricardo Bigolin Lanfredi et al. Reflacx, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays. Scientific data, 9(1):350, 2022. 2, 3, 11DOI
  6. [6]Louis Blankemeier, Ashwin Kumar, Joseph Paul Cohen, Jiaming Liu, Longchao Liu, Dave Van Veen, Syed Jamal Safdar Gardezi, Hongkun Yu, Magdalini Paschali, Zhihong Chen, et al. Merlin: a computed tomography vision-language foundation model and dataset. Nature, 652(8112):1318–1328, 2026. 4, 5, 6, 15, 17
  7. [7]Aurelia Bustos et al. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797, 2020. 1DOI
  8. [8]Xi Chen, Xiao Wang, Soravit Changpinyo, et al. Pali: A jointly-scaled multilingual language-image model. arXiv, 2022. 1arxiv.org/abs/2209.06794
  9. [9]Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pages 424–432. Springer, 2016. 4, 5, 6, 15arxiv.org/abs/1606.06650
  10. [10]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 4, 14
  11. [11]Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geoffrey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical image analysis, 67:101857, 2021. 2DOI
  12. [12]Trafton Drew, Melissa Le-Hoa Vo, Alex Olwal, Francine Jacobson, Steven E Seltzer, and Jeremy M Wolfe. Scanners and drillers: characterizing expert visual search through volumetric images. Journal of vision, 13(10):3–3, 2013. 2DOI
  13. [13]Bradley Efron. Bootstrap methods: another look at the jack-knife. In Breakthroughs in statistics: Methodology and distribution, pages 569–593. Springer, 1992. 5
  14. [14]Zenzi M Griffin and Kathryn Bock. What the eyes say about speaking. Psychological science, 11(4):274–279, 2000. 2pubmed.ncbi.nlm.nih.gov/11273384
  15. [15]Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv, 2020. 1arxiv.org/abs/2003.10286
  16. [16]Chihcheng Hsieh, Chun Ouyang, et al. Mimic-eye: Integrating mimic datasets with reflacx and eye gaze for multimodal deep learning applications. PhysioNet (version 1.0. 0), 2023. 2
  17. [17]Jeremy Irvin, Pranav Rajpurkar, Michael Ko, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proc. of the AAAI, pages 590–597, 2019. 1arxiv.org/abs/1901.07031
  18. [18]Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, et al. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv, 2019. 1
  19. [19]Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy Wu, Matthew Tong, Arjun Sharma, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth Krupinski, and Mehdi Moradi. Eye Gaze Data for Chest X-rays (version 1.0.0), 2020. 2
  20. [20]Linda C Kelahan, Allan Fong, Joseph Blumenthal, Swaminathan Kandaswamy, Raj M Ratwani, and Ross W Filice. The radiologist’s gaze: mapping three-dimensional visual search in computed tomography of the abdomen and pelvis. Journal of Digital Imaging, 32(2):234–240, 2019. 2DOI
  21. [21]Yoonsoo Kim, Jinge Wu, Yusuf Abdulle, Yue Gao, and Honghan Wu. Enhancing human-computer interaction in chest x-ray analysis using vision and language model with eye gaze patterns. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 184–194. Springer, 2024. 2arxiv.org/abs/2404.02370
  22. [22]Harold L Kundel, Calvin F Nodine, Elizabeth A Krupinski, and Claudia Mello-Thoms. Using gaze-tracking data and mixture distribution analysis to support a holistic model for the detection of cancers on mammograms. Academic radiology, 15(7):881–886, 2008. 2DOI
  23. [23]Jason J Lau, Soumya Gayen, et al. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. 1DOI
  24. [24]Chunyuang Li, Cliff Wong, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Adv. Neural Inf. Process. Syst., 36:28541–28564, 2023. 1arxiv.org/abs/2306.00890
  25. [25]Weixiong Lin, Ziheng Zhao, et al. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In Proc. of MICCAI, pages 525–536. Springer, 2023. 1arxiv.org/abs/2303.07240
  26. [26]Bo Liu, Li-Ming Jin, Chao Liu, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In Proc. IEEE Int. Symp. Biomed. Imaging (ISBI), pages 1650–1654. IEEE, 2021. 1arxiv.org/abs/2102.09542
  27. [27]Ze Liu, Yutong Lin, Yue Cao, et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. of ICCV, 2021. 14arxiv.org/abs/2103.14030
  28. [28]Chong Ma, Hanqi Jiang, Wenting Chen, Yiwei Li, Zihao Wu, XiaoWei Yu, Zhengliang Liu, Lei Guo, Dajiang Zhu, Tuo Zhang, et al. Eye-gaze guided multi-modal alignment for medical representation learning. Advances in Neural Information Processing Systems, 37:6126–6153, 2024. 2arxiv.org/abs/2403.12416
  29. [29]Xueyan Mei, Zelong Liu, Philip M Robson, et al. Radimagenet: an open radiologic deep learning research dataset for effective transfer learning. Radiology: Artificial Intelligence, 4(5):e210315, 2022. 15DOI
  30. [30]Cristian Minnoccheri, Emily Wittrup, Kayvan Najarian, and Ryan Stidham. Representation geometry shapes task performance in vision-language modeling for ct enterography. arXiv preprint arXiv:2604.13021, 2026. 18
  31. [31]Andriy Myronenko, Dong Yang, Baris Turkbey, et al. Reasoning visual language model for chest x-ray analysis. arXiv, 2025. 1arxiv.org/abs/2510.23968
  32. [32]Trong Thang Pham, Akash Awasthi, Saba Khan, et al. Ct-scangaze: A dataset and baselines for 3d volumetric scanpath modeling. In Proc. of ICCV, pages 21732–21743, 2025. 2, 11arxiv.org/abs/2507.12591
  33. [33]Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In Proc. of ICML, pages 8748–8763, 2021. 1arxiv.org/abs/2103.00020
  34. [34]Philip Sedgwick. Spearman’s rank correlation coefficient. Bmj, 349, 2014. 4DOI
  35. [35]Andrew Sellergren, Sahar Kazemzadeh, et al. Medgemma technical report. arXiv, 2025. 1, 7, 18arxiv.org/abs/2507.05201
  36. [36]Donald F Swinehart. The beer-lambert law. Journal of chemical education, 39(7):333, 1962. 3DOI
  37. [37]Gemma Team, Aishwarya Kamath, Johan Ferret, et al. Gemma 3 technical report. arXiv, 2025. 1
  38. [38]Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv, 2023. 1arxiv.org/abs/2302.13971
  39. [39]A Van der Gijp, CJ Ravensloot, H Jarodzka, et al. How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education, 22(3):765–787, 2017. 2DOI
  40. [40]Maya Varma, Ashwin Kumar, Rogier van der Sluijs, et al. Medvae: Efficient automated interpretation of medical images with large-scale generalizable autoencoders, 2025. 6arxiv.org/abs/2502.14753
  41. [41]Yuli Wang, Jian Peng, Yuwei Dai, Craig Jones, Haris Sair, Jinglai Shen, Nicolas Loizou, Jing Wu, Wen-Chi Hsu, Mal- iha Imami, et al. Enhancing vision-language models for medical imaging: bridging the 3d gap with innovative slice selection. Advances in Neural Information Processing Systems, 37:99947–99964, 2024. 1, 2, 4, 5, 15DOI
  42. [42]Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3876–3887, 2022. 1arxiv.org/abs/2210.10163
  43. [43]David S Wooding. Fixation maps: quantifying eye-movement traces. In Proceedings of the 2002 symposium on Eye tracking research & applications, pages 31–36, 2002. 2
  44. [44]Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044, 2025. 7, 18arxiv.org/abs/2506.07044
  45. [45]Luca Zedda et al. Radio dino: A foundation model for advanced radiomics and ai-driven medical imaging analysis. Comput. Biol. Med., 195:110583, 2025. 15DOI
  46. [46]Sheng Zhang, Yanbo Xu, Naoto Usuyama, et al. A multi-modal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI, 2(1):AIoa2400640, 2025. 1, 4, 5, 7, 15, 17DOI
  47. [47]Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, et al. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv, 2023. 1arxiv.org/abs/2305.10415
  48. [48]Yuhao Zhang, Hang Jiang, et al. Contrastive learning of medical visual representations from paired images and text. In Proc. of MLHC, pages 2–25. PMLR, 2022. 1

Paper details

vlnc:W_by8pmzp4
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents