WorksComputer Science
GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification
Abstract
Early and accurate glaucoma detection is critical to prevent irreversible vision loss, yet existing AI methods often rely on unimodal inputs and lack interpretability. We present GlaBoost, a multimodal gradient boosting framework that unifies three complementary signals for glaucoma risk prediction: fundus image embeddings from a pretrained convolutional encoder,free-text neuroretinal rim assessments encoded by a transformer-based language model, and structured ophthalmic biomarkers. These modalities are fused into a single representation and classified by an enhanced XGBoost model.On two real-world annotated datasets, GlaBoost consistently outperforms unimodal and generic multimodal baselines. Feature importance analysis highlights the cup-to-disc ratio, rim thinning, and the ISNT rule as the dominant predictors, yielding clinically consistent and interpretable decisions. GlaBoost offers a transparent and scalable foundation for multimodal decision support in ophthalmology.
Introduction
Glaucoma is a progressive optic neuropathy that ranks among the leading causes of irreversible blindness worldwide [20, 21, 22]. Early and accurate diagnosis is critical but remains challenging due to the disease’s complex presentation and its reliance on heterogeneous diagnostic evidence—fundus images [23], anatomical measurements [19], and expert clinical descriptions [18].
Recent AI-based approaches, particularly those built on convolutional neural networks (CNNs), have achieved strong performance in image-based glaucoma detection [19], [28], [30], [32], [35], [36]. However, these systems typically operate on a single visual modality and fail to incorporate the structured biomarkers and free-text annotations that ophthalmologists routinely rely on [18], [24]. This unimodal bias limits both diagnostic accuracy and clinical interpretability.
To bridge this gap, we propose GlaBoost, a multimodal learning framework that fuses fundus image embeddings, structured ophthalmic biomarkers, and transformer-encoded clinical narratives within an enhanced XGBoost classifier [4]. By explicitly modeling expert textual descriptions alongside visual and numeric features, GlaBoost captures the semantic context that underlies clinical reasoning.
Our main contributions are summarized as follows:
We propose the GlaBoost, a multimodal XGBoost-based framework that jointly leverages fundus images, structured anatomical biomarkers, and free-text clinical descriptions for glaucoma risk prediction.
We design an effective pipeline for encoding free-text ophthalmic observations via a pretrained language model, and quantify their contribution to diagnostic performance through ablation studies.
We demonstrate, across two real-world datasets and a comprehensive set of unimodal, multimodal, and large language model baselines, that multimodal fusion improves both predictive accuracy and feature-level interpretability in glaucoma diagnosis.
Related Work
AI for Glaucoma
Most existing glaucoma AI systems are built around CNN-based architectures [21], [22], including U-Net variants [32], and have been applied to detection, segmentation, and reconstruction across diverse retinal imaging modalities [19], [23], [28], [29], [35]. Recent progress in large language models (LLMs) and vision-language models (VLMs) has further enabled the incorporation of textual evidence into medical AI [1], [2], [13], [25]. Nevertheless, current glaucoma AI remains predominantly image-driven [19], [28], [35], while clinical diagnosis depends on interpretable biomarkers such as intraocular pressure (IOP) and cup-to-disc ratio [6], [20], a mismatch that constrains both clinical adoption and trust.
Clinical Decision Making
In practice, ophthalmologists synthesize fundus imaging, patient history, and free-text descriptions of optic nerve head morphology to reach a diagnosis [6], [20]. Such textual descriptions encode semantic cues, e.g., rim pallor, notching, and bayoneting, that are difficult to recover from pixel-level features alone [18], [24]. Although recent explainable AI techniques offer interpretability through feature attribution or attention-based visualization [39], they are typically applied to a single modality and therefore cannot align with the inherently multimodal nature of clinical reasoning. GlaBoost is designed to close this gap by jointly modeling imaging, structured biomarkers, and expert narratives within a single interpretable framework.
Dr. Cheng Huang, Zeyu Han, Weizheng Xie and Dr. Jia Zhang are with the Department of Computer Science, Southern Methodist University, Dallas, TX 75205, USA.
Dr. Karanjit Kooner and Dr. Jui-Kai Wang are with the Department of Ophthalmology, UT Southwestern Medical Center, Dallas, TX 75390, USA.
Dr. Tsengdar Lee is with the High Performance Computing Program, National Aeronautics and Space Administration, Washington, DC 20546, USA.
†Corresponding Author: [email protected]
Methodology: GlaBoost
Data Input: let denote the dataset, where each instance consists of multimodal features and a binary label indicating the presence of glaucoma. We decompose the input feature vector as:
where is the textual embedding, represents structured clinical features, and are visual features extracted from fundus images. is the data based on human subjective judgment, not a label, and in certain datasets, it is not required and is considered optional.
Based on this input of data, as shown in Figure 1 GlaBoost have 4 stages: (1) Structured Feature Extraction; (2) Image Feature Extraction; (3) Textual Feature Extraction and (4) Multimodal Fusion and Classification.

Figure 1. The Architecture of GlaBoost
Structured Feature Extraction
Each instance contains a clinical report field description where we extract two or one types of structured features:
Machine-Readable Fundus Biomarker: Within each report, the
fundus_featuresfield provides structured annotations of retinal morphology. We extract clinically established indicators, including optic disc size (small, normal, large) and the cup-to-disc ratio (). Categorical features are converted via one-hot encoding, and missing values are explicitly encoded as additional categories to ensure model compatibility, as formulated in Eq. (2):
where extracts raw fundus-related attributes, and denotes a preprocessing function that performs one-hot encoding, normalization of continuous values, and explicit handling of missing entries. The resulting vector provides a fixed-dimensional representation of structured clinical information.
Human-Evaluated Indicator (Optional): We extract two subjective diagnostic indicators provided by clinical annotators:
Glaucoma_risk_assessmentandConfidence_level. The glaucoma risk assessment is treated as a categorical judgment and encoded using one-hot vectors, while the confidence level is retained as a continuous score in . Both indicators are processed using the same feature extraction formulation as the structured data stage to obtain .
Image Feature Extraction
We apply a pretrained CNN, ResNet-152 [15] to extract image features from each retinal fundus photograph :
The network is initialized with ImageNet-pretrained weights and extracts high-level image representations from the penultimate layer. All convolutional layers are frozen to reduce overfitting, with only the final projection layers optionally fine-tuned. Input images are resized to , normalized using ImageNet statistics, and optionally augmented during training with random horizontal flips and color jitter.
Textual Feature Extraction
Each clinical report contains unstructured narrative fields such as some description, which often include nuanced spatial and qualitative assessments of optic disc morphology. To encode this textual modality, we use a Transformer-based language encoder , mBERT [9]:
where denotes the input text token sequence and mean-pooling aggregates contextualized token embeddings. The model is initialized with domain-general weights and remains frozen to avoid overfitting due to limited textual supervision. We apply the following preprocessing to : (1) Lowercasing and punctuation normalization; (2) Truncation or padding to a maximum sequence length and (3) Removal of medically irrelevant boilerplate text (optional).
Multimodal Fusion and Classification
The final feature vector is constructed by concatenating all modalities:
We use an XGBoost classifier [4] to predict the label:
and optimize it with logistic loss:
where is a regularization term controlling model complexity.
Dataset
Ethics and Data Governance. The experimental procedures involving human subjects described in this paper were approved by the Institutional Review Board (IRB) of the University of Texas Southwestern Medical Center (UTSW), in accordance with the Helsinki Declaration of 1975 (revised in 2000). As the study is retrospective, the IRB waived the requirement for written informed consent. All patient records and fundus images used in this work were fully de-identified prior to analysis, and no protected health information was accessible to the modeling pipeline.
Public Dataset: TABLE I shows a representative sample from the public dataset, which is organized as JSON-formatted clinical reports paired with fundus images. During preprocessing, categorical fields such as
optic_disc_sizeand continuous fields such ascup_to_disc_ratioare mapped to binary indicators (True/False) following established clinical thresholds for glaucoma diagnosis [6]. The free-text fieldneuroretinal_rimis retained as an unstructured description and processed through the textual encoder.
The two fields glaucoma_risk_assessment and confidence_level encode an annotator’s preliminary risk tier and self-rated confidence, respectively. We treat them as auxiliary clinician-derived inputs rather than ground-truth labels. A representative fundus image is shown in Figure 2a. The dataset is publicly available1.

Figure 2. The Sample of Image from Public and Private Dataset
| Field | Value |
optic_disc_size | large |
cup_to_disc_ratio | 0.8 |
isnt_rule_followed | false |
rim_pallor | true |
rim_color | pale |
bayoneting | true |
sharp_edge | true |
laminar_dot_sign | true |
notching | true |
rim_thinning | true |
additional_observations | null |
neuroretinal_rim | (free-text description) |
glaucoma_risk_assessment | high risk |
confidence_level | 0.9 |
Table 1. Representative Sample from the Public Dataset
Private Dataset: TABLE II shows a representative sample from the private UTSW dataset, which is derived from clinical OCT reports. Each record contains 16 biomarkers organized into three anatomical categories, RNFL, ONH, and GCC, measured independently for OD and OS eyes, together with the corresponding IE.
Status Encoding Each biomarker is associated with a clinical status assigned by the OCT analyzer: Within Normal, Borderline, or Outside Normal. Borderline values lie within the normal reference range but approach the pathological boundary, and therefore carry diagnostic significance that is distinct from both ends. We encode each biomarker status as an ordinal score corresponding to , preserving the underlying clinical ordering. Unless a biomarker is explicitly marked as Within Normal in the report, Borderline cases are retained as a distinct third category rather than being collapsed into the Normal class.
Annotation All UTSW records are reviewed by two board-certified glaucoma specialists from the Department of Ophthalmology, UT Southwestern Medical Center, who provide the final diagnostic labels used as ground truth in this study. A representative fundus image is shown in Figure 2b.
Abbreviations: RNFL: retinal nerve fiber layer (AR: Average; SR: Superior; IR: Inferior; I-ER (S−I): intra-eye Superior−Inferior difference). ONH: optic nerve head (A-O / V-O / H-O: cup-to-disc area / volume / horizontal ratio; RA: rim area; DA: disc area; CVO: cup volume). GCC: ganglion cell complex (A-G: Average; S-G: Superior; I-F: Inferior; I-EG (S−I): intra-eye Superior−Inferior difference; FLV: focal loss volume; GLV: global loss volume). OD: right eye; OS: left eye; IE: inter-eye difference.
Experimental Setup
Baseline Model: we have chosen these following models as baseline models: RNN [11], Bi-RNN [34], LSTM [16], Bi-LSTM [16, 34], GRU [5], BERT [9], Transformer [39], Mamba [7, 12], RoBERTa [26], CNN [15], ViT [10], and ConViT [8]. Except for the last three CV models, CNN [15], ViT [10], and ConViT [8], all others are primarily designed for NLP tasks. We also choose the machine learning models: Linear Regression [14], Logistic Regression [17], Random Forest [3] and XGBoost [4]. For parameter optimization, we selected tool Optuna2 as the optimization method.
| LLM | Version | DR | LR | BS | MSL | Epoch | GC |
| GPT | 4O | 8 | 512 | 5 | 1.0 | ||
| GPT | O3 | 8 | 512 | 5 | 1.0 | ||
| LLaMA | 3.1-405B | 8 | 512 | 5 | 1.0 | ||
| LLaMA | 3.1-70B | 8 | 512 | 5 | 1.0 | ||
| DeepSeek | R1 | 8 | 512 | 5 | 1.0 | ||
| DeepSeek | V3 | 8 | 512 | 5 | 1.0 |
Table 3. Configuration for Fine-tuning with MLP Head
| Model | Image | Words | Factor | Risk | Sure | ACC | PRE | F1 | Image | Factor | ACC | PRE | F1 | Cause |
| Linear Regression | 45.72 | 51.03 | 48.26 | 34.29 | 39.25 | 36.59 | Yes | |||||||
| Logistic Regression | 64.37 | 67.29 | 65.83 | 58.74 | 59.24 | 58.95 | Yes | |||||||
| Random Forest | 78.93 | 81.77 | 80.30 | 69.97 | 72.35 | 71.09 | Yes | |||||||
| XGBoost | 98.22 | 97.86 | 98.04 | 97.83 | 98.33 | 98.08 | Yes | |||||||
| GlaBoost | 98.73 | 99.01 | 98.83 | 99.21 | 98.89 | 99.10 | Yes | |||||||
| RNN | 95.62 | 96.72 | 96.17 | 95.50 | 95.38 | 95.44 | Yes | |||||||
| Bi-RNN | 96.72 | 97.84 | 97.28 | 97.58 | 97.04 | 97.31 | Yes | |||||||
| LSTM | 97.44 | 98.34 | 97.89 | 96.65 | 98.27 | 97.45 | Yes | |||||||
| Bi-LSTM | 97.74 | 97.00 | 97.37 | 96.73 | 96.79 | 96.76 | Yes | |||||||
| GRU | 97.08 | 96.90 | 96.99 | 97.76 | 97.78 | 97.77 | Yes | |||||||
| BERT | 97.77 | 98.39 | 98.08 | 98.72 | 98.02 | 98.37 | Yes | |||||||
| Transformer | 97.86 | 98.90 | 98.38 | 98.96 | 98.11 | 98.53 | Yes | |||||||
| Mamba | 98.96 | 98.29 | 98.63 | 98.68 | 97.57 | 98.12 | Yes | |||||||
| RoBERTa | 98.10 | 98.57 | 98.33 | 97.59 | 99.32 | 98.45 | Yes | |||||||
| GlaBoost | 98.73 | 99.01 | 98.83 | 99.21 | 98.89 | 99.10 | Yes | |||||||
| CNN | 98.37 | 99.01 | 98.69 | 98.82 | 98.24 | 98.52 | No | |||||||
| ViT | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | No | |||||||
| ConViT | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | No | |||||||
| GlaBoost | 99.17 | 98.56 | 98.91 | 98.37 | 98.52 | 98.41 | No | |||||||
| CNN-RNN | 95.41 | 96.17 | 95.79 | 95.34 | 97.24 | 96.28 | Yes | |||||||
| CNN-LSTM | 96.96 | 96.42 | 96.69 | 95.53 | 95.45 | 95.49 | Yes | |||||||
| CNN-BERT | 97.02 | 96.61 | 96.81 | 97.43 | 96.49 | 96.96 | Yes | |||||||
| CNN-Transformer | 96.56 | 96.81 | 96.68 | 97.15 | 95.11 | 96.12 | Yes | |||||||
| ViT-Transformer | 95.73 | 96.90 | 96.31 | 96.91 | 96.71 | 96.81 | Yes | |||||||
| ViT-Mamba | 96.83 | 97.38 | 97.10 | 97.38 | 95.64 | 96.50 | Yes | |||||||
| ConViT-Transformer | 97.03 | 97.33 | 97.18 | 96.94 | 96.67 | 96.80 | Yes | |||||||
| ConViT-Mamba | 96.97 | 96.48 | 96.72 | 97.41 | 96.88 | 97.14 | Yes | |||||||
| GlaBoost | 98.36 | 98.52 | 98.71 | 98.96 | 99.03 | 98.91 | Yes | |||||||
| GPT-4O | 96.13 | 96.02 | 97.65 | 96.34 | 96.57 | 96.43 | Yes | |||||||
| GPT-O3 | 97.82 | 94.74 | 94.35 | 95.78 | 96.00 | 95.89 | Yes | |||||||
| DeepSeek-R1 | 94.38 | 95.26 | 94.72 | 95.23 | 95.45 | 95.31 | Yes | |||||||
| DeepSeek-V3 | 96.27 | 94.4 | 95.85 | 94.81 | 95.13 | 94.95 | Yes | |||||||
| LlaMA-3.1-405B | 96.33 | 95.49 | 96.76 | 94.18 | 94.42 | 94.32 | Yes | |||||||
| LlaMA-3.1-70B | 97.94 | 96.87 | 94.94 | 94.49 | 94.80 | 94.61 | Yes | |||||||
| GlaBoost | 98.36 | 98.52 | 98.71 | 98.96 | 99.03 | 98.91 | Yes | |||||||
| GPT-4O | 96.77 | 96.44 | 95.77 | 94.81 | 94.37 | 94.52 | Yes | |||||||
| GPT-O3 | 96.13 | 95.72 | 96.61 | 95.14 | 95.67 | 94.99 | Yes | |||||||
| DeepSeek-R1 | 95.57 | 97.29 | 96.65 | 93.68 | 95.22 | 94.01 | Yes | |||||||
| DeepSeek-V3 | 96.69 | 97.24 | 96.24 | 94.75 | 93.93 | 94.32 | Yes | |||||||
| LlaMA-3.1-405B | 97.19 | 97.29 | 97.01 | 93.52 | 94.13 | 93.78 | Yes | |||||||
| LlaMA-3.1-70B | 97.11 | 96.42 | 94.84 | 95.89 | 95.14 | 95.32 | Yes | |||||||
| GlaBoost | 97.52 | 96.83 | 96.13 | 95.42 | 94.76 | 94.93 | Yes | |||||||
| GPT-4O | 93.91 | 93.83 | 93.86 | 94.78 | 94.90 | 94.83 | Yes | |||||||
| GPT-O3 | 94.76 | 94.35 | 94.59 | 94.12 | 94.21 | 94.17 | Yes | |||||||
| DeepSeek-R1 | 94.15 | 94.08 | 94.11 | 93.79 | 93.92 | 93.85 | Yes | |||||||
| DeepSeek-V3 | 95.12 | 94.86 | 95.01 | 93.47 | 93.59 | 93.53 | Yes | |||||||
| LlaMA-3.1-405B | 93.52 | 93.30 | 93.38 | 93.05 | 93.12 | 93.09 | Yes | |||||||
| LlaMA-3.1-70B | 93.14 | 93.01 | 93.07 | 92.82 | 92.91 | 92.86 | Yes | |||||||
| GlaBoost | 95.82 | 95.41 | 95.63 | 95.21 | 95.38 | 95.29 | Yes | |||||||
| GPT-4O | 94.92 | 94.37 | 94.65 | - | - | - | - | |||||||
| GPT-O3 | 95.11 | 94.50 | 94.80 | - | - | - | - | |||||||
| DeepSeek-R1 | 95.72 | 95.03 | 95.37 | - | - | - | - | |||||||
| DeepSeek-V3 | 94.48 | 93.70 | 94.09 | - | - | - | - | |||||||
| LlaMA-3.1-405B | 95.63 | 94.80 | 95.21 | - | - | - | - | |||||||
| LlaMA-3.1-70B | 94.65 | 93.94 | 94.29 | - | - | - | - | |||||||
| GlaBoost | 96.00 | 95.46 | 95.73 | - | - | - | - |
Table IV. TABLE IV: Comparison of GlaBoost Against Machine Learning, NLP, Vision, Dual-Stream Multimodal, and Large Language Model Baselines on Public and Private Datasets
Large Language Model: we also use LLMs for evaluation, fine-tuning them with a classification head to align with the downstream task. They are divided into open source and closed source models. The open source model is LlaMA family [2], [37], [38] and DeepSeek family [13], [25], and the closed source model is GPT family [1], [31]. Regarding open-source LLM execution, DeepSeek family (R1, V3) [13], [25] were evaluated via their respective official APIs provided by the LLM developers. The LlaMA family (3.1-70B, 3.1-405B) [2], [37], [38] were accessed and run using the LlaMA-API platform. So do closed-source LLMs, like GPT family (4O, O3) [1], [31]. For the fine-tuning with the classification head (MLP, 2-layer and its size is 256 [33]), as shown in TABLE III it involves appending a task-specific classification head to the LLM and jointly optimizing both components on the binary classification objective. By updating the model parameters during training, it enables the LLM to learn task-specific representations and decision boundaries. The Optimizer is the AdamW [27]. The Decay Strategy is the Cosine w/10% warm-up. Finally, the loss function is the Cross-Entropy.
Note: Dropout Rate = DR; Learning Rate = LR; Batch Size = BS; Max Sequence Length = MSL; Gradient Clipping = GC.
Implementation
Hyperparameter: we implemented the multimodal classifier using XGBoost with standard settings, including a learning rate of 0.05, maximum tree depth of 6, 100 estimators,
and logloss as the evaluation metric, with a fixed random seed for reproducibility. Image features were extracted using a ResNet-152 pretrained on ImageNet, with frozen convolutional layers and inputs resized to 224×224. Textual features were obtained using a transformer-based mBERT model kept frozen during training, with inputs normalized, truncated to 128 tokens, and pooled to form sentence embeddings.
Hardware: the hardware specifications for training and testing include 2 Tesla V100 GPUs (2 × 32GB), 64GB of RAM, 8 CPU cores per node, and a total of 6 nodes.
Evaluation Metrics: We evaluate classification performance using accuracy (ACC), precision (PRE), and F1-score (F1). Accuracy measures the proportion of correctly classified samples, precision reflects the reliability of positive predictions, and F1-score provides a balanced assessment under class imbalance. The best results are highlighted in bold, and the second-best are underlined (×100%).
Experimental Result
Comparative Experiment
TABLE IV summarizes results across all baseline families. GlaBoost achieves the best accuracy, precision, and F1-score on both datasets. Vision-only models attain competitive numbers but cannot reason over structured biomarkers or clinical narratives, and are therefore marked “No” in the Cause column. Dual-stream multimodal encoders degrade as the input modality count grows, indicating that naïve feature concatenation is harder to optimize than tree-based fusion under limited data. Recent LLMs trail GlaBoost on most settings, suggesting that the diagnostic signal is dominated by a few structured biomarkers rather than long-form textual reasoning. Including clinician preliminary assessments raises performance for both LLM baselines and GlaBoost; we report results with and without such inputs side-by-side in TABLE IV to keep their contribution transparent rather than treating them as ground truth.
Ablation Experiment
Public Dataset: as shown in TABLE V, we evaluate the impact of different dataset components on model performance, both in isolation and in combination with other components. Using image-only inputs without any textual corpus, the model achieved performance comparable to that of multimodal approaches. However, it lacked clinical interpretability: while it could identify the presence of glaucoma, it failed to provide insights into the underlying biomarkers. Human evaluation factors, even when used in isolation, exert a measurable influence on the model’s performance. In scenarios where the model already achieves high precision, incorporating these human-derived factors often serves as the icing on the cake, further enhancing performance. Interestingly, when only images and vocabulary are provided, the model tends to ”speak for itself by looking at the picture”, leveraging both visual features and text-derived embeddings to construct a representation of ”glaucoma” for classification purposes.
| Data Category | Judgment | Public Dataset | Cause | |||||
| Image | Words | Factor | Risk | Sure | ACC | PRE | F1 | |
| - | - | - | - | |||||
| 8.72 | 7.51 | 6.94 | No | |||||
| 6.38 | 5.87 | 6.01 | No | |||||
| 9.45 | 8.36 | 7.89 | No | |||||
| 97.69 | 97.84 | 97.55 | Yes | |||||
| 97.98 | 98.22 | 97.81 | Yes | |||||
| _98.36_ | _98.71_ | _98.22_ | Yes | |||||
| 98.73 | 99.01 | 98.83 | Yes | |||||
| 46.27 | 49.13 | 47.12 | No | |||||
| 47.85 | 50.72 | 48.66 | No | |||||
| _48.94_ | _51.03_ | _49.71_ | No | |||||
| 50.67 | 53.45 | 51.89 | Yes | |||||
| 48.12 | 50.04 | 49.23 | Yes | |||||
| 49.03 | 51.67 | 50.11 | Yes | |||||
| 50.45 | 52.31 | 51.18 | Yes | |||||
| 52.37 | 54.89 | 53.61 | Yes | |||||
| _99.17_ | _98.56_ | _98.91_ | No | |||||
| 98.97 | 98.42 | 98.73 | Yes | |||||
| 99.01 | 98.44 | 98.80 | No | |||||
| 99.22 | 98.63 | 98.96 | Yes | |||||
| 95.71 | 95.86 | 95.74 | Yes | |||||
| 95.89 | 96.12 | 95.91 | Yes | |||||
| 96.21 | 96.18 | 96.10 | Yes | |||||
| _96.49_ | _96.37_ | _96.33_ | Yes | |||||
| 93.71 | 94.03 | 93.82 | No | |||||
| 94.21 | 94.08 | 94.02 | Yes | |||||
| 94.12 | 93.87 | 93.97 | Yes | |||||
| 94.45 | 94.28 | 94.36 | Yes | |||||
| 96.00 | 95.46 | 95.73 | Yes | |||||
| 95.82 | 95.41 | 95.63 | Yes | |||||
| 97.52 | 96.83 | 96.13 | Yes | |||||
| 98.36 | 98.52 | 98.71 | Yes |
Table V. Impact of Input Modalities of Public Dataset
| Data Category | Private Dataset | Cause | ||||
| Image | Factor | Borderline | ACC | PRE | F1 | |
| - | - | - | - | |||
| 95.21 | _95.38_ | _95.29_ | Yes | |||
| _95.42_ | 94.76 | 94.93 | No | |||
| 98.96 | 99.03 | 98.91 | Yes | |||
| 92.84 | 96.12 | 94.45 | Yes | |||
| 93.67 | 94.75 | 94.21 | Yes | |||
| 94.91 | 93.43 | 94.16 | Yes | |||
| 97.38 | 97.01 | 97.21 | Yes |
Table VI. Impact of Input Modalities of Private Dataset
Private Dataset: As shown in Table 2 and VI, our dataset includes a Borderline category. Compared with the public dataset (12×2 variables), Table 2 contains approximately three times more parameter names, with substantially higher feature dimensionality (32×3 for single-eye and 48×3 for both-eye data). In the comparative experiments, Borderline cases are treated as normal by default, whereas in this analysis they are considered a separate third class, resulting in an increase in variable dimensionality by factors of 4 or 2.7, respectively. As feature dimensionality increases, computational cost rises while classification accuracy degrades. Incorporating image features mitigates this effect by improving model fitting, leading to higher final accuracy.
| Category | Biomarker | |||
| RNFL | AR (m) | 97 | 91 | 6 |
| RNFL | SR (m) | 94 | 92 | 2 |
| RNFL | IR (m) | 99 | 89 | 10 |
| RNFL | I-ER (S−I) (m) | −5 | 3 | −8 |
| ONH | A-O | 0.28 | 0.51 | −0.23 |
| ONH | V-O | 0.46 | 0.79 | −0.33 |
| ONH | H-O | 0.62 | 0.74 | −0.12 |
| ONH | RA () | 1.44 | 1.23 | 0.21 |
| ONH | DA () | 2.01 | 2.52 | −0.51 |
| ONH | CVO () | 0.043 | 0.299 | −0.256 |
| GCC | A-G (m) | 85 | 83 | 2 |
| GCC | S-G (m) | 81 | 83 | −2 |
| GCC | I-F (m) | 88 | 83 | 5 |
| GCC | I-EG (S−I) (m) | −7 | 0 | −7 |
| GCC | FLV | 0.87 | 0.47 | 0.40 |
| GCC | GLV | 10.76 | 12.50 | −1.74 |
Table 2. Sample from the UTSW Private Dataset
GlaBoost: As shown in Table VII, we evaluate the individual components of GlaBoost through a series of ablation studies. Different CNN architectures are compared for image feature extraction, key mBERT hyperparameters are examined for textual feature modeling, and the influence of XGBoost parameters on biomarker-based classification is analyzed. Due to the relative simplicity of the classification task, CNN backbones such as ResNet-101 already achieve near-saturated accuracy, with deeper architectures yielding marginal gains. Textual representations extracted by mBERT improve
| Model | Module / Function | Data Category: Image | Data Category: Words | Data Category: Factor | Judgment: Risk | Judgment: Sure | Public Dataset: ACC | Public Dataset: PRE | Public Dataset: F1 | Data Category: Image | Data Category: Factor | Private Dataset: ACC | Private Dataset: PRE | Private Dataset: F1 | Cause |
| CNN | + ResNet-18 | ✓ | ✗ | ✗ | ✗ | ✗ | 91.20 | 89.10 | 90.14 | ✓ | ✗ | 90.80 | 91.10 | 90.95 | No |
| + ResNet-34 | ✓ | ✗ | ✗ | ✗ | ✗ | 93.50 | 91.80 | 92.64 | ✓ | ✗ | 93.00 | 92.50 | 92.75 | No | |
| + ResNet-50 | ✓ | ✗ | ✗ | ✗ | ✗ | 94.80 | 93.60 | 94.19 | ✓ | ✗ | 94.50 | 94.10 | 94.30 | No | |
| + ResNet-101 | ✓ | ✗ | ✗ | ✗ | ✗ | 100.0 | 99.37 | 99.68 | ✓ | ✗ | 100.0 | 100.0 | 100.0 | No | |
| + ResNet-152 | ✓ | ✗ | ✗ | ✗ | ✗ | 100.0 | 100.0 | 100.0 | ✓ | ✗ | 100.0 | 100.0 | 100.0 | No | |
| + ResNet-200 | ✓ | ✗ | ✗ | ✗ | ✗ | 100.0 | 100.0 | 100.0 | ✓ | ✗ | 100.0 | 100.0 | 100.0 | No | |
| mBERT | + Embedding | ✗ | ✓ | ✓ | ✗ | ✗ | 87.79 | 88.31 | 88.05 | ✗ | ✓ | 87.90 | 87.46 | 87.68 | No |
| + Attention | ✗ | ✓ | ✓ | ✗ | ✗ | 91.26 | 89.83 | 90.54 | ✗ | ✓ | 90.77 | 91.02 | 90.89 | No | |
| + Dual Weight | ✗ | ✓ | ✓ | ✗ | ✗ | 90.38 | 88.96 | 89.66 | ✗ | ✓ | 89.50 | 89.93 | 89.71 | No | |
| XGBoost | + Default | ✗ | ✗ | ✓ | ✓ | ✓ | 93.91 | 94.27 | 94.09 | ✗ | ✓ | 94.02 | 93.88 | 93.95 | Yes |
| + Max_depth (10) | ✗ | ✗ | ✓ | ✓ | ✓ | 94.62 | 95.13 | 94.87 | ✗ | ✓ | 94.91 | 94.45 | 94.68 | Yes | |
| + Subsample (0.8) | ✗ | ✗ | ✓ | ✓ | ✓ | 94.35 | 93.89 | 94.12 | ✗ | ✓ | 94.41 | 94.18 | 94.29 | Yes | |
| + Random State | ✗ | ✗ | ✓ | ✓ | ✓ | 95.02 | 94.71 | 94.86 | ✗ | ✓ | 95.13 | 94.96 | 95.04 | Yes | |
| + Fusion | ✗ | ✗ | ✓ | ✓ | ✓ | 95.74 | 94.38 | 95.05 | ✗ | ✓ | 94.67 | 95.36 | 95.01 | Yes | |
| GlaBoost | + ResNet-152 | ✓ | ✗ | ✗ | ✗ | ✗ | 94.88 | 95.36 | 95.11 | ✓ | ✗ | 95.22 | 94.95 | 95.08 | No |
| + mBERT | ✓ | ✓ | ✓ | ✗ | ✗ | 93.42 | 92.85 | 93.13 | ✓ | ✓ | 93.05 | 93.28 | 93.17 | No | |
| + XGBoost | ✓ | ✓ | ✓ | ✓ | ✓ | 94.36 | 93.74 | 94.05 | ✓ | ✓ | 93.89 | 94.22 | 94.05 | Yes | |
| + All | ✓ | ✓ | ✓ | ✓ | ✓ | 98.36 | 98.52 | 98.71 | ✓ | ✓ | 98.96 | 99.03 | 98.91 | Yes |
Table VII. Ablation Study of the GlaBoost
with parameter tuning, leading to consistent performance enhancements. For XGBoost, increasing model depth and varying random seeds improve both accuracy and robustness as feature dimensionality grows. Finally, the selected components are integrated into the full GlaBoost framework and jointly optimized, resulting in the best overall performance.
Analysis
As shown in Table VIII, feature importance analysis highlights the cup_to_disc ratio and related attributes, including rim thinning and cup volume, as the most influential factors for glaucoma diagnosis across datasets. This is clinically well established, as an increased cup-to-disc (C/D) ratio reflects optic nerve damage due to retinal ganglion cell loss, with values above 0.6 or marked inter-eye asymmetry () raising strong suspicion of glaucoma [6, 20]. Similar diagnostic relevance is observed for the superior SRNFL. In addition, the included glaucoma risk assessment from the public dataset reflects human judgment that extends beyond a limited set of structured variables and implicitly incorporates visual cues not captured by existing annotations. This suggests that the non-image features in the public dataset are inherently incomplete.
| Public Dataset: Feature | Public Dataset: Importance | Public Dataset: Value | Private Dataset: Feature | Private Dataset: Importance | Private Dataset: Value |
| isnt_rule_followed | 0.5180 | True | Superior RNFL (SR) | 0.3424 | 91 - 96 |
| cup_to_disc_ratio | 0.2140 | False | Cup/Disc H. Ratio (H-O) | 0.3109 | 0.69 - 0.72 |
| rim_thinning | 0.2020 | True | Average GCC (A-G) | 0.1723 | 81 - 85 |
| glaucoma_risk_assessment | 0.0088 | very healthy | Cup Volume (CVO) | 0.1216 | 0.095 - 0.127 |
| rim (embedding) | 0.0132 | thin (word) | FLV | 0.0128 | 0.61 - 0.78 |
Table VIII. Statistics of Impact Factor (Top 5)
Conclusion
We presented the GlaBoost, a multimodal gradient boosting framework that fuses fundus images, structured ophthalmic biomarkers, and free-text clinical descriptions for glaucoma risk stratification. Ablation analysis shows that the structured and textual modalities together recover most of the diagnostic signal even without any clinician-derived input, while feature importance analysis identifies clinically established predictors, the cup-to-disc ratio, rim thinning, and the ISNT rule, as the dominant contributors, in line with ophthalmologic practice.
Beyond the present setting, several directions remain open: adopting ophthalmology-specific foundation models for image encoding, replacing the general-purpose text encoder with biomedical language models, calibrating output probabilities for stratified clinical decisions, and validating GlaBoost across multi-site external cohorts.
Acknowledgments
This research work is partially sponsored by NASA under Grant No. 80NSSC22K0144, UTSW under GMO No. 241213, and the National Institutes of Health (NIH) under Grant No. 1R01AG083179-01.
References
- [1]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
- [2]Meta AI. Llama 3 technical report. https://ai.meta.com/llama/, 2024. Accessed: August 2025.
- [3]Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
- [4]Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 785–794. ACM, 2016.DOI
- [5]Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder–decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
- [6]Dominic M. Choo, Harper Henderson, Mahija Gunjipalli, and Karanjit S. Kooner. Glaucoma and retinal vein occlusion, pages 11–38. Nova Science Publishers, Inc., February 2025.
- [7]Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
- [8]Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning (ICML), pages 2286–2296. PMLR, 2021.
- [9]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.arxiv.org/abs/1810.04805
- [10]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
- [11]Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179–211, 1990.
- [12]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
- [13]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.DOI
- [14]Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
- [15]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, 2016.arxiv.org/abs/1512.03385
- [16]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
- [17]David W. Hosmer and Stanley Lemeshow. Applied Logistic Regression. John Wiley & Sons, New York, 2nd edition, 2000.
- [18]Cheng Huang, Junhao Shen, Beichen Hu, Mohammad Ausaf Ali Haqqani, Tsengdar Lee, Karanjit Kooner, Ning Zhang, and Jia Zhang. Semantic and visual attention-driven multi-lstm network for automated clinical report generation. In Arash Shaban-Nejad, Martin Michalowski, and Simone Bianco, editors, AI for Health Equity and Fairness: Leveraging AI to Address Social Determinants of Health, volume 1164 of Studies in Computational Intelligence, pages 233–248. Springer, 2024.DOI
- [19]Cheng Huang, Weizheng Xie, Tsengdar J. Lee, Jui-Kai Wang, Karanjit Kooner, Ning Zhang, and Jia Zhang. Glagan: A generative unsupervised model for high-precision segmentation of retinal main vessels toward early detection of glaucoma, 2025.
- [20]Yifan Huang, Dmitry Plotnikov, Han Wang, et al. Gwas-by-subtraction reveals an iop-independent component of primary open angle glaucoma. Nature Communications, 15:8962, 2024.DOI
- [21]Liu Li, Mai Xu, Hanruo Liu, Yang Li, Xiaofei Wang, Lai Jiang, Zulin Wang, Xiang Fan, and Ningli Wang. A large-scale database and a cnn model for attention-based glaucoma detection. IEEE transactions on medical imaging, 39(2):413–424, 2019.
- [22]Liu Li, Mai Xu, Xiaofei Wang, Lai Jiang, and Hanruo Liu. Attention based glaucoma detection: A large-scale database and cnn model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10571–10580, 2019.
- [23]Mingchao Li, Kun Huang, Qiuzhuo Xu, Jiadong Yang, Yuhan Zhang, Zexuan Ji, Keren Xie, Songtao Yuan, Qinghuai Liu, and Qiang Chen. Octa-500: a retinal dataset for optical coherence tomography angiography study. Medical image analysis, 93:103092, 2024.DOI
- [24]Yihan Lin, Qian Tang, Hao Wang, Cheng Huang, Ekong Favour, Xiangxiang Wang, Xiao Feng, and Yongbin Yu. Attention enhanced network with semantic inspector for medical image report generation. In 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI), pages 242–249, 2023.DOI
- [25]Aixin Liu, Bei Feng, Bing Xue, Binxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.arxiv.org/abs/2412.19437
- [26]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.arxiv.org/abs/1907.11692
- [27]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101.arxiv.org/abs/1711.05101
- [28]Yan Luo, Muhammad Osama Khan, Yu Tian, Min Shi, Zehao Dou, Tobias Elze, Yi Fang, and Mengyu Wang. Fairvision: Equitable deep learning for eye disease screening via fair identity scaling, 2024.arxiv.org/abs/2310.02492
- [29]Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, Yi Fang, and Mengyu Wang. Fairclip: Harnessing fairness in vision-language learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2403.19949
- [30]Yan Luo, Min Shi, Yu Tian, Tobias Elze, and Mengyu Wang. Harvard glaucoma detection and progression: A multimodal multitask dataset and generalization-reinforced semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20471–20482, 2023.arxiv.org/abs/2308.13411
- [31]OpenAI. Gpt-4o: Openai’s multimodal flagship model. https://openai.com/index/gpt-4o, 2024. Model ID: gpt-4o (o3); Accessed: August 1, 2025.
- [32]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351, pages 234–241. Springer, 2015.arxiv.org/abs/1505.04597
- [33]David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533–536, 1986.
- [34]Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997.DOI
- [35]Yu Tian, Min Shi, Yan Luo, Ava Kouhana, Tobias Elze, and Mengyu Wang. Fairseg: A large-scale medical image segmentation dataset for fairness learning using segment anything model with fair error-bound scaling. In International Conference on Learning Representations (ICLR), 2024. Accepted as an ICLR 2024 conference paper.DOI
- [36]Yu Tian, Congcong Wen, Min Shi, Muhammad Muneeb Afzal, Hao Huang, Muhammad Osama Khan, Yan Luo, Yi Fang, and Mengyu Wang. Fairdomain: Achieving fairness in cross-domain medical image segmentation and classification. In European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2407.08813
- [37]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozier, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
- [38]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmin Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.arxiv.org/abs/2307.09288
- [39]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017.arxiv.org/abs/1706.03762