WorksComputer Science
Enhancing RAG-based Medical VQA with Gated Multimodal Representations
Abstract
Medical Vision-Language Models (Med-LVLMs) have shown strong performance in Medical Visual Question Answering (MVQA), yet they may still produce factually inconsistent responses. Retrieval-Augmented Generation (RAG) can mitigate this issue by incorporating external clinical evidence; however, retrieval is often guided by a single modality, potentially overlooking the joint semantics of medical images and clinical questions. We introduce a multimodal RAG framework that learns a joint query representation through a Gated Multimodal Unit, enabling evidence retrieval to leverage information from both modalities while keeping the underlying Med-LVLM frozen. We evaluate the proposed approach on benchmark datasets. Results show that the multimodal query representation consistently outperforms a reference Med-LVLM and remains competitive with state-of-the-art methods. Our findings demonstrate the potential of multimodal query representations for more effective evidence retrieval in Medical VQA.
Introduction
Progress in Medical Large Vision-Language Models (Med-LVLMs) has enabled promising interactive diagnostic applications [15]. However, these models may still produce factually inconsistent responses, particularly in tasks such as Medical Visual Question Answering (Medical VQA), report generation, and clinical decision support, where generated outputs may not align with medical knowledge or reference answers. Retrieval-Augmented Generation (RAG) can mitigate this issue by incorporating external clinical evidence into the inference process. In this context, traditional RAG combines a parametric model with a non-parametric retriever, typically using Maximum Inner Product Search, to select the top- relevant documents from a dense knowledge base [6]. However, in multimodal scenarios, retrieval is often guided by a single modality, even when the input includes both a medical image and a clinical question [15]. This may lead to misalignment of the evidence in the query, where the retrieved passages partially reflect the multimodal clinical context.
We propose a multimodal query representation for retrieval in Medical VQA. Our approach incorporates a Gated Multimodal Unit (GMU) [1] to combine visual and textual features, producing a joint image-question embedding. The learned representation is used as the query-vector to retrieve the top- relevant textual passages from a vector database, while keeping the Med-LVLM frozen. By focusing on retrieval representation, our method aims to better align the multimodal input with the retrieved clinical evidence. Experimental results on benchmark Medical VQA datasets show that the proposed multimodal retrieval representation boosts the performance of a reference Med-LVLM, and remains competitive with state-of-the-art models. Our contributions are as follows:
Introducing a novel methodology for learning multimodal query representations in a RAG framework by integrating a GMU into the retrieval stage, explicitly modeling the interaction between the involved modalities.
Proposing four joint representation learning loss-function configurations that emphasize different aspects of the available information.
Performing an extensive experimental evaluation of the proposed methodology on benchmark Medical VQA datasets, showing its competitiveness in challenging scenarios.
Related work
RULE [16] and MMed-RAG [15] retrieve textual reports using mainly visual information. RULE uses a CLIP-based image-text retriever and a factuality risk-control strategy to select the number of contexts ; while MMed-RAG uses domain-aware retrievers and adaptive context selection. Both further apply preference tuning to reduce over-reliance on retrieved information. FactMM-RAG [13] follows a similar visual-centric formulation for report generation: the approach employs MARVEL [19] to represent medical images and reports in a shared embedding space. Thus, although its retriever is trained using paired visual and textual information, the query used at inference remains based on the visual modality. RAMM [17] fuses the image-question representation with retrieved image-text pairs through a retrieval-attention module, and MOTOR [11] uses grounded captions and optimal transport during re-ranking. MIRA [14] weights modalities dynamically over a multimodal index, and MM-RAG-MedVQA [12] builds a query from the image and a generated caption, but leaves the question out of the retrieval representation, incorporating it only at generation time. Hence, multimodal fusion has been explored for processing evidence, but not as a joint representation to be used directly as the initial retrieval query.
Proposed method
The proposed framework is depicted in Figure 1, it comprises five steps grouped into two stages. Each step is described in the remainder of this section.

Figure 1. Overview of the proposed framework for Medical VQA.
GMU Training (Multimodal Fusion). To learn multimodal query representations, we use training samples comprising a medical image, its associated clinical
question, and the corresponding medical report. Image–question pairs are the input to the GMU [1], while the reports serve as supervision in the Report-Centric configuration and, later, as the contexts of the knowledge base. Visual and textual embeddings, and , are extracted with frozen encoders under two setups: (i) a dual-encoder with MobileNetV2 [10] for images and PubMedBERT [4] for text; and (ii) BiomedCLIP [18]. Given these embeddings, the GMU first projects each modality into a shared latent space of dimension :
where , , , and are learnable parameters. A gate then weights the contribution of each modality:
The joint representation is
is an adaptive, per-dimension mixture of visual and textual information. We train the GMU with a contrastive objective that pulls toward its positive and pushes it away from negatives, under four configurations that differ in the alignment objective. This model maps every image-question pair to its own multimodal embedding , i.e., the contribution of each modality is instance-specific. Four variants to selecting positive samples () in a InfoNCE loss [9], function were designed:
Baseline. The joint representation is learned directly from the multimodal fusion, without an additional semantic anchor.
Fusion-Image (F-I). The positive is the embedding of the corresponding medical image; negatives are non-matching images.
Fusion-Question (F-Q). The positive is the embedding of the associated clinical question; negatives are unrelated questions.
Fusion-Report (F-R). The positive is the embedding of the clinical report associated with the image–question pair; negatives are non-matching reports.
Knowledge Indexing (per configuration). The Knowledge Base is built offline from the medical reports the training data, where each report is a retrievable context. Since the query lives in the joint space learned by the GMU, the contexts must be embedded in that same space for their cosine similarity to be meaningful. Therefore, for each configuration, every report is encoded with the text encoder used for the questions (PubMedBERT or BiomedCLIP), yielding , and passed through the corresponding trained GMU with a zero vector as the visual input (). The resulting -dimensional embeddings are stored in a FAISS [3] index, so that one index is built per configuration. At inference time, the query is embedded with the same GMU and the top- most similar contexts are retrieved.
Test Example Encoding. At inference time, each example of the test split is converted into a multimodal query. The image and the question are encoded with the same frozen encoders used during training, and the resulting embeddings are fed to the trained GMU, Eq. (3).
Retrieval. Given the multimodal query q of a test example, we search the knowledge index of the corresponding configuration by computing its cosine similarity against every context embedding : and retrieve the contexts with the highest scores, where determines the number of medical reports provided as additional evidence to the Med-LVLM. Since the amount of retrieved information may influence answer generation, we evaluate different values of to analyze their impact on performance, using as the default setting. The text of the retrieved reports is then passed to the generation stage.
Generation & Evaluation. We use a simple in-context prompt for answering with LLaVA-Med-1.5 [7], kept frozen (no fine-tuning): the top- retrieved contexts are combined with the image and the question in the prompt.
Experiments and results
Experimental settings
The GMU module is trained and evaluated using three publicly available medical imaging datasets: MIMIC-CXR [5], IU-XRAY [2], and Harvard-FairVLMed [8], see Table 1. All experiments were performed in Google Colaboratory Pro using T4, L4, and A100 GPUs. We used LLaVA-Med-1.5 7B [7] as the Med-LVLM. We evaluated two encoders: (i) a dual-encoder setup, with MobileNetV2 [10] for images and PubMedBERT [4] for text; and (ii) BiomedCLIP [18], which maps visual and textual inputs into a shared embedding space. For retrieval, textual report embeddings were indexed offline using FAISS [3].
Table 1. Datasets and binary VQA splits used in this work.
Results
Table 2 presents the experimental results in terms of Accuracy, F1-score, and AUROC (0–100 scale), considering two encoder settings and the four GMU training configurations. Fusion-Image, Fusion-Question, and Fusion-Report are abbreviated as F-I, F-Q, and F-R, respectively. We include the following references: LLaVA-Med-1.5 (the MedLVLM at the inference stage without RAG), two unimodal retrieval baselines, in which the query is built only from the clinical question (text-only) or only from the medical image (image-only) and the retrieval-only configurations of RULE [16] and MMed-RAG [15].
| Method | IU-Xray Acc | IU-Xray F1 | IU-Xray AUC | MIMIC-CXR Acc | MIMIC-CXR F1 | MIMIC-CXR AUC | FairVLMed Acc | FairVLMed F1 | FairVLMed AUC |
| GMU with dual encoder | |||||||||
| Baseline | 62.79 | 55.78 | 70.92 | 73.79 | 81.86 | 67.12 | 72.11 | 81.80 | 70.90 |
| F-R | 66.33 | 59.33 | 74.64 | 74.94 | 81.97 | 69.66 | 72.39 | 82.24 | 68.08 |
| F-Q | 61.74 | 55.78 | 71.00 | 76.73 | 82.48 | 73.05 | 78.79 | 86.55 | 78.34 |
| F-I | 59.28 | 54.84 | 70.03 | 76.01 | 82.51 | 71.23 | 87.58 | 92.90 | 70.25 |
| GMU with BiomedCLIP | |||||||||
| Baseline | 76.06 | 65.31 | 85.56 | 82.61 | 86.41 | 88.32 | 90.72 | 94.80 | 71.26 |
| F-R | 75.47 | 62.88 | 85.59 | 82.41 | 86.24 | 88.15 | 88.47 | 93.26 | 89.30 |
| F-Q | 72.75 | 62.88 | 85.42 | 82.41 | 86.52 | 87.62 | 87.22 | 92.40 | 89.50 |
| F-I | 62.63 | 73.80 | 73.64 | 79.98 | 85.14 | 73.32 | 90.11 | 94.48 | 73.32 |
| Baselines and Prior Work | |||||||||
| LLaVA-Med-1.5 | 75.47 | 64.04 | 67.46 | 75.79 | 80.49 | 68.84 | 63.03 | 74.11 | 63.05 |
| Text-only retrieval | 78.45 | 69.00 | 89.31 | 81.90 | 85.77 | 87.24 | 70.46 | 80.71 | 70.72 |
| Image-only retrieval | 86.83 | 74.12 | 91.05 | 73.25 | 76.55 | 83.32 | 49.86 | 61.26 | 67.60 |
| RULE{sup}* | 87.84 | 78.00 | 85.78 | 83.92 | 87.49 | 83.44 | 87.12 | 92.89 | 77.08 |
| MMed-RAG{sup}* | 89.54 | 80.72 | 87.13 | 83.57 | 88.49 | 85.08 | 87.94 | 92.78 | 80.81 |
Table 2. Results on closed-ended Medical VQA. F-R, F-Q and F-I denote the GMU configurations. Results marked with * correspond to the retrieval-only configurations of RULE [16] and MMed-RAG [15], i.e., retrieved contexts added to the Med-LVLM prompt without preference fine-tuning, taken from the original papers. Best results per column are in bold.
BiomedCLIP outperforms the dual encoder in every configuration of our method, with average gains of +3.9 to +14.1 points (the largest in AUROC on MIMIC-CXR). Also, the encoder directly influences the contribution of the loss configuration: with the dual encoder, the anchored variants often improve the AUROC of the Baseline, whereas with BiomedCLIP the Baseline is already strong. This may be due to the fact that BiomedCLIP’s image and text embeddings are already aligned when fed to the GMU, while the GMU must also bridge the modality gap when using the dual (separate-modalities) encoder. After the Baseline configuration, F-R and F-Q perform similarly, while F-I was the configuration with lowest performance, still obtaining AUC above 73.
Compared with LLaVA-Med-1.5 without retrieval (i.e., same method but no RAG), the GMU variants improve most metrics on MIMIC-CXR and FairVLMed, with gains of up to 27.7 points in Accuracy and 26.5 in AUROC on FairVLMed; on IU-Xray, improvements are mostly limited to AUROC. Improvements are also reported for the baselines implementing unimodal RAG (rows 10-11): considerable improvements are observed for FairVLMed, moderate ones for MIMIC-CXR, and lower performance than image-only RAG was obtained for IU-Xray.
These results clearly illustrate the benefits of using the learned multimodal representations for RAG. Compared with RULE and MMed-RAG, our method obtains mixed results: outperforming the references in some metrics and datasets, and achieving lower performance in others. Although the results are not conclusive, they indicate that our method achieves performance comparable to that of these state-of-the-art approaches. Further investigation is needed to identify the conditions under which our method offers consistent improvements.
Conclusions
We introduced a method for learning multimodal representations for supporting RAG in Medical VQA. A GMU learns to weight the contribution of visual and textual information in the query, enhancing the RAG stage by providing useful information about the interaction of modalities. The experimental evaluation shows the competitiveness of our proposed methodology, yet further work is needed to characterize the configurations under which improvements are consistent.
Table 3 shows an ablation study on the value of . According to this table, generally results in the best performance, while often reduces F1-score and AUROC. This suggests that retrieving multiple reports provides richer clinical evidence for the frozen Med-LVLM.
| Model | IU-Xray Acc | F1 | AUC | MIMIC-CXR Acc | F1 | AUC | FairVLMed Acc | F1 | AUC | |
| Baseline | 1 | 62.39 | 56.44 | 82.74 | 80.36 | 85.75 | 86.85 | 89.06 | 93.82 | 57.48 |
| 3 | 75.16 | 64.79 | 85.45 | 82.75 | 86.60 | 88.16 | 90.18 | 94.54 | 68.33 | |
| 5 | 76.06 | 65.31 | 85.56 | 82.61 | 86.41 | 88.32 | 90.72 | 94.80 | 71.26 | |
| F-R | 1 | 62.08 | 56.28 | 82.64 | 80.19 | 85.64 | 86.74 | 89.45 | 94.02 | 89.23 |
| 3 | 74.81 | 64.31 | 85.51 | 82.49 | 86.37 | 87.95 | 85.93 | 91.55 | 89.20 | |
| 5 | 75.47 | 62.88 | 85.59 | 82.41 | 86.24 | 88.15 | 88.47 | 93.26 | 89.30 | |
| F-Q | 1 | 53.60 | 51.77 | 81.84 | 76.61 | 83.71 | 85.81 | 88.84 | 93.85 | 85.82 |
| 3 | 74.42 | 64.11 | 85.01 | 82.55 | 86.49 | 87.64 | 87.15 | 92.40 | 89.17 | |
| 5 | 72.75 | 62.88 | 85.42 | 82.41 | 86.52 | 87.62 | 87.22 | 92.40 | 89.50 | |
| F-I | 1 | 59.47 | 70.51 | 77.97 | 75.05 | 81.27 | 71.16 | 87.62 | 93.29 | 55.64 |
| 3 | 60.45 | 71.57 | 74.87 | 75.23 | 81.33 | 71.49 | 89.97 | 94.43 | 67.55 | |
| 5 | 62.63 | 73.80 | 73.64 | 79.98 | 85.14 | 73.32 | 90.11 | 94.48 | 73.32 |
Table 3. Ablation study for the number of retrieved contexts .
References
- [1]Arevalo, J., Solorio, T., Montes-y Gómez, M., González, F.A.: Gated multimodal units for information fusion. In: Proc. of International Conference on Learning Representations (ICLR) (2017), https://openreview.net/forum?id=HJxK5Zsxx
- [2]Demner-Fushman, D., Kohli, M., Rosenman, M., Shooshan, S., Rodriguez, L., Antani, S., Thoma, G., McDonald, C.: Preparing a collection of radiology examinations for distribution and retrieval. In: Proc. of AMIA. pp. 361–370 (2016)
- [3]Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library (2024)
- [4]Gu et al., Yu: Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthcare 3(1) (Oct 2021)
- [5]Johnson, A., Pollard, T., Mark, R., Berkowitz, S., Horng, S.: MIMIC-CXR Database. PhysioNet (Jul 2024)
- [6]Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: Proc. of NeurIPS. vol. 33, pp. 9459–9474 (2020)
- [7]Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In: Proc. of NeurIPS. vol. 36, pp. 28541–28564 (2023), article 1240
- [8]Luo, Y., Shi, M., Khan, M.O., Afzal, M.M., Huang, H., Yuan, S., Tian, Y., Song, L., Kouhana, A., Elze, T., et al.: Fairclip: Harnessing fairness in vision-language learning. arXiv preprint arXiv:2403.19949 (2024)
- [9]van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018), https://arxiv.org/abs/1807.03748
- [10]Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proc. of CVPR. pp. 4510–4520 (2018)arxiv.org/abs/1801.04381
- [11]Shaaban, M.A., Saleem, T.J., Papineni, V.R.K., Yaqub, M.: Motor: Multimodal optimal transport via grounded retrieval in medical visual question answering. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2025. Lecture Notes in Computer Science, vol. 15965, pp. 459–469. Springer Nature Switzerland (2025). https://doi.org/10.1007/978-3-032-04978-0_44
- [12]Shaaban, M.A., Zarei, M.R., Khan, A., Akkasi, A., Yakub, M., Komeili, M.: Towards multimodal retrieval-augmented generation for medical visual question answering. Research Square preprint (2025). https://doi.org/10.21203/rs.3.rs-7752202/v1
- [13]Sun, L., Zhao, J., Han, M., Xiong, C.: Fact-aware multimodal retrieval augmentation for accurate medical radiology report generation. In: Proc. 2025 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) (2025)
- [14]Wang, J., Ashraf, T., Han, Z., Laaksonen, J., Anwer, R.M.: Mira: A novel framework for fusing modalities in medical rag. In: Proceedings of the 33rd ACM International Conference on Multimedia. p. 6307–6315. MM ’25, Association for Computing Machinery, New York, NY, USA (2025). https://doi.org/10.1145/3746027.3755760, https://doi.org/10.1145/3746027.3755760
- [15]Xia, P., Li, D., Wang, Z., Deng, C., Peng, X., Ma, T., Liu, J., Wang, Y., Zhang, S.: Mmed-rag: Multimodal retrieval-augmented generation for medical vision-language tasks. In: Proc. of the International Conference on Learning Representations (ICLR 2025) (2025)
- [16]Xia, P., Zhu, K., Li, H., Zhu, H., Li, Y., Li, G., Zhang, L., Yao, H.: Rule: Reliable multimodal rag for factuality in medical vision-language models. In: Proc. of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 1081–1093. Association for Computational Linguistics, Miami, Florida, USA (2024)arxiv.org/abs/2407.05131
- [17]Yuan, Z., Jin, Q., Tan, C., Zhao, Z., Yuan, H., Huang, F., Huang, S.: Ramm: Retrieval-augmented biomedical visual question answering with multimodal pre-training. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 547–556. Association for Computing Machinery (2023). https://doi.org/10.1145/3588178.3611830
- [18]Zhang, et al. Sheng: A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI 2(1), AIoa2400640 (2025)DOI
- [19]Zhou, T., Mei, S., Li, X., Liu, Z., Xiong, C., Liu, Z., Gu, Y., Yu, G.: MARVEL: Unlocking the multi-modal capability of dense retrieval via visual module plugin. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14608–14624. Association for Computational Linguistics, Bangkok, Thailand (Aug 2024). https://doi.org/10.18653/v1/2024.acl-long.783, https://aclanthology.org/2024.acl-long.783/