betaFeedback

WorksComputer Science

Multimodal AI in Healthcare: Review of Vision-Language Foundation Models for Real-World Medical Applications

Abstract

The emergence of foundation models has marked a transformative shift in AI, enabling robust generalization across diverse downstream tasks through putative zero-shot learning. Large Language Models and Vision Language Models have demonstrated strong capabilities in tasks such as image interpretation, report generation, and question answering by effectively learning from multimodal data-images paired with associated text often with minimal supervision. In the healthcare domain, this ability to align visual and textual information reduces the reliance on extensive manual annotations, as models can leverage existing clinical reports and imaging data to learn meaningful representations. This integration holds promise for improving diagnostic support, treatment planning, and overall patient care, even in data constrained settings. In this review, we provide a definitive taxonomy of the medical VLM landscape, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting edge frontiers of Dense Pixel Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning Incentivized (RL) architectures. We critically examine the "medical bottleneck" identifying the persistent challenges of data scarcity, the "hallucination" risks in generative diagnostics, the computational strain of 3D volumetric processing, and the lack of standardized, clinically grounded evaluation metrics.

Introduction

The landscape of artificial intelligence has undergone a fundamental shift with the emergence of Foundation Models (FMs), large-scale architectures, typically based on the Transformer paradigm, that are pre-trained on vast, heterogeneous datasets. Unlike traditional deep learning (DL) models that rely on task-specific supervision, FMs leverage self-supervised learning to develop robust, transferable representations. This allows them to excel in downstream tasks through zero-shot or few-shot learning, effectively navigating the "long tail" of rare conditions where labeled data is historically sparse [1, 2]. The scalability of these models enables the capture of emergent properties—capabilities that arise only at high computational and data scales—thereby reducing the prohibitive costs of manual annotation that have long bottlenecked AI deployment in specialized fields.

This surge in FMs was catalyzed by breakthroughs in Large Language Models (LLMs), such as GPT-4 [3] and LLaMA-3 [4]. These models demonstrated that massive-scale pre-training on natural language could yield sophisticated reasoning and context-awareness. However, human intelligence and professional domains, particularly medicine, are inherently multimodal. The integration of linguistic reasoning with visual perception has led to the development of Vision-Language Models (VLMs), which bridge the gap between structured textual knowledge and unstructured pixel data. By aligning these modalities in a shared latent space, VLMs facilitate a more holistic understanding of complex environments.

In the clinical domain, the utility of VLMs is transformative. Medical diagnostics rarely rely on a single data source; rather, they require the synthesis of medical imaging (e.g., MRI, CT, X-ray) and clinical narratives (e.g., radiology reports, electronic health records). Specialized medical VLMs, such as Med-Flamingo ([5]) and LLaVA-Med ([6]), utilize contrastive learning or generative prefix-tuning to achieve semantic alignment between imaging features and medical terminology. This enables high-order tasks such as automated medical report generation, visual question answering (VQA) for clinical decision support, and cross-modal retrieval. By leveraging pre-trained knowledge from general-domain datasets and refining it with domain-specific corpora, these models address the critical challenge of "data silos" and expert-label scarcity in healthcare ([7]).

This review provides a systematic taxonomy of recent advancements in medical vision-language foundation models. We begin by dissecting the underlying architectures—categorizing them by their fusion strategies and backbone structures. Furthermore, we analyze key clinical applications, including diagnostic classification, semantic segmentation, and interactive VQA. Finally, we address the prevailing challenges including hallucinatoy outputs, data privacy, and the “domain gap” between natural and medical images.

Vision-Language Multimodal Foundation Models

VLMs represent a significant advancement in the integration of computer vision and natural language processing, enabling AI systems to perform a wide range of vision tasks with unprecedented accuracy in the medical domain. These models are designed to simultaneously comprehend and process visual and textual data, bridging the gap between visual and linguistic understanding. By combining image- and text-encoders, VLMs can extract meaningful features from medical images and associate them with relevant clinical information.

Figure 1 presents a generic block diagram of a foundation model, illustrating how different input data modalities are processed by multiple encoders, and their encoded representations are fused with text tokens before being passed into a large language model backbone. This integrated design governs the foundation models’ outputs, enabling a wide range of applications such as visual question answering, image captioning, and medical diagnosis. In this section, we delve into notable research efforts aimed at integrating different types of foundation models into medical applications, showcasing their transformative potential in healthcare. We specifically discuss recent advancements categorized based on the architectures of foundation models, each bringing unique capabilities and challenges to the field.

Figure 1. Figure 1: Overview of the foundation model pipeline in the medical domain. Step 1 involves building a multimodal dataset by integrating visual and language data. In Step 2, this dataset is used to fine-tune a foundation model capable of processing both modalities. Step 3 illustrates the various medical downstream tasks, including feature classification, segmentation, report generation, and question answering, on which the foundation model can be applied.

Figure 2 illustrates the temporal trends across different model categories, highlighting the growth in model size (in terms of parameters), the scale of training data, and the evolution of dataset availability over time. It also visualizes the distribution and size of datasets associated with each architectural category. In addition, Table 1 provide a comprehensive overview of the foundation models developed within each category, detailing their target backbones and supported tasks, including classification, segmentation, report generation, and question answering.

Evolution of foundation models in medicine, dataset size distribution, and dataset releases over years

Figure 2. A. Evolution of foundation models over time, showing the relationship between the progress and model size. The size of each data point indicates the training dataset size, demonstrating the varying computational and data requirements across different models. B. Distribution of dataset sizes across different foundation model architectures, highlighting the substantial variation in them. C. Dataset releases over the years across architectures, revealing that considerable datasets have been released in recent years.

ArchitectureModel (Year)ParamsBackbones (Vision / Language)Training StatusTasks
ContrastiveSimVLM (2021)∼86MViT / Transformer LMAdaptedC,R,Q
CheXzero (2022)∼120MViT-B/16 / TransformerAdapted (CLIP)C
BiomedCLIP (2023)∼120MViT-L/16 / PubMedBERTAdapted (CLIP)C,S,R,Q
BioViL-T (2023)∼110MResNet-50 + ViT / CXR-BERTMixed (Scratch Text)C
PMC-CLIP (2023)∼120MViT-B/16 / PubMedBERTAdapted (CLIP)C,Q
MedCLIP (2024)∼120MSwin-T or ResNet / Domain BERTAdaptedC,S,R,Q
UniMed-CLIP (2025)∼120MMetaCLIP ViT-B / BioMed-BERTAdaptedC,S,R,Q
GenerativeMedical-VisualBERT (2019)∼110MCNN / VisualBERTAdaptedC,Q
MedGPT (2021)∼124MTransformer LMScratch-trainedQ
MedPaLM2 (2023)∼340BMed-PaLM / PaLMAdaptedC,R,Q
Med-Flamingo (2023)∼80BFlamingo Vision Encoder / LLMAdaptedC,R,Q
XrayGPT (2023)∼150MMedCLIP / VicunaAdapted EncodersC,Q
BiomedGPT (2023)∼182MBERT-style / GPT-styleAdaptedC,R,Q
LLaVA-Med (2024)∼13BCLIP ViT-L/14 / Vicuna-7BAdapted (LLaVA)Q
Med-PaLM M (2024)∼562BViT / PaLMAdapted (PaLM)C,S,R,Q
LLaVA-Rad (2024)∼7BBiomedCLIP-CXR / VicunaAdaptedR,Q
RadFM (2024)∼7BSwin Transformer / LLaMAAdaptedR,Q
LAMM-Med (2024)∼7BCLIP ViT-L / LLaMA-2AdaptedC,R,Q
GMAI-VLM (2024)∼7BViT / LLaMA-based LLMAdaptedC,R,Q
SigPhi-Med (2025)4.2BCLIP / Phi-2AdaptedC,S,R,Q
OmniV-Med (2025)∼7BRoPE-SigLIP / LLM backboneAdaptedC,R,Q
VILA-M3 (2025)∼8B–13BViT / LLaMA-3AdaptedC,R,Q
Dense GroundingSAM-MED 2D (2023)∼636MSAM ViT Encoder / N/AAdapted (SAM)S
MedSAM (2024)∼636MViT / N/AAdapted (SAM)S
VISTA3D (2025)–SegResNet / TransformerMixed (Trained)S
MedRegA (2025)∼7BInternVL variant / LLMAdaptedC,S,R,Q
Expert/MoEMed-MoE (2024)∼3.6BCLIP ViT-L/14 / Phi-2Adapted (LLaVA)C,S,R,Q
MedPLIB (2025)∼12BViT-L/336 + ViT / MoE LLMAdapted (CLIP/SAM)C,S,R,Q
Med3DVLM (2026)∼7.6BDCFormer / ClinicalBERTAdaptedC,S,R,Q
Reasoning/RLMedVLM-R1 (2025)∼2BQwen2-VL-2B (Unified)Adapted (RL)Q
Med-R1 (2026)∼7BQwen2-VL (Unified)Adapted (RL)Q
MediX-R1 (2026)∼8B–30BBaseline MLLM (Unified)Adapted (RL)C,R,Q
Legacy/VisionSwinUNETR (2022)∼27MSwin + UNET / N/AScratch-trainedS
Clinical-BERT (2022)∼110MBERT / N/AAdaptedQ
RETFound (2023)∼86MViT-Large / N/AScratch (SSL)C

Table 1. Overview of medical foundation models grouped by architectural paradigm and ordered chronologically within each category. The table summarizes model scale, backbone architectures, training strategy, and supported downstream tasks including classification (C), segmentation (S), report generation (R), and visual question answering (Q).

The Contrastive Alignment Paradigm

The contrastive architectural paradigm, popularized by the Contrastive Language-Image Pre-training (CLIP) framework [8], focuses on establishing a shared latent space where visual and textual representations are semantically aligned. The core mechanism involves a dual-encoder architecture that employs a contrastive loss function to maximize the cosine similarity between matched image-text pairs while minimizing the similarity for all other mismatched pairs in a given training batch. These models are highly prized for their zero-shot capabilities, allowing them to classify pathologies or retrieve relevant clinical cases based purely on natural language descriptions without requiring fine-tuning on labeled examples for every new condition.

Dual-Encoder Foundations and Adaptation Strategies

BiomedCLIP stands as a prominent example of the contrastive paradigm specifically tailored for the biomedical domain. Released by Microsoft Research in 2023, BiomedCLIP [9] addresses the fundamental “domain gap” that renders web-trained VLMs ineffective in clinical settings. While the original CLIP utilizes a general-domain GPT-2 text encoder, BiomedCLIP is adapted from the CLIP framework by replacing the language backbone with PubMedBERT [10]. PubMedBERT is a language model pre-trained on the vast corpus of biomedical literature in PubMed, enabling the text encoder to natively understand clinical nomenclature, anatomical relations, and pathological nuances. For its vision backbone, BiomedCLIP utilizes a Vision Transformer [11] (specifically the ViT-L/16 architecture) adapted to handle higher image resolutions (up to 336 pixels) and captures fine-grained visual details essential for medical diagnosis. BiomedCLIP was not trained from scratch in the traditional sense but is an advanced adaptation of the CLIP approach, pre-trained on the PMC-15M dataset [12], a public collection of 15 million image-text pairs extracted from 4.4 million scientific articles.

MedCLIP [13] further refines the contrastive paradigm by addressing the “false negative” problem inherent in standard CLIP training. In clinical datasets, reports from different patients may describe identical findings (e.g., "normal chest radiograph"), yet a standard contrastive loss would treat an image from Patient A and a report from Patient B as a negative pair even if they are semantically identical. MedCLIP utilizes a dual-encoder architecture with Swin Transformer [14] or ResNet [15] backbones for vision and domain-specific BERT for text, adapted from pre-trained checkpoints. The model’s defining innovation is the decoupling of images and texts for pre-training, which scales usable data in a combinatorial manner, and the implementation of a semantic matching loss that leverages medical ontologies like UMLS to soft-label pairs.

Specialized Radiology and Temporal Contrastive Variants

CheXzero [16] represents a highly specialized application of contrastive learning for chest radiograph interpretation. It adopts the CLIP architecture using a ViT-B/16 vision backbone and a Transformer-based text encoder. CheXzero is an adaptation of general-domain CLIP weights, fine-tuned on approximately 200,000 pairs of chest X-ray images and radiology reports from the MIMIC-CXR [17] dataset. This adaptation strategy allowed the model to match the diagnostic accuracy of human radiologists in a zero-shot multi-label classification setting, demonstrating that explicit labels are unnecessary when high-quality clinical reports are available for self-supervised alignment.

BioViL-T [18] introduces temporal awareness into the contrastive paradigm, recognizing that radiologists’ workflows often involve comparing current scans with historical baselines. The model utilizes a hybrid vision backbone composed of a ResNet-50 [15] for initial feature extraction from images at each time point and a Vision Transformer to aggregate these features across the temporal dimension. Its text backbone is CXR-BERT-general [19], which is first pre-trained from scratch via masked language modeling on PubMed abstracts and clinical notes and then continually pre-trained for multimodal alignment. This model is essentially an adaptation of BERT and ResNet/ViT components, explicitly accounting for prior images and reports during both training and fine-tuning to achieve state-of-the-art performance on progression classification and phrase grounding.

The Generative Multimodal Paradigm

The generative architectural paradigm, or the Multimodal Large Language Model (MLLM) framework, integrates high-capacity vision encoders with power-

ful large language model (LLM) decoders to facilitate interactive clinical reasoning and open-ended conversation. Unlike contrastive models that produce simple similarity scores, generative Med-VLMs utilize an “encoder-connector-decoder" structure where visual tokens are projected into the embedding space of the LLM, allowing the model to “see" images as part of its prompt stream. This facilitates complex diagnostic assistance, structured report generation, and multi-turn visual dialogues that ground LLM logic in specific radiologic or pathologic findings.

LLaVA-based Architectures and Curriculum Learning

LLaVA-Med [6] is the seminal generative model for biomedicine, representing an adaptation of the general-domain Large Language and Vision Assistant (LLaVA). It utilizes a CLIP ViT-L/14 vision backbone to extract patch-level embeddings and a 7B-parameter LLM backbone, typically Vicuna-7B [20] or LLaMA [4]. LLaVA-Med employs a two-stage curriculum learning method: Stage 1 involves concept alignment by training a lightweight linear projection layer while keeping the vision encoder and LLM frozen; Stage 2 involves full-blown instruction-tuning of the LLM and the projector using GPT-4 generated multimodal dialogues. This adaptation process allows LLaVA-Med to transition from a general-purpose model to a specialized medical visual chatbot in less than 15 hours of training on a single 8-GPU node. Further architectural diversification is seen in models like LLaVA-Rad [21] and CXR-LLaVA [22]. LLaVA-Rad is adapted by integrating a domain-specific vision encoder, BiomedCLIP-CXR (pre-trained on 697,000 chest X-rays), with a Vicuna-7B-v1.5 language backbone using an MLP projector. In contrast, CXR-LLaVA’s creators developed a custom image encoder (ViT-L/16) from scratch to handle grayscale chest radiograph textures specifically, which is then integrated with the LLaMA-2 7B language model through a linear multimodal projector that maps 1024-dimensional tokens into a 4096-dimensional space.

Modular Generative Assistants and Multi-Agent Frameworks

XrayGPT [23] exemplifies a modular adaptation strategy by aligning a pre-trained specialized medical vision encoder (MedCLIP) with a fine-tuned medical LLM (Vicuna) using a simple linear transformation layer. The model focuses on chest radiograph summarization, leveraging approximately 217,000 interactive summaries generated from radiology reports to ground its conversational semantics.

Med-PaLM M [24], developed by Google, is a large-scale generalist multimodal system built upon the Pathways Language Model (PaLM) architecture. It integrates visual information through two Vision Transformer (ViT) models of different parameter sizes—pre-trained on a classification dataset of 4 billion images—and utilizes the PaLM-E framework to handle interleaved sequences of multimodal inputs, including clinical language, imaging, and genomics, using the same set of model weights. Med-PaLM 2 [26] further refines this by leveraging base model improvements (PaLM 2) and medical domain fine-tuning to achieve excellent-level performance on USMLE-style questions and clinical topic datasets. SigPhi-Med [27] proposes a lightweight vision–language assistant for biomedical applications by replacing large language models with a smaller language backbone utilizing the Phi-2 backbone for vision.

The Dense Grounding & Pixel-Level Paradigm

The Promptable Foundation Models

As Med-VLMs advance, there is a shift from holistic image-level understanding toward fine-grained pixel-level perception and grounding. This architectural paradigm enables “promptable" systems where models can respond to specific coordinates, bounding boxes, or textual references to identify anatomical regions and lesions. These models often utilize specialized mask decoders or region-aware encoders to bridge the gap between abstract diagnostic language and the spatial reality of clinical scans.

MedSAM [28] is a fundamental adaptation of the Segment Anything Model (SAM) [29] for the medical domain. It retains the original ViT encoder and mask decoder architecture but is fine-tuned on a colossal dataset of over 1.5 million image-mask pairs covering 10 imaging modalities and 30 cancer types. This adaptation allows MedSAM to delineate anatomical structures and pathological findings across diverse modalities—from ultrasound to histopathology—with minimal user prompting. VISTA3D [30] is a unified foundation model designed specifically for 3D medical imaging segmentation, addressing the limitations of 2D slice-by-slice processing. Its vision backbone is based on SegResNet, integrated with a transformer-based prompt encoder that accepts class indices or 3D point clicks. VISTA3D utilizes a novel 3D supervoxel method to distill knowledge from 2D pre-trained backbones (like SAM) into a 3D volumetric context. The model is trained on 11,454 CT volumes using a multi-stage recipe that systematically trains automatic segmentation for 127 supported classes and zero-shot interactive segmentation for novel structures.

Multi-Granular Pixel Grounding and Region-Aware Assistant

MedPLIB [31] introduces an end-to-end MLLM architecture capable of pixel-level visual grounding and reasoning. Its visual encoder is a dual-stream design: a CLIP ViT-L/336 vision tower for semantic features and a Visual Transformer with medical adapter layers (from SAM-Med2D) for pixel-level features. MedPLIB integrates these features with a Mixture-of-Experts (MoE) LLM that employs specialized experts for VQA and pixel grounding tasks. The model supports arbitrary pixel-level prompts—points, bounding boxes, and free-form shapes—and introduces a “" token to identify features necessary for generating segmentation masks via a specialized mask decoder. MedRegA [32] and Anatomy-VLM [33] focus on region-aware alignment to mimic human-centric diagnostic workflows. MedRegA is a bilingual generalist model adapted from powerful general-domain backbones (e.g., InternVL), fine-tuned on the MedRegInstruct [32] dataset to handle tasks like grounded report generation and text-to-region detection across 8 modalities. Anatomy-VLM employs an object detection encoder to localize anatomical structures and pathology features, which are then enriched with structured knowledge descriptions before being aligned via anatomy-wise contrastive learning.

The Expert-Centric & Sparse Computation Paradigm

The Modular Mixture-of-Experts Paradigm

To mitigate the computational costs of scaling Med-VLMs while enhancing their ability to handle heterogeneous clinical data, researchers have adopted Mixture-of-Experts (MoE) and modular architectural paradigms. These models replace traditional dense feed-forward networks with a set of sparsely activated experts, where a routing mechanism dynamically selects the most appropriate expert for a given input—such as an MRI expert for neuroimaging or a pathology expert for histology slides. Med-MoE [49] is a lightweight framework designed to reduce parameter activation by 30%–50% while maintaining performance comparable to much larger models. Utilizing small-scale LLM backbones like Phi-2 (2.7B) or StableLM (1.7B), Med-MoE incorporates multiple domain-specific experts along with an “always-on” meta-expert that captures global medical information. The model is an adaptation of LLaVA-style frameworks, where a modality-aware router is pre-trained to identify imaging types (CT, MRI, pathology, X-ray) and direct the token stream to the relevant expert. MedMoE focuses specifically on adapting local visual representations based on the diagnostic context and report type. It routes multi-scale features through expert branches conditioned on the accompanying clinical narrative. Using a Swin Transformer backbone to derive feature pyramids, MedMoE enables spatially adaptive attention that emphasizes regions of the image aligned with the semantic granularity of the associated report.

Modular 3D Volumetric Processing

Med3DVLM [50] addresses the inherent complexity of 3D volumetric data through architectural modularity. It integrates three key innovations: (1) DCFormer, an efficient encoder that uses decomposed 3D convolutions to capture volumetric features with reduced complexity; (2) a SigLIP-based sigmoid language-image pre-training scheme for stable alignment; and (3) an MLP-Mixer projector that effectively fuses low- and high-level image features with LLM text embeddings. Med3DVLM is an adaptation of the LLaVA framework, utilizing the DCFormer-small variant and ClinicalBERT [51] as backbones, pre-trained on 120,084 3D medical images. The table below details the routing and expert configurations of prominent MoE and modular models.

DatasetYearSizeOpen-sourceTasks
BraTS [34]2020∼10K\sim10K images✓S
CheXpert [35]2019∼224K\sim224K chest X-rays✓C
MS-CXR-T [36]20221162 images✓C
21K medical images [37]202321K images✓S
CX-XHR [38]202085,184 images✓C
COVID-19 CT [39]202040,175 images✓C
MedMD [40]202316M samples×C,S,R
MEH-MIDAS [41]202337,401 images✓C
RSNA Pneumonia [42]2018∼30K\sim30K chest X-rays✓C,S
Colossal Clean Crawled Corpus [43]2023–✓C,S,R
MIMIC-III [44]2016∼58K\sim58K ICU admissions✓R,Q
MultiMedQA [24]202312,723 QA pairs✓Q
MTB [5]2023–✓R,Q
PMC-OA [45]2023∼3.9M\sim3.9M full-text articles✓R,Q
MIMIC-CXR [17]2019∼377K\sim377K images, ∼227K\sim227K reports✓R
VQA-RAD [46]2018∼315\sim315 images, 3K3K QA pairs✓Q
ALIGN [43]2021∼1.8B\sim1.8B image-text pairs✓C,S,R
PMC-15M [12]202315M figure-caption pairs✓C,S,R,Q
Path-VQA [47]2020∼5K\sim5K images, ∼33K\sim33K QA pairs✓C,Q
SLAKE [48]2021∼642\sim642 images, ∼14K\sim14K QA pairs✓C,S,Q

Table 2. Table 2: Summary of datasets used in foundation models for medicine. Tasks are abbreviated as C (Classification), S (Segmentation), R (Report Generation), and Q (Question Answering).

The Generative Diffusion & Reconstruction Paradigm

The diffusion paradigm replaces autoregressive decoding with iterative denoising, offering a flexible alternative for text and image generation. DiffuMamba (or DiffuApriel) is a masked diffusion language model built on a bidirectional Mamba-2 backbone, matching Transformer-based diffusion quality with up to 8.2x8.2x higher throughput on long sequences. In image generation, RoentGen [52] is an adaptation of the Stable Diffusion (SD) pipeline, fine-tuned on real chest X-rays to synthesize high-fidelity scans conditioned on textual reports. MSTG (Mamba-Spatial-Temporal Generator) integrates CNNs and Mamba within a diffusion probabilistic model to generate 4D temporal medical images, effectively modeling spatial and sequential dependencies.

The Reasoning-Incentivized / RL Paradigm

A major criticism of perception-oriented Med-VLMs is their lack of sophisticated reasoning capabilities required for real-world clinical contexts. Supervised fine-tuning often results in models that memorize surface patterns rather than developing robust understanding. To address this “reasoning gap," researchers have introduced reinforcement learning (RL) frameworks designed to incentivize step-by-step logic and clinical guideline alignment. MedVLM-R1 [53] and Med-R1 [54] utilize Group Relative Policy Optimization (GRPO) to cultivate emergent reasoning capabilities without requiring expensive, explicit reasoning annotations. MedVLM-R1 is adapted from the Qwen2-VL-2B [55] backbone and is fine-tuned on radiological images (MRI, CT, X-ray) to generate explicit rationales alongside final answer selections. This model demonstrates that compact architectures can outperform generalist giants like Qwen2-VL-72B by focusing on rule-based rewards for structural compliance and accuracy. Med-R1 also leverages the Qwen2-VL backbone and GRPO, supporting 8 diverse imaging modalities and key diagnostic tasks like lesion grading and biological attribute analysis. Med-R1 explores the nuances of the "thinking" process, revealing that for certain cross-domain tasks, omitting intermediate rationales can actually improve generalization by reducing overfitting to specific reasoning styles.

The Data Infrastructure

The technical success of these architectural paradigms is inextricably linked to the curation of large-scale biomedical datasets. PMC-15M [12] has emerged as the global standard for contrastive and generative pre-training, providing 15 million image-text pairs extracted from publicly available scientific articles. This dataset solves the “supervision bottleneck" by moving beyond the narrow focus of radiology to include pathology, microscopy, and biomedical illustrations, enabling the development of generalist Med-VLMs like BiomedGPT and Med-PaLM M. MIMIC-CXR [17] remains the definitive resource for radiology-specific models like CheXzero, BioViL-T, and CXR-LLaVA. Its combination of high-resolution DICOM images and detailed free-text reports allows models to learn the fine-grained relationship between visual evidence and clinical impression. The recent release of M3D-Data further expands this infrastructure to 3D volumetric analysis, providing 120,084 CT/MRI volumes paired with radiology reports for models like Med3DVLM. Table 2 summarizes the commonly used datasets for medical VLMs along with their task based utility.

Application of Medical VLMs

Visual Question Answering & Report Generation

One common application of foundation models and medical vision-language models is their integration into chatbots, facilitating interactive question answering and clinical report generation. These foundation models have demonstrated promising performance and assistive capabilities in generating clinical reports. For instance, ClinicalBERT ([51]) and RadioBERT ([56]), which build upon the BERT architecture, exhibit remarkable effectiveness in disease prediction and report generation based on user-input symptoms and test results.

While BERT is widely used for medical chatbots and report generation, GPT-based models have gained traction following the success of ChatGPT. Models such as GPT-2, GPT-3.5, and GPT-4 ([57]) have been employed to generate radiology reports based solely on patient information and imaging findings, with their performance validated by practicing radiologists, indicating their applicability in clinical settings.

Additionally, BiomedGPT ([58]) is a GPT-based model extensively fine-tuned for medical tasks, enabling end-to-end question answering and report generation. BiomedGPT attains the highest accuracy for question answering and has a decent performance for report generation.

Another significant effort involves analyzing the performance of existing open-source chatbots like ChatGPT and Claude for question-answering. Studies indicate that ChatGPT and Claude can extract valuable information from clinical notes and patient data, providing relevant and accurate answers. These advancements underscore the potential of foundation models in improving medical chatbot functionality and clinical workflows.

Pathology Classification

One of the most common applications of deep learning models in computer vision is classification. Generalized classification models typically rely on heavily annotated datasets, which are often scarce in the medical domain. Consequently, VLMs targeting classification tasks employ zero-shot or few-shot learning as viable alternatives. For instance, CheXzero ([16]) has demonstrated strong performance in pathology classification on chest X-ray (CXR) ([35]) images using single-shot learning with minimal annotations, achieving accuracy comparable to that of radiologists. CheXzero attains a high AUC score across multiple diagnostic tasks. Similarly, CXR-CLIP ([36]) utilizes image-text pairs along with general prompts for both classification and report generation. Other strategies to address the need for extensively labeled images in medical classification include optimized fine-tuning methods that leverage positive pair loss to enhance performance alongside zero-shot learning. The widespread adoption of zero- or few-shot learning techniques for medical image classification has resulted in powerful VLMs being effectively utilized for not only question answering and report generation but also for classification tasks. This evolution underscores the potential of these models to overcome challenges associated with limited annotated datasets in the medical field, thereby improving diagnostic accuracy and efficiency.

Medical Image Segmentation

Medical image segmentation is a crucial aspect of medical diagnostics, playing a key role in non-invasive information retrieval, visualization, and analysis. Several models, including MedSAM [28] and Swin UNETR [71], have fine-tuned existing state-of-the-art architectures like SAM and U-Net for medical imaging applications. These models have demonstrated remarkable performance not only in larger object segmentation but also in achieving benchmark results for polyp segmentation. The ViT-based Swin UNETR model, for example, has achieved the highest scores in the BraTS 2021 [34] segmentation challenge, highlighting its effectiveness in brain tumor segmentation. Both MedSAM and Swin UNETR achieve comparable F1 scores on medical segmentation tasks, with SAM-Med2D performing closely behind.

Another significant model is nnU-Net [72], which automates the configuration of U-Net for various medical imaging tasks, optimizing performance across diverse datasets. Furthermore, the use of Generative Adversarial Networks (GANs) in models like CycleGAN [73] has shown promise in tasks such as domain adaptation and enhancing image quality for segmentation purposes.

While significant progress has been made in developing VLMs and corresponding datasets for 2D medical image segmentation, recent research is increasingly focusing on extending these capabilities to 3D segmentation tasks [74]. Vista3D [30] presents a unified foundation model for both 2D and 3D medical image segmentation, emphasizing versatility across modalities while minimizing the need for extensive 3D annotations. To address the scarcity of large-scale annotated 3D datasets, Mosaic3D [75] was recently introduced. It combines a 3D encoder trained via contrastive learning with a lightweight mask decoder, enabling open-vocabulary semantic and instance segmentation in 3D medical volumes. In parallel, SAM-Med3D [76] proposes a cross-dimensional adaptation strategy that leverages 2D SAM features for effective 3D segmentation, highlighting the potential of 2D-to-3D knowledge transfer. These advancements underscore the growing emphasis on generalizable 3D foundation models in medical imaging, particularly in data-scarce environments.

Overall, these advancements in medical image segmentation highlight the potential of both traditional and deep learning-based methods to improve diagnostic accuracy and facilitate better clinical decision-making across various imaging modalities.

Bridging the Gap Between Foundation Models and Real-World Healthcare Practitioners

The integration of foundation models into healthcare presents a transformative opportunity to improve clinical decision-making and streamline workflows. While these models have demonstrated substantial promise in tasks such as diagnostics, report generation, and medical image analysis, their successful deployment in real-world clinical settings requires not only technical proficiency but also alignment with clinical workflows and healthcare-specific requirements ([77]).

Vision-language multimodal foundation models are designed to generalize across diverse medical imaging modalities and tasks by jointly learning from visual and textual inputs. However, their broad scope often limits clinical applicability due to the inherent heterogeneity in imaging techniques, anatomical structures, and diagnostic requirements. Consequently, there has been a growing focus on developing specialized vision-language models that are optimized for particular clinical domains, leveraging targeted datasets and task-specific objectives. Table 3 summarizes key vision-language foundation models designed for domain-specific medical applications.

YearModelModalityTaskOrgan
2022RoentGen [52]X-rayGenerationChest
ChexZero [16]X-rayInterpretationChest
2023RadFM [59]RadiologyInterpretationTissue
RetFound [60]OphthalmologyInterpretationEye
X-RayGPT [23]X-rayInterpretationChest
PLIP [61]HistopathologyAnalysisTissue
2024CONCH [62]HistopathologyClassificationTissue
EchoCLIP [63]EchocardiogramInterpretationHeart
Merlin [64]CT3D TomographyAbdomen
CheXagent [65]X-rayInterpretationChest
VisionUnite [66]OphthalmologyMultimodal analysisEye
EyeCLIP [67]OphthalmologyInterpretationEye
2025FLAIR [68]OphthalmologyInterpretationEye
MUSK [69]Whole-body ImagingClassificationGeneral
NasVLM [70]Nasal EndoscopyDiagnosisNose

Table 3. Domain-specific vision-language foundation models categorized by year, modality, task, and organ system.

CONCH ([62]), for example, is a vision-language foundation model developed for histopathology. Trained on a diverse dataset that includes histopathology images, biomedical text, and over 1.17 million image–captions pairs, CONCH achieves state-of-the-art performance on classification, segmentation, and captioning tasks across 14 histology datasets. Similarly, PLIP ([61]) is a vision-language model trained on pathology images from OpenPath, showing a 2.5% improvement in F1 score over prior baselines and excelling in image-text retrieval and classification tasks.

EchoCLIP ([63]) is designed for echocardiography, leveraging ultrasound images and expert interpretations to facilitate automated and accurate echocardiogram analysis. Merlin ([64]), a 3D vision-language model, integrates structured electronic health records (EHRs) with unstructured radiology reports for the interpretation of abdominal CT scans. The model has shown strong performance in internal validation and real-world data settings.

RoentGen ([52]) is a vision-language model for chest X-ray (CXR) generation. It synthesizes realistic, diverse CXR images controllable via free-form radiological text prompts, expanding the potential for data augmentation and medical image simulation. Similarly focused on chest X-ray analysis, CheXAgent ([65]) specializes in disease classification and diagnosis, emphasizing model transparency and decision interpretability.

In ophthalmology, FLAIR ([68]) and EyeCLIP ([67]) address retinal and general eye disease diagnosis, respectively, utilizing multimodal training data to enable image-text alignment for tasks such as disease classification and report generation. VisionUnite also focuses on ophthalmology, incorporating multimodal representations to support diagnosis across a broader range of eye conditions.

In the context of oncology, MUSK ([69]) is a domain-specific foundation model trained for cancer diagnosis and treatment prediction, leveraging both imaging and clinical text. NasVLM ([70]) addresses nasal disease diagnosis by integrating ENT-specific medical images and clinical descriptions, facilitating automated classification and report generation in rhinology.

These specialized models build upon general-purpose foundation architectures and adapt them to meet the unique demands of specific clinical domains. While they demonstrate superior performance on curated benchmarks and in-domain datasets, their translation to real-world deployment is still hindered by challenges such as generalization across institutions, lack of prospective validation, and integration into clinical decision support systems.

In real-world settings, medical practitioners do not rely on a single source of information; instead, they utilize multiple images, reports, and test results to arrive at accurate diagnoses. Collaborations between AI developers and clinicians are essential to create user-friendly interfaces that seamlessly integrate FMs into existing systems, ensuring that these tools support rather than disrupt clinical decision-making ([78]). With their ability to process and reason across multimodal data including imaging, text-based reports, and structured clinical records FMs are increasingly being integrated into real-world healthcare environments. However, effectively leveraging these systems requires substantial training and curriculum development to ensure that practitioners understand both their capabilities and limitations [79, 80]. When appropriately integrated and supported with adequate training, studies suggest that FM-assisted workflows can significantly enhance clinical performance, with reports indicating improvements of up to 27.5% in diagnostic reasoning scores [81]. This highlights the potential of FMs not only as automated tools but as collaborative decision-support systems that augment clinicians’ analytical capabilities.

Beyond assisting physicians in diagnosis and clinical decision-making, medical foundation models are increasingly being applied to human activity recognition (HAR) tasks to support remote patient monitoring [82]. By analyzing multimodal data streams—such as wearable sensor signals, video, and physiological measurements—these models can detect patient behaviors, track rehabilitation progress, and identify abnormal activities such as falls or mobility impairments. Such capabilities make remote monitoring systems more informative and proactive, enabling clinicians to intervene earlier and provide more personalized care, particularly for elderly patients or individuals with chronic conditions.

Although foundation models (FMs) have demonstrated significant capabilities in both research and clinical settings, they are not without limitations. Notably, issues such as hallucinations and embedded biases remain important concerns. Hallucinations may lead models to generate plausible but incorrect medical information, while biases arising from imbalanced or non-representative training data can affect the reliability and fairness of model predictions. Although substantial research efforts are underway to mitigate these challenges through improved training strategies, better dataset curation, and enhanced evaluation frameworks [83], these issues have not yet been fully resolved. Consequently, the integration of FMs into real-world clinical applications must be approached with caution, ensuring appropriate safeguards, validation protocols, and human oversight. By addressing these challenges and fostering collaboration between clinicians, researchers, and technology

developers, we can bridge the gap between foundation models and their practical application in healthcare, ultimately improving patient outcomes and enhancing the efficiency of medical services.

Challenges and Opportunities

The transition of vision-language foundation models into clinical practice is fundamentally constrained by the high-dimensional complexity of medical data and the stringent reliability requirements of safety-critical environments. A primary algorithmic challenge lies in the scarcity of diverse, high-quality annotated datasets for rare pathologies, which often results in models that struggle with the "long-tail" distribution of diseases. While generative frameworks like RoentGen attempt to address this through the synthesis of realistic, text-controllable radiographs for data augmentation, ensuring the anatomical fidelity of synthesized pixels across heterogeneous imaging modalities remains a significant technical hurdle. Furthermore, the inherent black-box nature of large-scale transformer architectures limits their interpretability, as standard attention maps frequently fail to provide a medically grounded justification for a model’s diagnostic output.

Recent technical shifts toward region-specific grounding, such as those implemented in MedRegA, represent a necessary move toward identifying fine-grained anatomical structures to improve model transparency and clinical trust. The deployment of these models also faces substantial systemic and computational obstacles. Privacy regulations such as HIPAA and GDPR restrict the large-scale data aggregation required to train dense models, necessitating the exploration of decentralized paradigms like Federated Learning and the implementation of Differential Privacy layers to facilitate cross-institutional collaboration.

Computationally, the quadratic complexity of self-attention mechanisms poses a significant barrier to processing high-resolution 3D volumetric data, such as CT or MRI scans, in real-time clinical workflows. To mitigate these resource requirements, the field is increasingly adopting sparse architectures like the Mixture-of-Experts (MoE) framework utilized in Med-MoE and MedPLIB, which selectively activates only a subset of parameter experts during inference. These efficient designs, coupled with the emergence of lightweight "Small Language Models" like SigPhi-Med [27], offer a viable path for democratizing expert-level AI diagnostics in resource-constrained environments where massive compute infrastructure is unavailable.

Looking toward the horizon of General-purpose Medical AI (GMAI) [1, 2], the integration of reinforcement learning represents a critical evolution in enhancing the logical reasoning of multimodal systems. Emerging paradigms such as Group Relative Policy Optimization (GRPO), employed by models like Med-R1, incentivize logical reasoning paths over simple linguistic pattern matching, thereby reducing the risk of clinically dangerous hallucinations. This shift enables a future where foundation models act as agentic clinical assistants capable of autonomous health monitoring and precision medicine. By synthesizing multi-sensory inputs—including imaging, structured electronic health records, and genomic data—into a unified multimodal latent space, these models could facilitate highly personalized treatment ecosystems tailored to individual genetic profiles. The successful

realization of this future vision hinges on the technical refinement of these models to handle complex clinical inquiries with the consistency and precision required to augment, rather than replace, human expertise. Figure 3 summarizes the challenges, opportunities, and future prospects of the foundation model in the medical domain.

Figure 3. Overview of the Challenges, Opportunities, and Future Prospects for Foundation Model in the medical domain.

Conclusion

In this review article, we explored the evolution of foundation models in the medical domain, highlighting their growing impact on various healthcare applications. We examined the development of general and vision-language models, the datasets that fuel their training, and their role in key downstream tasks such as medical image segmentation, disease classification, report generation, and clinical question answering. While these models demonstrate remarkable potential in automating and enhancing medical workflows, challenges remain in terms of data heterogeneity, model interpretability, domain adaptation, and regulatory considerations. Addressing these challenges will require continued advancements in model design, data curation, and validation frameworks. Moving forward, interdisciplinary collaboration between AI researchers, clinicians, and regulatory bodies will be essential to ensure the safe, effective, and ethical deployment of foundation models in real-world medical settings, ultimately improving patient outcomes and the efficiency of healthcare delivery.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References

  1. [1]Wasif Khan, Seowung Leem, Kyle B See, Joshua K Wong, Shaoting Zhang, and Ruogu Fang. A comprehensive survey of foundation models in medicine. arXiv preprint arXiv:2406.10729, 2024.arxiv.org/abs/2406.10729
  2. [2]Bobby Azad, Reza Azad, Sania Eskandari, Afshin Bozorgpour, Amirhossein Kazerouni, Islem Rekik, and Dorit Merhof. Foundational models in medical imaging: A comprehensive survey and future vision. arXiv preprint arXiv:2310.18689, 2023.arxiv.org/abs/2310.18689
  3. [3]Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681–694, 2020.DOI
  4. [4]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.arxiv.org/abs/2302.13971
  5. [5]Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmiа, Jure Leskovec, Cyril Zakкa, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023.arxiv.org/abs/2307.15189
  6. [6]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.arxiv.org/abs/2306.00890
  7. [7]Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.arxiv.org/abs/2304.00685
  8. [8]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  9. [9]Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023.arxiv.org/abs/2303.00915
  10. [10]Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3 (1):1–23, October 2021. ISSN 2637-8051. doi: 10.1145/3458754. URL http://dx.doi.org/10.1145/3458754.
  11. [11]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929.
  12. [12]Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI, 2(1):AIoa2400640, 2025.DOI
  13. [13]Taha Koleilat, Hojat Asgariandehkordi, Hassan Rivaz, and Yiming Xiao. Medclip-sam: Bridging text and image towards universal medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 643–653. Springer, 2024.
  14. [14]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baing Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL https://arxiv.org/abs/2103.14030.
  15. [15]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385.
  16. [16]Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6(12): 1399–1406, 2022.
  17. [17]Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.DOI
  18. [18]Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15016–15027, 2023.arxiv.org/abs/2301.04558
  19. [19]Aaron Nicolson, Jason Dowling, and Bevan Koopman. Longitudinal data and a semantic similarity reward for chest x-ray report generation, 2024. URL https://arxiv.org/abs/2307.09758.
  20. [20]Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6, 2023.
  21. [21]Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, Hany Awadalla, Julia Gong, Houdong Hu, Jianwei Yang, Chunyuang Li, Jianfeng Gao, Yu Gu, Cliff Wong, Mu Wei, Tristan Naumann, Muhao Chen, Matthew P. Lungren, Akshay Chaudhari, Serena Yeung-Levy, Curtis P. Langlotz, Sheng Wang, and Hoifung Poon. A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings. Nature Communications, 16(1), April 2025. ISSN 2041-1723. doi: 10.1038/s41467-025-58344-x. URL http: //dx.doi.org/10.1038/s41467-025-58344-x.
  22. [22]Seowoo Lee, Jiwon Youn, Hyungjin Kim, Mansu Kim, and Soon Ho Yoon. Cxr-llava: a multimodal large language model for interpreting chest x-ray images. European Radiology, 35(7):4374–4386, 2025.DOI
  23. [23]Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models, 2023. URL https: //arxiv.org/abs/2306.07971.
  24. [24]Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaeckermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. NEJM AI, 1(3):AIoa2300138, 2024.DOI
  25. [26]Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950, 2025.DOI
  26. [27]Feizhong Zhou, Xingyue Liu, Qiao Zeng, Zhuhan Li, and Hanghuang Xiao. Siphimed: A lightweight vision-language assistant for biomedicine. J. of Biomedical Informatics, 167(C), July 2025. ISSN 1532-0464. doi: 10.1016/j.jbi.2025.104849. URL https://doi. org/10.1016/j.jbi.2025.104849.DOI
  27. [28]Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medi cal images. Nature Communications, 15(1):654, 2024.
  28. [29]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.arxiv.org/abs/2304.02643
  29. [30]Yufan He, Pengfei Guo, Yucheng Tang, Andriy Myronenko, Vishwesh Nath, Ziyue Xu, Dong Yang, Can Zhao, Benjamin Simon, Mason Belue, Stephanie Harmon, Baris Turkbey, Daguang Xu, and Wenqi Li. Vista3d: A unified segmentation foundation model for 3d medical imaging, 2024. URL https://arxiv.org/abs/2406.05285.
  30. [31]Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. Towards a multimodal large language model with pixel-level insight for biomedicine, 2025. URL https://arxiv.org/abs/2412.09278.
  31. [32]Lehan Wang, Haonan Wang, Honglong Yang, JiajI Mao, Zehong Yang, Jun Shen, and Xiaomeng Li. Interpretable bilingual multimodal large language model for diverse biomedical tasks, 2025. URL https://arxiv.org/abs/2410.18387.
  32. [33]Difei Gu, Yunhe Gao, Mu Zhou, and Dimitris Metaxas. Anatomy-vlm: A fine-grained vision-language model for medical interpretation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2838–2847, 2026.arxiv.org/abs/2511.08402
  33. [34]Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(10):1993–2024, 2014.DOI
  34. [35]Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019.arxiv.org/abs/1901.07031
  35. [36]Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Max Ilse, Daniel Coelho de Castro, Benedikt Boecking, Harshita Sharma, K Enza Bouzid, Anton Schwaighofer, et al. Ms-cxr-t: Learning to exploit temporal structure for biomedical vision-language processing, 2023.arxiv.org/abs/2301.04558
  36. [37]Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sam-med2d. arXiv preprint arXiv:2308.16184, 2023.arxiv.org/abs/2308.16184
  37. [38]Yifan Wu, Hayden Gunraj, Chi-en Amy Tai, and Alexander Wong. Covidx cxr-4: An expanded multi-institutional open-source benchmark dataset for chest x-ray image-based computer-aided covid-19 diagnostics. arXiv preprint arXiv:2311.17677, 2023.arxiv.org/abs/2311.17677
  38. [39]Xingyi Yang, Xuehai He, Jinyu Zhao, Yichen Zhang, Shanghang Zhang, and Pengtao Xie. Covid-ct-dataset: a ct scan dataset about covid-19. arXiv preprint arXiv:2003.13865, 2020.arxiv.org/abs/2003.13865
  39. [40]Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308.02463, 2023.arxiv.org/abs/2308.02463
  40. [41]Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. A foundation model for generalizable disease detection from retinal images. Nature, 622(7981):156–163, 2023.DOI
  41. [42]Linghua Wu, Jing Zhang, Yilin Wang, Rong Ding, Yueqin Cao, Guiqin Liu, Changsheng Lilufu, Baowei Xie, Shanping Kang, Rui Liu, et al. Pneumonia detection based on rsna dataset and anchor-free deep learning detector. Scientific Reports, 14(1):1929, 2024.
  42. [43]Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pre-training with weak supervision. arXiv preprint arXiv:2108.10904, 2021.arxiv.org/abs/2108.10904
  43. [44]Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii. In Proceedings of the ACM conference on health, inference, and learning, pages 222–235, 2020.DOI
  44. [45]Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ye Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. Springer, 2023.arxiv.org/abs/2303.07240
  45. [46]Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018.DOI
  46. [47]Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering, 2020. URL https://arxiv.org/abs/2003.10286.
  47. [48]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering, 2021. URL https://arxiv.org/abs/2102.09542.
  48. [49]Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhuo Liu. Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models, 2024. URL https://arxiv.org/abs/2404.10237.
  49. [50]Yu Xin, Gorkem Can Ates, Kuang Gong, and Wei Shao. Med3dvlm: An efficient vision-language model for 3d medical image analysis, 2025. URL https://arxiv.org/abs/2503.20047.
  50. [51]Bin Yan and Mingtao Pei. Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2982–2990, 2022.DOI
  51. [52]Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Malgorzata Polacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay Chaudhari. Roentgen: vision-language foundation model for chest x-ray generation. arXiv preprint arXiv:2211.12737, 2022.arxiv.org/abs/2211.12737
  52. [53]Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2025: 28th International Conference, Daejeon, South Korea, September 23–27, 2025, Proceedings, Part VII, pages 337–347, Berlin, Heidelberg, 2025. Springer-Verlag. ISBN 978-3-032-04980-3. doi: 10.1007/978-3-032-04981-0_32. URL https://doi.org/10.1007/978-3-032-04981-0_32.
  53. [54]Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, Yuheng Li, Konstantinos Psonis, and Xiaofeng Yang. Med-rl: Reinforcement learning for generalizable medical reasoning in vision-language models. IEEE Transactions on Medical Imaging, pages 1–1, 2026. doi: 10.1109/TMI.2026.3661001.
  54. [55]Weizhi Wang, Yu Tian, Linjie Yang, Heng Wang, and Xifeng Yan. Open-qwen2vl: compute-efficient pre-training of fully-open multimodal llms on academic resources. arXiv preprint arXiv:2504.00595, 2025.arxiv.org/abs/2504.00595
  55. [56]Navdeep Kaur and Ajay Mittal. Radiobert: A deep learning-based system for medical report generation from chest x-ray images using contextual embeddings. Journal of biomedical informatics, 135:104220, 2022.DOI
  56. [57]Konstantinos I Roumeliotis and Nikolaos D Tselikas. Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6):192, 2023.DOI
  57. [58]Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D. Davison, Hui Ren, Jing Huang, Chen Chen, Yuyin Zhou, Sunyang Fu, Wei Liu, Tianming Liu, Xiang Li, Yong Chen, Lifang He, James Zou, Quanzheng Li, Hongfang Liu, and Lichao Sun. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine, 30(11):3129–3141, August 2024. ISSN 1546-170X. doi: 10.1038/s41591-024-03185-2. URL http://dx.doi.org/10.1038/s41591-024-03185-2.
  58. [59]Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463, 2023.
  59. [60]Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. A foundation model for generalizable disease detection from retinal images. Nature, 622(7981):156–163, 2023.DOI
  60. [61]Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307–2316, 2023.DOI
  61. [62]Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863–874, 2024.DOI
  62. [63]Matthew Christensen, Milos Vukadinovic, Neal Yuan, and David Ouyang. Vision–language foundation model for echocardiogram interpretation. Nature Medicine, 30(5):1481–1488, 2024.DOI
  63. [64]Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs–3, 2024.DOI
  64. [65]Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208, 2024.
  65. [66]Zihan Li, Diping Song, Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E Kinahan, and Yu Qiao. Visionunite: A vision-language foundation model for ophthalmology enhanced with clinical knowledge. arXiv preprint arXiv:2408.02865, 2024.arxiv.org/abs/2408.02865
  66. [67]Danli Shi, Weiyi Zhang, Jiancheng Yang, Siyu Huang, Xiaolan Chen, Mayinuer Yusufu, Kai Jin, Shan Lin, Shunming Liu, Qing Zhang, et al. Eyeclip: A visual-language foundation model for multi-modal ophthalmic image analysis. arXiv preprint arXiv:2409.06644, 2024.arxiv.org/abs/2409.06644
  67. [68]Julio Silva-Rodriguez, Hadi Chakor, Riadh Kobbi, Jose Dolz, and Ismail Ben Ayed. A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision. Medical Image Analysis, 99:103357, 2025.
  68. [69]Jinxi Xiang, Xiyue Wang, Xiaoming Zhang, Yinghua Xi, Feyisop Eweje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, et al. A vision–language foundation model for precision oncology. Nature, pages 1–10, 2025.DOI
  69. [70]Xueli Liu, Wentao Gong, Xiao Chen, Zhen Li, Yinlong Liu, Li Wang, Quan Liu, Xicai Sun, Xiaofeng Liu, Xinrong Chen, et al. Vision-language foundation model for generalizable nasal disease diagnosis using unlabeled endoscopic records. Pattern Recognition, 165:111646, 2025.DOI
  70. [71]Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger Roth, and Daguang Xu. Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images, 2022. URL https://arxiv.org/abs/2201.01266.
  71. [72]Huan Minh Luu and Sung-Hong Park. Extending nn-unet for brain tumor segmentation. In International MICCAI brainlesion workshop, pages 173–186. Springer, 2021.arxiv.org/abs/2112.04653
  72. [73]Amjad Almahairi, Sai Rajeșhwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Augmented cyclegan: Learning many-to-many mappings from unpaired data. In International conference on machine learning, pages 195–204. PMLR, 2018.arxiv.org/abs/1802.10151
  73. [74]Shizhan Gong, Yuan Zhong, Wen ao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou. 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306.13465, 2023.DOI
  74. [75]Junha Lee, Chunghyun Park, Jaesung Choe, Yu-Chiang Frank Wang, Jan Kautz, Minsu Cho, and Chris Choy. Mosaic3d: Foundation dataset and model for open-vocabulary 3d segmentation, 2025. URL https://arxiv.org/abs/2502.02548.
  75. [76]Haoyu Wang, S izheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, Bin Fu, Shaoting Zhang, Junjun He, and Yu Qiao. Sam-med3d: Towards general-purpose segmentation models for volumetric medical images, 2024. URL https://arxiv.org/abs/2310.15161.
  76. [79]Isabel Tretow, Moritz Schwebel, Stefan Feuer riegel, Theresa Treffers, and Isabell M. Welpe. The effect of llm assistance on diagnostic accuracy: A meta-analysis. medRxiv, 2025. doi: 10.64898/2025.12.11.25341476. URL https://www.medrxiv.org/content/early/2025/12/12/2025.12.11.25341476.DOI
  77. [80]Isabel Tretow, Moritz Schwebel, Stefan Feuer riegel, Theresa Treffers, and Isabell M Welpe. The effect of llm assistance on diagnostic accuracy: A meta-analysis. medRxiv, pages 2025–12, 2025.DOI
  78. [81]Ihsan Ayyub Qazi, Ayesha Ali, Asad Ullah Khawaja, Muhammad Junaid Akhtar, Ali Zafar Sheikh, and Muhammad Hamad Alizai. Large language model diagnostic assistance for physicians in a lower-middle-income country: a randomized controlled trial. Nature Health, 1(2): 198–205, 2026.DOI
  79. [82]Abderrazek Abid, Thanh-Cong Ho, and Fakhri Karray. Vision language models for dynamic human activity recognition in healthcare settings. In International Work-Conference on Bioinformatics and Biomedical Engineering, pages 35–45. Springer, 2025.DOI
  80. [83]Zhihong Zhu, Yunyan Zhang, Xianwei Zhuang, Fan Zhang, Zhongwei Wan, Yuyan Chen, QingqingLong QingqingLong, Yefeng Zheng, and Xian Wu. Can we trust ai doctors? a survey of medical hallucination in large language and large vision-language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6748–6769, 2025.

Paper details

vlnc:W_87y2azk7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents