WorksComputer Science
BAPLe++: Cross-Model Transferability and Prompt-Level Defense of Backdoors in Medical Foundation Models
Abstract
Medical foundation models (Med-FMs) are increasingly adapted to clinical tasks through prompt learning, which tunes a small set of prompt tokens while the backbone remains frozen. Recent work has shown that this efficiency introduces a vulnerability: jointly optimizing learnable prompts and an imperceptible noise trigger during few-shot adaptation implants a backdoor that leaves clean performance intact but steers triggered inputs to an attacker-chosen class. Since prompts are compact and readily shared, this raises two questions: whether a backdoored prompt remains effective on a different Med-FM, and whether it can be repaired without retraining the backbone. We address both in BAPLe++. First, we present the first study of cross-model transferability of prompt-based backdoors, quantified by two proposed metrics, Clean Accuracy Difference (CAD) and Backdoor Accuracy Difference (BAD). Second, we propose DePLe (Defense via Prompt Learning), which uses the infected prompt as a fixed anchor and learns a new prompt on a small clean set that preserves the clean task while diverging from the compromised embedding region. Experiments on four Med-FMs and six chest X-ray and histopathology datasets show that backdoors transfer across models, but asymmetrically and in a dataset-dependent manner. Compared with four training-free input transformations, DePLe reduces backdoor accuracy more consistently while preserving, and in several settings improving, clean accuracy. Our code can be accessed on https://github.com/Ufaqkhan/Baple-Plus-Plus
Introduction
Medical Foundation Models (Med-FMs) [18, 6, 19, 7] have emerged as a transformative paradigm in medical imaging, enabling robust performance across diverse modalities and clinical tasks. By pretraining on large-scale medical image-text datasets, CLIP-like Med-FMs [18, 6, 19, 7] learn aligned visual and semantic representations, which can be readily adapted to a wide range of downstream tasks such as disease classification, histopathology analysis, and radiology report generation. This ability to transfer knowledge with minimal labeled data has made Med-FMs increasingly attractive for clinical deployment, particularly in settings where annotated medical data are scarce. As a result, Med-FMs are rapidly becoming central components in decision-support pipelines and automated imaging analysis.
As Med-FMs transition from research prototypes to clinical practice, ensuring their trustworthiness has become a pressing challenge. These models are often trained or adapted using publicly available datasets and open-source checkpoints, which promote reproducibility but also introduce attack surfaces such as data poisoning and checkpoint tampering. Combined with the scale and limited interpretability of Med-FMs, these factors complicate auditing and risk assessment. The concern is well founded, since deep learning models are known to be susceptible to adversarial manipulation [4, 13, 2]. Among such threats, backdoor attacks are particularly insidious. An adversary poisons a small fraction of the training or fine-tuning data by embedding subtle visual patterns, known as triggers, into selected samples, so that the compromised model behaves normally on clean inputs but consistently misclassifies triggered inputs into an attacker-specified target class. This dual behavior makes backdoors difficult to detect with standard validation on clean data, which is particularly concerning in clinical workflows where an undetected misclassification may adversely affect patient care. Systematic evaluation under well-defined adversarial and data-poisoning threat models is therefore a necessary, though not sufficient, step toward the safe deployment of Med-FMs in safety-critical clinical settings.
Recently, BAPLe (Hanif et al., [5]) introduced backdoor attacks on Med-FMs during prompt learning, which demonstrated for the first time that Med-FMs can be backdoored during the prompt learning phase. Unlike earlier backdoor attacks that rely on retraining large models with substantial amounts of poisoned data, BAPLe demonstrates that introducing learnable prompt tokens into the text encoder and imperceptible noise triggers into image inputs during few-shot prompt learning is sufficient to achieve over 90% attack success across multiple Med-FMs and datasets, while maintaining accuracy on clean samples. These findings highlight a critical vulnerability, i.e., parameter-efficient adaptation techniques such as prompt learning widely adopted due to their efficiency in data-limited settings can serve as an attack vector.
However, important gaps remain. First, prior work such as BAPLe focuses on attack construction, without evaluating attack effectiveness under source-to-target Med-FM transfer. In practice, institutions may deploy distinct foundation models trained on different pretraining corpora (e.g., X-ray-based MedCLIP vs. pathology-focused PLIP) ([18, 6]). Whether a backdoor crafted on one Med-FM transfers to another remains unknown, yet this question is crucial for real-world risk assessment. Second, the literature lacks a systematic investigation of defense mechanisms for mitigating backdoor attacks in Med-FMs under few-shot prompt learning setup. Third, defenses originally proposed for natural image models, including input denoising, activation clustering, and adversarial training, remain largely unexplored in the Med-FM context, where limited labeled data and frozen backbone architectures impose unique constraints.
In this work, we address these gaps by presenting BAPLe++, which investigates both cross-model transferability of backdoor attacks and proposes a robust defense mechanism to sanitize infected prompts. We further compare this prompt repair with four training-free input-transformation defenses. Our experiments reveal that backdoors can persist beyond the model in which they are introduced and may propagate across institutional deployments, posing systemic risks to medical AI. At the same time, prompt-space repair suppresses backdoor behavior more consistently than generic input preprocessing while largely preserving clean performance. These contributions position BAPLe++ as a step toward the trustworthy and secure adoption of Med-FMs in clinical practice.
Our contributions are threefold:
We are the first to examine whether backdoors introduced via prompt learning in one Med-FM persist when transferred to and evaluated on other Med-FMs. To quantify this effect, we introduce two metrics: Clean Accuracy Difference (CAD), which measures the change in clean performance after prompt transfer, and Backdoor Accuracy Difference (BAD), which measures how much attack success is retained when moving from the source to the target Med-FM.
We propose DePLe, a parameter-efficient defense that learns robust prompts using the backdoor-infected prompts as fixed anchors, which mitigates backdoor behavior without modifying the frozen Med-FM backbone.
We provide a systematic evaluation of prompt-based backdoor attacks and defenses across four Med-FMs (MedCLIP, BioMedCLIP, PLIP, and QuiltNet) and six chest X-ray and histopathology datasets, benchmarking our defense against four training-free input transformations (Gaussian Blur, JPEG-30, Feature Squeeze, and Pixel Deflection). We find that (i) prompt-based backdoors transfer across Med-FMs, but asymmetrically and in a dataset-dependent manner, (ii) prompt-space repair reduces backdoor accuracy more consistently than input transformations while preserving clean accuracy, and (iii) input transformations trade clean accuracy for robustness in ways that vary strongly across models and datasets.
Related Work
Medical Foundation Models
Medical Foundation Models (Med-FMs) have become central to modern medical image analysis, offering transferable representations that reduce the reliance on large, task-specific annotated datasets. By pre-training on large-scale medical image-text corpora, these models are able to generalize across diverse downstream tasks such as disease classification, lesion detection, and pathology recognition. Among the most influential, MedCLIP (Wang et al., [18]) adapts the CLIP (Radford et al., [15]) framework to chest X-rays, aligning radiology reports with image features for zero-shot classification. BioMedCLIP (Zhang et al., [19]) extended this approach by training on millions of medical image-caption pairs, significantly improving cross-modal transfer. In the histopathology domain, PLIP (Huang et al., [6]) and QuiltNet (Ikezogwo et al., [7]) introduced pre-training strategies tailored for cell and tissue-level analysis. These advances demonstrate the versatility and clinical utility of Med-FMs across imaging modalities.
Prompt Learning
Prompt learning has emerged as an effective alternative to full fine-tuning, enabling efficient adaptation of foundation models while keeping model backbone parameters frozen. Methods such as CoOp (Zhou et al., [20]) demonstrated that learning a set of tunable prompt tokens can effectively adapt vision-language models such as CLIP (Radford et al., [15]). This prompt-learning paradigm was subsequently transferred to the medical domain, leading to models such as MedCLIP (Wang et al., [18]) and PLIP (Huang et al., [6]). Despite their efficiency, the reliance on frozen encoders and parameter-efficient adaptation mechanisms such as prompt learning introduces new security risks, making medical foundation models particularly vulnerable to adversarial manipulation.
Backdoor Attacks
Backdoor attacks aim to implant hidden behaviors into models such that clean inputs behave normally, but the presence of a trigger causes targeted misclassification. Early work such as BadNets (Gu et al., [4]) showed how simple patch triggers could compromise classifiers, followed by Trojaning attacks (Liu et al., [12]) and WaNet (Nguyen & Tran, [13]) which improved stealth and generalizability. In the medical domain, FIBA (Feng et al., [2]) introduced backdoors tailored to medical images, while BadCLIP (Liang et al., [11]) extended the idea to vision-language models by coupling image triggers with prompt embeddings. In medical imaging, BAPLe (Hanif et al., [5]) was the first to demonstrate that prompt learning, while effective in few-shot settings, can also serve as an attack surface, allowing backdoors to be injected via learnable prompts under data-scarce conditions.
Adversarial Transferability and Defense
Adversarial transferability, where perturbations crafted on one model fool another, is well studied in vision networks (Khan et al., [10]). Optimizations such as RAP (Qin et al., [14]; Khan et al., [10]) and GRA (Zhu et al., [21]) improve transfer across architectures by reshaping the loss landscape and the gradient signal used to craft perturbations. Whether the same effect arises in the backdoor setting, where the transferred artifact is a learned prompt rather than an input perturbation, has not been examined for medical foundation models. In that setting, many practical defenses remain training-heavy or architecture-specific. By contrast, test-time preprocessing defenses such as blur, compression, and filtering are attractive because they are easy to deploy, but their robustness/utility trade-offs for Med-FMs remain poorly understood.
Method
Preliminaries
Zero-Shot Classification for Med-FM
Since Med-FMs, particularly vision-language models (VLMs), are pre-trained to align images with corresponding textual descriptions, they are inherently suited for zero-shot classification tasks. Zero-shot inference in VLMs refers to making predictions on new, unseen data without specific training. We denote a VLM by , where and are the image and text encoders, respectively. For classification in zero-shot scenario, the image is first passed to the image encoder , resulting in a -dimensional feature vector . Similarly, on the text encoder side, each class label is wrapped within the class-specific text template, such as:
Each text prompt is fed to the text encoder , yielding text feature vector . The relationship between the image’s feature vector and the text prompt feature vector is quantified using cosine similarity, , to evaluate the image’s alignment with th class. Class with the highest similarity score is selected as the predicted class label , i.e.
Prompt Learning
Textual class descriptions are central to zero-shot inference in VLMs, but manually crafted prompts can lead to performance variability and sensitivity. Prompt learning addresses this by introducing auxiliary learnable parameters on top of the text encoder , optimized in few-shot settings to steer the frozen model’s response and reduce manual engineering [20]. Formally, prompt learning methods optimize to enhance downstream performance and generalization:
where represents an image-label pair from few-shot training dataset , denotes the model’s prediction conditioned on the learnable prompt parameters , and is the task-specific loss function.
Backdoor Attack
A backdoor attack involves embedding a visible/hidden trigger (a small random or patterned patch) within a deep learning model during its training or fine-tuning phase. When the model encounters this trigger in the input data during inference, it produces a predefined output while performing normally on clean data. In a supervised classification task, a normally trained classifier maps a clean input image to a label . Parameters are learned from a training dataset where and . In a typical backdoor attack, the training dataset is split into clean and poison subsets , where . In , each sample is transformed into a backdoor sample , where is the backdoor injection function and denotes the target label function. During the training/fine-tuning phase of backdoor attacks, the victim classifier is trained/fine-tuned on a mix of the clean dataset and the poisoned dataset . Formally, this task can be formulated as the following objective function:
where denotes the cross-entropy loss. After training, behaves similarly on clean input as the original classifier (trained entirely on clean data), yet alters its prediction for the backdoor image to the target class , i.e. and .
Threat Model
We describe the objectives and capabilities of the attacker and the defender.
Attacker’s Objective and Capabilities
The attacker aims to embed a targeted backdoor into a Med-FM during downstream adaptation, so that the adapted model behaves normally on clean inputs but predicts an attacker-chosen target class whenever a trigger is present. The backdoor must remain stealthy under standard validation, which relies only on clean data. Following prior work on medical backdoor attacks [2], we assume the attacker can poison only a small fraction of the downstream training data and can apply the trigger to inputs at inference time. Because annotated medical data are scarce and the attacker is computationally limited, full fine-tuning of the Med-FM is impractical.
The attacker therefore exploits prompt learning, which changes model behavior while keeping the backbone frozen. We consider two realistic scenarios. In the first, the attacker acts as a malicious service provider: a user supplies clean labeled data and receives a task-specific prompt that has been backdoored during training. In the second, the attacker distributes backdoored prompts through public repositories (e.g., Hugging Face), where users may integrate them with their own Med-FMs. In both cases, the backdoor is introduced through the prompt, allowing the attacker to compromise deployments without modifying model weights or accessing private training pipelines.
Defender’s Objective and Capabilities
The defender aims to deploy a Med-FM safely for a downstream medical task, preserving high clean accuracy while preventing triggered inputs from being steered to the target class. The defense must work under the same clinical constraints as adaptation itself, namely limited labeled data and a frozen backbone. We assume the defender has access to the pre-trained Med-FM, the deployment prompt, which may be compromised, and a small clean labeled set for the task. Like the attacker, the defender cannot retrain or modify the backbone.
Since the backdoor is introduced through the prompt rather than the model weights, the defender operates entirely at the prompt level. Instead of discarding the suspect prompt or retraining the model, the defender learns a new prompt that preserves clean-task utility while suppressing backdoor Algorithm 1 BAPLe - Backdoor Attack via Prompt Learning
Notations NumSamples=, BatchSize=, NumBatches=, Train Dataset: , Vision and Text Encoders: , Image: , Learnable Trigger Noise: , Perturbation Budget: , Backdoor Patch: , Backdoor Injection Function: , Target Label Function: , Learnable Prompt Context: , Number of Classes: , Text Prompts: where , Cosine Similarity Function:
for to NumEpochs do
for to NumBatches do
Sample clean and poisoned mini-batch: ,
▷ Prediction scores on clean images
▷ Prediction scores on poisoned images
▷ Compute cross-entropy loss on the mini-batch
▷ Update prompt parameters
▷ Update trigger noise
▷ Apply budget on noise
end for
end for
behavior, using the suspect prompt as a reference for the compromised region of the embedding space.
Backdoor Attack using Prompt Learning (BAPLe)
BAPLe (Hanif et al. [5]), as shown in Figure 1(a), is a backdoor attack method that exploits the multimodal nature of medical vision-language models (Med-VLMs). For computational efficiency, it injects a small set of learnable parameters (text prompts) into the input space of the model’s text encoder while keeping the backbone frozen during training on a poisoned downstream dataset. The prompt-based mechanism is particularly well suited to few-shot settings: by tailoring the input space, it enables a Med-FM to leverage pretrained multimodal knowledge for data-scarce tasks without requiring extensive retraining or large medical datasets. To further increase attack effectiveness, the method also adds imperceptible and learnable noise as a backdoor trigger to the input images, which subtly increases the model’s sensitivity to backdoor activation. By combining learnable text prompts in the text encoder with an imperceptible image trigger, this approach takes advantage of the full multimodal capabilities of Med-VLMs.
Figure 1. Overview of BAPLe++. (a) Backdoor Attack: a learnable prompt context and an imperceptible noise trigger are jointly optimized while both encoders remain frozen, so that triggered inputs are steered toward the target label . (b) Cross-model Transferability: the noise and prompt learned on a source Med-FM are transferred, unchanged, to a target Med-FM , and transferability is measured with CAD and BAD. (c) Defense: the infected prompts are kept as fixed anchors, and robust prompts are learned by minimizing , which preserves clean-task alignment while pushing the prompts away from the compromised embedding region. Without the defense, a triggered input is classified as the backdoor label; after sanitization, it is assigned the true label. The Med-FM backbone remains frozen throughout
Zero-shot inference in a Med-FM depends on hand-engineered text prompts, and prompt learning therefore replaces fixed templates with a small set of learnable token embeddings . For classes we form class-specific text inputs where , i.e., learnable prompt context is prepended to the class names in the input token embedding space. Denote the frozen image and text encoders by and and a similarity scoring function by . The model logit vector on a clean image and on a backdoored image are written compactly as
To inject a backdoor, BAPLe jointly optimizes learnable prompt tokens and an imperceptible noise trigger via a bi-objective that balances clean utility and targeted misclassification:
Here and are the clean and poisoned data subsets, maps labels to the attacker-chosen target, denotes an optional spatial patch (logo) and denotes the operation that inserts/combines the patch with the image. Note that both the image and text encoders remain frozen during optimization of Equation (4); only and are updated. Algorithm 1 outlines the attack steps.
Cross-Model Transferability
To evaluate whether backdoors implanted in one Med-FM can persist when transferred to another, we study cross-model transferability. In this setting, an attacker first learns prompts and a noise trigger on a source model . These artifacts are then applied directly, without re-optimization, to a different target model during inference. This setup reflects realistic scenarios where compromised prompts or triggers may propagate across institutions employing distinct Med-FMs. We quantify transferability using two metrics: Clean Accuracy Difference (CAD) and Backdoor Accuracy Difference (BAD). Given a test set , CAD measures the change in clean performance of the target model after transferring the learned text prompt from source model , while BAD measures the change in backdoor success rate when transferring both the prompt and trigger from source model to target model . Formally, CAD and BAD are defined as follows:
where and denote prompts learned from the source and target models, and are their respective triggers, represents the attacker-chosen target label, and is indicator function. An overview of cross-model transferability is shown in Figure 1(b). For a highly transferable attack, both CAD and BAD should be close to zero. A near-zero CAD means clean accuracy is preserved when a backdoored prompt is transferred from the source to the target model. A near-zero BAD means the backdoor remains equally effective, so source-infected prompts trigger the target model as strongly as prompts optimized directly on it.
Defense via Prompt Learning (DePLe)
Since the backdoor resides in the learned prompt rather than in the model weights, we mitigate it at the prompt level. The Med-FM backbone stays frozen, and the defender has only a small clean labeled set . Rather than discarding the potentially infected prompt , we keep it as a fixed anchor that marks a compromised region of the text-embedding space. During the attack, is optimized jointly with the trigger , so its class embeddings are positioned such that triggered image features align with the target class. We therefore learn a freshly initialized prompt on while pushing its class embeddings away from those induced by . This retains clean-task semantics while discouraging the prompt from re-learning the trigger-aligned configuration. Formally, with and , we define a clean-task loss and a feature-divergence term:
where returns -normalized embeddings, so is bounded. The robust prompt context is obtained by solving the following objective. A high-level overview of the proposed defense is shown in Figure 1(c).
Experimental Setup
Implementation Details
All experiments were conducted on a single NVIDIA RTX 3090Ti GPU with 24 GB of memory. We adopt a 32-shot setting, selecting 32 random samples per class, and use a batch size of 16. For prompt-based training, we set the learning rate to . A poisoning rate of 5% was applied across all datasets. The learnable noise trigger is bounded by an budget of , and a fixed backdoor patch of size is inserted at the bottom-left corner of each image. Each class is considered as a possible attack target, and final results are averaged across all target classes to ensure fairness in evaluation.
Datasets
We conduct experiments across six benchmark medical imaging datasets spanning both radiology and histopathology domains. For radiology, we utilize COVID ([16]), a chest X-ray dataset for COVID-19 detection, RSNA18 ([17]), a large-scale chest X-ray dataset for pneumonia classification, and MIMIC ([8]), a clinically curated chest X-ray dataset (single-label samples only). For histopathology, we include Kather ([9]), a dataset for colorectal cancer tissue classification, PanNuke ([3]), a pan-cancer nuclei classification dataset, and DigestPath ([1]), which focuses on digestive system pathology for tumor detection. These datasets span two imaging modalities (radiology and pathology) and provide a comprehensive testbed for studying both attack persistence and defense effectiveness.
Medical Foundation Models (Med-FMs)
We evaluate four representative Med-FMs that are widely used in the literature. MedCLIP ([18]) leverages contrastive learning on chest X-rays and radiology reports, while BioMedCLIP ([19]) is pretrained on the large-scale PMC-15M image-text pairs. For histopathology, PLIP ([6]) is a pathology-focused foundation model trained on extensive histopathology corpora, and QuiltNet ([7]) is a recent pathology vision-language model designed with improved patch-level alignment. All models adopt the CLIP-style dual-encoder architecture, where the backbones remain frozen and only prompt embeddings are trained during adaptation.
Metrics
We report four metrics. Clean Accuracy (CA) is the classification accuracy on unperturbed test samples and reflects the model’s utility for normal clinical use. Backdoor Accuracy (BA) is the fraction of triggered test inputs classified as the attacker-specified target class; test samples whose true label is the target class are excluded. A high BA indicates a successful attack, and a reduction in BA after a defense indicates effective mitigation. Two difference-based metrics quantify cross-model transferability. Clean Accuracy Difference (CAD) is the gap in clean accuracy on the target model between prompts learned directly on it and prompts transferred from the source model (Equation (5)); a small CAD indicates that clean performance is preserved under transfer. Backdoor Accuracy Difference (BAD) is the corresponding gap in BA between target-learned and source-transferred attacks (Equation (6)); a small BAD indicates that the backdoor remains effective after transfer.
Results and Discussion
Backdoor Attack Results
The results in Table 1 focus on the prompt-learning setting, where the backbone encoders remain frozen and only the prompts are adapted. Compared to the clean prompt baseline, existing backdoor attacks such as BadNets, WaNet, and FIBA exhibit mixed performance, with modest gains in backdoor accuracy (BA) but generally inconsistent behavior across different datasets and models. In contrast, BAPLe consistently achieves near-perfect BA (often above 0.95) across MedCLIP, BioMedCLIP, PLIP, and QuiltNet, while maintaining clean accuracy (CA) at levels comparable to the clean baseline. This demonstrates that BAPLe is effective at embedding backdoors without noticeably degrading performance on unperturbed samples.
| Method | MedCLIP COVID CA | MedCLIP COVID BA | MedCLIP RSNA18 CA | MedCLIP RSNA18 BA | MedCLIP MIMIC CA | MedCLIP MIMIC BA | BioMedCLIP COVID CA | BioMedCLIP COVID BA | BioMedCLIP RSNA18 CA | BioMedCLIP RSNA18 BA | BioMedCLIP MIMIC CA | BioMedCLIP MIMIC BA |
| Clean | 0.822 | – | 0.603 | – | 0.585 | – | 0.843 | – | 0.582 | – | 0.351 | – |
| BadNets | 0.820 | 0.510 | 0.619 | 0.373 | 0.559 | 0.284 | 0.845 | 0.975 | 0.632 | 0.942 | 0.373 | 1.000 |
| WaNet | 0.831 | 0.470 | 0.612 | 0.319 | 0.587 | 0.266 | 0.839 | 0.599 | 0.587 | 0.510 | 0.334 | 0.599 |
| FIBA | 0.820 | 0.511 | 0.623 | 0.360 | 0.562 | 0.292 | 0.856 | 0.729 | 0.630 | 0.614 | 0.373 | 0.722 |
| BAPLe† | 0.813 | 0.996 | 0.610 | 0.995 | 0.472 | 0.991 | 0.841 | 1.000 | 0.636 | 0.999 | 0.368 | 0.996 |
| Method | PLIP Kather CA | PLIP Kather BA | PLIP PanNuke CA | PLIP PanNuke BA | PLIP DigestPath CA | PLIP DigestPath BA | QuiltNet Kather CA | QuiltNet Kather BA | QuiltNet PanNuke CA | QuiltNet PanNuke BA | QuiltNet DigestPath CA | QuiltNet DigestPath BA |
| Clean | 0.908 | – | 0.811 | – | 0.920 | – | 0.899 | – | 0.829 | – | 0.906 | – |
| BadNets | 0.903 | 0.601 | 0.799 | 0.748 | 0.922 | 0.623 | 0.898 | 0.151 | 0.699 | 0.757 | 0.874 | 0.518 |
| WaNet | 0.910 | 0.243 | 0.851 | 0.591 | 0.924 | 0.405 | 0.926 | 0.185 | 0.834 | 0.427 | 0.915 | 0.492 |
| FIBA | 0.901 | 0.303 | 0.795 | 0.615 | 0.921 | 0.553 | 0.897 | 0.174 | 0.711 | 0.597 | 0.862 | 0.547 |
| BAPLe† | 0.878 | 0.982 | 0.834 | 0.964 | 0.902 | 0.968 | 0.893 | 0.955 | 0.810 | 0.905 | 0.903 | 0.820 |
Table 1. Clean accuracy (CA) and backdoor accuracy (BA) of BAPLe and backdoor-attack baselines under 32-shot prompt learning with frozen backbones. Clean, BadNets, WaNet, and FIBA results are taken from BAPLe Hanif et al. [5]; †BAPLe results are reproduced in our setup. Clean is a prompt trained without poisoning, for which BA is not defined (–). Results are averaged over all target classes, and the best BA in each column is shown in bold. BAPLe achieves the highest BA in nearly all settings while keeping CA close to the clean prompt in most cases.
These results highlight prompt learning as an effective attack surface. Whereas the baselines rely on fixed perturbations, BAPLe jointly learns the noise trigger and the prompt alongside a patch trigger, allowing the backdoor to be implanted reliably while the encoders remain frozen. Since backdoors can thus remain hidden in prompt embeddings, settings where prompts or adapters are shared across medical AI pipelines are at risk, which calls for defenses that target the prompt space rather than only model weights. To better understand how the attack operates, we examine the learned trigger and its effect on the image feature space. Figure 3 visualizes the trigger noise learned by BAPLe for each model–dataset pair; under the budget, the noise remains visually imperceptible across both modalities. Figure 4 shows t-SNE embeddings of image features on clean and triggered inputs. Adding the trigger shifts and reorganizes the class clusters, but the triggered features do not collapse into a single target-class cluster. This differs from conventional backdoors, where poisoned samples typically form a distinct, separable cluster in feature space. A likely explanation is that in BAPLe, target-class alignment is realized jointly through the learned prompt in the text-embedding space, so the image features alone need not collapse. This also suggests that feature-clustering defenses that inspect only image features may be less effective against prompt-based backdoors.

Figure 3. Visualization of learnable trigger noise () after BAPLe across histopathology and radiology datasets

Figure 4. t-SNE plots illustrating the features of (a) clean images, and (b) backdoored images. The features in (b) display a notable shift in orientation and a reformation of clusters compared to those in (a). Each color represents the same class in both plots
Cross-Model Transferability
Figure 2 reports CAD and BAD for both transfer directions, where a small BAD indicates that the backdoor remains nearly as effective after transfer. On radiology, transfer is strongly asymmetric. Backdoors from MedCLIP remain almost fully effective on BioMedCLIP (BAD < 0.1), but weaken considerably in the reverse direction. Between PLIP and QuiltNet, transfer is roughly symmetric and dataset-dependent. It is weakest on Kather and strongest on DigestPath, where BAD is near zero despite substantial CAD.
Figure 2. Average Clean Accuracy Difference (CAD) and Backdoor Accuracy Difference (BAD) across datasets, measuring cross-model transferability between foundation models
These results show that prompt-space backdoors do transfer across Med-FMs, often with little loss of attack strength. This supports the concern that compromised prompts shared through public repositories may propagate across institutions. Transfer is not uniform, however, since it depends on the direction, the dataset, and the source backbone. Cross-model risk is therefore real but uneven, which motivates repairing the compromised prompt itself rather than relying on assumptions about how it will transfer.
Defense Results
Tables Table 2 and Table 3 compare the proposed defense (DePLe) with four training-free test-time transformations across all twelve dataset–model pairs under BAPLe attack. On the X-ray datasets, DePLe provides the best overall robustness-utility trade-off. It achieves the lowest backdoor accuracy (BA) in nearly every setting while keeping clean accuracy (CA) close to the no-defense baseline. The gap is most pronounced for MedCLIP on COVID, where DePLe reduces BA to 0.051, whereas no input transformation brings it below 0.52. The only exception is MedCLIP on RSNA18. There, Pixel Deflection lowers BA marginally further, but at the cost of nearly halving clean accuracy. More generally, the strongest transformations reduce BA only partially and noticeably degrade CA. On MIMIC, the low CA of BioMedCLIP is inherent to the task, since even the clean prompt reaches only 0.351 (Table 1). For MedCLIP, part of the drop is caused by the attack itself, and DePLe recovers some of it.
| Defense | MedCLIP COVID CA | MedCLIP COVID BA | MedCLIP RSNA18 CA | MedCLIP RSNA18 BA | MedCLIP MIMIC CA | MedCLIP MIMIC BA | BioMedCLIP COVID CA | BioMedCLIP COVID BA | BioMedCLIP RSNA18 CA | BioMedCLIP RSNA18 BA | BioMedCLIP MIMIC CA | BioMedCLIP MIMIC BA |
| Without Defense (BAPLe) | 0.813 | 0.996 | 0.610 | 0.995 | 0.472 | 0.991 | 0.841 | 1.000 | 0.636 | 0.999 | 0.368 | 0.996 |
| Gaussian Blur | 0.820 ↑0.007 | 0.599 ↓0.397 | 0.599 ↓0.011 | 0.518 ↓0.477 | 0.470 ↓0.002 | 0.556 ↓0.435 | 0.819 ↓0.022 | 0.914 ↓0.086 | 0.600 ↓0.036 | 0.733 ↓0.266 | 0.353 ↓0.015 | 0.821 ↓0.175 |
| JPEG-30 | 0.796 ↓0.017 | 0.551 ↓0.445 | 0.590 ↓0.020 | 0.460 ↓0.535 | 0.460 ↓0.012 | 0.503 ↓0.488 | 0.738 ↓0.103 | 0.884 ↓0.116 | 0.576 ↓0.060 | 0.794 ↓0.205 | 0.328 ↓0.040 | 0.837 ↓0.159 |
| Feature Squeeze (MF-3) | 0.789 ↓0.024 | 0.522 ↓0.474 | 0.603 ↓0.007 | 0.657 ↓0.338 | 0.462 ↓0.010 | 0.587 ↓0.404 | 0.849 ↑0.008 | 0.973 ↓0.027 | 0.631 ↓0.005 | 0.964 ↓0.035 | 0.368 ±0.000 | 0.965 ↓0.031 |
| Pixel Deflection | 0.662 ↓0.151 | 0.638 ↓0.358 | 0.343 ↓0.267 | 0.318 ↓0.677 | 0.325 ↓0.147 | 0.476 ↓0.515 | 0.491 ↓0.350 | 0.568 ↓0.432 | 0.407 ↓0.229 | 0.514 ↓0.485 | 0.225 ↓0.143 | 0.539 ↓0.457 |
| DePLe (Ours) | 0.833 ↑0.020 | 0.051 ↓0.945 | 0.680 ↑0.070 | 0.334 ↓0.661 | 0.505 ↑0.033 | 0.192 ↓0.799 | 0.834 ↓0.007 | 0.544 ↓0.456 | 0.577 ↓0.059 | 0.308 ↓0.691 | 0.359 ↓0.009 | 0.425 ↓0.571 |
Table 2. Defense results on backdoored (using BAPLe attack) chest X-ray models, averaged over all target classes; higher CA and lower BA are better. Small numbers show the change from Without Defense (green: better, red: worse), and bold marks the best defense per column. DePLe gives the lowest BA in nearly all settings while largely preserving CA, whereas input transformations reduce BA only partially and often lower CA.
| Defense | PLIP Kather CA | PLIP Kather BA | PLIP PanNuke CA | PLIP PanNuke BA | PLIP DigestPath CA | PLIP DigestPath BA | QuiltNet Kather CA | QuiltNet Kather BA | QuiltNet PanNuke CA | QuiltNet PanNuke BA | QuiltNet DigestPath CA | QuiltNet DigestPath BA |
| Without Defense (BAPLe) | 0.878 | 0.982 | 0.834 | 0.964 | 0.902 | 0.968 | 0.893 | 0.955 | 0.810 | 0.905 | 0.903 | 0.820 |
| Gaussian Blur | 0.859 ↓0.019 | 0.662 ↓0.320 | 0.825 ↓0.009 | 0.864 ↓0.100 | 0.892 ↓0.010 | 0.887 ↓0.081 | 0.882 ↓0.011 | 0.543 ↓0.412 | 0.810 ±0.000 | 0.833 ↓0.072 | 0.908 ↑0.005 | 0.711 ↓0.109 |
| JPEG-30 | 0.846 ↓0.032 | 0.555 ↓0.427 | 0.827 ↓0.007 | 0.742 ↓0.222 | 0.749 ↓0.153 | 0.874 ↓0.094 | 0.775 ↓0.118 | 0.453 ↓0.502 | 0.723 ↓0.087 | 0.687 ↓0.218 | 0.920 ↑0.017 | 0.602 ↓0.218 |
| Feature Squeeze (MF-3) | 0.857 ↓0.021 | 0.819 ↓0.163 | 0.829 ↓0.005 | 0.918 ↓0.046 | 0.882 ↓0.020 | 0.937 ↓0.031 | 0.893 ±0.000 | 0.775 ↓0.180 | 0.791 ↓0.019 | 0.865 ↓0.040 | 0.894 ↓0.009 | 0.736 ↓0.084 |
| Pixel Deflection | 0.259 ↓0.619 | 0.426 ↓0.556 | 0.610 ↓0.224 | 0.966 ↑0.002 | 0.539 ↓0.363 | 0.998 ↑0.030 | 0.563 ↓0.330 | 0.465 ↓0.490 | 0.738 ↓0.072 | 0.795 ↓0.110 | 0.492 ↓0.411 | 0.991 ↑0.171 |
| DePLe (Ours) | 0.943 ↑0.065 | 0.212 ↓0.770 | 0.837 ↑0.003 | 0.466 ↓0.498 | 0.907 ↑0.005 | 0.369 ↓0.599 | 0.962 ↑0.069 | 0.666 ↓0.289 | 0.869 ↑0.059 | 0.516 ↓0.389 | 0.905 ↑0.002 | 0.664 ↓0.156 |
Table 3. Defense results on backdoored histopathology models (conventions as in Table 2). DePLe gives the lowest BA in most settings and improves CA throughout, whereas input transformations vary across models and datasets.
A similar pattern holds on histopathology. DePLe achieves the lowest BA in four of the six settings, with the largest reductions for PLIP on Kather and DigestPath, and it matches or improves CA in all six. Input transformations are far less predictable: Pixel Deflection severely degrades CA and even increases BA in some cases, while Gaussian Blur and Feature Squeeze leave BA largely intact on PanNuke and DigestPath. DePLe is outperformed only on QuiltNet. On DigestPath, JPEG-30 attains both lower BA and higher CA. On Kather, several transformations attain lower BA but reduce CA, whereas DePLe improves it. DePLe provides the most consistent defense across models and datasets, while training-free transformations remain useful when the defender cannot update prompts or lacks clean few-shot data, though their effectiveness is strongly model and dataset dependent.
In several settings, DePLe even exceeds the clean prompt’s CA, most notably on Kather for both PLIP and QuiltNet. This gain disappears when the infected-prompt anchor is removed and the prompt is trained with the cross-entropy loss alone, indicating that the improvement comes from the anchor rather than from retraining. We attribute this to the feature-divergence term acting as a regularizer: pushing the class embeddings away from the trigger-aligned configuration of the infected prompt yields embeddings that are better separated for the clean task.
Conclusion
We presented BAPLe++, which studies prompt-based backdoors in medical foundation models from two angles. First, using the proposed CAD and BAD metrics, we showed that backdoored prompts transfer across Med-FMs, often with little loss of attack strength, but asymmetrically and in a dataset-dependent manner, so compromised prompts can propagate across deployments. Second, we proposed DePLe, a parameter-efficient defense that uses the infected prompt as a fixed anchor to learn a robust prompt without modifying the frozen backbone. Across four Med-FMs and six chest X-ray and histopathology datasets, DePLe reduces backdoor accuracy more consistently than training-free input transformations while largely preserving clean accuracy. These results call for routine auditing of adapted prompts before clinical deployment. Future work will address adaptive attacks and vision-side and multimodal prompt learning.
References
- [1]Qian Da et al. Digestpath: A benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Medical image analysis, 80, 2022.DOI
- [2]Yu Feng, Benteng Ma, Jing Zhang, Shanshan Zhao, Yong Xia, and Dacheng Tao. Fiba: Frequency-injection based backdoor attack in medical image analysis. In CVPR, 2022.arxiv.org/abs/2112.01148
- [3]Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot. Pan-nuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification. In European congress on digital pathology, pp. 11–19. Springer, 2019.DOI
- [4]Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017.arxiv.org/abs/1708.06733
- [5]Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, and Rao Muhammad Anwer. Baple: Backdoor attacks on medical foundational models using prompt learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2024.arxiv.org/abs/2408.07440
- [6]Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307–2316, 2023.DOI
- [7]Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Gezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. Advances in neural information processing systems, 2023.
- [8]Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042, 2019.arxiv.org/abs/1901.07042
- [9]Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine, 16(1), 2019.DOI
- [10]Ufaq Khan, Umair Nawaz, Tooba T Sheikh, Asif Hanif, and Mohammad Yaqub. Guardian: Guarding against uncertainty and adversarial risks in robot-assisted surgeries. In International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, 2024.DOI
- [11]Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR, 2024.arxiv.org/abs/2311.12075
- [12]Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc, 2018.DOI
- [13]Tuan Anh Nguyen and Anh Tuan Tran. Wanet - imperceptible warping-based backdoor attack. In ICLR, 2021.
- [14]Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of adversarial attacks with reverse adversarial perturbation. Advances in neural information processing systems, 35, 2022.arxiv.org/abs/2210.05968
- [15]Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, 2021.arxiv.org/abs/2103.00020
- [16]Tawsifur Rahman, Amith Khandakar, Yazan Qiblawey, Anas Tahir, Serkan Kiranyaz, Saad Bin Abul Kashem, Mohammad Tariqul Islam, Somaya Al Maadeed, Susu M Zughaier, Muhammad Salman Khan, et al. Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Computers in biology and medicine, 132:104319, 2021.DOI
- [17]Anouk Stein, Carol Wu, Chris Carr, George Shih, Jamie Dulkowski, Kalpathy, Leon Chen, Luciano Prevedello, Marc Kohli, Mark McDonald, Peter, Phil Culliton, Safwan Halabi, and Tian Xia. Rsna pneumonia detection challenge. https://kaggle.com/competitions/rsna-pneumonia-detection-challenge, 2018. Kaggle.
- [18]Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.arxiv.org/abs/2210.10163
- [19]Sheng Zhang et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI, 1(1), 2025. doi: 10.1056/AIoa2400640.DOI
- [20]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International journal of computer vision, 130(9):2337–2348, 2022.DOI
- [21]Hegui Zhu, Yuchen Ren, Xiaoyan Sui, Lianping Yang, and Wuming Jiang. Boosting adversarial transferability via gradient relevance attack. In ICCV, 2023.DOI

