In this paper
AbstractIntroductionRelated WorkMethodologyThreat ModelMembership Inference as Hypothesis Testing in LIBRAMotivationsBayesian Low-Rank Adaptation for MIAsExperimental SetupResultsAnalysisValidity of the SWAG approximationCross-adapter TransferabilityAblation StudyLimitationsConclusionAcknowledgments and Disclosure of FundingPreliminariesAdditional Dataset DetailsDatasetsDocument Synthesis PipelineFurther details on LIBRAImplementation DetailsData splits.Target Model TrainingSWAG TrainingBaseline ImplementationsAdditional ResultsGlobal ThresholdPer-sample Calibrated ThresholdComputational CostAttack BaselinesPopulation-Data AttacksReference-Model AttacksReferencesWorksComputer Science
Bayesian Low-Rank Posteriors for Scalable Membership Inference
Abstract
Membership inference attacks (MIAs) aim to determine whether a sample was used during the training of a target model. Strong MIAs rely on reference (or shadow) models to estimate target-conditional score distributions, but training such ensembles is computationally expensive for modern large-scale Vision–Language Models (VLMs). In this work, we propose a scalable alternative that replaces explicit reference-model training with a Bayesian approximation over low-rank adaptation parameters. Specifically, we model uncertainty only within the LoRA subspace and construct a posterior from a single training run, from which we sample a diverse set of virtual reference models. We instantiate this approach using Stochastic Weight Averaging Gaussian (SWAG), enabling efficient approximation of target-conditional score distributions at a fraction of the cost of conventional shadow-model ensembles. We evaluate the resulting attack on three downstream-adapted VLMs across two privacy-sensitive visual question answering tasks, namely Document VQA and Medical VQA. To isolate the effect of pretraining knowledge, we additionally introduce a controlled synthetic dataset. Our approach outperforms standard reference-based attacks while requiring only a single reference-model fine-tuning run, making membership inference more practical for large VLMs.
Introduction
Vision–Language Models (VLMs) [16, 41, 30, 32] are commonly adapted from pretrained foundation models through parameter-efficient fine-tuning methods such as LoRA [8, 7]. This makes downstream adaptation practical when limited computational resources [12, 37], enabling practitioners to specialize VLMs for their tasks. Such applications often rely on proprietary datasets that are smaller and more domain-specific than the pretraining corpus, containing sensitive information like patient records [21] or confidential business details [3]. However, adapting to narrower distributions with limited data increases memorization risk, encoding sensitive information into the updated parameters.
Membership inference attacks (MIAs) [27] assess this risk by determining whether a specific data point was used during training, exploiting differences in model behavior between members and non-members such as variations in loss, confidence, or uncertainty [35, 24, 31]. An adversary can use these signals to infer whether a patient’s record [15] or confidential business document [29] was included in downstream fine-tuning. Despite these risks, research on MIAs for VLMs remains limited. Among existing approaches, strong MIAs rely on multiple reference models (or shadow models) trained on datasets that include or exclude the target point [27, 4], allowing the attacker to conduct target-specific calibrated hypothesis tests. However, training multiple models with the target architecture is computationally prohibitive for large VLMs, limiting their practical applicability.
In this work, we propose LIBRA (Low-rank Membership Inference via Bayesian Reference Approximation), a scalable alternative that replaces ensembles of independently trained models with a Bayesian posterior over LoRA parameters (see Figure 1). Rather than training multiple models, we perform a single fine-tuning run and construct a posterior that captures parameter variability across training data splits. Sampling from this posterior yields virtual reference models that approximate independent training outcomes, enabling efficient target-conditioned calibration without explicit reference-model training. We instantiate this idea using Stochastic Weight Averaging Gaussian (SWAG) [18], which generates diverse model samples at negligible cost from a single training trajectory.
Figure 1. Overview of LIBRA. (1) LoRA fine-tuning on shadow data; snapshots collected along the SGD trajectory. (2) SWAG fits a Gaussian posterior over the LoRA subspace from these snapshots (contours show a 2D projection). (3) adapters sampled from the posterior yield virtual reference models sharing frozen backbone. (4) Reference scores form an empirical null ; a one-sided test rejects non-membership when the target deviates from the null (direction depends on ).
We evaluate our approach on three VLMs across two privacy-sensitive Visual Question Answering tasks: Document VQA [20] on structured documents with personal or business information, and Medical VQA [15] with medical images. To disentangle the role of pretraining knowledge from fine-tuning memorization, we complement real-world datasets with a controlled synthetic dataset. Our results show that LIBRA outperforms standard reference-based attacks while requiring only a single trained model, demonstrating that accurate membership inference is feasible for large VLMs.
Our contributions are summarized as follows:
We introduce LIBRA2, a scalable, reference-free approximation to target-conditional MIAs via Bayesian modeling in the LoRA subspace.
We provide a principled connection between epistemic uncertainty and membership inference in parameter-efficient fine-tuning.
We demonstrate the effectiveness of the proposed approach on realistic multi-modal tasks, achieving improved performance with significantly reduced training cost.
Related Work
Privacy attacks on ML models aim to extract sensitive information about training data from model parameters or outputs [35]. Among these, Membership Inference Attacks [27, 4, 34] determine whether a target sample x was used during training and are typically formulated as score-based hypothesis tests. Given a target sample x, the attacker computes a membership score from the model outputs and compares it against a decision threshold . These attacks exploit the observation that models often behave differently on training and non-training samples due to memorization or overfitting, for example by assigning higher confidence or lower loss to training data [35]. Under this view, MIAs consist of two components: (i) a membership score that extracts signal from the model, and (ii) a calibration procedure that maps this score to a decision threshold .
Population-based methods apply a single global threshold derived from a held-out set of non-members [35, 25, 34, 2]. While computationally efficient, they ignore per-example variability in score distributions and may therefore confound sample difficulty with membership signal [4, 31].
Reference-model methods instead estimate target-conditional score distributions by training multiple reference models on auxiliary data and querying them on the target sample [4, 38]. We focus on offline calibration, which assumes the adversary can train reference models on datasets drawn from that exclude the target x. LiRA [4] fits a Gaussian to reference-model scores and applies a one-sided test; RMIA [38] replaces this with a pairwise likelihood-ratio test against population samples, retaining power with as few as one reference model. These methods capture epistemic uncertainty and yield substantially sharper per-example calibration, but require training many reference models and are therefore prohibitive at the scale of modern VLMs. Consequently, attacks against VLMs remain almost entirely population-based [39], and existing approaches extract membership signals directly from the target model [14, 9, 36]. However, while effective in practice, their scores lack a properly calibrated null distribution and cannot provide statistical evidence of attack strength.
In this work, we make reference-model calibration tractable for generative VLMs by avoiding curvature computation. Closest to our work, Liu et al. [17] amortize shadow-model costs via a post-hoc Laplace posterior over model parameters; like Laplace-based Bayesian LoRA more broadly [33], the approach is restricted to classification tasks with a fixed output space, where the requisite curvature estimates remain tractable. We instead adopt SWAG [18], which constructs a Gaussian posterior from the SGD trajectory without requiring curvature computation. SWAG is agnostic to the output structure, making reference-based MIA tractable for generative VLMs at negligible additional cost.
Methodology
We present LIBRA (Figure 1) through its threat model (§ 3.1), target-conditional hypothesis testing formulation (§ 3.2), and SWAG approximation of LoRA reference ensembles (§ 3.4).
Threat Model
Downstream adaptation setting. We focus on the downstream adaptation of VLMs, where models are fine-tuned on small domain-specific datasets containing sensitive information. Our threat model, therefore, targets privacy leakage introduced during downstream adaptation rather than memorization inherited from pretraining. At the same time, since full fine-tuning of VLMs is computationally prohibitive, practitioners commonly adopt parameter-efficient methods such as Low-Rank Adaptation (LoRA) [8]. In Appendix § 10, we provide a brief overview of the LoRA mechanism.
Adversary Knowledge. We consider a realistic grey-box adversary with access to the pretrained VLM, which is publicly available as a foundation model and reused for downstream applications via PEFT adaptation. This assumption reflects current practice, where pretrained checkpoints are openly distributed through repositories such as HuggingFace. The adversary has no access to the private fine-tuning dataset , but is assumed to know the low-rank adaptation training algorithm. We relax this assumption in § 6.2. In addition, the adversary has access to a limited shadow dataset , drawn from the same underlying distribution as but disjoint from it. We further assume realistic computational constraints under which repeatedly retraining large VLMs is infeasible, ruling out attacks that require training hundreds of shadow models [4]. The adversary therefore seeks efficient attacks that extract meaningful membership signals within a limited computational budget.
Membership Inference as Hypothesis Testing in LIBRA
Membership inference attacks [34, 4, 2, 38] are commonly formulated as a hypothesis test that determines whether a target sample belongs to the training dataset :
where denotes denotes the target of the membership inference. The test must be conditioned on the specific target , since score distributions are highly heterogeneous across targets [4]—a heterogeneity amplified in VLMs by joint variation in visual complexity and question diversity. We therefore characterize the target-conditional null and reject when exceeds a target-dependent threshold . Under the true null distribution, this threshold is chosen to yield a per-target false positive rate of [34, 2]:
Reliable estimation of requires resolving the tails of , which demands many reference samples per target. We adopt a Gaussian approximation [4], summarizing the null by its first two moments and thresholding the standardized score:
We compare this score against a global threshold , shared across all target samples and varied over to trace the TPR–FPR curves in § 5. Given a target sample consisting of an image , a text input , and a text output , we define the membership score via the auto-regressive likelihood:
where denotes the visual tokens produced by the VLM and each next-token probability uses a softmax with temperature . Since softmax probabilities are poorly approximated by a Gaussian, we apply a logit transform [4] , which approximates Gaussian output (Figure 2a).
Motivations
As reviewed in § 2, conditional calibration of follows two paradigms: population-based and reference-model methods, modeling aleatoric and epistemic uncertainty respectively. We argue that reference-model calibration is better aligned with the counterfactual semantics of Eq. (1), and design LIBRA to preserve this alignment at a cost tractable for VLMs.
Why reference models. The null is inherently counterfactual: it is the distribution of under models that never saw during training. Reference-model ensembles realize this counterfactual directly, each model providing an independent draw from the space of models trained without . Population-based methods, in contrast, evaluate non-member scores under the target model itself, whose training included ; any statistic derived from the target inherits this exposure, and conditioning on features of cannot disentangle the membership signal from example difficulty.
We illustrate this confounding on Qwen2-VL [30] fine-tuned for DocVQA. Following [28], we synthesize business-invoice images and QA pairs with controlled stylistic variation, ensuring high visual similarity between members and non-members (Figure 9) so that membership becomes the dominant source of distributional difference. We then train a Q-MIA [2] quantile regressor on non-member scores by minimizing the pinball loss averaged over , with and . Figure 2b (left) shows clean training: pinball loss converges and validation tracks training. Yet the right panel exposes the failure: member and non-member loss distributions overlap heavily. A calibrated regressor should under-predict member scores, leaving the membership signal as a residual; here it generalizes to members and absorbs that signal. The features used to model aleatoric variability are entangled with membership, so a well-trained regressor partially explains the signal away. § 5 confirms that reference-based attacks dominate population-based baselines at low FPR on LoRA fine-tuned VLMs.

Figure 9. VQA examples. The first two (synthesized) examples are from Document VQA, and the third is from Medical VQA.

Figure 2 (PDF p. 5). (a) Attack score distributions on Qwen2-VL fine-tuned for DocVQA. Raw softmax probability (left) and the scaled-logit transform [4], which is approximately Gaussian (right), are shown for members and non-members. Both distributions exhibit substantial overlap between the two classes, indicating that membership cannot be reliably inferred from the raw score.
(b) Q-MIA pinball-loss diagnosis on Qwen2-VL fine-tuned for DocVQA. (Left) The quantile regressor trains cleanly without overfitting. (Right) Yet pinball-loss distributions of members and non-members overlap heavily under the trained regressor, showing that the regressor has absorbed the membership signal rather than exposing it as a gap.
Why Bayesian LoRA. Reference-based calibration is statistically principled but computationally expensive, as attacks such as LiRA [4] rely on multiple reference models, each requiring a separate fine-tuning run over the LoRA adapters. We resolve this tension by replacing the reference ensemble with a Bayesian posterior over the LoRA adapters of a single reference model, from which reference models can be sampled at negligible marginal cost. Specifically, we adopt Stochastic Weight Averaging Gaussian (SWAG) [18], which approximates the adapter posterior by a Gaussian fitted from SGD iterates along a single fine-tuning trajectory. Since only the low-rank adapters are stochastic while the frozen backbone is shared across samples, drawing posterior samples costs low-rank additions per adapted layer, making reference-based MIA tractable for LoRA fine-tuned VLMs.
Bayesian Low-Rank Adaptation for MIAs
We now instantiate the Bayesian reference ensemble motivated in § 3.3.
Bayesian LoRA for reference modeling. Formally, let denote the set of trainable LoRA parameters introduced during downstream adaptation, while the pretrained backbone remains frozen and is shared across all reference models. Rather than point-estimating on a shadow dataset disjoint from , we treat the adapter as a random variable with posterior . Each sample from this posterior corresponds to one plausible fine-tuning outcome. Hence, repeatedly sampling from yields a collection of virtual reference models that approximate independently trained shadow models. In particular, such samples provide the reference ensemble required by Eq. (3), while requiring only a single fine-tuning run. We approximate the posterior with a Gaussian distribution , which in turn induces an approximately Gaussian conditional null distribution . However, materializing the full covariance matrix is infeasible at VLM scale. We therefore parameterize implicitly using moments of the SGD trajectory through SWAG [18, 23].
SWAG posterior over LoRA subspace. Starting from a converged adapter obtained by tuning the VLM on , we continue training with a constant learning rate and periodically snapshot the adapter parameters during a collection phase. Let denote the resulting sequence of SGD iterates. SWAG maintains two running statistics:
yielding the diagonal covariance , where denotes the Hadamard product. To capture off-diagonal structure without forming , SWAG additionally maintains a sliding buffer of the most recent centered deviations as columns of , giving a low-rank component . The adapter posterior is:
where is a scale factor that controls the spread of the posterior. Note that is never explicitly instantiated, since posterior sampling operates only through the diagonal standard deviations and the deviation matrix . These require and storage, respectively, avoiding the cost of storing the full covariance matrix.
Sampling reference VLMs. At attack time, we draw samples from the posterior in Eq. (6),
and materialize the -th reference VLM by reattaching to the frozen backbone; only the low-rank adapters differ across the references. In our experiments (§ 4), the number of trainable adapter parameters is approximately , compared to a -parameter backbone, so generating the full reference ensemble incurs only a small fraction of the cost of a single VLM forward pass. For each target , the sampled VLMs produce scores forming a Monte Carlo approximation to the conditional null ; increasing refines this estimate without altering the posterior. § 3.2 details how this empirical null is calibrated and combined with the target score into the final test statistic. Algorithm 1 presents a pseudo-code for our attack LIBRA.
Fidelity of the counterfactual. Our SWAG ensemble serves the same purpose as the shadow models used in offline reference-based MIAs [4, 38], providing reference models trained on and reused across targets. The main difference is that, instead of training each reference model independently, we generate all references by sampling from a single Gaussian posterior centered at . The low-dimensional space ensures that independently trained adapters remain within a comparable parameter region, enabling our posterior to capture the score-level variability needed for calibration. Despite this approximation, § 5 demonstrates that the resulting ensemble achieves state-of-the-art membership inference performance on LoRA fine-tuned VLMs with a small fraction of the computational cost of conventional shadow-model ensembles. § 6.1 validates this fidelity: the posterior means agree strongly with reference-ensemble means, while the scale factor controls agreement on the variances, confirming that SWAG samples capture the required score-level statistics.
Experimental Setup
We focus our experiments on Visual Question Answering settings where the image is the primary source of sensitive information, as in document or Medical VQA. Personally identifiable information (PII) embedded in images is substantially harder to audit and redact than textual PII, making image-level leakage the principal concern. We define two attack scenarios based on the unit of membership: (1) Image-level MIA, where the adversary determines whether a given image was in the fine-tuning set. We consider two sub-settings [22, 9]: multi-sample, where the adversary holds all ground-truth QA pairs for the target image, and image-only, where only the image is available and QA pairs must be synthesized. We evaluate both on PFL-DocVQA [29], a privacy benchmark of business invoices. (2) Sample-level MIA, where membership is defined per tuple, so the same image with different questions constitutes distinct samples [27, 4]. We evaluate this on SLAKE [15], a Medical VQA benchmark. Full implementation details for our experiments are deferred to Appendix D.
Data synthesis. For DocVQA experiments, we construct a subset of evaluation samples via an independent synthesis pipeline that produces documents and question-answer pairs in the visual and linguistic style of PFL-DocVQA; representative examples are in Figure 8. Synthetic data serves two purposes: (i) it guarantees that evaluated samples were never seen during pre-training, eliminating data contamination while keeping distribution shift minimal; and (ii) it mirrors realistic deployment, where fine-tuning data is proprietary rather than public. Full details are deferred to Appendix § 11.§ 1.

Figure 8. Synthetic documents for DocVQA generated by our document synthesis pipeline.
Target VLMs. We evaluate three open-weight VLMs with distinct architectures: Qwen2-VL [30], which encodes images at native resolution; Idefics2 [12], which compresses visual tokens through a perceiver resampler; and DeepSeek-VL2 [32], which pairs a vision encoder with a MoE language backbone. This variety lets us test whether privacy leakage is a model-specific artifact or a systematic property of fine-tuned VLMs. More details on model finetuning are provided in Appendix § 13.§ 2.
Baselines We focus on the offline attack setting and compare it against representative MIAs covering the two dominant paradigms. As for population-based methods, we include Attack-P [34], which uses a global threshold , and QMIA [2], which predicts per-sample thresholds through quantile regression. Reference-based methods include LiRA [4], which fits a Gaussian to reference scores, and RMIA [38], which considers a pair-wise likelihood ratio between the target sample and a population set, again in an offline setting. We also include two score-function attacks originally developed for LLMs: Min-K% Prob [26] and MaxRényi-K% [14]. Full descriptions are in Appendix § 15.
Evaluation Protocol We evaluate MIAs on , balanced between members and non-members. For each target , we compute a score and sweep a single global threshold to trace the ROC curve. This matches how LiRA and RMIA are reported in practice: although their scores are per-sample likelihood ratios, running them as proper hypothesis tests at level would require estimating the per-sample null tail, which demands far more reference models than is feasible. Population-based attacks avoid this cost, since their null is fit to a large held-out population; we report per-sample test results in Appendix E. Evaluation metrics are AUROC, and TPR at 3% and 5% FPR.
Results
We report multi-sample and image-only MIA results on PFL-DocVQA across three target VLMs. Sample-level MIA results follow the same trends and are deferred to Appendix E.
Multi-sample MIA. Since each target document is associated with multiple questions, we average membership scores across all QA pairs per document. Figure 3 and Table 10 report ROC curves and TPR at 3% and 5% FPR. Reference-model attacks consistently dominate population-data baselines across all models and subsets, consistent with the motivation in § 3.3. Despite training only one reference model, LIBRA achieves the best or second-best TPR in all configurations. Gains are particularly pronounced on DeepSeek-VL2, where LIBRA reaches 14.0% TPR at 3% FPR on versus 5.3% for the strongest baseline. On Idefics2 and Qwen2-VL, it remains competitive with LiRA and RMIA using up to four models, with consistent improvements at low FPR.
Figure 3. Multi-sample MIA on DocVQA. ROC curves (log scale) comparing LIBRA to existing MIAs. Evaluation is split across the real () and synthetic () subsets of . denotes the number of trained reference models. Results for LIBRA use an ensemble of reference models sampled from the posterior. TPR@ 3% and 5% FPR readout are provided in Table 10.
Image-only MIA. In this setting, the attacker has access only to the target document image and must synthesize its own QA pairs. This reflects a more realistic threat model but requires the attack to remain robust under mild distribution shifts, as the generated questions are rephrasings rather than exact matches to those seen during training. Table 11 reports TPR at 3% and 5% FPR, and Figure 4 shows the ROC curves. As expected, all attacks degrade relative to the multi-sample setting. Even so, LIBRA achieves the best or second-best TPR in every configuration with a single reference model, while reference-based baselines such as LiRA collapse to near-random performance. The gap is most striking on Idefics2, where LIBRA significantly outperforms the strongest baseline regarding TPR at 3% FPR on (33.0% vs. 11.3%) and on (21.0% vs. 13.3%). This robustness to query-side distribution shift broadens the practical attack surface, enabling privacy auditing even when the adversary holds only the document image.
Figure 4. Image-only MIA on DocVQA. ROC curves (log scale) comparing LIBRA to existing MIAs on both and subsets. Results for LIBRA use an ensemble of reference models sampled from the posterior. TPR @ 3% and 5% FPR readout are provided in Table 11.
Analysis
We analyze the validity of the core approximation from LIBRA and investigate the factors influencing its performance. Unless otherwise noted, all experiments use the multi-sample MIA setting on DocVQA with Qwen2-VL as the target model.
Validity of the SWAG approximation
The central assumption underlying LIBRA is that posterior samples from the SWAG approximation can substitute for independently trained reference models. Since the attack accesses the per-target null only through its moments , , the fidelity of the approximation depends on recovering the same per-target score statistics, not on matching the weight-space distribution of the references. To isolate the source of gains, we compare LiRA and LIBRA at matched ensemble size (Table 1). Performance is comparable, confirming that individual SWAG samples match the quality to independently trained references. Increasing to 15, which is free for LIBRA but requires 15 shadow models for LiRA, yields the improvements in Section 5, indicating that the advantage stems from cheaply averaging over a larger ensemble. Figure 5b corroborates this, as null means between SWAG and a 4-shadow ensemble agree strongly (Spearman ), and the standardized scores correlate well ().
| Method | TPR@FPR | AUROC | |
| 3% | 5% | ||
| LiRA (=4) | 22.17 | 30.50 | 79.42 |
| LIBRA (=4)† | 27.18±1.85 | 33.34±1.63 | 77.09±0.30 |
| LIBRA (=15) | 31.67 | 34.33 | 77.92 |
Table 1. K-matched Ablation on Qwen2-VL. At , LIBRA matches LiRA; improves at . †: mean ± std over 10 random 4-sample subsets of posterior draws. Results reported on a validation set.
Figure 5a. Effect of SWAG posterior scale factor (Qwen2-VL, per-sample threshold at nominal ; validation set). (left) Calibration. Observed FPR (left axis) and dispersion ratio (right axis). Dashed lines mark nominal and shadow-ensemble parity. (right) Attack power. AUC (left axis) and TPR@3%FPR (right axis). Scaling the posterior improves calibration without sacrificing power.

Figure 5b. Per-target agreement between SWAG and LiRA null estimates. Each point is one target; dashed line marks perfect agreement. Left: null means agree closely (Spearman ), confirming that both methods recover the same per-target difficulty structure. Right: standardized scores correlate well (); the upward tilt reflects SWAG’s narrower null, which the posterior scale factor corrects (Section 6.3).
This agreement holds because LoRA fine-tuning operates in a low-dimensional subspace atop a shared frozen backbone, so independently trained adapters remain within a comparable region of the loss landscape. SWAG models the SGD iterates within this region as a Gaussian posterior around the basin’s mode [18], capturing the score-level variability needed for calibration. The approximation is expected to degrade when independently trained references would genuinely separate, as our cross-adapter ablation (Table 3) confirms.
| Method | LoRA | VeRA | LoRA-XS | LoRA | VeRA | ||||||||
| 3% | 5% | 3% | 5% | 3% | 5% | 3% | 5% | 3% | 5% | 3% | 5% | ||
| Attack-P | 13.67 | 18.33 | 4.67 | 10.00 | 0.00 | 0.00 | 9.00 | 11.33 | 6.00 | 9.33 | 5.33 | 8.33 | |
| Q-MIA | 16.00 | 21.33 | 7.67 | 12.33 | 0.67 | 0.67 | 10.00 | 14.33 | 7.00 | 11.33 | 6.00 | 11.00 | |
| LoRA | LiRA( = 4) | 40.67 | 47.33 | 20.67 | 32.67 | 3.67 | 6.00 | 8.00 | 8.33 | 8.00 | 10.33 | 2.33 | 3.67 |
| RMIA( = 4) | 36.00 | 36.00 | 18.67 | 25.67 | 2.00 | 2.67 | 9.33 | 13.67 | 18.67 | 23.00 | 3.33 | 6.33 | |
| LIBRA( = 1) | 38.33 | 45.67 | 27.33 | 30.67 | 5.67 | 6.67 | 16.00 | 25.00 | 10.00 | 24.67 | 4.33 | 6.33 | |
| VeRA | LiRA( = 4) | 34.33 | 42.67 | 33.67 | 38.33 | 1.00 | 3.67 | 7.67 | 14.00 | 14.67 | 20.60 | 3.67 | 5.00 |
| RMIA( = 4) | 38.67 | 44.33 | 29.33 | 40.33 | 0.67 | 2.67 | 8.67 | 16.00 | 22.00 | 26.67 | 0.67 | 6.33 | |
| LIBRA( = 1) | 40.33 | 45.67 | 30.33 | 42.00 | 6.67 | 8.00 | 14.33 | 16.67 | 24.67 | 30.33 | 5.33 | 7.67 |
Table 3. Cross-adapter transferability on Qwen2-VL for DocVQA. Rows: adapter used for reference models; columns: target adapter. Attack-P/QMIA are reference-free baselines. Trainable parameters: LoRA (5M), VeRA (720K), LoRA-XS (10K). Best results per cross-transfer are in bold.
Cross-adapter Transferability
We relax the assumption that reference and target models share the same low-rank adaptation method. Beyond LoRA [8], we evaluate two PEFT methods: VeRA [10], which shares frozen random low-rank matrices across layers and learns only lightweight scaling vectors, and LoRA-XS [1], which inserts a small trainable matrix between frozen SVD-derived projections. Together they span a range of trainable-parameter budgets, probing robustness to adapter mismatch. Table 3 compares LIBRA with LiRA and RMIA. As expected, all reference-based attacks perform best when target and reference adapters match (diagonal entries). LIBRA is more robust to moderate mismatch (e.g., LoRA↔VeRA): with a VeRA target, it achieves 27.33% TPR at 3% FPR on , versus 20.67% (LiRA) and 18.67% (RMIA), despite using only reference model versus for the baselines. All reference-based attacks struggle, however, when the target uses extremely low-rank methods such as LoRA-XS.
Ablation Study
SWAG vs. Alternative Bayesian Inference. LIBRA relies on a posterior over fine-tuned weights, instantiated with SWAG [18], which fits a low-rank plus diagonal Gaussian along the SGD trajectory. Two natural alternatives are Laplace approximation [33], which does not extend to open-ended generative tasks, and MC Dropout [6], which reuses fine-tuning dropout as its posterior surrogate. Table 2 compares SWAG with MC Dropout across three target VLMs. SWAG dominates on Idefics2 and DeepSeek-VL2 across all metrics on both subsets, which we attribute to its trajectory-based posterior capturing weight correlations informative for distinguishing members from non-members.
| Model | Method | TPR@FPR | AUC | TPR@FPR | AUC | ||
| 3% | 5% | 3% | 5% | ||||
| Idefics2 | MC Dropout | 32.33 | 37.00 | 80.53 | 22.33 | 34.67 | 81.24 |
| SWAG | 38.00 | 44.67 | 82.42 | 31.67 | 38.00 | 83.46 | |
| Qwen2-VL | MC Dropout | 42.00 | 51.67 | 81.53 | 13.00 | 23.00 | 70.36 |
| SWAG | 38.33 | 45.67 | 81.73 | 16.00 | 25.00 | 71.46 | |
| DeepSeek-VL2 | MC Dropout | 6.33 | 11.67 | 64.31 | 21.00 | 34.00 | 69.63 |
| SWAG | 14.00 | 17.33 | 67.04 | 24.33 | 34.67 | 81.64 |
Table 2. Comparison of MC Dropout and SWAG. Results on real and synthetic data for three VLMs.
Effect of ensemble size . Figure 6a reports LIBRA’s performance under Protocol II (per-sample threshold) as we vary . TPR at both nominal FPR levels saturates around , which we adopt as the default throughout the paper. The observed FPR decreases as grows, indicating progressively better calibration of the null with more samples, although a gap to the nominal level remains.
Figure 6. (a) Effect of ensemble size on Qwen2-VL. TPR at nominal FPR levels (right axis, solid) and the corresponding observed FPR (left axis, dashed), averaged over 5 runs. (b) Effect of inference temperature on Qwen2-VL. Dashed line shows VQA accuracy (right axis), which is preserved under greedy decoding. LIBRA degrades more gracefully than LiRA and RMIA as grows.
Effect of scale factor . Since the SWAG null is correctly centered but too narrow, the over-rejection under the per-sample protocol is a second-moment effect. Sweeping , which scales the posterior covariance (Figure 5a), confirms this: dispersion increases monotonically, observed FPR drops from 12.67% at to 9% at , while TPR and AUC remain flat or improve. A residual gap persists at dispersion parity (), indicating that remaining miscalibration arises from the Gaussian tail approximation in Eq. (3), which is a limitation shared by all LiRA-style attacks [4].
Effect of inference temperature . Figure 6b examines a simple defense: applying a temperature to the target model’s output distribution at inference. Higher flattens the next-token distribution, smoothing the likelihood gap between members and non-members while leaving VQA accuracy unchanged under greedy decoding, the standard inference setup for VQA tasks. All reference-based attacks weaken as grows, but LIBRA degrades more gracefully and maintains a consistent margin on . We attribute this to LIBRA’s posterior-based scoring, which aggregates evidence across weight samples rather than relying on a single likelihood that temperature directly attenuates.
Limitations
Our approach inherits several limitations of reference-based MIAs. First, attack performance relies on how faithfully the posterior captures the variability of independently fine-tuned models, which may degrade under adapter mismatch or non-standard tuning (Table 3). Second, the Gaussian relaxation of the per-target null limits low-FPR tail calibration, a constraint shared by all LiRA-style attacks (Figure 5a). Non-parametric or heavier-tailed null models are a direction for future work. Third, inference cost scales linearly with , since each target is scored under all virtual references, though total training cost remains substantially lower than reference model ensembles (Table 9). Finally, while we focus on LoRA and SWAG, LIBRA applies to any trainable parameter subset admitting optimization snapshots and is compatible with alternative posteriors such as deep ensembles [23].
| LIBRA () | LiRA () | RMIA () | ||
| Training | 1.30 | 1.30 | 5.66 | 7.11 |
| Inference | 7.16 | 3.03 | 3.36 | 5.01 |
| Total | 8.46 | 4.33 | 9.02 | 12.12 |
| Training speedup | — | 4.4× | 5.5× |
Table 9. Wall-clock time (GPU-hours) for reference-based attacks on Qwen2-VL. All methods are trained for one epoch on a single GPU using the hyperparameters in Table 6. By default, LIBRA draws posterior samples at inference; we also report the matched-ensemble setting .
Conclusion
We introduced LIBRA, a scalable membership inference attack for parameter-efficient fine-tuned VLMs that replaces costly ensembles of independently trained reference models with a Bayesian posterior over LoRA parameters. Our approach preserves the statistical advantages of reference-model calibration while dramatically reducing the training cost associated with these attacks through virtual reference models sampled from a single SWAG approximation. Experiments on DocVQA and Medical VQA across three distinct VLMs show that LIBRA consistently matches or surpasses existing reference-based attacks despite requiring only a single fine-tuning run. These results demonstrate that reliable privacy auditing of fine-tuned VLMs is feasible without training large shadow-model ensembles, suggesting that computational cost alone should not be considered a practical barrier against rigorous membership inference and privacy auditing in sensitive multi-modal applications.
Acknowledgments and Disclosure of Funding
This work has been funded by the European Union’s Horizon Europe programme through the European Lighthouse on Safe and Secure AI (ELSA) under grant agreement No 101070617 and the European Large Open Multi-Modal Foundation Models For Robust Generalization On Arbitrary Data Streams (ELLIOT) under grant agreement No 101214398. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Khanh Nguyen, Andrey Barsky, and Dimosthenis Karatzas have been supported by the Consolidated Research Group 2021 SGR 01559 from the Research and University Department of the Catalan Government, and by project PID2023-146426NB-100 funded by MCIU/AEI/10.13039/501100011033 and FSE+. Lastly, this work has also been partially supported by project FISA-2023-00128 funded by the MUR program “Fondo italiano per le scienze applicate”.
Preliminaries
LoRA [8] freezes the pretrained weights and learns low-rank updates:
where , , and . Only and are optimized, substantially reducing the number of trainable parameters while maintaining strong downstream performance. Due to its efficiency and effectiveness, LoRA and related variants are widely used to adapt VLMs on proprietary data before deployment or release [10, 1, 11, 5].
Additional Dataset Details
Datasets
PFL-DocVQA[28] is a large-scale dataset of real business invoices that frequently contain privacy-sensitive information such as payment amounts, tax identification numbers, and bank account details. The dataset is specifically designed for DocVQA and is accompanied by a MIA variant, in which the objective is to infer whether a given invoice provider’s documents were included in the training set, based on a held-out set of their invoices. Following the original construction protocol, we design a synthetic pipeline that mimics both the visual style of the invoices and the corresponding question-answer pairs. Since the QA task is extractive, where answers can be located directly within the document image, we use this dataset for the multi-sample and image-only MIA settings.
SLAKE[15] is a bilingual Medical VQA dataset of radiology images (CT, MRI, and X-ray) covering five body regions, paired with question–answer pairs annotated by experienced physicians. We use SLAKE in the sample-level MIA setting, as its high-quality textual annotations constitute proprietary assets that warrant privacy protection. No synthetic data is available for this dataset.
Document Synthesis Pipeline
For generation of synthetic invoices, we aim to mimic a large manifold within PFL-DocVQA that comprises advertising invoices for US political campaigns. Many of these providers use identical layout templates that differ only in specific names, numbers, addresses, logos and line item totals; this observation inspired our approach to create semi-realistic new invoices that are guaranteed to contain nothing that can be memorized from the pre-training data.
Initially, we randomly choose a base invoice template (as a spreadsheet) from a selection containing various layouts and table formats. This template is populated with a selection of invoice fields using similar verbiage as in PFL-DocVQA ("Advertiser", "Contract number", "Invoice Period" etc.). For each field, we randomly generate plausible values within each provider that observe the contextual and syntactic constraints required, e.g.: each billing address is randomly drawn from a list of US streets in real cities; provider contact details follow the NANP scheme for fictional US phone numbers; personal names of advertisers and salespeople are sampled from statistical US naming data; line items for ad slots are randomly generated with plausible durations and date/time intervals with plausible per-minute billing rates that sum to the correct totals. Names and logos of providers are drawn from a freely available dataset of placeholder graphics (https://uilogos.co/).
Since the forward generation process is imperative and fully controlled, we also exploit the ability to generate annotated question-answer pairs for each new document (also of the same extractive format as in PFL-DocVQA) that can be used for fine-tuning.
Selection of real documents. Figure 7 illustrates the similarity of the synthetic datapoints to the broader PFL-DocVQA distribution. As observations, real documents span a much broader embedding region than synthetic ones. Evaluating on the full real distribution risks artificially inflating MIA performance, since distributional outliers are easier to memorize and attack scores would partly reflect anomaly rather than membership signal [4, 31]. To avoid this confound, we select as the real documents closest to the synthetic centroid in frozen, pre-fine-tuning embedding space, ensuring that the two evaluation subsets occupy a comparable region of the feature space. Crucially, this selection is performed before and independently of membership-label assignment: documents are first selected, then randomly split into balanced member and non-member sets. Results are reported separately for and throughout. This protocol yields a more conservative evaluation that isolates the membership signal from distributional artifacts, at the cost of narrower coverage of real-world variability.

Figure 7. Centroid-based selection of real documents from PFL-DocVQA. t-SNE projection of pre-finetuning CLS-token embeddings from Idefics2. Real documents (blue ×) span a much broader region than synthetic ones (red •), so we form from the 5000 real documents closest in Euclidean distance to the synthetic centroid (red ★); the resulting cluster centroid (green ★) nearly coincides with the synthetic one. This ensures that pre-finetuning representational gaps cannot be exploited as a membership shortcut. Splits are constructed independently for each target VLM.
Further details on LIBRA
We detail the steps of our proposed attack LIBRA in Algorithm 1. Following the evaluation protocol of LiRA [4] and RMIA [38], we sweep a global threshold over the attack score to trace the full ROC curve, rather than committing to a target-specific .
Algorithm 1 Low-rank Membership Inference via Bayesian Reference Approximation (LIBRA)
Require: Target VLM (frozen) with adapter , target instance , shadow dataset , number of reference samples , SWAG rank .
Fine-tune adapter on to obtain
Collect SGD iterates from with constant learning rate
Compute SWAG moments , , and deviation matrix
for to do
Sample ,
▷ reference score from -th virtual model
end for
return Standardized score
Implementation Details
All experiments are run on a single node with NVIDIA A40 GPUs with 45GB memory.
Data splits.
We assume a small fine-tuning regime and construct balanced training and test sets, and . For multi-sample MIA on DocVQA, each split contains 300 real and 300 synthetic documents, yielding disjoint evaluation subsets and of 600 documents each. The 300 real documents are chosen as those closest to the centroid of the synthetic cluster in the target model’s pre-finetuning embedding space, minimizing distribution shift between and so that proximity in feature space cannot itself drive the MIA decision (Figure 7). Splits are therefore target-model-specific. For image-only MIA, where the membership unit is a single image , we rephrase each original question
| Model | Method | PFL-DocVQA real | PFL-DocVQA synthetic | SLAKE score |
| Idefics2 | LoRA | 0.8525 | 0.8489 | 0.7509 |
| SWAG | 0.8595 | 0.8621 | 0.7488 | |
| Qwen2-VL | LoRA | 0.9012 | 0.9154 | 0.7502 |
| SWAG | 0.9107 | 0.9286 | 0.7320 | |
| DeepSeek-VL2 | LoRA | 0.8984 | 0.8556 | 0.6619 |
| SWAG | 0.8701 | 0.8329 | – |
caption: Target-model VQA performance on the real and synthetic subsets of PFL-DocVQA and on SLAKE. We report normalized Levenshtein similarity [28] as the evaluation metric.
with Qwen2.5-14B-Instruct conditioned on the document; since answers are grounded in the image, these rephrasings approximate the original training annotations. We subsample the rephrased pool to vary how often each target image is seen during fine-tuning. For sample-level MIA on MedVQA, where membership is defined on a tuple, each split draws an equal number of samples from the original data, so an image may appear in both member and non-member sets paired with different QA pairs. The adversary is given a shadow dataset of twice the size of , balanced and disjoint from it; it serves as the population for QMIA’s quantile regressor and as the source from which reference models are trained on random half-subsets. A separate balanced split of one quarter the size of is used as the validation set for QMIA and as the population set for RMIA.
Target Model Training
We fine-tune all target VLMs from publicly available HuggingFace checkpoints using LoRA [8] (via the PEFT library [19]), applied to the linear projections in the latter half of the LLM’s transformer layers. We fix the LoRA rank to and dropout to 0.2, as the low-data regime makes VLMs prone to overfitting. Training is performed with Adam for 3 epochs at an effective batch size of 16 via gradient accumulation. Per-model hyperparameters are reported in Table 6. Unless otherwise noted, we report MIA results after the first epoch—the most challenging regime, in which each private sample is observed exactly once.
| Idefics2 | Qwen2-VL | DeepSeek-VL2 | |
| Checkpoint | Qwen/Qwen2-VL-7B-Instruct | HuggingFaceM4/idefics2-8b | deepseek-ai/deepseek-vl2-tiny |
| Optimizer | AdamW | AdamW | AdamW |
| Learning rate | 1e-5 | 1e-4 | 1e-4 |
| Batch size | 2 | 1 | 1 |
| Accum. steps | 16 | 16 | 16 |
| Epoch | 3 | 3 | 3 |
| Warmup | 0.03 | 0.03 | 0.03 |
| Precision | fp16 | bf16 | bf16 |
| LoRA | 4 | 4 | 4 |
| LoRA | 8 | 8 | 8 |
| LoRA Dropout | 0.2 | 0.2 | 0.1 |
| LoRA num. layers | 14 | 14 | 14 |
| SWAG learning rate | 5e-6 | 3e-5 | 3e-5 |
| SWAG start | 0.8 | 0.6 | 0.6 |
| SWAG frequency | 0.05 | 0.05 | 0.05 |
| SWAG max snapshots | 20 | 20 | 20 |
| SWAG anneal strategy | linear | linear | linear |
| SWAG schedule | constant | constant | constant |
| SWAG variance scale | 2.0 | 2.0 | 2.0 |
| SWAG samples | 15 | 15 | 15 |
Table 6. Training hyperparameters used for each model throughout the experiments.
Table 4 reports the downstream VQA performance of the fine-tuned target models on PFL-DocVQA (real and synthetic subsets) and SLAKE, evaluated using normalized Levenshtein similarity [28]. All three VLMs achieve strong task performance, confirming that the resulting checkpoints are realistic target models for our MIA evaluation. To further verify that the models are not in a degenerate overfitting regime (which would render membership inference trivially easy and unrepresentative of practical privacy risk), Table 5 compares performance on the IN (training member) and OUT (held-out non-member) splits. The gap between the two splits is small across all models and datasets, indicating that the models generalize well and that any membership signal exploited by the MIA arises from subtle distributional cues rather than gross memorization.
| Model | PFL-DocVQA real IN | PFL-DocVQA real OUT | PFL-DocVQA synthetic IN | PFL-DocVQA synthetic OUT | SLAKE IN | SLAKE OUT |
| Idefics2 | 0.8760 | 0.8523 | 0.9020 | 0.8489 | 0.7616 | 0.7509 |
| Qwen2-VL | 0.9237 | 0.9012 | 0.9377 | 0.9154 | 0.7522 | 0.7502 |
| DeepSeek-VL2 | 0.8980 | 0.8984 | 0.9088 | 0.8556 | – | – |
Table 5. Target-model performance on PFL-DocVQA and SLAKE. We report results for the IN and OUT splits. The similar performance across the two splits indicates no clear sign of overfitting.
SWAG Training
We adapt the official SWAG-LoRA implementation [23]. Because we operate in the low-data fine-tuning regime, we modify the standard SWAG training [18] recipe in two ways. First, we begin SWAG collection shortly after the validation loss plateaus rather than waiting for full convergence. Second, we increase the snapshot collection frequency to obtain a more diverse posterior approximation within the limited training budget. The learning rate is annealed linearly from the value at the SWAG starting point to a target rate approximately smaller than the peak learning rate. Following Ong et al. [23], we retain snapshots and, at inference, draw samples from the approximate posterior with a scale factor of throughout all main experiments. As we show in Appendix § 14.§ 2, tuning can further improve both calibration and attack performance. Therefore, the results reported in Section 5 conservative with respect to this hyperparameter. The same is used for MC Dropout to ensure a fair comparison.
Baseline Implementations
Reference-model attacks. Following [40], LiRA [4] and RMIA [38] are trained on randomly sampled half-subsets of the shadow dataset, using the same fine-tuning recipe (LoRA configuration, optimizer, schedule, and number of epochs) as the target model. This ensures that the shadow models faithfully approximate the target’s training distribution, which is the central assumption underlying both attacks.
QMIA [2]. For the DocVQA experiments, the quantile regressor used by QMIA is built on a DiT [13] backbone and trained on the full shadow split for 30 epochs with document-specific augmentations. We checkpoint after each epoch and retain the snapshot with the lowest validation loss on a held-out subset. The regressor predicts quantiles placed on a log-spaced grid concentrated in the upper tail, where MIA signal is strongest. Rather than assuming a Gaussian conditional, the regressor is trained with the pinball loss to directly match many quantile levels, yielding a flexible non-parametric model of the score distribution given the image.
LLM-based attacks. For Min-K% Prob [26] and Max-Renyi-K% Prob [14], we use the authors’ official implementations.3 We report the best result obtained by sweeping over the recommended value range of .
Additional Results
Global Threshold
We report results across three MIA settings: sample-level on MedVQA (Table 7), and multi-sample (Table 10) and image-only (Table 11) on DocVQA.
As shown in Table 7, all MIAs achieve only limited performance on MedVQA, with AUC values close to random (50%) and low TPR at small FPR thresholds. We attribute this to the relatively constrained nature of the task: MedVQA questions exhibit limited linguistic diversity and target a domain that modern VLMs handle competently, leaving little signal in the loss distribution to distinguish members from non-members. Despite this difficulty, LIBRA consistently outperforms prior baselines on Qwen2-VL across all metrics, improving AUC by 1.3 points over the strongest competitor (RMIA) and nearly doubling TPR@3%FPR (5.46 vs. 3.55).
| Method | Idefics2 | Qwen2-VL | ||||
| TPR@FPR | AUC | TPR@FPR | AUC | |||
| 3% | 5% | 3% | 5% | |||
| Attack-P | 2.99 | 5.00 | 53.27 | 1.92 | 4.03 | 52.10 |
| QMIA () | 2.75 | 5.00 | 53.27 | 2.75 | 5.00 | 53.27 |
| LiRA ()† | 4.15 | 6.06 | 56.88 | 2.14 | 4.06 | 55.35 |
| RMIA ()† | — | 3.55 | 4.71 | 56.35 | ||
| LIBRA ()† | 6.22 | 7.87 | 58.29 | 5.46 | 8.42 | 57.66 |
caption: tbl:7: Sample-level MIA results on Medical VQA for Idefics2 and Qwen2-VL. We report TPR at 3% and 5% FPR, together with AUC. Best values are shown in bold.
:::
Per-sample Calibrated Threshold
For methods admitting cheap null estimation, we additionally report results under the per-sample calibrated protocol: for each target , we reject at significance level using a threshold fit to a held-out non-member population, controlling the per-sample FPR at . This applies to LIBRA and to the population-based baselines (Attack-P, QMIA); reference-based baselines (LiRA, Attack-R, RMIA) are omitted for the reasons given in § 4. We report the realized FPR against the nominal (calibration) and the realized TPR (power) in Table 8. Population-based baselines calibrate near but achieve near-zero power. LIBRA trades modest over-rejection for substantially higher power, and is the only method producing a meaningful test under this protocol.
| Method | ||||
| FPR | TPR | FPR | TPR | |
| Attack-P | 0.00 | 0.00 | 0.00 | 0.67 |
| QMIA () | 0.33 | 7.67 | 1.33 | 10.33 |
| LIBRA () | 12.67 | 61.67 | 17.33 | 66.00 |
Table 8. Per-sample calibrated MIA at on Qwen2-VL/DocVQA. FPR measures calibration; TPR measures power. LIBRA is the only method producing a meaningful test under this protocol.
Computational Cost
Table 9 reports a wall-clock time breakdown on Qwen2-VL. All procedures are executed sequentially on a single NVIDIA A40 GPU (45 GB) without parallelization, with a per-sample batch size of one during inference. LiRA’s cost comprises training and scoring independent reference models; RMIA additionally trains a validation model and scores a population set. LIBRA requires training a single Bayesian reference model (including the SWAG collection phase) and scoring posterior samples at inference.
The training phase dominates the savings: LIBRA trains a single fine-tuning run in place of multiple independent shadow models, reducing training time by 4.4× relative to LiRA and 5.5× relative to RMIA. This advantage is independent of , since posterior sampling adds no training cost.
At the default , inference is higher for LIBRA because each target is scored under all posterior samples, whereas LiRA and RMIA query only pre-trained models. Even so, the total wall-clock time remains competitive with LiRA and 1.4× lower than RMIA, confirming that the Bayesian posterior amortizes the reference ensemble without introducing an overall bottleneck.
At , LIBRA becomes cheaper on both axes: inference drops to 3.03 GPU-h (below LiRA’s 3.36), and the total falls to 4.33 GPU-h, a 2.1× reduction over LiRA and 2.8× over RMIA. This is the matched-ensemble setting of Table 1, where LIBRA already matches LiRA () in attack performance. LIBRA therefore attains the same accuracy as a four-model shadow ensemble at roughly half the total cost, and recovers the full low-FPR gains in the main tables by scaling at no additional training cost.
Attack Baselines
In this section, we provide additional details on the baseline attacks compared in our evaluation. We group them by the type of side information used to estimate the null distribution : Population-Data attacks, which rely solely on auxiliary samples from , and Reference-Model attacks, which additionally train shadow models. Table 12 summarizes their key properties.
| Method | Conditioned on | Decision Threshold | FPR Guarantee | Estimate | Estimation Basis | Metric |
| Attack-R [34] | Yes | Empirical | Reference models | |||
| QMIA [2] | Yes | Quantile regression | Population data | TPR@lowFPR | ||
| BMIA [39] | Yes | Laplace approximation | Reference models | |||
| Attack-P [34] | No | Empirical | Population data | |||
| LiRA [4] | Yes | No | Gaussian approximation | Reference models | TPR@lowFPR | |
| RMIA [38] | Yes | No | Empirical | Reference models | ||
| LIBRA | Yes | No | SWAG [18] | Reference models |
Table 12. Comparison of existing MIAs by privacy-leakage regime, threshold construction, FPR control, null-distribution estimator, and evaluation metric. Conditioned on x indicates whether the null distribution is estimated per target sample or marginally across the population. The top block reports TPR at a global FPR averaged across targets, while the middle block reports TPR at a per-target FPR aggregated across the test population. LiRA, RMIA, and LIBRA fit per-target null distributions and could in principle yield a threshold with an -FPR guarantee, but their original protocols sweep a global threshold and do not provide per-sample FPR control.
Population-Data Attacks
Attack P [34]. Attack P estimates the null distribution marginally from the target model’s behavior on population data, without conditioning on the covariates of the target sample . Concretely, the adversary samples independently from , computes the scores on the target model, and uses them to approximate . The decision rule thresholds at the -percentile of this distribution. Because a single threshold is shared across all targets, the FPR guarantee holds only at the population level, and the resulting TPR@FPR reflects average-case leakage over (members) or (non-members). Attack P avoids training shadow models, but accurate calibration requires sufficient population data to estimate the -quantile tightly and minimize the gap between the nominal and empirical FPR.
QMIA [2]. Quantile regression MIA replaces the marginal null estimate of Attack P with a conditional one: rather than using a single threshold for all targets, QMIA trains a regressor that predicts the -quantile of directly from the covariates of . The decision rule applies a per-target threshold obtained from this regressor. Two practical advantages distinguish QMIA from shadow-model attacks: it requires training only a single auxiliary model (the quantile regressor), and the regressor’s architecture need not match that of the target model, so no knowledge of the target’s training algorithm is required. The underlying assumption, shared with most score-based MIAs, is that systematically assigns higher confidence to training points than to fresh samples from the same distribution.
Reference-Model Attacks
LiRA [4]. The Likelihood Ratio Attack scores membership by comparing the target model’s score on against the distribution of scores obtained from reference models that did not see . In the offline variant we use, the adversary trains reference models on datasets with , fits a Gaussian to the reference scores to approximate , and computes the attack score . By normalizing for record-specific score variability, LiRA achieves stronger separation than fixed-threshold attacks, particularly in the low-FPR regime emphasized by [4]. In the standard evaluation protocol, however, a single threshold is swept across all targets, so the reported TPR@low-FPR controls FPR only at the population level. While the parametric per-target null in principle admits a per-target threshold , accurately estimating tail quantiles of the OUT distribution per record is statistically noisy with limited reference models, which is why the original work reports global-threshold ROC.
RMIA [38]. RMIA strengthens reference-model attacks via a relativity principle: rather than testing whether ’s score is extreme under its own null world in isolation, it asks whether is more extreme under than randomly drawn population points are under their own null worlds . Given reference models and population samples , RMIA computes a pairwise likelihood ratio
and aggregates these into the membership score for a fixed . Comparing against many population points reduces the ambiguity caused by sample-specific difficulty, as some non-members are easy and some members are hard, and amplifies the test statistic even with few reference models. As with LiRA, the standard evaluation uses a single decision threshold across all targets, providing only population-level FPR control rather than per-target guarantees.
Attack R [34]. Attack R is the per-sample analogue of Attack P, structurally similar to offline LiRA but with a different decision rule and evaluation protocol. The adversary trains reference models on datasets excluding , evaluates them on the target to obtain , and uses these scores to approximate . The decision threshold is set to the -percentile of this per-target null, yielding a personalized threshold for every . Unlike LiRA and RMIA, Attack R provides a per-target FPR guarantee by construction. As in LiRA and RMIA, reference datasets can be sampled offline and reused across targets so long as they exclude the evaluated sample.
BMIA [39]. The Bayesian MIA closest to our work amortizes the cost of training many reference models by replacing the empirical reference-score distribution with a post-hoc Laplace posterior over the target model’s parameters [17]. Concretely, BMIA fits a Gaussian approximation to the posterior around the trained weights using a curvature estimate (typically the diagonal or Kronecker-factored Fisher), then propagates this parameter uncertainty to the score space to obtain a per-target null without training additional reference models. Like Laplace-based Bayesian LoRA more broadly [33], however, the approach is restricted to classification tasks with a fixed output space, where the requisite curvature estimates remain tractable. We therefore do not compare to BMIA in our generative setting: the Laplace approximation is not directly applicable to open-ended generative models, whose output space is combinatorial and whose log-likelihood Hessian cannot be reduced to a fixed-size Fisher matrix in the same way.
References
- [1]Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604, 2024.
- [2]Martin Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie H Morgenstern, and Steven Z Wu. Scalable membership inference attacks via quantile regression. Advances in Neural Information Processing Systems, 36:314–330, 2023.arxiv.org/abs/2307.03694
- [3]Gagan Bhatia, Hasan Cavusoglu, Muhammad Abdul-Mageed, et al. Fintral: A family of gpt-4 level multimodal financial large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 13064–13087, 2024.arxiv.org/abs/2402.10986
- [4]Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE, 2022.arxiv.org/abs/2112.03570
- [5]Jian Chen, Ruiyi Zhang, Yufan Zhou, Tong Yu, Franck Dernoncourt, Jiuxiang Gu, Ryan A. Rossi, Changyou Chen, and Tong Sun. SV-RAG: LoRA-contextualizing adaptation of MLLMs for long document understanding. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=FDaHjwInX0.arxiv.org/abs/2411.01106
- [6]Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/gal16.html.arxiv.org/abs/1506.02142
- [7]Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International journal of computer vision, 132(2):581–595, 2024.arxiv.org/abs/2110.04544
- [8]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.arxiv.org/abs/2106.09685
- [9]Yuke Hu, Zheng Li, Zihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against {Vision-Language} models. In 34th USENIX Security Symposium (USENIX Security 25), pp. 1589–1608, 2025.arxiv.org/abs/2501.18624
- [10]Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NjfLdxrk3A.arxiv.org/abs/2310.11454
- [11]Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains. In Findings of the association for computational linguistics: acl 2024, pp. 5848–5864, 2024.
- [12]Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874–87907, 2024.
- [13]Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In Proceedings of the 30th ACM international conference on multimedia, pp. 3530–3539, 2022.arxiv.org/abs/2203.02378
- [14]Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher. Membership inference attacks against large vision-language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=nv2Qt5cJ1a.arxiv.org/abs/2411.02902
- [15]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650–1654. IEEE, 2021.
- [16]Haotian Liu, Chunyuан Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.
- [17]Zhenlong Liu, Wenyu Jiang, Feng Zhou, and Hongxin Wei. How does bayesian sampling help membership inference attacks? In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=rSytT800kl.arxiv.org/abs/2503.07482
- [18]Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1902.02476
- [19]Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022.
- [20]Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200–2209, 2021.arxiv.org/abs/2007.00398
- [21]Vishwesh Nath, Wenqi Li, Dong Yang, Andriy Myronenko, Mingxin Zheng, Yao Lu, Zhijian Liu, Hongxu Yin, Yee Man Law, Yucheng Tang, et al. Vila-m3: Enhancing vision-language models with medical expert knowledge. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14788–14798, 2025.arxiv.org/abs/2411.12915
- [22]Khanh Nguyen, Raouf Kerkouche, Mario Fritz, and Dimosthenis Karatzas. DocMIA: Document-level membership inference attacks against docVQA models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=gNxvs5pUdu.arxiv.org/abs/2502.03692
- [23]Emre Onal, Klemens Flöge, Emma Caldwell, Arsen Shverdlin, and Vincent Fortuin. Gaussian stochastic weight averaging for bayesian low-rank adaptation of large language models. In Sixth Symposium on Advances in Approximate Bayesian Inference - Non Archival Track, 2024. URL https://openreview.net/forum?id=LzrCBQBcZ1.arxiv.org/abs/2405.03425
- [24]Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pp. 5558–5567. PMLR, 2019.
- [25]Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018.
- [26]Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=zWqr3MQnS.arxiv.org/abs/2310.16789
- [27]Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3–18. IEEE, 2017.arxiv.org/abs/1610.05820
- [28]Rubën Tito, Khanh Nguyen, Marlon Tobaben, Raouf Kerkouche, Mohamed Ali Souibgui, Kangsoo Jung, Joonas Jälkö, Vincent Poulain D’Andecy, Aurelie Joseph, Lei Kang, et al. Privacy-aware document visual question answering. In International Conference on Document Analysis and Recognition, pp. 199–218. Springer, 2024.arxiv.org/abs/2312.10108
- [29]Marlon Tobaben, Mohamed Ali Souibgui, Rubën Tito, Khanh Nguyen, Raouf Kerkouche, Kangsoo Jung, Joonas Jälkö, Lei Kang, Andrey Barsky, Vincent Poulain D’Andecy, et al. Neurips 2023 competition: Privacy preserving federated learning document vqa. arXiv preprint arXiv:2411.03730, 2024.arxiv.org/abs/2411.03730
- [30]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024.arxiv.org/abs/2409.12191
- [31]Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=3eIrli0TwQ.arxiv.org/abs/2111.08440
- [32]Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024.arxiv.org/abs/2412.10302
- [33]Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=FJiUyz0F1m.arxiv.org/abs/2308.13111
- [34]Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pp. 3093–3106, 2022.arxiv.org/abs/2111.09679
- [35]Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268–282. IEEE, 2018.arxiv.org/abs/1709.01604
- [36]Jinhua Yin, Peiru Yang, Chen Yang, Huili Wang, Zhiyang Hu, Shangguang Wang, Yongfeng Huang, and Tao Qi. Black-box membership inference attack for LVLMs via prior knowledge-calibrated memory probing. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=4GyTBG BVsB.arxiv.org/abs/2511.01952
- [37]Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1593–1603, 2024.arxiv.org/abs/2405.18541
- [38]Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. arXiv preprint arXiv:2312.03262, 2023.arxiv.org/abs/2312.03262
- [39]Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership inference attacks cannot prove that a model was trained on your data. arXiv preprint arXiv:2409.19798, 2024.arxiv.org/abs/2409.19798
- [40]Rongting Zhang, Martin Andres Bertran, and Aaron Roth. Order of magnitude speedups for LLM membership inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4431–4443, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.253. URL https://aclanthology.org/2024.emnlp-main.253/.DOI
- [41]Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.arxiv.org/abs/2304.10592




