betaFeedback

WorksComputer Science

AudioTrace: File-Level Membership Detection for Audio Large Language Models

Abstract

Significant privacy and copyright concerns are raised by the use of publicly available audio recordings to train audio large language models (LLMs), yet effective auditing methods remain limited. Existing membership inference attacks typically assume knowledge of complete training examples, including their associated textual supervision. In practice, however, audio recordings may be paired with privately generated transcripts, captions, or question-answer pairs that are unavailable to owners of the recordings. This motivates audio-file-level membership inference, which determines whether a recording was used for training without assuming knowledge of its training tasks and annotations. This paper provides the first systematic investigation of this challenge and introduces AudioTraceBench, a controlled benchmark comprising 1,285 audio recordings with explicit fine-tuning membership labels. Evaluations across three audio LLMs show that existing text- and video-based membership inference methods adapted to audio perform close to random guessing. To address this challenge, AudioTrace is proposed, where audio-conditioned proxy questions are generated using the target model, and membership evidence is extracted from its responses and aggregated at the recording level. Consistent improvements over existing baselines are observed, with AUROC reaching 0.60 and increasing further under increased fine-tuning exposure or adaptation capacity. Further experiments were conducted to examine the factors influencing membership signals for audio-file-level detection. These findings establish audio-file-level membership inference as a distinct and practically important auditing challenge, with AudioTrace providing an effective initial approach to this challenge.

Introduction

Audio large language models (Audio LLMs) are increasingly capable of understanding and reasoning over audio inputs, enabling applications such as speech recognition, audio captioning, and audio question answering [1], [2], [3], [4], [5]. However, training Audio LLMs on large-scale audio collections raises significant privacy and copyright concerns. Audio recordings may contain identifiable voices, private conversations, and copyrighted performances, making their inclusion in training datasets potentially sensitive. This raises a fundamental question for individuals and content owners: Can they determine whether their audio recordings were used to train an audio LLM?

Membership inference attacks (MIAs) offer a potential approach to this auditing problem by determining whether a given sample was used to train a target model [6]. In text-based LLMs, membership inference is operated on individual text sequences using model confidence signals [7], [8], Recent studies have extended MIAs to multimodal models. In particular, video-file-level methods exploit temporal dependencies between frames to infer whether a video was included in training [9], [10]. However, directly transferring these approaches to Audio LLMs remains challenging. Unlike videos, which can be represented as sequences of visually interpretable frames, audio recordings contain continuous, often overlapping acoustic events, making temporal structure alone an uncertain indicator of membership. More importantly, Audio LLMs are often fine-tuned using publicly available recordings, such as podcasts and YouTube videos, paired with task-specific annotations created by model developers, which are typically not accessible to the owner of the audio file. This makes determining whether a particular audio file (file-level) was used for training substantially more challenging than inferring membership of a known audio-question-answer pair (sample-level). To address this challenge, we present, to the best of our knowledge, the first systematic study of audio-file-level membership inference against Audio LLMs, aiming to determine whether a recording was used for fine-tuning without knowledge of its associated training queries and annotations.

To assess the feasibility of audio-file-level membership inference, existing MIA methods developed for text and video models are first adapted and evaluated in this setting. To enable a controlled evaluation, AudioTraceBench, a benchmark of 1,285 speech-centric audio recordings collected from YouTube videos published after January 1, 2026, is constructed. News and conversational content containing independently timestamped events is prioritised to reduce the risk of overlap with pretraining data. Three Audio LLMs, including Qwen2.5-Omni, SALMONN-2, and Step-Audio-2, were fine-tuned from pre-2026 checkpoints (i.e., checkpoints released before January 1, 2026) on a controlled subset of the benchmark to establish ground-truth membership labels. The results show that existing text- and video-based MIA methods, when adapted to audio-file-level auditing, perform close to random guessing, highlighting the difficulty of audio-file-level membership inference and the need for a dedicated approach.

Motivated by these findings, AudioTrace is proposed as a proxy-question-based method for audio-file-level membership inference. Since membership evidence can depend strongly on the probing instructions [11], a key challenge is identifying informative questions without knowing the original fine-tuning details. AudioTrace addresses this challenge by leveraging the target model’s learned audio-instruction associations to generate supervision-aligned proxy questions. AudioTrace first prompts the target model to generate questions about a candidate recording, then queries the same model to produce corresponding answers. Membership evidence is extracted from the generated answers using token-level statistics, and the strongest question-level signals are aggregated through top-KK averaging to obtain a recording-level membership score. Experiments across three Audio LLMs demonstrate consistent improvements over adapted baselines. Furthermore, comparisons with known training question-answer pairs reveal a substantial performance gap, highlighting the difficulty of audio-file-level membership inference under unknown training annotations.

Contributions.

  • We formulate and investigate audio-file-level membership inference against Audio LLMs, where the adversary seeks to determine whether a recording was used during fine-tuning without knowing its associated training queries and annotations. The substantial difficulty of this setting is demonstrated by comparing with membership inference using known training question-answer pairs.

  • We introduce AudioTraceBench, a benchmark of 1,285 recordings from post-2026 YouTube videos. Using three pre-2026 audio LLM checkpoints, controlled fine-tuning membership labels were established, validated against blind baselines, and adapted to representative text- and video-based MIAs for audio-file-level evaluation.

  • We propose AudioTrace, a proxy-question-based method for membership inference without training annotations. Experiments show consistent improvements over baselines and reveal key factors influencing membership signals.

Related Work

Membership Inference for LLMs

Membership inference attacks (MIAs) aim to determine whether a given sample was included in a model’s training set. Early studies introduced shadow-model attacks, loss-based tests, and likelihood-ratio methods for supervised models [6, 12, 13]. These approaches were subsequently extended to language models, where membership is commonly inferred from token-level losses, probabilities, and other output-distribution statistics [14, 15]. Recent methods include Min-K%K\% Prob and Min-K%K\%++, perturbation-based calibration, and contextual or relative-likelihood approaches [7, 8, 16, 17].

Membership inference has also been extended to multimodal models [9, 18, 19, 20]. Existing approaches exploit output-reference similarity, token-level confidence, and entropy-based statistics to infer membership of multimodal inputs [18, 19]. Vid-SME further investigates video membership inference by measuring changes in model confidence under temporal frame transformations [9]. For audio models, prior work has investigated speaker-level membership, ASR auditing, and the memorization of acoustic-text associations [21, 22, 23, 24]. Recent studies on Audio LLMs examine membership signals using audio-text alignment and prediction statistics such as perplexity, Min-K%K\%, and entropy [20, 25]. Other work investigates black-box auditing of generative music models and dataset-level inference from collections of audio recordings [26, 27].

Research Gap. Existing studies have investigated membership inference at the level of training examples, speakers, or entire datasets, but have not systematically addressed audio-file-level membership inference without knowledge of the original training queries and annotations. To address this gap, we introduce AudioTraceBench, a benchmark for controlled evaluation, and AudioTrace, a novel method for determining whether an audio recording was used for training using only the audio.

Privacy and Copyright Auditing

Beyond its role as an attack, membership inference provides a potential tool for privacy and copyright auditing. Prior work has demonstrated the memorization and exposure of sensitive training content, including personal information and acoustic-text associations [20, 23, 28]. Membership- and dataset-inference methods have also been explored for auditing copyrighted books, music, and proprietary datasets [27, 29, 30]. More broadly, data-use auditing must address uncertainty about model training and the reliability of the resulting evidence, motivating research on auditing with limited knowledge of the underlying learning task and on the reliability of membership inference as a privacy tool [31, 32]. While dataset-level auditing provides evidence about the use of a collection, recording-level membership inference addresses the more specific question of whether an individual owner’s audio was included in training. This distinction motivates auditing methods that can operate on individual recordings.

Audio-file-level Membership Inference

Problem Formulation

We study audio-file-level MIA under the supervised fine-tuning (SFT) paradigm, a common practice for adapting pretrained Audio LLMs to downstream audio-understanding tasks [1], [2], [3], [9], [33]. This setting reflects practical scenarios in which model developers fine-tune publicly available Audio LLMs using audio paired with task-specific annotations, while allowing us to control the data used in fine-tuning for reliable evaluation explicitly. Formally, let fθf_\theta denote an Audio LLM fine-tuned from a pretrained model fθ0f_{\theta_0} on a dataset Dtrain={(ai,qi,yi)}i=1N\mathcal{D}_{\mathrm{train}} = \{(a_i, q_i, y_i)\}_{i=1}^{N}, where aia_i is an audio recording, qiq_i is a textual instruction, and yiy_i is the corresponding response. The same recording may appear in multiple training examples with different instructions and responses, spanning tasks such as automatic speech recognition, audio captioning, and question answering.

Unlike conventional example-level MIAs, which infer whether a specific audio-instruction-response tuple was used for training, audio-file-level MIA aims to determine whether a given recording aa appeared in the fine-tuning dataset, regardless of its associated supervision. Formally, we define the membership indicator as

m(a)=I[∃(q,y) such that (a,q,y)∈Dtrain].(1)m(a) = \mathbb{I}\left[\exists(q,y)\ \text{such that}\ (a,q,y) \in\mathcal{D}_{\mathrm{train}}\right]. \tag*{(1)}

An attacker A\mathcal{A} is given a candidate recording aa and query access to the target fine-tuned model fθf_\theta, and seeks to predict m(a)m(a). The attacker computes an audio-level membership score sA(a;fθ)s_{\mathcal{A}}(a; f_\theta) and makes a binary decision according to

m^(a)=I[sA(a;fθ)≥τ],(2)\hat{m}(a) = \mathbb{I}\left[s_{\mathcal{A}}(a; f_\theta) \ge\tau\right], \tag*{(2)}

where τ\tau is a decision threshold. The objective is to distinguish member from non-member recordings using evidence obtained from the target model, without requiring knowledge of the specific instructions used during fine-tuning.

Threat Model

Adversarial objective. We consider an external auditor, such as a content creator or recording owner, who seeks to determine whether a specific audio recording was used to fine-tune a target audio LLM. The adversary aims to infer the recording-level membership indicator m(a)m(a), rather than the membership of a particular audio-instruction-response tuple. The attacker targets membership in supervised fine-tuning datasets used to adapt pretrained LLMs to downstream audio-understanding tasks.

Adversarial knowledge. We assume the adversary possesses the candidate audio recording aa and knows the target model’s audio-text interaction interface, such as associated Python packages to run inference. However, the adversary has no knowledge of the target’s fine-tuning dataset, including the textual instructions and reference responses associated with aa. In particular, the adversary cannot assume access to the original transcripts, captions, or question-answer pairs used during training. We also do not assume access to the pretrained checkpoint fθ0f_{\theta_0} or other models sharing the target’s training history.

Adversarial capabilities. The adversary can repeatedly query the target model with a candidate recording and arbitrary textual instructions, and observe the generated responses. We distinguish two levels of model access: (i) response-only access, where the adversary observes only generated text; and (ii) probability access, where the adversary can additionally obtain token-level output probabilities or logits. Response-only access supports behavioral attacks based on the model’s predictions, while probability access enables likelihood- and entropy-based membership statistics. AudioTrace operates under the latter setting, using the target model to generate proxy questions and score its responses without accessing model parameters or gradients.

Evaluation scope. We evaluate this threat model under controlled supervised fine-tuning, where the inclusion of each candidate recording is explicitly determined by the experimenter. Membership labels and training annotations are used only to establish ground truth and evaluate attacks, and are not available to the adversary. We additionally consider access to the original training questions as a privileged comparison setting to quantify the difficulty introduced by unknown supervision.

Direct Audio Membership Inference

We first investigate whether existing membership inference methods developed for text and video can be adapted to audio-file-level auditing. We consider three families of approaches: (1) direct probability-based attacks, which infer membership from the model’s output distributions; (2) Audio-SME, adapted from Vid-SME, which exploits the temporal ordering of audio segments using Sharma–Mittal entropy; and (3) an audio adaptation of VideoMIA, which leverages temporal reasoning over the chronological structure of audio recordings to infer membership. All methods operate directly on the candidate recording without requiring its original training instructions or reference responses.

Direct probability-based attacks. Following the multi-modal probability-extraction approach of Vid-SME [9], we obtain the target model’s output distributions at positions corresponding to the input audio representation. Specifically, given an audio recording aa which is to be converted to a sequence of vectors sent to the LLM backbone. Let I(a)\mathcal{I}(a) denote the set of audio-associated positions in the model’s input sequence. We extract

pt(v)=softmax⁡(zt)v,t∈I(a),(3)p_t(v) = \operatorname{softmax}(z_t)_v,\qquad t \in\mathcal{I}(a), \tag*{(3)}

where ztz_t denotes the target model’s output logits and vv indexes its output vocabulary. Let T=∣I(a)∣T = \lvert\mathcal{I}(a)\rvert denote the number of selected positions. Unlike text-based MIAs, these positions do not have corresponding ground-truth vocabulary labels. We therefore assign each position a pseudo-label y~t=arg⁡max⁡vpt(v)\tilde{y}_t = \arg\max_v p_t(v), enabling confidence-based adaptations of conventional token-level metrics.

We first consider perplexity (PPL) and Min-K% probability [7]. PPL is computed as

PPL⁡(a)=exp⁡(−1T∑t∈I(a)log⁡pt(y~t)).(4)\operatorname{PPL}(a) = \exp\left(-\frac{1}{T}\sum_{t\in\mathcal{I}(a)}\log p_t(\tilde{y}_t)\right). \tag*{(4)}

For Min-K%, we select the fraction ρ\rho of positions with the lowest pseudo-label log-probabilities and average their negative log-probabilities:

SMinK,ρ(a)=−1∣Tρ∣∑t∈Tρlog⁡pt(y~t),(5)S_{\mathrm{MinK},\rho}(a) = -\frac{1}{|T_\rho|}\sum_{t\in T_\rho}\log p_t(\tilde{y}_t), \tag*{(5)}

where Tρ⊆I(a)T_\rho\subseteq\mathcal{I}(a) contains the selected positions.

We additionally evaluate entropy-based attacks, including Shannon entropy and Rényi entropy of different orders. For a distribution ptp_t, Rényi entropy is defined as

Hα(pt)=11−αlog⁡∑vpt(v)α,(6)H_\alpha(p_t) = \frac{1}{1-\alpha}\log\sum_v p_t(v)^\alpha, \tag*{(6)}

with Shannon entropy recovered as α→1\alpha\to1. We evaluate α∈{0.5,1,2,∞}\alpha\in\{0.5, 1, 2, \infty\} and aggregate the resulting position-wise entropy values using their mean or the Max-K% statistic, which averages the largest fraction ρ\rho of entropy values. These methods yield recording-level membership scores directly from audio-associated output distributions.

Audio-SME. We adapt Vid-SME [9] to audio recordings and refer to the resulting baseline as Audio-SME. Vid-SME measures membership evidence through changes in model uncertainty under temporal reversal. To extend this principle to audio, we divide a recording aa into consecutive fixed-length chunks of duration cc and reverse their order, obtaining areva^{\mathrm{rev}}. The samples within each chunk remain unchanged, preserving local acoustic content while disrupting the recording’s global temporal structure.

We separately extract the model’s audio-associated output distributions for aa and areva^{\mathrm{rev}}, denoted by ptp_t and ptrevp_t^{\mathrm{rev}}, respectively. Following the original Vid-SME implementation, we compute Sharma–Mittal entropy with fixed parameters q=2q=2 and r=1r=1, corresponding to second-order Rényi entropy. The position-wise entropy difference is

Δt(a)=H2(pt)−H2(ptrev).(7)\Delta_t(a) = H_2(p_t) - H_2(p_t^{\mathrm{rev}}). \tag*{(7)}

Audio-SME obtains an audio-level membership score by aggregating {Δt(a)}\{\Delta_t(a)\} using the mean or the Min-K% and Max-K% aggregation functions provided by Vid-SME. We also evaluate the difference between sequence-level entropy values computed from the mean probability distribution across audio-associated positions. This adaptation retains Vid-SME’s entropy-based scoring mechanism while replacing video-frame reversal with audio-chunk reversal.

VideoMIA adaptation. We further adapt VideoMIA [10], a response-only membership inference method that evaluates a model’s ability to reason about temporal relationships. Given a recording aa, we repeatedly construct temporal-ordering trials by sampling a sequence of consecutive context clips, followed by four non-overlapping candidate clips from later positions in the same recording. The candidate clips are randomly shuffled and assigned labels {A,B,C,D}\{\mathrm{A}, \mathrm{B}, \mathrm{C}, \mathrm{D}\}. For each trial jj, we prompt the target model to perform two tasks: (i) next-segment identification, which asks the model to identify the earliest candidate following the context; and (ii) temporal ranking, which asks the model to arrange all four candidates in chronological order. Both tasks are evaluated against the original temporal positions of the sampled clips.

For identification, we assign a score

uj={2−(dj−1),dj∈{1,2,3},0,otherwise,(8)u_j = \begin{cases} 2^{-(d_j-1)}, & d_j \in\{1,2,3\},\\ 0, & \text{otherwise}, \end{cases} \tag*{(8)}

where djd_j is the true chronological rank of the candidate selected by the model. For temporal ranking, we measure the proportion of correctly ordered candidate pairs:

vj=1(42)∑1≤k<l≤4I[π^j(k)<π^j(l)],(9)v_j = \frac{1}{\binom{4}{2}}\sum_{1\leq k<l\leq4}\mathbb{I}\left[\hat{\pi}_j(k)<\hat{\pi}_j(l)\right], \tag*{(9)}

where π^j(k)\hat{\pi}_j(k) denotes the predicted position of the candidate whose true chronological rank is kk. Invalid or unparseable predictions receive a score of zero. Finally, we aggregate the two task scores over JJ independently sampled trials:

SVideoMIA-Audio(a)=12J∑j=1J(uj+vj).(10)S_{\mathrm{VideoMIA\text{-}Audio}}(a) = \frac{1}{2J}\sum_{j=1}^{J}(u_j+v_j). \tag*{(10)}

This response-only baseline assesses whether an audio LLM’s temporal understanding provides evidence of recording-level training membership, without accessing token probabilities or training annotations.

AudioTrace

Direct audio-file-level MIAs attempt to detect membership signals from the output distributions of audio-associated positions or temporal dependencies. However, Audio LLMs are commonly fine-tuned to generate textual responses conditioned on both audio recordings and task-specific instructions. Consequently, the membership signal may be more apparent in the model’s responses to relevant questions than in its direct audio representations. In practice, the original training instructions are generally unknown to the attacker, making conventional question-answer-based membership inference methods difficult to apply.

To address this challenge, we propose AudioTrace, a proxy-question-based approach to audio-file-level membership inference. As illustrated in Figure 1, AudioTrace leverages the target audio LLM itself to construct questions about a candidate recording, obtains the target model’s responses to those questions, and aggregates their membership scores into a single file-level prediction. The method consists of three stages: proxy-question generation, response generation and metric collection, and file-level aggregation. Importantly, no stage requires access to the original training annotations.

Figure 1. AudioTrace membership inference method. AudioTrace comprises of 3 stages from left to right. It first generates proxy questions from the target audio LLM. Then, for each proxy question, the target LLM generates an answer and the membership signals, such as Min-K% and predictive entropy, are recorded. Finally, these signals are aggregated at audio-file-level using top-k average, producing one single membership score for each audio file.

Proxy-Question Generation

Membership signals in Audio LLMs may depend strongly on the probing instructions, but the original training questions are typically unavailable to the attacker. We therefore leverage the target model to generate proxy questions, hypothesising that its learned audio-instruction associations favour questions aligned with its training supervision, thereby eliciting stronger membership signals. Specifically, given an audio recording aa and the target model fθf_\theta, AudioTrace first generates a set of NN proxy questions about the recording. We provide aa together with a fixed question-generation instruction qgenq_{\mathrm{gen}} to the target model:

Q(a)={q1,…,qN}=Generate⁡fθ(a,qgen).(11)Q(a) = \{q_1, \ldots, q_N\} = \operatorname{Generate}_{f_\theta}(a, q_{\mathrm{gen}}). \tag*{(11)}

The instruction (Appendix 10.3) encourages the model to formulate questions based on the audio content, potentially covering different aspects such as spoken content, speakers, and background sounds. We do not explicitly remove repeated or semantically similar questions, as their recurrence may reflect the model’s preference for particular probing contexts. Such preferences may be informative for membership inference, though they themselves may not be a signal.

Unlike predefined transcription or captioning prompts, these proxy questions are conditioned on the individual recording and can probe multiple aspects of its content. Meanwhile, transcription and caption queries are also included in the set of proxy queries.

Response Generation and Metric Collection

We then query the same target model with the original recording aa and each generated question qjq_j, obtaining a corresponding answer:

y^j=Generate⁡fθ(a,qj),j=1,…,N.(12)\hat{y}_j = \operatorname{Generate}_{f_\theta}(a, q_j), \qquad j = 1, \ldots, N. \tag*{(12)}

This produces a collection of audio-question-answer triplets {(a,qj,y^j)}j=1N\{(a, q_j, \hat{y}_j)\}_{j=1}^{N}, where both the questions and answers are generated by the target model, without requiring access to the training set annotations or reference responses.

For each generated answer y^j=(y^j,1,…,y^j,Tj)\hat{y}_j = (\hat{y}_{j,1}, \ldots, \hat{y}_{j,T_j}), we extract the target model’s token-level output distributions conditioned on the audio recording and proxy question. Let

pj,t(v)=Pθ(v∣a,qj,y^j,<t)(13)p_{j,t}(v) = P_\theta(v \mid a, q_j, \hat{y}_{j,<t}) \tag*{(13)}

denote the probability of token vv at position tt of the generated answer. Unlike direct audio-position scoring, the generated response provides the actual tokens against which their probabilities can be evaluated.

We consider three families of membership statistics: Min-K% probability, Max Rényi entropy and Shannon entropy. For Min-K% , we select the fraction ρ\rho of answer tokens with the lowest log-probabilities. Let Tj,ρ\mathcal{T}_{j,\rho} denote the corresponding token positions. The question-level score is

sj,MinK=−1∣Tj,ρ∣∑t∈Tj,ρlog⁡pj,t(y^j,t).(14)s_{j,\mathrm{MinK}} = -\frac{1}{\lvert\mathcal{T}_{j,\rho}\rvert} \sum_{t \in\mathcal{T}_{j,\rho}} \log p_{j,t}(\hat{y}_{j,t}). \tag*{(14)}

For entropy-based scoring, we compute the Shannon entropy and Rényi entropy with a given α\alpha of the predictive distribution pj,tp_{j,t} at each answer position tt, denoted as H(pj,t)H(p_{j,t}). The question-level entropy score is obtained by averaging over the generated answer as follows.

sj,Ent=1Tj∑t=1TjH(pj,t).(15)s_{j,\mathrm{Ent}} = \frac{1}{T_j} \sum_{t=1}^{T_j} H(p_{j,t}). \tag*{(15)}

For Rényi entropy, we use the Max-K% statistics by averaging across the fraction ρ\rho of the answer positions with the largest entropy values. All scores are oriented such that larger values indicate higher uncertainty and, potentially, stronger non-member evidence of the given query-response pair, which aligns with the top-K aggregation next.

File-level Aggregation

Because different proxy questions may yield varying degrees of membership evidence, we aggregate their scores at the recording level rather than treating each question as an independent membership target. For a chosen metric M∈{MinK,Ent}M \in\{\mathrm{MinK}, \mathrm{Ent}\}, let {sj,M}j=1N\{s_{j,M}\}_{j=1}^{N} denote its question-level scores. We adopt Top-kk average aggregation, selecting the kk highest-scoring questions:

SM(a)=1k∑j∈TopK⁡k({si,M}i=1N)sj,M.(16)S_M(a) = \frac{1}{k}\sum_{j \in\operatorname{TopK}_k(\{s_{i,M}\}_{i=1}^{N})} s_{j,M}. \tag*{(16)}

The resulting score SM(a)S_M(a) is used as the recording-level membership score sA(a;fθ)s_A(a; f_\theta) defined in § 3.1. AudioTrace evaluates Min-K% and entropy-based variants separately, allowing each metric to provide an independent audio-file-level membership estimate. By aggregating evidence across multiple automatically generated questions, AudioTrace reduces its dependence on any single instruction and enables membership inference without knowing how the recording was annotated during fine-tuning.

AudioTraceBench

Reliable evaluation of audio-file-level MIA requires accurate membership labels and careful control of potential dataset biases. However, the complete training corpora of existing Audio LLMs are rarely disclosed, making it difficult to establish whether publicly available recordings were previously encountered by a model. To address this challenge, we construct AudioTraceBench, a controlled benchmark in which membership is explicitly determined through SFT. AudioTraceBench contains 1,285 unique recordings, randomly divided into 640 members and 645 non-members. Figure 2 illustrates the construction pipeline.

Data generation and curation pipeline for AudioTraceBench

Figure 2. Data generation and curation pipeline for AudioTraceBench. We first collect YouTube videos from time-sensitive, information-rich categories and extract their audio tracks. We then filter the recordings based on content recency, information density, and duration. Finally, we divide the filtered recordings into a member set for supervised fine-tuning and a non-member set held out for membership auditing.

Data Creation Principle

Temporal separation. We collect recordings from YouTube videos published after January 1, 2026, and evaluate audio LLM checkpoints obtained before 2026. To further reduce potential overlap with the models’ previous training data, we prioritize time-sensitive content, particularly news, interviews, and discussions referencing recent events. The categories of audio files and their distribution are provided in Figure 3. This temporal separation reduces the likelihood of prior exposure to the selected recordings, although publication dates alone cannot rule out reused content.

Distribution of the number of audio files across six different audio categories

Figure 3. Pie chart showing the distribution of the number of audio files across six different audio categories in AudioTraceBench.

Information-density filtering. After retrieving videos and extracting their audio, we apply captioning and transcription to assess the information content of each recording. Specifically, we decompose each generated caption into atomic facts and count the resulting factual statements. We retain recordings containing more than 20 atomic facts and more than 100 transcribed words, thereby favouring information-rich audio with substantial spoken content. We additionally restrict recording duration to at most five minutes to ensure compatibility with the evaluated Audio LLMs.

Controlled membership assignment. After filtering, we randomly partition the recordings into disjoint member and non-member sets, containing 640 and 645 recordings, respectively. Only member recordings are used for SFT, while non-members remain held out for auditing. Since both sets are sampled from the same collection and filtering pipeline, this design reduces systematic differences in data provenance and content characteristics, while providing explicit membership ground truth.

Multi-task annotation. For each member recording, we prompt Gemini 3.5 Flash to generate one transcription, one audio caption, and 5-10 question-answer (QA) pairs covering spoken content, speaker attributes, and background sounds, following practices adopted in many Audio LLMs [1], [2], [3]. On average, each member recording is associated with 7.45 QA pairs. These annotations represent a typical SFT dataset for Audio LLMs and are set to be unavailable during membership inference.

Dataset Statistics

Dataset composition. AudioTraceBench comprises 1,285 unique recordings, including 640 members (49.8%) and 645 non-members (50.2%). Figure 3 presents the distribution across six categories, which encompass diverse conversational settings, speaking styles, and acoustic environments.

Duration and supervision statistics. Member recordings have a mean duration of 138.3 ±\pm 35.2 seconds, compared with 136.7 ±\pm 35.0 seconds for non-members, indicating closely matched duration distributions.

Blind membership evaluation. To assess whether membership labels can be inferred from pre-existing model knowledge rather than the controlled SFT intervention, we evaluate the original pretrained checkpoints before fine-tuning, i.e. fθ0f_{\theta_0}. Using available QA annotations, we compute perplexity, Min-30% probability, and entropy-based membership scores. For Qwen2.5-Omni, the three methods achieve AUROCs of 0.4977, 0.4986, and 0.4977, respectively. The corresponding results for SALMONN-2 are 0.4930, 0.4975, and 0.4841. All results are approximately chance-level, indicating that these scoring methods do not readily distinguish the designated members from non-members before SFT.

Experimental Setup

Models and Fine-Tuning

Our experiments cover three audio-capable LLMs, including Qwen2.5-Omni [5], SALMONN 2 [3], and Step-Audio-2-mini [4]. All three models are at the 7B–8B parameter scale. For Qwen2.5-Omni and Step-Audio-2-mini, we use officially released checkpoints available before 2026 to establish temporal separation from AudioTraceBench. For SALMONN 2, its fully public training data and pipeline allow us to verify that all backbone models were trained on pre-2026 data. We initialise each model from its pretrained checkpoint and perform SFT using the member recordings in AudioTraceBench.

The training data for SFT contains audio captioning, speech transcription and QA, representing a typical fine-tuning setup. We adopt low-rank adaptation (LoRA) [34] for all three models, with rank r=128r = 128 and scaling factor α=256\alpha= 256. LoRA is applied to linear layers in the self-attention block as well as the feed-forward networks in the LLM backbone, while keeping the audio encoder part frozen. We use a learning rate of 2×10−52 \times10^{-5}, an effective batch size of 8. Each model is fine-tuned for three epochs, unless specified in the results for ablation studies. Training of each model is performed on a single NVIDIA A100 GPU, with a three-epoch run taking approximately 20 hours.

Inference Details and Baseline Setup

For AudioTrace, we generate 30 proxy questions per recording using temperature sampling with T=1T = 1. Adding one transcription and one caption query, the total number of proxy questions is 32 for each audio file. Answers to the proxy questions are generated using greedy decoding. We evaluate top-kk aggregation with a default value of k=2k = 2 and use Entropy, Min-kk% and Max Rényi Entropy with token selection fractions of {30%, 60%}. The corresponding configurations are specified in the results tables. For permutation-based methods such as Audio-SME and VideoMIA-Audio, we use an audio chunk size of 2 seconds. As a stronger baseline for membership inference, we design a fixed question bank containing 30 questions that apply to all audio files (see Appendix for the questions). The same aggregation as AudioTrace across questions is applied to obtain a single membership signal.

Greedy decoding is always applied to answer both fixed questions and generated questions. All inference configurations are fixed across member and non-member recordings. The code for fine-tuning, inference and scoring is provided1.

Evaluation Protocol

Membership inference is performed at the audio-file level, using the same 640 member and 645 non-member recordings across all target models and attacks. Each recording contributes a single membership score, regardless of the number of associated SFT examples or proxy questions. For methods producing multiple scores per recording, we apply their respective aggregation strategies. Unless explicitly designated as an analytical comparison, all attacks operate without knowledge of the original SFT questions or responses. We orient membership scores consistently such that higher values indicate stronger evidence of membership.

We report the area under the receiver operating characteristic curve (AUROC) and the true positive rate at a 5% false positive rate (TPR@5% FPR). AUROC evaluates overall discrimination between members and non-members, while TPR@5% FPR measures detection performance under a low false-positive budget. Random guessing corresponds to an expected AUROC of 0.5 and TPR@5% FPR of 0.05.

Results

Experiments are organised around four key questions to evaluate audio-file-level membership inference and investigate the factors influencing membership signals. (i) Can membership be inferred without training annotations? §7.1-§7.2 show that existing methods perform near chance, while AudioTrace achieves AUROCs of up to 0.60 across three Audio LLMs. Blind baselines further support the reliability of our benchmark. (ii) How do unknown supervision and audio content affect membership inference? §7.3 reveals a substantial performance gap between known and unknown training annotations, while perturbation experiments demonstrate the dependence of membership signals on acoustic content. (iii) Why do proxy questions matter? §7.4 shows that the generation and alignment of proxy questions with training supervision significantly influence membership detection. (iv) How do training configurations affect membership signals? §7.5 examines the effects of training tasks, sample exposure, and adaptation capacity, showing stronger membership signals under increased exposure and adaptation capacity.

Main Results on AudioTraceBench

The main results on AudioTraceBench across three different models fine-tuned on the member set are shown in Table 1. AUROC and TPR at 5% FPR are reported for direct audio-based baselines and AudioTrace. We highlight the following observations from the experiments.

Attack MethodSettingQwen2.5-Omni AUROCQwen2.5-Omni TPR@5%FPRSALMONN-2 AUROCSALMONN-2 TPR@5%FPRStep-Audio-2-mini AUROCStep-Audio-2-mini TPR@5%FPR
PPL0.49570.04190.49700.04030.52070.0636
Entropy0.49530.04190.49580.03720.52000.0698
Min-K%Min-10%0.49870.02790.49160.03880.51410.0589
Min-K%Min-30%0.49460.04810.49330.03100.51730.0667
Min-K%Min-60%0.49590.04590.49590.03720.52040.0636
Max-K% Rényi EntropyMax-10% (α=0.5\alpha= 0.5)0.50000.03410.50010.03880.51190.0604
Max-K% Rényi EntropyMax-30% (α=0.5\alpha= 0.5)0.49920.05120.49720.03410.51520.0589
Max-K% Rényi EntropyMax-60% (α=0.5\alpha= 0.5)0.49980.04650.49560.03260.51720.0651
Max-K% Rényi EntropyMax-10% (α=2\alpha= 2)0.50130.02480.49440.03720.51370.0558
Max-K% Rényi EntropyMax-30% (α=2\alpha= 2)0.49520.05270.49250.03100.51680.0667
Max-K% Rényi EntropyMax-60% (α=2\alpha= 2)0.49650.04650.49470.03720.51950.0636
Audio-SMEAudio-SME Max 10%0.51060.02640.50050.04190.51940.0465
Audio-SMEAudio-SME Max 30%0.51150.05120.50680.04340.52140.0574
Audio-SMEAudio-SME Max 60%0.50720.03720.51330.04810.52290.0620
VideoMIA-AudioTrial = 50.50270.03720.50150.04030.51330.0562
VideoMIA-AudioTrial = 150.50490.03370.51840.04960.51790.0602
Fixed Question BankEntropy0.52240.02950.53900.06200.52500.0576
Fixed Question BankMin-60%0.51990.03570.54400.04810.52130.0525
Fixed Question BankMax-60% Rényi-20.51590.03410.54550.05740.51780.0469
AudioTrace (ours)Entropy0.55050.05890.58090.08220.58970.0930
AudioTrace (ours)Min-30%0.55250.05890.58960.09610.59490.1054
AudioTrace (ours)Min-60%0.55550.05120.58810.09770.59390.1008
AudioTrace (ours)Max-30% Rényi-20.55370.05580.59170.09460.59580.1039
AudioTrace (ours)Max-60% Rényi-20.55630.05580.58990.10230.59480.0977
AudioTrace (ours)Combined0.55150.05740.59020.09920.59520.1039

Table 1. Audio-file-level membership inference performance on AudioTraceBench, where each model is fine-tuned on the member set of AudioTraceBench. We report AUROC and TPR at 5% FPR for all metrics. Audio-SME and VideoMIA-Audio are adaptations of existing video-level MIA methods to audio introduced in §3.3.

Existing audio-level methods perform close to random guessing. Across all three models, conventional probability-based methods, including PPL, entropy, Min-K%, and Max-K% Rényi entropy, achieve AUROCs close to 0.5. For example, the best AUROCs among these methods are 0.5013, 0.5001, and 0.5207 on Qwen2.5-Omni, SALMONN 2, and Step-Audio-2-mini, respectively. Compared to the conventional methods, the adaptation of Vid-SME and VideoMIA to Audio-SME and VideoMIA-Audio, respectively, achieved similarly limited improvements. The best Audio-SME setting achieved slightly better AUROC scores across the models. The best VideoMIA-Audio method merely improved upon random guessing and is worse than Audio-SME. The best-performing baseline is the fixed question bank, indicating the necessity of generation-based membership signal in audio-file-level auditing. However, all the baselines have TPRs at 5% FPR generally below random.

AudioTrace consistently improves audio-file-level membership inference, achieving significantly higher performance than conventional methods and the adapted Audio-SME and VideoMIA-Audio. Specifically, AudioTrace achieves 4.4%, 4.7% and 7.1% absolute AUROC improvements compared to the Fixed Question Bank baseline on Qwen2.5-Omni, SALMONN 2, and Step-Audio-2-mini, respectively. At 5% FPR, AudioTrace with combined metric achieves TPRs of 0.0574, 0.0992, and 0.1039, compared with 0.0295, 0.0620, and 0.0576 for the fixed question bank baseline. The improvement is particularly evident on SALMONN 2 and Step-Audio-2-mini, where AudioTrace approximately doubles the detection rate permitted by a 5% false-positive budget relative to the Fixed Question Bank baseline. These findings demonstrate that probing a recording through self-generated proxy questions can reveal membership information that is difficult to recover from direct audio-based membership signals or a pre-defined question set. While the top 2 scores are used by default, AudioTrace is robust to the choice of top-K, as shown in the analysis in §10.1 in the Appendix.

AudioTrace performance is robust across different metrics. As shown in Table 1, while single best results were achieved by different metrics, their differences were generally small, and are all well above the baselines. To achieve a single robust metric for auditing, we combine Min-30%, Min-60%, Max 30% Rényi-2 and Max 60% Rényi-2 metrics to give a single combined score for membership inference. The combined score does not achieve the best performance, but achieves the best balance between AUROC and TPR@5%FPR across the table. Although audio-file-level membership inference remains challenging, AudioTrace represents a meaningful step forward, demonstrating that membership signals can be recovered even without access to the original training annotations.

Black-box membership inference with AudioTrace. AudioTrace can operate in a black-box setting by drawing samples instead of directly measuring the token-level probabilities. Specifically, to validate the effectiveness under the black-box setting, eight samples were drawn per proxy question from the LLM, and the consistencies across the samples were measured pairwise using ROUGE-L [35] and BERTScore [36]. The pairwise consistency scores are averaged across all responses and are used as the membership inference signal, where a higher score indicates membership since the response distribution tends to be more concentrated around the training set responses. The results are shown in Table 2 for Qwen2.5-Omni, where similar performance is achieved without access to token probabilities.

Membership scoreAUROCTPR@5%FPR
Combined0.55150.0574
ROUGE-L0.52530.0731
BERT score0.54410.0592

Table 2. AudioTrace operating under black-box setting.

Blind Baseline

To assess whether the random membership split introduces detectable differences between members and non-members, we evaluate three blind classifiers using features extracted independently of the target Audio LLMs. The first uses a two-dimensional feature vector consisting of the average pitch (f0f_0) and duration. The second uses 768-dimensional text representations obtained by mean-pooling BERT-base embeddings [37] of the transcriptions. The third uses 512-dimensional acoustic representations obtained by averaging the feature encoder outputs of wav2vec 2.0 [38]. For each feature type, we train a linear binary classifier of the form σ(w⊤x+b)\sigma(\mathbf{w}^{\top}\mathbf{x} + b), where σ\sigma denotes the sigmoid function and w\mathbf{w} and bb are trainable parameters estimated by maximum likelihood. We use a random 50-50 train-test split of AudioTraceBench and report the test set performance.

As shown in Table 3, all three features achieve accuracies close to the 50% chance level, suggesting that the tested audio and content features provide limited evidence for distinguishing members from non-members. This supports the effectiveness of our randomised membership assignment in reducing readily detectable dataset-level confounding.

MethodAccuracy
Pitch and Duration51.25%
Text Embeddings52.77%
Acoustic Embeddings53.32%

Table 3. Blind baselines based on audio and content features. Binary classifiers are trained on each feature.

Challenge of Unknown Supervision Data

To investigate the source of membership signals, AudioTrace was compared against a privileged setting in which the original training questions and answers are known and are used the same way as how AudioTrace uses generated proxy questions. To investigate the association with the specific audio file, we further apply two audio perturbations, including highly destructive additive noise at 0 dB SNR and reconstruction using a discrete audio codec (DAC). The former substantially corrupts the acoustic signal to check if the membership signal is tied to specific text query-response pairs, while the latter modifies the waveform but retains the same auditory content when listening to it. Table 4 presents the results on Qwen2.5-Omni and SALMONN-2.

SettingMetricQwen2.5-Omni AUROCQwen2.5-Omni TPR@5%FPRSALMONN-2 AUROCSALMONN-2 TPR@5%FPR
AudioTrace - Known Training Set QA for Member Audio
Original AudioMin-60%0.81040.41060.82160.4044
Original AudioMax-60% Rényi-20.80920.41210.82060.3810
Original AudioCombined0.81520.44010.82340.4059
Original Audio + 0dB SNR NoiseMin-60%0.57880.05290.58500.0871
Original Audio + 0dB SNR NoiseMax-60% Rényi-20.57860.05290.58530.0731
Original Audio + 0dB SNR NoiseCombined0.58180.04510.58510.0855
DAC-Reconstructed AudioMin-60%0.80960.40770.82330.4122
DAC-Reconstructed AudioMax-60% Rényi-20.80850.41180.81790.3834
DAC-Reconstructed AudioCombined0.81330.43790.82150.4073
AudioTrace - Generated Proxy Questions
Original AudioMin-60%0.55550.05120.58810.0977
Original AudioMax-60% Rényi-20.55630.05580.58990.1023
Original AudioCombined0.55150.05740.59020.0992
Original Audio + 0dB SNR NoiseMin-60%0.50370.05740.52210.0543
Original Audio + 0dB SNR NoiseMax-60% Rényi-20.51130.05740.52280.0667
Original Audio + 0dB SNR NoiseCombined0.50610.05430.52160.0667
DAC-Reconstructed AudioMin-60%0.55450.05810.58810.0977
DAC-Reconstructed AudioMax-60% Rényi-20.55530.05810.58980.1023
DAC-Reconstructed AudioCombined0.55320.05960.59030.0992

Table 4. Comparison of known training set QA annotations against unknown annotation condition for Qwen2.5-Omni and SALMONN-2 models under corruptive noise and audio tokeniser reconstruction perturbations. AudioTrace is used across the table with the standard setting as before.

Knowledge of training QA substantially strengthens membership inference. On the original recordings, the privileged setting achieves AUROCs of up to around 0.82 for both Qwen2.5-Omni and SALMONN-2, respectively, with TPR@5% FPR reaching over 0.40 for both models. In contrast, without accessing the original training annotations, AudioTrace only achieves AUROCs of at most 0.56 and

0.59, with corresponding maximum TPRs of 0.06 and 0.10. This substantial gap indicates that the original question-answer pairs provide strong membership evidence that is difficult to recover when only the audio recording is available. Consequently, performance under known training QA supervision should not be interpreted as representative of practical audio-file-level membership inference.

Strong acoustic corruption largely eliminates AudioTrace’s membership signal. Adding noise at 0 dB SNR markedly reduces attack performance in both settings. For the privileged setting, the best AUROC decreases drastically, from 0.82 to 0.58 on Qwen2.5-Omni and from 0.82 to 0.59 on SALMONN-2. Nevertheless, a residual membership signal remains, with AUROCs consistently above 0.57 across the evaluated metrics. This residual signal is consistent with the model retaining some level of information about the specific textual question-answer pairs encountered during training, even when the audio is heavily corrupted.

The effect is more evident for AudioTrace. Its best AUROC decreases from 0.56 to 0.51 on Qwen2.5-Omni and from 0.59 to 0.52 on SALMONN-2, approaching random guessing performance. This finding suggests that AudioTrace’s membership evidence depends substantially on the acoustic information in the candidate recording. Unlike the privileged setting, AudioTrace cannot directly exploit knowledge of the original verbatim training questions and answers. The disappearance of its signal under severe audio corruption therefore provides evidence that AudioTrace probes recording-level membership, rather than merely detecting memorised textual question-answer pairs.

Audio reconstruction preserves membership evidence despite waveform changes. In contrast to additive noise, DAC reconstruction has little effect on both settings. For AudioTrace, the best AUROCs on reconstructed audio are 0.56 and 0.59 on Qwen2.5-Omni and SALMONN-2, compared with 0.57 and 0.59 on the original recordings. Similarly, the privileged setting maintains comparable AUROCs.

These results indicate that AudioTrace does not rely solely on an exact waveform match. Its membership signal survives DAC reconstruction, suggesting that relevant audio information is preserved. The perturbation experiments suggest that AudioTrace captures audio-dependent membership evidence that is robust to moderate reconstruction changes but vulnerable to severe acoustic corruption.

Ablation Study on Proxy Questions

Proxy questions can be generated either by the target model itself or by an external model. AudioTrace adopts the former approach, motivated by the hypothesis that a target model’s generated questions better reflect the knowledge acquired during training. To examine this design choice, we use Qwen3-Omni-30B-A3B-Instruct [39] as an external question generator, producing 30 proxy questions per recording, and apply the same AudioTrace scoring and aggregation procedure. We further quantify the affinity between proxy questions and the original training questions using rescaled BERTScore [36]. Specifically, for each proxy question, we compute its BERTScore against every training question associated with the same recording and take the maximum similarity as its affinity score. We then average these scores across proxy questions and report the resulting affinity alongside membership inference performance, measured using the Combined score, in Table 5.

ModelSource of QuerySimilarityAUROC
Qwen2.5-Omniself-gen.0.41090.5515
Qwen2.5-OmniQwen3-Omni gen.0.38700.5383
SALMONN-2self-gen.0.41660.5902
SALMONN-2Qwen3-Omni gen.0.38700.5562

Table 5. SIMILARITY TO THE TRAINING SET QUESTIONS AND INFLUENCE OF THE SOURCE OF QUERIES ON AUDITING PERFORMANCE. THE COMBINED SCORE IS USED TO MEASURE AUROC. WE ADOPT THE QWEN3-OMNI-30B-A3B-INSTRUCT MODEL TO GENERATE PROXY QUESTIONS AS A COMPARISON REFERENCE.

As shown in Table 5, self-generated proxy questions consistently achieve higher similarity to the original training questions and better membership inference performance than questions generated by an external model. For Qwen2.5-Omni, self-generated questions achieve a similarity score of 0.41 and an AUROC of 0.55, compared with 0.39 and 0.54 using Qwen3-Omni-generated questions. A similar trend is observed for SALMONN-2, where self-generated questions improve similarity from 0.39 to 0.42 and AUROC from 0.57 to 0.59. These results support our hypothesis that questions generated by the target model are more closely aligned with its training supervision, making them more effective for eliciting membership signals.

Influence of proximity between training questions and proxy questions. The consistent association between higher question similarity and improved AUROC in Table 5 suggests that proximity to the training-question distribution is an important factor. This finding further motivates the design choice of using the target model itself for proxy-question generation in AudioTrace. To further establish this correlation, we apply AudioTrace only to the MM most similar proxy queries and measure the performance variation against different MM, with results shown in Table 6.

ModelMMMax-60% R’enyi-2Combined
Qwen2.5-Omni30.59950.6209
Qwen2.5-Omni50.57910.5973
Qwen2.5-Omni100.55270.5669
Qwen2.5-Omni200.55680.5527
Qwen2.5-Omni300.55630.5515
SALMONN-230.67780.6822
SALMONN-250.62530.6285
SALMONN-2100.61790.6212
SALMONN-2200.59590.5983
SALMONN-2300.58990.5902

Table 6. AUROC OF DIFFERENT MODELS USING THE MM MOST SIMILAR PROXY QUERIES TO THE TRAINING SET QUESTIONS.

The results in Table 6 reveal that restricting AudioTrace to proxy questions more similar to the original training questions substantially improves membership inference. With only the three most similar questions (M=3M = 3), AudioTrace achieves its highest AUROCs of 0.62 and 0.68 using the Combined score on Qwen2.5-Omni and SALMONN-2, respectively. As MM increases to 30, the AUROCs decrease to 0.55 and 0.59, matching the results in the main table. A similar trend is observed with Max-60% R’enyi-2 scoring, particularly on SALMONN-2. These findings further support the relationship between proxy-question affinity and membership inference effectiveness, where questions semantically closer to the original training supervision tend to reveal stronger membership signals, while incorporating less similar questions dilutes the evidence.

Results in Table 5 and 6 highlight the importance of instruction alignment in audio-file-level membership inference. Questions more closely aligned with the original training supervision tend to elicit stronger membership signals, while target-model-generated questions exhibit higher training-question similarity than externally generated ones.

This supports our motivation for leveraging the target model itself to approximate informative probing instructions when the original training annotations are unavailable.

Impact of the number of proxy questions. We investigate the effect of the number of generated proxy questions on AudioTrace’s membership inference performance, as shown in Figure 4. For both Qwen2.5-Omni and SALMONN-2, AUROC generally improves as the number of proxy questions increases, but the gains gradually diminish when more samples are used. Specifically, increasing the question count from 3 to 50 improves AUROC from 0.54 to 0.55 on Qwen2.5-Omni and from 0.57 to 0.59 on SALMONN-2. Performance largely stabilises beyond 10 questions for Qwen2.5-Omni and 30 questions for SALMONN-2, suggesting diminishing returns from generating additional questions. tions. Moreover, the standard deviation generally decreases as more questions are generated, indicating greater stability as the number of sampled questions increases because more samples cover the possible question space more thoroughly.

Figure 4. Variation of AUROC against the number of proxy questions generated for Qwen2.5-Omni and SALMONN-2 using AudioTrace Combined score. The shaded area represents the single standard deviation range.

Impact of proxy question composition. Table 7 evaluates the contribution of different proxy-question types. Combining QA, captioning, and ASR achieves the highest AUROC on Qwen2.5-Omni and Step-Audio-2-mini, while remaining competitive on SALMONN 2 where captioning + ASR alone performs the best. The results suggest that different question types expose complementary membership evidence. By incorporating all three tasks, AudioTrace avoids relying on a single type of audio understanding and achieves consistently competitive performance without model-specific question selection, supporting our multi-task proxy-question generation design.

CompositionQwen2.5-OmniSALMONN-2Step-Audio-2-mini
Caption + ASR0.52760.60370.5549
QA0.54500.57230.5845
All0.55150.59020.5952

Table 7. IMPACT OF COMPOSITION OF THE PROXY QUESTIONS IN AUDIOTRACE MEASURED BY AUROC ACROSS ALL THREE MODELS. THE COMBINED SCORE IS USED AS THE MEMBERSHIP SCORE. FOR CAPTION + ASR, THE AVERAGE OF BOTH TASKS IS USED.

Ablation Study on Training Configurations

Influence of training data composition. To investigate how different composition in training data influences the performance of membership inference with AudioTrace method, Qwen2.5-Omni and SALMONN-2 were trained again just with audio captioning and speech recognition tasks, or just with QA. Other training configurations remained the same, and both models were trained for 3 epochs regardless of the training data composition. The performance of AudioTrace on those models are shown in Figure 5.

Figure 5. Variation of AUROC and TPR@5%FPR with different training data composition, including QA, CapASR (caption and transcription) or the combination of the two, which is the default setting.

As shown in the figure, AudioTrace achieves higher membership inference performance when QA supervision is included alongside captioning and ASR during fine-tuning. When restricted to captioning and ASR, AudioTrace still outperforms random guessing and direct audio-based baselines, but with reduced effectiveness. One possible explanation is that membership evidence becomes concentrated in captioning- and ASR-related queries, while less informative QA queries dilute the aggregated signal. In this setting, focusing on captioning and ASR queries may yield stronger membership evidence. However, the exact composition of the training data is generally unavailable to the auditor, making such task-specific selection impractical. This further motivates AudioTrace’s audio-file-level threat model, where membership must be inferred without knowledge of the training queries or supervision types.

We then investigate the influence of training exposure and adaptation capacity on audio-file-level membership inference, as shown in Figure 6. We vary the number of training epochs, reflecting repeated exposure to member recordings, and the LoRA rank, which controls the number of trainable parameters. To assess downstream utility, word error rate (WER) is measured on all 645 non-member recordings. Audio QA accuracy is evaluated on 2,000 question-answer pairs generated from these recordings by Gemini 3.5 Flash. Manual verification of 100 sampled questions yields 96% annotation accuracy, and Qwen3.5-9B is used as an LLM judge to assess answer correctness.

Ablation studies across training epochs and LoRA ranks for SALMONN-2 and Qwen2.5-Omni

Figure 6. Ablation studies on training configurations, including the number of repetitions of the samples (represented by the number of epochs), and the number of trainable parameters controlled by the LoRA rank, using (a) SALMONN-2 and (b) Qwen2.5-Omni.

Impact of sample repetitions in training. The AUROC of AudioTrace increases with increasing training epochs on both models, whereas Audio-SME remains close to random guessing. On SALMONN-2, the AUROC of AudioTrace increases from 0.56 to 0.63 over 5 epochs. Meanwhile, WER improves steadily, while QA accuracy declines with additional epochs. Similar improvements were found on Qwen2.5-Omni. These findings suggest that repeated exposure to member recordings strengthens the membership signal accessible through AudioTrace, even when downstream task performance exhibits different trends across tasks.

Impact of LoRA rank. Increasing the LoRA rank also strengthens the membership signal captured by AudioTrace. On SALMONN-2, AUROC increases from 0.56 to 0.67 as the rank increases from 32 to 512, while Audio-SME still remains substantially closer to random guessing. Similar improvements are observed on Qwen2.5-Omni. Notably, both WER and QA accuracy improve with increasing LoRA rank, indicating that stronger membership leakage can accompany improved downstream utility rather than necessarily resulting from degraded generalization.

Discussion

In this section, the broader implications of the findings are discussed for audio-file-level membership inference and the limitations of the proposed AudioTrace approach.

Membership inference beyond fixed scoring metrics. Existing membership inference research has primarily investigated how to extract membership evidence through prediction-based statistics [7], [13], [20]. Our findings highlight another important dimension: how the model is queried. The substantial gap between known- and unknown-supervision settings, alongside the relationship between proxy-question similarity and attack performance, suggests that membership evidence is sensitive to probing context. This observation connects to broader work on data-use auditing under limited knowledge of model training [31]. It also complements document-level membership inference research [29], highlighting the importance of aligning membership auditing with the unit of interest to data owners. Our results motivate further research on informative query selection when training-time details are unknown.

From membership inference to reliable auditing. Prior security research has emphasized that successful membership inference does not necessarily provide reliable evidence of training data use. Carlini et al. [13] highlight the importance of evaluating attacks at low false-positive rates, while recent work examines the reliability and coverage of membership inference for privacy auditing [32]. Distributional confounding is another challenge, particularly for audio data [20, 30]. Our controlled benchmark and blind baselines aim to mitigate these factors. Nevertheless, AudioTrace’s modest AUROC and low-FPR detection performance indicate that its scores should be interpreted as statistical evidence rather than definitive proof of recording membership. Developing more reliable and well-calibrated

membership evidence is therefore essential for practical privacy and copyright auditing.

Limitations and Future Work. Our study has several limitations. First, AudioTrace introduces a trade-off between inference efficiency and detection performance. Generating and evaluating multiple proxy questions requires approximately five times more inference computation than direct audio-level methods, but achieves higher AUROC. This additional cost may limit scalability in large-scale auditing scenarios. Second, our evaluation focuses on controlled, supervised fine-tuning, while real-world training may involve larger, more diverse datasets, different supervision strategies, and varying sample exposure. Evaluating AudioTrace in such settings remains important. Finally, identifying effective proxy questions without knowledge of training details remains challenging. Future work could explore adaptive query selection to improve detection, particularly at low false-positive rates.

Conclusion

We investigate audio-file-level membership inference against Audio LLMs without knowledge of the original training queries and annotations. We introduce AudioTraceBench, a controlled benchmark revealing the limitations of existing MIA methods, and propose AudioTrace to generate audio-conditioned proxy questions and aggregate response-based membership signals at the file level. AudioTrace consistently outperforms adapted baselines across three Audio LLMs. Further analyses reveal how unknown training annotations, query selection, and fine-tuning configurations influence membership leakage. These findings lay the foundation for audio-file-level auditing and motivate more reliable membership inference methods.

References

Fixed Question Bank

"What_is_the_main_topic_discussed_in_the_audio?",
"What_is_the_main_purpose_of_the_recording?",
"What_is_the_most_important_information_conveyed_in_the_audio?",
"What_are_the_key_points_discussed_in_the_recording?",
"What_is_the_main_conclusion_or_takeaway?",
"What_specific_issue_or_problem_is_being_discussed?",
"What_event_or_situation_is_described_in_the_audio?",
"What_are_the_main_facts_presented_in_the_recording?",
"What_specific_details_are_provided_about_the_main_topic?",

Ablation on Aggregation Top-K

We investigate the effect of the top-KK aggregation parameter in Figure 7. Across both Qwen2.5-Omni and SALMONN 2, AudioTrace consistently outperforms the Audio-SME baseline over all evaluated values of KK, indicating limited sensitivity to the aggregation parameter. Nevertheless, performance generally declines as more proxy questions are included, with absolute AUROC decreasing by approximately 1% and 2% on Qwen2.5-Omni and SALMONN 2, respectively.

Figure 7. Variations of AUROC against different top-KK aggregations on AudioTraceBench on Qwen2.5-Omni and SALMONN-2 using Max-60% Rényi-2 Entropy and the combined score.

One plausible explanation is that some proxy questions may be less aligned with the training question distribution and therefore yield weaker membership signals. Including less informative questions in the aggregation may dilute the stronger evidence provided by the most revealing questions, suggesting that aggregating a small number of high-scoring questions is a preferable approach.

Appendix

Ablation on Aggregation Top-K

We investigate the effect of the top-KK aggregation parameter in Figure 7. Across both Qwen2.5-Omni and

Ethics Considerations

Our study aims to support privacy and copyright auditing of Audio LLMs through membership inference. AudioTraceBench is constructed exclusively from publicly accessible YouTube videos. We screened the collected recordings and excluded those containing sensitive personal information to minimize potential privacy risks. To further mitigate privacy and copyright concerns, we will release only the original YouTube video links. Access to the benchmark through Hugging Face will be request-based.

We also acknowledge that AudioTrace could potentially be misused to infer the membership of recordings without their owners’ authorization. However, our experiments are conducted using locally deployed models under controlled fine-tuning conditions, without targeting third-party production services. Furthermore, the attack requires access to token-level output probabilities and currently offers limited reliability for individual-recording decisions. We believe the benefits of understanding and mitigating training-data privacy risks justify studying this attack under these controlled conditions. Our study does not identify a specific vulnerability in a deployed service, and therefore no vendor-specific vulnerability disclosure has been undertaken.

"What_examples_are_mentioned_to_illustrate_the_discussion?",
"Who_are_the_people_mentioned_in_the_audio?",
"What_organizations_or_companies_are_mentioned?",
"What_locations_or_places_are_mentioned?",
"What_dates_or_time_references_are_mentioned?",
"What_numbers,_quantities,_or_statistics_are_mentioned?",
"What_happened_at_the_beginning_of_the_recording?",
"What_events_or_developments_are_described_later?",
"What_outcome_or_result_is_discussed?",
"What_changes_or_developments_are_mentioned?",
"What_future_plans_or_predictions_are_discussed?",
"What_is_the_main_argument_or_claim_made_by_the_speaker?",
"What_reasons_are_given_to_support_the_main_argument?",
"What_evidence_is_provided_to_support_the_claims?",
"What_concerns_or_challenges_are_raised?",
"What_solutions_or_recommendations_are_proposed?",
"What_opinions_or_viewpoints_are_expressed?"
"Are_there_any_disagreements_or_contrasting_perspectives?",
"What_questions_are_asked_during_the_recording?",
"What_information_does_the_speaker_emphasize?",
"What_uncertainties_or_limitations_are_mentioned?"

Question-Generation Prompt

The following prompt is used for the target model to generate proxy questions in AudioTrace.

Question-Generation Prompt

Generate a question about the audio. The question can be about the content, the speaker or background audio information. Output the question directly. Question:

References

  1. [1]C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Towards general-purpose hearing abilities for large language models,” in Proc. ICLR, 2024.
  2. [2]Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-audio technical report,” arXiv: 2407.10759, 2024.arxiv.org/abs/2407.10759
  3. [3]X. Yang, X. Xu, W. Yu, S. Wang, C. Tang, T. Chiba, S. Hou, Z. Zhang, W. Wu, B. Li, G. Sun, C. Zhang, and P. Woodland, “Salmon-2: Advancing general-purpose hearing abilities with self-supervised representations,” arXiv:2607.17079, 2026.arxiv.org/abs/2607.17079
  4. [4]B. Wu et al., “Step-audio 2 technical report,” arXiv: 2507.16632, 2025.arxiv.org/abs/2507.16632
  5. [5]J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin, “Qwen2.5-omni technical report,” arXiv: 2503.20215, 2025.arxiv.org/abs/2503.20215
  6. [6]R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE Symposium on Security and Privacy (SP), 2017, pp. 3–18.arxiv.org/abs/1610.05820
  7. [7]W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, “Detecting pretraining data from large language models,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 51 826–51 843.arxiv.org/abs/2310.16789
  8. [8]J. Zhang, J. Sun, E. Yeats, Y. Ouyang, M. Kuo, J. Zhang, H. Yang, and H. Li, “Min-k%++: Improved baseline for pre-training data detection from large language models,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 64 845–64 862.arxiv.org/abs/2404.02936
  9. [9]Q. Li, R. Yu, and X. Wang, “Vid-sme: Membership inference attacks against large video understanding models,” Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2506.03179
  10. [10]J. Wang, Z. Gao, W. Wang, H. Wang, Z. Qin, L. Zhu, H. Fu, S. Wang, and T. Qi, “Black-box membership inference attacks for video training data in multimodal large language models,” in Proc. ACL, 2026.
  11. [11]R. Wen, Z. Li, M. Backes, and Y. Zhang, “Membership inference attacks against in-context learning,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 3481–3495.arxiv.org/abs/2409.01380
  12. [12]S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in 2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 2018, pp. 268–282.arxiv.org/abs/1709.01604
  13. [13]N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” in 2022 IEEE symposium on security and privacy (SP). IEEE, 2022, pp. 1897–1914.arxiv.org/abs/2112.03570
  14. [14]C. Song and V. Shmatikov, “Auditing data provenance in text-generation models,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 196–206.arxiv.org/abs/1811.00513
  15. [15]F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, “Quantifying privacy risks of masked language models using membership inference attacks,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 8332–8347.
  16. [16]F. Galli, M. Melis, and T. Cucinotta, “Noisy neighbors: Efficient membership inference attacks against llms,” in Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, 2024, pp. 1–6.arxiv.org/abs/2406.16565
  17. [17]C. Wang, Y. Wang, B. Hooi, Y. Cai, N. Peng, and K.-W. Chang, “Con-recall: Detecting pre-training data in llms via contrastive decoding,” in Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 1013–1026.arxiv.org/abs/2409.03363
  18. [18]P. Hu, Z. Wang, R. Sun, H. Wang, and M. Xue, “M4I: Multi-modal models membership inference,” Advances in Neural Information Processing Systems, vol. 35, pp. 1867–1882, 2022.DOI
  19. [19]Z. Li, Y. Wu, Y. Chen, F. Tonin, E. Abad Rocamora, and V. Cevher, “Membership inference attacks against large vision-language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 98 645–98 674, 2024.arxiv.org/abs/2411.02902
  20. [20]J.-K. Dong, Y.-X. Lin, and H.-Y. Lee, “Membership inference attacks against large audio language models,” arXiv preprint arXiv:2603.28378, 2026.arxiv.org/abs/2603.28378
  21. [21]G. Chen, Y. Zhang, and F. Song, “Slmia-sr: Speaker-level membership inference attacks against speaker recognition systems,” arXiv preprint arXiv:2309.07983, 2023.DOI
  22. [22]F. Teixeira, K. Pizzi, R. Olivier, A. Abad, B. Raj, and I. Trancoso, “Improving membership inference in asr model auditing with perturbed loss features,” arXiv preprint arXiv:2405.01207, 2024.arxiv.org/abs/2405.01207
  23. [23]L. Wang, Q. Thakkar, and R. Mathews, “Unintended memorization in large asr models, and how to mitigate it,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 4655–4659.arxiv.org/abs/2310.11739
  24. [24]M. Jagielski, O. Thakkar, and L. Wang, “Noise masking attacks and defenses for pretrained speech models,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 4810–4814.arxiv.org/abs/2404.02052
  25. [25]R. Cheng, Y. Ding, S. Cao, and Z. Wang, “Gibberish is all you need for membership inference detection in contrastive language-audio pretraining,” in Proceedings of the 2025 International Conference on Multimedia Retrieval, 2025, pp. 108–116.
  26. [26]Y. C. Liu, J. Yu, K. Cao, S. I. A. Meerza, T. Moyva, and J. Liu, “Auditing training data in generative music models via black-box membership inference,” in 2026 IEEE Symposium on Security and Privacy Workshops (SPW). IEEE, 2026, pp. 230–237.
  27. [27]J. Proboszcz, P. Kochanski, K. Korszun, D. Cricostomi, G. Strano, E. Rodolà, K. Deja, J. Dubinski, and K. Stankiewicz, “Membership and dataset inference attacks on large audio generative models,” in NeurIPS 2025 Workshop on Generative and Protective AI for Content Creation, 2025.arxiv.org/abs/2512.09654
  28. [28]N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in 30th USENIX security symposium (USENIX Security 21), 2021, pp. 2633–2650.arxiv.org/abs/2012.07805
  29. [29]M. Meeus, S. Jain, M. Rei, and Y.-A. de Montjoye, “Did the neurons read your book? document-level membership inference for large language models,” in 33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 2369–2385.
  30. [30]P. Maini, H. Jia, N. Papernot, and A. Dziedzic, “Llm dataset inference: Did you train on my dataset?” Advances in Neural Information Processing Systems, vol. 37, pp. 124 069–124 092, 2024.arxiv.org/abs/2406.06443
  31. [31]Z. Huang, N. Z. Gong, and M. K. Reiter, “A general framework for data-use auditing of ml models,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 1300–1314.DOI
  32. [32]Z. Wang, C. Zhang, Y. Chen, N. Baracaldo, S. R. Kadhe, and L. Yu, “Membership inference attacks as privacy tools: Reliability, disparity and ensemble,” in Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, 2025, pp. 1724–1738.DOI
  33. [33]G. Sun, W. Yu, C. Tang, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, Y. Wang, and C. Zhang, “video-salmonn: Speech-enhanced audio-visual large language models,” in Proc. ICML, 2024.arxiv.org/abs/2406.15704
  34. [34]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in Proc. ICLR, 2021.arxiv.org/abs/2106.09685
  35. [35]C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, Barcelona, Spain, Jul. 2004.
  36. [36]T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” in Proc. ICLR, 2020.arxiv.org/abs/1904.09675
  37. [37]J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019.arxiv.org/abs/1810.04805
  38. [38]A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. NeurIPS, 2020.arxiv.org/abs/2006.11477
  39. [39]J. Xu, Z. Guo et al., “Qwen3-omni technical report,” arXiv:2509.17765, 2025.arxiv.org/abs/2509.17765

Paper details

vlnc:W_uy4xz2tk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents