WorksComputer Science
TAC2: Timestamped Audio Captioning as composable functions
Abstract
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
Introduction
Human auditory perception separates overlapping acoustic signals before higher-level reasoning [6], grounding abstract semantics in physical perception [15, 5]. In contrast, modern Large Audio Language Models (LALMs) emphasize conversational generation [35, 11] (Xu et al. 2025), producing free-form text with limited structural grounding. Yet applications such as semantic search, precise audio editing, and automated archiving require queryable, temporally grounded representations of events, speakers, and paralinguistic cues.
Prior work on Timestamped Audio Captioning (TAC) [24] took a first step toward this goal by grounding audio descriptions in physical time, representing acoustic events with explicit temporal boundaries. However, existing timestamped captioning formulates description generation largely as a fixed, one-shot process that does not naturally accommodate richer perceptual structure or external priors. This motivates a natural question, Can richer, structured audio descriptions be built by composing simpler captioning functions, where the output of one conditions the next?
Figure 1. Overview of the two-stage TAC2 / TAC2-S framework. (1) Perceptual Foundation (TAC2): Predictions from weak teachers (SED, Diarization, Captioner) are merged via time-registered IoU matching to generate intermediate structured captions, which train a base adapter via the proposed Instruction-Input-Output (IIO) format. (2) Speech Elicitation (TAC2-S): Individual LoRA specialists (Stress, Emotion, Voice Profile, Acoustics) are trained on small gold corpora to elicit paralinguistic attributes from the pre-trained backbone. This cascade re-annotates the corpus, which is subsequently distilled into our unified student model, TAC2-S.
We propose TAC2, a foundation model that constructs granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid manually annotating such structured timelines at scale, we introduce a multi-view fusion pipeline that temporally aligns complementary predictions from off-the-shelf sound event detectors, diarizers, and captioners over a large real-world corpus, producing high-fidelity structural training targets. To construct this richer schema while retaining modular control, we introduce an Instruction-Input-Output (IIO) completion format that decomposes the structured caption into conditionable completion tasks. Each IIO task acts as a function of the audio, an instruction, and an optional input. A single call decodes the full description, while composing calls lets preceding predictions or external priors, such as high-quality transcripts, condition subsequent completions. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, formulating this expansion as latent capability elicitation. Motivated by probing evidence that these attributes are decodable from intermediate backbone representations, we train attention-only specialists on fewer than 10,000 labeled samples to expose them as structured outputs. These specialists annotate the corpus with expressive attributes, and the resulting cascade is distilled into the unified TAC2-S model.
The resulting structured representation enables two downstream paradigms. First, cascading TAC2-S with frozen text-only LLMs yields strong spoken understanding on MMSU ([38]) and MSU-Bench ([40]) across diverse reasoning backends, bridging perception and reasoning through a structured intermediate representation. Second, using TAC2-S to annotate in-the-wild audio produces a large-scale, paralinguistically enriched structured training corpus for text-to-audio generation. We show that this corpus enables explicit zero-shot control over event timing, speaker characteristics, and word-level emphasis.
In this work, our contributions are: (1) a scalable supervision framework that constructs granular, temporally grounded training targets by fusing complementary weak teachers, and extends them with paralinguistic attributes elicited by lightweight specialists trained on small labeled datasets (2) Instruction-Input-Output (IIO), a modular structured-completion interface that casts captioning tasks as composable functions, supporting both autonomous decoding and externally conditioned inference, enabling flexible construction and bootstrapping of comprehensive audio descriptions; and (3) TAC2 and its paralinguistic extension TAC2-S, which instantiate this framework and demonstrate strong fine-grained audio understanding together with temporally and paralinguistically controllable audio generation.
Related Work
Audio-Language Models and Grounded Perception. Recent Large Audio-Language Models (LALMs) [35, 11, 22] (Zhang et al. 2025) demonstrate strong instruction-following, yet benchmarks reveal a persistent gap between high-level reasoning and fine-grained acoustic perception [38, 31]. Existing models remain susceptible to temporal hallucinations and reliance on linguistic priors [23, 10], a limitation compounded by captioning datasets that summarize audio into short global descriptions [21, 13]. While prior timestamped captioners such as TAC [24] introduced explicit event boundaries, they primarily target event-level temporal grounding and are not designed to represent the richer structure of natural conversational audio, including speaker-attributed turns and overlapping events. In contrast, TAC2 uses multi-teacher temporal fusion to construct explicit, speaker-attributed acoustic timelines from in-the-wild audio, and represents them through IIO as a multi-granularity perceptual interface that supports both autonomous decoding and externally conditioned inference.
Spoken Paralinguistics and Latent Elicitation. Expressive speech attributes such as stress, intonation, emotion, and speaker characteristics are central to spoken understanding [28, 20]. Existing SpeechLLMs often model these attributes as isolated categorical tasks [37, 14] or rely on external signal-processing pipelines for acoustic features [39] (Yosha et al. 2025). Meanwhile, work on latent capability elicitation suggests that pretrained models can encode capabilities that become accessible through limited parameter adaptation [1, 12]. Building on this observation, TAC2-S uses lightweight specialists to expose paralinguistic information that is decodable from the pretrained audio backbone, and distills their predictions into a unified, multi-granularity acoustic schema.
Methodology
Our framework constructs a queryable, temporally grounded acoustic representation in two stages. Stage 1 builds TAC2, a granular, speaker-attributed physical timeline bootstrapped from complementary weak teachers over in-the-wild audio. Stage 2 extends this timeline into TAC2-S by eliciting latent paralinguistic capabilities from the pre-trained backbone using lightweight specialists, then distilling their annotations into a unified model.
Stage 1: Perceptual Foundation (TAC2)
Dense temporal annotation at the scale required to train a foundation model is prohibitively expensive. We therefore construct TAC2’s training targets through multi-view weak supervision and learn them using an Instruction-Input-Output (IIO) completion paradigm.
Multi-view Perceptual Fusion We programmatically construct TAC2’s target timelines by fusing predictions from specialized weak teachers, including audio captioning, speaker diarization, and sound event detection. While each teacher provides complementary task-specific predictions under a distinct schema, all predictions share physical time as a common coordinate. We exploit this common axis to align their disparate temporal boundaries and attributes (Figure 2).
Let the latent scene comprise acoustic events over the attribute schema . Each of specialized teachers produces a partial view , with a temporal support .
To fuse these views, we define an authority map that maps each attribute to its most reliable teacher. Furthermore, each acoustic event class is assigned a temporal owner that dictates its master temporal segmentation. Temporal agreement between two predictions is measured using their temporal Intersection over Union (IoU):
For each event class , let be its reference events drawn from its temporal owner . Against the time-ordered items of view , we compute a one-to-one partial alignment that maximizes total temporal overlap over matched references:
where is minimum temporal overlap threshold. We then construct the fused target timeline by taking each event’s temporal span from its temporal owner and each attribute from its designated authority:
where represents the class events of the temporal owner, and is a deterministic attribute selection function (for instance, longest string for open-text descriptions) that resolves cases where an annotator contributes multiple candidate values.
Instruction Input Output completion format Decoding a nested, multi-granularity timeline in a single pass imposes an all-or-nothing inference paradigm. In practice, however, downstream users may already possess high-quality partial signals such as an expert human transcript or output from a specialized diarization system. A standard end-to-end captioner cannot naturally incorporate these external priors to condition its remaining predictions, limiting its use as a modular component in larger audio pipelines.
We address this limitation with an Instruction-Input-Output (IIO) completion format, which fans out the target schema into completion tasks over the same audio (Figure 3). The INSTRUCTION specifies the task, the INPUT is either empty or contains teacher-forced context such as a transcript or speaker turns, and the OUTPUT is the remaining schema that we expect the model to generate. We train TAC2 as a LoRA adapter on a frozen pre-trained backbone , with adapted parameters , by minimizing the completion loss over the fanned-out tasks:
Figure 3. The IIO completion format. The fused target is decomposed into (INSTRUCTION, INPUT, OUTPUT) completion tasks. The INSTRUCTION specifies the task, INPUT provides optional teacher-forced context, and OUTPUT specifies the target attributes to generate. Structural tasks train TAC2 (caption, diarize, and a single-decode comprehensive task). Appending elicited speech tasks (stress, emotion, voice profile, acoustics) seamlessly extends this format to TAC2-S without changing the training procedure. At inference, the comprehensive task supports single-pass generation, while sequencing attribute-specific tasks with preceding outputs yields a modular multi-pass cascade.
where is the audio, the instruction, the optional teacher-forced context, and the target completion.
Two properties make IIO central to our approach. Firstly, it is extensible: new attributes enter the schema as additional completion tasks without changing the training procedure. Stage 2 (§ 3.2) exploits this property by appending elicited speech attributes to the same task fan-out (Figure 3, + TAC2-S rows). Second, each IIO task acts as a composable function , that is, the model prompted with the instruction of task . With an empty INPUT, the comprehensive task generates the full acoustic timeline in a single call, . Alternatively, attribute-specific tasks take preceding predictions or externally supplied priors as INPUT, so calls compose into a multi-pass cascade, for example . Likewise, injecting a high-quality transcript from an external ASR system allows subsequent attributes to be predicted while holding the transcript fixed. This same mechanism enables data bootstrapping as stronger external priors become available. We compare these inference modes in Section 4.1 and detail the per-task sampling mixture in Appendix D.
Stage 2: Speech-centric Elicitation (TAC2-S)
We next extend TAC2’s physical timeline with expressive speech attributes, including word stress, intonation, emotion, and speaker characteristics. Rather than learning these capabilities from scratch at scale, we elicit paralinguistic information already encoded within the pre-trained backbone using lightweight specialists, then distill their annotations into the unified model, TAC2-S.
Parameter-efficient Specialist Elicitation Clean paralinguistic labels are scarce at the scale required for our in-the-wild corpus. We therefore formulate this expansion as latent capability elicitation. Pre-trained models can encode capabilities internally that are not reliably expressed in their outputs, but can be surfaced through limited parameter updates (Donoway et al., [12]). We hypothesize that the pre-trained LALM backbone similarly encodes expressive acoustic traits that are poorly exposed through its textual generation.
We test this hypothesis through layer-wise linear probing of the intermediate hidden states of the Qwen3Omni-captioner model. For a binary expressive attribute , we fit an regularized
logistic probe at every layer by minimizing the cross-entropy:
where and is the sigmoid function. Our analysis reveals a gap between latent decodability and generative accessibility. Paralinguistic traits are strongly linearly decodable from the frozen backbone despite substantially weaker textual attribute generation (Table 6). Specialist adaptation leaves early- and mid-layer decodability largely unchanged, with gains concentrated in later layers. Appendix § 11 provides the layer-wise analysis.
| Attribute (metric) | Frozen backbone | + specialist |
| Word stress (top-word accuracy) | 0.60 | 0.81 |
| Accent (exact, 13 classes) | 0.37 | 0.62 |
| Accent (region) | 0.61 | 0.91 |
| Emotion (5 classes) | 0.15 | 0.59 |
Table 6. The specialist mitigates output mode collapse without altering early perception. Accuracy of the emitted textual attribute, frozen backbone vs. its specialist, on clips with ground-truth labels (word stress is scored as the top stressed word against Whistress ((Yosha et al., 2025)).
Motivated by this latent perception, we perform sample-efficient acoustic elicitation. More concretely, instead of using large datasets to teach the model how to perceive prosody from scratch, we train attention-only Low-Rank Adaptation (LoRA) specialists on small gold-standard datasets (K samples). Given the strong decodability of these attributes in the frozen backbone, these specialists primarily learn to expose the latent representations as explicit, structured output tags rather than learning the underlying acoustic features from scratch.
We train one LoRA specialist for each speech attribute (such as word stress, intonation, emotion, and voice profile) by adding a parameter efficient adapter to the frozen base :
where supplies a task-specific (teacher-forced) context, such as transcript or speaker list. The context allows each specialist to focus on predicting its target attribute rather than re-deriving the surrounding description. We detail the conditioning for specialists in Appendix § 10.
Once trained, we hot-load the specialists as co-existing LoRAs over the frozen Qwen3Omni-captioner to form our teacher cascade, TAC2-E. We deploy this over our in-the-wild corpus to augment the Stage 1 timelines with speech-centric, multi-attribute distillation targets.
Unified Student Distillation We distill TAC2-E into a single unified student, TAC2-S. Starting from the Stage 1 TAC2 weights , we learn a student LoRA :
where denotes the enriched targets produced by TAC2-E. The paralinguistic attributes are incorporated as additional fanned-out IIO tasks mixed with the Stage 1 objectives, allowing TAC2-S to learn multi-granularity completion across the distillation target.
Experiments
A good describer should encode attributes relevant for both reasoning and generation in its text output. Here, we evaluate our trained captioners in these two settings: audio understanding, by cascading TAC2-S with a frozen text-only LLM which answers questions based on TAC2-S’s generated description, and audio generation, by using its structured descriptions as paired supervision for text-to-audio generation.
Audio Understanding: validating schema fidelity
Under the Describe-then-Reason (DtR) protocol [24], TAC2-S converts audio into a structured acoustic schema that a frozen text-only LLM uses to answer multiple-choice questions. This helps us assess whether the schema preserves the perceptual information required for downstream reasoning. We evaluate on MMSU [38], the general audio benchmarks MMAR [26] and MMAU [31], and MSU-Bench [40]. TAC2-S achieves an 81.9 overall score on MMSU, outperforming existing LALMs and nearing
TAC2-S, the teacher cascade (82.9). The gain is concentrated in perception (Table 1): TAC2-S reaches 77.8 on MMSU perception, compared with 68.7 for Gemini-3.7 Flash, 63.8 for TAC, and 63.5 for Qwen3Omni-Captioner, while retaining strong reasoning performance (86.4). The attribute buildup in Figure 4 further localizes these gains, whereby adding individual schema attributes preferentially improves their associated perception tasks while leaving transcript-driven semantic tasks comparatively stable. This advantage is specific to fine-grained acoustic representation rather than general audio QnA alone. Frontier OmniLLMs remain weaker on fine-grained acoustics, timing, loudness, and phonetics, whether operating directly on audio or under the same protocol (Table 2, right). Beyond MMSU, TAC2-S remains competitive on the broader MMAR and MMAU benchmarks, reaching 68.0 and 75.6, respectively. On the speaker-centric MSU-Bench, it achieves 82.2 overall (79.2 Tier 1; 86.6 Tier 2), outperforming the baselines and TAC2-E (Table 3).3
| Model | DtR | MMSU Perception | MMSU Reasoning | MMSU Overall | MMAR Speech | MMAR Sound | MMAR Music | MMAR Overall | MMAU Speech | MMAU Sound | MMAU Music | MMAU Overall | Avg. |
| Kimi-Audio | ✗ | 44.8 | 75.7 | 59.8 | 58.5 | 49.7 | 33.0 | 48.0 | 62.2 | 75.7 | 66.8 | 68.2 | 58.7 |
| Qwen2.5-Omni | ✗ | 42.7 | 77.6 | 58.1 | 59.9 | 58.8 | 40.8 | 56.7 | 70.6 | 78.1 | 65.9 | 71.5 | 62.1 |
| StepAudio2 | ✗ | 42.9 | 73.2 | 57.6 | 61.2 | 54.6 | 42.2 | 56.8 | 68.2 | 79.3 | 68.4 | 72.7 | 61.9 |
| UAS-Audio | ✗ | 55.7 | 77.4 | 66.2 | 66.0 | 58.8 | 45.2 | 60.1 | 67.0 | 70.0 | 71.3 | 69.4 | 65.2 |
| Gemini-3.7 Flash | ✓ | 68.7 | 84.4 | 76.3 | 82.7 | 51.5 | 56.8 | 69.5 | 80.5 | 66.1 | 67.1 | 71.2 | 72.3 |
| Qwen3Omni-Capt. | ✓ | 63.5 | 83.4 | 73.1 | 77.6 | 59.4 | 54.4 | 69.2 | 84.1 | 73.0 | 68.0 | 75.0 | 72.4 |
| TAC | ✓ | 63.8 | 84.8 | 74.0 | 83.7 | 62.4 | 57.3 | 71.1 | 82.0 | 69.4 | 69.8 | 73.7 | 72.9 |
| TAC2-E | ✓ | 79.0 | 87.1 | 82.9 | 84.0 | 56.4 | 52.9 | 70.3 | 80.5 | 71.8 | 68.3 | 73.5 | 75.6 |
| TAC2-S | ✓ | 77.8 | 86.4 | 81.9 | 79.6 | 53.3 | 51.0 | 68.0 | 82.3 | 73.9 | 70.7 | 75.6 | 75.2 |
Table 1. Spoken and general audio understanding. Models in DtR ✓ rows emit a text schema as the description, which is read by a frozen text-only reasoner. Each score reflects the fidelity and coverage of audio perception in its generated description.
| Text-only reasoner | SP | MP |
| TAC2-S → Gemini3 | 81.3 | 81.9 |
| TAC2-S → Qwen3-Next | 78.8 | 79.7 |
| TAC2-S → Qwen3.8 | 80.5 | 80.9 |
| Tasks (MMSU) | SP | MP |
| Pitch comparison | 63.0 | 75.9 |
| Vocal-range comp. | 60.2 | 72.2 |
| Emotional context | 43.6 | 55.5 |
| Accent ident. | 59.6 | 69.3 |
| Accent | ||
| Cons-vowel | ||
| Content | ||
| Turn count | ||
| Disfluency | ||
| Emotion | ||
| Gender | ||
| Intonation | ||
| Near-homoph | ||
| Non-verbal | ||
| Pause | ||
| Plosive | ||
| Pitch cmp | ||
| Speaker ID | ||
| Prolong | ||
| Speech act | ||
| Duration | ||
| Stress | ||
| Speed cmp | ||
| Syllable | ||
| Spk count | ||
| Vocal range | ||
| Volume |
Table 2. Inference and Reasoner Robustness. Left: TAC2-S schemas under single-pass (SP) vs. multi-pass (MP) comprehensive decodes across text-only reasoners (top), and the MMSU tasks where MP helps most (bottom). Right: Perception fidelity. TAC2-S uses describe-then-reason; OmniLLM baselines process raw audio directly (left radar) or act as describers (right radar). TAC2-S (ours), Gemini 3.7, Qwen3Omni.
| System | T1 | T2 | Avg |
| Kimi-Audio | 41.0 | 47.0 | 43.0 |
| Qwen2.5-Omni | 19.0 | 21.0 | 19.0 |
| StepAudio2 | 44.0 | 46.0 | 44.0 |
| AudioFlamingo-3 | 40.0 | 38.0 | 39.0 |
| MiMoAudio | 52.0 | 64.0 | 56.0 |
| Gemini-3.7 Flash | 70.7 | 84.8 | 76.4 |
| Qwen3-Omni | 69.9 | 83.2 | 75.3 |
| TAC2-E | 78.2 | 87.3 | 81.9 |
| TAC2-S | 79.2 | 86.6 | 82.2 |
Table 3. Speaker-centric understanding on MSU-Bench. Evaluated on conversational multi-speaker scenarios in English/Mandarin. Left: Consistent with Table 1, TAC2-S uses a text reasoner. Right: Tier 1 (Identification) task accuracy. (AI: accent identification, AR: age recognition, GI: gender identification, ER: emotion recognition, SP: speaker profiling, SR: speaker retrieval, SVS: speaker-specific viewpoint summarization, SC: speaker counting, SV: speaker verification) OmniLLM baselines process raw audio directly (left radar) or act as describers (right radar). TAC2-S’s trace (blue) is superior across both, demonstrating robust speaker-centric perception. TAC2-S (ours), Gemini 3.7, Qwen3Omni.
Figure 4. Impact of individual IIO tasks. (left) Building TAC2-S’s description one attribute at a time, from a transcript-only base (who spoke, the words, and their timestamps). (right) For each attribute, the top-3 it lifts most, the gains land on the tasks that attribute governs and leave the transcript carried semantics tasks flat. Bar colour marks the added attribute (blue: stress and intonation, pink: emotion, orange: voice profile(demographics), green: pitch and loudness).
Inference and Reasoner Robustness
We ablate IIO decoding and the text reasoner in Table 2, left. Collapsing multi-pass (MP) inference, where each task conditions on the accumulated description, into a single (SP) decode incurs a modest MMSU drop (81.9 → 81.3). However, MP yields larger gains on perceptual tasks such as pitch (+12.9), vocal range (+12.0), and accent identification (+9.7). Holding the TAC2-S schema fixed and swapping the reasoner from Gemini3.5-Flash with Qwen3-Next or Qwen3.8-Omni retains similar MP performance (79.7 and 80.9, respectively). In contrast to TAC (Kumar et al., [24]), where scaling the reasoner improved results, this relative robustness to reasoner choice suggests that TAC2-S’s detailed schema already exposes much of the perceptual information needed for downstream reasoning. Together, these results show that composing IIO functions can improve fine-grained acoustic inference through intermediate conditioning, while the resulting schema remains effective across different text-only reasoners.
Generation: The Description as Paired Supervision for Audio Synthesis
Our second validation tests whether the structured perceptual representation is sufficiently grounded to be renderable back into audio. We use TAC2-S to relabel in-the-wild audio at scale, producing structured timestamped captions as paired supervision for a text-to-audio diffusion model. If the generated schemas faithfully capture the underlying scene, conditioning on its event timeline should reconstruct both what occurs and when it occurs (Figure 5).
We condition the generator on the schema of held-out clips and localize events in the synthesized audio using two frame-level audio-text localizers, FLAM ((Wu et al., 2025)) and PE-A-Frame ([36]). Temporal error is measured as the mean absolute offset between localizer peaks in the original and reconstructed audio (Table 4). Our generator provides the strongest temporal grounding, achieving 1.54s PE and 2.14s FLAM error, compared with 2.47s and 3.16s for the explicitly timestamp-conditioned PicoAudio2 ((Zheng et al., 2026)). Removing timestamps from the TAC2-S captions substantially degrades localization (1.54 → 2.93s PE; 2.14 → 3.16s FLAM), thereby demonstrating that the generator uses the predicted temporal structure.
| System | TAC2-S-as-judge Gen. ↑ | Age ↑ | Acc. ↑ | Emo. ↑ | Ev.R. ↑ | Sp.R. ↑ | WER ↓ | Temporal grounding PE ↓ | FLAM ↓ | PE-AV ↓ | Gemini-3-Pro judge vs. real ↑ | vs. cap. ↑ |
| PicoAudio2 ((Zheng et al., 2026)) | 0.44 | 0.35 | 0.24 | 0.28 | 0.27 | 0.09 | 1.15 | 2.47|2.77 | 3.16|3.50 | 0.52|0.51 | 1.39 | 1.61 |
| MiDashengLM-Gen ([33]) | 0.54 | 0.36 | 0.21 | 0.36 | 0.33 | 0.09 | 1.39 | 3.57|— | 3.78|— | 0.44|— | 1.54 | 1.60 |
| Seed Audio 1.0 ([8]) | 0.94 | 0.89 | 0.71 | 0.74 | 0.78 | 0.90 | 0.14 | 2.16|— | 2.62|— | 0.35|— | 3.53 | 4.05 |
| Ours (TAC2) | — | — | — | — | 0.80 | 0.94 | 0.12 | 1.67|2.47 | 2.13|2.77 | 0.35|0.35 | 3.38 | 4.01 |
| Ours (TAC2-S) | 0.99 | 0.84 | 0.69 | 0.77 | 0.80 | 0.93 | 0.14 | 1.54|2.93 | 2.14|3.16 | 0.30|0.33 | 3.95 | 4.52 |
Table 4. Structured captions generated by TAC2-S provide renderable supervision for audio synthesis. Temporal-grounding cells report with|without input timestamps. PE and FLAM measure mean localizer-peak offset in seconds; PE-AV is a text-free frame-wise semantic distance between original and reconstructed audio (real-vs.-real = 0). Gemini-3-Pro judgments are on a 1–5 scale.
Furthermore, we compare the recovered fields against the conditioning description using TAC2-S-as-judge. The reconstructed audio closely preserves speech content and speaker attributes, reaching 0.99 gender accuracy, 0.84 age accuracy, 0.69 accent accuracy, 0.77 emotion accuracy, 0.93 speech recall, and 0.14 WER. An audio-native Gemini-3-Pro judge rates our audio highest for both resemblance to the original (3.95) and adherence to the caption (4.52). We match the strongest commercial baseline, Seed Audio 1.0 ([8]), on speaker attributes, while Seed Audio 1.0 trails our generator on event timing and on both judgments. Additional details are in Appendix § 16.
Figure 5. TAC2-S-conditioned reconstruction. We caption real audio (top) with TAC2-S (middle), then condition the trained text-to-audio generator on it to synthesize new audio (bottom). The reconstruction reproduces speech and non-speech events, including background elements like a censor beep, engine rev, camera click, at times aligned with the real audio.
In Figure 6, we demonstrate that TAC2-S’s structured schema exposes explicit attributes as generation-time controls. For a fixed sentence, we vary only the emphasized word and its intensity in the TAC2-S’ caption. Without training a dedicated stress-control model, the synthesized stress peak follows the commanded word and increases with the requested emphasis level. Thus, paralinguistic attributes elicited for perception also become explicit control channels when the same representation is used as supervision for generation.
Figure 6. Word-level emphasis as a renderable control channel. Each column synthesizes the same sentence while varying the emphasized word in the TAC2-S caption (yellow). The bottom two rows increase emphasis from mild (<e>) to strong (<E>) while the cyan curve shows the contour. Bottom panels show WhiStress (Yosha et al., 2025) stress probability over time, with ⋆ marking the commanded word. The detected stress peak follows the targeted word and scales with emphasis intensity, demonstrating zero-shot emphasis control without a dedicated stress-control model.
Conclusion
We introduce TAC2 and its speech-centric extension, TAC2-S, for constructing queryable, temporally grounded acoustic representations from in-the-wild audio. Our framework bootstraps granular, speaker-attributed timelines at scale through multi-view weak supervision, models them as composable functions through the Instruction-Input-Output (IIO) completion format, and extends them with paralinguistic attributes through sample-efficient latent capability elicitation. The resulting unified model provides a structured interface between perception, reasoning, and generation. Paired with frozen text-only LLMs, it enables strong spoken audio understanding. Used as supervision for text-to-audio generation, it supports faithful reconstruction and explicit control over event timing, speaker attributes, and word-level emphasis. Overall, our results demonstrate that a high-fidelity, structured, temporally grounded acoustic timeline can function as both an interpretable reasoning-ready representation of audio and a renderable control interface for generation.
AI USE STATEMENT
The authors used generative AI to assist with language editing of the manuscript. We have reviewed and take responsibility for the final content of this work, including text produced with the aid of generative AI. All scientific content and conclusions were produced by the authors.
REPRODUCIBILITY STATEMENT
Our two-stage method is described in § 3, with architecture and training details (backbone, LoRA rank and adapted modules, optimizer, step counts for TAC2 and TAC2-S) in Appendix § 10. The time-registered IoU matching is formalized in Eq. (1)–(3) (Figure 2). The IIO format, and the per-task mixture and inference regimes in § 3.1, Figure 3, and Appendix § 12; The specialist elicitation, probing setup, and per-attribute gold datasets are detailed in § 3.2 and Appendix § 11. The text-to-audio validation (generator, evaluation set, metrics, baselines, and judge) and all evaluation protocols and judge prompts are provided in Appendices § 16 and § 17. Our backbone (Qwen3-Omni-Captioner), the specialist gold corpora (IEMOCAP ([7]), RAVDESS ([25]), MELD ([30]), Common Voice ([3])), and all benchmarks are publicly available. Our in-the-wild corpus uses proprietary licensed audio sources, instead we provide complete time-registered IoU matching to reproduce our results with alternative audio sources.
Figure 2. Programmatic data compilation via time-registered IoU matching. Predictions from three independent weak teachers (: audio captioner; : speaker diarizer; : sound event detector) are aligned on a shared time axis using temporal IoU with overlap gate (Eq. (1),2). Their aligned predictions form the fused target timeline (Eq. (3)), containing concurrent acoustic events with speaker attribution.
Summary of Appendix
Section A: Limitations and Future Work
Section B: Implementation Details
Section C: Elicitation: Per-Layer Probing
Section D: IIO Tasks
Section E: Segment- and pause-level localization
Section F: Long-form audio understanding
Section G: Extending the Describer to Audio-Visual Reasoning
Section H: Text-to-Audio Model and Temporal Grounding Setup
Section I: Prompts
Limitations and Future Work
Writing perception as an explicit textual description makes TAC2-S’s errors readily interpretable, tracing back to either an incorrect recognition of an existing attribute or a lack of schema coverage. Currently, TAC2-S’s schema prioritizes spoken attributes over non-speech and music events, a limitation reflected in benchmarks such as MMAR and MMAU. Interestingly, our current schema format enforces a temporal resolution tradeoff. Repeated events are merged into one span (for example, <speech>...</speech> from 1.4s to 3.8s, 5.4s to 6.7s) with comma-separated timestamps. While this fine temporal granularity helps TAC2-S act as a crucial reference for tasks like pause detection, disfluency detection, duration estimation etc. (see Table 2 (right), and Appendix § 13), it occasionally merges turns when consecutive voices are similar, causing the model to under-count unique speakers.
It is important to note that these bottlenecks reflect coverage limits rather than our architectural constraints. Our IIO proposal is inherently extensible, and new perceptual axes as well as upgraded teachers can be seamlessly integrated via teacher-forced bootstrapping. More broadly, human-designed schemas will eventually hit a coverage ceiling, oversight leading to bottleneck. In future, we aim to let descriptions evolve autonomously via reinforcement learning, using frozen text reasoners to reward the extraction of task-demanded cues.
Implementation details
TAC2 and the individual speech specialists are independently trained as rank-512, attention-only LoRA adapters over the same frozen Qwen3-Omni-30B captioner. At inference, the specialists can therefore be hot-loaded over this shared backbone to form the modular TAC2-E cascade. The unified TAC2-S is instead trained as an additional LoRA over the Stage 1 TAC2 weights.
Stage 1. We train TAC2 on approximately two million licensed, in-the-wild audio clips, each up to 30 seconds long. Each clip is paired with a fused target timeline constructed using the multi-view fusion procedure described in § 3.1. We combine specialized weak teachers: (1) a TAC-style sound event detector ([24]) for non-speech event spans and types; (2) a TAC-inspired diarizer ([24]) for speech turns and speaker identities; and (3) the Qwen3-Omni captioner ((Xu et al., 2025)) for transcripts, voice descriptions, and scene overviews. We additionally apply wav2vec2-based ([4]) forced alignment to obtain word-level timestamps. The per-type overlap gate of Eq. (2) is set to 0.2 IoU for sound-effect events and 0 for speech and other non-speech events (rely on the order-preserving alignment with a midpoint-proximity fallback). A separate 0.5 IoU gate de-duplicates overlapping event descriptions. We selected these values by manual inspection of the merged outputs on a small validation set of clips. The resulting targets are decomposed into the IIO tasks described in § 3.1, with training samples weighted proportionally to their speaker count. We optimize TAC2 with AdamW for 10,000 steps with a learning rate of , a 1,000-step linear warmup, and cosine decay.
Stage 2. We train each speech specialist on fewer than 10,000 labeled examples, using task-specific teacher-forced context so that the adapter predicts only its designated paralinguistic attribute. For word stress and intonation, we use the multilingual A560E dataset ((Wei et al., 2026)), spanning 10 languages, and are conditioned on the transcript, incorporating per-word relative pitch extracted via torchcrepe ((Morrison, 2023)). For emotion, we use IEMOCAP ([7]), RAVDESS ([25]), and MELD ([30]), predicting utterance-level emotion directly from the audio. For voice profile (gender, age, and accent), we use silver labels obtained from a per-speaker Qwen3-Omni probe over the licensed corpus, supplemented with a balanced Common Voice ([3]) accent shard to mitigate the US-accent skew of the silver labels, conditioned on the caption. Finally, relative pitch and loudness are derived from DSP measurements (CREPE F0 and RMS energy) ranked within each clip and conditioned on the temporal event list.
Once trained, these specialists are concurrently hot-loaded alongside TAC2 over the shared frozen backbone to form TAC2-E, which is subsequently used to re-annotate the complete Stage 1 corpus. A two-pass LLM rewriting step, using Qwen3-Next-80B, then rectifies the overview with the per-event descriptors from the hot-loaded specialists, for example to keep speaker count, gender, and accent consistent.
We initialize TAC2-S from the Stage 1 TAC2 weights and train an additional rank-512, attention-only LoRA on the enriched corpus. The elicited speech attributes are introduced as additional IIO tasks and mixed with the original Stage 1 objectives, preserving the same IIO fan-out used by TAC2.
We train TAC2-S for 2,000 steps using the same optimizer and learning rate as TAC2, with a 100-step linear warmup and cosine decay. The resulting model consolidates the modular predictions of TAC2-E into a single adapter while retaining the structural capabilities learned by TAC2.
Elicitation: Per-Layer Probing
Section 3.2 hypothesizes that the Qwen3-Omni backbone already encodes paralinguistic information internally, while our attention-only LoRA specialists primarily improve its accessibility to late-stage generation rather than learning new perceptual features. We examine this hypothesis through layer-wise linear probing before and after specialist adaptation.
Per-layer probe. For each paralinguistic attribute, we fit an -regularized logistic probe independently at each of the model’s 49 hidden layers. Representations are extracted at the final token position corresponding to the attribute’s generation step. We report 5-fold cross-validated AUC, with macro one-vs-rest averaging for multiclass attributes such as accent.
Three consistent patterns emerge (Table 5). First, every evaluated trait exhibits high linear decodability within the frozen backbone (peak AUCs between 0.89 and 0.99). Second, across the early and middle transformer blocks (depths 0–24), the representation quality of the backbone and the specialist are virtually identical (e.g., word stress AUC is 0.88 for the base versus 0.89 for the specialist), confirming that the adapter preserves early acoustic processing. Third, the adapter’s structural impact is strictly localized to the late layers (36–48), where it maximizes performance within the available headroom. This relative gain is most pronounced for relational traits that the base model struggles to articulate (e.g., relative loudness and pitch, yielding a +0.10 to +0.12 AUC lift). For traits already near the performance ceiling internally (e.g., accent at +0.01 lift), the specialist functions primarily to calibrate the final output distribution rather than enhance latent separability.
| Trait (granularity) | Backbone peak | Early (base / spec) | Late (base spec) | Lift |
| Relative loudness (segment) | 0.89 | 0.79 / 0.81 | 0.87 0.99 | +0.12 |
| Relative pitch (segment) | 0.90 | 0.80 / 0.82 | 0.89 0.99 | +0.10 |
| Word stress (word) | 0.93 | 0.88 / 0.89 | 0.91 0.99 | +0.08 |
| Accent (speaker) | 0.99 | 0.97 / 0.97 | 0.99 1.00 | +0.01 |
Table 5. The backbone (Qwen3Omni-captioner) possesses latent perception; the specialist isolates late-stage readouts. Early/Late: mean AUC over depths 0–24 / 36–48, backbone vs. specialist. Lift: the specialist’s late-stage gain.
Mitigating output collapse and bias. High internal decodability does not inherently translate to accurate generation. Post-training alignment heavily biases the decoding trajectory, causing the base model’s textual outputs to exhibit mode collapse toward majority classes. Table 6 contrasts the generated attribute accuracy of the frozen backbone against the specialist on a ground-truth evaluation set.
The baseline generation errors are highly systematic. Accent predictions overwhelmingly default to “United States” (emitted for 42% of samples compared to a 9% true prior), and emotion predictions degenerate to “neutral” (76% of samples, despite “neutral” not appearing as a valid ground-truth class). Concurrently, word stress exhibits a severe false-positive rate (1.5 stressed words per clip at 0.21 precision).
The specialist adapter effectively mitigates this generation bias, substantially recovering the true attribute distributions without altering the underlying perceptual manifold (consistent with the identical early-layer AUCs). Furthermore, the specialist’s residual accent errors primarily occur between geographically adjacent regions (e.g., Canada United States, Ireland United Kingdom, New Zealand Australia, Kenya Nigeria). Consequently, strict 13-class exact match accuracy underestimates the model’s true perceptual fidelity; evaluating at the broader regional level increases the specialist’s accuracy to 0.91, compared to 0.61 for the frozen backbone.
IIO tasks
The unified student model is trained using the extension of our proposed Instruction-Input-Output (IIO) format. During training, each example is formatted as a single completion task, sampled according to fixed probabilities. The mixture comprises per-attribute tasks which describe isolated paralinguistic features and one comprehensive task designed to output the entire structured description at once. The comparative attribute specialists always receive the speaker turns and transcript as teacher-forced input. Conversely, the caption and comprehensive tasks take no text input and decode directly from the raw audio (Table 7). To establish single-pass capabilities, we append the comprehensive task (empty INPUT) at a weight of 1.0 and reduced the caption task weight to 0.5. The comprehensive task adds single-pass capability while retaining the cascaded multi-pass inference regime.
| Task | Input | Weight | Output |
| comprehensive | none (audio) | 1.0 | full description (all sections) |
| caption | none (audio) | 0.5 | overview + events with transcript |
| stress | transcript turns | 0.5 | per-word stress and intonation |
| emotion | transcript turns | 0.5 | per-turn emotion |
| voice profile | transcript turns | 0.3 | per-speaker gender, age, accent |
| acoustics | transcript turns | 0.3 | per-segment pitch and loudness |
Table 7. IIO tasks during training. Each training sample is formulated as a single completion task, with weights denoting their respective sampling probabilities. The comparative speech specialists are conditioned on the speaker turns and transcript as teacher-forced inputs.
At inference, the dynamic IIO surface allows the model to operate in two distinct regimes. The multi-pass approach executes a cascaded sequence of five decodes. First, the caption task processes the raw audio to generate the scene overview and the timestamped events list, including the verbatim transcript. The predicted speaker turns from this initial pass are subsequently fed back as the INPUT conditioning for the four specialist decodes (stress, emotion, voice profile, and acoustics). This composed teacher-forcing ensures all fine-grained paralinguistic descriptors remain structurally aligned to the same master transcript. Conversely, the single-pass approach utilizes only the comprehensive task. By reading the raw audio with an empty INPUT IIO, it emits the complete structured description, spanning all non-speech events as well as speech attributes, in a single, continuous output block. Both inference approaches yield the identical structured schema.
Segment- and pause-level localization
TAC2’s training schema records a turn broken by a pause as a comma-separated list of active spans (for example, <speech>...</speech> from 1.4s to 3.8s, 5.4s to 6.7s). We test whether this supervision produces accurate segment- and pause-level localization on real conversational speech, and how it compares to dedicated aligners.
Specifically, we evaluate through the caption task (Appendix D) wherein we mark segment boundaries in the reference transcript with the delimiter (\\), and recover each segment’s predicted timestamps. A segment is a run of speech bounded by a pause, whether a change of speaker or the same speaker pausing mid-utterance.
We use the Buckeye corpus of conversational American English Pitt et al. [29] (eight development speakers, 50 recordings). We define a segment as a run of target-speaker words bounded by a silence of at least 0.1 s or an interviewer turn (provided in the dataset), and pack consecutive segments into windows of at most 20 s yielding 1,349 windows and 4,947 segments.
For reference we run two specialized alignment models on the same windows, the Qwen3 aligner Shi et al. [32] and a wav2vec2-based aligner Hwang et al. [18]. TAC2 localizes segment onsets to a 49 ms median boundary error and attains the highest boundary agreement at a 200 ms tolerance (0.93), the most consistent of the three systems and competitive with models built solely for alignment (Table 8, Figure 7). Importantly, TAC2 itself produces the segmentation, recovering the correct segment count 84% of the time, whereas the aligners require this as input.
Figure 7. Segment localization on conversational speech. The segments from the human annotation, TAC2, the Qwen3-FA aligner, and the wav2vec2-based aligner share the time axis. TAC2’s segment onsets track the human pause boundaries closely, while being competitive with the dedicated aligners. (Dotted lines mark the human segment onsets.)
Table 8. Segment- and pause-level localization on conversational speech. Median onset error (ms, ↓) and boundary F1 at 100/200 ms (↑). Count: fraction of windows where TAC2 proposes the correct segment count.
Long-form audio understanding
We test whether TAC2, trained on clips of at most 30 s, extends to recordings of up to 2.7 h on LongAudioSpan Huang et al. [17]. It contains 360 English and Chinese recordings in three duration tiers, with questions scored in three modes: multiple-choice Accuracy, open-ended Rubric, where an LLM judge checks both content and timing, and Chain, three linked questions about an edit inserted into the audio. We describe each recording in chunks of at most 30 s and join the descriptions on one timeline. A frozen text-only GPT-5.4 answers from this description alone, using the benchmark’s prompts and official scorers.
TAC2 outperforms every audio model in all three modes and tiers (Table 9), and on Accuracy nearly matches the dense caption the questions were written from (79.0 vs. 80.2). The largest gain is on Rubric, which asks when events happen. For closed-source audio models, the share of correctly placed timestamps falls from 42% on the shortest tier to 23% on the longest (Huang et al. [17]), whereas reading the TAC2 timeline, the reasoner places 78% and 70% correctly. On Chain, understanding answers are 17 points more accurate when the edit is located than when it is missed, more than for any closed-source audio model (at most 13 points), so the credit comes from grounding rather than guessing.
| Model | Size | Think | Limit | Accuracy (%) S | Accuracy (%) M | Accuracy (%) L | Accuracy (%) Avg | Rubric (%) S | Rubric (%) M | Rubric (%) L | Rubric (%) Avg | Chain (%) S | Chain (%) M | Chain (%) L | Chain (%) Avg |
| Text-only Baselines | |||||||||||||||
| GPT-5.4 (question-only) | – | ✓ | – | 41.1 | 41.1 | 40.6 | 40.9 | – | – | – | – | 8.9 | 11.9 | 10.3 | 10.4 |
| GPT-5.4 (transcript) | – | ✓ | – | 64.8 | 60.7 | 53.4 | 59.7 | 41.0 | 41.5 | 37.2 | 39.9 | 9.4 | 14.2 | 14.2 | 12.6 |
| GPT-5.4 (caption) | – | ✓ | – | 83.2 | 80.0 | 77.3 | 80.2 | 59.1 | 57.4 | 53.0 | 56.5 | 66.7 | 64.4 | 53.1 | 61.4 |
| Open-source Models | |||||||||||||||
| Phi-4-Multimodal | 5.6B | ✗ | 30m | 33.7 | 34.6 | 34.2 | 34.2 | 18.4 | 13.6 | 10.1 | 14.0 | 12.9 | 7.2 | 8.8 | 9.6 |
| AudioFlamingoNext | 8B | ✗ | 30m | 46.1 | 43.6 | 37.2 | 42.3 | 16.1 | 12.2 | 13.7 | 14.0 | 5.3 | 5.6 | 6.1 | 5.7 |
| ✓ | 30m | 43.3 | 38.9 | 35.0 | 39.1 | 26.9 | 20.3 | 16.3 | 21.2 | 7.5 | 11.1 | 13.1 | 10.6 | ||
| Voxtral-Mini | 3B | ✗ | 40m | 45.3 | 35.8 | 37.2 | 39.4 | 25.6 | 21.8 | 13.6 | 20.3 | 5.6 | 9.4 | 18.3 | 11.1 |
| Voxtral-Small | 24B | ✗ | 40m | 42.5 | 40.0 | 31.9 | 38.1 | 28.9 | 24.2 | 20.7 | 24.6 | 12.5 | 4.7 | 4.7 | 7.3 |
| MOSS-Audio | 8B | ✗ | 50m | 45.0 | 43.1 | 35.6 | 41.2 | 23.7 | 17.7 | 10.7 | 17.4 | 8.3 | 11.4 | 9.2 | 9.6 |
| ✓ | 50m | 52.2 | 52.8 | 38.6 | 47.9 | 30.1 | 15.4 | 11.6 | 19.0 | 13.9 | 15.3 | 11.4 | 13.5 | ||
| Baichuan-Omni-1.5 | 11B | ✗ | 80m | 46.9 | 46.9 | 36.4 | 43.4 | 19.0 | 12.5 | 9.7 | 13.7 | 11.9 | 14.4 | 14.7 | 13.7 |
| Qwen3-Omni | 30B (A3B) | ✗ | 80m | 53.9 | 54.4 | 46.2 | 51.5 | 31.3 | 24.3 | 20.6 | 25.4 | 20.5 | 19.2 | 13.2 | 17.6 |
| ✓ | 80m | 56.9 | 57.5 | 49.2 | 54.5 | 38.4 | 29.2 | 22.8 | 30.1 | 7.8 | 10.6 | 15.3 | 11.2 | ||
| Closed-source Models | |||||||||||||||
| Gemini-2.5-Pro | – | ✓ | – | 67.5 | 64.2 | 54.7 | 62.1 | 46.9 | 40.7 | 36.4 | 41.3 | 24.2 | 25.3 | 21.9 | 23.8 |
| Gemini-3-Flash | – | ✗ | – | 65.6 | 69.2 | 64.2 | 66.3 | 54.9 | 55.0 | 53.8 | 54.6 | 16.7 | 20.6 | 11.9 | 16.4 |
| Gemini-3.1-Pro | – | ✓ | – | 76.7 | 74.4 | 65.8 | 72.3 | 56.2 | 48.8 | 43.3 | 49.4 | 17.5 | 25.3 | 20.0 | 20.9 |
| Qwen3.5-Omni-Flash | – | ✗ | – | 73.9 | 71.4 | 58.9 | 68.1 | 50.3 | 40.9 | 35.9 | 42.4 | 23.1 | 19.2 | 18.3 | 20.2 |
| Qwen3.5-Omni-Plus | – | ✗ | – | 79.2 | 74.4 | 70.3 | 74.6 | 56.9 | 48.6 | 47.0 | 50.8 | 30.8 | 33.1 | 23.1 | 29.0 |
| Describe-then-Reason (ours) | |||||||||||||||
| TAC2 → GPT-5.4 (text-only) | 30B (A3B) | ✓ | – | 83.1 | 79.4 | 74.4 | 79.0 | 76.3 | 71.6 | 68.7 | 72.2 | 32.8 | 35.0 | 31.4 | 33.1 |
Table 9. Long-form audio understanding on LongAudioSpan [17]. Accuracy, Rubric and Chain over duration tiers S (10 to 30 min), M (30 to 60 min) and L (over 60 min); Avg is the mean over tiers. Baseline rows are from Huang et al. [17]; gray cells use audio truncated to the model’s input Limit (minutes). For Chain, the caption baseline sees the edits written into its caption. Our row pairs TAC2 (Size) with a frozen text-only GPT-5.4 reasoner. Best in bold, second underlined, among audio models and ours.
Extending the describer to audio-visual reasoning
The decoupled describe-then-reason paradigm is not limited to audio understanding but can readily be extended to serve as the acoustic foundation for an audio-visual describer. Following the TAC-V framework introduced by Kumar et al. [24], we generate an integrated audio-visual caption by pairing our audio describer with an off-the-shelf visual grounder (Qwen3-VL-32B). Specifically, the audio describer outputs timestamped acoustic events and speech, while a visual shot list delineates scene boundaries. The visual grounder then links acoustic events to their on-screen sources, leveraging visual context to resolve acoustic ambiguities. Finally, a frozen text reasoner answers downstream queries utilizing this unified multimodal caption alone.
We evaluate on four audio-visual reasoning benchmarks: Daily-Omni (Zhou et al., 2025), World-Sense (Hong et al., [16]), Video-Holmes (Cheng et al., [9]) and AVHBench (Sung-Bin et al., [34]). Table 10 replicates the comparative setup of Kumar et al. [24], contrasting native multimodal large language models (which process audio and video directly) against the cascaded describe-then-reason approach. The VLM + Qwen3 ablation establishes a baseline using visual-only captions devoid of acoustic descriptions. Substituting TAC2–S improves the cascade on all three reasoning benchmarks (+1.9, +0.4, +5.9 over TAC-V) and beats every native model on Daily-Omni and Video-Holmes. On AVHBench, TAC2–S-V leads or matches TAC-V on the three discriminative subtasks (audio-driven video hallucination AVH, video-driven audio hallucination VAH, and audio-visual matching AVM) but trails on the generative captioning subtask (AVC). AVC evaluates a generated caption against a single human reference using METEOR. Interestingly, we find that even a faithful paraphrase of the reference itself attains only METEOR. It is evident that wording differences alone therefore forfeit roughly two-thirds of the achievable score, compressing the absolute range for every system. We also noticed that the per-clip METEOR is high exactly when our caption’s opening subject and word choice happens to coincide with the reference and low otherwise. We believe the metric tracks phrasing agreement, not caption correctness. We form TAC2-S-V by substituting TAC2-S for TAC’s audio describer while keeping the same visual grounder and the reasoner, so the improvement is only attributable to our generated audio description.
| Benchmark | Model | Score | VLM + Qwen3 | TAC-V + Qwen3 | TAC-V + Gemini3 | TAC2-S-V + Qwen3 | TAC2-S-V + Gemini3 |
| Daily-Omni | Qwen3-Omni | 76.2 | |||||
| Gemini 2.5 Flash | 72.7 | 51.5 | 72.9 | 77.9 | 77.2 | 79.8 | |
| OmniVinci | 66.5 | ||||||
| World-Sense | Gemini 2.5 Pro | 65.1 | 37.4 | 45.7 | 58.6 | 55.8 | 59.0 |
| OmniVinci | 48.2 | ||||||
| Video-Holmes | Qwen3-Omni | 57.3 | 45.6 | 47.7 | 59.2 | 58.4 | 65.1 |
| AVHBench (AVH) | PandaGPT | 58.5 | 70.8 | 79.8 | 81.7 | 82.0 | 82.2 |
| AVHBench (VAH) | PandaGPT | 61.3 | 51.8 | 76.1 | 76.6 | 76.9 | 76.4 |
| AVHBench (AVM) | OneLLM | 60.1 | 50.5 | 56.7 | 61.6 | 72.6 | 77.5 |
| AVHBench (AVC) | Video-LLaMa | 14.0 | 12.9 | 22.6 | 20.6 | 14.1 | 13.8 |
Table 10. Audio-visual reasoning performance. Our TAC2-S-V pipeline substitutes TAC2-S for the prior TAC audio describer, keeping describe-then-reason pipeline constant. Daily-Omni, World-Sense, and Video-Holmes report multiple-choice accuracy. AVHBench reports accuracy for AVH/VAH/AVM and METEOR for the AVC subtask. Best per row in bold.
Text to audio model and temporal grounding setup
Generator. Derived from [19], our text-to-audio (T2A) architecture is a latent diffusion model that synthesizes 48 kHz audio conditioned on text captions. The base model is trained exclusively on structural TAC2 captions, which consist of a scene overview and a timestamped event list devoid of speaker characteristics. A subsequent fine-tuning phase incorporates enriched TAC2-S captions (include speaker demographics and word-level emphasis tags), that enable explicit control over speaker attributes while maintaining the temporal fidelity (see § 4.3). To evaluate the base T2A model, we deterministically ablate speaker metadata and emphasis tags from the enriched captions, retaining only the scene overview and timestamped events.
TAC2-S-as-judge. Beyond timing, we test whether the full schema survives synthesis with a describe–generate–re-describe approach. We re-caption the generated audio with TAC2-S and match the re-caption to the input caption field by field. We report accuracy for speaker gender, age, and accent and for emotion, recall for all events and for speech events, and transcript word error rate. A missing or mismatched attribute counts as an error, so every field is scored over the full clip set. This tests whether the commanded speaker attributes, events, and spoken content are recoverable from the generated audio, complementing the temporal and perceptual metrics below.
Temporal Grounding Evaluation. We construct a diverse evaluation suite of 300 held-out English clips encompassing six accents (United States, United Kingdom, Nigeria, India, Australia, and the Philippines). Each clip contains between three and ten distinct acoustic events, yielding 1,471 total events. To measure temporal fidelity without treating the generated text caption as a reliable ground truth, we utilize the original recording as a zero-distance reference. We employ FLAM (Wu et al., 2025) and PE-A-Frame [36], two frame-level audio-text models, to compute semantic similarity scores for every short-time frame against a given text query, extracting the peak activation frame as the predicted temporal location. For each event, we report the absolute temporal offset (in seconds) between the localizer’s peak in the generated audio and its corresponding peak in the original recording. This error is averaged exclusively over the events successfully localized in the original recording to ensure a reliable reference.
The peak-offset metrics localize each event against a text query. We additionally report a fully text-free measure: using PE-A-Frame’s audio tower alone, we embed the original and generated audio into one 1024- vector per 40 ms frame and report their mean per-frame cosine distance averaged over clips. The distance to original audio is zero, as reference. We believe this is a recall-heavy, global reconstruction measure. It ranks the systems identically to the peak-offset metrics and tracks the Gemini-3-Pro judge, supporting the view that the TAC2-S caption is a compact latent expressed in text from which the scene can be re-rendered.
Baselines. We benchmark against MiDashengLM-Gen Sun et al., [33], a recent text-to-audio scene generator and PicoAudio2 (Zheng et al., 2026), a model explicitly designed for timestamp control, evaluating it both with and without event timestamps to provide a clean intra-model ablation. Note that PicoAudio2 does not model speech. Finally, we evaluate Seed Audio 1.0 ByteDance Seed, [8], a highly capable commercial text-to-audio model prompted with the identical caption formatted as a scene overview followed by timestamped event lines.
Prompts
For MMSU, MMAR, and MMAU, for the audio-visual benchmarks, and for the open-weight reasoners in our robustness study (§ 4.1), we use the describe-then-reason prompts of Kumar et al. [24]. Below we provide the two prompts specific to this work: the describe-then-reason wrapper for MSU-Bench, and the audio-native judge used to evaluate text-to-audio synthesis.
MSU-Bench reasoner
We score MSU-Bench with its official evaluation harness Wang et al., [40], using its multiple-choice prompt (the question, the answer options, and a fixed output-format instruction) verbatim. We replace the audio with TAC2-S’s text description of the segment(s) referenced by each question, prepending it as shown in Figure 8. The reasoner has access to only the caption generated by our models.

Figure 8. The MSU-Bench describe-then-reason prompt. {description} is TAC2-S’s text description of the referenced segment(s). The output-format instruction is the official MSU-Bench harness prompt Wang et al., [40] used verbatim.
Text-to-audio perceptual judge (Gemini-3-Pro)
To conduct holistic perceptual evaluations, we employ Gemini-3-Pro as an audio-native judge. The model is queried greedily with one call per (system, clip) pair across the evaluation set. The judge evaluates systems across two rubrics. vs. real passes both the original recording and the generated candidate, prompting the model to rate their acoustic and thematic resemblance on a 1-5 scale. vs. caption passes the enriched TAC2-S text caption alongside the generated audio, prompting the model to evaluate how faithfully the candidate synthesizes the requested description. Since every system is judged against the same TAC2-S caption, this metric provides a direct, level-field cross-model comparison.

Figure 9. Gemini-3-Pro audio-native judge prompt for the vs. real rubric.

Figure 10. Gemini-3-Pro audio-native judge prompt for the vs. caption rubric. Since every system is judged against the same TAC2-S caption, this yields a level-field cross-model comparison.
References
- [1]Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp. 7319–7328, 2021.arxiv.org/abs/2012.13255
- [3]Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common Voice: A massively-multilingual speech corpus. In Proceedings of the Language Resources and Evaluation Conference (LREC), pp. 4218–4222, 2020.arxiv.org/abs/1912.06670
- [4]Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460, 2020.arxiv.org/abs/2006.11477
- [5]Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5185–5198, 2020.
- [6]Albert S Bregman. Auditory scene analysis: The perceptual organization of sound. The MIT press, 1990.
- [7]Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335–359, 2008.DOI
- [8]ByteDance Seed. Seed audio 1.0: Audio creation model. https://seed.bytedance.com/en/seedaudio1_0, 2026.
- [9]Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374, 2025a.arxiv.org/abs/2505.21374
- [10]Xize Cheng, Dongjie Fu, Chenyuhao Wen, Shannon Yu, Zehan Wang, Shengpeng Ji, Siddhant Arora, Tao Jin, Shiniij Watanabe, and Zhou Zhao. Aha-bench: Benchmarking audio hallucinations in large audio-language models. Advances in Neural Information Processing Systems, 2025b.DOI
- [11]Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024.arxiv.org/abs/2407.10759
- [12]Elizabeth Donoway, Hailey Joren, Arushi Somani, Henry Sleight, Julian Michael, Michael Deweese, John Schulman, Ethan Perez, Fabien Roger, and Jan Leike. Quantifying elicitation of latent capabilities in language models. Advances in Neural Information Processing Systems, pp. 150412–150443, 2025.DOI
- [13]Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736–740. IEEE, 2020.arxiv.org/abs/1910.09387
- [14]Xuelong Geng, Kun Wei, Qijie Shao, Shuiyun Liu, Zhennan Lin, Zhixian Zhao, Guojian Li, Wenjie Tian, Peikun Chen, Yangze Li, et al. Osum: Advancing open speech understanding models with limited resources in academia. arXiv preprint arXiv:2501.13306, 2025.arxiv.org/abs/2501.13306
- [15]Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335–346, 1990.DOI
- [16]Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. In International Conference on Learning Representations, volume 2026, pp. 52423–52443, 2026.arxiv.org/abs/2502.04326
- [17]Wen Huang, Yunfei Chu, Meng Gao, Haolin He, and Jin Xu. Longaudiospan: Spanning the duration and depth of audio comprehension. arXiv preprint arXiv:2608.26431, 2026.arxiv.org/abs/2608.26431
- [18]Jeff Hwang, Moto Hira, Caroline Chen, Xiaohui Zhang, Zhaoheng Ni, Guangzhi Sun, Pingchuan Ma, Ruizhe Huang, Vineel Pratap, Yuekai Zhang, et al. Torchaudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for pytorch. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1–9. IEEE, 2023.
- [19]Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohi Dibua, Yifan Gong, Yan Kang, and Dimitris N Metaxas. Dtop-p moe: Sparsity-controlled dynamic top-p moe for foundation model pre-training. arXiv preprint arXiv:2512.13996, 2025.arxiv.org/abs/2512.13996
- [20]Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. Speechcraft: A fine-grained expressive speech dataset with natural language description. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1255–1264, 2024.DOI
- [21]Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 119–132, 2019.DOI
- [22]Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024.arxiv.org/abs/2402.01831
- [23]Chun-Yi Kuan and Hung-yi Lee. Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2025.arxiv.org/abs/2410.16130
- [24]Sonal Kumar, Prem Seetharaman, Ke Chen, Oriol Nieto, Jiaqi Su, Zhepei Wang, Rithesh Kumar, Dinesh Manocha, Nicholas J Bryan, Zeyu Jin, and Justin Salamon. Tac: Timestamped audio captioning. arXiv preprint arXiv:2602.15766, 2026.arxiv.org/abs/2602.15766
- [25]Steven R. Livingstone and Frank A. Russo. The Ryerson audio-visual database of emotional speech and song (RAVDESS). PLoS ONE, 13(5):e0196391, 2018.
- [26]Ziyang Ma, Yinghao Ma, Yangqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. MMAR: A challenging benchmark for deep reasoning in speech, audio, music, and their mix. Advances in Neural Information Processing Systems, 38, 2025.arxiv.org/abs/2505.13032
- [28]Tu Anh Nguyen, Wei-Ning Hsu, Antony d’Aviro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05725, 2023.arxiv.org/abs/2308.05725
- [29]Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. Speech Communication, 45(1):89–95, 2005.DOI
- [30]Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pp. 527–536, 2019.
- [31]Sakshi Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. In International Conference on Learning Representations, volume 2025, pp. 84929–84964, 2025.
- [32]Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-asr technical report. arXiv preprint arXiv:2601.21337, 2026.arxiv.org/abs/2601.21337
- [33]Xingwei Sun, Heinrich Dinkel, Gang Li, Jiahao Mei, Yadong Niu, Zerui Han, Yuepeng Jiang, Jiahao Zhou, Lichun Fan, and Jian Luan. Midashenglm-gen: Unified audio scene generation via llm-driven autoregressive flow matching. arXiv preprint arXiv:2608.11804, 2026.
- [34]Kim Sung-Bin, Oh Hyun-Bin, Lee Jung-Mok, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Ahvbench: A cross-modal hallucination benchmark for audio-visual large language models. In International Conference on Learning Representations, volume 2025, pp. 24244–24271, 2025.
- [35]Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In International Conference on Learning Representations, volume 2024, pp. 16607–16629, 2024.
- [36]Apoorv Vyas, Heng-Jui Chang, Cheng-Fu Yang, Po-Yao Huang, Luya Gao, Julius Richter, Sanyuan Chen, Matthew Le, Piotr Dollár, Christoph Feichtenhofer, et al. Pushing the frontier of audio-visual perception with large-scale multimodal correspondence learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 30172–30182, 2026.
- [37]Chen Wang, Minpeng Liao, Zhongqiang Huang, Junhong Wu, Chengqing Zong, and Jiajun Zhang. Blsp-emo: Towards empathetic large speech-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19186–19199, 2024.arxiv.org/abs/2406.03872
- [38]Dingdong Wang, Junan Li, Jincenzi Wu, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, and Helen Meng. Mmsu: A massive multi-task spoken language understanding and reasoning benchmark. In International Conference on Learning Representations, volume 2026, pp. 31374–31410, 2026a.
- [39]Dingdong Wang, Shujie Liu, Tianhua Zhang, Youjun Chen, Jinyu Li, and Helen Meng. Emotion-thinker: Prosody-aware reinforcement learning for explainable speech emotion reasoning. ICLR, 2026b.
- [40]Shuai Wang, Zhaokai Sun, Zhennan Lin, Chengyou Wang, Zhou Pan, and Lei Xie. Msu-bench: Towards understanding the conversational multi-talker scenarios. Interspeech, 2026c.






