WorksComputer Science
CoTTAGE: An Explainable Judge for Text-to-Audio Generation
Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
Introduction
Reliable evaluation is fundamental to the development of text-to-audio (TTA) generation systems. As TTA models continue to improve, effective evaluation becomes increasingly important for guiding model selection and accelerating iterative development [40, 23, 37]. Among existing evaluation approaches, human assessment remains the gold standard, where annotators typically evaluate generated audio from two perspectives: relevance to
the text instruction (REL) and overall audio quality (OVL) [22, 14, 13] (Vyas et al. 2023; Majumder et al. 2024). While accurate, human evaluation is labor-intensive and difficult to scale, making it impractical for the rapid iteration cycles of TTA development. This motivates the development of automatic judges that can provide scalable evaluations while maintaining alignment with human perception.
Existing automatic evaluation methods remain insufficient for this purpose. For REL, CLAPScore [15] measures text-audio alignment using the cosine similarity between pretrained audio-language embeddings, but its correlation with human judgments remains limited [19]. To address this gap, RELATE [19] and XACLE [27] introduce human-annotated REL datasets, enabling the development of dedicated REL evaluators. In contrast, OVL lacks comparable training resources, as publicly available OVL annotations are limited to evaluation-scale datasets ([6, 20]).
Beyond accuracy, existing evaluators also provide limited explainability. Most methods produce only a scalar score ([15, 6, 30]), offering limited insight for diagnosing model failures and guiding model improvement. While proprietary frontier audio-language models (e.g., Gemini-3.1 Pro) can generate text explanations, their closed-source nature, high API costs, and privacy constraints render them impractical as scalable, reproducible evaluation benchmarks. In contrast, fine-grained natural-language feedback has been shown to facilitate both error analysis and model improvement in the text domain ([36]). This motivates the need for TTA judges that not only align with human perception but also provide natural-language rationales grounded in the audio.
We introduce CoTTAGE, a Chain-of-Thought Text-to-Audio Generation Evaluator. Instead of directly predicting a score, CoTTAGE first generates a natural-language rationale grounded in the acoustic evidence before assessing either REL or OVL, as illustrated in Figure 1. To enable OVL training, we collect human annotations for over 10K audio clips from the RELATE and XACLE training sets, complementing their REL annotations. CoTTAGE is trained in two stages: we first perform supervised fine-tuning (SFT) on human-MOS-guided reasoning traces, and then optimize the model via reinforcement learning (RL) using a Thurstone-based pairwise preference reward ([33]) to incorporate prediction uncertainty.
Figure 1. An example of text-audio relevance evaluation with different models. CLAPScore predicts only a scalar score, while Qwen2.5-Omni often hallucinates reasoning. In contrast, our CoTTAGE model provides an audio-grounded explanation highlighting a temporal-order mismatch.
Our contributions are three-fold:
We propose CoTTAGE, a reasoning-based TTA judge that generates audio-grounded rationales for REL and OVL assessment, enabling one model to assess both dimensions. CoTTAGE achieves the highest correlation with human judgments across all benchmarks and produces human-preferred rationales.
We collect and release human MOS annotations for OVL on over 10K audio clips from RELATE and XACLE, closing the training data gap for audio quality evaluation.
We show that the proposed Thurstone-based pairwise preference optimization complements pointwise supervised fine-tuning, yielding consistent gains across diverse in-domain and out-of-domain benchmarks.
Related Work
Text-to-Audio Evaluation
Text-to-audio (TTA) systems have advanced rapidly in recent years. Recent models achieve high-fidelity audio synthesis from free-form text while improving controllability, efficiency, and generation quality ([25, 24, 10, 12] (Yang et al. 2025; Zheng et al. 2026; Hung et al. 2026)).
As TTA systems become increasingly capable, reliable evaluation has emerged as a critical challenge. PAM ([6]) proposes a reference-free metric based on CLAP ([42]) to conduct audio quality assessment. AudioBERTScore ([20]) measures embedding similarity between synthesized and reference audio using pretrained audio encoders such as Audio Spectrogram Transformers ([9]). ELSA ([30]) combines global text-audio similarity with event-level alignment via language-queried source separation. RELATE ([19]) and XACLE ([27]) advance this line of research by introducing large-scale human-annotated datasets and benchmarks for training and evaluating automatic REL predictors. Beyond REL and OVL, TTA-Bench ([37]) has further broadened the evaluation landscape by incorporating dimensions such as fairness and generation efficiency. Unlike previous methods, CoTTAGE jointly evaluates text relevance and overall audio quality while providing natural-language rationales grounded in the audio.
Reasoning-based Audio Judges
Large audio language models (LALMs), typically comprising an audio encoder ([28]) and a large language model (LLM) backbone, have recently demonstrated remarkable performance across a wide range of audio understanding tasks ([31, 43, 8, 32, 26]). Their strong audio understanding capability has motivated growing interest in using LALMs as automatic judges of audio quality.
Early attempts have primarily focused on speech evaluation. QualiSpeech ([39]) first collects human-written justifications and demonstrates that LALMs can be fine-tuned for reasoning-based speech quality assessment. Subsequent work extends this direction by exploring instructionaware evaluation of speaking style [3], improving alignment with human judgments through LLM distillation and preference optimization [1], and multilingual evaluation with richer task formulations, including pairwise comparison and quality improvement suggestions [38]. These studies demonstrate that LALMs can serve as effective and scalable judges for speech quality.
OVL Annotation
Data Collection
While RELATE [19] and XACLE [27] provide training-scale human annotations for REL, publicly available human OVL labels on synthesized audio remain limited to small, evaluation-oriented test sets [6, 20]. The closest existing resource is the annotation used to train Audiobox-Aesthetics [34], which contains approximately 97K clips of audio, each rated by three human annotators. Although this corpus is closely related, several key differences limit its applicability to TTA evaluation. First, it has not been publicly released. Second, its annotations target a different objective: rather than assigning a single OVL score, annotators rated four aesthetic dimensions, Production Quality (PQ), Production Complexity (PC), Content Enjoyment (CE), and Content Usefulness (CU), which the authors show correlate only moderately with OVL on synthesized sound [34]. Third, it consists exclusively of real-world recordings, introducing a domain mismatch with synthesized audio.
We close this gap by collecting human OVL annotations for 10,362 training clips, including 2,862 clips from RELATE and 7,500 clips from XACLE. We recruit annotators through Amazon Mechanical Turk, an online participant recruitment platform, and collect five independent ratings for each clip. Details of the annotation protocol, quality assurance procedures, and worker payment are provided in Appendix § 12.
Analysis
To further justify the need for dedicated OVL annotations, we evaluate how well existing signals predict human OVL ratings. We consider human REL annotations and the four aesthetic dimensions predicted by the pretrained Audiobox-Aesthetics model [34], since the original human ratings used to train the model are not publicly available.
Figure Figure 2 reports the Pearson (LCC) and Spearman (SRCC) correlations between each signal and the mean human OVL rating. The two correlation coefficients closely agree, indicating that the observed trends are not driven by nonlinearity or outliers.
Figure 2. Pearson (LCC) and Spearman (SRCC) correlations between human OVL ratings and existing evaluation dimensions, including REL and the four Audiobox-Aesthetics dimensions. Error bars indicate bootstrap 95% confidence intervals.
REL shows only weak correlations with OVL on both RELATE and XACLE. A clip that faithfully matches its prompt is not necessarily perceived as high-quality audio, and vice versa. The relatively weak correlations indicate that text relevance and overall quality reflect different aspects of human perception. The four aesthetic dimensions show similarly weak correlations with OVL. Content Enjoyment is the strongest correlate with OVL, but its correlation is only comparable to that with REL. Production Complexity is essentially uncorrelated with OVL. Production Quality, despite being nominally the closest concept to audio quality, reaches only a moderate correlation with OVL on XACLE and performs substantially worse on RELATE. These results are also consistent with the moderate correlations between individual aesthetic dimensions and OVL reported in [34].
Overall, no existing signal correlates with human OVL beyond LCC = 0.34. Neither REL annotations nor existing aesthetic predictors provide an adequate substitute for dedicated OVL annotations.
Method
Figure Figure 3 illustrates the overall training pipeline of CoTTAGE. Our training pipeline consists of two stages: (1) supervised fine-tuning (SFT) to establish foundational reasoning and scoring capabilities (§ 4.1), and (2) reinforcement learning (RL) with an uncertainty-aware reward to further enhance the model (§ 4.2).

Figure 3. The overall training pipeline of CoTTAGE.
Supervised Fine-Tuning
SFT establishes the model’s basic reasoning and prediction capabilities.
Reasoning Data Construction. Existing general-purpose audio preference datasets provide limited supervision for reasoning-based judgment. To obtain reasoning supervision, we generate a reasoning trace for each audio clip while preserving its human-annotated MOS score. Specifically, we prompt Gemini-3.1 Pro with the audio and the ground-truth MOS score. For REL, we additionally provide the corresponding text instruction, since the score depends on text-audio alignment. For OVL, following the standard OVL annotation protocol (Vyas et al., 2023), no text instruction is provided because the evaluation concerns only the perceptual quality of the generated audio. The model then generates a rationale explaining how the score is determined by assessing REL and OVL. Providing the ground-truth score as part of the prompt encourages the generated rationale to remain consistent with the human annotation. The instruction we use is shown in Appendix § 11.
Training. Using this dataset, we first perform SFT to initialize the model with reasoning and scoring abilities. During training, the model is required to generate a rationale before predicting the final score. This encourages the model to justify its decision through an explicit and human-readable reasoning instead of directly regressing to a score. After the SFT stage, the model acquires basic reasoning and human preference prediction abilities.
Reinforcement Learning
Supervised fine-tuning provides only pointwise supervision, where each training sample is associated with a single MOS target. While effective for learning an initial evaluator, this formulation offers relatively sparse training signals. To further improve the judge, we optimize it with reinforcement learning based on pairwise comparisons. This allows each sample to contribute supervision through comparisons with other samples in the batch.
Thurstone Model. We adopt the grouped sampling mechanism of GRPO (Shao et al., [29]), where the policy generates reasoning trajectories for each audio input . Each trajectory produces a predicted score , and the resulting set of scores induces a predictive distribution over each sample, from which we estimate the mean and variance . The variance reflects the model’s inconsistency across reasoning trajectories, with larger values indicating higher uncertainty in the estimated quality score.
Inspired by prior work on image quality assessment (Zhang et al., 2021; Chen et al., [2]; You et al., 2025; Wu et al., [41]), we adopt the Thurstone model Thurstone, [33] to enable pairwise preference optimization. It converts uncertainty-aware score distributions into pairwise preference probabilities. These probabilities are then used as the reward signal for reinforcement learning. Specifically, the probability that audio is preferred over is defined as:
where is the standard normal cumulative distribution function and is a stability constant that prevents the denominator from vanishing. A larger combined variance naturally yields softer preference probabilities closer to 0.5, resulting in more conservative rewards for uncertain comparisons.
Mapping MOS to Preferences. The ground-truth preference is obtained by binarizing the human-annotated MOS into a comparative target:
Preference Reward. Given a batch of samples, the reward for sample is averaged over all pairwise comparisons with the other samples:
where is the Bhattacharyya coefficient between the predicted and target Bernoulli preferences, maximized exactly when .
Format Reward. During exploration, RL-tuned language models often suffer from format collapse, emitting unstructured text that cannot be parsed [29]. To enforce the desired reasoning structure, we add a rule-based format reward . The model earns 1 only if its output encloses the reasoning trace in <think>...</think> and the final score in <score>...</score>, and 0 otherwise:
Overall Reward. The final reward for the -th sampled response combines the two terms:
Experiments
Experimental Setup
Datasets. We train our judge model using the training sets of RELATE [19] and XACLE [27]. RELATE contains 2,862 training audio clips spanning both real recordings and TTA-generated samples, while XACLE provides an additional 7,500 training clips. Both datasets come with REL scores measuring the semantic alignment between each audio clip and its corresponding text instruction. We additionally collect OVL annotations for both datasets, as described in § 3.
We conduct evaluation on the official test sets of RELATE and XACLE to measure in-domain performance. To assess generalization, we further evaluate our model on the out-of-domain datasets PAM [6] and Clotho [7, 20], which differ from our training data in both annotator pools and data collection protocols.
Baselines. We compare our method against embedding-based baselines and general-purpose LALMs. The embedding-based baselines include CLAPScore [15] and ELSA [30]. We implement CLAPScore using the pretrained LAION-CLAP model [42]. The evaluated LALMs include Gemini-3.1 Pro, Audio Flamingo 3 [8], Qwen2.5-Omni-7B [43], and Qwen3-Omni-30B [43], where Gemini-3.1 Pro is also used to generate the reasoning labels during the SFT stage. For all LALMs, we prompt the models to first generate their natural-language reasoning before producing a final score.
We further compare against Audiobox-Aesthetics [34], a specialized audio quality assessment model. As discussed in § 3.1, it predicts four aesthetic dimensions (PQ, PC, CE, and CU) rather than a single OVL score. We therefore report each of the dimensions as a separate baseline for OVL prediction.
Implementation Details. We initialize our proposed model from Qwen2.5-Omni-7B [43], and train it with LoRA [11] for parameter-efficient fine-tuning. We set the LoRA rank to and the LoRA scaling factor to . During the SFT stage, the model is trained for 5 epochs with a learning rate of . For the RL stage, we set the learning rate to , the KL divergence coefficient to 0.04, and sample candidate responses per input. The overall reward is a weighted sum of the preference reward and the format reward with equal weight. All prompts and sweep parameters used in our experiments are provided in Appendix § 11. We conduct each experiment with three random seeds and report the median results in the following sections. Since the REL annotations from RELATE and XACLE are on a 0-10 scale, we linearly map them to a 1-5 scale to align with other benchmarks.
Main Results
Table 1 and Table 2 report the Pearson linear correlation coefficient (LCC) and Spearman rank correlation coefficient (SRCC) between model predictions and human ratings on REL and OVL, where CoTTAGE consistently achieves the highest LCC and SRCC across all benchmarks.
| XACLE | RELATE | PAM-Audio | Clotho | |||||
| LCC ↑ | SRCC ↑ | LCC ↑ | SRCC ↑ | LCC ↑ | SRCC ↑ | LCC ↑ | SRCC ↑ | |
| Embedding-based Methods | ||||||||
| CLAPScore | 0.472 | 0.398 | 0.437 | 0.411 | 0.493 | 0.488 | 0.396 | 0.378 |
| ELSA | 0.398 | 0.371 | 0.375 | 0.377 | 0.497 | 0.467 | 0.416 | 0.404 |
| Proprietary LALMs | ||||||||
| Gemini-3.1 Pro | 0.551 | 0.489 | 0.471 | 0.423 | 0.473 | 0.442 | 0.548 | 0.528 |
| Open-Source LALMs | ||||||||
| Audio Flamingo 3 | 0.344 | 0.300 | 0.215 | 0.193 | 0.297 | 0.230 | 0.376 | 0.339 |
| Qwen2.5-Omni-7B | 0.162 | 0.153 | 0.063 | 0.047 | 0.319 | 0.305 | 0.334 | 0.301 |
| Qwen3-Omni-30B | 0.399 | 0.374 | 0.256 | 0.256 | 0.373 | 0.353 | 0.438 | 0.444 |
| CoTTAGE (ours) | 0.685 | 0.652 | 0.594 | 0.591 | 0.657 | 0.663 | 0.607 | 0.602 |
| w/o RL | 0.601 | 0.572 | 0.519 | 0.516 | 0.606 | 0.590 | 0.536 | 0.531 |
Table 1. Correlation metrics (LCC and SRCC) between model predictions and human ratings on REL.
| PAM-Audio | Clotho | |||
| LCC ↑ | SRCC ↑ | LCC ↑ | SRCC ↑ | |
| Embedding-based Methods | ||||
| CLAPScore | 0.379 | 0.373 | 0.201 | 0.223 |
| PAM | 0.587 | 0.610 | 0.445 | 0.451 |
| ELSA | 0.387 | 0.363 | 0.305 | 0.283 |
| Regression-based Predictors | ||||
| Aesthetics-PQ | 0.355 | 0.363 | 0.465 | 0.450 |
| Aesthetics-PC | 0.092 | 0.127 | -0.148 | -0.074 |
| Aesthetics-CE | 0.464 | 0.502 | 0.317 | 0.340 |
| Aesthetics-CU | 0.396 | 0.405 | 0.487 | 0.471 |
| Proprietary LALMs | ||||
| Gemini-3.1 Pro | 0.404 | 0.473 | 0.512 | 0.502 |
| Open-Source LALMs | ||||
| Audio Flamingo 3 | 0.326 | 0.332 | 0.218 | 0.225 |
| Qwen2.5-Omni-7B | 0.221 | 0.206 | 0.271 | 0.329 |
| Qwen3-Omni-30B | 0.429 | 0.424 | 0.252 | 0.227 |
| CoTTAGE (ours) | 0.611 | 0.614 | 0.559 | 0.558 |
| w/o RL | 0.529 | 0.525 | 0.481 | 0.470 |
Table 2. Correlation metrics (LCC and SRCC) between model predictions and human ratings on OVL.
Are pretrained audio-text embeddings enough for TTA evaluation? CLAPScore and ELSA evaluate TTA outputs through the similarity of pretrained audio-text embeddings. While both achieve moderate correlation with human judgments, they consistently lag behind CoTTAGE. On XACLE, our model achieves an LCC of 0.685, compared with 0.472 for CLAPScore and 0.398 for ELSA, with similar gaps observed across the remaining benchmarks. For OVL, CLAPScore and ELSA consistently underperform CoTTAGE. While PAM performs competitively on the PAM-Audio benchmark, CoTTAGE achieves a slight improvement over PAM on PAM-Audio and demonstrates a larger advantage on Clotho, leading to the strongest overall performance across OVL benchmarks. These results suggest that pretrained audio-text embedding similarity provides a strong baseline, but reasoning-based evaluation offers more consistent alignment with human judgments across diverse TTA evaluation settings.
Can general-purpose LALMs serve as reliable judges? We investigate whether general-purpose LALMs can serve as TTA evaluators without task-specific training. Our results show that open-source LALMs remain unreliable. Qwen2.5-Omni, the backbone from which CoTTAGE is initialized, achieves an LCC of only 0.063 on RELATE, and even the stronger Qwen3-Omni performs on par with or below embedding-based metrics on most benchmarks. This indicates that strong audio understanding alone does not necessarily translate into reliable quality assessment. Overall, Gemini-3.1 Pro is the strongest baseline on REL, ranking first in three of the four benchmarks, and also demonstrates strong performance on OVL benchmarks. However, CoTTAGE consistently outperforms it across all benchmarks, improving the LCC from 0.551 to 0.685 on XACLE (REL) and from 0.404 to 0.611 on PAM-Audio (OVL).
Do Aesthetic ratings transfer to OVL prediction? We examine whether Audiobox-Aesthetics can serve as a substitute for a reliable OVL judge. Despite being trained on nearly ten times more annotated audio than CoTTAGE, none of its four aesthetic dimensions correlates strongly with human OVL ratings. Besides, the best-performing dimension varies across benchmarks. For example, CE achieves the highest correlation on PAM-Audio, whereas CU performs best on Clotho, suggesting that no single aesthetic axis consistently reflects the perceptual quality of synthesized audio. In contrast, CoTTAGE outperforms all four dimensions on PAM-Audio and Clotho. These findings support our analysis in § 3: the mismatch in the prediction target and the training domain limits the applicability of aesthetic ratings to TTA quality evaluation, highlighting the need for collected OVL annotations.
Does reinforcement learning help beyond SFT? Finally, comparing our full model with its SFT variant (w/o RL) shows that reinforcement learning consistently improves both LCC and SRCC across all REL benchmarks and OVL benchmarks. This demonstrates that the preference-based reward provides complementary supervision beyond supervised fine-tuning alone. Notably, the SFT variant already achieves strong performance in most cases, demonstrating that supervised fine-tuning provides a strong initialization for reasoning. Reinforcement learning further improves upon this initialization, yielding performance beyond what supervised training alone can achieve.
Explainable Chain-of-Thought
Human Evaluation
To assess how well the reasoning traces generated by CoTTAGE align with the audio content and human perception, we conduct a human preference evaluation. We randomly sample 50 examples from the PAM test set and collect reasoning traces for REL and OVL from Qwen2.5-Omni-7B, our SFT variant, Gemini-3.1 Pro, and CoTTAGE. Three human annotators independently perform blinded pairwise comparisons between CoTTAGE and each baseline, with the presentation order randomized. For each comparison, annotators select the reasoning trace that better reflects the audio content and
human perception, or indicate a tie when neither is preferred. We report the win rate of CoTTAGE (excluding ties), as well as the breakdown of wins, ties, and losses.
Results in Figure 4 show that annotators generally prefer CoTTAGE over the baselines when ties are excluded. The largest gains are observed against Qwen2.5-Omni-7B, with win rates of 75.7% on REL and 63.4% on OVL. Compared with the SFT variant, CoTTAGE achieves win rates of 55.1% on REL and 57.8% on OVL, suggesting that RL further improves the quality of reasoning traces beyond SFT. The comparison with Gemini-3.1 Pro is particularly competitive, as it is a stronger proprietary model that provides high-quality reasoning traces for distillation and performs well on holistic judgments. Despite this, CoTTAGE further improves REL, which requires fine-grained grounding between reasoning traces and audio content.

Figure 4. Results of human pairwise evaluation of generated scoring rationales. CoTTAGE is compared against Qwen2.5-Omni-7B, the SFT-only variant, and the proprietary model Gemini-3.1 Pro.
Qualitative Analysis
Table 3 demonstrates examples of rationales generated by CoTTAGE and Qwen2.5-Omni.
| Type | Instruction | Qwen2.5-Omni | CoTTAGE |
| REL Temporal | “A loud motor begins to pick up speed then cuts out and slowly stops.” Human-annotated MOS: 2.10 | The audio describes a motor that starts loudly, then slows down and eventually stops. The text also describes a motor that begins to pick up speed, then cuts out and slowly stops. The key elements of the motor’s behavior (starting loudly, slowing down, and stopping) are consistent between the audio and the text. Model predicted MOS: 4.50 | [...] Listening to the audio, there is a continuous, loud humming or buzzing sound that resembles a motor or drone. However, the audio remains at a constant, steady pitch and intensity from the beginning to the end. [...] there is no sense of the motor “picking up speed,” nor is there a sudden “cutting out” or stopping sound. [...] Therefore, the alignment between the text and the audio is very poor. Model predicted MOS: 1.80 |
| REL Counting | “A bug chirps once and then starts chirping repetitively.” Human-annotated MOS: 2.00 | The audio contains a continuous sound of a bug chirping, which aligns with the description of a bug chirping repetitively. The description does not mention the bug chirping once before starting to chirp repetitively. [...] Model predicted MOS: 4.50 | [...] The audio, however, consists of a continuous, repetitive bug chirping from the very beginning. It completely misses the initial “chirp once” action and plays the repetitive sound throughout its duration. [...] Model predicted MOS: 1.80 |
| OVL Distortion | “Rapid gunfire with male yelling.” Human-annotated MOS: 1.90 | [...] The gunshots are clear and distinct, indicating a high-quality sound effect. The man’s speech is also clear and understandable, [...] The overall audio quality is consistent with a real-world recording. Model predicted MOS: 4.50 | The audio clip suffers from severe quality degradations throughout its duration. The prominent sounds, which resemble loud bursts or synthetic impacts, are heavily distorted and clipped, resulting in a harsh, abrasive, and unnatural tone. [...] Model predicted MOS: 2.00 |
| OVL Artifacts | “A jackhammer drilling and vibrating continuously.” Human-annotated MOS: 2.60 | The audio clip contains a sound that resembles a vehicle engine idling, which is a common and recognizable sound in real-world recordings. The sound is clear and consistent, without any noticeable background noise or unnatural patterns. Therefore, the audio quality is excellent. Model predicted MOS: 5.00 | [...] The most prominent issue with this recording is the significant amount of background noise and static. A persistent hiss and low-frequency rumble are present throughout the entire clip, which masks the finer details of the mechanical sound. Additionally, the audio feels somewhat muffled and lacks high-frequency clarity, [...] Model predicted MOS: 2.60 |
Table 3. Qualitative comparison of natural-language rationales for REL and OVL. Green indicates claims supported by the audio, while red marks unsupported claims. Rationales are abbreviated. Full outputs are provided in Appendix § 13.
Reasoning on REL. For REL, a consistent failure mode emerges: Qwen2.5-Omni generates rationales that are primarily driven by caption rather than audio. Instead of verifying the described events against what it hears, Qwen2.5-Omni tends to restate the caption and then asserts agreement.
In the Temporal example, the audio is actually a steady, unchanging drone. Qwen2.5-Omni instead describes a motor that starts loudly, slows down, and eventually stops, matching the caption rather than the audio, and therefore predicts 4.50 for a clip with a human MOS of 2.1. In contrast, CoTTAGE identifies the continuous humming, notes that neineither the acceleration nor the final stop occurs, and predicts 1.80. The Counting example shows the same pattern. Although Qwen2.5-Omni recognizes the repetitive chirping, it incorrectly claims that the caption omits the initial isolated chirp and predicts 4.50 for a clip rated 2.0 by annotators. CoTTAGE instead points out that the repetitive chirping begins immediately and that the isolated chirp never occurs, producing a prediction of 1.80.
Reasoning on OVL. The OVL examples reveal a similar pattern. Qwen2.5-Omni frequently produces generic positive rationales such as “clear and distinct” and “without any noticeable background noise or unnatural sound patterns”, assigning high scores even to clips containing obvious synthesis artifacts. In contrast, CoTTAGE identifies the distorted and clipped impulsive sounds throughout the clip and predicts 2.00.
In the Artifacts example, Qwen2.5-Omni mistakes the jackhammer for an idling engine and still describes the recording as clean, leading to a prediction of 5.00 for a clip rated 2.6 by human annotators. CoTTAGE instead identifies the persistent hiss, low-frequency rumble, and muffled timbre, producing a prediction of 2.60.
Overall, these examples demonstrate that CoTTAGE achieves more faithful audio understanding, leading to more accurate and reliable TTA judgments. When Qwen2.5-Omni produces unsupported rationales, its predicted scores are often inaccurate as well. In contrast, CoTTAGE produces audio-grounded rationales that improve explainability by revealing the evidence behind each prediction, enabling easier diagnosis of TTA model failures and providing insights for improving training data and model design.
Conclusion
We present CoTTAGE, a reasoning-based evaluator that jointly assesses text relevance (REL) and overall quality (OVL) for text-to-audio generation. Following the reason-then-predict paradigm, CoTTAGE generates a natural-language rationale grounded in acoustic evidence before predicting each score. To enable OVL learning, we collect human-annotated OVL labels for over 10K clips from RELATE and XACLE, allowing a single model to assess both dimensions. Building on these resources, we train CoTTAGE via SFT on distilled reasoning traces, followed by RL with a Thurstone-based pairwise preference reward. Across six benchmarks spanning REL and OVL, CoTTAGE achieves the highest correlation with human judgments among all baselines, and its generated rationales are preferred by humans over those of the base model. We hope that our model and annotations will support further progress in this direction.
Limitations
While CoTTAGE provides an explainable evaluator aligned with human perception, we outline its key limitations below.
First, CoTTAGE relies on human MOS supervision from RELATE and XACLE. Their audio domains, caption styles, and annotator pools may not fully cover broader TTA scenarios, such as music-centric synthesis, speech-centric generation, or non-English instructions. While CoTTAGE demonstrates strong transferability to out-of-domain benchmarks such as PAM and Clotho, extending supervision to more diverse audio domains remains an important open direction.
Second, current evaluation benchmarks primarily consist of short audio clips (less than 30 seconds). Evaluating long-form audio generation (e.g., multi-minute soundscapes or extended compositions) requires tracking long-range acoustic dynamics and structural continuity over time, which is not fully addressed by short-clip evaluators.
Third, while CoTTAGE generates natural-language rationales to diagnose acoustic flaws and event ordering, it does not output exact temporal timestamps to pinpoint when specific acoustic events or artifacts occur. Incorporating explicit temporal grounding represents a promising step toward fine-grained error localization.
Ethics Statement
The human OVL annotations in this work were collected via Amazon Mechanical Turk with fair worker compensation aligned with standard crowdsourcing practices. The task involves rating environmental sound clips only, posing minimal risk to participants. Workers were informed that their anonymized ratings would be released for academic research, and no personally identifiable information was collected or stored.
Regarding broader impact, CoTTAGE provides scalable and reproducible feedback to guide TTA model development. However, directly optimizing a generative model solely against an automatic metric may lead to metric over-fitting. Therefore, as with any automatic metric, CoTTAGE should be used to complement rather than fully replace periodic human evaluation.
We do not identify any specific harm associated with our model or the collected annotations.
CoTTAGE is intended to complement rather than replace human evaluation, and the annotation task poses minimal risk to participants, from whom no personal information is collected.
Acknowledgments
This work was financially supported by Amazon AGI and the Ministry of Education (MOE) of Taiwan under the project Taiwan Centers of Excellence in Artificial Intelligence, through the NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE). We thank the National Center for High-performance Computing (NCHC) of the National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources. We also thank Chen-An Li from National Taiwan University for helpful discussions and feedback.
Disclosure of AI Usage
We used ChatGPT and Claude to polish the manuscript. The use of these AI tools did not influence the substantive content, data analysis, or scientific conclusions of the study and served exclusively as a writing aid. We have carefully checked the AI refinements to ensure that all contents are correct.
Implementation Details
Hyperparameter search. We sweep the learning rate separately for SFT and RL. For SFT, we consider the learning rates
while for RL, we sweep over
For both stages, we sweep the LoRA rank over with . We additionally sweep the RL sampling temperature over .
Prompts. We provide the full prompts used in Figure 5-Figure 8. Specifically, Figure 5 and Figure 6 show the evaluation prompts for REL and OVL, respectively, where the model is instructed to reason step by step before producing a final score between 1.00 and 5.00. To construct reasoning-augmented training data, we further design the prompts shown in Figure 7 and Figure 8, which condition the model on the ground-truth score and ask it to generate a reasoning trace that naturally leads to this score without explicitly revealing that the score is given. All prompts require the reasoning to be enclosed in <think></think> tags and the final score in <score></score> tags, rounded to two decimal places.

Figure 5. The prompt used for text relevance evaluation.

Figure 6. The prompt used for audio quality assessment.

Reasoning Data Construction Prompt - REL
You are an expert text-to-audio judge. Your job is to evaluate whether an audio faithfully represents the given text instruction.
Listen to the audio and compare it with the given text instruction. Carefully analyze how faithfully the audio conveys the content described in the instruction.
The ground-truth semantic alignment score has already been determined and must not be changed. Your task is to write a reasoning process that is consistent with this score. The reasoning should read as if you first analyzed the semantic alignment between the instruction and the audio, weighing supporting and conflicting evidence before arriving at a conclusion. Do not explicitly mention that the score is given or explain what the score represents. Instead, let the evidence naturally lead to the final judgment.
The semantic alignment score is on a scale from 1.00 to 5.00, where 1.00 means the audio does not match the instruction at all and 5.00 means the audio perfectly and faithfully represents the instruction.
Put your reasoning inside <think></think> and output the final score inside <score></score> rounded to two decimal places.
Text instruction: {instruction}
Ground-truth semantic alignment score: {gt_score}
Figure 7. The prompt used to construct reasoning-augmented training data about REL.

Figure 8. The prompt used to construct reasoning-augmented training data about OVL.
Human Annotation Details
We collect OVL annotations via Amazon Mechanical Turk (MTurk)3 for all audio clips in the training sets of RELATE and XACLE (2,862 and 7,500 clips, respectively; 10,362 in total). Each HIT contains 11 audio clips: 10 clips to be annotated and one anchor clip used for quality control, which is randomly drawn from an anchor pool and inserted at a random position so that workers cannot identify it. Each HIT is completed independently by five distinct workers. To ensure annotation quality, we set the following requirements for annotators:
Annotators must be MTurk Masters, which ensures high-quality workers.
Annotators must have an approval rate higher than 98%.
Annotators must have more than 10,000 approved tasks.
Annotators must be located in the United States to ensure familiarity with English.
Workers rate each clip on an integer scale from 1 to 5, where 1 denotes completely unnatural audio composed of background noise and unnatural sound patterns that does not resemble a real-world recording, and 5 denotes a completely natural, high-quality real-world recording. Following the standard OVL annotation protocol (Vyas et al., 2023), the interface presents only the audio without the corresponding text instruction, and workers are explicitly instructed to judge how the audio sounds rather than what it depicts. The instructions further recommend wearing headphones and inform workers in advance that each HIT contains a hidden quality-control clip and that failing this check may lead to rejection of the assignment. The final OVL score of each clip is the average of the five worker ratings. The full worker instructions, scoring rubric, and annotation interface are shown in Figure 9.

Figure 9. The MTurk annotation interface for OVL collection. For the annotation page, only the first three of the 11 clips are shown. We adopt the scoring rubric from Vyas et al. (2023).
Workers are paid $0.40 per HIT. To ensure that the payment meets minimum wage standards. The annotation task involves only rating the quality of machine-generated environmental audio and poses minimal risk to participants. We do not collect any personal information from workers, and worker identifiers are never released.
Annotation Interface. The annotation process consists of two stages. Upon accepting a HIT, workers first see an instruction page containing the task description, the full scoring rubric for the 1–5 scale, and the quality-control notice. After reading the instructions, workers click a “Start Evaluation” button to proceed to the annotation page. The annotation page lists the 11 audio clips in order, each accompanied by an audio player and five rating buttons corresponding to scores 1–5. To prevent workers from forgetting the scoring criteria during annotation, the interface provides a “View Criteria” button that reopens the rubric in a pop-up window at any time. A sticky progress bar at the bottom of the page displays the number of completed ratings in real time. The interface further requires that every clip be played at least once and rated before the assignment can be submitted, which prevents workers from rating clips without listening to them. Figure 9(a) and (Figure 9b) show the instruction page and the annotation page, respectively.
Quality Control. We employ three complementary quality-control mechanisms. First, we perform anchor-based checking. We construct an anchor pool consisting of clips with clearly high or clearly low audio quality, each of which is verified through internal listening tests to ensure that its reference score is unambiguous before being added to the pool. An assignment is rejected if the rating on its anchor clip deviates from the reference score by 2 or more points. Rejected assignments are republished and re-annotated by other workers, ensuring that every clip eventually receives five valid ratings. Second, we manage worker access through MTurk qualifications. Workers with consistently strong performance across batches, i.e., no rejections, high anchor accuracy, and high correlation with the consensus ratings, are granted a trusted qualification and notified early when new batches are published, whereas workers with poor records are assigned an exclusion qualification that blocks them from participating in subsequent batches. Third, we conduct distribution-based worker monitoring. On a 5-point scale, a low-effort strategy of compressing all ratings toward the middle of the scale largely evades the anchor-based check, since the resulting anchor errors rarely exceed the rejection threshold. To detect this behavior, we additionally monitor per-worker rating distributions (the exact-match rate on anchors and the standard deviation of ratings on real clips); workers exhibiting abnormal distributions are manually reviewed and, if confirmed, excluded from subsequent batches via the exclusion qualification.
Inter-Annotator Agreement. We report the inter-annotator agreement by ICC(1, 1), ICC(1, k) and Krippendorff’s . We also report the agreement scores of the existing REL annotations for reference.
As shown in Table 4, the ICC(1,) values for RELATE-OVL and XACLE-OVL are 0.699 and 0.704, indicating moderate-to-good reliability under standard guidelines [4, 21]. While the ICC(1, 1) and Krippendorff’s values are lower than those reported for the original REL annotations, this difference is expected. REL is grounded in a textual prompt and primarily evaluates semantic relevance. In contrast, OVL requires subjective judgments of overall audio quality and is therefore more susceptible to individual preferences and other confounding factors [5, 16]. Nevertheless, the ICC(1, ) values indicate that aggregating ratings across multiple annotators provides sufficiently reliable supervision for training.
| Dataset | Annotation | ICC(1,1) | ICC(1,) | |
| RELATE | REL | 0.449 | 0.762 | 0.446 |
| OVL | 0.317 | 0.699 | 0.317 | |
| XACLE | REL | 0.435 | 0.755 | 0.435 |
| OVL | 0.322 | 0.704 | 0.322 |
Table 4. Inter-annotator agreement of collected OVL annotations, compared with the original REL annotations. We report ICC(1,1), ICC(1,), and Krippendorff’s .
Dataset Statistics. Figure 10 shows the distributions of the mean human REL and OVL ratings on RELATE and XACLE. On both datasets, REL is heavily skewed toward the maximum rating of 5, while OVL is approximately unimodal and centered around 3. Many clips therefore receive near-perfect relevance ratings yet only average quality ratings, indicating that the two dimensions capture different aspects of human perception.
Figure 10. Distributions of the mean human REL and OVL ratings on RELATE and XACLE.
Qualitative Results
We provide qualitative comparisons of the natural-language rationales generated by Qwen2.5-Omni and CoTTAGE for REL and OVL in Table 5 and Table 6, respectively.
| Type | Instruction | Qwen2.5-Omni | CoTTAge |
| Temporal | “A loud motor begins to pick up speed then cuts out and slowly stops.” Human-annotated MOS: 2.10 | The audio describes a motor that starts loudly, then slows down and eventually stops. The text also describes a motor that begins to pick up speed, then cuts out and slowly stops. The key elements of the motor’s behavior (starting loudly, slowing down, and stopping) are consistent between the audio and the text. Model predicted MOS: 4.50 | The text instruction describes a sequence of events: a loud motor beginning to pick up speed, then cutting out, and finally slowly stopping. Listening to the audio, there is a continuous, loud humming or buzzing sound that resembles a motor or drone. However, the audio remains at a constant, steady pitch and intensity from the beginning to the end. It completely lacks the dynamic changes requested in the prompt: there is no sense of the motor “picking up speed,” nor is there a sudden “cutting out” or stopping sound. While the audio provides a basic motor-like noise, it fails to capture any of the specific actions or the temporal progression detailed in the instruction. Therefore, the alignment between the text and the audio is very poor. Model predicted MOS: 1.80 |
| Counting | “A bug chirps once and then starts chirping repetitively.” Human-annotated MOS: 2.00 | The audio contains a continuous sound of a bug chirping, which aligns with the description of a bug chirping repetitively. The description does not mention the bug chirping once before starting to chirp repetitively, but the continuous chirping is consistent with the description of repetitive chirping. Therefore, the audio aligns well with the text description. Model predicted MOS: 4.50 | The text instruction specifies a bug chirping once initially, followed by repetitive chirping. The audio, however, consists of a continuous, repetitive bug chirping from the very beginning. It completely misses the initial “chirp once” action and plays the repetitive sound throughout its duration. Because the temporal structure and the distinct events described in the prompt are absent, the audio fails to faithfully represent the instruction. Model predicted MOS: 1.80 |
| Multi-event | “Ducks quack with blowing wind and faint splashes.” Human-annotated MOS: 2.80 | The audio contains the sound of ducks quacking, which aligns with the text description. The presence of wind and faint splashes also suggests an outdoor setting, which is consistent with the idea of ducks in a natural environment. The audio captures the essence of the described scene, making the semantic alignment quite strong. Model predicted MOS: 4.50 | The text instruction specifies three distinct elements: ducks quacking, blowing wind, and faint splashes. Listening to the audio, the sound of ducks quacking is very prominent and clearly recognizable throughout the clip. However, the other elements described in the prompt are noticeably absent. There is no audible sound of blowing wind, nor are there any faint splashes to be heard. Because the audio successfully captures the primary subject (the ducks quacking) but completely misses the background environmental sounds (the wind and splashes), it only partially fulfills the text instruction. This results in a moderate alignment score. Model predicted MOS: 2.80 |
Table 5. Full natural-language rationales for REL, corresponding to Table 3, with an additional Multi-event example. Green indicates claims supported by the audio, while red marks unsupported claims.
| Type | Instruction | Qwen2.5-Omni | CoTTAGE | |
| OVL | Distortion | “Rapid gunfire with male yelling.”<br>Human-annotated MOS: 1.90 | The audio clip contains a series of gunshots followed by a man speaking. The gunshots are clear and distinct, indicating a high-quality sound effect. The man’s speech is also clear and understandable, suggesting good audio quality. There is no background noise or unnatural sound patterns present. The overall audio quality is consistent with a real-world recording.<br>Model predicted MOS: 4.50 | The audio clip suffers from severe quality degradations throughout its duration. The prominent sounds, which resemble loud bursts or synthetic impacts, are heavily distorted and clipped, resulting in a harsh, abrasive, and unnatural tone. This distortion significantly degrades the clarity and dynamic range of the recording. Additionally, there is a pervasive layer of background noise and static that further muddies the audio. The human voices heard towards the end are muffled, lack definition, and are difficult to discern clearly against the noisy background. The combination of intense clipping, high noise levels, and overall lack of fidelity makes the listening experience quite poor. These substantial recording artifacts and the absence of any clean audio elements strongly indicate a very low quality rating.<br>Model predicted MOS: 2.00 |
| OVL | Artifacts | “A jackhammer drilling and vibrating continuously.”<br>Human-annotated MOS: 2.60 | The audio clip contains a sound that resembles a vehicle engine idling, which is a common and recognizable sound in real-world recordings. The sound is clear and consistent, without any noticeable background noise or unnatural patterns. The quality of the audio is high, as it accurately represents the sound of an engine idling. Therefore, the audio quality is excellent.<br>Model predicted MOS: 5.00 | The audio presents a continuous mechanical sound, resembling a small engine or motor running. The most prominent issue with this recording is the significant amount of background noise and static. A persistent hiss and low-frequency rumble are present throughout the entire clip, which masks the finer details of the mechanical sound. Additionally, the audio feels somewhat muffled and lacks high-frequency clarity, giving it a slightly distant or enclosed characteristic. While the primary sound source remains identifiable, the heavy presence of noise and the overall lack of crispness severely degrade the listening experience, resulting in a below-average quality rating.<br>Model predicted MOS: 2.60 |
| OVL | High quality | “Baby crying then man speaking and humming.”<br>Human-annotated MOS: 4.20 | The audio clip contains a baby crying, which is a natural sound. However, the quality of the audio seems to be low, with some background noise and possibly some unnatural sound patterns. The audio does not sound like a high-quality real-world recording.<br>Model predicted MOS: 2.50 | The audio presents a baby crying, which is the primary sound source. The crying itself is captured with good clarity and sounds very natural, without any noticeable distortion, clipping, or digital artifacts. The vocalizations are distinct and easy to hear. However, there is a minor amount of background noise or room ambience present throughout the recording. While this background noise is not overly intrusive or distracting, it prevents the audio from sounding completely pristine or studio-quality. Overall, the recording is clean and the main subject is well-represented, but the slight background hiss slightly diminishes the overall fidelity.<br>Model predicted MOS: 4.00 |
Table 6. Full natural-language rationales for OVL, corresponding to Table 3, with an additional High quality example. Green indicates claims supported by the audio, while red marks unsupported claims.
References
- [1]Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhe-huai Chen, Chao Zhang, Chao-Han Huck Yang, and En-Siong Chng. 2025a. Audio large language models can be descriptive speech quality evaluators. In International Conference on Learning Representations, volume 2025, pages 24920–24934.arxiv.org/abs/2501.17202
- [2]Du Chen, Tianhe Wu, Kede Ma, and Lei Zhang. 2025b. Toward generalized image quality assessment: Relaxing the perfect reference quality assumption. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12742–12752.arxiv.org/abs/2503.11221
- [3]Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhen-dong Wang, Zhenyuan Yang, Hung-yi Lee, and Li-juan Wang. 2025. Audio-aware large language models as judges for speaking styles. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 467–480, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2506.05984
- [4]Domenic V Cicchetti. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment, 6(4):284.DOI
- [5]Erica Cooper and Junichi Yamagishi. 2023. Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speech. In Interspeech 2023, pages 1104–1108.arxiv.org/abs/2305.10608
- [6]Soham Deshmukh, Dareen Alharthi, Benjamin Elizalde, Hannes Gamper, Mahmoud Al Ismail, Rita Singh, Bhiksha Raj, and Huaming Wang. 2024. PAM: Prompting Audio-Language Models for Audio Quality Assessment. In Interspeech 2024, pages 3320–3324.arxiv.org/abs/2402.00282
- [7]Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740. IEEE.arxiv.org/abs/1910.09387
- [8]Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. In Advances in Neural Information Processing Systems, volume 38, pages 41819–41886. Curran Associates, Inc.arxiv.org/abs/2507.08128
- [9]Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. In Interspeech 2021, pages 571–575.
- [10]Gaëtan Hadjeres, Marc Ferras, Khaled Koutini, Benno Weck, Alexandre Bittar, Thomas Hummel, Zineb Lahrichi, Hakim Missoum, Joan Serrà, and Yuki Mitsufuji. 2026. Woosh: A sound effects foundation model. arXiv preprint arXiv:2604.01929.arxiv.org/abs/2604.01929
- [11]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuantai Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
- [12]Kuan-Po Huang, Bo-Ru Lu, Ho-Lam Chung, Shih-Hsin Wang, and Hung-yi Lee. 2026a. Fdaudio: Meanflow-anchored Fréchet-distance post-training for one-step text-to-audio generation. arXiv preprint arXiv:2607.10421.arxiv.org/abs/2607.10421
- [13]Kuan-Po Huang, Bo-Ru Lu, Byeonggeun Kim, Mihee Lee, Zalan Fabian, Renard Korżenowski, Qingming Tang, Greg Ver Steeg, Hung-yi Lee, Chieh-Chi Kao, and Chao Wang. 2026b. Fast text-to-audio generation with one-step sampling via energy-scoring and auxiliary contextual representation distillation. Preprint, arXiv:2605.00329.arxiv.org/abs/2605.00329
- [14]Kuan-Po Huang, Shu-wen Yang, Huy Phan, Bo-Ru Lu, Byeonggeun Kim, Sashank Macha, Qingming Tang, Shalini Ghosh, Hung-yi Lee, Chieh-Chi Kao, and Chao Wang. 2025. Impact: Iterative mask-based parallel decoding for text-to-audio generation with diffusion modeling. In International Conference on Machine Learning, pages 26002–26019. PMLR.arxiv.org/abs/2506.00736
- [15]Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023. Make-an-audio: Text-to-audio generation without prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916–13932. PMLR.arxiv.org/abs/2301.12661
- [16]Wen-Chin Huang, Erica Cooper, and Tomoki Toda. 2026c. Mos-bench: Benchmarking generalization abilities of subjective speech quality assessment models. IEEE Transactions on Audio, Speech and Language Processing, 34:2385–2397.DOI
- [19]Yusuke Kanamori, Yuki Okamoto, Taisei Takano, Shinnosuke Takamichi, Yuki Saito, and Hiroshi Saruwatari. 2025. Relate: Subjective evaluation dataset for automatic evaluation of relevance between text and audio. In Proc. Interspeech 2025, pages 3155–3159.arxiv.org/abs/2506.23582
- [20]Minoru Kishi, Ryosuke Sakai, Shinnosuke Takamichi, Yusuke Kanamori, and Yuki Okamoto. 2026. Audiobertscore: Objective evaluation of environmental sound synthesis based on similarity of audio embedding sequences. In Proceedings of the AAAI 2026 Workshop on Audio-Centric AI: Towards Real-World Multimodal Reasoning and Application Use Cases (Audio-AAAI), volume 312 of Proceedings of Machine Learning Research, pages 21–37. PMLR.arxiv.org/abs/2507.00475
- [21]Terry K Koo and Mae Y Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2):155–163.DOI
- [22]Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yanniv Taigman, and Yossi Adi. 2023. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2209.15352
- [23]Chun-Yi Kuan, Siwon Kim, Byeonggeun Kim, Suyoun Kim, Bo-Ru Lu, Qingming Tang, Ankur Gandhi, Hung-yi Lee, Chieh-Chi Kao, and Chao Wang. 2026. Improving text-to-audio instruction following via fine-grained feedback from audio-aware large language models. arXiv preprint arXiv:2607.13408.arxiv.org/abs/2607.13408
- [24]Xiquan Li, Junxi Liu, Wenxi Chen, Haina Zhu, Ziyang Ma, and Xie Chen. 2026. Resonate: Reinforcing text-to-audio generation via online feedback from large audio language models. arXiv preprint arXiv:2603.11661.arxiv.org/abs/2603.11661
- [25]Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. 2024. Audiodlm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2871–2883.
- [26]Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Hsueh Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuan-jun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, and 9 others. 2026. Desta2.5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment. IEEE Transactions on Audio, Speech and Language Processing, 34:2062–2076. Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. 2024. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 564–572, New York, NY, USA. Association for Computing Machinery.
- [27]Yuki Okamoto, Riki Takizawa, Minoru Kishi, Yusuke Kanamori, Noriyuki Tonami, Ryotaro Nagase, Shinnosuke Takamichi, and Keisuke Imoto. 2026. Xacle challenge 2026: The first x-to-audio alignment challenge. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21877–21879. IEEE.DOI
- [28]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeay, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.arxiv.org/abs/2212.04356
- [29]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.arxiv.org/abs/2402.03300
- [30]Shuntaro Suzuki, Kento Tokura, Daichi Yashima, Kanon Amemiya, Komei Sugiura, and Shinnosuke Takamichi. 2026. Elsa: Acoustic event-level semantic alignment for fine-grained reference-free text-to-audio evaluation. arXiv preprint arXiv:2606.17404.
- [31]Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Salmmon: Towards generating hearing abilities for large language models. In International Conference on Learning Representations, volume 2024, pages 16607–16629.
- [32]Core Team, Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhui Ren, Shuo Liu, Tao Guo, Weiji Zhuang, Xin Zhang, Xingchen Song, Yihan Yan, Yongzhe He, Cici, Bowen Shen, Chengxuan Zhu, Chong Ma, and 81 others. 2025. Mimo-audio: Audio language models are few-shot learners. Preprint, arXiv:2512.23808.
- [33]Louis L Thurstone. 1994. A law of comparative judgment. Psychological review, 101(2):266.
- [34]Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, Carleigh Wood, Ann Lee, and Wei-Ning Hsu. 2025. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. Preprint, arXiv:2502.05139.
- [36]Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2026a. Text2grad: Reinforcement learning from natural language feedback. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2505.22338
- [37]Hui Wang, Cheng Liu, Junyang Chen, Haoze Liu, Yuhang Jia, Shiwan Zhao, Jiaming Zhou, Haoqin Sun, Hui Bu, and Yong Qin. 2026b. Tta-bench: A comprehensive benchmark for evaluating text-to-audio models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33512–33520.
- [38]Hui Wang, Jinghua Zhao, Yifan Yang, Shujie Liu, Junyang Chen, Yanzhe Zhang, Shiwan Zhao, Jinyu Li, Jiaming Zhou, Haoqin Sun, Yan Lu, and Yong Qin. 2025a. Speechllm-as-judges: Towards general and interpretable speech quality evaluation. arXiv preprint arXiv:2510.14664.arxiv.org/abs/2510.14664
- [39]Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Yu Tsao, Junichi Yamagishi, Yuxuan Wang, and Chao Zhang. 2025b. Qualispeech: A speech quality assessment dataset with natural language reasoning and descriptions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23588–23609.
- [40]Zehan Wang, Ke Lei, Chen Zhu, Jiawei Huang, Sashuai Zhou, Luping Liu, Xize Cheng, Shengpeng Ji, Zhenhui Ye, Tao Jin, and Zhou Zhao. 2025c. T2A-feedback: Improving basic capabilities of text-to-audio generation via fine-grained AI feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23535–23547, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2505.10561
- [41]Tianhe Wu, Jian Zhou, Jie Liang, Lei Zhang, and Kede Ma. 2025. Visualquality-r1: Reasoning-induced image quality assessment via reinforcement learning to rank. Advances in Neural Information Processing Systems, 38:88167–88190.arxiv.org/abs/2505.14460
- [42]Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP.
- [43]Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-omni technical report. Preprint, arXiv:2503.20215.arxiv.org/abs/2503.20215


