betaFeedback

WorksComputer Science

A Systematic Study of Cross-Token Transfer for Data-Efficient Non-Verbal Text-to-Speech

Abstract

Non-verbal vocalizations (NVs), such as laughter and breath, carry affective cues that words alone do not convey. Yet NV data remains scarce because speech annotation pipelines often omit or normalize these events. Existing work has improved NV text-to-speech through model design and data curation, but whether one NV token can help a model learn another has received little systematic study. We investigate this cross-token transferability in a controlled CosyVoice 2 setup. We first trace a laughter-only scaling curve, then hold laughter at 200 or 1,600 examples while adding breath, sigh, cough, lip-smack, or neutral speech. With only 200 laughter examples, adding 3,200 cough examples raises end-to-end laughter F1 from 0.170 to 0.529, surpassing the 0.495 obtained from 1,600 laughter examples alone. At the 1,600-example anchor, cough reaches 0.695, nearly matching 6,400 laughter examples (0.697). Waveform-level Whisper-D scoring and a semantic-token transcriber show consistent trends. Cross-token transfer therefore offers a complementary route to low-resource NV generation.

Introduction

Spoken communication conveys more than words. Non-verbal vocalizations (NVs) such as laughter, breaths, sighs, cries, and coughs reveal speaker state, affect, and conversational intent [1, 2]. Laughter can signal warmth or empathy; filled pauses and breaths expose hesitation or control, while other bursts add rhythm and emphasis. Words establish content, but NVs shape how an utterance is understood, making explicit control important for expressive and socially natural TTS [3].

Unfortunately, the cues captured by a microphone often disappear during data preparation: conventional transcripts preserve lexical content while omitting NVs or collapsing them into inconsistent labels. Rare events such as crying receive even fewer reliable annotations, leaving far less usable supervision than ordinary speech.

Recent work has expanded NV datasets and improved controllable generation, as reviewed in § 2. One complementary question remains open: can a well-resourced NV token improve a different, low-resource token during TTS training? If cross-token transfer exists, abundant donor events could provide reusable acoustic structure, reduce the amount of target-specific annotation needed, and make rare NV capabilities cheaper to add. Knowing which donors transfer would also help data curation prioritize useful examples rather than simply collecting more speech.

We study this question through controlled scaling and transfer experiments with laughter as the target event. We first vary the amount of laughter data to establish its learning curve. We then fix the laughter budget and add breath, sigh, cough, or lip-smack examples, with neutral speech as a control. All conditions use the same CosyVoice 2 architecture and fine-tuning recipe, so observed differences can be attributed to training-data composition.

Our contributions are:

  • a controlled study of laughter scaling and cross-token transfer under a fixed TTS architecture;

  • evidence that 3,200 cough examples raise laughter F1 from 0.170 to 0.529 when only 200 laughter examples are available, outperforming an eightfold increase in target-only data; and

  • complementary evaluation with Whisper-D on generated waveforms and a learned transcriber that reads semantic tokens before waveform rendering.

Related Work

Data creation and NV-aware annotation. NV resources differ in how labels are obtained and how precisely events are aligned. Early event-aware ASR decoded words and acoustic social signals jointly, establishing a unified transcript representation [4, 5]. Recent pipelines turn recognition into a data-construction tool: NVSpeech bootstraps NV-aware annotation at scale, while WESR evaluates whether an event is both recognized and placed correctly within a transcript [6, 7]. Curated corpora offer finer supervision: NonverbalTTS aligns English NV and emotion labels, whereas SMIP-NV and MNV-17 provide multi-level Mandarin annotations [8, 9, 10]. NonVerbalSpeech-38K and SynParaSpeech trade some manual curation for scale through automatic annotation or generation [11, 12]. Beyond Words compares curriculum learning, data balancing, voice conversion, and auxiliary-token transfer for rare-event ASR [13]; NV-Bench and NVV-SuperBench expand synthesis evaluation across event types and quality dimensions [14, 15].

Modeling. EmoCtrl-TTS introduces detector-derived conditioning and time-varying emotion embeddings to control laughter and crying within a flow-matching TTS model [3]. Recent preference optimization instead uses an NV-aware recognition score to improve generation over 18 NV types [16]. Both directions improve how a system represents or optimizes NV behavior; expanding and cleaning training corpora improves the evidence available to it. Neither, however, isolates whether examples of one NV token help a scarce second token. Our study is therefore complementary: we hold the architecture and optimization recipe fixed, vary only target–donor data composition, and measure the effect both before and after waveform rendering.

Methodology

Framework

We build on the CosyVoice 2 generation pipeline [17], shown in Figure 1. The input text contains lexical tokens and explicit NV markers such as [laughter]. A prompt utterance supplies speaker and acoustic context. CosyVoice 2 converts the prompt into discrete semantic tokens and conditions an autoregressive semantic language model (LM) on both the prompt tokens and input text. The LM predicts the target semantic-token sequence1 μ=(μ1,…,μL)\boldsymbol{\mu} = (\mu_{1}, \ldots, \mu_{L}) with

Study overview showing CosyVoice generation, cross-token experiments, and complementary evaluation

Figure 1. Study overview: (a) CosyVoice generation, (b) controlled target scaling and cross-token transfer, and (c) complementary waveform and semantic-token evaluation.

Lsem=−∑l=1Llog⁡p(μl∣μ<l,tNV,p),(1)\mathcal{L}_{\mathrm{sem}} = - \sum_{l=1}^{L} \log p(\mu_{l} \mid\mu_{<l}, \mathbf{t}_{\mathrm{NV}}, \mathbf{p}), \tag*{(1)}

where tNV\mathbf{t}_{\mathrm{NV}} is the text sequence with NV markers and p\mathbf{p} denotes the prompt representation. A conditional flow decoder and vocoder render the predicted semantic tokens as a waveform. In every experiment, we fine-tune only the semantic LM and reuse the same tokenizer, flow decoder, and vocoder.

Case Studies

Collecting a large corpus for every affective burst is costly. Phonetic and physiological studies suggest that laughter, crying, and coughs draw on related respiratory and laryngeal mechanisms [1, 2]. Laughter, for example, consists of rhythmic glottal pulses superimposed on forced exhalation [18]. Although these events remain perceptually distinct, their shared airflow, voicing, and temporal structure creates a plausible basis for transfer.

Recent TTS evidence supports this possibility. EmoCtrl-TTS showed that embeddings from a laughter detector can support fine-grained, controllable crying synthesis [3]. This cross-event reuse motivates our data-level question: can examples labeled with an abundant donor token help the same semantic LM learn a scarce target token?

We test two controlled cases. The target scaling study trains on nested subsets containing 0, 200, 400, 800, 1,600, 3,200, or 6,400 laughter examples, establishing low-resource and near-saturation operating points. The cross-token transfer study fixes laughter at 200 or 1,600 examples and adds approximately 3,200 examples of breath, sigh, cough, or lip-smack. A neutral-speech mixture tests whether arbitrary additional audio is sufficient. At the 200-example anchor, cough ranges from 0 to 3,200 examples. All else remains fixed.

Evaluation

We measure whether the model generates laughter at the requested location using two complementary paths.

Semantic-token transcriber. We train a separate autoregressive transcriber that maps the semantic-token sequence μ\boldsymbol{\mu} to an NV-aware transcript y=(y1,…,yM)\mathbf{y} = (y_{1}, \ldots, y_{M}) containing words and inline NV markers. Its training objective mirrors Eq. (1):

Ltr=−∑m=1Mlog⁡p(ym∣y<m,μ).(2)\mathcal{L}_{\mathrm{tr}} = - \sum_{m=1}^{M} \log p(y_{m} \mid y_{<m}, \boldsymbol{\mu}). \tag*{(2)}

This model is used only for evaluation, not in the TTS generation path.

The waveform path renders the semantic tokens with the frozen flow decoder and vocoder, then applies Whisper-D{^2} [19] to the waveform. It therefore scores the complete synthesis pipeline. The token path applies our transcriber directly to the LM output, bypassing waveform rendering and its potential errors. Both paths produce an NV-aware transcript. We align each output with the reference prompt in text space and compute event-level precision, recall, and F1. Agreement between the two paths indicates that an effect originates in semantic-token generation and persists after waveform rendering; a gap helps localize errors to the acoustic back end. Crucially, this study measures laughter event retrieval in TTS rather than perceptual audio quality. Evaluating laughter retrieval under this setup serves as a controlled proxy to validate whether cross-event knowledge transfer successfully translates into predictable NV generation.

Experiments

Experimental Setup

Data and backbone. We use Libri-Light [20] as NV-neutral pre-training data and an internal emotional TTS corpus as the source of NV subsets. The backbone is CosyVoice 2-0.5B [17], and each run uses eight H100 GPUs. To reduce native NV priors, we reinitialize the autoregressive TTS LM and pre-train it on approximately 27k hours of neutral Libri-Light speech. Following CosyVoice 2, we initialize the backbone from Qwen2-0.5B [21], randomly initialize the speech-token layers and NV embeddings, and freeze the FSQ tokenizer, flow decoder, and vocoder. Pre-training uses AdamW with weight decay 0.01, attention dropout 0.1, gradient clipping at 1.0, and a cosine learning-rate schedule from 5×10−55 \times10^{-5} to 10−610^{-6} with 20k warm-up steps. The epoch-20 checkpoint obtains 4.7% WER on 50 speaker-disjoint LibriTTS test-clean utterances and produces no NVs on the evaluation prompts.

Fine-tuning sets. We normalize transcripts to eight canonical NV tags; examples include laughter, breath, sigh, cough, and lip-smack. Data budgets count tagged NV occurrences; the figures refer to each occurrence as one example. Unambiguous variants map to a canonical tag; unsupported or ambiguous annotations are removed. Pure-laughter utterances contain laughter and no other NV token. Donor utterances contain the selected donor event and no other NV token. The neutral control adds 3,000 Libri-Light utterances. All pools exclude a deterministic 1% speaker-disjoint development split. We fine-tune for 10 epochs with AdamW, learning rate 10−510^{-5}, weight decay 0.01, (β1,β2)=(0.9,0.95)(\beta_1,\beta_2) = (0.9, 0.95), ϵ=10−8\epsilon= 10^{-8}, attention dropout 0.1, and gradient clipping at 1.0. The learning rate decays to 10−610^{-6} after a 5% warm-up, and the step budget scales with set size so that every example is seen for 10 epochs.

Transcriber training. Our evaluator uses the released CosyVoice 2 LM with a Qwen2-0.5B backbone. We train it on the internal emotional corpus and English Libri-Light for 100k steps using AdamW, weight decay 0.01, attention dropout 0.1, and a learning rate that warms up for 5k steps to 10−410^{-4} and decays to 10−510^{-5}. The checkpoint reaches 4.88% WER on LibriTTS test-clean.

Evaluation set. Each checkpoint is evaluated on the same 400 prompts, including 100 laughter-positive prompts. We score generated waveforms with Whisper-D and LM outputs with the semantic-token transcriber. On oracle audio and oracle semantic tokens from this set, the two evaluators obtain laughter-retrieval F1 scores of 0.712 and 0.813, respectively. Inference uses deterministic F32 computation.

Results

Target-Only Scaling

Figure 2 establishes the laughter-only baseline. F1 rises from 0.170 at 200 examples to 0.495 at 1,600 and 0.667 at 3,200. Doubling the data once more yields only 0.030 absolute improvement, reaching 0.697 at 6,400. The curve places 200 examples in a data-starved regime and 1,600 near saturation.

Figure 2. Waveform-level laughter scaling. Performance rises quickly through 3,200 target examples and changes little from 3,200 to 6,400.

Cross-Token Transfer

Tables 1 and 2 compare donor tokens at the two laughter anchors. With 200 laughter examples, every NV donor improves on the 0.170 waveform F1 baseline, while neutral speech reduces F1 to 0.032. Cough performs best at 0.529, a 3.1× relative improvement that also exceeds the 0.495 achieved with 1,600 laughter examples alone. At the 1,600-example anchor, cough and lip-smack each reach 0.695, nearly matching the 0.697 result from 6,400 target examples. The neutral control confirms that arbitrary speech is insufficient.

DonorWaveform PWaveform RWaveform F1Semantic token PSemantic token RSemantic token F1
None.215.140.170.333.241.280
Breath.721.310.434.746.312.440
Sigh.628.270.378.831.348.490
Cough.622.460.529.731.482.581
Lip-smack.630.290.397.857.340.487
Neutral.083.020.032.250.050.083

Table 1. Cross-token transfer at the 200-laughter anchor. Each donor contributes 3,200 NV examples.

DonorWaveform PWaveform RWaveform F1Semantic token PSemantic token RSemantic token F1
None.481.510.495.678.582.626
Breath.759.630.689.859.560.678
Sigh.707.580.637.865.589.700
Cough.733.660.695.894.596.715
Lip-smack.747.650.695.914.603.726
Neutral.538.500.518.741.567.643

Table 2. Cross-token transfer at the 1,600-laughter anchor. Each donor contributes 3,200 NV examples.

The largest change appears in precision. At the 200-example anchor, cough raises waveform precision from 0.215 to 0.622 and recall from 0.140 to 0.460. The token evaluator shows the same pattern: precision rises from 0.333 to 0.731, while recall rises from 0.241 to 0.482. Donor NVs therefore help the semantic LM suppress spurious laughter while recovering more requested events.

Figure 3 varies the cough donor budget. The overall trend is positive but not strictly monotonic: waveform F1 changes from 0.245 at 200 cough examples to 0.233 at 400, then reaches 0.529 at 3,200. The token path follows the same broad pattern and reaches 0.568, although it generally gives a higher absolute score. The agreement localizes the effect to semantic-token generation and leaves a decoder gap.

Cough-donor scaling plots for waveform and semantic-token evaluators

Figure 3. Cough-donor scaling with 200 laughter examples. Small multiples separate the waveform and semantic-token evaluators so precision, recall, and F1 remain legible.

The spectrograms in Figure 4 illustrate the same difference. For the prompt “... ask for a weather update? [laughter] ...”, the target-only model produces diffuse, groan-like aspiration in the labeled interval. Adding cough data yields a sequence of clearer bursts. This pattern suggests shared respiratory and glottal dynamics, pending listening tests.

Spectrogram for 1,600 laughter examples

(a) 1,600 laughter examples

Spectrogram for 1,600 laughter plus 3,200 cough examples

(b) 1,600 laughter + 3,200 cough examples

Figure 4. Spectrograms for the same prompt. The red boxes mark the requested laughter interval. The target-only model produces diffuse aspiration, whereas the cough-augmented model produces structured bursts between 2.4 and 3.8 s.

Conclusion

We studied cross-token transfer while holding the TTS architecture fixed. Laughter-only performance showed diminishing returns beyond 3,200 examples. With 200 laughter examples, adding 3,200 cough examples increased waveform F1 from 0.170 to 0.529, exceeding 1,600 laughter examples alone. Waveform and semantic-token evaluation agreed on the main trends, placing the gain before waveform rendering and showing that it largely survives the acoustic decoder. We study one target and one backbone and measure event retrieval rather than perceptual quality; broader NV pairs and listening tests remain important.

References

References

  1. [1]Robert R Provine, Laughter: A scientific investigation, Penguin, 2000.
  2. [2]Klaus R Scherer, “Vocal communication of emotion: A review of research paradigms,” Speech communication, vol. 40, no. 1-2, pp. 227–256, 2003.DOI
  3. [3]Haibin Wu, Xiaofei Wang, Sefik Emre Eskimez, et al., “Laugh now cry later: Controlling time-varying emotional states of flow-matching-based zero-shot text-to-speech,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 690–697.arxiv.org/abs/2407.12229
  4. [4]Hirofumi Inaguma, Masato Mimura, Koji Inoue, Kazuyoshi Yoshii, and Tatsuya Kawahara, “An end-to-end approach to joint social signal detection and automatic speech recognition,” in Proc. IEEE ICASSP, 2018, pp. 5199–5203.
  5. [5]Nagisa Shione and Tatsuya Kawahara, “Automatic speech recognition using linguistic and verbal/nonverbal information,” IEEE Access, vol. 11, pp. 138985–138995, 2023.DOI
  6. [6]Huan Liao, Qinke Ni, Yuancheng Wang, Yiheng Lu, Haoyue Zhan, Pengyuan Xie, Qiang Zhang, and Zhizheng Wu, “Nvspeech: An integrated and scalable pipeline for human-like speech modeling with paralinguistic vocalizations,” arXiv preprint arXiv:2508.04195, 2025.arxiv.org/abs/2508.04195
  7. [7]Chenchen Yang, Kexin Huang, Liwei Fan, Qian Tu, Botian Jiang, Dong Zhang, Linqi Yin, Shimin Li, Zhaoye Fei, Qinyuan Cheng, and Xipeng Qiu, “WESR: Scaling and evaluating word-level event-speech recognition,” arXiv preprint arXiv:2601.04508, 2026.arxiv.org/abs/2601.04508
  8. [8]Maksim Borisov, Egor Spirin, and Daria Diatlova, “Non-verbaltts: A public english corpus of text-aligned non-verbal vocalizations with emotion annotations for text-to-speech,” arXiv preprint arXiv:2507.13155, 2025.arxiv.org/abs/2507.13155
  9. [9]Zhuojun Wu, Dong Liu, Juan Liu, et al., “SMIIP-NV: A multi-annotation non-verbal expressive speech corpus in Mandarin for llm-based speech synthesis,” in Proceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 12564–12570.
  10. [10]Jialong Mai, Jinxin Ji, Xiaofen Xing, et al., “MNV-17: A high-quality performative Mandarin dataset for nonverbal vocalization recognition in speech,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026, pp. 18312–18316.arxiv.org/abs/2509.18196
  11. [11]Runchuan Ye, Yixuan Zhou, Renjie Yu, Zijian Lin, Ke-han Li, Xiang Li, Xin Liu, Guoyang Zeng, and Zhiy-ong Wu, “A scalable pipeline for enabling non-verbal speech generation and understanding,” arXiv preprint arXiv:2508.05385, 2025.arxiv.org/abs/2508.05385
  12. [12]Bingsong Bai, Qihang Lu, Wenbing Yang, Zihan Sun, Yueran Hou, Peilei Jia, Songbai Pu, Ruibo Fu, Yingming Gao, Ya Li, and Jun Gao, “Synparasppeech: Automated synthesis of paralinguistic datasets for speech generation and understanding,” arXiv preprint arXiv:2509.14946, 2025.arxiv.org/abs/2509.14946
  13. [13]Gene Yang, Haibin Wu, Peng Su, Ruizhe Huang, Suwon Shon, Bach Do, Minxue Niu, Zhaoheng Ni, Shang-Wen Li, Florian Metze, Yossi Adi, Ming Sun, and Yuzong Liu, “Beyond words: Towards effective modeling of non-verbal vocalizations in asr,” arXiv preprint arXiv:2607.01563, 2026.arxiv.org/abs/2607.01563
  14. [14]Qinke Ni, Huan Liao, Dekun Chen, Yuxiang Wang, and Zhizheng Wu, “NV-Bench: Benchmark of nonverbal vocalization synthesis for expressive text-to-speech generation,” arXiv preprint arXiv:2603.15352, 2026.arxiv.org/abs/2603.15352
  15. [15]Liumeng Xue, Weizhen Bian, Jiahao Pan, Wenxuan Wu, Yilin Ren, Boyi Kang, Jingbin Hu, Ziyang Ma, Shuai Wang, Xinyuan Qian, Hung-yi Lee, and Yike Guo, “NVV-SuperBench: Beyond words, beyond quality—benchmarking nonverbal vocalizations in speech generation,” arXiv preprint arXiv:2604.16211, 2026.arxiv.org/abs/2604.16211
  16. [16]Haoyang Li, Chenglin Xu, Junchuan Zhao, Yuang Cao, Liumeng Xue, Yiwen Guo, and Eng Siong Chng, “Preference optimization for non-verbal vocalization synthesis,” arXiv preprint arXiv:2608.24163, 2026.arxiv.org/abs/2608.24163
  17. [17]Zhihao Du, Yuxuan Wang, Qian Chen, et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024.arxiv.org/abs/2412.10117
  18. [18]Jürgen Trouvain, “Laughing, breathing, clicking-the prosody of nonverbal vocalizations,” in Proceedings of the 7th international conference on Speech Prosody, 2014, pp. 598–602.DOI
  19. [19]Jordan Darefsky, Ge Zhu, and Zhiyao Duan, “Parakeet,” 2024.
  20. [20]Jacob Kahn, Morgane Riviere, Weiyi Zheng, et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2020, pp. 7669–7673.
  21. [21]An Yang, Baosong Yang, Binyuan Hui, et al., “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024.arxiv.org/abs/2407.10671

Paper details

vlnc:W_wt29tgan
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents