betaFeedback

WorksComputer Science

Constructing Dynamic Conversations with Neural Back-channel Generation and Voice-line Scheduling

Abstract

We present a novel approach to the problem of combining two voice-line channels of dyadic conversational audio into a unified conversation with controllable back-channeling and turn-taking dynamics. In voice performances for the game industry, the two tracks of voice-lines are typically recorded or synthesized in isolation. Our method takes two such unsynchronized tracks and uses a neural scheduler to sequence them into a single dynamic conversation, modifying both tracks by inserting or removing pauses and introducing voice overlap as appropriate. We further propose a back-channel predictor that is able to predict opportunity for back-channels for a given conversational turn's audio and content, which the scheduler can then incorporate. We evaluate our scheduler and predictor independently and in combination by: various quantitative comparison; a qualitative and perceptual comparison to prior art and naive baselines; a compelling set of results showing animator control over the diversity, extent and nature of back-channeling and turn dynamics.

CCS Concepts

• Computing methodologies → Animation; Neural networks.

Keywords

speech, audio-driven animation

ACM Reference Format:

Yifang Pan, Chris Landreth, and Karan Singh. 2026. Constructing Dynamic Conversations with Neural Back-channel Generation and Voice-line Scheduling. In The 19th ACM SIGGRAPH Conference on Motion, Interaction, and Games (MIG ’26), December 11–13, 2026, Charleston, SC, USA. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3828647.3857292

Introduction

Voice recognition, synthesis, and vocap (voice-capture driven character animation) solutions are now commonly used for games, film and virtual avatars [45]. A fundamental advantage of voice is scalability, due to its ease and flexibility of recording. Characters in games like Cyberpunk 2077 [13] are driven by tens of thousands of lines of dialogue recorded in over ten languages. This dialogue however, is typically recorded by voice actors in isolation, read from a script with little preparation or context [34]. Even for AAA games, “approximately 95% of the time, actors are recorded alone in the booth” (private communication [33]). Consequently, despite expressive individual voice acting, simply splicing these recordings together produces conversational turns that sound uniformly robotic, lacking the natural overlaps and pauses that characterize interactions. Further, conspicuously missing in such conversations, are listener interjections or back-channels. Back-channels are short verbal utterances like “mmm-hmmm”, “cool” or “maybe”, that can indicate listener engagement, enthusiasm, or skepticism, but above all keep the conversation flowing [8]. While a vocabulary of back-channels can sometimes be recorded separately, and manually edited into conversations by an animator [33], such an approach is tedious and does not scale to increasingly many hours of dialogue to be processed for games like Cyberpunk 2077.

We jointly address the scheduling and back-channel problem with a unified system, and present the first controllable neural solution to producing dynamic conversational turns from a script of independently recorded or TTS (text-to-speech) voice-lines. As seen in Figure 1, our framework operates on a script-ordered set of voice-lines (recorded by voice actors or synthesized with TTS), and produces natural two-speaker dialogue audio via a two-component pipeline. The Back-channel Predictor first determines where listener back-channels should occur and of what type, operating at the voice-line level. The Neural Scheduler then determines the precise timing of back-channels, turn transitions and voice overlap for both channels jointly. Both components combine learned models with algorithmic inference, and expose hyperparameters that afford fine-grained control over the back-channel frequency, valence, and conversational dynamics of the output at inference time.

Input turn-clips are processed by the Back-channel Predictor and sequenced with generated back-channels by the Neural Scheduler

Figure 1. Input turn-clips are processed by the Back-channel Predictor, which determines and synthesizes a candidate verbal back-channel. The Neural Scheduler then sequences the turn-clips and generated back-channels with appropriate temporal gaps or overlap into a conversational dialogue. Circled numbers indicate animator-centric parameters: (1) turn-clip splitting, and (2) a back-channel probability threshold to control back-channel frequency; (3) a back-channel vocabulary with varying valence, duration, and intensity; and (4) timing constraints to control pause and overlap in conversational turn-taking.

Outside of applications in game development, we further see our model as a stepping stone towards more natural and interactive voice LLMs. Successful text-LLMs like ChatGPT and Gemini increasingly provide voice-LLM support [46]. These models are trained on synthetic TTS data, and fine-tuned using small conversation datasets (e.g., Fisher or Switchboard [7]), and thus produce voice assistants that lack back-channeling and natural conversation dynamics. Our work can transform large text datasets into dynamic conversations using synthesized voices for use in training future conversational voice assistants, and other full-duplex model applications [40].

Evaluation. We evaluate each component of our model in isolation and the full pipeline end-to-end. The Back-channel Predictor achieves state-of-the-art performance on sentence-level back-channel opportunity prediction (section 4), and the Neural Scheduler produces conversational timing statistics closest to ground truth across nearly all turn-taking metrics (Table 3). Evaluated end-to-end, our system generates back-channels at the highest frequency while achieving the best temporal alignment with natural listener behavior on Full-Duplex Bench (Table 4), and is significantly preferred over PersonaPlex [40] in a perceptual user study (section 6).

Model∣ΔIPU∣\lvert\Delta\mathrm{IPU}\rvert∣ΔPause∣\lvert\Delta\mathrm{Pause}\rvert∣ΔGap∣\lvert\Delta\mathrm{Gap}\rvert∣ΔOverlap∣\lvert\Delta\mathrm{Overlap}\rvert
Naive Cascade4.35.50.93.6
dGSLM4.63.61.81.9
LSLM4.13.41.52.3
NTPP3.32.81.41.9
Moshi0.8_{0.8}16.06.88.32.1
Moshi0.9_{0.9}6.42.73.31.1
Moshi1.0_{1.0}0.30.60.30.8
Neural Scheduler0.10.10.60.4

Table 3. Turn-taking statistics (cumulated duration/min) for generated dialogues, compared to ground truth using mean absolute differences (∣Δ∣\lvert\Delta\rvert) between predicted and ground truth values. Lower ∣Δ∣\lvert\Delta\rvert values indicate better performance, with the best results highlighted in bold.

ModelTOR (↓)Freq (↑)JSD (↓)
Freeze-Omni0.6360.0010.997
Gemini0.0910.0120.896
Moshi1.0000.0010.957
dGSLM0.6910.0150.934
PersonaPlex0.3270.0250.649
Proposed (Ours)0.0000.0620.644

Table 4. Full-Duplex Bench back-channel evaluation results.

Contribution. Our principal contribution is, to our knowledge, the first end-to-end pipeline that combines two independently recorded voice-line channels into a unified dyadic conversation with natural, controllable back-channeling and turn-taking dynamics. Its key components are:

  1. A Neural Scheduler, based on our formulation of voice-line scheduling as next-token prediction, in which a full-duplex language model is given the content of every line and learns only its timing.

  2. A Back-channel Predictor, built around a multimodal classifier that predicts from a clip’s text and audio whether it invites a back-channel, and of what type.

  3. A search-based inference algorithm that uses the scheduler to score candidate timings, exposing animator-centric parameters for back-channel insertion and high-level control over conversational turn dynamics without retraining.

Related Work

We classify related work as pertaining to text-to-speech synthesis, conversational dynamics, and full-duplex language models.

Synthesizing Speech from Text

TTS has evolved from early sequence-to-sequence models [42, 50, 55] to efficient non-autoregressive architectures and GAN vocoders [21, 22, 36, 37]. Modern TTS is typically a two-stage process using discrete audio codecs [10 Large Language Models generate tokens from text {citep}] (Zeghidour et al. 2021)[Borsos et al. 2023a; Wang et al. 2023], which are decoded via diffusion {citep}[4, 43] or flow matching {citep}[Chen et al. 2024b; Du et al. 2024; Peng et al. 2024; Resemble AI 2025] into high-fidelity speech {citep}[5, 23]. Text alone however, is insufficient for dyadic conversation, which requires modeling speaker overlap {citep}[35] and listener back-channels {citep}[Anderson et al. 2025] absent in text transcripts. While joint-synthesis systems like MoonCast {citep}[20] and CoVoMix {citep}[61]` simulate basic turn-taking, they lack precise control over turn boundaries and do not model back-channels. We address these by using a neural scheduling system to combine single-speaker lines into natural two-channel dialogue, with explicit control over both turn-taking and back-channel dynamics.

Controlling Back-channels and Turn Dynamics

Back-channels provide essential grounding [8, 9], where continuers (e.g., mmmhmm) signal turn maintenance [Schegloff 1982] and emotional responses (e.g., oh!) invite elaboration [17, 48]. Inter-turn timing is equally communicative; while the modal gap for English is around 240 ms [44], deviations can signal competition [49], deception [39], or deliberation [Mc-Conaghie 2021]. Existing computational models approach these dynamics through multi-class classification [14, 25, 31, 53] or continuous Voice Activity Projection (VAP) [15, 19]. However, these statistical methods struggle with back-channel reliability and lack mechanisms for animator control. We address these limitations by combining statistical and procedural components, enabling explicit control over the timing and content of both turn transitions and back-channels.

Full-Duplex Language Models

The emergence of speech language models has made it possible to train systems that understand spoken input and respond through next-audio-token prediction [56, 57, 60]. However, these models process a single audio stream at a time, and are thus unable to generate conversational dynamics such as overlapping speech or back-channels. Full-duplex models address this by jointly modeling two token streams at all times. dGSLM [30] introduced a dual-tower Transformer trained on stereo telephone conversations, while LSLM [27] and NTPP [54] extended the paradigm with richer inter-stream interaction. Paired with streaming audio encoders and decoders, systems like Moshi [11] can process live user speech on one channel while generating responses on the other. However, the scarcity of dual-channel training data results in many of these models producing believable conversational dynamics but incoherent speech. Recent work such as PersonaPlex [40] addresses this through large scale TTS-synthesized training data, but is sparse in back-channeling. We complement this line of research, as our system can generate back-channel-rich dialogue data that could serve as training material for the next generation of full-duplex conversational agents.

Method Overview

Our framework operates on voice clips, which can either be lines recorded by a voice actor or a synthetic voice generated from text. In conversation, a turn is a contiguous stretch of spoken content by a single speaker that ends when the floor passes to the other. We refer to the voice audio of a turn (or a portion of it) as a voice clip. Given an ordered set of said voice clips, our system synthesizes natural two-speaker audio via a two-component pipeline (Figure 1). The first component, the Back-channel Predictor, determines where in the conversation listener back-channels should occur: a coarse, voice-clip classification of whether a back-channel response for the clip is warranted. The second component, the Neural Scheduler, then determines when precisely each voice line is played, jointly scheduling both the original input lines and any synthesized back-channels to produce natural turn transitions with voice overlap and back-channels. The pipeline exposes hyperparameters that afford fine-grained control over the conversational dynamics of the output at inference time (§ 7.1).

Back-channel Predictor

The Back-channel Predictor supplements the input turn clips with synthesized back-channel clips. At its core is a trained classifier that predicts for each voice-clip whether the listener should produce a back-channel and, if so, whether it should be a low-salience acknowledgment or a high-salience reactive response. Predictions are then used to produce a corresponding back-channel clip, which is added to the set of input clips to be scheduled.

We predict back-channels per voice clip rather than continuously over time, as in Voice Activity Projection (VAP) [15, 19]. Clips are the unit in which our input is authored, and we found in practice that VAP models predict whether a turn will continue well, but are poor at deciding when to insert a back-channel. The classifier therefore decides only whether a clip receives a back-channel, and the Neural Scheduler (§ 5) decides its precise timing.

Back-channel Prediction Classifier

Given a voice-clip, the classifier measures the semantic and auditory opportunity for a back-channel response, and if so, what type. We organize labels by emotional salience rather than functional or linguistic taxonomy, as back-channel choices are highly personal, depending on speaker intent [29], and varying with speaker-listener relationship [16]. This yields three classes:

  • No-back-channel (class 0): the clip presents no opportunity for a back-channel response.

  • Acknowledgment (class 1): the clip invites a neutral, low-salience response (e.g., uh-huh, yeah).

  • Reactive (class 2): the clip warrants an emotional, high-salience response (e.g., wow, really?, for sure, absolutely!).

This salience-based organization directly governs the behavioral impact of each back-channel on the perceived dynamics of the conversation. Once classified, a back-channel can be produced by sampling a voice line from a user-defined pool of back-channel audio, or generated through TTS in the same voice from a user-defined back-channel vocabulary.

Input Representation

The classifier is conditioned on a windowed, linearized view of the conversation, centered on a candidate back-channel site. For a target voice clip at index tt, the input is constructed as follows. A context window of ±k\pm k clips (we use k=5k = 5) around tt is extracted. Each clip within this window is prefixed with a speaker-relative tag: the speaker of the target clip is always labeled A, and the other speaker B, so the model learns speaker-role-invariant representations. Clips are delimited by </s> boundary tokens, and a special <bc_slot> token is appended to the target clip to mark the candidate insertion site. The classifier predicts the back-channel class at this marked position. Since prediction operates at the clip level, the density of candidate sites, and therefore the number of back-channels inserted into a conversation, can be controlled by varying the granularity at which longer turns are segmented into clips.

Earlier back-channel decisions are fed back into the context: a clip that received a back-channel is followed by its class as a tag token (e.g. [BC:Acknowledgment]), which lets the classifier account for recent back-channel density. An input might look like:

A: I was at Walmart today [BC:Acknowledgment] </s> A: Shopping for some clothes </s> B: I was there too </s> A: I didn’t see you there. <bc_slot> </s> B: We probably went at different times </s>

The target speaker’s waveform for the clip (capped at 15 s) is paired with the text input, providing prosodic cues.

Training Data

We source training examples from the natural split of the Seamless Interaction dataset, which provides dual-channel audio with sentence-level transcripts [1]. Back-channel annotations are derived automatically from text using a two-stage pipeline. First, a rule-based pre-filter identifies candidate clips by requiring that each be short (under 1.0s or at most 3 words), temporally isolated from the same speaker’s surrounding utterances (at least 0.3s silence on both sides), and preceded by the other speaker within 2 seconds. Candidates passing this filter are then submitted to a Qwen LLM, which classifies each into one of five back-channel types, Neutral, Positive, Emotional, Anticipation, and Confirmation, as defined in prior work [29], or rejects it as non-back-channel, using a ±10\pm10s conversational context window. These types are then collapsed into a 3-class scheme for training: No-back-channel, Acknowledgment (if Neutral or Positive), and Reactive (Emotional, Anticipation, or Confirmation). The pipeline yields 33,977 Reactive and 137,808 Acknowledgment back-channel clips. We additionally sample negative examples at a 5:1 ratio relative to positive instances, yielding 703,140 No-back-channel clips. The dataset is split 96/2/2 for train, validation, and test.

Model Architecture and Training

The classifier uses a late-fusion architecture combining a text encoder with a frozen speech encoder. The text branch is a fine-tuned RoBERTa-base encoder [26] extended with back-channel-specific special tokens. The audio branch uses a frozen HuBERT-base encoder [18] with a learnable weighted sum over all 13 hidden states, following the SUPERB protocol [58], downsampled via a stride-4 convolution to 12.5 Hz. A learned fusion CLS token is prepended to the concatenated text and audio representations, augmented with modality embeddings, and processed by a 2-layer Transformer encoder. The CLS output is passed through a linear head projecting to C=3C=3 classes. Full architecture details are available in our code repository.1

The model is trained with cross-entropy loss using inverse-square-root class weighting to mitigate the imbalance between No-back-channel and the two back-channel classes. HuBERT parameters remain frozen throughout; all other parameters are optimized jointly. Training proceeds for up to 10 epochs with early stopping on validation macro-F1 (patience of 3 epochs).

Classifier Evaluation

We compare our model against the state-of-the-art back-channel classifiers of Wang et al. and Lin et al.,2 along with ablated variants of our own model. Both Wang et al. and Lin et al. are originally formulated as tri-class classifiers over (continue, switch, back-channel); for a fair comparison we retrained both for the binary task [Back-channel, No-back-channel], and we ensure all models receive information regarding future context. All models are trained on our processed dataset (subsection 4.3) and evaluated on held-out data.

We report five metrics, all computed on the binary BC-vs-NoBC decision: P precision and R recall, F1, the harmonic mean of precision and recall at the model’s argmax decision; AUC, the area under the ROC curve, which measures how well the model ranks examples by back-channel likelihood independently of any threshold; and EER, the equal-error-rate operating point on the ROC curve where false-positive and false-negative rates coincide (EER ≈1−\approx1 - AUC for well-behaved curves). Higher F1 and AUC are better; lower EER is better. AUC and EER are derived from the calibrated score 1−P(No-back-channel)1 - P(\text{No-back-channel}), so they reflect the full probability output rather than the hardened prediction.

We additionally study four ablations, trained without corresponding input contexts: no audio silences the waveform; no future text drops context turns following the target; no past text drops context turns preceding the target; no current text removes the words on the target turn itself, leaving only its turn marker.

Table 1 reports results on the held-out test set. Our full model attains an AUC of 0.798, outperforming both prior baselines. Among ablations, removing audio causes the largest recall drop (0.49 →\rightarrow 0.25) with little effect on precision, indicating audio is primarily responsible for surfacing back-channel-eligible moments. Removing future or current text each costs 0.07–0.11 AUC, confirming that linguistic content in and immediately following the target turn carries substantial signal. By contrast, removing past context barely affects performance (Δ\DeltaAUC ≤\leq 0.03), suggesting the model relies far more on the current turn and its immediate continuation than on broader conversational history.

The classifier’s reliance on audio also suggests why the LLM annotator used to label our training data (subsection 4.3) is not a substitute for the classifier: it works from text alone, and it labels back-channels that a listener has already produced, whereas at inference time no listener response exists.

The tri-class breakdown (Table 2) shows the model handles No-back-channel well (F1 = 0.888) but struggles to separate the two BC subtypes. Acknowledgment retains an AUC (0.792) on par with the binary task, while Reactive is substantially harder (F1 = 0.229), reflecting both its lower frequency (400 examples) and greater linguistic and prosodic variability. This aligns with Onishi et al. [2023], who similarly find that fine-grained back-channel categories are inherently harder to predict. The inherent ambiguity of back-channel type motivates our decision to expose animator control as an explicit stage in the pipeline.

ModelF1 ↑P ↑R ↑AUC ↑EER ↓
Baselines
Wang et al.0.430.460.400.730.33
Lin et al.0.420.340.550.690.36
Ours
Full model0.510.530.490.800.28
w/o audio0.340.550.250.750.32
w/o future text0.380.360.400.690.37
w/o current text0.430.410.460.730.33
w/o past text0.480.520.450.770.30

Table 1. Binary back-channel classification results on held-out data. Higher F1, P, R, AUC are better; lower EER is better.

ClassF1 ↑P ↑R ↑AUC ↑EER ↓Support
No-back-channel0.890.880.900.800.288,891
Acknowledgment0.440.460.430.790.281,709
Reactive0.230.240.220.760.31400

Table 2. Per-class performance of our full model on the three-way merged task (No-back-channel / Acknowledgment / Reactive).

Neural Scheduler

While the Back-channel Predictor augments input voice-clips with back-channel clips, the Neural Scheduler orders the timing of all clips. The core of the Neural Scheduler is a full-duplex LLM fine-tuned on the task of scheduling an ordered set of voice lines. This section formally introduces the formulation of voice-clip scheduling, discusses how the model is trained, inference strategies, as well as evaluations of the scheduler.

Scheduling as Next Token Prediction

We formulate turn scheduling as a next token prediction problem by introducing two structurally distinct token regions: an input region that encodes the input turn-clips of each speaker, and an output region where the model generates interleaved output tokens, as shown in Figure 2.

Illustration of voice-line scheduling as a next-token prediction problem

Figure 2. Illustration of how voice-line scheduling can be formulated as a next-token prediction problem.

Input. Each speaker’s input is an independent, non-interleaved sequence of audio tokens. For speaker s∈{A,B}s \in\{A, B\}, the input s(s)\mathbf{s}^{(s)} is a flat sequence of discrete audio tokens structured into NsN_s clips by per-turn start and end marker tokens [sTn][\mathrm{sT}_n] and [eTn][\mathrm{eT}_n], with a silence token sil\mathrm{sil} separating consecutive clips:

s(s)=(sil;[sTn];cn(s);[eTn])n=1Ns(1)\mathbf{s}^{(s)} = \left(\mathrm{sil}; [\mathrm{sT}_n]; \mathbf{c}_n^{(s)}; [\mathrm{eT}_n]\right)_{n=1}^{N_s} \tag*{(1)}

where cn(s)\mathbf{c}_n^{(s)} denotes the speech tokens for clip nn of speaker ss, as illustrated in Figure 2. Each clip’s speech tokens are drawn from an audio tokenizer; the single silence token sil\mathrm{sil} between clips is a structural separator only, as it carries no temporal information and does not encode the actual duration of the pause, only that a clip boundary exists. As a result, the inputs are purely content representations: they specify what each speaker says and in what order, but are entirely stripped of timing. The two inputs generally have different lengths, reflecting the different number and duration of each speaker’s clips. The full input region is formed by concatenating both inputs sequentially, delimited by per-speaker boundary markers [sA][\mathrm{sA}], [eA][\mathrm{eA}], [sB][\mathrm{sB}], [eB][\mathrm{eB}]:

xin=([sA];s(A);[eA];[sB];s(B);[eB])(2)\mathbf{x}_{\mathrm{in}} = ([\mathrm{sA}]; \mathbf{s}^{(A)}; [\mathrm{eA}]; [\mathrm{sB}]; \mathbf{s}^{(B)}; [\mathrm{eB}]) \tag*{(2)}

Output. The output region xout\mathbf{x}_{\mathrm{out}} is where timing lives. It is a strictly interleaved, frame-aligned sequence of audio tokens:

xout=(a1(A),a1(B),a2(A),a2(B),…,aT(A),aT(B))(3)\mathbf{x}_{\mathrm{out}} = (a_1^{(A)}, a_1^{(B)}, a_2^{(A)}, a_2^{(B)}, \ldots, a_T^{(A)}, a_T^{(B)}) \tag*{(3)}

of length 2T2T, where each pair (at(A),at(B))(a_t^{(A)}, a_t^{(B)}) represents one frame (at 12.5 Hz) of both channels simultaneously, similar to the formulation in [54]. Frames with no active speech are assigned the silence token sil\mathrm{sil}, so the output region encodes the full temporal structure of the conversation including pauses, overlaps, turn transitions, through the pattern of speech and silence tokens across both channels.

The Neural Scheduler therefore learns the conditional distribution pθ(xout∣xin)p_\theta(\mathbf{x}_{\mathrm{out}} \mid\mathbf{x}_{\mathrm{in}}), where the input region supplies the what and the output region captures the when. Crucially, xout\mathbf{x}_{\mathrm{out}} contains the same speech tokens as xin\mathbf{x}_{\mathrm{in}}, but interleaved across both channels with silence tokens encoding the pauses, overlaps, and turn transitions that constitute conversational dynamics.

Training Objective

Standard next-token prediction (NTP) loss alone is insufficient for training the Neural Scheduler, which we demonstrate in subsection 5.5.1. The fundamental difficulty is a severe class imbalance inherent to the output token sequence xout\mathbf{x}_{\mathrm{out}}. Within a speech turn, the task of predicting the next token can be trivially solved by simply tracking the token indices in xin\mathbf{x}_{\mathrm{in}}. Structural transition events, turn changes, mutual pauses, and simultaneous speech are collectively rare relative to within-turn continuation steps. As a result, a model that consistently predicts “continue the current speaker” achieves deceptively low cross-entropy while failing to capture the very events that define conversational timing dynamics. We call this inertia bias: since same-token continuations dominate the training signal, the model gravitates toward repeating the current channel’s token and never acquires a reliable internal representation of when a transition is imminent. To counteract inertia bias, the Neural Scheduler is trained with a composite objective. The primary loss is a standard NTP cross-entropy over the 2T2T tokens of the output region:

LNTP=−12T∑t=1T∑s∈{A,B}log⁡pθ(at(s)∣xin,a<t(A),a<t(B))(4)\mathcal{L}_{\mathrm{NTP}}=-\frac{1}{2T}\sum_{t=1}^{T}\sum_{s\in\{A,B\}}\log p_{\theta}\left(a_{t}^{(s)}\mid\mathbf{x}_{\mathrm{in}},a_{<t}^{(A)},a_{<t}^{(B)}\right) \tag*{(4)}

The auxiliary duration head is a two-layer MLP from the transformer’s final hidden state to 32 classes (subsection 5.3), trained to predict the remaining silence duration at each position: how many more consecutive frames the current token will persist on that channel before a transition. Concretely, the ground-truth label at position (t,s)(t,s) is

dt(s)={0if at(s) is a speech token,min⁡(rt(s),31)if at(s)=sil,(5)d_{t}^{(s)}= \begin{cases} 0 & \text{if }a_{t}^{(s)}\text{ is a speech token},\\ \min\left(r_{t}^{(s)},31\right) & \text{if }a_{t}^{(s)}=\mathrm{sil}, \end{cases} \tag*{(5)}

where rt(s)r_{t}^{(s)} is the countdown of consecutive silence frames remaining at frame tt on channel ss, including the current frame. Class 0 indicates that the channel is producing speech (no silence stretch to measure); classes 1–31 encode the remaining silence duration in 80 ms increments, with class 31 acting as a catch-all for silences of 31 or more frames (≥2.48\ge2.48 s). This labeling injects gradient signal at every frame inside a silence stretch, compelling the model to build an explicit, future-facing estimate of when the current segment will end rather than reasoning myopically about the next single frame. The duration cross-entropy is

Ldur=CE⁡(hθ(xout),d)(6)\mathcal{L}_{\mathrm{dur}}=\operatorname{CE}\left(h_{\theta}(\mathbf{x}_{\mathrm{out}}),\mathbf{d}\right) \tag*{(6)}

The composite training objective is then a weighted sum of the two, in all of our experiments we use λdur=0.5\lambda_{\mathrm{dur}}=0.5

L=LNTP+λdurLdur(7)\mathcal{L}=\mathcal{L}_{\mathrm{NTP}}+\lambda_{\mathrm{dur}}\mathcal{L}_{\mathrm{dur}} \tag*{(7)}

Architecture

The Neural Scheduler must process both single-channel script tokens in xin\mathbf{x}_{\mathrm{in}} and dual-channel interleaved tokens in xout\mathbf{x}_{\mathrm{out}}. Rather than employing separate models per channel as in Moshi [11], we use a single decoder-only LLM that handles both regions, following the interleaved-output paradigm of NTPP [54] and SyncLLM [51]. The backbone is GLM-4-Voice [60], which shares its architecture with the Qwen family [57]. We make three modifications to adapt the base model to the scheduling task.

Channel embeddings. To help the model disambiguate between token regions and between speakers, we introduce a learned channel embedding ec∈Rd\mathbf{e}_{c}\in\mathbb{R}^{d} for each of three channel identifiers c∈{0,1,2}c\in\{0,1,2\}. Tokens in the input region xin\mathbf{x}_{\mathrm{in}} receive the embedding e0\mathbf{e}_{0}; tokens on channel A in the output region xout\mathbf{x}_{\mathrm{out}} receive e1\mathbf{e}_{1}; tokens on channel B receive e2\mathbf{e}_{2} (as shown in Figure 3). These embeddings are added to the query and key representations in each self-attention layer, enabling the model to distinguish between the single-channel input format and the two interleaved output channels throughout the network.

Illustration of positional id and channel id assignment for the corresponding embeddings

Figure 3. Illustration of positional id and channel id assignment for the corresponding embeddings.

Positional embeddings. We modify the rotary position encoding to reflect the temporal structure of each region. In the input region xin\mathbf{x}_{\mathrm{in}}, positions increment sequentially by one per token. In the output region xout\mathbf{x}_{\mathrm{out}}, the two tokens at each frame (at(A),at(B))(a_{t}^{(A)},a_{t}^{(B)}) share the same position index, so that the model’s notion of temporal order is consistent with the frame-aligned structure: position indices increment by one per frame rather than per token. Concretely, for a token at absolute index ii in the output region with dialogue offset δ=i−∣xin∣\delta=i-\lvert\mathbf{x}_{\mathrm{in}}\rvert, the position index is ∣xin∣+⌊δ/2⌋\lvert\mathbf{x}_{\mathrm{in}}\rvert+\lfloor\delta/2\rfloor.

Duration prediction head and vocabulary. We extend the model with the duration prediction head described in subsection 5.2: a two-layer MLP projecting from the hidden dimension to 32 duration classes (corresponding to countdown values 0, 1, …\ldots, 30 and a catch-all class 31 for durations ≥31\ge31 frames). Additionally, we expand the vocabulary with structural boundary tokens, per-speaker script delimiters ([sA], [eA], [sB], [eB]), dialogue delimiters ([sD], [eD]), and per-turn markers ([sTₙ], [eTₙ]), to demarcate the regions and turns described in subsection 5.1.

Dataset Processing

We train the Neural Scheduler on the audio data from the Seamless Interaction dataset [1], a large-scale collection of two-channel conversations recorded on separate microphone channels. Processing each recording into a training example proceeds through four stages: audio tokenization, input and output region construction, turn ID assignment, and duration labeling. We deliberately exclude the Fisher corpus [7] from training, reserving it exclusively for evaluation to avoid train-test contamination and to enable fair comparison against prior works.

Audio tokenization. Each channel is independently encoded into a sequence of discrete speech tokens using the WhisperVQ encoder from GLM-4-Voice [60]. A voice activity detection (VAD) model [47] is then applied to each channel to obtain per-channel speech timestamps. Non-speech frames identified by VAD are replaced with the silence token sil, yielding two token streams that faithfully represent both the acoustic content and the activity pattern of each speaker.

Input and output region construction. The input region xin\mathbf{x}_{\mathrm{in}} is constructed by encoding each channel’s speech content into a compact, non-interleaved script. Consecutive runs of silence tokens are collapsed into a single sil, retaining only the turn boundaries rather than the raw temporal gaps. Each turn’s content is delimited by start and end markers [sTₙ] and [eTₙ], with a single silence token separating consecutive turns. For turns containing internal pauses (e.g., mid-utterance hesitations), a single sil is inserted between the sub-segments within the turn markers as well. The two per-channel scripts are concatenated sequentially and wrapped by per-speaker boundary markers as described in subsection 5.1. The output region xout\mathbf{x}_{\mathrm{out}} is formed by interleaving the two silenced token streams at each frame, preserving the full temporal structure including pauses and overlaps. Long recordings are segmented into windows of at most 1 024 tokens (approximately 40 s of conversation at 12.5 Hz per channel). During training, the cross-entropy loss is applied only over the output region; the input prefix is masked.

Turn ID assignment. Turn IDs annotate the input region to delineate conversational turns across both speakers. To assign them, VAD-derived speech segments from both channels are jointly sorted by onset time, and temporally adjacent segments from the same speaker are sometimes (p=50%p = 50\%) merged into contiguous turns. A 50 % overlap rule then handles back-channels: if at least half of a segment’s duration overlaps with the temporal span of an already-assigned turn, the segment is absorbed into that turn rather than initiating a new one, meaning that back-channels share the same turn ID as the active speech turn. Turn IDs are renumbered contiguously after all merging is complete, producing the indices nn used in the turn markers [sTn\mathrm{sT}_n] and [eTn\mathrm{eT}_n].

Remaining duration labels. Each position in the output region is annotated with a remaining duration label as described in subsection 5.2. These labels provide per-frame supervision for the auxiliary duration prediction head.

Evaluation of Neural Scheduler

We evaluate the Neural Scheduler through an ablation of the duration prediction objective (subsubsection 5.5.1) and a quantitative comparison of conversational dynamics against baselines (subsubsection 5.5.2). Qualitative examples of its schedules (subsection 6.2) and the evaluation of the full system are presented in section 6.

Ablation: Duration Prediction Objective

Between-speaker intervals in generated and ground-truth dialogue

Figure 4 (PDF p. 7). Between-speaker intervals (negative: overlap; positive: gap) in generated vs. ground-truth dialogue. Without duration prediction (left), generated intervals collapse into a spike near zero; with it (right), they track ground truth.

Figure 4 ablates the auxiliary duration prediction objective described in subsection 5.2. Without it (left), inertia bias causes the generated between-speaker intervals to collapse into a narrow spike just above zero, as the model learns to perpetually continue the current token rather than anticipate transitions. Adding duration prediction (right) resolves this collapse, confirming that the auxiliary objective successfully counteracts inertia bias. Notably, the resulting distribution closely tracks ground truth across both overlaps and gaps, suggesting that the model has learned realistic turn timing dynamics.

Between-speaker intervals comparing NTP objective only with NTP objective plus silence prediction objective

Figure 4. Between-speaker intervals (negative: overlap; positive: gap) in generated vs. ground-truth dialogue. Without duration prediction (left), generated intervals collapse into a spike near zero; with it (right), they track ground truth.

5.5.2 Quantitative Results. We quantitatively evaluate our Neural Scheduler’s understanding of conversational dynamics following the established protocol of NTPP [54] and Moshi [11], where models are prompted with 10 seconds of audio and tasked to complete the conversation on unseen Fisher corpus [7] samples. Performance is measured by comparing corpus statistics against ground truth across four VAD-derived turn-taking metrics: an IPU (Inter-Pausal Unit) is a contiguous speech segment on a single channel; a Pause is a within-speaker silence where the same speaker resumes; a Gap is a between-speaker silence followed by a turn change; and an Overlap is any interval of simultaneous activity on both channels. For each metric, we report the mean absolute difference ∣Δ∣\lvert\Delta\rvert in cumulated duration (seconds per minute) against ground truth, where lower values indicate closer alignment with natural conversational dynamics.

We compare against Moshi [11] at temperatures (0.8, 0.9, 1.0), NTPP [54], dGSLM [30], LSLM [27], and the naive cascade baseline from [54], with baseline numbers taken directly from the respective publications under the same evaluation conditions. All models are evaluated on the same held-out test split of the Fisher corpus [7].

Table 3 shows that our Neural Scheduler produces conversational dynamics closest to ground truth across nearly all metrics, outperforming both cascaded and full-duplex baselines. This is consistent with the nature of our approach: rather than generating content and timing jointly, the scheduler operates on pre-synthesized voice lines with known linguistic content, leaving timing as the sole degree of freedom to optimize.

Our formulation also sidesteps the intelligibility degradation observed in existing full-duplex models like Moshi [11], which relies on softmax temperature to trade off turn-taking naturalness against speech quality during token inference, higher temperatures improve timing behavior at the cost of intelligibility. Because our model schedules fixed voice lines rather than generating tokens, it is immune to this tradeoff entirely.

Inference

At inference time, the Neural Scheduler operates as a scoring function rather than an autoregressive generator. Unlike autoregressive generation, where token identities are unknown, the tokens of xoutx_{\mathrm{out}} are fully determined by the onset positions of the voice-line clips: each frame either advances a clip or emits silence. This means that scheduling reduces to a search over onset assignments rather than a generation problem. As the Neural Scheduler is trained on natural conversation, its NLL reflects how consistent a given timing arrangement is with the conversational dynamics it has observed: arrangements with awkward gaps or poorly timed overlaps will receive high NLL, while natural-sounding ones will not. We therefore use NLL directly as a proxy for timing naturalness. Scheduling then proceeds by searching over candidate onset positions and committing the one that achieves the lowest NLL, effectively asking the model to judge arrangements rather than produce them. This formulation has three appealing properties: the speech content of every voice line is already fixed at inference time, so there is no risk of drift from the original voice lines; the search is fully deterministic, producing reproducible schedules; and the search space is explicitly parameterized, making the scheduler directly controllable without any modification to model weights, as we demonstrate in subsection 7.1.

Naively searching over all joint onset assignments is intractable, so we schedule greedily in three sequential passes, each conditioned on the placements committed by the previous one: (1) primary turns, (2) back-channels, and (3) a local re-timing correction. Algorithm 1 summarizes the procedure, Figure 5 illustrates it on a small example, and we detail each pass below. All timings are expressed in frames (token steps) at the model’s 12.5 Hz rate.

Figure 5. An illustrative example of the three passes. Pass 1 places the turns, Pass 2 inserts the back-channel bb with a brief pause in A’s turn (dotted), and Pass 3 re-times when A resumes. Dashed boxes mark each clip’s search range.

Scoring a placement. Given a set of committed onsets, we materialize the corresponding interleaved output region xoutx_{\mathrm{out}} by advancing the assigned clip on each channel at each frame and emitting sil elsewhere, then evaluate its NLL under the frozen model conditioned on xinx_{\mathrm{in}}. Because moving a single clip only perturbs timing locally, we score a window around the clip under consideration, its token span padded by a context margin of cc frames on each side, rather than re-scoring the full dialogue; this keeps each candidate evaluation to one cheap forward pass. We write NLL(⋅\cdot) for this windowed score, where lower is more natural. We write S\mathcal{S} for the set of placements committed so far and t@ot@o for clip tt placed with its onset at frame oo, so NLL(S∪{t@o}\mathcal{S}\cup\{t@o\}) scores the current schedule with tt added at oo. For a placed clip, onset(⋅\cdot), end(⋅\cdot) and dur(⋅\cdot) give its first frame, end frame and length.

Pass 1: Primary turns. We first remove all back-channels and place the primary turns in conversational order. Each turn is anchored at the end of the most recently committed turn plus a nominal inter-turn gap gg (the first turn is anchored at gg), and we scan candidate onsets over a window of ±W\pm W frames around the anchor at stride σ\sigma, committing the onset of lowest windowed NLL. A hard floor prevents a turn from overlapping any previously committed turn on the same channel, while cross-channel overlap is permitted, which is what allows the model to schedule natural turn-transition overlap. This window is also the primary point of animator control (subsection 7.1): tightening or shifting the per-speaker bounds forces simultaneous speech or widens inter-turn gaps into deliberate thinking pauses. Turns that contain internal pauses are optionally scheduled sub-segment by sub-segment, searching a short silence between consecutive sub-segments within a per-speaker range.

Pass 2: Back-channels. Back-channels are placed one at a time, each within the span of its host turn. For a back-channel of duration dd hosted by a turn ending at frame hendh_{\mathrm{end}}, we search onsets from the host’s start up to hend−∣rd∣h_{\mathrm{end}}-\lvert r d\rvert, so that at least a fraction rr of the back-channel overlaps the host (multiple back-channels on one host are ordered after one another). Rather than an absolute NLL, each candidate is scored by its NLL difference Δ=NLL(with back-channel)−NLL(baseline without it)\Delta= \mathrm{NLL}(\text{with back-channel})-\mathrm{NLL}(\text{baseline without it}) against the current schedule, isolating the back-channel’s own contribution; we commit the minimizing onset. For the first back-channel on a given host we additionally search a host-accommodation strategy jointly with the onset: leave the host unchanged so the back-channel overlaps ongoing speech (no-pause), or split the host at the insertion point with a silence gap of dd (full) or ⌊d/2⌋\lfloor d/2\rfloor (half). When a splitting strategy wins, the inserted silence is propagated as a rigid shift to every turn and back-channel scheduled at or after the split point, so downstream timing is preserved.

Pass 3: Local re-timing. A host split in Pass 2 can leave the remainder of the host turn timed awkwardly relative to the interjection. For each host turn that received a back-channel, we identify break-points, the sub-segment (or, if the back-channel lands in the host’s final segment, the following speaker’s turn) immediately after each back-channel. We freeze everything before a breakpoint qq, search a new onset for it that minimizes NLL, and rigidly shift all subsequent material by the resulting delta; the accumulated shift then propagates to all downstream turns. The search runs from the end of the preceding clip on qq’s channel up to onset(qq) + Dq+ED_q + E, where DqD_q is the total duration of the back-channels before qq on its host and EE bounds the extra silence that may be added, so the resumption can move earlier or later. This mirrors how a speaker naturally adjusts the resumption of their turn after a listener interjects. The scan bounds, gap gg, overlap ratio rr, extra gap EE, and context cc are the only free parameters; the exact values used in our experiments, together with the full implementation, are available in our code repository.

Pipeline Evaluation

To our knowledge, ours is the first computational work to jointly address conversational synthesis through back-channel insertion and turn-clip scheduling. For this reason, we evaluate each component of our framework individually against relevant baselines in section 4 and section 5. We assess the full pipeline in three ways: (1) evaluating back-channel generation and placement using the full-duplex benchmark [24]; (2) presenting illustrative conversational examples synthesized via text-to-speech; and (3) conducting a perceptual study to assess the overall system quality.

Full-Duplex Bench

We evaluate the joint performance of the back-channel generation and scheduling pipeline against existing full-duplex dialogue

Algorithm 1 Three-pass greedy scheduling. SS is the schedule built so far and t@ot@o denotes clip tt placed at onset frame oo. Exact search ranges are given in the text, and the full procedure in our code release (https://github.com/EvanSamaa/NeuralSchedularRelease).

Require: primary turns P\mathcal{P} in script order; back-channels B\mathcal{B}, each with a host turn; frozen scheduler pθp_{\theta}

  1. S←∅S \leftarrow\emptyset

  2. Pass 1: primary turns

  3. for each turn t∈Pt \in\mathcal{P} do

  4. W←\mathcal{W} \leftarrow onsets near the end of the previous turn

  5. add t@ot@o to SS, with o=arg⁡min⁡o′∈WNLL⁡(S∪{t@o′})o = \arg\min_{o' \in\mathcal{W}} \operatorname{NLL}(S \cup\{t@o'\})

  6. end for

  7. Pass 2: back-channels

  8. for each b∈Bb \in\mathcal{B} with host turn hh do

  9. W←\mathcal{W} \leftarrow onsets that overlap hh

  10. pick o∈Wo \in\mathcal{W} and a pause for hh with lowest NLL

  11. add b@ob@o to SS; pause hh at oo and shift later clips

  12. end for

  13. Pass 3: local re-timing

  14. for each clip qq that resumes speech after a back-channel do

  15. W←\mathcal{W} \leftarrow onsets shortly after the previous clip

  16. move qq to its lowest-NLL onset in W\mathcal{W}; shift later clips

  17. end for

  18. return xoutx_{\mathrm{out}} rendered from SS

systems using the Full-Duplex Bench evaluation framework [24]. The benchmark assesses back-channel behavior along three axes: the Turn-Override Rate (TOR), which measures the proportion of back-channels that exceed a duration threshold and thus intrude on the main speaker’s turn; Frequency (Freq), which measures the average rate of back-channel production per second; and the Jensen-Shannon Divergence (JSD) between the predicted back-channel temporal distribution and ground truth, where lower values indicate better alignment with natural back-channel timing.

As shown in Table 4, our proposed system achieves the best performance across all three back-channel metrics. A TOR of 0.000 indicates that none of our generated back-channels override the

main speaker; this is trivial, as our architecture only inserts back-channels. Our system also produces the highest back-channel frequency at 0.062 per second while achieving the lowest JSD of 0.644, suggesting that our model not only is capable of generating more back-channels but also schedules them in a manner most consistent with natural conversational behavior.

Qualitative Examples

We qualitatively compare our model against baselines on both the back-channel generation and turn scheduling tasks. For back-channel generation, we compare against PersonaPlex [40], the state-of-the-art full-duplex audio language model, using examples randomly sampled from the full-duplex benchmark. Figure 6 visualizes the voice activity of back-channels inserted by each system. PersonaPlex exhibits two recurring failure modes: either deferring the back-channel until after the speaker has finished (Example A), or taking over the floor during the speaker’s turn (Example B). In contrast, our model consistently inserts back-channels either during the speaker’s turn or within natural pauses, better reflecting the timing of human listener responses.

Voice activity timelines for two dialogue examples

Figure 6. Voice activity timelines for two dialogue examples comparing back-channels generated by PersonaPlex and our proposed model.

Turn Scheduling. As no dedicated turn-scheduling baseline exists, we compare against a pause-based heuristic that concatenates voice lines with a fixed inter-turn pause of 200 ms, an approach used in prior work both for synthesizing two-channel speech datasets [40] and as an evaluation baseline [30, 54]. We present examples across two settings: casual conversations synthesized with ChatterBox TTS [38], and more dramatic exchanges sourced from in-the-wild performances from movies and TV shows. As Figure 7 shows, the naive heuristic produces uniform inter-turn gaps throughout, whereas our scheduler generates varied timing with natural overlaps and pauses, yielding turn-taking dynamics that are markedly less mechanical.

Voice activity timelines comparing naive and proposed turn scheduling

Figure 7. Voice activity timelines for one dialogue, comparing the turn scheduling of the naive pause-based baseline (top) with that of our proposed model (bottom).

User Study

While our qualitative examples allow readers to assess output quality, we further conduct a forced-choice preference study (n=20n = 20 participants) to evaluate perceptual naturalness more rigorously. We showed 10 back-channeling examples, 5 turn-taking examples for conversational speech, and 5 for dramatic speech. In both back-channel and turn-taking conditions, participants listened to pairs of stimuli (ours vs. baseline) presented in random order and selected which sounded more natural, with instructions to attend to conversational timing rather than voice quality.

For the back-channel generation examples, our model was significantly preferred over PersonaPlex (67.5%, 95% CI [60.7%, 73.6%], p<0.001p < 0.001). Preference for turn scheduling is more nuanced. For casual conversational examples, preference for our model over the naive baseline was at chance (53.0%, 95% CI [43.3%, 62.5%], p=0.617p = 0.617), suggesting that timing differences are difficult to perceive in dialogue with relatively flat turn-taking dynamics. We suspect this reflects the nature of the test stimuli: casual chat and customer service exchanges tend to feature simple, sequential turn structure where the distinction between uniform and natural gaps is subtle. For more dramatic examples featuring real voice-actor recordings, where timing expressivity is more perceptually salient, preference for our model was significant (64.3%, 95% CI [52.6%, 74.5%], p=0.022p = 0.022), suggesting that the scheduler’s benefits become perceptually apparent when the source material has richer conversational dynamics. Additional dramatic examples are presented in the supplementary video.

Discussion

Animator Control

A key advantage of the search-based inference formulation is that the scheduler’s behavior is directly controllable through its search hyperparameters, without modifying model weights.

We demonstrate four axes of control in Figure 8 and the supplementary video. The nature of back-channels can be varied by swapping the back-channel candidates, ranging from short unobtrusive acknowledgements (e.g., “nice,” “uh-huh”) to longer, substantially significant responses that cause the primary speaker’s turn to be retimed by Pass 3. The number of back-channels is controlled by adjusting the prediction threshold, with lower thresholds producing more frequent back-channels. Overlapping speech can be induced by tightening onset position constraints to force simultaneous activity on both channels. Finally, the search window can be widened for a speaker to introduce natural thinking pauses between turns.

Four axes of animator control

Figure 8. Four axes of animator control: (1) back-channel content, by swapping candidates; (2) back-channel frequency, by lowering the prediction threshold; (3) speech overlap, by tightening onset constraints; and (4) thinking pauses, by widening one speaker’s search window.

Limitations

While the separation of timing and content is a core strength of our approach, it also limits how back-channels are chosen. The Back-channel Predictor decides whether a clip invites a back-channel and of which class, but not its wording: the back-channel itself is drawn from a vocabulary supplied by the animator. Our system therefore does not produce context-specific back-channels that are semantically grounded in what is being said, unlike voice-LLM-based methods such as PersonaPlex [40] which jointly model content and timing. This is a property of the Back-channel Predictor rather than of the Neural Scheduler, which simply schedules the clips it is given. The separation is deliberate: it keeps the content of every line, including back-channels, under the animator’s direct control, so that the nature and valence of back-channels can be specified by curating the candidate pool. The cost is that this requires manual intervention rather than automatic inference from context. A natural direction for future work is a hybrid approach that uses an LLM to select semantically appropriate back-channel candidates while delegating timing decisions to our scheduler. A second limitation is that our model operates on fixed voice-line clips and cannot introduce speech disfluencies such as repetitions or stutters, which are also important components of natural conversation. Incorporating disfluency generation into the voice-line synthesis stage can be another promising direction for future work.

Conclusion

We presented the first controllable neural solution for combining independently recorded voice lines into dynamic conversations with natural turn-taking and back-channeling. By decoupling timing from content, our Neural Scheduler produces conversational dynamics that closely match ground truth while remaining directly controllable: overlap density, back-channel frequency, and character expressiveness can all be specified without any modification to model weights. We believe this separation of timing and content is an unexplored direction for conversational synthesis, with applications ranging from game dialogue production to training data generation for future voice assistants.

References

References

  1. [1]Vasu Agrawal, Akiniinyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, Preveen Chowdary, Joe Chuang, Antony D’Avviro, Jon Daly, Ning Dong, Mark Duppenthaler, Cynthia Gao, Jeff Girard, Martin Gleize, Sahir Gomez, Hongyu Gong, Srivathsan Govindarajan, Brandon Han, Sen He, Denise Hernandez, Yordan Hristov, Rongjie Huang, Hirofumi Inaguma, Somya Jain, Raj Janardhan, Qingyao Jia, Christopher Klaiber, Dejan Kovachev, Moneish Kumar, Hang Li, Yilei Li, Pavel Litvin, Wei Liu, Guangyao Ma, Jing Ma, Martin Ma, Xutai Ma, Lucas Mantovani, Sagar Miglani, Sreyas Mohan, Louis-Philippe Morency, Evonne Ng, Kam-Woh Ng, Tu Anh Nguyen, Amia Oberai, Benjamin Peloponin, Juan Pino, Jovan Popovic, Omid Poursaee, Fabian Prada, Alice Rakototarin, Rakesh Ranjan, Alexander Richard, Christophe Ropers, Safiyyah Saleem, Vasu Sharma, Alex Shcherbyna, Jia Shen, Jie Shen, Anastasis Stathopoulos, Anna Sun, Paden Tomasello, Tuan Tran, Arina Turkatemko, Bo Wan, Chao Wang, Jeff Wang, Mary Williamson, Carleigh Wood, Tao Xiang, Yilin Yang, Julien Yao, Chen Zhang, Jiemin Zhang, Xinyue Zhang, Jason Zheng, Pavlo Zhyzherya, Jan Zikes, and Michael Zollhoefer. 2025. Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset. arXiv:2506.22554 [cs.CV] https://arxiv.org/abs/2506.22554
  2. [4]Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023b. SoundStorm: Efficient Parallel Audio Generation. arXiv preprint arXiv:2305.09636. https://arxiv.org/abs/2305.09636
  3. [5]Sanyuan Chen, Shuqi Liu, Ziyang Lu, Zonghang Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. 2024a. VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesis. arXiv preprint arXiv:2406.05370. https://arxiv.org/abs/2406.05370
  4. [7]Christopher Cieri, David Miller, and Kevin Walker. 2004. The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Maria Teresa Lino, Maria Francisca Xavier, Fátima Ferreira, Rute Costa, and Raquel Silva (Eds.). European Language Resources Association (ELRA), Lisbon, Portugal. https://aclanthology.org/L04-1500/DOI
  5. [8]Herbert H. Clark and Susan E. Brennan. 1991. Grounding in communication. In Perspectives on Socially Shared Cognition, Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley (Eds.). APA Books, 127–149.DOI
  6. [9]Herbert H. Clark and Edward F. Schaefer. 1989. Contributing to Discourse. Cognitive Science 13, 2 (1989), 259–294. doi:10.1207/s15516709cog1302_7DOI
  7. [10]Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High Fidelity Neural Audio Compression. arXiv preprint arXiv:2210.13438. https://arxiv.org/abs/2210.13438
  8. [11]Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: A Speech-Text Foundation Model for Real-Time Dialogue. arXiv:2410.00037 [eess] doi:10.48550/arXiv.2410.00037.
  9. [13]Pif Edwards, Chris Landreth, Mateusz Popławski, Robert Malinowski, Sarah Watling, Eugene Fiume, and Karan Singh. 2020. JALI-Driven Expressive Facial Animation and Multilingual Speech in Cyberpunk 2077. In ACM SIGGRAPH 2020 Talks. ACM, Virtual Event USA, 1–2. doi:10.1145/3388676.3407339DOI
  10. [14]Erik Ekstedt and Gabriel Skantze. 2020. TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2981–2990. https://arxiv.org/abs/2010.10874
  11. [15]Erik Ekstedt and Gabriel Skantze. 2022. Voice Activity Projection: Self-supervised Learning of Turn-taking Events. In Proc. Interspeech 2022. 5190–5194. https://arxiv.org/abs/2205.09812
  12. [16]Yoshinori Fukunaga, Ryota Nishimura, Kengo Ohta, and Norihide Kitaoka. 2025. Backchannel Prediction for Natural Spoken Dialog Systems Using General Speaker and Listener Information. In Interspeech 2025. ISCA, 1078–1082. doi:10.21437/Interspeech.2025-1316
  13. [17]John Heritage. 1984. A change-of-state token and aspects of its sequential placement. In Structures of Social Action: Studies in Conversation Analysis, J. Maxwell Atkinson and John Heritage (Eds.). Cambridge University Press, 299–345. doi:10.1017/CBO9780511665868.020DOI
  14. [18]Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhtia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv:2106.07447 [cs] doi:10.48550/arXiv.2106.07447
  15. [19]Koji Inoue, Bing’er Jiang, Erik Ekstedt, Tatsuya Kawahara, and Gabriel Skantze. 2024. Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection. arXiv preprint arXiv:2401.04868. https://arxiv.org/abs/2401.04868
  16. [20]Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, and Xiangyang Li. 2025. MoonCast: High-Quality Zero-Shot Podcast Generation. arXiv preprint arXiv:2503.14345. https://arxiv.org/abs/2503.14345
  17. [21]Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech. In International Conference on Machine Learning (ICML). https://arxiv.org/abs/2106.06103
  18. [22]Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2010.05646
  19. [23]Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, and Nima Mesgarani. 2023. StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2306.07691
  20. [24]Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, and Hung-yi Lee. 2025b. Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities. arXiv:2503.04721 [cs] doi:10.48550/arXiv.2503.04721arxiv.org/abs/2503.04721
  21. [25]Yuxin Lin et al. 2025a. Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals. In ACL. https://arxiv.org/abs/2505.12654
  22. [26]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (Sept. 2019).arxiv.org/abs/1907.11692
  23. [27]Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. 2024. Language Model Can Listen While Speaking. arXiv:2408.02622 [cs] doi:10.48550/arXiv.2408.02622
  24. [29]Akira Morikawa, Ryo Ishii, Hajime Noto, Atsushi Fukayama, and Takao Nakamura. 2022. Determining Most Suitable Listener Backchannel Type for Speaker’s Utterance. In Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents (IVA ’22). Association for Computing Machinery, New York, NY, USA, 1–3. doi:10.1145/3514197.3549619
  25. [30]Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2022. Generative Spoken Dialogue Language Modeling. Transactions of the Association for Computational Linguistics (2022). https://arxiv.org/abs/2203.16502
  26. [31]Toshiki Onishi, Naoki Azuma, Shunichi Kinoshita, Ryo Ishii, Atsushi Fukayama, Takao Nakamura, and Akihiro Miyata. 2023. Prediction of Various Backchannel Utterances Based on Multimodal Information. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents (IVA ’23). Association for Computing Machinery, New York, NY, USA, 1–4. doi:10.1145/3570945.3607298
  27. [33]Pixel Audio Inc. 2026. Pixel Audio: Full Spectrum Audio Design & Immersive Sound Design. https://pixelaudio.ca/. Accessed: May 9, 2026.
  28. [34]Kevin Redding. 2022. How to Become a Video Game Voice Actor. https://www.backstage.com/magazine/article/video-game-voice-actor-guide-75034/.
  29. [35]Andrew G. Reece et al. 2023. The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation. Science Advances (2023). doi:10.1126/sciadv.adf3197DOI
  30. [36]Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2021. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2006.04558
  31. [37]Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. FastSpeech: Fast, Robust and Controllable Text to Speech. In Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/1905.09263
  32. [38]Resemble AI. 2025. Chatterbox-TTS. https://github.com/resemble-ai/chatterbox. GitHub repository.
  33. [39]Edward Reynolds and Johanna Rendle-Short. 2011. Cues to Deception in Context: Response Latency/Gaps in Denials and Blame Shifting. The British Journal of Social Psychology 50, 3 (Sept. 2011), 431–449. doi:10.1348/014466610X520104DOI
  34. [40]Rajarshi Roy, Jonathan Raiman, Sang-gil Lee, Teodor-Dumitru Ionescu, Robert Kirby, Sungwon Kim, Jaehyeon Kim, and Bryan Catanzaro. 2026. PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models. arXiv:2602.06053 [cs] doi:10.48550/arXiv.2602.06053
  35. [42]Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. 2018. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://arxiv.org/abs/1712.05884
  36. [43]Kai Shen, Ziqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2024. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2304.09116
  37. [44]Tanya Stivers, N. J. Enfield, Penelope Brown et al. 2009. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences 106, 26 (2009), 10587–10592. doi:10.1073/pnas.0903616106
  38. [45]Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. 2021. A Survey on Neural Speech Synthesis. arXiv preprint arXiv:2106.15561. https://arxiv.org/abs/2106.15561
  39. [46]Gemini Team. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CV] https://arxiv.org/abs/2312.11805
  40. [47]Silero Team. 2024. Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier. https://github.com/snakers4/silero-vad.
  41. [48]Jackson Tolins and Jean E. Fox Tree. 2014. Addressee backchannels steer narrative development. Journal of Pragmatics 70 (2014), 152–164. doi:10.1016/j.pragma.2014.06.006DOI
  42. [49]Carol Trimboli and Michael B. Walker. 1984. Switching pauses in cooperative and competitive conversations. Journal of Experimental Social Psychology (1984). doi:10.1016/0022-1031(84)90027-1DOI
  43. [50]Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. WaveNet: A Generative Model for Raw Audio. In 9th ISCA Workshop on Speech Synthesis (SSW9). https://arxiv.org/abs/1609.03499
  44. [51]Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. 2024. Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yasel Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 21390–21402. doi:10.18653/v1/2024.emnlp-main.1192DOI
  45. [53]Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, Andreas Stolcke, and Venkatesh Ravichandran. 2024. Turn-Taking and Backchannel Prediction with Acoustic and Large Language Model Fusion. arXiv:2401.14717 [cs] doi:10.48550/arXiv.2401.14717arxiv.org/abs/2401.14717
  46. [54]Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Irwin King, Liang Chen, and Peilin Zhao. 2025. NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction. arXiv:2506.00975 [cs] doi:10.48550/arXiv.2506.00975
  47. [55]Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zhongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017. Tacotron: Towards End-to-End Speech Synthesis. In Proceedings of Interspeech. https://arxiv.org/abs/1703.10135
  48. [56]Zhifei Xie and Changqiao Wu. 2024. Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming. arXiv:2408.16725 [cs] doi:10.48550/arXiv.2408.16725DOI
  49. [57]Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jincheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. 2025. Qwen3-Omni Technical Report. arXiv:2509.17765 [cs] doi:10.48550/arXiv.2509.17765DOI
  50. [58]Shu-wen Yang, Po-Han Chi, Yun-Sung Chuang, Cheng-I. Jeff Lai, Kushal Lakotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2021. SUPERB: Speech Processing Universal PERFormance Benchmark. arXiv:2105.01051 [cs] doi:10.48550/arXiv.2105.01051
  51. [60]Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot. arXiv:2412.02612 [cs] doi:10.48550/arXiv.2412.02612DOI
  52. [61]Leyiang Zhang et al. 2025. CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching. arXiv preprint arXiv:2506.00885. https://arxiv.org/abs/2506.00885

Paper details

vlnc:W_vbwa663k
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents