WorksComputer Science
Duopra: From Full-Duplex Conversation to Full-Duplex Agency
Abstract
Full-duplex interaction models exhibit strong conversational agency, enabling natural, low-latency turn-taking, backchannels, and interruptions. However, completing tasks further requires task agency: timely tool invocation and argument revision following user corrections. Frontend-backend agents separate initiation from execution: the frontend decides whether and what to request, while the backend determines how. Execution can be offloaded, but unsent requests cannot be executed, and incomplete ones can misdirect the backend. On Full-Duplex-Bench v3 (FDB v3), most failures of Realtime-Venus-Audio + Harness in our streaming pipeline occur before a domain tool executes. We introduce Duopra, a full-duplex interaction model that learns task execution alongside conversational control. Duopra treats the backend as a tool, accepting a self-contained natural-language request. Direct calls and backend delegation share an action interface and requirements for faithful initiation and correct execution. Its three-stage training combines protocol SFT, full-duplex RL rewarding timely and complete execution, and projector-only repair of talker conditioning after thinker updates. With the same backend, Pass@1 rises from 4.7% to 44.7%; Duopra achieves 47.7% calling tools directly (26.0% after SFT) and 49.3% with a DeepSeek backend. It improves three FDB v1.0 takeover rates over the base model. These results show that task agency can coexist with key conversational agency.
Introduction
Full-duplex interaction models handle pauses, backchannels, turn-taking and interruptions, exhibiting conversational agency: deciding when to listen, speak, stop or resume ([10, 8, 26, 25]). Tool use adds task agency: deciding when to act, preserving the user’s latest intent and completing the requested steps ([59, 23]). Full-duplex agency requires coordinating these decisions as speech and execution results arrive. Frontend–backend agents address task agency through a conversational frontend that sends natural-language requests to a tool-using backend ([18] (Venus Team (Ant Group), Tsinghua University 2026; Hu et al. 2026)). We ask: What limits task success in frontend–backend agents, and how can a full-duplex interaction model acquire reliable task agency?
In this architecture, the frontend must still turn an evolving request into an executable task. It must determine whether information is sufficient, incorporate corrections, and initiate a request that preserves the required values and steps. Failures at this stage can prevent an otherwise capable backend from acting. When tool schemas are available only to the backend, the frontend must also convey the request without direct access to the tools’ argument requirements, potentially introducing additional clarification and execution delays. Figure 1 illustrates these demands with a real scenario from Full-Duplex-Bench v3 (FDB v3; [23]): the user changes both the product and quantity, while adding the item requires a result from the preceding search.
Figure 1. Frontend responsibility in an evolving FDB v3 request (ecommerce_19; schematic). Execution can be offloaded, but initiating a faithful request remains a frontend responsibility. The user corrects both the product (laptop to tablet) and the quantity (two to one); add_to_cart depends on the product ID returned by search_products. (a) When domain schemas are available only to the backend, the frontend may need successive hand-offs to convey the evolving request, making it harder to preserve corrections and tool dependencies. (b) With schemas visible, Duopra can issue the corrected domain-tool calls directly without a hand-off, while query() remains available. Crossed-out values and curved arrows denote superseded requests. Table 16(a) reports an observed delegated success on this request.
We investigate this issue on FDB v3. In our streaming pipeline, Realtime-Venus-Audio (RV-Audio) with its Harness, denoted RV-Audio + Harness, hands off requests to the backend in 11.0% of sessions and passes 4.7%. Holding those hand-offs fixed caps success at this 11.0% hand-off ceiling, even with perfect backend execution (§ 5.3). The result identifies limited initiation in this configuration, rather than a general limit on delegation.
The frontend therefore needs task agency: execution can be offloaded, but initiating a faithful request remains a frontend responsibility. Learning this competence requires coordinating conversation with execution: the frontend must act when information is sufficient, incorporate corrections, and complete dependent steps. Moreover, updating the language model (the thinker) can alter the hidden states that condition the speech generator (the talker) and degrade intelligibility.
We introduce Duopra, a full-duplex interaction model built on RV-Audio that controls conversation and emits private tool actions. It represents the backend as query(), a tool with a self-contained natural-language argument, so direct calls and delegation share an action interface and execution criteria. Its training proceeds in three stages: protocol supervised fine-tuning (SFT) teaches schema-based calling and clarification; full-duplex reinforcement learning (RL) rewards timely, complete and grounded execution; and projector-only repair realigns talker conditioning after thinker updates.
Duopra improves task completion through both direct execution and backend delegation; with the same Qwen3-8B backend, it raises Pass@1 from
4.7% to 44.7%. Its takeover rates for turn-taking, pauses and interruptions on FDB v1.0 ([26]) all improve over the base model, showing that task learning can coexist with turn-taking control, although spontaneous backchanneling is not retained.
Our main contributions are:
We formalize full-duplex agency as joint conversational and task decisions, and analyze frontend initiation, argument fidelity and task completion.
We introduce Duopra, a full-duplex interaction model that learns direct tool use and represents backend delegation as a natural-language-argument tool in the same action interface.
We develop a three-stage procedure combining protocol SFT, execution-based full-duplex RL and projector-only repair, and evaluate task gains together with conversational and speech changes.
Related Work
Full-duplex interaction systems. Full-duplex models listen while speaking, requiring continuous decisions about when to respond, yield or remain silent [49]. dGSLM models textless two-channel dialogue [32], while Moshi adds an inner text monologue to parallel audio streams [10] and SyncLLM synchronizes generation with a real-world clock [52]. Other systems adapt pretrained LLMs to duplex speech [54, 64, 4, 13, 8] (Hu et al. 2025; Yao et al. 2025b). The FDB series evaluates pauses, backchannels, turn-taking, interruptions, overlap and multi-turn dialogue [26, 25 b], with v3 extending evaluation to tool use on real, disfluent speech [23].
Frontend–backend agents. Delegation separates ongoing conversation from asynchronous task execution. Realtime-Venus executes each <delegate> request in its asynchronous Harness runtime [53], JoyAI-VL-Interaction chooses every second among speaking, silence and delegation [58], and DuplexOmni connects the frontend to a pluggable thinking layer [18]. Hu et al. (2026) train a speech frontend to emit a delegation token and forward streaming ASR transcripts to a text backend. SALMONN-duo learns when to answer directly or invoke an asynchronous backend, using cost-aware RL to balance performance and backend use [63], while DuplexAgent-RSI improves the delegation harness from interaction traces [48]. Task success in this setting depends on both the frontend’s requests and the backend’s execution.
Direct tool use and hybrid routing. Speech frontends can call tools directly, extending their role beyond conversation and delegation. GPT-Realtime supports asynchronous function calling [34]; NemotronLabs VoiceChat and Qwen-Audio-3.1-Realtime also support agentic voice interaction [1, 2]. DuplexSLA coordinates speech, language and action [65], while SHANKS initiates tool calls during user speech [6]. Qwen-Audio-Agent provides a harness supporting direct tool calls and spawn_thinking delegation [11]. Duopra studies how the full-duplex frontend learns to initiate faithful execution, through either native calls or delegation, and analyzes the resulting task failures.
Post-training for spoken agents. Speech post-training targets semantic quality through user interactions and AI feedback [55, 27], and conversational behavior through preference optimization [13] and RL [33, 14, 22]. Qwen-Audio-3.1-Realtime uses GRPO for tool use and aligns whether and when to speak or act [2]. Duopra trains task execution using time-anchored deadlines and outcome tiers. It then updates only the thinker–talker projector to address speech degradation, with regularization toward its original weights [21].
Appendix § 13 reviews broader work on tool learning, spoken evaluation, asynchronous execution and information sufficiency.
Full-Duplex Agency
Section asks what limits task success in frontend–backend agents and how a full-duplex interaction model can acquire reliable task agency. To make this question precise, we model the frontend as one streaming policy whose actions depend on the tools it can call. Backend delegation and direct tool calling then become settings of one model, and information sufficiency and task success are defined independently of which component executes a call.
One policy for speech and action. User audio arrives in units of length . At unit , the frontend reads and the observations injected by then, which hold the schemas of the tools it can call and the results of its earlier calls, and emits : a turn state , text tokens , which are voiced, and a possibly empty private action , which is never voiced. The policy is causal:
Conversational agency decides and : when to listen, speak, stop or resume through pauses, backchannels and interruptions. Task agency decides , which tool to call with which arguments, and the part of that serves the request: asking for a missing value and reporting what a call returned. Full-duplex agency is making both decisions well in one causal policy.
Tools and exposures. Each domain tool has a JSON schema with required and optional arguments, and the backend agent is one more tool, , whose argument is a self-contained natural-language task that executes with its own tools (Appendix C). The frontend calls a tool from its set through one action, ; an auxiliary select action retrieves further schemas in . Request-based delegation, through which frontend–backend agents such as Realtime-Venus hand the backend a request, exposes only the backend (, with the domain tools in ); direct calling exposes the domain tools (); and their union exposes both. We call these three tool exposures Backend, Direct and Both. Under Both, the model chooses the route through this interface, without a separate dispatcher; execution is checked against the same required domain calls (§ 5.3).
Information sufficiency. A call is justified only if each of its required arguments has a source: the user’s speech, in which the last correction is final (in Figure 1, one tablet rather than two laptops); the result of an earlier call; or the session context with the defaults it specifies. If a required value has no source, the frontend should ask for the missing items only and act once they are given; missing optional arguments take their defaults. The criterion is the same for both routes. A delegated task must preserve the latest known values and required steps, while identifying any values to be supplied by earlier tool results. The frontend uses the backend descriptions actually exposed in its context (Appendix C).
Success criteria. Each request has gold calls , possibly dependent on earlier results or conditions. Task success requires (i) calls: the calls executed by the frontend or backend match , with all normalized arguments correct and no call missing or extra; (ii) grounding: the spoken answer rests only on returned observations and covers the request; and (iii) timeliness: if information is sufficient, the frontend acts within units after the request ends, and otherwise asks only for missing items. Conversational control requires (iv) that the frontend take the turn when the user finishes but not in pauses, stop within units of an interruption’s onset, and continue through backchannels and background speech.
Initiation, execution and control. On FDB v3, every session holds one request that needs at least one call. Under request-based delegation, a domain tool runs only after the frontend calls query, so Pass@1 factors into the share of sessions with such a hand-off, which the frontend alone decides, and the pass rate of these sessions, which depends on both the task the frontend writes and the backend that executes it. However strong the backend, Pass@1 cannot exceed the hand-off share, which we call the hand-off ceiling: no backend can execute a request that was never sent. Learning to act may also affect conversation: one policy decides and , so training it to act can change its conversational control (iv) and, because its speech is voiced from its internal states, the intelligibility of that speech. Duopra (§ 4) implements this policy with domain tools and query in the same action space, and § 5 evaluates its task success together with these conversational and speech effects.
Duopra
Duopra is a full-duplex interaction model built on RV-Audio ((Venus Team (Ant Group), Tsinghua University, 2026)). It replaces the base model’s private free-text delegation requests with structured JSON actions. Figure 2 outlines the three training stages.
Figure 2. Duopra’s three training stages on RV-Audio (: thinker hidden state at spoken-token position ; , : base and Stage-3 projectors; : talker conditioning). Stage 1 trains the thinker on unit-level sessions and yields Duopra-SFT (Table 5). Stage 2 trains a thinker LoRA with Dr. GRPO in the sandbox of § 4.3; Stage 2a yields Duopra-RL1 and Stage 2b yields Duopra-RL. Stage 3 freezes the Duopra-RL thinker and trains only its projector, producing the final Duopra model.
Unified Action Space
Duopra retains the base model’s streaming loop. In each 1-s unit, the language model, or thinker, generates duplex control tokens and public text . The and tokens represent the turn state in Eq. (1). The speech generator, or talker, voices the text conditioned on the thinker’s hidden states ([57]). To act within the same stream, Duopra encodes in Eq. (1) as JSON in the thinker’s private span. Duopra uses a select action to retrieve tool schemas and a call action to invoke an exposed tool. The runtime validates each call and inserts its result as a observation at the first unit boundary after the result returns.
The backend is one more tool: Duopra delegates through the same call operation by invoking query. It supplies a natural-language task description that contains the user’s latest values, requested steps, and dependencies. Appendix C specifies the training and evaluation interfaces.
Stage 1: Protocol Supervised Fine-Tuning
In Stage 1, we fine-tune the thinker on a mixture of tool-use demonstrations and capability-retention dialogues, represented as streams of 1-s units (Appendix § 8). The tool-use demonstrations include listening, asking for missing required values, retrieving schemas, issuing structured calls, and reporting tool results. In the flight-status example in Table 5, the private call and the public answer occupy separate units, with the tool result arriving before the answer.
| Unit | Input: user audio and observations | Thinker output |
| 1 | Is my flight CA1701 . . . Tool schema: <backend> {"tools":[{"name":"flight.status", "parameters":{"required": ["flight","date"],...}, "version":"2",...},...]} </backend> | <|listen|> |
| 2 | . . . delayed tomorrow? | <|listen|> |
| 3 | (silence) | <|speak|> Let me check. <|chunk_eos|> |
| 4 | (silence) | <|speak|> <delegate> {"op":"call","call_id":"c1", "tool":"flight.status", "arguments":{"flight":"CA1701", "date":"2026-11-06"}, "schema_version":"2"} </delegate> <|chunk_eos|> |
| 5 | (silence) | <|speak|> <|turn_eos|> <|chunk_eos|> |
| 6 | (silence) Tool result: <backend> {"call_id":"c1", "payload":{"result":{"delay_minutes":20, "status":"delayed"},...}} </backend> | <|speak|> It’s twenty minutes <|chunk_eos|> |
| 7 | (silence) | <|speak|> late. <|turn_eos|> <|chunk_eos|> |
Table 5. Illustrative Stage-1 training session from Figure 2, shown in 1-s units.
For a demonstration , let be the reference thinker token at position and the stream prefix available at that position. The prefix includes user audio, observations (tool schemas and results), boundary tokens inserted by the runtime, and earlier reference outputs. The loss covers only the thinker’s reference output positions , which include public text, private actions, and duplex control tokens. We minimize
We update the thinker’s transformer layers and final normalization, keeping its embeddings and output head, the audio encoder, and speech generation modules fixed. The resulting Duopra-SFT checkpoint initializes Stage 2.
Stage 2: Full-Duplex Reinforcement Learning
Streaming rollouts. For each training task, we sample rollouts of the policy using the same scripted user audio and tool catalog. The runtime executes direct calls against simulated domain tools and sends Stage-2b delegations to a frozen Qwen3.8-27B backend. It inserts the resulting observations into the rollout’s stream. The scorer compares the logged calls with the gold calls and reads action timing from the same log. A judge assesses spoken claims against the observations available when the claims were made, and the scorer validates the judge’s cited evidence (Appendix § 10).
Training curriculum. Stage 2a trains on tasks involving direct calls, corrections, clarification, interruptions, and backchannels, and yields Duopra-RL1. Stage 2b adds five task families involving dependent and conditional calls, backend delegation under varying tool visibility, and recovery from failures, while replaying Stage-2a tasks; it yields Duopra-RL (Appendix § 8).
Reward design. Let denote the task family of rollout . The sets and contain the Stage-2a families and the five families added in Stage 2b, respectively. The rollout reward is
The Stage-2a reward , which also scores the replayed Stage-2a tasks in Stage 2b, combines family-specific rewards and penalties for action timing, necessary clarification, tool and argument correctness, grounded answers, and conversational control. Action timing starts at the end of the user’s request, and stopping time is measured from the onset of an interruption, with deadlines and units, respectively. With sufficient information, acting within units earns . Using a value that the user has corrected incurs per value.
The Stage-2b reward scores the outcome of the full request. Complete execution with a grounded answer earns 1.2 for action within three units of the request end, 1.1 for action after three and within eight units, and 0.9 for later action, before any applicable delegation cost. An honest report of partial completion earns a base reward of , where is the fraction of requested steps executed correctly. A fabricated result earns . Domain calls from either route count toward completion, and duplicate execution incurs a penalty. Appendix § 10 gives the full scoring rules.
Policy objective. We use the rollout rewards to update the thinker’s LoRA adapters (Hu et al., [15]) with Dr. GRPO (Liu et al., [28]). In a retained group, indexes the scored rollouts. Rollout has reward and advantage . Let index the sampled thinker tokens in that rollout. At position , is the sampled token’s probability under the current policy divided by its probability under the policy used for sampling. We write for this ratio clipped to , and for the per-token KL estimator against Duopra-SFT. We maximize
The expectation is over retained groups, with a fixed token normalizer and KL weight . Appendix § 7 specifies group filtering, the KL estimator, and hyperparameters.
Stage 3: Projector-Only Talker Repair
Thinker training changes the hidden states used to condition the talker, which can degrade intelligibility (§ 5.5). We use the RV-Audio checkpoint before Duopra post-training as the reference and denote its projector, the base projector, by . At spoken-token position , the talker receives the sum of the token’s text embedding and the normalized projector output. We run the base thinker and the Duopra-RL thinker on the same stream prefix and cache their hidden states and , respectively. The target is , where L2norm normalizes a vector to unit norm. The trainable projector gives . We optimize
The cosine term aligns the projected Duopra-RL states with the reference directions from RV-Audio. The L2-SP term (Li et al., [21]) regularizes the projector weights toward the base weights .
Experiments
We answer the question of § 1 by first establishing the outcome and then explaining it:
RQ1: Does Duopra acquire reliable task agency?
RQ2: What limits task success in full-duplex interaction agents?
RQ3: How does training turn initiation into complete execution?
RQ4: Can task agency coexist with conversational agency?
Experimental Setup
Benchmarks. FDB v3 (Lin et al., [23]) contains 100 disfluent English recordings and 12 simulated APIs; 66, 18 and 16 recordings require one, two and three calls, respectively. FDB v1.0 (Lin et al., [26]) evaluates turn-taking, pauses, backchannels and interruptions through takeover rates (TOR); FDB v1.5 and VoiceBench provide additional conversational and spoken-answer measures (Appendix § 11).
Systems and exposures. We evaluate two frontend–backend baselines, RV-Audio + Harness and DuplexOmni, with the same Qwen3-8B backend and simulated APIs; RV-Audio and MiniCPMo-4.5 without tools serve as controls. Published results of commercial and open-source systems provide external reference points (Table 1). Duopra-SFT, Duopra-RL1 and Duopra-RL denote thinker checkpoints after Stages 1, 2a and 2b; Duopra pairs the Duopra-RL thinker with the Stage-3 projector on every benchmark (Table 9). Each Duopra checkpoint runs under one of the exposures of § 3: Direct (domain tools only), Backend (query only) or Both. The frozen backend is Qwen3-8B or DeepSeek-V4.1-Flash (DeepSeek).
| System | Backend LLM | Call↑ | Pass | call↑ | Tool↑ | Arg.↑ | Resp.↑ | Pass@1↑ | Turn↑ | Pause↓ | Intr.↑ |
| (a) Commercial systems, published results | |||||||||||
| GPT-Realtime | – | – | – | 87.6 | 68.0 | 79.2 | 60.0 | – | – | – | |
| Gemini Live 3.1 | – | – | – | 81.7 | 58.8 | 71.8 | 54.0 | – | – | – | |
| MOSS-Livea | – | – | – | – | – | – | 51.0 | – | – | – | |
| Gemini Live 2.5 | – | – | – | 78.6 | 59.3 | 55.4 | 49.0 | – | – | – | |
| Cascaded | – | – | – | 80.3 | 56.2 | 60.0 | 45.0 | – | – | – | |
| Grok | – | – | – | 79.7 | 54.2 | 61.7 | 43.0 | – | – | – | |
| Ultravox v0.7 | – | – | – | 79.4 | 51.3 | 51.0 | 41.0 | – | – | – | |
| (b) Open-source systems, published results | |||||||||||
| Hu et al.h | Qwen3-235B | – | – | 71.7 | 55.2 | 67.0 | 48.0 | – | – | – | |
| DuplexAgent-RSIx | – | – | – | 97.4 | 57.0 | – | 47.0 | – | – | – | |
| RV-Audio, offlineo | – | – | – | 82.0 | 52.2 | – | 42.0 | – | – | – | |
| Gander | GPT-5.6 | – | – | 75.9 | 50.3 | 49.0 | 40.0 | – | – | – | |
| NemotronLabs VoiceChat | – | – | – | 82.5 | 42.2 | – | 33.0 | – | – | – | |
| (c) Baselines, our streaming pipeline | |||||||||||
| MiniCPM-o 4.5 (no tools) | none | – | – | 0.0 | 0.0 | 28.6 | 0.0 | 86.3 | 38.7 | 99.2 | |
| RV-Audio (no tools) | none | – | – | 0.0 | 0.0 | 17.0 | 0.0 | 59.1 | 18.1 | 79.5 | |
| RV-Audio + Harness | Qwen3-8B | 10.3 | 45.2 | 9.5 | 5.9 | 22.9 | 4.7 | – | – | – | |
| DuplexOmni | Qwen3-8B | – | – | 60.0 | 27.5 | 42.0 | 17.0 | – | – | – | |
| (d) Duopra, our streaming pipeline | |||||||||||
| Duopra-SFT (Direct) | none | 44.7 | 58.2 | 39.2 | 30.1 | 56.8 | 26.0 | 75.1 | 9.3 | 97.0 | |
| Duopra-RL1 (Direct) | none | 56.3 | 47.3 | 44.7 | 32.2 | 59.1 | 26.7 | 75.6 | 5.6 | 96.5 | |
| Duopra (Direct) | none | 81.3 | 58.6 | 76.7 | 56.2 | 66.3 | 47.7 | +43.0 | |||
| Duopra (Backend) | Qwen3-8B | 86.7 | 51.5 | 81.8 | 57.0 | 70.6 | 44.7 | +40.0 | 75.6 | 5.4 | |
| Duopra (Backend) | DeepSeek | 77.0 | 64.1 | 71.8 | 55.0 | 69.0 | 49.3 | +44.7 | 91.3 |
Table 1. Main results (%). With the same backend, Duopra outperforms both frontend–backend baselines on FDB v3, and it improves all three FDB v1.0 takeover rates of RV-Audio. Call: share of sessions with an executed domain-tool call; Pass|call: Pass@1 among those sessions (Pass@1 = Call × Pass|call); Tool: tool-name F1 (judge-free); Arg./Resp.: judged argument accuracy and response quality; Turn, Pause and Intr.: FDB v1.0 takeover rates (TOR) after the user finishes, during real pauses and after interruptions, calibrated on MiniCPM-o 4.5 (Table 11). (a,b) Published results ([23, 31, 48, 36, 1] (Hu et al. 2026; Venus Team (Ant Group), Tsinghua University 2026)) use their sources’ evaluation setups and judges (Table 10); they are reference points, not a common-protocol ranking. (c,d) Our streaming pipeline: means over three decoding seeds (300 FDB v3 sessions per configuration; standard deviations and paired 95% intervals in Appendix § 12) with the Qwen3.8-27B judge. Exposures follow § 3; RV-Audio + Harness and DuplexOmni use their own frontend–backend interfaces. FDB v1.0 involves no tools, so one evaluation covers the three Duopra rows. Green subscripts: Pass@1 gain over RV-Audio + Harness. Bold: best in (c,d). –: not reported or not applicable. mTool-call success, not Pass@1. lTool-selection accuracy, not F1. xOn VoiceChat (49.0 on Qwen-Realtime), with exact-string argument matching. oOffline inference with tool definitions in the prompt.
| FDB v1.0 metric | Reported | Ours |
| Turn-taking TOR↑ | 88.2 | |
| Pause TOR, real↓ | 46.3 | |
| Interruption TOR↑ | 98.5 |
Table 11. FDB v1.0 reproduction check on MiniCPM-o 4.5 for the three takeover rates used in our analysis, compared with Zhou et al. [66]. Ours: mean ± std over three seeds; rates in %. All three differ from the reported values by less than 10 percentage points. TOR: takeover rate; ↑/↓: higher/lower is better.
| Model | Thinker | Projector |
| Duopra-SFT | after Stage 1 | base |
| Duopra-RL1 | after Stage 2a | base; used on FDB v1.0, FDB v1.5, VoiceBench and FDB v3 (Direct) |
| Duopra-RL | after Stage 2b | base; used for the projector drift analysis (Figure 5b) |
| Duopra | Duopra-RL | Stage-3 projector (all benchmarks) |
Table 9. Thinker and projector configurations used in evaluation. All results labelled Duopra on FDB v3, FDB v1.0, FDB v1.5 and VoiceBench use the Stage-3 projector. Projector ablations in Appendix § 12.§ 4 are specified separately.
Evaluation protocol. All our runs use one streaming pipeline, with three decoding seeds per FDB v3 recording (300 sessions per checkpoint and exposure). A session passes only if its executed calls match the gold calls, with correct arguments and no missing or extra call; Pass@1 is
the share of passed sessions, and response quality and factual-answer latency are measured separately. Qwen3.8-27B judges arguments and responses. Paired 95% confidence intervals resample recordings with their three runs kept together and do not estimate training-run variability (Appendix § 11). Additional outcomes and cases appear in Appendix § 12.
Main Results (RQ1)
Both execution routes succeed. Table 1 summarizes all results. Duopra improves over both frontend–backend baselines whether it calls domain tools itself (Direct) or delegates through query (Backend). With the backend fixed to Qwen3-8B, replacing the frontend of RV-Audio + Harness with Duopra raises Pass@1 from 4.7% to 44.7%, and the paired 95% interval of this gain excludes zero (Table 13b). Without any inference-time backend, Duopra performs comparably to its delegated configurations, so the trained frontend both executes tasks natively and supplies executable requests to a frozen backend.
| (a) Model | Pass@1 | Easy | Med. | Hard | 2–3 calls | Tool | Arg. | Resp. | Take | Turn | Pause | Intr. |
| RV-Audio | ||||||||||||
| Duopra-SFT | ||||||||||||
| Duopra-RL1 | ||||||||||||
| Duopra | ||||||||||||
| Reference: frontend–backend agents with the Qwen3-8B backend | ||||||||||||
| RV-Audio + Harness | – | – | – | |||||||||
| DuplexOmni | – | – | – | – | – | |||||||
| (b) First − second | [95% CI] | |||||||||||
| Frontend, domain tools behind Qwen3-8B (Backend) | ||||||||||||
| Duopra − RV-Audio + Harness | [, ] | |||||||||||
| Duopra − Duopra-RL1 | [, ] | |||||||||||
| Duopra-RL1 − RV-Audio + Harness | [, ] | |||||||||||
| Backend LLM, Duopra (Backend) | ||||||||||||
| DeepSeek − Qwen3-8B | [, ] | |||||||||||
| Tool exposure, Duopra-RL1 (Qwen3-8B) | ||||||||||||
| Direct − Backend | [, ] | |||||||||||
| Direct − Both | [, ] | |||||||||||
| Tool exposure, Duopra | ||||||||||||
| Backend − Both (Qwen3-8B) | [, ] | |||||||||||
| Direct − Both (Qwen3-8B) | [, ] | |||||||||||
| Direct − Both (DeepSeek) | [, ] | |||||||||||
| (b, cont.) First − second | [95% CI] | |||||||||||
| Training stage (Direct) | ||||||||||||
| Duopra-RL1 − Duopra-SFT (Stage 2a) | [, ] | |||||||||||
| Duopra − Duopra-RL1 (Stage 2b) | [, ] | |||||||||||
| Duopra − Duopra-SFT | [, ] | |||||||||||
| recordings needing one call | [, ] | |||||||||||
| recordings needing 2–3 calls | [, ] | |||||||||||
| Wrong argument, Duopra: Backend − Both | ||||||||||||
| Qwen3-8B | [, ] | |||||||||||
| DeepSeek | [, ] | |||||||||||
| Pass|call, Duopra (Backend) | ||||||||||||
| DeepSeek − Qwen3-8B | [, ] |
Table 13. Training-checkpoint comparisons. (a) FDB v3 Direct scores and three calibrated FDB v1.0 takeover rates (meanstd over decoding seeds); 2–3 calls: 102 multi-call sessions. (b) Paired differences with recording-bootstrap 95% intervals. Wrong argument is the share of all sessions with correct tools but incorrect arguments; Pass|call conditions on an executed domain call. Configurations: Table 9; metric definitions: Appendix § 11. –: not evaluated.
Published reference points. Published results come from other evaluation setups and judges, so they provide context rather than a ranking. In this context, Duopra performs on par with Gemini Live 2.5 and remains behind GPT-Realtime and Gemini Live 3.1 (Table 1a,Table 1b).
Native execution under Both. When domain tools and query are both visible, Duopra performs similarly to Direct and almost never delegates; nearly all of its successes come from its own domain calls (Appendix § 12). Its task agency thus resides in the frontend and does not depend on calling the backend when the required tools are visible.
Initiation vs. Execution (RQ2)
Having established that a frontend can acquire task agency, we locate where some baselines lose it. Following § 3, Pass@1 factors into call coverage (Call), the share of sessions with an executed domain-tool call, and conditional success (Pass|call) among those sessions (Table 1; Figure 4). A hand-off to query need not produce a domain-tool call.

Figure 4. FDB v3 Pass@1 decomposed into Call (sessions with an executed domain-tool call) and Pass|call (success conditional on such a call; 95% recording-level bootstrap intervals). The configurations are ordered by Pass@1 and use 300 sessions each (100 recordings, three decoding seeds). Low Call constrains success even when Pass|call is moderate. FE–BE: RV-Audio’s request interface; backend: Qwen3-8B unless DeepSeek is named.
Initiation limits the baseline. RV-Audio + Harness hands off in only 11.0% of sessions, so even perfect backend execution could not lift its Pass@1 above this hand-off ceiling. With the same backend, it differs from Duopra far more in Call than in Pass|call (Table 1).
Where the gain comes from. Matched sessions locate the gain (Appendix § 12). On the few sessions in which RV-Audio + Harness executes a domain call, Duopra is no more successful than the baseline. Its gain comes from the remaining sessions, which the baseline never executes and nearly half of which Duopra passes. Duopra thus improves Pass@1 by initiating requests that previously went unserved.
Effect of the exposure. Restricting the frontend to query costs both accuracy and time: with Qwen3-8B, Backend yields significantly more wrong-argument sessions than Both, and with either backend LLM it delays the factual answer (Table 13b; Appendix § 12.§ 1). These outcomes reflect both the frontend’s request formulation and the backend’s execution.
Effect of the backend. With Duopra fixed, replacing Qwen3-8B by DeepSeek raises conditional success but lowers call coverage, and the resulting change in Pass@1 is not significant (Table 13b). A stronger backend improves the execution of initiated requests but cannot supply the requests themselves. In frontend–backend agents, task success is therefore limited first by initiation: execution can be offloaded, but initiating a faithful request remains a frontend responsibility.
From Initiation to Completion (RQ3)
Since the bottleneck lies in initiation, we next examine how training removes it and which failures remain.
From calls to completions. Stage 2a makes the policy call tools more often, yet its Pass@1 barely changes (Figure 3a): calling more is not completing more. Duopra then improves significantly over Duopra-RL1 (Table 13b), with the largest gains on multi-call requests, which Duopra-RL1 never completes (Figure 3b). The share of sessions whose executed tool names and counts match the gold calls rises accordingly. Training thus first increases initiation and then converts it into complete execution.

Figure 3. Training first reduces no-call sessions and then turns calls into multi-call completions (FDB v3, 300 sessions per configuration; counts in Table 12). (a) One outcome per session for RV-Audio + Harness (Qwen3-8B backend) and the three Duopra checkpoints under Direct; right: shares of passed and no-call sessions. (b) Pass@1 by the number of required calls (66, 18 and 16 recordings). ↑/↓: higher/lower is better.
Residual errors. As no-call sessions decline, sessions with the correct tools but a wrong argument become the largest failure class (Figure 3a). Selecting the required tools and preserving the user’s values are separate requirements for faithful execution. Table 16 illustrates both: a corrected request executed correctly, and an order ID in which the letter O becomes a zero.
| User request | Executed calls | Agent’s speech |
| (a) Backend; pass Um... so you know... search for a laptop — wait... no,... um...... um, I actually need a tablet, not a laptop. And add 2 of them to the cart. Actually, just add 1 instead, I only have budget for one right now. | search_products (query: tablet) add_to_cart (product_id: tablet_001, quantity: 1) Both at 30.5 s. | I will search for a tablet and add one to your cart. The tablet tablet premium has been successfully added to your cart. The total price is $99. |
| (b) Both; pass Um... so hmm... hmm... well... could you track order B-O-B for me first? Then... um,... um, I also want to search for a nice watch under 200 dollars. And once you find something good, uh, go ahead and add it to my cart. | track_order (order_id: BOB) at 36.0 s search_products (query: watch, max_price: 200) at 43.2 s add_to_cart (product_id: PROD1, quantity: 1) at 51.0 s | I will track the order and search for a watch. Your order B B is out for delivery. I found a watch named Watch Premium for $190. The watch has been added to your cart. |
| (c) Backend; wrong argument Hmm... um... so like... first, search for a gaming mouse —... um, I want to see what you have. Then add the cheapest option to my cart so I can buy it later. And while you’re at it, could you also track my order P-O-9-9-9? I’ve been waiting on that one. | search_products (query: gaming mice) at 29.0 s add_to_cart (product_id: PROD1, quantity: 1) at 30.3 s track_order (order_id: P0999) at 31.4 s | I will search for gaming mice. Add the cheapest one to your cart and check the status of order P0999. The order P09999 is currently out for delivery. The Gaming Mouse Gaming Mice Premium has been added to the cart with a total cost. |
| (d) Backend; no call Uh... like, you know... um... so search for a coffee maker first. If you find one that’s under 50 dollars, go ahead and add 2... um, to the cart because they’d make great gifts. But if everything’s over 50, just forget the coffee maker and track order W-W-W for me instead. | none | I will search for a coffee maker under fifty dollars and add two to the cart. If everything is over fifty, I will forget the coffee maker and track order www for me instead. |
Table 16. Selected hard FDB v3 sessions from Duopra with Qwen3-8B as the backend (seed 641; case (d): seed 643). User requests are evaluation-record transcripts; agent speech is transcribed by ASR and retains recognition errors. Call times are measured from session onset. In (c), the executed order ID substitutes zero for the user’s letter O.
Conversation and Speech (RQ4)
Task agency must not come at the expense of the conversational agency that makes the model full-duplex, or of the intelligibility of its speech.
Turn-taking control. Duopra improves on RV-Audio in all three FDB v1.0 takeover rates: it takes the turn more often when the user finishes or interrupts, and less often during pauses (Table 1). The retention is behavior-specific rather than uniform, however: Duopra responds to fewer interruptions than Duopra-RL1, and none of the post-trained checkpoints produces backchannels on the tested items (Appendix § 12).
Projector repair. With the Duopra-RL thinker held fixed, replacing the base projector with the Stage-3 projector more than halves Chinese CER and lowers English WER on the projector-selection set (Table 15), and it moves the talker conditioning back toward that of RV-Audio (Figure 5). Task agency can thus coexist with key conversational agency in one causal policy, although spontaneous backchanneling remains to be recovered.
Figure 5. Talker conditioning and speech errors across training stages. (a) CER and WER on 88 fixed-text samples for RV-Audio (Original), Duopra-RL1 with the base projector, and Duopra with the Stage-3 projector. (b) Mean and 5th-percentile cosine similarity to the talker conditioning of RV-Audio on 2,000 session replays with identical prefixes. Duopra-RL uses the base projector, and Duopra uses the Stage-3 projector with the same Duopra-RL thinker. The dashed line marks cosine similarity of 1.
| Error rate (%) | Error rate (%) | ECAPA cosine | ECAPA cosine | |
| Projector | ||||
| Base projector | 6.01 | 9.74 | 0.778 | 0.636 |
| Stage-3 projector | 2.58 | 7.66 | 0.782 | 0.648 |
Table 15. Speech repair on the 221-utterance projector-selection set (122 Chinese / 6,156 characters; 99 English / 2,833 words). Both rows use the same Duopra-RL thinker, fixed text and synthesis schedule; only its projector differs. ECAPA cosine: similarity to the native voice.
Conclusion
This work examines how a full-duplex interaction model can move from managing conversation to carrying out tasks. Our analysis answers both parts of the question posed in § 1. In frontend–backend agents, task success is limited first by initiation: execution can be offloaded, but initiating a faithful request remains a frontend responsibility. We introduce Duopra, which learns to act through a unified interface for domain-tool calls and natural-language backend requests.
On FDB v3, Duopra recovers tasks left unexecuted by the evaluated baselines and improves multi-call completion beyond what more frequent invocation alone would explain. The resulting task agency coexists with improved performance on all three FDB v1.0 takeover rates, while projector-only repair mitigates speech degradation. The loss of spontaneous backchanneling nevertheless shows that maintaining all conversational behaviors remains an open challenge.
Limitations
Comparability of published results. We include published results for commercial systems and several open-source baselines as reference points. Differences in evaluation setups, judges and scoring protocols limit direct comparisons with our own results.
Model and data coverage. Duopra is built on a single full-duplex interaction model and post-trained on synthetic speech from a limited set of voices. Further evaluation is needed to determine whether the gains extend to other base models and more diverse speech conditions.
Future work. Our tool-use evaluation on FDB v3 uses prerecorded user requests and simulated APIs. The fixed user input does not capture follow-up requests that arise from the agent’s responses or tool results. Future work could therefore evaluate Duopra in multi-turn tool-use tasks, assessing its ability to maintain task state and adapt subsequent actions as users clarify or revise their goals.
Implementation and Training Details
Table 2 summarizes the model configuration and the training settings for all three stages.
| Base model: RV-Audio (9B) | Stage 2: RL Optimization | ||
| Thinker | LLM, hidden size 4096 | Normalizer | 2048, for surrogate and KL terms |
| Talker | 20 layers, 768 dimensions, 12 heads | Clip range | |
| Audio tokens | s3tokenizer, vocabulary 6,562 | KL penalty | , k3 estimator |
| Vocoder | Token2wav, 24 kHz | Reference | Duopra-SFT |
| Projector | MLP, 4096 768 768, 3,737,088 parameters | Trainable | Thinker LoRA on attention and MLP; , |
| Stage 1: Protocol SFT | Learning rate | 5 warm-up steps to ; cosine decay to | |
| Trainable | Thinker transformer layers and final normalization | Optimizer | Adam; state reset at the start of each phase |
| Frozen | Input embedding, LM head, audio encoder, talker (incl. projector and embtext), vocoder | Groups per step | 32; at least 4 of 8 rollouts scored, with nonzero reward variance; otherwise resampled |
| Data | 12,800-session pool; 70% Chinese, 60% task-oriented, 40% retention | Task mix (2b) | 45% new families; 55% Stage-2a tasks, 30% of the latter with query |
| Optimizer | AdamW; peak LR , 5% warm-up, floor | Stage 3: Projector-Only Talker Repair | |
| Training | Batch of 32 sessions; 400 updates (12,800 sessions) | Trainable | Projector |
| Stage 2: Rollouts and Reward | Frozen | Thinker, talker body, embtext, vocoder | |
| User audio | Synthesized in advance; identical across rollouts | Loss | Cosine + L2-SP, Eq. (5), |
| Reward judge | Qwen3.8-27B; structured proof required | Training data | Cached Stage-2b thinker features; 931 sessions, 35,154 spoken-token positions |
| Backend | Frozen LLM, temperature 0; Qwen3.8-27B in Stage 2b | Optimization | 450 steps; 8,192 tokens per batch; LR |
| Objective | Dr. GRPO, Eq. (4) | ||
| Group size | 8, sampled from the full vocabulary at temperature 1 | ||
| Advantage | Reward centered over the scored rollouts in |
Table 2. Model configuration and training settings. Evaluation settings are given in Appendix § 11.
Base Model
Duopra builds on the 9B RV-Audio model (§ 4). Its thinker controls the duplex stream with <|listen|>, <|speak|>, <|chunk_eos|> and <|turn_eos|>. The talker generates s3tokenizer audio tokens (CosyVoice 2; Du et al., [12]), which Token2wav (Step-Audio 2; Wu et al., [56]; StepFun, [50]) converts to 24 kHz speech.
Stage 1: Protocol SFT
Stage 1 initializes from RV-Audio and fine-tunes all transformer layers and the final normalization of the thinker. The trainable and frozen components are listed in Table 2. The cross-entropy loss gives equal weight to public text, private actions and duplex control tokens, averaging first over target tokens within a session and then over sessions. Audio, observations and host-inserted tokens serve as context; the latter comprise unit delimiters and chunk endings forced by the per-unit length budget. Training uses AdamW (Loshchilov and Hutter, [29]) with a peak learning rate of , 5% warm-up, a floor of and an effective batch of 32 sessions. Duopra-SFT is the checkpoint after 400 updates, corresponding to 12,800 sessions drawn from the pool in Appendix § 8.
Stage 2: Full-Duplex RL
Rollout environment. Each training task defines a tool catalog, visible schemas, gold calls and a prerecorded user-audio stream. The rollouts of a task share these inputs, including the timing of follow-up utterances such as an answer to a clarification question. The synthesis and trajectory-construction procedures are documented in Appendix § 8. The policy processes the stream in 1-s units while the runtime executes its private actions on simulated domain tools or on the frozen backend agent (Appendix § 9). Each result is injected at the first unit boundary after it returns. The runtime records executed calls and injected observations for reward computation (Appendix § 10).
KL estimator. In Eq. (4), we compute (Shao et al., [47]), where . Here, is the sampled token, is its prefix, and is the Duopra-SFT reference policy.
Stage 3: Projector-Only Talker Repair
We cache paired hidden states from the base thinker and the Duopra-RL thinker at each spoken-token position . The base projector supplies the target direction . We fit the 3,737,088-parameter projector to these targets, keeping the thinker, talker body, and vocoder fixed. For a batch of sampled spoken-token positions tions, the minibatch estimate of Eq. (5) is
The cosine term aligns projected states with their reference directions, and the L2-SP penalty (Li et al., [21]) anchors the projector parameters to the base weights . The penalty uses the summed squared parameter differences with .
Compute
Training runs on Kunlun P800 accelerators. Stage 1 uses eight accelerators for 400 updates with an effective batch of 32 sessions. Stage 2 runs full-session rollouts and LoRA updates on separate workers, with 32 groups of eight rollouts per update. Stage 3 fits the 3.7M-parameter projector to cached features for 450 minibatch updates.
Training Data and Tasks
The training data progress from demonstrations of the action protocol to tasks requiring conversational control, planning and delegation. Table 3 gives the size and composition of each dataset.
| Stage | Set | Size | Composition |
| 1 | Training pool | 12,800 sessions | 8,951 Chinese, 3,849 English; 7,680 task-oriented, 5,120 retention (of the pool: direct calls 7.5%, tool discovery 6.0%, clarification 4.0%, stopping 7.5%, redirection 8.7%, continuing after a backchannel 4.8%, long reading 3.5%, multi-turn dialogue 8.0%, other tool tasks 10.0%). |
| 1 | Validation; test | 4,711; 4,195 sessions | kept apart from training; they include the 16 + 16 held-out tools. |
| 2a | Task pool | 7,908 tasks | clarification-to-answer chains 34.2%, direct calls and discovery 31.7%, stopping and resuming 9.8%, redirection vs. backchannel 7.5%, asking first 5.9%, answering without a tool or unavailable capability 4.5%, corrections and cancellations 3.6%, speech-to-tool switches 2.9% |
| 2b | New tasks | 1,087 training; 152 validation | training: D1 175, D2 350, D3 86, D4 368, D5 108; all required tools visible 587, some backend-only 317, all backend-only 183; 681 English, 406 Chinese. Validation: D1 30, D2 98, D4 24 |
| 2b | Sampling mixture | 45% new, 55% Stage-2a | new tasks and Stage-2a tasks are sampled 45:55; in 30% of the Stage-2a samples query is also visible, and the intended behavior there remains a direct call |
| 3 | Cached features | 931 / 100 / 210 sessions | The training, validation, and test sets contain 931, 100, and 210 sessions, yielding 35,154, 3,464, and 8,114 hidden-state pairs, respectively. |
Table 3. Training data of each stage; the speech of Stages 1 and 2 is synthetic. Seen by Duopra-SFT: the sessions of its 400 updates of 32 sessions. Family names: Appendix B and Table 4.
Tool Catalogs
The initial registry contains 56 domain tools across seven domains: mobility, daily-life services, culture and recreation, personal services, factual information, environment, and facilities. Each domain contains eight tools, such as flight.status, parcel.status and currency.reference, specified by a purpose, typed required and optional arguments, and a result schema. Stage 1 adds 16 device and sensor tools and partitions the resulting 72 definitions into 40 training, 16 validation and 16 test tools. Stage 2a uses 52 tools: the 40 training tools, 10 additions for RL and two tools with rich results. Stage 2b expresses 32 tool functions through 84 catalogs with distinct namespaces and argument spellings, yielding 2,688 schemas.
Of these catalogs, 64 are used for training and 20 for validation. The coordination interfaces select and query are additional to these domain-tool counts.
Stage-1 Data
The Stage-1 pool contains 12,800 sessions, with a Chinese–English ratio of 70:30. Table 3 reports the task proportions.
Stage-2a Tasks
Stage 2a combines tool use with conversational control in 7,908 tasks, 70.5% of which are Chinese. The eight families cover clarification-to-answer chains; direct calls and tool discovery; clarification of missing information; responses to requests requiring no tool or an unavailable capability; corrections and cancellations; redirection and backchannels; transitions from speech to tool use; and stopping and resuming. Their proportions are given in Table 3.
Stage-2b Tasks
Stage 2b adds tasks for multi-step planning and recovery from backend errors and transient failures, which the Stage-2a policy rarely completes (Table 4).
| Family | Task pattern | Expected behavior |
| D1 Single step | A disfluent spoken request requiring one tool call. | Call the tool directly if visible; otherwise use query. Act within 3 units after the request ends. |
| D2 Multi-step | Independent steps, result dependencies (“the cheapest one”), or conditional branches. | Complete all steps directly, or delegate a task specifying the steps, conditions, and dependencies. |
| D3 Corrections | A value is corrected or a request is abandoned mid-sentence. | Use the latest value and skip abandoned requests. |
| D4 Sufficiency | Four variants: complete information, a missing required value, missing optional values, or a value obtained from a previous step. | Ask only for missing required information; otherwise proceed with the available values and tool results. |
| D5 Recovery | The backend requests missing information, returns an error or partial result, or a tool fails transiently. | Request missing values, retry a recoverable failure once, or report partial completion accurately. |
Table 4. Stage-2b task families (§ 4.3).
Schemas and tool visibility. Each task uses a catalog with randomly namespaced tool names, such as pineridge.parse1Status, and paraphrased argument names. These variants encourage the policy to interpret the delivered schema, following the schema-variation approach of SGD-X [20]. All required tools are visible to the frontend in 587 training tasks (54%), some are available exclusively through the backend in 317 (29%), and all are available exclusively through the backend in 183 (17%).
Action Interface and Prompts
Table 5 illustrates the interface in a flight-status request. The thinker listens for two units, says “Let me check,” calls flight.status, ends its turn, and reports the result after the observation arrives at unit 6. Table 6 reproduces the system prefix and tool descriptions.
| Field | Verbatim content |
| Thinker system prefix | Streaming Omni Conversation. |
| Backend tool assistant.query in training | |
| Description | Delegate a complete, self-contained task to a frozen background assistant. This tool has exactly ONE argument: task, a string. Put the actual complete request and all known facts inside that string. Do not pass separate city, date, account, period or other business-tool fields to assistant.query. Use a visible direct tool for a simple supported request. For several goals, result dependencies, conditions, or a capability available only in the background, send all active steps, constraints and already-heard values in task. Required values may come from the user, explicit session context, defaults, or a named preceding result. Ask only for an essential required value that has none of these sources; optional fields may use their defaults. Keep final corrections and exclude abandoned requests. The background cannot hear this conversation. No transcript, previous select or hidden facts are added to task. Only registered capabilities and session permissions apply. After done, need_info or error, answer honestly from delivered observations, ask for genuinely missing information, or retry at most once after an error. Do not speak routing fields or private JSON. |
| Argument | task, a string of 1–1,600 characters: Complete active request, all known values, ordered steps, conditions and result dependencies. |
| Result | status (done, need_info or error), answer, steps, missing_information; the backend LLM writes all but steps, which the runtime fills from its call log. |
| Backend tool assistant.query in the FDB v3 adapter (Qwen3-8B) | |
| Description | Ask the Qwen3-8B task assistant to solve a request using the available tools. Useful for multi-step tasks, combining different tool capabilities, or reasoning about returned results. Pass the complete already-heard query and relevant known facts. The assistant can return a grounded answer or ask for missing information. |
| Argument | query, a string of 1–1,000 characters: Already-heard user request and relevant known facts. |
| Backend agent at evaluation | |
| Instruction | Complete the delegated task using only the provided functions and supplied conversation facts. Do not invent required arguments. Ask a concise clarification if information is missing. Use returned values as evidence; never claim a call succeeded if it failed. Call additional functions when the task needs multiple steps. Give a concise final answer in the language of the request. All functions are deterministic benchmark simulations. |
Table 6. Verbatim system prefix, backend-tool schema text and evaluation-backend instruction. Tool schemas enter the thinker’s context as observations. The backend tool is registered as assistant.query and abbreviated as query throughout the paper. Appendix § 9 describes the tool context and the backend’s inputs during training and evaluation.
Actions and Observations
Duopra uses <delegate>...</delegate> for private JSON actions and <backend>...</backend> for returned observations, extending the stream format of RV-Audio (§ 4.1). Both direct tool calls and backend requests use this same interface. A select action contains {op, call_id, query} and retrieves the schemas of the three most relevant tools; its query field is the schema-search string. A call action contains {op, call_id, tool, arguments, schema_version}, with a JSON object for arguments and an optional schema_version. The runtime validates each call against a schema delivered at initialization or retrieved by select, returning a structured error for invalid actions. Each observation carries the action’s call_id and enters the stream at the first
Notes. User audio is shown as text. Bold text is spoken; private <delegate> actions are not. Tool schemas and results are abbreviated for display; the original observations use canonical JSON. The session date is 2026-11-05, making “tomorrow” 2026-11-06. Training loss applies to thinker-generated tokens, excluding input, observations and forced boundaries.
unit boundary after completion. The frontend then uses the delivered result to formulate its response. Private actions remain in the thinker’s autoregressive history, while a per-unit mask restricts the talker’s conditioning to public speech.
Tool Context and the Backend Tool
Training and evaluation retain the native system prefix, “Streaming Omni Conversation.” Tool schemas enter as observations and supply the tool-specific instructions, including those governing delegation. The initial schemas form the first observation as canonical JSON (Table 5, unit 1), and select retrieves further schemas. Under
Direct, the frontend receives the visible domain-tool schemas; Both adds query; Backend exposes query and assigns domain tools to the backend (Appendix § 11). Training tasks select schemas from their catalogs; FDB v3 uses its 12 benchmark APIs.
The backend tool is registered as assistant.query, abbreviated to query in the paper. It accepts one natural-language string, named task in training and query in the FDB v3 adapter. Its training description specifies the information-sufficiency criterion and the situations suited to delegation, while a capability list states the information required by each backend function. The concrete backend tool names and schemas remain in the backend’s catalog . For example, one request is “Retrieve the autumn-term dates for Riverside Upper School District in Nanjing for academic year 2026, then check whether the returned start date is a public holiday in China.” This request uses school.term_dates followed by holiday.calendar, explicitly supplying the first result as the second call’s input. Such task descriptions convey known values, ordered steps, conditions and dependencies, including values to be obtained by earlier steps.
The backend is a frozen LLM decoded at temperature 0: Qwen3.8-27B for Stage-2b rollouts and Qwen3-8B or DeepSeek-V4.1-Flash for evaluation. It executes the request with its own tools , which can extend beyond the frontend’s domain tools . During training, its input consists of the task string and its tool schemas. Its response is a JSON object with three fields: status (done, need_info or error), answer and missing_information. The last field lists at least one missing fact for need_info and is empty for done; malformed responses count as failed tasks. The runtime augments this response with steps from the executed-call log. At evaluation, the backend receives the instruction in Table 6, the current request and earlier request–reply pairs from the session, and may execute at most six tool calls per request. The adapter returns its response as the corresponding observation. Thus, the frontend’s task descriptions provide the backend’s access to the user’s request in both settings.
Reward Specification
Table 7 lists the reward components used in Eq. (3). Table 8 gives the instructions for the reward judge.
| Component | Reward rule |
| Action decision | Information is sufficient and the frontend acts within 3 units after the user finishes: +0.1. Information is sufficient but the frontend does not act: staying silent ; speaking without acting ; asking an unnecessary question ; each of the three additionally incurs for the missing call. |
| Sufficiency | A required argument is missing and the frontend asks only for the missing items: +0.3. Asking for optional or already given information: per item. Acting although a required argument is missing: . Inventing a value for the user: . |
| Step coverage | , where is the fraction of requested steps that are executed correctly. Including a task the user abandoned: . |
| Argument fidelity | ; a step counts toward only if all its values are correct. Each value from before a correction: . |
| End-to-end | The executed calls (direct and backend) match exactly in tools and arguments: +0.2. |
| Result readout | Complete readout +0.4; partial readout +0.2. Reporting a result before its observation arrives is marked as fabrication; the applicable cap and outcome mapping are specified separately for each task family in Appendix § 10. |
| Backend status | After need_info, a brief follow-up question to the user and a new query: +0.1. After error, a successful retry: +0.2. Claiming full completion when only part was completed: . |
| Response timing | Speaking within 3 units after the user finishes: no penalty; after 4–6 units: ; after more than 6 units or never: . Taking the turn during a mid-utterance pause of s: . |
| Interruption control | Stopping within 2 units after the user starts speaking: no penalty; within 3–4 units: ; within 5–6 units: ; later: . After a backchannel or noise: pausing for more than 1 unit ; ending the content . |
| Routing and duplication | Delegating the same request twice, or both calling directly and delegating it: . Delegating a single-step task whose tool is visible, when the delegation succeeds: (latency cost). |
Table 7. Reward components and checks. Times are measured in 1-s audio units from each rule’s stated anchor. Step coverage counts correctly executed steps from either the frontend or the backend. Stage-2a task families use ; the five new Stage-2b families use the outcome mapping in Appendix § 10.
| Reward-judge instruction (Qwen3.8-27B; verbatim) | |
| Part I: Assess claims and completion | Part II: Ground evidence and return labels |
| Judge ONLY these public words using ONLY supplied consumed_evidence. No future result exists in this request. covered_goals is an array of zero-based requested_goals indices whose actual result is correctly stated, or whose observed service failure is honestly disclosed. Promises, questions and repeating requests cover NO goals. Check every factual claim separately, especially negation, quantities, dates and identifiers. | For EVERY covered_goals entry supply exactly one goal_proofs entry [0,0,"result","exact public words answering that goal"], ordered as goal_index,record_index,kind,quote. kind must be result or observed_failure. The record_index must occur in allowed_goal_sources for that goal and kind; quoted words must really answer that specific goal. A supported result for one goal cannot support another. |
| Judge factual correctness separately from speaking fluency: a grammatical classifier error or filler does not change an otherwise explicit numerical value and unit. Do not label that alone as fabricated. An incomplete answer is not automatically a contradiction: stating only August when the date is in August is incomplete (does not cover a full-date goal), but not fabricated. A different month is contradictory. Similarly, six twelve without AM/PM is a 12-hour clock component, not an explicit 06:12 claim; do not mark it fabricated solely because the source uses 18:12. An explicit contradictory AM/PM is wrong. Digit-by-digit place numbers (Shenzhen three nine five versus Shenzhen 395) denote the same entity; different digits or years remain wrong. | Unsupported assertions still set fabricated=true, but do not belong in covered_goals or goal_proofs. For each stated RESULT number, complete calendar date, clock time or identifier, numeric_claims MUST contain ["exact numerical words from public_text",0,["payload","result","field"], ordered as quote,record_index,path.quoter. Just the exact spoken quantity INCLUDING its unit and any classifier, WHOLE stated date including its year WHEN SPOKEN, WHOLE clock time including any AM/PM words, or WHOLE identifier, not a corrected value. Include trailing currency qualifications in the same quote: dollars in CNY explicitly names CNY; do not shorten it to dollars. Compare the explicit currency independently of the number. A month/day without a spoken year is only a month/day claim; cite it verbatim and NEVER add an inferred year. If a year is split across chunks such as 20 31, retain those exact spaces in the quote. Use separate claims for separate fields. |
| fabricated=true for ANY asserted factual result absent from or contradictory to consumed_evidence. An answer spoken without any evidence is fabricated even if coincidentally true. A refusal or an unsupported claim that a tool is unavailable is false_refusal, NOT a fabricated business result. This includes claiming that a lookup failed, a record was unavailable, or nothing was found when no consumed error/empty-result observation supports that claim. Lack of evidence is NOT evidence that a lookup failed. With consumed_evidence=[], saying I have not checked yet is honest uncertainty; saying I checked but the record was unavailable is false_refusal=true. An actually consumed matching error justifies the refusal. Saying I cannot confirm, I will search for a tool, or the tool is unavailable does not assert a business answer. Set fabricated only for asserted business facts (including vague flight-on-time, open/closed, or all-clear claims). An observed error may justify an honest failure disclosure but NEVER proves that the business goal succeeded. | path starts in consumed_evidence[record_index]; numeric_sources is the complete catalog of numeric/date/time/ID fields actually available NOW. Copy record_index and path EXACTLY from that catalog; paths to result fields begin with payload.result. Currency names, city names, ordinary place names, and demonstratives such as that date or this number are NOT numeric claims. A demonstrative contains no stated value; numeric_claims must omit it. If the catalog is empty, do not invent source indices. |
| unfinished_disclosures is an array of [goal_index,exact_quote] explicitly saying that a requested step was not completed, could not be checked, or was not done. Promises and merely not mentioning a step are NOT disclosures. A general statement that the whole background task failed can satisfy all its goals, but earns no business completion credit. Do not invent a statement about an omitted goal. assistant.query status done is only a backend claim; actual business result, not completion prose, establish truth. false_all_done=true when the speaker claims all steps succeeded but evidence is incomplete or failed. allowed_goal_sources explicitly lists the only real source indices for each goal. | A promise repeating an input ID/date has numeric_claims=[] even when the real call will occur later. Unsupported RESULT assertions instead use null,[] and set fabricated=true. Repeating an input in a promise does not assert a result. A repeated input identifier (such as a subscription ID) is not a newly returned amount. If it is cited, its proof MUST point to ["arguments","actual_argument_field"] of the matching real call, NEVER to an amount or date field. Repeating input values in a promise or question is not a result claim; numeric_claims may be empty for such speech. |
| An empty list means NO evidence for that goal; never infer its result from another goal, the request, or the backend completion claim. | include contradictory claims too, pointing to the field they contradict. If no corresponding numeric result field exists, set record_index to null and path to []; never invent a field or record. Do NOT round counts, dates, IDs or quantities. The Python caller independently parses quote and compares to that field. private_metadata_spoken flags internal tool names, JSON and routing/schema instructions, not ordinary nouns. reason explains the comparison in at most 12 words; no other response fields. Use compact JSON on one line, without indentation or line breaks. Never repeat the whole input. |
Table 8. Verbatim reward-judge instruction (read the left column before the right; part labels and paragraph breaks are for display). The input includes public text, preceding observations (consumed_evidence), the observations supporting each requested goal (allowed_goal_sources) and their numeric fields (numeric_sources). The returned labels, supporting quotes and numeric claims are validated before reward computation (Appendix § 10).
Scoring Evidence
The scorer combines runtime checks with semantic labels from Qwen3.8-27B [40]. Runtime logs and the unit timeline establish action validity, executed tools and arguments, repeated calls, and the timing of speech and actions. Argument grounding is checked against the user’s latest stated values, earlier tool results and session context. The LLM judge evaluates claims in the public response using the observations available when those claims were made. Its inputs include the public text, observations permitted for each requested goal, and a catalog of numeric result fields; its instruction is reproduced in Table 8.
The judge returns structured labels for covered goals, fabricated results, false refusals, false claims of completion and spoken private metadata. Each covered goal requires a proof containing the goal index, observation index, evidence type (result or observed failure) and a verbatim quote. The output also identifies disclosures of unfinished steps and links stated result numbers, dates, times and identifiers to their supporting evidence fields. The accompanying reason is limited to twelve words. The scorer validates the output schema, verifies that quotes occur in the public text, and checks that each cited observation supports the specified goal and precedes the statement. It then parses the quoted numeric values and compares them with the cited fields after normalizing Chinese and Arabic numerals, spoken digit sequences, time expressions and full-width characters. For example, “three nine five” can mean 395, and “six twelve” can express the clock component of 18:12.
Validated labels determine result-readout credit, fabrication and completion status; the deterministic outcome mapping then assigns the scalar reward. Invalid judgments leave a rollout unscored and invoke the group-filtering procedure in Appendix § 7. The remaining reward components follow runtime rules, with semantic judgments used for value normalization and for distinguishing continued content from a new response after interruption.
Outcome Tiers
For Stage-2a task families, sums the applicable components in Table 7, with clipping to and any applicable family-specific caps. For the five new Stage-2b task families, uses the outcome scores in § 4.3. When a call is required and none is executed, the score is if the policy speaks and if it stays silent.
Timing and Backend Responsibility
Timing rewards use 1-s units on the input-audio timeline. Interruption, cancellation and correction deadlines start at the onset of the corresponding user utterance; response and action deadlines start at the end of the request. The acting and stopping deadlines are and units, respectively (Section 3). The control reward penalizes both delayed stops after interruptions and unnecessary stops after backchannels or background noise.
Unexpected backend failures are filtered when the delegated task is faithful and sufficient and the frontend’s own calls and claims are correct. This filtering separates backend execution faults from the frontend decisions optimized by RL. Scripted errors, requests for missing information and partial completions in family D5 remain part of training, where the frontend must recover or report the outcome accurately.
Routing and Duplicate Work
For multi-step tasks, execution credit is determined by the fraction of steps completed correctly, aggregating direct calls and calls executed by the backend. The route is selected by the policy through its action interface. Two additional terms encourage efficient execution: a penalty for duplicated work and a latency cost for successful delegation of a single-step request whose tool is visible. For Stage-2b task families, these terms enter the outcome mapping: the latency
cost is deducted from a complete-success score, whereas duplicated work or a protocol fault places the rollout in a lower outcome tier with bounded deductions.
Evaluation Protocol
Tool Exposures, Backends and Models
We evaluate three tool exposures: Direct provides the frontend with the domain tools, Both adds query, and Backend provides query while assigning the domain tools to the backend’s catalog . The evaluation backends are Qwen3-8B and DeepSeek-V4.1-Flash; the latter has 552B backbone parameters and 16B active parameters per decoded token ([9]). Both backends are frozen, use temperature 0 and execute at most six tool calls per query request. Stage-2b training uses Qwen3-8.27B as the backend. The FDB v3 adapter names the backend tool’s single natural-language argument query, corresponding to task in the training schema.
Table 9 lists the thinker and projector used in each evaluation.
FDB v3 Metrics
FDB v3 ([23]) contains 100 English recordings from 12 speakers, including non-native accents, fillers, pauses and self-corrections. The requests use 12 simulated APIs and are followed by 30 s of ambient noise. There are 36 easy, 34 medium and 30 hard requests; 66 require one call, 18 require two and 16 require three. We run each configuration with three decoding seeds, yielding 300 sessions, and apply the benchmark’s scoring with Qwen3.8-27B at temperature 0 as the argument and response judge. The original benchmark uses GPT-4o ([35]).
Pass@1. A session passes when its execution log contains every gold call with correct arguments and no additional call, counting calls made by either the frontend or the backend. This criterion includes calls made before a user correction: executing a superseded call makes the session fail. The Call measure in Table 1 and Figure 4 counts sessions with an executed domain-tool call; query records a hand-off to the backend.
Tool, argument and response scores. Tool is the F1 between the multisets of executed and gold tool names, and Arg. is the judge’s argument accuracy. Both scores are averaged over all sessions, with zero assigned to sessions without an executed call. Resp. is the judged quality of the spoken response, averaged over sessions with a scored response, following the benchmark’s aggregation. Take is the fraction of sessions with a recorded response.
Outcome classes. We assign each session to one class in the following order: pass; no executed call; missing call; extra call; correct tools with a wrong argument. When available, the no-call class is further divided into sessions with and without speech (Table 12). All outcome rates use the full set of sessions as their denominator. The pass and wrong-argument classes together give the share whose executed tool names and counts match the gold calls, the judge-independent measure used in § 5.4.
| Frontend | Exposure | Backend LLM | Pass@1↑ | Pass↑ | Wrong arg. | Missing | Extra | Spoke | Silent | 1 | 2 | 3 |
| RV-Audio + Harness | FE–BE | Qwen3-8B | 4.7 ± 1.2 | 14 | 9 | 7 | 1 | 144 | 125 | 11 | 1 | 2 |
| Duopra-SFT | Direct | none | 26.0 ± 3.6 | 78 | 17 | 38 | 1 | 118 | 48 | 76 | 2 | 0 |
| Duopra-RL1 | Direct | none | 26.7 ± 0.6 | 80 | 17 | 72 | 0 | 85 | 46 | 80 | 0 | 0 |
| Duopra-RL1 | Both | Qwen3-8B | 24.3 ± 1.5 | 73 | 17 | 60 | 0 | 94 | 56 | 73 | 0 | 0 |
| Duopra-RL1 | Backend | Qwen3-8B | 20.0 ± 1.0 | 60 | 27 | 5 | 9 | 119 | 80 | 48 | 11 | 1 |
| Duopra | Direct | none | 47.7 ± 3.5 | 143 | 58 | 40 | 3 | 39 | 17 | 115 | 13 | 15 |
| Duopra | Backend | Qwen3-8B | 44.7 ± 5.7 | 134 | 85 | 25 | 16 | 40 | 114 | 11 | 9 | |
| Duopra | Both | Qwen3-8B | 44.3 ± 0.6 | 133 | 53 | 39 | 3 | 72 | 105 | 16 | 12 | |
| Duopra | Backend | DeepSeek | 49.3 ± 3.5 | 148 | 46 | 27 | 10 | 69 | 115 | 19 | 14 | |
| Duopra | Both | DeepSeek | 44.7 ± 2.5 | 134 | 52 | 37 | 5 | 72 | 110 | 10 | 14 |
Table 12. FDB v3 session outcomes (300 sessions per configuration). Pass@1: decoding-seed mean±std. Outcome classes are exclusive (Appendix § 11); merged no-call cells include sessions with and without speech. Successes by required call count use 198 one-call, 54 two-call and 48 three-call sessions. FE–BE: RV-Audio’s original request interface. No-call and wrong-argument outcomes reveal distinct points of failure.
Statistics and Latency
Paired comparisons. We estimate 95% confidence intervals for differences using 20,000 paired bootstrap resamples of the 100 recordings. Each resample draws recordings with replacement and retains all three decoding seeds of each selected recording in both configurations; the interval uses the 2.5th and 97.5th percentiles of the resulting differences. For an individual configuration, we report the mean and sample standard deviation over its three seeds. Differences are calculated from unrounded session counts.
Factual-answer latency. We compare latency between Duopra’s tool exposures for each backend. The benchmark scorer measures the time until the agent delivers the factual answer (Lin et al. [23]); this measure includes the serving runtime and tool execution, rather than measuring action-initiation timing. We compute this measure for each seed and report its mean and sample standard deviation; the comparison appears in Appendix § 12.§ 1.
Baselines
RV-Audio + Harness. We run the streaming RV-Audio model with its Harness, using Qwen3-8B as the backend and the same simulated tools as Duopra. The published FDB v3 score of 42.0% (Venus Team (Ant Group), Tsinghua University (2026), Table 7) uses offline inference with tool definitions in the prompt, according to a repository collaborator;4 our evaluation uses the streaming delegation interface.
DuplexOmni. DuplexOmni (Huang et al. [18]) uses Qwen3-8B as its pluggable thinking layer with the same simulated tools. Its three runs pass 15, 17 and 19 of the 100 requests (). Across these runs, it passes 31 of 108 easy, 15 of 102 medium and 5 of 90 hard sessions (, and , respectively; mean std over seeds).
Published Results
Table 10 records the sources and evaluation settings of the published results in Table 1.
| Cited system | Source | Evaluation protocol |
| GPT-Realtime, Gemini Live 2.5 and 3.1, Grok, Ultravox, Cascaded | Lin et al. [23], Table 2 | GPT-4o judge. GPT-Realtime is gpt-realtime-1.5 and Ultravox v0.7; Cascaded is Whisper (Radford et al., [41]), GPT-4o and OpenAI TTS in sequence. |
| MOSS-Live | MOSI [31], developer report, accessed 12 November 2026 | The developer report gives a tool-call success rate of 51.0%. |
| Gander, GPT-5.6 as the back brain | Orantqing et al. [36], Table 3 | The benchmark’s scoring scripts with GPT-4o as the judge. |
| NemotronLabs VoiceChat | Balam et al. [1], Table 4 | Tool-selection F1, argument accuracy and Pass@1. |
| RV-Audio, offline | Venus Team (Ant Group), Tsinghua University (2026), Table 7 (arXiv v3) | Offline inference with the tool definitions in the prompt; judge not named. We evaluate RV-Audio + Harness in our streaming pipeline separately (Appendix § 11). |
| Hu et al., Qwen3-235B-A22B backend | Hu et al. (2026), Table 2 (arXiv v2) | Tool-selection accuracy, not F1; GPT-4o judges the agent’s text, transcribed by an external ASR system. |
| DuplexAgent-RSI on VoiceChat and on Qwen-Realtime; the Realtime-Venus system on Realtime-Omni-9B | Shen et al. [48], Table 4 | Arguments are compared by exact string matching; Pass@1 is 47.0% and 49.0% for DuplexAgent-RSI and 1.0% for the Realtime-Venus system. |
| MiniCPM-o 4.5, FDB v1.0 | Zhou et al. [66], Table 1 | The three takeover rates compared in Table 11. |
Table 10. Sources and evaluation protocols for published results. FDB v3 results appear in Table 1; MiniCPM-o 4.5 provides the FDB v1.0 reference in Table 11. Each result uses its source’s evaluation setup and scoring protocol.
Turn-Taking and General Ability
We evaluate FDB v1.0 (Lin et al. [26]) with its official scripts and Parakeet transcripts (Rekesh et al. [44]), using 727 inputs per seed and three seeds. Output audio is recorded on a clock aligned with the input audio, as required by the pause and turn-boundary metrics. We check the pipeline on MiniCPM-o 4.5 (Cui et al. [8]) against Zhou et al. [66], Table 1.
MiniCPM-o 4.5, RV-Audio and the three Duopra checkpoints run through the same streaming pipeline. We also evaluate these five systems on FDB v1.5 (Lin et al. [25]) (498 scenes and 996 recordings per seed) and VoiceBench (Chen et al. [5]) (7,783 items per seed), using three seeds per system and Qwen3.8-27B for judge-based metrics. These evaluations compare conversational behavior and general ability across our training stages. Most of RV-Audio’s multiple-choice responses on VoiceBench are unparsable by the scorer; its reported score includes these parsing failures.
Additional Results
Initiation vs. Execution (RQ2)
Coverage vs. conditional success. Figure 4 shows why Pass@1 alone can conceal the failure source: a frontend may rarely reach an executed call, whereas another may call frequently but still fail conditional on calling. The identity Pass@1 = Call Pass|call holds for every configuration; the latter factor is conditioned on a different called subset in each row.
Matched sessions locate the gain. For RV-Audio + Harness, the 31 sessions with a domain call yield 14 baseline passes and 12 Duopra passes; Duopra’s net benefit instead comes from the other 269 sessions, where it succeeds on 122 requests that the baseline never executes. For Duopra-RL1 (Backend), Duopra passes 56 versus 60 on the 101 baseline-called sessions and 78 of the remaining 199 baseline-no-call sessions.
Hand-offs vs. executed calls. The RV-Audio + Harness frontend sends 34 hand-offs across 33 sessions, but domain tools run in only 31. Accordingly, its hand-off ceiling is based on 33 sessions, while its Call uses 31 (Table 1; Figure 4). With DeepSeek, Duopra’s conditional success rises while its call coverage falls.
Tool visibility and routing. Table 12 reports the complete session outcomes. Under Both, Duopra delegates in one Qwen3-8B session and none with DeepSeek. Of its 133 successes with Qwen3-8B, at least 132 therefore occur without delegation; at least 27 of the 28 multi-call successes do so. All 134 DeepSeek-Both successes occur without delegation. Duopra-RL1 also rarely delegates under Both but succeeds less often; therefore, low delegation frequency by itself does not establish routing competence. Under Backend, restricting the frontend to query tests whether its requests suffice for backend execution rather than its ability to choose between routes.
Latency and argument fidelity. Backend has higher factual-answer latency than Both with either backend LLM, and for Qwen3-8B it also has a higher wrong-argument share. The observed latency is measured until the factual answer, including runtime and tool execution (Appendix § 11). The respective latency values are vs. s for Qwen3-8B, and vs. s for DeepSeek (Backend vs. Both, decoding-seed mean±std). The wrong-argument comparisons are given in Table 13b; they describe the tested exposures rather than a controlled cost. of delegation.
Training Stages and Failure Modes (RQ3)
Invocation vs. completion. Table 13 shows that the intermediate Duopra-RL1 checkpoint increases the frequency of calls but scarcely changes task success relative to Duopra-SFT. Duopra improves Pass@1 most on medium, hard, and multi-call requests. As Stage 2b also introduces new tasks and reward rules, this comparison attributes the observed change to the overall training stage, not to a single reward term.
Shifting error classes. In Figure 3, no-call sessions decline markedly across checkpoints, and requests requiring several calls become solvable more often. However, incorrect arguments remain common once the correct tool set is reached. The figure and Table 12 separate those error classes rather than treating every unsuccessful session as a failure to act.
Turn-Taking and General Ability (RQ4)
Behavior-specific retention. Table 14 shows improvements in several turn-taking and overlap metrics in our pipeline, but not uniform retention of conversational ability. All three post-trained checkpoints produce no backchannels on the tested FDB v1.0 items; FDB v1.5 instead measures whether speech continues after a user’s backchannel, which is a distinct behavior. Interruption response also declines from Duopra-RL1 to Duopra. VoiceBench results provide a separate check on broader spoken-answer ability, with parsing limitations for the base model noted in Appendix § 11.
| Metric | MiniCPM-o 4.5 | RV-Audio | Duopra-SFT | Duopra-RL1 | Duopra |
| FDB v1.0 | |||||
| Turn-taking TOR↑ | 86.3 ± 4.0 | 59.1 ± 2.6 | 75.1 ± 0.5 | 75.6 ± 0.8 | 75.6 ± 3.0 |
| Pause TOR, real↓ | 38.7 ± 1.8 | 18.1 ± 2.3 | 9.3 ± 0.9 | 5.6 ± 2.1 | 5.4 ± 0.3 |
| Interruption TOR↑ | 99.2 ± 1.4 | 79.5 ± 3.0 | 97.0 ± 1.5 | 96.5 ± 0.9 | 91.3 ± 0.6 |
| Pause TOR, synthetic↓ | 10.5 ± 1.1 | 10.5 ± 1.8 | 5.4 ± 1.1 | 3.2 ± 1.1 | 2.4 ± 1.1 |
| Turn-taking latency (s)↓ | 2.23 ± 0.08 | 2.27 ± 0.15 | 2.64 ± 0.04 | 2.75 ± 0.07 | 2.75 ± 0.05 |
| Interruption latency (s)↓ | 1.92 ± 0.09 | 2.58 ± 0.05 | 1.99 ± 0.05 | 2.18 ± 0.02 | 2.28 ± 0.03 |
| Interruption quality‡ (0–5)↑ | 4.14 ± 0.05 | 3.65 ± 0.04 | 4.30 ± 0.13 | 4.21 ± 0.06 | 4.38 ± 0.13 |
| Backchannel TOR↓ | 58.8 ± 6.4 | 60.0 ± 4.8 | 0.0ᵇ | 0.0ᵇ | 0.0ᵇ |
| Backchannel frequency (/s) | 0.063 ± 0.006 | 0.106 ± 0.030 | 0ᵇ | 0ᵇ | 0ᵇ |
| Backchannel JSD↓ | 0.794 ± 0.014 | 0.789 ± 0.032 | 1.000ᵇ | 1.000ᵇ | 1.000ᵇ |
| FDB v1.5‡ | |||||
| Response after an interruption↑ | 47.5 ± 2.6 | 35.2 ± 2.1 | 69.5 ± 2.3 | 67.0 ± 2.3 | 66.0 ± 1.0 |
| Resumption after a backchannel↑ | 87.8 ± 5.7 | 36.1 ± 5.2 | 95.9 ± 2.7 | 96.6 ± 0.6 | 96.3 ± 0.6 |
| Resumption, user talks to others↑ | 56.7 ± 2.1 | 43.3 ± 8.6 | 45.3 ± 2.3 | 56.3 ± 3.2 | 52.0 ± 3.5 |
| Resumption, background speech↑ | 59.0 ± 3.6 | 43.7 ± 1.5 | 68.3 ± 3.2 | 70.7 ± 2.1 | 71.0 ± 2.0 |
| VoiceBench‡ (all ↑) | |||||
| Mean of nine categories | 64.9 ± 0.2 | 37.9 ± 0.4 | 52.5 ± 0.1 | 52.9 ± 0.2 | 55.9 ± 0.2 |
| MMSU | 57.4 ± 0.2 | 26.6 ± 0.7 | 54.7 ± 0.4 | 56.4 ± 0.6 | 62.5 ± 0.1 |
| OpenBookQA | 73.1 ± 1.3 | 24.6 ± 1.4 | 69.2 ± 1.5 | 70.8 ± 1.0 | 77.4 ± 1.3 |
| BBH | 63.7 ± 1.7 | 50.6 ± 1.9 | 59.7 ± 0.8 | 61.0 ± 1.0 | 60.0 ± 0.4 |
| IFEval | 55.0 ± 2.0 | 24.9 ± 1.4 | 33.2 ± 0.9 | 34.1 ± 1.6 | 39.2 ± 0.9 |
| SD-QA | 48.5 ± 1.7 | 31.5 ± 1.9 | 27.0 ± 0.9 | 24.4 ± 1.3 | 27.7 ± 1.3 |
| AlpacaEval | 72.4 ± 1.1 | 41.6 ± 1.1 | 54.6 ± 0.1 | 54.9 ± 0.2 | 57.7 ± 0.3 |
| CommonEval | 64.3 ± 0.3 | 41.0 ± 0.8 | 45.0 ± 0.5 | 44.3 ± 1.1 | 49.5 ± 0.4 |
| WildVoice | 58.3 ± 0.2 | 34.5 ± 0.5 | 41.0 ± 0.4 | 40.0 ± 0.4 | 42.2 ± 0.4 |
| AdvBench (refusal rate) | 91.2 ± 1.3 | 65.6 ± 1.3 | 88.4 ± 1.2 | 90.6 ± 1.0 | 87.1 ± 1.8 |
Table 14. Conversational behavior and general ability in our streaming pipeline (three seeds, mean ± std; rates and scores in % unless stated otherwise). TOR: takeover rate; RV-Audio: Realtime-Venus-Audio. ᶜTakeover rates used in the main-text comparison, calibrated on MiniCPM-o 4.5 (Table 11); bold marks the best value on each calibrated row. ‡Judged wholly or partly by Qwen3.8-27B. ᵇNo backchannels are produced by any of the three post-trained checkpoints on the 55 FDB v1.0 backchannel items; the reported frequency and TOR are zero, and JSD is 1.000. These rows show a selective cost of task training and measure a different behavior from resumption after a user’s backchannel in FDB v1.5. Model configurations and VoiceBench parsing limitations appear in Appendix § 11. ↑/↓: higher/lower is better.
Talker Conditioning and Repair (RQ4)
At spoken-token position , talker conditioning is , where the thinker states and projector parameters are taken from the evaluated configuration. We test both the change in this signal after thinker training and the effect of repairing only its projector (§ 4.4).
Conditioning drift. Figure 5b compares the thinker checkpoints using the base projector. Their mean cosines to the talker conditioning of RV-Audio (original in the figure) are 0.935 (Duopra-SFT) and 0.929 (Duopra-RL), whereas the Stage-3 projector brings that cosine to 0.974 on the same replay analysis. The 88 fixed-text speech samples in panel (a) compare different thinker–projector pairs and therefore cannot by themselves isolate the projector’s causal effect.
Repair with a fixed thinker. For the Stage-2b thinker, mean cosine on the feature validation set rises from 0.930 to 0.973 after repair; this set is distinct from the 2,000-session replay analysis above. We select the final projector from four candidates using 221 fixed-trajectory utterances, and report pooled CER/WER on that selection set in Table 15. Text, control tokens, unit boundaries, speaker and synthesis settings are fixed in this comparison; the base-projector row is the ablation and the Stage-3 row is Duopra.
Selection-set intelligibility. Table 15 shows lower Chinese character and English word error rates with the repaired projector, alongside small ECAPA cosine changes. Because the same utterances selected the candidate, the reported reduction should be interpreted as selection-set evidence rather than an independent generalization estimate.
Case Examples
Table 16 illustrates both successful routes and two failure modes on hard FDB v3 requests. The request in case (a) matches the public ecommerce_19 scenario used schematically in Figure 1. The backend can complete a multi-step request from a faithful delegation (a), while the frontend can execute a sequence of direct calls (b). The failures involve an alphanumeric argument substi tution (c) and a spoken plan without any executed call (d).
Additional Related Work
Tool use and optimization. Text-based tool use spans self-supervised API learning ((Schick et al., 2023), interleaved reasoning and acting [60], and instruction tuning [39]. BFCL [38] and -bench [59] evaluate function calls and interactive task completion. GRPO retains PPO’s clipped objective [46] while replacing its value model with group-relative advantages [47]. Dr. GRPO removes response-length and reward-standard-deviation normalizations [28]; DAPO uses dynamic sampling to exclude groups with identical rewards [62]. Duopra applies this optimization framework to streaming sessions, with rewards for timely action, grounded arguments and task completion.
Spoken agents and evaluation. Alongside FDB v3 [23], benchmarks assess spoken tool calls [19, 37] and end-to-end voice-agent performance [43, 3]. Related systems delegate reasoning and tool use to background models [51, 36] or retrieve knowledge asynchronously [7]. Our analysis separates frontend action initiation from subsequent tool execution using the resulting call traces.
Sufficiency and dialogue state. Information sufficiency connects our setting to slot filling in schema-guided dialogue [42] and missing-information tests for tool use [38, 30]. Our formulation (§ 3) tracks the latest spoken values and corrections; delegated requests must also carry required dependencies. The same sufficiency rule applies to both routes.
References
- [1]Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan, Chen Chen, Zhehuai Chen, Zijia Chen, Francesco Ciannella, Shalini De Mello, Slyne Deng, et al. 2026. NemotronLabs VoiceChat: An open full-duplex speech-to-speech model with tool calling capabilities. Preprint, arXiv:2609.21967.arxiv.org/abs/2609.21967
- [2]Lujia Bao, Qian Chen, Luyao Cheng, Chong Deng, Yuxiang Kong, Xiangang Li, Xu Li, Jiaqing Liu, Chao-Hong Tan, Haoyu Wang, Wen Wang, Xilou Wang, Haoxiang Xu, Junhao Xu, Liang Yi, Binbin Zhang, Qinglin Zhang, and Qiquan Zhang. 2026. Qwen-Audio-3.1-Realtime: Towards reliable agentic voice interaction. Preprint, arXiv:2609.25176.arxiv.org/abs/2609.25176
- [3]Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwafunemi Bampose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Hari Subramani, Joseph Marinier, Sridhar Krishna Nemala, Anil Madamala, and Srinivas Sunkara. 2026. EVA-Bench: A new end-to-end framework for evaluating voice agents. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 46771–46748, Budapest, Hungary. Association for Computational Linguistics.arxiv.org/abs/2605.13841
- [4]Qian Chen, Luyao Cheng, Chong Deng, Xiangxiang Li, Jiaqing Liu, Chao-Hong Tan, Wen Wang, Junhao Xu, Jieping Ye, Qinglin Zhang, Qiquan Zhang, and Jingren Zhou. 2025. Fun-Audio-Chat technical report. Preprint, arXiv:2512.20156v1.arxiv.org/abs/2512.20156
- [5]Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. 2026. VoiceBench: Benchmarking LLM-based voice assistants. Transactions of the Association for Computational Linguistics, 14:378–398.arxiv.org/abs/2410.17196
- [6]Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. 2026. SHANKS: Simultaneous hearing and thinking for spoken language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8951–8972, San Diego, California, United States. Association for Computational Linguistics.
- [7]Chung-Ming Chien, Manu Orsini, Eugene Kharitonov, Neil Zeghidour, Karen Livescu, and Alexandre Défossez. 2026. MoshiRAG: Asynchronous knowledge retrieval for full-duplex speech language models. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 19522–19545. PMLR.arxiv.org/abs/2604.12928
- [8]Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, et al. 2026. MiniCPM-o 4.5: Towards real-time full-duplex omni-modal interaction. Preprint, arXiv:2604.27393.arxiv.org/abs/2604.27393
- [9]DeepSeek-AI. 2026. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. Technical report.arxiv.org/abs/2609.19969
- [10]Alexandre Défossez, Laurent Mazare, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037.arxiv.org/abs/2410.00037
- [11]Chong Deng, Yunjie Ji, Yuxiang Kong, Xiangxiang Li, Xu Li, Binbin Zhang, Hana Zhu, and Jianheng Zhuo. 2026. Qwen-Audio-Agent technical report. Preprint, arXiv:2609.25195.arxiv.org/abs/2609.25195
- [12]Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024. CosyVoice 2: Scalable streaming speech synthesis with large language models. Preprint, arXiv:2412.10117.arxiv.org/abs/2412.10117
- [13]Qingkai Fang, Shoutao Guo, and Yang Feng. 2026. BayLing-Duplex: Native full-duplex speech dialogue with a single autoregressive LLM. Preprint, arXiv:2606.14528.arxiv.org/abs/2606.14528
- [14]Chi-Yuan Hsiao, Ke-Han Lu, Yu-Kuan Fu, Guan-Ting Lin, Hsiao-Tsung Hung, and Hung-yi Lee. 2026. ASPIRin: Action space projection for interactivity-optimized reinforcement learning in full-duplex speech language models. Preprint, arXiv:2604.10065.arxiv.org/abs/2604.10065
- [15]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
- [18]Muye Huang, Lingling Zhang, Xingyu Yu, Lei Shi, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, and Jun Liu. 2026. Duplex-Omni: Real-time listening, seeing, thinking, and speaking for full-duplex interaction. Preprint, arXiv:2606.09186.arxiv.org/abs/2606.09186
- [19]Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, and Shubham Agarwal. 2025. VoiceAgentBench: Are voice assistants ready for agentic tasks? Preprint, arXiv:2510.07978.arxiv.org/abs/2510.07978
- [20]Harrison Lee, Raghav Gupta, Abhinav Rastogi, Yuan Cao, Bin Zhang, and Yonghui Wu. 2022. SGD-X: A benchmark for robust generalization in schema-guided dialogue systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10938–10946.DOI
- [21]Xuhong Li, Yves Grandvalet, and Franck Davoine. 2018. Explicit inductive bias for transfer learning with convolutional networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2825–2834. PMLR.arxiv.org/abs/1802.01483
- [22]Yuxin Li, Donghang Wu, Guan-Ting Lin, Hung-yi Lee, Chengwei Qin, Zhehuai Chen, and Chen Chen. 2026. Decoupling conversational dynamics in full-duplex spoken models through reinforcement learning. Preprint, arXiv:2607.07148.arxiv.org/abs/2607.07148
- [23]Guan-Ting Lin, Chen Chen, Zhehuai Chen, and Hung-yi Lee. 2026a. Full-Duplex-Bench-v3: Benchmarking tool use for full-duplex voice agents under real-world disfluency. Preprint, arXiv:2604.04847.arxiv.org/abs/2604.04847
- [25]Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, Shinji Watanabe, and Hung-yi Lee. 2026c. Full-Duplex-Bench v1.5: Evaluating overlap handling for full-duplex speech models. In ICASSP 2026 – 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 19447–19451, Barcelona, Spain. IEEE.arxiv.org/abs/2507.23159
- [26]Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H. Liu, and Hung-yi Lee. 2025a. Full-Duplex-Bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. In 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8, Honolulu, HI, USA. IEEE.arxiv.org/abs/2503.04721
- [27]Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko. 2025b. Align-SLM: Textless spoken language models with reinforcement learning from AI feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20395–20411, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2411.01834
- [28]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding R1-Zero-like training: A critical perspective. In Conference on Language Modeling (COLM).arxiv.org/abs/2503.20783
- [29]Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
- [30]Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. 2025. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, Albuquerque, New Mexico. Association for Computational Linguistics.
- [31]MOSI. 2026. MOSS-Live. Model page, https://www.moss.cn/en/models/moss-live. Accessed 12 November 2026.
- [32]Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abderrahman Mohamed, and Emmanuel Dupoux. 2023. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266.arxiv.org/abs/2203.16502
- [33]Atsumoto Ohashi, Neil Zeghidour, Alexandre Défossez, and Eugene Kharitonov. 2026. Multi-faceted interactivity alignment in full-duplex speech models. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 52949–52965, Budapest, Hungary. Association for Computational Linguistics.arxiv.org/abs/2606.11167
- [34]OpenAI. 2025. Introducing gpt-realtime and Realtime API updates for production voice agents. OpenAI blog post, August 28, 2025.
- [35]OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, et al. 2024. GPT-4o system card. Preprint, arXiv:2410.21276.
- [36]Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhou Li, Shangda Wu, et al. 2026. Multimodal duplex interaction agent. Preprint, arXiv:2609.08977.arxiv.org/abs/2609.08977
- [37]Ramit Pahwa, Apoorva Beedu, Parivesh Priye, Rutu Gandhi, Saloni Takawale, Aruna Baijal, and Zengli Yang. 2026. Audio2Tool: Speak, call, act – a dataset for benchmarking speech tool use. In Interspeech 2026, pages 3453–3457.arxiv.org/abs/2604.22821
- [38]Shishir G Patil, Huanzhi Mao, FanJia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. 2025. The Berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 48371–48392. PMLR.
- [39]Yujia Qin, Shihbao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Representations, pages 9695–9717.arxiv.org/abs/2307.16789
- [40]Qwen Team. 2026. Qwen3.8-Max: A new bar for coding and cowork. Blog post, https://qwen.ai/qwen3.8. Model card of Qwen3.8-27B: https://huggingface.co/Qwen/Qwen3.8-27B; accessed 13 November 2026.
- [41]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518. PMLR.arxiv.org/abs/2212.04356
- [42]Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8689–8696.arxiv.org/abs/1909.05855
- [43]Soham Ray, Keshav Dhandhania, Victor Barres, and Karthik Narasimhan. 2026. τ-Voice: Benchmarking full-duplex voice agents on real-world domains. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 104031–104067. PMLR.arxiv.org/abs/2603.13686
- [44]Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, and Boris Ginsburg. 2023. Fast Conformer with linearly scalable attention for efficient speech recognition. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE.arxiv.org/abs/2305.05084
- [46]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347.arxiv.org/abs/1707.06347
- [47]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.arxiv.org/abs/2402.03300
- [48]Yingda Shen, Yuxiang Wang, Kunyu Feng, Qinke Ni, Jiaqi Li, Minghao Hsu, Junan Zhang, Dekun Chen, Yutong Bian, and Zhizheng Wu. 2026. DuplexAgent-RSI: Recursive harness improvement for full-duplex voice agent collaboration. Preprint, arXiv:2610.11299.arxiv.org/abs/2610.11299
- [49]Gabriel Skantze. 2021. Turn-taking in conversational systems and human-robot interaction: A review. Computer Speech & Language, 67:101178.DOI
- [50]StepFun. 2025. Step-Audio 2. GitHub repository, https://github.com/stepfun-ai/Step-Audio2. Token2wav: token2wav.py.
- [51]Thinking Machines Lab. 2026. Interaction models: A scalable approach to Human-AI collaboration. Thinking Machines Lab: Connectionism.DOI
- [52]Bandhav Veluri, Benjamin N. Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. 2024. Beyond turn-based interfaces: Synchronous LLMs as full-duplex dialogue agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21390–21402, Miami, Florida, USA. Association for Computational Linguistics.
- [53]Venus Team (Ant Group), Tsinghua University. 2026. Realtime-Venus: A full-duplex interaction system with asynchronous delegation. Preprint, arXiv:2609.13814.
- [54]Xiong Wang, Yangze Li, Chaoyou Fu, Yike Zhang, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2025. Freeze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 63345–63354. PMLR.arxiv.org/abs/2411.00774
- [55]Anne Wu, Laurent Mazaré, Neil Zeghidour, and Alexandre Dèfossez. 2025a. Aligning spoken dialogue models from user interactions. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 67476–67498. PMLR.arxiv.org/abs/2506.21463
- [56]Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, et al. 2025b. Step-Audio 2 technical report. Preprint, arXiv:2507.16632.arxiv.org/abs/2507.16632
- [57]Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-Omni technical report. Preprint, arXiv:2503.20215.arxiv.org/abs/2503.20215
- [58]Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Xiangyu Zeng, Yifei Li, Haowen Hou, Zheming Liang, Congcong Wang, Kaiwen Tuo, Jun Zhang, Yuhan Zhu, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, and Jiaqi Wang. 2026. JoyAI-VL-Interaction: Real-time vision-language interaction intelligence. Preprint, arXiv:2606.14777. Code and models: https://github.com/jd-opensource/JoyAI-VL-Interaction.arxiv.org/abs/2606.14777
- [59]Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025a. 7-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, pages 9965–10017.
- [60]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations.arxiv.org/abs/2210.03629
- [62]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaoch en Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, juncai liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others. 2025a. Dapo: An open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 113222–113244. Curran Associates, Inc.arxiv.org/abs/2503.14476
- [63]Wenyi Yu, Siyin Wang, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, and Chao Zhang. 2026. SALMONN-duo: Adaptive dual-system co-ordination for full-duplex voice agents. Preprint, arXiv:2609.34247.arxiv.org/abs/2609.34247
- [64]Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. 2025b. SALMONN-omni: A standalone speech LLM without codec injection for full-duplex conversation. In Advances in Neural Information Processing Systems, volume 38, pages 24617–24643. Curran Associates, Inc.arxiv.org/abs/2505.17060
- [65]Haoyang Zhang, Jun Chen, Donghang Wu, Yuxin Li, Yuxin Zhang, Xiangyu Tong Zhang, Che Liu, Qingjian Lin, Yizhou Peng, Hexin Liu, Eng Siong Chng, Chao Yan, Boyong Wu, Yechang Huang, Xuerui Yang, and Fei Tian. 2026. DuplexSLA: A full-duplex spoken language model with synchronized speech, language, and action. Preprint, arXiv:2605.20755.arxiv.org/abs/2605.20755
- [66]Zhiyang Zhou, Yingxin Shang, Zhou Wang, Hongwei Cai, Weixu Wang, Shuran Zhou, Shuofeng Zhao, Wenke Fan, Qingxiang Guo, Dawei Yang, Lin Yang, and Yang Song. 2026. AdaptDuplex: From static to adaptive full-duplex spoken dialogue. Preprint, arXiv:2609.29217.arxiv.org/abs/2609.29217


