WorksComputer Science
XBot-Omni: An Omni-Modal Model for Embodied Understanding and Interaction
Abstract
Embodied interaction requires an agent to integrate multimodal perception with verbal and nonverbal behaviors to effectively communicate its intentions, including speech, body motion, and facial expressions. We present XBot-Omni, a Thinker–Talker–Actor architecture that extends the Omni paradigm to robotic modalities, including body motion and facial expressions, with X'' representing the extensibility to robotic-specific modalities. The Thinker processes text, images, audio, and video to formulate contextually appropriate responses, which are then realized by the Talker and Actor through speech and coordinated bodily expressions, respectively. In this report, we innovatively introduce the concept of Actor'' as a dedicated component for embodied agents: a coordinated, streaming module that leverages the native representations of the Thinker and Talker within an end-to-end pipeline, introducing only modest latency overhead. This unified architecture enables multimodal generation conditioned on omni-modal inputs directly to prevent information loss in staged designs. We comprehensively evaluate XBot-Omni across seven benchmarks, demonstrating strong multimodal understanding capabilities on both general-domain and robot-centric videos, as well as high-quality, natural, and well-coordinated motion generation. Furthermore, our results validate its capability to generate embodied multimodal responses directly from omni inputs, meeting the diverse requirements of embodied interaction. XBot-Omni enables robots not only to perceive the omni world but also to express themselves through speech and bodily behaviors, extending Omni models from conversational interaction to embodied communication in physical environments.
Introduction
In embodied interaction, the meaning of nonverbal behavior, such as body gestures and facial expressions, depends on the surrounding dialogue, visual scene, and its timing relative to speech. For example, a pointing gesture must indicate the intended object as the corresponding phrase is spoken. Recent Omni models, including Qwen3.5-Omni-Plus (Team, 2026) and Gemini 3.1 Pro ([6]), integrate text, vision, and audio for multimodal understanding and response generation, but do not directly generate body motion. Extending them to embodied interaction requires deciding which actions to perform and coordinating them with speech.
Co-speech methods generate holistic motion ([12]) or strengthen gesture semantics through retrieval ([33, 29]). A common conversational pipeline passes synthesized speech to an audio-to-gesture (A2G) module, as in RoboGesture ([24]). However, speech-only conditioning can lose intent or scene information not explicitly expressed in the reply, while still requiring waveform release and audio feature extraction even during incremental processing of an ongoing response.
Native representation sharing offers tighter integration: MIBURI ([15]) supports streaming gestures, Motion-Omni ([13]) jointly generates speech and full-body motion offline, and DyaPlex ([16]) couples full-duplex speech with streaming motion and partner-motion perception. Building on this direction, an embodied Omni model should also use multimodal response context to guide explicit gesture semantics while coordinating motion with unfolding speech.
We introduce XBot-Omni, extending the Omni paradigm to an expandable set of robotic modalities, including body motion and facial expression, with “X” denoting this extensibility. Its Thinker–Talker–Actor architecture combines Omni understanding, native speech conditioning, and explicit gesture-semantic control for streaming motion generation (Table 1). The Thinker interprets text, images, audio, and video to infer response intent and scene context, while the Talker generates streaming speech. Rather than treating motion as a downstream synthesis task, the Actor directly conditions on native Thinker representations for semantic and scene-referential guidance, and native Talker states for speech-coordinated movement, alongside a complementary branch for facial expression.
Table 1. Comparison of representative Omni and speech–motion models. Omni input: text/audio/image/video; Motion: direct body-motion generation; Streaming: online motion generation; Native: direct speech-generator feature conditioning; Semantics: explicit gesture-semantic supervision, retrieval, or control; Robot deploy.: physical robot deployment. Checks: reported support; crosses: outside the described design; dashes: unreported or inapplicable, including motion-related features without motion output.
The Actor addresses three challenges through its Motion Generator, Semantic Action Controller, and Speech–Action Aligner. First, the Motion Generator incrementally consumes native Talker states, bypassing waveform release and audio re-encoding for low-latency motion generation. Second, as semantically specific gestures are underrepresented in common co-speech datasets ([15]), the Semantic Action Controller derives explicit action requirements from Thinker representations, grounding gestures in communicative intent and visual context. For example, “the apple is right there” should elicit pointing toward the intended object rather than generic rhythmic motion. Finally, the Speech–Action Aligner associates Thinker-derived semantic events with speech progression estimated from Talker states, coordinating gesture preparation and expression with the corresponding spoken phrases.
To support context-aware embodied interaction, we curate interaction videos and collect robot-view recordings for Thinker training, capturing perceptual context, dialogue, and response decisions. For Actor training, we collect synchronized speech–motion recordings to learn expressive movements and their temporal coordination with speech. The training pipeline combines supervised fine-tuning, reinforcement learning, and multi-teacher on-policy distillation for the Thinker, with speech-conditioned motion generation, semantic control, and spatial grounding for the Actor. Visually grounded pointing further illustrates how multimodal understanding can guide both spoken references and accompanying bodily expression.
Our evaluation spans seven benchmarks and examines multimodal understanding, embodied motion generation, visually grounded motion, and integrated system responsiveness. The Thinker achieves leading or state-of-the-art performance across four benchmarks, demonstrating strong capabilities in audio-visual understanding, multi-turn interaction, egocentric spatial reasoning, and embodied grounding. In particular, on Daily-Omni ([36]), it achieves 86.30% overall accuracy, outperforming Qwen3.5-Omni-Plus, Gemini-3.1-Pro, and Doubao-Seed-2.1-Lite. Together, these results demonstrate the Thinker’s strong performance in embodied scenarios. For motion generation, the Actor achieves a Fréchet Gesture Distance (FGD) of 1.351 and an overall semantic quality (SQ) of 1.383 on a 0–3 scale, compared with 0.608 for the strongest baseline under our machine-assisted semantic evaluation. We further evaluate visually grounded motion, while the integrated system achieves a median first-motion latency of 527 ms.
The following sections describe the architecture, training, and evaluation of XBot-Omni, whose central features are:
An embodied interaction paradigm for Omni models. We unify multimodal perception and reasoning with speech, body motion, and facial expression to enable semantically coherent and temporally aligned embodied responses.
A Thinker–Talker–Actor architecture. We integrate the Actor with the native representations of the Thinker and
Talker. Its body branch jointly supports motion generation, semantic action control, and speech–action alignment, while a complementary facial branch handles facial generation.
Interaction-centric data and training. We curate interaction videos, robot-view recordings, and synchronized speech–motion data, and develop a training pipeline combining supervised fine-tuning, reinforcement learning, and multi-teacher on-policy distillation for the Thinker, and speech-conditioned motion generation, semantic control, and spatial grounding for the Actor.
Comprehensive evaluation across embodied interaction capabilities. We evaluate XBot-Omni on seven benchmarks spanning four key dimensions of embodied interaction. The results demonstrate that XBot-Omni achieves leading performance, highlighting its strong and comprehensive embodied interaction capabilities.
Architecture
Overview
XBot-Omni adopts a Thinker–Talker–Actor architecture to connect multimodal understanding with embodied response generation (Figure 2). Given text, images, audio, video, and dialogue history, the Thinker interprets the interaction and formulates a contextual response. The Talker realizes the response as streaming speech, while the Actor produces accompanying body motion and facial expressions. The three components organize a single interaction response from the interpretation of incoming observations to its verbal and nonverbal expression.
Figure 2. Thinker–Talker–Actor architecture of XBot-Omni. Thinker representations provide response semantics and scene context, while native Talker states provide speech-generation conditions. For motion generation, semantic control translates interaction context into gesture requirements, including spatial pointing targets. These requirements and speech–action alignment guide an autoregressive generator whose tokens are decoded into body motion. The facial expression pathway combines a chunk transformer with a flow-matching generator. Output illustrations are schematic.
The architecture separates response formulation from modality-specific generation while maintaining coordination across output modalities. Speech unfolds as an acoustic sequence, body motion evolves continuously from the current pose, and facial expressions change throughout the response. These differences require the generated behaviors to share communicative context while remaining coordinated with the progression of speech. To achieve this, the Actor draws on native Thinker representations for interaction context and communicative intent, and native Talker states for speech-dependent conditioning. This connection allows the multimodal understanding used to formulate a response to also guide its bodily expression. Object pointing is one application: the scene-dependent referent becomes part of the action requirements passed to the Actor.
Thinker: Multimodal Understanding and Reasoning
Text, Audio, Images, and Video. The Thinker processes text, audio, images, and video through modality-specific pathways, mapping inputs into a unified sequence of representations consumed by the backbone. Text inputs are tokenized using the backbone tokenizer, while images and individual video frames are transformed into visual representations by a dedicated vision encoder. Following the Qwen3.5-Omni design ((Team, 2026)), audio inputs, including standalone recordings and video audio tracks, are resampled and converted into mel-spectrograms. The resulting acoustic features are processed by an AuT initialized from Qwen3-Omni ((Team, 2026)) and mapped to the backbone’s representation space through a lightweight projector. The backbone integrates these observations with dialogue context to formulate a response, while its hidden states provide semantic and contextual representations for both speech and embodied response generation. This multimodal context links referring expressions to objects and spatial relationships in the observed scene, informing the action requirements passed to the Actor.
Audio-visual Timestamps. Following Qwen3.5-Omni (Team, 2026), we expose absolute temporal information by prepending a formatted textual timestamp, expressed in seconds, to each video or audio-visual temporal patch. This representation makes observation time explicit in the input sequence and reduces reliance on temporal position indices alone, whose magnitudes and spacing can vary with sequence length and frame rate. The textual timestamps provide a common temporal reference for acoustic and visual observations, supporting the interpretation of events and their relationships over time.
Talker: Streaming Speech Generation
Autoregressive Speech Generation. The Talker follows the speech-generation architecture of Qwen3.5-Omni (Team, 2026), converting Thinker-generated responses into streaming speech. As shown in Figure 2, it comprises a hybrid temporal backbone and a depth predictor. The backbone uses a hybrid MoE design to model speech-token progression, while the depth predictor performs multi-token prediction (MTP) over the residual codebooks of each frame, conditioned on the backbone states. These tokens are subsequently decoded into acoustic waveforms by a speech codec decoder, enabling incremental speech generation without requiring the complete response to be synthesized in advance.
Native Speech Representations for Embodied Generation. In addition to speech tokens, the Talker exposes its evolving hidden states during autoregressive generation. These representations provide speech-related conditioning information before waveform decoding, allowing the Actor to access the ongoing speech-generation process without a separate audio encoding stage. In XBot-Omni, this internal interface enables the Actor to operate alongside streaming speech generation rather than waiting for synthesized audio.
Actor: Embodied Response Generation
Body Motion Generation. Using the shared Thinker representations and evolving Talker states, the Actor generates body motion through a semantically controlled, speech-conditioned autoregressive pathway. The Semantic Action Controller derives explicit gesture requirements from Thinker representations, linking interaction context to the meaning conveyed by the movement. For object pointing, these requirements identify the intended referent, and visual target localization supplies its spatial conditions. The Speech–Action Aligner associates these semantic events with the progression of the spoken response. Together with adapted Talker states and motion history, these conditions guide the Motion Generator, which draws on elements of MIBURI’s autoregressive formulation [15]. The generator produces discrete motion tokens, which are decoded into continuous body trajectories. For pointing, the target is expressed in the Actor’s body-centric frame, allowing the same generation pathway to realize spatially specified gestures alongside other communicative movements. This arrangement incorporates communicative gestures into the ongoing motion-generation process rather than treating them as isolated motion segments.
Facial Expression Generation. The Actor also generates facial expressions through a separate pathway conditioned on Thinker representations and native Talker states. These inputs provide response context and evolving speech-related features, respectively. A chunk transformer organizes them into temporally structured conditions, which guide a flow-matching generator to produce continuous facial-expression sequences. Unlike the autoregressive motion-token generation used for body movement, this pathway employs a distinct generative mechanism while remaining connected to the same communicative context and unfolding speech.
Coordinated Nonverbal Generation from Native Representations
The Actor generates body motion and facial expression as coordinated components of an embodied response. Native Thinker states carry the multimodal context and communicative intent used to formulate that response, while native Talker states provide evolving speech features and progress cues. Body generation combines these inputs with explicit action requirements, temporal conditions, and motion history, allowing interaction context to shape the meaning expressed by the movement. Facial generation organizes the shared response representations into chunk-level conditions for continuous expression. Both output streams are placed on a common speech timeline, coordinating body gestures and facial changes with each other and with the associated spoken content.
Motion Generator
Let and denote the native Thinker and Talker state streams. Their indices follow response-text and speech-generation positions, respectively. The Motion Generator accesses Talker states during native speech-token prediction. A recurrent causal adapter maps the available states into the motion-conditioning space, producing speech features while the native speech decoder continues to generate audio. This connection preserves direct access to the Talker’s internal representations and removes waveform release and audio re-encoding from the motion-conditioning path.
At motion step , generation is conditioned on the adapted speech history, the active semantic events and their temporal conditions, and the motion already produced:
Here, is a tuple of discrete motion codes, contains the adapted Talker states available at that motion step, and collects the active event conditions constructed by the Semantic Action Controller and Speech–Action Aligner. These conditions specify the gesture’s meaning and timing, together with the target location when the gesture refers to an object in the visual scene.
The generated codes are decoded into articulated body motion and also form the history for subsequent predictions. Temporal state persists across steps, allowing a new action requirement to take effect within the ongoing movement. The upper- and lower-body codec outputs are assigned motion timestamps on the response timeline, connecting the generated sequence to its eventual presentation with speech and facial expression. For robot-specific pointing execution, the spatial target and generated reference motion are passed to a kinematic adapter. It converts the target to the robot’s base frame and uses the reference posture to guide inverse kinematics, aligning the pointing direction with the target while preserving motion continuity. The platform-specific configuration is described in § 4.4.
Semantic Action Controller
The Semantic Action Controller converts response intent, interpreted in the multimodal interaction context, into explicit events that guide the Motion Generator. A causal semantic readout uses Thinker states together with features of the emitted response tokens to identify gesture categories and their associated response spans. Each event specifies a category , supported semantic parameters , an availability mask , and an associated text span. These describe how the gesture expresses its meaning, including the participating hand, direction, and, for a referential pointing event, the target’s 3D location in the Actor’s body-centric frame.
Object pointing provides a concrete application of this event interface. The Thinker’s multimodal context identifies which visual object the response refers to. Target localization, combined with depth estimation and geometric processing, provides its 3D position, which is transformed into the body-centric frame before motion generation. This location becomes part of the event’s semantic conditions. In Figure 1, “left” specifies a directional gesture, whereas “the apple is right there” associates the pointing event with the apple identified in the scene. The latter turns a scene-dependent reference into a visible gesture that helps the interlocutor identify the object being discussed. Separate event slots retain concurrent requirements and their individual progress.
Figure 1. Embodied interaction capabilities of XBot-Omni. We demonstrate XBot-Omni on two robot platforms, PrimeBOT Q1 ([18]) and AGIBOT X2 ([1]). The three examples illustrate: (1) multimodal synchronization, coordinating speech, body motion, and facial expressions to convey shyness; (2) semantic motion, aligning directional gestures with spoken references; (3) visually grounded motion, generating referential gestures toward objects identified in the visual scene. Colored speech segments and corresponding robot behaviors highlight their semantic and temporal relationships.
At motion step , an active event also has a start indicator and elapsed time , whose relation to speech is established by the Speech–Action Aligner. We concatenate the category, semantic parameter, and temporal representations into the event condition
The time encoder combines the start signal with elapsed-time and periodic features. These conditions modulate the hidden activations of the temporal and depth networks:
Here, is the active event set, is the no-event condition, and the map produces feature-wise scale and shift coefficients. Separate maps apply this modulation in the temporal and depth networks, combining concurrent events relative to the no-event condition.
The controller maintains an event while its meaning is being expressed. Completion predictions from the temporal hidden state distinguish the expressive core from readiness to release or transition. For example, a wave requires its characteristic movement in addition to raising the hand. These predictions inform subsequent event updates after the current motion step is generated, linking continued activation and handover to the evolving motion state.
Speech–Action Aligner
The Speech–Action Aligner relates the event spans identified from Thinker states to speech progression estimated from native Talker states. A causal recurrent encoder converts Talker states into normalized queries . Available response-text positions provide normalized keys , constructed from their Thinker features and text-unit embeddings. Their compatibility scores are combined with a forward-transition model to estimate speech progress:
The compatibility score applies to available text positions, with scale . The transition probabilities allow progress to remain at a position or advance. An additional frontier state represents text positions not yet available and uses the mean visible-position score as its emission. The state space expands as response text arrives; a nondecreasing readout reports the estimated visible position in speech.
Let be the text anchor associated with event , and let be the speech timestamp associated with Talker step . For a visually grounded pointing event, the target location specifies where the movement is directed, while the associated referring phrase provides the text anchor for when that movement should be expressed. Once the anchor is located, its estimated speech time and the event’s motion start are related by
Here, indexes the currently observed Talker steps, is a category-specific preparation lead, and is the earliest eligible motion timestamp when the event and alignment information are available. The preparation lead relates the requested action onset to its semantic anchor; the generation frontier determines when the request can enter the motion stream. The resulting start signal and elapsed time provide the temporal fields in Equation (2), guiding how the gesture develops toward its expressive core.
Speech progression updates can refine the timing of pending events. During presentation, decoded body frames and facial-expression frames are scheduled against the same speech timeline. This common reference connects the event-level timing above with the coordination of both visible outputs and the associated audio.
Facial Expression Generation
The facial branch uses a Chunk Transformer to organize response semantics and speech features into temporal conditions, followed by a conditional flow generator (Lipman et al. [11]):
Here, and collect the native states available for facial chunk , is its conditioning sequence, and is a noise input. The flow generator produces the facial-expression sequence from these conditions. Its frames carry timestamps on the common speech timeline, coordinating facial changes with the corresponding body motion and spoken content.
Data and Training Strategy
Large-Scale, High-Quality Embodied Interaction Data
Data Requirements. Training XBot-Omni for real-world embodied interactions requires data that captures not only rich audio-visual information but also fine-grained human behaviors, underlying intentions, and temporal dependencies across modalities. However, existing audio-visual datasets primarily focus on short clips or isolated events, often lacking the continuous behavioral and interaction dynamics of real-world scenarios. Meanwhile, embodied datasets largely emphasize action recognition and task execution, with limited coverage of natural conversations, social interactions, and real-time response behaviors. To bridge these gaps, we aim to construct a large-scale dataset that combines multimodal richness, temporal continuity, and diverse embodied interactions, supporting models in learning to perceive, understand, and respond to dynamic real-world environments.
Data Collection. We integrate publicly available datasets with privately collected recordings to cover a broad range of physical activities, natural conversations, and human–robot interactions. Specifically, we incorporate embodied activity datasets, which provide extensive coverage of egocentric daily activities, object manipulation, tool use, and complex procedural tasks. These datasets offer rich supervision for understanding action sequences, object state changes, and task contexts. To complement these sources, we collect private human–human and human–robot interaction recordings spanning natural conversations, social exchanges, collaborative activities, and robot-assisted tasks. These recordings preserve synchronized speech, environmental sounds, visual observations, gestures, facial expressions, and interaction dynamics, particularly speaker transitions, conversational turn-taking, and response timing. Together, public and private data provide complementary coverage of embodied manipulation and natural multimodal interactions.
Agent-Driven Annotation. Processing heterogeneous data sources presents challenges in capturing fine-grained multimodal details, preserving complex temporal structures, and producing reliable annotations at scale. To address these challenges, we develop a fully automated, agent-driven data construction pipeline. The agent adaptively orchestrates specialized tools for speech recognition, speaker diarization, object detection and tracking, action recognition, and audio-visual event localization to extract timestamped multimodal information. It then organizes continuous recordings into temporally structured interaction episodes while preserving contextual dependencies. For egocentric manipulation videos, we further extract task goals, action sequences, object state changes, and behavioral intentions to characterize the progression of physical activities. Based on these structured annotations, the pipeline automatically generates diverse instruction-following and question-answering samples covering scene understanding, audio-visual relationships, temporal reasoning, and intention understanding. Finally, cross-modal consistency checks and evidence-based verification against the original recordings are applied to filter unreliable annotations, yielding high-quality embodied interaction data with fine-grained temporal alignment and rich multimodal supervision.
Thinker Training
The post-training of the Thinker follows a three-stage pipeline designed to progressively develop and integrate multimodal interaction capabilities. First, supervised fine-tuning (SFT) establishes a unified foundation for multimodal perception, reasoning, and interaction understanding. However, different embodied interaction tasks require specialized reasoning capabilities and task-specific optimization objectives. We therefore train multiple domain-specific expert policies through reinforcement learning (RL), allowing each expert to specialize in its designated task domain. Finally, multi-teacher on-policy distillation (MOPD) consolidates these specialized capabilities into a single student policy.
Stage 1: Supervised Fine-Tuning (SFT). Supervised fine-tuning is conducted on a unified mixture of text, image, video, audio, and audio-visual instruction data collected or constructed for first-person human–robot interaction scenarios. This stage develops the Thinker’s ability to understand ongoing interactions from egocentric observations, including conversational-context modeling, temporal event understanding, reference resolution, speaker and addressee identification, target-person selection, and intention recognition from verbal and nonverbal cues. Particular attention is given to proactive-interaction decisions, such as determining whether the robot should respond, when it should enter or remain outside an interaction, and which person a response should address. The Thinker and its associated modality adapters are updated, while the Talker, action head, and expression head remain frozen. The resulting SFT checkpoint provides the shared initialization for both the specialized experts in Stage 2 and the student in Stage 3.
Stage 2: Specialized Expert Training via Reinforcement Learning. Starting from the shared SFT checkpoint, we independently train a set of specialized expert policies , where each expert focuses on a designated task domain within embodied multimodal interaction. We employ Group Relative Policy Optimization (GRPO) [20] to optimize each expert using its domain-specific training data and reward criteria. This stage separates the development of individual capabilities, allowing each expert to be optimized independently before capability integration.
We employ two complementary types of reward signals, according to the requirements of each task:
Rule-based rewards. For tasks with objectively verifiable outcomes, predefined criteria evaluate the correctness of generated responses and interaction decisions, providing direct feedback on task completion and compliance with output requirements.
Model-based rewards. For conversational and embodied behaviors that cannot be fully captured by deterministic rules, we adopt an LLM-as-a-judge protocol. The evaluator assesses the generated response or interaction decision based on the interaction history and key multimodal observations. When available, reference answers, target actions, and environment-state annotations provide additional grounding for the evaluation.
After RL training, the specialized experts are frozen and serve as teachers for the subsequent MOPD stage.
Stage 3: Multi-Teacher On-Policy Distillation (MOPD). We employ MOPD [14] to consolidate the capabilities of the specialized experts into a single Thinker. The student policy is initialized from the Stage 1 SFT checkpoint and trained on a mixture of task-domain datasets. For each input, the student generates its own response trajectory, and the expert associated with that input’s task domain provides token-level supervision along the generated trajectory.
Formally, let denote the input distribution for task domain , and let denote its sampling probability, with . An input contains the interaction context and the available observations in the relevant modalities. The student first samples a response trajectory . At each student-visited prefix , the corresponding frozen expert produces a next-token distribution conditioned on the same input and prefix. The student is optimized by minimizing the token-level reverse Kullback–Leibler divergence:
Here, remains fixed throughout distillation, while denotes the trainable student parameters.
By obtaining expert supervision on student-generated trajectories, MOPD trains the Thinker on prefixes encountered under its own policy. Domain-specific routing supplies supervision from the corresponding specialist, while updates across domains integrate these capabilities into shared student parameters. The resulting Thinker serves as a single policy for multimodal understanding, reasoning, and embodied interaction.
Actor Training
The body branch is trained on our privately collected co-speech and semantic-motion data. The co-speech recordings provide natural speech–motion pairs and motion contexts, while the semantic recordings cover a range of semantic categories. We combine these complementary sources to learn natural co-speech behavior and explicit semantic control.
Actor training progressively connects native speech representations to continuous, semantically controlled motion. The body branch follows three stages: native-state adaptation establishes speech conditioning and temporal alignment; semantic-event training develops complete gesture execution; and generated-history training improves continuity under the model’s own motion context. The complementary facial branch uses a conditional flow-matching objective under the same native response context. Together, these components train the Actor to realize communicative intent conveyed through native Omni representations. § 4.4 applies this foundation to object pointing, where scene context determines the referent communicated by the gesture.
Stage 1: Native-State Adaptation and Speech Alignment. The first stage connects the body model to the fixed Talker through a causal conditioning adapter. Training pairs associate Talker states with features extracted from the corresponding generated audio. Let denote the adapter and the set of valid paired steps. The adapter is optimized using normalized feature regression:
Here, is the conditioning dimension and contains feature-wise standard deviations estimated from the training data; division is element-wise. The Talker and motion codec remain frozen during adaptation. Audio-derived features provide offline supervision, while inference uses the available Talker states directly. This stage gives the motion model access to speech features without waiting for waveform release and re-encoding.
The Speech–Action Aligner learns speech-position scores from native states and available response-text features. Offline acoustic alignment provides position labels, with confidence weights controlling their contribution. For source with speech steps, let be the aligned text position and its supervision weight. The position objective is
Here, contains sources with valid supervision, and is the softmax distribution over the position scores defined in § 3, with unavailable text positions masked. A label beyond the visible prefix is assigned to the frontier state. Invalid labels contribute zero weight; normalization by source length preserves the reduced contribution of sparsely supervised sources. These machine-aligned labels are used only during training. At inference, the learned scores are combined with the forward-transition model to estimate speech progress and schedule semantic events.
Stage 2: Semantic-Event Motion Learning. With native speech conditioning established, this stage trains the motion-control modules and completion heads to express explicit gesture requirements. We construct semantic-motion examples by combining privately recorded gesture segments with natural co-speech motion contexts and varied preceding poses. Each event supplies its category, supported attributes, start signal, and elapsed time, while the target trajectory covers preparation, the expressive core, any required hold or repetition, and the transition to subsequent motion.
Let be the reference token at motion step and codebook . The motion objective is a weighted token cross-entropy:
Here, contains the active semantic and temporal conditions, is the motion history used for prediction, and masks invalid targets. The groups and weights balance temporal and codebook-depth prediction, including active body channels. Each counts valid targets across the logical update; empty groups are omitted. Initially, contains reference tokens from preceding motion steps and earlier codebooks within the current step.
Completion supervision distinguishes the gesture’s expressive core from readiness to release or transition. Let be the predicted completion probability for event-step sample and head . For valid labels , the balanced objective is
Here, contains valid samples with label for head , and contains its nonempty label classes. Positive and negative examples are normalized separately, preventing long incomplete prefixes from dominating the completion signal. The set contains heads with valid labels; examples with no valid completion labels contribute only to the motion objective.
The semantic motion stage optimizes
The motion term learns how to realize the requested gesture, while the completion term learns when its meaning has been expressed and when handover is possible. Full reference trajectories and durations are supervision targets; they are not online conditioning inputs. Natural examples remain in the mixture to retain ordinary co-speech behavior between explicit events.
Stage 3: Generated-History and Compositional Training. The final body-training stage broadens the contexts in which semantic events must succeed, including concurrent requests and model-generated motion histories. It mixes reference-history examples with examples whose temporal prefixes are generated by the corresponding model branch. For the latter, the prediction history becomes
where denotes generated tokens from preceding steps, while earlier codebooks within the current step remain teacher-forced. Equation (12) is retained with this history substituted into the motion objective. Thus, the model learns to realize the same event requirements from motion contexts affected by its own temporal prediction errors.
Completion labels are masked when they cannot be justified for the generated trajectory: a reference completion time alone does not imply that a diverging generated gesture has finished. Natural co-speech recordings and their derived windows or retimed variants remain in the same data partition. Development selection considers free-running decoded motion, semantic execution, and natural-motion retention. This stage develops the continuity and event-handover behavior needed to place successive requests within an ongoing response.
Facial-Expression Learning. For the complementary facial branch in § 3, the Chunk Transformer forms a condition from native Thinker and Talker states. Given a target facial chunk and Gaussian noise , conditional flow matching uses the interpolated state and the objective
Here, is flow time and . The velocity field learns to transform noise into facial motion conditioned on the response; flow time is distinct from the speech timestamps assigned to the resulting frames. This objective complements discrete body-token prediction, while both branches use the common response timeline for presentation. The spatial extension below specializes the body’s semantic event conditions and retains its motion and completion objectives.
Perception-to-Action Training
Perception-to-action training builds on the Thinker and Actor to connect multimodal understanding with embodied response generation. This stage extends the learned semantic event interface with task-specific conditions, allowing interaction context to guide both what a gesture expresses and how it unfolds with speech. We use Spatial Grounding and Pointing Training (SGPT) as a practical application, connecting visual target identification to a complete, spatially directed gesture within the ongoing response.
Stage 1: Target Grounding. Supervised observation–instruction pairs establish the correspondence between referring expressions and visual targets. Target localization is combined with depth estimation and geometric processing to obtain a metric position, which is transformed into the Actor’s body-centric frame:
Here, denotes the grounding result, the estimated depth, the camera intrinsics, and the camera-to-body transformation. The position supplies the spatial parameters of the semantic event, while the associated referring phrase provides its speech-alignment anchor. Directly supplied 3D coordinates use the same body-centric conditioning interface.
Stage 2: Task-Conditioned Event Learning. Building on the semantic-event and generated-history training in § 4.3, we continue Actor-side training to incorporate grounded targets into the existing event representation. The target position supplies the spatial parameters of , while native Thinker representations, Talker states, and motion history continue to provide communicative context, speech conditioning, and temporal continuity. The Speech–Action Aligner associates the event with its referring phrase, and completion control governs its continuation and handover. Target-conditioned demonstrations are mixed with natural co-speech examples using the existing Actor objective:
Here, denotes the trainable Actor-side parameters, contains pointing demonstrations, contains natural co-speech motion, and denotes the pointing sampling probability. The event conditions include spatial targets where available. Equation (12) continues to supervise motion prediction and event completion, extending the learned semantic control to scene-dependent pointing within ongoing motion. The Thinker and Talker remain frozen, following the preceding Actor training schedule.
Robot-Specific Execution. The generated human-motion representation and spatial target are converted into robot joint trajectories through a platform-specific kinematic adapter. X2 and Q1 use their respective coordinate transformations, kinematic models, pointing-axis calibration, and execution constraints to align the pointing ray while maintaining motion continuity. SGPT thus provides a concrete application of the broader perception-to-action training procedure through the Actor’s shared semantic event and motion-generation interfaces.
System Integration
System integration follows a staged training schedule to combine the frozen Thinker and Talker with the trained Actor for response generation. After multi-teacher on-policy distillation, the consolidated Thinker and the Talker are frozen, and Actor training proceeds with this fixed upstream model pair. This ordering establishes the semantic and speech-generation representations used to condition the Actor, avoiding changes to their source models during Actor training. The specialized experts are used only during Thinker training; the integrated system uses the consolidated Thinker as a single policy.
The Actor is trained through its semantic and speech-state interfaces while the native Thinker–Talker response pathway remains fixed. The coordinate-conditioned stage of SGPT follows the same Actor-side training arrangement, extending semantic events with spatial targets while retaining the fixed native response pathway. At inference, visual grounding supplies the target parameters, and the Actor combines them with the response semantics and evolving speech features through its shared event interface. This pathway carries the context used to understand and formulate a response into its bodily realization, with object pointing as one concrete application. The connection is learned on the Actor side while the upstream Thinker and Talker remain fixed.
Evaluation
Thinker Evaluation
We evaluate the Thinker on four benchmarks covering audiovisual understanding, multi-turn interaction, egocentric proximity, and embodied reasoning. Together, these tasks assess the perception and contextual reasoning needed to interpret human requests in dynamic environments.
General audiovisual understanding. Daily-Omni [36] evaluates audiovisual understanding across six capabilities using 30-second and 60-second videos. As shown in Table 2, XBot-Omni achieves 86.30% overall accuracy, outperforming the strongest baseline, Qwen3.5-Omni-Plus, by 1.56 percentage points. It also leads in every capability and both duration groups. The largest gains are in audiovisual alignment (+2.97 points) and event sequencing (+2.65 points), indicating improved association of acoustic and visual events and reasoning about their temporal order.
| Model | AV Align. | Comp. | Context | Event Seq. | Infer. | Reason. | 30s | 60s | Overall |
| Qwen3.5-Omni-Plus | 84.42 | 88.55 | 82.72 | 83.95 | 84.67 | 85.96 | 83.41 | 86.36 | 84.74 |
| Gemini 3.1 Pro Preview | 83.61 | 86.26 | 79.79 | 82.35 | 81.17 | 84.57 | 81.76 | 84.00 | 82.79 |
| Doubao Seed 2.1 Lite | 81.51 | 83.97 | 79.79 | 83.33 | 80.52 | 83.43 | 80.37 | 84.18 | 82.12 |
| XBot-Omni | 87.39 | 90.08 | 82.90 | 86.60 | 85.06 | 86.29 | 84.39 | 88.55 | 86.30 |
Table 2. Audiovisual understanding on Daily-Omni across six capability dimensions and two video-duration groups (30s and 60s). All scores are accuracy (%); the best result in each column is shown in bold.
Multi-turn embodied interaction. ReTurn [22] evaluates how models retain and update interaction context over five conversational turns, covering Reconfirm, Reground, Retrieve, and Rebind tasks. Table 3 shows that XBot-Omni achieves 84.00% overall accuracy, a 13.96-point gain over the strongest baseline. Improvements are particularly pronounced on Reground (67.20%, +37.60 points) and Rebind (88.48%, +33.44 points), where earlier interpretations or object references must be revised. Performance remains below the best baseline on Reconfirm and Retrieve, indicating that the overall advantage is driven primarily by contextual updating rather than uniformly higher accuracy across tasks.
| Model | Overall | Reconfirm | Reground | Retrieve | Rebind |
| MCQ: Long Conversations | |||||
| Doubao-Seed-2.1-Lite | 70.04 | 98.24 | 29.60 | 97.28 | 55.04 |
| Qwen3.5-Omni-Flash | 53.56 | 97.12 | 5.60 | 87.04 | 24.48 |
| XBot-Omni | 84.00 | 87.20 | 67.20 | 93.12 | 88.48 |
Table 3. Multi-turn conversational understanding on ReTurn across Reconfirm, Reground, Retrieve, and Rebind tasks, measured by multiple-choice accuracy (%) over five native interactive turns.
Egocentric proximity understanding. EgoProx [9] assesses approximate and relative object proximity from an egocentric viewpoint under Intention, Exploration, and Exploitation settings. As shown in Table 4, XBot-Omni leads on five of the six metrics. Approximate proximity accuracy reaches 65.54% under Intention and 62.57% under Exploration, exceeding the best baselines by 25.64 and 20.86 points, respectively. Relative proximity accuracy also improves in these settings by 3.21 and 7.85 points. Under Exploitation, XBot-Omni obtains the best approximate proximity score (69.43%), while its relative proximity score (57.72%) trails the best baseline by 0.14 points. The results therefore show the clearest gains in approximate proximity estimation, especially under Intention and Exploration.
| Model | Intention | Exploration | Exploitation | |||
| Approx. | Rel. | Approx. | Rel. | Approx. | Rel. | |
| Qwen3.5-Omni-Plus | 36.53 | 37.43 | 40.11 | 29.32 | 64.69 | 57.86 |
| Doubao Seed 2.1 Lite | 39.90 | 35.67 | 41.71 | 26.70 | 63.51 | 53.10 |
| XBot-Omni | 65.54 | 40.64 | 62.57 | 37.17 | 69.43 | 57.72 |
Table 4. Embodied proximity reasoning on EgoProx under Intention, Exploration, and Exploitation settings. Approx. and Rel. report approximate and relative proximity accuracy (%), respectively.
Embodied reasoning and grounding. ERIBench [8] evaluates intent inference, safety-boundary understanding, spatial relations, and visual grounding in embodied interaction. Table 5 shows that XBot-Omni leads on three of the four dimensions: safety boundaries (68.76%, +6.51 points), spatial relations (59.17%, +20.99 points), and grounding (69.81%, +2.27 points). The strongest improvement is in spatial reasoning, consistent with the gains observed on EgoProx. Intent inference is weaker than the strongest baseline (85.51% versus 91.45%), suggesting that the improvements in spatial understanding do not extend equally to all forms of intent reasoning.
| Model | Intent Inference | Safety Boundary | Spatial Relation | Grounding |
| Qwen3.5-Omni-Plus | 91.45 | 57.94 | 38.18 | 67.54 |
| Doubao-Seed-2.1-Lite | 90.05 | 62.25 | 25.66 | 56.20 |
| XBot-Omni | 85.51 | 68.76 | 59.17 | 69.81 |
Table 5. Embodied reasoning on ERIBench, covering intent inference, safety boundary understanding, spatial relations, and grounding. All scores are accuracy (%), with grounding evaluated at IoU ≥ 0.5; the best result in each column is shown in bold.
Motion Quality and Coordination
Evaluation setup. We compare XBot-Omni with EMAGE ([12]), GestureHYDRA ([29]), Semantic Gesticulator ([33]), and RoboGesture ([24]). For general motion evaluation, XBot-Omni reads 500 transcript excerpts from the official BEAT2 test split ([12]), yielding 499 valid speech samples. Baselines use the same generated audio and/or transcripts; our Actor uses corresponding native states. Released implementations, including Semantic Gesticulator’s Qwen-based retrieval route, are evaluated in a common body representation with fingers, face, and unsupported joints neutralized.
Motion metrics comprise beat consistency (BC), Fréchet Gesture Distance (FGD), and motion diversity (DIV). BC describes rhythmic agreement and DIV within-clip movement variation, without a preferred ranking direction. FGD compares generated-motion features against 265 human-motion recordings from the BEAT2 test split using a shared encoder and preprocessing; lower values indicate closer distributions, without clipwise pairing.
The separate semantic set has 119/120 valid short responses across ten targets: direction (left, right, up, down), spatial extent (large, small), and interaction (greeting, farewell, thanks, uncertainty). There are twelve responses per target except small (eleven after a shared speech failure). Gemini 3.8 Flash ([7]) observes anonymized, silent front/oblique motion videos at 4 fps without method identities, transcripts, or target labels. Its observations are matched locally to the intended target, yielding for absent, ambiguous or alternative-only, clear but partial or uncertain in completeness, and clear complete expression, respectively. Overall SQ averages target means:
where contains the ten targets and their scorable responses; family scores average their constituent target means. SQ measures expression quality across all scorable responses. CT instead measures speech alignment within the subset that has clear compatible expression () and valid speech alignment. It counts expressive-core overlap with MMS-aligned cue spans expanded by seconds:
where is 1 for an overlap and 0 otherwise. CT is expressed as a percentage, as reported in Table 6. Cue annotations are evaluation-only.
| Method | BC | FGD ↓ | DIV | Dir. | Extent | Inter. | Overall | CT ↑ (n/d) |
| EMAGE | 0.761 | 1.389 | 5.120 | 0.083 | 0.045 | 1.063 | 0.467 | 70.59% (12/17) |
| GestureHYDRA | 0.324 | 3.944 | 1.807 | 0.000 | 0.000 | 0.458 | 0.183 | 42.86% (3/7) |
| Semantic Gesticulator | 0.500 | 9.817 | 7.122 | 0.313 | 0.042 | 1.188 | 0.608 | 56.52% (13/23) |
| RoboGesture | 0.116 | 5.759 | 2.771 | 0.000 | 0.500 | 0.354 | 0.242 | 40.00% (2/5) |
| XBot-Omni | 0.622 | 1.351 | 5.423 | 1.750 | 1.000 | 1.208 | 1.383 | 85.19% (46/54) |
Table 6. Motion quality, semantic quality (SQ, 0–3), and conditional semantic timing (CT) under shared XBot-Omni-generated speech. Dir., Extent, and Inter. denote the three semantic families. Bold marks the lowest FGD and highest SQ values; BC and DIV are descriptive. CT reports timely/eligible counts and is not ranked because eligible subsets differ across methods.
Results. Table 6 shows that XBot-Omni achieves the lowest FGD and the highest overall and family-level SQ among the compared methods. Its overall SQ reaches 1.383, exceeding the strongest baseline, Semantic Gesticulator (0.608), by 0.775 points on the 0–3 scale. Under the same spoken content, this result indicates more recognizable and complete target-compatible body expression, supporting the Actor’s ability to turn communicative requirements into visible gestures.
The advantage extends across all three semantic families. For direction, XBot-Omni scores 1.750, compared with a best baseline score of 0.313; for spatial extent, it reaches 1.000, compared with 0.500. These results show stronger expression of directional and size-related meaning, which is central to communicating spatial information through body motion. For interaction gestures, XBot-Omni also obtains the highest family score (1.208), complementing its spatial-expression capabilities with conversational behavior. The family-level results therefore demonstrate gains across different kinds of communicative intent rather than an improvement confined to one gesture family.
For general motion quality, XBot-Omni achieves an FGD of 1.351, below EMAGE (1.389), GestureHYDRA (3.944), RoboGesture (5.759), and Semantic Gesticulator (9.817). This is the closest match to the BEAT2 human-motion feature distribution under the shared protocol. Together with the SQ results, it shows that stronger semantic expression is accompanied by a favorable motion-distribution score. BC (0.622) and DIV (5.423) additionally characterize the generated motion’s rhythmic agreement and movement variation; these descriptive measures complement the distributional and semantic assessments.
The Actor also produces more outputs eligible for semantic timing: 54 of 119 responses contain clear compatible expression with valid alignment, compared with 5–23 for the baselines. Of these, 46 are timely, giving a CT of 85.19%, compared with reported baseline rates of 40.00–70.59%. These results indicate that the Actor combines broader coverage of clear target-compatible gestures with speech-aligned execution in this cohort. CT is conditional on each method’s eligible outputs, so it is not an overall success rate or a paired comparison on an identical subset. The results are machine-assisted measurements on controlled short responses, without human-preference or streaming-latency claims.
Visually Grounded Motion
We examine object pointing as a practical case of using Omni understanding for embodied communication. SGPT connects visual target localization with coordinate-conditioned motion generation. We evaluate these capabilities separately through spatial perception benchmarks and a pointing test measuring accuracy, timing, and motion quality. The results characterize the components needed to express a visual reference; they do not by themselves measure end-to-end interaction success or isolate the effect of native Omni features.
Target grounding and spatial localization. Pointing toward a target first requires recovering its location from visual input. We evaluate spatial perception on TraceSpatial-Bench [35] using 2D and 3D start/end localization success rates, and on SUN RGB-D [21] using 3D object detection AP at IoU 0.25. The TraceSpatial-Bench 3D criterion uses a 20 cm tolerance; SUN RGB-D results cover five object categories. The two evaluations use their respective benchmark-specific perception configurations.
As shown in Table 7, XBot-Omni achieves 67.00% and 68.00% 3D localization success for start and end positions, improving over the strongest corresponding baselines by 24.00 and 29.00 points. Its 2D scores remain below the strongest baseline, indicating that the advantage is concentrated in 3D localization. On SUN RGB-D, XBot-Omni reaches 35.04% five-category [email protected], compared with 34.15% for the best baseline, and obtains the highest AP for chairs and sofas. These results indicate substantial improvements in 3D target localization and a smaller gain in category-averaged object detection.
| Method | 2D SR Start | 2D SR End | 3D SR Start | 3D SR End | Chair | Table | Bed | Toilet | Sofa | mAP |
| Qwen3.5-Omni-Flash | 50.00 | 46.00 | 43.00 | 39.00 | 10.21 | 9.63 | 49.36 | 19.55 | 24.14 | 22.58 |
| Qwen3.5-35B-A3B | 78.00 | 56.00 | 26.00 | 25.00 | 17.85 | 16.96 | 56.84 | 43.67 | 35.41 | 34.15 |
| XBot-Omni | 47.00 | 53.00 | 67.00 | 68.00 | 27.16 | 16.84 | 49.69 | 37.55 | 43.98 | 35.04 |
Table 7. Spatial perception on TraceSpatial-Bench and SUN RGB-D, evaluated by 2D/3D localization success rates (%) and 3D object detection [email protected] (%), respectively.
Coordinate-conditioned pointing. We evaluate whether SGPT improves the Actor’s ability to turn target coordinates into spatially accurate gestures with appropriate timing. The evaluation uses 100 development clips selected for high human-reference STG scores from the public spatially grounded gesture benchmark [4]. We report spatial accuracy (Score), temporal alignment (Score), and joint spatio-temporal performance (STG). Human motion serves as a reference, while the Actor without SGPT provides the ablation baseline.
We also assess motion quality using pointing-arm FGD, pointing-arm jerk, L1 velocity diversity, and beat alignment. Pointing-arm FGD compares reference and generated motion distributions using a fixed encoder after masking non-pointing-arm rotations. Jerk measures third-order central differences of root-relative arm joint positions (mm/frame), while velocity diversity captures within-clip variation in full-body frame-to-frame displacement. Beat alignment measures agreement between audio onsets and motion beats. These metrics complement, rather than replace, target-directed accuracy.
As shown in Table 8, SGPT increases Score from 0.357 to 0.990, Score from 0.654 to 0.851, and STG from 0.241 to 0.843. Spatial accuracy approaches the human reference (0.999), whereas temporal alignment remains below the human score of 0.985. Motion-quality changes are more modest: jerk decreases from 0.558 to 0.530, and velocity diversity moves closer to the human reference (0.696 to 0.650 versus 0.618). However, FGD rises slightly from 0.485 to 0.498, and beat alignment decreases from 0.432 to 0.423. The results therefore support improved spatial and temporal control of pointing gestures, without a uniform improvement in motion quality.
| Method | Spatial | Temporal | Joint | Motion Quality | |||
| Score ↑ | Score ↑ | STG ↑ | FGD ↓ | Jerk ↓ | Vel. Div. | Beat Align. ↑ | |
| Human reference | 0.999 | 0.985 | 0.984 | 0.389 | 0.502 | 0.618 | 0.393 |
| XBot-Omni w/o SGPT | 0.357 | 0.654 | 0.241 | 0.485 | 0.558 | 0.696 | 0.432 |
| XBot-Omni | 0.990 | 0.851 | 0.843 | 0.498 | 0.530 | 0.650 | 0.423 |
Table 8. Coordinate-conditioned pointing on 100 selected public development clips. Human motion is included as a reference. Bold indicates the better of the two generated models for metrics with a specified direction.
Related Work
Omni-LLMs. Omni-LLMs aim to unify multimodal understanding and generation across vision, audio, and language. Early approaches explore different strategies for modality unification. NExT-GPT (Wu et al., [25] connects a language model with modality-specific encoders and decoders, while AnyGPT (Zhan et al., [31] represents heterogeneous modalities as discrete token sequences. Subsequent models, including VITA (Fu et al., [5], Mini-Omni2 (Xie and Wu, [26], and Baichuan-Omni-1.5 (Li et al., [10], further extend these capabilities toward interactive audio-visual dialogue. Recent advances increasingly emphasize temporal alignment and streaming generation. Qwen2.5-Omni (Xu et al., [27] introduces time-aligned multimodal positional encoding and a Thinker–Talker architecture, which Qwen3-Omni (Xu et al., [28] further enhances with mixture-of-experts modeling and efficient speech synthesis. OmniVinci (Ye et al., [30] improves joint audio-visual understanding through representation alignment and temporal encoding, while MiniCPM-o 4.5 (Cui et al., [3] enables simultaneous perception and response within a unified streaming framework. Despite these advances, general-purpose Omni-LLMs primarily focus on multimodal perception and text or speech generation, without natively producing embodied body motion grounded in multimodal scene understanding. We introduce XBot-Omni, a Thinker–Talker–Actor architecture that extends Omni to coordinated speech and body-motion generation.
Embodied Interaction and Vision-Grounded Motion Generation. Speech-driven motion generation has traditionally focused on the relationship between acoustic structure, linguistic semantics, and body movement. EMAGE (Liu et al., [12] explores holistic co-speech motion generation. Semantic Gesticulator (Zhang et al., [33] and GestureHY-DRA (Yang et al., [29] investigate semantics-aware co-speech gesture synthesis. RoboGesture (Wang et al., [24] and LiveGesture (Saleem et al., [19] further investigate semantic motion generation for embodied interaction and streaming gesture generation. More recent studies integrate bodily behavior into conversational models. MIBURI (Mughal et al., [15] explores causal streaming gesture generation for spoken dialogue, ViBES (Zhang et al., [32] incorporates bodily behavior into a speech-language model, Motion-Omni (Ma et al., [13] jointly generates speech and full-body motion, and ProAct (Zhang et al., [34] combines streaming motion generation with behavioral reasoning. Beyond body motion, DiffSHEG (Chen et al., [2] explores holistic facial expression and gesture generation, while the GENEA Challenge series (Nagy et al., [17] advances the evaluation of speech-driven gesture generation. Despite these developments, directly incorporating visual scene understanding into the motion-generation process of an Omni model, particularly for generating gestures grounded in visually identified targets, remains underexplored. To the best of our knowledge, XBot-Omni is the first Omni-LLM-based embodied interaction framework to directly integrate scene-level visual understanding into communicative body-motion generation, enabling visually grounded and target-directed gestures.
Conclusion
In this work, we present XBot-Omni, an omni-modal model for embodied understanding and interaction that extends the Omni paradigm to robotic modalities, enabling coordinated generation of speech, body motion, and facial expressions. Built on the Thinker–Talker–Actor framework, XBot-Omni connects multimodal perception and reasoning with embodied response generation through native Thinker representations and evolving Talker states. The Actor further integrates semantic action control, speech–action alignment, and autoregressive motion generation to translate communicative intent into context-aware bodily expressions coordinated with unfolding speech. A complementary facial generation branch extends embodied responses beyond body motion. To support these capabilities, we construct large-scale embodied interaction data through an agent-driven annotation pipeline and develop a unified training strategy that combines supervised fine-tuning, specialized reinforcement learning, multi-teacher on-policy distillation, and staged Actor training. Furthermore, we introduce visually grounded motion to connect scene understanding with target-directed gestures, enabling robots to express visual references through physical actions. These advances provide a unified framework for connecting multimodal understanding, conversational response generation, and embodied expression. We hope XBot-Omni provides a strong foundation for future research on general-purpose embodied agents that can perceive, reason, communicate, and interact naturally in real-world environments.
References
Core Contributors and Contributions
Core Contributors: Kerui Gu, Xing Fu, Shuoyang Sun, Cheng Peng, Jingru Duan, Huayu Gu, Shaoli Huang.
| Overall Architecture | Kerui Gu, Xing Fu, Shuoyang Sun, Cheng Peng |
| Thinker-Talker Training | Xing Fu, Huayu Gu, Kerui Gu |
| Actor Training | Shuoyang Sun, Cheng Peng, Jingru Duan, Kerui Gu |
| Omni-Understanding Data | Xing Fu, Zhenyu Chen, Huayu Gu, Shuoyang Sun, Jingru Duan, Kerui Gu |
| Omni-Motion Data | Shuoyang Sun, Cheng Peng, Huayu Gu, Cheng Wen, Kairui Chen, Kerui Gu |
| Evaluation | Xing Fu, Shuoyang Sun, Jingru Duan, Cheng Peng |
| Robot Deployment | Cheng Peng, Jingru Duan, Xing Fu, Rui Zhen, Kerui Gu |
| Simulation | Cheng Peng, Huayu Gu, Shuoyang Sun, Jiaxin Huang |
| Advisor | Zhihui Peng |
| Project Lead | Shaoli Huang |
References
- [1]AGIBOT. AGIBOT X2: Humanoid robot. Official Product Website, 2026. URL https://www.agibot.com/products/X2. Accessed: 2026-11-12.
- [2]Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation. In CVPR, 2024.arxiv.org/abs/2401.04747
- [3]Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, et al. MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction. arXiv preprint arXiv:2604.27393, 2026. URL https://arxiv.org/abs/2604.27393.
- [4]Anna Deichler, Rishabh Dabral, Fethiye Irmak Dogan, Anindita Ghosh, and Jonas Beskow. A benchmark for spatially grounded gesture generation. arXiv preprint arXiv:2610.03105, 2026.arxiv.org/abs/2610.03105
- [5]Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, et al. VITA: Towards Open-Source Interactive Omni Multimodal LLM. arXiv preprint arXiv:2408.05211, 2024. URL https://arxiv.org/abs/2408.05211.
- [6]Google. Gemini 3.1 Pro Preview. Official model documentation, 2026. URL https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview. Accessed November 12, 2026.
- [7]Google DeepMind. Gemini 3.8 Flash. Model card, September 2026. URL https://deepmind.google/models/model-cards/gemini-3-8-flash/. Published September 2, 2026; accessed November 12, 2026.
- [8]Huayu Gu, Menghan Xia, Shaoli Huang, Wenjie Xuan, Juhua Liu, and Bo Du. ERIBench: A comprehensive benchmark for embodied reference interaction in human-robot scenarios. Manuscript available from the official project repository, 2026. URL https://github.com/HuayuGu/ERIBench/blob/main/ERIBench.pdf.
- [9]Jinzhao Li, Yinuo Chen, Dongxu Piao, Panwang Pan, Yifan Yu, Dong Wang, Honglei Yan, Liang Yue, Shaofei Wang, Yixin Chen, Siyuan Huang, and Miao Liu. EgoProx: Evaluating MLLMs on egocentric 3D proximity reasoning across a cognitive hierarchy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23751–23762, 2026. URL https://openaccess.thecvf.com/content/CVPR2026/html/Li_EgoProx_Evaluating_MLLMs_on_Egocentric_3D_Proximity_Reasoning_Across_a_CVPR_2026_paper.html.arxiv.org/abs/2605.24456
- [10]Yadong Li, Jun Liu, Tao Zhang, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, et al. Baichuan-Omni-1.5 Technical Report. arXiv preprint arXiv:2501.15368, 2025. URL https://arxiv.org/abs/2501.15368.
- [11]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/forum?id=PqvMRDCJ19t.arxiv.org/abs/2210.02747
- [12]Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. EMAGE: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/html/Liu_EMAGE_Towards_Unified_Holistic_Co-Speech_Gesture_Generation_via_Expressive_Masked_CVPR_2024_paper.html.arxiv.org/abs/2401.00374
- [13]Chengqian Ma, Wei Tao, Haoyu Zhang, and Yiwen Guo. Motion-Omni: End-to-end joint speech and full-body motion for spoken dialogue. arXiv preprint arXiv:2609.04250, 2026a. URL https://arxiv.org/abs/2609.04250.
- [14]Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, and Fuli Luo. Modp: Multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406, 2026b. doi: 10.48550/arXiv.2606.30406. URL https://arxiv.org/abs/2606.30406.
- [15]M. Hamza Mughal, Rishabh Dabral, Vera Demberg, and Christian Theobalt. Miburi: Towards expressive interactive gesture synthesis. In Computer Vision and Pattern Recognition (CVPR), 2026.arxiv.org/abs/2603.03282
- [16]Koki Nagano, Hongyu Liu, Seonwook Park, Tianye Li, Amrita Mazumdar, Christian Jacobsen, Shengze Wang, Michael Stengel, Rajarshi Roy, Ka Chun Cheung, Simon See, and Shalini De Mello. DyaPlex: Full-duplex speech-motion model for dyadic interaction. In Advances in Neural Information Processing Systems, 2026. URL https://research.nvidia.com/labs/amri/projects/DyaPlex/.arxiv.org/abs/2606.03874
- [17]Rajmund Nagy, Silvia Arellano García, Hendric Voss, Mihail Tsakov, Taras Kucherenko, Youngwoo Yoon, and Gustav Eje Henter. The GENEA challenge 2026: A large-scale disentangled evaluation of speech-driven gesture generation on the seamless interaction dataset. arXiv preprint arXiv:2608.10839, 2026. URL https://arxiv.org/abs/2608.10839.
- [18]PrimeBOT. PrimeBOT Q1: Humanoid robot. Official Product Website, 2026. URL https://www.primebot.com/q1.html. Accessed: 2026-11-12.
- [19]Muhammad Usama Saleem, Mayur Jagdishbhai Patel, Ekkasit Pinyowantupong, Zhongxing Qin, Li Yang, Hongfei Xue, Ahmed Helmy, Chen Chen, and Pu Wang. LiveGesture: Streamable co-speech gesture generation model. arXiv preprint arXiv:2604.10927, 2026. URL https://arxiv.org/abs/2604.10927.
- [20]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [21]Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A RGB-D scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 567–576, 2015. URL https://openaccess.thecvf.com/content_cvpr_2015/html/Song_SUN_RGB-D_A_2015_CVPR_paper.html.DOI
- [22]Shuoyang Sun, Kerui Gu, Hao Fang, Shaoli Huang, and Bin Chen. When history helps and hurts: Selective history use across multimodal turns, 2026. URL https://arxiv.org/abs/2610.11948.
- [24]Zifan Wang, Ziang Ren, Pengyang Shi, Zirui Wang, Chenghuai Lin, Tianze Wang, Zekun Qi, Liangliang Zhao, He Wang, and Li Yi. Robogesture: Real-time semantic-aligned co-speech gestures generation for humanoid interaction, 2026. URL https://arxiv.org/abs/2608.28693.
- [25]Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NExT-GPT: Any-to-Any Multimodal LLM. In Proceedings of the International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2309.05519.
- [26]Zhifei Xie and Changqiao Wu. Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities. arXiv preprint arXiv:2410.11190, 2024. URL https://arxiv.org/abs/2410.11190.
- [27]Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni technical report. arXiv preprint arXiv:2503.20215, 2025a.arxiv.org/abs/2503.20215
- [28]Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-Omni Technical Report. arXiv preprint arXiv:2509.17765, 2025b. URL https://arxiv.org/abs/2509.17765.
- [29]Quanwei Yang, Luying Huang, Kaisiyuan Wang, Jiazhi Guan, Shengyi He, Fengguo Li, Hang Zhou, Lingyun Yu, Yingying Li, Haocheng Feng, and Hongtao Xie. GestureHYDRA: Semantic co-speech gesture synthesis via hybrid modality diffusion transformer and cascaded-synchronized retrieval-augmented generation. In IEEE/CVF International Conference on Computer Vision, ICCV 2025, Honolulu, HI, USA, October 19-25, 2025, pages 12615–12625. IEEE, 2025. doi: 10.1109/ICCV51701.2025.01172. URL https://doi.org/10.1109/ICCV51701.2025.01172.
- [30]Hanrong Ye, Chao-Han Huck Yang, Arushi Goel, Wei Huang, Ligeng Zhu, Yuanhang Su, Sean Lin, An-Chieh Cheng, Zhen Wan, Jinchuan Tian, et al. OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2510.15870.
- [31]Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. URL https://arxiv.org/abs/2402.12226.
- [32]Juze Zhang, Changan Chen, Xin Chen, Heng Yu, Tiange Xiang, Ali Sartaz Khan, Shrinidhi K. Lakshmikanth, and Ehsan Adeli. ViBES: A conversational agent with behaviorally-intelligent 3D virtual body. arXiv preprint arXiv:2512.14234, 2025. URL https://arxiv.org/abs/2512.14234.
- [33]Zeyi Zhang, Tenglong Ao, Yuyao Zhang, Qingzhe Gao, Chuan Lin, Baoquan Chen, and Libin Liu. Semantic gesticulator: Semantics-aware co-speech gesture synthesis. ACM Trans. Graph., 43(4):136:1–136:17, 2024. doi: 10.1145/3658134. URL https://doi.org/10.1145/3658134.
- [34]Zeyi Zhang, Zixi Kang, Ruijie Zhao, Yusen Feng, Biao Jiang, Hanyu Ji, and Libin Liu. ProAct: Harnessing streaming motion generation and agentic reasoning for real-time embodied social interaction. arXiv preprint arXiv:2602.14048, 2026. URL https://arxiv.org/abs/2602.14048v2. Version 2.
- [35]Enshen Zhou, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Mengzhen Liu, Yi Han, Yuheng Ji, Huajie Tan, Jiawei He, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Lu Sheng, and Shanghang Zhang. Towards spatial trace with reasoning in vision-language models for robotics, 2026. URL https://arxiv.org/abs/2512.13660v4.
- [36]Ziwei Zhou, Rui Wang, Zuxuan Wu, and Yu-Gang Jiang. Daily-Omni: Towards audio-visual reasoning with temporal alignment across modalities. arXiv preprint arXiv:2505.17862, 2025.arxiv.org/abs/2505.17862

