Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

Introduction

The rapid progress of multimodal large language models (MLLMs) is reshaping Embodied AI, driving a growing line of generalist embodied models that harness MLLMs’ multimodal understanding and reasoning capabilities for physical-world perception and interaction. Representative systems such as PaLM-E [35], RoboBrain [57, 13], RynnBrain [30] and Vesta [10] demonstrate the potential of this paradigm. However, while modern MLLMs have developed strong capabilities across modalities such as images, video, and audio, their support for diverse physical signals remains limited.

The real world contains rich information that can be captured by embodied sensory modalities, such as depth, thermal, tactile, and radar. These modalities provide complementary physical cues about geometry, contact, temperature, and dynamics beyond basic modality inputs. As Figure 1 illustrates, such cues can clarify physical states that remain ambiguous in basic observations. Accordingly, recent work has explored three main strategies for incorporating sensory modalities into language-centric models. (i) Specialists develop dedicated interfaces or adaptation mechanisms for individual sensory modalities [70, 65] (Asif Imran Shouborno et al. 2025). (ii) Binders [44] (Zhu et al. 2024) use image or language representations as semantic anchors to align different modalities within a shared representation space. (iii) Generalists [47, 88, 66] (Wu et al. 2024) integrate multiple modalities through modality-specific encoders, projectors, or routing modules. Despite these differences, most sensory modality adaptation is still concentrated at the modality interface, before entering the LLM. A more detailed discussion of related work is provided in Appendix § 5.

This emphasis largely reflects the maturity gap in modality interfaces between embodied sensory and widely studied basic modalities. On the one hand, the latter benefit from well-established modality-specific encoders, web-scale multimodal data, and widely adopted alignment recipes [96, 26] (Tschannen et al. 2025), whereas many embodied sensory modalities still lack comparable resources, making sensory–language grounding a primary challenge. On the other hand, despite heterogeneous low-level representations, these modalities ultimately rely on shared high-level understanding and reasoning, motivating a unified LLM backbone. For basic modalities, mature interfaces have further enabled omni-modal models to move beyond and introduce modality-dependent computation within the LLM [69, 56] (Shukor et al. 2025). For embodied sensory modalities, unified MLLM-side specialization remains limited: existing methods largely share the same parameters across representations with substantially different structures, statistics, and spatiotemporal characteristics. Prior studies suggest that such parameter sharing across modalities or tasks can introduce conflicting learning signals [21, 71, 48]. We therefore view unified embodied sensory modeling across two levels: semantic grounding at the modality–language interface, followed by computational specialization within the shared LLM.

In this paper, we present UniSense, a unified MLLM for embodied sensory understanding. UniSense supports diverse embodied sensory modalities through lightweight modality tokenizers, a unified encoder, modality projectors, and a shared LLM. Building on this architecture, UniSense follows a two-stage training process to address the above challenges. (i) Aligning different sensory representations with language through caption supervision, making physical sensory information accessible to the LLM, and then (ii) tuning modality-specific experts within the LLM, allowing different sensory modalities to specialize beyond the shared backbone computation.

To support UniSense, we develop a data construction pipeline that applies modality-specific preprocessing to heterogeneous sensory observations and pairs the processed inputs with modality-aware captions and instruction–response pairs. This results in a large-scale training corpus of 1M samples spanning 8 embodied sensory modalities, alongside general multimodal data. We further evaluate UniSense on 12 benchmarks covering depth, point cloud, IMU, tactile, thermal, radar, motion, and smell. Experiments show that UniSense outperforms general multimodal and specialized models on most benchmarks. Our contributions are summarized as follows:

• We present UniSense, a unified MLLM that supports text, image, video, audio, and diverse physical sensing modalities within a shared multimodal architecture.

• We develop a two-stage training strategy that aligns and specializes sensory processing, enabling language-accessible sensory representations and modality-specific specialization within a shared LLM.

• We curate UniSense–1M, a multimodal training corpus spanning 8 embodied sensory modalities, and evaluate UniSense on 12 benchmarks for embodied sensory understanding, demonstrating its effectiveness across heterogeneous sensing tasks.

Method

Problem Definition

We consider a unified embodied sensory understanding task, where a multimodal LLM supports heterogeneous modalities under a shared framework. Let $\mathcal{M}$ denote the set of supported non-text modalities, including basic modalities (image, video, and audio), and embodied sensory modalities $\mathcal{M}_s \subseteq\mathcal{M}$. In this work, $\mathcal{M}_s$ comprises depth, point cloud, IMU, tactile, thermal, radar, motion, and smell. For modality $m \in\mathcal{M}$, we denote its input by $x^m$. The language instruction provided to the model is denoted by $q$.

Our goal is to incorporate embodied sensory modalities in $\mathcal{M}_s$ into a MLLM while considering both how their representations are grounded in the model’s language space and how they are subsequently processed within the shared backbone.

Model Architecture

Figure 2 illustrates the architecture of UniSense, which consists of modality-specific tokenizers, a unified encoder, modality-specific projectors, and a shared LLM. We build UniSense on pretrained Qwen3.5 models [95] at 2B and 9B scales, with preliminary multimodal initialization on image and audio data but no additional video-specific training (see Appendix § 7.§ 3 and §C.6).

For an embodied sensory modality $s \in\mathcal{M}_s$, the input $x^s$ is first transformed into encoder-compatible tokens by a tokenizer $\mathcal{T}_s$. We adopt two tokenization pathways: pixel-aligned modalities (e.g., depth, thermal, and tactile) reuse the standard visual patch tokenizer, whereas non-pixel modalities (e.g., point cloud, IMU, and radar) employ dedicated modality-specific tokenizers. The resulting tokens are then processed by a unified encoder $\mathcal{E}$ and mapped to the hidden space of the language model through a modality-specific projector $\mathcal{P}_s$:

where $\mathbf{Z}^{s}$ denotes the resulting sensory representation. Image, video, and audio are processed through their corresponding input pathways. Accordingly, we use $\mathbf{Z}^{m}$ to denote the resulting representation of any non-text modality $m \in\mathcal{M}$ provided to the LLM. Detailed tokenizer and projector designs are provided in Appendix §C.2.

The resulting modality representations $\{\mathbf{Z}^{m}\}$ are combined with the language instruction $\mathbf{q}$ and processed by the LLM to generate the response. Although these interfaces make heterogeneous inputs compatible with the pretrained backbone, they do not ensure effective language grounding or appropriate processing. We address these two aspects through the two training stages described in §2.3 and §2.4.

Sensory Representation Alignment

Unlike basic modalities, which have been well studied, embodied sensory modalities lack comparably mature language-aligned representations, making their physical semantics difficult to ground in the target MLLM. Prior work aligns heterogeneous modalities in shared embedding spaces [44] (Zhu et al. 2024), while generative–contrastive frameworks further show that generative supervision and contrastive learning provide complementary semantic signals (Yu et al., 2022; Xiao et al., 2025). However, such alignment is typically defined outside the representation structure of the target MLLM that ultimately consumes these modality features. Recent MLLM studies suggest that grounding modality representations toward the pretrained linguistic structure of the target LLM can improve their integration [84, 19].

To align embodied sensory representations with language, we construct a modality-aware caption for each sensory input that explicitly verbalizes its physical information. We use each caption to supervise sensory representations in two complementary ways: conditional captioning encourages sensory representations to retain physical semantics recoverable through the target MLLM’s native language interface, while contrastive alignment uses representations induced by the target MLLM itself to establish explicit sensory–language correspondence. The overall design is illustrated in Figure 3, with examples of modality-aware captions provided in Appendix §E.

For a sensory modality $s \in\mathcal{M}_s$, let $\mathbf{Z}^{s} \in\mathbb{R}^{N_s \times d}$ denote its projected sensory representation and $\mathbf{C}=(c_1,\ldots,c_L)$ the corresponding modality-aware caption.

Conditional Captioning. We impose language-level supervision on sensory representations by predicting their paired modality-aware captions. Given the projected sensory tokens $\mathbf{Z}^{s}$, a modality-specific caption decoder $\mathcal{C}^{s}$ predicts each token of $\mathbf{C}$ using causal self-attention over preceding caption tokens and cross-attention to $\mathbf{Z}^{s}$. The decoder reuses the token embeddings and vocabulary projection of the target LLM, encouraging $\mathbf{Z}^{s}$ to preserve physical semantics that can be recovered through the model’s native language interface. We optimize this branch with the standard autoregressive token-level cross-entropy loss, denoted by $\mathcal{L}_{\mathrm{cap}}$.

Contrastive Alignment. We further establish representation-level correspondence between sensory inputs and their captions using representations induced by the target LLM. Specifically, $\mathbf{Z}^{s}$ and the embedded caption $\operatorname{Emb}(\mathbf{C})$ are independently processed by the same first $k$ frozen LLM layers, followed by modality-specific aggregation and separate lightweight projection heads:

Here, $\rho_s$ and $\rho_c$ denote the sensory and caption aggregation operations, respectively, $\phi_s$ and $\phi_c$ are separate projection heads, and $\operatorname{Norm}(\cdot)$ denotes $\ell_2$ normalization. For a batch of $B$ sensory–caption pairs, representations from the same sensory instance form a positive pair, while those from different sensory instances serve as in-batch negatives. We optimize the resulting representations with a standard symmetric InfoNCE loss, denoted by $\mathcal{L}_{\mathrm{con}}$.

Modality-Specific Expert Tuning

The first training stage grounds embodied sensory representations in the language space of the target MLLM. However, these heterogeneous representations are still processed by the same decoder parameters, which may introduce interference across modalities or tasks [99, 71]. While recent multimodal models introduce modality- or task-specific computation within the backbone [69, 56], embodied sensory modeling remains largely focused on modality-side adaptation [47, 70] (Zhou & Yang 2026).

We therefore introduce modality-specific capacity within the shared LLM, while keeping self-attention shared and specializing only the feed-forward sublayers. As shown in Figure 4, the base FFN remains as the shared transformation, with modality-specific residual experts providing specialized processing.

Modality-Aware Routing. We organize sensory-specific computation through two levels of routing. For each sensory modality $s$, we maintain a pool of $E$ lightweight residual experts $\{R_{s,i}\}_{i=1}^{E}$. A modality router first directs each sensory token to its corresponding expert pool, after which a token-wise router $g_s$ further determines token-dependent expert contributions. This allows tokens from the same modality to receive different residual transformations.

Residual Expert Adaptation. For a hidden state $\mathbf{h}$ associated with sensory modality $s$, the modality-specific residual transformation is defined as:

where $\mathbf{w}_s=[w_{s,1},\ldots,w_{s,E}]$ denotes the normalized expert weights, $r$ is the low-rank dimension, and $\alpha$ controls the residual scale. Each $R_{s,i}$ follows the SwiGLU-based FFN structure of the LLM backbone [95], with a low-rank parameterization.

The resulting residual is then added token-wise to the output of the shared pretrained FFN:

where $\mathcal{F}_{\mathrm{base}}$ denotes the frozen FFN shared across modalities. Thus, each token receives an input-dependent sensory-specific correction while retaining the shared pretrained transformation.

The residual expert modules are introduced only in selected full-attention Transformer layers, while the remaining layers stay unchanged. We further use balance and diversity regularization to stabilize expert utilization and discourage redundant residual transformations. More details are provided in Appendix § 7.5.

(a) Image-text benchmarks.
Model	TextVQA$_{\mathrm{val}}$	MME	MMBench
General MLLMs
Qwen3-VL-8B-Instruct	–	2243	84.5
GLM-4.5V	72.0	2423.8	88.2
GLM-4.1V-9B	79.6	2445.8	85.8
Kimi-VL-A3B-2506	77.7	2352.7	84.4
InternVL3.5-8B	78.2	2380.6	79.5
InternVL3.5-30B-A3B	80.5	2461.9	84.8
UniSense
UniSense-2B	79.5	2247.9	79.6
UniSense-9B	83.6	2466.0	87.5
(b) Audio-image-text benchmarks.
Model	TextVQA$_{\mathrm{s}}$	DocVQA$_{\mathrm{s}}$	ChartQA$_{\mathrm{s}}$
General MLLMs
Intern-Omni	69.1	79.9	56.0
Qwen3-Omni-30B-A3B	70.8	76.4	–
Lyra-Mini-3B	73.4	74.8	40.7
Lyra-Base-9B	80.0	85.5	61.0
UniSense
UniSense-2B	73.8	80.0	53.8
UniSense-9B	80.4	87.2	68.0
(c) Video-text benchmarks.
Model	Video-MME	EgoSchema
General MLLMs
InternVL2.5-7B	64.2	51.5
InternVL3.5-2B	58.4	50.8
InternVideo2.5-7B	65.1	63.9
Lyra-Base-9B	62.8	63.2
Qwen2-VL-2B	55.6	54.9
Qwen2.5-VL-7B	65.1	65.0
UniSense
UniSense-2B	59.4	49.8
UniSense-9B	69.8	66.4

Training

After the preliminary image–audio initialization described in §2.2, we train UniSense on a unified collection of embodied sensory datasets spanning 8 sensory modalities. Each sensory sample is paired with a modality-aware caption used for sensory representation alignment. Detailed data sources and sampling strategies are provided in §B.2.

Training proceeds in two stages. The first aligns sensory representations with language, while the second tunes modality-specific experts within the shared LLM.

Alignment Stage. We freeze the pretrained LLM backbone and optimize the newly introduced sensory tokenizers, projectors, caption decoders, and contrastive projection heads. Given a sensory input and its instruction, $\mathcal{L}_{\mathrm{LM}}$ is the autoregressive cross-entropy loss over the target response tokens. The paired modality-aware caption further provides the captioning objective $\mathcal{L}_{\mathrm{cap}}$ and the contrastive objective $\mathcal{L}_{\mathrm{con}}$ introduced in §2.3. The overall objective is:

where $\lambda_{\mathrm{align}}$ controls the overall contribution of the alignment objectives, and $\lambda_{\mathrm{con}}$ determines the relative weight of the contrastive objective.

Tuning Stage. The pretrained LLM backbone remains frozen, while we optimize the sensory modules, modality routers, and residual experts. We retain the language-modeling and captioning objectives, but disable the contrastive branch to permit task-driven specialization. We further apply the load-balancing loss $\mathcal{L}_{\mathrm{bal}}$ and expert-diversity loss $\mathcal{L}_{\mathrm{div}}$ to regularize expert learning. The second-stage objective is:

where $\lambda_{\mathrm{align}}$, $\lambda_{\mathrm{bal}}$, and $\lambda_{\mathrm{div}}$ weight the captioning, load-balancing, and expert-diversity terms, respectively. Detailed settings are provided in Appendix §C.6.

Experiments

Experimental Setup

Due to space limitations, we provide additional experimental details in the appendix:

• Datasets and benchmarks. Details of UniSense–1M, including data sources and preparation, together with benchmark splits, evaluation protocols, and metrics, are provided in Appendix §B.2.

• Implementation and training. Model configurations, optimization settings, freezing schedules, and training hyperparameters are provided in Appendix §C.

Model	PC	PSS	POM
Closed-Source Models
GPT-4o	30.9	38.1	22.6
Gemini-2.5-Pro	26.7	25.1	23.5
General MLLMs
Qwen2.5-VL-7B	21.5	23.1	19.9
LLaVA-OneVision-7B	35.4	28.4	29.1
TokenFlow-7B	38.1	32.6	29.5
Tactile-Specialized Models
Octopi-7B	45.5	48.0	22.2
UniTac-3B	55.0	58.9	42.1
UniTac-7B	57.3	59.2	64.6
UniSense
UniSense-2B	58.4	72.8	45.5
UniSense-9B	61.9	73.4	53.3

Main Results

In this section, we evaluate UniSense on general multimodal tasks and benchmarks spanning 8 embodied sensory modalities. These evaluations examine both its general multimodal capability and its ability to handle heterogeneous physical sensing signals. Unless otherwise specified, higher scores indicate better performance across all reported metrics.

General Multimodal Understanding. UniSense-9B achieves 83.6 on TextVQA (Singh et al., 2019) and 2466.0 on MME [38], leading the compared models on both benchmarks. It also performs strongly on speech-conditioned image and video understanding (Table 1), indicating that sensory training preserves its basic-modality capabilities.

Depth and Point Cloud Understanding. Table 2 and Table 3 evaluate spatial and 3D understanding. UniSense-9B achieves 70.1 on the SpatialBench [12] 4-task aggregate and 80.5 accuracy on RoboSpatial-Home (Song et al., 2025) without benchmark-specific training, surpassing both general-purpose MLLMs and spatial-specialized models. On ScanQA [6], UniSense-9B achieves 74.3 CIDEr using point clouds alone, which increases to 99.6 when combined with images, outperforming all compared baselines. This improvement shows that newly introduced sensory inputs can be naturally composed with basic modality observations to exploit complementary geometric and appearance information.

Model	Pos.	Exist.	Count.	Reach.	Size	4-Task Agg.	Agg.	Config.
Closed-Source Models
GPT-4o	61.8	90.0	85.2	51.7	40.0	68.1	65.3	77.2
Claude-3.5	55.9	65.0	72.2	–	26.7	55.1	–	–
General MLLMs
LLaVA-1.5-7B	44.1	45.0	82.8	–	30.0	49.5	–	–
LLaVA-NeXT-7B	47.1	75.0	84.0	–	20.0	55.1	–	68.3
Qwen2-VL-7B	–	–	–	–	–	–	–	76.4
Qwen2.5-VL-3B	55.9	80.0	76.4	–	25.0	58.8	–	–
Qwen2.5-VL-7B	61.8	80.0	87.1	–	30.0	64.3	–	78.1
Spatial-Specialized Models
SpatialVLM-3B	52.9	80.0	77.1	–	28.3	58.6	–	–
Bunny-Phi2-3B	50.0	75.0	89.4	51.7	26.7	58.7	57.5	–
SpatialBot-Phi2-3B	61.8	80.0	91.7	55.0	26.7	64.6	62.9	–
Bunny-Llama3-8B	50.0	75.0	91.7	38.3	23.3	58.5	55.0	–
SpatialBot-Llama3-8B	53.0	75.0	90.4	45.0	20.0	58.6	56.2	–
SpaceThinker-3B	50.0	82.5	70.0	57.5	22.5	55.3	56.5	–
SpaceOm	67.7	92.5	75.0	55.0	37.5	68.1	65.8	–
SSR-7B	64.7	85.0	90.2	–	28.3	66.7	–	–
VPT	–	–	–	–	–	–	–	77.2
SpatialVTS	–	–	–	–	–	–	–	70.7
UniSense
UniSense-2B	67.7	80.0	89.2	58.3	31.7	67.2	65.5	73.2
UniSense-9B	76.5	80.0	91.1	58.3	28.3	70.1	68.0	80.5

Model	Input	CIDEr	MET.	R-L	B-1	B-4
General MLLMs
InternVL2-8B	img	62.5	14.5	34.3	–	3.3
Qwen2-VL-7B	img	53.9	11.4	29.3	27.8	3.0
LLaVA-Video	MV-Img	88.7	17.7	44.6	–	3.1
3D-Specialized Models
ScanQA	xyz	64.9	13.1	33.3	30.2	10.1
LL3DA	xyz	76.8	15.9	37.3	–	13.5
Scene-LLM	xyz	80.0	16.6	40.0	–	12.0
ChatScene	xyz+img	87.7	18.0	41.6	43.2	14.3
GPT4Scene	MV-Img	90.9	17.7	43.6	43.4	14.6
ViewMind3D	MV-Img	73.4	15.3	37.6	35.8	–
UniSense
UniSense-2B	xyz	65.1	21.1	34.0	36.4	11.0
UniSense-2B	xyz+img	90.0	27.5	44.6	44.3	12.7
UniSense-9B	xyz	74.3	23.4	36.9	39.2	12.4
UniSense-9B	xyz+img	99.6	30.2	47.5	48.3	15.5

Model	Zero-Shot	Fine-Tuning
General MLLMs
ImageBindLLM-7B	24.2	48.5
PandaGPT-7B	26.8	41.3
PandaGPT-13B	34.3	46.0
Thermal-Specialized Models
Infrared-LLaVA-7B	56.6	79.0
UniSense
UniSense-2B	–	86.8
UniSense-9B	70.2	89.9

Model	Input	Bin. F1	Mul. F1	R-L	MET.
Motion-Specialized Models
Naive	TS	35.6	9.6	6.0	5.8
ChatTS	TS	34.8	29.1	4.6	10.6
xQA-Llama	GT	78.2	81.5	9.1	36.8
xQA-Qwen	GT	78.4	88.0	17.9	37.0
xQA-Llama	AE	78.3	81.2	9.0	36.8
xQA-Qwen	AE	78.5	88.0	17.8	36.9
UniSense
UniSense-2B	TS	89.7	97.3	82.1	72.3
UniSense-9B	TS	97.4	99.6	85.0	74.2

Model	Acc@1	F1
Smell-Specialized Models
MLP	26.8	24.7
CNN	52.7	50.5
LSTM	57.9	56.0
ScentFormer	56.1	55.5
UniSense
UniSense-2B	65.2	63.4
UniSense-9B	74.5	72.2

IMU, Radar, and Motion Understanding. Across four IMU activity-recognition datasets, UniSense-9B outperforms LLaSA (Asif Imran Shoubornо et al., 2025) on three tasks and matches its F1 on Shoaib (Table 5). On radar, it outperforms HoloLLM (Zhou & Yang, 2026) in both action recognition and captioning, with a 21.6-point gain in caption METEOR (Table 7a). On QuAnTS [34], it answers motion questions directly from skeleton sequences, outperforming baselines given ground-truth actions or pretrained action features (Table 7b).

Model	HHAR	MotionSense	Shoaib	UCI
Closed-Source Models
GPT-4o	7.0 / 14.0	8.0 / 26.0	14.0 / 35.0	4.0 / 16.0
General LLMs
Vicuna-7B	5.0 / 12.0	9.0 / 15.0	8.0 / 11.0	4.0 / 25.0
IMU-Specialized Models
GenHAR	63.0 / –	81.0 / –	74.0 / –	79.0 / –
LanHAR	70.0 / –	76.0 / –	71.0 / –	75.0 / –
LLaSA-7B	84.0 / 88.0	83.0 / 85.0	81.0 / 84.0	72.0 / 82.0
UniSense
UniSense-2B	92.0 / 92.0	75.0 / 79.0	72.0 / 76.0	87.0 / 87.0
UniSense-9B	97.0 / 97.0	87.0 / 89.0	81.0 / 85.0	90.0 / 90.0

Model	Human Action QA	Action Caption
General MLLMs
Tokenpacker	4.7	6.0
Honeybee	1.5	10.4
OneLLM	1.1	2.2
ImageBind	24.0	17.5
Radar-Specialized Models
HoloLLM	61.4	24.5
UniSense
UniSense-2B	61.0	46.0
UniSense-9B	65.2	46.1

Tactile, Thermal, and Smell Understanding. We further test UniSense on tactile, smell, and thermal modalities in Tables 4, 7c, and 6. On Octopi (Yu et al., 2024), UniSense-9B leads on property comparison and superposition, reaching 61.9 and 73.4. On Infrared-Benchmark [60], it reaches 70.2 zero-shot and 89.9 after fine-tuning, outperforming Infrared-LLaVA [60] by 13.6 and 10.9 points. For smell recognition [37], UniSense-9B reaches 74.5 Acc@1 and 72.2 Macro F1, surpassing specialized recognition models.

Ablation Studies

All ablations use the Qwen3.5-9B backbone with the same training and evaluation settings. Additional analyses and detailed configurations are provided in Appendix § 7.7 and Appendix § 8.

Component Ablation. Table 8 evaluates the two training stages. Sensory representation alignment consistently improves the baseline, while modality-specific expert tuning brings further gains across all evaluated benchmarks, showing that language grounding and modality-specific specialization provide complementary benefits.

Setting	Proj.	Alignment	Experts	RoboSpatial-Home Config.	ScanQA CIDEr	IMU-4-Task Avg. F1	Infrared-Benchmark Acc.	MMFi-VQA Acc.
Baseline	✓			74.0	96.7	86.0	88.2	60.1
+ Alignment	✓	✓		77.2	97.7	88.0	89.0	63.5
Full	✓	✓	✓	80.5	99.6	89.0	89.9	65.2

Alignment Objectives. Figure 5(a) compares conditional captioning and contrastive alignment. The two objectives show different strengths across modalities, while their combination achieves the best overall performance. This consistent improvement indicates that the two objectives provide complementary supervision and jointly produce better-aligned sensory representations.

Expert Specialization. Figure 5(b) compares shared and modality-specific expert pools. Modality-specific experts consistently outperform the shared alternative across all evaluated modalities, supporting specialized computation for heterogeneous sensory inputs. The gains are particularly evident on SpatialBench-Speech ([12]) and ScanQA ([6]) with xyz+Image inputs, suggesting that modality-specific computation reduces interference across heterogeneous modalities and better preserves complementary cross-modal information.

Conclusion

We present UniSense, a unified MLLM for understanding diverse embodied sensory signals beyond basic multimodal inputs. By grounding heterogeneous sensory representations in language and introducing modality-specific processing within the shared LLM, UniSense enables 8 physical sensing modalities to be modeled under a common architecture. Extensive experiments show that this unified formulation performs strongly across heterogeneous sensory tasks and can effectively integrate sensory signals with existing multimodal inputs. Taken together, these results suggest that pretrained MLLMs can move beyond serving as interfaces for new modalities, providing a shared foundation for understanding heterogeneous physical signals without treating each modality as an isolated system. More broadly, our findings point toward embodied multimodal models that share a common reasoning backbone while retaining the specialization required by different sensing processes.

Related Work

Generalist Embodied Models. Recent advances in MLLMs have motivated a growing line of generalist embodied models that leverage pretrained multimodal backbones for unified perception and reasoning in physical environments. Early systems such as PaLM-E [35] demonstrated the potential of integrating embodied observations into large language models, while subsequent models such as RoboBrain [57, 13], RynnBrain [30], and Vesta [10] further broaden this paradigm across diverse physical-world tasks and environments. More recent efforts further emphasize spatial-temporal perception and high-level reasoning [87] (Zhao et al. 2025; Team et al. 2025a; Zhang et al. 2026a; Yuan et al. 2026), while benchmarks such as OpenEQA [80] and FoMER [33] reflect the growing focus on embodied reasoning. In parallel, vision-language-action models map visual observations directly to low-level robot control [42, 11, 62], targeting action generation rather than the high-level sensory understanding considered in this work. Despite this progress, these generalist models still rely largely on basic modalities such as vision and audio, and their extensions beyond these modalities are typically restricted to a single additional modality. Heterogeneous physical signals such as depth, contact, temperature, inertial dynamics, and fine-grained geometry are therefore rarely available to a single embodied reasoner, even though dedicated models exist for many of them individually.

Heterogeneous Multimodal LLMs. A parallel line of work extends LLMs to increasingly diverse input modalities. Image-text pretraining first provides a language-anchored semantic space through large-scale cross-modal alignment (Yu et al., 2022; Tschannen et al., 2025), which subsequent binding-based methods [44] (Zhu et al. 2024) reuse as a pivot to align heterogeneous modalities. Unified multimodal frameworks [88, 66] instead attach multiple modality encoders to a common LLM, enabling a broader set of input types and tasks within one framework. Audio and speech models likewise benefit from large-scale paired supervision, providing mature language-aligned interfaces for acoustic inputs and spoken instructions [26, 53, 43]. Other works develop dedicated interfaces for individual physical signals, including depth [12, 23, 31], point clouds [51, 45] (Xu et al. 2024), inertial and time-series data [70] (Asif Imran Shouborno et al. 2025), touch [40], thermal [60, 14], Thushara et al., 2026), radar [65] (Zhou & Yang 2026), smell (Zhou et al., 2025c,b), and motion (Jiang et al., 2025; Wang et al., 2026a,b). However, sensory data generally receive far less linguistic supervision than web-scale image-text data, making their physical content difficult to expose fully to language-based reasoning, and each of these interfaces is designed for one modality in isolation. In contrast, UniSense uses caption-guided supervision to make sensory semantics language-accessible and covers 8 embodied sensory modalities within a single model.

Multimodal Fusion and Specialization. Beyond modality alignment, multimodal models also differ in how heterogeneous representations are integrated into language reasoning. Encoder-projector architectures [74, 67, 29] typically perform late fusion, offering a simple interface but limiting deeper and fine-grained cross-modal interaction. A related line of work unifies modalities on the encoder side, reusing a shared encoder or a pre-aligned binding space as a single interface to the language model [47] (Wu et al. 2024), which makes modality extension straightforward but leaves modality-specific structure largely unmodeled. Naive early-fusion models [16] (Zhou et al. 2024; Wang et al. 2024b) process multiple modalities within a shared representation space, enabling richer interaction but potentially increasing cross-modal interference. Recent sparse architectures seek a better balance between sharing and specialization: mixture-of-experts methods [99, 69, 73] (Zhou et al. 2025a) introduce specialized expert pathways, with low-rank expert variants [21] reducing the cost of such specialization, while Mixture-of-Transformers and related architectures [71, 32, 100] more explicitly decouple modality-specific computation while retaining shared attention. These mechanisms, however, have been developed primarily for semantically well-aligned modalities such as image, video, and audio. UniSense instead targets weakly grounded embodied sensory signals, keeping self-attention shared while adding lightweight modality-specific residual experts to selected feed-forward layers.

Datasets and Evaluation Protocols

Supplementary Citations

Due to space limitations in the main text, the citations of the evaluation benchmarks and compared baselines reported in Tables 1–8 are collected in this section instead of in the corresponding captions. Table B9 lists the evaluation benchmarks, grouped by modality following the organization of §B.2. Table B10 and Table B11 list the closed-source and general multimodal baselines and the modality-specialized baselines, respectively, using the same model groups as the main-text tables. Model variants that differ only in parameter scale share a single citation and are therefore listed once.

Modality	Benchmark	Citation	Benchmark	Citation
	General Multimodal Benchmarks
Image-Text	TextVQA	Singh et al. (2019)	MME	Fu et al. (2025a)
	MMBench	Liu et al. (2024c)
Audio-Image-Text	DocVQA	Mathew et al. (2021)	ChartQA	Masry et al. (2022)
Video-Text	Video-MME	Fu et al. (2025b)	EgoSchema	Mangalam et al. (2023)
	Embodied Sensory Benchmarks
Depth	SpatialBench	Cai et al. (2025)	RoboSpatial-Home	Song et al. (2025)
Point Cloud	ScanQA	Azuma et al. (2022)	CLEVR3D-REAL	Yan et al. (2023)
IMU	HHAR	Stisen et al. (2015)	MotionSense	Malekzadeh et al. (2019)
	Shoaib	Shoaib et al. (2014)	UCI-HAR	Anguita et al. (2013)
Thermal	Infrared-Benchmark	Jiang et al. (2024)	IF-Bench	Zhang et al. (2026c)
Tactile	Octopi	Yu et al. (2024)	VTBS	Xie et al. (2025)
Radar	MM-Fi	Yang et al. (2023)	HoloLLM CrossEnv	Zhou & Yang (2026)
Motion	QuAnTS	Divo et al. (2025)
Smell	SmellNet-Base	Feng et al. (2026)

Model	Citation	Model	Citation
	Closed-Source Models
GPT-4o	Hurst et al. (2024)	Claude-3.5	Anthropic (2024)
Gemini-2.5-Pro	Comanici et al. (2025)
	General MLLMs
LLaVA-1.5	Liu et al. (2024a)	LLaVA-NeXT	Liu et al. (2024b)
LLaVA-OneVision	Li et al. (2025a)	LLaVA-Video	Zhang et al. (2025a)
Qwen2-VL	Wang et al. (2024a)	Qwen2.5-VL	Bai et al. (2025b)
Qwen3-VL	Bai et al. (2025a)	InternVL2	Chen et al. (2024d)
InternVL2.5	Chen et al. (2024d)	InternVL3.5	Wang et al. (2025a)
InternVL3	Zhu et al. (2025)	InternVideo2.5	Wang et al. (2025b)
Kimi-VL	Team et al. (2025b)	GLM-4.1V	Hong et al. (2025)
GLM-4.5V	Hong et al. (2025)	Vicuna	Chiang et al. (2023)
TokenFlow	Qu et al. (2025)	Tokenpacker	Li et al. (2025b)
Honeybee	Cha et al. (2024)
	General Omnimodal MLLMs
ImageBind	Girdhar et al. (2023)	ImageBind-LLM	Han et al. (2023)
PandaGPT	Su et al. (2023)	OneLLM	Han et al. (2024)
Intern-Omni	OpenGVLab (2024)	Qwen3-Omni	Xu et al. (2025)
Lyra	Zhong et al. (2025)

Group	Model	Citation	Model	Citation
Spatial	SpatialVLM	Chen et al. [17]	Bunny	He et al. [49]
	SpatialBot	Cai et al. [12]	SpaceThinker	Batra et al. [9]
	SpaceOm	Remyx AI (2025)	SSR	Liu et al. [77]
	VPT	Yu et al. (2025)	SpatialVTS	Liang et al. [72]
3D	ScanQA	Azuma et al. [6]	Scene-LLM	Fu et al. [41]
	LL3DA	Chen et al. [20]	ChatScene	Huang et al. [54]
	ViewMind3D	Chiang et al. [24]	GPT4Scene	Qi et al. [92]
	ReferIt3D	Achlioptas et al. [1]	BEV+MCAN	Yan et al. (2023)
	PointQA	Khan & Gaddam [61]	TransVQA3D	Yan et al. (2023)
	E2E-MPCQA	Qorsham et al. [93]
Tactile	Octopi	Yu et al. (2024)	UniTac	Tu et al. (2026)
IMU	GenHAR	Hong et al. [52]	LanHAR	Yan et al. (2025)
	LLaSA	Asif Imran Shouborno et al. (2025)
Thermal	Infrared-LLaVA	Jiang et al. [60]
Radar	HoloLLM	Zhou & Yang (2026)
Motion	Naive	Divo et al. [34]	ChatTS	Xie et al. (2024)
	xQA	Divo et al. [34]
Smell	MLP, CNN, LSTM and ScentFormer	Feng et al. [37]

Datasets and Benchmarks

This section describes the datasets and evaluation protocols used for each modality. We first detail the training data sources and preparation, with the overall data composition summarized in Figure B6, Figure B6, and then describe the evaluation benchmarks, data splits, input settings, and metrics. Unless otherwise specified, we follow the official benchmark splits and scoring protocols. During evaluation, UniSense receives only the benchmark-specified inputs, without access to training-only captions or auxiliary semantic annotations.

General Multimodal

Following the multimodal initialization pipeline of Zhong et al. (2025), we first establish the image–speech–text capabilities of UniSense before sensory training. Speech alignment uses Common Voice (Ardila et al., [4]) and LibriSpeech (Panayotov et al., [91]), comprising approximately 1.19M speech–text pairs. We then perform image–speech–text instruction tuning on a mixture of Lyra-SFT (Zhong et al., 2025) and selected subsets of FineVision (Wiedmann et al., 2025), containing approximately 3.12M conversations. The mixture consists primarily of Lyra-SFT together with additional FineVision image–text data.

We evaluate the general multimodal capabilities on image–text, image–speech, and video benchmarks. Specifically, we use TextVQA (Singh et al., 2019), MME (Fu et al., [38]), and MMBench (Liu et al., [78]) for image–text understanding; the speech-conditioned variants of TextVQA, DocVQA (Mathew et al., [85]), and ChartQA (Masry et al., [83]) for image–speech understanding; and Video-MME (Fu et al., [39]) and EgoSchema (Mangalam et al., [82]) for video understanding. We follow the standard evaluation protocols for all benchmarks, with Video-MME evaluated without subtitles.

Depth

For depth training, we use SpatialQA (Cai et al., [12]), SPAR (Zhang et al., 2026b), and CAVQA (Daxberger et al., [31]). We additionally construct depth-oriented training samples from COCO (Mishra et al., [86]), Visual Genome (VG) (Krishna et al., [64]), and SA-1B (Kirillov et al., [63]). These datasets cover diverse depth-centric scenes and task formats, including spatial question answering, 3D reasoning, captioning, and general multimodal instruction tuning. We synthesize speech with CosyVoice (Du et al., [36]) for SpatialQA, SPAR, CA-VQA, and part of the constructed depth subset. For samples without depth maps, we generate them using Depth Anything V2 (Yang et al., 2024).

For sensory representation alignment, each RGB–D sample is paired with a depth-aware description generated offline by Qwen3-Omni-30B-A3B-Instruct (Xu et al., 2025): the model first extracts objects from RGB and then generates a depth-focused caption conditioned on the RGB image, depth map, and extracted objects.

We evaluate depth understanding on SpatialBench and RoboSpatial-Home (Song et al., 2025) using paired RGB–D observations. SpatialBench comprises five subtasks: positional reasoning, object existence, counting, reaching, and size comparison. We report each subtask score, together with 4-Task Agg., the sample-weighted aggregate over Position, Existence, Counting, and Size, and Agg., the aggregate over all five subtasks. All scores follow the official SpatialBench evaluation protocol. We further construct SpatialBench-Speech by converting the question text into speech to assess the robustness of depth reasoning to spoken queries. For RoboSpatial-Home, we report zero-shot accuracy on the Configuration split without benchmark-specific fine-tuning.

Point Cloud

For point-cloud training, we use CLEVR3D-REAL (Yan et al., 2023), ScanQA (Azuma et al., [6]), and SceneVerse (Jia et al., [58]). Together, these datasets cover scene-level descriptions, question answering, and compositional reasoning over indoor 3D scenes. We use the official training splits of ScanQA and CLEVR3D-REAL. For ScanQA, we further construct RGB-augmented examples using single-view and four-view images from MV-ScanQA (Mo et al., [87]), yielding both point-cloud-only and point-cloud–image training inputs. For sensory representation alignment, scene-level descriptions from SceneVerse are matched to ScanQA and CLEVR3D-REAL samples by scene identity.

We evaluate point-cloud understanding on ScanQA and CLEVR3D-REAL (results reported in Table D15), using their official validation splits. On ScanQA, we evaluate both point-cloud-only ($xyz$) and point-cloud–image ($xyz+img$) inputs. Following the standard ScanQA protocol, we report CIDEr, METEOR (MET.), ROUGE-L (R-L), BLEU-1 (B-1), and BLEU-4 (B-4) for open-ended 3D question answering. CLEVR3D-REAL evaluates compositional reasoning over point-cloud scenes across existence, counting, comparison, query-attribute, query-object, and compare-attribute questions. We report accuracy for each question category, together with Overall, denoting accuracy over all test questions, and Cls. Avg., denoting the unweighted average accuracy across question categories.

IMU

For IMU training, we use the SensorCaps and OpenSQA annotations (Asif Imran Shoubornо et al., 2025) associated with four human activity recognition datasets: HHAR (Stisen et al., 2015), MotionSense (Malekzadeh et al., [81]), Shoaib (Shoaib et al., 2014), and UCI-HAR (Anguita et al., [2]). SensorCaps provides a natural-language description for each IMU window and is used for sensory representation alignment, while OpenSQA provides question–answer pairs covering activity-related understanding and reasoning. We construct the training and evaluation sets using stratified windowlevel hold-out within each source dataset, reserving approximately 100 windows per activity class for evaluation and using the remaining windows for training, with no overlap between the two splits. This results in approximately 23.8k training windows and 2.5k held-out evaluation windows, with multiple OpenSQA questions associated with each training window.

We evaluate IMU understanding as open-ended activity classification on the held-out windows from HHAR, MotionSense, Shoaib, and UCI-HAR, following the LLaSA (Asif Imran Shouborno et al. (2025)) classification protocol. For each dataset, the model is provided with the IMU sequence and generates the predicted class name directly from the input. We report results separately for the four datasets using accuracy and macro-averaged precision, recall, and F1.

Tactile

For tactile training, we use Octopi (Yu et al. (2024)) and the Universal Visuo-Tactile Video Understanding corpus (VTBS) (Xie et al. (2025)). Both datasets follow object-level splits, such that objects used for evaluation are excluded from training. Following the official data generation protocols, we construct 50k instruction–response pairs from each corpus, covering tactile property description, comparison, selection, and object–property reasoning, resulting in approximately 100k training examples in total. The original GelSight (Yuan et al. (2017)) videos are converted into a unified image-based representation by first selecting informative motion frames and then uniformly sampling three RGB frames from each video, which are treated as tactile observations during training. For sensory representation alignment, we extract the descriptive property text from the generated responses and use it as the corresponding tactile caption.

We evaluate on the official held-out-object splits of Octopi using the same three-frame tactile representation as in training. Octopi contains 498 questions across property comparison (PC), property superlative selection (PSS), and property–object matching (POM). Following the official protocol, we report accuracy separately for the three tasks.

Thermal

For thermal training, we use a partial subset of IR-TD from IRGPT (Cao et al. [14],) the thermal subset of VS-TDX (Chung et al. [27]), and Infrared-LLaVA (Jiang et al. [60]). Together, these datasets cover thermal captioning, question answering, recognition, counting, and localization. For IR-TD, we use its caption, instruction-QA, and task-oriented training subsets, and pair thermal images with the corresponding RGB observations when available. For sensory representation alignment, we use the paired RGB–thermal captions provided by IR-TD.

We evaluate thermal understanding on IF-Bench (Zhang et al. (2026)) (results reported in Table D14), and the Benchmark from Infrared-LLaVA, using thermal inputs for both benchmarks. IF-Bench is evaluated in a zero-shot setting and groups its tasks into three categories: coarse-grained perception (Coarse), fine-grained perception (Fine), and image reasoning (Reason). We report accuracy for each of the ten tasks and the overall average. Infrared-LLaVA Benchmark contains six infrared question-answering tasks covering localization, aerial counting, pedestrian counting, recognition, scene understanding, and safety assessment. We report the average accuracy across the six tasks under both zero-shot and fine-tuning settings.

Radar

For radar training, we use mmWave point-cloud sequences from MM-Fi (Yang et al. (2023)) together with the language annotations introduced by HoloLLM (Zhou & Yang (2026)). We follow the HoloLLM Cross-Environment protocol, using environments E01–E03 for training and holding out E04 for evaluation, such that the training and test sets contain disjoint environments and subjects. The training split contains approximately 12.6k temporal radar segments. Based on the HoloLLM action annotations, we construct a mixture of action captions and question–answering examples, and further augment the instruction data with laterality questions and multiple-choice questions targeting confusable action categories, resulting in approximately 121k training examples. The original HoloLLM captions are further refined offline using GPT-5.1 [89] with action labels as guidance. The refined captions are used for sensory representation alignment.

We evaluate radar understanding following the HoloLLM Cross-Environment protocol on the held-out E04 environment. We report results on two tasks using radar-only input: Human Action QA and Action Caption. For Human Action QA, the model predicts one of the predefined human action categories from the radar sequence, and performance is measured using accuracy following the HoloLLM matching protocol. For Action Caption, the model generates a free-form description of the observed action, and we report METEOR as the primary metric.

Motion

For skeleton-motion training, we use QuAnTS [34], which contains synthetic SMPL [79] motion sequences paired with binary, multiple-choice, and open-ended question answering. We follow the official 80/10/10 trajectory-level split and use only the training partition, corresponding to 24k motion sequences and 120k QA pairs. For sensory representation alignment, we use the scene-level textual descriptions provided by QuAnTS as auxiliary captions.

We evaluate on the official QuAnTS test split, which contains 3k held-out motion trajectories and 15k questions spanning binary, multiple-choice, and open-ended QA. At inference time, the model receives only the skeleton-motion sequence and the corresponding question, without access to scene descriptions or ground-truth action sequences. We report binary and multiple-choice performance using accuracy and macro-averaged precision, recall, and F1, while open-ended responses are evaluated using ROUGE-L and METEOR.

Smell

For smell training, we use SmellNet-Base [37], which contains 50 single-substance categories recorded with a six-channel metal-oxide (MOX) sensor array. We follow the official day-wise split, using recordings from the first five days for training and the sixth day for evaluation, yielding approximately 2.5k training windows and 0.5k held-out test windows after temporal segmentation. We segment the aligned sensor readings into fixed-length windows and normalize each channel using training-set statistics. From these windows, we construct approximately 50.4k instruction samples spanning classification, reasoning, and captioning, and generate temporal descriptions that exclude the substance identity for the alignment stage.

We evaluate on the held-out day-6 split of SmellNet-Base (502 windows across 50 classes), using training-set normalization statistics. Given a sensor sequence and the full candidate set, the model predicts the substance name; we report $\mathrm{Acc}@1$ and macro F1.

Implementation Details

Baselines

We compare UniSense with three groups of baselines: (i) closed-source models, including models from the GPT, Claude and Gemini families; (ii) general multimodal models, such as the Qwen-VL and InternVL families; and (iii) modality-specialized models designed for specific sensory modalities or tasks. For each benchmark, we follow its standard evaluation protocol and report the corresponding metrics.

Tokenizers and Projectors

We adopt two tokenization pathways according to the structure of each sensory input.

Pixel-aligned modalities. Depth, tactile, and thermal signals are represented as image-like inputs and processed using the default Qwen visual preprocessing. They are divided into patches by the pretrained visual patch embedding and then encoded by the shared vision encoder and merger.

Non-pixel modalities. Point cloud, IMU, radar, motion, and smell inputs have distinct spatial or temporal structures that cannot be directly processed by visual patch embedding. We therefore introduce a lightweight tokenizer for each modality to map its native input into a sequence of embeddings with the same hidden dimension as the visual patch tokens. These embeddings are directly fed into the shared vision encoder and merger. The tokenizer design for each modality is described below.

Point cloud. Each point cloud $x^{\mathrm{pc}}$ is represented by normalized XYZ coordinates and RGB features. We sample 32,768 points from each scene and organize them into 1,024 local groups, each containing 32 neighboring points. Within each group, the point features are augmented with coordinates relative to the group center. A point-wise $1 \times1$ convolution projects the resulting features to the vision hidden dimension, followed by max pooling over the local neighborhood:

IMU. Each IMU input $x^{\mathrm{imu}}$ consists of six channels corresponding to tri-axial accelerometer and gyroscope measurements. Each sequence contains 120 temporal steps. Inputs of a different length are linearly resampled to this length before encoding. We employ a length-preserving 1D convolution with kernel size 10 to map the input sequence to the vision hidden dimension:

The tokenizer produces 120 IMU tokens, which are subsequently processed by the shared vision encoder and reduced to 30 tokens by the vision merger in $\mathcal{E}$.

Radar. Each radar input $x^{\mathrm{radar}}$ is represented as a temporal sequence of point sets, where each point contains five attributes: 3D coordinates, Doppler, and intensity. Each segment is uniformly sampled to at most 32 frames, with each frame containing at most 64 points. For each frame, the original point features are augmented with coordinates relative to the masked centroid. A point-wise projection followed by masked max pooling produces a frame-level representation:

The resulting frame-level representations are temporally interpolated to 64 radar tokens, which are subsequently processed by the shared vision encoder and reduced to 16 tokens by the vision merger in $\mathcal{E}$. Padded points are excluded from both centroid computation and feature pooling.

Motion. Each skeleton–motion input $x^{\mathrm{motion}}$ is represented as a sequence of 24 skeleton joints with 3D coordinates. Each frame is flattened into a 72-dimensional feature vector, and each sequence is linearly resampled to 320 temporal steps. We employ a length-preserving 1D convolution with kernel size 10 to map the input sequence to the vision hidden dimension:

The tokenizer produces 320 motion tokens, which are subsequently processed by the shared vision encoder and reduced to 80 tokens by the vision merger in $\mathcal{E}$.

Smell. Each smell input $x^{\mathrm{smell}}$ consists of temporal measurements from six metal-oxide gas sensors. Each sequence contains 100 temporal steps, and each channel is standardized using its training-set mean and standard deviation. We employ a length-preserving 1D convolution with kernel size 10 to project the signal sequence to the vision hidden dimension:

The tokenizer produces 100 smell tokens, which are subsequently processed by the shared vision encoder and reduced to 25 tokens by the vision merger in $\mathcal{E}$.

Modality-specific projectors. After the shared vision encoder and merger, each sensory modality $s \in\mathcal{M}_s$ is equipped with an independent residual projector $\mathcal{P}_s$. Let $\mathbf{H}^{s} = [\mathbf{h}^{s}_{1}, \ldots, \mathbf{h}^{s}_{N_s}]$ denote the merged representation of modality $s$, where $\mathbf{h}^{s}_{i} \in\mathbb{R}^{d_{\mathrm{llm}}}$. The projector is applied independently to each token:

and the projected sensory representation is $\mathbf{Z}^{s} = [\mathbf{z}^{s}_{1}, \ldots, \mathbf{z}^{s}_{N_s}] = \mathcal{P}_s(\mathbf{H}^{s})$. Here, $\mathbf{W}^{s}_{1} \in\mathbb{R}^{d_{p} \times d_{\mathrm{llm}}}$ and $\mathbf{W}^{s}_{2} \in\mathbb{R}^{d_{\mathrm{llm}} \times d_{p}}$, with $d_{p} = 768$. The output layer is zero-initialized, such that the projector starts as an identity mapping.

Basic Modality Processing

Unlike embodied sensory inputs, basic modalities retain the processing pathways inherited from the pretrained multimodal backbone.

Image pathway. RGB images are processed by the native Qwen3.5 visual encoder and merger. The resulting image tokens are directly inserted into the language sequence without an additional learnable projector, and the entire pathway remains unchanged during sensory training.

Speech pathway. Speech signals are encoded using Whisper-large-v3-turbo [97]. The resulting acoustic features are mapped to the language hidden dimension through a speech projector. The Whisper encoder remains frozen, while the speech projector is pretrained to align acoustic representations with the language model.

Multimodal initialization. Before sensory training, UniSense is initialized from an image–speech–text instruction-tuned Qwen3.5 checkpoint, with its original image and speech pathways retained.

Sensory Representation Alignment

We provide additional implementation details for the sensory representation alignment stage introduced in § 2.3. Unless otherwise specified, the following settings correspond to the Qwen3.5-9B configuration.

Conditional captioning. We use a lightweight 6-layer caption decoder with a hidden dimension of 1,024 and 16 attention heads. The decoder reuses the frozen token embeddings and language-modeling head of Qwen, while its remaining parameters are trainable. Cross-attention uses only the projected tokens of the target sensory modality as memory, and the decoder output is used solely for auxiliary caption reconstruction rather than the main response-generation pathway. The caption decoder remains trainable in both sensory-training stages.

Contrastive alignment. The projected sensory tokens and caption tokens are independently processed by the first $k$ frozen layers of the language backbone. In the default configuration, we use $k = 8$. We apply mean pooling over valid sensory tokens and use the final valid caption token as the textual representation. The two representations are then passed through separate LayerNorm and linear projection heads, mapped to 512 dimensions, and $\ell_2$-normalized before computing the symmetric InfoNCE loss.

Representations are gathered across all distributed workers to construct global in-batch negatives. Pairs corresponding to the same underlying sensory instance or sharing an identical normalized caption are excluded from the negative set to reduce false-negative supervision. The contrastive branch is used only during the first sensory-training stage and is disabled after the transition to modality-specific expert tuning.

Modality-Specific Expert Tuning

During the second sensory-training stage, modality-specific residual experts are activated in the selected full-attention decoder layers, while the pretrained self-attention and base FFN remain frozen. For the 9B model, expert modules are inserted into the eight full-attention layers at indices 3, 7, 11, 15, 19, 23, 27, and 31.

Expert configuration. Each embodied sensory modality uses an independent router and a pool of $E = 3$ residual experts. The router produces softmax weights over all experts in the corresponding modality-specific pool. Text, image, and speech tokens bypass these expert pools and retain the original pretrained FFN computation.

Residual parameterization. Each expert applies low-rank residual adaptations to the gate, up, and down projections of the backbone SwiGLU FFN. We use rank $r = 32$ and scaling factor $\alpha= 64$, corresponding to a residual scaling factor of $\alpha/r = 2$.

Regularization. We use router load-balancing together with z-loss regularization, with a combined weight of $10^{-3}$. An additional expert-diversity objective is applied with a target weight of 0.07, which is linearly increased from zero during the early expert-tuning stage to stabilize optimization.

Phase	Trainable modules	Base LR	Epochs
Speech alignment	Speech projector	$1 \times10^{-3}$	1
Multimodal initialization	LLM LoRA, speech projector	$2 \times10^{-4}$	1
Alignment Stage	Tokenizers, projectors, caption decoder, contrastive heads	$1 \times10^{-4}$	1
Tuning Stage	Tokenizers, projectors, caption decoder, routers, experts	$1 \times10^{-4}$	2

Training Details

Unless otherwise specified, the following settings correspond to the Qwen3.5-9B configuration; the 2B model follows the same two-stage training procedure with scale-specific configurations.

Multimodal initialization. We construct a common multimodal initialization checkpoint before sensory training. We first connect a frozen Whisper encoder to the language model by pretraining the speech projector. We then perform image–speech–text instruction tuning, where the language model is adapted with rank-64 LoRA while the speech projector remains trainable. No additional video-specific training is introduced during these preliminary stages. These stages establish the basic-modality pathways and are not counted as part of the two-stage sensory training of UniSense.

Two-stage training. Starting from the merged multimodal checkpoint, the pretrained language backbone and modality encoders remain frozen throughout sensory training. In the alignment Stage, we train the sensory tokenizers, modality projectors, caption decoders, and contrastive projection heads for 1 epoch using language-modeling, caption-reconstruction, and contrastive objectives.

The tuning Stage continues optimizing the sensory tokenizers, projectors, and caption decoders, while activating the modality-specific routers and residual experts. The contrastive branch is disabled after the stage transition, whereas caption supervision is retained together with the expert regularization objectives. Stage 2 is trained for 2 additional epochs.

Loss configuration. During the alignment Stage, the caption and contrastive objectives are jointly applied with the language-modeling objective. We set the contrastive weight to $\lambda_{\mathrm{con}} = 0.5$ and use an overall alignment weight of $\lambda_{\mathrm{align}} = 0.08$ for all modalities. In the tuning Stage, the contrastive objective is removed while caption reconstruction is retained. The router regularization is weighted by $10^{-3}$, and the expert-diversity weight is linearly increased from 0 to 0.07 during the early expert-tuning stage.

Optimization. We use AdamW with cosine learning-rate decay and a 5% warm-up ratio. Modality projectors use a learning rate of $1.5 \times10^{-4}$, while the remaining trainable sensory modules use a base learning rate of $1 \times10^{-4}$. The maximum sequence length is 2,048 for speech alignment and 4,096 for both multimodal initialization and sensory training. All models are trained in bfloat16 on 4 NVIDIA H200 GPUs.

Ablation Settings

All ablation experiments are conducted with the Qwen3.5-9B backbone. Unless otherwise specified, we keep the training data, modality tokenizers and projectors, optimization schedule, and evaluation protocol unchanged across variants.

Component ablation. We evaluate the contributions of sensory representation alignment and modality-specific expert tuning using the three configurations summarized in Table C13. All variants follow the same two-stage training schedule. The baseline uses only the language-modeling loss on downstream instruction–response pairs during the alignment stage, followed by a single low-rank residual FFN shared across all sensory modalities without routing during the tuning stage(Figure C7a). We then introduce conditional captioning and contrastive alignment while retaining the same shared adaptation. The full model further replaces the shared residual FFN with modality-specific routers and expert pools.

Setting	Alignment Stage	Tuning Stage
Baseline	$\mathcal{L}_{\mathrm{LM}}$	Shared residual FFN
Representation alignment	$\mathcal{L}_{\mathrm{LM}} + \mathcal{L}_{\mathrm{cap}} + \mathcal{L}_{\mathrm{con}}$	Shared residual FFN
Full	$\mathcal{L}_{\mathrm{LM}} + \mathcal{L}_{\mathrm{cap}} + \mathcal{L}_{\mathrm{con}}$	Modality-specific experts

The shared residual FFN is activated only in the tuning Stage, following the same training schedule as the modality-specific experts. This keeps the amount of training data and the number of optimization steps consistent across configurations. We report results on representative benchmarks spanning depth, point cloud, IMU, thermal and radar.

Alignment objectives. All variants retain $\mathcal{L}_{\mathrm{LM}}$, the language-modeling loss on downstream instruction–response pairs. We independently enable the two auxiliary objectives during the alignment stage, yielding four settings: (1) task LM only; (2) task LM with conditional captioning ($\mathcal{L}_{\mathrm{cap}}$); (3) task LM with contrastive alignment ($\mathcal{L}_{\mathrm{con}}$); and (4) task LM with both objectives. All variants use the same sensory front ends, training data, and optimization settings.

Expert architectures. Starting from the sensory representation-aligned setting, we further examine how expert sharing affects the tuning Stage adaptation. We compare a Shared Expert Pool (Figure C7b), where all sensory modalities route over a common pool of 3 residual experts, with Modality-Specific Expert Pools (Figure C7c), where each modality uses an independent router and expert pool. Both variants start from the same representation-aligned initialization and retain the frozen pretrained base FFN.

Additional Experiments

Additional Benchmark Results. We further provide additional results on thermal and point-cloud understanding in Table D14 and Table D15. UniSense-9B reaches 81.03 average accuracy on IF-Bench and 50.1 class-average accuracy on CLEVR3D-REAL, demonstrating strong performance across both benchmarks.

Expert placement. We study expert placement on two modalities, depth and radar, comparing four strategies: all Transformer layers, the early or late half of the full-attention layers, and all full-attention layers. Table D16 shows that expert placement matters more than simply increasing its layer coverage. While depth exhibits some benchmark-specific preference, restricting experts to full-attention layers yields the most robust performance overall, particularly for radar where it consistently outperforms all alternative placements.

tbl:D16: Effect of expert placement. We vary the Transformer layers equipped with modality-specific experts while keeping all other training settings unchanged. Early-half and Late-half denote placing experts in the first and second halves of the full-attention layers, respectively. Full-attn. denotes our default setting, where experts are attached only to full-attention layers.

Visualization of Training Samples

This section presents representative training examples across sensory modalities: depth (Figs. E8 and E9), point cloud (Figs. E10 and E11), IMU (Figs. E12 and E13), thermal (Figs. E14 and E15), tactile (Figs. E16 and E17), radar (Figs. E18 and E19), motion (Figs. E20 and E21), and smell (Figs. E22 and E23). Each example shows the sensory input, its instruction–response pair, and the modality-aware caption used for sensory representation alignment.

Depth. The following examples show paired RGB–depth inputs, instruction–response supervision, and depth-aware captions used for sensory representation alignment.

Point Cloud. Each point cloud is rendered from two viewpoints for visualization only; the model receives the underlying 3D point cloud as input. Each example also shows its instruction–response pair and scene-level caption used for sensory representation alignment.

IMU. Each IMU sequence is visualized as accelerometer and gyroscope curves for presentation only; the model receives the underlying six-channel time series. Each example also shows its instruction–response pair and sensor caption used for sensory representation alignment.

Thermal. Each example shows a thermal image, its instruction–response pair, and a temperature-aware caption used for sensory representation alignment.

Tactile. Each tactile observation is represented by three sampled frames. Single-object examples contain one frame triplet, whereas multi-object examples contain a separate triplet for each object. The corresponding instruction–response pair and tactile property caption are shown below.

Radar. Each radar sequence is visualized using sampled point-cloud frames and signal heatmaps for presentation only; the model receives the underlying temporal mmWave point clouds. Each example also shows its instruction–response pair and action caption used for sensory representation alignment.

Motion. Each skeleton-motion sequence is visualized using sampled poses and joint trajectories for presentation only; the model receives the underlying sequence of 3D joint coordinates. Each example also shows its instruction–response pair and temporal caption used for sensory representation alignment.

Smell. Each smell sensor sequence is visualized as six-channel response curves for presentation only; the model receives the underlying numerical measurements. Each example also shows its instruction–response pair and sensor caption used for sensory representation alignment.