betaFeedback

WorksComputer Science

Lie Routing Vision Language Models for View-Robust 3D Understanding

Abstract

Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVLM, a Lie routing framework for view-robust 3D understanding. Motivated by the rotational component of viewpoint change, Lie routing adapts visual representations by applying local SO(3)\mathrm{SO}(3) transformations to projected features within learned three-dimensional subspaces. Geometry tokens extracted by VGGT predict local rotation parameters and channel-wise routing coefficients. A closed-form Padé–Cayley map transforms projected visual features, whose identity-relative corrections are projected back and gated into the original visual and geometry features. This design retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments. Experiments on 3D captioning, grounding, detection, and spatial reasoning show consistent improvements over additive fusion across training scales and backbones. Controlled ablations examine the roles of geometric conditioning and transformation structure. Viewpoint analyses further reveal systematic variation in the predicted routing states alongside more localized target responses and stronger task performance across viewpoint changes.

Introduction

3D Multimodal Large Language Models (MLLMs) extend visual-language reasoning to scene-level tasks such as object captioning, language-guided localization, and spatial question answering. Early approaches primarily connect language models to explicit 3D representations, including point clouds and object-level scene features [37, 6, 22]. These representations provide direct spatial structure, while their reliance on reconstructed scene inputs motivates complementary approaches that learn 3D understanding from RGB observations.

Recent 3D MLLMs derive geometry directly from multi-view RGB observations, enabling 3D understanding without requiring preconstructed 3D scene models. Pretrained geometry encoders such as VGGT [47] extract scene structure and cross-view correspondences, and recent methods inject these features into MLLM visual representations for 3D scene understanding and spatial reasoning [42, 24, 39]. This learned geometry injection paradigm preserves the scalability of video-based MLLMs while providing strong spatial context. Despite these advances, geometry is primarily used to augment visual representations, while how it should drive viewpoint-adaptive changes in visual semantics remains underexplored.

We observe that a fixed scene entity and query can yield substantially different target-related responses across viewpoints. As camera pose, apparent scale, and visibility change, the response may weaken, become diffuse, or shift toward irrelevant regions, a phenomenon we term viewpoint-induced semantic drift. We visualize it using query-relevant activations, obtained by projecting final visual tokens through the vocabulary head and averaging their positive support over the target-phrase tokens. Viewpoint change between frames ii and jj is measured by the SO(3) geodesic distance drot(i,j)=arccos⁡((tr⁡(Ri⊤Rj)−1)/2)d_{\mathrm{rot}}(i,j) = \arccos((\operatorname{tr}(R_i^{\top}R_j)-1)/2) [16, 26]. Taking the first frame containing the referred target as reference rr, we define ΔR=∣Vs∣−1∑t∈Vsdrot(r,t)\Delta R = |V_s|^{-1}\sum_{t\in V_s}d_{\mathrm{rot}}(r,t) over the remaining views VsV_s, with both distances reported in degrees. Figure 1(a) varies viewpoint along a fixed camera trajectory while holding the scene, target, query, checkpoint, and reference view constant, showing that Lie routing preserves more localized responses. Figure 1(b) relates task performance to ΔR\Delta R, with LieVLM maintaining stronger performance as viewpoint variation increases.

3D scene viewpoint variation and task robustness across viewpoints

Figure 1. 3D scene viewpoint variation exposes semantic instability and downstream performance degradation. (a) Without routing, query-relevant activations become diffuse and less target centered across views, while Lie routing preserves stable target responses. Warmer regions indicate stronger support for the target phrase. (b) Fitted task trends show that performance degrades as sample-level viewpoint variation ΔR\Delta R increases, while LieVLM maintains stronger performance across higher levels of viewpoint variation. For grounding, raw IoU visualizes the continuous quantity underlying [email protected]. Additional examples are provided in Appendix § 10.§ 1.

To address this challenge, we introduce LieVLM, which adapts visual semantics through geometry-conditioned Lie routing. Camera rotations are described by SO(3), motivating us to use the same structure for feature adaptation across viewpoints. Lie routing applies learned rotations to three-dimensional groups of projected visual features, changing their directions while preserving their lengths. Concretely, geometry tokens predict local SO(3) generators and channel-wise routing coefficients. A Padé–Cayley realization converts the generators into local rotations, and only the resulting identity-relative corrections are projected back into the MLLM representation. The corrections are gated and fused with the original visual features and direct geometry features before language generation. This design combines structured feature adaptation with retained semantic and spatial-context paths, without requiring relative camera poses as inputs to the routing module.

We evaluate LieVLM on 3D captioning, grounding, video object detection, and spatial reasoning [9, 4, 36, 38]. LieVLM consistently improves over matched additive geometry fusion, with gains that generalize across spatial benchmarks, training scales, and stronger multimodal backbones. Controlled analyses further connect these improvements to structured geometry conditioned transformations and robustness to viewpoint variation.

Our contributions are summarized as follows:

  • We characterize viewpoint-induced semantic drift in 3D MLLMs through cross-view target-response diagnostics and its association with downstream task performance.

  • We develop Lie Routing Vision Language Models (LieVLM), combining local so(3)\mathfrak{so}(3) prediction, rotation-preserving residual transformations, and channel-wise gating to adapt pretrained visual semantics across viewpoints.

  • We demonstrate consistent gains across 3D scene understanding and spatial reasoning, with controlled evaluations across training scales and backbones and targeted analyses of routing structure and viewpoint robustness.

Related Work

3D Vision Language Models. 3D VLMs incorporate explicit spatial inputs such as point clouds and object representations [37, 6, 22, 44], or derive geometric priors from image sequences [42, 34, 35]. Reconstruction-aware models further couple video understanding with 3D reconstruction [5, 11, 20], establishing geometry as an important representation for 3D reasoning. Recent RGB-based systems additionally combine semantic and geometry encoders through camera-guided fusion [41]. Our work builds on these interfaces and studies geometry-conditioned routing of pretrained visual semantics across viewpoints.

3D Scene Understanding and Spatial Reasoning. Scene understanding benchmarks evaluate localization, grounding, and captioning [4, 9, 36]. Spatial reasoning benchmarks assess depth, distance, viewpoint, and multiview relations [38, 33, 12, 39], while instruction datasets develop these capabilities from embodied observations [3, 40, 31]. Multi-view grounding methods further aggregate view-dependent evidence through inter-view attention and learned view prototypes [14]. These efforts motivate our focus on semantic stability and task robustness under viewpoint variation.

Geometry in Neural Representations. Prior work spans conditional feature modulation through FiLM [30], explicit geometry-induced transformations in multi-view attention via GTA [28], and equivariant representations under SO(3)\mathrm{SO}(3) or Euclidean symmetries [10, 25, 13, 32]. Orthogonal and Cayley parameterizations further provide structure-preserving learned transformations [17]. We instead study geometry-conditioned Lie routing in pretrained 3D MLLMs, where learned geometry tokens predict token-wise local SO(3)\mathrm{SO}(3) actions and channel-wise gates over latent subspaces for viewpoint-adaptive routing without explicit relative pose inputs.

Method

Preliminaries

3D Geometry in MLLMs. Consider a video or multi-view observation {It}t=1T\{I_t\}_{t=1}^{T} of a 3D scene. A standard MLLM maps each frame into semantic visual tokens Ft2d∈RN×HF_t^{2d} \in\mathbb{R}^{N \times H}, while the geometry encoder provides aligned geometry tokens Ft3d∈RN×HF_t^{3d} \in\mathbb{R}^{N \times H}, where HH denotes the visual token width at the MLLM interface. Existing 3D-enhanced VLMs commonly combine the two streams through additive or attention-based feature injection,

F~t=Ft2d+B(Ft3d),(1)\widetilde{F}_t = F_t^{2d} + B(F_t^{3d}), \tag*{(1)}

where B(⋅)B(\cdot) is a learned geometry projection. The 3D perception fundamentally concerns how scene observations transform across viewpoints. Beyond supplying spatial context, our goal is to adapt visual semantics to viewpoint variation through structured latent transformations.

Lie Groups. We briefly review Lie groups, Lie algebras, and the Special Orthogonal group SO(3)\mathrm{SO}(3). For more in-depth details, please refer to [15, 16].

Definition 3.1 (Lie Group). A Lie group is a smooth manifold GG equipped with a group operation ⊙\odot such that the multiplication map m:(x,y)↦x⊙ym:(x,y)\mapsto x\odot y and the inverse map i:x↦x−1i:x\mapsto x^{-1} are smooth.

Definition 3.2 (Lie Algebra). Let GG be a Lie group with identity ee. Its Lie algebra is the tangent space at ee, i.e., g=TeG\mathfrak{g}=T_eG, which locally linearizes GG through the exponential map exp⁡:g→G\exp:\mathfrak{g}\to G.

Framework of the proposed LieVLM

Figure 2. Framework of the proposed LieVLM. Lie-State Elicitation extracts latent operator states and routing coefficients from geometry tokens. Lie Group Routing transforms projected visual tokens, and routed fusion combines semantic, geometric, and operator paths before language generation.

The set of 3D rotations, known as the Special Orthogonal group SO(3), forms a Lie group

SO(3)={R∈R3×3∣RTR=I,det⁡(R)=1},so(3)={Ω∈R3×3∣ΩT=−Ω},(2)\mathrm{SO}(3)=\{R \in\mathbb{R}^{3 \times3}\mid R^{\mathsf{T}}R=I,\det(R)=1\},\quad\mathfrak{so}(3)=\{\Omega\in\mathbb{R}^{3 \times3}\mid\Omega^{\mathsf{T}}=-\Omega\}, \tag*{(2)}

where so(3)\mathfrak{so}(3) is the Lie algebra of SO(3), namely the space of 3×33 \times3 skew-symmetric matrices. The standard hat operator (⋅)∧:R3→so(3)(\cdot)^{\wedge}:\mathbb{R}^{3}\to\mathfrak{so}(3) identifies an axis-angle vector ω\omega with its skew-symmetric matrix Ω=ω∧\Omega=\omega^{\wedge}, whose exponential exp⁡(Ω)\exp(\Omega) gives the corresponding rotation in SO(3).

Lie Routing. Lie groups provide a natural framework for modeling geometric transformations. Since the geometry tokens Ft3dF_t^{3d} encode scene structure, viewpoint variation, and cross-view correspondence, they should not only serve as additional geometric context, but also indicate how semantic states should be adjusted under viewpoint changes. Conventional additive fusion injects a geometry-dependent feature, such as B(Ft3d)B(F_t^{3d}), into the visual representation. Although this enriches tokens with 3D information, it does not explicitly define an operation on the semantic state. Lie routing utilizes geometry features to predict local rotations of projected visual representations. The routed branch retains only the change induced by these transformations relative to the identity,

Ft2d↦Ft2d+Δop(Ft2d;Ft3d).(3)F_t^{2d}\mapsto F_t^{2d}+\Delta_{\mathrm{op}}\left(F_t^{2d};F_t^{3d}\right). \tag*{(3)}

Here, Ft2dF_t^{2d} denotes the semantic state and Ft3dF_t^{3d} provides the geometric conditioning. The term Δop\Delta_{\mathrm{op}} denotes only the semantic change induced by the operator, rather than the complete transformed state. By construction, this correction vanishes when the induced operator is the identity. We implement this correction using local so(3)\mathfrak{so}(3) generators and a rational SO(3) parameterization, preserving the pretrained semantic path while geometry governs viewpoint dependent semantic transformations.

Lie-State Elicitation

For a video or multi-view observation {It}t=1T\{I_t\}_{t=1}^{T} of a 3D scene, VGGT [34] extracts 3D geometry features. These features are aligned to the visual token grid and projected to the MLLM interface width HH, yielding geometry tokens F3d∈RN×HF^{3d}\in\mathbb{R}^{N\times H}. Alignment details are provided in Appendix § 9.§ 1. Layer normalization and two prediction heads then produce the latent rotation parameters and channel-wise routing coefficients,

Z=LN⁡(F3d),ω=tanh⁡(WωZ)∈RN×Do,γ=σ(WγZ+bγ)∈(0,1)N×H.(4)Z=\operatorname{LN}(F^{3d}),\qquad\omega=\tanh(W_{\omega}Z)\in\mathbb{R}^{N\times D_o},\qquad\gamma=\sigma(W_{\gamma}Z+b_{\gamma})\in(0,1)^{N\times H}. \tag*{(4)}

Here, Wω:RH→RDoW_{\omega}:\mathbb{R}^{H}\to\mathbb{R}^{D_o} predicts latent operator states, while Wγ:RH→RHW_{\gamma}:\mathbb{R}^{H}\to\mathbb{R}^{H} predicts token and channel wise routing coefficients. The learnable routing bias bγb_{\gamma} is initialized to −4-4, biasing γ\gamma toward a low activation regime, since σ(−4)≈0.018\sigma(-4)\approx0.018 when WγZ=0W_{\gamma}Z=0. We set Do=3⌊H/3⌋D_o=3\lfloor H/3\rfloor and define M=Do/3M=D_o/3 triplet groups, with γ\gamma gating the routed transformation at each token and output channel.

To construct structured generators, we reshape the predicted state as ω∈RN×M×3\omega\in\mathbb{R}^{N\times M\times3}, with ωn,m=(ωn,mx,ωn,my,ωn,mz)\omega_{n,m}=(\omega^{x}_{n,m},\omega^{y}_{n,m},\omega^{z}_{n,m}), where n∈{1,…,N}n\in\{1,\ldots,N\} indexes visual tokens and m∈{1,…,M}m\in\{1,\ldots,M\} indexes triplet subspaces. The partition is applied after projection to the operator space, rather than directly to the backbone hidden dimension. Each triplet defines a learned local latent subspace. The triplet partition is defined over learned projected channels and assumes no predefined semantic ordering of channels. Each ωn,m\omega_{n,m} parameterizes an element of so(3)\mathfrak{so}(3) acting on the corresponding semantic triplet, with camera-dependent geometry provided by the aligned geometry tokens, and its exponential defines an SO(3)\mathrm{SO}(3) structured transformation in the latent subspace.

The latent operator state ωn,m\omega_{n,m} is mapped to a triplet-wise so(3)\mathfrak{so}(3) generator through the hat map:

Ωn,m=[0−ωn,mzωn,myωn,mz0−ωn,mx−ωn,myωn,mx0]∈so(3).(5)\Omega_{n,m} = \begin{bmatrix} 0 & -\omega_{n,m}^{z} & \omega_{n,m}^{y} \\ \omega_{n,m}^{z} & 0 & -\omega_{n,m}^{x} \\ -\omega_{n,m}^{y} & \omega_{n,m}^{x} & 0 \end{bmatrix} \in\mathfrak{so}(3). \tag*{(5)}

The geometry encoder produces triplet-wise so(3)\mathfrak{so}(3) generators over the visual tokens. The output is defined as the Lie state SLie=(Ω,γ)S_{\mathrm{Lie}}=(\Omega,\gamma), Ω=(Ωn,m)∈so(3)N×M\Omega=(\Omega_{n,m})\in\mathfrak{so}(3)^{N\times M}, where Ωn,m\Omega_{n,m} specifies the local operator for the mm-th latent group of token nn, and γ\gamma controls the routed contribution at each token.

Lie-Group Routing

Given the Lie state SLieS_{\mathrm{Lie}} elicited from geometry tokens, we use it to route the 2D semantic representation through geometry conditioned operators with SO(3)\mathrm{SO}(3) structure. We first project the visual tokens into the operator space V=Wv(LN⁡(F2d))∈RN×DoV=W_v(\operatorname{LN}(F^{2d}))\in\mathbb{R}^{N\times D_o}, where Wv:RH→RDoW_v:\mathbb{R}^{H}\to\mathbb{R}^{D_o}. We then partition VV into M=Do/3M=D_o/3 semantic triplets {vn,m∈R3}n,m\{v_{n,m}\in\mathbb{R}^{3}\}_{n,m}, and pair each vn,mv_{n,m} with its corresponding so(3)\mathfrak{so}(3) generator Ωn,m\Omega_{n,m}. The exponential map defines a standard SO(3)\mathrm{SO}(3) action for each generator and admits the Rodrigues closed form [29]. We adopt a diagonal Padé realization [18] that preserves the local rotation structure and yields a closed-form, rationally normalized correction. Its structural properties and local sensitivity are analyzed in Appendix § 8. Its action is expressed through the first- and second-order generator terms:

an,m=Ωn,mvn,m=ωn,m×vn,m,(6)a_{n,m}=\Omega_{n,m}v_{n,m}=\omega_{n,m}\times v_{n,m}, \tag*{(6)}
bn,m=Ωn,m2vn,m=ωn,m×(ωn,m×vn,m)=ωn,m(ωn,mTvn,m)−∥ωn,m∥22vn,m.(7)b_{n,m}=\Omega_{n,m}^{2}v_{n,m}=\omega_{n,m}\times(\omega_{n,m}\times v_{n,m})=\omega_{n,m}(\omega_{n,m}^{\mathsf{T}}v_{n,m})-\lVert\omega_{n,m}\rVert_{2}^{2}v_{n,m}. \tag*{(7)}

Here, an,ma_{n,m} and bn,mb_{n,m} are the first- and second-order generator actions on the semantic triplet. Let R1,1(Ωn,m)R_{1,1}(\Omega_{n,m}) denote the Padé 1/1 approximation to exp⁡(Ωn,m)\exp(\Omega_{n,m}), which is equivalent to the Cayley action reviewed in Appendix § 7. LieVLM retains only the residual action relative to the identity,

cn,mpade=(R1,1(Ωn,m)−I)vn,m=an,m+12bn,m1+14∥ωn,m∥22.(8)c_{n,m}^{\mathrm{pade}}=(R_{1,1}(\Omega_{n,m})-I)v_{n,m}=\frac{a_{n,m}+\frac{1}{2}b_{n,m}}{1+\frac{1}{4}\lVert\omega_{n,m}\rVert_{2}^{2}}. \tag*{(8)}

The corresponding transformed triplet is vn,m+cn,mpadev_{n,m}+c_{n,m}^{\mathrm{pade}}, while only the correction cn,mpadec_{n,m}^{\mathrm{pade}} enters the routed branch. We reshape all triplet-wise corrections into δpade∈RN×Do\delta^{\mathrm{pade}}\in\mathbb{R}^{N\times D_o}. A bias-free output projection maps the routed representation back to the multimodal token space,

Δop(F2d;F3d)=Wo(δpade)∈RN×H,(9)\Delta_{\mathrm{op}}(F^{2d};F^{3d})=W_o(\delta^{\mathrm{pade}})\in\mathbb{R}^{N\times H}, \tag*{(9)}

where Wo:RDo→RHW_o:\mathbb{R}^{D_o}\to\mathbb{R}^{H}. The visual tokens F2dF^{2d} provides the content to be routed, while F3dF^{3d} conditions the so(3)\mathfrak{so}(3) generators and routing strengths. This produces a geometry-conditioned operator correction that is later fused with the original semantic path and geometry anchor.

Geometry-Aware Routed Fusion

The Lie Group Routing module produces the operator correction Δop(F2d;F3d)\Delta_{\mathrm{op}}(F^{2d};F^{3d}), which captures geometry-conditioned semantic transformation under the induced so(3)\mathfrak{so}(3) generators. However, geometry tokens also contain 3D contextual information beyond rotational variation, such as scene layout, structural priors, and cross-view correspondence. We therefore fuse the original 2D semantic path with two complementary geometry paths: a direct 3D context path and a Lie-routed transformation path. The final geometry-aware visual representation is computed as

Fout=F2d+F3d+γ⊙Δop(F2d;F3d),(10)F^{\mathrm{out}}=F^{2d}+F^{3d}+\gamma\odot\Delta_{\mathrm{op}}(F^{2d};F^{3d}), \tag*{(10)}

where ⊙\odot denotes element-wise multiplication and γ∈(0,1)N×H\gamma\in(0,1)^{N\times H} is the token- and channel-wise routing coefficient predicted from the geometry tokens in Eq. (4). The three terms in Eq. (10) play complementary roles. The first term F2dF^{2d} preserves the pretrained 2D visual-semantic representation. The second term F3dF^{3d} provides direct 3D contextual information, and γ⊙Δop(F2d,F3d)\gamma\odot\Delta_{\mathrm{op}}(F^{2d}, F^{3d}) introduces a geometry-conditioned correction to the semantic representation. The direct geometry path keeps 3D evidence available to the language model, while the Lie-routed path uses the geometry-conditioned operators from Sec. 3.3 to model transformation-aware semantic variation. The routing coefficient γ\gamma adaptively controls the Lie routed contribution for each token and channel.

This fusion design preserves the pretrained semantics and direct geometry paths throughout optimization, while the bounded routing coefficient γ\gamma modulates only the Lie-routed transformation. The routing head is learned jointly with the task objective, allowing the transformation strength to adapt across visual tokens and output channels.

Finally, FoutF^{\mathrm{out}} is used as the visual representation passed to the MLLM backbone for language generation. For a video or multi-view input, the same fusion rule is applied to each frame’s aligned token grid, and the resulting fused visual tokens are concatenated with text embeddings following the standard MLLM interface. The model is trained end-to-end with the task objective without requiring explicit supervision on ω\omega, Ω\Omega, or γ\gamma.

Experiments

3D Scene Understanding

Datasets & Evaluation Protocols. We evaluate LieVLM on 3D visual grounding, 3D dense captioning, and 3D video object detection. For grounding, we use the ScanRefer dataset [4] under the protocol of SPAR [40], with Mask3D proposal refinement applied consistently to methods SPAR, VG LLM, and LieVLM. For dense captioning, we use the Scan2Cap dataset [9] and generate object descriptions conditioned on Mask3D proposals from LEO [23], following prior work [23, 43, 44]. For 3D video object detection, we adopt the VG LLM protocol on the EmbodiedScan dataset [36]. We report [email protected]/0.5 for ScanRefer, CIDEr, BLEU-4, METEOR, and ROUGE under the @0.5 criterion for Scan2Cap, and precision, recall, and F1 at IoU 0.25 over 20 common classes for detection. All models are trained for one epoch with Adam, a batch size of 64, a warmup ratio of 0.03, and a peak learning rate of 1×10−51 \times10^{-5} followed by linear decay. The 2D visual encoder, 3D geometry encoder, and multimodal connector are frozen, while the learnable routing bias is initialized to bγ=−4b_{\gamma}=-4 and fixed as the default across all main experiments.

Baselines. The baselines are grouped according to how 3D geometry is incorporated. For ScanRefer and Scan2Cap, we compare with explicit 3D-input methods, including ScanRefer [4], Scan2Cap [9], 3D-LLM [19], L3DA [6], Chat-3D v2 [21], Grounded 3D-LLM [7], LEO [23], Chat-Scene [22], LLaVA-3D [44], and Video-3D LLM [43]. We also include geometry-injected MLLMs, including SPAR [40], 3DRS [24], and VG LLM [42]. For 3D video object detection, we compare against Qwen2.5-VL [1] and VG LLM [42]. Tables 1–3 use published VG LLM results, while controlled analyses use our reproduced 8B baselines.

Results on 3D Dense Captioning. Table 1 reports the Scan2Cap results. LieVLM achieves the best CIDEr score of 82.7, surpassing Video-3D LLM by 2.7 points and LLaVA-3D by 3.5 points. Compared with geometry-injected RGB/video baselines, LieVLM improves over VG LLM-8B from 80.0 to 82.7 in CIDEr while remaining competitive across the other captioning metrics. These results support the effectiveness of Lie routing for object-level captioning.

The quantitative results of 3D dense captioning on Scan2Cap [9]. Underline denotes overall best result, and bold denotes best results among RGB-based methods.

Method[email protected]↑[email protected]↑[email protected]↑[email protected]↑
Explicit 3D Input
Scan2Cap [9]39.123.322.044.8
LL3DA [6]65.236.826.055.0
Chat-3D-v2 [21]63.931.8--
Grounded 3D-LLM [7]70.235.0--
LEO [23]72.438.227.958.1
Chat-Scene [22]77.136.5--
LLaVA-3D [44]79.241.130.263.4
Video-3D LLM [43]80.040.228.561.7
RGB-Based Models
3DRS [24]76.439.627.357.1
VG LLM-4B [42]78.640.928.662.4
VG LLM-8B [42]80.041.528.962.6
LieVLM-8B (Ours)82.741.728.962.9

Table 1.

Results on 3D Visual Grounding. Table 2 shows the ScanRefer results. LieVLM reaches 64.5 [email protected] and 57.2 [email protected]. Under the same proposal refinement protocol, it improves over the published VG LLM-8B results by 6.9 and 6.3 points, respectively, demonstrating stronger grounding performance with geometry conditioned operator routing. Results in Figure 3 show that the predictions closely overlap with the ground-truth boxes, demonstrating accurate language-guided 3D localization.

The results of 3D Visual grounding on ScanRefer. [4]. Underline denotes overall best, and bold denotes best among RGB-based methods.

Model[email protected][email protected]
Explicit 3D Input
ScanRefer [4]37.324.3
Chat-3D v2 [21]35.930.4
Grounded 3D-LLM [7]47.944.1
ChatScene [22]55.550.2
LLaVA-3D [44]54.142.4
Video-3D LLM [43]58.151.7
RGB-Based Models
SPAR [40]48.843.1
3DRS [24]55.251.1
VG LLM-4B [42]53.547.5
VG LLM-8B [42]57.650.9
LieVLM-8B (Ours)64.557.2

Table 2.

Qualitative results of LieVLM for 3D visual grounding

Figure 3. Qualitative results of LieVLM for 3D visual grounding.

Results on 3D Video Object Detection. Table 3 evaluates 3D video object detection under the 4-frame and 6-frame settings. LieVLM improves F1 from 41.2 to 45.8 in the 4-frame setting and from 40.0 to 44.1 in the 6-frame setting. Matched three-seed results in Table 7 further confirm consistent improvements, with detailed statistics in Appendix § 10.§ 2. The category-level effects vary across objects, with particularly strong gains on table, bed and bathtub.

The quantitative results on 3D video object detection on EmbodiedScan [36].

Methodchaircabinettablebincouchbedbathtubtoilet20 Common Classes
[email protected][email protected][email protected]
4-Frame Setting
Qwen2.5-VL [1]41.211.636.530.241.168.236.668.734.631.032.5
VG LLM-8B [42]54.017.146.539.847.074.142.182.543.439.641.2
LieVLM-8B (Ours)53.521.356.843.055.182.975.986.248.644.145.8
6-Frame Setting
Qwen2.5-VL [1]36.110.632.725.040.764.638.468.631.828.029.6
VG LLM-8B [42]48.717.944.838.546.475.840.483.243.538.740.0
LieVLM-8B (Ours)50.019.654.235.551.783.779.986.547.941.944.1

Table 3.

VariantScan2Cap [email protected]ScanRefer [email protected]EmbodiedScan 4F [email protected]EmbodiedScan 6F [email protected]
Additive Only80.07±0.1257.62±0.1841.18±0.2440.05±0.13
Visual-Conditioned Routing80.71±0.1859.56±0.2342.48±0.1941.69±0.22
Generic Cross-Attention80.62±0.1759.08±0.2442.36±0.1841.48±0.21
Learnable 3×33 \times3 Generator80.11±0.1457.91±0.2041.56±0.1540.71±0.18
Spectral-Controlled 3×33 \times3 Generator80.76±0.1559.34±0.2242.61±0.1641.72±0.20
Rational-Controlled 3×33 \times3 Generator81.48±0.1661.27±0.2543.83±0.1742.76±0.21
so(3)\mathfrak{so}(3)–Padé (Ours)82.69±0.1564.56±0.3145.76±0.1344.10±0.26

Table 7. Operator structure ablation under matched training. Results are mean ± std over three seeds.

Qualitative results for 3D video object detection

Figure 4. Qualitative results for 3D video object detection.

General Spatial Reasoning

Datasets & Evaluation Protocols. We train on a 3.05M example generalist mixture constructed from the public data sources released with CA-MLLM [39], covering video instruction following,

@page 7/35 starts_mid=0 ends_mid=1 spatial question answering, 3D grounding and captioning, 3D detection, and object centric spatial reasoning. We evaluate Qwen2.5-VL-7B variants on SPAR-Bench Full [40], CV-Bench-3D [33], and BLINK-Spatial [12], and Qwen3-VL-8B variants on VSI-Bench [38], without benchmark-specific fine tuning. SPAR-Bench reports difficulty and camera sensitive categories, CV-Bench-3D and BLINK-Spatial report accuracy by spatial subtask, and VSI-Bench combines numerical Mean Relative Accuracy with multiple choice accuracy. The training composition is provided in Appendix § 9.§ 3.

MethodData SizeCV-Bench-3D Avg.CV-Bench-3D DepthCV-Bench-3D Dist.BLINK-Spatial Avg.BLINK-Spatial DepthBLINK-Spatial SpatialBLINK-Spatial Multi View
Prior MLLMs and Spatial Models
GPT-4oN/A83.0--67.674.269.259.4
Gemini-2.5-ProN/A90.891.090.771.687.991.635.3
Qwen2.5-VL-7BN/A77.384.576.573.879.086.755.6
Qwen2.5-VL-72BN/A81.088.781.068.679.885.340.6
SPAR-8B2M89.1------
VG LLM-4B297K91.3--68.479.871.354.1
CA-MLLM-4B3.05M90.789.891.577.077.466.487.2
Controlled 3.05M Setting
VG LLM-8B3.05M92.692.193.175.983.577.466.6
LieVLM-8B (Ours)3.05M94.393.994.677.184.078.169.1
+1.7+1.7+1.6+1.2+0.5+0.7+2.5

Table 5. Comparison on CV-Bench-3D and BLINK-Spatial under the 3.05M setting.

Baselines. We compare with representative generalist MLLMs and spatially specialized models, and include SenseNova-SI [2] as an external Qwen3 scale reference. Within each controlled comparison, VG LLM and LieVLM share the same language and geometry backbones, training data, optimization, and evaluation, and differ only in additive geometry fusion or Lie routing.

Generalization across Benchmarks. Table 4 and 5 show consistent matched gains across every reported metric on SPAR-Bench Full, CV-Bench-3D, and BLINK-Spatial. LieVLM reaches the highest SPAR-Bench overall score with its largest gain on viewpoint change inference, and achieves the strongest CV-Bench-3D results across both depth and distance. The improvement further transfers to BLINK-Spatial, providing evidence beyond the original ScanNet centered evaluation suite.

MethodData SizeAvg.LowMediumHighCamMotionViewChgIDistI-OO-MVObjRel-OC-MVSpImag-OO-MV
Prior MLLMs and Spatial Models
InternVL2.5-8B [8]N/A36.2829.4631.8843.8028.006.3256.8551.7539.50
LLaVA-OV-7B [27]N/A31.2021.7926.1340.1430.259.4755.0637.2535.01
Qwen2.5-VL-7B [1]N/A33.0728.7522.9740.2728.756.8351.4944.7531.93
SPAR-8B [40]2M63.2565.5363.0161.2973.0037.1479.7664.0053.50
VG LLM-4B [42]297K60.3652.8151.3572.9263.0025.8263.6981.2571.99
CA-MLLM-4B [39]3.05M68.3559.9460.4281.7473.0031.0168.7587.5081.23
Controlled 3.05M Setting
VG LLM-8B [42]3.05M67.1859.0859.3180.0770.1832.1170.4284.9377.65
LieVLM-8B (Ours)3.05M68.57 +1.3959.81 +0.7360.74 +1.4381.21 +1.1470.92 +0.7435.26 +3.1571.58 +1.1685.34 +0.4178.27 +0.62

Table 4. Comparison on SPAR-Bench Full under the 3.05M setting. Prior model results follow published evaluations, and controlled rows use identical training and evaluation.

Scalability across Backbones and Data. Table 6 shows that the matched benefit of Lie routing persists from the original Qwen2.5 setting to the scaled Qwen3 setting. With Qwen3-VL-8B and the 3.05M mixture, LieVLM raises the VSI-Bench average from 60.6 to 62.8 and improves seven of eight categories, including all four multiple choice categories shown in the table. The 8M SenseNova-SI results provide additional scale context, while the controlled Qwen3 comparison shows that the benefit of Lie routing persists with a stronger backbone and 10.3 times more training data. Full VSI-Bench category results and multi-seed statistics are provided in Appendix § 10.§ 3.

Understanding Lie Routing

Routing Viewpoint Correspondence. A premise of Lie routing is that latent transformations should vary systematically with viewpoint while preserving target-related semantics. For same-scene frame pairs, with tokens matched by shared indices on the aligned visual grid, we relate the SO⁡(3)\operatorname{SO}(3) camera rotation distance drotd_{\mathrm{rot}} to the mean ℓ2\ell_2 change in within-space normalized geometry features, cross-attention residuals, and latent operator states. As shown in Figure 5, operator state variation shows the strongest monotonic association with camera rotation among the representations considered, reaching Spearman ρ=0.870\rho= 0.870 compared with 0.452 for geometry features and 0.576 for cross-attention residuals. The stronger rank association indicates that the predicted transformation states vary systematically with viewpoint under this diagnostic. This characterizes the adaptive behavior of Lie routing and complements the target-localization and task-level evidence in Figure 1.

Routing viewpoint correspondence plots

Figure 5. Routing viewpoint correspondence. Spearman correlation ρ\rho between physical camera rotation and internal representation change across same scene frame pairs.

Operator Structure. To disentangle transformation structure, conditioning source, and response control, we compare variants under the same backbone, direct geometry path, training, and evaluation, with implementation details in Appendix C.1. Additive Only removes routing branch, while Visual-Conditioned Routing predicts the operator state and gate from F2dF^{2d} instead of F3dF^{3d}. Generic cross-attention replaces the structured transformation with cross-attention from F2dF^{2d} to F3dF^{3d}, and Learnable 3×33 \times3 Generator applies an unconstrained geometry-conditioned matrix Gn,mgen=Gn,mvnvmG_{n,m}^{\mathrm{gen}} = G_{n,m}v_n v_m, with spectral- and rational-controlled variants isolating response regulation. As shown in Table 7, cross-attention yields modest gains and the unconstrained generator remains comparable to additive fusion, while spectral and rational controls provide progressive but limited improvements. Visual-conditioned routing likewise recovers only part of the gain. The geometry-conditioned so(3)\mathfrak{so}(3)–Padé formulation remains consistently strongest, supporting the joint role of geometric conditioning and structured operator realization.

Operator Realization and Routing. We further examine the numerical realization of the structured operator and its routing bias initialization. The Rodrigues baseline uses the closed-form action with small-angle series for coefficient evaluation near the identity, providing an exact exponential-action reference. As shown in Table 8 and Table 9, the Padé variants achieve the strongest overall results among the evaluated realizations. Padé 1/1 gives the highest CIDEr and detection F1, while bγ=−4b_\gamma= -4 yields the best CIDEr and detection precision and bγ=−3b_\gamma= -3 gives higher recall. Appendix B characterizes its rotation structure, response normalization, and local sensitivity.

MethodScan2Cap [email protected]Scan2Cap [email protected]3D Video Detection P25_{25}3D Video Detection R25_{25}3D Video Detection F125_{25}
Rodrigues81.741.447.842.744.5
Taylor-181.241.347.242.343.9
Taylor-281.641.547.642.844.2
Taylor-381.941.447.942.944.6
Padé-2/282.441.648.443.345.3
Padé-1/1 (Ours)82.741.748.644.145.8

Table 8. Comparison of numerical realizations for the proposed Lie Group Routing operator.

bγb_\gammaScan2Cap [email protected]Scan2Cap [email protected]3D Video Detection P25_{25}3D Video Detection R25_{25}3D Video Detection F125_{25}
081.841.447.743.745.2
−182.041.547.943.945.4
−382.541.748.444.245.7
−482.741.848.644.145.8
−682.341.648.244.045.6

Table 9. Sensitivity to the initial routing bias bγb_\gamma in Lie State Elicitation.

Conclusion

We presented LieVLM, a Lie routing framework for view-robust 3D MLLMs. Rather than treating geometry as auxiliary context alone, LieVLM uses geometry tokens to predict local so(3)\mathfrak{so}(3) states and routing coefficients that govern structured transformations of pretrained visual semantics across viewpoints. Through Lie-State Elicitation, Lie-Group Routing, and Geometry-Aware Routed Fusion, the model injects gated corrections while preserving the original visual-semantic, thereby mitigating viewpoint-induced semantic drift. Experiments across 3D scene understanding and spatial reasoning show consistent gains over additive geometry fusion, with benefits that generalize across benchmarks, training scales, and stronger backbones. Controlled ablations and viewpoint analyses further connect these gains to geometry-conditioned routing and structured transformation design.

Acknowledgements This work is supported by Zhejiang Leading Innovative and Entrepreneur Team Introduction Program (2024R01007). This work is also supported by The “Pioneer” and “Leading Goose” Research and Development Program of Zhejiang (2025C02077). Jingjun Yi sincerely thanks Prof. Pascal Mettes (University of Amsterdam) for his extensive help in revising the manuscript, greatly improving its clarity and presentation.

Supplementary Material

Background: Exponential Map on SO(3)SO(3)

This section provides background on the numerical realization of the exponential action on SO(3)SO(3). We first recall the exact Rodrigues formula, and then review Taylor and Padé approximations, which are commonly used to implement matrix exponentials in practice.

Rodrigues Formula

Let Ω=ω∧∈so(3)\Omega= \omega^{\wedge} \in\mathfrak{so}(3) and θ=∥ω∥2\theta= \lVert\omega\rVert_2. The exponential map from so(3)\mathfrak{so}(3) to SO(3)SO(3) admits the classical Rodrigues formula,

exp⁡(Ω)=I+A(θ)Ω+B(θ)Ω2,(11)\exp(\Omega) = I + A(\theta)\Omega+ B(\theta)\Omega^2, \tag*{(11)}

where, for θ>0\theta> 0, the scalar coefficients are

A(θ)=sin⁡θθ,B(θ)=1−cos⁡θθ2.(12)A(\theta) = \frac{\sin\theta}{\theta}, \qquad B(\theta) = \frac{1-\cos\theta}{\theta^2}. \tag*{(12)}

For a vector v∈R3v \in\mathbb{R}^3, the corresponding action is

exp⁡(Ω)v=v+A(θ)Ωv+B(θ)Ω2v.(13)\exp(\Omega)v = v + A(\theta)\Omega v + B(\theta)\Omega^2v. \tag*{(13)}

The Rodrigues formula is exact in real arithmetic, but direct evaluation of its coefficient ratios requires care near θ=0\theta= 0. Both ratios have removable singularities with limits

lim⁡θ→0A(θ)=1,lim⁡θ→0B(θ)=12.(14)\lim_{\theta\to0} A(\theta) = 1, \qquad\lim_{\theta\to0} B(\theta) = \frac{1}{2}. \tag*{(14)}

We set A(0)=1A(0) = 1 and B(0)=1/2B(0) = 1/2 by continuous extension. Taylor expansions provide regular local approximations for evaluating the coefficients near zero. The apparent singularities therefore belong to the ratio expressions, while the exponential map itself remains smooth.

Taylor Expansion

Taylor expansion [18] provides the most direct power-series realization of the matrix exponential. It follows from the analytic definition of the exponential map, which represents exp⁡(Ω)\exp(\Omega) as an infinite sum of weighted powers of the generator Ω\Omega. Truncating this series yields a polynomial approximation whose accuracy improves near the identity as more orders are retained. For Ω∈so(3)\Omega\in\mathfrak{so}(3) and v∈R3v \in\mathbb{R}^3, the exponential action can be written as

exp⁡(Ω)v=∑k=0∞1k!Ωkv=v+Ωv+12Ω2v+16Ω3v+⋯ .(15)\exp(\Omega)v = \sum_{k=0}^{\infty}\frac{1}{k!}\Omega^k v = v + \Omega v + \frac{1}{2}\Omega^2v + \frac{1}{6}\Omega^3v + \cdots. \tag*{(15)}

A finite Taylor approximation of order KK is therefore

ΦTaylor−K(Ω,v)=∑k=0K1k!Ωkv.(16)\Phi_{\mathrm{Taylor-}K}(\Omega,v) = \sum_{k=0}^{K}\frac{1}{k!}\Omega^k v. \tag*{(16)}

This polynomial form is smooth around the identity and avoids the explicit trigonometric ratios in Rodrigues formula, making it suitable for small-angle implementations. However, finite Taylor truncations control the response only through the truncation order and ∥Ω∥\lVert\Omega\rVert, without the rational normalization introduced by Padé approximations.

Padé Approximation

Padé approximation [18] provides a rational-function realization of a given analytic function. Given a scalar function f(x)f(x), its (m,n)(m,n) Padé approximant takes the form

Rm,n(x)=Pm(x)Qn(x)=a0+a1x+⋯+amxm1+b1x+⋯+bnxn,(17)R_{m,n}(x) = \frac{P_m(x)}{Q_n(x)} = \frac{a_0 + a_1x + \cdots+ a_mx^m}{1 + b_1x + \cdots+ b_nx^n}, \tag*{(17)}

where the coefficients are chosen such that the Taylor expansion of Rm,n(x)R_{m,n}(x) matches that of f(x)f(x) up to the highest possible order around the expansion point. Compared with Taylor truncation, Padé approximation uses both numerator and denominator polynomials, and therefore provides a rational rather than purely polynomial approximation.

For the matrix exponential, Padé approximation replaces the infinite exponential series with a rational matrix function. The commonly used diagonal Padé forms approximate exp⁡(Ω)\exp(\Omega) by a ratio of two matrix polynomials. In particular, the Padé-1/1 form is

R1,1(Ω)=(I−12Ω)−1(I+12Ω),(18)R_{1,1}(\Omega)=\left(I-\frac{1}{2}\Omega\right)^{-1}\left(I+\frac{1}{2}\Omega\right), \tag*{(18)}

where (⋅)−1(\cdot)^{-1} denotes the matrix inverse, usually implemented by solving a linear system. This form is exactly the Cayley map applied to Ω/2\Omega/2. A higher-order diagonal variant is the Padé-2/2 form:

R2,2(Ω)=(I−12Ω+112Ω2)−1(I+12Ω+112Ω2).(19)R_{2,2}(\Omega)=\left(I-\frac{1}{2}\Omega+\frac{1}{12}\Omega^2\right)^{-1}\left(I+\frac{1}{2}\Omega+\frac{1}{12}\Omega^2\right). \tag*{(19)}

Both Padé-1/1 and Padé-2/2 are rational approximations to exp⁡(Ω)\exp(\Omega). When Ω∈so(3)\Omega\in\mathfrak{so}(3) is skew-symmetric, these diagonal Padé forms preserve the rotation structure because the denominator polynomial is the transpose of the numerator polynomial. Thus, they define valid SO(3)\mathrm{SO}(3) actions while approximating the exact exponential map.

Theoretical Analysis

This section analyzes the local sensitivity and structural properties of the operator realizations. We examine direct Rodrigues evaluation near the identity, establish Jacobian bounds for Padé actions, and compare the rotation structure and response magnitude of finite Taylor and Padé forms.

Direct Rodrigues Evaluation Near the Identity

Rodrigues formula expresses the exact exponential action through the coefficients A(θ)A(\theta) and B(θ)B(\theta). Their direct derivative expressions involve subtracting nearly equal quantities near θ=0\theta=0, which can amplify rounding errors during evaluation. We examine this computational form while retaining the continuous limits of the underlying functions.

Specifically, differentiating A(θ)=sin⁡θθA(\theta)=\frac{\sin\theta}{\theta}, B(θ)=1−cos⁡θθ2B(\theta)=\frac{1-\cos\theta}{\theta^2} gives

dA=θcos⁡θ−sin⁡θθ3(ωTdω),dB=θsin⁡θ−2(1−cos⁡θ)θ4(ωTdω).(20)dA=\frac{\theta\cos\theta-\sin\theta}{\theta^3}(\omega^{\mathsf{T}}d\omega),\qquad dB=\frac{\theta\sin\theta-2(1-\cos\theta)}{\theta^4}(\omega^{\mathsf{T}}d\omega). \tag*{(20)}

The scalar coefficients in Eq. (20) have well-defined limits:

lim⁡θ→0θcos⁡θ−sin⁡θθ3=−13,lim⁡θ→0θsin⁡θ−2(1−cos⁡θ)θ4=−112.(21)\lim_{\theta\to0}\frac{\theta\cos\theta-\sin\theta}{\theta^3}=-\frac{1}{3},\qquad\lim_{\theta\to0}\frac{\theta\sin\theta-2(1-\cos\theta)}{\theta^4}=-\frac{1}{12}. \tag*{(21)}

Local Jacobian Bounds for Padé Routing

We establish a local Jacobian bound for the full Padé 1/1 action underlying the operator correction. The derivative is taken with respect to the predicted operator state while holding the visual feature fixed. The argument extends to general Padé forms under a uniform nonsingularity condition on the denominator.

Proposition B.1 (Local Jacobian bound). Let Ω=ω∧∈so(3)\Omega=\omega^{\wedge}\in\mathfrak{so}(3) and define the Padé-1/1 routing action as

Φ1,1(ω,v)=v+Ωv+12Ω2v1+14∥ω∥22.(22)\Phi_{1,1}(\omega,v)=v+\frac{\Omega v+\frac{1}{2}\Omega^2v}{1+\frac{1}{4}\lVert\omega\rVert_2^2}. \tag*{(22)}

For any ρ>0\rho>0, there exists a constant Cρ<∞C_{\rho}<\infty such that, for all ∥ω∥2≤ρ\lVert\omega\rVert_2\leq\rho and v∈R3v\in\mathbb{R}^3,

∥∂Φ1,1(ω,v)∂ω∥≤Cρ∥v∥2.(23)\left\lVert\frac{\partial\Phi_{1,1}(\omega,v)}{\partial\omega}\right\rVert\leq C_{\rho}\lVert v\rVert_2. \tag*{(23)}

The Jacobian norm is the operator norm induced by the Euclidean norm.

Proof. Let

q(ω)=1+14∥ω∥22,N(ω,v)=Ωv+12Ω2v.(24)q(\omega)=1+\frac{1}{4}\lVert\omega\rVert_2^2,\qquad N(\omega,v)=\Omega v+\frac{1}{2}\Omega^2v. \tag*{(24)}

Then Φ1,1(ω,v)=v+N(ω,v)/q(ω)\Phi_{1,1}(\omega,v)=v+N(\omega,v)/q(\omega). For any perturbation direction h∈R3h\in\mathbb{R}^3, since Ω=ω∧\Omega=\omega^\wedge depends linearly on ω\omega, we have

DΩ[h]=h∧.(25)D\Omega[h]=h^\wedge. \tag*{(25)}

Therefore, the differentials of the numerator and denominator are

DN(ω,v)[h]=h∧v+12(h∧Ωv+Ωh∧v),(26)DN(\omega,v)[h]=h^\wedge v+\frac{1}{2}(h^\wedge\Omega v+\Omega h^\wedge v), \tag*{(26)}
Dq(ω)[h]=12ωTh.Dq(\omega)[h]=\frac{1}{2}\omega^\mathsf{T}h.

Since ∥x∧∥2=∥x∥2\lVert x^\wedge\rVert_2=\lVert x\rVert_2 for any x∈R3x\in\mathbb{R}^3, when ∥ω∥2≤ρ\lVert\omega\rVert_2\leq\rho, we have

∥N(ω,v)∥2≤(ρ+12ρ2)∥v∥2,\lVert N(\omega,v)\rVert_2\leq\left(\rho+\frac{1}{2}\rho^2\right)\lVert v\rVert_2,
∥DN(ω,v)[h]∥2≤(1+ρ)∥h∥2∥v∥2,(27)\lVert DN(\omega,v)[h]\rVert_2\leq(1+\rho)\lVert h\rVert_2\lVert v\rVert_2, \tag*{(27)}
q(ω)≥1,∣Dq(ω)[h]∣≤12ρ∥h∥2.q(\omega)\geq1,\qquad\lvert Dq(\omega)[h]\rvert\leq\frac{1}{2}\rho\lVert h\rVert_2.

Using the above relations, the directional derivative of Φ1,1\Phi_{1,1} can be bounded as follows:

∥DΦ1,1(ω,v)[h]∥2=(1)∥DN(ω,v)[h]q(ω)−N(ω,v)Dq(ω)[h]q(ω)2∥2≤(2)∥DN(ω,v)[h]∥2q(ω)+∥N(ω,v)∥2∣Dq(ω)[h]∣q(ω)2≤(3)(1+ρ)∥h∥2∥v∥2+12ρ(ρ+12ρ2)∥h∥2∥v∥2=((1+ρ)+12ρ(ρ+12ρ2))∥h∥2∥v∥2=Cρ∥h∥2∥v∥2,(28)\begin{aligned} \lVert D\Phi_{1,1}(\omega,v)[h]\rVert_2 &\mathrel{\overset{(1)}{=}} \left\lVert\frac{DN(\omega,v)[h]q(\omega)-N(\omega,v)Dq(\omega)[h]}{q(\omega)^2}\right\rVert_2\\ &\mathrel{\overset{(2)}{\leq}} \frac{\lVert DN(\omega,v)[h]\rVert_2}{q(\omega)} +\frac{\lVert N(\omega,v)\rVert_2\lvert Dq(\omega)[h]\rvert}{q(\omega)^2}\\ &\mathrel{\overset{(3)}{\leq}} (1+\rho)\lVert h\rVert_2\lVert v\rVert_2 +\frac{1}{2}\rho\left(\rho+\frac{1}{2}\rho^2\right)\lVert h\rVert_2\lVert v\rVert_2\\ &=\left((1+\rho)+\frac{1}{2}\rho\left(\rho+\frac{1}{2}\rho^2\right)\right)\lVert h\rVert_2\lVert v\rVert_2\\ &=C_\rho\lVert h\rVert_2\lVert v\rVert_2, \tag*{(28)} \end{aligned}

where Cρ=(1+ρ)+12ρ(ρ+12ρ2)<∞C_\rho=(1+\rho)+\frac{1}{2}\rho\left(\rho+\frac{1}{2}\rho^2\right)<\infty.

The derivation of Eq. (28) follows.

(1) follows from the quotient rule applied to Φ1,1(ω,v)=v+N(ω,v)/q(ω)\Phi_{1,1}(\omega,v)=v+N(\omega,v)/q(\omega).

(2) follows from the triangle inequality and norm sub-multiplicativity.

(3) follows from the bounds in Eq. (27) and the fact that q(ω)≥1q(\omega)\geq1.

Taking the supremum over all perturbation directions with ∥h∥2=1\lVert h\rVert_2=1 gives

∥∂Φ1,1(ω,v)∂ω∥≤Cρ∥v∥2.(29)\left\lVert\frac{\partial\Phi_{1,1}(\omega,v)}{\partial\omega}\right\rVert\leq C_\rho\lVert v\rVert_2. \tag*{(29)}

This completes the proof.

Remark B.2 (Extension to Padé-m/nm/n)

The above proof uses Padé-1/1 as the default case in our main model, but the same argument applies to a general Padé-m/nm/n routing action under a local nonsingularity condition. Let

Pm(Ω)=∑i=0mpiΩi,Qn(Ω)=∑j=0nqjΩj,q0=1,(30)P_m(\Omega)=\sum_{i=0}^{m}p_i\Omega^i,\qquad Q_n(\Omega)=\sum_{j=0}^{n}q_j\Omega^j,\qquad q_0=1, \tag*{(30)}

and define

Φm,n(ω,v)=Qn(Ω)−1Pm(Ω)v.(31)\Phi_{m,n}(\omega,v)=Q_n(\Omega)^{-1}P_m(\Omega)v. \tag*{(31)}

Assume that, for all ∥ω∥2≤ρ\lVert\omega\rVert_2\leq\rho, the denominator matrix is locally nonsingular:

σmin⁡(Qn(Ω))≥αm,n,ρ>0.(32)\sigma_{\min}(Q_n(\Omega))\geq\alpha_{m,n,\rho}>0. \tag*{(32)}

We further define the coefficient-dependent quantities

Pm,ρ=∑i=0m∣pi∣ρi,Pm,ρ′=∑i=1mi∣pi∣ρi−1,Qn,ρ′=∑j=1nj∣qj∣ρj−1.(33)P_{m,\rho}=\sum_{i=0}^{m}|p_i|\rho^i,\qquad P'_{m,\rho}=\sum_{i=1}^{m}i|p_i|\rho^{i-1},\qquad Q'_{n,\rho}=\sum_{j=1}^{n}j|q_j|\rho^{j-1}. \tag*{(33)}

For any perturbation direction h∈R3h \in\mathbb{R}^3, since DΩ[h]=h∧D\Omega[h]=h^\wedge and ∥h∧∥2=∥h∥2\lVert h^\wedge\rVert_2=\lVert h\rVert_2, we have

∥DΩk[h]∥2=∥∑ℓ=0k−1Ωℓh∧Ωk−1−ℓ∥2≤kρk−1∥h∥2.(34)\lVert D\Omega^k[h]\rVert_2=\left\lVert\sum_{\ell=0}^{k-1}\Omega^\ell h^\wedge\Omega^{k-1-\ell}\right\rVert_2\leq k\rho^{k-1}\lVert h\rVert_2. \tag*{(34)}

Therefore,

∥Pm(Ω)v∥2≤Pm,ρ∥v∥2,∥DPm(Ω)[h]v∥2≤Pm,ρ′∥h∥2∥v∥2,∥DQn(Ω)[h]∥2≤Qn,ρ′∥h∥2,∥Qn(Ω)−1∥2≤αm,n,ρ−1.(35)\begin{aligned} \lVert P_m(\Omega)v\rVert_2&\leq P_{m,\rho}\lVert v\rVert_2,\\ \lVert DP_m(\Omega)[h]v\rVert_2&\leq P'_{m,\rho}\lVert h\rVert_2\lVert v\rVert_2,\\ \lVert DQ_n(\Omega)[h]\rVert_2&\leq Q'_{n,\rho}\lVert h\rVert_2,\\ \lVert Q_n(\Omega)^{-1}\rVert_2&\leq\alpha^{-1}_{m,n,\rho}. \tag*{(35)} \end{aligned}

Using these bounds, the directional derivative of Φm,n\Phi_{m,n} can be bounded as

∥DΦm,n(ω,v)[h]∥2=(1)∥−Qn−1(DQn[h])Qn−1Pmv+Qn−1(DPm[h])v∥2≤(2)∥Qn−1∥2∥DPm[h]v∥2+∥Qn−1∥22∥DQn[h]∥2∥Pmv∥2≤(3)(Pm,ρ′αm,n,ρ+Qn,ρ′Pm,ραm,n,ρ2)∥h∥2∥v∥2=Cm,n,ρ∥h∥2∥v∥2.(36)\begin{aligned} \lVert D\Phi_{m,n}(\omega,v)[h]\rVert_2 &\overset{(1)}{=}\left\lVert-Q_n^{-1}(DQ_n[h])Q_n^{-1}P_mv+Q_n^{-1}(DP_m[h])v\right\rVert_2\\ &\overset{(2)}{\leq}\lVert Q_n^{-1}\rVert_2\lVert DP_m[h]v\rVert_2+\lVert Q_n^{-1}\rVert_2^2\lVert DQ_n[h]\rVert_2\lVert P_mv\rVert_2\\ &\overset{(3)}{\leq}\left(\frac{P'_{m,\rho}}{\alpha_{m,n,\rho}}+\frac{Q'_{n,\rho}P_{m,\rho}}{\alpha_{m,n,\rho}^2}\right)\lVert h\rVert_2\lVert v\rVert_2\\ &=C_{m,n,\rho}\lVert h\rVert_2\lVert v\rVert_2. \tag*{(36)} \end{aligned}

where

Cm,n,ρ=Pm,ρ′αm,n,ρ+Qn,ρ′Pm,ραm,n,ρ2.(37)C_{m,n,\rho}=\frac{P'_{m,\rho}}{\alpha_{m,n,\rho}}+\frac{Q'_{n,\rho}P_{m,\rho}}{\alpha_{m,n,\rho}^2}. \tag*{(37)}

The derivation of (36) follows.

(1) follows from the product rule and the inverse differential identity D(Q−1)=−Q−1(DQ)Q−1D(Q^{-1})=-Q^{-1}(DQ)Q^{-1}.

(2) follows from the triangle inequality and norm sub-multiplicativity.

(3) follows from the bounds in (35).

Taking the supremum over all perturbation directions with ∥h∥2=1\lVert h\rVert_2=1 gives

∥∂Φm,n(ω,v)∂ω∥≤Cm,n,ρ∥v∥2.(38)\left\lVert\frac{\partial\Phi_{m,n}(\omega,v)}{\partial\omega}\right\rVert\leq C_{m,n,\rho}\lVert v\rVert_2. \tag*{(38)}

Thus, the Padé action has a locally bounded Jacobian under the stated denominator condition. For Padé 1/1, the correction used in the main model has the strictly positive scalar denominator $1+\lVert\omega\rVert^2/4.

Remark B.3. The bound characterizes the sensitivity of the exact Padé action to the predicted operator state. The implemented triplet correction is cn,mpade=Φ1,1(ωn,m,vn,m)−vn,mc^{\mathrm{pade}}_{n,m}=\Phi_{1,1}(\omega_{n,m},v_{n,m})-v_{n,m}. For fixed vv, subtracting vv leaves the Jacobian with respect to ω\omega unchanged, so the same bound applies to the implemented correction. This result concerns input sensitivity in exact arithmetic, independently of finite precision rounding error.

Structure and Response of Taylor and Padé Actions

Finite Taylor and diagonal Padé forms provide smooth realizations of the operator action near the identity. We compare their rotation structure and the magnitude of the induced semantic change.

First, finite Taylor truncations do not generally preserve the SO(3) constraint. Let TK(Ω)=∑k=0K1k!ΩkT_K(\Omega)=\sum_{k=0}^{K}\frac{1}{k!}\Omega^k. Since Ω⊤=−Ω\Omega^\top=-\Omega, its orthogonality defect is

TK(Ω)⊤TK(Ω)−I=TK(−Ω)TK(Ω)−I.(39)T_K(\Omega)^\top T_K(\Omega)-I=T_K(-\Omega)T_K(\Omega)-I. \tag*{(39)}

This defect is composed of higher-order powers of Ω\Omega. For example,

T1(Ω)TT1(Ω)−I=−Ω2,(40)T_1(\Omega)^{\mathsf{T}}T_1(\Omega)-I=-\Omega^2, \tag*{(40)}
T2(Ω)TT2(Ω)−I=14Ω4,T_2(\Omega)^{\mathsf{T}}T_2(\Omega)-I=\frac{1}{4}\Omega^4,
T3(Ω)TT3(Ω)−I=−112Ω4−136Ω6.T_3(\Omega)^{\mathsf{T}}T_3(\Omega)-I=-\frac{1}{12}\Omega^4-\frac{1}{36}\Omega^6.

These expressions quantify the orthogonality defect of finite Taylor truncations. The defect depends on both the truncation order and generator magnitude, and finite truncation does not guarantee a rotation matrix for arbitrary generators. As ∥Ω∥2=∥ω∥2\lVert\Omega\rVert_2=\lVert\omega\rVert_2 increases, these residual high-order terms become more visible, so Taylor truncations do not strictly define rotation matrices unless additional projection or re-orthogonalization is applied.

In contrast, diagonal Padé forms preserve the orthogonal structure for skew-symmetric generators. A diagonal Padé approximation can be written as

Rℓ,ℓ(Ω)=Qℓ(Ω)−1Pℓ(Ω),Qℓ(Ω)=Pℓ(−Ω).(41)R_{\ell,\ell}(\Omega)=Q_{\ell}(\Omega)^{-1}P_{\ell}(\Omega),\qquad Q_{\ell}(\Omega)=P_{\ell}(-\Omega). \tag*{(41)}

Since ΩT=−Ω\Omega^{\mathsf{T}}=-\Omega, we have Qℓ(Ω)=Pℓ(Ω)TQ_{\ell}(\Omega)=P_{\ell}(\Omega)^{\mathsf{T}}. Therefore,

Rℓ,ℓ(Ω)TRℓ,ℓ(Ω)=Qℓ(Ω)Pℓ(Ω)−1Qℓ(Ω)−1Pℓ(Ω)=I,(42)R_{\ell,\ell}(\Omega)^{\mathsf{T}}R_{\ell,\ell}(\Omega)=Q_{\ell}(\Omega)P_{\ell}(\Omega)^{-1}Q_{\ell}(\Omega)^{-1}P_{\ell}(\Omega)=I, \tag*{(42)}

where the last equality follows because Pℓ(Ω)P_{\ell}(\Omega) and Qℓ(Ω)Q_{\ell}(\Omega) are polynomials of the same matrix and hence commute. If Qℓ(tΩ)Q_{\ell}(t\Omega) remains nonsingular for all t∈[0,1]t\in[0,1], continuity from Rℓ,ℓ(0)=IR_{\ell,\ell}(0)=I further gives det⁡Rℓ,ℓ(Ω)=1\det R_{\ell,\ell}(\Omega)=1. In particular, the Padé 1/1 and Padé 2/2 forms used here have nonsingular denominators for real skew-symmetric generators and therefore yield matrices in SO(3)\mathrm{SO}(3).

Second, Taylor truncations are purely polynomial. For the finite Taylor approximation in Eq. (16), the operator induced change satisfies

∥ΦTaylor-K(Ω,v)−v∥2≤(∑k=1K∥ω∥2kk!)∥v∥2.(43)\left\lVert\Phi_{\mathrm{Taylor\text{-}K}}(\Omega,v)-v\right\rVert_2\le\left(\sum_{k=1}^{K}\frac{\lVert\omega\rVert_2^k}{k!}\right)\lVert v\rVert_2. \tag*{(43)}

By contrast, the Padé-1/1 routing form has the bound

∥Φ1,1(ω,v)−v∥2≤∥ω∥2+12∥ω∥221+14∥ω∥22∥v∥2.(44)\left\lVert\Phi_{1,1}(\omega,v)-v\right\rVert_2\le\frac{\lVert\omega\rVert_2+\frac{1}{2}\lVert\omega\rVert_2^2}{1+\frac{1}{4}\lVert\omega\rVert_2^2}\lVert v\rVert_2. \tag*{(44)}

The denominator in (44) provides a rational normalization of the linear and quadratic generator actions. Finite Taylor forms avoid direct evaluation of the Rodrigues coefficient ratios, while Padé 1/1 additionally preserves the rotation structure and introduces rational normalization of the operator response.

Experimental Setting and Details

Implementation Details

LieVLM uses VGGT-1B as the geometry encoder and Qwen2.5-VL-7B as the default language backbone. For the scaled spatial reasoning experiments in § 4.2, we additionally instantiate matched VG LLM and LieVLM variants with Qwen3-VL-8B-Instruct. Within each matched setting, the two models use identical initialization, training data, optimization, and evaluation, and differ only in additive geometry fusion or Lie routing. For the original setting, we train for one epoch using Adam with a batch size of 64. The learning rate is warmed up with a ratio of 0.03 to a peak value of 1×10−51\times10^{-5}, and then linearly decayed to zero. At each iteration, one data source is sampled from the mixed training corpus to form the batch. During training, the 2D visual encoder, the 3D geometry encoder, and the multimodal connector are kept frozen. We optimize the language backbone together with the proposed LieVLM modules, including the rotation head, routing head, and routed fusion layers. The output projection is applied to the correction cn,mpade=R1,1(Ωn,m)vn,m−vn,mc_{n,m}^{\mathrm{pade}}=R_{1,1}(\Omega_{n,m})v_{n,m}-v_{n,m}, rather than to the full transformed triplet. The routing head Wγ:RH→RHW_{\gamma}:\mathbb{R}^{H}\to\mathbb{R}^{H} predicts a token and channel wise gate γ∈(0,1)N×H\gamma\in(0,1)^{N\times H}, which modulates the routed feature after the output projection WoW_o. The learnable routing bias bγb_\gamma is initialized to −4-4 and optimized jointly with the routing head. For the spectral-controlled 3×33 \times3 generator in Table 7, we retain the prediction head and parameter count of the unconstrained dynamic-matrix baseline but normalize each predicted residual matrix as

Gˉn,m=τGn,mmax⁡(τ,∥Gn,m∥2),cn,mspec=Gˉn,mvn,m.(45)\bar{G}_{n,m} = \frac{\tau G_{n,m}}{\max(\tau,\lVert G_{n,m}\rVert_2)}, \qquad c^{\mathrm{spec}}_{n,m} = \bar{G}_{n,m}v_{n,m}. \tag*{(45)}

We set τ=23/7≈1.309\tau= 2\sqrt{3/7} \approx1.309, matching the maximum spectral norm of R1,1(Ω)−IR_{1,1}(\Omega)-I under the component-wise bounded state ω=tanh⁡(WωZ)\omega= \tanh(W_\omega Z). The baseline otherwise shares the direct geometry path, routing gate, output projection, initialization, and training protocol with the unconstrained generator. For the rational-controlled 3×33 \times3 generator, we retain the same unconstrained prediction head but replace the hard spectral cap with

cn,mrat=Gn,mvn,m1+18∥Gn,m∥F2.(46)c^{\mathrm{rat}}_{n,m} = \frac{G_{n,m}v_{n,m}}{1+\frac{1}{8}\lVert G_{n,m}\rVert_F^2}. \tag*{(46)}

The factor 1/81/8 matches the Padé-1/1 denominator for a skew-symmetric generator because ∥ω∧∥F2=2∥ω∥22\lVert\omega^\wedge\rVert_F^2 = 2\lVert\omega\rVert_2^2. This variant retains the same direct geometry path, routing gate, output projection, parameter count, and training protocol. For visual-conditioned routing, we retain the complete so(3)\mathfrak{so}(3)–Padé routing architecture and direct geometry path, but replace the conditioning input in Eq. (4) with Z2d=LN⁡(F2d)Z^{2d} = \operatorname{LN}(F^{2d}). Both ω\omega and γ\gamma are therefore predicted from visual semantics rather than geometry tokens, with all other components and training settings unchanged. The MLLM interface width is H=3584H = 3584 for Qwen2.5-VL-7B and H=4096H = 4096 for Qwen3-VL-8B. We set Do=3⌊H/3⌋D_o = 3\lfloor H/3\rfloor, giving (Do,M)=(3582,1194)(D_o, M) = (3582, 1194) and (4095,1365)(4095, 1365) for the two backbones, respectively. The projections WvW_v and WωW_\omega map HH to DoD_o, WγW_\gamma maps HH to HH, and the bias free WoW_o maps DoD_o back to HH. The four projection matrices contain 3HDo+H23HD_o + H^2 weights, corresponding to 51.36M and 67.10M parameters for the two settings, excluding biases. Following the geometry fusion interface of VG LLM [42], we adopt the same VGGT-to-MLLM token alignment and projection procedure without modification. The aligned VGGT features are projected to the interface width HH before fusion. The resulting geometry tokens F3dF^{3d} share the token indexing and channel dimension of F2dF^{2d}. The same aligned geometry representation supplies both the direct geometry path and Lie State Elicitation, so geometry injection and operator prediction operate on a common token grid. All experiments are conducted on 8 H100 80G GPUs. For 3D scene understanding, training takes approximately 12 hours for LieVLM. For the original 297K VG LLM spatial reasoning setting, training takes approximately 9 hours.

Data Preparation

We use publicly available datasets and follow the VG LLM [42] preprocessing protocol to convert them into the input-output format required by LieVLM.

3D Scene Understanding. For ScanRefer and Scan2Cap, we sample 32 RGB frames per scene. For 3D video object detection, each sample contains a 4-frame clip at 1 FPS, with annotations in the first-frame coordinate system.

SPAR-7M. We process SPAR-7M with the official implementation2, retaining spatial QA samples with valid visual inputs and rendering visual markers using the released pipeline. We exclude navigation samples because of their variable-length image sequences.

LLaVA-Video-178K. We use the LLaVA-Hound split as general video instruction data, sampling frames at 2 FPS and keeping 4–8 frames per video.

297K Spatial Reasoning Setting. Following VG LLM [42], we use 234K SPAR-7M samples and 63K LLaVA-Hound samples, corresponding to 3% and 25% of the two source datasets. The same composition and preprocessing are used for VG LLM and LieVLM.

Large Scale Spatial Reasoning Setting

Training Mixture. Following the public data sources and annotations released with CA-MLLM, we construct a 3.05M example generalist mixture covering video instruction following, spatial question answering, 3D grounding and captioning, 3D detection, and object centric spatial reasoning. We retain our own architecture and training pipeline and do not use the camera ray embeddings, explicit camera intrinsics, camera specific augmentation, or UniDepth distillation introduced by CA-MLLM. Table 10 summarizes the complete composition.

Data ComponentPublic SourcesSamplingExamples
General video instructionLLaVA Hound split of LLaVA Video 178KFull released split0.26M
Large scale spatial QASPAR subsets from ScanNet, ScanNet++, Structured3D, and RxR10% of each training subset1.63M
3D grounding and captioningScanRefer and Scan2CapOfficial training splits0.07M
Other spatially grounded tasksScanNet, 3RScan, Matterport3D, ARKitScenes, Objectron, and SUN RGB-DReleased annotations1.09M
Total––3.05M

Table 10. Composition of the 3.05M generalist training mixture.

Matched Training and Evaluation. The Qwen2.5 variants start from the same Qwen2.5-VL-7B and VGGT-1B checkpoints, while the Qwen3 variants start from the same Qwen3-VL-8B-Instruct and VGGT-1B checkpoints. Within each pair, VG LLM and LieVLM use identical training examples, training settings, and evaluation implementations. The only architectural difference is additive geometry fusion or Lie routing. The evaluated checkpoints receive no benchmark specific fine tuning.

Data Exposure. The complete pretraining and post training data of Qwen2.5-VL are not publicly auditable, so prior exposure to ScanNet or ScanNet++ cannot be excluded. Such exposure may affect the absolute scores of both variants. Within each matched comparison, VG LLM and LieVLM use the same base checkpoint and instruction tuning data, so this uncertainty does not create a differential data exposure advantage for LieVLM. CV-Bench-3D and BLINK-Spatial further provide evaluation outside the original ScanNet centered benchmark suite.

Evaluation Details

Following VG LLM [42], we evaluate LieVLM with LMMs-Eval3 and use customized task interfaces when necessary. During inference, all responses are generated with greedy decoding.

3D Dense Captioning. For Scan2Cap, we follow the common two-stage evaluation protocol [44, 43]. Object proposals are first obtained from the Mask3D detections provided by LEO [23]. We transform each proposal center into the reference-frame coordinate system using the corresponding extrinsic matrices [36], and then prompt the model to describe the object conditioned on this coordinate. Generated captions are evaluated with CIDEr, BLEU-4, ROUGE, and METEOR using the COCO caption evaluation toolkit4.

3D Visual Grounding. For ScanRefer, the model predicts both the frame where the referred object is visible and the corresponding oriented 3D bounding box in that frame’s camera coordinates. The predicted box is transformed into the world coordinate system before evaluation. Since the original ground-truth boxes are axis-aligned, we append zero rotation angles to obtain oriented boxes for IoU computation. We then report [email protected] and [email protected]. For the viewpoint trend in Figure 1(b), we retain the corresponding raw IoU value before thresholding to obtain a continuous per-frame measure. For proposal refinement, we compare the predicted box with Mask3D proposals from LEO [23] and select the proposal with the highest IoU.

3D Video Object Detection. For each predicted box, we perform category-wise greedy matching against unused ground-truth boxes. A prediction is counted as a true positive if its best unmatched ground-truth box of the same category exceeds the IoU threshold of 0.25. Matched ground-truth boxes are removed from later matching. Precision, recall, and F1 are then computed for each category and averaged over the 20 common classes.

Spatial Reasoning. For VSI-Bench [38], we use the default LMMs-Eval configuration. Each video is uniformly sampled into 32 frames. Numerical-answer tasks are evaluated with Mean Relative Accuracy, while multiple-choice tasks are evaluated with standard accuracy.

Additional Experimental Results

Controlled Viewpoint Responses

Figure 6 extends the controlled analysis in Figure 1a with four additional scene target query examples. Across the wooden shelf, white sink, tan bed, and black chair cases, responses without routing vary substantially across viewpoints and often become diffuse or shift away from the target. Lie routing consistently preserves more localized target responses across the near, medium, and far views, supporting the same cross view behavior observed in the main teaser.

Controlled visualization of viewpoint-robust representations across five scene-query-target pairs, showing reference, near, medium, and far views with and without Lie routing

Figure 6. Additional controlled viewpoint response examples extending Figure 1(a). query-relevant activations are shown across reference, near, medium, and far views without and with Lie routing. Warmer regions indicate stronger support for the target phrase. Red wireframes denote projected ground truth boxes used only for visualization and [email protected].

SceneTargetRoutingNearMediumFar
scene0011_00Long tableWithout Routing44.62.749.5
scene0011_00Long tableLie Routing81.649.462.8
scene0064_00Wooden shelfWithout Routing82.664.734.0
scene0064_00Wooden shelfLie Routing86.274.086.7
scene0246_00White sinkWithout Routing73.540.615.6
scene0246_00White sinkLie Routing95.171.580.7
scene0645_00Tan bedWithout Routing48.811.530.6
scene0645_00Tan bedLie Routing83.228.270.6
scene0700_00Black chairWithout Routing48.331.98.9
scene0700_00Black chairLie Routing76.877.853.8
MeanWithout Routing59.630.327.7
MeanLie Routing84.660.270.9

Table 11. Target overlap of query-relevant activations under controlled viewpoint variation. [email protected] reports the percentage of top response locations inside the projected target box.

Activation Localization. Following the controlled setting in Figure 1(a), each example fixes the scene, target, query, checkpoint, and reference view, chosen as the first frame containing the target, and varies only the viewpoint along the same camera trajectory. Near, medium, and far views are selected by increasing reference relative rotation distance drot(r,v)d_{\mathrm{rot}}(r,v). Let Hv(p)H_v(p) denote the final decoder feature at visual token pp, and let QtQ_t contain the target phrase tokens. The query-relevant activation is

Av(p)=∣Qt∣−1∑c∈Qt[WcTHv(p)]+,(47)A_v(p)=\lvert Q_t\rvert^{-1}\sum_{c\in Q_t}\left[W_c^{\mathsf{T}}H_v(p)\right]_+, \tag*{(47)}

where WcW_c is the vocabulary projection for token cc, and [x]+=max⁡(x,0)[x]_+=\max(x,0). Let CvC_v denote the top 20 percent activation locations and let BvB_v denote the projected target region. We measure target localization using

PBO@0.2=100∣Cv∩Bv∣∣Cv∣.(48)\mathrm{[email protected]}=100\frac{\lvert C_v\cap B_v\rvert}{\lvert C_v\rvert}. \tag*{(48)}

PBO quantifies target localization within each view rather than monotonic degradation with viewpoint distance. The projected target boxes are used only for visualization and evaluation, and are not provided to the model. Lie routing improves [email protected] in all 15 evaluated views and increases the overall mean from 39.2 to 71.9. The improvements remain consistent across the near, medium, and far viewpoint groups.

3D Video Object Detection on EmbodiedScan

Category-Level Results. Table 12 and Table 13 provide category-level F1 scores for 3D video object detection. LieVLM improves over Qwen2.5-VL on nearly all categories, confirming the benefit of geometry-aware modeling for cross-frame 3D localization. Compared with VG LLM, LieVLM also shows clear gains on many spatially prominent categories, including table, couch, bed, bathtub, toilet, refrigerator, and computer in the 4-frame setting, as well as chair, table, desk, bed, bathtub, toilet, stove, and computer in the 6-frame setting. These improvements suggest that Lie-Group Routing helps preserve object orientation and spatial consistency when detections are aggregated into the first-frame coordinate system.

Table 12: The detailed results on 3D video object detection in 4-frame setting. We report the F1 score for all categories.

ObjectQwen2.5-VLVG LLMLieVLM (Ours)
chair41.254.053.5
pillow13.222.925.7
cabinet11.617.121.3
table36.546.556.8
lamp9.018.923.3
couch41.147.055.1
desk34.541.045.4
stand33.746.150.2
bed68.274.182.9
backpack33.341.449.5
bathtub36.642.175.9
ottoman0.00.00.0
dresser38.249.247.5
bin30.239.843.0
toilet68.782.586.2
refrigerator38.553.660.0
stove61.268.668.5
microwave21.333.324.1
monitor21.334.029.7
computer12.011.416.5

Table 12.

Table 13: The detailed results on 3D video object detection in 6-frame setting. We report the F1 score for all categories.

ObjectQwen2.5-VLVG LLMLieVLM (Ours)
chair36.148.750.0
pillow12.418.922.8
cabinet10.617.919.6
table32.744.854.2
lamp8.015.918.7
couch40.746.451.7
desk34.432.345.7
stand31.347.343.1
bed64.675.883.7
backpack28.640.446.3
bathtub38.440.479.9
ottoman2.75.30.1
dresser36.444.546.6
bin25.038.535.5
toilet68.683.286.5
refrigerator33.262.059.6
stove55.969.577.5
microwave7.825.819.6
monitor20.331.725.6
computer5.211.615.3

Table 13.

We also observe several remaining challenges. For categories with limited visual evidence or ambiguous geometry, such as ottoman, microwave, and monitor, LieVLM does not always improve over VG LLM. In particular, ottoman remains difficult in both frame settings, likely due to sparse instances and weak appearance cues. These results indicate that operatorized geometry substantially improves common object localization, while rare or small objects still require stronger supervision or more reliable visual evidence.

Multi-Seed Reliability. We further evaluate the stability of the matched VG LLM and LieVLM comparison over three random seeds. Table 14 provides detailed three-seed detection statistics consistent with the F1 results in Table 7. LieVLM improves all six aggregate precision, recall, and F1 metrics in every matched run, with 4-frame and 6-frame F1 gains of 4.58 ±\pm 0.11 and 4.05 ±\pm 0.13, respectively.

SettingMetricVG LLMLieVLMPaired GainPositive Seeds
4-Frame[email protected]43.19 ±\pm 0.2548.26 ±\pm 0.14+5.07 ±\pm 0.113/3
[email protected]39.44 ±\pm 0.2343.78 ±\pm 0.12+4.34 ±\pm 0.113/3
[email protected]41.18 ±\pm 0.2445.76 ±\pm 0.13+4.58 ±\pm 0.113/3
6-Frame[email protected]43.35 ±\pm 0.1447.58 ±\pm 0.28+4.23 ±\pm 0.143/3
[email protected]38.57 ±\pm 0.1241.60 ±\pm 0.24+3.03 ±\pm 0.123/3
[email protected]40.05 ±\pm 0.1344.10 ±\pm 0.26+4.05 ±\pm 0.133/3

Table 14. Multi-seed reliability on EmbodiedScan. Results are mean ±\pm std over three matched random seeds.

Detailed VSI-Bench Results

Full Category Comparison. Table 15 complements the compact scaling comparison in Table 6 with all eight VSI-Bench categories and a broader set of prior MLLMs and spatial models. The Qwen3 matched comparison improves seven of eight categories, while the original Qwen2.5 setting provides complementary three seed evidence. Category results are unavailable for the two external Qwen3 scale references, for which only the reported benchmark averages are shown.

MethodData SizeRel. Dist.Rel. Dir.RouteAppr.Overall Avg.
Prior MLLMs and Spatial Models
VG LLM-4B297K44.645.633.536.447.3
CA-MLLM-4B3.05M50.745.531.420.246.8
Qwen3-VL-8BN/A----57.9
SenseNova-SI Qwen3-VL-8B8M----64.8
Controlled Lie Routing Comparisons
VG LLM-8B297K46.640.732.459.250.6
LieVLM-8B (Ours)297K49.1 +2.542.9 +2.233.5 +1.159.8 +0.652.4 +1.7
VG LLM-Qwen33.05M57.852.245.069.960.6
LieVLM-Qwen3 (Ours)3.05M60.6 +2.855.5 +3.348.8 +3.873.7 +3.862.8 +2.2

Table 6. VSI-Bench comparison across backbone and scales with multiple choice categories.

MethodAvg.Obj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. Order
Prior MLLMs and Spatial Models
GPT-4o34.046.25.343.838.237.041.331.528.5
Gemini-1.5-Pro45.456.230.964.143.651.346.336.034.6
Gemini-2.5-Pro52.748.235.671.351.758.942.446.866.7
InternVL2-8B34.623.128.748.239.836.730.729.939.6
VILA-1.5-40B31.222.424.848.722.740.525.731.532.9
LongVA-7B29.238.016.638.922.233.143.325.415.7
LLaVA-NeXT-Video-72B40.948.922.857.435.342.436.735.048.6
SAT-LLaVA-Video-7B----47.341.137.136.140.4
RynnEC-7B45.858.525.454.942.744.2-38.730.5
CA-MLLM-4B46.871.339.350.465.950.745.531.420.2
Qwen3-VL-8B57.9--------
SenseNova-SI64.8--------
Controlled Qwen2.5 Setting with 297K Training
VG LLM-8B50.667.937.758.662.046.640.732.459.2
LieVLM-8B (Ours)52.4 +1.769.6 +1.740.5 +2.858.7 +0.164.7 +2.749.1 +2.542.9 +2.233.5 +1.159.8 +0.6
Controlled Qwen3 Setting with 3.05M Training
VG LLM-Qwen360.674.450.964.370.357.852.245.069.9
LieVLM-Qwen3 (Ours)62.8 +2.275.4 +1.052.7 +1.864.2 -0.171.5 +1.260.6 +2.855.5 +3.348.8 +3.873.7 +3.8

Table 15. Detailed VSI-Bench comparison across model and scaling settings. Green values denote changes over the corresponding controlled VG LLM baseline.

Multi Seed Reliability. The original Qwen2.5 setting is additionally evaluated over three matched random seeds. LieVLM improves all eight categories in every run, increasing the average from 50.62±0.1850.62 \pm0.18 to 52.35±0.1152.35 \pm0.11. Table 16 reports the complete uncertainty, paired gains, and positive seed counts.

Table 16: Multi seed reliability on VSI-Bench. Results are mean ±\pm standard deviation over three matched seeds, with paired gains computed seed wise.

MetricVG LLM-8BLieVLM-8BPaired GainPositive Seeds
Obj. Count67.88±\pm0.2669.58±\pm0.14+1.70±\pm0.123/3
Abs. Dist.37.68±\pm0.2440.50±\pm0.11+2.82±\pm0.133/3
Obj. Size58.60±\pm0.0658.70±\pm0.09+0.10±\pm0.033/3
Room Size61.98±\pm0.2164.69±\pm0.13+2.71±\pm0.093/3
Rel. Dist.46.57±\pm0.2249.11±\pm0.10+2.54±\pm0.123/3
Rel. Dir.40.69±\pm0.2242.90±\pm0.15+2.20±\pm0.083/3
Route Plan32.39±\pm0.2033.50±\pm0.08+1.11±\pm0.123/3
Appr. Order59.20±\pm0.0759.80±\pm0.12+0.60±\pm0.053/3
Average50.62±\pm0.1852.35±\pm0.11+1.72±\pm0.073/3

Table 16.

Robustness to Sparse Views and Geometry Perturbations

Sparse Input Views. We further evaluate whether the advantage of Lie routing persists when fewer views are available. Frames are uniformly subsampled from the same temporal interval while preserving the scene, query, temporal span, and endpoint views, and VG LLM and LieVLM receive identical frame subsets at every density. As shown in Table 17, LieVLM remains stronger across all evaluated input densities. The performance gap is preserved as the observations become sparser, with larger gains under the lowest density on Scan2Cap and EmbodiedScan.

Table 17: Performance under decreasing input view density. Relative density is measured with respect to the task specific full input setting. Both models receive identical frame subsets sampled from the same temporal interval. The full-density results are from the runs used for the sparse-view study, whereas Table 7 reports three-seed averages.

TaskFramesRelative DensityVG LLMLieVLM
Scan2Cap [email protected]32100%80.0682.70
Scan2Cap [email protected]1650%74.3177.08
Scan2Cap [email protected]825%66.4270.71
ScanRefer [email protected]32100%50.8757.21
ScanRefer [email protected]1650%48.3454.85
ScanRefer [email protected]825%42.6649.36
EmbodiedScan 6F [email protected]6100%40.7844.41
EmbodiedScan 6F [email protected]466.7%38.0542.41
EmbodiedScan 6F [email protected]350%34.4739.32

Table 17.

Geometry Perturbations. We separately measure the sensitivity of LieVLM to errors in the predicted geometry by perturbing F3dF^{3d} while keeping the visual tokens F2dF^{2d} fixed. Additive feature noise with σ=0.10\sigma= 0.10 and token dropout with p=30%p = 30\% are applied to the trained checkpoints without corruption specific retraining, and each stochastic setting is averaged over three independent realizations. As shown in Table 18, LieVLM retains at least 95.5% of clean performance under both perturbations, indicating tolerance to moderate errors in the geometry representation.

Table 18: Inference time robustness to geometry perturbations. Values in brackets report performance retained relative to the clean setting. The clean results are from the runs used for the perturbation study, whereas the multi-seed detection results are reported in Table 14.

ConditionSeverityScanRefer [email protected]EmbodiedScan 4F [email protected]EmbodiedScan 6F [email protected]
Clean–57.20 [100%]45.90 [100%]44.40 [100%]
Geometry Feature Noiseσ=0.10\sigma= 0.1055.87 [97.7%]44.93 [97.9%]43.31 [97.5%]
Geometry Token Dropoutp=30%p = 30\%54.76 [95.7%]43.84 [95.5%]42.69 [96.1%]

Table 18.

More Visualization

We provide additional qualitative results for 3D visual grounding in Figure 7. The examples cover diverse object categories and spatial expressions, including cabinets, armchairs, pictures, trash cans, chairs, doors, carts, windows, and tables. Across these cases, LieVLM is able to associate the language query with the correct temporal evidence and localize the referred object in 3D space. The projected boxes generally align well with the target regions across different views, showing that the model can handle object attributes, relative positions, and view-dependent appearances. We also observe that challenging cases mainly arise from depth ambiguity, partial visibility, and thin or distant objects, where the 2D projection can still look plausible while the 3D box orientation or depth is less accurate.

Additional visualization results for 3D visual grounding

Figure 7. Additional visualization results for 3D visual grounding.

Figure 8 presents additional results for 3D video object detection. Compared with VG LLM, LieVLM produces more stable 3D boxes across frames and better preserves the spatial orientation of objects.

Additional visualization results for 3D video object detection

Figure 8. Additional visualization results for 3D video object detection.

In several examples, VG LLM detects the major objects but predicts boxes with incorrect poses or misses objects under viewpoint changes. In contrast, LieVLM recovers more complete object sets and yields boxes that are better aligned with the visible geometry. These qualitative comparisons further support the role of Lie-Group Routing in reducing view-induced semantic drift and improving cross-frame 3D localization.

Broader Impact

LieVLM aims to improve the view robustness of 3D multimodal reasoning from RGB videos. This capability can benefit applications that require reliable spatial understanding across changing viewpoints, such as embodied navigation, assistive robotics, indoor mapping, and human-robot interaction. By reducing the dependence on explicit 3D sensors or reconstructed scene inputs, the framework may also make 3D scene understanding more accessible in resource-constrained settings. At the same time, LieVLM inherits several risks from its underlying MLLM backbone and geometry encoder. The model may still hallucinate objects or spatial relations, especially under poor visibility, severe occlusion, motion blur, or inaccurate geometry estimation. Since its predictions can influence downstream spatial decisions, the model should not be used as the sole source of truth in safety-critical systems without verification, calibration, and human oversight. The base language and vision models may also carry dataset biases, and improved 3D reasoning could be misused for intrusive surveillance or unauthorized spatial analysis. We therefore encourage responsible deployment, careful evaluation across environments and populations, and transparency about failure cases when releasing 3D MLLM systems.

Limitations

LieVLM uses the same geometry representation for direct 3D context and latent operator prediction, so reconstruction or alignment errors can affect both paths. The routing gate and Padé normalization constrain the influence of moderate perturbations, but severe geometry degradation can still reduce the reliability of routed corrections. The current formulation uses local SO(3)\mathrm{SO}(3) actions within learned three dimensional channel groups. The rotation constraint applies to these local action matrices. The learned projections and routing gates do not impose equivariance or composition consistency with respect to physical camera transformations on the complete model. These actions provide rotation oriented structure but do not explicitly represent translation, non rigid motion, or long range interactions, and the latent groups should not be interpreted as physical coordinates. The routing heads and output projections add computation relative to additive fusion. Current evaluations focus mainly on static indoor scenes, leaving dynamic objects, outdoor environments, and long horizon trajectories for future study.

Usage of Large Language Models

In this work, Large Language Models (LLMs) are only used for polishing the writing. They are not involved in research ideation, experimental design, data analysis, or the formulation of conclusions. All substantive intellectual contributions are made by the authors. The authors have checked the content throughout the paper after polishing, and take full responsibility for the current form.

NeurIPS Paper Checklist

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction covers the theoretical and technical contribution, the developed method and the experimental contribution.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

  1. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: At the end of the conclusion section, a limitation discussion is provided.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

  1. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] Justification: Yes, the assumptions are given in the form of Lema in the Prelimaries section. A complete proof is given in the Supplementary material.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, the baseline model, technical details, hyper-parameter and configuration are detailed in the submission for reproducibility.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets this paper uses are publicly available, and the source code is promised to be public once published.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details are given at the beginning of the experimental section.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following prior works in this field, the evaluation protocols on the corresponding datasets do NOT require a report of the error bar.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The hardware, especially the GPU requirement, is listed in the supplementary materials.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: This paper focuses on a fundamental task of machine learning and conducts experiments on publicly available datasets.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impact of this work has been discussed at the end of the conclusion section. We do not envision negative societal impact could be brought by this work.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: This work focuses on a fundamental problem in machine learning and conducts experiments on standard datasets. We do not envision such risks.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, all the assets have been properly cited, with a license to use for academia and no commercial purpose.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: This paper does not release new assets.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: The core method development in this research of this paper does not involve LLMs as any important, original, or non-standard components.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., & al., . et . (2025). Qwen2.5-VL technical report. 2502.13923
  2. [2]Cai, Z., Wang, R., Gu, C., Pu, F., Xu, J., Wang, Y., Yin, W., Yang, Z., Wei, C., Zhou, T., & al., . et . (2026). Scaling spatial intelligence with multimodal foundation models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7879–7890.
  3. [3]Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024.arxiv.org/abs/2401.12168
  4. [4]Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. ScanRefer: 3D object localization in RGB-D scans using natural language. In European Conference on Computer Vision, pages 202–221. Springer, 2020.arxiv.org/abs/1912.08830
  5. [5]Haijier Chen, Bo Xu, Shoujian Zhang, Haoze Liu, Jiaxuan Lin, and Jingrong Wang. Vid-LLM: A compact video-based 3D multimodal LLM with reconstruction-reasoning synergy. arXiv preprint arXiv:2509.24385, 2025.arXiv
  6. [6]Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LL3DA: Visual interactive instruction tuning for omni-3D understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26428–26438, 2024.arxiv.org/abs/2311.18651
  7. [7]Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3D-LLM with referent tokens. arXiv preprint arXiv:2405.10370, 2024.arXiv
  8. [8]Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., & al., . et . (2024). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. 2412.05271
  9. [9]Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2Cap: Context-aware dense captioning in RGB-D scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3193–3203, 2021.arxiv.org/abs/2012.02206
  10. [10]Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector Neurons: A general framework for SO(3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12200–12209, 2021.arxiv.org/abs/2104.12229
  11. [11]Fan, Z., Zhang, J., Li, R., Zhang, J., Chen, R., Hu, H., Wang, K., Qu, H., Zhou, S., Wang, D., & al., . et . (2025). VLM-3R: Vision-language models augmented with instruction-aligned 3D reconstruction. 2505.20279
  12. [12]Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Chiou, W.-C., Ma, R., & Krishna, S. (2024). Blink: Multimodal large language models can see but not perceive. European Conference on Computer Vision, 148–166.
  13. [13]Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3)-Transformers: 3D roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970–1981, 2020.arxiv.org/abs/2006.10503
  14. [14]Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15326–15337. IEEE, 2023.DOI
  15. [15]Hall, B. C. (2013). Lie groups, Lie algebras, and representations. In Quantum Theory for Mathematicians (pp. 333–366).
  16. [16]Hartley, R., Zisserman, A., & al., . et . (2003). Multiple view geometry in computer vision, volume 2.
  17. [17]Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled Cayley transform. In International Conference on Machine Learning, pages 1969–1978. PMLR, 2018.arxiv.org/abs/1707.09520
  18. [18]Nicholas J Higham. Functions of matrices: theory and computation. SIAM, 2008.
  19. [19]Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3D-LLM: Injecting the 3D world into large language models. Advances in Neural Information Processing Systems, 36:20482–20494, 2023.arxiv.org/abs/2307.12981
  20. [20]Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, and Jiangmiao Pang. G²VLM: Geometry grounded vision language model with unified 3D reconstruction and spatial reasoning. arXiv preprint arXiv:2511.21688, 2025.arxiv.org/abs/2511.21688
  21. [21]Huang, H., Wang, Z., Huang, R., Liu, L., Cheng, X., Zhao, Y., Jin, T., & Zhao, Z. (2023). Chat-3D v2: Bridging 3D scene and large language models with object identifiers (Vol. 2, Number 3, p. 6).arXiv
  22. [22]Huang, H., Chen, Y., Wang, Z., Huang, R., Xu, R., Wang, T., Liu, L., Cheng, X., Zhao, Y., Pang, J., & al., . et . (2024). Chat-Scene: Bridging 3D scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37, 113991–114017.
  23. [23]Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3D world. arXiv preprint arXiv:2311.12871, 2023.arxiv.org/abs/2311.12871
  24. [24]Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han. 3DRS: Mllms need 3D-aware representation supervision for scene understanding. arXiv preprint arXiv:2506.01946, 2025.arxiv.org/abs/2506.01946
  25. [25]Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. LieTransformer: Equivariant self-attention for Lie groups. In International Conference on Machine Learning, pages 4533–4543. PMLR, 2021.arxiv.org/abs/2012.10885
  26. [26]Du Q Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35(2):155–164, 2009.DOI
  27. [27]Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., & al., . et . (2024). LLaVA-OneVision: Easy visual task transfer.arXiv
  28. [28]Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. GTA: A geometry-aware attention mechanism for multi-view transformers. In International Conference on Learning Representations, 2024.arxiv.org/abs/2310.10375
  29. [29]Murray, R. M., Li, Z., & Sastry, S. S. (2017). A mathematical introduction to robotic manipulation. https://doi.org/CRC Press
  30. [30]Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.arxiv.org/abs/1709.07871
  31. [31]Ray, A., Duan, J., Brown, E., Tan, R., Bashkirova, D., Hendrix, R., Ehsani, K., Kembhavi, A., Plummer, B. A., Krishna, R., & al., . et . (2024). SAT: Dynamic spatial aptitude training for multimodal language models.arXiv
  32. [32]Garcia Satorras, V., Hoogeboom, E., & Welling, M. (2021). E(n) equivariant graph neural networks. International Conference on Machine Learning, 9323–9332.
  33. [33]Tong, S., Brown, E., Wu, P., Woo, S., Middeppogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., & al., . et . (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. Advances in Neural Information Processing Systems, 37, 87310–87356.
  34. [34]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5294–5306, 2025.arxiv.org/abs/2503.11651
  35. [35]Wang, S., Leroy, V., Cabon, Y., Childlovskii, B., & Revaud, J. (2024). DUST3R: Geometric 3D vision made easy. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20697–20709.
  36. [36]Wang, T., Mao, X., Zhu, C., Xu, R., Lyu, R., Li, P., Chen, X., Zhang, W., Chen, K., Xue, T., & al., . et . (2024). EmbodiedScan: A holistic multi-modal 3D perception suite towards embodied AI. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19757–19767.
  37. [37]Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. PointLLM: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pages 131–147. Springer, 2024.arxiv.org/abs/2308.16911
  38. [38]Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025). Thinking in space: How multimodal large language models see, remember, and recall spaces. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10632–10643.
  39. [39]Gongjie Zhang, Wenhao Li, Quanhao Qian, Jiuniu Wang, Deli Zhao, Shijian Lu, and Ran Xu. On the generalization capacities of MLLMs for spatial intelligence. arXiv preprint arXiv:2603.06704, 2026.arxiv.org/abs/2603.06704
  40. [40]Zhang, J., Chen, Y., Xu, Y., Huang, Z., Mei, J., Chen, C., Zhou, Y., Yuan, Y.-J., Cai, X., Huang, G., & al., . et . (2025). From flatland to space: Teaching vision-language models to perceive and reason in 3D. Advances in Neural Information Processing Systems, 38, 171909–171941.
  41. [41]Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, and Zizhuang Wei. SpaceMind: Camera-guided modality fusion for spatial reasoning in vision-language models. arXiv preprint arXiv:2511.23075, 2025.arxiv.org/abs/2511.23075
  42. [42]Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. Learning from videos for 3D world: Enhancing MLLMs with 3D vision geometry priors. arXiv preprint arXiv:2505.24625, 2025.arxiv.org/abs/2505.24625
  43. [43]Duo Zheng, Shijia Huang, and Liwei Wang. Video-3D LLM: Learning position-aware video representation for 3D scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8995–9006, 2025.arxiv.org/abs/2412.00493
  44. [44]Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. LLaVA-3D: A simple yet effective pathway to empowering lmms with 3D-awareness. arXiv preprint arXiv:2409.18125, 2024.arxiv.org/abs/2409.18125

Paper details

vlnc:W_9g263wrr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Large Language Models (LLMs)

View statement
Looking for related papers…
Contents