betaFeedback

WorksComputer Science

PhysMoE: Fast Spatial Grounding with Single-Token Coordinate Representation in Modality-Decoupled VLMs

Abstract

Embodied Vision-Language Models (VLMs) are backbones for robotic planning and manipulation. However, for coordinate prediction in grounding tasks, existing architectures suffer from two distinct bottlenecks: discrete tokenization disrupts spatial continuity with high decoding latency, while standard point regression fails on multi-contact affordances and irregularly shaped targets. To resolve these challenges, we propose PhysMoE, introducing three core contributions: (1) sharing lower layers for multimodal context while routing upper layers to specialized spatial (pixel) and linguistic (text) experts by physical modality, eliminating gradient interference; (2) encoding each coordinate as a single token by reusing the visual encoder's learned positional embeddings, injecting 2D spatial priors to natively align coordinates with visual features and ease training; and (3) differentiable Signed Distance Field (SDF) supervision on pointing, accommodating affordances and irregular geometries without costly mask decoders. Compared to the base model, PhysMoE achieves a 15.7% average gain and an 8.31x throughput speedup across 9 coordinate-based grounding benchmarks (trajectory, affordance, and localization), while preserving multimodal understanding and reasoning across 11 text-based embodied benchmarks.

Figure 1. Overview of the PhysMoE framework. (a) Comparison between conventional coupled architectures and our decoupled approach. (b) The proposed PhysMoE framework featuring continuous pixel position encoding, Semi-MoE token routing with decoupled linguistic (FFNlang_{\mathrm{lang}}) and spatial (FFNpix_{\mathrm{pix}}) experts, and Mask-SDF supervision.

Introduction

Embodied Vision-Language Models (VLMs) have emerged as powerful backbones for robotic planning and control [6, 7, 15, 65]. An effective embodied agent must not only understand language instructions and reason over visual scenes, but also ground decisions into fine-grained 2D coordinates—such as identifying grasp contact points, object placement regions, and dynamic manipulation trajectories [30, 59].

Despite rapid advances, existing autoregressive VLMs struggle to balance continuous spatial manipulation with high-level multimodal reasoning due to two fundamental bottlenecks:

1. Fragmented Spatial Tokenization and Modality Interference in Dense Decoders. Mainstream architectures represent coordinates either as character-level text strings [6, 8] or discrete spatial bin tokens [38]. Under standard VLM sub-word tokenizers, a single coordinate pair such as "[342, 518]" is partitioned into a sequence of isolated character and delimiter tokens (e.g., [, 3, 4, 2, ,, 5, 1, 8, ]). This representation breaks the holistic spatial integrity of pixel positions and imposes an unnecessary representational burden on semantic understanding. Moreover, generating coordinates autoregressively requires 8 to 10 sequential decoding passes per point, severely inflating inference latency and introducing quantization errors. When these disparate tasks are processed through standard dense decoders where all tokens share identical Feed-Forward Networks (FFNs), high-magnitude spatial regression gradients interfere with pre-trained linguistic weights, creating modality conflicts between categorical text generation and geometric modeling.

2. Multi-Peak Affordance Ambiguity. Physical affordances rarely correspond to an isolated, unique point. For tasks such as grasping a utensil handle or placing a bowl onto a table, multiple contact locations across the object part are equally functional, forming a distinct multi-peak distribution. Conventional point grounding enforces single-point regression (e.g., ℓ1\ell_1 or ℓ2\ell_2 distance to an arbitrary center). This formulation is fundamentally mismatched with multi-peak targets: it penalizes the model for predicting any valid contact mode away from the center and often forces the network to regress to a blurred average of multiple modes that lands in invalid physical space. Conversely, full mask prediction models like Qwen3-VL-Seg [57] introduce substantial compute and memory overhead.

To resolve these challenges, we propose PhysMoE, an embodied vision-language model featuring three synergistic innovations: Firstly, we represent a 2D coordinate as a unified token ⟨x,y⟩\langle x,y\rangle and derive their representations by reusing the pre-trained learned positional embeddings of the visual encoder. This preserves holistic 2D spatial integrity and simplify model training. Secondly, we keep the lower decoder layers fully shared, and in the top KK decoder layers we preserve shared attention across all tokens for unified scene context while routing tokens between a linguistic expert (FFNlang_{\mathrm{lang}}) and a specialized spatial expert (FFNpix_{\mathrm{pix}}) paired with decoupled output heads (a continuous coordinate regression head and a discrete LM head for language), resolving modality interference. Thirdly, we formulate a differentiable Euclidean Signed Distance Field (ESDF) loss to accommodate multi-peak affordances without auxiliary mask decoders. By sampling continuous coordinates directly from target ESDF maps built from binary masks, our asymmetric loss rewards points falling anywhere inside valid affordance regions while providing steep restorative gradients for exterior predictions.

Our main contributions are summarized as follows:

  1. We encode each 2D coordinate as a single token by reusing the visual encoder’s learned positional embeddings, aligning coordinates with visual features to ease training.

  1. We propose a PhysMoE architecture that shares lower layers while routing the top KK layers to modality-specific linguistic and spatial experts, eliminating gradient interference.

  1. We introduce differentiable ESDF supervision on pointing, accommodating multi-contact affordances and irregular targets without mask decoders.

  1. Across 20 benchmarks, PhysMoE achieves a 15.7% average gain and an 8.31× speedup on 9 coordinate-based grounding tasks while preserving reasoning across 11 text-based embodied benchmarks.

Related Work

Embodied Vision-Language Models. Multimodal foundation models [2, 6, 39, 48] have inspired a surge of embodied VLMs for robotic planning and control [7, 15, 65]. Backbones such as RoboBrain [46], RynnBrain [30], HY-Embodied-VLM [23], and Embodied-R1.5 [61] advance spatial scene understanding and physical reasoning. However, these systems typically handle spatial coordinates either through discrete text generation or separate downstream policies. In contrast, PhysMoE unifies continuous physical manipulation and conversational reasoning within an autoregressive architecture via an expert-decoupled design.

Coordinate Encodings in Multimodal Backbones. Spatial coordinates are conventionally tokenized as character-level text strings (e.g., partitioned into individual tokens [,3,4,…)[ , 3, 4, \ldots) [6, 8, 52] or discretized spatial bins [10, 38]. While compatible with standard vocabularies, discrete encodings suffer from artificial quantization errors, sequence bloat, and language degradation during fine-tuning. Recently, Molmo [13] demonstrated point pointing via direct coordinate prediction. Expanding on this direction, our ablation studies reveal that reusing the pre-trained learned 2D positional embeddings of the visual encoder via bilinear interpolation yields superior spatial accuracy and faster optimization than training coordinate embeddings from scratch or using fixed sinusoidal Fourier encodings.

Modality Decoupling and Multi-Peak Affordance Learning. Mixture of Experts (MoE) [12, 18, 25, 44] is widely used to scale capacity. In multimodal contexts [31, 32], experts typically route across general data-driven clusters. In contrast, our PhysMoE enforces a structural division of labor: self-attention remains shared across all tokens to preserve unified context, while top-layer FFNs disjunctively isolate linguistic generation (FFNlang_{\text{lang}}) from continuous spatial regression (FFNpix_{\text{pix}}). Furthermore, physical affordance learning [14, 20, 35] is inherently a multi-peak problem, where an actionable surface contains multiple valid contact points. Standard single-point regression penalizes valid alternate modes or averages them into in-valid regions. Inspired by implicit signed distance fields in 3D geometry [37] and robotics [36, 40], we introduce 2D Truncated Euclidean Signed Distance Fields (ESDF) as a continuous, differentiable loss. This resolves the multi-peak ambiguity by softly supervising regional affordances without introducing heavy mask decoders (e.g., SAM [26]).

Methodology

In this section, we present the architecture and training methodology of PhysMoE. We first detail the Semi-MoE decoupled architecture with dynamic routing (§ 3.1), describe the continuous pixel position representation and bilinear positional encoding (§ 3.2), introduce the differentiable Mask-SDF map supervision for multi-peak targets (§ 3.3), and outline the two-stage training curriculum (§ 3.4).

Semi-MoE Decoupled Architecture

Standard dense transformers process all tokens through identical Feed-Forward Networks (FFNs), causing acute gradient interference between discrete categorical text classification and continuous coordinate regression. To resolve this conflict while retaining global visual-linguistic context, we introduce the Semi-MoE architecture (Figure 2).

Figure 2. PhysMoE Architecture, Region-Based Supervision, and Continuous Coordinate Embedding. (a) Input RGB image and prompt. (b) Sequence tokenization. (c) Semi-MoE layers with shared self-attention and dynamically routed language/spatial experts (FFNlang\mathrm{FFN}_{\mathrm{lang}}, FFNpix\mathrm{FFN}_{\mathrm{pix}}). (d) Decoupled prediction heads for language tokens and 2D coordinates. (e) Region-based Mask-SDF supervision for continuous point prediction, penalizing exterior predictions while accommodating multi-peak affordance regions without mode averaging. (f) Continuous coordinate embedding via bilinear interpolation over visual positional embeddings PvisP_{\mathrm{vis}} and linear adapter projection for subsequent decoding steps.

Layer Partitioning and Shared Token Mixer. Let the Transformer backbone comprise LL sequential layers (l=0,…,L−1l = 0, \ldots, L - 1). We partition the decoder into two functional stages:

  • Shared Context Layers (l=0,…,L−K−1l = 0, \ldots, L - K - 1): The initial L−KL - K layers operate as standard dense Transformer blocks where visual, textual, and spatial tokens attend to one another freely, building rich multimodal contextual representations.

  • Semi-MoE Routed Layers (l=L−K,…,L−1l = L - K, \ldots, L - 1): Across the top KK layers (e.g., K=8K = 8), we maintain the shared self-attention mixer across all tokens to preserve uninterrupted global receptive fields. However, the subsequent feed-forward computation is decoupled into specialized experts.

Dynamic Token Router. At layer L−KL - K, a lightweight router MLP R\mathcal{R} inspects the normalized hidden state ht∈Rdtext\mathbf{h}_t \in\mathbb{R}^{d_{\text{text}}} and predicts binary routing logits:

rt=W2GELU⁡(W1ht+b1)+b2∈R2,(1)\mathbf{r}_t = \mathbf{W}_2 \operatorname{GELU}(\mathbf{W}_1\mathbf{h}_t + \mathbf{b}_1) + \mathbf{b}_2 \in\mathbb{R}^2, \tag*{(1)}

where rt,0r_{t,0} and rt,1r_{t,1} denote the routing propensities for linguistic versus spatial coordinate generation, respectively. During training, routing is teacher-forced using the ground-truth modality ct∗∈{0,1}c_t^* \in\{0, 1\}:

Lrouter=CE⁡EOS(rt,ct∗;weos),(2)\mathcal{L}_{\mathrm{router}} = \operatorname{CE}_{\mathrm{EOS}}(\mathbf{r}_t, c_t^*; w_{\mathrm{eos}}), \tag*{(2)}

weighted by an End-of-Sequence factor weosw_{\mathrm{eos}}. During inference, hard discrete gating is executed: ct=arg⁡max⁡(rt)c_t = \arg\max(\mathbf{r}_t).

Modality-Specialized Experts. In the top KK routed layers, after shared self-attention, each token is processed by a modality-specific feed-forward network according to ctc_t: linguistic tokens (ct=0c_t = 0) pass through FFNlang_{\text{lang}}, whereas spatial tokens (ct=1c_t = 1) are routed to FFNpix_{\text{pix}}. This decoupling isolates continuous spatial regression gradients from textual semantic representations.

Decoupled Dual Prediction Heads. At the final layer LL, hidden states are mapped to predictions via two task-specific heads:

  • LM Head: A linear projection Wlm∈Rdtext×V\mathbf{W}_{\mathrm{lm}} \in\mathbb{R}^{d_{\text{text}} \times V} predicting logits over vocabulary VV, supervised via cross-entropy:

LLM=CE⁡EOS(Wlmht,yt+1∗;weos).(3)\mathcal{L}_{\mathrm{LM}} = \operatorname{CE}_{\mathrm{EOS}}(\mathbf{W}_{\mathrm{lm}}\mathbf{h}_t, y_{t+1}^*; w_{\mathrm{eos}}). \tag*{(3)}
  • Pixel Head: An MLP that directly regresses continuous normalized 2D coordinates:

p^t=σ(Wpix,2GELU⁡(Wpix,1ht+bpix,1)+bpix,2)∈[0,1]2,(4)\hat{\mathbf{p}}_t = \sigma\left(\mathbf{W}_{\mathrm{pix},2}\operatorname{GELU}(\mathbf{W}_{\mathrm{pix},1}\mathbf{h}_t+\mathbf{b}_{\mathrm{pix},1})+\mathbf{b}_{\mathrm{pix},2}\right) \in[0,1]^2, \tag*{(4)}

where σ(⋅)\sigma(\cdot) is the Sigmoid activation enforcing normalized coordinate bounds [0,1][0, 1].

Continuous Pixel Position Representation and Encoding

Representation Syntax. Within conversational dialogues and instructions, 2D coordinates are formatted as ⟨∣x,y∣⟩\langle|x,y| \rangle, where x,y∈[0,1]x, y \in[0,1] are floating-point values normalized by image width and height. Our custom tokenizer maps each coordinate to a single unified token, which preserves prompt conciseness in inputs and bypasses the multi-token autoregressive decoding traditionally required for coordinate serialization. Furthermore, treating coordinates as unified geometric units allows the tokenizer to flexibly support arbitrary pixel delimiter tokens (e.g., brackets, parentheses, or task-specific tags) without altering model weights.

Coordinate Encoding via Visual Positional Embeddings. To embed continuous coordinates (x,y)∈[0,1]2(x,y) \in[0,1]^2 into dense vectors that share the visual encoder’s spatial frame, we reuse the pre-trained learned 2D positional embedding grid Pvis∈RG×G×dvisP_{\mathrm{vis}} \in\mathbb{R}^{G \times G \times d_{\mathrm{vis}}} of the Vision Transformer (e.g., G=48G = 48). Rather than quantizing coordinates into discrete grid bins, we query PvisP_{\mathrm{vis}} via bilinear interpolation and project the resulting feature into the language hidden dimension Rdtext\mathbb{R}^{d_{\mathrm{text}}} with a linear layer (Figure 2f):

epix(x,y)=WembedBilinearInterpolate⁡(Pvis,(x,y))+bembed.(5)e_{\mathrm{pix}}(x,y) = W_{\mathrm{embed}} \operatorname{BilinearInterpolate}(P_{\mathrm{vis}},(x,y)) + b_{\mathrm{embed}}. \tag*{(5)}

As demonstrated in our ablation studies (§ 4.3), reusing the visual encoder’s learned positional embeddings provides an essential geometric inductive bias that natively aligns coordinates with visual token representations and eases training.

Differentiable ESDF Supervision for Point Prediction

Resolving Multi-Peak Ambiguity in Affordance Grounding. In real-world robotic manipulation, actionable affordances and placement regions span continuous 2D zones M⊂[0,1]2M \subset[0,1]^2, forming an inherently multi-peak distribution with multiple valid contact locations. Conventional point grounding enforces single-point regression (e.g., ℓ1\ell_1 or Smooth L1 distance to an arbitrary mask centroid). This penalizes the model for predicting any valid contact mode away from the center and often forces it to output an arithmetic centroid that lands outside valid physical space. To resolve this without costly mask decoders, we propose ESDF supervision (Figure 2e).

Euclidean Signed Distance Field Formulation. As illustrated in Figure 2e, for a target affordance region M⊂[0,1]2M \subset[0,1]^2 enclosed by boundary ∂M\partial M, the signed distance ϕM(p)\phi_M(p) of any 2D point p∈[0,1]2p \in[0,1]^2 measures its Euclidean distance to the nearest boundary:

ϕM(p)={−min⁡q∈∂M∥p−q∥2,if p∈M (interior),+min⁡q∈∂M∥p−q∥2,if p∉M (exterior).(6)\phi_M(p) = \begin{cases} -\min_{q \in\partial M} \lVert p-q \rVert_2, & \text{if } p \in M\ \text{(interior)},\\ +\min_{q \in\partial M} \lVert p-q \rVert_2, & \text{if } p \notin M\ \text{(exterior)}. \end{cases} \tag*{(6)}

where q∈∂Mq \in\partial M represents a point on the mask boundary ∂M\partial M. Thus, ϕM(p)<0\phi_M(p) < 0 inside the mask, ϕM(p)=0\phi_M(p) = 0 on the boundary, and ϕM(p)>0\phi_M(p) > 0 outside.

Differentiable Bilinear Grid Sampling. Given the predicted continuous coordinate p^=(x^,y^)∈[0,1]2\hat{\mathbf{p}} = (\hat{x},\hat{y}) \in[0,1]^2 emitted by the continuous coordinate head, we sample the scalar signed distance ϕ(p^)\phi(\hat{\mathbf{p}}) directly from the target ESDF map Φ∈RS×S\Phi\in\mathbb{R}^{S \times S} via bilinear grid sampling:

ϕ(p^)=grid_sample⁡(Φ,2p^−1;align_corners⁡=True⁡).(7)\phi(\hat{\mathbf{p}}) = \operatorname{grid\_sample}(\Phi, 2\hat{\mathbf{p}} - 1; \operatorname{align\_corners} = \operatorname{True}). \tag*{(7)}

Because grid_sample⁡\operatorname{grid\_sample} is differentiable with respect to sampling coordinates, gradients ∂ϕ∂p^\frac{\partial\phi}{\partial\hat{\mathbf{p}}} flow directly into p^\hat{\mathbf{p}}, enabling end-to-end backpropagation without discrete approximation.

Asymmetric Continuous Loss Objective. We formulate an asymmetric loss Lsdf\mathcal{L}_{\mathrm{sdf}} over the signed distance ϕ\phi:

Lsdf(ϕ)={0.2⋅exp⁡(25.0⋅ϕ),if ϕ≤0 (inside),0.2+5.0⋅ϕ+8.0⋅ϕ2,if ϕ>0 (outside).(8)\mathcal{L}_{\mathrm{sdf}}(\phi) = \begin{cases} 0.2 \cdot\exp(25.0 \cdot\phi), & \text{if } \phi\le0\ \text{(inside)},\\ 0.2 + 5.0 \cdot\phi+ 8.0 \cdot\phi^2, & \text{if } \phi> 0\ \text{(outside)}. \end{cases} \tag*{(8)}

This formulation ensures C1C^1 smoothness at the boundary ϕ=0\phi= 0. For exterior points (ϕ>0\phi> 0), the linear-quadratic penalty exerts an increasingly strong inward pull as distance grows. Crucially, assigning a small, exponentially decaying loss to interior points (ϕ≤0\phi\le0) gently pushes predictions near the boundary further inward for a safer margin and more robust localization, while rapidly vanishing with depth to accommodate arbitrary interior contact peaks without center-seeking bias.

Joint Training Objective. The total multi-task objective combines language modeling, routing supervision, and coordinate loss:

L=LLM+Lrouter+λcoordLcoord+λesdfLsdf,(9)\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \mathcal{L}_{\mathrm{router}} + \lambda_{\mathrm{coord}}\mathcal{L}_{\mathrm{coord}} + \lambda_{\mathrm{esdf}}\mathcal{L}_{\mathrm{sdf}}, \tag*{(9)}

where Lcoord\mathcal{L}_{\mathrm{coord}} denotes Smooth L1 loss applied to fixed coordinate targets (e.g., trajectories), Lsdf\mathcal{L}_{\mathrm{sdf}} supervises regional affordance masks, and λcoord\lambda_{\mathrm{coord}}, λesdf\lambda_{\mathrm{esdf}} balance the terms.

Two-Stage Training Curriculum

Model Initialization. We instantiate PhysMoE at two scales: our 4B model is initialized from Qwen3.5-4B, and our 9B model is initialized from RynnBrain 1.1-9B [30] (which itself is initialized from Qwen3.5-9B). Across the top K=8K = 8 semi-MoE layers, the spatial expert FFNpix\mathrm{FFN}_{\mathrm{pix}} is initialized by copying weights from the corresponding pre-trained language FFNtext\mathrm{FFN}_{\mathrm{text}}, whereas the remaining newly introduced spatial modules—including the router R\mathcal{R}, continuous coordinate regression head, and coordinate projection MLP (MLPpix_embed\mathrm{MLP}_{\mathrm{pix\_embed}})—are randomly initialized.

Stage 1: Expert Cold-Start and Routing Warmup. To prevent randomly initialized spatial heads and routers from destabilizing the pre-trained backbone, we first freeze the vision encoder, merger, and token embeddings. We apply LoRA to the bottom Transformer layers (layers 0–3) while fully training the spatial modules (the continuous coordinate head, MLPpix_embed\mathrm{MLP}_{\mathrm{pix\_embed}}, R\mathcal{R}, and FFNpix\mathrm{FFN}_{\mathrm{pix}}) to stabilize routing boundaries and align continuous coordinate embeddings.

Stage 2: Full Multi-Layer LoRA and Spatial Specialization. In Stage 2, we merge the Stage 1 checkpoint and expand LoRA adaptation across all language layers while continuing full fine-tuning of FFNpix\mathrm{FFN}_{\mathrm{pix}}, R\mathcal{R}, and the coordinate regression head over our full multimodal mixture, yielding our final PhysMoE model.

Experiments

In this section, we present an extensive empirical evaluation of PhysMoE. Following the evaluation protocol established in the project benchmark suite, we assess our models across 20 diverse benchmarks comprising 9 coordinate-based spatial grounding tasks and 11 text-based embodied understanding and reasoning tasks.

Experimental Setup

Training Data Preparation. As summarized in Table 1, we curate a massive and balanced multi-task training mixture of 12 million (12M) samples across two complementary domains: general multimodal cognition and embodied physical grounding. For spatial grounding datasets that initially provide only sparse point or bounding-box (bbx) annotations, we prompt SAM3 [26, 41] with the ground-truth annotations to automatically extract high-quality binary object and affordance masks. These binary masks are subsequently transformed into continuous ESDF maps to provide dense boundary-aware supervision under our Mask-SDF objective (Sec. 3.3).

Task ModalityTraining Data Sources
General Multimodal Cognition
General UnderstandingVisual7W [64], Visual Genome [27]
Spatial UnderstandingRefSpatial [62], RoboSpatial [45], RoboPoint [58]
General LocalizationEgoObjects [63], LVIS [21], PixMo-Points [13], V3Det [49]
Embodied Physical Grounding & Planning
Embodied UnderstandingFSD [59], Robo2VLM [9]
Grounding & AffordanceRAGNet [51], AffGrasp [11], HOI4D [33], RoboAfford [22], ShareRobot [24], GraspNet [17], VL-Grasp [34], GraspClutter6D [5]
Trajectory PredictionMolmoAct [28], A0 [53]
Reasoning & PlanningRoboVQA [43], ShareRobot [24]
Total Multi-Task Mixture: ∼12.0 Million Training Samples

Table 1. Comprehensive multi-task training mixture (∼12M samples) for PhysMoE. Datasets are structured across general multimodal cognition and embodied physical grounding. For composite suites (e.g., ShareRobot [24]), we incorporate their respective domain subsets. Point and bounding-box annotations in affordance and contact tasks are converted into binary masks via SAM3 [26, 41] for continuous Mask-SDF supervision.

Evaluation Benchmarks and Prompt Formatting. Our evaluation spans two complementary domains:

  1. Coordinate-Based Spatial Grounding Tasks (9 Benchmarks): Tasks that require the model to output 2D pixel coordinates or ordered coordinate sequences in image space. This suite covers affordance and interaction pointing on Part-Affordance-2K [60], RoboAfford [22], VABench-Point [59], and Where2Place [58]; referring and spatial localization on PhyBench [54], RefSpatial-Bench [62], and RoboRefit [34]; and trajectory prediction on ShareRobot-Trajectory [24] and VABench-Visual-Trace [59].

  1. Text-Based Embodied Understanding & Reasoning Tasks (11 Benchmarks): Tasks where the model responds in natural language, such as multiple-choice answers, numerical estimates, or textual action plans. These benchmarks encompass scene spatial perception and relational QA on BLINK [19], CV-Bench [47], EmbSpatial [16], and SAT [42]; **multi-view and video

spatio-temporal reasoning on MindCube [50], ViewSpatial [29], and VSI-Bench [56]; and physical dynamics and embodied task planning on COSMOS [4], ERQA [1], ERQA-PLUS [55], and RoboVQA [43].

Because PhysMoE directly outputs complete, continuous 2D coordinate units rather than character-by-character token strings, it does not require verbose prompt constraints specifying coordinate output formatting. Accordingly, we omit verbose format instructions on coordinate-based tasks and simply append Type: point/bbox/trajectory to specify the target output for our model.

Metrics. All scores are reported on a standard 0–100 scale (higher is better). Point grounding tasks are evaluated using micro-averaged F1=2PμRμ/(Pμ+Rμ)×100F_1 = 2P_\mu R_\mu/(P_\mu+R_\mu)\times100, where precision Pμ=∑iHi/∑iPiP_\mu= \sum_i H_i/\sum_i P_i measures the proportion of predicted points falling inside valid ground-truth masks, and recall Rμ=∑imin⁡(Mi,Gi)/∑iGiR_\mu= \sum_i \min(M_i,G_i)/\sum_i G_i measures the proportion of covered targets (PiP_i, GiG_i are prediction and target counts; HiH_i, MiM_i are hit and matched counts for sample ii) For trajectory benchmarks (ShareRobot-Trajectory and VABench-Visual-Trace), following standard evaluation protocols, trajectory accuracy is computed as 100×(1−mean⁡RMSE/1000)100 \times(1 - \operatorname{mean}\mathrm{RMSE}/1000). The best and second-best scores among parameter-comparable models are highlighted in dark blue ( ) and light blue ( ), respectively.

Baselines. We evaluate PhysMoE at both 4B and 9B scales against representative open-source and published embodied foundation models. At the 4B scale, we compare with the base backbone Qwen3.5 (4B), RynnBrain 1.0 (4B) [30], and RoboBrain 2.5 (4B) [46]. At the 8B–9B (and MoE) scale, our competitors include RoboBrain 2.5 (8B) [46], RynnBrain 1.1 (9B) [30], Cosmos3-Nano (8B) [3], Embodied-R1.5 (8B) [61], and the large-scale mixture-of-experts model HY-Embodied-VLM-1.0 (30A3B) [23]. Note that some results for Embodied-R1.5, Cosmos3-Nano, and HY-Embodied-VLM in Table 2 and Table 3 are adopted from HY-Embodied-VLM-1.0 [23], while all other models are evaluated locally under unified benchmark protocols.

BenchmarkQwen3.5 (4B)Rynn1.0 (4B)RoboBrain (4B)Ours (4B)RoboBrain (8B)Rynn1.1 (9B)Cosmos3 (8B)Embodied (8B)HY (30A3B)Ours (9B)
Part-Affordance17.915.534.857.225.142.932.282.663.782.2
PIOBench49.458.556.852.160.262.254.461.665.360.7
RefSpatial-Bench39.756.853.353.854.962.044.452.453.455.2
RoboAfford65.673.177.473.572.572.876.260.671.574.8
RoboRefit81.061.083.582.082.481.955.477.288.282.0
VABench-Point37.82.345.846.79.613.245.261.459.762.7
Where2Place58.770.939.371.969.072.771.073.065.072.6
ShareRobot-Traj77.581.786.885.985.280.965.569.276.787.1
VABench-Trace79.584.383.587.286.787.981.689.879.790.2
Average56.356.062.467.860.664.158.469.869.274.2

Table 2. Comprehensive Evaluation on Coordinate-Based Spatial Grounding Tasks. Scores are on a 0–100 scale (↑\uparrow). [blue] and [light blue] denote the best and second-best results among parameter-comparable models, respectively. Model abbreviations: Rynn1.0 (RynnBrain1.0), Rynn1.1 (RynnBrain1.1), RoboBrain (RoboBrain2.5), Cosmos3 (Cosmos3-Nano), Embodied (Embodied-R1.5), HY (HY-Embodied-VLM-1.0).

BenchmarkQwen3.5 (4B)Rynn1.0 (4B)RoboBrain (4B)Ours (4B)RoboBrain (8B)Rynn1.1 (9B)Cosmos3 (8B)Embodied (8B)HY (30A3B)Ours (9B)
BLINK78.882.278.075.580.985.082.477.887.380.6
CV-Bench86.289.287.478.287.687.788.086.889.788.7
EmbSpatial77.380.675.968.977.981.980.076.082.781.9
MindCube35.055.534.046.229.381.132.827.970.079.0
SAT69.365.364.071.366.076.754.060.778.078.0
VSI-Bench53.065.741.556.144.166.050.459.258.965.3
ViewSpatial39.754.541.646.239.155.252.043.753.352.1
COSMOS60.858.253.960.858.464.567.168.666.962.9
ERQA46.043.840.242.844.843.045.037.360.847.0
ERQA-PLUS82.877.381.880.681.083.683.680.782.280.8
RoboVQA45.461.938.962.943.760.950.260.841.263.6
Average61.366.757.962.759.371.462.361.870.170.9

Table 3. Evaluation on Text-Based Embodied Understanding & Reasoning Tasks. [blue] and [light blue] denote the best and second-best results among parameter-comparable models, respectively.

Main Quantitative Results and Analysis

Table 2 and Table 3 evaluate PhysMoE across 9 coordinate-based spatial grounding tasks and 11 text-based embodied understanding and reasoning tasks, validating two key properties of our framework:

1. Strong Gains on Coordinate-Based Spatial Grounding. As shown in Table 2, PhysMoE substantially improves over its base models at both scales—rising from 56.3 to 67.8 on 4B (over Qwen3.5-4B) and from 64.1 to 74.2 on 9B (+15.7% relative over RynnBrain 1.1)—while also out-performing specialized embodied VLMs including the 3× larger 30B HY-Embodied-VLM (69.2). The advantage of our coordinate encoding is most evident on trajectory prediction, the most pixel-context-rich task: each waypoint is predicted conditioned on previously generated ones, so the model must read its own coordinates back as context. Since each coordinate is a single token embedded in the visual encoder’s positional space, earlier waypoints stay aligned with image patches instead of being fragmented into digit tokens. Accordingly, PhysMoE-9B ranks first on both ShareRobot-Traj (87.1) and VABench-Trace (90.2).

2. PhysMoE Enhances Spatial Grounding While Preserving Text-Based Reasoning. Comparing Table 2 and Table 3 relative to each model’s initialization backbone demonstrates that our PhysMoE architecture prevents coordinate regression from interfering with pre-trained linguistic representations. By sharing lower layers for multimodal context while isolating spatial regression gradients within FFNpix_{\mathrm{pix}} across the top KK layers, PhysMoE achieves substantial gains in coordinate grounding (+11.5 points on 4B and +10.1 points on 9B) while maintaining text-based reasoning performance on par with its base models: PhysMoE-4B slightly improves over Qwen3.5-4B (62.7 vs. 61.3), and PhysMoE-9B closely matches RynnBrain 1.1 (70.9 vs. 71.4).

Ablation Studies

To validate our core design choices, we conduct ablation analyses on the 4B model variant (PhysMoE-4B), investigating continuous coordinate token encoding and maskbased supervision objectives.

1. Analyzing Coordinate Token Encoding via Learnable Gating. We initially considered three candidate schemes for encoding a coordinate token (x,y)∈[0,1]2(x, y) \in[0, 1]^2: reusing the visual encoder’s learned positional embeddings (epe\mathbf{e}_{\mathrm{pe}}), sinusoidal Fourier frequency encoding (ef\mathbf{e}_{\mathrm{f}}), and local visual patch features sampled at that coordinate (ei\mathbf{e}_{\mathrm{i}}). To allow the model to automatically learn which encoding scheme is most suitable, we combine the three candidates via a learnable gated fusion layer during training:

e~fused=wpeepe+wfef+wiei,\tilde{\mathbf{e}}_{\mathrm{fused}} = w_{\mathrm{pe}}\mathbf{e}_{\mathrm{pe}} + w_{\mathrm{f}}\mathbf{e}_{\mathrm{f}} + w_{\mathrm{i}}\mathbf{e}_{\mathrm{i}},
efused=∥epe∥2∥e~fused∥2e~fused,(10)\mathbf{e}_{\mathrm{fused}} = \frac{\lVert\mathbf{e}_{\mathrm{pe}}\rVert_2}{\lVert\tilde{\mathbf{e}}_{\mathrm{fused}}\rVert_2}\tilde{\mathbf{e}}_{\mathrm{fused}}, \tag*{(10)}

where epe\mathbf{e}_{\mathrm{pe}} projects the interpolated visual positional embedding Pvis\mathcal{P}_{\mathrm{vis}} (xrefEq. (5)); ef\mathbf{e}_{\mathrm{f}} maps 12-frequency sinusoidal Fourier features of (x,y)(x,y) (frequencies 2kπ2^k\pi for k=0,…,11k = 0,\ldots,11) through a two-layer MLP; and ei\mathbf{e}_{\mathrm{i}} applies an affine-free LayerNorm and linear projection to pre-merger visual patch features sampled at (x,y)(x,y). Crucially, we rescale e~fused\tilde{\mathbf{e}}_{\mathrm{fused}} to match the ℓ2\ell_2 norm of epe\mathbf{e}_{\mathrm{pe}} so that the fusion branches only adjust the embedding direction without altering the magnitude scale expected by the language model, making the gating weights directly reflect relative contributions.

Analyzing the learned gating weights and branch norm shares (∥wkek∥2/∑j∥wjej∥2\lVert w_k\mathbf{e}_k\rVert_2 / \sum_j \lVert w_j\mathbf{e}_j\rVert_2) at convergence reveals distinct behaviors:

  • Learned Visual PE Dominates (>97%> 97\%): Reusing the visual backbone’s learned positional embeddings accounts for over 97% of the representation magnitude, as Pvis\mathcal{P}_{\mathrm{vis}} inherently encodes 2D metric geometry aligned with visual tokens.

  • Image Features Provide Marginal Context (∼3%\sim3\%): Pre-merger visual patch features contribute only ∼3%;\sim3\%; while offering semantic appearance cues, patch-level features lack the fine spatial resolution required for exact coordinate localization.

  • Fourier Encoding Fails to Contribute (<1%< 1\%): The learned Fourier weight vanishes to less than 1% (wf≈0w_f \approx0), as generic sinusoidal encodings operate in an unaligned domain disjoint from the visual representation manifold.

These results confirm that directly reusing the visual backbone’s learned PE provides the most effective coordinate representation, rendering auxiliary Fourier embeddings and patch feature injections unnecessary.

2. Effectiveness of Differentiable ESDF Supervision on Pointing. Under identical training data and architecture, Table 4 compares single-point centroid regression (ℓ1\ell_1) against our differentiable ESDF supervision. By accommodating arbitrary interior contact points on multi-contact affordances and irregular geometries rather than forcing predictions toward a single centroid, ESDF supervision consistently improves RoboRefit (79.2% →\rightarrow 82.0%), Part-Affordance (55.5% →\rightarrow 57.2%), and RoboAfford (72.3% →\rightarrow 73.5%). Furthermore, although ESDF is not directly applied to trajectory sequences, ShareRobot-Traj also improves (85.5% →\rightarrow 85.9%) due to more accurate localization of trajectory start and target points.

Supervision ObjectivePart-Afford.RoboRefitShareRobot-TrajRoboAfford
(a) Without Mask (Centroid ℓ1\ell_1)55.579.285.572.3
(b) With Mask-SDF (Ours)57.282.085.973.5

Table 4. Ablation Study on Supervision Objectives (Evaluated on 4B Model). Comparison of single-point centroid regression versus our differentiable Mask-SDF supervision across affordance grounding, referring expressions, and trajectory prediction.

Inference Efficiency and Pixel Throughput

In physical robot manipulation, low latency and high inference throughput are essential for responsive, closed-loop execution. In Table 5, we benchmark the pixel inference speed across representative embodied foundation models on an NVIDIA RTX 5090 GPU under identical evaluation settings. To construct a representative evaluation suite, we sample 1,000 QA instances from the benchmarks where the primary target output consists of 2D pixel coordinates (spanning point grounding, bounding boxes, and motion trajectories).

ModelCoord. DecodingThroughput (samples/s) ↑\uparrow
RynnBrain1.0-4B [30]Autoregressive Tokens5.28
RoboBrain2.5-4B [46]Autoregressive Tokens3.41
RynnBrain1.1-9B [30]Autoregressive Tokens2.61
EmbodiedR1.5-8B [61]Autoregressive Tokens2.81
RoboBrain2.5-8B [46]Autoregressive Tokens1.58
Ours-4BDirect Regression31.88 (6.04×)
Ours-9BDirect Regression23.34 (8.31×)

Table 5. Pixel Inference Speed Comparison. Measured in samples/second (samples/s ↑\uparrow). Speedups in parentheses are computed conservatively relative to the fastest competing baseline at each model scale.

Here, the evaluation metric is defined as pixel inference speed measured in samples per second (samples/s), where a complete coordinate tuple alongside its formatting delimiters constitutes one sample. As shown in Table 5, existing autoregressive models serialize coordinates into multiple discrete tokens, requiring iterative autoregressive decoding that bottlenecks throughput to only 1.58–5.28 samples/s.

In contrast, PhysMoE represents each 2D coordinate as a single token and predicts it in one decoding step via the coordinate head, bypassing multi-token character serialization entirely. Notably, even when benchmarked against the fastest competing baseline in each parameter class, Ours-9B attains a pixel inference throughput of 23.34 samples/s (8.31× over EmbodiedR1.5-8B), while Ours-4B reaches 31.88 samples/s (6.04× over RynnBrain1.0-4B). This substantial acceleration provides critical responsiveness for high-frequency robotic control loops without sacrificing spatial precision.

Conclusion

In this paper, we introduced PhysMoE, an embodied vision-language model that harmonizes continuous physical manipulation with high-level multimodal reasoning. By decoupling top-layer FFNs via a Semi-MoE framework while preserving shared self-attention, PhysMoE eliminates gradient interference between categorical text generation and geometric regression. Furthermore, by reusing pre-trained visual positional embeddings for continuous coordinate encoding and formulating differentiable Mask-SDF supervision for multi-peak affordances, our model achieves sub-pixel precision without auxiliary mask decoders. Extensive evaluations across 20 benchmarks validate that PhysMoE achieves state-of-the-art trajectory accuracy and outstanding affordance grounding while preserving conversational reasoning fidelity.

References

  1. [1]Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini Robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025.arxiv.org/abs/2503.20020
  2. [2]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
  3. [3]N. Agarwal, A. Ali, J. Allen, M. Antolini, A. Aubame, A. Azzolini, J. Bai, M. Bala, Y. Balaji, J. Bapst, et al. Cosmos 3: Omnidomodal world models for physical ai. arXiv preprint arXiv:2606.02800, 2026.arxiv.org/abs/2606.02800
  4. [4]Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jiniu Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-Reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558, 2025.arxiv.org/abs/2503.15558
  5. [5]Seunghyeok Bae, Jooseon Lee, Kangmin Kim, Heeseon Rho, Geonhyup Lee, Raeyoung Kang, Sangbeom Lee, Sangjun Noh, Youngjin Lee, Taeyeop Lee, et al. GraspClutter6D: A large-scale real-world dataset for robust perception and grasping in cluttered scenes. IEEE Robot. Autom. Lett., 2025.arxiv.org/abs/2504.06866
  6. [6]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Rongyao Fang, Chang Gao, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  7. [7]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Davis, Chelsea Ding, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Ho, et al. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2212.06817
  8. [8]Keqin Chen, Zhao Zhao, Binyuan Peng, et al. Shikra: Unleashing multimodal llm’s referential dialogue magic. In arXiv preprint arXiv:2306.15195, 2023.arxiv.org/abs/2306.15195
  9. [9]Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R Sanketi, and Ken Goldberg. Robo2VLM: Visual question answering from large-scale in-the-wild robot manipulation datasets. In Adv. Neural Inform. Process. Syst., 2025.arxiv.org/abs/2505.15517
  10. [10]Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2Seq: A language modeling framework for object detection. In Int. Conf. Learn. Represent., 2022.arxiv.org/abs/2109.10852
  11. [11]Y. Chen et al. AffGrasp: Task-oriented affordance grounding for robotic grasping. In IEEE/RSJ Int. Conf. Intell. Robots Syst., 2024.
  12. [12]Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Wang, Xingkai Yu, Y. Wu, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1157–1172, 2024.arxiv.org/abs/2401.06066
  13. [13]Matt Deitke, Christopher Clark, Sangho Lee, Rohun Schmitz, Vasu Sehgal, et al. Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models. In IEEE Conf. Comput. Vis. Pattern Recog., 2025.arxiv.org/abs/2409.17146
  14. [14]Thanh-Toan Do, Anh Nguyen, and Ian Reid. AffordanceNet: An end-to-end deep learning approach for keypoint-based affordance detection. In IEEE Int. Conf. Robot. Autom., pages 5840–5847, 2018.DOI
  15. [15]Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. In ICML, 2023.arxiv.org/abs/2303.03378
  16. [16]Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proc. Annu. Meet. Assoc. Comput. Linguist., pages 346–355, 2024.arxiv.org/abs/2406.05756
  17. [17]Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. GraspNet-1Billion: A large-scale benchmark for general object grasping. In CVPR, pages 11444–11453, 2020.DOI
  18. [18]William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 23(120):1–39, 2022.arxiv.org/abs/2101.03961
  19. [19]Xingyu Fu, Yushi Hu, Bangzhe Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In Eur. Conf. Comput. Vis., pages 148–166, 2024.arxiv.org/abs/2404.12390
  20. [20]James J Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979.
  21. [21]Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5356–5364, 2019.arxiv.org/abs/1908.03195
  22. [22]Xiaoshuai Hao, Yingbo Tang, Lingfeng Zhang, Yanbiao Ma, Yunsheng Diao, Ziyu Jia, Wenbo Ding, Hangjun Ye, and Long Chen. RoboAfford++: A generative ai-enhanced dataset for multimodal affordance learning in robotic manipulation and navigation. arXiv preprint arXiv:2511.12436, 2025.arxiv.org/abs/2511.12436
  23. [23]Hunyuan Embodied Team. Hy-Embodied-VLM-1.0: Efficient physical-world agents, 2026.arxiv.org/abs/2607.12894
  24. [24]Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. RoBrain: A unified brain model for robotic manipulation from abstract to concrete. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1724–1734, 2025.arxiv.org/abs/2502.21257
  25. [25]Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.arxiv.org/abs/2401.04088
  26. [26]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Int. Conf. Comput. Vis., pages 4015–4026, 2023.arxiv.org/abs/2304.02643
  27. [27]Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32–73, 2017. 6
  28. [28]Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, et al. MolmoAct: Action reasoning models that can reason in space. In IEEE Int. Conf. Robot. Autom., 2026. 6arxiv.org/abs/2508.07917
  29. [29]Dingming Li, Hongxing Li, Zixuan Wang, Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, et al. ViewSpatial-Bench: Evaluating multi-perspective spatial localization in vision-language models. In Eur. Conf. Comput. Vis., 2026. 6arxiv.org/abs/2505.21500
  30. [30]Kehan Li, Bohan Hou, Minghao Zhu, Tianyi Zhang, Zesen Cheng, Zhikai Wang, Sicong Leng, Xin Li, Xiao Lin, Biying Yao, et al. RynnBrain 1.1: Towards more capable and generalizable embodied foundation model. arXiv preprint arXiv:2607.17977, 2026. 2, 5, 6, 8arxiv.org/abs/2607.17977
  31. [31]Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-MoE: Scaling unified multimodal large language models with mixture of experts. IEEE Trans. Pattern Anal. Mach. Intell., 47(5):3424–3439, 2025. 3DOI
  32. [32]Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Mुनan Ning, and Li Yuan. MoE-LLaVA: Mixture of experts for large vision-language models. IEEE Transactions on Multimedia, 2025. 3arxiv.org/abs/2401.15947
  33. [33]Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4d egocentric dataset for category-level human-object interaction. In IEEE Conf. Comput. Vis. Pattern Recog., pages 21013–21022, 2022. 6arxiv.org/abs/2203.01577
  34. [34]Yuhao Lu, Yuxin Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. VL-Grasp: A 6-dof interactive grasp policy for language-oriented objects in cluttered indoor scenes. In IEEE/RSJ Int. Conf. Intell. Robots Syst., pages 976–983, 2023. 5, 6arxiv.org/abs/2308.00640
  35. [35]Tushar Nagarajan, Christoph Li, and Kristen Grauman. Grounded human-to-robot affordance transfer from unconstrained videos. In Int. Conf. Comput. Vis., pages 4118–4127, 2019. 3
  36. [36]Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto. Voxblox: Incremental 3d euclidean and signed distance fields for on-board mav planning. IROS, pages 1366–1373, 2017. 3DOI
  37. [37]Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 165–174, 2019. 3arxiv.org/abs/1901.05103
  38. [38]Zhiliang Peng, Wenhuai Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. In Int. Conf. Learn. Represent., 2024. 2, 3arxiv.org/abs/2306.14824
  39. [39]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., pages 8748–8763, 2021.
  40. [40]Nathan Ratliff, Matt Zucker, J Andrew Bagnell, and Siddhartha Srinivasa. CHOMP: Covariant hamiltonian optimization for motion planning. IJRR, 32(9-10):1164–1193, 2013. 3DOI
  41. [41]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolnick, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. In Int. Conf. Learn. Represent., 2025. 5, 6arxiv.org/abs/2408.00714
  42. [42]Arijit Ray, Jiafei Duan, Ellis L Brown II, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A Plummer, Ranjay Krishna, et al. SAT: Dynamic spatial attitude training for multimodal language models. In Conf. Lang. Model., 2025. 5arxiv.org/abs/2412.07755
  43. [43]Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. RoboVQA: Multimodal long-horizon reasoning for robotics. In IEEE Int. Conf. Robot. Autom., pages 645–652, 2024. 6arxiv.org/abs/2311.00899
  44. [44]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR, 2017. 3arxiv.org/abs/1701.06538
  45. [45]Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. RoboSpatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. In IEEE Conf. Comput. Vis. Pattern Recog., pages 15768–15780, 2025. 6arxiv.org/abs/2411.16537
  46. [46]Huajie Tan, Enshen Zhou, Zhiyu Li, Yijie Xu, Yuheng Ji, Xiansheng Chen, Cheng Chi, Pengwei Wang, Huizhu Jia, Yulong Ao, et al. RoboBrain 2.5: Depth in sight, time in mind. arXiv preprint arXiv:2601.14352, 2026. 2, 6, 8arxiv.org/abs/2601.14352
  47. [47]Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrain-1: A fully open, vision-centric exploration of multimodal llms. In Adv. Neural Inform. Process. Syst., pages 87310–87356, 2024. 5arxiv.org/abs/2406.16860
  48. [48]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2arxiv.org/abs/2302.13971
  49. [49]Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3Det: Vast vocabulary visual detection dataset. In Int. Conf. Comput. Vis., pages 19844–19854, 2023. 6arxiv.org/abs/2304.03752
  50. [50]Qineng Wang, Baiqiao Yin, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. MindCube: Spatial mental modeling from limited views. In Int. Conf. Learn. Represent., 2026. 6arxiv.org/abs/2506.21458
  51. [51]Z. Wu et al. RAGNet: Robotic affordance grounding with vision-language models. In IEEE Int. Conf. Robot. Autom., 2025. 6
  52. [52]Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for diverse vision tasks. In IEEE Conf. Comput. Vis. Pattern Recog., pages 11624–11635, 2024. 3arxiv.org/abs/2311.06242
  53. [53]Rongtao Xu, Jian Zhang, Minghao Guo, Youpeng Wen, Haoting Yang, Min Lin, Jianzheng Huang, Zhe Li, Kaidong Zhang, Liqiong Wang, et al. A₀: An affordance-aware hierarchical model for general robotic manipulation. In ICCV, 2025. 6arxiv.org/abs/2504.12636
  54. [54]Haotian Xue, Yunhao Ge, Yu Zeng, Zhaoshuo Li, Ming-Yu Liu, Yongxin Chen, and Jiaojiao Fan. Point-It-Out: Benchmarking embodied reasoning for vision language models in multi-stage visual grounding. arXiv preprint arXiv:2509.25794, 2025. 5arxiv.org/abs/2509.25794
  55. [55]Hong Yang and Basura Fernando. ERQA-Plus: A diagnostic benchmark for reasoning in embodied ai. arXiv preprint arXiv:2606.17639, 2026. 6arxiv.org/abs/2606.17639
  56. [56]Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In IEEE Conf. Comput. Vis. Pattern Recog., pages 10632–10643, 2025. 6arxiv.org/abs/2412.14171
  57. [57]Yuan Yao, Qiushi Yang, Humen Zhong, Jiangning Wei, Yifang Men, Shuai Bai, Miaomiao Cui, and Zhibo Yang. Qwen3-vl-seg: Unlocking open-world referring segmentation with vision-language grounding. arXiv preprint arXiv:2605.07141, 2026. 2arxiv.org/abs/2605.07141
  58. [58]Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairan Murali, Arsalan Mousavian, and Dieter Fox. RoboPoint: A vision-language model for spatial affordance prediction for robotics. In Conf. Robot Learn., pages 3824–3843, 2024. 5, 6arxiv.org/abs/2406.10721
  59. [59]Yifu Yuan, Haiqin Cui, Yibin Chen, Zibin Dong, Fei Ni, Longxin Kou, Jinyi Liu, Pengyi Li, Yan Zheng, and Jianye Hao. From seeing to doing: Bridging reasoning and decision for robotic manipulation. In Int. Conf. Learn. Represent., 2026. 2, 5, 6arxiv.org/abs/2505.08548
  60. [60]Yifu Yuan, Haiqin Cui, Yaoting Huang, Yibin Chen, Fei Ni, Zibin Dong, Pengyi Li, Yan Zheng, Hongyao Tang, and Jianye Hao. Embodied-R1: Reinforced embodied reasoning for general robotic manipulation. In Int. Conf. Learn. Represent., 2026. 5arxiv.org/abs/2508.13998
  61. [61]Yifu Yuan, Yaoting Huang, Xianze Yao, Yutong Li, Shuo-heng Zhang, Linqi Han, Pengyi Li, Ji Jangeng Sun, Wenting Jia, Zhao Zhang, et al. Embodied-R1.5: Evolving physical intelligence via embodied foundation models. arXiv preprint arXiv:2606.11324, 2026. 2, 6, 8arxiv.org/abs/2606.11324
  62. [62]Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. RoboRefer: Towards spatial referring with reasoning in vision-language models for robotics. In Adv. Neural Inform. Process. Syst., pages 32209–32286, 2025. 5, 6arxiv.org/abs/2506.04308
  63. [63]Chenchen Zhu, Fanyi Xiao, Andres Alvarado, Mohammad Babaeizadeh, Zhicheng Yan, Rajesh Ranjan, Naveen Harikrishnan, Fei-Fei Li, Rakesh Yan, et al. EgoObjects: A large-scale egocentric dataset for fine-grained object understanding. In Int. Conf. Comput. Vis., pages 20110–20120, 2023. 6arxiv.org/abs/2309.08816
  64. [64]Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7W: Grounded question answering in images. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4995–5004, 2016. 6arxiv.org/abs/1511.03416
  65. [65]Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Welker, Ayzaan Wahid, Quan Vuong, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. CoRL, 2023. 2

Paper details

vlnc:W_nc8dd6ke
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents