betaFeedback

WorksComputer Science

Xiaomi-Lens-0 Technical Report

Abstract

Multimodal large language models (MLLMs) answer questions about whole images well, yet they still struggle with fine-grained region understanding. A key reason is that text-only supervision never specifies which regions support an answer. To close this gap, we present Xiaomi-Lens-0, a 4B MLLM built on Qwen3.5 that predicts masks from its own visual representations. Specifically, a lightweight decoder combines vision-encoder patch features with the language model's mask-token states, termed visual latents, so that mask supervision reaches the shared backbone without an additional segmentation encoder. Because each latent decodes to a mask, the model can interleave reasoning with region-specific latents and thus think with visual latents, while grounded reasoning traces supervise the intermediate regions referenced during chain-of-thought. As a result, across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in referring comprehension and achieves the highest segmentation cIoU among compared models. Moreover, it surpasses Qwen3.5 across all perception benchmarks. Homepage: https://seerray-lab.github.io/Xiaomi-Lens-0

Xiaoai Plus, SeerRay Team

See Contributors for the full author list.

Homepage: https://seerray-lab.github.io/Xiaomi-Lens-0/

Contents

1 Introduction

2 Method 2.1 Mask Prediction from Shared Visual Features 2.2 Model Architecture 2.2.1 Shared Visual Features and Visual Latents 2.2.2 Mask Decoding 2.2.3 Joint Language and Mask Supervision 2.3 Thinking with Visual Latents 2.4 Training Recipe 2.4.1 Stage 1: Mask Decoder Pretraining 2.4.2 Stage 2: Supervised Fine-Tuning

3 Data Pipeline 3.1 Referring-Expression Training Data 3.1.1 Visual Grounding 3.1.2 Instruction Synthesis 3.1.3 Consistency Filtering 3.2 Grounded Reasoning Data 3.2.1 Question–Answer Synthesis 3.2.2 Question–Answer Verification 3.2.3 Trace Generation and Mask Annotation 3.3 LensBench 3.3.1 Benchmark Design 3.3.2 Benchmark Construction 3.3.3 Consistency Filtering

4 Experiments 4.1 Evaluation Setup 4.2 Main Results 4.2.1 Referring Expression Comprehension 4.2.2 Referring Expression Segmentation 4.2.3 Multi-Instance Grounding 4.2.4 Reasoning Segmentation and Embodied Referring 4.2.5 Fine-Grained Perception 4.3 LensBench Results 4.4 Ablation Study

5 Related Work

6 Conclusion

Appendix

Introduction

Visual grounding, which links natural-language expressions to image regions, is a foundational capability of multimodal large language models (MLLMs) [1, 79], supporting embodied intelligence [28] and multimodal agent systems [6, 12]. Specifically, referring expression comprehension (REC) and referring expression segmentation (RES) evaluate grounding directly, requiring the model to output the referenced target as a bounding box or a pixel-level mask, respectively. Beyond these tasks, visual questions about object counting, attributes, and spatial relations also depend on grounding: answering them requires pinpointing the referenced regions and perceiving their visual details [54, 59].

However, current MLLMs still struggle with fine-grained region understanding, often miscounting nearby objects or misidentifying the attributes of small regions [23, 45]. We attribute this to text-only supervision, under which the image serves only as a conditioning input and the training signal comes entirely from the text answer. Such supervision does not explicitly specify which image regions support the answer or how far those regions extend. By contrast, segmentation masks provide explicit spatial targets by labeling which pixels belong to a referenced region [17], making them a direct source of supervision for region extent.

Although existing MLLMs for RES [39, 60, 75] already train with mask supervision, most obtain the spatial detail for mask prediction from an additional encoder: LISA [18] and Sa2VA [71] rely on a hierarchical segmentation encoder [17, 40], and Qwen3-VL-Seg [66] fuses multi-scale features from a lightweight supplementary image encoder. Others decode masks from the MLLM’s own visual features: PixelLM [41] decodes from multiple layers of its vision encoder, and MLLMSeg [50] fuses shallow vision-encoder features with the language model’s visual tokens. However, both keep the vision encoder frozen, so the mask loss does not reach the shared visual features. Moreover, across both lines, mask supervision is treated as a means to segmentation and evaluated mainly on RES and grounding, leaving unexplored whether it can also improve fine-grained perception and visual reasoning. Therefore, we investigate whether dense mask supervision can improve fine-grained perception and visual reasoning by requiring the MLLM to predict masks from its own visual representations.

In this report, we develop Xiaomi-Lens-0, a 4B MLLM built on Qwen3.5 [37] that predicts region masks directly from its own visual representations, bringing pixel-level supervision into the backbone shared with language generation. Concretely, the model reads spatial features from the patch grid of its vision encoder [10], taken before the spatial merger, without an additional hierarchical visual encoder or multi-scale feature fusion. To specify the target region, it generates a special mask token [18] whose contextual state, termed a visual latent, represents the region requested by the instruction or referenced in the preceding response. A lightweight mask decoder combines this visual latent with the shared visual features to predict the corresponding mask. Joint language and mask training updates both the shared visual features and the language model states that identify target regions, thereby giving the representations used for perception and language generation explicit supervision over region extent.

In this architecture, the mask decoder also serves as a visual latent decoder. This allows the MLLM to interleave textual reasoning with region-specific visual latents [21, 65], each of which is decoded into a mask that explicitly grounds the corresponding reasoning evidence in the image. We refer to this formulation as thinking with visual latents. To train this capability, we construct grounded reasoning traces [26, 42] that pair intermediate region references with masks, and jointly supervise both the textual responses and the decoded masks. This extends pixel-level supervision to intermediate regions referenced in chain-of-thought (CoT) [55], making their spatial extent explicit and inspectable through decoded masks.

We evaluate Xiaomi-Lens-0 across six tasks (Figure 1): REC, RES, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained perception. It improves REC over its Qwen3.5 backbone on all eight splits of RefCOCO, RefCOCO+ [67], and RefCOCOg [30, 33], with an average gain of 5.6 points, and achieves the highest cumulative intersection over union (cIoU) among the compared models on every RES split. For multi-instance grounding on COCO 2017 [24], it also achieves the highest F1 averaged over intersection over union (IoU) thresholds from 0.50 to 0.95. Beyond spatial prediction, it outperforms its backbone on every perception benchmark. To evaluate referring under conditions that existing benchmarks rarely cover, we also build LensBench, which crosses explicit and implicit instructions with single and multiple targets across general, aerial, driving, and panoramic imagery, with every paired unit human-reviewed for target completeness and instruction–mask agreement (§ 3.3). Xiaomi-Lens-0 achieves the highest cIoU among the compared models in the aerial, driving, and panoramic domains, although implicit instructions remain challenging, particularly when selecting multiple targets in panoramic scenes. Finally, ablations show that grounded reasoning traces further increase REC and RES scores.

Examples of Xiaomi-Lens-0 across six tasks

Figure 1. Examples of Xiaomi-Lens-0 across six tasks: referring expression comprehension, referring expression segmentation, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained perception. The model localizes targets with boxes or masks and answers questions about counting, attributes, and spatial relations.

Our main contributions can be summarized as follows:

  • Mask prediction from shared visual representations. We develop Xiaomi-Lens-0, a 4B MLLM that decodes region masks from the patch grid of its own vision encoder, taken before the spatial merger, and the visual latents produced by its language model, without an additional hierarchical vision encoder or multi-scale feature fusion. Joint language and mask training therefore supervises region extent directly in the representations shared with language generation.

  • Thinking with visual latents. We interleave textual reasoning with visual latents that the mask decoder maps to masks, and construct grounded reasoning traces that extend pixel-level supervision to intermediate regions in CoT.

  • LensBench. We introduce a human-reviewed referring benchmark that pairs single and multiple targets with explicit and implicit instructions across four visual domains, enabling paired comparisons of target selection and mask prediction under varying instruction semantics.

  • Mask supervision also improves perception. Beyond the spatial gains above, Xiaomi-Lens-0 outperforms its Qwen3.5 backbone on every fine-grained perception benchmark. Moreover, ablations show that adding mask supervision to box supervision further improves all three evaluated general perception benchmarks and the REC validation splits of RefCOCO, RefCOCO+, and RefCOCOg.

Method

Xiaomi-Lens-0 predicts masks directly from its own visual representations, so mask supervision must reach the same features and states that support language generation. As shown in Figure 2, the shared vision encoder supplies merged tokens to the language model and its patch-level feature grid to a lightweight mask decoder. The decoder combines this grid with a region-specific visual latent from the language model, and joint training propagates the mask loss into both. We first motivate decoding from the shared encoder (§ 2.1), then describe the architecture and joint objective (§ 2.2), extend mask prediction to intermediate regions through thinking with visual latents (§ 2.3), and present the two-stage training recipe (§ 2.4).

Figure 2. Xiaomi-Lens-0 architecture. The language model consumes spatially merged visual tokens. Each generated <|mask|> token has a contextual state that queries the encoder’s patch-level features through a lightweight mask decoder. Patch-wise compatibility scores are reassembled into a spatial grid and upsampled with two transposed-convolution layers.

Mask Prediction from Shared Visual Features

Mask prediction requires spatial features that describe region extent and a query that identifies the target. In LISA and Sa2VA, the spatial features come from an additional hierarchical segmentation encoder, and Qwen3-VL-Seg fuses multi-scale features from a supplementary image encoder. Under these designs, the MLLM’s own visual representation is not required to encode the region detail that the mask loss supervises. To supervise the shared representation directly, we decode masks from a single patch grid obtained before the MLLM’s spatial merger, without an additional encoder or multi-scale fusion. This choice preserves the encoder’s patch resolution and provides the mask loss with a direct path to the visual features used for language generation.

The patch grid of a Vision Transformer (ViT) [10] provides spatially indexed features enriched by attention across the image. Its residual structure expresses the final representation as the initial patch features plus successive transformations:

xl=xl−1+Fl(xl−1),xL=x0+∑l=1LFl(xl−1),(1)x_l = x_{l-1} + \mathcal{F}_l(x_{l-1}), \qquad x_L = x_0 + \sum_{l=1}^{L} \mathcal{F}_l(x_{l-1}), \tag*{(1)}

where x0x_0 contains the initial patch features and Fl\mathcal{F}_l is the residual transformation at layer ll. The identity mapping xl−1x_{l-1} keeps the fine-grained signals produced by earlier layers from being overwritten by higher-level semantic abstractions. Instead, those signals propagate through the network and accumulate into xLx_L, so the final representation retains a high-mutual-information pathway to the input image.

LongCat-Next formalizes this property as intrinsic information recovery and tests it by pairing frozen encoders with a lightweight pixel decoder [31]. Specifically, semantic encoders with residual pathways reconstruct coarse image structure despite receiving no reconstruction supervision, and a randomly initialized ViT even reconstructs better than its pretrained counterpart, which indicates that recoverability follows from the network structure rather than from learned semantics. Moreover, QwenViT [1], without its spatial merger, behaves like a plain ViT that maintains a single patch resolution and still recovers coarse image structure, whereas the merger degrades fidelity because its aggressive downsampling discards fine-grained, high-frequency detail. Because the patch grid is taken before the merger, it preserves the spatial structure required for mask prediction while remaining part of the encoder’s shared computation. We therefore feed this patch-level grid to the mask decoder and query it with region-specific contextual states from the language model, allowing a single shared grid to support multiple mask predictions.

Model Architecture

Shared Visual Features and Visual Latents

Language generation and mask prediction begin with the same image encoding. Given an image I∈R3×H×WI \in\mathbb{R}^{3 \times H \times W}, the vision encoder EvE_v produces a spatial grid of patch features, which we flatten for notational convenience:

V0=Ev(I)=[v1,…,vNp]∈RNp×dv,Np=hvwv,(2)V^0 = E_v(I) = [v_1, \ldots, v_{N_p}] \in\mathbb{R}^{N_p \times d_v}, \qquad N_p = h_v w_v, \tag*{(2)}

where (hv,wv)(h_v, w_v) denotes the patch-grid resolution. Following the Qwen3.5 backbone [37], an s×ss \times s spatial merger groups neighboring patch features, and the visual projector ϕ\phi maps the merged features to the language-model width dld_l:

V=ϕ(Merge⁡s(V0))∈RNv×dl,Nv=Np/s2.(3)V = \phi(\operatorname{Merge}_s(V^0)) \in\mathbb{R}^{N_v \times d_l}, \qquad N_v = N_p / s^2. \tag*{(3)}

The autoregressive language model processes the merged tokens VV together with the system and user tokens. At decoding step tt, the language-model head maps the contextual state ht∈Rdl\mathbf{h}_t \in\mathbb{R}^{d_l} to a next-token prediction.

To mark each region for mask decoding, we add a special <|mask|> token to the output vocabulary. For its kk-th occurrence at response position tkt_k, we take the contextual state mk=htk∈Rdl\mathbf{m}_k = \mathbf{h}_{t_k} \in\mathbb{R}^{d_l} as the visual latent. The token identity is shared across occurrences, but each state depends on the image, the instruction, and the preceding response, so it represents the region requested by the instruction or referenced in the preceding response. A response can contain several mask tokens, each supplying the query for a separate mask.

Mask Decoding

The mask decoder scores each patch of the grid V0\mathbf{V}^{0} (Equation (2)) for the requested target. A lightweight Transformer layer first refines these features with global visual context, and a linear projection then maps each patch to a mask feature space of width dmd_m:

F=Pv(Tmask(V0))=[f1,…,fNp]∈RNp×dm.(4)\mathbf{F}=P_v\left(\mathcal{T}_{\mathrm{mask}}(\mathbf{V}^{0})\right)=[\mathbf{f}_1,\ldots,\mathbf{f}_{N_p}]\in\mathbb{R}^{N_p\times d_m}. \tag*{(4)}

Next, we project the mask-token state into the same space, qk=Pm(mk)∈Rdm\mathbf{q}_k=P_m(\mathbf{m}_k)\in\mathbb{R}^{d_m}, and use it as a dynamic classifier over patch locations. Its compatibility with patch pp is

zk,p=qkTfpdm,p=1,…,Np.(5)z_{k,p}=\frac{\mathbf{q}_k^{\mathsf{T}}\mathbf{f}_p}{\sqrt{d_m}},\qquad p=1,\ldots,N_p. \tag*{(5)}

Finally, we reassemble the NpN_p scores into the original patch layout and apply two transposed convolution layers to produce a dense mask logit map. Applying a sigmoid gives

M^k=σ(Umask(Reassemble⁡hv,wv(zk)))∈[0,1]H×W,(6)\widehat{\mathbf{M}}_k=\sigma\left(\mathcal{U}_{\mathrm{mask}}\left(\operatorname{Reassemble}_{h_v,w_v}(\mathbf{z}_k)\right)\right)\in[0,1]^{H\times W}, \tag*{(6)}

where Umask\mathcal{U}_{\mathrm{mask}} is the two-layer upsampling head and σ\sigma is the sigmoid function. Because all regions share the same image features, we compute F\mathbf{F} once per image and reuse it for every visual latent, changing only the query qk\mathbf{q}_k. Beyond the shared vision encoder, the decoder consists of only one Transformer refinement layer, two linear projections, and a two-layer upsampling head.

Joint Language and Mask Supervision

Pixel-level mask supervision gives each visual latent an explicit target region. For a training example with KK supervised <|mask|> occurrences, the kk-th occurrence is paired with a binary target mask Mk\mathbf{M}_k. The mask objective combines pixel-wise binary cross-entropy and Dice loss [32]:

Lmask=1K∑k=1K[λbceLbce(M^k,Mk)+λdiceLdice(M^k,Mk)],(7)\mathcal{L}_{\mathrm{mask}}=\frac{1}{K}\sum_{k=1}^{K}\left[\lambda_{\mathrm{bce}}\mathcal{L}_{\mathrm{bce}}(\widehat{\mathbf{M}}_k,\mathbf{M}_k)+\lambda_{\mathrm{dice}}\mathcal{L}_{\mathrm{dice}}(\widehat{\mathbf{M}}_k,\mathbf{M}_k)\right], \tag*{(7)}

where λbce\lambda_{\mathrm{bce}} and λdice\lambda_{\mathrm{dice}} weight the two terms. The cross-entropy term supervises individual pixels, and the Dice term supervises region overlap. Adding the mask objective to the next-token prediction loss gives the joint objective:

L=Lntp+Lmask.(8)\mathcal{L}=\mathcal{L}_{\mathrm{ntp}}+\mathcal{L}_{\mathrm{mask}}. \tag*{(8)}

During joint training, the mask loss is propagated to the vision encoder via F\mathbf{F} and to the language model via mk\mathbf{m}_k, while Lntp\mathcal{L}_{\mathrm{ntp}} supervises the response sequence. Region extent must therefore be encoded in the shared image features and in the contextual states that identify targets. Since text answers are generated from these same representations, mask supervision also improves fine-grained perception, as § 4.6 shows.

Thinking with Visual Latents

Because each visual latent is a contextual state of the language model, the mask decoder also decodes visual latents produced during reasoning. Specifically, in thinking with visual latents, the response interleaves textual reasoning with <|mask|> tokens placed at region references. Each token’s contextual state represents the region discussed in the preceding text, and the decoder maps that state to a mask over the shared image grid. A counting trace, for example, can enumerate individual instances with one visual latent each, and a spatial comparison can assign separate latents to the objects being compared. Each latent is tied to an explicit image region.

We supervise these intermediate regions using grounded reasoning traces that pair region references with target masks (§ 3.2). The next-token loss supervises the reasoning text, spatial references, and final answer, while the mask loss supervises the extent of each annotated region. The joint objective in Equation (8) supervises both final segmentation targets and intermediate regions in a CoT.

During inference, mask decoding is an output branch alongside autoregressive generation: subsequent text is conditioned on the image and the preceding response tokens, and no crops or region features are fed back to the language model. Thus, the decoded masks make the regions behind intermediate statements inspectable, while training extends pixel-level supervision to the states that represent those regions.

Training Recipe

We train in two stages. Stage 1 pretrains the mask decoder to map frozen representations to regions, and Stage 2 jointly fine-tunes the backbone and decoder. Figure 3 summarizes the data mixtures. The in-house data, detailed in § 3, provide masks for referring-expression targets and for intermediate regions in grounded reasoning traces.

Figure 3. Training data for Stage 1 mask decoder pretraining (left) and Stage 2 supervised fine-tuning (right). Donut charts show the proportions of public and in-house data; the adjacent lists give the public datasets and their sample counts.

Stage 1: Mask Decoder Pretraining

We freeze the pretrained backbone and optimize only the mask decoder with Equation (7), so that it learns to decode regions from fixed image features and visual latents. The training mixture combines in-house data with public sources covering objects, parts, and referring expressions, including Objects365 [43], LVIS [13], PACO [38], PhraseCut [57], RefCOCO, RefCOCO+, and RefCOCOg. These sources expose the decoder to regions at different scales and levels of specificity before joint adaptation.

Stage 2: Supervised Fine-Tuning

We then unfreeze the backbone and optimize the full model with Equation (8). The language loss supervises the response sequences, and the mask loss updates both the shared visual features and the visual latents. The data combine in-house referring-expression and grounded reasoning examples with COCO 2017, RefCOCO, RefCOCO+, RefCOCOg, RoboRefIt [28], and ReasonSeg [18].

Data Pipeline

The data pipeline supplies the two forms of region supervision used in § 2. Specifically, referring-expression data pair instructions with final target masks to supervise REC and RES outputs (§ 3.1), whereas grounded reasoning data attach masks to the regions referenced at intermediate steps of a CoT to supervise thinking with visual latents (§ 3.2). Both pipelines fix or verify the region annotation before accepting the accompanying language. We then present LensBench (§ 3.3), which evaluates target selection and region extent as instruction semantics, target cardinality, and visual domain vary.

Referring-Expression Training Data

Referring-expression data provide explicit region targets for REC and RES. We first establish a box and mask for each candidate object, then generate instructions that describe it, and finally verify that each instruction selects the annotated region (Figure 4). Fixing the target before instruction synthesis allows several descriptions of the same object to share a single spatial annotation.

Figure 4. RES data pipeline. (a) Candidate objects are localized, their descriptions are checked against image crops, and masks are generated. (b) The masked view and original image guide referring-instruction synthesis. (c) Re-grounding and contextual verification check that each instruction identifies its intended target.

Visual Grounding

We establish candidate regions through object discovery and crop-level verification. Given an image II, object discovery returns boxes bib_i with short descriptions did_i. Before generating masks or instructions, a crop–text check Vcrop(X,t)∈{0,1}V_{\mathrm{crop}}(X,t) \in\{0,1\} verifies each description against its image crop. The accepted candidates are

CI={(bi,di)∈D(I)∣Vcrop(Crop⁡(I,bi),di)=1},(9)C_I = \{(b_i,d_i) \in\mathcal{D}(I) \mid V_{\mathrm{crop}}(\operatorname{Crop}(I,b_i),d_i)=1\}, \tag*{(9)}

where D(I)\mathcal{D}(I) is the set of discovered box–description pairs. For each (bi,di)∈CI(b_i,d_i) \in C_I, mask generation produces Mi=S(I,bi,di)∈{0,1}H×WM_i = \mathcal{S}(I,b_i,d_i) \in\{0,1\}^{H \times W}.

Instruction Synthesis

We generate referring instructions from both the target appearance and its scene context. For each retained target, a masked view I~i=I⊙Mi\tilde{I}_i = I \odot M_i preserves the target pixels and sets the background to black. The generator then receives this view to identify the referent and the original image to describe its relation to surrounding objects:

Qi={qij}j=1Ki=G(I,I~i),(10)Q_i = \{q_{ij}\}_{j=1}^{K_i} = \mathcal{G}(I,\tilde{I}_i), \tag*{(10)}

where KiK_i is the number of expressions generated for target ii. The expressions identify the target through location, appearance, action, function, or contrasts with neighboring objects; a blanket, for example, can be described by its pattern or its use for warmth. Concise descriptions, negation, and quantity references, such as distinguishing one object within a counted group, further diversify the wording while preserving the single annotated target.

Consistency Filtering

We retain instructions that agree with the target in both localization and meaning. Re-grounding each qijq_{ij} in the original image must return exactly one box, R(I,qij)={bij′}\mathcal{R}(I,q_{ij})=\{b'_{ij}\}, whose IoU with the candidate box meets a threshold. In addition, a separate check Vref(I,I~i,qij)∈{0,1}\mathcal{V}_{\mathrm{ref}}(I,\tilde{I}_{i},q_{ij})\in\{0,1\} uses the original image and masked view to assess whether the expression uniquely describes the target and its contextual relations. With an IoU threshold τ\tau, the retained instructions are

Q~i={qij∈Qi | R(I,qij)={bij′}, IoU⁡(bij,bij′)≥τ, Vref(I,I~i,qij)=1}.(11)\tilde{Q}_{i}=\left\{q_{ij}\in Q_{i}\ \middle|\ \mathcal{R}(I,q_{ij})=\{b'_{ij}\},\ \operatorname{IoU}(b_{ij},b'_{ij})\geq\tau,\ \mathcal{V}_{\mathrm{ref}}(I,\tilde{I}_{i},q_{ij})=1\right\}. \tag*{(11)}

The semantic check screens for references to nearby instances or properties absent from the target, complementing the box-overlap criterion. For each retained instruction, the final box is the tight bounding box of the mask, box⁡(Mi)\operatorname{box}(\mathcal{M}_{i}). The resulting sample (I,qij,box⁡(Mi),Mi)(I,q_{ij},\operatorname{box}(\mathcal{M}_{i}),\mathcal{M}_{i}) pairs the same referent with box and pixel-level supervision.

Grounded Reasoning Data

Grounded reasoning data supervise the intermediate regions used in thinking with visual latents (§ 2.3). They target the region-level failures that motivate this work, such as miscounting nearby objects and misidentifying the attributes of small regions. Each visual question answering (VQA) example pairs an image and question with a reasoning trace, a final answer, and masks for the trace’s region references. We first synthesize question–answer pairs, verify the answers independently, and then construct traces with mask-annotated region references (Figure 5). The spatial training signal thus extends from final segmentation targets to the regions discussed along the reasoning trace.

Figure 5. VQA data pipeline. (a) After image and region screening, an MLLM generates question–answer pairs with multiple-choice or numeric answers, optionally referring to regions by temporary IDs. (b) Rule-based filters remove flawed questions, options are shuffled, and an independent checker must reproduce the reference answer. (c) A reasoning model generates traces with box references. Traces that are complete, use valid boxes, and match the verified answer are paired with masks whose bounding boxes agree with the referenced boxes.

Question–Answer Synthesis

We synthesize questions about counting, attributes, and spatial relations, each of which requires perceiving specific regions. After screening images and available region annotations, we assign temporary IDs to retained boxes so that questions can distinguish similar instances or combine information across regions. An MLLM generates question–answer pairs with multiple-choice or numeric answers, providing a common format for exact-match verification.

Question–Answer Verification

We verify the answers before generating reasoning traces. Rule-based filters remove malformed or duplicate options, near-duplicate questions, and wording that reveals the answer; multiple-choice options are then shuffled to remove position cues. Next, an independent checker answers each remaining question from the image without seeing the reference answer, receiving region hints only when the question names them. We retain a pair when its predicted option label or numeric value exactly matches the reference. Final examples use the original image, retaining ID markers only for regions named in the question.

Trace Generation and Mask Annotation

We generate reasoning traces that explicitly reference the image regions discussed at each intermediate step. For each verified pair, a reasoning model produces candidate traces that interleave text with <bbox> references and end with an answer. We retain complete traces with valid box references and an answer matching the verified reference. When several traces pass, we select the shortest to limit redundancy; pairs with no passing trace are discarded.

After trace selection, each accepted box reference is paired with an existing instance mask or, when unavailable, a mask generated by SAM 3 [4]. We accept the mask when the IoU between its tight bounding box and the referenced box exceeds a threshold, and append a <|mask|> token to the reference. Its contextual state becomes the visual latent supervised by that mask through Equation (7). Figure 6 shows counting and spatial-relation traces with masks attached to the regions named at each step.

Figure 6. Thinking with visual latents. The top example identifies the position of food relative to a woman. The middle example localizes people in two boats and sums the per-boat counts. The bottom example localizes and counts individual picture frames. Colored boxes and masks show the regions referenced in each reasoning trace.

LensBench

Referring benchmarks such as RefCOCO pair each expression with a single target in everyday photographs and usually name the target’s category. To evaluate a broader range of referring conditions, LensBench examines how target selection and mask prediction vary when the instruction describes targets indirectly, requests several instances, or comes from a different visual domain. It covers general, aerial, driving, and panoramic imagery, with 300 complete evaluation units per domain. Each unit pairs one image with two target masks and four instructions crossing single and multiple targets with explicit and implicit wording (Figure 7). The benchmark contains 1,200 paired units and 4,800 instruction-level samples.

Figure 7. LensBench design and construction. (a) Examples from four domains combine single and multiple targets with explicit and implicit instructions. Each unit contains two ground-truth masks and four instructions. (b) Visual grounding, instance curation, and grouping establish the targets before instruction synthesis. Automated checks of masks, instructions, and re-grounded instances inform human review of each complete unit.

Benchmark Design

Each unit separates the number of requested instances from the way they are described. Single instructions select one instance, and multiple instructions select at least two distinct instances. Explicit instructions name the category and specify the target set through attributes, relations, or quantity. Implicit instructions describe the targets through function, purpose, or commonsense relations, omitting the category and direct synonyms so that selection requires an image-supported inference.

For an image II, let GG be a set of target instances from the same category, with ∣G∣≥2|G| \ge2, and let f∈Gf \in G be a distinguishable focus instance. The target masks are

Msingle=Mf,Mmultiple=⋁i∈GMi,(12)M_{\mathrm{single}} = M_f, \qquad M_{\mathrm{multiple}} = \bigvee_{i \in G} M_i, \tag*{(12)}

where ⋁\bigvee denotes pixel-wise logical OR. We generate one explicit and one implicit instruction for each mask, yielding four evaluation samples from the same image. The two instruction types share the same ground-truth mask within each cardinality, enabling paired comparisons.

Benchmark Construction

We fix the target instances and masks before generating instructions. Candidate annotations follow the visual-grounding procedure in § 3.1.1; instance curation verifies identity and visible extent and resolves duplicates. Masks are stored at their original resolution using lossless run-length encoding.

Within each image, we group countable instances by category. Each group must contain every visible instance within its intended scope, without unrelated objects or duplicate physical instances. We then select a focus instance f∈Gf \in G and fix both target masks before generating instructions. The generator receives the original image, a crop containing the focus instance and its surroundings, and target metadata:

Qu={quj}j=14=G(I,Crop⁡(I,cf),zu),(13)Q_u = \{q_{uj}\}_{j=1}^{4} = \mathcal{G}(I, \operatorname{Crop}(I,c_f), z_u), \tag*{(13)}

where u=(I,G,f)u=(I,G,f) indexes the unit and cfc_f expands box⁡(Mf)\operatorname{box}(M_f) to include surrounding context. The indices j=1,2,3,4j = 1, 2, 3, 4 correspond to single–explicit, single–implicit, multiple–explicit, and multiple–implicit, respectively. The metadata zuz_u gives the canonical category, group size, and focus bounding box box⁡(Mf)\operatorname{box}(M_f). Each instruction must preserve its assigned target set and semantic granularity.

Consistency Filtering

Automatic screening checks mask quality, instruction semantics, and agreement with independently re-grounded instances. The mask check Vmask(u)\mathcal{V}_{\mathrm{mask}}(u) assesses identity and visible extent, detects duplicate physical instances, and verifies the union of individual masks. The instruction check Vref(u,q)\mathcal{V}_{\mathrm{ref}}(u,q) uses individual and union masks to assess whether the wording is natural, self-contained, precise about the target set, and consistent with its assigned explicit or implicit condition. The intended instance set is Gq={f}G_q=\{f\} for single and Gq=GG_q=G for multiple.

To check whether each instruction recovers the intended targets, re-grounding R(I,q)\mathcal{R}(I,q) receives only the original image and instruction, with no target annotations. Its output retains duplicate predictions. We match predicted boxes to target instances one to one, maximizing total IoU, and denote the prediction assigned to instance ii by bqi′b'_{q_i}. Let δq\delta_q count pairs of identical normalized prediction boxes. With an IoU threshold τ\tau, the instructions passing all automatic checks are

Q~u={q∈Qu | ∣R(I,q)∣=∣Gq∣,δq=0,min⁡i∈GqIoU⁡(box⁡(Mi),bqi′)≥τ,Vmask(u)=1,Vref(u,q)=1}.(14)\widetilde{\mathcal{Q}}_u = \left\{ q \in\mathcal{Q}_u \ \middle|\ \begin{aligned} &|\mathcal{R}(I,q)| = |G_q|,\quad\delta_q = 0,\\ &\min_{i\in G_q}\operatorname{IoU}(\operatorname{box}(M_i), b'_{q_i}) \ge\tau,\\ &V_{\mathrm{mask}}(u) = 1,\quad V_{\mathrm{ref}}(u,q) = 1 \end{aligned} \right\}. \tag*{(14)}

A unit passes automatic screening when all four instructions pass, that is, Q~u=Qu\widetilde{\mathcal{Q}}_u = \mathcal{Q}_u. Uncertain judgments and failed requests are recorded separately to inform human review.

Finally, human reviewers make the final selection or revision decision for each complete unit. They inspect the original image, individual instance masks, both target masks, and all four instructions together to check target completeness and instruction–mask agreement. Only complete four-sample units enter LensBench, preserving the paired comparisons. Each sample is (I,q,box⁡(Mq),Mq)(I, q, \operatorname{box}(M_q), M_q), where Mq=⋁i∈GqMiM_q = \bigvee_{i\in G_q} M_i is the mask assigned to instruction qq.

Experiments

The experiments test the question raised in § 1: whether dense mask supervision, applied to masks predicted from the MLLM’s own visual representations, improves fine-grained perception and visual reasoning. We address three questions. (1) Spatial prediction: without an additional encoder for spatial features, how well does Xiaomi-Lens-0 predict boxes and masks compared with methods that use one? (2) Fine-grained perception: do the gains extend to counting, attribute, and spatial-relation questions answered in text? (3) Attribution: what do mask supervision and grounded reasoning traces each contribute? § 4.2 addresses the first two questions across six tasks, § 4.5 examines instruction semantics and visual domains on LensBench, and § 4.6 addresses the third.

Evaluation Setup

The evaluation covers REC, RES, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained perception. All scores are percentages, and differences are reported in percentage points. Main-model results use the equal-weight average of three Stage 2 checkpoints, with a generation limit of 16,384 tokens.

  • REC and RES. We use the validation, TestA, and TestB splits of RefCOCO and RefCOCO+ [67], and the validation and test splits of RefCOCOg [30, 33]. For REC, [email protected] is the percentage of predicted boxes that have an IoU of at least 0.5 with the target box. A complementary box-grounding evaluation reports F1 at IoU thresholds of 0.50 and 0.95, and the mean F1 over thresholds from 0.50 to 0.95 in steps of 0.05 (F1@mIoU). RES uses cIoU on the same eight splits.

  • Multi-instance grounding. COCO 2017 [24] evaluates localization of multiple object instances. We report recall, precision, and F1 at box IoU thresholds of 0.50 and 0.95, along with their mIoU aggregates.

  • Reasoning segmentation and embodied referring. ReasonSeg [18] evaluates target selection from reasoning-dependent instructions, and RoboRefIt [28] evaluates referring expressions in embodied scenes. We report gIoU and cIoU on the ReasonSeg validation and test sets and on RoboRefIt TestA and TestB, following the ReasonSeg convention, in which gIoU is the mean per-image IoU.

  • Fine-grained perception. The English and Chinese MME-RealWorld sets (MME-RW-en and MME-RW-cn) [77] and CV-Bench (CV-B) [47] assess general perception. V* [59] and the 4K and 8K HR-Bench sets (HR-4K and HR-8K) [54] assess high-resolution perception. CountQA [45] and ColorBench (ColorB) [23] assess counting and color recognition. All perception scores measure answer accuracy, with no box or mask output.

For NN image–instruction pairs with binary prediction and target masks PiP_i and GiG_i and nonzero unions, the benchmark definitions are

gIoU⁡=1N∑i=1N∣Pi∩Gi∣∣Pi∪Gi∣,cIoU⁡=∑i=1N∣Pi∩Gi∣∑i=1N∣Pi∪Gi∣(15)\operatorname{gIoU} = \frac{1}{N} \sum_{i=1}^{N} \frac{\lvert P_i \cap G_i \rvert}{\lvert P_i \cup G_i \rvert}, \qquad\operatorname{cIoU} = \frac{\sum_{i=1}^{N} \lvert P_i \cap G_i \rvert}{\sum_{i=1}^{N} \lvert P_i \cup G_i \rvert} \tag*{(15)}

where ii indexes image–instruction pairs. gIoU weights each pair equally, while cIoU pools pixel intersections and unions across the dataset and therefore weights large targets more heavily.

On LensBench (§ 3.3), we report mask cIoU and [email protected] by instruction type and visual domain. Unlike REC, [email protected] here counts pairs whose predicted mask has an IoU of at least 0.5 with the target mask. For multiple targets, both metrics use the unions of predicted and target instance masks.

Main Results

Referring Expression Comprehension

Xiaomi-Lens-0 improves box localization over its Qwen3.5 backbone on all eight REC splits, with an average gain of 5.6 points (Table 1). In particular, the mean gain is largest on RefCOCO+, at 7.6 points, compared with 4.4 points on both RefCOCO and RefCOCOg. RefCOCO+ excludes absolute location words [67], so instances must be distinguished by appearance and attributes. Within this dataset, the TestB split shows the largest backbone-relative gain, at 9.4 points.

MethodsRefCOCO ValRefCOCO TestARefCOCO TestBRefCOCO+ ValRefCOCO+ TestARefCOCO+ TestBRefCOCOg ValRefCOCOg Test
Grounding-DINO-L [25]90.693.288.282.889.075.986.187.0
Florence-2 [61]93.495.392.088.392.983.691.291.7
LISA-LLaVA-7B [18]85.488.882.674.279.568.479.380.4
GSVA-Vicuna-7B [60]86.389.283.872.878.868.081.681.8
PixelLM-LLaVA-7B [41]89.892.286.483.287.078.984.686.0
VisionLLM v2-Vicuna-7B [58]87.991.284.377.683.870.282.984.1
Text4Seg-InternVL2-8B [20]90.393.487.585.289.979.585.485.4
UFO-InternVL2-8B [46]91.493.888.285.790.779.786.887.4
MLLMSeg-InternVL2.5-4B [50]91.693.387.686.090.079.888.388.6
InternVL-3.5-4B [53]92.594.388.287.692.381.689.689.3
Qwen2.5-VL-3B [2]89.191.784.082.488.074.185.285.7
Qwen3-VL-4B [1]90.792.286.782.989.475.687.387.7
Qwen3-VL-Seg-4B [66]92.794.689.887.892.182.289.189.0
Qwen3.5-4B [37]90.492.387.184.888.979.188.088.2
Youtu-VL-4B [56]93.695.290.890.193.985.492.292.9
Xiaomi-Lens-0-4B94.696.292.392.194.988.592.592.4

Table 1. REC results on RefCOCO and RefCOCO+ [67], and RefCOCOg [30, 33]. Scores are [email protected] (%, higher is better). Best scores are bold; second-best scores are underlined.

Among the compared models, Xiaomi-Lens-0 ranks first on seven splits and second on RefCOCOg test. It exceeds Youtu-VL-4B [56], the strongest baseline on each RefCOCO+ split, by 2.0, 1.0, and 3.1 points on validation, TestA, and TestB, while Youtu-VL-4B retains a 0.5-point lead on RefCOCOg test.

Localization across IoU thresholds. The gains persist at stricter overlap thresholds (Table 2). F1@mIoU reaches 83.3, 81.1, and 81.1 on RefCOCO, RefCOCO+, and RefCOCOg, respectively, ranking first among the compared models and exceeding the backbone by 8.8, 11.6, and 9.5 points. Moreover, at IoU 0.95, the backbone-relative gains range from 15.5 to 16.8 points, so the improvement covers precise box placement as well as target selection.

Table 2 REC results across IoU thresholds on RefCOCO and RefCOCO+ [67], and RefCOCOg [30, 33]. [email protected] and [email protected] use the indicated IoU thresholds; F1@mIoU is the mean F1 over thresholds from 0.50 to 0.95 in steps of 0.05. All scores are percentages (higher is better). Best scores are bold; second-best scores are underlined.

MethodsRefCOCORefCOCO+RefCOCOg
[email protected][email protected]F1@mIoU[email protected][email protected]F1@mIoU[email protected][email protected]F1@mIoU
GroundingDINO [25]51.223.344.349.220.941.458.023.249.7
Rex-Omni [15]84.432.971.476.929.564.184.734.772.3
LocateAnything Fast [51]92.544.080.684.439.573.188.540.776.2
LocateAnything Hybrid [51]92.744.381.485.840.675.189.441.577.7
LocateAnything Slow NTP [51]92.438.480.285.235.773.988.735.676.1
Qwen2.5-VL7B_{7\mathrm{B}} [2]82.417.766.973.016.459.375.315.660.3
Qwen3-VL2B_{2\mathrm{B}} [1]88.326.772.481.224.366.585.925.769.8
Qwen3-VL4B_{4\mathrm{B}} [1]92.132.877.686.830.973.488.832.374.7
Qwen3-VL8B_{8\mathrm{B}} [1]91.129.975.886.429.072.088.728.873.3
Qwen3.54B_{4\mathrm{B}} [37]90.129.174.584.127.869.588.427.071.6
Qwen3.59B_{9\mathrm{B}} [37]92.035.178.387.533.974.689.533.575.3
RynnBrain1.1 [22]82.620.965.872.519.257.584.321.567.6
SenseNova-Vision [14]90.145.179.784.441.774.389.244.778.7
MiMo-VL-SFT7B_{7\mathrm{B}} [62]86.114.566.376.813.158.984.914.364.8
MiMo-VL-RL7B_{7\mathrm{B}} [62]90.414.268.584.813.664.487.713.164.8
BAGEL [9]79.425.165.268.320.554.577.722.262.3
RynnBrain [7]74.013.655.665.511.748.976.514.357.9
Xiaomi-Lens-04B94.6<u>44.6</u>83.392.143.681.192.5<u>43.8</u>81.1

Table 2.

Referring Expression Segmentation

RES directly tests whether masks decoded from the shared patch-level grid can match methods that obtain spatial features from an additional encoder. Under this setup, Xiaomi-Lens-0 achieves the highest cIoU among the compared models on all eight RES splits (Table 3), averaging 81.4 versus 78.6 for Qwen3-VL-Seg4B_{4B} [66], the strongest baseline on average. Margins over the best competing score on each split range from 0.9 to 2.1 points on RefCOCO, 3.1 to 4.9 on RefCOCO+, and 2.8 to 2.9 on RefCOCOg. Xiaomi-Lens-0 also exceeds the dedicated segmentation model SAM 3 [4] by 7.0 to 12.9 points.

Table 3 RES results on RefCOCO and RefCOCO+ [67], and RefCOCOg [30, 33]. Scores are cIoU (%, higher is better). Best scores are bold; second-best scores are underlined.

MethodsRefCOCORefCOCO+RefCOCOg
ValTestATestBValTestATestBValTest
LISALLaVA−7B_{\mathrm{LLaVA-7B}} [18]74.979.172.365.170.858.167.970.6
GSVAVicuna−7B_{\mathrm{Vicuna-7B}} [60]77.278.973.565.969.659.872.773.3
PixelLMLLaVA−7B_{\mathrm{LLaVA-7B}} [41]73.076.568.266.371.758.369.370.5
VisionLLM v2Vicuna−7B_{\mathrm{Vicuna-7B}} [58]76.679.374.364.569.861.570.771.2
Text4SegInternVL2−8B_{\mathrm{InternVL2-8B}} [20]79.281.775.672.877.966.574.075.3
UFOInternVL2−8B_{\mathrm{InternVL2-8B}} [46]78.079.775.672.376.866.673.774.3
MLLMSegInternVL2.5−4B_{\mathrm{InternVL2.5-4B}} [50]79.581.477.775.478.770.878.1<u>78.4</u>
SAM 3 [4]75.577.671.067.371.163.473.474.0
UniPixelQwen2.5−VL−3B_{\mathrm{Qwen2.5-VL-3B}} [26]80.582.676.974.378.968.476.377.0
Youtu-VL4B_{4\mathrm{B}} [56]80.782.078.476.2<u>79.6</u><u>71.4</u>76.576.6
Qwen3-VL-Seg4B_{4\mathrm{B}} [66]<u>82.3</u><u>83.7</u><u>79.1</u><u>76.2</u>80.270.8<u>78.2</u>78.1
Xiaomi-Lens-04B83.484.681.279.983.376.381.181.2

Table 3.

MLLMSegInternVL2.5-4B [50] also decodes masks from the MLLM’s vision-encoder features, and Xiaomi-Lens-0 exceeds it by 2.8 to 5.5 points across the eight splits. On both RefCOCOg RES splits, Xiaomi-Lens-0 also exceeds Youtu-VL4B by 4.6 cIoU points. Figure 9 and Figure 10 in Appendix § 10 illustrate the resulting box and mask predictions.

Figure 9. Box predictions for referring expressions based on appearance, action, relative size, and spatial relations. Each orange box marks the instance selected by the model.

Figure 10. Mask predictions for targets identified by referring expressions. Orange overlays delineate the selected fruit wedge, laptop, kite, and cow. The accompanying expressions specify the intended referent in each scene.

Multi-Instance Grounding

Beyond single-target referring, Xiaomi-Lens-0 also localizes multiple object instances on COCO 2017 (Table 4). Under the mIoU aggregate, it achieves the highest F1 among the compared models, at 63.9, which is 0.9 points above the grounding foundation model GroundingPI [68] and 3.3 points above GroundingDINO [25]. At IoU 0.50, its F1 of 80.4 trails only these two models. However, the order reverses at IoU 0.95: its F1 of 27.4 exceeds GroundingPI by 4.6 points and GroundingDINO by 1.1 points, trailing only SenseNova-Vision [14] and LocateAnything Hybrid [51]. Therefore, its lead over GroundingPI comes from tighter box placement at strict thresholds rather than from recovering more instances. Figure 11 illustrates predictions for repeated and overlapping objects.

MethodsIoU 0.50IoU 0.95mIoU
RecallPrecisionF1RecallPrecisionF1RecallPrecisionF1
GroundingDINO [25]<u>79.6</u>83.281.424.728.126.361.163.760.6
Rex-Omni [15]69.981.075.017.921.719.753.959.456.3
LocateAnything Fast [51]66.273.369.624.726.925.850.959.353.1
LocateAnything Hybrid [51]74.780.877.6<u>26.6</u><u>28.7</u><u>27.6</u>52.465.259.1
LocateAnything Slow NTP [51]75.0<u>81.9</u>78.325.027.226.054.7<u>65.1</u>59.4
Qwen2.5-VL7B_{7B} [2]57.262.959.912.213.212.744.749.146.3
Qwen3-VL2B_{2B} [1]63.567.165.321.822.622.242.644.942.5
Qwen3-VL4B_{4B} [1]66.170.668.323.724.924.344.947.846.5
Qwen3-VL8B_{8B} [1]65.769.567.523.925.124.544.847.346.3
RynnBrain1.1 [22]36.560.545.511.616.513.628.745.335.2
SenseNova-Vision [14]79.180.579.827.230.228.663.155.457.5
GroundingPI [68]81.480.9<u>81.2</u>22.922.822.8<u>63.2</u>62.8<u>63.0</u>
MiMo-VL-SFT7B_{7B} [62]65.572.268.710.011.110.545.750.548.0
MiMo-VL-RL7B_{7B} [62]65.674.569.89.210.39.744.550.247.2
BAGEL [9]65.675.270.111.013.011.946.347.246.0
RynnBrain [7]21.349.929.84.910.96.715.535.421.5
Xiaomi-Lens-04B_{4B}<u>79.6</u>81.180.427.227.727.463.464.563.9

Table 4. Multi-instance grounding results on COCO 2017 [24]. Recall, precision, and F1 are reported at IoU thresholds of 0.50 and 0.95 and as mIoU aggregates. All scores are percentages. Best scores are bold; second-best scores are underlined.

Multi-object box predictions in scenes with repeated instances and multiple categories

Figure 11. Multi-object box predictions in scenes with repeated instances and multiple categories. Labels identify the predicted category and distinguish instances within each category.

Reasoning Segmentation and Embodied Referring

Target selection becomes harder when the instruction identifies the target through its function or scene context. Xiaomi-Lens-0 leads the compared models in both gIoU and cIoU on the two ReasonSeg splits (Table 5). On the test split, it reaches 68.0 gIoU and 62.5 cIoU, exceeding the strongest baseline, Qwen3-VL-based InstructSAM [72], by 6.9 and 1.5 points. On validation, the cIoU margin is 0.4 points over LISA++LLaVA−7B_{\mathrm{LLaVA-7B}} [64], at 68.5 versus 68.1.

MethodsReasonSegRoboRefIt
ValTestTestATestB
gIoUcIoUgIoUcIoUgIoUcIoUgIoUcIoU
LISALLaVA−7B\mathrm{LISA}_{\mathrm{LLaVA-7B}} [18]52.954.047.348.436.129.928.722.7
LISA++LLaVA−7B\mathrm{LISA++}_{\mathrm{LLaVA-7B}} [64]64.268.157.059.558.751.553.547.8
PixelLMLLaVA−7B\mathrm{PixelLM}_{\mathrm{LLaVA-7B}} [41]44.737.444.039.745.238.438.231.4
Sa2VAInternVL2−2B\mathrm{Sa2VA}_{\mathrm{InternVL2-2B}} [71]59.260.056.955.856.829.034.010.2
Sa2VAInternVL2−8B\mathrm{Sa2VA}_{\mathrm{InternVL2-8B}} [71]65.261.159.557.858.727.936.19.0
X-SAMPhi−3−3.8B\mathrm{X\text{-}SAM}_{\mathrm{Phi-3-3.8B}} [48]56.632.957.841.041.210.325.14.0
InstructSAMQwen3−VL−2B\mathrm{InstructSAM}_{\mathrm{Qwen3-VL-2B}} [72]62.565.061.161.082.558.474.455.5
Xiaomi-Lens-04B\mathrm{Xiaomi\text{-}Lens\text{-}04B}71.268.568.062.587.088.481.581.2

Table 5. Reasoning segmentation on ReasonSeg [18] and embodied referring on RoboRefIt [28]. Scores are gIoU and cIoU (%, higher is better; Equation (15)). Best scores are bold; second-best scores are underlined.

For embodied referring on RoboRefIt, Xiaomi-Lens-0 reaches 87.0 gIoU and 88.4 cIoU on TestA, and 81.5 gIoU and 81.2 cIoU on TestB. It exceeds InstructSAM, the strongest listed gIoU baseline, by 4.5 and 7.1 points on the respective splits. Figure 12 and Figure 13 illustrate target selection through function, appearance, and spatial context.

Figure 12. Predictions from functional descriptions. Each example pairs the instruction and response with the predicted boxes and masks in orange. The spacecraft and packaging traces explicitly consider multiple candidate referents.

Figure 13. Object selection from requests that combine category, appearance, and position. Orange boxes and masks show the predictions alongside the reasoning and response.

Fine-Grained Perception

To test whether the gains extend beyond spatial outputs, the perception benchmarks evaluate the region-level failures identified in § 1 through text answers alone. Xiaomi-Lens-0 exceeds its Qwen3.54B_{4B} backbone and every other compared open-source model on all eight benchmarks in Table 6. § 4.6 isolates the contribution of mask supervision to these gains.

MethodsGeneral PerceptionHigh-Resolution PerceptionCounting and Color
MME-RW-enMME-RW-cnCV-BV*HR-4KHR-8KCountQAColorB
Qwen2.5-VL7B\mathrm{Qwen2.5\text{-}VL}_{7B} [2]60.858.375.378.571.667.918.976.4
MiMo-VL-RL7B\mathrm{MiMo\text{-}VL\text{-}RL}_{7B} [62]63.459.884.381.274.472.928.382.8
Qwen3-VL4B\mathrm{Qwen3\text{-}VL}_{4B} [1]63.563.685.080.178.372.928.181.6
Qwen3-VL8B\mathrm{Qwen3\text{-}VL}_{8B} [1]66.066.785.486.478.974.629.082.8
MiniCPM-V-4.59B\mathrm{MiniCPM\text{-}V\text{-}4.5}_{9B} [70]58.256.280.270.269.963.623.479.8
Qwen3.54B\mathrm{Qwen3.5}_{4B} [37]66.560.385.482.281.575.643.882.7
Xiaomi-Lens-04B\mathrm{Xiaomi\text{-}Lens\text{-}04B}69.767.588.187.482.580.650.684.4

Table 6. General and fine-grained perception results. All scores are answer accuracy (%, higher is better). Benchmark names and abbreviations are defined in the evaluation setup. Best scores are bold; second-best scores are underlined.

Counting and color recognition. CountQA and ColorB directly test counting and attribute recognition. CountQA accuracy rises from 43.8 to 50.6, a 6.8-point gain over the backbone, and ColorB improves by 1.7 points to 84.4. Figure 14 and Figure 15 show thinking with visual latents on such questions, where region references in the reasoning trace link intermediate statements to decoded masks (Appendix § 10).

Region references in spatial comparison and vehicle-color recognition

Figure 14. Region references in spatial comparison (top) and vehicle-color recognition (bottom). Matching colors link the response tokens to image regions. The enlarged view shows the small targets in the second example.

Figure 15. Instance-by-instance counting of climbing stones (top) and leaves (bottom). The traces contain five and six region references, respectively. Each colored token block corresponds to a box and a mask.

General and high-resolution perception. Backbone-relative gains are 3.2 points on MME-RW-en, 7.2 on MME-RW-cn, and 2.7 on CV-B. On the high-resolution benchmarks, which require recognizing small regions in large images, gains range from 1.0 point on HR-4K to 5.2 on V*, with a 5.0-point improvement on HR-8K. Xiaomi-Lens-0 also exceeds Qwen3-VL8B_{8\mathrm{B}} [1] on all three high-resolution sets.

LensBench Results

LensBench examines region prediction as instruction semantics and visual domains vary. Table 7 reports mask cIoU and [email protected] by instruction type and domain, pooling single and multiple targets. The evaluation covers 300 samples in each of the 16 domain–cardinality–instruction partitions, totaling 4,800 instructions from 1,200 paired units. Explicit and implicit wording share the same target mask within each cardinality. We evaluate all compared models under the same instructions, and each model predicts masks directly from the image and instruction.

MethodsExplicit cIoUExplicit [email protected]Implicit cIoUImplicit [email protected]General cIoUGeneral [email protected]Aerial cIoUAerial [email protected]Driving cIoUDriving [email protected]Panoramic cIoUPanoramic [email protected]
GPT-5.6-Sol [35]63.873.261.270.068.467.942.258.568.481.857.178.3
Gemini-3.6-Flash [11]61.474.561.070.363.665.850.162.566.881.060.880.3
MiMo-V2.5 [63]51.452.247.446.854.245.232.638.650.552.845.461.4
Seed-2.1-Pro [3]62.170.255.964.560.656.750.158.566.877.459.976.7
Qwen3.8-Max [36]67.669.264.563.874.279.733.562.558.057.760.866.2
Kimi-K3 [16]29.439.128.333.038.526.113.844.947.231.113.442.1
GLM-5.3-Flash [73]44.337.543.533.348.926.328.033.451.647.530.634.4
DeepSeek-V4-Flash-Vision-Exp [8]41.830.741.127.346.128.125.928.943.927.932.831.0
SAM 3 [4]17.338.115.419.523.824.84.327.327.735.111.127.9
Xiaomi-Lens-071.485.054.770.063.979.157.179.668.679.462.372.1

Table 7. LensBench marginal results by instruction type and visual domain, pooling single and multiple targets. Scores are mask cIoU and [email protected] (%, higher is better); [email protected] uses a mask IoU threshold of 0.5, and multiple targets use the unions of predicted and target instance masks. All models are evaluated by us with the same instructions and predict masks directly. Best scores are bold; second-best scores are underlined.

Xiaomi-Lens-0 achieves the highest cIoU among the compared models in three of the four domains: 57.1 on aerial, 68.6 on driving, and 62.3 on panoramic. The corresponding margins over the strongest baseline in each domain are 7.0, 0.2, and 1.5 points. Aerial imagery shows the largest gains in both metrics: the strongest baselines reach 50.1 cIoU and 62.5 [email protected], compared with 57.1 and 79.6 for Xiaomi-Lens-0. By contrast, on general imagery, Qwen3.8-Max [36] leads with 74.2 cIoU and 79.7 [email protected], while Xiaomi-Lens-0 reaches 63.9 and 79.1.

Across instruction types, implicit instructions remain harder than explicit ones. Xiaomi-Lens-0 leads on explicit instructions with 71.4 cIoU and 85.0 [email protected]. However, on implicit instructions, its cIoU falls to 54.7, ranking fifth behind Qwen3.8-Max at 64.5, GPT-5.6-Sol [35] at 61.2, Gemini-3.6-Flash [11] at 61.0, and Seed-2.1-Pro [3] at 55.9. By contrast, its [email protected] of 70.0 ties GPT-5.6-Sol for second behind Gemini-3.6-Flash at 70.3. The decline from explicit to implicit instructions is 16.7 points in cIoU and 15.0 points in [email protected].

Examining instruction type and cardinality together reveals a particularly large drop on panoramic imagery with multiple targets. [email protected] falls from 81.3 for explicit instructions to 48.7 for implicit instructions, and cIoU falls from 71.8 to 48.9. These results identify implicit selection of multiple targets in panoramic scenes as a remaining challenge.

Instruction-conditioned target selection. Figure 8 complements the aggregate results with visual comparisons under the same instructions. In general imagery, Xiaomi-Lens-0 selects the bitten pastry using its exposed filling, while SAM 3 selects several other pastries. In driving scenes, it identifies the foreground metal enclosure from its warning labels, padlock, and relative position, and distinguishes the requested right-hand bollard from nearby posts. Moreover, the panoramic bus example shows a mask that follows the target’s curved appearance under image distortion. These cases illustrate how appearance, spatial context, and instruction semantics jointly determine the selected region across domains.

Qualitative comparison on LensBench across general, aerial, driving, and panoramic imagery

Figure 8. Qualitative comparison on LensBench across general, aerial, driving, and panoramic imagery. Each row gives the domain, target cardinality, and instruction type alongside the query. Columns show the input image, predictions from the compared models, and the ground truth (GT); red overlays indicate the selected regions. The examples illustrate attribute- and function-based target selection, recovery of multiple instances, and differences in mask extent.

Multiple targets and region extent. The multi-target examples distinguish recovering the requested set from delineating its members. Xiaomi-Lens-0 covers both hanging lamps and both boats near the shore. For the implicit aerial query, it selects the two parallel docking structures while separating them from the ships and intervening water. In comparison, SAM 3 also includes ships, and MiMo-V2.5 extends the selected region into the water between the structures. This example illustrates the mask-level differences behind the aerial results: identifying the vicinity of the targets still leaves their spatial extent to resolve. Finally, the panoramic laundry example shows successful selection of the requested row from a functional description. Together with the aggregate decline on implicit multi-target instructions, these cases motivate examining target completeness and mask extent alongside box localization.

Ablation Study

The ablations attribute the gains to the two components introduced in § 2. The BBox-only variant is trained on the same Stage 2 grounding and reasoning data with the mask loss disabled, isolating pixel-level mask supervision. The Grounding-only variant keeps mask supervision but omits grounded reasoning data, isolating thinking with visual latents. Both tables include the original backbone as a reference.

Mask supervision and perception. Box supervision alone already improves both REC and perception over the backbone (Table 8). The BBox-only variant gains 3.1, 4.9, and 2.2 validation REC points on RefCOCO, RefCOCO+, and RefCOCOg, and reaches 68.8 on MME-RW-en, 66.1 on MME-RW-cn, and 87.9 on CV-B, gains of 2.3, 5.8, and 2.5 points.

Stage 2 SupervisionReferring Expression ComprehensionGeneral Perception
BBoxMaskRefCOCORefCOCO+RefCOCOgMME-RW-enMME-RW-cnCV-B
✗✗90.484.888.066.560.385.4
✓✗93.589.790.268.866.187.9
✓✓94.692.192.569.767.588.1

Table 8. Spatial supervision ablation on REC and general perception. Scores are validation-set [email protected] and answer accuracy (%), respectively. The first row is the Qwen3.5-4B backbone [37].

Adding mask supervision to this box-supervised setting raises all six metrics further. The full model’s REC advantage over BBox-only ranges from 1.1 to 2.4 points, and its perception advantage from 0.2 to 1.4 points. The REC advantage is largest on RefCOCO+, at 2.4 points, and smallest on RefCOCO, at 1.1 points. Since RefCOCO+ excludes absolute location words, mask supervision helps most where the target must be separated from its neighbors by appearance and shape, which a box encodes only coarsely. Because the two variants differ only in the mask loss, these differences answer the question posed in § 1: supervising region extent in the MLLM’s own representations improves box localization and fine-grained perception expressed in text.

Takeaway 1: Mask supervision specifies target extent beyond what a box provides, helping most when appearance must disambiguate the target. Its gains on general perception benchmarks show that it improves the shared visual features.

Grounded reasoning traces. Grounding-only training already yields strong referring performance (Table 9). Its validation REC scores of 94.5, 91.5, and 92.0 exceed the backbone by 4.1, 6.7, and 4.0 points on RefCOCO, RefCOCO+, and RefCOCOg, respectively. Its RES scores are within 0.8 cIoU points of the full model on each dataset.

DataReferring Expression ComprehensionReferring Expression Segmentation
GroundingReasoningRefCOCORefCOCO+RefCOCOgRefCOCORefCOCO+RefCOCOg
✗✗90.484.888.0———
✓✗94.591.592.082.779.180.6
✓✓94.692.192.583.479.981.1

Table 9. Training data ablation on REC and RES. Scores are validation-set [email protected] and cIoU (%), respectively. The first row is the Qwen3.5-4B backbone [37]; — denotes unavailable results.

Building on this grounding-only setting, the full model adds grounded reasoning traces, which supervise the intermediate regions of thinking with visual latents (§ 2.3). Relative to Grounding-only, its REC scores are 0.1 to 0.6 points higher, and its RES scores 0.5 to 0.8 points higher across the three validation splits. The gains concentrate in mask quality: the RES improvement matches or exceeds the REC improvement on every dataset. Mask supervision on intermediate regions in a CoT thus also benefits final-target prediction.

Takeaway 2: Grounding data drives most of the referring gain, while grounded reasoning traces mainly sharpen final masks. Supervising intermediate regions thus refines region extent.

Related Work

Visual Grounding in MLLMs

MLLMs commonly ground language by generating box coordinates in the text stream, as in Shikra, Ferret, and CogVLM [5, 52, 68, 69, 74]; Florence-2 casts grounding and other vision tasks as sequence prediction [61]. Under this interface, spatial supervision reaches the model only through coordinate tokens, and the extent of the referenced region remains implicit. Youtu-VL extends autoregressive supervision to visual targets alongside language and examines how such supervision shapes multimodal learning [56]. For pixel-level grounding, RES makes region extent explicit by requiring a pixel-level mask for the referenced target.

Segmentation with MLLMs

Most MLLM-based segmentation systems obtain spatial features from a dedicated segmenter. LISA conditions a SAM-based branch on a generated token [17, 18]; LISA++ adds instance-aware outputs, and GSVA handles multiple or empty targets [60, 64]. GLaMM and OMG-LLaVA interleave masks with grounded conversation [39, 75], and Sa2VA couples an MLLM with SAM 2 for image and video tasks [40, 71]. Seg-Zero trains a reasoning model to prompt a frozen segmenter [27], and InstructSAM learns parallel instance queries that condition SAM 3 [4, 72]. Qwen3-VL-Seg replaces the segmenter with a lightweight supplementary encoder and combines multi-scale feature fusion with box-guided decoding and iterative mask refinement [66].

Alongside these dedicated segmentation branches, another line derives spatial predictions from the MLLM’s own representations or output interface. PixelLM decodes masks from features at multiple layers of its vision encoder [41], and MLLMSeg decodes them from shallow vision-encoder features fused with the language model’s visual tokens [50]; both keep the vision encoder frozen. ClawMachine refers to entities through groups of visual tokens, and PaDT maps patch-derived reference tokens to boxes or masks with a lightweight decoder [29, 44]. Text4Seg represents segmentation through textual descriptors, while UFO combines a language interface with embedding retrieval [20, 46]. Both lines evaluate the benefit of mask supervision mainly on segmentation and grounding. To study the effect beyond spatial prediction, Xiaomi-Lens-0 decodes masks from a single patch-level grid of the shared vision encoder and evaluates whether the resulting supervision also improves fine-grained perception, as measured by text answers.

Grounded and Latent Visual Reasoning

Fine-grained visual reasoning often requires details beyond a global image description. To capture such details, Cambrian-1 integrates high-resolution features through a spatially aware connector [47]; V* searches for relevant regions, and Divide, Conquer, and Combine processes high-resolution patches with textual summaries [54, 59]. OpenAI o3 thinks with images through cropping, zooming, and other image transformations [34], and DeepEyes, Pixel Reasoner, Chain-of-Focus, and Mini-o3 learn or control such image interactions during visual search [19, 49, 76, 78].

Complementing such image interactions, other work either grounds intermediate reasoning steps explicitly or carries them in visual form. Visual CoT supervises intermediate box references [42], and UniPixel links intermediate mask pointers to object-centric reasoning [26]. Meanwhile, Mirage interleaves latent visual tokens with text during multimodal reasoning [65], and LVR performs reasoning in the visual embedding space [21]. Thinking with visual latents connects these directions: each region reference produces a visual latent that the shared mask decoder maps to a mask, and the mask loss on that latent updates the backbone. Intermediate regions thus receive pixel-level supervision and remain inspectable through the decoded masks.

Conclusion

We presented Xiaomi-Lens-0, a 4B MLLM that decodes region masks from the patch-level grid of its shared vision encoder and from visual latents, the language-model states of generated mask tokens. The mask loss therefore reaches the representations used for language generation, and thinking with visual latents extends this pixel-level supervision to the regions named in a grounded CoT.

Without an additional spatial encoder, Xiaomi-Lens-0 achieves the highest cIoU on all eight RES splits, improves REC over its Qwen3.5 backbone on every split, and achieves the highest threshold-averaged F1 for multi-instance grounding on COCO 2017. Moreover, it surpasses the backbone on every fine-grained perception benchmark; the BBox-only ablation attributes a further gain on all evaluated REC and perception metrics to mask supervision, and grounded reasoning further raises REC and RES scores. On LensBench, it leads in cIoU on aerial, driving, and panoramic imagery, although implicit instructions remain harder, especially for multiple targets in panoramic scenes.

Contributors

Xiaomi-Lens-0 was developed by the SeerRay Team. Core contributors, contributors, and supervisors are listed below. Names are ordered alphabetically by surname within each group, without indicating the relative extent of individual contributions. Corresponding authors are marked with 1.

Core Contributors

  • Pei Fu1

  • Yike Liu

  • Yue Sun

  • Hanmo Wang

  • Xiuwen Xi

  • Ruoceng Zhang

  • Shaojie Zhang

Contributors

  • Tingwei Huang

  • Chao Jiang

  • Hang Li

  • Bingrui Liu

  • Zhenbo Luo

  • Heng Qu

  • Changjian Xu

  • Ming Xu

Supervisors

  • Jian Luan1

Appendix

Conversation Format and Examples

The examples below illustrate how spatial predictions are serialized in a response. A <bbox> block contains box coordinates, and a mask block contains the special <|mask|> token. The contextual state at this token is the visual latent passed to the decoder in § 2.2.

Explicit Instruction: Box Output The clothing description directly identifies the requested person. The response returns a single box after an empty reasoning block.

[
    {
        "role": "user",
        "content": "<image>\nLocate all instances in the image that match the description \"the lady with the blue shirt\". Report bbox coordinates."
    },
    {
        "role": "assistant",
        "content": "<think>\n\n</think>\n\n<bbox>[244, 469, 557, 745]</bbox>"
    }
]

Implicit Instruction: Box and Mask Outputs The instruction specifies a protective function that requires the model to infer the target category. The response identifies and describes the selected helmet, then returns its box and a mask token whose visual latent represents that target.

[
    {
        "role": "user",
        "content": "<image>\nLocate all instances in the image that match the description \"If we want to protect our heads when participating in contact sports like football or baseball, what object in the picture should we wear?\". Report bbox coordinates and masks."
    },
    {
        "role": "assistant",
        "content": "<think>\nTo protect the head in contact sports, the appropriate item to wear is a helmet. In the center of the image, sitting on a wooden shelf, is a glossy blue helmet featuring the number 99 and an Oilers logo. This object is clearly a piece of protective headgear. While there are other padded items visible on the left and right edges, the blue helmet in the middle is the most distinct and complete example of the required safety equipment. Therefore, this central blue region is the correct target to identify.\n</think>\n\n<bbox>[190, 169, 756, 924]</bbox><mask><|mask|></mask>"
    }
]

Qualitative Analysis

The qualitative examples connect the quantitative results in § 4 to target selection, region extent, and intermediate region references. Grounding examples show the boxes and masks predicted for requested targets. Complementing these grounding examples, VQA examples show how thinking with visual latents makes intermediate regions inspectable: each region discussed in the reasoning trace is linked to its decoded mask and to the final answer. For these VQA visualizations, we replay the stored completion through the same model to obtain the contextual states used for mask decoding.

Referring Expression Grounding and Multi-Object Localization Figure 9 and Figure 10 illustrate target selection and region delineation from appearance, action, relative size, and spatial relations. The masks distinguish requested instances from nearby objects, including a particular fruit wedge and a cow beside another cow. Figure 11 shows multi-instance predictions, where individual boxes distinguish repeated or overlapping objects such as oranges in a bowl and surfboards in a rack.

Grounding Functional Descriptions Figure 12 illustrates target selection from functional descriptions, including objects used to cut grass or add citrus flavor to seafood. The responses connect each requested function to visible candidates. For example, the spacecraft trace considers two candidates before selecting the foreground object, while the gift-packaging example returns both boxes.

Instruction

As an AI assistant, please help me identify the object that is specifically designed to orbit around the Earth and perform specific tasks or complete missions.

Model Output

Thinking

Looking at the image, I see two spacecraft. The one in the foreground is a large satellite with solar panels and a cylindrical body, clearly designed for orbiting Earth. The one in the background appears to be a space station (like the ISS) with solar arrays. Both are designed to orbit Earth, but the question asks for "the object" singular. The foreground object is the most prominent satellite. However, the space station also orbits Earth. Let me think about which is "specifically designed to orbit around the Earth and perform specific tasks or complete missions." Both fit. The foreground satellite is the clearest example of a satellite. The background is the ISS, a space station. The question likely refers to the satellite in the foreground as the primary answer, but the ISS also orbits. Hmm.

The phrasing "the object" suggests one. The foreground satellite is the most typical "satellite" object. I'll localize the foreground satellite.

Response

<bbox>[364, 278, 882, 926]</bbox><mask><|mask|></mask>

Instruction

When we want to surprise someone with a gift, we often put it in decorative wrapping. Can you help me identify an item in the image that could be utilized to package a gift?

Model Output

Thinking

Looking at the image, I see two perfume boxes: a Hermès "Un Jardin Sur Le Nil" box on the left and a Chanel N°5 box on the right. These are the two main objects. The question asks about an item that could be utilized to package a gift. Perfume boxes themselves are often used as gift packaging or are the gift itself. Both boxes are candidates. The left box is the Hermès one, the right is the Chanel one. Both are rectangular boxes that could serve as packaging. I'll include both boxes as they are the items in the image that could be used for gift packaging.

Response

<bbox>[213, 7, 476, 753]</bbox><mask><|mask|></mask>

<bbox>[587, 79, 785, 829]</bbox><mask><|mask|></mask>

Instruction

What item would be useful for maintaining a well-groomed lawn, particularly when it comes to regularly cutting the grass?

Model Output

Thinking

The image shows a riding lawn mower, which is the primary tool for cutting grass. The mower is orange and black, with a seat, steering wheel, and a cutting deck at the front. The cutting deck is where the grass is actually cut. The question asks for the item useful for maintaining a well-groomed lawn, particularly for cutting grass. The most direct answer is the mower itself, or more specifically, the cutting deck. However, the question is a bit general. Let's look at the options. The mower is the main object. The cutting deck is part of it. The question asks "What item..." The mower is the item. Let's localize the mower.

Response

<bbox>[129, 79, 885, 945]</bbox><mask><|mask|></mask>

Instruction

Seafood dishes often include a tangy condiment that enhances the flavor. What item in the picture could be squeezed onto the seafood as a tangy flavor enhancer?

Model Output

Thinking

Looking at the image, I see a plate of grilled shrimp/prawns. On the left side, there's a small white bowl containing a red sauce (likely a chili or tomato-based sauce). In the bottom left corner, there's a lime wedge. The question asks about a tangy flavor enhancer that can be squeezed. A lime wedge is the classic tangy citrus element that you squeeze onto seafood. The red sauce is a condiment but not typically "squeezed"—it's already in a bowl. The lime wedge is the item you squeeze.

Response

<bbox>[0, 591, 162, 868]</bbox><mask><|mask|></mask>

Object Selection in Cluttered Scenes Figure 13 shows referring requests that combine object identity with attributes or relative position in cluttered scenes. These cues select a cup, a beverage can, a book, and a cleanser among nearby objects. Specifically, positional phrases distinguish the requested cup and cleanser from other instances of the same category, and the masks delineate the selected objects.

Instruction

could you please pass me the white cup on the left

Model Output

Thinking

The image shows a blue chair seat holding several items, including a small blue bottle, a white paper cup, a white plastic cup, a black container, and a red can. The description asks for the white cup on the left. Looking at the arrangement, there is a white paper cup positioned to the left of the small blue bottle and the other white cup. This cup has a blue rim and some text on it, distinguishing it from the plain white plastic cup further to the right. Therefore, the target is the white paper cup located on the left side of the cluster of objects.

Response

<bbox>[408, 412, 497, 567]</bbox><mask><|mask|></mask>

Instruction

would you please hand me the cylindrical can of pepsi cola on the right

Model Output

Thinking

The image shows a wooden table cluttered with various items, including several beverage cans and bottles. I need to find the specific cylindrical can of Pepsi Cola located on the right side of the scene. Scanning the table from left to right, there is a red can, a blue Pepsi can, a yellow can, and then further to the right, there is another blue can with the Pepsi logo visible. This blue can is positioned towards the right side of the table, near a white mug and some other small objects. It matches the description of a cylindrical Pepsi Cola on the right.

Response

<bbox>[711, 515, 848, 760]</bbox><mask><|mask|></mask>

Instruction

can you hand me the old book with a black and white cover

Model Output

Thinking

The target object is a paperback book positioned in the lower right quadrant of the image, lying flat on the white surface. It features a distinct black and white cover design with the title "SHERLOCK HOLMES" clearly visible in large serif font. The cover art depicts a historical scene with figures and a boat, consistent with the description of an old book. This book is situated next to a yellow jar and a white bar of soap, distinguishing it from the other items like the green soap dish and the white robot device.

Response

<bbox>[416, 488, 812, 888]</bbox><mask><|mask|></mask>

Instruction

would you please hand me the facial cleanser on the left

Model Output

Thinking

The image shows a bathroom sink area with several toiletries arranged on the countertop. There are two white bottles with red caps and two dark tubes. The description asks for the facial cleanser on the left. Looking at the arrangement, there is a dark tube standing upright on the far left side of the counter, near the wall. This tube is positioned to the left of the other items, including the other dark tube and the white bottles. Therefore, the target is the dark tube located at the leftmost position among the toiletries.

Response

<bbox>[408, 558, 538, 925]</bbox><mask><|mask|></mask>

Spatial Comparison and Color Recognition Figure 14 illustrates visual latents associated with spatial comparison and color recognition. The first trace localizes chairs and bicycles before comparing their horizontal positions and selecting answer A. The second identifies a small vehicle in the wider scene and selects answer D, orange. Matching colors connect each region reference to its box and decoded mask, and an enlarged view shows the small target.

Counting with Instance References Figure 15 illustrates instance-by-instance counting with visual latents. The traces enumerate five spatially separated climbing stones and six leaves, several of which overlap, and then return the corresponding counts. Matching colors link each enumerated item to its predicted box and mask, so the region behind each counting step can be inspected.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.arxiv.org/abs/2511.21631
  2. [2]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025.arxiv.org/abs/2502.13923
  3. [3]ByteDance Seed Team. Seed2.1 officially released: Advancing AI productivity, June 2026. URL https://seed.bytedance.com/en/blog/seed2-1-officially-released-advancing-ai-productivity.
  4. [4]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. SAM 3: Segment anything with concepts. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2511.16719
  5. [5]Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal LLM’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.arxiv.org/abs/2306.15195
  6. [6]Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 9313–9332, 2024. URL https://aclanthology.org/2024.acl-long.505.
  7. [7]Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jianping Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, et al. RynnBrain: Open embodied foundation models. arXiv preprint arXiv:2602.14979, 2026. URL https://arxiv.org/abs/2602.14979.
  8. [8]DeepSeek-AI. DeepSeek-V4-Flash-Vision-Expert release: Multimodal API now live, August 2026. URL https://api-docs.deepseek.com/news/news260821/.
  9. [9]Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683, 2025. URL https://arxiv.org/abs/2505.14683.
  10. [10]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2010.11929.
  11. [11]Google DeepMind. Gemini 3.6 Flash: Model card, 2026. URL https://deepmind.google/models/model-cards/gemini-3-6-flash/.
  12. [12]Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.05243.
  13. [13]Agrim Gupta, Piotr Dollár, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. URL https://arxiv.org/abs/1908.03195.
  14. [14]Xiaoyang Han, Jianhua Li, Kewang Deng, Zukai Chen, Xuanke Shi, Sihan Wang, Boxuan Li, Linyan Wang, Siyi Xie, Xin You, et al. Vision as unified multimodal generation. arXiv preprint arXiv:2607.06560, 2026. URL https://arxiv.org/abs/2607.06560.
  15. [15]Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, and Lei Zhang. Get anything via next point prediction. arXiv preprint arXiv:2510.12798, 2025. URL https://arxiv.org/abs/2510.12798.
  16. [16]Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
  17. [17]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.arxiv.org/abs/2304.02643
  18. [18]Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024.arxiv.org/abs/2308.00692
  19. [19]Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-03: Scaling up reasoning patterns and interaction turns for visual search. arXiv preprint arXiv:2509.07969, 2025. URL https://arxiv.org/abs/2509.07969.
  20. [20]Mengcheng Lan, Chaofeng Chen, Yue Zhou, Jiaxiang Xu, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Text4Seg: Reimagining image segmentation as text generation. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2410.09855
  21. [21]Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen, and Zicheng Liu. Latent visual reasoning. In International Conference on Learning Representations, 2026.arxiv.org/abs/2509.24251
  22. [22]Kehan Li, Bohan Hou, Minghao Zhu, Tianyi Zhang, Zesen Cheng, Zhikai Wang, Sicong Leng, Xin Li, Xiao Lin, Biying Yao, et al. RynnBrain 1.1: Towards more capable and generalizable embodied foundation model. arXiv preprint arXiv:2607.17977, 2026. URL https://arxiv.org/abs/2607.17977.
  23. [23]Yijun Liang, Ming Li, Chenrui Fan, Ziyue Li, Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj, Jiuhai Chen, Fuxiao Liu, and Tianyi Zhou. ColorBench: Can VLMs see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness. arXiv preprint arXiv:2504.10514, 2025. URL https://arxiv.org/abs/2504.10514.
  24. [24]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014. URL https://arxiv.org/abs/1405.0312.
  25. [25]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In Proceedings of the European Conference on Computer Vision, pages 38–55. Springer, 2024.arxiv.org/abs/2303.05499
  26. [26]Ye Liu, Zongyang Ma, Junfu Pu, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. UniPixel: Unified object referring and segmentation for pixel-level visual reasoning. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2509.18094
  27. [27]Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-Zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025. URL https://arxiv.org/abs/2503.06520.
  28. [28]Yuhao Lu, Yixuan Fan, Beixing Deng, Fangfu Liu, Yali Li, and Shengjin Wang. VL-Grasp: a 6-DoF interactive grasp policy for language-oriented objects in cluttered indoor scenes. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 976–983, 2023. URL https://arxiv.org/abs/2308.00640.
  29. [29]Tianren Ma, Lingxi Xie, Yunjie Tian, Boyu Yang, and Qixiang Ye. ClawMachine: Learning to fetch visual tokens for referential comprehension. arXiv preprint arXiv:2406.11327, 2024.arxiv.org/abs/2406.11327
  30. [30]Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. URL https://arxiv.org/abs/1511.02283.
  31. [31]Meituan LongCat Team, Bin Xiao, Chao Wang, et al. LongCat-Next: Lexicalizing modalities as discrete tokens. arXiv preprint arXiv:2603.27538, 2026.
  32. [32]Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. arXiv preprint arXiv:1606.04797, 2016. URL https://arxiv.org/abs/1606.04797.
  33. [33]Varun K. Nagaraja, Vlad I. Morariu, and Larry S. Davis. Modeling context between objects for referring expression understanding. In European Conference on Computer Vision, 2016. URL https://arxiv.org/abs/1608.00525.
  34. [34]OpenAI. Thinking with images, April 2025. URL https://openai.com/index/thinking-with-images/.
  35. [35]OpenAI. Previewing GPT-5.6 Sol, September 2026. URL https://openai.com/index/previewing-gpt-5-6-sol/.
  36. [36]Qwen Team. Qwen3.8, 2026. URL https://qwen.ai/blog?id=qwen3.8.
  37. [37]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
  38. [38]Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. PACO: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. URL https://arxiv.org/abs/2301.01795.
  39. [39]Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. GLaMM: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.arxiv.org/abs/2311.03356
  40. [40]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. URL https://arxiv.org/abs/2408.00714.
  41. [41]Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. PixelLM: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024.arxiv.org/abs/2312.02228
  42. [42]Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual CoT: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2403.16999
  43. [43]Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. URL https://openaccess.thecvf.com/content_ICCV_2019/html/Shao_Objects365_A_Large-Scale_High-Quality_Dataset_for_Object_Detection_ICCV_2019_paper.html.DOI
  44. [44]Yongyi Su, Haojie Zhang, Shijie Li, Nanqing Liu, Jingyi Liao, Junyi Pan, Yuan Liu, Xiaofen Xing, Chong Sun, Chen Li, et al. Patch-as-decodable-token: Towards unified multi-modal vision tasks in MLLMs. arXiv preprint arXiv:2510.01954, 2025.arxiv.org/abs/2510.01954
  45. [45]Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. CountQA: How well do MLLMs count in the wild. arXiv preprint arXiv:2508.06585, 2025. URL https://arxiv.org/abs/2508.06585.
  46. [46]Hao Tang, Chenwei Xie, Haiyang Wang, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. UFO: A unified approach to fine-grained visual perception via open-ended language interface. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2503.01342
  47. [47]Shengbang Tong, Ellis Brown, Pengchao Wu, Sanghyun Woo, Manoj Middelpogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. arXiv preprint arXiv:2406.16860, 2024. URL https://arxiv.org/abs/2406.16860.
  48. [48]Hao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, and Xiaodan Liang. X-SAM: From segment anything to any segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 26187–26196, 2026.arxiv.org/abs/2508.04655
  49. [49]Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel Reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning. arXiv preprint arXiv:2505.15966, 2025. URL https://arxiv.org/abs/2505.15966.
  50. [50]Jingchao Wang, Zhijian Wu, Dingjiang Huang, Yefeng Zheng, and Hong Wang. Unlocking the potential of MLLMs in referring expression segmentation via a light-weight mask decoder. arXiv preprint arXiv:2508.04107, 2025.arxiv.org/abs/2508.04107
  51. [51]Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, Jan Kautz, Lei Zhang, and Zhiding Yu. LocateAnything: Fast and high-quality vision-language grounding with parallel box decoding. arXiv preprint arXiv:2605.27365, 2026. URL https://arxiv.org/abs/2605.27365.
  52. [52]Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. CogVLM: Visual expert for pretrained language models. In Advances in Neural Information Processing Systems, volume 37, pages 121475–121499, 2024.arxiv.org/abs/2311.03079
  53. [53]Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025.arxiv.org/abs/2508.18265
  54. [54]Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. arXiv preprint arXiv:2408.15556, 2024. URL https://arxiv.org/abs/2408.15556.
  55. [55]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2201.11903
  56. [56]Zhixiang Wei, Yi Li, Zhehan Kan, Xinghua Jiang, Zuwei Long, Shifeng Liu, Hongze Shen, Wei Liu, Xiaoyu Tan, Haojia Lin, et al. Youtu-VL: Unleashing visual potential via unified vision-language supervision. arXiv preprint arXiv:2601.19798, 2026.arxiv.org/abs/2601.19798
  57. [57]Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. PhraseCut: Language-based image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. URL https://arxiv.org/abs/2008.01187.
  58. [58]Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Ping Luo, Yu Qiao, and Jifeng Dai. VisionLLM v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2406.08394
  59. [59]Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal LLMs. arXiv preprint arXiv:2312.14135, 2023. URL https://arxiv.org/abs/2312.14135.
  60. [60]Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSVA: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024.arxiv.org/abs/2312.10103
  61. [61]Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4818–4829, 2024.arxiv.org/abs/2311.06242
  62. [62]Xiaomi LLM-Core Team. MiMo-VL technical report. arXiv preprint arXiv:2506.03569, 2025.
  63. [63]Xiaomi MiMo Team. MiMo-V2.5, June 2026. URL https://mimo.mi.com/models/zh-CN/mimo-v2.5.
  64. [64]Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. LISA++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240, 2023.arxiv.org/abs/2312.17240
  65. [65]Zeyuan Yang, Xueyang Yu, Delin Chen, Maohao Shen, and Chuang Gan. Machine mental imagery: Empower multimodal reasoning with latent visual tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 33510–33520, 2026.arxiv.org/abs/2506.17218
  66. [66]Yuan Yao, Qiushi Yang, Humen Zhong, Jiangning Wei, Yifang Men, Shuai Bai, Miaomiao Cui, and Zhibo Yang. Qwen3-VL-Seg: Unlocking open-world referring segmentation with vision-language grounding. arXiv preprint arXiv:2605.07141, 2026.arxiv.org/abs/2605.07141
  67. [67]Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision, 2016. URL https://arxiv.org/abs/1608.00272.
  68. [68]Qize Yu, Lianrui Fan, Boyu Chen, Jiaqi Liang, Xini Ding, Yue Chen, Zetian Song, Yuran Wang, Yi Zou, Kaixuan Wang, et al. GroundingPI: A grounding foundation model towards physical intelligence with visual primitives. arXiv preprint arXiv:2609.39601, 2026. URL https://arxiv.org/abs/2609.39601.
  69. [69]Qize Yu, Lianrui Fan, Bowen Ping, Xini Ding, Zetian Song, Junbo Niu, Kaixuan Wang, Tianxiang Chen, Yue Chen, Minghua He, et al. GroundAnything: Reconciling parallel decoding with precise visual grounding at flash speed. arXiv preprint arXiv:2609.39600, 2026. URL https://arxiv.org/abs/2609.39600.
  70. [70]Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, et al. MiniCPM-V 4.5: Cooking efficient MLLMs via architecture, data, and training recipe. arXiv preprint arXiv:2509.18154, 2025.arxiv.org/abs/2509.18154
  71. [71]Haobo Yuan, Xiangtai Li, Tao Zhang, Yueyi Sun, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, et al. Sa2VA: Marrying SAM 2 with LLaVA for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025.DOI
  72. [72]Yuqian Yuan, Wentong Li, Zhaocheng Li, Yutong Lin, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang, and Wenqiao Zhang. InstructSAM: Segment any instance with any instructions. arXiv preprint arXiv:2605.26102, 2026. URL https://arxiv.org/abs/2605.26102.
  73. [73]Z.AI. GLM-5.3-Flash: Overview, 2026. URL https://docs.z.ai/guides/vlm/glm-5.3-flash.
  74. [74]Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973, 2024.arxiv.org/abs/2404.07973
  75. [75]Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. OMG-LLaVA: Bridging image-level, object-level, pixel-level reasoning and understanding. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.19389
  76. [76]Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaowen Zhang, Yang Liu, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, et al. Adaptive Chain-of-Focus reasoning via dynamic visual search and zooming for efficient VLMs. arXiv preprint arXiv:2505.15436, 2025. URL https://arxiv.org/abs/2505.15436.
  77. [77]Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. MME-RealWorld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024. URL https://arxiv.org/abs/2408.13257.
  78. [78]Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. DeepEyes: Incentivizing “thinking with images” via reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2505.14362
  79. [79]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.arxiv.org/abs/2504.10479

Paper details

vlnc:W_2vp6x6xj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents