WeVisOne-VL: Unified Visual Perception from Objects to Pixels

Project Page: https://tencent.github.io/WeVisOne-VL

Code: https://github.com/Tencent/WeVisOne-VL

Weights: https://huggingface.co/collections/tencent/wevisone-vl

Introduction

Perceiving and understanding the visual world has long been a central pursuit of computer vision [1, 4, 25, 32, 45, 46, 70, 94]. An intelligent visual system must be able to recognize what is present in an image, identify which object a user is referring to, and determine where these objects exist. Such capabilities establish the foundation for a wide range of real-world applications, such as content creation [65], industrial inspection [37], autonomous driving, and embodied intelligence [18].

Over the past decade, computer vision has made remarkable progress toward this goal. From conventional detection and segmentation models (e.g., YOLO [86] and DETR [4]), to open-world perception models (e.g., Grounding-DINO [67] and SAM [45]), and more recently to large Vision-Language Models (VLMs) [1], this field is being driven toward more unified, general, and powerful visual systems. This evolution has substantially broadened the concepts that a model can recognize and the instructions that it can follow. However, when both performance and efficiency are considered, current models are still far from satisfactory for real-world deployment. They often hallucinate non-existent objects, misinterpret user intent when encountering unfamiliar object descriptions, or overlook non-salient instances, leading to incomplete recall.

In this report, we introduce WeVisOne-VL (WeChat Vision All-in-One), a frontier vision-language model that unifies object-level and pixel-level visual perception through a single natural-language interface. The model casts diverse visual tasks as structured next-token prediction and incorporates a lightweight pixel decoder for generating dense outputs. Built on Qwen3-VL [1], we instantiate our models at two scales: 2B and 4B. We train the models on approximately 100M images with fine-grain annotations using a multi-stage curriculum. The key designs of WeVisOne-VL are summarized as follows.

Data Engine. We develop a unified data engine that connects semantic discovery with instance-aligned supervision for object detection, referring localization, segmentation, and matting. To guide targeted acquisition and task-specific annotation, the engine organizes category concepts and compositional referring expressions in a shared knowledge graph. It then combines complementary localization experts to generate spatial candidates and consolidates their predictions through visual-language verification of query–instance correspondence. Building on these verified localizations, the engine generates binary masks to extend supervision to the pixel level while preserving target identity across tasks, with manual spot checks for additional quality control. For matting, it further leverages expert predictions and curated data to support a dedicated annotator that provides continuous alpha supervision. Finally, the engine uses reviewed model errors to guide subsequent acquisition, annotation correction, and hard-example sampling, enabling iterative refinement of the training data.

Model Design. WeVisOne-VL consists of a VLM backbone and a lightweight pixel decoder. Built on Qwen3-VL [1], WeVisOne-VL casts a broad range of vision tasks as structured next-token prediction, generating object labels and spatial coordinates autoregressively. For compact and efficient output, we introduce a group of dedicated tokens (from <O> to <1000>) to encode normalized coordinates [41]. For dense prediction tasks, the model generates a special <OBJ> token after each predicted bounding box, whose hidden state serves as a latent representation of the corresponding instance. The pixel decoder combines this instance representation with multi-level features from the shared ViT to produce predictions over the full image canvas. In this way, WeVisOne-VL supports both discrete and dense predictions without relying on a separate large segmentation model [48, 62, 68, 122].

Training Pipeline. We train WeVisOne-VL in a curriculum pipeline consisting of three-stage Supervised Fine-Tuning (SFT) followed by mixed-task Reinforcement Learning (RL). Stage 1 establishes task alignment, teaching the model to follow task instructions and produce well-formed structured outputs. Stage 2 introduces high-resolution and dense perception data to improve the recall of small and crowded objects. Stage 3 further incorporates pixel-level tasks, extending the model from discrete localization to dense prediction. For RL, we first select samples on which the SFT model is most uncertain, then apply Group Relative Policy Optimization (GRPO) over the mixture of tasks. This on-policy optimization effectively improves the localization accuracy, while suppressing repeated or hallucinated predictions.

Across multiple visual benchmarks, WeVisOne-VL demonstrates highly competitive performance against both state-of-the-art general-purpose models and specialized vision systems. It exhibits consistently strong capabilities on both object-level and pixel-level tasks, including object detection, fine-grained referring, text recognition and language-guided pixel prediction. In addition, it also shows strong potential in challenging real-world scenarios, such as maintaining high recall in crowded scenes and correctly rejecting queries when the target is absent. By unifying these capabilities into a single model, we offer a powerful, efficient, and versatile vision-language foundation for the open-source community, accelerating the development of truly practical and reliable visual systems.

WeVisOne-VL Data Engine

Data construction for unified visual perception must support both category-level recognition and context-dependent target selection, while preserving instance associations across language and spatial supervision. To this end, we build the WeVisOne-VL Data Engine (WDE). WDE draws on GRIT [78], W3D [90], LAION [49], SA-1B [45], and several task-specific datasets, and its annotation inventory comprises over 100 million images and approximately 3.6 billion bounding-box annotations.

Figure 2 shows an overview of WDE. We first organize category concepts and referring expressions in a shared knowledge graph, which guides targeted image acquisition (§ 2.1). The acquired image–query pairs then go through task-specific localization and semantic verification, which establish the correspondence between each query and its target instances (§ 2.2 and § 2.3). We keep these correspondences when generating binary masks (§ 2.4), and further extend the instance annotations to continuous alpha supervision with a dedicated matting pipeline (§ 2.6). Finally, model errors reviewed after each training round guide subsequent acquisition and annotation refinement (§ 2.7). Below, we describe each component in detail.

Concept Discovery and Targeted Acquisition

Semantic discovery produces two complementary annotations: category-level concepts and compositional referring expressions. For detection, we use an optimized Tokenize Anything model [76] to discover object categories. For referring tasks, we prompt Qwen3-VL [1] to generate referring expressions that specify targets through attributes, states, actions, and relations. Category concepts support category-based localization, while referring expressions support selecting particular instances from their context. Both serve as annotation queries, and their correspondence to image regions is established in the subsequent localization stages.

We organize both category concepts and referring expressions in a shared knowledge graph. The graph is initialized from resources including Wikipedia² and WordNet [73], and represents canonical category names, aliases, taxonomic relations, and whole–part relations. A candidate term is first linked to existing nodes through text retrieval and vision-language verification, and is considered for addition to the graph only when it is supported by stable visual evidence. During normalization, we merge equivalent category terms while keeping the descriptive constraints that determine the scope of referring expressions.

This semantic organization also supports targeted acquisition. Concept names and aliases serve as textual retrieval queries, verified instances provide visual retrieval cues, and historical errors indicate where additional coverage is needed. We set sampling priorities according to concept representation, image domain, target scale, and challenging conditions such as occlusion and dense object arrangements, and confusable categories and recurring failures in interpreting referring expressions further inform acquisition. In this way, annotation effort is directed toward both semantic gaps and visual conditions that remain difficult for the current model. The acquired image–query pairs are then sent to the detection or referring annotation pipeline.

Detection Annotation

Conventional object detectors excel at detecting densely distributed objects but struggle to recognize long-tail categories and open-world concepts. By contrast, VLMs offer strong recognition capabilities but often exhibit low recall in dense object scenarios. Therefore, for each object category, we first utilize detectors to generate exhaustive proposals, and then use a VLM for post verification. Specifically, we obtain proposals in parallel from SAM3 [5] and an in-house open-vocabulary detector based on a DINO-style architecture [123], which we refine iteratively. For each image–query pair, the predictions of both models are merged into a shared candidate pool. This design reduces our reliance on any single proposal source and uses the variation across detectors to pursue broader instance coverage.

For detection, we obtain proposals in parallel from two models: SAM3 [5] and an in-house open-vocabulary detector based on a DINO-style architecture [123], which we refine iteratively. For each image–query pair, the predictions of both models are merged into a shared candidate pool. This design reduces our reliance on any single proposal source and uses the variation across detectors to pursue broader instance coverage.

We then use Qwen3-VL [1] to verify whether each proposed region matches the category query given the image content. Based on its judgments, we select candidates, remove duplicates, and resolve conflicting predictions. For the retained instances, we keep the bounding boxes produced by the localization models. These boxes serve as detection supervision and also define the target instances used in the subsequent segmentation annotation.

Referring Annotation

Referring annotation requires the predicted regions to satisfy the compositional meaning of an expression, including the conditions that distinguish the intended targets within a scene. We build this stage on the SAM3 Agent framework [5] and add Rex-Omni [41] as an additional localization expert. Within the same expression-conditioned workflow, SAM3 and Rex-Omni both propose candidate regions, providing complementary localization evidence for the same expression.

Qwen3-VL then consolidates these proposals by checking whether each one matches the expression in its image context. The check covers the attributes, states, actions, and relations stated in the expression, and distinguishes the requested targets from objects that are mentioned only as contextual references. Whether an expression refers to a whole object or one of its parts is likewise determined by the expression itself. Conflicting proposals are resolved through the same semantic check. The accepted results bind each expression to an individual target or a target set, so that the annotation preserves the selection constraints of the query.

Mask Annotation

For each localized instance, we use SAM2 [85] to predict a binary mask of its visible region. Each mask inherits the query of the box it is generated from. Each instance thus has both a box and a mask under the same query. With category queries, the boxes supervise detection and the masks supervise category-conditioned segmentation. With referring expressions, the boxes supervise referring localization and the masks supervise referring segmentation.

We manually spot-check a subset of these annotations, checking whether each mask and box are linked to the correct instance and whether the masks are accurate. Defects found in this check are corrected through the refinement process in § 2.7. We also record the source of every annotation, so that affected labels can be traced and regenerated.

Hard-Example Construction

Some distinctions may not be adequately represented by broad semantic coverage alone, and we construct hard examples to focus on them. Candidate queries are derived from confusable concepts in the knowledge graph, visually similar categories, controlled changes to descriptive conditions, and observed model failures. These sources provide plausible alternatives that can only be handled by precisely interpreting both the category and the contextual constraints of a query.

A query is negative only when no object in the image satisfies all of its conditions, and this cannot be decided from the query or the detector output alone. For example, changing “red car” to “blue car” does not mean the image contains no blue car, and a detector that returns nothing for a query may simply have missed the target. Therefore, we review each candidate negative against the whole image, checking the attributes, actions, and relations in the query. If a matching target turns out to exist, the query is sent back to positive annotation.

Whether an example is hard is determined by the behavior of the current model. A reviewed negative becomes a hard negative when the model nevertheless predicts a target region for it. Such examples carry an empty target set, so they teach the model to reject the query without introducing any box or pixel targets. We retain the erroneous predictions as evidence of the underlying confusion. Hard positives, in turn, capture missed targets, incorrect referent selection, and difficult localization under crowding or occlusion. We keep their reliable annotations, so that these examples strengthen the model’s sensitivity to true targets while hard negatives teach it to reject. Sample difficulty is reassessed for each model version, and the sampling emphasis is adjusted toward persistent and newly emerging failure modes.

Matting Data Engine

We build a dedicated matting pipeline within WDE to provide WeVisOne-VL with alpha supervision across the diverse objects and scenes in our instance corpus. On the one hand, high-quality matting annotations are expensive to acquire, which limits their scale and coverage. Compositing foregrounds onto new backgrounds can enlarge the training set, but mismatched lighting and unnatural boundary transitions limit how well models trained on these images generalize to real scenes [56, 116]. On the other hand, our instance annotations already identify each target and its spatial extent, but their binary masks lack the continuous alpha values needed for detailed boundary supervision [116]. Extending them to matting therefore requires recovering fine details while preserving the target specified by each description, even when the reference mask has a coarse or inaccurate boundary. To this end, we combine expert-assisted annotation, dedicated annotator training, and corpus re-annotation, producing alpha labels that remain associated with the original instance descriptions and bounding boxes.

Expert-assisted Annotation. For each instance with binary mask $S$, we obtain two alpha candidates on the same crop: the soft foreground mask $\alpha^{B}$ from BiRefNet [125] and the alpha matte $\alpha^{M}$ from MatAnyone2 [116]. We initialize MatAnyone2 with $S$ after dilation and erosion, repeat the crop as a short frame sequence, and take the final alpha prediction after warmup. BiRefNet, in contrast, predicts foreground from the RGB crop without any instance prompt, so its output may correspond to a different object or cover only part of the intended target. We therefore use its binarized foreground $B = \mathbf{1}[\alpha^{B} \ge\tau_{\alpha}]$ only to check spatial consistency with $S$. Specifically, we set $\tau_\alpha= 0.5$ and accept a BiRefNet candidate only if all of the following criteria hold: $|B| \ge8$, $\operatorname{IoU}(B,S) \ge0.5$, coverage $|B \cap S|/|S| \ge0.5$, foreground precision $|B \cap S|/|B| \ge0.8$, area ratio $|B|/|S| \in[0.5,2]$, bounding-box IoU of at least 0.7, and a centroid offset no greater than 0.15 times the box diagonal of $S$. For an accepted candidate, we further retain only the connected components of $\alpha^B \ge0.01$ that have at least 10% of their area overlapping a slightly dilated $S$. The alpha values within these components are preserved without clipping to $S$, allowing fine structures to extend beyond the coarse boundary. If the BiRefNet candidate is rejected, or fewer than eight foreground pixels remain at threshold $\tau_\alpha$ after filtering, we fall back to $\alpha^M$ when $\operatorname{IoU}(\mathbb{1}[\alpha^M \ge\tau_\alpha],S) \ge0.9$; otherwise, we retain $S$. We record the selected source for each instance, together with any MatAnyone2 fallback to $S$ caused by small crops or inference failures, so that model predictions can be distinguished from binary fallbacks.

Matting Annotator Training. We train a dedicated matting annotator by sampling instances with initial labels and high-quality matting data in equal proportions, combining broad instance coverage with detailed alpha supervision. Inspired by SAM2Matting [91], we adopt a frozen segmentation backbone with a trainable matting head. Specifically, we augment a frozen SAM2.1-Base+ model [85] with a new lightweight alpha refinement head. The head fuses multi-scale image features, prompt-mask guidance injected at each decoding stage, and local detail features from a shallow RGB stem, using top-down decoding to directly predict an alpha matte. During training, prompt masks are obtained by binarizing the target alpha, taking the instance segmentation mask, randomly perturbing these masks, or segmenting the target with box-prompted SAM2. The objective combines multi-scale $L_1$ and Laplacian alpha losses with foreground/background supervision and a prompt-consistency term applied only to unperturbed object-level prompts. Additional transition-region losses are applied to samples with continuous alpha labels, excluding small targets.

Corpus Re-annotation. Finally, we apply the trained annotator to all annotated instances in the training corpus, using the existing instance annotations to specify each target. Its predictions replace the initial labels and serve as the final alpha pseudo-labels for the matting training described in § 4. Since each matte remains linked to its original instance description and bounding box, language grounding and matting are supervised on the same target.

Training Feedback and Data Refinement

After each training round, we analyze model errors to guide both data acquisition and annotation refinement. We organize the errors by concept, image domain, target scale, occlusion, task, and failure type, and apply clustering and deduplication to select representative cases for review. This concentrates annotation effort on recurring failures and coverage gaps.

The review distinguishes errors in model prediction from errors in supervision. If the annotation is reliable, we keep the example unchanged; it may receive greater sampling emphasis or serve as a retrieval seed for related images. If the review identifies an annotation defect, we correct or regenerate the affected supervision before including it in a subsequent data version. In other words, model disagreement only triggers investigation, and annotations are changed only after review.

Different failure modes lead to different updates. Missed targets guide additional acquisition and hard-positive sampling. Referent confusion motivates instance contrasts and more descriptive queries, while false positive responses contribute to hard-negative construction. Defects in boxes, masks, or cross-task instance associations are returned to the corresponding annotation stage for correction.

Each data release records the annotation provenance, the model version that exposed each error, and the associated review and revision history. Unresolved cases are excluded from training. Original images, their identified near duplicates, and the derived task records stay within the same data partition across updates. Reviewed feedback data can enter subsequent training versions, whereas a fixed monitoring set is reserved for internal regression and excluded from both training and threshold calibration. Formal benchmarks are kept isolated from training and from feedback-driven data construction, so that evaluation remains independent as WDE evolves.

Model Design

WeVisOne-VL is built upon Qwen3-VL [1] with two key extensions. (1) Task formulation: it introduces dedicated tokens for normalized coordinates and task-specific tags, yielding a compact and efficient output format. (2) Model architecture: it incorporates a lightweight decoder that transforms discrete token representations into dense predictions for segmentation and matting. Below, we provide the details of these two designs.

Task Formulation

We formulate all visual tasks as structured next-token prediction. Inspired by Rex-Omni [41] and LocateAnything [104], we introduce a group of dedicated tokens from <0> to <1000> to represent normalized coordinates, enabling higher token efficiency. As shown in Figure 3, WeVisOne-VL localizes queried targets by sequentially generating object references and spatial coordinates. For tasks requiring bounding boxes, we encode the spatial coordinates as $[x_0, y_0, x_1, y_1]$. For tasks requiring point outputs, we use $[x_0, y_0]$ as the coordinates. If a queried target is absent from the image, its coordinate field is filled by None. For pixel-level tasks such as segmentation and matting, WeVisOne-VL additionally generates an <OBJ> token immediately after each box-coordinate quartet, inside the same <box> field, to provide the target representation for pixel decoding.

Model Architecture

Given an image and an instruction, WeVisOne-VL first encodes the image with the Qwen3-VL vision encoder and autoregressively generates the structured response described above. For pixel-level tasks, it invokes a lightweight decoder for each generated <OBJ> token. The decoder takes the token’s hidden state, its predicted box, and multi-level features from the Qwen3-VL vision encoder. The decoder draws image context from the shared visual features and spatial guidance from the predicted boxes, eliminating the need for a separate large segmentation model such as SAM [45].

The decoder consists of three components: a visual feature adapter that organizes intermediate ViT features into a shared multi-level memory, a deformable query decoder that combines each <OBJ> representation with its box prior, and an instance-conditioned dense prediction stage that converts the updated query and visual features into a full-image prediction. Segmentation and matting share these components, while matting uses an additional lightweight refinement module for fine boundaries and fractional opacity. The entire decoder, including the refinement module, contains about 9.6M parameters.

Visual Feature Adapter

To preserve fine-grained visual information for dense prediction, we extract $L$ pre-merging features $\{F_l \in\mathbb{R}^{C \times H_p \times W_p}\}_{l=1}^{L}$ from the intermediate ViT layers, serving as multi-level visual features. For each level $l$, the features are mapped to a $d$-dimensional decoder space and combined with positional and level information:

where $W_l \in\mathbb{R}^{d \times C}$ is a learnable projection, $P_{\mathrm{2D}} \in\mathbb{R}^{d \times H_p \times W_p}$ is the 2D positional encoding, and $e_l \in\mathbb{R}^{d}$ is a learnable level embedding. The visual feature adapter is composed of $T$ blocks. Given the input $X_l^{(t)}$, the $t$-th block computes as follows:

The final outputs $\{X_l^{(T)}\}_{l=1}^{L}$ across different levels are then flattened along the spatial dimensions and concatenated to derive the visual memory $\mathcal{M} \in\mathbb{R}^{LH_pW_p \times d}$.

Box-guided Query Decoder

Our query decoder follows the design of Deformable DETR [129], but differs in the initialization of instance queries: instead of using static learned embeddings, our queries are instantiated directly from the VLM outputs. Consider a response containing $N$ predicted instances, each represented by a bounding box followed by an <OBJ> token. For the $i$-th instance, let $h_i$ be the hidden state of its <OBJ> token and $b_i = [x_{i1}, y_{i1}, x_{i2}, y_{i2}]$ its normalized box. We initialize the content query as $q_i = W_h h_i$ and the positional query as $p_i = \operatorname{MLP}(\phi(b_i))$, where $\phi$ denotes Fourier encoding.

Our decoder consists of six deformable blocks, each containing a self-attention module, a multi-scale deformable cross-attention module, and a feed-forward network. In the self-attention module, the $N$ instance queries interact with one another, where $q_i + p_i$ serves as both the query and the key, and $q_i$ serves as the value. In cross-attention, each instance query aggregates visual features from the memory $M$ through multi-scale deformable attention conditioned on its reference box $r_i = \left[\frac{x_{i1}+x_{i2}}{2}, \frac{y_{i1}+y_{i2}}{2}, x_{i2}-x_{i1}, y_{i2}-y_{i1}\right]$.

Instance-Conditioned Dense Prediction

Given the multi-level visual features from the visual feature adapter and the query embeddings from the box-guided query decoder, this stage fuses the visual features, predicts an initial mask for each instance, and refines it across multiple scales.

Multi-level Pixel Fusion. Deep features encode high-level semantic information, while shallow features preserve fine-grained spatial details. To combine visual information from different levels, we first fuse the adapted visual features $\left\{X_l^{(T)}\right\}_{l=1}^{L}$ from top to bottom:

To recover color, texture, and edge details that may be lost in the fused visual feature, we additionally resize the input image $I$ to $1024 \times1024$ and feed it into a two-layer CNN to extract a shallow feature map. Then, we upsample $G_1$ to the same resolution $512 \times512$ and combine the two features by element-wise addition:

Dynamic Mask Head. Let $\tilde{q}_i$ denote the query representation of the $i$-th instance derived from the box-guided query decoder. We first apply an MLP to $\tilde{q}_i$ to generate an instance-aware dynamic kernel, which is then applied to $P$ to produce pixel-wise logits:

Coarse-to-fine Refinement. A single mask prediction is often insufficient for thin structures and translucent boundaries. Starting from $M_i^{(0)} = M_i$, we progressively refine the mask through three upsampling stages ($j = 1,2,3$). At stage $j$, we upsample the previous stage feature $F_i^{(j-1)}$ and concatenate it with the resized image $I$, pixel feature $P$, and previous mask predictions $C_i^{(j)}$:

where $\operatorname{Up}$ denotes transposed convolution, $\operatorname{Interp}_j$ denotes bilinear interpolation to the resolution of stage $j$, and $\mathbin{\|}$ denotes channel concatenation. We then modulate $U_i^{(j)}$ with the instance query $\tilde{q}_i$ via channel-wise scale and shift, and pass it through residual convolutional blocks $f_{\mathrm{rc}}^{(j)}$ to predict a logit residual:

Finally, the output prediction is obtained by $A_i = \operatorname{Sigmoid}\left(M_i^{(3)}\right)$.

Training Pipeline

We train WeVisOne-VL with a three-stage Supervised Fine-Tuning (SFT) curriculum followed by mixed-task Reinforcement Learning (RL). The curriculum first establishes a shared output interface, then improves high-resolution and dense perception, and finally connects autoregressive localization with pixel-level prediction. After SFT, RL directly optimizes the quality of the complete prediction rather than the likelihood of individual output tokens. The data composition across the training pipeline is summarized in Table 1.

Stage	Samples	Task or data-source distribution
SFT Stage 1	90.2M samples	Detection 67.2%; Referring 20.1%; Counting 2.7%; Pointing 3.1%; OCR 2.3%; Caption 3.0%; Segmentation 1.7%
SFT Stage 2	40.5M samples	Detection 66.4%; Referring 18.8%; Counting 3.0%; Pointing 2.2%; OCR 1.7%; Caption 2.2%; Segmentation 5.7%
Segmentation Warm-up	250M samples	Public 37.3%; WDE 62.7%
SFT Stage 3	41.8M samples	Detection 37.3%; Referring 6.2%; Counting 1.4%; Pointing 1.0%; OCR 0.7%; Caption 1.0%; Segmentation 52.4%
Matting Fine-tuning	130M samples	Public 0.2%; WDE 99.8%
RL	178K samples	Detection 44.0%; Dense det. 20.8%; Referring 17.5%; OCR 7.9%; Segmentation 5.8%; Pointing 4.0%

Curriculum Supervised Fine-Tuning

Stage 1: Task Alignment. This stage adapts the model to the new tokens and task-specific structured outputs. Starting from Qwen3-VL with an extended vocabulary, we freeze the vision encoder and visual-language projector and train only the language model on 90M examples covering all target tasks for 13K optimization steps. The model learns to follow different task instructions, produce the structured outputs, and return None when the target is absent. For pixel-task examples, we impose only next-token prediction supervision (NTP) without activating the pixel decoder, thereby establishing an initial correspondence between each predicted instance and its <OBJ> token. In this stage, we use 16–512 visual tokens per image and limit the context length to 4,096 tokens. Notably, early training can be dominated by extremely dense samples (e.g., containing hundreds of boxes in a single response), making the model highly prone to repetitive output collapse. To mitigate this issue, we primarily use samples with at most 50 boxes or points per category during this stage.

Stage 2: Resolution and Density Scaling. Building on Stage 1, we unfreeze the vision encoder, projector, and language model. This stage serves two purposes. First, we raise the maximum input resolution to 1,960 visual tokens, preserving the visual details for recognizing small objects. Second, we increase the maximum box/point number within one response to 500, exposing the model to crowded scenes for dense detection. The maximum context length is extended to 8,192 tokens. We train on approximately 40M examples for 9K optimization steps. During this stage, pixel-level examples remain supervised solely by NTP.

Stage 3: Alternating Token–Pixel Optimization. This stage aims to equip the VLM with pixel-level perception by establishing the <OBJ> token as the bridge between the VLM and the lightweight pixel decoder. To initialize the pixel decoder before alternating token–pixel training, we first jointly train the SFT-S2 VLM and the randomly initialized decoder on 250M segmentation samples for 100K steps with the following segmentation loss function:

where the focal term emphasizes hard pixels under foreground–background imbalance, while the Dice term directly optimizes mask overlap. After warm-up, we retain only the decoder weights and the learned <OBJ> token embedding. We attach them to the original Stage 2 checkpoint and jointly train the VLM and decoder for 9,000 steps. To preserve localization ability, we alternate NTP and pixel-level tasks at a 1:1 ratio, which can be expressed as the following loss function:

After the alternating training, we specialize the pixel decoder for continuous alpha prediction while keeping the VLM frozen. We use 130M matting samples to further finetune the decoder for 100K steps. Following standard practice in image matting [29], we combine $L_1$, gradient, and Laplacian losses to supervise alpha estimation, and additionally employ $\mathcal{L}_{\mathrm{hard}}$ to emphasize difficult foreground and background pixels:

Mixed-Task Reinforcement Learning

Although SFT provides a strong initialization, its token-level likelihood objective cannot directly optimize task rewards. We therefore perform mixed-task reinforcement learning on an uncertainty-filtered corpus.

Uncertainty-Aware Sample Selection. GRPO requires relative advantages to provide learning signals. Before RL training, we therefore use the SFT model for data sampling. For each candidate prompt $x$, we sample a group of responses and compute their corresponding task rewards. Let $\mu_x$ and $\sigma_x$ denote the mean and standard deviation of the group rewards, respectively. The uncertainty score for each sample is defined as follows:

This score favors moderately difficult samples with high reward variance, on which the model succeeds in some rollouts but fails in others. Such samples have high uncertainty and yield stronger relative-advantage signals, thereby improving the efficiency of RL training. Using this score, we select approximately 140K samples for RL training. We supplement this mixture with targeted box-supervised examples and pointing data, yielding 178K examples in total. Table 1 shows the complete task distribution. Data selection and reward details are provided in Appendices § 11.§ 3 and § 12.§ 3, respectively.

GRPO with Grounding Rewards. On the resulting training corpus, we perform GRPO using grounding rewards computed over complete structured responses. These rewards directly evaluate the grounding quality of the model predictions rather than individual tokens. For each selected image–prompt pair $x$, we sample $G = 8$ complete responses $\{o_i\}_{i=1}^{G}$ from the current policy and parse each response into a set of predicted objects. Let $\hat{B}_i$ and $B^*$ denote the predicted and ground-truth box sets, respectively. We perform one-to-one Hungarian matching between predicted and ground-truth boxes within each semantic class, and denote the set of matched pairs as $\mathcal{M}_i$. The grounding reward for response $o_i$ is defined as:

This reward measures localization quality, while penalizing both missed and duplicate objects. Cubing the IoU makes the reward more sensitive to localization improvements at high IoU values, encouraging more precise box predictions. For segmentation and matting, RL uses only box-based rewards, without evaluating predicted masks or alpha mattes. For referring segmentation, a prediction is valid only when its box is followed by the required <OBJ> token. For negative samples, the reward is 1 only when the model predicts no object.

Experiments

We evaluate WeVisOne-VL across five object-level tasks (object detection, referring object detection, object counting, object pointing, and OCR) and two pixel-level tasks (referring segmentation and image matting). At inference, we use greedy decoding with temperature 0 and limit each image to 1,960 visual tokens. We primarily compare with the Qwen series as general-purpose VLM baselines and with two representative VLM-based grounding models, Rex-Omni [41] and LocateAnything [104]. For Qwen3-VL, we evaluate the Instruct variants; for Qwen3.5, we enable thinking mode because it performs better in our evaluation.

Main Results

Main Results

Object Detection

Benchmarks. We evaluate common object detection on COCO [63], long-tailed object detection on LVIS [30], and dense and tiny-object detection on Dense200 [41] and VisDrone [19].

Metrics. Average Precision (AP) relies on calibrated confidence scores and is therefore not directly applicable to generative VLMs. Following Rex-Omni [41] and LocateAnything [104], we report F1 scores. A predicted box is regarded as a true positive if its $\operatorname{IoU}$ with an unmatched ground-truth box of the same category exceeds a threshold. Precision and recall are computed per category and averaged, and F1 is their harmonic mean. We report F1 at $\operatorname{IoU}=0.5$, at $\operatorname{IoU}=0.95$, and averaged over $\operatorname{IoU}$ thresholds from 0.5 to 0.95 with a step of 0.05 (denoted as “Mean”).

Results. As shown in Table 2, WeVisOne-VL consistently outperforms VLMs at comparable model scales in mean F1 across general, long-tailed, and dense detection benchmarks. The gains are especially pronounced on Dense200 and VisDrone, where accurate recognition must be maintained over many small and crowded instances. Its strong performance under the strict $\operatorname{IoU}=0.95$ criterion further indicates that the improvement reflects both broad object coverage and precise localization.

Table: Results on general object detection benchmarks COCO and LVIS, and dense object detection benchmarks Dense200 and VisDrone. COCO and LVIS are evaluated in a zero-shot setting. We report F1 at $\operatorname{IoU}=0.5$, $\operatorname{IoU}=0.95$, and averaged over $\operatorname{IoU}$ thresholds from 0.5 to 0.95 (Mean). Bold and <u>underline</u> denote the best and second-best results among VLMs.

Method	COCO			LVIS			Dense200			VisDrone
	0.5	0.95	Mean	0.5	0.95	Mean	0.5	0.95	Mean	0.5	0.95	Mean
Specialized Detectors
GroundingDINO-T [67]	69.8	23.0	56.6	47.7	22.7	38.8	36.9	19.7	33.1	55.2	3.9	38.5
Vision-Language Models
Qwen3-VL-2B [1]	62.9	13.1	45.2	58.5	17.8	40.8	14.7	1.8	10.0	37.0	1.5	23.1
Qwen3-VL-4B [1]	66.6	14.9	48.7	61.3	19.5	44.0	20.9	3.7	15.2	49.5	1.8	31.0
Qwen3-VL-8B [1]	65.9	14.8	47.8	59.7	19.6	42.9	19.5	2.9	13.8	47.7	2.0	29.5
Qwen3.5-4B [81]	67.6	12.9	48.2	61.1	17.8	42.2	28.4	2.3	18.9	50.3	1.3	29.9
Qwen3.5-9B [81]	68.9	13.8	49.5	62.2	19.4	43.7	26.2	2.5	18.0	51.0	1.4	31.3
Rex-Omni-3B [41]	72.0	15.9	52.9	64.3	20.7	46.9	78.4	10.3	58.3	61.6	1.5	35.8
LocateAnything-3B [104]	70.1	<u>19.3</u>	54.7	62.3	31.1	50.7	74.0	<u>18.5</u>	58.7	63.0	<u>3.2</u>	39.9
WeVisOne-VL-2B	<u>72.6</u>	18.7	<u>55.2</u>	<u>69.5</u>	27.5	<u>53.3</u>	<u>79.1</u>	14.1	<u>61.4</u>	<u>64.5</u>	2.8	<u>40.4</u>
WeVisOne-VL-4B	74.0	19.5	57.1	71.1	<u>28.8</u>	55.3	86.9	23.2	71.3	68.1	3.9	44.7

Referring Object Detection

Benchmarks. We evaluate referring object detection on the public RefCOCOg [72], HumanRef [39], Ref-Adv [17], PR-Bench [22], DVG-Bench [127], and Ref-L4 [6] benchmarks. We also include RefCOCOg-Rej [58], in which each modified expression has no matching object in the image. Furthermore, we introduce WeRef-Bench, an in-house benchmark comprising 2,880 English and Chinese queries. It covers single-target referring, multi-target referring, and absent-target queries, providing more challenging and practical evaluations than the public positive, single-target benchmarks.

Metrics. For most benchmarks, we report F1 Mean as the primary metric. For RefCOCOg-Rej, we report the Rejection Rate, where a prediction is correct only when no box is produced. For WeRef-Bench and PR-Bench, we report both F1 Mean and Rejection Rate to evaluate target-present and absent-target queries, respectively.

Results. Table 3 shows that WeVisOne-VL consistently outperforms existing VLMs across single-target, multi-target, and absent-target settings. These results demonstrate that WeVisOne-VL localizes referred objects more accurately and rejects absent-target queries more reliably.

Method	RefCOCOg			WeRef-Bench		PR-Bench		HumanRef	Ref-Adv	DVG	Ref-L4
	val	test	Rej	Mean	Rej	Mean	Rej
Specialized Detectors
GroundingDINO-T [67]	45.9	46.8	0.0	21.1	14.7	25.2	0.0	25.2	19.7	24.1	23.1
Vision-Language Models
Qwen3-VL-2B [1]	70.8	71.0	1.7	30.8	22.8	41.6	4.3	63.2	34.7	35.2	67.9
Qwen3-VL-4B [1]	74.1	74.3	18.8	46.3	61.9	49.9	10.8	69.4	44.4	42.4	72.5
Qwen3-VL-8B [1]	74.6	74.9	22.5	36.7	18.3	50.3	7.1	71.6	45.0	43.4	72.4
Qwen3.5-4B [81]	73.3	73.5	18.9	48.1	71.6	49.6	8.7	75.8	44.2	40.2	70.4
Qwen3.5-9B [81]	74.3	74.6	15.2	50.7	70.7	50.8	8.3	77.8	47.3	42.5	72.0
Rex-Omni-3B [41]	73.6	74.3	0.5	50.3	43.3	42.7	0.2	79.9	40.9	33.2	65.2
LocateAnything-3B [104]	76.7	77.6	0.6	51.7	73.0	44.5	0.2	78.7	43.5	34.7	74.3
WeVisOne-VL-2B	77.1	78.2	50.2	73.2	92.9	49.8	37.0	77.4	43.6	40.8	75.1
WeVisOne-VL-4B	80.8	81.8	67.4	78.7	96.4	54.1	52.6	80.7	50.5	45.2	78.1

Object Counting

Benchmarks. We evaluate object counting on CountQA [100], PixMo-Count [13], CLOC [51], KubriCount [66], and MixCount [20]. CountQA and PixMo-Count focus on counting in natural images, while CLOC evaluates cross-domain generalization. KubriCount stresses multi-grained target specification with controlled distractors, and MixCount targets fine-grained open-vocabulary counting in mixed-object scenes.

Metrics. For models evaluated through detection output, we use the number of predicted target boxes as the count; for models that return a count directly, we extract the numerical answer from the response. We report exact-match accuracy: the percentage of queries for which the predicted count equals the ground-truth count. Thus, an error of even one object is counted as incorrect, and higher values are better.

Results. Table 4 shows that WeVisOne-VL achieves the highest exact-match accuracy on every reported benchmark and split. It reaches 93.7%/89.6% on PixMo-Count validation/test and 42.0%/40.4% on MixCount validation/test, indicating accurate counts in both natural images and fine-grained mixed-object scenes.

Method	CountQA	PixMo-Count		CLOC		KubriCount		MixCount
		val	test	val	test	testA	testB	val	test
Specialized Detectors
Grounding DINO-T [67]	9.6	44.8	48.3	12.4	9.5	5.2	6.5	1.7	2.8
Vision-Language Models
Qwen3-VL-2B [1]	20.4	61.3	60.4	16.3	15.3	4.3	7.8	1.8	2.7
Qwen3-VL-4B [1]	29.2	69.6	65.6	19.7	18.6	7.1	12.1	3.5	4.3
Qwen3-VL-8B [1]	22.4	68.2	59.4	18.5	17.2	6.6	10.8	3.2	3.5
Qwen3.5-4B [81]	40.0	80.6	76.9	21.1	21.7	10.0	16.2	3.9	5.2
Qwen3.5-9B [81]	41.6	82.6	80.4	23.1	22.4	11.9	19.1	5.8	7.0
Rex-Omni-3B [41]	37.6	82.2	71.9	19.0	22.3	13.5	18.8	9.9	11.4
LocateAnything-3B [104]	52.4	88.7	82.8	22.8	23.9	18.5	22.4	12.5	15.2
WeVisOne-VL-2B	47.9	90.9	88.1	27.5	26.9	29.5	32.4	30.1	32.2
WeVisOne-VL-4B	57.1	93.7	89.6	28.2	28.3	37.7	40.1	42.0	40.4

Object Pointing

Benchmarks. We evaluate object pointing on two dedicated pointing benchmarks, PointBench [7] and PixMo-Points [13]. We also evaluate pointing on the detection benchmarks COCO, LVIS, Dense200, and VisDrone, following the protocol of Rex-Omni [41].

Metrics. A predicted point is correct when it falls inside the corresponding ground-truth mask (or the annotated target box where masks are unavailable). Each target contributes at most one true positive, so duplicate points reduce precision and missed targets reduce recall. We report F1@Point as the harmonic mean of precision and recall. For detection-derived benchmarks, precision and recall are first averaged across categories.

Results. As shown in Table 5, WeVisOne-VL achieves the best results on both PointBench and PixMo-Points, demonstrating its ability to point directly to target objects. On the detection-derived benchmarks, it also performs strongly across general detection benchmarks, showing that this ability extends to diverse scenes.

Method	PointBench	PixMo-Points	COCO	LVIS	Dense200	VisDrone
Qwen3-VL-2B [1]	61.2	62.1	72.7	61.9	29.5	35.7
Qwen3-VL-4B [1]	57.1	71.3	74.5	65.8	37.7	45.6
Qwen3-VL-8B [1]	62.5	72.2	77.5	67.7	42.2	46.4
Qwen3.5-4B [81]	61.5	59.2	72.7	62.4	7.4	38.3
Qwen3.5-9B [81]	66.4	62.6	74.4	64.8	10.5	40.6
Rex-Omni-3B [41]	<u>69.9</u>	72.9	80.5	70.8	82.5	58.9
LocateAnything-3B [104]	56.9	68.6	83.9	76.6	87.6	60.4
WeVisOne-VL-2B	62.9	<u>73.2</u>	85.8	<u>78.9</u>	83.2	<u>62.4</u>
WeVisOne-VL-4B	71.9	74.7	<u>85.4</u>	79.5	<u>87.0</u>	64.4

OCR

Benchmarks. We evaluate text detection and recognition on HierText [71], ICDAR2015 [43], TotalText [8], and SROIE [35]. These benchmarks cover dense document text, incidental scene text, curved text, and structured receipts.

Metrics. We formulate OCR as a grounding task in which the recognized transcription serves as the category label. A prediction is considered correct only when its text exactly matches the ground-truth transcription and its box satisfies the IoU threshold. We then compute precision, recall, and F1 using the same matching procedure as object detection, and report F1 at IoU=0.5, at IoU=0.95, and averaged over thresholds from 0.5 to 0.95.

Results. As shown in Table 6, WeVisOne-VL consistently outperforms both specialist OCR systems and general-purpose VLMs across all evaluated benchmarks. Its strong performance across dense documents, incidental and curved scene text, and structured receipts demonstrates robust end-to-end text perception under diverse layouts and visual conditions.

Method	HierText			ICDAR2015			TotalText			SROIE
	0.5	0.95	Mean	0.5	0.95	Mean	0.5	0.95	Mean	0.5	0.95	Mean
Specialist OCR Systems
PP-OCRv5 [12]	45.2	3.4	30.5	38.2	1.2	25.6	40.2	0.7	25.7	77.7	5.6	58.6
Vision-Language Models
Qwen3-VL-2B [1]	21.4	0.3	11.0	15.7	0.0	8.5	32.6	1.1	20.5	40.2	0.2	18.6
Qwen3-VL-4B [1]	39.9	0.9	22.7	52.0	0.8	30.0	48.3	3.3	33.0	69.0	1.0	40.4
Qwen3-VL-8B [1]	45.3	0.9	25.7	45.8	0.3	27.1	53.2	3.2	35.4	75.8	1.1	43.3
Qwen3.5-4B [81]	14.4	0.4	8.0	17.4	0.1	7.8	28.8	0.6	16.6	14.7	0.0	7.1
Qwen3.5-9B [81]	17.5	0.5	10.0	28.1	0.0	13.4	49.9	1.6	30.8	31.8	0.4	16.7
Rex-Omni-3B [41]	45.9	1.4	28.0	45.2	0.3	28.1	56.6	3.9	40.6	72.0	1.5	44.8
LocateAnything-3B [104]	39.2	4.1	29.4	37.8	0.3	24.1	58.9	5.1	43.3	52.8	3.4	40.3
WeVisOne-VL-2B	44.3	2.5	<u>30.5</u>	<u>65.3</u>	<u>2.0</u>	<u>42.6</u>	<u>61.5</u>	2.7	42.5	<u>85.3</u>	<u>4.4</u>	<u>62.1</u>
WeVisOne-VL-4B	51.6	5.3	39.0	69.1	2.3	47.8	70.1	6.9	52.0	87.4	7.3	68.1

Referring Segmentation

Given an image and a natural-language expression, referring segmentation predicts a binary mask of the described objects, extending localization from box-level predictions to pixel-level dense outputs.

Benchmarks. We evaluate referring segmentation on RefCOCO and RefCOCO+ [120], RefCOCOg [72], ReasonSeg [48] and Ref-ZOM [34]. ReasonSeg uses implicit queries that require visual reasoning and world knowledge. Ref-ZOM further includes expressions that refer to zero, one, or multiple targets.

Metrics. For the RefCOCO family and Ref-ZOM, we report cumulative intersection over union (cIoU) over the entire evaluation set, computed as the total intersection between predicted and ground-truth masks divided by their total union across all samples. For ReasonSeg, we report generalized IoU (gIoU) on the validation and test splits. For Ref-ZOM, which contains negative expressions that refer to no target, we additionally report the rejection rate (Rej).

Results. As shown in Table 7, WeVisOne-VL improves over all baselines across every RefCOCO-family split. It also demonstrates strong potential for reasoning segmentation even without explicit chain-of-thought (CoT) reasoning. Furthermore, the results on Ref-ZOM reveal that existing segmentation models tend to output a mask even when the target is absent, lacking the ability to reject negative expressions. In contrast, WeVisOne-VL can not only output accurate masks when the target is present, but also correctly reject the expression when it is absent.

Method	RefCOCO			RefCOCO+			RefCOCOg		ReasonSeg		Ref-ZOM
	val	testA	testB	val	testA	testB	val	test	val	test	cIoU	Rej
LISA-13B (ft) [48]	76.0	78.8	72.9	65.0	70.2	58.1	69.5	70.5	<u>65.0</u>	<u>61.3</u>	-	-
GSVA-7B [110]	77.2	78.9	73.5	65.9	69.6	59.8	72.7	73.3	50.5	-	66.6	41.9
LaSagnA-7B [106]	76.8	78.7	73.8	66.4	70.6	60.1	70.6	71.9	-	-	57.4	0.7
Text4Seg-7B [50]	79.3	81.9	76.2	72.1	77.6	66.1	72.1	73.9	-	-	54.5	25.9
VRS-HQ-7B [27]	73.5	77.5	69.5	61.7	67.6	54.3	66.7	67.5	55.2	51.7	63.0	13.9
UniPixel-7B [68]	<u>80.8</u>	<u>83.0</u>	77.4	75.3	<u>80.1</u>	70.0	76.4	<u>77.1</u>	60.5	-	62.9	0.3
Youtu-VL-4B [107]	80.7	82.0	78.4	76.2	79.6	71.4	76.5	76.6	-	-	61.0	1.3
WeVisOne-VL-2B	79.7	82.9	78.1	73.1	79.0	66.3	74.4	74.9	60.9	56.9	73.4	<u>62.4</u>
WeVisOne-VL-4B	81.4	84.5	80.1	76.5	81.1	72.8	77.7	77.8	71.2	66.8	<u>72.2</u>	86.0

Promptable Matting

Matting predicts a continuous alpha matte that preserves fine structures and partial transparency, such as hair and fur. Existing matting evaluations typically assume a salient foreground [44, 54] or provide a ground-truth mask to specify the target [91, 115]. To evaluate promptable matting in a more practical setting, we use boxes, category names, and referring expressions to specify the target object.

Benchmarks and Baselines. For box prompts, we use P3M-500-NP [54], AM-2K [56], and PPM-100 [44]. For text prompts, we use our in-house benchmark WeRefMat. For the three established benchmarks, all methods use the same ground-truth box as prompt, and all baselines are run from their official checkpoints. We evaluate three SAM2-Matting [91] variants based on SAM2.1 [85] and SAM3 [5]. For box prompts, SAM2-Matting first uses SAM2.1 or SAM3 to predict a mask from the ground-truth box and then performs matting, while WeVisOne-VL constructs a sequence prefix from the ground-truth box, generates the subsequent <OBJ> token, and passes its hidden state to the pixel decoder to predict the alpha matte. On WeRefMat, we compare with SAM3’s native text grounding and two MLLM-based localization settings, in which Qwen3-VL-8B-Instruct or Qwen3.5-9B first localizes the target and passes the resulting box to SAM2-Matting.

Metrics. We report mean absolute difference (MAD), mean squared error (MSE), gradient error (Grad), connectivity error (Conn), and sum of absolute differences (SAD) between the predicted and ground-truth alpha mattes. Following common practice, MAD and MSE are multiplied by $10^3$. Lower values indicate better performance for all five metrics.

Results. As shown in Table 8, WeVisOne-VL is competitive with task-specific matting models under box prompts and consistently improves over the SAM2-Matting baseline. On WeRefMat, it performs strongly with both category and referring prompts, showing that language-conditioned target selection and boundarysensitive transparency estimation can be handled effectively within a unified model.

Dataset	Prompt	Method	MAD$\downarrow$	MSE$\downarrow$	Grad$\downarrow$	Conn$\downarrow$	SAD$\downarrow$
P3M-500-NP	Box	Matte Anything [117]	24.4	20.4	20.9	40.4	40.9
		SAM2-Matting (SAM2.1-T) [91]	22.7	18.2	20.0	37.8	38.1
		SAM2-Matting (SAM2.1-B+) [91]	19.0	15.0	18.3	31.0	31.4
		SAM2-Matting (SAM3) [91]	15.4	11.4	17.6	25.1	25.8
		WeVisOne-VL-2B	6.6	3.1	<u>10.7</u>	10.6	11.2
		WeVisOne-VL-4B	<u>7.3</u>	<u>3.8</u>	10.5	<u>11.7</u>	<u>12.6</u>
AM-2K	Box	Matte Anything [117]	13.2	8.7	13.9	22.0	23.1
		SAM2-Matting (SAM2.1-T) [91]	11.0	6.2	21.7	18.4	18.8
		SAM2-Matting (SAM2.1-B+) [91]	11.1	6.3	20.0	18.8	19.0
		SAM2-Matting (SAM3) [91]	12.1	7.5	19.1	19.5	20.1
		WeVisOne-VL-2B	<u>5.7</u>	<u>2.2</u>	<u>8.1</u>	<u>8.9</u>	<u>9.7</u>
		WeVisOne-VL-4B	5.0	1.6	7.2	7.5	8.5
PPM-100	Box	Matte Anything [117]	10.5	6.7	66.3	100.1	102.4
		SAM2-Matting (SAM2.1-T) [91]	9.7	5.9	68.6	95.3	97.1
		SAM2-Matting (SAM2.1-B+) [91]	9.3	5.7	64.0	90.3	92.9
		SAM2-Matting (SAM3) [91]	10.8	6.9	65.9	100.6	103.2
		WeVisOne-VL-2B	5.3	1.8	48.4	45.5	49.3
		WeVisOne-VL-4B	<u>6.1</u>	<u>2.4</u>	<u>48.5</u>	<u>53.2</u>	<u>57.2</u>
WeRefMat	Category	SAM2-Matting (SAM3) [91]	70.8	64.9	45.0	245.6	241.6
		Qwen3-VL-8B + SAM2-Matting	80.7	74.4	53.6	270.5	267.1
		Qwen3.5-9B + SAM2-Matting	77.8	71.4	51.9	265.4	261.8
		WeVisOne-VL-2B	<u>17.7</u>	<u>13.4</u>	<u>35.6</u>	<u>63.6</u>	<u>65.7</u>
		WeVisOne-VL-4B	16.4	12.5	35.0	60.3	62.0
	Referring	SAM2-Matting (SAM3) [91]	51.9	50.3	34.3	162.1	162.2
		Qwen3-VL-8B + SAM2-Matting	23.0	21.1	22.6	78.3	78.1
		Qwen3.5-9B + SAM2-Matting	22.4	20.6	22.1	82.5	82.3
		WeVisOne-VL-2B	<u>15.8</u>	<u>14.2</u>	<u>19.4</u>	<u>65.3</u>	<u>66.2</u>
		WeVisOne-VL-4B	9.8	8.4	15.9	37.8	39.0

Qualitative Analysis

We present qualitative comparisons between WeVisOne-VL and representative baselines in Figure 5–Figure 12. Each figure is organized by task, and the selected examples below are discussed from top to bottom.

Object detection. Figure 5 covers degraded visibility, adjacent instances, and dense scenes. In the first example, both baselines detect only the foreground elephant, while WeVisOne-VL also recovers the distant elephants partly obscured by dust. Across the carrot and dense fruit examples, the baselines tend to merge adjacent instances into coarse-grained boxes, whereas WeVisOne-VL more often preserves instance-level localization.

Referring object detection. Figure 6 highlights five aspects of referring object detection: spatial position, world knowledge, fine-grained attributes, negative attributes, and absent-target queries. The first four examples require interpreting ordinal position, grounding a person’s name, distinguishing headphone color, and excluding people wearing sunglasses, respectively. In these cases, WeVisionOne-VL successfully selects the correct target, while the baselines select distractors or include additional people. The final query specifies a person wearing a white shirt and red shorts, but no one in the image satisfies both conditions. Both baselines still predict a box, whereas WeVisionOne-VL correctly rejects the query.

Object counting. Figure 7 shows the importance of the queried counting unit. The first query asks for cups on the left, and the second asks for pairs of footwear rather than individual shoes. The remaining queries specify green balls and fake green apples among distractors. The baselines often count objects outside the requested subset or count individual shoes instead of pairs; in the last example, Rex-Omni reports 537 rather than 42. WeVisOne-VL gives the ground-truth count in all four examples.

Object pointing. Figure 8 shows dense, relational, attribute-based, and fine-grained pointing queries. In the first example, WeVisionOne-VL places points across the heart-shaped balloons, while LocateAnything misses most instances and Rex-Omni produces redundant points on some balloons. The second query asks for two people gazing into each other’s eyes: both baselines select other people, whereas WeVisionOne-VL points to the indicated pair. In the third example, Rex-Omni points to multiple pigs and LocateAnything selects a distractor, while WeVisionOne-VL identifies the pig facing upward. In the final example, both baselines point to the handle, whereas WeVisionOne-VL places its point on the queried curved hook.

OCR. Figure 9 presents a shop sign with curved lettering, followed by a receipt and an instruction sheet with dense small text. On the shop sign, the baselines recognize some prominent words but miss the curved lettering, whereas WeVisionOne-VL localizes and transcribes the curved words. On the receipt, all methods read several lines, but WeVisionOne-VL covers more of the lower text. The difference is more pronounced on the instruction sheet, where WeVisionOne-VL localizes substantially more of the small text than either baseline.

Referring segmentation. Figure 10 compares the models on attribute, relational, multi-instance, and absent-target queries. In the first three examples, the expressions refer to the woman with gray hair in front, the dog with lowered ears, and the white-and-red vase on the wood crate. LISA and UniPixel select distractors or incomplete regions, whereas WeVisOne-VL segments the referred objects. The fourth query specifies the leftmost remote and the third remote from the right; WeVisOne-VL segments both, while LISA produces fragmented masks and UniPixel selects a different remote. The final query describes a deer beside a fire hydrant, neither of which appears in the image. Both baselines produce masks, while WeVisOne-VL returns no mask.

Box-prompted matting. Figure 11 shows five box-prompted matting examples: a rabbit, a camel, and three people. In the first two rows, WeVisOne-VL retains the rabbit’s whiskers and more of the camel’s fine coat along the boundary. The third and fifth rows highlight flyaway and curly hair strands, which appear smoother in the baseline mattes. In the fourth row, SAM2-Matting makes the apron logo transparent, leaving a hole in the subject, while WeVisOne-VL keeps the logo and apron opaque.

Text-prompted matting. Figure 12 presents category prompts for chickens and icebergs (first two examples), followed by referring prompts for the people in black and the balloon bearing “Daikin” (last two). WeVisOne-VL retains the visible chickens, gives the icebergs a more opaque matte, and mattes both intended people and the named balloon. The Qwen3.5 + SAM2-Matting pipeline selects only one of the two people in black, while SAM2-Matting (SAM3) misses them and includes several distractors in its balloon result.

Grounded caption. In Figure 13, we show how WeVisOne-VL can describe an image in detail while grounding the descriptions with bounding boxes. For example, in the baseball scene, it describes the batter’s clothing and swinging action, as well as the catcher and umpire behind him, and links these descriptions to the corresponding image regions. In the outdoor-table scene, it identifies the two people and individual items on the table. Such captions could be valuable for VLM training, as they provide not only detailed descriptions but also explicit connections between text and image regions, helping models learn to associate words and phrases with specific visual content.

Related Work

Visual Grounding. Open-vocabulary detectors [28, 61, 67] align object localization with category names and referring expressions. Grounded multimodal models then brought spatial outputs into a language interface. Kosmos-2 [78] represents text–box links with location tokens, while Florence-2 [111] casts detection, grounding, and other vision tasks as prompt-conditioned sequence generation. Molmo/PixMo [13] further showed the value of explicit point supervision for grounding and counting. More recently, Rex-Omni [41] uses quantized coordinate tokens and geometry-aware reinforcement learning for detection, referring, pointing, and OCR. LocateAnything [104] decodes whole boxes and points in parallel to improve grounding speed and precision.

Referring Segmentation. SAM [45] and SAM 2 [85] established promptable mask prediction, and SAM 3 [5] extends this line to concept-prompted detection, segmentation, and tracking. To connect language reasoning with pixels, LISA [48] maps a generated segmentation token to a mask decoder, GLaMM [84] interleaves text and grounded masks, and GSVA [110] supports multiple referents and explicit rejection of absent targets. Later work explores different output representations and task mixtures: Text4Seg [50] serializes masks as text, and UniPixel [68] integrates masks and pixel-level reasoning. Sa2VA [122] couples an MLLM with SAM 2, while Youtu-VL [107] trains visual and language tokens under unified autoregressive supervision.

Instance Matting. Image matting estimates continuous opacity for fine structures and translucent boundaries. Automatic methods such as MODNet [44] and P3M [54] focus on portrait foregrounds. Matte Anything [117] and Matting Anything [53] use promptable segmentation to guide alpha prediction, while In-Context Matting [29] specifies the foreground through a prompted reference image. SAM2-Matting [91] adds matting components to a segmentation tracker. For video, MatAnyone [115] propagates the matte of a target specified by a first-frame mask. MatAnyone 2 [116] improves detail and temporal consistency through learned quality evaluation and larger real-world training data. Many matting evaluations assume a salient foreground [44, 54] or provide a ground-truth mask prompt [115]. We evaluate promptable matting in multi-object scenes using boxes, category names, and referring expressions to specify targets, with WeVisOne-VL predicting their continuous alpha mattes.

Limitations

In this work, we focus on fundamental visual perception, which entails several trade-offs. First, the limited amount of general-purpose training data may compromise the general capabilities relative to the base model. Second, for inference efficiency, we train the model to directly produce structured outputs without chain-of-thought (CoT) reasoning [21, 69], which potentially limits its ability to tackle complex reasoning tasks. Third, our current approach is limited to single-turn prediction. It is promising to integrate an agent framework to progressively refine visual understanding through multi-turn tool interactions [38, 62].

Beyond these limitations, a key challenge lies in the real-world complexity. The object prompts in benchmarks are typically clear, well-formed, and unambiguous. However, real-world user instructions are often vague and hesitant. Variations in contexts or communication styles can make a straightforward task, such as locating the target, significantly more challenging. Therefore, strong intent reasoning is essential for a reliable vision system. In future work, we will continue to improve WeVisOne-VL toward addressing these limitations.

Conclusion

We introduced WeVisOne-VL, a vision-language model that unifies object-level (e.g., object detection, referring localization, counting, and OCR) and pixel-level (e.g., referring segmentation, and image matting) tasks through a structured next-token interface. Dedicated coordinate tokens encode instance predictions, while an <OBJ> token connects each localized instance to a lightweight pixel decoder for masks and alpha mattes. A three-stage supervised curriculum develops task alignment, high-resolution dense perception, and pixel-level prediction. After that, we apply mixed-task reinforcement learning to optimize the complete grounding outputs. Across the evaluated benchmarks, WeVisOne-VL achieves strong results on both object-level and pixel-level tasks. It improves mean detection F1 and counting accuracy over comparable-scale VLMs, advances text grounding and referring localization, and surpasses the evaluated baselines for referring segmentation and image matting. Qualitative examples further show that the model can localize individual instances in crowded scenes, ground expressions involving multiple objects, and recognize when a requested target is absent. By unifying these capabilities within a single language-guided model, WeVisOne-VL provides a foundation for practical visual systems.

Appendix

This appendix describes the training corpus of WeVisOne-VL and provides implementation details.

WeVisOne-VL Training Corpus

We describe the data used for SFT, segmentation and matting training, and RL below.

SFT Data

Table 9 summarizes our SFT data by task and annotation source, including public annotations and annotations generated by the WeVisOne-VL Data Engine (WDE).

Task	Source	Datasets	S1	S2	S3
Detection	Public	Objects365 [89], OpenImages-V6 [47], V3Det [102], DeepFashion2 [24], ODinW-35 [52], EgoObjects [128], PACO [82], PartImageNet++ [59], BDD100K [119], nuImages [74], VisDrone [19], CrowdHuman [88], SKU-110K [26], PartImageNet [31], MOT20Det [14], MOT17Det [15], OWDOD [93], FSCD-147 [75]	18.4	16.2	8.4
Detection	WDE	CC3M, GRIT, LAION, SA-1B	42.2	10.7	7.2
Referring	Public	RefCOCO [120], gRefCOCO [33], Flickr30k [79], HumanRef-CoT [40], HumanParts [60], $D^3$ [112], OpenRef [109], Unsplash [104]	6.3	2.1	1.0
Referring	WDE	GRIT, CC3M, LAION	11.8	5.5	1.6
Counting	Public	CLOC [51], MixCount [20], KubriCount [66], PixMo [13], OWDOD [93], FSC-147 [83], FSCD-147 [75]	2.4	1.2	0.6
Pointing	Public	PixMo [13], Molmo2 [10], CLOC [51], FSC-147 [83]	2.8	0.9	0.4
OCR	Public	IDL-OCR [2], BLIP3-OCR [113], ArT [9], COCO-Text [101], LSVT [99], TextOCR [95], ReCTS [124], HierText [71], RCTW [92], ICDAR2015 [43], WildReceipt [96], SROIE [35], ICDAR2013 [42]	2.1	0.7	0.3
Caption	Public	RefSpatial [126], Inst-IT [77], VG-PhraseCut [108], Flickr30k [118], COCONut-PanCap [16], PixMo [13], LLM-Seg40K [103], PanCap, Rea2Seg [23]	2.7	0.9	0.4
Segmentation	Public	RefCOCO [120], RefCOCO+ [120], RefCOCOg [72], ReVOS [114], Ref-YouTube-VOS [87]	0.9	0.9	1.6
Segmentation	WDE	CC3M, GRIT, LAION, SA-1B	0.6	1.4	20.3
Total			90.2	40.5	41.8

Segmentation and Matting Data

Table 10 summarizes the data sources for segmentation and matting training. Public matting datasets provide high-resolution images with high-quality alpha mattes. During matting fine-tuning, we sample public and WDE-annotated data at a ratio of 1:4 within each batch.

Task	Source	Datasets	Size (M)
Segmentation	Public	SA-1B [45], RefCOCO [120], RefCOCO+ [120], RefCOCOg [72], ReVOS [114], Ref-YouTube-VOS [87]	171.4
Segmentation	WDE	CC3M, GRIT, LAION	288.4
Matting	Public	AM-2K [56], Distinctions-646 [80], UHRIM [64], HHM-50K [98], IHIM-50K [36], RefMatte [57], AIM-500 [55], CelebA-Hair [11], Transparent-460 [3], SIMD [97], P3M-10K (train) [54], Real-Human [105], RealWorldPortrait-636 [121]	0.3
Matting	WDE	CC3M, SA-1B	129.8

Reinforcement Learning Data

We apply uncertainty-aware sample selection (§ 4) to the SFT data using the SFT checkpoint to construct the RL training set. The final RL set contains 178K examples: 44.0% detection, 20.8% dense detection, 17.5% referring localization, 7.9% OCR, 5.8% referring segmentation, and 4.0% pointing. Table 1 shows the complete 178K-example RL task distribution, including the additional localization and pointing examples.

Implementation Details

Training Hyperparameters

Table 11 summarizes the hyperparameters that differ across the three SFT stages and RL training. SFT-S3 refers to joint token–pixel training. We use 128 GPUs for SFT and 160 GPUs for RL. All four configurations use AdamW with a weight decay of 0.01, a maximum gradient norm of 1.0, and BF16 precision. We use cosine learning-rate decay for SFT and constant or cosine schedules for RL, depending on the configuration. For SFT, we use sequence packing and specify batch sizes by the number of tokens per GPU. OCR examples in SFT-S2 and SFT-S3 use 16–256 visual tokens to accommodate long transcriptions.

Hyperparameter	SFT-S1	SFT-S2	SFT-S3	RL
Trainable modules	LLM	VLM	VLM + Dec.	LLM
Learning rate: LLM	$2 \times10^{-5}$	$5 \times10^{-6}$	$5 \times10^{-6}$	$1 \times10^{-6}$
Learning rate: vision encoder	–	$5 \times10^{-7}$	$5 \times10^{-6}$	–
Learning rate: projector	–	$5 \times10^{-6}$	$5 \times10^{-6}$	–
Learning rate: pixel decoder	–	–	$1 \times10^{-5}$	–
Warmup ratio	5%	5%	3%	–
Training steps	13,000	8,995	9,062	593
Max. sequence length	4,096	8,192	8,192	4,096 + 4,096
Visual tokens / image	16–512	64–1,960	64–1,960	64–1,960
Token budget / GPU	28,672	24,576	24,576	24,576
Distributed training	ZeRO-2	ZeRO-2	ZeRO-2	FSDP2 + vLLM

Pixel Decoder Training

Warm-up. After SFT-S2, we connect the VLM to a randomly initialized pixel decoder and jointly train them for 100,000 steps with segmentation supervision. The decoder learning rate is $1 \times10^{-6}$ with 1,000 warmup steps. The decoder predicts masks at $512 \times512$ resolution. After warm-up, we retain only the trained decoder and <OBJ> embedding and attach them to the original SFT-S2 checkpoint for joint training. The other VLM weights updated during warm-up are discarded.

Joint training. During SFT-S3, we alternate NTP and segmentation steps at a 1:1 ratio to jointly train the SFT-S2 checkpoint with the warmed-up decoder. The VLM and decoder learning rates are $5 \times10^{-6}$ and $1 \times10^{-5}$, respectively, with a 3% warmup ratio. For each packed sequence on each GPU, we randomly sample 24 <OBJ> tokens and pass their corresponding representations to the decoder for segmentation prediction, while retaining the full text sequence for NTP.

Matting fine-tuning. After training is complete, we freeze the VLM and fine-tune the decoder for matting for 100,000 steps at an output resolution of $1024 \times1024$. We use a learning rate of $5 \times10^{-7}$ for the decoder and $2.5 \times10^{-6}$ for the coarse-to-fine refinement branch, with 200 warmup steps. Each GPU processes up to eight <OBJ> tokens. The matting loss is defined in § 4.

The warm-up and matting phases use AdamW with cosine learning-rate decay, a weight decay of 0.05, and a maximum gradient norm of 5.0. Both phases use BF16 precision and DDP, with a token budget of 98,304 per GPU, a maximum sequence length of 8,192, and 16–1,024 visual tokens per image.

RL Rewards

For detection, dense detection, and referring localization, we use the grounding reward defined in § 4. For segmentation and matting, RL also uses only the box-based grounding reward; no reward is computed from the predicted masks or alpha mattes. For pointing, we compute F1 through one-to-one matching between predicted points and ground-truth masks, counting a match when the point lies inside the mask. Referring segmentation requires each predicted box to be followed by an <OBJ> token, while OCR uses its transcription reward. When the target is absent, the reward is 1 only if the model predicts no object.