betaFeedback

WorksComputer Science

Sketch-Omni: Instance-Level Sketch-Based Visual Prompting for Perception MLLMs

Abstract

Promptable perception, from GroundingDINO to Rex-Omni, puts one model behind one prompt interface and returns coordinates for any localisation task. Every model in that line was built for photographs; none has been given a sketch. We present Sketch-Omni, to our knowledge the first perception MLLM built to work on sketches and from them. On a sketch, the sketch is the image, and text prompts detect, point, resolve object and spatial referring expressions, and ground affordances in it. From a sketch, the sketch is the prompt, and a drawing of one object localises that object in a photograph, even among several of the same category, a query that neither a category name nor a photo exemplar can pose. Two obstacles stand in the way. A photo-trained model's attention is tuned to colour and texture, which a sketch does not have, and no dataset pairs a drawing with the one instance it depicts among others of its kind. We build the training data instead of collecting it. An edge detector turns every labelled photograph into a sketch-like drawing that keeps the photograph's labels, and two constructions produce sketch-to-photo instance pairs from existing data. On top of this, photo-guided relation alignment passes each photograph and its edge sketch through the same model and trains the sketch pass to reproduce how the photograph pass relates phrases to patches and patches to each other. Not one training label is drawn by hand. Sketch-Omni reaches 91.4 F1 on sketch-prompted localisation against 81.8 for the strongest general-purpose MLLM, with a lead that doubles when the photograph holds several instances of the queried category, and it is best in 39 of 40 evaluation cells across six tasks. Code, model, and a hand-annotated benchmark of 2754 sketch-to-photo links will be released.

Introduction

Perception has converged on a prompt. GroundingDINO [25] opened a detector’s vocabulary with a text prompt. T-Rex2 [17] let a box or an exemplar stand in for the words. Rex-Omni [19] folded detection, pointing, referring, and visual prompting into one MLLM that answers every prompt with coordinate tokens, and general-purpose MLLMs (Bai et al., 2025b;a; Yue et al., 2025; Wu et al., 2024) now ship grounding as standard. All of these models were built for photographs. None has been given a sketch. This paper brings sketch into the interface, in both directions (Figure 1). Sketch-Omni works on sketches: the sketch is the image, and text prompts detect, point, resolve object and spatial referring expressions, and ground affordances in it. We call these five the sketch-native tasks. And it works from sketches: the sketch is the prompt, and given a drawing of one object and a photograph, the model localises that object in the photograph. To our knowledge it is the first perception MLLM to support both directions.

Figure 1. Sketch-Omni. Top, from a sketch: the sketch is the prompt, and a drawing of one object localises that object in a photograph. Bottom, on a sketch: the sketch is the image, and text prompts detect, point, resolve object and spatial referring expressions, and ground affordances in it. One model, one coordinate-token interface.

Sketch deserves this because it is not another kind of image. A photograph records what stood in front of a camera. A sketch records what a person has in mind: a product not yet built, a suspect never photographed. It is specific: a drawing shows the shape of one particular object, which is why fine-grained sketch retrieval has worked for a decade [49, 46]. And it is cheap: a few strokes, drawn in seconds, before any search begins. Put these together and a sketch can ask a question no other prompt can. A category name returns every instance of the category. A photo exemplar does the same, and it exists only if the object has already been photographed. A sketch, drawn from memory, asks for that one among several of its kind (Figure 2). Chowdhury et al. [8] first showed that a standalone detector can use a sketch this way. No perception MLLM can: its visual prompts are boxes and exemplars, and each stands for a category. We bring the query into the prompt interface as instance-level sketch-based visual prompting (instance VP), and it is where a sketch model should prove itself: on photographs with several instances of the queried category, where a name no longer helps.

Figure 2. Category-level versus instance-level visual prompting. A photo exemplar returns every instance of its category (blue); a sketch returns only the one it depicts (red).

Two things stand in the way. The first is the model. A photo-trained MLLM grounds a phrase by relating its tokens to image patches, and it relates the patches of one object to each other. Both relations rest on attention tuned to the colour and texture of photographs. A sketch has neither: most of the canvas is blank, and every cue is a line. On a sketch, Rex-Omni’s attention spreads over empty space (Figure 5), and a general-purpose MLLM finds roughly the right region but not the strokes that bound it (§ 4.4). The second is data. Photographic grounding is trained on tens of millions of labelled photographs ([25, 19]). Sketch has nothing comparable: single-object drawings each paired with one photo ((Sangkloy et al., 2016)), scene sketches without boxes ([7]), and instance links found by retrieval rather than by hand ([12]). Nobody will box-annotate a million drawings.

Figure 5. Cross-modal and intra-modal relations on free-hand sketches. Cross-modal maps show phrase-query similarity to every visual key, and intra-modal maps the red-marked patch’s query against every visual query. All maps are from layer 22.

Sketch-Omni answers both with two ingredients. The first is supervision built from photographs. An edge detector ([45]) turns any labelled photograph into a sketch-like line drawing that keeps the photograph’s boxes, phrases, and affordance labels; run over Objects365, RefCOCOg, and AGD20K, this supervises all five sketch-native tasks at no labelling cost. For instance VP we construct the pairs no dataset has: a Sketchy drawing whose paired photo is tiled into a canvas beside other photos of its category, and an edge-sketch crop of one annotated object in an Objects365 photograph, to be found again in that photograph. The second is photo-guided relation alignment, which teaches the model to read a sketch the way it already reads a photograph. Grounding rests on two relations: how much each phrase token attends to each image patch, and how the patches of one object relate to each other. Both are about structure, and structure is exactly what a photograph and its edge sketch share. So each labelled photograph passes through the frozen model and its edge sketch through the LoRA-adapted one, and the sketch pass is trained to reproduce the photograph pass’s two relations at every layer and head. The features are left free to differ, since a sketch is not a photograph; only the relations are matched. Not one training label is drawn by hand.1 The only hand annotation is for evaluation: 2754 sketch-to-photo instance links on FS-COCO, the first such benchmark we know of.

On instance VP, Sketch-Omni reaches 91.4 F1 at IoU 0.5 on FS-COCO, against 81.8 for the strongest general-purpose MLLM, Qwen3-VL-8B, given the same two images; its own base model, whose visual prompts must be boxes inside the target photograph, scores 4.0. The lead over Qwen3-VL-8B is 8.4 F1 when the photograph holds one instance of the queried category and 17.2 when it holds several (Sec. 4.3): the advantage sits exactly where a name fails. Across the six tasks Sketch-Omni is best in 39 of 40 table cells, and where general-purpose MLLMs collapse at tight boxes (Qwen3-VL-8B: 75.3 F1 at IoU 0.5 on free-hand detection, 23.0 at IoU 0.9) it holds 65.7. The alignment improves every one of the 14 ablation cells, by up to 4.3 F1 on referring, the task that depends most on a phrase selecting its region (Table 1, App. F).

What this opens. Sketch research has built one model per task: a recogniser, a retriever, a segmenter. Sketch-Omni replaces the model with a prompt. Any task the grounding literature poses to a photograph can now be posed to a drawing, in the same words; this paper poses six. For grounding research, sketch offers three things: a visual prompt that names an instance without a photograph of it; a modality added to a perception MLLM without a single new label; and a test of how much a model’s grounding leans on appearance, since a sketch is a photograph with its appearance removed, and the IoU 0.9 column is the score.

Related Work

Promptable perception. GroundingDINO (Liu et al., [25]) made a DETR-style detector (Zhang et al., [53]) open-set by conditioning it on a text prompt, T-Rex2 (Jiang et al., [17]) added visual prompts beside the text, and ChatRex (Jiang et al., [18]) carried prompted perception into a multimodal LLM. In parallel, Pix2Seq (Chen et al., [6]) cast detection as language modelling over coordinate tokens. Autoregressive coordinate prediction was subsequently adopted by grounding MLLMs (Peng et al., [30]; Chen et al., [5]; You et al., [48]; Zhan et al., [52]; Wang et al., [42]; Xiao et al., [44]) and is now built into general-purpose MLLMs (Bai et al., [3]; Wu et al., [43]; Deitke et al., [9]; Yue et al., [51]; Guo et al., [13]). Rex-Omni (Jiang et al., [19]) closes the loop, unifying detection, pointing, referring, OCR, keypoints, visual prompting, and GUI grounding under one coordinate-token vocabulary. Where language cannot name a target, they take clicks, boxes, masks, and regions of interest as prompts (Kirillov et al., [20]; Zou et al., [56]; Zhang et al., [54]; Lin et al., [24]; Yang et al., [47]), each placed on the image being perceived, so the object is already in view, or taken from another photograph as a category exemplar (Jiang et al., [17]). The whole line assumes captured images to perceive and photographs to prompt with. Sketch-Omni adds a modality to this line, not a task.

Sketch as a query. Sketch research has argued for the drawing as a query since its datasets were built (Eitz et al., [11]; Sangkloy et al., 2016; Ha & Eck, [15]; Gao et al., [12]; Chowdhury et al., [7]), chiefly through sketch-based image retrieval (Yu et al., 2023; Dey et al., [10]; Lin et al., [22]), and fine-grained SBIR (Yu et al., [49]; Sain et al., [33]) established that a drawing identifies an instance rather than a category. Retrieval returns whole images, though. Tripathi et al. [38, 39] moved the query inside the image, localising sketched objects in natural images, and Chowdhury et al. [8] build a zero-shot detector on a CLIP-initialised [31] SBIR backbone that takes the sketch as an instance-aware query. O3SLM (Gupta et al., [14]) places the sketch query in an LVLM trained on image–sketch–instruction triplets for localisation, counting, retrieval, and question answering. Its sketch stands for a category, since a drawing is paired with any image that holds the drawn class and localisation returns every instance of it, the behaviour of the photo exemplar in Figure 2, and it never perceives within a sketch. The instance-level query therefore exists only in standalone detectors that take no language and serve one task.

Perceiving within sketches. Understanding what a sketch contains has been left to single-task models. Scene sketches have been segmented semantically, recently with an open vocabulary (Bourouis et al., [4]), and by instance, where InkLayer (Tang et al., [37]) fine-tunes GroundingDINO on scene sketches class-agnostically, collapsing every label to object, so the detector survives the domain gap but the prompt does not. SketchFusion (Koley et al., [21]) fuses foundation-model features into a universal sketch encoder and StructXLIP (Ruan et al., [32]) aligns CLIP embeddings against structural cues, but none is prompted beyond a category list, and none points, refers, or grounds affordances on a drawing. Sketch-Omni is where the two lines meet.

Sketch-Omni

Sketch-Omni adds a modality to Rex-Omni [19] and leaves its interface unchanged. Stage A (Sec. 3.1) transfers the base model’s grounding to sketch input through photo-guided relation alignment while training the five sketch-native tasks, and Stage B (Sec. 3.2) builds instance VP on that shared reading from constructed supervision.

The interface casts perception as next-token prediction. A textual prompt TT and the visual inputs form one context, and coordinates are discretised into 1000 bins, each a dedicated coordinate token (App. A). Sketch-native perception makes the sketch the image input, and the prompt TT lists the phrases to be grounded. A LoRA [16] update Δθ\Delta\theta on the frozen pretrained weights θ0\theta_0 defines the sketch path and θ0\theta_0 alone the photo path, written SS and PP for the input each runs on. Both training stages fit Δθ\Delta\theta with a next-token prediction loss LNTP\mathcal{L}_{\mathrm{NTP}} on the target sequence, and they interleave under a single optimiser, alternating at a fixed step interval (Sec. 4.2).

Stage A: photo-guided relation alignment

Stage A trains the five sketch-native tasks, detection, pointing, object referring, spatial referring, and affordance grounding, from two sources. A labelled photograph yields its pixel-aligned edge sketch through a learned detector [45], so the sketch trains under supervision carried over from the photograph. Since a sketch cannot show colour or texture, a lexicon filter [32] strips appearance terms from each phrase; only the phrases it alters are re-grounded on the photograph by the base model, whose answers become their labels, and every other label stays as annotated. What stays invariant across the two renderings is the scene structure, which patches belong to one object and which region a phrase selects, so the alignment supervises these relations and leaves the features free: it never asks the sketch path to look like the photo path, only to relate its tokens as the photo path relates its own, and the sketch path still encodes strokes.

Which relations to align. Grounding rests on two kinds of relation. A phrase has to select its region, and attention already computes this as the cross-modal query-to-key relation, which Lcross\mathcal{L}_{\mathrm{cross}} aligns across the two paths. The parts of one object have to hold together, and for this we borrow the intra-modal query-to-query relation underlying DeCLIP’s context features [41], which Lintra\mathcal{L}_{\mathrm{intra}} aligns in the same way. Both losses compare corresponding token positions between the two readings of one scene, not samples in a batch as in relation-based distillation [29, 40].

How the relations are matched. The photo path runs without gradients, so the alignment adds one forward pass per Stage-A step (Figure 3). The sketch keeps the photograph’s prompt, target sequence, and patch grid, so a relation in one path has an exact counterpart in the other.

Sketch-Omni overview diagram showing photo-guided relation alignment and the two-stage architecture

Figure 3. Sketch-Omni overview. Left, both stages share one LoRA-adapted backbone under a single optimiser. Right, photo-guided relation alignment matches the sketch path’s token relations to the frozen photo path’s via Lcross\mathcal{L}_{\mathrm{cross}} and Lintra\mathcal{L}_{\mathrm{intra}}, with solid grids denoting queries and dashed grids keys. S(l)S^{(l)} and P(l)P^{(l)} are the two paths’ hidden states at layer ll, and ΔWq(l)\Delta W_q^{(l)}, ΔWk(l)\Delta W_k^{(l)} the LoRA updates to the frozen projections Wq(l)W_q^{(l)}, Wk(l)W_k^{(l)}.

At every language-model decoder layer ll, the sketch path computes QS,(l)Q^{S,(l)} and KS,(l)K^{S,(l)} through the adapted projections and the photo path computes QP,(l)Q^{P,(l)} and KP,(l)K^{P,(l)} through the frozen ones (Figure 3). We align every head of every layer, where each key head is shared by a group of query heads. Cosine similarities are computed after Qwen2.5-VL’s rotary position embeddings [3] are applied. Let V\mathcal{V} hold the positions of the image tokens, F⊆V\mathcal{F} \subseteq\mathcal{V} those whose patch overlaps any annotated box, and T\mathcal{T} the positions of the phrases in the prompt, all taken on the input side of the teacher-forced sequence. Lcross\mathcal{L}_{\mathrm{cross}} keeps every visual position as a key, which lets it penalise phrase similarity that falls on blank regions. Lintra\mathcal{L}_{\mathrm{intra}} restricts its queries to F\mathcal{F}, which keeps the grouping signal on foreground object patches. For simplicity, qiSq_i^S and kjSk_j^S denote path SS’s query and key at positions ii and jj, with the layer and head indices suppressed. With Φ(u,v)=⟨u,v⟩/(∥u∥∥v∥)\Phi(u,v)=\langle u,v\rangle/(\lVert u\rVert\lVert v\rVert), D\mathcal{D} the decoder layers, and HH the number of heads,

Lcross=1∣D∣∑l∈D∑h=1H1∣T∣∣V∣∑i∈T, j∈V(Φ(qiS,kjS)−Φ(qiP,kjP))2(1)\mathcal{L}_{\mathrm{cross}} = \frac{1}{|\mathcal{D}|} \sum_{l\in\mathcal{D}} \sum_{h=1}^{H} \sqrt{ \frac{1}{|\mathcal{T}||\mathcal{V}|} \sum_{i\in\mathcal{T},\,j\in\mathcal{V}} \left(\Phi(q_i^S,k_j^S)-\Phi(q_i^P,k_j^P)\right)^2 } \tag*{(1)}
Lintra=1∣D∣∑l∈D∑h=1H1∣F∣2∑i,j∈F(Φ(qiS,qjS)−Φ(qiP,qjP))2(2)\mathcal{L}_{\mathrm{intra}} = \frac{1}{|\mathcal{D}|} \sum_{l\in\mathcal{D}} \sum_{h=1}^{H} \sqrt{ \frac{1}{|\mathcal{F}|^2} \sum_{i,j\in\mathcal{F}} \left(\Phi(q_i^S,q_j^S)-\Phi(q_i^P,q_j^P)\right)^2 } \tag*{(2)}

With Lalign=λcrossLcross+λintraLintra\mathcal{L}_{\mathrm{align}}=\lambda_{\mathrm{cross}}\mathcal{L}_{\mathrm{cross}}+\lambda_{\mathrm{intra}}\mathcal{L}_{\mathrm{intra}}, Stage A optimises

LA=LNTP+Lalign(3)\mathcal{L}_A=\mathcal{L}_{\mathrm{NTP}}+\mathcal{L}_{\mathrm{align}} \tag*{(3)}

jointly over the five sketch-native tasks. The two weights are given in § 4.2.

Stage B: instance-level visual prompting

Stage B places two images in the context, a target photograph first and a query sketch second, followed by a prompt naming the sketch as the query (App. § 7). The answer is a match between the sketch’s tokens and the photograph’s, which is what the shared reading built in Stage A makes possible. No dataset annotates the supervision this needs, a free-hand drawing tied to the one instance it depicts inside a larger scene, so we build it in two ways.

The first construction uses sketch–photo pair datasets, in which a free-hand drawing is paired with a photograph of a single object. We tile the paired photograph, together with crops of the same category, onto a low-clutter background, so the drawing designates exactly one instance among them, and the category name alone cannot identify the target. The second construction supplies a real scene, with its clutter and occlusion, by deriving pairs from ordinary detection annotations. Given an annotated object, we crop its region from the photograph’s edge sketch and use the crop as the query, asking the model to localise that instance in the untouched photograph. One construction trains tolerance to variation in drawing style, the other precise shape matching in real scenes. Full construction details are given in App. § 8.

When an input holds both a photograph and a sketch, in Stage B and at inference, the vision encoder runs once per image in the interface’s fixed image order, with the LoRA update disabled for the photograph and applied for the sketch; the language model applies it over the whole sequence.

Stage B trains with next-token prediction alone, interleaved with Stage A, so the alignment keeps acting on the sketch path while the prompting capability forms on top of it. Alongside the two-image samples, detection and pointing train in this stage on multi-object canvases composited from single-object free-hand drawings, where prompts naming absent categories are answered with an explicit None. A Group Relative Policy Optimisation (GRPO) [36] pass with localisation rewards then refines the model (App. § 13), and training sources for both stages are given in § 4.1. If shape is what lets a query pick out its instance, the advantage over a competent baseline should widen once same-category distractors are present (§ 4.3).

Experiments

Datasets

Stage A, sketch-native perception. Each sketch-native task takes photographs from a photo-grounding dataset and feeds the matching HED edge sketch [45] as the input. Objects365 [35] supervises detection and pointing, and box centres serve as the pointing targets. RefCOCOg [27] supervises object referring and spatial referring, which name an instance by what it is or by where it stands, and AGD20K [26] affordance grounding.

Stage B, instance-level sketch-based visual prompting. Stage B trains instance-level visual prompting on Sketchy (Sangkloy et al., 2016), which pairs free-hand sketches with single-object photos. One positive crop is tiled alongside same-category distractor crops onto a low-clutter Objects365 background. No Sketchy sketch of a SketchyCOCO foreground category enters this training, and any training sketch that also appears as a SketchyCOCO query is removed. The second construction draws on Objects365 [35], where the query is an annotated object’s edge sketch cropped at its box and the target is the source photograph.

Evaluation. We evaluate against base Rex-Omni [19] and baselines appropriate to each task. Detection, referring, and pointing are compared against the general perception MLLMs Qwen2.5-VL [3], Qwen3-VL [2], MiMo-VL [51], and DeepSeek-VL2 [43]. Instance VP is compared against the same four, where each model receives the query sketch and the target photograph as two images. Since these four MLLMs are not specifically trained on sketches, we also compare against a pipeline that pairs a detector and a sketch retrieval model: GroundingDINO [25] proposes boxes in the target photograph under a fixed vocabulary prompt shared by every query, and ZSE-SBIR [22] ranks them against the query sketch. Affordance is compared against the pointing specialists Molmo [9], RoboPoint [50], and RoboRefer [55]. Referring and affordance are evaluated on HED sketches. These are pixel-aligned with the source photographs, so we reuse the photographs’ ground truth. Detection and pointing are evaluated on free-hand sketches from FS-COCO ([7]) and on HED sketches rendered from COCO ([23]) under Rex-Omni’s protocol.

Method[email protected][email protected][email protected]
Qwen2.5-VL-3B59.3/55.853.6/51.148.8/47.5
Qwen3-VL-8B68.3/84.566.5/81.862.5/76.9
MiMo-VL-7B63.6/48.261.2/45.557.2/40.1
DeepSeek-VL2-S53.0/43.150.3/36.046.5/26.0
Rex-Omni18.1/5.713.0/4.04.7/1.4
G-DINO + ZSE-SBIR69.5/55.663.3/51.157.9/47.1
Sketch-Omni (full)74.0/93.671.0/91.467.7/87.1
w/o align73.1/91.269.8/88.666.7/84.2

Table 1. Instance-level sketch VP. Each cell is SketchyCOCO / FS-COCO, and the lower block ablates the alignment. Bold means best and <u>underline</u> means second best.

Free-hand evaluation has no such transfer and requires new labels. SketchyCOCO ([12]) pairs real photographs with composited scene sketches. However, three properties limit what a margin on it can show: (1) Its queries are linked to their instances by a retrieval model (Sangkloy et al., 2016), so the links are only as accurate as that model. (2) The queries come from a small pool, so the same drawing recurs across different photographs. (3) The queried instance is typically the photograph’s dominant object, covering 17.1% of it at the median against 8.8% on FS-COCO, so localising the most salient object succeeds without attending to the query. FS-COCO provides what SketchyCOCO cannot, genuine free-hand scene sketches drawn from memory after viewing the paired photograph, but it provides neither object boxes nor instance links. We therefore annotate its test split with boxes on the sketch as ground truth for detection and pointing, and with sketch-to-photo instance links for VP, where each query is the region of the scene sketch that the annotated box covers (App. § 9). The evaluation comprises 1340 SketchyCOCO and 2754 FS-COCO sketch queries, with qualitative results in Figure 4.

Qualitative instance-level sketch VP

Figure 4. Qualitative instance-level sketch VP. Each row shows a free-hand query sketch, the target photograph with the ground-truth box, and one panel per model.

Metrics. Given predicted and ground-truth boxes, we match predictions to ground truth one-to-one and greedily by Intersection-over-Union (IoU). Recall and precision are computed over all matches in an evaluation set, and F1 is their harmonic mean. Detection and referring report IoU 0.5, IoU 0.9, and the mean across thresholds from 0.5 to 0.95 with step size 0.05, shown in the avg columns. Instance VP reports IoU 0.3, 0.5, and 0.7. Pointing and affordance test containment instead. For pointing, a prediction counts as correct when it falls inside the target’s segmentation mask, or its box where no mask exists. For affordance, it must fall inside the ground-truth region obtained by thresholding the heatmap at half its peak (App. § 10).

Implementation Details

We initialise θ0\theta_0 from Rex-Omni ([19]), a Qwen2.5-VL-3B ([3]) backbone, and train rank-64 LoRA adapters Δθ\Delta\theta ([16]) on all linear layers of every language-model block and on the query, key, value, and output projections of the vision encoder, with θ0\theta_0 frozen. The two stages alternate every 70 optimiser steps at a Stage-A to Stage-B ratio of 2:3, and the Stage-A objective (Eq. (3)) uses λcross=0.5\lambda_{\mathrm{cross}}=0.5 and λintra=0.25\lambda_{\mathrm{intra}}=0.25. Decoding at inference is greedy. Every configuration, ablations included, is refined by the same final GRPO pass following Rex-Omni. Optimiser and GRPO settings are given in App. § 13.

Instance-Level Sketch-Based Visual Prompting

Sketch-Omni outperforms every baseline at all three IoU thresholds (Table 1). Base Rex-Omni fails at this task: its visual prompting takes reference boxes inside the target image and returns every similar object, so it has no instance-level query mode and seldom answers the two-image prompt. Sketch-Omni leads the strongest MLLM by 4.5 and 9.6 F1 at IoU 0.5 on SketchyCOCO and FS-COCO, and the assembled pipeline by 7.7 and 40.3, since its retrieval stage ranks proposals by category and cannot pick one instance out of several. The lower block ablates the alignment. The w/o align row is the same recipe trained without LalignL_{\mathrm{align}}, and the constructed supervision alone establishes the capability, outperforming every baseline in every cell. The alignment provides a consistent further improvement of 0.9 to 2.9 points in all six cells. Figure 5 shows how base Rex-Omni and Sketch-Omni differ on a sketch-native detection prompt: cross-modal similarity from the phrase spreads over blank regions under Rex-Omni and concentrates on the queried object under Sketch-Omni, and intra-modal similarity propagates from a token across its instance only under Sketch-Omni. Figure 4 shows the failure modes: a baseline that misses returns a same-category distractor, another salient object, or nothing, where Sketch-Omni localises the instance the sketch depicts.

Resolving same-category distractors. Instance VP is defined by discrimination among same-category distractors, which single-instance scenes never exercise. We therefore split both benchmarks by the number of annotated instances of the queried category in the photograph, Easy when it holds exactly one and Hard when it holds two or more. Counts and all MLLM baselines are in App. E. Table 2 shows that on FS-COCO-Hard, Sketch-Omni reaches 80.3 F1 at IoU 0.5 against 63.1 for the strongest baseline, 17.2 above it. SketchyCOCO shows the same ordering with a smaller margin, since its same-category instances differ little from one another and leave less for a query to separate. The two models are within one point of each other on SketchyCOCO-Easy. On FS-COCO, whose queries are from the target’s own strokes, Sketch-Omni leads by 7.9 to 8.8 F1 even on Easy, and the lead grows with the IoU threshold as a faithful query improves the box itself.

MethodSubset[email protected][email protected][email protected]
Qwen3-VL-8BEasy91.0/86.990.2/84.386.7/79.5
Hard50.5/66.247.8/63.143.4/56.8
Sketch-OmniEasy91.7/94.889.5/92.786.1/88.3
Hard55.3/83.351.4/80.348.2/77.0
ΔEasy+0.7/+7.9−0.7/+8.4−0.6/+8.8
Hard+4.8/+17.1+3.6/+17.2+4.8/+20.2

Table 2. Stratified instance VP. Each cell is SketchyCOCO / FS-COCO on the Easy and Hard subsets of § 4.3, and the Δ rows report Sketch-Omni minus Qwen3-VL-8B.

Sketch-Native Tasks

Sketch-Omni achieves the best recall, precision, and F1 in every detection column of Table 3, with qualitative results in Figure 6. On free-hand sketches, Sketch-Omni attains its largest margin at [email protected], leading base Rex-Omni by 9.6 and the best general-purpose MLLM by 42.7. Sketch-Omni retains 62.1 of its 81.9 recall as the threshold tightens from 0.5 to 0.9, while Qwen3-VL drops from 70.2 to 21.4, the signature of grounding tuned to appearance: the region is found and the strokes are not. On edge sketches, Sketch-Omni leads the best baseline by 23.6 in precision at IoU 0.5. On pointing, Sketch-Omni outperforms every model on free-hand sketches.

MethodDetection [email protected]Detection [email protected]Detection [email protected]Detection [email protected]Detection [email protected]Detection [email protected]Detection R@avgDetection P@avgDetection F1@avgPointing F1@Point
Qwen2.5-VL-3B48.8/24.269.7/41.457.4/30.613.3/6.519.1/13.215.7/8.730.6/14.843.7/27.436.0/19.272.2/36.3
Qwen3-VL-8B70.2/34.381.2/55.675.3/42.421.4/9.924.8/17.723.0/12.747.5/22.055.0/36.751.0/27.581.7/47.9
MiMo-VL-7B39.6/19.143.5/30.241.5/23.41.9/2.52.1/5.62.0/3.416.6/8.918.3/16.317.4/11.561.8/33.8
DeepSeek-VL2-S57.8/31.562.3/50.060.0/38.611.6/9.312.5/15.812.0/11.733.8/19.936.5/32.335.1/24.7-
Rex-Omni74.7/24.386.0/55.180.0/33.752.4/7.260.4/18.956.1/10.463.5/15.573.1/37.568.0/22.084.9/37.9
Sketch-Omni81.9/34.592.1/79.286.7/48.162.1/15.869.8/37.365.7/22.272.4/25.781.4/59.776.6/36.089.9/44.5

Table 3. Sketch-input detection and pointing. Each cell is FS-COCO / COCO.

Figure 6. Qualitative results on sketch input. Two examples each of detection and pointing, object referring, spatial referring, and affordance grounding, with box and point colours matching the phrase colours. The referring expressions and their ground-truth boxes are newly annotated for these examples, and DeepSeek-VL2-S outputs boxes only.

On object and spatial referring (Table 4), Sketch-Omni achieves the best result in every cell, and attains its largest margin at IoU 0.9, where object F1 reaches 43.1/42.7 against 25.4/25.9 for the strongest baseline. Every general-purpose MLLM loses average F1 on spatial expressions relative to its own object subset, up to 4.3 points for the strongest, while Sketch-Omni widens its margin from 11.4/13.1 to 16.0/17.7 and resolves spatial expressions better than object ones, 77.6 against 75.0 F1 at IoU 0.5 on val, which is what a drawing that keeps layout and drops appearance should do.

MethodObject referring [email protected]Object referring [email protected]Object referring F1@avgSpatial referring [email protected]Spatial referring [email protected]Spatial referring F1@avg
Qwen2.5-VL-3B51.5/50.815.8/16.733.7/33.150.3/50.013.6/12.530.2/30.2
Qwen3-VL-8B62.2/62.423.7/25.543.8/44.361.7/62.521.1/21.942.3/42.8
MiMo-VL-7B43.5/45.72.3/3.420.0/22.441.7/42.71.8/2.418.4/18.6
DeepSeek-VL2-S69.1/66.125.4/25.948.2/46.567.6/65.121.1/20.243.9/43.0
Rex-Omni52.0/51.220.2/20.336.3/35.754.2/54.419.4/19.837.1/37.5
Sketch-Omni75.0/74.643.1/42.759.6/59.677.6/76.743.3/43.159.9/60.7

Table 4. Sketch-input referring on RefCOCOg. Each cell is val / test. Object and spatial referring are told apart by the presence of a spatial term in the expression.

On AGD20K affordance grounding (Table 5), the 3B Sketch-Omni outperforms every pointing specialist on both splits, improving base Rex-Omni by 14.9 and 13.3. The alignment accounts for 0.4 to 4.3 F1 of the gain across the five tasks (App. § 12). Across Tables 1, Table 3, Table 4, and Table 5, Sketch-Omni is best in 39 of 40 cells.

MethodSeenUnseen
Molmo-7B-D53.656.8
Molmo-72B56.858.6
RoboPoint13.813.9
RoboRefer-2B40.643.7
Rex-Omni51.361.0
Sketch-Omni66.274.3

Table 5. Affordance grounding on AGD20K. Point-in-region F1 on the Seen and Unseen splits, against pointing specialists.

Conclusion

Sketch-Omni brings promptable perception to sketch, on the drawing and from it. One diagnosis, that a sketch keeps structure and drops appearance, gives the recipe: construct the supervision, and align relations rather than features, with no new training label. The evidence concentrates where a name fails, among several instances of one category, and there Sketch-Omni leads the strongest baseline by 3.6 F1 on SketchyCOCO and 17.2 on FS-COCO at IoU 0.5. For sketch research, every perception task is now a prompt away; for grounding, sketch is a prompt that carries identity, a modality that costs no labels, and a test of what grounding leans on.

Acknowledgements

The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI [28] is operated by the University of Bristol and is funded by the UK Government’s Department for Business, Innovation, Science and Trade (DBIST) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Token Interface of the Base Model

Rex-Omni [19] casts every perception task as conditional sequence generation over one shared vocabulary, so detection, pointing, referring, and visual prompting differ only in their prompt and output content.

Input grammar. A visual input is encoded by the vision encoder into NvN_v patch tokens under Qwen2.5-VL’s dynamic-resolution convention, so NvN_v varies with the input’s size and aspect ratio. The patch tokens are wrapped in the image delimiters ⟨im⟩\langle\mathrm{im}\rangle and ⟨/im⟩\langle/\mathrm{im}\rangle, then concatenated with the NtN_t tokens of the textual prompt TT,

x=[⟨im⟩,v1,…,vNv,⟨/im⟩,w1,…,wNt].(4)\mathbf{x} = [\langle\mathrm{im}\rangle, v_1, \ldots, v_{N_v}, \langle/\mathrm{im}\rangle, w_1, \ldots, w_{N_t}]. \tag*{(4)}

For sketch-native tasks TT carries phrases p1,…,pMp_1,\ldots,p_M, as in “Detect p1,…,pMp_1,\ldots,p_M in the sketch.”, where MM is the number of phrases and a phrase may be a category name, a referring expression, or an affordance verb. Sketch VP uses a fixed prompt, since the query is carried by the sketch span rather than by words. The position sets that § 3 pairs index this sequence: V\mathcal{V} holds the positions of v1,…,vNvv_1,\ldots,v_{N_v}, T\mathcal{T} those of the tokens of p1,…,pMp_1,\ldots,p_M within w1,…,wNtw_1,\ldots,w_{N_t}, and F⊆V\mathcal{F}\subseteq\mathcal{V} those whose patch overlaps an annotated box.

Output grammar. Coordinates are normalised to the image and quantised into 1000 bins, each a dedicated coordinate token in the vocabulary. Every task answers with one structured token sequence, built from a descriptive phrase, coordinate tokens, and special tokens that demarcate the two,

⟨ref⟩ PHRASE ⟨/ref⟩⏟phrase⟨box⟩ COORDS ⟨/box⟩⏟geometry,(5)\underbrace{\langle\mathrm{ref}\rangle\ \mathrm{PHRASE}\ \langle/\mathrm{ref}\rangle}_{\text{phrase}}\quad \underbrace{\langle\mathrm{box}\rangle\ \mathrm{COORDS}\ \langle/\mathrm{box}\rangle}_{\text{geometry}}, \tag*{(5)}

where PHRASE names the object the coordinates stand for, and COORDS lists them, [x0,y0,x1,y1][x_0,y_0,x_1,y_1] per box and [x0,y0][x_0,y_0] per point. Several instances of one phrase share a single geometry segment, comma-separated and ordered by increasing x0x_0, and commas likewise join the blocks of different phrases. A phrase with no match in the input carries a literal None inside the geometry segment.

Visual prompting in the base model. Rex-Omni prompts visually with boxes rather than with a second image. One or more reference boxes inside the target image are quantised into the same coordinate tokens as an output box and passed in the prompt as a list under a single object key, and the released template reads “Given reference boxes {“object_1”: [“c1c2c3c4”, . . . ]} indicating one or more objects, find all similar objects in the image and output their bounding boxes”. The answer lists every box of the indicated category. The query therefore names a category by example and lives in the image it queries, which is why it has no way to ask for one instance among several of that category (Sec. 4.3).

Sketch-VP extension. The sketch-VP input extends Eq. (4) to two delimited visual spans, the target photograph first and the query sketch second, with the prompt specifying the query sketch,

xVP=[⟨im⟩ v1P,…,vNPP,⟨/im⟩, ⟨im⟩ v1S,…,vNSS,⟨/im⟩, w1,…,wNt],(6)\mathbf{x}_{\mathrm{VP}} = [\langle\mathrm{im}\rangle\ v^P_1,\ldots,v^P_{N_P},\langle/\mathrm{im}\rangle,\ \langle\mathrm{im}\rangle\ v^S_1,\ldots,v^S_{N_S},\langle/\mathrm{im}\rangle,\ w_1,\ldots,w_{N_t}], \tag*{(6)}

where v1P,…,vNPPv^P_1,\ldots,v^P_{N_P} are the patch tokens of the target photograph and v1S,…,vNSSv^S_1,\ldots,v^S_{N_S} those of the query sketch. The output grammar is Eq. (5) unchanged, with the queried instance’s category name filling the reference segment and the answer box expressed in the target photograph’s coordinate frame. Sketch-Omni therefore inherits Rex-Omni’s decoding, parsing, and evaluation stack verbatim.

Training Data Construction

Composited sketch scenes for Stage B detection and pointing. The scenes composite single-object drawings from three sources, each contributing a different kind. QuickDraw [15] holds 345 categories of doodles drawn under a twenty-second limit, TU-Berlin [11] 250 categories drawn under a much looser thirty-minute limit, and Sketchy (Sangkloy et al., 2016) 125 categories drawn from memory after a brief view of a photograph, so mixing them exposes Stage B to the range of abstraction a free-hand query can take. All three store strokes as vectors. Their class names are unified into a self-derived vocabulary of 415 categories, the canonicalised union of the three: names are lowercased, de-pluralised, and passed through a synonym table that merges variants such as aeroplane and airplane. For each instance, a source dataset is first sampled uniformly, then a category within that source, then a drawing, so the mix stays near one third per source and QuickDraw’s far larger pool cannot dominate. Each scene holds 2 to 12 instances from 2 to 6 categories on a canvas whose long side varies from 768 to 1152 pixels (Figure 7). Every drawing is rendered from its stroke data directly at its final canvas size rather than rescaled, so stroke width stays constant across object sizes. A placement is rejected when the new box overlaps a placed one above 0.5 IoU, or when either box contains more than 0.8 of the other, so no object ends up hidden inside another. Prompts list the present categories together with sampled absent ones, which the target answers with None. Scenes are generated procedurally during training rather than stored.

Composited free-hand sketch scenes

Figure 7. Composited free-hand sketch scenes. Four scenes with their ground-truth boxes, ordered by instance count. Canvas size and aspect ratio are sampled per scene.

Sketchy stitched canvases for Stage B. Sketchy provides photo-sketch pairs but no bounding boxes, so each photo receives a tight box from GroundingDINO [25] prompted with its category name. A square canvas with side sampled between 768 and 1280 pixels tiles 1 to 7 such photos drawn from up to 4 categories (Figure 8). Instance-level samples place several tiles of the queried category, so the query competes with a same-category distractor. Tiles are cropped off-centre within the margin around the tight box, resized across a stratified size range, colour jittered, alpha-feathered at their borders, and placed so that no tile overlaps a previously placed object region. Every ground-truth box therefore stays fully visible. Backgrounds are Objects365 photographs screened to contain none of the pasted categories. Query sketches are inverted to white-on-black so their polarity matches the HED sketches of Stage A.

Stitched Sketchy canvases showing query sketches and canvases with queried instances, same-category distractors, and other objects

Figure 8. Stitched Sketchy canvases. Each group holds the query sketch and the canvas it is asked about. Green marks the queried instance, orange a same-category distractor, and grey every other object.

Objects365 HED-sketch queries for Stage B. For the Objects365 augmentation, HED is applied to the whole photograph, and the query is cropped from the resulting edge sketch at one annotated object’s box, with a random padding of 2.5 to 7.5% of the box size per side. Boxes with area below 96×9696 \times96 pixels are skipped. The model is then asked to localise that instance in the untouched source photograph (Figure 9).

Objects365 HED-sketch queries

Figure 9. Objects365 HED-sketch queries. Each group holds a photograph with the queried box and the HED-sketch crop taken from that box, which serves as the query. Sketches are drawn black on white here, and training uses the inverted polarity.

SketchyCOCO overlap filter. SketchyCOCO’s foreground sketches were themselves retrieved from QuickDraw, TU-Berlin and Sketchy, the three pools the constructions above draw from, so an evaluation query could otherwise appear among the training drawings. Two levels of filtering separate them. At the category level, every Sketchy pair whose category is one of SketchyCOCO’s 14 foreground categories is dropped from training: airplane, bicycle, car, cat, cow, dog, elephant, fire hydrant, giraffe, horse, motorcycle, sheep, traffic light and zebra. At the instance level, every SketchyCOCO evaluation query is matched against all three pools and every drawing it matches is denied to both constructions. A drawing’s signature is its ink cropped to the tight box, stretched to 32×3232 \times32, Gaussian blurred and L2 normalised, and a candidate is denied once its cosine similarity to a query reaches 0.92, with the horizontal flip of each query matched as well. The stretch removes the aspect distortion SketchyCOCO applies when fitting a sketch into an instance box, and the blur absorbs the stroke-width and antialiasing differences between its renders and ours.

FS-COCO Annotation

Procedure. Two people annotated the official FS-COCO test split and a third checked every annotation, using a browser tool built for the purpose. It shows the photograph, with COCO’s boxes drawn on it, beside the rendered sketch, and records every box and link as it is drawn, so each link is made with both sides in view.

Boxes on the sketch. Labels come from the COCO 80 categories extended with five that appear in these sketches but COCO does not name: tree, building, cloud, road sign, and tower. Every sketched object of one of these categories receives a box, tightened to the outer edge of its strokes. A box is drawn only when the strokes inside it are recognisable as that object on their own, without looking at the photograph. The ground truth is therefore self-contained.

Sketch-photo links. The photograph’s boxes are COCO’s own annotations and stay untouched, so linking runs in one direction only. A sketched object is linked to a photographed instance when the two depict the same object, and the link is kept only when the sketch crop alone identifies that instance uniquely. An instance annotated in the photograph that also appears in the sketch receives a box on the sketch side and is then linked, while a sketched object with no counterpart among the photograph’s annotations stays unlinked and serves the detection and pointing ground truth alone. A sketched object whose category COCO does not annotate therefore carries no query. This yields the 2754 query-target pairs of § 4.1.

Category distribution. The annotation holds 11674 boxes over 68 categories on 2907 sketches, 4.0 boxes per sketch, and the queries span 59 of those categories. Queries per category have a median of 15 against a mean of 47. The distribution is long-tailed (Figure 10), as people drawing a scene from memory return to a few familiar objects far more often than to the rest.

Figure 10. Per-category counts of the FS-COCO annotation. Categories are ranked by count and the vertical axis is logarithmic. The dashed line marks the median.

Affordance Evaluation Protocol

AGD20K supplies a heatmap for each image and affordance verb, and its official metrics compare predicted and ground-truth heatmap distributions. Every model in Table 5 emits points rather than a heatmap, so we score containment instead. The heatmap is binarised at half its peak value. Each connected region of the result becomes one ground-truth target, since one verb can apply to several instances in an image. Predicted points are matched to these targets one-to-one and greedily, as boxes are matched in § 4.1, and a match holds when the point falls inside its target. The label a model attaches to a point is ignored, so extra points count against precision. Recall and precision are computed over all matches, and F1 is their harmonic mean.

Stratified Instance VP Across All Baselines

Table 6 extends the stratification of § 4.3 to every MLLM baseline. The split yields 689 Easy and 651 Hard queries on SketchyCOCO, and 2452 Easy and 302 Hard queries on FS-COCO. Base Rex-Omni performs near floor on this task, as Table 1 shows, and is omitted. Every model loses F1 from Easy to Hard. Sketch-Omni leads every MLLM baseline on both Hard subsets and on FS-COCO-Easy, and is on par with Qwen3-VL-8B on SketchyCOCO-Easy.

Table 6: Stratified instance VP, all baselines. Each cell is SketchyCOCO / FS-COCO, same protocol as Table 2.

MethodSubset[email protected][email protected][email protected]
Qwen2.5-VL-3BEasy75.7/57.470.3/52.664.9/48.8
Qwen2.5-VL-3BHard42.3/43.536.3/39.132.1/37.2
Qwen3-VL-8BEasy91.0/86.990.2/84.386.7/79.5
Qwen3-VL-8BHard50.5/66.247.8/63.143.4/56.8
MiMo-VL-7BEasy83.9/51.982.9/48.979.0/43.0
MiMo-VL-7BHard47.3/30.543.8/28.939.8/26.1
DeepSeek-VL2-SEasy65.2/45.162.8/37.758.3/27.0
DeepSeek-VL2-SHard42.0/30.339.1/25.035.9/19.3
Sketch-OmniEasy91.7/94.889.5/92.786.1/88.3
Sketch-OmniHard55.3/83.351.4/80.348.2/77.0

Table 6. Stratified instance VP, all baselines. Each cell is SketchyCOCO / FS-COCO, same protocol as Table 2.

Stage-A Alignment on the Sketch-Native Tasks

Table 7 evaluates the w/o align configuration of Table 1 on the five sketch-native tasks, isolating the alignment’s effect on the tasks it operates on during Stage A. The largest gain falls on object referring, 4.3 points. This is consistent with the role of Lcross\mathcal{L}_{\mathrm{cross}}, which aligns how a phrase selects its region.

Table 7: Stage-A alignment on the sketch-native tasks. Detection and referring report [email protected], pointing F1@Point, affordance F1, all under the protocol of the main tables. Referring uses the test splits.

MethodDetection FS-COCODetection COCOPointing FS-COCOPointing COCOReferring objReferring spatAffordance SeenAffordance Unseen
Sketch-Omni (full)86.748.189.944.574.676.766.274.3
w/o align86.345.289.242.470.373.163.970.8

Table 7. Stage-A alignment on the sketch-native tasks. Detection and referring report [email protected], pointing F1@Point, affordance F1, all under the protocol of the main tables. Referring uses the test splits.

Training Details

Supervised stages. Both stages fit the LoRA update Δθ\Delta\theta with a single AdamW optimiser at an initial learning rate of 5×10−55 \times10^{-5} with cosine decay, weight decay 0.01, and gradient-norm clipping at 1.0, at an effective batch size of 64. The adapters use scaling factor 64 and no dropout. Training runs in bfloat16 with gradient checkpointing under ZeRO stage 1, for about 1 day on 32 H100 GPUs.

Task mixture. Stage A and Stage B alternate in blocks of 50 optimiser steps, two Stage-A blocks to every three Stage-B blocks. Within a block the tasks are sampled at the rates below, which give the share of all optimiser steps in the last column. Object and spatial referring share the referring stream and together receive probability 0.25.

StageTaskWithin stageOverall
Adetection0.250.10
Apointing0.250.10
Areferring0.250.10
Aaffordance0.250.10
Binstance VP, stitched canvases1/30.20
Binstance VP, crop queries1/30.20
Bcomposited-scene detection1/60.10
Bcomposited-scene pointing1/60.10

Table 1 (PDF p. 19).

GRPO refinement. The final GRPO pass follows Rex-Omni’s recipe and runs for about 8 hours on 32 H100 GPUs on the same task mixture as the supervised stages, with 8 rollouts per prompt at temperature 1.0 and top-pp 0.99, a global batch size of 32, learning rate 5×10−65 \times10^{-6}, PPO clip 0.2, and KL coefficient 0.01. Box tasks are rewarded by the IoU between the predicted and ground-truth boxes, and pointing tasks by whether the predicted point lands inside the ground-truth box or mask.

References

  1. [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a.arxiv.org/abs/2511.21631
  2. [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025b.arxiv.org/abs/2502.13923
  3. [4]Ahmed Bourouis, Judith Ellen Fan, and Yulia Gryaditskaya. Open vocabulary semantic scene sketch understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2312.12463
  4. [5]Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal LLM’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.arxiv.org/abs/2306.15195
  5. [6]Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2109.10852
  6. [7]Pinaki Nath Chowdhury, Aneeshan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song. FS-COCO: Towards understanding of freehand sketches of common objects in context. In European Conference on Computer Vision (ECCV), 2022.
  7. [8]Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. What can human sketches do for object detection? In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2303.15149
  8. [9]Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and PixMo: Open weights and open data for state-of-the-art vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
  9. [10]Sounak Dey, Pau Riba, Anjan Dutta, Josep Lladós, and Yi-Zhe Song. Doodle to search: Practical zero-shot sketch-based image retrieval. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  10. [11]Mathias Eitz, James Hays, and Marc Alexa. How do humans sketch objects? Acm Transactions on Graphics (TOG), 2012.DOI
  11. [12]Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. SketchyCOCO: Image generation from freehand scene sketches. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  12. [13]Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seedl.5-VL technical report. arXiv preprint arXiv:2505.07062, 2025.
  13. [14]Rishi Gupta, Mukilan Karuppasamy, Shyam Marijit, Aditay Tripathi, and Anirban Chakraborty. O3SLM: Open weight, open data, and open vocabulary sketch-language model. In AAAI Conference on Artificial Intelligence (AAAI), 2026. Preprint. Under review.
  14. [15]David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1704.03477
  15. [16]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.09685
  16. [17]Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-Rex2: Towards generic object detection via text-visual prompt synergy. In European Conference on Computer Vision (ECCV), 2024a.arxiv.org/abs/2403.14610
  17. [18]Qing Jiang, Gen Luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. ChatRex: Taming multimodal LLM for joint perception and understanding. arXiv preprint arXiv:2411.18363, 2024b.arxiv.org/abs/2411.18363
  18. [19]Qing Jiang, Junan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, and Lei Zhang. Detect anything via next point prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.arxiv.org/abs/2510.12798
  19. [20]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2304.02643
  20. [21]Subhadeep Koley, Tapas Kumar Dutta, Aneeshan Sain, Pinaki Nath Chowdhury, Ayan Kumar Bhunia, and Yi-Zhe Song. SketchFusion: Learning universal sketch features through fusing foundation models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
  21. [22]Fengyin Lin, Mingkang Li, Da Li, Timothy Hospedales, Yi-Zhe Song, and Yonggang Qi. Zero-shot everything sketch-based image retrieval, and in explainable style. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2303.14348
  22. [23]Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.arxiv.org/abs/1405.0312
  23. [24]Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable MLLMs to comprehend what you want. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2403.20271
  24. [25]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2303.05499
  25. [26]Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning affordance grounding from exocentric images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.arxiv.org/abs/2203.09905
  26. [27]Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.arxiv.org/abs/1511.02283
  27. [28]Simon McIntosh-Smith, Sadaf R. Alam, and Christopher Woods. Isambard-AI: a leadership class supercomputer optimised specifically for artificial intelligence. arXiv preprint arXiv:2410.11199, 2024.
  28. [29]Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.arxiv.org/abs/1904.05068
  29. [30]Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. In International Conference on Learning Representations (ICLR), 2024. Preprint. Under review.arxiv.org/abs/2306.14824
  30. [31]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00020
  31. [32]Zanxi Ruan, Songqun Gao, Qiuyu Kong, Yiming Wang, and Marco Cristiani. StructXLIP: Enhancing vision-language models with multimodal structural cues. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.arxiv.org/abs/2602.20089
  32. [33]Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, and Yi-Zhe Song. Cross-modal hierarchical modelling for fine-grained sketch based image retrieval. In British Machine Vision Conference (BMVC), 2020.arxiv.org/abs/2007.15103
  33. [35]Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.DOI
  34. [36]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepskmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
  35. [37]Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, and Maneesh Agrawala. Instance segmentation of scene sketches using natural image priors. In SIGGRAPH Conference Papers ’25, 2025.DOI
  36. [38]Aditay Tripathi, Rajath R. Dani, Anand Mishra, and Anirban Chakraborty. Sketch-guided object localization in natural images. In European Conference on Computer Vision (ECCV), 2020.arxiv.org/abs/2008.06551
  37. [39]Aditay Tripathi, Anand Mishra, and Anirban Chakraborty. Query-guided attention in vision transformers for localizing objects using a single sketch. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024.arxiv.org/abs/2303.08784
  38. [40]Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.arxiv.org/abs/1907.09682
  39. [41]Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. DeCLIP: Decoupled learning for open-vocabulary dense perception. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2505.04410
  40. [42]Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyu Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. CogVLM: Visual expert for pretrained language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2311.03079
  41. [43]Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. DeepSeek-VL2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024.
  42. [44]Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  43. [45]Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In IEEE International Conference on Computer Vision (ICCV), 2015.
  44. [46]Peng Xu, Timothy M. Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. Deep learning for free-hand sketch: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.arxiv.org/abs/2001.02600
  45. [47]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyu Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441, 2023. Preprint. Under review.
  46. [48]Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Referer and ground anything anywhere at any granularity. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.07704
  47. [49]Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen-Chang Loy. Sketch me that shoe. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.DOI
  48. [50]Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. RoboPoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406.10721, 2024.arxiv.org/abs/2406.10721
  49. [51]Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, et al. MiMo-VL technical report. arXiv preprint arXiv:2506.03569, 2025.arxiv.org/abs/2506.03569
  50. [52]Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. In European Conference on Computer Vision (ECCV), 2024.arxiv.org/abs/2311.14552
  51. [53]Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2023.
  52. [54]Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for referring and grounding with large language models. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2404.07973
  53. [55]Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong, Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, et al. RoboRefer: Towards spatial referring with reasoning in vision-language models for robotics. Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2506.04308
  54. [56]Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2304.06718

Paper details

vlnc:W_2aavpsz9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 14

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents