WorksComputer Science
LocateAnyMesh: Grounding 3D Mesh Queries in 2D Images with Vision-Language Models
Abstract
Textured 3D meshes specify object geometry and appearance, making them a natural query for finding a particular asset in an image. We introduce LocateAnyMesh, a vision-language model that uses the mesh itself as part of its prompt. A frozen 3D encoder maps colored surface points to a compact token sequence, which a learned connector places alongside image and instruction tokens. We evaluate two grounding models with PPAT mesh tokens: LAM-Select selects image-region proposals, while LAM-Direct predicts coordinates directly. Neither requires rendered query views or view-template matching. New objects require a single encoder pass and no object-specific fine-tuning. LAM-Select improves localization on the reported real-image suites and the official BOP YCB-V evaluation. Secondary experiments compare mesh encoders, and a LAM-Direct study isolates surface color. A single-seed LAM-Select continuation supplies pointing and mesh-only description evidence without replacing the reported localization model. The results support mesh tokens as a query representation, with remaining limitations in presence decisions, identity binding, proposal coverage, and descriptive fidelity.
Introduction
Grounding VLMs such as Florence-2, Rex-Omni, and LocateAnything generate spatial coordinates from images and language prompts [13, 26, 29]. A textured mesh specifies an asset’s geometry and surface color directly. A phrase such as the green figurine may identify one target in Figure 1, but distinguishing similar assets in words can require a much longer query. We ask whether a vision–language model can use the mesh itself to locate an asset in a scene or describe it.
Figure 1. A 3D mesh is a direct input to a VLM. Mesh tokens encode geometry and surface color; mesh views visualize the assets for the reader. (a) Localization in an unseen HomebrewedDB scene [14]. (b) Pointing to the queried fan in a synthetic ABO scene. (c) A literal first-sentence excerpt of a mesh-only description. All three panels use LAM-Direct, initialized from LocateAnything-3B: a localization checkpoint for (a) and a separate multitask checkpoint for (b,c). These selected illustrations are not performance estimates; errors and controlled evaluations are reported in Section C.3.
LocateAnyMesh conditions a grounding VLM on direct mesh input. We compare two variants, LAM-Direct and LAM-Select, initialized from the released LocateAnything-3B [26] and VLX-Seek-1.5-10B (om-ai-lab, 2026) grounding checkpoints, respectively. Both use PPAT mesh tokens. A frozen OpenShape-RGB PointBERT encoder [17, 31] maps the colored surface to 65 tokens; a trainable MLP projects them into the same sequence as the instruction and any image tokens (Fig. 3). We name the variants by their output mechanism: LAM-Direct generates box coordinates, whereas LAM-Select chooses among detector proposals.
| Variant | Base model | LLM | Params. | Vision encoder(s) | Output | External detector at eval. |
| LAM-Direct | LocateAnything-3B | Qwen2.5-3B-Instruct | 3B class | MoonViT-SO-400M | Box coordinates | None |
| LAM-Select | VLX-Seek-1.5-10B | Qwen3.5 | 10B class | Qwen3.5-VL + C-RADIO | Region references | WeDetect-Base-Uni, top 100 |
Table 1. The two LocateAnyMesh variants, named by output mechanism. Parameters are released full-system size classes, not adapter-inclusive audits. Both variants use frozen PPAT mesh tokens.
Model-based detectors also accept a CAD asset and an image, but render the asset into views and match their descriptors to image regions [15, 19, 24]. We feed mesh tokens directly to the language decoder. LAM-Select selects proposals from an image-only detector; the query mesh does not need rendered views. Each new object requires one encoder pass.
Our main localization benchmark compares both systems on held-out HOPE, LM-O, and HB objects and on seen YCB-V objects. Every YCB-V detection AP uses the official 900-image BOP target list with amodal scoring (§ 5).
A single-seed multitask continuation of the LAM-Select localization checkpoint provides the main pointing and mesh-only description studies (Figure 7 and Figure 8). It is used only for capability evidence; the localization tables retain the original checkpoint. Separate LAM-Direct capability studies and the original LAM-Select transfer probes remain in Section C.3.

Figure 7. Mesh-only descriptions from R and M10k@10k. Fixed before output inspection: zero-based positions 0, 37, 75, 112, 150, 187, 225, 262 in the evaluation-ID sort of all 300 identities, without quality ranking. Each row shows view_0 for the reader and the literal correct-mesh responses; only terminal EOS is omitted. The models receive mesh tokens and the instruction, but no image or render.

Figure 8. M10k@10k native pointing on real BOP images. The fixed gallery takes the first two present query IDs in each suite after sorting, without using outcomes: three are hits and five are misses. Text is the literal model response with only terminal EOS omitted. Coordinates use the size-minus-one pixel convention. Orange points and green visible-mask outlines determine the displayed hit/miss labels; the crops, masks, and labels are evaluation-only reader aids and were not model inputs.
Contributions.
Direct mesh input to two grounding VLMs. A shared PPAT query representation conditions proposal selection or coordinate generation through a learned connector (§ 3).
Representation comparisons and grounding errors. Within LAM-Select, we compare PPAT and Uni3D-S and audit proposal, presence, and identity errors. A matched LAM-Direct intervention tests query color (Sections B.4, 5.3 and 5.4).
Related Work
Generative grounding VLMs. Florence-2 [29] unifies captioning, detection, and grounding through task prompts and sequence generation. Rex-Omni [13] combines coordinate-token outputs with text and image-region prompts. LocateAnything [26] predicts the coordinates within each box in parallel. VLX-Seek (om-ai-lab, 2026) uses region proposals for grounding. LocateAnyMesh extends this line of work on the input side: native mesh tokens condition proposal selection in LAM-Select and direct-coordinate generation in LAM-Direct.
Figure 2 separates the supplied query from its network representation. CNOS [19], SAM-6D [15], and CAD-Prompted SAM3 [24] accept a CAD asset and a scene image, a setting evaluated by BOP [10], but use rendered query views. CAD-Prompted SAM3 returns instance masks that also define boxes. Universal-prompt detectors [2, 27] accept 2D prompts. Exemplar matching methods such as OS2D and the visual-prompt modes of T-Rex2 condition on 2D examples [12, 21]. PointLLM [30] generates language from colored point clouds, while Kyvo [23] models structured 3D scenes and evaluates both synthetic and real-world settings. Here, the supplied 3D asset serves as a query for image grounding.
Method
Task. Mesh-prompted grounding takes an RGB image and a textured 3D asset as input, and localizes matching instances with boxes or explicitly refuses an absent query. Queries identify individual assets, including objects unseen during grounding training.

Figure 2. Same CAD input, different network representations. Model-based detection and LocateAnyMesh ground a supplied CAD asset in a scene image using rendered views and native 3D tokens, respectively.
Figure 3. LAM-Select, initialized from VLX-Seek-1.5-10B. A frozen 3D encoder and a trainable connector place mesh features alongside image, region, and instruction tokens. An image-only detector supplies candidate boxes. The Qwen3.5 decoder generates region IDs or an explicit refusal; selected IDs map back to the candidate boxes. Gray modules are frozen; orange modules are trained.
We implement LocateAnyMesh with two pretrained grounding systems, both using frozen PPAT mesh tokens. LAM-Direct initializes from the released LocateAnything-3B checkpoint Wang et al. [26], inheriting its Qwen2.5-3B-Instruct decoder, MoonViT image encoder, and direct-coordinate output. LAM-Select initializes from the released VLX-Seek-1.5-10B checkpoint om-ai-lab (2026), inheriting its Qwen3.5 text decoder, image towers, region encoder, and autoregressive region-ID output. A learned connector adds mesh features to each system’s input sequence. We name the variants by output mechanism: LAM-Direct generates box coordinates directly, whereas LAM-Select selects detector proposals. They also differ in base model, scale, language model, vision towers, and training; this is a complete-system comparison, not an isolated language-model ablation. Table 6 and Table 7 compare their output representations and training settings.
| LAM-Select + PPAT | LAM-Direct + PPAT | |
| Added tokens | <MESH_CONTEXT> (id 248182) | <MESH_CONTEXT>, <mesh>, </mesh> (ids 152681–152683) |
| Mesh span | 65 <MESH_CONTEXT> positions | <mesh>, 65 <MESH_CONTEXT> positions, </mesh> |
| Visual input | Native image tokens and one <objN><objfeat> pair per candidate | Inherited <img>, <IMG_CONTEXT> tokens |
| Output | <ground>mesh</ground> then <objects> enclosing region IDs, followed by </objects> | <box><x1><y1><x2><y2></box>, integer coordinates in [0, 1000] |
| Multiple boxes | Multiple <objN> references within the object span | Concatenated <box>...</box> blocks; main benchmark caps output at one box |
| Absent | None inside the object span | <box>none</box>; plain-text none, id 6697 |
| Request | All candidate regions matching the mesh | Single-instance locate template |
Table 6. Token contracts for the two PPAT systems.
| LAM-Select + PPAT | LAM-Direct + PPAT (arm A) | |
| Initialization | VLX-Seek-1.5-10B | LocateAnything-3B |
| Text decoder | Qwen3.5, 32 layers, width 4,096 | Qwen2.5, 36 layers, width 2,048 |
| Frozen mesh features | PPAT, 65 tokens of width 512 | PPAT, 65 tokens of width 512 |
| Connector widths | ||
| LoRA / / dropout | 16 / 32 / 0 | 64 / 128 / 0.05 |
| LoRA targets | All language-model linear layers | + gate, up, down |
| Optimizer / decay | AdamW / 0.01 | AdamW / 0 |
| Peak learning rate | ||
| Warmup updates | 1,000 | 916 |
| Schedule / horizon | Linear / 523,223 updates | Cosine / 45,823 updates |
| Reported checkpoint | 31,434 updates | 45,823 updates |
| Per-GPU batch / accum. | 1 query / 1 | 1 pack / 1 |
| Global batch | 8 queries | 8 packs |
| Delivered exposure | 251,472 queries | 366,584 packs |
| Sequence caps | 4,096 tokens/query | 8,192/pack; 4,096/sample |
| Precision / clip / seed | BF16 / 1.0 / 42 | BF16 / 1.0 / 42 |
| Activation checkpointing | Language layers | Non-reentrant |
| Training loss | Answer-token cross entropy | Coordinate weight 5.0; null 0.1; box-reg 1.0 |
Table 7. Training settings of the two headline PPAT systems.
From a textured mesh to 65 tokens. We sample ten thousand colored points from the mesh surface and normalize their coordinates to the unit sphere. A frozen OpenShape-RGB PointBERT encoder, PPAT-B32 Liu et al. [17]; ); Yu et al. [31], maps their coordinates and RGB values to 65 tokens. A trainable LayerNorm–MLP connector maps each width-512 token to the LAM-Select decoder’s width of 4,096. A secondary Uni3D-S Zhou et al. [32] comparison is reported in Section B.4. New assets require one encoder pass; its tokens can be cached for reuse. Mesh onboarding is excluded from downstream inference timings and remains unmeasured on GB200 (Section A.5).
Image regions and decoding. At inference, WeDetect-Base-Uni supplies up to 100 image-only candidate boxes. LAM-Select’s frozen image towers and native region encoder produce a feature token for each candidate. The language decoder receives image context, region IDs and features, the mesh span, and the instruction. Causal @page 3/36 starts_mid=0 ends_mid=1

Figure 2 (PDF p. 3). Same CAD input, different network representations. Model-based detection and LocateAnyMesh ground a supplied CAD asset in a scene image using rendered views and native 3D tokens, respectively.
Figure 3 (PDF p. 3). LAM-Select, initialized from VLX-Seek-1.5-10B. A frozen 3D encoder and a trainable connector place mesh features alongside image, region, and instruction tokens. An image-only detector supplies candidate boxes. The Qwen3.5 decoder generates region IDs or an explicit refusal; selected IDs map back to the candidate boxes. Gray modules are frozen; orange modules are trained.
We implement LocateAnyMesh with two pretrained grounding systems, both using frozen PPAT mesh tokens. LAM-Direct initializes from the released LocateAnything-3B checkpoint Wang et al. [26], inheriting its Qwen2.5-3B-Instruct decoder, MoonViT image encoder, and direct-coordinate output. LAM-Select initializes from the released VLX-Seek-1.5-10B checkpoint om-ai-lab (2026), inheriting its Qwen3.5 text decoder, image towers, region encoder, and autoregressive region-ID output. A learned connector adds mesh features to each system’s input sequence. We name the variants by output mechanism: LAM-Direct generates box coordinates directly, whereas LAM-Select selects detector proposals. They also differ in base model, scale, language model, vision towers, and training; this is a complete-system comparison, not an isolated language-model ablation. Table 6 and Table 7 compare their output representations and training settings.
From a textured mesh to 65 tokens. We sample ten thousand colored points from the mesh surface and normalize their coordinates to the unit sphere. A frozen OpenShape-RGB PointBERT encoder, PPAT-B32 Liu et al. [17]; ); Yu et al. [31], maps their coordinates and RGB values to 65 tokens. A trainable LayerNorm–MLP connector maps each width-512 token to the LAM-Select decoder’s width of 4,096. A secondary Uni3D-S Zhou et al. [32] comparison is reported in Section B.4. New assets require one encoder pass; its tokens can be cached for reuse. Mesh onboarding is excluded from downstream inference timings and remains unmeasured on GB200 (Section A.5).
Image regions and decoding. At inference, WeDetect-Base-Uni supplies up to 100 image-only candidate boxes. LAM-Select’s frozen image towers and native region encoder produce a feature token for each candidate. The language decoder receives image context, region IDs and features, the mesh span, and the instruction. Causal autoregression emits one or more region IDs, or None. Each selected ID returns its proposal’s coordinates. The mesh is never rendered for the model, and ground-truth boxes are not used to form inference candidates. Localization recall is limited by proposal coverage as well as the decoder’s selections.
LAM-Select training. We train the mesh connector and language-model LoRA adapters [11]; the base weights, image towers, region encoder, and mesh encoder remain frozen. Training prompts request all matching instances, with explicit absent queries. Training candidates are ground-truth visible boxes, and the objective is causal cross entropy on the answer and termination tokens. This creates a candidate-distribution difference from evaluation, which uses detector proposals. The LAM-Select reference uses 31,434 updates; Section A.1 gives its training settings and encoder control.
Experimental Setup
Training data
We train on Google Scanned Objects [7], Amazon Berkeley Objects [3], two UID-disjoint Objaverse 1.0 tabletop waves [4], and BOP YCB-V [5, 9, 28]. The LAM-Select recipe also includes an Objaverse PPAT union. Its PPAT and Uni3D runs share query order and an absent-sampling probability of 0.25, with localization supervision only. Source mixtures and sampling details are in Table 5 and Section A.4; Figure 4 shows examples and mesh categories.
Figure 4. Training examples and mesh categories. Examples and the asset inventory come from the earlier LAM-Direct data analysis. Blue boxes mark training targets; GSO meshes are scanned. The histogram counts each eligible mesh ID once across all sources, including YCB-V. Native labels are supplemented by categories inferred from descriptions; smaller groups form “Other categories” (Section A.4).
| Source | Prepared query rows |
| GSO v2 | 1,106,785 |
| ABO | 978,042 |
| YCB-V PBR | 703,878 |
| Objaverse PPAT union | 627,079 |
| YCB-V real | 622,876 |
| Objaverse tabletop v2tex | 73,784 |
| Objaverse tabletop wave2 | 73,340 |
| Total | 4,185,784 |
Table 5. LAM-Select prepared training manifest. Weighted query rows sampled with replacement. LAM-Select’s PPAT reference and Uni3D control consume the same order; the reference checkpoint consumes only part of the manifest. These counts are not unique-image or unique-mesh counts and do not describe LAM-Direct’s separately sampled training recipe.
Held-out splits and contamination. Synthetic recipes define object-level held-out splits; YCB-V retains its 21 seen identities. An embedding-space audit finds duplicate assets across training and evaluation IDs in all three simulated suites and may miss others (Section A.4). The retained synthetic scores use the original splits, so they do not establish fully held-out identity transfer. The audit flags no pairs in the real BOP suites; BOP suites other than YCB-V are evaluation-only.
Evaluation protocol
Evaluation protocol
We evaluate HOPE (Tyree et al., [25]), LM-O (Brachmann et al., [1]), HB (Kaskman et al., [14]), and YCB-V. Table 9 and Section A.6.1 give the cohorts and sampling rules. Each query pairs an image with a mesh; we query present classes and up to 8 seeded absent classes per image, or all 21 models on YCB-V.
| Suite | Split | Obj. | Scenes | Images eval. | Present queries | LAM-Select + PPAT | LAM-Direct + PPAT |
| YCB-V BOP | official targets | 21 | 12 | 900 | 4,123 | ✓ | ✓ |
| LM-O | lmo/test | 8 | 1 | 300 | 2,319 | ✓ | ✓ |
| HOPE | hope/val | 28 | 10 | 50 | 775 | ✓ | ✓ |
| HB | hb/val_prim. | 33 | 13 | 300 | 1,553 (2,400 abs.) | ✓ | ✓ |
| GSO/ABO/Objv. | h/o renders | 97 / 405 / 385 | – | 300 | – | ✓ |
Table 9. Evaluation cohorts and coverage of the two PPAT systems. A check marks a measured result; means not evaluated in the accepted record. Shared cohort counts alone do not establish parity; saved image/query inputs and the scorer have been matched for HOPE, LM-O, and HB. Absent queries use up to eight sampled classes per image; YCB-V queries all 21 models under official BOP amodal scoring; the other rows use the internal visible-box protocol. HB and HOPE use validation splits with public ground truth. Simulated rows retain the original splits and the contamination limitation in Section 4.
Detection and scoring. LAM-Select uses greedy autoregressive proposal selection and retains all selected boxes; a top-one diagnostic keeps the highest-scored selection. LAM-Direct uses the single-instance prompt and caps decoding at one box. Invalid outputs contribute no detections but count as failures, not valid refusals. The reported LAM-Select localization evaluations contain no invalid answers.
We report COCO , macro-averaged over classes (Lin et al., [16]). AP confidence comes from decoder-token probabilities, not detector scores, and is not calibrated as a presence probability (Section A). The house protocol uses modal boxes for instances at least 10% visible, pooling scores per dataset. Official YCB-V results use the 900-image target list, mask-derived amodal boxes, and ignored low-visibility instances (Section A.6.2). Only this official protocol supports comparison with BOP leaderboard AP; house scores use different images and box targets.
We also report present-target hits and absent-query false positives. The supplemental presence-discrimination ratio excludes HB development scene 000001 (Eq. (1)).
Uncertainty. Neither headline system has a multi-seed evaluation. Results are tied to their checkpoint and inference route, without seed-based confidence intervals; small rerun differences do not estimate training uncertainty.
Results
Localization and inference cost
With frozen PPAT mesh tokens, LAM-Select improves house mAP over LAM-Direct on the three held-out real-image suites (Table 2). This comparison changes backbone, proposals, training exposure, and output cardinality; it evaluates complete systems rather than isolating the backbone. The secondary PPAT–Uni3D-S comparison is in Section B.4.
| System | HOPE | LM-O | HB |
| LAM-Direct | 0.155 | 0.181 | 0.303 |
| LAM-Select | 0.371 | 0.388 | 0.366 |
Table 2. Two grounding systems with PPAT mesh tokens. House visible-box COCO mAP on 50 HOPE and 300 images per other suite; HB includes all 13 scenes. LAM-Select uses optimized autoregressive inference and retains all selected boxes; its top-one HOPE mAP is 0.319. Other LAM-Select cells are unchanged by top-one filtering on this cohort. LAM-Direct uses one-box decoding; image/query inputs and the scorer are matched on all three suites. These are internal-protocol results; Table 3 reports official BOP YCB-V separately.
Official BOP YCB-V. On the official 900-image target list, LAM-Select reaches amodal AP of 0.7152 with PPAT, compared with 0.6384 for LAM-Direct with the same mesh encoder (Table 3). YCB-V identities are seen during training, so these results measure test-image performance rather than transfer to unseen objects.
| Grounding system | AP | AP50 | AP75 | GB200 s/img , | Repeat range s/img |
| LAM-Direct | 0.6384 | 0.7885 | 0.7112 | 0.322 | 0.309–0.324 |
| LAM-Select | 0.7152 | 0.9150 | 0.8183 | 0.682 | 0.652–0.837 |
Table 3. Official BOP YCB-V detection with PPAT. Both systems use the same 900 target images across 12 scenes and query all 21 objects per image (18,900 queries). Ground truth uses mask-derived amodal boxes and BOP low-visibility ignore handling. AP is COCO AP@[.50:.05:.95]. LAM-Select uses its 31,434-update checkpoint; LAM-Direct uses arm A at 45,823 updates. These complete-system comparisons are not compute-matched. Timing is the reported GB200 batch-21 pilot on eight images: approximate seconds/image is 21/median queries/s, the median of three repeat-level cohort means; the range spans those repeats. Timing and AP use different cohorts; LAM-Direct timing uses 10,000 updates, and LAM-Select timing uses cached proposals. The full encoder comparison is in Table 14; timer boundaries and malformed-answer accounting are in Section B.6.
Blackwell inference cost. On the reported eight-image GB200 pilot, batching increases throughput for both systems (Table 17 and Table 18). Timings exclude mesh encoding and live proposal detection; LAM-Select uses cached proposals. Figure 5 combines these pilot times with the separate official AP evaluation. For LAM-Direct, accuracy and timing also use different checkpoints.
Figure 5. YCB-V accuracy and runtime context with Blackwell timings for our models. Seen- and unseen-object leaderboard points have separate external-only Pareto frontiers. Our PPAT models use official 900-image BOP AP on the vertical axis and reported GB200 batch-21 time from the eight-image pilot on the horizontal axis (21/median queries/s). These coordinates come from different cohorts; LAM-Direct also uses different checkpoints for accuracy (45,823 updates) and timing (10,000). LAM-Select uses 31,434 updates for both, with cached proposals in the timed pilot. External methods retain their reported hardware and timer boundaries. Our models saw YCB-V objects during training and are excluded from both frontiers. This is reference context, not a controlled speed–accuracy comparison. Table 14 reports Uni3D-S accuracy and timing.
| System | Encoder | Queries/s | Queries/s | Approx. s/img | Peak alloc. GiB |
| LAM-Direct | PPAT | 7.89 | 65.3 | 0.322 | 7.9–8.5 |
| LAM-Direct | Uni3D-S | 7.70 | 59.9 | 0.351 | 7.9–8.4 |
| LAM-Select | PPAT | 2.07 | 30.8 | 0.682 | 21.4 |
| LAM-Select | Uni3D-S | 1.57 | 23.8 | 0.882 | 21.4 |
Table 17. Reported Blackwell GB200 timing. Eight images, 21 queries per image, three repeats. Throughput is the median across repeats. Approximate batch-21 seconds/image is 21 divided by the reported throughput, the median of the repeat-level cohort means, not the median of individual-image latencies. Peak allocated memory covers the reported batch sweep, including loading and warmup. Baseline timings retain malformed answers: 2/168 queries for PPAT and 1/168 for Uni3D-S per repeat. Full batch results and timer boundaries are in Section B.6.
| System | Encoder | range | |||||
| LAM-Direct | PPAT | 7.89 | 21.5 | 35.8 | 44.2 | 65.3 | 64.9–67.9 |
| LAM-Direct | Uni3D-S | 7.70 | 20.4 | 33.9 | 44.4 | 59.9 | 52.7–63.4 |
| LAM-Select | PPAT | 2.07 | 6.86 | 12.5 | 16.5 | 30.8 | 25.1–32.2 |
| LAM-Select | Uni3D-S | 1.57 | 5.45 | 9.9 | 14.1 | 23.8 | 22.8–29.8 |
Table 18. Table 18: Reported GB200 throughput by query-batch size. Median queries/s across three repeats on the same eight-image cohort. Batch size counts mesh queries sharing one image. The final column is the observed minimum–maximum over the three batch-21 repeats; ranges for other batches were not supplied. Malformed baseline answers remain in the query count.
Several detectors on BOP’s seen-object board (Hodan et al., [10]) report higher AP and lower runtime than our models, with different hardware and timer boundaries. Our models saw YCB-V objects during training; their AP does not establish unseen-object leaderboard superiority. Transfer evidence comes from the held-out real suites (Table 2); leaderboard scope is in Section B.7.
Multitask capability continuation
All localization results above continue to use R, the 31,434-update LAM-Select checkpoint. For capability figures only, we continue the same weights for 10,000 updates with an 80/15/5 locate/describe/point mixture. This
M10k@10k run has one seed and does not replace R in any localization table.
Mesh-only description. Neither R nor M10k@10k receives an image for this task. In a judge-assisted, blinded review of 300 held-out-caption objects, R already produces grounded, non-generic mesh-only descriptions on 52.7% of correct-mesh inputs. M10k@10k reaches 63.0%, a paired gain of 10.3 percentage points with a 95% identity-grouped bootstrap interval of [3.7, 16.7]. Hidden positive and negative controls score 86.7% and 5.7%, respectively. These are assistant judgments, not human accuracy; protocol and controls are in Section C.3.1.
Native pointing. On 2,000 present and 533 absent real-BOP queries, M10k@10k has 99.6% strict-format validity, 35.7% present visible-mask hit rate, and 26.6% absent false positives. The present hit rate is 60.6% on seen YCB-V objects; the unseen LM-O, HB-validation, and HOPE-validation suites reach 17.8%, 29.6%, and 34.8%. Thus the continuation supplies native point answers, but unseen-object pointing and absence rejection remain weak.
Earlier LAM-Direct capability studies and R’s image-plus-mesh description and single-prompt point probes remain in Section C.3. The capability continuation does not change the reported localization model.
Query color ablation
ISO-1 isolates query-surface color by setting every point’s RGB to gray. The two runs share code, configuration, training steps, frozen encoder, and sampled geometry; only query color changes. They use the earlier LAM-Direct in-graph stack, so this intervention does not estimate the color effect in LAM-Select.
Removing color reduces mAP by 79.6% on LM-O, 57.8% on HOPE, and 27.2% on HB (Table 19). This matched intervention supports a role for surface color under the tested encoder and recipe. One run per arm does not estimate the effect across seeds. This pair has no official BOP YCB-V evaluation. Figure 12 illustrates the intervention on two similarly shaped statues.
| Query point cloud | LM-O | HOPE | HB | Discrim | GSO | ABO | Objv. |
| Color twin | 0.198 | 0.122 | 0.244 | 2.356 | 0.266 | 0.302 | 0.257 |
| Gray twin | 0.040 | 0.052 | 0.177 | 1.798 | 0.124 | 0.241 | 0.208 |
| gray vs. color |
Table 19. Table 19: LAM-Direct ISO-1 color and gray twins. Identical configuration, LoRA , and 10,000 updates; token stores differ only in –force-gray. HB mAP uses strided-300 over 13 scenes; Discrim uses scenes 2–13. Both arms re-score bit-identically; training-run uncertainty remains (§ 4.3). Both checkpoints use the retired in-graph stack and are compared with each other. Neither headline checkpoint has a matched color/gray training pair under this protocol.
Figure 12. Query color separates similar statues. (a–f) Two mesh queries in one HB image; the gray twin gives identical outputs in (c,f). Only the ISO-1 twins isolate query color; arm A is shown for context. (g) A refusal in another frame. (h) All 23 frames containing both statues.
Proposal, presence, and identity errors
We audit native LAM-Select predictions on HB’s 276 images from 12 non-development scenes: 1,481 present and 2,208 absent queries (§ 4.3), using PPAT. Every saved selected box maps to its proposal, with zero invalid decodes. Figure 6 compares LAM-Select and LAM-Direct on matched inputs. The complete encoder comparison is in Table 10.
Figure 6. Two grounding systems on matched held-out HB queries. LAM-Direct and LAM-Select both use PPAT. Every column uses the same RGB image and mesh query. Blue boxes are predictions; dashed green boxes are visible targets. All predicted boxes are shown, and hits require IoU . Rows illustrate shared success, LAM-Select-only success, LAM-Direct-only success, a LAM-Select wrong-object selection, and absent-query disagreement. They were chosen by a fixed median rule within these outcome strata, not sampled to estimate success rates. Selection used all three systems; this view shows only the PPAT columns. Mesh previews are reader aids, not image inputs. Matched test inputs do not make this a controlled backbone ablation. More examples: Section C, including the same cases with Uni3D-S in Figure 10.
| Outcome | PPAT | Uni3D-S |
| Matched target instances | 747 | 826 |
| No matching proposal | 68 | 68 |
| Covered, but refused | 470 | 406 |
| Covered, but selection missed | 196 | 181 |
| Absent-query false positives | 311 | 352 |
Table 10. Full mesh-encoder comparison of LAM-Select HB errors. Native visible-box outputs on 276 non-development images at IoU , with maximum one-to-one matching. The first four rows partition 1,481 target instances; absent false positives use 2,208 queries. Both runs have zero invalid outputs and no multi-box answers on this cohort. Higher is better for matched instances; lower is better for errors.
Proposal coverage is not sufficient. At IoU 0.5, let be the number of target instances, the maximum one-to-one matching to available proposals, and the corresponding matching to selected boxes. Table 4 partitions into matched instances, unavailable proposal capacity (), and the remaining covered capacity (), split by refusal or nonempty selection. Most missed targets have usable proposals. This is an IoU-specific recall accounting, not an AP decomposition or a causal diagnosis: a valid refusal alone cannot distinguish failure to recognize the target from a decision not to commit.
| Outcome | Instances / queries |
| Matched target instances | 747 |
| No matching proposal | 68 |
| Covered, but refused | 470 |
| Covered, but selection missed | 196 |
| Absent-query false positives | 311 |
Table 4. LAM-Select with PPAT on non-development HB. Native visible-box outputs at IoU , with maximum one-to-one matching. The first four rows partition 1,481 target instances; absent false positives use a separate denominator of 2,208 queries. This run has zero invalid outputs and no multi-box answers on this cohort. Lower counts are better for errors, higher for matched instances.
Wrong-object selections. Among covered queries with no selected target hit, 130/196 PPAT outputs overlap another annotated object at IoU . These are annotation-level wrong-object matches, not semantic identity judgments. The corresponding diagnostics for HOPE and LM-O are in Section B.1.
Mesh substitution does not reliably redirect grounding. On a fixed query-swap diagnostic, both co-visible targets are followed correctly in only 1/18 and 5/18 pairs for LAM-Direct with PPAT and LAM-Select with PPAT, respectively. Each pair keeps the image and instruction fixed; LAM-Select also keeps its proposal set fixed. Many original source queries already fail, and these controls remain in the denominator. Changing the answer is insufficient: both answers must hit the corresponding target without hitting the other. Uni3D-S, exact-set, and absent-donor controls are reported in Section B.2. These selected cases expose unreliable paired grounding despite proposal coverage, not a population failure rate or a causal diagnosis of every miss.
Conclusion
LocateAnyMesh uses native mesh tokens to condition two grounding VLMs. With PPAT tokens, LAM-Select selects image-region proposals and improves house-protocol and official YCB-V accuracy over direct-coordinate LAM-Direct. Paired mesh substitutions expose unreliable target following even with proposal coverage. already generates mesh-only descriptions; a single-seed multitask continuation raises judge-assisted grounding from 52.7% to 63.0%. Its 35.7% present-mask point hit rate and 26.6% absent false-positive rate leave native pointing unreliable. These capability results do not replace as the reported localization model.
Implementation, Data, and Evaluation
LAM-Direct and LAM-Select adapt the complete pretrained LocateAnything-3B ([26]) and VLX-Seek-1.5-10B (om-ai-lab, 2026) systems, respectively. Earlier experiments retain their checkpoint-specific settings.
LAM-Select implementation
VLX-Seek-1.5-10B combines a Qwen3.5 decoder, its primary vision tower, and an auxiliary C-RADIO tower; native region encoding uses both towers. Our connector is LayerNorm(), Linear(, 4096), GELU, Linear(4096, 4096), with for PPAT and for Uni3D-S. The wider Uni3D connector adds 2,098,176 parameters. Pretrained weights and the added mesh-placeholder vocabulary row are frozen; projected mesh features replace its input embeddings. Table 7 gives decoder dimensions and LoRA settings.
The PPAT reference and Uni3D control share those settings and the same prepared query order. Both use the intermediate 31,434-update checkpoint, before the scheduled training horizon.
Targets and inference. The prepared localization task requests all instances and assigns targets to ground-truth visible-box candidates. Prompt labels are masked; answer-token cross entropy includes the terminal token. The answer format in Table 6 ends with <|im_end|>. Inference replaces training candidates with raster-ordered WeDetect proposals and uses no ground truth. The reported optimized route shares image, region, and prefix computation across groups of up to 21 mesh queries, with fused kernels and LoRA merged in memory. Box confidence is the FP32 softmax probability of the selected region-ID token at its autoregressive generation step.
LAM-Direct implementation
Table 6 and Table 7 compare the two PPAT systems. The details below describe LAM-Direct and its earlier ablation regimes. Its learning rates scale with the square root of the batch-size ratio; neither system uses vision LoRA.
Training hardware and software. We use NVIDIA A100-SXM4-80GB GPUs with the NVIDIA PyTorch 25.06 container, PyTorch 2.8.0a0, Transformers 4.57.1, PEFT 0.12.0, and DeepSpeed 0.15.4. Training uses ZeRO-1 optimizer sharding. AdamW uses and . Online packing uses a buffer of 32 samples; images are resized aspect-preservingly to a 1,280-pixel long edge. Other training settings are in Table 7.
LAM-Direct mesh encoding and decoding
Point sampling and color. The surface sampler is seeded by the mesh identifier’s CRC32, keeping sampled geometry fixed across arms. Each point receives the mean vertex color of its face; missing or constant colors use gray 0.5. ISO-1 applies this gray substitution with the geometry sample held fixed (§ 5.3).
Encoder and connector. PPAT-B32, the RGB PointBERT variant of OpenShape ([17, 31]), returns the full pre-pool sequence with its CLIP-alignment head ([22] removed. Set abstraction to 64 centroids and a 12-layer, width-512 transformer map to . The token-wise connector is LayerNorm, 512 2048, GELU, then 2048 2048. Uni3D ([32]) emits 65 width-1024 tokens; its connector’s parameter count is matched within 0.026%. The prompt order is image, instruction, mesh span, then assistant turn.
Training objective. Training is the base next-token objective with coordinate positions up-weighted 5× and a soft-argmax auxiliary that applies smooth- and GIoU to read off the coordinate logits (loss weights in Table 7).
Table 6: Token contracts for the two PPAT systems. Both use 65 frozen PPAT features, projected into their decoder width. Added vocabulary embeddings are frozen; projected features replace the mesh placeholders. LAM-Select selects proposed regions; LAM-Direct predicts coordinates. Point and selection templates listed below are LAM-Direct interfaces, not measured LAM-Select capabilities.
Table 7: Training settings of the two headline PPAT systems. LAM-Select is evaluated at an intermediate checkpoint; LAM-Direct uses the completed 100K arm A. “100K” denotes identity scale, not update count. A query and a packed training sequence are different exposure units; these are not compute-matched runs. For LAM-Direct, decoder LoRA and the connector train 124,982,272 of 3,668.3 M parameters (3.4%).
Attention, decoding, and score. Prompt attention is causal. A six-position spatial prediction block attends to the prompt and within the block, predicting coordinates in parallel. For multiple instances, decoding continues after each </box> until termination or the configured box limit. The main benchmark AP and runtimes use a limit of one box. We append <box> to the prefix for present and absent queries alike; the model can still emit none. A box AP score is the mean maximum token probability across its four coordinate positions.
LAM-Direct prompt grammar. Coordinate tokens <0>...<1000> (ids 151677–152677) and <box>, <ref>, <text_mask>, and <null> are inherited. Point answers reuse <box><x><y></box>. The following templates retain the original strings; <mesh-k> expands to the mesh span in Table 6. The benchmark uses the first template.
<image-1> Locate the object that matches this 3D shape: <mesh-1>
<image-1> Locate every instance of the object that matches this 3D shape: <mesh-1>
<image-1> The marked object matches which of these shapes: {placeholders}
Earlier LAM-Direct regimes. The 5,000/10,000/20,000-update ablations retain the LAM-Direct optimizer settings, with 200 warmup updates and cosine decay over their own horizon. Some early arms use LoRA rank 32 rather than 64. Eight packed sequences per update contain approximately 54–57 samples, supplied by eight A100-80GB GPUs with accumulation 1, or four with accumulation 2. All use seed 42; the data seed is unset in early ablations and 42 in later ablations and the 100K reference.
Table 8: LAM-Direct ablation-anchor training sources. Rows count annotations; effective rows apply the repeat factor. Objects are pool / usable after excluding 97 GSO and 405 ABO evaluation identities (502 total). YCB-V supplies 21 models in real photos and BlenderProc PBR renders; the other four sources use Isaac Sim or tabletop renders. Unqueryable assets in these frames serve as distractors. These inventory counts are not verified LAM-Select pool counts; LAM-Select’s prepared query-row composition is reported separately in Table 5.
| Source | Objects pool/use | Rows | Rep. | Effective rows (share) |
ycbv_train_real | 21/21 | 623,479 | 1.0 | 623,479 (14.90%) |
ycbv_train_pbr | 21/21 | 702,649 | 1.0 | 702,649 (16.79%) |
gso_v2 | 971/874 | 5,080 | 218.0 | 1,107,440 (26.46%) |
abo | 4,047/3,642 | 2,397 | 408.0 | 977,976 (23.36%) |
objv2tex | 11,909/11,909 | 109,043 | 3.40 | 371,022 (8.86%) |
objv_wave2 | 11,909/11,909 | 118,505 | 3.40 | 403,216 (9.63%) |
| Total | 28,836/28,334 (+21) | 1,561,153 | 4,185,782 |
Table 8.
Training-data composition
Table 8 describes the LAM-Direct ablation anchor; the sampling and task routing below cover that pipeline and its 100K extension. LAM-Select instead uses the prepared query rows in Table 5. The LAM-Direct 100K configuration adds a disjoint Objaverse source, expanding from 28,334 to 129,835 mesh identities, plus the 21 YCB-V models.
Sampling and task routing. Rendered sources use a train-time composer; YCB-V uses stored examples. Each composed sample has query slots, independently replaced by absent queries with probability 0.25. A per-sample choice selects single-instance or every-instance localization with probability 0.5/0.5, including for arm A’s localization samples. Every-instance targets concatenate all matching boxes. Repeated instances occur in 85% of objv2tex frames.
The anchor requests locate:0.54, point:0.18, select_dot:0.18, describe:0.10. Missing task stores fall back to locate; an instrumented run rewrote 54.4% of requested select samples. Delivered fractions were not recorded for any 10,000-step run. Arm A uses a manifest-defined per-source router without fallback and receives no point or describe samples. The PPAT and Uni3D appearance-token paths bypass feature-noise augmentation.
Category provenance. Category summaries count each eligible source/mesh ID once, including absent-query candidates. Native labels and object names are supplemented by noun rules over Cap3D captions [18], stored Qwen mesh descriptions, and product titles. Ambiguous entries remain unassigned. These are approximate metadata categories, not visually verified labels or training-sampling frequencies.
Contamination audit. Synthetic-source recipes define object-level held-out splits, with load-time ID-disjointness checks. We compare each evaluation CAD against the training pool in the frozen mesh encoder’s L2-normalized CLS space and manually review pairs above cosine similarity 0.90. Reported contamination counts are lower bounds. The audit flags no pairs in LM-O, HOPE, HB, T-LESS, TUD-L, IC-BIN [6], or ITODD [8]; their maximum top-1 similarity is 0.864.
Mesh onboarding
Mesh encoding and descriptor construction are separate from cached-token inference (Section B.6). PPAT and matched CNOS component timings have not been measured on GB200. The reported inference times exclude mesh encoding and cannot supply an onboarding measurement.
Evaluation protocols
Cohorts and sampling
Table 9 lists the saved evaluation cohorts. Official YCB-V AP uses the 900 target images with all 21 object queries per image. The Blackwell timing pilot uses a separate eight-image subset and house scoring; its checkpoints and timing exclusions are given in Section B.6. Historical 300-image visible-box scores cannot substitute for Table 9: Evaluation cohorts and coverage of the two PPAT systems. A check marks a measured result; means not evaluated in the accepted record. Shared cohort counts alone do not establish parity; saved image/query inputs and the scorer have been matched for HOPE, LM-O, and HB. Absent queries use up to eight sampled classes per image; YCB-V queries all 21 models under official BOP amodal scoring; the other rows use the internal visible-box protocol. HB and HOPE use validation splits with public ground truth. Simulated rows retain the original splits and the contamination limitation in § 4.
official BOP results. HB samples strided frames across all scenes; LM-O retains the first frames selected when its manifest was built.
HB Discrim excludes development scene 000001, leaving 12 scenes with 1,481 present and 2,208 absent queries. This population is distinct from the all-scene HB mAP cohort.
Official BOP scoring
The official YCB-V target list spans 12 test scenes. Ground truth uses mask-derived amodal boxes; instances below 10% visibility are ignored. COCO AP@[.50:.05:.95] is macro-averaged over objects. Querying every object on every image yields 18,900 queries, 78.2% absent. Ground truth contains 4,125 instances, including 2 ignored instances.
LAM-Select preserves every selected box and its generation-token score; LAM-Direct uses its batched fast route. The latter has 26 invalid answers; both LAM-Select encoder variants have none. Invalid outputs contribute no detections and never count as valid refusals. We report each evaluation route’s own AP; small numerical changes between reruns are not an uncertainty estimate. On GDRNPPDet_PBRReal’s published detections, the scorer returns 0.85215 AP against the reported 0.852; all twelve COCO statistics agree within .
Matched internal evaluation
LAM-Direct uses the LAM-Select manifests for 1,175 HOPE and 2,400 LM-O queries and the same visible-box AP kernel, yielding mAP 0.15465 and 0.18144. It has 21 and 132 invalid answers, respectively, all with empty decoded boxes. Together with the matched HB query audit, this matches query selection and scoring across systems, while architectures and output cardinalities differ.
Internal HB uses public val_primesense: 13 scenes, 33 objects and modal boxes. BOP-Classic HB instead uses 3 scenes and 16 objects from withheld test_primesense, with amodal boxes. These protocols are not interchangeable. External comparisons use official YCB-V AP (Table 14); missing official evaluations remain marked as unmeasured.
| System | Encoder | AP | AP50 | AP75 | GB200 s/img | Repeat range s/img |
| LAM-Direct | PPAT | 0.6384 | 0.7885 | 0.7112 | 0.322 | 0.309–0.324 |
| LAM-Select | PPAT | 0.7152 | 0.9150 | 0.8183 | 0.682 | 0.652–0.837 |
| LAM-Select | Uni3D-S | 0.7134 | 0.9110 | 0.8162 | 0.882 | 0.705–0.921 |
Table 14. Official BOP YCB-V encoder accuracy and reported Blackwell time. All 900 target images, 12 scenes and 18,900 image–object queries; mask-amodal COCO AP@[.50:.05:.95] with BOP low-visibility ignore handling. LAM-Select checkpoints have 31,434 updates; LAM-Direct arm A has 45,823. All systems use the same official target list and scoring protocol for AP. Times are approximate median cohort-mean seconds/image from three repeats of the separate eight-image GB200 batch-21 pilot; ranges span those repeats. The LAM-Direct timing checkpoint has 10,000 updates rather than 45,823; LAM-Select timing uses cached proposals and excludes live detection. These are assembled accuracy/runtime summaries; see Section B.6.
Presence discrimination
A commit is a valid predicted box. At natural decoding, we measure how often a model answers present queries relative to absent ones:
A value of 1.0 means committing is independent of presence. A committed box can still identify the wrong object; a hit requires target IoU . When no absent query is answered, we report for the ratio. HB excludes development scene 000001, as specified above.
Supporting Quantitative Results
The two headline systems are LAM-Select and LAM-Direct, both with PPAT. Historical experiments below retain their original checkpoints and conditions; they do not constitute matched ablations of both headline systems. Official YCB-V AP uses the 900-image BOP protocol with mask-derived amodal boxes and visibility-ignore handling. The separate Blackwell speed pilot (Section B.6) reports house mAP on eight images. Other datasets retain the internal protocols specified in each caption.
Grounding errors across suites
Table 10 retains both mesh encoders for the HB instance accounting in § 5.4. They share image/query inputs and ordered proposals. Uni3D-S recovers more present targets but emits more boxes on absent queries. Among covered queries with no selected target hit, 130/196 PPAT and 129/181 Uni3D-S outputs overlap another annotated object at IoU .
Table 11 applies the same accounting to HOPE and LM-O. HOPE includes multi-instance queries; its instance counts differ from query counts. All native boxes are retained, and both runs have zero invalid outputs.
| Suite | Encoder | Matched | Uncovered | Covered refused | Covered missed | Absent FP / queries |
| HOPE | PPAT | 600 | 0 | 26 | 294 | 245/400 |
| Uni3D-S | 577 | 0 | 83 | 260 | 272/400 | |
| LM-O | PPAT | 1,414 | 249 | 415 | 241 | 11/81 |
| Uni3D-S | 1,241 | 249 | 528 | 301 | 25/81 |
Table 11. LAM-Select instance and absence errors on HOPE and LM-O. Native visible-box outputs on 50 and 300 images, respectively, at IoU 0.5. Matched, uncovered, covered/refused and covered/missed counts sum to the number of target instances (920 on HOPE; 2,319 on LM-O). The final column counts absent queries emitting a box, not the number of false-positive boxes. These single-threshold counts do not decompose AP.
The instance partition does not count extra predictions on queries with a target hit. HOPE has multi-box outputs; LM-O and non-development HB do not.
Earlier LAM-Direct wrong-object matches. On HB’s non-development scenes, 142 of arm A’s 163 present-query predictions with target IoU below 0.10 overlap a different annotated object at IoU . This direct-coordinate diagnostic is separate from the proposal-conditioned accounting above.
Paired mesh-binding controls
We fix 36 distinct images before inference, 12 each from HOPE, LM-O, and non-development HB. Each suite contributes six co-visible and six absent donors. Half of each donor group uses scene-balanced uniform selection; half uses feature-neighbor proxies, not visually verified lookalikes. All source and co-visible donor instances have proposal coverage at IoU 0.5, and source–donor GT-box IoU is below 0.1. This is a selected diagnostic favorable to proposal selection.
Each pair changes only the valid mesh tokens and their cache identity, holding the image, native generic instruction, decoder settings, batch size one, and batch position fixed. LAM-Select receives identical ordered proposals in both conditions; LAM-Direct remains proposal-free. Native prompts and output cardinalities differ between architectures.
| System | Co-visible donor / 18 | Absent donor / 18 | ||
| Primary | Exact set | Primary | Exact set | |
| LAM-Direct + PPAT | 1 | 0 | 9 | 9 |
| LAM-Select + PPAT | 5 | 3 | 11 | 9 |
| LAM-Select + Uni3D-S | 4 | 3 | 10 | 9 |
Table 12. Paired target following under mesh substitution. Success counts out of 18 pairs in each donor group, at IoU . Co-visible success requires a source hit without a donor hit before substitution, and a donor hit without a source hit afterward. Absent-pair success requires a source hit followed by a valid empty answer. Primary success allows unrelated extra boxes; exact-set success requires a one-to-one match for every target instance and no extras in both conditions. Failed source controls and invalid answers remain in the denominators.
| System | GSO | ABO | Objaverse |
| LAM-Direct + PPAT | 0.305 | 0.353 | 0.320 |
Table 13. Synthetic-suite transfer of the selected LAM-Direct checkpoint. Arm A at 45,823 updates, with PPAT and one-box decoding. House visible-box mAP on 300 strided images per held-out suite, with max_neg_per_image=8. These use the original held-out splits before duplicate removal (Section A.4). This single configuration does not measure the effect of training-set scale.
Source controls fail on 14/18, 10/18, and 10/18 co-visible pairs in table order; thus paired failures cannot all be attributed to substitution. Answer sets change on 14/18, 14/18, and 12/18 pairs, far more often than both targets are followed correctly. LAM-Direct has two invalid condition outputs among 72; both LAM-Select runs have none. Invalid answers never count as valid rejection. Four co-visible and two absent pairs contain multiple target instances, which prevents exact-set success under LAM-Direct’s native one-box limit. These small, selected counts do not establish a reliable model ranking or isolate the backbone’s effect.
LAM-Direct synthetic transfer
LAM-Select mesh-encoder comparison
PPAT remains the primary mesh encoder; Tables 14 and 15 report the secondary Uni3D-S comparison. The two models share training-query order, seed, optimizer settings and schedule (Section A.1). This compares each encoder with its required connector. One training trajectory per encoder does not establish a statistically significant ordering. PPAT leads on HOPE, LM-O and YCB-V, while Uni3D-S leads on HB but emits more absent-query false positives (Table 10). HOPE top-one AP is 31.90% for PPAT and 26.81% for Uni3D-S; LM-O and HB AP are unchanged by the top-one restriction.
LAM-Direct multiple-instance localization
Arm A receives the same image–mesh inputs under three prompt/cap settings (Table 16). The HOPE panel is fixed before inference, spans ten scenes and shares views; it is not a representative benchmark. Present targets have at least 50% visibility and visible boxes covering at least 0.5% of the image. All eligible targets are matched one-to-one at IoU .
| Prompt | Cap | Recall | Precision | Both found / 48 | Single exact / 8 | Absent FP / 8 |
| Single | 1 | 24.0 | 48.9 | 0 | 4 | 7 |
| Single | 4 | 29.2 | 21.9 | 4 | 2 | 7 |
| Every | 4 | 41.7 | 22.2 | 10 | 0 | 7 |
Table 16. LAM-Direct prompt and box-limit comparison on selected HOPE queries. On 48 two-instance queries, recall (%) uses 96 targets and precision (%) uses all valid predicted boxes. Both-found counts require well-formed answers but allow extras. Single exact means one matched box with no extras; absent FP counts queries emitting a box (eight controls each). LAM-Select’s native multi-box benchmark results use a different query population and cannot fill this pilot.
No two-instance query yields an exact-set-correct answer in any mode. Larger caps recover more instances but add false positives (Figure 9). Box precision and recall retain valid boxes from two malformed every-instance answers; these contribute three matched and four extra boxes, but cannot count as well-formed answers.

Figure 9. LAM-Direct multiple-instance localization on real HOPE images. Every-instance prompt, four-box limit. Three selected examples find both targets with extra boxes; the fourth misses one target. Blue boxes show every prediction, dashed green boxes show all targets, and insets show the queried mesh. Display crops retain all boxes.
Blackwell batch scaling
The GB200 pilot evaluates eight YCB-V images with 168 queries (35 present, 133 absent), after 21 excluded warmup queries on a separate image. Table 17 and Table 18 report three fresh-process repeats per configuration. The LAM-Direct encoder pair has 10,000 updates, unlike the 45,823-update PPAT accuracy checkpoint; the LAM-Select pair has 31,434 updates in both evaluations. The pilot’s timing cohort differs from official 900-image AP.
Timing scope. CUDA-synchronized timing includes shared image processing, vision and prefix work, and grouped query inference; it excludes image/token file reads, loading, warmup, scoring, mesh encoding and live proposal detection. LAM-Select uses 100 cached proposals per image; LAM-Direct uses detector-free one-box decoding. The reported environment is NGC PyTorch 26.01, PyTorch 2.10 and CUDA 13.1 on GB200. The benchmark uses GPU 0 while light matrix multiplication occupies GPUs 1–3; shared power and CPU activity may affect timing. Peak allocated memory includes loading and warmup. Approximate seconds/image is 21/median queries/s, the median of the three repeat-level cohort means, not individual-image latency. Repeat ranges are observed minima and maxima, not confidence intervals.
Leaderboard comparison scope
Quality accounting. Reported house mAP is 0.832 for LAM-Select PPAT and 0.826 for Uni3D-S at every batch size. LAM-Direct ranges across batches are 0.717–0.722 and 0.677–0.681; the report supplies no exact batch-21 values. Its PPAT model returns two malformed absent-query answers per 168 queries; Uni3D-S returns one. These remain in the timing denominator and do not count as valid abstentions. The small cohort, training budgets and decoder paths limit this pilot to batching and execution cost.
Leaderboard comparison scope
Figure 5 shows points from the 2026-09-06 snapshot of BOP’s seen-object and unseen-object 2D detection boards. The separate external-only frontiers use 10 timed seen-object entries and 18 named unseen-object entries. Shaded regions show the area dominated by each external frontier. Our models use YCB-V training identities and remain excluded from both frontiers. The shared test images and AP definition do not equalize training exposure or establish eligibility for the unseen-object task.
External timing and exclusions. BOP’s published Time is the average time to predict all objects in an image. Our GB200 values cover all 21 YCB-V queries with the scope in Section B.6. External runtimes retain their original hardware and timing scopes without normalization; the plot does not support controlled speed claims. The submission named only “.” is excluded because it has no identifiable method name, paper or code. Named unpublished entries with positive reported time remain. Three seen-object entries omit runtime and therefore have no cost-axis coordinate: DLZDet-PBR1, Extended FCOS-PBR (v2), and DLZDet-PBR+Real. One-time mesh onboarding is excluded from these image/query times (Section A.5).
Additional Qualitative Results
Matched systems and encoder control. Figure 10 adds the Uni3D-S control to the main gallery’s PPAT pair. Selection joins saved outputs by image and mesh ID, checks identical ground truth, and excludes HB’s
Figure 10. Matched HB detections with the full encoder comparison. LAM-Direct with PPAT and LAM-Select with PPAT or Uni3D-S receive identical image/query inputs. Blue boxes are predictions; dashed green boxes are visible targets. Hits require IoU . All predictions are shown. These are the same five selected cases as Figure 6; selection used all three systems, including the Uni3D-S control. Mesh previews are reader aids, not image inputs.
| Encoder package | HB | LM-O | HOPE | commit | Discrim | |
| PPAT-B32 (anchor) | 0.282 | 0.140 | 0.117 | 0.637 | 0.210 | 3.030 |
| Uni3D-S | 0.265 | 0.271 | 0.150 | 0.700 | 0.271 | 2.585 |
| Uni3D-G (Y-up) | 0.304 | 0.139 | 0.166 | 0.603 | 0.208 | 2.901 |
Table 20. Table 20: LAM-Direct frozen encoder comparison. House visible-box mAP. The historical Uni3D variants have no official YCB-V evaluation; headline-system scores appear in Table 14. All arms use 10,000 updates, seed 42, and LoRA . The first two change only the encoder package; Uni3D-G also changes canonical axis, dtype, and topology. HB commit, FP, and Discrim use scenes 2–13; HB mAP uses all 13. LAM-Select’s separate encoder experiment uses a different schedule (Table 15).
| Mesh encoder | HOPE | LM-O | HB (all) | HB (no dev.) |
| PPAT | 0.3707 | 0.3880 | 0.3658 | 0.3404 |
| Uni3D-S | 0.3160 | 0.3337 | 0.4559 | 0.4219 |
Table 15. Secondary mesh-encoder comparison within LAM-Select. Native house visible-box mAP at 31,434 updates on 50 HOPE images and 300 images each for LM-O and HB. Both encoders share queries and detector proposals. HB’s development-excluded column uses 276 images; the full HB column includes all 13 scenes. Official YCB-V scores and the separate Blackwell timings are in Table 14. These are encoder-plus-connector comparisons, separate from the two-system comparison in Table 2.
development scene. Five prespecified strata cover shared success, both LAM-Select variants succeeding where LAM-Direct fails, the reverse, a verified LAM-Select wrong-object selection, and absent-query disagreement. Present targets occupy 2.5–40% of the image; we select the lower-median target area within each stratum, breaking ties by query ID. The absent stratum uses the lower-median query ID. Wrong-object matches require target IoU below 0.1 and overlap of at least 0.5 with another annotated object. These balanced strata do not estimate outcome frequencies.
Other panels use earlier LAM-Direct dumps, with literal outputs and pixel boxes.
Outcomes across suites
Figure 11 uses LAM-Direct 100K arm A, with strided-300 dumps for GSO and Objaverse. Hits and mislocalizations use median IoU; misses use median relative target area. Absent-query examples use the lowest object ID in the hit’s image when available, otherwise the median image–object pair.
Figure 11. Localization across datasets. Solid orange boxes show predictions; dashed green boxes show ground truth. Insets show query meshes, and text below each image gives the model output. Hits have IoU .
HB annotations permit the wrong-object checks in Section B.1; low target IoU alone does not verify such a match. Synthetic scenes annotate only 1.2–2.2 of roughly 25 objects per frame, precluding the same comparison.
Color and object identity
For Figure 12, we select the closest of 528 HB CAD pairs by sorted bounding-box extents and a scale-normalized radial descriptor before inspecting predictions. The frame is outcome-selected from 23 co-present frames: it maximizes the minimum of four IoUs, requiring identical gray-twin outputs and both other models to localize both queries at IoU ≥ 0.5. Across these frames, the gray twin never localizes both queries correctly; LAM-Direct arm A does so on 8.
Pointing and description
The LAM-Direct studies use a separate 20,000-step multitask checkpoint with PPAT and rank-64 LoRA, not either headline localization checkpoint. A later LAM-Select continuation supplies the main capability figures (Section C.3.1). Historical tests of R, the reported localization model, cover native point-prompt transfer and image-plus-mesh description (Sections C.3.2 and C.3.3).
Mesh-only descriptions. With no image input, Describe this 3D shape: <mesh-1> elicits paired answers for 120 held-out IDs (40 each from ABO, GSO, and Objaverse), using the correct mesh or a random same-source swap. Greedy decoding has a 256-token limit. All 240 outputs are valid and nonempty; every pair differs. IDs are disjoint from training lists, but geometric deduplication is unverified and swaps may change category, so fine-grained identity is not isolated.
A blinded gpt-6-astra judge scores agreement with stored Qwen3-VL-8B-Instruct captions (Table 21). Correct meshes raise same-asset judgments by 9.17 percentage points (95% paired bootstrap interval [4.17, 14.17]), below the predeclared 20-point success threshold. Reference captions can misstate geometry, so agreement is not visual correctness.
| System / source | Agreement rating / 10 | Same-asset judgments | ||
| Correct mesh | Random swap | Correct mesh | Random swap | |
| LAM-Direct + PPAT, separate multitask checkpoint | ||||
| ABO () | 3.73 | 1.18 | 7/40 | 0/40 |
| GSO () | 3.63 | 1.30 | 5/40 | 1/40 |
| Objaverse () | 1.98 | 0.43 | 0/40 | 0/40 |
| All () | 3.11 | 0.97 | 12/120 | 1/120 |
Table 21. Mesh-only reference agreement under correct and swapped inputs. Blinded Astra judgments against Qwen3-VL-8B-Instruct captions, with the target reference and instruction held fixed. Measured rows use the LAM-Direct multitask checkpoint; no LAM-Select result in this table uses this caption-agreement rubric. The newer LAM-Select study uses blinded visual review (Section C.3.1). This is not a comparison between the two headline localization checkpoints.
Visual review. Before scoring, we fix 24 meshes: eight evenly spaced sorted held-out IDs per source. Blinded Astra reviews compare each description with three target renders, which only the judge and reader see. The judge flags major contradictions in 15/24 correct-mesh and 24/24 swapped outputs; only 2/24 correct-mesh outputs are rated largely faithful. These are exploratory model judgments, not calibrated accuracy. Of 16 hidden controls, all eight negatives and two nominal positives are flagged. The latter incorrectly call a U-shaped sofa L-shaped and remain in the record as defective controls.
Pointing audits. On 32 ABO cases with held-out object IDs in possibly seen scenes, the checkpoint returns 26 points, five refusals, and one malformed answer. Only 9/32 hit the target box, including four boundary hits: this is coarse localization, not keypoint accuracy.
A real-image check selects 32 YCB-V test and 32 HB validation queries before inference, spread across scenes and excluding HB’s development scene. Targets have at least 70% visibility and visible boxes covering at least 2.5% of the image. With the same checkpoint, prompt, and decoder, all 64 answers are <box>none</box>. YCB-V supplies localization training only; these selected targets do not define a representative benchmark.
Examples and scope. The compact galleries select successes and failures at 3:1, not their audit frequencies. Pointing hits lie strictly inside the target box and on the object; description labels assess only the shown excerpts, not the full responses below. The boot response has a minor shaft-shape error beyond its excerpt. Other galleries retain fixed-print-set cases. Joint capability at the headline localization accuracy remains untested.

Figure 13. Pointing in synthetic ABO scenes. The LAM-Direct multitask checkpoint’s blue points hit three target boxes and miss one. Dashed green boxes mark the targets; insets show the query meshes. These selected cases do not estimate accuracy.
a Bottle — Full response
The object is a cylindrical container with a slightly tapered base and a flat, circular top surface. The main body is a smooth, continuous cylinder, and the top surface is a flat, circular disc. The base is a flat, circular disc that is wider than the top surface and sits flush with the bottom of the cylinder. The overall proportions suggest a tall, slender form, with the height being significantly greater than the diameter of the top and bottom circular surfaces.
b Box — Full response
The object is a rectangular prism with a consistent, uniform height and a rectangular base. Its top and bottom faces are identical rectangles, and its four vertical sides are also rectangular, forming a closed, box-like structure. The front face is the largest surface, while the side faces are narrower and appear to be vertical planes. The overall proportions suggest a tall, slender box with a relatively small depth compared to its height and width, giving it a vertical orientation. The edges are straight and meet at right angles, forming a closed, geometric volume.
c Rug — Full response
The object is a flat, rectangular slab with a consistent thickness throughout, appearing as a single, solid piece. Its silhouette is defined by four straight edges and four corners, forming a parallelogram-like shape when viewed from the side. The top surface is a perfect rectangle, and the bottom surface is a perfect rectangle, parallel to the top. The object has no visible curvature or protrusions, presenting a simple, geometric form with no indentations, curves, or complex structures.
d Perforated disc — Full response
The object is a sphere composed of two distinct hemispherical halves connected along a common vertical axis, forming a closed, smooth, and continuous surface. The two halves are joined at their equator, which is a perfect circle, and the entire structure is symmetrical along this axis, maintaining a constant radius throughout its entire volume.
GSO | fixed audit position 14
Target mesh → generated description
The object is a low-profile athletic shoe with a streamlined, elongated silhouette that tapers toward the toe and widens slightly at the heel. It features a distinct upper section that wraps around the foot, with a visible collar and a reinforced toe cap. The sole is thick and segmented, with a pronounced heel counter and a forefoot that curves upward to meet the upper. The midsole is a continuous, slightly raised volume that connects seamlessly to the heel and toe, forming a unified base. The outsole is a distinct, multi-layered structure with a pronounced heel bumper and a textured, segmented outsole that extends slightly beyond the heel. The overall proportions suggest a design optimized for ground contact and dynamic movement, with the upper section appearing to be a single, continuous piece that wraps around the foot, and the sole providing a stable, multi-directional base.
Swapped mesh → generated description
The object is a stylized, blocky figure with a rounded, bulbous head that tapers slightly toward the top, a short, stout neck, and a wide, slightly curved body that curves inward at the midsection before flaring out into a rounded, bulbous tail-like appendage. The head is proportionally large relative to the body, with a distinct, rounded top and a small, pointed snout. The limbs are short and stubby, with the arms ending in small, rounded hands and the legs ending in small, rounded feet. The overall silhouette is compact and squat, with the head and body forming a single, continuous volume that is wider than it is tall, and the tail extending outward and slightly upward from the rear of the body.
Objaverse | fixed audit position 17
Target mesh → generated description
The object is a rectangular prism with a slightly tapered top edge and a flat base, forming a box-like structure. It has a distinct front face, a back face, and two side faces, all meeting at right angles. The front face is larger than the side faces, and the top surface is parallel to the front face, creating a box-like silhouette. The side faces are vertical and rectangular, connecting the front and back faces. The overall proportions suggest a taller, narrower form with a relatively small depth compared to its height and width.
Swapped mesh → generated description
The object is a stylized humanoid figure with a rounded, bulbous head that tapers slightly toward the top, a short, cylindrical torso, and two short, cylindrical arms extending from the sides of the torso. The head is proportionally large relative to the body, with a distinct, slightly curved top and a small, rounded protrusion at the front. The arms are short and end in small, rounded hands. The figure is depicted in a seated posture with the legs positioned close together, and the overall silhouette is compact and blocky, with smooth, continuous curves connecting the head, torso, and limbs.
Figure 18. Mesh-only descriptions: GSO and Objaverse. Multitask checkpoint, correct mesh (left) and random swap (right), with full literal responses. Renders are for the reader only. Together with Figure 17, these panels show four cases from the fixed audit.
Figure 17. Mesh-only descriptions: ABO and GSO. Multitask checkpoint, correct mesh (left) and random swap (right), with full literal responses. Renders illustrate the inputs; the model receives only mesh tokens. These are two cases from the fixed audit.
Pointing with the final T-DM checkpoint
20,000-step multitask checkpoint · ABO object-identity split
Prompt: Point to the object that matches this 3D shape: <mesh-1>
Same scene and prompt; the control substitutes an absent object's mesh.
Figure 19. Pointing: a hit and a miss. Multitask checkpoint, ABO cases 18 and 22 from the fixed print set. Each scene and instruction is held fixed while the mesh is swapped. The original-query point lies inside the target box in case 18 and outside it in case 22. Literal outputs include the swapped-query responses. These selected cases do not estimate accuracy.
LAM-Select M10k capability continuation
M10k starts from R at update 31,434 and runs for 10,000 additional updates with one seed. Its delivered examples follow an 80/15/5 locate/describe/point mixture: 64,000, 12,000, and 4,000 examples. The run uses the same PPAT mesh representation. It is evaluated for capability figures only; R remains the model in every localization result and table.
For mesh-only description, both R and M10k@10k receive the mesh tokens and an instruction, with no image, regions, labels, caption, or reference render in the model inputs. The panel contains 300 held-out-caption identities. Ten blinded assistant reviewers score correct and same-source, same-category swapped-mesh outputs against three renders, with hidden positive and negative controls. A grounded, non-generic response must contain at least one supported intrinsic attribute, no major contradiction, no refusal, and no generic-only description. Positive and negative controls reach 86.7% and 5.7% grounded. R scores 52.7% on correct meshes without description-task training; M10k@10k scores 63.0%. Their paired difference is 10.3 percentage points with a 95% identity-grouped bootstrap interval of [3.7, 16.7]. These are judge-assisted visual-review results, not calibrated human accuracy. Swapped-mesh grounded rates are 31.7% for R and 44.3% for M10k@10k; correct-minus-swapped gaps are 21.0 and 18.7 percentage points, respectively. These results do not establish stronger identity binding or a statistically significant change in the gap.
For pointing, the fixed real-BOP panel contains 2,000 present and 533 absent queries. M10k@10k produces strict point syntax on 99.6% of attempts, hits the queried object’s visible mask on 35.7% of present attempts, and emits a point on 26.6% of absent attempts. Present hit rates are 60.6% on YCB-V, 17.8% on LM-O, 29.6% on HB validation, and 34.8% on HOPE validation. Coordinates decode with . Invalid, refused, and failed attempts stay in the denominators. Masks, crops, and hit labels are evaluation-only and never enter the model.
R native point-prompt transfer
The 31,434-step checkpoint receives the description probe’s 32 fixed present-target image–mesh pairs, eight per suite, without proposals, labels, ground-truth boxes, reference crops, or additional task training. One instruction requests <point>x,y</point> with integer coordinates in [0, 1000], or <point>none</point> if identification fails. Coordinates start at the top-left, increasing rightward and downward, and map to image pixels as . Greedy generation permits 96 new tokens with one attempt per query, no prompt search or retries, and no conflicting grounding instruction.
There are 0/32 exact-format answers. Two contain complete point tags preceded by an unclosed <ground> wrapper; the other 30 have no complete point tag. Both emitted points miss the manifest boxes and original HOPE visible/full boxes and masks (Figure 20); inspection neither repairs wrappers nor extracts points from free-form numbers. This single-prompt, present-target probe provides no successful native pointing example, does not test absence, and does not establish a general LAM-Select pointing result or isolate the cause of the retained grounding syntax.
Figure 20. R native point-prompt failures. These are the only two answers containing complete point-coordinate tags among 32 fixed present-target queries to the localization checkpoint. The displayed points come directly from the generated coordinates, scaled from [0, 1000] to the original image dimensions. Both miss the target’s visible box and mask. Point markers, target-box overlays, and enlarged detail crops are reader aids; the model receives only the unannotated image, mesh features, and instruction. Responses are literal, including the incomplete grounding wrapper; only the end-of-sequence marker is omitted. No answer meets the exact requested format. This one-prompt transfer test does not establish general pointing performance.
R image-plus-mesh description
The main LAM-Select + PPAT checkpoint at 31,434 steps receives an image, 65 mesh tokens, and an instruction to describe the queried object’s visible appearance or abstain. There is no additional mesh-description supervision; labels, ground-truth boxes, reference crops, and proposal candidates are excluded. Greedy decoding permits 96 new tokens. This is image-conditioned, not mesh-only description.
The cohort has eight original present queries per suite (HOPE, LM-O, HB, YCB-V), same-image substitutions with another present mesh, and absent-image controls (Table 22). All 96 answers are nonempty and EOS-terminated, but 95/96 retain grounding markup. All 32 present-mesh substitutions change the answer, establishing sensitivity rather than successful identification.
| Condition | Identity S / U / C | Refusal-like text | Appropriate abstention | |
| Original present mesh | 32 | 11 / 18 / 3 | 0 | – |
| Other present mesh | 32 | 7 / 17 / 4 | 4 | – |
| Absent image, same suite | 25 | – | 16 | 13 |
| Absent image, cross suite | 7 | – | 6 | 5 |
Table 22. Exploratory LAM-Select image-plus-mesh description controls. Counts use all attempts in each condition. Assistant identity judgments are supported (S), uncertain (U), or contradicted (C); the four remaining present-mesh substitutions are refusal-like outputs with identity rated not applicable. Appropriate abstention counts only affirmative assistant judgments, retaining negative, uncertain, and not-applicable cases in . Refusal-like text includes bare no/None, but excludes <objects>None</objects>: no proposals were supplied, so this field is not evidence of abstention. – denotes a measure not reported for that condition. These are review counts, not calibrated human accuracy.
Each item has one assistant review, with partial blinding and no inter-rater calibration. Reviewers see the input image and reviewer-only target-reference crops and visible-instance masks. Identity support does not guarantee correct attributes. Presence follows dataset-qualified raw annotations, including invisible instances; cross-suite controls change image domain and lack an exhaustive physical-identity crosswalk. These selected, dependent controls are not a representative score, and no matched LAM-Direct or Uni3D run uses this protocol. Figure 21 shows post-hoc selected examples.
Figure 21. R descriptions: supported, generic, and unsupported responses. Each panel shows the complete input image and a positive-reference crop of the queried identity for the reader only; the crop is neither a mesh render nor a model input. Panels (a,b) hold the image and instruction fixed while substituting another present mesh: a supported label becomes a generic answer, whose identity is uncertain rather than demonstrably wrong. Panels (c,d) show an appropriate refusal and an unsupported description under same-suite absent-image controls. Text is the verbatim content of <ground>, with grounding/object wrappers and the end-of-sequence marker omitted. Examples were selected after review by the lowest review ID within these outcome strata, pairing (a) with its mesh substitution; they illustrate behavior, not its frequency.
Figure 14. Mesh-only description. Each row shows three correct excerpts and one incorrect excerpt from the LAM-Direct multitask checkpoint. Mesh renders are for the reader; the model receives no image input. Full responses: Figure 15 and Figure 16.

Figure 15. Full mesh descriptions (a–d). The first description row of Figure 14, with three views per mesh and complete literal responses. Correctness labels in the compact gallery assess only the shown excerpts.

Figure 16. e Saucer — Full response. The object is a shallow, circular dish with a slightly raised rim, featuring a central, raised, and slightly concave central depression. The rim is uniformly thick and extends outward from the central depression, forming a continuous, smooth outer edge. The overall form is symmetrical and flat, with the central depression creating a central void. The proportions suggest a low, wide bowl with a shallow depth, where the diameter is significantly greater than its height, giving it a broad, open appearance.
f Chair — Full response. The object is a chair with a continuous, gently curved backrest that seamlessly transitions into the seat, forming a unified, enveloping silhouette. The seat is a low, rectangular cushioned plane that sits flush with the base, providing a stable foundation. The armrests are integrated into the sides of the backrest, curving outward and downward from the back to the front, creating a smooth, continuous contour that follows the contour of the backrest. The chair is supported by four slender, tapered legs that splay outward from the base, angled slightly outward and downward, and are connected by a horizontal stretcher bar near the bottom for stability. The overall proportions are compact and low-slung, with the backrest being the dominant vertical element, and the seat and armrests extending outward to provide a balanced, enclosed form.
g Bed — Full response. The object is a rectangular bed frame with a flat, rectangular mattress base and a headboard that is taller than the base and has a curved, slightly rounded top edge. The headboard is attached to the base at the front and back, forming a continuous vertical surface. The bed frame has four short, straight legs that are positioned at each corner, elevating the base slightly from the ground. The headboard is connected to the base via a horizontal support bar that runs along the top edge of the headboard, connecting it to the base. The overall structure is symmetrical along its longitudinal axis, with the headboard and base forming a single, unified rectangular volume.
h Table — Full response. The object is a rectangular prism with a flat top surface and four vertical legs, each attached to the underside of the top and to the corners of the base. The legs are straight and vertical, extending downward from the corners of the base. The top surface is a single, flat plane. The base is a solid, rectangular prism with four corners, and the legs are positioned to support the top surface at its center. The overall proportions suggest a low, wide, and stable structure, with the legs being relatively short in comparison to the length of the base.
Figure 16: Full mesh descriptions (e–h). The second description row of Figure 14, in the same order. These selected examples do not replace the fixed visual audit.
References
- [1]Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6D Object Pose Estimation Using 3D Object Coordinates. In ECCV, pages 536–551, 2014. 5DOI
- [2]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. SAM 3: Segment Anything with Concepts. arXiv preprint arXiv:2511.16719, 2025. 2arxiv.org/abs/2511.16719
- [3]Jasmine Collins, Shubham Goel, Kenan Deng, A chleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. ABO: Dataset and Benchmarks for Real-World 3D Object Understanding. In CVPR, pages 21094–21104, 2022. 4arxiv.org/abs/2110.06199
- [4]Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A Universe of Annotated 3D Objects. In CVPR, pages 13142–13153, 2023. 4arxiv.org/abs/2212.08051
- [5]Maximilian Denninger, Dominik Winkelbauer, Martin Sundermeyer, Wout Boerdijk, Markus Knauer, Klaus H. Strobl, Matthias H umt, and Rudolph Triebel. BlenderProc2: A Procedural Pipeline for Photorealistic Rendering. Journal of Open Source Software, 8(82):4901, 2023. 4DOI
- [6]Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim. Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd. In CVPR, pages 3583–3592, 2016. 16arxiv.org/abs/1512.07506
- [7]Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. In ICRA, pages 2553–2560, 2022. 4arxiv.org/abs/2204.11918
- [8]Bertram Drost, Markus Ulrich, Paul Bergmann, Philipp Härtinger, and Carsten Steger. Introducing MVTec ITODD – A Dataset for 3D Object Recognition in Industry. In ICCV Workshops, pages 2200–2208, 2017. 16DOI
- [9]Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiri Matas, and Carsten Rother. BOP: Benchmark for 6D Object Pose Estimation. In ECCV, pages 19–35, 2018. 4arxiv.org/abs/1808.08319
- [10]Tomas Hodan, Martin Sundermeyer, Yann Labbe, Van Nguyen Nguyen, Gu Wang, Eric Brachmann, Bertram Drost, Vincent Lepetit, Carsten Rother, and Jiri Matas. BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objects. In CVPR Workshops, pages 5610–5619, 2024. 2, 5arxiv.org/abs/2403.09799
- [11]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022. 4arxiv.org/abs/2106.09685
- [12]Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy. In ECCV, pages 38–57, 2024. 2arxiv.org/abs/2403.14610
- [13]Qing Jiang, Junnan Huo, Xingyu Chen, Yuda Xiong, Zhaoyang Zeng, Yihao Chen, Tianhe Ren, Junzhi Yu, and Lei Zhang. Detect Anything via Next Point Prediction. arXiv preprint arXiv:2510.12798, 2025. 1, 2arxiv.org/abs/2510.12798
- [14]Roman Kaskman, Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic. HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects. In ICCV Workshops, pages 2767–2776, 2019. 1, 5arxiv.org/abs/1904.03167
- [15]Jiehong Lin, Lihua Liu, Dekun Lu, and Kui Jia. SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation. In CVPR, pages 27906–27916, 2024. 2arxiv.org/abs/2311.15707
- [16]Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. In ECCV, pages 740–755, 2014.arxiv.org/abs/1405.0312
- [17]Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding. In NeurIPS, 2023.arxiv.org/abs/2305.10764
- [18]Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3D Captioning with Pretrained Models. In NeurIPS, 2023.arxiv.org/abs/2306.07279
- [19]Van Nguyen Nguyen, Thibault Groueix, Georgy Ponimatkin, Vincent Lepetit, and Tomas Hodan. CNOS: A Strong Baseline for CAD-based Novel Object Segmentation. In ICCV Workshops (R6D), pages 2126–2132, 2023.arxiv.org/abs/2307.11067
- [21]Anton Osokin, Denis Sumin, and Vasily Lomakin. OS2D: One-Stage One-Shot Object Detection by Matching Anchor Features. In ECCV, pages 635–652, 2020.arxiv.org/abs/2003.06800
- [22]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, pages 8748–8763, 2021.arxiv.org/abs/2103.00020
- [23]Aadarsh Sahoo, Vansh Tibrewal, and Georgia Gkioxari. Aligning Text, Images, and 3D Structure Token-by-Token. arXiv preprint arXiv:2506.08002, 2025.arxiv.org/abs/2506.08002
- [24]Zhenran Tang, Rohan Nagabhirava, and Changliu Liu. CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objects. arXiv preprint arXiv:2602.20551, 2026.arxiv.org/abs/2602.20551
- [25]Stephen Tyree, Jonathan Tremblay, Thang To, Jia Cheng, Terry Mosier, Jeffrey Smith, and Stan Birchfield. 6-DOF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmark. In IROS, pages 13081–13088, 2022.arxiv.org/abs/2203.05701
- [26]Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, Jan Kautz, Lei Zhang, and Zhidong Yu. LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding. arXiv preprint arXiv:2605.27365, 2026.arxiv.org/abs/2605.27365
- [27]Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General Object Foundation Model for Images and Videos at Scale. In CVPR, pages 3783–3795, 2024.arxiv.org/abs/2312.09158
- [28]Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. In Robotics: Science and Systems (RSS), 2018.arxiv.org/abs/1711.00199
- [29]Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. In CVPR, pages 4818–4829, 2024.arxiv.org/abs/2311.06242
- [30]Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. PointLLM: Empowering Large Language Models to Understand Point Clouds. In ECCV, pages 131–147, 2024.arxiv.org/abs/2308.16911
- [31]Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling. In CVPR, pages 19291–19300, 2022.arxiv.org/abs/2111.14819
- [32]Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3D: Exploring Unified 3D Representation at Scale. In ICLR, 2024.arxiv.org/abs/2310.06773













