betaFeedback

WorksComputer Science

SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?

Abstract

Spatial reasoning is a fundamental aspect of human cognition, yet it remains a major challenge for contemporary vision-language models (VLMs). Prior work largely relied on synthetic or LLM-generated environments with limited task designs and puzzle-like setups, failing to capture the real-world complexity, visual noise, and diverse spatial relationships that VLMs encounter. To address this, we introduce SpatiaLab, a comprehensive benchmark for evaluating VLMs' spatial reasoning in realistic, unconstrained contexts. SpatiaLab comprises 1,400 visual question-answer pairs across six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each with five subcategories, yielding 30 distinct task types. Each subcategory contains at least 25 questions, and each main category includes at least 200 questions, supporting both multiple-choice and open-ended evaluation. Experiments across diverse state-of-the-art VLMs, including open- and closed-source models, reasoning-focused, and specialized spatial reasoning models, reveal a substantial gap in spatial reasoning capabilities compared with humans. In the multiple-choice setup, InternVL3.5-72B achieves 54.93% accuracy versus 87.57% for humans. In the open-ended setting, all models show a performance drop of around 10-25%, with GPT-5-mini scoring highest at 40.93% versus 64.93% for humans. These results highlight key limitations in handling complex spatial relationships, depth perception, navigation, and 3D geometry. By providing a diverse, real-world evaluation framework, SpatiaLab exposes critical challenges and opportunities for advancing VLMs' spatial reasoning, offering a benchmark to guide future research toward robust, human-aligned spatial understanding. SpatiaLab is available at: https://spatialab-reasoning.github.io/.

Introduction

Spatial reasoning is a fundamental aspect of human cognition that involves understanding spatial layouts, imagining and manipulating objects, and navigating environments. It supports applications ranging from robotics, autonomous driving and augmented/virtual reality to geospatial analytics, computer graphics, human–computer interaction, and education [32] (Qi et al. 2024; Qi et al. 2025). For artificial agents, autonomous systems that perceive and act in dynamic environments, reliable spatial competence requires interpreting geometric relations, depth cues, occlusion patterns, and agent-centric perspectives in scenes that are visually noisy and structurally complex [6, 9] (Trope & Liberman 2010; Ramalho et al. 2018). Recent vision–language models have advanced multimodal representation and language grounding, but their spatial judgments remain fragile in realistic environments [13] (Kosoy et al. 2025; Pothiraj et al. 2025).

Existing spatial benchmarks, however, are narrow and simplified. Most focus on elementary tasks such as binary spatial relations, low-resolution depth categories, or schematic navigation cues, often in synthetic or simplified settings that reduce visual clutter and homogenize objects [21, 34, 36, 22, 8] (Szymanska et al. 2024; Shiri et al. 2024; Song et al. 2024; Yang et al. 2024; Qi et al. 2025). These controlled setups lessen perceptual and reasoning demands, leading to apparent saturation while masking failures under distribution shift. Critical challenges such as robust occlusion inference, scale-consistent judgments across viewpoints, and planning under partial observability remain under-sampled [13, 9] (Kosoy et al. 2025; Pothiraj et al. 2025). As a result, models that perform well on existing benchmarks often fail on integrated, multi-step inference tasks, limiting safe deployment in real-world environments where spatial errors carry practical consequences.

A meaningful evaluation must therefore span the all core axes of spatial reasoning: relative position, depth and occlusion, orientation, size and scale, navigation, and 3D geometry. Unlike current task-isolated setups, human cognition integrates these dimensions seamlessly under noise and ambiguity [18, 7, 15] (Zitkovich et al. 2023; Yuan et al. 2024; Qi et al. 2025). Robust benchmarks should therefore test both choice and generation, benchmark against human performance, and draw from real-world scenes that reveal failures masked by simulator datasets-principles that motivate our benchmark.

To address the above needs, we introduce SPATIALAB, a benchmark designed to assess spatial reasoning of vision–language models in realistic, unconstrained visual contexts, as outlined in Figure 1. SPATIALAB provides a balanced suite of visual question–answer items covering a broad taxonomy of spatial tasks and supports both multiple-choice and open-ended evaluation formats. The benchmark emphasizes photographic diversity, complex spatial relational queries, and tasks that require integrating depth, perspective, and 3D structure. We evaluate a wide collection of state-of-the-art VLMs, including open- and closed-source models and reasoning-focused systems, compare them to human performance, and perform extensive error analysis. Our key contributions are as follows:

Figure 1. Overview of SPATIALAB. The benchmark addresses limitations of prior datasets (left), introduces 1,400 visual QA pairs spanning 5 categories and 30 subcategories (center), and enables systematic evaluation through multiple-choice and open-ended tasks. It features diverse task and image complexity, with varied object counts, layers, lighting, textures, relations, and materials (right).

  1. We introduce SPATIALAB, a large-scale benchmark of 1,400 visual question–answer pairs spanning six principal categories and thirty task types in realistic, unconstrained contexts. With at least 200 questions per category and 25 per subcategory, it ensures balanced coverage and supports both multiple-choice (SPATIALAB-MCQ) and open-ended (SPATIALAB-OPEN) evaluation.

  1. We perform a comprehensive evaluation of 25+ state-of-the-art vision–language models: open- and closed-source, reasoning-oriented, and spatially specialized reasoning models against human baselines, exposing a consistent and significant gap in spatial reasoning.

  1. We deliver in-depth error analyses and diagnostic studies that uncover systematic failure modes in depth perception, navigation, occlusion, and 3D geometry, yielding concrete insights and actionable directions for advancing robust, human-aligned spatial understanding.

  1. We explore multiple strategies to enhance spatial reasoning in VLMs, including supervised fine-tuning (SFT), chain-of-thought (CoT) prompting, CoT with self-reflection, and multi-agent architectures (SPATIOXOLVER) with multi-step reasoning, analyzing their benefits and limitations across multiple-choice and open-ended tasks.

In the multiple-choice setting, leading models achieve roughly 50–55% accuracy, while human annotators exceed 85%. In open-ended generation, model performance drops by another 10–25 percentage points, with the best systems near 40% versus about 65% for humans. Error analysis reveals concentrated failures in occlusion inference, scale-consistency, and multi-step navigational planning, indicating limits in compositional spatial representations and reasoning. Our comprehensive study highlights which interventions provide robust gains, which amplify existing biases, and the critical role of perceptual grounding for generalizable spatial reasoning. By providing tasks, baseline results, and diagnostic analyses, SPATIALAB establishes a foundation for progress and reproducible comparison in spatial reasoning evaluation.

Preliminaries: Spatial Reasoning in VLMs

Visual–spatial reasoning, the ability to perceive, represent, and manipulate spatial relations among objects, plays a central role in human cognition and in enabling embodied agents to navigate and act in the world (Palmer, 1999; Chabris et al., 2006). Formally, we can describe a scene as a set of objects O={o1,…,on}O = \{o_1, \ldots, o_n\} with attributes (e.g., position, orientation, size) in a world coordinate system WW. Visual–spatial reasoning then seeks to learn a mapping f:(I,Q)→Rf : (I, Q) \to R, where II is an image (or image sequence), QQ is a linguistic query, and RR is a structured representation of spatial relations R={r(oi,oj)∣oi,oj∈O}R = \{r(o_i, o_j) \mid o_i, o_j \in O\} capturing predicates such as left-of, occludes, or supports. For vision–language models (VLMs), learning ff is highly challenging because it requires integrating noisy visual perception with compositional linguistic abstraction under embodied constraints such as perspective, gravity, and physical interaction [17, 1].

Early approaches to spatial reasoning in VLMs have largely relied on basic spatial relations or narrow task designs (Cheng et al., 2024; Du et al., 2024; Szymanska et al., 2024), often situated in synthetic datasets like CLEVR [35] or structured question–answering benchmarks such as GQA [33] or puzzle-like simulated setups such as MGT (Stogiannidis et al., 2025) and OmniSpatial (Stogiannidis et al., 2025), as we can notice in Table 11. While useful for controlled evaluation, these settings oversimplify spatial structure, omit environmental noise, and neglect dynamic factors such as temporal changes or viewpoint shifts. To overcome these limitations, we argue for the need to benchmark spatial reasoning in rich, real-world contexts that capture scene complexity, temporal variation, and multiple perspectives. Such settings expose VLMs to challenges they would encounter in embodied or interactive applications, including robotics, AR/VR, and autonomous navigation (Mu et al., 2023; Collaboration et al., 2023).

DatasetE.T.C.DomainSourceA.SizeCountDf.Cmpl.Venue/YearTypeDepthA.L.Md.Best BaselineB.B.P.
ScanQAY7IndoorE.D.M80041KHECVPR’22CClose-endedLLI--
Visual SpatialY7R.W.E.D.T10K10KEETACL’23MCQ,Bin.LMILXMERT70.10
What’s upN6IndoorE.D.T5K5KEEEMNLP’23CMCQSHiIXVLM-COCO60.40
EmbSpatial-BenchY6IndoorE.D.T2.2K3.6KHEACL’24CMCQ,%MSIQwen-VL-Max49.11
Space3D-BenchN6IndoorE.D.M2111KMmEECCV’24WMixLSIRAG3D-Chat66.80
SpatialRGPT-BenchN12MixE.D.T1.5K1.5KMmENeurIPS’24CMCQ,Bin.MLIGPT-4o57.83
BLINK-SpatialY14MixE.D.M7.3K3.8KHEECCV’24CMCQ,Bin.MMIGPT-4o59.03
Spatial-MMN4InternetInternetT2.3K2.3KEMmEMNLP’24COpenLHiIGPT-4 Vision63.82
RoboSpatialY4IndoorE.D.T1M3MHECVPR’25COpen,Bin.LSILEO71.90
SpatialVLMN2WebE.D.T10M2BHECVPR’24COpenLSIGPT-4V68.00
VSI-BenchY8IndoorM,E.D.T2885KMmMmCVPR’25CMCQ,Nm.MMVGemini-1.5 Pro45.40
OmniSpatialY50InternetInternetM,P13871.5KHMm2025MCQ,Bin.MSIo3-2025-04-1656.33
Mind the GapN6MixAI-GenM,P1.8K1.8KEMm2025MixMSIInternVL2.5-26B48.83
VLM4DN4MixMixM1K1.8KEHICCV’25MCQHiMMixGemini-2.5 Pro62.00
SPATIALAB-MCQY30MixMix,MM1.2K+1.5KHH2025MCQHiHMixInternVL3.5-72B54.93
SPATIALAB-OPENY30MixMix,MM1.2K+1.5KHVH2025OpenHiHMixGPT-5-mini40.93

Table 1. Comparison of spatial reasoning benchmarks.

In SPATIALAB, we organize spatial reasoning into six principal categories, Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each linked to a distinct cognitive faculty studied in psychology and spatial cognition (Baddeley, 1998; Previc, 1998).

As shown in Figure 1 and Appendix § 12, each category is further decomposed into five subcategories, producing a taxonomy of thirty task types that isolate fine-grained reasoning skills, such as directional alignment, occlusion inference, or stability prediction. This classification scheme is motivated by the need to capture the full spectrum of visual–spatial reasoning challenges. By systematically breaking down complex reasoning into measurable and testable units, we can evaluate not only whether models “solve” spatial tasks but also which specific dimensions remain fragile or underdeveloped [23, 31, 26] (Yang et al. 2023).

Moreover, we adopt two complementary evaluation formats: multiple-choice and open-ended question answering. The multiple-choice format probes recognition and fine discrimination among distractors, while the open-ended format measures generative reasoning and compositional description [18] (Liu et al. 2025; Zheng et al. 2022). Together, these formats reflect real-world reasoning demands and expose different strengths and weaknesses in model capabilities. By combining a principled taxonomy, diverse real-world imagery, and dual evaluation formats, SpatiaLab provides a structured framework for systematic benchmarking. This setup enables fine-grained model comparison, alignment with human-level performance, and deeper insights into the development of robust spatial cognition in artificial agents.

SpatiaLab Benchmark

Spatial reasoning remains a persistent challenge for vision–language models (VLMs), especially in naturalistic conditions where occlusion, perspective distortion, and noisy backgrounds co-occur. To address this, we present SpatiaLab, a benchmark designed to systematically evaluate VLMs’ capacity for complex spatial cognition under realistic visual conditions. The benchmark comprises 1,400 visual question–answer pairs organized into six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each further divided into five subcategories, yielding 30 distinct task types. Every subcategory includes at least 25 QA pairs, while each top-level category contributes a minimum of 200, ensuring balanced coverage across the taxonomy. SpatiaLab supports both multiple-choice and open-ended evaluation, enabling assessment of discriminative accuracy as well as generative reasoning and explanatory competence, thus aligning with broader multimodal evaluation practices while emphasizing spatial-specific reasoning. High visual diversity, fine-grained annotation, and rigorous quality control make SpatiaLab a reliable and scalable testbed, and Figure 2 illustrates representative examples from the six categories, showcasing both open-ended and multiple-choice formats that capture the richness and difficulty of real-world spatial reasoning tasks.

Figure 2. Representative examples from six categories in open-ended and MCQ Tasks.

Benchmark Construction

Image Collection. The image corpus was curated using three complementary strategies: (i) automated large-scale web crawling, (ii) targeted online retrieval of scene-specific images, and (iii) manual snapshots collected in diverse indoor and outdoor environments (Figure 3). This multi-source pipeline ensures diversity across object categories, environmental conditions, and spatial configurations. To explicitly capture natural variability, images were profiled along six meta-dimensions: lighting condition (high contrast, low contrast, shadows, and reflective settings), texture complexity (uniform, patterned, and complex), edge complexity (sharp and smooth boundaries), *dominant spatial relation (stacked, scattered, aligned), material type (transparent, translucent, opaque, reflective), and gravity constraints (normal, floating, and unconstrained). Each dimension was systematically represented in the final corpus, ensuring that the benchmark reflects a broad spectrum of real-world, in-the-wild scenarios rather than simplified or synthetic setups. Complexity analysis of the corpus reveals that on average images contain 21.48 objects, of which 11.88 are partially visible, distributed across 3.23 distinct depth layers. Spatial reference chains (e.g., “object A is left of B, which is behind C”) contain 2.07 links, indicating the presence of multi-step relational reasoning. Also, 16.71% of the questions require mental rotation; 56.07% require object specificity (specific color, pattern, or other details). Such statistics posits that SPATIALAB captures cluttered, layered, and relationally rich scenes, which are characteristic of real-world spatial reasoning, underrepresented in synthetic datasets.

Data creation pipeline for SPATIALAB

Figure 3. Data creation pipeline for SPATIALAB. Images are collected via web crawling, targeted search, and manual snapshots, followed by structured annotation of spatial question–answer pairs.

Annotation. Annotation proceeded in three phases. First, annotators underwent targeted training (details are available in Appendix § 13.§ 2) to ensure consistency in interpreting spatial categories and subcategories. Second, each image was paired with one or more spatial QA tasks covering a range of reasoning types such as proximity gradients, alignment patterns, occlusion inference, and stability prediction. Questions were designed to balance perceptual grounding with higher-order inference (e.g., predicting stability under gravity constraints). Third, QA pairs were encoded in both multiple-choice (04 options) and open-ended formats, ensuring comparability across evaluation paradigms. In total, it produces around 1,500 spatial visual QA pairs, across six categories and 30 subcategories.

Review and Quality Control. To ensure reliability, annotation outputs underwent a three-tier review and validation pipeline. In the first stage, QA items were checked for semantic validity and alignment with the designated subcategory. In the second stage, independent annotators verified correctness of both the question phrasing and the designated answer, focusing on eliminating ambiguity and enforcing task clarity. In the third stage, a gold-standard annotation round was conducted, establishing a set of high-confidence QA pairs for evaluation. All metadata was removed from the images, and the dataset was carefully curated to avoid copyright concerns, though minor oversights may still occur. This procedure yielded a final benchmark of 1,400 validated QA items.

Experiments and Evaluation

Models. We categorize the evaluated models into four groups. Proprietary models include GPT-4o-mini (OpenAI, 2024), GPT-5-nano, Gemini-2-Flash and Gemini-2.5-Flash ([24]), and Claude 3.5 Haiku ([2]). Open-source models comprise Intern-VL3 (1B, 2B, 4B) (Zhu et al., 2025), Qwen-VL2.5 (3B, 7B, 32B, 72B Instruct) ([5]), GLM 4.5V-3.06B and GLM-4.1V-9B-Thinking ([24]), Gemma-3 (4B, 27B) ([24]), Llama-3.2 (11B, 90B) Vision-Instruct ([25]), and Step-3-321B-MoE (StepFun, 2025). Reasoning models include o4-mini (OpenAI, 2025), Gemini-2-Flash and Gemini-2.5-Flash thinking variants ([24]), and Kimi-VL-A3B-Thinking-2506 ([24]). Spatial reasoning specialists comprise SpaceOm, SpaceThinker-Qwen2.5-VL-3B, and SpaceQwen2.5-VL-3B-Instruct (Chen et al., 2024a). For GLM-4.5V-106B-MoE, GLM-4.1V-9B-Thinking, and Step-3-321B-MoE we only performed open-ended evaluation, as these models are not instruction-tuned and were trained primarily on open-ended tasks, resulting in excessive refusals during MCQ evaluation. For ease of comparison in our results tables, we use color coding: proprietary, open-source up to 7B, larger than 7B, proprietary reasoning, open-source reasoning, spatial specialists, and human baseline.

Evaluation Process and Metrics. We measure accuracy as the primary metric for both MCQ and open-ended tasks. For MCQ evaluation, we use direct prompting, where models return the option number corresponding to their selected answer. Responses are automatically checked against the correct answer. Detailed prompts and details are provided in Appendix § 14.§ 1. For open-ended evaluation, models are prompted to generate free-form answers, which are then assessed using a large language model judge. Prompts and details are available Appendix § 14.§ 2. To evaluate the performance of LLM Judge with humans, we performed LLM-human agreement analysis too, as detailed in Appendix § 14.§ 2. Table 5 demonstrates that the LLM judge (Gemini-2.5-Flash) achieves substantial agreement with human annotators, with a Cohen’s kappa of 0.738 against the majority vote and 0.681–0.795 against individual annotators. Raw accuracy against the majority vote is 0.880, while Fleiss’ kappa among human annotators is 0.774, indicating strong reliability. For additional details on experiment setup, prompts, and evaluation procedures, see Appendix § 14.

Categoryκ\kappaMajorityκ\kappaHuman1κ\kappaHuman2κ\kappaHuman3Acc. (majority)Fleiss κ\kappa (all)
Overall0.7380.7380.6810.7950.8800.774
3D Geometry1.0001.0000.8751.0001.0000.916
Depth & Occlusion0.8050.8050.8990.6970.9090.871
Orientation0.5000.5000.4260.5880.8100.571
Relative Positioning0.8330.6570.6760.8330.9170.778
Size & Scale0.4170.5710.4170.6960.7140.704
Spatial Navigation0.8150.8150.6671.0000.9330.773

Table 5. Agreement metrics between Gemini-2.5-Flash as an automatic judge and three human annotators across all categories. The table reports Cohen’s kappa with the majority vote and each annotator, raw accuracy against the majority, and Fleiss’ kappa among all annotators. Bold values highlight the highest agreement within each category.

Improving Visual Reasoning Capabilities. To enhance spatial reasoning on our diverse benchmark, we explore reasoning strategies including inherent reasoning in the VLMs (Appendix § 18.§ 1), Chain-of-Thoughts (CoT) (Appendix § 18.§ 2), CoT with self-reflection (Appendix § 18.§ 3), supervised fine-tuning (SFT) (Appendix § 18.§ 4), and multi-agent systems (Appendix § 18.§ 5). For SFT, specifically, Qwen-VL2.5-3B-Instruct was fine-tuned on 40% of the dataset with stratified sampling, improving generalization across relational and geometric tasks. Evaluation on the remaining 60% showed reduced errors in navigation and orientation, with additive gains when integrating CoT and SFT. However, agentic reasoning revealed that while orientation benefits substantially, other categories stagnate or degrade, underscoring that deeper perceptual and spatial grounding is required beyond multi-step reasoning.

Findings and Analysis

Overall Results

SpatialLab-MCQ. Table 2 demonstrates model performances in multiple choice evaluation on SpatialLab-MCQ. We observe substantial variation in performance across models, tasks, and families: overall model accuracies span roughly 30–55% (random choice = 25%), while the human baseline is 87.57%, indicating large headroom. InternVL3.5-72B, an open source model, is the best performing model, with 54.93% accuracy. Reasoning- and instruction-tuned models (e.g., o4-mini: 53.21%, Gemini-2.5-Flash-Thinking: 52.93%, GPT-5-mini: 54.29%) and some very large open-source models (e.g., InternVL3.5-72B: 54.93%, Llama-3.2-90B: 50.36%) rank near the top, but model scale alone is not determinative: for example, Llama-3.2-11B attains only 30.50% despite its substantial size. Category-wise, Orientation and 3D Geometry are where many strong models excel (several models exceed 60% on orientation or 3D geometry), whereas Spatial Navigation, Depth & Occlusion, and Size & Scale are more variable and generally harder across the board. Notably, the spatially specialized models (SpaceOm, SpaceThinker, SpaceQwen) achieve only mid-40s to low-40s overall, so specialization does not automatically translate to higher MCQ accuracy across our diverse question set. Depth & Occlusion and Size & Scale show mixed results, some architectures handle occlusion relatively well while others fail, pointing to differing inductive biases or training data. These patterns imply that instruction-tuning and reasoning capability particularly help geometric and orientation reasoning, while multi-step grounding tasks like navigation remain an open weakness.

Model3D Geom. (#238)Dep. & Occu. (#259)Orientation (#202)Relat. Posit. (#212)Size & Scale (#252)Spati. Navig. (#237)Overall (#1400)
Random Choice25.0025.0025.0025.0025.0025.0025.00
Proprietary Models
GPT-4o-mini47.0639.0047.0347.1749.6049.7946.50
GPT-5-mini48.7454.8360.4062.7444.8456.5454.29
Gemini-2.0-Flash47.0655.2153.9658.0254.3746.8452.50
Gemini-2.5-Flash44.9648.2648.0256.1342.4651.0548.29
Gemini-2.5-Pro47.4850.1949.5058.4943.6552.3250.07
Claude 3.5 Haiku42.4442.0846.5346.2335.7145.9942.93
Mistral Medium 3.146.6449.8147.5261.7941.6741.7747.93
Open-Source Models
InternVL3.5-1B33.6132.4323.2737.2631.7530.8031.64
InternVL3.5-2B34.0331.6631.6840.5732.5432.4933.71
Qwen-VL2.5-3B-Instruct41.1835.5246.0440.0947.2239.2441.43
InternVL3.5-4B42.8642.8642.0854.7236.5142.1943.29
Gemma-3-4B-it43.7034.3646.5345.7537.3037.9740.57
Qwen-VL2.5-7B-Instruct42.8637.8442.5746.2342.0635.4441.00
Llama-3.2-11B-Vision-Instruct26.4730.5020.3042.9230.5632.0730.50
Gemma-3-27B-it43.2840.1548.0254.2548.0247.2646.57
Qwen-VL2.5-32B-Instruct41.1840.1546.5345.2845.2441.7743.21
InternVL3.5-72B50.0057.1453.4766.0449.2154.8554.93
Qwen-VL2.5-72B-Instruct47.0648.6551.9854.2543.6548.9548.86
Llama-3.2-90B-Vision-Instruct46.2252.1250.5058.9646.8348.5250.36
Reasoning Models
o3-mini50.0054.8351.9861.3238.8950.2150.93
o4-mini-medium51.2658.3054.9564.1540.8751.4853.21
Gemini-2-Flash-Thinking37.8241.3141.5845.7550.4043.0443.36
Gemini-2.5-Flash-Thinking45.8053.6752.9756.6055.1653.5952.93
Kimi-VL-A3B-Thinking-250642.8641.3140.5951.4239.6841.3542.71
Spatial Reasoning Models
SpaceOm42.4438.6148.0237.7442.8639.2441.36
SpaceThinker-Qwen2.5VL-3B40.3437.8447.0338.2143.2537.9740.64
SpaceQwen2.5-VL-3B-Instruct31.5135.1437.6237.7450.7947.2640.14
Human Baseline93.7074.1391.5891.5188.8987.7687.57

Table 2. Multiple Choice Evaluation Accuracy (%)(↑\uparrow) on SpatialLab-MCQ by Question Categories.

SPATIALAB-OPEN. Open-ended evaluation scores presented in Table 3 are substantially lower than MCQ results and show a wide spread: overall accuracy ranges from about 9.6% (InternVL3.5-1B) up to 40.9% (GPT-5-mini), while the human baseline sits at 64.9%, indicating a large gap to close. Proprietary models generally lead the leaderboard on open-ended tasks (e.g., GPT-5-mini 40.93%, GPT-4o-mini 26.00%, Gemini-2.5-Flash 30.93%), but reasoning-tuned variants stand out in particular, the o4-mini (37.86%) and Gemini-2.5-Flash-Thinking (32.77%) perform noticeably better than many baseline open-source counterparts, suggesting instruction/CoT-style tuning substantially improves generative outputs. Most open-source small models cluster at the bottom (many in the teens; InternVL3.5-1B = 9.64%, Qwen-VL2.5-3B = 12.93%), whereas some very large open models improve to the low-20s (e.g., InternVL3.5-72B = 23.36%, Qwen-VL2.5-72B = 24.64%), showing that scale alone yields only modest gains. Spatial-specialist models (SpaceOm, SpaceThinker, SpaceQwen) attain only 10–13% overall, implying that architecture or task specialization does not automatically translate to robust open-ended generation across our diverse question set. Across subtasks we observe relatively better performance on orientation and certain relative-position items for top models (e.g., GPT-5-mini Rel.Pos = 49.53%, Ori = 37.13%), while depth & occlusion, size & scale, and especially spatial navigation remain consistently hard (many models score <30% on these subtasks). This pattern reinforces the earlier finding that multi-step grounding and perceptual reasoning are the bottlenecks for open generative answers.

Model3D Geom. (#238)Dep. & Occu. (#259)Orientation (#202)Relat. Posit. (#212)Size & Scale (#252)Spati. Navig. (#237)Overall (#1400)
Proprietary Models
GPT-4o-mini23.5316.6023.2730.6617.8621.9426.00
GPT-5-mini45.3834.7537.1349.5342.4637.1340.93
Gemini-2.0-Flash31.9324.3227.2331.1326.1924.4727.43
Gemini-2.5-Flash34.0326.6431.6838.6826.5929.5430.93
Gemini-2.5-Pro37.1445.4536.3637.1423.9124.4433.61
Claude 3.5 Haiku26.0518.9224.7525.9420.2421.1022.64
Mistral Medium 3.125.2119.3121.7829.2515.0816.8821.00
Open-Source Models
InternVL3.5-1B05.8809.659.9013.6809.1310.1309.64
InternVL3.5-2B12.1811.2010.8923.5811.9018.1414.50
Qwen-VL2.5-3B-Instruct15.558.4915.3510.8518.259.2812.93
InternVL3.5-4B19.3317.7615.8419.8116.2718.9918.00
Gemma-3-4B-it20.1713.1314.8523.5815.0819.8317.64
Qwen-VL2.5-7B-Instruct15.1315.8320.3027.8315.8719.8318.86
Llama-3.2-11B-Vision-Instruct16.8116.9922.2825.0013.4918.5718.57
Gemma-3-27B-it22.6916.2224.7534.4322.6221.9423.43
Qwen-VL2.5-32B-Instruct16.3909.6514.8516.9809.9213.5013.36
InternVL3.5-72B22.6920.4620.3031.6019.8426.1623.36
Qwen-VL2.5-72B-Instruct26.8920.8525.2530.6624.6020.6824.64
Llama-3.2-90B-Vision-Instruct22.6923.1721.2928.3021.8327.0024.00
GLM-4.5V-106B-MoE31.0920.4625.2526.4224.2124.4725.21
Reasoning Models
o3-mini39.0830.1229.7039.1541.2730.3835.00
o4-mini-medium40.7632.8232.1842.9244.0534.1837.86
Gemini-2-Flash-Thinking31.0927.4131.1934.4329.3729.5430.36
Gemini-2.5-Flash-Thinking37.1445.4536.3637.1421.7422.2232.77
Kimi-VL-A3B-Thinking-250613.4514.2911.8817.9212.7018.5714.79
GLM-4.1V-9B-Thinking21.4319.3126.7324.5319.8422.7822.21
Step-3-321B-MoE32.3518.5327.7227.3629.7627.8527.14
Spatial Reasoning Models
SpaceOm12.6106.9515.8411.7918.6512.2412.93
SpaceThinker-Qwen2.5VL-3B13.4509.2717.8210.3819.4410.1313.36
SpaceQwen2.5-VL-3B-Instruct12.6103.8613.8609.4311.9011.3910.36
Human Baseline73.5350.1970.3069.8165.4862.8764.93

Table 3. Open-ended Evaluation Accuracy (%)(↑) on SPATIALAB-OPEN by Question Categories.

Performance Drop in Open-Ended Evaluation Compared to MCQ

Across 25 models, the average MCQ→\toopen-ended performance gap in SpatialLab is 23.0% (σ\sigma = 5.5%), with subtask means ranging from 22.89% (spatial navigation) to 24.57% (3D geometry). Specialist spatial reasoning models exhibit the largest gaps (about 27%), particularly in spatial navigation (up to 36.68%) and orientation (34.44%), while reasoning-oriented models achieve smaller gaps (around 19%) and lower subtask variance, reflecting the stabilizing effect of instruction-tuning and CoT decoding. Correlations confirm that spatial navigation dominates overall disparity (Pearson rr = 0.99), with orientation (rr = 0.83) and 3D geometry (rr = 0.79) also contributing. Negative or near-zero gaps (e.g., Llama-3.2-11B: -1.98 in depth & occlusion; o4-mini: -3.18 in relative position) indicate that poorly designed MCQ distractors can misrepresent true competence. We hypothesize that format sensitivity arises from MCQ’s structural advantage, specialization bias toward categorical tasks, sequential reasoning challenges in spatio-navigation, and calibration differences in generative output. These findings suggest that MCQ alone can overestimate practical spatial reasoning ability. We recommend complementing MCQ with open-ended evaluations, auditing distractors, employing stepwise generation or instruction-tuning, and reporting per-subtask diagnostics to capture true model performance and reduce format-dependent artifacts. More details are in Appendix § 17 and Table 18.

Table 18: Performance gap between MCQ and Open-ended (%) (↓) across spatial categories on SPATIALAB. A lower value indicates that a model performs comparably in both setups, whereas a higher value reflects greater disparity. We bold and underline the highest gap within each model group.

Model3D Geom. (#238)Dep. & Occu. (#259)Orientation (#202)Relat. Posit. (#212)Size & Scale (#252)Spati. Navig. (#237)Overall (#1400)
Proprietary Models
GPT-4o-mini22.4023.7616.5131.7427.8520.5023.53
GPT-5-mini20.0823.2713.212.3819.4113.363.36
Gemini-2.0-Flash30.8926.7326.8928.1822.3725.0715.13
Gemini-2.5-Flash21.6216.3417.4515.8721.5117.3610.93
Claude 3.5 Haiku23.1621.7820.2915.4724.8920.2916.39
Mistral Medium 3.130.5025.7432.5426.5924.8926.9321.43
Open-Source Models (Small)
InternVL3.5-1B22.7813.3723.5822.6220.6722.0027.73
InternVL3.5-2B20.4620.7916.9920.6414.3519.2121.85
Qwen-VL2.5-3B-Instruct27.0330.6929.2428.9729.9628.5025.63
InternVL3.5-4B25.1026.2434.9120.2423.2025.2923.53
Gemma-3-4B-it21.2331.6822.1722.2218.1422.9323.53
Qwen-VL2.5-7B-Instruct22.0122.2718.4026.1915.6122.1427.73
Llama-3.2-11B-Vision-Instruct13.51-1.9817.9217.0713.5011.939.66
Gemma-3-27B-it23.9323.2719.8225.4025.3223.1420.59
Qwen-VL2.5-32B-Instruct30.5031.6828.3035.3228.2729.8524.79
InternVL3.5-72B36.6833.1734.4429.3728.6931.5727.31
Qwen-VL2.5-72B-Instruct27.8026.7323.5919.0528.2724.2220.17
Llama-3.2-90B-Vision-Instruct28.9529.2130.6625.0021.5226.3623.53
Reasoning Models
o4-mini25.4822.7721.23-3.1817.3015.3510.50
Gemini-2-Flash-Thinking13.9010.3911.3221.0313.5013.006.73
Gemini-2.5-Flash-Thinking8.2216.6119.4633.4231.3720.168.66
Kimi-VL-A3B-Thinking-250627.0228.7133.5026.9822.7827.9229.41
Spatial Reasoning Models
SpaceOm31.6632.1825.9524.2127.0028.4329.83
SpaceThinker-Qwen2.5VL-3B28.5729.2127.8323.8127.8427.2826.89
SpaceQwen2.5-VL-3B-Instruct31.2823.7628.3138.8935.8729.7818.90

Table 18.

Error Analysis

SpatialLab-MCQ. Our analysis (Appendix § 15) shows that VLMs achieve selective peaks but lack holistic spatial competence. Closed-source models reach the highest scores (e.g., 85.71% in Stacking Orientation), yet collapse to near-chance in Relative Size Comparison. Open-source scaling improves ceilings (80.56% in Corner/Angle Positioning) but does not prevent catastrophic failures (2.0% in Object Rotation). Reasoning-augmented and spatially specialized models add localized gains, such as in occlusion inference or navigation, but remain capped below 55% in many physical abstraction tasks (Gravity Effects, Stability Prediction). Error distributions further reveal systematic weaknesses in embodied reasoning (Tool Handedness), recursive relational chaining (Pathway Existence), and non-local cues (Reflective Surfaces), while stronger results in Obstacle Avoidance suggest shortcut exploitation rather than robust planning. Taken together, these patterns confirm that current VLMs rely heavily on surface-level correlations and lack stable encodings of orientation, physics, and compositional logic. Progress will likely require both richer training distributions (e.g., embodied and physics-driven data) and architectural mechanisms for geometric and reference-frame grounding.

SpatialLab-Open. Error analysis of SpatialLab-Open (Appendix § 16) reveals that even top-performing closed-source models (e.g., GPT-5-mini: 58.14% in Directional Relations) collapse on tasks like Proximity Gradients, indicating that dominance remains task-specific. Large open-source models achieve strong results in Depth & Occlusion (e.g., 60.0%) but still fall to 9.3% on Proximity, showing that scale alone cannot ensure robustness. Smaller models exhibit isolated strengths (e.g., 26.0% in Relative Size Comparison) but often break down completely (e.g., 0.0% in Betweenness), reflecting the limitations of compact architectures. Reasoning-tuned systems show promise, Gemini-2.5-Flash-Thinking reaches 75.0% in Tool Handedness, yet similar models fall to single digits, revealing fragile integration. Specialized spatial models perform worst overall, rarely exceeding 20% and frequently failing on core relational tasks. Failure patterns highlight systemic weaknesses in occlusion handling, orientation, and multi-step relational chaining, while partial success in size–scale cues suggests reliance on superficial correlations. These results point to three root causes: missing representations for spatial analogies, architectural limits tied to scale and training diversity, and brittle reasoning pipelines underscoring the requirement for a unified geometric encodings, physics-aware reasoning, and embodied data to move from isolated peaks toward consistent spatial competence.

Qualitative Error Analysis. To better understand model weaknesses, we conducted a qualitative error analysis (see Appendix § 19) and found that failures cluster into a small number of recurring classes rather than being random noise. Common errors include spatial mislocalization, where models confuse referents in crowded scenes; perspective and scale mistakes, where reliance on object-size priors overrides image-based cues; and occlusion or ordering failures, especially with thin or partially hidden structures. We also observe attribute confusion, such as mixing perceptual properties with functional ones, and open-ended rationalizations that generate fluent but visually ungrounded narratives. These errors are consistent across architectures and prompting styles, suggesting structural biases in current VLMs rather than dataset-specific noise. Importantly, failure rates spike when multiple cues must be fused, such as combining depth ordering with relative size, and confidence calibration is especially poor in open-ended settings. Our diagnostic protocols confirm that models often ignore minimal but decisive visual features, rely on brittle heuristics, and fail to update when counterfactual edits are applied. The root causes point to insufficient object-centric binding, lack of geometric supervision, and training objectives that reward plausibility over grounding. Together, these findings highlight that current VLMs achieve strong coarse perception but struggle with multi-cue integration and grounded reasoning, underscoring the need for geometry-aware supervision, multi-scale feature retention, and verification pipelines.

CategoryMCQ BeforeMCQ AfterMCQ GainOpen-ended BeforeOpen-ended AfterOpen-ended Gain
3D Geometry30.0737.767.6920.2823.783.50
Depth & Occlusion23.8732.909.0343.2345.812.58
Orientation23.3336.6713.3339.6742.152.48
Relative Positioning26.1927.781.5951.1850.39−0.79
Size & Scale27.6344.7417.1118.4214.47−3.95
Spatial Navigation22.3838.4616.0836.6239.442.82
Grand Total25.6336.5910.9734.4035.481.07

Table 4.

Performance Improvement Approaches

Inherent Reasoning Mechanisms. We notice that reasoning-enabled models consistently outperform their baselines in both MCQ and open-ended formats, with the largest gains in relational and orientation tasks (e.g., +13.1% in Relative Positioning for MCQ). While reasoning helps stabilize open-ended performance, improvements remain uneven across categories (notably declining in Size & Scale), underscoring that reasoning modules boost logical consistency but do not fully solve grounding or scale sensitivity. Detailed per-category comparisons are provided in Appendix § 18.§ 1.

Chain-of-Thought (CoT) Prompting. Across models, CoT prompting provides little benefit and often reduces accuracy (Table 19), with orientation being the only category showing consistent gains. This indicates that while step-by-step reasoning helps with directional alignment, it fails in tasks requiring robust perceptual grounding such as depth, scale, and spatial navigation. Unlike in textual reasoning, CoT here tends to amplify flawed priors rather than correct them, underscoring that solving SPATIALAB requires perceptual understanding beyond logical chaining (details in Appendix § 18.§ 2).

CategoryInternvVL-3-78b (MCQ) w/o CoTInternvVL-3-78b (MCQ) with CoTInternvVL-3-78b (MCQ) GainInternvVL-3-78b (Open) w/o CoTInternvVL-3-78b (Open) with CoTInternvVL-3-78b (Open) GainGemini-2.5-Flash (MCQ) w/o CoTGemini-2.5-Flash (MCQ) with CoTGemini-2.5-Flash (MCQ) GainGemini-2.5-Flash (Open) w/o CoTGemini-2.5-Flash (Open) with CoTGemini-2.5-Flash (Open) Gain
3D Geometry52.0052.000.0028.0028.000.0048.0020.00-28.0060.0028.00-32.00
Depth & Occlusion60.0060.000.0016.0012.00-4.0048.0028.00-20.0040.0016.00-24.00
Orientation60.0064.004.0024.0028.004.0056.0024.00-32.000.0032.0032.00
Relative Positioning80.0072.00-8.0044.0056.0012.0060.0036.00-24.0050.0032.00-18.00
Size & Scale56.0044.00-12.0028.0020.00-8.0048.0060.0012.0025.008.00-17.00
Spatial Navigation64.0064.000.0048.0040.00-8.0068.0052.00-16.0040.0028.00-12.00
Overall62.0059.33-2.6731.3330.67-0.6654.6736.67-18.0038.2424.00-14.24

Table 19. Performance with CoT prompting across categories for InternvVL-3-78b and Gemini-2.5-Flash. Values are accuracies (%) and Gains represent the difference between with and without CoT prompting.

Chain-of-Thought (CoT) with Self-Reflection. Adding self-reflection to CoT prompting, where models are explicitly prompted to review their previous reasoning step-by-step and correct mistakes (see Figure 8 in Appendix for prompt), yields modest benefits in multiple-choice settings, especially for geometry and depth. However, this fails to generalize to open-ended tasks (Appendix § 18.§ 3). This confirms that reflective reasoning cannot compensate for missing perceptual grounding, highlighting a core limitation of current VLMs.

Chain-of-thought (CoT) with Self-Reflection PromptsVector (PDF)
SpatiaLab-MCQ : CoT > Self-Reflection : Prompt for Answering Model (Self-Reflection)

You are a spatial reasoning assistant. Review your previous reasoning and response below for the given question and options. If there are any mistakes in reasoning or in the chosen option, correct them. Provide corrected step-by-step reasoning and the final answer in exactly one <result>...</result> tag. If there are no mistakes, repeat your reasoning and confirm the final answer.

Question: {question}
Options: 1. {opt1} 2. {opt2} 3. {opt3} 4. {opt4}

Previous model output:
{prev_output}

Provide corrected reasoning followed by the final answer tag, and nothing else.

SpatiaLab-Open : CoT > Self-Reflection : Prompt for Answering Model (Primary Ans)

You are a spatial reasoning assistant.
Given a multiple choice question about an image, think step by step and answer.

Provide your reasoning in 2-4 sentences, then provide the final answer on a separate line like this:

Reasoning: <your reasoning here>
Answer: <your answer here>

Question:
{question}

SpatiaLab-Open : CoT > Self-Reflection : Prompt for Answering Model (Self Reflection Ans)

You are a spatial reasoning assistant and a strict self-reflector.
You will receive the question, the model's reasoning, and the model's answer.
Check the reasoning and answer for any mistakes. If mistakes exist, correct them.
Provide the corrected reasoning and answer in the same format:

Reasoning: <corrected reasoning here>
Answer: <corrected answer here>

Question:
{question}

Model reasoning:
{reason}

Model answer:
{answer}

Figure 8. Chain-of-thought (CoT) with Self-Reflection Prompts.

Supervised Fine-Tuning (SFT). As shown in Figure 4, supervised fine-tuning (SFT), conducted on Qwen2.5-VL-3B-Instruct using a stratified 40% split of SPATIALAB (~560 samples), consistently improves MCQ accuracy across all spatial reasoning categories. However, it offers little, and sometimes negative, transfer to open-ended tasks. This divergence suggests overfitting to task-specific answer distributions, reinforcing concerns that SFT often teaches to the test rather than fostering generalizable reasoning (Wang et al., 2025). The sharp drop in open-ended performance points to biased internal representations, tuned for categorical discrimination but unstable in generative settings. A likely cause is catastrophic forgetting of linguistic priors during fine-tuning [30], compounded by objectives that favor discrete-choice alignment over spatial reasoning in language. Overall, while SFT enhances performance in structured formats, it risks creating an algorithmic straightjacket, boosting accuracy in constrained tasks while stalling progress in naturalistic spatial reasoning. More details are available in Appendix § 18.§ 4.

SFT performance trends over epochs and performance comparison across categories for MCQ and Open-ended formats

Figure 4. SFT training results: (a) learning trends over epochs, and (b) final accuracy values.

AI Agents for Spatial Reasoning (SPATIOXOLVER). To explore agentic capabilities, we employ SPATIOXOLVER, a multi-agent system adapted and extended from Xolver [28] designed to perform structured spatial reasoning on images. This framework decomposes complex visual reasoning into specialized sub-tasks managed by dedicated agents, including modules for object segmentation, attribute extraction, spatial relation mapping, and transformation tracking. By consolidating these outputs into a unified structured representation, the system attempts to propagate low-level perceptual cues through higher-order relational inferences. We implemented the pipeline using Gemini-2.5-Flash with low temperature to ensure deterministic outputs during the multi-stage refinement process. Experimental results demonstrate substantial gains in the Orientation category, improving by +8.00% in MCQ and an impressive +36.00% in open-ended evaluation. However, this success is not universal; categories such as Depth & Occlusion and Spatial Navigation suffered severe declines of -24.00% and -12.00% in open evaluation, respectively. These drops indicate that when inherent perceptual priors—such as depth or occlusion cues—are weak, multi-step reasoning tends to reinforce misconceptions rather than correct them. Ultimately, these findings suggest that while agentic workflows enhance tasks reducible to sequential alignment, they cannot compensate for a lack of fundamental perceptual grounding in complex 3D environments. More details are available in Appendix § 18.§ 5.

Discussion

Spatial reasoning is central to embodied intelligence, yet most benchmarks rely on synthetic scenes, templated questions, or narrow task scopes that overlook real-world visual variability. We introduce SPATIALAB, a benchmark of 1,400 visual question–answer pairs covering six major spatial categories and 30 subcategories, including 3D geometry, occlusion, and navigation. The benchmark uses both Multiple-Choice and Open-Ended formats to separate option-based pattern use from genuine generative reasoning. Its quality is supported by a multi-stage review process and complexity checks covering lighting, texture, and spatial diversity. Evaluations of more than 25 contemporary models and human baselines show that SPATIALAB functions as a precise diagnostic tool for identifying weaknesses in current vision–language systems.

Detailed error analysis uncovers systematic failures in handling transparency, multi-step navigation, and occlusion, indicating that models often rely on surface-level texture cues rather than internalized 3D geometric representations. While interventions such as Supervised Fine-Tuning (SFT) yielded clear gains in structured MCQ tasks, they simultaneously triggered instability in open-ended reasoning, suggesting a ”catastrophic forgetting” of linguistic priors when optimizing for discrete answers. Furthermore, reasoning-based approaches like Chain-of-Thought proved effective for sequential tasks like Orientation but failed to correct fundamental perceptual errors in Depth and Scale, often amplifying hallucinations instead of resolving them. Agentic decompositions showed similar trade-offs, improving specific alignment tasks while degrading performance in holistic scene understanding contexts. These divergent behaviors confirm that scaling laws and reasoning scaffolds alone are insufficient; achieving human-level spatial intelligence will require architectural innovations that explicitly integrate geometric grounding and physics-aware representations.

Concluding Remarks

Spatial reasoning remains a core bottleneck for VLMs, particularly in visually complex, real-world settings. SPATIALAB introduces 1,400 diverse QA pairs across six spatial categories and thirty sub-categories, enabling fine-grained evaluation via both MCQ and open-ended formats. Despite applying several performance-improving methods, our experiments reveal substantial gaps between state-of-the-art models and human performance, with open-ended generation showing an additional 10–25% drop relative to multiple-choice accuracy. Error analysis reveals persistent failures in depth, occlusion, 3D geometry, and multi-step navigation, reflecting reliance on superficial cues over grounded spatial representations. SPATIALAB exposes these limitations and provides a structured testbed for diagnosing spatial competence. Our benchmark enables holistic evaluation and provides actionable insights for developing VLMs with human-aligned spatial understanding. Future progress will require richer training signals (e.g., embodied interaction, action-conditioned feedback) and architectures capable of geometric grounding, relational chaining, and consistent multi-scale reasoning.

Ethics Statement

We have taken multiple steps to ensure ethical compliance throughout the development of this work. All metadata associated with the images was removed to protect privacy, and any personally identifiable features (such as human faces or license plates) were minimized or excluded wherever possible. The dataset was carefully curated to avoid copyright concerns, with diligent attention paid to sourcing and licensing, though minor oversights may occur despite best efforts. The benchmark construction process was designed with transparency and fairness in mind, ensuring that no demographic groups were singled out or negatively represented. We adhered to the ICLR Code of Ethics and confirm that no human subjects or sensitive personal data were involved.

Reproducibility Statement

To facilitate reproducibility, we provide a detailed description of the dataset construction process, annotation pipeline, and evaluation protocol in the main text and appendix. All filtering, quality-control, and validation steps are explicitly documented, with clear definitions of task categories and annotation criteria. Hyperparameters, model configurations, and training procedures are reported in the appendix to support consistent replication of our results. SPATIALAB is available at: https://spatialab-reasoning.github.io/.

Acknowledgment

We sincerely thank Mahir Absar Khan and Enjamamul Haque Eram for their assistance with data collection. We also acknowledge the Qatar Computing Research Institute (QCRI) for providing computing resources and technical support. Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspregen, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailien Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayaraj, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman,

Related Work

Spatial reasoning in multimodal models. Spatial reasoning, the capacity to understand geometric structures and relational layouts in visual scenes, is central to embodied intelligence. Early multimodal benchmarks such as CLEVR [35] and GQA [33] tested compositional reasoning with synthetic or curated templates, which allowed controlled analysis but provided limited ecological validity. With the advent of large vision–language models (VLMs) [1] (Li et al. 2023; OpenAI 2023), spatial reasoning has been revisited in more naturalistic datasets, though systematic weaknesses remain. Despite strong performance on captioning and visual QA, these models often fail in tasks requiring depth ordering, occlusion inference, or consistent reference-frame alignment. This discrepancy highlights a broader problem: scaling alone does not guarantee progress in structured spatial understanding.

Spatial reasoning benchmarks. A range of benchmarks have recently been proposed to probe these gaps (Table 1). SCANQA [3] was among the first embodied QA datasets, requiring reasoning about 3D indoor environments, but its limited scale (800 scenes) and relatively shallow complexity restrict broader generalization. Datasets such as VISUAL SPATIAL (Liu et al., 2023) and WHAT’S UP [36] expanded scope to real-world or household contexts, but relied on template-driven questions, leading to reduced linguistic variability and an overrepresentation of simple binary or multiple-choice relationships. More recent efforts, including EMBSPATIAL-BENCH [21] and SPACE3D-BENCH (Szymanska et al., 2024), increased task variety yet remained constrained by scale (2.2K and 211 samples, respectively) and annotation style (template-heavy or manual but narrow). SPATIALRGPT-BENCH and BLINK-SPATIAL introduced mixed domains and up to 14 categories, improving diversity, but still emphasized either static snapshots or limited multi-view simulation.

Embodiment, difficulty, and diversity. The trend toward embodiment is visible in datasets like ROBOSPATIAL (Song et al., 2024), which scales up to 1M scenes with robot-centric questions. However, its template-based annotation makes questions less natural and easier to game with heuristic shortcuts. Similarly, SPATIALVLM [12] offers unprecedented scale (billions of QA pairs), but its two-category coverage renders it unsuitable for fine-grained diagnostic evaluation. On the other end, high-quality manual datasets such as VSI-BENCH (Yang et al., 2024) achieve improved annotation fidelity, but with only 288 scenes, they remain insufficient for training or robust cross-model comparison. In terms of complexity, most existing benchmarks cap at “medium” (Mm in Table 1), rarely extending to the “hard” or “extremely hard” levels that require multi-step causal inference. This leaves a wide evaluation gap for next-generation models.

Evaluation protocols. Equally important are evaluation types and analysis depth. Most prior work employs close-ended QA or binary judgments (e.g., VISUAL SPATIAL, EMBSPATIAL-BENCH), which simplify scoring but restrict the expressivity of model outputs. A few datasets such as ROBOSPATIAL, SPATIAL-MM, and SPATIALLAB-OPEN introduce open-ended evaluation, but only SPATIALLAB-OPEN couples this with extremely high complexity, manual puzzle-style annotations, and comprehensive analysis of errors. This enables a deeper understanding of systematic weaknesses beyond accuracy, including calibration and cross-model divergence, which existing datasets rarely capture. Without this granularity, performance gains can be misleading, as models may overfit to dataset-specific biases rather than generalizing robust spatial competence.

Positioning of SPATIALLAB. SPATIALLAB directly addresses these limitations. Compared to prior datasets that span between 2 and 14 task categories, SPATIALLAB covers 30 distinct categories across geometry, occlusion, orientation, relative positioning, size and scale, and navigation, offering unprecedented task diversity. Unlike template-based benchmarks, all questions are manually authored and puzzle-inspired, ensuring linguistic richness and reducing shortcut opportunities. Furthermore, SPATIALLAB is fully embodied and multi-modal, including both static and dynamic visual contexts, thus bridging gaps between robotics-style datasets (ROBOSPATIAL) and web-scale corpora (SPATIALVLM). As shown in Table 1, even the strongest baselines, such as InternVL3.5-72B and GPT-5-mini, perform well below 60%, with the open-ended variant yielding only 40.93% accuracy. This large performance gap demonstrates the difficulty of the benchmark and its utility for driving future research.

Toward next-generation evaluation. By combining high difficulty, broad coverage, manual annotation, and open-ended tasks, SPATIALAB extends the landscape of spatial reasoning benchmarks from template-driven, narrow evaluations to comprehensive, embodied testing. This design makes it possible not only to assess whether a model answers correctly, but also to identify systematic error patterns, probe causal and counterfactual reasoning, and measure calibration reliability. In this sense, SPATIALAB provides a rigorous foundation for the next phase of multimodal research, complementing datasets like MIND THE GAP and VLM4D that explore temporal or generative aspects but remain narrower in coverage. We expect future work to leverage SPATIALAB as a high-bar diagnostic tool, pushing models beyond surface-level recognition toward genuine grounded spatial intelligence.

Detailed Analysis of Limitations in Prior Benchmarks

To situate SPATIALAB within the broader landscape, we conducted a detailed examination of existing spatial reasoning benchmarks. Earlier efforts have contributed important foundations, yet they present structural and methodological constraints that limit their value for evaluating current VLMs. These constraints fall into three areas: synthetic-domain bias, limited task scope, and restrictive evaluation formats. Table 4 summarizes the major limitations identified.

Synthetic and Template-Based Constraints. Benchmarks such as CLEVR and GQA rely on synthetic environments or template-generated linguistic structures. Although these settings offer controlled conditions for isolating compositional reasoning, the visual domain lacks real-world complexity and the linguistic space is narrow. Models can rely on template regularities rather than grounded spatial interpretation. Related simulated benchmarks such as RoboSpatial face similar issues, resulting in a large sim-to-real gap.

Domain and Scale Limitations. Datasets grounded in 3D reconstruction, including ScanQA and Space3D-Bench, provide depth supervision but remain restricted to indoor scenes. This prevents assessment in outdoor or mixed environments that are essential for embodied agents. High-quality manual datasets such as VSI-Bench offer richer annotation but remain small in scale, limiting their usefulness for high-resolution subcategory analysis.

Taxonomic and Evaluation Narrowness. Large-scale datasets such as SpatialVLM cover broad tasks but lack detailed taxonomies, making them unsuitable for diagnosing specific weaknesses in orientation, occlusion, or layout reasoning. Many recent benchmarks, including EmbSpatial-Bench and What’s Up, rely heavily on Binary or MCQ formats. These formats can inflate performance by enabling option elimination rather than requiring generative spatial reasoning. SPATIALAB mitigates these issues by combining MCQ and Open-Ended formats across a detailed taxonomy of 30 spatial subcategories.

BenchmarkDomainEval. FormatTaxonomyPrimary Limitations Identified
CLEVR and GQASyntheticClosed-EndedRelationalArtificial visual domain; template-driven language; prone to shortcut learning.
ScanQAIndoor 3DOpen-EndedRelationalRestricted to indoor point clouds; limited scale (<1K scenes).
SpatialVLMWeb DataOpen-EndedGeneralCovers only a small number of high-level spatial categories.
EmbSpatial-BenchIndoorMCQ / Binary6 CategoriesRelies on closed-ended formats; limited complexity in reasoning chains.
RoboSpatialSimulatorHybridEmbodiedTemplate-generated questions in simulated scenes produce a sim-to-real gap.
Space3D-BenchIndoorMCQ6 CategoriesVery small scale (~200 samples) prevents robust statistical evaluation.
OmniSpatialMixedMCQ / BinaryMetricEmphasizes metric outputs; limited support for explanatory reasoning.
SpatialLab (Ours)In-the-WildMCQ + Open30 Sub-catsAddresses identified limitations with real-world images, detailed taxonomy, and diverse evaluation formats.

Table 4 (PDF p. 22). Limitations identified in representative prior spatial benchmarks compared with SPATIALAB. Limitations are categorized by domain constraints, evaluation format, and task taxonomy.

Task Design and Descriptions

Understanding spatial reasoning requires more than simple recognition of objects; it depends on how those objects relate, align, and interact in physical space. To capture this complexity, we introduce SpatialLab, a benchmark designed to test fine-grained aspects of spatial understanding across diverse categories. The tasks in SpatialLab reflect how humans naturally describe and reason about environments, ranging from relative positioning (e.g., What’s to the left of the microwave?) to geometric inference (e.g., Would this stack fall if another book is added?). Each category is further broken into subcategories that highlight subtle but essential dimensions of reasoning, such as occlusion, orientation, navigation, and physical effects like gravity. By framing each subtask in accessible, everyday terms, SpatiaLab encourages models to go beyond object detection and engage in structured spatial interpretation. This design allows us to probe how well systems generalize across contexts that matter for robotics, AR/VR, autonomous driving, and human-computer interaction. The following sections provide short, task-level descriptions for every category and subcategory in SpatiaLab.

Figure 5. Categories and subcategories of spatial reasoning in SPATIALAB. Each category decomposes into five subcategories, yielding thirty task types in total.

The following outlines the rationale and real-world utility guiding each task design, per category:

Relative Positioning

Directional Relations. Directional relations describe how objects are positioned relative to one another, such as left, right, above, or below. Understanding these relations enables models to interpret spatial layouts more naturally, just like how humans describe scenes. For example, identifying what lies immediately to the left of an appliance helps in scene understanding and robot navigation.

Proximity Gradients. Proximity reasoning involves recognizing which objects are closest or farthest from a reference point. This is key for questions like “Which object is nearest to the doorway?” Models capable of estimating gradients of closeness can support navigation tasks and assistive technologies.

Alignment Patterns. Alignment concerns whether objects follow a certain orientation or order, such as being stacked vertically or arranged in a straight line. Recognizing alignment patterns helps in understanding organization in environments like shelves, tables, or warehouses. Such reasoning is also critical for tasks like robotic sorting and automated inspection.

Betweenness Relationships. Betweenness focuses on identifying objects that occupy the middle ground between two references. For instance, asking what sits between the lamp and the armchair requires reasoning about spatial continuity. This helps in navigation, retrieval, and situational awareness tasks.

Corner/Angle Positioning. Corners and angled positions often serve as reference anchors in spatial reasoning. Questions such as “What’s positioned in the back-right corner?” require the model to localize objects within constrained regions. This kind of reasoning is vital in robotics for mapping and placement.

Depth & Occlusion

Layering Order. Layering requires distinguishing which objects appear in front versus behind. For example, identifying the frontmost item on a table reflects depth reasoning. This is particularly useful in AR/VR applications where visual stacking affects user experience.

Partial Occlusion. Partial occlusion occurs when one object hides part of another from view. Recognizing the hidden item behind a coffee cup, for instance, requires filling in missing information. This strengthens robustness in perception systems operating in cluttered environments.

Complete Occlusion Inference. Sometimes objects are fully hidden, yet clues like shadows or bulges reveal their presence. Inferring what lies under a blanket requires the model to use indirect evidence. Such reasoning mirrors human inference in uncertain visual conditions.

Transparency Effects. Transparent surfaces, such as glass or water, allow partial visibility of objects behind them. Understanding what’s visible through a fish tank requires blending occlusion reasoning with transparency perception. This plays an important role in simulation, graphics, and embodied AI.

Reflective Surfaces. Reflective reasoning asks models to interpret mirrors, shiny metals, or glass reflections. Identifying what is reflected in a mirror involves indirect spatial mapping. This is essential for safety in autonomous vehicles and situational awareness in robotics.

Orientation

Cardinal Direction. Cardinal reasoning relates to compass directions like north, south, east, and west. Asking which way a front door faces requires integrating global orientation cues. This is critical for navigation and geospatial alignment.

Object Rotation. Rotation reasoning measures how much an object has been turned around an axis. Estimating that a chair is rotated by some degrees is crucial for accurate 3D modeling. This helps robots align objects correctly for manipulation.

Facing Direction. Facing direction specifies the orientation of an object’s front side. For example, determining whether a monitor faces north requires mapping its surface orientation to global references. This is important in surveillance and robotics.

Stacking Orientation. Stacking involves understanding whether objects are piled vertically, horizontally, or in mixed orientations. Recognizing such layouts ensures stability in warehouses and homes. It is also critical for safe robotic stacking.

Tool Handedness. Certain tools, like scissors or gloves, are designed for left- or right-handed use. Identifying handedness requires fine-grained shape and orientation perception. This enhances assistive robotics in human-centered environments.

Size & Scale

Relative Size Comparison. Relative comparisons assess whether one object is larger, taller, or smaller than another. Asking if a vase is taller than a statue requires proportional reasoning. This is crucial for packing, fitting, and recognition tasks.

Perspective Distortion. Perspective can make distant objects appear smaller than nearby ones, even if the actual size differs. Identifying illusions like a faraway car looking smaller than a nearby bicycle requires compensating for depth cues. This ability makes models more robust to camera perspectives.

Distance-Size Correlation. Here, size judgments depend on distance estimation. For instance, a nearby bicycle appearing larger than a far car should not confuse true size reasoning. This helps in navigation, AR/VR, and visual grounding.

Scale Consistency. Scale reasoning involves checking whether object sizes are reasonable in context. A mouse larger than a chair signals inconsistency, which models must detect. This strengthens realism in simulated environments.

Shadow-Size Projection. Shadows provide indirect cues about object size and light source position. Estimating how tall a light must be based on shadow length requires geometric reasoning. This is useful in graphics, architecture, and physics-based modeling.

Spatial Navigation

Pathway Existence. Pathway reasoning checks if a clear route exists between two points. For example, determining if one can reach the desk without moving chairs tests navigability. This ability underpins indoor robot planning.

Obstacle Avoidance. Avoidance requires finding routes that minimize collisions. Asking for the clearest path from door to window directly maps to robotic and human navigation. It is essential in autonomous driving and indoor mobility aids.

Viewpoint Visibility. From a given viewpoint, certain objects may be blocked or visible. For example, checking what is obscured when sitting on the couch requires simulating perspective. This strengthens AR/VR realism and robotic vision.

Spatial Sequence. Sequence reasoning identifies the order of encounters along a path. Walking from the kitchen to the balcony, one might pass the dining table first. Such reasoning mimics human navigation memory.

Accessibility Constraints. Constraints identify which objects are blocked from access due to obstacles. For instance, a cabinet may be unreachable because of a table in front. This is crucial for assistive AI and ergonomic design.

3D Geometry

Volume Comparison. Comparing volume asks which object occupies more three-dimensional space. Judging whether a box is larger than a sphere helps in packing and containment tasks. This is important for logistics and simulation.

Stability Prediction. Stability reasoning predicts whether adding or moving objects will topple a stack. For instance, checking if placing a book will collapse a pile mirrors human intuition. This supports safety in robotics and construction.

Shape Projection. Projection tasks ask what shape an object would cast under certain rotations or lighting. For example, a sphere always projects a circle, while a cube may project a square or hexagon. Such reasoning strengthens geometric understanding.

Spatial Containment. Containment checks whether one object can fit inside another. For example, reasoning if a ball can fit in a box supports packing and design. This task aligns with intuitive geometry.

Gravity Effects. Gravity reasoning predicts motion outcomes under physical forces. For example, a rolling ball may stop under a table due to friction and barriers. This strengthens real-world simulation capabilities.

More Details on Benchmark Construction

Annotator Recruitment

We recruited annotators aged between 24 and 30 years, all of whom were engineering graduates. They participated voluntarily and worked independently without external incentives. Before starting the annotation process, we provided structured training sessions to ensure clarity of task requirements and consistency across annotators. We also maintained all ethical and quality control standards throughout the process. This approach ensured that the resulting annotations were both reliable and aligned with the dataset design.

Annotator Training and Protocol

Before beginning annotation, we designed a structured training program to ensure consistency and reliability across the full taxonomy of tasks. We first introduced annotators to the six core categories (Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry) and their thirty subcategories. For each subcategory, we provided curated examples that illustrated the corresponding spatial relations (e.g., directional relations, occlusion inference, stability prediction) along with counter-examples to highlight common pitfalls. Beyond the taxonomy, we trained annotators to account for six meta-dimensions of image variability: lighting, texture complexity, edge density, dominant relation, material type, and gravity constraints, so that questions explicitly leveraged visual diversity. We demonstrated how these factors interact in real-world settings, such as reasoning about occlusion under low-contrast lighting or maintaining scale consistency under perspective distortion.

We also introduced the two evaluation modes: multiple-choice and open-ended. In multiple-choice settings, annotators learned to design precise distractors to test fine-grained discrimination, while in open-ended settings they practiced formulating prompts that encourage compositional reasoning. To ground this training, we ran calibration sessions in which annotators generated both types of questions for sample images and iteratively refined their designs with feedback. To situate task design within the broader research landscape, we reviewed prior benchmarks (e.g., SpatialBot-Bench, EmbSpatial, SpatialMM) with annotators and analyzed their limitations. These resources often emphasize simplified relations, synthetic or staged scenes, and narrow reasoning types, while neglecting natural complexity such as shadows, reflections, or gravity-sensitive stability. This gap analysis underscored the need for richer, more ambiguous, and dynamically layered spatial contexts.

Finally, we provided detailed demonstrations of the annotation protocol. These demonstrations specified how to select images with sufficient spatial complexity, how to map questions unambiguously to subcategories, and how to avoid trivial formulations. We instructed annotators to prioritize tasks requiring multi-step reasoning, such as depth layering, relational chains, or cross-view scale inference, over binary labels. Through this systematic training and iterative feedback, we ensured that the final dataset balanced taxonomic coverage with real-world relevance.

More details on Experiments and Evaluation

Multiple Choice Evaluation

The prompts used for the answer model are shown in Figure 6. We set temperature to 0 and max_tokens to 10, while keeping all other generation parameters at their default values.

Figure 6. Prompts used for evaluation.

Open-Ended Evaluation

The prompts used for both the answer model and the judge model are provided in Figure 6. We set temperature to 0.7 and max_tokens to 1200, with all other parameters kept at their default values.

Evaluation of LLM-as-a-Judge for Open-ended Evaluation

To evaluate Gemini-2.5-Flash as an automatic judge, we selected a balanced set of 240 samples covering all categories. Each sample included the evaluation setup (question, ground truth, and model output), which was scored independently by three human annotators. Annotators assigned binary scores: 1 for correct and 0 for incorrect. We compared the LLM’s judgments against the human annotations using multiple complementary agreement metrics, computed both overall and per category. Specifically, we employed Cohen’s Kappa, Accuracy with Majority Vote, and Fleiss’ Kappa, defined as follows.

Cohen’s Kappa. Cohen’s kappa (κ\kappa) measures pairwise agreement between two raters while correcting for chance agreement. For binary task, it is defined as κ=po−pe1−pe\kappa= \frac{p_o-p_e}{1-p_e}, where pop_o is the observed agreement and pep_e is the expected agreement by chance: pe=∑c∈{0,1}pc(1)pc(2)p_e = \sum_{c\in\{0,1\}}p_c^{(1)}p_c^{(2)}. Here, pc(1)p_c^{(1)} and pc(2)p_c^{(2)} are the marginal probabilities that each rater assigns class cc. An κ=1\kappa= 1: perfect agreement; κ=0\kappa= 0: agreement equals chance; κ<0\kappa< 0: systematic disagreement) We computed Cohen’s kappa between the LLM and each annotator individually (κHuman1\kappa_{\mathrm{Human1}}, κHuman2\kappa_{\mathrm{Human2}}, κHuman3\kappa_{\mathrm{Human3}}), as well as between the LLM and the majority vote of the annotators (κmajority\kappa_{\mathrm{majority}}).

Accuracy with Majority Vote. We also measured raw agreement between the LLM and the annotator majority. Accuracy is defined as Accuracy=1N∑i=1N1(yiLLM=yimaj)\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\left(y_i^{\mathrm{LLM}}=y_i^{\mathrm{maj}}\right), where yiLLMy_i^{\mathrm{LLM}} is the LLM’s decision for sample ii, and yimajy_i^{\mathrm{maj}} is the majority vote among annotators. Unlike Cohen’s kappa, accuracy does not correct for chance agreement; it simply reflects the proportion of identical labels.

Fleiss’ Kappa. To assess inter-annotator reliability, we computed Fleiss’ kappa (κF\kappa_F), which generalizes Cohen’s kappa to multiple raters. Let nijn_{ij} denote the number of annotators assigning category jj to item ii, and define Pi=1n(n−1)∑jnij(nij−1)P_i = \frac{1}{n(n-1)}\sum_j n_{ij}(n_{ij}-1), where nn is the number of annotators. The observed agreement is Pˉ=1N∑i=1NPi\bar{P} = \frac{1}{N}\sum_{i=1}^{N} P_i, and the expected agreement is Pˉe=∑jpj2\bar{P}_e = \sum_j p_j^2, with pj=1N∑inijp_j = \frac{1}{N}\sum_i n_{ij}. Fleiss’ kappa is then computed as κF=Pˉ−Pˉe1−Pˉe\kappa_F = \frac{\bar{P}-\bar{P}_e}{1-\bar{P}_e}. Higher values indicate stronger inter-annotator reliability beyond chance.

Interpretation Table 5 shows that the LLM judge (Gemini-2.5-Flash) demonstrates strong overall agreement with human annotators, achieving a Cohen’s kappa of 0.738 with the majority vote and 0.681–0.795 against individual annotators. Raw accuracy against the majority is 0.880 overall, while Fleiss’ kappa among human annotators is 0.774, indicating substantial consistency. Certain categories, such as 3D Geometry, reach perfect agreement with κ=1.000\kappa= 1.000 and accuracy of 1.000. Depth & Occlusion also shows high agreement (κmajority=0.805\kappa_{\mathrm{majority}} = 0.805, accuracy = 0.909). Categories like Orientation (κ=0.500\kappa= 0.500, accuracy = 0.810) and Size & Scale (κ=0.417\kappa= 0.417, accuracy = 0.714) show moderate agreement but remain within acceptable reliability ranges.

These results indicate that Gemini-2.5-Flash performs comparably to human evaluators in scoring correctness, with Cohen’s kappa values mostly in the substantial range (0.61–0.80) and some categories reaching almost perfect agreement (higher than 0.81). Fleiss’ kappa of 0.774 confirms that human annotators themselves are highly consistent, validating the reliability of the reference scores. The LLM achieves perfect or near-perfect agreement in clearly defined categories such as 3D Geometry (κ=1.000\kappa= 1.000, accuracy = 1.000) and Spatial Navigation (κ=0.815\kappa= 0.815, accuracy = 0.933), demonstrating accurate pattern recognition. Moderate agreement in categories like Orientation and Size & Scale reflects task complexity and inherent subjectivity rather than model shortcomings. Overall, it shows that LLM-as-judge shows robust reliability and can serve as a practical automated scoring tool, closely mirroring human evaluation.

Error Analysis of SpatiaLab-MCQ

Sub-Category-wise Quantitative Error Analysis

Closed-source Models. Closed-source models (Table 6) produce some of the strongest results across the benchmark. The best case is GPT-4o-mini, which reaches 85.71% in Stacking Orientation, the highest score overall. Gemini-2.0-Flash performs well in occlusion and projection, achieving 64.0% in Partial Occlusion and 64.71% in Shadow-Size Projection. Similarly, GPT-5-mini scores 68.0% in Tool Handedness and 55.71% in Layering Order. Weaknesses remain, as Claude 3.5 Haiku only reaches 32.0% in Relative Size Comparison. On average, closed-source systems outperform open-source ones by 15–25 percentage points across several categories. However, within-model variation remains large, often exceeding 50 percentage points. We therefore conclude that while closed-source training pipelines deliver higher ceilings, they do not solve the challenge of uniform spatial competence.

CategorySub-CategoryGPT-4o-miniGPT-5-miniGemini-2.0-FlashGemini-2.5-FlashClaude 3.5 HaikuMistral Medium 3.1
3D GeometryGravity Effects52.050.046.052.052.046.0
Shape Projection54.048.050.050.046.060.0
Spatial Containment36.046.044.038.034.038.0
Stability Prediction52.3852.3859.5250.045.2452.38
Volume Comparison41.347.8339.1334.7834.7836.96
Depth and OcclusionComplete Occlusion Inference61.5469.2361.5461.5461.5446.15
Layering Order32.8655.7150.042.8632.8645.71
Partial Occlusion42.054.064.050.044.052.0
Reflective Surfaces34.054.046.048.044.044.0
Transparency Effects32.044.060.044.036.062.0
OrientationCardinal Direction26.048.044.030.044.038.0
Facing Direction45.2457.1457.1447.6235.7145.24
Object Rotation40.060.050.046.038.044.0
Stacking Orientation85.7177.1462.8665.7168.5762.86
Tool Handedness52.068.064.064.056.056.0
Relative PositioningAlignment Patterns44.1958.1455.8153.4946.5158.14
Betweenness Relationships51.0663.8370.2163.8344.6861.7
Corner/Angle Positioning47.2263.8950.058.3350.077.78
Directional Relations48.8479.0767.4469.7753.4969.77
Proximity Gradients44.1948.8444.1934.8837.2144.19
Size and ScaleDistance-Size Correlation56.056.056.042.034.042.0
Perspective Distortion50.040.046.036.038.034.0
Relative Size Comparison42.044.058.046.032.038.0
Scale Consistency47.0645.147.0639.2233.3341.18
Shadow-Size Projection52.9439.2264.7149.0241.1852.94
Spatial NavigationAccessibility Constraints56.160.9865.8565.8548.7856.1
Obstacle Avoidance38.048.034.034.046.040.0
Pathway Existence54.3541.343.4845.6539.1336.96
Spatial Sequence66.078.048.062.056.044.0
Viewpoint Visibility36.054.048.050.040.034.0

Table 6. Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub-Categories for Closed-source Models. We bold and underline the best score within each model category.

Open-source Large Models. When scaling to large open-source models (Table 7), we see clear improvements but also severe fragilities. The InternVL3.5-72B achieves 80.56% in Corner/Angle Positioning, one of the highest open-source results. Both InternVL3.5-72B and llama-3.2-90b-vision-instruct reach 66.67% in Complete Occlusion Inference. Yet weaknesses persist, such as the llama-3.2-11b-vision-instruct scoring only 2.0% in Object Rotation. Gains are also evident in Stacking Orientation, where llama-3.2-90b-vision-instruct scores 71.43%, but this comes with losses in other categories. The within-model spread often exceeds 70 percentage points, highlighting brittle specialization. Gravity reasoning remains modest at 56.0%. We find that scaling yields higher peaks but does not resolve structural weaknesses. Our conclusion is that size amplifies selective strengths while leaving foundational representation gaps intact.

CategorySub-Categoryllama-3.2-11b-vision-instructGemma-3-27B-itqwen-2.5-vl-32b-instructInternVL3-5-72Bqwen-2.5-vl-72b-instructllama-3.2-90b-vision-instruct
3D GeometryGravity Effects22.048.054.052.052.056.0
Shape Projection32.042.044.050.046.050.0
Spatial Containment32.034.036.044.046.036.0
Stability Prediction33.3354.7638.152.3847.6252.38
Volume Comparison13.0439.1332.6152.1743.4836.96
Depth and OcclusionComplete Occlusion Inference30.7761.5458.9766.6748.7266.67
Layering Order34.2934.2937.1450.050.038.57
Partial Occlusion32.034.044.060.048.058.0
Reflective Surfaces28.038.026.054.050.044.0
Transparency Effects26.040.040.060.046.062.0
OrientationCardinal Direction22.048.034.046.048.040.0
Facing Direction21.4342.8652.3854.7650.050.0
Object Rotation2.040.038.048.048.044.0
Stacking Orientation28.5765.7165.7168.5762.8671.43
Tool Handedness40.048.052.056.056.056.0
Relative PositioningAlignment Patterns27.9155.8148.8453.4955.8153.49
Betweenness Relationships42.5555.3255.3265.9661.761.7
Corner/Angle Positioning58.3355.5636.1180.5650.075.0
Directional Relations60.4762.7951.1683.7262.7965.12
Proximity Gradients27.9141.8632.5648.8439.5341.86
Size and ScaleDistance-Size Correlation36.050.054.052.044.046.0
Perspective Distortion22.042.036.046.034.052.0
Relative Size Comparison28.048.040.048.052.046.0
Scale Consistency29.4147.0641.1843.1437.2537.25
Shadow-Size Projection37.2552.9454.956.8650.9852.94
Spatial NavigationAccessibility Constraints29.2758.5443.975.6153.6643.9
Obstacle Avoidance24.038.038.046.042.040.0
Pathway Existence28.2654.3536.9639.1354.3547.83
Spatial Sequence34.052.056.066.054.062.0
Viewpoint Visibility44.036.034.050.042.048.0

Table 7. Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub-Categories for Open-source Large Models. We bold and underline the best score within each model category.

Open-source Small Models. We observe that open-source small models (Table 8) display sharp contrasts in performance across sub-categories. For example, qwen-2.5-vl-7b-instruct reaches 74.29% in Stacking Orientation, while InternVL3.5-1B collapses to 12.0% in Cardinal Direction. Scores in Pathway Existence remain modest, with the best models tied at 34.78%. Gravity-related reasoning is similarly weak, with only 50.0% achieved by InternVL3.5-4B and Gemma-3-4B-it. The within-model spread often exceeds 60 percentage points, showing brittle generalization. These patterns suggest that smaller models are highly cue-dependent, performing well where visual templates are strong but failing in tasks requiring deeper spatial abstractions. We interpret this inconsistency as evidence that representation learning is uneven across sub-categories. Overall, we find that small models provide localized competence but lack robustness across the full spectrum of spatial reasoning.

CategorySub-CategoryInternVL3.5-1BInternVL3.5-2Bqwen-2.5-vl-3b-instructInternVL3.5-4BGemma-3-4B-itqwen-2.5-vl-7b-instruct
3D GeometryGravity Effects46.038.048.050.050.050.0
3D GeometryShape Projection30.042.038.042.040.044.0
3D GeometrySpatial Containment30.028.034.034.036.034.0
3D GeometryStability Prediction30.9535.7150.052.3845.2452.38
3D GeometryVolume Comparison30.4326.0936.9636.9647.8334.78
Depth and OcclusionComplete Occlusion Inference30.7730.7741.0364.133.3341.03
Depth and OcclusionLayering Order30.024.2930.032.8622.8627.14
Depth and OcclusionPartial Occlusion38.030.038.054.046.042.0
Depth and OcclusionReflective Surfaces38.034.036.032.042.040.0
Depth and OcclusionTransparency Effects26.042.036.040.032.044.0
OrientationCardinal Direction12.018.036.024.038.030.0
OrientationFacing Direction21.4328.5752.3840.4842.8638.1
OrientationObject Rotation12.028.042.034.042.034.0
OrientationStacking Orientation45.7157.1462.8668.5768.5774.29
OrientationTool Handedness40.036.040.060.048.048.0
Relative PositioningAlignment Patterns27.9130.2339.5344.1941.8641.86
Relative PositioningBetweenness Relationships38.338.346.8165.9646.8153.19
Relative PositioningCorner/Angle Positioning50.055.5647.2269.4455.5650.0
Relative PositioningDirectional Relations51.1658.1427.9160.4746.5148.84
Relative PositioningProximity Gradients20.9323.2639.5334.8839.5337.21
Size and ScaleDistance-Size Correlation32.020.040.036.038.036.0
Size and ScalePerspective Distortion26.040.042.038.044.044.0
Size and ScaleRelative Size Comparison34.032.046.038.030.042.0
Size and ScaleScale Consistency27.4535.2941.1831.3727.4541.18
Size and ScaleShadow-Size Projection39.2235.2966.6739.2247.0647.06
Spatial NavigationAccessibility Constraints36.5929.2751.2241.4639.0243.9
Spatial NavigationObstacle Avoidance24.026.038.038.038.036.0
Spatial NavigationPathway Existence34.7830.4332.6134.7834.7828.26
Spatial NavigationSpatial Sequence24.042.040.056.042.044.0
Spatial NavigationViewpoint Visibility36.034.036.040.036.026.0

Table 8. Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub-Categories for Open-source Small Models. We bold and underline the best score within each model category.

Reasoning Models. Reasoning-enhanced models (Table 9) show notable benefits in select tasks but remain inconsistent overall. o4-mini reaches 65.12% in Alignment Patterns and 56.0% in Gravity Effects, outperforming most open baselines. Gemini-2.0-Flash-Thinking achieves 62.75% in Shadow-Size Projection and 52.17% in Volume Comparison, demonstrating stronger geometric reasoning. The strongest results come from Gemini-2.5-Flash-Thinking, which scores 71.79% in Complete Occlusion Inference and 70.21% in Betweenness Relationships. Yet the same model fails in others, such as 34.0% in Cardinal Direction and 34.0% in Shape Projection. This variance shows that reasoning mechanisms improve depth in certain sub-categories but do not broaden coverage. Within-model spreads frequently exceed 35 percentage points. We conclude that reasoning augmentation adds localized capacity but leaves structural gaps in spatial understanding. Overall, these models demonstrate promise but not generalizable competence.

CategorySub-Categoryo4-miniGemini-2.0-Flash-ThinkingGemini-2.5-Flash-ThinkingKimi-VL-A3B-Thinking-2506
3D GeometryGravity Effects56.038.046.050.0
Shape Projection50.036.040.034.0
Spatial Containment56.028.048.042.0
Stability Prediction50.035.7150.050.0
Volume Comparison43.4852.1745.6539.13
Depth and OcclusionComplete Occlusion Inference69.2351.2871.7951.28
Layering Order57.1430.041.4335.71
Partial Occlusion50.042.058.040.0
Reflective Surfaces54.040.052.042.0
Transparency Effects64.050.054.042.0
OrientationCardinal Direction46.034.050.028.0
Facing Direction50.042.8654.7645.24
Object Rotation56.036.048.030.0
Stacking Orientation71.4357.1457.1465.71
Tool Handedness56.044.060.044.0
Relative PositioningAlignment Patterns65.1241.8644.1934.88
Betweenness Relationships65.9655.3270.2163.83
Corner/Angle Positioning75.055.5661.1147.22
Directional Relations65.1244.1965.1260.47
Proximity Gradients51.1632.5641.8648.84
Size and ScaleDistance-Size Correlation44.042.056.044.0
Perspective Distortion40.044.050.036.0
Relative Size Comparison46.046.058.032.0
Scale Consistency39.2256.8652.9443.14
Shadow-Size Projection35.2962.7558.8243.14
Spatial NavigationAccessibility Constraints56.141.4653.6631.71
Obstacle Avoidance32.044.044.036.0
Pathway Existence45.6541.347.8334.78
Spatial Sequence64.048.068.054.0
Viewpoint Visibility60.040.054.048.0

Table 9. Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub-Categories for Reasoning Models. We bold and underline the best score within each model category.

Spatial Reasoning Models. Spatially specialized models (Table 10) reveal targeted improvements but lack generality. SpaceQwen2.5-VL-3B-Instruct achieves 48.0% in Distance-Size Correlation and 49.02% in Scale Consistency, clearly surpassing many open-source baselines in size–scale reasoning. The same model leads in navigation tasks, with 53.66% in Accessibility Constraints and 44.0% in Obstacle Avoidance. SpaceOm and SpaceThinker-Qwen2.5VL-3B reach 50.0% in Stability Prediction, matching expectations for domain-specific reasoning. However, performance drops sharply in other tasks, such as only 24.0% for Spatial Containment. Rarely do these models exceed 55.0%, even in their strongest areas. The spread across sub-categories frequently surpasses 30 percentage points. We find that specialization drives narrow peaks but does not produce holistic competence. Our interpretation is that domain targeting enhances select abilities but fails to integrate them into a robust, general framework.

CategorySub-CategorySpaceOmSpaceThinker-Qwen2.5VL-3BSpaceQwen2.5-VL-3B-Instruct
3D GeometryGravity Effects48.046.032.0
Shape Projection38.036.036.0
Spatial Containment40.036.024.0
Stability Prediction50.050.028.57
Volume Comparison36.9634.7836.96
Depth and OcclusionComplete Occlusion Inference38.4635.938.46
Layering Order30.032.8624.29
Partial Occlusion44.046.038.0
Reflective Surfaces44.044.038.0
Transparency Effects40.032.042.0
OrientationCardinal Direction40.038.030.0
Facing Direction59.5254.7650.0
Object Rotation40.042.028.0
Stacking Orientation62.8662.8640.0
Tool Handedness40.040.048.0
Relative PositioningAlignment Patterns39.5344.1939.53
Betweenness Relationships44.6842.5553.19
Corner/Angle Positioning41.6741.6738.89
Directional Relations25.5825.5820.93
Proximity Gradients37.2137.2134.88
Size and ScaleDistance-Size Correlation40.044.048.0
Perspective Distortion40.038.042.0
Relative Size Comparison42.046.048.0
Scale Consistency33.3333.3349.02
Shadow-Size Projection58.8254.966.67
Spatial NavigationAccessibility Constraints48.7846.3453.66
Obstacle Avoidance36.032.044.0
Pathway Existence32.6134.7841.3
Spatial Sequence48.044.050.0
Viewpoint Visibility32.034.048.0
CategorySub-Category012345Total
3D GeometryGravity Effects97323125
Shape Projection46095731
Spatial Containment69636131
Stability Prediction56422322
Volume Comparison65654228
3D Geometry Total303319212014137
Depth & OcclusionComplete Occlusion Inference04063417
Layering Order98786543
Partial Occlusion22575223
Reflective Surfaces65574128
Transparency Effects46325727
Depth & Occlusion Total212520302319138
OrientationCardinal Direction171073432
Facing Direction23853122
Object Rotation46735429
Stacking Orientation2302119
Tool Handedness21431112
Orientation Total112029201311104
Relative PositioningAlignment Patterns34444120
Betweenness Relationships31225720
Corner/Angle Positioning01323312
Directional Relations14123213
Proximity Gradients46652326
Relative Positioning Total11161615171691
Size & ScaleDistance-Size Correlation65611726
Perspective Distortion87336431
Relative Size Comparison55744429
Scale Consistency76745231
Shadow-Size Projection57211420
Size & Scale Total313025131721137
Spatial NavigationAccessibility Constraints32154318
Obstacle Avoidance269610235
Pathway Existence210414324
Spatial Sequence32327825
Viewpoint Visibility22598531
Spatial Navigation Total122222233321133
Grand Total116146131122123102740

Table 10. Multiple-choice Evaluation Accuracy on SPATIALAB-MCQ by Question Sub-Categories for Spatial Reasoning Models. We bold and underline the best score within each model category.

Overall Discussion. Across all categories (Table 8, Table 7, Table 6, Table 10, Table 9), we observe consistent fragmentation. Small open-source models peak at 74.29% in Stacking Orientation but collapse to 12.0% in Cardinal Direction. Larger models raise ceilings, such as 80.56% in Corner/Angle Positioning, but still fail catastrophically with 2.0% in Object Rotation. Closed-source systems achieve the strongest scores, with GPT-4o-mini reaching 85.71% in Stacking Orientation, yet still fall to 32.0% in Relative Size Comparison. Specialized models show strengths in navigation (53.66%) and stability (50.0%) but remain capped below 55% overall. Reasoning models demonstrate gains, such as 71.79% in Complete Occlusion Inference, but others remain at 34.0%. Within-model spreads often exceed 50 percentage points, showing brittle internal consistency. Sub-categories

requiring physical abstraction, Gravity Effects, Stability Prediction, Spatial Containment, consistently score below 35–40% regardless of category. We attribute these failures to two root causes: the absence of stable encodings of orientation and physics, and an over-reliance on surface-level correlations. The wide variance, from 85.71% highs to 2.0% lows, suggests models are pattern-matching rather than reasoning robustly about space. Scaling, proprietary training, and reasoning augmentations each provide benefits but none produce general coverage. Instead, each approach yields isolated peaks that coexist with glaring blind spots. The collective picture is one of fragmented competence rather than integrated ability. We conclude that progress in spatial reasoning will require architectural changes aimed at embedding consistent geometric grounding and physics-aware mechanisms. In sum, our results show both remarkable selective gains and the persistence of fundamental gaps in holistic spatial intelligence.

Model Failure Analysis

The distribution across categories and sub-categories in tbl:11 reveals several notable patterns in model performance. Categories such as 3D Geometry and Size & Scale show relatively balanced distributions, yet both retain a significant proportion of failures where no model answered correctly, suggesting that even basic geometric invariances are not reliably internalized. In contrast, Depth & Occlusion exhibits a heavier tail in the mid-range (2–4 correct), pointing to partial but inconsistent reasoning about layered scenes. Orientation emerges as a weak category overall, with especially low totals in Stacking Orientation and Tool Handedness, which indicates that reasoning about subtle directional or embodied affordances is poorly captured. Relative Positioning presents a mixed picture: while some tasks like Proximity Gradients show moderate success, others such as Corner/Angle Positioning have many near-zero correct counts, reflecting brittleness in compositional reasoning over reference frames. Interestingly, Spatial Navigation shows comparatively stronger performance, particularly in Obstacle Avoidance and Viewpoint Visibility, but still has significant errors in Pathway Existence where relational chaining is needed. When aggregated, no category achieves uniformly high accuracy, and the Grand Total shows an even spread across 0–5 bins, suggesting that success is highly task-dependent rather than reflecting a general spatial competence. Overall, models display isolated strengths but lack systematic reliability across spatial dimensions.

These results suggest that current vision–language models struggle with several root causes tied to representational and reasoning bottlenecks. High error rates in Stacking Orientation and Tool Handedness indicate limited grounding in physics-based and embodied reasoning, consistent with models trained primarily on static internet imagery rather than interactive or task-driven data. The difficulty in Corner/Angle Positioning and Pathway Existence reflects inadequate multi-step relational chaining, pointing to weaknesses in their ability to handle recursive spatial logic. The inconsistency in Transparency Effects and Reflective Surfaces underscores the challenge of handling visual phenomena that require non-local cues, such as secondary reflections or occluded geometry. By contrast, comparatively better performance in Obstacle Avoidance suggests that models may have learned shortcuts from dataset biases, such as frequent co-occurrence patterns of walkable paths, rather than genuine planning ability. Taken together, these failures highlight that model training regimes overemphasize correlation-driven visual-language alignments while underrepresenting embodied, counterfactual, and physics-based reasoning. Addressing these issues likely requires richer training distributions (e.g., embodied simulation data, physics-augmented learning) and architectural innovations that can explicitly model reference frames and relational constraints. Without such advances, models will continue to show brittle competence that breaks down under compositional or physically grounded tasks.

Error Analysis of SpatiaLab-Open

Sub-Category-wise Quantitative Error Analysis

Closed-source Models (Table 12). We observe that closed-source models achieve the strongest overall performance in the open-ended evaluation, yet their results remain uneven across sub-categories. GPT-5-mini is the clear leader, consistently securing the best scores in 3D Geometry and Relative Positioning, with peaks of 58.14% in Directional Relations and 55.81% in Alignment Patterns. In contrast, Gemini-2.5-Flash and Claude 3.5 Haiku register very weak results in Proximity Gradients (16.28%) and Perspective Distortion (12.0%), respectively. This wide gap highlights the fragility of even the strongest models when tasked with reasoning over subtle, continuous visual cues. We believe these disparities stem from the differential sophistication of models’ spatial reasoning pipelines, where some architectures capture relational and geometric structure more effectively than others. Tasks such as Proximity Gradients are particularly challenging, as they require analogical, graded reasoning rather than discrete classification. The fact that GPT-5-mini succeeds where others fail suggests that scaling and proprietary training corpora confer distinct advantages. Nevertheless, even the leader shows vulnerabilities, pointing to a systemic issue: closed-source dominance is relative, not absolute, and the models still fall short of robust, human-like spatial reasoning.

Open-source Large Models (Table 13). For large open-source models, we find a more distributed performance landscape, with different systems excelling in isolated sub-categories. llama-3.2-90b-vision-instruct achieves 60.0% in Depth and Occlusion, while GLM-4.5V-106B-MoE leads in Spatial Containment at 38.0%. Despite these highlights, several models collapse on Proximity Gradients, with qwen-2.5-vl-32b-instruct and qwen-2.5-vl-72b-instruct both scoring only 9.3%. This indicates that architectural scaling helps but does not guarantee coverage across difficult continuous-reasoning tasks. We hypothesize that much of this variation arises from heterogeneous training data: models exposed to richer depth and geometry corpora perform better in occlusion and containment tasks, while others generalize poorly. The persistent weakness on Proximity Gradients suggests that these systems are not optimized for subtle spatial analogies and distance scaling. We also observe higher inter-model variance in this group than among closed-source models, underscoring the impact of open-source development heterogeneity. Overall, large open-source models show encouraging peaks but inconsistent coverage, suggesting that scale alone is insufficient without targeted pre-training and fine-tuning.

Open-source Small Models (Table 14). Small open-source models predictably lag behind their larger counterparts, but their results reveal noteworthy localized strengths. qwen-2.5-vl-3b-instruct achieves 18.0% in Perspective Distortion and 26.0% in Relative Size Comparison, while Gemma-3-4B-it leads in Spatial Containment (24.0%) and Stability Prediction (21.43%). Yet these successes are offset by catastrophic failures: InternVL3.5-2B scores 0.0% in Betweenness Relationships, and InternVL3.5-1B manages only 2.0% in Shape Projection. We attribute this extreme variability to limited capacity and restricted training diversity, which prevent small models from learning robust representations of geometric and relational principles. Sub-categories that demand precise reasoning about orientation and hierarchy trigger complete breakdowns, as small misinterpretations translate into total failure. Nevertheless, their ability to occasionally outperform larger models in narrow tasks suggests that even small-scale training can instill isolated competencies. This points to the value of specialization but also confirms the structural ceiling of compact architectures. In short, small open-source models demonstrate pockets of promise but remain highly brittle, with overall accuracy limited by size and data constraints.

CategorySub-CategoryInternVL3.5-1BInternVL3.5-2Bqwen-2.5-vl-3b-instructInternVL3.5-4BGemma-3-4B-itqwen-2.5-vl-7b-instruct
3D GeometryGravity Effects4.06.014.024.016.022.0
3D GeometryShape Projection2.08.010.04.010.00.0
3D GeometrySpatial Containment6.04.012.020.024.08.0
3D GeometryStability Prediction2.3811.914.2919.0521.4319.05
3D GeometryVolume Comparison15.2232.6128.2630.4330.4328.26
Depth and OcclusionComplete Occlusion Inference2.565.1310.2617.9520.5123.08
Depth and OcclusionLayering Order5.717.1410.014.2911.4314.29
Depth and OcclusionPartial Occlusion14.08.012.016.04.010.0
Depth and OcclusionReflective Surfaces16.016.06.014.018.026.0
Depth and OcclusionTransparency Effects10.020.04.028.014.08.0
OrientationCardinal Direction12.016.016.026.020.024.0
OrientationFacing Direction9.524.7628.5714.297.1419.05
OrientationObject Rotation4.010.012.08.014.016.0
OrientationStacking Orientation17.1411.438.5717.1420.028.57
OrientationTool Handedness8.012.08.012.012.012.0
Relative PositioningAlignment Patterns13.9527.9132.5611.6341.8634.88
Relative PositioningBetweenness Relationships12.7717.020.019.1517.0225.53
Relative PositioningCorner/Angle Positioning22.2230.562.7822.2219.4422.22
Relative PositioningDirectional Relations11.6325.589.334.8823.2637.21
Relative PositioningProximity Gradients9.318.69.311.6316.2818.6
Size and ScaleDistance-Size Correlation6.010.014.016.020.012.0
Size and ScalePerspective Distortion8.012.018.016.012.016.0
Size and ScaleRelative Size Comparison10.016.026.024.010.016.0
Size and ScaleScale Consistency7.849.811.7611.767.8411.76
Size and ScaleShadow-Size Projection13.7311.7621.5713.7325.4923.53
Spatial NavigationAccessibility Constraints4.8826.837.3217.0719.5119.51
Spatial NavigationObstacle Avoidance16.016.012.018.028.026.0
Spatial NavigationPathway Existence0.08.72.178.717.398.7
Spatial NavigationSpatial Sequence12.022.08.026.020.028.0
Spatial NavigationViewpoint Visibility16.018.016.024.014.016.0

Table 14. Open-ended Evaluation Accuracy on SPATIALAB-OPEN by Question Sub-Categories for Open-source Small Models. We bold and underline the best score within each model category.

Reasoning Models (Table 15). Reasoning-oriented models present a sharper internal divide, with Gemini-2.5-flash-thinking standing far above its peers. It records the best scores in nearly every sub-category, including Tool Handedness at 75.0% and Complete Occlusion Inference at 57.14%. By contrast, Kimi-VL-A3B-Thinking-2506 struggles dramatically, scoring just 6.0% in Spatial Containment and lagging behind across the board. We believe this gap arises from differences in how these models integrate multi-modal streams into coherent reasoning frameworks. Gemini-2.5-flash-thinking appears capable of forming and querying internal world models, enabling it to infer object interactions and physical properties more effectively. In contrast, weaker models rely heavily on surface-level pattern matching, which proves inadequate for compositional or causal reasoning. The consistently high performance of Gemini-2.5 across categories suggests that explicit reasoning mechanisms offer tangible benefits for spatial understanding. However, its failure to dominate Shape Projection and Betweenness Relationships shows that even advanced reasoning modules are not yet universally effective. Thus, while reasoning models hold promise, the category itself remains heterogeneous and far from solved.

CategorySub-Categorygemini-2.0-flash-thinkinggemini-2.5-flash-thinkingKimi-VL-A3B-Thinking-2506Step3
3D GeometryGravity Effects32.040.014.038.0
3D GeometryShape Projection24.00.010.030.0
3D GeometrySpatial Containment36.050.06.030.0
3D GeometryStability Prediction26.1942.8614.2933.33
3D GeometryVolume Comparison36.9644.4423.9130.43
Depth and OcclusionComplete Occlusion Inference33.3357.1417.9533.33
Depth and OcclusionLayering Order20.020.014.2914.29
Depth and OcclusionPartial Occlusion28.050.08.014.0
Depth and OcclusionReflective Surfaces34.063.6410.016.0
Depth and OcclusionTransparency Effects26.057.1422.020.0
OrientationCardinal Direction40.020.018.026.0
OrientationFacing Direction35.7137.514.2928.57
OrientationObject Rotation18.040.04.022.0
OrientationStacking Orientation22.860.05.7134.29
OrientationTool Handedness44.075.020.032.0
Relative PositioningAlignment Patterns46.5166.6725.5841.86
Relative PositioningBetweenness Relationships27.6620.012.7725.53
Relative PositioningCorner/Angle Positioning36.1150.011.1111.11
Relative PositioningDirectional Relations39.5344.4427.9137.21
Relative PositioningProximity Gradients23.2616.6711.6318.6
Size and ScaleDistance-Size Correlation32.057.1414.022.0
Size and ScalePerspective Distortion18.011.1110.030.0
Size and ScaleRelative Size Comparison28.011.1114.036.0
Size and ScaleScale Consistency33.3320.011.7631.37
Size and ScaleShadow-Size Projection35.2918.1813.7329.41
Spatial NavigationAccessibility Constraints31.7145.4517.0741.46
Spatial NavigationObstacle Avoidance26.010.016.018.0
Spatial NavigationPathway Existence23.9125.013.0419.57
Spatial NavigationSpatial Sequence34.012.530.036.0
Spatial NavigationViewpoint Visibility32.012.516.026.0

Table 15. Open-ended Evaluation Accuracy on SPATIALAB-OPEN by Question Sub-Categories for Reasoning Models. We bold and underline the best score within each model category.

Spatial Reasoning Models (Table 16). Spatial reasoning models, though explicitly designed for the task, record the lowest results overall. Performance is highly fragmented: several models sit at 16.0% in Gravity Effects, 8.0% in Shape Projection, and 14.29% in Stability Prediction. SpaceQwen2.5-VL-3B-Instruct stands out slightly with 23.91% in Volume Comparison and 13.73% in Scale Consistency, but these scores remain far below competitive thresholds. At the other extreme, Betweenness Relationships and Corner/Angle Positioning collapse entirely, with models recording 0.0% and 2.78%, respectively. These results suggest that the models’ smaller capacity and narrowly focused training severely constrain their representational flexibility. We interpret the failures as evidence that narrow specialization without sufficient capacity or data breadth cannot yield generalizable spatial reasoning. Particularly on geometric and positional tasks, even slight representational errors lead to total breakdowns. Consequently, despite their focus, spatial reasoning models are least effective, underscoring the necessity of both scale and diversity in achieving robust performance.

CategorySub-CategorySpaceOmSpaceThinker-Qwen2.5VL-3BSpaceQwen2.5-VL-3B-Instruct
3D GeometryGravity Effects14.012.016.0
3D GeometryShape Projection8.08.06.0
3D GeometrySpatial Containment8.010.010.0
3D GeometryStability Prediction14.2914.297.14
3D GeometryVolume Comparison19.5723.9123.91
Depth and OcclusionComplete Occlusion Inference7.6910.267.69
Depth and OcclusionLayering Order8.5711.434.29
Depth and OcclusionPartial Occlusion10.012.04.0
Depth and OcclusionReflective Surfaces4.04.02.0
Depth and OcclusionTransparency Effects4.08.02.0
OrientationCardinal Direction18.024.08.0
OrientationFacing Direction23.8123.8116.67
OrientationObject Rotation20.020.018.0
OrientationStacking Orientation8.578.5717.14
OrientationTool Handedness0.04.08.0
Relative PositioningAlignment Patterns32.5632.5623.26
Relative PositioningBetweenness Relationships4.262.130.0
Relative PositioningCorner/Angle Positioning2.782.782.78
Relative PositioningDirectional Relations11.636.9811.63
Relative PositioningProximity Gradients6.986.989.3
Size and ScaleDistance-Size Correlation18.018.016.0
Size and ScalePerspective Distortion16.020.02.0
Size and ScaleRelative Size Comparison22.020.08.0
Size and ScaleScale Consistency9.811.7613.73
Size and ScaleShadow-Size Projection27.4527.4519.61
Spatial NavigationAccessibility Constraints7.327.329.76
Spatial NavigationObstacle Avoidance10.08.014.0
Spatial NavigationPathway Existence6.526.528.7
Spatial NavigationSpatial Sequence16.012.016.0
Spatial NavigationViewpoint Visibility20.016.08.0

Table 16. Open-ended Evaluation Accuracy on SPATIALLAB-OPEN by Question Sub-Categories for Spatial Reasoning Models. We bold and underline the best score within each model category.

Overall Discussion

Across all categories, closed-source (Table 12), open-source large (Table 13), open-source small (Table 14), reasoning (Table 15), and spatial reasoning (Table 16), we find a consistent pattern of fragmented competence and systemic weaknesses. Closed-source models perform best overall, with GPT-5-mini achieving highs of 58.14% in Directional Relations and 55.81% in Alignment Patterns, yet even they collapse on Proximity Gradients. Large open-source models show impressive peaks, such as 60.0% in Depth and Occlusion, but still fail at just 9.3% in Proximity Gradients, illustrating that scale without specialization remains insufficient. Small open-source models confirm this further, with rare localized strengths (e.g., 26.0% in Relative Size Comparison) but catastrophic lows like 0.0% in Betweenness Relationships. Reasoning models demonstrate that explicit reasoning mechanisms can significantly improve results, with Gemini-2.5-flash-thinking reaching 75.0% in Tool Handedness, yet peers like Kimi-VL-A3B fall to single digits, exposing the fragility of less-integrated reasoning frameworks. Spatial reasoning models, despite their specialization, underperform the most, with scores often below 15% and total failures on Betweenness Relationships. Taken together, these results highlight three root causes: (1) a lack of robust representations for continuous and analogical spatial cues, (2) structural limitations tied to scale and training diversity, and (3) brittle reasoning pipelines that prevent models from generalizing across tasks. We conclude that no current approach, whether proprietary scaling, open-source expansion, reasoning integration, or specialization, delivers comprehensive coverage. The frequent performance gaps exceeding 50 percentage points within a single model further confirm that progress is piecemeal. Future work must thus focus on unifying robust geometric encodings, physics-informed reasoning, and cross-task transfer, moving beyond isolated gains to achieve consistent spatial intelligence.

CategorySub-CategoryGPT-5-miniGemini-2.0-FlashGemini-2.5-FlashClaude 3.5 HaikuMistral Medium 3.1
3D GeometryGravity Effects48.036.030.024.020.0
3D GeometryShape Projection40.018.022.016.016.0
3D GeometrySpatial Containment50.038.050.036.036.0
3D GeometryStability Prediction50.026.1935.7130.9526.19
3D GeometryVolume Comparison39.1341.332.6123.9128.26
Depth and OcclusionComplete Occlusion Inference46.1535.938.4628.2125.64
Depth and OcclusionLayering Order30.022.8621.4317.1415.71
Depth and OcclusionPartial Occlusion38.020.028.014.014.0
Depth and OcclusionReflective Surfaces32.028.032.022.022.0
Depth and OcclusionTransparency Effects32.018.018.016.022.0
OrientationCardinal Direction34.040.034.026.030.0
OrientationFacing Direction28.5728.5726.1919.0514.29
OrientationObject Rotation42.014.028.030.014.0
OrientationStacking Orientation37.1420.031.4322.8628.57
OrientationTool Handedness48.036.044.024.024.0
Relative PositioningAlignment Patterns55.8134.8837.2137.2134.88
Relative PositioningBetweenness Relationships48.9423.431.9123.421.28
Relative PositioningCorner/Angle Positioning44.4430.5641.6716.6736.11
Relative PositioningDirectional Relations58.1446.5158.1434.8837.21
Relative PositioningProximity Gradients39.5320.9325.5816.2818.6
Size and ScaleDistance-Size Correlation48.032.036.020.022.0
Size and ScalePerspective Distortion48.024.022.020.012.0
Size and ScaleRelative Size Comparison36.026.028.018.016.0
Size and ScaleScale Consistency43.1421.5723.5319.619.8
Size and ScaleShadow-Size Projection37.2527.4523.5323.5315.69
Spatial NavigationAccessibility Constraints41.4631.7139.0224.3921.95
Spatial NavigationObstacle Avoidance40.022.022.026.014.0
Spatial NavigationPathway Existence23.9115.2221.7410.8719.57
Spatial NavigationSpatial Sequence52.032.034.024.020.0
Spatial NavigationViewpoint Visibility28.022.032.020.010.0

Table 12. Open-ended Evaluation Accuracy on SPATIALAB-OPEN by Question Sub-Categories for Closed-source Models. We bold and underline the best score within each model category.

CategorySub-Categoryllama-3.2-11b-vision-instructGemma-3-27B-itqwen-2.5-vl-32b-instructqwen-2.5-vl-72b-instructllama-3.2-90b-vision-instructGLM-4.5V-106B-MoE
3D GeometryGravity Effects14.022.012.030.024.028.0
3D GeometryShape Projection10.020.010.020.016.020.0
3D GeometrySpatial Containment14.030.018.022.020.038.0
3D GeometryStability Prediction23.8130.9516.6721.4326.1938.1
3D GeometryVolume Comparison23.9110.8726.0941.328.2632.61
Depth and OcclusionComplete Occlusion Inference25.6420.5112.8225.6430.7733.33
Depth and OcclusionLayering Order8.5715.7110.014.2915.7110.0
Depth and OcclusionPartial Occlusion12.016.08.014.016.014.0
Depth and OcclusionReflective Surfaces26.016.08.034.032.024.0
Depth and OcclusionTransparency Effects18.014.010.020.022.026.0
OrientationCardinal Direction32.034.020.038.028.030.0
OrientationFacing Direction11.926.1914.2921.4311.916.67
OrientationObject Rotation26.016.010.018.028.030.0
OrientationStacking Orientation17.1425.7117.1425.7157.137.14
OrientationTool Handedness20.020.012.020.032.024.0
Relative PositioningAlignment Patterns23.2646.5120.9346.5130.2324.23
Relative PositioningBetweenness Relationships34.0431.9117.0225.5338.329.79
Relative PositioningCorner/Angle Positioning27.7836.1111.1127.7827.7816.67
Relative PositioningDirectional Relations30.2339.5325.5834.8825.5832.56
Relative PositioningProximity Gradients9.318.69.318.618.620.93
Size and ScaleDistance-Size Correlation20.026.012.026.036.026.0
Size and ScalePerspective Distortion10.018.010.018.032.014.0
Size and ScaleRelative Size Comparison8.026.014.032.022.024.0
Size and ScaleScale Consistency11.7621.573.9219.6121.5723.53
Size and ScaleShadow-Size Projection17.6521.579.827.4523.5327.45
Spatial NavigationAccessibility Constraints21.9526.8326.8334.1539.0224.39
Spatial NavigationObstacle Avoidance12.018.012.020.030.022.0
Spatial NavigationPathway Existence19.5721.746.5217.3915.2217.39
Spatial NavigationSpatial Sequence22.028.012.022.028.034.0
Spatial NavigationViewpoint Visibility18.016.012.012.024.024.0

Table 13. Open-ended Evaluation Accuracy on SPATIALAB-OPEN by Question Sub-Categories for Open-source Large Models. We bold and underline the best score within each model category.

Model Failure Analysis

The distribution in Table 17 highlights persistent weaknesses across categories, with a notable skew toward the lower bins (0–2 models correct). 3D Geometry is broadly challenging, with Shape Projection (19 items with zero correct) and Stability Prediction (10 items with zero correct) showing especially poor outcomes, suggesting models lack reliable physical and geometric reasoning. Depth and Occlusion is the hardest category overall, accumulating the highest zero-correct counts (72), particularly in Layering Order (20 zero) and Transparency Effects (16 zero), both requiring nuanced handling of occlusion and non-local cues. Orientation exhibits similar struggles: while Cardinal Direction achieves some mid-range success, Stacking Orientation (11 zero) and Tool Handedness (5 zero, none with perfect success) underscore the brittleness of embodied directional reasoning. Relative Positioning reveals widespread inconsistency, with tasks such as Betweenness Relationships and Proximity Gradients rarely solved by more than two or three models, while Corner/Angle Positioning shows low but somewhat balanced performance. Size and Scale appears relatively stronger, with multiple subcategories (e.g., Scale Consistency with 6 full successes) showing that some models internalize coarse perspective and scaling cues, though failures remain high in Shadow-Size Projection (17 zero). Spatial Navigation is split: while Spatial Sequence and Obstacle Avoidance show occasional full-model success, Pathway Existence (16 zero) and Accessibility Constraints (10 zero) highlight persistent difficulty in multi-step relational chaining. Overall, the grand total reveals that across 356 items, zero-correct dominates (356), while only 74 items are solved by all five models, confirming that consistent spatial competence remains elusive.

CategorySub-Category012345
3D GeometryGravity Effects1373412
Shape Projection1956251
Spatial Containment1043623
Stability Prediction1037302
Volume Comparison1134342
3D Geometry Total632223181210
Depth & OcclusionComplete Occlusion Inference754252
Layering Order201410633
Partial Occlusion1485542
Reflective Surfaces1541525
Transparency Effects1677312
Depth & Occlusion Total723827211514
OrientationCardinal Direction8100152
Facing Direction1355132
Object Rotation12106412
Stacking Orientation1151211
Tool Handedness541420
Orientation Total49341312127
Relative PositioningAlignment Patterns753111
Betweenness Relationships1157213
Corner/Angle Positioning762132
Directional Relations544420
Proximity Gradients10103324
Relative Positioning Total40301911910
Size & ScaleDistance-Size Correlation759415
Perspective Distortion1266852
Relative Size Comparison1474224
Scale Consistency1475336
Shadow-Size Projection1744432
Size & Scale Total642928211419
Spatial NavigationAccessibility Constraints1054012
Obstacle Avoidance1558225
Pathway Existence1667311
Spatial Sequence1433335
Viewpoint Visibility1339531
Spatial Navigation Total682231131014
Grand Total356175141967274

Table 17. Distribution of question counts by the number of models (0–5) that correctly answered each item across categories and sub-categories in SPATIALAB-OPEN. “0” indicates no model answered correctly, while “5” indicates only five models succeeded to answer these among all models tested.

These findings indicate that the root causes of failure stem less from dataset design and more from the representational and reasoning limitations of current models. First, the dominance of errors in Depth and Occlusion reflects a reliance on superficial appearance features rather than structured scene representations: for example, models may lack onto surface textures rather than reasoning about hidden layers, leading to systematic errors in Layering Order. Failures in Orientation tasks such as Tool Handedness expose the absence of embodied affordance modeling, since pretrained internet imagery often lacks explicit annotations of left/right handedness or stacking physics. The brittleness in Relative Positioning and Spatial Navigation points to deficiencies in chaining multi-hop spatial relations; for instance, Pathway Existence requires integrating local connectivity with global layout, which is not well captured by attention-based co-occurrence priors. On the other hand, partial success in Size and Scale suggests that models exploit statistical regularities in perspective distortion or scale consistency, though they collapse under less frequent cases like shadows. These patterns collectively show that current models succeed when cues are salient and correlate with training priors but fail when reasoning requires counterfactual, compositional, or embodied inference. Addressing these issues will require architectural advances that explicitly model reference frames, occlusion layers, and physical dynamics, as well as training data enriched with embodied simulations and counterfactual augmentations. Without these, progress in spatial reasoning will remain fragmented, with models performing well on surface-level cues but breaking down on structured, real-world spatial logic.

Analysis of MCQ vs. Open-Ended Performance Gaps

In this section we provide a detailed analysis of the performance gap between multiple-choice (MCQ) and open-ended evaluation formats across spatial reasoning categories in SPATIALAB, as demonstrated in Table 18.

Quantitative snapshot

Across 25 models, the average performance gap between MCQ and open-ended formats is 23.0% with a standard deviation of 5.5%. Dataset-wide means per subtask are as follows: 3D geometry (24.57), depth & occlusion (23.45), orientation (23.70), relative positioning (23.11), size & scale (23.42), and spatial navigation (22.89), yielding an overall average of 23.01. The overall gap ranges from a minimum of 11.93 to a maximum of 31.57 (median ≈ 23.34). At the family level, spatial reasoning models exhibit the largest average gap (27.03), while reasoning-oriented models show the smallest (19.11). Proprietary models have lower gaps on average (21.09) compared to many long open-source models (24.51). Notably, the human baseline shows an overall gap of 22.64. The highest gaps are observed in InternVL3.5-72B (31.57%), Qwen-VL2.5-32B (29.85%), and SpaceQwen2.5-VL-3B-Instruct (29.78%). The lowest belong to Llama-3.2-11B (11.93%), Gemini-2-Flash-Thinking (13.00%), and GPT-5-mini (13.36%). Correlational analysis reveals that spatial navigation dominates the overall gap (Pearson r=0.99r = 0.99), with orientation (r=0.83r = 0.83) and 3D geometry (r=0.79r = 0.79) also showing strong alignment.

Patterns and observations

Three broad patterns emerge. First, specialist spatial reasoning models exhibit the largest MCQ→open-ended gaps, suggesting they are optimized for classification-style supervision and less robust to free-form generation. By contrast, reasoning-focused models display smaller gaps, reflecting the stabilizing effect of chain-of-thought-style training on generative outputs. Second, spatial navigation is the strongest predictor of overall disparity: models that falter on open-ended navigation reasoning drive the aggregate gap. Orientation and 3D geometry subtasks also contribute significantly, while relative position and size/scale vary more idiosyncratically across models. Third, scale alone does not explain robustness: smaller models such as Llama-3.2-11B achieve among the lowest gaps, while very large models such as InternVL3.5-72B show the highest. Finally, several negative or near-zero gaps appear (e.g., depth & occlusion for Llama-3.2-11B at −1.98, relative position for o4-mini at −3.18), indicating that in some cases open-ended evaluation is more reliable than MCQ, likely due to misleading distractors.

Hypotheses on root causes

We hypothesize several interacting factors that underlie the observed MCQ→open-ended performance gaps in SPATIALAB, each supported by statistical observations:

  1. MCQ structural advantage. MCQ format constrains the output space, allowing models to exploit surface cues or eliminate distractors. Across models, the mean MCQ score is systematically higher than the open-ended score, with the average gap at 23.0% and σ=5.5%\sigma= 5.5\%. Negative gaps observed for Llama-3.2-11B in depth & occlusion (−1.98) and o4-mini in relative position (−3.18) indicate that open-ended generation can occasionally outperform MCQ when distractors are misleading, reinforcing the notion that MCQ can artificially inflate or distort measured performance.

    Recommendation: To address this issue, we suggest complementing MCQ evaluations with open-ended tasks and subtask-level diagnostics, which can more accurately capture model competence and reduce overestimation caused by option-based cues.

  1. Specialization bias in spatial reasoning models. Spatial reasoning models exhibit the largest average gaps (27.03%) despite being optimized for spatial tasks, suggesting that they may have been trained primarily on categorical or synthetic selection tasks rather than free-form generation. For instance, InternVL3.5-72B, a 72B parameter spatial reasoning model, shows the highest overall gap (31.57%), particularly in spatial navigation (36.68%) and orientation (34.44%), highlighting a reliance on classification-like supervision. Correlations between subtask gaps and overall gap (SpNav: r=0.99r = 0.99, Orientation: r=0.83r = 0.83) indicate that these format-dependent failures are concentrated in sequential and geometric reasoning tasks.

    Recommendation: To mitigate this bias, we recommend fine-tuning or instruction-tuning on open-ended spatial generation tasks, which can improve model robustness to generative formats and reduce reliance on categorical supervision.

  1. Instruction-tuning and stepwise decoding. Reasoning-oriented models (e.g., Gemini-2-Flash-Thinking, o4-mini) demonstrate smaller average gaps (around 19.11%) and lower variance across subtasks. For example, Gemini-2-Flash-Thinking has an overall gap of 13.0% and consistently modest subtask gaps (Difference in 3D Geometry: 13.9%, Difference Relative Positioning: 21.03%), suggesting that chain-of-thought or stepwise decoding helps models produce stable open-ended responses. This is further supported by reduced correlation between subtask gap variance and overall gap within this family, implying that instruction-tuned reasoning mitigates format sensitivity.

    Recommendation: To address format sensitivity, we suggest adopting chain-of-thought or stepwise decoding strategies and instruction-tuning for open-ended spatial tasks, which can improve generative consistency and reduce MCQ dependence.

  1. Spatio-navigation stresses sequential grounding. Spatial navigation gaps are the strongest predictor of overall MCQ→open-ended disparity (Pearson r=0.99r = 0.99). Models with high performance gaps in spatial navigation, such as SpaceQwen2.5-VL-3B-Instruct (Spatial Navigation gap = 35.87%), also exhibit large overall gaps (29.78%), indicating that multi-step referential reasoning is disproportionately affected in open-ended formats. This suggests that MCQ options simplify the reasoning chain, masking underlying weaknesses in sequential grounding and relational computation.

    Recommendation: To better capture sequential reasoning abilities, we recommend including dedicated open-ended spatio-navigation tasks and employing targeted multi-step reasoning supervision, which can improve grounding and reduce overestimation by MCQ.

  1. MCQ distractor artifacts. Some negative or near-zero gaps indicate that poorly designed distractors can suppress measured MCQ performance. For example, o4-mini shows a negative relative position gap (−3.18%) while maintaining positive gaps in other subtasks, implying that distractor quality can artificially depress MCQ accuracy. Conversely, models like Qwen-VL2.5-32B-Instruct exhibit very high MCQ→\toopen gaps (overall = 29.85%) across multiple subtasks, suggesting that MCQ distractors can also exaggerate perceived competence in selection-based tasks. Recommendation: To address this, we suggest careful auditing and refinement of MCQ distractors and complementing them with open-ended evaluation, which can yield more accurate and unbiased measures of model spatial reasoning ability.

Practical implications

Our findings show the importance of reporting both MCQ and open-ended results. Exclusive reliance on MCQ inflates estimates of practical spatial competence, with average disparities around 23%. Subtask-level breakdowns are essential, as spatial navigation disproportionately drives observed differences. Diagnostic evaluations that disentangle classification versus generative reasoning can better isolate error modes. Moreover, instruction-tuning and decoding strategies that encourage stepwise reasoning appear promising for reducing MCQ dependence. Finally, careful design and auditing of MCQ distractors are necessary to ensure fair comparisons across formats.

Details on Improving Visual Reasoning Capabilities

To enhance spatial reasoning performance on our diverse visual question benchmark, we explored multiple complementary strategies.

Inherent Reasoning Mechanisms

We next examine the role of built-in reasoning mechanisms by comparing standard models with their reasoning-enabled counterparts (Table 2, Table 3). Across MCQ evaluations, reasoning consistently boosts performance, though the magnitude varies by category. For instance, Gemini-2.5-Flash-Thinking improves over its non-reasoning variant by +4.64% overall (52.93% vs. 48.29%), with particularly strong gains in Relative Positioning (+13.1%) and Orientation (+8.6%). A similar trend holds for o4-mini-medium, which reaches the highest overall MCQ accuracy (53.21%), suggesting that explicit reasoning scaffolds are especially effective in structured, discrete-answer settings.

Open-ended evaluation reveals a more nuanced picture. Here, reasoning-equipped models again outperform their baselines, but the gains are uneven. Gemini-2.5-Flash-Thinking improves slightly over Gemini-2.5-Flash (32.77% vs. 30.93%), but the largest leap is observed in o4-mini-medium (37.86%), which significantly surpasses all non-reasoning models. These results indicate that reasoning mechanisms help mitigate the instability seen in free-form generation, especially in complex categories such as 3D Geometry and Spatial Navigation. However, gains are not universal, reasoning models sometimes show regressions, e.g., Gemini-2.5-Flash-Thinking drops sharply in Size & Scale (21.74% vs. 26.59%), echoing our earlier finding that generative settings amplify brittleness.

Overall, these comparisons highlight that explicit reasoning modules substantially enhance structured MCQ performance and can stabilize open-ended responses, but their benefits are uneven across categories. The results suggest that reasoning mechanisms strengthen logical consistency and relational chaining, yet they do not fully resolve the challenges of scale sensitivity, occlusion, and free-form linguistic grounding.

Chain-of-Thought (CoT) Prompting

To examine whether structured reasoning improves spatial performance, we further evaluated models under Chain-of-Thought (CoT) prompting. For each of the 30 sub-categories in SPATIALLAB, we sampled 5 items (seed fixed to 42) to construct a balanced test set covering all major spatial phenomena. This design ensures fair coverage while keeping evaluation computationally tractable. Example CoT prompts are shown in Figure 7, which illustrate the step-by-step reasoning scaffolds provided to the models. The setup allows us to isolate the contribution of CoT reasoning from model scale or training.

Chain-of-thought promptsVector (PDF)
SpatiaLab-MCQ : CoT : Prompt for Answering Model

You are a spatial reasoning assistant. Given a multiple choice question about an image,
think step by step and answer with a single number (1-4) corresponding to the option in exactly one
<result>...</result> tag.
Question:
{question}
Options:
1. {opt1}
2. {opt2}
3. {opt3}
4. {opt4}

Final answer format: <result>1</result>
SpatiaLab-Open : CoT : Prompt for Answering Model

You are a spatial reasoning assistant.
Given a multiple choice question about an image, think step by step and the following question in 2-4 sentences.

Question:
{question}

Figure 7. Chain-of-thought (CoT) Prompts.

Our analysis (Table 19) shows that orientation is the only category where CoT prompting reliably improves performance, while most other categories experience either stagnation or decline. This suggests that reasoning helps when the task is reducible to logical alignment of directions, but fails when more complex perceptual priors (e.g., depth, scale, relative positioning) are required. We anticipate this happens because models lack inherent knowledge about 3D perception and geospatial relationships; they are not fundamentally “aware” of these concepts in the way humans are. As a result, when CoT prompting is applied, the step-by-step reasoning does not correct misconceptions but instead reinforces them, propagating and amplifying errors across reasoning steps.

This finding highlights that our benchmark is non-trivial: it cannot be solved merely by adding multi-step reasoning to existing models. Unlike textual reasoning tasks where CoT reliably improves outcomes, our benchmark exposes deeper architectural limitations in perceptual and spatial grounding. In categories like depth and size, incorrect priors mean that every additional reasoning step moves the model further from the correct solution. The sharp performance drops, particularly for Gemini in MCQ settings, demonstrate how fragile reasoning becomes when it is not supported by robust perceptual knowledge.

Thus, our task design is a strength, it goes beyond surface-level reasoning and probes whether models possess a genuine understanding of geospatial and 3D relational concepts. Solving it requires more than chaining logic; it demands embeddings and training that capture core aspects of perception and spatial cognition. In this way, our benchmark challenges models at a fundamental level and highlights an important gap in current architectures.

Chain-of-Thought (CoT) Prompting with Self-Reflection

We further extend the CoT setup by adding a self-reflection stage, where the model is prompted to review and revise its initial reasoning before finalizing an answer (see Figure 8). The evaluation protocol remains identical to the standard CoT setting, using 5 balanced samples per sub-category (seed 42). This design isolates the effect of self-reflection while keeping the comparison directly aligned with baseline CoT performance.

Table 20: Performance with CoT vs. CoT + Self-Reflection (CoT-SR) across categories for InternVL-3-78b and Gemini-2.5-Flash. Values are accuracies (%) and Gains represent the difference between CoT-SR and CoT.

CategoryInternVL-3-78b (MCQ)InternVL-3-78b (Open)Gemini-2.5-Flash (MCQ)Gemini-2.5-Flash (Open)
CoTCoT-SRGainCoTCoT-SRGainCoTCoT-SRGainCoTCoT-SRGain
3D Geometry52.0056.004.0028.0020.00-8.0020.0052.0032.0028.0024.00-4.00
Depth & Occlusion60.0044.00-16.0012.0016.004.0028.0052.0024.0016.0016.000.00
Orientation64.0064.000.0028.0032.004.0024.0048.0024.0032.0036.004.00
Relative Positioning72.0064.00-8.0056.0028.00-28.0036.0072.0036.0032.0032.000.00
Size & Scale44.0056.0012.0020.0012.00-8.0060.0056.00-4.008.0012.004.00
Spatial Navigation64.0056.00-8.0040.0024.00-16.0052.0064.0012.0028.0028.000.00
Overall59.3356.67-2.6630.6722.00-8.6736.6757.3320.6624.0024.670.67

Table 20.

Adding self-reflection to Chain-of-Thought (CoT) prompting produced mixed and highly model-dependent outcomes. For InternVL-3-78b, overall accuracy declined slightly (−2.66%-2.66\% MCQ, −8.67%-8.67\% open-ended), with sharp drops in categories like depth, relative positioning, and navigation, suggesting that reflection often reinforced errors instead of correcting them. Some localized gains were observed, such as +12% in Size & Scale (MCQ), but these were inconsistent and offset by large degradations (−28% in relative positioning open-ended). In contrast, Gemini-2.5-Flash showed strong MCQ gains (+20.66%), with particularly large improvements in 3D Geometry (+32%) and Depth & Occlusion (+24%), though open-ended improvements remained negligible (+0.67%). These results highlight a key asymmetry: reflection helps in multiple-choice contexts where models can prune unlikely options, but does not transfer to free-form generation where uncertainty propagates into ungrounded rationales. We also find that the categories most improved by reflection are those where geometric consistency is central (orientation, depth, size), while tasks requiring relational integration across multiple objects (relative positioning, navigation) degrade. Overall, reflection functions as a stabilizer in structured choice settings but fails to reliably enhance naturalistic reasoning.

This observed inconsistency mirrors our earlier findings with plain CoT prompting: reasoning enhancements only succeed when the model has stable perceptual anchors to ground its steps. Self-reflection adds an explicit verification pass, but this mechanism is only as strong as the model’s internal priors. When depth cues, occlusion ordering, or coordinate frames are poorly encoded, the second pass amplifies misconceptions rather than repairing them. This explains why Gemini, which exhibits stronger perceptual grounding, benefits under MCQ conditions, while InternVL-3-78b suffers more frequent degradations. The broader root cause lies in a pretraining mismatch: current VLMs import reasoning templates from language modeling but lack the embedded 3D and geospatial representations that humans leverage. Without these anchors, reflection cannot perform true error correction; instead, it functions as a linguistic filter that polishes reasoning syntax without improving perceptual grounding. As a result, reflective reasoning loops provide modest utility in discrete answer formats but collapse under open-ended demands where factual verification is absent.

These findings confirm that our benchmark shows a fundamental gap in current architectures: surface-level reasoning layers alone cannot compensate for missing spatial cognition, and meaningful progress will require deeper integration of geometric and perceptual structure. Moreover, because VLLMs are commonly trained on web data, the concepts that are well-represented online are those where models tend to perform better, whereas poorly represented spatial and 3D concepts suffer, reinforcing these performance disparities.

Supervised Fine-Tuning (SFT)

Setup and Implementation

We fine-tuned the Qwen 2.5 VL 3B instruction-following model (unsloth/Qwen2.5-VL-3B-Instruct) using a parameter-efficient approach with LoRA [29]. To accommodate limited GPU resources, we loaded the model in 4-bit precision (load_in_4bit=True) and inserted LoRA adapters into attention and feed-forward network projection layers. The LoRA hyperparameters, including rank, scaling factor, dropout, and target modules, are summarized in Table 21. Gradient checkpointing was enabled to reduce memory usage, and a fixed random seed ensured reproducibility.

HyperparameterValueNotes
LoRA rank (rr)16Low-rank update capacity
LoRA alpha16Scaling factor
LoRA dropout0No dropout applied
Gradient checkpointingEnabledLibrary-specific flag
LoRA random seed3407Adapter initialization

Table 21. LoRA Hyperparameters for SFT

For tokenization, we followed the Qwen-2.5 chat template to maintain consistent formatting between training and inference. Each training example was converted into a single text sequence, and dynamic padding was applied with DataCollatorForSeq2Seq. Training used an effective batch size of 4 (per-device batch size 1 with gradient accumulation of 4). Trainer and optimization hyperparameters, including learning rate, optimizer, weight decay, and scheduler, are listed in Table 22. The training loop consisted of four passes, each spanning a single epoch, yielding four saved checkpoints.

HyperparameterValue
Base modelunsloth/Qwen2.5-VL-3B-Instruct
Max sequence length2048
Load in 4-bitTrue
Per-device train batch size1
Gradient accumulation steps4
Effective batch size4
Learning rate2e-4
Optimizerpaged_adamw_8bit
Weight decay0.01
LR schedulerlinear
Logging steps1
Seed42
Tokenizer templateQwen-2.5

Table 22. Trainer and Optimization Hyperparameters for SFT

Overall, this setup allowed us to efficiently fine-tune a large vision-language model on limited GPU resources while maintaining high adaptation capacity.

Supervised Fine-Tuning (SFT) Performance

The learning trends in Figure 4a reveal notable differences between the MCQ and Open-ended evaluation modes over training epochs. We observe that MCQ accuracy begins at a lower baseline of 25.63% but rises consistently, converging near 36.59% by the fourth epoch. This steady improvement suggests that the model benefits strongly from additional supervised fine-tuning, particularly in tasks that constrain answers within a limited candidate space. By contrast, Open-ended evaluation starts higher at 34.40% but declines sharply by the second epoch, dropping to as low as 12.62%. Only in later epochs does it partially recover, reaching 35.48% at the end. This instability points to a sensitivity in open-ended generation that is not present in the MCQ setting, where the structure of multiple-choice questions provides more stable optimization signals. We interpret the fluctuations in open-ended performance as evidence that fine-tuning may overfit to MCQ-like reasoning patterns at the expense of free-form language generalization. The dotted baselines in the plot further highlight how the MCQ mode achieves gains beyond its initial standing, whereas Open-ended remains approximately stagnant. Overall, the figure illustrates that SFT predominantly favors MCQ tasks in terms of stability and consistent accuracy gains.

The tabulated results in Table 4b complement the figure by providing a detailed before-and-after breakdown across six spatial reasoning categories. Across MCQ tasks, we find that performance improved in every category, with gains ranging from a modest 1.59% in Relative Positioning to a substantial 17.11% in Size and Scale. The Grand Total gain for MCQ stands at 10.97%, confirming that the fine-tuning procedure effectively enhances structured-answer performance. In contrast, Open-ended results are far more mixed, with both positive and negative changes. Categories such as 3D Geometry and Spatial Navigation show small but positive gains (3.50% and 2.82%, respectively), yet Size and Scale exhibits a significant drop of −3.95%. Relative Positioning also decreases by −0.79%, reflecting that fine-tuning can inadvertently harm certain reasoning modes in free-form contexts. The overall Open-ended gain is only 1.07%, which is statistically negligible compared to MCQ improvements. These results confirm that the effectiveness of SFT is highly sensitive to the evaluation paradigm. By isolating the categories, we underscore where training helps, where it fails, and how performance benefits differ between constrained and unconstrained outputs.

A deeper inspection reveals striking disparities between categories and formats. For MCQ, Size and Scale and Spatial Navigation yield the highest improvements, suggesting that models trained with multiple-choice prompts are especially effective at learning tasks requiring scale discrimination and navigational reasoning. However, in Open-ended form, Size and Scale actually declines, showing that the same learned representations do not transfer well to generative contexts. Similarly, Relative Positioning barely improves in MCQ and worsens in Open-ended, highlighting a consistent weakness in relational reasoning regardless of format. Conversely, Orientation shows strong MCQ gains (+13.33%) and small Open-ended improvements, suggesting that some categories generalize more robustly. We hypothesize that the divergence between the two formats may stem from the structural scaffolding MCQs provide, which reduces the need for the model to generate precise linguistic formulations. Open-ended tasks, by contrast, require simultaneous mastery of reasoning and natural language fluency, which amplifies weaknesses. The trends indicate that while fine-tuning strongly enhances pattern recognition in discrete-choice settings, it introduces volatility and even degradation when free expression is required. This duality makes clear that training interventions optimized for MCQs may not generalize seamlessly to unconstrained reasoning.

The implications of these findings are twofold. First, we show that SFT as currently applied provides strong benefits for structured tasks but limited or even negative transfer to open-ended reasoning. This limitation is not only an artifact of fine-tuning, but also symptomatic of how contemporary VLMs are pretrained. Existing pretraining pipelines heavily emphasize broad image–text alignment, object recognition, and captioning, yet they provide little explicit supervision on compositional spatial structures, occlusion layers, or geometric transformations. As a result, models learn to excel at surface-level associations while failing to acquire robust representations of depth, orientation, or scale. The instability we observe in open-ended spatial reasoning suggests that the model’s learned priors are fragile: without multiple-choice scaffolding, they revert to shallow pattern-matching instead of grounded inference. Categories such as Size and Scale and Relative Positioning, where declines are most pronounced, point to missing inductive biases in pretraining, biases that humans develop through embodied interaction with 3D environments. Another contributing factor is the reliance on synthetic or instruction-following datasets that flatten spatial diversity into puzzle-like abstractions, preventing models from generalizing to real-world clutter, noise, and complexity. These findings underscore the need to rethink both pretraining and fine-tuning for spatial reasoning, incorporating datasets and objectives that reflect naturalistic visual–spatial challenges. Ultimately, we argue that without targeted reforms, current VLMs risk reinforcing an algorithmic straightjacket: they appear competent under constrained evaluations yet remain brittle and unaligned with the embodied, flexible reasoning that humans bring to real-world spatial understanding.

SFT Dynamics Analysis

To investigate the stability of Supervised Fine-Tuning (SFT) on spatial reasoning tasks, we conducted a multi-seed validation study (N=5N = 5, seeds A–E) using the Qwen-2.5-VL-3B-Instruct model. The training trajectory, summarized in Table 23, reveals a distinct divergence in performance dynamics between Multiple-Choice Question (MCQ) and Open-Ended evaluation formats.

EpochTypeSeed ASeed BSeed CSeed DSeed E
0 (Base)MCQ25.63%25.63%25.63%25.63%25.63%
Open34.40%34.40%34.40%34.40%34.40%
1MCQ29.56%35.83%24.13%27.63%26.53%
Open22.50%23.20%12.50%14.00%24.10%
2MCQ34.33%37.76%27.33%31.83%34.43%
Open12.62%25.30%28.80%23.80%16.65%
3MCQ34.68%37.33%32.86%32.23%34.73%
Open28.69%22.22%30.34%32.90%28.30%
4MCQ36.59%38.48%34.43%34.36%34.83%
Open35.48%34.59%33.92%34.70%35.40%

Table 23. Multi-seed validation of SFT dynamics (N=5N = 5). While MCQ performance improves steadily, Open-Ended performance exhibits a systematic “U-shaped” collapse and recovery, indicating representational instability in generative spatial tasks.

Divergent Learning Trajectories. As illustrated in Table 23 and Figure 9, performance on SPATIAlAB-MCQ exhibits a monotonic improvement, rising from a baseline of 25.63% to an average of 35.74% at Epoch 4. This indicates that the model successfully adapts to the discriminative nature of the task, learning to map visual features to constrained output options with increasing reliability.

Figure 9. SFT Dynamics Analysis.

In sharp contrast, SPATIALAB-OPEN displays a pronounced U-shaped trajectory across all seeds. Rather than steady improvement, the model suffers an immediate and significant performance collapse in the early epochs (dropping from 34.40% to an average of 19.26% at Epoch 1). While performance recovers in later epochs (converging to ≈34.82% at Epoch 4), the final gain over the zero-shot baseline is negligible (+0.42%).

Representational Brittleness and Alignment Tax. We attribute this phenomenon to the fragility of spatial representations in current VLMs when subjected to fine-tuning. The initial collapse in open-ended accuracy suggests catastrophic forgetting of the linguistic priors required for fluent, grounded generation. As the model optimizes for the specific instruction-following format of the training data, it overfits to the structural logic of the tasks (benefiting MCQ) at the expense of the flexible, generative spatial grounding required for open-ended descriptions.

The subsequent recovery phase represents the model re-learning to articulate spatial concepts within the new distribution, yet it fails to surpass the initial baseline significantly. This dynamic underscores a critical finding: SFT is effective for teaching models to select correct spatial answers, but it does not fundamentally enhance the underlying generative spatial representation needed for robust open-ended reasoning.

Performance Gain on External Benchmarks

To assess whether the representations learned from SPATIALAB generalize beyond the benchmark’s specific format, we evaluated the transferability of our fine-tuned model to three external spatial reasoning datasets: OMNISPATIAL, SPACE (Multimodal), and MIND THE GAP. We employed the Qwen2.5-VL-3B-Instruct model, fine-tuned on SPATIALAB via the SFT protocol detailed in Appendix H.4.1, and evaluated it on these external benchmarks in a zero-shot setting.

As shown in Table 24, fine-tuning on SPATIALAB yields substantial and consistent performance improvements across all evaluated domains, ranging from +5.24% to +11.70%. The most significant gain was observed on MIND THE GAP (+11.70%), suggesting that our benchmark’s emphasis on occlusion and spatial continuity effectively addresses the reasoning gaps targeted by that dataset. Similarly, improvements on OMNISPATIAL (+7.05%) and SPACE (+5.24%) confirm that the geometric and relational concepts encoded in our taxonomy align with broader definitions of spatial intelligence in the literature. These results validate the quality of the SPATIALAB data. While the primary purpose of this work is to serve as a rigorous diagnostic benchmark, the strong transferability demonstrates that the dataset does not merely encourage overfitting to a specific question template. Instead, the diversity of the 30 sub-categories forces the model to acquire robust, transferable spatial representations that generalize to unseen distributions and task formats.

BenchmarkBase AccuracyAfter SFTImprovement
OmniSpatial40.30%47.35%+7.05%
SPACE (Multimodal)23.43%28.67%+5.24%
Mind The Gap35.86%47.56%+11.70%

Table 24. Transfer learning performance of Qwen2.5-VL-3B-Instruct on external benchmarks after fine-tuning on SPATIALAB. The consistent gains indicate that the model learns generalizable spatial representations rather than merely overfitting to the source dataset’s format.

AI Agents for Spatial Reasoning

We developed SPATIOXOLVER, a multi-agent system adapted and extended from Xolver (Hosain et al. [28]), to perform structured spatial reasoning on images within the SPATIOLAB evaluation. The system decomposes complex visual reasoning into specialized sub-tasks, each managed by a dedicated agent, allowing for focused processing of objects, attributes, relations, symmetries, and transformations. This modular design not only improves accuracy and interpretability but also supports multi-step reasoning, enabling the system to propagate low-level perceptual cues through higher-order relational and transformational inferences. The architecture, agent implementation, and their coordinated interaction are detailed in the following sections, with the full set of prompts provided in Figure 10, Figure 11, and Figure 12.

Figure 10. Prompts for SPATIOXOLVER (Part 1).

!Spatial Relation Agent Prompt; Grouping Symmetry Agent Prompt; Transformation Tracking Prompt. You are a Spatial Relation Specialist. Extract spatial relationships among objects. Your focus: 1. Core Relations: - left_of, right_of, above, below; - inside, surrounding; - touching, overlapping; - aligned, centered. 2. Precision: - Always specify directionality (ObjA left_of ObjB is not the same as ObjB left_of ObjA); - Use consistent object IDs. 3. Structural Patterns: - Detect repeated relations (e.g., "each circle is above a square"); - Note spacing and alignment. 4. Output Format: Triples in form: (ObjectA, Relation, ObjectB). Example: (Obj1, left_of, Obj2); (Obj3, inside, Obj4). You are a Grouping and Symmetry Specialist. Identify higher-order structures from objects and relations. Relations: {relation_list}. Your focus: 1. Grouping: - Clusters of objects (rows, columns, triangles, grids); - Repeated patterns. 2. Symmetry: - Vertical, horizontal, rotational; - Mirror or reflective symmetry; - Translational repetition. 3. Output Format: JSON with two fields: "groups" and "symmetries". Example: { "groups": ["Obj1, Obj2, Obj3 form a horizontal row"], "symmetries": ["Vertical symmetry across Obj2"] }. You are a Transformation Tracker. Analyze changes across multiple frames of an image sequence. Your focus: 1. Types of Transformations: - Movement (left, right, up, down); - Scaling (larger, smaller); - Rotation (degrees); - Shading or color change; - Shape change (square → circle); - Appearance/disappearance. 2. Output Format: Per-object transformation logs: Obj1: Frame1 → Frame2 (rotated 90° clockwise); Obj2: Frame2 → Frame3 (moved right) bbox=0.18,0.14,0.82,0.88

Figure 11. Prompts for SPATIOXOLVER (Part 2).

Figure 12. Prompts for SPATIOXOLVER (Part 3).

Representation Standardization Agent Prompt

You are a Symbolic Abstraction Specialist. Consolidate all perception results into a standardized structured format.

Inputs:

  • Objects: {object_list}

  • Attributes: {attributes_json}

  • Relations: {relation_list}

  • Groups & Symmetries: {grouping_symmetry}

  • Transformations: {transformations}

Your focus:

  1. Create a unified JSON representation containing:

    • Objects with attributes

    • Relations as triples

    • Groups and symmetries

    • Transformations across frames

  1. Ensure consistency in IDs and formatting

  1. Example Output: { "objects": [...], "relations": [...], "groups": [...], "symmetries": [...], "transformations": [...]}

MCQ Spatial Option Prompt

You are a Spatial Reasoning Expert. Analyze the image data and select the correct option for the multiple-choice question.

Image Analysis Data:

  • Objects and Attributes: {attributes_json}

  • Spatial Relations: {relations_list}

  • Groups & Symmetries: {grouping_symmetry_json}

  • Transformations Across Frames: {transformations_json}

MCQ: Question: {question_text} Options: {options_text}

Instructions:

  • Consider shapes, positions, shading, rotations, symmetries, and transformations.

  • Identify the pattern or rule in the sequence of images.

  • Select only the correct option Only (1, 2, 3, 4 ).

  • Do NOT provide explanations, reasoning, or extra text. Only the option letter.

Output format: "<option_number>"

MCQ Spatial Option Prompt

You are a Spatial Reasoning Expert. Analyze the image data and give correct answer for the question.

Image Analysis Data:

  • Objects and Attributes: {attributes_json}

  • Spatial Relations: {relations_list}

  • Groups & Symmetries: {grouping_symmetry_json}

  • Transformations Across Frames: {transformations_json}

MCQ: Question: {question_text}

Instructions:

  • Consider shapes, positions, shading, rotations, symmetries, and transformations.

  • Identify the pattern or rule in the sequence of images.

Agents

Base Visual Analysis Agent. Base Visual Language Model (VLM) agent initiates the pipeline by generating a detailed textual description of the input image, capturing object types—including geometric shapes, natural elements, and artificial icons—alongside attributes such as size, orientation, color, and shading. Relative positions like leftmost, center, and top are encoded explicitly, and the prompt emphasizes exhaustive detail without summarization. Images are converted to data URIs and sent with the prompt to a Gemini-2.5-based model, whose output is parsed into a structured bullet-list using regex-based extraction. This structured description serves as the foundation for downstream agents, providing a comprehensive perceptual anchor that encodes both explicit and implicit spatial cues. The approach ensures that even subtle visual features, such as partial occlusions or minor shape variations, are retained for later relational and transformational reasoning.

Object Segmentation Agent. Object Segmentation agent converts the verbose textual description into a discrete list of uniquely identified objects, differentiating between visually similar instances. Each object receives a stable identifier (e.g., Obj1, Obj2) to maintain consistent references across subsequent reasoning steps. The agent uses carefully formatted prompts and calls the same Gemini-2.5 model to structure the description, while post-processing splits lines by object ID to enforce consistency. This structured representation reduces ambiguity in downstream attribute and relation extraction, enabling precise mapping of complex scenes. By explicitly isolating objects, the agent also facilitates detection of overlaps, repetitions, and hierarchical groupings that may not be evident in raw descriptions.

Attribute Extraction Agent. Attribute Extraction agent translates each object into a structured JSON capturing core properties such as shape, size, color, shading, orientation, and positional descriptors. Prompts enforce strict JSON output, and responses are parsed using json.loads(), with fallback defaults to ensure robustness against model inconsistencies. This structured encoding allows quantitative and qualitative comparisons of objects, supporting later stages of spatial reasoning. By preserving both absolute and relative measures, the agent maintains fidelity to the original perceptual content, while also normalizing descriptors for computational consistency. The JSON structure enables programmatic downstream operations, such as calculating symmetries or detecting transformations.

Spatial Relation Agent. The Spatial Relation agent encodes object interactions as triples (ObjectA,Relation,ObjectB), capturing directional relations (e.g., left_of, above, inside) and higher-order alignments like rows, grids, or repetition patterns. Consistent object IDs are enforced to prevent relational ambiguity, and parsing splits output lines starting with parentheses, followed by automated consistency checks. This step transforms static object descriptions into a relational graph that encodes spatial topology and hierarchy. The resulting structure allows reasoning over both pairwise interactions and more complex scene configurations, which is critical for tasks like symmetry detection, grouping, and multi-step navigation of spatial layouts.

Grouping and Symmetry Agent. Grouping and Symmetry agent abstracts higher-order patterns from object lists and spatial relations, identifying clusters, rows, grids, and symmetries—including vertical, horizontal, rotational, and translational. Outputs are requested in structured JSON with keys groups and symmetries, with empty structures as fallbacks to maintain downstream compatibility. This agent captures emergent scene regularities that are not apparent at the object or pairwise relation level, providing critical context for pattern recognition and reasoning. By formalizing structural redundancies and symmetries, the agent supports both perceptual inference and anticipatory reasoning for dynamic transformations.

Transformation Tracking Agent. For multi-frame sequences, the Transformation Tracking agent logs object-level changes including translations, scaling, rotations, color/shading shifts, and appearance or disappearance events. Structured logs are extracted line-by-line using regex patterns that capture directional arrows, producing a temporal representation of the scene. This encoding allows the system to track not just static spatial configurations but also dynamic evolution, supporting reasoning about motion, causality, and object interactions over time. The agent ensures that transformations are linked to consistent object IDs, enabling coherent cross-frame reasoning and predictive modeling.

Representation Standardization Agent. Finally, the Representation Standardization agent consolidates all outputs into a unified JSON format containing objects, relations, groups, symmetries, and transformations. Prompts enforce consistent object IDs and structured formatting, while JSON parsing failures revert to empty dictionaries to ensure pipeline robustness. By centralizing all perceptual, relational, and temporal information, this agent creates a comprehensive scene representation that is both human-interpretable and machine-readable. The resulting unified structure serves as a versatile foundation for downstream tasks, including reasoning evaluation, spatial pattern analysis, and multi-frame prediction.

Open-Ended Spatial Reasoning Agent. The standardized representation feeds into the reasoning agent, which answers multiple-choice or pattern recognition questions. Inputs include objects, attributes, relations, groups, symmetries, and transformations. The model considers geometric patterns, relational alignment, and dynamic transformations to generate or select the correct answer.

Technical Implementation Details

For our technical implementation, we use Gemini-2.5-Flash with a low temperature (0.1) to ensure deterministic outputs and perform multiple iterations for refinement. API calls are handled via OpenRouter, supporting both text and image inputs, while parsing of model response relies on regex and JSON-based extraction to generate structured outputs. All interactions are logged in JSON format for reproducibility, and the final output is a unified JSON per image containing comprehensive information on objects, relations, symmetries, and transformations. This multi-agent decomposition allows each specialized model to focus on a constrained sub-task, enhancing overall accuracy, interpretability, and robustness in structured spatial reasoning.

Evaluation and Analysis

The evaluation protocol remains identical to the standard CoT and CoT-SR setting, using 5 balanced samples per sub-category (seed 42), making a total of 150 samples.

The evaluation of Gemini-2.5-Flash with agents highlights both strengths and limitations of agentic reasoning in perceptual–geometric tasks. Orientation is the only category where agents provide consistent and substantial gains, improving by +8.00% in MCQ and an impressive +36.00% in open evaluation. This pattern suggests that agent-based multi-step reasoning is effective when the task can be decomposed into sequential alignment of directional cues, where structured deliberation offers a clear advantage over single-step outputs.

CategoryMCQ NormalMCQ With AgentsMCQ GainOpen Evaluation NormalOpen Evaluation Agent ResultsOpen Evaluation Gain
3D Geometry48.00%44.00%-4.00%60.00%48.00%-12.00%
Depth & Occlusion48.00%44.00%-4.00%40.00%16.00%-24.00%
Orientation56.00%64.00%+8.00%0.00%36.00%+36.00%
Relative Positioning60.00%60.00%0.00%50.00%36.00%-14.00%
Size & Scale48.00%52.00%+4.00%25.00%24.00%-1.00%
Spatial Navigation68.00%64.00%-4.00%40.00%28.00%-12.00%
Overall54.67%57.33%+2.66%35.83%31.33%-4.50%

Table 25. Agent Performance Analysis

In contrast, most other categories exhibit negative or neutral gains, revealing that agent reasoning does not universally enhance performance. For example, depth & occlusion suffers a severe -24.00% drop in open evaluation, while spatial navigation decreases by -12.00%. These tasks require deeper perceptual grounding in three-dimensional space, and the absence of such priors means that reasoning steps tend to reinforce misconceptions rather than correct them. Similarly, relative positioning shows stagnation in MCQ and a -14.00% decline in open evaluation, indicating that agent loops fail to add robustness when relational understanding is inherently weak.

Some categories show marginal improvements, such as size & scale (+4.00% in MCQ), but the gains are not stable and do not transfer to open evaluation. Even in 3D geometry, agents fail to provide a boost, with -4.00% and -12.00% drops in MCQ and open evaluation, respectively. These results underscore that our benchmark is complex and not solvable simply by layering multi-step agentic reasoning on top of existing models. Instead, it requires a deeper integration of geospatial understanding and perceptual representations, which current architectures lack.

Overall, the mixed results emphasize that agentic workflows alone are insufficient for advancing performance in this domain. While orientation tasks benefit from structured reasoning, most categories expose the models’ lack of inherent awareness of perceptual concepts. This makes errors propagate through multi-step reasoning, amplifying performance drops. Consequently, our benchmark serves as a strong stress test, highlighting that future progress will require embedding genuine perceptual and geospatial priors rather than relying solely on agent-based reasoning.

Qualitative Error Analysis and Error Patterns

This section presents a focused qualitative analysis of model failures observed in our visual reasoning benchmark. We explain each error class, provide diagnostic root cause analysis, identify recurring patterns, and describe verification protocols. The discussion aims to be both interpretable and actionable, so it can inform future architectural and evaluation design.

Overview

Across the sample set, models achieve strong coarse perception, they can reliably recognize object categories, colors, and simple binary relations, but they break down systematically when multiple cues must be integrated. Failures are not random noise but cluster into a small set of interpretable classes: spatial mislocalization, perspective and scale mistakes, occlusion and ordering errors, attribute confusion, and ungrounded open-ended rationalization. These classes recur across architectures and prompting strategies, which suggests that the underlying causes are common inductive biases in current vision–language pipelines rather than isolated training artifacts. Importantly, the same error often appears in both large-scale proprietary systems and small open-source baselines, indicating structural weaknesses shared across the model family. In the following, we unpack each error class with representative examples and root-cause hypotheses.

Error taxonomy and analysis

Spatial mislocalization and reference confusion.

Observation. When queries require distinguishing one object from a set of visually similar candidates, models frequently misidentify the target. For example, if asked to point out “the leftmost red cube among three stacked cubes,” the model may incorrectly output the middle cube, influenced by lighting highlights or central positioning. This problem becomes particularly acute in cluttered environments such as kitchen counters or crowded table scenes, where multiple overlapping objects share the same color and shape. In such cases, the model’s errors are not random: it tends to privilege objects that appear most salient due to size, brightness, or centrality, even when these cues contradict the explicit spatial instructions in the query. These consistent patterns highlight that the failures are systematic, not accidental.

Analysis. The recurring mistakes suggest that models lack a robust object-centric representation, or what cognitive science would call an “index” that persists across reasoning steps. Instead of maintaining a stable pointer to the correct candidate throughout the reasoning chain, models appear to drift, often reassigning attention to whichever object seems most visually striking at intermediate layers. This leads to reasoning errors such as starting with the leftmost cube but ending the chain with the brightest or largest cube. The inability to anchor spatial reference consistently makes tasks involving relative comparisons, like “the ball behind the second chair”, particularly fragile. In essence, the model operates more like a salience detector than a referential reasoner.

Root cause. At the architectural level, feature pooling operations, whether via attention or convolution, tend to mix contextual information from neighboring regions. While this is beneficial for global context, it erases the fine-grained, object-specific identity needed to distinguish between similar candidates. Once pooled, features of two adjacent red cubes may be nearly indistinguishable in the latent space. Without an explicit mechanism, such as slot-based representations or pointer tokens, the network collapses distinct entities into blended embeddings. This structural limitation prevents reliable referent tracking across reasoning steps and explains why models consistently confuse dense, overlapping objects.

Perspective and scale mistakes.

Observation. Tasks involving metric reasoning about relative size, fit, or scale are inconsistently handled. On simple comparisons like “is a car larger than a bicycle,” models typically succeed because such relationships align with strong statistical priors. However, when faced with near-threshold decisions such as “will the small sphere fit inside the slightly larger cylinder,” errors become frequent and unpredictable. A common pattern is reliance on canonical object sizes: for example, predicting that a door must be taller than a truck even in a rendering where the truck is scaled down. Another failure occurs in unusual visualizations, such as toy miniatures or architectural models, where familiar proportions are deliberately inverted. In these cases, models cling to their priors and ignore the actual geometric evidence visible in the image.

Analysis. These behaviors indicate that models rely heavily on distributional statistics of object categories rather than performing true geometric inference. Instead of reasoning about projection geometry, relative bounding box sizes, or depth cues, they substitute their world knowledge of typical scales. This shortcut works well in everyday contexts but fails catastrophically in edge cases that deviate from the training distribution. For instance, when an unusually small airplane is rendered next to a person, the model still predicts the airplane as larger. Such errors highlight the lack of grounding in pixel-level metric cues and the absence of consistent use of perspective or vanishing points.

Root cause. Current architectures do not include explicit geometric reasoning modules that preserve projection-aware relationships. Supervision for scale and fit tasks is limited, so models learn correlations between object category labels and size but not the underlying geometry. Furthermore, training objectives prioritize overall accuracy rather than fine-grained metric consistency, meaning that scale-sensitive errors are under-penalized. Without inductive biases for 3D reasoning or targeted supervision, the model defaults to memorized priors about canonical sizes rather than interpreting the actual perspective geometry in the image.

Occlusion and ordering failures.

Observation. Determining which object is in front, behind, or partially hidden remains a core weakness. Models often misreport the foreground object when two animals overlap, or when semi-transparent barriers like glass or fences are involved. For instance, in an image where a cat sits partially behind a curtain, models may incorrectly describe the curtain as being occluded by the cat. Thin or fragile structures, such as wires, tree branches, or railings, are particularly error-prone: they are frequently omitted altogether, leading to wrong answers in questions like “what object lies behind the fence.” These mistakes occur consistently, especially when occlusion cues are subtle or when the depth difference is small.

Analysis. Such failures indicate that models treat depth and occlusion as weak, secondary signals. While the pixel data contains clear occlusion cues (e.g., edge boundaries, T-junctions), the intermediate features learned by the model do not retain them with sufficient resolution. Aggregation layers blur boundaries and downweight thin structures, effectively removing the very cues necessary to resolve ordering. This is why models succeed in gross depth distinctions (foreground vs. background) but collapse in fine-grained cases where partial occlusion must be inferred. In practice, they guess rather than infer, with systematic biases toward treating the visually larger or brighter object as foreground.

Root cause. The architecture lacks explicit mechanisms for multi-scale feature retention and has no dedicated supervision for depth or occlusion reasoning. Thin or partially occluded signals are easily lost when features are pooled across large receptive fields. Furthermore, training data often underrepresents subtle occlusion cases, especially those involving transparency or fine boundaries. As a result, the learned representations flatten layered scenes into 2D appearance maps, preventing robust construction of depth-aware relational reasoning.

Attribute confusion and semantic swap.

Observation. Models sometimes conflate perceptual attributes (such as color, texture, or material) with functional or semantic roles. A classic example is misjudging whether a crosswalk is safe: models may base the answer solely on the presence of painted stripes while ignoring the state of the traffic light or approaching vehicles. Similarly, a surface with a metallic texture may be misclassified as “steel” even if it is painted plastic. In tool-use contexts, the presence of handles or familiar shapes may lead the model to infer functionality that the object does not possess. These confusions are frequent in tasks requiring fine-grained visual grounding of attributes.

Analysis. These errors reveal that language priors strongly bias the model’s predictions when visual input is ambiguous. The decoder favors plausible narratives based on co-occurrence in the training data: “crosswalks are safe when stripes are visible,” “shiny textures correspond to metal,” and so on. This shortcut allows models to produce semantically coherent answers even when they do not inspect the relevant visual evidence. The result is superficially correct reasoning in familiar contexts but systematic errors in atypical or adversarial scenarios. In practice, this manifests as overfitting to textual stereotypes rather than grounding in perceptual details.

Root cause. The underlying training objective, maximizing likelihood of fluent and plausible text, does not enforce visual grounding. As long as the generated answer is semantically reasonable, the loss is minimized, even if the reasoning is incorrect. Consequently, the model is incentivized to substitute common-sense priors for genuine perceptual inference. Without auxiliary losses or verification steps to tie textual claims to image regions, models will continue to swap perceptual and semantic attributes.

Open-ended rationalization without verification.

Observation. For explanatory or why-style questions, models often generate coherent but unsupported narratives. For example, when asked “why is the object leaning,” a model might answer “because the surface is slippery” even though slipperiness cannot be visually inferred. In another case, a tilted shelf might be explained as “due to an earthquake,” a claim entirely outside the evidence in the image. These errors highlight a tendency to fill gaps with imaginative but ungrounded rationales, producing fluent explanations that mislead users about the true evidence.

Analysis. The issue stems from a decoupling between text generation and visual grounding. The model generates high-quality language that reads convincingly, but there is no mechanism ensuring that these rationales are tied to actual visual features. Confidence calibration compounds the problem: models often assign high certainty to fabricated explanations, giving the false impression of reliability. As a result, open-ended outputs may look authoritative while being factually baseless. This mode of failure is particularly concerning in safety-critical domains where plausible but false rationales could misguide decision-making.

⊛ Root cause. Training is dominated by language-modeling objectives that reward fluency and coherence but not factual grounding. Decoders are optimized to produce the most probable continuation of text given the prompt, without constraints linking the explanation back to specific image evidence. The lack of an explicit verification layer or grounding requirement means that models default to linguistic plausibility over perceptual truth. Without additional architectural or training constraints, open-ended rationalizations will continue to prioritize surface coherence at the expense of evidential accuracy.

Error patterns and cross-model signals

When comparing failures across different models, several consistent patterns appear, revealing insights into the underlying mechanisms of errors. One clear trend is that disagreements between models are systematic rather than random: some models consistently rely on geometric cues such as relative size, depth ordering, and occlusion relationships, while others fall back on linguistic priors or common-sense assumptions derived from training text. This leads to predictable divergences, for example when two models give different answers about which object is “in front” despite seeing the same scene. Errors also spike when multiple cues must be integrated simultaneously. Tasks that require reasoning about occlusion, relative scale, and perspective together are particularly challenging; for instance, understanding whether a larger object partially occluding a smaller one is in front or behind often triggers consistent mispredictions. Calibration is another noticeable issue: high-confidence predictions frequently correspond to incorrect answers, especially in open-ended explanations or complex spatial queries. These patterns indicate that errors arise from how models balance perceptual grounding and linguistic plausibility. Different architectures exhibit distinct tendencies, some over-rely on visual geometry, others overfit to language correlations, so failures are not just isolated mistakes but reflect deeper trade-offs in reasoning strategies. This also implies that any attempt to improve performance should account for both sources of bias simultaneously rather than treating errors as independent events.

Mitigation strategies

Given these error patterns, several targeted strategies can be pursued to reduce failures without requiring complete architectural redesign. For instance, geometry-aware supervision can be incorporated by adding auxiliary tasks like depth prediction, occlusion ordering, or overlap estimation. This encourages models to respect spatial constraints and improves consistency across predictions. Multi-scale feature retention is another approach: by using edge-aware pooling or attention mechanisms, models can maintain information from thin or partially occluded structures that would otherwise vanish in standard downsampling. Coordinate anchoring can help resolve confusion between viewer-centric and object-centric frames by explicitly representing reference points for relative positioning. Counterfactual augmentation can also be applied, where distractor objects are systematically swapped or perturbed, forcing the model to rely on the true referent cues rather than heuristics. Lightweight verification modules can be added on top of existing outputs, cross-checking claims against visual or geometric constraints to filter implausible predictions. Each of these strategies can be implemented incrementally, providing clear paths to reduce systematic failures while keeping the core model largely unchanged.

In addition to these architectural and training augmentations, supervised fine-tuning (SFT) offers a practical pathway to instill more reliable spatial reasoning. By curating datasets that explicitly highlight failure cases, such as near-threshold scale comparisons, occlusion ambiguities, or reference resolution traps, SFT can directly expose models to the edge scenarios where they otherwise default to heuristics. Carefully designed instruction-style fine-tuning further encourages step-by-step reasoning, prompting the model to verbalize intermediate spatial relations rather than skipping to a guess. This not only strengthens grounding but also improves interpretability, as errors become traceable to specific reasoning steps. Moreover, SFT can be paired with counterfactual or adversarial augmentations, ensuring that the model does not simply memorize but instead learns invariances to distractors and context shifts. Importantly, these interventions require no wholesale changes to the architecture: they adapt existing models by reshaping their inductive biases through supervision. In this sense, SFT serves as a lightweight yet powerful complement to geometry-aware and verification-based strategies.

In this work, we additionally performed SFT using 40% of the training data and observed clear performance gains, particularly on multiple-choice questions. This suggests that targeted supervision helps models better internalize spatial cues and resolve distractor confusions when answer options are well-defined. However, improvements on open-ended responses were more limited. The model continues to produce fluent but weakly grounded explanations, highlighting that SFT alone cannot fully address rationalization without verification. These findings reinforce the need to combine SFT with geometry-aware supervision, counterfactual augmentations, and lightweight verifiers to achieve balanced progress across both structured and free-form tasks.

Our analysis shows that failures are not arbitrary but structured and interpretable, emerging from recurring patterns rather than random noise. This indicates that VLM errors follow systematic trajectories tied to the inductive biases of their architectures and training regimes. For example, geometric reasoning often collapses when occlusion cues are weak, and in such cases language priors take over, producing answers that sound plausible but lack visual grounding. Similarly, scale inference drifts when canonical object size priors dominate over pixel-level metric cues, leading to consistent misjudgments in near-threshold scenarios. These findings highlight that the gap is not simply one of capacity but of representation: current models lack explicit mechanisms to anchor visual features to grounded geometric relations.

By adopting the proposed verification protocols and targeted mitigation strategies, researchers can address these weaknesses directly. Geometry-aware supervision and counterfactual augmentation would help models resist shortcuts, while multi-scale feature retention ensures thin structures and partial occlusions are not discarded. SFT, when carefully applied, can further strengthen structured reasoning by aligning model predictions with task constraints, though it must be combined with grounding objectives to avoid overfitting to discrete-choice formats. Importantly, verification modules provide a lightweight way to enforce consistency between claimed relations and image evidence, counteracting the tendency to rationalize without checking. Taken together, these steps chart a practical pathway toward systems that reason more like humans: robust to clutter, interpretable in their failures, and capable of explaining not only what they predict but why.

Representative Benchmark Samples Across All Sub-Categories

We present representative benchmark samples for all 30 sub-categories in SPATIAlAB, illustrating the diverse spatial reasoning challenges covered. These samples highlight the variety of visual phenomena, including 3D geometry, occlusion, orientation, relational positioning, size and scale, and navigation tasks. Together, they provide a concrete view of the types of reasoning each model must perform to succeed across the benchmark.

Figure 13. 3D Geometry: Gravity Effects

Question: From which direction would the wind cause the liquid in the glass which is in the front to spill onto the ground first?

Options

  1. From the left side

  2. From the right side

  3. From the front

  4. From the back

Gold MCQ Answer: 1

Gold Open ended Answer: From the left side

MCQ Answers:

• GPT-5-mini: 4 • Gemini-2.5-Flash: 4 • InternVL3.5-72B: 1 • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

• GPT-5-mini: A gust blowing from the back of the patio (the top of the image, toward the viewer) would push the liquid forward over the near edge and make it spill first. Winds from the sides would tend to slosh it sideways rather than directly over the front rim.

• Gemini-2.5-Flash: The glass in the front is the one on the right side of the coffee table. Given its position, a wind blowing from the left (from the direction of the wooden cabinet and tall plant) would likely cause the liquid to spill first. This is because the glass is positioned with its opening facing somewhat towards the left, making it more susceptible to a gust from that direction.

• InternVL3.5-72B: The wind would need to come from the upper-right direction to cause the liquid in the glass on the front right of the table to spill first. This is due to the positioning of the glass towards the front edge of the table, closest to the right side of the image.

• Kimi-VL-A3B-Thinking-2506: Okay, let's tackle this question. The user is asking about the direction the wind would have to come from to spill the liquid in the front glass. Looking at the position of the glass: it's on the left front edge of the coffee table. The table seems to have a flat surface. If the wind blows from the left side (from the viewer's left, which would be the front of the image if the viewer. Figure 13: Some examples from the benchmark (3D Geometry: Gravity Effects) and evaluation results.

Figure 14. 3D Geometry: Shape Projection. Question: If the hanging light casts a shadow on the wall, which outline might appear? Options: 1. A rectangle narrowing at the bottom From the right side 2. A vertical cylinder 3. A circle with fuzzy edges 4. A triangle pointing downward. Gold MCQ Answer: 2. Gold Open ended Answer: A vertical cylinder. MCQ Answers: GPT-5-mini: 4; Gemini-2.5-Flash: 1; InternVL3.5-72B: 1; Kimi-VL-A3B-Thinking-2506: 1. Open-ended Answers: GPT-5-mini: You would see the pendant’s dome-shaped profile: a rounded semicircle or oval shadow with a thin vertical line or narrow neck above where the cord hangs. The shadow would fall on the wall to the right of the window, roughly over the nightstand. Gemini-2.5-Flash: The hanging light is a simple, cylindrical pendant. If it casts a shadow on the wall, the outline would likely be a circular or elliptical shape, depending on the angle of the light source relative to the wall. Given its position near the window, the shadow might also be influenced by ambient light or other objects in the room. InternVL3.5-72B: The hanging light fixture is likely to cast a shadow resembling its own shape onto the wall. Given its cylindrical form, the shadow would appear as a soft-edged, elongated circle or oval, depending on the angle of the light source relative to the wall. Kimi-VL-A3B-Thinking-2506: Okay, let's tackle this question. The user is asking which outline a hanging light would cast on the wall if it casts a shadow. The wall near the window is light-colored, so the shadow would be visible. Since the shade is cylindrical, the shadow might be a cylinder. But maybe the outline is of the lampshade's perimeter. Alternatively, if the light is hanging from a chain, the shadow could be more elongated. But based on typical light fixtures, a cylindrical shade would cast a circular or oval shadow. The answer should be the outline of the hanging light's shade. Some examples from the benchmark (3D Geometry: Shape Projection) and evaluation results.

Figure 15. 3D Geometry: Spatial Containment

Question: Look at the tall pot in the background with the wide mouth (top left). could the small jug from the center of the photo fit completely inside it?

Options

  1. Yes, easily with room to spare

  2. No, the small jug hand will stuck

  3. No, the small jug is too tall.

  4. Yes, but it would be a perfect fit with no room left

Gold MCQ Answer: 2

Gold Open ended Answer: No, the small jug hand will stuck

MCQ Answers:

  • GPT-5-mini: 1

  • Gemini-2.5-Flash: 1

  • InternVL3.5-72B: 1

  • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

  • GPT-5-mini: No — the tall pot's mouth appears too narrow compared with the small jug's widest part and its handle. The jug wouldn't pass through the opening to sit completely inside.

  • Gemini-2.5-Flash: No, the small jug from the center of the photo could not fit completely inside the tall pot in the background. While the tall pot has a wide mouth, the jug's overall height and width appear to be too large to be fully contained within the pot's interior. The jug's handle would also likely prevent it from fitting snugly.

  • InternVL3.5-72B: The tall pot in the background with the wide mouth appears large enough to potentially accommodate the small jug from the center of the photo. Considering the proportions and the wide opening of the tall pot, it is likely that the small jug could fit completely inside it.

  • Kimi-VL-A3B-Thinking-2506: Okay, let's see. The question is about whether the small jug in the center of the photo could fit completely inside the tall pot in the background with the wide mouth (top left). However, without explicit scale references, this conclusion relies on typical proportional relationships between similar pottery forms. Based on typical proportions, the small jug’s height is likely shorter than the tall pot’s, and its width seems narrow enough to fit.

Figure 15: Some examples from the benchmark (3D Geometry: Spatial Containment) and evaluation results.

Figure 16. Some examples from the benchmark (3D Geometry: Volume Comparison) and evaluation results.

Figure 17. Some examples from the benchmark (3D Geometry: Shape Projection) and evaluation results.

3D Geometry: Shape Projection

Question: If you look at the seating arrangement from above, what shape does it form?

Options

  1. A perfect square

  2. A round shape

  3. A rectangle

  4. An irregular shape

Gold MCQ Answer: 4

Gold Open ended Answer: An irregular shape

MCQ Answers:

  • GPT-5-mini: 4

  • Gemini-2.5-Flash: 4

  • InternVL3.5-72B: 4

  • Kimi-VL-A3B-Thinking-2506: 4

Open-ended Answers:

  • GPT-5-mini: Viewed from above, the seating is arranged in a U-shape (a horseshoe) around the central fire pit. The L-shaped sectional plus the two facing chairs create that U configuration.

  • Gemini-2.5-Flash: The seating arrangement, when viewed from above, primarily forms an L-shape with the large sectional sofa. This L-shape is complemented by two individual chairs placed opposite the open ends of the sectional. Together, the furniture creates a U-shaped configuration around the central fire pit, designed for communal gathering.

  • InternVL3.5-72B: From above, the seating arrangement forms a U-shape around the central fire pit. The sectional sofa curves around the pit, leaving an open side opposite the tall fence, creating a semi-enclosed space.

  • Kimi-VL-A3B-Thinking-2506: Okay, let's tackle this question. The user wants to know the shape formed by the seating arrangement when viewed from above. When viewed from above, the configuration clearly forms a right-angled L-shape, combining two parallel sections at a 90-degree angle. The symmetry of the layout and the placement of furniture around the fire pit reinforce this geometric form. The seating arrangement forms a right-angled L-shape when viewed from above.

Figure 18. Some examples from the benchmark (Spatial Navigation: Accessibility Constraints) and evaluation results.

Spatial Navigation: Accessibility Constraints

Question: The notebook at the leftmost corner of the table is inaccessible because of which object?

Options

  1. Chair

  2. Vase

  3. Table

  4. Mug with pens and pencil in it

Gold MCQ Answer: 4

Gold Open ended Answer: Mug with pens and pencil in it

MCQ Answers:

  • GPT-5-mini: 1

  • Gemini-2.5-Flash: 4

  • InternVL3.5-72B: 4

  • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

  • GPT-5-mini: The leftmost corner notebook is blocked by a wooden chair. The chair sits directly in front of the desk, preventing easy access to the notebook.

  • Gemini-2.5-Flash: The notebook at the leftmost corner of the table is inaccessible because of the chair. The chair is positioned directly in front of the notebook, blocking direct access to it. To reach the notebook, one would first need to move the chair out of the way.

  • InternVL3.5-72B: The notebook at the leftmost corner of the table is inaccessible because of the chair, which is positioned in front of it. The chair's backrest and seat block direct access to the notebook.

  • Kimi-VL-A3B-Thinking-2506: Alright, let's tackle this spatial reasoning question. So, the question is asking why the notebook at the leftmost corner of the table is inaccessible. The chair is positioned near the leftmost corner of the table, and given its placement, it's reasonable to conclude that the chair is blocking access to the notebook. The chair's presence in that space would make it difficult to reach the notebook from the left side without moving the chair first. The chair is the object that's making the notebook at the leftmost corner inaccessible.

Figure 19. Some examples from the benchmark (Spatial Navigation: Obstacle Avoidance) and evaluation results.

Figure 20. Some examples from the benchmark (Spatial Navigation: Pathway Existence) and evaluation results.

Spatial Navigation: Pathway Existence

Question: Which walkways are the safest to cross according to the picture?

Options

  1. Top and bottom

  2. Top and left

  3. Left and right

  4. Bottom and right

Gold MCQ Answer: 1

Gold Open ended Answer: Top and bottom

MCQ Answers:

  • GPT-5-mini: 2

  • Gemini-2.5-Flash: 3

  • InternVL3.5-72B: 1

  • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

  • GPT-5-mini: The marked white crosswalks at each corner of the intersection (the painted lines across the road) are the safest to use. They are signalized and where pedestrians are waiting, unlike cutting across mid-block or diagonally through the intersection.

  • Gemini-2.5-Flash: The walkways marked with white lines, commonly known as crosswalks, are the safest to cross. These designated areas indicate where pedestrians have the right-of-way or are expected to cross the street. In the image, several people are visible using these marked crosswalks, suggesting they are the intended and therefore safest routes for pedestrians.

  • InternVL3.5-72B: The crosswalks aligned with the pedestrian signals at the corners are the safest places to cross, as they are equipped with traffic signals ensuring vehicles are stopped. The designated pedestrian areas with painted lines across the streets are clearly marked for safety. Additionally, sidewalks provide a safe space for walking along the roads.

  • Kimi-VL-A3B-Thinking-2506: Okay, let's tackle this question. The user wants to know which walkways are the safest to cross based on the image provided. The safest walkways to cross, based on the image, are the marked crosswalks equipped with pedestrian signals. Specifically, the crosswalk at the bottom-left corner of the intersection is safest because it features a visible pedestrian signal indicating walk, reducing ambiguity with traffic flow.

Figure 21. Some examples from the benchmark (Spatial Navigation: Viewpoint Visibility) and evaluation results.

Figure 22. Some examples from the benchmark (Spatial Navigation: Spatial Sequence) and evaluation results.

Figure 23. Some examples from the benchmark (Size and Scale: Scale Consistency) and evaluation results.

Figure 24. Some examples from the benchmark (Size and Scale: Shadow-Size Projection) and evaluation results.

Size and Scale: Shadow-Size Projection

Question: If one light is rotated so that they are perpendicular to each other, how will the shadow look over the wall?

Options

  1. It will cover almost all of the wall

  2. The shadow will only take the size of the light frame as it's actual size

  3. There won't be any shadow

  4. Shadow will smaller than the actual light body

Gold MCQ Answer: 1

Gold Open ended Answer: It will cover almost all of the wall

MCQ Answers:

  • GPT-5-mini: 4

  • Gemini-2.5-Flash: 1

  • InternVL3.5-72B: 1

  • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

  • GPT-5-mini: If one lamp is rotated 90° so the two beams are perpendicular, you'll get two elongated oval patches of light on the wall at right angles to each other. Where the ovals overlap it will be brightest; outside them the wall will be darker, producing an L-shaped darker region and two perpendicular shadow lobes cast by the fixture.

  • Gemini-2.5-Flash: If one light is rotated to be perpendicular to the other, the shadow cast on the wall would change significantly. The current shadows are likely elongated and somewhat parallel due to the lights' current orientation. With one light rotated, the shadows would intersect at a more acute angle, creating a more complex and possibly darker overlapping shadow where the light beams cross.

  • InternVL3.5-72B: If one light is rotated so that the two lights are perpendicular to each other, the shadows on the wall will intersect at a right angle, creating intersecting patterns. The light beams will cast shadows in perpendicular directions, resulting in a cross-like effect where the shadows overlap. The intensity and shape of the shadows will depend on the orientation and angle of the lights relative to the wall.

  • Kimi-VL-A3B-Thinking-2506: Okay, let's try to figure out this question. The user is asking about rotating a light so that they're parpassword to each other and how the shadow on the wall would look. If they're further away, the shadow would be smaller.

Figure 25. Some examples from the benchmark (Size and Scale: Perspective Distortion) and evaluation results.

Figure 26. Size and Scale: Relative Size Comparison

Question: The ash-colored, thicker coiled spring will be surpassed by which coiled springs if all are kept in the standing position?

Options

  1. Right bottom corner

  2. Right top corner

  3. Left top corner

  4. Bottom mid spring

Gold MCQ Answer: 1

Gold Open ended Answer: Right bottom corner

MCQ Answers: • GPT-5-mini: 1 • Gemini-2.5-Flash: 4 • InternVL3.5-72B: 1 • Kimi-VL-A3B-Thinking-2506: 2

Open-ended Answers: • GPT-5-mini: The ash-colored thicker spring in the center will be shorter than the thin, taller silver spring just behind it (top-center) and the long, dark-gray spring at the right edge when all are stood upright. Those two coils extend above the ash-colored spring. • Gemini-2.5-Flash: The ash-colored, thicker coiled spring, located near the center-right of the image, would be surpassed by the longer, thinner coiled spring positioned in the bottom right corner. Additionally, the two smaller, conical springs in the upper left quadrant would also be taller if fully extended and standing. These springs vary in height and coil density, indicating different potential applications. • InternVL3.5-72B: The ash-colored, thicker coiled spring in the image will be surpassed in height by the two taller springs on its right. One of these taller springs is also ash-colored, while the other has a slightly darker hue. Both of these springs have more coils and are visibly longer when standing upright. • Kimi-VL-A3B-Thinking-2506: Okay, let's try to figure out this question. The image shows a bunch of different coiled springs. The question is asking which coiled springs would be surpassed by the ash-colored, thicker one if all are kept in the standing position The thinner, tightly coiled springs (smaller in diameter or fewer coils) would be surpassed in height by the ash-colored, thicker spring when all are kept upright.

Figure 26: Some examples from the benchmark (Size and Scale: Relative Size Comparison) and evaluation results.

Figure 27. Some examples from the benchmark (Size and Scale: Relative Size Comparison) and evaluation results.

Depth & Occlusion : Complete Occlusion Inference

Question: What is creating the striped shadow on the wall?

Gold Open ended Answer: Blinds

Gold MCQ Answer: 2

Options

  1. The sunflower

  2. Blinds

  3. Window

  4. Not applicable

MCQ Answers:

  • GPT-5-mini: 2

  • Gemini-2.5-Flash: 2

  • InternVL3.5-72B: 2

  • Kimi-VL-A3B-Thinking-2506: 2

Open-ended Answers:

  • GPT-5-mini: Sunlight streaming through window blinds (slatted shades) is casting the diagonal striped shadow on the wall. The slats break the light into parallel bands, producing the striped pattern.

  • Gemini-2.5-Flash: The striped shadow on the wall is created by sunlight passing through a window with horizontal blinds. The blinds block portions of the light, casting distinct parallel lines onto the wall. This effect is common when direct sunlight enters a room through slatted window coverings.

  • InternVL3.5-72B: The striped shadow on the wall is created by blinds or window slats. The sunlight is passing through the gaps between the horzontal blinds, casting parallel lines of light and shadow. This pattern is often seen when light is filtered through window coverings with narrow openings. The alignment and spacing of the lines suggest that the blinds are partially open, allowing the sun to project the shadow pattern onto the wall.

  • Kimi-VL-A3B-Thinking-2506: The striped shadow on the wall is created by sunlight passing through a set of window blinds or shutters with narrow, evenly spaced slats. The parallel lines of the shadow result from the angular interaction between the light source (likely the sun) and the rigid, horizontal slats of the blinds, casting consistent geometric patterns onto the wall. The uniformity and sharpness of the shadows suggest the light is direct and possibly filtered through the gaps between the slats.

Figure 28. Some examples from the benchmark (Depth and Occlusion: Complete Occlusion Inference) and evaluation results.

Figure 29. Some examples from the benchmark (Depth and Occlusion: Layering Order) and evaluation results.

Benchmark example for Depth & Occlusion: Partial OcclusionVector (PDF)

Depth & Occlusion : Partial Occlusion

Question: What is the color of the ball behind the white colored square shaped device?

Options

  1. Cyan

  2. Yellow

  3. Red

  4. Not applicable

Gold MCQ Answer: 1

Gold Open ended Answer: Cyan

MCQ Answers:

  • GPT-5-mini: 1

  • Gemini-2.5-Flash: 2

  • InternVL3.5-72B: 1

  • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

  • GPT-5-mini: The ball directly behind the white, square-shaped device is blue (a cyan/teal colored ball). It sits slightly to the left and behind the device.

  • Gemini-2.5-Flash: The ball directly behind the white-colored square-shaped device (GameBoy) is teal. There is also a yellow-green ball further to the right of the GameBoy.

  • InternVL3.5-72B: The ball behind the white, square shaped device is blue.

  • Kimi-VL-A3B-Thinking-2506: The color of the ball behind the white square-shaped device is blue.

Figure 30. Some examples from the benchmark (Depth and Occlusion: Partial Occlusion) and evaluation results.

Figure 31. Depth & Occlusion : Reflective Surfaces

Question: Which object is the shiniest among all the objects in the image?

Options

  1. Kettle

  2. Tea glass

  3. Table clock

  4. Lamp chimney

Gold MCQ Answer: 4

Gold Open ended Answer: Lamp chimney

MCQ Answers: • GPT-5-mini: 1 • Gemini-2.5-Flash: 4 • InternVL3.5-72B: 1 • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers: • GPT-5-mini: The frontmost object is a small orange street-cleaning cart with a blue plastic bin mounted on it. A broom is attached to the cart and it sits on the pavement in the foreground. • Gemini-2.5-Flash: The frontmost object in the picture is the yellow cart with a blue crate on top, which a man in a yellow and black suit is sitting on. Its wheels are closest to the viewer, indicating its position in the foreground. The people sitting on the bench behind the cart are further back in the scene. • InternVL3.5-72B: The frontmost object in the image is a cleaning cart with wheels. It has a blue container attached to it and a broom standing upright. There's also a noticeable yellow brush or shovel handle next to the container. The cart appears to be positioned on a sidewalk in an urban environment. • Kimi-VL-A3B-Thinking-2506: The perspective is from the ground level, looking along the sidewalk. The cart is positioned in the foreground, closest to the viewer. The frontmost object is a high-visibility work cart.

Figure 31: Some examples from the benchmark (Depth and Occlusion: Reflective Surfaces) and evaluation results.

Figure 32. Some examples from the benchmark (Depth and Occlusion: Transparency Effects) and evaluation results.

Figure 33. Some examples from the benchmark (Orientation: Cardinal Direction) and evaluation results.

Figure 34. Some examples from the benchmark (Orientation: Facing Direction) and evaluation results.

Figure 35. Some examples from the benchmark (Orientation: Object Rotation) and evaluation results.

Figure 36. Some examples from the benchmark (Orientation: Stacking Orientation) and evaluation results.

Figure 37. Some examples from the benchmark (Orientation: Tool Handedness) and evaluation results.

Representative Benchmark Samples Across All Sub-Categories

Relative Positioning : Alignment Patterns

Question: Which of the objects is the closest to the centerline of the image?

Options

  1. Rice

  2. Nut

  3. Avocado

  4. Berries

Gold MCQ Answer: 4

Gold Open ended Answer: Berries

MCQ Answers:

  • GPT-5-mini: 4

  • Gemini-2.5-Flash: 4

  • InternVL3.5-72B: 4

  • Kimi-VL-A3B-Thinking-2506: 3

Open-ended Answers:

  • GPT-5-mini: The red bowl of berries (raspberries and blueberries) is closest to the image's centerline. It sits almost directly in the middle of the composition, slightly left of the exact center..

  • Gemini-2.5-Flash: The red bowl containing raspberries and blueberries is the closest object to the centerline of the image. Its position is almost perfectly centered horizontally, and it is located in the upper-middle section of the image vertically. While other objects like the avocado halves are also quite central, the red bowl's overall placement makes it the most aligned with the image's central axis.

  • InternVL3.5-72B: In the image, the avocado halves are the closest to the centerline. One avocado is placed near the top center, and the other is slightly below it to the right, straddling the vertical centerline of the image.

  • Kimi-VL-A3B-Thinking-2506: The bowl of mixed berries is closest to the centerline.

Figure 38. Some examples from the benchmark (Relative Positioning: Alignment Patterns) and evaluation results.

Figure 39. Relative Positioning : Betweenness Relationships

Question: What is the object between the building and the street lamp?

Options

  1. A monument of various statues

  2. Statue

  3. Tree

  4. Humans

Gold MCQ Answer: 1

Gold Open ended Answer: A monument of various statues

MCQ Answers:

  • GPT-5-mini: 1

  • Gemini-2.5-Flash: 1

  • InternVL3.5-72B: 1

  • Kimi-VL-A3B-Thinking-2506: 1

Open-ended Answers:

  • GPT-5-mini: Between the building and the street lamp stands a stone statue of a person. It appears to be one of the sculpted figures that are part of the fountain in the foreground.

  • Gemini-2.5-Flash: The object located between the building and the street lamp is a large, ornate monument. This monument features multiple levels adorned with various statues and intricate carvings. At its very top, there is a prominent statue of a person.

  • InternVL3.5-72B: The object between the building and the street lamp is a detailed stone monument or fountain. It features several statues, including a prominent figure standing on top and additional figures around its structure. The monument is ornate, with decorative elements and possibly bas-reliefs on its sides, and it appears to be situated in an open, possibly public area.

  • Kimi-VL-A3B-Thinking-2506: The object between the building and the street lamp is a detailed stone monument or fountain. The monument is ornate, with decorative elements and possibly bas-reliefs on its sides, and it appears to be situated in an open, possibly public area.

Some examples from the benchmark (Relative Positioning: Betweenness Relationships) and evaluation results.

Figure 40. Relative Positioning : Corner/Angle Positioning

Question: What is in the top left corner of the plant?

Options

  1. A lobster

  2. A crab

  3. An octopus

  4. Nothing

Gold MCQ Answer: 2

Gold Open ended Answer: A crab

MCQ Answers:

  • GPT-5-mini: 2

  • Gemini-2.5-Flash: 2

  • InternVL3.5-72B: 2

  • Kimi-VL-A3B-Thinking-2506: B

Open-ended Answers:

  • GPT-5-mini: A crab is in the top left corner of the plant, perched on the rim of the fishbowl. It’s hanging over the water with its claws visible.

  • Gemini-2.5-Flash: The plant in the image is a type of seaweed or coral, and it is contained within a glass bowl. In the top left corner of this plant, there is a branch of the plant extending upwards and slightly to the left. This branch is a dark, greenish-brown color.

  • InternVL3.5-72B: In the top left corner of the plant, there is a small crab that is perched on edge of the bowl, outside the water. It is gripping the rim with its legs, and its body is partially inside the bowl. The background shows the open water, providing contrast to the scene.

  • Kimi-VL-A3B-Thinking-2506: The answer is the crab

caption: Some examples from the benchmark (Relative Positioning: Corner/Angle Positioning) and evaluation results.

Figure 41. Some examples from the benchmark (Relative Positioning: Directional Relations) and evaluation results.

Figure 42. Some examples from the benchmark (Relative Positioning: Proximity Gradients) and evaluation results.

Statistical Robustness and Dataset Stability

To ensure that the benchmark results are not artifacts of random initialization or dataset noise, we evaluate both model robustness across runs and dataset stability across repeated trials. Such analyses are essential for small to medium scale benchmarks, where minor fluctuations can lead to misleading fine-grained conclusions [19, 20].

Let NN denote the number of items questions in the benchmark and R=3R = 3 the number of independent runs of the same model. Each run uses a distinct random seed while preserving all other hyperparameters.

Model Robustness

For checking model evaluation robustness, we perform several statistical studies, as described below.

Multiple Run Averages and Deviations

Let ai,r∈{0,1}a_{i,r} \in\{0,1\} denote the correctness of item i∈{1,…,N}i \in\{1,\ldots,N\} in run r∈{1,2,3}r \in\{1,2,3\}. The per-item average accuracy is:

aˉi=1R∑r=1Rai,r(1)\bar{a}_i = \frac{1}{R}\sum_{r=1}^{R} a_{i,r} \tag*{(1)}

and its corresponding standard deviation is:

σi=1R−1∑r=1R(ai,r−aˉi)2(2)\sigma_i = \sqrt{\frac{1}{R-1}\sum_{r=1}^{R}(a_{i,r}-\bar{a}_i)^2} \tag*{(2)}

The overall mean accuracy per run is:

Ar=1N∑i=1Nai,r(3)A_r = \frac{1}{N}\sum_{i=1}^{N}a_{i,r} \tag*{(3)}

and its standard deviation across runs is:

σA=1R−1∑r=1R(Ar−Aˉ)2(4)\sigma_A = \sqrt{\frac{1}{R-1}\sum_{r=1}^{R}(A_r-\bar{A})^2} \tag*{(4)}

where Aˉ=1R∑r=1RAr\bar{A} = \frac{1}{R}\sum_{r=1}^{R} A_r is the overall mean accuracy across all runs.

Intra-Class Correlation (ICC)

To quantify agreement between runs, we compute the Intra-Class Correlation coefficient (Shrout & Fleiss, 1979), using a two-way mixed-effects model:

ICC⁡(3,k)=σbetween2σbetween2+σwithin2k(5)\operatorname{ICC}(3,k) = \frac{\sigma_{\mathrm{between}}^2}{\sigma_{\mathrm{between}}^2 + \frac{\sigma_{\mathrm{within}}^2}{k}} \tag*{(5)}

where σbetween2\sigma_{\mathrm{between}}^2 is the variance between items and σwithin2\sigma_{\mathrm{within}}^2 is the residual variance across runs. ICC values above 0.75 indicate good reliability, while values above 0.9 imply excellent consistency (Koo & Li, 2016).

Dataset Stability and Internal Consistency

We also analyze the stability of the dataset itself, whether question difficulty remains consistent across runs.

Resampling Study

To further assess robustness, we perform a resampling analysis. From the total of C=30C = 30 subcategories, we randomly sample subsets of size S∈{20,25}S \in\{20, 25\} for each sub-category from each of the R=3R = 3 runs, across both models and evaluation types (MCQ and open-ended).

For each run rr and subset size SS, the accuracy is computed as:

Ar,S=1S∑i=1Sai,r,S,(6)A_{r,S} = \frac{1}{S}\sum_{i=1}^{S} a_{i,r,S}, \tag*{(6)}

where ai,r,Sa_{i,r,S} denotes the accuracy for subcategory ii in run rr under subset size SS.

The mean accuracy across runs for each subset size is:

Aˉ∗S=1R∑r=1RAr,S,(7)\bar{A}_{*S} = \frac{1}{R}\sum_{r=1}^{R} A_{r,S}, \tag*{(7)}

and the corresponding standard deviation:

σS=1R−1∑r=1R(Ar,S−AˉS)2.(8)\sigma_S = \sqrt{\frac{1}{R-1}\sum_{r=1}^{R}(A_{r,S}-\bar{A}_S)^2}. \tag*{(8)}

As a complementary check, the Wilcoxon signed-rank test assesses median differences between paired samples. Let dr=Ar,20−Ar,25d_r = A_{r,20} - A_{r,25} be the difference in accuracies for run rr, excluding ties (dr≠0d_r \ne0). The test statistic is computed as:

W=min⁡(W+,W−),(9)W = \min(W^{+}, W^{-}), \tag*{(9)}

where W+W^{+} and W−W^{-} are the sums of signed ranks of positive and negative differences, respectively. The standardized zz-score is given by:

z=W−n(n+1)4n(n+1)(2n+1)24,(10)z = \frac{W-\frac{n(n+1)}{4}}{\sqrt{\frac{n(n+1)(2n+1)}{24}}}, \tag*{(10)}

where nn is the number of nonzero pairs.

If the resulting pp-value satisfies p>0.05p > 0.05, we again fail to reject H0H_0, confirming that the accuracies from 20- and 25-sample subsets are statistically indistinguishable.

Item-Level Consistency

For each item ii, its difficulty can be defined as di=1−aˉ∗id_i = 1 - \bar{a}_{*i}. If item difficulty is consistent, the correlation between runs should be high:

ρ∗r1,r2=Cov⁡(a∗,r1,a∗,r2)σr1σr2(11)\rho_{*r_1,r_2} = \frac{\operatorname{Cov}(a_{*,r_1}, a_{*,r_2})}{\sigma_{r_1}\sigma_{r_2}} \tag*{(11)}

A high mean pairwise correlation (ρˉ>0.8\bar{\rho} > 0.8) suggests stable item difficulty and reliable dataset behavior.

Cronbach’s Alpha

To measure overall internal consistency, we compute Cronbach’s alpha [16]:

α=RR−1(1−∑r=1Rsr2sT2)(12)\alpha= \frac{R}{R-1}\left(1-\frac{\sum_{r=1}^{R}s_r^2}{s_T^2}\right) \tag*{(12)}

where sr2s_r^2 is the variance of run rr and sT2s_T^2 is the variance of the total scores aggregated across runs. Values of α≥0.9\alpha\ge0.9 indicate excellent internal consistency (Tavakol & Dennick, 2011).

Empirical Results

We perform the analysis on two models: one large proprietary model, Gemini-2.5-Flash, and one small open-source model, Qwen-2.5-vl-7b-Instruct, thereby covering both large- and small-scale models as well as proprietary and open-source paradigms. We also conduct all tests in both multiple-choice (MCQ) and open-ended evaluation formats (treated as binary: correct or incorrect). For statistical analysis, we used the following Python packages and functions: scipy.stats2 for wilcoxon test; pingouin3 for intraclass correlation and cronbach alpha; and numpy4’s numpy.corrcoef for correlations. (Virtanen et al., 2020) (Vallat, 2018) [27]

Model Evaluation Robustness

Multiple Run Averages and Deviations

We analyzed the stability of model accuracy across R=3R = 3 independent runs. The standard deviation of accuracy (σA\sigma_A) across runs is negligible (<0.6%< 0.6\%), indicating that the reported scores are precise estimates unaffected by random seeding noise.

ModelFormatMean Accuracy (Aˉ\bar{A})Std Dev (σA\sigma_A)
Qwen-2.5-vl-7bMCQ0.40790.0021
Gemini-2.5-FlashMCQ0.51930.0054
Qwen-2.5-vl-7bOpen0.19360.0047
Gemini-2.5-FlashOpen0.31240.0042

Table 26. Model Accuracy and Variance across 3 Independent Runs.

The consistently low standard deviations across all models and evaluation formats indicate that performance estimates are highly stable and reproducible, with minimal sensitivity to stochastic run-to-run variation. This stability suggests that observed accuracy differences meaningfully reflect underlying model capability rather than noise introduced by random initialization or decoding. Notably, although open-ended evaluation exhibits slightly higher variance than MCQ, the absolute magnitude remains negligible, confirming that free-form generation is also robustly assessed under the current protocol. These findings imply that the dataset size and scoring methodology are sufficient to yield convergent accuracy estimates without requiring extensive repeated runs. Consequently, single-run evaluations are likely to be reliable for large-scale comparisons, while multi-run analyses primarily serve to validate robustness. Overall, this strengthens the empirical credibility of the reported results and supports confident cross-model and cross-format performance interpretation.

Intra-Class Correlation (ICC)

We computed ICC⁡(3,k)\operatorname{ICC}(3,k) to quantify the reliability of the scoring mechanism.

ModelFormatICC(3,k)Interpretation
Qwen-2.5-vl-7bMCQ0.988Excellent
Gemini-2.5-FlashMCQ0.990Excellent
Qwen-2.5-vl-7bOpen0.983Excellent
Gemini-2.5-FlashOpen0.981Excellent

Table 27. Intra-Class Correlation (ICC) Reliability Scores.

The consistently high ICC(3,k) scores demonstrate that the evaluation exhibits near-perfect reliability across independent runs, regardless of model size or response format. This indicates that item-level difficulty ordering is preserved, and that performance fluctuations do not alter relative rankings among samples. Such strong agreement suggests that the scoring pipeline is internally consistent and not sensitive to stochastic variation in model outputs. Importantly, this rules out random guessing as a dominant factor, even in open-ended generation, where variability is typically higher. From an evaluation standpoint, these results imply that the benchmark provides a dependable signal for assessing model capability. Consequently, longitudinal comparisons and fine-grained ablations can be conducted with high confidence in measurement stability.

Overall, the combined evidence from multiple-run variance analysis and ICC reliability demonstrates that the evaluation is highly robust to stochastic effects. Both absolute performance (low standard deviation) and relative performance (near-perfect ICC) remain stable across independent runs, confirming that results are not driven by random seeding or decoding noise. This robustness ensures that observed performance gaps across models and formats reflect genuine capability differences. As a result, the evaluation framework provides a reliable and reproducible basis for comparative analysis and benchmarking.

Dataset Stability and Internal Consistency

Resampling Study To ensure statistical reliability, this resampling procedure is repeated over 1,000 independent trials, each time drawing new random subsets of size S=20S = 20 and S=25S = 25 from the available subcategories. Repeating the test at this scale stabilizes the distribution of accuracy differences and pp-values, ensuring that conclusions are not driven by a particular random draw. The consistent failure to reject H0H_0 across all trials therefore provides strong evidence that the observed robustness is systematic rather than incidental.

ModelFormatMean pp-value% Trials p>0.05p > 0.05Conclusion
Qwen-2.5-vl-7bMCQ0.290100%No Sig. Diff.
Gemini-2.5-FlashMCQ0.285100%No Sig. Diff.
Qwen-2.5-vl-7bOpen0.294100%No Sig. Diff.
Gemini-2.5-FlashOpen0.294100%No Sig. Diff.

Table 28. Resampling Study (1,000 Trials): Wilcoxon Signed-Rank Test (S=20S = 20 vs. S=25S = 25).

The resampling results provide strong empirical evidence that accuracy estimates have already converged at relatively small subcategory sizes. The consistently high mean (p)-values and the absence of statistically significant differences across all 1,000 trials indicate that performance metrics are insensitive to moderate reductions in sample size. This stability holds uniformly across models and evaluation formats, suggesting that neither model scale nor output structure introduces additional sampling uncertainty. Importantly, these findings imply that the variance in accuracy is dominated by model behavior rather than data sparsity at the subcategory level. As a result, the current subcategory size (≈50\approx50) can be considered statistically redundant, offering a substantial safety margin beyond the minimum required for reliable estimation. This supports the validity of the benchmark design and suggests that future extensions could reallocate samples across categories without compromising statistical robustness.

Item-Level Consistency and Cronbach’s Alpha We measured the mean pairwise Pearson correlation (ρˉ\bar{\rho}) between runs and Cronbach’s Alpha to assess internal consistency.

ModelFormatMean Pairwise ρˉ\bar{\rho}Cronbach’s α\alpha
Qwen-2.5-vl-7bMCQ0.9660.988
Gemini-2.5-FlashMCQ0.9720.990
Qwen-2.5-vl-7bOpen0.9500.983
Gemini-2.5-FlashOpen0.9460.981

Table 29. Item-Level Consistency Metrics.

The extremely high mean pairwise correlations (>0.94> 0.94) demonstrate strong item-level stability, indicating that the relative difficulty of individual questions is preserved across independent runs. This consistency shows that model responses are systematically structured rather than subject to random variation, even under stochastic decoding. The similarly high Cronbach’s α\alpha values (>0.9> 0.9) further confirm that the dataset exhibits excellent internal coherence, with items contributing reliably to the overall accuracy measure. Notably, the slight reductions observed in open-ended formats are expected given their greater expressive flexibility, yet remain well within the range of excellent reliability. Together, these results indicate that the benchmark functions as a unified and internally consistent measurement instrument. Consequently, model performance differences can be confidently attributed to true capability variation rather than instability at the item level.

Overall Discussion

Our expanded statistical testing shows our evaluiton and datset is robust ICC and Cronbach’s α\alpha scores consistently exceed 0.98, and the standard deviation of accuracy across runs is negligible. Furthermore, the resampling study confirms that even smaller subsets produce statistically indistinguishable results from the full set, proving that our subcategory sizes (∼50\sim50) are sufficient for robust conclusions. These results demonstrate that the granular analysis in the paper reflects genuine model capability differences, not statistical noise.

Our expanded statistical testing shows that the evaluation protocol and dataset are highly robust, with ICC and Cronbach’s α\alpha scores consistently exceeding 0.98, indicating near-perfect reliability and internal consistency. The standard deviation of accuracy across runs remains negligible (<0.6%< 0.6\%), confirming that reported model performance is stable and not influenced by stochastic effects such as random seeding or decoding variability. In addition, the resampling study demonstrates that even substantially smaller subsets (N=20N = 20 and N=25N = 25) yield statistically indistinguishable accuracy estimates compared to the full set, establishing that the chosen subcategory size (∼25\sim25) is well beyond the threshold required for convergence. Collectively, these results show that both absolute scores and relative performance rankings are stable across runs and sampling regimes. This statistical robustness ensures that fine-grained comparisons across models, formats, and subcategories are meaningful. Consequently, the granular analyses reported in the paper reflect genuine differences in model capability rather than artifacts of sampling variance or evaluation noise.

References

  1. [1]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreiro, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.arxiv.org/abs/2204.14198
  2. [2]Anthropic. Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet, October 2024. URL https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card-October-Addendum.pdf. Accessed: 2025-09-11.
  3. [3]Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129–19139, 2022.arxiv.org/abs/2112.10482
  4. [5]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://arxiv.org/abs/2502.13923.
  5. [6]Yoav Bar-Anan, Nira Liberman, and Yaacov Trope. The association between psychological distance and construal level: evidence from an implicit association test. Journal of experimental psychology: General, 135(4):609, 2006.
  6. [7]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Davis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ich­ter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspian Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong T. Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023.arxiv.org/abs/2212.06817
  7. [8]Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. CoRR, abs/2406.13642, 2024.arxiv.org/abs/2406.13642
  8. [9]Zhonggang Cai, Yubo Wang, Qingping Sun, Ruisi Wang, Chenyang Gu, Wangj Yin, Zhiqian Lin, Zhitao Yang, Chen Wei, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Jiaqi Li, Xiangyu Fan, Hanming Deng, Lewei Lu, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, and Lei Yang. Has gpt-5 achieved spatial intelligence? an empirical study, 2025. URL https://arxiv.org/abs/2508.13142.
  9. [12]Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ich­ter, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 14455–14465. IEEE, 2024b.arxiv.org/abs/2401.12168
  10. [13]Shiqi Chen, Tongyao Zhu, Ru­ochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor G­eya, Junchian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for vlms? an attention mechanism perspective on focus areas. CoRR, abs/2503.01773, 2025.arxiv.org/abs/2503.01773
  11. [15]Open X.-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alexander Herzog, Alex Irpan, Alexander Khazatsky, Anant Raj, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Brian Ich­ter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Gregory Kahn, Hao Su, Haoshu Fang, Haochen Shi, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mordatch, Ilija Radosavovic, and et al. Open x-embodiment: Robotic learning datasets and RT-X models. CoRR, abs/2310.08864, 2023.
  12. [16]Lee J. Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297–334, September 1951. ISSN 1860-0980. doi: 10.1007/bf02310555. URL http://dx.doi.org/10.1007/BF02310555.
  13. [17]Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Niebner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In 2017 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2432–2443. IEEE Computer Society, 2017.
  14. [18]Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Thompson, Quan Vuong, Tianyu He, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In ICML, 2023.arxiv.org/abs/2303.03378
  15. [19]Chris Drummond. Replicability is not reproducibility: nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, volume 1, Montreal, Canada, 2009. National Research Council of Canada.
  16. [20]Chris Drummond and Nathalie Japkowicz. Warning: statistical benchmarking is addictive: kicking the habit in machine learning. Journal of Experimental and Theoretical Artificial Intelligence, 22(1):67–80, March 2010. ISSN 1362-3079. doi: 10.1080/09528130903010295. URL http://dx.doi.org/10.1080/09528130903010295.
  17. [21]Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Empspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Short Papers, Bangkok, Thailand, August 11-16, 2024, pp. 346–355. Association for Computational Linguistics, 2024.
  18. [22]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: multimodal large language models can see but not perceive. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXIII, volume 15081 of Lecture Notes in Computer Science, pp. 148–166. Springer, 2024.arxiv.org/abs/2404.12390
  19. [23]Howard E Gardner. Frames of mind: The theory of multiple intelligences. Basic books, 2011.
  20. [24]Gemma Team, Google DeepMind. Gemma 3 technical report, 2025. URL https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf. Accessed: 2025-09-11.
  21. [25]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Srivankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattaï, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Isham Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecan Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Junteng Jia, Kalyan Vasudean Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pashupuletti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gouguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shejfel, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Chipin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspregen, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailien Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratchandani, Prithish Vuyraj, Qian Liang, Rachad Alaao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayak, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  22. [26]Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023.arxiv.org/abs/2211.11559
  23. [27]Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357–362, September 2020. doi: 10.1038/s41586-020-2649-2. URL https://doi.org/10.1038/s41586-020-2649-2.
  24. [28]Md Tanzib Hosain, Salman Rahman, Md Kishor Morol, and Md Rizwan Parvez. Xolver: Multi-agent reasoning with holistic experience learning just like an olympiad team, 2025. URL https://arxiv.org/abs/2506.14234.
  25. [29]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685.
  26. [30]Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1416–1428, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.77. URL https://aclanthology.org/2024.acl-long.77/.DOI
  27. [31]Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022.arxiv.org/abs/2207.05608
  28. [32]Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In CoRL, 2024b.arxiv.org/abs/2409.01652
  29. [33]Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700–6709. Computer Vision Foundation / IEEE, 2019.arxiv.org/abs/1902.09506
  30. [34]Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. Omnispatial: Towards comprehensive spatial reasoning benchmark for vision language models, 2025. URL https://arxiv.org/abs/2506.03135.
  31. [35]Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016. URL https://arxiv.org/abs/1612.06890.
  32. [36]Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 9161–9175. Association for Computational Linguistics, 2023.

Paper details

vlnc:W_69vsaps4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents