betaFeedback

WorksComputer Science

CueClash: Frontier Vision-Language Models Judge 3-D Relations Close to the Picture Plane

Abstract

A vision-language model (VLM) can judge 3-D relations from the picture or from the ground layout. In photographs the two usually agree. We separate them with CueClash, rendered scenes in which the 2-D cue for each of five relations is right on exactly half the statements. The shortcut gap is accuracy when the cue is right minus accuracy when it is wrong. For Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash at low reasoning effort it is +18 to +37 percentage points. It is +7 to +29 without two relations whose cue-wrong items are confounded with borderline cases or occlusion. Given ground coordinates and each relation's definition, they answer 99 to 100% correctly. Pixel measurements with camera depths remove most of the gap for objects fitting through openings. The failure lies in the step from the picture to the layout. The maps that the models draw of the scene from above mostly keep the depth order but, like the picture, shorten depth. GPT-5.5's maps stay close to a copy of the picture. Opus 5.5's and Gemini's maps undo about 0.6 of this shortening. Gemini's then shrink all depths by a further uniform factor. Opus 5.5 follows the picture mainly on fits, and its maps correct depth about as much as human observers of real scenes do. The three models also lose accuracy on CV-Bench photographs where a reliable 2-D cue misleads. In two open models the layout information reaches the language model, but their maps lose much of it.

Introduction

Is the pink cube in Figure 1 narrow enough to pass between the purple cylinder and the orange sphere? In the picture it looks too wide, because the cube spans 132 pixels and the gap only 60. On the ground the gap is 1.85 units and the cube 1.12, so the cube fits with room to spare. The picture is wrong about the widths only because the cube stands nearer the camera than the gap does. Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash, which we call the frontier models, all say that it does not fit. Shown a to-scale map of the same scene from above, all three say that it does.

The same fits item read in the picture plane and on the ground plane

Figure 1. The same fits item read in the picture plane and on the ground plane. (a) In the image the gap spans 60 px and the nearer cube 132 px. (b) On the ground plane the gap is 1.85 units and the cube 1.12, so the cube fits. (c) Opus 5.5, GPT-5.5 and Gemini at low reasoning effort answer False from the picture and True from the ground plane. Given pixel measurements and camera depths, which nearly determine the answer, only Gemini still answers False. Only the input changes between the rows. Over the 600 medium items their pooled gaps are +21.3, +35.0 and +24.7 points from the picture and +4.0, +5.3 and +4.7 from the top-down map (Table 6).

Spatial benchmarks for VLMs ask many questions of this kind [37, 12, 52] (Tong et al. 2024a; Yang et al. 2025a), but their scores cannot tell whether an answer came from the picture or from the scene. In natural photographs the two rarely come apart. The taller silhouette usually belongs to the taller object, just as silhouettes that overlap usually belong to objects that touch. A model that reads relations straight off the image can then score about as well as one that recovers the layout [23]. The usual remedy, giving the model a depth map, coordinates or a bird’s-eye view, adds a second confound, because it changes what the model is told and the form in which it is told at the same time.

We study the question with CueClash, a benchmark built on a controlled cue conflict. The five relations are closer, taller, touching, between and fits (Table 1). For each relation we name the 2-D cue that a reader of the picture would use, such as comparing pixel widths for fits. We balance the statements so that this cue is right on exactly half of them. Items on which the cue gives the right answer are cue-right items, and the others are cue-wrong items (Figure 2). A model that only reads the cue then scores 50%, the same as a coin. A model’s shortcut gap is its accuracy on cue-right items minus its accuracy on cue-wrong items, in percentage points. The gap measures how far the answers follow the picture. The items come in three difficulty tiers, easy, medium and hard.

One cue-right and one cue-wrong true item per relation from the medium tier

Figure 2. One cue-right (top) and one cue-wrong (bottom) true item per relation from the medium tier. In the bottom row the named 2-D cue gives the wrong answer, which is what the shortcut gap tests. The cues are pixel area for closer, pixel height for taller, masks within 3 px for touching, the angle test on mask centroids for between and pixel gap against pixel width for fits.

relation3-D truth2-D cue (balanced)reject unless
closer (control)camera depth of A’s centre << B’sA covers more pixels than Bdepth ratio ≥1.15\ge1.15
tallerheight(A) >> height(B)pixel height of A >> Bheight ratio ≥1.25\ge1.25
touchingsurface distance d<0.02d < 0.02visible masks within 3 pxd<0.02d < 0.02 or d>0.5min⁡(sA,hA,sB,hB)d > 0.5\min(s_A,h_A,s_B,h_B)
betweenmax⁡(∠CAB,∠CBA)≤10∘\max(\angle CAB,\angle CBA) \le10^\circ on the groundsame test on visible-mask centroids (20∘20^\circ)≤10∘\le10^\circ or ≥30∘\ge30^\circ
fitsfootprint gap(B,C) ≥\ge width(A)pixel gap(B,C) ≥\ge pixel width(A)ratio ≥1.2\ge1.2 or ≤0.8\le0.8

Table 1. The five relations. Truth is read from the scene graph and the 2-D cue from the rendered view’s visible masks, and candidates inside the reject band are discarded. The relation closer is the control. ss is size, hh height. In between, C is the middle object. In fits, A is the object that passes between B and C. Formal definitions and hand-checked cases are in Appendix § 18.

We then give the models the same scene in other forms. The perspective form gives exact pixel measurements, optionally with camera depths, which describe the picture, while the ground-plane form gives a top-down map or ground coordinates, which describe the scene. A deterministic solver that reads exactly the text a model received shows how far each form determines the answer, so a gap that remains under a sufficient input can be attributed to the form of that input. In a last task the models draw the scene from above. We compare these drawn maps with the picture-plane copy, the layout obtained by reading positions straight off the image.

The design lets us ask five questions in turn. Do the answers follow the picture (RQ1)? If they do, is the information missing from the inputs (RQ2a), or the ability to reason about the relations (RQ2b)? What layout do the models recover from the picture (RQ3)? Does the gap appear in photographs, where the picture rarely misleads (RQ4)? In open models, does the information that the picture carries reach the language model (RQ5)?

The failure lies in the step from the picture to the layout. At low reasoning effort the frontier models follow the picture, with gaps of +18 to +37 points (RQ1, § 5.1). Without between and touching, whose cue-wrong items are confounded with borderline cases or occlusion, the gaps are +7 to +29. On the cue-wrong fits items GPT-5.5 falls below chance. High reasoning effort narrows the fits gap for Opus 5.5 and GPT-5.5 without closing it. Exact pixel measurements with camera depths close most of the fits gap, so most of that failure lies in getting from the picture to measurements and depths (RQ2a, § 5.2). The between errors persist, because the models apply the between rule to image-plane angles and read the word “between” loosely. Given ground coordinates and the definition of each relation, the same models answer 99 to 100% of items correctly (RQ2b, § 5.3).

The drawn maps mostly keep the depth order but compress depth relative to width (RQ3, § 5.4). On a constancy index from 0 for the picture-plane copy to 1 for the scene, GPT-5.5’s maps score −0.01 and 0.12 on the medium and hard tiers. Opus 5.5’s score 0.60 and 0.63, near the partial constancy reported for human observers of real scenes.

On CV-Bench photographs the frontier models show gaps of +10 to +22 points where the lower object in the frame is the farther one (RQ4, § 5.5). In Qwen3-VL-4B and Qwen3-VL-8B a linear probe trained on the true layout reads each object’s ground position from the language model’s internal states about as well as from exact pixel measurements (RQ5, § 5.6). The same models’ drawn maps keep only 0.25 and 0.31 of the true depth-to-width ratio.

This paper makes four contributions.

  • CueClash, a benchmark of rendered scenes that balances the 2-D cue against the truth for five relations on three difficulty tiers, with the shortcut gap as its measure (§ 3).

  • A way to locate the failure. The same scene is given in the perspective form and in the ground-plane form, and a solver shows how far each input determines the answer (§ 5.2 and § 5.3).

  • An account of the layout that the models recover. Drawn maps scored with a constancy index show that the frontier models keep most of the depth order but compress depth, each to a different degree (§ 5.4).

  • Two further checks. The gap appears on CV-Bench photographs where a 2-D cue misleads (§ 5.5). Probes of two open models on the rendered scenes show the picture’s information reaching the language model (§ 5.6).

Related work

Cue conflict and balanced evaluation. Cue-conflict stimuli set two cues against each other to show which one a network follows, as in the texture–shape study of Geirhos et al. [22]. Visual question answering has long balanced answers across paired images [26] (Zhang et al. 2016) and shifted answer priors between training and test [1, 38], while minimal pairs isolate one factor at a time [25] (Thrush et al. 2022). SpatialSense collected spatial relations adversarially against a baseline that reads the 2-D configuration (Yang et al., 2019). Among spatial benchmarks, 3DSRBench avoids questions that the shortcut “higher in 3-D is higher in 2-D” would answer [52]. We keep such items and balance them instead, so that the shortcut is right on half of every relation and its use can be measured.

Spatial benchmarks and depth cues. Benchmarks built from photographs judge relations such as front and behind without metric 3-D truth [46, 37, 21], while newer suites attach 3-D annotations at scale [10] (Tong et al. 2024a; Yang et al. 2025a).

Work on the cues themselves finds monocular depth information in vision encoders ([18, 15]) and size constancy in VLMs that depends strongly on model scale (Sun et al., 2026). VLMs also order depth near chance on real and rendered images ([47]) and tend to treat higher in the frame as farther ([57] (Zou and Marttinen 2026)). Height in the frame is the angular-declination cue of human distance perception, which is valid for objects on a common visible ground ([59]).

The closest designs set a 2-D cue against depth order. Min et al. [57] report an accuracy gap between items consistent with the cue that far objects sit higher in the frame and items that contradict it, which is the depth-order case of our shortcut gap. Zou and Marttinen (2026) set apparent size and height in the frame against depth on renders and photographs. They show that depth order stays decodable when the answer is wrong, and they correct some answers by writing the model’s own depth readout back into its object representations. Our closer edits in Section 5.6 repeat their result on our scenes. What is new here is the extension of the conflict from depth order to four relations that compare sizes, contacts and alignments across depth. We also add inputs that keep the information fixed while its form changes, each checked by a solver. The drawn maps are scored with the same statements. On photographs we extend the depth-order split of Min et al. [57] on CV-Bench to the frontier models and to relative distance (Appendix § 11 covers the remaining benchmarks).

Supplied inputs and maps. Depth helps VLMs more as text than as an image (Vo et al., 2026), depth-layered captions also help (Roy et al., 2025), and models given naively fused geometry tokens still lean on 2-D cues (Zhang et al., 2026a). Coordinates are hard to use without explicit association of objects and names ([12] (Yuan et al. 2026)). Bird’s-eye-view prompting is an established lever (Zhu et al., 2025; Qi et al., 2026), although MindCube finds that a ground-truth map helps only when the model reasons over it (Wang et al., 2026). These interventions change the information and its form together. Because our solver measures how far each input determines the answer, a gap that remains under a sufficient input can be attributed to the form in which the information arrives.

Probing. A linear probe shows what a representation makes available to a simple reader without showing that the model uses it ([2, 5]). Control tasks test how selective a probe is ([30]), and a reference reader of the input tells what the representation adds ([31]). In language models, linear probes recover truth and world coordinates ([27, 56]), sometimes beyond what the answers express (Orgad et al., 2025). In VLMs the language model often fails to use what the vision encoder provides ([20, 49]). Image information reaches the object and text tokens at identifiable layers ([58, 36] (Zhang et al. 2025b)). Attention to the wrong regions is associated with some spatial errors ([11]), and a latent map of the 3-D scene can be extracted and steered (Wang and Gao, 2026).

Our probes build on two studies. Min et al. [57] probe spatial representations in VLMs and Zou and Marttinen (2026) decode depth order in models whose answers are wrong, from object features averaged over each object’s mask. Our object features follow the construction of Zou and Marttinen (2026). Liao et al. [45] show that probes and steering can overstate what a VLM takes from the image. We probe the full ground-plane layout and compare it with a reader of the 2-D measurements and with the same model’s drawn map. We score all three with the same statements. We also compare a probe at the last token of the prompt with an input-only reader of the 2-D measurements.

The CueClash benchmark

CueClash consists of rendered scenes paired with true or false statements.

Scenes

Each scene holds four to eight convex primitives (cubes, spheres, cylinders and cones) standing on a grey ground plane, lit by one sun lamp and rendered at 768×768768 \times768 with Blender Cycles ([7]) from a perspective camera 8 to 14 units away. Every object has its own colour, so that “the red cube” names one object. Sizes are drawn independently of shape, which leaves familiar sizes uninformative. Following CLEVR, PTR and Super-CLEVR-3D ([35, 33] (Wang et al. 2023)), the truth of every statement is read from the scene graph, while the statements take the true/false format of NLVR (Suhr et al., 2017, 2019). Instance masks come from an analytic ray caster over the same primitives, which makes the 2-D cues exact (generator settings in Appendix § 10).

Relations, truth and cues

Each of the five relations in Table 1 has a 3-D truth computed from the scene graph and a 2-D cue computed from the visible masks. Candidates close to the decision threshold fall into a reject band and are discarded, so that every statement is clearly true or false in the scene geometry. Half of the statements are negated and each template has three paraphrases, which leaves the wording uninformative about the label. The default prompt does not state the thresholds.

The relation closer serves as a control, because its natural trap, that lower in the frame means nearer, is almost always right for objects standing on a ground plane seen from above. It agrees with the truth on all 480 closer items of the benchmark. We balance closer on pixel area instead, the size cue that Zou and Marttinen (2026) found models follow, and read a zero gap on it only as evidence that pixel area is not used.

Balancing

For each relation we draw scenes and enumerate statements until the four cells, true or false crossed with cue-right or cue-wrong, each hold exactly a quarter of the items, with at most two statements per scene. Several cue-wrong cells almost never arise from random placement, so the sampler is steered to build them. It may hide a contact behind a third object, or place a near gap seen side-on next to a wider object farther away. These constructions change which scenes are drawn but never the labels, and they are recorded so that the analysis can control for them. As a result, every cue-wrong touching item is an occlusion case. Each of three tiers (easy, medium and hard) has 600 items, with a second medium split drawn from an independent seed. The tiers differ in clutter, camera elevation, shadows and occlusion.

Premise checks. The cue scores exactly 50% on every split and is right or wrong on exactly the intended items, which a unit test verifies. Under random placement on the medium tier the cues of taller, touching, between and fits agree with the truth on 82–99% of candidate statements, so a model that leans on them would be behaving sensibly on natural images (Landy et al., [42]; Knill and Richards, [39]). Blender renders of per-object masks from the meshes match the analytic masks with a median per-object intersection over union (IoU) of 0.9998. Relabelling the 11 items whose cue flips under the renders moves no gap by more than 2.5 points (Appendix § 18).

Measurements

Models and prompts. Each item pairs one image with one statement, under a prompt that says the statement concerns the 3-D scene and asks for True or False. This default input, the picture and the statement only, is the image condition. Unparseable answers count as wrong. The frontier models, Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash, run at low reasoning effort on the image condition of every tier (Table 2) and on every medium-tier input. They are the only models run on every input, so the main analyses use them. Grok 4.7, Nova 2 Pro, Pixtral Large, Qwen3-VL-235B, Llama 4 Maverick and Gemma 3 27B run at the provider default on the inputs of Table 3. Grok 4.7 behaves like the frontier models on the image condition, and Figure 4 shows it beside them.

Table 2: Accuracy and shortcut gap (points, 95% CI) on the image condition, five relations pooled. Each tier has 600 items and one run per model. The frontier models ran at low reasoning effort and the other models at the provider default. Random is a random-answer baseline. The frontier models’ answers track the picture plane on every tier. The 2-D cue scores 50 / +100 by construction. The last column of each tier is the mean gap over closer, taller and fits, which leaves out between and touching.

modeleasy acceasy gap [95% CI]easy w/o between, touchingmedium accmedium gap [95% CI]medium w/o between, touchinghard acchard gap [95% CI]hard w/o between, touching
2-D cue50.0+100.0–50.0+100.0–50.0+100.0–
Random52.0+3.3 [−5, +11]–51.0−4.7 [−13, +3]–50.7−2.7 [−10, +5]–
Claude Opus 5.584.5+19.7 [+14, +25]+7.285.7+21.3 [+16, +27]+11.784.2+17.7 [+12, +23]+8.9
GPT-5.575.8+37.0 [+31, +43]+29.478.5+35.0 [+29, +41]+27.276.0+28.7 [+22, +35]+27.2
Gemini 3.6 Flash81.2+24.3 [+18, +30]+16.781.7+24.7 [+19, +30]+20.678.5+23.7 [+18, +30]+22.2
Pixtral Large60.5+11.0 [+3, +20]–60.5+13.7 [+6, +22]–56.8+15.7 [+8, +23]–
Qwen3-VL-235B66.8+26.3 [+19, +34]–69.8+23.7 [+16, +31]–69.0+24.7 [+18, +32]–
Llama 4 Maverick53.2+15.7 [+8, +23]–56.5+9.0 [+1, +16]–54.3+12.7 [+5, +20]–
Gemma 3 27B50.3+10.0 [+2, +18]–51.8+15.0 [+7, +23]–53.7+11.3 [+4, +19]–
Grok 4.772.8+31.0 [+25, +38]+25.676.5+33.0 [+27, +39]+26.176.3+31.3 [+25, +38]+28.3
Nova 2 Pro60.2+13.7 [+6, +22]–62.5+26.3 [+19, +34]–63.8+17.0 [+9, +24]–

Table 2.

Table 3: Accuracy (%) / shortcut gap (points) under each input on the medium tier, one run on 600 items. The ground-plane form (top-down map, coordinates) brings the frontier models to gaps of +4 to +6 and Grok 4.7 to +5 to +18, while the five others stay at low accuracy. Frontier models used low reasoning effort, others the provider default. Columns marked + add that input to the image. Text only and coordinates only drop it. Camera depth and ground coordinates are text. The second view uses the 465 items with the same truth in both views and its own camera’s cue (Appendix Table 32). Appendix Table 24 gives frontier and Pixtral Large CIs.

modelimage+ CoTtext only+ pixel measurements+ depth map+ camera depth (text)+ top-down map+ coordinatescoordinates onlysecond view
2-D cue50 / +100–––––––––
Claude Opus 5.586 / +2186 / +1750 / −486 / +1587 / +1389 / +1893 / +496 / +596 / +686 / +3
GPT-5.578 / +3578 / +3350 / +180 / +3179 / +3089 / +1693 / +594 / +492 / +683 / +30
Gemini 3.6 Flash82 / +2581 / +2546 / +280 / +2480 / +2184 / +1892 / +593 / +593 / +483 / +9
Pixtral Large60 / +1462 / +2250 / +156 / +2360 / +964 / +358 / +959 / +1463 / +557 / +16
Qwen3-VL-235B70 / +2472 / +1849 / +364 / +3068 / +2268 / +1566 / +1768 / +1556 / +868 / +24
Llama 4 Maverick56 / +970 / +2752 / −155 / +2052 / +1158 / +051 / +265 / +355 / +058 / +1
Gemma 3 27B52 / +1558 / +2548 / −152 / +1552 / +1755 / +155 / +652 / +955 / +853 / +12
Grok 4.776 / +3374 / +3249 / +153 / +2868 / +3362 / +2480 / +1891 / +582 / +1075 / +21
Nova 2 Pro62 / +2670 / +2548 / −455 / +2258 / +2062 / +1657 / +1263 / +1257 / +564 / +18

Table 3.

Shortcut gap by relation for frontier models and Grok 4.7

Figure 4. The shortcut gap by relation (points, image condition) for the frontier models and Grok 4.7, one row per model and tier. The gap is concentrated in fits, between and touching. Each cell prints its gap, and darker red marks a larger positive gap. Each cell has 120 items and one run, the frontier models at low reasoning effort and Grok 4.7 at the provider default. The relation closer is the control.

We do not interpret the gaps of a model whose accuracy stays below 70% even with ground coordinates, since its failure to reason about the relations is then general. We call this the coordinates rule. All calls use provider-default sampling. Low reasoning effort is the low setting of each provider’s reasoning control. Appendix § 16 prints every prompt and Appendix Table 40 lists the model identifiers.

The shortcut gap. With truth YY, cue HH and answer MM, the gap is g=P(M=Y∣H=Y)−P(M=Y∣H≠Y)g = P(M=Y \mid H=Y) - P(M=Y \mid H\ne Y), reported in percentage points (points). Because the cue is right on half of the items, a gap of gg means that the answers agree with the cue on 50+g/2%50 + g/2\% of them. GPT-5.5’s medium gap of +35.0 points, for example, corresponds to agreement on 67.5%. The gap without between and touching is the mean of the closer, taller and fits gaps. Per-relation gaps rest on 120 items per tier, so their 95% intervals are about ±15 points.

A positive gap does not by itself prove reliance on the cue, since an observer with noisy but faithful geometry would also show one if cue-wrong items were harder for other reasons. We address this by adjusting the gap for item difficulty with a penalised logistic regression ([19]) and, more directly, with the input forms. An observer that corrects the picture only part of the way toward the scene, as human observers do ([41] (Thouless 1931)), would also show a positive gap. The gap measures how far the answers follow the picture, and § 5.4 measures how far each model’s layout corrects it. Intervals are 95% bootstrap intervals, resampled over scenes for paired comparisons ([17]). Appendix § 10 lists the covariates and the correction for multiple comparisons.

Input forms. On the medium tier we present the same scene in further inputs, listed in full in Appendix Table 24. Pixel measurements, the perspective form, describe the picture exactly. They list as text each object’s visible box, centroid, ground-contact pixel and visible area, optionally with its camera depth, the depth along the camera axis (Figure 3). The perspective form leaves the conversion from pixels and depths to ground distances to the model. One variant adds a depth map as a second image. The ground-plane form describes the scene instead, either as a to-scale top-down map or as ground coordinates.

The perspective form of the running example

Figure 3. The perspective form of the running example. Equal cells in the picture cover very different areas of ground. (a) The picture with a 64-pixel grid and the prompts’ visible box, ground-contact pixel and camera depth for the three objects in the statement. (b) The grid carried to the ground plane through the exact image-to-ground homography, the projective map from pixels to ground points. The two shaded cells are equal in the picture, but the far cell covers 2.94 square units of ground and the near cell 0.32. The gap, farther from the camera than the cube and seen at a slant, spans 60 px against the cube’s 132 px although it is wider on the ground.

Table 24: All inputs on the medium tier with 95% scene-bootstrap CIs. Second-view rows cover the 465 view-invariant items and use the second camera’s own cue.

modelinputnnacccue rightcue wronggap [95% CI]
Claude Opus 5.5image60085.796.375.0+21.3 [+16, +27]
Claude Opus 5.5image + CoT60086.595.078.0+17.0 [+12, +22]
Claude Opus 5.5image + tools (available)60086.394.378.3+16.0 [+11, +21]
Claude Opus 5.5image + tools (required first)60086.094.777.3+17.3 [+12, +23]
Claude Opus 5.5image + stated thresholds60093.299.087.3+11.7 [+8, +16]
Claude Opus 5.5image + numeric definitions only60089.096.082.0+14.0 [+9, +19]
Claude Opus 5.5image + reject bands only60092.398.386.3+12.0 [+8, +16]
Claude Opus 5.5image + numeric definitions + bands60094.298.789.7+9.0 [+5, +13]
Claude Opus 5.5image + verbal definitions60088.095.380.7+14.7 [+10, +19]
Claude Opus 5.5text only60050.348.352.3−4.0 [−11, +4]
Claude Opus 5.5image + exact pixel measurements (text)60086.594.079.0+15.0 [+10, +20]
Claude Opus 5.5pixel measurements, no image60086.392.380.3+12.0 [+7, +18]
Claude Opus 5.5image + pixel measurements + camera depth (text)60089.297.081.3+15.7 [+11, +21]
Claude Opus 5.5image + depth map (no legend)60086.793.380.0+13.3 [+8, +19]
Claude Opus 5.5image + depth map (metric legend)60086.594.778.3+16.3 [+11, +22]
Claude Opus 5.5image + camera depth (text)60088.797.779.7+18.0 [+14, +23]
Claude Opus 5.5image + top-down map60093.095.091.0+4.0 [−0, +8]
Claude Opus 5.5image + top-down map, no labels60092.595.789.3+6.3 [+2, +11]
Claude Opus 5.5image + ground coordinates (text)60095.798.093.3+4.7 [+1, +8]
Claude Opus 5.5ground coordinates only (no image)60095.598.792.3+6.3 [+3, +10]
Claude Opus 5.5ground coordinates only + verbal definitions60097.599.096.0+3.0 [+1, +5]
Claude Opus 5.5ground coordinates only + numeric definitions60099.099.099.0+0.0 [−2, +2]
Claude Opus 5.5second view (own-camera cue)46586.586.983.8+3.1 [−6, +12]
GPT-5.5image60078.596.061.0+35.0 [+29, +41]
GPT-5.5image + CoT60078.394.762.0+32.7 [+26, +39]
GPT-5.5image + stated thresholds60080.897.364.3+33.0 [+27, +39]
GPT-5.5image + numeric definitions only60078.794.762.7+32.0 [+25, +38]
GPT-5.5image + reject bands only60080.597.763.3+34.3 [+28, +40]
GPT-5.5image + numeric definitions + bands60077.796.758.7+38.0 [+32, +44]
GPT-5.5image + verbal definitions60077.093.760.3+33.3 [+27, +39]
GPT-5.5text only60050.350.750.0+0.7 [−7, +9]
GPT-5.5image + exact pixel measurements (text)60079.795.064.3+30.7 [+25, +37]
GPT-5.5pixel measurements, no image60081.591.072.0+19.0 [+13, +25]
GPT-5.5image + pixel measurements + camera depth (text)60085.897.074.7+22.3 [+17, +27]
GPT-5.5image + depth map (no legend)60078.793.763.7+30.0 [+24, +36]
GPT-5.5image + depth map (metric legend)60081.094.367.0+26.7 [+21, +32]
GPT-5.5image + camera depth (text)60088.896.781.0+15.7 [+11, +21]
GPT-5.5image + top-down map60093.095.790.3+5.3 [+1, +9]
GPT-5.5image + top-down map, no labels60092.796.389.0+7.3 [+3, +12]
GPT-5.5image + ground coordinates (text)60093.896.091.7+4.3 [+1, +8]
GPT-5.5ground coordinates only (no image)60092.595.789.3+6.3 [+2, +11]
GPT-5.5ground coordinates only + verbal definitions60095.596.394.7+1.7 [−1, +5]
GPT-5.5ground coordinates only + numeric definitions600100.0100.0100.0+0.0 [+0, +0]
GPT-5.5second view (own-camera cue)46583.287.757.4+30.3 [+19, +42]
Gemini 3.6 Flashimage60081.794.069.3+24.7 [+19, +30]
Gemini 3.6 Flashimage + CoT60080.793.368.0+25.3 [+19, +31]
Gemini 3.6 Flashimage + stated thresholds60084.095.073.0+22.0 [+17, +27]
Gemini 3.6 Flashimage + numeric definitions only60085.295.075.3+19.7 [+15, +25]
Gemini 3.6 Flashimage + reject bands only60084.595.373.7+21.7 [+16, +27]
Gemini 3.6 Flashimage + numeric definitions + bands60083.895.772.0+23.7 [+18, +30]
Gemini 3.6 Flashimage + verbal definitions60081.593.769.3+24.3 [+19, +30]
Gemini 3.6 Flashtext only60046.547.345.7+1.7 [−7, +9]
Gemini 3.6 Flashimage + exact pixel measurements (text)60080.292.368.0+24.3 [+18, +30]
Gemini 3.6 Flashpixel measurements, no image60079.787.771.7+16.0 [+10, +23]
Gemini 3.6 Flashimage + pixel measurements + camera depth (text)60082.293.371.0+22.3 [+17, +28]
Gemini 3.6 Flashimage + depth map (no legend)60079.590.069.0+21.0 [+15, +27]
Gemini 3.6 Flashimage + depth map (metric legend)60079.390.768.0+22.7 [+17, +29]
Gemini 3.6 Flashimage + camera depth (text)60084.393.375.3+18.0 [+12, +23]
Gemini 3.6 Flashimage + top-down map60092.094.389.7+4.7 [+0, +9]
Gemini 3.6 Flashimage + top-down map, no labels60091.596.087.0+9.0 [+4, +13]
Gemini 3.6 Flashimage + ground coordinates (text)60093.295.790.7+5.0 [+1, +9]
Gemini 3.6 Flashground coordinates only (no image)60092.894.791.0+3.7 [−0, +8]
Gemini 3.6 Flashground coordinates only + verbal definitions60095.095.394.7+0.7 [−3, +4]
Gemini 3.6 Flashground coordinates only + numeric definitions60099.599.399.7−0.3 [−2, +1]
Gemini 3.6 Flashsecond view (own-camera cue)46582.683.975.0+8.9 [−2, +20]
Pixtral Largeimage60060.567.353.7+13.7 [+6, +22]
Pixtral Largeimage + CoT60062.573.751.3+22.3 [+15, +30]
Pixtral Largeimage + verbal definitions60054.560.348.7+11.7 [+4, +20]
Pixtral Largetext only60050.050.749.3+1.3 [−7, +9]
Pixtral Largeimage + exact pixel measurements (text)60056.067.744.3+23.3 [+16, +31]
Pixtral Largeimage + depth map (no legend)60060.364.756.0+8.7 [+1, +16]
Pixtral Largeimage + camera depth (text)60063.765.362.0+3.3 [−4, +11]
Pixtral Largeimage + top-down map60057.862.353.3+9.0 [+2, +16]
Pixtral Largeimage + ground coordinates (text)60058.866.051.7+14.3 [+6, +22]

Table 24.

Other inputs. Further prompts state the numeric definition of each relation, so that an error cannot come from a looser reading of a word such as “between”. Other inputs change the prompt wording with stated thresholds or verbal definitions, or ask for chain-of-thought (CoT). High reasoning effort uses a 16,384-token reasoning budget. In the tool input Opus 5.5 gets a zoom tool and a Python interpreter over the pixels. A second camera view shows the scene from another angle. Appendix Table 24 lists each input, and Appendix Table 27 gives the effort runs.

low efforthigh effort (16k budget, single run)high effort (4k budget + retries)
modelaccgap [95% CI]accgap [95% CI]accgap [95% CI]
Claude Opus 5.585.7+21.3 [+16, +27]88.0+16.7 [+12, +22]87.8+15.0 [+10, +20]
GPT-5.578.5+35.0 [+29, +41]83.7+25.3 [+20, +31]82.3+30.0 [+24, +36]
Gemini 3.6 Flash81.7+24.7 [+19, +30]83.0+17.3 [+12, +23]80.3+23.3 [+17, +29]

Table 27. Accuracy (%) and gap (points) at low and high reasoning effort on the medium image condition. The middle columns are the high-effort run with a 16,384-token budget, which gives the high-effort results in the text. The last two columns are a second high-effort run with a 4,096-token budget, in which truncated responses were sampled again. The lower block gives the fits gap and the accuracy on the cue-wrong fits items at low and high effort (16,384-token run), with the 95% interval of the change in the gap.

Reference readers. For every text input, a deterministic solver reads exactly the text the model saw and applies the benchmark’s rule, which shows whether the input determines the answer. The size-constancy reader scales each pixel width and gap by its image distance below the horizon [59] (Sedgwick 1986) and makes no correction for foreshortening.

The measures that only the maps, the photographs and the open models need are defined where they are used, in § 5.4, § 5.5 and § 5.6.

Results

Do the answers follow the picture? (RQ1)

The answers of the frontier models follow the picture, most clearly on fits. On the image condition these models reach 76 to 86% accuracy with gaps of +18 to +37 points (Table 2). The pattern is plainest for GPT-5.5, which on the medium tier is 96.0% correct on cue-right items and 61.0% correct on cue-wrong items. The gaps hold on the independently generated split (+20 to +35) and move by about a point across three repeated runs (Appendix Table 36 and Table 26).

modelrepeatsaccuraciesgapsgap s.d.item-level agreement
Claude Opus 5.5385.7/86.5/86.2+21.3/+19.7/+19.01.292.8
GPT-5.5378.5/77.7/78.8+35.0/+36.0/+34.30.889.5
Gemini 3.6 Flash381.7/82.0/80.8+24.7/+22.7/+21.71.588.8

Table 26. Three runs of each frontier model on the medium image condition, with accuracy (%), gap (points), the standard deviation of the gap and item-level agreement (%) between runs.

modelimage seed 7image seed 11ground coordinates only seed 7ground coordinates only seed 11image + top-down map seed 7image + top-down map seed 11
Claude Opus 5.585.7 / +21.383.8 / +19.795.5 / +6.395.7 / +2.793.0 / +4.093.7 / +4.0
GPT-5.578.5 / +35.076.8 / +35.092.5 / +6.391.8 / +2.393.0 / +5.393.5 / +4.3
Gemini 3.6 Flash81.7 / +24.779.3 / +22.792.8 / +3.792.2 / +3.092.0 / +4.792.0 / +6.0

Table 36. Accuracy (%) / gap (points) on the first medium split (seed 7) and on a second medium split generated from seed 11 with the same settings.

Grok 4.7 behaves like the frontier models. The five other models score lower. Nova 2 Pro, Pixtral Large, Llama 4 Maverick and Gemma 3 27B score 51.8–62.5% on the medium tier and Qwen3-VL-235B scores 69.8%. None of them exceeds 68% even with ground coordinates, which points to a general failure to reason about the relations. By the coordinates rule of § 4 we do not interpret their gaps. No model beats chance on text alone (46 to 52%).

The gap is concentrated in the relations that compare sizes across depth or depend on what an occluder hides (Figure 4). It reaches its largest values for fits, at +18 to +68 points. On the medium tier GPT-5.5 answers the cue-wrong half of fits at 33%, worse than a coin, just as it does on the item of Figure 1. The gaps for between and touching follow at +37 to +47 and +13 to +50, while the control relation closer stays at −2 to +0. The pooled gap therefore leans on these two relations. For Opus 5.5 they supply 67–78% of the summed per-relation gaps. Without them the frontier gaps are +7 to +29 points, and +7 to +12 for Opus 5.5, which shows no taller gap (−3 to +7 across the three tiers and the second medium split). For Opus 5.5 the picture-plane errors that we can interpret are concentrated in fits.

The two cue-wrong halves of fits test different abilities (Table 4). Where the object does not fit but looks narrower than the gap, it stands farther from the camera than the gap, and an observer that scales pixel extents by distance answers correctly. The size-constancy reader of § 4 is right on 87 to 97% of these items. Where the object fits but looks too wide, the gap usually runs along the line of sight and is foreshortened. The same reader is then right on only 3 to 17%, which gives it a gap of +42 to +53 points. On the first cell Opus 5.5 scores 80 to 87% against the reader’s 87 to 97%. On the second it is well above the reader (50 to 73%). GPT-5.5 is far below the reader on the first cell (47 to 50%), a lack of size scaling would produce. It also scores low on the second (10 to 27%), where the reader fails too. Gemini is also below the reader on the first cell (33 to 63%).

tierreadercue-right fitscue-right does not fitcue-wrong fitscue-wrong does not fit
easysize-constancy reader971001797
easyClaude Opus 5.5100907380
easyGPT-5.593932750
easyGemini 3.6 Flash97937063
mediumsize-constancy reader97100393
mediumClaude Opus 5.51001005383
mediumGPT-5.5971002047
mediumGemini 3.6 Flash100905743
hardsize-constancy reader100100787
hardClaude Opus 5.5100975087
hardGPT-5.5931001047
hardGemini 3.6 Flash100974733

Table 4. Accuracy (%) on the four fits cells of each tier, split by whether the pixel-width cue is right and whether the object fits (30 items per cell, one run at low reasoning effort). The size-constancy reader of § 4 scales each pixel width and gap by its image distance below the horizon and makes no correction for foreshortening. It fails where the object fits but looks too wide, because the gap then runs along the line of sight. GPT-5.5 and Gemini fall short of it on the cue-wrong items where the object does not fit, the cell that tests size scaling.

If cue-wrong items were simply harder, holding difficulty fixed would remove the gap. For fits it does not. Adjusting for distance from the decision threshold, occlusion, pixel area, clutter, camera elevation and the construction of each scene leaves medium-tier fits gaps of +24 [+14, +34] for Opus 5.5, +56 [+42, +69] for GPT-5.5 and +29 [+15, +45] for Gemini. Every interval on the medium and hard tiers excludes zero, while on the easy tier those of Opus 5.5 and Gemini reach it (Appendix Table 35). The pooled gaps after adjustment are +10 to +29 points with every interval above zero (Appendix Table 30), and the fits gap survives on the items whose covariate values occur among both cue-right and cue-wrong items (Appendix Table 34). The relation between is the exception. In its false half the cue-wrong items are exactly the items near the reject band, so no adjustment can separate the cue from closeness to threshold. We treat between as confounded and fits as the case free of this confound (Appendix § 9).

High reasoning effort narrows the fits gap without removing it. It moves the pooled gap by −10 to −5 points (Appendix Table 27) but the fits gap by −22 to −15. Gemini’s change has an interval that reaches zero ([−32, +1]). Opus 5.5’s fits gap falls from +32 to +10 points, with a 95% interval of [−35, −8] for the change. GPT-5.5’s change has an interval of [−38, −3]. Its accuracy on the cue-wrong fits items rises from 33% to 57%, so its below-chance score holds only at low effort.

Is the information missing? (RQ2a)

The information is not missing from the inputs, since exact pixel measurements with camera depths close most of the fits gap and the between gap that remains comes from how the models use the numbers. If the picture simply did not carry enough information, an input that does carry it should remove the gap. Pixel measurements test this. Given exact pixel measurements as text and no image, the models score 80 to 86% (Appendix Table 24), close to their accuracy on the image condition, which shows that they read and use the numbers. A zoom tool and a Python interpreter over the pixels move Opus 5.5’s pooled gap by −5.3-5.3 [−9-9, −2-2], although the same model measures visible boxes accurately when that is all it is asked to do (median IoU 0.977).

Adding each object’s camera depth makes closer and between exactly solvable and fits solvable on 97% of items, as the solver confirms (Table 5). Table 6 gives the pooled gap under each input. Table 7 gives the fits and between gaps. With pixel measurements and camera depths the fits gaps of Opus 5.5, GPT-5.5 and Gemini fall from +32+32, +65+65 and +45+45 points to +10+10, +17+17 and +20+20. The between gaps stay at +28+28 to +37+37, although this input determines every between answer (Table 5).

input (text)closertallertouchingbetweenfits
visible-region centres (text)–––72†–
ground-contact pixels (text)–––94†–
pixel measurements (text)100†50†62†94†65†
pixel measurements + camera depth (text)10069†96†10097†
ground-contact pixels + homography (text)–––100–
pairwise surface distances (text)––100––
ground coordinates (text)100100100100100

Table 5. Solver accuracy (%) on the 120 medium items of each relation, which shows how far each text input determines the answer. With camera depths, pixel measurements determine closer and between exactly and fits nearly. The solver applies the benchmark’s rule to exactly the text in the prompt. Rows with a single entry are inputs given for that relation only (Appendix § 10). Visible-region centres are the mask centroids. The homography maps each pixel to its ground point. Surface distances are given for each pair of objects. †: the input lacks a needed quantity (object sizes, the ground mapping), so the solver uses the best simple estimate with thresholds fitted on the easy tier.

inputClaude Opus 5.5 gapClaude Opus 5.5 ∆ vs image [95% CI]GPT-5.5 gapGPT-5.5 ∆ vs image [95% CI]Gemini 3.6 Flash gapGemini 3.6 Flash ∆ vs image [95% CI]
image+21.3–+35.0–+24.7–
stated thresholds+11.7−9.7 [−14, −5]+33.0−2.0 [−7, +3]+22.0−2.7 [−7, +2]
verbal definitions (no bands)+14.7−6.7 [−10, −3]+33.3−1.7 [−7, +4]+24.3−0.3 [−5, +5]
pixel measurements (text)+15.0−6.3 [−11, −2]+30.7−4.3 [−10, +1]+24.3−0.3 [−6, +5]
pixel measurements, no image+12.0−9.3 [−14, −4]+19.0−16.0 [−22, −10]+16.0−8.7 [−16, −1]
pixel measurements + camera depth (text)+15.7−5.7 [−10, −2]+22.3−12.7 [−19, −6]+22.3−2.3 [−8, +4]
depth map+13.3−8.0 [−12, −4]+30.0−5.0 [−10, −0]+21.0−3.7 [−9, +1]
depth map, metric legend+16.3−5.0 [−9, −2]+26.7−8.3 [−13, −4]+22.7…
camera depth (text)+18.0−3.3 [−7, +0]+15.7−19.3 [−24, −14]+18.0−6.7 [−12, −2]
top-down map+4.0−17.3 [−22, −12]+5.3−29.7 [−36, −24]+4.7−20.0 [−26, −14]
top-down map, no labels+6.3−15.0 [−20, −10]+7.3−27.7 [−34, −22]+9.0−15.7 [−22, −10]
ground coordinates (text)+4.7−16.7 [−21, −12]+4.3−30.7 [−37, −25]+5.0−19.7 [−26, −14]
ground coordinates, no image+6.3−15.0 [−20, −10]+6.3−28.7 [−34, −23]+3.7−21.0 [−27, −15]
ground coordinates, no image, verbal definitions+3.0−18.3 [−23, −13]+1.7−33.3 [−39, −27]+0.7−24.0 [−30, −17]
ground coordinates, no image, numeric definitions+0.0−21.3 [−27, −16]+0.0−35.0 [−40, −29]−0.3−25.0 [−30, −19]

Table 6. Gap (points) under each input on the medium tier and its paired change against the image condition, 95% scene-bootstrap CI, one run per model on the 600 items at low reasoning effort. The two rows after the image row change only the prompt wording. Verbal definitions state each rule in words without the reject bands. The metric legend is a numeric colour bar on the depth map. The top-down map without labels drops the text labels on the footprints. The ground-plane form nearly removes the gap. With numeric definitions, ground coordinates give 99.0, 100.0 and 99.5% accuracy. Pixel measurements narrow the gap in the point estimates, with several intervals reaching zero.

inputClaude Opus 5.5 fitsClaude Opus 5.5 betweenGPT-5.5 fitsGPT-5.5 betweenGemini 3.6 Flash fitsGemini 3.6 Flash between
image+32+45+65+43+45+43
pixel measurements, with the image+15+33+68+30+42+37
pixel measurements, no image−3+23+42+18+18+15
pixel measurements + camera depths, with the image+10+33+17+28+20+37
ground coordinates, no image, no definitions+0+28+0+23−2+10
ground coordinates, no image, numeric definitions−2+0+0+0+0+0

Table 7. The fits and between gaps (points) under the inputs of Section 5.2 on the medium tier, 120 items per relation, one run per model at low reasoning effort. Pixel measurements and ground coordinates are given as text. Table 5 gives how far each input determines the answer.

What closes the fits gap differs by model. For GPT-5.5 it is the camera depths. For Opus 5.5 the pixel measurements already bring most of the drop. For Gemini removing the image does as much as adding depths (Table 7). Most of the failure therefore lies before the arithmetic, in getting from the picture to measurements and depths.

The pixel-measurement inputs come with the image. The same measurements without the image give smaller fits gaps for every model (Table 7). Part of the residual may be the picture overriding the text. Pixel measurements with camera depths also raise some gaps above the image condition (Opus 5.5 touching from +27 to +38, GPT-5.5 taller from +17 to +35, Gemini taller from +17 to +28, Gemini touching from +18 to +27). A depth map shown as a second image shifts the pooled gap by only −8 to −4 (Table 6).

The between residual shows what the models do with such numbers. Given ground-contact pixels and an exact angular rule for between, they apply the rule to image-plane angles. GPT-5.5’s answers agree with that image-plane reading on 99.2% of items. On this input they still score 92 to 94%, because the reject band leaves few items on which image-plane and ground-plane angles disagree. The models compute on the measurements as given and do not carry them onto the ground. Part of the residual is also the looser reading of the word. Ground coordinates without the numeric definition leave between gaps close to those under pixel measurements with camera depths for Opus 5.5 and GPT-5.5, though not for Gemini (Table 7). Stating the numeric definitions alongside the image moves the gap by −7 to −3 points (Appendix Table 14). Adding the reject bands lowers Opus 5.5’s pooled gap from +21.3 to +9.0 points. The bands add no geometry, although they reveal how the items were sampled.

Can the models reason on the ground plane? (RQ2b)

The models can apply the relations once the layout is given, because describing the scene on the ground nearly removes the gap. The top-down map narrows it to +4 to +5 points and ground coordinates without the image narrow it to +4 to +6 (Table 6). When the prompt also states each relation’s numeric definition (last row of Table 6), coordinates give 99 to 100% accuracy and no gap farther than 0.3 points from zero. This includes the contacts hidden behind other objects, which Opus 5.5 gets right on only 10% of items from the picture. The relations are within the models’ reach once the layout is given, which leaves the recovery of the layout as the step that fails.

What layout do the models recover? (RQ3)

The frontier models recover a layout that keeps most of the depth order but compresses depth toward the picture, by an amount that differs between models.

Task and measures. In a separate task the model sees only the image and returns every object’s ground position and footprint diameter as JSON, in a top-down frame of its choosing. We call the result its drawn map. We align each drawn map to the truth by a similarity transform, a shift, rotation and uniform scaling, and re-derive the statements from the aligned positions. The picture-plane copy, the ground-contact pixels read as top-down coordinates, is aligned in the same way. Each frontier model’s drawn map is also fed back to it as text, either as drawn or with its depth axis stretched by a factor estimated on the other half of the scenes. The gap of the statements re-derived from a map is its map gap.

A map’s depth-to-width ratio is the spread of its positions along the camera axis over their spread across it, relative to the truth. A ratio below 1 means that depth is compressed. The constancy index of Thouless (1931) is the median over scenes of ln⁡(m/c)/ln⁡(1/c)\ln(m/c) / \ln(1/c), with mm the map’s depth-to-width ratio and cc the copy’s. It is 0 for the picture-plane copy and 1 for the scene. It is the part of the picture’s compression that a typical map undoes, measured on a log scale. A second measure fits ln⁡m=a+bln⁡c\ln m = a + b \ln c over each model’s medium and hard scenes. The share corrected is 1−b1 - b, the part of the picture’s compression that the map undoes at any level of compression. The factor eae^a is any further uniform shrinking of depth, which lowers the index but not the share corrected. We also count the scenes in which the aligned map lies closer to the picture-plane copy than to the scene.

The drawn maps put most objects in the right depth order, with median rank correlations along the camera axis of at least 0.94 on the medium tier (Appendix § 9). They compress depth relative to width (Table 8). Statements about which object is closer, re-derived from the maps, are right on 93, 77 and 73% of the cue-wrong items on the medium tier for Opus 5.5, GPT-5.5 and Gemini. On the hard tier they are right on 72% for Opus 5.5. For GPT-5.5 and Gemini they are right on 38% and 48% of cue-wrong items, although the depth rank correlations of their maps stay at 0.94 and 0.90. On the medium tier GPT-5.5’s and Gemini’s maps keep 0.61 and 0.63 of the true depth-to-width ratio, close to the 0.61 of the picture-plane copy, and Opus 5.5’s maps keep 0.83.

modelscenes parsedRMSE (units, median)depth-to-width ratio [95% CI]constancy index [95% CI]share corrected% compressedcue-rightcue-wronggapclosertouchingbetweenfits
medium tier
Claude Opus 5.5304/3040.240.83 [0.80, 0.84]0.60 [0.58, 0.63]0.64888060+20−7+0+47+42
GPT-5.5304/3040.460.61 [0.60, 0.65]−0.01 [−0.07, 0.07]0.18877538+38+0+42+65+43
Gemini 3.6 Flash304/3040.440.63 [0.60, 0.65]0.07 [0.01, 0.12]0.58967443+30+3+17+55+47
Pixtral Large303/3040.860.62 [0.58, 0.65]––864833+15−8+23+33+15
Qwen3-VL-235B304/3040.780.38 [0.36, 0.39]––1005427+27−13+40+47+33
Llama 4 Maverick294/3040.900.56 [0.51, 0.59]––895332+22−10+32+37+28
Gemma 3 27B304/3040.890.66 [0.61, 0.69]––825340+12−7+12+17+28
Grok 4.7224/3040.390.75 [0.72, 0.79]0.37 [0.30, 0.45]0.60837053+18−3+4+45+23
Nova 2 Pro304/3040.860.40 [0.38, 0.44]––985230+23−5+25+48+23
hard tier
Claude Opus 5.5302/3020.430.64 [0.63, 0.66]0.63 [0.61, 0.65]0.64987350+23+2+13+35+42
GPT-5.5302/3020.780.35 [0.33, 0.37]0.12 [0.08, 0.15]0.18975725+32+8+40+55+23
Gemini 3.6 Flash301/3020.710.47 [0.46, 0.50]0.37 [0.35, 0.41]0.58996433+30+9+30+51+30
Pixtral Large302/3021.110.44 [0.41, 0.46]––944323+20+0+30+40+10
Qwen3-VL-235B301/3021.110.20 [0.19, 0.21]––1004413+31+4+48+60+10
Llama 4 Maverick283/3021.100.37 [0.34, 0.40]––964621+25+6+32+48+12
Gemma 3 27B301/3021.170.48 [0.44, 0.50]––944528+18+9+25+31+5
Grok 4.7206/3020.630.54 [0.52, 0.58]0.50 [0.43, 0.55]0.60876342+21−2+28+37+26
Nova 2 Pro302/3021.130.24 [0.22, 0.25]––1004421+23+5+33+43+12

Table 8. Drawn maps, one run per model (frontier models at low reasoning effort, others at provider default). All frontier maps compress depth (ratio below 1). The constancy index puts GPT-5.5 lowest, Opus 5.5 highest. § 5.4 defines the depth-to-width ratio (1 = undistorted), the constancy index and the share corrected (– if not computed, Appendix Tables 18 and 19). The picture-plane copy’s ratio is 0.61 (medium) and 0.30 (hard). RMSE is the median position error after alignment. Relation columns re-derive the statements from the map with true sizes, scoring those inside a reject band as wrong (accuracy in %, map gaps in points). Heights are not mapped, so taller is left out. % compressed counts scenes with a ratio below 1.

Figure 5 shows a scene in which every map narrows a gap below the width of an object that in fact fits through it. All three models answer False from the image and True from coordinates or the top-down map. Among the 60 medium scenes with a cue-wrong fits item and parsed maps from all three models, 5 have that item wrong from the image and from every map. The scene of Figure 5 is the only one of the five whose statement is true.

One scene drawn from above by each model

Figure 5. One scene drawn from above by each model. The statement “The yellow sphere would fit through the gap between the orange sphere and the pink cylinder” is true. (a) In the picture the gap is 16 px and the sphere 164 px. (b) On the ground the gap is 2.10 units and the sphere 1.67. (c) The picture-plane copy. (d–f) Each frontier model’s drawn map from the image alone. Every layout is aligned to the truth by a similarity transform and drawn with the true footprint sizes, with dashed circles at the true positions. Panel titles give each layout’s depth-to-width ratio and gap width in units. Each map compresses depth and narrows the gap below the sphere’s width.

The constancy index orders the frontier models the same way on both tiers, with GPT-5.5 lowest and Opus 5.5 highest. GPT-5.5’s maps score −0.01-0.01 [−0.07-0.07, 0.070.07] on the medium tier and 0.120.12 [0.080.08, 0.150.15] on the hard tier. Gemini’s score 0.070.07 [0.010.01, 0.120.12] and 0.370.37 [0.350.35, 0.410.41], and Opus 5.5’s 0.600.60 [0.580.58, 0.630.63] and 0.630.63 [0.610.61, 0.650.65] (Table 8). The picture-plane copy keeps half as much depth on the hard tier (copy ratios 0.610.61 and 0.300.30).

The fit of ln⁡m\ln m on ln⁡c\ln c separates three patterns (Appendix Table 19). Opus 5.5 corrects a fixed share of about 0.640.64 of the compression with no further bias (intercept −0.01-0.01 on the log scale). GPT-5.5 corrects only 0.180.18. Gemini corrects a share like that of Opus 5.5 (0.580.58) but shrinks every map in depth by a further factor of 0.780.78 (intercept −0.25-0.25 [−0.30-0.30, −0.19-0.19] on the log scale), which on the medium tier happens to land on the copy. Distances to the copy and to the scene also separate Opus 5.5 from the other two. After similarity alignment, GPT-5.5’s maps lie closer to the picture-plane copy than to the scene on 78%78\% and 92%92\% of medium and hard scenes, Gemini’s on 83%83\% and 88%88\%, and Opus 5.5’s on only 41%41\% and 34%34\%.

Human observers show a similar partial correction, known as phenomenal regression (Thouless, 1931). Monocular judgements of the shape of objects on the ground fall roughly midway between the retinal and the true values, an index near 0.50.5 by our reading of Figure 2 of Loomis and Philbeck [50]. Opus 5.5’s maps have a constancy index close to this value. GPT-5.5’s maps have a far lower index. People also compress depth relative to width in real scenes [51], and in pictures they recover depth only up to a scaling and shear [40, 41].

The maps reproduce the gap in aggregate but not item by item. On the medium tier, statements re-derived from them show gaps of +20+20 to +38+38. They agree with the answers given from the image on 7070–75%75\% of items (Appendix Table 37). On the cue-wrong fits items, though, the two are only weakly related (correlations of −0.18-0.18 to +0.23+0.23 on 60 items per model). Part of the map gap also comes from scoring as wrong the statements whose re-derived relation falls inside the reject band, which happens on 2929 to 38%38\% of cue-wrong items and 1818 to 23%23\% of cue-right ones. Without these items the map gaps are +14+14 to +36+36, and Opus 5.5’s between map gap falls from +47+47 to +3+3. The answers and the maps may be two separate outputs that each lean on the picture.

When a model reads its drawn map back as text, it answers as a solver reading that map would on 100%100\% of items. Stretching each map’s depth axis by a factor estimated on the other half of the scenes changes the fits gap by −17-17 to −10-10 points and the between gap by −2-2 to +3+3 (Appendix § 9). This suggests that uniform compression of depth explains part of the fits errors and none of the between errors.

Among the other models, Grok 4.7’s maps correct a share of 0.600.60 of the compression, close to Opus 5.5’s 0.640.64, but show a gap of only +18+18 against +33+33 for its answers. The five other models’ maps order depth less reliably, and the Qwen3-VL-235B maps run depth the wrong way (Appendix § 9). The frontier models recover most of the depth order but compress depth. GPT-5.5 stays close to the picture.

Do photographs show the same gap? (RQ4)

The frontier models lose accuracy on the CV-Bench photographs in which a 2-D cue misleads.

Data and cues. The 3-D subset of CV-Bench (Tong et al., 2024a) asks which of two marked objects is closer to the camera (Depth) and which of two objects is closer to a third (Distance), on 600 photographs per task from indoor, street and synthetic indoor scenes. Its items are not balanced against a cue, so we name the cues a reader of the picture would use and score the shortcut gap against each of them. For Depth these are the larger box and the lower box, whose bottom edge is lower in the frame. For Distance it is the smaller pixel distance from each object’s box centre to the centre of the reference box. The frontier models, Qwen3-VL-4B and Qwen3-VL-8B answer every item with one letter. We compare the correlation of the answers with each cue against the correlation of that cue with the truth (Brunswik, 1956). A cue that the answers follow more closely than it tracks the truth is used beyond its validity. As a check on the Depth cue, a small rendered tier puts objects on pedestals, so that height in the frame is wrong about depth on 16 of its 32 closer items (the elevated tier of Appendix § 15).

Natural photographs rarely set the picture against the scene, so a photographic benchmark can test the gap only on the few items where a cue misleads. Min et al. [57] split CV-Bench Depth by whether the farther object sits higher in the frame and report the resulting gap for open models. We extend the test to the frontier models and to the Distance task. CV-Bench Depth has 88 items on which the box lower in the frame belongs to the farther object. These 15% of items hold 60 to 82% of the frontier models’ Depth errors, and the gap under this cue is +10 to +22 points with intervals that exclude zero (Table 9). Their answers correlate with this cue at 0.73 to 0.75, slightly above the cue’s own correlation with the truth of 0.71, so their errors lean toward the cue (Brunswik, 1956).

modelDepth: gap [95% CI] under each cue: acc.larger boxlower boxDistance: acc.smaller pixel distance
Claude Opus 5.598.2+0 [−2, +3]+10 [+4, +17]95.3+19 [+12, +27]
GPT-5.595.0+1 [−2, +5]+18 [+10, +27]90.3+48 [+38, +57]
Gemini 3.6 Flash95.7+3 [−0.3, +7]+22 [+13, +31]91.7+28 [+20, +37]
Qwen3-VL-4B95.8+3 [−0.5, +6]+6 [+0.1, +13]88.7+34 [+25, +43]
Qwen3-VL-8B94.3−1 [−5, +3]+7 [+0.4, +14]90.8+44 [+35, +53]
cue right on–59.5%85.3%–80.2%
cue-wrong items–24388–119

Table 9. The shortcut gap on the 3-D subset of CV-Bench, in points, under each 2-D cue, on 600 photographs per task. The cues are not balanced. The last rows give how often each cue is right and on how many items it is wrong. Where the lower box is the farther one, every frontier model loses accuracy, and the Qwen intervals end within half a point of zero. The size of the box carries no reliable gap, with no interval above zero, as for closer in the rendered scenes. Where the object nearer the reference in the picture is farther from it in the scene, every model loses accuracy on Distance too. Intervals are 95% item-bootstrap intervals.

The pedestal scenes test height in the frame directly. There the frontier models answer 16, 16 and 15 of the 16 cue-wrong closer items correctly (Appendix § 15). Each of the two perfect scores has a 95% interval of 79 to 100%. The Depth errors may therefore come from photographs in which the common support of the two objects is hard to see.

On Distance the models lean on the cue more than it deserves. The object nearer the reference box in the picture is also nearer to it in the scene on 80.2% of items. On the 119 items where it is not, the frontier models show gaps of +19 to +48 points, and these items hold 74 to 98% of their Distance errors. GPT-5.5’s answers correlate with this cue at 0.79, above the cue’s own correlation with the truth of 0.60, and follow it on 48% of the items where it is wrong. Leaving out any one of the three source datasets keeps every frontier point estimate at or above +7 points on Depth and +17 on Distance (Appendix Table 20).

The larger box carries no reliable gap, just as pixel area carries none for closer in the renders. On Depth the answers are consistent with cue use. On Distance they use the cue more than its validity warrants.

Does the information reach the language model? (RQ5)

In the two probed models the information that the picture carries reaches the language model, while their drawn maps lose much of it.

Models and probes. The drawn maps leave open whether the information that the picture carries is lost before it reaches the language model or afterwards. The frontier models are closed, so we follow the information inside Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct, the two probed models, run in bfloat16. A hidden state is the vector of numbers that the language model computes for one token at one layer. Image tokens are the tokens into which the vision encoder turns the picture, one for each 32×3232 \times32-pixel cell. A layer is one of the language model’s stacked processing steps. Layer 0 is the output of the vision merger, which the language model receives as input. The merger is the small network that joins neighbouring vision-encoder features into image tokens.

A probe is a linear regression trained to read a known quantity from hidden states. If so simple a reader can recover a quantity, the hidden states make it available, whether or not the model uses it. Our probes are ridge regressions, cross-validated over scenes, that read each visible object’s ground depth and lateral position from the mean hidden state of the image tokens that cover its visible mask. The masks come from the generator, so the probes receive a segmentation that the model does not. Qwen3-VL also adds features from three intermediate vision layers to the first three layers of the language model (DeepStack, Bai et al., [4]), so the language model receives visual information at more than one layer.

The probes cover 7,002 objects in 1,252 scenes. The same probe applied to each object’s exact pixel measurements is the reference, the reader of the 2-D measurements. A probe on the hidden states can read depth better than the reader of the 2-D measurements, because the hidden states encode depth in a form closer to linear than the raw measurements do (Appendix § 14). Further probes read object positions at the object’s name in the statement and the truth of the statement at the answer position. The answer position is the last token of the prompt, whose next-token distribution gives the answer. The two probed models are scored by the log-probabilities of True and False.

Edits. An edit adds a chosen vector to a hidden state. We then check whether the answer moves. To test whether the answers use the layout we fit encoding directions for ground depth and picture position (Haufe et al., [28]). An encoding direction is the least-squares map from a quantity to the hidden state, the direction along which the hidden states move as the quantity grows. A closer edit swaps the two objects’ values along one of these directions. A between edit moves the middle object to the midpoint of the other two, on the ground or in the picture. Each edit is made at the size that the swap or move implies and at four times that size, in the image tokens that the objects cover. We read the change in the log-odds of True over False against random edits of the same norm as the combined depth-and-picture edit. Appendix § 12 gives the full methods and results.

Qwen3-VL-4B answers 61.8% of medium items correctly with a gap of +14 points, and the maps it draws keep 0.25 of the true depth-to-width ratio (Table 10). A probe of its hidden states recovers far more than its drawn map shows. At layer 16 the probe recovers each object’s relative depth with R2R^2 0.93 [0.92, 0.93], against 0.87 [0.86, 0.88] for the reader of the 2-D measurements. A map decoded from the states is not compressed, as for any reader trained on the true positions. Depth is decodable with R2R^2 0.86 already at layer 0, the output of the vision merger.

readerdepth R2R^2 [95% CI]lateral R2R^2depth-to-width ratioacc.gap [95% CI]fits gap
Hidden states, layer 160.93 [0.92, 0.93]0.941.0180.2+8 [+2, +14]+27
Reader of the 2-D measurements (ridge)0.87 [0.86, 0.88]0.961.0087.8+6 [+1, +11]+28
Visible boxes only, ridge0.86 [0.84, 0.87]0.960.9779.5+20 [+13, +26]+37
Gradient-boosted trees on 2-D measurements0.95 [0.94, 0.95]0.98––––
Qwen3-VL-4B’s drawn map––0.2531.5+16+15
Qwen3-VL-4B’s answers from the image–––61.8+14+2
Qwen3-VL-8B hidden states, layer 280.89 [0.88, 0.90]0.910.9970.8+13 [+6, +20]+23
Qwen3-VL-8B’s drawn map––0.3138.1+23+25
Qwen3-VL-8B’s answers from the image–––63.3+14−10

Table 10. Readers of the ground-plane layout for Qwen3-VL-4B and Qwen3-VL-8B. The hidden states give a layout close in R2R^2 to the reader of the 2-D measurements, with no compression of depth, as for any reader trained on the true positions, while the model’s drawn map and answers do much worse. The reader of the 2-D measurements is the same ridge probe applied to exact pixel measurements. The next two rows give it the visible boxes only or use gradient-boosted trees instead. R2R^2 pools the 7,002 objects of the three tiers and the second medium split. Maps, aligned as in § 5.4, and statements are scored on the 600 medium items. Intervals are 95% scene-bootstrap intervals.

The probe therefore shows that the information in the picture reaches the language model, not that the model forms a better layout than the picture provides. The comparison with the drawn maps is between a probe trained on the true layout and a zero-shot answer. Statements re-derived from the 4B’s maps are right on 31.5%, below chance, in part because many of its maps put objects at nearly the same depth, inside the reject band of closer. The compression thus partly reflects a failure at the map task itself. Qwen3-VL-8B, a robustness check within the same family, behaves the same way, with a depth R2R^2 of 0.89 and drawn maps that keep 0.31 of the ratio. Its lower value at layer 0 (R2R^2 0.61) fits the loss of spatial information at the merger that Maheshwari et al. [53] report for a model of the same size.

The information also reaches the words of the statement and the answer position. At the last token of each object’s name a probe reads its depth with R2R^2 0.88. On the cue-wrong statements outside closer, a probe at the answer position recovers the truth on 65.8% of items against 51.0% for the model’s answers (difference +15 [+9, +20] points). An input-only reader of the 2-D measurements (gradient-boosted trees), which never sees the model, does as well (65.2%, Appendix § 12.§ 1).

Editing the hidden states shows that the answers depend weakly on the image tokens. Swapping the ground depths of the two objects of a closer statement moves the answers both ways at layers 8 and 12, as in the depth write-back of Zou and Marttinen (2026). The log-odds change by +1.2 to +1.7 on the statements the swap makes true and by −2.4 to −2.0 on those it makes false. At layer 12 the swap at four times the size of the change flips 6.7 to 8.8% of the answers, against 2.1 to 2.9% for a random edit (differences of +5.8 [+2.9, +9.2] points in the 4B and +4.6 [+1.7, +7.9] in the 8B). Its effect is gone by layer 20 (Appendix Tables 15 and 17). For between, picture edits move the answers further than ground edits (rises of +0.20 to +0.44 against +0.10 to +0.16), although the picture edits are also larger in norm (Appendix Table 17). Ground edits flip answers no more often than random edits, so these edits give no evidence that between is read on the ground. The edits cannot separate ground depth from picture position, because an object’s depth and its row in the picture are strongly correlated in these scenes (Appendix § 12).

The two probed models fail the coordinates rule of § 4. Given the frontier models’ coordinate prompts without an image, they answer 53.0% and 55.8% of items correctly, and 53.3% and 54.3% with the numeric definitions, below their accuracy on the image condition (Table 11). Their largest gaps on the image condition are on taller and touching (Table 11).

modelacc.gapclosertallertouchingbetweenfitsacc.gapacc. / gap
Qwen3-VL-4B61.8+13.7+3+33+18+12+253.0+3.353.3 / +0.7
Qwen3-VL-8B63.3+14.0+10+22+30+18−1055.8+3.754.3 / +5.3

Table 11. Behaviour of the two probed models on the 600 medium items, scored by the log-probabilities of True and False. The image condition gives accuracy (%), the pooled gap and the gap per relation. The coordinate inputs, which have no image, reuse the frontier models’ coordinate prompts verbatim. Ground coordinates do not help either model, so by the coordinates rule of § 4 their gaps are not interpreted.

Discussion

What the evidence locates. Each manipulation removes one candidate explanation. Balancing the cue rules out a gap that comes from the label distribution, and the difficulty adjustment rules out the measured difficulty covariates as the source of the fits gap on the medium and hard tiers. The ground-plane form rules out an inability to apply the relations. The perspective form splits the remaining step into reading measurements and depths from the picture and carrying them onto the ground. Getting from the picture to measurements and depths accounts for most of the fits gap, since text inputs that supply them close most of it. The between errors persist with exact measurements, where the models apply the rule to image-plane angles and read the word loosely. From the picture, part of the between gap may arise earlier, in where the models anchor objects (see the paragraph on 2-D cues below). The maps show the layout that this step produces, with depth compressed toward the picture by an amount that differs between models. The maps do not predict the answers item by item (§ 5.4), so the compression they show may accompany the failure without causing it.

For fits the pattern is consistent with comparing a gap and an object without rescaling them to a common depth, the problem of size constancy (Holway and Boring, [32]; Sun et al., 2026). This holds for GPT-5.5 and Gemini in the cell that tests size scaling. The errors of Opus 5.5 lie mainly in the foreshortened cell, where the object fits but looks too wide and the size-constancy reader also fails. In the two probed models the probes show the information reaching the language model and the edits show the answers reading the image tokens weakly.

Which 2-D cues? Each relation’s cue is the simplest 2-D reading, which is not always the cue that human picture perception relies on [14] (Sedgwick 1986). A reader that uses only each object’s lowest visible point solves 98, 80 and 58% of the cue-wrong between items on the easy, medium and hard tiers, yet the frontier between gaps on the easy tier are +42 to +47 points. The between gap may therefore reflect where the models anchor objects in the picture as well as whether they carry the picture onto the ground (Appendix § 13). Benchmarks that balance a cue should balance the one a competent observer of pictures would use, such as ground contact or height below the horizon, or they risk scoring good picture reading as 3-D understanding.

Limitations

Scope. For the frontier models the results describe behaviour under different inputs, not the internal computation. A sufficient input locates the failing step in behaviour. A small gap on CueClash shows that a model’s answers do not follow the picture, which is one part of 3-D understanding. The headline numbers are at low reasoning effort. High effort changes the fits gap by −22 to −15 points (§ 5.1).

Scenes. The scenes are rendered primitives on a plane, which is what allows exact balancing. The check on photographs covers depth order and relative distance, on the 88 and 119 CV-Bench items where the lower-box and distance cues are wrong. These items are not adjusted for difficulty, so the photographic gaps are an upper bound on the cue’s influence. Testing taller, touching, between and fits on photographs needs metric annotations and items balanced against the cue.

Human reference. We collected no human judgements of these pictures. The human constancy value we quote comes from real scenes. Appendix § 22 gives a protocol for a study with the same pictures.

between and touching. In the false half of between the cue-wrong items lie near the decision threshold. In every cue-wrong contact of touching a third object hides the contact point, and we do not show that the picture alone determines these answers. Their gaps may mix word use and occlusion with geometry, which is why we also report the gap without these two relations.

Probed models. The two probed models are small open models of one family. Neither passes the coordinates rule of § 4, so we use them to follow where the picture’s information goes and do not interpret their gaps. The probes and the edits are linear. They show what a simple reader can use, not the computation the model performs.

Conclusion

The failure lies in the step from the picture to the layout. GPT-5.5 recovers a layout close to the picture plane. Gemini 3.6 Flash’s layout corrects part of the way and adds a uniform compression of depth. Opus 5.5’s maps have a constancy index close to the one reported for human observers of real scenes, although its answers still show a picture-plane gap on fits. All three read the numbers in inputs that carry the information and can apply the relations once the layout is given. The layouts they draw from the picture keep most of the order of the objects in depth but compress the distances. In the two probed models the hidden states keep the information that the picture carries, while their drawn maps lose much of it.

Accuracy on natural images cannot show this, because there the picture and the scene agree. On the few CV-Bench images where they disagree, the same models lose accuracy. Spatial evaluations can follow CueClash and balance a 2-D cue against the truth. We suggest the cue that a competent observer of pictures would use, with the resulting gap reported next to accuracy and the reasoning effort. Work on 3-D reasoning in VLMs should target this step. For agents that must judge clearance from a single frame, supplying exact ground coordinates from depth sensing or a map removes most of the failure we measure.

Further analyses

This appendix keeps the full per-model results that the main text summarises as ranges.

Main results. The frontier models’ answers track the picture plane. Opus 5.5, GPT-5.5 and Gemini 3.6 Flash reach 76 to 86% accuracy with gaps of +18 to +37 points (Table 2). On medium, GPT-5.5 is 96.0% correct on cue-right items and 61.0% on cue-wrong items. The second medium split, from an independent seed (seed 11), gives gaps of +19.7, +35.0 and +22.7 (Appendix Table 36). Three repeats move the pooled gap by about a point (Appendix Table 26), and tier changes it little (Table 2). Grok 4.7 patterns with the frontier models (76.5%, +33.0) and Qwen3-VL-235B sits between (69.8%, +23.7). Nova 2 Pro (60–64%) and three open-weight models (50.3–60.5%) score lower and gain little from ground coordinates (Pixtral 58.8%, Llama 64.7%, Gemma 51.7%), so their errors are not specific to the picture plane. By the coordinates rule of § 4 we do not read their gaps or those of Qwen3-VL-235B. Text-only accuracy is 46 to 52% for every model, so the statement wording alone does not give away the label.

Where the gap sits. The gap is concentrated in the relations that compare quantities across depth or behind occluders (Figure 4). Per-relation 95% CIs are about ±15 points. fits is +18 to +68, and GPT-5.5 answers its cue-wrong half at 33%, below chance, as on the item of Figure 1. between is +37 to +47 and touching +13 to +50. taller is small for Opus 5.5 and moderate for the other two. The control relation closer is −2 to +0.

Harder items or the cue? If cue-wrong items were only harder, the gap should vanish once difficulty is held fixed. For fits it does not. The pooled adjusted gaps are +10 to +29 points with intervals excluding zero for every frontier model and tier (Appendix Table 30). Appendix Table 35 gives the adjusted gap of each relation. On the fits items that share covariate support, the gap is +38 to +66 raw and +26 to +59 after inverse-propensity weighting (Appendix Table 34). Within tertiles of the margin (the distance from the decision threshold), the pooled gap is +18 to +42 on the smallest margins and −4 to +22 on the largest. This grading is consistent with a cue-combination observer ([42]), but cannot be settled this way. Its adjusted gap is +25, +29 and +28 under the shared-slope model, +34, +31 and +33 with inverse-propensity weighting, and +17, +8 and +11 in the within-relation model with a binary construct flag (Appendix Table 34). In its false half, cue status and nearness to the reject band coincide almost exactly, so no regression can separate the cue from the margin. We report between as confounded and take fits as the case free of the margin confound.

Information or form? Ground-plane form removes the gap. Perspective form that carries nearly the same information removes only part of it (Table 5 and Table 6). Exact pixel measurements as text leave accuracy close to the image condition (86.3, 81.5 and 79.7% without the image), so the models read the text. Adding camera depth makes closer and between exactly solvable and fits solvable to 97% by the solver. The models then cut the fits gap from +32, +65 and +45 to +10, +17 and +20, while between stays at +33, +28 and +37. A depth map as a second image changes the pooled gap by −8 to −4. The top-down map, a second image with labelled footprints, narrows the pooled gap to +4 to +5, as coordinates do without the image (+4 to +6, and +2.7, +2.3 and +3.0 on the second medium split). With coordinates and the numeric definitions, accuracy is 99 to 100% and no gap is more than 0.3 points from zero. Table 6 gives the gap under each input with its interval, and Table 7 the fits and between gaps. The models reason correctly on the ground plane and convert a perspective description into it only partly.

The between rule. Given an exact rule, the models apply it to whatever points they receive. Given pixel points, they apply it to image-plane angles. In Table 12 we fix the numeric definition and change only the position input. Visible-region centres give 61.7, 61.7 and 60.0%, ground-contact pixels 94.2, 93.3 and 92.5%, and the homography or coordinates up to 100%. Applying the 10∘10^\circ rule to the image-plane angles of the same points scores 60.8 and 92.5%. GPT-5.5’s answers agree with that image-plane rule on 99.2% of items. Image-plane angles decide most items only because the reject band leaves nothing between 10∘10^\circ and 30∘30^\circ. Without the numeric rule, ground coordinates still leave between gaps of +28, +23 and +10, so the models’ own “between” is looser than ours. The between gap combines a rule looser than ours with the composition of its cue-wrong cell near the reject band (Appendix Table 38).

input (all with numeric definitions)Claude Opus 5.5GPT-5.5Gemini 3.6 Flashsolver
image87.5 / +18.3 [+7, +30]78.3 / +36.7 [+23, +50]82.5 / +28.3 [+16, +41]–
visible-region centres (text)61.7 / +50.0 [+35, +65]61.7 / +53.3 [+39, +68]60.0 / +53.3 [+39, +68]72†
ground-contact pixels (text)94.2 / +5.0 [−3, +14]93.3 / +6.7 [−2, +16]92.5 / +5.0 [−4, +15]94†
ground-contact pixels + homography (text)99.2 / +1.7 [+0, +6]100.0 / +0.0 [+0, +0]91.7 / −3.3 [−14, +6]100
ground coordinates (text)100.0 / +0.0 [+0, +0]100.0 / +0.0 [+0, +0]100.0 / +0.0 [+0, +0]100

Table 12. Accuracy (%) / gap (points) [95% CI] of between under one rule (numeric definitions) and five position inputs (the image and four text inputs), medium tier (120 items). Text inputs have no image, and the solver column reads the same text. The models match the solver on each input. † marks an approximate solver (Table 5).

Contacts. The two best models’ touching gap sits on contacts hidden behind a third object (Table 13). Opus 5.5 answers apart pairs perfectly, whether or not they overlap in projection. It answers hidden contacts at 10%. With coordinates and no rule, hidden contacts are 83, 47 and 47% correct. With the numeric contact rule the same coordinates give 100, 100 and 97%. Exact surface distances give 100%. On the ground plane, contact is a computation the models perform once the rule is stated. From the picture, the contact point is hidden in 55 of 60 true contacts, a split that is close to forced for two convex solids that meet. A third object covers the region between the silhouettes in every cue-wrong contact, so whether these contacts can be resolved from the picture at all is left open (§ 7).

Table 13: touching on the medium tier, accuracy (%) by proposition truth (the truth of the relation before any negation) and cue status, and pooled accuracy / gap. Hidden contacts are in contact, with the visible masks more than 3 px apart because a third object covers the contact. Overlapping pairs are apart, with masks that overlap in projection. Each cell has 30 items. Coordinates with the numeric rule solve every cell.

modelinputmasks within 3 pxhiddenmasks apartoverlappingacc / gap
Claude Opus 5.5image631010010068 / +27
Claude Opus 5.5image + numeric definitions70201009772 / +27
Claude Opus 5.5coordinates only908310010093 / +3
Claude Opus 5.5coordinates only + verbal definitions907710010092 / +7
Claude Opus 5.5coordinates only + numeric definitions1001001009799 / +2
Claude Opus 5.5image + surface distances + numeric definitions100100100100100 / +0
GPT-5.5image8023975363 / +50
GPT-5.5image + numeric definitions5727975057 / +38
GPT-5.5coordinates only634710010078 / +8
GPT-5.5coordinates only + verbal definitions635010010078 / +7
GPT-5.5coordinates only + numeric definitions100100100100100 / +0
GPT-5.5image + surface distances + numeric definitions100100100100100 / +0
Gemini 3.6 Flashimage57331008769 / +18
Gemini 3.6 Flashimage + numeric definitions67371009374 / +18
Gemini 3.6 Flashcoordinates only674710010078 / +10
Gemini 3.6 Flashcoordinates only + verbal definitions604710010077 / +7
Gemini 3.6 Flashcoordinates only + numeric definitions939710010098 / −2
Gemini 3.6 Flashimage + surface distances + numeric definitions100100100100100 / +0

Table 13.

Definitions and bands. The stated-threshold prompt mixes each relation’s numeric rule with the reject bands, which tell the model which near-miss cases are absent. We separate the two in Table 14. For Opus 5.5, numeric definitions change the gap by −7.3 [−13, −2], bands by −9.3 [−14, −4] and both by −12.3 [−18, −7]. The two do not add. Their interaction is +4.3 [−1, +10] for Opus 5.5 and +7.0 [+1, +13] for Gemini. The bands carry sampling information rather than geometry. With bands alone, Opus 5.5 answers hidden contacts at 83% against 10% with the default prompt, because the bands imply that pairs that look close are in contact. For GPT-5.5 every paired interval includes zero. Headline numbers use the default prompt.

Table 14: Prompt factorial on the medium image condition, with the stated-threshold prompt split into numeric definitions and reject bands. Δgap is the paired change against the default prompt (scene bootstrap), and the last three columns are per-relation gaps. No prefix closes the gap.

modelprompt prefixacc / gap [95% CI]Δgap vs. default [CI]touchingbetweenfits
Claude Opus 5.5neither (default prompt)85.7 / +21.3 [+16, +27]–+27+45+32
Claude Opus 5.5numeric definitions only89.0 / +14.0 [+9, +19]−7.3 [−13, −2]+27+18+23
Claude Opus 5.5reject bands only92.3 / +12.0 [+8, +16]−9.3 [−14, −4]+7+27+22
Claude Opus 5.5numeric definitions + bands94.2 / +9.0 [+5, +13]−12.3 [−18, −7]+17+8+18
Claude Opus 5.5stated thresholds (original wording)93.2 / +11.7 [+8, +16]−9.7 [−14, −5]+15+15+20
Claude Opus 5.5verbal definitions88.0 / +14.7 [+10, +19]−6.7 [−10, −3]+20+28+20
GPT-5.5neither (default prompt)78.5 / +35.0 [+29, +41]–+50+43+65
GPT-5.5numeric definitions only78.7 / +32.0 [+25, +38]−3.0 [−9, +3]+38+37+68
GPT-5.5reject bands only80.5 / +34.3 [+28, +40]−0.7 [−5, +4]+53+43+63
GPT-5.5numeric definitions + bands77.7 / +38.0 [+32, +44]+3.0 [−2, +8]+53+43+75
GPT-5.5stated thresholds (original wording)80.8 / +33.0 [+27, +39]−2.0 [−7, +3]+53+38+58
GPT-5.5verbal definitions77.0 / +33.3 [+27, +39]−1.7 [−7, +4]+42+38+67
Gemini 3.6 Flashneither (default prompt)81.7 / +24.7 [+19, +30]–+18+43+45
Gemini 3.6 Flashnumeric definitions only85.2 / +19.7 [+15, +25]−5.0 [−10, −0]+18+28+47
Gemini 3.6 Flashreject bands only84.5 / +21.7 [+16, +27]−3.0 [−8, +2]+23+42+33
Gemini 3.6 Flashnumeric definitions + bands83.8 / +23.7 [+18, +30]−1.0 [−6, +4]+25+38+42
Gemini 3.6 Flashstated thresholds (original wording)84.0 / +22.0 [+17, +27]−2.7 [−7, +2]+22+35+40
Gemini 3.6 Flashverbal definitions81.5 / +24.3 [+19, +30]−0.3 [−5, +5]+17+38+52

Table 14.

Effort, tools and views. More reasoning and exact measurement narrow the pooled gap by a few points and leave it in place, although high effort changes the fits gap by −22 to −15 points (Section 5.1). High effort with a 16,384-token budget changes the gap by −10 to −5 points (Appendix Table 27). A zoom tool and a sandboxed Python interpreter over the pixel array change Opus 5.5’s gap by −5.3 [−9, −2], with tools used on 50% of items (Appendix Table 31). The measurements are accurate. Asked only to measure, Opus 5.5 returns visible-pixel boxes with a median IoU of 0.977. Its lowest row is within 2 px on 89% of 238 objects. Accurate measurement in the picture plane does not move the answer onto the ground plane. A checkerboard ground, a dense perspective texture, leaves the gap unchanged within noise (Appendix Table 28). On a second camera, scored against that camera’s own cue, the gaps are +3.1 [−6, +12], +30.3 [+19, +42] and +8.9 [−2, +20], on only 68 cue-wrong items (Appendix Table 32). GPT-5.5’s gap follows the cue to the new camera. Agreement between the two cameras is in Appendix Table 39.

The models’ drawn maps. Drawn from above, the frontier models’ maps sit at or between the picture-plane copy and the scene (Table 8). The picture-plane copy compresses depth to 0.61 of the lateral spread on medium and 0.30 on the hard tier. GPT-5.5’s and Gemini’s maps are compressed to 0.61 and 0.63 on medium, about as much as the copy, while Opus 5.5’s reach 0.83. Depth order is mostly right in all three (median rank correlation along the camera axis 1.00, 0.96 and 0.94). This is the pattern of phenomenal regression, a percept that moves only part of the way from the retinal image toward the real object (Thouless, 1931). It resembles the exocentric compression of human observers [51]. Statements re-derived from the aligned maps show gaps of +20 to +38. They match the direct answers on 70–75% of items (Mantel–Haenszel odds ratio 2.1–3.9 within relation × cue strata, Appendix Table 37). Fed their drawn maps as text, the models read them exactly as a solver does (100%). Stretching the depth axis by a factor estimated on the other half of the scenes changes the fits gap by −13.3 [−22, −5], −16.7 [−29, −5] and −10.0 [−21, +2], and the between gap by −1.7, +3.3 and +0.0 (Appendix Table 33). Uniform compression explains part of fits and none of between. Figure 5 shows one fits item on which every map compresses the gap below the object’s width. Grok 4.7 behaves like the frontier models but returns parseable maps for 224 of 304 scenes. The five other models’ maps order depth less reliably, and the Qwen3-VL-235B maps run depth the wrong way (median rank correlation −0.89 on medium).

Generator and statistics details

Scenes. Each scene has four to eight convex primitives (cube, sphere, cylinder, cone) resting on a matte grey ground plane, lit by one sun lamp with a random azimuth and 30–60° elevation, and viewed by a perspective camera with a 35–50 mm equivalent lens at 8–14 units from a jittered target near the scene centroid. Every object has a distinct colour from a palette of eight, so “the red cube” names exactly one object. Sizes are drawn log-uniformly in [0.3, 2.0] world units independently of shape, so familiar-size priors are useless. Perspective, ground contact, occlusion and shadows are the only depth and scale cues. Objects rest on the ground, do not interpenetrate, and lie inside the frame with at least 0.15% of the pixels and 35% of their silhouette visible. Images are rendered at 768 × 768 with Blender 5.0.1 Cycles (Blender Online Community, [7]). Depth maps and instance masks come from an analytic ray caster over the same primitives, so the 2-D statistics that define the cues are exact.

Relations. Table 1 defines the five relations. Each has a 3-D truth read from the scene graph, a 2-D cue computed from the visible instance masks, and a reject band around the decision threshold inside which candidates are discarded, so that the truth is unambiguous in the geometry. Surface distances are exact minimum distances between convex solids. In fits, the gap between the two objects is the minimum distance between their footprints, compared with the moving object’s narrowest width (cube side or diameter), and the gap must be clear of any third object by 0.25 units. The item of Figure 1 has a ratio of 1.64 on the ground, computed from the unrounded sizes, and a pixel ratio of 0.45. Half of all statements are negated and every template has three paraphrases, so no wording predicts the label. The default prompt does not state the numeric thresholds.

The control relation. The natural cue for closer, “lower in the frame means closer”, is close to veridical for objects on a ground plane seen from above. It agrees with the truth on all 480 closer items of the benchmark and on 100% of candidate pairs under random placement (Appendix Table 22). This is an empirical property of our scene distribution and the 15% depth margin, not a general theorem. We balance closer on pixel area instead, the size-to-depth confound that Zou and Marttinen (2026) found models follow, and treat it as a control whose zero gap says only that pixel area is not used.

Balancing. For each relation we sample scenes and enumerate candidate statements until each of the four cells {true,false}×{cue agrees,disagrees}\{\text{true}, \text{false}\} \times\{\text{cue agrees}, \text{disagrees}\} holds exactly N/4N/4 items, half negated, with at most two statements per scene and never two about the same objects. Several cue-wrong cells almost never occur under random placement. The balancer therefore passes hints to the sampler, which constructs tangent pairs, occluders over a contact line or a base, near-collinear triples, and a near gap seen side-on with a wider far object. The constructs change the sampling distribution, not the labels. They are recorded per scene. Constructed objects are 80–97% of every cell, so construct membership enters the regression as a covariate. Appendix Table 25 gives the gap on constructed and natural items separately. Two side effects shape the interpretation. The cue-wrong touching cells are both occlusion configurations, a contact hidden behind a third object or an overlap in projection. The cue-wrong false between cell lies near the reject band, since all 30 of its triples deviate from the line by 30–60°, against 4 of the 30 cue-right false triples (Appendix Table 38). Generation is deterministic given a seed. Three tiers vary clutter, elevation, shadows and occlusion, each with 600 items (Appendix Table 21). A second medium split comes from an independent seed (Appendix Table 36). View-invariant relations are also rendered from a second camera.

Premise checks. Three checks run before any model. The 2-D cue scores exactly 50.0% overall and 100 / 0% on cue-right / cue-wrong items on every split, verified by a test. The named cues are informative under random placement. They agree with the truth on 82–99% of medium candidates. Because touching and between are rare at random, we also report their balanced accuracies (94 and 97%, Appendix Table 22). A learner that weights these cues is not irrational [42, 39]. The analytic masks agree with Blender renders of per-object masks from the tessellated meshes to a median per-object IoU of 0.9998. Relabelling the 11 items whose cue flips on the rendered masks moves no gap by more than 2.5 points (Appendix § 18). A datasheet is in Appendix § 24.

Prompts and models. Each item is one image and one statement. The instruction says the statement is about the 3-D scene, not the picture, and asks for one word, True or False, parsed leniently, with unparseable answers counted as wrong (Appendix Table 41). Every prompt is printed in Appendix § 16, with filled examples in Appendix § 19. Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash run on every tier for the image condition and on every medium-tier input, at low reasoning effort (we did not switch internal reasoning off), three times on the medium image condition and once at high effort with a 16,384-token budget. Grok 4.7, Nova 2 Pro, Pixtral Large, Qwen3-VL-235B, Llama 4 Maverick and Gemma 3 27B run on the inputs of Table 3. All calls go through one LiteLLM proxy with provider-default sampling and lossless PNG images. Appendix Table 40 lists requested and returned model identifiers, routes and dates.

Input forms. We also run the medium tier with the same scene in other forms. In perspective form we add, as text, every object’s visible bounding box, centroid, ground-contact pixel, visible area and hidden fraction, with or without the image, then the same measurements plus each object’s camera depth and the field of view. Figure 3 shows these measurements for the running example and what one pixel covers on the ground. A depth map is added as a second image, without and with a metric legend. In ground-plane form we add a to-scale top-down footprint map as a second image, with and without labels, or exact ground coordinates as text, with or without the image. The second camera view changes the picture but not the scene. For the frontier models we add a second set of inputs that isolates single steps. It splits the stated-threshold prompt into numeric definitions (each relation’s exact rule) and bands (the reject bands only). For between it supplies visible-region centres, ground-contact pixels, or ground-contact pixels with the exact image-to-ground homography, each without the image. For touching it supplies every pairwise surface distance. These all carry the numeric definitions, so they differ only in the input.

Solvers. For every text input a deterministic solver reads exactly the serialised text the model received, verified identical to the prompts sent to the model, and applies the benchmark’s rule (Table 5). Where the text lacks a needed quantity, such as object sizes in perspective form, the solver uses the best simple estimate with thresholds fitted on the easy tier. Such cells are marked approximate.

Map task. A separate task asks the model, from the image alone, for every object’s ground position and footprint diameter as JSON in a top-down frame of its choosing, with the scale set by naming one object’s diameter. We score maps by the root-mean-square error (RMSE) of their positions after similarity alignment to the scene graph, by the ratio of their depth spread to their lateral spread relative to the truth, and by re-deriving the split’s statements from the aligned positions with the true sizes. The alignment uses the truth, so the re-derived scores isolate position error. We align the picture-plane copy, the contact pixels read as top-down coordinates, the same way and compute its depth-to-width ratio. We also feed each frontier model’s drawn map back as text, raw or with its depth axis stretched by a factor estimated on the other half of the scenes.

Metrics. With truth YY, cue HH and answer MM, the shortcut gap is g=P(M=Y∣H=Y)−P(M=Y∣H≠Y)g = P(M=Y \mid H=Y) - P(M=Y \mid H\ne Y). On a split with equal halves P(M=H)=12+g2P(M=H) = \frac{1}{2} + \frac{g}{2}, so GPT-5.5’s medium gap of +35.0 means its answers agree with the cue on 67.5% of items. The gap describes alignment with the cue. A geometric observer with noisy evidence also shows a positive gap if cue-wrong items are harder for other reasons, which is why the adjustment, the overlap checks and the input forms are needed before reading it as reliance. We resample items for pooled intervals, which gives the same 95% intervals as resampling scenes because a scene holds at most two items (Opus 5.5: [+16, +27] against [+16, +27]). Every paired comparison resamples scenes 2,000 times [17]. The difficulty-adjusted gap is the average marginal effect of the cue being right in a Firth-penalised logit [19] with margin, depth ratio, smallest pixel area, largest occlusion, object count, elevation, polarity (whether the statement is negated), construct membership, relation, and interactions of cue status with relation and margin. We call this pooled fit the shared-slope model, because each covariate has one slope for all relations. The within-relation model fits the same logit to each relation separately. The primary endpoints are the three frontier models’ pooled medium gaps, with Bonferroni-corrected 98.3% intervals of [+15, +28], [+28, +42] and [+18, +31]. Relation-level contrasts are secondary. Of 45 per-relation image gaps, 28 survive a Holm correction (32 unadjusted, 2.2 expected under a global null).

Further related work

Cue conflict and balanced evaluation. Cue-conflict stimuli set two cues against each other to reveal which one a network uses, as in the texture–shape contrast of Geirhos et al. [22]. Balanced-pair VQA asks one question of two scenes with opposite answers [26] (Zhang et al. 2016). Prior-shift sets change the answer distribution [1] or score rare pairs separately [38]. Minimal pairs isolate one factor [29] (Thrush et al. 2022). Rel3D builds minimally contrastive pairs with 3-D truth [25]. NLVR supplies our statement format (Suhr et al., 2017, 2019), and CLEVR and PTR the rendered-primitive lineage [35, 33]. 3DSRBench balances with complementary image pairs and filters questions that the shortcut “higher in 3-D is higher in 2-D” answers [52]. These designs balance labels, remove shortcut items or conflict two cues. CueClash keeps the shortcut items and balances a named 2-D cue against the 3-D truth, so the picture plane and the ground plane disagree on half of every relation.

Spatial benchmarks and depth cues. VSR, What’s Up and GQA collect spatial relations judged by people from photographs, including front and behind, without metric 3-D truth [46, 37, 34]. The spatial subsets of BLINK and the 2-D subset of CV-Bench are similar, while CV-Bench’s 3-D subset asks depth order and relative distance on Omni3D images [21] (Tong et al. 2024a). SpatialVLM, SpatialRGPT-Bench, SpatialBot, Spatial457, EmbSpatial, SPACE and VSi-Bench scale 3-D-annotated questions [10, 12, 9, 16] (Wang et al. 2025; Ramakrishnan et al. 2025; Yang et al. 2025a). SpatialEval compares text-only and visual inputs (Wang et al., 2024). COMFORT scores consistency across frames of reference (Zhang et al., 2025a) and SpinBench under viewpoint change (Zhang et al., 2026b). Low-level failures of VLM vision are documented by Rahmanzadehgerdevi et al. (2024), MMVP (Tong et al., 2024b) and GeoMeter [3], occlusion failures by CAPTURе (Pothiraj et al., 2025), O-Bench [48] and Maheshwari et al. (2026), who localise the loss in one model to the vision merger. DepthCues probes monocular depth cues in vision encoders [15], perceptual-constancy probes test size constancy in VLMs (Sun et al., 2026), and O3-D varies pictorial depth cues one at a time [47]. Their vertical-position bias matches the angular-declination cue of human distance perception [59] and the behaviour of self-supervised depth networks (van Dijk and de Croon, 2019). Min et al. [57] and Zou and Marttinen (2026) are the closest designs. Each puts a cue in conflict with depth order, on renders and photographs. Zou and Marttinen (2026) also show that depth remains decodable when the answer is wrong. CueClash extends the conflict to four further relations and adds input forms that hold the information fixed.

Supplied inputs, maps and text. Depth helps a VLM as text and can hurt as an image (Vo et al., 2026). Layered depth captions also help (Roy et al., 2025). Coordinates are hard to use without explicit object association [12] (Yuan et al. 2026). VLMs also over-trust text that conflicts with the image (Deng et al., 2025). Bird’s-eye-view (BEV) prompting is an established lever. Struct2D and GPT4Scene prompt with BEV images and object marks (Zhu et al., 2025; Qi et al., 2026), VST shows a teacher model BEV renderings to generate part of its reasoning data (Yang et al., 2025b), VSI-Bench’s ground-truth cognitive map lifts relative-distance accuracy from 46% to 66–78% (Yang et al., 2025a), and 3ViewSense argues for a view-consistent representation (Zhan et al., 2026). MindCube finds a ground-truth map lowers accuracy unless the model reasons over it (Wang et al., 2026). TopViewRS finds VLMs weak on realistic top-view maps ([43]). Our top-down input is at the easy end of that spectrum. Our homography input supplies the image-to-ground mapping of inverse perspective mapping ([54]). These studies vary the form and the information together. Our inputs hold the information fixed, as a solver checks, and vary only the form.

Open models in full

Probes. § 5.6 introduces the probes, and Appendix § 14 gives their settings. Qwen3-VL-4B-Instruct is of the same family as Qwen3-VL-235B, and every probe is repeated on Qwen3-VL-8B-Instruct. Object features are read at every fourth decoder layer. Image tokens precede the text, so under the causal mask these states do not depend on the statement. Besides the reader of the 2-D measurements, the reference includes gradient-boosted trees that see the measurements of every object in the scene. Decoded maps are scored with the code that scores the drawn maps. The same model also answers every medium item and draws a map under the map prompt. The probe at the answer position uses the 1,200 items of both medium seeds. It is scored on the cue-wrong items and compared with a probe trained on labels permuted within each relation and cue cell. Any supervised reader of the input can beat a model, so we also train the probe on cue-right items only and compare it with an input-only reader, gradient-boosted trees on the 2-D measurements of the statement’s objects with the relation, the polarity and the cue’s verdict, cross-validated over the same scenes.

Steering. To test whether the answers use the layout we edit the hidden states of both open models and read the change in the log-odds of True over False. At each layer we fit encoding directions, the least-squares map from an object’s ground depth, lateral position, picture position and size to its hidden state ([28]), and add them to the states as in work on steering language models ([44] (Zou et al. 2023)). The directions are fitted on the states of all scenes, the steered ones included, which matters little with 7,002 objects. A closer edit swaps the two objects’ values along one group of directions, either ground depth, picture position (column, row and size), both together, which keeps the scene physically consistent, or lateral position as a control. A between edit moves the middle object to the midpoint of the other two, either on the ground or in the picture. Each edit is added to the image tokens the object covers or to the tokens of its name in the statement, at layers 8, 12, 16, 20 and 24 and at one and four times the size that the swap or move implies, on the 240 closer and 240 between items of the two medium seeds. Random edits with the norm of the combined edit, isotropic or within the top 64 principal components of the states, control for disruption. We report the change separately on the items whose log-odds the edit should raise and those it should lower. A random edit does not move both groups the same way. It pulls the log-odds toward zero, so the split partly follows the sign of the base answer, and we also compare each edit with the random edit within strata of the base log-odds (below). We also report the share of answers that cross to the side the edit implies. Appendix Table 17 gives every layer and edit.

Probes

Qwen3-VL-4B is weak on the task, answering 61.8% of medium items correctly with a gap of +14 points, and the maps it draws keep 0.25 of the true depth-to-width ratio, with 57% of them running depth the wrong way (Table 10). At layer 16 a linear probe recovers each object’s relative depth with R2R^2 0.93, compared with 0.87 for the same probe on the exact 2-D measurements and 0.95 for gradient-boosted trees given the measurements of every object in the scene (Figure 6). The difference is +0.060 [+0.050, +0.070] in a paired scene bootstrap, and +0.023 [+0.012, +0.034] for the 8B. Probes trained on the scenes of the easy split and both medium splits and tested on the hard tier reach R2R^2 0.79 for the 4B and 0.78 for the 8B, against 0.66 for the reader of the 2-D measurements, when the penalty is chosen by cross-validation grouped by scene. A map decoded from the hidden states is not compressed, with a depth-to-width ratio of 1.01, and statements read from it show a gap of +8 points. Any reader trained on the true positions is uncompressed in this sense. The ridge reader of the 2-D measurements gives a ratio of 1.00.

Linear probes on Qwen3-VL-4B’s object representations by decoder layer

Figure 6. Linear probes on Qwen3-VL-4B’s object representations, by decoder layer. Ground-plane depth (red) and lateral position (blue) are decodable at every layer and peak in the middle of the network. Picture-plane position (grey) is decoded better still. Horizontal lines give the same ridge readout applied to the exact 2-D measurements, and to the visible boxes alone. R2R^2 is within scene and cross-validated over 1,252 scenes.

The hidden states keep about as much of the layout as an exact reading of the picture does, while the model’s drawn map and answers keep far less. The probe adds little to the input itself, and depth is decodable with R2R^2 0.86 already at layer 0, the output of the vision merger (§ 5.6). For depth order, Zou and Marttinen (2026) found the analogous dissociation between a decodable readout and the answer, most strongly on rendered scenes. Three cautions bound the claim. Decodability shows that the information is available to a linear reader, not that the model computes with it (Belinkov, [5]), and the probed model is far weaker than the frontier models, whose internals we cannot inspect. Its answers also lack the frontier models’ fits gap (+2 points for the 4B and −10 for the 8B), so the probes follow the information path and do not reach the fits failure. The decoded map also keeps a fits gap of +27 points, about as large as the exact 2-D measurements give, so the hidden states do not hold a sharper layout than the picture itself provides.

The layout also reaches the words of the statement. At the last token of each object’s name the probe reads the object’s depth with R2R^2 0.88 at layer 20, close to the 0.92 of the image tokens and below the 0.94 of a polynomial reader of the exact 2-D measurements centred within the scene. At layer 0 the same probe gives 0.00, as it should, since the name has not yet attended to the image. The truth of the statement is partly present at the answer position as well. On the items whose cue is wrong, a probe at layer 24 recovers the truth on 71.2% of items [67.6, 74.7], while the model’s own answers are right on 54.8% and probes trained on labels permuted within each relation and cue cell score 49.6% over 20 draws (SD 2.6, at most 55.8). Part of this advantage comes from closer, whose cue-wrong items the probe gets 92.5% right. Without closer the probe is right on 65.8% of the 480 cue-wrong items and the model on 51.0%, a difference of +15 points [+9, +20]. The difference is positive at every layer from 20 to 36, between +8 and +15 points, and at most +3 before layer 20. The advantage does not show that the model forms an answer and then ignores it (Appendix Table 16). The input-only reader plays for the answer probe the role that an image ablation plays for depth probes (Zou and Marttinen, 2026). It never sees the model and is right on 65.2% of the same items. Trained only on the 480 cue-right items of each fold, the probe at the answer position beats the answers by +3.5 points [−1.6, +8.5], against +7.0 for random subsets of the same size (SD 2.0 over 30 draws), so leaving out the cue-wrong items changes it by −3.5 points [−6.9, +0.1] at equal training size, an interval that reaches zero. With growing training sets its advantage rises through +3, +7, +10 and +15 points (240, 480, 720 and 960 items) without levelling off. Choosing each training set and epoch first leaves it at +15, and at +14.2 with the grid of the inverse penalty C^\hat{C} extended to C=10C = 10, since the search often picks the edge of the original grid. Choosing the layer within each fold as well gives +14.4 [+8.5, +19.9], so the advantage does not depend on picking layer 24 after the fact. A reader trained on the labels of cue-wrong items learns the relation from the state at the answer position, as one can from the measurements alone. The state carries the information. The probe does not show what the model computes from it.

Qwen3-VL-8B shares the 4B’s family, items and analysis, with the answer and name layers chosen on the 4B and the image-probe layer chosen again on the 8B, so it is a robustness check rather than independent evidence. It gives the same overall picture with some differences (Table 10). It answers 63.3% of medium items correctly with a gap of +14 points. A probe at layer 28 reads depth with R2R^2 0.89 [0.88, 0.90], above the 0.87 of the reader of the 2-D measurements. The decoded map keeps a depth-to-width ratio of 0.99. The maps the 8B model draws keep 0.31 [0.29, 0.32] of the true ratio and run depth the wrong way in 98% of the 296 maps with spread on both axes, with a median depth rank correlation of −0.89. The order in depth is nearly perfect apart from its sign, which looks like copying the rows of the picture into the map, since nearer objects sit lower in the frame. The statements re-derived from the maps align them with reflection allowed, so the sign does not affect them. Unlike the 4B, the 8B decodes depth less well at layer 0 (R2R^2 0.61), and its answers show no fits gap. Its name tokens carry depth with R2R^2 0.88 at layer 20. On the cue-wrong items outside closer its probe at the answer position is right on 64.8% against 51.7% for its answers, a difference of +13 points [+8, +19] that stays between +7 and +13 from layer 20 to 36. The input-only reader matches it here too (+14 points over the answers), and trained on cue-right items only the probe gains +2.7 [−2.6, +7.7], a change of −2.9 [−6.5, +0.4] against random subsets of the same size. Choosing its layer within each fold gives +12.5 [+7.0, +17.8].

Steering

A decodable layout need not be the one the answer reads, and editing it tests this directly (Table 15). In the image tokens of both models, swapping the ground depths of the two objects in a closer statement moves the answer the way the swap should. At layers 8 and 12 the log-odds change by +1.2 to +1.7 on the statements the swap makes true and by −2.4 to −2.0 on those it makes false. Swapping the objects’ picture positions instead changes them by −1.9 to −1.1 on the second group and by only −0.1 to +0.2 on the first, which is what a random edit of comparable norm also does. Per unit of edit norm the depth swap changes the signed log-odds by 0.057 to 0.071, against 0.010 to 0.015 for the picture swap and 0.004 to 0.009 for a random edit. Swapping lateral positions changes nothing (at most 0.09). The effect fades in later layers, with rises of +0.30 to +0.56 at layer 16. At layer 20 none of the depth, picture or ground edits to the image tokens moves either group by more than 0.20, so by then the answer no longer reads the image tokens, at about the depth where image information is reported to stop flowing into the text tokens ([36] (Zhang et al. 2025b)). In the taxonomy of Liao et al. [45], the closer edits fall in the regime in which a decodable variable is also used with the right sign. The effect is real but small. The depth swap at four times the size of the change moves 6.7 to 9.6% of the answers to the side it implies at layers 8 and 12, against 2.1 to 6.2% for the random edit. This repeats on our scenes the depth write-back of Zou and Marttinen (2026).

relationeditimage tokens 4B riseimage tokens 4B fallimage tokens 8B riseimage tokens 8B fallname tokens 4B risename tokens 4B fallname tokens 8B risename tokens 8B fall
closerground depth+1.23−2.00+1.48−2.09−0.39−4.95−0.32−5.59
closerpicture position−0.13−1.40+0.08−1.09−2.12−9.30−2.00−8.68
closerlateral position−0.02−0.09−0.06−0.06−0.27−0.63−0.38−0.61
closerrandom (top PCs)+0.06−0.90+0.01−0.54−0.44−1.60−0.56−1.36
betweenground position+0.15−0.15+0.16−0.17−0.51−0.26−0.52−0.22
betweenpicture position+0.20−0.20+0.34−0.27−0.95−0.07−0.60−0.16
betweenrandom (top PCs)−0.00−0.08−0.04−0.02−0.06−0.05+0.01+0.01

Table 15. Steering Qwen3-VL-4B and 8B at layer 12 with edits four times the size of the change. Entries are mean changes in the log-odds of True over False, on the items whose log-odds the edit should raise (rise) and lower (fall). Bold rises have 95% item-bootstrap intervals above zero, uncorrected for the number of cells. A ground-depth swap in the image tokens moves closer answers both ways. The between edits in the image tokens move its answers both ways by smaller amounts, while the other edits mostly lower every answer, as a disruption does. Appendix Table 17 gives every layer, the swap of both depth and picture position, an isotropic random edit and the share of flipped answers.

Strata of the base answer. Because random edits pull the log-odds toward zero, we also compare the depth swap at four times the size of the change with the random top-PC edit within strata of the base answer. In the 4B at layer 12 the swap moves the log-odds toward the answer it implies by +2.26 on items first answered False and +1.39 on items first answered True, against +0.42 and +0.50 for the random edit. In the 8B the values are +2.04 and +1.70 against +0.45 and +0.22. The share of answers flipped the intended way exceeds the random edit’s by +3.3 [−0.8, +7.5] and +5.8 [+2.9, +9.2] points in the 4B at layers 8 and 12, and by +5.0 [+1.7, +8.3] and +4.6 [+1.7, +7.9] in the 8B, so three of the four intervals exclude zero.

The edits do not separate ground depth from picture position. In these scenes an object’s depth correlates with its picture row at r=−0.94r = -0.94. Its lateral position, column, row and size explain $R^2 0.90 of its depth, so the depth direction carries only the part of depth that the picture does not predict. Swapping it alone gives a state that no real scene produces. The swap of both depth and picture position, which keeps the scene consistent, moves the answers about as far at layer 8 (rises of +0.85 to +1.47) and less at layer 12 (+0.32 to +0.44). For closer, the control relation, the answer reads the objects’ depth from the image tokens, without telling whether it reads it on the ground or in the picture.

For between, a relation with a gap in the 8B and a smaller, uncertain one in the 4B (Table 11), the ground edit has no advantage over the picture edit. Moving the middle object to the midpoint of the other two raises the log-odds by +0.10 to +0.16 when the move is made on the ground and by +0.20 to +0.44 when it is made in the picture, and lowers them by −0.17 to −0.11 and −0.37 to −0.20 on the statements the move should make false, with effects per unit of norm of 0.011 to 0.018 and 0.015 to 0.026. Comparisons per unit of norm are rough, because the directions are fitted to variables in different units. Each edit moves the answers both ways, with every interval excluding zero at the 95% level taken one interval at a time, so the answer reads the image tokens for between as well. The effects are small. The picture edit moves the answers about twice as far, and the ground edit flips 0.8 to 3.3% of answers at layer 12 against 0.0 to 4.2% for a random edit. An edit to one object may miss whatever the model compares. Edits to the name tokens move no answer the intended way in either model. They lower the log-odds whatever the edit should do. A depth swap changes them by −5.6 to −4.2 on the statements it should make false and by −0.5 to −0.3 on those it should make true. The position carried by the name tokens is decodable, but these edits give no evidence that the answer reads it there.

Picture cues that human observers use

Each relation’s cue is the simplest 2-D reading, which is not always the cue that human picture perception relies on ([14] (Sedgwick 1986)). People place objects on the ground by their points of contact ([24]). For between our cue uses the centroids of the visible masks. The same 20° rule applied to the projected footprint centres, which lie hidden under the objects and so are not visible to the models, is right on 92.5% of medium items and 85% of the cue-wrong ones, because collinearity of ground points survives projection. A reader that sees only visible points, the horizontal centre of each object’s visible box at its lowest visible row, is right on 98, 89 and 78% of easy, medium and hard items and 98, 80 and 58% of the cue-wrong ones. The models weight centroid collinearity and give contact collinearity almost no weight (Appendix Table 29), which suggests that their between errors come from placing objects by their image regions rather than by their contacts. For taller the human cue is the horizon ratio, the object’s image height divided by the image distance from its base to the horizon, which gives its height in units of the camera’s height when the picture plane is vertical ([6] (Sedgwick 1986)). Our cameras are pitched down, the visible top of a cube or cylinder seen from above is its far top edge, and occlusion can raise the lowest visible row, so our reader is approximate and its accuracy a lower bound for an exact reader. On the easy and medium tiers it also uses the true horizon row, which lies outside the frame. The horizon lies inside the frame in 0, 2.3 and 99.7% of easy, medium and hard scenes, and a reader of the horizon ratio solves 23, 42 and 88% of the cue-wrong taller items on those tiers. GPT-5.5’s and Gemini’s taller gaps are smallest on the hard tier (+13 and +8), where the horizon is visible. Shadows, a strong human cue to contact ([55]), are rendered on the easy and medium tiers only, so the tiers vary shadows, elevation and occlusion together.

Probe details

Model and features. We run Qwen3-VL-4B-Instruct from Hugging Face in bfloat16 on Apple-silicon MPS for features, behaviour and the drawn map, and on an NVIDIA L40S for steering, and Qwen3-VL-8B-Instruct in bfloat16 on an L40S throughout, with identical code and settings. The image processor is fixed at 768 × 768, so that each merged visual token covers a 32 × 32-pixel cell and the image yields 24 × 24 tokens. Each scene gets one forward pass with the fixed prompt “Describe the scene.” Because the image tokens precede the prompt, their states are the same for any statement. For each object we weight the image-token states by the fraction of each cell that its visible instance mask covers and average them, at decoder layers 0, 4, . . . , 36, where layer 0 is the output of the vision merger. Qwen3-VL adds the features of vision layers 5, 11 and 17, each through its own merger, to the outputs of the first three decoder layers (DeepStack, [4]), so the image-token states from layer 4 on also contain intermediate vision features. Objects with no visible pixel have no representation and are dropped, along with scenes that keep fewer than three objects, which leaves 7,002 objects in 1,252 scenes from the easy, hard and both medium splits.

Targets and readers. Targets are each object’s ground position in the camera’s horizontal frame, split into depth along the camera’s forward direction and lateral position across it, both centred within the scene. Features are centred within the scene in the same way. Ridge regressions with the penalty chosen by internal cross-validation from 10−110^{-1} to 10510^{5} are fitted on standardised features, with 5-fold cross-validation grouped by scene. The readers of the 2-D measurements see each object’s projected footprint centre, visible box, centroid, square root of visible area, width and height, expanded to second-order polynomial terms for the ridge reader. The visible-only reader drops the footprint centre, which can lie behind an occluder. The gradient-boosted reader (scikit-learn HistGradientBoostingRegressor, 600 iterations, learning rate 0.05) also receives the scene mean and standard deviation of every measurement, so that it can infer the camera from the arrangement of the picture. R2R^2 intervals resample scenes 1,000 times.

Picture-plane targets and the residual. The same probes read the picture-plane copy, the footprint centre’s image column and row, with R2R^2 of 0.98 and 0.99 at layer 16, above the ground-plane values at every layer (Figure 6). The hidden states also predict 0.48 [0.44, 0.51] of the depth variance that the reader of the 2-D measurements leaves unexplained. We do not read this as information beyond the picture plane, because the gradient-boosted reader of the same measurements reaches a depth R2R^2 of 0.95 [0.94, 0.95], above the hidden-state probe. The residual shows only that the hidden states encode depth in a form closer to linear than the raw measurements do.

Behaviour and the drawn map. For behaviour we feed the benchmark’s image-condition prompt for each of the 600 medium items and compare the next-token log-probabilities of “True” and “False”. An exact tie counts as False, which happens on 15 of the 600 items for the 4B and on 12 for the 8B. For the map we use the map prompt of Appendix § 16 verbatim with greedy decoding and a 700-token budget, and parse and score the responses with the same code as for the other models. Of the 304 maps, 304 parse. In 55 of them every object shares one xx or one yy coordinate, so the depth-to-width ratio is taken over the other 249, while the statement scores use all parsed maps. The median depth rank correlation is −0.14 (median absolute value 0.50 with ties broken by order, and 0.46 with tied coordinates given their average rank, since most of these maps repeat a yy value), and 57% of the 249 maps with spread on both axes place farther objects at smaller yy although the prompt asks for yy to increase away from the camera. For the 8B the median absolute value is 0.89. The statement scores align each map with reflection allowed, so a reversed axis does not lower them. Statements re-derived from the drawn maps are 31.5% correct, which is below chance. Their gap is +16 points.

Probes on the statement. For the 1,200 items of both medium seeds we run the image-condition prompt once and keep the hidden states at decoder layers 0, 2, . . . , 36 at two kinds of position. The answer position is the last token of the prompt, a lower next-token distribution gives the model’s answer. A mention position is the last token of an object’s name, at its last occurrence in the statement. The mention probe is the ridge reader above, fitted on the 2,880 mentions of objects with a visible pixel and compared with the same reader on those objects’ image-token states and with the polynomial reader of their 2-D measurements. The probes at the answer position are logistic regressions on standardised states, with the inverse penalty chosen once at layer 18 by 3-fold grouped cross-validation from 10−410^{-4} to 10−110^{-1} and then fixed for every layer, and with 5-fold cross-validation grouped by scene. The permuted control shuffles the truth labels within each relation and cue cell and is fitted in the same way. Intervals resample scenes 1,000 times.

Controls for the probe at the answer position. Table 16 scores every reader on the cue-wrong items outside closer. The input-only reader is scikit-learn’s HistGradientBoostingClassifier (300 iterations, learning rate 0.05) on the 11 measurements of each of the statement’s objects and their pairwise differences, with the relation, the polarity, the cue’s verdict and the number of objects, cross-validated over the same five scene folds as the probe. The cue-right probe is fitted on the cue-right items of each training fold. The comparison probe is fitted on a random subset of each training fold of the same size. The nested probe chooses its penalty by 3-fold grouped cross-validation inside each training fold.

Table 16: Readers of the statement’s truth on the cue-wrong items outside closer (480 items), with the difference from the model’s answers and 95% scene-bootstrap intervals. The input-only reader, which never sees the model, does as well as the probe at the answer position. Trained without cue-wrong items, the probe scores a few points lower at equal training size, with intervals that reach zero. Its advantage grows with training size.

readerQwen3-VL-4B acc.Qwen3-VL-4B minus answers [95% CI]Qwen3-VL-8B acc.Qwen3-VL-8B minus answers [95% CI]
model’s answers51.0–51.7–
answer-position probe, layer 24 (as in the text)65.8+14.8 [+9.5, +20.0]64.8+13.1 [+7.5, +18.6]
same probe, CC chosen within each training fold65.8+14.8 [+9.5, +20.0]65.4+13.7 [+8.1, +19.4]
same probe, trained on cue-right items only54.6+3.5 [−1.6, +8.5]54.4+2.7 [−2.6, +7.7]
same probe, trained on a random subset of that size (mean ± SD of 30 draws)58.1+7.0 ± 2.057.3+5.7 ± 1.3
cue-right only minus random subset, paired–−3.5 [−6.9, +0.1]–−2.9 [−6.5, +0.4]
same probe, layer chosen within each training fold65.4+14.4 [+8.5, +19.9]64.2+12.5 [+7.0, +17.8]
input-only reader (no model)65.2+14.2 [+8.4, +20.2]65.2+13.5 [+7.9, +19.9]

Table 16.

Steering grid. Table 17 gives every image-token edit at four times the size of the change. Edits at the size of the change show the same pattern at about a third of the size. For the image tokens the directions are fitted at each layer on the object states of the probe, by least squares from the scene-centred ground depth, lateral position, centroid column and row and square root of visible area to the state, and the edit is added to every image token that the object’s visible mask covers by at least 30%. For the name tokens the directions are fitted on the state at the last token of each name and added to every token of the name, which may explain part of the disruption these edits cause. The two random edits take the norm of the combined edit of depth and picture position.

Table 17: Image-token steering of Qwen3-VL-4B and 8B at four times the size of the change. Each cell gives the mean change in the log-odds of True over False on the items whose log-odds the edit should raise and on those it should lower. Bold cells have 95% item-bootstrap intervals that exclude zero in the intended direction for both the rise and the fall. The last column gives the percentage of answers at layer 12 that cross to the side the edit implies and away from it.

modelrelationeditlayer 8layer 12layer 16layer 20layer 24flips at 12 (%)
4Bcloserground depth+1.37 / −2.01+1.23 / −2.00+0.30 / −0.77+0.04 / −0.08+0.01 / −0.038.8 / 1.7
4Bcloserpicture position+0.24 / −1.87−0.13 / −1.40−0.00 / −0.28−0.00 / −0.14+0.02 / −0.073.8 / 3.8
4Bcloserboth+1.47 / −3.36+0.44 / −2.19+0.54 / −0.93−0.04 / −0.14+0.02 / −0.047.1 / 2.9
4Bcloserlateral position−0.02 / −0.09−0.02 / −0.09−0.11 / −0.04+0.03 / −0.05−0.01 / −0.030.4 / 0.4
4Bcloserrandom (isotropic)+0.20 / −0.58+0.02 / −0.59−0.05 / −0.27+0.00 / −0.07+0.02 / −0.051.2 / 2.5
4Bcloserrandom (top PCs)+0.11 / −1.19+0.06 / −0.90−0.03 / −0.31+0.03 / −0.07+0.01 / −0.032.9 / 2.1
4Bbetweenground position+0.16 / −0.11+0.15 / −0.15+0.12 / −0.10−0.01 / −0.02−0.01 / −0.033.3 / 0.0
4Bbetweenpicture position+0.20 / −0.25+0.20 / −0.20+0.06 / −0.10−0.01 / −0.02−0.03 / −0.045.0 / 0.4
4Bbetweenboth+0.43 / −0.25+0.33 / −0.22+0.06 / −0.11−0.00 / −0.03−0.03 / −0.044.6 / 1.2
4Bbetweenrandom (isotropic)−0.03 / −0.03−0.01 / −0.02+0.01 / −0.07−0.00 / −0.01−0.03 / −0.021.7 / 0.8
4Bbetweenrandom (top PCs)−0.06 / −0.10−0.00 / −0.08−0.05 / −0.05−0.03 / −0.05−0.02 / −0.034.2 / 0.8
8Bcloserground depth+1.66 / −2.36+1.48 / −2.09+0.56 / −0.73+0.02 / −0.05−0.01 / −0.006.7 / 0.0
8Bcloserpicture position+0.07 / −1.78+0.08 / −1.09+0.16 / −0.62−0.04 / −0.20−0.03 / −0.012.1 / 2.5
8Bcloserboth+0.85 / −3.09+0.32 / −1.52+0.26 / −0.83−0.09 / −0.18−0.01 / −0.014.6 / 2.5
8Bcloserlateral position−0.04 / −0.02−0.06 / −0.06−0.04 / −0.00−0.05 / −0.03−0.03 / +0.011.2 / 0.0
8Bcloserrandom (isotropic)−0.14 / −0.31−0.05 / −0.530.01 / −0.44−0.03 / −0.04−0.01 / +0.010.0 / 2.1
8Bcloserrandom (top PCs)+0.09 / −0.63+0.08 / −0.54+0.03 / −0.46−0.05 / −0.10−0.03 / −0.002.1 / 2.1
8Bbetweenground position+0.10 / −0.14+0.16 / −0.17+0.11 / −0.11+0.00 / −0.02−0.01 / −0.020.8 / 2.1
8Bbetweenpicture position+0.44 / −0.37+0.34 / −0.27+0.08 / −0.13−0.04 / −0.00−0.01 / +0.002.5 / 2.5
8Bbetweenboth+0.54 / −0.27+0.33 / −0.21+0.10 / −0.14−0.02 / −0.02−0.01 / +0.012.5 / 2.5
8Bbetweenrandom (isotropic)−0.01 / −0.01−0.01 / −0.04−0.02 / −0.01−0.02 / −0.01−0.01 / −0.000.4 / 2.5
8Bbetweenrandom (top PCs)−0.05 / −0.02−0.04 / −0.02+0.07 / +0.01−0.01 / −0.01−0.02 / −0.020.0 / 2.5

Table 17.

The probe and CV-Bench code will accompany the code release.

Constancy of the maps and the photographic items

Constancy index. For each scene we compute the depth-to-width ratio mm of a model’s map and the ratio cc of the picture-plane copy, both as in Table 8, and the index ln⁡(m/c)/ln⁡(1/c)\ln(m/c)/\ln(1/c) of Thouless (1931), which is 0 for a map that copies the picture and 1 for an undistorted map. Scenes whose copy is nearly undistorted (c≥0.95c \ge0.95) would be dropped, but no scene on either tier reaches that value. The index has no upper bound, and 12 to 13% of Opus 5.5’s and GPT-5.5’s medium-tier maps exceed 1, so we summarise it by the median. Grok 4.7’s maps cannot be scored on 80 of 304 medium and 96 of 302 hard scenes, and its index uses the rest. Table 18 gives the median over scenes with a 2,000-draw scene-bootstrap interval, and the median ratio within each tertile of camera elevation. The copy’s ratio rises with elevation, roughly as the sine of the angle at which the camera sees the ground. Opus 5.5’s index stays near 0.6 on both tiers and in every tertile, while GPT-5.5’s stays near the copy on the medium tier.

tiermapscenesconstancy index [95% CI]lowmiddlehighmedian ratio
mediumpicture-plane copy–00.510.600.690.61
Claude Opus 5.53040.60 [0.58, 0.63]0.760.830.880.83
GPT-5.5304−0.01 [−0.07, 0.07]0.530.610.720.61
Gemini 3.6 Flash3040.07 [0.01, 0.12]0.570.610.700.63
Grok 4.72240.37 [0.30, 0.45]0.650.750.810.75
elevation tertiles: below 32°, 32–39°, above 39°
hardpicture-plane copy–00.240.300.360.30
Claude Opus 5.53020.63 [0.61, 0.65]0.620.630.680.64
GPT-5.53020.12 [0.08, 0.15]0.290.370.400.35
Gemini 3.6 Flash3010.37 [0.35, 0.41]0.460.460.490.47
Grok 4.72060.50 [0.43, 0.55]0.500.550.570.54
elevation tertiles: below 13°, 13–17°, above 17°

Table 18. Constancy of the drawn maps. The index is the median over scenes of ln⁡(m/c)/ln⁡(1/c)\ln(m/c)/\ln(1/c), with mm the map’s depth-to-width ratio and cc the picture-plane copy’s, so that 0 is a copy of the picture and 1 the scene. Opus 5.5’s index is about 0.6 on both tiers, while GPT-5.5’s medium-tier maps are copies of the picture.

A two-parameter fit. The index assumes that a map corrects a fixed share of the picture’s compression with no further bias. We test this by regressing ln⁡m\ln m on ln⁡c\ln c over the medium and hard scenes of each model, ln⁡m=a+bln⁡c\ln m = a + b\ln c, where 1−b1-b is the share corrected and aa a uniform bias (Table 19). Opus 5.5 and Grok 4.7 have intercepts near zero and correct 0.64 and 0.60 of the compression. GPT-5.5 corrects 0.18. Gemini corrects 0.58 but has an intercept of −0.25 [−0.30, −0.19], a further uniform compression of depth by a factor of 0.78.

mapscenesintercept aa [95% CI]slope bb [95% CI]share corrected 1−b1-b
Claude Opus 5.5606−0.01 [−0.04, +0.02]0.36 [0.32, 0.39]0.64
GPT-5.5606−0.01 [−0.07, +0.05]0.82 [0.76, 0.89]0.18
Gemini 3.6 Flash605−0.25 [−0.30, −0.19]0.42 [0.36, 0.48]0.58
Grok 4.7430−0.06 [−0.14, +0.02]0.40 [0.29, 0.49]0.60

Table 19. Constancy of the drawn maps as a fit of ln⁡m=a+bln⁡c\ln m = a + b\ln c over the medium and hard scenes, with 2,000-draw scene-bootstrap intervals. A copy of the picture has a=0a = 0 and b=1b = 1, and an undistorted map has a=0a = 0 and b=0b = 0.

Photographic items by source. Table 20 splits the CV-Bench gaps of Table 9 by source dataset and recomputes them with each source left out. The nuScenes street scenes give the largest Depth gaps for every model except GPT-5.5, on only 12 cue-wrong items, and without them the frontier Depth gaps are +7, +17 and +16 points for Opus 5.5, GPT-5.5 and Gemini.

task (cue)modelSUN RGB-DnuScenesHypersimwithout SUN RGB-Dwithout nuSceneswithout Hypersim
Depth (lower box)items (cue-wrong)200 (36)200 (12)200 (40)
Claude Opus 5.5+8 [+0, +19]+32 [+7, +58]+5 [−0, +12]+11 [+5, +21]+7 [+1, +13]+14 [+5, +25]
GPT-5.5+33 [+17, +50]+29 [+4, +57]+3 [−4, +11]+8 [+0, +17]+17 [+9, +26]+31 [+17, +44]
Gemini 3.6 Flash+22 [+8, +36]+57 [+30, +82]+11 [+1, +22]+21 [+11, +33]+16 [+8, +25]+30 [+18, +44]
Qwen3-VL-4B+2 [−1, +8]+35 [+9, +61]+3 [−3, +11]+9 [+1, +19]+3 [−1, +7]+9 [+0, +19]
Qwen3-VL-8B+6 [+0, +14]+39 [+12, +66]+3 [−3, +11]+8 [−1, +20]+4 [+0, +10]+11 [+1, +22]
Distance (pixel distance)items (cue-wrong)200 (14)200 (65)200 (40)
Claude Opus 5.5+14 [+0, +36]+16 [+7, +26]+24 [+11, +38]+20 [+12, +27]+22 [+11, +33]+17 [+9, +26]
GPT-5.5+14 [+0, +36]+65 [+54, +77]+30 [+18, +45]+52 [+43, +61]+26 [+15, +39]+57 [+46, +68]
Gemini 3.6 Flash+7 [+0, +21]+36 [+23, +49]+19 [+6, +32]+30 [+21, +40]+17 [+7, +28]+33 [+23, +44]
Qwen3-VL-4B+7 [+0, +21]+51 [+38, +64]+9 [−2, +21]+35 [+26, +45]+10 [+2, +20]+47 [+36, +58]
Qwen3-VL-8B+0 [+0, +0]+71 [+59, +81]+15 [+5, +28]+50 [+40, +59]+11 [+4, +20]+59 [+48, +70]

Table 20. The CV-Bench shortcut gap (points, 95% item-bootstrap intervals) within each source dataset and with each source left out. The first row of each block gives the number of items and, in parentheses, of cue-wrong items per source.

Photographs in more detail. The cue-wrong items hold 9 of Opus 5.5’s 11 Depth errors, 18 of GPT-5.5’s 30 and 20 of Gemini’s 26, and 24 of 28, 57 of 58 and 37 of 50 of their Distance errors. nuScenes supplies 65 of the 119 cue-wrong Distance items. CV-Bench releases no 3-D depths or distances, so the gaps cannot be stratified by the 3-D ratio without the Omni3D annotations, which we did not use. Accuracy on CV-Bench Depth is 98.2% for Opus 5.5 and at least 95.0% for the other frontier models. The lower-box gap is in line with the entanglement of height in the frame with distance that Min et al. [57] report. The larger box, which is right on only 59.5% of items, carries a gap of at most +3 points, with no interval above zero. The two probed models show gaps of +34 to +44 points on Distance. The two Depth cues do not add up, which is inconsistent with an observer that adds them and points to effects of item type. Accuracy is lowest where only the lower box misleads (79.2/62.5/62.5% for Opus 5.5, GPT-5.5 and Gemini on 24 items) and higher where the larger box misleads as well (93.8/85.9/82.8% on 64 items). An observer that weights a cue right on 85.3% of photographs will err where it fails (Brunswik, 1956), so these errors show the cue in use without showing that its use is irrational. In the rendered closer items height in the frame cannot be tested, because every object rests on the ground. The elevated tier, a small pilot split with objects on pedestals and stacks, balances closer on height in the frame, and there the frontier models answer 16, 16 and 15 of its 16 cue-wrong closer items correctly (Appendix Table 23). Where the support surface is visible, they do not follow height in the frame on these 16 items. Their Depth answers correlate with the lower-box cue at 0.73 to 0.75, slightly above the cue’s own correlation with the truth of 0.71, so their errors lean toward the cue. On Distance, Opus 5.5, GPT-5.5 and Gemini correlate with the pixel-distance cue at 0.67, 0.79 and 0.69 against the cue’s own correlation with the truth of 0.60, and follow it on 20, 48 and 31% of the items where it is wrong.

Prompts

Placeholders in braces are filled per item, and Appendix § 19 shows filled examples. The second-view input uses the image prompt on the other render.

image (default). Look at the image and decide whether the statement is true or false. The statement is about the actual 3D scene shown in the image (real sizes, distances and positions in the scene), not about the 2D picture. Statement: {statement} Answer with exactly one word: True or False.

chain of thought (suffix replacing the one-word instruction). Reason step by step about the 3D geometry first. Then give your final answer on its own last line, in exactly the form ’Answer: True’ or ’Answer: False’.

stated thresholds: numeric definitions and reject bands (prepended to the image prompt). Definitions used by the statements: ’closer to the camera/viewer’ compares the distance from the camera to each object’s centre (the statements only use pairs whose distances differ by at least 15%). ’Taller’ compares the objects’ real heights (at least 25% apart). ’Touching/in contact’ means the surfaces are within 0.02 units; ’not touching’ items are at least half the smaller object’s size apart. ’Between’ means the object’s ground position is within 10 degrees of the straight ground line joining the other two, as seen from either endpoint; ’not between’ items are at least 30 degrees off that line. ’Fits/can pass through the gap’ means the free gap between the two objects’ footprints on the ground is at least 1.2 times the moving object’s footprint width; ’does not fit’ means it is at most 0.8 times.

numeric definitions only, no bands (prepended, used in the prompt factorial of Table 14 and in the single-step inputs of Appendix § 10). Definitions used by the statements: ’closer to the camera/viewer’ compares the depth of each object’s centre along the camera’s viewing direction. ’Taller’ compares the objects’ real heights. ’Touching/in contact’ means the two objects’ surfaces are within 0.02 units of each other. ’Between’ means that, on the ground, the angle at each of the two end objects between the line to the other end object and the line to the middle object is at most 10 degrees. ’Fits/can pass through the gap’ means the free gap between the two objects’ footprints on the ground is wider than the moving object’s footprint width.

reject bands only (prepended, used in the prompt factorial of Table 14). About the statements: they only use clear cases. Objects compared for closeness differ in depth by at least 15%, and objects compared for height differ by at least 25%.

Objects described as not touching are at least half the smaller object’s size apart. Objects described as not between two others are at least 30 degrees off the line joining them. In statements about fitting through a gap, the gap is either at least 1.2 times or at most 0.8 times the moving object’s width.

verbal definitions without numbers or bands (prepended). Definitions used by the statements: ’closer to the camera/viewer’ compares how far each object’s centre is from the camera. ’Taller’ compares the objects’ real heights. ’Touching/in contact’ means the two objects’ surfaces actually meet; objects that merely look close or overlap in the picture but have space between them are not touching. ’Between’ means the object lies on (or very near) the straight line joining the other two; an object clearly off that line is not between them. ’Fits/can pass through the gap’ means the free space between the two objects is wider than the moving object; if the object is wider than the space it does not fit.

text only. Decide whether the following statement about a scene is true or false. No image is provided, so answer as best you can from the statement alone. Statement: {statement} Answer with exactly one word: True or False.

pixel measurements. You are shown a rendered 3D scene and, below, exact 2D measurements of every object in the picture, in pixels (image is {w}x{h}; x grows to the right, y grows downward). For each object: the bounding box of its visible pixels, the centroid of its visible pixels, the image position of the point where it touches the ground, the number of visible pixels, and the fraction of it hidden by other objects; then the minimum pixel distance between every pair of visible silhouettes (0 = overlapping). Use the image and these measurements to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. 2D measurements: {geometry} Statement: {statement} Answer with exactly one word: True or False.

pixel measurements, no image. Below are exact 2D measurements of every object in a picture of a 3D scene, in pixels (image is {w}x{h}; x grows to the right, y grows downward; the camera looks slightly down at objects resting on a flat ground). For each object: the bounding box of its visible pixels, the centroid of its visible pixels, the image position of the point where it touches the ground, the number of visible pixels, and the fraction of it hidden by other objects; then the minimum pixel distance between every pair of visible silhouettes (0 = overlapping). No image is provided. Decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. 2D measurements: {geometry} Statement: {statement} Answer with exactly one word: True or False.

pixel measurements + camera depth. You are shown a rendered 3D scene and, below, exact measurements of every object: its 2D measurements in the picture, in pixels (image is {w}x{h}; x grows to the right, y grows downward): the bounding box of its visible pixels, the centroid of its visible pixels, the image position of the point where it touches the ground, the number of visible pixels and the fraction hidden by other objects; and its exact distance from the camera in scene units (the same units as the objects’ sizes), for its centre and for its ground-contact point. The camera’s horizontal field of view is {hfov:.1f} degrees. Use the image and these measurements to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Measurements: {geometry} Statement: {statement} Answer with exactly one word: True or False.

depth map (no legend). You are shown two images. The first is a rendered 3D scene. The second is the exact ground-truth depth map of the same view, colour-coded: bright yellow is closest to the camera, green is intermediate, dark blue/purple is farthest, and black is sky. Use both images to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Statement: {statement} Answer with exactly one word: True or False.

depth map with metric legend. You are shown two images. The first is a rendered 3D scene. The second is the exact ground-truth depth map of the same view with a numeric colour bar: the colour of every pixel encodes its distance from the camera in scene units (the same units as the objects’ sizes), as read off the bar; the camera’s horizontal field of view is {hfov:.1f} degrees. Use both images to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Statement: {statement} Answer with exactly one word: True or False.

camera depth (text). You are shown a rendered 3D scene and, below, the exact distance from the camera to each object, in scene units (the same units as the objects’ sizes). Use the image and the distances to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Camera distances: {geometry} Statement: {statement} Answer with exactly one word: True or False.

top-down map. You are shown two images. The first is a rendered 3D scene. The second is an exact top-down map of the same scene: each object’s footprint on the ground is drawn to scale in the object’s colour and labelled, the grid lines are one scene unit apart, and the camera’s position and viewing direction are marked. Heights are not shown on the map. Use both images to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Statement: {statement} Answer with exactly one word: True or False.

top-down map without labels. You are shown two images. The first is a rendered 3D scene. The second is an exact top-down map of the same scene: each object’s footprint on the ground is drawn to scale in the object’s colour (no text labels), the grid lines are one scene unit apart, and the camera’s position and viewing direction are marked. Heights are not shown on the map. Identify the objects by their colours. Use both images to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Statement: {statement} Answer with exactly one word: True or False.

ground coordinates (text) with image. You are shown a rendered 3D scene and, below, the exact ground-truth geometry of every object in it. Coordinates are in a world frame with the ground plane at z=0z = 0, in the same length units as the sizes; the camera position is given too. Use the image and the geometry together to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Scene geometry: {geometry} Statement: {statement} Answer with exactly one word: True or False.

ground coordinates only (no image). Below is the exact geometry of every object in a 3D scene. Coordinates are in a world frame with the ground plane at z=0z = 0, in the same length units as the sizes; the camera position is given too. No image is provided. Decide whether the statement about this scene is true or false. Scene geometry: {geometry} Statement: {statement} Answer with exactly one word: True or False.

visible-region centres only (no image). Below is the image position, in pixels, of the centre of each object’s visible region in a picture of a 3D scene (image is {w}x{h}; x grows to the right, y grows downward; the camera looks down at objects resting on a flat ground). No image is provided. Decide whether the statement about the 3D scene is true or false. Visible-region centres: {geometry} Statement: {statement} Answer with exactly one word: True or False.

ground-contact pixels only (no image). Below is the exact image position, in pixels, of the point where each object’s footprint centre meets the ground, in a picture of a 3D scene (image is {w}x{h}; x grows to the right, y grows downward; the camera looks down at objects resting on a flat ground). No image is provided. Decide whether the statement about the 3D scene is true or false. Ground-contact points: {geometry} Statement: {statement} Answer with exactly one word: True or False.

ground-contact pixels + homography (no image). Below is the exact image position, in pixels, of the point where each object’s footprint centre meets the ground, in a picture of a 3D scene (image is {w}x{h}; xx grows to the right, yy grows downward), and the exact homography HH that maps image pixels to ground-plane coordinates in scene units: for a pixel (u,v)(u, v), compute (X,Y,W)=H(u,v,1)(X, Y, W) = H (u, v, 1) and the ground point is (X/W,Y/W)(X/W, Y/W). No image is provided. Decide whether the statement about the 3D scene is true or false. Ground-contact points: {geometry} Statement: {statement} Answer with exactly one word: True or False.

pairwise surface distances with image. You are shown a rendered 3D scene and, below, the exact minimum distance between the surfaces of every pair of objects in it, in scene units (the same units as the objects’ sizes; 0 means the surfaces meet). Use the image and the distances to decide whether the statement is true or false. The statement is about the actual 3D scene (real sizes, distances and positions), not about the 2D picture. Surface distances: {geometry} Statement: {statement} Answer with exactly one word: True or False.

drawn map fed back (no image, raw or depth-corrected layout). Below is a top-down layout of a 3D scene: each object’s ground position (x,y)(x, y) and footprint diameter, in scene units. xx increases to the right as seen from the camera and yy increases away from the camera. No image is provided. Decide whether the statement about this scene is true or false. Layout: {geometry} Statement: {statement} Answer with exactly one word: True or False.

map reconstruction task (separate task, JSON output). Look at the image of a 3D scene. All objects rest on a flat ground plane. Estimate the layout of the scene as seen from directly above: give each object’s ground position (x,y)(x, y) in a top-down coordinate frame of your choosing, in units of the scene (take the {ref} as having footprint diameter {refsize:.2f} units to set the scale), together with its footprint diameter. xx should increase to the right as seen from the camera and yy should increase away from the camera. Objects: {names}. Answer with a JSON object only, of the form {{"<object name>": {{"x": <number>, "y": <number>, "diameter": <number>}}, ...}} with one entry per object and no other text.

tool access, variant 1 (tools available, system prompt, answer tool offered from the first turn). You judge statements about 3D scenes from a single rendered image. You may use tools to inspect the image: zoom into regions and run Python over the pixel array to measure things (positions, extents, colours, apparent sizes). You have no access to ground truth. Reason about the actual 3D geometry (real sizes, distances and positions), not the 2D picture, and finish by calling answer. Use at most 10 tool calls.

tool access, variant 2 (tools required first, system prompt, answer tool withheld for two turns). You judge statements about 3D scenes from a single rendered image. You may use tools to inspect the image: zoom into regions and run Python over the pixel array to measure things (positions, extents, colours, apparent sizes). You have no access to ground truth. Reason about the actual 3D geometry (real sizes, distances and positions), not the 2D picture. Before answering, measure the quantities the statement depends on (e.g. silhouette extents, ground-contact pixel rows, gaps, apparent sizes) with the tools, and use perspective reasoning to convert them to 3D. Finish by calling answer. Use at most 10 tool calls.

tool access, user turn (both variants, the image prompt without its last line, followed by). Inspect the image with the tools as needed, then call answer.

tool access, variant 2 nudge (sent if the model answers in prose during the first two turns). Before answering, inspect the image with crop_zoom or measure with python. The answer tool becomes available after you have inspected.

Additional tables

Table 21: Difficulty tiers (generator seed 7). “Scenes sampled” counts scenes the balancer had to draw to fill all 20 cells.

tierobjectselevation (°)shadowsocclusionitemsscenesscenes sampled
easy4–545–60onlow6003431402
medium5–725–45onmedium600304315
hard6–88–20offhigh600302313

Table 21.

Table 22: Base-rate reliability of each 2-D cue over all candidate statements in 150 freshly sampled scenes per tier under random placement. Entries are agreement / balanced accuracy / prevalence of the true proposition (%), with the number of candidates in parentheses. The last column gives the agreement of the lower-in-the-frame cue for closer.

tierclosertallertouchingbetweenfitscloser: lower in frame
easy63 / 63 / 50 (532)87 / 87 / 50 (1754)96 / 98 / 3 (1008)100 / 99 / 3 (2741)89 / 91 / 63 (1722)100 (532)
medium58 / 58 / 50 (1534)91 / 91 / 50 (3062)89 / 94 / 2 (1824)99 / 97 / 3 (7830)82 / 84 / 59 (4172)100 (1534)
hard52 / 52 / 50 (2314)95 / 95 / 50 (3770)84 / 89 / 1 (2208)94 / 89 / 3 (10181)76 / 82 / 66 (5453)100 (2314)

Table 22.

Table 23: Passage relation and elevated tier, two small pilot splits, image condition, every model. A passage statement asks whether one object could be slid in a straight line to another without touching any other object. The elevated tier puts objects on pedestals and stacks and balances closer on height in the frame.

modelpassage (64 items)elevated tier (96 items)
accgap [95% CI]accgap [95% CI]
Claude Opus 5.589.1+15.6 [+1, +31]100.0+0.0 [+0, +0]
GPT-5.584.4+12.5 [−5, +30]96.9+6.2 [+0, +14]
Gemini 3.6 Flash81.2+6.2 [−12, +25]96.9+6.2 [+0, +14]
Pixtral Large57.8+3.1 [−22, +28]62.5+20.8 [+1, +40]
Qwen3-VL-235B64.1+3.1 [−20, +27]89.6+12.5 [+0, +25]
Llama 4 Maverick56.2−6.2 [−30, +17]71.9+14.6 [−4, +32]
Gemma 3 27B57.8+3.1 [−21, +28]56.2+16.7 [−2, +36]
Grok 4.775.0+12.5 [−9, +34]89.6+20.8 [+10, +34]
Nova 2 Pro62.5+0.0 [−22, +22]70.8+4.2 [−15, +23]

Table 23.

modelinputnnacccue rightcue wronggap [95% CI]
Pixtral Largeground coordinates only (no image)60063.366.060.7+5.3 [−2, +12]
Pixtral Largesecond view (own-camera cue)46556.658.942.6+16.3 [+3, +29]

Table 43.

tallertouchingbetweenfits
modelnaturalconstructednaturalconstructednaturalconstructednaturalconstructed
Claude Opus 5.5−3 (35)+3 (145)+0 (13)+31 (167)+57 (8)+41 (172)+23 (12)+27 (168)
GPT-5.5+20 (35)+21 (145)+69 (13)+38 (167)+57 (8)+43 (172)+78 (12)+62 (168)
Gemini 3.6 Flash+11 (35)+17 (145)+0 (13)+22 (167)+51 (8)+42 (172)+23 (12)+46 (168)

Table 25. Gap (points) on items whose referenced objects were placed by a sampler construct versus the natural remainder, pooled over the three tiers (number of cue-wrong items in parentheses).

fits gap (points)cue-wrong fits accuracy (%)
modellow efforthigh effortchange, 95% CIlow efforthigh effort
Claude Opus 5.5+32+10[−35, −8]6888
GPT-5.5+65+43[−38, −3]3357
Gemini 3.6 Flash+45+30[−32, +1]5068

Table 44.

modelplain ground accplain ground gapcheckerboard ground acccheckerboard ground gap [95% CI]
Claude Opus 5.585.7+21.386.8+18.3 [+13, +24]
GPT-5.578.5+35.079.2+31.0 [+25, +37]
Gemini 3.6 Flash81.7+24.781.8+23.7 [+18, +30]

Table 28. Accuracy (%) and gap (points) on the same medium scenes and statements with a plain and a checkerboard ground (image condition).

relationregressor (members)agree.VIFClaude Opus 5.5GPT-5.5Gemini 3.6 FlashGrok 4.7Qwen3-VL-235BPixtral LargeNova 2 ProLlama 4 MaverickGemma 3 27B
tallertruth1.4+4.0+2.6+4.1+1.8+1.0+0.6+0.8+0.5+0.1
pixel cues (pixel height, area)50/481.1+0.9+1.3+1.9+1.3+1.0+0.3+0.9+0.4+0.5
geometry-faithful (depth-corrected height, top y)69/711.5+0.3+1.1+0.6+1.0+1.2+0.6+0.9−0.5+0.4
touchingtruth2.2+2.2+0.7+0.9+0.6+0.7+0.2+0.9+0.3−0.8
silhouette cues (mask distance, bbox overlap)50/501.3+1.3+1.4+0.5+1.5+1.0+0.6+1.0+1.1+0.2
geometry-faithful (base points close)822.5−0.2+0.3+0.6+0.2+0.4+0.2+0.3+0.3+1.0
betweentruth3.9+3.3+2.0+2.2+1.7+0.6+0.2+0.4+0.1+0.5
image-plane collinearity (centroid collinear, bbox centre collinear, x between)50/63/641.3+3.5+2.8+2.7+2.4+0.9+0.0+0.7+0.1+0.3
geometry-faithful (base collinear)924.4+0.3−0.2−0.5−0.5−0.3+0.1−0.3+0.3−0.4
fitstruth1.6+2.3+1.0+1.1+0.7+0.3+0.6−0.2+0.0+0.3
pixel cues (pixel gap vs width, base gap vs width)50/561.8+1.5+1.9+1.1+1.0+0.4+0.8+0.0+0.3+0.2
geometry-faithful (depth-corrected)722.5+1.9+0.8+0.5+0.5−0.4−0.4+0.5−0.2−0.2

Table 29. Alternative cues on the medium tier. Geometry-faithful cues are those that respect perspective. A base point is a ground-contact pixel, top y is the image row of the visible top, x between tests the middle object’s image column and bbox is the visible box. The column “agree.” gives each member cue’s agreement (%) with the un-negated truth on the split’s items (the balanced cue is 50 by construction). Model columns are Firth-logit coefficients of the model’s answer on truth and on one composite per cue set (the mean of the members’ ±1 codes). VIF is the variance inflation factor in the Opus 5.5 fit. The relation closer is omitted because its fit is saturated.

Checks of the rules, renderer, solvers and adjustment

Formal predicates. Objects are convex primitives resting on the ground plane z=0z = 0 with footprint centre pi∈R2\mathbf{p}_i \in\mathbb{R}^{2}, size sis_i (cube side or diameter), height hih_i and, for cubes, yaw ψi\psi_i. Let d(⋅,⋅)d(\cdot,\cdot) be the exact minimum Euclidean distance between two solids (0 if they meet), dFd_F the minimum distance between two footprints in the ground plane, and z(⋅)z(\cdot) the depth of a point along the camera’s viewing axis. With margins as in Table 1, the propositions are

closer⁡(A,B):z(cA)<z(cB),ci=(pi,hi/2),taller⁡(A,B):hA>hB,touching⁡(A,B):d(A,B)<0.02,between⁡(C;A,B):max⁡(∠(pC−pA,pB−pA),∠(pC−pB,pA−pB))≤10∘,fits⁡(A;B,C):dF(B,C)/sA≥1.2 (true),≤0.8 (false)\begin{aligned} \operatorname{closer}(A,B)&: z(c_A)<z(c_B),\quad c_i=(p_i,h_i/2),\\ \operatorname{taller}(A,B)&: h_A>h_B,\\ \operatorname{touching}(A,B)&: d(A,B)<0.02,\\ \operatorname{between}(C;A,B)&: \max\bigl(\angle(p_C-p_A,p_B-p_A),\angle(p_C-p_B,p_A-p_B)\bigr)\le10^\circ,\\ \operatorname{fits}(A;B,C)&: d_F(B,C)/s_A\ge1.2\ \text{(true)},\le0.8\ \text{(false)} \end{aligned}

reject if max⁡(zA,zB)/min⁡(zA,zB)<1.15\max(z_A,z_B)/\min(z_A,z_B)<1.15, reject if max⁡(hA,hB)/min⁡(hA,hB)<1.25\max(h_A,h_B)/\min(h_A,h_B)<1.25, reject if 0.02≤d≤0.5min⁡(sA,hA,sB,hB)0.02\le d\le0.5\min(s_A,h_A,s_B,h_B), reject if 10∘<⋅<30∘10^\circ<\cdot<30^\circ, reject otherwise, if dF<0.1d_F<0.1, or if any other footprint is within 0.25 of the gap segment.

The between rule is angular. A point 0.2 units off the line but next to an endpoint is not between (a case in the test suite), and a point beyond an endpoint on the line is not between. The fits rule compares the minimum footprint gap with the narrowest width. It is a clearance test at the gap, not a general collision-free path. The 2-D cues apply the same rules to the visible masks of the main view, with pixel area (closer), visible height (taller), mask distance ≤3\le3 px (touching), the angular rule on visible-mask centroids at 20∘20^\circ (between), and pixel gap ≥\ge pixel width (fits). “Overlapping” means a mask distance of 0 on the analytic instance mask, where each pixel belongs to the nearest surface.

Hand-checked cases. The test suite covers spheres, cubes, rotated cubes, cylinders and cones at known distances. It covers unequal tangent spheres, which touch at horizontal centre separation 2r1r22\sqrt{r_1r_2} while their footprint discs overlap, so footprint overlap is not contact. It also covers cones touching at their bases, the angular between cases above, a third object inside a fit gap (rejected), the round trip of the image-to-ground homography and exact recovery of every label from the coordinate, homography and surface-distance serialisations on a sample split.

Renderer consistency. The 2-D statistics come from an analytic ray caster, while the models see Blender renders of tessellated meshes (96×48 sphere, 128-sided cylinders and cones). We re-rendered all 304 scenes of the first medium split (seed 7) with flat emission colours, no lights and a near-zero pixel filter, so the silhouettes of the actual meshes classify exactly. Per-object visible masks agree with the analytic masks to a median IoU of 0.9998 (1st percentile 0.998). Recomputing the balanced cues on the Blender masks flips 11 of 600 cue decisions, all at the 3-pixel touching rule, the pixel-width rule of fits or the pixel-height rule of taller. Relabelling those items changes the frontier models’ medium gaps to +19.4, +32.5 and +22.8, against +21.3, +35.0 and +24.7.

Solvers. The solvers of Table 5 regenerate each input’s serialisation with the evaluation code. On 80 sampled items per input the regenerated text is identical to the prompt sent to the model. The coordinate serialisation (two decimals) recovers all labels, including contact under rounding. The pixel-plus-depth serialisation recovers closer and between exactly. With intrinsics from the field of view and a central principal point, each ground-contact pixel and its camera depth give the contact point in camera coordinates, and the angles of between are invariant to the camera pose. This serialisation lacks object sizes, so fits, touching and taller need size estimates from pixel extents.

modeltierraw gapadjusted gap [95% CI]raw gap by margin tertile (T1/T2/T3)adjusted gap by margin tertile
Claude Opus 5.5easy+19.7+10.3 [+5, +15]+29 / +4 / −4+19 / +2 / −2
Claude Opus 5.5medium+21.3+14.6 [+11, +18]+33 / +10 / +0+28 / +10 / +0
Claude Opus 5.5hard+17.7+12.3 [+7, +17]+18 / +12 / +12+16 / +12 / +3
GPT-5.5easy+37.0+28.6 [+21, +35]+40 / +34 / +3+29 / +27 / +14
GPT-5.5medium+35.0+27.5 [+22, +33]+42 / +22 / +9+31 / +27 / +14
GPT-5.5hard+28.7+23.0 [+17, +29]+37 / +27 / +22+28 / +27 / +17
Gemini 3.6 Flasheasy+24.3+12.7 [+8, +18]+36 / +12 / +2+23 / +8 / −0
Gemini 3.6 Flashmedium+24.7+16.7 [+12, +22]+39 / +19 / +3+29 / +19 / +4
Gemini 3.6 Flashhard+23.7+16.4 [+12, +21]+34 / +22 / +8+29 / +19 / +5

Table 30. Difficulty-adjusted gap (points, image condition), the average marginal effect of the cue being right in a Firth logit with difficulty and construct covariates and interactions. CIs resample scenes 2,000 times. The tertile columns give the raw and conditional adjusted gap on the items of each rank-based within-relation tertile of the margin, T1 the smallest (touching excluded).

inputaccgap [95% CI]Δgap vs. low [95% CI]calls/item% used% python% zoomtools usednot usedclosertallertouchingbetweenfits
image, no tools (low effort)85.7+21.3 [+16, +27]———————+0+3+27+45+32
image, no tools (high effort, 16k)88.0+16.7 [+12, +22]−4.7 [−9, −1]——————+0+2+30+42+10
tools available86.3+16.0 [+11, +21]−5.3 [−9, −2]0.550223077 / +1596 / +11+0+0+22+40+18
tools required first86.0+17.3 [+12, +23]−4.0 [−8, −1]0.655253280 / +1893 / +13+0+2+22+42+22

Table 31. Tool-access intervention (Opus 5.5, medium tier). Δgap is the paired change against the low-effort image condition (2,000 scene-bootstrap draws). The high-effort row is the 16,384-token run. “% used” is the share of episodes with at least one tool call. The by-episode columns split accuracy / gap by whether tools were used, a self-selected split.

modelview 1view 2Δ [95% CI]gap, view 1gap, view 2gap, view 1gap, view 2
Claude Opus 5.581.986.5+4.5 [+1, +8]+26.4 [+20, +33]+11.4 [+6, +17]−7.4 [−16, +1]+3.1 [−6, +12]
GPT-5.572.783.2+10.5 [+6, +15]+44.0 [+37, +51]+11.8 [+5, +18]+2.5 [−9, +14]+30.3 [+19, +42]
Gemini 3.6 Flash76.882.6+5.8 [+2, +10]+30.7 [+23, +38]+12.2 [+5, +19]−8.3 [−17, +2]+8.9 [−2, +20]

Table 32. Second-view accuracy and shortcut gap on the same view-invariant medium items under two groupings, by the first camera’s cue (the balanced split, 231 cue-right / 234 cue-wrong) and by the second camera’s own cue (397 / 68, not balanced). Δ is the paired accuracy change from view 1 to view 2. CIs resample scenes (2,000).

Filled prompt examples

Item medium–0008 (between, cue-wrong, statement true). Statement: “The orange cylinder is between the green cube and the red sphere.”

pixel measurements (inserted at {geometry}):

  • green cube: bbox x 538–644, y 336–444 (width 107 px, height 109 px); centroid (590, 389); ground-contact point (585, 418); visible pixels 8631; hidden 0%

  • red sphere: bbox x 276–340, y 366–441 (width 65 px, height 76 px); centroid (307, 401); ground-contact point (316, 434); visible pixels 3485; hidden 23%

  • orange cylinder: bbox x 348–438, y 273–449 (width 91 px, height 177 px); centroid (396, 353); ground-contact point (393, 424); visible pixels 13038; hidden 10%

  • yellow cone: bbox x 350–402, y 356–526 (width 101 px, height 171 px); centroid (351, 463); ground-contact point (352, 492); visible pixels 9883; hidden 0%

  • blue cylinder: bbox x 429–546, y 441–612 (width 118 px, height 172 px); centroid (485, 525); ground-contact point (481, 572); visible pixels 17088; hidden 0% Silhouette distances: green cube / red sphere: 197 px; green cube / orange cylinder: 104 px; green cube / yellow cone: 155 px; green cube / blue cylinder: 37 px; red sphere / orange cylinder: 16 px; red

modelitemsimage + numeric definitions (acc / gap)drawn mapdrawn map, depth-correctedΔ\Deltagap [95% CI]answers = solver (%)
Claude Opus 5.524087.1 / +20.877.1 / +35.879.2 / +28.3−7.5 [−12, −3]100
GPT-5.524072.1 / +52.567.9 / +50.870.4 / +44.2−6.7 [−15, +2]100
Gemini 3.6 Flash24077.9 / +37.567.5 / +30.073.3 / +25.0−5.0 [−14, +4]100

Table 33. The model’s drawn map fed back as text (no image, numeric definitions), fits and between items of the medium tier. “Depth-corrected” stretches the map’s depth axis by the inverse of the median compression estimated on the other half of the scenes. Δ\Deltagap is the paired change from the drawn map to the depth-corrected map (scene bootstrap). The table pools the two relations, and § 5.4 gives the change for each. The last column is the share of items on which the model’s answer equals a solver reading the same layout.

relationmodel% offnn supported (cue right/wrong)rawsupportedIPWbinary construct [CI]construct type [CI]
tallerClaude Opus 5.59102 (43/59)+3+3+3+3 [−6, +13]+5 [−2, +14]
tallerGPT-5.59102 (43/59)+17+16+13+10 [−1, +27]+15 [+1, +33]
tallerGemini 3.6 Flash9102 (43/59)+17+17+15+14 [+4, +29]+15 [+4, +33]
touchingClaude Opus 5.52087 (42/45)+27+29+31+32 [+15, +44]+33 [+18, +48]
touchingGPT-5.52087 (42/45)+50+46+51+39 [+20, +57]+39 [+21, +59]
touchingGemini 3.6 Flash2087 (42/45)+18+20+16+19 [+3, +34]+19 [+2, +40]
betweenClaude Opus 5.54103 (51/52)+45+42+34+17 [−4, +25]+13 [−8, +35]
betweenGPT-5.54103 (51/52)+43+40+31+8 [−8, +25]+7 [−15, +23]
betweenGemini 3.6 Flash4103 (51/52)+43+42+33+11 [−4, +26]+10 [−7, +33]
fitsClaude Opus 5.52670 (31/39)+32+38+26+28 [+17, +40]+29 [+18, +46]
fitsGPT-5.52670 (31/39)+65+66+59+63 [+51, +76]+58 [+42, +76]
fitsGemini 3.6 Flash2670 (31/39)+45+46+43+41 [+26, +58]+47 [+32, +61]

Table 34. Common support for the difficulty adjustment (medium, image). The column “% off” gives the share of items with a propensity of being cue-right outside [0.05, 0.95] under a Firth propensity model on the difficulty covariates and proposition truth. The gaps are raw, raw on items with propensity in [0.1, 0.9], and inverse-propensity weighted (weights trimmed at [0.05, 0.95]). The adjusted gaps come from a within-relation Firth outcome model without the interaction of cue status with margin, with construct membership as a binary flag or as construct type and scene-bootstrap CIs.

sphere / yellow cone: 1 px; red sphere / blue cylinder: 114 px; orange cylinder / yellow cone: 1 px; orange cylinder / blue cylinder: 23 px; yellow cone / blue cylinder: 28 px

pixel measurements + camera depth (inserted at {geometry}):

  • green cube: bbox x 538–644, y 336–444 (width 107 px, height 109 px); centroid (590, 389); ground-contact point (585, 418); visible pixels 8631; hidden 0%; distance from camera: centre 9.91, ground-contact point 10.12

  • red sphere: bbox x 276–340, y 366–441 (width 65 px, height 76 px); centroid (307, 401); ground-contact point (316, 434); visible pixels 3485; hidden 23%; distance from camera: centre 9.65, ground-contact point 9.88

  • orange cylinder: bbox x 348–438, y 273–449 (width 91 px, height 177 px); centroid (396, 353); ground-contact point (393, 424); visible pixels 13038; hidden 10%; distance from camera: centre 9.58, ground-contact point 10.03

  • yellow cone: bbox x 302–402, y 356–526 (width 101 px, height 171 px); centroid (351, 463); ground-contact point (352, 492); visible pixels 9883; hidden 0%; distance from camera: centre 8.65, ground-contact point 9.10

  • blue cylinder: bbox x 429–546, y 441–612 (width 118 px, height 172 px); centroid (485, 525); ground-contact point (481, 572); visible pixels 17088; hidden 0%; distance from camera: centre 7.88, ground-contact point 8.20

Silhouette distances: green cube / red sphere: 197 px; green cube / orange cylinder: 104 px; green cube / yellow cone: 155 px; green cube / blue cylinder: 37 px; red sphere / orange cylinder: 16 px; red sphere / yellow cone: 1 px; red sphere / blue cylinder: 114 px; orange cylinder / yellow cone: 1 px; orange cylinder / blue cylinder: 23 px; yellow cone / blue cylinder: 28 px

camera depth (text) (inserted at {geometry}):

  • green cube: centre at 9.91 units from the camera, ground-contact point at 10.12

  • red sphere: centre at 9.65 units from the camera, ground-contact point at 9.88

  • orange cylinder: centre at 9.58 units from the camera, ground-contact point at 10.03

  • yellow cone: centre at 8.65 units from the camera, ground-contact point at 9.10

  • blue cylinder: centre at 7.88 units from the camera, ground-contact point at 8.20

ground coordinates (inserted at {geometry}):

  • green cube: ground position (x=0.01, y=−0.63), side 0.76, rotated 74 deg about the vertical, height 0.76

  • red sphere: ground position (x=1.57, y=1.89), diameter 0.80, height 0.80

  • orange cylinder: ground position (x=1.21, y=1.12), diameter 0.91, height 1.61

  • yellow cone: ground position (x=0.57, y=2.13), diameter 1.01, height 1.61

  • blue cylinder: ground position (x=−1.02, y=1.86), diameter 0.94, height 1.12

  • camera: position (x=−5.10, y=6.09, z=6.02), looking at the scene centre

visible-region centres (inserted at {geometry}):

  • green cube: (589.7, 389.2)

  • red sphere: (306.6, 401.0)

modeltiercloser: raw / shared [CI] / withintaller: raw / shared [CI] / withintouching: raw / shared [CI] / withinbetween: raw / shared [CI] / withinfits: raw / shared [CI] / within
Claude Opus 5.5easy−2 / −2 [−7, +0] / −3+5 / −0 [−9, +7] / +0+35 / +27 [+12, +40] / +14+42 / +20 [+8, +32] / +12+18 / +7 [−8, +18] / +7
Claude Opus 5.5medium+0 / +0 [−1, +3] / −0+3 / +1 [−4, +5] / +3+27 / +23 [+10, +35] / +27+45 / +25 [+16, +34] / +23+32 / +24 [+14, +34] / +27
Claude Opus 5.5hard+0 / −0 [−1, +0] / +0−3 / −6 [−14, −0] / −5+25 / +19 [+9, +31] / +24+37 / +22 [+9, +35] / +22+30 / +26 [+14, +40] / +38
GPT-5.5easy+0 / −0 [−1, +0] / +0+33 / +25 [+16, +35] / +24+50 / +49 [+34, +64] / +49+47 / +35 [+19, +49] / +21+55 / +34 [+13, +51] / +41
GPT-5.5medium+0 / −0 [−1, +0] / −0+17 / +11 [+1, +21] / +11+50 / +41 [+25, +56] / +38+43 / +29 [+12, +44] / +10+65 / +56 [+42, +69] / +61
GPT-5.5hard+0 / −0 [−6, +4] / +2+13 / +10 [+3, +17] / +15+22 / +19 [+6, +34] / +20+40 / +30 [+15, +45] / +25+68 / +57 [+40, +72] / +68
Gemini 3.6 Flasheasy−2 / −2 [−7, +0] / −3+23 / +16 [+8, +24] / +17+28 / +22 [+6, +36] / +2+43 / +20 [+8, +32] / +14+28 / +9 [−6, +22] / +4
Gemini 3.6 Flashmedium+0 / −0 [−1, +0] / −0+17 / +11 [+5, +18] / +14+18 / +15 [+1, +29] / +18+43 / +28 [+15, +41] / +17+45 / +29 [+15, +45] / +40
Gemini 3.6 Flashhard+0 / −0 [−1, +0] / +0+8 / +5 [+1, +11] / +22+13 / +11 [+0, +23] / +12+38 / +22 [+6, +35] / +26+58 / +44 [+32, +56] / +48

Table 35. Per-relation gap (points, image condition), raw / adjusted under the shared-slope model [95% CI] / adjusted under the within-relation model (Appendix B). This model keeps the interaction of cue status with margin, which Appendix Table 34 drops, so the two tables differ by a few points.

  • orange cylinder: (395.9, 352.8)

  • yellow cone: (351.2, 462.9)

  • blue cylinder: (485.2, 525.1)

ground-contact pixels (inserted at {geometry}):

  • green cube: (584.9, 417.7)

  • red sphere: (316.0, 434.1)

  • orange cylinder: (392.7, 423.8)

  • yellow cone: (352.5, 491.7)

  • blue cylinder: (481.2, 572.5)

ground-contact pixels + homography (inserted at {geometry}):

  • green cube: (584.9, 417.7)

  • red sphere: (316.0, 434.1)

  • orange cylinder: (392.7, 423.8)

  • yellow cone: (352.5, 491.7)

  • blue cylinder: (481.2, 572.5)

Homography H (rows): [-0.0184154, -0.0347219, 25.2953] [-0.0239956, 0.0356756, -2.59666] [2.04533e-19, 0.0041513, 1]

Item medium-0007 (touching, cue-wrong, statement true). Statement: “There is a gap between the pink cylinder and the cyan cylinder.”

pixel measurements (inserted at {geometry}):

  • red cylinder: bbox x 493–587, y 230–312 (width 95 px, height 83 px); centroid (539, 271); ground-contact point (538, 286); visible pixels 6559; hidden 0%;

  • pink cylinder: bbox x 287–393, y 172–375 (width 107 px, height 204 px); centroid (336, 260); ground-contact point (346, 345); visible pixels 15999; hidden 18%;

  • purple sphere: bbox x 435–484, y 277–371 (width 50 px, height 95 px); centroid (461, 324); ground-contact point (436, 361); visible pixels 2917; hidden 59%;

  • cyan cylinder: bbox x 355–446, y 268–455 (width 92 px, height 188 px); centroid (399, 357); ground-contact point (399, 427); visible pixels 14932; hidden 3%;

  • blue cone: bbox x 367–436, y 428–529 (width 70 px, height 102 px); centroid (401, 490); ground-contact point (402, 504); visible pixels 4279; hidden 0%

Silhouette distances: red cylinder / pink cylinder: 100 px; red cylinder / purple sphere: 19 px; red cylinder / cyan cylinder: 48 px; red cylinder / blue cone: 160 px; pink cylinder / purple sphere: 41 px; pink cylinder / cyan cylinder: 1 px; pink cylinder / blue cone: 71 px; purple sphere / cyan cylinder: 1 px; purple sphere / blue cone: 70 px; cyan cylinder / blue cone: 1 px

pixel measurements + camera depth (inserted at {geometry}):

  • red cylinder: bbox x 493–587, y 230–312 (width 95 px, height 83 px); centroid (539, 271); ground-contact point (538, 286); visible pixels 6559; hidden 0%; distance from camera: centre 10.79, ground-contact point 10.02

  • pink cylinder: bbox x 287–393, y 172–375 (width 107 px, height 204 px); centroid (336, 260); ground-contact point (346, 345); visible pixels 15999; hidden 18%; distance from camera: centre 9.58, ground-contact point 10.14

  • purple sphere: bbox x 435–484, y 277–371 (width 50 px, height 95 px); centroid (461, 324); ground-contact point (436, 361); visible pixels 2917; hidden 59%; distance from camera: centre 9.64, ground-contact point 9.95

  • cyan cylinder: bbox x 355–446, y 268–455 (width 92 px, height 188 px); centroid (399, 357); ground-contact point (399, 427); visible pixels 14932; hidden 3%; distance from camera: centre 8.73, ground-contact point 9.23

modelscenes parseditemsbothdirect onlymap onlyneitheragree %ϕ\phiMH odds ratio% in bandparsedparse-incl.
Claude Opus 5.5304/304480305913054750.342.423+20+20
GPT-5.5304/3044802451182790700.382.131+38+38
Gemini 3.6 Flash304/3044802601202179710.393.928+30+30
Grok 4.7224/304353188753159700.332.325+18+16

Table 37. Item-level correspondence between the direct image answer and the answer re-derived from the model’s drawn map (medium tier, taller excluded because heights are not mapped). The Mantel–Haenszel odds ratio pools the 2×22 \times2 tables within relation ×\times cue status strata, and ϕ\phi is the correlation between the two correctness indicators. “% in band” is the share of re-derived relations that fall inside a reject band, which count as wrong. The map gap is given on parsed scenes and with unparsed scenes counted as wrong.

modelinput≤10∘\le10^\circ (true)3030–40∘40^\circ4040–60∘60^\circ≥60∘\ge60^\circ
Claude Opus 5.5image100 (60)18 (17)24 (17)100 (26)
Claude Opus 5.5verbal definitions100 (60)53 (17)47 (17)100 (26)
Claude Opus 5.5stated thresholds98 (60)71 (17)82 (17)100 (26)
Claude Opus 5.5coordinates only100 (60)29 (17)59 (17)100 (26)
Claude Opus 5.5coordinates only + verbal definitions100 (60)76 (17)94 (17)100 (26)
Claude Opus 5.5coordinates + image100 (60)35 (17)88 (17)100 (26)
GPT-5.5image98 (60)12 (17)12 (17)96 (26)
GPT-5.5verbal definitions75 (60)41 (17)41 (17)100 (26)
GPT-5.5stated thresholds88 (60)35 (17)47 (17)100 (26)
GPT-5.5coordinates only100 (60)29 (17)65 (17)100 (26)
GPT-5.5coordinates only + verbal definitions100 (60)100 (17)94 (17)100 (26)
GPT-5.5coordinates + image100 (60)47 (17)82 (17)100 (26)
Gemini 3.6 Flashimage97 (60)18 (17)18 (17)100 (26)
Gemini 3.6 Flashverbal definitions90 (60)41 (17)47 (17)100 (26)
Gemini 3.6 Flashstated thresholds92 (60)35 (17)47 (17)100 (26)
Gemini 3.6 Flashcoordinates only98 (60)41 (17)71 (17)100 (26)
Gemini 3.6 Flashcoordinates only + verbal definitions97 (60)100 (17)100 (17)100 (26)
Gemini 3.6 Flashcoordinates + image100 (60)41 (17)76 (17)100 (26)

Table 38. between accuracy (%) by deviation of the third object from the line, medium tier, with item counts. The 3030–60∘60^\circ bands hold all 30 cue-wrong false items and 4 cue-right ones. The ≥60∘\ge60^\circ band holds the other 26 cue-right false items.

  • blue cone: bbox x 367–436, y 428–529 (width 70 px, height 102 px); centroid (401, 490); ground-contact point (402, 504); visible pixels 4279; hidden 0%; distance from camera: centre 8.22, ground-contact point 8.52

Silhouette distances: red cylinder / pink cylinder: 100 px; red cylinder / purple sphere: 19 px; red cylinder / cyan cylinder: 48 px; red cylinder / blue cone: 160 px; pink cylinder / purple sphere: 41 px; pink cylinder / cyan cylinder: 1 px; pink cylinder / blue cone: 71 px; purple sphere / cyan cylinder: 1 px; purple sphere / blue cone: 70 px; cyan cylinder / blue cone: 1 px

camera depth (text) (inserted at {geometry}):

  • red cylinder: centre at 10.79 units from the camera, ground-contact point at 10.92

  • pink cylinder: centre at 9.58 units from the camera, ground-contact point at 10.14

  • purple sphere: centre at 9.64 units from the camera, ground-contact point at 9.95

  • cyan cylinder: centre at 8.73 units from the camera, ground-contact point at 9.23

  • blue cone: centre at 8.22 units from the camera, ground-contact point at 8.52

ground coordinates (inserted at {geometry}):

  • red cylinder: ground position (x=1.41, y=0.89), diameter 0.98, height 0.40

  • pink cylinder: ground position (x=−0.73, y=1.76), diameter 0.97, height 1.70

  • purple sphere: ground position (x=−0.33, y=0.91), diameter 0.92, height 0.92

  • cyan cylinder: ground position (x=−1.29, y=−0.57), diameter 0.76, height 1.51

  • blue cone: ground position (x=−1.99, y=−0.06), diameter 0.59, height 0.90

  • camera: position (x=−6.41, y=−3.75, z=6.37), looking at the scene centre

pairwise surface distances (inserted at {geometry}):

  • red cylinder / pink cylinder: 1.341

  • red cylinder / purple sphere: 0.794

  • red cylinder / cyan cylinder: 1.857

  • red cylinder / blue cone: 2.748

  • pink cylinder / purple sphere: 0.000

  • pink cylinder / cyan cylinder: 0.457

  • pink cylinder / blue cone: 1.435

  • purple sphere / cyan cylinder: 0.188

  • purple sphere / blue cone: 1.235

  • cyan cylinder / blue cone: 0.265

Table 39: Two-view consistency on the view-invariant relations (2-D cue on every tier, models on the medium tier). Agreement is the share (%) of items answered identically on both cameras.

modeltiernnagreementboth correctcue flipsagreement on flipsagreement on non-flips
2-D cueeasy47662.648.31780.0100.0
2-D cuemedium46556.145.62040.0100.0
2-D cuehard41652.444.21980.0100.0
Claude Opus 5.5medium46586.977.620479.992.3
GPT-5.5medium46574.865.420457.488.5
Gemini 3.6 Flashmedium46580.069.720471.187.0

Table 39.

Table 40: Models, the model identifiers returned by the providers and the reasoning effort. All calls used provider-default sampling, PNG images and a 4,096-token completion budget, except the 16,384-token high-effort and map runs.

modelmodel identifierreasoning effort
Claude Opus 5.5claude-opus-5-5low, high
Gemini 3.6 Flashgemini-3.6-flashlow, high
Grok 4.7global.xai.grok-4.7provider default
Gemma 3 27Bgoogle.gemma-3-27b-itprovider default
Qwen3-VL-235Bqwen.qwen3-vl-235b-a22bprovider default
Nova 2 Prous.amazon.nova-2-pro-preview-20251202-v1:0provider default
Llama 4 Maverickus.meta.llama4-maverick-17b-instruct-v1:0provider default
Pixtral Largeus.mistral.pixtral-large-2502-v1:0provider default
GPT-5.5gpt-5.5-2026-04-24low, high

Table 40.

Generation details

Occlusion level bounds the scene’s maximum per-object occlusion, at most 0.45 for low, within [0.10,0.60][0.10, 0.60] for medium and within [0.20,0.65][0.20, 0.65] for high. Referenced objects must be at least 50% visible. The disc of object positions is shrunk to what the lens can frame (min⁡(6,0.72dtan⁡(hfov/2))\min(6, 0.72 d \tan(\mathrm{hfov}/2)), with dd the camera distance and hfov\mathrm{hfov} the horizontal field of view). The sampler uses four constructs. They are an exact-tangent pair (bisection on the convex surface distance), an occluder over a contact line or an object’s base (prisms preferred, with the height chosen so that the sight line is blocked), a collinear triple with short endpoints and a tall middle object or tall endpoints and a short offset object, and a near gap seen side-on with a far object slightly wider than the gap. The between thresholds are 10∘10^\circ (true) and 30∘30^\circ (false) with the 2-D test at 20∘20^\circ. The full rules and the log of hand-inspected items will accompany the code release.

Parse rates and token usage

Appendix Table 41 gives the parse and truncation rates, and Appendix Table 42 the calls and tokens.

Table 41: Unparseable, API-error and budget-truncated response rates (%) on the image condition, all tiers pooled.

modelresponsesunparseable (%)API error (%)truncated (%)
Claude Opus 5.518000.00.00.0
Claude Opus 5.5 (high effort)6000.00.00.0
GPT-5.518000.00.00.0
GPT-5.5 (high effort)6000.30.00.3
Gemini 3.6 Flash18000.00.00.0
Gemini 3.6 Flash (high effort)6000.00.00.7
Pixtral Large18000.00.00.0
Qwen3-VL-235B18000.00.00.0
Llama 4 Maverick18000.00.00.0
Gemma 3 27B18000.00.00.0
Grok 4.718002.20.02.2
Nova 2 Pro18000.00.00.0

Table 41.

modelcallsinput tokens (M)output tokens (M)
Claude Opus 5.521,49126.85.66
Claude Opus 5.5 (high effort)1,2001.10.32
GPT-5.520,19117.511.17
GPT-5.5 (high effort)1,2000.93.58
Gemini 3.6 Flash20,19125.68.81
Gemini 3.6 Flash (high effort)1,2001.42.45
Pixtral Large8,43121.50.44
Qwen3-VL-235B8,4316.60.33
Llama 4 Maverick8,43126.42.56
Gemma 3 27B8,4314.00.24
Grok 4.78,4316.513.05
Nova 2 Pro8,4318.30.33
total116,059

Table 42. Calls and tokens over every input on the rendered tiers. The total leaves out the CV-Bench calls and the calls to the two probed models, which ran locally.

Proposed human study protocol (not run)

The code release will include a rating interface that shows an item with no labels, records a True / False / cannot-tell answer and a confidence, and a scorer that computes the same metrics as for models, including the adjusted gap, plus Cohen’s κ\kappa between raters. The planned design is at least ten naive raters on the 600 medium items, single-view presentation, confidence and response time recorded, majority vote, and the human raw and adjusted gaps reported beside the models, with the 49 items that are cue-wrong in both views as the difficulty control cell. Viewing should be monocular, with a chin rest at the distance at which the image subtends its rendered horizontal field of view, since perceived depth in pictures changes with viewing distance [13]. A second, free-viewing condition and a map-drawing version of the task would place human and model compression on the constancy index of Appendix § 15. This study was not run, and no other study in the paper replaces it.

Error examples

We show three cue-wrong items that Opus 5.5 gets wrong with chain of thought, the median-length failure in each of three relations. The rationales are illustrative outputs, not a faithful trace of the computation. On a fits item the model estimates “the cube’s width (about 100 px at that depth)” and “the gap (about 60–70 px)” and attempts a perspective correction, which it gets wrong. On a between item it reasons on the ground plane and then accepts a “roughly midway” offset, a looser test than our 10∘10^{\circ} rule. On a touching item it concludes from a “margin [of] only a few pixels”. Appendix Figure 7 shows the three items.

Three cue-wrong items answered wrongly by Claude Opus 5.5 with chain of thought

Figure 7. Cue-wrong items answered wrongly by Claude Opus 5.5 with chain of thought (medium tier). From left to right, fits, between and touching.

fits, medium-0130. The pink cube would fit through the gap between the purple cylinder and the orange sphere. Truth: true, 2-D cue: false, model: false. Rationale excerpt: “[...] If it were moved back to the depth of the cylinder and sphere, perspective would shrink it to roughly 100 px. 4. Comparison: The cube’s width (about 100 px at that depth) is clearly larger than the gap (about 60–70 px). Even rotated to its narrowest orientation, a cube’s minimum width is its side length, so it cannot pass through.”

between, medium-0185. The green cube lies in between the purple cone and the orange cylinder. Truth: false, 2-D cue: true, model: true. Rationale excerpt: “[...] So on the actual ground plane the offset is somewhat larger than it looks, but still modest relative to the distance between the cone and the cylinder. The objects form a rough chain on the ground: cone, then sphere, then cube, then cylinder. The cube is not exactly on the line, but it sits roughly midway between the cone and the cylinder, so I judge the statement true.”

touching, medium-0580. The green sphere and the cyan cylinder are in contact with each other. Truth: true, 2-D cue: false, model: false. Rationale excerpt: “[...] So they are also offset in depth, which increases the true ground-plane distance further. Conclusion: These measurements suggest a small gap between the sphere and the cyan cylinder, with the yellow cylinder near both of them. The margin is only a few pixels, so this is a close call, but they most likely do not touch.”

Datasheet

Motivation. CueClash was created to measure whether vision-language models answer 3-D spatial statements from scene geometry or from image-plane regularities. It was built by the author at Scale AI. No external funding is involved.

Composition. Three splits (easy, medium, hard) of 600 true/false statements each from seed 7, a second 600-item medium split from seed 11, 1,252 rendered scenes of 4–8 primitives across these four splits, plus a 600-item checkerboard-ground re-render of the medium scenes and a 50-item sample inspected by hand. Each item carries the statement, its truth, the 2-D cue’s prediction, whether the cue is right or wrong, relation, margin, polarity, paraphrase variant, referenced object indices, the rendered image (768×768768 \times768 PNG), a 16-bit depth map, a colourised depth visualisation, and for view-invariant relations a second-camera render with its own cue prediction. Each scene carries the full scene graph (object shape, colour, position, size, height, yaw), camera and light parameters, per-object analytic 2-D statistics (visible mask area, occlusion, centroid, ground-contact pixel, pixel area, depth) and the list of sampler constructs used. Every model response used in this paper will be released as a JSON record (prompt, raw response, parsed answer, token usage, latency). No people, personal data or offensive content appear, and all images are synthetic.

Collection process. Scenes are drawn by a seeded rejection sampler (Appendix § 20). Statements are enumerated exhaustively per scene, labelled from the scene graph and selected by an exact balancer. Rendering uses Blender 5.0.1 Cycles, and masks and depth come from an analytic ray caster over the same primitives. Model responses were collected in October 2026 through an OpenAI-compatible proxy with the parameters in § 4. No human annotation was used. A rating interface for a human study will be released. No ratings were collected.

Preprocessing and labelling. None beyond the generator. Truth and cue labels are computed, not annotated. The reject bands in Table 1 discard near-threshold candidates.

Uses. Intended for evaluating 3-D spatial understanding of VLMs and for studying cue reliance. The generator can produce arbitrarily many fresh items from a new seed, so training on the benchmark splits defeats the purpose and is discouraged. Not intended as a measure of performance on natural photographs.

Distribution and maintenance. Code (generator, evaluation harness, analysis, rating tool) will be released under the MIT licence, and rendered data, labels and model responses under CC BY 4.0. Code, data and model responses will be hosted in a public repository. A DOI and Croissant metadata will accompany the data release. The author maintains the benchmark, and issues and new-model submissions will be accepted through that repository. Version 1.0 will hold the four splits evaluated here, the easy, medium and hard splits from seed 7 and the second medium split from seed 11. Any change to generation rules or margins will be a new version with its own results table.

Ethical considerations. The data are synthetic and carry no privacy or consent risk. The main ethical consideration is interpretive. The shortcut gap is a one-sided instrument (a zero gap shows no cue-right advantage for the named cue, not that the cue is never used), and we ask users not to read a small gap at chance-level accuracy as 3-D competence. The gap measures alignment of answers with a cue, and reading it as reliance needs the input controls described in the paper.

References

  1. [1]Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4971–4980, 2018. URL https://arxiv.org/abs/1712.00377.
  2. [2]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2016. URL https://arxiv.org/abs/1610.01644.
  3. [3]Shehreen Azad, Yash Jain, Rishit Garg, Yogesh S. Rawat, and Vibhav Vineet. Understanding depth and height perception in large visual-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2025. URL https://arxiv.org/abs/2408.11748.
  4. [4]Shuai Bai et al. Qwen3-VL technical report, 2025. URL https://arxiv.org/abs/2511.21631.
  5. [5]Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022.arxiv.org/abs/2102.12452
  6. [6]Marco Bertamini, Tyrone L. Yang, and Dennis R. Proffitt. Relative size perception at a distance is best at eye level. Perception & Psychophysics, 60(4):673–682, 1998. doi: 10.3758/BF03206054.DOI
  7. [7]Blender Online Community. Blender – a 3D modelling and rendering package. https://www.blender.org, 2025. Version 5.0.1 (bpy Python module).
  8. [9]Wenxiao Cai, Jaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. SpatialBot: Precise spatial understanding with vision language models. In IEEE International Conference on Robotics and Automation (ICRA), 2025. arXiv:2406.13642.arxiv.org/abs/2406.13642
  9. [10]Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14555–14465, 2024. URL https://arxiv.org/abs/2401.12168.
  10. [11]Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for VLMs? An attention mechanism perspective on focus areas. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 9910–9932. PMLR, 2025. URL https://proceedings.mlr.press/v267/chen25cr.html.arxiv.org/abs/2503.01773
  11. [12]An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. SpatialRGPT: Grounded spatial reasoning in vision-language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/f38cb4cf9a5eaa92b3cfa481832719c6-Abstract.html.arxiv.org/abs/2406.01584
  12. [13]Emily A. Cooper, Elise A. Piazza, and Martin S. Banks. The perceptual basis of common photographic practice. Journal of Vision, 12(5):8, 2012.DOI
  13. [14]James E. Cutting and Peter M. Vishton. Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In William Epstein and Sheena Rogers, editors, Perception of Space and Motion, volume 5 of Handbook of Perception and Cognition, pages 69–117. Academic Press, San Diego, CA, 1995. doi: 10.1016/B978-012240530-3/50005-5.DOI
  14. [15]Duolikun Danier, Mehmet Aygün, Changjian Li, Hakan Bilen, and Oisin Mac Aodha. DepthCues: Evaluating monocular depth perception in large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20049–20059, 2025. Ailin Deng, Tri Cao, Zhirui Chen, and Bryan Hooi. Words or vision: Do vision-language models have blind faith in text? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3867–3876, 2025. arXiv:2503.02199.
  15. [16]Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346–355. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-short.33. URL https://aclanthology.org/2024.acl-short.33/.DOI
  16. [17]Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1): 1–26, 1979. doi: 10.1214/aos/1176344552.DOI
  17. [18]Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  18. [19]David Firth. Bias reduction of maximum likelihood estimates. Biometrika, 80(1):27–38, 1993. doi: 10.1093/biomet/80.1.27.DOI
  19. [20]Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. Hidden in plain sight: VLMs overlook their visual representations. In Conference on Language Modeling (COLM), 2025. URL https://openreview.net/forum?id=qQb1JLrwoI.arxiv.org/abs/2506.08008
  20. [21]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. doi: 10.1007/978-3-031-73337-6_9. URL https://arxiv.org/abs/2404.12390.
  21. [22]Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.arxiv.org/abs/1811.12231
  22. [23]Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1):665–673, 2020. doi: 10.1038/s42256-020-00257-z.DOI
  23. [24]James J. Gibson. The Perception of the Visual World. Houghton Mifflin, Boston, 1950.
  24. [25]Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3D: A minimally contrastive benchmark for grounding spatial relations in 3D. In Advances in Neural Information Processing Systems (NeurIPS), 2020. arXiv:2012.01634.arxiv.org/abs/2012.01634
  25. [26]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi: 10.1109/CVPR.2017.670. URL https://arxiv.org/abs/1612.00837.
  26. [27]Wes Gurnee and Max Tegmark. Language models represent space and time. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=jE8xbmvF1n.arxiv.org/abs/2310.02207
  27. [28]Stefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, and Felix Bißmann. On the interpretation of weight vectors of linear models in multivariate neuroimaging. NeuroImage, 87:96–110, 2014. doi: 10.1016/j.neuroimage.2013.10.067.DOI
  28. [29]Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3635–3644. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.findings-acl.318. URL https://aclanthology.org/2021.findings-acl.318/.DOI
  29. [30]John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, 2019.arxiv.org/abs/1909.03368
  30. [31]John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher D. Manning. Conditional probing: Measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1626–1639, 2021.
  31. [32]Alfred H. Holway and Edwin G. Boring. Determinants of apparent visual size with distance variant. The American Journal of Psychology, 54(1):21–37, 1941. doi: 10.2307/1417790.DOI
  32. [33]Yining Hong, Li Yi, Joshua B. Tenenbaum, Antonio Torralba, and Chuang Gan. PTR: A benchmark for part-based conceptual, relational, and physical reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2112.05136
  33. [34]Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi: 10.1109/CVPR.2019.00686. URL https://arxiv.org/abs/1902.09506.
  34. [35]Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901–2910, 2017. URL https://arxiv.org/abs/1612.06890.
  35. [36]Omri Kaduri, Shai Bagon, and Tali Dekel. What’s in the image? A deep-dive into the vision of vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549–14558, 2025. URL https://openaccess.thecvf.com/content/CVPR2025/html/Kaduri_Whats_in_the_Image_A_Deep-Dive_into_the_Vision_of_CVPR_2025_paper.html.arxiv.org/abs/2411.17491
  36. [37]Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? Investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9161–9175. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.568. URL https://aclanthology.org/2023.emnlp-main.568/.
  37. [38]Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should VQA expect them to? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2776–2785, 2021. doi: 10.1109/CVPR46437.2021.00280. URL https://arxiv.org/abs/2006.05121.
  38. [39]David C. Knill and Whitman Richards, editors. Perception as Bayesian Inference. Cambridge University Press, Cambridge, UK, 1996. ISBN 9780521461092. doi: 10.1017/CBO9780511984037.DOI
  39. [40]Jan J. Koenderink, Andrea J. van Doorn, and Astrid M. L. Kappers. Surface perception in pictures. Perception & Psychophysics, 52(5):487–496, 1992. doi: 10.3758/BF03206710.DOI
  40. [41]Jan J. Koenderink, Andrea J. van Doorn, Astrid M. L. Kappers, and James T. Todd. Ambiguity and the ‘mental eye’ in pictorial relief. Perception, 30(4):431–448, 2001.pubmed.ncbi.nlm.nih.gov/11383191
  41. [42]Michael S. Landy, Laurence T. Maloney, Elizabeth B. Johnston, and Mark Young. Measurement and modeling of depth cue combination: In defense of weak fusion. Vision Research, 35(3):389–412, 1995. doi: 10.1016/0042-6989(94)00176-M.DOI
  42. [43]Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, and Ivan Vulić. TopViewRS: Vision-language models as top-view spatial reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1786–1807, 2024. doi: 10.18653/v1/2024.emnlp-main.106. arXiv:2406.02537.DOI
  43. [44]Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 41451–41530, 2023. doi: 10.52202/075280-1797. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/81b8390039b7302c909cb769f8b6cd93-Abstract-Conference.html.DOI
  44. [45]Chih-Ting Liao, Fei Shen, Xin Cao, and Tat-Seng Chua. Decodable is not grounded: A vision-ablation arbiter for VLM spatial reasoning, 2026. URL https://arxiv.org/abs/2606.31257. arXiv preprint.
  45. [46]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. doi: 10.1162/tacl_a_00566. URL https://aclanthology.org/2023.tacl-1.37/.DOI
  46. [47]Yiqian Liu, Iuliia Kotseruba, and John K. Tsotsos. Disentangling pictorial cue understanding from language bias in VLMs via depth ordering task. In Proceedings of the European Conference on Computer Vision (ECCV), 2026a. URL https://arxiv.org/abs/2607.01503.
  47. [48]Zhaocheng Liu, Kaiwen Gao, Shuyi Liang, Bin Xiao, Limeng Qiao, Lin Ma, and Tingting Jiang. Beyond the visible: Benchmarking occlusion perception in multimodal large language models. arXiv preprint arXiv:2508.04059, 2025.arxiv.org/abs/2508.04059
  48. [49]Zhining Liu, Ziyi Chen, Hui Liu, Chen Luo, Xianfeng Tang, Suhang Wang, Jingying Zeng, Zhenwei Dai, Zhan Shi, Tianxin Wei, Hanqing Lu, Benoit Dumoulin, and Hanghang Tong. Seeing but not believing: Probing the disconnect between visual attention and answer correctness in VLMs. In International Conference on Learning Representations (ICLR), 2026b. URL https://openreview.net/forum?id=JA7fWA9e.arxiv.org/abs/2510.17771
  49. [50]Jack M. Loomis and John W. Philbeck. Is the anisotropy of perceived 3-D shape invariant across scale? Perception & Psychophysics, 61(3):397–402, 1999. doi: 10.3758/BF03211961.DOI
  50. [51]Jack M. Loomis, José A. Da Silva, Naofumi Fujita, and Sergio S. Fukushima. Visual space perception and visually directed action. Journal of Experimental Psychology: Human Perception and Performance, 18(4):906–921, 1992. doi: 10.1037/0096-1523.18.4.906.DOI
  51. [52]Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso M. de Melo, and Alan Yuillie. 3DSRBench: A comprehensive 3D spatial reasoning benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6924–6934, 2025. URL https://arxiv.org/abs/2412.07825.
  52. [53]Animesh Maheshwari, Divyansh Sahu, and Nishit Verma. Do vision-language models understand 3D scenes or just catalogue objects? arXiv preprint arXiv:2605.20448, 2026.arxiv.org/abs/2605.20448
  53. [54]Hanspeter A. Mallot, Heinrich H. Bülthoff, James J. Little, and Stefan Bohrer. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological Cybernetics, 64(3):177–185, 1991. doi: 10.1007/BF00201978.DOI
  54. [55]Pascal Mamassian, David C. Knill, and Daniel Kersten. The perception of cast shadows. Trends in Cognitive Sciences, 2(8):288–295, 1998. doi: 10.1016/S1364-6613(98)01204-2.DOI
  55. [56]Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM), 2024. URL https://openreview.net/forum?id=aajyYHjisk.arxiv.org/abs/2310.06824
  56. [57]Cheolhong Min, Jaeyun Jung, Daeun Lee, Hyeonseong Jeon, Yu Su, Jonathan Tremblay, Chan Hee Song, and Jaesik Park. Why far looks up: Probing spatial representation in vision-language models. In Proceedings of the European Conference on Computer Vision (ECCV), 2026. arXiv:2605.30161.arxiv.org/abs/2605.30161
  57. [58]Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=chanJGoa7f.arxiv.org/abs/2410.07149
  58. [59]Teng Leng Ooi, Bing Wu, and Zijiang J. He. Distance determined by the angular declination below the horizon. Nature, 414(6860):197–200, 2001. doi: 10.1038/35102562.

Paper details

vlnc:W_2qr5tr92
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents