betaFeedback

WorksAgricultural and Biological Sciences

Vision-Language Models Reproduce a Human Mental-Extrapolation Bias in the Watanabe Illusion

Abstract

Vision-language models (VLMs) are increasingly used for visual judgments, yet whether they share human perceptual biases, and where in the model such biases arise, is unclear. We addressed this with the Watanabe Illusion, a stimulus not previously studied quantitatively: an oblique segment inside a circle whose extension geometrically hits the topmost of 11 dots, yet human observers (n = 131) reported a dot near the middle of the column. Five VLMs (Qwen2.5-VL-32B and 7B, Claude Sonnet 4.6, GPT-5.5 Instant, Gemini 3.5 Flash), given the identical stimulus and wording, all deviated in the same direction. In the open-weight Qwen2.5-VL-32B, linear probes on the vision encoder recovered the line angle and the target dot six to eight times more accurately than its behavioral responses. A layer-wise logit lens showed that the correct answer dominates in the middle layers of the language model and is overwritten in late layers only under a visual-judgment frame. Naming the illusion, requesting suppression of prior knowledge, forcing a visual chain of thought, and 390 semantic preambles left the response center unchanged; only disclosing the correct answer moved it. VLMs may therefore carry human perceptual biases at the language-generation stage, where they resist declarative correction.

Impact statement

Vision-language models encode a line’s true direction accurately yet report the same mis-extrapolated target as human observers, a bias imposed in late language layers and unaffected by prompting.

Introduction

Vision-language models (VLMs), large models that take images as input and respond in natural language, have proliferated rapidly and are beginning to be used routinely for visual-judgment tasks, including medical imaging, technical-drawing interpretation, and measurement support. Yet little is known about how these models “see” a stimulus, and in particular whether they share the biases specific to human perception.

Visual illusions are a classical tool for exposing the dissociation between perception and physical reality, and susceptibility to them has become one criterion for gauging how human-like a system’s vision is. For instance, a growing body of research evaluates artificial neural networks as models of human vision by testing whether they reproduce illusory percepts (e.g., Watanabe et al., [11]; Lotter et al., [4]; Ueda et al., [7]). Studies presenting illusion stimuli to VLMs have also emerged in recent years; benchmarks spanning multiple classical illusions have reported that VLM responses partially agree with human perception, but that it is difficult to determine whether this agreement reflects perceptual understanding or a surface pattern derived from the training data ([6, 12]). In particular, Ullman [8] showed that VLMs erroneously judge stimuli that resemble illusions but produce no illusory effect to be “illusions,” raising the possibility that VLM responses to illusions reflect the recollection of illusion commentary contained in the training data. Two gaps remain in the existing literature. First, because classical illusions are accompanied by countless explanations and human response data on the internet, it is difficult to determine whether a VLM response is “the result of looking at the image” or “the result of recalling a description of the illusion.” Second, few studies have localized the origin of illusion responses through direct analysis of the model’s internal representations.

This paper addresses both gaps by examining a single stimulus in depth. The stimulus is the Watanabe Illusion (Figure 1a), which one of the authors has made public on Figshare [10]. As a related phenomenon, Bouma and Andriessen [3] reported that the orientation of line segments close to horizontal or vertical is perceived as closer to horizontal or vertical, and showed a similar bias in judgments of collinearity with a distant dot. That study did not, however, treat the effect as an illusion, and it has since been cited mainly in the context of tilt normalization and the oblique effect. In a short phenomenological report that re-evaluates this phenomenon as an illusion, Usui and Kitaoka [9] depicted the present stimulus as a structural variant. No quantitative study has addressed the stimulus itself, and the present paper is its first detailed report. These works are rarely cited, and Usui and Kitaoka [9] in particular was published after the release of Qwen2.5-VL-32B (March 2025), the model used here for internal analysis. Contamination through the training data is therefore expected to be limited. None of the four VLMs we tested (Qwen2.5-VL-32B, Claude Opus 4.7, GPT-5.5, Gemini 3.5 Flash) could name this stimulus as a known illusion, whereas the three commercial frontier models named the classical illusions (Müller-Lyer, Poggendorff, Kanizsa) accurately (Appendix 1).

We therefore used the Watanabe Illusion as the test stimulus and carried out (i) a survey of the illusion in human observers (n=131n = 131), (ii) a behavioral survey of five VLMs under an identical protocol, (iii) localization of the origin of the bias through internal analysis of the open-weight model Qwen2.5-VL-32B, and (iv) tests of whether declarative interventions and 390 semantic interventions can shift the bias.

Results

Human observers show a mental-extrapolation bias in the Watanabe Illusion

The stimulus (104.jpg, Figure 1a) contains, inside a circle at the lower-left corner, a line segment inclined 23.5∘23.5^\circ from the horizontal; 11 equally spaced dots run down the right edge. The geometric extension of the segment intersects the topmost dot (dot 1), but no extension line or guide is drawn, so the observer must extrapolate the intersection from the apparent trajectory of the segment alone. In an online experiment, participants answered two independent questions asking which dot the line would hit, counted from the top (Q1Q_1) and from the bottom (Q2Q_2). Illusion magnitude was summarized by the Illusion Index (IIII), which is 0 for a veridical response and 10 for the maximal illusion (Materials and methods).

Summaries of human Q1Q_1, Q2Q_2, and IIII are shown in Figure 1c–e (Human), with detailed statistics in Appendix § 3.5—table 1. Although the line physically passes through dot 1, human observers (n = 131) on average reported the 4.65th dot from the top (Q1Q_1, SD = 2.63) and the 6.36th dot from the bottom (Q2Q_2, SD = 2.86). The mean IIII was 4.15 (SD = 2.28). Only 4.6% of participants (6 of 131) gave a veridical response (Q1Q_1 = 1). We call this systematic downward deviation the mental-extrapolation bias. It agrees with the phenomenological observation of Usui and Kitaoka [9] for this stimulus (observers tend to point to the third or fourth dot from the top), and it is in the same direction as the report of Bouma and Andriessen [3] that near-horizontal line segments are perceived as closer to horizontal. At the group level the answers were internally consistent: an observer identifying a single dot should produce Q1+Q2Q_1 + Q_2 = 12, and the human mean was 11.01 (SD = 3.07). They were also asymmetric, with II Q2II\ Q_2 (4.64) exceeding II Q1II\ Q_1 (3.65) by about one dot. The recognition probe in Appendix 1 confirms that this stimulus is rare within the training distribution of contemporary VLMs (0 correct namings in 80 naming trials across four VLMs, against up to 100% for the classical-illusion controls).

VLMs reproduce the same bias as humans

Figure 1b shows the phenomenon in a single image. We passed 104.jpg to the image-generation function of ChatGPT (GPT-5.5) and asked only that it add a star to the dot the line segment hits. In the output image, although the line segment was nearly preserved (23.3°), the star was placed near dot 6, within the range of the behavioral bias of humans and VLMs. Anyone can replicate this demonstration in a few minutes. We then quantified the phenomenon under a controlled protocol in which five VLMs received the same wording and the same independent-trial structure as the human participants (Materials and methods).

Figure 1c–e shows the mean ± SD of Q1Q_1, Q2Q_2, and IIII for all six data sources (humans and VLMs). The group means of IIII were 4.91 for Qwen-32B, 4.95 for Qwen-7B, 5.17 for Sonnet 4.6, 6.04 for Gemini 3.5 Flash, and 7.78 for GPT-5.5 Instant. All five deviated in the same direction as humans (4.15) and lay far from veridical (IIII = 0): each model reported a dot in the middle-to-lower range rather than dot 1.

Veridical responses (Q1=1Q_1 = 1) occurred in only 2 of the 299 trials pooled across the five VLMs (both from Qwen-7B). The directional asymmetry observed in humans (∥Q2∥>∥Q1∥\lVert Q_2 \rVert> \lVert Q_1 \rVert) was also reproduced at both Qwen sizes (32B: +1.38; 7B: +2.36; humans: +0.99).

Qwen-32B compressed the spread of its responses to about one quarter of human inter-individual variability (∥I∥\lVert I \rVert SD = 0.63, 0.28× the human value), whereas the 7B model of the same series and architecture retained variance close to that of humans (Q1Q_1 SD = 2.51, 0.95× the human value of 2.63; ∥I∥\lVert I \rVert SD = 1.75). The variance of the three commercial models lay between the two (∥I∥\lVert I \rVert SD = 0.75–1.29). How much a model compresses response variance therefore differs greatly between models, even within one model series. All statistics are collected in Appendix § 3.5—table 2.

The vision encoder retains accurate information about the line’s inclination

To test whether visual encoding preserves the angle of the segment, we directly probed the vision encoder of the same model whose behavior we had measured (Qwen2.5-VL-32B), using linear probes trained on a parametric stimulus set of 68 images (Materials and methods).

The line angle was recovered with R2=0.985R^2 = 0.985 and MAE = 2.23° from the final ViT block (Representation A; Figure 2a), and performance was comparable at the Patch Merger output that the language model actually receives (R2=0.981R^2 = 0.981, MAE = 2.43°). The dot number indicated by the line segment was recovered with sub-dot accuracy, MAE = 0.61 dots (Representation C; Figure 2b). Stimulus condition (presence of the circle, length of the segment) had little effect on probe accuracy.

Linear probes on the Qwen2.5-VL-32B vision encoder

Figure 2. Linear probes on the Qwen2.5-VL-32B vision encoder recover the stimulus geometry accurately. Parametric stimulus set (n=68n = 68 images), ridge regression with leave-one-out cross-validation. (a) Angle probe (Representation A = final ViT block): recovers the true line angle with R2=0.985R^2 = 0.985, MAE = 2.23∘2.23^\circ. (b) Dot-number probe (Representation C = Patch Merger flattened): recovers the dot number indicated by the line with R2=0.974R^2 = 0.974, MAE = 0.61 dots. This is roughly six to eight times more accurate than the behavioral error of the same model (4.22–5.60 dots; Appendix § 3.5—table 3), showing that the bias is not present at the visual-encoding stage.

We compared this internal accuracy with the behavior of the same model (Appendix § 3.5—table 3). The behavioral error was 4.22 dots for Q1Q_1 and 5.60 dots for Q2Q_2, roughly six to eight times the linear read-out error from the encoder (0.71 dots). The encoder therefore passes accurate information to the language model, and the bias is most likely introduced downstream of that interface.

The bias arises in the late layers of the language model

To localize where within the language model the bias arises, we applied a layer-wise logit lens ( (Belrose et al., 2023; nostalgebraist, 2020) ) to the 64 layers of Qwen2.5-VL-32B, prefilling the assistant response with one of four fixed continuations and reading out the probabilities of the digit tokens “1”–“9” at each layer (n=10n = 10 trials per continuation; Materials and methods):

  • v1 “The line would hit the” (visual-judgment baseline)

  • v2 “The topmost dot is the” (concept-recall control)

  • v3 “The number five is the” (irrelevant-prior control)

  • v4 “The line would hit dot number” (visual judgment + numerical cue)

Figure 3 shows the layer ×\times digit-probability curves for the four conditions. In v1, v2, and v4 alike, the correct digit “1” rises steeply in the middle layers (layers 35–45) and briefly becomes the argmax (peak P≈0.40P \approx0.40–0.500.50). The model thus appears to form the correct candidate partway through processing, and what happens to it afterwards depends on the continuation. Under the concept-recall frame (v2), “1” persists to the final layer (P=0.94P = 0.94). Under the visual-judgment frames (v1, v4), “1” is systematically overwritten in the late layers (~45–63), and the final layer commits to a distribution centered on “5.” In v3, “5” is carried from the middle layers onward and persists. This suggests that a context-dependent override mechanism in the late layers of the language model, triggered when the input is framed as a visual judgment, rewrites the correct middle-layer candidate into a response consistent with human reports. Final-layer values are given in Appendix § 3.5—table 4.

Layer-wise logit lens across the 64 layers of the Qwen2.5-VL-32B language model

Figure 3. Layer-wise logit lens across the 64 layers of the Qwen2.5-VL-32B language model. Each panel shows, for one continuation (v1–v4), the within-layer softmax probabilities of digit tokens 1–9 at each layer (mean of n=10n = 10 trials per continuation). The correct answer “1” is the bold red line and the canonical bias “5” is the bold blue line. In v1, v2, and v4, “1” rises to argmax dominance in the middle layers (~35–45). Under the concept-recall frame (v2), “1” persists to the final layer (P=0.94P = 0.94), whereas under the visual-judgment frames (v1, v4) it is overwritten in the late layers (~45–63) and the final layer commits to a distribution centered on “5.” In v3, “5” is carried from the middle layers onward (control). Final-layer values are in Appendix § 3.5—table 4.

The bias resists declarative and semantic interventions

We tested whether the late-layer override could be disabled by declarative linguistic interventions, using an eight-condition prompt-cancellation experiment (n=20n = 20 each, Q1; Figure 4). Naming the illusion (P1: “Watanabe Illusion”; P2: “a visual illusion”), requesting the suppression of prior knowledge (P3: “Ignore any prior knowledge… answer based only on the line’s pixel-level geometry”), and forcing a visual chain of thought (P5) all failed to move the response outside the baseline band (P0 mean 5.45 ±\pm 1 SD). An irrelevant context (P7) behaved likewise. The only manipulations that moved the response were direct content interventions: a counter-hint (P4) produced an incomplete shift of −1.25-1.25 dots, and only disclosure of the correct answer (P6) collapsed the response to near-veridical (mean 1.35) (condition-wise statistics in Appendix § 3.5—table 5). The full text of every prompt is recorded in Appendix 3.

Mean Q1 responses across eight prompt-cancellation conditions

Figure 4. Declarative prompt interventions do not move the biased response. Mean Q1Q_1 responses of Qwen2.5-VL-32B across the eight prompt-cancellation conditions (n = 20 trials each). Bars are mean ± SD, black points are individual trials, the dashed line is the P0 control mean (5.45), the shaded band is P0 ± 1 SD, and the red line marks veridical (Q1Q_1 = 1). Naming the illusion (P1, P2), requesting the suppression of prior knowledge (P3), forcing a visual chain of thought (P5), and an irrelevant context (P7) all fail to move the response outside the P0 ± 1 SD band. The counter-hint (P4) produces an incomplete shift (−1.25 dots), and only disclosure of the correct answer (P6) collapses the response to near-veridical. Statistics are descriptive; condition-wise values are in Appendix § 3.5—table 5, and the prompts in Appendix 3.

Although the model is fully capable of generating the words “ignore prior knowledge,” that request does not reach the process that produces the bias. The bias does not reside at the level of declarative knowledge that the language model can introspect on, name, and rewrite. Semantic interventions drawn from a much broader space did not shift it either. In nine conditions (n≈130n \approx130 each, one-shot design) in which 390 preambles describing the observer’s identity (a sniper, a poet, an octopus), living environment (dense fog, zero gravity), or a neutral situation were placed in the system or user message, the response mean remained within the narrow range 5.52–6.40, and no preamble moved the response center toward veridical (Appendix § 3.5—table 6). Placing the preamble in the system message increased response variance (maximum SD=1.75SD = 1.75, 67% of the human inter-individual SD of 2.63), but the center did not move, and no condition reached the human inter-individual variance.

Discussion

What the results suggest

All five VLMs showed a bias in the same direction as humans, of comparable or greater magnitude. Because the illusion is rare in the VLMs’ training distribution, these responses are unlikely to be a memory of a particular illusion (Appendix 1). In Qwen2.5-VL-32B, the vision encoder retained accurate angle information, and the bias arose through a context-dependent override in the late layers of the language model. Neither declarative interventions nor 390 semantic interventions moved the response center.

Although the vision encoder retains accurate information, a VLM trained on human-derived data (human descriptions of and annotations on images) may hold human perceptual biases at the language-generation stage, where they are deeply embedded and hard to remove. The training labels of a VLM are thought to reflect not pixel-level geometry but what human observers “report having seen.” The fact that all VLMs showed a bias in the same direction despite the Watanabe Illusion being a rare stimulus suggests that this “knowledge” is not a memory of a particular illusion but a distillation of the statistical structure of human perceptual responses. The tilt normalization shown by Bouma and Andriessen [3] is a general property of human perception that is not restricted to particular figures. Even when an individual stimulus is unknown, if such general properties are reflected in the training data through human descriptions, a VLM can show a bias in the same direction as humans for that unknown stimulus.

Application risks and contributions to basic science

VLMs are being introduced into practical image analysis in medicine, manufacturing, and measurement. Our results show that even when accurate information exists inside the model, the linguistic response can be systematically distorted by a human-like bias, and that this bias cannot be removed by prompt-level warnings or requests for suppression. When VLM visual judgments are used in practice, this class of bias needs to be investigated comprehensively, beyond illusion stimuli.

If VLMs distill human report statistics, they may also be useful as models of human vision. The dissociation observed here (accurate information exists internally, yet the report is biased; and the bias is not under the control of declarative knowledge) parallels the human experience of “seeing an illusion even while knowing it is an illusion.” Because the layer where this transformation occurs can be located in a model, the same question can be put to humans: do early measurements that precede verbal report carry veridical information?

Limitations

(i) The behavioral comparison rests on a single stimulus (104.jpg, 23.5°, and the angle dependence of the bias has not been tested. Future experiments should use stimuli in which the physically correct dot lies in the middle of the column. (ii) The internal analysis is limited to the open-weight Qwen2.5-VL-32B, and whether the same mechanism operates inside commercial VLMs can only be inferred from the agreement in behavioral patterns. (iii) The values for commercial VLMs are a snapshot of default web-UI behavior as of May 2026. (iv) The success of a linear probe establishes the presence of information, but not its absence. (v) The variance value for 7B is based on a single n=60n = 60 run, and confirmation of run-to-run stability would be desirable. (vi) The human n=131n = 131 reflects between-subject variability, whereas the VLM variance derives from stochastic decoding of a single model, so this asymmetry remains in any comparison of the two variances. (vii) Because Usui and Kitaoka [9] depicts this stimulus, we cannot rule out that it is included in the training data of commercial models trained after that paper’s publication. In the recognition probe of Appendix 1, however, none of the models could name the stimulus.

Materials and methods

Key resources table

Reagent type (species) or resourceDesignationSource or referenceIdentifiersAdditional information
Software, algorithmQwen2.5-VL-32B-InstructBai et al. [1]Hugging Face: Qwen/Qwen2.5-VL-32B-InstructOpen weights, BF16
Software, algorithmQwen2.5-VL-7B-InstructBai et al. [1]Hugging Face: Qwen/Qwen2.5-VL-7B-InstructOpen weights, BF16
Software, algorithmClaude Sonnet 4.6; Claude Opus 4.7Anthropic—Web UI (behavior); API (recognition probe)
Software, algorithmGPT-5.5 Instant; GPT-5.5OpenAI—Web UI (behavior); API (recognition probe)
Software, algorithmGemini 3.5 FlashGoogle—Web UI (behavior); API (recognition probe)
Software, algorithmGorilla Experiment BuilderGorillahttps://app.gorilla.sc/openmaterials/1339686 (demonstratioOnline human experiment
Reagent type (species) or resourceDesignationSource or referenceIdentifiersAdditional information
n)
Software, algorithmAnalysis and experiment codeThis paperGitHub: https://github.com/eijwat/watanabe-illusion-vlm; Zenodo: doi:10.5281/zenodo.23096369See Appendix 4
OtherWatanabe Illusion stimulus (104.jpg)Watanabe [10]doi:10.6084/m9.figshare.5838606539 × 264 px, 23.5°

Table 7.

Stimulus

The stimulus (104.jpg, Figure 1a) is a rectangle (539 × 264 px). Inside a circle at the lower-left corner, a line segment inclined 23.5∘23.5^\circ from the horizontal is drawn, and 11 equally spaced dots are arranged from top to bottom along the right edge. The segment is designed so that its geometric extension intersects the topmost dot (dot 1). No extension line or guide is drawn; the observer must extrapolate the intersection solely from the apparent trajectory of the segment.

Human experiment

In an online psychophysical experiment (Gorilla platform), participants were presented with 104.jpg and asked to answer the following two questions independently.

  • Q1Q_1 (from top): Estimate which dot on the right edge the line inside the circle would hit if extended. Answer like ‘the Xth dot from the top.’

  • Q2Q_2 (from bottom): Estimate which dot on the right edge the line inside the circle would hit if extended. Answer like ‘the Xth dot from the bottom.’

After the exclusions described under Participants, restricting the analysis to participants who answered both Q1Q_1 and Q2Q_2 with values within [1, 11] yielded data from 131 participants. The experiment as run in Gorilla can be tried in an online demonstration (https://app.gorilla.sc/openmaterials/1339686).

Participants. Participants were recruited through Prolific (https://www.prolific.com/). The main experiment was run on 23 April 2026 with 198 participants before exclusion; a pilot test with 94 participants was run on 16 April 2026, and its data were not used. Participants were paid at an hourly rate of £12.00. Before analysis, participants were excluded if they failed the attention check, if their answer to an additional question on the angle of the line (not analyzed in this study) was invalid, or if they had participated more than once. Demographic data from the Prolific profiles were available for the 195 participants whose submissions were approved on Prolific: self-reported sex was 50.3% male (n = 98), 49.2% female (n = 96), and 0.5% not reported (n = 1), and age was 33.38 ± 10.25 years (mean ± SD; male 33.37 ± 10.21, female 33.50 ± 10.34).

Ethics. The experimental procedure received approval from the Life Science Research Ethics Committee of the National Institutes of Natural Sciences (Project ID: EC01-078), and informed consent was obtained from all participants. All methods were carried out in accordance with relevant guidelines and regulations.

Illusion Index

Illusion magnitude is expressed by the Illusion Index (IIII):

II=(Q1−1)+(11−Q2)2II = \frac{(Q_1 - 1) + (11 - Q_2)}{2}

A veridical response (Q1=1Q_1 = 1, Q2=11Q_2 = 11) gives II=0II = 0, and the maximal illusion (Q1=11Q_1 = 11, Q2=1Q_2 = 1) gives II=10II = 10. The direction-specific sub-indices are IIQ1=Q1−1II_{Q_1}=Q_1-1 and IIQ2=11−Q2II_{Q_2}=11-Q_2.

VLM behavioral experiment

We presented the VLMs with the same wording and the same independent-trial structure as the human experiment (Q1Q_1 and Q2Q_2 asked in separate turns with no conversation history). We tested five VLMs:

  • Open weights (n = 60 each): Qwen2.5-VL-32B and 7B [1] (BF16, temperature = 1.0, fixed per-trial seeds). Only the model ID differed between the two runs (same script and prompt), enabling a comparison

across model sizes. For both sizes, we confirmed via image reconstruction from the processor output that the stimulus was received at native dynamic resolution without distortion.

  • Commercial frontier models (web UI, default mode, n=60n = 60 each): Claude Sonnet 4.6, GPT-5.5 Instant, and Gemini 3.5 Flash. Each trial was conducted in a new temporary chat with memory storage turned off. We used the web UI because it reflects “the VLM that people actually experience.” Data were collected in May 2026.

The image-generation demonstration (Figure 1b) passed 104.jpg to the image-generation function of ChatGPT (GPT-5.5) with a request only to add a star to the dot the line segment hits.

Statistics

All six data sources were processed with the same response parser, the same directional normalization (responses given in the opposite counting direction were mapped by 11−value+111 - \text{value} + 1), and the sample standard deviation (ddof = 1). Because a response could not be extracted on some trials, the Illusion Index and the sum of the two answers were computed from the trials in which both answers were extracted, pairing them by trial (paired n=58n = 58 for Qwen-7B and 5959 for Gemini). Results are reported descriptively as mean ±\pm SD (with medians and ranges in Appendix § 3.5); no inferential tests were applied.

Vision encoder probes

We generated a parametric stimulus set (1717 angles ×4\times4 conditions =68= 68 images, generate_stimuli.py) and trained linear probes with ridge regression under leave-one-out cross-validation on the final ViT block output (Representation A) and the Patch Merger output (Representations B and C) of Qwen2.5-VL-32B. Targets were the line angle and the dot number indicated by the line. Probe accuracy is reported as R2R^2 and mean absolute error (MAE).

Logit lens

For each of the four continuations (v1–v4, Results), the assistant response was prefilled with the continuation and, at the final token position, the within-layer softmax probabilities of the digit tokens “1” through “9” were read out across all 64 layers of the language model of Qwen2.5-VL-32B (Belrose et al., 2023; nostalgebraist, 2020); n=10n = 10 trials per continuation.

Prompt cancellation

Eight conditions (P0–P7) ×\times n=20=160n = 20 = 160 single-turn trials on Qwen2.5-VL-32B-Instruct, Q1Q_1 only, no conversation history, seed = (prompt index) ×\times 1000 + trial index. Each prompt was the fixed preamble followed by the condition-specific body (Appendix 3).

Semantic preamble intervention

A set of 390 preambles (130 in each of three categories: Character, Environment, and Neutral) was placed in either the system or the user message in nine conditions (A–I) on Qwen2.5-VL-32B-Instruct, Q1Q_1 only. The design is one-shot: each condition comprises 130 trials, each with a different preamble and a different seed, and no preamble is repeated. Condition definitions, seed scheme, and message structure are given in Appendix 3.

Recognition probe

Four stimuli (104.jpg and three classical illusions) were presented to Qwen2.5-VL-32B (A100) and, via official APIs without conversation history, to Claude Opus 4.7, GPT-5.5, and Gemini 3.5 Flash, with three question types in independent trials (n=10n = 10 per cell, 480 trials in total). Details are given in Appendix 1.

Generation settings and computing environment

Open-weight runs used temperature = 1.0, top_p = 1.0, repetition_penalty = 1.05, max_new_tokens = 512, and a fixed per-trial seed, on an NVIDIA A100 80GB (x86_64) and an NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory, ARM64, CUDA 13.0). Full details are given in Appendix 4.

Use of generative AI

The experimental design, analysis code, and documentation were developed in collaboration with Claude (Anthropic). ChatGPT (OpenAI) also played a substantial role in generating ideas. Both Claude and ChatGPT additionally participated in this study as objects of experimentation. The human authors take full responsibility for the scientific content, the integrity of the data, the interpretations drawn from the results, and any remaining errors.

Additional information

Funding

FunderGrant reference numberAuthor
Japan Society for the Promotion of Science (JSPS)Grants-in-Aid for Scientific Research (A, B, and C)Eiji Watanabe
Japan Society for the Promotion of Science (JSPS)KAKENHI JP21H04426Akiyoshi Kitaoka
Japan Science and Technology Agency (JST)SPRING JPMJSP2104Kyohei Ueda

Table 8.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Competing interests

The authors declare that no competing interests exist.

Author contributions

Eiji Watanabe: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization, Project administration, Funding acquisition

Kyohei Ueda: Methodology, Validation, Formal analysis, Investigation, Data curation, Funding acquisition

Kagayaki Kato: Software, Validation, Resources

Masaki Watabe: Software, Validation, Writing – original draft, Writing – review & editing, Visualization

Akiyoshi Kitaoka: Conceptualization, Validation, Resources, Supervision, Funding acquisition

Ethics

Human subjects: the experimental procedure received approval from the Life Science Research Ethics Committee of the National Institutes of Natural Sciences (Project ID: EC01-078), and informed consent was obtained from all participants.

Data availability

All code, stimuli, cleaned human response data, and raw model outputs are available at GitHub (https://github.com/eijwat/watanabe-illusion-vlm) and archived at Zenodo (https://doi.org/10.5281/zenodo.23096369); Appendix 4 maps every result in this paper to the files that reproduce it. The raw Gorilla exports of the human experiment are not released because they contain participant-identifying information; the cleaned responses used for all analyses (data/human/human_2nd_test.csv) are released. The main stimulus is available on Figshare (doi:10.6084/m9.figshare.5838606).

The following dataset was generated:

Watanabe E, Ueda K, Kato K, Watabe M, Kitaoka A. 2026. watanabe-illusion-vlm: code, stimuli and data for “Vision-Language Models Reproduce a Human Mental-Extrapolation Bias in the Watanabe Illusion”. Zenodo. doi:10.5281/zenodo.23096369

Figures

The Watanabe Illusion and responses of human and VLM observers

Figure 1. The Watanabe Illusion and the responses of human and VLM observers. (a) Main stimulus 104.jpg: the line segment, partially visible inside the circle, geometrically passes through the topmost dot (dot 1) of the column of 11 dots along the right edge. (b) Demonstration with a generative probe: the output when ChatGPT was asked only to add a star to the dot the line segment hits. Although the line segment is preserved at 23.3°, the star was placed near dot 6. (c–e) Q1Q_1 (c), Q2Q_2 (d), and the Illusion Index IIII (e) for six data sources: humans (n = 131 participants), Qwen2.5-VL-32B (n = 60 trials), Qwen2.5-VL-7B (n = 60; paired n = 58), Claude Sonnet 4.6 (n = 60), GPT-5.5 Instant (n = 60), and Gemini 3.5 Flash (n = 60; paired n = 59). Points are means, bars are ± SD (sample SD, ddof = 1), and the red line marks veridical (Q1Q_1 = 1, Q2Q_2 = 11, IIII = 0). All VLMs deviate from veridical in the same direction as humans. Statistics are descriptive; full values are in Appendix § 3.5—table 2.

Appendix

Recognition probe: the Watanabe Illusion is an unknown stimulus for contemporary VLMs

This appendix tests whether VLM responses are generated “by looking at the image content” or “retrieved from descriptions of the Watanabe Illusion in the training data.” To address it, we measured whether four VLMs could identify 104.jpg as the Watanabe Illusion, using three classical illusions as positive controls.

Protocol. Four stimuli (104.jpg, Müller-Lyer, Poggendorff, Kanizsa; the three classical stimuli are shown in Appendix 1—figure 1) were presented to each VLM, and three types of question were posed in independent trials: (QV1) yes/no familiarity, (QV2) forced naming of a known illusion, and (QV3) description + recognition. Search was prohibited (“Do not search the web.”). The models tested were Qwen2.5-VL-32B (A100), Claude Opus 4.7, GPT-5.5, and Gemini 3.5 Flash (the three commercial models via API, because the authors’ web-UI accounts are contaminated by conversation history about this illusion). n=10n = 10 per cell, 480 trials in total.

Positive-control stimuli for the recognition probe: (a) Müller-Lyer illusion, (b) Poggendorff illusion, and (c) Kanizsa illusion

Figure 1 (PDF p. 22). Positive-control stimuli for the recognition probe. (a) The Müller-Lyer illusion. (b) The Poggendorff illusion. (c) The Kanizsa illusion (illusory contours). All were presented under the same protocol as 104.jpg. Images are from Wikimedia Commons: (a) “Müller-Lyer Illusion - MathWorld version.svg” (public domain); (b) “Poggendorff illusion.svg” and (c) “Kanizsa triangle.svg” (both by Fibonacci, CC BY-SA 3.0).

Main result (Appendix 1—table 1).

Appendix 1—table 1. Correct-naming rates for the naming questions (QV2 + QV3, n=20n = 20 per cell). Claude Opus 4.7 and GPT-5.5 named all three classical illusions correctly 100% of the time. By contrast, the Watanabe Illusion was never named correctly in any of the 80 naming trials pooled across the four VLMs. Their performance on the classical illusions rules out the alternative that VLMs cannot name illusions in general, so this result is strong evidence that 104.jpg is unknown within the training distribution of today’s major VLMs.

ModelWatanabeMüller-LyerPoggendorffKanizsa
Qwen2.5-VL-32B0/20 (0%)2/20 (10%)2/20 (10%)0/20 (0%)
Claude Opus 4.70/20 (0%)20/20 (100%)20/20 (100%)20/20 (100%)
GPT-5.50/20 (0%)20/20 (100%)20/20 (100%)20/20 (100%)
Gemini 3.5 Flash0/20 (0%)13/20 (65%)20/20 (100%)20/20 (100%)

Table 9.

Misnamings reflect structural proximity. The Watanabe stimulus was misnamed as another illusion in 13 trials. The illusion names given in those trials (counted once per name when a trial gave several) were Poggendorff 8, Müller-Lyer 5, and Kanizsa 1. Poggendorff was the most frequent misnaming, and both illusions share the task structure of “mentally estimating the extension of a line segment whose visible extent is limited.” VLMs extract the visual structure of the stimulus to some degree, but have not learned the particular name tied to that structure.

Statistical tables {#sec:Appendix 2}

Appendix 2—table 1. Human reference data (n = 131).

MeasureVeridicalMean (SD)Median
Q1Q_1 (from top)14.65 (2.63)4
Q2Q_2 (from bottom)116.36 (2.86)7
IIQ1=Q1−1I I Q_1 = Q_1 - 103.65 (2.63)3

Table 10.

MeasureVeridicalMean (SD)Median
II Q2=11−Q2II\,Q_{2}=11-Q_{2}04.64 (2.86)4
IIII04.15 (2.28)4.00
Sum Q1+Q2Q_{1}+Q_{2}1211.01 (3.07)12

Table 11.

MeasureHuman (n = 131)Qwen-32 B (n = 60)Qwen-7B (n = 59; paired 58)Sonnet 4.6 (n = 60)GPT-5.5 Instant (n = 60)Gemini 3.5 Flash (n = 59/60)
Q1Q_{1} — mean (SD)4.65 (2.63)5.22 (0.90)4.78 (2.51)4.30 (1.53)8.03 (1.34)7.10 (1.86)
Q2Q_{2} — mean (SD)6.36 (2.86)5.40 (0.83)4.86 (2.32)3.97 (1.73)2.48 (0.54)5.00 (1.52)
II Q1II\,Q_{1} — mean (SD)3.65 (2.63)4.22 (0.90)3.78 (2.51)3.30 (1.53)7.03 (1.34)6.10 (1.86)
II Q2II\,Q_{2} — mean (SD)4.64 (2.86)5.60 (0.83)6.14 (2.32)7.03 (1.73)8.52 (0.54)6.00 (1.52)
IIII — mean (SD)4.15 (2.28)4.91 (0.63)4.95 (1.75)5.17 (1.13)7.78 (0.75)6.04 (1.29)
IIII — range[0.00, 9.50][3.50, 6.50][1.50, 8.50][2.00, 7.00][5.00, 9.00][2.50, 7.50]

Table 2. Appendix 2—table 2. Distributional statistics for the six data sources. All sources were processed with the same parser, the same directional normalization, and the sample standard deviation (ddof = 1). Because of extraction failure on some trials, the Illusion Index and the sum of the two answers use the trials in which both answers were extracted, paired by trial (paired n = 58 for Qwen-7B and 59 for Gemini).

MeasureHuman (n = 131)Qwen-32B (n = 60)Qwen-7B (n = 59; paired 58)Sonnet 4.6 (n = 60)GPT-5.5 Instant (n = 60)Gemini 3.5 Flash (n = 59/60)
Sum Q1Q_{1} + Q2Q_{2} — mean (SD)11.01 (3.07)10.62 (1.19)9.66 (3.38)8.27 (2.36)10.52 (1.38)12.08 (2.23)

Table 2 (PDF p. 25).

Appendix 2—table 3. Behavioral error vs. internal probe error (Qwen2.5-VL-32B).

MeasureVeridicalBehavioral response (mean)Behavioral errorInternal probe error (n = 68)Amplification
Q1Q_{1} (dot from top)15.224.22 dots0.71 dots5.9×
Q2Q_{2} (dot from bottom)115.405.60 dots0.71 dots7.9×

Table 3. Behavioral error vs. internal probe error (Qwen2.5-VL-32B).

Appendix 2—table 4. Logit lens: final-layer digit-token probabilities (mean of n = 10 each).

Digitv1 (“would hit the”)v2 (“topmost dot is the”)v3 (“number five is the”)v4 (“would hit dot number”)
10.0390.9440.0290.090
40.1770.0040.0200.190
50.2920.0070.8580.190
60.2010.0070.0140.168
(others: 2, 3, 7–9)≤ 0.14≤ 0.012≤ 0.03≤ 0.12

Table 4. Logit lens: final-layer digit-token probabilities (mean of n = 10 each).

PromptDescriptionMeanSDDifference from P0
P0Control5.450.89(reference)
P1Naming as “Watanabe Illusion”5.351.04−0.10
P2Naming as “visual illusion”5.350.99−0.10
P3Prior-knowledge suppressio n5.201.15−0.25
P4Counter-hint4.201.01−1.25
P5Forced visual chain of thought5.101.65−0.35
P6Disclosure of the correct answer1.350.67−4.10
P7Irrelevant context5.550.83+0.10

Table 5. Prompt cancellation (n=20n = 20 each, Q1Q_1).

Table 6. Semantic preamble intervention (PIE, one-shot, n≈130n \mathrel{\approx} 130 each, Q1Q_1).

ConditionCategoryPlacementnvalidn_{\mathit{valid}}MeanSDSD / human SD
Human——1314.652.631.00
A(baseline)—1295.520.740.28
CCharacteruser1265.671.110.42
EEnvironmentuser1305.650.990.38
GNeutraluser1305.520.940.36
BCharactersystem1256.281.750.67
DEnvironmentsystem1276.201.570.60
FNeutralsystem1286.401.550.59
HChar + Envsystem1226.361.660.63
IChar + Envuser1275.831.220.46

Table 12.

Full text of the prompts used (Figure 4, Appendix 2—tables 5 and 6)

This appendix records, in reproducible form, the text actually given to the model in the prompt-cancellation experiment (Figure 4, Appendix 2—table 5) and the semantic preamble intervention experiment (PIE, Appendix 2—table 6). Both are single-turn trials on Qwen2.5-VL-32B-Instruct (A100) with no conversation history.

The fixed part shared by all conditions

Both experiments share the following fixed preamble and fixed Q1 body (the same strings as Q1 in the n=60n = 60 behavioral experiment).

PREAMBLE_FIXED

Look at the image below straight on from the front. Please answer based on your visual impression alone — do not use any tools such as Python, image processing, a ruler, or a protractor to measure.

Q1_BODY

Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

ItemValue
modelQwen/Qwen2.5-VL-32B-Instruct
do_sampleTrue
temperature1.0
top_p1.0
repetition_penalty1.05
max_new_tokens512
stimulus104.jpg (identical in every trial)
conversation historynone (each trial independent)

Table 1 (PDF p. 28). Appendix 3—table 1. Generation settings (common to both experiments).

Prompt cancellation (Figure 4, Appendix § 3.5—table 5)

Script run_qwen25vl_cancellation_a100_q1.py. Eight conditions × n=20=160n = 20 = 160 trials, Q1 only. seed = (prompt index) × 1000 + trial_idx (P0 = 0–19, P1 = 1000–1019, and so on to P7 = 7000–7019).

The prompt is assembled as follows; the message consists of a single user turn (image + text).

final_prompt = PREAMBLE_FIXED

  • "\n\n"

  • <condition-specific body>

The <condition-specific body> is, except for P0, “the intervention sentence + a single space + Q1_BODY.”

P0: control

Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P1: name_specific (naming the particular illusion)

The image below shows the Watanabe Illusion. Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P2: name_generic (naming the class of illusion)

The image below contains a visual illusion. Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P3: knowledge_suppress (suppression of prior knowledge)

Ignore any prior knowledge about visual illusions or typical geometric figures. Answer based only on the line’s pixel-level geometry. Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P4: counter_hint (a counter-hint toward the top)

For figures like this one, the line typically hits a dot near the top of the right edge. Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P5: force_visual_cot (forced visual chain of thought)

First describe the line’s slope in your own words. Then Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P6: ground_truth_disclose (disclosure of the correct answer) The correct answer is the 1st dot from the top. Independently, what is your own visual estimate? Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

P7: random_distractor (irrelevant context)

While listening to instrumental music in a quiet library on a Tuesday afternoon, Estimate which dot on the right edge the line in the circle would hit if extended. Answer like ‘the Xth dot from the top.’

The capital letter in “Then Estimate” in P5 arises from string concatenation; the string is recorded here exactly as it was at run time.

Semantic preamble intervention (PIE, Appendix 2—table 6)

The scripts are run_qwen25vl_pie_oneshot_a100.py (conditions A–G) and run_qwen25vl_pie_combined_a100.py (conditions H and I). The preamble set is pie_prompts_v2.csv (390 preambles = 3×1303 \times130). The design is one-shot: one condition = 130 trials, each trial using a different preamble and a different seed. This mirrors the human data available at design time (n≈130n \approx130), in which each participant answered once; the final human analysis set (n=131n = 131) is of almost the same size. No preamble is repeated.

The question text (invariant across conditions)

Q1_USER_TEXT = PREAMBLE_FIXED + "\n\n" + Q1_BODY

That is, the question part of the user message is the two sentences above concatenated in that order, and it was never altered regardless of the presence or placement of a preamble.

Conditions and message structure

ConditionCategoryPlacementseed rangeMessage structure
A(baseline)—10000–10129user: [image, Q1_USER_TEXT]
BCharactersystem11000–11129system: preamble /
ConditionCategoryPlacementseed rangeMessage structure
user: [image, Q1_USER_TEXT]
CCharacteruser12000–12129user: [image, preamble + "\n\n" + Q1_USER_TEXT]
DEnvironmentsystem13000–13129same form as B
EEnvironmentuser14000–14129same form as C
FNeutralsystem15000–15129same form as B
GNeutraluser16000–16129same form as C
HCharacter + Environmentsystem17000–17129system: concatenated preamble
ICharacter + Environmentuser18000–18129user: concatenated preamble + Q1_USER_TEXT

Table 2 (PDF p. 30). PIE conditions and message structure.

The concatenation rule for H and I:

combined_preamble = <character preamble> + "\n\n" + <environment preamble>

Within each category, the 130 preambles are used in ascending order of (subcategory_code, numeric index). Because subcategory codes sort lexicographically, the actual order is C1-N, C1-S, C1-V, C1-X for Character; C2-P, C2S, C2-V, C2-X for Environment; and C3-D, C3-E, C3-H, C3-S for Neutral. The pairing in H and I is a deterministic one-to-one correspondence based on this order.

trialCharacterEnvironment
0C1-N-01 (Lyric poet)C2-P-01 (Zero gravity)
1C1-N-02 (Tax accountant)C2-P-02 (Underwater city)
………
129C1-X-32 (Crystalline lifeform)C2-X-32 (Inside a stained glass window)

Table 3 (PDF p. 32). Pairing of preambles in conditions H and I.

Composition of the preamble set

CategorySubcategoryID prefixnFirst item (run order)
Characterneutral_professionC1-N33Lyric poet
spatial_cognitionC1-S32Architect
vision_specializedC1-V33Savanna hunter
extreme_personaC1-X32Thousand-year-old sage
Environmentphysical_environmentC2-P32Zero gravity
social_environmentC2-S33Quiet library
visual_environmentC2-V33Dense fog
CategorySubcategoryID prefixnFirst item (run order)
environment
extreme_environmentC2-X32Four-dimensional space
Neutraldaily_activityC3-D33Sunday coffee
emotion_moodC3-E32Calm satisfaction
hobby_activityC3-H32Balcony gardening
static_situationC3-S33Waiting for a train

Table 4 (PDF p. 32). Composition of the preamble set.

Each category consists of four subcategories totaling 130 preambles (33 + 32 + 33 + 32). Preamble length is 16–38 words (mean 26.7). IDs run from 01 within each subcategory, with no duplicates and no gaps. The Neutral category deliberately excludes spatial and visual vocabulary. The first design carried 25 preambles per subcategory, 300 in total; to obtain a one-to-one correspondence with the human n≈130n \approx130 available at design time, 7–8 were added to each subcategory (ID 26 onward, 90 in total), giving 390. The 300 from the first version are unchanged in both ID and wording.

Example preambles (one from each of the 12 subcategories)

IDFull text
C1-N-01You are a lyric poet who spends most days searching for the precise word to describe a feeling. Numbers and measurements rarely enter your thoughts; what matters is rhythm and image.
IDFull text
C1-S-01You are an architect with three decades of experience designing complex buildings. You instinctively read lines, angles, and proportions in everything you see, and your mind constantly reconstructs spaces in three dimensions.
C1-V-01You are a hunter from a nomadic tribe in the African savanna. Your eyes are trained to spot prey at extreme distances, and you can see fine details that most people would miss in the shimmering heat haze.
C1-X-01You are a sage who has lived for a thousand years on a remote mountain. You have watched dynasties rise and fall, and you perceive the world with a patience that no mortal could possess.
C2-P-01You live in a world without gravity. Objects drift freely in every direction, and your sense of “up” and “down” has dissolved into a pure three-dimensional awareness of space.
C2-S-01You spend your days in the silent reading room of an ancient library. The only sounds are the rustle of paper and the distant ticking of a clock, and your attention has become slow and careful.
C2-V-01You live in a forest where dense fog never lifts. You have learned to see the world through soft, diffuse light, where edges blur and distances are estimated by sound and intuition
IDFull text
rather than by sight.
C2-X-01You live in a four-dimensional space. What three-dimensional beings call “lines” and “planes” are merely shadows of the richer structures you perceive natively.
C3-D-01You are sipping a warm cup of coffee on a quiet Sunday morning. The newspaper is folded beside you, and soft sunlight gently fills the kitchen.
C3-E-01You feel a quiet sense of satisfaction. Nothing dramatic has happened today, but several small things went well, and you find yourself unusually at peace.
C3-H-01You are tending to a small herb garden on your balcony. You have just finished watering the basil and are pinching back the mint to encourage new growth.
C3-S-01You are standing on a station platform, waiting for a train that is six minutes away. A mild breeze moves through the station, and you have nothing in particular to do.

Table 5 (PDF p. 33). Example preambles.

The full text of all 390 preambles is released in the repository as pie_prompts_v2.csv.

A completed prompt (condition I, trial 0: C1-N-01 + C2-P-01)

You are a lyric poet who spends most days searching for the precise word to describe a feeling. Numbers and measurements rarely enter your thoughts; what matters is rhythm and image.

You live in a world without gravity. Objects drift freely in every direction, and your sense of "up" and "down" has dissolved into a pure three-dimensional awareness of space.

Look at the image below straight on from the front. Please answer based on your visual impression alone — do not use any tools such as Python, image processing, a ruler, or a protractor to measure.

Estimate which dot on the right edge the line in the circle would hit if extended. Answer like 'the Xth dot from the top.'

Notes

  • Both experiments target Q1 (from top) only; Q2 was not run.

  • As a remnant of the first design, pie_prompts_v2.csv contains a full_prompt column (all 390 rows using the connective “Now,”), but the execution scripts do not read it. The scripts read only the preamble column and use the Q1_USER_TEXT above for the question part. The wording in the full_prompt column is therefore not the string used in this experiment.

Appendix 4. Reproduction map of code and data

All code, stimuli, and raw data are available at GitHub (https://github.com/eijwat/watanabe-illusion-vlm) and archived at Zenodo (https://doi.org/10.5281/zenodo.23096369). The repository is organized as follows.

watanabe-illusion-vlm/
    stimuli/        stimulus images and ground truth
    scripts/        experiment and analysis scripts
    prompts/        prompt definitions and protocols
    data/human/     cleaned responses of the human experiment (Gorilla)
    data/manual_logs/  manual logs of commercial models
    results/        raw output of each experiment
    requirements.txt

Below we list, experiment by experiment, the programs and data required to reproduce every result in this paper.

Stimuli and experimental materials

FileContents
stimuli/104.jpgMain stimulus (539×264539 \times264 px, segment 23.5∘23.5^\circ, 11 dots on the right edge, MD5 9cba67ee1a9ccb213dcab20206e4135e)
scripts/generate_stimuli.pyParametric stimulus generation (17 angles ×\times 4 conditions = 68 images)
stimuli/stimulus_metadata.csvGround truth for the 68 images (angle, intersection coordinates, dot count, circle parameters)
scripts/measure_stimulus_v2.pyVerification script measuring segment angle, dot positions, and the correct dot directly from the image
scripts/make_fig1.py – make_fig4.py, make_figA1.pyDrawing of Figures 1–4 and Appendix 1—figure 1 (correspondence below)
stimuli/control_muller_lyer.png, control_poggendorff.png, control_kanizsa.pngClassical-illusion positive controls (Appendix 1—figure 1; from Wikimedia Commons, with sources and licenses recorded in the repository README)
stimuli/G1.png, G2.png, G3.pngThe screens as actually presented in the human experiment (Gorilla)
stimuli/104_jpg_ChatGPT_Image.pngOutput image of the generative probe (Figure 1b)

Table 1 (PDF p. 37).

The 68 parametric stimuli can be regenerated from generate_stimuli.py and stimulus_metadata.csv.

Human experiment (Figure 1c–e, Appendix § 3.5—table 1)

FileContents
data/human/human_1st_test.csv, human_2nd_test.csvCleaned human responses (Q1Q_1, Q2Q_2, Q3Q_3)
FileContents
data/human/human_2nd_test_Q1Q2.csvQ1/Q2Q_1/Q_2 complete-case extract

Table 2 (PDF p. 37).

The human reference data used in the body is human_2nd_test.csv; the n=131n = 131 cases in which both Q1Q_1 and Q2Q_2 were answered within the range [1, 11] are the analysis set. The raw Gorilla exports are not released because they contain participant-identifying information.

VLM behavioral experiment (Figure 1c–e, Appendix § 3.5—tables 1 and 2)

Open-weight models:

FileContents
scripts/run_qwen25vl_n60_a100_q1q2.pyn=60n = 60 behavioral experiment for Qwen2.5-VL-32B / 7B (A100)
scripts/run_qwen25vl_n60_spark_q1q2.pyThe same, DGX Spark version
results/qwen25vl_n60_a100_q1q2/Raw output for 32B (n60_progress.json, input_verification_meta.json)
results/qwen25vl_n60_a100_q1q2_7b/Raw output for 7B (same files)

Table 13.

Commercial models (manual web-UI runs):

FileContents
prompts/commercial_vlm_protocol.mdProtocol for the commercial web-UI experiment
prompts/prompts.txtManual run procedure and full text of the presented prompts
data/manual_logs/<br>manual_log_template_n60.mdFull responses of the three commercial models (canonical data)
scripts/parse_manual_log.pyManual log to JSON conversion

Table 14.

Integrated analysis:

FileContents
scripts/analyze_commercial_vlm.pyCanonical analysis pipeline. Integrates the six sources with one parser, one directional normalization and ddof = 1, and outputs the integrated statistics (stats_extended.json, table2_extended.md) and the material for fig:1c–e. Source of the numbers in Appendix 2—tables 1 and 2
scripts/plot_vlm_vs_human.py, plot_qwen25vl_n60_v2_vs_human.pySingle-model versus human comparison figures
scripts/analyze_n60_q1q2.py, analyze_bias.py, analyze_qwen25vl_n60_v2.pyAuxiliary analyses

The outputs of the analysis (statistics JSON, tables, figures) are not included in the repository; they are produced by running the scripts above against the raw output.

The reproduction procedure is as follows. For the two Qwen sizes, run run_qwen25vl_n60_a100_q1q2.py. For the three commercial models, record the web-UI responses in a manual log following commercial_vlm_protocol.md and prompts.txt, then convert to JSON with parse_manual_log.py. Passing the five JSON files and human_2nd_test.csv to analyze_commercial_vlm.py then produces Appendix 2—tables 1 and 2 and the material for fig:1c–e. The response JSON files of the three commercial models (n60_progress_sonnet46.json, n60_progress_gpt55.json, n60_progress_gemini35flash.json) are derivatives of the manual logs and are therefore not included in the repository; they are generated from the manual logs by parse_manual_log.py.

Vision encoder probe (Figure 2, Appendix 2—table 3)

FileContents
scripts/analyze_qwen25vl_vit_a100.pyRidge linear probes on ViT / Patch Merger output (leave-one-out)
scripts/check_vit_letterbox_shape.pyVerification of the input preprocessing
FileContents
shape
scripts/analyze_vision_encoders.pyAnalysis across several encoders
scripts/decoder_model.py, train_decoder.pyDecoder-type probe (auxiliary)
results/qwen25vl_vit_probe_a100/ probe_results.csv, probe_results.jsonR2R^2 and MAE per probe ×\times condition (source of the numbers in Figure 2 and Appendix § 3.5—table 3)

Running this requires the 68 images produced by generate_stimuli.py together with stimulus_metadata.csv. The ViT / Patch Merger representation cache representations.npz (about 567 MB) is not included in the repository because of its size; it is regenerated by running the same script.

Logit lens (Figure 3, Appendix § 3.5—table 4)

FileContents
scripts/run_qwen25vl_logit_lens_a100.py (v1)continuation “The line would hit the”
scripts/run_qwen25vl_logit_lens_v2_a100.pycontinuation “The topmost dot is the”
scripts/run_qwen25vl_logit_lens_v3_a100.pycontinuation “The number five is the”
scripts/run_qwen25vl_logit_lens_v4_a100.pycontinuation “The line would hit dot number”
scripts/plot_logit_lens_4panel.pyDraws the layer ×\times digit-probability curves for the four conditions
scripts/analyze_intermediate_layers.pyIntermediate-layer analysis (auxiliary)
results/qwen25vl_logit_lens_v\{1..4\}_a100/ layer_logits.npz, layer_logits_meta.jsonLayer-wise logits and probabilities (source of the numbers in Figure 3 and Appendix § 3.5—table 4)

Table 15.

For each continuation, n=10n = 10 trials, reading the within-layer softmax probabilities of digit tokens 1–9 at the final token position across the 64 layers of the language model.

Prompt cancellation (Figure 4, Appendix 2—table 5)

FileContents
scripts/ run_qwen25vl_cancellation_a100_q1.pyThe eight conditions × n=20n = 20 run (full prompt text in Appendix 3)
scripts/plot_cancellation_v1.pyDrawing and computation of the condition-wise statistics
results/qwen25vl_cancellation_v1_a100/Full responses of the 160 trials (cancellation_progress.json). The condition-wise mean / SD (cancellation_stats.json, source of the numbers in Appendix 2—table 5) is generated from this file by plot_cancellation_v1.py

Table 16.

Semantic preamble intervention (PIE, Appendix 2—table 6)

FileContents
prompts/pie_prompts_v2.csvThe 390 preambles (Appendix 3)
prompts/pie_prompts_v1.mdPreamble design document (first version, 300 preambles)
scripts/ run_qwen25vl_pie_oneshot_a100.pyConditions A–G (n=130n = 130 each, 910 trials)
scripts/ run_qwen25vl_pie_combined_a100.pyConditions H and I (n=130n = 130 each, 260 trials)
results/pie_oneshot_20260527_005108/Raw output for conditions A–G (config.json, responses.jsonl, condition_summary.csv)
results/pie_combined_20260527_043814/Raw output for conditions H and I (same files)

Table 17.

Recognition probe (Appendix 1, Appendix 1—table 1)

FileContents
scripts/run_qwen25vl_recognition_stage2_a100.pyRecognition probe for Qwen2.5-VL-32B (44 stimuli ×3\times3 questions ×n=10\times n = 10)
scripts/run_commercial_recognition_stage2.pyAPI runs for the three commercial models (Claude Opus 4.7, GPT-5.5, Gemini 3.5 Flash)
scripts/analyze_recognition_stage2.pyAggregation of naming rates, yes/no, and the misnaming breakdown
results/qwen25vl_recognition_stage2_a100/Full responses of Qwen (progress.json, responses.jsonl, summary.csv)
results/commercial_recognition_stage2/Full responses and aggregate CSV of the three commercial models (source of the numbers in Appendix 1—table 1)

Table 1 (PDF p. 42).

Running the commercial APIs requires the environment variables ANTHROPIC_API_KEY, OPENAI_API_KEY, and GOOGLE_API_KEY (or GEMINI_API_KEY). The total number of API calls is 360.

Correspondence between the figures and their sources

FigureDrawing scriptInput
Figure 1scripts/make_fig1.pystimuli/104.jpg, stimuli/104_jpg_ChatGPT_Image.png, stats_extended.json from analyze_commercial_vlm.py
Figure 2scripts/make_fig2.pyfig_angle_probe.png and fig_spatial_probe.png from analyze_qwen25vl_vit_a100.py

Table 2 (PDF p. 42).

FigureDrawing scriptInput
Figure 3scripts/make_fig3.pyresults/qwen25vl_logit_lens_v{1..4}_a100/layer_logits.npz and layer_logits_meta.json
Figure 4scripts/make_fig4.pycancellation_stats.json output by plot_cancellation_v1.py
Appendix 1—figure 1scripts/make_figA1.pystimuli/control_muller_lyer.png, control_poggendorff.png, control_kanizsa.png

caption:

:::

None of the drawing scripts writes a figure number or title into the image; the legend carries the explanation. Colors are assigned consistently across all figures, with crimson (#dc143c) for the geometric ground truth and pale cyan (#e5ffff) for the reference band (the middle-layer band in Figure 3, the P0 ± 1 SD band in Figure 4).

Figures 1, Figure 3, and Figure 4 are drawn directly from numerical data. Figure 2 alone is composed by taking panels out of the output figures of the angle probe and the spatial probe; given a prediction-value output from analyze_qwen25vl_vit_a100.py it could likewise be replaced by direct drawing.

Execution environment

ItemContents
Open-weight run machine 1NVIDIA A100 80GB (x86_64), Docker image watanabe-illusion:a100
Open-weight run machine 2NVIDIA DGX Spark (GB10 Grace Blackwell, 128GB unified memory, ARM64, CUDA 13.0), Docker image watanabe-illusion:phase5
ModelsQwen/Qwen2.5-VL-32B-Instruct, Qwen/Qwen2.5-VL-7B-Instruct (BF16, native dynamic resolution)

Table 18.

ItemContents
Generation settings (behavioral and intervention experiments)temperature = 1.0, top_p = 1.0, repetition_penalty = 1.05, max_new_tokens = 512, fixed per-trial seed
Commercial models (behavioral experiment)Web UI default mode, memory storage off, each trial in a new temporary chat (as of May 2026)
Commercial models (recognition probe)Official API, no conversation history
Human experimentGorilla online experiment platform
StatisticsSample standard deviation (ddof = 1); directional normalization maps value to 11−value+111 - \mathrm{value} + 1
Dependenciesrequirements.txt (A100 / Spark)

Table 19.

References

  1. [1]Bai S, Chen K, Liu X, Wang J, Ge W, Song S, Dang K, Wang P, Wang S, Tang J, Zhong H, Zhu Y, Yang M, Li Z, Wan J, Wang P, Ding W, Fu Z, Xu Y, Ye J, Zhang X, Xie T, Cheng Z, Zhang H, Yang Z, Xu H, Lin J. 2025. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. DOI: https://doi.org/10.48550/arXiv.2502.13923
  2. [3]Bouma H, Andriessen JJ. 1968. Perceived orientation of isolated line segments. Vision Research 8:493–507. DOI: https://doi.org/10.1016/0042-6989(68)90091-6
  3. [4]Lotter W, Kreiman G, Cox D. 2020. A neural network trained for prediction mimics diverse features of biological neurons and perception. Nature Machine Intelligence 2:210–219. DOI: https://doi.org/10.1038/s42256-020-0170-9
  4. [6]Shahgir HS, Sayeed KS, Bhattacharjee A, Ahmad WU, Dong Y, Shahriyar R. 2024. IllusionVQA: A challenging optical illusion dataset for vision language models. arXiv preprint arXiv:2403.15952. DOI: https://doi.org/10.48550/arXiv.2403.15952
  5. [7]Ueda K, Sinapayen L, Watanabe E. 2026. Predictive networks generate motion-induced color illusions. Scientific Reports 16:27582. DOI: https://doi.org/10.1038/s41598-026-57953-w
  6. [8]Ullman T. 2024. The illusion-illusion: Vision language models see illusions where there are none. arXiv preprint arXiv:2412.18613. DOI: https://doi.org/10.48550/arXiv.2412.18613
  7. [9]Usui K, Kitaoka A. 2025. Three-dot misalignment illusion and related new illusions: A rediscovery of Bouma and Andriessen’s (1968) oblique effect as a strong illusion. Journal of Illusion 5:10908. DOI: https://doi.org/10.47691/joi.v5.10908
  8. [10]Watanabe E. 2018. Watanabe Illusion (2010). DOI: https://doi.org/10.6084/m9.figshare.5838606
  9. [11]Watanabe E, Kitaoka A, Sakamoto K, Yasugi M, Tanaka K. 2018. Illusory motion reproduced by deep neural networks trained for prediction. Frontiers in Psychology 9:345. DOI: https://doi.org/10.3389/fpsyg.2018.00345
  10. [12]Zhang Y, Pan J, Zhou Y, Pan R, Chai J. 2023. Grounding visual illusions in language: Do vision-language models perceive illusions like humans? In: Bouamor H, Pino J, Bali K (Eds). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. p. 5718–5728. DOI: https://doi.org/10.18653/v1/2023.emnlp-main.348

Paper details

vlnc:W_rxyb62ub
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Agricultural and Biological Sciences

Claude, ChatGPT

View statement
Looking for related papers…
Contents