WorksComputer Science
Learning Grounded Reasoning from Human Gaze
Abstract
Human visual reasoning is driven by systematic exploration and extraction of task-relevant visual evidence. Human gaze provides a window into this process, making gaze a useful source of guidance for visually-grounded reasoning in vision-language models (VLMs). However, human gaze also includes noise and redundancies that may transfer poorly to VLMs. In this work, we investigate how human gaze data can be effectively leveraged to improve visual reasoning in VLMs, while maintaining alignment with human behavior. We collect 7,109 human gaze trials from a diverse and challenging set of visual reasoning tasks, and train grounded reasoning models with different forms of gaze guidance. Across evaluations in chart understanding, visual search, and counting, we find that using human gaze as a prior for grounded reasoning consistently outperforms supervision using the full gaze sequence. Based on these findings, we introduce GGR (Gaze-Guided Reasoner), which achieves greater human alignment and more reliable visual grounding while matching or outperforming leading grounded-reasoning baselines in accuracy. Reinforcement learning further improves task accuracy, gaze alignment, and grounding quality despite using no gaze supervision. Behavioral analyses show that gaze supervised models exhibit more visual exploration before arriving at an answer. Our results suggest that the value of human gaze lies in guiding spatial exploration and shaping reasoning behavior without requiring complete imitation of human behavior.
Data Models Code
Figure 1. GGR achieves higher human gaze alignment and better visual grounding while maintaining strong task performance. (Left) Mean accuracy versus human fixation log-likelihood relative to uniform ( SEM); open/filled circles denote pre/post-RL models. (Middle) Human gaze, Base RL, and GGR grounding traces on a held-out example. (Right) Supported grounded reasoning claims with points on/near target for GGR-RL and No-Gaze-RL ( SEM). All results on Qwen3-VL-8B-Instruct.
Introduction
Visual reasoning is fundamentally a process of selective exploration. A reasoner must decide where to look, ground abstract concepts to locations in the visual world, and use that evidence to determine where to look next. Humans perform this visual exploration via their gaze. When comparing two series in a chart, for example, a person may look from the legend to one curve, trace it to a relevant point, inspect the corresponding axis value, and then repeat the process for the other curve. When searching for a person in a crowded scene, humans move their eyes to different people in the scene to check for identifying attributes and reject distracting content. Human gaze therefore provides a naturally grounded record of task-directed visual exploration.
This signal is particularly appealing for training vision-language models (VLMs). Although chain-of-thought reasoning allows VLMs to decompose a problem into textual steps, those steps may not remain faithfully connected to the image. Models may confuse which features belong to which objects [5, 4], or hallucinate visual evidence, producing an internally coherent trace whose intermediate claims are unsupported by the image [48] (Ma et al. 2026). Grounded reasoning methods address such failures by associating reasoning steps with explicit image locations, forcing models to cite evidence they are referring to in text [16]. Prior methods have explored training this grounded reasoning using tree search distillation or on-policy reinforcement learning (RL) to discover sequences of spatially grounded reasoning steps [39, 6]. These approaches can discover useful visual reasoning strategies, but depend on the model itself to ground reasoning steps accurately and propose exploration locations.
Human behavior offers a complementary source of guidance. Progress in language-model reasoning has benefited from human-written solutions and step-level annotations [9, 25]. Beyond answer correctness, the behaviors demonstrated in chain-of-thought—such as verification and backtracking—can shape subsequent improvement through RL [13]. For visual reasoning, gaze records precise spatial regions that people look at while exploring an image and processing visual evidence. Could gaze recordings provide similarly rich supervision for grounded reasoning?
In this work, we investigate which forms of gaze supervision support both strong visual reasoning and closer alignment with human evidence selection. Evaluating both outcomes is important to understand under what conditions gaze guidance transfers to VLMs. Task accuracy does not reveal whether models spatially explore similarly to humans, while gaze similarity alone does not establish that the transferred behavior supports effective reasoning. Human fixations also reflect perceptual and memory constraints that VLMs do not share [17, 2], so reproducing them in full may not improve both aims.
We systematically study how different constraints on gaze supervision transfer to VLMs in terms of task performance and human alignment. We collect 7,109 eye-tracking trials while human participants perform challenging chart understanding, counting, and visual search tasks. For each collected trial, we then provide an annotation model with the set of human fixations, along with the image and question, and have it annotate grounded reasoning under various constraints on the use of the provided human gaze (e.g., annotate every fixation or a selected subset of fixations). We finetune a compact model on the constructed datasets and evaluate them on challenging visual reasoning benchmarks. Our findings demonstrate that selective use of gaze is more effective than supervising every fixation for jointly improving both gaze alignment and task performance.
Based on our findings, we introduce GGR (Gaze-Guided Reasoner), which uses human fixations as a spatial prior for constructing grounded reasoning supervision. After gaze-guided supervised finetuning, GGR-SFT obtains the highest task performance and gaze alignment relative to all grounded baselines considered. After RL on GGR-SFT without gaze supervision, gaze-guided trained models further improve on human-gaze alignment, obtaining higher alignment than the strongest baselines while maintaining competitive task performance. Beyond these gains in gaze alignment, GGR produces reasoning whose visual claims correspond more reliably to the reported image locations compared to no-gaze baselines. GGR-RL also exhibits a distinct reasoning behavioral profile, expressing more initial exploration and reference-based navigation than no-gaze baselines. These results point to human gaze as a promising and largely untapped source of guidance for building AI systems whose visual reasoning is both effective and accurately grounded in the evidence people use.
Related work
Grounded reasoning in VLMs. Difficulties in binding object features in multi-object scenes have motivated recent methods that train VLMs to associate their reasoning steps with explicit image locations [5, 4]. Point-based supervision can improve object-centric processing [11, 16], and points or bounding boxes can be incorporated directly into reasoning traces to improve performance across broader grounded reasoning tasks. Such supervision is straightforward when the task specifies what to identify, but more complex multi-step tasks require discovering a sequence of relevant regions. ViGoRL-MCTS constructs grounded traces through tree search before applying RL [39], while related methods use answer-based rewards to learn where to gather visual evidence, including which regions to crop or zoom into [6, 46]. These approaches support flexible grounding, although the strategies available for training depend on model-generated proposals. We use human gaze to guide grounded reasoning trace proposal, comparing against ViGoRL-MCTS to assess the relative benefit of human- vs. model-guided traces.
Eye movements and visual reasoning. Eye movements constrain how people gather visual information in response to task demands. Different questions about the same image elicit different fixation patterns [49], and during natural behavior, gaze is directed toward locations relevant to the current reasoning step, often just before the information is needed [17, 22]. Fixation sequences also provide evidence about the coordination of perceptual and cognitive operations [18], including the interplay between reading and interpreting chart elements [7] and the shifts in attention associated with problem solving [15]. Repeated fixations can allow people to consult external information instead of retaining it in memory [1]. Ballard et al. (1997) describe such fixations as pointers connecting cognitive operations to locations in the world, emphasizing how eye movements help organize the use of visual evidence over time.
Learning from human behavior. Human behavioral signals have been used to train neural networks toward closer correspondence with human processing [26, 12, 31, 42, 3, 30]. Such correspondence is not captured by accuracy alone, since comparable performance can conceal different solution strategies [14, 19]. Human gaze has been used to supervise attention to task-relevant information. One line of work aggregates fixations into spatial salience maps, supervising where a model attends without retaining the order [33, 37, 36]. A second line retains ordered fixation trajectories to supervise sequential visual processing, as explored in recent radiology models [24, 23, 55]. However, agreement with human attention does not consistently predict task accuracy [41], and fixation sequences vary across observers and reflect constraints that VLMs do not share [47, 19]. Our work examines gaze as selective guidance for reasoning-trace generation, evaluating task performance and behavioral alignment separately.
Method
Problem setup
Our objective is to learn a vision-language model (VLM) that obtains strong visual reasoning performance while selecting visual evidence aligned with human gaze behavior. We consider a training dataset , where is an image, is a natural-language question, is the ground-truth answer, and is a human fixation trace recorded during problem solving. Each denotes a fixation location in image coordinates. Additional details of human data collection can be found in § 3.2.3.
We represent a grounded reasoning step as , pairing a textual thought with an image location that identifies the visual evidence it refers to. A trajectory consists of these steps followed by a predicted answer . Our goal is to design a learning algorithm that maps the training dataset to a grounded reasoning policy , with the objective of improving answer accuracy and gaze alignment on unseen examples. At inference time, conditions only on the image and question . Human fixations are available only during training.
Learning Grounded Reasoning from Human Gaze.
Our goal is to learn a grounded reasoning trajectory , where is the evidence proposal and is the reasoning supervision. While human fixation traces contain useful task-directed information, they may not themselves be ideal supervision as the evidence proposal for a VLM. A fixation may correspond to identifying a target, inspecting an attribute, rejecting a distractor, verifying an intermediate conclusion, or simply revisiting previously observed evidence. Moreover, human gaze paths reflect perceptual and memory constraints that do not apply to VLMs.
Creating a strong visual reasoning trace therefore requires deciding which fixation locations should be associated with a reasoning step, and whether to allow grounding at additional locations. We investigate how these choices affect task performance and human gaze alignment. To do so, we use an annotation model (Qwen3.5-122B-A10B) to construct grounded reasoning traces under different constraints on the use of human gaze. We detail this process below.
Constructing Gaze-Guided Reasoning Supervision
We provide a strong annotation model (Qwen3.5-122B-A10B) with the image , question , and fixation coordinate list , and prompt it to generate a grounded reasoning trace , where is guided by (Figure 2b). We investigate different forms of gaze guidance in § 4.1. These include constraints on whether each fixation must be annotated with a reasoning step, and if the annotation model can add additional regions outside the human fixations. Importantly, we do not provide the reference answer while the annotation model constructs the reasoning trace. The model must derive an answer from the image and selected evidence rather than generate a post-hoc rationale for the ground-truth answer ([50]).
Figure 2. Learning grounded reasoning with gaze guidance. (a) Eye gaze was recorded while human participants solved visual reasoning problems. (b) We give the image , question , and fixation list to an annotation model, which annotates regions guided by human fixations to form a grounded reasoning trace. (c) We train on these traces using SFT followed by RL, where each grounded step pairs a reasoning statement with its grounding location . (d) We evaluate the models on task performance, gaze alignment, reasoning behavior, and spatial grounding accuracy.
Sampling and verification. For each trial, we sample 25 candidate trajectories. We apply rejection sampling ([45]), giving the image, reasoning trace, and answer back to the annotation model to have it flag samples for 1) wrong answers, 2) redundancy in the model’s response, 3) inconsistency between thinking and final answer, and 4) self-contradictions. We remove any sample from training which triggers any of these flags. The accepted trajectories form our gaze-guided SFT dataset. Annotation prompts and details are provided in Appendix § 6.§ 2.
Training the Gaze-Guided Grounded Reasoner
After obtaining the gaze-guided grounded reasoning supervision from the process in § 3.2.1, we train a base VLM to mimic the grounded traces, followed by reinforcement learning using task accuracy rewards without gaze guidance. Our training pipeline is shown in Figure 2c. We detail each stage below.
Supervised fine-tuning. Using the fixation-guided dataset we obtain from § 3.2.1, we train a compact model with supervised finetuning on the dataset to encourage grounding aligned with gaze behavior. The resulting model thus learns to mimic the gaze-guided reasoning in the training traces, using only the image and question at inference time.
Reinforcement Learning. We then apply on-policy reinforcement learning to GGR-SFT to refine the grounded model behavior without human gaze. We follow the Vero [38] recipe, and use GSPO [54] with clip-higher, no KL penalty, and soft overlong penalty. The total reward combines 95% answer accuracy and 5% format compliance (<think> tags, boxed answer, and at least one valid coordinate). Our accuracy reward uses text matching for short answers, math-verify for numerical answers, and absolute error decreasing exponentially for counting rewards.
Implementation details. For supervised finetuning of our models and baselines, we train five epochs and choose the best checkpoint as determined by the validation accuracy. We use the 698 human trials as our validation set and only select based on accuracy. For reinforcement learning, all models train 400 steps, taking the best checkpoint by validation accuracy. For RL, we use 100,000 Vero chart QA examples [38], 9,667 visual search examples from PixelReasoner and Visual Probe subsets in Vero [38], and 7,785 counting examples curated from FSC147, REC-8K, PixMo, and RPINE. We sample batches with equal representation from each domain. We provide SFT and RL training configurations in Table A1 and Table A2, respectively. Full SFT and RL hyperparameters are provided in Appendix § 6.§ 4.
| Setting | Configuration |
| Learning rate | |
| Schedule | Cosine decay with 3% warmup |
| Weight decay | 0 |
| Gradient clipping | Max norm 1.0 |
| Training epochs | 5 |
| Batch size | 64 |
| Maximum sequence length | 32,768 tokens |
| Image pixel bounds | 921,600–8,388,608 |
| Precision / parallelism | BF16 / DeepSpeed ZeRO-3 across 8 GPUs |
Table A1. SFT training configuration.
| Setting | Configuration |
| Learning rate | 20-step warmup to ; linear decay to at step 400 |
| Prompt batch / optimization minibatch | 256 / 128 |
| Responses per prompt | 8 |
| GSPO clipping | , |
| Rollout sampling | Temperature 1.0, top- 1.0 |
| Maximum prompt / response length | 16,384 / 14,336 tokens |
| Precision / parallelism | BF16 / FSDP2 across 8 GPUs |
Table A2. RL training configuration.
Collecting Human Gaze Traces
To collect human gaze traces for reasoning annotation, we recruited participants to solve visual reasoning problems while we tracked their eye gaze (Figure 2a). Participants completed chart question answering, counting, and visual search trials while participating in the study. Images were scaled so that their longer dimension occupied 1,267 pixels, with questions presented below the image. Real-time synchronization between PsychoPy and the eye tracker allowed participants to zoom into or out of the image around their current gaze location. Participants could also place markers at their gaze location to keep track of counted objects or previously searched regions, reducing working-memory demands during extended trials. We collected 7,109 trials (2,145 visual search, 2,652 counting, 2,312 chart) in total with fixation coordinate traces for each trial. We use 6,411 trials for SFT training and 698 for held-out evaluation trials. Additional details on data collection are in Appendix § 6.§ 1.
Results
Our experiments aim to understand how human gaze supervision affects visual reasoning performance and gaze alignment in vision-language models. As shown in Figure 2d, we seek to understand how each condition and baseline performs on task accuracy, gaze alignment, spatial grounding accuracy, and chain of thought behavior.
Visual reasoning evaluation suite. We curate a suite of challenging and diverse benchmarks to test the ability of our method and baselines on novel problems in chart understanding, counting, and visual search. We evaluate on ChartQA [28] and ChartQA-Pro [29] for chart understanding, CountBenchQA [32] and CountQA [43] for counting, and MME-RealWorld-Lite [52] and Visual Probe [21] for visual search in high resolution images.
Gaze alignment. We measure spatial correspondence between model grounding points and human fixations. We form a Gaussian kernel density estimate over model coordinates with standard deviation and uniform background weight , then compute the mean log-likelihood of human fixations relative to that of a uniform spatial baseline (following Kümmmerer et al. (2022)). For each model tested, we fit via grid search. Of the 698 held-out trials, 336 are used to calibrate the gaze-alignment metric and 362 to evaluate it. Behavioral analyses use all 698 held-out trials. The metric does not capture fixation order, duration, or the model’s internal attention. See Appendix § 6.6 for additional details.
Human gaze supervision is most effective when used as a flexible prior
To systematically study the impact of different forms of gaze-guided supervision on task performance and human alignment, we train on data generated by conditioning a strong model (Qwen3.5-122B-A10B) on gaze while varying the constraints as outlined in § 3. In particular, we compare the following conditions: (1) the annotation model annotates each fixation with a textual reasoning step (Gaze-All), (2) the annotation model selects a strict subset of the human fixations and annotates each selected fixation with a reasoning step (Gaze-Subset), (3) the annotation model uses the fixation list as a prior, choosing which fixations to annotate and grounding additional regions when needed (Gaze Guided Reasoner; GGR), (4) we directly use fixation coordinates interleaved with transcribed human think aloud (Raw-Gaze), (5) we use the Gaze-All condition with fixation lists shuffled across examples (Shuffled), (6) the annotation model generates grounded reasoning without human fixation guidance (No-Gaze). We report the results in Table 1.
| Condition | SFT Chart | SFT Counting | SFT Search | SFT Overall | SFT Gaze alignment [bits/fix] | SFT RL Chart | SFT RL Counting | SFT RL Search | SFT RL Overall | SFT RL Gaze alignment [bits/fix] |
| Human split-half | – | – | – | – | – | – | – | – | ||
| Controls | ||||||||||
| No-Gaze | ||||||||||
| Shuffled | – | – | – | – | – | |||||
| Human-gaze supervision | ||||||||||
| Raw-Gaze | ||||||||||
| Gaze-All | ||||||||||
| Gaze-Subset | ||||||||||
| GGR |
Table 1. Comparing strategies for human-gaze supervision. Task accuracy (%) and human-gaze alignment after SFT and subsequent RL on Qwen3-VL-8B-Instruct. Accuracy averages the two benchmarks in each domain or all six overall. Reported is Avg@3 (± SEM) across three evaluation runs and gaze alignment is mean ± SEM in bits/fix. Benchmark-level results are in Appendix Table A7.
GGR improves gaze alignment and reasoning quality without sacrificing performance
We investigate which constraints on human gaze supervision lead to improvements in task performance and gaze alignment. We first compare Gaze-All, where we require a textual explanation at every fixation, with Gaze-Subset, where we allow the annotation model to select which fixations to include. After SFT on Qwen3-VL-8B-Instruct, overall accuracy increases from 55.9% with Gaze-All to 62.8% with Gaze-Subset (Table 1). We observe improvements on all six benchmarks (Appendix Table A7). Although gaze alignment is higher with Gaze-All than with Gaze-Subset (1.04 versus 0.82 bits/fix), task accuracy is lower. Gaze-All exhibits significant re-examination behavior (increase of 54.8 percentage points relative to the base model after RL; ; Figure 5a), which often leads to self-confusion and repeated loops. These results suggest that selective use of gaze is more effective than supervising every fixation.

Figure 5. Behavioral changes under gaze-guided and no-gaze training. a. Changes relative to Qwen3-VL-8B-Instruct on six behaviors: visual exploration, re-examination, task decomposition, reference-based navigation, result verification, scope definition, in percentage points of behavior presence. b. Reasoning-length changes relative to the same base model in tokens. Open and filled markers denote selected SFT and RL checkpoints. Mean SEM ().
Finding 1: Selective use of gaze is more effective than supervising every fixation.
We next ask whether the selected evidence should be restricted only to locations that humans fixated. With Gaze-Subset, fixations can be omitted but cannot include evidence from other regions. This restriction may exclude evidence that the model finds useful, since a person may not look directly at every region that could support the answer. To test whether this additional flexibility improves reasoning, we use gaze as a prior in GGR (Gaze-Guided Reasoner). We treat human fixations as a source of candidate visual evidence by providing the annotation model with the fixation list, while allowing the model to select which regions to include in a grounded reasoning trace, and add additional regions when needed.
After SFT, overall accuracy is 2.5 percentage points higher with GGR-SFT than with Gaze-Subset. After RL without gaze supervision, overall accuracy remains 1.5 percentage points higher with GGR-RL than with Gaze-Subset, while improving gaze alignment (1.15 GGR-RL versus 0.99 Gaze-Subset bits/fix; ). Both conditions remain more accurate overall than Gaze-All (Table 1). We observe the same ordering after SFT with Gemma 3 4B and MiMo-VL-7B, with overall accuracy highest for GGR-SFT, followed by Gaze-Subset and Gaze-All (Appendix Tables A9 and A10). We thus find using gaze as a prior (GGR) improves task accuracy and gaze alignment most among conditions tested.
Finding 2: Selective use of gaze is more effectively implemented as a prior rather than as a strict subset.
Our results suggest that better reasoning and greater human-gaze alignment can coexist, but stronger alignment alone does not imply better performance. After SFT, gaze alignment is higher with Gaze-All than with GGR-SFT (1.04 versus 0.84 bits/fix; ), although overall accuracy is lower. We observe the same pattern with Raw-Gaze-SFT (Table 1). After removing the correspondence between gaze and the task by shuffling fixation traces across examples, gaze alignment falls to 0.01 bits/fix, while overall accuracy remains similar (55.8% versus 55.9% for Gaze-All). We therefore find that task-specific gaze guidance matters for human alignment even when accuracy changes little.
GGR improves gaze alignment and reasoning quality without sacrificing performance
Building on the findings of the previous section, we now examine how our best gaze-guided method, GGR, compares to grounded reasoning baselines that do not use gaze guidance. We compare with ViGoRL-MCTS (Sarch et al., [39]), for which we generate the training traces through tree search, No-Gaze, where we use the same annotation model and procedure as GGR to generate grounded reasoning without providing human fixations, and several open and closed general reasoning models (Clark et al., [8]; Singh et al., [40]; Qwen Team, [34]) (Table 2).
| Condition | ChartQA Pro | ChartQA | Count BenchQA | CountQA | MME RealWorld Lite | Visual Probe | Overall | Gaze Alignment [bits/fix] |
| Human split-half | – | – | – | – | – | – | – | |
| Reference models | ||||||||
| GPT-5-Nano | 56.0 | 80.1 | 75.4 | 25.7 | 49.8 | 41.5 | 54.8 | – |
| Molmo2-O-7B | 31.4 | 75.2 | 89.4 | 32.1 | 44.4 | 34.8 | 51.2 | – |
| DeepEyes-7B | 36.3 | 76.7 | 80.0 | 18.7 | 49.0 | 40.6 | 50.2 | – |
| Vero-Qwen3I-8B | 63.0 | 91.9 | 92.7 | 33.4 | 57.2 | 47.0 | 64.2 | – |
| Qwen3-VL-8B-Instruct | ||||||||
| Supervised fine-tuning (SFT) | ||||||||
| ViGoRL-MCTS | ||||||||
| No-Gaze | ||||||||
| GGR-SFT | ||||||||
| Reinforcement learning (RL) | ||||||||
| Point-Grounded RL | ||||||||
| Ungrounded RL | – | |||||||
| No-Gaze-SFT RL | ||||||||
| GGR-RL |
Table 2. GGR compared with existing models and training baselines. Task performance and human-gaze alignment for reference models, SFT methods, and RL methods. All SFT and RL methods use Qwen3-VL-8B-Instruct. Task scores report Avg@3 ( SEM) across three evaluation runs. Overall is the unweighted mean of the six benchmarks. Gaze alignment is mean SEM in bits/fix.
After SFT, GGR obtains the highest overall accuracy among the models considered, exceeding the strongest baseline, No-Gaze, by 1.1 percentage points in overall performance and 0.13 bits/fix in gaze alignment (). After RL, GGR-RL improves overall accuracy from 65.3% to 68.8% and gaze alignment from 0.84 to 1.15 bits/fix (), matching No-Gaze in performance while achieving higher gaze alignment (1.15 versus 0.99 bits/fix; ). GGR therefore maintains the accuracy of the strongest baseline while grounded reasoning is more aligned with human gaze.
Finding 3: GGR maintains the accuracy of the strongest baseline while being more aligned with human gaze.
Given that gaze provides precise candidate locations for grounding as humans reason, we further investigate whether gaze-guided supervision improves the spatial grounding and reasoning quality. We use GPT-6.1-Sol-xlhigh to assess grounding locations and their associated reasoning steps (). We validate the model ratings with two human raters ( samples; 87–90% agreement with model; Appendix Table A5). Full methods are in Appendix § 6.7.
Compared to our method without gaze guidance (No-Gaze), GGR obtains a higher rate of supported claims that are correctly grounded after both SFT and RL, with gains of 4.5% and 4.4%, respectively (Figure 3; Appendix Figure A6; ). GGR-RL additionally achieves better grounding accuracy regardless of whether the claim is supported (Table A6 Point on/near target; 84.1% with GGR-RL compared with 79.8% for No-Gaze-RL; ). Our results demonstrate that gaze guidance improves the rate of supported reasoning claims and accuracy of the visual grounding.

Figure 3. GGR improves visual grounding quality compared to the no gaze condition. Paired GGR minus No-Gaze differences in visual grounding after RL. Reported is mean ± bootstrapped 95% CI. SFT results are in Appendix Figure A6.
| Stage | No-Gaze | GGR |
| Supported claim + point on/near target | ||
| SFT | 58.7 [56.5, 60.9] | 63.3 [61.0, 65.5] |
| Post-RL | 70.5 [68.4, 72.6] | 74.9 [72.8, 77.0] |
| Point on/near target | ||
| SFT | 68.3 [66.1, 70.3] | 72.7 [70.7, 74.8] |
| Post-RL | 79.8 [77.9, 81.7] | 84.1 [82.3, 85.8] |
Table A6. Visual-grounding condition rates. Judgements of 1,500 matched questions (250 per benchmark; 6,000 outputs). The evaluation assesses one randomly sampled coordinate occurrence and its local claim. We report average (%) and 95% CI. Bold marks the higher rate in each row.
Finding 4: GGR produces visual reasoning claims that correspond more reliably to the reported image locations.
Gaze-guided supervision improves alignment coverage and precision
In section 4.2, we established that GGR improves gaze alignment compared to the no gaze conditions. In this section, we investigate in what ways gaze-guided supervision improves this gaze alignment. We examine gaze coverage and precision between the model and human locations. We define gaze coverage as the fraction of human fixations within a fixed radius of a model grounding location and gaze precision as the fraction of unique model points that are within a fixed radius of a human fixation. We use 100 normalized coordinate units as the radius.
Compared to the initial base model, GGR-SFT significantly increases both coverage and precision (Figure 4a; 41.9% GGR-SFT vs. 27.0% Base-CoT coverage; 83.5% GGR-SFT vs. 51.2% Base-CoT precision). Relative to ViGoRL-MCTS (Sarch et al. [39]) and our method without gaze guidance (No-Gaze-SFT), GGR-SFT also improves both coverage and precision (Figure 4a; all six comparisons ). We additionally consider a point-budget-matched coverage comparison, using the first points in the reasoning trace. Under this setup, GGR-SFT maintains improvements over both ViGoRL-MCTS and the No-Gaze-SFT condition (Figure 4c). Thus, GGR-SFT improves both coverage and precision under full and budget matched comparisons to no-gaze baselines.
Figure 4. Gaze alignment and coverage. (a) Coverage versus precision at a radius of 100 normalized coordinate units. (b,c) Alignment and coverage at point budgets for GGR-SFT, No-Gaze SFT, and ViGoRL-MCTS. Open and filled markers denote SFT and RL, respectively. Reported is mean ± SEM.
Finding 5: Relative to supervised baselines not using gaze, GGR-SFT increases both gaze coverage and precision.
After applying RL without the use of gaze supervision, GGR-RL remains the model with the highest coverage and precision among those tested (Figure 4a; full coverage 60.2% and precision 83.7%, coverage at is 54.8% GGR-RL vs. 49.6% No-Gaze-RL and 41.3% Point-Grounded RL; Appendix Figure A7b). RL on GGR-SFT largely improves coverage (41.9% → 60.2%), while precision remains high (83.5% → 83.7%). The same trend in coverage and precision persists when doing RL on the No-Gaze-SFT (Figure 4a; coverage 38.2% → 52.0%; precision 78.2% → 79.8%), while Point-Grounded RL improves both coverage and precision similarly (Figure 4a).
Finding 6: RL on GGR-SFT increases gaze coverage while maintaining precision.
Gaze-guided supervision changes reasoning behavior
To examine how gaze-guided training alters reasoning behavior, we analyze how the chains of thought behaviors of GGR-RL and No-Gaze-RL change relative to the base model (Qwen3-VL-8B-Instruct) we are training. Following Gandhi et al. [13], for each behavior, we use a strong LLM (Qwen3.5-122B-A10B) to count occurrences in each chain of thought (held-out set, ). We report six behaviors which we determined based on inspecting the raw human fixation traces. These behaviors include visual exploration, re-examination, task decomposition, reference-based navigation, result verification, and scope definition. Full behavior definitions are provided in Appendix Table A4. Presence denotes the average behavior presence across reasoning traces. We report the results in Figure 5.
We find that GGR-RL increases exploration and task decomposition compared to the base model and the no gaze condition. After RL, GGR-RL increases exploration by 3.9 and task decomposition by 18.7 percentage points of behavior presence relative to the base model, compared with 1.0 and −4.1 for No-Gaze RL (Figure 5a). Reference-based navigation also increases (+8.1) compared with −4.3 for No-Gaze RL. Scope definition remains near the base model (−1.1) compared with +6.4 for No-Gaze RL.
On the other hand, gaze guidance reduces result verification and re-examination compared to the condition without gaze guidance. Result verification and re-examination change by −6.6 and +2.8 percentage points, respectively, for GGR-RL compared with +25.7 and +15.7 for No-Gaze RL (Figure 5a; ). These findings suggest that GGR-RL does more visual exploration before arriving at an answer, while the No-Gaze-RL condition spends more time verifying an already established answer. We provide an example of this behavior difference in Figure 6.

Figure 6. Example of GGR eliciting useful grounded exploration compared to baselines. On this example, Point-Grounded RL grounds few locations and commits to a location, Gaze-All re-evaluates many locations, and GGR explores relevant locations before finding the correct answer.
Finding 7: GGR-RL does more visual exploration before arriving at an answer, while the No-Gaze-RL condition arrives at an answer quickly but spends more reasoning effort verifying the answer.
Discussion
In this work, we investigated how human gaze can be transferred to modern vision-language models. Across chart understanding, counting, and visual search, we find that gaze is most useful as a spatial prior for grounded reasoning. Our findings motivate GGR, which uses a strong model to annotate reasoning conditioned on the gaze locations. GGR obtains strong visual reasoning task performance while being more aligned with human gaze and producing more reliable visual claims compared to grounded reasoning baselines that do not use gaze guidance.
Our results suggest that human behavior can be useful to a system with very different perceptual and cognitive constraints, but how the behavior is transferred matters. Supervision that annotated every fixation with a reasoning step (Gaze-All) significantly underperformed using a subset of gaze (Gaze-Subset) and using gaze as a prior (GGR). This suggests that the effectiveness of human guidance depends on how it is transferred and represented during training (§ 4.1). A sequence of fixations is a useful learning signal, but requiring a reasoning step for each fixation in a grounded reasoning trace is an ineffective way to transfer that signal to modern VLMs. This aligns with prior work that uses an abstraction of human behavior, rather than the raw behavior itself, as the supervision signal ([37, 44, 36]). Humans have attention and memory constraints that are missing from modern VLMs, and these constraints may require selective use of the human behavior when transferring it to a VLM with different constraints.
Our results also show that human guidance can leave persistent behavioral and grounding changes after further training without gaze. After doing RL on GGR-SFT, we find persistent grounding accuracy and behavior changes compared to running the same method but without gaze guidance (§ 4.3 and § 4.5). This is consistent with prior work showing that reinforcement learning amplifies and shapes behaviors already present in the base model ([13, 53]). We show that gaze-guided supervised learning improves grounding reliability and amplifies visual exploration, and these behaviors persist after subsequent reinforcement learning.
To understand grounded reasoners in greater depth, we argue for measuring multiple metrics beyond accuracy, as similar accuracy does not establish similar strategies. We investigate multiple dimensions of model output in our work, including task success, alignment to gaze behavior, grounded reasoning reliability, and chain of thought reasoning behavior. We show that optimizing for one of these metrics does not imply predictable changes in the other metrics. For example, while finetuning on the raw gaze trace improved gaze alignment significantly, task performance dropped below the base model it was trained on. In fact, we show that improved reasoning trace quality (§ 4.3) is not reflected in accuracy compared to the baseline without gaze, consistent with prior work showing dissociations between task performance and model behavior ([14, 12]). Task accuracy alone offers a narrow perspective on visual reasoning behavior.
Several areas for future work remain. While we examined multiple base models (§ 4.1), we did not fully investigate how different pretraining data or architectures interact with the usefulness of the gaze supervision. Furthermore, the VLMs tested have significantly different constraints relative to human observers. Future work should examine how biologically-plausible attention and working memory constraints change the effectiveness of gaze supervision.
Acknowledgments. This work was supported by Princeton Research Computing and Princeton Language and Intelligence (PLI) resources, including the Della high-performance computing cluster.
Ethics statement. This work involves research with human participants. The eye-tracking study was conducted under a protocol approved by an institutional review board (IRB). All participants provided informed consent before participation and were informed that participation was voluntary and that they could withdraw at any time.
Appendix
Additional Human Data Collection Details
Source datasets. Figure A1 shows the source-dataset composition of 7,109 trials, with 6,411 used for training and 698 for validation. We had 58 participants in total and compensated $20 per hour.
Vector (PDF)| Source dataset | Trials (%) |
| Visual Probe | 1,879 (26.4%) |
| PixMo | 1,028 (14.5%) |
| TallyQA | 750 (10.6%) |
| New counting (reviewed) | 637 (9.0%) |
| EvoChart | 377 (5.3%) |
| CoSyn charts | 355 (5.0%) |
| ChartQA difficulty | 300 (4.2%) |
| CoSyn diagrams | 280 (3.9%) |
| PixelReasoner | 266 (3.7%) |
| ArXivQA | 255 (3.6%) |
| ECD VQA | 250 (3.5%) |
| ReachQA | 187 (2.6%) |
| InfographicVQA | 157 (2.2%) |
| CoSyn tables | 151 (2.1%) |
| OODVQA | 130 (1.8%) |
| Multihop counting | 107 (1.5%) |
Figure A1. Source-dataset composition (; 6,411 training and 698 validation trials). Counts and percentage shares are shown for each source.
Data collection. English-speaking participants with normal or corrected-to-normal vision were recruited under an institutional review board-approved protocol. All participants provided informed consent and were informed that participation was voluntary and that they could withdraw at any time. Stimuli were presented using PsychoPy on an Asus VG27AQ LCD monitor operating at pixels and 120 Hz. Participants viewed the display from approximately 89.5 cm with their head stabilized using a chin rest. Right-eye gaze was recorded using an SR Research EyeLink 1000 Plus desktop system at 1,000 Hz. Calibration and validation were performed before each session using the standard nine-point EyeLink array, with drift-correction trials interleaved every 10–20 task trials.
Participants were instructed to verbalize their reasoning and final answer before proceeding to the next trial. Trial audio was recorded and transcribed using Whisper large-v3 ([35]), with word-level timestamps. These verbalizations were not used to generate training traces because participants varied substantially in how extensively they explained their reasoning.
Eye-tracking recordings were processed at the fixation-event level. Finalized EDF files were converted to an ASCII event representation using EyeLinkIO (version 0.3.0) and the EyeLink EDF API, retaining the fixation onset times, durations, and mean screen-space gaze positions recorded by EyeLink. No additional offline fixation detection, duration filtering, or fixation merging was applied. Recorded synchronization markers aligned the PsychoPy and tracker clocks, and fixation times were expressed relative to trial onset. Fixations ending before trial onset were discarded, while those spanning trial onset were truncated to their within-trial duration.
To place gaze from different viewing states in a common reference frame, screen-space fixation positions were mapped to original-image coordinates by inverting the stimulus display geometry and applying the logged zoom level and viewport center active at fixation onset. Fixations mapped outside the original image were discarded, and retained coordinates were rounded to image pixels. No additional interpolation of missing gaze samples or explicit blink-window filtering was performed. Trials explicitly flagged as bad were excluded; participant-level exclusions were made only when manual inspection indicated particularly poor eye-tracking data quality.
Model Annotation Details
GGR: Single-Stage Annotation
For GGR, the teacher receives the image , question , and fixation coordinate list , and directly generates a grounded reasoning trace in a single stage. The fixation list serves as a spatial prior where the teacher may omit supplied fixations and ground additional locations when needed. The reference answer is withheld during trace generation.
Listing A.1 gives the gaze-conditioned prompt.
Listing A.1: GGR: Gaze-Conditioned Annotation Prompt
You are given:
1. an image,
2. a question,
3. human fixation points recorded while a person solved the task.
Your goal is to generate a visually grounded reasoning trace with inline grounding coordinates that supports the final answer.
IMPORTANT: Treat the human fixation points only as a SOFT SPATIAL PRIOR over where useful visual evidence may be located.
The fixation points are NOT target grounding coordinates and are NOT in one-to-one correspondence with reasoning steps or objects.
Use the fixation points to guide where you inspect the image, but determine every grounding coordinate from the IMAGE ITSELF.
For each reasoning step:
1. Identify the actual visual entity or region that supports the claim.
2. Place the grounding coordinate on that entity or region.
3. Do not copy or reuse a fixation coordinate unless it is also an appropriate visual grounding.
4. If the fixation information and the visible image disagree, trust the image.
5. Do not force every fixation to appear in the reasoning.
6. Do not make up grounding coordinates that are not supported by the visible image just to support the final answer.
Output:
• Output only the visible reasoning trace with inline grounding coordinates formatted as (x, y) followed by exactly one \boxed{<answer>}.
• Do not make any reference to the ’fixation list’ in your final output.
No-Gaze SFT: Single-Stage Annotation
For No-Gaze SFT, the teacher receives the image and question , and directly generates a grounded reasoning trace in a single stage without human fixation guidance.
Listing A.2 gives the no-fixation prompt.
Listing A.2: No-Gaze SFT: Annotation Prompt
You are given:
1. an image, and
2. a question.
Your goal is to answer the question from the visible image with a visually grounded reasoning trace and inline grounding coordinates.
For each reasoning step:
1. Identify the actual visual entity or region that supports the claim.
2. Place the grounding coordinate on that entity or region.
3. Trust the visible image when determining the evidence and coordinates.
4. Do not make up visual evidence or grounding coordinates.
Output:
• Output only the visible reasoning trace with inline grounding coordinates formatted as (x, y) followed by exactly one \boxed{<answer>}.
Gaze-All: Three-Stage Annotation
Gaze-All uses three annotation stages. First, the annotation model assigns a visual description to each candidate fixation in , identifying the object or region at that location. Second, conditioned on the question , it annotates the task relevance of these regions, such as identifying a target, ruling out a distractor, or checking for an omitted object. Third, the model uses the image, question, and annotated regions to generate candidate reasoning traces, with an accompanying reasoning step for every fixation. We exclude the participant’s raw think-aloud transcript from this generation stage and withhold the reference answer , requiring the model to derive its answer from the visual evidence.
Gaze-Subset: Three-Stage Annotation
Gaze-Subset uses the same three-stage annotation procedure as Gaze-All, but the final stage generates a reasoning trace using a strict subset of the supplied fixations, with no additional grounding points. Only Gaze-All and Gaze-Subset use the three-stage procedure; GGR and No-Gaze SFT generate traces directly from their respective inputs.
Listing A.3 gives the final-stage trace-generation prompt for Gaze-Subset.
Rejection and Visual Verification
We apply the same filtering procedure to candidate traces from GGR, No-Gaze SFT, Gaze-All, and Gaze-Subset, using answer checks and image-grounded assessments of reasoning quality. The checks remove candidates with incorrect answers, invalid point references, excessive repetition, or commentary about the annotation process. A subsequent visual assessment checks whether the trace identifies the correct objects and attributes, supports its claims with appropriate spatial references, and avoids counting errors, contradictions, or unsupported conclusions. Accepted traces are converted into training targets containing the grounded reasoning and final answer.
Baseline Details
Raw-Gaze-SFT/RL. We process the raw fixation sequence and think aloud into a single reasoning chain. We create a flattened list of (x, y) coordinates and interleave the think aloud in between the coordinate list at locations that align based on the Whisper transcriptions. Each flattened list + think aloud + final answer is converted into a training target.
Listing A.3: Gaze-Subset: Final-Stage Reasoning Trace Prompt
Write a task-directed visual reasoning trace from the image, question, and annotated human fixation points.
POINT SELECTION
Select useful supplied points. Omit others and use them in any semantic order.
Cite at least one supplied point using its
<N>placeholder.Cite individual points, not coordinate ranges or grouped IDs.
Describe the scene directly; do not narrate the human scanpath or annotation process.
REASONING
Ground observations in the image and keep them relevant.
Verify object identity, attributes, counts, and relationships.
A correction or one verification pass is acceptable. Avoid repeated recounting, irrelevant exploration, and loops.
Never mention reference answers, judges, or previous attempts.
OUTPUT
Return only the reasoning trace followed by exactly one \boxed{answer}. Do not include <think> tags or text after the answer.
Image: {image}
Question: {question}
Annotated fixations:
<1> {visual label} — {task-related function}
<2> {visual label} — {task-related function}
...
ViGoRL-MCTS. Following Sarch et al. [39], we generated spatially grounded reasoning trajectories with Qwen3.5-122B-A10B using Monte Carlo Tree Search (MCTS), with 16 simulations per example, four children per expansion, and four rollouts per child. The teacher received the image and question and generated reasoning steps grounded in image coordinates, while reference answers were used to score completed rollouts and guide search. We retained answer-correct trajectories and constructed correction examples by pairing unsuccessful and successful continuations from the same node. After deduplication, we selected up to 25 trajectories per example.
Training and Evaluation Details
SFT Training Protocol. All conditions use the same system prompt (Listing A.4) during training and evaluation. We fine-tune using supervised next-token prediction on the reasoning trace and final answer. Training uses full-parameter fine-tuning of the language model with the vision encoder frozen.
RL Training Protocol. We use the system prompt in Listing A.4. We train with GSPO for 400 training steps. Each batch contains 256 prompts balanced across chart, counting, and search tasks. Advantages do not use standard-deviation normalization. We do not apply KL regularization nor an entropy bonus. We use full-finetuning of all parameters. RL hyperparameters are detailed in Table A2.
The reward combines task-specific answer correctness (weight 0.95) and valid point-grounded reasoning format (weight 0.05); invalid formatting also zeros the correctness component. For counting, the correctness reward is for absolute errors below four and zero otherwise.
Evaluation Protocol.
Listing A.4: SFT and RL System Prompt
You are a helpful, conversational assistant tasked with answering a question about an image. Generate a reasoning trace with reasoning grounded in points in the image ((x, y) coordinates) to help you answer.
Reasoning: A detailed, free-flowing chain of thought enclosed in
<think>and</think>tags, with reasoning grounded in points in the image ((x, y) coordinates). While reasoning, explicitly model these cognitive behaviors:Planning: Outline possible approaches before committing.
Exploration: Consider multiple image regions or interpretations.
Evaluation: Compare alternatives and verify against visual evidence.
Reflection: Revisit earlier ideas if they might still be viable.
Error Correction: Acknowledge mistakes if grounded reasoning reveals an incorrect path.
Response: After reasoning and reaching a conclusion, generate a clear, conversational response. After the response, indicate your final answer using
\boxed{}.
Evaluation Protocol
We detail our evaluation protocol in Table A3. We use lmms-eval [51] for all evaluations.
| Setting | Configuration |
| Chart benchmarks | ChartQA, ChartQA Pro |
| Counting benchmarks | CountQA, CountBenchQA |
| Search benchmarks | MME-RealWorld-Lite (Vero subsample), Visual Probe |
| Decoding | Temperature 1.0, top- 0.95, top- 20, presence penalty 1.5 |
| Output budget | 16,384 tokens |
| Image preprocessing | Min 2M and max 10M |
Table A3. Evaluation configuration.
Behavioral Analysis. We analyze reasoning traces from held-out gaze evaluations using Qwen3.5-122B-A10B as a text-only behavioral coder, with temperature zero and thinking disabled. The coder counts distinct, explicitly expressed occurrences in each rollout. We report two category scores and four individual behaviors (Table A4). We average three rollouts per example and report paired differences from the base model (mean SEM across 698 examples).
| Category / behavior | Coded behaviors / definition |
| Exploration | Exploring a different image region for visual evidence. |
| Re-examination | Revisiting of earlier observations and verification of provisional or final results. |
| Task decomposition | Dividing the task into organized intermediate steps or milestones. |
| Reference-based navigation | Uses a landmark or salient object to orient the search. |
| Result verification | Checks a provisional or final result for completeness, consistency, arithmetic, and compliance with the task. |
| Scope definition | Defines inclusion, exclusion, boundary, or ambiguity rules before processing. |
Table A4. Plotted behavioral categories and individual behaviors. Counts describe behaviors expressed in generated reasoning, rather than independently verified visual actions.

Figure A2. Behavioral contrasts relative to base in occurrences per 100 reasoning words. Categories, individual behaviors, and conditions match Figure 5a, which reports behavior presence. Reported is mean SEM across paired example differences after averaging three rollouts per example ().
Fixation Alignment Metric.
For example , let denote human fixation , and let denote grounding point from model response at checkpoint . Coordinates lie in the normalized image plane . After excluding invalid and out-of-bounds coordinates, we construct
where indexes training condition and is the number of valid reported points.
We score each response by its mean log-likelihood relative to uniform:
We jointly fit separately for each condition using 336 calibration examples. The candidate grid is
Selection maximizes calibration-set mean alignment.
Time-resolved and order-aware alignment. Our spatial alignment score does not take the ordering of fixations into account. The below three analyses extend this score to include fixation order.
Time-resolved alignment. We split the sequence of fixations into five ordered parts with equal numbers of fixations, and compute the alignment score on each part. Here, we discard the first two fixations (which reflect the central starting position) and exclude scanpaths with fewer than ten remaining fixations.
Order score. We compute the Goodman–Kruskal between fixation time and the position of the fixated region in the model’s response (1 if regions are fixated in the order the model mentions them, for the reverse order). Here, a region is a grounding point more than 50 units from every earlier region, each fixation is assigned to its nearest region within 100 units, and we require at least three assigned fixations on at least two regions.
ScanMatch-style similarity. Following ScanMatch [10], we align the sequence of regions with the sequence of human dwells (runs of consecutive fixations within 50 units of their centroid) using the Needleman–Wunsch algorithm. Here, a region and a dwell at distance score . To separate order from spatial overlap, we repeat the alignment with the regions shuffled (mean of ten shuffles) and report the difference per region for responses with at least two regions.
Human reference. For scanpaths with at least 24 fixations, we summarize the odd-numbered fixations by their ten largest dwells, ordered by first visit, and score this summary against the even-numbered fixations.
Additional Experiments
We compare models using paired permutation tests over source images. All p-values are two-sided and Holm-corrected for multiple comparisons.
Condition-fitted gaze alignment. Figure A3 reports alignment across supervision conditions before and after RL, complementing the precision and coverage analysis in Figure 4a.
Figure A3. Condition-fitted gaze alignment. Alignment after SFT (open bars) and after RL (filled bars). The dashed line denotes human split-half alignment; the dotted line denotes the uniform reference. Reported is mean SEM.
Time-resolved and order-aware gaze alignment. With shared parameters (Appendix § 6.6), GGR is more aligned than No-Gaze overall (Figure A4a) and in every fifth of the human scanpath (Figure A4b–Figure A4c; all ). ScanMatch-style similarity is also higher for GGR (SFT and RL both ).
Figure A4. Gaze alignment over viewing time and order agreement. (a) Alignment with shared parameters (, ). (b) Alignment within each fifth of the human scanpath. (c) Paired GGR minus No-Gaze difference in each fifth, with bootstrapped 95% CIs. (d) Order score. The dashed line denotes the human split-half reference (0.19). Open and filled markers denote SFT and RL, respectively. Panels a, b, and d report mean SEM.
We additionally compare gaze correspondence in teacher-generated reasoning supervision on 100 trials held out from training. Using the same teacher model, we compare GGR with Gazette [30] to obtain reasoning annotations. GGR achieves higher ScanMatch similarity (0.172 versus 0.103; paired difference 0.069, 95% CI [0.053, 0.085]) and lower human-to-trace spatial coverage distance (0.147 versus 0.207).
Figure A5 provides additional plots of GGR and Point-Grounded RL across RL steps for unique points output, human fixation coverage, and human fixation precision on the held out set. Figure A2 reports the behavioral contrasts in Figure 5a in occurrences per 100 reasoning words.

Figure A5. Spatial behavior across RL training. Reported is mean SEM across images. Open markers denote SFT, while filled markers denote RL.
Visual grounding. Figure A6 reports SFT differences; Table A6 reports condition rates. We used GPT-6.1-Sol to assess one sampled point, when present, and its associated claim. We provided the question, image, marked crop, and surrounding text. We assessed placement as on, near, off, or unclear, and the claim as supported, contradicted, or unclear. For joint grounding, we required a supported claim and a point on or near the target.
Two participants judged the same 100 outputs from 25 questions across six benchmarks. They could not see the conditions, reference answers, or assistant ratings. We compare their judgments to GPT-6.1-Sol in Table A5.
| Measure | H1 vs. assistant | H2 vs. assistant | H1 vs. H2 |
| Point on/near target | 90 (0.534) | 90 (0.561) | 92 (0.512) |
| Claim: supported, contradicted, unclear | 89 (0.437) | 87 (0.306) | 91 (0.275) |
| Supported claim + point on/near target | 86 (0.572) | 84 (0.525) | 90 (0.608) |
Table A5. Agreement with humans. Agreement (%) with Cohen’s in parentheses.

Figure A6. GGR improves visual grounding quality compared to the no gaze condition. Paired GGR minus No-Gaze differences in visual grounding after SFT. Reported is mean ± bootstrapped 95% CI.
Benchmark-level results for gaze supervision. Table A7 expands Table 1 with results on each of the six benchmarks after SFT and RL.
| Condition | ChartQA Pro | ChartQA | Count BenchQA | CountQA | MME RealWorld Lite | Visual Probe | Overall | Gaze Alignment [bits/fix] |
| Supervised fine-tuning (SFT) | ||||||||
| Controls | ||||||||
| No-Gaze | 59.6 ± 0.1 | 90.5 ± 0.0 | 94.9 ± 0.5 | 40.5 ± 0.3 | 54.2 ± 0.2 | 45.6 ± 0.9 | 64.2 ± 0.2 | 0.71 ± 0.05 |
| Shuffled | 49.1 ± 0.3 | 89.2 ± 0.1 | 82.6 ± 2.0 | 20.2 ± 0.5 | 51.5 ± 0.2 | 42.5 ± 1.1 | 55.8 ± 0.4 | 0.01 ± 0.00 |
| Human-gaze supervision | ||||||||
| Raw-Gaze | 31.2 ± 0.2 | 76.1 ± 0.2 | 85.0 ± 1.2 | 24.9 ± 0.3 | 41.9 ± 0.7 | 41.9 ± 0.7 | 50.2 ± 0.3 | 0.96 ± 0.06 |
| Gaze-All | 50.2 ± 0.2 | 88.4 ± 0.2 | 83.3 ± 0.5 | 22.8 ± 0.1 | 50.8 ± 0.6 | 39.8 ± 0.8 | 55.9 ± 0.2 | 1.04 ± 0.06 |
| Gaze-Subset | 58.6 ± 0.1 | 91.0 ± 0.2 | 93.3 ± 0.4 | 32.9 ± 0.4 | 55.2 ± 0.8 | 46.0 ± 1.0 | 62.8 ± 0.2 | 0.82 ± 0.05 |
| GGR-SFT | 58.4 ± 0.5 | 90.8 ± 0.0 | 95.0 ± 0.4 | 41.8 ± 0.3 | 57.7 ± 0.1 | 48.0 ± 0.7 | 65.3 ± 0.2 | 0.84 ± 0.06 |
| Reinforcement learning (RL) | ||||||||
| Controls | ||||||||
| No-Gaze-SFT → RL | 63.6 ± 0.4 | 92.0 ± 0.1 | 95.1 ± 0.0 | 52.9 ± 0.1 | 57.3 ± 0.3 | 51.8 ± 0.2 | 68.8 ± 0.1 | 0.99 ± 0.07 |
| Shuffled | – | – | – | – | – | – | – | – |
| Human-gaze supervision | ||||||||
| Raw-Gaze | 59.8 ± 0.4 | 90.9 ± 0.3 | 89.7 ± 0.4 | 31.4 ± 0.5 | 54.4 ± 0.2 | 53.5 ± 0.5 | 63.3 ± 0.1 | 1.02 ± 0.06 |
| Gaze-All | 58.8 ± 0.2 | 91.9 ± 0.1 | 93.3 ± 0.1 | 38.1 ± 0.5 | 58.1 ± 0.4 | 51.5 ± 0.5 | 65.3 ± 0.1 | 1.03 ± 0.07 |
| Gaze-Subset | 61.1 ± 0.3 | 92.0 ± 0.1 | 94.8 ± 0.1 | 45.0 ± 0.3 | 57.8 ± 0.1 | 53.1 ± 0.7 | 67.3 ± 0.1 | 0.99 ± 0.07 |
| GGR-RL | 64.0 ± 0.7 | 91.9 ± 0.3 | 95.9 ± 0.1 | 50.7 ± 0.5 | 59.6 ± 0.2 | 50.8 ± 0.6 | 68.8 ± 0.2 | 1.15 ± 0.07 |
Table A7. Benchmark-level comparison of gaze-supervision strategies. Conditions, metrics, and bolding follow Table 1. Overall is the unweighted mean of the six benchmarks.
GGR ablations. Table A8 compares GGR-SFT with ablated GGR supervision.
| Condition | ChartQA Pro | ChartQA | Count BenchQA | CountQA | MME RealWorld Lite | Visual Probe | Overall | Gaze Alignment [bits/fix] |
| GGR-SFT | 58.4 ± 0.5 | 90.8 ± 0.0 | 95.0 ± 0.4 | 41.8 ± 0.3 | 57.7 ± 0.1 | 48.0 ± 0.7 | 65.3 ± 0.2 | 0.84 ± 0.06 |
| Coordinate-only | 34.79 | 84.68 | 93.28 | 40.31 | 48.83 | 49.51 | 58.57 | 0.7784 ± 0.0542 |
| Reasoning-only | 55.69 | 89.92 | 93.48 | 32.46 | 54.98 | 44.70 | 61.87 | 0.0018 ± 0.0019 |
| Shuffled | 56.9 | 89.6 | 95.1 | 38.4 | 55.1 | 45.2 | 63.4 | 0.6799 ± 0.0513 |
Table A8. GGR-SFT ablations. Task accuracy (%) and human-gaze alignment. Overall is the unweighted mean of all six benchmarks.
SFT with additional base models. Table A9 and Table A10 report SFT task performance and human-gaze alignment for Gemma 3 4B and MiMo-VL-7B under the conditions in Table 1.
| Condition | ChartQA Pro | ChartQA | Count BenchQA | CountQA | MME RealWorld Lite | Visual Probe | Overall | Gaze Alignment [bits/fix] |
| Base model | ||||||||
| Gemma 3 4B + CoT | ||||||||
| Human-fixation SFT | ||||||||
| Gaze-All | ||||||||
| Gaze-Subset | ||||||||
| GGR-SFT |
Table A9. SFT task performance and gaze alignment with Gemma 3 4B. Condition names follow Table 1. Task scores report Avg@3 ( SEM). We bold SFT task means within 0.5 percentage points of the best SFT mean in each column, and the best available SFT gaze-alignment mean.
| Condition | ChartQA Pro | ChartQA | Count BenchQA | CountQA | MME RealWorld Lite | Visual Probe | Overall | Gaze Alignment [bits/fix] |
| Base model | ||||||||
| MiMo-VL-7B + CoT | ||||||||
| Human-fixation SFT | ||||||||
| Raw-Gaze-SFT | ||||||||
| Gaze-All | ||||||||
| Gaze-Subset | ||||||||
| GGR-SFT |
Table A10. SFT task performance and gaze alignment with MiMo-VL-7B. Condition names follow Table 1. Task scores report Avg@3 ( SEM). We bold SFT task means within 0.5 percentage points of the best SFT mean in each column, and the best SFT gaze-alignment mean.
Point-budget analysis. Figure A7 reports GGR-RL, No-Gaze RL, and Point-Grounded RL at the point budgets used in Figure 4b–Figure 4c.
Figure A7. Additional point-budget results. (a) Alignment and (b) coverage for GGR-RL, No-Gaze RL, and Point-Grounded RL using up to the first unique output points. Metrics and axis limits match Figure 4b–Figure 4c. Reported is mean SEM.
Additional Trial Examples
Figure A8. Human gaze and model grounding traces from Point-Grounded RL and GGR on additional held-out examples across our three datasets.
Examples showing internal attention tracking grounded reasoning. We extracted decoder-to-image attention while GGR-RL predicted each coordinate pair, averaging across all attention heads and tokens in the pair at a fixed layer (21 for counting and 24 for the chart). We mapped these weights onto image patches, normalized each map by its maximum, and overlaid it on the original image. Black circles mark the emitted coordinates (Figure A9).
Figure A9. Examples showing internal attention tracking grounded reasoning in GGR-RL. Internal attention (red) and emitted coordinates (black circles).
References
- [1]Dana H. Ballard, Mary M. Hayhoe, and Jeff B. Pelz. Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7(1):66–80, 1995. doi: 10.1162/jocn.1995.7.1.66. URL https://doi.org/10.1162/jocn.1995.7.1.66.
- [2]Dana H. Ballard, Mary M. Hayhoe, Polly K. Pook, and Rajesh P. N. Rao. Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20(4):723–742, 1997. doi: 10.1017/S0140525X97001611.DOI
- [3]Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K. Eckstein, Noémi Éltető, Thomas L. Griffiths, Susanne Haridi, Akshay K. Jagadish, Li Ji-An, Alexander Kipnis, Sreejan Kumar, Tobias Ludwig, Marvin Mathony, Marcelo Mattar, Alireza Modirshanechi, Surabhi S. Nath, Joshua C. Peterson, Milena Rmus, Evan M. Russek, Tankred Saanum, Johannes A. Schubert, Luca M. Schulze Buschoff, Nishad Singhi, Xin Sui, Mirko Thalmann, Fabian J. Theis, Vuong Truong, Vishaal Udandarao, Konstantinos Voudouris, Robert Wilson, Kristin Witte, Shuchen Wu, Dirk U. Wulff, Haodong Xiong, and Eric Schulz. A foundation model to predict and capture human cognition. Nature, 644(8078):1002–1009, August 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09215-4. URL https://doi.org/10.1038/s41586-025-09215-4.
- [4]Nicholas Budny, Kia Ghods, Declan Campbell, Raja Marjieh, Amogh Joshi, Sreejan Kumar, Jonathan D. Cohen, Taylor W. Webb, and Thomas L. Griffiths. Visual serial processing deficits explain divergences in human and vlm reasoning, 2026. URL https://arxiv.org/abs/2509.25142.
- [5]Declan Campbell, Sunayana Rane, Tyler Giallanzа, Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas L Griffiths, Jonathan D Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37:113436–113460, 2024.arxiv.org/abs/2411.00238
- [6]Meng Cao, Haoze Zhao, Can Zhang, Xiaojun Chang, Ian Reid, and Xiaodan Liang. Ground-r1: Incentivizing grounded visual reasoning via reinforcement learning. arXiv preprint arXiv:2505.20272, 2025.arxiv.org/abs/2505.20272
- [7]Patricia A. Carpenter and Priti Shah. A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology: Applied, 4(2):75–100, 1998. doi: 10.1037/1076-898X.4.2.75.DOI
- [8]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, Vincent Shao, Yue Yang, Weikai Huang, Ziqi Gao, Taira Anderson, Jianrui Zhang, Jitesh Jain, George Stoica, Winson Han, Ali Farhadi, and Ranjay Krishna. Molmo2: Open weights and data for vision-language models with video understanding and grounding, 2026. URL https://arxiv.org/abs/2601.10611.
- [9]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.arxiv.org/abs/2110.14168
- [10]Filipe Cristino, Sebastiaan Mathôt, Jan Theeuwes, and Iain D. Gilchrist. ScanMatch: A novel method for comparing fixation sequences. Behavior Research Methods, 42(3):692–700, 2010.DOI
- [11]Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 91–104. IEEE, 2025.
- [12]Thomas Fel, Ivan F. Rodriguez Rodriguez, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. In Advances in Neural Information Processing Systems, volume 35, pages 9432–9446. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/3d681cc4487b97c08e5aa67224dd74f2-Abstract-Conference.html.arxiv.org/abs/2211.04533
- [13]Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https://arxiv.org/abs/2503.01307.
- [14]Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In Advances in Neural Information Processing Systems, volume 33, pages 13890–13902, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/9f6992966d4c363ea0162a056cb45fe5-Abstract.html.arxiv.org/abs/2006.16736
- [15]Elizabeth R Grant and Michael J Spivey. Eye movements and problem solving: Guiding attention guides thought. Psychological science, 14(5):462–466, 2003.
- [16]Udith Haputhanthri, Declan Campbell, Rim Assouel, Jonathan D. Cohen, and Taylor W. Webb. Binding visual features point by point, 2026. URL https://arxiv.org/abs/2605.25427.
- [17]Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in Cognitive Sciences, 9(4):188–194, 2005. doi: 10.1016/j.tics.2005.02.009. URL https://www.sciencedirect.com/science/article/abs/pii/S1364661305000598.DOI
- [18]Marcel Adam Just and Patricia A. Carpenter. Eye fixations and cognitive processes. Cognitive Psychology, 8(4):441–480, 1976. doi: 10.1016/0010-0285(76)90015-3. URL https://www.sciencedirect.com/science/article/pii/0010028576900153.DOI
- [19]Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Ulas Bagci, and Alessandro Bruno. Matched outcomes, divergent gaze: How foveated mllms search compared to humans, 2026. URL https://arxiv.org/abs/2608.16514.
- [21]Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning patterns and interaction turns for visual search, 2025. URL https://arxiv.org/abs/2509.07969.
- [22]Michael F. Land and Mary Hayhoe. In what ways do eye movements contribute to everyday activities? Vision Research, 41(25–26):3559–3565, 2001. doi: 10.1016/S0042-6989(01)00102-X.DOI
- [23]Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang, Yue Yang, Tao Wang, Dominic C Marshall, Yingying Fang, and Guang Yang. Seeing through experts’ eyes: a foundational vision-language model trained on radiologists’ gaze and reasoning. npj Artificial Intelligence, 2026.
- [24]Yiwei Li, Zihao Wu, Yanjun Lv, Hanqi Jiang, Weihang You, Zhengliang Liu, Dajiang Zhu, Xiang Li, Quanzheng Li, Tianming Liu, et al. Thinking with gaze: Sequential eye-tracking as visual reasoning supervision for medical vlms. arXiv preprint arXiv:2603.06697, 2026.arxiv.org/abs/2603.06697
- [25]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.arxiv.org/abs/2305.20050
- [26]Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/forum?id=BJgLJg3R9KQ.arxiv.org/abs/1805.08819
- [28]Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022. URL https://arxiv.org/abs/2203.10244.
- [29]Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadiku Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartqapro: A more diverse and challenging benchmark for chart question answering, 2025. URL https://arxiv.org/abs/2504.05506.
- [30]Sounak Mondal, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. Gaze-to-text generation: Beyond categorical decoding of human attention. In European Conference on Computer Vision, pages 411–430. Springer, 2026.arxiv.org/abs/2607.23917
- [31]Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael C Mozer, Klaus-Robert Müller, Thomas Unterthiner, and Andrew K Lampinen. Aligning machine and human visual representations across abstraction levels. Nature, 647(8089):349–355, 2025.DOI
- [32]Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten, 2023. URL https://arxiv.org/abs/2302.12066.
- [33]Tingting Qiao, Jianfeng Dong, and Duanqing Xu. Exploring human-like attention supervision in visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.arxiv.org/abs/1709.06308
- [34]Qwen Team. Owen3.5: Towards native multimodal agents. February 2026. URL https://qwen.ai/blog?id=qwen3.5.
- [35]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518. PMLR, 2023. URL https://proceedings.mlr.press/v202/radford23a.html.arxiv.org/abs/2212.04356
- [36]Ali Salamatian, Amirhossein Abaskohi, Wan-Cyuan Fan, Mir Rayat Imtiaz Hossain, Leonid Sigal, and Giuseppe Carenini. Chartgaze: Enhancing chart understanding in lvms with eye-tracking guided attention refinement. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12104–12124, 2025.arxiv.org/abs/2509.13282
- [37]Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum. Efficiently guiding imitation learning agents with human gaze. arXiv preprint arXiv:2002.12500, 2020.arxiv.org/abs/2002.12500
- [38]Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, and Zhuang Liu. Vero: An open rl recipe for general visual reasoning. arXiv preprint arXiv:2604.04917, 2026a. URL https://arxiv.org/abs/2604.04917.
- [39]Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning, 2026b. URL https://arxiv.org/abs/2505.23678.
- [40]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025.arxiv.org/abs/2601.03267
- [41]Ekta Sood, Fabian Kögel, Florian Strohm, Prajit Dhar, and Andreas Bulling. Vqa-mhug: A gaze dataset to study multimodal neural attention in visual question answering. In Proceedings of the 25th Conference on Computational Natural Language Learning, page 27–43. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.conll-1.3. URL http://dx.doi.org/10.18653/v1/2021.conll-1.3.
- [42]Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, et al. Getting aligned on representational alignment. Transactions on Machine Learning Research, 2025. URL https://openreview.net/forum?id=Hiq7lUh4Yn.
- [43]Jayant Sravan Tamarapalli, Rynaa Grover, Nilay Pande, and Sahiti Yerramilli. Countqa: How well do mlms count in the wild?, 2025. URL https://arxiv.org/abs/2508.06585.
- [44]Chaitanya Thammineni, Hemanth Manjunatha, and Ehsan T Esfahani. Selective eye-gaze augmentation to enhance imitation learning in atari games. Neural Computing and Applications, 35(32):23401–23410, 2023.
- [45]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.arxiv.org/abs/2307.09288
- [46]Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning, 2025. URL https://arxiv.org/abs/2505.15966.
- [47]Niklas Wilming, Torsten Betz, Tim C. Kietzmann, and Peter König. Measures and limits of models of fixation selection. PLoS ONE, 6(9):e24038, 2011. doi: 10.1371/journal.pone.0024038.DOI
- [48]Zhongxing Xu, Chengzhi Liu, Qingyue Wei, Juncheng Wu, James Zou, Xin Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models. Advances in Neural Information Processing Systems, 38:82878–82905, 2026.
- [49]Alfred L. Yarbus. Eye Movements and Vision. Plenum Press, New York, 1967.DOI
- [50]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.arxiv.org/abs/2203.14465
- [51]Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. In Findings of NAACL, 2025a.
- [52]Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?, 2025b. URL https://arxiv.org/abs/2408.13257.
- [53]Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining, 2025. URL https://arxiv.org/abs/2504.07912.
- [54]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071.
- [55]Chunzheng Zhu, Yangfang Lin, Shen Chen, Yijun Wang, and Jianxin Lin. Medeyes: Learning dynamic visual focus for medical progressive diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 13916–13924, 2026.







