betaFeedback

WorksComputer Science

FoAViS: Analyzing Learned Fixation Strategies in a Foveated Active Vision System

Abstract

Human vision is foveated and active: resolution is highest at the fovea and falls off towards the periphery, and fixations are actively directed to locations of interest. Building a machine system that processes information in the same way lets us study gaze behavior in a controlled setting where the system's components and objectives are known. The emergence of gaze characteristics can therefore be examined systematically, which may in turn inform our understanding of biological vision. This requires a model that can sample like the fovea, handle various tasks from classification to dense prediction, and be trained concurrently with a fixation policy. In our study, we present such a system and study its capabilities and the fixation policies that emerge from it. Our system, FoAViS, samples visual input by adopting the established log-polar foveation and accumulates information in a 2D spatial memory. Due to its 2D structure, the same memory can be read out globally for classification or densely for semantic segmentation, without any architectural change. Fixations are drawn from a 2D map by a policy trained concurrently with the task. We find that learned fixation policies select more informative, task-specific locations than non-learned heuristics. These policies also shape the task representations in a beneficial way: for both classification and segmentation, models trained concurrently with a policy outperform models trained with random locations, even when the latter are provided with learned scanpaths. Moreover, these scanpaths exhibit similarities to human gaze behavior by capturing aspects of empirical scanpath characteristics. Our work marks an important step away from standard passive, single-pass architectures, towards biologically grounded models of vision. It opens a path for studying goal-directed gaze behavior and information representation in the visual system.

Introduction

Traditionally, artificial neural networks that receive a static image as input have been used to gain insights into biological vision: networks trained on tasks such as object recognition develop representations that align with computations along the ventral stream ([108]; [66]; [14]), and such correspondences emerge even in self-supervised models never optimized for biological fidelity ([75]). While these models have proven valuable, they remain structurally mismatched to the system they aim to explain: they process the whole image at once, passively, often in a single feedforward pass (but see [51]), typically with large monolithic architectures that neglect the hierarchical and modular organization of the visual system (but see [5]; [21]) as well as top-down feedback, and cross-area coordination ([91, 76]). In contrast, biological vision is not a passive camera, but an active exploratory process. Because the visual field pairs a high-resolution foveal bottleneck with a heavily undersampled periphery ([84]), the visual system must deploy rapid saccades sequentially towards informative regions. It is therefore forced to solve two coupled problems in tandem: identifying what is present at the current fixation to update an ongoing spatiotopic representation of the scene ([36]), and using low-resolution peripheral cues to decide where to look next. This way of perceiving the world gives rise to characteristic fixation patterns ([109, 54, 41]) and neural representations ([50, 33, 73]). For an artificial model, learning behavioral and representational strategies that resemble those of biological vision is likely facilitated by analogous inductive biases: an eccentricity-dependent sampling strategy, a spatiotopic memory capable of integrating sequential glimpses over time, and an eye movement policy trained across diverse visual tasks.

While active vision has been explored in computational research, existing models diverge considerably across several core design axes (Table 1). At the sensory level, many architectures prioritize computational efficiency over biological plausibility: they typically rely on rectangular crops ([71, 7, 103]) or multi-scale patch pyramids ([90 Pourrahimi &] (Seif & Tyutelaars 2020) Bashivan, 2025 rather than preserving the eccentricity-dependent resolution falloff of human vision ([59, 35]). Furthermore, the memory component of previous works often flattens or averages away spatial structure ([62, 1, 68, 103]). While some models incorporate an explicit spatial memory, they remain constrained by low spatial resolution ([103]) or limited representational capacity ([58]), restricting their scope to spatially coarse tasks like image classification. A high-capacity memory that stays coordinate-aligned across glimpses would preserve scene topography and lend itself naturally to a broad range of tasks such as classification, segmentation or visual search. Recent concurrent work moves in this direction by integrating a high-resolution spatial memory into an active vision transformer (Berreby et al., [7]). However, this model is trained using only random locations rather than a learned, task-driven policy, and has no support for eccentricity-dependent acuity. In general, we are not aware of any active vision model combining biologically motivated foveation, an explicit spatiotopic memory, and the ability to learn both image-level and dense tasks concurrently with a fixation policy. Such a system would provide a natural testbed for studying how characteristic fixation strategies and representations emerge under different task constraints.

ModelBiologically Motivated FoveationExplicit 2D Spatial MemoryTrainable Fixation PolicyGlobal Tasks (e.g., Cls.)Dense Tasks (e.g., Seg.)Concurrent Policy and Task Training
RAM (Mnih et al., [62])∼\sim (Patch Pyramid)×\times (1D Recurrent)✓✓×\times✓
DRAM (Ba et al., [1])∼\sim (Patch Pyramid)×\times (1D Recurrent)✓✓×\times✓
Cheung et al. [20]✓ (Learned Sampling)×\times (1D Recurrent)✓✓×\times✓
Ren & Zemel [79]×\times (Rect. Crops)∼\sim (Prediction-Masks Memory)✓✓✓✓
Saccader (Elsayed et al., [27])×\times (Feature Sampling)×\times (Mean Pooling)✓✓×\times✓
Attend-and-Segment (Seifi & Tuytelaars, [89])∼\sim (Patch Pyramid)✓ (Multi-Level Memory)×\times (Uncertainty Map)×\times✓×\times (No Policy)
GFNet (Wang et al., [102])×\times (Rect. Crops)×\times (1D Recurrent)✓✓×\times×\times (Multi-Stage)
GAE (Seifi et al., [90])∼\sim (Patch Pyramid)✓ (Multi-Level Memory)×\times (Attention Map)✓✓×\times (No Policy)
TNet (Papadopoulos et al., [69])×\times (Rect. Crops)×\times (Mean Pooling)✓✓×\times✓
Lukanov et al. [59]✓ (Log-Polar Warp)×\times (None)×\times (Feature Saliency)✓×\times×\times (No Policy)
FoveaTer (Jonnalagadda et al., [45])×\times (Feature Sampling)∼\sim (Token Bank)×\times (Attention Map)✓×\times×\times (No Policy)
STAM (Rangrej et al., [74])×\times (Rect. Crops)∼\sim (Token Bank)✓✓×\times✓
NeVA (Schwinn et al., 2022)∼\sim (Gaussian Blending)∼\sim (Pixel Memory)✓∼\sim×\times×\times (Frozen Task Model)
Li et al. [57]∼\sim (Exponential Mask)✓ (Conv. Recurrent)✓✓×\times✓
SimGlim (Jha et al., [44])×\times (Rect. Crops)∼\sim (Token Bank)×\times (Recon. Error Map)×\times∼\sim (Reconstruction)×\times (No Policy)
AME (Pardyl et al., [70])∼\sim (Patch Pyramid)∼\sim (Token Bank)×\times (Entropy)✓✓×\times (No Policy)
Killick et al. [47]✓ (Retina-Like Sampling)×\times (None)✓✓×\times✓
Liu et al. [58]∼\sim (Two-Level Sampling)∼\sim (Low Capacity)✓✓×\times×\times (Multi-Stage)
AdaGlimpse (Pardyl et al., [71])×\times (Rect. Crops)∼\sim (Token Bank)✓✓✓∼\sim (Alternating)
TORE (Olszewski et al., [68])×\times (Rect. Crops)∼\sim (Token Bank)×\times (Entropy)✓∼\sim (Reconstruction)×\times (No Policy)
GVSM (Pourrahimi & Bashivan, [73])∼\sim (Patch Pyramid)×\times (1D Recurrent)✓✓×\times×\times (Multi-Stage)
AdaptiveNN (Wang et al., [103])×\times (Rect. Crops)∼\sim (Low Res.)✓✓×\times✓
CanViT (Berreby et al., [7])×\times (Rect. Crops)✓ (High-Res Latent)×\times (Heuristic, Entropy)✓×\times×\times (No Policy)
FoAViS (ours)✓ (Log-Polar Sampling)✓ (High-Res LSTM)✓ (2D map + RL)✓✓✓

Table 1. Comparison of active vision architectures across core design axes. Entries are ordered chronologically by publication year. Columns indicate whether a requirement is fully supported (✓), partially supported (∼), or not supported (×). Entries reflect each model as described and evaluated in its original publication. Some models could plausibly be extended to meet further requirements, but this was not explored in the respective work. FoAViS is discussed in more detail in § 3.

In our work, we introduce FoAViS, a Foveated Active Vision System designed around these core principles. Visual input is sampled in a log-polar pattern, adopting the radial undersampling of biological vision. Information from multiple such glimpses is integrated over time using an allocentric, spatially aligned 2D memory, which can be seen as a computational analogue to spatiotopic updating (Golomb & Mazer, [36]). The 2D memory is compatible with standard task heads, so it supports both global, image-level tasks like classification and dense, pixel-level tasks like semantic segmentation without any architectural change. Furthermore, our architecture incorporates a fixation policy that is trained concurrently with the task.

We find that FoAViS learns to direct fixations towards informative image regions (Figure 1A) and develops distinct fixation patterns across different tasks and datasets (Figure 1B). In addition, the policy exhibits inhibition of return (IoR) (Itti et al., [41]), suppressing re-fixations of recently visited locations. These specialized gaze behaviors emerge naturally from downstream task optimization as solutions under biologically inspired active vision constraints. They also translate directly into performance gains: concurrently optimizing the visual representation and the fixation policy consistently outperforms architectures trained on random locations, even when the latter are evaluated using the learned policy at test time (Figure 1C). This underscores the importance of co-adapting the visual representation and fixation strategy.

Figure 1. Emergent fixation strategies and model performance under concurrent policy and task training. (A) Example scanpaths directing fixations towards informative image areas. (B) Cumulative distribution over fixations two to four (excluding the initial center fixation) across 10k (C-MNIST and ImageNet), 5k (COCO-Stuff) and 2k (ADE20K) images, showing task- and dataset-dependent spatial priors. The colorbar indicates likelihood relative to a uniform baseline, with values reflecting the factor by which probability exceeds chance. The distributions are consistent across models trained with different random seeds (Appendix § 16.§ 2). (C) Downstream model performance comparing models trained concurrently with their policy and evaluated with it against models trained on random locations, evaluated either with random locations or via transfer of the learned policy.

Related work

Active vision is about controlling a sensor to gather relevant information about a scene in an efficient way [4]. While the scene of interest could in general be a video or a real-world setting, we focus on the simplest case in our work: a single fixed image that is viewed through a sequence of fixations, a paradigm long used to study human vision [12, 109]. By introducing Recurrent Models of Visual Attention (RAM), Mnih et al. [62] were the first to provide a systematic deep learning framework for this paradigm, drawing on earlier works that processed visual scenes sequentially [83, 41, 30, 55, 26]. Notably, this first deep learning framework already laid out the full set of components that more modern active vision models are still composed of:

  • a foveation module that turns a chosen location into a glimpse,

  • a backbone that accepts that glimpse as input,

  • a memory module that accumulates information across glimpses,

  • a fixation policy for choosing where to look next, and

  • a task module that turns the memory into a task-specific output, e. g. a classification.

Subsequent work has explored different choices for these components, for better task performance, lower compute, or closer correspondence to biological vision. Not all of these components carry equal weight as design decisions. The backbone and task module are usually kept simple and reflect whatever was standard at the time rather than being tailored to the active vision setting. The choices specific to active vision, and the ones that most determine what a model is capable of doing, are the foveation module, the memory module, and the fixation policy. In § 2.1 to § 2.3, we examine prior work along these three axes. In § 2.4, we position FoAViS within the broader landscape of gaze models and highlight its value for providing a normative perspective.

Foveation module

Models differ considerably in how a fixation is turned into network input. While some formulations stay close to the geometry of biological foveation, others depart from it substantially. Among the latter, one line of active vision models feeds rectangular crops directly to the backbone, with no foveation applied within a glimpse. To be able to explore larger scenes with this formulation, crops cover different extents of the image but are always resized to the same number of pixels: the first glimpse is typically the whole image at low resolution, giving a coarse overview, with later crops smaller and finer [102, 69, 71, 103, 7]. Any foveal character thus comes from the changing crop size across glimpses, not the sampling geometry of a single glimpse.

Another widespread choice is the multi-scale patch pyramid, following RAM [62]: several crops of different real extents are taken around the fixation point, resized to a common size, and passed to the backbone jointly [62, 89, 90, 70, 73]. Each glimpse then carries both a fine view of the fixated point and a coarser view around it, but the falloff is discrete rather than the smooth space-variant compression of the retina.

Another strategy avoids sampling the input image altogether: the backbone first processes the full, non-foveated image, and the foveation is applied to the resulting feature map [45, 27]. For example, FoveaTer [45] passes the whole image through a shallow convolutional backbone, giving a 14×1414 \times14 grid of feature vectors, each summarizing a local neighborhood of the image. For each fixation, this map is then pooled around the corresponding grid position: the vectors immediately around it are kept as they are, while those further away are averaged together in increasingly large blocks. Averaging discards the detail, so resolution falls off with distance from the fixation without the input image itself ever being foveated. This is in contrast to biological vision, where foveation acts on the input itself.

Closer to biological foveation are methods that sample the image on a space-variant grid. In primates, retinal eccentricity maps approximately logarithmically onto primary visual cortex, so that central acuity is preserved while the periphery is progressively compressed [84, 85]. The log-polar mapping formalizes this as an image representation ([104, 81, 11]. The image is sampled along concentric rings whose spacing grows with eccentricity, yielding a regular grid whose axes are angle and log-eccentricity. Lukanov et al. [59] adopt such a mapping in an active vision model. However, two caveats come with this mapping: (1) It is not translation-equivariant, so convolutions applied afterwards are not translation-equivariant in image space either (instead, they are equivariant to rotation and scaling around the center of fixation). (2) Because the samples are mapped onto a rectangular grid, their radial and angular spacing differ. The sampling is therefore locally anisotropic, and the degree of anisotropy varies with eccentricity.

Two recent methods aim to resolve both caveats while keeping the eccentricity-dependent sampling. Killick et al. [47] sample on an irregular point set, denser at the center and sparser towards the periphery but locally close to isotropic throughout. The samples are not remapped onto a radial-angular grid but processed directly with a graph convolution, so translation equivariance is preserved. Blauch et al. [9] use a sampling pattern that directly implements the retino-cortical mapping of Schwartz [85], which is locally isotropic by construction. Here too the samples form no grid, and features are computed with kk-nearest-neighbor convolutions. They provide a complete framework for combining this foveation with convolutional and transformer backbones, though it has not yet been demonstrated in an active vision system.

All of the sampling patterns discussed so far are hand-crafted beforehand. In contrast, Cheung et al. [20] let the model learn how to sample the image under a constrained budget. They discover that classic biological foveation, with high resolution in the center and lower resolution towards the periphery, emerges entirely on its own, provided the model can only shift its gaze across the image rather than zoom in and out.

Despite the mentioned drawbacks, we adopt the log-polar sampling scheme (see § 3.1), following Lukanov et al. [59]. It is space-variant like the retina, partially invertible, and produces a rectangular grid that can be processed by off-the-shelf convolutional backbones without modification. The framework of Blauch et al. [9] offers a promising alternative, but emerged concurrently with our work.

Memory module

Active vision models differ in how their memory accumulates information across glimpses and how much spatial structure it preserves. In the simplest case, per-glimpse features or predictions are pooled into a single vector with no spatial layout ([69, 27]. The pooling operation does not depend on what has already been seen, so earlier glimpses cannot change how later ones accumulate information. The memory therefore only has limited capacity to capture relationships between different glimpses.

In contrast, there is a line of models that replace the pooling step with a learned recurrent update, while still using a state vector without spatial layout. This is the setting of early active vision models ([62, 1]) and of sequential routing architectures ([102]). The recent GVSM (Pourahmadi & Bashivan, 2025) also follows this design, integrating visual information across sequential fixations through a dynamic internal state, implemented either as an LSTM or a causal transformer.

Another line of research keeps no fixed-size memory at all but instead accumulates a growing collection of per-glimpse entries. STAM ([74]), AME ([70]), SimGlim ([44]) and AdaGlimpse ([71]) accumulate raw image patches and re-encode the entire growing sequence through the full transformer from scratch at each step. In contrast, TORE ([68]) splits the transformer into an extractor and an aggregator, processing each glimpse through the extractor once to cache its intermediate midway tokens, which the aggregator then reuses as new glimpses arrive.

A fourth group keeps a fixed-size 2D memory, preserving spatial layout but storing output or image content rather than semantic features. For example, Ren & Zemel [79] accumulate predicted instance masks in a 2D ConvLSTM memory and pass them as an additional input channel in subsequent timesteps to provide top-down feedback. Other models iteratively uncover the input image itself ((Schwinn et al., 2022), so the spatial memory holds raw pixel values that serve as input to a downstream task such as classification.

Closest to our setting are models whose memory stores semantic features. Attend-and-Segment ([89]) and its successor ([90]) extract multi-scale feature maps from each glimpse and write them deterministically into a persistent allocentric 2D spatial memory at coordinates corresponding to the glimpse’s position in the image. Instead of deterministic coordinate-based writes, Liu et al. [58] perform dynamic memory updates using a shallow ConvLSTM as a joint encoder and recurrent state. Yet, its constrained channel capacity restricts the expressive power of the stored representations. Similarly, AdaptiveNN (Wang et al. [103]) maintains a dynamically updated, spatially organized state, but operates only at a low spatial resolution tailored for non-spatial, image-level tasks. CanViT (Berreby et al. [7]) instead maintains a high-resolution latent memory (termed a canvas in their work) that is updated by cross-attention with the current glimpse. In combination with distilling DINOv3 (Siméoni et al. [91]) features, this memory formulation has been shown to successfully integrate fine-grained semantic information over multiple glimpses to perform tasks like semantic segmentation.

Our memory (see § 3.3) also employs a global 2D spatial layout that accumulates semantically rich features at high resolution. Before updating the memory, incoming foveal features are mapped to the image reference frame to maintain precise topographic alignment across glimpses. Memory updates are then implemented as a per-position LSTM with shared weights, enabling step-by-step visualization and analysis of the gating dynamics. Because the memory preserves deep features at a fixed spatial resolution, it can be read out directly for both image-level classification and dense semantic segmentation without architectural modifications.

Fixation policy

Existing active vision models generally determine where to look next through one of three paradigms: non-learned selection rules derived from network activations, end-to-end differentiable coordinate prediction, or stochastic policies that choose the next fixation. Both learned paradigms are forms of reinforcement learning, but they differ in how they are trained.

In non-learned selection, fixation targets are derived directly from internal activation maps or spatial heuristics calculated during the primary task’s forward pass. By simply reusing existing feature representations derived from previous visual input, these approaches bypass the need for a dedicated policy network. Prior work has extracted non-learned fixation maps from averaged backbone feature activations (Lukanov et al. [59]), accumulated self-attention weights (Seifi et al. [90]; Jonnalagadda et al. [45]), spatial uncertainty maps (Seifi & Tuyltaars, [89]), or the entropy of an internal representation (Pardyl et al., [70]; Olszewski et al. [68]; Berreby et al. [7]). Because selection is decoupled from training, models can even alter policies between training and inference: For example, CanViT (Berreby et al., [7]) trains on random locations but switches to an entropy-guided policy at inference to increase task performance. Because such heuristics rely on representation byproducts rather than task optimization, they may result in suboptimal fixation sequences.

This limitation can be addressed by formulating fixation selection as an end-to-end differentiable pipeline. In this framework, the model predicts continuous coordinates and extracts visual features through soft interpolation kernels (Jaderberg et al., [42]) centered at the predicted locations (Ren & Zemel, [79]; Schwinn et al. (2022); Li et al. [57]; Killick et al. [47]). For instance, Schwinn et al. (2022) regress the next fixation location and center a Gaussian mask around it that smoothly blends the high-resolution image with a low-resolution version, assigning decreasing weight towards the periphery based on the mask’s values. Because small shifts in predicted location continuously alter the extracted visual features, error gradients flow directly back from the downstream task loss to optimize the location predictor. In recent literature, however, these continuous formulations are less commonly adopted, as fixation selection via stochastic policies has been empirically found to be easier to optimize.

In these approaches, past observations are mapped to a spatial probability distribution from which the next fixation location is drawn. Although differentiable formulations exist for both the spatial cropping (Jaderberg et al., [42]) and the stochastic selection of gaze targets (Jang et al., [43]; Maddison et al. [60]; Cordonnier et al. [23]), these models are typically optimized using policy gradient methods, which associate each selected fixation with a scalar reward based on downstream task performance. Early foundational architectures like RAM (Mnih et al., [62]) and DRAM (Ba et al., [1]) established this formulation by predicting continuous Cartesian coordinates and drawing fixations from 2D Gaussian distributions centered on them, using policy gradients with a baseline network to reduce variance. More recent continuous approaches have found that active vision models benefit from receiving coarse global information before attending to details: AdaGlimpse ([71]) uses Soft Actor-Critic to dynamically adjust both the location and the scale of each glimpse, enabling the agent to learn an emergent strategy of starting with an overview before zooming in on details. AdaptiveNN ([103]) builds this in explicitly by supplying an initial low-resolution version of the full image. In contrast to continuous coordinate approaches, discretized formulations simplify spatial search by evaluating candidate locations over fixed grids: Saccader ([27]) restricts fixations to a pre-segmented grid of candidate patches, and similarly, GVSM ([73]) discretizes the scene into a 10×1010 \times10 grid to output a 100-dimensional categorical distribution via spatial softmax.

Following this discretized approach, we similarly draw fixation locations from a spatial probability map rather than regressing continuous coordinates (see § 3.4), which gives the model a natural mechanism for expressing uncertainty about where to look next. To optimize this fixation policy, we adopt the standard policy gradient formulation.

Mechanistic, data-driven, and normative accounts of gaze behavior

Human saccadic trajectories are highly task-dependent, a phenomenon established by Yarbus [109] and corroborated quantitatively across subsequent studies ([10]). Existing computational models of this behavior largely fall into three paradigms. First, mechanistic models formulate interpretable saccade-selection dynamics, progressing from early architectures based on saliency, winner-take-all, and inhibition of return ([41]) to activation-inhibition dynamical systems like SceneWalk ([28, 87]).

Second, data-driven frameworks directly fit image-to-gaze mappings ([52]), moving from free-viewing models like DeepGaze III ([53]) to task-conditioned successors such as ScanDiff ([17]). Bridging these two paradigms, recent hybrid approaches in free viewing use deep networks as discovery tools to identify what mechanistic models are missing ([24]).

Third, a normative tradition asks neither how gaze is produced mechanistically nor how it can be empirically predicted, but what an observer should look at under a given task and visual constraints ([64, 13, 33, 93, 99, 37, 40]). In this framework, fixations follow directly from an explicit objective, such as maximizing information gain or task reward. For example, the task of the Bayesian ideal searcher ([64]) is to locate a known target in a noisy background through successive views with a foveated sensor. Each fixation is then derived mathematically as the one that most reduces uncertainty about the target’s location, and humans were found to search nearly as efficiently as this optimal strategy.

While these prior works provide normative accounts in simplified settings, such as using synthetic visual stimuli or searching for a single known target, they do not scale to natural images and tasks. Deep active vision models instead learn fixation policies directly from raw pixels and a task loss, letting us study emergent visual behavior in complex scenes under controlled computational constraints. In our work, the underlying architecture remains constant across tasks, such that any divergence in the resulting fixation policy directly reflects the objective. We investigate this specialization (§ 5.2) and also show how established gaze characteristics emerge within this framing (§ 5.3). While traditional scanpath models must explicitly engineer mechanisms like IoR to suppress re-fixation ([41, 107, 19, 95]), our learned policies exhibit emergent IoR-like re-fixation avoidance ([72, 48]) without explicit reward or penalty, a behavior also exhibited by the recent GVSM ([73]).

Method

FoAViS operates as an autoregressive loop over TT steps, where a step consists of the following five components: (1) a log-polar foveation module that samples the image around a fixation point with resolution decreasing towards the periphery, producing a foveated view we refer to as a glimpse, (2) a Res-UNet backbone that extracts a feature map from this glimpse, (3) an LSTM-cell memory module applied per position that accumulates these features across glimpses in a unified image reference frame, (4) a policy head that selects the next fixation point based on the current memory features (omitted for the last step), and (5) a task head that produces a task prediction based on the memory features. Figure 2 provides an overview of the full pipeline, where the circled numbers 1–5 in green in the figure correspond to the five components listed above. In § 3.1 to § 3.5, we describe these components in more detail, followed by the training procedure in § 3.6.

Figure 2. Overview of FoAViS unrolled over three glimpses. At each step, the current fixation serves as the center for a foveation module that extracts a log-polar glimpse ① (sampling grid shown in white). The glimpse is encoded by the backbone ② (feature map visualized via PCA), backprojected to Cartesian image space, and accumulated into a 2D spatial memory using a shared per-position LSTM ③ (memory state visualized via PCA). The memory state is then read out by the policy head ④ to produce a spatial probability map (logits visualized as a heatmap) from which the next fixation is drawn (purple number), while the task head ⑤ uses it to generate the task prediction. Appendix § 14 shows these stages over full example scanpaths as visualizations.

Foveation module

The model observes the scene through a glimpse (Figure 2, ①) that is obtained by applying a log-polar foveation module. We first center the Cartesian image coordinates (x,y)(x,y) around the current fixation point ft=(ftx,fty)f_t = (f_t^x, f_t^y), writing xc,t=x−ftxx_{c,t} = x - f_t^x and yc,t=y−ftyy_{c,t} = y - f_t^y. Here and throughout the paper, tt denotes the time index of the fixation. We then map the centered coordinates to log-polar coordinates:

ρt=log⁡xc,t2+yc,t2,αt=atan2⁡(yc,t,xc,t) mod 2π,(1)\rho_t = \log\sqrt{x_{c,t}^{2} + y_{c,t}^{2}}, \qquad\alpha_t = \operatorname{atan2}(y_{c,t}, x_{c,t}) \bmod2\pi, \tag*{(1)}

where atan2 is the two-argument arctangent, so αt\alpha_t is the angle of (x,y)(x,y) around the fixation point. Finally, we sample the image on a fixed grid of RR radial ×\times AA angular points, reading off pixel values by bilinear interpolation. We use R=A=64R = A = 64, resulting in a fixed number of 64×64=409664 \times64 = 4096 sampled pixels for each glimpse. The grid is uniform in angle α\alpha over [0,2π][0, 2\pi] and uniform in log-radius ρ\rho, extending from the fixation out to a maximum radius equal to the image side length SS (all images are square), i. e. ρmax⁡=log⁡S\rho_{\max} = \log S. As a result, even a fixation at the very edge of the image still yields samples from the opposite side. Because the grid is uniform in log-radius, sample points are packed densely near the fixation point and spread out towards the periphery, encoding high resolution at the center and increasingly lower resolution towards the periphery. For quantitative characteristics of the sampling pattern, we refer to Appendix § 9.§ 2.

Backbone

The input to the backbone (Figure 2, ②) is the 64×6464 \times64 glimpse obtained as described in § 3.1, which is in log-polar coordinates. Therefore, the features it computes are also in log-polar coordinates. We adopt a Res-UNet encoder-decoder architecture (details in Appendix § 9.§ 1) whose encoder consists of four stages of two residual blocks (He et al., [38]). With each stage, the number of channels doubles (starting at 64) and the spatial resolution is halved. The decoder likewise consists of four stages, each fusing the skip connection from the corresponding encoder stage and applying one residual block, restoring the full input spatial resolution. Unlike standard U-Net decoders, which typically reduce the number of channels towards the output, our decoder maintains a channel number of 128 throughout, ensuring that the recovered high-resolution features remain semantically rich. While stronger backbones would likely yield a better performance, our focus in this work was on designing a versatile active vision method, so we kept the backbone simple.

Memory module

The memory module (Figure 2, ③) aggregates information from multiple glimpses over time in a unified image reference frame. Since the backbone operates in log-polar coordinates, its 64×6464 \times64 output obtained as described in § 3.2, must first be mapped back to the image reference frame by inverting the log-polar transformation (Cao et al., [16]). Since the memory is processed by further modules (§ 3.4 and § 3.5), we want it to be compact and to have a fixed size across varying image resolutions. We therefore do not map the backbone features to every individual image pixel. Instead, we map them onto a fixed 64×6464 \times64 grid as follows: For each grid cell, we take its center point in image coordinates, compute its log-polar coordinates relative to the current fixation (Equation (1)), and bilinearly interpolate the backbone features. Grid points that are not covered by the glimpse that the backbone receives as input (§ 3.1) are set to zero. The resulting feature map Ft\mathbf{F}_t is passed to the memory module: a standard LSTM cell applied independently at every spatial position,

Ht=LSTM⁡θ(Ht−1,Ft),H0=0,(2)\mathbf{H}_t = \operatorname{LSTM}_{\theta}(\mathbf{H}_{t-1}, \mathbf{F}_t), \qquad\mathbf{H}_0 = 0, \tag*{(2)}

where Ht\mathbf{H}_t is a 64×6464 \times64 grid of 128-dimensional hidden vectors that we refer to as the memory features (cell state omitted from notation for brevity). Here and throughout the paper, θ\theta denotes all learnable parameters of the model, and we subscript each learned module with θ\theta accordingly.

Policy head

The memory features Ht\mathbf{H}_t (§ 3.3) aggregate information in a unified image reference frame, where each part of the image is represented at a different resolution depending on how close it was to the fovea during past glimpses. They therefore encode both what is in the scene and how well each region has been resolved so far, making them the ideal basis for deciding where to look next. Specifically, we use a policy head (Figure 2, ④) that turns the memory features into a probability distribution over candidate fixation points. It consists of a lightweight convolutional projection ψθ\psi_\theta (three 128-channel convolutions and a final 1×11 \times1 convolution to a single channel; details in Appendix § 9.§ 1), followed by a softmax over all spatial positions,

πθ(⋅∣Ht)=softmax⁡(ψθ(Ht)),ft+1∼πθ(⋅∣Ht),(3)\pi_\theta(\cdot\mid\mathbf{H}_t) = \operatorname{softmax}(\psi_\theta(\mathbf{H}_t)), \qquad f_{t+1} \sim\pi_\theta(\cdot\mid\mathbf{H}_t), \tag*{(3)}

yielding a 64×6464 \times64 probability map from which the next fixation ft+1f_{t+1} is drawn. Predicting a full spatial distribution, rather than regressing a coordinate directly, gives the model a natural mechanism for expressing uncertainty about where to look next.

Task head

Analogous to the policy head, a task head ϕθ\phi_\theta (Figure 2, ⑤) produces a task prediction by reading out from the memory features (§ 3.3):

y^t=ϕθ(Ht),(4)\hat{y}_t = \phi_\theta(\mathbf{H}_t), \tag*{(4)}

where y^t\hat{y}_t is a vector or spatial map of logits for classification or segmentation, respectively. Because the memory is a standard 2D spatial feature map, it can be processed by whatever head the task requires. For classification, a convolutional projection (two 256-channel convolutions) is followed by global average pooling and a two-layer MLP. For semantic segmentation, a convolutional projection (a 128-channel convolution and a 1×11 \times1 convolution to the number of classes) maps each memory position to class logits, which are bilinearly upsampled to the input resolution. Details for both heads are given in Appendix § 9.§ 1. Task heads for other tasks such as visual search, object detection or instance segmentation would also be supported by the framework, though this is not part of the present work.

Model training

The components of § 3.1 to § 3.5 form a single computational loop that can be unrolled over multiple glimpses. We train the model using a combination of a supervised loss for the differentiable task prediction path (foveation module, backbone, memory, and task head) and a policy-gradient loss for the fixation policy. The remainder of this section defines each loss term and how they combine into the total objective.

Episode structure. An episode unrolls the model over TT steps. A step of the model consists of foveating at the current fixation (§ 3.1), encoding the resulting glimpse with the backbone (§ 3.2), updating the memory (§ 3.3), and reading out from the memory using the policy head (§ 3.4) and task head (§ 3.5). The first fixation f1f_1 is fixed as the image center. At each step t=1,…,T−1t = 1, \ldots, T - 1 the model then selects the next fixation ft+1∼πθ(⋅∣Ht)f_{t+1} \sim\pi_\theta(\cdot\mid\mathbf{H}_t) (Equation (3)) and produces a task prediction y^t\hat{y}_t, while at the final step t=Tt = T it produces only the task prediction y^T\hat{y}_T and no further fixation. The model therefore takes only T−1T - 1 actions, the last at t=T−1t = T - 1, in contrast to standard reinforcement learning notation.

Task loss. The task predictions y^t\hat{y}_t (Equation (4)) are supervised using a cross-entropy loss:

Ltask(θ)=1T∑t=1TLCE(y^t,y).(5)\mathcal{L}_{\mathrm{task}}(\theta) = \frac{1}{T} \sum_{t=1}^{T} \mathcal{L}_{\mathrm{CE}}(\hat{y}_t, y). \tag*{(5)}

For classification, this is the standard cross-entropy loss. For semantic segmentation, the cross-entropy is calculated per pixel and averaged over all spatial positions. Supervising at every timestep t=1,…,Tt = 1, \ldots, T encourages the model to provide an accurate prediction after any number of glimpses. Each of the loss terms is used to train the entire differentiable path, unrolled over multiple glimpses, via backpropagation through time. Supervising only the final step (t=Tt = T) yields comparable performance at the training horizon, but performance degrades when the model is unrolled for fewer or more steps (Appendix § 19.§ 1).

Reward. A fixation should be rewarded for the improvement it brings. We therefore define the reward of the fixation selected at step tt as the resulting reduction in the task loss:

rt=LCE(y^t,y)−LCE(y^t+1,y),t=1,…,T−1,(6)r_t = \mathcal{L}_{\mathrm{CE}}(\hat{y}_t, y) - \mathcal{L}_{\mathrm{CE}}(\hat{y}_{t+1}, y), \qquad t = 1, \ldots, T - 1, \tag*{(6)}

such that the reward grows with the size of the improvement. For simplicity, we credit each fixation with this reward directly, instead of accumulating future rewards into a discounted return as is often done in reinforcement learning. Equivalently, this can be regarded as a discounted return with a discount factor of zero. In our experiments, non-myopic return formulations generalized worse when the model is unrolled for more steps at inference time (see Appendix § 19.§ 2).

We apply the following standardization to the rewards:

r~t=rt−E[rt]σ[rt],(7)\tilde{r}_t = \frac{r_t - \mathbb{E}[r_t]}{\sigma[r_t]}, \tag*{(7)}

where E[⋅]\mathbb{E}[\cdot] and σ[⋅]\sigma[\cdot] are the empirical mean and standard deviation over the samples of the current batch at the same step. This equalizes the influence of early and late fixations, whose raw rewards differ considerably in magnitude because an early fixation reduces the task loss far more than a late one. It further prevents the reward scale from drifting as the task loss decreases over training.

Policy loss. The policy πθ(⋅∣Ht)\pi_\theta(\cdot\mid\mathbf{H}_t) of Equation (3) defines a distribution over the 64×6464 \times64 grid of candidate fixations, from which the next fixation ft+1\mathbf{f}_{t+1} is drawn. We optimize it via policy gradient Williams [105]. Intuitively, the update makes a fixation more probable when its standardized reward is positive, and less probable otherwise. Concretely, this is achieved by minimizing the following loss function,

Lpolicy(θ)=−∑t=1T−1sg⁡[r~t]log⁡πθ(ft+1∣sg⁡[Ht]),(8)\mathcal{L}_{\mathrm{policy}}(\theta) = - \sum_{t=1}^{T-1} \operatorname{sg}[\tilde{r}_t] \log\pi_\theta(\mathbf{f}_{t+1} \mid\operatorname{sg}[\mathbf{H}_t]), \tag*{(8)}

where sg⁡[⋅]\operatorname{sg}[\cdot] denotes the stop-gradient (stop-grad) operator. It is applied to r~t\tilde{r}_t so that the reward only weights the gradient rather than receiving one itself. It further is applied to Ht\mathbf{H}_t to confine the policy gradient to the policy head, leaving backbone and memory to be shaped by the task loss alone. For a motivation of why we opted for this design, we refer to § 5.4.

Total objective. The full training objective is the sum of the task and policy loss:

L(θ)=Ltask(θ)+Lpolicy(θ).(9)\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{task}}(\theta) + \mathcal{L}_{\mathrm{policy}}(\theta). \tag*{(9)}

Both terms are optimized simultaneously, but they act on disjoint sets of parameters because of the stop-grad in Equation (8). The policy loss updates only the policy head, while the backbone, memory module, and task head are trained by the task loss alone. The foveation module is parameter-free.

Experiments

Datasets

We train and evaluate on two image-classification datasets, ImageNet [25] and Cluttered MNIST (C-MNIST), and two dense semantic-segmentation datasets, ADE20K [110] and COCO-Stuff [15]. ImageNet is a large-scale natural-image recognition benchmark spanning 1000 object categories, with 1.28M training and 50 000 validation images [25]. C-MNIST is a custom 10-class digit-recognition task in which one MNIST digit [56] is placed as target on a larger canvas alongside five EMNIST letter distractors [22]. Targets come from 60 000 training and 10 000 validation digits, distractors from 124 800 training and 20 800 validation letters. Samples are generated on the fly rather than drawn from a fixed pool, so there is no fixed image count. We refer to Appendix § 10 for the exact generation procedure. Similar datasets derived from MNIST [56] have already been used in prior active vision works [18, 103]. We include C-MNIST as a controlled setting in which the discriminative signal is small and spatially localized, so that the task can only be solved by actively fixating the target. ADE20K is a scene-parsing benchmark with 150 semantic classes, comprising 20 210 training and 2000 validation images. COCO-Stuff densely labels the COCO 2017 images with 171 semantic classes (80 thing and 91 stuff), comprising 118 287 training and 5000 validation images.

Trained models

All active vision experiments use the architecture and general training recipe described in § 3 with a fixed number of T=4T = 4 glimpses. The classification models were trained from random initialization, whereas the segmentation models were finetuned from the last checkpoint of the ImageNet classification model trained on random locations due to the limited size of the segmentation datasets. All models were trained three times under an identical training procedure. Complete optimization details, hyperparameters, and dataset preprocessing for all training settings can be found in Appendix § 9.§ 3. All tasks are trained in two different settings: The learned-policy setting uses the full hybrid objective of § 3.6, minimized concurrently: the policy-gradient term supervises the policy head and the task loss supervises the task head, memory and backbone. Since both objectives are minimized concurrently, this setting allows the policy to adapt to the task features and vice versa.

The random-location setting instead draws fixations uniformly at random over the image, so the policy head is unused and the policy-gradient terms are absent during optimization. The backbone, memory, and task head are trained under the task loss. As in the learned-policy setting, the first fixation is fixed at the image center, and only the remaining T−1T - 1 fixations are drawn at random. The random-location setting was proposed by Berreby et al. [7] as a way to train a policy-agnostic active vision model. We therefore include it in our work to compare its performance to the learned-policy setting.

We also train passive baselines that consist of the Res-UNet backbone (§ 3.2) and task head (§ 3.5) of FoAViS, but have no foveation, memory, or policy. Instead, they process the whole image at 128 × 128 pixels in a single forward pass. The baselines are thus architecturally comparable to FoAViS and receive as many pixels as it does across four 64 × 64 glimpses (of which fewer are unique for FoAViS due to foveal oversampling; Appendix § 9.§ 2). They also share the training recipe of FoAViS, which includes initializing the segmentation baselines from the ImageNet baseline. Details are given in Appendix § 11.

Results and discussion

FoAViS processes visual scenes sequentially through a foveated sensory bottleneck, accumulates the resulting features into a coordinate-aligned 2D memory, and learns downstream task predictions concurrently with a fixation policy. We evaluate this framework across four benchmarks for classification and segmentation with a focus on how models perform when being supplied with their native (§ 5.1) vs. other policies (§ 5.2) at test time. We then go on to analyze the emergent scanpath properties of our models, comparing them to results typically observed in human scanpath data (§ 5.3), and conclude with an analysis of problems encountered during joint policy and task optimization (§ 5.4).

In our work, we focus on policy and task learning in a biologically inspired architecture and investigate the properties of the learned policies, instead of optimizing benchmark performance. For completeness, we provide the results of prior active vision architectures on ImageNet and ADE20K alongside our models in Appendix § 12. Direct comparability across these models is limited, however, as they differ substantially in input size, pixel budget, model size, pretraining schemes, and overall architecture.

Native policy evaluation

We first run each model with the fixation policy it was trained with – the learned-policy model with its own policy head, and the random-location model with locations drawn uniformly across the image. This provides a direct measure of performance under the training conditions of each model.

Across all four benchmarks, models trained concurrently with a fixation policy consistently outperform their counterparts trained and evaluated on random locations (Table 2). At the training budget of T=4T = 4 glimpses, learning where to look provides a clear advantage across both image-level tasks (+8.8% Acc on C-MNIST; +4.8% Acc on ImageNet) and dense tasks (+2.5 mIoU on ADE20K; +1.7 mIoU on COCO-Stuff). This result shows that these models successfully learn to focus on informative regions, as is also supported by qualitative results (Figure 1 and Appendix § 14.§ 2). With a single glimpse, i.e. only the initial center fixation, random-location and learned-policy models perform similarly, since no fixation has yet been chosen by the policy. Beyond the training budget, performance continues to scale upward for all benchmarks except C-MNIST, which saturates early (see Appendix § 13 for full scaling curves).

C-MNISTImageNetADE20KCOCO-Stuff
# GlimpsesAcc (%)Acc (%)mIoU (%)Px Acc (%)mIoU (%)Px Acc (%)
Baseline trained and evaluated with full images
–99.570.530.774.429.760.8
Trained and evaluated with random locations
176.6±0.276.6 \pm0.259.0±0.159.0 \pm0.119.7±0.319.7 \pm0.366.0±0.166.0 \pm0.120.2±0.220.2 \pm0.251.4±0.251.4 \pm0.2
490.6±0.390.6 \pm0.363.7±0.163.7 \pm0.121.8±0.421.8 \pm0.468.1±0.068.1 \pm0.023.2±0.223.2 \pm0.254.7±0.154.7 \pm0.1
693.4±0.293.4 \pm0.264.8±0.164.8 \pm0.122.7±0.022.7 \pm0.068.9±0.168.9 \pm0.124.0±0.124.0 \pm0.155.7±0.155.7 \pm0.1
Trained and evaluated with its own learned policy
176.4±0.276.4 \pm0.258.8±0.258.8 \pm0.219.4±0.419.4 \pm0.466.0±0.266.0 \pm0.220.3±0.220.3 \pm0.251.3±0.051.3 \pm0.0
499.4±0.099.4 \pm0.068.5±0.368.5 \pm0.324.3±0.124.3 \pm0.169.9±0.269.9 \pm0.224.9±0.324.9 \pm0.356.4±0.256.4 \pm0.2
699.4±0.199.4 \pm0.169.4±0.369.4 \pm0.324.9±0.224.9 \pm0.270.5±0.370.5 \pm0.325.9±0.325.9 \pm0.357.5±0.257.5 \pm0.2

Table 2. Native policy evaluation. A passive full-image baseline (top), models trained with random locations (middle), and models trained concurrently with a policy (bottom). Values report the mean ±\pm std across three random seeds (single run for passive baselines). All active vision models were trained with a fixed horizon of T=4T = 4, but are evaluated at different # Glimpses. For the learned-policy models, test-time fixations were selected deterministically via argmax from the policy’s probability map. Passive baselines receive the full image at 128×128128 \times128 pixels.

We compare FoAViS to passive full-image baselines with a similar architecture and pixel budget at T=4T = 4 glimpses (see § 4.2) and find that it underperforms them (−2.0% Acc on ImageNet; −6.4 mIoU on ADE20K; −4.8 mIoU on COCO-Stuff). One contributing factor may be that fewer of the pixels it receives are unique due to foveal oversampling (Appendix § 9.§ 2). The gap is larger for segmentation, possibly because this task requires a detailed semantic understanding of the full image, which can be hard to obtain from a sequence of foveated glimpses. Two design choices may also put FoAViS at a disadvantage relative to the passive baselines: (1) Its backbone applies convolutions in log-polar coordinates, whereas convolutional architectures were developed for Cartesian images. (2) The per-position LSTM memory updates each location independently, limiting how information is combined across glimpses.

Policy transfer evaluation

Because our model learns to fixate on task-relevant regions, a baseline trained and evaluated on random locations naturally underperforms. However, Berreby et al. [7] show that random-location models can benefit substantially from more sophisticated policies at test time. Conversely, it remains an open question how robust models trained concurrently with a policy are when evaluated with other fixation strategies. To investigate this, we evaluate every model under non-native policies. That is, a model receives externally supplied fixation points instead of selecting them with its native policy. We consider two types of such policy transfer.

In heuristic policy transfer, fixations are selected either independently of image content or based on internal activation maps of the model. Specifically, we evaluate five strategies (see Appendix § 15): (1) uniform random locations, (2) a fixed triangle scan pattern, (3) a fixed quadrant scan pattern, (4) a saliency-based heuristic inspired by Lukanov et al. [59] combined with a hardcoded IoR mechanism (suppression radius of a quarter of the image width), and (5) an entropy-based heuristic inspired by Berreby et al. [7] using the same hardcoded IoR mechanism.

In learned policy transfer, fixations come from another learned-policy model: it is run on the images to be evaluated, and the fixation sequences it selects (probability map argmax) are used as forced fixations for the receiving model. The receiving model can be another learned-policy model or the random-location model.

Policy transfer to the random-location model. We find that models trained on random locations can be successfully paired with more elaborate policies to achieve improved performance (Table 3). For example, on ImageNet, both the triangle policy (65.0%) and the saliency policy (65.6%) outperform random locations (63.7%). The learned ImageNet policy yields further gains (66.3%), surpassing the quadrant policy (65.9%) despite the latter using an additional glimpse. While segmentation models also improve with heuristic and learned policies, the learned policies provide no additional benefit over the heuristic ones. Interestingly, even when evaluated with the corresponding learned policy, the model trained on random locations fails to match the performance of models trained concurrently with a policy on ImageNet (66.3% vs. 68.5%), ADE20K (22.8 vs. 24.3 mIoU), and COCO-Stuff (24.1 vs. 24.9 mIoU). This demonstrates that the co-adaptation of representations to the fixation policy during training is beneficial. We see two possible, non-exclusive explanations: (1) an encoder trained concurrently with a policy is exposed to more informative data during training and thus learns better representations, and (2) the memory can learn to aggregate information more effectively across glimpses when these are directed to semantically meaningful locations. The result also puts the findings of Berreby et al. [7] into perspective, who report that policy-agnostic training benefits from sophisticated policies at test time, but do not investigate concurrent training of encoder and policy.

Evaluation policy# GlimpsesImageNet Acc (%)ImageNet mIoU (%)ADE20K Px Acc (%)ADE20K mIoU (%)COCO-Stuff Px Acc (%)
random locations463.7±0.163.7 \pm0.121.8±0.421.8 \pm0.468.1±0.068.1 \pm0.023.2±0.223.2 \pm0.254.7±0.154.7 \pm0.1
triangle465.0±0.165.0 \pm0.122.6±0.122.6 \pm0.168.9±0.168.9 \pm0.124.2±0.224.2 \pm0.255.8±0.155.8 \pm0.1
quadrant565.9±0.165.9 \pm0.123.1±0.223.1 \pm0.269.5±0.169.5 \pm0.124.8±0.224.8 \pm0.256.6±0.256.6 \pm0.2
saliency + IoR465.6±0.265.6 \pm0.2––––
entropy + IoR4–22.4±0.322.4 \pm0.368.5±0.168.5 \pm0.123.3±0.223.3 \pm0.255.1±0.155.1 \pm0.1
ImageNet policy466.3±0.166.3 \pm0.122.8±0.322.8 \pm0.368.7±0.168.7 \pm0.124.0±0.124.0 \pm0.155.0±0.155.0 \pm0.1
ADE20K policy464.7±0.164.7 \pm0.122.8±0.322.8 \pm0.369.0±0.169.0 \pm0.123.6±0.023.6 \pm0.055.3±0.055.3 \pm0.0
COCO-Stuff policy464.9±0.264.9 \pm0.222.9±0.322.9 \pm0.368.9±0.168.9 \pm0.124.1±0.224.1 \pm0.255.6±0.155.6 \pm0.1
random locations463.3±0.163.3 \pm0.121.7±0.221.7 \pm0.268.3±0.168.3 \pm0.123.4±0.223.4 \pm0.254.7±0.154.7 \pm0.1
triangle465.3±0.165.3 \pm0.122.9±0.222.9 \pm0.269.3±0.269.3 \pm0.224.5±0.124.5 \pm0.156.0±0.156.0 \pm0.1
quadrant566.1±0.266.1 \pm0.223.4±0.423.4 \pm0.469.9±0.269.9 \pm0.225.2±0.225.2 \pm0.256.9±0.156.9 \pm0.1
saliency + IoR466.7±0.266.7 \pm0.2––––
entropy + IoR4–22.9±0.422.9 \pm0.469.1±0.369.1 \pm0.323.9±0.123.9 \pm0.155.7±0.255.7 \pm0.2
ImageNet policy468.5±0.368.5 \pm0.323.7±0.223.7 \pm0.269.4±0.269.4 \pm0.224.9±0.124.9 \pm0.155.5±0.155.5 \pm0.1
ADE20K policy465.5±0.165.5 \pm0.124.3±0.124.3 \pm0.169.9±0.269.9 \pm0.224.3±0.124.3 \pm0.155.8±0.155.8 \pm0.1
COCO-Stuff policy465.5±0.565.5 \pm0.524.0±0.424.0 \pm0.469.8±0.369.8 \pm0.324.9±0.324.9 \pm0.356.4±0.256.4 \pm0.2

Table 3. Table 3: Policy transfer evaluation. Models trained with random locations (top block) and models trained on a specific dataset with their own learned policy (bottom block) are evaluated with heuristic policies (top sub-block) and with learned policies (bottom sub-block). Shaded cells indicate evaluation with a model’s native policy. Values report the mean ±\pm std across three models trained with different random seeds. The heuristic policies are defined in Appendix § 15.

Policy transfer to the learned-policy model. When evaluating a model trained concurrently with a policy under other fixation strategies, we find that performance generally degrades. On ImageNet, this drop is especially pronounced: its native policy performance (68.5%) substantially exceeds that of the closest alternatives, namely the saliency-based policy (66.7%) and the learned ADE20K and COCO-Stuff policies (65.5% each). Evaluating it on random locations (63.3%) yields worse performance than the model trained directly on random locations (63.7%). Furthermore, under the triangle policy (65.3%), it performs roughly on par with the random-location model evaluated under the same policy (65.0%). A similar pattern is observed for the segmentation models. These results indicate that the backbone of a model trained concurrently with a policy specializes to the input distribution induced by that policy.

Properties of learned policies

Beyond task performance, we analyze the behavioral characteristics of the scanpaths generated by our models, contrasting them across tasks and comparing them against human scanpaths from MIT1003 (Judd et al., [46]). These human scanpaths were recorded with a display diagonal of approximately 35∘35^\circ of visual angle (Kümmerer et al., [53]), roughly 28∘×21∘28^\circ\times21^\circ for typical 1024×7681024 \times768 pixel images. In contrast, our log-polar foveation architecture operates directly in pixel space without an explicit mapping to visual degrees.

Consequently, this comparison should be regarded as an initial reference point to contextualize model behavior and motivate future research on human and model scanpaths. We first evaluate standard scanpath statistics following Kümmerer et al. [53]. We then quantify how consistent learned policies are across different random seeds for the same task, and how much policies differ across tasks. Finally, we examine the IoR characteristics of our trained policies.

Cumulative fixation distribution. A behavior common to all our trained models is to avoid fixating at the very edge of an image, reflected by the dark border in the cumulative fixation distributions (Figure 1B). This can be explained by models learning to look around efficiently: Since the glimpse captures a region around the exact fixation point in sufficiently high resolution, fixating at the very edge would leave much of that informative region outside the image. Beyond this shared behavior, different tasks and datasets yield characteristic fixation distributions. For instance, segmentation models allocate fewer fixations to the top or bottom of an image, most notably on ADE20K, likely because these regions tend to contain less informative content such as sky or ground. Furthermore, segmentation models tend to explore peripheral regions, whereas ImageNet models concentrate fixations in a ring around the center, reflecting the object-centered nature of the dataset. A tendency to direct fixations towards the image center is well documented in humans and more pronounced than in our models (see Appendix § 16.§ 2), remaining robust across different visual inputs and tasks Tatler, [96]. Appendix § 16.§ 2 shows these distributions for all three seeds, separately for each step, and with each policy applied to the images of the other datasets.

Saccade amplitude. We examine the distribution of saccade amplitudes Bahill et al., [2] across fixations f2f_2 to f4f_4 following the initial center fixation f1f_1 (Figure 3A). For all models, the initial saccade from f1f_1 to f2f_2 is constrained in amplitude by the central starting position. After this initial saccade, classification models exhibit smaller amplitudes than segmentation models. This difference can be explained by their respective objectives: while classification requires focusing on a central object of interest, segmentation models need to obtain a comprehensive understanding of the scene. For C-MNIST in particular, exploration decreases sharply across fixations as the target digit is found in an increasing number of cases. Compared to our models, humans consistently make shorter saccades, which might be explained by several characteristics and constraints of human eye movements that favor shorter saccades and are not included in our model. First, saccade duration increases with amplitude Bahill et al., [3], prolonging the period during which vision is degraded van Beers, [101]. Second, larger saccades are subject to greater endpoint variability, making them less reliable van Beers, [101]. Third, in natural viewing, large gaze shifts are typically accomplished by combined eye and head movements Bahill et al., [2]; Freedman [32]. Lastly, short saccades in humans may serve local exploration or correct a preceding saccade, and tend to follow after shorter fixation durations Tatler & Vincent, [97]. Foveated vision has also been proposed to favor short saccades Tatler & Vincent, [98], though the absence of similarly short saccades in our foveated models suggests that foveation alone might not be sufficient to produce such behavior.

Figure 3. Scanpath statistics of the learned policies. Results are reported for a single seed, but remain consistent across training runs with different seeds (Appendix § 16.§ 1). Unless stated otherwise, metrics for our models are calculated for the standard training horizon of three fixations (f2f_2, f3f_3, f4f_4) after an initial center fixation (f1f_1). The human dataset (MIT1003) was also recorded with an initial center fixation and metrics were calculated for all scanpaths that consist of at least three fixations afterwards. (A) Saccade amplitude across first three saccades. (B) Saccade direction across first three saccades. (C) Mean amplitude of the next saccade as a function of the preceding one (amplitudes standardized within each scanpath). Our models were unrolled for nine fixations (f2f_2–f10f_{10}) starting from a center fixation (f1f_1). For MIT1003, we selected scanpaths with at least nine fixations after the center fixation and truncated them to these nine. The plot is based on the saccades between f2f_2 and f10f_{10}. The initial saccade from the center to f2f_2 is excluded because its starting point is fixed. (D) Turning angle between two consecutive saccade vectors, measured between the first and second saccade (at f2f_2) and between the second and third saccade (at f3f_3).

Saccade amplitude autocorrelation. We study saccade amplitude autocorrelation Tatler & Vincent, [97]; Wilming et al. [106] by computing the mean amplitude of the next saccade as a function of the preceding one, standardizing amplitudes using the mean and standard deviation within each scanpath to account for variability across images (Figure 3C). Due to the standardization, a negative association is observed even for a uniformly random policy when scanpaths are relatively short, growing stronger as length decreases (see Appendix § 16.§ 3 for a more detailed analysis). We therefore benchmark against this random policy using scanpaths of nine fixations. On MIT1003 Judd et al., [46], human observers display a clear negative association beyond that of the random policy. This finding aligns with prior work reporting a weak negative association between consecutive saccade amplitudes Kümmerer et al., [53]; Malem-Shinitski et al. [61], consistent with periods of local scanning separated by large relocations to new scene regions Tatler & Vincent, [97]. In contrast, our models exhibit no substantial negative association beyond that of the random policy. Thus, while our models do explore individual informative image regions in more detail (Figure 1 and Appendix § 14.§ 2), they lack the human pattern of long saccades that relocate to a new region followed by short saccades that explore it. As short saccades following long ones may serve local exploration or correct a preceding saccade Tatler & Vincent, [97]; Kümmerer et al. [53], and tend to follow short fixations Tatler & Vincent, [97], a framework that also models fixation durations might favor the emergence of this pattern.

Saccade direction. Regarding direction [98], all models trained on natural images show a bias towards horizontal (left/right) and vertical (up/down) saccades, whereas diagonal saccades occur less frequently (Figure 3B). The strength of these biases differs between models: the horizontal bias is particularly strong for the ADE20K model, while the vertical bias is most pronounced for the ImageNet model. The most likely explanation for these biases is that information of interest in natural images is often arranged along horizontal or vertical axes. For example, objects frequently have a vertical orientation, which explains why the vertical bias is strongest for the object-centric ImageNet and why the synthetic C-MNIST dataset does not exhibit any horizontal or vertical biases. In the human scanpath literature, similar biases are well documented [31, 53]. In fact, the pattern observed on MIT1003 closely resembles those that emerge naturally in a task-driven way in our models. Prior work likewise attributes the human horizontal bias partly to image content [31, 53, 77]. However, observer-related factors also contribute [65, 78, 49]. For instance, diagonal saccades require more effort than horizontal or vertical ones and therefore tend to be avoided by humans [49]. Our model serves as a flexible framework where such constraints could be added or omitted selectively to observe how different factors contribute to the observed pattern.

Saccade turn angle. We examine the turn angle between two consecutive saccades [80, 92, 106], which represents the angular change in direction from one fixation to the next. Across all models, saccade probability increases with absolute turn angle, with exact reversals of 180∘180^\circ being the most likely (Figure 3D). This tendency is particularly pronounced for the ADE20K model, which is also the only model that exhibits a slight continuation bias at f3f_3, favoring subsequent saccades in the same direction near 0∘0^\circ. On MIT1003, we also observe both biases, consistent with established findings in human scanpaths [97, 106, 53]. Reversals have been partly attributed to image boundaries [106]: even a scanpath model based solely on saliency and inhibition of return produces frequent turns near 180∘180^\circ, simply because reaching an edge naturally turns the gaze back [86]. In addition, humans tend to return to informative locations [92, 106]. Continuation, in contrast, has been attributed to oculomotor bias [92] (Bays & Husain 2012), small corrective saccades [97] and lingering presaccadic attention [86], which stays at the same retinal position and thus points further along the same direction after a saccade has landed. For our models, image boundaries most likely explain the high rate of reversals, especially given their tendency towards large saccade amplitudes. The continuation bias on ADE20K likely reflects the model’s tendency to explore images horizontally (Figure 1B), which may also be one factor that contributes to the pattern observed in humans.

Argmax fixation distance. Beyond aggregate scanpath statistics, we evaluate cross-task policy similarity and within-task consistency across random seeds using a conditional evaluation scheme adapted from scanpath modeling [52, 53]. In this scheme, models are scored on next-fixation predictions conditioned on identical fixation histories. While prior work performs the conditional evaluation using continuous predictions like log-likelihood, our policies are unregularized and highly peaked with clear maxima, making such density-based scores ill-conditioned. Instead, we evaluate spatial distance between predicted fixations directly under the following protocol: For each dataset, scanpaths are first generated by a native reference model trained on that dataset. Fixations are selected via argmax⁡\operatorname{argmax} from policy probability maps across nine fixations f2f_2–f10f_{10}, starting from an initial center fixation f1f_1. These scanpaths are then fed as forced fixation sequences to models trained on (1) other tasks or (2) the same task with different random seeds. At each step, we measure the argmax⁡\operatorname{argmax} distance defined as the Euclidean distance between the argmax⁡\operatorname{argmax} location predicted by that model’s policy probability map and the reference fixation (Figure 4).

Figure 4. Argmax fixation distance. Euclidean distance between fixations from the native reference model and fixations from evaluated models conditioned on the native model’s fixation history, computed across nine fixations f2f_2–f10f_{10} following the initial center fixation f1f_1 (see § 5.3 for details). Distances are expressed as a percentage of image width (all images are square, so width equals height). Lower values indicate stronger policy alignment. Colored lines indicate comparison models (different tasks or the same task with different random seeds). The gray line denotes the expected distance when locations are drawn uniformly at random. Shaded bands show ±1\pm1 std over three model pairs. The dotted line marks the T=4T = 4 glimpses used in training.

Models trained on the same task across random seeds exhibit consistently lower argmax distances than models trained on different tasks, demonstrating stronger overall policy alignment. Nevertheless, within-task argmax distances remain non-trivial, measuring at least 20% from f2f_2 onward. This suggests that no single canonical policy emerges, arguably because multiple distinct fixation strategies can effectively tackle the same task. Compared to a uniformly random policy, learned policies are in general more similar to each other, indicating that they share some general principles of where to look next given the same history of past fixations (e.g. saliency and inhibition of return). We also find that alignment decreases as forced fixation history grows, suggesting that the optimal next fixation becomes increasingly ambiguous. In Appendix § 16.§ 4, we compare policies based on two further similarity metrics and obtain consistent results.

Emergent inhibition of return. Human scanpaths naturally avoid looking back at recently visited locations, a mechanism known as inhibition of return ([41]). To test whether our models learn a similar suppression mechanism, we evaluate whether visiting a location depresses its policy logits at subsequent steps. Specifically, models trained with T=4T = 4 glimpses are unrolled for an extended horizon of Teval=10T_{\mathrm{eval}} = 10 steps (t=1,…,10t = 1, \ldots, 10), starting from the initial center fixation f1f_1 at t=1t = 1. We then track the policy logit at each fixation coordinate fkf_k (k∈{2,…,10}k \in\{2, \ldots, 10\}), which is visited at step t=kt = k, across all steps tt of the episode. These values are then grouped by their relative lag t−kt-k to the visit and averaged over all fixations and images. To make values comparable across timesteps and images, we consider z-standardized logits over all spatial positions of the 64×6464 \times64 probability map.

For the three natural-image datasets, we find that the logits of a location fkf_k gradually increase up until the point where the location is actually fixated (Figure 5). Right after the fixation, the logits for that location sharply decrease and then partially recover over the following steps, making a revisit more likely further into the future, a pattern similarly reported for human gaze behavior ([72]). The same pattern also holds for every visit index separately (Appendix § 16.§ 5).

Figure 5. Emergent inhibition of return. zz-standardized policy logits at visited locations as a function of relative lag t−kt-k (see § 5.3 for details). Thin lines show individual models trained with different random seeds, while bold lines indicate their mean. Negative / positive lags correspond to steps before / after the fixation. Lower / higher logits reflect reduced / increased selection probability at that location.

Challenges in joint task and policy optimization

In primate vision, gaze is guided by priority maps represented in a dedicated oculomotor network of cortical and subcortical areas ([29, 8]), that interacts with the ventral, object-recognition pathway in both directions ([82, 63]). In contrast, we train the policy on top of memory features that are optimized solely through the task loss (see § 3.6), as is also done in AdaptiveNN ([103]). As a result, FoAViS lacks a structural counterpart capable of self-organizing representations specifically for fixation selection. To allow the policy to shape intermediate representations, a natural first option is to let the policy gradient also backpropagate through the memory and backbone. We refer to this setting as joint task and policy optimization, as both objectives then shape the backbone and memory. We experimented with this setting extensively and consistently observed optimization issues that we aim to raise awareness of in the following. These issues can be clearly illustrated by ablating the stop-grad operation while keeping all other components and objectives from § 3 fixed. In that case, we observe substantially degraded task performance across ImageNet, ADE20K, and COCO-Stuff compared to our default stop-grad setting (Figure 6A). A likely cause is high variability in the policy gradient without stop-grad (Figure 6B), though the precise underlying mechanism requires further investigation (see Appendix § 18 for a more detailed analysis). Empirically, we find that most of the performance can be recovered by adding entropy regularization to the probability distribution output by the policy head (see Appendix § 18.§ 5 for implementation details), which also stabilizes the relative magnitude between policy and task gradient. However, task performance under entropy regularization still tends to fall short of our default stop-grad setting (Figure 6A), suggesting that the optimization issues are not fully resolved. At the same time, the resulting policies slightly exceed the performance of their stop-grad counterparts when transferred to a model trained on random locations (Table J.2). Our findings highlight the complexity of joint task and policy optimization, where performance trade-offs can be subtle, as illustrated by the entropy-regularized setting. Further research is needed to explore both architectural designs and optimization algorithms that enable stable, efficient, and effective learning of task and policy in active vision systems.

Figure 6. Training dynamics under different optimization settings. (A) Task performance across models trained with stop-grad (see § 3.6), without stop-grad, and without stop-grad alongside entropy regularization. The triangle marks the checkpoint evaluated in panel (B), matched at roughly 150k training iterations. (B) Distribution of the ratio qq of policy to task gradient magnitudes for backbone and memory parameters, as a kernel density estimate over log⁡10q\log_{10} q with one value per minibatch, computed over 78 (ImageNet, batch size 64) or 312 (ADE20K and COCO-Stuff, batch size 16) training minibatches, i. e. about 5k images per model. The stop-grad model yields zero policy gradient to these components and is omitted.

Conclusion

Summary

We presented FoAViS, an active vision model that consists of log-polar foveation, a backbone and 2D spatial memory, as well as a task and policy head (§ 3.1 to § 3.5), and systematically studied its gaze behavior across different tasks. The system can simultaneously learn where to look and what it sees (§ 3.6). We found that the learned policy selects more informative regions than random or heuristic baselines (§ 5.1). Furthermore, the model’s representations develop concurrently with its own learned policy, having a positive effect on model performance (§ 5.2). Beyond task performance, our models exhibited structured, task-specific gaze behaviors that reproduce key aspects of human scanpath statistics while also showing notable divergences (§ 5.3). These empirical patterns, along with phenomena such as inhibition of return, emerged naturally from concurrent task-policy optimization without explicit supervision, providing a normative perspective on gaze behavior. Finally, we drew attention to optimization challenges encountered during joint task and policy training, raising the broader question of how to structure and optimize multi-component active vision systems (§ 5.4).

Limitations and outlook

FoAViS is built on top of a simple Res-UNet with roughly 13.5M parameters and is trained from scratch with a relatively low compute budget. Unsurprisingly, it therefore trails the raw performance of higher-capacity models that leverage larger, more powerful model architectures (AdaptiveNN; [103]), more extensive training, and/or strong distilled features (CanViT; [7]) as a starting point (see also Appendix § 12). Our model would certainly benefit from integrating such techniques to obtain stronger downstream task performance, as a faithful computational model of primate vision should competently solve a broad range of visual tasks. However, evaluating active vision models purely on task performance relative to compute neglects other critical axes along which such systems need to be judged when serving as models of biological vision. Specifically, we identify three key dimensions: (1) architectural alignment with biological vision (foveation, spatial memory, and cortical areas), (2) behavioral alignment with human scanpaths, and (3) representational alignment, which has so far primarily been studied in single-pass vision models.

Architectural alignment. Our model implements a log-polar image sampling and learns both image-level and dense vision tasks concurrently with an active vision policy. While these are important properties for an active vision system, our architecture still lacks several core characteristics of biological vision. First, our log-polar sampling approximates retinal sampling only qualitatively, sacrificing local isotropy to map the foveated representation onto a Cartesian grid for standard convolutions. Recent work provides more realistic foveation mechanisms that could be used instead ([9]). A further limitation is that our model has no notion of time: every glimpse receives the same amount of computation. In humans, fixation durations vary and partly adapt to the processing demands of the currently fixated region ([39]; [67]; [106]). This temporal dimension likely shapes several of the scanpath statistics analyzed in § 5.3 ([97]; [92]; [106]). To address this, the policy could explicitly predict fixation duration, using architectural mechanisms like variable network depth to allocate compute dynamically. Furthermore, our model currently encodes each glimpse strictly bottom-up before merging it with the 2D scene memory. Consequently, the backbone cannot leverage top-down feedback from information accumulated across previous glimpses, a core characteristic of primate vision [6, 34]. Recent work has started to explore top-down feedback in an active vision architecture [7]. Moreover, our 2D spatial memory is designed for static scenes. When dealing with dynamic environments, keeping an exhaustive map is neither practical nor biologically plausible, leaving open what such a memory should store and in which reference frame. Finally, our architecture does not faithfully reflect distinct cortical areas and their communication, a factor critical for representational alignment.

Behavioral alignment. Prior work has established that training fixation policies concurrently with global or dense tasks is feasible across a wide range of foveation mechanisms and architectures [62, 1, 71, 103], which is further corroborated by our results. However, core architectural and optimization questions remain open: Which parts of the network should be responsible for the task prediction, for fixation selection, or for both, and how should information flow between them, potentially inspired by the dorsal and ventral streams? How do we optimize these objectives together without running into optimization instability? And what algorithms should we use to train policies for various tasks in the most efficient and effective way? Answering these questions will bring us closer to achieving an unbiased normative account of gaze behavior under different task constraints, providing a powerful tool to study similarities and differences between machine and human gaze behavior. Observed behavioral differences can serve as guidance to uncover systematic biases and inefficiencies as well as their underlying causes, which may in turn help us design systems that behave more like humans. Notable differences could potentially be reverse-engineered into a machine vision architecture through dedicated pretraining schemes, structural changes, or additional loss terms. Accurate behavioral alignment is arguably also important for adequate representational alignment, as the incoming visual inputs shape how representations emerge in the brain.

Representational alignment. While correspondence between deep artificial networks and the ventral visual stream has been widely studied in passive models [108, 14], researchers have only recently begun addressing neural questions in active vision [73, 100, 94]. A primary obstacle has been the scarcity of datasets that jointly record gaze trajectories and multi-area neural population activity during unconstrained active viewing. Future experiments in the visual system of non-human primates should aim to close this gap, which could pave the way for a new generation of computational models of the primate visual system. With our work, we aim to provide a foundation for developing such a system, while also highlighting the challenges that remain on the way. Ultimately, this line of research opens a path towards comprehensive in silico models, enabling systematic manipulations to investigate the interactions between eye movements, internal representations, and behavioral goals in biological vision.

AI Usage Statement

We used generative models and agentic tools for writing and debugging code, running experiments, and editing the manuscript. We thoroughly verified all results produced by these models and take full responsibility for the content of this paper.

Acknowledgments

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – GRK2906 – project number 502807174. Yohai-Eliel Berreby was funded by a Unifying Neuroscience and Artificial Intelligence - Québec (UNIQue) PhD Excellence Scholarship and by a Natural Sciences and Engineering Research Council of Canada (NSERC) Canada Graduate Research Scholarship – Doctoral program. Haoze Zhu was funded by the German Federal Ministry of Research, Technology and Space (BMFTR) under grant 01GQ2501A (PRISMA-V). The authors gratefully acknowledge the computing time granted by the Resource Allocation Board and provided on the supercomputer Emmy/Grete at NHR-Nord@Göttingen as part of the NHR infrastructure. The calculations for this research were conducted with computing resources under the project nim00018.

Implementation details

This appendix provides further information about the architecture and training procedure. The architecture is presented in more detail in Appendix § 9.§ 1. Quantitative characteristics of the log-polar sampling are presented in Appendix § 9.§ 2. Further information on the training and evaluation protocol is given in Appendix § 9.§ 3.

Architecture

Tables A.1 and A.2 summarize the backbone and the recurrent memory with its task heads. The ImageNet classification model has about 15.3M parameters: 13.5M backbone, 1.2M classification head, 0.44M policy head and 0.13M recurrent memory. The C-MNIST model has 15.0M (0.95M classification head), and the semantic segmentation models have 14.2M (0.17M segmentation head). The backbone uses instance normalization throughout to cope with input statistics that differ across glimpses.

Details on the retinal layout of a glimpse

This subsection describes how the foveation module (§ 3.1) distributes its samples over the image. Each glimpse has 64×6464 \times64 samples: 64 rings spaced evenly in log⁡r\log r, from r=1r = 1 px to the image width SS, with 64 samples spaced evenly in angle on each ring. Because the rings are spaced evenly in log⁡r\log r, the distance between neighboring samples grows in proportion to the distance from the fixation.

Close to the fixation, neighboring samples are less than a pixel apart, so the image is oversampled: up to 11 px from the fixation for 224×224224 \times224 images and 10 px for 512×512512 \times512 images. This disk covers less than 1% of the image (about 360 pixels) but receives 44% of the samples, so about 1430 of the 4096 samples (35%) read pixels that are already sampled (29% for 512×512512 \times512). We did not optimize this allocation, since the exact design of the foveation is not the focus of this work. A different sampling scheme would leave the principles of the architecture unchanged. Beyond the disk, neighboring samples are more than a pixel apart. To reduce aliasing there, each sample is the mean of a 4×44 \times4 block of a grid sampled 4× finer, rather than a single read at its position. Because the outermost ring has radius SS, part of the grid falls outside the image and reads zeros: 11% of the samples for a fixation at the image center (10% for 512×512512 \times512), 53% at the middle of an edge and 75% in a corner.

Both resolutions follow the same rule. Because the rings run from 1 px to SS, the gap between neighboring rings at a given distance from the fixation depends on the image size only through log⁡S\log S, which grows by only 15% from 224 to 512 although the image is more than twice as wide. At the same distance from the fixation in pixels, the rings of the 512 grid are therefore only about 15% further apart, and the spacing along a ring is the same, so in pixels the two grids sample the image nearly alike. Relative to the image they differ, because a pixel is a smaller part of a larger image: the oversampled disk covers 2.0% of the image width at 512 and 4.9% at 224 (Figure A.1).

Training and evaluation details

Optimization. Table A.3 lists the hyperparameters. The learning rate rises linearly during warmup, stays constant and is then halved at fixed intervals.

Fixations and policy. The first fixation is the image center. During training, each further fixation is drawn from the policy’s probability map (Equation (3)) and placed uniformly at random inside the drawn cell of the 64×6464 \times64 grid. The rewards are standardized per step over the samples on each GPU (Equation (7); 16 samples for ImageNet due to multi-GPU training, the whole batch otherwise). In the random-location setting, every fixation after the first is drawn uniformly from the whole image, and the policy head is neither used nor trained. Validation during training draws fixations as above; reported task metrics are instead evaluated with the center of the most probable cell as fixation (argmax, see Evaluation below).

Data. ImageNet images are normalized with ImageNet statistics. In training, they are randomly resized and cropped to 224×224224 \times224 (scale [0.08,1][0.08, 1]) and flipped horizontally; in evaluation, the shorter side is resized to

Table A.1: Backbone architecture. The backbone takes a log-polar transformation of the input image. Each ResBlock uses 3×33 \times3 convolutions, with a 1×11 \times1 projection on the skip connection where needed. In a ResBlock, each convolution is followed by Instance Normalization and the first also by a ReLU; the second ReLU follows the residual addition. The bottleneck operates at the encoder’s final resolution. In the decoder, each “Conv 1×11 \times1 fuse” row concatenates the incoming features with the corresponding encoder feature map (channel counts in parentheses) and projects them to the decoder width, again followed by Instance Normalization and ReLU.

ComponentLayerRes. changeOut ch.Output size
ResBlock (stride 1)–6464×6464 \times64
ResBlock (stride 1)–6464×6464 \times64
EncoderResBlock (stride 2)↓ 212832×3232 \times32
ResBlock (stride 1)–12832×3232 \times32
ResBlock (stride 2)↓ 225616×1616 \times16
ResBlock (stride 1)–25616×1616 \times16
ResBlock (stride 2)↓ 25128×88 \times8
ResBlock (stride 1)–5128×88 \times8
BottleneckResBlock (stride 1)–1288×88 \times8
Conv 1×11 \times1 fuse (128+512)–1288×88 \times8
ResBlock (stride 1)–1288×88 \times8
UpConv (stride 2)↑ 212816×1616 \times16
Conv 1×11 \times1 fuse (128+256)–12816×1616 \times16
ResBlock (stride 1)–12816×1616 \times16
DecoderUpConv (stride 2)↑ 212832×3232 \times32
Conv 1×11 \times1 fuse (128+128)–12832×3232 \times32
ResBlock (stride 1)–12832×3232 \times32
UpConv (stride 2)↑ 212864×6464 \times64
Conv 1×11 \times1 fuse (128+64)–12864×6464 \times64
ResBlock (stride 1)–12864×6464 \times64
Inverse log-polarLog-polar → Cartesian 64×6464 \times64 grid (bilinear)–12864×6464 \times64

Table A.1.

Table A.2: Recurrent memory and task heads. The recurrent memory takes its past state and the current backbone output. It is an LSTM cell applied independently at each of the 64×6464 \times64 spatial positions and unrolled over the TT glimpses; the weights are shared across positions, and each position carries its own state. The policy, classification, and segmentation heads take the memory features Ht\mathbf{H}_t as input, the policy head through a stop-gradient. KK is the number of classes.

ComponentLayerRes. changeOut ch.Output size
Recurrent memoryStandard LSTM cell applied at every position–12864×6464 \times64
Policy headConv 3×33 \times3 + ReLU–12864×6464 \times64
Conv 3×33 \times3 + ReLU–12864×6464 \times64
Conv 3×33 \times3 + ReLU–12864×6464 \times64
Conv 1×11 \times1 → probability map–164×6464 \times64
Classification headConv 3×33 \times3 (stride 2) + ReLU↓ 225632×3232 \times32
Conv 3×33 \times3 (stride 1) + ReLU–25632×3232 \times32
Global avg-pool↓ 322561×11 \times1
Linear + ReLU–2561×11 \times1
Linear → classes–KK1×11 \times1
Segmentation headConv 3×33 \times3 + ReLU–12864×6464 \times64
Conv 1×11 \times1 → classes–KK64×6464 \times64
Bilinear upsample (×8\times8)↑ 8KK512×512512 \times512

Table A.2.

Figure A.1. Sampling density of the glimpse at the classification (224 × 224) and segmentation (512 × 512) resolutions. The spacing between neighboring samples, roughly the smallest detail a glimpse resolves, is plotted against the distance from the fixation, both as a percentage of the image width. Up to 11 px (224) and 10 px (512) from the fixation, the image is oversampled: neighboring samples are less than a pixel apart. The plot floors the spacing at one pixel there (plateaus, labeled full resolution), since samples closer than a pixel add no detail.

HyperparameterImageNetC-MNISTADE20KCOCO-Stuff
InitializationrandomrandomImageNet random-location modelImageNet random-location model
OptimizerAdamW, weight decay 0.01 (all parameters)AdamW, weight decay 0.01 (all parameters)AdamW, weight decay 0.01 (all parameters)AdamW, weight decay 0.01 (all parameters)
β1\beta_1, β2\beta_2, ϵ\epsilon0.99, 0.999, 10−810^{-8}0.99, 0.999, 10−810^{-8}0.99, 0.999, 10−810^{-8}0.99, 0.999, 10−810^{-8}
Learning rate10−410^{-4}10−410^{-4}2×10−42 \times10^{-4}2×10−42 \times10^{-4}
Warmup2000 steps (linear)2000 steps (linear)2000 steps (linear)2000 steps (linear)
Epochs15040050072
LR constant for (epochs)8923330044
then halved every (epochs)13 (5×)33 (6×)50 (4×)8 (4×)
GPUs4111
Batch size (total)64 (16 per GPU)641616
Input resolution224 × 224224 × 224512 × 512512 × 512
Glimpses (TT)4444
Gradient-norm clip20202020
Precisionbf16 mixedbf16 mixedfp32fp32

Table A.3. Training hyperparameters. Random initialization uses PyTorch’s defaults. The segmentation models take over the backbone and memory of the ImageNet random-location model; their policy and segmentation heads are newly initialized.

256 and the central 224 × 224 crop is taken. C-MNIST canvases (Appendix § 10) are normalized with MNIST statistics and not augmented. Segmentation images are augmented in training as follows. A rectangular region covering 40 to 100% of the image area, with a width-to-height ratio between 0.75 and 1.33, is cut out at a random position and resized to 512 × 512. The region is thus stretched or squashed by up to a factor of 1.33, and its aspect ratio is not preserved. The result is flipped horizontally with probability 0.5, color-jittered (brightness, contrast and saturation 0.4, hue 0.1) with probability 0.8, converted to grayscale with probability 0.1, and rotated by a random angle of up to ±10∘\pm10^\circ, which leaves the corners empty (filled with label 0, see below). In evaluation, the whole image is resized to 512 × 512, again without preserving the aspect ratio. Label maps undergo the same geometric transforms, with nearest-neighbor interpolation.

Segmentation loss and labels. For segmentation, the task loss and the reward (Equations (5) and (6)) use the per-pixel cross-entropy, averaged over the image. Unlabeled pixels (label 0 in ADE20K, 255 in COCO-Stuff) are excluded from loss, reward and metrics. ADE20K has 150 classes and COCO-Stuff 171. For COCO-Stuff, two settings were by mistake taken over from datasets in which label 0 is a background class. In COCO-Stuff, label 0 is instead the class person, while unlabeled pixels have label 255 and are excluded as described above. (1) Label 0 is weighted by 0.1 in the cross-entropy loss (a setting from an earlier setup in which the background class was down-weighted by this factor instead of excluded, applied here by mistake). In COCO-Stuff, this down-weights person pixels, in the task loss and hence in the reward. (2) The image corners that the training-time rotation leaves empty are labeled 0, i.e. as background. In COCO-Stuff, they are labeled as person. Both mistakes affect all COCO-Stuff models in the same way, so comparisons between these models remain valid, and we therefore did not retrain them. Evaluation is not affected.

Evaluation. Every reported task metric comes from the last checkpoint of a run, evaluated with argmax fixations on the full validation set (ImageNet 50 000, C-MNIST 10 000, ADE20K 2000 and COCO-Stuff 5000 images). Classification is scored by accuracy. Segmentation is scored by mIoU and pixel accuracy, computed from one confusion matrix accumulated over the whole validation set; mIoU averages over the classes that occur in the labels or the predictions. Both are computed at 512×512512 \times512, the resolution the models see, not at the original image resolution. Because every image is squashed to a square, this differs from the standard protocol, which resizes the predicted logits to each image’s native resolution before scoring. Scored that way, mIoU changes by −0.5-0.5 on ADE20K and by +0.1+0.1 on COCO-Stuff. The shift is almost the same for models trained with a learned policy and with random locations, so their difference barely moves (from 2.49 to 2.59 mIoU on ADE20K and from 1.76 to 1.77 on COCO-Stuff), and since all models are scored the same way, no comparison in this paper changes. Table D.1, which compares with published numbers, uses the native-resolution score. Where a cell reports mean ±\pm std, it is over three models trained with different random seeds. Some analyses in Appendices § 18 and § 19 deviate from the evaluation protocol described above (see Appendix § 17.§ 2 for details).

Compute. One ImageNet model trains in about three days on four A100 or H100 GPUs, one ADE20K or COCO-Stuff model in about two days and one C-MNIST model in about 20 hours, each on a single GPU.

C-MNIST validation images

Figure B.1. C-MNIST validation images, the first four of the validation set, with the target digit boxed and its label above. The other characters are EMNIST letters.

Cluttered MNIST

C-MNIST (§ 4.1) is generated on the fly. Each image is a 224×224224 \times224 single-channel canvas (224=8×28224 = 8 \times28) with a black background; there is no stored pool of images. The target is one MNIST digit at its native 28×2828 \times28 size, placed at a uniformly random position that keeps it fully inside the canvas (top-left corner uniform in {0,…,196}2\{0, \ldots, 196\}^{2}), and its digit class is the label, giving 10 classes. The target covers 282/2242≈1.6%28^{2}/224^{2} \approx1.6\% of the canvas. Five clutter characters are added, each a 28×2828 \times28 letter drawn uniformly from EMNIST letters and placed independently in the same way; letters are never a class. Target and clutter are combined by a pixel-wise maximum, without rejecting overlaps, so letters can partly cover the target and each other.

Training images take their targets from the MNIST training split (60 000 images) and their letters from the EMNIST letters training split. Position and clutter are drawn anew every time a sample is loaded, so each of the 400 training epochs, of 60 000 images each, sees new images. Validation images use the MNIST test split (10 000 images) and the EMNIST letters test split. Each is a deterministic function of its sample index, which seeds the choice and placement of its clutter, so all evaluations and models see identical validation images. Pixel values are scaled to [0,1][0, 1] and normalized with the MNIST mean and standard deviation (0.1307 and 0.3081); no augmentation is applied. Figure B.1 shows four validation images.

Passive baseline

Table 2 compares the active models with a passive baseline that sees the whole image at once. The passive baseline is the non-foveated counterpart of FoAViS. It follows the design of our backbone and uses the same data pipeline (splits, image sizes and augmentations) as the active models, and differs from them in seeing the whole image in a single forward pass, without foveation, memory or policy.

For classification, the baseline is our full Res-UNet backbone (Table A.1) followed by the classification head of the active model (Table A.2), which reads the backbone output instead of the memory (14.5M parameters for C-MNIST and 14.7M for ImageNet, against 15.0M and 15.3M for the active models, which add memory and policy head; Appendix § 9.§ 1). For segmentation, the baseline is the same backbone with the segmentation head of the active model (13.7M parameters, against 14.2M).

Both baselines receive the image downsampled to 128×128128 \times128, the pixel count of four glimpses, so they use as many pixels and as much backbone computation as four glimpses of the active model (but see Appendix § 9.§ 2 where foveal oversampling is quantified which leads to fewer unique pixels seen in FoAViS). The Res-UNet predicts at its input resolution, so the task head reads a 128×128128 \times128 grid, twice the resolution of the 64×6464 \times64 memory of the active model; the segmentation logits are upsampled to 512×512512 \times512 in the same way. Like the active segmentation models, the segmentation baselines are initialized from an ImageNet model, here the classification baseline: its backbone (encoder and decoder) initializes the Res-UNet, and only the segmentation head starts from random initialization. The active segmentation models additionally take over the memory of the ImageNet random-location model (Table A.3).

All baselines are trained with the same learning rate schedule, number of epochs and batch size as the corresponding active model (Table A.3): AdamW (weight decay 0.01) and cross-entropy, a learning rate of 10−410^{-4} (classification) or 2×10−42 \times10^{-4} (segmentation) with a linear warmup over the first 2000 steps, held constant and then halved at the same epochs as for the active model, for 400 (C-MNIST), 150 (ImageNet), 500 (ADE20K) and 72 (COCO-Stuff) epochs, with batch size 64 (classification) or 16 (segmentation) on the same number of GPUs. The baselines use AdamW with β1=0.9\beta_1 = 0.9 (PyTorch’s default) instead of the active models’ 0.99.

ModelBackboneInit.Input res.Pixels (%)GlimpsesADE20K mIoU (%)ImageNet Acc (%)
DRAM ([1])ResNet-50ImageNet (3-stage)2242224^294.5a94.5^{a}8–67.5b67.5^{b}
Saccader ([27])NASNetImageNet (3-stage)3312331^269.9a69.9^{a}6–75.0
GFNet ([102])ResNet-50ImageNet2242224^291.85–75.9
TNet ([69])BagNet-77from scratch2242224^270.96–74.6
FoveaTer ([45])PatchConvNetResNet-18 + DeiT-S2242224^2n/a3–76.7
Lukanov et al. (2021)Xception-based CNNfrom scratch2562256^210.0a10.0^{a}1–65.2c65.2^{c}
STAM ([74])DeiT-BDeiT (distilled)2242224^255.127–80.8
AME (MAE) ([70])ViT-LMAE128×256128 \times25618.8824.4–
AME (SETR) ([70])ViT-LSETR128×256128 \times25618.8827.6–
Liu et al. (2024)ResNet-101ImageNet (not tuned)2242224^241.35–73.7
AdaGlimpse ([71])ViT-BMAE + glimpse pretr.2242224^236.7 / 28.68 / 1425.777.5
AdaptiveNN ([103])DeiT-Sfrom scratch2882288^262.8d62.8^{d}4.15d4.15^{d}–82.2
CanViT-B ([7])ViT-BDINOv3 distillation5122512^2131 / 10021 / 1645.984.5
FoAViS (ours), ImageNet modelRes-UNetfrom scratch2242224^232.74–68.5
FoAViS (ours), ADE20K modelRes-UNetImageNete^{e}5122512^26.3423.8–

Table D.1. Published results of prior active vision models on ADE20K and ImageNet. Each model is listed with its best published result. These works cannot be compared in a controlled way: backbone, initialization, glimpse budget, input resolution and evaluation protocol all differ. Pixels: pixels used for the task, summed over all glimpses and divided by the pixels of the model’s own input image (Input res.). Pixels are counted as delivered, after any down-scaling, not by the area they cover. A glance or overview pass, if a model uses one, counts at its own size and is included in Glimpses. Overlapping glimpses are not de-duplicated, so a budget can exceed 100%. Two values in a cell are segmentation / classification. A dash marks a benchmark the model does not report.

Published results of prior active vision models

Table D.1 places our results among published active vision models. Each model is listed with its best published result, together with the input resolution, number of glimpses and pixel budget at which it was obtained.

Additional information on individual entries of Table D.1. Saccader and DRAM choose their glimpse locations from a view of the whole image: Saccader’s network processes the full image at 2242224^2, and DRAM’s location network also receives the two wider, lower-resolution crops of each glimpse, the widest spanning the whole image. Only the glimpses reach the classifier, and only they are counted (note a of Table D.1). Saccader itself reports 29.5%, the image area its six glimpses cover after removing overlaps; counted as in the other rows (six 1132113^2 glimpses on 3312331^2 images), it is 69.9%. DRAM was evaluated by the Saccader authors ([27]), whose appendix describes its ImageNet training in three stages (two pretraining stages, then full training), counted as for Saccader; its accuracy is taken from the table of TNet ([69]). Where a paper gives some variants only in plots (GFNet’s other backbones, Saccader’s other glimpse counts), we list its best printed result. For FoveaTer we list its main model; a variant in its appendix that applies six transformer layers to the full feature map before foveating reaches 78.3%. Lukanov’s accuracy and budget are from its own paper, on ImageNet ILSVRC2010. Its result table does not restate the protocol, so we take the one the paper gives for all its models: two 81281^2 foveated glimpses on 2562256^2 images, the first at the center and the second placed by one saccade. The model keeps no memory, so only the second glimpse is classified, and only it is counted, as for DRAM and Saccader (note a); counting both would give 20%. Liu et al. [58] list the same result at 27%. STAM reports pixels rather than glimpses; its 27 glimpses are 27.7K pixels divided by its 32232^2 glimpse size. AME’s retina-like glimpses are counted after their peripheral down-scaling: eight 48248^2 glimpses on 128×256128 \times256 images deliver 18.75% of the pixels while covering 56.25% of the image area. AME’s supplementary states this setting for segmentation without naming the weights; we assume it for both variants, which share one table and the same retinal baselines. AdaGlimpse lists the run as full-resolution glimpses with 56.25%; we follow AME’s own materials. AdaptiveNN uses no fixed number of fixations but concludes observation adaptively once the evidence suffices, so its glimpse count and budget are averages, computed from the 3.15 “#Visual Fixations (average)” its paper reports at this operating point. Its paper states ImageNet-pretrained initialization only for its other tasks; for ImageNet it follows the full-length DeiT and ResNet training schedules, and its released code builds the model without pretrained weights, so we list it as trained from scratch. CanViT’s ADE20K result uses a coarse-to-fine viewing policy chosen at inference, a 64264^2 canvas instead of the trained 32232^2 and a frozen linear probe, at 21 glimpses; its ImageNet result is fine-tuned and peaks at 16 glimpses.

Our models. Our rows use the learned policy at the trained budget of four glimpses, averaged over three models trained with different random seeds. The budget counts the 4⋅6424 \cdot64^2 samples of the four glimpses against the 2242224^2 (ImageNet) or 5122512^2 (ADE20K) pixels of the image. The ADE20K mIoU of the table is scored at each image’s native resolution, the standard protocol; elsewhere segmentation numbers are scored at 512×512512 \times512 (Appendix § 9.§ 3), which is why the numbers in Table D.1 are not the same as the ones reported in Table 2.

Comparability. The numbers in Table D.1 are not comparable. The models differ at once in backbone, pretraining, input resolution, number and size of glimpses and evaluation protocol. Nor does a small pixel budget imply low computational cost: How much compute a model uses depends as much on the architecture that processes the pixels as on how many it samples. A fair comparison would need controlled benchmarks at matched input resolutions, pixel budgets and number of glimpses with clearly defined performance metrics. Beyond task performance and efficiency, comparisons of active vision models that aim to be biologically faithful may also benefit from the axes of alignment discussed in § 6.

Figure E.1. Unrolling beyond the training horizon. Every model is trained at T=4T = 4 (dotted rule) and unrolled to each budget at inference; nothing is retrained. One color per dataset, one line style per setting. Solid is each dataset’s own learned policy, dashed a model trained and evaluated with random locations, dash-dotted each seed’s learned policy transferred to that same seed’s random-location model, and dotted the learned-policy model evaluated with random locations. All four are measured by one protocol (last checkpoints; learned policies fixate the argmax of their probability map, random locations are drawn uniformly once per evaluation, so these points can differ slightly from the corresponding table cells in Table 2 and Table 3). At Teval=1T_{\mathrm{eval}}=1 the four curves collapse to two, since the first fixation is the fixed image center and no policy has acted yet. Shading is ±1\pm1 std over three models trained with different random seeds, measured on this protocol in every panel.

Unrolling beyond the training horizon

This appendix evaluates how models trained with four glimpses perform when they are unrolled for more glimpses at inference. The number of glimpses is fixed during training, but nothing in the architecture fixes it at inference. Because the memory is a spatial feature map rather than a vector, extra glimpses keep writing into the locations the policy selects, so the model can simply be unrolled for longer. Figure E.1 evaluates this: every model is trained with T=4T = 4 glimpses per episode and then unrolled to each budget TevalT_{\mathrm{eval}} with no retraining. Fixations were chosen via argmax of the probability map of the policy.

The figure shows each learned-policy model and each random-location model with its native policy, and two policy transfers (§ 5.2): each seed’s learned policy transferred to that same seed’s random-location model, and the learned-policy model evaluated with random locations.

With their own learned policy, the models on ImageNet, ADE20K and COCO-Stuff keep improving past the budget they were trained at, peak at Teval=8T_{\mathrm{eval}} = 8–1010 and then decay slightly, staying above their value at the trained budget up to Teval=20T_{\mathrm{eval}} = 20; C-MNIST is saturated from the trained budget on.

With more glimpses, random locations catch up: the margin of the learned policy over random locations narrows on every dataset. On the learned-policy model it stays positive over the whole range; transferred to the random-location model, it falls to about zero or below at the largest budgets on ADE20K and COCO-Stuff.

Visualizations

This appendix shows single episodes: what the model receives at each step (Appendix § 14.§ 1) and how its policy, memory and prediction develop (Appendix § 14.§ 2).

Selection of the examples. The examples are chosen, not drawn at random, to show how the task prediction can change across glimpses. For one learned-policy model per dataset (the one shown in Figure 1), we ran one episode on each image of a validation pool (C-MNIST: all 10 000, ImageNet: a random 5000, ADE20K: all 2000, COCO-Stuff: a random 3000). Each of several criteria keeps the episodes that meet a condition and ranks them by a score, and the twenty highest-ranked episodes of each criterion became candidates (up to 120 per classification dataset and 100 per segmentation dataset). For classification:

  1. wrong after the first glimpse and correct at the end, ranked by the rise in the ground-truth probability from the first to the last glimpse;

  2. the predicted class changes at the last glimpse, ranked by how much the ground-truth and final-class probabilities change at that glimpse;

  3. the final class differs from the first prediction and gains at least 40% of probability in one step, ranked by that gain;

  4. at least three different classes are predicted over the episode, ranked by their number;

  5. correct at some glimpse but wrong at the end, ranked by the drop in the ground-truth probability from its highest value before the last glimpse;

  6. never correct, with a final class that differs from the first, ranked by the rise in the final-class probability.

For segmentation:

  1. all episodes, ranked by the gain in per-image mIoUmIoU (plus half the gain in pixel accuracy) from the first to the last glimpse;

  2. final pixel accuracy higher than after the second glimpse, ranked by the largest share of pixels relabeled at a later glimpse;

  3. pixel accuracy reaches at least 30% at some glimpse, ranked by the share of pixels relabeled over the episode;

  4. images with at least 70% labeled pixels and at least three ground-truth classes, ranked by the final pixel accuracy;

  5. all episodes, ranked by the drop in pixel accuracy from its highest value before the last glimpse.

From the candidates, we picked four examples per dataset by hand, so not every criterion is represented. One of the four examples was always selected to be a failure. In the order of Appendix § 14.§ 2, the examples come from criteria 3, 2, 4 and 5 for ImageNet, 2, 2, 4 and 5 for C-MNIST, 4, 1, 1 and 5 for ADE20K and 3, 1, 4 and 5 for COCO-Stuff. The panels therefore show unusual episodes rather than typical ones. On average, the task prediction improves with every glimpse.

Visualization of the foveation

Figures F.1 to F.4 show what the model receives at each step for the first two examples of each dataset in Appendix § 14.§ 2 (same scanpaths, one column per glimpse). The rows are (1) the glimpse in log-polar coordinates, i.e. the backbone input (log-radius from left to right, so the fovea is the left edge; gray: outside the image); (2) the glimpse mapped back to image coordinates, for illustration only; (3) the best view so far: each pixel from the glimpse whose fixation is closest to it, which sampled it most finely, as the sample spacing grows with the distance from the fixation (Appendix § 9.§ 2); the outline marks where the current glimpse samples at least twice as finely as all earlier ones, a disk around the new fixation pushed away from the earlier ones; (4) the backbone features in log-polar coordinates; (5) the same features mapped into image coordinates, the input of the memory (§ 3.3). Rows (4) and (5) show the first three principal components of the features, one basis per row and example. The cyan cross marks the fixation.

Figure F.1. Foveation, ImageNet. Same examples and scanpaths as Figure F.5 (top: Japanese spaniel, bottom: paddlewheel). Rows as described in Appendix § 14.§ 1.

Figure F.2. Foveation, C-MNIST. Same examples and scanpaths as Figure F.6 (ground truth 5 and 4). Rows as described in Appendix § 14.§ 1.

Figure F.3. Foveation, ADE20K. Same examples and scanpaths as Figure F.7. Rows as described in Appendix § 14.§ 1.

Figure F.4. Foveation, COCO-Stuff. Same examples and scanpaths as Figure F.8. Rows as described in Appendix § 14.§ 1.

Visualization of policy, memory and prediction across glimpses

Each figure follows one episode: a validation example unrolled along a scanpath drawn from its policy, one column per step. The rows show the policy heatmap (as logits, since the probabilities are too peaked to show more than their maximum; the cyan cross marks the fixation), the memory features projected onto their first three principal components (one basis per example), the change of the memory ∣ΔH∣|\Delta H|, and the task prediction after that glimpse. Unlike the other rows, the heatmap in column tt is the probability map that chose ftf_t, computed at step t−1t-1; column t=1t=1 has none, since f1f_1 is the image center.

Memory updates. The ∣ΔH∣|\Delta H| row is ∣Ht−Ht−1∣|\mathbf{H}_t-\mathbf{H}_{t-1}| averaged over the 128 channels, on one color scale per example. At t=1t=1 the change covers the whole image, since H0=0\mathbf{H}_0=\mathbf{0}. In the later columns of the ImageNet and C-MNIST examples, the largest changes lie near the fixation of that glimpse: a disk around it on ImageNet (Figure F.5, top) and the characters close to it on C-MNIST (Figure F.6, top). On the segmentation examples, the later changes are faint compared with the first one on the shared color scale (Figures F.7 and F.8).

Classification. The task prediction is shown as the top-5 class probabilities, with the ground-truth class in green. Percentages refer to the four predictions in order.

Semantic segmentation. The task prediction is shown as the predicted map with its pixel accuracy, followed by a row that marks which pixels changed label relative to the previous step: green where a pixel became correct, orange where it became wrong, gray where it changed between two wrong labels. Percentages are rounded and refer to the four predictions in order.

Figure F.5. Policy, memory and prediction, ImageNet (T=4T = 4). Top: ground truth Japanese spaniel. The first two predictions name the related Blenheim spaniel (23, 84%); after the third glimpse the prediction switches to Japanese spaniel (92%) and keeps it (91%). Bottom: the predicted class alternates between flagpole (33%), paddlewheel (20%) and flagpole again (39%); the last glimpse raises the ground truth paddlewheel to 86%.

Figure F.5 (PDF p. 45). Policy, memory and prediction, ImageNet (continued). Top: four predictions, four classes, all at low confidence: coho (28%), sturgeon (13%), cowboy hat (9%), and the ground truth gondola (23%) only at the last glimpse. Bottom, failure: lion at 95% after the first glimpse loses probability at every step (32, 23, 2%) and grey fox is the predicted class from the second glimpse on (42, 56, 68%).

Figure F.6. Policy, memory and prediction, C-MNIST (T=4T = 4). Top: 22 for the first three predictions (52, 40, 55%) while the ground truth 55 stays at 9–14%; the last glimpse switches to 55 at 100%. Bottom: 99 with rising confidence (26, 46, 87%) while the ground truth 44 stays at 10–15%; the last glimpse switches to 44 at 100%.

Figure F.6 (PDF p. 47). Policy, memory and prediction, C-MNIST (continued). Top: 99 (64%), 11 (63%), 22 (51%), then the ground truth 99 again at 100%. Bottom, failure: 99 at 73% after the first glimpse; the three later fixations land on the same location and the prediction moves to 88 (96, 94, 85%), with the ground truth at 4–15%.

Figure F.7. Policy, memory and prediction, ADE20K (T=4T = 4). Top: pixel accuracy 97–99% at every step and per-image mIoU from 93% to 96%; later glimpses relabel at most 2% of the pixels. Bottom: pixel accuracy 80% after the first glimpse and 98% after the second, which relabels 30% of the pixels; per-image mIoU 12 →\to 40%.

Figure F.7 (PDF p. 49). Policy, memory and prediction, ADE20K (continued). Top: pixel accuracy 73, 85, 87, 86% and per-image mIoU 22→49%22 \to49\%, with less relabeling at each step (14, 5, 3% of the pixels). Bottom, failure: 93% pixel accuracy after the first glimpse; the next two glimpses relabel 8% and 13% of the pixels, mostly from correct to wrong, and accuracy falls to 88% and 76% (78% at the end).

Figure F.8. Policy, memory and prediction, COCO-Stuff (T=4T = 4). Top: 60, 49 and 63% of the pixels change label at the three later glimpses; pixel accuracy 7, 23, 28, 70%. Bottom: pixel accuracy changes little (95, 98, 98, 99%) while per-image mIoU rises from 33% to 82%.

Figure F.8 (PDF p. 51). Policy, memory and prediction, COCO-Stuff (continued). Top: pixel accuracy 82% after the first glimpse and 98% after the second, which relabels 16% of the pixels (98% at the end); per-image mIoU 27 →\to 55%. Bottom, failure: pixel accuracy above 99% after the first glimpse, then 41, 4 and 3%; the second and third glimpses relabel 59% and 37% of the pixels.

Evaluation policyImageNet Acc (%) learnedImageNet Acc (%) randomADE20K mIoU (%) learnedADE20K mIoU (%) randomCOCO-Stuff mIoU (%) learnedCOCO-Stuff mIoU (%) random
saliency64.5 ± 0.263.7 ± 0.2––––
saliency + IoR (r=12.5%r=12.5\%)66.7 ± 0.265.2 ± 0.2––––
saliency + IoR (r=25%r=25\%)66.7 ± 0.265.6 ± 0.2––––
entropy––21.9 ± 0.421.8 ± 0.223.2 ± 0.222.7 ± 0.1
entropy + IoR (r=12.5%r=12.5\%)––22.8 ± 0.322.3 ± 0.223.7 ± 0.123.1 ± 0.1
entropy + IoR (r=25%r=25\%)––22.9 ± 0.422.4 ± 0.323.9 ± 0.123.3 ± 0.2
memory saliency, relu(H\mathbf{H})62.5 ± 0.362.5 ± 0.620.3 ± 0.320.8 ± 0.421.2 ± 0.221.3 ± 0.1
memory saliency, relu(H\mathbf{H}) + IoR (r=12.5%r=12.5\%)64.2 ± 0.363.7 ± 0.420.7 ± 0.321.1 ± 0.421.9 ± 0.222.1 ± 0.2
memory saliency, ∣H∣|\mathbf{H}|62.8 ± 0.062.2 ± 0.620.3 ± 0.320.8 ± 0.421.2 ± 0.121.3 ± 0.1
memory saliency, ∣H∣|\mathbf{H}| + IoR (r=12.5%r=12.5\%)64.2 ± 0.263.4 ± 0.420.9 ± 0.221.0 ± 0.422.0 ± 0.222.1 ± 0.2

Table G.1. Further heuristic policies. Accuracy (ImageNet) and mIoU (ADE20K, COCO-Stuff) after T=4T=4 glimpses for the heuristic variants that Table 3 does not show. The IoR radius rr is given as a fraction of the image width (Appendix § 15). The r=25%r=25\% rows are the ones reported in Table 3, repeated here to compare the two IoR radii. Saliency is run on ImageNet only and entropy on the segmentation datasets only (Appendix § 15). Learned: learned-policy models, random: random-location models, each evaluated with the heuristic of the row. Mean ± std over three models trained with different random seeds, last checkpoints.

Heuristic fixation policies

Policy transfer (§ 5.2) compares learned policies with hand-designed ones; this appendix defines the heuristics. Table G.1 gives the variants that the main text does not report, after four glimpses, and Figures G.1 to G.3 show, for two validation images per dataset, the fixations of each heuristic and the map each choice was made from.

Common to all heuristics. The heuristics are used at evaluation only and replace the policy head; nothing is retrained, and they are applied to learned-policy and random-location models alike. The first fixation f1f_1 is the image center. After glimpse tt, a heuristic computes a score map MtM_t from the model’s own state, and the next fixation ft+1f_{t+1} is the center of the highest-scoring cell. All maps are computed from the memory features Ht∈R128×64×64\mathbf{H}_t \in\mathbb{R}^{128 \times64 \times64} after glimpse tt (Equation (2)), which are aligned with the image and are the input of every head.

Content-independent policies. These policies use no score map, so their fixations depend neither on the image nor on the model’s state. Random locations draws every fixation after f1f_1 independently and uniformly from the whole image. Triangle (T=4T=4) visits the three vertices of an equilateral triangle around the image center, each 32.5% of the image width from the center (the model sees square images, so width and height are equal): first the vertex below the center, then the upper left and the upper right one. Quadrant (T=5T=5) visits the centers of the four image quadrants, in the order top left, top right, bottom left, bottom right.

Saliency (ImageNet only). The saliency heuristic of the main text (Table 3) is the bottom-up map of Lukanov et al. [59]: the channel mean of the last convolutional feature map before global pooling. In our classification head this is the output Ut\mathbf{U}_t of its second convolution after the ReLU (256 × 32 × 32, Table A.2), so Mt=1256∑cUt,cM_t = \frac{1}{256}\sum_c U_{t,c} on a 32 × 32 grid. The map is non-negative due to the ReLU.

Memory saliency, relu(H) and ∣H∣|\mathbf{H}|. These variants take the same channel mean directly on the memory features Ht\mathbf{H}_t (64 × 64). Because Ht\mathbf{H}_t is signed, they treat negative activations in two ways: relu(H\mathbf{H}), Mt=1128∑cmax⁡(Ht,c,0)M_t = \frac{1}{128}\sum_c \max(H_{t,c},0), counts only positive activations, as after the ReLU of the original, and ∣H∣|\mathbf{H}|, Mt=1128∑c∣Ht,c∣M_t = \frac{1}{128}\sum_c |H_{t,c}|, counts both signs as activity.

Heuristic policies on ImageNet

Figure G.1. Heuristic policies on ImageNet. The learned-policy model evaluated with different policies on two validation images (left and right). Each row is one evaluation policy: the model’s own (native) policy, saliency, saliency with IoR at r=12.5%r = 12.5\% and at r=25%r = 25\%, and the two memory-saliency variants relu(H\mathbf{H}) and ∣H∣|\mathbf{H}|, both with IoR at r=12.5%r = 12.5\%. For each image, the first column shows the four fixations of the policy, numbered in order (1 is the initial center fixation). The next three columns show the map from which the next fixation was chosen after glimpses 1, 2 and 3, with brighter colors for higher values. In these maps, the cyan cross marks the chosen fixation (the argmax of the map, or in the native-policy row the fixation drawn from the probability map), white crosses mark earlier fixations, and dashed circles the regions excluded by IoR. The native-policy row shows the policy logits.

Entropy (ADE20K, COCO-Stuff). The entropy heuristic of the main text (Table 3) fixates where the current segmentation is least certain. The segmentation head is applied to Ht\mathbf{H}_t at 64×6464 \times64, before upsampling to the image size, and a softmax over classes gives pt(k∣x,y)p_t(k \mid x,y), the probability of class kk at cell (x,y)(x,y) after glimpse tt. The score of a cell is the entropy −∑kpt(k∣x,y)log⁡pt(k∣x,y)-\sum_k p_t(k \mid x,y)\log p_t(k \mid x,y) in nats, averaged over non-overlapping 4×44 \times4 blocks into 16×1616 \times16 candidate locations, and the next fixation is the center of the block with the highest score. The heuristic uses the model’s own prediction only, not the label. It is inspired by the entropy-guided policy of CanViT [7], which computes the entropy in the same way. The difference is how the next fixation is chosen: CanViT only considers tiles it has not visited yet and moves from large to small tiles over the episode, whereas we simply fixate the location with the highest entropy in the whole image, always with a glimpse of the same size.

Inhibition of return (+IoR). Every map can be combined with inhibition of return: cells whose center lies within radius rr of any earlier fixation, the initial center fixation included, are excluded from the argmax. We give rr as a fraction of the image width. Every map is run with r=12.5%r = 12.5\%. The best heuristic per task

Heuristic policies on ADE20K

Figure G.2. Heuristic policies on ADE20K. As in Figure G.1, with entropy in place of saliency in rows 2 to 4. The entropy map is shown on its 16×1616 \times16 block grid.

(saliency on ImageNet, entropy on ADE20K and COCO-Stuff) is also run with r=25%r = 25\%, and the main text reports these runs (Table 3).

Heuristic policies on COCO-Stuff

Figure G.3. Heuristic policies on COCO-Stuff, as in Figure G.2.

Properties of learned policies

Supporting analyses for § 5.3: the scanpath statistics for all three seeds (Appendix § 16.§ 1); each policy’s spatial fixation prior across seeds, image domains and steps (Appendix § 16.§ 2); the saccade-amplitude autocorrelation for different scanpath lengths (Appendix § 16.§ 3); the per-fixation policy comparison under two further metrics (Appendix § 16.§ 4); and inhibition of return split by visit index (Appendix § 16.§ 5).

Scanpath statistics across seeds

The main text reports scanpath statistics for one seed per dataset; Figures H.1 to H.4 show all three. On every dataset, the statistics of the three seeds are similar.

Figure H.1. Scanpath statistics across seeds, C-MNIST (10 000 validation images per seed; fixations drawn from the policy). Panels as in Figure 3; the three models trained with different random seeds share the dataset color and differ by line style (seed 1 solid, 2 dashed, 3 dotted) and marker. Thick gray line, in every panel: a uniform random policy (initial center fixation, then independent uniform fixations; simulated), the geometric reference for an image of this shape, the same simulation in every dataset’s figure and in Figure 3. The yy-axes are scaled per dataset, so the same gray line looks flatter where the models’ curves are more peaked (e.g. the direction row of ADE20K against that of ImageNet).

Figure H.2. Scanpath statistics across seeds, ImageNet (10 000 validation images per seed; fixations drawn from the policy). Panels, line styles and the random-policy reference as in Figure H.1.

Figure H.3. Scanpath statistics across seeds, ADE20K (2000 validation images per seed; fixations drawn from the policy). Panels, line styles and the random-policy reference as in Figure H.1.

Figure H.4. Scanpath statistics across seeds, COCO-Stuff (5000 validation images per seed; fixations drawn from the policy). Panels, line styles and the random-policy reference as in Figure H.1.

Cumulative fixation distributions across seeds for C-MNIST, ImageNet, ADE20K, and COCO-Stuff; rows show seed 1, seed 2, and seed 3

Figure H.5. Cumulative fixation distribution across seeds. Figure 1B for the three seeds (rows) of our default models; seed 1 is the model shown in Figure 1. Each panel is the mean probability map over the validation pool, with fixations drawn from the policy, averaged over the maps after glimpses 1 to 3, which choose fixations 2 to 4 (the initial center fixation is fixed and therefore left out, as is the map after the fourth glimpse). Colors are clipped per panel at the 99.5th percentile. Images per seed: C-MNIST 10 000, ImageNet 10 000, ADE20K 2000, COCO-Stuff 5000.

More detailed analyses on cumulative fixation distributions

Figure 1B in the main paper shows each policy’s cumulative fixation prior on its own dataset. Figure H.5 repeats it for all three seeds. Figure H.6 applies each policy to each image domain, policy along the rows, image domain along the columns, so the fixation distribution of one policy can be compared across image domains, and the distributions of different policies on the same images. Figure H.7 splits the prior of Figure 1B by step, including human observers.

Cumulative fixation distributions across ImageNet, ADE20K, and COCO-Stuff image domains for ImageNet, ADE20K, and COCO-Stuff policies

Figure H.6. Cumulative fixation distributions with the policy moved across image domains. Each cell is the mean probability map over the image pool, with fixations drawn from the policy, averaged over the maps after glimpses 1 to 3, which choose fixations 2 to 4 (the initial center fixation is fixed and therefore left out, as is the map after the fourth glimpse), i.e. the quantity Figure 1B plots per dataset. Rows fix the policy (the learned-policy models shown in Figure 1), columns fix the image domain; images per column: ImageNet 10 000, ADE20K 2000, COCO-Stuff 5000. Each model sees its own training resolution and all priors are compared in the shared 64×6464 \times64 memory space. The diagonal (colored frame) is each policy on its own training domain.

Fixation distributions at each step for models and human observers across datasets

Figure H.7. Fixation distribution at each step, for the models and human observers. Figure 1B, split by step: one row per dataset (top four: models; bottom: human observers), one column per step from t=1t = 1 to t=6t = 6, and the cumulative map. The models are trained with T=4T = 4 and unrolled to Teval=7T_{\mathrm{eval}} = 7 glimpses with fixations drawn from the policy: after the initial center fixation f1f_1, the map at step tt chooses ft+1f_{t+1}, so the six steps place f2f_2 to f7f_7. Each map is the mean of the per-step distributions over the validation images (C-MNIST and ImageNet 10 000, ADE20K 2000, COCO-Stuff 5000); the color scale is shared within a row and clipped at the 99.5th percentile. The cumulative column averages all six steps (t=1t = 1 to 66), more than the cumulative map of Figure 1B (t=1t = 1 to 33). Bottom row: MIT1003 free viewing; column tt is the density of the fixation ft+1f_{t+1}, as a 64×6464 \times64 histogram blurred by one cell, and nn is the number of fixations per panel.

Saccade-amplitude autocorrelation by scanpath length

The autocorrelation plot of Figure 3C standardizes the saccade amplitudes within each scanpath, which makes the results dependent on the scanpath length (see next paragraph). Standardization is still necessary in depicting the relationship to account for differences in saccade amplitudes across images. Figure H.8 shows the same plot as in Figure 3C for different scanpath lengths. For all plots, the mean amplitude of the next saccade is shown as a function of the preceding one, both zz-standardized within their scanpath. The last panel is identical to Figure 3C. It can be clearly seen that the slopes of the curves depend on the scanpath length, which is why the uniformly random policy is needed as a baseline to interpret the results. An intuition of this dependence is illustrated below.

The curve of the random policy slopes downward, more steeply for short scanpaths. Its slope reflects the correlation between consecutive amplitudes after they are standardized within each scanpath. This correlation results from two opposing effects: (1) Consecutive saccades share a fixation, so even when fixations are placed at uniformly random locations, their raw amplitudes are positively correlated at roughly +0.11+0.11. (2) Standardizing within a scanpath works against this: subtracting the scanpath’s own mean makes its deviations sum to zero, so a saccade above that mean has to be balanced by others below it. For nn independent amplitudes this gives every pair of deviations a correlation of −1/(n−1)-1/(n - 1), i.e. −0.5-0.5 for three saccades and −0.14-0.14 for eight, and the bias vanishes as nn grows. For short scanpaths it dominates: under the random policy the correlation of consecutive standardized amplitudes is −0.48-0.48 with three saccades, −0.19-0.19 with five and −0.06-0.06 with eight. It crosses zero at about twelve saccades and approaches the raw +0.11+0.11 for long scanpaths (+0.10+0.10 with 100). Part of the downward slope of every curve therefore comes from the standardization itself, which is why the random policy at each scanpath length is needed for comparison.

Figure H.8. Autocorrelation of saccade amplitudes. Mean amplitude of the next saccade as a function of the preceding one, both z-standardized within each scanpath (own mean subtracted, divided by own std), smoothed with a Gaussian kernel (bandwidth 0.15) and drawn over the range that lies within the 2nd to 98th percentile of the preceding amplitude for every source in the panel. Each panel uses the fixations f2f_2 to fk+1f_{k+1} for one k=4,…,9k = 4, \ldots, 9, i.e. k−1k - 1 saccades. The initial saccade from the center f1f_1 to the next fixation f2f_2 is excluded since its starting position is fixed. For MIT1003, f2f_2 to fk+1f_{k+1} are the first kk fixations after the initial center fixation, and each panel uses all scanpaths with at least kk of them (counts in the panel titles). One learned-policy model per dataset (the one shown in Figure 1, trained with four glimpses) is unrolled for nine fixations (f2f_2 to f10f_{10}), drawn from the policy after the initial center fixation, and cut to f2f_2 to fk+1f_{k+1}. Thick gray line: uniform random policy.

Figure H.9. Per-fixation policy alignment: near hits. Fraction of images on which the two policies’ argmax fixations lie within 5% of the image width of each other (about three cells of the 64×6464 \times64 probability map; higher = more aligned). It thresholds the per-image distances that Figure 4 averages. On every dataset and at every fixation, the two seeds of the same task give the most near hits, the other tasks’ policies fewer and uniform fixations the fewest. Layout as Figure 4: every model is unrolled for nine fixations (f2f_2–f10f_{10}) after the initial center fixation; the dotted line marks the end of the T=4T = 4 glimpses used in training. Shading: ±1 std over three model pairs, each averaged over the validation images (ImageNet 10 000, ADE20K 2000, COCO-Stuff 5000).

Figure H.10. Per-fixation policy alignment: probability-map correlation. Correlation between the two policies’ next-fixation probability maps, each first averaged over the images (higher = more aligned): the least localized of the three metrics, and a general impression of where each policy puts its mass rather than of where its peak falls for a specific image. Layout and shading as in Figure H.9.

Additional per-fixation policy comparisons

Figures H.9 and H.10 compare the policies under the protocol of Figure 4 with two further similarity metrics. Near hits (Figure H.9) take the same per-image argmax distances as Figure 4 but only count whether the two fixations lie within 5% of the image width of each other. The probability-map correlation (Figure H.10) compares the policies globally: it first averages probability maps across all images, and then correlates these averaged probability maps, so it reflects where each policy puts its mass rather than where it fixates on a given image.

Figure H.11. Inhibition of return, split by visit index. The quantity of Figure 5 with one row per visited fixation fkf_k (k=2,…,10k = 2, \ldots, 10), one panel per dataset and a shared color scale. Fixations are drawn from the policy. Values are means over the three seeds of each of our default models. The black vertical line separates the steps before the visit (negative lags) from the visit and the steps after it.

Figure H.12. Inhibition of return on C-MNIST, shown on its own because its values are an order of magnitude larger: its probability maps put nearly all their mass on a single cell. The policy logit zz at the location of f2f_2, the first fixation after the initial center fixation, tends to drop sharply at the visit (z=20.9→15.4z = 20.9 \to15.4) and stays low afterwards (seen in panel B). From f3f_3 on, the policy tends to keep returning to the same location, and the logit at the visited cell drops by at most 1.7 (seen in panel B), so the marginal (seen in panel A) shows little suppression. This reflects the fact that the policy has a tendency to keep looking at the digit once it found it. Inhibition of return therefore does not emerge in the same way in this setting where discriminatory information is highly localized. Values are means over the three seeds of each of our default models. In (A), thin lines are the individual seeds. The black vertical line separates the steps before the visit (negative lags) from the visit and the steps after it.

Inhibition of return, split by visit index

The main paper reports this effect marginally, averaged over visits. Figure H.11 shows the same quantity with one row per visit index: the drop at the visit and the partial recovery afterwards appear in every row. Figure H.12 shows C-MNIST separately.

ChoiceDefaultAlternativeAnalyzed in
Policy gradientstop-gradient on Ht\mathrm{H}_t ((8))full gradientAppendix J
Entropy regularizationnoneentropy target (H⋆=3.5\mathcal{H}^{\star}=3.5 nats)Appendix J.5
Task supervisionmean per-step CE ((5))last step onlyAppendix K.1
Rewardrt=CEt−CEt+1r_t = \mathrm{CE}_t - \mathrm{CE}_{t+1} ((13))−CET-\mathrm{CE}_T for every fixationAppendix K.2

Table I.1. Alternatives to the training recipe investigated in Appendices J and K. For each choice of the training recipe, the table gives the setting used in our default models, the alternative we compare it with, and the subsection that reports the results. CEt=LCE(y^t,y)\mathrm{CE}_t = \mathcal{L}_{\mathrm{CE}}(\hat{y}_t,y) is the task loss after glimpse tt.

Overview of the training-recipe ablations

Appendices J and K examine alternatives to choices of the training recipe that the main text presents. Appendix I.1 lists the choices that these appendices vary on purpose, each with the setting of our default models and the alternative examined. Appendix I.2 lists all the ways in which a run in Appendices J and K can differ from our default models. Besides the choices varied on purpose, these include differences due to the design of an analysis or to saving compute, for example by using runs from earlier stages of model development.

The choices examined

Our default models, i.e. the models of the main results (§ 5.1 to § 5.3), are trained and evaluated as described in § 3.6 and Appendix § 9.§ 3. This main protocol is summarized at the beginning of Appendix § 17.§ 2. Appendices J and K vary four main choices of the training recipe of the main protocol, listed in Table I.Table 1: (1) whether the policy gradient reaches backbone and memory, (2) whether entropy regularization is added, (3) how the task is supervised, and (4) how a fixation is rewarded. For each choice, the table gives the setting of our default models, the alternative we examine, and the appendix in which the alternative is analyzed. By default, the stop-gradient keeps the policy gradient out of backbone and memory, since letting it through was shown to be detrimental in § 5.4. Appendix § 18 examines this choice in more detail, together with entropy regularization (choices (1) and (2)). As further remedies without the stop-gradient, Appendix § 18.§ 5 also tests a smaller policy-loss weight and a smaller policy head. Appendix § 19 compares the task supervision and the reward with their end-of-episode alternatives (choices (3) and (4)).

Deviations from the main protocol

The main protocol is how our default models are trained and evaluated. Training uses the recipe of § 3.6 with the hyperparameters of Table A.3: the task loss at every step ((5)), the per-step reward of (13) without a baseline, its standardization per step ((14)), no entropy regularization, and the stop-gradient on the memory features ((8)). Evaluation uses the last checkpoint with argmax fixations (Appendix § 9.§ 3).

Deviations. The runs in Appendices J and K differ from the main protocol for three reasons: (1) The alternative under investigation, such as the policy gradient on backbone and memory. (2) The design of an analysis, such as scoring policies on a common frozen model. (3) Saving compute: for example, some runs are trained only as long as needed to show a difference, or some runs come from earlier stages of model development (where the protocol differed) and were not rerun under the main protocol to save compute. Table I.Table 2 lists all deviations.

The main protocol is how our default models …

DeviationDescription
D1Terminal rewardEvery fixation receives the same reward, −CE-\mathrm{CE} at the last step, instead of its own per-step improvement (Equation (6))
D2Terminal supervisionThe task loss is computed only on the prediction after the last glimpse, instead of after every glimpse (Equation (5))
D3Level normalizerThe reward is divided by its mean absolute value, pooled over batch and step, instead of being standardized per step (Equation (7))
D4Gradient on backbone and memoryThe policy gradient reaches backbone and memory, instead of stopping at the memory features (stop-gradient in Equation (8))
D5Frozen representationOnly the policy head trains, on a frozen backbone, memory and task head, instead of all parameters; accuracy then depends on the fixations alone
D6Early epochEvaluated at a matched epoch before convergence, instead of at the end of the full schedule (Table A.3)
D7Best checkpointThe maximum over validation evaluations, instead of the last checkpoint (Appendix § 9.§ 3)
D8Baseline presentA running-mean or learned value baseline is subtracted from the reward, instead of no baseline
D9Different pretrained modelsThe compared models are finetuned from different pretrained models, instead of one common starting point, so part of a difference between them can come from these starting points rather than from the setting being compared
D10Policy-transfer scoreA common frozen receiving model’s accuracy under externally supplied fixations, instead of the model’s own metric
D11Reduced budgetFewer GPUs, a 24 h wall clock or a truncated schedule, instead of the setup of Table A.3
D12Logged validation scoreThe validation score logged during training, with fixations drawn from the policy, instead of an evaluation with argmax fixations (Appendix § 9.§ 3)
D13Different learning-rate scheduleA constant learning rate, or one halved whenever the validation score stops improving, instead of the schedule of Table A.3
D14Entropy regularizationEntropy regularization −λHHˉ-\lambda_H\bar{H} with a fixed weight λH=5×10−3\lambda_H = 5 \times10^{-3} is added to the loss, instead of none

Table I.2. Numbered deviations from the main protocol. Every figure and table caption in Appendices § 18 and § 19 lists the numbers of the deviations that apply to its runs. Investigated settings without a number (the small policy head, the entropy target and the policy-loss weight λ\lambda) are named in the captions themselves. A result given only in the text either belongs to the runs of a cited figure or table, or lists its deviations in a footnote.

C-MNIST Acc (%)ImageNet Acc (%)ADE20K mIoU (%)COCO-Stuff mIoU (%)
with stop-grad99.3 ± 0.061.624.3 ± 0.224.9 ± 0.2
w/o stop-grad98.8 ± 0.125.518.1 ± 0.817.9 ± 0.5
w/o stop-grad + small head95.161.423.124.1
w/o stop-grad + entropy99.361.923.823.5

Table J.1. Task performance with the policy gradient on backbone and memory. With stop-grad (reference): the policy gradient stops at the memory features (our default models). w/o stop-grad: no stop-gradient. w/o stop-grad + small head, w/o stop-grad + entropy: no stop-gradient, with one of the two variants of Appendix § 18.§ 5 (a 129-parameter policy head or an entropy target of 3.5 nats). On C-MNIST, ADE20K and COCO-Stuff, with stop-grad are our default models. On ImageNet, with stop-grad is our default ImageNet model after 75 of its 150 epochs, and the other rows are trained only for these 75 epochs. ±: std over seeds where several exist. Within each column the runs share epochs, learning-rate schedule, batch size and initialization and differ only in the setting named in each row. Deviations (Table I.Table 2): gradient on backbone and memory (D4; all rows except with stop-grad), reduced budget (D11; ImageNet), logged validation score (D12).

The policy gradient on backbone and memory

Our default models keep the policy gradient out of backbone and memory with a stop-gradient on the memory features (Equation (8)). § 5.4 then ablates this stop-gradient operation. This appendix adds additional information to this ablation: it shows what goes wrong when the policy gradient reaches backbone and memory, and which mitigations appear to help and which appear not to. Appendix § 18.§ 1 reports the drop in task performance without the stop-gradient. Appendix § 18.§ 2 finds that worse fixations, gradient-norm clipping and the AdamW optimizer state are unlikely to explain this drop, which suggests that the policy gradient on backbone and memory itself causes it. Appendix § 18.§ 3 follows how this gradient changes once the policy sharpens (meaning its logits grow so large that it fixates almost deterministically, though still at image-dependent locations). Appendix § 18.§ 4 describes a separate problem of the runs trained in bf16. Appendix § 18.§ 5 tests potential remedies: a smaller policy-loss weight λ\lambda, a small policy head and an entropy target. Appendix § 18.§ 6 summarizes the findings and contextualizes our solution of using stop-gradient to prevent optimization issues. Appendix § 18.7 derives the bound used in Appendix § 18.§ 3.

The central comparison of this appendix (Table J.1) is between runs with and without the stop-gradient that are otherwise trained alike. On C-MNIST, ADE20K and COCO-Stuff, the runs with stop-gradient are our default models. On ImageNet, the run with stop-gradient is our default ImageNet model after 75 of its 150 epochs, before its learning rate is first halved, so it reaches a lower accuracy than in Table 2. To save compute, the runs without stop-gradient are trained only for these 75 epochs. Later subsections also use other runs. How these differ from our default models is listed in the captions of the figures and tables that show them, or in a footnote for runs reported only in the text (Table I.Table 2).

Without the stop-gradient, task performance drops

Without the stop-gradient (w/o stop-grad), task performance drops on all four datasets (Table J.1): accuracy by 36.1% on ImageNet and 0.5% on C-MNIST, and mIoU by 6.2 on ADE20K and 7.0 on COCO-Stuff. In the models without the stop-gradient, the policy-loss weight λ\lambda is 0.1 by default. The analyses of the following subsections cover ImageNet, ADE20K and COCO-Stuff, where the drop is large.

Worse fixations, clipping and the optimizer state are unlikely to explain the drop in task performance

Without the stop-gradient, the runs of Table J.1 differ from those with it in one respect: the policy gradient also reaches backbone and memory. The drop in task performance can therefore come from this gradient directly, or from what changes it induces: the fixations the policy chooses, how often gradient-norm clipping acts, and the AdamW optimizer state. Whether worse fixations explain the drop is checked with the accuracy after the first glimpse and with the policy transfers of Table J.2; whether clipping or the optimizer state explains it, with statistics logged during training.

C-MNISTImageNetADE20KCOCO-Stuff
Acc (%)Δ\DeltaAcc (%)Δ\DeltamIoU (%)Δ\DeltamIoU (%)Δ\Delta
random locations90.5–63.6–22.2–23.0–
with stop-grad98.3+7.866.0+2.523.0+0.823.9+0.8
w/o stop-grad98.3+7.864.9+1.423.2+1.024.1+1.1
w/o stop-grad + small head97.7+7.266.3+2.823.0+0.824.3+1.3
w/o stop-grad + entropy98.4+7.966.2+2.723.3+1.124.3+1.3

Table J.2. The policies of Table J.1, transferred to one random-location model. In Table J.1 each row changes both the policy and backbone and memory, so a low cell there can come from either. Here the receiving model is frozen: in every row, the dataset’s default random-location model is evaluated on its validation set, and only the fixations f2f_2–f4f_4 it is given change (f1f_1 is always the image center). Row 1 is that model under the uniform-random policy it was trained with, so each Δ\Delta is the value of the policy alone. Deviations (Table I.Table 2): gradient on backbone and memory (D4; the policies without stop-gradient), policy-transfer score (D10), reduced budget (D11; the ImageNet policies).

Worse fixations do not play a major role. In all our models, which locations are fixated changes task performance by a few points: our default ImageNet model, for example, loses 5.2% when it is evaluated with random locations instead of its own policy (Table 3). Even this drastic change of the fixations costs far less than the 36.1% of accuracy that the ImageNet model loses without the stop-gradient, which suggests that the drop is not mainly the result of worse fixations. Also, after the first glimpse, which is always at the image center and so does not depend on the policy, the ImageNet model without stop-gradient of Table J.1 reaches only 21.4% accuracy against 52.1% for the model with stop-gradient. Transferred to a frozen random-location model, the policy trained without stop-gradient improves on random locations about as much as the policy trained with stop-gradient on three datasets, and 1.1% less on ImageNet (1.4 against 2.5%; Table J.2). Most of the 36.1% accuracy drop on ImageNet therefore cannot be the result of worse fixations.

Clipping and optimizer state are also unlikely to affect task performance substantially. Large policy gradients could slow learning through two side effects, without having a negative effect on backbone and memory per se. (1) Gradient-norm clipping (Table A.3) scales down the whole update, including the task part, whenever the gradient norm exceeds its threshold. (2) AdamW divides each step by the square root of a running average of the squared gradients. This average forgets slowly, so a single large gradient keeps it high for many steps and makes the following steps smaller, including those of the task gradient. This might influence optimization since we show that the policy gradient on backbone and memory arrives in rare, large bursts without the stop-gradient (Appendix § 18.§ 3). However, we find that neither of these factors explains the drop in task performance. As for (1), clipping cannot have started the drop: on ImageNet, the run without stop-gradient of Table J.1 is already behind at epoch 4 in the validation score logged during training (15.3% against 26.9%), before clipping appears in the logged minibatches (one per epoch) at epoch 5. On ADE20K, a run without stop-gradient with a 10×\times smaller policy-loss weight (λ=0.01\lambda= 0.01 instead of the default 0.1; Figure J.1) falls behind the run with stop-gradient after about eight epochs, although clipping acts on none of its steps up to epoch 9 and on at most 2% afterwards. Switching clipping off does not help either: in an earlier ablation on ImageNet (single runs, stopped after about 16 epochs), two otherwise identical runs, both without stop-gradient, reached 23.1% without clipping and 24.4% with it, against 49.0% for the run with stop-gradient in the same set at the same epoch.6 As for (2), smaller AdamW steps would act like a lower learning rate, so a run that falls behind should show a clearly larger square root of the running average, by which each step is divided, than runs that keep learning. On backbone and memory this square root is within about 1.5×1.5\times of that of the run with stop-gradient in the λ=0.01\lambda= 0.01 run, which falls behind, and within about 2×2\times in the two variants of Appendix § 18.§ 5, which keep learning (compared at matched checkpoints within the first 37 epochs). Smaller AdamW steps therefore do not seem to explain the drop in task performance either.

Figure J.1. Training variants on ADE20K. Validation mIoU of the run with stop-gradient, a run without stop-gradient at policy-loss weight λ=0.1\lambda= 0.1 and one at λ=0.01\lambda= 0.01, otherwise identical. First 65 epochs; single runs. Deviations (Table I.Table 2): gradient on backbone and memory (D4; models without stop-gradient), reduced budget (D11; the run at λ=0.01\lambda= 0.01 ends at epoch 64), logged validation score (D12).

The policy gradient arrives in rare, large bursts once the policy sharpens

Appendix J.2 suggests that the policy gradient on backbone and memory causes the drop in task performance. This subsection describes how this gradient develops during training without the stop-gradient. It first gives an upper bound on the size of this gradient (derived in Appendix J.7). Two factors of the bound can change during training: the sharpness of the policy (how close it comes to fixating one location per image with certainty) and the norm of the policy head’s Jacobian. For a sharp policy, the bound predicts that most samples send almost no gradient and a few send a large one; the Jacobian norm sets how large these few gradients can be. The subsection then follows both factors through training and shows how the gradient is distributed over the samples.

What the bound predicts. For one fixation drawn from the policy (one image and one decision step), the policy gradient at the memory features Ht\mathbf{H}_t has norm at most λB∣r~t∣∥Jt∥2(1−πf)\frac{\lambda}{B} \lvert\tilde{r}_t \rvert\lVert\mathbf{J}_t \rVert\sqrt{2}(1-\pi_f) (Equation (12), derived in Appendix J.7). Here λ\lambda is the weight of the policy loss, BB the batch size, r~t\tilde{r}_t the standardized reward, Jt\mathbf{J}_t the Jacobian of the policy logits with respect to Ht\mathbf{H}_t, and πf\pi_f the probability of the drawn fixation ft+1f_{t+1}, one cell of the 64×6464 \times64 grid. Since ∣r~t∣≤n−1\lvert\tilde{r}_t \rvert\le\sqrt{n-1} for the nn samples standardized together and 1−πf≤11-\pi_f \le1, only ∥Jt∥\lVert\mathbf{J}_t \rVert can grow without limit. We call it the Jacobian norm of the policy head: it sets how strongly the head amplifies a change of the memory features into a change of the logits, and equally how strongly it amplifies the policy gradient on its way back into the memory features. For a sharp policy, the most likely cell has a probability close to 11: if it is drawn, 1−πf≈01-\pi_f \approx0 and the gradient is almost zero, and only the rare samples of another cell give 1−πf≈11-\pi_f \approx1 and a gradient whose size is set by the Jacobian norm. Once the policy has sharpened, the bound thus predicts that the gradient into the memory arrives in rare, large bursts.

The policy sharpens and the Jacobian norm of its head grows. Without the stop-gradient, the policies sharpen within a few epochs: their entropy falls to 0.010.01–0.20.2 nats, also in the ADE20K run with a 10×10\times smaller policy-loss weight (λ=0.01\lambda= 0.01; Figure J.2A). The range of their logits (largest minus smallest logit of a probability map) is much larger than with the stop-gradient and by far the largest on ImageNet (Table J.3), whose runs, unlike those on ADE20K and COCO-Stuff, are trained in bf16 (Appendix § 18.§ 4). The Jacobian

logged validation score (D12), different learning-rate schedule (D13), entropy regularization (D14). In addition, these runs use an earlier version of the backbone and a policy head with a single 1×11 \times1 convolution. norm of the policy head also grows, both with and without the stop-gradient, but far more without it: on ImageNet, it rises between the first and the last epoch from 137 to 22 982 without the stop-gradient (about 170-fold) and from 50 to 1063 with it (about 21-fold; Table J.3).

w/o stop-gradwith stop-grad
Logit range
ImageNet82 946360
ADE20K1802240
COCO-Stuff3184616
Jacobian norm of the policy head, ImageNet
first epoch13750
last epoch22 9821063

Table J.3. Logit range and Jacobian norm of the policy head, without and with stop-gradient. Logit range: largest minus smallest policy logit of one probability map, median over 500 validation images and the three decision steps, final checkpoints. Jacobian norm: ∥Jt∥\lVert J_t\rVert, the largest singular value of the Jacobian of the 64 ×\times 64 policy logits with respect to the memory features (128 ×\times 64 ×\times 64), computed by power iteration without forming the matrix; median over 32 validation images and the three decision steps, at the first and the last epoch; ImageNet only. Models as in Table J.1. Deviations (Table I.Table 2): gradient on backbone and memory (D4; models without stop-gradient), reduced budget (D11; ImageNet).

The gradient arrives in bursts. As the bound predicts, once the policies without stop-gradient have sharpened, the median per-sample gradient into the memory is essentially zero: on most probability maps one cell holds almost all of the probability, so 1−πf≈01-\pi_f \approx0 whichever cell is drawn. Almost all of the gradient comes from the few maps (2–6%) on which the policy is still undecided, mostly between two cells. There, the less likely cell is sometimes drawn despite its low probability, and such a sample has a large 1−πf1-\pi_f and therefore a large gradient. These samples are rare (0.2–1.4% of all samples) but carry 34–55% of the summed gradient (checkpoints of Figure J.3).

Per minibatch, the policy gradient on backbone and memory of the models without stop-gradient is either negligible or many times the task gradient there: the ratio of the two norms is below 0.1 on 30–50% of the minibatches and above 10 on 2–18%. On the most extreme minibatch, the policy gradient is 140 times the task gradient. The ratio is computed as in Figure 6B but on 40 training minibatches at the checkpoints of Figure J.3. Its direction is unrelated to that of the task gradient: the cosine between the two has a median of −0.064 to 0.004 over training minibatches on the final checkpoints of Table J.1.

A separate problem in bf16: rounding ties the largest logits

The classification runs (ImageNet, C-MNIST) are trained in bf16. On ImageNet, where it was measured, this adds a second problem to the bursts of Appendix J.3, caused by rounding of the largest policy logits. bf16 stores a logit zz with ∣z∣∈[2e,2e+1)\lvert z\rvert\in[2^e,2^{e+1}), for an integer exponent ee, in steps of 2e−72^{e-7} (256 at ∣z∣≈4×104\lvert z\rvert\approx4 \times10^4). With logits as large as those of the models without stop-gradient (Table J.3), the kk largest logits can round to the same value. Then 1−πf≈(k−1)/k1-\pi_f \approx(k-1)/k on every sample from such a map, and the gradient of each of these samples, not only of the rare ones of Appendix J.3, grows with the Jacobian norm ∥Jt∥\lVert J_t\rVert.

On the checkpoints of the ImageNet run without stop-gradient of Table J.1, we compute the gradients once in fp32 and once in bf16. The ratio qq of the policy to the task gradient norm on backbone and memory agrees until epoch 1 (1.3 vs. 1.2) and then separates. In fp32 it falls below 0.01 from epoch 19, as the policy sharpens. In bf16 it rises to 61, with about 10–25% of the probability maps having a tied maximum. In training, which ran in bf16, gradient-norm clipping acted on every logged minibatch from epoch 5 on (71 of 75; one minibatch is logged per epoch).

fp32 does not have this problem. On ADE20K and COCO-Stuff, which are trained in fp32, we compute the per-sample policy gradient into the memory features once in fp32 and once in fp64, on 12 and 11 checkpoints spread over training of the runs without stop-gradient of Table J.1 (16 validation images, 8 fixations drawn per probability map). The two agree within 2% on 16 of the 23 checkpoints and within about a factor of two on all, with no trend over training. The runs without stop-gradient on these datasets lose task performance all the same (Table J.1). bf16 thus enlarges the policy gradient on ImageNet but does not alone cause the drop in task performance; computing the policy logits in fp32 avoids the ties.

Figure J.2. Policy entropy and the entropy weight during training. Models as in Table J.1 on ImageNet, ADE20K and COCO-Stuff (one model each); on ADE20K also the run without stop-gradient with a 10×10 \times smaller policy-loss weight (λ=0.01\lambda= 0.01) of Figure J.1. (A) policy entropy (dashed: uniform policy, log⁡4096≈8.3\log4096 \approx8.3 nats; dotted: the entropy target of 3.5 nats of the entropy-target variant). (B) the entropy weight λH\lambda_H set by the update rule of the entropy-target variant (dotted: the bounds 10−610^{-6} and 10−110^{-1} within which it is kept); in the second half of training it is about 1.6×10−41.6 \times10^{-4} on ImageNet, 1.0×10−31.0 \times10^{-3} on ADE20K and 3.6×10−53.6 \times10^{-5} on COCO-Stuff. One logged minibatch per epoch; bold lines are rolling medians (window a tenth of the run, at least nine epochs), faint lines the raw values. Deviations (Table I.Table 2): gradient on backbone and memory (D4; all models except the one with stop-gradient), reduced budget (D11; ImageNet and the run at λ=0.01\lambda= 0.01).

Potential remedies

This subsection tests potential remedies for the drop in task performance without the stop-gradient. The first scales down the policy gradient on backbone and memory with a 10×10 \times smaller policy-loss weight λ\lambda. The other two are variants that act on the two properties that shape this gradient (Appendix § 18.§ 3): the Jacobian norm ∥Jt∥\lVert J_t\rVert of the policy head (the norm of the Jacobian of the policy logits with respect to the memory features), which sets how large a burst can be, and the sharpness of the policy, which makes the gradient arrive in bursts. A small policy head limits the Jacobian norm, and with fewer weights its policy may also sharpen less; an entropy target limits the sharpening directly. Each of the three changes only this one thing in the runs without stop-gradient of Table J.1. We compare their task performance, the value of their policies and how their policy gradient reaches backbone and memory with those of the models with and without stop-gradient.

Smaller policy-loss weight. The run without stop-gradient is repeated on ADE20K with a 10×10 \times smaller policy-loss weight, λ=0.01\lambda= 0.01 instead of 0.10.1, and otherwise identically (single run). The policy head receives only the policy gradient, and AdamW normalizes each parameter’s step by its own gradient scale, so its steps stay about the same. Backbone and memory also receive the task gradient, so for them the smaller λ\lambda scales down the policy gradient against the task gradient: it acts as a partial stop-gradient.

Figure J.3. Bursts of the policy gradient. Distribution of the per-sample gradient norm ∥∂log⁡πθ(ft+1∣Ht)/∂Ht∥\lVert\partial\log\pi_\theta(f_{t+1} \mid H_t)/\partial H_t \rVert into the memory features, for the model without stop-gradient and the two variants of Appendix J.5, as a kernel density estimate over log⁡10\log_{10} of the norm. One sample is one fixation drawn from one map (8 per map; 64 ImageNet or 32 segmentation validation images), computed in fp32. Samples with a gradient of exactly zero cannot be placed on the log axis and are left out of the curves, so each curve integrates to the share of samples with a nonzero gradient. Each panel prints the share of zeros for the model without stop-gradient (e.g. 70% on ImageNet). Epochs as in the panel titles; single runs. Deviations (Table I.Table 2): gradient on backbone and memory (D4), early epoch (D6; the epochs in the panel titles), reduced budget (D11; ImageNet).

Small head. The policy head (Table A.2) is replaced by a single 1×11 \times1 convolution from the 128-channel memory to one logit per cell (128 weights ww and a bias, 129 parameters). Its Jacobian norm ∥Jt∥\lVert J_t \rVert is then at most the norm ∥w∥\lVert w \rVert of these weights. It can then grow only with the norm of these 128 weights, not with a product over four layers as in the default head. The head was sized so that its policy reaches about the entropy that the entropy target condition (below) enforces (2.0–3.5 nats, Figure J.2A), so the small head limits the sharpening as well.

Entropy target. The policy head is kept, and entropy regularization (§ 5.4), −λHHˉ-\lambda_H\bar{H}, is added to the loss, with Hˉ\bar{H} the policy entropy averaged over the batch and the decision steps. To hold the entropy at a target H⋆=3.5H^\star= 3.5 nats, λH\lambda_H is multiplied after every training step by exp⁡(η(H⋆−Hˉ))\exp(\eta(H^\star-\bar{H})), with step size η=10−2\eta= 10^{-2}, and kept within [10−6,10−1][10^{-6},10^{-1}]: it grows while the policy is sharper than the target and shrinks while it is flatter. We use a target rather than a fixed weight because the weight that holds 3.5 nats differs by about an order of magnitude between datasets and drifts within a run (Figure J.2B). With the entropy held at the target, the policy cannot become sharp: 1−πf1-\pi_f is not, as in Appendix § 18.§ 3, almost always near 0 and occasionally near 1.

Results. A smaller policy-loss weight only delays the drop in task performance. The policy sharpens at about the same speed as at λ=0.1\lambda= 0.1 (Figure J.2A). On ADE20K, the run at λ=0.01\lambda= 0.01 stays within about 1 mIoU of the run with stop-gradient for eight epochs, while the run without stop-gradient at λ=0.1\lambda= 0.1 falls behind from the first epoch (Figure J.1). After about eight epochs, the run at λ=0.01\lambda= 0.01 falls behind the run with stop-gradient as well (11.5 against 17.4 mIoU at epoch 32). Trivially, a much smaller weight would eventually behave like the stop-gradient, but we did not investigate further configurations.

The small head and the entropy target keep the entropy at 2.0–3.5 nats (Figure J.2A), and gradient-norm clipping almost never acts on them. They recover the full task performance on ImageNet (61.4% and 61.9% against 61.6% with stop-gradient) and most of it on the segmentation datasets, 80–92% of the drop (Table J.1). On C-MNIST, the entropy target matches the model with stop-gradient (99.3%), while the small head loses 4.2% (down to 95.1%). Transferred to a frozen random-location model, their policies improve on random locations about as much as the policy with stop-gradient or slightly more (Table J.2).

Both variants also change how the policy gradient reaches backbone and memory: it is spread evenly over the samples instead of arriving in bursts (Figure J.3). Measured as in Appendix § 18.§ 3, on the checkpoints of Figure J.3, it stays within 0.2–6× the task gradient on every minibatch. The variants thus receive a steady policy gradient on backbone and memory. In many minibatches it is larger than in the runs without stop-gradient, where it is often negligible and large only in rare bursts (Appendix § 18.§ 3). Yet the variants lose much less task performance. Policy gradient on backbone and memory may therefore not necessarily be harmful in itself. The severe drop goes together with its arrival in rare, large bursts.

Summary and the stop-gradient in our default models

What goes wrong. Without the stop-gradient, task performance drops on all four datasets (Appendix § 18.§ 1). Worse fixations, clipping and the optimizer state are unlikely to explain this drop (Appendix § 18.§ 2), which suggests that the policy gradient on backbone and memory itself causes it. During training, the policy sharpens and the Jacobian norm of its head grows, so backbone and memory receive rare, large bursts of policy gradient in directions unrelated to the task gradient (Appendix § 18.§ 3). On ImageNet, which is trained in bf16, rounding ties the largest policy logits, so that many samples send a large gradient (Appendix § 18.§ 4).

What prevents it. The stop-gradient prevents the drop in task performance. Without it, a small policy head or an entropy target recovers most of the task performance (Appendix § 18.§ 5). Both keep the policy from sharpening.

What does not. A 10× smaller policy-loss weight λ\lambda only delays the drop in task performance (Appendix § 18.§ 5), and gradient-norm clipping does not prevent it (Appendix § 18.§ 2). The runs trained in fp32 (ADE20K, COCO-Stuff), in which the largest policy logits do not round to the same value as they do in bf16, lose task performance as well (Appendix § 18.§ 4).

What remains open. Which property of the bursts (their size, their direction, their frequency, effect on optimizer) harms backbone and memory is not identified, and neither is the precise reason why the policy becomes so sharp. The run with a smaller policy-loss weight (Appendix § 18.§ 5) is a single run on ADE20K. The other factors tested (worse fixations, clipping, optimizer state) are unlikely to explain the drop in task performance, but further contributing factors are not excluded.

The choice of the stop-gradient for our default models. The stop-gradient is one option among several and may not necessarily be the best one in general. Arguably, it is a sound baseline and needs no per-dataset tuning. Also, we found that it performs at least as well as the small head and the entropy target (they match it on ImageNet and lose 0.5–1.4 mIoU on segmentation; single runs). The other variants warrant further attention in future research.

Derivation of the bound

For completeness, this subsection derives the bound used in Appendix § 18.§ 3.

Setting. sg⁡[⋅]\operatorname{sg}[\cdot] is the stop-gradient: it passes its argument unchanged in the forward pass and lets no gradient through in the backward pass. The model without stop-gradient drops the sg⁡[⋅]\operatorname{sg}[\cdot] around Ht\mathbf{H}_t in Equation (8), weights its policy loss by λ=0.1\lambda= 0.1 and averages it over the batch of size BB:

λLpolicy=−λB∑b=1B∑t=1T−1sg⁡[r~t]log⁡πθ(ft+1∣Ht),\lambda\mathcal{L}_{\mathrm{policy}} = -\frac{\lambda}{B}\sum_{b=1}^{B}\sum_{t=1}^{T-1}\operatorname{sg}[\tilde{r}_t]\log\pi_{\theta}(f_{t+1}\mid\mathbf{H}_t),

where r~t\tilde{r}_t, ft+1f_{t+1} and Ht\mathbf{H}_t belong to sample bb (index dropped); ft+1f_{t+1} is the drawn fixation, one cell of the 64 × 64 grid. The policy is the softmax over the cells of the policy head’s output (Equation (3)); ψθ\psi_\theta is the convolutional projection of the policy head (§ 3.4), which maps the memory features to one value per cell kk. Written out,

log⁡πθ(ft+1∣Ht)=[ψθ(Ht)]ft+1−log⁡∑jexp⁡([ψθ(Ht)]j).(10)\log\pi_{\theta}(f_{t+1}\mid\mathbf{H}_t) = [\psi_{\theta}(\mathbf{H}_t)]_{f_{t+1}} - \log\sum_j \exp\left([\psi_{\theta}(\mathbf{H}_t)]_j\right). \tag*{(10)}

Ht\mathbf{H}_t thus enters only through ψθ(Ht)\psi_\theta(\mathbf{H}_t). These values are the policy logits, zt=ψθ(Ht)\mathbf{z}_t = \psi_\theta(\mathbf{H}_t) with entries zkz_k, so that πt=softmax⁡(zt)\boldsymbol{\pi}_t = \operatorname{softmax}(\mathbf{z}_t) is the policy written as a vector, with entries πk\pi_k, and eft+1\mathbf{e}_{f_{t+1}} is the one-hot vector of the drawn cell. To shorten notation, we write z\mathbf{z}, π\boldsymbol{\pi} and f\mathbf{f} for zt\mathbf{z}_t, πt\boldsymbol{\pi}_t and ft+1\mathbf{f}_{t+1}.

The chain rule. Since the term of one sample and step depends on Ht\mathbf{H}_t only through the logits zt\mathbf{z}_t, its gradient is a product of three factors: the derivative of the loss term with respect to log⁡πf\log\pi_f, of log⁡πf\log\pi_f with respect to the logits, and of the logits with respect to the memory features,

∂Lpolicy∂Ht=∂Lpolicy∂log⁡πf(∂log⁡πf∂zt∂zt∂Ht)T⏟step 2step 3\frac{\partial\mathcal{L}_{\mathrm{policy}}}{\partial\mathbf{H}_t} = \frac{\partial\mathcal{L}_{\mathrm{policy}}}{\partial\log\pi_f} \underbrace{\left( \frac{\partial\log\pi_f}{\partial\mathbf{z}_t} \frac{\partial\mathbf{z}_t}{\partial\mathbf{H}_t} \right)^{\mathsf{T}}}_{\text{step 2}\qquad\text{step 3}}

with the first factor labelled step 1.

Step 1: loss term. r~t\tilde{r}_t is held constant by sg⁡[⋅]\operatorname{sg}[\cdot], so ∂Lpolicy/∂log⁡πf=−λBr~t\partial\mathcal{L}_{\mathrm{policy}}/\partial\log\pi_f = -\frac{\lambda}{B}\tilde{r}_t.

Step 2: log-softmax. Writing out the log-softmax and differentiating with respect to one logit zkz_k,

log⁡πf=zf−log⁡∑jezj,\log\pi_f = z_f - \log\sum_j e^{z_j},
∂log⁡πf∂zk=1[k=f]−ezk∑jezj=1[k=f]−πk,(11)\frac{\partial\log\pi_f}{\partial z_k} = 1[k=f] - \frac{e^{z_k}}{\sum_j e^{z_j}} = 1[k=f] - \pi_k, \tag*{(11)}
∂log⁡πf∂z=ef−π.\frac{\partial\log\pi_f}{\partial\mathbf{z}} = \mathbf{e}_f - \boldsymbol{\pi}.

Raising the drawn cell’s logit raises log⁡πf\log\pi_f by 1−πf1-\pi_f; raising any other logit lowers it by that cell’s probability, through the softmax denominator.

Step 3: policy head. The last factor is the Jacobian Jt=∂zt/∂Ht\mathbf{J}_t = \partial\mathbf{z}_t/\partial\mathbf{H}_t of the policy head. Together,

∂Lpolicy∂Ht=−λBr~tJtT(eft+1−πt).\frac{\partial\mathcal{L}_{\mathrm{policy}}}{\partial\mathbf{H}_t} = -\frac{\lambda}{B}\tilde{r}_t \mathbf{J}_t^{\mathsf{T}}(\mathbf{e}_{f_{t+1}}-\boldsymbol{\pi}_t).

This gradient is added to that of the task loss at Ht\mathbf{H}_t and backpropagated through memory and backbone; with the stop-gradient, Ht\mathbf{H}_t enters the policy loss as a constant and the term is zero. Through the memory recurrence, the policy terms of later steps also reach Ht\mathbf{H}_t; the bound below applies to each term.

The bound. The operator norm ∥Jt∥\lVert\mathbf{J}_t\rVert (the Jacobian norm of the policy head) bounds how much JtT\mathbf{J}_t^{\mathsf{T}} can stretch a vector: ∥JtTv∥≤∥Jt∥∥v∥\lVert\mathbf{J}_t^{\mathsf{T}}\mathbf{v}\rVert\le\lVert\mathbf{J}_t\rVert\lVert\mathbf{v}\rVert for every vector v\mathbf{v}. For v=ef−π\mathbf{v}=\mathbf{e}_f-\boldsymbol{\pi},

∥ef−π∥2=(1−πf)2+∑k≠fπk2≤2(1−πf)2,\lVert\mathbf{e}_f-\boldsymbol{\pi}\rVert^2 = (1-\pi_f)^2+\sum_{k\ne f}\pi_k^2 \le2(1-\pi_f)^2,

because the probabilities of the other cells sum to 1−πf1-\pi_f, and the sum of their squares is at most the square of that sum, (1−πf)2(1-\pi_f)^2. Together,

∥∂Lpolicy∂Ht∥≤λB∣r~t∣∥Jt∥2(1−πft+1).(12)\left\lVert\frac{\partial\mathcal{L}_{\mathrm{policy}}}{\partial\mathbf{H}_t}\right\rVert \le\frac{\lambda}{B}|\tilde{r}_t|\lVert\mathbf{J}_t\rVert\sqrt{2}(1-\pi_{f_{t+1}}). \tag*{(12)}

Of the three factors, ∣r~t∣≤n−1|\tilde{r}_t|\le\sqrt{n-1} for the nn samples standardized together (Equation (7)), and 1−πf≤11-\pi_f\le1; only the Jacobian norm ∥Jt∥\lVert\mathbf{J}_t\rVert of the policy head is unbounded.

Random locationsLearned policy
TevalT_{\mathrm{eval}}Per-stepLast onlyΔ\DeltaPer-stepLast onlyΔ\Delta
158.7152.77+5.9455.2546.23+9.02
4 (trained)62.9263.81−0.8964.6664.34+0.32
1065.4964.64+0.8565.2164.14+1.07
2065.7363.18+2.5564.1362.19+1.94

Table K.1. Supervision at every step against supervision at the last step only. ImageNet accuracy (%) of models trained with T=4T = 4 glimpses and unrolled to TevalT_{\mathrm{eval}} glimpses, each evaluated with the fixations it was trained with (random locations or its native policy). Δ\Delta is the accuracy in column Per-step minus that in column Last only. The models use an earlier version of the backbone, so their accuracies are not comparable with those of the other tables. Deviations (Table I.Table 2): terminal reward (D1), terminal supervision (D2; the last-only variants), level normalizer (D3), gradient on backbone and memory (D4), best checkpoint (D7; random-location pair), baseline present (D8), different pretrained models (D9), different learning-rate schedule (D13), entropy regularization (D14). In addition, the learned last-only model is evaluated with fixations drawn from its policy instead of argmax fixations.

Per-step and end-of-episode signals in the loss

Two choices of the default training recipe give a learning signal at every step rather than only at the end of an episode: the task loss supervises the prediction of each step (Appendix § 19.§ 1), and the reward credits each fixation with its own improvement (Appendix § 19.§ 2). This appendix compares each choice with its end-of-episode counterpart, at the trained budget and with fewer or more glimpses at inference. The runs come from earlier stages of model development, and the captions in this appendix list how they differ from our default models (Table I.Table 2).

Per-step supervision in the task loss

The task loss supervises the prediction of every step (Equation (5)). The alternative examined here supervises only the last step: it replaces Equation (5) by the cross-entropy of the final prediction, LCE(y^T,y)\mathcal{L}_{\mathrm{CE}}(\hat{y}_{T}, y), as in RAM (Mnih et al. [62]). Other weightings of the steps are possible but are not examined here. Both pairs of models in Table K.1 were trained before we introduced the stop-gradient (deviation D4, Table I.Table 2). Since the choice acts on the task loss, which trains backbone and memory either way, we expect the comparison to carry over to our default setting.

At the trained budget, the choice makes no consistent difference. With fewer or more glimpses, per-step supervision is clearly ahead. With a single glimpse, it leads by 5.94% with random locations and by 9.02% with a learned policy, since only the per-step models had their early predictions supervised. With 20 glimpses, it leads by 2.55% with random locations and by 1.94% with a learned policy, and both models supervised at the last step only have fallen below their accuracy at the trained budget. Per-step task supervision thus appears to promote the model’s ability to perform well when unrolled for fewer or more glimpses than it was trained with.

Per-step reward in the policy loss

The policy is trained by policy gradient (Equation (8)). By default, the reward of the fixation selected at step tt is the improvement it produced (Equation (6)),

rt=LCE(y^t,y)−LCE(y^t+1,y),t=1,…,T−1.(13)r_t = \mathcal{L}_{\mathrm{CE}}(\hat{y}_t, y) - \mathcal{L}_{\mathrm{CE}}(\hat{y}_{t+1}, y), \qquad t = 1,\ldots,T-1. \tag*{(13)}

The alternative is the terminal reward, which gives every fixation the negative task loss at the last step as reward,

rtterm=−LCE(y^T,y),t=1,…,T−1.(14)r_t^{\mathrm{term}} = -\mathcal{L}_{\mathrm{CE}}(\hat{y}_T, y), \qquad t = 1,\ldots,T-1. \tag*{(14)}
TevalT_{\mathrm{eval}}Random locationsTerminalPer-stepΔ\Delta
158.8958.8958.890.00
261.1763.12 ±\pm 0.0663.47 ±\pm 0.02+0.35 ±\pm 0.08
4 (trained)63.5465.27 ±\pm 0.4765.89 ±\pm 0.07+0.62 ±\pm 0.40
664.6564.97 ±\pm 0.1766.50 ±\pm 0.06+1.53 ±\pm 0.23
865.2664.56 ±\pm 0.1666.57 ±\pm 0.01+2.01 ±\pm 0.15
1065.8064.21 ±\pm 0.2066.58 ±\pm 0.04+2.36 ±\pm 0.15

Table K.2. Reward rule and horizon on ImageNet. Accuracy (%) with policy heads trained on a frozen random-location model and evaluated with it, unrolled to TevalT_{\mathrm{eval}} glimpses. The column Random locations shows that model with its own, randomly drawn locations. Terminal and Per-step are means over two policy heads trained with different random seeds; ±\pm is half the difference between the two. Δ\Delta is the accuracy in column Per-step minus that in column Terminal, computed per seed and then averaged. Deviations (Table I.Table 2): terminal reward (D1; the terminal column), level normalizer (D3), frozen representation (D5), baseline present (D8), policy-transfer score (D10), reduced budget (D11; one GPU, 24 epochs of 2000 steps). In addition, the policy head has one instead of three 3×33 \times3 convolutions.

Table K.2 compares the two rules on ImageNet below, at and past the trained budget. We train only the policy head, on a frozen ImageNet model that was trained with random locations, and evaluate both policies with this same model. Differences in performance between the two rules therefore come purely from the fixations they produce. For each rule, two policy heads with one instead of three 3×33 \times3 convolutions are trained with different random seeds for 24 epochs of 2000 minibatches (learning rate 10−410^{-4}, constant for eight epochs, then halved every four).

At the trained budget, the reward rule makes a small difference: the policy trained with the per-step reward is ahead by 0.62%. For longer rollouts, the difference gets bigger. Up to Teval=10T_{\mathrm{eval}} = 10, the policy trained with the terminal reward falls from 65.27 to 64.21% while random locations climb from 63.54 to 65.80%, so its margin over them turns from +1.73 to −1.59%. The policy trained with the per-step reward keeps rising, to 66.58%, and stays above random locations (+0.78%). Both seeds show this pattern: with the terminal reward, each falls below random locations from Teval=8T_{\mathrm{eval}} = 8 on, and with the per-step reward, each stays above them. Like per-step task supervision (Appendix § 19.§ 1), the per-step reward thus appears to promote the model’s ability to perform well when unrolled for more glimpses than it was trained with. The reason for the advantage of the per-step forms, and how per-step supervision compares to terminal supervision when using other training algorithms and optimization settings, remain to be investigated.

References

  1. [1]Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In 3rd International Conference on Learning Representations (ICLR), 2015.DOI
  2. [2]A Terry Bahill, Deborah Adler, and Lawrence Stark. Most naturally occurring human saccades have magnitudes of 15 degrees or less. Investigative Ophthalmology, 14(6):468–469, 1975a.pubmed.ncbi.nlm.nih.gov/1132942
  3. [3]A Terry Bahill, Michael R Clark, and Lawrence Stark. The main sequence, a tool for studying human eye movements. Mathematical Biosciences, 24(3-4):191–204, 1975b. doi: 10.1016/0025-5564(75)90075-9.DOI
  4. [4]Ruzena Bajcsy. Active perception. Proceedings of the IEEE, 76(8):966–1005, 1988. doi: 10.1109/5.5968.DOI
  5. [5]Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. In Advances in Neural Information Processing Systems, volume 34, pp. 25164–25178. Curran Associates, Inc., 2021.
  6. [6]Moshe Bar, Karim S Kassam, Avniel Singh Ghuman, Jasmine Boshyan, Annette M Schmid, Anders M Dale, Matti S Hämäläinen, Ksenija Marinkovic, Daniel L Schacter, Bruce R Rosen, et al. Top-down facilitation of visual recognition. Proceedings of the National Academy of Sciences, 103(2):449–454, 2006. doi: 10.1073/pnas.0507062103.DOI
  7. [7]Yohai-Eliel Berreby, Sabrina Du, Audrey Durand, and B. Suresh Krishna. CanViT: Toward active-vision foundation models. In Advances in Neural Information Processing Systems, 2026. arXiv:2603.22570.arxiv.org/abs/2603.22570
  8. [8]James W Bisley and Koorosh Mirpour. The neural instantiation of a priority map. Current Opinion in Psychology, 29:108–112, 2019. doi: 10.1016/j.copsyc.2019.01.002.DOI
  9. [9]Nicholas M. Blauch, George A. Alvarez, and Talia Konkle. FOVI: A biologically-inspired foveated interface for deep vision models. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of PMLR, pp. 8446–8469, 2026.arxiv.org/abs/2602.03766
  10. [10]Ali Borji and Laurent Itti. Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision, 14(3):29, 2014. doi: 10.1167/14.3.29.DOI
  11. [11]C Braccini, Giuseppe Gambardella, Giulio Sandini, and Vincenzo Tagliasco. A model of the early stages of the human visual system: Functional and topological transformations performed in the peripheral visual field. Biological Cybernetics, 44(1):47–58, 1982. doi: 10.1007/BF00353955.
  12. [12]Guy Thomas Buswell. How People Look at Pictures: A Study of the Psychology of Perception in Art. University of Chicago Press, Chicago, 1935.
  13. [13]Nicholas J Butko and Javier R Movellan. Infomax control of eye movements. IEEE Transactions on Autonomous Mental Development, 2(2):91–107, 2010. doi: 10.1109/tamd.2010.2051029.DOI
  14. [14]Santiago A. Cadena, Konstantin F. Willeke, Kelli Restivo, George Denfield, Fabian H. Sinz, Matthias Bethge, Andreas S. Tolias, and Alexander S. Ecker. Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks. PLoS Computational Biology, 20(5):e1012056, 2024. doi: 10.1371/journal.pcbi.1012056.
  15. [15]Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1209–1218, 2018. doi: 10.1109/cvpr.2018.00132.DOI
  16. [16]Jie Cao, Chun Bao, Qun Hao, Yang Cheng, and Chenglin Chen. LPNet: Retina inspired neural network for object detection and recognition. Electronics, 10(22):2883, 2021. doi: 10.3390/electronics10222883.
  17. [17]Giuseppe Cartella, Vittorio Cuculo, Alessandro D’Amelio, Marcella Cornia, Giuseppe Boccignone, and Rita Cucchiara. Modeling human gaze behavior with diffusion models for unified scanpath prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16206–16216, 2025. doi: 10.1109/iccv51701.2025.01504.DOI
  18. [18]Gang Chen. Where to look: A unified attention model for visual recognition with reinforcement learning. arXiv preprint arXiv:2111.07169, 2021.arxiv.org/abs/2111.07169
  19. [19]Zhenzhong Chen and Wanjie Sun. Scanpath prediction for visual attention using IOR-ROI LSTM. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pp. 642–648, 2018. doi: 10.24963/ijcai.2018/89.DOI
  20. [20]Brian Cheung, Eric Weiss, and Bruno Olshausen. Emergence of foveal image sampling from learning to attend in visual scenes. In 5th International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1611.09430
  21. [21]Minkyu Choi, Kuan Han, Xiaokai Wang, Yizhen Zhang, and Zhongming Liu. A dual-stream neural network explains the functional segregation of dorsal and ventral visual pathways in human brains. In Advances in Neural Information Processing Systems, volume 36, pp. 50408–50428. Curran Associates, Inc., 2023. doi: 10.52202/075280-2193.DOI
  22. [22]Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926. IEEE, 2017. doi: 10.1109/ijcnn.2017.7966217.DOI
  23. [23]Jean-Baptiste Cordonnier, Aravindh Mahendran, Alexey Dosovitskiy, Dirk Weissenborn, Jakob Uszkoreit, and Thomas Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2351–2360, 2021. doi: 10.1109/cvpr46437.2021.00238.DOI
  24. [24]Federico D’Agostino, Lisa Schwetlick, Matthias Bethge, and Matthias Kümmerer. What moves the eyes: Doubling mechanistic model performance using deep networks to discover and test cognitive hypotheses. In Advances in Neural Information Processing Systems, volume 38, pp. 63059–63088. Curran Associates, Inc., 2025. doi: 10.52202/085713-1895.DOI
  25. [25]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255. IEEE, 2009. doi: 10.1109/cvpr.2009.5206848.DOI
  26. [26]Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural Computation, 24(8):2151–2184, 2012. doi: 10.1162/neco_a_00312.DOI
  27. [27]Gamaleldin F. Elsayed, Simon Kornblith, and Quoc V. Le. Saccader: Improving accuracy of hard attention models for vision. In Advances in Neural Information Processing Systems, volume 32, pp. 702–714. Curran Associates, Inc., 2019.arxiv.org/abs/1908.07644
  28. [28]Ralf Engbert, Hans A Trukenbrod, Simon Barthélmé, and Felix A Wichmann. Spatial statistics and attentional dynamics in scene viewing. Journal of Vision, 15(1):14, 2015. doi: 10.1167/15.1.14.DOI
  29. [29]Jillian H Fecteau and Douglas P Munoz. Salience, relevance, and firing: a priority map for target selection. Trends in Cognitive Sciences, 10(8):382–390, 2006. doi: 10.1016/j.tics.2006.06.011.DOI
  30. [30]Pedro F Felzenszwalb, Ross B Girshick, and David McAllester. Cascade object detection with deformable part models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2241–2248. IEEE, 2010. doi: 10.1109/cvpr.2010.5539906.DOI
  31. [31]Tom Foulsham, Alan Kingstone, and Geoffrey Underwood. Turning the world around: Patterns in saccade direction vary with picture orientation. Vision Research, 48(17):1777–1790, 2008. doi: 10.1016/j.visres.2008.05.018.DOI
  32. [32]Edward G Freedman. Coordination of the eyes and head during visual orienting. Experimental Brain Research, 190(4):369–387, 2008. doi: 10.1007/s00221-008-1504-8.DOI
  33. [33]Karl Friston, Rick A Adams, Laurent Perrinet, and Michael Breakspear. Perceptions as hypotheses: saccades as experiments. Frontiers in Psychology, 3:151, 2012. doi: 10.3389/fpsyg.2012.00151.DOI
  34. [34]Charles D Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5):350–363, 2013. doi: 10.1038/nrn3476.DOI
  35. [35]Andrey Gizdov, Shimon Ullman, and Daniel Harari. Seeing more with less: human-like representations in vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4408–4417, 2025. doi: 10.1109/cvpr52734.2025.00416.DOI
  36. [36]Julie D Golomb and James A Mazer. Visual remapping. Annual Review of Vision Science, 7(1):257–277, 2021. doi: 10.1146/annurev-vision-032321-100012.DOI
  37. [37]Mary Hayhoe and Dana Ballard. Modeling task control of eye movements. Current Biology, 24(13):R622–R628, 2014. doi: 10.1016/j.cub.2014.05.020.DOI
  38. [38]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. doi: 10.1109/cvpr.2016.90.DOI
  39. [39]John M Henderson and Graham L Pierce. Eye movements during scene viewing: Evidence for mixed control of fixation durations. Psychonomic Bulletin & Review, 15(3):566–573, 2008. doi: 10.3758/pbr.15.3.566.
  40. [40]David Hoppe and Constantin A Rothkopf. Multi-step planning of eye movements in visual search. Scientific Reports, 9:144, 2019. doi: 10.1038/s41598-018-37536-0.
  41. [41]Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11):1254–1259, 1998. doi: 10.1109/34.730558.
  42. [42]Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Advances in Neural Information Processing Systems, volume 28, pp. 2017–2025. Curran Associates, Inc., 2015.arxiv.org/abs/1506.02025
  43. [43]Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. In 5th International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1611.01144
  44. [44]Abhishek Jha, Soroush Seifi, and Tinne Tuytelaars. SimGlim: Simplifying glimpse based active visual reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 269–278, 2023. doi: 10.1109/wacv56688.2023.00035.
  45. [45]Aditya Jonnalagadda, William Yang Wang, B. S. Manjunath, and Miguel P. Eckstein. FoveaTer: Foveated transformer for image classification. arXiv preprint arXiv:2105.14173, 2021.arxiv.org/abs/2105.14173
  46. [46]Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. Learning to predict where humans look. In Proceedings of the IEEE 12th International Conference on Computer Vision (ICCV), pp. 2106–2113. IEEE, 2009. doi: 10.1109/iccv.2009.5459462.DOI
  47. [47]George Killick, Paul Henderson, Paul Siebert, and Gerardo Aragon-Camarasa. Foveation in the era of deep learning. In Proceedings of the 34th British Machine Vision Conference (BMVC), 2023.arxiv.org/abs/2312.01450
  48. [48]Raymond M Klein and W Joseph MacInnes. Inhibition of return is a foraging facilitator in visual search. Psychological Science, 10(4):346–352, 1999. doi: 10.1111/1467-9280.00166.
  49. [49]Damian Koevoet, Laura Van Zantwijk, Marnix Naber, Sebastiaan Mathôt, Stefan Van der Stigchel, and Christoph Strauch. Effort drives saccade selection. eLife, 13:RP97760, 2025. doi: 10.7554/eLife.97760.DOI
  50. [50]Eileen Kowler. Eye movements: The past 25 years. Vision Research, 51(13):1457–1483, 2011. doi: 10.1016/j.visres.2010.12.014.DOI
  51. [51]Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Jonathan Presc​​ott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel K Yamins, and James J DiCarlo. Brain-like object recognition with high-performing shallow recurrent ANNs. In Advances in Neural Information Processing Systems, volume 32, pp. 12805–12816. Curran Associates, Inc., 2019.arxiv.org/abs/1909.06161
  52. [52]Matthias Kümmerer and Matthias Bethge. State-of-the-art in human scanpath prediction. arXiv preprint arXiv:2102.12239, 2021.arxiv.org/abs/2102.12239
  53. [53]Matthias Kümmerer, Matthias Bethge, and Thomas SA Wallis. DeepGaze III: Modeling free-viewing human scanpaths with deep learning. Journal of Vision, 22(5):7, 2022. doi: 10.1167/jov.22.5.7.DOI
  54. [54]Matthias Kümmerer, Harneet Singh Khanuja, and Matthias Bethge. Modeling saliency dataset bias. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22077–22088, 2025. doi: 10.1109/ICCV51701.2025.02050.DOI
  55. [55]Hugo Larochelle and Geoffrey E Hinton. Learning to combine foveal glimpses with a third-order Boltzmann machine. In Advances in Neural Information Processing Systems, volume 23, pp. 1243–1251. Curran Associates, Inc., 2010.
  56. [56]Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791.
  57. [57]Jason Li, Nicholas Watters, Hansem Sohn, and Mehrdad Jazayeri. Modeling human eye movements with neural networks in a maze-solving task. In Gaze Meets Machine Learning Workshop, volume 210 of PMLR, pp. 98–112, 2023.
  58. [58]Jiayang Liu, Yiming Bu, Daniel Ts’o, and Qinru Qiu. Improved efficiency based on learned saccade and continuous scene reconstruction from foveated visual sampling. In 12th International Conference on Learning Representations (ICLR), 2024.
  59. [59]Hristofor Lukanov, Peter König, and Gordon Pipa. Biologically inspired deep learning model for efficient foveal-peripheral vision. Frontiers in Computational Neuroscience, 15:746204, 2021. doi: 10.3389/fncom.2021.746204.
  60. [60]Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations (ICLR), 2017.
  61. [61]Noa Malem-Shinitski, Manfred Opper, Sebastian Reich, Lisa Schwetlick, Stefan A Seelig, and Ralf Engbert. A mathematical model of local and global attention in natural scene viewing. PLoS Computational Biology, 16(12):e1007880, 2020. doi: 10.1371/journal.pcbi.1007880.
  62. [62]Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, volume 27, pp. 2204–2212. Curran Associates, Inc., 2014.arxiv.org/abs/1406.6247
  63. [63]Tirin Moore and Katherine M Armstrong. Selective gating of visual signals by microstimulation of frontal cortex. Nature, 421(6921):370–373, 2003. doi: 10.1038/nature01341.
  64. [64]Jiri Najemnik and Wilson S Geisler. Optimal eye movement strategies in visual search. Nature, 434(7031):387–391, 2005. doi: 10.1038/nature03390.
  65. [65]Jiri Najemnik and Wilson S Geisler. Eye movement statistics in humans are consistent with an optimal search strategy. Journal of Vision, 8(3):4, 2008. doi: 10.1167/8.3.4.
  66. [66]Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L. K. Yamins. Task-driven convolutional recurrent models of the visual system. In Advances in Neural Information Processing Systems, volume 31, pp. 5290–5301. Curran Associates, Inc., 2018.arxiv.org/abs/1807.00053
  67. [67]Antje Nuthmann, Tim J Smith, Ralf Engbert, and John M Henderson. CRISP: a computational model of fixation durations in scene viewing. Psychological Review, 117(2):382–405, 2010. doi: 10.1037/a0018924.DOI
  68. [68]Jan Olszewski, Dawid Damian Rymarczyk, Piotr Wójcik, Mateusz Pach, and Bartosz Zieliński. TORE: Token recycling in vision transformers for efficient active visual exploration. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 8606–8616, February 2025. doi: 10.1109/wacv61041.2025.00834.DOI
  69. [69]Athanასios Papadopoulos, Paweł Korus, and Nasir Memon. Hard-attention for scalable image classification. In Advances in Neural Information Processing Systems, volume 34, pp. 14694–14707. Curran Associates, Inc., 2021.arxiv.org/abs/2102.10212
  70. [70]Adam Pardyl, Grzegorz Rypeść, Grzegorz Kurzejamski, Bartosz Zieliński, and Tomasz Trzciński. Active visual exploration based on attention-map entropy. In Edith Elkind (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pp. 1303–1311. International Joint Conferences on Artificial Intelligence Organization, August 2023. doi: 10.24963/ijcai.2023/145.DOI
  71. [71]Adam Pardyl, Michał Wronka, Maciej Wołczyk, Kamil Adamczewski, Tomasz Trzciński, and Bartosz Zieliński. AdaGlimpse: Active visual exploration with arbitrary glimpse position and scale. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision – ECCV 2024, Lecture Notes in Computer Science, pp. 112–129, Cham, 2024. Springer Nature Switzerland. doi: 10.1007/978-3-031-72664-4_7.DOI
  72. [72]Michael I Posner and Yoav Cohen. Components of visual orienting. In Herman Bouma and Don G Bouwhuis (eds.), Attention and Performance X: Control of Language Processes, pp. 531–556. Erlbaum, Hillsdale, NJ, 1984.
  73. [73]Motahareh Pourrahimi and Pouya Bashivan. Neural signatures of associational cortex emerge in a goal-directed model of visual search. bioRxiv preprint 2025.06.06.658387, 2025. doi: 10.1101/2025.06.06.658387.DOI
  74. [74]Samruddhi B Rangrej, Chetan L Srinidhi, and James J Clark. Consistency driven sequential transformers attention model for partially observable scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2508–2517, 2022. doi: 10.1109/cvpr52688.2022.00255.DOI
  75. [75]Joséphine Raugel, Marc Szafraniec, Huy V Vo, Camille Couprie, Patrick Labatut, Piotr Bojanowski, Valentin Wyart, and Jean-Rémi King. Disentangling the factors of convergence between brains and DINOv3. In 14th International Conference on Learning Representations (ICLR), 2026.
  76. [76]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädl e, Chloé Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In 13th International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2408.00714
  77. [77]Stephanie M Reeves and Jorge Otero-Millan. Horizontal saccade bias results from combination of saliency anisotropies and egocentric biases. Scientific Reports, 16(1):6027, 2026. doi: 10.1038/s41598-026-35572-9.DOI
  78. [78]Stephanie M Reeves, Emily A Cooper, Raul Rodriguez, and Jorge Otero-Millan. Head orientation influences saccade directions during free viewing. eNeuro, 9(6):ENEURO.0273–22.2022, 2022. doi: 10.1523/eneuro.0273-22.2022.DOI
  79. [79]Mengye Ren and Richard S. Zemel. End-to-end instance segmentation with recurrent attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 293–301, 2017. doi: 10.1109/cvpr.2017.39.DOI
  80. [80]Lars OM Rothkegel, Hans A Trukenbrod, Heiko H Schütt, Felix A Wichmann, and Ralf Engbert. Influence of initial fixation position in scene viewing. Vision Research, 129:33–49, 2016. doi: 10.1016/j.visres.2016.09.012.DOI
  81. [81]Giulio Sandini and Vincenzo Tagliasco. An anthropomorphic retina-like structure for scene analysis. Computer Graphics and Image Processing, 14(3):365–372, 1980. doi: 10.1016/0146-664x(80)90026-x.DOI
  82. [82]Jeffrey D Schall, Anne Morel, Dana J King, and Jean Bullier. Topography of visual cortex connections with frontal eye field in macaque: convergence and segregation of processing streams. The Journal of Neuroscience, 15(6):4464–4487, 1995. doi: 10.1523/JNEUROSCI.15-06-04464.1995.DOI
  83. [83]Jürgen Schmidhuber and Rudolf Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2(1–2):125–134, 1991. doi: 10.1142/s012906579100011x.DOI
  84. [84]Eric L. Schwartz. Spatial mapping in the primate sensory projection: analytic structure and relevance to perception. Biological Cybernetics, 25(4):181–194, 1977. doi: 10.1007/bf01885636.DOI
  85. [85]Eric L Schwartz. Computational anatomy and functional architecture of striate cortex: a spatial mapping approach to perceptual coding. Vision Research, 20(8):645–669, 1980. doi: 10.1016/0042-6989(80)90090-5.DOI
  86. [86]Lisa Schwetlick, Lars Oliver Martin Rothkegel, Hans Arne Trukenbrod, and Ralf Engbert. Modeling the effects of perisaccadic attention on gaze statistics during scene viewing. Communications Biology, 3(1):727, 2020. doi: 10.1038/s42003-020-01429-8.DOI
  87. [87]Lisa Schwetlick, Daniel Backhaus, and Ralf Engbert. A dynamical scan-path model for task-dependence during scene viewing. Psychological Review, 130(3):807–840, 2023. doi: 10.1037/rev0000379.DOI
  88. [89]Soroush Seifi and Tinne Tuytelaars. Attend and segment: Attention guided active semantic segmentation. In Computer Vision – ECCV 2020, pp. 305–321. Springer, 2020. doi: 10.1007/978-3-030-58595-2_19.DOI
  89. [90]Soroush Seifi, Abhishek Jha, and Tinne Tuytelaars. Glimpse-attend-and-explore: Self-attention for active visual exploration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16117–16126, 2021. doi: 10.1109/iccv48922.2021.01583.DOI
  90. [91]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeon Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3. arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
  91. [92]Tim J Smith and John M Henderson. Facilitation of return during scene viewing. Visual Cognition, 17(6-7):1083–1108, 2009. doi: 10.1080/13506280802678557.DOI
  92. [93]Nathan Sprague, Dana Ballard, and Al Robinson. Modeling embodied visual behaviors. ACM Transactions on Applied Perception, 4(2):11, 2007. doi: 10.1145/1265957.1265960.DOI
  93. [94]Philip Sulewski, Carmen Amme, Peter König, Martin N Hebart, and Tim C Kietzmann. Active visual semantics: A large-scale MEG and eye-tracking dataset for understanding visual intelligence in action. arXiv preprint arXiv:2609.01055, 2026.arxiv.org/abs/2609.01055
  94. [95]Wanjie Sun, Zhenzhong Chen, and Feng Wu. Visual scanpath prediction using IOR-ROI recurrent mixture density network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):2101–2118, 2021. doi: 10.1109/tpami.2019.2956930.DOI
  95. [96]Benjamin W Tatler. The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions. Journal of Vision, 7(14):4, 2007. doi: 10.1167/7.14.4.DOI
  96. [97]Benjamin W. Tatler and Benjamin T Vincent. Systematic tendencies in scene viewing. Journal of Eye Movement Research, 2(2):5, 2008. doi: 10.16910/jemr.2.2.5.DOI
  97. [98]Benjamin W Tatler and Benjamin T Vincent. The prominence of behavioural biases in eye guidance. Visual Cognition, 17(6-7):1029–1054, 2009. doi: 10.1080/13506280902764539.DOI
  98. [99]Benjamin W Tatler, Mary M Hayhoe, Michael F Land, and Dana H Ballard. Eye guidance in natural vision: Reinterpreting salience. Journal of Vision, 11(5):5, 2011. doi: 10.1167/11.5.5.DOI
  99. [100]Sushrut Thorat, Adrien Doerig, Alexander Kroner, Carmen Amme, and Tim C Kietzmann. Predicting upcoming visual features during eye movements yields scene representations aligned with human visual cortex. Science Advances, 12(43):eae1955, 2026. doi: 10.1126/sciadv.aee1955.DOI
  100. [101]Robert J van Beers. Saccadic eye movements minimize the consequences of motor noise. PLoS ONE, 3(4):e2070, 2008. doi: 10.1371/journal.pone.0002070.DOI
  101. [102]Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: A dynamic approach to reducing spatial redundancy in image classification. In Advances in Neural Information Processing Systems, volume 33, pp. 2432–2444. Curran Associates, Inc., 2020.arxiv.org/abs/2010.05300
  102. [103]Yulin Wang, Yang Yue, Yue Yue, Huanqian Wang, Haojun Jiang, Yizeng Han, Zanlin Ni, Yifan Pu, Minglei Shi, Rui Lu, Qisen Yang, Andrew Zhao, Zhuofan Xia, Shiji Song, and Gao Huang. Emulating human-like adaptive vision for efficient and flexible machine visual perception. Nature Machine Intelligence, 7(11):1804–1822, 2025. doi: 10.1038/s42256-025-01130-7.DOI
  103. [104]Carl FR Weiman and George Chaikin. Logarithmic spiral grids for image processing and display. Computer Graphics and Image Processing, 11(3):197–226, 1979. doi: 10.1016/0146-664x(79)90089-3.DOI
  104. [105]Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, 1992. doi: 10.1007/BF00992696.DOI
  105. [106]Niklas Wilming, Simon Harst, Nico Schmidt, and Peter König. Saccadic momentum and facilitation of return saccades contribute to an optimal foraging strategy. PLoS Computational Biology, 9(1):e1002871, 2013. doi: 10.1371/journal.pcbi.1002871.DOI
  106. [107]Calden Wloka, Iuliia Kotseruba, and John K Tsotsos. Active fixation control to predict saccade sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3184–3193, 2018. doi: 10.1109/cvpr.2018.00336.DOI
  107. [108]Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619–8624, 2014. doi: 10.1073/pnas.1403112111.DOI
  108. [109]A. L. Yarbus. Eye Movements and Vision. Springer US, Boston, MA, 1967. doi: 10.1007/978-1-4899-5379-7.DOI
  109. [110]Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. International Journal of Computer Vision, 127(3):302–321, 2019. doi: 10.1007/s11263-018-1140-0.DOI

Paper details

vlnc:W_jkubjtn7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative models, agentic tools

View statement
Looking for related papers…
Contents