WorksAgricultural and Biological Sciences
Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition
Abstract
Computational models of cognition have traditionally relied on abstract symbolic inputs that approximate sensory experiences with simplified vector representations. This simplification reflects historical computational constraints and a long-held assumption that the details of sensory realism are incidental to the study of cognitive processes. Recent advances have enabled the development of sensory-realistic models capable of performing diverse cognitive tasks, yet whether sensory realism meaningfully shapes cognition has not been formally established. We address this gap through a large-scale empirical comparison of abstract and sensory-realistic neural network models trained on a battery of memory-dependent decision-making tasks. Our results show that sensory-realistic models mirror human behavioral response patterns significantly more than their abstract counterparts, though a substantial gap to human internal behavioral consistency remains even for the best tested models. By analyzing the internal dynamics of these systems, we find that the learned recurrent weights are not the main source of this divergence. In fact, frozen abstract-trained weights driven by sensory-realistic inputs recover the human consistency of sensory-realistic models. The resulting dynamics are also within the seed-to-seed variability of sensory-realistic models. Together, our findings argue that across a broad set of visual working-memory tasks, sensory realism accounts for a substantial portion of the model-human behavioral agreement, more than the choice of sequence architecture and comparable to the choice of vision encoder. These results suggest that findings obtained from abstract task models in neuroscience should be more carefully interpreted in light of their potential behavioral limitations.
Introduction
Since their inception, neural network models have been a key tool in understanding computations underpinning perception, memory, and decision-making in the brain [1, 2, 3, 4, 5]. In modeling cognition, the stimuli in these models are typically abstract, low-dimensional codes (e.g. one-hot or scalar values) rather than the high-dimensional sensory signals that biological brains receive from the external world. This convention historically rests on two grounds. First, developing cognitive models capable of processing raw sensory data had been computationally infeasible. Training and studying large neural network models at a scale that would enable processing of natural inputs such as high-resolution images was not possible until recently. Second, and perhaps more importantly, it is widely assumed that sensory observations are abstracted when they reach the brain regions implicated in cognitive processing. This viewpoint has been supported by many studies reporting neurons in the primate associational cortices that encode stimuli in a primarily categorical fashion [6, 7, 8]. As a result, most models of cognitive behavior have treated stimuli as idealized signals stripped of their rich perceptual content [9, 10, 11, 12, 13, 14]. Beyond these computational and theoretical grounds, the reliance on abstract stimuli also inherits a long experimental tradition in human and animal studies [15, 16, 17, 18, 19], where cognitive paradigms are deliberately built from impoverished stimuli such as dots, gratings, or colored patches, and models are then designed to mirror these paradigms [20, 11].
Given the long history of abstract models in cognitive neuroscience, many in the field may assume that these models reproduce cognitive behavior as effectively as sensory-realistic models. However, this assumption has not been tested, and it is not known whether sensory-realistic models of cognition are needed at all. Recent advances in developing large-scale neural network models have made it possible to build sensory-realistic cognitive models. Within the past decade, numerous studies have demonstrated the capabilities of these models in capturing behavioral and neural responses during perceptual and language tasks [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]. While several studies have begun applying these models to memory and decision-making settings [33, 34, 35], it remains unclear whether sensory realism matters. Importantly, no prior work has systematically asked whether the format of the input could modulate the cognitive behavior that emerges, nor whether the internal solutions these models discover depend on the input geometry. The answer may have important implications for neuroscience, as a substantial body of mechanistic claims about circuit dynamics, representational geometry, and the algorithms underlying decision-making is grounded in models trained on abstract inputs.
We address this gap through a large-scale comparison of abstract-input and sensory-realistic neural network models, evaluated on a battery of memory-dependent decision-making tasks for which we also collected human behavioral data1. We analyze the fine-grained pattern of errors humans and different models commit, and the internal recurrent dynamics that underlie their decisions. We systematically vary the sequence architecture, vision backbone, and model size, and study their effects on behavior. The results converge on the finding that input realism is not an incidental implementation detail but a determinant of the behavior that models produce. Sensory-realistic models track human behavior more closely than abstract-input models, and substitution experiments locate this difference in the inputs rather than in the learned recurrent weights.
Our main contributions are as follows:
We show that across a range of memory-dependent decision-making tasks, models trained with sensory-realistic inputs reproduce human behavioral responses significantly more closely than their abstract-input counterparts. Yet a substantial gap remains between the best tested model and human internal behavioral consistency.
We show that pretrained vision-language models, despite approaching or even exceeding human task performance, fall behind most task-trained sensory-realistic models at reproducing the pattern of human responses.
We show that recurrent weights trained on abstract inputs, when driven by sensory-realistic inputs, recover the human consistency of natural models and produce dynamics within the seed-to-seed variability of natural models.
We benchmark how neural network design choices such as sequential architecture, vision backbone, and model size affect human-model behavioral alignment.
Related Work
Sensory-Realistic Models in Neuroscience. With parallel GPU computing and the availability of large-scale datasets such as ImageNet [36, 37, 38, 39, 40], neural networks have scaled up, allowing their adoption as candidate models of neural processing in the brain. Early work demonstrated that deep convolutional networks trained on natural images can predict neural responses along the primate ventral visual stream at an unprecedented level [41, 21, 22, 23, 24, 25] while replicating the well-known hierarchy of functional selectivity along the visual pathway [42, 43, 44, 45, 46, 47]. This sensory-realistic modeling approach has since been extended to simulate neural activity across a broad range of modalities and domains, including audition [26, 27], olfaction [28, 29], oculomotor control [48], and language [30, 31, 32].
This shift toward naturalistic inputs has been partially driven by their potential applications. Models operating on natural stimuli are directly applicable to real-world downstream tasks [49], enable control of neural responses through closed-loop interventions and stimulus optimization [50, 51], and provide a framework for explaining large-scale organizational principles in the brain such as cortical topography and functional specialization [52, 53, 54]. They also generalize more broadly without a predefined input space [55, 56].
Models of Cognition. The question of whether the format of a representation matters for cognition has a long history in cognitive science. In the classic imagery debate, pictorialists argued that mental images are depictive and preserve the spatial layout of what they represent [57, 58, 59], while descriptivists argued that they are built from abstract, language-like descriptions [60, 61]. Here, we ask a related question for neural network models of cognition: whether the format of the input shapes the behavior and internal solutions that these models learn. Neural networks have also been used to understand neural computations in higher-level cognitive behaviors, spanning a wide range of paradigms including decision-making [11, 62, 63, 9, 64], working memory [13, 20], and associative memory [14, 65]. Much of this work, however, has been constrained to simplified setups where sensory observations and behavioral paradigms are delivered in symbolic form. For example, sensory evidence along different experimental dimensions is typically encoded as a single-channel scalar sequence, while task identities, rules, and contextual cues are supplied as one-hot codes [9, 66, 12, 11, 13]. More recently, a smaller but growing class of cognitive models has begun incorporating sensory realism, operating directly on high-dimensional natural inputs such as pixels or natural images in tasks of visual working memory and decision-making [33, 34, 35].
Methods
Problem Formulation
Each task trial consists of frames, each presenting a stimulus. At the designated response frames, the model must produce a binary decision comparing stimulus information across the frames according to the task rule. In both Abstract and Natural models, task identity is provided as a one-hot vector , concatenated to each frame in a trial. For humans and Vision-Language models (VLMs), the task rule is provided in natural language (see A.1.3 for details and Figure 6, Figure 7, Figure 8, Figure 9 for examples).

Figure 6. An example trial of 1-Back Category task

Figure 7. An example trial of InterDMS - ABAB - Position - Identity task

Figure 8. An example trial of CtxDMS - Category - Identity - Position task

Figure 9. An example trial of DMS-position task
Task Dataset
Stimuli are sampled from the HvM dataset [67, 68], which contains 5,760 images of objects positioned on naturalistic backgrounds (e.g. mountains, beaches, forests). The dataset contains 8 object categories (cars, planes, boats, fruits, faces, animals, chairs, and tables), each with 8 distinct object identities rendered in multiple viewing conditions. There are 3 variation levels, with each level introducing more extreme viewing conditions, making the objects more difficult to distinguish. Each image is described by discrete features: category , identity , and spatial position on a grid, and continuous features: rotation angles , and scale . See A.1.2 for stimuli examples.
We excluded the 100 images shown to human participants from model training and validation. The remaining 5,660 images are split into training and validation sets at a 3:1 ratio within each object and variation level (4,271 and 1,389 images). As a result, every object appears in training, and validation measures how well models generalize to unseen views of these objects. We chose this split so that the two model types can be compared fairly. Since Abstract models receive object identity as a one-hot vector (§ 3.3), if some objects were held out from training, the input weights for their identity dimensions would never be updated, and Abstract models would receive no useful identity information for these objects. Natural models do not face this problem, because their pretrained vision encoder produces informative features for images of any object.
We created 43 tasks from four task families: Delayed match-to-sample (DMS), NBack, Interleaved DMS (InterDMS), and contextual decision-making (CtxDMS) using iWISDM [69] (see Table 1 and Table 6). DMS tasks are the basic unit tasks, where an observer judges whether a sample stimulus and a test stimulus match on a target feature (position, identity, or category). In NBack tasks, at each step , the observer performs a DMS judgement by comparing the current stimulus with the stimulus on frame . In InterDMS tasks, the observer performs two independent DMS subtasks whose frames are interleaved. In CtxDMS tasks, the observer performs two DMS subtasks where the target feature of the second subtask depends on the outcome of the first.
| Family | Description | # Tasks | Task IDs |
| NBack | -back task for each feature | 9 | 1–9 |
| InterDMS | Two interleaved DMS sub-tasks (patterns: AABB, ABAB, ABBA) | 27 | 10–36 |
| CtxDMS | Contextual DMS: first comparison gates the feature to use in the second comparison | 4 | 37–40 |
| DMS | Simple delayed match-to-sample | 3 | 41–43 |
Table 1. Summary of the 43 DMS tasks.
| Family | # of frames | Param. | Action frame | ID | Comparison dimension |
| NBack | 6 | 2–6 | 1 | position | |
| NBack | 6 | 2–6 | 2 | identity | |
| NBack | 6 | 2–6 | 3 | category | |
| NBack | 6 | 3–6 | 4 | position | |
| NBack | 6 | 3–6 | 5 | identity | |
| NBack | 6 | 3–6 | 6 | category | |
| NBack | 6 | 4–6 | 7 | position | |
| NBack | 6 | 4–6 | 8 | identity | |
| NBack | 6 | 4–6 | 9 | category | |
| InterDMS | 4 | AABB | 2, 4 | 10 | position, position |
| InterDMS | 4 | AABB | 2, 4 | 11 | position, identity |
| InterDMS | 4 | AABB | 2, 4 | 12 | position, category |
| InterDMS | 4 | AABB | 2, 4 | 13 | identity, position |
| InterDMS | 4 | AABB | 2, 4 | 14 | identity, identity |
| InterDMS | 4 | AABB | 2, 4 | 15 | identity, category |
| InterDMS | 4 | AABB | 2, 4 | 16 | category, position |
| InterDMS | 4 | AABB | 2, 4 | 17 | category, identity |
| InterDMS | 4 | AABB | 2, 4 | 18 | category, category |
| InterDMS | 4 | ABAB | 3, 4 | 19 | position, position |
| InterDMS | 4 | ABAB | 3, 4 | 20 | position, identity |
| InterDMS | 4 | ABAB | 3, 4 | 21 | position, category |
| InterDMS | 4 | ABAB | 3, 4 | 22 | identity, position |
| InterDMS | 4 | ABAB | 3, 4 | 23 | identity, identity |
| InterDMS | 4 | ABAB | 3, 4 | 24 | identity, category |
| InterDMS | 4 | ABAB | 3, 4 | 25 | category, position |
| InterDMS | 4 | ABAB | 3, 4 | 26 | category, identity |
| InterDMS | 4 | ABAB | 3, 4 | 27 | category, category |
| InterDMS | 4 | ABBA | 3, 4 | 28 | position, position |
| InterDMS | 4 | ABBA | 3, 4 | 29 | position, identity |
| InterDMS | 4 | ABBA | 3, 4 | 30 | position, category |
| InterDMS | 4 | ABBA | 3, 4 | 31 | identity, position |
| InterDMS | 4 | ABBA | 3, 4 | 32 | identity, identity |
| InterDMS | 4 | ABBA | 3, 4 | 33 | identity, category |
| InterDMS | 4 | ABBA | 3, 4 | 34 | category, position |
| InterDMS | 4 | ABBA | 3, 4 | 35 | category, identity |
| InterDMS | 4 | ABBA | 3, 4 | 36 | category, category |
| CtxDMS | 3 | – | 3 | 37 | position category identity |
| CtxDMS | 3 | – | 3 | 38 | position identity category |
| CtxDMS | 3 | – | 3 | 39 | identity position category |
| CtxDMS | 3 | – | 3 | 40 | category identity position |
| DMS | 2 | – | 2 | 41 | position |
| DMS | 2 | – | 2 | 42 | identity |
| DMS | 2 | – | 2 | 43 | category |
Table 6. Complete listing of all 43 tasks. Each task is defined by a family-specific parameter (memory lag for NBack, interleaving pattern for InterDMS) and its comparison dimension(s); for CtxDMS the arrows give the feature ordering. Raw frames are the natural sequence length; all tasks are padded to 6 frames in multi-task training. Action frames are the frames that require a response (1-based).
Model Types
We use three types of models in our analyses (see Figure 1): (1) Abstract models: sequence models that receive symbolic vectors in their inputs; (2) Natural models: sequence models that receive a sequence of images; (3) Vision-Language Models (VLMs): pretrained transformer models that receive a natural-language description of the task and sequence of images in their input prompts.
Figure 1. Schematic of the Experimental Setup. Models receive both task and stimulus information in their inputs. Task indicator is provided in either textual or one-hot while stimuli are presented in their naturalistic or abstract formats. In the abstract format, information about the primary object in the scene is provided as a combination of one-hot (category, object identity, position) and scalar values. Three groups are studied: humans, neural network models trained on abstract or natural inputs, and pretrained vision-language models. Behavioral responses are collected from each group and used to estimate the choice probabilities for each task trial. These probabilities are then used to calculate behavioral consistency with humans using different metrics.
Natural input. Natural input consists of a sequence of images. Each image is processed by the frozen pretrained vision encoder.
The encoder maps each image into a sequence of token embeddings , where is the resulting number of patches from a grid and is the embedding dimension. We then apply attention pooling [70] on to obtain a single image embedding , and concatenate it with a one-hot task index to produce our pre-projection input vector . Formally,
where denotes the vision model with parameters .
Abstract input. Abstract input consists of a sequence of one-hot and continuous encodings of the object information in the images. Specifically, at each time step ,
where is a one-hot category vector, is a one-hot identity vector (8 identities in 8 categories), is a one-hot position vector over the spatial grid, is the rotation angle each axis, is the object scale, and is the same one-hot task index. To remove differences between natural input and abstract input, we propose dimensionality and visual uncertainty matched abstract models (A.2.3).
Model architecture. Both Natural and Abstract models project their respective inputs into dimensions with a linear layer followed by LayerNorm [71] and non-linearity (e.g. GELU [72]):
For every time step , passes through a sequence model followed by a linear output head :
where is the hidden state at time , and is the model’s class logits.
Evaluation Pipeline
To make the model-human comparisons feasible, we selected 100 stimuli from the HvM dataset and created 100 stimulus pairs that were balanced across all three object features (category, identity, position). Using this fixed set, we generated trials for 12 Amazon Mechanical Turk (MTurk) tasks (see A.1.3), with 20 trials for three 1Back tasks and 100 trials for the other nine tasks, totaling 960 task trials. These 12 tasks are a subset of the 43 tasks models were trained on.
The behavioral tasks performed in this study were reviewed and approved by the Research Ethics Board (REB) in accordance with ethical standards for research involving human subjects. We recruited 20 unique participants for each task on Amazon Mechanical Turk (MTurk). Each task page included instructions, example HvM stimuli, and an example trial. Participants then completed the experiment on an interactive website built with PsychoJS (see A.1.1 and A.1.3 for example trials and details of recruitment).2
To investigate the alignment between humans and VLMs, we evaluated four state-of-the-art VLMs of different sizes (see A.4). For trained Natural and Abstract models, we evaluated 6 random seeds per architecture on the 12 MTurk tasks. Because model inputs were padded to six frames, we appended delay frames to MTurk trials so that the final action frame always occurred at the sixth frame.
Behavioral Analysis
After evaluating on the 12 MTurk tasks (defined in A.1.3), we represent the model responses as matrices (directly from ). The first dimension indexes the 18 elementary task conditions, the second indexes the 100 trial stimulus pairs, and the final dimension indexes the two response logits (match and non-match). The 18 elementary task conditions arise by decomposing compositional tasks into their constituent DMS parts. For example, an InterDMS-ABBA trial is split into two elementary subtasks, InterDMS-AA and InterDMS-BB, each corresponding to a distinct stimulus pair. Applying this decomposition to the 12 MTurk tasks yields 18 elementary task conditions, whose responses are grouped and sorted into the same set of 100 MTurk stimulus pairs for all models, VLMs, and human observers. For humans and VLMs, we obtain binary output matrices . To quantify how closely model response patterns resemble those of humans, we apply the following behavioral metrics on the above matrices.
Human Consistency. Following Rajalingham et al. [73], we define human consistency as the Pearson correlation between model and human response vectors across trials (see A.2.1). For models, we apply softmax to to get the probability of correct class per choice. For humans and VLMs, we use respectively and estimate the class probabilities by calculating the fraction of correct choices. For each task, we split the 20 subjects into two groups of 10, treating each group’s average as a noisy estimate of the true response probability. We mirror the procedure for trained model logits, partitioning the 6 seeds into groups of 3. We report the mean over splits in Fisher- space, with the standard deviation over splits as error bars. Algorithm 1 describes this process in detail.
Representational Similarity Analysis (RSA). Using the same set of stimulus pairs on all MTurk tasks allows us to group behavioral patterns and measure how similarly two models organize the task/stimulus space. We performed two distinct RSA analyses: cross-task and cross-stimulus. With cross-task RSA, we compare response patterns across tasks with a fixed set of stimuli, and with cross-stimulus RSA, we compare response patterns across stimuli with a fixed set of tasks. See A.2.7.
Error consistency. We additionally compute Cohen’s as a measure of trial-by-trial error overlap [74]: , where is the fraction of trials on which the model and a human subject give the same response (both correct or both incorrect), and is the expected overlap by chance given their respective accuracies and .
Dynamical Similarity Analysis
To compare the recurrent structures learned by different models, we apply Dynamical Similarity Analysis (DSA) [75] to the hidden states from our trained models. Unlike , has shape since model inputs were padded to six frames, and there are 20 trials for the three 1Back tasks plus 100 trials for the other nine tasks. For each task and model seed, we project the states onto their top principal components. We then apply Hankel alternative view of Koopman (HAVOK) [76] to learn a linear operator that advances the delay-embedded state one step. Finally, we compute the pairwise distances between the learned operators for each model seed with Procrustes Analysis on Vector Fields. We report the mean SEM over the 12 tasks and all seed pairs. See A.2.8 for more details.
Training
We train both Abstract and Natural models using cross-entropy loss on batches of random trials from all tasks. A new set of trials is generated for each epoch while balancing for task type to mitigate learning biases. We use the AdamW optimizer [77] with a cosine-decay learning rate scheduler. To compare across different model architectures, we kept model parameters on the same order of magnitude. RNN [78] had , while all other sequential modules (LSTM [79], GRU [80], Transformer [81], BERT [82]) had . Natural models additionally use label smoothing of 0.2 (Table 5). The attention pool and the transformer layers use 8 attention heads.
Table 5: Training configuration.
| Item | Value |
| Input dimension | 171 for natural models (image embedding of size 128 + one-hot task embedding of size 43), 128 for abstract models |
| Hidden dimension | RNN: 256, All other models: 128 |
| Layers | 1 |
| Learning rate | : 3e-4 for Natural models, 3e-3 for Abstract models |
| : 1e-4 for Natural models, 3e-4 for Abstract models | |
| Label smoothing | 0.2 for natural models, none for abstract |
| Attention head | 8 |
| Optimizer | AdamW |
| Loss function | Cross-entropy loss |
| Epochs | 10 for Abstract models, 20 for Natural models |
| Train size | 256000 |
| Validation size | 64000 |
| Batch size | 128 |
| Weight decay | 1e-4 |
| Dropout | 0.1 for FC (Natural only), 0.3 for FC |
| Gradient clipping | 2.0 |
| GPU | 1 x NVidia H100 SXM5 (80GB) for Natural, None for Abstract |
| Execution time | 4 hours for Natural, 1 hour for Abstract |
Table 5.
Results
Natural models better replicate human behavioral responses
We first compare human behavioral alignment in Abstract and Natural models across 5 sequential architectures: RNN, GRU, LSTM, BERT (decoder only), and vanilla transformer (decoder only). In this section, we use ResNet-50 [37] pre-trained with CLIP [70] as the vision backbone. Abstract models reached significantly higher validation accuracy compared to Natural ones (0.93-0.99 vs. 0.70-0.81). To control for this difference, we select Abstract checkpoints with validation accuracy closest to 80%, matching the Natural models’ accuracy range on the MUTurk dataset. Across the 5 sequential architectures tested, final checkpoints do not have higher human consistency except for RNNs (Figure 13a).
Figure 13. Comparison of Abstract models with matched accuracy and final accuracy. (a) Correlation with human behavior. (b) Task accuracy. Empty orange bars are model checkpoints with 75% validation accuracy, orange bars with backward slashes are 80% matched accuracy, and orange bars with forward slashes are final model checkpoints. The blue bar is the ResNet-CLIP LSTM natural model. Although accuracy on the MTurk trials increases (right figure), the gap in behavioral alignment persists (left figure).
Regardless of the sequential architecture, models trained on naturalistic stimuli (both task trained and VLMs) have higher human consistency than Abstract models (Figure 2a). Natural models exceed their abstract counterparts by 0.199 (95% CI [0.170, 0.229], paired , ), and do so in all 18 task conditions (paired , ). Regressing image-based visual discriminability out of the human and model responses changes this gap by at most 0.019 (see A.2.6).
Figure 2. Comparison of Abstract and Natural models across sequential architectures. (a) Correlation with human behavior. Gray line and zone indicates human internal behavioral consistency, in Eq. (1). Bars show averaged over 20 random split-halves in Fisher- space; error bars show the standard deviation across split-halves. (b) Task accuracy. Mean and standard deviation reported as error bars, 6 seeds for models and 20 participants for humans. VLMs (left to right): nemotron-nano-12b-v2, nemotron-nano-12b-v2-no-thinking, qwen3-vl-30b, qwen3-vl-235b, and gemini-3.1-pro-preview (surpasses human-level accuracy).
To dissect this discrepancy, we perform feature-level analysis by correlating responses that only compare categories (30%), identities (33%), and positions (37%). Natural models are more human-consistent than Abstract models on every feature. Within the three features, Abstract models come closest on identity trials. VLMs are the most human-consistent on category trials, but fall below Natural models on identity trials (Figure 18). Notably, gemini-3.1-pro-preview exceeds human accuracy on every feature (on identity only marginally, 0.821 vs. 0.814). It has the lowest category consistency among the VLMs but the highest position consistency of any model except the Natural LSTM (0.369 vs. 0.381). This is in contrast to reports that VLMs struggle on spatial reasoning [83, 84, 85, 86, 87, 88, 69].

Figure 18. Feature-level human consistency and accuracy of trained task models and VLMs. For every architecture, the Natural model is more human-consistent than the Abstract on every feature, and both model types are most human-consistent on identity trials. Among Natural models, BERT and the Transformer lead on identity, the Transformer on category, and the LSTM on position. VLMs are the most human-consistent models on category trials (0.39–0.58) but fall below every Natural model on identity trials. On position trials, the human consistency of Abstract models is 0 for every architecture except BERT (0.01). We hypothesize that Abstract models solve position tasks from the one-hot position code, whereas Natural models, VLMs and humans judge position from the object’s location over an ambiguous grid on a naturalistic background. Abstract models are evaluated at the 80%-matched checkpoint; markers, error bars and the human reference follow Figure 2.
We correlate probability vectors across trials within each task family to examine how task paradigm affects human consistency (Figure 19). Natural LSTMs are the most human-like in InterDMS and DMS trials, Transformers lead on CtxDMS, and four Natural architectures are within 0.015 of each other on NBack. Among Abstract models, BERT has the highest human consistency in every task family. On DMS trials (DMS-position only) every Abstract model has zero human consistency.
Figure 19. Task-family human consistency and accuracy by sequence architecture. (a, b) NBack, (c, d) InterDMS, (e, f) CtxDMS and (g, h) DMS; left of each pair , right MTurk accuracy. For every architecture, the Natural model is more human-consistent than the Abstract in every task family. Among Natural models, the LSTM leads on InterDMS and DMS, and the Transformer on CtxDMS, while four architectures are within 0.015 of each other on NBack. Among Abstract models, BERT leads in every family. On DMS trials (the DMS-position task only) every Abstract model has zero correlation with humans. Abstract models are evaluated at the 80%-matched checkpoint; markers, error bars and the human reference follow Figure 2.
We performed RSA to identify which properties of a model’s response structure predict its consistency with humans. Cross-task RSA measures whether responses to a given stimulus pair are similar regardless of task type. We find that Natural models have much higher cross-task RSA alignment values than Abstract models (paired , ; Figure 11b). Cross-stimulus RSA, in contrast, measures how similarly a model responds to each stimulus pair (across the 12 tasks) relative to all other stimulus pairs. We find that it is also a strong proxy for human consistency (). Moreover, the human consistency measure was strongly correlated with the error-consistency metric [74] (Figure 10).
Figure 10. Correlation of behavioral metrics. Across models, human consistency is almost linearly related to error consistency (Pearson ) and much less to accuracy (), which motivates measuring model behavior beyond accuracy.

Figure 11. Cross-task and cross-stimulus RSA. (a) Human cross-task RSA matrix shows clusters of high correlation in tasks using the same stimuli features, e.g. 1back position and InterDMS position. Since identity tasks are related to category tasks, we also see correlation between the response vectors in these cross-task comparisons. (b) Correlation between cross-task RSA and human consistency. Abstract models (orange) have much lower alignment values than natural models (blue, green, and purple colors match the colors used in Figure 17). (c) Human cross-stimulus RSA matrix. If two stimuli are similar in category, identity, and position, then they would have high correlation across tasks. (d) Correlation between cross-stimulus RSA and human consistency. Cross-stimuli RSA alignment predicts human consistency better than cross-task RSA.
Sweeping over a range of vision encoder architectures, we find that vision encoders pretrained with CLIP [70] generally have higher human consistency and accuracy than their counterparts trained on ImageNet with supervision (Figure 17a). Overall, DINOv3 [89] is the most human-consistent vision encoder. As a negative control, a frozen, randomly initialized ResNet-CLIP barely learns the tasks (MTurk accuracy 0.52) and its human consistency drops to almost zero (see A.2.4). We also notice a positive correlation between the linear separability of a vision model’s features on the HvM dataset (linear SVM) and its human consistency on category tasks (Figure 12a). Beyond , increasing the hidden dimension raises the Natural LSTM’s accuracy but not its human consistency, and the Natural Transformer’s consistency decreases with size (Figure 17).
Figure 12. Feature-level linear-SVM accuracy and behavioral metric correlation.
Figure 17. Comparison of behavioral alignment across vision encoders, sequential architectures (LSTM vs. Transformer), hidden dimension sizes, and vision encoder sizes. Hatch patterns indicate pretraining method: empty bars for supervised ImageNet [36], forward slash for CLIP, backslash for DINOv1 [111], crossed diagonal lines for DINOv3 [89], and horizontal lines for masked image modeling (MIM) [112]. (a) Correlation with human behavior. (b) Task accuracy. (c) Effect of sequential module hidden dimension size on human alignment. (d) Effect of hidden size on task accuracy. Increasing model representation size increases accuracy, but does not increase behavioral alignment. For natural transformers, the human alignment declines steadily. (e) Effect of vision encoder size (number of parameters, log scale) on human alignment, for Natural LSTMs with ConvNeXt (supervised ImageNet-22k; Tiny to XLarge) and EVA02-CLIP (Base, Large, Enormous) encoders. (f) Effect of vision encoder size on task accuracy. Neither behavioral alignment nor accuracy increases consistently with encoder size.
To test whether simple properties of the natural input explain this behavioral gap, we matched the dimensionality and the visual difficulty of the Abstract input to those of the Natural input (see A.2.3). Both changes reduced the behavioral alignment of Abstract models. At similar validation accuracy, matching the input dimensionality lowered human consistency from 0.183 to 0.115, and adding noise to mimic perceptual uncertainty lowered it to zero (Figure 15). This suggests that the gap is not explained by input dimensionality or perceptual uncertainty alone.

Figure 15. Comparison of Abstract models with dimensionality and perceptual uncertainty matching. (a) Correlation with human behavior decreases as more noise is injected. (b) Task accuracy also decreases as more noise is injected. Each level shows 6 seeds at their final checkpoints, including seeds that did not learn the task. Responses are averaged over 20 noise draws.
Why do Natural models better replicate human behavior?
Sec. 4.1 shows that Natural models better capture human behavioral patterns, but the source of this gap remains unclear. We reason that input geometry and learned recurrent computations can each contribute to this gap. To disentangle them, we conduct substitution analyses on the LSTM models (ResNet-CLIP for Natural). We freeze modules trained on one modality, and retrain the remaining modules on the other. We denote the sequence modules of natively trained Abstract and Natural models as and . For example, a model loads a frozen and trains the input and output projections on naturalistic input. In , the output projection is also frozen, so only the input projection is trained. In , only the output projection is trained. Because Abstract and Natural inputs differ in dimensionality, substitution uses the dimensionality-matched Abstract models (see A.2.3). As a baseline, we also retrain the projections with a frozen, randomly initialized . All Abstract models in this section use final checkpoints rather than the accuracy-matched checkpoints. See Figure 14 for accuracy-matched substitution results.
Figure 14. Behavioral metrics of accuracy-matched Abstract and Natural models with substituted components. Same conditions and colors as Figure 3, but with accuracy-matched Abstract checkpoints. Models receiving abstract input (orange) are evaluated at the checkpoint closest to 80% validation accuracy, and the Abs → Nat substitutions (blue) load the sequence module of the 80%-matched Abstract model. (a) Human consistency . (b) Task accuracy on the MTurk trials. All trained models fall within 0.03 of each other in accuracy, except the untrained- Natural baseline. With accuracy controlled, the input still dominates behavior alignment. Every model receiving naturalistic input, including the untrained- baseline, is more consistent with humans than every model receiving abstract input. A natural barely changes abstract-input models: falls slightly below the native Abstract model (0.147 vs. 0.183), to the level of an untrained . Keeping the natural readout as well raises consistency only to 0.221, below every model receiving naturalistic input. The 80%-matched abstract supports higher consistency than an untrained but lower than , whereas the fully trained module in Figure 3 matches , so the contribution of an abstract sequence module grows with its training.
Surprisingly, the model matches the human consistency of the natively trained Natural model (blue triangle vs. blue circle in Figure 3a) and exceeds its task accuracy (Figure 3b). In the other direction, raises human consistency only slightly over the native Abstract model (orange triangle vs. red circle), and barely above an untrained (orange cross). Freezing as well further increases consistency on abstract input but lowers it on naturalistic input. also matches the native Natural model in cross-task and cross-stimulus RSA alignment, while all substitutions on abstract input remain well below it (Figure 20).
Figure 3. Behavioral metrics of Abstract and Natural models with substituted components. From left to right, warm-colored bars show , , and untrained ; cold-colored bars show , , and untrained . Abstract models use final checkpoints. (a) Correlation with human behavior. The gray line and band indicate human internal behavioral consistency, in Eq. (1). Bars show averaged over 20 random split-halves in Fisher-z space; error bars show the standard deviation across split-halves. On Abstract input, substitution improves human consistency only marginally (orange triangle and square vs. red circle: 0.138 and 0.188 vs. 0.082), barely above the untrained (0.126). On natural input, recovers the human consistency of the native model (blue triangle vs. circle: 0.446 vs. 0.452), well above the untrained (0.274), whereas reaches only 0.258 (blue square). (b) Task accuracy; mean and standard deviation over 6 seeds for models and 20 participants for humans. With an untrained , models still reach 0.78 MTurk accuracy on abstract input but only 0.61 on natural input.
Figure 20. Representational similarity analysis (RSA) of substituted models. The y-axis shows the Spearman correlation between the upper-triangular entries of the model and human representational dissimilarity matrices (RDMs).
In dimensionality-matched models, substitutions show the same asymmetry as above: approaches the native Natural model, while stays at the level of an untrained . Retraining only the input projections , square markers) recovers behavioral alignment and retains task accuracy. However, retraining only the output projection drops task accuracy to chance in both directions, and reduces human consistency (diamond markers, Figure 16).
Figure 16. Behavioral metrics of dimensionality-matched Abstract and Natural models with substituted components. Same conditions and colors as Figure 3, but with dimensionality-matched Abstract models and substitution (diamond markers). Bars below zero are not drawn.
In these experiments, human consistency follows the input modality more than the origin of the recurrent weights. For a model receiving naturalistic input, a trained is needed, but whether it was trained on abstract or naturalistic input matters little. For a model receiving abstract input however, an untrained already reaches 0.78 MTurk accuracy, yet no brings its human consistency to the level of Natural models. This asymmetry is consistent with how the two inputs represent the stimuli. Human errors are reliable at the level of single trials, so they depend on the specific stimuli. One-hot Abstract inputs make every stimulus equidistant, so no driven by them can reproduce these stimulus-specific errors, regardless of the input it was trained on. In contrast, a pretrained vision encoder represents stimuli visually with task-specific features mixed with background and viewpoint in the same feature space. A capable of solving the task inherits the geometry of this representation in its errors.
Under natural inputs, Abstract recurrent weights produce dynamics similar to Natural models
Neural networks trained on symbolic inputs have been widely used to model cognitive behaviors, particularly to inform how the neural dynamics of these networks (and potentially the brain) solve decision-making tasks. Our previous experiments showed that while Abstract models poorly replicate human behavior, their consistency improves when the projections are retrained on natural inputs. Two mechanisms are possible. Either the two models converge to similar recurrent computations, allowing trained on abstract input to transfer to natural input, or the retrained input projection compensates for differences in the recurrent computation, reaching human-like behavior through different neural dynamics. To distinguish between these two mechanisms, we used DSA to measure the distance between the models’ internal dynamics.
Figure 4b supports the first mechanism. When the Abstract is frozen inside a model that receives ResNet embeddings and only the input and output projections are retrained, its dynamics lie within the seed-to-seed variability of native Natural models, for and most other DSA settings (see A.2.8). The retrained projection alone does not produce this match in dynamics, since an untrained stays far from the native models. The dynamical similarity is also not inherited from the input, since the projected input of this model remains far from that of native Natural models (Figure 22b). This suggests that, when driven by natural inputs, the trained Abstract produces dynamics similar to those of Natural models, which may support the human-like behavior of this model. Freezing the readout as well breaks the match: the dynamics of stay above the seed-to-seed variability of native Natural models in every DSA setting we tested.
Figure 4. Dynamic distances of retrained models to native models. (a). In models that receive abstract input, and substitution lie above the noise floor at every dimension but well below the untrained . (b). In models that receive ResNet embedding input, from 5 dimensions on, substitution is at the noise floor, while and the untrained sit far above it, with farthest. Distances are between sequence-core hidden states. Gray bands and dashed lines indicate native models’ seed-to-seed distances, which is the noise floor. Error bars indicate standard error of mean over tasks and seed pairs.
Figure 22. Dynamic distances of retrained models to native models in the projected input. Each panel shows the DSA angular distance between each retrained model and the native models of the same input modality, computed on the projected input , against the number of PCA components. (a). In models that receive abstract input, is at the noise floor at every dimension, while and the untrained lie just above it. (b). In models that receive ResNet embedding input, lies at or just above the noise floor, is well above it, and the untrained is farthest. Closeness in does not predict closeness in the hidden state (Figure 4). With abstract input, is at the noise floor here but not in . With ResNet input, is far from native here but at the noise floor in , while shows the reverse.
The match of also holds in dimensionality-matched Abstract models, where, unlike with one-hot inputs, also stays within the seed-to-seed variability of native Natural models (Figure 23d). This parallels the behavioral alignment of in dimensionality-matched models (Figure 16). Substituting instead places the model far above the floor with either input, and the models perform at chance. A trained thus needs an input projection adapted to the new modality.
Figure 23. Dynamic distances of retrained dimensionality-matched models to native models. Each panel shows the DSA angular distance between each retrained model and the native models of the same input modality, against the number of PCA components, computed on the projected input (a, b) or the hidden state (c, d). (a). In models that receive dimensionality-matched abstract input, is at the noise floor in at every dimension and lies above it, while the untrained and are far above it, with farthest. (b). In models that receive ResNet embedding input, and are at the noise floor in from 5 dimensions on, while and the untrained are far above it, with the untrained farthest. (c). In of models that receive abstract input, and lie above the noise floor at every dimension but well below the untrained , and is farthest. (d). In of models that receive ResNet embedding input, and are at the noise floor from 5 dimensions on, while the untrained sits far above it and is farthest.
In models receiving abstract input, substitution recovers native dynamics partly. Natural driven by abstract inputs stays above the seed-to-seed variability of native Abstract models at every dimension (Figure 4a, Figure 23c), even when the projected input matches the native models (Figure 22a, Figure 23a).
Together, these results show that recurrent weights learned from symbolic inputs can work with sensory-realistic inputs and, given an input projection trained on them, produce dynamics that we cannot distinguish from those of models trained on natural inputs. This is consistent with the view that the behavioral gap between Abstract and Natural models depends more on their input representations than on their recurrent weights.
Discussion and Limitations
Through substitution analyses, we further showed that most of this improvement can be recovered by freezing the recurrent weights of an abstract sequence model and retraining only its input and output projections on naturalistic stimuli. This suggests that task input matters more for behavioral alignment than the input on which recurrent weights are trained. However, our experiments do not identify which properties of input representation are responsible.
Current models explained only a portion of the reliable structure in human behavior; the best-performing models accounted for 64% of the behavioral consistency observed across human participants. The remaining gap likely reflects factors absent from the present framework such as reward-history-dependent adaptation [90], metacognitive monitoring [91], individual differences in learning strategy [92], and the linguistic and conceptual priors accumulated through a lifetime of naturalistic experience [93, 94]. Closing this gap might require models that integrate sensory realism with richer learning histories, more structured inductive biases, and more flexible forms of adaptation.
Notably, large VLMs reproduced a substantial fraction of human behavioral consistency despite receiving no task-specific training, suggesting that broad multimodal pretraining produces representations that transfer non-trivially to structured cognitive demands [95, 96]. Fine-tuning such models on larger collections of compositional cognitive tasks may provide a practical route toward closing the remaining behavioral gap. More broadly, future work should examine whether in-context adaptation [97, 98], continual learning [99], or reinforcement learning from environmental feedback [100] better approximate the conditions under which human-like cognitive behavior is acquired.
Limitations. Several limitations should be addressed in future work. We derived the human behavioral consistency from a relatively small participant pool of 20 per task. While this sample size is sufficient for reliable group-level estimates, larger and more demographically diverse cohorts would yield more stable measures of human behavioral consistency and enable analyses of individual differences in model-human alignment.
Our comparison also changes several factors at once. Natural models use vision encoders pre-trained on large image datasets, and with a frozen random encoder they do not learn the tasks (Appendix § 7.2.4), so we cannot separate the contribution of pretraining from image input. Training Abstract models on inputs that gradually move toward the vision encoder space of each would directly test this account. Also, our tasks are binary match/non-match judgments over a single stimulus set, so the line between perceptual and cognitive contributions is not sharp.
Although we attempted to eliminate extrinsic differences between abstract and natural input by matching their dimensionality and visual difficulty (see A.2.3), it remains unclear what intrinsic geometric properties of sensory realism drive behavioral alignment. Our dimensionality-matched control uses random up-sampling projections that keep the one-hot encoding in an equidistant simplex, whereas natural images are known to occupy a low-dimensional manifold within a high-dimensional pixel space [101, 102].
Future work could address this with class-conditional optimal transport, morphing each one-hot encoding toward the empirical distribution of natural embeddings of its category. This would replace our binary comparison with a dose-response curve of behavioral alignment against sensory realism.
Furthermore, behavioral alignment alone cannot determine whether models and humans rely on similar underlying computations. Future work should couple behavioral metrics with comparisons of brain and model neural dynamics. Fixed-point analyses could reveal whether naturalistic training reshapes the attractor structures underlying task performance [66, 103], while representational similarity analysis, centered kernel alignment, and linear probing could identify where and how task-relevant variables emerge and align with neural population responses [104, 105, 106]. Together, these analyses would clarify whether sensory realism merely improves behavioral fit or also steers models toward more human-like computational mechanisms.
Acknowledgments and Disclosure of Funding
We thank the Amazon Mechanical Turk participants who took part in our experiments, and the reviewers and area chair, whose feedback improved this paper. This research was supported by the NSERC Discovery grant RGPIN-2021-03035 and the CIHR Project Grant PJT-191957. P.B. was supported by the FRQ-S Research Scholars Junior 1 grant 310924 and the William Dawson Scholar award. X.L. was supported by a CAMBAM fellowship, and L.G. by a Masters Excellence Scholarship from UNIQUE (Unifying Neuroscience and Artificial Intelligence in Québec). This research was enabled in part by support provided by Calcul Québec (calculquebec.ca) and the Digital Research Alliance of Canada (alliancecan.ca), and by funding from the Canada Foundation for Innovation (project 42730). The funders had no role in study design, data collection and analysis, the decision to publish, or the preparation of the manuscript. The authors declare no competing interests.
Appendix
Tasks
Here, we detail our task space, task generation pipeline, sensory-realistic task stimuli, and human task evaluation.
Task Description
We define 43 DMS tasks that compare three stimulus features: category, identity, and position. The tasks are organized by their structure into four families (Table 1). For a complete breakdown of each task, see Table 6.
NBack (tasks 1–9). A sequence of six frames is shown. At each step , the model must report whether frame matches frame on the target feature, yielding up to decision frames per trial (see Figure 6).
InterDMS (tasks 10–36). Two independent DMS sub-tasks (see Figure 7), one for feature and one for feature , are interleaved across four stimulus frames according to one of three temporal patterns: AABB (both frames precede both frames), ABAB (frames alternate between the two sub-tasks), and ABBA (the sub-task is nested inside the sub-task).
CtxDMS (tasks 37–40). Three frames are presented. The first comparison (frames 1 and 2) tests a context feature . Based on whether the context match is positive or negative, the second comparison (frames 2 and 3) tests one of two possible test features ( or ), requiring the model to flexibly route its decision (see Figure 8).
DMS (tasks 41–43). Two frames are presented. The model observes a sample stimulus and must judge whether the immediately following test stimulus matches on the target feature (position, identity, or category, see Figure 9).
HvM Dataset
At training time, trials are generated on-the-fly. For each frame, the appropriate feature constraints are applied as a Boolean mask over all eligible HvM rows, and a stimulus is sampled uniformly from the resulting candidate set. With 50% probability, the target feature value is preserved from the reference frame (match); otherwise a new value is drawn that differs on that feature (non-match), while all unconstrained features vary freely.
The image below illustrates the three variation levels in the High-Variation Model (HvM) dataset [68] used for the behavioral and computational experiments.
Variation-level balancing. Variation level 6 contains more images per object than levels 0 and 3, and would otherwise be sampled at a disproportionately high rate. To counteract this and reduce sampling bias toward variation level 6, we assign sampling weights of 0.5 to all var-6 candidates relative to 1.0 for other variation levels, making var-6 images approximately half as likely to be drawn.
Task generation. Each task is defined by a fixed logic that determines which stimulus pairs are compared and along which feature dimension. For instance, InterDMS–ABBA–position–category compares the position of the first and last stimuli (response on frame 4), and also the category of the second and third stimuli (response on frame 3). The predefined task logic allows us to sample the ground truth action first (match vs. no-match), and then find stimuli that satisfy the task conditions. This generation procedure guarantees balanced action labels and ensures the diversity of stimulus features.
Figure 5. Examples of the HvM stimuli seen by sensory-realistic models at 3 variation levels. Variation 0 (var-0): No variation. Objects are centered, with a fixed neutral view angle and fixed scale. Variation 3 (var-3): Medium variation. Objects undergo moderate translation, rotation, and scaling. Variation 6 (var-6): High variation. Objects undergo large translation, rotation, and scaling, presenting the most challenging visual condition.
Human Trials
The 12 MTurk task IDs are: {1, 2, 3, 19, 23, 27, 28, 32, 36, 37, 40, 41}. On the interactive website for human trial collection, each frame was shown for a fixed duration, and participants respond via their keyboard ( = match; = non-match) at the response frames. Before the main session, for each task, participants completed a learning session (2 trials for 1Back, 3 for DMS, 5 for InterDMS, and 8 for CtxDMS tasks; 11 early CtxDMS participants received 5) with per-frame on-screen task instructions and response feedback.
From 16 objects (2 per category), we randomly selected 100 images: 6 per object plus 1 more for 4 of the objects. Of these, 95 are high-variation images (31 from variation 3 and 64 from variation 6) and 5 are variation-0 images (centered, unscaled) from 5 unique objects. To enable direct comparison across all 12 tasks, we construct a single set of 100 unique stimulus pairs that are jointly balanced for category, identity, and position match distributions (50% match / 50% non-match for each feature). NBack tasks additionally require temporally chained pairs; for those tasks we greedily construct chains that approximately maintain feature balance.
To enforce quality control, participants whose accuracy on the main session fell below 65% (approximately above chance level of 50%) were rejected and excluded from analysis. For task 40 (CtxDMS: category-identity-position), which is the most demanding variant, the threshold was relaxed to 60% due to its high difficulty (16% of recruited workers met the standard threshold). MTurk workers who passed this quality control received $5 per task.
Algorithm 1 Split-Half Consistency Evaluation
Require: Model logits , human binary responses
Ensure: Consistency scores
for each split indices and do
end for
return
Additional Results
Human consistency
Rajalingham et al. [73] define a noise-corrected human consistency as:
where is the Pearson correlation between model and human response vectors, correlation between model split-halves, and is the Pearson correlation between human split-halves. We report the uncorrected throughout, clipped at zero after averaging over split-halves (, so a reported 0 means ), and show as its ceiling. is not used. Statistical tests and the residualization analysis use the unclipped values.
We perform 20 random split-halves, compute , , , take the Fisher -transform of each correlation, and average across splits. Since we are primarily interested in the behavioral alignment between models and humans and less interested in the model’s internal reliability across seeds, we visualize the uncorrected directly in Sec. 4, and show as the noise-correction ceiling.
For significance tests, we correlate each seed’s response probabilities with the mean human response, average over seeds within a model, and pair over the five sequence architectures or the 18 task conditions.
Accuracy matched vs. final Abstract model
When training Abstract models, we realized their validation accuracies quickly approached 100% and their logits were almost deterministic. Since human consistency is computed on a model’s per-trial confidence, comparing humans against the final model checkpoints is unfair. To account for this, we performed the analyses in Sec. 4.1 using intermediate model checkpoints with around 80% validation accuracy (most model seeds reached this in the first training epoch). In Figure 13, we show that regardless of the selected model checkpoint, the behavioral gap still exists between abstract and natural models. In sections 4.2 and 4.3 however, we performed analyses with the final model checkpoints. This is because the Abstract models have not yet formed stable recurrent dynamics in the first epoch. Using DSA, we find that their dynamics are close to that of a random .
Dimensionality and visual difficulty matching
To identify the key properties of sensory realism that drive human behavioral alignment, we ran additional experiments, isolating the effect of extrinsic and intrinsic properties.
To assess the effect of dimensionality, we up-projected the one-hot embedding into the same number of dimensions as the vision encoder . For instance, ResNet CLIP produces image tokens of dimension , so our pipeline applies
a fixed random projection , an independent map for each of the 49 tokens. The resulting abstract embedding thus matches in the number of dimensions. This also has the benefit of connecting the up-projected abstract input to the identical attention pool mechanism in natural models (which produces ), and allows both models to share the same input projection ().
To match the perceptual uncertainty of sensory-realistic stimuli, we add Gaussian noise to the up-projected tokens of every stimulus frame, resampled at each presentation, with standard deviation , where is the RMS of the stimulus’s normalized rotation and scale, , and is the noise level. These models are evaluated at their final checkpoints, since the noise level affects their accuracy, and each trial’s response is averaged over 20 seeded noise draws.
At comparable validation accuracy (about 0.75), projecting the one-hot input to the natural dimensionality lowers human consistency from (one-hot) to 0.115 (dimensionality-matched), and adding noise (, final validation accuracy 0.76) lowers human consistency to 0, against 0.452 for the Natural LSTM. Across noise levels (Figure 15), falls from 0.151 without noise to 0.050 at , where MTurk accuracy matches humans (0.779 vs. 0.777). At higher noise levels, some seeds never leave a plateau near 0.62 validation accuracy, so the two highest noise levels mostly reflect failed training.
Frozen, randomly initialized vision encoder
To test how much the Natural models rely on pretrained visual features, we replaced the pretrained ResNet-CLIP encoder of Natural LSTM with its default timm random initialization (shared by all six seeds) and kept it frozen. All other modules were trained with the same recipe (Table 5). These models barely learn the tasks (Table 2). Their final validation accuracy is 0.596 ± 0.009, against 0.771 ± 0.029 with the pretrained encoder, and MTurk accuracy is 0.516 ± 0.012. Their human consistency is near zero overall (): 0 on category and identity trials and 0.101 on position trials. Frozen random features are therefore insufficient for the downstream modules to learn the tasks.
Effect of architecture on behavioral consistency
Using the timm package [107], we train and evaluate LSTMs and Transformers using various frozen vision encoders. To test whether a larger vision encoder brings Natural models closer to human behavior, we trained Natural LSTMs (, 6 seeds) on two families of frozen encoders of increasing size (Figure 17e). The first is ConvNeXt [108] pretrained with supervision on ImageNet-22k [38], in five sizes from 28M (Tiny) to 348M (XLarge) parameters. All five share the pretraining data, recipe and input resolution (), so only the encoder size changes. The second is EVA02-CLIP [109, 110] at Base (86M), Large (303M) and Enormous (4.3B) sizes.
Human consistency does not increase with encoder size in either family. Across a 12-fold range of ConvNeXt sizes, stays between 0.363 and 0.388 and MTurk accuracy between 0.683 and 0.701, with no consistent trend. For EVA02-CLIP, decreases slightly from 0.438 (Base) to 0.415 (Large) and 0.405 (Enormous), while MTurk accuracy peaks at Large (0.723, 0.756 and 0.730). The encoder family matters more than its size. At nearly equal size, EVA02-CLIP Base reaches against 0.388 for ConvNeXt Base, and it exceeds every ConvNeXt, including XLarge.
The two families differ in both architecture and pretraining objective, so this comparison does not isolate either factor, although it agrees with the advantage of CLIP pretraining in Figure 17a. All eight
| Encoder | Val. acc. | MTurk acc. | All | Category | Identity | Position |
| Pretrained | 0.771 ± 0.029 | 0.706 ± 0.021 | 0.452 | 0.329 | 0.534 | 0.381 |
| Random | 0.596 ± 0.009 | 0.516 ± 0.012 | 0.025 | 0.000 | 0.000 | 0.101 |
Table 2. Natural LSTM with a frozen pretrained or a frozen randomly initialized ResNet-CLIP encoder (6 seeds). Accuracy: mean ± SD over seeds; : mean over 20 split-halves.
encoders remain far below the human consistency ceiling (), and none exceeds the ResNet-CLIP LSTM used in the main text ().
Visual discriminability does not explain the behavioral gap
Natural models and humans see the same images, so their agreement could reflect shared stimulus difficulty rather than shared task computation. To test this, we regressed an image-based estimate of each trial’s visual discriminability out of every human and model response vector and recomputed . For trial , let be the cosine similarity between the two stimuli in the feature space of a vision encoder. A similar pair is easy to call a match but hard to call a non-match, so within each of the 18 task conditions we sign by the ground truth and center it within trial type, , where is the correct answer and is the mean similarity of trials of that type in the condition. Centering makes orthogonal to the match/non-match label, so the regression removes differences in visual uncertainty between stimulus pairs but keeps the task structure differences between match and non-match trials. Within each condition, each human subject’s response vector and each model seed’s response vector are replaced by its residual , with . Since depends only on the stimuli, humans and models are projected identically, while each responder’s slope is fitted separately. Removing the per-condition intercept changes by itself, so we compare against a null in which is permuted within each condition before the same residualization (10 permutations), and report the effect of discriminability as the residualized gap minus the null gap.
| Covariate | Unregressed | Residualized | Null | Effect | |
| ResNet-CLIP | 0.029 | 0.229 | 0.249 | 0.252 ± 0.001 | −0.003 |
| DINOv3 | 0.042 | 0.229 | 0.233 | 0.252 ± 0.002 | −0.019 |
Table 3. Natural minus Abstract difference in (mean over the five sequence models; Abstract models at the 80%-matched checkpoint) before and after regressing out visual discriminability. Null: mean ± SD over 10 within-condition permutations of the covariate. Effect: residualized minus null.
We computed with two encoders: ResNet-CLIP and DINOv3. The covariate explains (ResNet-CLIP) and 0.042 (DINOv3) of the variance in per-trial human accuracy (mean over conditions), and the two similarity estimates correlate at across trials. Table 3 reports the difference between the mean of the five Natural and the five Abstract sequence models before residualization. Relative to the null, ResNet-CLIP discriminability changes this gap by and DINOv3 discriminability by (8% of the gap); per feature, the largest change is on category trials. With the final Abstract checkpoints, the gap (0.291) changes by and . Visual discriminability, as estimated by either encoder, thus accounts for at most a small part of the behavioral gap.
RSA
Following the same split-half procedure for human consistency, for each random split-half, we average each partitioned group’s response, resulting in a pair of vectors of shape . We then calculate the average pairwise correlation either over the task space, or over the stimulus space, producing , . We can then measure alignment with humans by taking the Pearson correlation between the upper triangles of the human and model . Since we use the same 100 stimulus pairs for evaluation, given a stimulus pair, a human-like
model would produce similar logits in InterDMS-AA-Position and 1-Back-Position tasks, but dissimilar logits in InterDMS-AA-Position and 1-Back-Category tasks.
Figure 21 shows the cross-task matrices of the human participants, the Natural LSTM and the Abstract LSTM, and the difference of each model’s matrix from the human one.
Figure 21. Cross-task response structure of humans and LSTM models. (a) Human cross-task matrix (Appendix § 7.2.7); rows and columns are the 18 task conditions, in the same order in every panel. (b, d) of the Natural LSTM (ResNet-CLIP) and of the Abstract LSTM at its 80%-matched checkpoint, on the same scale as (a). (c, e) Model minus human, on the shared scale at the right. Green cells are condition pairs the model correlates more strongly than humans do, pink cells less strongly. The diagonal is the split-half reliability of each condition, not 1.
DSA
For each task and model seed, we project the states onto their top principal components (time steps as samples, no whitening) and report results for . We then apply Hankel alternative view of Koopman (HAVOK) [76] with 2 delays and rank 3, selected by a preliminary grid search over delays and ranks . Finally, we compute the angular Procrustes distance between the operators of every pair of model seeds after optimal orthogonal alignment (Adam, 200
iterations, learning rate . Unlike the Wasserstein distance between eigenvalue spectra, which at rank 3 compares only three eigenvalues, it compares the full operators. We report the mean SEM over the 12 tasks and all seed pairs. Note that similar dynamics should not be read as similar recurrent weights, since different weights can produce similar dynamics and similar weights can produce different dynamics under different inputs.
Figure 24 shows the pairwise DSA distances between the native and substituted models of § 4.3 (untrained omitted), averaged over the 12 tasks.

Figure 24. Pairwise DSA distances between native and substituted models. (a) DSA pairwise distance heatmap on . (b) DSA pairwise distance heatmap on . (c) MDS [113] applied to the pre-computed pairwise distance matrix of . (d) MDS applied to the pairwise distance matrix of .
Sensitivity to the DSA settings. We repeated the analysis of § 4.3 in 19 configurations, varying one setting at a time around the default above with : ; 1 or 3 delays; rank 2 or 4; the Euclidean Procrustes distance or the Wasserstein distance between eigenvalue spectra instead of the angular distance; 500 or 1000 iterations at learning rate , or 1000 iterations at or ; and random seeds 0, 7, 42, and 2025 instead of 2026. We only kept configurations that our trials support: with delays, a 6-frame trial gives one-step transitions, so the rank cannot exceed , which rules out ranks above 4 with 2 delays and above 3 with 3 delays. We did not vary PCA whitening. HAVOK fits the operator on the leading right singular vectors of the Hankel matrix, which form an orthonormal basis regardless of whether the input components are whitened, so whitening changes only which rank-3 subspace is kept. Because whitening gives every component unit variance, that subspace no longer follows the dominant directions of the trajectories, so whitened DSA measures a different quantity, and one that depends on .
For each comparison and configuration, we test the per-task distances with a one-sided paired Wilcoxon signed-rank test over the 12 tasks (), where the distance between two models is the mean over all pairs of their seeds and a model’s seed floor is the mean over pairs of its own seeds. A model is within a floor when its distance is not significantly above it, which is a failure to reject rather than evidence of equivalence. The optimizer settings and random seeds change every task-averaged distance by at most 0.012 and leave every comparison in Table 4 unchanged relative to the default setting, so the table and the counts below cover the other 11 configurations.
Table 4: DSA sensitivity, number of configurations in which each comparison holds. Each column varies one setting around the default setting of the main text (, 2 delays, rank 3, angular distance), which is counted once, in the column. The floor is the mean distance between seeds of the native model of the input modality; “within” means not significantly above it (one-sided paired Wilcoxon signed-rank test over the 12 tasks, ). The 8 optimizer and seed configurations give the default outcome in every row and are not shown.
| Comparison | Feature | 2–50 | Delays 1, 3 | Rank 2, 4 | Distance Eucl./Wass. | Total |
| Natural input | ||||||
| within floor | 4/5 | 1/2 | 1/2 | 2/2 | 8/11 | |
| Untrained farther than | 5/5 | 2/2 | 1/2 | 1/2 | 9/11 | |
| above floor | 5/5 | 2/2 | 0/2 | 2/2 | 9/11 | |
| within floor | 3/5 | 1/2 | 1/2 | 1/2 | 6/11 | |
| Dim.-matched , within floor | 4/5 | 1/2 | 1/2 | 2/2 | 8/11 | |
| Abstract input | ||||||
| within floor | 5/5 | 2/2 | 1/2 | 2/2 | 10/11 |
Table 4.
Under natural input, stays within the Natural floor in 8 of the 11 configurations and leaves it with , 3 delays or rank 2, where its distance above the floor is 21–60% of the untrained ’s (11% at the default setting); the dimensionality-matched and fail in the same three. An untrained stays farther from the native models in 9 (all but rank 2 and the Wasserstein distance), and the projected input of stays above the floor in 9 (all but ranks 2 and 4). is above the floor in all 11, but its projected input is within the floor in only 6. The dimensionality-matched is above the floor in all 11 for both input modalities, but it is not consistently farther from the native models than an untrained : with natural input the comparison is borderline ( at the default setting, 5 of 11 configurations and 5 of the 8 optimizer and seed reruns), and with abstract input it holds in 2 of 11. Under abstract input, is above the Abstract floor in all 11, and its projected input is within that floor in 10 (all but rank 2). It is closer to the native Abstract models than an untrained on average (0.300 vs. 0.350 at the default setting) but not consistently across tasks (1 of 11), whereas the dimensionality-matched is closer in 8 of the 11 configurations.
Training Settings
We detail training configuration in Table 5. For dimensionality matched and visual difficulty matched Abstract models, their input dimension is 171 and they are trained for 10 epochs; everything else stayed the same as Natural models. Substituted models use the training recipe of the native models of the modality they are retrained on (e.g., uses the Abstract recipe).
VLM Evaluation Details
We evaluated 4 state-of-the-art models: Gemini 3.1 Pro Preview [114], Qwen3 VL 235B A22B Thinking, Qwen3 VL 30B A3B Thinking [115], and Nemotron Nano 12B 2 VL [116]. All models were accessed via OpenRouter API. All 4 models were tested in reasoning mode, while Nemotron was additionally evaluated without reasoning by adding no-thinking to the system prompt for comparison against the reasoning baseline.
To ensure a fair and rigorous comparison between human participants and Vision-Language Models (VLMs), we designed an evaluation pipeline that closely emulated the interactive, sequential nature of the behavioral experiment. This section details the prompting structure and trial emulation methods used for all evaluated VLMs.
VLMs were provided with the same task instructions as human participants. For all the trials, images and instructions from previous steps were kept in the context of the VLMs. To ensure that the models strictly adhered to the two-alternative forced-choice format, we implemented an iterative response filtering and retry mechanism: any trial where the VLM generated a response at an action frame that was not exactly “x” or “b” was removed and rerun until the model successfully produced a correctly formatted response.
Sequential Trial Emulation
Unlike standard VLM benchmarks where an entire task is presented in a single prompt, our pipeline presented each trial sequentially. For each trial, the VLM maintained a continuous context window across the sequence of frames.
At the start of every trial, the model was initialized with a System Prompt that contained the global task instructions. Then, for each frame in the sequence ():
The model was provided with the visual stimulus (the image for frame , encoded in base64).
The model was provided with a Per-Frame Instruction (text prompt) corresponding to the current step in the trial logic.
These text prompts were sourced identically from the instructions provided to human participants during their interactive learning sessions. At action frames (where a response was required), the model was forced to output its decision based on the history of frames observed so far.
Prompting Structure
System Prompts The system prompt served as the global task instruction, mirroring the “Welcome Screen” text shown to human participants before beginning the experiment. To format this for VLMs, we made minor adjustments (e.g. replacing “Press X” with “Respond ’x’”). The model was explicitly constrained to output only the target characters.
For example, the system prompt for the Contextual Delayed Match-to-Sample (Category Object/Position) task was:
You will perform a Contextual Delayed Match-to-Sample task.
Each trial consists of a sequence of 3 images. You will make two comparisons, the second comparison depends on the result of the first:
Category match:
Compare the 2nd frame with the 1st frame.
Remember if they are from the same category (for example, both planes).
Identity/Position match:
Compare the 3rd frame with the 2nd frame.
If first comparison was a match, then compare their IDENTITY:
Otherwise, compare their POSITION:
If they match according to the relevant feature, respond ’x’.
Otherwise, respond ’b’.
The categories include Cars, Planes, Boats, Fruits, Faces, Animals, Chairs, and Tables.
Only output ’x’ or ’b’.
(A similar structure was used across all 12 tasks, accurately reflecting their specific task rules and contingencies).
Per-Frame Instructions To emulate the learning phase, the VLM was given explicit, step-by-step instructions alongside each image frame. These per-frame instructions guided the model on whether it should encode the current stimulus into working memory or execute a comparison.
For instance, in an Interleaved Delayed Match-to-Sample ABAB Position–Position trial, the 4 frames were presented sequentially with the following instructions:
Frame 1 (A):
Image: [Stimulus Image 1]
Instruction: Remember the POSITION of this object.
Frame 2 (B):
Image: [Stimulus Image 2]
Instruction: Remember the POSITION of this object.
Frame 3 (A - Action Frame):
Image: [Stimulus Image 3]
Instruction: Compare POSITION with the FIRST image. Respond ’x’ if same, ’b’ if different.
Frame 4 (B - Action Frame):
Image: [Stimulus Image 4]
Instruction: Compare POSITION with the SECOND image. Respond ’x’ if same, ’b’ if different.
By maintaining this strict sequential history and providing per-frame behavioral anchors, we ensured the VLMs experienced the temporal and cognitive demands of the tasks as close to human participants as possible.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction explicitly state the paper’s main contributions and clearly delineate the scope of the work.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We acknowledge that the analysis does not exhaustively cover all possible model variants, and the experiments are limited by available computational resources, which may restrict the breadth of evaluations.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [N/A].
Justification: The paper does not include theoretical results.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides all necessary details to reproduce the main experimental results, including model architectures, training procedures, hyperparameters, and evaluation protocols. Additionally, we release the code and data to further facilitate reproducibility.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release the full codebase along with the data and accompanying instructions to reproduce all experimental results in the camera-ready version.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: All training and evaluation details, including data splits, hyperparameters, optimization strategies, and model configurations, are specified in the methods section and further elaborated in the appendix, providing sufficient information to understand and interpret the reported results.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistical analyses are performed wherever possible, with the methodology and sources of variability clearly described in the corresponding sections.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The paper provides sufficient information on the computational resources used for each experiment.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: The authors have reviewed and strictly adhered to the NeurIPS Code of Ethics in all aspects of this research. MTurk workers were compensated fairly for their time.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: The paper poses no such risk.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets, code, and models used in this paper are properly credited with appropriate citations, and their respective licenses and terms of use are acknowledged and respected. We will use CC-BY 4.0.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: The paper does not release new assets.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The paper includes full details of the platform, behavioral task instructions given to participants, compensation information, and relevant screenshots, all provided in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper does not pose potential risks, IRB approval has been obtained.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: The core method development does not involve LLMs as an original components.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
References
- [1]F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386–408, 1958. ISSN 1939-1471. doi: 10.1037/h0042519. Place: US.DOI
- [2]Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193–202, 1980. ISSN 03401200. doi: 10.1007/BF00344251. Number: 4 ISBN: 0340-1200.DOI
- [3]J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558, April 1982. doi: 10.1073/pnas.79.8.2554. URL https://www.pnas.org/doi/abs/10.1073/pnas.79.8.2554.DOI
- [4]Teuvo Kohonen. Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43(1):59–69, January 1982. ISSN 1432-0770. doi: 10.1007/BF00337288. URL https://doi.org/10.1007/BF00337288.
- [5]Chr. von der Malsburg. Self-Organization of Orientation Sensitive Cells in the Striate Cortex. Kybernetik, 14:85–100, 1973.
- [6]David J. Freedman, Maximilian Riesenhuber, Tomaso Poggio, and Earl K. Miller. Categorical Representation of Visual Stimuli in the Primate Prefrontal Cortex. Science, 291:312–317, 2001. doi: 10.1126/science.291.5502.312. Number: 2001.DOI
- [7]David J. Freedman, M Riesenhuber, T Poggio, and E K Miller. A comparison of primate prefrontal and inferior temporal cortices during visual categorization. J Neurosci, 23(12):5235–5246, 2003. URL http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=12832548. Number: 12.
- [8]Sruthi K Swaminathan and David J Freedman. Preferential encoding of visual categories in parietal cortex compared with prefrontal cortex. Nature Neuroscience, 15(2):315–320, February 2012. ISSN 1097-6256, 1546-1726. doi: 10.1038/nn.3016. URL http://www.nature.com/articles/nn.3016. Number: 2.DOI
- [9]Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(7474):78–84, 2013.DOI
- [10]Warasinee Chaisangmongkon, Sruthi K. Swaminathan, David J. Freedman, and Xiao Jing Wang. Computing by Robust Transience: How the Fronto-Parietal Network Performs Sequential, Category-Based Decisions. Neuron, 93(6):1504–1517.e4, 2017. ISSN 10974199. doi: 10.1016/j.neuron.2017.03.002. URL http://dx.doi.org/10.1016/j.neuron.2017.03.002. Number: 6.
- [11]Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297–306, 2019.DOI
- [12]H. Francis Song, Guangyu R Yang, and Xiao-Jing Wang. Training excitatory-inhibitory recurrent neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biology, 12(2):e1004792, 2016.DOI
- [13]Randall C. O’Reilly and Michael J. Frank. Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18(2): 283–328, 2006. ISSN 08997667. doi: 10.1162/089976606775093909. Number: 2.DOI
- [14]Qihong Lu, Uri Hasson, and Kenneth A Norman. A neural network model of when to retrieve and encode episodic memories. eLife, 11:e74445, feb 2022. ISSN 2050-084X. doi: 10.7554/eLife.74445. URL https://doi.org/10.7554/eLife.74445.
- [15]Gustav Theodor Fechner. Elemente der Psychophysik. Breitkopf und Härtel, Leipzig, 1860. 2 vols.
- [16]David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, 1962. doi: 10.1113/jphysiol.1962.sp006837.DOI
- [17]F. W. Campbell and J. G. Robson. Application of Fourier analysis to the visibility of gratings. The Journal of Physiology, 197(3):551–566, 1968. doi: 10.1113/jphysiol.1968.sp008574.DOI
- [18]William T. Newsome, Kenneth H. Britten, and J. Anthony Movshon. Neuronal correlates of a perceptual decision. Nature, 341(6237):52–54, 1989. doi: 10.1038/341052a0.DOI
- [19]International Brain Laboratory. A brain-wide map of neural activity during complex behaviour. Nature, 645(8079):177–191, 2025. doi: 10.1038/s41586-025-09235-0.DOI
- [20]Christopher J. Cueva, Alex Saez, Encarni Marcos, Aldo Genovesio, Mehrdad Jazayeri, Ranulfo Romo, C. Daniel Salzman, Michael N. Shadlen, and Stefano Fusi. Low-dimensional dynamics for working memory and time encoding. Proceedings of the National Academy of Sciences, 117(37):23021–23032, September 2020. doi: 10.1073/pnas.1915984117. URL https://www.pnas.org/doi/full/10.1073/pnas.1915984117.DOI
- [21]Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619–8624, 2014.DOI
- [22]Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib J Majaj, and James J DiCarlo. Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS computational biology, 10(12): e1003963, 2014.DOI
- [23]Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS computational biology, 10(11): e1003915, 2014.DOI
- [24]Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356–365, 2016.DOI
- [25]N. Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J DiCarlo, and Nancy Kanwisher. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications, 12(1):5540, 2021.
- [26]Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H McDermott. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630–644, 2018.DOI
- [27]Yuanning Li, Gopala K Anumanchipalli, Abdelrahman Mohamed, Peili Chen, Laurel H Carney, Junfeng Lu, Jinsong Wu, and Edward F Chang. Dissecting neural computations in the human auditory pathway using deep neural networks for speech. Nature Neuroscience, 26(12):2213–2225, 2023.DOI
- [28]Peter Y Wang, Yi Sun, Richard Axel, LF Abbott, and Guangyu Robert Yang. Evolving the olfactory system with machine learning. Neuron, 109(23):3879–3892, 2021.
- [29]Farzaneh Taleb, Miguel Vasco, Antonio Ribeiro, Mårten Björkman, and Danica Kragic. Can transformers smell like humans? Advances in Neural Information Processing Systems, 37:72032–72060, 2024.
- [30]Charlotte Cau cheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022.DOI
- [31]Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2105646118, 2021.
- [32]Joséphine Raugel, Stéphane d’Ascoli, Jérémy Rapin, Valentin Wystar, and Jean-Rémi King. Scaling and context steer LLMs along the same computational path as the human brain. arXiv preprint arXiv:2512.01591, 2025.arxiv.org/abs/2512.01591
- [33]Christopher J Bates, George A Alvarez, and Samuel J Gershman. Scaling models of visual working memory to natural images. Communications Psychology, 2(1):3, 2024.
- [34]Xiaoxuan Lei, Takuya Ito, and Pouya Bashivan. Geometry of naturalistic object representations in recurrent neural network models of working memory. Advances in Neural Information Processing Systems, 37:100604–100629, 2024.arxiv.org/abs/2411.02685
- [35]Paul I Jaffe, Gustavo X Santiago-Reyes, Robert J Schafer, Patrick G Bissett, and Russell A Poldrack. An image-computable model of speeded decision-making. Elife, 13:RP98351, 2025.
- [36]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
- [37]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- [38]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
- [39]Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition, April 2015. URL http://arxiv.org/abs/1409.1556. arXiv:1409.1556 [cs].
- [40]Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper with Convolutions, September 2014. URL http://arxiv.org/abs/1409.4842. arXiv:1409.4842 [cs].
- [41]Daniel L.K. Yamins, Ha Hong, Charles Cadieu, and James J. DiCarlo. Hierarchical Modular Optimization of Convolutional Networks Achieves Representations Similar to Macaque IT and Human Ventral Stream. Advances in neural information processing systems, 26(February):1–9, 2013. ISSN 10492558. doi: 10.5465/AMBPP.2014.13529abstract. Number: February ISBN: 1884902049.
- [42]David H Hubel and Torsten N Wiesel. Receptive fields of single neurones in the cat’s striate cortex. The Journal of physiology, 148(3):574, 1959.
- [43]David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):215–243, 1968.
- [44]Anitha Pasupathy and Charles E. Connor. Responses to contour features in macaque area V4. Journal of Neurophysiology, 82(5):2490–2502, 1999. doi: 10.1152/jn.1999.82.5.2490. URL https://doi.org/10.1152/jn.1999.82.5.2490. PMID: 10561421.
- [45]A Pasupathy and C E Connor. Shape representation in area V4: position-specific tuning for boundary conformation. Journal of Neurophysiology, 86(5):2505–2519, 2001. ISSN 0022-3077. doi: 10.1152/jn.2001.86.5.2505. Number: 5 ISBN: 0022-3077.DOI
- [46]Charles G Gross, C E Rocha-Miranda, and D B Bender. Visual properties of neurons in inferotemporal cortex of the Macaque. Journal of Neurophysiology, 35:96–111, 1972. ISSN 0022-3077. doi: 10.1152/jn.1972.35.1.96. ISBN: 0022-3077.DOI
- [47]Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception. Journal of Neuroscience, 17(11):4302–4311, June 1997. ISSN 0270-6474, 1529-2401. doi: 10.1523/JNEUROSCI.17-11-04302.1997. URL https://www.jneurosci.org/content/17/11/4302.DOI
- [48]Motahareh Pourrahimi and Pouya Bashivan. Neural signatures of associational cortex emerge in a goal-directed model of visual search. bioRxiv, 2025.DOI
- [49]Johannes Mehrer, Ben Lonnqvist, Anna Mitola, Paolo Papale, and Martin Schrimpf. Model-guided microstimulation steers primate visual behavior. In The Fourteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2510.03684
- [50]Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364(6439):eaav9436, 2019.
- [51]Edgar Y. Walker, Fabian H. Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G. Fahey, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 2019. ISSN 15461726. doi: 10.1038/s41593-019-0517-x. URL http://dx.doi.org/10.1038/s41593-019-0517-x.
- [52]Eshed Margalit, Hyodong Lee, Dawn Finzi, James J DiCarlo, Kalanit Grill-Spector, and Daniel LK Yamins. A unifying framework for functional organization in early and higher ventral visual cortex. Neuron, 112(14):2435–2451, 2024.DOI
- [53]Amirozhan Dehghani, Xinyu Qian, Asa Farahani, and Pouya Bashivan. Credit-based self organizing maps: training deep topographic networks with minimal performance degradation. In The Thirteenth International Conference on Learning Representations, 2024.
- [54]Xinyu Qian, Amir Ozhan Dehghani, Asa Borzabadi Farahani, and Pouya Bashivan. Local lateral connectivity is sufficient for replicating cortex-like topographical organization in deep neural networks. Nature Communications, 2026.
- [55]Yifei Ren and Pouya Bashivan. How well do models of visual cortex generalize to out of distribution samples? PLOS Computational Biology, 20(5):e1011145, May 2024. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1011145. URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011145.DOI
- [56]Eric Y. Wang, Paul G. Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa A. Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, Dat Tran, Jiakun Fu, Casey M. Schneider-Mizell, Nuno Maçarico da Costa, R. Clay Reid, Forrest Collman, Nuno Maçarico da Costa, Katrin Franke, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, Fabian H. Sinz, and Andreas S. Tolias. Foundation model of neural activity predicts response to new stimulus types. Nature, 640(8058):470–477, April 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-08829-y. URL https://www.nature.com/articles/s41586-025-08829-y.DOI
- [57]Stephen M. Kosslyn, Thomas M. Ball, and Brian J. Reiser. Visual images preserve metric spatial information: Evidence from studies of image scanning. Journal of Experimental Psychology: Human Perception and Performance, 4(1):47–60, 1978. doi: 10.1037/0096-1523.4.1.47.DOI
- [58]Stephen M. Kosslyn. Image and Mind. Harvard University Press, Cambridge, MA, 1980.
- [59]Stephen M. Kosslyn, William L. Thompson, and Giorgio Ganis. The Case for Mental Imagery. Oxford University Press, New York, 2006.
- [60]Zenon W. Pylyshyn. What the mind’s eye tells the mind’s brain: A critique of mental imagery. Psychological Bulletin, 80(1):1–24, 1973. doi: 10.1037/h0034650.DOI
- [61]Zenon W. Pylyshyn. Mental imagery: In search of a theory. Behavioral and Brain Sciences, 25(2):157–182, 2002. doi: 10.1017/S0140525X02000043.DOI
- [62]Arup Sarma, Nicolas Y. Masse, Xiao Jing Wang, and David J. Freedman Freedman. Task-specific versus generalized mnemonic representations in parietal and prefrontal cortices. Nature Neuroscience, 19(1):143–149, 2015. ISSN 15467126. doi: 10.1038/nn.4168. Number: 1.DOI
- [63]Nicolas Y. Masse, Guangyu R. Yang, H. Francis Song, Xiao-Jing Wang, and David J. Freedman Freedman. Circuit mechanisms for the maintenance and manipulation of information in working memory. Nature Neuroscience, 22(7):1159–1167, July 2019. ISSN 1546-1726. doi: 10.1038/s41593-019-0414-3. URL https://www.nature.com/articles/s41593-019-0414-3.DOI
- [64]Laura N. Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 27(7):1349–1363, July 2024. ISSN 1546-1726. doi: 10.1038/s41593-024-01668-6. URL https://www.nature.com/articles/s41593-024-01668-6.DOI
- [65]Jelena Sučević and Anna C Schapiro. A neural network model of hippocampal contributions to category learning. eLife, 12:e77185, December 2023. ISSN 2050-084X. doi: 10.7554/eLife.77185. URL https://doi.org/10.7554/eLife.77185.
- [66]David Sussillo and Omri Barak. Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural computation, 25(3):626–649, 2013.
- [67]N. J. Majaj, H. Hong, E. A. Solomon, and J. J. DiCarlo. Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance. Journal of Neuroscience, 35(39):13402–13418, 2015. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.5181-14.2015. URL http://www.jneurosci.org/cgi/doi/10.1523/JNEUROSCI.5181-14.2015. Number: 39 ISBN: 1529-2401.DOI
- [68]Rishi Rajalingham, Kailyn Schmidt, and James J DiCarlo. Comparison of object recognition behavior in human and monkey. Journal of Neuroscience, 35(35):12127–12136, 2015.
- [69]Xiaoxuan Lei, Lucas Gomez, Hao Yuan Bai, and Pouya Bashivan. iWISDM: Assessing instruction following in multimodal models at scale. In Vincenzo Lomonaco, Stefano Melacci, Tinne Tuytelaars, Sarath Chandar, and Razvan Pascanu, editors, Proceedings of The 3rd Conference on Lifelong Learning Agents, volume 274 of Proceedings of Machine Learning Research, pages 457–480. PMLR, 29 Jul–01 Aug 2025. URL https://proceedings.mlr.press/v274/lei25a.html.arxiv.org/abs/2406.14343
- [70]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
- [71]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer Normalization. arXiv preprint arXiv:1607.06450, 2015. arXiv: 1607.06450v1.arxiv.org/abs/1607.06450
- [72]Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
- [73]Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. Journal of Neuroscience, 38(33):7255–7269, 2018.
- [74]Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. Advances in neural information processing systems, 33:13890–13902, 2020.arxiv.org/abs/2006.16736
- [75]Mitchell Ostrow, Adam Eisen, Leo Kozachkov, and Ila Fiete. Beyond geometry: Comparing the temporal structure of computation in neural circuits with dynamical similarity analysis. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 33824–33837. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/6ac807c9b296964409b277369e56521a-Paper-Conference.pdf.
- [76]Steven L Brunton, Bingni W Brunton, Joshua L Proctor, Eurika Kaiser, and J Nathan Kutz. Chaos as an intermittently forced linear system. Nature communications, 8(1):19, 2017.
- [77]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101.
- [78]D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation, page 673–695. MIT Press, Cambridge, MA, USA, 1988. ISBN 0262010976.
- [79]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9 (8):1735–1780, 1997.
- [80]Kyunghyun Cho, Bart Van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1724–1734, 2014.
- [81]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
- [82]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019.
- [83]Zihan Weng, Lucas Gomez, Taylor Whittington Webb, and Pouya Bashivan. Caption this, reason that: VLMs caught in the middle, 2025. URL https://arxiv.org/abs/2505.21538.
- [84]Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li. Why is spatial reasoning hard for VLMs? an attention mechanism perspective on focus areas, 2025. URL https://arxiv.org/abs/2503.01773.
- [85]Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models, 2024. URL https://arxiv.org/abs/2406.14852.
- [86]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. ISSN 2307-387X. doi: 10.1162/tacl_a_00566. URL http://dx.doi.org/10.1162/tacl_a_00566.
- [87]Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning, 2023. URL https://arxiv.org/abs/2310.19785.
- [88]Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li. An empirical analysis on spatial reasoning capabilities of large multimodal models, 2024. URL https://arxiv.org/abs/2411.06048.
- [89]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025.
- [90]Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):408–422, 2019.
- [91]Stephen M Fleming and Raymond J Dolan. The neural basis of metacognitive ability. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1594):1338–1349, 2012.
- [92]Anne GE Collins and Michael J Frank. Within-and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory. Proceedings of the National Academy of Sciences, 115(10):2502–2507, 2018.
- [93]Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
- [94]Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011.
- [95]Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721, 2023.
- [96]Andrew K I Lampinen, Nicholas Roy, Ishita Dasgupta, Stephanie CY Chan, Allison Tam, James Mcclelland, Chen Yan, Adam Santoro, Neil C Rabinowitz, Jane Wang, et al. Tell me why! explanations support learning relational and causal structure. In International Conference on Machine Learning, pages 11868–11890. PMLR, 2022.
- [97]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- [98]Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in neural information processing systems, 35:30583–30598, 2022.
- [99]Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028–1040, 2020.
- [100]Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature neuroscience, 21(6):860–868, 2018.
- [101]Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Hk4_qw5xe.
- [102]Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
- [103]Matthew D Golub and David Sussillo. FixedPointFinder: A Tensorflow toolbox for identifying and characterizing fixed points in recurrent neural networks. Journal of open source software, 3(31):1003, 2018.
- [104]Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.
- [105]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMLR, 2019.
- [106]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.arxiv.org/abs/1610.01644
- [107]Ross Wightman. PyTorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
- [108]Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976–11986, 2022.
- [109]Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024.
- [110]Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023.arxiv.org/abs/2303.15389
- [111]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
- [112]Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=p-BhZSZ59o4.
- [113]Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1–27, 1964.
- [114]Google DeepMind. Gemini 3.1 pro. https://deepmind.google/models/model-cards/gemini-3-1-pro/, February 2026. Model card. Released February 19, 2026 (Preview).
- [115]Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
- [116]NVIDIA. NVIDIA nemotron nano v2 vl, 2025. URL https://arxiv.org/abs/2511.03929.















