betaFeedback

WorksComputer Science

Recurrent Vision Language Models Solve the Shell Game

Abstract

Tracking a hidden object through a video is effortless for humans, yet it remains a bottleneck for vision language models (VLMs). In the shell game, where a ball is hidden under one of several identical cups that are then swapped, VLMs that answer directly stay near chance, thinking models fail on long games, and fine-tuning learns only the lengths it has seen. We argue that the limit is serial computation: each swap must be composed with the last result, and a VLM of fixed depth has only a fixed number of state transitions for it. To address this, we propose RecurrentVLM, which makes every attention layer of a pretrained VLM recur over time by computing the keys and values of finished frame groups from the layer's own output. With a single answer token, RecurrentVLM reaches 100% on VET-Bench and 99.6% on MET-Bench, and 99% on 50-swap games after training on at most 30 swaps. We also observe several interesting phenomena: length generalization emerges suddenly at an aha moment; without recurrence, each swap costs an attention layer; the ball's state forms in the first four attention layers; and recurrence keeps the latest state at the cost of details of earlier moments. These results suggest that recurrent VLMs are a potential solution for tracking the states of unobserved objects in video. Code will be released.

Introduction

Understanding a video often requires knowing where things are when they cannot be seen. Humans keep track of hidden objects from infancy [5], and video assistants, embodied agents, and world models need the same ability to follow objects through occlusion [77, 79]. For vision language models (VLMs), which have advanced rapidly on video benchmarks [4, 70], tracking unobserved objects is thus a basic test of how well these models understand what happens in a video. Yet VLMs still fail this test. The shell game is its minimal form: a ball is hidden under one of several identical cups, the cups are swapped many times, and the viewer must say where the ball is (Figure 1). Since the cups look the same, no single frame shows the answer. Answering directly, closed and open VLMs alike stay near the 33% chance level of VET-Bench [53] and MET-Bench [16]. Although they see every frame, they still cannot tell where the ball ends up after the swaps.

Figure 1. Current VLMs struggle with the shell game. Claude-Opus-5.5 thinks for thousands of tokens and still errs; fine-tuned Qwen3.5-4B drifts on games longer than in training. RecurrentVLM stays correct at every swap with one token.

Existing remedies follow three routes, each with its own weakness. Thinking models write the ball’s position as text [60, 83]: Claude-Opus-5.5 solves the 5-swap cup games with about 1,000 tokens per video but drops to 34% with 16 swaps. Agents call external trackers [86], but the VLM itself still lacks the ability, and trackers cannot follow objects that jump between frames. Fine-tuning Qwen3.5-4B on the task solves the short benchmark games but stays at 37% and 38% on 16-swap cup games and 50-swap shell games. In short, none lets the VLM follow the ball reliably and efficiently.

This raises a question: do VLMs fail because they lack data, or because their structure lacks something? Fine-tuning shows that data alone is not enough; the structure may be the limit, because the computation is serial: each swap permutes the cups, and the answer is the composition of all permutations. A transformer of fixed depth performs only a bounded number of state transitions per token, too few to compose arbitrarily many permutations [51, 61], and linear recurrences with eigenvalues in [0,1][0, 1], such as the Gated DeltaNet (GDN) layers [88] of Qwen3.5, cannot track permutations either [33]. A human, in contrast, keeps the ball’s position in mind and updates it at every swap, a state transition over time. This suggests that what VLMs lack is not data but such state transitions.

To test this hypothesis, we propose RecurrentVLM, which performs these state transitions inside its attention layers. In every attention layer, RecurrentVLM computes the keys and values of finished frame groups from the layer’s own output instead of its input, following the Recurrent Transformer [65] but recurring once per frame group rather than once per token, so each layer passes its state forward at every frame group while the tokens of a group stay parallel. Frame-wise supervision, which asks for the ball’s position after every frame group, further helps the model learn these transitions. Trained with the same data and losses as the fine-tuned Qwen3.5-4B, RecurrentVLM answers with a single token and reaches 100% on VET-Bench and 99.6% on MET-Bench, and keeps 91% and 99% on these longer games.

Detailed analyses further reveal several interesting phenomena: length generalization emerges suddenly at an aha moment; without recurrence, each swap costs an attention layer; the ball’s state forms in the first four attention layers; and recurrence keeps the latest state but overwrites details of earlier moments.

In summary, our contributions are as follows:

  • We propose RecurrentVLM, a simple yet effective VLM that performs state transitions over time inside its attention layers, enabling it to track unobserved objects.

  • RecurrentVLM achieves state-of-the-art accuracy on VET-Bench and MET-Bench with a single answer token and generalizes to games longer than in training.

  • We provide in-depth analyses of why recurrence works, how it emerges, and what it costs, offering insights for future VLMs that track the states of unobserved objects.

Related Work

Video language models. VLMs have become the main tool for video understanding: they connect a visual encoder to a language model [2, 52], extend it to videos [13, 42, 47, 50, 55], and read a video as a sequence of frame tokens [4, 15, 28, 70, 81]. Video benchmarks cover broad skills [25, 46], causal and temporal actions [85], and questions that a single frame cannot answer [17, 54, 76], where TOMATO finds a 57.3-point gap between humans and the best model. They reason with chain of thought [60, 83], step supervision [49, 63], latent reasoning [21, 31, 36], or agents that reason over clips and crops [48, 57]. On the shell game, SGCoT [53] fine-tunes Molmo2 [15] to write the cup trajectory, MET-Bench trains Mamba3 [28] with reinforcement learning, and Track, Then Bind [86] calls SAM 2. Yet all of them rely on text or external tools, whereas RecurrentVLM tracks hidden objects within the VLM itself.

Visual state tracking. Visual state tracking asks a model to follow hidden states that change over time. Synthetic videos test compositional and causal reasoning [30, 89], trackers learn object permanence through occlusion [75, 79, 82], the Perception Test [66] films real cup games, and video world models are tested on unobserved states [77]. VET-Bench [53] and MET-Bench [16] hide the ball under identical cups, cards, or shells and find VLMs near chance. Memories for long videos [6, 12, 78] and trackers [14, 72] store what was seen, not a state that depends on every swap. Probes and causal interventions [1, 26, 58] locate board states [45, 80], entity states [41, 43, 69], and permutation algorithms [44], and loss drops after plateaus [37, 68] have been traced to forming circuits [62, 64] or hidden progress [8]. In contrast, this paper studies how a VLM that answers directly can learn to track such hidden states over the course of a video.

Recurrence in transformers. Recurrence lets a model carry a state over time. Without it, fixed-depth transformers compute only shallow circuits per token [35, 59, 61] and learn shortcuts that break on longer sequences [20, 51] unless the data format and position encoding help [3, 56, 91]; linear recurrences share this limit [73] unless their eigenvalues may be negative [33]. Segment-level recurrence and memory [11, 18, 22, 23, 39, 71, 84] carry hidden states across segments, fixed-size recurrences [9, 34, 40, 67, 87, 88] replace attention, and test-time memories update their weights while reading [10, 90]. Looped and recurrent-depth models [19, 24, 27, 29, 32, 74] repeat layers in depth instead of in time, and recurrent networks extrapolate by running more iterations [7]. The Recurrent Transformer [65] computes past keys and values from each layer’s output. RecurrentVLM extends this recurrence to video: rather than recurring once per token on text, it recurs once per frame group inside a pretrained VLM.

Method

RecurrentVLM adds a recurrent KV cache to the attention layers of a pretrained VLM (Figure 2) and is trained with frame-wise supervision (Figure 3).

Figure 2. Recurrent KV cache. (a) A standard attention layer of the LLM caches keys and values of its input tokens. (b) We cache keys and values of the layer’s own output, once per frame group, so every attention layer becomes a recurrence over time. (c) Each query reads keys and values of earlier tokens’ inputs in (a), but of earlier frame groups’ outputs in (b).

Figure 3. Frame-wise supervision. Each frame group ends with its final token, <|vision_end|>. In training, its output must name the ball’s current position (boxed), and the output after the question is the answer; at test time, only the answer is generated.

Setting. A video is a sequence of frame groups G1,…,GTG_1, \ldots, G_T followed by a question qq. A frame group holds the visual tokens of two consecutive frames, which the vision encoder of Qwen3.5 merges into one temporal patch (0.25 s at 8 fps); on MET-Bench, each swap image forms one group. We build on Qwen3.5-4B, whose language model stacks 24 Gated DeltaNet (GDN) layers [88] and 8 attention layers, one after every three GDN layers; we number the attention layers 1 to 8.

Recurrent KV cache. To carry a state over time, each attention layer reads its own past outputs. Let xix_i and ziz_i be the input and output of an attention layer at token ii, where ziz_i follows both the attention and the MLP with their residuals. A standard layer computes keys and values from its input, ki=WKxˉik_i = W_K\bar{x}_i and vi=WVxˉiv_i = W_V\bar{x}_i, where xˉi\bar{x}_i is the normalized input. Information from a past frame therefore reaches a later token only through the inputs of deeper layers, and the number of state transitions it can undergo is bounded by the depth (Figure 2a). Inspired by the Recurrent Transformer [65], once a frame group GsG_s ends, RecurrentVLM recomputes its keys and values from the layer’s own output with the same projections, forming the recurrent KV cache,

k~j=WKzˉj,v~j=WVzˉj,j∈Gs,(1)\tilde{k}_j = W_K\bar{z}_j,\qquad\tilde{v}_j = W_V\bar{z}_j,\qquad j \in G_s, \tag*{(1)}

keeping the rotary position of token jj. A query in group GtG_t attends to the recurrent KV cache of all earlier groups and to the standard keys and values of its own group, so the output of GtG_t depends on the outputs of earlier groups in the same layer,

zGt=f(xGt,zG1,…,zGt−1).(2)z_{G_t} = f\left(x_{G_t}, z_{G_1}, \ldots, z_{G_{t-1}}\right). \tag*{(2)}

Each attention layer thus updates its state once per frame group, and the number of state transitions grows with the video, not with the depth (Figure 2b). Within a group, tokens are still processed in parallel.

Frame-wise supervision. Frame-wise supervision is an auxiliary loss that helps the model learn the recurrence, since the final answer alone gives only one training signal per video. We therefore also supervise the last token of every frame group, <|vision_end|>, whose output must name the ball’s current position sts_t, e.g. 3, 1, …, 2 in Figure 3, while the output after the question is the answer aa,

L=−log⁡p(a∣G≤T,q)−∑t=1Tlog⁡p(st∣G≤t).(3)\mathcal{L} = -\log p(a \mid G_{\leq T}, q) - \sum_{t=1}^{T} \log p(s_t \mid G_{\leq t}). \tag*{(3)}

The baseline uses this loss as well, so the two models differ only in the recurrent KV cache.

Training and inference. Training needs neither a curriculum nor full backpropagation through time. Gradients through the recurrent KV cache stop after 8 groups, while the forward pass carries the state through all of them. The number of swaps is sampled uniformly, so training teaches one step of the rule and the recurrence repeats it. At inference, RecurrentVLM answers with a single token, so the state is never written out as text.

Experiments

We answer four questions in turn: does recurrence solve the shell game (§ 4.2), why (section 4.3), how it emerges (section 4.4), and what it costs (section 4.5)?

Setup

Benchmarks. VET-Bench [53] has 50 cup and 50 card videos with 5 swaps, read at 8 fps; MET-Bench [16] has 500 shell games of 10 swap images, scored with its official prompt and parser. For longer games, we render cup videos with 16 swaps, Cup×16, with the VET-Bench generator and cut the first 100 MET-Bench test games at 30, 50, and 100 swaps, of which Shell×50 is the main one. All questions have three options (chance 33.3%).

Training and baseline. Training videos come from the official VET-Bench generator (1–8 swaps of cups or cards) and the MET-Bench training split (1–30 swaps), mixed at

0.35, 0.15, and 0.5; no test video is used. The baseline is Qwen3.5-4B fine-tuned with the same data and losses but without recurrence, trained for 5,000 steps as it converges more slowly; RecurrentVLM is trained for 2,500. Both use LoRA [38] (rank 16), learning rate 10−410^{-4} with cosine decay, and 4 videos per step on 4 RTX A5000 GPUs; all models answer directly, and unparsed answers are wrong.

Main results

We test RecurrentVLM against existing methods, on longer games, and with more cups and balls.

Recurrence solves the shell game. We compare RecurrentVLM with closed, open, fine-tuned, and tool-using models on VET-Bench, MET-Bench, and the longer games (Table 1). Models that answer directly only guess, with 15–42% on every set, around the 33% chance level. Thinking reaches 84–100% on VET-Bench but pays with hundreds of tokens per video: Claude-Opus-5.5 still falls to 34% and 40% on the longer games, and GPT-6 Sol holds up at 78% and 97% with 608 tokens. Track, Then Bind [86] solves VET-Bench with tool calls but not shell games.

MethodSourceTypeVET-Bench CupVET-Bench CardMET-Bench ShellLonger Cup×16Longer Shell×50Cost ↓ Tokens
Random––33.333.333.333.333.3–
Closed models
Claude-Opus-5Anthropic 2026direct answer32.038.036.442.032.011
Claude-Opus-5.5Anthropic 2026thinking (high)96.010097.634.040.0984
GPT-6 SolOpenAI 2026direct answer30.026.034.032.035.09
GPT-6 SolOpenAI 2026thinking (high)84.092.099.078.097.0608
Open models
Qwen3-VL-8B [70]Tech. rep. 2025direct answer22.028.033.034.015.0549
Qwen3-VL-32B [70]Tech. rep. 2025direct answer30.030.033.638.033.0846
Molmo2-8B [15]Tech. rep. 2026direct answer26.040.031.431.0–4
GLM-4.6V-Flash [81]Tech. rep. 2025direct answer30.034.032.028.034.067
Gemma3-4B [28]Tech. rep. 2025direct answer26.028.032.826.035.06
Qwen3.5-4BModel card 2026direct answer32.040.030.841.032.06
Fine-tuned or tool-using
Molmo2-SGCoT [53]EMNLP 2026trajectory CoT, SFT92.090.033.670.033.0368
Gemma3-4B [16]ICML 2026direct answer, GRPO––90.4–––
Open-o3-Video-v2-8B [57]arXiv 2026RL crop tool26.030.036.832.029.0325
VideoINSTA [48]EMNLP-F 2024clip-reasoning agent42.028.0–33.343.3–
Track, Then Bind [86]arXiv 2026SAM2 tracking agent100100–83.3–982
Baseline (Qwen3.5-4B)Oursdirect answer, SFT10010091.637.038.01
RecurrentVLM (Qwen3.5-4B)Oursdirect answer, SFT10010099.691.099.01

Table 1. Table 1 Recurrence solves the shell game. Accuracy (%) of existing methods on VET-Bench [53], MET-Bench [16], and longer games. Only RecurrentVLM stays above 90 on both longer sets. VET-Bench of Open-o3-Video, VideoINSTA and Track, Then Bind is from Yang et al. [86], MET-Bench of the GRPO-trained Gemma3 from Cohen and Mooney [16]; all else is ours. Baseline: Qwen3.5-4B fine-tuned on our data without recurrence. Tokens: generated per VET-Bench video. Bold: best per column; –: not available or not applicable.

After supervised fine-tuning, the baseline learns the short games, with 100% on VET-Bench and 91.6% on MET-Bench, but stays at 37% and 38% on Cup×16 and Shell×50. RecurrentVLM reaches 99.6% on MET-Bench and 91% and 99% on these longer games. After fine-tuning, RecurrentVLM thus solves both benchmarks and their longer games with a direct, single-token answer.

Length generalization. Solving the test games could still mean fitting their lengths, so we test up to three times the training length. Trained on up to 30 shell swaps, RecurrentVLM reaches 98%, 99%, and 84% with 30, 50, and 100 shell swaps (Figure 4d). The baseline does not even fit its own training range: it is near chance at 30 shell swaps (Figure 4d) and on Cup×16 (Table 1). Recurrence extrapolates to longer games, and fine-tuning does not.

Figure 4. Length generalization emerges with an aha moment. (a, b) Training loss and accuracy on shell games: RecurrentVLM shows an abrupt drop in loss within 300 steps (shaded), an aha moment also seen in multi-step tasks [37], while the baseline improves slowly. (c) Accuracy on 50 swaps, longer than in training, jumps right after the aha moment. (d) Accuracy against the number of swaps (gray: training lengths). Fine-tuned and zero-shot: Qwen3.5-4B.

More cups and more balls. A single ball is the simplest state, so we also hide 1–3 balls under 3 or 5 cups, ask about one ball after the video, and supervise every ball after each swap (Table 2). Trained only on these shell games, with twice the shell data of Table 1, both models answer at least 90% up to the 30 swaps seen in training. At 50 swaps, the baseline falls to 20–44%, while RecurrentVLM keeps 87–100%. Recurrence also extrapolates with more cups and more balls.

SwapsModel3 cups, balls5 cups, balls
123123
30Qwen3.5-4B (fine-tuned)909194959390
RecurrentVLM100100100969893
50Qwen3.5-4B (fine-tuned)384432292520
RecurrentVLM10099100879292

Table 2. More cups and more balls. Accuracy (%) on our MET-Bench-style shell games with 3 or 5 cups and 1–3 balls, 100 games each (random: 33.3 and 20.0); both models train on the same mix of all settings, with up to 30 swaps per training game.

Observation 1. Recurrence learns the rule. Direct answers guess, thinking pays with hundreds of tokens, and fine-tuning fits only the lengths it has seen. Recurrence carries what it learns to longer games and more objects.

Why does recurrence solve the shell game?

We mainly explore this question with the following three experiments. First, we compare different recurrence mechanisms on the shell game. Second, we probe the ball position layer by layer across the attention layers. Third, we observe how the training loss of both models changes.

Different recurrence mechanisms. We add each mechanism to the baseline and train it with the same data and losses (Table 3). Running the model twice, as a Looped Transformer [19, 24], adds depth but no state transitions over time and reaches only 57.4% on MET-Bench. Negative eigenvalues let the GDN layers [33] solve MET-Bench (98.8%) but not the longer games (30% and 49%). Memory tokens passed between frame groups [11] cost 6.8× per step and stay at chance on VET-Bench and the longer games; per-token recurrence [65] would be 50× slower.

MethodApplied toRecurrent stateTrain timeBenchmarksLonger games
VETMETCup×16Shell×50
Random–––33.333.333.333.3
Qwen3.5-4B (fine-tuned)–GDN, eigenvalues in [0,1][0, 1]1.0×10091.637.038.0
+ Looped Transformer [19, 24]whole model, twicenone1.8×10057.431.042.0
+ Negative-eigenvalue GDN [33]24 GDN layersGDN, eigenvalues in [−1,1][-1, 1]1.0×94.098.830.049.0
+ Recurrent Memory Transformer [11]input tokensmemory tokens6.8×36.069.635.034.0
+ Recurrent Transformer [65]8 attention layerslayer outputs, per token∼50×––––
+ RecurrentVLM8 attention layerslayer outputs, per frame group2.5×10099.691.099.0

Table 3. Table 3 Different recurrence mechanisms. Accuracy (%) on the benchmarks and longer games of Table 1. Each row adds one mechanism to the baseline, Qwen3.5-4B (fine-tuned), trained with the same data and losses. Recurrent state: what is carried over time beyond the KV cache; Train time: per step, relative to the baseline; per-token recurrence is about 50× slower and not trained (–).

Recurrence over frame groups reaches the longer games at 2.5× the cost per step and with half the steps. More depth is thus no substitute for state transitions over time.

Probing the ball across attention layers. To see where each model holds the ball, we train a linear probe, a linear classifier on the output of one layer, to read the ball position after each swap of 100-swap shell games, for every layer of both models; the probes are fit on the first 30 swaps and tested on all 100 (Figure 5a,Figure 5b).

Figure 5. Figure 5 Probing the ball across attention layers. (a, b) Linear probes reading the ball position from every layer output (three GDN layers between attention layers) after each swap of 100-swap games, fit on the first 30 (inset: swaps 1–10). RecurrentVLM reads the ball after attention layer 4 at all swaps, while the baseline needs one more attention layer per about two swaps, which limits it to 10 swaps. (c) Memory transfer: midway through game A, we copy part of game B’s recurrent state and count answers switching to B.

Interestingly, we observe that in the baseline, the first layer from which the ball can be read moves one attention layer deeper after about every two swaps (correlation 0.96), and after about ten swaps no layer is left. In RecurrentVLM, the ball is readable after attention layer 4 at all 100 swaps (94–96% on average, 87% over the last 10), and its per-swap accuracy, the accuracy of its frame-wise answers, is 95.6%, against only 41.7% for the baseline.

We hypothesize that, without recurrence, each swap is combined with the last result one attention layer higher (Figure 6). Each swap thus costs the baseline an attention layer, while recurrence keeps the ball in the same attention layer at every one of the 100 swaps.

Figure 6. Figure 6 Our hypothesis of why depth runs out. Each swap is combined with the last result. RecurrentVLM keeps it in one layer; the baseline reads earlier swaps one layer lower, so the result climbs and swap 6 finds only raw swaps (dashed).

An aha moment during training. Finally, we check how the training loss changes during training (Figure 4a–Figure 4c). We observe that the loss of RecurrentVLM on shell games stays on a plateau for about 1,200 steps and then drops to near zero within 300 steps, while its accuracy on unseen 50-swap games jumps from 35% at step 1,000 to 86% at step 1,500 and 99% at step 2,500. The loss of the baseline decreases slowly, and its accuracy stays at 30–38% on 50-swap games for all 5,000 steps. We call this jump the aha moment: length generalization emerges suddenly, not gradually over training.

Figure 7. Tracking the aha moment. (a) Over training: share of swaps on which the frame-wise answer equals the model’s own last answer updated by the current swap (black), and per-swap accuracy on swaps 31–50 (red). (b) Accuracy on 50-swap games with one head of attention layer 4 switched off; any single head of attention layer 3 or 5 keeps at least 60%. (c) Per-swap accuracy with head 2 of attention layer 4 off.

Figure 8. Shallow recurrence suits continuous motion; deep recurrence carries jumps further. Accuracy after 2,500 steps with recurrence in only 4 of the 8 attention layers (shallow, middle, deep: 1–4, 3–6, 5–8).

Observation 2. Recurrence adds state transitions over time, not depth. Without recurrence, each swap costs an attention layer, so the ball is lost after about ten swaps. Recurrence gives every frame group its own state transition, and this ability emerges suddenly at an aha moment during training.

What happens at the aha moment?

To understand this emergence, we focus on the aha moment (figure 7). First, we ablate the components of RecurrentVLM. Second, we track the model before and after the aha moment. Third, we probe inside attention layer 4. Finally, we locate where the learned state is stored.

Ablating components. We first ask which components the aha moment needs (table 4a, b, figure 8). Without frame-wise supervision, RecurrentVLM stays at chance, and without the recurrent KV cache it learns only the short cup games within 2,500 steps. With recurrence in only 4 of the 8 attention layers, the shallow and middle layers learn the cup games, which move continuously, but not the shell games, which jump by a whole swap per frame group, and the deep layers learn neither fully (figure 8). The aha moment thus needs frame-wise supervision and recurrence in all eight attention layers during training.

VariantVETMETCup×16Shell×50
RecurrentVLM10099.691.099.0
(a) no recurrent KV cache90.032.035.037.0
(b) no frame-wise supervision36.033.835.032.0
(c) recurrent KV cache off34.034.840.030.0
(d) keep last 8 frame groups98.099.697.091.0
(e) keep last 4 frame groups38.091.837.055.0

Table 4. Ablating components. Accuracy (%) after 2,500 steps, one change to RecurrentVLM per row: (a, b) in training, (c–e) only at inference; (a) is the baseline of Table 1 at step 2,500. Shaded: unseen lengths.

Tracking the aha moment. We then follow the frame-wise answers through training (figure 7a). The share of swaps on which the frame-wise answer equals the model’s own last answer updated by the current swap rises from 28% at step 1,000 to 92% at step 1,240, before accuracy follows; the baseline stays at 41%.

Six of the 16 heads of attention layer 4 carry this reuse: switching off any one of them drops the accuracy on 50-swap games to 28–57%, and the ball is lost after the first swaps (figure 7b, c). In short, the model first learns to reuse its own last answer and update it with the current swap, through these six heads of attention layer 4.

Probing inside attention layer 4. If the update rule is in place by step 1,240, long-game accuracy should already be high, so we look inside attention layer 4 (Table 5, Figure 9). At the aha moment, attention layers 3–7 shift toward the current and previous frame groups; in attention layer 4, the share on the previous frame group rises from 16% before to 29% after the aha moment.

Figure 9. Attention moves to the latest frame groups at the aha moment. Share (%) of image-token attention on the same frame group (0), the previous three, and older ones (gray), per attention layer, averaged over swaps 21–50 of 50-swap games.

ModelPoint in layer 4SwapLast pos.New pos.
Qwen3.5-4B (fine-tuned)Input1003834
Qwen3.5-4B (fine-tuned)After attention1003436
Qwen3.5-4B (fine-tuned)After MLP1003636
RecurrentVLM during ahaInput1003738
RecurrentVLM during ahaAfter attention1008363
RecurrentVLM during ahaAfter MLP1008580
RecurrentVLM after ahaInput1003738
RecurrentVLM after ahaAfter attention10010095
RecurrentVLM after ahaAfter MLP10010099

Table 5. Probing inside attention layer 4. Accuracy (%) of linear probes on held-out 50-swap games at three points of the layer: the swap is seen at its input, the last ball position is fetched by its attention, and the new position is then computed from both by its MLP.

Probes read the current swap at the input of attention layer 4, the last position after its attention, and the new position after its MLP. During the aha moment, its attention already fetches 83% of the last positions, but its MLP still computes

@page 7/11 starts_mid=0 ends_mid=1

Figure 7 (PDF p. 7). Tracking the aha moment. (a) Over training: share of swaps on which the frame-wise answer equals the model’s own last answer updated by the current swap (black), and per-swap accuracy on swaps 31–50 (red). (b) Accuracy on 50-swap games with one head of attention layer 4 switched off; any single head of attention layer 3 or 5 keeps at least 60%. (c) Per-swap accuracy with head 2 of attention layer 4 off.

Figure 8 (PDF p. 7). Shallow recurrence suits continuous motion; deep recurrence carries jumps further. Accuracy after 2,500 steps with recurrence in only 4 of the 8 attention layers (shallow, middle, deep: 1–4, 3–6, 5–8).

Observation 2. Recurrence adds state transitions over time, not depth. Without recurrence, each swap costs an attention layer, so the ball is lost after about ten swaps. Recurrence gives every frame group its own state transition, and this ability emerges suddenly at an aha moment during training.

What happens at the aha moment?

To understand this emergence, we focus on the aha moment (figure 7). First, we ablate the components of RecurrentVLM. Second, we track the model before and after the aha moment. Third, we probe inside attention layer 4. Finally, we locate where the learned state is stored.

Ablating components. We first ask which components the aha moment needs (table 4a, b, figure 8). Without frame-wise supervision, RecurrentVLM stays at chance, and without the recurrent KV cache it learns only the short cup games within 2,500 steps. With recurrence in only 4 of the 8 attention layers, the shallow and middle layers learn the cup games, which move continuously, but not the shell games, which jump by a whole swap per frame group, and the deep layers learn neither fully (figure 8). The aha moment thus needs frame-wise supervision and recurrence in all eight attention layers during training.

VariantVETMETCup×16Shell×50
RecurrentVLM10099.691.099.0
(a) no recurrent KV cache90.032.035.037.0
(b) no frame-wise supervision36.033.835.032.0
(c) recurrent KV cache off34.034.840.030.0
(d) keep last 8 frame groups98.099.697.091.0
(e) keep last 4 frame groups38.091.837.055.0

Table 4 (PDF p. 7). Ablating components. Accuracy (%) after 2,500 steps, one change to RecurrentVLM per row: (a, b) in training, (c–e) only at inference; (a) is the baseline of Table 1 at step 2,500. Shaded: unseen lengths.

Tracking the aha moment. We then follow the frame-wise answers through training (figure 7a). The share of swaps on which the frame-wise answer equals the model’s own last answer updated by the current swap rises from 28% at step 1,000 to 92% at step 1,240, before accuracy follows; the baseline stays at 41%.

Six of the 16 heads of attention layer 4 carry this reuse: switching off any one of them drops the accuracy on 50-swap games to 28–57%, and the ball is lost after the first swaps (figure 7b, c). In short, the model first learns to reuse its own last answer and update it with the current swap, through these six heads of attention layer 4.

Probing inside attention layer 4. If the update rule is in place by step 1,240, long-game accuracy should already be high, so we look inside attention layer 4 (Table 5, Figure 9). At the aha moment, attention layers 3–7 shift toward the current and previous frame groups; in attention layer 4, the share on the previous frame group rises from 16% before to 29% after the aha moment.

Probes read the current swap at the input of attention layer 4, the last position after its attention, and the new position after its MLP. During the aha moment, its attention already fetches 83% of the last positions, but its MLP still computes the new position for only 80% of the swaps, against 99% of the swaps after the aha moment.

Meanwhile, the number of swaps the model tracks before its per-swap accuracy falls below 80% grows from about 13 at step 1,200 to 25 at step 1,240, 37 at step 1,400, 66 at step 1,480, and over 100 from step 2,000. Copying the final MLPs of attention layer 4 and of the three GDN layers below it into the model of step 1,240 raises its per-swap accuracy on swaps 31–50 from 50% to 82%, while its final attention gives only 60%. The rule is thus in place before the jump, and the jump makes its update nearly exact.

Observation 3. The aha moment extends tracking. Evidence. In the same 300 steps, the model starts to reuse its last answer and tracks from about 13 to over 100 swaps. Inference. The update rule forms first; accuracy jumps once the MLP of attention layer 4 computes the new position reliably.

Locating the learned state.

Finally, to locate the learned state, we use memory transfer: copying game B’s recurrent KV cache into game A midway switches 98% of A’s later answers to B, those of the last frame group alone 83%, and the state of all GDN layers none (Figure 5c). At test time, recurrence in attention layers 1–4 alone keeps a per-swap accuracy of 98.8% on swaps 31–50, whereas switching it off only in attention layer 4 leaves 37.7%. Without recurrence, every game falls back to chance, and keeping only the last 8 frame groups lowers Shell×\times50 from 99% to 91% (table 4c–e). Together with the probes, which read the ball right after attention layer 4 (Figure 5a), the learned state thus lives in the recurrent KV cache of attention layer 4, mostly of the latest frame group.

Observation 4. Every layer learns, one layer stores. All eight attention layers must recur to learn the rule; once it is learned, the state lives in attention layer 4 alone.

What does recurrence cost?

General video understanding. We fine-tune Qwen3.5-4B with and without recurrence on the Perception Test [66] training split (Table 6). RecurrentVLM is 1.8 points lower overall: better at the state after an action, such as adversarial actions (+5.8), but worse at recalling an earlier moment, such as the order of events (−7.7): recurrence keeps the latest state but overwrites earlier details.

Question typeQwen3.5-4B zero-shotQwen3.5-4B fine-tunedRecurrentVLM fine-tuned
Perception Test (2000)64.878.576.7 (−1.8)
Advantage: the state after an action
Adversarial action (104)42.356.762.5 (+5.8)
Object attributes (98)57.174.578.6 (+4.1)
State recognition (166)62.776.579.5 (+3.0)
Task completion (166)50.073.576.5 (+3.0)
Disadvantage: recalling an earlier moment
Sequencing (299)56.977.669.9 (−7.7)
Object recognition (252)70.285.378.2 (−7.1)
Action counting (138)38.471.766.7 (−5.0)
Feature matching (153)72.581.777.1 (−4.6)

Table 6. General video understanding. Accuracy (%) on the Perception Test [66] validation set by skill after fine-tuning; gray: question counts; parentheses: change vs. fine-tuned Qwen3.5-4B.

Inference cost. We time both models from loading a cup video at 8 fps to the one-token answer (Table 7). RecurrentVLM takes 1.21.2–1.7×1.7\times the time of Qwen3.5-4B for videos of 12–128 s, with the same peak memory. The cost stays small because recurrence runs inside each attention layer: a frame group waits only for the previous group of the same layer, not for a pass through the whole model, as memory tokens do at 6.8×6.8\times the training cost (Table 3).

Video FramesVideo Sec.Latency (s) Qwen3.5-4BLatency (s) RecurrentVLMMem. (GB)
98121.01.2 (1.2×)9.1
258322.43.2 (1.4×)10.0
514644.97.3 (1.5×)11.4
1,02612810.618.4 (1.7×)14.4

Table 7. Inference cost. Seconds per answer for 8-fps cup videos on one RTX A5000 (ratio to Qwen3.5-4B in parentheses) and peak GPU memory, which is the same for both models.

Observation 5. Recurrence trades detail for state. It keeps the latest state at the cost of details of earlier moments.

Conclusion

We present RecurrentVLM, which adds a recurrent KV cache to a pretrained VLM so that its attention layers perform state transitions over time, without writing the state as text or calling external tools. With a single answer token, RecurrentVLM achieves state-of-the-art accuracy on VET-Bench and MET-Bench and generalizes to games longer than in training, where fine-tuning without recurrence fails. These results support our hypothesis that VLMs lack state transitions over time rather than data. We further find that length generalization emerges suddenly at an aha moment, that each swap costs an attention layer without recurrence, and that the ball’s state forms in the first four attention layers. We hope that recurrent VLMs inspire more work on tracking the states of unobserved objects throughout a video.

References

  1. [1]G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR), Workshop Track, 2017.arxiv.org/abs/1610.01644
  2. [2]J.-B. Alayrac, J. Donahue, P. Luc, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2204.14198
  3. [3]C. Anil, Y. Wu, A. Andreassen, et al. Exploring length generalization in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2207.04901
  4. [4]S. Bai, K. Chen, X. Liu, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025.arxiv.org/abs/2502.13923
  5. [5]R. Baillargeon. Object permanence in 3 1/2- and 4 1/2-month-old infants. Developmental Psychology, 23(5):655–664, 1987.
  6. [6]I. Balažević, Y. Shi, P. Papalampidi, et al. Memory consolidation enables long-context video understanding. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2402.05861
  7. [7]A. Bansal, A. Schwarzschild, E. Borgnia, Z. Emam, F. Huang, M. Goldblum, and T. Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overfitting. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  8. [8]B. Barak, B. L. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2207.08799
  9. [9]M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. xLSTM: Extended long short-term memory. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.04517
  10. [10]A. Behrouz, P. Zhong, and V. Miroknin. Titans: Learning to memorize at test time. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2501.00663
  11. [11]A. Bulatov, Y. Kuratov, and M. Burtsev. Recurrent memory transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2207.06881
  12. [12]J. Chen, Z. Lv, S. Wu, et al. VideoLLM-online: Online video large language model for streaming video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2406.11816
  13. [13]Z. Chen, J. Wu, W. Wang, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  14. [14]H. K. Cheng and A. G. Schwing. XMem: Long-term video object segmentation with an Atkinson-Shiffrin memory model. In European Conference on Computer Vision (ECCV), 2022.arxiv.org/abs/2207.07115
  15. [15]C. Clark et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2026. arXiv:2601.10611.arxiv.org/abs/2601.10611
  16. [16]V. Cohen and R. Mooney. MET-Bench: Multimodal entity tracking for evaluating the limitations of vision-language and reasoning models. In International Conference on Machine Learning (ICML), 2026. arXiv:2502.10886.arxiv.org/abs/2502.10886
  17. [17]D. Cores, M. Dorkenwald, M. Mucientes, C. G. M. Snoek, and Y. M. Asano. Lost in time: A new temporal benchmark for VideoLLMs. In British Machine Vision Conference (BMVC), 2025.
  18. [18]Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.arxiv.org/abs/1901.02860
  19. [19]M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser. Universal transformers. In International Conference on Learning Representations (ICLR), 2019.arxiv.org/abs/1807.03819
  20. [20]G. Delétang, A. Ruoss, J. Grau-Moya, et al. Neural networks and the Chomsky hierarchy. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2207.02098
  21. [21]Y. Deng, K. Prasad, R. Fernandez, P. Smolensky, V. Chaudhary, and S. Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460, 2023.arxiv.org/abs/2311.01460
  22. [22]S. Ding, J. Shang, S. Wang, Y. Sun, H. Tian, H. Wu, and H. Wang. ERNIE-Doc: A retrospective long-document modeling transformer. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.arxiv.org/abs/2012.15688
  23. [23]A. Fan, T. Lavril, E. Grave, A. Joulin, and S. Sukhbaatar. Addressing some limitations of transformers with feedback memory. arXiv preprint arXiv:2002.09402, 2020.arxiv.org/abs/2002.09402
  24. [24]Y. Fan, Y. Du, K. Ramchandran, and K. Lee. Looped transformers for length generalization. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2409.15647
  25. [25]C. Fu, Y. Dai, Y. Luo, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2405.21075
  26. [26]A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.arxiv.org/abs/2106.02997
  27. [27]J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kaikhura, A. Bhatale, and T. Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2502.05171
  28. [28]Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  29. [29]A. Giannou, S. Rajput, J.-y. Sohn, K. Lee, J. D. Lee, and D. Paliopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning (ICML), 2023.arxiv.org/abs/2301.13196
  30. [30]R. Girdhar and D. Ramanan. CATER: A diagnostic dataset for compositional actions and temporal reasoning. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1910.04744
  31. [31]S. Goyal, Z. Ji, A. S. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan. Think before you speak: Training language models with pause tokens. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.02226
  32. [32]A. Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983, 2016.arxiv.org/abs/1603.08983
  33. [33]R. Grazzi, J. Siems, A. Zela, J. K. H. Franke, F. Hutter, and M. Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2411.12537
  34. [34]A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Conference on Language Modeling (COLM), 2024.arxiv.org/abs/2312.00752
  35. [35]M. Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics (TACL), 2020.DOI
  36. [36]S. Hao, S. Sukhbaatar, D. Su, X. Li, Z. Hu, J. Weston, and Y. Tian. Training large language models to reason in a continuous latent space. In Conference on Language Modeling (COLM), 2025.arxiv.org/abs/2412.06769
  37. [37]D. T. Hoffmann, S. Schrodi, J. Bratulić, N. Behrmann, V. Fischer, and T. Brox. Eureka-moments in transformers: Multi-step tasks reveal softmax induced optimization problems. In International Conference on Machine Learning (ICML), pages 18409–18438, 2024.arxiv.org/abs/2310.12956
  38. [38]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.09685
  39. [39]D. Hutchins, I. Schlag, Y. Wu, E. Dyer, and B. Neyshabur. Block-recurrent transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2203.07852
  40. [40]A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning (ICML), 2020.arxiv.org/abs/2006.16236
  41. [41]N. Kim and S. Schuster. Entity tracking in language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023.arxiv.org/abs/2305.02363
  42. [42]B. Li, Y. Zhang, D. Guo, et al. LLaVA-OneVision: Easy visual task transfer. Transactions on Machine Learning Research (TMLR), 2025.arxiv.org/abs/2408.03326
  43. [43]B. Z. Li, M. Nye, and J. Andreas. Implicit representations of meaning in neural language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.arxiv.org/abs/2106.00737
  44. [44]B. Z. Li, Z. C. Guo, and J. Andreas. (How) do language models track state? In International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2503.02854
  45. [45]K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.13382
  46. [46]K. Li, Y. Wang, Y. He, et al. MVBench: A comprehensive multimodal video understanding benchmark. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2311.17005
  47. [47]K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao. VideoChat: Chat-centric video understanding. Science China Information Sciences, 68(10), 2025.arxiv.org/abs/2305.06355
  48. [48]R. Liao, M. Erler, H. Wang, G. Zhai, G. Zhang, Y. Ma, and V. Tresp. VideoInSTA: Zero-shot long video understanding via informative spatial-temporal reasoning with LLMs. In Findings of the Association for Computational Linguistics: EMNLP, 2024.arxiv.org/abs/2409.20365
  49. [49]H. Lightman, V. Kosaraju, Y. Burda, et al. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2305.20050
  50. [50]B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.arxiv.org/abs/2311.10122
  51. [51]B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang. Transformer learn shortcuts to automata. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.10749
  52. [52]H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  53. [53]T. Liu and W. S. Lee. Can vision-language models solve the shell game? In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4519–45221, 2026.arxiv.org/abs/2603.08436
  54. [54]Y. Liu, S. Li, Y. Liu, Y. Wang, S. Ren, L. Li, S. Chen, X. Sun, and L. Hou. TempCompass: Do video LLMs really understand videos? In Findings of the Association for Computational Linguistics: ACL, 2024.arxiv.org/abs/2403.00476
  55. [55]M. Maaz, H. Rasheed, S. Khan, and F. S. Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024.arxiv.org/abs/2306.05424
  56. [56]S. McLeish, A. Bansal, A. Stein, et al. Transformers can do arithmetic with the right embeddings. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2405.17399
  57. [57]J. Meng, Z. Bao, Y. Tan, H. Wen, et al. Open-o3-Video-v2: Seek, ground, and reason with verifiable spatio-temporal evidence. arXiv preprint arXiv:2610.17656, 2026.arxiv.org/abs/2610.17656
  58. [58]K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2202.05262
  59. [59]W. Merrill and A. Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics (TACL), 2023.arxiv.org/abs/2207.00729
  60. [60]W. Merrill and A. Sabharwal. The expressive power of transformers with chain of thought. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.07923
  61. [61]W. Merrill, J. Petty, and A. Sabharwal. The illusion of state in state-space models. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2404.08819
  62. [62]N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2301.05217
  63. [63]M. Nye, A. J. Andreassen, G. Gur-Ari, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.arxiv.org/abs/2112.00114
  64. [64]C. Olsson, N. Elhage, N. Nanda, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.arxiv.org/abs/2209.11895
  65. [65]C.-A. Oncescu, D. Morani, S. Jelassi, A. Meterez, M. Kwun, and S. Kakade. The recurrent transformer: Greater effective depth and efficient decoding. arXiv preprint arXiv:2604.21215, 2026.arxiv.org/abs/2604.21215
  66. [66]V. Pătrăucean, L. Smaira, A. Gupta, et al. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.arxiv.org/abs/2305.13786
  67. [67]B. Peng, E. Alcaide, Q. Anthony, et al. RWKV: Reinventing RNNs for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP, 2023.arxiv.org/abs/2305.13048
  68. [68]A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.arxiv.org/abs/2201.02177
  69. [69]N. Prakash, T. Rott Shaham, T. Haklay, Y. Belinkov, and D. Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In International Conference on Learning Representations (ICLR), 2024.
  70. [70]Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.
  71. [71]J. W. Rae, A. Potapenko, S. M. Jayakumar, et al. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1911.05507
  72. [72]N. Ravi, V. Gabeur, Y.-T. Hu, et al. SAM 2: Segment anything in images and videos. In International Conference on Learning Representations (ICLR), 2025.
  73. [73]Y. Sarrof, Y. Veitsman, and M. Hahn. The expressive capacity of state space models: A formal language perspective. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
  74. [74]N. Saunshi, N. Dikkala, Z. Li, S. Kumar, and S. J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2502.17416
  75. [75]A. Shamsian, O. Kleinfeld, A. Globerson, and G. Chechik. Learning object permanence from video. In European Conference on Computer Vision (ECCV), 2020.arxiv.org/abs/2003.10469
  76. [76]Z. Shangguan, C. Li, Y. Ding, Y. Zheng, Y. Zhao, T. Fitzgerald, and A. Cohan. TOMATO: Assessing visual temporal reasoning capabilities in multimodal foundation models. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.23266
  77. [77]J. Shin, Y. Hong, J. Park, and X. Huang. Can video world models track unobserved world states? arXiv preprint arXiv:2608.30692, 2026.arxiv.org/abs/2608.30692
  78. [78]E. Song, W. Chai, G. Wang, et al. MovieChat: From dense token to sparse memory for long video understanding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.arxiv.org/abs/2307.16449
  79. [79]P. Tokmakov, J. Li, W. Burgard, and A. Gaidon. Learning to track with object permanence. In International Conference on Computer Vision (ICCV), 2021.arxiv.org/abs/2103.14258
  80. [80]S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel. Chess as a testbed for language model state tracking. In AAAI Conference on Artificial Intelligence (AAAI), 2022.arxiv.org/abs/2102.13249
  81. [81]V Team, W. Hong, et al. GLM-4.5V and GLM-4.1V-Thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025.
  82. [82]B. Van Hoorick, P. Tokmakov, S. Stent, J. Li, and C. Vondrick. Tracking through containers and occluders in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2305.03052
  83. [83]J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2201.11903
  84. [84]Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2203.08913
  85. [85]J. Xiao, X. Shang, A. Yao, and T.-S. Chua. NExT-QA: Next phase of question-answering to explaining temporal actions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.arxiv.org/abs/2105.08276
  86. [86]D. Yang, X. Li, M. Liu, Y. Wang, and Y. Xiong. Ask about the right moment: Training-free object identity reasoning for video-language models. arXiv preprint arXiv:2610.31471, 2026.arxiv.org/abs/2610.31471
  87. [87]S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing linear transformers with the delta rule over sequence length. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2406.06484
  88. [88]S. Yang, J. Kautz, and A. Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2412.06464
  89. [89]K. Yi, C. Gan, Y. Li, P. Kohli, J. Wu, A. Torralba, and J. B. Tenenbaum. CLEVRER: Collision events for video representation and reasoning. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1910.01442
  90. [90]T. Zhang, S. Bi, Y. Hong, K. Zhang, F. Luan, S. Yang, K. Sunkavalli, W. T. Freeman, and H. Tan. Test-time training done right. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2505.23884
  91. [91]H. Zhou, A. Bradley, E. Littwin, et al. What algorithms can transformers learn? A study in length generalization. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16028

Paper details

vlnc:W_8pygqkw5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents