betaFeedback

WorksComputer Science

Out of Sight, Into Memory: Compact Visual Latents Retain Local Evidence in Ultra-High-Resolution Remote Sensing

Abstract

Question answering over ultra-high-resolution (UHR) remote sensing scenes poses a tension for multimodal large language models: global overviews obscure small objects and local attributes, whereas detailed crops consume substantial context. Compressing inspected crops into short sequences of continuous visual latents offers a compact memory once crop tokens are discarded, yet diagnostic studies on natural images find that models often ignore latent content, leaving unclear whether compact latents can retain fine-grained targets lost in downsampled UHR remote sensing scenes. To isolate evidence retention from region selection, we encode high-resolution crops into latent memory tokens for frozen backbones and evaluate both their predictive utility and content dependence. Across two generations of Qwen-VL on LRS-GRO, a 64-token memory achieves 7.6- to 9.9-fold compression of retained crop tokens and improves accuracy by 2.5 to 4.8 percentage points over the downsampled global image alone, recovering most of the high-resolution crop gain with consistent behavior on XLRS-Bench. Substituting memory from another image or another region of the same scene reduces accuracy by 5.8 to 7.4 percentage points and yields lower accuracy than removing memory altogether, demonstrating that predictions depend on the encoded regional evidence. Restoring the source crop within the same model reduces the accuracy loss caused by zeroing or removing memory. These findings establish that compact visual latents retain useful local evidence in UHR remote sensing and identify evidence availability as a key factor governing their contribution.

Introduction

Ultra-High-Resolution (UHR) Earth observation scenes routinely contain hundreds of millions of pixels, with small objects and fine-grained spatial relationships distributed across vast geographic areas. When multimodal large language models (MLLMs) process a downsampled global overview of such a scene, small targets and local attributes often become indistinguishable. Recent UHR remote sensing methods address this limitation through active perception and selective zooming, inspecting high-resolution crops of task-relevant regions. However, retaining the visual tokens of each inspected crop consumes substantial context budget, creating a bottleneck when reasoning over fine-grained details under limited token capacity [13, 19]. Compressing an observed crop into a small set of continuous visual latents [6, 11, 20, 21] and discarding its original tokens offers a natural mechanism for compact visual memory in UHR remote sensing.

Whether MLLMs genuinely read visual evidence stored in such compact latents, however, remains uncertain. Diagnostic studies of latent visual reasoning in general vision domains have shown that replacing latent tokens with uninformative vectors or altering their visual content often has little effect on prediction accuracy [7, 12, 17]. Such observations indicate that accuracy gains associated with a latent pathway do not necessarily arise from the visual information it carries. At the same time, they point to a key structural distinction between general domain images and UHR remote sensing: evidence availability, defined as whether the task-relevant information carried by a latent can also be recovered from other inputs in the context. When a global image preserves sufficient visual cues, the model retains an alternative route to the answer, consistent with findings that masking input evidence increases reliance on latents [17]. In UHR remote sensing, downsampling the global scene removes fine-grained local details, leaving compact memory as the sole retained source of zoomed-in visual evidence once crop tokens are discarded.

We investigate whether this natural evidence scarcity enables compact visual latents to serve as an effective, content-dependent memory for UHR question answering. To separate the retention and reading of local evidence from the choice of where to look, we fix crop locations using region annotations (Figure 1). Keeping both the vision encoder and the language model frozen, we first calibrate a shared 8-token format prefix on global overviews to absorb answer formatting and label priors across all evaluated conditions, and then train only a lightweight resampler [1, 10] of approximately 10M parameters to map each high-resolution crop into KK latent memory tokens. At inference, original crop tokens are removed, and the frozen model answers from the question, the downsampled global image, the shared format prefix, and the KK memory tokens.

Figure 1. Writing, keeping and probing the memory. A crop of the annotated region is encoded and compressed by a resampler into KK memory tokens, after which the crop tokens are discarded. In the scarce context the language model sees the question, the thumbnail and the memory, and cannot look again at the crop. In the redundant context the crop pixels are added, so the model can look again. ZERO, DROP, SWAP-IMAGE and SWAP-REGION act on the memory only.

Across two generations of Qwen-VL [2, 3] on the local questions of LRS-GRO [13], compact visual memory delivers substantial context savings alongside consistent accuracy gains. With K=64K = 64 memory tokens, accuracy improves by 2.5 percentage points on Qwen2.5-VL and 4.8 percentage points on Qwen3-VL relative to using only the downsampled global image. This representation replaces full-resolution crop encodings averaging 634 and 486 tokens, achieving 9.9-fold and 7.6-fold compression of retained crop context while recovering approximately two thirds of the full-crop gain on Qwen2.5-VL and matching full-crop accuracy on Qwen3-VL. It also outperforms matched-budget alternatives on LRS-GRO and transfers directly on Qwen3-VL to XLRS-Bench [18].

Targeted interventions on the memory further establish that predictions depend on the specific regional evidence it encodes, and that this dependence varies with evidence availability. Holding token count, token positions, and model weights fixed, substituting memory from another image reduces accuracy by 7.4 percentage points on Qwen2.5-VL and 6.2 percentage points on Qwen3-VL, while substituting memory from another annotated region of the same scene reduces accuracy by 6.0 and 5.8 percentage points, respectively. In both models, mismatched memory yields lower accuracy than removing the memory altogether, confirming that the model actively reads the retained local content. Restoring the source crop to the context within the same trained model reduces the accuracy loss caused by zeroing or removing memory by 2.7 to 3.3 percentage points on LRS-GRO, with even larger reductions on XLRS-Bench. Thus, the contribution of the same visual latent shifts according to whether its local evidence is accessible elsewhere in the context.

Our contributions are summarized as follows:

  • We demonstrate that compact visual latents efficiently retain fine-grained local evidence in UHR remote sensing. Across Qwen2.5-VL and Qwen3-VL on LRS-GRO, a 64-token memory achieves 7.6-fold to 9.9-fold compression of retained crop tokens, improves accuracy by 2.5 to 4.8 percentage points over the global image alone, and recovers most or all of the high-resolution crop gain, with consistent cross-dataset transfer on Qwen3-VL to XLRS-Bench.

  • We establish causal dependence on encoded regional content through controlled substitutions and matched-budget baselines. Replacing memory with latents from another image or another region within the same UHR scene reduces accuracy by 5.8 to 7.4 percentage points and harms prediction more than removing it altogether.

  • We identify evidence availability as a key factor governing visual latent reliance in MLLMs. Restoring the source crop within the same model substantially attenuates the accuracy loss of zeroing or removing memory, explaining why the loss of fine detail in downsampled UHR overviews activates latent memory use.

Related Work

Ultra-High-Resolution Remote Sensing. UHR remote sensing requires models to interpret small objects and local attributes distributed across large scenes. Benchmarks such as XLRS-Bench [18] and LRS-VQA [14] highlight the difficulty of preserving subtle local details under constrained visual processing budgets. Existing methods improve access to relevant regions through selective processing. Luo et al. [14] combine a dynamic image pyramid with tile selection guided by text and token pruning, while ZoomEarth [13] and Wang et al. [19] employ adaptive cropping, zooming, and visual tools to acquire local visual evidence. Whereas these approaches focus on selecting where to inspect a scene, retaining the visual tokens of inspected crops still consumes substantial context. Our work complements region selection by demonstrating that acquired local evidence remains useful when compressed into compact visual latents after the original crop tokens are discarded.

Visual Token Compression and Memory. Visual token compression produces compact representations of visual inputs to reduce computational and context costs. Perceiver [10] aggregates dense inputs into a latent array, while Flamingo [1] uses a Perceiver Resampler to produce a fixed number of visual tokens. Token Merging [4] combines similar tokens in vision transformers, and FastV [5] prunes redundant visual tokens within multimodal language models. Related work on visual memory, including MA-LMM [8] and MovieChat [16], retains information from earlier frames to support the understanding of long videos. In our setting, a resampler with learned queries encodes crops at high resolution into compact latent memories. Beyond evaluating accuracy under reduced token budgets, we examine whether model predictions genuinely depend on the regional visual information preserved in these tokens.

Latent Visual Reasoning and Diagnostics. Latent visual reasoning introduces continuous tokens as intermediate visual representations within multimodal language models [6, 11, 21]. Methods such as Monet [20] train models to generate these representations through visual supervision and distillation to support reasoning in latent space. Alongside advances in latent construction, diagnostic studies have examined whether model predictions depend on the visual content of these representations. Li et al. [12] identify weak connections between input images, latent representations, and final predictions in evaluated models. Guo et al. [7] distinguish the content of latent slots from boundary markers and formatting, showing that substantial performance gains can persist without the slot content. Viveiros et al. [17] connect latent bypass to visual evidence already accessible in the input and demonstrate increased reliance when relevant image regions are masked during training. CVRR [15] makes the recurrent latent state the required visual pathway for prediction and shows that restoring an alternative visual route partially compensates for corrupted latent content. These findings indicate that the contribution of visual latents depends on both their encoded content and the surrounding visual context. Extending this perspective to UHR remote sensing, we show that compact latent memories preserve local details obscured in the downsampled global overview and that model predictions rely on this regional content. Furthermore, restoring the source crop within the same model reduces reliance on memory, supporting evidence availability as an explanation for the contribution of visual latents.

Visual Memory Testbed and Protocol

To evaluate whether compact visual latents preserve useful local evidence in UHR remote sensing, we construct a controlled question answering testbed. This section details the memory construction pipeline, the two context conditions and causal interventions used to probe content reliance, and the evaluation protocol.

Compact Crop Memory Construction

Given a question qq about a UHR scene, the model receives a downsampled global image resized to a maximum side length of 1,024 pixels. Although this overview preserves global scene layout, small targets and local attributes often become indistinguishable at this scale. To supply fine local detail while separating memory retention from region localization, we extract a crop from the annotated region associated with each question. Following the LRS-GRO implementation [13], we enlarge the annotated bounding box by 50%, enforce a minimum side length of 512 pixels at native resolution, and resize the crop so that its longer side does not exceed 1,024 pixels.

The frozen vision encoder maps the crop to a sequence of visual features, which are compressed into a memory mm of KK continuous tokens in the embedding space of the language model (Figure 1). Our resampler [1, 10] consists of 2 layers with internal dimension 512, 8 attention heads, feed-forward width 2,048, and 64 learned queries; for any budget K∈{4,8,16,32,64}K \in\{4, 8, 16, 32, 64\} (K=64K = 64 by default), it selects the

first KK queries. Each layer applies cross attention from the queries to the crop features followed by a feedforward network, projecting back to the hidden dimension of the language model (3,584 on Qwen2.5-VL, totaling 10.0M trainable parameters). Its output projection is initialized to zero and added as a residual to spatially pooled crop features, so the module reduces exactly to the Average Pooling baseline before training.

Throughout training, both the vision encoder and the language model remain strictly frozen without LoRA [9] or backbone tuning. Using 2,117 local training questions from 228 scenes and 223 validation questions from 54 scenes in LRS-GRO, we run two stages with AdamW, batch size 1 with 8 gradient accumulation steps, gradient clipping at 1.0, and a constant learning rate without warmup. To prevent trained memory from winning merely by adapting output formatting or answer priors on a frozen backbone, Stage 1 initializes an 8-token format prefix (28.7K parameters on Qwen2.5-VL and 32.8K on Qwen3-VL) from the first 8 token embeddings of “Answer with a short lowercase phrase and no punctuation.” and trains it for 3 epochs at a learning rate of 3×10−43 \times10^{-4} with zero weight decay using answer cross entropy on global overviews and questions alone, selecting the checkpoint with lowest validation cross entropy. This 8-token format prefix is then frozen and inserted after the visual or memory tokens and before the question in every evaluated condition and intervention. Stage 2 updates only the resampler (10.0M parameters on Qwen2.5-VL and 10.5M on Qwen3-VL) for 4 epochs at a learning rate of 1×10−41 \times10^{-4} with weight decay 0.01, minimizing answer cross entropy combined with a counterfactual margin loss and logit distillation over the top 64 tokens from uncompressed crop inputs.

Diagnostic Contexts and Interventions

Let cc denote the answering context accompanying the memory mm. We distinguish overall predictive utility from causal reliance on memory content. Predictive utility is measured by comparing accuracy against a baseline that receives only the question and the downsampled global overview. Reliance is measured within the same trained model by intervening on mm while holding cc and all model weights fixed. For an intervention that replaces mm with m~\tilde{m}, we define paired reliance in percentage points on identical questions as:

Rel⁡=Acc⁡c,m−Acc⁡c,m~.(1)\operatorname{Rel} = \operatorname{Acc}_{c,m} - \operatorname{Acc}_{c,\tilde{m}}. \tag*{(1)}

A positive value of Rel indicates that corrupting or removing the memory reduces prediction accuracy.

Evidence availability refers to whether the local information carried by mm can also be recovered from the surrounding context cc. We evaluate how evidence availability shapes memory reliance by comparing two answering contexts at inference time (Figure 1):

  • Scarce context (Remembering). Original crop tokens are discarded, so cc contains only the question and the downsampled global overview. Because the model cannot inspect the crop pixels again, mm serves as the sole retained carrier of zoomed local evidence.

  • Redundant context (Looking again). Original crop tokens from which mm was written are restored to cc alongside the same memory mm. Because the zoomed evidence is simultaneously accessible as pixels, comparing reliance across the two contexts while keeping model weights, questions, and memories fixed isolates the effect of evidence availability within a single model, providing an inference counterpart to the masked training analysis of Viveiros et al. [17].

We apply four targeted interventions to mm. ZERO replaces every memory token with a zero vector, and DROP removes the memory tokens from the input sequence; together they test whether the retained pathway contributes necessary information. SWAP-IMAGE replaces mm with memory encoded from a different scene, whereas SWAP-REGION replaces mm with memory encoded from another annotated region within the same scene. Because SWAP-IMAGE and SWAP-REGION preserve the exact number, sequence positions, and feature distribution of the memory tokens as well as the trained pathway that consumes them, they isolate dependence on regional visual content from potential artifacts caused by sequence truncation.

Benchmarks, Controls, and Scoring Protocol

We conduct our primary evaluation on LRS-GRO [13], covering 7,393 local test questions across 839 UHR scenes using Qwen2.5-VL-7B-Instruct [3] and Qwen3-VL-8B-Instruct [2], which tokenize visual inputs at 28 and 32 pixels per token, respectively. To test generalization across datasets, we evaluate the Qwen3-VL model trained on LRS-GRO directly without further training on 1,657 questions with reference boxes from 856 scenes in XLRS-Bench [18], denoted XLRS-Bench-lite, applying the identical crop extraction rule to each reference box.

All evaluated conditions share the frozen backbone, question, downsampled global overview, and frozen 8-token Stage 1 format prefix. Thumbnail, High-Res Crop, Low-Res Pixels, Average Pooling, and Text Notes introduce zero additional trainable parameters beyond this shared prefix: Thumbnail provides no crop, High-Res Crop supplies uncompressed crop tokens, and the three compact baselines at budget KK compress the crop via image downsampling, spatial feature pooling (the exact zero-parameter initialization of Memory), or truncated descriptions generated by the frozen backbone without the prefix (at most KK tokens). Soft Prompt places KK static learned vectors independent of the crop at the memory token positions using answer cross entropy; because it has far fewer parameters than the resampler, it tests adding KK learned continuous slots without crop input, whereas full resampler parameter capacity is controlled in § 4.2 via ZERO and SWAP-IMAGE using the identical Stage 2 checkpoint. Across tables, reported token counts measure only the regional evidence payload per question, excluding the global overview, the shared 8-token format prefix, and region label text.

All runs use greedy decoding under a fixed generation configuration and share a deterministic scoring protocol. When a question specifies candidate answers, we score the likelihood assigned to each candidate rather than parsing generated text; otherwise, we apply official benchmark matching rules. Likelihood scoring prevents output formatting failures from confounding memory interventions: in a pilot on XLRS-Bench using generative extraction, ZERO lowered accuracy by 19.8 percentage points largely through invalid output formats. All comparisons are paired on identical test questions, which remain strictly excluded from model development and checkpoint selection. We report 95% confidence intervals from a cluster bootstrap over scenes to account for correlation among questions sharing the same scene, considering an intervention effect statistically supported when the lower bound of its 95% confidence interval exceeds zero.

Empirical Findings

We organize our empirical analysis around three progressive questions: whether compact visual memory preserves fine local details under tight token budgets, whether predictions genuinely depend on the regional visual content stored in memory, and why UHR remote sensing activates latent reading where general vision settings often exhibit bypass. Unless stated otherwise, results use the default budget of K=64K = 64 tokens.

Compact Memory Preserves Local UHR Evidence

Table 1 evaluates predictive utility and retained context cost when original crop tokens are discarded. On LRS-GRO, adding K=64K = 64 latent memory tokens raises accuracy over Thumbnail from 54.6% to 57.1% (+2.5 percentage points) on Qwen2.5-VL and from 57.2% to 62.0% (+4.8 percentage points) on Qwen3-VL. These 64 tokens replace full crop representations averaging 634 tokens on Qwen2.5-VL and 486 tokens on Qwen3-VL, compressing retained crop context by factors of 9.9 and 7.6, respectively. Despite this compression, Memory recovers roughly two thirds of the full crop improvement on Qwen2.5-VL (57.1% vs. 58.4%) and matches High-Res Crop on Qwen3-VL (62.0% vs. 61.6%, a paired difference of 0.4 [−0.4, 1.2] points). When transferred directly to XLRS-Bench-lite without additional training, Memory on Qwen3-VL improves accuracy over Thumbnail by 12.2 percentage points

ConditionLRS-GRO Qwen2.5-VL TokensLRS-GRO Qwen2.5-VL Acc.LRS-GRO Qwen3-VL TokensLRS-GRO Qwen3-VL Acc.XLRS-Bench-lite Qwen3-VL TokensXLRS-Bench-lite Qwen3-VL Acc.
Thumbnail054.6057.2036.3
Soft Prompt6453.86457.76437.5
Low-Res Pixels6455.86459.86452.9
Text Notes4754.946.558.049.439.4
Average Pooling6454.96458.76443.6
Memory6457.16462.06448.5
High-Res Crop63458.448661.6321.853.7

Table 1. Predictive utility and retained context cost. Accuracy (%) on LRS-GRO (7,393 questions from 839 scenes) and XLRS-Bench-lite (1,657 questions from 856 scenes). All conditions include the question, the downsampled global overview, and the shared frozen 8-token Stage 1 format prefix. Compact representations use a budget of K=64K = 64 tokens, whereas High-Res Crop retains the uncompressed crop tokens as an upper reference. Tokens counts only the regional evidence payload per question beyond this shared context (excluding global overview tokens, the 8-token format prefix, and region label text). Bold values mark the highest accuracy within the budget of K=64K = 64 tokens.

Figure 2. Accuracy across token budgets on LRS-GRO. Accuracy of Memory, Low-Res Pixels, Text Notes, and Average Pooling on Qwen3-VL in the scarce context for K∈{4,8,16,32,64}K \in\{4, 8, 16, 32, 64\}. Horizontal reference lines indicate Thumbnail and High-Res Crop. Error bars show 95% confidence intervals from a cluster bootstrap over scenes.

(from 36.3% to 48.5%), preserving approximately 70% of the High-Res Crop gain while compressing an average of 321.8 crop tokens by a factor of 5.0.

Comparisons against controls at matched budgets confirm that these improvements cannot be explained by adding static learned token slots without crop input. Soft Prompt yields 53.8% on Qwen2.5-VL and 57.7% on Qwen3-VL on LRS-GRO, and 37.5% on XLRS-Bench-lite, remaining near Thumbnail across all settings (while § 4.2 rules out the 10.0M to 10.5M resampler parameters by evaluating the identical checkpoint under ZERO and SWAP-IMAGE).

Finding 1. Compact visual latents retain local UHR evidence at a fraction of the original crop token cost, recovering most or all of the accuracy gain of uncompressed crops and outperforming alternative representations across budgets on the training domain.

Predictions Depend on Regional Visual Content

To verify that accuracy gains arise from reading visual information stored in memory rather than from structural formatting cues, Table 2 measures paired reliance under targeted interventions in the scarce context. Eliminating memory information consistently degrades performance: on LRS-GRO, ZERO lowers accuracy by 4.5 [3.6, 5.3] percentage points on Qwen2.5-VL and 5.2 [4.4, 6.0] points on Qwen3-VL, while DROP reduces accuracy by 3.5 [2.6, 4.4] and 4.6 [3.9, 5.4] points, respectively. On XLRS-Bench-lite, both ZERO and DROP reduce Qwen3-VL accuracy by 11.8 percentage points, returning performance to the level of Thumbnail alone (36.6% to 36.7% vs. 36.3%).

ConditionLRS-GRO Qwen2.5-VL Acc.LRS-GRO Qwen2.5-VL RelLRS-GRO Qwen3-VL Acc.LRS-GRO Qwen3-VL RelXLRS-Bench-lite Qwen3-VL Acc.XLRS-Bench-lite Qwen3-VL Rel
Memory57.10.062.00.048.50.0
ZERO52.64.5 [3.6, 5.3]56.75.2 [4.4, 6.0]36.711.8 [9.2, 14.3]
DROP53.63.5 [2.6, 4.4]57.34.6 [3.9, 5.4]36.611.8 [9.3, 14.4]
SWAP-IMAGE49.77.4 [6.5, 8.3]55.86.2 [5.4, 7.0]32.316.1 [13.5, 18.7]
SWAP-REGION{sup}†`50.76.0 [4.9, 7.1]56.15.8 [4.9, 6.7]32.016.3 [13.5, 19.3]

Table 2. Dependence on memory content. Accuracy (%) and paired reliance Rel (percentage points) at K=64K = 64 in the scarce context. Rel equals the accuracy of the matched Memory reference minus the intervention accuracy, with 95% confidence intervals from a cluster bootstrap over scenes in brackets. Results on the complete evaluation set use the samples in Table 1. †SWAP-REGION uses a matched Memory reference on the subset of scenes containing multiple annotated regions: 5,295 questions from 514 scenes on LRS-GRO and 1,212 questions from 446 scenes on XLRS-Bench-lite, where reference Memory accuracy on the XLRS-Bench-lite subset is 48.3%. Bold Rel values highlight interventions that substitute memory content. Paired differences are computed prior to rounding.

Content substitutions provide a stricter test by preserving token count, sequence positions, and latent distributions within the identical trained model while altering only the encoded region. Replacing memory with latents from another scene (SWAP-IMAGE) reduces accuracy by 7.4 [6.5, 8.3] points on Qwen2.5-VL and 6.2 [5.4, 7.0] points on Qwen3-VL on LRS-GRO, and by 16.1 [13.5, 18.7] points on XLRS-Bench-lite. Even when global scene context is shared, replacing memory with latents from another annotated region of the same UHR image (SWAP-REGION) lowers accuracy by 6.0 [4.9, 7.1] and 5.8 [4.9, 6.7] points on LRS-GRO, and by 16.3 [13.5, 19.3] points on XLRS-Bench-lite. Across every model and dataset, both SWAP-IMAGE and SWAP-REGION produce lower accuracy than removing memory altogether (DROP), pushing predictions below the Thumbnail baseline. Incorrect regional latents actively mislead the model, demonstrating that predictions rely on the specific local visual evidence encoded in memory.

Finding 2. Model predictions causally depend on the regional visual content of latent memory: substituting latents from another image or another region within the same UHR scene degrades accuracy even more than removing memory altogether.

Evidence Scarcity Activates Latent Reading

The strong content sensitivity observed above contrasts sharply with diagnostic findings in general vision domains.

As summarized in Table 3, substituting latents from another image or injecting uninformative tokens on standard visual benchmarks typically alters accuracy by at most two to four percentage points [17], and donor latents from the same image rarely steer model predictions [7]. In our scarce UHR setting, analogous interventions consistently induce drops of 4.5 to 7.4 percentage points on LRS-GRO and up to 16.3 points on XLRS-Bench-lite. Unlike natural images where global views often retain sufficient cues to answer local queries, downsampling a UHR Earth observation scene removes small targets and subtle attributes, leaving compact memory as the only carrier of local visual evidence.

InterventionGeneral domainOurs (LRS-GRO)
Content of another imageChange between −4 and +2 points across six model versions on V* and BLINK [17]−7.4 and −6.2 points
Uninformative contentAccuracy changes by at most 2 points under an intervention with uninformative tokens for each model and dataset [17]−4.5 and −5.2 points under ZERO
Content of the same imageAnswer follows the donor in 5.3% to 12.7% of cases, below the 25% chance level [7]−6.0 and −5.8 points

Table 3. Contextualizing our findings alongside prior latent diagnostics. Results for the general domain are taken from the cited studies. Our values report accuracy changes in percentage points relative to Memory on LRS-GRO for Qwen2.5-VL and Qwen3-VL, respectively. Because models, benchmarks, and latent sources differ, only the direction and qualitative magnitude of the effect are compared.

To test whether evidence availability directly governs this behavior within the same model, we restore the source crop tokens alongside the memory at inference time (the redundant context) while keeping all model weights, questions, global overviews, and memories unchanged (Table 4). On LRS-GRO, restoring the crop reduces accuracy loss under ZERO by 3.2 [2.4, 4.1] percentage points on Qwen2.5-VL and 2.7 [2.0, 3.5] points on Qwen3-VL, and reduces loss under DROP by 3.3 [2.4, 4.2] and 2.9 [2.1, 3.8] points, respectively. On XLRS-Bench-lite, restoring the source crop on Qwen3-VL reduces reliance under ZERO and DROP by 14.9 [12.2, 17.6] and 19.0 [16.3, 21.7] percentage points; in this redundant setting, removing the transferred memory (DROP) raises accuracy by 7.2 points. Providing direct pixel access to the local region thus supplies an alternative route to the visual evidence and substantially lowers the cost of zeroing or removing memory.

DatasetModelInterventionScarceRedundantReduction
LRS-GROQwen2.5-VLZERO4.5 [3.6, 5.3]1.2 [0.5, 1.9]3.2 [2.4, 4.1]
DROP3.5 [2.6, 4.4]0.2 [−0.5, 0.9]3.3 [2.4, 4.2]
SWAP-REGION†6.0 [4.9, 7.1]3.7 [2.8, 4.7]2.3 [1.4, 3.3]
Qwen3-VLZERO5.2 [4.4, 6.0]2.5 [1.8, 3.1]2.7 [2.0, 3.5]
DROP4.6 [3.9, 5.4]1.7 [1.1, 2.3]2.9 [2.1, 3.8]
SWAP-REGION†5.8 [4.9, 6.7]2.1 [1.4, 2.8]3.7 [2.8, 4.6]
XLRS-Bench-liteQwen3-VLZERO11.8 [9.2, 14.3]−3.1 [−5.4, −0.9]14.9 [12.2, 17.6]
DROP11.8 [9.3, 14.4]−7.2 [−9.3, −5.1]19.0 [16.3, 21.7]
SWAP-REGION†16.3 [13.5, 19.3]16.1 [13.3, 18.9]0.2 [−2.1, 2.7]

Table 4. Memory reliance decreases when source evidence is restored. Paired reliance Rel in percentage points at K=64K = 64. The scarce context contains the question, global overview, and memory; the redundant context additionally restores the source crop. Reduction equals Relscarce−Relredundant\mathrm{Rel}_{\mathrm{scarce}} - \mathrm{Rel}_{\mathrm{redundant}}, computed from unrounded paired results, with 95% confidence intervals from a cluster bootstrap over scenes in brackets. Negative Rel indicates that the intervention increases accuracy. Bold values highlight reductions under ZERO and DROP as well as negative reliance in the redundant context. †SWAP-REGION uses the subsets specified in Table 2 with matched questions across both contexts.

At the same time, access to source pixels does not make the model impervious to conflicting memory content. In the redundant context on LRS-GRO, SWAP-REGION still lowers accuracy by 3.7 [2.8, 4.7] points on Qwen2.5-VL and 2.1 [1.4, 2.8] points on Qwen3-VL, while on XLRS-Bench-lite reliance under SWAP-REGION remains virtually unchanged at 16.1 [13.3, 18.9] points (a reduction of 0.2 [−2.1, 2.7] points). Restoring source pixels therefore diminishes the necessity of memory when it is absent, while predictions remain sensitive to mismatched regional content.

Finding 3. Evidence availability directly modulates reliance on visual latents within a single model: restoring high-resolution crop pixels substantially reduces the cost of zeroing or removing memory, explaining why natural detail loss in downsampled UHR overviews activates latent reading.

Discussion

Visual Memory and Evidence Reconciliation. Our experiments establish that compact visual latents written from inspected crops serve as genuine local memory in UHR remote sensing. After original crop tokens are removed, retaining memory improves accuracy, and substituting its regional content degrades predictions across both benchmarks. Pairing predictive utility with causal content sensitivity addresses a central concern raised by Guo et al. [7], who show that accuracy gains from latent tokens need not imply that their contents store visual information. Complementing diagnostics of latents generated as intermediate reasoning states, our results confirm a concrete memory function when latents encode observed local crops obscured in the global overview. Restoring the source crop further links this function to surrounding visual context, extending the account of Viveiros et al. [17] to retained crop representations while uncovering an operational asymmetry: access to source pixels reduces the penalty of zeroing or dropping memory, yet does not protect predictions against mismatched regional latents (SWAP-REGION), particularly on XLRS-Bench-lite (Table 4). For sequential remote sensing agents, deciding when to retain past crop memories and verifying their consistency with newly inspected pixels therefore represent distinct design requirements.

Representation Choices Across Domains and Tasks. Comparisons across datasets and question types clarify where learned query resampling is most effective. On XLRS-Bench-lite, 64 memory tokens on Qwen3-VL preserve roughly 70% of the accuracy gain provided by High-Res Crop over Thumbnail at fivefold token compression (Table 1) while exhibiting strong content dependence (Table 2). Under the same budget of 64 tokens, however, Low-Res Pixels achieves higher accuracy without relying on resampling queries learned on LRS-GRO, indicating that the relative advantage of learned compression over spatial downsampling varies under domain shift. Task granularity introduces a related distinction on LRS-GRO: although Qwen3-VL with Memory matches High-Res Crop in overall accuracy, its accuracy on object counting questions drops to 32.1% compared with 41.0% for High-Res Crop. Learned query resampling effectively condenses regional semantics and attributes into a small token budget, whereas tasks requiring discrete instance enumeration or direct transfer across imagery distributions benefit from preserving explicit spatial structure.

Integration with Active Perception and Scope. Fixing crop locations through region annotations or reference boxes separates the retention and reading of local evidence from localization errors. Combining compact latent memory with learned region selection [13, 19] to support reasoning across multiple inspected regions is a natural next step for UHR Earth observation systems. Evaluating architectures beyond two generations of the Qwen-VL family will also clarify how broadly these behaviors hold across multimodal backbones. Finally, our compression ratios measure the visual context retained within the language model, because the vision encoder still processes each crop prior to resampling, benchmarking overall inference latency remains an important direction for deployment.

Conclusion

Reasoning over UHR remote sensing imagery under constrained token budgets requires preserving fine local details that disappear in downsampled global overviews without retaining hundreds of tokens per inspected crop. Across two generations of Qwen-VL on LRS-GRO and transfer evaluation on XLRS-Bench, compressing crops at high resolution into compact latent memories recovers most or all of the accuracy benefit of uncompressed crops at a fraction of their context cost. Controlled content substitutions confirm that model predictions rely on the encoded regional visual evidence rather than added token capacity, while restoring source crop pixels within the same model demonstrates that evidence availability governs reliance on memory. These findings establish compact visual latents as an effective mechanism for retaining observed local evidence in UHR remote sensing. Looking ahead, whereas current evaluations focus on individual local regions, complex Earth observation tasks often require comparing and aggregating evidence across several crops within a vast scene, where retaining uncompressed tokens quickly becomes infeasible. Extending our compact memory framework and diagnostic protocol to settings with multiple crops, together with constructing new UHR benchmarks for reasoning across multiple retained regions, represents a promising direction for future work.

References

  1. [1]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tendga Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, pages 23716–23736. Curran Associates, Inc., 2022. 2, 3arxiv.org/abs/2204.14198
  2. [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Chen, Lianghao Deng, Wei Ding, Chang Gao, Chunjang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhao-hai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Jun-yang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuanchen Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. 2, 4
  3. [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun-yang Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 4arxiv.org/abs/2502.13923
  4. [4]Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023. 2arxiv.org/abs/2210.09461
  5. [5]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Computer Vision – ECCV 2024, pages 19–35, 2024. 2arxiv.org/abs/2403.06764
  6. [6]Shuai Dong, Siyuan Wang, Xingyu Liu, Chenglin Li, Haowen Hou, and Zhongyu Wei. Interleaved latent visual reasoning with selective perceptual modeling. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29316–29335. Association for Computational Linguistics, 2026. 1, 3
  7. [7]Jiawei Guo, Yu Chen, Xiang Wang, Shuai Li, Xinpei Zhao, Huaxing Liu, Shuai Dong, Feifei Zhai, and Yu Zhou. Beyond visual memory: Mechanistic diagnostics of latent visual reasoning. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 6493–6519. Association for Computational Linguistics, 2026. 1, 3, 7, 8arxiv.org/abs/2606.01287
  8. [8]Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. MA-LLM: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13504–13514, 2024. 2
  9. [9]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 4arxiv.org/abs/2106.09685
  10. [10]Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Proceedings of the 38th International Conference on Machine Learning, pages 4651–4664. PMLR, 2021. 2, 3arxiv.org/abs/2103.03206
  11. [11]Bangzhen Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen, and Zicheng Liu. Latent visual reasoning. In International Conference on Learning Representations, 2026. 1, 3arxiv.org/abs/2509.24251
  12. [12]You Li, Chi Chen, Yanghao Li, Fanhu Zeng, Kaiyu Huang, Jinan Xu, and Maosong Sun. Imagination helps visual reasoning, but not yet in latent space. In Proceedings of the 43rd International Conference on Machine Learning, pages 66255–66269. PMLR, 2026. 1, 3arxiv.org/abs/2602.22766
  13. [13]Ruixun Liu, Bowen Fu, Jiayi Song, Kaiyu Li, Wanchen Li, Lanxuan Xue, Hui Qiao, Weizhan Zhang, Deyu Meng, and Xiangyong Cao. ZoomEarth: Active perception for ultra-high-resolution geospatial vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 34877–34888, 2026. 1, 2, 3, 4, 8arxiv.org/abs/2511.12267
  14. [14]Junwei Luo, Yingying Zhang, Xue Yang, Kang Wu, Qi Zhu, Lei Liang, Jingdong Chen, and Yansheng Li. When large vision-language model meets large remote sensing imagery: Coarse-to-fine text-guided token pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9206–9217, 2025. 2arxiv.org/abs/2503.07588
  15. [15]Suhyeong Park, Junho Jung, and Jaewoo Kang. Reason through the latent! making latent visual reasoning necessary. arXiv preprint arXiv:2609.06746, 2026. 3arxiv.org/abs/2609.06746
  16. [16]Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. MovieChat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221–18232, 2024. 2arxiv.org/abs/2307.16449
  17. [17]André G. Viveiros, Nuno Gonçalves, André F. T. Martins, and Matthias Lindemann. What’s holding back latent visual reasoning? In Advances in Neural Information Processing Systems, 2026. 1, 3, 4, 7, 8arxiv.org/abs/2605.18445
  18. [18]Fengxiang Wang, Hongzhen Wang, Zonghao Guo, Di Wang, Yulin Wang, Mingshuo Chen, Qiang Ma, Long Lan, Wenjing Yang, Jing Zhang, Zhiyuan Liu, and Maosong Sun. XLRS-Bench: Could your multimodal LLMs understand extremely large ultra-high-resolution remote sensing imagery? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14325–14336, 2025. 2, 4
  19. [19]Fengxiang Wang, Jiangnan Huang, Mingshuo Chen, Yueying Li, Yang Shi, Junwei Luo, Haoyu Wang, Yansheng Li, Jing Zhang, Haiyan Zhao, and Wenjing Yang. Beyond zooming: Learning multi-tool visual reasoning for ultra-high-resolution remote sensing. arXiv preprint arXiv:2607.25993, 2026. 1, 2, 8arxiv.org/abs/2607.25993
  20. [20]Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, and Yisen Wang. Monet: Reasoning in latent visual space beyond image and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12030–12040, 2026. 1, 3arxiv.org/abs/2511.21395
  21. [21]Zeyuan Yang, Xueyang Yu, Delin Chen, Maohou Shen, and Chuang Gan. Machine mental imagery: Empower multimodal reasoning with latent visual tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 33510–33520, 2026. 1, 3arxiv.org/abs/2506.17218

Paper details

vlnc:W_nmj84mxc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents