betaFeedback

WorksComputer Science

Interpreting DeepStack-Style Vision-Language Models: Do Input or Deeper-Layer Visual Tokens Drive Answers?

Abstract

DeepStack-style vision-language models feed visual tokens to the language model at its input, and again at deeper layers. We call these the main entry and the later entries. What do the later entries add when the model already has the image? We give the entries versions that differ in one object's color and see which one the answer follows. On photographs, the original DeepStack follows the later entries, while Qwen3-VL-8B/32B follow the main entry. Does weakening the main entry give the later features more influence? On synthetic images, we keep this conflict and reduce color saturation at the main entry only. The unchanged later features gain a stronger effect on the relative scores of the two colors, in all four models, including Granite Vision. This holds even under mild desaturation, when accuracy without the conflict stays at 100%. To test whether later features can also correct errors, we degrade both entries and restore only the later ones. The same blur costs both Qwen models 50 percentage points of accuracy on stripe orientation. The 32B model recovers 48 points and the 8B model only 8. Accuracy with one clean entry predicted which entry each model can lose, and five of six tested predictions held on new GQA attribute questions. With the other entry zeroed, Qwen3-VL-8B loses only 0.3 points on its main entry alone, and DeepStack at most 3.45 points on either entry alone. In split inference on these questions, DeepStack can then send only its main-entry features at higher precision. This takes 49% fewer bytes than all entries at a shared precision, at similar accuracy.

Introduction

Recent vision-language models such as Qwen3-VL [37] and Granite Vision [16] build on the DeepStack design [30]: visual tokens enter the language model at its input and again at deeper layers. We call these inputs the main entry and the later entries (main and later for short). In the original DeepStack model (DeepStack-L-Vicuna-7B, DeepStack below), the later entries carry a higher-resolution view. In Qwen3-VL, they carry intermediate vision-encoder features. Benchmark gains show that later entries can help, but not what they do in a trained model. What do later entries contribute once the model has al- *These authors contributed equally. †Corresponding authors. ready received the image? They could decide the answer when the entries disagree, gain influence when main’s image is degraded, repair the resulting errors, or duplicate main, so that a model can lose one entry. Both entries normally receive the same image, so a correct answer cannot tell these roles apart. The roles matter for interpreting, compressing, and redesigning these models. Split inference, for example, sends every entry’s coded features between machines [12].

We therefore give main and the later entries different versions of one image (Figure 1). In the conflict test, one object is recolored for main only or for later only, and a question about its color shows which entry the answer follows. Earlier analyses instead remove DeepStack-style features altogether [37, 38] or patch hidden states that already mix the entries [7]. We use photographs, synthetic images of shapes and short text, GQA, and DocVQA. The conflict test covers nine checkpoints: the Qwen lineage (five Qwen3-VL sizes and two Cosmos3 models derived from them [32]), Granite Vision, and DeepStack. The other tests cover Qwen3-VL-8B/32B (Qwen8/32), Granite, and DeepStack.

Figure 1. Each entry can receive a different version of one photo. Answers follow the later entries in DeepStack and the main entry in Qwen3-VL-8B/32B (Qwen8/32). (a) One photograph edit: main keeps the original photo, and only the bus’s later-entry features come from the recolored photo. (b) Share of target answers that turn blue when only main or only later gets the edit, across the 128 photograph edits. At most four control answers change in any condition. All seven Qwen3-VL and Cosmos3 checkpoints favor main (Table 2).

Later entries can dominate photograph answers. When only DeepStack’s later entries receive the recolored object, 63% of answers about it change, against 5% when only main does. Qwen8/32 show the reverse: main changes 65–74% of these answers, and later changes none. All seven Qwen-lineage checkpoints favor main, and Granite leans later, more weakly. Models sharing one design thus give the deciding role to opposite entries.

Later influence depends on the evidence already supplied through main. Qwen’s later entries change no photograph answer, but they are not idle. On synthetic color images, we degrade only main’s image and keep conflicting later features unchanged. The degradation strengthens their effect on the log-probability margin between the two colors in all four models, even where it causes no errors. In Qwen32, later features with a conflicting stripe orientation switch 20% of target answers with clean main and 81% with mildly blurred main. Without the conflict, every answer stays correct, so accuracy alone would miss these changes.

Later entries can compensate for degraded main, with model and task limits. Gaining influence is not the same as repairing errors. When both entries receive a degraded image, restoring clean later features recovers part of the lost GQA accuracy in all four models. Although Qwen8 and Qwen32 lose equal accuracy at the strongest stripe blur, clean later features restore almost all of it in Qwen32 and little in Qwen8. Qwen8/32 recover little on short text and DocVQA.

Which entry a model can lose differs by model. If one entry can replace the other, a model may not need both. Recovery patterns predicted which entry each model can lose, and five of six tested predictions held on further GQA attribute questions. Qwen8 keeps its accuracy with later

zeroed. DeepStack scores at least 76.05% with either entry zeroed, against 79.50% with both intact, so here its later entries decide conflicts but add little. In split inference, sending only its main entry takes 49% fewer bytes than coding all entries, at similar accuracy (§ 5.5).

Across the four tests, models that share the DeepStack design use their later entries differently, and accuracy on matching inputs does not show it.

Related work

Visual interfaces. DeepStack adds spatial visual features at several decoder depths [30], Qwen3-VL adds intermediate encoder features [37], Cosmos3 keeps the Qwen3-VL interface after continued training [32], and Granite uses a DeepStack variant [16]. We keep each model’s own entry layout (§ 4). Other interfaces, object benchmarks, and intervention caveats appear in Supplementary Sec. J.

Internal visual pathways. Interventions inside vision-language models trace how visual information is carried and used. Causal tracing restores clean visual states after corruption [35]. Patch and attention interventions study object-local information [31]. The location of an OCR routing bottleneck depends on architecture [42]. Global views and local crops can play different roles in localization and classification [40]. Rahman patches activations in video models. Their order-recovery gap persists after the DeepStack-style features are removed [38]. We instead vary each entry separately under conflicting sources and degraded images (Table 1).

StudyWhat is separated?Question or finding
Zhang et al. [50]Image, question, and final-token positionsTwo stages transfer global and question-relevant visual information into question representations.
Salazar et al. [39]Direct image-token and text-mediated pathwaysPathway use depends on task and prompt; text mediation can support fallback.
Ghosh et al. [11]Conflicting image and text sourcesGeneration and likelihood margins reveal task-dependent reliance shifts under degradation, including after accuracy-loss calibration.
Rahman [38]Visual sequence and positional orderSimilar temporal scores conceal different channel reliance and failure modes.
Cui et al. [7]Visual ordering and language-model backup representationsVisual ordering supports spatial binding; amplifying it improves natural-image spatial judgments.
Our workMain and later visual entriesLater dominates in DeepStack, main in Qwen3-VL; main quality changes later influence; recovery and entry removal vary by model.

Table 1. Related analyses differ in the information sources and routes they separate. We vary each entry’s evidence separately to distinguish influence (under conflict or degraded main), recovery, and which entry a model can lose.

Spatial binding. Position-ID and spatial-ID interventions redirect feature retrieval and spatial judgments [3, 20]. Cui et al. patch Qwen3-VL’s early-layer states and link the effect to its DeepStack-style injection [7]. Those states mix upstream computation with the injected features, whereas we change each entry’s input separately. We find that Qwen3-VL’s later entries barely change color answers. This result is therefore compatible with their finding that visual ordering supports spatial binding.

Following an attribute through visual entries

One visual change, two object queries

Sources aa and bb contain the same objects and layout, and only one object’s attribute differs. The target query asks about this edited object, and the control query about the unchanged object. Both receive the same intervention (Figure 1). At entry ee, with features veav_e^a and vebv_e^b from the two sources, selected positions receive

vea+k(veb−vea),(1)v_e^a + k(v_e^b-v_e^a), \tag*{(1)}

where k=0k=0 keeps source aa and raw strength k=1k=1 installs source bb. We intervene on main alone or on all later entries together. On photographs, the other entry keeps the original image (Supplementary Table 5). A local transfer selects the tokens that overlap the edited object’s bounding box. Being contextualized, these tokens can carry information from outside the box. Other positions keep the surrounding scene (the background). Background strength α\alpha is defined in § 4, and α=1\alpha= 1 leaves the image intact.

Both queries are scored toward the edited value: for a red-to-blue edit, toward cbc_b = “blue” rather than cac_a = “red”. In every synthetic set, the edit gives the target the control’s value, so the control’s correct answer is already cbc_b. The same holds for 74 of 128 photograph controls (Supplementary Sec. B). For query qq in intervention state zz, the candidate score is yqscore=log⁡p(cb∣q,z)−log⁡p(ca∣q,z)y_q^{\mathrm{score}} = \log p(c_b \mid q,z) - \log p(c_a \mid q,z), summed over answer tokens without length normalization. Generation uses yqgen=1[c^q=cb]y_q^{\mathrm{gen}} = \mathbf{1}[\hat{c}_q = c_b] for a valid answer. Scores measure graded support, and generation the answer. The parser normalizes case and whitespace and accepts more forms than the scored strings, so the readouts can disagree without an intervention (Supplementary Sec. A.3). Scores are not pooled across models or answer formats.

Let g∈{main,later}g \in\{\mathrm{main},\mathrm{later}\} be main or all later entries together. In state zaz_a, gg receives source aa (k=0k = 0), and in zbz_b it receives the intervention (k=1k = 1 at raw strength). The directed effect on query qq is Eq,g=yq(zb)−yq(za)E_{q,g} = y_q(z_b) - y_q(z_a). Relative object selectivity and its entry contrast are

Rg=Etarget,g−Econtrol,g,ΔR=Rmain−Rlater.(2)R_g = E_{\mathrm{target},g} - E_{\mathrm{control},g}, \qquad\Delta R = R_{\mathrm{main}} - R_{\mathrm{later}}. \tag*{(2)}

Positive RgR_g means that the target moves toward cbc_b more than the control, not that only the target moves. We therefore report both components. Generation effects are net changes toward cbc_b, not accuracy or flip rates. A correctly answered control already answers cbc_b, so generation cannot detect a shift toward cbc_b that both queries share. Scores still register the control’s response. Subtraction cancels a shared shift only if both queries move equally, so scores limit a shared bias but do not exclude it.

Position and intervention magnitude

Local transfers test whether an effect follows the object or the image position. The transferred features and their mask go to the edited object’s position (original destination) or to the same position in the other image half (opposite destination). The destination effect is RgR_g at the opposite minus RgR_g at the original destination. We measure it in the original (M0M0) and exchanged (M1M1) layouts. Exchange changes every entry’s unselected positions but not the transferred features. If relative selectivity follows the object currently at a position, the destination effect reverses. The spatial interaction is its M1M1 value minus its M0M0 value, paired within scenes (Supplementary Sec. A.6). Because main’s input also changes under exchange, the interaction tests whether effects follow the current object, not which entry carries that object.

Magnitude controls test whether feature size alone explains an entry contrast. Each entry’s magnitude is the RMS of veb−veav_e^b-v_e^a divided by that of the decoder hidden state receiving the entry. Several later entries combine by root sum of squares (RSS) or by sum (L1). To match main, we keep main at raw strength and rescale the later difference to main’s magnitude. Matching to later reverses the roles (Supplementary Sec. A.5).

Conditional influence and recovery

To test whether later influence depends on main’s image, main receives a clean (CC) or degraded (DD) version of aa. Later receives one of two fixed endpoints, computed once and reused under CC and DD. Clean source-aa features give the non-conflict input, and the conflict endpoint installs source bb (k=1k = 1) on the edited object’s region. Only the decoder states built from main’s image change with main quality. The change in later influence is Rlater(D)−Rlater(C)R_{\mathrm{later}}(D) - R_{\mathrm{later}}(C), not the entry contrast ΔR\Delta R.

Recovery involves no conflict: main and later each receive the clean or degraded version of the same image at all positions, giving four cells. All cells are scored against the clean image’s answer and use the same degraded image. Let AuvA_{uv} be accuracy with main quality uu and later quality vv (for benchmarks, the metric of § 4). With main degraded, clean later features give a gain GDG_D, and IrecI_{\mathrm{rec}} is GDG_D minus their gain with main clean:

GD=ADC−ADD,Irec=GD−(ACC−ACD).(3)G_D = A_{DC} - A_{DD}, \qquad I_{\mathrm{rec}} = G_D - (A_{CC} - A_{CD}). \tag*{(3)}

Accuracy is main-limited when it tracks main’s quality alone (ACD≈ACCA_{CD} \approx A_{CC}, ADC≈ADDA_{DC} \approx A_{DD}), later-limited when it tracks later’s quality alone (ADC≈ACCA_{DC} \approx A_{CC}, ACD≈ADDA_{CD} \approx A_{DD}), and either-sufficient when one clean entry suffices (ACD≈ADC≈ACCA_{CD} \approx A_{DC} \approx A_{CC}). In the last case, GDG_D and IrecI_{\mathrm{rec}} approach the joint loss ACC−ADDA_{CC} - A_{DD}. Each label applies to one metric, answer format, and level, and only where the joint loss is clear. For the same task and degradation, conflict and recovery share the CCCC and DCDC inputs. Thus ACCA_{CC} and ADCA_{DC} are the accuracies on the non-conflict input.

Experimental setup

Materials. The photograph panel contains 64 photographs from GQA and Visual Genome, chosen without looking at model outputs [15, 22]. In each, two checked, nonoverlapping objects are recolored in turn on a 384×384384 \times384 canvas, with a component mask restricting each edit to one object. This yields 128 red-to-blue image pairs and 256 queries. The controlled panel has 64 synthetic templates of two shapes each. They cross shape pairs, layouts, and attribute assignments, and color and stripe versions share geometry. Local transfers vary layout, destination, entry, and background strength α∈{0.08,1}\alpha\in\{0.08, 1\}. Background strength blends the object textures of a template, or a whole photograph, toward gray. Layout exchange swaps the two halves of a template and mirrors a photograph. Supplementary Sec. A gives construction and filtering details.

The fixed-contrast experiment (§ 3.3) scales image contrast by 0.12 without resizing. It uses the same 64 photographs and 32 further templates for color, stripe orientation, the position of a mark within one shape (relative position), and reading short text (512 queries). It also blurs GQA and DocVQA images, which causes little loss (Supplementary Sec. F).

The calibrated experiment desaturates color, blurs stripes and relative position (scaled to the stimulus), and lowers the resolution of text, GQA, and DocVQA. On a separate calibration set, we chose three levels per task using only clean accuracy and accuracy with both entries degraded. With both entries degraded, models should keep a median of 0.75, 0.5, and 0.25 of their clean above-chance accuracy at these levels. These shared levels and the degraded images are the same for all models. Evaluation uses other items: 32 templates per synthetic task (balanced for object side and answer), 192 GQA validation-balanced images per question group, and 192 Single-Document DocVQA validation documents [15, 28]. GQA answers are matched after normalizing case and punctuation, and DocVQA uses the organizers’ ANLS (average normalized Levenshtein similarity). Other metrics are sensitivity analyses (Supplementary Sec. G).

Models and interfaces. We evaluate five Qwen3-VL sizes (2B, 4B, 8B, 30B-A3B, 32B), the Cosmos3-Nano/Super understanding towers, Granite Vision 4.1-4B, and DeepStack-L-Vicuna-7B, all in BF16 on NVIDIA GPUs [16, 30, 33, 34, 37]. The Cosmos3 towers descend from Qwen8/32 [32], so the seven Qwen-lineage checkpoints share one interface and give lineage coverage, not independent architectures.

Qwen’s main entry uses the final encoder features, and its three later entries use intermediate encoder features. Separate learned projectors (mergers) give all four the same visual grid and token count, and the later features are added to the outputs of the first three decoder layers. Granite’s main entry also uses the final encoder features. Its seven later entries are three intermediate-layer entries and four spatial Q-Former entries, and its main and spatial branches reduce their visual grids to the same 12 × 12 output. DeepStack instead uses one CLIP layer and one shared projector for a global main view and four spatial later entries (groups). The groups are added to the outputs of its first four decoder layers. Rectangular synthetic inputs activate only two of the four groups, and the model’s preprocessing zero-pads the rest. On square photographs, DeepStack’s main entry sees one 336-pixel view as 576 tokens. Its four later groups take 576 tokens each from a 672-pixel canvas. Together they hold four times main’s tokens, at twice its linear sampling density. Supplementary Sec. A gives exact checkpoints and geometry.

Readouts and inference. Outside the benchmarks, each query has three answer formats. Direct answers name the attribute (“blue”). Identity and cyclic labels give the number of a listed option. The options are in fixed (identity) or shifted (cyclic) order, and with two options the shift reverses them. Greedy generation allows 32 tokens, or 64 for benchmarks. Items are never selected by whether a model answers them correctly. Self-patching, which reinstalls a source’s own features, checks that interventions are delivered as specified.

Comparisons are paired within photographs, templates, or source documents, which define nn, so repeated queries do not enlarge it. Conflict contrasts use complete cases within each edit: every answer the contrast needs must be valid. The nn for ΔR\Delta R can therefore differ from that for RmainR_{\mathrm{main}} or RlaterR_{\mathrm{later}}. Benchmark questions are averaged within source images or documents first, and recovery accuracy counts invalid answers as incorrect. Brackets give 95% intervals: descriptive normal for both panels and for benchmarks, Student-tt for templates and photographs in the fixed-contrast and calibrated experiments, and paired bootstrap in § 5.5. All intervals are unadjusted. Apart from the predictive fixed in advance in § 5.5, analyses are exploratory.

Results

Entry effects follow the object at each position

Before comparing entries, we check what a local transfer acts on. Relative selectivity changes when the transferred features move to the other object’s position. Exchanging the two objects reverses this change (Figure 2). Since both queries receive the same intervention, the object now at the transferred position decides which query responds more. For Qwen32’s later color scores, for example, the paired spatial interaction is 31.994 [29.954, 34.035].

Figure 2. Entry effects follow the object at each position. Each point uses the 64 color templates of the controlled panel, direct answers, and background strength α=0.08\alpha= 0.08. Each model has its own score scale. Features and their region mask are placed at the original (Orig.) or opposite (Opp.) coordinate, in the original layout (M0M0) or with the objects exchanged (M1M1). Numbers label Qwen8’s small later effects. Error bars are descriptive normal 95% intervals; most are smaller than the markers.

DeepStack’s later entries show the same reversal in color generation. On stripes, paired comparisons in all three answer formats favor a larger spatial interaction for main in Qwen8/32 and for later in Granite. At full background strength (α=1\alpha= 1), the Cosmos3 comparisons follow the Qwen direction (Supplementary Secs. B, C, and I2).

DeepStack follows later, Qwen follows main

On photographs, answers follow the later entries in DeepStack (ΔR=−0.547\Delta R = -0.547 [−0.649, −0.445]) and the main entry in all seven Qwen-lineage checkpoints (Table 2). Granite leans later, more weakly (−0.086 [−0.160, −0.012]).

Table 2. On photographs, answers follow the later entries in DeepStack and the main entry in Qwen3-VL. Generation selectivity is R=Etarget−EcontrolR = E_{\mathrm{target}} - E_{\mathrm{control}}; positive ΔR\Delta R means larger relative selectivity at main. Conditions are M0M0, α=1\alpha= 1, direct answers, and raw k=1k = 1. Cells are means [descriptive normal 95% intervals]. Each estimate uses nn complete photographs out of 64. Main is one entry; later contains three entries for the Qwen lineage, seven for Granite, and four for DeepStack on these images. Shaded: models used in § 5.3–§ 5.5.

CheckpointRmainR_{\mathrm{main}} [95% interval]nnRlaterR_{\mathrm{later}} [95% interval]nnΔR\Delta R [95% interval]nn
Qwen3-VL-2B0.414 [0.313, 0.515]640.008 [−0.027, 0.042]640.406 [0.307, 0.506]64
Qwen3-VL-4B0.694 [0.621, 0.766]62−0.008 [−0.023, 0.008]630.702 [0.632, 0.771]62
Qwen3-VL-8B0.645 [0.563, 0.728]620.000 [0.000, 0.000]640.645 [0.563, 0.728]62
Qwen3-VL-30B-A3B0.516 [0.427, 0.604]630.016 [−0.022, 0.053]640.500 [0.409, 0.591]63
Qwen3-VL-32B0.730 [0.651, 0.810]630.000 [0.000, 0.000]640.730 [0.651, 0.810]63
Cosmos3-Nano0.605 [0.523, 0.687]620.000 [0.000, 0.000]640.605 [0.523, 0.687]62
Cosmos3-Super0.754 [0.681, 0.827]630.008 [−0.008, 0.023]630.746 [0.670, 0.822]63
Granite Vision 4.1-4B0.180 [0.113, 0.246]640.266 [0.184, 0.347]64−0.086 [−0.160, −0.012]64
DeepStack-L-Vicuna-7B0.047 [0.011, 0.083]640.594 [0.496, 0.691]64−0.547 [−0.649, −0.445]64

Table 2.

Target and control exemplars explain the contrast. On the photographs with complete results for both entries,

Qwen8’s main entry changes 81/124 target answers toward bb and one control answer. Qwen32’s main entry changes 93/126 targets and one control. Neither model changes any answer through its later entries. Later interventions still raise target scores more than control scores, by 0.686 [0.523, 0.848] in Qwen8 and 1.302 [1.056, 1.549] in Qwen32 (Supplementary § 17.§ 5). DeepStack’s later entries change 80/128 targets and four controls, and its main entry changes six targets and no controls (Supplementary Figure 8). The contrast holds when we keep only queries that the model first answers red (Supplementary Table 7; Supplementary Sec. B).

Answer formats and magnitude matching keep the direction in Qwen8, Qwen32, Cosmos3-Super, and DeepStack. In every format, Qwen8, Qwen32, and Cosmos3-Super have ΔR\Delta R intervals above zero for scores and generation, at raw strength and under all four magnitude matchings (to main’s or later’s magnitude, by RSS or L1; Supplementary Sec. E). DeepStack’s intervals are below zero in all 15 comparisons per measure (five settings in three formats). At raw strength, Granite’s contrasts favor later or have intervals containing zero in every format, most clearly for generation with identity labels (−0.219 [−0.300, −0.137]). Under L1 matching, however, its score contrast for direct answers reverses sign (Supplementary Sec. E).

Later influence depends on main quality

A weak later effect under clean main need not stay weak when main is degraded. On synthetic color, reducing main contrast or saturation increases the score selectivity of fixed later features in all four models. In the fixed-contrast experiment, Rlater(D)−Rlater(C)R_{\mathrm{later}}(D) - R_{\mathrm{later}}(C) has intervals above zero for all four models in all three formats (32 templates; Supplemen-

tary Table 12). The later features are identical under both main qualities, so the change must come from the decoder states built from main’s image. On the calibrated experiment’s evaluation templates, all 36 color score changes again have intervals above zero (four models, three formats, three levels). At the strongest desaturation, generation selectivity with direct answers also rises in all four models, with intervals above zero (Qwen32: 0.422 [0.276, 0.567]).

Accuracy does not show this increase. At the mildest shared desaturation level, all four models answer every query correctly with the non-conflict input, whether main is clean or degraded (ACC=ADC=1A_{CC}=A_{DC}=1; direct answers, 32 templates). Yet the fixed later conflict has a larger score effect under degraded main in every model, with all four paired intervals above zero. Generation selectivity also increases, by 0.188 [0.060, 0.315] in Granite and 0.375 [0.238, 0.512] in DeepStack (Supplementary Sec. G).

A large response change at perfect accuracy. At the mildest stripe blur, Qwen32 answers every query correctly in all four recovery cells. Degrading main alone lowers the correct answer’s log odds only from 19.6 to 18.5. With clean main, the later features switch 0.203 of target answers toward bb (Figure 3b). With degraded main, the same features switch 0.812, a rise of 0.609 [0.500, 0.719]. The rise is again 0.609 with identity labels and 0.469 [0.340, 0.598] with cyclic labels, and it also holds for relative position (Supplementary § 17.§ 6). Candidate scores already show the change. The later intervention’s mean effect on the target score rises from 8.8 to 27.6, while its effect on the control moves only from −1.1 to −1.5. The paired selectivity change is 19.163 [16.308, 22.017]. If main and later contributions simply added, the fixed later change would have the same score effect under both main states. So the dependence is not only a by-product of turning scores into answers. The direction varies outside synthetic color. Degrading main lowers later score selectivity for Qwen8’s stripes at the strongest blur (−1.681-1.681 [−2.397-2.397, −0.965-0.965]) and for Granite’s relative position. On photographs with direct answers, lowering main contrast increases later score selectivity in Qwen8/32, with an uncertain change in DeepStack and a negative change in Granite. The Qwen increase barely reaches answers: Qwen32’s generation selectivity rises by only 0.008 [−0.027-0.027, 0.043] (Supplementary Sec. F). With clean main, later interventions on photographs change Qwen8/32’s candidate scores little: a median 1.04% of the pre-intervention score in Qwen8 and 2.35% in Qwen32, against 77.85% in Granite (Supplementary Sec. I.5).

Figure 3. Recovery depends on model and task; fixed later features gain influence under mild main blur. (a) Strongest calibrated level per task. Cells name main first (C clean, D degraded); hatched cells have degraded later input. Generation accuracy on 32 synthetic templates in the answer format named under each model, GQA accuracy on 192 images, and ANLS on 192 DocVQA documents. Qwen32’s stripe cells are identical under identity labels. (b) Qwen32 stripes, direct answers, as main blur grows (σ\sigma as a fraction of the stripe period): DC and DD accuracy and the later conflict effect RR (Eq. (2), later inputs fixed). Error bars are descriptive, unadjusted 95% intervals. Zero-SE cells have none. Complete values: Supplementary Sec. G.

Later recovery depends on model and task

Clean later features also recover part of the GQA accuracy lost to degradation in all four models. We restore later under degraded main at the strongest shared levels: a 5-pixel long side for attributes and 10 for spatial questions. Across models, GDG_D is 0.070–0.271 for attributes and 0.057–0.279 for spatial questions. For both tasks and all four models, the GDG_D and IrecI_{\mathrm{rec}} intervals lie above zero, so each gain exceeds the gain under clean main (Supplementary Sec. G).

On GQA attributes, the four cells show which entry limits accuracy in each model. Over the three shared levels, Qwen32’s GDG_D rises from 0.060 to 0.146 and 0.224. Qwen8’s is only 0.070 at 5 pixels. At that level, Qwen32’s IrecI_{\mathrm{rec}} is 0.190 [0.123, 0.257]. DeepStack is either-sufficient: degrading either entry alone costs at most 0.042 of its 0.289 joint loss, with Irec=0.229I_{\mathrm{rec}} = 0.229 [0.167, 0.291]. Granite is closer to later-limited: degrading later alone costs 0.198 of its 0.266 joint loss, leaving IrecI_{\mathrm{rec}} between 0.062 and 0.070 across levels.

Equal stripe losses, different recovery. With the same degraded images and direct answers, Qwen8 and Qwen32 both lose 0.500 stripe accuracy (paired loss difference 0.000 [−0.065-0.065, 0.065]). Clean later features restore 0.484 [0.453, 0.516] of it in Qwen32 but only 0.078 [−0.060-0.060, 0.216] in Qwen8, a paired difference of 0.406 [0.266, 0.547] (Figure 3a). Qwen8 is closer to main-limited, and Qwen32 is close to either-sufficient. With identity labels, Qwen32 again recovers stripes but not text. Both tasks fall from 1.000 to 0.500 when both entries are degraded (paired loss difference 0.000 [−0.171-0.171, 0.171]), and clean later features restore 0.000 [−0.046-0.046, 0.046] of text accuracy (Figure 3a). The recovery difference between stripes and text is 0.484 [0.429, 0.540]. Qwen8’s text gain has no interval above zero at any level or in any format. Granite’s relative-position accuracy is later-limited: degrading later alone lowers ACDA_{CD} to 0.500, close to ADDA_{DD}. Clean later features therefore help about as much under either main quality, so their IrecI_{\mathrm{rec}} interval contains zero despite a large recovery of 0.406 [0.299, 0.513] (Supplementary Sec. G).

Little DocVQA recovery in Qwen8/32. At a 180-pixel long side, Qwen8 and Qwen32 lose 0.516 and 0.461 DocVQA ANLS when both entries are degraded, and their accuracy is main-limited: GDG_D never exceeds 0.030, and every IrecI_{\mathrm{rec}} interval contains zero. Qwen3-VL reports DocVQA and InfoVQA gains from adding DeepStack-style injection in pretraining [37], whereas we change the inputs of fixed, post-trained models. Granite’s IrecI_{\mathrm{rec}} is positive under ANLS but negative under lowercase exact matching (Supplementary Sec. G). DeepStack’s accuracy looks later-limited, but the lower resolution removes more of its high-resolution later input than its global main view.

Which entry a model can lose differs by model

We test whether the entry relationships of § 5.4 predict which entry each model can lose. This matters for split inference (§ 1). Heidari et al. [12] code Qwen3-VL’s four entry tensors at one shared quantization parameter (QP; larger is coarser) and leave the split of bytes across entries to future work. We remove an entry’s image information by setting its features to zeros or to their mean over 48 calibration images. We also code each entry tensor with NNCodec [4], an implementation of the ISO/IEC neural network coding standard, and count bitstream bytes per image. A zeroed entry still runs through the model, so bytes measure features, not latency or memory. We use 1,000 new GQA images with 1,838 attribute questions (Granite: 999 images). Accuracies here are percentages, and differences are percentage points (Supplementary Sec. H).

Which entry suffices depends on the model, as in § 5.4, and on the replacement value (Table 3). With later zeroed, Qwen8 scores 72.20% against 72.50% for the full input (−0.30-0.30 [−1.55-1.55, 0.95]). With main set to its mean, Granite scores 69.77% against 73.27% (−3.50-3.50 [−5.16-5.16, −1.85-1.85]). With either entry zeroed, DeepStack scores at least 76.05% against 79.50% (at worst −3.45-3.45 [−4.80-4.80, −2.10-2.10]). The replacement matters: zeroing main instead lowers Granite from 69.77% to 11.66% but raises DeepStack from 29.35% to 76.05%. Before these images were scored, we predicted which single entry each model can keep: main for Qwen8, later for Granite, and either entry for Qwen32 and DeepStack. Kept alone, each should cost at most 25% (Qwen8, Granite) or 50% (Qwen32, DeepStack) of the gap between the full input and both entries zeroed. With zeros, five of six tested predictions held, and Granite’s failed (Supplementary Sec. H.2).

Main inputLater inputQwen8Qwen32GraniteDeepStack
imageimage72.5072.7573.2779.50
imagezeros72.2071.9051.6077.85
imagemean71.9572.6558.8175.70
zerosimage54.4567.6011.6676.05
meanimage50.5065.4069.7729.35
zeroszeros31.5543.201.0011.45
meanmean43.1043.3042.2910.35

Table 3. The entry that keeps accuracy alone differs by model and replacement. GQA attribute accuracy (%) on 1,000 new images (Granite: 999 images). Each entry receives the image’s features, zeros (all its features set to zero), or its mean feature over 48 calibration images. Intervals: Supplementary Table 19.

Sending only DeepStack’s main entry roughly halves the bytes at similar accuracy. With later zeroed, main alone at QP −4 takes 255.08 KB and scores 77.45%. All entries at shared QP −2 take 503.26 KB and score 76.35%. That is 49.31% fewer bytes at a difference of 1.10 [−0.40, 2.55]. We fixed this comparison after an exploratory run had shown the same pattern and after the first results on these images, which included uncoded main alone (77.85%; Supplementary Sec. H.3). Post hoc, at a similar size, all entries at shared QP 0 (243.13 KB) score 42.65% (Supplementary Table 22). A second comparison starts from the scale-normalized setting, which scales each later entry’s quantization step with its feature scale (Supplementary Sec. H.1). We selected a configuration on calibration images before testing. It keeps main at QP −2 and codes later two QP steps coarser than this setting. Against this setting at the same main QP, it takes 39.80% fewer bytes (278.19 against 462.14 KB) and scores 75.80% (−0.45 [−1.40, 0.50]). At equal main QP, dropping later removes about 74% of the bytes and lowers accuracy by 1.45 to 3.60 points (Supplementary Table 23).

Granite and Qwen8/32 show no coding gain that we can credit to entry roles. Qwen8/32 do not consistently beat a shared QP at the tested budgets (Supplementary Sec. H.5). Granite is closer to later-limited, so we predicted that its later entries should be coded more finely. This prediction failed (Supplementary Sec. H.6). A different configuration, found post hoc, moves Granite’s bytes from main to later. On 531 held-out images, it scores 69.30% against 66.67% for a shared QP at nearly equal bytes (2.64 [0.47, 4.80]; Supplementary Table 24). But the scale-normalized setting codes later the same way. On the 468 images where the configuration was found, it already scored 2.46 [0.11, 4.81] points above the shared QP.

Discussion and limitations

What later inputs contribute. An entry’s place in the architecture does not fix its role in the trained model. Later entries change 63% of photograph answers about the edited object in DeepStack and none in Qwen8/32. Their influence also depends on main. In Qwen32, the same later stripe features switch 20% of target answers with clean main and 81% with mildly blurred main.

Using the findings. The entry a model needs cannot be read off its architecture: on GQA attributes, Qwen8 needs main, DeepStack needs either entry, and Granite needs later when main is set to its mean. A system should test its own checkpoint and task if it drops or codes entry features, as in split inference. The recovery test predicted five of six single-entry outcomes on new images.

Differences between the designs. Feature construction, projectors, group sizes, sampling, and injection depth differ across the designs and covary with training, so we cannot say which of them makes DeepStack follow later and Qwen follow main. Our interventions show how answers depend on the evidence at each entry, not the mechanism behind it. A stronger later effect under degraded main does not show that a model detects the degradation.

Limitations. The equal-loss comparisons are equal only in sample mean, and the stripe-text comparison also changes the stimulus and the failure mode. Qwen’s direct color accuracy saturates at contrast 0.12, Granite’s direct stripe recognition is weak, and several recovery patterns hold only for direct answers. The photograph panel covers red-to-blue edits of 64 images, and GQA and DocVQA results come from selected validation subsets. GQA accuracy that survives a 5-pixel long side may reflect language priors or coarse visual cues, which we did not separate. Removal and coding results come mostly from GQA attributes. Dropping DeepStack’s later entries costs 1.65 points on these attribute images. In exploratory runs on an Apple-silicon GPU, it costs 3.91 points on GQA spatial questions and 9.48 ANLS points on DocVQA (Supplementary Sec. H.3).

Conclusion

We gave each entry of DeepStack-style models a different version of one image. This test shows what the later entries contribute, and the answer differs between models that share the design. Later entries decide photograph answers in DeepStack, and main decides them in Qwen3-VL. In both, later features can gain influence when main is degraded. On GQA attribute questions, Qwen8 can lose its later entries and DeepStack either entry. In split inference, DeepStack can then send only its main entry, which roughly halves the bytes. Accuracy on matching inputs shows none of this. The test needs no training and applies to any model with several visual entries.

LLM Usage Statement

OpenAI Codex and Anthropic Claude Code assisted research planning, code and experiment execution, analysis, figures, literature review, and substantive manuscript drafting and editing. The authors are responsible for all content and results.

Evidence includes two quality experiments: a fixed-contrast experiment (Appendix § 14) and calibrated degradation curves on held-out units (Appendix § 15). A coding experiment removes or codes each entry’s features on new GQA images and DocVQA documents (Appendix § 16). This supplement gives intervention and setup definitions, supporting figures, paired analyses of the conflict panels, the quality, recovery and coding results that the main text cites, and further experiments (Appendix § 17). Complete estimates for every model, task, readout, level and metric of the two quality experiments, and for every configuration and comparison of the coding experiment, are in the data folder of the code repository (Appendix § 19). Reference numbers follow the shared bibliography in the main paper.

Methods and reproducibility details

Checkpoints and native visual entries

Table 4 identifies the nine fixed checkpoints. All complete the photograph, synthetic local, whole-source, and dose panels; Qwen8, Qwen32, and Granite also complete the native-entry decomposition, giving 39 model/panel analyses. This three-model subset was chosen before any decomposition result existed. The quality, removal and coding experiments use these three models and original DeepStack. The five Qwen3-VL models and two Cosmos3 understanding towers represent seven checkpoints within one multidepth interface family, not seven independent architectures [33, 34, 37]. Cosmos Nano and Super initialize from Qwen3-VL-8B and 32B and continue training the vision encoder, projectors, and language model [32]; their system names do not specify the loaded understanding tower’s parameter count.

Qwen3-VL-2B/4B take intermediate vision features from encoder indices 5, 11, and 17; the remaining Qwen/Cosmos towers use 8, 16, and 24. Separate learned mergers produce later representations added after decoder blocks 0, 1, and 2. Main uses the final vision representation, and the output grids align. Granite maps vision indices −1, −7, −13, −19 to decoder entries 0, 3, 6, and 9 [16]. Entries 12, 15, 18, and 21 use final-layer features with four phase-seeded Q-Former branches. Each branch compresses a 24×2424 \times24 grid to 12×1212 \times12; within an 8×88 \times8 window, a phase changes query initialization but cross-attention can read the whole window.

Original DeepStack uses CLIP hidden-state index −2 and a shared projector for its global 336×336336 \times336 main view and dense spatial groups [30]. Four later positions enter at decoder layers 1–4. Square photograph inputs activate four groups, whereas rectangular synthetic inputs activate two and retain two officially zero-padded groups; each active group has 576 tokens. “All later” always includes all native later positions. These comparisons preserve native geometry and jointly vary feature depth, spatial sampling, projector, token count, entry count, decoder depth, and training. They do not isolate architecture as a causal factor. Primary inference uses BF16 CUDA with architecture-specific adapters; the quality experiments’ batching checks are reported in Appendix § 14. The photograph and controlled panel experiments ran on single NVIDIA A100 GPUs.

Three implementations of DeepStack-style injection. The nine checkpoints implement multi-layer injection in three independent ways. Qwen3-VL names its later entries DeepStack [37], and Granite’s model card describes its design as “a variant of Deepstack” in which visual features are added to the language-model hidden states at multiple layers [16]. The three differ in what the later entries add to main. In original DeepStack, main is one 336-pixel global view of 576 tokens, whereas the later groups take 336-pixel crops from a 672-pixel canvas (up to 1008 pixels on DocVQA), 576 tokens each, through the shared projector. On square photographs the later entries therefore carry four times main’s tokens at twice its linear sampling density. Qwen’s later entries share main’s grid and token count and differ in encoder depth and merger. Granite’s intermediate entries likewise read main’s final-layer features and 24×2424 \times24 grid and compress them to the same 12×1212 \times12 output; they differ from main in query phase, branch weights, and injection depth. Only DeepStack’s later entries thus add tokens or input resolution.

Photographs, synthetic scenes, and intervention support

The photograph panel contains 64 images from fixed 2019 GQA scene graphs and Visual Genome sources [15, 22], excluding 320 photographs used during method development. Development experiments used other materials; Appendix § 17 reports the informative ones, and none is pooled with the main panels. Selection and visual review do not inspect model responses. Each image has two checked referents with nonoverlapping boxes and is fitted, without object cropping, to a 384 × 384 gray canvas. Within each box, a fixed HSV rule selects the largest connected color component; recoloring changes hue to red or blue while preserving saturation, value, and all other pixels. This is controlled recoloring rather than instance segmentation. Editing each object gives 128 edits and 256 target/control queries, always directed red-to-blue. Queries identify the referent without naming its tested color.

The pixel-edit mask differs from intervention support: selected visual tokens follow the object box and can include surrounding context. Backgrounds are original or horizontally mirrored photographs (M0/M1M0/M1), blended toward gray at strength α=1\alpha= 1 or .08. Mirroring need not move an object onto the other referent’s coordinates. Raw interventions cover both backgrounds and strengths; dose matching to the same native target uses M0M0, α=1\alpha= 1.

The paired synthetic panel uses a fixed seed and 64 templates: two shape pairs, two layouts, two assignments, and eight replicates. Color (red/blue) and stripe (vertical/horizontal) versions share geometry on a 384 × 256 canvas. Stripes have period 14 and bar width 6. Each object is edited in turn, with target and control queries. Local conditions cross entry group, original/opposite destination, M0/M1M0/M1, and α∈{.08,1}\alpha\in\{.08, 1\}. Here attenuation blends object texture toward gray while preserving shape and background. M1M1 exchanges equal image halves pixel for pixel; no extra shape transformation is applied. Across layouts, the local donor features and their mask at a given destination stay fixed, while the scene supplying main and the unselected later tokens changes. The layout contrast therefore tests current object–position correspondence without isolating main’s contribution to it. Moving an intervention transports its contextualized features and token mask together.

Whole-source conditions use both strengths at one, four layout pairs, and all main/later source assignments. Dose and native-entry decomposition use a 32-template subset. A deterministic hash ranking, balanced across shape, layout, and assignment, selects it without using outcomes. Dose retains full-source support, M0/M0M0/M0, and both fixed-source assignments; decomposition crosses layouts and strengths. All repeated edits, queries, formats, and conditions remain clustered by their shared photograph or template.

Readouts, pairing, and intervals

For the edited object’s directed a→ba \to b attribute change, the score is yqscore=log⁡p(cb∣q,z)−log⁡p(ca∣q,z)y_q^{\mathrm{score}} = \log p(c_b \mid q,z) - \log p(c_a \mid q,z) and valid generation is yqgen=1[c^q=cb]y_q^{\mathrm{gen}} = \mathbf{1}[\hat{c}_q = c_b]. This direction is retained for control queries and reversed numeric mappings. We form paired source effects Eq=yq(zb)−yq(za)E_q = y_q(z_b) - y_q(z_a), selectivity Rq=Etarget,q−Econtrol,qR_q = E_{\mathrm{target},q} - E_{\mathrm{control},q}, and the entry contrast ΔR=Rmain−Rlater\Delta R = R_{\mathrm{main}} - R_{\mathrm{later}}. Generation effects are net changes toward bb, generally neither accuracy nor total transition rates. Positive RR can reflect target increases, control decreases, or both.

Direct parsing normalizes case and surrounding whitespace and permits an optional final period, but requires a complete candidate phrase. Numeric parsing requires one unambiguous legal label; options (1)/(2) appear in identity or reversed (cyclic) order. Generation is greedy with a 32-token limit in the conflict panels. Invalid, truncated, missing, and unmatched outcomes remain unavailable there; native correctness is a separate diagnostic, not an inclusion filter. Appendix § 14 specifies the accuracy policies of the quality experiments and their 64-token benchmark setting.

Candidate scoring sums the declared continuation’s token log probabilities without length normalization or appended EOS. It therefore measures a narrower event than the generation parser accepts: a lower-case phrase excludes its capitalized spelling, while a phrase and that phrase plus a period can overlap as continuation events. Their probabilities cannot simply be summed into mutually exclusive semantic answers. Alternative complete-answer scoring has not been run, and raw score scales are not pooled across models, attributes, or formats.

Contrasts are formed within edits and averaged over both edited objects only when all required observations are valid. Each estimate reports its complete-unit nn; single-cell denominators cannot replace a paired contrast’s intersection. Conflict-panel intervals use xˉ±1.96s/n\bar{x} \pm1.96s/\sqrt{n}. Quality-experiment synthetic/photo estimates use Student-tt quantiles, while benchmark estimates retain the scorer’s normal approximation. All are descriptive 95% intervals without multiplicity adjustment. Contrasts are estimated from paired unit differences, not aggregate interval endpoints. Zero-containing intervals do not establish equivalence, and zero standard errors describe the observed sample.

Fixed source tables and role exchange

For fixed physical main/later layouts and strengths, let yuvy_{uv} denote the response with source assignments u,vu,v. We distinguish

Dfixed=yba−yab,J=ybb−yba−yab+yaa.(4)D_{\mathrm{fixed}} = y_{ba} - y_{ab}, \qquad J = y_{bb} - y_{ba} - y_{ab} + y_{aa}. \tag*{(4)}

The first compares conflicting assignments; the second measures output nonadditivity. With the other source fixed to aa or to bb, subtracting entry effects gives the same $D_{\mathrm{fixed}} algebraically, not two replications. In a role exchange, the changing source’s layout and strength follow its identity as it moves between entries. Mixed-layout exchanges span different physical tables. Their DswapD_{\mathrm{swap}} cannot replace DfixedD_{\mathrm{fixed}}, JJ, or the query-relative ΔR\Delta R.

Operational dose and native-entry decomposition

Throughout the supplement, dose means the intervention magnitude of § 3.2 in the main paper, and dose matching means matching that magnitude. A selected entry installs vea+k(veb−vea)v_e^a+k(v_e^b-v_e^a) on its specified tokens; k=1k=1 replaces source features and k>1k>1 extrapolates. Equal replacement need not give equal magnitudes across entries. For each edited object and role/context, a canonical forward pass on a direct target query measures the post-entry residual heah_e^a on the aa side. On selected tokens, define

re=RMS⁡(veb−vea)RMS⁡(hea)(5)r_e=\frac{\operatorname{RMS}(v_e^b-v_e^a)}{\operatorname{RMS}(h_e^a)} \tag*{(5)}
ρRSS(G)=∑e∈Gre2,ρL1(G)=∑e∈Gre.\rho_{\mathrm{RSS}}(G)=\sqrt{\sum_{e\in G}r_e^2},\qquad\rho_{\mathrm{L1}}(G)=\sum_{e\in G}r_e.

L1 is a sum of entry RMS ratios, not an elementwise tensor norm; RSS assumes no cross-layer orthogonality. A main-native target keeps main raw and scales later to main’s native dose; a later-native target reverses these roles. Target/control queries and answer mappings reuse the selected tokens, canonical denominators, and multiplier. Delivery is checked at inference precision; unmatched cases remain unavailable. Matching equates the declared measure, not semantic information or every query’s residual normalization. Since L1 exceeds RSS for multi-entry groups, it reduces later’s multiplier at the main-native target or increases main’s at the later-native target. These are different physical interventions; their raw anchors must remain visible.

Decomposition starts from a common baseline BB, installing donor-aa ROI features at every entry over a fixed background. Only the selected group receives its increment. One complete forward pass on BB supplies all denominators; the common-low target is the minimum native singleton ratio across main and all later entries. A group’s multiplier is that target divided by its RSS or L1 aggregate. Qwen8/32 test three singletons, their three pairs, and all later entries. Granite tests seven singletons, triples (3,6,9)(3, 6, 9), (15,18,21)(15, 18, 21), (3,12,21)(3, 12, 21), and all seven later entries. The spatial triple excludes layer 12.

At raw strength only, C=Rall later−∑eR{e}C=R_{\mathrm{all\ later}}-\sum_eR_{\{e\}} compares identical joint and singleton increments from BB. For Granite, G=R{15,18,21}−R{3,6,9}G=R_{\{15,18,21\}}-R_{\{3,6,9\}} compares equal-count spatial and intermediate subsets, and H=Gstripe−GcolorH=G_{\mathrm{stripe}}-G_{\mathrm{color}} is their paired attribute interaction. A separate experiment measures M21∣15,18=R{15,18,21}−R{15,18}M_{21|15,18}=R_{\{15,18,21\}}-R_{\{15,18\}} on the fixed-contrast templates, keeping entries 15 and 18 fixed. Neither CC nor this conditional marginal assigns additive neural contributions. Matched groups have different multipliers and are not used for this additive contrast. Qwen32’s direction-aligned layout comparison is −C(M1)−C(M0)-C(M1)-C(M0), whether or not its components are separately antisymmetric. Granite’s equal-count comparison is Gattribute=R{15,18,21}−R{3,6,9}G_{\mathrm{attribute}}=R_{\{15,18,21\}}-R_{\{3,6,9\}}, with paired interaction H=Gstripe−GcolorH=G_{\mathrm{stripe}}-G_{\mathrm{color}}. These output contrasts do not establish neural synergy or separate feature type from decoder depth and dose.

Intervention specification

Local transfers cross original and exchanged layouts (M0/M1)(M0/M1) with original and opposite destinations (o/p)(o/p). The spatial interaction

Ig=[Rg(M1,p)−Rg(M1,o)]−[Rg(M0,p)−Rg(M0,o)](6)I_g=[R_g(M1,p)-R_g(M1,o)]-[R_g(M0,p)-R_g(M0,o)] \tag*{(6)}

tests whether destination effects change with the object currently occupying that position; the entry difference Imain−IlaterI_{\mathrm{main}}-I_{\mathrm{later}} is paired directly. Table 5 lists, for each experiment family, the source identity, intervened entries, spatial scope, dose, readouts and inference unit. Designed contrasts vary specified factors within a family: the entry set for ΔR\Delta R, the multiplier for matched ΔR\Delta R, scene layout and destination for II, main quality for Iquality=Rlater(D)−Rlater(C)I_{\mathrm{quality}}=R_{\mathrm{later}}(D)-R_{\mathrm{later}}(C), main and later quality for GDG_D and IrecI_{\mathrm{rec}}, the readout for cross-format differences, and the checkpoint for Qwen8 versus Qwen32 at one level. Comparisons across families, such as photo conflict against synthetic recovery or stripes against text, change several elements at once and are descriptive.

Family (estimands)Source identityEntries intervenedSpatial scopeDose / multiplierReadouts, kindsUnit
Photo conflict (RR, ΔR\Delta R)a/ba/b: referent recolored red/blue, clean; unchanged entry and non-ROI tokens show the original or mirrored photo at α∈{1,.08}\alpha\in\{1,.08\}, edited object in its native colormain alone, or all native later (Qwen 3, Granite 7, DeepStack 4)ROI, fixed donor box (Qwen: median 6% of tokens)raw k∈{0,1}k \in\{0,1\}; at M0M_0, α=1\alpha=1 also RSS/L1 opposite-native kkD/I/C; S+Gphotograph (64)
Local transfer (II, Imain−IlaterI_{\mathrm{main}}-I_{\mathrm{later}})donors a/ba/b (M0M_0, α=1\alpha=1), clean, over the source-aa scene at M0/M1M_0/M_1, α∈{1,.08}\alpha\in\{1,.08\}main alone, or all native laterROI at the original or opposite half; features and mask move togetherraw replacementD/I/C; S+Gscene or template (64)
Whole source (DfixedD_{\mathrm{fixed}}, JJ)main and later each aa or bb (four assignments), clean; four layout pairs; α=1\alpha=1main and all later, jointlyallrawD/I/C; S+G; target query onlytemplate (64)
Synthetic dose (matched ΔR\Delta R)changed entry a→ba \to b; other entry fixed to aa or bb; M0/M0M_0/M_0, α=1\alpha=1main alone, or all native laterallraw k∈{0,1}k \in\{0,1\}; RSS/L1 opposite-native kkD/I/C; S+Gtemplate (32)
Groups (RGR_G, CC, GG, HH)common B: donor-aa ROI at every entry over the source-aa scene (M0/M1M_0/M_1, α∈{1,.08}\alpha\in\{1,.08\}); the selected group moves toward clean bbsingletons, pairs or triples, all laterROIraw k=1k=1; common-low RSS/L1D/I/C; S+Gtemplate (32)
Granite marginal (M21∣15,18M_{21|15,18})main and later base clean aa; donor clean bb (fixed-contrast templates)entry 21 added to fixed {15,18}\{15,18\}ROIk=1k=1D/I/C; S+Gtemplate (32)
Fixed-contrast conflict (Rlater(C)R_{\mathrm{later}}(C), Rlater(D)R_{\mathrm{later}}(D), IqualityI_{\mathrm{quality}})main: C or D (contrast .12) of aa; later: clean aa with ROI replaced by clean bb at k=1k=1all native laterROI; non-ROI later tokens stay clean aake=1k_e=1 (exact replacement)D/I/C; S+Gtemplate (32 per task); photograph (64)
Fixed-contrast recovery (AuvA_{uv}, GDG_D, IrecI_{\mathrm{rec}})one image aa, C or D separately at main and later; no bb; D: contrast .12 (synthetic, photos) or blur radius 2 (GQA, DocVQA)main and all later, as whole inputsallnoneD/I/C accuracy (S, G); free-form GQA exact, DocVQA ANLStemplate (32 per task); photograph (64); GQA image (256 per group); document (256)
Calibrated conflict (as fixed-contrast)as fixed-contrast; D is a calibrated level (desaturation, relative blur, lower resolution)all native later (DeepStack: 2 active groups)ROI (median 19–26% of later tokens); non-ROI later tokens stay clean aak=1k=1D/I/C; S+Gtemplate (32 per task); synthetic only
Calibrated recovery (as fixed-contrast)as fixed-contrast at each calibrated level; CC shared across levelsmain and all later, as whole inputsallnoneD/I/C accuracy (S, G); free-form GQA exact, DocVQA ANLStemplate (32 per task); GQA image (192 per group); document (192)
Removal and coding (accuracy, bytes; Appendix H)one image aa, no bb; each entry unchanged, zeros, calibration mean, or coded and decoded by NNCodecmain, all native later, or both; each entry separatelyallnone; one QP per entryaccuracy (G); free-form GQA exact, DocVQA ANLSGQA image (1,000; Granite 468 and 531); document (474)

Table 5. Intervention specification of each experiment family. a/ba/b: sources differing only in the edited object’s attribute; C/D: clean/degraded versions of aa. ROI: visual tokens overlapping the edited object’s box, one index set at every changed entry (DeepStack: one set per crop group); “all”: every visual position. Readouts D/I/C: direct, identity-label, cyclic-label; kinds S/G: candidate score and greedy generation. In both quality experiments the conflict aa endpoint is the same visual input as recovery cell CC (clean main) or DC (degraded main), and conflict cells never degrade later input.

Quality checks and sensitivity analyses

Unavailable responses and native capability

In the photograph panel, Qwen8 and Qwen32 have 37 and 11 invalid generation records; their reference paired contrasts retain 62 and 63 photographs, versus Granite’s 64. Some weak-background local color interactions retain only 34 and 53 complete templates, respectively. The full-strength stripe interaction retains 64 in all three models. These differences arise from the observations required by each contrast, not a change in the planned sample.

For Qwen8 and Qwen32, the entry decomposition contains all 78,336 planned observations in each answer measure. Qwen8 has 217 unparseable, EOS-terminated generations and no recorded truncation; its cell and paired-contrast counts range over 20–32 and 19–32 templates. Qwen32 has 319 truncated responses plus one unparseable EOS-terminated response, giving ranges 24–32 and 23–32. Score estimates and every raw/common-low reference in Table 10 retain 32 templates. The reference table’s completeness does not extend to the whole grid.

Table caption: At common-low dose, neither Qwen model keeps generation selectivity. Entry decomposition for color, direct answers, M0M0, α=1\alpha= 1. Each cell gives a mean above its descriptive normal 95% interval; every estimate uses 32 complete templates. All-later contains the three later entries and excludes main. The rightmost column within each endpoint is the directly paired Rmain−RallR_{\mathrm{main}} - R_{\mathrm{all}}. Rows labeled Low use common-low matching: the minimum native singleton dose at the shared source-a baseline. Under RSS (shaded rows), the small matched score contrast is positive for Qwen8 and negative for Qwen32.

DoseScore (n=32n = 32 per estimate)Generation (n=32n = 32 per estimate)
RmainR_{\mathrm{main}}RallR_{\mathrm{all}}Main − allRmainR_{\mathrm{main}}RallR_{\mathrm{all}}Main − all
Qwen8
Raw (k=1k = 1)27.286 [23.282, 31.290]0.269 [0.148, 0.389]27.017 [23.022, 31.013]0.563 [0.418, 0.707]0.000 [0.000, 0.000]0.563 [0.418, 0.707]
Low RSS0.217 [0.124, 0.310]0.102 [0.044, 0.159]0.115 [0.033, 0.197]0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Low L10.217 [0.124, 0.310]0.070 [−0.005, 0.146]0.147 [0.050, 0.244]0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Qwen32
Raw (k=1k = 1)38.314 [35.941, 40.688]0.667 [0.532, 0.802]37.647 [35.250, 40.044]0.813 [0.707, 0.918]0.000 [0.000, 0.000]0.813 [0.707, 0.918]
Low RSS0.067 [0.020, 0.113]0.115 [0.054, 0.177]−0.048 [−0.093, −0.004]0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Low L10.067 [0.020, 0.113]0.097 [0.043, 0.151]−0.030 [−0.071, 0.010]0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]

Table 10.

Nano full-source and DeepStack local outcomes have no missing, invalid, truncated, or nonfinite records (n=64n = 64). Super full-source and both Cosmos dose panels likewise retain all 64 and 32 templates, respectively. Super’s photograph panel has 37 invalid generations, no missing/truncated/unmatched records, and n=62n = 62–64 (reference n=63n = 63). Nano and Super local color have 852 and 1,032 invalid records among 29,184 each, without truncation. Ordinary effects use n=6n = 6–64 and 8–64; spatial contrasts use 0–64. At α=.08\alpha= .08 with direct answers, neither has a complete template for the paired entry interaction: Nano’s separate main/later interactions have n=1/6n = 1/6, and Super has neither. At full strength, color comparisons are complete; local stripe generation and all local scores retain 64 throughout.

For both Cosmos checkpoints, full-strength native color is correct and valid on all 128 queries in all formats. Native stripe correctness is 98/81/87 for Nano and 114/115/116 for Super under direct/identity/cyclic readouts, each out of 128 repeated queries from 64 templates. DeepStack’s dose panel has 34 invalid color and 32 invalid stripe generations, including 13 truncated stripe outputs under amplification; affected estimates use n=28n = 28–31. The Qwen8, Qwen32 and Granite dose panels have no invalid, truncated, or unmatched observations. Their maximum canonical dose discrepancy is 0.595% for Granite, 0.191% for Qwen8, and 0.080% for Qwen32. This checks the declared anchor, not every query-specific normalization.

Photograph exclusion and answer transitions

In photograph vg-2389325, the textual reference and the bounding box are inconsistent. A post hoc analysis excludes the whole photo in Qwen2/4/30, Nano, Super, and DeepStack. The exclusion preserves the main entry-ordering conclusions and leaves the original primary results intact. One Qwen2 direct-generation RR interval newly contains zero; two Qwen4 control intervals newly exclude zero. For Super, reference ΔR\Delta R changes from 0.746 [0.670, 0.822], n=63n = 63, to 0.750 [0.673, 0.827], $n = 62. All 24 entry comparisons remain positive in each measure, but two control-related intervals newly contain zero and one near-zero control-score mean changes sign. Component invariance is therefore not implied.

In the photograph reference condition (M0M0, α=1\alpha= 1, direct generation, raw dose), Table 6 uses each model’s common complete-photograph set. Main/later target-control means are (.653, .008)/(0,0)(0, 0) for Qwen8, (.738, .008)/(0,0)(0, 0) for Qwen32, (.188, .008)/(.266, 0) for Granite, and (.047, 0)/(.625, .031) for DeepStack. They reconstruct the paired contrasts before rounding; separately complete entry columns can use different sets.

ModelEntryTarget changesControl changesPhotos with target change
Qwen8Main81/1241/12455/62
Qwen8Later0/1240/1240/62
Qwen32Main93/1261/12659/63
Qwen32Later0/1260/1260/63
GraniteMain24/1281/12822/64
GraniteLater34/1280/12828/64
DeepStackMain6/1280/1286/64
DeepStackLater80/1284/12854/64

Table 6. Table 6. Main changes Qwen answers and later changes DeepStack answers. Granite’s answers change with both entries, and controls almost never change. Counts use each model’s common complete-photograph set. Query pairs and photographs are different denominators: each photograph supplies two target-query pairs and two control-query pairs. All observed changes are toward bb.

Qwen8/32 also show no later changes among all 128 target and 128 control pairs that are complete for the later-entry intervention alone. Under the reported matched RSS condition, later target changes instead number one and seven. For unavailable reference answers, we assign each binary endpoint 0 or 1, consistently across its repeated uses. Over the full 64 photos, this gives ΔR\Delta R extrema [0.640625, 0.664063] and [0.726563, 0.734375]. These are deterministic point-estimate bounds, not sampling intervals.

Control composition and room to move

Both queries are scored toward bb, so a control whose answer is already bb cannot move toward bb in generation. All 128 photograph edits run red to blue; 54 unchanged referents are red and 74 blue (34 red–blue, 20 blue–blue and 10 red–red photographs). Table 7 therefore restricts changes to queries that answer red with source aa. Controls with room still rarely change: at most 4 of 63, for DeepStack’s later entry, and each such change is a blue control misread as red at baseline. Across models, only one natively red control changes (Qwen8 main, 1 of 52). This removes the explanation that controls were all at the generation ceiling, but equal starting answers need not mean equal margins, so it does not exclude a shared bias. Candidate scores can move on both queries, and they show the same contrast in the reference cell. For Qwen8 and Qwen32 main, control effects are −0.049-0.049 [−0.280,0.182-0.280, 0.182] and 0.228 [−0.017,0.473-0.017, 0.473] against target effects of 18.204 and 21.631. For DeepStack main and later, they are 0.120 [0.058, 0.182] and 0.402 [0.239, 0.565] against 1.094 and 3.368. However, subtraction cancels a shared shift only if it moves both queries equally. Restricted to photographs whose control is red, ΔR\Delta R is 0.512 [0.364, 0.659], 0.558 [0.412, 0.705], −0.523-0.523 [−0.665,−0.380-0.665, -0.380] and −0.159-0.159 [−0.286,−0.033-0.286, -0.033] for Qwen8, Qwen32, DeepStack and Granite (edit-level complete cases); with blue controls it is 0.708, 0.806, −0.537-0.537 and −0.009-0.009, so the lower Qwen values with red controls come from the target component. When red targets and red controls start from nearly equal log odds, targets change in 40 of 52, 42 of 52 and 34 of 53 cases (Qwen8 main, Qwen32 main, DeepStack later) against 1 of 52, 0 of 52 and 0 of 53 controls. Photograph-level score effects are 19.370, 24.523 and 4.019 for targets against −0.192-0.192, 0.342 and 0.290 for controls. DeepStack’s direct generations are capitalized, whereas its candidate scores use lower-case words; we rescored 8 paired queries from 2 photographs using capitalized candidates. This raised the candidate probability mass from 0.000263 to 0.997632 and kept all 88 b−ab - a signs, all 16 selectivity signs and all 8 main-minus-later signs.

ModelEntryPhotosTarget, at riskTarget, allControl, at riskControl, all
Qwen8Main6281/9581/1241/541/124
Later0/540/1240/540/124
Qwen32Main6393/10793/1261/561/126
Later0/560/1260/560/126
GraniteMain6424/7724/1281/591/128
Later34/9234/1280/580/128
DeepStackMain646/656/1280/620/128
Later80/10780/1284/634/128

Table 7. Table 7. Controls that can still move toward blue rarely do. Answer changes toward blue in the photograph reference cell (raw strength, M0M0, full background, direct generation, common complete photographs). “At risk” counts queries that answer red at the aa endpoint of that entry’s intervention, where the other entry shows the original photograph, and so can still move toward blue (the counts therefore differ between entries); “all” repeats the full denominators of Table 6. No answer moves from blue to red. Post hoc counts from the paired endpoints.

The synthetic sets differ. In every template of the controlled panel and of both quality experiments, the two objects take different values and the edit gives the target the control’s value. All 256 control rows per set therefore have correct answer bb, and source bb always shows two objects sharing the queried value. A correctly answered control can then move only away from bb in generation, and generation control components cannot reveal a shared shift toward bb. Candidate scores can: in the calibrated color task, for example, DeepStack’s direct control effect is 0.418 against a target effect of 1.322 with clean main and 1.804 against 4.546 at the strongest desaturation. The local-transfer reversals are unaffected, because there the moved features land on the control and its answers do change (Appendix § 11). Controls whose value equals aa would require other materials.

Supporting figures and source-conflict results

The following figures cover the three models of the group analyses and show component, readout, and dose differences. At full strength, original layouts, and direct color generation, target-query DfixedD_{\mathrm{fixed}} is 0.914 [0.863, 0.966] for Qwen30 and −0.789-0.789 [−0.850,−0.728-0.850, -0.728] for DeepStack (n=64n = 64). Nano’s fixed-aa generation ΔR\Delta R is 1.000 [1.000, 1.000] for color and 0.586 [0.473, 0.698] for stripes. The latter combines Rmain=0.531R_{\mathrm{main}} = 0.531 [0.408, 0.654] and Rlater=−0.055R_{\mathrm{later}} = -0.055 [−0.162,0.053-0.162, 0.053], whereas its stripe DfixedD_{\mathrm{fixed}} is 0.797 [0.736, 0.858]. Super’s color ΔR\Delta R is also 1.000 [1.000, 1.000]; its stripe ΔR=0.352\Delta R = 0.352 [0.260, 0.443] differs from Dfixed=0.414D_{\mathrm{fixed}} = 0.414 [0.323, 0.506] because control effects contribute to selectivity. These are 64-template estimates; saturated color intervals do not imply certainty outside the sample. With mixed layouts, 21 of DeepStack’s 24 role-selectivity intervals include zero despite later effects on both queries. Fixed-source influence, role exchange, and relative selectivity remain distinct (Appendix § 9.§ 4).

Additional native-entry comparisons

Composition and direction alignment

Figure 8 shows the common-photo target/control components and the joint, singleton-sum, and aligned residual estimates. Table 8 reports Qwen32’s raw composition residual for stripes, and Table 9 reports its paired layout difference after direction alignment. The residual reverses sign with layout; all aligned-difference intervals include zero, without establishing equivalence. For α=.08/1\alpha= .08/1, the aligned joint-generation differences are 0.141 [0.022, 0.259] and 0.250 [0.104, 0.396]; singleton-sum differences are 0.203 [0.039, 0.367] and 0.156 [0.007, 0.305]. The residual subtracts these changes. Later entry 2 instead has differences −0.016/0-0.016/0, so neither invariant components nor a changed neural combination rule follows from the residual alone. Qwen8’s corresponding full-strength stripe generation intervals include zero.

Layoutα\alphaScore CC [95% interval]Generation CC [95% interval]nn
Original.088.655 [6.608, 10.701]0.313 [0.198, 0.427]32
Original15.508 [3.476, 7.541]0.172 [0.050, 0.293]32
Exchanged.08−8.351 [−10.702, −6.000]−0.250 [−0.382, −0.118]32
Exchanged1−4.282 [−6.060, −2.504]−0.266 [−0.411, −0.120]32

Table 8. Qwen32’s stripe composition residual reverses sign with layout. Raw composition residual CC with direct answers. Each value is a paired template-level estimate over 32 complete templates, with a descriptive normal 95% interval.

Readoutα\alpha−C(M1)−C(M0)-C(\mathrm{M1}) - C(\mathrm{M0})95% interval
Generation.08−0.063[−0.220, 0.095]
Generation10.094[−0.095, 0.283]
Score.08−0.304[−2.496, 1.888]
Score1−1.226[−3.311, 0.858]

Table 9. After direction alignment, no layout difference in Qwen32’s residual is resolved. Stripe composition residual with the direction of spatial correspondence aligned. Each contrast pairs the two layouts within the same 32 templates.

Three-panel plots of directed answer change and aligned effects

Figure 8. Strong object-directed changes and nonadditive joint effects. (a) Natural photos, M0M0, α=1\alpha= 1, direct generation, k=1k = 1: target and control components use the same complete photos within each model (62/63/64/64 for Qwen8/Qwen32/Granite/DeepStack). Each later point changes the native later-entry collection. The strong Qwen-main and DeepStack-later effects mostly come from the target query. (b,c) Qwen32 stripes, direct readout, raw dose, 32 templates: joint later effect, sum of the three singleton effects, and their directly paired residual CC. Rows give layout and background strength. For M1M1 (marked *), all plotted effects and interval endpoints are multiplied by −1-1 to align spatial direction; this does not assert layout equivalence. Bars are descriptive normal 95% intervals. Generation is a directed answer change, not task accuracy.

Granite subsets and individual entries

For spatial (15, 18, 21) versus layer (3, 6, 9) subsets at M0M0, α=1\alpha= 1, and raw dose, the score interaction HH is 0.280 [0.215, 0.344], 0.120 [0.033, 0.208], and 0.192 [0.096, 0.287] under direct, identity, and cyclic readouts. Generation interactions are −0.016-0.016 [−0.085, 0.054], 0.125 [−0.007, 0.257], and 0.016 [−0.067, 0.098], all n=32n = 32. At α=.08\alpha= .08, score intervals remain positive and generation intervals contain zero. Identity-label stripe GG is 0.156 [0.045, 0.268] at full strength, versus color GG = 0.031 [−0.030, 0.093]; at .08, stripe GG = 0.156 [0.036, 0.276], and cyclic stripe GG = 0.063 [0.004, 0.121]. Group differences within stripes therefore need not yield resolved attribute interactions.

Native direct-score stripe correctness is 64/128 and 65/128 for the two sources, versus generation’s 72/128 and 68/128. Label formats have higher native correctness and retain the score interaction. We divide each attribute’s group difference by its mean all-later score effect. This gives direct full-strength H=0.315H = 0.315, with paired bootstrap interval [0.266, 0.367] (10,000 template resamples with a fixed seed, denominators recomputed). This split preserves the direction under one output scaling; it does not remove capability, dose, or depth confounds. The separate conditional marginal in Appendix § 14 holds entries 15 and 18 fixed while adding 21, rather than combining endpoints measured in separate experiments.

At M0M0, α=.08\alpha= .08, Qwen32’s direct-score differences, later entry 0 minus later entry 1, are −0.309-0.309 [−0.445, −0.172] for color and −0.686-0.686 [−1.340,−0.033][-1.340, -0.033] for stripes at raw dose. Common-low matching gives −0.014-0.014 [−0.105,0.077][-0.105, 0.077] and −0.068-0.068 [−0.232,0.096][-0.232, 0.096], all n=32n = 32. Qwen8’s corresponding matched differences are −0.226-0.226 [−0.320,−0.132][-0.320, -0.132] and −0.422-0.422 [−0.682,−0.162][-0.682, -0.162]; its generation intervals contain zero (n=25/32n = 25/32). RSS/L1 singleton interventions are physically identical, so agreement is not independent confirmation. Other layouts can change the ordering.

Figure 4. Across all three answer formats, stripe spatial interactions favor main in Qwen8 and Qwen32, and favor later in Granite. Generation interaction IgI_g of Eq. (6) at full background strength under direct answers and two label formats, with 64 complete templates. The main and later interactions and their directly paired entry difference are separate estimates.

Granite also varies by condition. At M0M0, α=.08\alpha= .08, raw direct-score stripe selectivity is 0.0760.076 [0.043,0.109][0.043, 0.109] for main and 0.5950.595 [0.550,0.640][0.550, 0.640] for entry 21. Under common-low RSS, the color difference, main minus entry 15, is −0.0757-0.0757 [−0.0912,−0.0603][-0.0912, -0.0603]. With exchanged layout and full-strength color, main score selectivity is 0.8650.865 [0.670,1.060][0.670, 1.060] but generation selectivity is −0.250-0.250 [−0.348,−0.152][-0.348, -0.152]. These conditional effects do not assign fixed semantic roles to layers.

Figure 6. Candidate scores give the same Qwen ordering, whereas Granite’s score contrast is negative under RSS matching and positive under L1. Score counterpart of Figure 5, with model-specific score scales: target, control and relative effects, then the paired entry contrast. Granite’s L1 generation contrast remains uncertain. The same photographs underlie the repeated dose settings.

Figure 5. On photographs, Qwen8 and Qwen32 favor main at every dose, while Granite’s contrast is near zero or negative. Generation components and dose controls for Qwen8, Qwen32 and Granite: target and control effects, relative selectivity RR and the directly paired entry difference ΔR\Delta R. Counts are complete photographs, not repeated intervention records. Table 2 covers all nine checkpoints.

Dose and low-strength boundaries

Photograph directions across formats and doses. For the photographs at M0M0 and full background, each checkpoint has 15 entry contrasts per measure: three answer formats crossed with raw strength and the RSS and L1 targets at main- and later-native doses. Qwen8, Qwen32 and Cosmos3-Super have positive ΔR\Delta R intervals in all 15 cells for both scores and generation, as do the nine further raw cells with layout exchange or weak background. DeepStack’s intervals are negative in all 15 cells and all nine further cells for both measures; its generation contrasts range from −0.578-0.578 to −0.281-0.281, the weakest being cyclic labels at the main-native L1 target, −0.281-0.281 [−0.360-0.360, −0.203-0.203]. Granite has 3 positive, 9 negative and 3 zero-containing score cells and 1, 9 and 5 generation cells; every positive cell uses L1 matching, and its identity-label raw generation contrast is −0.219-0.219 [−0.300-0.300, −0.137-0.137].

A small contrast can compare two selective entries. In Qwen32 stripes, fixed α\alpha, and direct generation, main-native RSS gives Rmain=0.531R_{\mathrm{main}} = 0.531 [0.400, 0.663] and Rlater=0.578R_{\mathrm{later}} = 0.578 [0.425, 0.731], but paired ΔR=−0.047\Delta R = -0.047 [−0.181,0.088-0.181, 0.088] (n=32n = 32; see Figure 9a). Conversely, for Nano stripes with fixed α\alpha and the later-native RSS target, generation selectivity is zero in both groups but score $\Delta R = 0.679 [0.415, 0.943].

Figure 9. Dose matching can tie two selective entries or reverse their order in one cell. These full-source conditions use direct generation, M0/M0M0/M0, both source strengths α=1\alpha= 1, and 32 complete shared templates per point. (a) With Qwen32 stripes and fixed source aa, main-native RSS leaves both RmainR_{\mathrm{main}} and RlaterR_{\mathrm{later}} positive while their directly paired ΔR\Delta R interval contains zero; main-native L1 changes the ordering. (b) With Cosmos3-Super stripes at the later-native target, fixed source aa yields a negative ΔR\Delta R under RSS and a positive one under L1; fixed source bb yields positive differences under both measures. The negative Super cell is one condition in a larger grid and is not treated as an independent replicated reversal. Generation is the net change in the frequency of the edited object’s bb answer. Bars are descriptive normal 95% intervals; ΔR\Delta R intervals are directly paired.

Figure 7. Later-native matching removes the Qwen main advantage on synthetic color but not on photographs. All 45 three-model generation reference points: raw dose and both native targets under RSS/L1 for natural photographs, synthetic color, and synthetic stripes. Synthetic comparisons use 32 templates, M0/M0, and fixed source aa; natural comparisons use their actual complete-photo counts. Material, intervention support, and sample set differ across panels, so their amplitude differences do not isolate a causal effect of material type.

Super supplies an isolated negative cell among its matched stripe conditions (Figure 9b). With fixed aa and the later-native target, RSS score/generation differences are −1.601-1.601 [−2.357-2.357, −0.845-0.845] and −0.156-0.156 [−0.276-0.276, −0.036-0.036]; L1 gives 2.3682.368 [1.7671.767, 2.9682.968] and 0.2030.203 [0.1060.106, 0.3000.300]. Fixing bb under RSS also gives positive generation 0.2810.281 [0.1360.136, 0.4270.427], all n=32n = 32. Across 24 matched conditions, score intervals are positive/zero-containing/negative in 18/5/1 cells, and generation in 17/6/1. The negative generation cell has four positive, 14 tied, and 14 negative template contrasts. Selection from this exploratory grid is not accounted for by its interval.

Qwen32 stripes favor main under L1 at both targets.

Granite fixed-aa color generation moves from −0.406-0.406 [−0.518-0.518, −0.295-0.295] at raw dose to 0.2970.297 [0.2000.200, 0.3940.394] under main-native L1, while fixed-bb L1 remains later-favoring. Qwen30 identity-label color at the later-native target has score ΔR=−0.17\Delta R = -0.17 [−0.75-0.75, 0.410.41] under RSS and 6.026.02 [4.814.81, 7.247.24] under L1. DeepStack color remains later-favoring across all 24 matched settings in both measures, whereas matched direct stripe intervals contain zero. Scalar matching therefore constrains magnitude explanations without establishing a context-free ordering.

Table 10 retains raw/common-low color comparisons. Both Qwen checkpoints have zero net generation selectivity in both groups at common-low strength. Even so, their score differences remain distinct: Qwen8 stays positive; Qwen32 gives −0.048-0.048 [−0.093-0.093, −0.004-0.004] under RSS and an L1 interval containing zero. This small score-only reversal is an exploratory boundary. These comparisons use a different baseline and target from photograph dose controls.

Quality-conditioned influence and recovery

Design, materials, and interpretation

The conflict comparison changes main quality while fixing later’s clean-aa baseline, region mask, source difference, and multiplier. Target and control receive the same intervention; Iquality=Rlater(D)−Rlater(C)I_{\mathrm{quality}} = R_{\mathrm{later}}(D) - R_{\mathrm{later}}(C) compares their relative effect across main states. Recovery instead supplies clean/degraded versions of one image without conflicting attribute sources. In CC/CD/DC/DDCC/CD/DC/DD, the first letter denotes main quality. Native performance is ACCA_{CC}, later recovery is GD=ADC−ADDG_D = A_{DC} - A_{DD}, and the interaction is Irec=GD−(ACC−ACD)I_{\mathrm{rec}} = G_D - (A_{CC} - A_{CD}). A positive gain need not yield a positive interaction.

Synthetic materials use a fixed seed and 32 shared templates for color, stripe orientation, within-shape relative position, and short text. Two single-object edits and two queries per edit give 128 queries per task, 512 overall. Position marks lie within one shape; this task differs from exchanging two objects. The 64 reused photographs give 256 queries under new quality conditions, not an independent material replication. Both sets use contrast 0.12. Image dimensions are retained; identical degraded pixels recur across the four cells, with processed grids and visual hashes recorded.

In all 512 synthetic queries, the clean-aa correct option follows object side (left: first; right: second). Reversing numeric labels preserves predictability. Edited-bb target answers flip while controls remain unchanged, so the same rule does not hold across both sources. Native success therefore need not demonstrate attribute recognition independent of position. Within-query pairing identifies effects on these stimuli but does not exclude interactions between degraded evidence and a position–answer shortcut; recovery cannot isolate which visual information returns. An always-one-option baseline scores 0.5 with zero standard error, but matching that endpoint does not identify a model’s strategy. The calibrated experiment in Appendix § 15 balances all four side/option combinations on separate templates.

Table 11 preserves all four cells for color. Qwen’s direct and identity conditions remain saturated, while cyclic labels expose recovery. Under cyclic labels, Granite’s later features also help with clean main; DeepStack’s positive cyclic interaction partly reflects a negative clean-main benefit. The cells distinguish these cases from recovery that comes solely from better degraded-main performance.

Table caption: Qwen’s color accuracy stays at ceiling except under cyclic labels. Synthetic-color generation accuracy (%) at contrast 0.12. C and D denote clean and degraded sources, with main first and later second. All 32 templates contribute in every format; invalid answers count as incorrect. Paired contrasts and intervals for every task are in fixed_contrast_recovery.csv.

ModelFormatCCCDDCDD
Qwen8Direct100.0100.0100.0100.0
Identity100.0100.0100.0100.0
Cyclic100.0100.059.448.4
Qwen32Direct100.0100.0100.0100.0
Identity100.0100.0100.0100.0
Cyclic100.0100.0100.070.3
GraniteDirect100.098.4100.093.8
Identity100.0100.0100.0100.0
Cyclic92.271.992.250.0
DeepStackDirect100.0100.0100.084.4
Identity100.0100.0100.092.2
Cyclic82.896.995.384.4

Table 11.

Under cyclic labels, Granite’s color gain is GD=0.422G_D = 0.422 [0.355, 0.488], with Irec=0.219I_{\mathrm{rec}} = 0.219 [0.107, 0.330]; DeepStack’s direct-color gain is 0.156 [0.071, 0.241]. The cross-format comparison retains all planned synthetic-color queries from the same 32 templates and counts invalid outputs as incorrect, without selecting complete cases per readout. For Granite, the cyclic-minus-direct difference in GDG_D is 0.359 [0.277, 0.442], whereas the corresponding difference in IrecI_{\mathrm{rec}} is 0.172 [0.063, 0.280] (paired estimates in fixed_contrast_format_pairs.csv). Both are estimated from within-template differences. The first measures the change in degraded-main benefit; the second subtracts the change in clean-main benefit. Readout dependence therefore concerns the four-cell response pattern, not only whether one recovery interval excludes zero.

Execution, readouts, and pairing

All four models complete synthetic/photo quality and recovery; Granite additionally completes the conditional marginal. Experiments and benchmarks use separately validated batched implementations. Inference uses A100 80GB GPUs and BF16: Qwen/Granite/DeepStack use PyTorch 2.1.1/2.10.2/1.2 and Transformers 5.8.0/5.4.0/4.39.3, respectively. Conditions are batched when text and geometry match. Two jobs shared one GPU only after a check: a completed unit re-executed alongside a running job reproduced all 768 of its observations bitwise, on each of the two machines used. The final runs comprise 176,512 observations. Scalar reuse requires identical complete visual tensors; residual states, attention states, and key/value caches are not shared across main conditions.

Batched and unbatched checks on selected units agree on all 384 DeepStack synthetic decisions and token strings. Granite and Qwen32 agree on all 384 decisions, with exact tokens on 379/380; Granite’s five differences append words after the same labels. Qwen8 on photographs agrees on 95/96 decisions and token strings; one near-tie margin changes from −0.0234 to 0.0615. A separate synthetic check agrees on decisions and 370/384 token strings. Maximum candidate log-probability differences are 0, 0.307, 0.526, and 0.707 for DeepStack, Granite, Qwen32, and Qwen8 on photographs. These document BF16 batch-shape sensitivity, not equality across all runs. No paired DocVQA batching check was performed; unbatched runs and answer-format checks remain separate.

Direct, identity, and cyclic readouts are retained. Native/recovery accuracy treats invalid answers as incorrect over all planned queries, with valid-only sensitivity estimates. Conflict and marginal effects require valid paired endpoints and report actual counts: Granite direct-text marginal has 25 complete templates, versus 32 for its other displayed generation marginals. Native and recovery cells were not planned for the Granite conditional-marginal runs and are not available. Entries 15 and 18 keep identical increments; only entry 21 changes from source-aa baseline to its k=1k = 1 increment. The native accuracy shown with the marginal (Table 13) comes from the separate quality runs, not from a matched marginal endpoint.

TaskReadoutnnNativeMarginal
ColorDirect321.0000.000 [0.000, 0.000]
Identity321.0000.000 [0.000, 0.000]
Cyclic320.9220.156 [0.071, 0.241]
Relative positionDirect320.9220.094 [0.022, 0.165]
Identity320.8280.000 [−0.046, 0.046]
Cyclic320.9840.000 [0.000, 0.000]
StripesDirect320.5160.000 [0.000, 0.000]
Identity321.0000.266 [0.154, 0.378]
Cyclic320.6560.031 [−0.047, 0.110]
TextDirect250.8910.000 [0.000, 0.000]
Identity321.0000.000 [0.000, 0.000]
Cyclic320.6090.016 [−0.016, 0.047]

Table 13. Entry 21 adds selectivity in three of the twelve task and readout cells. Granite entry-21 marginal given fixed entries 15 and 18: paired target-minus-control generation effect, not a share of a joint effect. Native accuracy comes from the separate fixed-contrast condition. Units and tt intervals follow that panel; score marginals and components are in fixed_contrast_granite_marginal.csv.

Readout differences subtract formats within shared templates before estimating intervals. Primary generation comparisons for synthetic color use all planned queries within each template. Elsewhere, shared templates can contain different valid query sets across formats; those comparisons are flagged and are not the main readout evidence. All cell outputs, including invalid and truncated responses, remain in the observations.

Benchmark subsets and scoring

GQA 1.2 validation-balanced sampling uses a fixed seed and at most two questions per image: 465 attribute questions on 256 images and 472 spatial questions on 256 images, with eight shared images. Before sampling, we remove 2,663 candidate questions on images used in the other experiments of this paper. DocVQA 1.0 official validation sampling uses the same seed: 495 questions from 256 original documents and 296 pages, clustered by document. These are validation subsets, not full test-set evaluations.

Pillow Gaussian blur (radius 2.0) is applied before native processing without resizing the source. Qwen uses 4,096–262,144 pixels; Granite uses native 384 × 384 tiling; DeepStack uses CLIP 336 × 336 processing and spatial crops. Generation is greedy with 64 new tokens and sequence limit 4,096. All four cells share questions and references; all 12 panels retain 256 planned clusters without missing observations (Table 14).

TaskModelCCCDDCDDGDG_DIrecI_{\mathrm{rec}}
DocVQAQwen80.8000.7940.7800.785−0.004 [−0.018, 0.009]−0.010 [−0.031, 0.010]
Qwen320.7940.7960.7860.800−0.014 [−0.030, 0.002]−0.012 [−0.032, 0.007]
Granite0.8470.8400.8500.8390.011 [−0.003, 0.024]0.004 [−0.011, 0.019]
DeepStack0.3740.3570.3630.3430.021 [−0.001, 0.042]0.003 [−0.014, 0.021]
GQA attr.Qwen80.7050.7090.6700.6520.018 [−0.004, 0.040]0.021 [−0.010, 0.053]
Qwen320.7190.7130.7010.6890.012 [−0.014, 0.038]0.006 [−0.026, 0.038]
Granite0.0000.0040.0000.014−0.014 [−0.024, −0.004]−0.010 [−0.018, −0.001]
DeepStack0.0000.0000.0000.0000.000 [0.000, 0.000]0.000 [0.000, 0.000]
GQA spatialQwen80.6040.5920.5720.5530.020 [−0.001, 0.040]0.008 [−0.020, 0.035]
Qwen320.5760.5720.5390.564−0.025 [−0.053, 0.002]−0.029 [−0.062, 0.003]
Granite0.0000.0020.0000.002−0.002 [−0.006, 0.002]0.000 [0.000, 0.000]
DeepStack0.0000.0000.0000.0000.000 [0.000, 0.000]0.000 [0.000, 0.000]

Table 14. Blur at radius 2 costs little GQA or DocVQA accuracy, and no positive later recovery is resolved.

The GQA instruction (a single lowercase word or short phrase) was selected once for all models, after checking answer formats on clean inputs and before any recovery comparison. Strict equality remains primary. Named sensitivities lowercase and strip whitespace, then additionally remove terminal periods/exclamation/question marks and surrounding quotes. Granite attribute recovery is 0.0859 [0.0479, 0.1240] under the first rule but zero [−0.0277, 0.0277] under the second; spatial recovery changes from 0.1016 [0.0653, 0.1378] to 0.0410 [0.0141, 0.0679]. Punctuation-aware interactions contain zero. Zero clean accuracy under strict matching therefore cannot establish absent visual competence in Granite and DeepStack; DeepStack’s lowercase-and-strip native accuracies are 0.777/0.705 for attributes/spatial questions.

DocVQA uses organizer-framework ANLS: the best normalized Levenshtein similarity across references after low-casing/stripping. Values below 0.5 are set to zero, but exactly 0.5 is retained.1 The strict paper-cutoff sensitivity excludes that boundary; lowercase-and-strip exact matching is separate. ANLS is not binary accuracy. For DeepStack/Granite/Qwen8/Qwen32, joint losses ACC−ADDA_{CC} - A_{DD} are 0.0308 / 0.0079 / 0.0156 / − 0.0059, and later recoveries are 0.0205/0.0106/− 0.0045/− 0.0139, all recovery intervals containing zero. Small induced losses restrict conclusions about recovery from substantial damage; Appendix § 15 calibrates larger losses on separate units.

Complete estimates

The files fixed_contrast_recovery.csv, fixed_contrast_conflict.csv, fixed_contrast_granite_marginal.csv, fixed_contrast_format_pairs.csv, and fixed_contrast_benchmarks.csv (Appendix § 19) retain all task/readout conditions, target/control components, four cells, recovery interactions, conditional marginals, and named benchmark sensitivities, with original clustering and estimate-specific denominators. Accuracy policies distinguish invalid-as-zero, scoring returned strings, and complete valid four-cell sets where available. Intervals follow Appendix § 9.§ 3; none of these results is presented as confirmatory.

Conflict quality interactions

Table 12 gives the quality interactions on synthetic color for scores and generation, and on photographs for scores. For generation on photographs with direct answers, IqualityI_{\mathrm{quality}} is 0.062 [0.010, 0.115] for Qwen8, 0.008 [−0.027, 0.043] for Qwen32, 0.023 [−0.045, 0.092] for Granite, and −0.039 [−0.080, 0.001] for DeepStack. Only Qwen8’s interval lies entirely above zero.

ModelSynthetic colorPhotographs
Score, directScore, identityScore, cyclicGen., directScore, direct
Qwen81.715 [1.479, 1.951]1.047 [0.801, 1.293]1.879 [1.385, 2.373]0.000 [0.000, 0.000]0.182 [0.026, 0.339]
Qwen322.834 [2.381, 3.288]2.521 [1.982, 3.060]4.024 [2.869, 5.178]0.000 [0.000, 0.000]0.489 [0.232, 0.746]
Granite1.103 [0.897, 1.309]1.153 [0.942, 1.364]0.465 [0.311, 0.619]0.234 [0.132, 0.337]−0.080 [−0.148, −0.013]
DeepStack0.680 [0.558, 0.803]1.369 [1.228, 1.510]1.014 [0.786, 1.243]0.188 [0.060, 0.315]0.055 [−0.015, 0.125]

Table 12. Degrading main raises later score selectivity on synthetic color in all four models. Quality interaction Iquality=Rlater(D)−Rlater(C)I_{\mathrm{quality}} = R_{\mathrm{later}}(D) - R_{\mathrm{later}}(C) in the fixed-contrast experiment: the change in later target-minus-control selectivity when main is degraded to contrast 0.12 while the later features stay fixed. Score values are candidate log odds; generation values are directed answer-frequency changes. Synthetic color uses 32 templates and photographs 64, with complete units only. Intervals are descriptive, unadjusted 95% tt intervals.

Granite conditional marginal

Table 13 gives the generation effect of adding Granite’s entry 21 while entries 15 and 18 stay fixed.

Standard-task cells

Table 14 gives the four cells for GQA and DocVQA under blur, with later recovery GDG_D and its interaction IrecI_{\mathrm{rec}}.

Recovery along calibrated degradation curves

Design and units

The calibrated experiment asks how later recovery changes with each model’s measured loss. It keeps the four models, the four synthetic tasks, and the recovery and conflict definitions of Appendix § 14. A single contrast level is replaced by three calibrated degradation levels per task, GQA and DocVQA receive the same treatment, and evaluation uses units that played no part in calibration.

Calibration units only choose the levels. Synthetic calibration uses 16 templates (256 queries). GQA attribute and spatial calibration reuse 48 images each from the fixed-contrast experiment (87 and 85 questions), and DocVQA calibration reuses 48 of its documents (56 pages, 94 questions); this is named reuse, not replication. Evaluation uses held-out units throughout. The 32 synthetic templates per task (128 queries per task) balance object side against the answer option: each of the four combinations occurs in 32 rows per task. Each template has two edits with a target and a control query each; both edits share one clean image, so recovery has 64 distinct template–object items per task. Each item is counted twice, which leaves template means unchanged. This balance removes the clean-reference association described in Appendix § 14. GQA 1.2 validation-balanced sampling (at most two questions per image) gives 192 images per question group, with 349 attribute and 340 spatial questions. Single-Document DocVQA 1.0 validation sampling gives 192 documents, 216 pages, and 373 questions, at most two per document. All sampling uses fixed seeds. Benchmark selection excludes all benchmark units of the fixed-contrast experiment and the photograph panel, so calibration and evaluation units are disjoint. These are selected validation subsets, not leaderboard evaluations.

Degradation families and level selection

Each task uses one degradation family scaled to its stimulus. Color keeps a fraction of chroma with luminance unchanged. Stripes and relative position use Gaussian blur, with σ\sigma relative to the stripe period or to the dot separation. Text, GQA, and DocVQA reduce resolution, expressed as equivalent font size for text and as long side for images and pages. Each level is one image file derived from the clean image with Pillow 12.2.0; degraded files keep the clean pixel size, and all four models receive the same file. Calibration runs only the clean cell and the jointly degraded cell of each candidate level, under every readout. Recovery and conflict cells were never run on calibration units, so no recovery or conflict effect could inform the choice.

For each model, retained ability at a level is r=(ADD−chance)/(ACC−chance)r=(A_{DD}-\text{chance})/(A_{CC}-\text{chance}), averaged over native-capable readouts. A readout is native-capable when ACCA_{CC} exceeds chance by at least 0.2(1−chance)0.2(1-\text{chance}), and models without such a readout are excluded from that task’s median. The shared grid consists of three ordered levels. Their cross-model median rr values jointly minimize the summed distance from 0.75, 0.5, and 0.25, with ties toward milder levels. GQA calibration uses case-and-punctuation normalization; DocVQA calibration uses organizer ANLS. The GQA metric was fixed before any run, because answer formatting had made strict GQA accuracy zero for Granite and DeepStack. Table 15 lists the selected levels.

TaskLevelMedian rrQ8Q32Gr.DS
Color0.20.8650.8120.9170.7260.998
0.10.5830.5620.7500.4820.603
0.050.2860.2810.3750.0730.290
Stripes0.370.8670.8671.0000.214NN
0.40.4750.4750.7710.143NN
0.430.0210.0210.0000.143NN
Rel. position0.10.7270.7270.9520.254NN
0.20.4550.4550.7530.031NN
0.30.1820.1820.4880.062NN
Text40.8030.8030.9380.290NN
30.4270.4270.5620.061NN
2.50.2290.3190.2290.061NN
GQA attr.28 px0.7430.7420.7430.6580.870
10 px0.6220.5450.4860.6990.714
5 px0.5620.5000.4430.6300.623
GQA spatial20 px0.7160.7430.8940.6890.648
14 px0.6810.6860.7420.6220.676
10 px0.6520.6290.7580.6220.676
DocVQA360 px0.8010.8950.9010.7060.628
256 px0.4770.6130.5820.3570.372
180 px0.2570.2620.3070.1550.251

Table 15. Shared degradation levels chosen on calibration units from clean and jointly degraded cells only.

The initial candidate levels left text and most stripe levels at ceiling and barely affected GQA. Heavier or intermediate levels were therefore derived from the same clean images: six text levels (font sizes 4 to 1), four intermediate stripe levels, and six GQA levels (long sides 28 to 5 pixels). The decision to add them was based on calibration accuracy alone. The resulting grids have known limits. The most severe stripe level is at the floor (median r=0.021r=0.021), and the mildest relative-position level falls below its target (median r=0.727r=0.727). GQA keeps a median rr of at least 0.562 even at a 5-pixel long side, so its grid spans moderate loss only. Whether language priors or coarse visual cues sustain this accuracy was not tested.

A secondary, model-specific design also evaluates, for each model and task, the calibration level whose rr is nearest 0.5 (calibrated_capability_levels.csv). Most coincide with grid levels; four were run outside the grid. These rows form a separate estimand and are not pooled with the shared grid.

Execution, geometry, and scoring

Inference ran in BF16 on A100 80GB GPUs, with the checkpoints of Table 4 and the software versions of Appendix § 14. Before analysis, every run passed four technical checks: native and self-patched scores differ by 0, self-patched generations are identical, processed geometry is identical across all sources of a query, and every cell receives its intended visual source. All 609 observations computed on both of the two machines used were bitwise identical. Generation allows 32 new tokens for synthetic tasks and 64 for benchmarks. Synthetic tasks, so geometry alone does not account for differences in loss. Loss is therefore measured per model rather than inferred from the level.

Hub identifierRevision prefix
Qwen/Qwen3-VL-2B-Instruct89644892e4d8
Qwen/Qwen3-VL-4B-Instructebb281ec70b0
Qwen/Qwen3-VL-8B-Instruct0c351dd01ed8
Qwen/Qwen3-VL-32B-Instruct0cfaf48183f5
Qwen/Qwen3-VL-30B-A3B-Instruct9c4b90e1e4ba
nvidia/Cosmos3-Nanoe59a53c25979
nvidia/Cosmos3-Superf543c56225b2
ibm-granite/granite-vision-4.1-4b37d591f06319
menglc/deepstack-l-vicuna-7b76f7d20cd835

Table 4. Hub identifiers and revision prefixes for the nine checkpoints.

The same file does not give every model the same input. Qwen8/32 process DocVQA pages at a median long side of 576 pixels and Granite at 1152 pixels, so a given level removes a larger share of Granite’s processed resolution. DeepStack’s main entry receives one 336 × 336 global view, whereas its later groups receive 336-pixel crops from a 672-pixel canvas (up to 1008 pixels on DocVQA). Its 360-pixel DocVQA level therefore barely changes main (ADC=ACC=0.334A_{DC} = A_{CC} = 0.334, ACD=ADD=0.277A_{CD} = A_{DD} = 0.277) and is not interpreted, and its lower levels remove more of the later input than of the main input. Apart from this main view, no evaluated level leaves a processed input unchanged, and processed geometry is identical across the clean and degraded sources of each query. Although Qwen8 and Qwen32 share one processed geometry, they lose different amounts on the synthetic tasks, so geometry alone does not account for differences in loss. Loss is therefore measured per model rather than inferred from the level.

Synthetic recovery cells count invalid answers as incorrect over all 128 planned queries, while conflict endpoints exclude invalid generations. Conflict cells start from the clean-aa later input. At k=1k = 1 on every native later entry, they install source bb only on the edited object’s later tokens (a median 19–26% of later tokens across models); their source-aa endpoint is the same input as recovery cell CC or DC. Recovery cells change all later tokens. Synthetic Λ=ACC−ADD\Lambda= A_{CC} - A_{DD} is a difference of two means and has no interval; GDG_D, IrecI_{\mathrm{rec}}, and conflict effects use descriptive Student-tt intervals over templates. Benchmarks retain all 192 clusters in every cell. GQA is read with case-and-punctuation normalization, the calibration metric fixed before the runs; the official strict matching, which equals it for Qwen8/32, is kept as a format-sensitive variant. DocVQA uses organizer-framework ANLS. Both score invalid outputs as zero. Named sensitivities are lowercase-and-strip GQA matching, strict-cutoff ANLS and lowercase exact matching for DocVQA, and valid-four-cell and returned-string policies. Benchmark Λ\Lambda, GDG_D, and IrecI_{\mathrm{rec}} are paired estimates with normal intervals clustered by image or document. Truncated generations are the only invalid benchmark outputs; they are most frequent for DeepStack on DocVQA, with at most six in one cell.

Granite and DeepStack capitalize nearly all GQA answers (“Right” for the reference “right”): 349 of 349 and 348 of 349 clean attribute answers, and all 340 clean spatial answers for each model, against none for Qwen8/32. Strict matching therefore scores their clean answers zero, and Granite’s negative strict-match loss reflects a change of answer format under degradation. Case-and-punctuation normalization, which equals strict matching for Qwen8/32, is the comparable metric. At Granite’s 5-pixel attribute level, lowercase-and-strip matching shows extra loss only in cells with degraded later input. The extra loss disappears once trailing punctuation is removed; this format effect is not counted as loss.

Reading the curves

On GQA attribute and spatial questions, the four models and three shared levels give 24 estimates per contrast under the preselected case-and-punctuation metric with invalid outputs scored as zero. All GDG_D and IrecI_{\mathrm{rec}} means are positive; 23 recovery intervals and 21 interaction intervals are above zero. At the strongest shared levels (5-pixel long side for attributes, 10 for spatial questions), all eight intervals for each contrast are above zero. Conditions share images and do not count as independent replications. With joint loss Λ=ACC−ADD\Lambda= A_{CC} - A_{DD} and single-entry losses UM=ACC−ADCU_M = A_{CC} - A_{DC} and UL=ACC−ACDU_L = A_{CC} - A_{CD}, GD=Λ−UMG_D = \Lambda- U_M and Irec=Λ−UM−ULI_{\mathrm{rec}} = \Lambda- U_M - U_L. No readout is primary; the main-text figure shows the strongest level of selected cells, and Figure 10 plots GDG_D against Λ\Lambda for four tasks. Where ADD=ACCA_{DD} = A_{CC}, no net loss is observed to recover, although GDG_D is still computed. At chance, two-choice accuracy can fall further only through systematically wrong answers, which occur here only in non-native readouts. Because GD=Λ−UMG_D = \Lambda- U_M and both axes share ADDA_{DD}, a rising curve is partly definitional and its sampling errors are correlated; Table 16 therefore gives the single-entry losses UMU_M and ULU_L. Models and readouts with clean accuracy below chance plus 0.2(1−chance)0.2(1 - \text{chance}) count as non-native. They are listed but not interpreted: DeepStack on stripes, relative position, and text; Granite’s direct and cyclic stripes; Qwen8’s label readouts for relative position; and Granite’s cyclic relative position.

Figure 10. Equal losses do not give equal recovery. On text and DocVQA, Granite recovers much of its loss while Qwen8 and Qwen32 recover little; on identity-label stripes, Qwen32 recovers nearly all of its loss and Qwen8 little. Later recovery GD=ADC−ADDG_D = A_{DC} - A_{DD} against each model’s joint loss Λ=ACC−ADD\Lambda= A_{CC} - A_{DD} at the three shared levels, joined in severity order (generation; organizer ANLS for DocVQA). On the dashed diagonal ADC=ACCA_{DC} = A_{CC}. Both axes share ADDA_{DD}, so their sampling errors are positively correlated. Also, GD=Λ−UMG_D = \Lambda- U_M: a point on the diagonal can be either-sufficient or later-limited, and only ULU_L separates the two. Bars are descriptive 95% intervals for GDG_D; zero-SE points have none, and loss intervals are omitted. Hollow points: no net loss observed, or DeepStack’s 360-pixel DocVQA level, which barely changes main. DeepStack is not natively capable of stripes or text.

Table 16. Single-entry losses separate main-limited, later-limited and either-sufficient cells. Post hoc single-entry losses and item changes for selected cells of the calibrated experiment, recomputed from the stored model outputs without new inference. UM=ACC−ADCU_M = A_{CC} - A_{DC} and UL=ACC−ACDU_L = A_{CC} - A_{CD} carry paired intervals over templates (Student-tt) or images and documents (normal); Λ=GD+UM\Lambda= G_D + U_M. Changes count items whose score rises or falls from DD to DC: the 64 distinct template–object items for synthetic cells (duplicated recovery rows were checked to be identical) and all questions for benchmarks. C+P: case-and-punctuation matching.

ModelCellΛ\LambdaUMU_MULU_LChanges
Qwen32Stripes 0.43, identity0.5000.016 [−0.016, 0.047]0.000 [0.000, 0.000]32 up, 1 down of 64
Qwen32Text 2.5, identity0.5000.500 [0.329, 0.671]0.000 [0.000, 0.000]1 up, 1 down of 64
Qwen32Stripes 0.43, direct0.5000.016 [−0.016, 0.047]0.000 [0.000, 0.000]32 up, 1 down of 64
Qwen8Stripes 0.43, direct0.5000.422 [0.269, 0.574]0.031 [−0.013, 0.076]12 up, 7 down of 64
Qwen32Text 2.5, direct0.4060.406 [0.258, 0.554]0.000 [0.000, 0.000]1 up, 1 down of 64
Qwen32Color 0.05, direct0.2500.000 [0.000, 0.000]0.000 [0.000, 0.000]16 up, 0 down of 64
DeepStackColor 0.05, direct0.4060.000 [0.000, 0.000]0.016 [−0.016, 0.047]26 up, 0 down of 64
GranitePosition 0.3, direct0.4530.047 [−0.023, 0.117]0.484 [0.429, 0.540]28 up, 2 down of 64
GraniteText 2.5, direct0.3910.094 [−0.003, 0.190]0.391 [0.315, 0.466]21 up, 2 down of 64
GraniteGQA attr. 5 px, C+P0.2660.005 [−0.033, 0.043]0.198 [0.143, 0.253]116 up, 28 down of 349
DeepStackGQA attr. 5 px, C+P0.2890.018 [−0.000, 0.036]0.042 [0.005, 0.079]112 up, 18 down of 349
Qwen8DocVQA 180 px, ANLS0.5160.498 [0.446, 0.550]0.025 [0.002, 0.048]22 up, 7 down of 373
Qwen32DocVQA 180 px, ANLS0.4610.454 [0.404, 0.503]−0.002 [−0.029, 0.025]30 up, 23 down of 373
GraniteDocVQA 180 px, ANLS0.6150.234 [0.189, 0.280]0.217 [0.175, 0.260]216 up, 14 down of 373
DeepStackDocVQA 180 px, ANLS0.1850.019 [−0.013, 0.051]0.151 [0.114, 0.188]97 up, 16 down of 373

Table 16.

Several patterns depend on readout. Granite reads stripes near chance with direct answers but accurately with identity labels. There, degrading only its later input lowers accuracy about as far as degrading both (ACDA_{CD} from 0.500 to 0.594, ADDA_{DD} from 0.484 to 0.578), so GDG_D is large while IrecI_{\mathrm{rec}} stays near zero. With label readouts, Qwen8’s color loss reaches 0.469 while GDG_D stays at or below 0.078, and Qwen32’s ADCA_{DC} falls to 0.891 and 0.875 at the strongest desaturation. On text, Qwen8/32 show little recovery across readouts, with no positive gain resolved by the descriptive intervals; one label-readout interval lies below zero. Granite recovers part of its direct text loss but nearly all of its label-readout text losses, again with ACDA_{CD} near ADDA_{DD}.

UMU_M and ULU_L separate the patterns. Granite’s relative-position, text, and identity-label stripe accuracy is later-limited. DeepStack on GQA is either-sufficient: ACDA_{CD} stays near ACCA_{CC}, and IrecI_{\mathrm{rec}} intervals exclude zero at every level of both question groups. Granite’s GQA attribute accuracy at 5 pixels is closer to later-limited. On DocVQA, Granite’s single-entry losses sum to less than the joint loss under both ANLS variants. Under lowercase exact matching its interaction reverses (−0.107 [−0.149, −0.064] at 180 pixels). Removing only a final period restores 0.081 [0.008, 0.153], whereas removing final exclamation or question marks, or outer quotes, leaves the reversal (Table 17); Granite ends 0.914 of answers with a period when only later input is degraded, against 0.682 with both entries clean. Normalized exact interactions are 0.112, 0.128, and 0.081 from mild to severe, so growth with loss is an ANLS description only. Pattern labels are therefore specific to the metric. Qwen32’s main-limited text accuracy coexists with later effects on candidate scores: its score IqualityI_{\mathrm{quality}} is 0.535 [0.304, 0.765] at the strongest level.

Table 17. At similar loss, Qwen32 recovers more on stripes than on text, and more than Qwen8 on stripes. Post hoc paired contrasts and the Granite DocVQA format check. Contrasts subtract within shared templates or documents before the interval is computed. The format check rescores the stored 180-pixel answers with the experiment’s scorer: each row removes one kind of final mark before exact matching, C+P applies all three, and the last row gives the share of answers ending in a period.

Paired contrastGDG_D differenceΛ\Lambda difference
Qwen32 stripes minus text, identity0.484 [0.429, 0.540]0.000 [−0.171, 0.171]
Qwen32 stripes minus text, direct0.484 [0.429, 0.540]0.094 [−0.054, 0.242]
Qwen32 minus Qwen8 stripes, direct0.406 [0.266, 0.547]0.000 [−0.065, 0.065]
Qwen32 minus Qwen8 DocVQA 180 px−0.011 [−0.036, 0.014]−0.055 [−0.102, −0.009]
Granite DocVQA, 180 pxCC / CD / DC / DDIrecI_{\mathrm{rec}}
Organizer ANLS0.828/0.611/0.594/0.2140.163 [0.104, 0.221]
Lowercase exact0.307/0.049/0.159/0.008−0.107 [−0.149, −0.064]
Exact, final period removed0.839/0.523/0.523/0.1280.081 [0.008, 0.153]
Exact, final ! or ? removed0.310/0.049/0.159/0.008−0.109 [−0.152, −0.067]
Exact, outer quotes removed0.310/0.049/0.159/0.008−0.109 [−0.152, −0.067]
C+P exact (all three)0.841/0.523/0.523/0.1280.078 [0.005, 0.151]
Ends with a period0.682/0.914/0.776/0.8880.120 [0.069, 0.170]

Table 17.

Generation conflict effects use a bounded answer indicator, so plateaus or zeros can reflect that bound. Recovery and IqualityI_{\mathrm{quality}} measure different uses of later input and need not agree in direction. Each model’s later input is a different

Table 14. Blur at radius 2 costs little GQA or DocVQA accuracy, and no positive later recovery is resolved. Fixed-contrast standard-task cells under Gaussian blur (radius 2.0), over 256 image or document clusters per task. The primary metric is strict matching for GQA and organizer ANLS for DocVQA, with invalid outputs scored zero. GDG_D is later recovery with degraded main and IrecI_{\mathrm{rec}} its difference from the clean-main gain; intervals are descriptive normal 95% intervals. Named metric sensitivities and invalid-answer policies are in fixed_contrast_benchmarks.csv. Granite and DeepStack capitalize their GQA answers, so their strict GQA rows are a format floor.

Table 15. Shared degradation levels chosen on calibration units from clean and jointly degraded cells only. Retained ability is r=(ADD−chance)/(ACC−chance)r = (A_{DD} - \text{chance})/(A_{CC} - \text{chance}), with chance 0.5 for two-choice tasks and 0 for free-form tasks; each model averages its native-capable readouts. The three levels jointly minimize the summed distance of the cross-model median rr from 0.75, 0.5 and 0.25. Levels give retained chroma (color), blur σ\sigma relative to the stripe period or dot separation, equivalent font size, or long side. NN marks non-native models excluded from the median. Q8, Q32, Gr. and DS abbreviate the four models.

intervention, with its own number, depth, and source of injected entries; equal GDG_D across models therefore does not indicate use of the same information.

Complete estimates

Tables 16 and 17 add post hoc diagnostics recomputed from the stored model outputs; they are exploratory. Table 18 gives the color conflict scores that the main text cites. The files calibrated_synthetic_recovery.csv, calibrated_synthetic_conflict.csv, calibrated_benchmarks.csv and calibrated_capability_levels.csv (Appendix K) retain every model, task, readout, kind, level, metric and invalid-answer policy of the calibrated experiment, including target and control conflict components and the four levels run outside the shared grid. Their values are the unrounded estimates from the same analysis that produced the paper’s tables; nothing is re-estimated. As elsewhere, if one interval excludes zero and another includes it, the pair is not read as a difference.

Removing and coding entry features

This appendix gives the design and the complete results of the coding experiment in Section 5.5 of the main paper. We remove one entry’s image information by replacing its features with zeros or with a calibration mean, or we quantize and code each entry tensor with NNCodec and count bitstream bytes. With later zeroed, Qwen8, Qwen32 and DeepStack lose at most 1.65 points of GQA attribute accuracy, whereas Granite keeps most of its accuracy from later alone only when main is replaced by its mean. Five of six single-entry predictions fixed in advance hold. Coding only DeepStack’s main entry at QP −4 takes 49.31% fewer bytes than coding all entries at shared QP −2, with a difference of 1.10 [−0.40, 2.55] points, in a comparison fixed after the first confirmation results. One held-out image, a fixed Granite configuration scores 2.64 points above a shared QP at

intervention, with its own number, depth, and source of injected entries; equal GDG_D across models therefore does not indicate use of the same information. nearly equal bytes. We credit this to the configuration, not the role diagnosis, whose prediction fails. Qwen shows no consistent advantage over a shared QP.

Design and units

What is coded. Each entry hands the language model one feature tensor per image, after the vision encoder and the projector or merger, with one row per visual token. Qwen8 and Qwen32 have one main and three later tensors (at most 256 tokens of width 4,096 or 5,120), Granite one main and seven later tensors (344 to 1,032 tokens of width 2,560 on the calibration images, depending on its tiling), and DeepStack one main and two or four later tensors, depending on image geometry (576 tokens of width 4,096 each). Decoded

features are cast to BF16 and enter through the native entries (Appendix § 9).

Codec and configurations. The codec is NNCodec 2.1.3 [4], an implementation of the ISO/IEC Neural Network Coding (NNC) standard [21], with plain uniform quantization (no dependent or rate-distortion-optimized quantization) and its entropy coder. Each tensor is coded as its own bitstream at exactly the requested quantization parameter (QP). Larger QP is coarser: the step size is 1 at QP 0, and four QP steps double it. A configuration is written main | later. An integer is the QP of main or of every later entry, so q∣qq \mid q is a shared QP, the setting in which Heidari et al. [12] code Qwen3-VL’s four entry tensors. Z sets an entry to zeros. In the scalenormalized setting nn, later entry ii is coded at QP q+diq + d_i, where qq is main’s QP and di=round⁡(4log⁡2(σi/σmain))d_i = \operatorname{round}(4 \log_{2}(\sigma_i/\sigma_{\mathrm{main}})) compares the standard deviation of the entry’s elements with main’s on 48 calibration images. The offsets did_i are −4, −3, −3 for Qwen8, −5, −2, −7 for Qwen32, −12 to −6 for Granite and 0, 0, +1, +1 for DeepStack. The settings n+kn+k and n−kn-k code every later entry kk QP units coarser or finer than nn. These offsets are relative to nn and not to main: Granite’s fixed configuration 0 | n+4n+4 still codes every later entry

ModelReadoutRR, clean mainIqualityI_{\mathrm{quality}}, level 1Level 2Level 3
Qwen8Direct0.242 [0.145, 0.338]1.921 [1.615, 2.227]2.535 [2.215, 2.855]2.658 [2.275, 3.041]
Identity−0.042 [−0.138, 0.054]1.864 [1.278, 2.451]2.654 [1.833, 3.475]1.771 [0.418, 3.124]
Cyclic−0.011 [−0.179, 0.157]1.731 [1.358, 2.104]2.584 [1.729, 3.439]1.856 [0.863, 2.848]
Qwen32Direct0.531 [0.361, 0.702]3.609 [2.820, 4.398]5.788 [4.945, 6.632]9.552 [8.255, 10.850]
Identity0.169 [0.096, 0.242]3.466 [2.538, 4.393]6.792 [5.669, 7.914]10.383 [8.488, 12.278]
Cyclic0.133 [0.015, 0.251]3.336 [2.093, 4.578]5.742 [4.849, 6.636]9.409 [7.891, 10.927]
GraniteDirect4.470 [4.122, 4.819]0.537 [0.298, 0.776]1.154 [0.862, 1.445]2.006 [1.618, 2.394]
Identity3.954 [3.127, 4.781]0.258 [0.005, 0.511]0.840 [0.452, 1.229]1.633 [1.082, 2.184]
Cyclic3.482 [2.569, 4.394]0.452 [0.152, 0.753]1.100 [0.650, 1.550]1.623 [1.045, 2.202]
DeepStackDirect0.904 [0.723, 1.085]0.549 [0.425, 0.673]1.178 [0.970, 1.386]1.838 [1.658, 2.017]
Identity0.977 [0.758, 1.196]0.813 [0.642, 0.984]1.691 [1.468, 1.913]2.578 [2.415, 2.741]
Cyclic0.932 [0.791, 1.073]0.625 [0.528, 0.722]1.289 [1.069, 1.508]2.215 [1.978, 2.452]

Table 18. Table 18. Degrading main raises later color score selectivity for every model, readout and level: all 36 IqualityI_{\mathrm{quality}} intervals lie above zero. Color conflict scores in the calibrated experiment. RR is the later target-minus-control effect on the candidate score with clean main; Iquality=RI_{\mathrm{quality}} = R(degraded main) − RR(clean main) at the three shared desaturation levels of Table 15, mild to severe. Intervals are descriptive, unadjusted 95% Student-tt intervals over 32 templates. Every task, readout, kind, level and the target and control components are in calibrated_synthetic_conflict.csv.

Bytes and replacements. Bytes are the actual NNCodec bitstream bytes of the coded tensors, headers included. Each question takes the bytes of its image or page, bytes are averaged within each image or document and then over units, and 1 KB is 1,000 bytes. A removed entry counts zero bytes, unchanged features count as uncoded BF16 (2 bytes per element), and inactive DeepStack groups are never counted.

The mean vectors and offset tables are fixed side information and are not counted. “Mean” sets every position of an entry to one vector, the per-image token mean averaged over 48 calibration images (for DocVQA, the 56 page images of 48 calibration documents). For DeepStack’s third and fourth later groups, it is averaged over the 24 of these images where the groups are active. Neither replacement depends on the test image. Zeroing replaces a tensor and leaves the branch in place, so we claim nothing about speed or memory.

Data, scoring and statistics. Test units come from GQA validation-balanced attribute questions and Single-Document DocVQA validation. We excluded every image or document used in the other experiments of this paper, ordered the remaining units by a fixed-seed hash, and took them in that order with at most two questions per unit. This gives 1,000 GQA images with 1,838 questions (the confirmation images) and all 474 remaining DocVQA documents with 907 questions on 518 pages. Calibration uses the 240 units per task of the calibrated experiment (Appendix § 15), its 48 calibration and 192 evaluation units (87 and 349 GQA attribute questions). The 48 calibration units give the mean vectors, the 48 GQA attribute calibration images give the offsets djd_j for all tasks, and all 240 units select configurations. Generation is greedy with at most 64 new tokens and batch size 1. GQA answers are scored by exact match after caseand-punctuation normalization (Appendix § 15), DocVQA by organizer-framework ANLS×100, and invalid outputs score zero. Scores are averaged within each image or document and then over units. Intervals are paired bootstrap 95% percentile intervals over images (documents for DocVQA), 4,000 resamples, with resampling weights drawn once per image set and shared by every estimate on that set. A prediction holds when its whole interval lies on the predicted side of zero.

What was fixed in advance. Before any confirmation image was scored, a protocol fixed 31 configurations for each of four models and three tasks (GQA attribute, GQA spatial and DocVQA): the seven single-entry configurations of Table 19 and four families of six coded configurations (shared QP, scale-normalized, later coarser than n, later finer than n). At that point the confirmation units had been used only for feature extraction and coding. The protocol also fixed ten single-entry predictions, the role-direction predictions and a selection rule. The rule defines a high and a low budget, the calibration bytes of the cheapest shared-QP configuration that reaches 95% or 60% of the way from both entries zeroed to the full input on calibration units, and takes in each family the configuration with the highest calibration score within each budget.

Main inputLater inputQwen8Qwen32Granite, 999DeepStack
Images or documents / questions1,000 / 1,8381,000 / 1,838999 / 1,8361,000 / 1,838
imageimage72.50 [70.40, 74.50]72.75 [70.60, 74.85]73.27 [71.17, 75.33]79.50 [77.60, 81.40]
imagezeros72.20 [70.14, 74.15]71.90 [69.70, 74.00]51.60 [49.15, 53.90]77.85 [75.80, 79.75]
imagemean71.95 [69.80, 73.90]72.65 [70.50, 74.65]58.81 [56.46, 61.11]75.70 [73.55, 77.70]
zerosimage54.45 [51.95, 56.85]67.60 [65.30, 69.80]11.66 [10.11, 13.26]76.05 [74.00, 78.00]
meanimage50.50 [48.00, 52.90]65.40 [63.10, 67.65]69.77 [67.57, 71.97]29.35 [27.30, 31.50]
zeroszeros31.55 [29.30, 33.80]43.20 [40.85, 45.55]1.00 [0.60, 1.45]11.45 [10.00, 13.05]
meanmean43.10 [40.80, 45.50]43.30 [40.90, 45.60]42.29 [39.94, 44.54]10.35 [8.90, 11.85]
Main inputLater inputGranite, 531Granite, 468Qwen32 DocVQA
Images or documents / questions531 / 970468 / 866474 / 907
imageimage74.86 [72.03, 77.68]71.47 [68.38, 74.57]78.64 [76.09, 81.10]
imagezeros53.11 [49.81, 56.50]49.89 [46.58, 53.31]76.05 [73.30, 78.64]
imagemean59.89 [56.69, 63.28]57.59 [54.06, 61.11]76.99 [74.38, 79.47]
zerosimage12.24 [10.17, 14.41]11.00 [8.87, 13.25]7.09 [5.57, 8.58]
meanimage71.19 [68.27, 74.11]68.16 [64.85, 71.47]4.04 [2.85, 5.31]
zeroszeros0.75 [0.28, 1.41]1.28 [0.64, 2.03]6.60 [5.22, 7.98]
meanmean42.56 [39.55, 45.76]41.99 [38.57, 45.51]5.34 [4.07, 6.71]

Table 19. Table 19. Which entry suffices depends on the model and the replacement. GQA attribute accuracy on new images when an entry’s features are replaced; Qwen32 DocVQA reports ANLS × 100 on all 474 remaining DocVQA documents. “image” keeps the entry’s features unchanged; “zeros” sets all of them to zero; “mean” sets every position to the entry’s mean feature vector over 48 calibration images (for DocVQA, the 56 page images of 48 calibration documents), so neither replacement depends on the test image. Qwen8, Qwen32 and DeepStack are evaluated on the 1,000 confirmation images. Granite’s first 468 images come from the confirmation runs; its seven configurations were then run on the 531 remaining images, which Granite had not seen, and 999 pools the two. Values in percent. Intervals are paired bootstrap 95% intervals over images (documents for DocVQA), 4,000 resamples.

Of the twelve cells (four models, three tasks), four were completed: Qwen8, Qwen32 and DeepStack on attribute questions, and Qwen32 on DocVQA. We ended the confirmation runs after seeing the first confirmation results. At that point Granite had 468 complete attribute images (Granite’s first 468 images below), and the other seven cells had not been run, so there is no GQA spatial result for any model. We then fixed six follow-up runs and their comparisons before running them. Four ran main alone with later zeroed, each compared with the shared QP at the same main QP: DeepStack, Qwen8 and Qwen32 on attribute questions, and Qwen32 on DocVQA. The fixed comparisons also include DeepStack −4 | Z against −2 | −2, DeepStack −2 | Z against −2 | n+2 and Qwen8 0 | Z against 2 | 2. The other two ran on the 531 confirmation images that Granite had not seen: Granite’s seven single-entry configurations, and the fixed Granite pair of coded configurations (Appendix § 16.§ 4). Nothing was added or reselected from these results. Tables mark each estimate as fixed in the protocol (L), a fixed follow-up (F) or post hoc (P). The Qwen32 byte decomposition and the training comparisons with plain fine-tuning are also post hoc.

Runs. Confirmation and follow-up runs used BF16 on NVIDIA A100 80GB GPUs, with the software versions of Appendix § 14. An unmodified pass through each model’s adapter matched the native model, and our GPU implementation of the quantizer reproduced NNCodec’s decoded values exactly for all four models. Exploratory runs used BF16 on an Apple-silicon GPU and the same 240 calibration units. We cite them as exploratory and do not pool them with confirmation results.

What these results do not show. No configuration was tested for near-lossless coding, and bytes count coded features, not end-to-end latency or memory. The comparisons do not establish a general role-based coding method, an allocation that needs no calibration or is optimal, or an advantage over a generic configuration search, and no rule for splitting bytes between entries follows from them. Training shows no role-specific benefit. The results also do not say which entry to transmit first under limited bandwidth: Granite’s two single-entry options differ sevenfold in raw payload.

Single-entry accuracy and predictions fixed in advance

Which entry suffices depends on the model and on the replacement (Table 19 and Table 20). With later zeroed, Qwen8 scores 72.20% against 72.50% for the full input (−0.30 [−1.55, 0.95]), Qwen32 71.90% against 72.75% and DeepStack 77.85% against 79.50%. Granite loses 14.46 to 21.67 points without its later entries, but with later alone it scores 69.77% against 73.27% when main is replaced by its mean (−3.50 [−5.16, −1.85]) and 11.66% when main is zeroed. DeepStack depends on the replacement value in the opposite way: 76.05% with main zeroed and 29.35% with main replaced by its mean. Under mean replacement, keeping main rather than later gains 21.45 points for Qwen8, and keeping later rather than main gains 10.96 points for Granite. The 999 Granite images are its first 468 images and the 531 held-out images, and the two groups agree: this Granite difference is 11.30 [7.91, 14.60] points on the 531 held-out images and 10.58 [6.94, 14.10] on the 468.

ComparisonQwen8Qwen32Granite, 999DeepStack
Later zeroed minus full−0.30 [−1.55, 0.95]−0.85 [−2.25, 0.50]−21.67 [−24.27, −19.22]−1.65 [−3.05, −0.20]
Later mean-replaced minus full−0.55 [−1.80, 0.65]−0.10 [−1.40, 1.15]−14.46 [−16.82, −12.06]−3.80 [−5.35, −2.35]
Main zeroed minus full−18.05 [−20.55, −15.70]−5.15 [−6.95, −3.40]−61.61 [−64.02, −59.16]−3.45 [−4.80, −2.10]
Main mean-replaced minus full−22.00 [−24.50, −19.50]−7.35 [−9.20, −5.55]−3.50 [−5.16, −1.85]−50.15 [−52.75, −47.55]
Main kept minus later kept, mean21.45 [18.85, 24.05]7.25 [5.35, 9.10]−10.96 [−13.51, −8.46]46.35 [43.60, 49.05]
Main kept minus later kept, zeros17.75 [15.35, 20.20]4.30 [2.40, 6.15]39.94 [37.49, 42.49]1.80 [0.20, 3.40]
ComparisonGranite, 531Granite, 468
Later zeroed minus full−21.75 [−25.14, −18.27]−21.58 [−25.32, −17.84]
Later mean-replaced minus full−14.97 [−18.27, −11.58]−13.89 [−17.41, −10.47]
Main zeroed minus full−62.62 [−66.10, −59.23]−60.47 [−64.00, −56.94]
Main mean-replaced minus full−3.67 [−5.93, −1.41]−3.31 [−5.77, −0.85]
Main kept minus later kept, mean−11.30 [−14.60, −7.91]−10.58 [−14.10, −6.94]
Main kept minus later kept, zeros40.87 [37.48, 44.35]38.89 [35.15, 42.52]
Other comparisonModel, images
Main zeroed minus both meanGranite, 999−30.63 [−33.13, −28.03]
Later zeroed minus fullQwen32 DocVQA−2.58 [−4.11, −1.14]

Table 20. Paired differences between single-entry configurations. Percentage points (ANLS×100 points for DocVQA), same images and replacements as Table 19. “Main kept minus later kept” compares keeping only main with keeping only the later entries when the removed entry is replaced by its mean or by zeros. The kept entries differ in raw size, so this is not a comparison at equal bytes. Status: L, fixed in the protocol before any confirmation image was scored; F, follow-up comparison fixed after the first confirmation results and before the follow-up runs; P, post hoc. In this table the first five rows for Granite on all three image sets are F (the follow-up stated the mean comparison with the opposite sign), and every other entry is P. Intervals are paired bootstrap 95% intervals over images (documents for DocVQA), 4,000 resamples.

Six of the ten single-entry predictions have data, and five hold (Table 21). All use zeros as the replacement. Granite’s prediction for later alone fails on all three image sets, and on the 999 images later alone loses 0.852 of the full visual gain. The protocol kept zeros for this prediction although it had already failed on the calibration units (0.891 of the gain) Table 23. DeepStack paired comparisons on the 1,000 confirmation images. Byte change is the KB of the first configuration relative to the second, in percent; the difference is in accuracy points. Notation as in Table 22. The protocol predicted no role direction for this model and task, so the comparisons of the selected configuration carry no predicted sign. Status codes and intervals as in Table 20.

ModelEntry keptImagesffLoss/gainMarginOutcomeStatus
Qwen8main1,0000.250.0079.94 [8.68, 11.24]holdsL
Qwen32main1,0000.500.02913.93 [12.20, 15.55]holdsL
Qwen32later1,0000.500.1749.63 [7.78, 11.53]holdsL
Granitelater4680.250.861−42.92 [−45.91, −39.90]failsL
Granitelater5310.250.845−44.09 [−46.99, −41.17]failsF
Granitelater9990.250.852−43.54 [−45.58, −41.44]failsF
DeepStackmain1,0000.500.02432.38 [30.73, 34.00]holdsL
DeepStacklater1,0000.500.05130.58 [28.90, 32.18]holdsL

Table 21. Single-entry predictions fixed in the protocol: five of six hold. Each prediction states that keeping one entry (the other replaced by zeros) loses at most a fraction ff of the full visual gain, the accuracy of the full input minus that with both entries zeroed. Loss/gain is the observed loss as a fraction of that gain. The margin, in percentage points, is ff times the gain minus the loss; a prediction holds when the whole interval lies above zero. Granite’s prediction fails on its first 468 images, on the 531 held-out images and on both together. The four predictions that do not (Qwen8 on GQA spatial and on DocVQA, main kept, f=0.25f = 0.25; DeepStack on GQA spatial, each entry kept, f=0.5f = 0.5) belong to cells that were not run. Status codes and intervals as in Table 20.

Table 22. DeepStack: every coded configuration on the 1,000 confirmation images. GQA attribute accuracy in percent; KB per image (1 KB = 1,000 bytes of NNCodec bitstream, zeroed entries count 0). Configurations are written main | later: an integer is the QP of main or of every later entry (shared QP); nn is the scale-normalized setting, in which later entry ii is coded at QP q+diq+d_i with did_i its rounded scale offset (0, 0, +1, +1 for DeepStack); n+kn+k and n−kn-k code the later entries kk QP units coarser or finer than nn. Larger QP is coarser, and four QP steps double the quantization step. Z sets all later features to zero. *Selected on calibration images at the high budget. The uncoded full input scores 79.50%. Status codes and intervals as in Table 20.

ConfigurationStatusKB totalKB mainKB laterAccuracy
Shared QP
−4 | −4L1,002.32255.08747.2478.95 [77.00, 80.90]
−2 | −2*L503.26131.07372.1976.35 [74.30, 78.35]
−1 | −1L350.2992.11258.1867.75 [65.50, 70.05]
0 | 0L243.1364.28178.8642.65 [39.90, 45.25]
2 | 2L71.4317.8053.6216.40 [14.50, 18.30]
4 | 4L42.129.5332.6027.05 [24.90, 29.35]
Scale-normalized
−4 | nL924.20255.08669.1278.95 [77.00, 80.95]
−2 | n*L462.14131.07331.0676.25 [74.20, 78.25]
−1 | nL321.7192.11229.6067.70 [65.45, 70.05]
0 | nL211.3964.28147.1141.75 [39.05, 44.40]
2 | nL66.1617.8048.3515.55 [13.70, 17.40]
4 | nL40.309.5330.7726.45 [24.30, 28.70]
Later coarser than scale-normalized
−2 | n+2*L278.19131.07147.1175.80 [73.70, 77.85]
−2 | n+4L179.43131.0748.3574.90 [72.80, 77.05]
0 | n+2L112.6364.2848.3535.70 [33.10, 38.30]
0 | n+4L95.0564.2830.7734.50 [31.95, 37.10]
4 | n+2L33.169.5323.6424.90 [22.75, 27.05]
4 | n+4L26.919.5317.3826.85 [24.70, 29.20]
Later finer than scale-normalized
−2 | n−4L1,132.93131.071,001.8677.15 [75.10, 79.20]
−2 | n−2L800.19131.07669.1276.60 [74.50, 78.65]
0 | n−4L733.3964.28669.1247.80 [45.00, 50.65]
0 | n−2*L395.3464.28331.0645.35 [42.55, 48.10]
4 | n−4L156.649.53147.1138.85 [36.30, 41.40]
4 | n−2L57.889.5348.3528.05 [25.80, 30.35]
Main only, later entries zeroed
−4 | ZF255.08255.080.0077.45 [75.40, 79.45]
−2 | ZF131.07131.070.0074.90 [72.85, 77.05]
−1 | ZF92.1192.110.0064.15 [61.75, 66.60]

Table 22.

ComparisonKB, firstKB, secondByte change (%)DifferenceStatus
Main only minus shared QP at the same main QP
−4Z minus −4−4255.081,002.32−74.55−1.50 [−2.90, −0.15]F
−2Z minus −2−2131.07503.26−73.95−1.45 [−2.95, 0.05]F
−1Z minus −1−192.11350.29−73.70−3.60 [−5.35, −1.90]F
Across main QP
−4Z minus −2−2255.08503.26−49.311.10 [−0.40, 2.55]F
Configuration selected on calibration images
−2nn+2 minus −2nn278.19462.14−39.80−0.45 [−1.40, 0.50]L
−2nn+2 minus −2−2278.19503.26−44.72−0.55 [−1.55, 0.45]L
−2Z minus −2nn+2131.07278.19−52.88−0.90 [−2.25, 0.45]F
Post hoc
−4Z minus −2nn255.08462.14−44.801.20 [−0.25, 2.65]P

Table 23. DeepStack paired comparisons on the 1,000 confirmation images.

Table 24. Granite: the fixed pair on held-out images and the scale-normalized configuration with the same later setting. GQA attribute accuracy in percent, KB per image. The pair 0 | nn+4 versus −4 | −4 was found on Granite’s first 468 images and fixed before the 531 held-out images were scored; the 468 images are discovery data and are not pooled with the 531. Granite’s scale offsets are −12 to −6, so in 0 | nn+4 every later entry is coded finer than main. 4 | nn codes the later entries exactly as 0 | nn+4 does and main four QP units coarser. It was not run on the 531 images. Notation as in Table 22. Status codes and intervals as in Table 20.

ImagesConfigurationKB mainKB laterKB totalAccuracy
Configurations
531 held-out0nn+4187.23208.40395.6369.30 [66.20, 72.32]
531 held-out−4−4305.9086.99392.8966.67 [63.84, 69.77]
468 discovery0nn+4189.32209.77399.0966.56 [63.25, 69.87]
468 discovery−4−4309.1887.35396.5363.14 [59.72, 66.56]
468 discovery4nn63.38209.77273.1565.60 [62.29, 68.91]
Paired differences
ImagesComparisonStatusByte change (%)Difference
531 held-out0nn+4 minus −4−4F0.702.64 [0.47, 4.80]
468 discovery0nn+4 minus −4−4P0.653.42 [1.28, 5.45]
468 discovery4nn minus −4−4P−31.122.46 [0.11, 4.81]
468 discovery0nn+4 minus 4nnP46.110.96 [−0.96, 2.88]

Table 24. Granite: the fixed pair on held-out images and the scale-normalized configuration with the same later setting.

and the mean replacement had scored higher there.

A separate rule picks, among the four single-entry options, the one with the highest calibration accuracy. It was written after the exploratory results for these two cells were known and was not adopted by the protocol. On the 48 calibration images it selects main with zeroed later for Owen8 (68.75%) and later with a mean-filled main for Granite (72.92%), and both are also the best of the four options on the new images, an agreement that is not a test.

The single-entry comparison concerns function, not rate. Granite’s seven later tensors together hold seven times main’s feature elements, yet after coding at a shared QP main takes most of the bytes (305.90 of 392.89 KB at −4 | −4 on the 531 images, Table 24). On DocVQA, Qwen32 scores 76.05 against 78.64 ANLS points with later zeroed (−2.58 [−4.11, −1.14]), whereas with main zeroed it scores 7.09, close to the 6.60 with both entries zeroed. The protocol fixed no single-entry prediction for this cell.

DeepStack: bytes and accuracy

DeepStack’s shared-QP accuracy falls monotonically with bytes down to 71.43 KB (QP 2, 16.40%) and is higher again at 42.12 KB (QP 4, 27.05%, Table 22). The curve is thus not monotone below about 75 KB. DeepStack configurations are compared directly in pairs. The main paper reports two readouts (Table 23). First, main alone at QP −4 with later zeroed takes 255.08 KB for 77.45%, against 503.26 KB for 76.35% at shared QP −2: 49.31% fewer bytes and a difference of 1.10 [−0.40, 2.55] points. An exploratory run on the 240 calibration units had shown about 494 KB for shared QP −2 against 254 KB for main alone at QP −4, both within three points of the full input. We fixed this pair after the first confirmation results, which already included uncoded main alone (77.85%), main’s bytes at QP −4 and the score of the shared configuration, and before −4 | Z itself was run. Second, the protocol rule selected −2 | nn+2 at the high budget. It takes 278.19 KB for 75.80%, 39.80% fewer Table 25. Qwen: main only and scale-normalized coding against shared QP. Score in percent (ANLS×100 for DocVQA), KB per image or document, differences in points. Main-only rows keep main at the stated QP and zero the later entries. They are compared with the shared configuration at the same main QP, and the byte change is relative to that configuration. Every row is also compared with the shared-QP curve at the same KB, read by linear interpolation in log KB between the two neighboring shared points and recomputed in each bootstrap resample. Of the seven main-only differences, five intervals contain zero, one lies wholly above zero (Qwen32 attribute, 12 | Z) and one wholly below (Qwen32 DocVQA, 10 | Z). Of the 13 scale-normalized differences, no interval lies wholly above zero and eight lie wholly below. Where the curve bends, linear interpolation can fall below it and overstate a positive difference: 12 | Z lies between 14 | 14 (5.33 KB, 57.10) and 12 | 12 (18.90 KB, 69.25). Five scale-normalized configurations lie outside the byte range of the shared points and are omitted. The shared-QP points are plotted in Figure 11 and listed in coding_configurations.csv. Two further Qwen8 comparisons: 0 | Z minus 2 | 2 is 0.55 [−0.70, 1.85] points with 17.98% more bytes (F), and the configuration selected on calibration images at the high budget, 2 | n+4 (45.64 KB, 70.10), minus 2 | 2 is −0.25 [−1.20, 0.65] points with 12.86% fewer bytes (L). Uncoded full input: Qwen8 72.50, Qwen32 72.75, Qwen32 DocVQA 78.64. Notation as in Table 22. Status codes and intervals as in Table 20.

Figure 11. Accuracy against coded bytes on the confirmation images. Each point is one configuration: GQA attribute accuracy on 1,000 images, or ANLS×100 on 474 DocVQA documents, against NNCodec bytes per image or document on a log scale. Shared QP codes main and every later entry at one QP; scale-normalized codes later entry ii at QP q+diq + d_i; other later offsets code the later entries coarser or finer than scale-normalized; main only codes main and sets the later entries to zero (0 bytes). Thin bars are paired bootstrap 95% intervals over images or documents; the dashed line is the uncoded full input. Rings mark the DeepStack pair reported in § 5.5 (main only at QP −4, shared QP −2). DeepStack’s values are in Table 22, and the Qwen main-only and scale-normalized values in Table 25. Every point, including Granite’s configurations on 468 images, is listed in coding_configurations.csv.

bytes than the scale-normalized −2 | n at the same main QP (−0.45 [−1.40, 0.50]) and 44.72% fewer than −2 | −2 (−0.55 [−1.55, 0.45]). The protocol had classed this cell as one without a predicted role direction, so these comparisons carry no predicted sign. The protocol also set an accuracy target, 95% of the gap between both entries zeroed and the full input, which is 76.10% on these images. The shared and scale-normalized configurations at main QP −2 reach it (76.35% and 76.25%), and the selected −2 | n+2 falls just below it (75.80%).

At an equal main QP, the later entries still contribute. Zeroing them changes accuracy by −1.50 [−2.90, −0.15] points at QP −4, −1.45 [−2.95, 0.05] at QP −2 and −3.60 [−5.35, −1.90] at QP −1, with about 74% fewer bytes each time. Post hoc, main alone at QP −4 scores 1.20 [−0.25, 2.65] points above −2 | n with 44.80% fewer bytes.

The cost of removing an entry depends on the task. On the new attribute images, keeping main alone changes accuracy by −1.65 [−3.05, −0.20] points and keeping later alone by −3.45 [−4.80, −2.10]. Both differences are post hoc: the protocol fixed these configurations but not their differences from the full input. Exploratory runs on an Apple-silicon GPU, with zeros and on the 192 evaluation units of Appendix G, gave −3.91 [−7.03, −1.04] for main alone and 0.26 [−3.91, 4.43] for later alone on GQA spatial questions. On DocVQA, where the full input scores only 34.56 ANLS points, they gave −9.48 [−13.42, −5.56] for main alone and −6.03 [−9.99, −2.28] for later alone. These four differences are exploratory, not confirmation results, and their intervals are paired bootstrap intervals over the 192 units, computed for those runs.

Granite: a fixed pair on held-out images

On the 531 held-out images, 0 | n+4 scores 69.30% at 395.63 KB and the shared −4 | −4 scores 66.67% at 392.89 KB: 0.70% more bytes and a difference of 2.64

[0.47, 4.80] points (Table 24). Relative to the shared QP, the configuration moves bytes from main to later.

The pair was found post hoc. 0 | n+4 is one of the 31 protocol configurations, but the selection rule did not pick it. We first read its advantage over −4 | −4 on Granite’s first 468 images (3.42 [1.28, 5.45]), among several comparisons with shared-QP configurations, and then fixed this single pair and scored it on the 531 images that Granite had not seen.

We credit the gain to this configuration and not to the role diagnosis. On the 468 discovery images, the scale-normalized 4 | n, which codes the later entries exactly as 0 | n+4 does and main four QP units coarser, already scores 2.46 [0.11, 4.81] points above −4 | −4 with 31.12% fewer bytes, and 0 | n+4 minus 4 | n is 0.96 [−0.96, 2.88]. Both comparisons are post hoc. An exploratory run on the 192 evaluation images of Appendix § 15 that coded only the later entries, all at QP −8, with main replaced by its mean, did not differ from −4 | −4 at matched bytes (395.51 against 395.24 KB, 0.00 [−4.69, 4.43]).

Qwen: no consistent advantage over a shared QP

At the same main QP, zeroing Qwen’s later entries lowers the point estimate at every tested QP, by 0.50 to 3.20 points, and saves 4.74% to 55.34% of the bytes (Table 25). Against the shared-QP curve at the same bytes (post hoc), the interval of one of the seven main-only differences lies wholly above zero, Qwen32 attribute 12 | Z (1.74 [0.69, 2.79]), at a bend where linear interpolation of the full label can fall below the curve, and one lies wholly below, Qwen32 DocVQA 10 | Z (−1.54 [−2.95, −0.20]). Of the 13 scale-normalized differences, no interval lies wholly above zero and eight lie wholly below.

ConfigurationFamilyKBScoreDifferenceByte change (%)CurveDifference
Qwen8, GQA attribute (1,000 images)
0 | Zmain only61.7970.90 [68.75, 72.95]−1.55 [−2.80, −0.25]−55.3470.710.19 [−0.94, 1.41]
2 | Zmain only33.7868.80 [66.65, 70.90]−1.55 [−2.85, −0.25]−35.5168.010.79 [−0.38, 1.94]
6 | Zmain only6.3854.75 [52.45, 57.10]−0.50 [−1.35, 0.35]−9.0754.730.02 [−0.80, 0.87]
0 | nscale-normalized304.3272.30 [70.15, 74.35]72.84−0.54 [−1.46, 0.37]
2 | nscale-normalized156.7971.85 [69.70, 73.90]72.51−0.66 [−1.78, 0.47]
4 | nscale-normalized66.7669.65 [67.50, 71.80]70.87−1.22 [−2.53, 0.08]
6 | nscale-normalized18.2557.40 [55.15, 59.75]63.71−6.31 [−7.91, −4.68]
8 | nscale-normalized4.8149.60 [47.30, 52.00]53.17−3.57 [−4.88, −2.29]
Qwen32, GQA attribute (1,000 images)
6 | Zmain only105.4171.75 [69.55, 73.85]−1.20 [−2.55, 0.15]−47.0972.30−0.55 [−1.76, 0.71]
12 | Zmain only14.7468.60 [66.40, 70.80]−0.65 [−1.70, 0.45]−22.0166.861.74 [0.69, 2.79]
10 | nscale-normalized143.6172.70 [70.55, 74.85]72.620.08 [−1.07, 1.24]
12 | nscale-normalized49.9071.20 [69.00, 73.35]71.53−0.33 [−1.60, 0.97]
14 | nscale-normalized12.9761.15 [58.80, 63.55]65.64−4.49 [−6.08, −2.88]
16 | nscale-normalized4.4547.25 [44.90, 49.60]54.67−7.42 [−9.20, −5.68]
Qwen32, DocVQA (474 documents, ANLS×100)
6 | Zmain only140.0074.86 [72.23, 77.50]−1.61 [−3.08, −0.17]−19.1974.750.11 [−1.26, 1.48]
10 | Zmain only49.7952.46 [49.36, 55.54]−3.20 [−4.67, −1.82]−4.7453.99−1.54 [−2.95, −0.20]
10 | nscale-normalized115.7560.10 [57.15, 63.01]73.21−13.12 [−15.63, −10.64]
12 | nscale-normalized44.9738.33 [35.45, 41.26]50.50−12.18 [−14.32, −9.96]
14 | nscale-normalized12.5514.46 [12.50, 16.47]19.70−5.24 [−6.54, −3.93]
16 | nscale-normalized4.748.52 [6.98, 10.08]10.10−1.57 [−2.35, −0.79]

Table 25.

The configurations selected on calibration units give the same picture. For Qwen8, the rule selected 2 | n+4 at the high budget: 12.86% fewer bytes than 2 | 2 and a difference of −0.25 [−1.20, 0.65]. For Qwen32 attribute questions it selected 8 | n+4 (105.75 KB, 72.50%), which used 46.91% fewer bytes than the shared configuration selected at the same budget, 6 | 6 (199.21 KB, 72.95%). Most of this difference, 88.64% of the bytes, lies between the shared points 6 | 6 and 8 | 8. At the fixed main QP, 8 | n+4 saved 9.13% of the bytes of 8 | 8 with a difference of 0.10 [−0.80, 1.00] (post hoc).

Role-direction comparisons and training pilot

Role allocation. The protocol classified each cell from the single-entry losses of the calibrated experiment (Appendix § 15): main-limited (Qwen8 on all three tasks, Qwen32 on DocVQA), later-limited (Granite attribute, DeepStack DocVQA), without a predicted direction (Qwen32 and DeepStack attribute, DeepStack spatial), or descriptive only. The prediction was that the role allocation beats its mirror (Table 26). It holds at both budgets for Qwen8 attribute (25.90 [23.25, 28.50] and 10.80 [8.95, 12.85]) and Qwen32 DocVQA (30.91 [27.86, 34.02] and 25.91 [23.12, 28.73]), and fails for Granite on the 468 complete images (−29.27 [−33.33, −25.11] and −11.43 [−14.53, −8.33]). The other cells with a role prediction were not run. The two arms of a pair are not matched in bytes: at Granite’s high budget the mirror used 614.73 KB and the role allocation 212.91 KB. Beating the mirror also does not imply beating a shared QP: Qwen8’s high-budget role configuration is the 2 | n+4 that scored −0.25 [−1.20, 0.65] against 2 | 2.

Model, taskBudgetRoleKBScoreMirrorKBScoreDifferenceOutcome
Qwen8, attributehigh2 | n+445.6470.1010 | n−412.4444.2025.90 [23.25, 28.50]holds
Qwen8, attributelow6 | n+46.7754.7510 | n−22.9843.9510.80 [8.95, 12.85]holds
Qwen32, DocVQAhigh8 | n+4117.4972.1512 | n−291.8041.2430.91 [27.86, 34.02]holds
Qwen32, DocVQAlow12 | n+230.3934.9216 | n−422.269.0025.91 [23.12, 28.73]holds
Granite, attribute (468)high8 | n−4212.9140.920 | n+2614.7370.19−29.27 [−33.33, −25.11]fails
Granite, attribute (468)low12 | n−436.519.628 | n+219.7221.05−11.43 [−14.53, −8.33]fails

Table 26. Role-direction predictions fixed in the protocol: they hold for Qwen and fail for Granite. For a model whose main entry carried most of the accuracy (Qwen8 attribute, Qwen32 DocVQA) the role allocation codes the later entries coarser than scale-normalized (n+k); for Granite, whose later entries carried most of it, it codes them finer (n−k). The mirror reverses the sign of kk. At each budget each allocation takes its best configuration on calibration images within the byte cap, so the two compared configurations need not have equal bytes. The high and low budgets are the cheapest shared-QP configurations reaching 95% and 60% of the gap between both entries zeroed and the full input on calibration images. The prediction was role minus mirror above zero. Granite has only its first 468 images, so the protocol’s analysis was not run for it; its differences are computed with the same bootstrap as the other rows. Score in percent (ANLS×100 for DocVQA), KB per image or document, difference in points. All rows are L. Notation as in Table 22. Status codes and intervals as in Table 20.

Training pilot. A pilot asked whether fine-tuning on coded features helps, and whether role-specific offsets help more than generic ones. For Qwen8 and Granite, we trained rank-16 LoRA adapters on the language model, with the vision side frozen, for one pass over 8,000 GQA training attribute questions on 4,352 images (500 optimizer steps of 16 samples), disjoint from all evaluation units. Plain fine-tuning sees the full input. Three arms see coded features with the later entries offset by ∣k∣∈{2,4}|k| \in\{2, 4\} QP units from n, with a random sign (generic), the role sign, or the opposite sign (anti-role). For each sample the three arms share the image, ∣k∣|k| and a byte cap, which is the bytes of that image at 0∣n0 \mid n, 2∣n2 \mid n, 4∣n4 \mid n or 6∣n6 \mid n. Each arm takes the finest main QP that fits the cap, so actual bytes differ slightly (role-to-anti-role byte ratio 0.971 for Qwen8, 1.021 for Granite). All arms were evaluated on the 192 evaluation images of Appendix § 15 (349 questions) at fixed configurations that give every arm the same bytes. The pilot was not evaluated on the confirmation images, so it is exploratory.

At 8∣n8 \mid n (4.89 KB), a coarser level than any training budget, Qwen8’s generic arm scored 67.97% against 51.30% after plain fine-tuning (16.67 [10.94, 22.14], post hoc). The role-specific arm did not differ from the generic arm (−0.78 [−4.95, 3.13]) and scored below the anti-role arm (−3.91 [−7.55, −0.26]). For Granite, all four fine-tuned arms scored 75.78% at 4∣n4 \mid n. All pilot scores and comparisons are in coding_configurations.csv and coding_comparisons.csv.

Complete estimates

coding_single_entry.csv holds every single-entry score, paired difference and prediction margin on each image set. coding_configurations.csv holds every configuration on every image set with its bytes per entry and score, including Granite’s 31 configurations on the 468 images, the shared-QP points of Figure 11 and both training checkpoints (150 and 500 steps). coding_comparisons.csv holds every paired comparison with both byte sizes. Values are unrounded and carry intervals and status codes (Appendix § 19). The exploratory Apple-silicon results quoted here are not in these files.

Additional experiments

This appendix reports results that the main text cites only briefly or not at all. Sections I.1–I.4 describe exploratory experiments, together with further readouts of the main spatial and whole-source panels; Sections I.5–I.7 mainly give further readouts and cells of the main experiments. Exploratory results are not pooled with the main panels.

Removing later entries on a benchmark

We removed Qwen3-VL-4B’s three later entries on all 1,901 labeled BLINK validation items [9]. Each entry was replaced by zeros at its own injection site, with main unchanged. We scored every complete option label under the eight on/off combinations; this run used BF16 on an Apple-silicon GPU. Removing all three lowered accuracy only from 49.18% to 47.29% and the mean correct-option margin by 0.1391 nats (SE 0.0307), but it changed 251 predictions: 115 from correct to wrong, 79 from wrong to correct, and 57 between wrong options.

Removing one entry changed 140 to 147 predictions with mean margin changes between −0.0153 and −0.0044, and 442 items changed under at least one combination. Item responses were entry-specific: on the 1,789 items other than the 112 analyzed first, repeated removals of the same entry correlated at 0.41 across items, against 0.10 for different entries, and keeping any one entry recovered 50% to 73% of the margin lost by removing all three. Changed predictions concentrated in near-tie items, which made up 17% of these 1,789 items but gave 68% to 75% of single-entry changes.

Item effects also depended on the answer format. On 204 items with an explicit letter format, removal lowered accuracy from 119 to 110 (Qwen3-VL-4B) and from 117 to 105 (Qwen3-VL-8B) with options in their original order, but only from 96 to 94 and from 91 to 88 with cyclically permuted options, and the item-level changes correlated at 0.049 and 0.0004 between the two orders. Removal therefore gives small net changes in accuracy, many offsetting item changes and format-dependent item effects. The conflict and recovery experiments instead vary entry inputs and read effects through target and control queries. Appendix § 16 removes entries in Qwen8, Qwen32, Granite and DeepStack, on new GQA images and DocVQA documents, with a calibration mean as well as zeros as the replacement. A post hoc analysis of 135 Visual Similarity items also found that single later entries act conditionally: in Qwen3-VL-8B, exchanging the main input raised the third entry’s effect by 0.2673 nats (SE 0.0803).

Spatial correspondence across designs and models

Three further designs test whether entry effects follow the object currently at a position. First, a factorial crossed the receiving scene’s layout (M0/M1M0/M1), the donor’s layout and the placement of later-entry tokens. One interaction separates current scene correspondence from the original coordinates. In Qwen3-VL-4B and 8B, it was above zero in 12 of 12 attribute, background and readout strata. Real Visual Genome crops recomposed into two-object canvases gave the same pattern in Qwen3-VL-4B and in Qwen3-VL-8B’s unit means. The factorial was repeated with Granite (all later entries and its two entry groups), Qwen32, Qwen30 and both Cosmos3 checkpoints. A least-squares decomposition on three templates (current scene layout, original coordinates, donor identity) gave a scene-layout component above zero in all 84 pooled conditions, whereas the original-coordinate component included zero in 205 of 210 label-order-specific rows; donor identity also contributed in some conditions.

Second, three-object scenes placed objects in three columns or in three cells of a 2×22 \times2 grid, with an exact cyclic layout exchange. All 24 predefined statistics were positive, with intervals above zero, for Granite, Qwen8 and Qwen32. With direct answers at α=.08\alpha= .08, the layout interaction in three columns was 5.64 [5.11, 6.18], 3.5 [3.07, 3.93] and 12.5 [11.2, 13.9], and real crops recomposed into the same layouts gave 3.52 [3.1, 3.94], 1.47 [1.14, 1.8] and 8.8 [7.52, 10.1]. These crops change color, instance and context together. Third, a held-out model of the local panels predicted each signed effect from whether the destination holds the edited object. Relative to an identity baseline, it reduced the cross-validated error of RR from 2.075 to 0.716 (Granite), 11.479 to 4.751 (Qwen8) and 20.047 to 5.441 (Qwen32), close to the error of training-condition means; this comparison was specified after the effects had been seen.

The prediction analysis also compares current object correspondence with a fixed-destination explanation. Both use the same number of parameters, with separate gains for each entry. They fit one scene layout identically but make different predictions when the objects exchange positions. In each of four folds, we fit one layout on 48 scenes and predict the other layout on 16 different scenes. All queries and conditions from a scene stay in the same fold. Holding out M0M0, the mean absolute error of direct-answer score selectivity RR falls from 4.146 to 0.772 in Granite, 22.627 to 4.847 in Qwen8 and 40.051 to 5.294 in Qwen32. Holding out M1M1 gives the same ordering, as do the later-only comparisons at both background strengths. These errors compare explanations within each model’s score scale. Layout exchange changes main and the unselected later background together while keeping the local donor and its destination fixed. The prediction comparison therefore supports current scene correspondence, not an independent effect of main on binding. This exploratory analysis tests the controlled left–right layout exchange, with its competing models specified after inspecting the original effects.

For the local transfers of the main paper’s Figure 2, the score interactions of Granite, Qwen8 and Qwen32 were 12.124 [11.771, 12.476], 6.345 [5.755, 6.935] and 31.994 [29.954, 34.035], and the generation interactions with direct answers at α=.08\alpha= .08 were 1.56 (SE 0.114, 32 of 32 scenes) for Granite, 1.80 (SE 0.134, 32 of 32) for Qwen32 and 0.385 (SE 0.0890, 26 of 32) for Qwen8. Among the added checkpoints, Qwen30’s main-entry color selectivity changed from 7.08 [5.88, 8.28] at the original destination to −4.74 [−5.66, −3.83] at the opposite one, with identity labels at full background strength; DeepStack’s later-entry color generation selectivity changed from 0.789 [0.724, 0.854] to −0.367 [−0.460, −0.274], and it reversed again under the exchanged layout (−0.398 [−0.494, −0.303] and 0.750 [0.681, 0.819]).

Whole-image source competition

A source-competition panel re-rendered the 64 two-object scenes with a gray cross between the objects and gave the main entry and all later entries independent sources, layouts and strengths. It asked identity questions (“What color is the circle?”) and relation questions about the side of the cross. For identity questions about the edited object, the role contrast D=Emain−ElaterD = E_{\mathrm{main}} - E_{\mathrm{later}} was negative in all 64 conditions for Granite (60 intervals below zero) and positive in all 64 for Qwen8 and Qwen32 (all intervals above zero); in the reference condition, its score values were −0.881 [−0.975, −0.786], 38.397 [36.539, 40.256] and 45.001 [44.622, 45.381]. Qwen’s later effect remained measurable at full strength, with intervals [0.137, 0.378] (Qwen8) and [0.372, 0.596] (Qwen32). The source four-cell interaction J=γub−γua−γab+γaaJ = \gamma_{ub} - \gamma_{ua} - \gamma_{ab} + \gamma_{aa} of the score was 1.1910 [0.9871, 1.3949], 0.1482 [−0.0002, 0.2966] and 1.2169 [0.9391, 1.4947]; in generation it was 0.2812 [0.1292, 0.4333] for Granite and 0 for both Qwen models. Their answers changed with the main source and never with the later source, so a zero JJ does not mean that neither entry is preferred. Whole-source conflicts on the controlled scenes complement the Qwen30 and DeepStack whole-source results of Appendix § 9.§ 4. With direct answers, generation ΔR\Delta R was −0.406 [−0.478, −0.334] for Granite, 1.000 for Qwen8 and 0.977 [0.950, 1.003] for Qwen32 with color, and −0.094 [−0.166, −0.022], 0.711 [0.614, 0.808] and 0.453 [0.356, 0.550] with stripes.

Dose curves on a dense subset

On an outcome-independent subset of 16 whole-image scenes, raw multipliers k∈{0,.25,.5,1,2,4}k \in\{0, .25, .5, 1, 2, 4\} traced dose curves in 12 backgrounds. The curves were not monotone: Qwen8’s main score effect was 39.171 at k=1k = 1 and 37.734 at k=4k = 4, Qwen32’s later effect rose from 0.315 at k=1k = 1 to 40.768 at k=4k = 4, and Granite’s main and later effects changed order between k=2k = 2 (6.689 against 8.280) and k=4k = 4 (7.769 against 7.195). Across backgrounds, the Qwen main score fell from k=1k = 1 to k=4k = 4 in 8 of 12, whereas the main generation mean did not change in 9 of 12 (Qwen8) and 11 of 12 (Qwen32), although individual answers still changed (5 of 377 and 8 of 380). The delivered magnitudes stayed within 0.99264 and 1.01284 of their canonical values. Matched magnitudes therefore compare entries at one dose rather than along a linear dose scale.

Later entries move scores without changing answers

On the photographs, Qwen8/32 later interventions change no target answer in the reference condition, although their later score selectivity is positive: 0.686 [0.523, 0.848] and 1.302 [1.056, 1.549]. A 32-photograph later-only panel measured shifts of the candidate margin. For Qwen32’s direct answers about the edited object in the reference background, the median absolute shift was 0.762, against a median starting margin of 19.504; over all pairs the median shift was 1.039% (Qwen8) and 2.349% (Qwen32) of the starting margin, against 77.849% for Granite, whose answers changed in 126 of 456 pairs. Rescaled to main’s native magnitude, Qwen’s later generation effects remain small (0.008 [−0.007, 0.023] and 0.055 [0.016, 0.093] under RSS), whereas main rescaled to later’s magnitude keeps selectivity of 0.153 and 0.246; magnitude accounts for part of the raw contrast but not for its direction. On these photographs, Qwen’s later features thus move its evidence without crossing the answer threshold. In this situation, a change of main quality can reveal their influence (main paper, § 5.3).

Conditional influence in other cells

The accuracy-invisible change of the main paper’s Figure 3b recurs in other cells of the calibrated experiment. For Qwen32 stripes at the mildest blur, IqualityI_{\mathrm{quality}} is 0.609 [0.500, 0.719] in generation and 18.642 in score under identity labels, and 0.469 [0.340, 0.598] and 15.969 under cyclic labels. For Qwen32 relative position at the mildest blur, all four accuracies are again 1 while later generation selectivity rises from 0.125 to 0.422 (Iquality=0.297I_{\mathrm{quality}} = 0.297 [0.161, 0.433]); its score IqualityI_{\mathrm{quality}} grows with blur (4.662, 9.460 and 9.801). For Qwen32 color at the mildest desaturation, the change appears only in scores (3.609 [2.820, 4.398]). The dependence is model- and task-specific: Qwen8’s stripe score IqualityI_{\mathrm{quality}} falls to −1.681 [−2.397, −0.965] at the strongest blur, and Granite’s relative-position score IqualityI_{\mathrm{quality}} is negative at all three levels (−0.253, −0.621 and −0.752).

In the cell of Figure 3b, the target’s starting margin barely changes (−19.567 to −18.522) while its target effect rises from 8.819 to 27.627, so the change is not a starting-margin artifact; for Qwen32 color, by contrast, the starting margin shrinks from −23.73 to −10.03 as the target effect grows from 0.47 to 8.91, so margin dependence must be judged per cell. On shared units, conflict and recovery can move together: at the strongest desaturation, Qwen32’s later color selectivity rises from 0.000 to 0.422 (Iquality=0.422I_{\mathrm{quality}} = 0.422 [0.276, 0.567]) while clean later features restore 0.250 [0.148, 0.352] of the color loss; for Qwen32 text, both stay near zero.

Recovery in other cells

Further cells extend the recovery patterns of the main paper’s § 5.4. On relative position at the strongest blur, Qwen32 recovered 0.188 [0.099, 0.276] of a 0.203 loss (Irec=0.109I_{\mathrm{rec}} = 0.109 [0.034, 0.185]) and Qwen8 0.172 [0.054, 0.290] of a 0.250 loss, while Qwen8 recovered color only partly and non-monotonically (GG of 0.047, 0.000 and 0.125 at losses of 0.047, 0.125 and 0.281). On GQA, Qwen accuracy is not purely main-limited as it is on DocVQA: Qwen32’s attribute recovery rose with loss (0.060, 0.146 and 0.224 at losses of 0.122, 0.219 and 0.302; Irec=0.190I_{\mathrm{rec}} = 0.190 [0.123, 0.257] at the strongest level), and Qwen8’s spatial interactions excluded zero at every level.

The model-specific capability-calibrated levels give another pair of similar losses with different recovery: on GQA spatial questions, Qwen8 lost 0.234 [0.169, 0.300] at a 7-pixel long side and recovered 0.008 [−0.023, 0.039], whereas Qwen32 lost 0.263 [0.195, 0.331] at 5 pixels and recovered 0.185 [0.127, 0.243]; the levels differ, and these rows form a separate estimand. The equal-loss comparison of Qwen32 and Qwen8 on stripes holds for direct answers only: under identity labels the recovery difference is 0.500 [0.363, 0.637], but Qwen8’s loss is 0.391 (paired loss difference 0.109 [−0.041, 0.259]). In the fixed-contrast experiment, Qwen’s direct and identity color cells saturate, whereas cyclic labels expose recovery of 0.109 [0.034, 0.185] (Qwen8) and 0.297 [0.207, 0.387] (Qwen32); DeepStack’s cyclic IrecI_{\mathrm{rec}} of 0.250 [0.113, 0.387] arises because clean later features lower its accuracy when main is clean (0.828, against 0.969 with degraded later).

Supplementary related work

Visual interfaces and evaluation controls. LLaVA projects visual-grid features into language embeddings [26]. BLIP-2 and Honeybee study learned-query interfaces and locality-preserving compression [5, 23]. Flamingo offers repeated cross-attention access, while Cambrian-1 and DeepStack supply spatial visual features at multiple language-model depths [2, 30, 44]. Qwen3-VL instead projects intermediate encoder features into early decoder layers [37]. Cosmos3 retains this interface after continued training [32]. Granite Vision describes its design as “a variant of DeepStack” in which visual features are added to the language-model hidden states at multiple layers [16]. We preserve these native differences when comparing entries.

CLEVR, Visual Genome, and GQA support controlled or annotated object questions [15, 18, 22]. Compositional tests expose failures to preserve object–attribute assignments, while spatial benchmarks vary relations or object positions [19, 25, 43, 48]. Our paired queries use the same visual intervention to separate the edited object’s response from an unchanged object’s response.

Intervention interpretation. Activation patching and interchange interventions test causal output effects or correspondence with a proposed computation [10, 29, 46]. Corruption, grouping, metric, direction, and redundancy affect their interpretation [13, 27, 49]. We specify those choices and retain magnitude controls. We also separate candidate scores from generated answers: increased target evidence can stay below an answer-selection threshold [47]. In a text-only setting, language models likewise linearly encode that a math or code question is structurally answerable, but often still answer it [8].

Additional visual-reliance controls. MMVP tests difficult visual distinctions associated with similar CLIP representations [45]; VALSE and SugarCrepe use language-controlled contrasts to limit textual shortcuts [14, 36]. FiVL compares answer-region dependence with same-area random masks [1]. These controls motivate retaining native capability and target/control components, without identifying the same internal intervention. Visual concepts can emerge in frozen text models [41]; image-token editing and transfers further distinguish available information from its use [6, 17]. POPE balances yes/no object-presence questions [24]. Our attribute-generation and continuation-score endpoints have different answer events, conditioned on native entry, spatial correspondence, and receiving state.

Data use and availability

We reuse existing datasets and checkpoints, with no new participants or human judgments. DocVQA was obtained through its official access process. Recolored photographs are counterfactual stimuli. The photograph in Figure 1 is River Brent at Brentford by Maxwell Hamilton (https://www.flickr.com/photos/mualphachi/3736968562/), licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/). We cropped it and recolored the bus.

The data folder in the code repository holds the complete estimates of Appendices § 14, § 15 and § 16 as twelve CSV files, with a README that defines every column. For the two quality experiments, the files give the unrounded estimates from the same analysis that produced the paper’s tables, with descriptive 95% intervals; a zero standard error is flagged explicitly in the fixed-contrast files and shown as an empty interval in the calibrated files. Nothing is re-estimated. Rows identify model, task, readout, response kind, degradation level, metric and invalid-answer policy, so any estimate quoted in the paper can be traced to one row. The three coding files are recomputed from the per-question observations with the paired bootstrap of Appendix § 16.§ 1, and their rows identify model, task, image set and configuration. The repository is at https://github.com/yucheng-du/interpreting-deepstack-style-vlms.

References

  1. [1]Estelle Aflalo, Gabriela Ben Melech Stan, Tiep Le, Man Luo, Shachar Rosenman, Sayak Paul, Shao-Yen Tseng, and Vasudev Lal. FiVL: A framework for improved vision-language alignment through the lens of training, evaluation and explainability. arXiv preprint arXiv:2412.14672, 2024.arxiv.org/abs/2412.14672
  2. [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: A visual language model for few-shot learning. In Advances in Neural Information Processing Systems, pages 23716–23736, 2022.arxiv.org/abs/2204.14198
  3. [3]Rim Assouel, Declan Campbell, Yoshua Bengio, and Taylor Webb. Visual symbolic mechanisms: Emergent symbol processing in vision language models. In International Conference on Learning Representations, 2026. 2arxiv.org/abs/2506.15871
  4. [4]Daniel Becking, Paul Haase, Heiner Kirchhoffer, Karsten Müller, Wojciech Samek, and Detlev Marpe. NNCodec: An open source software implementation of the neural network coding ISO/IEC standard. In ICML 2023 Workshop Neural Compression: From Information Theory to Applications, 2023. 7
  5. [5]Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal LLM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13817–13827, 2024.arxiv.org/abs/2312.06742
  6. [6]Ido Cohen, Daniela Gottesman, Mor Geva, and Raja Giryes. Performance gap in entity knowledge extraction across modalities in vision language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29095–29108. Association for Computational Linguistics, 2025.arxiv.org/abs/2412.14133
  7. [7]Kelly Cui, Nikhil Prakash, Shoval Messica, Ayush Raina, David Bau, Antonio Torralba, and Tamar Rott Shaham. The dual mechanisms of spatial variable binding in vision-language models. arXiv preprint arXiv:2603.22278, 2026. 2, 3arxiv.org/abs/2603.22278
  8. [8]Yucheng Du and Xiyang Hu. Recognition–refusal misalignment in LLMs: Why models answer structurally unanswerable questions. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 24766–24794. Association for Computational Linguistics, 2026.DOI
  9. [9]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, 2024.arxiv.org/abs/2404.12390
  10. [10]Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, 2021.arxiv.org/abs/2106.02997
  11. [11]Rodela Ghosh, Aviral Gupta, and Guangjing Wang. Which source wins? Task-dependent reliance in vision-language models. arXiv preprint arXiv:2608.17205, 2026. 3arxiv.org/abs/2608.17205
  12. [12]Reza Heidari, Hamed R. Tavakoli, and Juho Kannala. Compressing AI traffic: Standardized neural network coding of visual-token representations in split vision-language inference. arXiv preprint arXiv:2609.01200, 2026. 2, 7arxiv.org/abs/2609.01200
  13. [13]Stefan Heიმersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255, 2024.arxiv.org/abs/2404.15255
  14. [14]Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. SugarCrepe: Fixing hackable benchmarks for vision-language compositionality. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2306.14610
  15. [15]Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019. 4arxiv.org/abs/1902.09506
  16. [16]IBM Granite Vision Team. Granite 4.1 Vision. Model card, 2026. Accessed 2026-10-28. 1, 2, 4
  17. [17]Nick Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. In International Conference on Learning Representations, 2025.arxiv.org/abs/2410.02762
  18. [18]Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2910, 2017.arxiv.org/abs/1612.06890
  19. [19]Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s “up” with vision-language models? Investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9161–9175. Association for Computational Linguistics, 2023.
  20. [20]Raphi Kang, Hongqiao Chen, Georgia Gkioxari, and Pietro Perona. Linear mechanisms for spatiotemporal reasoning in vision language models. In International Conference on Learning Representations, 2026. 2arxiv.org/abs/2601.12626
  21. [21]Heiner Kirchhoffer, Paul Haase, Wojciech Samek, et al. Overview of the neural network compression and representation (NNR) standard. IEEE Transactions on Circuits and Systems for Video Technology, 32(5):3203–3216, 2022.DOI
  22. [22]Ranjay Krishna, Yuke Zhu, Oliver Groth, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017. 4arxiv.org/abs/1602.07332
  23. [23]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, pages 19730–19742. PMLR, 2023.arxiv.org/abs/2301.12597
  24. [24]Yifan Li, Yifan Du, Kun Zhou, Jinping Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305. Association for Computational Linguistics, 2023.arxiv.org/abs/2305.10355
  25. [25]Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023.
  26. [26]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, pages 34892–34916, 2023.
  27. [27]Aleksandar Makelov, Georg Lange, Atticus Geiger, and Neel Nanda. Is this the subspace you are looking for? An interpretability illusion for subspace activation patching. In International Conference on Learning Representations, 2024.arxiv.org/abs/2311.17030
  28. [28]Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2200–2209, 2021.arxiv.org/abs/2007.00398
  29. [29]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2202.05262
  30. [30]Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. DeepStack: Deeply stacking visual tokens is surprisingly simple and effective for LMMs. In Advances in Neural Information Processing Systems, 2024.
  31. [31]Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. In International Conference on Learning Representations, 2025.
  32. [32]NVIDIA. Cosmos 3: Omnimodal world models for physical AI. arXiv preprint arXiv:2606.02800, 2026.
  33. [33]NVIDIA. Cosmos3-Nano. Model card, 2026. Accessed 2026-10-28.
  34. [34]NVIDIA. Cosmos3-Super. Model card, 2026. Accessed 2026-10-28.
  35. [35]Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision-language mechanistic interpretability: A causal tracing tool for BLIP. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 2856–2861, 2023.
  36. [36]Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Lacer Calixto, and Albert Gatt. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8253–8280. Association for Computational Linguistics, 2022.
  37. [37]Qwen Team. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025.
  38. [38]Farrukh Rahman. What does a temporal benchmark score measure? Decomposing channel use in video VLM evaluation. arXiv preprint arXiv:2607.12304, 2026.
  39. [39]Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, and Constanza Fierro. Pathways of visual information flow in vision-language models. In Advances in Neural Information Processing Systems, 2026.
  40. [40]Timothy Schaumlöffel, Martina G. Vilas, and Gemma Roig. Mechanisms of object localization in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 31356–31365, 2026.
  41. [41]Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 2862–2867, 2023.
  42. [42]Jonathan Steinberg and Oren Gal. Where vision becomes text: Locating the OCR routing bottleneck in vision-language models. arXiv preprint arXiv:2602.22918, 2026.arxiv.org/abs/2602.22918
  43. [43]Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022.
  44. [44]Shengbang Tong, Ellis Brown, Penghao Wu, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In Advances in Neural Information Processing Systems, pages 87310–87356, 2024.
  45. [45]Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024.
  46. [46]Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, 2020.
  47. [47]Zeyu Wang and Xinming Xu. Knowing isn’t always saying: When do spatial encodings reach answers in vision-language models? In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 52683–52705. Association for Computational Linguistics, 2026.arxiv.org/abs/2608.22916
  48. [48]Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023.
  49. [49]Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In International Conference on Learning Representations, 2024.
  50. [50]Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-modal information flow in multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19781–19791, 2025.

Paper details

vlnc:W_xzydpg4y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Codex, Claude Code

View statement
Looking for related papers…
Contents