betaFeedback

WorksComputer Science

Layer-Matched Sparse Attention for Low-Redundancy 3D Reconstruction

Abstract

VGGT reconstructs cameras and geometry from unposed images in one pass, but its global attention grows quadratically with the number of views. Training-free accelerators thin this attention and work well on redundant video, but lose pose accuracy when views share little geometry. This paper traces these losses to two kinds of cross-view information that a uniform sparse approximation can fail to preserve together: diffuse layers aggregate distributed context, whereas matching layers depend on query-specific interactions that carry little attention mass. A per-layer statistic measures the log-mass missed by a mean token. Its row-free proxy is the logit variance, and it selects the estimator for each layer. Diffuse layers use shared, importance-sampled hub keys, with log-weights that make the softmax numerator and denominator unbiased. Matching layers stay exact: every aggressive thinning tested cost accuracy. At equal cost, the selected layers outperform the four shifted windows tested. On 13 unseen low-redundancy scenes with 32 views, LMSA stays within 1.4 AUC@5 of dense attention, compared with 10.3 for AVGGT and 4.5 for SAF3R; its difference from contemporaneous GPA-VGGT is not significant, and only its gains over AVGGT pass both scene-level tests. With 100 views, it stays within 1.9 AUC@5 while its aggregator runs 2.0×2.0\times faster. Used post hoc to choose the layers GPA-VGGT protects, the statistic improves it by 1.2 AUC@5 for 5% more FLOPs. Applied untuned to π3π^3, it beats AVGGT but remains further from dense accuracy.

Introduction

Feed-forward reconstruction transformers [26, 44, 45, 46, 48, 52] predict cameras, depth and point maps for a set of unposed images in one forward pass. VGGT [44] alternates attention within each frame with global attention over the tokens of all NN views, and the global part grows as N2N^{2}. A fast-growing family of training-free accelerators therefore thins global attention [12, 30, 36, 40, 41, 43, 47]. Many rely on the premise that cross-view information is redundant, so that most cross-view keys can be dropped, merged or summarised with little effect on geometry. The premise often holds for overlapping video but breaks when views are few and far apart, a setting this paper calls low-redundancy and measures by ρ\rho, the mean number of views that see a point. The resulting failure is severe: on ETH3D courtyard (ρ≈7.7\rho\approx7.7), AVGGT [40], keeping one key per 3×33 \times3 patch cell (σ=9\sigma=9), lowers the pose AUC@5 of a frozen VGGT-1B from 58.3 to 8.1 and its AUC@30 from 90.9 to 34.8, averaged over five frame subsets.

AVGGT [40] observed that VGGT’s early global layers form no cross-view correspondences, its middle layers align the views and its late layers only refine. Thinning one group at a time (Table 1) shows that these roles require different cross-view information. The middle layers give the clearest example. Across three low-redundancy development scenes, removing the model’s small correspondence-key sets from layers 10–16 (on TUM fr1_room, 30 of 33,312 keys per query) costs 17.7–49.2 AUC@5. Removing as many of the heaviest other keys costs 6.0–23.0, even though they carry 2.3–3.4× more attention mass; removing random keys costs nothing. The keys with the highest query-specific lift cost the most, although only 13% lie in correspondence windows (Fig. 2c). A key can therefore carry little attention mass yet be critical to geometric reconstruction.

Scene (ρ\rho)E (0–8)B (10–16)L (9, 17–23)all
TUM fr1_room (4.3)5.611.7−0.413.3
ETH3D courtyard (7.7)33.128.55.450.2
ETH3D office (5.5)1.56.03.512.3
TUM fr3_long_office (15.4)0.54.414.822.1
7-Scenes office (13.0)−1.10.81.61.8

Table 1. Three groups fail for different reasons. AUC@5 lost by frozen VGGT-1B when one group of global layers is thinned: AVGGT-style and the rest stay dense: E, layers 0–8 made frame-local; B, layers 10–16 on a σ=9\sigma=9 lattice with a mean token; L, layers 9 and 17–23 likewise. Mean of 5 frame subsets, N=32N=32 (N=22N=22 for ETH3D office); ρ\rho as in § 3.1; bold: largest single-group loss.

The other layers fail in different ways. Making early layers 0–8 frame-local, as AVGGT does, costs 33.1 AUC@5 on courtyard because these layers carry distributed cross-view context. Layers 9 and 17–23 are diffuse again, but thinning them accounts for 14.8 of the 22.1 points AVGGT loses on the redundant TUM fr3_long_office scene: its mean token under-counts the dropped mass (Section 3.4).

Attention approximation is therefore an estimation problem with two regimes. A diffuse layer needs an accurate estimate of a sum over many comparable terms, whereas a matching layer needs query-specific terms that a small shared sample does not contain. One per-layer statistic separates them: the log-mass Γ\Gamma that a mean token misses from a query’s softmax denominator is about half the logit variance s2s^{2} when the logits spread little (Prop. 1), and s2s^{2} needs only the key covariance, not attention rows. A cut-off of 1.5 nats (s2≈3s^{2} \approx3), chosen on the development scenes and fixed before any test run, labels layers 11–16 as matching; that these layers need exact attention, not just a better sample, is an empirical finding that § 3.3 tests.

LMSA (layer-matched sparse attention, Figure 1) gives each regime its own estimator. Diffuse layers attend to their own frame, the special tokens and a few hundred shared hub keys, importance-sampled per input from the attention of 512 pilot queries, with log-weights that make the softmax numerator and denominator unbiased. Their per-query cost hardly grows with NN, and they remove both the early failure and the late one, which is consistent with the mean token’s under-weighting of the dropped keys. Matching layers keep exact attention: every aggressive way of thinning them that was tested, from truncated lattices with routed correspondence blocks to merging most keys, cost accuracy once the other layers were sparse (§ 3.3); the routed design is kept as a faster variant, LMSA-fast.

Figure 1. LMSA on a frozen VGGT-1B. (a) NN unposed views (ETH3D courtyard [35]), each PP patch and 5 special tokens. (b) Frame attention (grey) is unchanged. The offline statistic Γˉℓ\bar{\Gamma}_{\ell} (black bars, once per model, Section 3.3) sets each global layer’s estimator: above 1.5 (dotted), matching layers (blue) keep exact attention; diffuse layers (orange) use 256 or 1024 shared hubs. Block width: keys per query at N=32N=32, log scale. (c) Cameras, point cloud and depth predicted by LMSA. (d) A diffuse-layer query (black dot) attends to its own view, all special tokens and mm hubs drawn per input (Section 3.4); hub counts to scale, positions illustrative. (e) In a matching layer every token is a key (magenta: true correspondences). (f) Exact cost (§ 2.8); pale: the same layers at dense cost.

On 13 unseen low-redundancy scenes at N=32N=32, named before any test run, LMSA stays within 1.4 AUC@5 of dense attention (95% CI [0.7, 2.0]), the lowest mean loss of the training-free methods tested, against 10.3 to 18.9 for AVGGT (σ=4\sigma=4, 9), which is 7–18% faster, and 4.5 for SAF3R; with 100 views it stays within 1.9 AUC@5 while its aggregator runs 2.0× faster. At equal cost, the statistic’s six exact layers beat the four shifted windows of six tested (§ 3.3). The contemporaneous GPA-VGGT, as fast, loses 2.4, so the margin of LMSA (1.0, CI [−0.3, 2.5]) is not significant, and on 8 unseen redundant scenes GPA-VGGT loses 0.1 against 0.7 (§ 3.2); protecting the statistic’s layers 11–16 instead of 13–16 improves GPA-VGGT by 1.2 AUC@5 (CI [0.4, 2.1], post hoc).

The paper makes three contributions:

  1. An interpretable criterion for choosing the attention estimator per layer: controlled interventions show that reconstruction needs both distributed context and query-specific interactions that carry little attention mass, and the mean-token log-mass deficit, with a row-free logit-variance proxy, separates the layers that need each; simpler concentration statistics agree (§ 2.2 and 3.3).

  1. LMSA: probability-corrected shared hubs in diffuse layers and exact attention in matching layers, with no training and no known poses (Sections 3.4 and § 2.7).

  1. Validation: on 21 pre-named test scenes LMSA has the lowest mean loss of the training-free methods tested; at equal cost its exact layers beat the four shifted windows tested, the statistic also improves GPA-VGGT on the same backbone (post hoc), and a registered test on NRGBD, Sintel and long video confirms a variant with more hubs (§ 3).

Related Work

Feed-forward 3D reconstruction. DuSt3R [46] and MAsT3R [26] regress point maps from image pairs; VGGT [44], π3\pi^{3} [48] and Fast3R [52] process many views jointly with global attention, and CUT3R [45] with a persistent state. Bratulić et al. [5] show that correspondence and epipolar structure emerge in the intermediate-layer attention of VGGT and related models, the behaviour that the matching layers of § 2.7 rely on. LMSA keeps these models frozen and changes only which keys their global attention reads.

Accelerating global attention in 3D transformers. FastVGGT [36], HTTM [47], Co-Me [9] and, with fine-tuning, LiteVGGT [38] merge tokens; QuantVGGT [16] quantizes the model, which is complementary. AVGG T [40] analyses VGGT’s global attention layer by layer and finds that early global layers form no cross-view correspondences, middle layers align the views and late layers only refine; it makes the global layers below a hand-set boundary frame-local and, in later layers, keeps one key per σ\sigma-cell, the reference frame and a count-biased mean token. LMSA does not claim the role split, which AVGGT and RegimeVGGT establish; it asks which estimator each layer’s information needs, choosing between a shared sample and exact attention by a per-layer statistic, and replaces the frame-local early layers, which fail on low-redundancy input (Table 1), and the biased mean token by probability-corrected shared hubs. Block-sparse attention [43] keeps key blocks chosen by pooled scores, and HeSS [23] redistributes its budget among the heads of each layer by a gradient-based sensitivity score; RegimeVGGT [30] analyses VGGT’s layer regimes and merges tokens and subsamples keys and values on a layer-wise schedule, choosing how much to compress rather than which estimator to use, and Good Token Hunting [54] selects frames and prunes tokens per layer. Other methods sparsify whole frames [27], as does the contemporaneous BlazeVGGT [53], partition the views into chunks [33], or train and fine-tune [20, 34, 49]. The contemporaneous GPA-VGGT [41] adds to the early frame attention a shared cross-frame set of per-head cluster summaries with a ++ log(size) bias, as in the proportional attention of ToMe [4], and merges keys and values later; its merging operator is used in the merged variants of § 3.3. The hubs of LMSA are an importance sample with probability-correcting log-weights rather than summaries, and the same estimator serves the late layers, where it removes an estimation bias (Section 3.4). SAF3R [12] assigns each head one of several sparse patterns from offline calibration and, for correspondence heads, keeps the top-KK keys per frame by feature similarity.

Sparse, sampled and summarised attention. HyperAttention [18], Scatterbrain [7] and Support Basis [2] (beyond the bounded-entry regime of [3]) keep large entries exactly and estimate the rest by uniform samples, low-rank factors or polynomials, MagicPIG [10] estimates attention by hashed sampling, and vAttention [14] adds a uniform sample of the remaining keys to exact top-kk keys; Skeinformer [8] samples key columns from a pilot, the mechanism closest to the hubs of LMSA. FMMformer [32], Fast Multipole Attention [22], Multipole Semantic Attention [31] and Multipole Attention [19] treat distant or dissimilar keys coarsely (low-rank, hierarchical or by cluster centroids), the last also picking keys for exact attention by centroid. In LLMs and video models the estimator is chosen per head, by retrieval heads [42, 51], offline search or a divergence statistic [21, 25], sampled queries [56] or online profiling [50], and Kascade [15] computes exact top-kk keys only in anchor layers chosen on a development set. LMSA makes such a choice per layer from a closed-form statistic calibrated offline on development scenes, without gradients or a search over layer windows, and pairs it with a probability-corrected estimator.

Statistical physics and hubness. The random-energy model [13] has a freezing transition at which a partition function becomes dominated by its extreme terms; it has been mapped onto attention at initialisation [17] and onto retrieval in dense associative memories [29], and LMSA uses it at inference to motivate a choice of estimator. Hubs that attract many queries are a known failure of similarity retrieval, corrected in text–image matching by the inverted softmax [28]; makes the global layers below a hand-set boundary frame-local and, in later layers, keeps one key per σ\sigma-cell, the reference frame and a count-biased mean token.

the popularity–specificity split of § 2.7 follows the same factorisation.

Method

Setting

VGGT-1B [44] cuts each of NN images into PP patch tokens and adds five special tokens (one camera, four register), so a frame has T=P+5T=P+5 tokens (T=1041T=1041 for 392×518 images). It alternates 24 frame blocks with 24 global blocks, numbered 0–23, of 16 heads of width dh=64d_h=64; LMSA changes only the global blocks. Fix one query token tt with query vector qtq_t, one head and one layer. A key uu with key vector kuk_u and value vuv_u has logit lu=⟨qt,ku⟩/dhl_u=\langle q_t,k_u\rangle/\sqrt{d_h} and weight wu=eluw_u=e^{l_u}, and dense global attention over the set U\mathcal{U} of all NTNT tokens returns

ot=∑u∈Uwuvu∑u∈Uwu.(1)o_t=\frac{\sum_{u\in\mathcal{U}}w_uv_u}{\sum_{u\in\mathcal{U}}w_u}. \tag*{(1)}

ZZ: partition function

A sparse layer that misestimates ZZ rescales every kept key, even correctly chosen ones. Every sparse layer in this paper, of LMSA and of the baselines alike, keeps an exact key set K(t)\mathcal{K}(t) and replaces the dropped set D(t)=U∖K(t)D(t)=\mathcal{U}\setminus\mathcal{K}(t) by a few stand-in keys W(t)\mathcal{W}(t), each with a log-weight bub_u:

o^t=∑u∈Kwuvu+∑u∈Webuwuvu∑u∈Kwu+∑u∈Webuwu.(2)\hat{o}_t=\frac{\sum_{u\in\mathcal{K}}w_uv_u+\sum_{u\in\mathcal{W}}e^{b_u}w_uv_u}{\sum_{u\in\mathcal{K}}w_u+\sum_{u\in\mathcal{W}}e^{b_u}w_u}. \tag*{(2)}

The factor ebue^{b_u} states how many dropped keys the stand-in uu represents. AVGGT’s mean token, for example, is one stand-in with the mean key kˉD\bar{k}_D and mean value vˉD\bar{v}_D over DD and b=ln⁡∣D∣b=\ln|D|; truncated attention has W=∅\mathcal{W}=\varnothing; and the hubs of LMSA are sampled keys with importance weights (Section 3.4). Costs are counted in units: one unit is one frame’s TT keys for every query, so a dense global layer costs NN units.

Why one sparse pattern fails

In the terms of Eq. (2), the three failures of Table 1 are three different estimation errors. In the early layers AVGGT keeps only the own frame, so the cross-view part of a diffuse sum is missing; adding back the special tokens of all frames still loses 24.2 AUC@5 on courtyard. In the middle layers a small set of correspondence keys matters for pose far beyond its attention mass, and the lattice misses them. In layers 9 and 17–23 the sum is diffuse again, but the mean token under-counts it; the size of this under-count is the statistic that the method builds on.

Figure 2. One statistic assigns the layer roles; attention mass is not importance. (a) Log-mass deficit Γ\Gamma of a count-biased mean token against the row-free logit variance s2s^2 (Eqs. (3) and (5)), for 1.15M (query, layer, head) triples from layers 9–23 of VGGT-1B on 8 scenes (median, IQR); it follows s2/2s^2/2 up to s2≈10s^2 \approx10 and falls below it at larger variance (grey), as the Gaussian REM, or light-tailed or correlated logits, would. (b) Offline layer statistic Γˉℓ\bar{\Gamma}_{\ell} (median head, mean of the 5 development scenes), coloured by the estimator LMSA assigns; dotted: the empirical cut-off (1.5 nats). (c) AUC@5 lost when one key set is removed from layers 10–16 only (three low-redundancy development scenes, 5 seeds; Table S15); in brackets, the attention mass removed.

One statistic: the mean-token deficit

For a query tt and its dropped set DD, the mean-token deficit Γt\Gamma_t compares the true mass of the dropped keys with the mass that a count-biased mean token gives them:

Γt=ln⁡∑u∈Dwu⏟true log-mass of D−(ln⁡∣D∣+lˉD)⏟log-weight of the mean token.(3)\Gamma_t = \underbrace{\ln\sum_{u\in D} w_u}_{\text{true log-mass of }D} - \underbrace{\left(\ln|D| + \bar{l}_D\right)}_{\text{log-weight of the mean token}} . \tag*{(3)}

The mean token’s log-weight adds the count bias ln⁡∣D∣\ln|D| to the logit of the mean key, lˉD=⟨qt,kˉD⟩/dh\bar{l}_D = \langle q_t,\bar{k}_D\rangle/\sqrt{d_h}. Because logits are linear in the key, this is also the mean logit 1∣D∣∑u∈Dlu\frac{1}{|D|}\sum_{u\in D} l_u. Γt=0\Gamma_t=0 means that the mean token reproduces the dropped mass; Γt>0\Gamma_t>0 means that it gives only e−Γte^{-\Gamma_t} of it.

Proposition 1 (Mean-token deficit). Let pτ(u)∝eτlup_{\tau}(u) \propto e^{\tau l_u} on DD for τ∈[0,1]\tau\in[0,1]. (i) Exact: for some ξ∈(0,1)\xi\in(0,1), Γt=12Var⁡pξ(l)≥0\Gamma_t = \frac{1}{2}\operatorname{Var}_{p_{\xi}}(l) \ge0, with equality iff the logits are constant on DD. (ii) Small spread: with ΣK\Sigma_K the dh×dhd_h \times d_h covariance of the keys in DD,

Γt≈12st2,st2=Var⁡p0(l)=qtTΣKqtdh.(4)\Gamma_t \approx\frac{1}{2}s_t^2,\qquad s_t^2 = \operatorname{Var}_{p_0}(l) = \frac{q_t^{\mathsf T}\Sigma_K q_t}{d_h}. \tag*{(4)}

Here p0p_0 is uniform on DD; the proof (App. § 7.1.1) writes Γ\Gamma as a Taylor remainder. The useful consequence is Eq. (4): the logit variance s2s^2 needs one key covariance per layer and head and O(dh2)O(d_h^2) work per query and no attention row, hence row-free.

Random-energy reading. A classical model shows where Γ≈s2/2\Gamma\approx s^2/2 breaks.

Assumption 1 (Random-energy model, REM [13]). The n=∣D∣n=|D| dropped logits of a query behave like independent Gaussians with variance s2s^2.

For large nn, Assumption 1 gives two phases separated by the freezing variance 2ln⁡n2\ln n:

Γ≈{12s2,s2<2ln⁡n(high temp.),2s2ln⁡n−ln⁡n,s2≥2ln⁡n(frozen).(5)\Gamma\approx \begin{cases} \frac{1}{2}s^2, & s^2 < 2\ln n \quad\text{(high temp.)},\\ \sqrt{2s^2\ln n}-\ln n, & s^2 \ge2\ln n \quad\text{(frozen)}. \end{cases} \tag*{(5)}

In the frozen phase the largest logits carry ZZ, and the correction s2/2s^2/2 overshoots by (s−2ln⁡n)2/2(s-\sqrt{2\ln n})^2/2. On VGGT (Fig. 2a), measured Γ\Gamma stays within 0.3 nats of s2/2s^2/2 up to s2≈10(rankcorrelation0.986)s^2 \approx10 (rank correlation 0.986), consistent with Eq. (4), and then bends below it, earlier than the REM’s 2ln⁡n≈20.82\ln n \approx20.8, as correlated or light-tailed logits would; the REM thus motivates the cut-off rather than setting it. Below freezing, an importance sample estimates ZZ and the numerator of Eq. (1) without bias, while a mean token is short by about s2/2s^2/2 nats; above it, a shared sample of fixed size misses the extreme keys. Proposition 1 is exact, Eq. (4) approximate, and the cut-off below empirical, validated in Section 4.3: Γ\Gamma measures how badly a mean token fails; that high-Γ\Gamma layers need exact attention rather than a better sample is empirical.

Selector. For each global layer ℓ\ell, the selector computes Γt\Gamma_t for 512 random patch queries per pilot scene and frame subset, with DD set to all patch keys of the other frames, and takes the median over heads and then the mean over pilot queries and pilot scenes to obtain the pilot statistic Γˉℓ\bar{\Gamma}_{\ell}, a layer-level aggregate of the per-query Γt\Gamma_t. Layer ℓ\ell is a matching layer if Γˉℓ≥1.5\bar{\Gamma}_{\ell}\ge1.5 nats and keeps exact attention (§ 2.7); otherwise it is a diffuse layer and uses hubs (Section 3.4). The cut-off was chosen on the development scenes of VGGT after the factorial of Table 1 and fixed before any composed or test run. Γˉℓ\bar{\Gamma}_{\ell} is 1.83–9.21 in layers 11–16 and at most 1.00 elsewhere (Fig. 2b), so the matching set is 11–16, on each of the five development scenes. The row-free form gives the same split: among layers 9–23, the median-head s2s^2 is at least 3 (= 2×1.52\times1.5) in exactly layers 11–16 and at most 1.6 elsewhere. The cut-off (1.5 nats, s2≈3s^2\approx3) lies well below freezing: only layer 14 is frozen, and what the matching layers need are query-specific keys rather than heavy ones (§ 2.7). The selector runs once per model, offline, on the development scenes, and is applied to a new model without retuning (on π3\pi^3 it misses its target, § 3.5); at inference only the hub pilot runs.

Diffuse layers: importance-sampled hubs

In a diffuse layer every query receives the same shared set of stand-in keys, drawn once per layer and input in three steps. Hub pilot (per input): 512 random patch queries R\mathcal{R} compute their full attention and estimate each key’s column mass μ^j=1∣R∣∑t∈RAˉ(t,j)\hat{\mu}_j = \frac{1}{|\mathcal{R}|}\sum_{t\in\mathcal{R}}\bar{A}(t,j), where Aˉ\bar{A} is the head-averaged attention. Sample: over the patch tokens CC, the proposal is pj∝μ^j+ϵ/∣C∣p_j \propto\hat{\mu}_j + \epsilon/|C|, @page 4/29 starts_mid=1 ends_mid=1

Figure 2 (PDF p. 4). One statistic assigns the layer roles; attention mass is not importance. (a) Log-mass deficit Γ\Gamma of a count-biased mean token against the row-free logit variance s2s^2 (Eqs. (3) and (5)), for 1.15M (query, layer, head) triples from layers 9–23 of VGGT-1B on 8 scenes (median, IQR); it follows s2/2s^2/2 up to s2≈10s^2 \approx10 and falls below it at larger variance (grey), as the Gaussian REM, or light-tailed or correlated logits, would. (b) Offline layer statistic Γˉℓ\bar{\Gamma}_{\ell} (median head, mean of the 5 development scenes), coloured by the estimator LMSA assigns; dotted: the empirical cut-off (1.5 nats). (c) AUC@5 lost when one key set is removed from layers 10–16 only (three low-redundancy development scenes, 5 seeds; Table S15); in brackets, the attention mass removed.

size of this under-count is the statistic that the method builds on.

One statistic: the mean-token deficit

For a query tt and its dropped set DD, the mean-token deficit Γt\Gamma_t compares the true mass of the dropped keys with the mass that a count-biased mean token gives them:

Γt=ln⁡∑u∈Dwu⏟true log-mass of D−(ln⁡∣D∣+lˉD)⏟log-weight of the mean token.(6)\Gamma_t = \underbrace{\ln\sum_{u\in D} w_u}_{\text{true log-mass of }D} - \underbrace{\left(\ln|D| + \bar{l}_D\right)}_{\text{log-weight of the mean token}} . \tag*{(6)}

The mean token’s log-weight adds the count bias ln⁡∣D∣\ln|D| to the logit of the mean key, lˉD=⟨qt,kˉD⟩/dh\bar{l}_D = \langle q_t,\bar{k}_D\rangle/\sqrt{d_h}. Because logits are linear in the key, this is also the mean logit 1∣D∣∑u∈Dlu\frac{1}{|D|}\sum_{u\in D} l_u. Γt=0\Gamma_t=0 means that the mean token reproduces the dropped mass; Γt>0\Gamma_t>0 means that it gives only e−Γte^{-\Gamma_t} of it.

Proposition 1 (Mean-token deficit). Let pτ(u)∝eτlup_{\tau}(u) \propto e^{\tau l_u} on DD for τ∈[0,1]\tau\in[0,1]. (i) Exact: for some ξ∈(0,1)\xi\in(0,1), Γt=12Var⁡pξ(l)≥0\Gamma_t = \frac{1}{2}\operatorname{Var}_{p_{\xi}}(l) \ge0, with equality iff the logits are constant on DD. (ii) Small spread: with ΣK\Sigma_K the dh×dhd_h \times d_h covariance of the keys in DD,

Γt≈12st2,st2=Var⁡p0(l)=qtTΣKqtdh.(7)\Gamma_t \approx\frac{1}{2}s_t^2,\qquad s_t^2 = \operatorname{Var}_{p_0}(l) = \frac{q_t^{\mathsf T}\Sigma_K q_t}{d_h}. \tag*{(7)}

Here p0p_0 is uniform on DD; the proof (App. § 7.1.1) writes Γ\Gamma as a Taylor remainder. The useful consequence is Eq. (4): the logit variance s2s^2 needs one key covariance per layer and head and O(dh2)O(d_h^2) work per query and no attention row, hence row-free.

Random-energy reading. A classical model shows where Γ≈s2/2\Gamma\approx s^2/2 breaks.

Assumption 1 (Random-energy model, REM [13]). The n=∣D∣n=|D| dropped logits of a query behave like independent Gaussians with variance s2s^2.

For large nn, Assumption 1 gives two phases separated by the freezing variance 2ln⁡n2\ln n:

Γ≈{12s2,s2<2ln⁡n(high temp.),2s2ln⁡n−ln⁡n,s2≥2ln⁡n(frozen).(8)\Gamma\approx \begin{cases} \frac{1}{2}s^2, & s^2 < 2\ln n \quad\text{(high temp.)},\\ \sqrt{2s^2\ln n}-\ln n, & s^2 \ge2\ln n \quad\text{(frozen)}. \end{cases} \tag*{(8)}

In the frozen phase the largest logits carry ZZ, and the correction s2/2s^2/2 overshoots by (s−2ln⁡n)2/2(s-\sqrt{2\ln n})^2/2. On VGGT (Fig. 2a), measured Γ\Gamma stays within 0.3 nats of s2/2s^2/2 up to s2≈10(rankcorrelation0.986)s^2 \approx10 (rank correlation 0.986), consistent with Eq. (4), and then bends below it, earlier than the REM’s 2ln⁡n≈20.82\ln n \approx20.8, as correlated or light-tailed logits would; the REM thus motivates the cut-off rather than setting it. Below freezing, an importance sample estimates ZZ and the numerator of Eq. (1) without bias, while a mean token is short by about s2/2s^2/2 nats; above it, a shared sample of fixed size misses the extreme keys. Proposition 1 is exact, Eq. (4) approximate, and the cut-off below empirical, validated in Section 4.3: Γ\Gamma measures how badly a mean token fails; that high-Γ\Gamma layers need exact attention rather than a better sample is empirical.

Selector. For each global layer ℓ\ell, the selector computes Γt\Gamma_t for 512 random patch queries per pilot scene and frame subset, with DD set to all patch keys of the other frames, and takes the median over heads and then the mean over pilot queries and pilot scenes to obtain the pilot statistic Γˉℓ\bar{\Gamma}_{\ell}, a layer-level aggregate of the per-query Γt\Gamma_t. Layer ℓ\ell is a matching layer if Γˉℓ≥1.5\bar{\Gamma}_{\ell}\ge1.5 nats and keeps exact attention (§ 2.7); otherwise it is a diffuse layer and uses hubs (Section 3.4). The cut-off was chosen on the development scenes of VGGT after the factorial of Table 1 and fixed before any composed or test run. Γˉℓ\bar{\Gamma}_{\ell} is 1.83–9.21 in layers 11–16 and at most 1.00 elsewhere (Fig. 2b), so the matching set is 11–16, on each of the five development scenes. The row-free form gives the same split: among layers 9–23, the median-head s2s^2 is at least 3 (= 2×1.52\times1.5) in exactly layers 11–16 and at most 1.6 elsewhere. The cut-off (1.5 nats, s2≈3s^2\approx3) lies well below freezing: only layer 14 is frozen, and what the matching layers need are query-specific keys rather than heavy ones (§ 2.7). The selector runs once per model, offline, on the development scenes, and is applied to a new model without retuning (on π3\pi^3 it misses its target, § 3.5); at inference only the hub pilot runs.

Diffuse layers: importance-sampled hubs

In a diffuse layer every query receives the same shared set of stand-in keys, drawn once per layer and input in three steps. Hub pilot (per input): 512 random patch queries R\mathcal{R} compute their full attention and estimate each key’s column mass μ^j=1∣R∣∑t∈RAˉ(t,j)\hat{\mu}_j = \frac{1}{|\mathcal{R}|}\sum_{t\in\mathcal{R}}\bar{A}(t,j), where Aˉ\bar{A} is the head-averaged attention. Sample: over the patch tokens CC, the proposal is pj∝μ^j+ϵ/∣C∣p_j \propto\hat{\mu}_j + \epsilon/|C|, where the floor ϵ=10−3\epsilon=10^{-3} keeps every key reachable; mm i.i.d. draws from pp give the distinct hubs H\mathcal{H} with multiplicities cjc_j.

Attend: a query tt in frame ii uses Eq. (2) with the exact set K(t)=Fi∪S\mathcal{K}(t)=\mathcal{F}_i\cup\mathcal{S}, the query’s own frame Fi\mathcal{F}_i and the set S\mathcal{S} of all special tokens, and the stand-ins W(t)=H∖Fi\mathcal{W}(t)=\mathcal{H}\setminus\mathcal{F}_i with the hub log-weight

bj=ln⁡cj⏟draws of j−ln⁡(mpj)⏟expected draws of j.(9)b_j=\underbrace{\ln c_j}_{\text{draws of }j}-\underbrace{\ln(m p_j)}_{\text{expected draws of }j}. \tag*{(9)}

Proposition 2 (Unbiased hubs). For any function ff of a key, because E[cj]=mpj\mathbb{E}[c_j]=mp_j,

E[∑j∈Webjf(j)]=∑j∈C∖Fif(j).\mathbb{E}\left[\sum_{j\in\mathcal{W}}e^{b_j}f(j)\right]=\sum_{j\in\mathcal{C}\setminus\mathcal{F}_i}f(j).

With f(j)=wjvjf(j)=w_jv_j and f(j)=wjf(j)=w_j, both the numerator and the denominator of the dropped part are unbiased.

A key drawn cjc_j times enters once with weight cj/(mpj)c_j/(mp_j), which corrects for both its draws and its selection probability. The output is a ratio of the two estimates and is not itself unbiased; its bias is of order 1/m1/m (App. § 7.1.3). By contrast, the mean token under-counts the denominator by the factor e−Γe^{-\Gamma} for every query, regardless of sample size (Prop. 1). Because H\mathcal{H} is shared, a hub layer needs no per-query gathers (App. § 7.§ 1.1) and costs about 1.6 units for m=256m=256 and 2.4 for m=1024m=1024 at N=32N=32, a cost that grows with NN only through the special tokens. The statistic chooses the estimator, not mm: LMSA uses m=256m=256 in layers 0–8 and m=1024m=1024 in layers 9, 10 and 17–23, set on the development scenes, where 256 hubs in the later layers lose 11.4 AUC@5 on courtyard (2 subsets).

The late wall as estimation bias. The same deficit offers an explanation for why AVGGT’s late layers fail on redundant scenes. These layers keep the reference frame dense, a σ=9\sigma=9 lattice elsewhere and one mean token for the rest. If φ=∑Dwu/Z\varphi=\sum_D w_u/Z is the share of the true mass that falls on dropped keys, Eqs. (2) and (3) inflate every kept key by the factor

κ=ZZ^=[(1−φ)⏟kept share+φe−Γ⏟dropped share, under-counted]−1≥1,(10)\kappa=\frac{Z}{\widehat{Z}}=\left[\underbrace{(1-\varphi)}_{\text{kept share}}+\underbrace{\varphi e^{-\Gamma}}_{\text{dropped share, under-counted}}\right]^{-1}\geq1, \tag*{(10)}

with Z^\widehat{Z} the estimated denominator; κ\kappa equals 1 only if φ=0\varphi=0 or Γ=0\Gamma=0. In late layers Γ\Gamma is small, but the inflation acts in one direction and falls mostly on the dense reference frame, so it predicts a systematic camera error rather than noise.

Finding 1 (Late-layer failure). With layers 9 and 17–23 thinned AVGGT-style (per-query mean token, other layers dense), adding the second-cumulant term of Eq. (4) (capped at 3 nats) to the mean token’s logit and the first-order term Cov⁡D(v,l)\operatorname{Cov}_D(v,l) to its value recovers 9.7 of the 15.0 AUC@5 points lost on TUM fr3_long_office and cuts the rotation deviation from the dense prediction from 1.01∘1.01^\circ to 0.22∘0.22^\circ (2 subsets). Replacing the dense reference frame, the lattice and the mean token by the own frame and 1024 hubs, which carry no such bias, leaves losses of at most 1.7 AUC@5 on the five development scenes (5 subsets) and a rotation deviation of 0.12∘0.12^\circ on TUM fr3_long_office.

The hub arm also changes the kept keys, so it supports the bias explanation without isolating it, and the cumulant correction serves only as a diagnosis: in the frozen layer 14 it overshoots by 3–9 nats, as Eq. (5) predicts.

Matching layers stay exact

A matching layer keeps dense attention, K(t)=U\mathcal{K}(t)=\mathcal{U} and W(t)=∅\mathcal{W}(t)=\varnothing in Eq. (2), at a price of NN units per layer. § 3.3 explains why: the truncated key sets tested there fail once the diffuse layers are also sparse.

Query-specific relevance, not popularity. Writing attention as A(t,j)=πjL(t,j)A(t,j)=\pi_jL(t,j) separates the column mass πj\pi_j of key jj, its popularity and the quantity that hubs sample, from the lift L(t,j)L(t,j), its specificity to the query. Removing in layers 10–16, per query, as many highest-lift keys as the query has correspondence keys, which hold 1–14% of the attention mass per layer, drives AUC@5 to 0.05 / 0.13 / 0.59 on the three low-redundancy development scenes, although only 13% of these keys lie in correspondence windows; removing a comparable 7–11% of mass chosen by attention weight costs far less (§ 1). A shared sample of popular keys is therefore unlikely to contain what these layers need.

Truncation amplifies whatever it keeps. A sparse matching layer is in effect truncated attention: the smallest layer statistic here is Γ^≃1.8\widehat{\Gamma}\simeq1.8, and a query with that deficit would give its mean token only e−1.8≈17%e^{-1.8}\approx17\% of the mass it stands for (an illustration, not a bound on individual queries), so Eq. (10) amplifies the kept keys by an estimated 2.0–3.8×\times for a σ=9\sigma=9 lattice. With the right keys this sharpening helps: a σ=4\sigma=4 lattice with oracle correspondence keys matches dense accuracy as the only sparse group (loss −1.38-1.38, CI [−3.81,0.29-3.81,0.29]). But it applies equally to every error in the kept keys, including errors that sparse upstream layers pass in to the few query-specific, low-mass keys these layers depend on (App. § 10.§ 1.1). The cheaper LMSA-fast nevertheless replaces exact matching layers by a σ=4\sigma=4 lattice plus routed correspondence blocks (App. § 10.§ 3).

Definition 1 (LMSA). For a frozen alternating-attention transformer, one offline dense pass on development scenes gives Γ^ℓ\widehat{\Gamma}_\ell for every global layer ℓ\ell. If Γ^ℓ≥1.5\widehat{\Gamma}_\ell\geq1.5 nats, layer ℓ\ell is matching and keeps exact attention, K(t)=U\mathcal{K}(t)=\mathcal{U}. Otherwise it is diffuse: a query in frame ii attends exactly to Fi∪S\mathcal{F}_i\cup\mathcal{S} and to mm shared hubs drawn from the per-input hub pilot’s column mass, weighted by Eq. (9). On VGGT-B this gives matching layers 11–16, m=256m=256 in layers 0–8 and m=1024m=1024 in layers 9–10 and 17–23.

Cost

The cost of LMSA follows from the units of its layers. Per token, a forward pass costs

G(N)=2.02⏟rest of net+0.004264(∑ℓuℓ+6N∑ℓ)⏟const in Nexact layers.(11)G(N)=\underbrace{2.02}_{\text{rest of net}}+\underbrace{0.004264\left(\sum_\ell u_\ell+\frac{6N}{\phantom{\sum}\vphantom{\ell}}\right)}_{\substack{\text{const in }N\\\text{exact layers}}}. \tag*{(11)}

GFLOP, where 2.02 GFLOP is the cost outside global attention, uℓu_\ell the number of units of diffuse layer ℓ\ell, and 0.004264 GFLOP the cost of one unit per token. Because the 18 diffuse layers have a nearly fixed cost, the attention cost approaches a quarter of dense as NN grows. In this model, LMSA needs 2.99 / 3.82 / 4.76 / 7.35 GFLOP per token at N=32N=32 / 64 / 100 / 200, against 5.29 / 8.57 / 12.25 / 22.49 for dense (Table 3).

MethodGFLOP / token↓ N=32N=32100Speed-up vs. dense↑ 3264100T&T loss↓ N=100N=100
Dense5.2912.251.001.001.000
AVGGT (σ=4\sigma=4)2.643.771.541.992.417.10
AVGGT (σ=9\sigma=9)2.382.941.632.222.7812.61
GPA-VGGT2.964.791.401.732.001.61
SAF3R2.603.671.341.802.195.65
FastVGGT––1.221.581.8913.96
LMSA2.994.761.431.762.041.90
LMSA-fast2.493.141.512.002.504.55

Table 3. Cost, speed and accuracy at scale. GFLOP per token from the exact cost model (Eq. (11)) at the reference T=1041T=1041; speed-up of the aggregator over dense attention timed in the same session; AUC@5 lost against dense attention on 5 Tanks and Temples scenes at N=100N=100. Absolute times: Tab. S10. Bold: best sparse value per column.

Experiments

The experiments ask whether the design holds on unseen scenes (§ 3.2), why the matching layers must stay exact (§ 3.3), what it costs and how far the selector transfers (§ 3.4 and § 3.5).

Protocol

Model and inputs. All runs use the released VGGT-1B [44] with unchanged weights, the aggregator under bf16 autocast and the camera and depth heads in fp32. Each scene contributes N=32N=32 views at 518 px, or n−4n-4 of its nn views if it has fewer (e.g., 22 of 26 for ETH3D office), and a seed is one frame subset (App. § 8.§ 1); every number is a mean over 5 seeds unless marked. Ground-truth cameras only build and order the subsets, identically for every method, and never enter inference; in a random order the loss of LMSA changes by +0.08 AUC@5 (CI [−1.05, 1.21], App. § 8.§ 1).

Redundancy and scenes. The redundancy ρ\rho of a configuration is one plus the mean number of other views in which a patch’s point is visible, with visibility taken from the dense model’s own depth and cameras because Tanks and Temples has no ground-truth depth; a configuration is low-redundancy if ρ<10.5\rho< 10.5. The 13 unseen low-ρ\rho test scenes come from ETH3D [35] (8), Tanks and Temples [24] (4) and TUM [39] (1), and the 8 unseen high-ρ\rho test scenes from TUM, 7-Scenes [37], DTU [1] and Tanks and Temples (lists in App. § 8.§ 1.1). The 5 development scenes are TUM fr1_room, ETH3D courtyard and office (low ρ\rho), and TUM fr3_long_office and 7-Scenes office (high ρ\rho).

Development protocol. Every design choice in § 2 was made on the development scenes, with acceptance thresholds fixed before the corresponding runs (App. § 8.§ 3). Registered tests had their plan and decision rules hashed before any run (App. § 8.§ 3). The 21 test scenes were named before any test run, and the frozen recipe was evaluated on them once, for Table 2; later analyses on them are marked post hoc; development results, including the variants that failed, are reported separately (App. § 10.§ 2, § 10.§ 1). Three high-ρ\rho test scenes (7-Scenes chess, DTU 24, 114) appear in the attention audit of Fig. 2a but in no end-to-end decision.

MethodUnseen low-ρ\rho (13 scenes) AUC@5↑AUC@30↑AbsRel↓Unseen high-ρ\rho (8 scenes) AUC@5↑AUC@30↑AbsRel↓Dev. (5) AUC@5↑GFLOP/tok.↓
Dense VGGT-1B57.485.44.3472.992.63.9549.25.29
AVGGT (σ=4\sigma=4) [40]47.181.14.0867.491.43.8735.22.64
AVGGT (σ=9\sigma=9) [40]38.572.84.3262.490.53.7929.22.38
GPA-VGGT [41]55.083.84.6572.892.63.9347.82.96
SAF3R [12]52.884.35.4871.192.23.8041.52.60
FastVGGT [36]41.079.74.7267.591.43.8634.3–
LMSA (ours)56.084.74.6872.192.43.9548.62.99
LMSA-fast (ours)52.783.93.9771.192.13.9545.32.49

Table 2. Main results on the 21 pre-named test scenes, the frozen recipe’s single evaluation (N=32N=32, scene means of 5 seeds), and on the development scenes. AbsRel ×100, without Tanks and Temples (no depth). GFLOP/token at N=32N=32: exact cost model (reference T=1041T=1041) for VGGT-based recipes, measured kept fractions otherwise. Bold: best sparse value per column (not a significant difference).

Metrics and statistics. AUC@k averages, over thresholds 1..k degrees, the fraction of frame pairs whose larger rotation or translation-direction error is below the threshold. AbsRel is the median over frames of the per-frame depth AbsRel after median scaling; Tanks and Temples has COLMAP poses and no depth. A loss is dense minus method, paired by seed; 95% intervals come from a hierarchical bootstrap over scenes, then seeds (10,000 resamples).

Baselines and implementation. AVGGT [40] runs at σ∈{4,9}\sigma\in\{4,9\} in a re-implementation, since no official code is released, and need not reach its published accuracy (App. § 8.§ 2); the contemporaneous GPA-VGGT [41] runs at its paper defaults, SAF3R [12] at its repository default (lazy top-4, shipped head assignment) and FastVGGT [36] at its default (90% token merging, aimed at ∼103\sim10^3 views). All methods share checkpoint, precision, frame subsets and software environment (App. § 8.§ 2), and each baseline but FastVGGT reproduces the dense predictions bit for bit with sparsity disabled (FastVGGT within 0.2 AUC@5). SAF3R’s head assignment was calibrated on 11 ETH3D training scenes that include courtyard, office and seven of the eight unseen low-ρ\rho ETH3D scenes, which favours it on ETH3D. Development runs use a masked reference implementation of LMSA and the rows of Table 2 a fused one, whose pooled losses do not differ significantly (App. § 7.§ 1.1).

Main results

Table 2 compares LMSA with dense attention and training-free baselines on scenes unused in the design. On the 13 unseen low-redundancy scenes, LMSA loses 1.38 AUC@5 (95% CI [0.71, 2.01]) and 0.70 AUC@30 ([0.17, 1.26]) against dense attention. It has the lowest mean loss of the training-free methods tested. GPA-VGGT loses 2.42 ([1.28, 3.71]), SAF3R 4.53 ([1.73, 7.73]), AVGGT(4) and AVGGT(9) 10.27 and 18.91, and FastVGGT 16.42. LMSA gains 8.89 AUC@5 over AVGGT(4) (CI [6.45, 11.97]) and 17.54 over AVGGT(9) (CI [13.16, 23.11]). Its gain over GPA-VGGT is 1.04 (CI [−0.34, 2.54]) at 1.01× its FLOPs. This misses the pre-registered target of +5 over the best baseline (App. § 8.§ 3); at equal cost the difference is not significant. Only the gains over AVGGT pass both scene-level tests after Holm correction (App. § 9.§ 1). Post hoc (App. § 12.§ 1), the advantage over GPA-VGGT lies below ρ=10.5\rho= 10.5 (Spearman −0.52, CI [−0.78, −0.14]; −0.48, [−0.72, −0.01], with ρ\rho from ground truth, under which no low-ρ\rho test scene changes side) and comes mainly from ETH3D, and the worst unseen low-ρ\rho scene costs 2.9 AUC@5, against 6.8 for GPA-VGGT. On the 8 unseen redundant scenes, GPA-VGGT is more accurate (loss 0.08, CI [−0.20, 0.42], against 0.73, [0.25, 1.27]), and LMSA gains 9.77 over AVGGT(9) (CI [3.23, 15.78]). The routed variant, LMSA-fast, loses 4.65 for 17% fewer FLOPs. Post hoc, more hubs (2% more FLOPs) halve this low-redundancy loss to 0.66 (App. § 12.§ 1.1; confirmed on fresh data in § 3.4). Depth suffers slightly (AbsRel×100 4.34 to 4.68 at low redundancy), and frame-local layers 0–8 or AVGGT’s late layers raise the low-redundancy loss to 4.84 or 5.54 (App. § 11.§ 3).

Why the matching layers stay exact

In a pre-registered design study on the development scenes (Table S19, D.3), sparsifying layers 11–16 with a σ=4\sigma=4 lattice and routed blocks (LMSA-fast) lost 11.6 AUC@5 on courtyard (2 seeds), above the threshold of 4. Denser lattices and fewer exact layers each failed on at least one scene; exact layers 11–16 lose 1.10 AUC@5 at low redundancy (CI [−0.54, 2.86]).

Layers 11–16fr1courtETH-ofr37Slow-ρ\rhoG100G_{100}
exact (LMSA)1.291.100.920.710.351.104.76
σ=4\sigma=4 + blocks (fast)†^{\dagger}−1.2111.551.735.561.794.023.14
σ=9\sigma=9 + blocks†^{\dagger}6.4116.773.387.442.968.852.82
σ=2\sigma=2 + blocks1.994.921.322.060.562.743.77
13–15 exact, rest σ=4\sigma=44.103.691.092.150.862.964.05
14 exact, rest σ=4†\sigma=4^{\dagger}0.3610.284.334.501.534.993.44

Table S19. Design study (development scenes): AUC@5 lost against dense when layers 11–16 are exact or sparse and all other layers use hubs. fr1: TUM fr1_room; court: ETH3D courtyard; ETH-o: ETH3D office; fr3: TUM fr3_long_office; 7S: 7-Scenes office; low-ρ\rho: pooled over the first three; G100G_{100}: GFLOP per token at N=100N=100. 5 seeds, †^{\dagger}: seeds 0–1. “Blocks”: routed correspondence blocks. Red: above the pre-registered acceptance threshold (4 on the low-ρ\rho scenes, 3 on the others). Further arms: Table S16. Bold: the frozen recipe and its pooled low-ρ\rho loss, the lowest of all arms.

Finding 2 (Sparse matching layers). Every aggressive way of thinning layers 11–16 that was tested costs accuracy on low-redundancy views once the other layers are sparse: truncated lattices with routed blocks cost 3.27 AUC@5 on the test scenes (LMSA-fast against LMSA, CI [1.34, 5.43]), merging similar keys costs 1.6 and 5.7 when 50 and 20% of them remain, per-query top-k frame selection costs 4.2, and shared hubs cost 27.8; only a mild merge that keeps 80% is not significantly worse (0.5), for little saving (post hoc, App. § 10.§ 4).

An ablation (App. § 11.§ 3) shows each part matters: without log-weights the low-ρ\rho test loss is 36 AUC@5, and cut-offs of 2.0 or Post hoc, GPA-VGGT loses 1.61 on the same runs, not significantly different from LMSA (CI of the difference [−2.00, 1.77]), and SAF3R 5.65. On long, redundant 7-Scenes video (App. § 9.§ 4), LMSA loses 0.60 AUC@5 (CI [−0.27, 1.87]) with every third frame (N=333N=333, 7 sequences) at 2.7× the dense speed, where GPA-VGGT exceeds dense by 0.82; with 800 frames it stays within 1.6 AUC@5 of dense on each of 4 sequences at 3.3×, whereas AVGGT, at 5.1–8.1×, departs from it by up to 7.9. In a confirmatory test registered before it ran, on NRGBD, Sintel and these sequences (none used in design), the variant with more hubs stays on average within 0.41 AUC@5 of dense attention and closer to it than LMSA, and one merging 20% of the keys in layers 11–16 stays within 1 AUC@5 of LMSA at 1.16× its speed for N=333N=333 (App. § 9.7). On 9 outdoor RealEstate10K clips [55] (post hoc; all redundant) LMSA loses 0.22 AUC@5, not significantly different from GPA-VGGT, against 2.11, 5.14 and 2.49 for AVGGT(4), AVGGT(9) and SAF3R (App. § 9.6).

3.0 nats cost 2.9 or 11.0 more. At equal cost, the statistic’s layers also beat the four shifted windows tested (registered before it ran; App. § 11.§ 1): windows 9–14, 10–15, 12–17 and 13–18 lose 8.51, 4.19, 2.72 and 4.61 AUC@5 at low redundancy, against 1.38, each significantly more after Holm correction. Entropy and effective key count pick the same layers, so the choice is not an artefact of the deficit formula (used for its link to normalisation error and its row-free proxy); the six layers with most cross-frame mass lose 27.5 more (App. § 11.§ 1.1).

Cost, time and scale

LMSA needs 1.8× fewer FLOPs than dense at N=32N=32 and 3.1× fewer at $N=200 ({xref}Table 3); because its six matching layers stay dense, the ratio tends to 4 as $N$ grows. With every method timed in one session per $N$ ({xref}Table 3), the aggregator of LMSA is about 2% faster than GPA-VGGT’s and takes 7–18% longer than AVGGT(4)’s, and LMSA-fast is on par with AVGGT(4); end to end, with the heads, LMSA is 2–5% slower than GPA-VGGT at $N=32$–100 ({xref}Tab. S10). Accuracy at $N=100$ was measured on five Tanks and Temples scenes (four at low $\rho$), with 5 strided seeds giving only 3–5 distinct subsets ({xref}App. C.4). There LMSA loses 1.90 AUC@5 (CI [1.11, 2.81]) and 0.34 AUC@30 against dense attention while running 2.0× faster on the aggregator; AVGGT(4) and AVGGT(9) lose 7.10 and 12.61, so LMSA gains 5.20 over AVGGT(4) (CI [2.88, 8.35]) at an indicative 1.14× its time ({xref}Table 3`). Post hoc, GPA-VGGT loses

Transfer to a second architecture

A dense pilot pass of the original π3\pi^{3} [48] on the development scenes labelled its 18 global layers with the VGGT cut-offs before any sparse run: layers 11–15 and, only just (Γˉ=1.54\bar{\Gamma}=1.54), layer 7 are matching. With these layers exact and hubs elsewhere, the rule loses 4.87 AUC@5 against dense π3\pi^{3} on the 13 unseen low-ρ\rho scenes (CI [2.45, 7.51], 2 seeds each); AVGGT(4) loses 13.81 (gain 8.93, CI [4.07, 13.95]). The transfer is thus preliminary: the rule beats AVGGT but misses the pre-registered target of a loss within 3 AUC@5 on the unseen scenes, and a pre-registered group-level test failed on the borderline lay 7 (App. § 12.§ 2); GPA-VGGT and SAF3R were not run on π3\pi^{3}. Layer 7, which the row-free proxy does not select, loses nothing with hubs, so a statistic just above the cut-off is not sufficient.

Limitations and Conclusion

Efficient attention for 3D reconstruction must preserve each layer’s information, not merely attention mass. Probability-corrected hubs preserve diffuse context; no tested aggressive thinning preserved the few high-deficit matching layers’ query-specific interactions.

LMSA is not significantly more accurate than contemporaneous GPA-VGGT. At equal FLOPs, its low-redundancy margin (1.0 AUC@5) misses the pre-registered +5; GPA-VGGT is better on redundant views. LMSA is about as fast as GPA-VGGT and takes 13–18% longer than AVGGT(4) at N=64N=64–100100; its six exact layers bound its FLOP advantage over dense attention by 4×4 \times (App. § 10.6). More hubs reduce run-to-run hub-sampling noise; π3\pi^{3} transfer is preliminary.

LLM Usage Statement

Large language models were used in preparing this paper, under the authors’ supervision, for two purposes, both declared in the submission form. The tool used was ChatGPT (OpenAI). First, drafting: some passages were generated from outlines, derivations and result tables supplied by the authors, and then edited by the authors. Second, editing: existing prose was revised for concision, consistent terminology and clarity. The authors read, corrected and approved every generated or edited passage before including it. The authors developed the research problem, conceptual framework, hypotheses, theoretical results and benchmark design before any generative AI was used. AI was not used to originate ideas or hypotheses, to find or select related work, to derive or check proofs, or to produce any reported result. Every citation was checked against its source, and every reported number comes from the reported experiments. The authors take full responsibility for the manuscript and all its claims, results and artifacts.

Appendices

This supplement has six parts: method details, the evaluation protocol, detailed results, the evidence behind the design, the validation of the layer selector, and robustness. It starts with the derivations behind § 2 and then gives the implementation and evaluation details needed to reproduce the results.

Method details

Derivations

These derivations prove the identity for the mean-token deficit Γ\Gamma, show which properties the hub estimator has, and state where both results stop applying. Each derivation is set out in numbered steps, one relation per line, with the reason for a step in grey on its right.

Colour key. Blue: exact (kept) keys; orange: stand-ins and hubs; magenta: the mean-token deficit Γ\Gamma; green: the row-free logit variance s2s^2.

Notation. Fix one query tt, one head and one layer, and drop tt from the notation. For a key uu with value vuv_u and a key set XX,

wu=eluw_u=e^{l_u}
ZX=∑u∈XwuZ_X=\sum_{u\in X}w_u
AX=∑u∈XwuvuA_X=\sum_{u\in X}w_uv_u
mX=AX/ZXm_X=A_X/Z_X

Proof of Proposition 1

Step 1: a cumulant generating function. For τ∈[0,1]\tau\in[0,1], define the log-mean of the tilted weights and the tilted distribution on DD:

Φ(τ)=ln⁡1∣D∣∑u∈Deτlu\Phi(\tau)=\ln\frac{1}{|D|}\sum_{u\in D}e^{\tau l_u}
pτ(u)=eτlu∑u′∈Deτlu′.p_\tau(u)=\frac{e^{\tau l_u}}{\sum_{u'\in D}e^{\tau l_{u'}}}.

At the two ends of the interval,

Φ(0)=0,\Phi(0)=0,
Φ(1)=ln⁡ZD−ln⁡∣D∣.\Phi(1)=\ln Z_D-\ln|D|.

Step 2: its derivatives. Differentiating,

Φ′(τ)=Epτ[l],\Phi'(\tau)=\mathbb{E}_{p_\tau}[l],
Φ′′(τ)=Var⁡pτ(l)≥0.\Phi''(\tau)=\operatorname{Var}_{p_\tau}(l)\ge0.

At τ=0\tau=0 the distribution p0p_0 is uniform on DD, so

Φ′(0)=lˉD=⟨q,kˉD⟩dh.\begin{aligned} \Phi'(0)=\bar l_D \\ &=\frac{\langle q,\bar k_D\rangle}{\sqrt{d_h}}. \end{aligned}

Step 3: Γ\Gamma is a Taylor remainder. By the definition in Eq. (3), for some ξ∈(0,1)\xi\in(0,1),

Γ=Φ(1)−Φ(0)−Φ′(0)=12Φ′′(ξ)=12Var⁡pξ(l).\begin{aligned} \Gamma=\Phi(1)-\Phi(0)-\Phi'(0) \\ &=\frac{1}{2}\Phi''(\xi) \\ &=\frac{1}{2}\operatorname{Var}_{p_\xi}(l). \end{aligned}

Step 4: sign and equality. Hence Γ≥0\Gamma\ge0, and Γ=0\Gamma=0 iff Var⁡pξ(l)=0\operatorname{Var}_{p_\xi}(l)=0. Since pξp_\xi has full support on DD, this happens iff ll is constant on DD.

Step 5: the row-free approximation. With ΣK\Sigma_K the uniform covariance of the keys in DD,

Φ′′(0)=Var⁡p0(l)=1∣D∣∑u∈D(lu−lˉD)2=qTΣKq/dh=s2.\begin{aligned} \Phi''(0)=\operatorname{Var}_{p_0}(l) \\ &=\frac{1}{|D|}\sum_{u\in D}(l_u-\bar l_D)^2 \\ &=q^\mathsf{T}\Sigma_Kq/d_h=s^2. \end{aligned}

If the logit spread is small, Φ′′\Phi'' varies little on [0,1][0,1], so

Γ≈12Φ′′(0)=s2/2.\Gamma\approx\frac{1}{2}\Phi''(0)=s^2/2.

Step 6: a range bound. Hoeffding’s lemma gives, with Δ=max⁡Dl−min⁡Dl\Delta=\max_D l-\min_D l the logit range,

0≤Γ≤Δ2/8.0\le\Gamma\le\Delta^2/8.

The random-energy model

Setup. Let the nn logits be lu=lˉ+sgul_u=\bar l+s g_u, with gug_u i.i.d. standard normal. Then the density of keys, the largest key and the mass are

#{u:gu∈[x,x+dx]}≈ne−x2/2 dx/2π,\#\{u:g_u\in[x,x+dx]\}\approx n e^{-x^2/2}\,dx/\sqrt{2\pi},
xmax⁡≈2ln⁡n,x_{\max}\approx\sqrt{2\ln n},

(no keys beyond)

Z≈elˉ∫xmax⁡ne−x2/2+sxdx2π.Z\approx e^{\bar l}\int^{x_{\max}}n e^{-x^2/2+sx}\frac{dx}{\sqrt{2\pi}}.

The exponent peaks at x=sx=s.

Annealed (high-temperature) phase, s<2ln⁡ns<\sqrt{2\ln n}. The peak lies inside the range, and ZZ is dominated by the ≈ne−s2/2\approx n e^{-s^2/2} keys near x=sx=s, which are many:

ln⁡Z≈ln⁡n+lˉ+s2/2,\ln Z\approx\ln n+\bar l+s^2/2,
Γ=s2/2.\Gamma=s^2/2.

Frozen phase, s≥2ln⁡ns\ge\sqrt{2\ln n}. The peak lies beyond the largest key, and ZZ is dominated by the keys at the edge:

ln⁡Z≈lˉ+s2ln⁡n,\ln Z\approx\bar l+s\sqrt{2\ln n},
Γ=s2ln⁡n−ln⁡n.\Gamma=s\sqrt{2\ln n}-\ln n.

The second-cumulant formula then overshoots by

ln⁡n+s2/2−s2ln⁡n=(s−2ln⁡n)2/2.\begin{aligned} \ln n+s^2/2-s\sqrt{2\ln n} \\ &=(s-\sqrt{2\ln n})^2/2. \end{aligned}

Sample size. For a uniform sample of mm keys in the annealed phase, the relative variance of the estimate of ZZ is (es2−1)/m(e^{s^2}-1)/m. This suggests m∝es2m\propto e^{s^2}, but the saved 256-sample estimators show no such dependence (§ 4). The REM therefore serves only to classify layers, not to size samples.

Proof of Proposition 2 (unbiased hubs)

Step 1: sampling. Let pp be the proposal on the patch tokens CC, and let cjc_j count how often jj appears among mm i.i.d. draws, so

E[cj]=mpj.\mathbb{E}[c_j]=mp_j.

Step 2: the hub weights. For a query in frame ii, the hub part of Eq. (2) uses

ebji=cjmpj,j∈H∖Fi.e^{b_j^i}=\frac{c_j}{mp_j},\qquad j\in\mathcal{H}\setminus\mathcal{F}_i.

Draws that land in the query’s own frame are discarded, because that frame is already exact.

Step 3: unbiased sums. For any ff (the Hansen–Hurwitz estimator),

E[∑j∈C∖Ficjmpjf(j)]=∑j∈C∖FiE[cj]mpjf(j)=∑j∈C∖Fif(j).\mathbb{E}\left[\sum_{j\in C\setminus F_i}\frac{c_j}{mp_j}f(j)\right]=\sum_{j\in C\setminus F_i}\frac{\mathbb{E}[c_j]}{mp_j}f(j) =\sum_{j\in C\setminus F_i}f(j).

With f(j)=wjf(j)=w_j and f(j)=wjvjf(j)=w_jv_j, both estimates are unbiased:

E[Z^D]=ZD,(dropped mass)\mathbb{E}[\widehat{Z}_D]=Z_D,\qquad\text{(dropped mass)}
E[A^D]=AD.(dropped numerator)\mathbb{E}[\widehat{A}_D]=A_D.\qquad\text{(dropped numerator)}

Step 4: the output ratio. The output is a ratio of two unbiased estimates, so its bias is O(1/m)O(1/m) and its variance is O(1/m)O(1/m), with a constant that falls as pp approaches the attention row. The column-mass proposal approaches this ideal for all queries at once: it is the attention row averaged over pilot queries.

Contrast: the mean token. The mean token instead has, deterministically,

Z^D=ZDe−Γ,\widehat{Z}_D=Z_D e^{-\Gamma},

and this bias does not shrink with any sample size.

Real-data check (post hoc, registered before it ran). The proposition was tested directly on VGGT-1B activations: 5 development scenes × 2 seeds, 2048 queries, all 16 heads and all 18 hub layers, with 131,072 hub draws per layer and mm from 64 to 2048.

  • Unbiased sums. In every layer group the mean of Z^D/ZD−1\widehat{Z}_D/Z_D-1 is within 1.6×10−51.6\times10^{-5} of 0, and every 95% CI covers 0; the CIs of the numerator also cover 0.

  • Variance. The output variance falls as 1/m1/m: log–log slopes −0.99, −0.97 and −1.00 for layers 0–8, 9–10 and 17–23.

  • Ratio bias. At the frozen mm it is at most 0.11% of the output’s mean squared error, so the hub error is almost entirely variance.

  • A prediction not supported as registered. The squared bias was predicted to fall as 1/m21/m^2. In layers 9–10 its slope is −1.42, and in the other groups the bias is too small to resolve beyond m=128m=128 under the registered aggregation.

In a post hoc check, the leading-order constant of the O(1/m)O(1/m) bias matches its analytic value within 1% in layers 0–8 and 17–23; in layers 9–10, which approach that regime more slowly, it reaches 0.85 of the analytic value. Because the hub error is almost entirely variance, and that variance falls as the number of draws grows, drawing more hubs, not changing the proposal, is the effective way to reduce it (Section F.2).

Error identity and the inflation factor

Setting. Let U=K∪˙DU=K\mathbin{\dot\cup}D, and let an estimator replace (ZD,AD)(Z_D,A_D) by (Z^D,A^D)(\widehat{Z}_D,\widehat{A}_D) with Z^D>0\widehat{Z}_D>0. Write

m^D=A^D/Z^D,(stand-ins’ mean value)\widehat{m}_D=\widehat{A}_D/\widehat{Z}_D,\qquad\text{(stand-ins’ mean value)}
ϕ^=Z^D/(ZK+Z^D),(their mass share)\widehat{\phi}=\widehat{Z}_D/(Z_K+\widehat{Z}_D),\qquad\text{(their mass share)}
γ=ln⁡(Z^D/ZD).(their log-mass error)\gamma=\ln(\widehat{Z}_D/Z_D).\qquad\text{(their log-mass error)}

Claim. The output error splits into a mean-value error and a mass error:

o^−o=ϕ^(m^D−mD)+(1−e−γ)(mD−o).(12)\widehat{o}-o=\widehat{\phi}(\widehat{m}_D-m_D)+(1-e^{-\gamma})(m_D-o). \tag*{(12)}

Proof. Let Z^=ZK+Z^D\widehat{Z}=Z_K+\widehat{Z}_D. Then

Z^o^=AK+Z^Dm^D,\widehat{Z}\widehat{o}=A_K+\widehat{Z}_D\widehat{m}_D,
Z^o=AK+ZDmD+(Z^D−ZD)o.\widehat{Z}o=A_K+Z_Dm_D+(\widehat{Z}_D-Z_D)o.

Subtracting the two lines,

Z^(o^−o)=Z^D(m^D−mD)+(Z^D−ZD)(mD−o),\widehat{Z}(\widehat{o}-o)=\widehat{Z}_D(\widehat{m}_D-m_D) +(\widehat{Z}_D-Z_D)(m_D-o),

and dividing by Z^\widehat{Z} gives Eq. (12).

Reading. The first bracket term is the error of the stand-ins’ mean value; the second is the error of their mass. For the mean token γ=−Γ\gamma=-\Gamma, so the mass error is

(1−eΓ)(mD−o),(1-e^{\Gamma})(m_D-o),

which vanishes only if Γ=0\Gamma=0. Every kept key’s weight is multiplied by

κ=ZZ^=1(1−φ)+φe−Γ,\kappa=\frac{Z}{\widehat{Z}}=\frac{1}{(1-\varphi)+\varphi e^{-\Gamma}},

which is Eq. (10). Truncation (Z^D=0\widehat{Z}_D=0) is the limit Γ→∞\Gamma\to\infty, with κ=1/(1−φ)\kappa=1/(1-\varphi) and o^=mK\widehat{o}=m_K.

Full recipe

This section specifies the frozen recipe, that is, the exact configuration fixed before the main evaluation, and how it is implemented. Development runs use a masked reference implementation of LMSA and its rows in Table 2 a fused one, whose AUC@5 loss differs from the reference by −0.38 (CI [−1.45, 0.84]) pooled over the development scenes but by 1.78 per run.

Table S1 gives both recipes. The random generator of the pilot for layer ℓ\ell uses the seed 100003(ℓ+1)+m100003(\ell+1)+m, so every run can be reproduced exactly. The frozen recipe has no router, no correspondence blocks and no lattice-density controller; LMSA-fast, in contrast, uses a router and routed correspondence blocks.

LayersKey set per query (query in frame ii)Units
0–8Own frame (all tokens) + special tokens of every frame + 256 hubs importance-sampled from a 512-query pilot (head-mean full attention, column mass, floor 10−3/∣C∣10^{-3}/\lvert C\rvert, log-weight ln⁡(cj/(mpj))\ln(c_j/(mp_j))); hubs in the own frame get no weight1.6
9, 10Own frame + special tokens + 1024 hubs (same estimator)2.4
11–16Exact dense attention (the layers with Γˉℓ≥1.5\bar{\Gamma}_{\ell} \ge1.5)NN
17–23As layers 9–102.4
LMSA-fast, 11–16Eq. (S2): frame 0 + special tokens + σ=4\sigma=4 lattice + diagonal + mean token + switched 6×10 blocks9–10
LMSA-fast, routerOnce, on layer 9’s attention input; top-4 cosine per (query, frame) at full width; 4×8 tiles; 3×3-smoothed displacement mode; switch: share of the tile’s top-1 votes in the smoothed peak ≥0.3\ge0.3≈0.5N\approx0.5N

Table S1. LMSA on VGGT-1B (frozen definition) and LMSA-fast. T=1041T=1041 tokens per frame at 518 px, 16 heads, dh=64d_h=64. Units at N=32N=32 from the exact cost model (one unit = one frame’s keys for every query). Bold: the layers kept exact.

Implementation. The hubs H\mathcal{H} are the same for every query, so a hub layer needs no per-query gathering of keys. Instead, it runs one dense attention call per frame over the query’s own frame Fi\mathcal{F}_i and one call, shared by all queries, over the special tokens and hubs [S;H][S;\mathcal{H}]; the two partial results are then merged by log-sum-exp. A hub layer costs 1+5(N−1)/T+(m+256)/T1+5(N-1)/T+(m+256)/T units, where the four terms are the own frame, the special tokens, the hubs and the pilot. The pilot term counts one pass of query–key products for the 512 pilot queries, 512/(2T)=512/(2T)=

256/TT units. The fused implementation computes these products twice (once for the softmax normaliser and once for the column mass), which adds another 256/TT units per hub layer, about 0.02 GFLOP per token over the 18 hub layers, under 1% of the total. The GFLOP figures in the paper use the single-pass count; the measured times include the actual cost. Only the special-token term grows with NN, and it is small, so the cost of a hub layer stays nearly constant as more frames are added. The test rows and all timings use a fused implementation. Per frame, it issues one FlashAttention [11] call over the own frame and one shared Triton kernel over the special tokens and hubs. It matches the masked reference implementation to within 1.5×10−51.5 \times10^{-5} per layer in fp32, so the two implementations give the same model. For models without a reference frame, a frame-0 rule was also defined: keep a frame dense in the matching layers if its pilot column-mass share exceeds 1.5× the uniform share. This rule is only logged, because it has no effect when the matching layers are already exact.

Per-layer statistics of VGGT-1B

This section reports, for every layer of the model, the statistics that the selector uses or that help to interpret its choice.

Table S2 lists the statistics per layer. Three observations in it matter for the method. First, the two forms of the selector agree on layers 9–23: the pilot rule Γˉ≥1.5\bar{\Gamma} \ge1.5 and the row-free rule (median-head s2≥3s^2 \ge3) both pick layers 11–16. This also explains why Table 1 and Fig. 2c treat layers 10–16 as the middle group: those experiments were run before the statistic existed, and the statistic then classified layer 10 as diffuse (Γˉ=1.00\bar{\Gamma}=1.00), so the method gives it 1024 hubs rather than exact attention. Second, layer 21 is diffuse at the median head, but 5 of its 16 heads have s2≥3s^2 \ge3, so a few of its heads are concentrated. This fits two earlier observations: this layer needs 1024 rather than 256 hubs, and the corrected mean token alone still left a 5.4-point loss on TUM fr3. Third, the correspondence window carries at most 10% of the attention mass, even at layer 14, and yet removing it is catastrophic (Table S15). Attention mass is therefore the wrong criterion for choosing which keys to keep exact: keys that receive little mass can still carry the information the layer uses.

LayerΓˉ\bar{\Gamma}s2s^2<3<3≥7\ge7TubeRole
00.12––––hubs, 256
10.05––––hubs, 256
20.30––––hubs, 256
30.71––––hubs, 256
40.78––––hubs, 256
50.86––––hubs, 256
60.92––––hubs, 256
70.91––––hubs, 256
80.25––––hubs, 256
90.781.28160.00–hubs, 1024
101.001.57150.000.006hubs, 1024
112.664.9240.290.029exact
122.766.8240.390.030exact
132.826.5940.470.050exact
149.2126.6430.730.098exact
154.088.3230.490.038exact
161.833.1560.040.008exact
170.530.88160.00–hubs, 1024
180.030.05160.00–hubs, 1024
190.280.44160.00–hubs, 1024
200.060.11160.00–hubs, 1024
210.481.04110.12–hubs, 1024
220.010.03160.00–hubs, 1024
230.070.12160.00–hubs, 1024

Table S2. Per-layer statistics behind Fig. 2. Γˉ\bar{\Gamma}: pilot statistic (median head, mean of 5 development scenes). s2s^2: median-head row-free logit variance. “<3<3”: heads (of 16) with s2<3s^2 < 3. “≥7\ge7”: share of (query, head) pairs with s2≥7s^2 \ge7. Tube: median mass on the 3×3 correspondence window (low-ρ\rho). Layers 0–8 were not in the attention audit. Bold: the layers kept exact (those above the cut-off) and their Γˉ\bar{\Gamma}.

Table S15. Which keys matter in layers 10–16. AUC@5 lost against dense when a key set is removed from layers 10–16 only (5 seeds, low-ρ\rho development scenes). “Correspondence”: the model’s own 3×3 correspondence windows. “Heaviest”: as many of the highest-attention non-correspondence keys. “Highest lift”: as many keys ranked by A(t,j)/πjA(t,j)/\pi_j. “Random”: as many random keys. Mass: head-averaged attention mass removed (for highest lift: range over layers). Bold: the two largest losses per scene.

Removed setfr1courtETH-oMass
Correspondence22.149.217.70.03–0.04
Heaviest6.523.06.00.07–0.11
Highest lift29.758.242.50.01–0.14
Random0.1−0.30.0−

Table S15.

Protocol

Reproducibility details

This section documents the registered protocol: how frame subsets are drawn and ordered, how images and ground truth are prepared, and how the redundancy ρ\rho is computed.

Frame subsets. Each seed ss selects a different subset of frames from a scene. For TUM, 7-Scenes and Tanks and Temples, seed ss takes N=32N=32 frames at a fixed stride, starting at frame round⁡(s⋅stride⁡/5)\operatorname{round}(s \cdot\operatorname{stride}/5). The stride is 20 for TUM (or the largest stride that fits; only frames with depth and pose within 20 ms

are used), 15 for 7-Scenes sequence 1, and ⌊n/32⌋\lfloor n/32 \rfloor over the nn registered Tanks and Temples images in filename order. For ETH3D and DTU, seed ss draws views without replacement with NumPy default_rng([20261111, s, \sum \operatorname{ord}(\text{scene key})]). The drawn view nearest the subset’s camera centroid becomes frame 0, and the other views follow in order of camera distance. The exceptions are seed 0 of courtyard and the DTU scans, which take the 32 views nearest the view closest to the centroid of all cameras. The ground-truth cameras serve only to build and order these benchmark subsets. They are never an input to any model, and every method receives the same images in the same order, so dense attention and all sparse methods share the same frame 0. For TUM, 7-Scenes and Tanks and Temples, the order is the recording or file order, which uses no pose information.

Random input order (post hoc control, registered before it ran). The ETH3D subsets are ordered using camera geometry, so the reference frame is a central view. This control tests whether that geometry-informed order favours any method. The 8 unseen low-ρ\rho ETH3D test scenes were rerun with the same views in a random order, so that frame 0 is an arbitrary view (5 seeds, 40 stored permutations), for dense attention, LMSA, GPA-VGGT and AVGGT(4). Each loss is measured against dense attention run on the same order (Table S3). The loss of LMSA changes by only +0.08 AUC@5 (CI [−1.05, 1.21]), which lies inside the registered band of ±1.5, and the loss of GPA-VGGT changes by −0.19. Only AVGGT(4) degrades, by 3.48 AUC@30 (CI [0.24, 7.34]); this is expected, because this method keeps all of frame 0 in its later layers and so depends on which view comes first. As a result, under the random order the lead of LMSA over AVGGT(4) grows (from 8.87 to 10.66 AUC@5), while its lead over GPA-VGGT changes little (from 1.44 to 1.17). One deviation from the plan occurred. One run of GPA-VGGT on the original order differed from its main-table run by up to 0.53°, slightly beyond the planned parity bound of 0.5°. A registered amendment showed that this difference matches GPA-VGGT’s own run-to-run spread, which arises because its merging uses atomic operations; its rows are therefore kept and marked as a deviation. Scoring the random-order predictions in the original frame order gives the same picture (LMSA +0.15, GPA-VGGT +0.11, AVGGT(4) +2.44 AUC@5).

MethodOriginal orderRandom orderChange
LMSA1.571.65+0.08 [−1.05, 1.21]
GPA-VGGT{sup}†`3.012.83−0.19 [−2.08, 1.35]
AVGGT(4)10.4412.31+1.87 [−1.02, 4.95]

Table S3. Random input order on the 8 unseen low-ρ\rho ETH3D scenes (5 seeds): AUC@5 loss† against dense attention on the same order, mean and 95% CI; change = random minus original. Bold: lowest loss per column and the change for LMSA.

In every subset, VGGT’s reference frame is frame 0. Preprocessing. Images are resized bicubically to a width of 518 px and a height rounded to a multiple of 14, and the camera intrinsics are rescaled per axis to match. The resulting sizes are 392×518 for TUM, 7-Scenes and DTU (so T=1041T=1041 tokens per frame), 350×518 for ETH3D and 294×518 for Tanks and Temples. Ground-truth depth is brought to the model’s resolution without hole filling. TUM and 7-Scenes depth is resized by nearest neighbour; DTU depth comes from the 160×128 MVSNet depth maps; ETH3D depth is a per-pixel minimum-depth splat of the aligned laser scans. Tanks and Temples has COLMAP poses but no depth. Redundancy. The redundancy ρ\rho measures how many views see each surface patch. To compute it, every patch of every view is back-projected from its centre, at the median predicted depth of its pixels, using the dense model’s cameras. The patch counts as visible in another view if it lands in front of that camera, inside the image, and within 15% of the median predicted depth of the surrounding 5×5 pixels. No confidence threshold is applied. Selector for a new model. The selector computes Γ\Gamma on 512 pilot rows, which are uniformly random patch queries drawn per scene and subset. The same rows are used in every layer, and they are separate from the hub pilot. Γ\Gamma is computed in fp32 from the post-RoPE queries and keys. Its median over heads is averaged over rows and then over the pice-scene-subsets, which are the five development scenes (on VGGT, subsets 0–4 for the low-ρ\rho scenes and 0–1 for the others; on π3\pi^3, subsets 0–2, after a parity check of the masked pass). Finding 1. The corrected arm of this finding keeps, in layers 9 and 17–23, the keys of the uncorrected arm (frame 0, special tokens, σ=9\sigma=9 lattice and self term) and its per-query mean token. It then corrects the mean token in two ways: it adds min⁡(12s2,3)\min(\frac{1}{2}s^2, 3) to the token’s logit, and it adds the first-order term Cov⁡p(v,l)\operatorname{Cov}_p(v,l) to the token’s value. The covariances are computed per layer and head over the patches that lie off the lattice and outside frame 0. Code will be released.

Scenes

This section lists the development and test scenes used in the registered evaluation.

All lists were fixed before the composed method was first run, based on a redundancy scan of 70 configurations. Unseen low-ρ\rho (13): ETH3D [35] delivery_area, electro, kicker, playground, relief, relief_2, terrace and terrains; Tanks and Temples [24] Barn, Caterpillar, Courthouse and Meetingroom (full span); TUM [39] fr1_plant. Unseen high-ρ\rho (8): TUM fr2_desk and fr3_structure_texture_far; 7-Scenes [37] chess, fire and pumpkin; DTU [1] scans 24 and 114; Tanks and Temples Ignatius. Development (5): TUM fr1_room, ETH3D courtyard and office; TUM fr3_long_office and 7-Scenes office. Held out (10; used only to fit a threshold of an earlier, routed recipe): ETH3D facade, pipes and meadow; TUM fr1_teddy, fr1_desk and fr3_cabinet; 7-Scenes heads, redkitchen and stairs; Tanks and Temples Truck. The Tanks and Temples low-ρ\rho scenes have low redundancy by this count, but they are easy for the model: dense VGGT already scores 90–91 AUC@5 on Barn, Caterpillar and Meetingroom (5-seed means). Differences between methods are therefore expected to be small on these scenes. ETH3D scenes with fewer than 36 images use n−4n-4 of their nn images (office 22 of 26; kicker, relief and relief_2 27 of 31; terrace 19 of 23).

Baselines

This section documents how each baseline was implemented, where the setup deviates from the authors’ own, and which methods were not run.

AVGGT. No official code is released for AVGGT [40], so a re-implementation is used. It makes layers 0–8 frame-local, meaning that each query attends only to its own frame. In layers 9–23, each query keeps the special tokens, frame 0 and one patch per σ\sigma-cell (σ=4\sigma=4: 2×22\times2; σ=9\sigma=9: 3×33\times3), plus a self column for dropped queries; all dropped tokens are replaced by one count-biased mean key. The constants follow the AVGGT paper: frame-local layers 0–8; the first patch of each 2×22\times2 (σ=4\sigma=4) or 3×33\times3 (σ=9\sigma=9) window; frame 0 uncompressed; all queries and special tokens kept; and the self term plus one mean key and value for the dropped patches, all under a shared softmax. The only constant that the AVGGT paper does not state is the +ln⁡ndropped+\ln n_{\mathrm{dropped}} bias on the mean key. This re-implementation does not show AVGGT at its best. At N=32N=32 it loses 5.5 AUC@5 on the unseen redundant scenes (on TUM fr2_desk, from 88.9 to 74.5). In contrast, the AVGGT paper reports accuracy close to dense VGGT at 333 and 800 frames on its own benchmarks (e.g., 7-Scenes AUC@30 77.4 against 74.2 at 800 frames) [40]. The likely causes are the smaller NN used here, the absence of official code and the unstated constant above.

GPA-VGGT. GPA-VGGT [41] is contemporaneous work (arXiv, 27 October 2026); it is run with its paper defaults. In global blocks 0–8, each query keeps its own frame, all special tokens and 20 cluster summaries per frame and head, with a +log⁡(size)+\log(\mathrm{size}) bias that accounts for how many tokens each summary stands for. In blocks 9–23, it keeps frame 0, the special tokens and one random token per 2×22\times2 window, and merges every other token into its most similar kept token. Blocks 13–16 are treated more gently: they merge only 20% of their tokens.

SAF3R. SAF3R [12] is run with its repository default for VGGT. Each of the 384 global heads is assigned one fixed pattern: frame 0 only, strided with anchors, query-probe top-kk, own frame only, or per-frame top-4. The assignment comes from an offline calibration on 11 ETH3D training scenes, as listed in its repository’s calibration configuration (the SAF3R paper reports 10 sequences).

FastVGGT. FastVGGT [36] is run with its repository defaults (official code, commit 6526e27) and no tuning. It merges tokens in all 24 global blocks with merge ratio 0.9, which means that 90% of the tokens of each block, queries included, are merged into kept tokens. The other defaults are a 2×22\times2 destination stride, protected tokens enabled and merge seed 33. FastVGGT was added after the main results, under a plan registered before its first run. It is designed for about 10310^3 views, and its authors evaluate it at that scale, so its rows here, at N=32N=32 and 100, need not reflect its best case.

Not run. Other training-free accelerators were not run in the time available, among them RegimeVGGT [30], whose code is public. Their published speed-ups and accuracies use other checkpoints, hardware, sequence lengths and timing conventions, so they cannot be compared with Table S10 or Table 2.

MethodGFLOP / token N=32N=32GFLOP / token 100Agg. time (s) 32Agg. time (s) 64Agg. time (s) 100Full time (s) 32Full time (s) 64
Dense5.2912.255.2715.0231.325.7116.05
AVGGT (σ=4\sigma=4)2.643.773.437.5412.983.908.63
AVGGT (σ=9\sigma=9)2.382.943.226.7611.263.727.88
GPA-VGGT2.964.793.768.6815.694.039.41
SAF3R2.603.673.948.3414.284.419.34
FastVGGT†––4.209.4116.554.6010.11
LMSA2.994.763.688.5415.354.229.69
LMSA-fast2.493.143.487.5112.553.948.50

Table S10. Cost and time. GFLOP per token from the exact cost model (Eq. (11)). Times: one session per NN in which every method, dense included, was timed interleaved on the same input (1 warm-up and 3 timed rounds, median, CUDA events) on one 16GB GPU. “Agg.”: aggregator only, including the hub pilot; “full”: aggregator (which includes the DINOv2 patch embedding) plus camera and depth heads, i.e. the whole network; image loading and resizing are excluded (at N=100N=100, in a separate session: dense 32.26 s, LMSA 17.19 s, GPA-VGGT 16.84 s). †Own sessions, in which dense took 5.14, 14.88 and 31.30 s (aggregator). Bold: lowest sparse value per column.

Deviations from the authors’ setups. The GPA-VGGT and SAF3R repositories default to the tracker-fixed VGGT checkpoint, and the FastVGGT authors also evaluate on it; here, in contrast, VGGT-1B is used for all methods. FastVGGT casts its whole model to bf16 and normalises its input in bf16. Here it runs at the precision shared by all methods, but keeps its own normalisation and its per-block cache release. As a result, unlike the other baselines, FastVGGT with merging disabled

SceneMethodAUC@5AUC@30AbsRel
TUM fr1_roomdense29.1577.900.0418
TUM fr1_roomSAF3R23.0273.500.0414
TUM fr1_roomGPA-VGGT26.9476.130.0429
ETH3D courtyarddense54.4090.220.0198
ETH3D courtyardSAF3R30.9780.340.0245
ETH3D courtyardGPA-VGGT41.3784.690.0222

Table S4. Table S4. Baseline bring-up check (single seed, N=32N=32, one environment). Not used for any claim; the main comparison is Table 2. Bold: the better baseline per scene and metric.

does not reproduce dense attention bit for bit; it stays within 0.2 AUC@5 and 0.3 AUC@30 of it on TUM fr1_room (largest rotation difference 0.33°). GPA-VGGT also runs at the precision shared by all methods here (fp32 weights, bf16 autocast in the aggregator, fp32 heads) rather than in fp16. On TUM fr1_room this changes its AUC@5 by 0.37 (single seed). All rows of Table 2 run in one software environment. This matters because dense predictions differ by up to about 1 AUC@5 between the two environments used, so mixing environments would add a spurious difference between methods. Dense attention and SAF3R give bit-identical outputs from run to run, but GPA-VGGT does not, because its key merging and pooling use atomic scatter operations. Its largest output difference between runs is 4.6, on aggregator outputs of magnitude up to about 111 (N=32N=32). For timing, GPA-VGGT and SAF3R were run in the same sessions as all other rows of Table S10. GPA-VGGT was timed with bf16 autocast; fp16 gives the same aggregator times within 0.4%. FastVGGT was added later and was therefore timed in its own sessions, together with dense attention and LMSA; its speed-ups in Table 3 are relative to the dense run of those sessions.

Pre-registered bars

This section records the pre-registered bars, that is, the pass/fail thresholds written down before each experiment, and separates confirmatory results from post-hoc analyses. The main table and the fresh-data test are confirmatory; the remaining analyses are post hoc unless they are identified above as registered.

What “registered” means. A test is called registered when its plan was written into a file before any of its runs: the scenes and frame subsets, the methods, the metrics, the predictions and the rule that decides the outcome. The plan file was then hashed with SHA-256, so any later change to it would change the hash; amendments were written as new, separately hashed files that point to the original. Table S5 lists the first 12 hexadecimal digits of each hash and the time the plan was fixed (UTC+6). The bars of the main evaluation and of the design study are the exception: they were fixed in the design specification before the test runs, but that document was not hashed on its own.

TestPlan hash (prefix)Fixed
π3\pi^3 predictions, newer release20947731e395Nov 13, 15:02
π3\pi^3 predictions, original checkpointd338191c0597Nov 13, 15:44
π3\pi^3 transfer evaluation5ed86d7cbc44Nov 13, 18:19
Selector applied to GPA-VGGT1d5d99725695Nov 13, 21:37
7-Scenes long sequencesd360a911d5f0Nov 13, 21:52
Confirmatory test (first version)2ee8727dd5a1Nov 14, 04:31
800 frames21e5457f0f36Nov 14, 00:49
Equal-cost windowsff4462e4c988Nov 14, 15:53
Random input order5d4caa447be6Nov 14, 22:31
Simpler selectors782dc763645bNov 14, 22:57
Hub estimator on real data3d57a51caf67Nov 15, 01:37
RealEstate10K outdoord7ac3d273aedNov 15, 05:00

Table S5. Registered plans. SHA-256 prefix of each plan file and the time it was fixed, before any run of that test (2026, UTC+6). Bold: the confirmatory test.

All bars were written before the corresponding runs. They were set for four stages.

  • Composed pilot (seeds 0–1). The loss must be ≤4\leq4 AUC@5 on each low-ρ\rho development scene and ≤3\leq3 on the redundant ones; the interaction between groups must be ≤1.5\leq1.5; the in-pass router must stay within 0.03 vote recall and +0.03 orphan share of the dense-pass router; and the late layers may add ≤1.5\leq1.5. A stop rule

ends the pilot if the low-ρ\rho mean loss exceeds 5 or any scene exceeds 8; in that case, any swap-back (returning one group to dense attention) that recovers more than 2.5 names the group to fix.

  • Repair (seeds 0–1 screen, seeds 0–4 confirmation). The loss bars are the same. The cheapest arm that passes the screen is selected and confirmed on 5 seeds; if it fails, the next screen-passer is tried; exact layers 11–16 are the fallback.

  • Main table. On the unseen low-ρ\rho scenes, the loss must be within 3 AUC@5 and 2 AUC@30 of dense attention (CI upper bounds ≤5\le5 and ≤4\le4); the gain over the best baseline must be ≥+5\ge+5 AUC@5 at ≤1.10×\le1.10\times its FLOPs (CI lower bound >0> 0); and on the unseen high-ρ\rho scenes the loss must be within 3 of dense attention.

  • Transfer. The groups predicted as matching for π3\pi^3 must carry ≥50%\ge50\% of the loss.

Outcomes. Pilot. The pilot fired its stop rule (courtyard 11.55). Its swap-backs named layers 11–16 as the group to fix: returning them to dense attention recovered 3.66 pooled, the only recovery above 2.5. The router bars passed. Repair. The repair screen passed the σ=2\sigma=2 lattice and exact layers 13–15, and failed exact layer 14 (courtyard 10.28). In the confirmation on more seeds, the σ=2\sigma=2 lattice then failed (courtyard 4.92), and so did exact layers 13–15 (TUM fr1 4.10). The fallback, exact layers 11–16, passed every bar and became the frozen recipe. Main table. The bars for closeness to dense accuracy pass: on the unseen low-ρ\rho scenes the loss is 1.38 AUC@5 (CI upper bound 2.01) and 0.70 AUC@30 (upper bound 1.26), and on the unseen high-ρ\rho scenes it is 0.73. On the high-ρ\rho scenes, the gain over AVGGT(9) is 9.77 (CI [3.23, 15.78]). The bar over the best baseline fails. On the unseen low-ρ\rho scenes the best baseline is GPA-VGGT, and LMSA gains only 1.04 AUC@5 over it (CI [−0.34, 2.54]) at 1.01× its FLOPs. This gain is short of the required +5, and its CI includes zero. The larger gains over SAF3R (3.15, CI [0.20, 6.39], at 1.15× its FLOPs) and over AVGGT(4) (8.89, CI [6.45, 11.97], at 1.13×) do not satisfy the bar either, for two reasons: the bar is set against the best baseline only, and both comparisons exceed 1.10× the baseline’s FLOPs. Transfer. The group rule fails on the original π3\pi^3: it predicted layer 7 and the late group as matching, but only the late group was observed to be (Section § 2). With the rule applied untuned, the target of a loss within 3 AUC@5 fails on the unseen low-ρ\rho scenes (4.87, CI [2.45, 7.51]) and passes on the development scenes (2.53, CI [0.40, 4.96]).

Results in detail

Per-scene results and intervals

This section breaks the main evaluation down by scene and by dataset and gives the uncertainty intervals and scene-level tests behind it.

Tables S6 and S7 complete Table 2: the first gives each method’s paired difference to dense attention with intervals, and the second gives every scene. Table S8 groups the test scenes by dataset. LMSA has the lowest loss on ETH3D, Tanks and Temples and DTU; GPA-VGGT has the lowest on TUM; and AVGGT(4) has the lowest on 7-Scenes. On that dataset, however, every method except AVGGT(9) and SAF3R stays within 1 AUC@5 of dense attention, so the differences there are small. On the unseen low-ρ\rho scenes, the paired AUC@5 gain of LMSA is:

Table S8. Per-dataset comparison. AUC@5 lost against dense attention, mean over each dataset’s unseen test scenes of the 5-seed scene means (N=32N=32; in brackets, how many of the scenes are low-ρ\rho). Negative: above dense attention. Bold: lowest loss in the row. Descriptive: datasets with 2–3 scenes carry no interval. *A separate post-hoc test (Section C.6): 9 outdoor test clips, 5 frame subsets each, all of high redundancy; LMSA-fast was not run.

DatasetScenesDense AUC@5AVGGT(4)AVGGT(9)GPA-VGGTSAF3RLMSALMSA-fast
ETH3D8 (8)43.410.4418.933.013.141.574.84
Tanks and Temples5 (4)84.19.4018.781.255.501.103.78
TUM RGB-D3 (1)82.111.8119.200.734.461.045.04
7-Scenes3 (0)38.4−1.08−0.46−0.212.040.800.64
DTU2 (0)99.97.3614.130.060.300.050.06
RealEstate10K outdoor*9 (0)33.22.115.14−0.762.490.22—

Table S8.

  • 8.89 over AVGGT(4) (CI [6.45, 11.97]);

  • 17.54 over AVGGT(9) (CI [13.16, 23.11]);

  • 3.15 over SAF3R (CI [0.20, 6.39]);

  • 1.04 over GPA-VGGT (CI [−0.34, 2.54]), an interval that includes zero.

The difference to GPA-VGGT is uneven across scenes. On the unseen low-ρ\rho scenes, LMSA is ahead on ETH3D terrace (5.6), relief (4.1), playground (3.1) and relief_2 (2.1) and on Tanks and Temples Courthouse (3.3); it is behind on ETH3D terrains (−3.6); and it is within 1 AUC@5 on the other seven. On the unseen high-ρ\rho scenes, in contrast, LMSA is behind GPA-VGGT on seven of eight, by at most 1.4. On ETH3D terrains, SAF3R exceeds dense attention by 6.0. This scene is not unseen for this baseline: its head assignment was calibrated on ETH3D training scenes that include terrains (Section B.3). Table S13 tests whether these gains hold at the scene level, treating the 13 scene means as the samples. The Holm correction runs over the four baselines that were part of the main evaluation from the start (AVGGT(4), AVGGT(9), SAF3R and GPA-VGGT); FastVGGT was added later and is not part of these tests. After a Holm correction over the four baselines, the results are as follows. The gains over AVGGT(4) and AVGGT(9) are significant in both tests. The gain over SAF3R is significant only in the signed-rank test, which is robust to the terrains outlier. The gain over GPA-VGGT is significant in neither test. On the 8 high-ρ\rho test scenes, no comparison survives the correction; GPA-VGGT is ahead of LMSA on 7 of them (raw p=0.023p = 0.023 in both tests).

Point-map metrics

This post-hoc analysis asks whether the pose results are also reflected in the quality of the predicted point maps.

This analysis of existing predictions is post hoc; its plan was registered before scoring. It uses the 9 unseen low-ρ\rho scenes that have ground-truth geometry (8 ETH3D scenes and TUM fr1_plant, 5 seeds each). Point maps were scored with the accuracy, completeness and normal-consistency metrics of AVGGT and π3\pi^3, with 5 cm voxels for ETH3D and 2 cm for TUM. The Chamfer distance is the mean of accuracy and completeness, in metres, so lower is better. Before scoring, each prediction must be aligned to the ground truth, and two alignments are reported (Table S9). Alignment A fits a similarity transform between the predicted and ground-truth camera centres, so its Chamfer distance also carries pose error. Alignment B fits the transform on per-pixel point correspondences, which isolates the quality of the geometry itself.

A (cameras)B (points)
LMSA − dense+0.007 [−0.014, 0.029]+0.001 [−0.014, 0.015]
LMSA-fast − dense−0.057 [−0.116, −0.010]+0.034 [−0.010, 0.095]
GPA-VGGT − dense+0.066 [0.018, 0.130]+0.011 [−0.006, 0.035]
SAF3R − dense+0.064 [0.031, 0.098]+0.020 [0.005, 0.038]
AVGGT(4) − dense+0.120 [0.033, 0.219]–
AVGGT(9) − dense+0.283 [0.108, 0.541]–
LMSA − GPA-VGGT−0.059 [−0.121, −0.009]−0.010 [−0.045, 0.016]
LMSA − SAF3R−0.057 [−0.097, −0.021]−0.019 [−0.044, 0.003]

Table S9. Point maps on the 9 unseen low-ρ\rho scenes with ground truth (post hoc). Change in Chamfer distance (m) against dense attention, and paired contrasts, with hierarchical 95% CIs; negative is better. A: alignment on camera centres (includes pose error); B: alignment on points. Bold: LMSA against dense attention, and its two contrasts that are significant (alignment A).

Table S6. Paired differences to dense on the unseen scenes (pooled; hierarchical bootstrap 95% CI), N=32N=32, 5 seeds. Loss = dense − method; Δ\DeltaAbsRel = method − dense. Bold: lowest value per column.

MethodAUC@5 loss, low-ρ\rhoAUC@30 loss, low-ρ\rhoAUC@5 loss, high-ρ\rhoΔ\DeltaAbsRel×100\times100, low-ρ\rho
AVGGT (σ=4\sigma=4)10.27 [7.98, 13.22]4.30 [2.25, 6.85]5.50 [1.40, 9.53]-0.26
AVGGT (σ=9\sigma=9)18.91 [14.64, 24.51]12.63 [7.73, 18.59]10.50 [4.01, 16.65]-0.02
GPA-VGGT2.42 [1.28, 3.71]1.67 [0.50, 3.09]0.08 [−0.20, 0.42]+0.31
SAF3R4.53 [1.73, 7.73]1.09 [−0.52, 2.73]1.73 [0.60, 3.20]+1.14
LMSA1.38 [0.71, 2.01]0.70 [0.17, 1.26]0.73 [0.25, 1.27]+0.34
LMSA-fast4.65 [2.74, 6.72]1.54 [0.71, 2.49]1.81 [0.24, 3.67]-0.37

Table S6.

Table S7. Per-scene AUC@5 (5-seed means, N=32N=32 unless the scene has fewer views). Rows are generated from the main-table results file. Bold: highest sparse value per scene (ties all bold).

SplitSceneDenseAVGGT(4)AVGGT(9)GPA-VGGTSAF3RLMSALMSA-fast
Unseen low-ρ\rhoETH3D delivery_area25.114.88.924.124.424.619.0
Unseen low-ρ\rhoETH3D electro61.248.135.558.044.558.354.6
Unseen low-ρ\rhoETH3D kicker90.284.879.089.188.688.486.7
Unseen low-ρ\rhoETH3D playground25.817.99.020.921.524.025.8
Unseen low-ρ\rhoETH3D relief23.513.14.518.621.222.719.0
Unseen low-ρ\rhoETH3D relief_210.14.91.77.07.39.07.2
Unseen low-ρ\rhoETH3D terrace81.557.436.774.778.980.468.8
Unseen low-ρ\rhoETH3D terrains29.822.820.430.835.727.227.3
Unseen low-ρ\rhoT&T Barn90.581.470.589.185.488.485.6
Unseen low-ρ\rhoT&T Caterpillar90.282.373.589.984.489.584.6
Unseen low-ρ\rhoT&T Courthouse58.845.935.055.846.859.159.6
Unseen low-ρ\rhoT&T Meetingroom91.081.674.789.986.889.488.5
Unseen low-ρ\rhoTUM fr1_plant68.057.550.566.661.467.058.8
Unseen high-ρ\rhoTUM fr2_desk88.974.565.887.988.287.088.6
Unseen high-ρ\rhoTUM fr3_str._tex._far89.378.972.489.783.489.283.8
Unseen high-ρ\rho7-Scenes chess46.947.646.347.045.646.247.0
Unseen high-ρ\rho7-Scenes fire52.355.755.652.548.751.151.2
Unseen high-ρ\rho7-Scenes pumpkin15.915.014.516.214.615.414.9
Unseen high-ρ\rhoDTU scan 24100.089.782.799.999.8100.099.8
Unseen high-ρ\rhoDTU scan 11499.895.388.899.899.499.799.8
Unseen high-ρ\rhoT&T Ignatius89.982.372.889.489.588.683.4
DevelopmentTUM fr1_room29.823.116.528.323.429.825.6
DevelopmentETH3D courtyard58.521.08.153.136.457.152.5
DevelopmentETH3D office43.233.930.744.039.542.840.5
DevelopmentTUM fr3_long_office87.570.865.586.782.886.581.7
Development7-Scenes office26.927.225.126.625.526.925.9

Table S7.

Under both alignments, the point-map differences between LMSA and dense attention are not statistically significant. The normals of LMSA are slightly worse, however (normal consistency −0.006, CI [−0.010, −0.003], under A). Under A, LMSA is 0.059 m better than GPA-VGGT and 0.057 m better than SAF3R. Under B, these gaps shrink to 0.010 and 0.019 m and are no longer significant, which means that most of the gap under A comes from pose error rather than from worse geometry. Under B, only SAF3R remains significantly worse than dense attention (+0.020 m). LMSA-fast has a lower Chamfer distance than dense attention under A, but this is an artefact of aligning on its less accurate cameras, not better geometry. Two ETH3D scenes (relief_2 and relief) carry 73% of this improvement, and under B, which does not depend on the cameras, its change is +0.034 m. All arms were scored from float32 depth; storing depth as float16, as was done for the later runs, changes the Chamfer distance by at most 10−310^{-3} m. Further metrics for every arm and split are released with the code: rotation and translation accuracy at 5, 15 and 30 degrees, pose AUC at the same thresholds, ATE and RPE.

Cost and timing

This section reports the measured and modelled cost of the evaluated methods.

Tanks and Temples at N=100

This registered evaluation asks how the method performs on longer inputs, with more frames per subset, on the Tanks and Temples scenes.

Each of the five scenes of Table 3 was run with 5 seeds under the strided rule of Section B.1. At N=100N=100, however, the strided rule gives only 4, 3, 5, 3 and 3 distinct frame subsets for Barn, Caterpillar, Courthouse, Meetingroom and Ignatius. Some subsets are therefore repeated, and repeated subsets enter the means more than once. On the four low-ρ\rho scenes LMSA loses 2.17 AUC@5 (CI [1.33, 3.09]); on Ignatius it loses 0.80; and on the held-out Truck, scored separately, it loses 0.17. Over all five scenes it gains 10.71 AUC@5 over AVGGT(9) at an indicative 1.32× its time. The times in this run are medians of single timed forwards, so they serve only as indications. GPA-VGGT and SAF3R were run afterwards (post hoc) on the same scenes, seeds and dense reference. Over the five scenes, GPA-VGGT loses 1.61 AUC@5 (CI [0.16, 4.19]) and SAF3R loses 5.65 (CI [2.89, 8.93]). The paired difference between LMSA and GPA-VGGT is −0.29 (CI [−1.77, 2.00]), which is not significant, whereas LMSA is 3.76 ahead of SAF3R (CI [1.38, 6.55]). Per scene group, the losses of LMSA, GPA-VGGT and SAF3R are:

  • 2.17, 1.91 and 6.75 on the four low-ρ\rho scenes;

  • 0.80, 0.40 and 1.28 on Ignatius;

  • 0.17, −0.61 and 1.39 on the held-out Truck, so on Truck GPA-VGGT is ahead of LMSA by 0.77 (CI [0.13, 1.43]); with each distinct frame subset counted once, this lead is 0.99 (CI [−0.08, 2.09]).

Repeated subsets give bit-identical runs, so they can also be counted only once. Doing so moves every five-scene mean by at most 0.2 AUC@5: LMSA loses 1.95 (CI [1.16, 2.82]), GPA-VGGT 1.60 (CI [0.10, 4.20]) and SAF3R 5.52. The difference between LMSA and GPA-VGGT stays non-significant (−0.35, CI [−1.83, 1.96]).

VGGT is −0.29 (CI [−1.77, 2.00]), which is not significant, whereas LMSA is 3.76 ahead of SAF3R (CI [1.38, 6.55]). Per scene group, the losses of LMSA, GPA-VGGT and SAF3R are:

  • 2.17, 1.91 and 6.75 on the four low-ρ\rho scenes;

  • 0.80, 0.40 and 1.28 on Ignatius;

  • 0.17, −0.61 and 1.39 on the held-out Truck, so on Truck GPA-VGGT is ahead of LMSA by 0.77 (CI [0.13, 1.43]); with each distinct frame subset counted once, this lead is 0.99 (CI [−0.08, 2.09]).

Long sequences: 333 and 800 frames

This evaluation measures accuracy and runtime on long video sequences, where the cost of dense attention matters most. Following the AVGGT protocol [40], 7-Scenes test sequences were run in two settings, both with the frozen recipe and no retuning. The first uses every third frame (N=333N=333 on six sequences, 167 on stairs). The second uses 800 frames on four sequences: chess-03, office-02, fire-03 and redkitchen-03, the last of which was held out from all design choices. Dense attention at N=800N=800 needs more than 16 GB of GPU memory. Dense attention, and any other method that also exceeds this memory, was therefore run by a chunked engine that writes activations to disk between segments and is bit-identical to the reference forward; the reported times include this disk I/O. At these lengths, the official implementations of GPA-VGGT and SAF3R also exceed 16 GB. For N=333N=333, GPA-VGGT was therefore run through a port that applies its head-merging step two heads at a time, which is arithmetically identical to the original. At N=16N=16, 32 and 100, the port matches the official outputs within twice GPA-VGGT’s own run-to-run variation over three official runs. This criterion was adopted after a single-pair version of it failed at N=16N=16 on a low noise sample. SAF3R, and GPA-VGGT at N=800N=800, were not run. At N=333N=333, LMSA loses 0.60 AUC@5 against dense attention (paired 95% CI [−0.27, 1.87]) and 0.16 AUC@30, and its aggregator runs 2.7× faster. For comparison, AVGGT(4) loses 0.06 and 0.45 at 3.9×, and AVGGT(9) loses 1.69 and 1.08 at 5.4×. GPA-VGGT exceeds dense attention by 0.82 AUC@5 (CI [0.20, 1.77]) and exceeds LMSA by 1.42 (CI [0.20, 3.55]), at 2.9× (2.6× if the first call in each process, which includes kernel autotuning, is kept). This is in line with the advantage of GPA-VGGT on redundant views at N=32N=32. At N=800N=800 (Table S11), LMSA stays within 1.6 AUC@5 of dense attention on every sequence and runs 3.3× faster. AVGGT is faster still, at 5.1–8.1×, but it is less faithful to dense attention: its AUC@5 departs from dense attention by up to 7.9, upward on fire-03 and redkitchen-03, and its AUC@30 falls below dense attention on three of the four sequences. These speed-ups include the disk I/O of the chunked runs, which affects all dense runs and one run each of AVGGT(4) and LMSA; without that I/O, the speed-ups are 3.3, 5.1 and 7.8×. The AVGGT paper reports a similar effect, with AVGGT(9) above dense VGGT at 800 frames (7-Scenes AUC@30 77.4 against 74.2). This suggests that frame-local pooling can act as a regulariser on long redundant video; LMSA, which aims to reproduce dense attention, does not show this effect. LMSA-auto is a variant that selects the exact layers per sequence from the in-pass Γˉ\bar{\Gamma}, rather than using the frozen exact layers. At this length it changes only layer 16, whose Γˉ\bar{\Gamma} lies at 1.41–1.74 and falls below the cut-off on three of the four sequences; in contrast, Γˉ\bar{\Gamma} is 1.9–8.2 for layers 11–15 and at most 0.97 for the hub layers. The confirmatory test of Section C.7 also covers these N=333N=333 sequences.

Methodchessofficefirekitchen∣Δ∣5|\Delta|_{5}Δ30\Delta_{30}Speed-up
AVGGT(4)−0.85+0.96+7.85+4.283.48−0.115.1×
AVGGT(9)−1.92+0.32+5.93+5.043.30−0.908.1×
LMSA−0.33+0.94−1.57+1.351.05+0.593.3×
LMSA-auto−0.13+0.94−0.97+1.350.85+0.703.4×

Table S11. 800 frames on four 7-Scenes sequences, one frame subset each. Per-sequence AUC@5 minus that of dense attention (positive = above dense), mean ∣Δ∣|\Delta| of AUC@5, mean Δ\Delta of AUC@30, and mean end-to-end speed-up over dense attention. Dense AUC@5: 43.2, 15.4, 53.1 and 6.7. Bold: the two lowest mean absolute differences (both LMSA variants), the largest departure from dense attention and the largest speed-up.

Every tenth frame (post hoc). This run uses N=100N=100 frames taken at stride 10 from each of the 18 test sequences (7 scenes). LMSA loses 0.53 AUC@5 against dense attention (paired 95% CI [0.14, 0.94]), whereas GPA-VGGT is 0.41 above dense attention (CI [−0.03, 0.91]). AVGGT(4), AVGGT(9) and SAF3R lose 1.15, 3.57 and 1.58, with CIs that include 0. Mean end-to-end times are 16.0 s for LMSA, 16.6 s for GPA-VGGT and 30.8 s for dense attention. So, as at stride 3, GPA-VGGT is slightly more accurate than LMSA on this redundant video, at a similar time. Back at stride 3, LMSA-auto chose a different layer set from the frozen recipe on 2 of the 7 sequences, and its mean AUC@5 is 34.12, against 34.24 for the frozen recipe, so choosing layers per sequence made almost no difference.

Outdoor scenes: RealEstate10K

This test asks whether LMSA keeps its accuracy on outdoor scenes, a domain that none of the main-table datasets covers well. It is post hoc relative to the main table, but its scene list, frame rule, methods and predictions were registered before it ran, and it is reported in full.

Data and scene selection. RealEstate10K [55] consists of real-estate video clips with camera poses estimated from the videos. The test clips come from the preprocessed release of the pixelSplat authors [6]; their full test split could not be downloaded, so the candidate pool is the 41 test clips of their published subset. Every clip was labelled indoor or outdoor from thumbnails before any model was run, and the test set is every outdoor clip with at least 32 frames: 9 clips of yards, facades, pools, porches and decks. Each of 5 frame subsets takes N=32N=32 frames spread over the whole clip. The poses are known only up to scale, so the scores use the relative rotation and translation direction, as in the main table; the dataset has no depth.

Result. LMSA loses 0.22 AUC@5 against dense attention (95% CI [−1.55, 2.33]) and 0.40 AUC@30 (CI [−0.51, 1.33]), so on these clips it is practically as accurate as dense attention. GPA-VGGT is 0.76 above dense attention, and the difference between the two methods is not significant (+0.98, CI [−1.73, 4.05]). AVGGT(4), AVGGT(9) and SAF3R lose 2.11, 5.14 and 2.49; only the gap to AVGGT(9) is significant (−4.92, CI [−9.89, −0.20]). Per clip, LMSA loses more than 3 AUC@5 on one clip (at most 7.0, a covered porch), as does GPA-VGGT, whereas AVGGT(9) does so on four clips.

What this test does not show. Every clip is of high redundancy (ρ\rho between 16 and 30), because a real-estate video revisits the same space many times. The test therefore shows that LMSA carries over to outdoor scenes, but it does not test the low-redundancy regime that the main table targets. With 9 clips the intervals are wide, and GPA-VGGT’s negative loss comes mostly from one clip with moving people, on which it beats dense attention by 14.4. Without that clip and one clip that fades to white, LMSA loses 0.82 (CI [−0.85, 3.19]) and GPA-VGGT 1.05 (CI [−0.22, 2.71]). The frames come from YouTube videos and are used for research only; the poses are released under CC BY 4.0.

Confirmatory test on fresh datasets

This registered test asks whether two variants of LMSA generalise to fresh datasets and long video. The two variants were chosen after the main results, so they were tested on data not used for any design choice, with their predictions registered before the runs. LMSA-Accurate uses more hubs (1024 in layers 0–8, 2048 elsewhere; Section F.2) to reduce sampling noise. LMSA-Efficient saves cost by merging 20% of the keys in layers 11–16 with the operator of GPA-VGGT (Section D.5). The test uses three datasets: NRGBD (9 scenes, 3 keyframe subsets each, N≈120N\approx120), MPI Sintel (14 sequences, 5 random 32-frame subsets each) and the N=333N=333 7-Scenes sequences above. Both variants and LMSA were run with three hub draws per subset, and the intervals come from a nested bootstrap over scenes, subsets and draws (10,000 resamples). The registered predictions were:

Table S13. Scene-level tests on the 13 unseen low-ρ\rho scenes: dd = AUC@5 of LMSA minus that of the baseline, per scene (5-seed means). Two-sided Wilcoxon signed-rank and exact sign-flip tests on dd; Holm correction over the four baselines within each test. Bold: significant after Holm correction.

BaselineMean ddAheadWilcoxon (Holm)Sign-flip (Holm)
AVGGT (σ=4\sigma=4)+8.8913/13<0.001 (0.001)<0.001 (0.001)
AVGGT (σ=9\sigma=9)+17.5413/13<0.001 (0.001)<0.001 (0.001)
GPA-VGGT+1.048/130.340 (0.340)0.165 (0.165)
SAF3R+3.1511/130.017 (0.034)0.064 (0.128)

Table S13.

The Table S14 shows the two diffuse groups in this isolated setting. In the early layers, adding the special tokens of other frames to the own frame is not enough to recover accuracy. A lattice with a mean token nearly closes the gap on the low-ρ\rho scenes but costs 1.0 on TUM fr3, whereas 256 hubs close the gap on every scene tested. In the late layers, the AVGGT key set loses 15.0 AUC@5 on fr3. Correcting the mean token’s deficit Γ\Gamma recovers most of this loss (Finding 1), which supports the deficit as an explanation without isolating it. 1024 hubs remove the deficit without any correction term, whereas 256 hubs are too few and lose 11.4 on courtyard. One caveat remains: the 1024-hub courtyard loss of 1.0 has a seed-level 95% upper bound of 4.4, which is above the pre-registered bound of 4.

Table S14. Diffuse layers in isolation. AUC@5 lost against dense when only the named layers are sparse; development scenes, 5 seeds, †: 2 seeds. “mean”: AVGGT’s count-biased mean token. Bold: the key sets of LMSA.

Layers and key setfr1courtETH-ofr37S
Early, 0–8 (own frame exact)
frame-local (AVGGT)5.633.11.50.5−1.1
+ specials5.924.2−0.1−1.0
+ specials, σ=16\sigma=16 lattice, mean−0.61.6−1.0−0.4
+ specials, 256 hubs (LMSA)0.20.9−0.10.0
Late, 9 and 17–23
σ=9\sigma=9 lattice, frame 0, mean†−4.55.615.0−
+ cumulant correction†−3.62.55.4−
own, specials, 256 hubs†−11.42.01.8−
own, specials, 1024 hubs (LMSA)1.01.00.31.20.1

Table S14.

The Table S14 shows the two diffuse groups in this isolated setting. In the early layers, adding the special tokens of other frames to the own frame is not enough to recover accuracy. A lattice with a mean token nearly closes the gap on the low-ρ\rho scenes but costs 1.0 on TUM fr3, whereas 256 hubs close the gap on every scene tested. In the late layers, the AVGGT key set loses 15.0 AUC@5 on fr3. Correcting the mean token’s deficit Γ\Gamma recovers most of this loss (Finding 1), which supports the deficit as an explanation without isolating it. 1024 hubs remove the deficit without any correction term, whereas 256 hubs are too few and lose 11.4 on courtyard. One caveat remains: the 1024-hub courtyard loss of 1.0 has a seed-level 95% upper bound of 4.4, which is above the pre-registered bound of 4.

Further evidence for the design

This post-hoc evidence examines which keys the matching layers rely on and why the other components of the design were chosen.

Why the design

Diffuse layers in isolation

This section examines the diffuse layers on their own: each diffuse group is made sparse while every other layer stays dense, so that its loss can be measured without interference from the other groups.

The Table S15 shows which key sets matter in layers 10–16. “Correspondence” causes substantial losses, as does “highest lift,” whereas removing random keys has almost no effect.

All of these predictions held (Table S12). The predictions concern mean differences, however, and the 95% intervals of LMSA-Accurate reach −0.78, −0.65 and −0.55 AUC@5. The test therefore does not establish equivalence to dense attention within 0.5; it shows only that the mean differences fall inside that bar. LMSA-Accurate is slightly below dense attention on NRGBD (−0.41, CI [−0.78, −0.10]), which is within the bar. It is closer to dense attention than LMSA on every dataset (LMSA: −1.10, −0.62 and −0.89). Its point maps lie within 1.3% of dense attention, and on Sintel its depth error equals that of dense attention (AbsRel 0.290). LMSA-Efficient is 1.16× faster than LMSA at N=333N=333 (aggregator, steady state), 1.11× at N≈120N\approx120 and 1.02× at N≤32N\le32, so its saving grows with the number of frames. LMSA-Accurate costs no more than LMSA (0.95–1.00× its time). One registered baseline prediction failed in part. AVGGT(9) has the lowest AUC@5 everywhere, as predicted, but SAF3R is above AVGGT(4) on NRGBD and Sintel instead of being among the lowest. On Sintel, GPA-VGGT is slightly above dense attention (28.51 against 28.27 AUC@5), as it was on the 7-Scenes video.

Table S12. Confirmatory test (registered before the runs; mean and nested 95% CI). Left: LMSA-Accurate minus dense attention. Middle: LMSA-Efficient minus LMSA. Right: draw-to-draw SD of AUC@5 within a subset. Bold: the pose-accuracy difference of LMSA-Accurate to dense attention (first column), and the lowest draw SD per dataset.

Accurate − denseEfficient − LMSADraw SD of AUC@5
DatasetAUC@5AUC@30AUC@5AUC@30Acc.Eff.LMSA
NRGBD−0.41 [−0.78, −0.10]−0.08+0.32 [−0.25, 0.94]+0.070.681.441.40
Sintel−0.30 [−0.65, 0.02]−0.08+0.27 [−0.38, 0.93]+0.041.302.012.04
7-Scenes+0.00 [−0.55, 0.48]−0.06+0.55 [−0.77, 2.44]−0.000.790.851.11

Table S12.

Figure S1. Reconstructions on two ETH3D scenes [35]. Each panel shows the predicted depth, unprojected with the predicted cameras (86% most confident pixels), and the predicted cameras (frusta in the column colour), both mapped to the ground-truth frame by the least-squares similarity transform between predicted and ground-truth camera centres; black frusta are the ground-truth cameras, and all panels of a row share one viewpoint. Top: courtyard, a development scene, subset 0. Bottom: terrace, an unseen low-redundancy test scene, subset 2. In each scene the subset is the one on which GPA-VGGT loses the most AUC@5 against dense attention (13.0 and 14.6 points), chosen by GPA-VGGT’s loss, not by the result of LMSA. Titles give that subset’s pose AUC@5 and, in brackets, its difference from dense VGGT-1B. On courtyard the cameras of AVGGT(4) are visibly displaced and its facades blurred; smaller pose errors are hard to see at this scale. Courtyard was used to design LMSA, so its +9.4 there is not a test result.

Keys. Table S15 collects the knockouts, in which one set of keys is removed from the matching layers and the loss is measured. Removing the correspondence windows, which hold very little attention mass, costs far more than removing as many of the heaviest keys, and removing random keys costs nothing. The highest-lift keys are those whose attention from a given query most exceeds their column mass, that is, the keys most specific to that query. Removing them is the most destructive of all, even though only 13% of them lie in correspondence windows.

Early layers. Making only layers 0–2, 3–5 or 6–8 frame-local costs 18.4, 12.7 and 17.2 AUC@5 on courtyard. Each block of three early layers therefore carries cross-view information on wide-baseline input, so no early block can simply be restricted to its own frame.

Proposal of the importance sample. A layer-local audit measured the output error of each layer on its own, for the far-field keys of layers 10–16. In it, 256 samples drawn in proportion to the pilot’s column mass have about half the output error of 256 uniform samples (0.109 against 0.203), so the column mass is a useful proposal.

Shared versus per-query blocks. This test asks whether one block of keys shared by a tile of queries can replace a separate window for each query. Blocks of 6×10 keys centred by the oracle lose 0.93 AUC@5 (CI [−0.84, 2.80]) relative to per-query 3×3 oracle windows, pooled over the low-ρ\rho scenes. On courtyard alone, the loss is 2.62 (CI [−0.31, 5.55]). The pooled result therefore supports shared blocks, but the support is weaker per scene.

Denser lattices instead of routing. Without correspondence keys, the lattice must be dense to stay accurate: at σ\sigma=16, the loss relative to σ\sigma=9 with oracle keys is 3.86 AUC@5 (CI [1.45, 6.64]) on the low-ρ\rho scenes. Making every visible frame dense also recovers less than oracle 3×3 windows do (courtyard 49.3 vs. 54.7), so on courtyard adding whole frames helps less than adding the few correct keys.

Refining the router later does not help. The router finds correspondences from the features of one layer, so a natural question is whether deeper features would find them better. The best correspondence readout from a sparse pass recalls 0.450 of the 3×3 targets at layer 9 and also 0.450 at layer 12, so routing from deeper features gains nothing.

Composition study details

This pre-registered composition study asks how the layer groups behave when each is made sparse alone and when all are made sparse together.

Table S16 gives the arms behind § 3.3: arms in which a single group is sparse, and arms in which one group of LMSA-fast is returned to dense attention. Three results stand out. The diffuse groups cost little when made sparse alone. The sparse matching layers alone already exceed the thresholds on courtyard and TUM fr3_long_office. Only returning layers 11–16 to exact attention removes most of the composed loss, which points to the matching layers as the group to keep exact.

Armfr1courtETH-ofr37Slow-ρ\rho
LMSA-fast with one group dense
layers 0–8 dense0.362.101.305.911.611.25
layers 9–10 dense0.915.302.517.661.552.91
layers 11–16 dense0.10−0.931.170.000.770.11
layers 17–23 dense−0.286.112.294.681.672.71
only the named layers sparse
0–8 (256 hubs)−0.201.290.56−0.160.280.55
9–10 (1024 hubs)−0.58−0.480.09−0.14−0.46−0.33
11–16 (σ=4\sigma=4, blocks)−0.324.151.775.401.331.87
17–23 (1024 hubs)0.750.360.821.370.990.64

Table S16. Composition study: swap-backs and single groups. AUC@5 lost against dense, development scenes, seeds 0–1. Top: LMSA-fast with one group returned to dense attention. Bottom: only the named group sparse. Bold: in the TUM long-office and pooled low-ρ\rho columns, the matching-layer rows have the lowest loss of the top block and the highest of the bottom block.

Sequence of the study. The study ran in three pre-registered steps, each with acceptance thresholds and a stop rule (Sec. B.4). (1) In the composed pilot (2 seeds), LMSA-fast lost 11.55 on courtyard (both seeds above 11.5), which fired the stop rule. Returning layers 11–16 to dense attention recovered 3.66 AUC@5 pooled; this was the only group whose return recovered more than 2.5, so it was named as the group to repair. (2) Repairs were tried in a fixed order, cheapest passing arm first. A σ=2\sigma=2 lattice passed the 2-seed screen but then lost 4.92 on courtyard over 5 seeds. Exact layers 13–15 lost 4.10 on TUM fr1_room over 5 seeds. Exact layer 14 alone lost 10.28 on courtyard already in the screen. (3) The fallback, exact layers 11–16, passed every threshold on 5 seeds. The later fused runs give a smaller courtyard loss for LMSA-fast than the masked pilot did. In the fused 5-seed runs of Table 2, LMSA-fast loses 5.9 on courtyard, close to the sum of its single-group losses (5.3); seed 1 alone loses 1.81 there, against 11.53 in the masked pilot. Table S17 gives AUC@30 for the main rows of Table S19. It shows that the AUC@30 loss of the sparse recipes also comes almost entirely from layers 11–16. Depth is barely affected: AbsRel changes by at most 0.009 in every arm. Table S18 shows how much the loss of LMSA varies from seed to seed: the 5-seed means pass the bars, but two individual seeds lose more than 7 AUC@5. The composed loss could in principle come from the sparse early layers disturbing the router of LMSA-fast, but they do not: vote recall is 0.6806 in the composed pass against 0.6805 in a dense pass, and the gated orphan share is 0.0668 against 0.0670. The pilot’s key selections (hub

Layers 11–16fr1courtETH-ofr37Slow-ρ\rho
exact (LMSA)1.410.240.810.140.550.82
σ=4\sigma=4 + blocks†^{\dagger}4.062.887.010.951.934.65
σ=2\sigma=2 + blocks2.761.212.680.370.582.22
13–15 exact7.100.803.620.411.273.84
only 11–16 sparse†^{\dagger}5.651.256.830.911.414.57

Table S17. AUC@30 lost against dense in the composition study (development scenes; †^{\dagger}: seeds 0–1). Bold: lowest loss per column.

Scenes0s1s2s3s4
TUM fr1_room0.52−0.320.817.26−1.81
ETH3D courtyard−3.511.658.192.46−3.31
ETH3D office−0.092.42−1.13−0.173.55
TUM fr3_long_office−2.502.50−0.521.902.18
7-Scenes office1.010.520.360.20−0.32

Table S18. LMSA per seed (AUC@5 lost against dense, seeds 0–4, masked implementation). Bold: the two largest seed-level losses.

counts, orphan shares, frame-0 shares) are also nearly identical between the composed and single-group passes. The extra loss of the composed pass therefore arises in the activations, not in a different choice of keys; per-layer errors were not measured.

LMSA-fast: the routed variant

This section describes LMSA-fast, the faster variant that routes correspondence keys instead of keeping the matching layers exact, and the evidence behind its switch. LMSA-fast keeps the diffuse layers of LMSA and replaces the exact matching layers 11–16 by the sparse key set

K(t)=F0∪S∪Λ4∪{t}∪⋃j≠iBj(u),(13)\mathcal{K}(t)=\mathcal{F}_0\cup\mathcal{S}\cup\Lambda_4\cup\{t\}\cup\bigcup_{j\ne i}\mathcal{B}_j(u), \tag*{(13)}

plus a count-biased mean token over the remaining keys. Here F0\mathcal{F}_0 is the reference frame, S\mathcal{S} the special tokens, Λ4\Lambda_4 one patch per 2×22\times2 cell in every frame, and tt the query itself. Bj(u)\mathcal{B}_j(u) is the routed block of the query’s 4×84\times8 tile uu in another frame jj; it is included only when its switch is on. The query’s own frame contributes only its lattice keys, because keeping the own frame dense was found to cost 7.9 AUC@5 on courtyard.

Router. The router finds, for each tile of queries, where the corresponding region lies in every other frame. It runs once per pass on the input of layer 9’s attention, with patch tokens normalised to unit length. For every patch query and every other frame jj, it finds the four most similar tokens by cosine similarity. The hits of all queries in a tile then vote for a displacement in frame jj; the vote histogram is smoothed with a 3×33\times3 box, and its mode Δj∗(u)\Delta_j^*(u) is taken as the tile’s displacement. The block Bj(u)\mathcal{B}_j(u) is the tile’s footprint shifted by Δj∗(u)\Delta_j^*(u) and dilated by one token (6×106\times10 keys), and it is shared by all 32 queries of the tile. An orphan query is one with no exact correspondence key in any view. Without gating, the blocks cut the share of orphan queries from 0.371 / 0.193 / 0.288 to 0.035 / 0.021 / 0.076 on TUM fr1 / courtyard / ETH3D office at σ=4\sigma=4, so most queries regain a correspondence key. The router’s search grows as N2N^2 (54 / 214 / 521 ms at N=32N=32 / 64 / 100); letting only 8 of the 32 queries per tile vote still keeps block recall at 0.637. The router’s input is unaffected by the sparse early layers: its vote recall is 0.681 both in the composed pass and in a dense pass. Not every routed block is right, and a wrong block can harm accuracy. A decision-theoretic switch (Section D.4.1) therefore keeps a block only when its chance of being right exceeds the break-even precision p⋆=H/(H+og)p^\star= H/(H+og). On courtyard p⋆=3.4%p^\star= 3.4\%, so even fairly unreliable blocks pay off; on redundant scenes (o→0o \to0), p⋆→1p^\star\to1, so blocks are almost never worth keeping.

Break-even switch

Consider one query and one routed slot with probability pp of holding the query’s correspondence. If the query is an orphan (probability oo) and the slot is right, the slot removes it from the orphan share. Pose accuracy rises by gg per unit of orphan share, so the expected gain is p o gp\,o\,g. If the slot is wrong (probability 1−p1-p), its keys are amplified like right ones and cost HH. The expected value is p o g−(1−p)Hp\,o\,g-(1-p)H, and it is positive iff

p>p⋆=HH+og.(14)p > p^\star= \frac{H}{H+og}. \tag*{(14)}

The numerator is the cost of a wrong slot and ogog the expected gain of a right one. The gain saturates per query, because one right block is enough to rescue an orphan, whereas the harm adds up over every wrong slot. On redundant input there are almost no orphans (o→0o \to0), so p⋆→1p^\star\to1 and the switch should stay off. The three inputs gg, HH and oo are estimated on courtyard at σ=4\sigma=4 from 5-seed runs. Oracle correspondences remove an orphan share of 0.193 and gain 8.07 AUC@5, so g=8.07/0.193=42g=8.07/0.193=42. Similarity keys placed only in the 24.3 slots per query whose frame does not see the point cost 6.87, so H=6.87/24.3=0.283H=6.87/24.3=0.283. With o=0.193o=0.193 this gives p⋆=3.4%p^\star=3.4\%. Figure S2 shows the curve, and Table S20 lists the switch signals measured on saved router outputs. Of these signals, the vote-peak rule is the one used by LMSA-fast. The break-even accounting is consistent with earlier runs: on courtyard it predicts net changes of −1.05-1.05 and +0.3+0.3 AUC@5 for ungated similarity windows and vote blocks, against −0.69-0.69 and −0.02-0.02 measured. The model’s predicted gain for the vote-peak switch (≈+3.8\approx+3.8) is only an estimate, because the switched blocks were measured only inside the composed recipe (Table S19), not on their own.

Figure S2. Break-even precision p⋆=1/(1+r)p^\star= 1/(1+r) of a routed slot (Eq. (14)) against the benefit-to-harm ratio r=og/Hr=og/H. Black dots show the development scenes. Horizontal lines show measured slot precisions; above the curve, switching a slot on pays. On courtyard the per-slot rule predicts a net loss for always-on slots (−1.05-1.05 AUC@5, Table S20), so the switch helps by removing wrong slots. On 7-Scenes office it should be off.

Table S20. Switch signals on saved layer-9 router outputs (courtyard unless stated). “Right-slot precision” counts a slot as right if its frame sees the point and its window holds the target. The last column is the net AUC@5 change on courtyard predicted by the break-even model, not a measurement. The vote-peak gate is a block-level rule; for it, and for the in-degree rule, only visibility precision is available. Bold: the rule used by LMSA-fast, the predicted net loss of always-on slots, and the highest predicted net among the signals with a right-slot precision.

SignalSlots keptVisible keptRight-slot precision, fr1 / court / ETH-oWrong / queryPredicted net
Always on1.001.000.06 / 0.09 / 0.1224.3−1.05
Mutual nearest neighbour at layer 90.210.380.20 / 0.20 / 0.363.9+2.63
Layer-12 attention top-1 agrees with hit0.250.360.20 / 0.20 / 0.335.5+1.98
same, inside the epipolar band0.290.430.23 / 0.20 / 0.366.0+2.53
Mutual NN and layer-12 agreement0.070.170.41 / 0.35 / 0.591.2+1.93
Hit lands on a non-hub (in-degree ≤ 2)0.350.53visibility 0.327.4+2.2
Vote-peak share ≥ 0.3 (LMSA-fast)0.420.68visibility 0.348.6≈ +3.8 (estimate)

Table S20.

Thinner matching layers

This post-hoc study tests how far the matching layers can be thinned before accuracy suffers.

These tests are post hoc: their plans were registered after the results of Table 2 were in, before any of their own runs, with predictions and no pre-registered bar. They ask whether layers 11–16 can be thinned in other ways than the truncated lattices of Section 4.3. In every arm, the other layers keep the key sets of LMSA (hub256 in layers 0–8, hub1024 in 9–10 and 17–23), so only the matching layers change. The merge arms apply the key merging of GPA-VGGT in layers 11–16 at merge ratios 0.2, 0.5 and 0.8, so that 80, 50 and 20% of the keys remain. As a check, the merge arm at ratio 0 reproduces LMSA bit for bit. The top-kk arm uses, in layers 11–16 only, the per-query top-kk frame selection of SAF3R heads, which keeps 21–77% of the keys. Its layers were checked to match SAF3R’s own forward bit for bit. Every arm ran on the 5 development and 21 unseen test scenes with 5 seeds and was scored as in Table 2 (Table S21).

Table S21. Thinner matching layers (post hoc), N=32N=32, 5 seeds. Keys: share of keys kept in layers 11–16. FLOPs: reduction against dense attention at N=100N=100. Loss: AUC@5 lost against dense attention on the 13 unseen low-ρ\rho scenes. Δ\Delta: paired AUC@5 difference to LMSA (negative = less accurate) with hierarchical 95% CIs. Red: CI below zero. Bold: the lowest and highest loss, and the only thinner arm that is not significantly worse on the low-ρ\rho scenes.

Layers 11–16KeysFLOPsLossΔ\Delta low-ρ\rhoΔ\Delta high-ρ\rho
Exact (LMSA)100%2.58×1.38––
Merge 0.280%2.86×1.86−0.49 [−1.24, 0.12]+0.06 [−0.35, 0.49]
Merge 0.550%3.49×3.00−1.62 [−2.53, −0.80]+0.22 [−0.20, 0.67]
Merge 0.820%4.25×7.11−5.74 [−7.57, −3.93]−0.43 [−1.14, 0.13]
Top-k frames21–27%4.34×5.60−4.22 [−5.70, −2.91]−0.75 [−1.77, 0.33]
Shared hubs (1024)own frame + 10245.40×29.17−27.79 [−34.62, −21.22]−14.88 [−23.04, −7.49]

Table S21.

On the low-ρ\rho scenes, accuracy falls steadily as more keys are removed, relative to LMSA:

  • keeping 80% of the keys is not significantly worse than exact attention;

  • keeping 50% costs 1.6 AUC@5, and keeping 20% costs 5.7;

  • top-kk frame selection costs 4.2, which is no better than SAF3R with its own early and late layers (−1.07-1.07, CI [−3.58-3.58, 1.48]).

The most extreme arm replaces layers 11–16 by the shared hub keys of the diffuse layers, so that no layer stays exact. It costs 27.8 AUC@5 on the low-ρ\rho scenes and 14.9 even on the high-ρ\rho scenes, yet its aggregator runs only 1.1, 1.4 and 1.5× faster than that of LMSA at N=32N=32, 64 and 100 (same session). This means that the matching layers depend on query-specific keys, which a shared sample cannot provide, and that keeping these layers exact is cheap relative to their effect on accuracy. On the high-ρ\rho scenes, all other arms stay within 0.8 AUC@5 of LMSA, so thinning matters mainly where views overlap little. The three arms that use over 4× fewer FLOPs than dense attention at N=100N=100 lose 4.2, 5.7 and 27.8 AUC@5 against LMSA on the low-ρ\rho scenes. A fourth arm that keeps accuracy there retains at least 80% of the keys in the matching layers, so its cost stays quadratic in NN. In other words, no tested way of thinning the matching layers both keeps accuracy and removes their quadratic cost. A final arm replaces the early and late layers of SAF3R by the key sets of LMSA. It recovers none of the baseline’s low-ρ\rho loss, so that loss therefore sits in its matching layers.

Why the matching layers resist thinning

This post-hoc analysis asks why the matching layers resist thinning, by looking directly at their attention.

This attention audit is post hoc. Its plan, screening rule and amendments were registered before its GPU runs, and its results gate no claim. It asks whether any selection of keys, that is, any rule that keeps a small subset of keys exactly, could replace exact attention in layers 11–16. The candidate rules borrow ideas from other fields:

  • an epipolar band around each query’s line in every other frame, computed from ground-truth poses (an oracle);

  • reuse of each query’s heaviest keys across layers;

  • sharing heavy keys among the queries of a 4×8 tile;

  • block-wise upper bounds on qTkq^{T}k;

  • a per-(query, head) switch between exact attention and hubs driven by s2s^2.

In each case, the keys outside the selected set were covered by the unbiased hub estimator of LMSA, so a rule only has to find the keys that matter most. To evaluate the rules, exact fp32 attention was computed on the 5 development scenes with 2 seeds, for 2048 random patch queries in each of the 16 heads of layers 11–16. A rule was screened as viable if, even for the worst 10% of (query, head) pairs, it kept at least 90% of the attention mass with at most 25% of the keys.

No rule passed in any of the six layers.

  • Epipolar band. Even with true poses, a band of ±5 patches (15% of the keys) kept only 21–52% of the mass of the median peaked pair, where a peaked pair is one with Γrow≥1.5\Gamma_{\mathrm{row}} \ge1.5.

  • Each query’s own 256 heaviest keys (an oracle that no practical rule can beat) kept only about 60%.

  • Reuse across layers. Heavy keys do not persist from layer to layer: a query’s heaviest cross-frame key matched that of layer 11 in under 2% of cases, so keys chosen in one layer are a poor guess for another.

  • Block bounds and the s2s^2 switch. Both reach exact accuracy, but only by keeping 51–99% of the keys, which saves at most 12% of the FLOPs of LMSA.

The switch uses the proxy s2≥3s^2 \ge3 in place of Γrow≥1.5\Gamma_{\mathrm{row}} \ge1.5; the two agree on 89–96% of (query, head) decisions (κ=0.71\kappa= 0.71–0.91). Table S22 shows why these rules fail. Even where attention is peaked, it is spread over many keys: half the mass of a peaked pair lies on 48–269 keys, and 90% lies on 349–2540, out of about 33,000. Most of this mass falls on other frames, and the geometric structure, although visible, is small. On peaked pairs, the 3×3 window around the true correspondence is enriched up to 74-fold in layer 14, meaning that it receives far more attention than its size alone would give it, yet it holds only 1.6–12% of the cross-frame mass; a ±9-patch epipolar band (16% of the keys) holds 29–56%. In contrast, layers 10 and 17, which LMSA serves with hubs, are mostly diffuse and show no such concentration. The matching layers thus sit between the two cases that cheap approximations handle well: for the approximations tried, their attention is too broad for any small key set tested, and too focused for a shared sample. The bound of about 4× is therefore an observed bottleneck under the training-free approximations tried, not a lower bound on sparse matching attention. It suggests that learned compression, rather than selection, is the route past it.

Table S22. Breadth of attention in the matching layers (post hoc; 5 development scenes × 2 seeds, 2048 random queries, all heads). Peaked: (query, head) pairs with Γrow≥1.5\Gamma_{\mathrm{row}} \ge1.5. neffn_{\mathrm{eff}}: exponential of the attention entropy, median. k50k_{50}, k90k_{90}: median number of keys holding 50% and 90% of the mass. Band and window: share of the cross-frame mass inside the ground-truth epipolar band (±9 patches, 16% of cross-frame keys) and the 3×3 correspondence window (0.17%). Bold: the most concentrated matching layer, which still needs hundreds of keys to hold most of its mass.

LayerPeaked shareneffn_{\mathrm{eff}} (peaked)k50k_{50}k90k_{90}Cross-frameBand ±93×3 windowneffn_{\mathrm{eff}} (diffuse)
10 (hubs)0.22348049158060.930.270.01610231
110.77112115820610.880.320.04510537
120.79151022325400.900.380.0448972
130.73113817217340.870.470.07912590
140.91284483490.850.560.12010437
150.87100717113020.890.430.05011332
160.60176026924780.920.290.01611921
17 (hubs)0.08277541939440.930.210.00318871

Selector validation

Equal-cost windows of exact layers

This registered control, post hoc relative to the main table, compares the selected window with four shifted windows.

This control is post hoc: it was added after the main results, but its plan, arms and decision rules were registered before it ran, and no arm could replace the method. The ablation (Section E.4) shows that layers 11–16 beat fewer or more exact layers, but that comparison changes the cost as well as the placement. This control therefore asks a sharper question: do layers chosen by the statistic beat other placements of six exact layers at exactly the same cost? Each arm keeps six consecutive global layers exact and treats every other layer as LMSA does (256 hubs in layers 0–8, 1024 elsewhere). As a result, all arms cost 2.993 GFLOP per token at N=32N=32 and 4.758 at N=100N=100, and they differ only in where the exact layers sit. The arm with layers 11–16 is LMSA itself; its 65 test runs reproduce the main-table predictions bit for bit. The registered outcome rule labels layers 11–16 “best” if they have the lowest mean loss and the 95% CI of every other arm’s difference to them excludes 0.

Table S23. Equal-cost windows of six exact layers (AUC@5 lost against dense attention, mean and 95% hierarchical-bootstrap CI; Δ: arm minus layers 11–16). Development: 5 scenes × 5 seeds; test: the 13 unseen low-ρ\rho scenes × 5 seeds. All arms: 2.99 GFLOP per token at N=32N=32. Bold: the selected window, which has the lowest loss.

Exact layersDevelopment (5)Unseen low-ρ\rho (13)
LossΔLossΔ
9–147.60+7.07 [2.70, 13.10]8.51+7.14 [5.36, 9.18]
10–152.89+2.36 [0.80, 4.89]4.19+2.81 [1.68, 4.11]
11–16 (LMSA)0.53–1.38–
12–173.03+2.50 [0.31, 4.73]2.72+1.34 [0.49, 2.31]
13–186.09+5.56 [0.72, 12.31]4.61+3.24 [1.91, 4.85]

Layers 11–16 meet the rule on both the development and the unseen test scenes, so the registered verdict is that the selected window is best among the five tested. All four differences remain significant after Holm correction over the four comparisons (largest adjusted pp 0.001 on the test scenes, 0.025 on the development scenes, hierarchical bootstrap). The cost of a misplaced window grows with the shift: shifting the window by one layer costs 1.3 to 2.8 AUC@5 on the test scenes, and shifting it by two layers costs 3.2 to 7.1. AUC@30 gives the same ordering, with every difference excluding 0. For depth, the windows 12–17 and 13–18 are not significantly different from 11–16 (AbsRel×100 −0.10, CI [−0.33, 0.07] and −0.06, CI [−0.39, 0.18]), whereas 9–14 and 10–15 are worse. The summed Γˉℓ\bar{\Gamma}_{\ell} inside each window was recorded before the runs (11–16: 23.4; 10–15: 22.5; 12–17: 21.2; 9–14: 19.2; 13–18: 18.5). The accuracy ranking follows this sum, with two exceptions: 9–14 and 13–18 swap places, and on the test scenes 10–15 and 12–17 also swap.

Simpler layer statistics as selectors

This control asks whether simpler per-layer statistics, given the same budget of six exact layers, choose layers as good as those chosen by Γˉℓ\bar{\Gamma}_{\ell}.

This control is post hoc: it was added after the main results, but its statistic definitions, selection rule, arms and decision rules were registered before it ran, and no arm could replace the method. For each query and head, the statistics are computed over the patch keys XX of the other frames:

  • the row-free logit variance s2s^2;

  • the attention entropy HXH_X of the softmax restricted to XX;

  • the effective number of keys neffn_{\mathrm{eff}}, the exponential of the entropy over all keys;

  • the share of attention mass on XX.

The first three measure how concentrated a layer’s attention is; the last measures how much attention goes to other frames. They are aggregated in the same way as Γˉℓ\bar{\Gamma}_{\ell} (mean over 2048 random queries, median over heads, mean over 2 seeds and the 5 development scenes). Each statistic keeps exact the six layers it ranks first: highest s2s^2, lowest HXH_X, lowest neffn_{\mathrm{eff}}, or highest cross-frame mass. Every other layer is treated as in LMSA. Computing these statistics needed one dense measurement pass, which computes no accuracy metric. The selected sets were frozen after that pass and before any accuracy run, so no selection could be tuned on accuracy.

The three concentration statistics select exactly layers 11–16, both on the pooled development scenes and on each of the 5 scenes separately. Their rank correlations with Γˉℓ\bar{\Gamma}_{\ell} over the 24 layers are 0.995 (s2s^2), −0.996 (HXH_X) and −0.994 (neffn_{\mathrm{eff}}); the negative signs only reflect that low entropy and few effective keys mean high concentration. This agreement is expected, because all three measure the same property: Γ\Gamma is the divergence KL⁡(UX∥pX)\operatorname{KL}(U_X \mathbin{\|} p_X) of the attention from uniform attention, and ln⁡∣X∣−HX\ln|X|-H_X is the reverse divergence. Cross-frame mass, in contrast, selects layers 0, 1, 4, 8, 22 and 23, which have no layer in common with 11–16. It is even anti-correlated with Γˉℓ\bar{\Gamma}_{\ell} (Spearman −0.82), because the matching layers put slightly more of their mass on their own frame.

Since the three concentration statistics pick the same layers as Γˉℓ\bar{\Gamma}_{\ell}, only the mass-selected set needed an accuracy run. The mass-selected layers cost 0.4% more than the selected layers and lose far more accuracy (Table S24). They are worse on all 13 test scenes and all 5 development scenes, and the difference also holds for AUC@30 (+20.65, CI [12.73, 28.98], test) and AbsRel. With only one comparison, Holm correction leaves pp at the bootstrap floor of 0.0002. The registered verdict is therefore that layers 11–16 are best. The run with layers 11–16 reproduces the main-table predictions bit for bit. This control shows that the selection follows from how concentrated a layer’s attention is, a property that several equivalent statistics measure. Selecting by cross-frame attention mass, by contrast, keeps exact the layers that the tested recipe can thin, and leaves the layers that need exact attention to be thinned. One limit remains: the control does not test a per-layer sensitivity selector, because no single-layer thinning measurement exists for all 24 layers.

Exact layersDev. (5)Test (13)
11–16 (Γˉ\bar{\Gamma}, s2s^2, HXH_X, neffn_{\mathrm{eff}})0.531.38
0, 1, 4, 8, 22, 23 (mass)25.2128.91
Δ\Delta+24.68 [13.76, 39.68]+27.54 [21.03, 34.42]

Table S24. Mass-selected against Γˉ\bar{\Gamma}-selected exact layers (AUC@5 lost against dense attention, mean and 95% hierarchical-bootstrap CI; Δ\Delta: arm minus layers 11–16). Development: 5 scenes × 5 seeds; test: the 13 unseen low-ρ\rho scenes × 5 seeds. GFLOP per token at N=32N=32: 2.993 (11–16) and 3.005 (mass). Bold: the layers selected by Γˉ\bar{\Gamma}, which have the lower loss.

The selector applied to GPA-VGGT

This post-hoc test asks whether the selector helps GPA-VGGT on the same backbone.

This test is post hoc: its plan was registered after the results of Table 2 were in, before any of its own runs, and it has no pre-registered bar. It asks whether the two parts of LMSA that can be moved to another method, the layer selector and the hubs, improve a published training-free baseline. GPA-VGGT already protects layers 13–16 by merging only 20% of their keys, and it summarises keys more coarsely in the other layers. Three arms modify it:

  • GPA+sel extends this protection to the matching set that the statistic selects, layers 11–16; this is a change of configuration only;

  • GPA+hubs replaces the early cluster summaries of GPA-VGGT in layers 0–8 by the hub256 key sets of LMSA;

  • GPA+sel+hubs applies both changes.

Each arm ran on the 5 development and the 13 unseen low-ρ\rho scenes with 5 seeds and was scored as in Table 2 (Table S25). Protecting the selected set improves GPA-VGGT by 1.20 AUC@5 and brings it level with LMSA, at the price of 5% more FLOPs and 1–6% more aggregator time at N=32N=32–100. The development scenes agree (+1.34, CI [0.14, 2.90]). The hubs alone, in contrast, leave GPA-VGGT unchanged (+0.15), so in this baseline the gain comes from the choice of layers, not from the early estimator. The test therefore supports the statistic as a layer selector that transfers to another method on the same backbone. It does not show that hubs are better than the early cluster summaries of GPA-VGGT.

ArmLossΔ\Delta vs GPA-VGGTΔ\Delta vs LMSAAbsRelGFLOP/tok
GPA-VGGT2.42–−1.04.04652.96
LMSA1.38+1.04 [−0.34, 2.54]–.04682.99
GPA+sel1.21+1.20 [0.41, 2.12]+0.16 [−0.73, 1.00].04513.10
GPA+hubs2.26+0.15 [−0.30, 0.60]−0.89 [−2.33, 0.31].04942.96
GPA+sel+hubs1.13+1.28 [0.03, 2.55]+0.24 [−0.41, 0.90].04803.10

Table S25.

Component ablation

This post-hoc ablation measures how much each component of the frozen recipe contributes, by changing it and measuring the loss.

The development-scene arms were planned before the test runs. The test-scene arms are post hoc, added after the main results, and have no pass/fail rule. Each arm changes one component of the frozen recipe and keeps everything else unchanged (Table S26), so any change in loss can be attributed to that component.

ComponentVariantDev lossLow-ρ\rho test lossGFLOP/token
LMSA (frozen recipe)0.54 [−0.61, 1.81]1.38 [0.71, 2.01]2.99
Early layers 0–8own frame only8.41 [0.10, 22.26]–2.97
own frame + special tokens5.69 [0.24, 14.18]–2.97
+ count-biased mean token0.95 [−0.69, 3.03]–2.97
Hub estimatoruniform proposal (weights kept)1.34 [0.00, 3.64]3.34 [2.26, 4.74]2.97
no log-weight correction24.22 [9.73, 38.90]36.17 [25.88, 46.31]2.99
systematic / heavy-tail / quantile draws0.61–0.86–2.99
Hub count128 / 5122.36 [0.69, 5.40]–2.97
1024 / 20480.62 [−0.12, 1.43]0.66 [0.00, 1.26]3.06
Late layers 17–23AVGGT(9)5.12 [1.41, 9.85]–3.07
256 hubs3.14 [0.99, 6.73]–2.97
frame 0 exact0.93 [0.05, 1.87]–3.03
Layers 9–10dense0.77 [0.00, 1.70]–3.25
AVGGT(9)1.26 [0.34, 2.39]–3.02
Matching layers 11–16shared hubs (1024)25.56 [14.06, 41.46]29.17 [22.41, 36.17]2.24
σ\sigma=4 lattice + tubes (LMSA-fast)3.91 [2.04, 5.71]4.652.49
σ\sigma=4 lattice, no tubes4.38 [1.81, 6.70]–2.41
σ\sigma=2 lattice + tubes1.96 [0.83, 3.47]–2.68
Selector cut-off1.0 nats (exact 10–16)1.61 [0.28, 3.54]–3.12
2.0 nats (exact 11–15)3.90 [2.15, 6.40]4.27 [3.12, 5.61]2.87
3.0 nats (exact 14, 15)12.48 [3.30, 25.53]12.34 [9.73, 15.45]2.49

Table S26.

The ablation separates the contributions as follows.

  • Importance-weight correction. The hubs are drawn non-uniformly, so each must be reweighted by its log-weight. Without this correction the loss rises to 24 AUC@5 on the development scenes and 36 on the test scenes, so the hubs are useless without it.

  • Pilot proposal. Drawing hubs uniformly, while keeping the correct weights, costs 2.0 more on the test scenes than drawing them from the pilot’s column mass. Variance-reducing draws (systematic, heavy-tail, quantile) change little.

  • Early layers. They need cross-frame context: restricting them to the own frame, with or without special tokens, is costly. A count-biased mean token recovers most of the loss, and the hubs recover the rest.

  • Later diffuse layers. In layers 17–23, 1024 hubs beat both 256 hubs and AVGGТ’s lattice. Making layers 9–10 dense brings no gain for 8% more FLOPs.

  • Matching layers. Every sparse key set tested in the matching layers costs 1.4–25 more than exact attention, and tubes (correspondence windows) recover only part of what a lattice loses.

  • Selector cut-off. The 1.5-nat cut-off sits at the knee of the accuracy curve. Raising it to 2.0 nats removes layer 16 from the exact set and costs 2.9 on the test scenes; raising it to 3.0 nats keeps only layers 14 and 15 exact and costs 11.0. Lowering it to 1.0 adds layer 10 to the exact set without benefit.

Robustness and generality

Post-hoc robustness analysis

This post-hoc analysis tests whether the main conclusions are robust to how redundancy is measured, to which datasets are included, and to the worst scenes rather than the mean.

This analysis is post hoc: it was not part of the pre-registered plan, was run after all results of Table 2 were in, and has no pre-registered bar. It asks how the per-scene difference between LMSA and the two strongest baselines depends on redundancy, and how large each method’s tail losses are. A scene’s ρ\rho is the mean over the frame subsets on which it was measured (one to five per scene; § 3.1), and every per-scene value is a 5-seed mean.

Redundancy from ground truth. ρ\rho is computed from the dense model’s own depth and cameras. A poorly reconstructed scene could therefore look less redundant than it really is, which could place it on the wrong side of the split. To check this,

Table S25. The selector applied to GPA-VGGT (post hoc), 13 unseen low-ρ\rho scenes, NN=32, 5 seeds. Loss: AUC@5 lost against dense attention; Δ\Delta: paired AUC@5 difference (positive = the arm is more accurate), with hierarchical 95% CIs. Bold: the lowest loss and AbsRel, and the gain of GPA+sel over GPA-VGGT, the main result of this test.

Table S26. Component ablation, NN=32, 5 seeds. AUC@5 lost against dense attention, with hierarchical 95% CIs. Dev: 5 development scenes. Test: the 13 unseen low-ρ\rho scenes (post hoc). Each row changes one component of the frozen recipe (first row). Bold: the frozen recipe and the lowest test loss.

ρ\rho was recomputed with the same code from ground-truth depth and poses, which are available for ETH3D, TUM, 7-Scenes and DTU but not for Tanks and Temples. The ground-truth ρ\rho has a Spearman correlation of 0.98 with the model-based ρ\rho over 58 frame subsets (0.99 over the 21 scenes with ground truth, 16 test and 5 development), so the two measures rank scenes almost identically. No unseen low-ρ\rho scene crosses the 10.5 split; the highest is ETH3D relief at 9.75. One high-ρ\rho test scene, 7-Scenes pumpkin, falls just below the split (from 12.60 to 10.36), partly because Kinect depth has holes; it stays above the split when targets without depth count as visible (12.10). On the 16 test scenes with ground truth, the correlation of the gain over GPA-VGGT with ρ\rho is −0.55 (CI [−0.78, −0.12]) with the model-based ρ\rho and −0.48 (CI [−0.72, −0.01]) with the ground-truth ρ\rho. With ground-truth ρ\rho, the mean gain is +1.1 below the split and −0.6 above it, so the pattern is similar with either measure of ρ\rho.

Leaving out one dataset. This check asks whether any single dataset drives the result. On the 13 unseen low-ρ\rho scenes, the loss of LMSA against dense attention stays between 1.07 and 1.52 AUC@5 when ETH3D (8 scenes), Tanks and Temples (4) or TUM (1) is left out, so the loss itself does not depend on one dataset. The gains over the baselines do. The gain over GPA-VGGT comes mainly from ETH3D: it is 1.44 on ETH3D alone, and 0.39 (CI [−0.83, 2.14]) without it. The gain over SAF3R, in contrast, comes mainly from Tanks and TUM: it is 5.69 (CI [3.05, 9.60]) without ETH3D, and its CI includes 0 without either of them.

Dependence on redundancy. On the 21 test scenes, the per-scene difference between LMSA and GPA-VGGT falls as ρ\rho grows (see Figure S3; Spearman −0.52, bootstrap 95% CI [−0.78, −0.14]). Below the split ρ\rho = 10.5, LMSA is ahead by 1.0 AUC@5 on average (13 scenes, ahead on 8); above it, LMSA is behind by 0.6 (8 scenes, ahead on 1). With the 5 development scenes included, the correlation is −0.43 (CI [−0.69, −0.07]). This correlation reflects a contrast between the two sides of the split rather than a trend within either side: within each side it is −0.18 below and +0.21 above, and neither is significant. Against SAF3R, the test-scene correlation is weaker and its CI includes 0 (Figure S3; −0.37, CI [−0.72, 0.09]; means +3.2 below the split and +1.0 above). Tail losses. A user who runs a method on one scene cares about its worst case as well as its mean, and the methods differ more in their tails than in their means (Table S27, Figure S3). On the 13 unseen low-ρ\rho scenes, the worst scene costs LMSA 2.9 AUC@5 (ETH3D electro) and costs GPA-VGGT 6.8 (terrace); the worst 5% of single runs cost them 5.9 and 10.2. On the six scenes on which GPA-VGGT itself loses at least 3 AUC@5 (ETH3D electro, playground, relief, relief_2 and terrace, and courtyard), LMSA is ahead of it by 0.2 to 5.6, so LMSA is ahead where GPA-VGGT struggles most. The whole analysis is descriptive: the split ρ=10.5\rho= 10.5 was fixed by the protocol (§ 3.1), but the comparisons themselves were chosen after the results were known, so they are not tests of a prior hypothesis.

Figure S3. Post-hoc robustness analysis (not pre-registered). (a) AUC@5 of LMSA minus that of GPA-VGGT on the 21 test scenes (filled) and the 5 development scenes (open) of Table 2 against the scene’s redundancy ρ\rho (log axis); shaded: ρ<10.5\rho< 10.5; dashed lines and box: mean on each side of the split and Spearman correlation with a bootstrap 95% CI, over the test scenes only. (b) The same against SAF3R. (c) Loss against dense attention on the 13 unseen low-ρ\rho scenes: worst scene (bar) and 90th-percentile scene (tick).

Unseen low-ρ\rho (13 scenes)All 26 scenes
Methodmean90th pct. sceneworst scene95th pct. runworst 5% of runsworst scene
AVGGT (σ=4\sigma=4)10.313.124.2 (terrace)20.925.237.5 (courtyard)
AVGGT (σ=9\sigma=9)18.925.344.8 (terrace)38.047.050.4 (courtyard)
GPA-VGGT2.44.96.8 (terrace)6.310.26.8 (terrace)
SAF3R4.510.916.7 (electro)18.322.622.0 (courtyard)
LMSA1.42.52.9 (electro)5.25.92.9 (electro)
LMSA-fast4.68.712.7 (terrace)10.214.512.7 (terrace)

Table S27. Tail losses (post hoc): AUC@5 lost against dense attention. Scene losses use 5-seed means; run losses use single scene-seed runs (65 on the unseen low-ρ\rho scenes; “worst 5%” is the mean of the 4 worst). Dense loses 0 by definition. Bold: lowest per column.

Sources of run-to-run variance

The loss of LMSA varies from run to run for two reasons: the random hub draw, and the choice of frame subset. This post-hoc analysis measures how much of the variation each source causes.

On the five development scenes, LMSA was rerun with two frame subsets per scene and three hub draws per subset, giving 30 runs; a new draw changes the seed of both the hub pilot and the hub sample. A nested method-of-moments decomposition then splits the variance of the per-run AUC@5 loss into its sources (Table S28). It attributes 84% of the variance to the hub draw (SD 2.28), 16% to the frame subset (SD 1.00) and none to the scene, so hub sampling noise is the main source. Most of the draw variance comes from one courtyard subset, whose three draws span 12.7 AUC@5; without courtyard, the draw SD is 1.14 (70%). Over the 21 test scenes, with one draw per subset, the scene term explains 3.5% of the per-run variance. Since the noise comes from sampling, drawing more hubs should reduce it. More hubs (1024 in layers 0–8 and 2048 in layers 9–10 and 17–23) indeed lower the draw SD by 44%, from 2.28 to 1.27, at an unchanged mean loss (0.36 against 0.39) and for 2.2% more FLOPs at N=32N=32. With more hubs, the frame subset becomes the dominant source of variation.

ComponentLevelsVar.SDShare
Scene50.00†0.00^{\dagger}0.000% [0, 23]
Frame subset101.001.0016% [0, 66]
Hub draw305.212.2884% [34, 100]
Total per run306.212.49100%
Hub draw, more hubs‡^{\ddagger}301.621.2737% [19, 100]

Table S28. Variance components of the per-run AUC@5 loss of LMSA: 5 development scenes × 2 frame subsets × 3 hub draws, nested method of moments; share with a bootstrap 95% interval over scenes. †^{\dagger}Negative estimate (−0.70) truncated at 0. ‡^{\ddagger}With more hubs; share of that arm’s total (4.41). Bold: the dominant component, the hub draw, and its SD with more hubs.

More hubs on the test scenes (post hoc). The same variant with more hubs was then run on all 21 test scenes with 5 seeds. It loses 0.66 AUC@5 on the unseen low-ρ\rho scenes (CI [0.00, 1.26]) and 0.20 on the high-ρ\rho scenes (CI [0.00, 0.43]), against 1.38 and 0.73 for LMSA. Its paired gains over LMSA (0.72, CI [−0.16, 1.66], and 0.53, CI [−0.03, 1.17]) are not significant, however. On the low-ρ\rho scenes, the variant leads GPA-VGGT by 1.76 AUC@5 (CI [0.60, 3.17]) and 1.45 AUC@30 (CI [0.44, 2.78]); on the high-ρ\rho scenes, it does not differ significantly from GPA-VGGT (−0.12, CI [−0.54, 0.34]). Against the stronger GPA-VGGT with the selected layers (Section E.3), its lead shrinks to 0.55 AUC@5 (CI [−0.15, 1.27]) and 0.75 AUC@30 (CI [0.06, 1.67]). On its worst unseen low-ρ\rho scene (ETH3D electro), the variant loses 2.05 AUC@5, against 2.94 for LMSA and 6.81 for GPA-VGGT. Because the variant was chosen after the development results, these numbers are not a pre-registered test.

Other samplers at equal cost (post hoc). Instead of drawing more hubs, the noise could in principle be reduced by drawing the same number of hubs more cleverly. A layer-local audit tested this by comparing the current sampler with five alternatives on the 18 hub layers of the frozen recipe, using the 5 development scenes, 2 seeds and 2048 random queries, with 8 independent draws each. All alternatives keep the numerator and denominator unbiased, and the plan and its continue rules were registered before the run.

  • Fixed share of top-mass keys. Keeping 10, 25 or 50% of the top-mass keys exactly and sampling the rest from the renormalised residual raised the output error by 3, 13 and 33%, so it made the error worse.

  • Other proposals. An RMS proposal and a head-balanced proposal changed the error by less than 0.2%, and a value-aware proposal lowered it by only 1.4%.

  • More draws. The output error fell as m−1/2m^{-1/2} with the number of draws; twice the hubs gave 0.71 of the error.

The reason the proposal matters so little is that it is close to flat relative to the draw budget: about 254 of 256 draws are distinct keys, so almost no draw is wasted on a repeated key. Drawing more hubs is therefore the only effective lever, which is why the post-hoc variant above changes the number of hubs rather than the sampler.

Transfer to π3\pi^3

This preliminary transfer evaluation asks whether the selector, applied without tuning to another model, π3\pi^3, picks the layers that need exact attention; the sparse runs use the original π3\pi^3 checkpoint.

Table Table S29 gives the prediction behind § 3.5. The selector was applied to two π3\pi^3 checkpoints: the newer release, whose values the table lists first, and the original checkpoint, on which the sparse runs were made. Newer release. Per layer, the selector labels layers 11–15 as matching (Γˉ≥1.5\bar{\Gamma} \ge1.5), layers 0, 1 and 17 as plain (Γˉ<0.2\bar{\Gamma} < 0.2), and the rest as diffuse; the row-free rule s2≥3s^2 \ge3 picks the same layers 11–15. ∗^{*}Group labels follow the majority of their layers. The middle group (three diffuse and

Group (π3\pi^3 layers)Γˉ\bar{\Gamma}PredictedLoss shareOrig. π3\pi^3 (Γˉ\bar{\Gamma})
early (0–6)0.76diffuse0.6%diffuse (0.82)
middle (7)1.42diffuse0%matching (1.54)
middle (8–12)1.64diffuse∗^{*}14.6%diffuse∗^{*} (1.70)
late (13–17)2.96matching84.8%matching (3.02)

Table S29. Selector on π3\pi^3. Γˉ\bar{\Gamma} (group mean) and predicted role for the Pi3X release, and for the original π3\pi^3 checkpoint (last column); loss share: share of the summed positive AUC@5 loss when only that group uses hubs (original checkpoint, 5 development scenes × 3 subsets). Groups are mapped from VGGT’s by relative depth (VGGT 0–8 / 9–10 / 11–16 / 17–23). Predictions were written and hashed before any sparse run. Bold: groups labelled matching, and the observed loss share of the late group.

two matching layers) is therefore labelled diffuse, although its mean Γˉ\bar{\Gamma} is 1.64. Pass rule. The pre-registered pass rule compares the predicted groups with the observed ones. The observed matching set is the smallest set of groups that carries at least half of the summed positive AUC@5 loss when each group in turn uses hubs, and the rule passes if this set equals the predicted matching set. Original checkpoint. For the original π3\pi^3 checkpoint, the selector labels layers 7 and 11–15 as matching, with layer 7 at Γˉ=1.54\bar{\Gamma}=1.54, just above the cut-off. The row-free rule picks only layers 11–15, and the predicted matching groups are layer 7 and layers 13–17. Outcome on the original checkpoint. The predicted matching groups were layer 7 and layers 13–17, but the observed set is layers 13–17 alone, which carry 84.8% of the summed loss. Under hubs, layer 7 loses −0.11 AUC@5, that is, nothing. The pre-registered rule therefore fails because of this one borderline layer (Γˉ=1.54\bar{\Gamma}=1.54), which shows that a statistic just above the cut-off is not sufficient to make a layer need exact attention. The untuned rule (exact layers 7 and 11–15, hubs elsewhere) loses 4.87 AUC@5 against dense π3\pi^3 on the unseen low-ρ\rho scenes and 2.53 on the development scenes (§ 3.5).

Three frame subsets (post hoc). To check the result on more data, the same untuned rule was run on frame subsets 0–2: 39 runs on the unseen low-ρ\rho scenes and 24 on the high-ρ\rho test scenes, scored against the registered bars. On the unseen low-ρ\rho scenes, it loses 4.71 AUC@5 (CI [2.48, 7.21]), again above the registered target of 3, against 13.05 for AVGGT(4) and 28.93 for AVGGT(9). Its gain of 8.35 over AVGGT(4) (CI [3.73, 12.95]) does not meet the registered cost clause, because the rule uses about 2.1× AVGGT(4)’s attention cost. On the high-ρ\rho test scenes, it loses 2.13 (CI [0.40, 4.03]), which passes the registered accuracy bar, but its gain over AVGGT(4) there (1.02, CI [−4.16, 5.91]) is not significant. Because both the low-ρ\rho accuracy target and the cost clause are missed, the transfer result remains preliminary.

References

  1. [1]Henrik Aanes, Rasmus Ramsøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjørholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, 120 (2):153–168, 2016.
  2. [2]Maryam Aliakbarpour, Vladimir Braverman, Junze Yin, and Haochen Zhang. Support basis: Fast attention beyond bounded entries. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 325–333, 2026.arxiv.org/abs/2510.01643
  3. [3]Josh Alman and Zhao Song. Fast attention requires bounded entries. In Advances in Neural Information Processing Systems (NeurIPS), pages 63117–63135, 2023.arxiv.org/abs/2302.13214
  4. [4]Daniel Boyla, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations (ICLR), 2023.
  5. [5]Jelena Bratulić, Sudhanshu Mittal, Thomas Brox, and Christian Rupprecht. On geometric understanding and learned priors in feed-forward 3D reconstruction models. arXiv preprint arXiv:2512.11508, 2025.
  6. [6]David Charatan, Sızhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelSplat: 3D Gaussian splats from image pairs for scalable generalizable 3D reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19457–19467, 2024.
  7. [7]Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. In Advances in Neural Information Processing Systems (NeurIPS), pages 17413–17426, 2021.
  8. [8]Yifan Chen, Qi Zeng, Dilek Hakkani-Tur, Di Jin, Heng Ji, and Yun Yang. Sketching as a tool for understanding and accelerating self-attention for long sequences. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 5187–5199, 2022.arxiv.org/abs/2112.05359
  9. [9]Yutian Chen, Yuheng Qiu, Ruogu Li, Jay Patrikar, and Sebastian Scherer. Co-Me: Confidence guided token merging for visual geometric transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14590–14599, 2026.arxiv.org/abs/2511.14751
  10. [10]Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. MagicPIC: LSH sampling for efficient LLM generation. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.16179
  11. [11]Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), pages 16344–16359, 2022.arxiv.org/abs/2205.14135
  12. [12]Jianing Deng, Yuanzhe Li, Jialu Wang, Song Wang, Tianlong Chen, Huanrui Yang, and Jingtong Hu. SAF3R: Dynamic sparse attention for feed-forward 3D reconstruction transformers. In European Conference on Computer Vision (ECCV), pages 42–59, 2026.arxiv.org/abs/2607.03612
  13. [13]Bernard Derrida. Random-energy model: An exactly solvable model of disordered systems. Physical Review B, 24(5):2613–2626, 1981.
  14. [14]Aditya Desai, Kumar Agrawal, Shuo Yang, Alejandro Cuadron Lafuente, Luis Gaspar Schroeder, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. VAttention: Verified sparse attention via sampling. In International Conference on Learning Representations (ICLR), pages 52253–52284, 2026.
  15. [15]Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra, and Ramachandran Ramjee. Kascade: A practical sparse attention method for long-context LLM inference. arXiv preprint arXiv:2512.16391, 2025.arxiv.org/abs/2512.16391
  16. [16]Weilun Feng, Haotong Qin, Mingqiang Wu, Chuangyang Yang, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Yulun Zhang, Michele Magno, and Yongjun Xu. Quantized visual geometry grounded transformer. In International Conference on Learning Representations (ICLR), pages 59817–59838, 2026.arxiv.org/abs/2509.21302
  17. [17]Alessio Giorldandino and Sebastian Goldt. Two failure modes of deep transformers and how to avoid them: A unified theory of signal propagation at initialisation. In International Conference on Learning Representations (ICLR), pages 63101–63127, 2026.
  18. [18]Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P. Woodruff, and Amir Zandieh. HyperAttention: Long-context attention in near-linear time. In International Conference on Learning Representations (ICLR), pages 39310–39325, 2024.arxiv.org/abs/2310.05869
  19. [19]Coleman Hooper, Sebastian Zhao, Luca Manolache, Sehoon Kim, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Multipole attention for efficient long context reasoning. In Advances in Neural Information Processing Systems (NeurIPS), pages 41954–41977, 2025.arxiv.org/abs/2506.13059
  20. [20]David Huang, Guile Wu, Chengjie Huang, Bingbing Liu, and Dongfeng Bai. TurboVGGT: Fast visual geometry reconstruction with adaptive alternating attention. In Advances in Neural Information Processing Systems (NeurIPS), 2026.
  21. [21]Huiqiang Jiang, Yuchen Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In Advances in Neural Information Processing Systems (NeurIPS), pages 52481–52515, 2024.
  22. [22]Yanming Kang, Giang Tran, and Hans De Sterck. Fast multiple attention: A scalable multilevel attention mechanism for text and images. Neural Networks, 206:109847, 2027.DOI
  23. [23]Youngsung Kim, Wooseok Song, Jaihyun Lew, Hun Hwangbo, Jaehoon Lee, and Sungho Yoon. Head sensitivity score for sparsity redistribution in VGGT. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 36509–36517, 2026.arxiv.org/abs/2603.25336
  24. [24]Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4):78:1–78:13, 2017.
  25. [25]Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. FlexPrefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2502.20766
  26. [26]Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3D with MASt3R. In European Conference on Computer Vision (ECCV), pages 71–91, 2024.arxiv.org/abs/2406.09756
  27. [27]Xinze Li, Pengxu Chen, Yiquan Wang, Weifeng Su, and Wentao Cheng. S-VGGT: Structure-aware subscene decomposition for scalable 3D foundation models. In IEEE International Conference on Multimedia and Expo (ICME), 2026.arxiv.org/abs/2603.17625
  28. [28]Fangyu Liu and Rontian Ye. A strong and robust baseline for text-image matching. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 169–176, 2019.arxiv.org/abs/1906.01205
  29. [29]Carlo Lucibello and Marc Mézard. Exponential capacity of dense associative memories. Physical Review Letters, 132(7):077301, 2024.DOI
  30. [30]Shuo Lyu, Jinhao You, Zhuohang Lyu, Tanxuan Li, Zibo Zhao, Jiaxiang Hu, Kai Tang, and Yichen Guo. RegimeVGGT: Layer-wise spatially preserving redundancy removal for visual geometry grounded transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2026.arxiv.org/abs/2606.18439
  31. [31]Rupert Mitchell and Kristian Kersting. Multipole semantic attention: A fast approximation of softmax attention for pretraining. In International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2509.10406
  32. [32]Tan Nguyen, Vai Suliafu, Stanley Osher, Long Chen, and Bao Wang. FMMformer: Efficient and flexible transformer via decomposed near-field and far-field attention. In Advances in Neural Information Processing Systems (NeurIPS), pages 29449–29463, 2021.
  33. [33]Jinsoo Park, Donggyu Choi, Ahyun Seo, Minsu Cho, and Jeany Son. Diversity-aware view partitioning for scalable VGGT. In European Conference on Computer Vision (ECCV), pages 547–564, 2026.arxiv.org/abs/2607.01885
  34. [34]Weijing Ren, Xiao Tan, and Kai Han. Speed3R: Sparse feed-forward 3D reconstruction models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings, pages 119–128, 2026.arxiv.org/abs/2603.08055
  35. [35]Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2538–2547, 2017.DOI
  36. [36]You Shen, Zhipeng Zhang, Yansong Qu, Xiawu Zheng, Jiayi Ji, Shengchuan Zhang, and Liujun Cao. FastVGGT: Fast visual geometry transformer. In International Conference on Learning Representations (ICLR), 2026.
  37. [37]Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2930–2937, 2013.DOI
  38. [38]Zhijian Shu, Cheng Lin, Tao Xie, Wei Yin, Ben Li, Zhiyuan Pu, Weize Li, Yao Yao, Xun Cao, Xiaoyang Guo, and Xiao-Xiao Long. LiteVGGT: Boosting vanilla VGGT via geometry-aware cached token merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 36422–36432, 2026.arxiv.org/abs/2512.04939
  39. [39]Jürgen Sturm, Nikolaus Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of RGB-D SLAM systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 573–580, 2012.DOI
  40. [40]Xianbing Sun, Zhikai Zhu, Zhengyu Lou, Bo Yang, Jinyang Tang, Liqing Zhang, He Wang, and Jianfu Zhang. AVGG T: Rethinking global attention for accelerating VGGT. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 251–260, 2026.arxiv.org/abs/2512.02541
  41. [41]Muyeol Sung, Jangwon Kim, and Jongwoo Lim. GPA-VGGT: Geometry-preserving acceleration of VGGT. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2027. arXiv:2610.37000.arxiv.org/abs/2610.37000
  42. [42]Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. RazorAttention: Efficient KV cache compression through retrieval heads. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2407.15891
  43. [43]Chung-Shien Brian Wang, Christian Schmidt, Jens Piekenbrinck, and Bastian Leibe. Block-sparse global attention for efficient multi-view geometry transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14546–14555, 2026.arxiv.org/abs/2509.07120
  44. [44]Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. VGGT: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5294–5306, 2025.arxiv.org/abs/2503.11651
  45. [45]Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3D perception model with persistent state. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10510–10522, 2025.arxiv.org/abs/2501.12387
  46. [46]Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3D vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20697–20709, 2024.arxiv.org/abs/2312.14132
  47. [47]Weitian Wang, Lukas Meiner, Rai Shubham, Cecilia De La Parra, and Akash Kumar. HTTM: Head-wise temporal token merging for faster VGGT. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26379–26388, 2026.arxiv.org/abs/2511.21317
  48. [48]Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jianqiao Pang, Chunhua Shen, and Tong He. π³: Permutation-equivariant visual geometry learning. In International Conference on Learning Representations (ICLR), 2026.
  49. [49]Zipeng Wang and Dan Xu. FlashVGGT: Efficient and scalable visual geometry transformers with compressed descriptor attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21826–21835, 2026.arxiv.org/abs/2512.01540
  50. [50]Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, and Song Han. Sparse Video-Gen: Accelerating video diffusion transformers with spatial-temporal sparsity. In International Conference on Machine Learning (ICML), pages 68208–68224, 2025.arxiv.org/abs/2502.01776
  51. [51]Guangxuan Xiao, Jianming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.10819
  52. [52]Jianing Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3R: Towards 3D reconstruction of 1000+ images in one forward pass. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21924–21935, 2025.arxiv.org/abs/2501.13928
  53. [53]Shujun Yao, Junhui Lin, Congying Han, Tiande Guo, Ziwen Liu, and Bonan Li. BlazeVGGT: 13× faster VGGT at 2,000 frames via camera token without training. arXiv preprint arXiv:2610.29454, 2026.arxiv.org/abs/2610.29454
  54. [54]Shuhong Zheng, Michael Oechsle, Erik Sandström, Marie-Julie Rakotosao­na, Federico Tombari, and Igor Gilitschenski. Good token hunting: A hitchhiker’s guide to token selection for visual geometry transformers. arXiv preprint arXiv:2605.23892, 2026.arxiv.org/abs/2605.23892
  55. [55]Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Transactions on Graphics, 37 (4):65:1–65:12, 2018.arxiv.org/abs/1805.09817
  56. [56]Qianchao Zhu, Jiangfei Duan, Chang Chen, Xiuhong Li, Siran Liu, Guanyu Feng, Xin Lv, Chuanfu Xiao, Dahua Lin, and Chao Yang. SampleAttention: Near-lossless acceleration of long context LLM inference with adaptive structured sparse attention. In Proceedings of Machine Learning and Systems (MLSys), 2025.arxiv.org/abs/2406.15486

Paper details

vlnc:W_dsvxrp74
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

ChatGPT (OpenAI)

View statement
Looking for related papers…
Contents