betaFeedback

WorksComputer Science

Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation

Abstract

Causal masking obstructs the column balancing used by optimal-transport (OT) attention. We study an unbalanced OT operator that uses only current and earlier rows and characterize its induced positional prior. For bounded scores and any fixed balancing strength below one, its fixed point assigns vanishing mass to every fixed window as context grows. With constant scores, the prior has an exact scale-invariant form. We classify positional encodings by whether they prevent this dilution. We also derive finite-length window-mass bounds for log-distance priors with exponents near one. Preregistered experiments use 25M- and 100M-parameter language models. The OT operator improves loss extrapolation over softmax, while exploratory comparisons favor an explicit distance bias at a fraction of the cost. Changing only the bias beyond the training length moves long-context loss in the predicted direction on all 30 tested checkpoints. During training, learned exponents approach one even when constrained above it. A margin chosen from prior calculations controls distant-mass growth in the tested setting.

Introduction

Softmax attention normalizes each query’s scores into a distribution over keys. In optimal-transport terms, softmax is one row-normalization step of the Sinkhorn algorithm for an entropic OT problem. It also solves a one-sided entropic OT problem exactly [21, 39]. Iterating Sinkhorn to convergence also enforces the column marginal [43]; with the uniform marginal used by Sinkformers, every key then receives the same total attention.

In causal language models, column balancing faces two obstructions (§ 2). First, the only lower-triangular doubly stochastic matrix is the identity, so balanced OT degenerates. Second, relaxing the column marginal is not enough: any column normalization whose column sums run over all rows makes the output of row tt depend on row t+1t+1, whose query is computed from the token that row tt is trained to predict. Under teacher forcing this leaks the label and lowers the validation loss artificially. Prefix column masses avoid this leakage [11].

We address both obstructions with an unbalanced Sinkhorn iteration whose column scaling for row tt uses only the prefix mass Mtj=∑i≤tPijM_{tj} = \sum_{i \le t} P_{ij}. It is prefix-consistent by construction, interpolates between softmax (ρ=0\rho= 0) and a strongly balancing regime (ρ→1\rho\to1), and at its fixed point takes the form

log⁡Ptj=log⁡Ktj−γlog⁡Mtj−lse⁡t,γ=ρ1−ρ\log P_{tj} = \log K_{tj} - \gamma\log M_{tj} - \operatorname{lse}_t,\qquad\gamma= \frac{\rho}{1-\rho}

where lse⁡t\operatorname{lse}_t is the log-normalizer of row tt. In a preregistered comparison at 100M parameters, this operator lowers the validation loss at twice the training length by 0.45 nats relative to softmax. It also beats log-nn softmax, which multiplies the scores by a factor growing like log tt [5, 28], by 0.30. The correction −γlog⁡M-\gamma\log M enters the logits as an additive bias. Exploratory comparisons suggest that its distance dependence accounts for the gain: an explicit learnable log-distance bias with softmax achieves lower loss, at 40% of the cost.

We analyze the implicit prior of the causal OT operator: the attention pattern it produces when content does not discriminate between keys. The far mass at a given context length is the attention that queries put on keys more than the training length back. Here attention dilutes if the mass in every fixed-length recent window vanishes as the context grows (see § 6 for prior terminology). A relative prior is summable if its weights, as a function of distance, have a finite sum; a fixed window then keeps a share bounded away from zero, and the far mass stays bounded. The OT prior is not a function of distance alone, so for it we speak of dilution only.

Contributions.

  1. Dilution. For scores bounded by BB, as in pre-norm Transformers, and any ρ<1\rho< 1, every entry of the fixed point satisfies Ptj≤(e2B/t)1−ρP_{tj} \le(e^{2B}/t)^{1-\rho} (Theorem 5), so the mass within any fixed-length recent window vanishes as the context grows: the implicit prior dilutes.

  1. Exact prior. With constant scores, tPt,⌊st⌋→(log⁡1/s)−ρ/Γ(1−ρ)tP_{t,\lfloor st\rfloor} \to(\log1/s)^{-\rho}/\Gamma(1-\rho) uniformly on compact subsets of (0,1)(0,1) (Theorems 6 and 7): the prior depends on the ratio t/jt/j, and near the diagonal it is an exact power law d−ρd^{-\rho} (Theorem 8). The same limit holds after any fixed number k≥2k \ge2 of iterations (Theorem 9), and with i.i.d. bounded scores the fixed point factors into this prior times softmax-like content weights (Proposition 10).

  1. Far mass of positional encodings. Writing positional schemes as potentials on the logits, we show that finite-dimensional group representations cannot produce a logarithmic kernel (Proposition 12). We classify common encodings by whether they keep a bounded far mass or dilute (Proposition 14), and give discrete near-boundary window-mass bounds under bounded content (Corollary 16).

  1. Experiments. The OT operator’s gain over softmax holds in a preregistered test at 100M parameters, and exploratory analyses trace it to the distance prior (§ 5.1). Across fifteen positional schemes, far mass and long-context loss group by the cases of Proposition 14, not by the form of the bias (§ 5.2). Changing a trained model’s bias only beyond the training length moves its long-context loss in the predicted direction on all 30 checkpoints, and a margin on a learned exponent bounds the far mass during training (§ 5.3); CABLE beats RoPE with YaRN on every seed at three training lengths (§ 5.2).

Scope. All theory concerns causal attention with a prefix-consistent operator. Without a causal constraint the column marginal is a genuine constraint and none of the results apply; we return to this in § 7. Our measure of extrapolation is the validation loss beyond the training length, which rewards not being disturbed by distant context rather than using it.

Background: attention as optimal transport under a causal mask

Attention. One head of a Transformer [48] maps queries qtq_t, keys kjk_j and values wjw_j (t,j=1,…,nt,j=1,\ldots,n; vv will denote a scaling vector) to outputs yt=∑jPtjwjy_t=\sum_j P_{tj}w_j with attention weights

Ptj=exp⁡(Stj/ε)∑j′∈Mtexp⁡(Stj′/ε)(j∈Mt),Stj=⟨qt,kj⟩d+btj,(1)P_{tj}=\frac{\exp(S_{tj}/\varepsilon)}{\sum_{j'\in\mathcal{M}_t}\exp(S_{tj'}/\varepsilon)}\quad(j\in\mathcal{M}_t),\qquad S_{tj}=\frac{\langle q_t,k_j\rangle}{\sqrt{d}}+b_{tj}, \tag*{(1)}

where dd is the head dimension, btjb_{tj} an optional additive positional bias, and Mt\mathcal{M}_t the keys that query tt may attend to: Mt={1,…,t}\mathcal{M}_t=\{1,\ldots,t\} under the causal mask of a language model, with support M={(t,j):j≤t}\mathcal{M}=\{(t,j):j\le t\}. With the kernel K=eS/εK=e^{S/\varepsilon} this is P=rownorm⁡(K)P=\operatorname{rownorm}(K), where rownorm divides each row of KK by its sum over Mt\mathcal{M}_t. We keep the temperature ε\varepsilon explicit (all models use ε=1\varepsilon=1) because it is the entropic regularization strength: row tt of softmax maximizes ∑j∈MtPtjStj+εH(Pt⋅)\sum_{j\in\mathcal{M}_t}P_{tj}S_{tj}+\varepsilon H(P_{t\cdot}), H(p)=−∑jpjlog⁡pjH(p)=-\sum_jp_j\log p_j, over probability vectors [29 §§2.2–2.3].

Entropic optimal transport and Sinkhorn. Entropic OT [7, 34], with cost −Stj-S_{tj}, sums this objective over the rows and constrains both marginals,

max⁡P≥0∑t,jPtjStj+εH(P)s.t.P1=a,PT1=b,(2)\max_{P\ge0}\sum_{t,j}P_{tj}S_{tj}+\varepsilon H(P)\quad\text{s.t.}\quad P\mathbf{1}=\mathbf{a},\quad P^{\mathsf{T}}\mathbf{1}=\mathbf{b}, \tag*{(2)}

with solution P=diag⁡(u)Kdiag⁡(v)P=\operatorname{diag}(u)K\operatorname{diag}(v). Sinkhorn’s algorithm alternates the row and column scalings,

u←a/(Kv),v←b/(KTu),(3)u\leftarrow\mathbf{a}/(Kv),\qquad v\leftarrow\mathbf{b}/(K^{\mathsf{T}}u), \tag*{(3)}

each step enforcing one marginal exactly (divisions are entrywise). With a=1\mathbf{a}=\mathbf{1} and the start v=1v=\mathbf{1} the first row step gives P=rownorm⁡(K)P=\operatorname{rownorm}(K): softmax is the first half iteration of Sinkhorn [39]. In the log domain log⁡vj\log v_j acts as a bias on key jj that lowers the logits of keys receiving more than their share bjb_j.

Unbalanced transport and ρ\rho. Replacing the hard column constraint in (2) by a penalty λKL⁡(PT1∥b)\lambda\operatorname{KL}(P^{\mathsf{T}}\mathbf{1}\mathbin{\|}\mathbf{b}) keeps the row step and damps the column step,

v←(b/KTu)ρ,ρ=λλ+ε∈[0,1](4)v\leftarrow(\mathbf{b}/K^{\mathsf{T}}u)^\rho,\qquad\rho=\frac{\lambda}{\lambda+\varepsilon}\in[0,1] \tag*{(4)}

[6 Table 1]. Only the fraction ρ\rho of the balancing bias is applied: ρ=1\rho=1 is balanced Sinkhorn and ρ=0\rho=0 returns softmax. The row constraint stays hard, so every output is a weighted average of the values. Relaxing it too would give row tt a total mass growing like a power of tt under a causal mask. Outputs beyond the training length would then be larger than any seen in training.

Two obstructions to the column constraint.

Lemma 1 (Triviality of causal doubly stochastic matrices). A nonnegative lower-triangular matrix with all row and column sums equal to one is the identity.

Proof. Column nn has a single entry PnnP_{nn}, so Pnn=1P_{nn}=1 and row nn has no other mass. Removing row and column nn leaves a matrix with the same property; induct.

Relaxing the column marginal avoids this degeneracy, but column normalization over all rows couples row tt to future queries. Under teacher forcing, these queries contain ground-truth future tokens, so their contributions to the column mass leak target information into row tt (§ 1). We call an operator prefix-consistent if row tt of its output depends only on rows 1,…,t1,\ldots,t of its input, and require it under a causal mask.

The prefix-consistent unbalanced operator. Following the prefix-mass construction of Foss [11], we replace the column mass by Mtj=∑i=jtPijM_{tj} = \sum_{i=j}^{t} P_{ij} (j≤tj \le t), the mass that column jj has received from rows 1,…,t1,\ldots,t. Starting from the all-ones array v(1)=1v^{(1)} = 1, iteration kk sets

P(k)=rownorm⁡(K⊙v(k)),log⁡v(k+1)=ρ(log⁡v(k)−log⁡M(k)),(5)P^{(k)} = \operatorname{rownorm}(K \odot v^{(k)}), \qquad\log v^{(k+1)} = \rho\left(\log v^{(k)} - \log M^{(k)}\right), \tag*{(5)}

where M(k)M^{(k)} is the prefix mass of P(k)P^{(k)}, ⊙\odot is the entrywise product, and log⁡\log acts entrywise. Unlike the vector vv of (3), v(k)v^{(k)} and M(k)M^{(k)} are lower-triangular arrays: the scaling that row tt applies to key jj depends on tt. Every iteration ends with a row normalization, so the output is row-stochastic; k=1k = 1 is softmax. The damping acts on the accumulated scaling vv, as in unbalanced Sinkhorn; damping the relative update P←P⊙M−ρP \leftarrow P \odot M^{-\rho} instead would keep the balanced fixed point, the identity of Lemma 1. At a fixed point log⁡v=−γlog⁡M\log v = -\gamma\log M with γ=ρ/(1−ρ)\gamma= \rho/(1-\rho):

Ptj=λtKtjMtj−γ(j≤t),∑j≤tPtj=1,P_{tj} = \lambda_t K_{tj} M_{tj}^{-\gamma}\quad(j \le t), \qquad\sum_{j \le t} P_{tj} = 1,

where λt>0\lambda_t > 0 is the normalizer of row tt. The operator solves no single OT problem: row i<ti < t is balanced against the prefix mass available at time ii. The correction −γlog⁡Mtj-\gamma\log M_{tj} acts as an additive bias. Under uniform attention Mtj=∑i=jt1/i≈log⁡(t/j)M_{tj} = \sum_{i=j}^{t} 1/i \approx\log(t/j) grows with distance because distant keys have received attention from more queries. The correction therefore down-weights distant keys; § 3 derives the resulting prior. We use 1/(1+γ)=1−ρ1/(1+\gamma) = 1-\rho and γ/(1+γ)=ρ\gamma/(1+\gamma) = \rho throughout, and write Λt=∑i≤tλi\Lambda_t = \sum_{i \le t} \lambda_i with Λ0=0\Lambda_0 = 0; Appendix § 9 collects the notation.

Bounded scores. In a pre-norm Transformer [54] every attention layer reads a LayerNorm output [3], whose norm is bounded by a constant of the model, so queries, keys and every quantity computed linearly from the input are bounded independently of the length (bounded inputs). The scores are bounded by BB if ∣Stj/ε∣≤B|S_{tj}/\varepsilon| \le B for all t,jt,j. This holds for dot-product scores with rotary encodings [45], which preserve norms, and fails for biases that decrease without bound in the distance, such as ALiBi’s −m(t−j)-m(t-j) [36] or a log-distance bias −clog⁡(1+t−j)-c\log(1+t-j). § 3 shows that attention dilutes under bounded scores. Avoiding dilution therefore requires a positional term with unbounded decay.

The implicit prior of causal OT attention

We derive the positional prior induced by the operator without an added positional encoding. The fixed point is computed row by row (§ 3.1); under bounded scores every fixed window loses its share of attention (§ 3.2); without content the prior has an explicit, scale-invariant shape (§ 3.3). The same limit holds with finite iterations and i.i.d. content (§ 3.4). All proofs are in Appendix § 10. Throughout, ρ∈[0,1)\rho\in[0,1) and PP is the fixed point (FP) unless stated otherwise; LL denotes a window length, which in the experiments is the training length.

A row recursion

References

  1. [1]Ben Anson, Xi Wang, and Laurence Aitchison. Scale-invariant attention. In Advances in Neural Information Processing Systems, 2025.
  2. [2]Kazuoki Azuma. Weighted sums of certain dependent random variables. Tôhoku Mathematical Journal, 19(3):357–367, 1967.DOI
  3. [3]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.arxiv.org/abs/1607.06450
  4. [4]Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, and Alexander I. Rudnicky. KERPLE: Kernelized relative positional embedding for length extrapolation. In Advances in Neural Information Processing Systems, 2022.arxiv.org/abs/2205.09921
  5. [5]David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. In Annual Meeting of the Association for Computational Linguistics, 2022.arxiv.org/abs/2202.12172
  6. [6]Lénaïc Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314):2563–2609, 2018. doi: 10.1090/mcom/3303.DOI
  7. [7]Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, 2013.
  8. [8]Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, 2021.arxiv.org/abs/2103.03404
  9. [9]Graham Everest, Alf van der Poorten, Igor Shparlinski, and Thomas Ward. Recurrence Sequences, volume 104 of Mathematical Surveys and Monographs. American Mathematical Society, 2003.DOI
  10. [10]Dylan B. Forde. Block-wise differentiable Sinkhorn attention: Tail-refinement gradients with a gap-aware dustbin bridge. arXiv preprint arXiv:2605.08123v3, 2026.arxiv.org/abs/2605.08123
  11. [11]David Tom Foss. Causal prefix-Sinkhorn attention: Leak-free column balancing for autoregressive transformers. doi:10.13140/RG.2.2.24121.74087, 2026. Working draft v0.4, August 18.DOI
  12. [12]Takumi Fukunaga and Hiroyuki Kasai. On the convergence of semi-relaxed Sinkhorn with marginal constraint and OT distance gaps. arXiv preprint arXiv:2205.13846, 2022.
  13. [13]Mahesh Godavarti. Why do accumulated transformations extrapolate? arXiv preprint arXiv:2606.24975, 2026.arxiv.org/abs/2606.24975
  14. [14]Ronald L. Graham, Donald E. Knuth, and Oren Patashnik. Concrete Mathematics: A Foundation for Computer Science. Addison-Wesley, 2nd edition, 1994.
  15. [16]Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30, 1963.
  16. [17]Greg Kamradt. Needle in a haystack - pressure testing LLMs. https://github.com/gkamradt/LLMTest_NeedleInAHaystack, 2023.
  17. [18]Michela Lapenna, Rita Fioresi, and Bahman Gharesifard. Sinkhorn doubly stochastic attention rank decay analysis. Transactions on Machine Learning Research, 2026.
  18. [19]Jarod Lévy, Mathurin Videau, Jad Yehya, Jean-Rémi King, Stéphane d’Ascoli, and Thomas Moreau. RoPE is dead, long live RoPE: Towards scalable data-aware positional encodings. arXiv preprint arXiv:2609.34556, 2026.
  19. [20]Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate. In International Conference on Learning Representations, 2025.
  20. [21]Elon Litman. Scaled-dot-product attention as one-sided entropic optimal transport. arXiv preprint arXiv:2508.08369, 2025.arxiv.org/abs/2508.08369
  21. [22]Elon Litman and Gabe Guo. You need better attention priors. In International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 75315–75339. PMLR, 2026.arxiv.org/abs/2601.15380
  22. [23]Yuxuan Liu. First passage attention. arXiv preprint arXiv:2610.22714, 2026.arxiv.org/abs/2610.22714
  23. [24]Carles Marín. Predicting how transformers attend: Analytic power-law theory, phase transitions, and practical compression tools. doi:10.5281/zenodo.19826343, April 2026. Technical report.
  24. [25]Manas Mehta, Fangcong Yin, and Greg Durrett. Randomized YaRN improves length generalization for long-context reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 30045–30055. Association for Computational Linguistics, 2026. doi: 10.18653/2026.findings-emnlp.1409.DOI
  25. [26]Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.16300
  26. [27]Timur Mudarissov, Mikhail Burtsev, Tatiana Petrova, and Radu State. Limitations of normalization in attention. In Advances in Neural Information Processing Systems, 2025.
  27. [28]Ken M. Nakanishi. Scalable-softmax is superior for attention. arXiv preprint arXiv:2501.19399, 2025.arxiv.org/abs/2501.19399
  28. [29]Vlad Niculae and Mathieu Blondel. A regularized framework for sparse and structured neural attention. In Advances in Neural Information Processing Systems, volume 30, 2017.arxiv.org/abs/1705.07704
  29. [30]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.arxiv.org/abs/2209.11895
  30. [32]Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2406.17557
  31. [33]Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In International Conference on Learning Representations, 2024.arxiv.org/abs/2309.00071
  32. [34]Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):355–607, 2019.arxiv.org/abs/1803.00567
  33. [35]Khiem Pham, Khang Le, Nhat Ho, Tung Pham, and Hung Bui. On unbalanced optimal transport: An analysis of Sinkhorn algorithm. In International Conference on Machine Learning, 2020.arxiv.org/abs/2002.03293
  34. [36]Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.arxiv.org/abs/2108.12409
  35. [37]Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025–7041. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.emnlp-main.473.DOI
  36. [38]Zhen Qin, Yiran Zhong, and Hui Deng. Exploring transformer extrapolation. In AAAI Conference on Artificial Intelligence, pages 18897–18905, 2024. doi: 10.1609/aaai.v38i17.29855.DOI
  37. [39]Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, 2022.arxiv.org/abs/2110.11773
  38. [40]Thibault Séjourné, François-Xavier Vialard, and Gabriel Peyré. Faster unbalanced optimal transport: Translation invariant Sinkhorn and 1-D Frank-Wolfe. In International Conference on Artificial Intelligence and Statistics, 2022.arxiv.org/abs/2201.00730
  39. [41]Thibault Séjourné, Gabriel Peyré, and François-Xavier Vialard. Unbalanced optimal transport, from theory to numerics. In Handbook of Numerical Analysis, volume 24, pages 407–471. Elsevier, 2023. doi: 10.1016/bs.hna.2022.11.003.DOI
  40. [42]Subham Singh, Ashutosh Mishra, and Subha Raut. Relative positions generalize, absolute positions memorize: An implicit-bias account of length generalization in attention. arXiv preprint arXiv:2607.18759, 2026.arxiv.org/abs/2607.18759
  41. [43]Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343–348, 1967.
  42. [44]Michal Spiegel, Michal Štefánik, Marek Kadlčík, Valeria Ruscio, Martin Fajčík, and Josef Kuchař. Attend or perish: Benchmarking attention on algorithmic reasoning. In Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 707–725. Association for Computational Linguistics, 2026. doi: 10.18653/2026.blackboxnlp-1.41.DOI
  43. [45]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/j.neucom.2023.127063.DOI
  44. [46]Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse Sinkhorn attention. In International Conference on Machine Learning, 2020.
  45. [47]Huy Tran, Max Milkert, and David Hyde. ASAP: Amortized doubly-stochastic attention via sliced dual projection. arXiv preprint arXiv:2605.12879, 2026.arxiv.org/abs/2605.12879
  46. [48]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.arxiv.org/abs/1706.03762
  47. [49]Pavlo Vaslyenko, Hugo Pitorro, André F. T. Martins, and Marcos Treviso. Long-context generalization with sparse attention. In International Conference on Learning Representations, 2026.
  48. [50]Ali Veisi, Hamidreza Amirzadeh, and Amir M. Mansourian. Context-aware biases for length extrapolation. In Conference on Empirical Methods in Natural Language Processing, pages 30374–30395, 2025. doi: 10.18653/v1/2025.emnlp-main.1545.DOI
  49. [51]Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. Softmax is not enough (for sharp size generalisation). In International Conference on Machine Learning, 2025.arxiv.org/abs/2410.01104
  50. [52]Nitin Vetchka. Towards infinite length extrapolation: A unified approach. arXiv preprint arXiv:2601.06113, 2026.
  51. [53]Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024.arxiv.org/abs/2309.17453
  52. [54]Rubin Xiong, Yun昌 Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, 2020.arxiv.org/abs/2002.04745
  53. [55]Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit Roy-Chowdhury, Jiasi Chen, and Samet Oymak. Selective attention: Enhancing transformer through principled context control. In Advances in Neural Information Processing Systems, volume 37, pages 11061–11086, 2024. doi: 10.52202/079017-0353.
  54. [56]Yifan Zhang, Zixiang Chen, Yifeng Liu, Zhen Qiu-Huiruo Yuan, Kangping Xu, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. Group representational position encoding. In International Conference on Learning Representations, 2026.arxiv.org/abs/2512.07805

Paper details

vlnc:W_746bed3j
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Anthropic Claude, Codex

View statement
Looking for related papers…
Contents