betaFeedback

WorksMathematics

Understanding axial attention in TSFMs through single location regression

Abstract

Recent advances in Time Series Foundation Models (TSFMs) increasingly rely on axial attention mechanisms, but despite their empirical success, the theoretical properties of axial attention remain largely unexplored. To bridge this gap, we extend the Single Location Regression framework introduced in Marion et al. [2025] to 2D structured data, in order to study sparse information settings where a target depends on a small number of tokens. We formulate an analytically tractable axial attention-type predictor and rigorously analyze its statistical properties and training dynamics. We prove that under a specific asymptotic regime and with knowledge of the underlying oracle parameters, our predictor achieves Bayes optimality, whereas a class of semi-linear simplifications strictly fails to do so, underlining the role of the inner non-linearity. We prove that for a specific temperature scaling, Projected Gradient Descent converges globally to the optimal oracle parameters. Extensive numerical experiments validate our theoretical findings and highlight the critical role of inverse temperature scheduling in empirical convergence.

Introduction

Recent years have witnessed a surge in the development of Time Series Foundation Models (TSFMs) leveraging transformer architectures [Vaswani et al., 2017] (e.g., Chronos-2 [1], Moirai 2.0 [Liu et al., 2025], TimesFM [5], TabPFN-TS [Hoo et al., 2025], Toto [3], PatchTST-FM [Wen et al., 2026]). However, applying standard global attention to high-dimensional multivariate time series presents a significant computational bottleneck. Standard attention treats inputs as a flattened 1D sequence, which scales quadratically with the total number of tokens and fails to explicitly leverage the natural 2D grid structure (time steps ×\times channels) of multivariate data. To address this, axial attention introduced in [Ho et al., 2019], and specifically for multivariate time series in [Zhang and Yan, 2023], has emerged as a powerful alternative already implemented in some of the state-of-the-art TSFMs, e.g., Chronos-2, TabPFN-TS and Toto, but also in task specific models such as CEReBrO [6], an EEG foundation model that utilizes alternating intra-channel temporal and inter-channel spatial attention and STAIT (Spatio-Temporal Alternating Iterative Transformer) [4] for remote sensing.

The axial-attention layer combines two token-mixing half-layers, as described in Figure 1. Formally, we consider our input time series as M×NM \times N matrices of tokens X=(Xij)1≤i≤M,1≤j≤N\mathbf{X} = (X_{ij})_{1 \le i \le M,1 \le j \le N}, taking values in the embedding (or latent) space Rd\mathbb{R}^d, where MM can be seen as the number of channels and NN as the number of time series patches. Generically, given as hyperparameters two real numbers λr,λc\lambda_r,\lambda_c, referred to as inverse temperatures, and an activation function σ:Rd→Rd\sigma: \mathbb{R}^d \to\mathbb{R}^d, the standard axial self-attention layer is parametrized by weight matrices Qr,Kr∈Rd×prQ_r,K_r \in\mathbb{R}^{d \times p_r}, Vr∈Rd×dV_r \in\mathbb{R}^{d \times d}, Qc,Kc,Vc∈Rd×pcQ_c, K_c, V_c \in\mathbb{R}^{d \times p_c}, and Oc∈Rpc×ocO_c \in\mathbb{R}^{p_c \times o_c}. The subscripts rr and cc represent row-wise and, respectively, column-wise operations. The mapping is defined as, for any j∈[N]j \in[N]:

Figure 1. Schematic view of an axial attention layer. Raw input tokens are first processed by a time-wise self-attention layer that isolates features along the rows. A subsequent channel-wise attention layer vertically blends these extracted features, effectively capturing global cross-channel and temporal context.

Tλr,λc,σ(Qr,c,Kr,c,Vr,c,Oc)(X)⋅j=σ(λcX⋅j(1/2)QcKcT(X⋅j(1/2))T)X⋅j(1/2)VcOc(1)T_{\lambda_r,\lambda_c,\sigma}^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}(\mathbf{X})_{\cdot j} = \sigma\left(\lambda_c \mathbf{X}_{\cdot j}^{(1/2)} Q_c K_c^{\mathsf{T}} \left(\mathbf{X}_{\cdot j}^{(1/2)}\right)^{\mathsf{T}}\right) \mathbf{X}_{\cdot j}^{(1/2)} V_c O_c \tag*{(1)}

where the initial row-wise computation is, for any i∈[M]i \in[M]:

Xi⋅(1/2)=σ(λrXi⋅QrKrTXi⋅T)Xi⋅Vr(2)\mathbf{X}_{i\cdot}^{(1/2)} = \sigma\left(\lambda_r \mathbf{X}_{i\cdot} Q_r K_r^{\mathsf{T}} \mathbf{X}_{i\cdot}^{\mathsf{T}}\right)\mathbf{X}_{i\cdot}V_r \tag*{(2)}

where we use the notations [n]={1,2,…,n}[n] = \{1, 2, \ldots, n\} for any positive integer nn, and Wi⋅W_{i\cdot} (resp. W⋅jW_{\cdot j}) to denote the ii-th line (resp. jj-th column) of a matrix WW. Note that in (2) (and throughout the paper when applied to a matrix) the activation function is used row-wise. When σ\sigma is the softmax function, this structure is one of the building blocks used in Chronos-2, TabPFN-TS and Toto, stripped of residual connections and normalization layers. Note that the order in which the row attention and column attention are performed has no impact on the results in our paper; we choose row attention followed by column attention just for presentation purposes.

Although the empirical success of these axial architectures is evident across various domains, their theoretical properties remain largely unexplored. For example, in Berthelier et al. [ [2], the authors have shown that pretrained Chronos-2 is not always capable of identifying simple relationships between targets and covariates in the token-wise sparse information setting. Our goal is to improve the theoretical understanding of axial attention by analyzing their ability to approximate and learn the optimal predictor in this setting. To this end, we build on the work of Marion et al. [2025], conducted in the context of classical transformers and motivated by natural language processing (NLP) tasks. In that paper, the authors introduced a new class of statistical models, namely Single Location Regression (SLR) models, through which they aimed to investigate the ability of standard attention layers to recover relevant information in a token-wise sparse information setting, where the target depends only on a relatively small number of tokens. They have provided theoretical guarantees for a modification of the standard transformer architecture, proving that one can recover optimal predictors for SLRs. This naturally raises a fundamental, yet unanswered, question: Can we achieve the same statistical and optimization performance with axial attention as we do with normal global attention? The answer is not obvious, since the information flow is much more restricted in an axial transformer, where each token only attends to tokens with which it shares an axis, as opposed to the free flow of information between any two tokens in a standard transformer.

Building upon the foundation established by Marion et al. [2025], we introduce a SLR model and extend the theoretical analysis of transformer architectures to axial attention mechanisms, as a method of assessing the theoretical capabilities of current state-of-the-art TSFMs. Our main contributions are structured as follows:

  • In § 2, we introduce an SLR statistical model for the token-wise sparse information setting adapted to 2D structured data (e.g., multivariate time series, tables). We then explain the connection between such a model and multivariate time series anomaly analysis, and finally, we introduce a trainable predictor and show that it corresponds to a slightly simplified axial attention layer.

  • In Section 3, we analyze the performance of the predictor introduced and prove that there exists an asymptotic regime in which it is statistically optimal for the SLR model investigated, when using the oracle parameters. Furthermore, we show that removing the inner non-linearity in our predictor renders it suboptimal.

  • In Section 4, we propose an optimization algorithm for our predictor and provide theoretical guarantees for its convergence, despite the non-convexity of the problem. We further conduct extensive numerical studies which reveal a complex optimization landscape that confirm our theoretical findings.

Our structure closely follows that of Marion et al. [2025], but our motivation (the study of TSFMs) and the technical challenges are, nevertheless, very different. Standard global attention benefits from symmetric token coupling, which facilitates the derivation of closed-form analytical solutions for the expected risk. By contrast, axial attention operates in a sequential manner. This decoupled architecture breaks the global symmetry and induces a cascading non-linear dependence between axes. As a result, the optimization landscape exhibits significant complexity, rendering standard closed-form approaches to statistical optimality and convergence of optimization methods intractable and necessitating novel analytical techniques.

SLR model and simplified axial attention predictor

In this section, we introduce an SLR model and underline its connection to multivariate time series anomaly analysis. Afterwards, we consider a simplification of axial attention that renders the statistical analysis in Section 3 tractable.

Single location regression model. We focus on the following statistical model, where the output Y∈RY \in\mathbb{R} is assumed to be given by

Y=XI0J0Tv∗+ξ,(3)Y = X_{I_0J_0}^{\mathsf{T}}v^* + \xi, \tag*{(3)}

where ξ∼N(0,ϵ2)\xi\sim\mathcal{N}(0,\epsilon^2) is independent noise and I0I_0, J0J_0 are discrete random variables over [M][M] and [N][N], respectively. No restriction on the distribution of (I0,J0)(I_0, J_0) is imposed unless otherwise specified. Conditionally on (I0,J0)(I_0,J_0), for (ℓ,γ2,ϵ2,k∗,v∗)∈R+×R+∗×R+∗×Sd−1×Sd−1(\ell,\gamma^2,\epsilon^2,k^*,v^*) \in\mathbb{R}_+ \times\mathbb{R}_+^* \times\mathbb{R}_+^* \times\mathbb{S}^{d-1} \times\mathbb{S}^{d-1},

{XI0J0∼N(ℓk∗,γ2Id),Xij∼N(0,Id),for (i,j)≠(I0,J0),\begin{cases} X_{I_0J_0} \sim\mathcal{N}(\ell k^*,\gamma^2 I_d),\\ X_{ij} \sim\mathcal{N}(0,I_d),\quad\text{for }(i,j)\ne(I_0,J_0), \end{cases}

where XijX_{ij} are independent for all (i,j)∈[M]×[N](i,j) \in[M] \times[N].

The parameters of (SLR) are the shift magnitude ℓ\ell, the localization and noise variances γ2\gamma^2 and ϵ2\epsilon^2, the unobserved projection vector v∗v^* and the localization vector k∗k^*, with both vectors taking values on the dd-dimensional unit sphere Sd−1\mathbb{S}^{d-1}. The target variable YY depends exclusively on a single embedded token within the M×NM \times N input matrix, positioned at random discrete coordinates (I0,J0)(I_0,J_0). This active token is statistically identifiable because its expected value is shifted along the direction k∗k^*. Consequently, the primary mathematical challenge of this task stems from its dynamic sparsity.

Remark 1. Unlike in Marion et al. [2025], we consider ℓ\ell to be decoupled from dd throughout our analysis, but note that for ℓ=d/2\ell= \sqrt{d/2} and γ2=1/2\gamma^2 = 1/2, all tokens have the same expected norm, i.e. E[∥XI0J0∥22]=E[∥Xij∥22]=d\mathbb{E}[\lVert X_{I_0J_0}\rVert_2^2] = \mathbb{E}[\lVert X_{ij}\rVert_2^2] = d, making them indistinguishable just based on their magnitude.

Connection to multivariate time series anomaly analysis. The Single Location Regression (SLR) model detailed above provides a natural theoretical framework for multivariate time series anomaly analysis, particularly when operating on learned representation embeddings XX. We consider system-wide anomalous events identifiable through correlated behaviors across multiple sensors, as pictured in Figure 2. We operate under the assumption that the embedding layer has already captured these correlated dynamics, localizing the signature of the system-wide anomaly into the embedding of a single primary driving sensor I0I_0 at a specific time step J0J_0. By investigating whether axial attention, which restricts information flow, can successfully recover the true parameters k∗k^* and v∗v^*, we can theoretically assess its ability to isolate and leverage localized anomalies.

Figure 2. Under normal operating conditions, the background embeddings behave as uninformative noise, or as a standard token, modeled by the standard normal distribution N(0,Id)\mathcal{N}(0,I_d). When an anomaly occurs in one sensor, its specific signature is encoded into the token XI0J0X_{I_0J_0}, characterized by a shifted mean ℓk∗\ell k^* and a localization variance γ2Id\gamma^2 I_d. The regression target YY represents a downstream consequence or system-level anomaly score that depends strictly on identifying this specific embedded state via the projection vector v∗v^*. Our goal is to investigate whether we can recover this target through an axial attention layer.

Furthermore, using axial attention layers instead of standard global attention is attractive also from a computational point of view. The global attention mechanism treats the input as a flattened 1D sequence of length M×NM \times N and this coupling requires O(M2N2)\mathcal{O}(M^2N^2) operations per layer, while axial attention only requires O(MN2+M2N)\mathcal{O}(MN^2+M^2N) (see Ho et al. [2019] for a derivation).

An analytically tractable simplification of axial attention for SLR. In what follows, we introduce a class of predictors that are direct simplifications of the axial attention layer. For an activation σ:Rd→Rd\sigma:\mathbb{R}^d \to\mathbb{R}^d and inverse temperatures λr,λc∈(0,∞)\lambda_r,\lambda_c \in(0,\infty), let

Tλr,λc,σ(k,v)(X):=σ(λcX(1/2)k)TX(1/2)v,where Xi(1/2):=σ(λrXik)TXi,(4)T_{\lambda_r,\lambda_c,\sigma}^{(k,v)}(\mathbf{X}) := \sigma\left(\lambda_c \mathbf{X}^{(1/2)}k\right)^\mathsf{T}\mathbf{X}^{(1/2)}v,\qquad\text{where }\mathbf{X}_i^{(1/2)} := \sigma\left(\lambda_r \mathbf{X}_i k\right)^\mathsf{T}\mathbf{X}_i, \tag*{(4)}

for i∈[M]i \in[M] and learnable parameters k,v∈Sd−1k,v \in\mathbb{S}^{d-1}. Note that we slightly overload the notation and use X(1/2)\mathbf{X}^{(1/2)} to denote the half step of both our predictor and of axial attention.

In the context of TSFMs, sequences are typically appended with designated [REG] tokens to act as readout tokens or attention sinks (see Ansari et al. [ [1], for example). Assuming that the input X\mathbf{X} is appended with a (N+1)(N+1)th column of row-aggregation [REG_ROW] tokens and a column aggregation [REG_COL] token at position (M+1,N+1)(M+1,N+1), we have the following lemma

Lemma 1. For pr=pc=1p_r=p_c=1, oc=1o_c=1, Oc=1O_c=1, together with shared keys Kr=KcK_r=K_c and Vr=IdV_r=I_d, we have the following equivalence:

Tλr,λc,σ(Qr,c,Kr,c,Vr,c,Oc)(X)M+1,N+1=Tλra,λca(1/2),σ(Kc,Vc)(X),T_{\lambda_r,\lambda_c,\sigma}^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}(\mathbf{X})_{M+1,N+1}=T_{\lambda_r a,\lambda_c a^{(1/2)},\sigma}^{(K_c,V_c)}(\mathbf{X}),

for some constants aa and a(1/2)a^{(1/2)}. The proof and further comments on the constraints are in Appendix § 6.

References

  1. [1]Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerron, Tony Hu, Junming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael Bohlke-Schneider. Chronos-2: From univariate to universal forecasting. arXiv preprint arXiv:2510.15821, 2025.arxiv.org/abs/2510.15821
  2. [2]Gaspard Berthelier, Mariia Baranova, Andrei-Tiberiu Pantea, Etienne Le Naour, Adrien Petralia, Tahar Nabil, and Themis Palpanas. Investigating simple target-covariate relationships for Chronos-2 and TabPFN-TS. In 1st ICLR Workshop on Time Series in the Age of Large Models, 2026.arxiv.org/abs/2605.12200
  3. [3]Ben Cohen, Emaad Khwaja, Youssef Doubli, Salahidine Lemaachi, Chris Lettieri, Charles Masson, Hugo Miccinilli, Elise Ramé, Qiqi Ren, Afshin Rostamizadeh, Jean Ogier du Terrail, Anna-Monica Toon, Kan Wang, Stephan Xie, Zongzhe Xu, Viktoriya Zhukova, David Asker, Ameet Talwalkar, and Othmane Abou-Amal. This time is different: An observability perspective on time series foundation models, 2025.arxiv.org/abs/2505.14766
  4. [4]Yukun Cui, Jiangshe Zhang, Haowen Bai, Zixiang Zhao, Lilun Deng, Shuang Xu, and Chunxia Zhang. STAT: A spatio-temporal alternating iterative transformer for multi-temporal remote sensing image cloud removal. Remote Sensing, 18(4), 2026. ISSN 2072-4292. doi: 10.3390/rs18040596.DOI
  5. [5]Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10148–10167. PMLR, 21–27 Jul 2024.arxiv.org/abs/2310.10688
  6. [6]Alexandru Dimofte, Glenn Anta Bucagu, Thorir Mar Ingolfsson, Xiaying Wang, Andrea Cossettini, Luca Benini, and Yawei Li. Cerebro: Compact encoder for representations of brain oscillations using efficient alternating attention. arXiv preprint arXiv:2501.10885, 2025.arxiv.org/abs/2501.10885

Paper details

vlnc:W_ncb24m93
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 3

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

LLMs

View statement
Looking for related papers…
Contents