Introduction
Recent years have witnessed a surge in the development of Time Series Foundation Models (TSFMs) leveraging transformer architectures [Vaswani et al., 2017] (e.g., Chronos-2 [1 ] , Moirai 2.0 [Liu et al., 2025], TimesFM [5 ] , TabPFN-TS [Hoo et al., 2025], Toto [3 ] , PatchTST-FM [Wen et al., 2026]). However, applying standard global attention to high-dimensional multivariate time series presents a significant computational bottleneck. Standard attention treats inputs as a flattened 1D sequence, which scales quadratically with the total number of tokens and fails to explicitly leverage the natural 2D grid structure (time steps × \times × channels) of multivariate data. To address this, axial attention introduced in [Ho et al., 2019], and specifically for multivariate time series in [Zhang and Yan, 2023], has emerged as a powerful alternative already implemented in some of the state-of-the-art TSFMs, e.g., Chronos-2, TabPFN-TS and Toto, but also in task specific models such as CEReBrO [6 ] , an EEG foundation model that utilizes alternating intra-channel temporal and inter-channel spatial attention and STAIT (Spatio-Temporal Alternating Iterative Transformer) [4 ] for remote sensing.
The axial-attention layer combines two token-mixing half-layers, as described in Figure 1 . Formally, we consider our input time series as M × N M \times N M × N matrices of tokens X = ( X i j ) 1 ≤ i ≤ M , 1 ≤ j ≤ N \mathbf{X} = (X_{ij})_{1 \le i \le M,1 \le j \le N} X = ( X ij ) 1 ≤ i ≤ M , 1 ≤ j ≤ N , taking values in the embedding (or latent) space R d \mathbb{R}^d R d , where M M M can be seen as the number of channels and N N N as the number of time series patches. Generically, given as hyperparameters two real numbers λ r , λ c \lambda_r,\lambda_c λ r , λ c , referred to as inverse temperatures , and an activation function σ : R d → R d \sigma: \mathbb{R}^d \to\mathbb{R}^d σ : R d → R d , the standard axial self-attention layer is parametrized by weight matrices Q r , K r ∈ R d × p r Q_r,K_r \in\mathbb{R}^{d \times p_r} Q r , K r ∈ R d × p r , V r ∈ R d × d V_r \in\mathbb{R}^{d \times d} V r ∈ R d × d , Q c , K c , V c ∈ R d × p c Q_c, K_c, V_c \in\mathbb{R}^{d \times p_c} Q c , K c , V c ∈ R d × p c , and O c ∈ R p c × o c O_c \in\mathbb{R}^{p_c \times o_c} O c ∈ R p c × o c . The subscripts r r r and c c c represent row-wise and, respectively, column-wise operations. The mapping is defined as, for any j ∈ [ N ] j \in[N] j ∈ [ N ] :
Figure 1. Schematic view of an axial attention layer. Raw input tokens are first processed by a time-wise self-attention layer that isolates features along the rows. A subsequent channel-wise attention layer vertically blends these extracted features, effectively capturing global cross-channel and temporal context.
T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) ⋅ j = σ ( λ c X ⋅ j ( 1 / 2 ) Q c K c T ( X ⋅ j ( 1 / 2 ) ) T ) X ⋅ j ( 1 / 2 ) V c O c (1) T_{\lambda_r,\lambda_c,\sigma}^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}(\mathbf{X})_{\cdot j}
=
\sigma\left(\lambda_c \mathbf{X}_{\cdot j}^{(1/2)} Q_c K_c^{\mathsf{T}} \left(\mathbf{X}_{\cdot j}^{(1/2)}\right)^{\mathsf{T}}\right)
\mathbf{X}_{\cdot j}^{(1/2)} V_c O_c \tag*{(1)} T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) ⋅ j = σ ( λ c X ⋅ j ( 1/2 ) Q c K c T ( X ⋅ j ( 1/2 ) ) T ) X ⋅ j ( 1/2 ) V c O c (1)
where the initial row-wise computation is, for any i ∈ [ M ] i \in[M] i ∈ [ M ] :
X i ⋅ ( 1 / 2 ) = σ ( λ r X i ⋅ Q r K r T X i ⋅ T ) X i ⋅ V r (2) \mathbf{X}_{i\cdot}^{(1/2)}
=
\sigma\left(\lambda_r \mathbf{X}_{i\cdot} Q_r K_r^{\mathsf{T}} \mathbf{X}_{i\cdot}^{\mathsf{T}}\right)\mathbf{X}_{i\cdot}V_r \tag*{(2)} X i ⋅ ( 1/2 ) = σ ( λ r X i ⋅ Q r K r T X i ⋅ T ) X i ⋅ V r (2)
where we use the notations [ n ] = { 1 , 2 , … , n } [n] = \{1, 2, \ldots, n\} [ n ] = { 1 , 2 , … , n } for any positive integer n n n , and W i ⋅ W_{i\cdot} W i ⋅ (resp. W ⋅ j W_{\cdot j} W ⋅ j ) to denote the i i i -th line (resp. j j j -th column) of a matrix W W W . Note that in (2) (and throughout the paper when applied to a matrix) the activation function is used row-wise. When σ \sigma σ is the softmax function, this structure is one of the building blocks used in Chronos-2, TabPFN-TS and Toto, stripped of residual connections and normalization layers. Note that the order in which the row attention and column attention are performed has no impact on the results in our paper; we choose row attention followed by column attention just for presentation purposes.
Although the empirical success of these axial architectures is evident across various domains, their theoretical properties remain largely unexplored. For example, in Berthelier et al. [ [2 ] , the authors have shown that pretrained Chronos-2 is not always capable of identifying simple relationships between targets and covariates in the token-wise sparse information setting. Our goal is to improve the theoretical understanding of axial attention by analyzing their ability to approximate and learn the optimal predictor in this setting. To this end, we build on the work of Marion et al. [2025], conducted in the context of classical transformers and motivated by natural language processing (NLP) tasks. In that paper, the authors introduced a new class of statistical models, namely Single Location Regression (SLR) models, through which they aimed to investigate the ability of standard attention layers to recover relevant information in a token-wise sparse information setting, where the target depends only on a relatively small number of tokens. They have provided theoretical guarantees for a modification of the standard transformer architecture, proving that one can recover optimal predictors for SLRs. This naturally raises a fundamental, yet unanswered, question: Can we achieve the same statistical and optimization performance with axial attention as we do with normal global attention? The answer is not obvious, since the information flow is much more restricted in an axial transformer, where each token only attends to tokens with which it shares an axis, as opposed to the free flow of information between any two tokens in a standard transformer.
Building upon the foundation established by Marion et al. [2025], we introduce a SLR model and extend the theoretical analysis of transformer architectures to axial attention mechanisms, as a method of assessing the theoretical capabilities of current state-of-the-art TSFMs. Our main contributions are structured as follows:
In § 2 , we introduce an SLR statistical model for the token-wise sparse information setting adapted to 2D structured data (e.g., multivariate time series, tables). We then explain the connection between such a model and multivariate time series anomaly analysis, and finally, we introduce a trainable predictor and show that it corresponds to a slightly simplified axial attention layer.
In Section 3, we analyze the performance of the predictor introduced and prove that there exists an asymptotic regime in which it is statistically optimal for the SLR model investigated, when using the oracle parameters. Furthermore, we show that removing the inner non-linearity in our predictor renders it suboptimal.
In Section 4, we propose an optimization algorithm for our predictor and provide theoretical guarantees for its convergence, despite the non-convexity of the problem. We further conduct extensive numerical studies which reveal a complex optimization landscape that confirm our theoretical findings.
Our structure closely follows that of Marion et al. [2025], but our motivation (the study of TSFMs) and the technical challenges are, nevertheless, very different. Standard global attention benefits from symmetric token coupling, which facilitates the derivation of closed-form analytical solutions for the expected risk. By contrast, axial attention operates in a sequential manner. This decoupled architecture breaks the global symmetry and induces a cascading non-linear dependence between axes. As a result, the optimization landscape exhibits significant complexity, rendering standard closed-form approaches to statistical optimality and convergence of optimization methods intractable and necessitating novel analytical techniques.
SLR model and simplified axial attention predictor
In this section, we introduce an SLR model and underline its connection to multivariate time series anomaly analysis. Afterwards, we consider a simplification of axial attention that renders the statistical analysis in Section 3 tractable.
Single location regression model. We focus on the following statistical model, where the output Y ∈ R Y \in\mathbb{R} Y ∈ R is assumed to be given by
Y = X I 0 J 0 T v ∗ + ξ , (3) Y = X_{I_0J_0}^{\mathsf{T}}v^* + \xi, \tag*{(3)} Y = X I 0 J 0 T v ∗ + ξ , (3)
where ξ ∼ N ( 0 , ϵ 2 ) \xi\sim\mathcal{N}(0,\epsilon^2) ξ ∼ N ( 0 , ϵ 2 ) is independent noise and I 0 I_0 I 0 , J 0 J_0 J 0 are discrete random variables over [ M ] [M] [ M ] and [ N ] [N] [ N ] , respectively. No restriction on the distribution of ( I 0 , J 0 ) (I_0, J_0) ( I 0 , J 0 ) is imposed unless otherwise specified. Conditionally on ( I 0 , J 0 ) (I_0,J_0) ( I 0 , J 0 ) , for ( ℓ , γ 2 , ϵ 2 , k ∗ , v ∗ ) ∈ R + × R + ∗ × R + ∗ × S d − 1 × S d − 1 (\ell,\gamma^2,\epsilon^2,k^*,v^*) \in\mathbb{R}_+ \times\mathbb{R}_+^* \times\mathbb{R}_+^* \times\mathbb{S}^{d-1} \times\mathbb{S}^{d-1} ( ℓ , γ 2 , ϵ 2 , k ∗ , v ∗ ) ∈ R + × R + ∗ × R + ∗ × S d − 1 × S d − 1 ,
{ X I 0 J 0 ∼ N ( ℓ k ∗ , γ 2 I d ) , X i j ∼ N ( 0 , I d ) , for ( i , j ) ≠ ( I 0 , J 0 ) , \begin{cases}
X_{I_0J_0} \sim\mathcal{N}(\ell k^*,\gamma^2 I_d),\\
X_{ij} \sim\mathcal{N}(0,I_d),\quad\text{for }(i,j)\ne(I_0,J_0),
\end{cases} { X I 0 J 0 ∼ N ( ℓ k ∗ , γ 2 I d ) , X ij ∼ N ( 0 , I d ) , for ( i , j ) = ( I 0 , J 0 ) ,
where X i j X_{ij} X ij are independent for all ( i , j ) ∈ [ M ] × [ N ] (i,j) \in[M] \times[N] ( i , j ) ∈ [ M ] × [ N ] .
The parameters of (SLR) are the shift magnitude ℓ \ell ℓ , the localization and noise variances γ 2 \gamma^2 γ 2 and ϵ 2 \epsilon^2 ϵ 2 , the unobserved projection vector v ∗ v^* v ∗ and the localization vector k ∗ k^* k ∗ , with both vectors taking values on the d d d -dimensional unit sphere S d − 1 \mathbb{S}^{d-1} S d − 1 . The target variable Y Y Y depends exclusively on a single embedded token within the M × N M \times N M × N input matrix, positioned at random discrete coordinates ( I 0 , J 0 ) (I_0,J_0) ( I 0 , J 0 ) . This active token is statistically identifiable because its expected value is shifted along the direction k ∗ k^* k ∗ . Consequently, the primary mathematical challenge of this task stems from its dynamic sparsity.
Remark 1. Unlike in Marion et al. [2025], we consider ℓ \ell ℓ to be decoupled from d d d throughout our analysis, but note that for ℓ = d / 2 \ell= \sqrt{d/2} ℓ = d /2 and γ 2 = 1 / 2 \gamma^2 = 1/2 γ 2 = 1/2 , all tokens have the same expected norm, i.e. E [ ∥ X I 0 J 0 ∥ 2 2 ] = E [ ∥ X i j ∥ 2 2 ] = d \mathbb{E}[\lVert X_{I_0J_0}\rVert_2^2] = \mathbb{E}[\lVert X_{ij}\rVert_2^2] = d E [∥ X I 0 J 0 ∥ 2 2 ] = E [∥ X ij ∥ 2 2 ] = d , making them indistinguishable just based on their magnitude.
Connection to multivariate time series anomaly analysis. The Single Location Regression (SLR) model detailed above provides a natural theoretical framework for multivariate time series anomaly analysis, particularly when operating on learned representation embeddings X X X . We consider system-wide anomalous events identifiable through correlated behaviors across multiple sensors, as pictured in Figure 2 . We operate under the assumption that the embedding layer has already captured these correlated dynamics, localizing the signature of the system-wide anomaly into the embedding of a single primary driving sensor I 0 I_0 I 0 at a specific time step J 0 J_0 J 0 . By investigating whether axial attention, which restricts information flow, can successfully recover the true parameters k ∗ k^* k ∗ and v ∗ v^* v ∗ , we can theoretically assess its ability to isolate and leverage localized anomalies.
Figure 2. Under normal operating conditions, the background embeddings behave as uninformative noise, or as a standard token, modeled by the standard normal distribution N ( 0 , I d ) \mathcal{N}(0,I_d) N ( 0 , I d ) . When an anomaly occurs in one sensor, its specific signature is encoded into the token X I 0 J 0 X_{I_0J_0} X I 0 J 0 , characterized by a shifted mean ℓ k ∗ \ell k^* ℓ k ∗ and a localization variance γ 2 I d \gamma^2 I_d γ 2 I d . The regression target Y Y Y represents a downstream consequence or system-level anomaly score that depends strictly on identifying this specific embedded state via the projection vector v ∗ v^* v ∗ . Our goal is to investigate whether we can recover this target through an axial attention layer.
Furthermore, using axial attention layers instead of standard global attention is attractive also from a computational point of view. The global attention mechanism treats the input as a flattened 1D sequence of length M × N M \times N M × N and this coupling requires O ( M 2 N 2 ) \mathcal{O}(M^2N^2) O ( M 2 N 2 ) operations per layer, while axial attention only requires O ( M N 2 + M 2 N ) \mathcal{O}(MN^2+M^2N) O ( M N 2 + M 2 N ) (see Ho et al. [2019] for a derivation).
An analytically tractable simplification of axial attention for SLR. In what follows, we introduce a class of predictors that are direct simplifications of the axial attention layer. For an activation σ : R d → R d \sigma:\mathbb{R}^d \to\mathbb{R}^d σ : R d → R d and inverse temperatures λ r , λ c ∈ ( 0 , ∞ ) \lambda_r,\lambda_c \in(0,\infty) λ r , λ c ∈ ( 0 , ∞ ) , let
T λ r , λ c , σ ( k , v ) ( X ) : = σ ( λ c X ( 1 / 2 ) k ) T X ( 1 / 2 ) v , where X i ( 1 / 2 ) : = σ ( λ r X i k ) T X i , (4) T_{\lambda_r,\lambda_c,\sigma}^{(k,v)}(\mathbf{X}) := \sigma\left(\lambda_c \mathbf{X}^{(1/2)}k\right)^\mathsf{T}\mathbf{X}^{(1/2)}v,\qquad\text{where }\mathbf{X}_i^{(1/2)} := \sigma\left(\lambda_r \mathbf{X}_i k\right)^\mathsf{T}\mathbf{X}_i, \tag*{(4)} T λ r , λ c , σ ( k , v ) ( X ) := σ ( λ c X ( 1/2 ) k ) T X ( 1/2 ) v , where X i ( 1/2 ) := σ ( λ r X i k ) T X i , (4)
for i ∈ [ M ] i \in[M] i ∈ [ M ] and learnable parameters k , v ∈ S d − 1 k,v \in\mathbb{S}^{d-1} k , v ∈ S d − 1 . Note that we slightly overload the notation and use X ( 1 / 2 ) \mathbf{X}^{(1/2)} X ( 1/2 ) to denote the half step of both our predictor and of axial attention.
In the context of TSFMs, sequences are typically appended with designated [REG] tokens to act as readout tokens or attention sinks (see Ansari et al. [ [1 ] , for example). Assuming that the input X \mathbf{X} X is appended with a ( N + 1 ) (N+1) ( N + 1 ) th column of row-aggregation [REG_ROW] tokens and a column aggregation [REG_COL] token at position ( M + 1 , N + 1 ) (M+1,N+1) ( M + 1 , N + 1 ) , we have the following lemma
Lemma 1. For p r = p c = 1 p_r=p_c=1 p r = p c = 1 , o c = 1 o_c=1 o c = 1 , O c = 1 O_c=1 O c = 1 , together with shared keys K r = K c K_r=K_c K r = K c and V r = I d V_r=I_d V r = I d , we have the following equivalence:
T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) M + 1 , N + 1 = T λ r a , λ c a ( 1 / 2 ) , σ ( K c , V c ) ( X ) , T_{\lambda_r,\lambda_c,\sigma}^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}(\mathbf{X})_{M+1,N+1}=T_{\lambda_r a,\lambda_c a^{(1/2)},\sigma}^{(K_c,V_c)}(\mathbf{X}), T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) M + 1 , N + 1 = T λ r a , λ c a ( 1/2 ) , σ ( K c , V c ) ( X ) , for some constants a a a and a ( 1 / 2 ) a^{(1/2)} a ( 1/2 ) . The proof and further comments on the constraints are in Appendix § 6 .
The main bottleneck in trying to analyze σ = softmax \sigma=\operatorname{softmax} σ = softmax in (4) is the softmax operation that induces a global coupling between the tokens, which acts as a major roadblock to rigorous statistical analysis. To bypass this issue without losing the core behavioral properties of the network, we substitute softmax with the component-wise nonlinear function erf ( x ) = 2 π ∫ 0 x e − u 2 d u \operatorname{erf}(x)=\frac{2}{\sqrt{\pi}}\int_0^x e^{-u^2}\,\mathrm{d}u erf ( x ) = π 2 ∫ 0 x e − u 2 d u , following Marion et al. [2025]. This adjustment leads to the definition of the analytically tractable predictor that we investigate throughout the remainder of the paper,
T λ r , λ c ( k , v ) ( X ) : = T λ r , λ c , erf ( k , v ) ( X ) = erf ( λ c X ( 1 / 2 ) k ) T X ( 1 / 2 ) v , where X i ( 1 / 2 ) : = erf ( λ r X i k ) T X i . (5) T_{\lambda_r,\lambda_c}^{(k,v)}(\mathbf{X}) := T_{\lambda_r,\lambda_c,\operatorname{erf}}^{(k,v)}(\mathbf{X}) = \operatorname{erf}\left(\lambda_c\mathbf{X}^{(1/2)}k\right)^\mathsf{T}\mathbf{X}^{(1/2)}v,\qquad\text{where }\mathbf{X}_i^{(1/2)}:=\operatorname{erf}\left(\lambda_r\mathbf{X}_ik\right)^\mathsf{T}\mathbf{X}_i. \tag*{(5)} T λ r , λ c ( k , v ) ( X ) := T λ r , λ c , erf ( k , v ) ( X ) = erf ( λ c X ( 1/2 ) k ) T X ( 1/2 ) v , where X i ( 1/2 ) := erf ( λ r X i k ) T X i . (5)
Statistical analysis
Having established the predictor T λ r , λ c ( k , v ) T_{\lambda_r,\lambda_c}^{(k,v)} T λ r , λ c ( k , v ) , we now turn to evaluating its statistical properties.
Asymptotic optimality of the oracle predictor
In this section, we follow the approach of Marion et al. [2025]: we first observe that our predictor only depends on the alignments between k k k , v v v , k ∗ k^* k ∗ and v ∗ v^* v ∗ , projecting the high-dimensional risk landscape down to five scalars, which is essential to the analysis of the training dynamics in Section 4. Second, under the assumption of orthogonality between k ∗ k^{*} k ∗ and v ∗ v^{*} v ∗ , we obtain a simplified formula for the risk. Finally, using this formula, we find an asymptotic regime in which our predictor is optimal. Although we follow the same structure and maintain similar notation as Marion et al. [2025], in order to optimize readability, the sequential nature of axial attention makes the results and the proofs more complicated as we lack a closed form formula for the risk. Detailed proofs for all theoretical claims in this section can be found in Appendix § 7 .
We quantify the model’s performance via its risk:
R λ r , λ c ( k , v ) : = E [ ( Y − T λ r , λ c ( k , v ) ( X ) ) 2 ] . (6) \mathcal{R}_{\lambda_r,\lambda_c}(k,v) := \mathbb{E}\left[\left(Y-T_{\lambda_r,\lambda_c}^{(k,v)}(\mathbb{X})\right)^2\right]. \tag*{(6)} R λ r , λ c ( k , v ) := E [ ( Y − T λ r , λ c ( k , v ) ( X ) ) 2 ] . (6)
Furthermore, we make the following assumption throughout the rest of the paper.
Assumption 1. The vectors k ∗ , v ∗ ∈ S d − 1 k^{*},v^{*}\in\mathbb{S}^{d-1} k ∗ , v ∗ ∈ S d − 1 are orthogonal to each other i.e. k ∗ T v ∗ = 0 k^{*T}v^{*}=0 k ∗ T v ∗ = 0 .
Note that enforcing orthogonality between k ∗ k^{*} k ∗ and v ∗ v^{*} v ∗ is a mild requirement in high-dimensional embedding spaces, commonly found in TSFMs. The following lemma establishes a reduced representation of the risk of our axial estimator based on the alignments between k k k , v v v , k ∗ k^{*} k ∗ and v ∗ v^{*} v ∗ .
Lemma 2. There exists a function R λ r , λ c < : R 5 → R \mathcal{R}^{<}_{\lambda_r,\lambda_c}:\mathbb{R}^{5}\to\mathbb{R} R λ r , λ c < : R 5 → R such that for any ( k , v ) ∈ ( S d − 1 ) 2 (k,v)\in(\mathbb{S}^{d-1})^{2} ( k , v ) ∈ ( S d − 1 ) 2 ,
R λ r , λ c ( k , v ) = R λ r , λ c < ( κ , ν , θ , η , ρ ) , (7) \mathcal{R}_{\lambda_r,\lambda_c}(k,v)=\mathcal{R}^{<}_{\lambda_r,\lambda_c}(\kappa,\nu,\theta,\eta,\rho), \tag*{(7)} R λ r , λ c ( k , v ) = R λ r , λ c < ( κ , ν , θ , η , ρ ) , (7)
where κ = k T k ∗ \kappa=k^{T}k^{*} κ = k T k ∗ , ν = v T v ∗ \nu=v^{T}v^{*} ν = v T v ∗ , θ = v T k ∗ \theta=v^{T}k^{*} θ = v T k ∗ , η = k T v ∗ \eta=k^{T}v^{*} η = k T v ∗ and ρ = k T v \rho=k^{T}v ρ = k T v . An expression for R λ r , λ c < \mathcal{R}^{<}_{\lambda_r,\lambda_c} R λ r , λ c < is given in Appendix § 7 .
Next, the Bayes risk for the regression task in (SLR) is bounded below by the variance ϵ 2 \epsilon^{2} ϵ 2 of the noise ξ \xi ξ . Building on Lemma 2, Proposition 3 demonstrates that our oracle estimator T λ r , λ c ( k ∗ , v ∗ ) T_{\lambda_r,\lambda_c}^{(k^{*},v^{*})} T λ r , λ c ( k ∗ , v ∗ ) recovers this theoretical lower bound under a specific asymptotic regime.
Proposition 3. Assume a joint asymptotic scaling where λ r ℓ → ∞ \lambda_r\ell\to\infty λ r ℓ → ∞ , λ c ℓ → ∞ \lambda_c\ell\to\infty λ c ℓ → ∞ and λ r = o ( M − 1 2 N − 1 2 ) \lambda_r=o\left(M^{-\frac{1}{2}}N^{-\frac{1}{2}}\right) λ r = o ( M − 2 1 N − 2 1 ) . Then,
R λ r , λ c ( k ∗ , v ∗ ) → ϵ 2 . \mathcal{R}_{\lambda_r,\lambda_c}(k^{*},v^{*})\to\epsilon^{2}. R λ r , λ c ( k ∗ , v ∗ ) → ϵ 2 .
Consequently, the oracle axial predictor T λ r , λ c ( k ∗ , v ∗ ) T_{\lambda_r,\lambda_c}^{(k^{*},v^{*})} T λ r , λ c ( k ∗ , v ∗ ) achieves asymptotic Bayes optimality.
Note that Proposition 3 does not impose any scaling on d d d , which is expected given Lemma 2. We provide a non-rigorous intuition for the imposed scalings. First, we wish X I 0 J 0 X_{I_{0}J_{0}} X I 0 J 0 to pass through the first layer asymptotically unaltered, that is,
X I 0 J 0 ≈ X I 0 J 0 ( 1 / 2 ) = erf ( λ r X I 0 J 0 T k ∗ ) X I 0 J 0 + ∑ j ≠ J 0 erf ( λ r X I 0 j T k ∗ ) X I 0 j . X_{I_{0}J_{0}}\approx X_{I_{0}J_{0}}^{(1/2)}=\operatorname{erf}\left(\lambda_r X_{I_{0}J_{0}}^{T}k^{*}\right)X_{I_{0}J_{0}}+\sum_{j\ne J_{0}}\operatorname{erf}\left(\lambda_r X_{I_{0}j}^{T}k^{*}\right)X_{I_{0}j}. X I 0 J 0 ≈ X I 0 J 0 ( 1/2 ) = erf ( λ r X I 0 J 0 T k ∗ ) X I 0 J 0 + j = J 0 ∑ erf ( λ r X I 0 j T k ∗ ) X I 0 j .
Therefore, we need λ r ℓ → ∞ \lambda_r\ell\to\infty λ r ℓ → ∞ to ensure that the argument of the first erf goes to ∞ \infty ∞ and λ r → 0 \lambda_r\to0 λ r → 0 to ensure that the others go to 0 0 0 . In an analogous manner, we wish X I 0 J 0 X_{I_{0}J_{0}} X I 0 J 0 to pass unaltered through the second layer, which leads to λ c ℓ → ∞ \lambda_c\ell\to\infty λ c ℓ → ∞ . Lastly, in the expansion of T λ r , λ c ( k ∗ , v ∗ ) T_{\lambda_r,\lambda_c}^{(k^{*},v^{*})} T λ r , λ c ( k ∗ , v ∗ ) , there appear ( M − 1 ) N (M-1)N ( M − 1 ) N noise terms of the form C ⋅ erf ( λ r X i j T k ∗ ) X i j T v ∗ C\cdot\operatorname{erf}(\lambda_rX_{ij}^{T}k^{*})X_{ij}^{T}v^{*} C ⋅ erf ( λ r X ij T k ∗ ) X ij T v ∗ , with C < 1 C<1 C < 1 , one for each ( i , j ) (i,j) ( i , j ) with i ≠ I 0 i\ne I_{0} i = I 0 . Multiple scalings could make the sum of these terms go to 0 0 0 , one of them being λ r = o ( M − 1 2 N − 1 2 ) \lambda_r=o\left(M^{-\frac{1}{2}}N^{-\frac{1}{2}}\right) λ r = o ( M − 2 1 N − 2 1 ) , as shown in the proof in Appendix § 7 .
Removing the inner non-linearity is suboptimal
To fully appreciate the asymptotic optimality of our oracle predictor, it is instructive to compare it against semi-linear models, which replace the inner non-linearity with a fixed linear combination layer governed by β ∈ R N \beta\in\mathbb{R}^{N} β ∈ R N . This justifies the necessity for the inner non-linearity in the axial attention layer in order to achieve asymptotic optimality. More precisely, consider the semi-linear predictor
T β , λ c ( k , v ) ( X ) : = erf ( λ c X ( 1 / 2 ) k ) T X ( 1 / 2 ) v , where X i ( 1 / 2 ) : = β T X i . (8) T_{\beta,\lambda_c}^{(k,v)}(\mathbb{X}):=\operatorname{erf}\left(\lambda_c\mathbb{X}^{(1/2)}k\right)^{T}\mathbb{X}^{(1/2)}v,\quad\text{where }\mathbb{X}_{i}^{(1/2)}:=\beta^{T}\mathbb{X}_{i}. \tag*{(8)} T β , λ c ( k , v ) ( X ) := erf ( λ c X ( 1/2 ) k ) T X ( 1/2 ) v , where X i ( 1/2 ) := β T X i . (8)
Then we have the following sub-optimality statement.
Proposition 4. Let p j = P ( J 0 = j ) p_j = \mathbb{P}(J_0 = j) p j = P ( J 0 = j ) for j ∈ [ N ] j \in[N] j ∈ [ N ] . Denote by S i = β T X i S_i = \beta^T X_i S i = β T X i . Then, for any function f : R d M → R f:\mathbb{R}^{dM} \to\mathbb{R} f : R d M → R in L 2 ( R d M ) L^2(\mathbb{R}^{dM}) L 2 ( R d M ) , the semi-linear predictor achieves a risk strictly larger than ϵ 2 \epsilon^2 ϵ 2 for any choice of β ≠ 0 \beta\ne0 β = 0 and hyperparameters M M M , N N N , ℓ \ell ℓ , λ c \lambda_c λ c i.e.
E [ ( Y − f ( S 1 , S 2 , … , S M ) ) 2 ] ≥ ϵ 2 + γ 2 ( 1 − ∑ j 0 = 1 N p j 0 γ 2 β j 0 2 γ 2 β j 0 2 + ∑ j ≠ j 0 β j 2 ) > ϵ 2 , \mathbb{E}\left[\left(Y-f(S_1,S_2,\ldots,S_M)\right)^2\right] \ge\epsilon^2+\gamma^2\left(1-\sum_{j_0=1}^{N}p_{j_0}\frac{\gamma^2\beta_{j_0}^2}{\gamma^2\beta_{j_0}^2+\sum_{j\ne j_0}\beta_j^2}\right)>\epsilon^2, E [ ( Y − f ( S 1 , S 2 , … , S M ) ) 2 ] ≥ ϵ 2 + γ 2 ( 1 − j 0 = 1 ∑ N p j 0 γ 2 β j 0 2 + ∑ j = j 0 β j 2 γ 2 β j 0 2 ) > ϵ 2 , if there is no j ∈ [ N ] j\in[N] j ∈ [ N ] with p j = 1 p_j=1 p j = 1 . We provide a proof of this statement in Appendix § 7 .§ 2 .
Remark 2. Discarding the outer non-linearity leads to a less restrictive information flow compared to our predictor, and it amounts to a simplification of the standard attention mechanism. Indeed,
T λ r , β ( k , v ) ( X ) : = ∑ i = 1 M β i ∑ j = 1 N erf ( λ r X i j T k ) X i j T v T_{\lambda_r,\beta}^{(k,v)}(X):=\sum_{i=1}^{M}\beta_i\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_r X_{ij}^{T}k\right)X_{ij}^{T}v T λ r , β ( k , v ) ( X ) := i = 1 ∑ M β i j = 1 ∑ N erf ( λ r X ij T k ) X ij T v is a modification of the trainable predictor proposed in Marion et al. [2025], which is asymptotically optimal for β i = 1 \beta_i=1 β i = 1 , i ∈ [ M ] i\in[M] i ∈ [ M ] , under the joint scaling λ r = o ( M − 1 2 N − 1 2 ) \lambda_r=o\left(M^{-\frac{1}{2}}N^{-\frac{1}{2}}\right) λ r = o ( M − 2 1 N − 2 1 ) and λ r ℓ → ∞ \lambda_r\ell\to\infty λ r ℓ → ∞ .
Gradient descent dynamics
The last section of our analysis is concerned with the theoretical training dynamics of our predictor (5) for the risk R λ r , λ c ( k , v ) R_{\lambda_r,\lambda_c}(k,v) R λ r , λ c ( k , v ) . While in practice training minimizes an empirical risk evaluated over a finite dataset, we limit our investigation of the empirical risk minimization to numerical simulations, which confirm our findings about the theoretical risk.
Convergence of gradient descent
Lemma 2 shows that on the manifold ( S d − 1 ) 2 (\mathbb{S}^{d-1})^2 ( S d − 1 ) 2 the risk can be described through 5 alignment metrics. We want to use this fact and formula since it renders the analysis tractable after further constraints. Therefore, it is natural to employ Projected Riemannian Gradient Descent (PGD) as an optimization algorithm for two main reasons: it keeps the parameters on the manifold and it guarantees that the Euclidean projection back onto the sphere remains well-defined by preventing the parameters from reaching the origin. We emphasize that this geometric constraint is necessary because our reduced risk formulation (7) is strictly valid only on this manifold.
Table 1.
Analyzing the dynamics of the resulting 5-dimensional system is intractable due to the highly complex nature of R λ r , λ c < R_{\lambda_r,\lambda_c}^{<} R λ r , λ c < . To enable a rigorous analysis, we restrict our focus to scenarios in which the parameters are initialized on the submanifold M \mathcal{M} M defined by
\mathcal{M}:=\left\{(k,v)\in(\mathbb{S}^{d-1})^2\mathrel{\left|} k^Tv^*=0,\ v^Tk^*=0,\ k^Tv=0\right.\right\}. \tag*{(9)}
Under Assumption 1, the true underlying parameters ( k ∗ , v ∗ ) (k^*,v^*) ( k ∗ , v ∗ ) naturally reside within M \mathcal{M} M and, crucially, the training parameters remain on the submanifold throughout the optimization process if they are initialized on it, from Lemma 6 in Appendix § 8 . Due to this invariance, we only need to characterize the dynamics of the system restricted to M \mathcal{M} M when initialized on this manifold. Although in practice this cannot be enforced, we show empirically in § 4.2 that penalizing a proxy for the distance to the manifold leads to similar dynamics. This lower-dimensional analysis is mathematically tractable and leads to the primary optimization result of this paper, which guarantees asymptotic recovery of the oracle parameters.
Theorem 5. Consider τ > 0 \tau> 0 τ > 0 . Then, there exists λ ˉ > 0 \bar{\lambda} > 0 λ ˉ > 0 and α ˉ > 0 \bar{\alpha} > 0 α ˉ > 0 such that for constant inverse temperatures λ > λ r > 0 \lambda> \lambda_r > 0 λ > λ r > 0 and λ c = τ λ r \lambda_c = \tau\lambda_r λ c = τ λ r , for any step size α ≤ α ˉ \alpha\le\bar{\alpha} α ≤ α ˉ and for a generic initialization ( k 0 , v 0 ) ∈ M (k_0, v_0) \in\mathcal{M} ( k 0 , v 0 ) ∈ M ,
( k t , v t ) → ± ( k ∗ , v ∗ ) as t → ∞ . (k_t, v_t) \to\pm(k^*, v^*) \quad\text{as } t \to\infty. ( k t , v t ) → ± ( k ∗ , v ∗ ) as t → ∞. The convergence to ( − k ∗ , − v ∗ ) (-k^*, -v^*) ( − k ∗ , − v ∗ ) is not a problem, as by symmetry around 0 0 0 , T λ r , λ c ( k ∗ , v ∗ ) = T λ r , λ c ( − k ∗ , − v ∗ ) T_{\lambda_r,\lambda_c}^{(k^*,v^*)} = T_{\lambda_r,\lambda_c}^{(-k^*,-v^*)} T λ r , λ c ( k ∗ , v ∗ ) = T λ r , λ c ( − k ∗ , − v ∗ ) .
Thus, the two results provide a sequential two-stage guarantee: parameter recovery is performed in an optimization-friendly temperature regime, followed by prediction in a statistically optimal inference regime. This separation is also used in our experiments, where the evaluation risk is computed using fixed inverse temperatures independently of the schedule used for optimization.
Understanding the dynamics - numerical experiments
In practice, one cannot use the theoretical risk for optimization using PGD, since there is no closed form formula for it. Furthermore, one also does not have knowledge of what values for λ r \lambda_r λ r and λ c \lambda_c λ c are small enough. In our experiments, we bypass these issues by using batch stochastic PGD for the optimization process and implementing a decreasing schedule for the inverse temperatures.
Figure 3 presents numerical results of the optimization obtained for initializations on the manifold using stochastic PGD. We perfectly recover the oracle parameters, up to numerical error, and there is no departure from the manifold, again, up to numerical error and noise, as was predicted by theory.
Figure 3. Dynamics of stochastic PGD to the oracle parameters over 20 independent runs. The solid curve represents the empirical median across the runs and the bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals (although not always visible). Left: The empirical risk obtained by our predictor on the training batches of data using both the training parameters k k k , v v v and the oracle parameters k ∗ k^* k ∗ , v ∗ v^* v ∗ . For computing the oracle and model losses we use the initial inverse temperatures. Middle-left: The alignment between the training and oracle parameters κ = k T k ∗ \kappa= k^T k^* κ = k T k ∗ , ν = v T v ∗ \nu= v^T v^* ν = v T v ∗ . Middle-right: The distance from the manifold M \mathcal{M} M of the training parameters ( k , v ) (k,v) ( k , v ) . Right: The schedule used for λ = λ r = λ c \lambda= \lambda_r = \lambda_c λ = λ r = λ c . The hyperparameters of the experiment are M = N = 5 \mathcal{M} = \mathbb{N} = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 and λ t = 5 ⋅ 10 − 2 / ( 1 + max ( t − 2 ⋅ 10 4 , 0 ) ⋅ 10 − 4 ) \lambda_t = 5 \cdot10^{-2}/(1+\max(t-2\cdot10^4,0)\cdot10^{-4}) λ t = 5 ⋅ 1 0 − 2 / ( 1 + max ( t − 2 ⋅ 1 0 4 , 0 ) ⋅ 1 0 − 4 ) .
We emphasize the fact that the schedule for the inverse temperatures is crucial for the dynamics. To understand this, in Figure 4 we run an experiment in which we train 4 predictors using different schedulers: 20k warm-up steps with λ c = λ r = 0.1 \lambda_c = \lambda_r = 0.1 λ c = λ r = 0.1 , followed by different values of constant inverse temperatures λ = λ c = λ r \lambda= \lambda_c = \lambda_r λ = λ c = λ r . The leftmost plots show two different behaviors for ν \nu ν and κ \kappa κ : ν \nu ν converges fast, usually before the warm up period ends, while the convergence of κ \kappa κ is highly dependent on the value of λ \lambda λ . The heat maps present the respective partial derivatives of R λ r , λ c < R_{\lambda_r,\lambda_c}^{<} R λ r , λ c < for the different values of λ \lambda λ . They show that for λ \lambda λ small enough, minus the gradient always points towards the global minima when far from the origin. For large values of λ \lambda λ , the dynamics push away from the origin much faster, but the gradients vanish and have the wrong sign close to the boundaries κ = ± 1 \kappa= \pm1 κ = ± 1 . As a consequence, we see that for those values, κ \kappa κ does not converge to one, while it does for λ = 0.01 \lambda= 0.01 λ = 0.01 . The warm start is needed, as for small λ \lambda λ , the dynamics would remain stuck around the axis κ = 0 \kappa= 0 κ = 0 where the values of the partial derivatives are extremely small, as visible on the plot.
Figure 4. (a) Alignment κ \kappa κ and the partial derivative ∂ κ R λ , λ < \partial_{\kappa} R_{\lambda,\lambda}^{<} ∂ κ R λ , λ < . (b) Alignment ν \nu ν and the partial derivative ∂ ν R λ , λ < \partial_{\nu} R_{\lambda,\lambda}^{<} ∂ ν R λ , λ < . Leftmost: Alignment dynamics during the training of our predictor with an inverse temperature warm-up period of 20k steps using λ c = λ r = 0.1 \lambda_c = \lambda_r = 0.1 λ c = λ r = 0.1 , followed by constant λ c = λ r = λ \lambda_c = \lambda_r = \lambda λ c = λ r = λ for λ ∈ { 0.01 , 0.03 , 0.05 , 0.07 } \lambda\in\{0.01, 0.03, 0.05, 0.07\} λ ∈ { 0.01 , 0.03 , 0.05 , 0.07 } . We plot the median across 20 runs for each value of λ \lambda λ . The dashed vertical line represents the warm up period. Center-left to right: The numerically computed gradients of R λ r , λ r < ( κ , ν ) R_{\lambda_r,\lambda_r}^{<}(\kappa, \nu) R λ r , λ r < ( κ , ν ) for λ r = λ c = λ ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_r = \lambda_c = \lambda\in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ r = λ c = λ ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } , N = 5 N = 5 N = 5 and ℓ = 100 \ell= 100 ℓ = 100 . The gradient, given by (34) , is approximated using the Monte Carlo method with a sample size of 10,000.
One last and very important problem remains: in practice we cannot enforce the initialization to be on the manifold M \mathcal{M} M , since that would require knowing the oracle parameters k ∗ k^{*} k ∗ and v ∗ v^{*} v ∗ . We can ensure that k T v = 0 k^{T}v = 0 k T v = 0 at initialization and penalize their inner product throughout the training. Furthermore, a noisy proxy for v T k ∗ v^{T}k^{*} v T k ∗ is 1 ℓ 2 ∑ i , j X i j \frac{1}{\ell^{2}} \sum_{i,j} X_{ij} ℓ 2 1 ∑ i , j X ij . Therefore, in the case of initialization outside the manifold, we impose k T v = 0 k^{T}v = 0 k T v = 0 at initialization and optimize for the penalized risk
R ‾ λ r , λ c ( k , v ) = R λ r , λ c ( k , v ) + μ 1 ( k T v ) 2 + μ 2 E [ ( v T ∑ i , j X i j ) 2 ] , (10) \overline{R}_{\lambda_r,\lambda_c}(k,v) = R_{\lambda_r,\lambda_c}(k,v) + \mu_1(k^{T}v)^2 + \mu_2 \mathbb{E}\left[\left(v^{T}\sum_{i,j}X_{ij}\right)^2\right], \tag*{(10)} R λ r , λ c ( k , v ) = R λ r , λ c ( k , v ) + μ 1 ( k T v ) 2 + μ 2 E ( v T i , j ∑ X ij ) 2 , (10)
where μ 1 , μ 2 ∈ ( 0 , ∞ ) \mu_1,\mu_2 \in(0,\infty) μ 1 , μ 2 ∈ ( 0 , ∞ ) are fixed penalization coefficients. A similar use of regularization to encourage the orthogonality conditions defining an invariant manifold is explored by Maulen Soto et al. [2025] for attention-based clustering, using a different penalty to handle initializations outside the manifold.
Indeed, Figure 5 presents the results obtained when optimizing for this penalized empirical risk from a generic random initialization on ( S d − 1 ) 2 (\mathbb{S}^{d-1})^2 ( S d − 1 ) 2 . The top row (Figure 5 a) displays the training dynamics for our trainable predictor (4, employing the erf non-linearity. We observe that the penalty terms effectively guide the parameters towards the invariant manifold M \mathcal{M} M throughout the training and the convergence is slower, even though we have used a higher initial inverse temperature. The lower row (Figure 5 b) presents the identical experimental setup applied to the predictor (3) with σ = softmax \sigma= \operatorname{softmax} σ = softmax . Although the training dynamics are not identical, both models converge to the manifold M \mathcal{M} M and there are phase transitions in the empirical loss curves for both models. In the case of the erf model, the phase transition appears to be correlated with ν \nu ν reaching near-perfect alignment, while in the case of the softmax model, the phase transition presents itself when both ν \nu ν and κ \kappa κ reach near-perfect alignment.
Figure 5. (a) Training dynamics of the trainable predictor (5) (i.e. (4) with σ = erf \sigma= \operatorname{erf} σ = erf ). (b) Training dynamics of the softmax predictor (4) with σ = softmax \sigma= \operatorname{softmax} σ = softmax . Figure 5: Convergence of stochastic PGD over 20 runs with random initializations on S d − 1 × S d − 1 \mathbb{S}^{d-1} \times\mathbb{S}^{d-1} S d − 1 × S d − 1 , optimizing for the penalized empirical risk (10) . The solid curve represents the empirical median across the runs. The hyperparameters of the experiment are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 , μ 1 = 10 \mu_1 = 10 μ 1 = 10 , μ 2 = 1 \mu_2 = 1 μ 2 = 1 . The inverse temperature schedules are λ t = 10 − 1 / ( 1 + max ( t − 5 ⋅ 10 3 , 0 ) ⋅ 10 − 4 ) \lambda_t = 10^{-1}/(1 + \max(t - 5 \cdot10^3, 0) \cdot10^{-4}) λ t = 1 0 − 1 / ( 1 + max ( t − 5 ⋅ 1 0 3 , 0 ) ⋅ 1 0 − 4 ) for the erf predictor and λ t = 10 − 1 / ( 1 + max ( t − 2 ⋅ 10 4 , 0 ) ⋅ 10 − 4 ) \lambda_t = 10^{-1}/(1 + \max(t - 2 \cdot10^4, 0) \cdot10^{-4}) λ t = 1 0 − 1 / ( 1 + max ( t − 2 ⋅ 1 0 4 , 0 ) ⋅ 1 0 − 4 ) for the softmax predictor. For computing the oracle and model losses we use the initial inverse temperatures.
As we have shown in the present experiments, the inverse temperature schedule is of the utmost importance to the performance of the training procedure, and finding a suitable one that performs well across all initializations adds a layer of complexity to the optimization process. A more in-depth analysis of and further details related to the numerical experiments can be found in Appendix § 10 .
Conclusion and discussion
We presented a rigorous theoretical framework for understanding axial attention in TSFMs by extending the SLR model to 2D grids. We proved that an analytically tractable simplification of axial attention achieves asymptotic Bayes optimality for the oracle parameters, whereas semi-linear models in which the inner non-linearity is removed strictly fail to do so. Furthermore, we established that Projected Gradient Descent converges to these optimal parameters for small enough inverse temperatures. Extensive numerical experiments validate our theoretical findings and highlight the critical role of inverse temperature scheduling in converging to the global minima within the non-convex optimization landscape.
Our statistical framework assumes fixed temporal and spatial dimensions, independent background tokens, and a target that depends on a single localized token. Relaxing these assumptions provides several natural directions for future work. In particular, incorporating autoregressive temporal dynamics and dense cross-channel correlations would better reflect the structure of real multivariate time series, while allowing the dimensions or number of informative tokens to grow would broaden the range of sparse information settings covered by the model. These extensions would introduce additional dependencies into both the statistical model and the optimization landscape and would therefore require analytical tools beyond those used in the present work.
It is also necessary to explicitly reiterate the architectural constraints that make the analysis possible. Our results are heavily dependent on a rank-1 constraint applied to the query and key projection matrices, as well as the substitution of the standard softmax activation with the erf function and the shared-key constraint K r = K c K_r = K_c K r = K c . The theoretical results in this paper apply to this simplified erf predictor, while our experiments with the corresponding softmax predictor provide encouraging evidence that similar optimization behavior persists beyond the activation function used in the analysis. Full TSFMs additionally contain realistic projection matrices, residual connections, normalization layers, and multiple axial-attention blocks; the implications of our results for these architectures are therefore qualitative. Nevertheless, our Single Location Regression model preserves the core structural challenge of identifying and exploiting a sparsely located informative token, making it highly relevant for the theoretical study of axial attention in TSFMs.
Finally, while our current focus is restricted to a single informative token, the same framework suggests a natural extension to multiple-token regression through multi-head attention. If the target depends on several informative locations, different heads could learn distinct localization directions and specialize in selecting different tokens, while their outputs could be combined to recover the target. Under suitable identifiability and separation assumptions, parts of the statistical analysis could then be applied head-wise, following an approach similar to the one developed here. The optimization problem would, however, become substantially richer: the heads would be trained jointly, their roles would be interchangeable, and several heads could compete for or collapse onto the same informative location. We leave this extension, together with the study of the corresponding statistical and optimization guarantees, for future work.
AI use statement. We have used generative AI tools to assist with writing and cleaning code, and with editing the manuscript. The authors reviewed all AI-assisted outputs and take responsibility for the final content of this work.
Acknowledgments. We thank Pierre Marion, whose insightful comments helped us improve the final version of this paper. The work of Andrei Tiberiu Pantea and Aymeric Dieuleveut is supported by French State aid managed by the Agence Nationale de la Recherche (ANR) under the France 2030 program ANR-23-IACL-0005 (Hi! PARIS).
Proof of Lemma 1 and further discussion about architectural constraints
We recall the definition of the standard axial self-attention layer.
T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) ⋅ j = σ ( λ c X ⋅ j ( 1 / 2 ) Q c K c T ( X ⋅ j ( 1 / 2 ) ) T ⏟ M × p c p c × M ) X ⋅ j ( 1 / 2 ) V c O c ⏟ M × p c p c × o c , T^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}_{\lambda_r,\lambda_c,\sigma}(X)_{\cdot j}
=
\sigma\left(
\underbrace{\lambda_c X_{\cdot j}^{(1/2)}Q_cK_c^T\left(X_{\cdot j}^{(1/2)}\right)^T}_{M\times p_c\quad p_c\times M}
\right)
\underbrace{X_{\cdot j}^{(1/2)}V_cO_c}_{M\times p_c\quad p_c\times o_c}, T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) ⋅ j = σ M × p c p c × M λ c X ⋅ j ( 1/2 ) Q c K c T ( X ⋅ j ( 1/2 ) ) T M × p c p c × o c X ⋅ j ( 1/2 ) V c O c ,
where the initial row-wise computation is:
X i ⋅ ( 1 / 2 ) = σ ( λ r X i ⋅ Q r K r T X i ⋅ T ⏟ N × p r p r × N ) X i ⋅ V r ⏟ N × d . X_{i\cdot}^{(1/2)}
=
\sigma\left(
\underbrace{\lambda_r X_{i\cdot}Q_rK_r^TX_{i\cdot}^T}_{N\times p_r\quad p_r\times N}
\right)
\underbrace{X_{i\cdot}V_r}_{N\times d}. X i ⋅ ( 1/2 ) = σ N × p r p r × N λ r X i ⋅ Q r K r T X i ⋅ T N × d X i ⋅ V r .
In order to justify our simplified estimator (4) , we explicitly map it to the mechanism of an actual axial attention block. As mentioned in the main paper, sequences are typically appended with designated [REG] tokens to act as either readout tokens or attention sinks in the context of TSFMs. Following this convention, we assume that the input’s final column consists of specialized [REG_ROW] tokens. To collapse the 2D grid into a single regression scalar, we then incorporate an additional [REG_COL] token at the specific index ( M + 1 , N + 1 ) (M + 1, N + 1) ( M + 1 , N + 1 ) , which acts as a global aggregator during the subsequent column-wise operation. We further assume the two [REG] tokens do not attend to themselves, which is a very light assumption, since a priori these tokens contain no information. Isolating the output for this specific token yields the following
X i , N + 1 ( 1 / 2 ) = ∑ l = 1 N σ ( λ r X [ R E G _ R O W ] T Q r K r T X i T ) l X i , l T V r = σ ( λ r a K r T X i T ) X i ⋅ ∈ R d X^{(1/2)}_{i,N+1}=\sum_{l=1}^{N}\sigma\left(\lambda_r X^{T}_{[\mathrm{REG\_ROW}]}Q_rK_r^{T}X_i^{T}\right)_lX_{i,l}^{T}V_r=\sigma\left(\lambda_raK_r^{T}X_i^{T}\right)X_{i\cdot}\in\mathbb{R}^{d} X i , N + 1 ( 1/2 ) = l = 1 ∑ N σ ( λ r X [ REG_ROW ] T Q r K r T X i T ) l X i , l T V r = σ ( λ r a K r T X i T ) X i ⋅ ∈ R d
if we assume that the projection V r ∈ R d × d V_r\in\mathbb{R}^{d\times d} V r ∈ R d × d is the identity matrix and encapsulate the query interaction as a vector a : = X [ R E G _ R O W ] T Q r ∈ R 1 × p r a:=X^{T}_{[\mathrm{REG\_ROW}]}Q_r\in\mathbb{R}^{1\times p_r} a := X [ REG_ROW ] T Q r ∈ R 1 × p r . By applying an analogous definition for the column-wise query a ( 1 / 2 ) : = X [ R E G _ C O L ] T Q c ∈ R 1 × p c a^{(1/2)}:=X^{T}_{[\mathrm{REG\_COL}]}Q_c\in\mathbb{R}^{1\times p_c} a ( 1/2 ) := X [ REG_COL ] T Q c ∈ R 1 × p c , the output of the final attention block becomes:
T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) M + 1 , N + 1 = ∑ i = 1 M σ ( λ c a ( 1 / 2 ) K c T X i , N + 1 ( 1 / 2 ) ) i ( X i , N + 1 ( 1 / 2 ) ) T V c O c T^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}_{\lambda_r,\lambda_c,\sigma}(X)_{M+1,N+1}=\sum_{i=1}^{M}\sigma\left(\lambda_ca^{(1/2)}K_c^{T}X^{(1/2)}_{i,N+1}\right)_i\left(X^{(1/2)}_{i,N+1}\right)^{T}V_cO_c T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) M + 1 , N + 1 = i = 1 ∑ M σ ( λ c a ( 1/2 ) K c T X i , N + 1 ( 1/2 ) ) i ( X i , N + 1 ( 1/2 ) ) T V c O c
To perfectly align this with our theoretical model, we enforce rank-1 restrictions on the projections, setting p r = p c = 1 p_r=p_c=1 p r = p c = 1 . With these constraints in place, along with o c = 1 o_c=1 o c = 1 , O c = 1 O_c=1 O c = 1 , and shared keys K r = K c K_r=K_c K r = K c , the full axial mechanism cleanly reduces to our formulated predictor:
T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) M + 1 , N + 1 = T λ r a , λ c a ( 1 / 2 ) , σ ( K c , V c ) ( X ) T^{(Q_{r,c},K_{r,c},V_{r,c},O_c)}_{\lambda_r,\lambda_c,\sigma}(X)_{M+1,N+1}=T^{(K_c,V_c)}_{\lambda_ra,\lambda_ca^{(1/2)},\sigma}(X) T λ r , λ c , σ ( Q r , c , K r , c , V r , c , O c ) ( X ) M + 1 , N + 1 = T λ r a , λ c a ( 1/2 ) , σ ( K c , V c ) ( X )
Discussion. In this paragraph, we address the main architectural constraints and modifications imposed in our analysis.
REG tokens. The addition of REG tokens is not a relevant modification, since any transformer layer whose output must be a single value, be it scalar of vector, will implement a global aggregation mechanism for the input tokens and will be trained for it. REG tokens remove the inductive biases that come with pooling methods or positional decoding and allow for a learned aggregation mechanism.
Rank-1 restriction on projection matrices. This constraint is not at all harsh in the context of SLR, where there exists a single informative direction in the latent space, namely k ∗ k^* k ∗ . Moreover, Yu et al. [2026] has shown that patch-based TSFMs admit low-rank approximations of the projection matrices.
Shared keys. K r = K c K_r=K_c K r = K c is indeed a strong constraint in practice. However, in the particular case of the studied SLR, there is a single special direction in the embedding space, k ∗ k^* k ∗ , and, heuristically, token selection should be made according to this direction.
Non-linearity. For the theoretical analysis, we have restricted the value of σ \sigma σ to erf, maintaining the same convention as in Marion et al. [2025]. However, only a couple of properties of the erf function are essential to the present proofs, such as global boundedness of erf and of its derivatives, the asymptotics lim x → ∞ erf ( x ) = 1 \lim_{x\to\infty}\operatorname{erf}(x)=1 lim x → ∞ erf ( x ) = 1 and lim x → − ∞ erf ( x ) = − 1 \lim_{x\to-\infty}\operatorname{erf}(x)=-1 lim x → − ∞ erf ( x ) = − 1 , erf ( x ) = 0 \operatorname{erf}(x)=0 erf ( x ) = 0 etc. Extending the proofs and the numerical analysis to other activation functions seems feasible and would be an interesting further research direction. Furthermore, the statistical role of the activation function in single-location regression is investigated by Duranthon et al. [2026], who establish advantages of softmax over alternative activations in a single-axis setting. Understanding how these comparisons extend to sequential axial attention is an interesting direction for future work.
Proofs for Section 3
In what follows, we shall denote by
R < ( κ , ν ) : = R < ( κ , ν , 0 , 0 , 0 ) , \mathcal{R}^{<}(\kappa,\nu):=\mathcal{R}^{<}(\kappa,\nu,0,0,0), R < ( κ , ν ) := R < ( κ , ν , 0 , 0 , 0 ) ,
the value of the risk when restricted to M \mathcal{M} M .
Proofs of Lemma 2 and of Proposition 3
We recall the formula for the risk
R λ r , λ c ( κ , ν ) = E [ ( Y − ∑ i = 1 M erf ( λ c ∑ j = 1 N erf ( λ r X i j T κ ) X i j T κ ) ∑ j = 1 N erf ( λ r X i j T κ ) X i j T ν ) 2 ] (11) \mathcal{R}_{\lambda_r,\lambda_c}(\kappa,\nu)=\mathbb{E}\left[\left(Y-\sum_{i=1}^{M}\operatorname{erf}\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}(\lambda_rX_{ij}^{T}\kappa)X_{ij}^{T}\kappa\right)\sum_{j=1}^{N}\operatorname{erf}(\lambda_rX_{ij}^{T}\kappa)X_{ij}^{T}\nu\right)^2\right] \tag*{(11)} R λ r , λ c ( κ , ν ) = E ( Y − i = 1 ∑ M erf ( λ c j = 1 ∑ N erf ( λ r X ij T κ ) X ij T κ ) j = 1 ∑ N erf ( λ r X ij T κ ) X ij T ν ) 2 (11)
Let us now denote by G i j u = X i j T u G_{ij}^{u}=X_{ij}^{T}u G ij u = X ij T u , for u ∈ R d u \in\mathbb{R}^{d} u ∈ R d . Note that we consider X I 0 J 0 = ℓ k ⋆ + γ X I 0 J 0 X_{I_0J_0}=\ell k^{\star}+\gamma X_{I_0J_0} X I 0 J 0 = ℓ k ⋆ + γ X I 0 J 0 and G I 0 J 0 u = X ~ I 0 J 0 T u G_{I_0J_0}^{u}=\widetilde{X}_{I_0J_0}^{T}u G I 0 J 0 u = X I 0 J 0 T u . Then, our predictor only depends on κ \kappa κ , θ \theta θ and the alignment random variables G i j u ⋆ G_{ij}^{u\star} G ij u ⋆ , G i j v G_{ij}^{v} G ij v , G i j v ⋆ G_{ij}^{v\star} G ij v ⋆ which are i.i.d. and follow the distribution
( G i j k G i j v G i j k ⋆ G i j v ⋆ ) ∼ N ( 0 , ( 1 ρ η ρ 1 ν η ν 1 ) ) . \begin{pmatrix} G_{ij}^{k} \\ G_{ij}^{v} \\ G_{ij}^{k\star} \\ G_{ij}^{v\star} \end{pmatrix}
\sim\mathcal{N}\left(0,\begin{pmatrix} 1 & \rho& \eta\\ \rho& 1 & \nu\\ \eta& \nu& 1 \end{pmatrix}\right). G ij k G ij v G ij k ⋆ G ij v ⋆ ∼ N 0 , 1 ρ η ρ 1 ν η ν 1 .
We further assume, without loss of generality, that I 0 = J 0 = 1 I_0=J_0=1 I 0 = J 0 = 1 . Then Y = γ G 11 v ⋆ + ξ Y=\gamma G_{11}^{v\star}+\xi Y = γ G 11 v ⋆ + ξ and, dropping the subscripts of R λ r , λ c \mathcal{R}_{\lambda_r,\lambda_c} R λ r , λ c , one indeed has
R ( κ , ν ) = E [ ( γ G 11 v ⋆ + ξ − T λ r , λ c ( κ , ν ) ( X ) ) 2 ] = ϵ 2 + E [ ( γ G 11 v ⋆ − T λ r , λ c ( κ , ν ) ( X ) ) 2 ] = ϵ 2 + γ 2 − 2 γ E [ G 11 v ⋆ T λ r , λ c ( κ , ν ) ( X ) ] + E [ ( T λ r , λ c ( κ , ν ) ( X ) ) 2 ] . (12) \begin{aligned}
\mathcal{R}(\kappa,\nu)=\mathbb{E}\left[\left(\gamma G_{11}^{v\star}+\xi-T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})\right)^2\right] \\
&=\epsilon^2+\mathbb{E}\left[\left(\gamma G_{11}^{v\star}-T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})\right)^2\right] \\
&=\epsilon^2+\gamma^2-2\gamma\mathbb{E}\left[G_{11}^{v\star}T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})\right]+\mathbb{E}\left[\left(T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})\right)^2\right]. \tag*{(12)}
\end{aligned} R ( κ , ν ) = E [ ( γ G 11 v ⋆ + ξ − T λ r , λ c ( κ , ν ) ( X ) ) 2 ] = ϵ 2 + E [ ( γ G 11 v ⋆ − T λ r , λ c ( κ , ν ) ( X ) ) 2 ] = ϵ 2 + γ 2 − 2 γ E [ G 11 v ⋆ T λ r , λ c ( κ , ν ) ( X ) ] + E [ ( T λ r , λ c ( κ , ν ) ( X ) ) 2 ] . (12)
where
T λ r , λ c ( κ , ν ) ( X ) = ∑ i = 2 M erf ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) ∑ j = 1 N erf ( λ r G i j k ) G i j v + erf ( λ c erf ( λ r κ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( erf ( λ r κ + λ r γ G 11 k ) ( θ ℓ + γ G 11 v ) + ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j v ) . (13) \begin{aligned}
T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})=\sum_{i=2}^{M}\operatorname{erf}\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{k}\right)\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{v} \\
&+\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_r\kappa+\lambda_r\gamma G_{11}^{k}\right)\left(\kappa\ell+\gamma G_{11}^{k}\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{k}\right) \\
&\cdot\left(\operatorname{erf}\left(\lambda_r\kappa+\lambda_r\gamma G_{11}^{k}\right)\left(\theta\ell+\gamma G_{11}^{v}\right)+\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{v}\right). \tag*{(13)}
\end{aligned} T λ r , λ c ( κ , ν ) ( X ) = i = 2 ∑ M erf ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) j = 1 ∑ N erf ( λ r G ij k ) G ij v + erf ( λ c erf ( λ r κ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( erf ( λ r κ + λ r γ G 11 k ) ( θ ℓ + γ G 11 v ) + j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j v ) . (13)
Since Equation (12) and Equation (13) only depend on the 5 parameters κ , ν , ρ , η , θ \kappa,\nu,\rho,\eta,\theta κ , ν , ρ , η , θ , this ends the proof of Lemma 2. On the manifold M \mathcal{M} M , where ρ = θ = 0 \rho=\theta=0 ρ = θ = 0 , we can simplify the expression by dropping the θ ℓ \theta\ell θ ℓ term in the last row of Equation (13) . We continue our proof by analyzing each term in Equation (12) individually.
The first term can be rewritten as
E [ G 11 v ⋆ T λ r , λ c ( κ , ν ) ( X ) ] = = ∑ i = 2 M E [ erf ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) ∑ j = 1 N erf ( λ r G i j k ) G i j v G 11 v ⋆ ] + E [ erf ( λ c erf ( λ r κ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) \begin{aligned}
\mathbb{E}\left[G_{11}^{v\star}T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})\right]= \\
&=\sum_{i=2}^{M}\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{k}\right)\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{v}G_{11}^{v\star}\right] \\
&+\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_r\kappa+\lambda_r\gamma G_{11}^{k}\right)\left(\kappa\ell+\gamma G_{11}^{k}\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{k}\right)\right.
\end{aligned} E [ G 11 v ⋆ T λ r , λ c ( κ , ν ) ( X ) ] = = i = 2 ∑ M E [ erf ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) j = 1 ∑ N erf ( λ r G ij k ) G ij v G 11 v ⋆ ] + E [ erf ( λ c erf ( λ r κ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k )
⋅ ( erf ( λ r κ + λ r γ G 11 k ) ( θ ℓ + γ G 11 v ) G 11 v ⋆ + ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j v G 11 v ⋆ ) ] . (14) \left.\cdot\left(\operatorname{erf}\left(\lambda_r\kappa+\lambda_r\gamma G_{11}^{k}\right)\left(\theta\ell+\gamma G_{11}^{v}\right)G_{11}^{v\star}+\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{v}G_{11}^{v\star}\right)\right]. \tag*{(14)} ⋅ ( erf ( λ r κ + λ r γ G 11 k ) ( θ ℓ + γ G 11 v ) G 11 v ⋆ + j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j v G 11 v ⋆ ) ] . (14)
On the manifold M \mathcal{M} M , this becomes:
E [ G 11 v ⋆ T λ r , λ c ( κ , ν ) ( X ) ] = = ∑ i = 2 M E [ erf ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) ∑ j = 1 N erf ( λ r G i j k ) G i j v G 11 v ⋆ ] + E [ erf ( λ c erf ( λ r κ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) (15) \begin{aligned}
\mathbb{E}\left[G_{11}^{v\star}T_{\lambda_r,\lambda_c}^{(\kappa,\nu)}(\mathsf{X})\right]= \\
&=\sum_{i=2}^{M}\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{k}\right)\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{v}G_{11}^{v\star}\right] \\
&+\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_r\kappa+\lambda_r\gamma G_{11}^{k}\right)\left(\kappa\ell+\gamma G_{11}^{k}\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{k}\right)\right. \tag*{(15)}
\end{aligned} E [ G 11 v ⋆ T λ r , λ c ( κ , ν ) ( X ) ] = = i = 2 ∑ M E [ erf ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) j = 1 ∑ N erf ( λ r G ij k ) G ij v G 11 v ⋆ ] + E [ erf ( λ c erf ( λ r κ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) (15)
⋅ [ erf ( λ c k ℓ + λ r γ G 11 k ) γ G 11 v G 11 v ∗ + ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j v G 11 v ∗ ] = ∑ i = 2 M ∑ j = 1 N E [ erf ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) erf ( λ r G i j k ) G i j v ] E [ G 11 v ∗ ] + E [ erf ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r k ℓ + λ r γ G 11 k ) ] γ E [ G 11 v G 11 v ∗ ] + ∑ j = 2 N E [ erf ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r G 1 j k ) G 1 j v ] E [ G 11 v ∗ ] = γ ν E [ erf ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r k ℓ + λ r γ G 11 k ) ] , (16) \begin{aligned}
&\cdot\left[\operatorname{erf}\left(\lambda_c k\ell+\lambda_r\gamma G_{11}^k\right)\gamma G_{11}^vG_{11}^{v*}+\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^vG_{11}^{v*}\right]\\
&=\sum_{i=2}^{M}\sum_{j=1}^{N}\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^k\right)G_{ij}^k\right)\operatorname{erf}\left(\lambda_rG_{ij}^k\right)G_{ij}^v\right]\mathbb{E}\left[G_{11}^{v*}\right]\\
&\quad+\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\left(k\ell+\gamma G_{11}^k\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^k\right)\right.\\
&\qquad\left.\cdot\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\right]\gamma\mathbb{E}\left[G_{11}^vG_{11}^{v*}\right]\\
&\quad+\sum_{j=2}^{N}\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\left(k\ell+\gamma G_{11}^k\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^k\right)\right.\\
&\qquad\left.\cdot\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^v\right]\mathbb{E}\left[G_{11}^{v*}\right]\\
&=\gamma\nu\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\left(k\ell+\gamma G_{11}^k\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^k\right)\right.\\
&\qquad\left.\cdot\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\right], \tag*{(16)}
\end{aligned} ⋅ [ erf ( λ c k ℓ + λ r γ G 11 k ) γ G 11 v G 11 v ∗ + j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j v G 11 v ∗ ] = i = 2 ∑ M j = 1 ∑ N E [ erf ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) erf ( λ r G ij k ) G ij v ] E [ G 11 v ∗ ] + E [ erf ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r k ℓ + λ r γ G 11 k ) ] γ E [ G 11 v G 11 v ∗ ] + j = 2 ∑ N E [ erf ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r G 1 j k ) G 1 j v ] E [ G 11 v ∗ ] = γ ν E [ erf ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r k ℓ + λ r γ G 11 k ) ] , (16)
where we have used the independence between G 11 v ∗ G_{11}^{v*} G 11 v ∗ and G i j v G_{ij}^v G ij v , G i j k G_{ij}^k G ij k .
Due to the independence between G i j v G_{ij}^v G ij v and G r s v G_{rs}^v G r s v for ( i , j ) ≠ ( r , s ) (i,j)\ne(r,s) ( i , j ) = ( r , s ) , there appear no cross-terms when expanding the expectation of T λ r , λ c ( k , v ) ( X ) 2 T_{\lambda_r,\lambda_c}^{(k,v)}(\mathbf{X})^2 T λ r , λ c ( k , v ) ( X ) 2 . Therefore, under ρ = 0 \rho=0 ρ = 0 , the second term simplifies as
E [ T λ r , λ c ( k , v ) ( X ) 2 ] = ∑ i = 2 M ∑ j = 1 N E [ erf 2 ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) ⋅ erf 2 ( λ r G i j k ) ( G i j v ) 2 ] + E [ erf 2 ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( erf 2 ( λ r k ℓ + λ r γ G 11 k ) ( θ ℓ + γ G 11 v ) 2 + ∑ j = 2 N erf 2 ( λ r G 1 j k ) ( G 1 j v ) 2 + 2 θ ℓ erf ( λ r k ℓ + λ r γ G 11 k ) ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j v ) ] , (17) \begin{aligned}
\mathbb{E}\left[T_{\lambda_r,\lambda_c}^{(k,v)}(\mathbf{X})^2\right]
&=\sum_{i=2}^{M}\sum_{j=1}^{N}\mathbb{E}\left[\operatorname{erf}^2\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^k\right)G_{ij}^k\right)\cdot\operatorname{erf}^2\left(\lambda_rG_{ij}^k\right)\left(G_{ij}^v\right)^2\right]\\
&\quad+\mathbb{E}\left[\operatorname{erf}^2\left(\lambda_c\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\left(k\ell+\gamma G_{11}^k\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^k\right)\right.\\
&\qquad\cdot\left(\operatorname{erf}^2\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\left(\theta\ell+\gamma G_{11}^v\right)^2+\sum_{j=2}^{N}\operatorname{erf}^2\left(\lambda_rG_{1j}^k\right)\left(G_{1j}^v\right)^2\right.\\
&\qquad\left.\left.+2\theta\ell\operatorname{erf}\left(\lambda_rk\ell+\lambda_r\gamma G_{11}^k\right)\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^k\right)G_{1j}^v\right)\right], \tag*{(17)}
\end{aligned} E [ T λ r , λ c ( k , v ) ( X ) 2 ] = i = 2 ∑ M j = 1 ∑ N E [ erf 2 ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) ⋅ erf 2 ( λ r G ij k ) ( G ij v ) 2 ] + E [ erf 2 ( λ c erf ( λ r k ℓ + λ r γ G 11 k ) ( k ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( erf 2 ( λ r k ℓ + λ r γ G 11 k ) ( θ ℓ + γ G 11 v ) 2 + j = 2 ∑ N erf 2 ( λ r G 1 j k ) ( G 1 j v ) 2 + 2 θ ℓ erf ( λ r k ℓ + λ r γ G 11 k ) j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j v ) ] , (17)
On the manifold M \mathcal{M} M (ρ = θ = η = 0 \rho=\theta=\eta=0 ρ = θ = η = 0 ), this becomes
E [ T λ r , λ c ( k , v ) ( X ) 2 ] = ∑ i = 2 M ∑ j = 1 N E [ erf 2 ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) ⋅ erf 2 ( λ r G i j k ) + E [ erf 2 ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( γ 2 erf 2 ( λ r κ ℓ + λ r γ G 11 k ) + ∑ j = 2 N erf 2 ( λ r G 1 j k ) ) ] \begin{aligned}
\mathbb{E}[T_{\lambda_r,\lambda_c}^{(k,v)}(\mathbf{X})^2]
&=\sum_{i=2}^{M}\sum_{j=1}^{N}\mathbb{E}[\operatorname{erf}^2(\lambda_c\sum_{j=1}^{N}\operatorname{erf}(\lambda_rG_{ij}^k)G_{ij}^k)\cdot\operatorname{erf}^2(\lambda_rG_{ij}^k) \\
&+\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_c\operatorname{erf}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)\left(\ell+\gamma G_{11}^{k}\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_r G_{1j}^{k}\right)G_{1j}^{k}\right)\right.
\\
\left.\cdot\left(\gamma^{2}\operatorname{erf}^{2}\left(\lambda_r\kappa\ell+\lambda_r\gamma G_{11}^{k}\right)+\sum_{j=2}^{N}\operatorname{erf}^{2}\left(\lambda_r G_{1j}^{k}\right)\right)\right]
\end{aligned} E [ T λ r , λ c ( k , v ) ( X ) 2 ] ⋅ ( γ 2 erf 2 ( λ r κ ℓ + λ r γ G 11 k ) + j = 2 ∑ N erf 2 ( λ r G 1 j k ) ) ] = i = 2 ∑ M j = 1 ∑ N E [ erf 2 ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) ⋅ erf 2 ( λ r G ij k ) + E [ erf 2 ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k )
In what follows, we shall prove that under the joint scaling λ r ℓ → ∞ \lambda_r\ell\to\infty λ r ℓ → ∞ , λ c ℓ → ∞ \lambda_c\ell\to\infty λ c ℓ → ∞ and λ r = o ( M − 1 2 N − 1 2 ) \lambda_r=o\left(M^{-\frac{1}{2}}N^{-\frac{1}{2}}\right) λ r = o ( M − 2 1 N − 2 1 ) , the risk converges to ϵ 2 \epsilon^{2} ϵ 2 . For the oracle parameters, κ = ν = 1 \kappa=\nu=1 κ = ν = 1 . Moreover, erf ( λ r x ) x ≥ 0 \operatorname{erf}(\lambda_r x)x\geq0 erf ( λ r x ) x ≥ 0 for all x ∈ R x\in\mathbb{R} x ∈ R and ∣ erf ( x ) ∣ ≤ 1 |\operatorname{erf}(x)|\leq1 ∣ erf ( x ) ∣ ≤ 1 . Therefore, Equation (14) implies
E [ G 11 ∗ T λ r , λ c ( k , ν ) ( X ) ] ≥ γ E [ erf ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) ) erf ( λ r ℓ + λ r γ G 11 k ) ] \mathbb{E}\left[G_{11}^{*}T_{\lambda_r,\lambda_c}^{(k,\nu)}(\mathsf{X})\right]\geq\gamma\mathbb{E}\left[\operatorname{erf}\left(\lambda_c\operatorname{erf}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)\left(\ell+\gamma G_{11}^{k}\right)\right)\operatorname{erf}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)\right] E [ G 11 ∗ T λ r , λ c ( k , ν ) ( X ) ] ≥ γ E [ erf ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) ) erf ( λ r ℓ + λ r γ G 11 k ) ]
− γ Φ ( γ G 11 k ≤ − ℓ ) = γ E [ erf ( λ c λ r erf ( G ) G ) erf ( G ) ] − γ Φ ( − ℓ γ ) , \begin{aligned}
-\gamma\Phi\left(\gamma G_{11}^{k}\leq-\ell\right) \\
&=\gamma\mathbb{E}\left[\operatorname{erf}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(G)G\right)\operatorname{erf}(G)\right]-\gamma\Phi\left(-\frac{\ell}{\gamma}\right),
\end{aligned} − γ Φ ( γ G 11 k ≤ − ℓ ) = γ E [ erf ( λ r λ c erf ( G ) G ) erf ( G ) ] − γ Φ ( − γ ℓ ) ,
where G ∼ N ( λ r ℓ , λ r 2 γ 2 ) G\sim\mathcal{N}\left(\lambda_r\ell,\lambda_r^{2}\gamma^{2}\right) G ∼ N ( λ r ℓ , λ r 2 γ 2 ) and Φ \Phi Φ is the cumulative distribution function of N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) .
Next, since ∣ erf ( x ) ∣ ≤ 1 |\operatorname{erf}(x)|\leq1 ∣ erf ( x ) ∣ ≤ 1 and erf 2 ( x ) ≤ 4 π x 2 \operatorname{erf}^{2}(x)\leq\frac{4}{\pi}x^{2} erf 2 ( x ) ≤ π 4 x 2 from Lemma 12 for all x ∈ R x\in\mathbb{R} x ∈ R , one has
∑ i = 2 M ∑ j = 1 N E [ erf 2 ( λ c ∑ j = 1 N erf ( λ r G i j k ) G i j k ) erf 2 ( λ r G i j k ) ] ≤ ∑ i = 2 M ∑ j = 1 N E [ erf 2 ( λ r G i j k ) ] ≤ ∑ i = 2 M ∑ j = 1 N 4 λ r 2 π E [ ( G i j k ) 2 ] = 4 π λ r 2 ( M − 1 ) N → 0 , \begin{aligned}
\sum_{i=2}^{M}\sum_{j=1}^{N}\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_c\sum_{j=1}^{N}\operatorname{erf}\left(\lambda_rG_{ij}^{k}\right)G_{ij}^{k}\right)\operatorname{erf}^{2}\left(\lambda_rG_{ij}^{k}\right)\right]
\leq\sum_{i=2}^{M}\sum_{j=1}^{N}\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_rG_{ij}^{k}\right)\right] \\
&\leq\sum_{i=2}^{M}\sum_{j=1}^{N}\frac{4\lambda_r^{2}}{\pi}\mathbb{E}\left[\left(G_{ij}^{k}\right)^{2}\right] \\
&=\frac{4}{\pi}\lambda_r^{2}(M-1)N\to0,
\end{aligned} i = 2 ∑ M j = 1 ∑ N E [ erf 2 ( λ c j = 1 ∑ N erf ( λ r G ij k ) G ij k ) erf 2 ( λ r G ij k ) ] ≤ i = 2 ∑ M j = 1 ∑ N E [ erf 2 ( λ r G ij k ) ] ≤ i = 2 ∑ M j = 1 ∑ N π 4 λ r 2 E [ ( G ij k ) 2 ] = π 4 λ r 2 ( M − 1 ) N → 0 ,
due to the scaling λ r = o ( M − 1 2 N − 1 2 ) \lambda_r=o\left(M^{-\frac{1}{2}}N^{-\frac{1}{2}}\right) λ r = o ( M − 2 1 N − 2 1 ) .
Finally, substituting κ = 1 \kappa=1 κ = 1 and using ∣ erf ( x ) ∣ ≤ 1 |\operatorname{erf}(x)|\leq1 ∣ erf ( x ) ∣ ≤ 1 and erf 2 ( x ) ≤ 4 π x 2 \operatorname{erf}^{2}(x)\leq\frac{4}{\pi}x^{2} erf 2 ( x ) ≤ π 4 x 2 again from Lemma 12 , one has
E [ erf 2 ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( γ 2 erf 2 ( λ r ℓ + λ r γ G 11 k ) + ∑ j = 2 N erf 2 ( λ r G 1 j k ) ) ] ≤ γ 2 E [ erf 2 ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) erf 2 ( λ r ℓ + λ r γ G 11 k ) + ∑ j = 2 N E [ erf 2 ( λ r G 1 j k ) ] ] ≤ γ 2 E [ erf 2 ( λ c λ r erf ( G ) G + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) erf 2 ( G ) ] + 4 π λ r 2 ( N − 1 ) , \begin{aligned}
\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_c\operatorname{erf}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)\left(\ell+\gamma G_{11}^{k}\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{k}\right)\right.
\\
\left.\cdot\left(\gamma^{2}\operatorname{erf}^{2}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)+\sum_{j=2}^{N}\operatorname{erf}^{2}\left(\lambda_rG_{1j}^{k}\right)\right)\right] \\
&\leq\gamma^{2}\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_c\operatorname{erf}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)\left(\ell+\gamma G_{11}^{k}\right)+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{k}\right)\right.
\\
\left.\operatorname{erf}^{2}\left(\lambda_r\ell+\lambda_r\gamma G_{11}^{k}\right)+\sum_{j=2}^{N}\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_rG_{1j}^{k}\right)\right]\right] \\
&\leq\gamma^{2}\mathbb{E}\left[\operatorname{erf}^{2}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(G)G+\lambda_c\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_rG_{1j}^{k}\right)G_{1j}^{k}\right)\operatorname{erf}^{2}(G)\right]+\frac{4}{\pi}\lambda_r^{2}(N-1),
\end{aligned} E [ erf 2 ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ( γ 2 erf 2 ( λ r ℓ + λ r γ G 11 k ) + j = 2 ∑ N erf 2 ( λ r G 1 j k ) ) ] erf 2 ( λ r ℓ + λ r γ G 11 k ) + j = 2 ∑ N E [ erf 2 ( λ r G 1 j k ) ] ] ≤ γ 2 E [ erf 2 ( λ c erf ( λ r ℓ + λ r γ G 11 k ) ( ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ≤ γ 2 E [ erf 2 ( λ r λ c erf ( G ) G + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) erf 2 ( G ) ] + π 4 λ r 2 ( N − 1 ) ,
where G ∼ N ( λ r ℓ , λ r 2 γ 2 ) G\sim\mathcal{N}\left(\lambda_r\ell,\lambda_r^{2}\gamma^{2}\right) G ∼ N ( λ r ℓ , λ r 2 γ 2 ) .
Aggregating everything and considering the oracle parameters k ∗ , ν ∗ k^{*},\nu^{*} k ∗ , ν ∗ , one now has that
R ( k ∗ , ν ∗ ) ≤ ϵ 2 + γ 2 − 2 γ 2 E [ erf ( λ c λ r erf ( G ) G ) erf ( G ) ] + 2 γ 2 Φ ( − ℓ γ ) + γ 2 E [ erf 2 ( λ c λ r erf ( G ) G + λ c ∑ j = 2 G 1 j k ≥ 0 N erf ( λ r G 1 j k ) G 1 j k ) erf 2 ( G ) ] + o ( 1 ) , \begin{aligned}
\mathcal{R}(k^{*},\nu^{*})\leq\epsilon^{2}+\gamma^{2}-2\gamma^{2}\mathbb{E}\left[\operatorname{erf}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(G)G\right)\operatorname{erf}(G)\right]+2\gamma^{2}\Phi\left(-\frac{\ell}{\gamma}\right) \\
&+ \gamma^2 \mathbb{E}\left[\operatorname{erf}^2\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(G)G+\lambda_c\sum_{\substack{j=2\\G_{1j}^k\ge0}}^{N}\operatorname{erf}(\lambda_rG_{1j}^k)G_{1j}^k\right)\operatorname{erf}^2(G)\right]+o(1),
\end{aligned} R ( k ∗ , ν ∗ ) ≤ ϵ 2 + γ 2 − 2 γ 2 E [ erf ( λ r λ c erf ( G ) G ) erf ( G ) ] + 2 γ 2 Φ ( − γ ℓ ) + γ 2 E erf 2 λ r λ c erf ( G ) G + λ c j = 2 G 1 j k ≥ 0 ∑ N erf ( λ r G 1 j k ) G 1 j k erf 2 ( G ) + o ( 1 ) ,
where G ∼ N ( λ r ℓ , λ r 2 γ 2 ) G\sim\mathcal{N}(\lambda_r\ell,\lambda_r^2\gamma^2) G ∼ N ( λ r ℓ , λ r 2 γ 2 ) .
To conclude the proof of Proposition 3, it is enough to prove
E [ erf ( λ c λ r erf ( G ) G ) erf ( G ) ] → 1 , \mathbb{E}\left[\operatorname{erf}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(G)G\right)\operatorname{erf}(G)\right]\to1, E [ erf ( λ r λ c erf ( G ) G ) erf ( G ) ] → 1 ,
as ℓ → ∞ \ell\to\infty ℓ → ∞ implies Φ ( − ℓ γ ) → 0 \Phi\left(-\frac{\ell}{\gamma}\right)\to0 Φ ( − γ ℓ ) → 0 . Since x ↦ erf ( λ c λ r erf ( x ) x ) erf ( x ) x\mapsto\operatorname{erf}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(x)x\right)\operatorname{erf}(x) x ↦ erf ( λ r λ c erf ( x ) x ) erf ( x ) is increasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) and less than 1 in absolute value, it is enough to consider
E [ erf ( λ c λ r erf ( G ) G ) erf ( G ) ] ≥ erf ( λ c λ r erf ( λ r ℓ 2 ) λ r ℓ 2 ) erf ( λ r ℓ 2 ) P ( G ≥ λ r ℓ 2 ) − P ( G < 0 ) = erf ( λ c ℓ 2 erf ( λ r ℓ 2 ) ) erf ( λ r ℓ 2 ) ( 1 − Φ ( − ℓ 2 γ ) ) − Φ ( − ℓ γ ) . \begin{aligned}
\mathbb{E}\left[\operatorname{erf}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}(G)G\right)\operatorname{erf}(G)\right]
&\ge\operatorname{erf}\left(\frac{\lambda_c}{\lambda_r}\operatorname{erf}\left(\frac{\lambda_r\ell}{2}\right)\frac{\lambda_r\ell}{2}\right)\operatorname{erf}\left(\frac{\lambda_r\ell}{2}\right)\mathbb{P}\left(G\ge\frac{\lambda_r\ell}{2}\right)\\
&\quad-\mathbb{P}(G<0)\\
&=\operatorname{erf}\left(\frac{\lambda_c\ell}{2}\operatorname{erf}\left(\frac{\lambda_r\ell}{2}\right)\right)\operatorname{erf}\left(\frac{\lambda_r\ell}{2}\right)\left(1-\Phi\left(-\frac{\ell}{2\gamma}\right)\right)\\
&\quad-\Phi\left(-\frac{\ell}{\gamma}\right).
\end{aligned} E [ erf ( λ r λ c erf ( G ) G ) erf ( G ) ] ≥ erf ( λ r λ c erf ( 2 λ r ℓ ) 2 λ r ℓ ) erf ( 2 λ r ℓ ) P ( G ≥ 2 λ r ℓ ) − P ( G < 0 ) = erf ( 2 λ c ℓ erf ( 2 λ r ℓ ) ) erf ( 2 λ r ℓ ) ( 1 − Φ ( − 2 γ ℓ ) ) − Φ ( − γ ℓ ) .
The conclusion now follows since λ r ℓ → ∞ \lambda_r\ell\to\infty λ r ℓ → ∞ , ℓ → ∞ \ell\to\infty ℓ → ∞ , λ c ℓ → ∞ \lambda_c\ell\to\infty λ c ℓ → ∞ .
Proof of Proposition 4
The proof aims to make explicit the loss of information from X X X to ( S 1 , S 2 , … , S M ) (S_1,S_2,\ldots,S_M) ( S 1 , S 2 , … , S M ) . To this end, one has
E [ ( Y − f ( S 1 , S 2 , … , S M ) ) 2 ] ≥ E [ Var ( Y ∣ S 1 , S 2 , … , S M ) ] ≥ E [ Var ( Y ∣ S 1 , S 2 , … , S M , I 0 , J 0 ) ] . \begin{aligned}
\mathbb{E}\left[(Y-f(S_1,S_2,\ldots,S_M))^2\right]\ge\mathbb{E}\left[\operatorname{Var}(Y\mid S_1,S_2,\ldots,S_M)\right] \\
&\ge\mathbb{E}\left[\operatorname{Var}(Y\mid S_1,S_2,\ldots,S_M,I_0,J_0)\right].
\end{aligned} E [ ( Y − f ( S 1 , S 2 , … , S M ) ) 2 ] ≥ E [ Var ( Y ∣ S 1 , S 2 , … , S M ) ] ≥ E [ Var ( Y ∣ S 1 , S 2 , … , S M , I 0 , J 0 ) ] .
Now, consider i 0 ∈ [ M ] i_0\in[M] i 0 ∈ [ M ] and j 0 ∈ [ N ] j_0\in[N] j 0 ∈ [ N ] . Conditional on ( I 0 , J 0 ) = ( i 0 , j 0 ) (I_0,J_0)=(i_0,j_0) ( I 0 , J 0 ) = ( i 0 , j 0 ) , S i S_i S i is independent of ( Y , S i 0 ) (Y,S_{i_0}) ( Y , S i 0 ) , for i ≠ i 0 i\ne i_0 i = i 0 . Then, one has
Var ( Y ∣ S 1 , S 2 , … , S N , I 0 = i 0 , J 0 = j 0 ) = Var ( Y ∣ S i 0 , I 0 = i 0 , J 0 = j 0 ) . (18) \operatorname{Var}(Y\mid S_1,S_2,\ldots,S_N,I_0=i_0,J_0=j_0)=\operatorname{Var}(Y\mid S_{i_0},I_0=i_0,J_0=j_0). \tag*{(18)} Var ( Y ∣ S 1 , S 2 , … , S N , I 0 = i 0 , J 0 = j 0 ) = Var ( Y ∣ S i 0 , I 0 = i 0 , J 0 = j 0 ) . (18)
For multivariate Gaussian random variables ( U , V ) (U,V) ( U , V ) , one has
Var ( U ∣ V ) = Σ U U − Σ U V Σ V V − 1 Σ V U , (19) \operatorname{Var}(U\mid V)=\Sigma_{UU}-\Sigma_{UV}\Sigma_{VV}^{-1}\Sigma_{VU}, \tag*{(19)} Var ( U ∣ V ) = Σ U U − Σ U V Σ V V − 1 Σ V U , (19)
where Σ A B \Sigma_{AB} Σ A B stands for Cov ( A , B ) \operatorname{Cov}(A,B) Cov ( A , B ) , for random vectors A A A and B B B .
Conditioned on I 0 = i 0 I_0=i_0 I 0 = i 0 and J 0 = j 0 J_0=j_0 J 0 = j 0 , one has:
X i 0 j 0 = ℓ k ∗ + γ Z j 0 X_{i_0j_0}=\ell k^*+\gamma Z_{j_0} X i 0 j 0 = ℓ k ∗ + γ Z j 0 and X i j 0 = Z j X_{ij_0}=Z_j X i j 0 = Z j for j ∈ [ N ] ∖ { j 0 } j\in[N]\setminus\{j_0\} j ∈ [ N ] ∖ { j 0 } with standard normal random variables Z 1 , … , Z N Z_1,\ldots,Z_N Z 1 , … , Z N ;
S i 0 = β j 0 ℓ k ∗ + γ β j 0 Z j 0 + ∑ j ≠ j 0 β j Z j S_{i_0}=\beta_{j_0}\ell k^*+\gamma\beta_{j_0}Z_{j_0}+\sum_{j\ne j_0}\beta_jZ_j S i 0 = β j 0 ℓ k ∗ + γ β j 0 Z j 0 + ∑ j = j 0 β j Z j ;
Y = γ Z j 0 T v ∗ + ξ Y=\gamma Z_{j_0}^{\mathsf T}v^*+\xi Y = γ Z j 0 T v ∗ + ξ .
One then has the following:
Var ( Y ) = γ 2 + ϵ 2 \operatorname{Var}(Y)=\gamma^2+\epsilon^2 Var ( Y ) = γ 2 + ϵ 2 ;
Cov ( S i 0 , Y ) = γ 2 β j 0 v ∗ \operatorname{Cov}(S_{i_0},Y)=\gamma^2\beta_{j_0}v^* Cov ( S i 0 , Y ) = γ 2 β j 0 v ∗ ;
Cov ( S i 0 , S i 0 ) = ( γ 2 β j 0 2 + ∑ j ≠ j 0 β j 2 ) I \operatorname{Cov}(S_{i_0},S_{i_0})=\left(\gamma^2\beta_{j_0}^2+\sum_{j\ne j_0}\beta_j^2\right)I Cov ( S i 0 , S i 0 ) = ( γ 2 β j 0 2 + ∑ j = j 0 β j 2 ) I .
Introducing in (19) , one obtains
Var ( Y ∣ S i 0 , I 0 = i 0 , J 0 = j 0 ) = γ 2 + ϵ 2 − γ 4 β j 0 2 γ 2 β j 0 2 + ∑ j ≠ j 0 β j 2 ∥ v ∗ ∥ 2 = ϵ 2 + γ 2 ( 1 − γ 2 β j 0 2 γ 2 β j 0 2 + ∑ j ≠ j 0 β j 2 ) . (20) \operatorname{Var}(Y\mid S_{i_0},I_0=i_0,J_0=j_0)=\gamma^2+\epsilon^2-\frac{\gamma^4\beta_{j_0}^2}{\gamma^2\beta_{j_0}^2+\sum_{j\ne j_0}\beta_j^2}\lVert v^*\rVert^2=\epsilon^2+\gamma^2\left(1-\frac{\gamma^2\beta_{j_0}^2}{\gamma^2\beta_{j_0}^2+\sum_{j\ne j_0}\beta_j^2}\right). \tag*{(20)} Var ( Y ∣ S i 0 , I 0 = i 0 , J 0 = j 0 ) = γ 2 + ϵ 2 − γ 2 β j 0 2 + ∑ j = j 0 β j 2 γ 4 β j 0 2 ∥ v ∗ ∥ 2 = ϵ 2 + γ 2 ( 1 − γ 2 β j 0 2 + ∑ j = j 0 β j 2 γ 2 β j 0 2 ) . (20)
Finally,
E [ Var ( Y ∣ S 1 , S 2 , … , S M , I 0 , J 0 ) ] = ∑ j 0 = 1 N ∑ i 0 = 1 M P ( I 0 = i 0 ∣ J 0 = j 0 ) p j 0 Var ( Y ∣ S i 0 , I 0 = i 0 , J 0 = j 0 ) = ∑ j 0 = 1 N p j 0 ( ϵ 2 + γ 2 ( 1 − γ 2 β j 0 2 γ 2 β j 0 2 + ∑ j ≠ j 0 β j 2 ) ) = ϵ 2 + γ 2 ( 1 − ∑ j 0 = 1 N p j 0 γ 2 β j 0 2 γ 2 β j 0 2 + ∑ j ≠ j 0 β j 2 ) ≥ ϵ 2 , \begin{aligned}
\mathbb{E}\left[\operatorname{Var}\left(Y \mid S_1,S_2,\ldots,S_M,I_0,J_0\right)\right]
&= \sum_{j_0=1}^{N}\sum_{i_0=1}^{M}\mathbb{P}(I_0=i_0\mid J_0=j_0)p_{j_0}\operatorname{Var}\left(Y\mid S_{i_0},I_0=i_0,J_0=j_0\right) \\
&= \sum_{j_0=1}^{N}p_{j_0}\left(\epsilon^2+\gamma^2\left(1-\frac{\gamma^2\beta_{j_0}^2}{\gamma^2\beta_{j_0}^2+\sum_{j\ne j_0}\beta_j^2}\right)\right) \\
&= \epsilon^2+\gamma^2\left(1-\sum_{j_0=1}^{N}p_{j_0}\frac{\gamma^2\beta_{j_0}^2}{\gamma^2\beta_{j_0}^2+\sum_{j\ne j_0}\beta_j^2}\right) \\
&\ge\epsilon^2,
\end{aligned} E [ Var ( Y ∣ S 1 , S 2 , … , S M , I 0 , J 0 ) ] = j 0 = 1 ∑ N i 0 = 1 ∑ M P ( I 0 = i 0 ∣ J 0 = j 0 ) p j 0 Var ( Y ∣ S i 0 , I 0 = i 0 , J 0 = j 0 ) = j 0 = 1 ∑ N p j 0 ( ϵ 2 + γ 2 ( 1 − γ 2 β j 0 2 + ∑ j = j 0 β j 2 γ 2 β j 0 2 ) ) = ϵ 2 + γ 2 ( 1 − j 0 = 1 ∑ N p j 0 γ 2 β j 0 2 + ∑ j = j 0 β j 2 γ 2 β j 0 2 ) ≥ ϵ 2 ,
with equality in the last step only if there exists j 0 j_0 j 0 such that p j 0 = 1 p_{j_0}=1 p j 0 = 1 and β j = 0 \beta_j=0 β j = 0 for j ≠ j 0 j\ne j_0 j = j 0 .
Proof of Theorem 5
In this section, we establish the global convergence of the Projected Gradient Descent (PGD) dynamics to the true oracle parameters. We follow the proof of Theorem 5 in Marion et al. [2025] as closely as possible while adapting it to our model T λ r , λ c ( k , v ) T_{\lambda_r,\lambda_c}^{(k,v)} T λ r , λ c ( k , v ) . The only result which requires a completely different and much more involved proof is that of Proposition 9. Throughout these proofs, C > 0 C>0 C > 0 denotes a generic constant that may vary from line to line but is strictly independent of the step size α \alpha α and the iteration step t t t . Moreover, we shall drop the subscript λ r , λ c \lambda_r,\lambda_c λ r , λ c and assume an implicit dependence on the inverse temperatures.
The theoretical guarantees of our optimization trajectory are established through the following five primary results. We first establish the invariance of the manifold to PGD steps.
Lemma 6. The manifold M \mathcal{M} M is invariant under the PGD dynamics (Algorithm 1), in the sense that if ( k t , v t ) ∈ M (k_t,v_t)\in\mathcal{M} ( k t , v t ) ∈ M , then ( k t + 1 , v t + 1 ) ∈ M (k_{t+1},v_{t+1})\in\mathcal{M} ( k t + 1 , v t + 1 ) ∈ M .
By restricting our analysis to the manifold M \mathcal{M} M , we can project the high-dimensional dynamics down to the scalar alignment variables κ t = k t T k ∗ \kappa_t=k_t^{\mathsf{T}}k^* κ t = k t T k ∗ and ν t = v t T v ∗ \nu_t=v_t^{\mathsf{T}}v^* ν t = v t T v ∗ . We then map the PGD updates to an autonomous dynamical system:
Lemma 7. When initialized on the manifold M \mathcal{M} M , the PGD iterations (Algorithm 1) can be reformulated as the autonomous discrete dynamical system ( κ t + 1 , ν t + 1 ) = g ( κ t , ν t ) (\kappa_{t+1},\nu_{t+1})=g(\kappa_t,\nu_t) ( κ t + 1 , ν t + 1 ) = g ( κ t , ν t ) , where g : [ − 1 , 1 ] 2 → [ − 1 , 1 ] 2 g\colon[-1,1]^2\to[-1,1]^2 g : [ − 1 , 1 ] 2 → [ − 1 , 1 ] 2 is given by
g ( κ , ν ) = ( κ − α ( ∂ κ R < ( κ , ν ) ) ( 1 − κ 2 ) 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) , ν − α ( ∂ ν R < ( κ , ν ) ) ( 1 − ν 2 ) 1 + α 2 ( ∂ ν R < ( κ , ν ) ) 2 ( 1 − ν 2 ) ) . (21) g(\kappa,\nu)=\left(\frac{\kappa-\alpha(\partial_\kappa R^{<}(\kappa,\nu))(1-\kappa^2)}{\sqrt{1+\alpha^2(\partial_\kappa R^{<}(\kappa,\nu))^2(1-\kappa^2)}},\frac{\nu-\alpha(\partial_\nu R^{<}(\kappa,\nu))(1-\nu^2)}{\sqrt{1+\alpha^2(\partial_\nu R^{<}(\kappa,\nu))^2(1-\nu^2)}}\right). \tag*{(21)} g ( κ , ν ) = ( 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) κ − α ( ∂ κ R < ( κ , ν )) ( 1 − κ 2 ) , 1 + α 2 ( ∂ ν R < ( κ , ν ) ) 2 ( 1 − ν 2 ) ν − α ( ∂ ν R < ( κ , ν )) ( 1 − ν 2 ) ) . (21)
With the dynamics formalized, we establish that the risk strictly decreases, forcing convergence to the stationary points:
Proposition 8. For a sufficiently small step size α \alpha α , the risk R < R^{<} R < is strictly decreasing along the PGD iterates unless it is at a fixed point. Furthermore, if ( κ , ν ) (\kappa,\nu) ( κ , ν ) is an accumulation point of the sequence ( κ t , ν t ) t ≥ 0 (\kappa_t,\nu_t)_{t\ge0} ( κ t , ν t ) t ≥ 0 , it must satisfy:
( 1 − κ 2 ) ∂ κ R < ( κ , ν ) = 0 and ( 1 − ν 2 ) ∂ ν R < ( κ , ν ) = 0. (22) (1-\kappa^2)\partial_\kappa R^{<}(\kappa,\nu)=0 \quad\text{and}\quad(1-\nu^2)\partial_\nu R^{<}(\kappa,\nu)=0. \tag*{(22)} ( 1 − κ 2 ) ∂ κ R < ( κ , ν ) = 0 and ( 1 − ν 2 ) ∂ ν R < ( κ , ν ) = 0. (22)
We then rigorously classify the geometry of these accumulation points, isolating the oracle states as the unique global minima:
Proposition 9. For λ c \lambda_c λ c and λ r \lambda_r λ r small enough, the fixed points of the dynamics, given by (22) , on [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 are ( ± 1 , ± 1 ) (\pm1,\pm1) ( ± 1 , ± 1 ) and ( 0 , 0 ) (0,0) ( 0 , 0 ) . Specifically:
( ± 1 , ∓ 1 ) (\pm1,\mp1) ( ± 1 , ∓ 1 ) are global maxima.
( ± 1 , ± 1 ) (\pm1,\pm1) ( ± 1 , ± 1 ) are global minima.
( 0 , 0 ) (0,0) ( 0 , 0 ) is a saddle point.
Finally, we characterize the local geometry around the saddle point to guarantee that random initializations escape it:
Proposition 10. For a sufficiently small step size α \alpha α , the mapping g g g is a local diffeomorphism around ( 0 , 0 ) (0,0) ( 0 , 0 ) , whose Jacobian matrix has one eigenvalue in ( 0 , 1 ) (0,1) ( 0 , 1 ) and one in ( 1 , ∞ ) (1,\infty) ( 1 , ∞ ) . Furthermore, g g g is injective on [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 .
The global convergence to the oracle states then follows directly from the exclusion of the saddle point and maxima. The proofs for all stated results are provided in the subsections below.
Proof of Lemma 6
To prove that the manifold M \mathcal{M} M is invariant under the Projected Gradient Descent (PGD) dynamics, we must first establish that the gradient of the risk does not "push" the parameters outside of M \mathcal{M} M . We do this by proving an auxiliary symmetry result regarding the cross-term derivatives.
Lemma 11. At any point ( κ , ν , θ , η , ρ ) (\kappa,\nu,\theta,\eta,\rho) ( κ , ν , θ , η , ρ ) on the manifold M \mathcal{M} M (i.e., where θ = η = ρ = 0 \theta=\eta=\rho=0 θ = η = ρ = 0 ), the partial derivatives of the risk with respect to the cross-terms vanish:
∂ θ R < = ∂ η R < = ∂ ρ R < = 0. \partial_{\theta}\mathcal{R}^{<}=\partial_{\eta}\mathcal{R}^{<}=\partial_{\rho}\mathcal{R}^{<}=0. ∂ θ R < = ∂ η R < = ∂ ρ R < = 0.
The proof follows by recalling that the risk can be reduced to a function R < \mathcal{R}^{<} R < depending only on the covariance structure of the Gaussian projections G i j k = X i j T k G_{ij}^{k}=X_{ij}^{T}k G ij k = X ij T k , G i j v = X i j T v G_{ij}^{v}=X_{ij}^{T}v G ij v = X ij T v , and G i j v ∗ = X i j T v ∗ G_{ij}^{v^*}=X_{ij}^{T}v^* G ij v ∗ = X ij T v ∗ . The relevant covariance matrix is:
( G i j k G i j v G i j v ∗ ) ∼ N ( 0 , ( 1 ρ η ρ 1 ν η ν 1 ) ) \begin{pmatrix}
G_{ij}^{k}\\
G_{ij}^{v}\\
G_{ij}^{v^*}
\end{pmatrix}
\sim\mathcal{N}\left(0,
\begin{pmatrix}
1 & \rho& \eta\\
\rho& 1 & \nu\\
\eta& \nu& 1
\end{pmatrix}\right) G ij k G ij v G ij v ∗ ∼ N 0 , 1 ρ η ρ 1 ν η ν 1
and the alignment between v v v and k ∗ k^* k ∗ is given by θ = v T k ∗ \theta=v^{T}k^* θ = v T k ∗ .
It is clear from Equation (11) that the risk is invariant to the sign inversion of ( v , v ∗ ) (v,v^*) ( v , v ∗ ) to ( − v , − v ∗ ) (-v,-v^*) ( − v , − v ∗ ) . This amounts to
R < ( κ , ν , θ , η , ρ ) = R < ( κ , ν , − θ , − η , − ρ ) . \mathcal{R}^{<}(\kappa,\nu,\theta,\eta,\rho)=\mathcal{R}^{<}(\kappa,\nu,-\theta,-\eta,-\rho). R < ( κ , ν , θ , η , ρ ) = R < ( κ , ν , − θ , − η , − ρ ) .
Taking the partial derivative with respect to θ \theta θ now yields the following result:
∂ θ R < ( κ , ν , θ , η , ρ ) = − ∂ θ R < ( κ , ν , − θ , − η , − ρ ) . \partial_{\theta}\mathcal{R}^{<}(\kappa,\nu,\theta,\eta,\rho)=-\partial_{\theta}\mathcal{R}^{<}(\kappa,\nu,-\theta,-\eta,-\rho). ∂ θ R < ( κ , ν , θ , η , ρ ) = − ∂ θ R < ( κ , ν , − θ , − η , − ρ ) .
Evaluating this at the manifold M \mathcal{M} M where θ = η = ρ = 0 \theta=\eta=\rho=0 θ = η = ρ = 0 , we immediately obtain ∂ θ R < ( κ , ν , 0 , 0 , 0 ) = 0 \partial_{\theta}\mathcal{R}^{<}(\kappa,\nu,0,0,0)=0 ∂ θ R < ( κ , ν , 0 , 0 , 0 ) = 0 . Identical arguments hold for ∂ η R < \partial_{\eta}\mathcal{R}^{<} ∂ η R < and ∂ ρ R < \partial_{\rho}\mathcal{R}^{<} ∂ ρ R < , concluding the proof of Lemma 11.
We now return to the proof of Lemma 6. By the chain rule, the Euclidean gradients of the risk with respect to k k k and v v v can be expanded in the basis of the parameter vectors:
∇ k R = ( ∂ κ R < ) k ∗ + ( ∂ η R < ) v ∗ + ( ∂ ρ R < ) v \nabla_k\mathcal{R}=(\partial_{\kappa}\mathcal{R}^{<})k^*+(\partial_{\eta}\mathcal{R}^{<})v^*+(\partial_{\rho}\mathcal{R}^{<})v ∇ k R = ( ∂ κ R < ) k ∗ + ( ∂ η R < ) v ∗ + ( ∂ ρ R < ) v
∇ v R = ( ∂ ν R < ) v ∗ + ( ∂ θ R < ) k ∗ + ( ∂ ρ R < ) k \nabla_v\mathcal{R}=(\partial_{\nu}\mathcal{R}^{<})v^*+(\partial_{\theta}\mathcal{R}^{<})k^*+(\partial_{\rho}\mathcal{R}^{<})k ∇ v R = ( ∂ ν R < ) v ∗ + ( ∂ θ R < ) k ∗ + ( ∂ ρ R < ) k
Applying Lemma 11, on the manifold M \mathcal{M} M (where θ = η = ρ = 0 \theta=\eta=\rho=0 θ = η = ρ = 0 ), the gradients simplify strictly to:
∇ k R ( k t , v t ) = ∂ κ R < ( κ t , ν t ) k ∗ ∇ v R ( k t , v t ) = ∂ ν R < ( κ t , ν t ) v ∗ (23) \begin{aligned}
\nabla_k\mathcal{R}(k_t,v_t)&=\partial_{\kappa}\mathcal{R}^{<}(\kappa_t,\nu_t)k^*\\
\nabla_v\mathcal{R}(k_t,v_t)&=\partial_{\nu}\mathcal{R}^{<}(\kappa_t,\nu_t)v^* \tag*{(23)}
\end{aligned} ∇ k R ( k t , v t ) ∇ v R ( k t , v t ) = ∂ κ R < ( κ t , ν t ) k ∗ = ∂ ν R < ( κ t , ν t ) v ∗ (23)
Recall the PGD updates for ( k t + 1 , v t + 1 ) (k_{t+1},v_{t+1}) ( k t + 1 , v t + 1 ) :
k t + 1 = k t − α ( I d − k t k t T ) ∇ k R ∥ k t − α ( I d − k t k t T ) ∇ k R ∥ 2 ≔ k ~ t + 1 c k k_{t+1}=\frac{k_t-\alpha(I_d-k_tk_t^T)\nabla_k\mathcal{R}}{\lVert k_t-\alpha(I_d-k_tk_t^T)\nabla_k\mathcal{R}\rVert_2}\coloneqq\frac{\widetilde{k}_{t+1}}{c_k} k t + 1 = ∥ k t − α ( I d − k t k t T ) ∇ k R ∥ 2 k t − α ( I d − k t k t T ) ∇ k R : = c k k t + 1
v t + 1 = v t − α ( I d − v t v t T ) ∇ v R ∥ v t − α ( I d − v t v t T ) ∇ v R ∥ 2 ≔ v ~ t + 1 c v v_{t+1}=\frac{v_t-\alpha(I_d-v_tv_t^T)\nabla_v\mathcal{R}}{\lVert v_t-\alpha(I_d-v_tv_t^T)\nabla_v\mathcal{R}\rVert_2}\coloneqq\frac{\widetilde{v}_{t+1}}{c_v} v t + 1 = ∥ v t − α ( I d − v t v t T ) ∇ v R ∥ 2 v t − α ( I d − v t v t T ) ∇ v R : = c v v t + 1
where c k , c v > 0 c_k,c_v>0 c k , c v > 0 are the respective normalization constants. To prove ( k t + 1 , v t + 1 ) ∈ M (k_{t+1},v_{t+1})\in\mathcal{M} ( k t + 1 , v t + 1 ) ∈ M , we must verify the three orthogonality conditions: ( v ∗ ) T k t + 1 = 0 (v^*)^Tk_{t+1}=0 ( v ∗ ) T k t + 1 = 0 , ( k ∗ ) T v t + 1 = 0 (k^*)^Tv_{t+1}=0 ( k ∗ ) T v t + 1 = 0 , and $v_{t+1}^{T}k_{t+1}=0. Condition 1: ( v ∗ ) T k t + 1 = 0 (v^*)^{T}k_{t+1}=0 ( v ∗ ) T k t + 1 = 0 . Using the simplified gradient (23) , we project k ~ t + 1 \tilde{k}_{t+1} k ~ t + 1 onto v ∗ v^* v ∗ :
( v ∗ ) T k ~ t + 1 = ( v ∗ ) T k t − α ∂ k R < ( ( v ∗ ) T k ∗ − ( v ∗ ) T k t ( k t T k ∗ ) ) . (v^*)^{T}\tilde{k}_{t+1}=(v^*)^{T}k_t-\alpha\partial_k\mathcal{R}^{<}\left((v^*)^{T}k^*-(v^*)^{T}k_t(k_t^{T}k^*)\right). ( v ∗ ) T k ~ t + 1 = ( v ∗ ) T k t − α ∂ k R < ( ( v ∗ ) T k ∗ − ( v ∗ ) T k t ( k t T k ∗ ) ) .
Because ( k t , v t ) ∈ M (k_t,v_t)\in\mathcal{M} ( k t , v t ) ∈ M , we have ( v ∗ ) T k t = 0 (v^*)^{T}k_t=0 ( v ∗ ) T k t = 0 . By Assumption 1, the oracle parameters are orthogonal, so ( v ∗ ) T k ∗ = 0 (v^*)^{T}k^*=0 ( v ∗ ) T k ∗ = 0 . Thus:
( v ∗ ) T k ~ t + 1 = 0 − α ∂ k R < ( 0 − 0 ) = 0 ⟹ ( v ∗ ) T k t + 1 = 0. (v^*)^{T}\tilde{k}_{t+1}=0-\alpha\partial_k\mathcal{R}^{<}(0-0)=0\Longrightarrow(v^*)^{T}k_{t+1}=0. ( v ∗ ) T k ~ t + 1 = 0 − α ∂ k R < ( 0 − 0 ) = 0 ⟹ ( v ∗ ) T k t + 1 = 0.
Condition 2: ( k ∗ ) T v t + 1 = 0 (k^*)^{T}v_{t+1}=0 ( k ∗ ) T v t + 1 = 0 . By an exactly symmetric argument using v ~ t + 1 \tilde{v}_{t+1} v ~ t + 1 and the fact that ( k ∗ ) T v t = 0 (k^*)^{T}v_t=0 ( k ∗ ) T v t = 0 , we obtain ( k ∗ ) T v t + 1 = 0 (k^*)^{T}v_{t+1}=0 ( k ∗ ) T v t + 1 = 0 .
Condition 3: v t + 1 T k t + 1 = 0 v_{t+1}^{T}k_{t+1}=0 v t + 1 T k t + 1 = 0 . We compute the inner product of the unnormalized updates v ~ t + 1 T k ~ t + 1 \tilde{v}_{t+1}^{T}\tilde{k}_{t+1} v ~ t + 1 T k ~ t + 1 :
v ~ t + 1 T k ~ t + 1 = ( v t − α ∂ ν R < ( v ∗ − ν t v t ) ) T ( k t − α ∂ k R < ( k ∗ − κ t k t ) ) = v t T k t − α ∂ k R < ( v t T k ∗ − κ t v t T k t ) − α ∂ ν R < ( ( v ∗ ) T k t − ν t v t T k t ) + α 2 ∂ k R < ∂ ν R < ( ( v ∗ − ν t v t ) T ( k ∗ − κ t k t ) ) . \begin{aligned}
\tilde{v}_{t+1}^{T}\tilde{k}_{t+1}
&=(v_t-\alpha\partial_\nu\mathcal{R}^{<}(v^*-\nu_tv_t))^{T}(k_t-\alpha\partial_k\mathcal{R}^{<}(k^*-\kappa_tk_t))\\
&=v_t^{T}k_t-\alpha\partial_k\mathcal{R}^{<}(v_t^{T}k^*-\kappa_tv_t^{T}k_t)-\alpha\partial_\nu\mathcal{R}^{<}((v^*)^{T}k_t-\nu_tv_t^{T}k_t)\\
&\quad+\alpha^2\partial_k\mathcal{R}^{<}\partial_\nu\mathcal{R}^{<}((v^*-\nu_tv_t)^{T}(k^*-\kappa_tk_t)).
\end{aligned} v ~ t + 1 T k ~ t + 1 = ( v t − α ∂ ν R < ( v ∗ − ν t v t ) ) T ( k t − α ∂ k R < ( k ∗ − κ t k t )) = v t T k t − α ∂ k R < ( v t T k ∗ − κ t v t T k t ) − α ∂ ν R < (( v ∗ ) T k t − ν t v t T k t ) + α 2 ∂ k R < ∂ ν R < (( v ∗ − ν t v t ) T ( k ∗ − κ t k t )) .
Since ( k t , v t ) ∈ M (k_t,v_t)\in\mathcal{M} ( k t , v t ) ∈ M , we already know v t T k t = 0 v_t^{T}k_t=0 v t T k t = 0 , v t T k ∗ = 0 v_t^{T}k^*=0 v t T k ∗ = 0 , and ( v ∗ ) T k t = 0 (v^*)^{T}k_t=0 ( v ∗ ) T k t = 0 . Thus, all the linear terms vanish. For the quadratic term, we expand:
( v ∗ − ν t v t ) T ( k ∗ − κ t k t ) = ( v ∗ ) T k ∗ − κ t ( v ∗ ) T k t − ν t v t T k ∗ + ν t κ t v t T k t . (v^*-\nu_tv_t)^{T}(k^*-\kappa_tk_t)=(v^*)^{T}k^*-\kappa_t(v^*)^{T}k_t-\nu_tv_t^{T}k^*+\nu_t\kappa_tv_t^{T}k_t. ( v ∗ − ν t v t ) T ( k ∗ − κ t k t ) = ( v ∗ ) T k ∗ − κ t ( v ∗ ) T k t − ν t v t T k ∗ + ν t κ t v t T k t .
Substituting the orthogonalities ( v ∗ ) T k ∗ = 0 (v^*)^{T}k^*=0 ( v ∗ ) T k ∗ = 0 , ( v ∗ ) T k t = 0 (v^*)^{T}k_t=0 ( v ∗ ) T k t = 0 , v t T k ∗ = 0 v_t^{T}k^*=0 v t T k ∗ = 0 , and v t T k t = 0 v_t^{T}k_t=0 v t T k t = 0 yields the desired orthogonality.
Therefore, v ~ t + 1 T k ~ t + 1 = 0 \tilde{v}_{t+1}^{T}\tilde{k}_{t+1}=0 v ~ t + 1 T k ~ t + 1 = 0 , meaning v t + 1 T k t + 1 = 0 v_{t+1}^{T}k_{t+1}=0 v t + 1 T k t + 1 = 0 . Having verified all three conditions, we conclude that ( k t + 1 , v t + 1 ) ∈ M (k_{t+1},v_{t+1})\in\mathcal{M} ( k t + 1 , v t + 1 ) ∈ M . q.e.d.
Proof of Lemma 7
Using the simplified gradient (23) valid on the manifold M \mathcal{M} M , we compute the unnormalized update for k k k :
k ~ t + 1 = k t − α ( I d − k t k t T ) ( ∂ k R < ) k ∗ = k t − α ( ∂ k R < ) ( k ∗ − κ t k t ) . \tilde{k}_{t+1}=k_t-\alpha(I_d-k_tk_t^{T})(\partial_k\mathcal{R}^{<})k^*=k_t-\alpha(\partial_k\mathcal{R}^{<})(k^*-\kappa_tk_t). k ~ t + 1 = k t − α ( I d − k t k t T ) ( ∂ k R < ) k ∗ = k t − α ( ∂ k R < ) ( k ∗ − κ t k t ) .
Projecting this unnormalized update onto k ∗ k^* k ∗ yields the numerator for κ t + 1 \kappa_{t+1} κ t + 1 :
( k ∗ ) T k ~ t + 1 = ( k ∗ ) T k t − α ( ∂ k R < ) ( ( k ∗ ) T k ∗ − κ t ( k ∗ ) T k t ) = κ t − α ( ∂ k R < ) ( 1 − κ t 2 ) . (k^*)^{T}\tilde{k}_{t+1}=(k^*)^{T}k_t-\alpha(\partial_k\mathcal{R}^{<})((k^*)^{T}k^*-\kappa_t(k^*)^{T}k_t)=\kappa_t-\alpha(\partial_k\mathcal{R}^{<})(1-\kappa_t^2). ( k ∗ ) T k ~ t + 1 = ( k ∗ ) T k t − α ( ∂ k R < ) (( k ∗ ) T k ∗ − κ t ( k ∗ ) T k t ) = κ t − α ( ∂ k R < ) ( 1 − κ t 2 ) .
To find the normalization constant c k = ∥ k ~ t + 1 ∥ 2 c_k=\|\tilde{k}_{t+1}\|_2 c k = ∥ k ~ t + 1 ∥ 2 , we apply the Pythagorean theorem. Since ( k ∗ − κ t k t ) (k^*-\kappa_tk_t) ( k ∗ − κ t k t ) is the projection of k ∗ k^* k ∗ onto the orthogonal complement of k t k_t k t , it is strictly orthogonal to k t k_t k t . Thus:
∥ k ~ t + 1 ∥ 2 2 = ∥ k t ∥ 2 2 + α 2 ( ∂ k R < ) 2 ∥ k ∗ − κ t k t ∥ 2 2 = 1 + α 2 ( ∂ k R < ) 2 ( 1 − κ t 2 ) . \begin{aligned}
\|\tilde{k}_{t+1}\|_2^2
&=\|k_t\|_2^2+\alpha^2(\partial_k\mathcal{R}^{<})^2\|k^*-\kappa_tk_t\|_2^2\\
&=1+\alpha^2(\partial_k\mathcal{R}^{<})^2(1-\kappa_t^2).
\end{aligned} ∥ k ~ t + 1 ∥ 2 2 = ∥ k t ∥ 2 2 + α 2 ( ∂ k R < ) 2 ∥ k ∗ − κ t k t ∥ 2 2 = 1 + α 2 ( ∂ k R < ) 2 ( 1 − κ t 2 ) .
Dividing the numerator by ∥ k ~ t + 1 ∥ 2 \|\tilde{k}_{t+1}\|_2 ∥ k ~ t + 1 ∥ 2 directly yields g 1 ( κ t , ν t ) g_1(\kappa_t,\nu_t) g 1 ( κ t , ν t ) . Analogously, one obtains the result for g 2 ( κ t , ν t ) g_2(\kappa_t,\nu_t) g 2 ( κ t , ν t ) .
Proof of Proposition 8
The theoretical risk R < \mathcal{R}^{<} R < is composed of smooth error functions evaluated over Gaussian expectations, ensuring it is C ∞ C^\infty C ∞ on the compact square [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 . Consequently, it is Λ \Lambda Λ -smooth, meaning we can bound its variation:
R < ( κ t + 1 , ν t + 1 ) − R < ( κ t , ν t ) ≤ ( ∂ κ R < ) ( κ t + 1 − κ t ) + ( ∂ ν R < ) ( ν t + 1 − ν t ) + Λ 2 [ ( κ t + 1 − κ t ) 2 + ( ν t + 1 − ν t ) 2 ] . (24) \mathcal{R}^{<}(\kappa_{t+1},\nu_{t+1})-\mathcal{R}^{<}(\kappa_t,\nu_t)\le(\partial_\kappa\mathcal{R}^{<})(\kappa_{t+1}-\kappa_t)+(\partial_\nu\mathcal{R}^{<})(\nu_{t+1}-\nu_t)+\frac{\Lambda}{2}\left[(\kappa_{t+1}-\kappa_t)^2+(\nu_{t+1}-\nu_t)^2\right]. \tag*{(24)} R < ( κ t + 1 , ν t + 1 ) − R < ( κ t , ν t ) ≤ ( ∂ κ R < ) ( κ t + 1 − κ t ) + ( ∂ ν R < ) ( ν t + 1 − ν t ) + 2 Λ [ ( κ t + 1 − κ t ) 2 + ( ν t + 1 − ν t ) 2 ] . (24)
Using the discrete mapping g g g from Lemma 7, we rewrite the step difference for κ \kappa κ :
κ t + 1 − κ t = κ t ( 1 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − 1 ) − α ( ∂ κ R < ) ( 1 − κ t 2 ) 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) \kappa_{t+1}-\kappa_t=\kappa_t\left(\frac{1}{\sqrt{1+\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)}}-1\right)-\frac{\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)}{\sqrt{1+\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)}} κ t + 1 − κ t = κ t ( 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) 1 − 1 ) − 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) α ( ∂ κ R < ) ( 1 − κ t 2 )
= − α ( ∂ κ R < ) ( 1 − κ t 2 ) + ( 1 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − 1 ) ( κ t − α ( ∂ κ R < ) ( 1 − κ t 2 ) ) . (25) \begin{aligned}
&=-\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\\
&\quad+\left(\frac{1}{\sqrt{1+\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)}}-1\right)\left(\kappa_t-\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right). \tag*{(25)}
\end{aligned} = − α ( ∂ κ R < ) ( 1 − κ t 2 ) + ( 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) 1 − 1 ) ( κ t − α ( ∂ κ R < ) ( 1 − κ t 2 ) ) . (25)
As a consequence,
∣ κ t + 1 − κ t + α ( ∂ κ R < ) ( 1 − κ t 2 ) ∣ ≤ ∣ 1 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − 1 ∣ ⋅ ∣ κ t − α ( ∂ κ R < ) ( 1 − κ t 2 ) ∣ ≤ α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) ∣ κ t − α ( ∂ κ R < ) ( 1 − κ t 2 ) ∣ ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . (26) \begin{aligned}
\left|\kappa_{t+1}-\kappa_t+\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right|
&\le\left|\frac{1}{\sqrt{1+\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)}}-1\right|\\
&\qquad\cdot\left|\kappa_t-\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right|\\
&\le\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)\left|\kappa_t-\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right|\\
&\le C\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2). \tag*{(26)}
\end{aligned} κ t + 1 − κ t + α ( ∂ κ R < ) ( 1 − κ t 2 ) ≤ 1 + α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) 1 − 1 ⋅ κ t − α ( ∂ κ R < ) ( 1 − κ t 2 ) ≤ α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) κ t − α ( ∂ κ R < ) ( 1 − κ t 2 ) ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . (26)
where the second inequality holds by Lemma 14 and the last bound holds since the function ( κ , ν ) ↦ ∣ κ − α ( ∂ κ R < ( κ , ν ) ) ( 1 − κ 2 ) ∣ (\kappa,\nu)\mapsto\left|\kappa-\alpha(\partial_\kappa\mathcal{R}^{<}(\kappa,\nu))(1-\kappa^2)\right| ( κ , ν ) ↦ κ − α ( ∂ κ R < ( κ , ν )) ( 1 − κ 2 ) is uniformly bounded for all α ≤ 1 \alpha\le1 α ≤ 1 . This bound has two implications. First,
( ∂ κ R < ) ( κ t + 1 − κ t ) + α ( ∂ κ R < ) 2 ( 1 − κ t 2 ) = ( ∂ κ R < ) ( ( κ t + 1 − κ t ) + α ( ∂ κ R < ) ( 1 − κ t 2 ) ) ≤ ∣ ( ∂ κ R < ) ∣ ∣ κ t + 1 − κ t + α ( ∂ κ R < ) ( 1 − κ t 2 ) ∣ ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . (27) \begin{aligned}
(\partial_\kappa\mathcal{R}^{<})(\kappa_{t+1}-\kappa_t)+\alpha(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)
&=(\partial_\kappa\mathcal{R}^{<})\left((\kappa_{t+1}-\kappa_t)+\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right)\\
&\le|(\partial_\kappa\mathcal{R}^{<})|\left|\kappa_{t+1}-\kappa_t+\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right|\\
&\le C\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2). \tag*{(27)}
\end{aligned} ( ∂ κ R < ) ( κ t + 1 − κ t ) + α ( ∂ κ R < ) 2 ( 1 − κ t 2 ) = ( ∂ κ R < ) ( ( κ t + 1 − κ t ) + α ( ∂ κ R < ) ( 1 − κ t 2 ) ) ≤ ∣ ( ∂ κ R < ) ∣ κ t + 1 − κ t + α ( ∂ κ R < ) ( 1 − κ t 2 ) ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . (27)
where we use the fact that ∣ ∂ κ R < ∣ |\partial_\kappa\mathcal{R}^{<}| ∣ ∂ κ R < ∣ is bounded and the bound (26) . Second, since the square function is Lipschitz on compact sets, we have
∣ ( κ t + 1 − κ t ) 2 − ( α ( ∂ κ R < ) ( 1 − κ t 2 ) ) 2 ∣ ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . \left|(\kappa_{t+1}-\kappa_t)^2-\left(\alpha(\partial_\kappa\mathcal{R}^{<})(1-\kappa_t^2)\right)^2\right|\le C\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2). ( κ t + 1 − κ t ) 2 − ( α ( ∂ κ R < ) ( 1 − κ t 2 ) ) 2 ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) .
Thus,
( κ t + 1 − κ t ) 2 ≤ α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) 2 + C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . (28) \begin{aligned}
(\kappa_{t+1}-\kappa_t)^2
&\le\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)^2+C\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)\\
&\le C\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2). \tag*{(28)}
\end{aligned} ( κ t + 1 − κ t ) 2 ≤ α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) 2 + C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) ≤ C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) . (28)
We obtain analogous bounds for ν \nu ν , namely
( ∂ ν R < ) ( ν t + 1 − ν t ) + α ( ∂ ν R < ) 2 ( 1 − ν t 2 ) ≤ C α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) (29) (\partial_\nu\mathcal{R}^{<})(\nu_{t+1}-\nu_t)+\alpha(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2)\le C\alpha^2(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2) \tag*{(29)} ( ∂ ν R < ) ( ν t + 1 − ν t ) + α ( ∂ ν R < ) 2 ( 1 − ν t 2 ) ≤ C α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) (29)
and
( ν t + 1 − ν t ) 2 ≤ C α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) . (30) (\nu_{t+1}-\nu_t)^2\le C\alpha^2(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2). \tag*{(30)} ( ν t + 1 − ν t ) 2 ≤ C α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) . (30)
Plugging the bounds (27) , (28) , (29) , (30) into Equation (24) , we obtain the desired inequality
R < ( κ t + 1 , ν t + 1 ) − R < ( κ t , ν t ) ≤ − α ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − α ( ∂ ν R < ) 2 ( 1 − ν t 2 ) + C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) + C α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) . \mathcal{R}^{<}(\kappa_{t+1},\nu_{t+1})-\mathcal{R}^{<}(\kappa_t,\nu_t)
\le-\alpha(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)-\alpha(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2)+C\alpha^2(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)+C\alpha^2(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2). R < ( κ t + 1 , ν t + 1 ) − R < ( κ t , ν t ) ≤ − α ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − α ( ∂ ν R < ) 2 ( 1 − ν t 2 ) + C α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) + C α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) .
By choosing the step size α ≤ 1 2 C \alpha\le\frac{1}{2C} α ≤ 2 C 1 , this ensures that
R < ( κ t + 1 , ν t + 1 ) − R < ( κ t , ν t ) ≤ − α 2 ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − α 2 ( ∂ ν R < ) 2 ( 1 − ν t 2 ) . \mathcal{R}^{<}(\kappa_{t+1},\nu_{t+1})-\mathcal{R}^{<}(\kappa_t,\nu_t)
\le-\frac{\alpha}{2}(\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)-\frac{\alpha}{2}(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2). R < ( κ t + 1 , ν t + 1 ) − R < ( κ t , ν t ) ≤ − 2 α ( ∂ κ R < ) 2 ( 1 − κ t 2 ) − 2 α ( ∂ ν R < ) 2 ( 1 − ν t 2 ) .
This shows that the risk is decreasing along the PGD iterates. Next, introducing R min < = min ( κ , ν ) ∈ [ − 1 , 1 ] 2 R < ( κ , ν ) \mathcal{R}^{<}_{\min}=\min_{(\kappa,\nu)\in[-1,1]^2}\mathcal{R}^{<}(\kappa,\nu) R m i n < = min ( κ , ν ) ∈ [ − 1 , 1 ] 2 R < ( κ , ν ) and using a telescopic sum, we have, for all T ≥ 0 T\ge0 T ≥ 0 ,
R < ( κ 0 , ν 0 ) − R min < ≥ R < ( κ 0 , ν 0 ) − R < ( κ T , ν T ) ≥ α 2 ∑ t = 0 T − 1 ( ( ∂ κ R < ) 2 ( 1 − κ t 2 ) + ( ∂ ν R < ) 2 ( 1 − ν t 2 ) ) . \begin{aligned}
\mathcal{R}^{<}(\kappa_0,\nu_0)-\mathcal{R}^{<}_{\min}\ge\mathcal{R}^{<}(\kappa_0,\nu_0)-\mathcal{R}^{<}(\kappa_T,\nu_T) \\
&\ge\frac{\alpha}{2}\sum_{t=0}^{T-1}\left((\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)+(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2)\right).
\end{aligned} R < ( κ 0 , ν 0 ) − R m i n < ≥ R < ( κ 0 , ν 0 ) − R < ( κ T , ν T ) ≥ 2 α t = 0 ∑ T − 1 ( ( ∂ κ R < ) 2 ( 1 − κ t 2 ) + ( ∂ ν R < ) 2 ( 1 − ν t 2 ) ) .
Since the left-hand side is finite, and the terms of the sum are nonnegative, we conclude that the series converges as T → ∞ T\to\infty T → ∞ . In particular, the generic term ( ∂ κ R < ) 2 ( 1 − κ t 2 ) + ( ∂ ν R < ) 2 ( 1 − ν t 2 ) (\partial_\kappa\mathcal{R}^{<})^2(1-\kappa_t^2)+(\partial_\nu\mathcal{R}^{<})^2(1-\nu_t^2) ( ∂ κ R < ) 2 ( 1 − κ t 2 ) + ( ∂ ν R < ) 2 ( 1 − ν t 2 ) of the series converges to 0 0 0 as t → ∞ t\to\infty t → ∞ . Therefore, the accumulation points ( κ ∞ , ν ∞ ) (\kappa_\infty,\nu_\infty) ( κ ∞ , ν ∞ ) satisfy
{ ∂ κ R < ( κ ∞ , ν ∞ ) = 0 or κ ∞ 2 = 1 , ∂ ν R < ( κ ∞ , ν ∞ ) = 0 or ν ∞ 2 = 1. \begin{cases}
\partial_\kappa\mathcal{R}^{<}(\kappa_\infty,\nu_\infty)=0 & \text{or}\quad\kappa_\infty^2=1,\\
\partial_\nu\mathcal{R}^{<}(\kappa_\infty,\nu_\infty)=0 & \text{or}\quad\nu_\infty^2=1.
\end{cases} { ∂ κ R < ( κ ∞ , ν ∞ ) = 0 ∂ ν R < ( κ ∞ , ν ∞ ) = 0 or κ ∞ 2 = 1 , or ν ∞ 2 = 1.
Inspecting identity (25) , we observe that the convergence of the general term also implies κ t + 1 − κ t → 0 \kappa_{t+1}-\kappa_t\to0 κ t + 1 − κ t → 0 . Similarly, ν t + 1 − ν t → 0 \nu_{t+1}-\nu_t\to0 ν t + 1 − ν t → 0 . q.e.d.
Proof of Proposition 9
First we recall the formula of the risk
R < ( κ , ν ) = = − 2 γ 2 E [ erf ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r κ ℓ + λ r γ G 11 k ) ] + γ 2 E [ erf 2 ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf 2 ( λ r κ ℓ + λ r γ G 11 k ) ] + E [ erf 2 ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) ⋅ ∑ j = 2 N erf 2 ( λ r G 1 j k ) ] + C , (31) \mathcal{R}^{<}(\kappa,\nu) =
\begin{aligned}
&= -2\gamma^{2}\mathbb{E}\left[\operatorname{erf}\left(\lambda_{c}\operatorname{erf}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right)\left(\kappa\ell+\gamma G^{k}_{11}\right)+\lambda_{c}\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_{r}G^{k}_{1j}\right)G^{k}_{1j}\right) \right.\\
&\qquad\left. \cdot\operatorname{erf}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right)\right]\\
&\quad+\gamma^{2}\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_{c}\operatorname{erf}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right)\left(\kappa\ell+\gamma G^{k}_{11}\right)+\lambda_{c}\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_{r}G^{k}_{1j}\right)G^{k}_{1j}\right)\right.\\
&\qquad\left. \cdot\operatorname{erf}^{2}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right)\right]\\
&\quad+\mathbb{E}\left[\operatorname{erf}^{2}\left(\lambda_{c}\operatorname{erf}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right)\left(\kappa\ell+\gamma G^{k}_{11}\right)+\lambda_{c}\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_{r}G^{k}_{1j}\right)G^{k}_{1j}\right)\right.\\
&\qquad\left. \cdot\sum_{j=2}^{N}\operatorname{erf}^{2}\left(\lambda_{r}G^{k}_{1j}\right)\right]+C,
\end{aligned} \tag*{(31)} R < ( κ , ν ) = = − 2 γ 2 E [ erf ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf ( λ r κ ℓ + λ r γ G 11 k ) ] + γ 2 E [ erf 2 ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ erf 2 ( λ r κ ℓ + λ r γ G 11 k ) ] + E [ erf 2 ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) ⋅ j = 2 ∑ N erf 2 ( λ r G 1 j k ) ] + C , (31)
where C C C does not depend on κ \kappa κ and ν \nu ν .
V ( x 1 , x 2 , … , x N ) : = erf ( λ c λ r ∑ j = 1 N erf ( x j ) x j ) (32) V(x_{1},x_{2},\ldots,x_{N}) := \operatorname{erf}\left(\frac{\lambda_{c}}{\lambda_{r}}\sum_{j=1}^{N}\operatorname{erf}(x_{j})x_{j}\right) \tag*{(32)} V ( x 1 , x 2 , … , x N ) := erf ( λ r λ c j = 1 ∑ N erf ( x j ) x j ) (32)
and by
U ( x 1 , x 2 , … , x N ) : = V ( x 1 , … , x N ) erf ( x 1 ) (33) U(x_{1},x_{2},\ldots,x_{N}) := V(x_{1},\ldots,x_{N})\operatorname{erf}(x_{1}) \tag*{(33)} U ( x 1 , x 2 , … , x N ) := V ( x 1 , … , x N ) erf ( x 1 ) (33)
Furthermore, we shall make a slight abuse of notation and denote by
V ( κ ) : = V ( λ r κ ℓ + λ r γ G 11 k , λ r G 12 k , … , λ r G 1 N k ) = erf ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c ∑ j = 2 N erf ( λ r G 1 j k ) G 1 j k ) . \begin{aligned}
V(\kappa) &:= V\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11},\lambda_{r}G^{k}_{12},\ldots,\lambda_{r}G^{k}_{1N}\right)\\
&= \operatorname{erf}\left(\lambda_{c}\operatorname{erf}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right)\left(\kappa\ell+\gamma G^{k}_{11}\right)+\lambda_{c}\sum_{j=2}^{N}\operatorname{erf}\left(\lambda_{r}G^{k}_{1j}\right)G^{k}_{1j}\right).
\end{aligned} V ( κ ) := V ( λ r κ ℓ + λ r γ G 11 k , λ r G 12 k , … , λ r G 1 N k ) = erf ( λ c erf ( λ r κ ℓ + λ r γ G 11 k ) ( κ ℓ + γ G 11 k ) + λ c j = 2 ∑ N erf ( λ r G 1 j k ) G 1 j k ) .
and by
U ( κ ) : = V ( κ ) ⋅ erf ( λ r κ ℓ + λ r γ G 11 k ) . U(\kappa):=V(\kappa)\cdot\operatorname{erf}\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G^{k}_{11}\right). U ( κ ) := V ( κ ) ⋅ erf ( λ r κ ℓ + λ r γ G 11 k ) .
One can easily observe that V ( κ ) V(\kappa) V ( κ ) is symmetric in all G 1 j k G^{k}_{1j} G 1 j k and in κ \kappa κ since erf ( x ) \operatorname{erf}(x) erf ( x ) is symmetric around 0 0 0 and that ∣ V ( κ ) ∣ ≤ 1 |V(\kappa)|\leq1 ∣ V ( κ ) ∣ ≤ 1 . Moreover, this implies that U ( κ ) U(\kappa) U ( κ ) is strictly increasing in κ \kappa κ and that ∣ U ( κ ) ∣ ≤ 1 |U(\kappa)|\leq1 ∣ U ( κ ) ∣ ≤ 1 . Finally, the risk can be rewritten as
R < ( κ , ν ) = − 2 γ 2 E [ U ( κ ) ] + γ 2 E [ U ( κ ) 2 ] + E [ V ( κ ) 2 ∑ j = 2 N erf 2 ( λ r G 1 j k ) ] + C . \mathcal{R}^{<}(\kappa,\nu)=-2\gamma^{2}\mathbb{E}[U(\kappa)]+\gamma^{2}\mathbb{E}[U(\kappa)^{2}]+\mathbb{E}\left[V(\kappa)^{2}\sum_{j=2}^{N}\operatorname{erf}^{2}\left(\lambda_{r}G^{k}_{1j}\right)\right]+C. R < ( κ , ν ) = − 2 γ 2 E [ U ( κ )] + γ 2 E [ U ( κ ) 2 ] + E [ V ( κ ) 2 j = 2 ∑ N erf 2 ( λ r G 1 j k ) ] + C .
Now, ( κ , ν ) (\kappa,\nu) ( κ , ν ) is a stationary point of the PGD dynamics if
{ ∂ κ R < ( κ , ν ) = 0 or κ 2 = 1 , ∂ ν R < ( κ , ν ) = 0 or ν 2 = 1. \begin{cases}
\partial_{\kappa}\mathcal{R}^{<}(\kappa,\nu)=0 & \text{or}\quad\kappa^{2}=1,\\
\partial_{\nu}\mathcal{R}^{<}(\kappa,\nu)=0 & \text{or}\quad\nu^{2}=1.
\end{cases} { ∂ κ R < ( κ , ν ) = 0 ∂ ν R < ( κ , ν ) = 0 or κ 2 = 1 , or ν 2 = 1.
The partial derivatives of the risk are
{ ∂ κ R < ( κ , ν ) = 2 γ 2 E [ ∂ κ U ( κ ) ( U ( κ ) − ν ) ] + 2 E [ ∂ κ V ( κ ) V ( κ ) ∑ j = 2 N erf 2 ( λ r G 1 j k ) ] , ∂ ν R < ( κ , ν ) = − 2 γ 2 E [ U ( κ ) ] . (34) \begin{cases}
\partial_{\kappa}\mathcal{R}^{<}(\kappa,\nu)=2\gamma^{2}\mathbb{E}\left[\partial_{\kappa}U(\kappa)(U(\kappa)-\nu)\right]+2\mathbb{E}\left[\partial_{\kappa}V(\kappa)V(\kappa)\sum_{j=2}^{N}\operatorname{erf}^{2}\left(\lambda_{r}G^{k}_{1j}\right)\right],\\
\partial_{\nu}\mathcal{R}^{<}(\kappa,\nu)=-2\gamma^{2}\mathbb{E}[U(\kappa)]. \tag*{(34)}
\end{cases} { ∂ κ R < ( κ , ν ) = 2 γ 2 E [ ∂ κ U ( κ ) ( U ( κ ) − ν ) ] + 2 E [ ∂ κ V ( κ ) V ( κ ) ∑ j = 2 N erf 2 ( λ r G 1 j k ) ] , ∂ ν R < ( κ , ν ) = − 2 γ 2 E [ U ( κ )] . (34)
We split our analysis in different cases.
Case 1. ∂ ν R < ( κ , ν ) = 0 \partial_{\nu}\mathcal{R}^{<}(\kappa,\nu)=0 ∂ ν R < ( κ , ν ) = 0 This implies that E [ U ( κ ) ] = 0 \mathbb{E}[U(\kappa)]=0 E [ U ( κ )] = 0 . It is clear that E [ U ( 0 ) ] = 0 \mathbb{E}[U(0)]=0 E [ U ( 0 )] = 0 and due to the fact that U U U is strictly increasing, κ = 0 \kappa=0 κ = 0 is also the unique such point.
Now,
S ( κ ) : = γ 2 E [ U ( κ ) 2 ] + E [ V ( κ ) 2 ∑ j = 2 N erf 2 ( λ r G 1 j k ) ] (35) S(\kappa):=\gamma^{2}\mathbb{E}\left[U(\kappa)^{2}\right]+\mathbb{E}\left[V(\kappa)^{2}\sum_{j=2}^{N}\operatorname{erf}^{2}\left(\lambda_{r}G_{1j}^{k}\right)\right] \tag*{(35)} S ( κ ) := γ 2 E [ U ( κ ) 2 ] + E [ V ( κ ) 2 j = 2 ∑ N erf 2 ( λ r G 1 j k ) ] (35)
is jointly symmetric in ( κ , ( G 1 j k ) j ∈ [ N ] ) \left(\kappa,\left(G_{1j}^{k}\right)_{j\in[N]}\right) ( κ , ( G 1 j k ) j ∈ [ N ] ) . Taking the expectation over ( G 1 j k ) j ∈ [ N ] \left(G_{1j}^{k}\right)_{j\in[N]} ( G 1 j k ) j ∈ [ N ] , we obtain that the term in (35) is symmetric in κ \kappa κ , so its derivative must satisfy ∂ κ S ( 0 ) = 0 \partial_{\kappa}S(0)=0 ∂ κ S ( 0 ) = 0 . This implies that
0 = ∂ κ R < ( 0 , ν ) = − 2 γ 2 ν E [ ∂ κ U ( 0 ) ] . 0=\partial_{\kappa}\mathcal{R}^{<}(0,\nu)=-2\gamma^{2}\nu\mathbb{E}\left[\partial_{\kappa}U(0)\right]. 0 = ∂ κ R < ( 0 , ν ) = − 2 γ 2 ν E [ ∂ κ U ( 0 ) ] .
But U U U is strictly increasing, which implies that ν = 0 \nu=0 ν = 0 .
Therefore, the only critical point of R < \mathcal{R}^{<} R < inside of [ − 1 , 1 ] 2 [-1,1]^{2} [ − 1 , 1 ] 2 is ( 0 , 0 ) (0,0) ( 0 , 0 ) . Moreover, since the Hessian
H R < ( 0 , 0 ) = ( ∂ κ κ R < − 2 γ 2 E [ ∂ κ U ( 0 ) ] − 2 γ 2 E [ ∂ κ U ( 0 ) ] 0 ) , \mathcal{H}_{\mathcal{R}^{<}}(0,0)=
\begin{pmatrix}
\partial_{\kappa\kappa}\mathcal{R}^{<} & -2\gamma^{2}\mathbb{E}\left[\partial_{\kappa}U(0)\right]\\
-2\gamma^{2}\mathbb{E}\left[\partial_{\kappa}U(0)\right] & 0
\end{pmatrix}, H R < ( 0 , 0 ) = ( ∂ κκ R < − 2 γ 2 E [ ∂ κ U ( 0 ) ] − 2 γ 2 E [ ∂ κ U ( 0 ) ] 0 ) ,
we obtain that ( 0 , 0 ) (0,0) ( 0 , 0 ) is a saddle point.
Case 2. ν = − 1 \nu=-1 ν = − 1 It is clear that ∂ κ R < ( 0 , − 1 ) > 0 \partial_{\kappa}\mathcal{R}^{<}(0,-1)>0 ∂ κ R < ( 0 , − 1 ) > 0 , since E [ U ( κ ) 2 ] \mathbb{E}[U(\kappa)^{2}] E [ U ( κ ) 2 ] and E [ V ( κ ) 2 ] \mathbb{E}[V(\kappa)^{2}] E [ V ( κ ) 2 ] are even functions and E [ U ( κ ) ] \mathbb{E}[U(\kappa)] E [ U ( κ )] is an increasing function in κ \kappa κ . Moreover, on κ ∈ [ 0 , 1 ] \kappa\in[0,1] κ ∈ [ 0 , 1 ] , the function R < ( κ , − 1 ) \mathcal{R}^{<}(\kappa,-1) R < ( κ , − 1 ) is clearly increasing in κ \kappa κ since E [ U ( κ ) ] \mathbb{E}[U(\kappa)] E [ U ( κ )] and E [ V 2 ( κ ) ] \mathbb{E}[V^{2}(\kappa)] E [ V 2 ( κ )] are.
Let us now denote by Z 1 : = κ ℓ + γ G 11 k Z_{1}:=\kappa\ell+\gamma G_{11}^{k} Z 1 := κ ℓ + γ G 11 k and by Z j : = G 1 j k Z_{j}:=G_{1j}^{k} Z j := G 1 j k , for j ∈ { 2 , 3 , … , N } j\in\{2,3,\ldots,N\} j ∈ { 2 , 3 , … , N } . Then, one has
∂ κ U ( κ ) = ∂ κ U ( λ r κ ℓ + λ r γ G 11 k , λ r G 12 k , … , λ r G 1 N k ) = λ r ℓ ∂ 1 U ( λ r Z 1 , … , λ r Z N ) , ∂ κ V ( κ ) = ∂ κ V ( λ r κ ℓ + λ r γ G 11 k , λ r G 12 k , … , λ r G 1 N k ) = λ r ℓ ∂ 1 V ( λ r Z 1 , … , λ r Z N ) . (36) \begin{aligned}
\partial_{\kappa}U(\kappa)&=\partial_{\kappa}U\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G_{11}^{k},\lambda_{r}G_{12}^{k},\ldots,\lambda_{r}G_{1N}^{k}\right)=\lambda_{r}\ell\,\partial_{1}U(\lambda_{r}Z_{1},\ldots,\lambda_{r}Z_{N}),\\
\partial_{\kappa}V(\kappa)&=\partial_{\kappa}V\left(\lambda_{r}\kappa\ell+\lambda_{r}\gamma G_{11}^{k},\lambda_{r}G_{12}^{k},\ldots,\lambda_{r}G_{1N}^{k}\right)=\lambda_{r}\ell\,\partial_{1}V(\lambda_{r}Z_{1},\ldots,\lambda_{r}Z_{N}). \tag*{(36)}
\end{aligned} ∂ κ U ( κ ) ∂ κ V ( κ ) = ∂ κ U ( λ r κ ℓ + λ r γ G 11 k , λ r G 12 k , … , λ r G 1 N k ) = λ r ℓ ∂ 1 U ( λ r Z 1 , … , λ r Z N ) , = ∂ κ V ( λ r κ ℓ + λ r γ G 11 k , λ r G 12 k , … , λ r G 1 N k ) = λ r ℓ ∂ 1 V ( λ r Z 1 , … , λ r Z N ) . (36)
By directly applying the chain rule to the expectation, the formula for the partial derivative of the risk becomes
∂ κ R < ( κ , − 1 ) = ∂ κ E [ γ 2 ( U ( κ ) 2 + 2 U ( κ ) ) + V ( κ ) 2 ∑ j = 2 N erf 2 ( λ r Z j ) ] = λ r ℓ E [ γ 2 ∂ 1 ( U ( λ r Z ) 2 + 2 U ( λ r Z ) ) + ∂ 1 ( V ( λ r Z ) 2 ) ∑ j = 2 N erf 2 ( λ r Z j ) ] = 2 λ r ℓ E [ γ 2 ∂ 1 U ( λ r Z ) ( 1 + U ( λ r Z ) ) + V ( λ r Z ) ∂ 1 V ( λ r Z ) ∑ j = 2 N erf 2 ( λ r Z j ) ] , (37) \begin{aligned}
\partial_{\kappa}\mathcal{R}^{<}(\kappa,-1)
&=\partial_{\kappa}\mathbb{E}\left[\gamma^{2}\left(U(\kappa)^{2}+2U(\kappa)\right)+V(\kappa)^{2}\sum_{j=2}^{N}\operatorname{erf}^{2}(\lambda_{r}Z_{j})\right]\\
&=\lambda_{r}\ell\,\mathbb{E}\left[\gamma^{2}\partial_{1}\left(U(\lambda_{r}Z)^{2}+2U(\lambda_{r}Z)\right)+\partial_{1}\left(V(\lambda_{r}Z)^{2}\right)\sum_{j=2}^{N}\operatorname{erf}^{2}(\lambda_{r}Z_{j})\right]\\
&=2\lambda_{r}\ell\,\mathbb{E}\left[\gamma^{2}\partial_{1}U(\lambda_{r}Z)\left(1+U(\lambda_{r}Z)\right)+V(\lambda_{r}Z)\partial_{1}V(\lambda_{r}Z)\sum_{j=2}^{N}\operatorname{erf}^{2}(\lambda_{r}Z_{j})\right], \tag*{(37)}
\end{aligned} ∂ κ R < ( κ , − 1 ) = ∂ κ E [ γ 2 ( U ( κ ) 2 + 2 U ( κ ) ) + V ( κ ) 2 j = 2 ∑ N erf 2 ( λ r Z j ) ] = λ r ℓ E [ γ 2 ∂ 1 ( U ( λ r Z ) 2 + 2 U ( λ r Z ) ) + ∂ 1 ( V ( λ r Z ) 2 ) j = 2 ∑ N erf 2 ( λ r Z j ) ] = 2 λ r ℓ E [ γ 2 ∂ 1 U ( λ r Z ) ( 1 + U ( λ r Z ) ) + V ( λ r Z ) ∂ 1 V ( λ r Z ) j = 2 ∑ N erf 2 ( λ r Z j ) ] , (37)
where U ( λ r Z ) U(\lambda_{r}Z) U ( λ r Z ) stands for U ( λ r Z 1 , … , λ r Z N ) U(\lambda_{r}Z_{1},\ldots,\lambda_{r}Z_{N}) U ( λ r Z 1 , … , λ r Z N ) and the same for the other functions.
We will now show that if λ c = τ λ r \lambda_{c}=\tau\lambda_{r} λ c = τ λ r , for λ r \lambda_{r} λ r small enough and the other hyperparameters fixed, the expectation above is strictly positive for all κ ∈ [ − 1 , 1 ] \kappa\in[-1,1] κ ∈ [ − 1 , 1 ] . We shall make use of the Taylor expansions
erf ( x ) = 2 π x + erf ′ ′ ′ ( ξ x ) 6 x 3 = 2 π x + O ( x 3 ) , \operatorname{erf}(x)=\frac{2}{\sqrt{\pi}}x+\frac{\operatorname{erf}^{\prime\prime\prime}(\xi_{x})}{6}x^{3}=\frac{2}{\sqrt{\pi}}x+O(x^{3}), erf ( x ) = π 2 x + 6 erf ′′′ ( ξ x ) x 3 = π 2 x + O ( x 3 ) ,
for some ξ x ∈ [ 0 , x ] \xi_{x}\in[0,x] ξ x ∈ [ 0 , x ] , and
e − x = 1 − e − ζ x x = 1 + O ( x ) , e^{-x}=1-e^{-\zeta_{x}}x=1+O(x), e − x = 1 − e − ζ x x = 1 + O ( x ) ,
for some ζ x ∈ [ 0 , x ] \zeta_{x}\in[0,x] ζ x ∈ [ 0 , x ] . Note that ∣ erf ′ ′ ′ ( x ) ∣ < 2.5 |\operatorname{erf}^{\prime\prime\prime}(x)|<2.5 ∣ erf ′′′ ( x ) ∣ < 2.5 , for any x ∈ R x\in\mathbb{R} x ∈ R and ∣ e − ζ x ∣ ≤ 1 |e^{-\zeta_{x}}|\leq1 ∣ e − ζ x ∣ ≤ 1 for any x ∈ [ 0 , ∞ ) x\in[0,\infty) x ∈ [ 0 , ∞ ) . This in turn implies the following asymptotic scalings
V ( λ r Z ) = erf ( τ λ r ∑ i = 1 N Z i erf ( λ r Z i ) ) = 4 τ π λ r 2 ∑ i = 1 N Z i 2 + O ( λ r 3 ) , V(\lambda_{r}Z)=\operatorname{erf}\left(\tau\lambda_{r}\sum_{i=1}^{N}Z_{i}\operatorname{erf}(\lambda_{r}Z_{i})\right)=\frac{4\tau}{\pi}\lambda_{r}^{2}\sum_{i=1}^{N}Z_{i}^{2}+O(\lambda_{r}^{3}), V ( λ r Z ) = erf ( τ λ r i = 1 ∑ N Z i erf ( λ r Z i ) ) = π 4 τ λ r 2 i = 1 ∑ N Z i 2 + O ( λ r 3 ) ,
U ( λ r Z ) = V ( λ r Z ) erf ( λ r Z 1 ) = 8 τ π π λ r 3 Z 1 ∑ i = 1 N Z i 2 + O ( λ r 4 ) , U(\lambda_r Z)=V(\lambda_r Z)\operatorname{erf}(\lambda_r Z_1)=\frac{8\tau}{\pi\sqrt{\pi}}\lambda_r^3 Z_1\sum_{i=1}^{N}Z_i^2+O(\lambda_r^4), U ( λ r Z ) = V ( λ r Z ) erf ( λ r Z 1 ) = π π 8 τ λ r 3 Z 1 i = 1 ∑ N Z i 2 + O ( λ r 4 ) ,
∂ 1 V ( λ r Z ) = 2 τ π e − τ 2 ( ∑ i = 1 N λ r Z i erf ( λ r Z i ) ) 2 ( 2 π λ r Z 1 e − λ r 2 Z 1 2 + erf ( λ r Z 1 ) ) = 2 τ π ( 1 − O ( λ r 2 ) ) ( 2 π λ r Z 1 ( 1 − O ( λ r 2 ) ) + 2 π λ r Z 1 ) = 8 τ π λ r Z 1 + O ( λ r 3 ) , \begin{aligned}
\partial_1V(\lambda_r Z)
&=\frac{2\tau}{\sqrt{\pi}}e^{-\tau^2\left(\sum_{i=1}^{N}\lambda_r Z_i\operatorname{erf}(\lambda_r Z_i)\right)^2}
\left(\frac{2}{\sqrt{\pi}}\lambda_r Z_1e^{-\lambda_r^2Z_1^2}+\operatorname{erf}(\lambda_r Z_1)\right)\\
&=\frac{2\tau}{\sqrt{\pi}}\left(1-O(\lambda_r^2)\right)\left(\frac{2}{\sqrt{\pi}}\lambda_r Z_1\left(1-O(\lambda_r^2)\right)+\frac{2}{\sqrt{\pi}}\lambda_r Z_1\right)\\
&=\frac{8\tau}{\pi}\lambda_r Z_1+O(\lambda_r^3),
\end{aligned} ∂ 1 V ( λ r Z ) = π 2 τ e − τ 2 ( ∑ i = 1 N λ r Z i erf ( λ r Z i ) ) 2 ( π 2 λ r Z 1 e − λ r 2 Z 1 2 + erf ( λ r Z 1 ) ) = π 2 τ ( 1 − O ( λ r 2 ) ) ( π 2 λ r Z 1 ( 1 − O ( λ r 2 ) ) + π 2 λ r Z 1 ) = π 8 τ λ r Z 1 + O ( λ r 3 ) ,
∂ 1 U ( λ r Z ) = 2 π e − λ r 2 Z 1 2 V ( λ r Z ) + ∂ 1 V ( λ r Z ) erf ( λ r Z 1 ) = 8 τ π π λ r 2 ∑ i = 1 N Z i 2 + 16 τ π π λ r 2 Z 1 2 + O ( λ r 3 ) = 8 τ π π λ r 2 ( ∑ i = 1 N Z i 2 + 2 Z 1 2 ) + O ( λ r 3 ) . (38) \begin{aligned}
\partial_1U(\lambda_r Z)
&=\frac{2}{\sqrt{\pi}}e^{-\lambda_r^2Z_1^2}V(\lambda_r Z)+\partial_1V(\lambda_r Z)\operatorname{erf}(\lambda_r Z_1)\\
&=\frac{8\tau}{\pi\sqrt{\pi}}\lambda_r^2\sum_{i=1}^{N}Z_i^2+\frac{16\tau}{\pi\sqrt{\pi}}\lambda_r^2Z_1^2+O(\lambda_r^3)\\
&=\frac{8\tau}{\pi\sqrt{\pi}}\lambda_r^2\left(\sum_{i=1}^{N}Z_i^2+2Z_1^2\right)+O(\lambda_r^3). \tag*{(38)}
\end{aligned} ∂ 1 U ( λ r Z ) = π 2 e − λ r 2 Z 1 2 V ( λ r Z ) + ∂ 1 V ( λ r Z ) erf ( λ r Z 1 ) = π π 8 τ λ r 2 i = 1 ∑ N Z i 2 + π π 16 τ λ r 2 Z 1 2 + O ( λ r 3 ) = π π 8 τ λ r 2 ( i = 1 ∑ N Z i 2 + 2 Z 1 2 ) + O ( λ r 3 ) . (38)
From these computations, it is also clear that every term O ( λ r n ) = C ( Z 1 , Z 2 , … , Z N ) λ r n O(\lambda_r^n)=C(Z_1,Z_2,\ldots,Z_N)\lambda_r^n O ( λ r n ) = C ( Z 1 , Z 2 , … , Z N ) λ r n , where E [ ∣ C ( Z 1 , … , Z N ) ∣ ] < ∞ \mathbb{E}[|C(Z_1,\ldots,Z_N)|]<\infty E [ ∣ C ( Z 1 , … , Z N ) ∣ ] < ∞ .
Finally, (37) becomes
∂ κ R < ( κ , − 1 ) = 16 τ ℓ γ 2 π π λ r 3 E [ ∑ i = 1 N Z i 2 + 2 Z 1 2 ] + O ( λ r 4 ) . \partial_\kappa R^{<}(\kappa,-1)=\frac{16\tau\ell\gamma^2}{\pi\sqrt{\pi}}\lambda_r^3\mathbb{E}\left[\sum_{i=1}^{N}Z_i^2+2Z_1^2\right]+O(\lambda_r^4). ∂ κ R < ( κ , − 1 ) = π π 16 τ ℓ γ 2 λ r 3 E [ i = 1 ∑ N Z i 2 + 2 Z 1 2 ] + O ( λ r 4 ) .
Since the coefficient of the leading term is positive, this concludes the proof that there exists λ r \lambda_r λ r small enough such that ∂ κ R < ( κ , − 1 ) > 0 \partial_\kappa R^{<}(\kappa,-1)>0 ∂ κ R < ( κ , − 1 ) > 0 for all κ ∈ [ − 1 , 1 ] \kappa\in[-1,1] κ ∈ [ − 1 , 1 ] .
Case 3. ν = 1 \nu=1 ν = 1 By leveraging the global symmetry of the theoretical risk function, specifically R < ( − κ , − ν ) = R < ( κ , ν ) R^{<}(-\kappa,-\nu)=R^{<}(\kappa,\nu) R < ( − κ , − ν ) = R < ( κ , ν ) , an analogous argument applies along the opposite boundary ν = 1 \nu=1 ν = 1 . The identical asymptotic smoothness guarantees ∂ κ R < ( κ , 1 ) < 0 \partial_\kappa R^{<}(\kappa,1)<0 ∂ κ R < ( κ , 1 ) < 0 for all κ ∈ [ − 1 , 1 ] \kappa\in[-1,1] κ ∈ [ − 1 , 1 ] .
Finally combining the results of the three cases, one obtains the conclusion.
Proof of Proposition 10
We verify the claims of the proposition sequentially.
Local diffeomorphism and eigenvalues at the origin. Let x = ( κ , ν ) x=(\kappa,\nu) x = ( κ , ν ) . Because the gradient of the risk vanishes at the origin ( ∂ κ R < ( 0 , 0 ) = 0 ) (\partial_\kappa R^{<}(0,0)=0) ( ∂ κ R < ( 0 , 0 ) = 0 ) and R < R^{<} R < is smooth, the squared partial derivative satisfies ( ∂ κ R < ( κ , ν ) ) 2 = O ( ∥ x ∥ 2 ) (\partial_\kappa R^{<}(\kappa,\nu))^2=O(\lVert x\rVert^2) ( ∂ κ R < ( κ , ν ) ) 2 = O (∥ x ∥ 2 ) . This allows us to write the Taylor expansion for the first component of g g g , denoted g 1 ( κ , ν ) g_1(\kappa,\nu) g 1 ( κ , ν ) :
g 1 ( κ , ν ) = κ − α ( ∂ κ R < ( κ , ν ) ) ( 1 − κ 2 ) 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) = κ − α ( ∂ κ R < ( κ , ν ) ) ( 1 − κ 2 ) 1 + O ( ∥ x ∥ 2 ) = ( κ − α ( ∂ κ R < ( κ , ν ) ) ( 1 − κ 2 ) ) ( 1 + O ( ∥ x ∥ 2 ) ) = κ − α ∂ κ R < ( κ , ν ) + O ( ∥ x ∥ 2 ) . \begin{aligned}
g_1(\kappa,\nu)
&=\frac{\kappa-\alpha(\partial_\kappa R^{<}(\kappa,\nu))(1-\kappa^2)}{\sqrt{1+\alpha^2(\partial_\kappa R^{<}(\kappa,\nu))^2(1-\kappa^2)}}\\
&=\frac{\kappa-\alpha(\partial_\kappa R^{<}(\kappa,\nu))(1-\kappa^2)}{\sqrt{1+O(\lVert x\rVert^2)}}\\
&=\left(\kappa-\alpha(\partial_\kappa R^{<}(\kappa,\nu))(1-\kappa^2)\right)(1+O(\lVert x\rVert^2))\\
&=\kappa-\alpha\partial_\kappa R^{<}(\kappa,\nu)+O(\lVert x\rVert^2).
\end{aligned} g 1 ( κ , ν ) = 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) κ − α ( ∂ κ R < ( κ , ν )) ( 1 − κ 2 ) = 1 + O (∥ x ∥ 2 ) κ − α ( ∂ κ R < ( κ , ν )) ( 1 − κ 2 ) = ( κ − α ( ∂ κ R < ( κ , ν )) ( 1 − κ 2 ) ) ( 1 + O (∥ x ∥ 2 )) = κ − α ∂ κ R < ( κ , ν ) + O (∥ x ∥ 2 ) .
Applying the same expansion to the second component g 2 ( κ , ν ) g_2(\kappa,\nu) g 2 ( κ , ν ) yields the Jacobian matrix of g g g evaluated at the origin, J g ( 0 , 0 ) = I 2 − α H R < ( 0 , 0 ) J_g(0,0)=I_2-\alpha H_{R^{<}}(0,0) J g ( 0 , 0 ) = I 2 − α H R < ( 0 , 0 ) . Because det ( H R < ( 0 , 0 ) ) = − 4 γ 4 ( E [ ∂ κ U ( 0 ) ] ) 2 \det(H_{R^{<}}(0,0))=-4\gamma^4(\mathbb{E}[\partial_\kappa U(0)])^2 det ( H R < ( 0 , 0 )) = − 4 γ 4 ( E [ ∂ κ U ( 0 )] ) 2 , from the proof of Proposition 9, the eigenvalues of H R < ( 0 , 0 ) H_{R^{<}}(0,0) H R < ( 0 , 0 ) have strictly opposite signs. Consequently, for a sufficiently small step size α > 0 \alpha>0 α > 0 , J g ( 0 , 0 ) J_g(0,0) J g ( 0 , 0 ) possesses one eigenvalue strictly inside the interval ( 0 , 1 ) (0,1) ( 0 , 1 ) and another strictly greater than 1 1 1 . This confirms that J g ( 0 , 0 ) J_g(0,0) J g ( 0 , 0 ) is invertible, which by the Inverse Function Theorem ensures g g g is a local diffeomorphism around ( 0 , 0 ) (0,0) ( 0 , 0 ) .
Differentiability and non-degenerate Jacobian on [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 . The mapping g g g is globally differentiable on [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 as it is a composition of smooth elementary functions. To prove its Jacobian is nowhere degenerate, we express the mapping as g ( x ) = x + α h ( x ) g(x)=x+\alpha h(x) g ( x ) = x + α h ( x ) . The first coordinate of h h h takes the form:
h 1 ( κ , ν ) = 1 α ( g 1 ( κ , ν ) − κ ) = κ α ( 1 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) − 1 ) − ∂ κ R < ( κ , ν ) ( 1 − κ 2 ) 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) . \begin{aligned}
h_1(\kappa,\nu)=\frac{1}{\alpha}(g_1(\kappa,\nu)-\kappa) \\
&=\frac{\kappa}{\alpha}\left(\frac{1}{\sqrt{1+\alpha^2(\partial_\kappa\mathcal{R}_{<}(\kappa,\nu))^2(1-\kappa^2)}}-1\right)-\frac{\partial_\kappa\mathcal{R}_{<}(\kappa,\nu)(1-\kappa^2)}{\sqrt{1+\alpha^2(\partial_\kappa\mathcal{R}_{<}(\kappa,\nu))^2(1-\kappa^2)}}.
\end{aligned} h 1 ( κ , ν ) = α 1 ( g 1 ( κ , ν ) − κ ) = α κ ( 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) 1 − 1 ) − 1 + α 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) ∂ κ R < ( κ , ν ) ( 1 − κ 2 ) .
We must prove that the gradient of h 1 ( κ , ν ) h_1(\kappa,\nu) h 1 ( κ , ν ) remains uniformly bounded for all α ≤ 1 \alpha\leq1 α ≤ 1 . Term 2 is a composition of functions with uniformly bounded gradients on the compact domain, so its gradient is trivially bounded. For Term 1, we introduce the auxiliary function q ( a , b ) = a α ( 1 1 + α 2 b − 1 ) q(a,b)=\frac{a}{\alpha}\left(\frac{1}{\sqrt{1+\alpha^2b}}-1\right) q ( a , b ) = α a ( 1 + α 2 b 1 − 1 ) and define the supremum B = sup [ − 1 , 1 ] 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) B=\sup_{[-1,1]^2}(\partial_\kappa\mathcal{R}_{<}(\kappa,\nu))^2(1-\kappa^2) B = sup [ − 1 , 1 ] 2 ( ∂ κ R < ( κ , ν ) ) 2 ( 1 − κ 2 ) . Term 1 can be expressed as q ( κ , ( ∂ κ R < ) 2 ( 1 − κ 2 ) ) q(\kappa,(\partial_\kappa\mathcal{R}_{<})^2(1-\kappa^2)) q ( κ , ( ∂ κ R < ) 2 ( 1 − κ 2 )) . Since the inner function is smooth and independent of α \alpha α , we only need to bound the partial derivatives of q q q . By Lemma 14, for any α ≤ 1 \alpha\leq1 α ≤ 1 :
∣ ∂ a q ( a , b ) ∣ = 1 α ∣ 1 1 + α 2 b − 1 ∣ ≤ α b ≤ B , \left|\partial_aq(a,b)\right|=\frac{1}{\alpha}\left|\frac{1}{\sqrt{1+\alpha^2b}}-1\right|\leq\alpha b\leq B, ∣ ∂ a q ( a , b ) ∣ = α 1 1 + α 2 b 1 − 1 ≤ α b ≤ B ,
∣ ∂ b q ( a , b ) ∣ = ∣ − α a 2 ( 1 + α 2 b ) 3 / 2 ∣ ≤ α 2 ≤ 1 2 . \left|\partial_bq(a,b)\right|=\left|-\frac{\alpha a}{2(1+\alpha^2b)^{3/2}}\right|\leq\frac{\alpha}{2}\leq\frac{1}{2}. ∣ ∂ b q ( a , b ) ∣ = − 2 ( 1 + α 2 b ) 3/2 α a ≤ 2 α ≤ 2 1 .
Since both partial derivatives are uniformly bounded, the gradient of h 1 h_1 h 1 is uniformly bounded. A symmetric argument applies to h 2 h_2 h 2 , confirming that the Jacobian matrix J h ( κ , ν ) J_h(\kappa,\nu) J h ( κ , ν ) is uniformly bounded over α ≤ 1 \alpha\leq1 α ≤ 1 . Because J g ( κ , ν ) = I 2 + α J h ( κ , ν ) J_g(\kappa,\nu)=I_2+\alpha J_h(\kappa,\nu) J g ( κ , ν ) = I 2 + α J h ( κ , ν ) , selecting a sufficiently small α \alpha α ensures the eigenvalues of J g J_g J g remain strictly positive, guaranteeing the Jacobian is nowhere degenerate.
Injectivity. The uniform bound on J h J_h J h implies that h h h is B L B_L B L -Lipschitz continuous on [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 for some constant B L > 0 B_L>0 B L > 0 independent of α \alpha α (for small α \alpha α ). Choose α < 1 / B L \alpha<1/B_L α < 1/ B L . Suppose there exist two distinct points x , y ∈ [ − 1 , 1 ] 2 x,y\in[-1,1]^2 x , y ∈ [ − 1 , 1 ] 2 such that g ( x ) = g ( y ) g(x)=g(y) g ( x ) = g ( y ) . This implies:
x + α h ( x ) = y + α h ( y ) ⟹ x − y = α ( h ( y ) − h ( x ) ) . x+\alpha h(x)=y+\alpha h(y)\implies x-y=\alpha(h(y)-h(x)). x + α h ( x ) = y + α h ( y ) ⟹ x − y = α ( h ( y ) − h ( x )) .
Taking the Euclidean norm of both sides yields:
∥ x − y ∥ 2 = α ∥ h ( y ) − h ( x ) ∥ 2 ≤ α B L ∥ x − y ∥ 2 . \lVert x-y\rVert_2=\alpha\lVert h(y)-h(x)\rVert_2\leq\alpha B_L\lVert x-y\rVert_2. ∥ x − y ∥ 2 = α ∥ h ( y ) − h ( x ) ∥ 2 ≤ α B L ∥ x − y ∥ 2 .
Because α B L < 1 \alpha B_L<1 α B L < 1 , this inequality forces ∥ x − y ∥ 2 < ∥ x − y ∥ 2 \lVert x-y\rVert_2<\lVert x-y\rVert_2 ∥ x − y ∥ 2 < ∥ x − y ∥ 2 , which is a strict contradiction. Therefore, g g g must be injective on the domain [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 .
Final details.
From Proposition 8, the distance between successive iterates ∥ ( κ t + 1 , ν t + 1 ) − ( κ t , ν t ) ∥ 2 \lVert(\kappa_{t+1},\nu_{t+1})-(\kappa_t,\nu_t)\rVert_2 ∥( κ t + 1 , ν t + 1 ) − ( κ t , ν t ) ∥ 2 tends to zero. Because the sequence ( κ t , ν t ) t ≥ 0 (\kappa_t,\nu_t)_{t\geq0} ( κ t , ν t ) t ≥ 0 lies within the compact set [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 , the set of its accumulation points must be connected. Given that the set of available fixed points is finite and isolated { ( 0 , 0 ) , ( ± 1 , ± 1 ) , ( ± 1 , ∓ 1 ) } \{(0,0),(\pm1,\pm1),(\pm1,\mp1)\} {( 0 , 0 ) , ( ± 1 , ± 1 ) , ( ± 1 , ∓ 1 )} , the sequence cannot oscillate and must uniquely converge to exactly one of them.
It remains to be shown that the probability of converging to the saddle point ( 0 , 0 ) (0,0) ( 0 , 0 ) or the maxima ( ± 1 , ∓ 1 ) (\pm1,\mp1) ( ± 1 , ∓ 1 ) is zero. Because the risk strictly decreases at every step (Proposition 8), convergence to the global maxima is impossible unless initialized exactly upon them.
To systematically rule out the saddle point ( 0 , 0 ) (0,0) ( 0 , 0 ) , we apply the Center-Stable Manifold Theorem [Shub, 2013, Theorem III.7]. Because J g ( 0 , 0 ) J_g(0,0) J g ( 0 , 0 ) has exactly one eigenvalue with magnitude strictly greater than 1 (Proposition 10 ), the local center-stable manifold W l o c c s W_{\mathrm{loc}}^{cs} W loc cs associated with the origin has codimension 1. In the 2-dimensional space [ − 1 , 1 ] 2 [-1,1]^2 [ − 1 , 1 ] 2 , a codimension-1 manifold is a curve, and its Lebesgue measure is strictly zero.
The stable set W s W^s W s , consisting of all initializations that converge to ( 0 , 0 ) (0,0) ( 0 , 0 ) under iterations of g g g , is contained within the countable union of pre-images of W l o c c s W_{\mathrm{loc}}^{cs} W loc cs :
W s ⊆ ⋃ T ≥ 0 g − T ( W l o c c s ) . (39) W^s\subseteq\bigcup_{T\geq0}g^{-T}(W_{\mathrm{loc}}^{cs}). \tag*{(39)} W s ⊆ T ≥ 0 ⋃ g − T ( W loc cs ) . (39)
Because g g g is injective and differentiable, its inverse preserves sets of measure zero. The countable union of measure-zero sets is also of measure zero.
Therefore, the set of all initializations on M \mathcal{M} M that converge to the saddle point ( 0 , 0 ) (0,0) ( 0 , 0 ) has Lebesgue measure zero. We conclude that for a generic random initialization ( k 0 , v 0 ) ∈ M (k_0,v_0)\in\mathcal{M} ( k 0 , v 0 ) ∈ M , the PGD sequence almost surely bypasses the saddle point and the maxima, converging strictly to the true oracle structure ( ± 1 , ± 1 ) (\pm1,\pm1) ( ± 1 , ± 1 ) .
Technical lemmas
In this section we list the technical lemmas used throughout the proofs in the Appendix.
Lemma 12. For u ∈ R u\in\mathbb{R} u ∈ R ,
erf 2 ( u ) ≤ 4 π u 2 . \operatorname{erf}^{2}(u)\leq\frac{4}{\pi}u^{2}. erf 2 ( u ) ≤ π 4 u 2 .
Proof. The proof relies on the concavity of erf \operatorname{erf} erf on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . Since erf \operatorname{erf} erf is an odd function, it is enough to prove the statement for u ≥ 0 u\geq0 u ≥ 0 . Since erf \operatorname{erf} erf is concave,
erf ( u ) ≤ erf ( 0 ) + erf ′ ( 0 ) u = 2 π u , \operatorname{erf}(u)\leq\operatorname{erf}(0)+\operatorname{erf}'(0)u=\frac{2}{\sqrt{\pi}}u, erf ( u ) ≤ erf ( 0 ) + erf ′ ( 0 ) u = π 2 u ,
which implies the conclusion.
Lemma 13. For G ∼ N ( 0 , σ 2 ) G\sim\mathcal{N}(0,\sigma^{2}) G ∼ N ( 0 , σ 2 ) and t ∈ R t\in\mathbb{R} t ∈ R ,
E [ erf ( t + G ) ] = erf ( t 1 + 2 σ 2 ) . \mathbb{E}[\operatorname{erf}(t+G)]=\operatorname{erf}\left(\frac{t}{\sqrt{1+2\sigma^{2}}}\right). E [ erf ( t + G )] = erf ( 1 + 2 σ 2 t ) .
Proof. One has
E [ erf ( t + G ) ] = ∫ 0 t E [ erf ′ ( s + G ) ] d s = ∫ 0 t E [ 2 π e − ( s + G ) 2 ] d s = ∫ 0 t ∫ − ∞ ∞ 2 π σ e − ( s + g ) 2 e − g 2 2 σ 2 d g d s = 2 π σ ∫ 0 t ∫ − ∞ ∞ e − 1 + 2 σ 2 2 σ 2 g 2 − 2 g s − s 2 d g d s = 2 π σ ∫ 0 t ∫ − ∞ ∞ e − ( g 1 + 2 σ 2 2 σ 2 + s 2 σ 2 1 + 2 σ 2 ) 2 e − 1 1 + 2 σ 2 s 2 d g d s = 2 π σ ∫ 0 t π 2 σ 2 1 + 2 σ 2 e − 1 1 + 2 σ 2 s 2 d s = 2 π ∫ 0 t 1 + 2 σ 2 e − u 2 d u = erf ( t 1 + 2 σ 2 ) . \begin{aligned}
\mathbb{E}[\operatorname{erf}(t+G)]
&=\int_{0}^{t}\mathbb{E}[\operatorname{erf}'(s+G)]\,\mathrm{d}s\\
&=\int_{0}^{t}\mathbb{E}\left[\frac{2}{\sqrt{\pi}}e^{-(s+G)^{2}}\right]\,\mathrm{d}s\\
&=\int_{0}^{t}\int_{-\infty}^{\infty}\frac{\sqrt{2}}{\pi\sigma}e^{-(s+g)^{2}}e^{-\frac{g^{2}}{2\sigma^{2}}}\,\mathrm{d}g\,\mathrm{d}s\\
&=\frac{\sqrt{2}}{\pi\sigma}\int_{0}^{t}\int_{-\infty}^{\infty}e^{-\frac{1+2\sigma^{2}}{2\sigma^{2}}g^{2}-2gs-s^{2}}\,\mathrm{d}g\,\mathrm{d}s\\
&=\frac{\sqrt{2}}{\pi\sigma}\int_{0}^{t}\int_{-\infty}^{\infty}e^{-\left(g\sqrt{\frac{1+2\sigma^{2}}{2\sigma^{2}}}+s\sqrt{\frac{2\sigma^{2}}{1+2\sigma^{2}}}\right)^{2}}e^{-\frac{1}{1+2\sigma^{2}}s^{2}}\,\mathrm{d}g\,\mathrm{d}s\\
&=\frac{\sqrt{2}}{\pi\sigma}\int_{0}^{t}\sqrt{\pi}\sqrt{\frac{2\sigma^{2}}{1+2\sigma^{2}}}e^{-\frac{1}{1+2\sigma^{2}}s^{2}}\,\mathrm{d}s\\
&=\frac{2}{\sqrt{\pi}}\int_{0}^{\frac{t}{\sqrt{1+2\sigma^{2}}}}e^{-u^{2}}\,\mathrm{d}u\\
&=\operatorname{erf}\left(\frac{t}{\sqrt{1+2\sigma^{2}}}\right).
\end{aligned} E [ erf ( t + G )] = ∫ 0 t E [ erf ′ ( s + G )] d s = ∫ 0 t E [ π 2 e − ( s + G ) 2 ] d s = ∫ 0 t ∫ − ∞ ∞ π σ 2 e − ( s + g ) 2 e − 2 σ 2 g 2 d g d s = π σ 2 ∫ 0 t ∫ − ∞ ∞ e − 2 σ 2 1 + 2 σ 2 g 2 − 2 g s − s 2 d g d s = π σ 2 ∫ 0 t ∫ − ∞ ∞ e − ( g 2 σ 2 1 + 2 σ 2 + s 1 + 2 σ 2 2 σ 2 ) 2 e − 1 + 2 σ 2 1 s 2 d g d s = π σ 2 ∫ 0 t π 1 + 2 σ 2 2 σ 2 e − 1 + 2 σ 2 1 s 2 d s = π 2 ∫ 0 1 + 2 σ 2 t e − u 2 d u = erf ( 1 + 2 σ 2 t ) .
Lemma 14. For u ≥ 0 u\geq0 u ≥ 0 ,
∣ 1 1 + u − 1 ∣ ≤ u . \left|\frac{1}{\sqrt{1+u}}-1\right|\leq u. 1 + u 1 − 1 ≤ u .
Proof. For u ≥ 0 u\geq0 u ≥ 0 , the term inside the absolute value is at most 0 0 0 , so we need to prove that
u + 1 1 + u ≥ 1. u+\frac{1}{\sqrt{1+u}}\geq1. u + 1 + u 1 ≥ 1.
This statement is obvious for u ≥ 1 u\geq1 u ≥ 1 . For u ∈ [ 0 , 1 ) u\in[0,1) u ∈ [ 0 , 1 ) , it is equivalent to
1 1 + u ≥ 1 − u ⟺ 1 1 + u ≥ ( 1 − u ) 2 ⟺ 1 ≥ ( 1 − u 2 ) ( 1 − u ) , \frac{1}{\sqrt{1+u}}\geq1-u\Longleftrightarrow\frac{1}{1+u}\geq(1-u)^{2}\Longleftrightarrow1\geq(1-u^{2})(1-u), 1 + u 1 ≥ 1 − u ⟺ 1 + u 1 ≥ ( 1 − u ) 2 ⟺ 1 ≥ ( 1 − u 2 ) ( 1 − u ) ,
which is clearly true since u ∈ [ 0 , 1 ) u\in[0,1) u ∈ [ 0 , 1 ) . One has equality only when u = 0 u=0 u = 0 .
Additional Experimental Details and Results
In this section, we provide supplementary empirical results that expand upon the optimization dynamics discussed in Section 4. Specifically, we analyze the training dynamics of the predictors without manifold penalties in Section E.2, provide the full results for the experiments in Figure 4 in Section E.3 and further investigate the optimization landscape for decoupled λ c \lambda_c λ c and λ r \lambda_r λ r in Section E.4, and for different values of ℓ \ell ℓ and N N N in Section E.5.
Implementation details and hyperparameters
Code availability. The complete codebase used to run all numerical simulations and generate the figures presented in this work can be found at https://github.com/anditp/Single-Location-Regression.git . All the numerical experiments were run on a MacBook Pro M1 on CPU. Each individual experiment (20 runs) in the paper takes less than an hour to run.
Distance to the invariant manifold. Throughout our experiments, we track the optimization trajectories to determine if they remain on, or converge to, the invariant manifold M \mathcal{M} M . Recall that M \mathcal{M} M is defined by the strict orthogonality constraints:
\mathcal{M} = \left\{(k,v) \in(\mathbb{S}^{d-1})^2 \mathrel{\left|} k^{T}v^{*}=0,\ v^{T}k^{*}=0,\ k^{T}v=0 \right.\right\}.
Consequently, for any parameter pair ( k , v ) ∈ ( S d − 1 ) 2 (k,v) \in(\mathbb{S}^{d-1})^2 ( k , v ) ∈ ( S d − 1 ) 2 , we compute its distance to M \mathcal{M} M continuously during training using the root sum of squares of the corresponding inner products:
d M ( k , v ) : = ( k T v ∗ ) 2 + ( v T k ∗ ) 2 + ( k T v ) 2 . (40) d_{\mathcal{M}}(k,v) := \sqrt{(k^{T}v^{*})^2 + (v^{T}k^{*})^2 + (k^{T}v)^2}. \tag*{(40)} d M ( k , v ) := ( k T v ∗ ) 2 + ( v T k ∗ ) 2 + ( k T v ) 2 . (40)
Hyperparameters. Table 1 summarizes the specific values and schedules used across all numerical experiments presented in Section 4 and Appendix § 10 . For the penalized empirical risk experiments, we introduced the weighting coefficients μ 1 \mu_1 μ 1 and μ 2 \mu_2 μ 2 to enforce the structural constraints.
Table 1 (PDF p. 26). Summary of hyperparameters used across the numerical experiments.
The batches for Stochastic PGD are generated dynamically and resampled at each step, since we are aiming to mimic PGD on the theoretical risk.
Optimization without Manifold Penalties
In Section 4, we demonstrated that optimizing the penalized empirical risk allows both the erf and softmax predictors to converge to the oracle parameters from generic random initializations. To isolate the effect of the penalty terms, we conduct an ablation study where both predictors are trained using the strictly unpenalized risk from generic initializations outside of M \mathcal{M} M .
Figure 6 presents the optimization trajectories for both T λ r , λ c ( k , v ) T_{\lambda_r,\lambda_c}^{(k,v)} T λ r , λ c ( k , v ) (erf predictor) and T λ r , λ c , s o f t m a x ( k , v ) T_{\lambda_r,\lambda_c,\mathrm{softmax}}^{(k,v)} T λ r , λ c , softmax ( k , v ) (softmax predictor) under these unconstrained conditions. Without the explicit geometric regularization provided by the penalties, the trajectories fail to always converge. For the erf predictor there are runs which get trapped in high loss regions. On the other hand, the optimization of the softmax predictor appears better behaved, with most of the runs converging in a similar manner as for the penalized version; however, there are some runs that get trapped in regions with high positive ν \nu ν and high negative κ \kappa κ , far from M \mathcal{M} M . This empirical finding underscores the practical necessity of the regularization terms when operating outside the theoretical guarantees of Theorem 5 .
Figure 6. (a) Dynamics for the erf predictor. (b) Dynamics for the softmax predictor. Figure 6: Optimization dynamics from 20 random initializations without the alignment penalties (μ 1 = μ 2 = 0 \mu_1 = \mu_2 = 0 μ 1 = μ 2 = 0 ). The solid curve represents the empirical median across the runs. The bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals. The parameters used are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 and λ t = 10 − 1 / ( 1 + max ( t − 5 ⋅ 10 4 , 0 ) ⋅ 10 − 4 ) \lambda_t = 10^{-1}/(1 + \max(t - 5 \cdot10^4, 0) \cdot10^{-4}) λ t = 1 0 − 1 / ( 1 + max ( t − 5 ⋅ 1 0 4 , 0 ) ⋅ 1 0 − 4 ) , for the erf predictor and λ t = 10 − 1 / ( 1 + max ( t − 2 ⋅ 10 4 , 0 ) ⋅ 10 − 4 ) \lambda_t = 10^{-1}/(1 + \max(t - 2 \cdot10^4, 0) \cdot10^{-4}) λ t = 1 0 − 1 / ( 1 + max ( t − 2 ⋅ 1 0 4 , 0 ) ⋅ 1 0 − 4 ) for the softmax predictor. For computing the oracle loss we use the initial inverse temperatures.
Full results for the experiment in Figure 4
In Section 4, we discussed the optimization dynamics under various constant inverse temperatures following an initial warm-up phase. Here, we present the full empirical results for this experiment.
As illustrated in Figure 7 , Figure 8 , Figure 9 , Figure 10 , the training procedure begins with a 20,000 step warm up using λ c = λ r = 0.1 \lambda_c = \lambda_r = 0.1 λ c = λ r = 0.1 . After this period, the inverse temperatures are held constant at values λ ∈ { 0.01 , 0.03 , 0.05 , 0.07 } \lambda\in\{0.01, 0.03, 0.05, 0.07\} λ ∈ { 0.01 , 0.03 , 0.05 , 0.07 } . We observe that without a meticulously chosen lambda schedule, not all the runs converged in the simulated 100,000 steps. For lower values of λ \lambda λ , more runs got stuck around the origin compared to the more consistent convergence obtained for larger λ \lambda λ s.
Figure 7. Training dynamics of the predictor with a 20k warm-up phase λ c = λ r = λ = 0.1 \lambda_c = \lambda_r = \lambda= 0.1 λ c = λ r = λ = 0.1 followed by constant inverse temperatures λ c = λ r = 0.01 \lambda_c = \lambda_r = 0.01 λ c = λ r = 0.01 . The solid curve represents the empirical median across 20 runs and the bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals (although not always visible). The parameters used are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 .
Figure 8. Training dynamics of the predictor with a 20k warm-up phase λ c = λ r = λ = 0.1 \lambda_c = \lambda_r = \lambda= 0.1 λ c = λ r = λ = 0.1 followed by constant inverse temperatures λ c = λ r = 0.03 \lambda_c = \lambda_r = 0.03 λ c = λ r = 0.03 . The solid curve represents the empirical median across 20 runs and the bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals (although not always visible). The parameters used are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 .
Figure 9. Training dynamics of the predictor with a 20k warm-up phase λ c = λ r = λ = 0.1 \lambda_c = \lambda_r = \lambda= 0.1 λ c = λ r = λ = 0.1 followed by constant inverse temperatures λ c = λ r = 0.05 \lambda_c = \lambda_r = 0.05 λ c = λ r = 0.05 . The solid curve represents the empirical median across 20 runs and the bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals (although not always visible). The parameters used are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 .
Figure 10. Training dynamics of the predictor with a 20k warm-up phase λ c = λ r = λ = 0.1 \lambda_c = \lambda_r = \lambda= 0.1 λ c = λ r = λ = 0.1 followed by constant inverse temperatures λ c = λ r = 0.07 \lambda_c = \lambda_r = 0.07 λ c = λ r = 0.07 . The solid curve represents the empirical median across 20 runs and the bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals (although not always visible). The parameters used are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 .
These full results empirically validate the need to employ a decreasing schedule for the inverse temperatures. A static λ \lambda λ forces a compromise between quickly escaping the saddle point at the origin and converging precisely at the global minima. A schedule dynamically adapts the landscape to first push the parameters away from the saddle point, and subsequently allows for fine-tuned convergence to the exact oracle parameters.
Gradient analysis for decoupled inverse temperatures
This subsection isolates the effects of decoupling the two inverse temperatures. We visualize the gradient vector field of the reduced risk R λ r , λ c < ( κ , ν ) R^{<}_{\lambda_r,\lambda_c}(\kappa,\nu) R λ r , λ c < ( κ , ν ) evaluated on the invariant manifold M \mathcal{M} M for multiple configurations of ( λ r , λ c ) (\lambda_r,\lambda_c) ( λ r , λ c ) .
Figure 11 , Figure 12 present the numerically computed gradients for various mixed combinations of λ c \lambda_c λ c and λ r \lambda_r λ r . By breaking the symmetry, we observe how the magnitude and direction of the gradients shift across the landscape. Specifically, two interesting phenomena appear:
Figure 11. (a) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ r = 0.1 \lambda_r = 0.1 λ r = 0.1 and λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_c \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } . (b) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ r = 0.1 \lambda_r = 0.1 λ r = 0.1 and λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_c \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } . (c) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ r = 0.01 \lambda_r = 0.01 λ r = 0.01 and λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_c \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } . (d) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ r = 0.01 \lambda_r = 0.01 λ r = 0.01 and λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_c \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ c ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } . Figure 11: Numerically computed gradients of the theoretical risk R λ r , λ c < ( κ , ν ) \mathcal{R}^{<}_{\lambda_r,\lambda_c}(\kappa,\nu) R λ r , λ c < ( κ , ν ) for λ r ∈ { 0.1 , 0.01 } \lambda_r \in\{0.1, 0.01\} λ r ∈ { 0.1 , 0.01 } and varying values of λ c \lambda_c λ c . The parameters used are M = N = 5 M = N = 5 M = N = 5 and ℓ = 100 \ell= 100 ℓ = 100 . The gradients, given by Equation (34) , are approximated using the Monte Carlo method with a sample size of 10,000 on a 100 × 100 100 \times100 100 × 100 discretization grid.
(a) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ c = 0.1 \lambda_c = 0.1 λ c = 0.1 and λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_r \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } .
(b) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ c = 0.1 \lambda_c = 0.1 λ c = 0.1 and λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_r \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } .
(c) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ c = 0.01 \lambda_c = 0.01 λ c = 0.01 and λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_r \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } .
(d) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ c = 0.01 \lambda_c = 0.01 λ c = 0.01 and λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } \lambda_r \in\{0.01, 0.03, 0.05, 0.07, 0.1\} λ r ∈ { 0.01 , 0.03 , 0.05 , 0.07 , 0.1 } .
Figure 12. Numerically computed gradients of the theoretical risk R λ r , λ c < ( κ , ν ) \mathcal{R}^{<}_{\lambda_r,\lambda_c}(\kappa,\nu) R λ r , λ c < ( κ , ν ) for λ c ∈ { 0.1 , 0.01 } \lambda_c \in\{0.1, 0.01\} λ c ∈ { 0.1 , 0.01 } and varying values of λ r \lambda_r λ r . The parameters used are M = N = 5 M = N = 5 M = N = 5 and ℓ = 100 \ell= 100 ℓ = 100 . The gradients, given by Equation (34) , are approximated using the Monte Carlo method with a sample size of 10,000 on a 100 × 100 100 \times100 100 × 100 discretization grid.
∂ ν R λ r , λ c < \partial_\nu R^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < behaves symmetrically with respect to both λ r \lambda_r λ r and λ c \lambda_c λ c ;
The same is not true for ∂ κ R λ r , λ c < \partial_\kappa R^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < . The shape of the gradient and most importantly of the 0 level curve of ∂ κ R < \partial_\kappa R^{<} ∂ κ R < varies significantly with the pairs ( λ r , λ c ) (\lambda_r,\lambda_c) ( λ r , λ c ) .
This analysis leads to another method of controlling the training dynamics, namely through the ratio τ = λ c / λ r \tau= \lambda_c/\lambda_r τ = λ c / λ r . Indeed, Figure 13 presents the optimization dynamics obtained for constant λ c = 0.01 \lambda_c = 0.01 λ c = 0.01 and λ r = 0.1 \lambda_r = 0.1 λ r = 0.1 . While 100,000 steps are not enough for convergence at the oracle parameters, the runs consistently move towards convergence. Moreover, the training dynamics are different from the ones studied in the main paper, as can be seen from the much faster increase of ∣ κ ∣ |\kappa| ∣ κ ∣ , closely mirroring the behavior of ∣ ν ∣ |\nu| ∣ ν ∣ .
Figure 13. Training dynamics of the predictor with constant λ c = 0.01 \lambda_c = 0.01 λ c = 0.01 and λ r = 0.1 \lambda_r = 0.1 λ r = 0.1 . The solid curve represents the empirical median across the runs and the bands represent the 0.05-0.95 and, respectively, the 0.25-0.75 quantile intervals (although not always visible). The parameters used are M = N = 5 M = N = 5 M = N = 5 , d = 20 d = 20 d = 20 , γ = 1 \gamma= 1 γ = 1 , ℓ = 100 \ell= 100 ℓ = 100 .
Gradient analysis for other hyperparameters N N N and ℓ \ell ℓ
In this section, we investigate the behavior of the partial derivatives of R < \mathcal{R}^{<} R < when varying N N N and ℓ \ell ℓ , for fixed λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 .
The values considered are not following the asymptotics required for Bayes optimality of the predictor, but better reflect real-world scenarios in which TSFMs are trained to take tens of channels and hundreds of time patches as input, while an individual token’s shift in the mean cannot be too large due to normalization layers. We restrict our analysis to exploring the gradient across multiple pairs of values ( N , ℓ ) (N,\ell) ( N , ℓ ) , as conducting full trainings for each pair would be computationally expensive.
As can be seen in Figure 14 , Figure 15 , Figure 16 , the effect of ℓ \ell ℓ overshadows that of N N N . Although for small ℓ \ell ℓ , a higher N N N appears to eliminate the problem of escaping the origin for κ \kappa κ , this disappears for larger ℓ \ell ℓ values.
Figure 14. (a) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 , ℓ = 10 \ell= 10 ℓ = 10 and N ∈ { 5 , 50 , 500 } N \in\{5, 50, 500\} N ∈ { 5 , 50 , 500 } . (b) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}\mathcal{R}^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 , ℓ = 10 \ell= 10 ℓ = 10 and N ∈ { 5 , 50 , 500 } N \in\{5, 50, 500\} N ∈ { 5 , 50 , 500 } . Numerically computed gradients of the theoretical risk R λ r , λ c < ( κ , ν ) \mathcal{R}^{<}_{\lambda_r,\lambda_c}(\kappa, \nu) R λ r , λ c < ( κ , ν ) for λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 , ℓ = 10 \ell= 10 ℓ = 10 and varying values of N N N . The gradients, given by Equation (34) , are approximated using the Monte Carlo method with a sample size of 10,000 on a 100 × \times × 100 discretization grid.
Figure 15. (a) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}R^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ r = λ c = 0.1 \lambda_r=\lambda_c=0.1 λ r = λ c = 0.1 , ℓ = 100 \ell=100 ℓ = 100 and N ∈ { 5 , 50 , 500 } N\in\{5,50,500\} N ∈ { 5 , 50 , 500 } . (b) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}R^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ r = λ c = 0.1 \lambda_r=\lambda_c=0.1 λ r = λ c = 0.1 , ℓ = 100 \ell=100 ℓ = 100 and N ∈ { 5 , 50 , 500 } N\in\{5,50,500\} N ∈ { 5 , 50 , 500 } . Figure 15: Numerically computed gradients of the theoretical risk R λ r , λ c < ( κ , ν ) R^{<}_{\lambda_r,\lambda_c}(\kappa,\nu) R λ r , λ c < ( κ , ν ) for λ r = λ c = 0.1 \lambda_r=\lambda_c=0.1 λ r = λ c = 0.1 , ℓ = 100 \ell=100 ℓ = 100 and varying values of N N N . The gradients, given by Equation (34) , are approximated using the Monte Carlo method with a sample size of 10,000 on a 100 × 100 100 \times100 100 × 100 discretization grid.
Figure 16. (a) Partial derivative ∂ κ R λ r , λ c < \partial_{\kappa}R^{<}_{\lambda_r,\lambda_c} ∂ κ R λ r , λ c < for λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 , ℓ = 1000 \ell= 1000 ℓ = 1000 and N ∈ { 5 , 50 , 500 } N \in\{5, 50, 500\} N ∈ { 5 , 50 , 500 } . (b) Partial derivative ∂ ν R λ r , λ c < \partial_{\nu}R^{<}_{\lambda_r,\lambda_c} ∂ ν R λ r , λ c < for λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 , ℓ = 1000 \ell= 1000 ℓ = 1000 and N ∈ { 5 , 50 , 500 } N \in\{5, 50, 500\} N ∈ { 5 , 50 , 500 } . Figure 16: Numerically computed gradients of the theoretical risk R λ r , λ c < ( κ , ν ) R^{<}_{\lambda_r,\lambda_c}(\kappa, \nu) R λ r , λ c < ( κ , ν ) for λ r = λ c = 0.1 \lambda_r = \lambda_c = 0.1 λ r = λ c = 0.1 , ℓ = 1000 \ell= 1000 ℓ = 1000 and varying values of N N N . The gradients, given by Equation (34) , are approximated using the Monte Carlo method with a sample size of 10,000 on a 100 × 100 discretization grid.
Furthermore, the higher ℓ \ell ℓ is, the higher the values of ∂ ν R < \partial_{\nu}R^{<} ∂ ν R < and the steeper its sign change around κ = 0 \kappa=0 κ = 0 .
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: We provide proofs for all the theoretical claims made and extensive numerical experiments both in the main text and in the supplemental material for all the empirically observed claims.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes] Justification: While not having a separate "Limitations" section in the paper, we discuss limitations of our work in the "Conclusions and limitations" section.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All the theoretical results and claims are accompanied by full proofs in the supplemental material.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the full code used for the numerical experiments in the paper. Furthermore, we provide the complete set of parameters for each experiment in the paper. Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is given as a resource in the supplemental materials.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: The hyperparameters used in all the experiments are listed in the supplemental material.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Empirical distributions are always plotted alongside the main curves.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Details regarding the compute resource are given in Appendix § 10 .
Guidelines:
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?
Answer: [Yes]
Justification: All the experiments are conducted using synthetically generated data. The authors don’t see any possible harm arising from the present research.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: The work in this paper is purely theory-oriented, aiming to better understand existing architectures. The authors see no societal impact related to the present paper.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A]
Justification: The authors don’t see any risk in the models used for this research.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [N/A]
Justification: No existing asset is used.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only asset released is code for the experiments in the main paper. The code is well documented in the opinion of the authors.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: LLMs were used only for writing, editing, or formatting purposes.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.