betaFeedback

WorksComputer Science

Spectral Bias Reduction in Local Gated ReLU Architectures

Abstract

Neural networks trained by gradient-based methods learn the low-frequency content of a target function faster than its high-frequency content. This spectral bias is well documented, but it is much less clear which architectural choices mitigate it and which cannot. We study the question for shallow ReLU models linear in their trainable parameters, where each mode is learned at a rate given by a Gram-matrix eigenvalue, and for a general symmetric bilinear form a(⋅,⋅)a(\cdot,\cdot), so that least-squares regression (a=a= the L2L^2 inner product) and variational or physics-informed losses (a=a= the H1H^1 energy form) are two instances of one analysis. Within this framework we propose gated local ReLU dictionaries, obtained by multiplying each ReLU feature by a hat partition-of-unity weight, in the spirit of partition-of-unity finite elements and overlapping Schwarz methods. The resulting Gram matrices are block-banded for any local bilinear form, with a block-Toeplitz interior whose matrix-valued symbol governs the spectrum. The two forms then behave very differently. For L2L^2 regression, the symbol is bounded away from zero, the ratio is uniform in the number of windows, and the measured spread of frequency-dependent learning rates falls from 1.9×1051.9\times10^{5} to 4.54.5 at matched model size. For the H1H^1 stiffness matrix, the symbol vanishes at zero frequency: gating leaves the Θ(N2)Θ(N^2) conditioning intact and inverts the frequency ordering instead of removing the bias. A preconditioner built from the same interior stencil will fix the spectral bias, showing that architecture alone is not enough. Experiments on 11D regression and Poisson problems confirm the dichotomy, and the regression effect persists when the hidden-layer parameters are trained as well.

Introduction

Neural networks trained by gradient-based optimization exhibit a pronounced spectral bias: they learn the low-frequency components of a target function substantially faster than the high-frequency ones [26, 5] (Xu, Zhang, Luo, Xiao and Ma 2020). The phenomenon has been observed across architectures, datasets, and tasks, and has been linked to generalization, training efficiency, the success of neural representations, and the well-known degradation of performance of physics-informed neural networks (PINNs) on multi-scale problems [30, 33, 34].

The dominant theoretical lens is the neural tangent kernel (NTK) [18]: in the infinite-width limit, training dynamics under gradient flow are governed by a fixed kernel whose eigenvalues decay rapidly with frequency, mechanically producing a low-frequency bias [5, 2, 4] (Du, Zhai, Poczos and Singh 2019). While powerful, this viewpoint offers limited architectural guidance: it explains why bias exists but is less direct about how to choose finite-width architectures that mitigate it. Architectural remedies in current practice – Fourier feature mappings, sinusoidal activations, polynomial nets, wavelet/KAN networks – are typically motivated by reshaping the NTK eigenspectrum by other means, often without finite-width spectral guarantees [30, 29, 21, 24]. In this work, we adopt a complementary, finite-dimensional viewpoint that puts the feature dictionary itself at the center of the analysis. We consider shallow approximations

fθ(x)=∑ℓ=1Nθℓφℓ(x),θ∈RN,(1)f_\theta(x) = \sum_{\ell=1}^{N} \theta_\ell\varphi_\ell(x), \qquad\theta\in\mathbb{R}^{N}, \tag*{(1)}

with ReLU-based features {φℓ}\{\varphi_\ell\}, trained by gradient descent on a quadratic loss. The model is simpler than a fully trainable neural network, but it captures the main mechanism of spectral bias in a transparent form: under gradient descent the error in the eigenbasis of the Gram feature matrix GG decays mode-by-mode at rates dictated by its eigenvalues [18, 23, 17]. Poor spectral conditioning of GG therefore directly translates into a large gap between the learning rates of the low- and high-frequency modes.

Our starting point is the recent observation by Hong et al. [17] that, for global ReLU feature dictionaries, the Gram matrix is closely related to a finite-element mass matrix and exhibits a quartic blow-up of its condition number with the number of features. This is precisely the source of the strong spectral bias of shallow ReLU networks. Motivated by classical ideas from numerical analysis – in particular the partition-of-unity finite element method [3] and overlapping Schwarz domain decomposition [12, 32] – we replace the global ReLU dictionary with a gated local variant in which each ReLU feature is multiplied by a partition-of-unity weight. The result is still linear in parameters, still expressible as a shallow ReLU-type network, but is spatially localized.

This change has a direct spectral consequence. The gated Gram matrix is block-banded, so its spectrum can be analyzed with the theory of block-Toeplitz matrices and their matrix-valued symbols [28, 7], a body of results that is standard in numerical linear algebra but little used in this context.

The symbol connects the architecture to the training dynamics. Its eigenvalues are indexed by a frequency, allowing us to obtain a learning rate for each frequency of the targeted function to be learned, and the ratio between the fastest and the slowest of these rates is a direct measure of spectral bias in the sense of Rahaman et al. [26]. Whether that ratio stays bounded as the network capacity grows, and what to do when it does not, is decided by the symbol alone.

Contributions. The contributions of this paper are as follows.

  1. A finite-dimensional, Gram-matrix-based framework for the spectral bias of linear-in-parameters shallow ReLU models, parameterized by a symmetric bilinear form a(⋅,⋅)a(\cdot,\cdot). Least-squares regression and variational or physics-informed losses are two instances of one analysis, which ties the frequency principle (Xu et al., 2020) and the activation-conditioning viewpoint of Hong et al. [17] to classical results on mass and stiffness matrices.

  1. A localized architecture, partition-of-unity gated ReLU, obtained by multiplying each ReLU feature by a hat partition of unity. It is still a shallow network, still linear in its trainable parameters, and its Gram matrix is block-banded whatever bilinear form the loss uses.

  1. A spectral characterization by block-Toeplitz analysis, and a bridge from it to the frequency principle. The theoretical argument is the same for every bilinear form, but the result depends on the symbol. Reading that symbol at the frequencies the gate grid supports gives a learning rate per frequency, whose extremes define a spectral-bias factor controlled by the symbol alone.

  1. A preconditioner can be naturally derived from the same symbol with the energy natural-gradient descent as its dense instance. Its construction shows that what the architecture cannot fix must be left to the optimizer.

Outline. Section 2 surveys spectral-bias and frequency-principle results, NTK-based analyses, architectural remedies, localized linear-in-parameters methods, and the partition-of-unity and block-Toeplitz background on which the framework draws. Section 3 sets up the linear-in-parameters model in the abstract bilinear-form setting, recalls how the Gram spectrum drives gradient descent, defines spectral bias formally, and recalls the global ReLU baseline. Section 4 introduces the PoU-gated dictionary and establishes its bilinear-form-independent block-banded structure. Section 5 develops the spectral framework: a scaling lemma, a block-Toeplitz decomposition with a matrix-valued symbol, a frequency-resolved reading of that symbol, and the two specializations to L2L^2 regression (where gating removes the growth of spectral bias with NN) and to the H1H^1 stiffness matrix (where it does not). Section 6 derives the preconditioner needed in the H1H^1. Section 7 sets up the experiments, fixes the diagnostic and reports the results in both regimes, and Section 8 discusses the implications and the pathways to deeper and higher-dimensional architectures.

Related work

Spectral bias and the frequency principle. The empirical observation that neural networks fit low-frequency content was made first independently by Rahaman et al. [26], who introduced the term spectral bias, and by Xu, Zhang and Xiao (2019); Xu et al. (2020), who phrased it as the frequency principle (F-PRINCIPLE). Both lines of work demonstrated, across architectures and datasets, that the relative error in Fourier space decays from low-frequency to high-frequency during training, often spanning several orders of magnitude in time scale. A recent overview is given by Xu, Zhang and Luo (2025). The phenomenon is now widely viewed as one of the principal implicit regularization mechanisms of gradient-based deep learning.

NTK-based analyses. The dominant theoretical framework for spectral bias is the neural tangent kernel (NTK) of Jacot et al. [18]. In the infinite-width limit, training a network by gradient flow is equivalent to kernel regression with the NTK; consequently, the modes of the target are learned at rates equal to the NTK eigenvalues. Basri et al. [5, 4] computed the NTK spectrum on the sphere and on non-uniform data and proved frequency-dependent convergence rates that decay polynomially (uniform data) or worse (non-uniform data). Arora et al. [2]; Du et al. (2019); Cao, Fang, Wu, Zhou and Gu [8] gave fine-grained convergence and generalization bounds in the same regime. These analyses establish the existence and the severity of spectral bias, but they describe a fixed kernel, and they say little about how a finite-width architecture should be modified to alleviate it. We take a complementary path: we work at finite width, with a fixed feature dictionary, and tie the bias to the symbol of a structured matrix that the architecture controls directly.

Architectural remedies. A substantial line of work seeks to mitigate spectral bias by reshaping the input or activation space. Fourier feature mappings (Tancik et al. [30]) and sinusoidal activations (Sitzmann et al. [29]) concentrate the mass of the kernel on a tunable frequency band and have become standard for implicit neural representations and PINNs. Kolmogorov–Arnold networks (KANs) and their wavelet variants (Liu et al. [21]; Meshir et al. [24]) control the frequency content through learned univariate basis functions, and their spectral behavior has itself been studied (Wang, Siegel, Liu and Hou [35]). A complementary line of work keeps the architecture fixed and shows how the choice of pointwise nonlinearity shapes the bias: Sah, Pyle, Anselmi and Patel (2025) derive, in the kernel regime, the spectral seminorm that a broad class of activations implicitly penalizes. For PINNs and neural operators specifically, Khodakaram, Oommen, Ahmadi Daryakenari, Beekenkamp and Karniadakis (2026) give a systematic empirical study of how architecture, activation, loss design and optimizer interact to produce spectral bias, and extract mitigation guidelines from it; the present paper can be read as supplying, for one architecture class, the closed-form spectral statement behind guidelines of that kind. Wang et al. [33, 34] analyze Fourier-feature PINNs through their NTK and propose multi-scale architectures specifically tailored to PDE problems. All of these approaches operate on the kernel/NTK side of the picture; the present work instead targets the Gram matrix of features of a shallow ReLU model and exploits classical ideas from approximation theory and conditioning of structured matrices.

Activation- and feature-conditioning viewpoints. The closest precursor to the present work is the activation-function study of Hong et al. [17], who isolate the role of the activation by analyzing least-squares problems for shallow networks with global features. They show that the ReLU mass matrix has a condition number scaling like N4N^4 in the number of features and that other activations (e.g. ReLUp^p, smoothed variants) yield more favorable spectra and reduce spectral bias. We use their global-ReLU result as a baseline and extend the conditioning program to a fundamentally different design choice, based on spatial gating, rather than a different point-wise nonlinearity. A related continuum-of-features point of view was developed by Ma, Wu et al. [22]; Mei et al. [23].

Localized linear-in-parameters methods for PDEs. The dictionary studied here is linear in its trainable parameters, and this places it close to a family of methods that solve PDEs by least squares on a fixed, randomly drawn or hand-designed feature set. Dong and Li [13] combine extreme learning machines with a domain decomposition, representing the solution on each subdomain by a local network whose hidden parameters are frozen and imposing continuity across subdomain interfaces; Chen, Chi, E and Yang [10] propose the random feature method, in which random features, a collocation treatment of the PDE and a penalty treatment of the boundary conditions are combined into a single least-squares problem. In both cases the discrete system is explicit, and its conditioning is therefore open to direct analysis, which is the property we exploit here. Our construction differs from these in that the locality is imposed by a partition of unity multiplying each feature, rather than by assigning separate networks to separate subdomains, so that the global dictionary stays conforming and no interface conditions are needed.

Partition of unity, domain decomposition, and block-Toeplitz spectra. Partition-of-unity methods [3] and overlapping Schwarz domain decomposition [12, 32] are classical tools in numerical analysis, where they are used to combine local approximation spaces into global ones with controlled overlap. Their spectral signatures are well understood through the theory of block-banded Toeplitz matrices and their matrix-valued symbols [31, 28, 7]. Although these tools are ubiquitous in the literature on numerical linear algebra and finite-elements, their use in the analysis of neural networks is much less common. Recent work on finite-basis PINNs (Moseley, Markham and Nissen-Meyer, 2023) introduces overlapping subdomain networks for PDEs, and Heinlein and Kapoor [16] observe that such a decomposition makes the otherwise dense Gauss–Newton system block-sparse, but neither provides a spectral analysis of the resulting matrices. Our contribution is to bring the block-Toeplitz machinery into the realm of the Gram matrix of a gated shallow ReLU network and to use it to derive sharp, spectral statements that translate directly into spectral-bias guarantees.

Domain-decomposed and random-feature architectures for SciML. The gated dictionary studied here is a linear-in-parameters relative of a family of domain-decomposed SciML architectures developed in earlier work of the author and her collaborators. Multilevel domain-decomposition-based architectures for PINNs [11] showed that combining overlapping subdomain networks with coarser levels restores scalability on high-frequency and multi-scale problems, mirroring the classical two-level Schwarz argument. Anderson, Dolean, Moseley and Pestana [1] replaced the subdomain networks with random features trained by least squares, which makes the resulting discrete systems – and hence their conditioning – explicit, and van Beek, Dolean and Moseley [6] showed that filtering the local feature sets is what keeps such domain-decomposed random-feature methods well conditioned as the decomposition is refined. These works give empirical evidence that localization, combined with an appropriate treatment of the coarse, low-frequency component, is what controls the conditioning of the resulting systems. What they do not provide, and what we aim at here, is a closed-form account of why that is so, and of the cases in which localization cannot help.

Problem formulation

Let D=[0,1]D=[0,1] and let f⋆∈L2(D)f^\star\in L^2(D) be a target function. We approximate f⋆f^\star by a model that is linear in trainable parameters,

fθ(x)=∑ℓ=1Nθℓφℓ(x),θ∈RN,(2)f_\theta(x) = \sum_{\ell=1}^{N}\theta_\ell\varphi_\ell(x), \qquad\theta\in\mathbb{R}^{N}, \tag*{(2)}

where the dictionary {φℓ}ℓ=1N⊂L2(D)\{\varphi_\ell\}_{\ell=1}^{N} \subset L^2(D) is fixed (its parameters, e.g. ReLU thresholds, are not trained). We work both in the continuous setting, with inner product ⟨g,h⟩:=∫01g(x)h(x) dx\langle g,h\rangle:= \int_0^1 g(x)h(x)\,\mathrm{d}x, and in an empirical setting with samples x1,…,xn∈Dx_1,\ldots,x_n \in D and ⟨g,h⟩n:=1n∑i=1ng(xi)h(xi)\langle g,h\rangle_n := \frac{1}{n}\sum_{i=1}^{n}g(x_i)h(x_i).

The training loss is the squared L2L^2 error,

J(θ)=12∥fθ−f⋆∥2,(3)J(\theta) = \frac{1}{2}\lVert f_\theta-f^\star\rVert^2, \tag*{(3)}

where ∥⋅∥\lVert\cdot\rVert stands for either the continuous L2(D)L^2(D) norm or its empirical version. Writing σ(z):=max⁡{z,0}\sigma(z):=\max\{z,0\} for the ReLU activation, our running concrete examples are:

  • the global ReLU dictionary φℓglob(x)=σ(x−sℓ)\varphi_\ell^{\mathrm{glob}}(x)=\sigma(x-s_\ell), with knots 0≤s1<⋯<sN<10\le s_1<\cdots<s_N<1;

  • the gated local ReLU dictionary introduced in § 4, where each σ(x−sℓ)\sigma(x-s_\ell) is multiplied by a partition-of-unity weight.

Remark 1. Although (2) is linear in θ\theta, this is not a degenerate model. With ReLU features it spans a large piecewise-linear function class, and with a fine knot grid it is a universal approximator on DD. The linear-in-θ\theta choice is what makes the spectral analysis tractable; it is also exactly the regime captured by the NTK limit of a wide network whose inner weights are frozen at initialization, so the conclusions transfer naturally to the weak-training regime of overparameterized networks.

Gram matrix and gradient-descent spectral filter. Define the Gram matrix G∈RN×NG \in\mathbb{R}^{N \times N} and the moment vector b∈RNb \in\mathbb{R}^{N} of the dictionary by

Gℓℓ′:=⟨φℓ,φℓ′⟩,bℓ:=⟨f⋆,φℓ⟩.(4)G_{\ell\ell'} := \langle\varphi_{\ell}, \varphi_{\ell'} \rangle,\qquad b_{\ell} := \langle f^\star, \varphi_{\ell} \rangle. \tag*{(4)}

A direct expansion of (3) gives J(θ)=12θTGθ−bTθ+cJ(\theta) = \frac{1}{2}\theta^\mathsf{T}G\theta- b^\mathsf{T}\theta+ c, with c=12∥f⋆∥2c = \frac{1}{2}\lVert f^\star\rVert^2. The matrix GG is symmetric positive semidefinite, and positive definite whenever the features are linearly independent in the chosen inner product.

Gradient descent with step size h>0h > 0 on (3) reads θ(ℓ+1)=θ(ℓ)−h(Gθ(ℓ)−b)\theta^{(\ell+1)} = \theta^{(\ell)} - h(G\theta^{(\ell)} - b). If GG is positive definite, with unique minimizer θ⋆=G−1b\theta^\star= G^{-1}b, the error e(ℓ):=θ(ℓ)−θ⋆e^{(\ell)} := \theta^{(\ell)} - \theta^\star satisfies the linear iteration

e(ℓ+1)=(I−hG)e(ℓ).(5)e^{(\ell+1)} = (I - hG)e^{(\ell)}. \tag*{(5)}

Lemma 1 (Spectral filter of gradient descent). Let G=UΛUTG = U\Lambda U^\mathsf{T} be the eigendecomposition of GG with Λ=diag⁡(λ1,…,λN)\Lambda= \operatorname{diag}(\lambda_1,\ldots,\lambda_N) and λ1≥⋯≥λN≥0\lambda_1 \geq\cdots\geq\lambda_N \geq0, and orthonormal eigenvectors {ur}\{u_r\}. Writing e(ℓ)=∑r=1Nαr(ℓ)ure^{(\ell)} = \sum_{r=1}^{N}\alpha_r^{(\ell)}u_r, the iteration (5) yields

αr(ℓ)=(1−hλr)ℓαr(0),r=1,…,N.(6)\alpha_r^{(\ell)} = (1-h\lambda_r)^\ell\alpha_r^{(0)},\qquad r=1,\ldots,N. \tag*{(6)}

If 0<h≤2/λ10 < h \leq2/\lambda_1 and GG is positive definite, then J(θ(ℓ))→J(θ⋆)J(\theta^{(\ell)}) \to J(\theta^\star).

The same statement holds in continuous time. The gradient flow of JJ,

θ˙(t)=−∇J(θ(t))=−(Gθ(t)−b),θ(0)=θ(0),(7)\dot{\theta}(t) = -\nabla J(\theta(t)) = -(G\theta(t)-b),\qquad\theta(0)=\theta^{(0)}, \tag*{(7)}

gives e˙=−Ge\dot{e}=-Ge for the error, and hence, in the same eigenbasis,

αr(t)=e−tλrαr(0),r=1,…,N.(8)\alpha_r(t) = e^{-t\lambda_r}\alpha_r(0),\qquad r=1,\ldots,N. \tag*{(8)}

Both filters—the discrete (1−hλr)ℓ(1-h\lambda_r)^\ell of (6) and the continuous e−tλre^{-t\lambda_r} of (8)—are monotone in λr\lambda_r and are governed by the same spectrum. We state results for the flow (7) from here onwards: the main reason is that no statement about learning rates is affected by the choice of hh, and ℓ\ell steps of gradient descent at the largest admissible step h=1/λ1h=1/\lambda_1 correspond to t=ℓ/λ1t=\ell/\lambda_1. Every conclusion transfers to the discrete iteration through (6).

Lemma 1 encodes exactly the link between the Gram spectrum and training dynamics that we will exploit throughout the paper: each eigenmode of GG is a direction in feature space that is learned at a rate λr\lambda_r, so a wide spread of {λr}\{\lambda_r\} creates a wide spread of learning timescales. The spectral-bias question becomes a question about spec⁡(G)\operatorname{spec}(G).

A working definition of spectral bias. To make the link between Lemma 1 and “learning low frequencies first” precise, one needs an appropriate notion of frequency. We use the eigenbasis of a kernel or Gram operator, which generalizes Fourier modes both in the continuum and on finite datasets. On R\mathbb{R}, frequency is naturally indexed by the Fourier transform; equivalently, by the eigenfunctions of a smoothing operator. The Gaussian convolution

(Tsf)(x):=(s/2π)1/2∫Re−s2(x−y)2f(y) dy(T_s f)(x) := \left(s/2\pi\right)^{1/2}\int_{\mathbb{R}} e^{-\frac{s}{2}(x-y)^2}f(y)\,\mathrm{d}y

has Fourier modes ϕω(x)=eiωx\phi_\omega(x)=e^{i\omega x} as eigenfunctions, with eigenvalues e−ω2/(2s)e^{-\omega^2/(2s)} that decay rapidly in ∣ω∣|\omega|. On bounded domains the eigenfunctions of the Laplacian or of a compact integral operator with smooth kernel give an analogous hierarchy, ordering modes from smooth (large eigenvalues) to oscillatory (small eigenvalues).

Data-driven frequencies via Gram matrices. On a finite dataset D={x1,…,xn}\mathcal{D}=\{x_1,\ldots,x_n\}, Fourier modes are not available, but the dictionary itself induces the kernel k(x,x′):=∑ℓ=1Nφℓ(x)φℓ(x′)k(x,x') := \sum_{\ell=1}^{N}\varphi_\ell(x)\varphi_\ell(x') whose matrix representation (on samples or in L2L^2) is the Gram matrix GG. The eigenvectors of GG define orthogonal modes of approximation; following Hong et al. [17], we adopt these as the relevant “frequencies” of the model.

Definition 1 (Spectral bias, in our setting). Let K\mathcal{K} be a symmetric positive semidefinite operator (a kernel or a Gram matrix) with eigenpairs Kϕr=μrϕr\mathcal{K}\phi_r=\mu_r\phi_r, μ1≥μ2≥⋯≥μN≥0\mu_1\ge\mu_2\ge\cdots\ge\mu_N\ge0. Expand a target f⋆=∑rar⋆ϕrf^\star=\sum_r a_r^\star\phi_r and the model output ft=∑rar(t)ϕrf_t=\sum_r a_r(t)\phi_r at training time tt. We say the dynamics exhibits spectral bias if ∣ar(t)−ar⋆∣\lvert a_r(t)-a_r^\star\rvert decays faster for modes with larger μr\mu_r than for modes with smaller μr\mu_r.

In the linear-in-parameters regime, Lemma 1 shows that gradient descent realizes exactly the spectral filter (1−hμr)ℓ(1-h\mu_r)^\ell in the Gram eigenbasis, and Definition 1 shows that spectral bias is controlled by spec⁡(G)\operatorname{spec}(G): it is severe when the μr\mu_r span many orders of magnitude, and mild when they are clustered. This is a first and perhaps superficial reading: it measures the spread of the whole spectrum, whereas what a practitioner observes is the evolution of the frequencies present in the target. § 5.3 refines the definition accordingly.

The global ReLU baseline. We briefly recall the conditioning of the standard global ReLU dictionary, which serves as our reference. With uniform knots sℓ=(ℓ−1)/Ns_\ell=(\ell-1)/N on [0,1][0,1], the continuous Gram matrix Gℓℓ′glob=∫01σ(x−sℓ)σ(x−sℓ′) dxG^{\mathrm{glob}}_{\ell\ell'}=\int_0^1\sigma(x-s_\ell)\sigma(x-s_{\ell'})\,\mathrm{d}x has explicit entries that are cubic in min⁡(sℓ,sℓ′)\min(s_\ell,s_{\ell'}) and is closely related to a finite-element mass matrix on a uniform grid ([17]).

Proposition 2 (Global ReLU spectrum, after Hong et al. [17]). For the uniform-knot global ReLU dictionary on [0,1][0,1] with NN features, the eigenvalues of GglobG^{\mathrm{glob}} satisfy

λN(Gglob)λj(Gglob)≍(Nj)4,j=1,…,N,(9)\frac{\lambda_N(G^{\mathrm{glob}})}{\lambda_j(G^{\mathrm{glob}})}\asymp\left(\frac{N}{j}\right)^4,\qquad j=1,\ldots,N, \tag*{(9)}

up to constants independent of NN. In particular, the condition number satisfies κ(Gglob)≍N4\kappa(G^{\mathrm{glob}})\asymp N^4.

Remark 2. The quartic blow-up in (9) is the central reason why shallow ReLU networks exhibit strong spectral bias: doubling NN shrinks the smallest non-zero eigenvalue by a factor of ∼16\sim16, so the slowest-learning mode becomes 16×16\times slower (per gradient-descent iteration) than before.

Gated local ReLU dictionaries

The architecture we propose retains ReLU activations but replaces the global feature dictionary by a spatially localized variant obtained by multiplying each ReLU feature by a partition-of-unity (PoU) weight. The construction is one-dimensional throughout this paper; we discuss tensor-product and higher-dimensional extensions in § 8.

Hat partition of unity. Fix an integer K≥2K\ge2 and define the uniform grid xk:=k/Kx_k:=k/K, k=0,…,Kk=0,\ldots,K, with mesh size h:=1/Kh:=1/K. The standard hat partition of unity on this grid consists of K+1K+1 piecewise-linear functions {ψk}k=0K\{\psi_k\}_{k=0}^K. Interior hats (1≤k≤K−11\le k\le K-1) are supported on [xk−1,xk+1][x_{k-1},x_{k+1}] and rise linearly from 00 at the endpoints to 11 at xkx_k; boundary hats ψ0,ψK\psi_0,\psi_K are the obvious one-sided versions. By construction

ψk≥0,∑k=0Kψk(x)=1for all x∈[0,1].(10)\psi_k\ge0,\qquad\sum_{k=0}^{K}\psi_k(x)=1\quad\text{for all }x\in[0,1]. \tag*{(10)}

Each ψk\psi_k overlaps only with its two nearest neighbors ψk±1\psi_{k\pm1}, a property that drives the entire spectral analysis.

Gated local ReLU dictionary. For each PoU index k∈{0,…,K}k\in\{0,\ldots,K\} we choose mm knots sk,1,…,sk,m∈supp⁡(ψk)s_{k,1},\ldots,s_{k,m}\in\operatorname{supp}(\psi_k) and define the gated local ReLU features

φk,j(x):=ψk(x)σ(x−sk,j),k=0,…,K,j=1,…,m,(11)\varphi_{k,j}(x):=\psi_k(x)\sigma(x-s_{k,j}),\qquad k=0,\ldots,K,\quad j=1,\ldots,m, \tag*{(11)}

giving a dictionary of total size N=(K+1)mN=(K+1)m. The associated shallow model is

fθgate,K(x)=∑k=0K∑j=1mθk,jφk,j(x),θ∈R(K+1)m.(12)f^{\mathrm{gate},K}_{\theta}(x)=\sum_{k=0}^{K}\sum_{j=1}^{m}\theta_{k,j}\varphi_{k,j}(x),\qquad\theta\in\mathbb{R}^{(K+1)m}. \tag*{(12)}

Like the global ReLU model of § 3, (12) is linear in the trainable parameters θ\theta; the inner parameters (knots sk,js_{k,j} and gate widths) are fixed by the architecture. The model can be implemented as a shallow network with a hand-crafted first layer that produces gated ReLU features and a linear second layer with weights θ\theta.

Remark 3 (Gating is not just a different activation). Replacing σ\sigma by a different scalar nonlinearity (e.g. ReLUp^{p} or hat-shaped activations) changes the smoothness of features but not their support. Gating, in contrast, leaves the activation untouched and changes only the support of each feature. As we show next, this is precisely the property that makes the Gram matrix block-banded and amenable to block-Toeplitz analysis. The activation-function and gating viewpoints are therefore complementary: the former reshapes within-feature behavior, the latter reshapes between-feature interactions.

Block-banded Gram structure. Define the gated Gram matrix

G(k,j),(k′,j′)gateK:=⟨φk,j,φk′,j′⟩.(13)G^{\mathrm{gate}K}_{(k,j),(k',j')} := \langle\varphi_{k,j},\varphi_{k',j'}\rangle. \tag*{(13)}

Because supp⁡(ψk)∩supp⁡(ψk′)=∅\operatorname{supp}(\psi_k) \cap\operatorname{supp}(\psi_{k'}) = \varnothing whenever ∣k−k′∣>1|k-k'|>1, the integrand in (13) vanishes identically beyond nearest neighbors. Ordering the unknowns gate by gate (all jj for k=0k=0, then all jj for k=1k=1, etc.) yields the following.

Proposition 3 (Block-banded structure). The Gram matrix (13) has the block-tridiagonal form

GgateK=(B0C00⋯0C0TB1C1⋱⋮0⋱⋱⋱0⋮⋱CK−2TBK−1CK−10⋯0CK−1TBK),(14)G^{\mathrm{gate}K} = \begin{pmatrix} B_0 & C_0 & 0 & \cdots& 0 \\ C_0^{\mathsf T} & B_1 & C_1 & \ddots& \vdots\\ 0 & \ddots& \ddots& \ddots& 0 \\ \vdots& \ddots& C_{K-2}^{\mathsf T} & B_{K-1} & C_{K-1} \\ 0 & \cdots& 0 & C_{K-1}^{\mathsf T} & B_K \end{pmatrix}, \tag*{(14)}

where each diagonal block Bk∈Rm×mB_k \in\mathbb{R}^{m\times m} encodes within-gate inner products and each off-diagonal block Ck∈Rm×mC_k \in\mathbb{R}^{m\times m} encodes the overlap between gates kk and k+1k+1. All other blocks are zero.

By contrast, the matrix GglobG^{\mathrm{glob}} of § 3 is dense: its (j,k)(j,k) entry is nonzero for every pair of indices. The block-banded form (14) has only O((K+1)m2)O((K+1)m^2) nonzero entries instead of O((Km)2)O((Km)^2).

Connection to finite elements and domain decomposition. The structure of (14) is familiar from numerical analysis. The PoU weights {ψk}\{\psi_k\} are exactly the standard P1P_1 finite-element basis on the uniform grid, and the gated features (11) can be viewed as enrichments of this P1P_1 basis by ReLU functions in the spirit of the partition-of-unity finite-element method [3]. The block-banded Gram matrix (14) is, mutatis mutandis, the mass matrix of an enriched P1P_1 space; its spectral conditioning is governed by exactly the tools used in classical numerical analysis. The next section makes this connection quantitative.

Spectral analysis: a unified framework

The block-banded structure of the gated Gram matrix derived in § 4 is purely a support property: it is inherited by any inner product on the dictionary, regardless of whether one weighs interactions by function values or by derivatives. In contrast, the eigenvalue scaling depends sharply on which inner product is used. We therefore analyze the gated Gram matrix in a unified way, parameterized by a symmetric bilinear form a(⋅,⋅)a(\cdot,\cdot). Two specializations are of central interest:

  • the L2L^2 inner product, which arises in least-squares regression on a target f∗f^*;

  • the H1H^1 semi-inner-product a(u,v):=∫01u′(x)v′(x) dxa(u,v):=\int_0^1 u'(x)v'(x)\,\mathrm{d}x, which arises in Galerkin discretizations of variational PDEs and in physics-informed losses.

The block-Toeplitz argument applies to both. The conclusions do not: gating bounds spectral bias uniformly in KK in the regression case, but it does not suffice in the stiffness case, which is what motivates the preconditioners of § 6.

Quadratic losses and the Gram matrix of a bilinear form

Let V⊂L2([0,1])V\subset L^2([0,1]) be a function space (typically L2L^2 or H01H_0^1) and a:V×V→Ra:V\times V\to\mathbb{R} a continuous symmetric bilinear form, positive semidefinite on VV. Given ℓ∈V′\ell\in V', the variational problem reads

θ∗=arg⁡min⁡θ∈RN Ja(θ),Ja(θ):=12a(fθ,fθ)−ℓ(fθ)(15)\theta^*=\underset{\theta\in\mathbb{R}^N}{\arg\min}\ J^a(\theta), \qquad J^a(\theta):=\frac{1}{2}a(f_\theta,f_\theta)-\ell(f_\theta) \tag*{(15)}

where fθf_\theta is given by (2) in the global case and (12) in the gated case. A direct expansion gives

Ja(θ)=12θTGaθ−bTθ+const,J^a(\theta) = \frac{1}{2}\theta^\mathsf{T}G^a\theta- b^\mathsf{T}\theta+ \mathrm{const},

with the Gram matrix of aa

Gℓℓ′a:=a(φℓ,φℓ′),bℓ:=ℓ(φℓ).(16)G^a_{\ell\ell'} := a(\varphi_\ell,\varphi_{\ell'}), \qquad b_\ell:= \ell(\varphi_\ell). \tag*{(16)}

Gradient descent on (15) obeys exactly the same iteration as before, e(ℓ+1)=(I−hGa)e(ℓ)e^{(\ell+1)} = (I-hG^a)e^{(\ell)}, with the more general GaG^a in place of the L2L^2 Gram matrix. Lemma 1 thus generalizes verbatim: each eigenmode of GaG^a is learned at a rate equal to the corresponding eigenvalue, and the spectral-bias question becomes a question about spec⁡(Ga)\operatorname{spec}(G^a).

The two specializations we consider are:

Regression (a=⟨⋅,⋅⟩L2a=\langle\cdot,\cdot\rangle_{L^2}). The form

aL2(u,v):=∫01uv dxa^{L^2}(u,v) := \int_0^1 uv\,\mathrm{d}x

gives Ga,L2=GglobG^{a,L^2}=G^{\mathrm{glob}} or GgateKG^{\mathrm{gate}K} as in Sections 3 and 4; this is the standard mass matrix. The linear functional is ℓ(v)=⟨f⋆,v⟩L2\ell(v)=\langle f^\star,v\rangle_{L^2}, recovering the L2L^2 regression loss (3).

Variational PDEs (a=a= energy form). For the model 1D Poisson problem −u′′=f-u^{\prime\prime}=f on (0,1)(0,1) with homogeneous Dirichlet conditions, the Galerkin formulation seeks u∈H01u\in H^1_0 such that

aH1(u,v):=∫01u′v′ dx=∫01fv dxa^{H^1}(u,v) := \int_0^1 u'v'\,\mathrm{d}x = \int_0^1 fv\,\mathrm{d}x

for all v∈H01v\in H^1_0. Restricting to the gated dictionary gives the stiffness matrix

Aℓℓ′gate,K:=∫01φℓ′(x)φℓ′′(x) dx.(17)A^{\mathrm{gate},K}_{\ell\ell'} := \int_0^1 \varphi'_\ell(x)\varphi'_{\ell'}(x)\,\mathrm{d}x. \tag*{(17)}

which is the relevant Gram matrix for the variational formulation. The same matrix appears in physics-informed losses for second-order PDEs, where it controls the Hessian of the energy term [34] (Moseley et al. 2023).

The block-banded structure of Proposition 3 holds for any bilinear form whose kernel is local enough that integrands vanish outside the union of supports: in particular, both L2L^2 and H1H^1 forms qualify, since each has only the support of φℓ⋅φℓ′\varphi_\ell\cdot\varphi_{\ell'} (resp. $\varphi'_\ell\cdot\varphi'_{\ell'}) as its domain of integration. We can therefore conduct the entire block-Toeplitz analysis once, in the abstract setting, and read off the consequences for each specialization.

Universal K−αK^{-\alpha} scaling lemma. Under the uniform setup of Section 4 (gates ψk(x)=ψ^((x−xk−1)/h)\psi_k(x)=\hat{\psi}((x-x_{k-1})/h) and knots sk,r=xk−1+hτrs_{k,r}=x_{k-1}+h\tau_r), a change of variables on each window extracts a common power of hh from every entry of the Gram matrix. The exponent depends only on the homogeneity of the integrand, so the argument covers any bilinear form of the form below.

Lemma 4 (K−αK^{-\alpha} scaling, abstract version). Let a(u,v)=∫01κ(u,v) dxa(u,v)=\int_0^1\kappa(u,v)\,\mathrm{d}x where κ\kappa is a (symmetric, bilinear) local pointwise functional homogeneous of degree β\beta in scalings u↦λuu\mapsto\lambda u. Assume that κ(φk,r,φk′,s)\kappa(\varphi_{k,r},\varphi_{k',s}), evaluated at gated features (11), has homogeneity −δ-\delta in hh relative to κ(φ^r,φ^s)\kappa(\hat{\varphi}_r,\hat{\varphi}_s) on the reference window [0,1][0,1]. Then there exists a dimensionless symmetric matrix G~aK,m\widetilde{G}^{K,m}_a, independent of KK, such that

Ga,gate K,m=K−αG~aK,m,α=1+β−δ.(18)G^{a,\mathrm{gate}\,K,m} = K^{-\alpha}\widetilde{G}^{K,m}_a,\qquad\alpha=1+\beta-\delta. \tag*{(18)}

The two cases of interest yield:

  • L2L^2 form a(u,v)=∫uv dxa(u,v)=\int uv\,\mathrm{d}x: each φk,r=ψkσ(x−sk,r)\varphi_{k,r}=\psi_k\sigma(x-s_{k,r}) contributes one factor of hh from the ReLU ramp under rescaling and one factor of hh from dx\mathrm{d}x; total α=3\alpha=3.

  • H1H^1 form a(u,v)=∫u′v′ dxa(u,v)=\int u'v'\,\mathrm{d}x: differentiation removes one factor of hh from each feature (the chain rule on ψ^′\hat{\psi}' contributes h−1h^{-1}, balancing the explicit hh); arithmetic gives α=1\alpha=1.

Proof. On window IkI_k, change variables x=xk−1+hξx=x_{k-1}+h\xi. For L2L^2, extracts h3h^3. For H1H^1, the derivative of a gated feature is φk,r′(x)=h−1ψ^′(ξ)(x−sk,r)+ψ^(ξ)Θ(x−sk,r)\varphi'_{k,r}(x)=h^{-1}\hat{\psi}'(\xi)(x-s_{k,r})+\hat{\psi}(\xi)\Theta(x-s_{k,r}), where Θ\Theta is the Heaviside function. The first term has explicit hh from the ReLU, divided by the chain-rule h−1h^{-1}, hence O(1)O(1) in hh; the second term is O(1)O(1) in hh as well. The product φk,r′φk′,s′\varphi'_{k,r}\varphi'_{k',s} is therefore O(1)O(1) in hh, and dx=h dξ\mathrm{d}x=h\,\mathrm{d}\xi contributes the only factor of hh. Total: α=1\alpha=1. □

The exponents α=3\alpha=3 (L2L^2) and α=1\alpha=1 (H1H^1) are well-known in finite-element analysis and reflect the standard mass-versus-stiffness scaling on a uniform mesh. Here they enter the spectral story directly: they are the rate at which all blocks of the gated Gram matrix shrink as K→∞K\to\infty.

Block-Toeplitz decomposition

The interior of the rescaled matrix G~aK,m:=KαGa,K,ma,gate\widetilde{G}^{K,m}_a := K^\alpha G^{a,\mathrm{gate}}_{a,K,m} is, by translation invariance of the uniform setup, a block-Toeplitz matrix with constant block-stencil (CT,B,C)(C^{\mathsf{T}}, B, C), where B,C∈Rm×mB, C \in\mathbb{R}^{m \times m} are determined by the gate profile, the knot pattern and the bilinear form aa, and are independent of KK. The boundary blocks differ because ψ0,ψK\psi_0, \psi_K are one-sided; the resulting correction has rank at most 4m4m.

Theorem 5 (Block-Toeplitz decomposition, abstract form). Under the assumptions of Lemma 4, define TK+1∈R(K+1)m×(K+1)mT_{K+1} \in\mathbb{R}^{(K+1)m \times(K+1)m} as the block-tridiagonal Toeplitz matrix with stencil (CT,B,C)(C^{\mathsf{T}}, B, C). Then

G~aK,m=TK+1+RK,m,rank⁡(RK,m)≤4m,∥RK,m∥≤MR,(19)\widetilde{G}^{K,m}_a = T_{K+1} + R_{K,m}, \qquad\operatorname{rank}(R_{K,m}) \le4m, \qquad\lVert R_{K,m} \rVert\le M_R, \tag*{(19)}

with MRM_R independent of KK. The Hermitian symbol

fa(ζ):=CTe−iζ+B+Ceiζ,(20)f_a(\zeta) := C^{\mathsf{T}}e^{-i\zeta} + B + Ce^{i\zeta}, \tag*{(20)}

has mm real eigenvalue branches λ1(fa(ζ))≤⋯≤λm(fa(ζ))\lambda_1(f_a(\zeta)) \le\cdots\le\lambda_m(f_a(\zeta)). With mfa:=min⁡ζλ1m_{f_a} := \min_{\zeta}\lambda_1 and Mfa:=max⁡ζλmM_{f_a} := \max_{\zeta}\lambda_m,

spec⁡(TK+1)⊂[mfa,Mfa],spec⁡(G~aK,m)⊂[mfa−MR,Mfa+MR].(21)\operatorname{spec}(T_{K+1}) \subset[m_{f_a}, M_{f_a}], \qquad\operatorname{spec}(\widetilde{G}^{K,m}_a) \subset[m_{f_a} - M_R, M_{f_a} + M_R]. \tag*{(21)}

As K→∞K \to\infty, the eigenvalues of TK+1T_{K+1} distribute themselves on the mm branches ζ↦λr(fa(ζ))\zeta\mapsto\lambda_r(f_a(\zeta)) [15, 7, 31].

Proof sketch. The block-Toeplitz part is translation-invariant by construction; the boundary correction is supported on the first and last mm rows/columns, hence rank ≤4m\le4m. Locating the spectrum of TK+1T_{K+1} inside the symbol range is the classical block-Toeplitz spectral theorem; the band-distribution claim is the corresponding asymptotic statement [15]. The Courant–Fischer perturbation by RK,mR_{K,m} is uniform in KK because ∥RK,m∥\lVert R_{K,m} \rVert is bounded independently of KK. Detailed calculations for m≤2m \le2 are given in Appendix § 12.

Theorem 5 applies to any local symmetric bilinear form; the stencils BB and CC, and hence the symbol faf_a, are the only data that change with the form. We now apply it to the L2L^2 and the H1H^1 case in turn, and the conclusions differ.

From the Toeplitz symbol to the frequency principle

The frequency principle of Rahaman et al. [26] and Xu et al. (2020) is a statement about frequencies: the Fourier mode of frequency qq is learned at a rate that decreases with qq. Theorem 5 is a statement about the spectrum of a block-Toeplitz matrix. The purpose of this subsection is to write down the translation between these two concepts.

Proposition 6 (Eigenpairs of the block-Toeplitz part). *Let TK+1T_{K+1} be as in Theorem 5, with symbol faf_a, branches λ1(fa(ζ))≤⋯≤λm(fa(ζ))\lambda_1(f_a(\zeta)) \le\cdots\le\lambda_m(f_a(\zeta)), and corresponding symbol eigenvectors wb(ζ)∈Rmw_b(\zeta) \in\mathbb{R}^m. Set

ζr:=rπK+2,r=1,…,K+1.(22)\zeta_r := \frac{r\pi}{K+2}, \qquad r = 1, \ldots, K+1. \tag*{(22)}

Then the eigenvectors of TK+1T_{K+1} are, blockwise, (sin⁡(kζr)wb(ζr))k=0K(\sin(k\zeta_r)w_b(\zeta_r))_{k=0}^{K}, with eigenvalues λb(fa(ζr))\lambda_b(f_a(\zeta_r)), b=1,…,mb = 1, \ldots, m.

Together with Lemma 1, this result says that the error along the eigenvector indexed by (r,b)(r,b) decays as e−tλb(fa(ζr))e^{-t\lambda_b(f_a(\zeta_r))}. What it does not say is which frequency that eigenvector represents. Fixing that is a matter of choice rather than an intrinsic mathematical fact, and the two indices rr and bb have to be matched separately.

The index rr counts oscillations across the gates. Block kk of the eigenvector is supported in [xk−1,xk+1][x_{k-1}, x_{k+1}] with xk=k/Kx_k = k/K, so on [0,1][0,1] the eigenvector behaves like sin⁡(πrx)\sin(\pi r x) and has rr half-waves, whereas sin⁡(2πqx)\sin(2\pi qx) has 2q2q; matching the counts gives r=2qr = 2q. The index bb labels the mm branches of the symbol. At a fixed ζr\zeta_r the mm eigenvectors of Proposition 6 all share the same gate-to-gate oscillation and differ only in the profile wb(ζr)∈Rmw_b(\zeta_r) \in\mathbb{R}^m, which is the shape within a single gate across its mm knots. Of these mm candidate eigenvectors we keep the one that is closest to the target mode, in the L2L^2 sense made explicit in (24).

Definition 2 (Frequency-resolved learning rate). For q≤K/2q \le K/2, set

ζq:=2πqK+2,(23)\zeta_q := \frac{2\pi q}{K+2}, \tag*{(23)}

write vq,b∈L2(D)v_{q,b} \in L^2(D) for the function represented by the bb-th eigenvector of Proposition 6 at ζq\zeta_q, expanded in the gated features φk,j=ψkσ(⋅−sk,j)\varphi_{k,j}=\psi_k\sigma(\cdot-s_{k,j}) of (11), and let b(q)b(q) be the index of the one closest to the target mode:

vq,b:=∑k=0K∑j=1msin⁡(kζq)(wb(ζq))jφk,j,b(q):=arg⁡max⁡b=1,…,m ⟨vq,b,sin⁡(2πq⋅)⟩∥vq,b∥.(24)v_{q,b}:=\sum_{k=0}^{K}\sum_{j=1}^{m}\sin(k\zeta_q)\left(w_b(\zeta_q)\right)_j\varphi_{k,j},\qquad b(q):=\underset{b=1,\ldots,m}{\arg\max}\ \frac{\left\langle v_{q,b},\sin(2\pi q\cdot)\right\rangle}{\lVert v_{q,b}\rVert}. \tag*{(24)}

The learning rate of frequency qq, under the gradient flow (7) of (15), is then

Ra(q):=λb(q)(fa(ζq))(25)R_a(q):=\lambda_{b(q)}\left(f_a(\zeta_q)\right) \tag*{(25)}

For m=1m=1 there is a single branch and b(q)≡1b(q)\equiv1.

The restriction q≲K/2q\lesssim K/2 is the Nyquist limit of the gate grid: the eigenvector changes sign at most once per gate, so ζq≤π\zeta_q\leq\pi. With (25) in hand the dictionary reads as follows.

block-Toeplitz objecttraining object
symbol variable ζ∈[0,π]\zeta\in[0,\pi]frequency qq, through ζq=2πq/(K+2)\zeta_q=2\pi q/(K+2)
branch value λb(fa(ζq))\lambda_b(f_a(\zeta_q))learning rate Ra(q)R_a(q) of the mode of frequency qq
faf_a bounded away from 00 and ∞\inftyno frequency ordering (Theorem 8)
fa(ζ)→0f_a(\zeta)\to0 as ζ→0\zeta\to0the lowest frequency is the slowest (Theorem 11)
Mfa/mfaM_{f_a}/m_{f_a}the spectral-bias factor (Proposition 7)

Table 6.

Every row of the table translates a property of the symbol into a property of training. The symbol is fixed by the architecture and the bilinear form together.

Definition 3 (Spectral-bias factor). For a band of frequencies Q⊆{1,…,⌊K/2⌋}Q\subseteq\{1,\ldots,\lfloor K/2\rfloor\}, the spectral-bias factor of the bilinear form aa on QQ is

Ba(Q):=max⁡q∈QRa(q)min⁡q∈QRa(q),(26)\mathcal{B}_a(Q):=\frac{\max_{q\in Q}R_a(q)}{\min_{q\in Q}R_a(q)}, \tag*{(26)}

the factor by which the slowest frequency in the band lags the fastest. Ba(Q)=1\mathcal{B}_a(Q)=1 means no frequency is preferred; spectral bias in the sense of Definition 1 is $\mathcal{B}_a(Q)\gg 1.

Proposition 7 (Symbol bounds control spectral bias). Let mfam_{f_a} and MfaM_{f_a} be as in Theorem 5. For every band QQ and every KK,

Ba(Q)≤κ(TK+1)≤Mfamfa,(27)\mathcal{B}_a(Q)\leq\kappa(T_{K+1})\leq\frac{M_{f_a}}{m_{f_a}}, \tag*{(27)}

with the convention that the right-hand side is +∞+\infty when mfa=0m_{f_a}=0. There are accordingly two regimes, separated by the single scalar mfam_{f_a}:

  1. mfa>0m_{f_a}>0. Ba(Q)\mathcal{B}_a(Q) is bounded by a constant determined by the stencil blocks BB and CC alone, for every band and every KK. The disparity between frequencies does not grow with the size of the model: adding windows adds resolvable frequencies without widening it.

  2. mfa=0m_{f_a}=0, attained at ζ=0\zeta=0. Then Ra(q)→0R_a(q)\to0 as ζq→0\zeta_q\to0, so the slowest mode in any band is the lowest frequency, and over the full resolvable range Q={1,…,⌊K/2⌋}Q=\{1,\ldots,\lfloor K/2\rfloor\} one has Ba(Q)=Θ(K2)\mathcal{B}_a(Q)=\Theta(K^2): the disparity grows quadratically with the size of the model.

Proof. Each Ra(q)R_a(q) is by Definition 2 a value λb(q)(fa(ζq))\lambda_{b(q)}(f_a(\zeta_q)), which by Proposition 6 is an eigenvalue of TK+1T_{K+1}; a ratio of two eigenvalues is at most κ(TK+1)\kappa(T_{K+1}), which gives the first inequality. The second is Theorem 5, which places spec⁡(TK+1)⊆[mfa,Mfa]\operatorname{spec}(T_{K+1})\subseteq[m_{f_a},M_{f_a}]. Part (i) is immediate, since mfam_{f_a} and MfaM_{f_a} depend only on BB and CC. For (ii), faf_a is smooth and vanishes at ζ=0\zeta=0, so Ra(q)≍ζq2≍q2/K2R_a(q)\asymp\zeta_q^2\asymp q^2/K^2 for q≪Kq\ll K while Ra(⌊K/2⌋)=O(1)R_a(\lfloor K/2\rfloor)=O(1), and the ratio is Θ(K2)\Theta(K^2).

L2L^2 regression: spectral bias bounded uniformly in KK

Specializing Theorem 5 to a=aL2a=a^{L^2}, for which α=3\alpha=3, gives the regression mass matrix. The blocks BB and CC for m=1m=1 and m=2m=2 are computed in Appendix § 12. The property of the symbol fL2f_{L^2} that drives everything in this subsection is that min⁡ζλ1(fL2(ζ))\min_{\zeta}\lambda_1(f_{L^2}(\zeta)) is strictly positive.

Theorem 8 (L2L^2 regression: KK-uniform eigenvalue and learning-rate bounds). Let a=aL2a=a^{L^2}, under the assumptions of Lemma 4. There exist constants 0<cmL2≤CmL2<∞0<c_m^{L^2}\le C_m^{L^2}<\infty, depending on mm but not on KK, such that

(a) (eigenvalues) every nonzero eigenvalue λ\lambda of GKgateG^{\mathrm{gate}}_K satisfies

cmL2K−3≤λ≤CmL2K−3,(28)c_m^{L^2}K^{-3}\le\lambda\le C_m^{L^2}K^{-3}, \tag*{(28)}

so that κ(GKgate)≤CmL2/cmL2\kappa(G^{\mathrm{gate}}_K)\le C_m^{L^2}/c_m^{L^2} uniformly in KK;

(b) (learning rates) mfL2>0m_{f_{L^2}}>0, so RL2(q)∈[mfL2,MfL2]R_{L^2}(q)\in[m_{f_{L^2}},M_{f_{L^2}}] for every frequency qq and every KK, and by Proposition 7

BL2(Q)≤MfL2mfL2for every band Q and every K.(29)\mathcal{B}_{L^2}(Q)\le\frac{M_{f_{L^2}}}{m_{f_{L^2}}}\qquad\text{for every band }Q\text{ and every }K. \tag*{(29)}

Part (b) is the statement that matters for training: the learning rate is bounded above and below independently of frequency, so gating does not merely bound the disparity of rates, it removes the frequency ordering itself. The constants are explicit for one and two knots per gate, as seen in the corollaries below.

Corollary 9 (one knot per gate, m=1m=1). With one knot per gate (s0=0s_0=0, sk=xk−1s_k=x_{k-1}), GKgate=h3(TK+1+RK+1)G^{\mathrm{gate}}_K=h^3(T_{K+1}+R_{K+1}) where TK+1T_{K+1} is tridiagonal Toeplitz with diagonal a0=11/15a_0=11/15 and off-diagonals a1=2/15a_1=2/15, and rank⁡(RK+1)≤3\operatorname{rank}(R_{K+1})\le3. The eigenvalues of TK+1T_{K+1} satisfy μr(TK+1)=a0+2a1cos⁡(rπ/(K+2))∈[7/15,1]\mu_r(T_{K+1})=a_0+2a_1\cos(r\pi/(K+2))\in[7/15,1]. The rescaled mass matrix GK:=h−3GKgate\mathcal{G}_K:=h^{-3}G^{\mathrm{gate}}_K satisfies, for all K≥2K\ge2,

spec⁡(GK)⊂[7−3460,1]≈[0.0195,1],κ(GK)≤4(7+34)≈51.3,(30)\operatorname{spec}(\mathcal{G}_K)\subset\left[\frac{7-\sqrt{34}}{60},1\right]\approx[0.0195,1],\qquad\kappa(\mathcal{G}_K)\le4(7+\sqrt{34})\approx51.3, \tag*{(30)}

while the learning rates occupy the much smaller interval

RL2(q)∈[715,1]for every q,BL2(Q)≤157≈2.14.(31)R_{L^2}(q)\in\left[\frac{7}{15},1\right]\quad\text{for every }q,\qquad\mathcal{B}_{L^2}(Q)\le\frac{15}{7}\approx2.14. \tag*{(31)}

The two intervals differ because (30) must also accommodate the rank-3 perturbation RK+1R_{K+1} of (51), whose eigenvectors correspond to no ζq\zeta_q: measured, κ(GK)→33.7\kappa(\mathcal{G}_K)\to33.7 while BL2≤15/7≈2.14\mathcal{B}_{L^2}\le15/7\approx2.14. The condition number is not an indicator of spectral bias.

Corollary 10 (two knots per gate, m=2m=2). With two knots per gate as in Appendix § 12, the blocks B=(11/157/607/601/30)B=\begin{pmatrix}11/15&7/60\\7/60&1/30\end{pmatrix}, C=(2/151/2000)C=\begin{pmatrix}2/15&1/20\\0&0\end{pmatrix} yield a symbol fL2(θ)f_{L^2}(\theta) with mfL2≈0.005405m_{f_{L^2}}\approx0.005405 and MfL2≈1.0279M_{f_{L^2}}\approx1.0279. With rank-4 boundary correction of operator norm ≤1/20\le1/20, every eigenvalue of GK(2)=K3GKgate\mathcal{G}^{(2)}_K=K^3G^{\mathrm{gate}}_K lies in [c2,C2][c_2,C_2] with c2≈0.005405c_2\approx0.005405, C2≈1.028C_2\approx1.028, and

RL2(q)∈[0.005405,1.0279],BL2(Q)≤190.2.(32)R_{L^2}(q)\in[0.005405,1.0279],\qquad\mathcal{B}_{L^2}(Q)\le190.2. \tag*{(32)}

Eigenvalues organize into two branches as K→∞K\to\infty, with at most four outliers.

Remark 4 (Effect of the gating). In this case, gating does not equalize the learning rates, but it removes the growth of the disparity with NN, from Θ(N4)\Theta(N^4) for the global dictionary to a bound independent of NN. That bound is uniform in KK but not in mm: e.g. we have 33.7, 190 and 9.4×1039.4\times10^3 at m=1,2,4m=1,2,4 (§ 7.1). Hence the design guideline of § 8.1: add windows, not knots.

H1H^1 stiffness: spectral bias persists

We repeat the construction with aH1(u,v)=∫01u′v′ dxa^{H^1}(u,v) = \int_0^1 u'v'\,\mathrm{d}x, obtaining the stiffness matrix Agate,KA^{\mathrm{gate},K} of (17); by Lemma 4 the universal scaling is now α=1\alpha= 1. The property of the symbol fH1f_{H^1} that drives everything here is the opposite of the one in § 5.3.1: min⁡ζλ1(fH1(ζ))\min_{\zeta} \lambda_1(f_{H^1}(\zeta)) is zero, attained at ζ=0\zeta= 0. We state this for one knot per gate, where the stencil is explicit.

Theorem 11 (H1H^1 stiffness: gating resolves neither the conditioning nor the bias). Let a=aH1a = a^{H^1} with one knot per gate, as in Corollary 9. Then:

(a) (stencil and symbol) the interior entries are

Akkgate,K=83h,Ak,k+1gate,K=−43h,Akk′gate,K=0 for ∣k−k′∣>1,(33)A^{\mathrm{gate},K}_{kk} = \frac{8}{3}h,\qquad A^{\mathrm{gate},K}_{k,k+1} = -\frac{4}{3}h,\qquad A^{\mathrm{gate},K}_{kk'} = 0\ \text{for } |k-k'| > 1, \tag*{(33)}

with one-sided boundary entries, so the symbol of h−1Agate,Kh^{-1}A^{\mathrm{gate},K} is fH1(ζ)=83(1−cos⁡ζ)f_{H^1}(\zeta) = \frac{8}{3}(1-\cos\zeta) and

mfH1=0 (at ζ=0),MfH1=163.(34)m_{f_{H^1}} = 0\ \text{(at } \zeta= 0\text{)},\qquad M_{f_{H^1}} = \frac{16}{3}. \tag*{(34)}

(b) (eigenvalues) μr(TK+1)=83(1−cos⁡rπK+2)\mu_r(T_{K+1}) = \frac{8}{3}\left(1-\cos\frac{r\pi}{K+2}\right), so μ1∼4π23K−2\mu_1 \sim\frac{4\pi^2}{3}K^{-2} while μK+1→16/3\mu_{K+1} \to16/3, and

κ(Agate,K)=Θ(K2)=Θ(N2);(35)\kappa(A^{\mathrm{gate},K}) = \Theta(K^2) = \Theta(N^2); \tag*{(35)}

(c) (learning rates)

RH1(q)=83(1−cos⁡2πqK+2)∼16π23q2K2(q≪K),(36)R_{H^1}(q) = \frac{8}{3}\left(1-\cos\frac{2\pi q}{K+2}\right) \sim\frac{16\pi^2}{3}\frac{q^2}{K^2}\qquad(q \ll K), \tag*{(36)}

which increases with qq and vanishes as q→0q \to0, so BH1({1,…,Q})→Q2\mathcal{B}_{H^1}(\{1,\ldots,Q\}) \to Q^2 at fixed band and BH1=Θ(K2)\mathcal{B}_{H^1} = \Theta(K^2) over the full resolvable range.

Proof. The stencil follows by the change of variables of Proposition 13, applied to φk′φk′′\varphi_k'\varphi_{k'}' in place of φkφk′\varphi_k\varphi_{k'}, and the symbol calculation is direct; (b) and (c) are then Proposition 6 and Definition 2 evaluated at fH1f_{H^1}.

Parts (b) and (c) are case (ii) of Proposition 7: the symbol annihilates the constant eigenvector, so the smallest eigenvalues of TK+1T_{K+1} collapse, and the slowest mode is the lowest frequency. Table 1 shows both growing at the same rate.

KKλmin⁡(h−1Agate,K)\lambda_{\min}(h^{-1}A^{\mathrm{gate},K})λmax⁡(h−1Agate,K)\lambda_{\max}(h^{-1}A^{\mathrm{gate},K})λmin⁡(TK+1)\lambda_{\min}(T_{K+1})λmax⁡(TK+1)\lambda_{\max}(T_{K+1})BH1\mathcal{B}_{H^1}
242^41.16×10−21.16 \times10^{-2}5.2855.2854.05×10−24.05 \times10^{-2}5.2935.29332.232.2
262^67.83×10−47.83 \times10^{-4}5.3305.3303.02×10−33.02 \times10^{-3}5.3305.330441441
282^84.99×10−54.99 \times10^{-5}5.3335.3331.98×10−41.98 \times10^{-4}5.3335.3336.74×1036.74 \times10^3
2102^{10}3.13×10−63.13 \times10^{-6}5.3335.3331.25×10−51.25 \times10^{-5}5.3335.3331.07×1051.07 \times10^5
2122^{12}1.96×10−71.96 \times10^{-7}5.3335.3337.84×10−77.84 \times10^{-7}5.3335.3331.70×1061.70 \times10^6
2142^{14}1.23×10−81.23 \times10^{-8}5.3335.3334.90×10−84.90 \times10^{-8}5.3335.3332.72×1072.72 \times10^7

Table 1. Gated stiffness matrix at m=1m = 1. The symbol vanishes at ζ=0\zeta= 0, so λmin⁡\lambda_{\min} decays as K−2K^{-2} while λmax⁡\lambda_{\max} saturates at 16/316/3, and the spectral-bias factor grows at the same Θ(K2)\Theta(K^2) rate: conditioning and frequency disparity degrade together.

Remark 5 (Effect of the gating). In the variational case, gating doesn’t change the growth rate: κ\kappa and BH1\mathcal{B}_{H^1} are both Θ(N2)\Theta(N^2), as for the global dictionary. What it changes is the direction of the ordering, making q=1q = 1 the slowest mode. The remedy is therefore not another dictionary but a preconditioner, which will be built in § 6.

Table 2 collects the two cases side by side.

Form aaL2L^2 inner productH1H^1 semi-inner-product
Gram matrixmass Ggate,KG^{\mathrm{gate},K}stiffness Agate,KA^{\mathrm{gate},K}
Universal scaleK−3K^{-3} (Lemma 4, α=3\alpha=3)K−1K^{-1} (α=1\alpha=1)
Symbol minimummfL2>0m_{f_{L^2}}>0mfH1=0m_{f_{H^1}}=0 at ζ=0\zeta=0
λmin⁡\lambda_{\min} scaleΘ(K−3)\Theta(K^{-3}); uniform after rescalingΘ(K−3)\Theta(K^{-3}); not uniform after rescaling
Condition no.κ≤CmfL2L2/cmfL2L2\kappa\le C^{L^2}_{m_{f_{L^2}}}/c^{L^2}_{m_{f_{L^2}}}, KK-uniformκ(Agate,K)=Θ(N2)\kappa(A^{\mathrm{gate},K})=\Theta(N^2)
Spectral-bias factor Ba\mathcal{B}_a≤MfL2/mfL2\le M_{f_{L^2}}/m_{f_{L^2}}, KK-uniform (2.14 at m=1m=1)Θ(K2)\Theta(K^2); ordering inverted
Rate at q→0q\to0bounded below by mfL2>0m_{f_{L^2}}>0RH1(q)∼16π23K2q2/K2→0R_{H^1}(q)\sim\frac{16\pi^2}{3K^2}q^2/K^2\to0

Table 2. The gated dictionary under the two bilinear forms. The Gram matrices share the same block-banded, block-Toeplitz structure; only the symbol differs, and its minimum mfam_{f_a} decides every row below it. The λmin⁡\lambda_{\min} row agrees by coincidence of the two scalings, Θ(K−3)⋅Θ(1)\Theta(K^{-3})\cdot\Theta(1) against Θ(K−1)⋅Θ(K−2)\Theta(K^{-1})\cdot\Theta(K^{-2}).

Preconditioning seen from the optimizer side

Proposition 7 sets a limit on what the architecture can do: the spectral-bias factor is bounded by Mfa/mfaM_{f_a}/m_{f_a}, and that bound says nothing once mfa=0m_{f_a}=0. By Theorem 11 this is the situation for aH1a^{H^1}, and changing the gate profile or the knot pattern does not help, because the symbol vanishes at ζ=0\zeta=0 for a reason that belongs to the bilinear form and not to the dictionary. We therefore leave the dictionary as it is and change the optimizer instead, replacing the gradient flow (7) by a preconditioned one.

The preconditioned rate. Let MM be symmetric positive definite and consider

θ˙=−M−1(Agate,Kθ−b).(37)\dot{\theta}=-M^{-1}\left(A^{\mathrm{gate},K}\theta-b\right). \tag*{(37)}

The error decays in the eigenbasis of M−1Agate,KM^{-1}A^{\mathrm{gate},K}, whose eigenvalues are those of the symmetric matrix M−1/2Agate,KM−1/2M^{-1/2}A^{\mathrm{gate},K}M^{-1/2}. If MM is itself block-Toeplitz with symbol gg, that matrix has interior symbol g−1/2fag−1/2g^{-1/2}f_ag^{-1/2}, with the eigenvalues of g−1fag^{-1}f_a, so Definition 2 and Proposition 7 apply verbatim to

RaM(q):=λb(q)(g(ζq)−1fa(ζq)),BaM(Q)=max⁡q∈QRaM(q)min⁡q∈QRaM(q).(38)R_a^M(q):=\lambda_b(q)\left(g(\zeta_q)^{-1}f_a(\zeta_q)\right),\qquad \mathcal{B}_a^M(Q)=\frac{\max_{q\in Q}R_a^M(q)}{\min_{q\in Q}R_a^M(q)}. \tag*{(38)}

The architecture fixes faf_a; the optimizer fixes gg. The whole of Section 5 therefore transfers, with g−1fag^{-1}f_a in place of faf_a, and the design question becomes the choice of gg.

The canonical instance is Energy Natural Gradient Descent ([25]), which projects the optimal function-space descent direction onto the model’s tangent space. For a linear-in-parameters model with the quadratic loss (15) this is (37) with M=Agate,KM=A^{\mathrm{gate},K} exactly, hence g=fag=f_a and g−1fa≡Ig^{-1}f_a\equiv I. The preconditioners below are computable approximations of that choice.

We consider three approximations to

P=Agate,K=(B0C00C0TB1C1⋱⋱⋱CK−2TBK−1CK−10CK−1TBK),P=A^{\mathrm{gate},K}= \begin{pmatrix} B_0 & C_0 & & 0\\ C_0^{\mathsf T} & B_1 & C_1 & \\ & \ddots& \ddots& \ddots\\ & & C_{K-2}^{\mathsf T} & B_{K-1} & C_{K-1}\\ 0 & & & C_{K-1}^{\mathsf T} & B_K \end{pmatrix},

namely the block diagonal preconditioner

Pd=(B00B1⋱BK−10BK),\mathcal{P}_d= \begin{pmatrix} B_0 & & & 0\\ & B_1 & & \\ & & \ddots& \\ & & & B_{K-1}\\ 0 & & & & B_K \end{pmatrix},

and the block Toeplitz and optimal block circulant Chan [9] preconditioners

Pt=(BC0CTBC⋮⋱⋱⋱⋱CTBC0CTB),Pc=(BK−1KCK−1KCTK−1KCTBK−1KC⋱⋱⋱K−1KCK−1KCTBK−1KCK−1KCTK−1KCB),\mathcal{P}_{t} = \begin{pmatrix} B & C & 0 \\ C^{\mathsf{T}} & B & C \\ \vdots& \ddots& \ddots& \ddots\\ & \ddots& C^{\mathsf{T}} & B & C \\ 0 & & C^{\mathsf{T}} & B \end{pmatrix}, \qquad \mathcal{P}_{c} = \begin{pmatrix} B & \frac{K-1}{K}C & & \frac{K-1}{K}C^{\mathsf{T}} \\ \frac{K-1}{K}C^{\mathsf{T}} & B & \frac{K-1}{K}C & \\ & \ddots& \ddots& \ddots\\ \frac{K-1}{K}C & & \frac{K-1}{K}C^{\mathsf{T}} & B & \frac{K-1}{K}C \\ & \frac{K-1}{K}C^{\mathsf{T}} & & \frac{K-1}{K}C & B \end{pmatrix},

where B∈Rm×mB \in\mathbb{R}^{m \times m} and C∈Rm×mC \in\mathbb{R}^{m \times m} are representative blocks, taken from an interior gate. Each of the three is inserted as MM in the preconditioned flow (37); when the same systems are solved directly, a preconditioned conjugate gradient iteration is the natural choice, and Theorem 12(b) below bounds its iteration count.

Of the three, only Pt\mathcal{P}_{t} is built from the symbol: its block rows carry the interior stencil (B,C)(B,C) of Theorem 5, so its symbol is g=fag=f_a, the choice that (38) makes trivial. It agrees with Agate,KA_{\mathrm{gate},K} on every interior block, exactly and not approximately, and the two can differ only on the boundary gates.

Theorem 12 (block-Toeplitz preconditioning). Let A:=Agate,KA := A_{\mathrm{gate},K} and let Pt\mathcal{P}_{t} be built from the interior stencil (B,C)(B,C). Then:

(a) (learning rates) g=fH1g=f_{H_1}, so g−1fH1≡Ig^{-1}f_{H_1}\equiv I and

RH1Pt(q)≡1,BH1Pt(Q)=1(39)R_{H_1}^{\mathcal{P}_{t}}(q) \equiv1,\qquad \mathcal{B}_{H_1}^{\mathcal{P}_{t}}(Q)=1 \tag*{(39)}

for every band QQ and every KK: the frequency dependence is removed exactly, and what remains is a boundary effect;

(b) (structure) A−PtA-\mathcal{P}_{t} is supported on the first and last block rows and columns, so

rank⁡(A−Pt)≤4m,Pt−1A=I+E,rank⁡(E)≤4m.(40)\operatorname{rank}(A-\mathcal{P}_{t})\le4m,\qquad \mathcal{P}_{t}^{-1}A=I+E,\qquad \operatorname{rank}(E)\le4m. \tag*{(40)}

Hence Pt−1A\mathcal{P}_{t}^{-1}A has at most 4m+14m+1 distinct eigenvalues, and in exact arithmetic conjugate gradients terminates in at most that many iterations, independently of KK.

Remark 6 (the observed rank). The rank is often in practice smaller than 4m4m and does not grow with mm: for the knot pattern of Appendix A we find rank⁡(A−Pt)=3\operatorname{rank}(A-\mathcal{P}_{t})=3 at every m∈{1,2,4,8}m\in\{1,2,4,8\} and K∈{15,31,63}K\in\{15,31,63\} tested.

Remark 7 (why only the Toeplitz preconditioner removes the spectral bias). The symbol of Pd\mathcal{P}_{d} is the constant BB, so B−1fH1B^{-1}f_{H_1} still vanishes at ζ=0\zeta=0; at m=1m=1 it leaves BH1\mathcal{B}_{H_1} exactly unchanged (Table 3). The symbol of Pc\mathcal{P}_{c} differs from fH1f_{H_1} at small ζ\zeta, which is where fH1f_{H_1} is smallest; it brings BH1\mathcal{B}_{H_1} down to a small factor, but one that still grows with KK. Only Pt\mathcal{P}_{t} removes the frequency dependence, and its condition number is uniform in KK when Dirichlet boundary conditions (BC) are used.

mmKKκ\kappa unpre.Pd\mathcal{P}_{d}Pt\mathcal{P}_{t}Pc\mathcal{P}_{c}BH1\mathcal{B}_{H^1} unpre.Pc\mathcal{P}_{c}
116108.4107.513.5310.6832.22.04
132423.4423.113.8811.551162.85
1641677167714.0718.964414.47
416425.3139.428.7822.744143.11
432852.6554.429.4024.084264.52
4643171221429.7226.349019.32
8164801145.159.2746.57469814.2
8324873559.160.5949.31480712.8
8647120221961.2751.94489924.8

Table 3. Stiffness matrix with u(1)=0u(1)=0 imposed, under the three preconditioners: condition number on the left, spectral-bias factor over the gate-resolved band on the right. Only Pt\mathcal{P}_{t} is uniform in KK in both. It gives BH1=1\mathcal{B}_{H^1}=1 exactly in every row, and Pd\mathcal{P}_{d} leaves BH1\mathcal{B}_{H^1} unchanged at m=1m=1, so both are omitted on the right.

Table 3 reports both measures. The Pt\mathcal{P}_{t} column of κ\kappa is constant in KK to three significant figures while BH1Pt=1\mathcal{B}_{H_1}^{\mathcal{P}_{t}}=1.

Remark 8 (architecture and optimizer are two complementary choices). Equation (38) separates the two levers: the dictionary sets the value of faf_a, the optimizer sets gg, and the quantity g−1fag^{-1}f_a is now representative of the learning rates. Where faf_a is bounded away from zero, g=Ig=I (i.e. no preconditioning) suffices; where it is not, no admissible faf_a will help tackling the spectral bias, and the burden passes to gg. This is the mechanism behind the finding of Khodakarami et al. [19] that curvature-aware optimizers reorder the spectral learning order of physics-informed models while architectural changes alone do not.

Numerical results

Two dictionaries are compared at matched parameter count: the global ReLU dictionary φℓ(x)=σ(x−sℓ)\varphi_{\ell}(x)=\sigma(x-s_{\ell}) of § 3, and the gated dictionary φk,j=ψk(σ(⋅−sk,j))\varphi_{k,j}=\psi_k(\sigma(\cdot-s_{k,j})) of § 4, whose double index we flatten to a single ℓ=1,…,N\ell=1,\ldots,N. In both cases the dictionary is fixed and only the coefficients θ\theta of fθ=∑ℓθℓφℓf_{\theta}=\sum_{\ell}\theta_{\ell}\varphi_{\ell} are trained, except in § 7.3, where the thresholds are trained as well.

Two problems are solved. For L2L^2 regression, the target is f⋆(x)=∑q=110Aqsin⁡(2πqx)f^{\star}(x)=\sum_{q=1}^{10}A_q\sin(2\pi qx) with Aq=1A_q=1, following Rahaman et al. [26]. For the Poisson problem we solve −u′′=f-u''=f on (0,1)(0,1) with u(0)=u(1)=0u(0)=u(1)=0 and f=∑q(2πq)2Aqsin⁡(2πqx)f=\sum_q(2\pi q)^2A_q\sin(2\pi qx), so that the exact solution is the same sum of sines and the same normalization applies.

The frequency content of the model. During training θ\theta evolves, but what we want to follow is the frequency content of the function fθ=∑ℓθℓφℓf_{\theta}=\sum_{\ell}\theta_{\ell}\varphi_{\ell}, so we first make that content explicit. For an integer frequency qq, set

(sq)ℓ:=2∫01φℓ(x)sin⁡(2πqx) dx,(cq)ℓ:=2∫01φℓ(x)cos⁡(2πqx) dx,(41)(s_q)_{\ell}:=2\int_{0}^{1}\varphi_{\ell}(x)\sin(2\pi qx)\,\mathrm{d}x,\qquad(c_q)_{\ell}:=2\int_{0}^{1}\varphi_{\ell}(x)\cos(2\pi qx)\,\mathrm{d}x, \tag*{(41)}

two vectors of RN\mathbb{R}^{N} fixed by the dictionary alone. The qq-th Fourier coefficient of the model, and the amplitude it carries at that frequency, are

f~q(θ):=2∫01fθ(x)e−2πiqx dx=cqTθ−i sqTθ,∣f~q(θ)∣=[(sqTθ)2+(cqTθ)2]1/2.(42)\widetilde{f}_q(\theta):=2\int_{0}^{1}f_{\theta}(x)e^{-2\pi iqx}\,\mathrm{d}x=c_q^{\mathsf{T}}\theta-i\,s_q^{\mathsf{T}}\theta,\qquad\lvert\widetilde{f}_q(\theta)\rvert=\left[(s_q^{\mathsf{T}}\theta)^2+(c_q^{\mathsf{T}}\theta)^2\right]^{1/2}. \tag*{(42)}

How that content evolves. With e(t):=θ(t)−θ⋆e(t):=\theta(t)-\theta^{\star} the flow gives e˙=−Gae\dot{e}=-G^{a}e, so by Lemma 1 the error decays mode by mode in the eigenbasis Gavr=μrvrG^{a}v_r=\mu_rv_r,

e(t)=∑r=1Ne−tμr⟨e(0),vr⟩vr.(43)e(t)=\sum_{r=1}^{N}e^{-t\mu_r}\langle e(0),v_r\rangle v_r. \tag*{(43)}

Applying (41) to (43) gives:

sqTe(t)=∑r=1Ne−tμrαq,r,αq,r:=(sqTvr)⟨e(0),vr⟩,(44)s_q^{\mathsf{T}}e(t)=\sum_{r=1}^{N}e^{-t\mu_r}\alpha_{q,r},\qquad\alpha_{q,r}:=(s_q^{\mathsf{T}}v_r)\langle e(0),v_r\rangle, \tag*{(44)}

and likewise for cqc_q. The frequency-qq content of the error is therefore governed not by one rate but by a mixture of all NN, weighted by αq,r\alpha_{q,r} – that is, by how much of frequency qq each eigenvector vrv_r carries.

Amplitude rather than error. One could follow either the error in the qq-th Fourier coefficient, sqTe(t)s_q^{\mathsf{T}}e(t), or the amplitude ∣f~q(t)∣\lvert\widetilde{f}_q(t)\rvert that the model has built up there. The error is of little use here. By (44) it is eventually dominated by its slowest term, sqTe(t)∼e−tμmin⁡αq,min⁡s_q^{\mathsf{T}}e(t)\sim e^{-t\mu_{\min}}\alpha_{q,\min}, so every frequency ends up converging at the same rate μmin⁡\mu_{\min}. The amplitude behaves differently. It rises from 0 towards ∣f~q(θ⋆)∣\lvert\widetilde{f}_q(\theta^{\star})\rvert at a rate set by the terms with the largest ∣αq,r∣\lvert\alpha_{q,r}\rvert. We therefore use the amplitude, exactly as Rahaman et al. [26] do: color encodes ∣f~q(t)∣/Aq\lvert\widetilde{f}_q(t)\rvert/A_q clipped to [0,1][0,1], on a single scale in every figure, with frequency horizontal and training time vertical as in their Figure 1.

Rise time, and a common clock. Each amplitude curve is summarized by the time it takes to reach 90% of its converged value,

tq:=inf⁡{t≥0:∣f~q(t)∣≥0.9∣f~q(θ⋆)∣}.(45)t_q:=\inf\{t\ge0:\lvert\widetilde{f}_q(t)\rvert\ge0.9\lvert\widetilde{f}_q(\theta^{\star})\rvert\}. \tag*{(45)}

We measure time in units of the fastest mode,

τ:=μmax⁡t,μmax⁡:=λmax⁡(Ga),(46)\tau:= \mu_{\max} t,\qquad\mu_{\max} := \lambda_{\max}(G^a), \tag*{(46)}

which has a concrete reading: τ\tau counts gradient steps taken at an admissible step size, so the three panels of a figure are on one clock.

The measured spectral-bias factor. Writing τq=μmax⁡tq\tau_q = \mu_{\max}t_q, we report

S(Q):=max⁡q∈Qτqmin⁡q∈Qτq,(47)\mathcal{S}(Q) := \frac{\max_{q\in Q}\tau_q}{\min_{q\in Q}\tau_q}, \tag*{(47)}

which is (26) with the predicted rates Ra(q)R_a(q) replaced by the measured rise times. It is not quite Ba\mathcal{B}_a, and it is not κ\kappa either; the three quantities differ only in what they range over.

modesoperator
Ba(Q)\mathcal{B}_a(Q): frequency modes onlyinterior symbol
S(Q)\mathcal{S}(Q): frequency modes onlyassembled GaG^a
κ(Ga)\kappa(G^a): all modesassembled GaG^a

Both relations follow. First, S≤κ(Ga)\mathcal{S} \le\kappa(G^a) always, because by (44) a rise time is bracketed by the fastest and the slowest eigenvalue that contributes to it. Second, S\mathcal{S} coincides with Ba\mathcal{B}_a when each frequency is carried by a single interior eigenvector, since if αq,r≠0\alpha_{q,r} \ne0 only for r=r(q)r = r(q) then (44) gives tq=ln⁡(10)/μr(q)t_q = \ln(10)/\mu_{r(q)}. Table 4 reports all three quantities for the configurations used below.

configurationS\mathcal{S} measuredBa\mathcal{B}_a predictedκ\kappa predicted
L2L^2, global (N=64N = 64)1.9×1051.9 \times10^5—6.7×1076.7 \times10^7
L2L^2, gated (K=31K = 31, m=2m = 2)4.54.51.541.54189189
H1H^1, global (N=64N = 64)2.232.23—824824
H1H^1, gated (K=31K = 31, m=2m = 2)11.011.010.510.5533533
H1H^1, gated +Pt+\mathcal{P}_t4.364.361114.014.0

Table 4. Measured against predicted, at the sizes of Figure 2 and Figure 4. Only S\mathcal{S} comes from the training runs; Ba\mathcal{B}_a is read from the symbol over q=1,…,10q = 1,\ldots,10, and κ\kappa from the assembled matrix, neither needing any training. Ba\mathcal{B}_a exists only for the gated, block-Toeplitz dictionaries, and predicts S\mathcal{S} closely where κ\kappa does not. The H1H^1 rows are the reduced system the dynamics run on, with the Dirichlet constraint eliminated.

Poisson / $H^1$ spectral bias with and without preconditioning

Figure 4. Poisson / H1H^1 with u(0)=u(1)=0u(0) = u(1) = 0 imposed; same metric and time axis as Figure 2. Left to right: global (spread 2.23); gated (spread 11.0, ordering inverted, q=1q = 1 slowest); gated with the block-Toeplitz preconditioner (spread 4.36); the rise times. The rising middle panel is RH1(q)∝q2R_{H^1}(q) \propto q^2 of Theorem 11(c); the preconditioner flattens it, as Theorem 12(a) requires.

Sections 5 and 6 already report Ba\mathcal{B}_a computed from the symbol, in closed form and without training (Table 1 and Table 3). What follows measures it in the dynamics.

The experiments that follow test different parts of the analysis. § 7.1 tests the eigenvalue statements. § 7.2 and § 7.4 test the frequency statements for the two bilinear forms, where the predictions are opposite: BL2\mathcal{B}_{L^2} bounded, against BH1=Θ(K2)\mathcal{B}_{H^1} = \Theta(K^2) with the ordering reversed, the second together with its preconditioned remedy. § 7.3 goes beyond the analysis by training the non-linear components as well, to see whether the conclusion survives once the dictionary is no longer fixed. § 7.5 varies the target, since S\mathcal{S} is a property of the pair (dictionary, target).

Conditioning

We begin with the eigenvalue statements, Theorems 8(a) and 11(b), before turning to the frequencies. The left panel of Figure 1 plots κ(G)\kappa(G) against the number of features for the L2L^2 mass matrix. A log–log fit gives slope 3.97 for the global dictionary, against the 4 of Proposition 2, and at most 0.02 for the gated dictionary at every mm tested.

Condition number against the number of features N

Figure 1. Condition number against the number of features NN. Left, the L2L^2 mass matrix: the global dictionary grows quartically (slope 3.97), the gated one stays flat at m=1,2,4m = 1, 2, 4. Right, the H1H^1 stiffness matrix: both grow with the same exponent (slopes ≈2\approx2). The predictions are Theorems 8(a) and 11(b).

The two curves separate quickly. By N≈256N \approx256 the global condition number has reached 1.68×10101.68 \times10^{10}, while the gated one has barely moved over the whole range: from 33.09 to 33.73 at m=1m = 1, from 179.9 to 190.1 at m=2m = 2, and from 9.17×1039.17 \times10^3 to 9.36×1039.36 \times10^3 at m=4m = 4. The values at the largest size agree with the limits of Corollaries 9 and 10, κ→33.73\kappa\to33.73 and κ→190.18\kappa\to190.18, to within 0.03%, which checks the analysis and the assembly at once.

The right panel repeats the study for the stiffness matrix, where the prediction is the opposite one. Global and gated grow with essentially the same exponent, fitted slopes 1.98, 2.02 and 1.96, all consistent with the Θ(N2)\Theta(N^2) of Theorem 11. Gating gains a constant factor of about three at fixed NN, 2.3×1032.3 \times10^3 against 6.7×1036.7 \times10^3 at N≈64N \approx64, and nothing beyond that.

Regression: gating compresses the frequency-rate spread

Figure 2 shows the amplitudes during training at matched size (N=64N = 64 global, K=31K = 31 with m=2m = 2 gated). The global dictionary reproduces the picture of Rahaman et al. [26]: a staircase in which each successive frequency darkens an order of magnitude later than the last, with a spread of 1.9×1051.9 \times10^5 between q=1q = 1 and q=10q = 10. For the gated dictionary the same ten frequencies rise together, with a spread of 4.5. The right-hand panel shows the same effect in the rise times, which climb through five decades for the global dictionary and stay inside a factor of five for the gated one.

Spectral bias reduction in gated ReLU networks

Figure 2. L2L^2 regression, linear in parameters. Color is ∣f~q(t)∣/Aq|\tilde{f}_q(t)|/A_q clipped to [0,1][0,1] against the dimensionless time τ=μmax⁡t\tau=\mu_{\max}t of (46), on a log axis shared across panels. Left: the global dictionary, showing the staircase of Rahaman et al. [26], spread 1.9×1051.9 \times10^5. Centre: the gated dictionary, all ten frequencies together, spread 4.5. Right: the rise times, whose spread is S\mathcal{S}.

The symbol predicts 1.54 for this band, and the remaining factor of three is the misalignment between the sines and the eigenvectors of GaG^a discussed above. It is small compared with the four orders of magnitude that separate the two dictionaries. What the figure shows is the content of Remark 4: gating does not equalize the rates, it stops their disparity from growing with NN.

The converged amplitudes call for one caveat. The gated dictionary represents every frequency of the target to within 0.2%, whereas the global one reaches only 0.58 of the target amplitude at q=10q = 10: at N=64N = 64 it is running out of approximation power at precisely the frequencies where it is also slowest. Speed and expressivity are therefore not fully separated in this comparison.

Regression with trainable networks

The experiments above fix the dictionary and train only the output layer. Figure 3 repeats the comparison with every hidden unit’s slope, threshold and output weight trained by full-batch gradient descent, at matched parameter count, and again with plain gradient descent, since an adaptive optimizer would rescale the very rates being measured.

L2 regression with trainable networks

Figure 3. L2L^2 regression with trainable networks, same metric and color scale as Figure 2. The gated network resolves all ten frequencies; the global network resolves only q=1q=1 within the budget, and open symbols in the right-hand panel mark the frequencies it never reaches.

The separation is larger than in the linear case. Within the same budget the gated network brings all ten frequencies to 90% of the target amplitude, taking between 1.25×1031.25 \times10^3 and 4.25×1034.25 \times10^3 iterations, and reaches a final training MSE of 6.4×10−36.4 \times10^{-3}. The global network reaches that level at q=1q=1 alone; the other nine frequencies never do, and its final MSE of 4.32 against a target variance of 5 means it has captured about a seventh of the signal.

The mechanism identified by the linear analysis therefore survives in the nonlinear case. In this case, it is clear that The Gram matrix changes along the trajectory and Section 5 no longer applies, so this comparison is empirical evidence rather than a verified prediction.

Poisson: gating relocates the bias, preconditioning removes it

Figure 4 applies the same diagnostic to the Galerkin system, and the picture is qualitatively different in the way the analysis predicts. For the global stiffness matrix, the spread is only 2.23, and the rise-time curve is U-shaped, with both the lowest and the highest frequencies slow. For the gated matrix, the spread rises to 11.0 and the ordering is inverted: q=1q=1 is now the slowest mode by a factor of eleven, against the 10.5 that (25) predicts for this band, an agreement to within 5%. As a result, in the variational case gating does not remove spectral bias: it moves the bottleneck to low frequency and makes it more severe.

The block-Toeplitz Pt\mathcal{P}_t of § 6 cuts the spread to 4.36, brings the q=1q = 1 rise time down by a factor of two and a half and flattens the curve across the band. Theorem 12(a) predicts BH1Pt=1\mathcal{B}^{\mathcal{P}_t}_{H^1} = 1 in the interior, so what remains is the boundary contribution. In the variational case the remedy is therefore preconditioning rather than gating, and it follows from the block structure already established.

Robustness to the choice of target

The spread is a property of the pair (dictionary, target), so a single target function is probably not representative for a general behaviour. Table 5 reports it for five targets of increasing difficulty, all at the sizes of Figure 2: the equal-amplitude sum of § 7.2; amplitudes increasing with frequency, as in the second setting of Rahaman et al. [26]; independent random phases, which remove any alignment between target and grid; a band of high frequencies k=6,…,10k = 6, \ldots, 10 only; and the two extremes k∈{1,10}k \in\{1, 10\} alone.

targetglobalgatedratio
equal amplitudes1.9×1051.9 \times10^{5}4.54.2×1044.2 \times10^{4}
increasing amplitudes5.8×1065.8 \times10^{6}6.68.9×1058.9 \times10^{5}
random phases5.0×1035.0 \times10^{3}5.98.4×1028.4 \times10^{2}
high band only (k=6,…,10k = 6, \ldots, 10)6.02.82.1
two extremes (k∈{1,10}k \in\{1, 10\})5.2×1035.2 \times10^{3}2.91.8×1031.8 \times10^{3}

Table 5. Spread S=max⁡qτq/min⁡qτqS = \max_q \tau_q / \min_q \tau_q for L2L^2 regression across five targets, at N=64N = 64 (global) and K=31K = 31, m=2m = 2 (gated). The gated dictionary stays within a factor of seven in every case.

The gated spread never leaves [2.8,6.6][2.8, 6.6], while the global one ranges over six orders of magnitude, from 6.0 to 5.8×1065.8 \times10^{6}, and exceeds the gated one in every case. The margin is widest for a broad band with unequal amplitudes, and narrowest – a factor of two – for the high-frequency band. The last case is instructive: when every frequency present is high, the global dictionary is uniformly slow rather than differentially slow, so its relative spread is small even though its absolute rates are not.

On the Poisson problem, the benefit of preconditioning is real but target-dependent: Pt\mathcal{P}_t cuts the gated spread from 11.0 to 4.36 on the equal-amplitude target and leaves it essentially unchanged, 10.6 against 10.9, on the increasing-amplitude one. Random phases are the revealing case. There the unpreconditioned spread depends strongly on the draw, ranging from 8.7 to 67.9 over five seeds, while the preconditioned one stays between 1.0 and 1.4 for every seed. Built from the interior stencil alone, Pt\mathcal{P}_t cancels the vanishing-symbol direction and nothing else. A particular right-hand side may excite, which is exactly the scope of the claim in § 6.

Discussion and conclusion

One architectural change – multiplying ReLU features by a partition-of-unity gate – makes the Gram matrix block-banded for any local bilinear form, and so places a matrix-valued symbol between the architecture and the training dynamics. That symbol is what decides the outcome, and the experiments of § 7 show it deciding at the level of the dynamics rather than of matrix spectra alone.

Where the symbol is bounded away from zero, gating alone suffices. For L2L^2 regression it flattens the condition-number curve against NN and compresses the spread of frequency-dependent rise times from 1.9×1051.9 \times10^5 to 4.54.5 at matched size, and the effect survives when the thresholds are trained as well: the gated network then resolves all ten frequencies of the target, the global one only the lowest.

Where the symbol vanishes, no amount of gating helps. For the H1H^1 and Poisson case gating leaves the Θ(N2)\Theta(N^2) growth untouched and does more than simply fail: it inverts the ordering, turning q=1q = 1 into the slowest mode and raising the spread from 2.232.23 to 11.011.0. The remedy comes from the same symbol rather than from another architecture – a preconditioner read off the interior stencil replaces faf_a by g−1fag^{-1}f_a and returns the spread to 4.364.36. What the architecture cannot do falls to the optimizer, and which of the two must act is decided by the bilinear form.

Neither conclusion is tuned to one target: across the five of Table 5 the gated spread stays within [2.8,6.6][2.8, 6.6] while the global one ranges from 6.06.0 to 5.8×1065.8 \times10^6. And the quantity carrying all of this is the spectral-bias factor, not the condition number; the two coincide only when the extreme eigenvalues lie at the extreme frequencies.

Design guidelines

The framework of § 5 turns the usual trial and error of picking a shallow architecture into a few explicit recommendations.

  • For L2L^2 losses, grow KK, not mm. Once the dictionary is gated, adding windows adds resolvable frequencies at no conditioning cost, whereas adding knots per window adds local approximation power without narrowing the gap between frequencies. Faced with slow high-frequency convergence from a global dictionary, the first thing to try is a gated one at comparable parameter count, before a different activation, a different loss weighting, or a larger network.

  • For H1H^1 losses, match the preconditioner to the bilinear form. The stiffness symbol vanishes at zero frequency whatever the gating, so the Θ(N2)\Theta(N^2) conditioning belongs to the second-order form and not to the choice between global and local features. What restores balanced convergence is a preconditioner built from the same block stencil that produces the block-Toeplitz structure in the first place (§ 6).

  • Do not use coordinate-adaptive optimizers to test spectral-bias claims. Adam’s per-coordinate normalization rescales exactly the per-mode rates that Lemma 1 identifies as the carrier of spectral bias, which is what the adaptive-kernel view of Kopaničáková and Riccietti [20] predicts a coordinate-wise preconditioner will do to a mode-diagonal problem. We therefore use plain gradient descent throughout. This concerns how the diagnostic should be read, and is not a claim that Adam should be avoided in practice.

Relation to the NTK and to general-purpose optimizers

We analyze plain gradient descent on a fixed dictionary rather than Adam or a quasi-Newton method, and the reason is worth stating, since much of the SciML optimization literature is about designing better optimizers for exactly these settings. The survey of Kopaničáková and Riccietti [20] illustrates this: every first- and second-order method used for such models is described by the same adaptive kernel Θk(Mk)=1mJkMk−1JkT\Theta_k^{(M_k)} = \frac{1}{m}J_kM_k^{-1}J_k^{\mathsf{T}}, with Mk=IM_k = I recovering gradient descent, diagonal second-moment estimates recovering Adam, and the Gauss–Newton or Fisher matrix recovering natural-gradient methods.

The mode-wise error then decomposes as ek=∑ici(1−αλi)kqie_k = \sum_i c_i(1-\alpha\lambda_i)^k q_i in the eigenbasis of Θk(Mk)\Theta_k^{(M_k)}, which is Lemma 1 with GaG^a in the role of Θk(I)\Theta_k^{(I)}. Every Mk≠IM_k \ne I is therefore an attempt to cluster that spectrum from the optimizer side rather than the architecture side, and holding Mk=IM_k = I while varying the architecture is what isolates the architectural half in closed form. The preconditioners of § 6 are themselves instances of Mk≠IM_k \ne I, distinguished from general-purpose optimizers only in being read directly off the block-banded structure of GaG^a instead of estimated adaptively – which is what makes them analyzable.

The same view sharpens the role of the two bilinear forms. Kopaničáková and Riccietti [20] report that for a PINN loss built from a PDE of order pp, the conditioning scales as κ≈(ωmax⁡/ωmin⁡)2p\kappa\approx(\omega_{\max}/\omega_{\min})^{2p}. Our L2L^2 (p=0p = 0) and H1H^1 (p=1p = 1) cases are the two smallest instances. Because the dictionaries here are finite-dimensional and explicit rather than infinite-width limits, we obtain that law constructively for shallow ReLU-type models, with the symbol responsible for each exponent written down in § 5.

Limitations and pathways forward

The analysis is deliberately narrow in three respects, each of which points to a next step rather than to an obstruction.

Everything here is one-dimensional. The block-Toeplitz machinery of § 5 generalizes to block-multilevel-Toeplitz structure in higher dimensions [28, 31], so carrying the partition-of-unity construction and its symbol to 2D is the natural route toward PDEs on domains.

The boundary treatment of § 4 is not entirely rigorous. A more accurate variant would sharpen the boundary-layer terms of the decomposition, but it would not change the interior symbol or anything that follows from it.

We did not extend the trainable-threshold bridge to the H1H^{1} case, where training the thresholds alongside a preconditioned output layer couples fast and slow dynamics in a way we did not find robust. More broadly, carrying the present closed-form analysis to deep architectures, where the Gram matrix itself evolves during training, would connect these results to the NTK evolution literature of § 2 and to the adaptive-kernel framework above. The symbol would no longer be fixed, but it would still be the object to track.

Acknowledgements

The author thanks Jennifer Pestana for helpful discussions on the spectral analysis of structured matrices and on preconditioning, which shaped the treatment in § 5 and § 6.

Declaration of competing interest

The author declares that she has no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data and code availability

All matrices in this paper are assembled in closed form from the stencils of Appendix § 12, and every experiment uses only NUMPY, SCIPY and MATPLOTLIB. The scripts that produce each figure and table, together with the assembly and diagnostic libraries they depend on, are available from the author on request.

Explicit interior and boundary stencils for m=1m = 1 and m=2m = 2

This appendix contains the explicit computations underlying Corollaries 9 and 10 and the proof sketch of Theorem 5. Throughout, h:=1/Kh := 1/K and σ(z):=max⁡{z,0}\sigma(z) := \max\{z,0\}. The integrals below are evaluated on the unit interval [0,1][0,1] in the continuous L2L^{2} inner product; the empirical version yields the same blocks up to discretization error and is omitted.

The case m=1m = 1

We choose a single knot per gate, s0:=0s_{0} := 0 and sk:=xk−1s_{k} := x_{k-1} for 1≤k≤K1 \leq k \leq K, and set φk(x):=ψk(x)σ(x−sk)\varphi_{k}(x) := \psi_{k}(x)\sigma(x-s_{k}). By Proposition 3 the Gram matrix GgateK∈R(K+1)×(K+1)G^{\mathrm{gate}K} \in\mathbb{R}^{(K+1)\times(K+1)} is tridiagonal up to boundary effects.

Proposition 13 (Explicit stencil, m=1m = 1). The entries of GgateKG^{\mathrm{gate}K} are

GkkgateK=1115h3,1≤k≤K−1,(48)G^{\mathrm{gate}K}_{kk} = \frac{11}{15}h^{3}, \qquad1 \leq k \leq K-1, \tag*{(48)}
Gk,k+1gateK=215h3,1≤k≤K−2,(49)G^{\mathrm{gate}K}_{k,k+1} = \frac{2}{15}h^{3}, \qquad1 \leq k \leq K-2, \tag*{(49)}
Gkk′gateK=0,∣k−k′∣>1,G^{\mathrm{gate}K}_{kk'} = 0, \qquad|k-k'|>1,

together with the boundary entries

G00gateK=130h3,G01gateK=120h3,GKKgateK=15h3,GK−1,KgateK=215h3.(50)G^{\mathrm{gate}K}_{00} = \frac{1}{30}h^{3},\qquad G^{\mathrm{gate}K}_{01} = \frac{1}{20}h^{3},\qquad G^{\mathrm{gate}K}_{KK} = \frac{1}{5}h^{3},\qquad G^{\mathrm{gate}K}_{K-1,K} = \frac{2}{15}h^{3}. \tag*{(50)}

Proof. For 1≤k≤K−11 \leq k \leq K-1, the feature φk=ψkσ(⋅−xk−1)\varphi_k = \psi_k \sigma(\cdot-x_{k-1}) is supported on Ik∪Ik+1=[xk−1,xk+1]I_k \cup I_{k+1} = [x_{k-1},x_{k+1}] and is a piecewise quadratic that vanishes at the endpoints. Splitting the integral ∫01φk2 dx\int_0^1 \varphi_k^2\,\mathrm{d}x into the contributions from IkI_k and Ik+1I_{k+1} and rescaling each to [0,1][0,1] via x=xk−1+hx~x=x_{k-1}+h\tilde{x} reduces every integral to a fixed polynomial integral on [0,1][0,1], with

three factors of hh extracted (two from the ReLU ramp σ(x−xk−1)=hξ\sigma(x-x_{k-1})=h\xi on the relevant subinterval, one from dx\mathrm{d}x). Summing the two contributions gives 1115h3\frac{11}{15}h^3, which is (48). The off-diagonal entry Gk,k+1gateKG^{\mathrm{gate}K}_{k,k+1} involves only the overlap window Ik+1I_{k+1} on which ψk\psi_k and ψk+1\psi_{k+1} are simultaneously nonzero; the same rescaling reduces it to 215h3\frac{2}{15}h^3, giving (49). The boundary entries (50) follow from the one-sided supports of ψ0\psi_0 and ψK\psi_K by the same procedure, with the only difference being that one of the two interior contributions is missing.

Toeplitz splitting and proof of Corollary 9

Define TK+1∈R(K+1)×(K+1)T_{K+1}\in\mathbb{R}^{(K+1)\times(K+1)} as the symmetric tridiagonal Toeplitz matrix with a0=11/15a_0=11/15 on the diagonal and a1=2/15a_1=2/15 on the sub-/super-diagonal. By Proposition 13,

GgateK=h3TK+1+h3RK+1,(51)G^{\mathrm{gate}K}=h^3T_{K+1}+h^3R_{K+1}, \tag*{(51)}

where RK+1R_{K+1} is supported on rows/columns {0,1,K}\{0,1,K\} with explicit entries (RK+1)00=130−a0=−710(R_{K+1})_{00}=\frac{1}{30}-a_0=-\frac{7}{10}, (RK+1)01=(RK+1)10=120−a1=−112(R_{K+1})_{01}=(R_{K+1})_{10}=\frac{1}{20}-a_1=-\frac{1}{12}, (RK+1)KK=15−a0=−815(R_{K+1})_{KK}=\frac{1}{5}-a_0=-\frac{8}{15}; all other entries vanish. Hence rank⁡(RK+1)≤3\operatorname{rank}(R_{K+1})\le3. The eigenvalues of TK+1T_{K+1} are explicit (Ekström et al. [15]), μr(TK+1)=a0+2a1cos⁡(rπ/(K+2))∈[a0−2a1,a0+2a1]=[7/15,1]\mu_r(T_{K+1})=a_0+2a_1\cos(r\pi/(K+2))\in[a_0-2a_1,a_0+2a_1]=[7/15,1].

Since rank⁡(RK+1)≤3\operatorname{rank}(R_{K+1})\le3, Weyl’s inequality for finite-rank perturbations gives λi(GK)≥μi+3(TK+1)≥7/15\lambda_i(\mathcal{G}_K)\ge\mu_{i+3}(T_{K+1})\ge7/15 for 1≤i≤K−21\le i\le K-2 (eigenvalues in decreasing order), so at most three eigenvalues of GK\mathcal{G}_K fall below 7/157/15. For the upper bound, Gershgorin applied to GK\mathcal{G}_K gives the disc [7/15,1][7/15,1] for the interior rows, while rows 00, 11 and KK have centres 1/301/30, 11/1511/15, 1/51/5 and radii 1/201/20, 11/6011/60, 2/152/15; hence λmax⁡(GK)≤1\lambda_{\max}(\mathcal{G}_K)\le1.

Gershgorin yields no useful lower bound: rows 00 and KK have discs [−1/60,1/12][-1/60,1/12] and [1/15,1/3][1/15,1/3], which lie below 7/157/15, the first one not even inside [0,∞)[0,\infty). Nor does the perturbation bound, since ∥RK+1∥2≈0.710>7/15\lVert R_{K+1}\rVert_2\approx0.710>7/15. The KK-uniform positive lower bound is obtained instead by a cell-wise coercivity argument. On IkI_k only φk−1\varphi_{k-1} and φk\varphi_k are supported, and the change of variables x=xk−1+hηx=x_{k-1}+h\eta turns them into the fixed pair h(1−η2,η2)h(1-\eta^2,\eta^2) for 2≤k≤K2\le k\le K, respectively h(η(1−η),η2)h(\eta(1-\eta),\eta^2) for k=1k=1, with local Gram matrices

h3M^=h3(8/152/152/151/5),h3M^(1)=h3(1/301/201/201/5),h^3\widehat{M}=h^3\begin{pmatrix}8/15&2/15\\2/15&1/5\end{pmatrix},\qquad h^3\widehat{M}^{(1)}=h^3\begin{pmatrix}1/30&1/20\\1/20&1/5\end{pmatrix},

whose smallest eigenvalues are (11−41)/30≈0.1532(11-\sqrt{41})/30\approx0.1532 and (7−34)/60≈0.01948(7-\sqrt{34})/60\approx0.01948. Since every index occurs in at least one cell,

v⊤GKv=∑k=1K(vk−1vk) ⁣⊤M^(k)(vk−1vk)≥min⁡kλmin⁡(M^(k))∑k=1K(vk−12+vk2)≥7−3460∥v∥2,v^\top\mathcal{G}_Kv=\sum_{k=1}^{K}\begin{pmatrix}v_{k-1}\\v_k\end{pmatrix}^{\!\top}\widehat{M}^{(k)}\begin{pmatrix}v_{k-1}\\v_k\end{pmatrix}\ge\min_k\lambda_{\min}\bigl(\widehat{M}^{(k)}\bigr)\sum_{k=1}^{K}(v_{k-1}^2+v_k^2)\ge\frac{7-\sqrt{34}}{60}\lVert v\rVert^2,

which is the lower bound in (30); the condition-number bound follows from 60/(7−34)=4(7+34)60/(7-\sqrt{34})=4(7+\sqrt{34}). The same argument applies verbatim for any mm and gives Theorem 8.

The case m=2m=2

For m=2m=2 we choose two knots per gate. With s0,1=0s_{0,1}=0, sk,1=xk−1s_{k,1}=x_{k-1}, sk,2=xks_{k,2}=x_k (1≤k≤K−11\le k\le K-1) and sK,1=xK−1s_{K,1}=x_{K-1}, the same change-of-variables computation as in the m=1m=1 case gives the explicit interior stencil:

Proposition 14 (Explicit stencil, m=2m=2). For 1≤k≤K−11\le k\le K-1, the diagonal block B=Bk∈R2×2B=B_k\in\mathbb{R}^{2\times2} and the off-diagonal block C=Ck∈R2×2C=C_k\in\mathbb{R}^{2\times2} (1≤k≤K−21\le k\le K-2) of GgateKG^{\mathrm{gate}K} satisfy

B=h3(11/157/607/601/30),C=h3(2/151/2000).(52)B=h^3\begin{pmatrix}11/15&7/60\\7/60&1/30\end{pmatrix},\qquad C=h^3\begin{pmatrix}2/15&1/20\\0&0\end{pmatrix}. \tag*{(52)}

The remaining within-block entries vanish, G(k,1),(k+1,2)gateK=G(k,2),(k+1,2)gateK=0G^{\mathrm{gate}K}_{(k,1),(k+1,2)}=G^{\mathrm{gate}K}_{(k,2),(k+1,2)}=0. Boundary blocks are

G(0,1),(0,1)gateK=130h3,G(0,1),(1,1)gateK=120h3,G(K,1),(K,1)gateK=15h3,G^{\mathrm{gate}K}_{(0,1),(0,1)}=\frac{1}{30}h^3,\qquad G^{\mathrm{gate}K}_{(0,1),(1,1)}=\frac{1}{20}h^3,\qquad G^{\mathrm{gate}K}_{(K,1),(K,1)}=\frac{1}{5}h^3,
G(K−1,j),(K,1)gateK=G(k,j),(k+1,1)gateK∣interior.(53)G^{\mathrm{gate}K}_{(K-1,j),(K,1)}=G^{\mathrm{gate}K}_{(k,j),(k+1,1)}\big|_{\mathrm{interior}}. \tag*{(53)}

Proof. Same argument as Proposition 13, applied separately to each pair of indices (j,j′)∈{1,2}2(j,j')\in\{1,2\}^2. The vanishing entry C12=0C_{12}=0 reflects the fact that sk,1=xk+1s_{k,1}=x_{k+1} falls strictly to the right of supp⁡(ψk)∩supp⁡(ψk+1)=Ik+1\operatorname{supp}(\psi_k)\cap\operatorname{supp}(\psi_{k+1})=I_{k+1}, so the corresponding integrand is zero on the overlap. The boundary blocks differ from B,CB,C only because ψ0\psi_0 and ψK\psi_K are one-sided, and the entries involving s0,2s_{0,2} or sK,2s_{K,2} that would be nontrivial in the interior are absent at the boundary.

Block-Toeplitz approximation and proof of Corollary 10. Let TK+1∈R2(K+1)×2(K+1)T_{K+1} \in\mathbb{R}^{2(K+1)\times2(K+1)} be the block-Toeplitz matrix with constant block-stencil (CT,B,C)(C^{\mathsf{T}}, B, C) given by (52) (rescaled by h−3h^{-3}, so we work directly with the dimensionless versions). The boundary correction RK:=h−3GKgate−TK+1cutR_K := h^{-3}G_K^{\mathrm{gate}} - T_{K+1}^{\mathrm{cut}}, where TK+1cutT_{K+1}^{\mathrm{cut}} denotes TK+1T_{K+1} with the first and last rows/columns removed, has rank at most 4 by direct inspection of (53); its operator norm satisfies ∥RK∥≤1/20\lVert R_K\rVert\le1/20.

The matrix-valued symbol is f(θ)=CTe−iθ+B+Ceiθf(\theta) = C^{\mathsf{T}}e^{-\mathrm{i}\theta} + B + Ce^{\mathrm{i}\theta}, which for the blocks (52) has two real eigenvalue branches θ↦λ1(f(θ))<λ2(f(θ))\theta\mapsto\lambda_1(f(\theta)) < \lambda_2(f(\theta)). Numerical evaluation gives mf=min⁡θλ1≈0.005405m_f = \min_\theta\lambda_1 \approx0.005405 and Mf=max⁡θλ2≈1.0279M_f = \max_\theta\lambda_2 \approx1.0279. By the symbol theorem of Ekström et al. [15]; Böttcher and Grudsky [7], every eigenvalue of TK+1T_{K+1} lies in [mf,Mf][m_f,M_f], and as K→∞K \to\infty the eigenvalues distribute themselves on the two branches (approximately uniform sampling of θ∈[−π,π]\theta\in[-\pi,\pi] on each branch). Combining with the rank-4 Courant–Fischer perturbation ∥RK∥≤1/20\lVert R_K\rVert\le1/20 yields the enclosure of Corollary 10 with c2≥mf−∥RK∥≈−0.044c_2 \ge m_f - \lVert R_K\rVert\approx-0.044, which is non-binding, and C2≤Mf+∥RK∥≈1.078C_2 \le M_f + \lVert R_K\rVert\approx1.078. The numerically observed values, c2≈0.005405c_2 \approx0.005405 and C2≈1.028C_2 \approx1.028, are far tighter than these analytic bounds; indeed c2c_2 reproduces mfm_f to four digits, so the rank-4 boundary perturbation does not move the bottom of the spectrum out of the symbol range at all.

References

  1. [1]Anderson, S., Dolean, V., Moseley, B., Pestana, J., 2026. ELM-FBPINNs: an efficient multilevel random feature method. Machine Learning for Computational Science and Engineering 2. ArXiv:2409.01949.arxiv.org/abs/2409.01949
  2. [2]Arora, S., Du, S., Hu, W., Li, Z., Wang, R., 2019. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks, in: Proceedings of the 36th International Conference on Machine Learning, PMLR. pp. 322–332.arxiv.org/abs/1901.08584
  3. [3]Babuška, I., Melenk, J.M., 1997. The partition of unity method. International Journal for Numerical Methods in Engineering 40, 727–758.doi.org/10.1002/(sici)1097-0207(19970228)40:4<727::aid-nme86>3.0.co;2-n
  4. [4]Basri, R., Galun, M., Geifman, A., Jacobs, D., Kasten, Y., Kritchman, S., 2020. Frequency bias in neural networks for input of non-uniform density, in: Proceedings of the 37th International Conference on Machine Learning, PMLR. pp. 685–694.
  5. [5]Basri, R., Jacobs, D., Kasten, Y., Kritchman, S., 2019. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems 32.
  6. [6]van Beek, J.W., Dolean, V., Moseley, B., 2026. Local feature filtering for scalable and well-conditioned domain-decomposed random feature methods. Computer Methods in Applied Mechanics and Engineering 449, 118583. ArXiv:2506.17626.
  7. [7]Böttcher, A., Grudsky, S.M., 2005. Spectral Properties of Banded Toeplitz Matrices. SIAM.DOI
  8. [8]Cao, Y., Fang, Z., Wu, Y., Zhou, D.X., Gu, Q., 2021. Towards understanding the spectral bias of deep learning, in: Zhou, Z.H. (Ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, International Joint Conferences on Artificial Intelligence Organization. pp. 2205–2211. URL: https://doi.org/10.24963/ijcai.2021/304, doi:10.24963/ijcai.2021/304.
  9. [9]Chan, T.F., 1988. An optimal circulant preconditioner for Toeplitz systems. SIAM Journal on Scientific and Statistical computing 9, 766–771.
  10. [10]Chen, J., Chi, X., E., W., Yang, Z., 2022. Bridging traditional and machine learning-based algorithms for solving PDEs: the random feature method. arXiv:2207.13380.
  11. [11]Dolean, V., Heinlein, A., Mishra, S., Moseley, B., 2024. Multilevel domain decomposition-based architectures for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering 429, 117116.
  12. [12]Dolean, V., Jolivet, P., Nataf, F., 2015. An Introduction to Domain Decomposition Methods. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA. doi:10.1137/1.9781611974065.ch1.
  13. [13]Dong, S., Li, Z., 2021. Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Computer Methods in Applied Mechanics and Engineering 387, 114129. doi:10.1016/j.cma.2021.114129.
  14. [15]Ekström, S.E., Furci, I., Serra-Capizzano, S., 2018. Exact formulae and matrix-less eigensolvers for block banded symmetric Toeplitz matrices. BIT Numerical Mathematics 58, 937–968.
  15. [16]Heinlein, A., Kapoor, T., 2025. Domain decomposition architectures and Gauss–Newton training for physics-informed neural networks. arXiv:2510.27018.
  16. [17]Hong, Q., Siegel, J.W., Tan, Q., Xu, J., 2022. On the activation function dependence of the spectral bias of neural networks. arXiv preprint arXiv:2208.004924.
  17. [18]Jacot, A., Gabriel, F., Hongler, C., 2018. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems 31.
  18. [19]Khodakarami, S., Oommen, V., Ahmadi Daryakenari, N., Beekenkamp, M., Karniadakis, G.E., 2026. Spectral bias in physics-informed and operator learning: analysis and mitigation guidelines. arXiv:2602.19265.
  19. [20]Kopaničáková, A., Riccietti, E., 2026. Introduction to optimization methods for training SciML models. arXiv:2601.10222. arXiv:2601.10222.
  20. [21]Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halvorsen, J., Soljacic, M., Hou, T., Tegmark, M., 2025. KAN: Kolmogorov–Arnold networks, in: Yue, Y., Garg, A., Peng, N., Sha, F., Yu, R. (Eds.), International Conference on Learning Representations, pp. 70367–70413.DOI
  21. [22]Ma, C., Wu, L., et al., 2020. Machine learning from a continuous viewpoint, I. Science China Mathematics 63, 2233–2266.
  22. [23]Mei, S., Montanari, A., Nguyen, P.M., 2018. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences 115, E7665–E7671.
  23. [24]Meshir, J.D., Palafox, A., Guerrero, E.A., 2025. On the study of frequency control and spectral bias in wavelet-based Kolmogorov–Arnold networks: a path to physics-informed KANs. arXiv preprint arXiv:2502.00280. Moseley, B., Markham, A., Nissen-Meyer, T., 2023. Finite basis physics-informed neural networks (FBPINNs): a scalable domain decomposition approach for solving differential equations. Advances in Computational Mathematics 49, 62.
  24. [25]Müller, J., Zeinhofer, M., 2023. Achieving high accuracy with PINNs via energy natural gradient descent, in: International Conference on Machine Learning, PMLR. pp. 25471–25485.arxiv.org/abs/2302.13163
  25. [26]Rahaman, N., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., Courville, A., 2019. On the spectral bias of neural networks, in: Chaudhuri, K., Salakhutdinov, R. (Eds.), Proceedings of the 36th International Conference on Machine Learning, pp. 5301–5310.arxiv.org/abs/1806.08734
  26. [28]Serra, S., 1999. Spectral and computational analysis of block Toeplitz matrices having nonnegative definite matrix-valued generating functions. BIT Numerical Mathematics 39, 152–175.DOI
  27. [29]Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G., 2020. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems 33, 7462–7473.arxiv.org/abs/2006.09661
  28. [30]Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R., 2020. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33, 7537–7547.arxiv.org/abs/2006.10739
  29. [31]Tilli, P., 1998. A note on the spectral distribution of Toeplitz matrices. Linear and Multilinear Algebra 45, 147–159.DOI
  30. [32]Toselli, A., Widlund, O.B., 2005. Domain Decomposition Methods — Algorithms and Theory. Springer.DOI
  31. [33]Wang, S., Wang, H., Perdikari, P., 2021. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering 384, 113938.DOI
  32. [34]Wang, S., Yu, X., Perdikari, P., 2022. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics 449, 110768.arxiv.org/abs/2007.14527
  33. [35]Wang, Y., Siegel, J.W., Liu, Z., Hou, T.Y., 2025. On the expressiveness and spectral bias of KANs, in: International Conference on Learning Representations.arxiv.org/abs/2410.01803

Paper details

vlnc:W_f246ef45
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents