betaFeedback

WorksComputer Science

Continual Classification Beyond the Kernel Regime: Harnessing the Sparsity Bias of L2 Regularization in Diagonal Linear Networks

Abstract

We study logistic regression for continual learning with linear predictors in a high-dimensional statistical setting. Existing theoretical work has largely focused on continual linear regression with variants of L2 regularization, motivated by their analytical simplicity and connection to the kernel regime of neural networks. We first derive the optimal L2 regularization strength for continual linear classification and characterize its generalization performance, recovering behavior analogous to prior results for continual linear regression. Next, we examine the effect of L2 regularization in deeper models using a minimal two-layer diagonal linear network. In the single-task setting, L2 regularization on the weights of such models is known to promote sparsity in the resulting predictor, yielding a rich, sparse feature-learning regime beyond the kernel regime (a similar effect arises in two-layer ReLU networks). However, in continual learning, where L2 regularization is typically centered on previous-task weights, this sparsity bias is no longer preserved. We therefore propose a weighted variant of L2 regularization that extends this beneficial sparsity bias to continual learning. We show theoretically and empirically that, for a sufficiently sparse teacher, our variant converges substantially faster than directly learning a linear predictor with conventional continual L2 or L1 regularization, trading some long-horizon bias for markedly improved performance over moderate task horizons.

Introduction

In continual learning (CL), a learner sequentially learns from a stream of tasks. The key challenge is to accumulate expertise by learning new tasks without forgetting previously seen ones [12] (McCloskey & Cohen 1989; Lopez-Paz & Ranzato 2017; Parisi et al. 2019). In modern large models, continual adaptation is especially important, as environments evolve over time (e.g., in user-facing agents, e-commerce, and social networks), requiring models to incorporate new knowledge while retaining previously acquired capabilities, whereas retraining from scratch requires enormous computational resources.

We theoretically study continual classification in a statistical setting. Specifically, our work provides a precise high-dimensional characterization of continual logistic regression with linear predictors under Gaussian features. Such statistical settings have recently facilitated theoretical advances in continual linear regression, shedding light on factors such as task similarity, model complexity, and regularization strength (e.g., (Lin et al., 2023; Zhao et al., 2025; Karpel et al., 2026). The framework we develop allows us, for the first time, to theoretically study both L2 and other regularizers in a statistical setting for continual classification, focusing on the effects of the number of learned tasks and the amount of sparsity in the system.

Regularization methods have been a central focus of the continual learning community (e.g., (Kirkpatrick et al. 2017; see survey in Shi et al. 2025). Typically, these works incorporate an additive L2 regularizer in the objective of task tt, i.e., minimize Lt(w)+λ∥w−wt−1∥M2\mathcal{L}_t(\mathbf{w}) + \lambda\|\mathbf{w} - \mathbf{w}_{t-1}\|_{\mathbf{M}}^2 over w\mathbf{w}, with Lt\mathcal{L}_t being the per-task loss, wt−1\mathbf{w}_{t-1} the solution to the previous task, and M\mathbf{M} typically being diagonal (e.g., [6]) or even simply isotropic, M=I\mathbf{M} = \mathbf{I} (e.g., Lubana et al., 2022).

While empirical works have focused on regularizing changes in the weights of neural networks (NNs), the theoretical literature on regularization has mostly analyzed shallow, linear models (e.g., Li et al., 2023, 2026; Cai & Diakonikolas, 2025; Levinstein et al., 2025; Karpel et al., 2026). Thus, it remains unclear whether even isotropic L2 regularization induces a similar effect or algorithmic bias in linear models as it does in NNs. For example, for a single task, minimizing L2 regularization over the weights of a two-layer ReLU network is equivalent to minimizing L1 regularization on the second layer (Neyshabur et al., 2014, 2015; Savarese et al., 2019). In the infinite-width limit, this further corresponds to sparse penalties on the predictor in function space (Savarese et al., 2019; Ongie et al., 2020). Similar effects arise in other homogeneous models, where L2 regularization on the weights of depth-hh models typically translates into “sparser” penalties, such as the Lp quasi-norm with p=2/hp = 2/h for linear networks [9] and certain deep ReLU networks (Savarese et al., 2019; Zhang & Wang, 2023).

Such explicit or implicit sparse penalties are often viewed as driving the rich feature-learning behavior of NNs, enabling them to outperform standard kernel methods (Woodworth et al., 2020; Parhi & Nowak, 2022). Even in simple linear settings, when a sparse solution exists, an L1 penalty yields sample complexity with only logarithmic dependence on the dimension dd, compared with the linear dependence under L2 regularization (Ng, 2004). In other words, sparsity bias may be a key mechanism enabling overparameterized models (e.g., NNs) to generalize.

So far, the effect of sparsity bias in continual learning has not been explored theoretically. Intuitively, sparse solutions may interfere less across tasks and thereby reduce forgetting. As our paper shows, this intuition can be effective, but only when the sparsity-inducing penalty is designed appropriately. A natural candidate is the L1 regularizer ∥w−wt−1∥1\lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_1, the direct analogue of the standard L2 penalty ∥w−wt−1∥22\lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_2^2. We prove that this L1 regularizer can exploit a sparse ground truth, with excess risk decaying nearly linearly in the sparsity factor ss. However, both of these continual regularizers are nonhomogeneous in w\mathbf{w}, making it hard to align them with the natural sparsity bias we observe in a single task of deeper models.

We design a continual regularizer that avoids this non-homogeneity pitfall, preserving a direct connection to the literature on sparsity bias. Specifically, we introduce a homogeneous and analyzable “sign weighted L1” regularizer in predictor space, defined as

f(w,wt−1)=∑i=1d∣wi∣[sign⁡(wi)wt−1,i]++ϵ.f(\mathbf{w},\mathbf{w}_{t-1}) = \sum_{i=1}^{d} \frac{|w_i|}{[\operatorname{sign}(w_i)w_{t-1,i}]_{+}+\epsilon}.

Intuitively, this regularizer promotes sparsity in the predictor while encouraging preservation of the coordinate-wise signs of the previous iterate (see Figure 1).

Coordinate-wise comparison of L1 and our sign-weighted L1 regularization

Figure 1. Coordinate-wise comparison of L1 and our sign-weighted L1 regularization. (a) Previously inactive coordinate (wt−1,i=0w_{t-1,i}=0). SW-L1 and L1 coincide, so activation is equally penalized in either direction. (b) Previously positive coordinate (wt−1,i>0w_{t-1,i}>0). SW-L1 penalizes same-sign updates less than sign flips. Here wt−1,i=ϵ=1w_{t-1,i}=\epsilon=1.

Importantly, our proposed regularizer arises naturally at the predictor level when weighted L2 regularization is applied to the parameter space of a two-layer diagonal linear network (DLN; Woodworth et al. 2020). Its benefit goes beyond this connection to the sparsity-bias literature: we show theoretically and empirically that this regularizer achieves a substantially better convergence rate and exhibits less forgetting than the direct L1 analogue.

Specifically, our contributions are:

  • We develop the first high-dimensional continual classification, deriving an analytic expression for the asymptotic performance of general regularized continual logistic regression.

  • In a 1-layer setting (where simply y^=wTx\hat{y} = \mathbf{w}^{\mathsf{T}}\mathbf{x}):

    • We derive the optimal L2 regularization strength and excess risk in continual linear classification in high dimensions. We show that the optimal strength scales with the task horizon TT as T/log⁡TT/\log T and the excess risk scales as log⁡T/T\log T/T, mirroring the linear regression case (Karpel et al., 2026).

    • For the direct L1 regularizer ∥w−wt−1∥1\lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_1, we establish global convergence guarantees with near-linear dependence on the sparsity factor ss, namely slog⁡(1s)s\log\left(\frac{1}{s}\right), but only polylogarithmic dependence on the task horizon TT.

  • In a two-layer DLN (where y^=∑jθ(1),jθ(2),jxj\hat{y} = \sum_j \theta_{(1),j}\theta_{(2),j}x_j):

    • We design a weighting scheme for L2 regularization on the weight space that induces a sign weighted L1 regularizer the predictor space.

    • We prove that the sign weighted regularizer attains a tight linear dependence on ss.

  • We empirically validate, using linear models and neural networks, that sign weighted L1 outperforms competing regularizers in the presence of sparsity.

Setting: Regularized Continual Logistic Regression

Continual classification tasks. In our setting, the learner receives a sequence of TT tasks {(Xt,yt)}t=1T\{(X_t,y_t)\}_{t=1}^{T}, each having nn samples. In each iteration tt a learner is exposed to the samples from the current task (Xt,yt)(X_t,y_t), with no access to data from previous tasks.

Statistical data model: Gaussian features and a noisy linear teacher. For each task tt, datapoints are sampled i.i.d. from the Gaussian distribution, i.e., {xt,i}i=1n∼iidN(0d,d−1I)\{x_{t,i}\}_{i=1}^{n} \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0_d,d^{-1}I) independently of all other tasks. Labels are drawn from a logistic model using a teacher vector w⋆∈Rd\mathbf{w}_{\star}\in\mathbb{R}^{d} whose entries are sampled independently from a distribution Π\Pi, and a label-flip noise level of ρ∈[0,12)\rho\in[0,\frac{1}{2}). That is, for any task t=1,…,Tt=1,\ldots,T and sample i=1,…,ni=1,\ldots,n, the label is drawn as

P(yt,i=1∣xt,i)=1−P(yt,i=0∣xt,i)=ρ+(1−2ρ)ext,iTw⋆1+ext,iTw⋆.(1)\mathbb{P}(y_{t,i}=1\mid x_{t,i})=1-\mathbb{P}(y_{t,i}=0\mid x_{t,i}) =\rho+(1-2\rho)\frac{e^{x_{t,i}^{\mathsf{T}}\mathbf{w}_{\star}}}{1+e^{x_{t,i}^{\mathsf{T}}\mathbf{w}_{\star}}}. \tag*{(1)}

This setting has been extensively studied in the deep learning theory literature [17] (Salehi et al. 2019; Thrampoulidis et al. 2020).

Studied algorithm: Regularized continual logistic regression. To learn sequentially without forgetting, we follow a large body of continual learning works that penalize deviations from the previous predictor (Kirkpatrick et al., 2017; Schwarz et al., 2018; Zhao et al., 2024) using weighted L2 regularization. Extending the theoretical analysis of continual learning with weighted L2 regularization on linear predictors (i.e., depth-one neural networks) to deep neural networks is nontrivial. We therefore consider a depth-hh diagonal linear network (DLN), which is a popular [2] (Woodworth et al. 2020; Nacson et al. 2022) minimal model for the effect of depth, of the form

w(θ)=θ(1)⊙⋯⊙θ(h),\mathbf{w}(\boldsymbol{\theta})=\boldsymbol{\theta}_{(1)}\odot\cdots\odot\boldsymbol{\theta}_{(h)},

where θ=[θ(1),…,θ(h)]∈Rd×h\boldsymbol{\theta}=[\boldsymbol{\theta}_{(1)},\ldots,\boldsymbol{\theta}_{(h)}]\in\mathbb{R}^{d\times h} are the parameters of all the layers, ⊙\odot denotes component-wise product, and for an input x\mathbf{x} the predictor is y^=wT(θ)x\hat{y}=\mathbf{w}^{\mathsf{T}}(\boldsymbol{\theta})\mathbf{x}. In this paper, we focus on two cases: h=1h=1 and h=2h=2.

Specifically, the objective function incorporates an additive regularization term f(θ,θt−1)f(\boldsymbol{\theta},\boldsymbol{\theta}_{t-1}) to discourage deviations from the previously learned predictor.

Algorithm 1Regularized Continual Logistic Regr.
Input:{(Xt,yt)}t=1T\{(X_t,y_t)\}_{t=1}^{T}, regularization strength λ>0\lambda>0
Initializeθ0=0d\boldsymbol{\theta}_0=0_d
For each iteration t=1,…,Tt=1,\ldots,T:θt=arg⁡min⁡θ∈Rd×h 1n∑i=1nlog⁡(1+ext,iTw(θ)eyt,ixt,iTw(θ))+λdf(θ,θt−1)\boldsymbol{\theta}_t=\underset{\boldsymbol{\theta}\in\mathbb{R}^{d\times h}}{\arg\min}\ \frac{1}{n}\sum_{i=1}^{n}\log\left(\frac{1+e^{x_{t,i}^{\mathsf{T}}\mathbf{w}(\boldsymbol{\theta})}}{e^{y_{t,i}x_{t,i}^{\mathsf{T}}\mathbf{w}(\boldsymbol{\theta})}}\right)+\frac{\lambda}{d}f(\boldsymbol{\theta},\boldsymbol{\theta}_{t-1})
Output:θT\boldsymbol{\theta}_T

Table 1.

We consider regularizers that are convex in their first argument and decompose additively across coordinates. That is, there exists f~:R×R→R≥0\tilde{f}:\mathbb{R}\times\mathbb{R}\to\mathbb{R}_{\geq0} such that,

f(θ,ν)=∑j=1df~(θj,νj)for all θ,ν∈Rd.f(\boldsymbol{\theta},\boldsymbol{\nu})=\sum_{j=1}^{d}\tilde{f}(\theta_j,\nu_j)\quad\text{for all }\boldsymbol{\theta},\boldsymbol{\nu}\in\mathbb{R}^{d}.

Notation. Throughout the paper, we write →d\xrightarrow{d} for convergence in distribution, and define the softplus function ϕ(z)=log⁡(1+ez)\phi(z)=\log(1+e^z). We write an=Θx(bn)a_n=\Theta_x(b_n) if there exist constants 0<cx<Cx<∞0<c_x<C_x<\infty, depending only on xx, such that cxbn≤an≤Cxbnc_xb_n\leq a_n\leq C_xb_n for all sufficiently large nn. Similarly, an=Ox(bn)a_n=O_x(b_n) and an=Ωx(bn)a_n=\Omega_x(b_n) denote the corresponding one-sided bounds. A function f:Rd→Rf:\mathbb{R}^{d}\to\mathbb{R} is called α\alpha homogeneous if f(αw)=αξf(w)f(\alpha\mathbf{w})=\alpha^\xi f(\mathbf{w}) for every α≥0\alpha\geq0, and w∈Rd\mathbf{w}\in\mathbb{R}^{d}. Its proximal operator is defined by

References

  1. [1]Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139–154, 2018.arxiv.org/abs/1711.09601
  2. [2]Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent. In ICML, 2021.arxiv.org/abs/2102.09769
  3. [3]Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183–202, 2009.DOI
  4. [4]Pierre C Bellec and Takuya Koriyama. Existence of solutions to the nonlinear equations characterizing the precise error of m-estimators. arXiv preprint arXiv:2312.13254, 2023.arxiv.org/abs/2312.13254
  5. [5]Mehdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006.11942, 2020.arxiv.org/abs/2006.11942
  6. [6]Frederik Benzing. Unifying importance based regularisation methods for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 2372–2396. PMLR, 2022.DOI
  7. [7]Xufeng Cai and Jelena Diakonikolas. Last iterate convergence of incremental methods as a model of forgetting. In The Thirteenth International Conference on Learning Representations, 2025.
  8. [9]Zhen Dai, Mina Karzand, and Nathan Srebro. Representation costs of linear neural networks: Analysis and design. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 26884–26896. Curran Associates, Inc., 2021.
  9. [11]Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Ŕabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1072–1080, 2021.arxiv.org/abs/2010.04003
  10. [12]Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026):768–774, 2024.DOI
  11. [13]Richard O. Duda, Peter E. Hart, and David G. Stork. Pattern Classification (2nd Edition). Wiley-Interscience, USA, 2000. ISBN 0471056693.
  12. [14]Itay Evron, Edward Moroshko, Gon Buzaglo, Maroun Khriesh, Badea Marjieh, Nathan Srebro, and Daniel Soudry. Continual learning in linear classification on separable data. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 9440–9484. PMLR, 23–29 Jul 2023.arxiv.org/abs/2306.03534
  13. [15]Jianqing Fan and Runze Li. Variable selection via non-concave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456):1348–1360, 2001.
  14. [17]Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, and Kaize Ding. Raptor: Ridge-adaptive logistic probes. arXiv preprint arXiv:2602.00158, 2026.arxiv.org/abs/2602.00158

Paper details

vlnc:W_wy3c3mk5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents