betaFeedback

WorksComputer Science

Classification with Large-Step Gradient Descent: Unifying Self-Stabilization and Convergence Rates

Abstract

Neural networks are often trained with gradient descent (GD) using step sizes that exceed classical stability thresholds, a regime where standard convergence theory no longer applies. Despite exhibiting non-monotonic loss and oscillatory dynamics, this regime frequently leads to faster convergence and better generalization. Existing theoretical perspectives offer complementary but incomplete insights: one identifies a self-stabilization mechanism that drives GD toward flat regions but does not yield convergence rates; another derives rigorous rates for specific losses but does not explain the underlying stabilization mechanism; and a third unifies these views for overparameterized least squares, but relies on the existence of a minimizer manifold, which is absent in classification settings. In this work, we extend this unified geometric perspective to exponential-tail losses, including logistic and cross-entropy losses in separable linear classification, where the loss infimum is not attained at any finite parameter. We identify a curvature-defined reference subspace that replaces the role of the minimizer manifold, yielding a coupled dynamical system in which orthogonal oscillations are damped by parallel progress along directions of decreasing sharpness. Under regularity conditions, which we verify for logistic and multiclass cross-entropy losses on symmetric data models, we derive explicit bounds for the iterates, not just the loss, covering both the unstable and stable regimes, and revealing a three-phase damping structure in the latter. Our analysis further uncovers a transient implicit bias induced by large step sizes: although GD converges asymptotically to the max-margin direction, large steps can leave persistent orthogonal residuals and, in certain geometries, amplify them, affecting finite-time alignment with the max-margin direction.

Introduction

Modern deep learning owes much of its success to optimization heuristics refined through extensive experimentation [1, 2, 3]. Notably, state-of-the-art results often depend on surprisingly simple algorithms such as gradient descent (GD), whose practical effectiveness persistently outpaces our theoretical understanding. One particularly intriguing illustration of this gap is the common use of large learning rates. Classical optimization theory traditionally advocates for small learning rates to ensure stable monotonic convergence [4, 5]. Yet, practitioners routinely exceed these conservative bounds, achieving faster convergence and often better generalization [6, 7, 8, 9, 10]. This counterintuitive regime, known as the edge of stability (EoS), is characterized by non-monotonic loss and oscillatory dynamics [11, 12]. The EoS can arise either from initializing training with a large learning rate, or through progressive sharpening, whereby the local curvature grows along the training trajectory until it reaches the stability boundary [12, 13, 14].

These empirical successes and observations raise fundamental theoretical questions: Why does operating near instability lead to faster convergence? Can we provide rigorous theoretical guarantees for convergence under these oscillatory dynamics?

Several theoretical perspectives have emerged in recent years [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]. Among the most insightful is the self-stabilization mechanism of [26], which shows that GD naturally adjusts to local curvature in response to large-step-size instabilities, preventing divergence. However, their assumptions are difficult to verify analytically and have only been confirmed numerically on specific architectures. Moreover, this framework does not establish convergence rates. Wu et al. [27, 28] take a complementary approach: they derive convergence rates for logistic-type losses through a direct analysis of the gradient structure. However, their arguments do not reveal the mechanistic role of self-stabilization and do not extend to other loss families without a new specialized analysis for each. MacDonald et al. [29] unify both perspectives for overparameterized least squares via a Taylor expansion around a manifold of minimizers, establishing a parallel-orthogonal decomposition that makes the self-stabilization mechanism explicit and simultaneously yields convergence rates. However, this approach fundamentally relies on the existence of a minimizer manifold, which is absent in the classification losses we target.

In this work, we extend this unified approach to exponential-tail losses, a class that includes logistic and cross-entropy losses. The high-level takeaway is:

We develop a unified framework for gradient descent on exponential-tail losses that makes self-stabilization quantitatively explicit and yields convergence rates for the iterates, while revealing a three-phase damping mechanism and a geometry-dependent transient implicit bias.

Specifically, we establish the following:

  1. We identify a subspace defined by the curvature structure of the loss as the geometric anchor for analyzing GD in the exponential-tail setting, and show that decomposing the dynamics relative to it yields a two-equation system: one governs the oscillations orthogonal to the subspace, and the other captures parallel drift along the subspace, where a third-order coupling acts as implicit gradient descent on the curvature, a key feature of the self-stabilization mechanism.

  1. Under certain regularity conditions, which we verify for logistic and cross-entropy under symmetric separable data, we provide a fine-grained convergence analysis with explicit rates for the iterates, not just the loss, revealing a three-phase damping mechanism for the oscillations.

  1. We identify a transient implicit bias of large step sizes: certain orthogonal components can be amplified during the unstable phase and persist at any finite horizon despite asymptotic directional convergence to the max-margin solution.

The remainder of the paper is structured as follows: we begin in § 2 with a review of the related literature. Sections 3–4 develop the general framework; Section 5 verifies it for logistic regression and multiclass classification with the cross-entropy loss; Section 6 empirically validates the theory.

Notation. Throughout, scalars are denoted by lowercase letters (e.g., η,λ\eta, \lambda) and vectors by lowercase bold letters (e.g., x,u\mathbf{x}, \mathbf{u}). We use ∥⋅∥\|\cdot\| for the Euclidean norm and ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle for the standard Euclidean inner product. For a subspace S⊆RdS \subseteq\mathbb{R}^{d}, S⊥S^{\perp} denotes its orthogonal complement, ΠS\Pi_{S} the orthogonal projector onto SS, and ISI_{S} the identity operator on SS. We write f≍gf \asymp g on a set SS if there exist constants c,C>0c,C > 0 such that cg(x)≤f(x)≤Cg(x)c g(x) \leq f(x) \leq C g(x) for all x∈Sx \in S. We use f≲gf \lesssim g and f≳gf \gtrsim g for the corresponding one-sided inequalities up to constants.

Related Work

The EoS phenomenon has motivated a rapidly growing literature on large-step-size training, progressive sharpening, benign oscillations, catapult dynamics, and step-size-dependent implicit bias [12, 30, 10, 31, 32, 33, 9, 11, 13, 14, 34, 20, 17, 35, 23, 22, 15, 18, 16, 36, 19, 25, 37]. We defer a broader discussion of this literature to Appendix § 9 and focus here on the works closest to ours:

General self-stabilization mechanisms. This line of work seeks architecture-independent mechanisms explaining why GD remains stable when the local quadratic model predicts instability. The most influential is the self-stabilization framework of [26]. The key analytical tool is a cubic Taylor

Self-stabilization mechanism?RatesLoss settingReference Geometry
Damian et al. [26]✓✗Abstract conditions1Stable Set
Wu et al. [27, 28]✗✓Logistic-likeMax-margin line
MacDonald et al. [29]✓✓Least squaresMinimizer manifold
This work✓✓Exponential-tailCurvature-defined subspace

Table 1. High-level comparison of related theoretical analyses of GD at the edge of stability.

expansion of the loss, centered at a constrained set defined by the sharpness threshold 2/η2/\eta, which shows that the third-order term induces a negative feedback on the sharpness, causing GD to behave like projected gradient descent under λmax⁡(θ)≤2/η\lambda_{\max}(\theta) \le2/\eta. While this provides a compelling explanation of EoS, it is not designed to yield end-to-end convergence rates and its hypotheses involve local, trajectory-dependent quantities that are difficult to verify analytically.

Quantitative convergence analyses for exponential-tail losses. Wu et al. [38, 27, 28] show that, for logistic regression and related classification losses, large-step GD can preserve the max-margin directional bias while achieving accelerated loss decay, with extensions to regularized objectives, NTK-regime models, and online SGD. Crawshaw and Liu [39] sharpen this in two dimensions, giving tight unstable-to-stable transition-time estimates via a fine-grained analysis of orthogonal oscillations. Our work takes a different perspective: rather than deriving rates from margin-based recursions, we connect convergence to the underlying self-stabilization mechanism. This connection enables a fine-grained analysis of the dynamics at the level of the iterates, not just the loss, while simultaneously providing a mechanistic interpretation of convergence. In particular, it reveals a new phenomenon: when the support vectors do not span the full data subspace, the dynamics can exhibit a transient implicit bias, with persistent, and in some cases amplified, orthogonal components.

Relationship to [29]. Our work is closest in spirit to MacDonald et al. [29], who bridge the gap between mechanistic and quantitative approaches for overparameterized least squares. Using a Taylor expansion of the gradient around a manifold of minimizers, they establish a parallel-orthogonal decomposition that makes the self-stabilization mechanism transparent while simultaneously yielding convergence rates. We extend this framework in two key directions. First, we replace the minimizer manifold with a curvature-defined reference subspace, allowing the analysis to apply in settings where no finite minimizer exists. This shift moves the theory beyond the overparameterized least-squares regime to exponential-tail losses such as logistic and cross-entropy. Second, our assumptions are formulated at the level of structural properties of the loss and its geometry, rather than tied to a specific objective, thereby enabling a unified treatment of a broader class of problems.

Setup and Motivation

We study gradient descent θt+1=θt−η∇L(θt)\theta_{t+1} = \theta_t - \eta\nabla\mathcal{L}(\theta_t) on a smooth loss L:Rd→R\mathcal{L} : \mathbb{R}^{d} \to\mathbb{R} with step size η>0\eta> 0. Two classical facts motivate the analysis. First, if ∇L\nabla\mathcal{L} is LL-Lipschitz on a convex set containing θt\theta_t and θt+1\theta_{t+1}, the descent lemma gives

L(θt+1)≤L(θt)−η(1−ηL2)∥∇L(θt)∥2,\mathcal{L}(\theta_{t+1}) \le\mathcal{L}(\theta_t) - \eta\left(1 - \frac{\eta L}{2}\right) \lVert\nabla\mathcal{L}(\theta_t) \rVert^2,

so the loss decreases for 0<η<2/L0 < \eta< 2/L whenever the gradient is nonzero [4]. Second, the Hessian governs the linearized GD map, whose Jacobian is I−η∇2L(θ)I - \eta\nabla^2 \mathcal{L}(\theta): a positive-curvature eigenmode with eigenvalue λ\lambda has multiplier 1−ηλ1 - \eta\lambda and is expansive when ηλ>2\eta\lambda> 2. The largest eigenvalue λmax⁡(∇2L(θ))\lambda_{\max}(\nabla^2 \mathcal{L}(\theta)), the sharpness at θ\theta, therefore sets the local stability threshold 2/η2/\eta. Under large step sizes, such as in the EoS regime, the sharpness exceeds 2/η2/\eta; the top modes are locally expansive and the loss oscillates non-monotonically [11]. Our analysis establishes convergence in exactly this setting, where classical theory provides no convergence guarantee [11]. Yet, the losses we study share a geometric structure that makes this possible: the parameter space decomposes into a low-curvature parallel direction along which the loss decreases, and high-curvature orthogonal directions along which the iterates oscillate. The interplay between these components produces a coupled dynamical system, whose structure we develop in this section and use to prove convergence guarantees in Section 4.

Figure 1. GD dynamics illustrating the flat subspace M\mathcal{M}, orthogonal oscillations, evolution of the loss, and parameter magnitudes.

Warmup Example

Consider the 2D loss function: L(x,y)=e−xy2\mathcal{L}(x,y)=e^{-x}y^2. The gradient and Hessian are given by:

∇L(x,y)=e−x(−y22y),∇2L(x,y)=e−x(y2−2y−2y2).(1)\nabla\mathcal{L}(x,y)=e^{-x}\begin{pmatrix}-y^2\\2y\end{pmatrix},\qquad\nabla^2\mathcal{L}(x,y)=e^{-x}\begin{pmatrix}y^2&-2y\\-2y&2\end{pmatrix}. \tag*{(1)}

When y=0y=0, the curvature vanishes in the xx direction and the loss is identically zero along xx. This suggests a natural decomposition of the parameter space into a low-curvature subspace M\mathcal{M}, the xx-axis, and its orthogonal complement, the yy-axis. Using this decomposition, we analyze GD with learning rate η\eta and initialization y0≠0y_0\ne0:

xt+1=xt+ηe−xtyt2,yt+1=yt(1−2ηe−xt).(2)x_{t+1}=x_t+\eta e^{-x_t}y_t^2,\qquad y_{t+1}=y_t(1-2\eta e^{-x_t}). \tag*{(2)}

These coupled dynamics reveal the self-stabilization phenomenon. Define the stability threshold x∗:=log⁡(η)x_*:=\log(\eta), so that 2ηe−x∗=22\eta e^{-x_*}=2 and let x0<x∗x_0<x_*. The three regimes are then:

  • Unstable phase (2ηe−xt>22\eta e^{-x_t}>2, i.e. xt<x∗x_t<x_*). Since ∣1−2ηe−xt∣>1\lvert1-2\eta e^{-x_t}\rvert>1, ∣yt∣\lvert y_t\rvert grows while alternating sign at each step. However, this growth is self-limiting: the first equation can be interpreted as gradient descent on e−xte^{-x_t} with ηyt2\eta y_t^2 acting as an effective step size. A large ∣yt∣\lvert y_t\rvert therefore accelerates forward movement in the xx direction, which simultaneously decreases e−xte^{-x_t} and weakens the instability.

  • Transition to stability (xtx_t crosses x∗x_*). Once xt≥x∗x_t\ge x_*, we have 2ηe−xt≤22\eta e^{-x_t}\le2 and hence ∣1−2ηe−xt∣≤1\lvert1-2\eta e^{-x_t}\rvert\le1: the yy-update becomes non-expanding. For 1<2ηe−xt<21<2\eta e^{-x_t}<2, the iterates still oscillate in sign but now contract. This stage can be described as an oscillatory stable regime.

  • Non-oscillatory contraction (2ηe−xt<12\eta e^{-x_t}<1, i.e. xt>x∗+log⁡2x_t>x_*+\log2). The multiplier 1−2ηe−xt∈(0,1)1-2\eta e^{-x_t}\in(0,1), so yty_t contracts without sign changes.

This illustrates the essential components of the unstable regime: oscillation in sharp directions, slow drift in flat directions, and an emergent coupling that stabilizes the dynamics. The subspace M\mathcal{M} (xx-axis), plays a central role in separating these behaviors. Two structural features of this example carry over to the general analysis. First, the orthogonal update yt+1=(1−2ηe−xt)yty_{t+1}=(1-2\eta e^{-x_t})y_t is an exact multiplicative map: the contraction/expansion factor depends on xtx_t but not on yty_t. Second, the sharpness on the parallel direction decays exponentially along the parallel direction, converting the qualitative observation sharpness drops below 2/η2/\eta into quantitative rates. One feature does not carry over: the loss restricted to the parallel direction L(x,0)\mathcal{L}(x,0) vanishes identically, so forward progress comes entirely from the curvature-reduction coupling. In the classification losses we analyze, the parallel gradient is nonvanishing and provides additional forward drift; the curvature-reduction term acts as an accelerant during the unstable phase, not the sole engine.

Subspace Decomposition and Coupled Dynamics

We now extend the warmup’s two-component structure to a more general setting. The guiding principle is that, near a distinguished one-dimensional subspace, the GD dynamics decompose into a drift parallel to the subspace and oscillations orthogonal to it. Expanding the gradient in these two directions reveals that the reduced dynamics are governed by a reduced loss along the parallel direction and a curvature operator in the orthogonal directions.

Subspace decomposition. Let v∈Rdv \in\mathbb{R}^{d} be a unit vector along which the curvature vanishes or decays and set M:=span⁡{v}\mathcal{M} := \operatorname{span}\{v\}. The subspace M\mathcal{M} is determined by the geometry of the objective: in the warmup example this was the xx-axis; in the symmetric classification models studied here, it is the max-margin direction. The losses we study have the property that ∇L(θ)\nabla\mathcal{L}(\theta) lies in a fixed subspace1^{1} N⊆Rd\mathcal{N} \subseteq\mathbb{R}^{d} containing M\mathcal{M} for all θ∈Rd\theta\in\mathbb{R}^{d}. Without loss of generality, we assume the dynamics are initialized in N\mathcal{N} and define U:=N∩M⊥\mathcal{U} := \mathcal{N} \cap\mathcal{M}^{\perp}, so that N=M⊕U\mathcal{N} = \mathcal{M} \oplus\mathcal{U} and every iterate can be written as

θ=θ∥v+θ⊥,θ∥:=⟨θ,v⟩∈R,θ⊥:=ΠUθ∈U.(3)\theta= \theta^{\parallel}v + \theta^{\perp}, \qquad\theta^{\parallel} := \langle\theta, v\rangle\in\mathbb{R}, \qquad\theta^{\perp} := \Pi_{\mathcal{U}}\theta\in\mathcal{U}. \tag*{(3)}

Similarly, the gradient decomposes into two components: ∇L∥:=ΠM∇L\nabla\mathcal{L}^{\parallel} := \Pi_{\mathcal{M}}\nabla\mathcal{L} and ∇L⊥:=ΠU∇L\nabla\mathcal{L}^{\perp} := \Pi_{\mathcal{U}}\nabla\mathcal{L} denoting the orthogonal projections onto M\mathcal{M} and U\mathcal{U} respectively.

Taylor expansion and coupled dynamics. Recall the GD update θt+1=θt−η∇L(θt)\theta_{t+1} = \theta_t - \eta\nabla\mathcal{L}(\theta_t) and consider a second-order Taylor expansion of the gradient ∇L(θt)\nabla\mathcal{L}(\theta_t) around the parallel direction point θt∥v\theta_t^{\parallel}v.2^{2} Projecting onto M\mathcal{M} and U\mathcal{U} under regularity conditions formalized in the next section, we obtain a coupled two-equation system for the GD iterates:

References

  1. [1]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. In Advances in Neural Information Processing Systems (NeurIPS), pages 1–9, 2016.arxiv.org/abs/1607.06450
  2. [2]Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 448–456, 2015.
  3. [3]Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv preprint, 2019.
  4. [4]Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
  5. [5]Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
  6. [6]Leslie N. Smith and Nicholay Topin. Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates. arXiv preprint, 2018.arxiv.org/abs/1708.07120
  7. [7]Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.arxiv.org/abs/1706.02677
  8. [8]Ildus Sadrtdinov, Maxim Kodryan, Eduard Pokonechny, Ekaterina Lobacheva, and Dmitry P Vetrov. Where do large learning rates lead us? Advances in Neural Information Processing Systems, 37:58445–58479, 2024.arxiv.org/abs/2410.22113
  9. [9]Guillaume Leclerc and Aleksander Madry. The two regimes of deep network training. arXiv preprint arXiv:2002.10376, 2020.arxiv.org/abs/2002.10376
  10. [10]Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218, 2020.arxiv.org/abs/2003.02218
  11. [11]Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021.arxiv.org/abs/2103.00065
  12. [12]Lei Wu, Chao Ma, et al. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31, 2018.
  13. [13]Lachlan Ewen MacDonald, Jack Valmadre, and Simon Lucey. On progressive sharpening, flat minima and generalisation, September 2023. arXiv:2305.14683 [cs, stat].arxiv.org/abs/2305.14683
  14. [14]Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability. Advances in Neural Information Processing Systems, 35:9983–9994, 2022.arxiv.org/abs/2207.12678
  15. [15]Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamics with a minimalist example. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.03294
  16. [16]Liming Liu, Zixuan Zhang, Simon Du, and Tuo Zhao. A minimalist example of edge-of-stability and progressive sharpening. arXiv preprint arXiv:2503.02809, 2025.arxiv.org/abs/2503.02809
  17. [17]Yuqing Wang, Zhenghao Xu, Tuo Zhao, and Molei Tao. Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023.arxiv.org/abs/2310.17087
  18. [18]Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In International Conference on Machine Learning, pages 17684–17744. PMLR, 2023.arxiv.org/abs/2305.13064
  19. [19]Xuxing Chen, Krishnakumar Balasubramanian, Promit Ghosal, and Bhavya Agrawalla. From stability to chaos: Analyzing gradient descent dynamics in quadratic regression. Transactions on machine learning research, 2024.arxiv.org/abs/2310.01687
  20. [20]Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (s) gd over diagonal linear networks: Implicit bias, large stepsizes, and edge of stability. Advances in Neural Information Processing Systems, 36:29406–29448, 2023.DOI
  21. [21]Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pages 948–1024. PMLR, 2022.arxiv.org/abs/2205.09745
  22. [22]Minhak Song and Chulhee Yun. Trajectory alignment: understanding the edge of stability phenomenon via bifurcation theory. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 71632–71682, 2023.arxiv.org/abs/2307.04204
  23. [23]Lei Chen and Joan Bruna. Beyond the edge of stability via two-step gradient updates. In International Conference on Machine Learning, pages 4330–4391. PMLR, 2023.arxiv.org/abs/2206.04172
  24. [24]Sungyoon Lee and Cheongjae Jang. A new characterization of the edge of stability based on a sharpness measure aware of batch gradient distribution. In The Eleventh International Conference on Learning Representations, 2023.
  25. [25]Avrajit Ghosh, Soo Min Kwon, Rongrong Wang, Saiprasad Ravishankar, and Qing Ou. Learning Dynamics of Deep Linear Networks Beyond the Edge of Stability, February 2025. arXiv:2502.20531 [stat].arxiv.org/abs/2502.20531
  26. [26]Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2209.15594
  27. [27]Jingfeng Wu, Peter L Bartlett, Matus Telgarsky, and Bin Yu. Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency. In The Thirty Seventh Annual Conference on Learning Theory, pages 5019–5073. PMLR, 2024.arxiv.org/abs/2402.15926
  28. [28]Jingfeng Wu, Peter Bartlett, Matus Telgarsky, and Bin Yu. Benefits of early stopping in gradient descent for overparameterized logistic regression. arXiv preprint arXiv:2502.13283, 2025.arxiv.org/abs/2502.13283
  29. [29]Lachlan Ewen MacDonald, Hancheng Min, Leandro Palma, Salma Tarmoun, Ziqing Xu, and Rene Vidal. Convergence rates for gradient descent on the edge of stability for overparametrised least squares. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2510.17506
  30. [30]Stanislaw Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. In International Conference on Learning Representations, 2018.arxiv.org/abs/1807.05031
  31. [31]Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020.arxiv.org/abs/2002.09572
  32. [32]Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018.arxiv.org/abs/1802.08770
  33. [33]Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1907.04595
  34. [34]Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pages 16270–16295. PMLR, 2022.
  35. [35]Yuhang Cai, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter Bartlett. Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization. Advances in Neural Information Processing Systems, 37:71306–71351, 2024.arxiv.org/abs/2406.08654
  36. [36]Miao Lu, Beining Wu, Xiaodong Yang, and Difan Zou. Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates, October 2023. arXiv:2310.17074 [cs].arxiv.org/abs/2310.17074
  37. [37]Alexandru Crăciun and Debarghya Ghoshdastidar. On the convergence of gradient descent for large learning rates. arXiv preprint arXiv:2402.13108, 2024.arxiv.org/abs/2402.13108
  38. [38]Jingfeng Wu, Vladimir Braverman, and Jason D Lee. Implicit bias of gradient descent for logistic regression at the edge of stability. Advances in Neural Information Processing Systems, 36:74229–74256, 2023.arxiv.org/abs/2305.11788
  39. [39]Michael Crawshaw and Mingrui Liu. Tight bounds for logistic regression with large stepsize gradient descent in low dimension. arXiv preprint arXiv:2602.12471, 2026.arxiv.org/abs/2602.12471
  40. [40]Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of Machine Learning Research, 2:265–292, 2001.
  41. [41]Hrithik Ravi, Clayton Scott, Daniel Soudry, and Yutong Wang. The implicit bias of gradient descent on separable multiclass data. Advances in neural information processing systems, 37:81324–81359, 2024.arxiv.org/abs/2411.01350

Paper details

vlnc:W_wwa595hk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents