betaFeedback

WorksComputer Science

Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule

Abstract

We study the implicit bias of Adam and its similarity to sign gradient descent on homogeneous models in binary classification tasks. Perhaps surprisingly, we uncover that these depend on the learning rate schedule. While previous works studied strongly decaying learning rate schedules η(t)=o(t1/L−1)η(t)=\mathrm o \left(t^{1/L-1}\right), where LL is the homogeneity degree, and showed ℓ∞\ell_\infty-margin-maximization biases, we focus on the more common complementary regime, including constant and weakly decaying schedules. For a polynomial schedule tαt^α and homogeneity degree LL, we show that when α>1L−12α> \frac 1L - \frac12, Adam has no trajectory-independent margin-maximization bias for deep linear networks (in the KKT sense): first, there is no non-trivial, locally Lipschitz, convex implicit regularizer in this setting, including any squared norm. Second, on width-11 networks, the result holds without requiring convexity, implying that any implicit regularizer for wide networks must be trivial when restricted to a width-11 network. Nevertheless, we positively characterize the implicit bias in this regime as maximizing a trajectory-dependent weighted-ℓ2\ell_2 margin for any LL-homogeneous model. The fundamental mechanism behind our results is convergence of Adam's preconditioner to a trajectory-dependent vector, which also allows showing that the limiting parameter direction depends on initialization. In the intermediate regime 1L−1≤α≤1L−12\frac 1L - 1 \leq α\leq \frac1L - \frac12 we prove an ℓ∞\ell_\infty bias, conditional on gradient directional convergence to a full-support vector. Of independent interest, we establish a loss decay rate for full batch Adam on general homogeneous models, whose leading order depends only on the second-moment parameter.

Introduction

The implicit bias or implicit regularization of optimizers refers to their tendency to favor solutions with certain properties, often associated with generalization, even when no explicit regularization is used in training. On classification tasks and homogeneous models (e.g. deep ReLU networks), optimizers such as gradient descent and the steepest descent family have been shown to possess margin-maximization implicit biases, selecting among all perfect-accuracy solutions those which maximize a certain confidence margin ([35, 45]).

Adam ([27]) and its derivatives are popular optimizers for training large language models (LLMs) and other deep neural networks. An improved understanding of their implicit biases is therefore essential to strengthening the theoretical foundations of modern AI. Earlier works ([53, 15, 19]) on linear and smooth homogeneous models trained with exponentially-tailed losses showed that Adam has a margin-maximization bias towards the ℓ∞\ell_\infty margin when the learning rate schedule η(t)\eta(t) undergoes a strong decay η(t)=o(t1/L−1)\eta(t) = o\left(t^{1/L-1}\right). Here, LL is the homogeneity degree, equal to the network depth for linear or ReLU networks. Moreover, these works show similarity between late-phase Adam and sign gradient descent (sign-GD), which also has a bias towards ℓ∞\ell_\infty-margin maximization ([45]). More broadly, the similarity between Adam and sign-GD (including variations with momentum) is assumed, derived or empirically tested in many prior works on optimization geometry, generalization, and optimizer design ([3, 54, 34, 31, 30, 33, 6]).

The schedule η(t)\eta(t) plays a key role, since it changes the effective memory horizon along the trajectory when the momentum hyperparameters β1,β2\beta_1,\beta_2 remain fixed ([5]), as is the case in practice. Motivated by the common constant and weakly decaying schedules, we extend the characterization of Zhang et al. [53]; Gronich and Vardi [19] to η(t)≍tα\eta(t) \asymp t^\alpha with α≥1L−1\alpha\ge\frac{1}{L} - 1, obtaining the following surprising results:

  1. When α>1L−12\alpha> \frac{1}{L} - \frac{1}{2}, including a constant learning rate for L>2L > 2, we prove nonexistence results for implicit regularizers of Adam. We formalize this by studying regularizers inspired by squared norms: locally Lipschitz functions R(θ)R(\theta), where θ\theta are the network’s parameters, satisfying for all separable datasets {(xi,yi)}\{(x_i,y_i)\} and Adam trajectories that directional limit points of θ\theta are proportional to KKT points of

min⁡θR(θ)s.t.∀i:yif(xi;θ)≥1.\min_{\theta} R(\theta) \quad\text{s.t.} \quad\forall i : y_i f(x_i;\theta) \ge1 .

(a) We first show that no non-constant R(θ)R(\theta) satisfies this definition for width-1 linear networks. Thus, Adam does not maximize any margin in width-1 linear networks.

(b) We then lift the construction to a deep linear network with general widths, proving triviality of any convex regularizer.

Both results rule out squared norms as implicit regularizers of Adam, implying there is no margin-maximization bias in the usual sense. In particular, Adam’s late-stage behavior differs significantly from that of sign-GD.

  1. Despite the above results implying that no single R(θ)R(\theta) regularizes all trajectories, we show on the flip side that Adam has a trajectory-dependent margin-maximization implicit bias. Specifically, directional limit points of Adam are KKT points of a weighted-ℓ2\ell_2 margin-maximization objective defined by a trajectory-dependent preconditioning diagonal matrix. This positive result, as well the negative ones, rely on a key mechanism of convergence of Adam’s second-moment preconditioner to a trajectory-dependent limit.

  1. In the intermediate range 1L−1≤α≤1L−12\frac{1}{L} - 1 \le\alpha\le\frac{1}{L} - \frac{1}{2}, we extend the result of Gronich and Vardi [19], and prove that Adam on homogeneous models maximizes the ℓ∞\ell_\infty margin. Our result is conditional on gradient directional convergence to a full-support vector.

  1. Of independent interest, we establish a loss decay rate for (full batch) Adam on general homogeneous models, whose leading order depends only on the second-moment parameter:

−log⁡L=c22t±o(t),(α>1L−1),(1)-\log\mathcal{L} = \frac{c_2}{2}t \pm o(t), \qquad\left(\alpha> \frac{1}{L} - 1\right), \tag*{(1)}
−log⁡L=c22t±O(log⁡t),(α>1L−12),(2)-\log\mathcal{L} = \frac{c_2}{2}t \pm O(\log t), \qquad\left(\alpha> \frac{1}{L} - \frac{1}{2}\right), \tag*{(2)}

where c2c_2 is the continuous analogue of −log⁡β2-\log\beta_2 (see § 3 for definitions). This both shows a direct connection between the value of Adam’s β2\beta_2 parameter and the loss decay rate, and demonstrates that this asymptotic rate is essentially independent of the learning rate schedule. In contrast, earlier results for the regime α<1L−1\alpha< \frac{1}{L} - 1 give the schedule-dependent rate −log⁡L≍tL(α+1)=o(t)-\log\mathcal{L} \asymp t^{L(\alpha+1)} = o(t) ([53, 19]); our analysis also derives this rate, under weaker assumptions.

Related work

An ℓ2\ell_2-margin-maximization implicit bias of gradient descent was first proved on linear predictors by Soudry et al. [43], and subsequent work investigated how architecture and initialization shape the implicit bias of certain deep networks ([24, 21, 52] (Moroshko et al. 2020)). General homogeneous models were studied by Lyu and Li [35], who showed that any directional limit point of the trajectory is the direction of a KKT point of the ℓ2\ell_2-max-margin problem, while Ji and Telgarsky [25] proved for homogeneous models definable in an o-minimal structure that the parameter direction converges under gradient descent. See Vardi [46] for a survey of the implicit bias of gradient descent.

A common family of non-Euclidean optimizers is steepest descent, whose special cases include sign gradient descent and spectral descent (with momentum: Signum [7] and idealized Muon [26], respectively). Gunasekar et al. [20] proved a margin-maximization bias for steepest descent on linear models, where the margin geometry is not ℓ2\ell_2 but the norm defining the optimizer. This was generalized by Tsilivis et al. [45] to homogeneous models, showing that any limit point of θt∥θt∥\frac{\theta_t}{\lVert\theta_t\rVert} is the direction of a KKT point of the appropriate max-margin problem. Specifically, this result implies that sign-GD has a bias towards ℓ∞\ell_\infty-margin maximization.

On Adam, Wang et al. [48] and Wang et al. [49] first established a bias toward ℓ2\ell_2-margin maximization. Their analysis considers a regime in which Adam’s stability constant ε\varepsilon asymptotically dominates the denominator, leading to gradient-descent-like behavior. Analyses omitting the stability constant include the works of Zhang et al. [53] and Fan et al. [15] establishing ℓ∞\ell_\infty-margin maximization on linear models (in the binary and multiclass settings, respectively), with a decaying learning rate, while Baek et al. [1] showed that deterministic batching can induce an implicit bias different from that of full-batch Adam. Recently, Gronich and Vardi [19] established an ℓ∞\ell_\infty bias for Adam on smooth, homogeneous networks under a learning-rate schedule satisfying η(t)=o(t1L−1)\eta(t)=o\left(t^{\frac{1}{L}-1}\right), where LL denotes the degree of homogeneity.

For AdamW, which incorporates explicit weight decay, Xie and Li [51] studied smooth models and losses and showed that trajectory limit points satisfy the KKT conditions for loss minimization subject to a bound on the ℓ∞\ell_\infty norm of the parameters.

Impossibility results for implicit bias have been proved before by Vardi and Shamir [47] for squared-loss regression with gradient flow on ReLU networks, and by Dauber et al. [13]; Suggala et al. [44] for other learning problems. Qian and Qian [42] and Wang et al. [48] observed a converging, trajectory-dependent preconditioner for AdaGrad in binary classification (on linear and homogeneous models, respectively), and Bai et al. [2] noticed the mechanism of second-moment convergence (which they term “decoupling”) in Adam on highly degenerate polynomials.

Problem setting

Throughout this work, we consider a binary classification task defined by a dataset {(xi,yi)}i=1m⊆Rd×{−1,1}\{(\mathbf{x}_i,y_i)\}_{i=1}^{m}\subseteq\mathbb{R}^d\times\{-1,1\}. We study models f(x;θ)f(\mathbf{x};\theta), parameterized by θ∈Rp\theta\in\mathbb{R}^p, which satisfy the following assumptions:

(M1) ff is locally Lipschitz in θ\theta and admits a chain rule (see Appendix § 10).

(M2) ff is LL-homogeneous for L≥1L\geq1: ∀z>0,x∈Rd,θ∈Rp: f(x;zθ)=zLf(x;θ)\forall z>0,\mathbf{x}\in\mathbb{R}^d,\theta\in\mathbb{R}^p: \ f(\mathbf{x};z\theta)=z^L f(\mathbf{x};\theta).

For example, deep linear networks, ReLU networks and convolutional networks (all without bias terms) of depth LL satisfy these conditions. The classification margin of a homogeneous model, with respect to a norm ∥⋅∥\lVert\cdot\rVert, is defined as

γ(θ)=min⁡i∈[m]yif(xi;θ∥θ∥).(3)\gamma(\theta)=\min_{i\in[m]}y_i f\left(\mathbf{x}_i;\frac{\theta}{\lVert\theta\rVert}\right). \tag*{(3)}

We write qi(θ)=yif(xi;θ)q_i(\theta)=y_i f(\mathbf{x}_i;\theta) and qmin⁡(θ)=min⁡iqi(θ)q_{\min}(\theta)=\min_i q_i(\theta) as shorthand.

We analyze Adam, where we follow Zhang et al. [53]; Fan et al. [15]; Gronich and Vardi [19]; Baek et al. [1], among others, by setting the stability constant ε\varepsilon to 00. This choice is motivated by the fact that in asymptotic analysis, a positive ε\varepsilon dominates the decaying denominator v^t\sqrt{\hat{v}_t}, while in practice ε\varepsilon typically remains negligible throughout optimization. We also follow Barakat and Bianchi [4]; Ma et al. [36]; Belotto da Silva and Gazeau [5]; Wang et al. [48]; Gronich and Vardi [19] in working with a continuous-time version of Adam, defined as follows. Consider an empirical loss L(θ)\mathcal{L}(\theta) and a choice of subgradients gt∈∂L(θt)g_t\in\partial\mathcal{L}(\theta_t), then:

dθtdt=−η(t)m^tv^t,(4)\frac{d\theta_t}{dt}=-\eta(t)\frac{\hat{m}_t}{\sqrt{\hat{v}_t}}, \tag*{(4)}
mt=EMA⁡c1[gt],m^t=EMA⁡c1[gt],vt=EMA⁡c2[gt2],v^t=EMA⁡c2[gt2].(5)m_t=\operatorname{EMA}_{c_1}[g_t],\quad\hat{m}_t=\operatorname{EMA}_{c_1}[g_t],\quad v_t=\operatorname{EMA}_{c_2}[g_t^2],\quad\hat{v}_t=\operatorname{EMA}_{c_2}[g_t^2]. \tag*{(5)}

Here we denote by EMA⁡c[g]\operatorname{EMA}_{c}[g], EMA⁡^c[g]\widehat{\operatorname{EMA}}_{c}[g] (respectively) the plain and bias-corrected exponential moving average (momentum) term for gg with a continuous smoothing parameter c>0c>0 analogous to the discrete parameter β∈[0,1)\beta\in[0,1) (or more precisely, analogous to −log⁡β-\log\beta; see Appendix § 11 for definitions and references). Note that division, square root and the EMA operations are applied coordinate-wise. Throughout this work, we assume c1≥c2c_1 \ge c_2, analogous to the common setting β1≤β2\beta_1 \le\beta_2. For any j∈[p]j \in[p] with v^t[j]=0\hat{v}_t[j] = 0 we set dθtdt[j]=0\frac{d\theta_t}{dt}[j] = 0; this is consistent with the practical ε>0\varepsilon> 0 setting of Adam, since v^t[j]=0\hat{v}_t[j] = 0 implies m^t[j]=0\hat{m}_t[j] = 0. We consider a polynomial learning rate schedule of the form

η(t)=η0(1+t)α,α>−1,η0>0.(6)\eta(t) = \eta_0(1+t)^\alpha,\qquad\alpha> -1,\qquad\eta_0 > 0 . \tag*{(6)}

Such learning rate schedules are ubiquitous in practice as well as theoretical analyses.

The losses we consider, following Gronich and Vardi [19], are log-concave, exponentially-tailed losses. Namely, we observe losses L(θ):Rp→R\mathcal{L}(\theta) : \mathbb{R}^p \to\mathbb{R} of the form

L(θ)=∑i=1mℓ(yif(xi;θ)),ℓ∈Le,(7)\mathcal{L}(\theta) = \sum_{i=1}^{m} \ell(y_i f(x_i;\theta)),\qquad\ell\in\mathcal{L}_{\mathrm{e}}, \tag*{(7)}

where Le\mathcal{L}_{\mathrm{e}} is the class of functions of the form ℓ(u)=e−φ(u)\ell(u) = e^{-\varphi(u)} for φ∈C2(R)\varphi\in C^2(\mathbb{R}) which is convex, strictly increasing and has bounded first and second derivatives (see Appendix § 10). Notably, the exponential and logistic losses (ℓ(u)=e−u\ell(u) = e^{-u} and ℓ(u)=log⁡(1+e−u)\ell(u) = \log(1+e^{-u}), respectively) are in Le\mathcal{L}_{\mathrm{e}}.

We introduce two trajectory assumptions:

References

  1. [1]Beomhan Baek, Minhak Song, and Chulhee Yun. Implicit bias of per-sample Adam on separable data: Departure from the full-batch regime. In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/c3596a5990c2cddb0b0de478df7aca65-Abstract-Conference.html.arxiv.org/abs/2510.26303
  2. [2]Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou, Zhi-Qin John Xu, and Yaoyu Zhang. Towards understanding adam convergence on highly degenerate polynomials, 2026. URL https://arxiv.org/abs/2603.09581.
  3. [3]Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent. arXiv preprint arXiv:2002.08056, 2020. URL https://arxiv.org/abs/2002.08056.
  4. [4]Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM Journal on Optimization, 31(1):244–274, 2021.arxiv.org/abs/1810.02263
  5. [5]André Belotto da Silva and Maxime Gazeau. A general system of differential equations to model first-order adaptive algorithms. Journal of Machine Learning Research, 21(129):1–42, 2020. URL https://jmlr.org/papers/v21/18-808.html.
  6. [6]Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024.arxiv.org/abs/2409.20325
  7. [7]Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International conference on machine learning, pages 560–569. PMLR, 2018.arxiv.org/abs/1802.04434
  8. [8]Jérôme Bolte, Aris Daniilidis, Adrian Lewis, and Masahiro Shiota. Clarke subgradients of stratifiable functions. SIAM Journal on Optimization, 18(2):556–572, 2007.DOI
  9. [9]Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv NikanKin, and Michal Irani. Deconstructing data reconstruction: Multiclass, weight decay and general losses. Advances in Neural Information Processing Systems, 36:51515–51535, 2023.arxiv.org/abs/2307.01827
  10. [10]Yuhang Cai, Kangjie Zhou, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter L. Bartlett. Implicit bias of gradient descent for non-homogeneous deep networks, 2025. URL https://arxiv.org/abs/2502.16075.
  11. [11]Matias D. Cattaneo, Jason Matthew Klusowski, and Boris Shigida. On the implicit bias of Adam. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5862–5906. PMLR, 2024. URL https://proceedings.mlr.press/v235/cattaneo24a.html.arxiv.org/abs/2309.00079
  12. [12]Frank H. Clarke. Generalized gradients and applications. Transactions of the American Mathematical Society, 205:247–262, 1975.DOI
  13. [13]Assaf Dauber, Meir Feder, Tomer Koren, and Roi Livni. Can implicit bias explain generalization? stochastic convex optimization as a case study. Advances in Neural Information Processing Systems, 33:7743–7753, 2020.arxiv.org/abs/2003.06152
  14. [14]Damek Davis, Dmitriy Drusvyatskiy, Sham M. Kakade, and Jason D. Lee. Stochastic Subgradient Method Converges on Tame Functions. Found. Comput. Math., 20(1):119–154, 2020.arxiv.org/abs/1804.07795
  15. [15]Chen Fan, Mark Schmidt, and Christos Thrampoulidis. Implicit bias of spectral descent and muon on multiclass separable data. arXiv preprint arXiv:2502.04664, 2025.arxiv.org/abs/2502.04664
  16. [18]Yuval Golbari, Navye Wasserman, Gal Vardi, and Michal Irani. Impmia: Leveraging implicit bias for membership inference attack. arXiv preprint arXiv:2510.10625, 2025.arxiv.org/abs/2510.10625
  17. [19]Eitan Gronich and Gal Vardi. The implicit bias of adam and muon on smooth homogeneous neural networks, 2026. URL https://arxiv.org/abs/2602.16340.
  18. [20]Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pages 1832–1841. PMLR, 2018a.arxiv.org/abs/1802.08246
  19. [21]Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018b.arxiv.org/abs/1806.00468
  20. [22]Sagi Habani, Yehonathan Refael, and Ofir Lindenbaum. Which optimizer leaks most? optimizer implicit bias as a training-data privacy controller, 2026. URL https://arxiv.org/abs/2610.18294.
  21. [23]Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. Advances in Neural Information Processing Systems, 35:22911–22924, 2022.arxiv.org/abs/2206.07758
  22. [24]Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv:1810.02032, 2018.arxiv.org/abs/1810.02032
  23. [25]Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176–17186, 2020.
  24. [26]Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon, 6(3):4, 2024.
  25. [27]Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.arxiv.org/abs/1412.6980
  26. [29]Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. arXiv preprint arXiv:2210.03820, 2022.
  27. [30]Frederik Kunstner and Francis Bach. Scaling laws for gradient descent and sign descent for linear bigram models under zipf’s law, 2025. URL https://arxiv.org/abs/2505.19227.
  28. [31]Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/350e718ff74062b4bac2c6ffd9e1ac66-Abstract-Conference.html.
  29. [32]Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 2002.
  30. [33]Bingrui Li, Wei Huang, Andi Han, Zhanpeng Zhou, Taiji Suzuki, Jun Zhu, and Jianfei Chen. On the optimization and generalization of two-layer transformers with sign gradient descent. In International Conference on Learning Representations, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/a1e0d6fa0c30b7d4f75dd9c7ed6189f2-Abstract-Conference.html.
  31. [34]Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/06960915ba8674c7a898ec0b472b80ff-Abstract-Conference.html.
  32. [35]Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890, 2019.
  33. [36]Chao Ma, Lei Wu, and Weinan E. A qualitative study of the dynamic behavior for adaptive gradient algorithms. In Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pages 671–692. PMLR, 2022. URL https://proceedings.mlr.press/v145/ma22a.html. Edward Moroshko, Blake E Woodworth, Suriya Gunasekar, Jason D Lee, Nati Srebro, and Daniel Soudry. Implicit bias in deep linear classification: Initialization scale vs training accuracy. Advances in neural information processing systems, 33:22182–22193, 2020.
  34. [37]Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pages 4683–4692. PMLR, 2019a.arxiv.org/abs/1905.07325
  35. [38]Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savaresse, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 3420–3428. PMLR, 16–18 Apr 2019b. URL https://proceedings.mlr.press/v89/nacson19b.html.arxiv.org/abs/1803.01905
  36. [39]Antonio Orvieto and Robert M. Gower. In search of Adam’s secret sauce. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://papers.neurips.cc/paper_files/paper/2025/hash/5bd9aa206d782e4e1f7ab5d177a10828-Abstract-Conference.html.arxiv.org/abs/2505.21829
  37. [40]Yakir Oz, Gilad Yehudai, Gal Vardi, Itai Antebi, Michal Irani, and Niv Haim. Reconstructing training data from models trained with transfer learning. In 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 1107–1135. IEEE, 2026.DOI
  38. [41]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
  39. [42]Qian Qian and Xiaoyuan Qian. The implicit bias of AdaGrad on separable data. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/3335881e06d4d23091389226225e17c7-Abstract.html.arxiv.org/abs/1906.03559
  40. [43]Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1–57, 2018.arxiv.org/abs/1710.10345
  41. [44]Arun Suggala, Adarsh Prasad, and Pradeep K Ravikumar. Connecting optimization and regularization paths. Advances in Neural Information Processing Systems, 31, 2018.
  42. [45]Nikolaos Tsilivis, Eitan Gronich, Julia Kempe, and Gal Vardi. Flavors of margin: Implicit bias of steepest descent in homogeneous neural networks, 2025. URL https://arxiv.org/abs/2410.22069.
  43. [46]Gal Vardi. On the implicit bias in deep-learning algorithms. Communications of the ACM, 66(6):86–93, 2023.arxiv.org/abs/2208.12591
  44. [47]Gal Vardi and Ohad Shamir. Implicit regularization in ReLU networks with the square loss. In Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 4224–4258. PMLR, 2021. URL https://proceedings.mlr.press/v134/vardi21b.html.arxiv.org/abs/2012.05156
  45. [48]Bohan Wang, Qi Meng, Wei Chen, and Tie-Yan Liu. The implicit bias for adaptive optimization algorithms on homogeneous neural networks. In International Conference on Machine Learning, pages 10849–10858. PMLR, 2021.arxiv.org/abs/2012.06244
  46. [49]Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, Zhi-Ming Ma, and Tie-Yan Liu. Does momentum change the implicit regularization on separable data? In Advances in Neural Information Processing Systems, volume 35, pages 26764–26776. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ab3f6bbe121a8f7a0263a9b393000741-Abstract-Conference.html.arxiv.org/abs/2110.03891
  47. [50]Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savaresse, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models, 2020. URL https://arxiv.org/abs/2002.09277.
  48. [51]Shuo Xie and Zhiyuan Li. Implicit bias of adamw: ℓ∞-norm constrained optimization. In International Conference on Machine Learning, pages 54488–54510. PMLR, 2024.arxiv.org/abs/2404.04454
  49. [52]Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. arXiv preprint arXiv:2010.02501, 2020.arxiv.org/abs/2010.02501
  50. [53]Chenyang Zhang, Difan Zou, and Yuan Cao. The implicit bias of adam on separable data. Advances in Neural Information Processing Systems, 37:23988–24021, 2024.arxiv.org/abs/2406.10650
  51. [54]Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C. Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/hash/9d9d4f495e875a2e075a1a4a6e1b9770f-Abstract.html.arxiv.org/abs/2010.07468

Paper details

vlnc:W_qh5kzgh3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI tools

View statement
Looking for related papers…
Contents