WorksComputer Science
Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?
Abstract
We study whether grokking, the delayed emergence of out-of-distribution generalization after apparent overfitting, alters training efficiency in deep neural networks, where efficiency is measured as accuracy per unit energy. Using two canonical settings, modular arithmetic and MNIST, we analyze two neural network models under 16-, 32-, and 64-bit precision and use Orthograd to test whether grokking produces noticeable deviations from the typically decreasing dynamics of training efficiency. We consider both short training runs of up to 500 epochs and long training runs of up to 20,000 epochs, and in both cases identify grokking as a sharp improvement in test accuracy after a sustained train-test gap. In the short-horizon experiments, applying Orthograd at progressively stronger levels during early training does not induce grokking, and efficiency remains smoothly decreasing. In the long-horizon experiments, Orthograd induces grokking, but the efficiency curves remain approximately monotonic and show no appreciable local increase at the grokking transition. These results suggest that, in our setting, grokking does not materially alter overall training-efficiency dynamics.
Introduction
Grokking refers to a training trajectory in which a model first attains very high training accuracy while test accuracy stagnates near chance, and only much later transitions—sometimes abruptly—to strong test accuracy with stable train accuracy. A growing body of work connects grokking to implicit regularization, representation phase transitions, circuit formation, and optimization regimes [1, 2, 3, 4]. In parallel, this paper develops grokking-based views of training efficiency dynamics i.e. an exploration of how the grokking phenomenon could alter the monotonically decreasing dynamics of efficiency, at two scales a short Section 3.2.1 and a long Section 3.2.2, with the experiments presented we intend to prove that training efficiency dynamics remain unphased even when grokking happens. This paper consist of the following: a brief literature review, then we present detailed experiments on modular addition and MNIST, compare standard training runs and long training runs, then analyze whether efficiency trends can change on both due to grokking. These experiment make …
Literature Review
For the research presented in this section this literature review demonstrates the breadth of deep learning research that is engaging with grokking and thereby motivates why grokking is an important phenomenon that should be considered as part of research on training efficiency in deep learning. Recent literature on grokking has contributed to underlying theoretical frameworks, experimental observations across different datasets and architectures, and proposed mechanisms for its occurrence. Some research has highlighted grokking as an interplay between memorization and compression phases during network training. Researchers have proposed different explanations such as changes in internal network complexity, shifts in linear regions of decision boundaries, and entanglement transitions in tensor-network representations.
Another contribution to the literature on grokking comes from Liu et al. which introduces the Linear Mapping Number (LMN), a complexity metric that generalizes the concept of linear regions for neural networks with any activation function. They argue that the LMN effectively captures the network’s compression process preceding generalization, thereby enenabling the observation/measurement of the grokking dynamics while training a DNN, something that traditional complexity measures like the norm cannot achieve. Liu et al. examines algorithmic tasks (modular arithmetic, permutation groups, and XOR operations), where they demonstrate LMN’s effectiveness in depicting the compression dynamics associated with grokking. In particular, the LMN reveals a previously unnoticed phenomenon in the XOR task, a double descent pattern, indicating two nearly degenerate solutions between which the network switches. Liu et al. main contribution is in positioning the LMN as an interpretable information-theoretic complexity measure aligned with the intrinsic computational mechanisms of neural networks, enhancing the theoretical understanding of generalization dynamics in deep learning.
Humayun et al. defined grokking as the delayed emergence of generalization in DNNs well after the model achieves near zero training error. The authors demonstrate that grokking is pervasive across different deep learning setups, such as CNNs trained on CIFAR10, Residual Networks (ResNets) on ImageNet, and even Transformer models on textual data. A novel contribution of Humayun et al. is the concept of “delayed robustness” wherein networks unexpectedly develop resilience to adversarial attacks long after achieving generalization (i.e., delayed robustness occurs after grokking if training is continued). The authors attribute grokking and delayed robustness to dynamic changes in the local complexity of the DNN’s input-output mapping, characterized by shifts in the density and positioning of linear regions within the input space. Notably, a crucial training phase identified as “region migration”, during which network nonlinearities reposition closer to decision boundaries, results in enhanced robustness and reduces local complexity around training points. Overall, the work by Humayun et al. uncovers a broader context for grokking and provides analytical tools to interpret network training dynamics, potentially guiding more effective and robust deep learning practices.
Prieto et al. also explore how deep neural networks suddenly generalize after extensive periods of overfitting. Prieto et al. introduces three primary innovations: StableMax, a numerically stable alternative to Softmax; ⊥Grad, an optimization method that avoids a harmful training direction called Na ive Loss Minimization (NLM); and the orthograd algorithm to induce grokking during a training. For our research the orthograd algorithm is the most relevant aspect of Prieto et al. research because we use it to induce grokking in our experiments. The basis for the orthograd algorithm is the concept of “Softmax Collapse” (SC), where the softmax calculation verges on numerical instability due to values approaching floating-point errors. Through experiments on modular arithmetic, sparse parity, and MNIST tasks, Prieto et al. demonstrate how inducing SC can induce networks to exhibit grokking without explicit regularization. Building on this observation Prieto et al. propose Loss Function Collapse (LFC) as a method to induce SC. LFC works by injecting large values into a DNN’s loss function such that the loss value is close to overflowing. Repeating this process multiple times causes SC and triggers grokking. Prieto et al. propose the orthograd algorithm based on LFC to induce grokking and we use this method to induce grokking when training both the perceptron and LeNet models in this chapter.
An open research question in the literature on orthograd and grokking is how early in a training cycle orthograd can induce grokking. As part of their work in this direction Prieto et al. have developed an experimental methodology for exploring the interaction between variable precision, SC, and grokking. The methodology involves running experiments at multiple levels of precision 16, 32 and 64 bit for the weights used in the neural models. In our experiments we examine how early in a training process orthograd can induce grokking, and to check whether the interactions between precision and grokking explored by Prieto et al. also affect how grokking and efficiency interact we adopted their experimental methodology and based our experimental work on their code base.
In conclusion, the current research on grokking presents diverse theoretical perspectives and empirical insights that collectively deepen our understanding of delayed generalization phenomena. In our experiments in Section 3 we will use the orthograd algorithm from Prieto et al. to try and induce grokking, we will also report on experiments that examine how early in a training process this algorithm can induce grokking.
Experimental Details and Results
The experiments performed in this section use two datasets: MNIST and modular addition 113, which is a common grokking task.
We evaluate two settings common in the grokking literature. Modular addition with prime modulus : inputs are integer pairs and the target is (or ). We use a small MLP and, for comparability with prior work, a perceptron baseline. MNIST digit classification with a LeNet–style CNN. For both, we track training and test accuracy, energy use per epoch, and the efficiency metric of Eq. (1). We examine 16-, 32-, and 64-bit arithmetic to test numerical-precision effects and run short horizons (up to 500 epochs) to probe early dynamics and long horizons (up to 20,000 epochs) to capture late grokking.
Experimental details
Datasets: for modular addition we use 100,000 pairs of values with a 60-40 split. For MNIST we use an 60k/10k split.
Models: an LeNet model a Convolutional Neural Network (CNN), and a Multilayer Perceptron (MLP).
Precisions: We repeat runs in float16/32/64 to assess numerical stability and its interaction with grokking.
Grokking detection: A grokking event is flagged when a train–test accuracy gap persists for a considerable amount of epochs, followed by a sustained test-accuracy rise that narrows the gap.
It should also be noted that the efficiency metric used in the experiments is an adapted version of the efficiency metric used in Cueto-Mendoza and Kelleher ((Equation 2)). Equation (1) is adapted to include a term for the testing accuracy of the model, as follows:
Where is the accuracy obtained on the -th epoch of training, and is the accuracy obtained when testing on the -th epoch, is the sum of the energy samples obtained up to that epoch of training, and
is the concatenation operation.
Similarly, as a consequence of the adjustments to the precision we make with the experimental methodology of Prieto et al. for each model type (LeNet and MLP) in each experiment three different versions of results are reported, the difference across these results being the precision used for the weights of the models, and the storage variables for system energy (W), accuracy and efficiency. The precisions used in the experiments were 16, 32 and 64 bit respectively.
Experimental Results
Early-time experiments (up to 500 epochs).
Within each training run the orthograd algorithm is initially triggered after 50 epochs. In other words, after 50 epochs we start to inject a large value into the loss function and continue doing this at each update while monitoring for sudden increase in generalization on the test set. If after 100 epochs we have not detected a sudden increase in generalization we increase the magnitude of the number injected into the loss function and again continue to monitor for increases in generalization. Finally if after 150 epochs of training we have not observed a significant increase in generalization we increase the magnitude of the number being injected yet again and continue training monitoring up to 500 epochs.
Late-time experiments (up to 20,000 epochs).
Following on from the results presented in the § 3.2.1 experiments in the experiments presented here we train the models for much longer, 20000 epochs in each case. Also, we apply the orthograd algorithm later in training. Specifically for each model and precision combination we randomly select an epoch between 5000 and 20000 and we initiate the orthograd algorithm during the training of the model at the selected epoch and continue applying it from then on until the end of training.
Tabular Data for Section 3.2.1
For this section we provide a table (see Table 1) containing the accuracy values near each application of the Orthograd algorithm, i.e., two epochs before and after 50 (47, 48, 51, 52); similarly for 100 and 150. These values are reported for each of the experiments show in Figure 1a, Figure 1b, Figure 1c, Figure 2a, Figure 2b, and Figure 2c.

Figure 1. The performance and efficiency per epoch (up to 500) with three applications of softmax collapse at 50, 100 and 150 epochs for an MLP, using the modulo 113 dataset, using 16,32 and 64-bit variable precision.

Figure 2. The performance and efficiency per epoch (up to 500) with three applications of softmax collapse at 50, 100 and 150 epochs for an LeNet CNN, using the MNIST dataset, using 16,32 and 64-bit variable precision.
| Epoch | Accuracy −2 | Accuracy −1 | Accuracy +1 | Accuracy +2 | |
| MLP 16-bit | 50 | 0.3198 | 0.3208 | 0.3228 | 0.3235 |
| MLP 16-bit | 100 | 0.3496 | 0.3516 | 0.3516 | 0.3517 |
| MLP 16-bit | 150 | 0.3506 | 0.3503 | 0.3516 | 0.3520 |
| MLP 32-bit | 50 | 0.31047714 | 0.31050083 | 0.31050990 | 0.3105144 |
| MLP 32-bit | 100 | 0.36048922 | 0.36049768 | 0.36049932 | 0.36050236 |
| MLP 32-bit | 150 | 0.40048927 | 0.40049290 | 0.40050557 | 0.40051800 |
| MLP 64-bit | 50 | 0.31098664593165456 | 0.31099036016668750 | 0.31100269995742355 | 0.31101400602066680 |
| MLP 64-bit | 100 | 0.34098711699073030 | 0.34099176252251040 | 0.34100344711960310 | 0.34101115044821220 |
| MLP 64-bit | 150 | 0.34094213766241880 | 0.34094782313983835 | 0.34095876696574884 | 0.34096879802585295 |
| LeNet 16-bit | 50 | 0.3005 | 0.3013 | 0.3013 | 0.3015 |
| LeNet 16-bit | 100 | 0.3691 | 0.3699 | 0.3704 | 0.3718 |
| LeNet 16-bit | 150 | 0.4097 | 0.4110 | 0.4111 | 0.4119 |
| LeNet 32-bit | 50 | 0.36098915 | 0.36099428 | 0.36100575 | 0.36102572 |
| LeNet 32-bit | 100 | 0.41099543 | 0.41099980 | 0.41100586 | 0.41101025 |
| LeNet 32-bit | 150 | 0.42098250 | 0.42100827 | 0.42100880 | 0.42103056 |
| LeNet 64-bit | 50 | 0.40099838981655235 | 0.40100167728465540 | 0.40100441104333820 | 0.41001773879209060 |
| LeNet 64-bit | 100 | 0.47048626152014684 | 0.47049831799038690 | 0.47050874309248315 | 0.47351139685790770 |
| LeNet 64-bit | 150 | 0.50149274260563670 | 0.50251099164355230 | 0.50151271764247890 | 0.50252116270772444 |
Table 1. Numerical results for the early-time grokking experiments. The leftmost column shows the model type MLP that performed the modulo 113 task; LeNet that performed the MNIST task. The columns accuracy value -2 and -1 record the accuracy two and one epoch before applying the Orthograd algorithm, respectively. Similarly, accuracy value +1 and +2 records the accuracy one and two epochs after applying the Orthograd algorithm, respectively.
Tabular Data for Section 3.2.2
In this section we provide a table (see Table 2) containing the mean Orthograd-application epochs and the corresponding mean grokking epochs for each of the experiments shown in Figure 3a, Figure 3b, Figure 3c, Figure 5a, Figure 5b, and Figure 5c.

Figure 3. The performance and efficiency per epoch for a Multi-Layer Perceptron, using a modulo-113 addition dataset, using a 16,32 and 64-bit variable precision.
Figure 5. The performance and efficiency per epoch for a LeNet CNN, using the MNIST dataset, using a 16,32 and 64-bit variable precision.
| Orthograd epoch | Grokking epoch | |
| MLP 16-bit | 12287 | 12329 |
| MLP 32-bit | 9731 | 9775 |
| MLP 64-bit | 7484 | 7523 |
| LeNet 16-bit | 11193 | 11252 |
| LeNet 32-bit | 14964 | 15041 |
| LeNet 64-bit | 17046 | 17105 |
Table 2. Results for the late-time grokking experiments (averaged over 3 runs).
Formal Proof
This section examines the conditions under which training efficiency is monotonically decreasing over training epochs, including steps at which accuracy increases abruptly. We consider the generalized efficiency ratio model
Where indexes epochs, represents the accuracy quantity used in the numerator, and represents cumulative energy consumption. When energy is cumulative, we also impose . Only epochs with positive cumulative energy are included in .
The arguments below concern Equation (2). Applying them to an efficiency expression involving training accuracy, test accuracy, and other variables requires an explicit identification of its numerator and denominator with and .
Definition 1 (Monotonic decrease). The sequence is non-increasing if for every . Equivalently, whenever . Throughout this section, monotonically decreasing means non-increasing; equality between successive values is permitted.
Proposition 1 (Monotonicity in the denominator). For any fixed and , .
Proof. Positivity gives . Multiplying by preserves the inequality and yields .
Proposition 1 holds with the numerator fixed. It does not establish monotonic decrease over time when the numerator and denominator both change. A separate condition on their changes is required.
We propose a minimal increase in energy described by
Since an absolute value is nonnegative, this condition implies , rather than merely .
Theorem 1 (Exact condition for monotonic decrease). *Let and satisfy and

Figure 4. Zoom on the region where the orthograd algorithm is applied, for a Multi-Layer Perceptron using a modulo-113 addition dataset, at 16, 32 and 64-bit variable precision.
for every . Define . Then is non-increasing if and only if
Proof. Fix an arbitrary time . Since , multiplication by this product preserves inequalities. Hence
Applying this equivalence at every time proves both directions. Furthermore, if , repeated application of the adjacent-step inequalities gives . For the required inequality is equality.
Theorem 1 characterizes the additional condition needed. It does not establish that a training process satisfies that condition; such a conclusion requires separate assumptions or results about the accuracy and energy sequences.
Corollary 1 (Comparison of changes). Define and . Under the assumptions of Theorem 1,
If additionally , this is equivalent to
Proof. Substitute and into (3). Canceling gives (4). When , division by yields (5).
For positive accuracy, the proportional increase in cumulative energy must therefore be at least as large as the proportional increase in accuracy. The undivided condition (4) also covers . In that case, non-increasing efficiency requires .
Corollary 2 (A sufficient condition with non-increasing accuracy). If and at every step, then efficiency is non-increasing.
Proof. For every ,

Figure 6. Zoom on the region where the orthograd algorithm is applied, for a LeNet CNN using the MNIST dataset, at 16, 32 and 64-bit variable precision.
The first inequality uses the ordering of the numerators and positivity of ; the second uses Proposition 1.
If grokking produces an abrupt increase in the accuracy quantity used by this model, efficiency can either increase or decrease. For example, a change from to multiplies accuracy by four. Efficiency is non-increasing at this step exactly when . Equality leaves efficiency unchanged; a larger energy ratio makes it decrease strictly. Accordingly, a justified statement for our efficiency model is:
For positive cumulative energy, training efficiency is non-increasing provided that at every training step, including steps at which grokking occurs.
To claim that efficiency always decreases regardless of grokking, this condition must be derived from several training process measurements. Verifying it on measured trajectories establishes the property for those trajectories, but does not prove it for all possible training runs. A computational proof our statements using LEAN theorem prover is provided at -.
Analysis and Discussion
The research question of this paper was whether the monotonic decrease property of the training efficiency function holds at any point during training in the presence of grokking. Our experimental results indicate that yes training efficiency does hold the monotonic property even if grokking occurs, and that this holds across multiple model architectures (LeNet CNNs and MLP), tasks and through different variable precisions.
A potential contributing factor to this is that grokking occurs so late in training that the cumulative sum of efficiency terms is so large in comparison to the accuracy gains that these gains are insignificant in the overall calculations of efficiency. In this context our findings from the experiments in § 3.2.1 are important because they provide empirical evidence that indicate that grokking cannot occur early in a training process.
Additionally, the comparative analysis between different architectures such as the Multi-Layer Perceptron (MLP) and LeNet Convolutional Neural Network (CNN) highlights intriguing architectural dependencies in grokking behavior. CNN architectures tend to start growing closer to the collapse point2, whereas MLPs display variability in the grokking start and intensity. In other words, CCN models appear more sensitive to the orthograd model then MLPs, and this is a potential basis for future research on grokking.
Limitations. Our efficiency proxy depends on hardware and implementation details; thus further experiments with different learning tasks and across computing systems are recommended [7, 8, 9, 10].
Conclusion
In conclusion, this paper has provided empirical evidence that shows that the grokking phenomena does not affect the general dynamics of efficiency in neural network training processes. Furthermore, our results show that early perturbations (collapse) of the loss function does not induce grokking nor modify the dynamics of efficiency. Consequently, this paper has provided evidence that overfitting is a requirement for grokking to appear on a training regime, as no grokking happens at early stages of training the behavior of efficiency is not modified. To further reinforce the fact that the will always be stable under grokking a mathematical proof of the statement is included in § 6.
By systematically identifying and analyzing grokking events across different architectures and numerical precisions, the thesis demonstrates that efficiency is stable through grokking. More specifically, Equation (1) is stable, through sudden increases in testing accuracy, which happen when a Neural Network groks. We hypothesize that the reason why grokking does not cause an increase in efficiency at later stages in training is that grokking typically occurs so late on the training process that accumulated energy cost dominates the efficiency calculation and even large increases in accuracy cannot overcome the accumulated cost.
Similarly, our experiments also provide evidence showing that grokking is not present at early stages of training (up 500 epochs) even when applying the state-of-the-art technique of Softmax Collapse, and this further support the argument that training efficiency is a monotonically decreasing function through all the training of a DNN, not only through early stages of training, but also at later stages of training (up to 20,000 epochs).
The experimental challenges encountered in terms of numerical instability across long training runs foregrounded the need to adjust and calibrate the energy sampling rate during extremely long training runs in order to avoid underflow and overflow errors when using the efficiency measurement methodology proposed by relevant works. These challenges point to the need for the development of architectural and numerical strategies that enhance the robustness of the efficiency framework against numerical instability in future research. However, the next chapter of this thesis develops the efficiency methodology in a different direction.
AI use statement
In this work, we used generative AI tools for the preparation of the manuscript, specifically for assistance with LaTeX formatting of the figure environments in Section 3.2. We have not used generative AI tools for research ideation, experimental design, the implementation or execution of the experiments, the analysis of the resulting data, or the derivation of the proof in Section 6 with the exception of the provided LEAN4 proof that was assisted by AI; the substantive text of this work was written by the authors. Retrieval, translation and data-collection uses of generative AI are not applicable to this work. We have reviewed all AI-assisted work. Concretely, the AI-assisted changes were limited to the layout of existing figures, namely the choice of float environment, panel widths and caption placement; the figures themselves, their underlying data and their captions’ content were produced by the authors, and every cross-reference and the compiled output were checked by the authors against the source data. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.
Acknowledgments
All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support. To preserve the anonymity, please include acknowledgments only in the camera-ready papers. The acknowledgments do not count against the 9-page page limit in the camera-ready.
References
- [1]Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, January 2022. URL http://arxiv.org/abs/2201.02177. arXiv:2201.02177 [cs].
- [2]Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, October 2023. URL http://arxiv.org/abs/2301.05217. arXiv:2301.05217 [cs].
- [3]Ziming Liu, Ziqian Zhong, and Max Tegmark. Grokking as Compression: A Nonlinear Complexity Perspective, October 2023. URL http://arxiv.org/abs/2310.05918. arXiv:2310.05918 [cs].
- [4]Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data, October 2023. URL http://arxiv.org/abs/2310.02541. arXiv:2310.02541 [cs].
- [7]Domenico Pomarico, Alfonso Monaco, Giuseppe Magnifico, Antonio Lacalamita, Ester Pantaleo, Loredana Bellantuono, Sabina Tangaro, Tommaso Maggipinto, Marianna La Rocca, Ernesto Picardi, Nicola Amoroso, Graziano Pesole, Sebastiano Stramaglia, and Roberto Bellotti. Grokking as an entanglement transition in tensor network machine learning, March 2025. URL http://arxiv.org/abs/2503.10483. arXiv:2503.10483 [quant-ph].
- [8]Hari K. Prakash and Charles H. Martin. Grokking and Generalization Collapse: Insights from \texttt{HTSR} theory, June 2025. URL http://arxiv.org/abs/2506.04434. arXiv:2506.04434 [cs].
- [9]Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the Edge of Numerical Stability, May 2025. URL http://arxiv.org/abs/2501.04697. arXiv:2501.04697 [cs].
- [10]Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The Complexity Dynamics of Grokking, December 2024. URL http://arxiv.org/abs/2412.09810. arXiv:2412.09810 [cs].
