WorksComputer Science
Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition
Abstract
Grokking describes understanding something deeply and intuitively. In neural networks, it is the phenomenon where a network generalizes long after memorizing its training data. This delay has been linked to a fall in the effective rank of the network's weight matrices, but whether this rank compression drives grokking or follows it is still unresolved. We study a mean-field MLP and a one-layer transformer trained on modular arithmetic and observe that: (i) Compression follows the onset, not memorization. Every transformer memorizes its training data within 250 steps, while the grokking onset (the first step at which test accuracy reaches 95%) varies from 6,500 to 21,000 steps. The steepest fall in effective rank comes about 900 steps after the onset, however late the onset occurs, and networks that memorize without grokking compress far less. (ii) Subspace crystallization. We call the span of the fewest top singular directions of a weight matrix that preserve 99% of test accuracy its sufficient subspace. Its size can shrink as the margin between the correct and the best wrong output grows, so we also count the directions needed to keep the outputs within 10%. In the transformer, this count falls during the transition from a quarter or more of the available directions to about a tenth, which we call subspace crystallization. (iii) Imposing low rank does not speed up grokking. Against a factorized full-rank control, capping the rank of the transformer's MLP input weights delays the onset, and capping the rank of its embedding stops 15 of 20 runs from grokking. Compression is therefore a signature of grokking rather than its cause.
Introduction
Grokking [29], memorization followed by delayed generalization, offers a rare window into how circuits emerge from gradient descent: a clean phase transition from a lazy, memorizing regime to a rich, generalizing one [17]. Substantial progress exists on what grokking networks compute [28, 5, 11], and a recurring observation is that the effective rank (ER) of their weight matrices falls as test accuracy jumps [21, 8, 18]. This has led to proposals that a pressure toward low-rank, small-norm or efficient circuits causes grokking [28, 20, 35]. We ask two questions. (Q1) Does this rank compression cause grokking, or follow it? (Q2) Does the learned solution lie in the subspace spanned by a few leading directions of each weight matrix?
Preprint. The usual evidence cannot settle these questions, for three reasons. First, rank compression is usually observed after two events, memorization (the first step at which training accuracy reaches 99%) and the rise in test accuracy (Figure 1a), so the timing alone cannot explain which of the two it follows. Second, timing without interventions cannot show cause. Lastly, the compactness of a solution is naturally measured by truncating each weight matrix to lower and lower rank until test accuracy drops. Keeping a weight matrix’s strongest singular directions gives the closest rank- matrix, which acts only within their span. The span of the fewest top directions that leave the test accuracy unchanged, which we call the sufficient subspace, is then taken as the subspace the solution needs, analogous to the intrinsic dimension of Li et al. [ [19]. Accuracy, however, only checks which output is largest. The margin between the correct-class output and the largest other output sets how far the outputs can move before a prediction changes. As margins grow, more directions can be deleted before any prediction changes, so the count can fall even though the network’s outputs still depend on those directions.

Figure 1. (a) Step of the steepest effective-rank decrease of against the grokking onset (left) and the step at which training accuracy reaches 99% (right). Blue: seeds 0–14. Orange: held-out seeds 15–29. Dashed line: identity.
(b) Number of directions of each weight matrix needed to keep the network’s outputs within 10% (left) and test accuracy (right), with each seed aligned at its own onset .
Compression follows the grokking onset, and the directions needed to keep the outputs fall during the transition (one-layer transformer, 29 grokked seeds). (a) Every seed fits its training data by step 250, yet compression follows the grokking onset across a threefold range of onset times. (b) The number of directions needed to keep the outputs within 10% falls during the rise in test accuracy and settles at 8–13 per matrix.
We address these questions with geometric metrics (effective rank, stable rank, sparsity, inverse participation ratio) and multiple interventions (truncation of weight matrices to their top, bottom or random singular directions, complement interventions, transplant patching with orthogonal alignment, a nuclear-norm penalty, rank caps compared with factorized controls, and optimizer swaps). We study modular addition mod 97 in two architectures with multiple random initializations. In the transformer, we analyze the embedding , the unembedding , and the input and output matrices and of its MLP block. We call the first step at which test accuracy reaches 95% the grokking onset. To separate memorization from the onset, we follow the transformer with a checkpoint every 250 steps and compare networks that grok with networks that memorize but never grok. To count directions without the margin, we judge each truncation by how much it changes the network’s outputs. To test cause, we limit the rank of a weight matrix throughout training and compare each limit with a factorized full-rank control. We test the timing results on held-out transformer seeds and on two further tasks, mod 97 and mod 113. Our results point to compression as a signature of grokking rather than its cause.
Contributions
Compression follows grokking, not memorization (§6). Every transformer memorizes its training data within 250 steps, but the steepest fall in the effective rank of comes about 900 steps after the grokking onset, whether the onset occurs at step 6,500 or 21,000. Networks that memorize without grokking lower this effective rank by 15–23%, against in those that grok. In the MLP, rank compression follows the onset in all 45 runs across three tasks. The timing replicates on 15 held-out seeds and, in the MLP, on two further tasks.
Subspace crystallization: the transformer’s solution narrows to a low-dimensional subspace (§5, §7). Before the transition, no weight matrix can be truncated below 27–53 directions without changing the network’s outputs by more than 10%. After the transition, 8–13 directions suffice (Figure 1b). We call this narrowing of the subspace that the outputs depend on subspace crystallization. It is consistent with the removal of memorizing components described by Nanda et al. [ [28]. We judge truncations by the outputs rather than only by the test accuracy. In the MLP trained with mean-squared error, truncation to 65 of 128 directions keeps the test accuracy, but keeping the outputs within 10% needs 122–127.
Imposing low rank before the transition does not speed up grokking (§6.4). If rank compression drove grokking, removing directions in advance should help. Instead, against a factorized full-rank control, capping at rank 32, 24 or 16, all above the 13 directions its solution needs, never lowers the mean onset, and capping prevents grokking in 15 of 20 runs. Compressing the MLP’s first layer with a nuclear-norm penalty before the onset also delays grokking, and comparing with the unfactorized network instead of the factorized control would wrongly suggest that low rank speeds grokking up, because the factorization changes the onset by itself.
We also report three secondary results. The late fall of effective rank in the MLP is set by AdamW’s : once the square root of its second-moment estimate falls below , its updates reduce to gradient descent with momentum and learning rate (§4). Different MLP seeds share their hidden representation up to an orthogonal rotation, but not with networks trained on another task or on random labels (§5). The Fourier inverse participation ratio (IPR), which measures how periodic the hidden units are [Doshi et al., [9]], rises before the onset but also in every network that never groks, so it cannot serve as a label-free early-warning signal (§ 6.5). Table 1 summarizes the evidence for the three contributions in both architectures.
| MLP | Transformer | |
| 1. Compression follows grokking, not memorization | ||
| Compression starts after test accuracy rises | 45 of 45 runs | 28 of 29 seeds |
| Fall in effective rank | 10% if grokked | 60 ± 5% if grokked, 15–23% if not |
| 2. Subspace crystallization: the transformer’s solution contracts into a low-dimensional subspace as it groks | ||
| Directions needed per matrix | Accuracy: from 110 to 65 of 128<br>Outputs: 122–127 of 128 (MSE) | Outputs: from 27–53 to 8–13, during the transition |
| 3. Imposing low rank does not speed up grokking | ||
| Rank caps against a factorized control (Table 4) | Caps delay the onset | : onset unchanged or delayed<br>: 15 of 20 runs fail to grok |
Table 1. Evidence for the three contributions in both architectures. Effective rank is measured on (MLP) and (transformer). Output counts are the fewest directions per matrix that change the network’s outputs by at most 10%.
Related Work
Grokking. Grokking was given its first mechanistic account by Nanda et al. [ [28], who identified the learned algorithm as a sparse Fourier cosine decomposition, formed in phases of memorization, circuit formation and cleanup. Chughtai et al. [ [5] extended this to general group composition, and Gromov [ [11] established exact solvability with analytic weights. Proposed causes of the delay include circuit efficiency [35], the weight norm [20], softmax collapse [30] and a transition from lazy to rich training [17]. Lei and Xu [ [18] describe grokking as construction followed by compression, tracked through the network’s Jacobian, consistent with our timing results. We add the separation of compression from memorization, negative controls and a replication on held-out seeds and tasks. Musat [ [27] proves that, in the limit of small learning rate and weight decay, gradient descent after memorization minimizes the weight norm on the zero-loss manifold. Our MLP lies outside this regime: late in training, AdamW acts as gradient descent with a large learning rate , and the weight norm grows (§ 4).
Complexity measures of grokking. Network complexity falls around the accuracy jump, whether measured by the linear mapping number [21] or by the compressed description length of the weights [8]. We test whether this fall drives grokking or follows it, and show that a count of directions judged by accuracy can register compression that the network’s outputs do not show (§ 5).
Low-rank bias and factorized controls. Gradient descent is biased toward low-rank solutions [13] and moves within a small subspace of parameter space [12]. Factorizing a matrix strengthens this bias [3] and accelerates optimization [2], so a rank cap written as a product changes the optimization as well as the rank. We therefore compare each cap with a factorized full-rank control (§ 6.4).
Sparse circuits and universality. Merrill et al. [ [24] describe grokking as a competition between a sparse generalizing subnetwork and a dense memorizing one, consistent with the few directions per matrix that the transformer’s solution keeps. Networks trained on modular arithmetic can learn different algorithms [38] or share one across architectures [5]. Our MLP seeds use different Fourier coordinates but share their hidden representation up to a rotation (§ 5). Appendix § 12 discusses further work.
Setup
Architecture 1: mean-field MLP. A single-hidden-layer MLP with 128 hidden units and quadratic activations, with and input , the concatenation of two one-hot vectors (, , i.i.d. initialization). The quadratic activation makes the Fourier computation explicit: for it doubles frequencies, , and applied to a sum of operand features it produces the products from which is read out. We train it with mean-squared error to one-hot targets, as in Gromov [ [11], using full-batch AdamW (lr , weight decay , ) on a 35% train split for steps, with 15 seeds. Five MLPs trained on randomly permuted labels serve as negative controls, and for the replication the MLP is also trained on and (15 seeds each).
Architecture 2: one-layer transformer. Four attention heads, , , , ReLU MLP, no layer normalization and biases fixed at zero, following Nanda et al. [ [28]. The inputs are and the prediction is read at the final position. We train it with cross-entropy, using full-batch AdamW (lr , weight decay 1.0, ) on a 30% train split for 25,000 steps, with 15 seeds. Fourteen of these seeds reach 95% test accuracy within this budget. The remaining seed, and three seeds each trained with weight decay 0.1 and 0.3, memorize the training set without generalizing and serve as negative controls. Seeds 15–29, trained only after the timing measure had been defined on seeds 0–14, form the held-out set of the replication, and all of them grok. For the time-resolved analyses, every run was repeated with a checkpoint every 250 steps. We analyze the embedding , the unembedding , the MLP projections and , and the attention projections , , , (heads stacked).
Timing. The grokking onset is the first logged step with test accuracy , and the memorization step the first logged step with training accuracy . A run counts as grokked if its final test accuracy is at least 95%. Runs described as never grokking did not reach this threshold within their training budget ( steps for the MLP, 25,000 for the main transformer runs). Throughout the paper, “before” and “after generalization” mean before and after .
Effective rank. Our primary metric is the effective rank [32], the exponential of the entropy of the normalized singular values . A matrix whose nonzero singular values are equal has . Appendix § 13 defines the other tracked metrics (stable rank, sparsity, spectral IPR, weight and gradient norms), and § 6.5 the Fourier IPR.
Code to reproduce all experiments is available at https://github.com/Lossfunk/Causal-geometric-grokking.
The Four-Stage Training Trajectory (MLP)
Figure 2 shows the four stages through the rates of change of effective rank and test accuracy. Numbers are means s.d. over 15 seeds, and Appendix Figure 3 shows all training metrics.

Figure 2. The four training stages of the MLP (seed 0): rate of change of the effective rank of (top) and of test accuracy (bottom), computed from the five-step moving average used for the steepest-decrease measure (Appendix § 16.§ 1). Stage 2 starts at the first step with test accuracy above 5%, Stage 3 at and Stage 4 at . Effective rank grows during memorization and the grokking transition, turns negative at and keeps falling through Stages 3 and 4.
Stage 1 (Memorization): training accuracy rises to by step 8,000 while test accuracy stays below 5%. Test loss rises before it falls, a double-descent signature [7]. Effective rank rises slightly, from 114.6 for the random initial to about 116.
Full sweep in Appendix C.2. Gradient descent at diverges.
Stage 2 (Grokking transition): test accuracy rises from 37.0 8.0% at step 10,000 to 95% at , and training accuracy reaches 100%. Effective rank rises only slightly (115.7 0.4 at step 10,000, 115.9 0.6 at step 12,500), while the hidden units become periodic in their inputs, as measured by the Fourier IPR (§ 6.5), and the gradient norm peaks just after .
Stage 3 (Consolidation): test accuracy is 100% and ER stays near its maximum until the compression onset , the first step at which it falls below 99% of its running maximum. Over the same interval the median margin grows from 0.105 0.017 at step 12,500 to 0.470 0.019 at step 20,000, and the directions of needed to keep 99% of test accuracy fall from 109.5 3.2 to 64.9 1.7, a consequence of the growing margin (§ 5). The stable rank of peaks around and starts to fall.
Stage 4 (Late compression): ER falls to 104.0 2.1 by step , and 99 2% of its total decrease happens after . The stable rank keeps falling, the weight norms grow throughout and the margin barely changes (0.599 to 0.611 between steps 50,000 and ). Observing this stage requires training well past the point where test accuracy saturates.
Stage 4 is set by the effective learning rate. Stage 4 is growth of the leading singular directions of against weight decay: between steps 20,000 and the largest singular value doubles (112 to 218) while the remaining 63 directions keep their size (Appendix C.1). The loss gradient that drives this growth is tiny ( at step 20,000), points increasingly outward along the weights (its cosine with rises from 0.10 to 0.89) and concentrates in the top 65 directions of (from 43% to 87% of its squared norm). For 87–100% of the coordinates, the square root of Adam’s second-moment estimate lies below , so the update is close to : AdamW acts as gradient descent with learning rate lr (Appendix C.2). Continuing the main runs from step 20,000 with other optimizers confirms that this effective learning rate, not Adam itself, governs Stage 4 (Table 2). At effective learning rates of and , AdamW and gradient descent reach the same weights. At , gradient descent reproduces most of Stage 4 and SGD with momentum reproduces all of it (effective rank 108.3 and 103.8, against 104.8 under AdamW). At , weight decay shrinks the norm and effective rank stays at 117.8, while strengthens Stage 4.
| Optimizer | Effective lr | ||||
| Start (step 20,000) | – | 726 | 111 | 115.1 | 65 |
| AdamW, | 658 | 85 | 117.8 | 65 | |
| Gradient descent | 662 | 85 | 117.8 | 65 | |
| AdamW, | 775 | 120 | 114.8 | 70 | |
| Gradient descent | 783 | 126 | 114.6 | 72 | |
| AdamW, (main) | 1,040 | 203 | 104.8 | 89 | |
| Gradient descent | 833 | 148 | 108.3 | 87 | |
| SGD with momentum 0.9 | 1,093 | 216 | 103.8 | 91 | |
| AdamW, | 1,582 | 318 | 94.7 | 98 | |
| AdamW, , no weight decay | 1,257 | 261 | 97.8 | 94 |
Table 2. Stage 4 tracks the effective learning rate. Main seeds 0–2 are continued from step 20,000 to $10^5 with each optimizer, and entries are means. The effective learning rate is lr for gradient descent, lr/() for SGD with momentum 0.9, and lr for AdamW (lr ). For , lr is an upper bound, since is comparable to . Weight decay is set so that the per-step shrinkage matches AdamW’s. : fewest directions that keep 99% of test accuracy.
Weight decay, loss and activation. The MLP (with MSE loss) groks without weight decay: from weight decay 0 to 0.1, barely moves (12,700 47 to 13,400 418), but the Stage-4 decrease in effective rank shrinks from 14.9 1.9% to 0.5 0.4% (9.7 1.5% at the default ), because weight decay restrains the growth of the leading directions. Trained with cross-entropy at learning rates and , the MLP groks in every seed ( and 10,100 962) and shows Stage-4 decreases of and . At lr it groks in only one of five seeds within steps. ReLU and GELU MLPs also grok, later than quadratic ones, with periodic first-layer weights and Stage-4 decreases of 14–17% (Appendix B.2). With the same hyperparameters, the quadratic MLP groks 2.6 to 3.5 times sooner than ReLU and GELU MLPs, so the quadratic activation speeds up grokking in our setting but is not required for the MLP to grok (Appendix B.4).
Subspace Crystallization: Locating the Sufficient Subspace
This section defines the sufficient subspace and the output count , and uses them to test whether the MLP crystallizes. The transformer’s crystallization is shown in § 7.
Definitions. We replace a weight matrix by its top- truncation and leave the rest of the network unchanged. is the smallest that keeps 99% of the test accuracy, and the span of the top- left singular vectors is the sufficient subspace. Because accuracy only checks which output is largest, can fall as margins grow. is the smallest with over the test inputs, where stacks the network’s outputs and, for cross-entropy models, each input’s mean logit is removed, since softmax ignores it. Subspace crystallization is a narrowing of . The leading directions carry the solution: truncation to the top directions keeps at least 99.26% of test accuracy, while random subspaces of equal rank keep 22.7% in the MLP and at most 12.5% in the transformer (Table 15).
| Matrix | Top- | Bottom- | Random subset | Random subspace | |
| MLP | / 128 | * | |||
| Transformer | / 128 | ‡ | |||
| Transformer | / 128 | ‡ | |||
| Transformer | / 97 | ||||
| Transformer | / 98 |
Table 15. Top-, bottom- and random- truncations at equal rank (per seed). Test accuracy in %, mean s.d. over 15 MLP seeds at step and 14 grokked transformer seeds. Random controls average 10 draws. Chance is 1.03%.
In the MLP, crystallization is only apparent. With mean-squared error, crystallization is only apparent. The sufficient subspace of narrows right after the onset, from at step 12,500 to at step 20,000, and widens again to by step (Table 13). Over the narrowing, however, the median margin grows more than fourfold and does not fall: it is before the onset, at step 20,000 and at the end (Table 3), above the of an untrained network and the of a network trained on random labels. The same holds on both replication tasks (Appendix D.4). The narrowing of therefore measured how far the predictions were from flipping, not how many directions the function uses. § 8 attributes the high to the loss: mean-squared error to one-hot targets requires reproducing every Fourier component of the outputs. The later widening of the sufficient subspace comes from the growth of the leading singular directions, which makes the trailing directions matter more for the outputs (Appendix C.1). The complement is unused for the predictions but is not inert: Gaussian noise added inside lowers accuracy more than the same noise inside (71% against 92%, Appendix D.3). With cross-entropy, which only requires the correct output to be largest, the MLP does crystallize, but late: just before and just after the onset, and (learning rate ) and () at the end of training (Table 3).
| Model | Checkpoint | Outputs within 10% | 99% of accuracy | Median margin |
| MSE (15 seeds) | step 0 (untrained) | – | ||
| step 11,000 (before onset) | ||||
| step 20,000 | ||||
| end () | ||||
| MSE, random labels (5) | end, training inputs | – | ||
| CE, lr (5) | last before onset | |||
| first after onset | ||||
| end | ||||
| CE, lr (5) | last before onset | |||
| first after onset | ||||
| end |
Table 3. and in the MLP (directions of out of 128, on test inputs unless noted, mean s.d.). : fewest directions that keep the outputs within 10%. : fewest directions that keep 99% of test accuracy. Cross-entropy (CE) models use centered logits.
| Step | Test acc. (%) | Freq. for 90% power | |||
| 10,000 | 37.0 8.0 | 97.5 3.2 | 120.5 3.5 | 115.7 0.4 | 40.8 0.4 |
| 11,000 | 67.3 8.9 | 95.2 2.1 | 119.3 3.2 | 115.8 0.4 | 40.3 0.7 |
| 12,500 | 95.7 2.7 | 81.7 3.2 | 109.5 3.2 | 115.9 0.6 | 39.7 0.5 |
| 15,000 | 100.0 0.0 | 59.6 1.8 | 77.0 3.6 | 115.6 0.8 | 38.5 0.6 |
| 20,000 | 100.0 0.0 | 50.8 2.5 | 64.9 1.7 | 114.4 1.1 | 38.2 1.1 |
| 30,000 | 100.0 0.0 | 56.4 2.4 | 69.3 1.7 | 110.9 1.5 | 38.3 0.9 |
| 50,000 | 100.0 0.0 | 68.3 2.6 | 81.5 2.4 | 107.0 1.9 | 36.1 1.4 |
| 100,000 | 100.0 0.0 | 80.5 3.7 | 89.1 2.3 | 104.0 2.1 | 33.7 1.9 |
Table 13. , , effective rank and frequency usage of over training (15 seeds).
Equivalent solutions across random initializations. MLP seeds implement the same solution in different coordinates. Each seed spreads its first-layer Fourier power over most of the 48 frequencies (45.2 0.6 hold 99%), with different strongest frequencies (Jaccard overlap 0.05 over 105 seed pairs), and the top-60 weight subspaces of different seeds overlap no more than random subspaces (0.466 0.027 against 0.487 0.005 in mean cosine of the principal angles, Appendix D.5). Yet after an orthogonal Procrustes rotation fitted on training inputs [34], one seed’s readout decodes another seed’s post-activations with 95.8 2.4% accuracy, against 1.0% without the rotation (Table 17). The rotation does not rescue a random network, a network trained on random labels or a network grokked on mod 97 (0.6–1.4%). Each seed thus implements the same task-relevant representation in a different orientation, consistent with Schiffman [2026], who find invariant algorithmic cores in independently trained transformers.
Table 17: Cross-seed transplants and linear CKA (MLP, final checkpoints unless noted, mean s.d.). Transplant: test accuracy of model ’s readout applied to model ’s post-activations after the map (chance 1.03%). CKA on all inputs.
| Model grokked model | Map | Test accuracy (%) |
| Grokked, other seed | fitted orthogonal | 95.82 2.38 |
| Grokked, other seed | none | 1.03 0.30 |
| Grokked, other seed | random orthogonal | 0.94 0.27 |
| Random initialization | fitted orthogonal | 0.61 0.12 |
| Random-label network | fitted orthogonal | 0.68 0.07 |
| Grokked on mod 97 (other task) | fitted orthogonal | 1.42 0.09 |
| Same seed at step 10,000 | none | 36.18 6.40 |
| Same seed at step 10,000 | fitted orthogonal | 25.09 5.77 |
| Linear CKA between grokked seeds | 0.096 0.019 | |
| a grokked network and its initialization | 0.089 0.010 | |
| two initializations | 0.142 0.002 |
Table 17.
Timing and Cause of Compression
We test whether compression drives grokking with three kinds of evidence: its timing across seeds, how much the effective rank decreases in networks that memorize without grokking, and interventions that impose low rank from initialization.
Compression Follows the Onset, Not Memorization
The transformer separates memorization from the transition, and compression follows the transition. Every one of the 29 grokked seeds (seeds 0–14 and the held-out seeds 15–29) fits its training set within the first 250 steps, while their onsets range from 6,500 to 21,000 steps. With weight decay 1.0, AdamW multiplies the weights by 0.999 per step, so by step 4,000 the initial weights have decayed to 2% of their size. Because random weights spread over all directions, this decay alone lowers the effective rank of , to a plateau at about 60% of its initial value, independently of grokking (Figure 9). We therefore measure compression only after step 4,000. After this decay, a second, sharp drop follows. The step at which this second drop is steepest follows the onset almost exactly (Pearson correlation , slope 1.02, 95% confidence interval 0.98–1.07), scattering by 506 steps around against 4,234 around the memorization step (Figure 1a). If compression followed memorization, would cluster at a fixed step.

Figure 9. Effective rank of (relative to step 0) and test accuracy of the 14 grokked transformer seeds against steps relative to . The effective rank falls steeply while the random initialization decays, plateaus at about 60% of its initial value, and drops again around .
Networks that memorize without grokking compress far less. From step 4,000 to the end of training, the ER of falls by 60 5% in grokked transformers but by only 15–23% in the seven that never grok, at a peak rate two to five times lower (Appendix E.2). In the MLP, whose onset and memorization time barely vary across seeds, the random-label networks reach 90% training accuracy without grokking, and their effective rank falls by 1.1 0.3%, against 10.5 1.8% in grokked MLPs. Comparing seeds at the same training step shows that low rank is acquired at the transition itself (Appendix Table 21): before their own onset, grokked seeds have as many directions as the seed that never groks, and after it far fewer.
Compression Does Not Start Before the Rise in Test Accuracy
In the MLP, the compression onset follows the grokking onset in every seed (Appendix Table 18 and Figure 11). All 15 seeds have , with mean lag 6,367 3,248 steps (one-sided , ), i.e. 50 25% of . Granger tests agree: changes in test accuracy predict changes in effective rank in all 15 seeds, and the reverse holds in at most 6 (Appendix E.1).
Figure 11. (a) vs. : all 15 seeds above the diagonal. (b) Mean cross-correlation ( s.d.) of and .
In the transformer, compression is part of the transition. We locate the start of the second drop as the first step of the stretch leading up to in which the decrease is at least 20% of its peak rate, using a centered five-point moving average that adds no delay (Appendix E.1). The drop starts 1,888 992 steps before the onset, but after test accuracy has first reached 10% in 28 of 29 seeds (median 1,750 steps after), at 34 22% test accuracy. It is steepest 905 506 steps after the onset (27 of 29 seeds), and 63% of it happens after the onset. In both architectures, then, compression does not start before test accuracy begins to rise. In the MLP it starts after the transition, and in the transformer during it.
Replication on Held-Out Tasks
We tested the timing results on seeds and tasks not used to define the measures, and report every test (Table 22). On the 15 held-out transformer seeds (H1), the steepest decrease came 850 524 steps after the onset (14 of 15 seeds, the fifteenth at the onset, one-sided , ), close to the 964 ± 499 steps of seeds 0–14, on which the measure had been defined. On mod 97 and mod 113, the MLP’s compression onset (H2) and steepest decrease (H3) followed the grokking onset in all 30 runs. The narrowing and re-widening of (H4, H5) and the effect of the factorization (H6) also replicated. With a Bonferroni correction over the nine -tests, every test except H4 on mod 113 () remains significant.
| Quantity | Transformer, held-out | MLP, mod 97 | MLP, mod 113 |
| H1: (Win) | (14/15)* | – | – |
| H2: | – | (15/15)* | (15/15)* |
| H3: () | – | (15/15)* | (15/15)* |
| H4: | – | (15/15)* | (12/15)* |
| H5: | – | (15/15)* | (15/15)* |
| H6: for at 128/64/32, and unfactorized | – | 2,000/2,667/4,667 and 12,833 | 2,000/3,000/4,500 and 15,333 |
Table 22. Replication tests (one-sided tests, mean s.d. in steps or directions, with the number of seeds showing a positive difference in parentheses).
Forcing Low Rank from Initialization
Forcing low rank before the transition does not bring grokking forward (Table 4, Table 25). We write a weight matrix as with inner dimension , initialized so that the entries of have the variance of the original initialization, and compare each rank cap with a factorized control whose inner dimension is at full rank. The control matters because the factorization alone changes the grokking time.
| Matrix | Grokked | Final test accuracy (%) | ||
| None (unfactorized) | — | 5/5 | 10,500 ± 1,777 | 100.0 ± 0.0 |
| 128 | 5/5 | 14,100 ± 4,095 | 99.6 ± 0.3 | |
| 32 | 5/5 | 14,350 ± 5,730 | 98.4 ± 1.6 | |
| 24 | 4/5 | 16,312 ± 3,332 | 95.5 ± 7.2 | |
| 16 | 3/5 | 16,417 ± 6,682 | 94.5 ± 7.7 | |
| 8 | 1/5 | 12,500 | 50.6 ± 39.7 | |
| 128 | 5/5 | 18,050 ± 5,917 | 100.0 ± 0.0 | |
| 32 | 1/5 | 15,750 | 31.8 ± 39.1 | |
| 24 | 4/5 | 16,312 ± 2,633 | 83.1 ± 37.4 | |
| 16 | 0/5 | — | 11.9 ± 9.5 | |
| 8 | 0/5 | — | 13.7 ± 11.8 |
Table 4. Rank caps in the transformer ( with inner dimension , 5 seeds per arm, averaged over the seeds that grok, mean ± s.d.). is the factorized control without a rank limit.
| Model | Grokked | Final test acc. (%) | Final ER() | |
| Unfactorized (main) | 3/3 | 12,833 289 | 100.0 0.0 | 104.9 1.1 |
| , (control) | 3/3 | 2,000 0 | 99.9 0.2 | 89.1 0.9 |
| , | 3/3 | 2,167 289 | 100.0 0.0 | 63.0 5.7 |
| , | 3/3 | 2,500 0 | 100.0 0.0 | 49.4 1.2 |
| , | 3/3 | 2,667 289 | 100.0 0.0 | 46.0 3.7 |
| , | 3/3 | 3,167 289 | 100.0 0.0 | 38.0 0.6 |
| , | 3/3 | 4,500 0 | 99.5 0.3 | 29.5 0.1 |
| , | 0/3 | – | 55.3 10.6 | 15.8 0.0 |
| , | 0/3 | – | 27.8 6.1 | 8.0 0.0 |
| No penalty | 5/5 | 12,700 447 | 100.0 0.0 | 104.9 1.6 |
| 5/5 | 12,800 274 | 100.0 0.0 | 103.5 2.2 | |
| 5/5 | 12,800 274 | 100.0 0.1 | 106.3 1.3 | |
| 5/5 | 16,200 447 | 100.0 0.0 | 100.7 3.4 | |
| 0/5 | – | 0.8 0.1 | 1.2 0.3 | |
| 0/5 | – | 1.1 0.1 | 110.8 0.4 |
Table 25. Forcing low rank from initialization (MLP). Rank-constrained models use 3 seeds and the penalty 5 seeds.
Transformer. The transformer’s function needs only about 13 directions of and 8 of , so caps above these numbers leave the solution’s capacity intact. Capping at 32 directions leaves the onset where the factorized control has it (14,350 ± 5,730 against 14,100 ± 4,095 steps), capping it at 24 or 16 delays it and makes it less reliable (four and three of five seeds grok), and capping it at 8, below the 13 directions the solution uses, prevents it in four of five seeds. Capping is worse: five of twenty capped runs grok. The four that grok at rank 24 do so at 16,312 ± 2,633 steps, against 18,050 ± 5,917 for the control. Whatever the cap, the capped matrix ends at an effective rank of 8–12, like its factorized control.
MLP. In the MLP the factorization alone accelerates grokking seven- to eight-fold ( for at full rank against 12,483 ± 340 for an unfactorized , logged every 50 steps, and 7.2 and 8.4 times on the two replication tasks), whereas it slows the transformer (14,100 ± 4,095 against 10,500 ± 1,777 steps for , Table 4). The factorization’s own effect is consistent with two known mechanisms. For gradient descent on linear networks, Arora et al. [ [2] show that factorizing a matrix preconditions the updates of the product, combining an adaptive learning rate with a form of momentum, which can accelerate convergence. Weight decay on and penalizes the nuclear norm of , since . This penalty is negligible at the MLP’s weight decay of but strong for the transformer’s , consistent with the factorization speeding up the MLP and slowing the transformer. Against the factorized control, rank limits delay grokking (2,450 ± 100 steps at , 4,133 ± 29 at ), and models with do not grok within steps. Compared with the unfactorized alone, low rank would wrongly appear to accelerate grokking (Appendix § 16.§ 4). Because the MLP’s solution uses most of its directions, these delays say less about compression than the transformer caps do. A nuclear-norm penalty with from step 0 compresses well before grokking (effective rank 78.5 ± 3.4 at step 11,500, against 115.9 ± 0.3) and delays grokking from 12,700 ± 447 to 16,200 ± 447 steps, and prevents learning (5 seeds per ).
IPR Is Not a Specific Early-Warning Signal
A hidden unit of the generalizing solution reads its input as a single cosine wave over the residues. The Fourier IPR measures how close the units are to this. Following Doshi et al. [ [9], with the magnitude of the discrete Fourier transform of unit ’s weights, normalized to unit norm,
We use the weights on operand only, where Doshi et al. [ [9] average over each unit’s input and output weights. A unit tuned to a single frequency has IPR 1/2, and unstructured weights give about 0.02–0.03. For the transformer we compute it for the embedding, over the 97 tokens. Doshi et al. [ [9] use it to separate generalizing from memorizing neurons and show that it rises during grokking. We test whether this rise can serve as a label-free early-warning signal of the onset.
IPR rises before the grokking onset, but also in every network that never groks, so it tracks the progress of training rather than an impending onset (Table 29). In the MLP, the Fourier IPR crosses a threshold at the end of whatever baseline window is chosen and fires in all five random-label networks. In the transformer, it fires before in nearly every grokked seed, but also in all seven runs that do not grok (Appendix § 16.§ 5).
| Model | Metric | Early | At window end | False alarms | Lead | |
| MLP | Fourier IPR () | 2,500 | 15/15 | 14/15 | 5/5 | 10,000 |
| 5,000 | 15/15 | 15/15 | 5/5 | 7,500 | ||
| 10,000 | 15/15 | 14/15 | 5/5 | 2,500 | ||
| MLP | Gradient norm | 2,500 | 15/15 | 0/15 | 5/5 | 9,000 |
| 5,000 | 15/15 | 0/15 | 0/5 | 2,500 | ||
| 10,000 | 3/15 | 0/15 | 3/5 | 0 | ||
| Transformer | Fourier IPR () | 2,500 | 14/14 | 0/14 | 7/7 | 7,625 |
| 5,000 | 14/14 | 0/14 | 7/7 | 5,250 | ||
| 10,000 | 9/9 | 6/14 | 7/7 | 1,000 | ||
| Transformer | Spectral IPR () | 2,500 | 14/14 | 0/14 | 7/7 | 3,125 |
| 5,000 | 14/14 | 0/14 | 7/7 | 1,625 | ||
| 10,000 | 9/9 | 5/14 | 3/7 | 750 |
Table 29. Label-free onset detection (threshold above the mean of the baseline window ). At window end: fires at the first logged step after the window. False alarms: fires in runs that never generalize (5 random-label MLP runs and 7 non-grokked transformer runs). Lead (median, steps).
Subspace Crystallization in the Transformer
Time-resolved results are means s.d. over the 29 grokked seeds, and final-checkpoint results over the 14 grokked seeds of 0–14.
The transformer crystallizes during the transition. With a checkpoint every 250 steps and each seed aligned at its own onset, narrows during the rise in test accuracy (Figure 1b, Table 5). Three thousand steps before the onset, at 16% test accuracy, is 27–53 per matrix. At the onset it is 11–17, and 3,000 steps later 9–14. For , and it is half-way down 740–960 steps before the onset, in 28 or 29 of 29 seeds, at about 60% test accuracy. narrows at the onset and keeps narrowing afterwards. The narrowing coincides with the cleanup phase of Nanda et al. [ [28]. Their restricted loss, which keeps only the key-frequency components of the logits, is half-way down 1,966 647 steps before the onset, as the generalizing circuit forms. Their excluded loss, which removes these components, is half-way up 1,026 646 steps before the onset, as the memorizing components are removed, at the same time as narrows. adds that the cleanup reduces the directions the outputs depend on, not only those the predictions depend on, and the MLP trained with mean-squared error shows no equivalent (§5). Transformers that never grok narrow only slowly and stay two to nine times wider (Appendix § 17.§ 2).
| Step | Test acc. (%) | Restricted | Excluded | ||||
| 15.7 | 2.09 | 4.8 | |||||
| 26.2 | 1.37 | 8.1 | |||||
| 57.7 | 0.64 | 14.7 | |||||
| 82.4 | 0.26 | 16.6 | |||||
| 0 | 96.9 | 0.12 | 16.2 | ||||
| 99.9 | 0.03 | 14.0 | |||||
| 100.0 | 0.12 | 18.4 |
Table 5. The transformer crystallizes during the transition (29 grokked seeds, each aligned at its own onset , with checkpoints every 250 steps). of each matrix (test inputs, mean s.d.), test accuracy, and the restricted and excluded training losses of Nanda et al. [ [28] (means).
The final solution uses few directions. At the final checkpoint, every analyzed matrix except is far below full rank (Table 30). is of 97 for , of 128 for and of 128 for , and the MLP post-activations need of 512 directions (3.4%, truncating the mean-centered activation matrix). These counts are not inflated by the margin: is 8.1–13.0 for , , and .
Table 30: Transformer dimensionality (14 grokked seeds, final checkpoint). , : singular directions carrying 95% and 99% of . : fewest directions that keep 99% of test accuracy. Attention projections are stacked across the four heads. Activations are truncated after mean-centering.
| Matrix | Max rank | |||
| (embedding) | 97 | |||
| 32 | / | |||
| 32 | ||||
| 128 | ||||
| 128 | ||||
| 128 | ||||
| (unembedding) | 98 | |||
| MLP post-activations | 512 |
Table 30.
Runs that never grok keep high-rank matrices. The seven runs that memorize without generalizing have far higher-rank MLP and unembedding matrices. Counting the singular directions that carry 99% of , they keep 68–84 for (grokked: 18.1 3.8), 61–79 for (11.8 2.5) and 62–73 for (23.8 2.8) (Table 33). With the comparison at matched steps in § 6.1 (Table 21), this shows that low rank in these matrices is acquired at the transition, not by training to zero loss under weight decay.
| Step | Group | ||||
| 10,000 | before own onset (17) | ||||
| after own onset (12) | |||||
| never groks (1) | 52 | 84 | 71 | 67 | |
| 12,500 | before own onset (9) | ||||
| after own onset (20) | |||||
| never groks (1) | 50 | 86 | 70 | 66 | |
| 15,000 | before own onset (8) | ||||
| after own onset (21) | |||||
| never groks (1) | 45 | 81 | 67 | 65 |
Table 21. Low rank is acquired at the transition (transformer, weight decay 1.0). Singular directions carrying 99% of at the same training step, for grokked seeds before and after their own onset and for the seed that never groks. Mean s.d., with the number of seeds in parentheses.
| Run | Train | Test | ||||||||
| wd 1.0, seed 12 | 100.0 | 8.2 | 42 | 4 | 4 | 30 | 42 | 68 | 62 | 62 |
| wd 0.1, seed 0 | 100.0 | 4.3 | 43 | 4 | 19 | 31 | 50 | 82 | 78 | 72 |
| wd 0.1, seed 1 | 100.0 | 1.8 | 43 | 4 | 18 | 31 | 51 | 81 | 78 | 73 |
| wd 0.1, seed 2 | 100.0 | 3.0 | 44 | 4 | 18 | 31 | 52 | 84 | 79 | 73 |
| wd 0.3, seed 0 | 100.0 | 13.2 | 34 | 4 | 4 | 31 | 43 | 74 | 61 | 62 |
| wd 0.3, seed 1 | 100.0 | 10.7 | 36 | 4 | 4 | 31 | 43 | 73 | 61 | 62 |
| wd 0.3, seed 2 | 100.0 | 9.1 | 37 | 4 | 4 | 30 | 43 | 77 | 65 | 65 |
Table 33. Non-grokked transformer runs: final accuracy (%) and (singular directions carrying 99% of ). Grokked seeds (mean): 30.4, 4.0, 4.0, 24.3, 21.3, 18.1, 11.8, 23.8.
Why the Transformer Crystallizes Further Than the MLP
We attribute this gap to the architecture. The transformer’s embedding uses about four frequencies: 4.2 0.7 of them hold 90% of its Fourier power. Each frequency requires two directions, a cosine and a sine, so this predicts about 8 directions, close to of , and (7.7–8.8, Table 31). The directions attention writes to nearly coincide with those reads from (mean cosine of the principal angles 0.923, against 0.248 for random subspaces), and equally so in runs that never grok (Appendix § 17). The MLP block therefore receives an input already confined to a few directions, and the directions its outputs do not need are removed during the transition, together with the memorizing components (§ 7). The MLP’s first layer instead reads both one-hot operands directly, so each frequency costs it four directions, and it spreads its weights over most of the 48 frequencies (§ 5). No MLP we trained crystallizes during the transition.
Table 31: Architecture comparison (final checkpoints, with 15 MLP seeds, 5 cross-entropy MLP seeds per learning rate and 14 grokked transformer seeds).
| Property | MLP (MSE) | MLP (CE) | Transformer |
| Loss, weight decay, training steps | MSE, , | CE, , | CE, 1.0, |
| of first-layer weights (of 128) | 48–51 | ||
| of first-layer weights (of 128) | 70–81 | ||
| When falls | never | late, after | during the transition |
| Frequencies holding 90% of first-layer power | () | – | () |
Table 31.
Other differences between the models, i.e. loss and the weight decay do not account for the gap. The loss explains only why the cross-entropy MLP ends with fewer representative directions than the mean-squared-error MLP. A one-hot target has equal power at all 97 Fourier components, so for a network that matches its targets, omitting one frequency changes the outputs by , more than allows. CE constrains only the ordering of the logits, and the CE MLP ends at –81, against 122–127 with MSE. Raising its weight decay to the transformer’s per-step shrinkage of speeds up grokking, from step 10,100 to 2,300, but does not make it crystallize: is about 120 just after the onset and ends at 118.4 instead of 69.6, and stronger decay prevents learning (Table 7). Only the CE MLP’s falls late, and its Stage-4 fall in effective rank is four times as large (37.9% against 9.7%, Table 6), which suggested that the late fall is Stage 4 itself (§ 4). Consistently, at every weight decay ends close to the effective rank of (69.6 against 71.1, 100.4 against 98.7, 118.4 against 115.7). The cross-entropy MLP’s late narrowing is thus a different process from the transformer’s crystallization.
| Variant | Grokked | Final ER | Stage-4 drop (%) | Fourier IPR | |
| Main | 5/5 | 12,700 ± 447 | 104.9 ± 1.6 | 9.7 ± 1.5 | 0.457 ± 0.009 |
| wd 0 | 5/5 | 12,700 ± 447 | 98.7 ± 2.2 | 14.9 ± 1.9 | 0.452 ± 0.010 |
| wd | 5/5 | 12,700 ± 447 | 99.3 ± 2.1 | 14.4 ± 1.9 | 0.453 ± 0.008 |
| wd 0.05 | 5/5 | 12,800 ± 274 | 113.7 ± 0.8 | 2.2 ± 0.4 | 0.466 ± 0.006 |
| wd 0.1 | 5/5 | 13,400 ± 418 | 116.7 ± 1.7 | 0.5 ± 0.4 | 0.476 ± 0.004 |
| CE, lr | 1/5 | 94,500 () | 114.1 ± 0.3 | – | 0.200 ± 0.008 |
| CE, lr | 5/5 | 27,500 ± 2,372 | 102.4 ± 0.6 | 10.9 ± 0.5 | 0.292 ± 0.009 |
| CE, lr | 5/5 | 10,100 ± 962 | 71.1 ± 2.2 | 37.9 ± 1.9 | 0.324 ± 0.011 |
| ReLU | 5/5 | 44,900 ± 4,615 | 80.8 ± 0.8 | 14.0 ± 1.3 | 0.332 ± 0.028 |
| GELU | 5/5 | 33,400 ± 1,140 | 79.2 ± 1.8 | 16.8 ± 1.3 | 0.474 ± 0.010 |
Table 6. MLP training variants (5 seeds each, mean ± s.d.). Main: mean-squared error, quadratic activation, weight decay (wd) . CE: cross-entropy. Grokked: test accuracy within steps. Stage-4 drop: relative decrease of effective rank from its peak after to its final value. Fourier IPR: § 6.5.
| lr· | Grokked | after | , end | , end | ER(), end | ||
| 0.01 | 5/5 | 10,100 ± 962 | 119.8 ± 0.4 | 69.6 ± 2.2 | 48.4 ± 2.5 | 71.1 ± 2.2 | |
| 0.03 | 5/5 | 4,500 ± 354 | 120.2 ± 0.8 | 100.4 ± 2.3 | 61.6 ± 3.2 | 98.7 ± 2.6 | |
| 0.1 | 5/5 | 2,300 ± 274 | 120.2 ± 0.8 | 118.4 ± 1.7 | 69.6 ± 2.6 | 115.7 ± 1.0 | |
| 0.3 | 0/5 | – | – | – | – | 1.0 ± 0.0 | |
| 1.0 | 0/5 | – | – | – | – | 1.0 ± 0.0 |
Table 7. Weight decay in the cross-entropy MLP (learning rate , 5 seeds each, mean ± s.d.). and of (of 128) on test inputs with centered logits, at the first checkpoint after and at step .
Discussion
What our measurements imply for rank analyses. Our results suggest four practices for measuring how many directions a network uses. First, count directions with a criterion that is independent of the margin, such as , and compare the count with that of untrained or memorizing networks: in our MLP, reported a narrowing that the outputs did not undergo. Second, report the training step with every count: the MLP’s grows from 65 to 89 after generalization, so it is not a fixed property of the trained model. Third, compare any intervention that imposes low rank with a control of the same parameterization: against the unfactorized MLP, rank caps would wrongly appear to accelerate grokking. Fourth, read weight-spectrum statistics such as IPR or effective rank as signals of grokking only next to negative controls, since they also move in networks that never grok. In the transformer, a few leading singular directions of each matrix carry the solution and truncating to them keeps the test accuracy, while random truncations of equal rank do not (Table 15, subject to the assumptions in Appendix § 15.§ 3). In the MLP, one seed’s hidden representation transfers to another after an orthogonal alignment fitted on its few thousand training inputs, although their weight subspaces overlap no more than random ones (§ 5).
Limitations. Our tasks are all modular arithmetic on cyclic groups, so a non-abelian group [5] would test the timing results more broadly. Whether language or vision models have a comparably clean sufficient subspace is a natural next question. Kaushik et al. [ [15] study low-dimensional weight subspaces in such models. Within our setting, the two architectures differ in more than their structure: they are also trained with different losses and for different lengths. We varied weight decay separately and found that it does not explain the difference in crystallization (Appendix § 13.§ 3), but the cross-entropy MLPs need larger learning rates than the mean-squared-error ones to grok within steps (one of five seeds groks at ). Finally, the Granger tests capture only temporal order, so our causal conclusions rest on the rank interventions.
Conclusion
We asked whether rank compression causes grokking (Q1) and whether the learned solution comes to lie in a few directions of each weight matrix (Q2). For Q1, compression follows the grokking onset rather than memorization: it tracks the onset across a threefold range of onset times while the training set is memorized within 250 steps, it does not start before test accuracy rises, and forcing low rank from initialization does not bring grokking forward. Compression is therefore a signature of grokking rather than its cause. For Q2, the answer depends on how directions are counted. Judged by the outputs, the transformer crystallizes during the transition, to 8–13 directions per matrix, at the same time as its memorizing components are removed. The MLP trained with mean-squared error does not: its narrows only because its margins grow, and its late compression tracks the optimizer’s effective learning rate. Counting directions by the outputs rather than by accuracy, and comparing any imposed low rank with a control of the same parameterization, is what makes these distinctions visible.
AI Use Statement
We used generative AI tools to retrieve and discover relevant literature and to draft and polish portions of the manuscript where appropriate. Any AI outputs were heavily edited by the authors. We did not use generative AI to formulate or refine the research hypotheses, develop the methodology, design the experiments, or interpret the results. The authors take full responsibility for the final content of this work, including its text, claims, code and data.
Further Related Work
Grokking beyond modular arithmetic. Grokking also occurs for adversarial robustness [14], in multi-hop reasoning [1] and in Recursive Feature Machines [23]. Walker et al. [2025] remove the delay with spherical constraints on the residual stream. Our factorized control shows a related effect: factorizing at full rank shortens the delay seven- to eight-fold in the MLP (§ 6.4).
Theories of the late phase. Lyu et al. [2024] prove that homogeneous networks with large initialization and small weight decay move from kernel-like dynamics early in training to a max-margin solution late in training, and Mohamadi et al. [2023] show that a margin-maximization bias explains grokking in two-layer quadratic networks on modular arithmetic, with sample complexity against for kernel methods. Their setting, an exponential-type loss with weight decay, differs from our mean-squared-error MLP, which groks at the same step without weight decay (Table 6). Morwani et al. [2024] show that Fourier features and irreducible representations emerge from implicit margin maximization, consistent with the Fourier structure our networks learn without any spectral penalty.
Weight geometry and representations. Li et al. [2018] train networks in low-dimensional random subspaces of parameter space, and Yunis et al. [2024] relate rank minimization, weight alignment and weight decay beyond grokking. Raghu et al. [2017] compare representations across networks with SVCCA, and our cross-seed results extend this comparison to grokking: the representations differ in raw coordinates and agree up to rotation. Sparse autoencoders [4, 6] decompose representations into features, a basis complementary to the singular directions used here.
Experimental Details and Training Variants
Data, Training and Metrics
Data and training. The MLP uses a 35% train split (3,293 of the 9,409 pairs) and logs metrics every 500 steps. The transformer uses a 30% split (2,822 pairs) and logs every 250 steps. Its vocabulary comprises tokens (the residues and the equals sign), so (the residue rows), , , , per head and (heads stacked). The transformer’s higher weight decay (1.0 against ) follows Nanda et al. [2023]. MLP variants use seeds 0–4, rank-constrained MLPs seeds 0–2, and transformer rank caps seeds 0–4. All experiments were run on NVIDIA RTX A5000 GPUs.
Training objectives. The MLP minimizes the mean-squared error to one-hot targets, with , and the transformer the cross-entropy over its output logits , .
Metrics. Stable rank is . Hoyer sparsity is over the entries of , and L0 sparsity the fraction of entries below in magnitude. The spectral IPR rises as spectral mass concentrates in a few directions. The gradient norm is the norm of the full gradient. For the transformer’s Fourier IPR, each of the 128 embedding dimensions is transformed over the 97 tokens.

Figure 3. Training dynamics of the MLP (seed 0): loss, weight norms, effective rank, accuracy, gradient norm, stable rank, and Hoyer sparsity of , and Fourier IPR, against training step.
Training Variants of the MLP
Table 6 and Figure 4 report the variants summarized in § 4.

Figure 4. Stage 4 across training variants (MLP, 5 seeds each, mean curves of test accuracy, effective rank of and, in (b, c), Fourier IPR).
CE at lr reaches final test accuracy, and its single grokked seed has a Stage-4 drop of 0.1%.
Weight Decay in the Cross-Entropy MLP
Table 7 trains the cross-entropy MLP at learning rate with five weight decays (5 seeds each, steps), including the runs of Table 6 at . With AdamW the weights shrink by lr · per step, so matches the transformer’s per-step decay of . and are computed for on test inputs with centered logits, at the first saved checkpoint after and at the end of training. At the weights decay to near zero, collapses to effective rank 1, and test accuracy stays at chance (1.0 ± 0.1%).
Role of the Nonlinearity
In the MLP, the quadratic activation turns sums of operand features into the products from which is read out. ReLU and GELU do not do this exactly, yet MLPs with these activations also grok, at and against , with periodic first-layer weights (Fourier IPR and , against ) and larger Stage-4 decreases in effective rank (Table 6). In the transformer, the ReLU of the MLP sublayer acts on inputs that attention has written into the subspace the MLP reads (alignment 0.923, Table 34), and the embedding concentrates 95% of its Fourier power on five frequencies, so the MLP sublayer operates on an already low-dimensional, low-dimensional input.
| Subspace pair | Grokked wd 1.0 () | Not grokked wd 0.1 () | Not grokked wd 0.3 () | Not grokked wd 1.0 () |
| Attention write () vs. MLP read () | ||||
| MLP read () vs. MLP write () | ||||
| MLP write () vs. unembedding read () | – | – | – | |
| MLP read vs. key frequencies of | – | – | – | |
| MLP write vs. key frequencies of | – | – | – | |
| Random baseline |
Table 34. Residual-stream alignment (mean cosine of principal angles between 11-dimensional subspaces). Not grokked: three seeds each at weight decay 0.1 and 0.3, and the one non-grokked seed at weight decay 1.0.
Stage 4: Weight Growth and the Optimizer
Why the Sufficient Subspace Widens
Table 8 and Figure 5 decompose the widening of the sufficient subspace for the main runs, and Table 9 repeats the measurement across weight decays. The top 65 directions of grow by half while the remaining 63 keep their size. The output change caused by removing the remaining directions, dominated by the cross term , roughly doubles while the margin grows by 30%, so truncation to the top 65 directions loses accuracy and widens. Stronger weight decay limits the growth of the top directions and, with it, both the fall in effective rank and the widening.
| Step | 20,000 | 30,000 | 50,000 | 100,000 |
| 64.9 ± 1.7 | 69.3 ± 1.7 | 81.5 ± 2.4 | 89.1 ± 2.3 | |
| 114.4 ± 1.1 | 110.9 ± 1.5 | 107.0 ± 1.9 | 104.0 ± 2.1 | |
| 731 ± 7 | 813 ± 14 | 937 ± 26 | 1059 ± 35 | |
| 531 ± 5 | 597 ± 11 | 685 ± 21 | 763 ± 27 | |
| 111.9 ± 3.4 | 132.2 ± 3.0 | 171.1 ± 10.4 | 217.8 ± 21.6 | |
| Size of top 65 | 659 ± 10 | 748 ± 18 | 878 ± 29 | 1002 ± 37 |
| Size of remaining 63 | 316 ± 10 | 318 ± 10 | 327 ± 13 | 344 ± 18 |
| Test acc., top-65 truncation (%) | 99.1 ± 0.4 | 97.7 ± 0.9 | 84.5 ± 5.5 | 57.7 ± 9.5 |
| Median correct-class output | 0.604 ± 0.017 | 0.696 ± 0.010 | 0.722 ± 0.008 | 0.732 ± 0.007 |
| Median margin | 0.470 ± 0.019 | 0.570 ± 0.012 | 0.599 ± 0.009 | 0.611 ± 0.008 |
| Output change, | 0.611 ± 0.015 | 0.699 ± 0.014 | 0.860 ± 0.063 | 1.153 ± 0.128 |
| Output change, | 0.254 ± 0.019 | 0.336 ± 0.022 | 0.445 ± 0.029 | 0.533 ± 0.047 |
Table 8. Why the sufficient subspace widens in Stage 4 (MLP, 15 seeds). is the median at step 20,000. Size: over the top 65 or the remaining 63 singular directions of . Output change: median change of the 97 outputs when the directions beyond the top 65 are removed, split into the two terms of .
| Weight decay | Size of remaining 63 | ||||
| 0 | 65.8 → 95.0 | 114.6 → 98.7 | 731 → 1,252 | 114 → 251 | 320 → 358 |
| 65.4 → 93.8 | 114.7 → 99.3 | 731 → 1,220 | 113 → 249 | 320 → 354 | |
| (main) | 65.2 → 88.6 | 115.0 → 104.9 | 727 → 1,043 | 111 → 205 | 320 → 347 |
| 64.0 → 73.0 | 115.7 → 113.7 | 710 → 825 | 101 → 151 | 321 → 342 | |
| 62.4 → 63.0 | 116.7 → 116.7 | 678 → 748 | 92 → 108 | 311 → 346 |
Table 9. Weight decay controls the widening (5 seeds each, step 20,000 → step , means). All runs reach 100% test accuracy.

Figure 5. Stage-4 widening. Left: size of the top 65 and the remaining 63 singular directions of over training (15 seeds). Right: at step 20,000 and at the end of training for each weight decay (5 seeds).
Gradient and Adam State
We continue each main seed for 1,000 AdamW steps from the checkpoints at steps 20,000, 50,000 and to rebuild the optimizer state, and then measure the full-batch gradient and Adam’s bias-corrected moments (Table 10). The gradient is small and becomes increasingly radial: its cosine with goes from to , so points outward and the loss would decrease if the weights grew. This radial component equals the Euler expression for this 3-homogeneous model to all printed digits. Almost every coordinate has , so the update is close to : AdamW acts as gradient descent with momentum and learning rate , not as sign descent (the mean of is 0.07–0.21, against 1 for sign descent). The radial part of this update exceeds the decay pull at step 20,000 and approaches it by the end, as the norm approaches an equilibrium. The gradient also concentrates in the top 65 directions of , the directions that grow.
| Step 20,000 | Step 50,000 | Step | |
| (= Euler expression) | |||
| Share of coordinates with | |||
| Median | |||
| Mean (1 = sign descent) | |||
| Radial push of the Adam update | |||
| Decay pull | |||
| Share of in the top 65 directions |
Table 10. Gradient and Adam state during Stage 4 (MLP, 15 seeds, mean s.d.). Radial push of the update and decay pull are per unit learning rate.
Optimizer Swap
Table 11 continues main seeds 0–2 from step 20,000 to with each optimizer. The weight decay is set so that the per-step shrinkage matches AdamW’s ( for gradient descent, for momentum 0.9, whose steady-state step is ). Runs with the same effective learning rate agree closely: SGD with momentum at 10 reproduces gradient descent at 100 to the printed digits, and AdamW with reproduces gradient descent at . Stage 4, the fall of effective rank with the growth of and the widening of , appears at effective learning rates of and above and strengthens as decreases. Test accuracy stays at 100% in every run.
Table 11: Continuations from step to (main seeds 0–2, means). The effective learning rate is for gradient descent, for AdamW and for SGD with momentum.
| Optimizer | Effective lr | ER | ||||
| At step 20,000 | – | 726 | 529 | 111 | 115.1 | 65 |
| AdamW, | 1,793 | 718 | 346 | 95.7 | 99 | |
| AdamW, | 1,582 | 874 | 318 | 94.7 | 98 | |
| AdamW, (main) | 1,040 | 748 | 203 | 104.8 | 89 | |
| AdamW, | 775 | 551 | 120 | 114.8 | 70 | |
| AdamW, | 658 | 469 | 85 | 117.8 | 65 | |
| AdamW, | 331 | 241 | 51 | 115.3 | 65 | |
| AdamW, , no weight decay | 1,257 | 917 | 261 | 97.8 | 94 | |
| SGD, (same decay) | 326 | 238 | 50 | 115.1 | 65 | |
| Gradient descent | 384 | 277 | 56 | 116.7 | 65 | |
| Gradient descent | 662 | 472 | 85 | 117.8 | 65 | |
| Gradient descent | 783 | 557 | 126 | 114.6 | 72 | |
| Gradient descent | 833 | 913 | 148 | 108.3 | 87 | |
| Gradient descent | diverges | |||||
| SGD + momentum, | 384 | 277 | 56 | 116.7 | 65 | |
| SGD + momentum, | 662 | 472 | 85 | 117.8 | 65 | |
| SGD + momentum, | 783 | 557 | 126 | 114.6 | 72 | |
| SGD + momentum, | 1,093 | 775 | 216 | 103.8 | 91 | |
| SGD + momentum, | 881 | 1,577 | 174 | 101.1 | 93 |
Table 11.
The Sufficient Subspace, Output Counts and Cross-Seed Equivalence
The Sufficient Subspace over Training
Table 12 reports of at four thresholds at the end of training, which ranges from 80 to 93 directions for thresholds from 90% to 99.9%. Table 13 and Figure 6 show its evolution over training. Before step 15,000, is relative to each checkpoint’s own test accuracy, which is below 100%. The number of frequencies holding 90% of the Fourier power of falls from 38.2 to 33.7 during Stage 4.

(a) (a) Test accuracy of truncated to its top directions (step ).

(b) (b) and (left axis) and effective rank (right axis) over training.
Figure 6. The sufficient subspace of the MLP (15 seeds). (a) Seed 0 and mean ± s.d. (b) and narrow sharply after the onset and widen again during Stage 4, while effective rank falls. does not narrow (Table 3).
| Threshold (fraction of clean accuracy) | (mean s.d.) |
| 90% | 80.5 3.7 |
| 95% | 84.5 3.2 |
| 99% | 89.1 2.3 |
| 99.9% | 93.1 2.7 |
Table 12. of at four thresholds (step , 15 seeds).
Temporal Stability
We compute the mean cosine of the principal angles between the top-89 left and right singular subspaces of at step and at the final checkpoint (, the median ), across all 15 seeds (Table 14). At initialization the overlap is at the random level. At it is only modestly above it (0.837 against 0.773 in the hidden space), and it keeps rising for tens of thousands of steps. The final sufficient subspace is therefore assembled mostly during Stages 3 and 4, not at the accuracy transition.
| Step | Hidden space () | Input space |
| 0 | 0.775 0.003 | 0.608 0.003 |
| 5,000 | 0.782 0.003 | 0.623 0.003 |
| 10,000 | 0.807 0.005 | 0.677 0.007 |
| 12,500 | 0.837 0.007 | 0.723 0.010 |
| 15,000 | 0.875 0.010 | 0.773 0.012 |
| 20,000 | 0.925 0.013 | 0.843 0.013 |
| 30,000 | 0.961 0.013 | 0.910 0.011 |
| 50,000 | 0.983 0.012 | 0.967 0.013 |
| Random | 0.773 0.003 | 0.606 0.003 |
Table 14. Overlap of the top-89 singular subspaces of at step with those at step (15 seeds). Hidden space: left singular vectors (the sufficient subspace ). Input space: right singular vectors.
Truncation Controls and Complement Interventions
*Top- and bottom- share about 50 directions when . ‡Keeping only the bottom directions of or of effectively removes the MLP’s contribution, hence the identical values.
Complement interventions. We intervene on the MLP’s pre-activations , with the sufficient subspace of each seed at step . Keeping only the -component gives 99.26 0.22% test accuracy and keeping only the -component 0.15 0.14%. Adding Gaussian noise of norm (per input) inside or inside gives 99.97 0.05% and 99.95 0.07% for , 92.42 4.22% and 70.97 9.69% for , 10.11 2.06% and 7.43 1.32% for , and 1.54 0.10% and 1.61 0.09% for . As a check of the pipeline, patching the clean hidden representation into a run on noise-corrupted inputs restores the clean output (100.00% in the MLP at corruption level 0.3, against 1.59% unpatched, and 99.92% in the transformer, against 18.52%), as it must in a network with one hidden layer.
Assumptions of the SVD probe. Equating high-variance singular directions with functional relevance assumes that those directions carry the computation rather than optimizer noise, representation drift or dense isotropic features. It also assumes linear, orthogonal structure, which can obscure sparse features stored in non-orthogonal superposition or on nonlinear manifolds. The controls of Table 15 test the first assumption in our setting: the high-variance singular directions carry the computation, while random subspaces of equal rank do not. In the MLP the distinction is weaker, because the sufficient subspace spans most of the directions.
Output Counts under Other Criteria
Table 16 gives the directions of needed under four output-based criteria: outputs within 10% () and 5%, per-input centered outputs within 10%, and per-input margins within 10%. Centering changes nothing for the MSE models. The margin criterion lies between accuracy and the full outputs: it narrows from 124 to 113 directions between steps 11,000 and 20,000 and widens again to 121, much less than (119 to 65 to 89). For the cross-entropy MLPs, every output-based criterion narrows only after the onset.
Table 16: Output counts in the MLP (directions of out of 128, mean over seeds, test inputs unless noted). and Out05: outputs within 10% and 5% (centered for CE). Cen10: per-input centered outputs within 10%. Marg10: per-input margins within 10%.
| Model | Checkpoint | Out05 | Cen10 | Marg10 | Median margin | ||
| MSE, main | step 0 | – | 116.4 | 123.8 | 116.3 | 102.2 | −0.004 |
| step 11,000 | 119.3 | 122.4 | 126.6 | 122.7 | 123.8 | 0.025 | |
| step 20,000 | 64.9 | 122.3 | 125.7 | 122.4 | 113.1 | 0.470 | |
| end | 89.1 | 127.3 | 127.9 | 127.3 | 121.4 | 0.611 | |
| end, training inputs | 88.0 | 127.3 | 127.9 | 127.5 | 121.1 | 0.649 | |
| MSE, random labels | end, training inputs | – | 123.0 | 127.0 | 123.2 | 116.6 | 0.110 |
| end, test inputs | – | 121.4 | 126.0 | 121.6 | 108.0 | −0.229 | |
| MSE, mod 97 | last before | 115.9 | 122.1 | 126.3 | 122.3 | 117.5 | 0.052 |
| first after | 97.5 | 122.5 | 126.3 | 122.5 | 112.7 | 0.155 | |
| step 20,000 | 64.7 | 121.7 | 125.1 | 121.7 | 112.4 | 0.481 | |
| end | 89.8 | 126.2 | 126.9 | 126.2 | 120.7 | 0.653 | |
| MSE, mod 113 | last before | 117.7 | 123.9 | 127.1 | 124.2 | 123.1 | 0.048 |
| first after | 79.4 | 124.3 | 127.1 | 124.4 | 115.3 | 0.232 | |
| step 20,000 | 70.9 | 124.3 | 126.9 | 124.3 | 114.1 | 0.275 | |
| end | 85.5 | 127.4 | 127.9 | 127.4 | 121.5 | 0.501 | |
| CE, lr | last before | 112.8 | 121.4 | 125.8 | 121.4 | 120.8 | 3.78 |
| first after | 91.8 | 119.0 | 125.4 | 119.0 | 112.6 | 5.86 | |
| end | 50.6 | 81.0 | 99.8 | 81.0 | 68.0 | 9.43 | |
| CE, lr | last before | 104.6 | 120.2 | 125.8 | 120.2 | 117.8 | 4.22 |
| first after | 98.6 | 119.8 | 125.8 | 119.8 | 115.6 | 4.86 | |
| end | 48.4 | 69.6 | 71.8 | 69.6 | 65.8 | 10.20 |
Table 16.
Cross-Seed Equivalence
For a source model and a receiving model , we compute ’s post-activations on ’s test inputs and decode them with ’s readout , either directly or after the orthogonal map fitted on ’s training inputs [34], and measure accuracy against the true labels mod 97 (Table 17). Rows with random-label models test specificity in one direction only: a readout trained on random labels cannot decode the true labels from any representation, so only transplants into a grokked readout are informative. The step-10,000 checkpoint of each seed (at least 99% training and test accuracy) already partly generalizes, so its rows do not isolate a memorizing representation.
Subspace overlap. Figure 7 shows the Fourier power spectrum of for six seeds. Overlaps are mean cosines of the principal angles between the top-60 weight subspaces of (in the 194-dimensional input space), and between 10-dimensional Fourier subspaces built from each seed’s five strongest frequencies (a cosine and a sine for each, with and treated as one frequency), over all 105 seed pairs, with random baselines simulated for the same dimensions. A few individual directions are shared: per seed pair, of the top-60 directions have a counterpart with cosine above 0.9, against 1.38 expected for random subspaces.

Figure 7. Fourier power spectra of for six seeds (share of power per frequency, and combined).
Phase structure and robustness. For each of the 128 hidden units, the phases of its dominant frequency on operands and are spread over the whole circle, and their differences are close to uniform (Figure 8a). There are no discrete phase families, consistent with the analytical solutions of Gromov [ [11], in which each unit’s phases are free parameters. Adding Gaussian noise with standard deviation times the standard deviation of each weight matrix to and lowers test accuracy to 71% at , 14% at and 5% at (Figure 8b, seed 0, step ).

(a) Phase alignment.

(b) Weight perturbation robustness.
Figure 8. Phase structure and robustness of the MLP solution (seed 0).
Timing, Replication and Rank Interventions
Timing of Compression
Table 18 summarizes the timing of compression in both architectures.
Steepest-decrease measure. The steepest decrease used in § 6 is the step of the most negative central-difference derivative of the effective rank after a centered moving average over five logged steps (np.convolve in same mode, then np.gradient). Each smoothed value belongs to the logged step at its center, so the smoothing adds no delay. The zero-padded average distorts the last two logged steps and the central difference a third, so we exclude the last four. On synthetic signals the measure places a ramp exactly and a sharp step two logging intervals early (Figure 10), so any bias shortens the measured lag. It falls after in 13 of 14 transformer seeds of 0–14 (the exception: 250 steps before ) and in 14 of 15 held-out seeds (the fifteenth at ). Figure 9 shows the effective rank of and the test accuracy of the 14 grokked transformer seeds aligned at . Counting the decay of the initialization, of the decrease of happens after (: 22%, : 27%, : 37%, : 38%, : 62%).

Figure 10. Synthetic check of the steepest-drop measure (logging every 250 steps). Left: a sharp step at step 12,000 is located two logging intervals early, because the derivative of the centered average is tied over several steps and the earliest is taken. Right: a 1,000-step ramp centered on step 12,000 is located exactly.
| MLP (15 seeds) | Transformer (29 grokked) | |
| Memorization step | 9,133 297 | 250 in all seeds |
| Grokking onset | 12,633 297 | 11,914 4,116 |
| Start of compression | : 6,367 3,248 (15/15 ) | : (28/29 after ) |
| Steepest ER decrease | 9,667 3,320 (15/15 ) | 905 506 (27/29 ) |
| Share of ER decrease after | 0.99 0.02 | 0.63 0.09 |
Table 18. Timing of compression (mean s.d., in steps). is the first step at which ER falls below 99% of its running maximum. and are the start and the steepest point of the effective-rank drop of after the initialization decay. is the first step with test accuracy . From the plateau after the initialization decay. Including the decay, 0.32 0.05.
Lead and lag in the MLP. Figure 11 shows that the compression onset follows the grokking onset in every MLP seed, and that the cross-correlation of the derivatives of effective rank and test accuracy has its trough at +12,500 steps.
Figure 11: Compression lags grokking in every MLP seed.
Granger tests. Granger causality [10] tests whether the past of one time series improves the prediction of another beyond that series’ own past. It measures temporal predictive asymmetry, not interventional causation. We use the first differences of effective rank and test accuracy, logged every 500 steps, with lag order chosen by the Akaike information criterion. Augmented Dickey–Fuller tests find these differences stationary in only 8 and 13 of 15 seeds, so we repeat the test after differencing further until the series are stationary in every seed. Changes in test accuracy predict changes in effective rank in every seed in both versions, and the reverse holds in at most 6 of 15 (Table 19). Since test accuracy is flat at 100% for most of training, these tests largely restate the order of the two pulses.
| Series | Null hypothesis | Med. | Rejected () | ||
| First differences | Compression Grokking | 2.84 | 0.133 | 0.059 | 6/15 |
| Grokking Compression | 24.77 | 0.003 | 15/15 | ||
| Stationary (ADF) | Compression Grokking | 2.44 | 0.257 | 0.126 | 4/15 |
| Grokking Compression | 19.77 | 0.003 | 15/15 |
Table 19. Granger tests (temporal predictive asymmetry) across 15 MLP seeds.
Networks That Memorize Without Grokking
Transformers. Table 20 compares the second effective-rank drop of in the 29 grokked transformer seeds with the seven runs that memorize the training set without generalizing. All runs memorize the training set by step 250. In grokked seeds, the effective rank of falls by between step 4,000 and the end of training, with a peak rate of units per 1,000 steps. In non-grokked runs it falls by 15–23%, at 2.1–4.6 units per 1,000 steps.
| Run | Final test acc. (%) | Peak rate (per 1,000 steps) | Relative change, step 4,000 end |
| Grokked (29 seeds) | |||
| wd 1.0, seed 12 | 8.2 | ||
| wd 0.1, seeds 0–2 | 1.8–4.3 | to | to |
| wd 0.3, seeds 0–2 | 9.1–13.2 | to | to |
Table 20. Effective rank of after the initialization decay (transformer). Peak rate: steepest smoothed decrease after step 4,000.
Start of the drop. The start of the drop is the first logged step of the contiguous stretch before in which the smoothed decrease is at least 20% of its peak rate. It falls between the steps at which test accuracy first exceeds 10% and 50% in most seeds: steps (28 of 29 non-negative, median 1,750) and (4 of 29 non-negative). Three seeds have from an early fluctuation of test accuracy, which inflates the mean of the first difference. Test accuracy at is , and 37% of the post-plateau decrease happens before .
Matched steps. Table 21 gives the variance counts at steps 10,000–15,000. At step 5,000, before any seed has generalized, the 29 grokked seeds and the non-grokked seed at the same weight decay have nearly the same counts ( against 75 for , against 70 for ), while the runs at weight decay 0.1 and 0.3 have higher counts (113 and 92 for ), a weight-decay effect. At step 20,000, 28 seeds have generalized and their counts are , , and for , , and , against 48, 78, 68 and 64 for the non-grokked seed.
MLPs trained on random labels. Figure 12 shows the effective rank and training accuracy of the five random-label MLPs next to the grokked ones. The random-label networks reach 90% training accuracy at step , end at , and their effective rank falls from to (), against to () in grokked networks.

Figure 12. MLPs that memorize without generalizing barely compress. Effective rank of (left) and training accuracy (right) for 15 grokked MLPs (blue) and 5 random-label MLPs (red).
Replication on Held-Out Seeds and Tasks
Data. H1 uses the transformer exactly as in the main experiments (weight decay 1.0, 25,000 steps) with seeds 15–29, which were trained only after the timing measure had been defined on seeds 0–14. H2–H5 use the MLP exactly as in the main runs, on and , with seeds 0–14 per task. H6 uses the same MLP with , , seeds 0–2 per task and rank. The analysis scripts are in the code repository.
Definitions. is the first logged step with test accuracy . A run counts as grokked if its final test accuracy is , and only grokked runs enter the tests. is the logged step with the most negative derivative of the five-point moving average of the effective rank, over logged steps from step 4,000 (transformer, ) or 1,000 (MLP, ) up to four logging intervals before the end. is the first logged step at which the effective rank of falls below 99% of its running maximum. is the smallest whose top- truncation of keeps of the test accuracy of checkpoint . is the first saved checkpoint , the first saved checkpoint that is not the final one, and the final checkpoint.
Tests. All tests are one-sided at and run separately per task, and every result is reported. H1: (one-sample -test, with the number of seeds with and an exact sign test). H2: and H3: (one-sample -tests). H4: and H5: (paired -tests). H6 (descriptive): the factorized full-rank model groks earlier than the unfactorized one, and increases as decreases from 128 to 64 to 32.
Results. Every test is significant at (Table 22, Table 24 and Table 23, and Table 27 for H6). With a Bonferroni correction over the nine -tests, H4 on mod 113 is not. For H1, the 95% confidence interval of is [560, 1,140] steps (median 750), and the exact sign test gives . Twice the number of frequencies holding 99% of the Fourier power of is for mod 97 (Fourier transform over the nonzero residues ordered by the discrete logarithm base 5) against , but for mod 113 against , so the frequency count does not predict in general (§ 8). Multiplication loses a little test accuracy late in training, from 100.0% at step 30,000 to at the end. , which H4 and H5 do not use, does not narrow on either task: it is 121.7–127.4 at every checkpoint (Table 16).
| mod 97 | mod 113 | |
| Grokked | 15/15 | 15/15 |
| 12,633 297 | 15,567 417 | |
| 18,933 1,498 | 22,267 4,629 | |
| H2: | 6,300 1,461 | 6,700 4,519 |
| 15/15, , | 15/15, , | |
| H3: | 9,800 2,274 | 11,867 6,741 |
| 15/15, , | 15/15, , | |
| H4: | 32.9 13.7 | 10.8 16.8 |
| 15/15, , | 12/15, , | |
| H5: | 25.1 2.9 | 16.9 3.1 |
| 15/15, , | 15/15, , | |
| Share of the ER() decrease after | ||
| at the end | ||
| Top / bottom / random-subspace truncation (%) | 99.10 / 36.84 / 20.68 | 99.08 / 27.93 / 33.03 |
Table 23. H2–H5 on the two further MLP tasks (15 seeds each, mean s.d.). Below each test, the number of seeds with a positive difference, the statistic and the one-sided .
| Seed | Seed | Seed | ||||||
| 15 | 7,000 | +500 | 20 | 17,500 | +500 | 25 | 17,750 | +250 |
| 16 | 12,500 | +1,000 | 21 | 8,000 | 0 | 26 | 8,750 | +500 |
| 17 | 7,500 | +750 | 22 | 13,000 | +1,000 | 27 | 15,750 | +750 |
| 18 | 17,000 | +1,000 | 23 | 16,500 | +1,750 | 28 | 8,500 | +750 |
| 19 | 7,500 | +750 | 24 | 10,250 | +2,000 | 29 | 8,250 | +1,250 |
Table 24. H1 per seed: held-out transformer seeds, and (steps).
Table 27: Factorization runs on the two further tasks (H6) (3 seeds per arm, unfactorized: main seeds 0–2).
| Task | Model | Grokked | Final test acc. (%) | Final ER() | |
| mod 97 | Unfactorized | 3/3 | 12,833 289 | 99.7 0.4 | 102.3 0.7 |
| , | 3/3 | 2,000 0$ | 99.3 0.7 | 85.6 1.4 | |||
| , | 3/3 | 2,667 289 | 100.0 0.0 | 47.9 0.2 | |
| , | 3/3 | 4,667 289 | 99.0 0.5 | 30.1 0.2 | |
| mod 113 | Unfactorized | 3/3 | 15,333 289 | 100.0 0.0 | 109.5 1.8 |
| , | 3/3 | 2,000 0$ | 100.0 0.0 | 89.9 2.1 | |||
| , | 3/3 | 3,000 0 | 100.0 0.0 | 47.0 5.1 | |
| , | 3/3 | 4,500 0 | 99.2 0.9 | 29.6 0.5 |
Table 27.
. The largest is 0.013 (H4, mod 113). H1–H3 use one-sample -tests, H4 and H5 paired -tests, and H6 is descriptive, with 3 seeds per arm. is the first checkpoint at or after , the first checkpoint at least 5,000 steps later, and the final checkpoint.
Truncations keep directions. Chance accuracy: 1.03% () and 0.88% (). For mod 113, is step 20,000 in 12 seeds. The three seeds with at step 15,000 narrow from 106, 111 and 115 directions to 66, 71 and 67.
Forcing Low Rank
Table 25 and Figure 13 report the MLP rank interventions summarized in § 6.4. Table 26 shows that the factorization alone biases toward low rank, and Table 27 repeats the factorization on the two further tasks. Table 28 and Figure 14 repeat the factorized control and two rank caps with logging every 50 steps, on the same seeds as above. The full-rank factorization groks , and times sooner than the unfactorized model on the three tasks. Its effective rank dips early, rises to its maximum at the onset and falls afterwards.

Figure 13. (a) Rank-constrained parameterization . (b) Nuclear-norm penalty from step 0. Figure 13: Forcing low rank from initialization (mean test accuracy and effective rank of against training step).
Table 26: The factorization biases toward low rank. Effective rank of with and of the unfactorized (seeds 0–2, mod 97). The factorized model groks at step 2,000, the unfactorized one at 12,833 289.
| Step | , | Unfactorized |
| 0 | 92.7 0.3 | 114.6 0.1 |
| 500 | 79.7 1.0 | 114.5 0.1 |
| 1,000 | 97.4 0.6 | 114.5 0.1 |
| 1,500 | 105.5 0.3 | 114.6 0.2 |
| 2,000 | 105.9 0.3 | 114.7 0.2 |
| 3,000 | 102.4 2.0 | 114.9 0.2 |
| 5,000 | 94.8 3.3 | 115.1 0.1 |
| 10,000 | 89.8 2.7 | 115.9 0.1 |
| 20,000 | 88.5 2.1 | 115.1 0.4 |
Table 26.
Table 28: Factorized MLPs logged every 50 steps (3 seeds each, mean s.d.). ER: effective rank of .
| Task | Model | ER minimum before (step) | ER at | ER at step 8,000 | |
| mod 97 | unfactorized | 12,483 340 | – | 116.2 0.2 | – |
| 1,667 58 | 76.9 0.7 (333) | 106.3 0.5 | 90.7 3.1 | ||
| 2,450 100 | 49.3 0.2 (267) | 60.5 0.1 | 52.5 6.1 | ||
| 4,133 29 | 28.2 0.2 (300) | 31.5 0.1 | 31.0 0.2 | ||
| mod 97 | unfactorized | 12,567 189 | – | 115.6 0.6 | – |
| 1,750 0 | 73.8 0.6 (300) | 106.0 1.3 | 87.3 1.5 | ||
| mod 113 | unfactorized | 15,233 293 | – | 117.6 0.2 | – |
| 1,817 29 | 78.6 0.0 (333) | 108.5 0.5 | 93.4 2.3 |
Table 28.

Figure 14. Effective rank of factorized MLPs ( with , 64 and 32, logged every 50 steps, 3 seeds each).
Label-Free Onset Detection
Table 29 reports the onset detectors summarized in Section 6.5, and Figure 16 the Fourier IPR of one MLP seed. In the MLP, the Fourier IPR fires at or just after the end of the baseline window, so its lead is set by the window, . Spectral IPR, effective rank and stable rank of change after (early in at most 2 of 15 seeds, with median leads from to steps). For the transformer, “early” is counted out of the seeds with , the only ones that can be early. Of 63 metric, window and threshold combinations tested on the MLP, only the gradient norm (window 5,000, threshold ) detected grokking early in all seeds without false alarms. Given the number of combinations, we regard this as exploratory.

Figure 16. Fourier IPR of (blue) and test accuracy (red), MLP seed 0.
Transformer: Additional Results
Dimensionality at the End of Training
Table 30 gives the variance counts and of every analyzed matrix, and Figure 15 the singular value spectra for one seed. Averaged over seeds at fixed steps, mixes seeds before and after their onsets: falls from at step 10,000 (mean test accuracy ) to at step 20,000, and from to . Aligned at each seed’s own onset (Table 5), narrows during the rise in test accuracy. Table 31 compares the final counts with those of the MLP.

Figure 15. Transformer singular value spectra (seed 0) for , , , , , , , and the MLP post-activations, with the 95% and 99% variance thresholds.
Dense Time Course
All transformer runs were repeated with a checkpoint every 250 steps. For each grokked seed we compute of , , and at every checkpoint within 3,000 steps of its , testing every from 1 upward. The restricted and excluded losses follow Nanda et al. [ [28] in output space: for each input, we keep (restricted) or remove (excluded) the components of the centered residue logits along and for the five key frequencies of , and evaluate the cross-entropy on the training set. For each curve, the midpoint is the first checkpoint at which the count falls below the midpoint between its value at the start of the window and its final value. The restricted loss uses the same rule, and the excluded loss the first checkpoint above the midpoint of its rise. The midpoints relative to are: +26 488 steps (11 of 29 seeds before ), −862 503 (29 of 29), −957 328 (29 of 29), −741 375 (28 of 29), restricted loss −1,966 647 and excluded loss −1,026 646.
Table 32 gives for the runs that never generalize. They narrow slowly, most at weight decay 0.3, where test accuracy also creeps up to 9–13%, but at the end of training they remain two to nine times wider than the grokked seeds.
| Run | Step | Test acc. (%) | ||||
| wd 1.0, seed 12 | 5,000 | 31 | 33 | 64 | 70 | 2.8 |
| end | 23 | 26 | 49 | 55 | 8.2 | |
| wd 0.1, seeds 0–2 | 5,000 | 45–47 | 42–43 | 74–77 | 81–84 | 1.0–2.8 |
| end | 35–38 | 35–37 | 67–69 | 78–83 | 1.8–4.3 | |
| wd 0.3, seeds 0–2 | 5,000 | 40–42 | 38–40 | 71–73 | 79–82 | 1.2–2.9 |
| end | 29–30 | 30–31 | 48–52 | 53–59 | 9.1–13.2 | |
| Grokked (29), + 3,000 | – | 8.6 2.0 | 13.8 1.7 | 9.0 1.8 | 8.9 1.7 | 100.0 |
Table 32. Runs that never generalize: (centered logits within 10%, test inputs) at step 5,000 and at the end of training, with grokked seeds 3,000 steps after their onset for comparison.
Runs That Never Grok and Attention Projections
Table 33 lists the seven transformer runs that did not reach 95% test accuracy within 25,000 steps. has rank 4 in all of them, and in those trained with weight decay of at least 0.3. At weight decay 0.1, where the random initialization has not yet decayed, keeps 18–19 directions. The rank-4 structure of and , one direction per head, is architectural rather than a signature of grokking: only the attention pattern of the final position, which always holds the “=” token, affects the output, so each head’s query is a single vector and the gradients of its query and key projections have rank one.
Residual-Stream Alignment
Consecutive components communicate through shared subspaces of the 128-dimensional residual stream, but this is not specific to grokking (Table 34). What attention writes is almost exactly what the MLP reads (0.923 0.035), and what the MLP writes is what the unembedding reads (0.885 0.078), whereas the MLP’s own read and write subspaces are only weakly aligned (0.330 0.050). The attention-to-MLP alignment is equally high in the non-grokked runs (0.839–0.947). For a matrix that reads from the residual stream (), its read subspace is spanned by its top- left singular vectors. For a matrix that writes to it (), its write subspace is spanned by its top- right singular vectors. For two subspaces with orthonormal bases , the principal angles are the arccosines of the singular values of , and we report the mean of their cosines, with key frequencies +1. The key-frequency subspace of is spanned by the images under of the cosine and sine of each of its five strongest frequencies and of the constant vector. Random baselines average 300 pairs of independent random subspaces.
Representational Geometry
Transformer logits are dominated by a few frequencies, while the MLP’s representations are not (Figure 17). Five frequencies carry 87.9% of the logit power (seed 0, after removing each input’s mean logit). The first two principal components of the transformer’s logits carry 54.9% and 31.4% of their variance and show clusters of nearby classes, and projected on the strongest frequency () the logits lie on a ring ordered by . The first two principal components of the MLP’s pre- and post-activations carry only 4.0%/3.5% and 6.5%/5.7% of the variance and show no ring, consistent with the dozens of frequencies the MLP uses (§ 8).

Figure 17. (a) Transformer logits: first two principal components (left) and projection on the dominant logit frequency (right), colored by .
(b) MLP pre-activations (left) and post-activations (right): first two principal components, colored by .
Representational geometry (seed 0).
References
- [1]Roman Abramov, Felix Steinbauer, and Gjergji Kasneci. Grokking in the wild: Data augmentation for real-world multi-hop reasoning with transformers. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 2025. URL https://proceedings.mlr.press/v267/abramov25a.html.arxiv.org/abs/2504.20752
- [2]Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. In International Conference on Machine Learning, 2018. URL https://mlanthology.org/icml/2018/arora2018icml-optimization/.
- [3]Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/c0c783b5fc0d7d808f1d14a6e9c8280d-Paper.pdf.arxiv.org/abs/1905.13655
- [4]Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/monosemantic-features/index.html.
- [5]Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/chughtai23a.html.arxiv.org/abs/2302.03025
- [6]Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK.arxiv.org/abs/2309.08600
- [7]Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent, 2023. URL https://arxiv.org/abs/2303.06173.
- [8]Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The complexity dynamics of grokking. Physica D: Nonlinear Phenomena, 2025. URL https://www.sciencedirect.com/science/article/pii/S0167278925003367.DOI
- [9]Darshil Doshi, Aritra Das, Tianyu He, and Andrey Gromov. To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.13061.
- [10]C. W. J. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 1969.
- [11]Andrey Gromov. Grokking modular arithmetic. arXiv:2301.02679, 2023. URL https://arxiv.org/abs/2301.02679.
- [12]Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018. URL https://arxiv.org/abs/1812.04754.
- [13]Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2023. URL https://arxiv.org/abs/2103.10427.
- [14]Ahmed Imtiaz Humayun, Randall Balestiero, and Richard Baraniuk. Deep networks always grok and here is why. In Proceedings of the 41st International Conference on Machine Learning, 2024. URL https://proceedings.mlr.press/v235/humayun24a.html.
- [15]Prakhar Kaushik, Shravan Chaudhari, Ankit Vaidya, Rama Chellappa, and Alan Yuille. The universal weight subspace hypothesis, 2025a. URL https://arxiv.org/abs/2512.05117.
- [17]Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=vt5mnLVIvo.
- [18]Ruian Lei and Zenglin Xu. Geometric compression in grokking: The three-stage modular dynamics of transformers, 2026. URL https://openreview.net/forum?id=27fc8hXB5N.
- [19]Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. URL https://arxiv.org/abs/1804.08838.
- [20]Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. In International Conference on Learning Representations, 2023a. URL https://arxiv.org/abs/2210.01117.
- [21]Ziming Liu, Ziqian Zhong, and Max Tegmark. Grokking as simplification: A nonlinear complexity perspective. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023b.
- [23]Neil Mallinar, Daniel Beaglehole, Libin Zhu, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Emergence in non-neural models: Grokking modular arithmetic via average gradient outer product. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://arxiv.org/abs/2407.20199.
- [24]William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. arXiv preprint arXiv:2303.11873, 2023. URL https://arxiv.org/abs/2303.11873.
- [27]Tiberiu Musat. The geometry of grokking: Norm minimization on the zero-loss manifold. arXiv:2511.01938, 2025. URL https://arxiv.org/abs/2511.01938.
- [28]Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2301.05217.
- [29]Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177, 2022. URL https://arxiv.org/abs/2201.02177.
- [30]Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the edge of numerical stability. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=TvfkSyHZRA.arxiv.org/abs/2501.04697
- [32]Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 15th European Signal Processing Conference (EUSIPCO), 2007.
- [34]Peter H. Schönemann. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 1966.DOI
- [35]Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency, 2023. URL https://arxiv.org/abs/2309.02390.
- [38]Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2306.17844.
