WorksComputer Science
Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer
Abstract
We study the local differential geometry of a Fourier-based modular-addition circuit in a one-layer transformer whose algorithm has previously been reverse-engineered. For each of the five key Fourier frequencies, we restrict the model's multilayer perceptron (MLP) contribution to a corresponding Fourier input plane and readout, and measure its directional Beltrami coefficient under continuous input perturbations. Whereas nondegenerate restrictions of ideal complex multiplication are conformal, the trained maps exhibit nonconformal distortion (median –), spatial organization in the Beltrami argument, and regions of local orientation reversal. The median across frequencies decreases from at initialization to after training. The key-frequency planes also show lower median distortion than most random-plane controls, although their distributions overlap and the coordinate frames differ in conditioning. Across 500 checkpoints in one training run, most of the aggregate reduction in Beltrami magnitude occurs before the principal increase in the measured Fourier-circuit formation. At the individual-input level, Beltrami magnitude shows negligible association with the correct-logit margin (). Late-checkpoint measurements across six runs show comparable distortion for most selected frequencies. These results provide a quantitative characterization of local deformation in a frequency-projected component of an identified neural circuit, complementary to its algorithmic reconstruction.
Affiliations
¹ AI Mechanisms Lab, Chile
² WriteWise, Chile
³ Facultad de Matemáticas, Pontificia Universidad Católica de Chile, Santiago, Chile
⁴ Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Germany
⁵ Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile
Corresponding authors: Eduardo Fuentes ([email protected]) and Héctor Allende-Cid ([email protected]).
Introduction
Mechanistic interpretability seeks to explain neural-network behavior by identifying the computations implemented by model components. Modular addition has provided a tractable setting in which such computations can be reconstructed. Small transformers trained on this task can exhibit grokking [9], with generalization emerging long after the training examples have been memorized. Nanda et al. [7] reverse-engineered a Fourier-based algorithm learned by a one-layer transformer, showing how discrete Fourier components and trigonometric identities implement modular addition. They validated the circuit through weights, activations, and Fourier-space ablations, and used it to study training dynamics. This reconstruction specifies an algorithm implemented by the network. It does not, however, determine the local differential properties of every component involved in that algorithm, particularly when its inputs are varied continuously rather than restricted to the discrete tokens seen during training.
Recent studies have also examined the geometry of learned neural representations. Engels et al. [4] identified circular features used for modular arithmetic in language models and tested their computational role through interventions. Gurnee et al. [5] studied how attention heads transform low-dimensional feature manifolds during a counting task. Zhong et al. [10] reported that changes in model hyperparameters and initialization can lead to qualitatively different algorithms for modular addition, including the Clock algorithm previously described by Nanda et al. [7] and the Pizza algorithm. In modular addition, Moisescu-Pareja et al. [6] found geometrically and topologically equivalent representations in models with different attention mechanisms. More generally, Brandon et al. [2] used Riemannian pullback metrics and curvature to analyze how networks learn discrete computations on continuous input manifolds. These studies connect computation to the geometry of learned representations. A different question concerns the local distortion of the map between an identified input plane and an internal readout. Even when the relevant representation and algorithm are known, that distortion must be measured from derivatives of the map itself.
The Fourier algorithm in modular addition provides a precise reference for this question. Multiplication of complex numbers represents addition of their phases, and the ideal map is holomorphic in either variable when the other is fixed. Its nondegenerate one-variable restrictions are therefore conformal. The corresponding transformer computation is not required to equal this ideal complex map under continuous perturbations of its embedding inputs. To quantify the difference, we use the Beltrami coefficient [1], defined from Wirtinger derivatives of a real-differentiable map. Its magnitude measures departure from conformality, and its argument determines the principal stretching axis; the sign of the Jacobian determinant indicates local orientation. Conformal methods have also been used to learn embeddings from distorted observations [8], while Beltrami coefficients have been used in deep-learning methods for image registration [3]. Here the object of measurement is instead an internal transformer component associated with a reconstructed Fourier circuit.
We examine the local differential geometry of the Fourier-based modular-addition circuit reconstructed by Nanda et al. [7]. For each of the five key Fourier frequencies, we restrict the transformer’s MLP contribution to a corresponding input plane and Fourier readout, and measure the resulting map using directional Beltrami coefficients. This allows us to characterize departures from conformality, local stretching directions, and orientation reversal. We compare median Beltrami magnitudes between key-frequency and random control planes, examine their evolution during training in one run, evaluate late checkpoints from six training runs, and test their association with the correct-logit margin. Our analysis focuses on the geometry of an identified internal computation rather than on the geometry of representations alone.
We find nonconformal distortion at all five key frequencies, with median Beltrami magnitudes below the Gaussian bilinear reference at each frequency and an overall decrease in the median across frequencies from initialization. The argument of the Beltrami coefficient exhibits spatial organization, with similar values at neighboring points, and orientation-reversing regions differ in extent across frequencies. In the main experiment, median Beltrami magnitudes on the key-frequency planes are lower than those of most random control planes, although the distributions overlap and differ in coordinate-frame conditioning. During training, Beltrami magnitude changes early, before much of the measured circuit formation, but does not follow a monotonic trajectory. At the level of individual inputs, it has little association with the correct-logit margin. Together, these results show that recovering a circuit’s algorithm does not fully characterize the local geometry of its implementation. Directional Beltrami analysis provides a complementary way to measure the deformation of an internal map whose computation has already been identified.
Mathematical Framework
The Fourier circuit identified by Nanda et al. [7] represents modular addition through trigonometric components of , consistent with multiplication of unit complex numbers. We study the local geometry of the corresponding computation, rather than the accuracy of its Fourier approximation. For each key frequency, we construct a two-input complex-valued map from the trained network, differentiate it with respect to one input at a time, and measure the resulting departure from conformality. This construction is interventional: the continuous inputs used for differentiation are not ordinary tokens.
Fourier coordinates and the internal two-input map
Let and let denote the embedding matrix restricted to numeric tokens. For the five key frequencies , set and define the real Fourier coefficient vectors
The two vectors are linearly independent for every at the analyzed checkpoint (verified numerically). They therefore define a real plane , together with the complex coordinate chart
Its induced complex structure satisfies and . The sign convention is chosen relative to the sine readout so that the ideal sum-frequency computation is represented by , rather than . The factor in Eq. (1) is fixed by discrete Fourier reconstruction: the frequency- component of the actual embedding of token is exactly . Thus the unit circle specifies the embedding’s actual single-frequency amplitude, not an arbitrary radius. Conformality is measured in these declared complex coordinates, not in a coordinate-free metric on the ambient ; the conditioning of the frequency frames is assessed in Appendix § 8.
Write for the trained network’s map from the residual-stream vectors at the two numeric-token positions to the MLP activation vector at the prediction position. The positional encodings at those positions are ; the residual stream of the fixed “=” token is held fixed inside . With , let be the unit-normalized sequences and , and define
The ten sine and cosine sequences for are orthonormal on . In particular, Eq. (3) defines the Fourier readout of the MLP contribution to the logits, not the transformer’s complete end-to-end logit map. The map studied below is
Only the numeric-token embeddings are replaced by their single-frequency components; the trained weights, positional encodings, attention computation, fixed prediction-position input, and MLP are otherwise retained. Equation (2.4) extends the unit-circle inputs to an open complex domain so that derivatives are defined. Points away from the circle are extrapolations under the declared linear chart, not observed training inputs.
Directional Beltrami distortion
For fixed or , consider the two families of coordinate-line restrictions
These are restrictions to complex coordinate lines; they are not assumed holomorphic. Given a real-differentiable map with , its Wirtinger derivatives are
At points where the slice is and , define
only when .
Applying Eq. (7) to the two restrictions yields , and their respective dilatations. The principal measurements in Section 3 use the first-input coefficient unless stated otherwise. The excluded critical locus comprises points of nondifferentiability and points where $\partial_z h=0; we do not assume that the actual restricted network is globally or prove that all such loci have measure zero.
The geometry is read directly from the real Jacobian . The identity
shows that corresponds to local orientation preservation, to reversal, and to zero determinant, wherever is defined. Unlike in a quasiconformal map, is not imposed on the network. We therefore report only on the orientation-preserving locus. For a nonsingular Jacobian with singular values , the orientation-independent axis ratio satisfies
It agrees with when but also measures finite distortion at orientation-reversing points; it diverges for a singular Jacobian. The argument of retains directional information discarded by : the local principal stretch axis is modulo , when that axis is defined. § 3 and Appendix § 7 report its spatial organization separately from the magnitude.
Coordinate dependence and exact references
Proposition 2.1 (Coordinate behavior). Let be and at the point of evaluation. Multiplying its complex-valued output by any leaves unchanged. Under an input reparametrization with ,
For the two-slot map the corresponding chart replacement gives , with the same relation for each directional coefficient at the transported point.
Thus the readout plane is invariant under complex-linear scaling and rotation, whereas input-plane changes transport the measurement to different physical points. Unit-modulus input rotations preserve the distribution of over the full torus; radial rescalings generally do not. We therefore use the fixed Fourier normalization in Eq. (1) throughout. The directional coefficients also obey if exactly; approximate slot symmetry in the trained model is not treated as this exact identity. Formal proofs and qualifications are given in Appendix § 6.
Two references calibrate the magnitude. For the ideal complex multiplication with , every nondegenerate coordinate slice is holomorphic and therefore and . At the opposite reference, a random real bilinear computation has an exact, distributional baseline:
Proposition 2.2 (Gaussian bilinear reference). Let be bilinear with independent standard-Gaussian tensor coefficients. For any fixed nonzero second input, the induced first-input Jacobian has independent centered Gaussian entries with a common variance. Then
These values follow because the squares of the holomorphic and antiholomorphic derivative magnitudes are independent, identically distributed multiples of . They are exact population statements for the specified Gaussian ensemble, not universal identities for every bilinear map. The proof is given in Appendix § 6. Together, ideal multiplication and this Gaussian reference make the observed distortion interpretable without conflating a global Fourier fit with local conformality.
Experimental procedures
We use the publicly released full_run_data.pth checkpoint series for a one-layer transformer trained on addition modulo . The model has residual width 128, four attention heads, and MLP width 512; the training fraction is 0.30 and weight decay is 1. The main trained-network measurements use epoch 40,000. Details of the model and checkpoint verification are given in Appendix § 10.
Automatic differentiation of with respect to the first residual-stream input gives . The directional Jacobian follows from the chain rule:
The second-input derivative is analogous. We compute from these matrices in float64, excluding numerically degenerate derivatives by a relative threshold of .} Regime A samples the unit torus on a grid ( for the random-plane comparison; for pointwise maps); Regime B uses actual numeric-token embeddings as basepoints while retaining the same Fourier-plane differentiation directions. Both are interventional measurements (Appendix § 7).
At epoch 40,000, we compare the five key frequencies with 20 norm-matched random input planes and the non-key Fourier frequency ; conditioning and scale checks are described in Appendix § 8. For replication, six independently trained runs are evaluated at epochs 40,000 and 40,100, selecting the five highest-embedding-Fourier-norm frequencies separately in each run (Appendix § 9). Training dynamics are measured at all 500 saved checkpoints (epochs 0–49,900 in steps of 100) and compared with the fraction of pair-centered logit variance explained by the five key Fourier frequencies. Finally, at epoch 9,400, we test the Spearman association between and the correct-logit margin of the full model on 8,939 held-out input pairs. Definitions, cutoff rules, and statistical checks for the last two analyses are provided in Appendix § 10.
Results
We characterize the geometry of the trained transformer’s Fourier computation, compare the measurements with their analytical and empirical references, and examine their evolution during training. Together, these analyses distinguish properties of the internal map from their relationship to the learned Fourier circuit.
The trained transformer exhibits nonconformal distortion
The Fourier multiplication algorithm identified by Nanda et al. [7] provides a conformal reference: ideal complex multiplication has wherever the relevant directional derivative is nonzero. We measured the directional Beltrami coefficient of the corresponding internal maps at the public epoch-40,000 checkpoint, whose reconstructed forward pass solves all 12,769 modular-addition pairs with 100% accuracy (mean cross-entropy ). Across the five key Fourier frequencies, median ranges from 0.3036 to 0.4455 on the Regime A grid (Table 1). The median ratios of the largest to smallest singular values of the directional Jacobian, , range from 1.8718 to 2.6070, showing that infinitesimal circles are generally deformed into ellipses.
| Frequency | Median | Median | Orientation preserved |
| 14 | 0.3468 | 2.0617 | 93.8% |
| 35 | 0.3902 | 2.2684 | 92.6% |
| 41 | 0.3036 | 1.8718 | 93.8% |
| 42 | 0.4455 | 2.6070 | 78.5% |
| 52 | 0.4001 | 2.2798 | 94.9% |
Table 1. Directional Beltrami distortion in the five key frequencies at epoch 40,000 (Regime A, grid). The singular-value ratio includes orientation-reversing points.
The departure from conformality includes a change in local orientation. The strict condition fails on the sampled domain for every key frequency. On the finer grid, the fractions of points with are 1.27%, 8.91%, 5.78%, 21.45%, and 4.08% for , respectively. These regions are visible in Figure 2A–B: the orientation-reversing region for is concentrated on the antipodal-input line and its immediately neighboring grid diagonals, while exhibits a much larger reversing region. At , all 208 such grid points fall within this three-diagonal band. The conventional dilatation is restricted to ; in the present measurements its censored medians (approximately 2) underestimate the median distortion of the full sampled domains, whose uncensored axis ratios are reported in Table 1 (Appendix § 7).
Figure 2. Pointwise directional Beltrami field of the trained transformer (epoch 40,000; Regime A; torus grid). (A, B) for and on the orientation-preserving locus; dark gray indicates (1.27% and 21.45%, respectively). (C, D) on the same locus using a cyclic colormap. Sharp boundaries reflect the piecewise-analytic structure induced by ReLU activation regions.
The distortion also exhibits spatial organization. The axial orientation of is defined by (mod ), with the angular analysis restricted to . Although the global axial resultant lengths are small (0.0124–0.2019), neighboring grid points have angular concentrations of 0.7715–0.9800, compared with permutation-null means of 0.0412–0.0524. The observed concentrations exceed those of all 1,000 spatial permutations, consistent with the angular fields shown in Figure 2C–D. The permutation comparison establishes local spatial organization, rather than global smoothness; full angular measurements and the separate equivariance check are provided in Appendix § 7.
To assess whether the measurements depend on the isolated Fourier-component inputs of Regime A, we evaluated the directional Beltrami coefficient around 40 actual embedding pairs (Regime B). Median values for are 0.3223, 0.4432, 0.3077, 0.4525, and 0.3550, respectively, broadly comparable with Regime A; orientation-preserving fractions range from 0.675 () to 1.000 (). Thus, similar distortions are also measured at real embedding basepoints, although the infinitesimal perturbation directions remain interventional (Appendix § 7).
Key Fourier frequencies show lower distortion than controls
Having identified nonconformal, spatially organized geometry in the trained model, we next compare its magnitude with initialization, random planes, and independent training runs. The aggregate distortion is substantially lower after training. At initialization, the median across the five per-frequency medians is , and the mean orientation-preserving fraction is 0.548; at epoch 40,000, these values are 0.3902 and 0.907 (Table 2, Block A). For reference, a real bilinear map with an i.i.d. standard-Gaussian directional Jacobian has population median and preserves orientation with probability (Appendix § 6). The aggregate shift is not uniform across frequencies: and begin at 0.4758 and 0.4035, respectively, relatively close to their final values.
| Block | Condition | Median | Orientation preserved |
| A | Gaussian bilinear reference (population) | 1.000 | 0.500 |
| A | Untrained weights, epoch 0 | 0.939 | 0.548 |
| A | Trained weights, epoch 40,000 | 0.390 | 0.907 |
| B | Five key-frequency planes | 0.302–0.435 | 0.903 |
| B | Twenty random control planes | 0.697 | 0.799 |
Table 2. Analytical and empirical controls. Block A uses the Regime A grid; Block B uses a separate plane-comparison experiment. Summary statistics have different per-plane aggregation rules; definitions and comparability checks are provided in Appendix § 8.
Within the trained model, the key Fourier planes also show lower distortion than random planes. On the separate grid, the key-frequency medians are 0.302–0.435, whereas the median of medians for twenty norm-matched random planes is 0.697 (Table 2, Block B). Three random controls (0.401, 0.430, and 0.431) overlap the key-frequency range, so the distributions are not fully separated. A rank comparison gives (); this is a descriptive comparison because the key frequencies were functionally selected and independence of control medians within the shared
Aggregation: In A, transformer values are medians of five frequency medians and orientation fractions are means; the Gaussian row gives population values. In B, the key value is a range and the orientation fraction a mean, whereas random-plane entries are medians of per-plane statistics (Appendix § 8).
network has not been established. The random controls also differ from the key planes in input-frame conditioning (median 1.34 versus 1.00–1.03) and readout-frame conditioning. Nevertheless, rescaling the twenty random planes over 0.5–2.0 gives median-of-medians of 0.687–0.735, while the corresponding rescaled key-frequency medians remain within 0.27–0.45; at 1.00 the random-plane result is 0.697. Construction, energy-overlap checks, and scale sensitivity are detailed in Appendix C.
The geometric measurements also recur across independent training runs. For each of six runs, we selected its five highest Fourier-norm frequencies and evaluated them at epochs 40,000 and 40,100. At epoch 40,000, 22 of 30 frequency-level medians lie in [0.30, 0.45], while four exceed 0.90 and one is below 0.30 (Figure 3). Depending on the numerical band used, the replication count varies from 20 to 27 of 30. The central concentration therefore recurs across runs without every frequency converging to the same geometric range. The complete replication figure, epoch comparison, and individual results are in Appendix D.
Figure 3. Median for the five highest Fourier-norm frequencies of each of six training runs, measured at epochs 40,000 and 40,100. Companion dilatation and band-sensitivity panels appear in Appendix D.
Taken together, the initialization, control-plane, and independent-run comparisons show that lower directional distortion is a recurring feature of the key Fourier representations after training.
Beltrami distortion decreases before most circuit formation
To determine when this geometric change develops relative to the learned Fourier circuit, we evaluated median on the 16 16 Regime A grid at all 500 saved checkpoints, from epoch 0 to 49,900. The aggregate reduction measured between initialization and the trained checkpoint occurs early in training. Median distortion across the five frequencies is 0.9394 at initialization, 0.4920 at epoch 200, and 0.3906 at epoch 1,400, compared with 0.3902 at epoch 40,000 (Figure 4A). During the main phase of Fourier-circuit formation (epochs 5,400–11,400, during which the circuit metric rises from 0.1 to 0.9), individual trajectories fluctuate without a common directional trend.
Figure 4. Geometric distortion, Fourier-circuit formation, and model behavior. (A) Median for five key frequencies and their median at all 500 checkpoints (Regime A, grid). (B) Fraction of pair-centered logit variance explained by the five key frequencies at those checkpoints. Dotted lines mark epochs 476, 2,390, and 9,830; the epoch axis is logarithmic above 100. (C) Spearman correlations of with correct-logit margins for 8,939 test pairs at epoch 9,400, alongside the embedding-norm baseline; shading indicates the 95th percentile of under 1,000 permutations.
The circuit metric follows a different trajectory (Figure 4B). At epoch 2,400, the five key frequencies explain 0.0277 of the pair-centered logit variance, compared with 0.9690 at epoch 40,000, corresponding to approximately 2.9% of the measured development of the circuit metric. By the same epoch, 74.5–96.6% of the measured change in has already occurred across the four frequencies with nondegenerate total change. The predefined temporal criterion required at least 60% of total geometric change before generalization onset in at least four of five frequencies; it was met in five of five frequencies under the primary window and four of five under the sensitivity window. The exception requiring care is : its total initial-to-final change is approximately 0.0035, making that fractional measure uninformative; excluding it yields four of four and three of four frequencies meeting the criterion, respectively. These results establish that the reduction in measured directional distortion precedes most of the development of the identified Fourier circuit (Appendix § 10).
We also tested whether the geometric measurement is associated with correct-logit margins on individual inputs. At epoch 9,400, using 8,939 held-out input pairs (test accuracy 0.5098; training accuracy 1.0000), the Spearman correlations for are −0.0260, −0.0146, −0.0070, −0.0224, and −0.0058, respectively (Figure 4C). Two correlations, at and , exceed the frequency-specific 95th percentile of under 1,000 label permutations (–0.021). Nevertheless, all satisfy , far below the preregistered association threshold of 0.20. The embedding-norm baseline has , and additional configurations yield a maximum of 0.052 (Appendix § 10). Thus, directional Beltrami magnitude does not provide a useful predictor of the correct-logit margin under the tested conditions.
Discussion
Nanda et al. [7] identified a Fourier-based algorithm for modular addition in a one-layer transformer by analyzing its weights, activations, and the effects of Fourier-space ablations. Here, we measure the directional Beltrami coefficient of the MLP map restricted, for each of the five key frequencies, to that frequency’s input plane and Fourier readout. We find nonconformal distortion, spatially organized variation in the Beltrami field, and regions of local orientation reversal. These measurements describe the differential geometry of a component of a computation whose algorithm has already been identified.
For ideal complex multiplication, , each nondegenerate one-variable restriction is conformal and has a zero Beltrami coefficient. The corresponding frequency-projected MLP maps of the trained transformer, however, have nonzero Beltrami coefficients and reverse orientation in parts of the sampled domain. This is consistent with Nanda et al.’s circuit reconstruction. Their analysis identifies how Fourier components and trigonometric identities implement modular addition; it does not imply that an individual projected MLP map is equal, as a differentiable function, to ideal complex multiplication under continuous embedding perturbations. The recovered algorithm and the local distortion of the projected MLP map are therefore distinct properties.
Other work has examined the geometry of representations learned by neural networks. Engels et al. [4] identified circular features involved in modular computations in language models. Gurnee et al. [5] showed that attention heads transform low-dimensional manifolds in a counting task. Moisescu-Pareja et al. [6] reported that architectures with uniform and trainable attention, associated with the Clock and Pizza interpretations introduced by Zhong et al. [10], learn geometrically and topologically equivalent representations and implement the same underlying algorithm. These studies examine the structure or functional use of representations. Our analysis instead measures derivatives of the map from Fourier input directions to an MLP readout at the same frequency. The geometry of a representation need not determine the distortion of this map.
Related geometric methods have also been applied to neural networks. Brandon et al. [2] analyzed Riemannian pullback metrics and curvature, while Chen et al. [3] applied Beltrami coefficients to neural image registration. We use the Beltrami coefficient to characterize the local deformation of a frequency-projected transformer component. Its magnitude is , the ratio of the magnitudes of the two Wirtinger derivatives; its argument determines a principal stretching direction, and the sign of the Jacobian determinant indicates orientation. The fields show local spatial organization; orientation reversal is confined to a narrow antipodal band at but covers a much larger region at . The five key Fourier planes also have lower median distortion than typical random-plane controls in the main experiment. Measurements from other training runs show medians in the same range as the main run for most selected frequencies at late checkpoints; the random-plane comparison was not repeated in those runs. Differences in coordinate-frame conditioning remain a possible contributor to the separation observed in the main experiment.
The training results show that directional distortion changes on a different timescale from the circuit measure. At epoch 1,400, the end of the memorization phase described by Nanda et al. [7], the median Beltrami magnitude across the five key frequencies is 0.3906, close to its value of 0.3902 at epoch 40,000. It does not remain constant: the median and individual frequency trajectories fluctuate during subsequent training. Most of the reduction from initialization occurs before the major increase in the fraction of pair-centered logit variance explained by the key Fourier frequencies. This finding identifies an early change in the geometry of the projected MLP map. Our circuit statistic differs from the restricted and excluded losses used by Nanda et al.; the temporal ordering does not by itself imply a causal relationship between distortion and circuit formation.
The input-level analysis tests a separate relationship. At epoch 9,400, Beltrami magnitude shows only weak rank correlations with the correct-logit margin across held-out input pairs. Although two correlations exceed their frequency-specific permutation thresholds, all are well below the predefined effect-size threshold. Thus, under the tested conditions, local distortion is not a useful predictor of the correct-logit margin for individual inputs. The two measurements concern different objects: the derivative of a frequency-projected MLP component and the output logits of the full model.
These findings were obtained from a one-layer transformer trained on modular addition. The temporal and input-level behavioral analyses use one training run, while measurements from six runs cover only late checkpoints. Regime B evaluates derivatives at actual embedding basepoints but along continuous Fourier directions. The random control planes differ from the key-frequency planes in input- and readout-frame conditioning. The analysis concerns two-dimensional sections of a projected MLP map rather than the full high-dimensional model. The input scale is fixed by the embedding reconstruction, and the transformation of the Beltrami coefficient under coordinate changes is given by Proposition 2.1.
A natural next step is to compare the directional distortion of circuits that learn equivalent representations. In particular, the uniform- and trainable-attention models examined by Moisescu-Pareja et al. [6] provide a test of whether equivalent representation geometry also entails similar local distortion. Further experiments with conditioning-matched controls, independent training trajectories, and targeted interventions could clarify the relationship between distortion and circuit computation. The Beltrami coefficient thus adds a quantitative measure of local deformation to the analysis of this Fourier-based modular-addition circuit.
Conclusion
Taken together, our results show nonconformal distortion and regions of reversed orientation in the frequency-projected MLP contribution of a transformer trained on modular addition. Directional Beltrami analysis reveals spatial variation in the magnitude and direction of this distortion. Across training, the median decreases early, before most of the measured increase in the Fourier circuit, but continues to fluctuate thereafter. In this model, does not predict the correct-logit margin for individual inputs under the tested conditions. The results indicate that the Beltrami coefficient provides a quantitative measure of the local geometry of a circuit whose Fourier algorithm has already been reconstructed. Future work should test other circuits and model architectures, examine independent training runs, and use interventions to determine whether these geometric changes affect the learned computation.
Mathematical Proofs
This appendix provides the derivations used in § 2. The coordinate construction and Fourier normalization are established first; the local Jacobian identities then connect the Beltrami coefficient with orientation and singular values. We finally prove Proposition 2.1 (coordinate behavior) and Proposition 2.2 (the Gaussian bilinear reference). Claims involving the directional Beltrami coefficient concern points at which the relevant coordinate slice is differentiable and its holomorphic derivative is nonzero. Numerical nondegeneracy of the measured Fourier frames is an empirical check, not an exact-arithmetic theorem.
Discrete Fourier coordinates and the choice of orientation
For and , the finite character sum gives
For , the product-to-sum identities and Eq. (A.1) imply
Both sine and cosine sequences have zero mean. Consequently, the normalized vectors and form an orthonormal family in . In particular, the five-frequency readout vectors in Eq. (3) are exactly orthonormal as functions of the output token. Their images need not be orthonormal, a distinct property of the learned weights.
Let and . Discrete Fourier inversion, with the pair and combined into a real component, yields
Thus the frequency- component is exactly , with . Equation (A.3) fixes the factor : replacing it by another common factor would no longer reconstruct that component on the unit circle. For each measured , the chart is an isomorphism onto provided and are linearly independent; this condition was checked numerically on the checkpoint.
The orientation is specified relative to the chosen sine readout. For and ,
Replacing both input charts by their conjugates changes the ideal coordinate expression to : its real component is unchanged, but the signs of both imaginary cross terms reverse. With the output sine convention held fixed, the positive cross terms of the Fourier multiplication mechanism select the orientation used in § 2. Simultaneously conjugating the input charts and output readout would instead constitute a global relabeling of this orientation, equivalent to exchanging frequencies and .
For completeness, let the ideal slice map be with . With fixed, has and . Thus and ; fixing gives the same conclusion for the second slot. At a zero fixed input, the slice is constant and its Beltrami coefficient is undefined under Eq. (7). This proves the ideal reference used in § 2 without attributing the property to the measured network.
Real Jacobians, orientation, and principal stretching
Let be at the evaluation point and write and . Its real differential acts on a complex increment as
Indeed, with and , Eq. (A.5) agrees with . Identifying with , the determinant of the real Jacobian is
If , then and
This proves Eq. (8) and the distinction between local orientation preservation (), reversal (), and a singular real Jacobian (), wherever is defined. When , the Beltrami coefficient used in this paper is undefined even if the real Jacobian is nonsingular.
For , Eq. (A.5) implies the sharp bounds
The phase of can align with or oppose them, so both bounds are attained. Hence the singular values are and . At points with and ,
which is Eq. (9). On this equals ; on it remains a finite axis ratio but is not defined under our convention. In particular, the axis ratio decreases towards 1 as after the singular threshold, and should not be interpreted as a monotone extension of .
When and the singular values differ, maximal stretching occurs when is positive real, giving the input-plane axis . For (a conformal linear differential) there is no distinguished principal axis. This establishes the directional interpretation used in § 2 and Appendix § 7.
Proof of Proposition 2.1: coordinate behavior
Let be and at the evaluation point. If for , linearity of the Wirtinger operators gives and . Their ratio is unchanged: . Both and (where defined) are therefore invariant under complex-linear output reparametrization.
For , let , , and suppose is with at . The Wirtinger chain rule and , yield
Dividing gives exactly
with rotated by where its argument exists. For in both input slots, the definition of the measured map gives . Applying Eq. (A.10) separately to each fixed-slot restriction proves Proposition 2.1 for both directional coefficients. In particular, these equalities compare transported points; they do not give fixed-coordinate invariance when .
If , the map bijectively preserves the product unit torus and its normalized Haar measure. Equation (A.11) therefore preserves the distribution of and, on its defined locus, of over the full torus. This population statement does not assert equality of finite-grid summaries under arbitrary rotations that fail to permute the sampled grid points.
Conjugating the complex-valued output reverses this orientation. Specifically, let . The Wirtinger derivatives satisfy
where is defined and nonzero. Conjugation also reverses the real-Jacobian determinant sign: . In the Fourier readout, reversing the output orientation (, with unchanged) is precisely such a conjugation. Thus the reported sign of orientation, and the magnitude , depend on that readout convention unless it is externally anchored.
Finally, if exactly, then for fixed the functions and coincide. Their Wirtinger derivatives coincide wherever defined, so . Approximate slot symmetry alone yields no such exact identity or quantitative stability bound.
Proof of Proposition 2.2: Gaussian bilinear reference
Consider a real bilinear map with tensor coefficients . Fix a nonzero second input . The first-slot restriction has real Jacobian with entries
The four entries are independent because they depend on disjoint tensor coefficients. Write and set
The four corresponding coefficient vectors in are orthogonal and have squared length 2. Joint Gaussianity therefore makes , , , independent variables. The complex representation of the first-slot differential is
Consequently and are independent and identically distributed rotationally symmetric complex Gaussians, and their squared magnitudes satisfy
The event has probability zero. Since and are independent identically distributed exponentials up to a common scale, is uniform on . Thus, for ,
Setting gives (there is no point mass at 1). Since the distribution function is continuous and strictly increasing, its unique median is also 1, proving Proposition 2.2. These are population results for this specified isotropic Gaussian tensor ensemble; neither conclusion is asserted for an arbitrary deterministic bilinear map.
Regularity and the directional chain rule
The map contains a softmax and a ReLU MLP. On any open region with fixed strict ReLU activation signs, its operations are compositions of affine and real-analytic functions, so its restrictions and their real Jacobians are well defined. We make no global assertion that the pullbacks of all ReLU boundaries, or the zero sets of , have measure zero on the specific four-dimensional input chart or its fixed-slot slices. Equations (2.5)–(2.12) are used only at points where the stated derivatives are defined.
Writing and , the derivative of is the constant matrix . At a differentiability point, the ordinary real chain rule therefore gives
with the analogous formula for the second slot. This is Eq. (12). It characterizes the measurement on the trained network; floating-point thresholds used to detect numerical degeneracy are computational conventions, not additional mathematical hypotheses.
Extended Geometric Measurements
The measurements summarized in § 3.1 characterize the directional Beltrami coefficient of the Fourier computation at epoch 40,000. This appendix gives the complementary statistics needed to interpret those measurements: the effect of restricting dilatation to orientation-preserving points, the spatial organization of arg , measurements at actual input-embedding basepoints, and the relationship between local coefficient jumps and ReLU activation-pattern changes. Unless stated otherwise, measurements use the five key frequencies and the declared Regime A embedding and readout frames.
Orientation reversal and the censoring of dilatation
The conventional quasiconformal dilatation is used only where . Consequently, summarizing over the orientation-preserving locus omits precisely those points where the directional map reverses orientation. To describe the entire sampled domain, we additionally report the ratio of the largest to smallest singular values of the real directional Jacobian. This axis ratio equals where and remains defined at the observed orientation-reversing points. It should not be interpreted as an orientation indicator; that information is tracked separately by the sign of the Jacobian determinant.
As shown in Table B1, the median full-grid axis ratio is higher than the median censored at all five frequencies. The largest difference is at (2.6070 versus 2.1472), for which 21.48% of the 16 × 16 grid is excluded from the summary. These comparisons quantify the effect of censoring in the present measurements; they do not establish that the difference must have the same sign for other maps or samples.
| Median () | Median | Difference | Excluded (%) | |
| 14 | 2.0437 | 2.0617 | +0.018 | 6.25 |
| 35 | 2.0932 | 2.2684 | +0.175 | 7.42 |
| 41 | 1.8648 | 1.8718 | +0.007 | 6.25 |
| 42 | 2.1472 | 2.6070 | +0.460 | 21.48 |
| 52 | 2.1582 | 2.2798 | +0.122 | 5.08 |
Table B1. Censored and uncensored median dilatation in Regime A (epoch 40,000; 16 × 16 grid). The median of uses only points with ; the median singular-value ratio uses the full grid.
On the finer 128 × 128 Regime A grid, the fractions with are 1.27%, 8.91%, 5.78%, 21.45%, and 4.08% (Figure B1A). For , all 208 such points lie on three toroidal diagonals: 128 on (grid-index offset 64), and 40 on each adjacent diagonal (offsets 63 and 65). No reversing point lies outside this band. The percentages characterize this sampled resolution, not a resolution-independent bound.

(a)

(b)
Figure B1. Orientation reversal and angular organization across the five key Fourier frequencies. (A) Percentage of evaluation points with on the 128 × 128 Regime A grid. (B) Observed concentration of the differences in arg between neighboring grid points on the 16 × 16 Regime A grid, restricted to , compared with the mean and 95th percentile of a spatial permutation null (1,000 permutations).
Measurements at real-embedding basepoints
Regime A evaluates the Fourier-component restriction on an artificial torus. Regime B instead fixes the basepoint at the embedding of an actual modular-addition input pair while taking derivatives along the same Fourier-plane directions. Forty sampled pairs were evaluated at each frequency, and percentile-bootstrap intervals for the median were formed using 2,000 replicates (Table B2).
| Median | 95% bootstrap CI | Fraction | |
| 14 | 0.3223 | [0.3035, 0.3481] | 1.000 |
| 35 | 0.4432 | [0.3112, 0.6419] | 0.875 |
| 41 | 0.3077 | [0.2942, 0.3212] | 1.000 |
| 42 | 0.4525 | [0.3632, 0.8923] | 0.675 |
| 52 | 0.3550 | [0.2642, 0.6046] | 0.975 |
Table B2. Directional Beltrami distortion around real input embeddings (Regime B). Each frequency is evaluated at 40 sampled input pairs; confidence intervals are percentile-bootstrap intervals for the median from 2,000 replicates. The derivative directions remain interventional.
The Regime B medians (0.3077–0.4525) are similar in magnitude to the Regime A estimates, although the orientation-preserving fractions vary between 0.675 and 1.000. This agreement shows that the measured magnitude does not require the basepoints themselves to be outside the model’s input distribution. It does not make the derivative measurements in-distribution: the infinitesimal Fourier-plane perturbations remain interventional. The 40-pair bootstrap characterizes uncertainty in the sampled median, not extreme values or global regularity.
Spatial organization of the Beltrami direction
The argument of the complex coefficient, , describes the direction of local anisotropy; because the associated geometric axis is axial, a small global angular resultant need not imply spatially unstructured distortion. On the Regime A grid, the global axial resultant lengths range from 0.0124 to 0.2019 (Table B3). To distinguish spatial arrangement from the marginal distribution of angles, we evaluated the concentration of angular differences between toroidally neighboring points, considering both grid directions and restricting the calculation to .
| Global axial resultant | Observed neighboring concentration | Permutation null mean | Permutation null 95th percentile | |
| 14 | 0.0134 | 0.9800 | 0.0412 | 0.0815 |
| 35 | 0.2019 | 0.7866 | 0.0524 | 0.1003 |
| 41 | 0.0124 | 0.8561 | 0.0420 | 0.0806 |
| 42 | 0.0722 | 0.8407 | 0.0470 | 0.0919 |
| 52 | 0.1592 | 0.7715 | 0.0447 | 0.0859 |
Table B3. Angular organization of the directional Beltrami field in Regime A (epoch 40,000). The global axial resultant summarizes angular concentration across the grid, whereas neighboring concentration measures spatial organization of differences in on the grid, restricted to , with toroidal neighbors in both directions. The spatial-permutation null uses 1,000 permutations.
Note. At every frequency, the observed neighboring concentration exceeded all 1,000 reported spatial-permutation values. The null reassigns angular values to grid positions while retaining their marginal distribution.
The observed neighboring concentrations are 0.7715–0.9800, whereas the permutation-null means are 0.0412–0.0524 (Figure B1B; Table B3). At each frequency, the observed statistic exceeds all 1,000 spatial permutations. Because the null preserves the collection of angular values while reassigning their positions, this result supports local spatial organization, not global smoothness or concentration around one preferred axis. We also tested whether the low global resultants were forced by a putative identity for . The median relative error on the Regime A grid was 0.77–1.38 across the five frequencies; the proposed identity does not hold in those measurements. The low global resultants therefore should not be attributed to that exact equivariance relation.
Additional consistency checks
Three further checks delimit the interpretation. First, exchanging the two input slots gives median differences between 0.0001 and 0.0018 across frequencies on the Regime A grid. The map itself was tested separately and was approximately, but not exactly, symmetric; closeness of two normalized derivatives does not imply closeness of the underlying functions. Second, moving from to changed the per-frequency median by at most 0.0216; it did not establish corresponding stability for maxima, upper quantiles, or orientation fractions. Third, in a basepoint-perturbation check limited to and , perturbations up to changed by 0.0000 at the stated precision, while perturbations of changed by at most 0.028. Those perturbation results cannot be generalized to the other three frequencies.
ReLU activation regions and local coefficient jumps
The fine-grid maps in Figure 2 of the main text contain sharp features. Because the transformer’s feedforward block is piecewise linear in its activations, we compared the complex-coefficient jumps across neighboring toroidal evaluation points with the associated changes in the ReLU activation pattern. For each of and , the check considers 32,768 directed neighbor pairs on the grid, one pair from every grid point along each of the two toroidal directions. It records and whether the activation patterns agree at the two endpoints.
Note. Spearman is the correlation between the Hamming distance of activation patterns and the magnitude of the complex-coefficient jump; the “top 1%” column is the fraction of the largest jumps with a changed pattern. These statistics are reported at the precision of the original ReLU check. They establish association, not a causal attribution of all jumps to ReLU boundaries.
At , the median jump is 0.0000 at the reported precision for unchanged patterns versus 0.0558 for changed patterns, with Spearman between activation-pattern Hamming distance and jump magnitude. The association is weaker at (), where the median jumps are 0.0261 and 0.0355, respectively. At both frequencies, all of the largest 1% of jumps connect points with different activation patterns (Table B4). These observations associate sharp spatial changes with activation-pattern transitions; they neither identify a unique causal mechanism for each jump nor show that orientation reversal itself is caused by a ReLU boundary.
| Same pattern (%) | Median jump same | Median jump changed | 99th pct. same | 99th pct. changed | Top 1% jumps with change | Spearman | |
| 14 | 62.8 | 0.0000 | 0.0558 | 0.0001 | 1.8951 | 1.000 | 0.731 |
| 42 | 22.8 | 0.0261 | 0.0355 | 0.1354 | 0.3589 | 1.000 | 0.275 |
Table B4. ReLU activation-pattern changes and local jumps of the complex directional Beltrami coefficient in Regime A (epoch 40,000; grid). For each , 32,768 directed toroidal neighboring pairs (one per grid point in each of two directions) are compared. Jumps are ; “same” and “changed” refer to equality of the feedforward ReLU activation patterns at the two points.
Control Analyses
Section 3.2 compares the key Fourier frequencies with randomly constructed input planes and with a sixth, non-key Fourier frequency. This appendix specifies the plane comparison, documents its limitations, and presents the scale-sensitivity analysis. These are comparisons of directional Beltrami measurements within the same trained model, not tests against an independently sampled population of networks.
Construction and aggregation of the control planes
We compared the five key Fourier frequencies with twenty random planes at the epoch-40,000 checkpoint. Each random input plane was formed from an orthonormal pair of functions on , as in the Fourier-plane construction, and rescaled to match the median input-frame norm of the key planes, . The reference comparison was evaluated on a grid. Norm matching is important because directional distortion can depend on the scale at which the input directions are probed. However, matching input norms does not make the input or readout frames equally conditioned.
The summaries in Table C1 require attention to their aggregation operators. The key-plane entry gives the range of five frequency-level medians, whereas the random-plane entry is the median of twenty plane medians. For orientation preservation, the reported key-plane figure is a mean over five frequency-level fractions and the random-plane figure is the median of twenty fractions. The two orientation entries therefore must not be treated as like-for-like estimators of one population quantity.
| Diagnostic | Key Fourier planes | Random control planes | |
| Number of planes | 5 | 20 | 1 |
| Median | 0.302–0.435 (range) | 0.697 (median of medians) | 1.219 |
| Orientation preserved | 0.903 (mean) | 0.799 (median) | 0.000 |
| Input-frame | 1.003–1.027 (range) | 1.3445 median; [1.048, 2.672] | Not reported |
| Readout-frame | 1.01–1.03 (range) | 1.05–2.79 (range) | Not reported |
| Key-subspace energy overlap | Not applicable | 0.0947 mean; 0.0944 median; [0.054, 0.142] | Not reported |
Table C1. Control-plane comparability and diagnostic summaries (epoch 40,000; grid). Key frequencies are . The sixth frequency is shown separately; its orientation sign depends on a declared, externally unanchored convention.
Separation, conditioning, and energy neutrality
The five key-frequency medians span 0.302–0.435, compared with 0.697 for the random-plane median of medians (Table C1). The difference is distributional rather than absolute: three random controls have medians 0.401, 0.430, and 0.431, overlapping the range of the five key frequencies. Their combined rank ordering gives a Mann–Whitney statistic (), which we report descriptively. The key frequencies were selected for their established functional role, not sampled randomly; independence between the twenty control measurements in the same trained network has also not been established. Consequently, this rank contrast should not be read as a conventional population-level inference.
Notes. Plane-wise summary operators differ: key-plane orientation preservation is the mean of five fractions, whereas the random-control value is the median of twenty fractions; these entries should not be directly compared as like-for-like estimates. The twenty controls are norm-matched in the input frame, but not conditioning-matched. The key-subspace energy fraction for random controls is close to the isotropic expectation . Three control-plane medians (0.401, 0.430, 0.431) fall in the key-frequency range, so there is no strict separation. The rank contrast (, ) is descriptive, not a population-level inferential test: the key modes were selected functionally, and independence among controls within the same trained network has not been established. Orientation at is reported under a convention without a direct sign anchor.
The random controls have median input-frame condition number (range 1.048–2.672), versus 1.003–1.027 for the key planes. Their readout-frame condition numbers span 1.05–2.79, versus 1.01–1.03 for the keys. Thus, lower key-frequency distortion cannot be uniquely attributed to functional circuit membership on the basis of norm matching alone: differences in frame conditioning remain an alternative contributor. A separate energy check found that the random input-plane pairs project onto the ten-dimensional key-frequency subspace with mean energy fraction 0.0947 (median 0.0944, range 0.054–0.142), close to the isotropic reference . This supports neutrality with respect to key-subspace energy, but does not remove the conditioning difference.
Sensitivity to the input-plane scale
Because the main comparison fixes one reference norm, we examined five scale multipliers (Figure C1). For random controls, the input frames were normalized to ; for each key frequency, its existing input frame was scaled by . These are two distinct scaling constructions, and is therefore the appropriate axis for comparing the resulting sensitivity curves, not a claim of identical input frames.
Figure C1. Scale sensitivity of the key Fourier planes and the twenty norm-matched random controls (epoch 40,000; 12 × 12 grid). The multiplier ranges from 0.50 to 2.00: random control input frames are normalized to (), while each key-frequency input frame is multiplied by relative to its original scale. (A) Each gray point is one random plane’s median ; dark diamonds are the medians of the twenty per-plane medians. Horizontal jitter is only for visibility. (B) Median for each of the five fixed key-frequency planes across scale multipliers; the dashed curve repeats the random-plane median-of-medians for comparison. Lines connect measurements of the same frequency at different scale multipliers. At , two to three individual random planes fall within the key-frequency range; none do at or 2.00. The random-group median remains higher at every tested multiplier.
Across the five multipliers, the random controls have median-of-medians 0.687, 0.735, 0.697, 0.708, and 0.712, respectively. The key-frequency medians remain approximately 0.27–0.45 across the tested scales. Individual overlap is retained at the three smaller multipliers: two to three random controls fall in the key-frequency range for , whereas none do at or 2.00. The random-control median remains above the key-frequency range at every tested multiplier. The conclusion is consequently robust in the tested sweep at the group-summary level, not a universal separation between individual planes or scales outside that range.
The additional Fourier frequency
The additional frequency is present in the embedding matrix but has much weaker readout vectors than the five functionally identified key frequencies. In the 12 × 12 plane comparison, its median is 1.219 and its sampled orientation-preserving fraction is zero (Table C1). Its determinant sign is constant on the sampled Regime A grid, unlike the mixtures of signs measured at the five key frequencies and the twenty random controls. The constancy of the sign is a geometric observation; the designation of the constant sign as orientation-reversing is contingent on a readout-orientation convention that has no direct external anchor for . Under the conjugate readout convention, every sampled determinant sign at flips and becomes (median approximately 0.820 instead of 1.219); the constancy of the sign is unchanged (Appendix § 6.§ 3). Neither the magnitude nor the chosen sign, considered alone, establishes a distinct algorithmic role for this frequency. We therefore report as a comparison case, not as another functional key frequency or a calibrated negative control.
Replication Across Training Runs
§ 3.2 reports that the reduced directional Beltrami distortion of the key Fourier representations recurs after independent retraining. This appendix describes how frequencies were selected within each run, presents the paired measurements at epochs 40,000 and 40,100, and makes explicit the criteria behind the range of replication counts. Replication here concerns the distribution of frequency-level geometric measurements, not the identity of a frequency index across independently initialized networks.
Independent retrainings and measurement protocol
Six independent seeds were trained with the architecture and hyperparameters of the main experiment (, model width 128, four attention heads, MLP width 512, training fraction 0.3, weight decay 1) to epoch 40,100. For each run, the five frequencies with the largest Fourier-component norms of the embedding matrix (columns 0–112) were selected using the same rule, and this selection was held fixed across the two measured checkpoints. The selected indices vary between seeds; equal values of in different runs should not be interpreted as identifying the same learned functional object.
Measurements followed Regime A on the angular grid. For each selected frequency and epoch, we record the full-grid median , the fraction of points with , and the median dilatation restricted to that orientation-preserving locus (Table D1). When this locus is empty, median is undefined rather than zero. The complete three-panel comparison is shown in Figure D1.

Figure D1. Geometric measurements from six independently trained seeds. Each run contributes its five highest-embedding-Fourier-norm frequencies, selected independently of the other seeds. Filled and hollow markers denote epochs 40,000 and 40,100; paired observations are joined. Top: median on the Regime A grid; the blue band marks and the dashed line marks . Bottom left: median on (blue band: 1.5–2.6); the gray strip identifies cases with undefined . Bottom right: counts per seed under five nested, two-epoch acceptance bands, from strict to permissive. Their exact definitions and aggregate totals (20–27 of 30) appear in Section D.3.
Table D1: Complete per-seed directional Beltrami measurements. Five independently selected frequencies per seed, evaluated at epochs 40,000 and 40,100 (Regime A, grid). The column is the median on ; is the fraction of grid points in that locus.
| Seed | Epoch 40,000 Med. | Epoch 40,000 Med. | Epoch 40,000 | Epoch 40,100 Med. | Epoch 40,100 Med. | Epoch 40,100 | |
| 1 | 3 | 1.0010 | 221 | 0.4961 | 0.9982 | 830 | 0.7539 |
| 6 | 0.2312 | 1.546 | 0.8828 | 0.2224 | 1.397 | 0.8711 | |
| 23 | 0.4983 | 2.816 | 0.8164 | 0.4584 | 2.356 | 0.7305 | |
| 35 | 0.9076 | 20.472 | 0.9688 | 1.1050 | — | 0.0000 | |
| 39 | 0.3192 | 1.936 | 0.9883 | 0.3544 | 1.969 | 0.9492 | |
| 2 | 6 | 0.3488 | 2.069 | 0.9844 | 0.3433 | 2.044 | 0.9883 |
| 7 | 0.4666 | 2.366 | 0.8125 | 0.4728 | 2.345 | 0.8086 | |
| 15 | 0.3251 | 1.954 | 0.9922 | 0.3258 | 1.954 | 0.9922 | |
| 27 | 0.3570 | 1.829 | 0.8086 | 0.3588 | 1.828 | 0.8086 | |
| 38 | 0.3943 | 1.939 | 0.8477 | 0.3944 | 1.988 | 0.8516 | |
| 3 | 9 | 0.3750 | 2.199 | 0.9961 | 0.3865 | 2.228 | 0.9805 |
| 18 | 0.9901 | 6.869 | 0.5078 | 0.8051 | 6.928 | 0.6328 | |
| 27 | 0.3854 | 1.830 | 0.8203 | 0.3515 | 1.824 | 0.8555 | |
| 53 | 0.3690 | 1.871 | 0.8398 | 0.3979 | 1.872 | 0.8516 | |
| 55 | 0.3929 | 1.966 | 0.8984 | 0.3486 | 1.862 | 0.8711 | |
| 4 | 11 | 0.4184 | 2.120 | 0.8984 | 0.3876 | 2.050 | 0.8945 |
| 25 | 0.3590 | 1.910 | 0.8281 | 0.3200 | 1.713 | 0.8516 | |
| 32 | 0.3614 | 2.132 | 1.0000 | 0.3515 | 2.084 | 1.0000 | |
| 40 | 0.3517 | 1.667 | 0.8398 | 0.3344 | 1.683 | 0.8594 | |
| 49 | 0.3635 | 2.142 | 1.0000 | 0.3603 | 2.126 | 1.0000 | |
| 5 | 9 | 0.3419 | 2.039 | 0.9961 | 0.3416 | 2.037 | 0.9961 |
| 34 | 0.3844 | 1.834 | 0.7891 | 0.3758 | 1.837 | 0.7891 | |
| 37 | 0.7122 | 3.943 | 0.7227 | 0.7442 | 4.300 | 0.7266 | |
| 38 | 0.3517 | 2.032 | 0.9766 | 0.3502 | 2.057 | 0.9844 | |
| 55 | 0.3222 | 1.760 | 0.8047 | 0.3222 | 1.745 | 0.7969 | |
| 6 | 7 | 0.3300 | 1.492 | 0.7812 | 0.3314 | 1.521 | 0.7852 |
| 11 | 0.3045 | 1.645 | 0.8477 | 0.2971 | 1.689 | 0.8477 | |
| 35 | 0.3246 | 1.961 | 1.0000 | 0.3261 | 1.963 | 0.9727 | |
| 41 | 0.4014 | 2.028 | 0.8203 | 0.4020 | 1.982 | 0.8125 | |
| 43 | 1.7480 | — | 0.0000 | 1.9400 | — | 0.0000 |
Table D1. Notes. A dash denotes undefined median because no grid point has . Fractions are computed from 256 grid points. The five frequency indices were selected separately for each training seed and are not comparable across seeds. Full-pair forward-pass accuracy (epochs 40,000 / 40,100): seed 1, 99.4675% / 99.7416%; seed 2, 99.9843% / 99.9843%; seed 3, 99.9530% / 99.9687%; seeds 4–6, 100% / 100%. Four runs were measured in one environment, and two were contributed externally without their checkpoints. The latter checksums were reported but not independently recomputed.
Concentration and departures from the central range
At epoch 40,000, 22 of the 30 frequency-level medians fall in ; one is below 0.30, three are in , and four exceed 0.90 (Table D1; Figure D1, top). The central concentration recurs across retrainings, but it is not universal: for example, seed 6 at has median and 1.9400 at the two checkpoints and no orientation-preserving grid point at either epoch. For seed 1, changes from 0.9076 to 1.1050, so its median is undefined at epoch 40,100. These cases are retained in both the visualization and the complete table instead of being censored from the replication summary.
The two checkpoints are only 100 epochs apart, and Figure D1 depicts their paired variation rather than treating them as independent observations. Moreover, a frequency can meet a criterion at epoch 40,000 without meeting it at both epochs. This distinction is necessary when interpreting the band counts below.
Sensitivity to the replication criterion
The 22/30 value in § 3.2 is a single-epoch count of medians in and does not restrict . The bottom-right panel of Figure D1 instead applies each specified criterion at both epochs 40,000 and 40,100 to the same thirty frequency-level records. With medians and ranges inclusive of their endpoints, the five criteria are:
Strict: med and : 20/30.
only: med with unrestricted: 21/30.
±0.05: med and : 23/30.
Tolerant: med and : 25/30.
only: med with unrestricted: 27/30.
These totals range from 20 to 27 of 30 because they answer different numerical questions; they do not contradict the 22/30 single-epoch observation. The six-run dataset does not by itself warrant a population-level significance claim, and the observed concentration is not evidence that lower distortion is unique to functional key modes relative to all alternative directions (see Appendix § 8).
Data provenance and scope of the replication
Four runs were trained and measured in one computational environment. Two additional runs were contributed with their measurement files but without the checkpoints; their SHA-256 checksums were reported but could not be recomputed from the received materials. This limits independent checkpoint-level verification for those two seeds, without changing the tabulated measurements. Training accuracy reached 100% for all runs; the separate full-pair forward-pass accuracies at the two reported epochs appear in the notes to Table D1. The experiment supports recurrence of the distributional pattern across these six retrainings, not a claim of identical geometry at every key frequency or of inferential certainty from six seeds.
Training Dynamics and Behavioral Validation
Section 3.3 reports two tests that delimit what directional Beltrami distortion can reveal about the trained transformer: its evolution relative to Fourier-circuit formation and its association with behavior on individual input pairs. This appendix specifies the checkpoint and comparison rules, presents the complete quantitative summaries, and separates evidence of temporal ordering or spatial organization from evidence of a predictive or causal link to the learned computation.
Checkpoints, geometric change, and comparison windows
We measured the median directional coefficient on the Regime A angular grid for each of the five key frequencies, using all 500 saved checkpoints from epoch 0 to 49,900 in steps of 100. The normalization in the temporal comparison uses epoch 40,000, the trained checkpoint evaluated in the main results, rather than the final saved epoch. For a checkpoint , the signed fraction of initial-to-final geometric change is
The ratio is not clipped to : it can be negative if distortion initially increases, or exceed one if it passes its epoch-40,000 level. Its interpretation depends on a non-negligible denominator.
The reference epochs were determined from the loss curves before the Beltrami trajectories were evaluated. Memorization was defined by the first training-loss crossing below (epoch 476); the onset and end of the test-loss generalization transition (epochs 2,390 and 9,830) were defined by a 5% and 95% decline, respectively, in the test-loss range measured from its post-memorization maximum at epoch 1,420 (test loss recorded every 10 epochs). Because checkpoints were saved every 100 epochs, epoch 2,300 is the last saved checkpoint strictly preceding the primary onset at 2,390. The declared geometric criterion required for at least four of the five key frequencies before that boundary. A second, earlier cutoff at epoch 500 was fixed as a sensitivity check. Epoch 2,400, which is just after the primary boundary, is reported separately to match the comparison in § 3.3 and Figure 4.
Checkpoint and script provenance. The source model is the publicly released full_run_data.pth archive (456,428,905 bytes; SHA-256 a151f02125a33d6647d7b27fce99ae145ff3b83abc0586a883acc73e8318d828). Its epoch-40,000 state was verified against all ordered input pairs, yielding 100% forward-pass accuracy and a mean cross-entropy of , as documented in the original experimental verification.
The original scripts for the circuit-variance and behavioral-margin analyses were not retained. The analysis scripts circuito.py and puente.py are reconstructions, not the original implementations; their recorded results reproduce the circuit fractions (0.0277 at epoch 2,400 and 0.9690 at epoch 40,000), the five reported Spearman coefficients, and test accuracy 0.509789. The archived numerical records used to construct Tables E1 and E2 are serie500.json, circuito500.json, and puente.json; no interpolation was applied. The maximum over additional configurations is documented in the experimental methods note rather than the primary puente.json summary.
Directional distortion before Fourier-circuit formation
The Fourier-circuit metric measures the fraction of pair-centered logit variance explained by the five key frequencies. For each of the ordered input pairs , the mean over the 113 output logits is first subtracted. Each resulting output-logit series is then centered across all input pairs and projected onto the ten orthonormalized directions and , for and . The metric is the sum of squared norms of these projections, over all output logits, divided by the sum of squared norms of the corresponding centered logit series.
Table E1 reports the frequency-level measurements and the Fourier-circuit metric at matched checkpoints. At epoch 2,300, the signed fractions exceed 0.60 for all five frequencies under the declared numerical rule. At epoch 500, four of five exceed the threshold: has , indicating a small increase rather than an accumulated reduction at that earlier checkpoint. For the five frequencies with a non-degenerate initial-to-final difference, the fractions at epoch 2,400 range from 0.745 to 0.966. They therefore substantially precede the main rise in the measured circuit metric.
| 14 | 0.4758 | 0.3468 | −0.102 | 0.746 | 0.745 |
| 35 | 1.1289 | 0.3902 | 0.859 | 0.963 | 0.966 |
| 41 | 2.2779 | 0.3036 | 0.957 | 0.950 | 0.948 |
| 42 | 0.9394 | 0.4455 | 0.693 | 0.777 | 0.773 |
| 52 | 0.4035 | 0.4001 | |||
| Epoch | Explained logit variance | ||||
| 0 | |||||
| 500 | 0.00963 | ||||
| 2,300 | 0.02669 | ||||
| 2,400 | 0.02775 | ||||
| 40,000 | 0.96902 |
Table E1. Geometric change and Fourier-circuit formation at reference checkpoints. () on the Regime A grid, and signed fractions defined in Eq. (30). Epoch 2,300 is the last saved checkpoint strictly before generalization onset (epoch 2,390); epoch 500 is the sensitivity cutoff, and epoch 2,400 is the nearest saved checkpoint afterward. () Fraction of pair-centered logit variance explained by the five key frequencies at the same checkpoints.
Notes. Under the primary cutoff, for 5/5 frequencies (4/4 after excluding ); at epoch 500 the count is 4/5 (3/4 without ). †For , , small relative to its observed training variation, so the reported ratios are numerically defined but not informative effect sizes. No fractions are truncated to [0, 1].
At epoch 2,400, the Fourier-circuit metric is 0.02775, compared with 0.96902 at epoch 40,000, or approximately 2.9% of the latter. Its much later increase is also apparent in Figure 4B; by contrast, the individual Beltrami trajectories are not uniformly monotonic during training (Figure 4A). The numerical criterion has an important limitation: the initial-to-final difference for is only 0.00345951, while the frequency’s median varies considerably across the training trajectory. Its very large fractions therefore pass the threshold mechanically rather than providing interpretable evidence of early change. Excluding preserves the primary count (4/4) but reduces the sensitivity count to 3/4. These comparisons document temporal ordering in this run; they neither identify what causes the geometric change nor show that the measured coefficient is used by the Fourier circuit.
Pairwise association with behavioral margins
The second test asks whether directional distortion predicts the correct-logit margin for an individual input pair. We chose the checkpoint by the previously fixed rule of test accuracy closest to 50%, without using the Beltrami values to select it. This gives epoch 9,400, with test accuracy 0.509789 and training accuracy 1.0000. The analysis uses all 8,939 held-out pairs, with the actual input pair as the base point for the directional evaluation, and no excluded observations. The association measure is Spearman’s rank correlation between and the correct-logit margin; the predeclared threshold for a substantively meaningful effect was .
| Spearman | Null 95th | Permutation | Exceeds null? | |
| 14 | −0.0260 | 0.0206 | 0.010 | Yes |
| 35 | −0.0146 | 0.0208 | 0.166 | No |
| 41 | −0.0070 | 0.0212 | 0.507 | No |
| 42 | −0.0224 | 0.0196 | 0.026 | Yes |
| 52 | −0.0058 | 0.0208 | 0.582 | No |
Table E2. Association of directional Beltrami magnitude with correct-logit margin. Spearman correlations for the five key frequencies at epoch 9,400 (8,939 held-out pairs; no exclusions). The null threshold is the frequency-specific 95th percentile of over 1,000 label permutations; the permutation -values are two-sided and unadjusted across frequencies.
Notes. Two frequencies ( and ) exceed their individual permutation-null 95th percentiles; no family-wise significance claim is made. None approaches the declared effect-size threshold. The non-geometric embedding-norm baseline, , yields against the same margin.
The maximum frequency-level correlation is . The two unadjusted, frequency-specific null crossings ( and ) are far below the declared effect-size threshold. The non-geometric embedding-norm baseline () is also small, but exceeds the magnitude of every Beltrami correlation. Three sensitivity analyses—epoch-9,900 test pairs, epoch-9,400 training pairs, and a full-logit variant—accompany the primary epoch-9,400 test-set analysis. Across these four configurations, the reported maximum is . Only that aggregate maximum, not the additional per-frequency estimates, is documented. Under the tested conditions, is not a useful predictor of pairwise correct-logit margins.
References
- [1]Ahlfors, L. V. (2006). Lectures on Quasiconformal Mappings (2nd ed.). American Mathematical Society. https://doi.org/10.1090/ulect/038.
- [2]Brandon, J., Chadwick, A., and Pellegrino, A. (2026). Emergent Riemannian geometry over learning discrete computations on continuous manifolds. Proceedings of Machine Learning Research, 282, 49–72. https://proceedings.mlr.press/v282/brandon26a.html.
- [3]Chen, Q., Li, Z., and Lui, L. M. (2021). A deep learning framework for diffeomorphic mapping problems via quasi-conformal geometry applied to imaging. arXiv:2110.10580. https://arxiv.org/abs/2110.10580.
- [4]Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. (2025). Not all language model features are one-dimensionally linear. International Conference on Learning Representations. https://arxiv.org/abs/2405.14860.
- [5]Gurnee, W., Ameisen, E., Kauvar, I., Targ, J., Pearce, A., Olah, C., and Batson, J. (2025). When models manipulate manifolds: The geometry of a counting task. Transformer Circuits. https://transformer-circuits.pub/2025/linebreaks/.arxiv.org/abs/2601.04480
- [6]Moisescu-Pareja, G., McCracken, G., Wiltzer, H., Létourneau, V., Daniels, C., Precup, D., and Love, J. (2026). On the geometry and topology of representations: The manifolds of modular addition. International Conference on Learning Representations. https://arxiv.org/abs/2512.25060.
- [7]Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. International Conference on Learning Representations. https://arxiv.org/abs/2301.05217.
- [8]Peterfreund, E., Lindenbaum, O., Dietrich, F., Bertalan, T., Gavish, M., Kevrekidis, I. G., and Coifman, R. R. (2020). Local conformal autoencoder for standardized data coordinates. Proceedings of the National Academy of Sciences, 117(49), 30918–30927. https://doi.org/10.1073/pnas.2014627117.
- [9]Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177. https://arxiv.org/abs/2201.02177.
- [10]Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J. (2023). The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2306.17844.



