WorksPhysics and Astronomy
Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking
Abstract
Learning in a neural network takes place on a high dimensional, non-convex loss landscape, the Hessian curvature of which dictates the learning dynamics. Despite its high dimensionality, the large number of architectural symmetries explains some of the geometric of this landscape, in particular flat directions, i.e. near-zero modes of the Hessian. These architectural symmetries have therefore been suggested as an organizing principle of deep learning. It remains an open question, though, wether there are additional organizing principles in the remaining degrees of freedom in the learning dynamic. Here we identify task-dependent contingent symmetries as such an additional organizing principle. We show that non-flat learning directions emerge due to explicit breaking of these contingent symmetries. We isolate this symmetry in an analytically solvable minimal model. Furthermore, we show that, by tuning an L2 penalty, signatures of the symmetry breaking become measurable. As a consequence of the same mechanism, adding noise to the data drives the model through a sequence of broken symmetry recoveries in the form of Baik-Ben-Arous-Péché (BBP) transitions. We demonstrate analytically for linear and numerically for deep linear networks that the contingent symmetry group remains explicit, and that the symmetry-breaking mechanism identifies which Hessian modes carry task-specific information. These signatures are measurable without a symmetry-group description and, hence, offer a route to study symmetry breaking as a more general principle in nonlinear networks and other learning systems.
Introduction
Understanding how neural networks learn and generalize is a central problem in machine learning, statistics, and physics. Research approaches this problem through optimization, statistical inference, and the geometry of high-dimensional loss landscapes. Statistical physics provides complementary concepts, including symmetries, fluctuations, and phase transitions. A common challenge is to connect task structure with the behavior of the learning system. Here we examine how task constraints restrict model freedom and when the resulting curvature in the loss landscape represents statistical information. Deep linear networks trained from small initialization acquire dominant singular directions on different timescales [21]. In nonlinear networks, generalization can improve abruptly after a long period of apparently successful fitting [19, 12]. Physical learning systems also develop structured responses as they adapt: their task performance and mechanical or electrical response become linked through changes in the relevant curvature operators [22, 23, 24]. A tractable model can isolate each feature constraint, the freedom it removes, and the resulting curvature. How to connect task structure with the behavior of the learning system is addressed by different communities using different methodologies. These include information and differential geometry, singular learning theory, statistical physics, theory of criticality and random matrix theory (RMT). Our work builds on all of these to some degree. We summarize the key findings, relevant for this work, in the following section.
Prior work
One line of work investigated the geometry of DNN error landscapes by varying the strength of an penalty. Ziyin and Ueda [27] derived an onset of learning as the penalty decreases: the transition is continuous with one hidden layer and discontinuous at greater depth. Ladewig et al. [14] showed analytically that in deep linear networks the onset is only the first of a cascade of transitions, each associated with a singular direction of the data. In nonlinear networks, Ersoy and Wiesner [11] identified further transitions at lower strengths, related them to changes in error-landscape curvature, and associated each phase with distinguishable aspects of the task, which they demonstrated both for synthetic and for real data (Ersoy et al., 2026). In the same framework, metastability and noise-driven escape from local minima have also been investigated and found to be a possible mechanism for delayed generalization in nonlinear DNNs [12]. Taken together, these results reveal a sequence of learning transitions, each associated with a geometric location and a Hessian spectrum signature.
A second line of work examined Hessian spectra of error landscapes at points of changing curvature directly. Empirical studies reported many small Hessian eigenvalues together with a smaller set of large eigenvalues [20]; random-matrix approaches explained aspects of the spectral structure and its dependence on the task and the parameter settings [18, 8]. In spiked random-matrix models, the Baik-Ben-Arous-Péché (BBP) transition and related results described when signal-associated outliers separate from the bulk and when their eigenvectors correlate with the underlying signal [4, 5]. Information geometry supplies a complementary local description through Fisher information [2, 16]. Kühn and Rosenow (2026) explain near-zero Hessian eigenvalues through the weak explicit breaking of architectural symmetries by nonlinearities and analyze the corresponding Fisher geometry.
A third line of work studies symmetries in neural-network parameter space and their consequences for learning, reviewed by Zhao et al. [30]. Ziyin [26] proved that mirror symmetries define fixed subspaces preserved by gradient descent and give the Hessian a corresponding block structure at symmetric configurations. Under additional assumptions, sufficiently strong weight decay favors minima in these subspaces; when the symmetry preserves the observation distribution, Fisher information is singular at the corresponding symmetric configurations. Symmetric configurations also reduce the expressivity of the linearized model and the dimension accessible to training [29]. In a recent position paper, Ziyin et al. [28] proposed symmetry breaking and restoration as organizing principles for hierarchical learning. Architectural symmetries hold independently of the data, whereas symmetries of a loss can depend on the task. In deep linear networks, architectural reparametrizations have a tractable geometry [17], and an penalty selects balanced parametrizations of a fixed function in the full-rank square case [15]. The contingent freedoms studied here preserve an auxiliary partial task but need not preserve the full objective.
Our Contribution
We ask: How does imposing a task feature remove a task-preserving freedom and create a positive Fisher mode, and how does regularization affect this relation? Answering this requires distinguishing a freedom left open by a partial task from a symmetry of a fixed objective and from a degeneracy at a particular parameter setting. It also requires separating the curvature of the data error from that of the penalized objective. Our contributions are fivefold. (i) We distinguish contingent symmetries, the freedoms a partially specified task leaves open, from architectural symmetries and task invariances, and introduce the feature filtration to isolate them. (ii) At a common regular exact solution, we show that imposing one independent feature constraint removes one such freedom and increases the Fisher rank by one, while architectural freedom survives. (iii) We show that an penalty can make reparametrization directions massive while they remain null directions of the Fisher metric: these penalty-lifted directions do not correspond to learning a feature. (iv) We combine the regularization cascade with spectral recovery thresholds in a learnability phase diagram over regularization strength and BBP noise amplitude, whose regions have the feature-subset labels of the filtration; for a uniform penalty the two probes order the features identically, while unequal penalty weights make the orderings compete and the boundaries cross. (v) We state the observables by which the account can be tested in networks whose symmetry group is unknown. § 2 defines the framework and filtration. § 3 develops the minimal example and its spectral interpretation. § 4 examines regularization and statistical recovery as probes of the resulting structure. § 5 extends the analysis to deep linear networks, and § 6 discusses nonlinear networks, other learning systems, and open questions.
Conceptual framework and method
The principle investigated here is that learning a feature restricts the choices left open by a task. To examine this idea, we need to distinguish the types of degrees of freedom and specify how a feature enters the analysis. We construct a sequence of partial tasks and compare their admissible solutions, symmetries, error curvature, and statistical distinguishability on the parameter space. This section defines the construction and the quantities to be compared. The relation between them is worked out in the minimal model of § 3.
Architectural, task, and contingent symmetries
Let be the model family. A configuration specifies the realized function ; the task specifies the requirements on that function. When a probability distribution is intended, we refer explicitly to the observation model introduced below. Consequently, two parameter settings can be equivalent for different reasons. We distinguish three sources of equivalence by what remains unchanged and by the conditions under which the equivalence holds.
Architectural symmetries. These are reparametrizations that leave the realized function unchanged:
They follow from the parametrization and hold independently of the task. For a two-layer linear network , the coordinates of the intermediate vector are its hidden units and is its hidden width. With end-to-end map , the reparametrizations include
Hidden-unit permutations are another example; homogeneous activations also permit suitable rescalings. The available transformations depend on the architecture. The fibers of a factorized linear network and their group actions provide a rigorous setting for this redundancy [17, 15].
Data and task symmetries. These express distinctions that the fixed task does not require the model to resolve. For example, in a classification task an input transformation may preserve the assigned class label. Such an invariance is specified by the data distribution and the objective; it need not be a symmetry of an arbitrary model’s predictions. To induce a symmetry of the learning objective, the corresponding transformation must preserve the loss and the relevant data measure. Its existence alone therefore does not establish a flat direction in parameter space. It remains part of the task when that task is fully specified.
Contingent symmetries. These preserve the features already imposed while allowing other aspects of the realized function to change. Their defining property is that a further feature constraint can distinguish configurations that were previously equivalent. They are the contingent symmetries investigated here. In linear networks some admit descriptions through transformations of both model and data coordinates. Their essential content is the preservation of a partial task. They can change the represented function and need not survive a further feature constraint. This distinction concerns the origin of the equivalence, rather than three necessarily disjoint groups. A transformation can preserve both the architecture’s function and every partial task. We therefore identify the architectural freedom first and then ask what additional freedom the partial task leaves available.
Task descriptions as coarse-grainings. Let be the set of realizable functions, and define the realization map by . A partial task retains scalar feature values , each depending only on the realized function. Its feature map is therefore well defined by
For prescribed target values , the scalar error is obtained through
These maps give descriptions containing no more detail than their inputs (Figure 1). Architectural transformations satisfy . Contingent transformations preserve while potentially changing the realized function. The maps perform the coarse-graining; the transformations preserve the descriptions retained by those maps. Their orbits lie within the corresponding fibers, where a fiber is the set of points having the same image. A fiber need not be a single symmetry orbit. The final scalar error discards further distinctions: equal error does not generally imply equal feature values.

Figure 1. Successive descriptions of a model configuration. The realization map sends a parameter configuration to its input-output function. Architectural symmetries change the parameters while leaving this function unchanged. The map retains only the feature values specified by the partial task. Contingent symmetries can change the function while preserving these values; their parameter-space orbits lie within fibers of . The final map compares the retained values with their targets and summarizes the discrepancies by a scalar error. Equal scalar errors need not correspond to equal feature values. Adding a feature replaces by the finer description , allowing some previously equivalent functions to be distinguished.
Feature filtration as an analytical construction
In the tractable construction, the task consists of features: components of the target that the model has to represent, with prescribed strengths , . We use the word only in this sense. The model’s coefficient along feature is a scalar functional ; imposing feature means requiring , and the model represents feature when . Each depends on the realized function, so architectural reparametrizations leave it unchanged. The conditions express what it means to represent the corresponding features correctly. For an aligned linear model, the desired linear input-output map and the represented map use the same fixed orthonormal input and output directions. Then . This is what we mean by alignment; no such decomposition is assumed for every learning problem. Choose an ordering of the constraints and define the auxiliary errors
with . The full task imposes all constraints; a partial task imposes only the specified subset, here the first . An exact fit of that partial task satisfies every retained equality, so .
Assume that the complete set is jointly realizable. Its partial-task solution sets, equivalently the global minimum sets of , are then
This nesting corresponds to a refinement of task descriptions. Let discard the last feature value. Then
For , the retained description is the single empty tuple. Functions with identical first feature values therefore have identical first values, whereas the converse need not hold. Discarding a feature coarse-grains the description; adding a feature refines it. The exact-solution sets are its target fibers,
This nesting is part of the construction: each partial task retains the constraints already imposed and adds one further requirement. Once the coefficient functionals, their target values, and their ordering are fixed, the chain is determined. Different orderings can give different intermediate sets. These are constraint sets, not generally vector subspaces or nested sets of exactly different ranks. The remaining coefficient functionals are absent from ; their values may vary to the extent allowed by the architecture and the retained constraints. Whether these choices admit a group description, and what changes when the next constraint is imposed, are questions for the subsequent analysis. The level means that no feature constraint has yet been imposed. It does not mean that every parameter setting realizes the zero function. Similarly, an unconstrained feature need not have a zero coefficient. This distinction separates the auxiliary partial task from a model that has failed to fit part of a fixed, fully specified task. The filtration is an analytical device for isolating the effect of a feature constraint. Its ordering does not prescribe the arrival of data or the trajectory of an optimizer. Whether regularization or statistical recovery produces a corresponding sequence is examined separately in the later sections.
Symmetry, curvature, and information
For each partial task we examine which transformations preserve its features, how the error responds to parameter perturbations, and which perturbations change the associated observation distribution. These are distinct descriptions, whose relationship must be established for the model under consideration. To specify the symmetry question, fix a common group of admissible invertible transformations of parameter space and define
The orbit is the set of configurations reached by these transformations from . When the action is smooth, a one-parameter subgroup has generator . The analysis will determine the groups and their orbits, and compare the independent tangent directions generated at a solution. We do not assume that an orbit is the entire minimum set or that every generator produces a nonzero tangent direction. We use explicit symmetry breaking for a change in the invariance of the objective and spontaneous symmetry breaking for the selection of a configuration with less symmetry than an unchanged objective [13, 1]. In the filtration, adding a feature constraint explicitly reduces the contingent group, , because the objective changes from to . A one-parameter action that leaves a configuration unchanged has and supplies no nonzero orbit tangent there. The distinct symmetry effects of regularization are introduced with the penalized objective in § 4.1. For the curvature calculation, define
At a stationary point, for a unit parameter variation is the directional curvature of the error . We call its zero-eigenvalue modes flat and its positive-eigenvalue modes massive. In the regularized sections we distinguish from . Fisher information is a separate quadratic form. A Hessian zero mode need not arise from an exact invariance. We also use the coefficient Jacobian to describe which coefficient values change under an infinitesimal parameter variation. The relation between this Jacobian, symmetry generators, and the Hessian is derived in § 3. To specify the information calculation, associate the partial task with independent Gaussian observations
with fixed variance. Let denote their joint density. The Fisher information is
It measures the local statistical distinguishability of nearby parameter settings within the specified observation model [7, 2, 16]. We call the pullback quadratic form the Fisher metric; it may be degenerate in redundant parameter coordinates. We compare its null and positive directions with those of the error Hessian. Their agreement is to be established under the stated conditions; it is not part of the definition. The distinction matters once a penalty is added: its curvature can make a direction massive without introducing additional statistical distinguishability.
Assumptions and analysis strategy
The exact local analysis assumes jointly realizable constraints, twice continuously differentiable coefficient functionals, and regular exact solutions: points of at which , so the retained constraint differentials are independent. Smooth group actions are used where symmetry generators are calculated. The Gaussian observation model fixes the statistical meaning of the Fisher calculation. Singular configurations, dependent constraints, and settings without an explicit group description require separate treatment. § 3 applies this construction to a two-feature factorized model and computes its minimum sets, symmetry actions, and curvature and information modes. We then keep the task fixed and examine how an penalty competes with its feature requirements (§ 4.1) and which curvature it adds without information (§ 4.1 and § 5.2). The BBP connection examines how finite noisy data resolve feature directions to different degrees, providing a statistical counterpart to exact feature inclusion (§ 4.2), and the two probes combine into a learnability phase diagram (Sec. 4.3). The later extensions distinguish the results established for linear networks from the proposed implications for nonlinear networks and other learning systems.
Explicit symmetry breaking produces positive Fisher information
We now realize the construction of § 2 in a model with four parameters and two features. Its purpose is to make the same event visible in three descriptions: each added feature constraint reduces the minimum set, removes a task-preserving freedom, and creates a positive Hessian and Fisher mode. The architectural freedom to represent a fixed function remains throughout. For the derivation of the explicit symmetry breaking when features are introduced and the resulting flat-to-massive transition in the spectra of the error Hessian and the Fisher metric, we use the filtration scheme introduced in § 2.2.
A minimal model and the partial tasks
The minimal model has four parameters and two coefficients, one per feature. Its realized predictor is , with
Take targets . Thus and . For whitened inputs, , the latter is also the population error against the noiseless target : . Here population error means an expectation over the input distribution, rather than an average over a finite sample. The diagonal factorization excludes hidden-unit mixing; the full matrix model is introduced in § 5. The architectural symmetry (2) reduces here to
under which is unchanged. For nonzero products its orbits are the hyperbolas in the plane, all points on a given hyperbola have the same coefficient : is constant along the curve and the Fisher metric has a null direction tangent to it. At an exact fit of an imposed feature, the transverse direction changes and has positive error curvature. Writing makes the pair explicit,
and , so the gauge tangent and the normal are orthogonal. They are respectively Hessian-flat and massive on the nonzero exact-fit hyperbola of an imposed feature; loss invariance along a curved orbit alone does not imply a zero ordinary Hessian eigenvalue away from stationarity. The two directions distinguish reparametrization along a fixed-product orbit from a change of the coefficient . The contingent symmetry appears while a feature is unconstrained. Before the second constraint is imposed, nothing in the objective depends on : at an exact fit of feature 1, every value of is compatible with the partial task, so varying it leaves . Once is imposed, it does. Explicitly, let consist of the independent rescalings , . The groups (5) are then
so , with the architectural symmetry (10) of both features. Through a point of with all , the -orbit has dimension , that of : it fills the solution set locally, and its tangents are exactly the null directions of the Hessian at that exact solution. The factor lost in the step , the rescaling of by , is the contingent symmetry that imposing the second feature breaks. The transition from one state to the other is the elementary breaking, and Figure 2 shows both sides of it.
Figure 2. Imposing features one at a time. Each row shows the loss with targets , in the factor coordinates (left) and (centre), and in the coefficient coordinates (right). Top row: only feature 1 is imposed, . Bottom row: both features are imposed, . Colours show the error on a shared logarithmic scale. A grey panel marks a feature that is not imposed, so the loss does not depend on its factors.
Factor panels: for an imposed feature, the dashed white hyperbola is the set of exact minima, and the white dot is the balanced representative . The arrow is normal to the hyperbola and changes ; its Hessian eigenvalue is . The arrow is tangent to the gauge orbit , which leaves unchanged, so . For a feature that is not imposed, both directions are unconstrained (purple), the dashed grey hyperbola is shown only for reference, and the black dot marks an arbitrary configuration.
Coefficient panels: the star marks the target and the open circle the represented coefficients . An imposed coefficient has curvature . A coefficient that is not imposed is a flat direction (purple double arrow), so is undetermined in the top row. With both features imposed, coincides with the target.
Reducing contingent symmetries generates a nested learning hierarchy
The contingent groups form a nested chain
maximal when no feature is imposed () and minimal when all are (). Imposing the -th feature is the step : the group loses transformations that change the new feature, and the set of realizations of the current accuracy, made up of orbits of equivalent parameter configurations, contracts to the set satisfying the additional constraint. Across the filtration this is explicit symmetry breaking in the sense of § 2.3: the objective itself loses invariance. Note that the modes are independent, so the ordering need not be unique. Instead of considering the models that capture feature 1 and feature 2 as a subset of the models that only capture feature 1, we can construct the nesting, starting with feature 2.
One independent constraint increases Hessian and Fisher rank by one
The added constraint detects a variation that previously left all retained coefficients unchanged. To count the resulting directions, consider a small parameter displacement : the coefficient Jacobian maps it to the first-order changes of the first coefficients. At a regular exact solution, the displacements that remain tangent to the solution set are therefore precisely those annihilated by ,
If the first constraints have independent differentials, the local dimension of this set is , and a further independent scalar constraint removes one tangent direction. At an exact solution of the first constraints,
To compare successive levels at the same point, take and set . Then
An old flat direction acquires curvature precisely when it changes the new feature to first order. If is independent of the preceding constraint gradients, the number of positive Hessian eigenvalues increases by one and the remaining tangent directions stay flat; the positive eigenvectors can rotate unless the constraint gradients are orthogonal. A continuous symmetry that preserves the imposed features has a generator with , hence at a fitted solution; after the additional constraint the same direction has curvature . For the Gaussian observations (7) the expected Fisher information is
so the curvature gained in (16) is a new direction of statistical distinguishability. In the minimal model the nonzero entries of the Jacobian row for feature is , so imposing it adds exactly the massive direction of (11) and leaves flat. Away from exact fit the error Hessian also contains residual-dependent terms, and a penalty contributes curvature of its own; neither can be identified with Fisher information (§ 4.1).
Feature-induced massive modes are emergent learning directions
The result of § 3.3 identifies the same newly constrained parameter variation in the error Hessian and in the observation distribution. For a Gaussian model with fixed noise scale, the mean-squared error is the negative log-likelihood up to an additive constant and a positive factor (Bishop, [7]): minimizing the loss is maximum-likelihood estimation. Consequently, the loss Hessian is proportional to the Hessian of the negative log-likelihood, whose expectation under the model is the Fisher-information matrix (Amari, [2]; Martens, [16]); the negative log-likelihood Hessian for the observed sample is the observed information, including when evaluated at an optimum (Efron and Hinkley, [9]). For the feature observations of Eq. (7), exact fit removes the residual term. For a correctly specified Gaussian population loss, fitting the conditional mean instead makes its expectation vanish. With matching sum or average conventions,
where is the noise variance. On locally identifiable statistical coordinates, Fisher information defines a Riemannian metric; its pullback to redundant parameters is generally singular (Amari, [2]): for a fixed parametrization, its large-eigenvalue directions are the parameter combinations most tightly constrained locally by the data, and its null directions have zero first-order statistical distinguishability. At singular points they may still be distinguishable at higher order. This turns the flat and massive directions of the filtration into statements about information. A symmetry direction is one along which the model distribution does not change, so by (18) it is a null direction of : the corresponding parameter combination is non-identifiable and carries zero information. Unbroken contingent and reparametrization symmetries give degenerate directions of the Fisher metric, the setting of singular learning theory [25]. In the minimal model at exact solutions with all factors nonzero they give all of them, Eq. (12); in deep linear networks they need not (§ 5.2). Conversely, a direction that becomes massive when a feature is imposed, Eq. (16), is a null direction of turning positive: that parameter variation now changes the observation distribution to first order, so its Fisher quadratic form is positive. This is local statistical distinguishability along a direction, not unique identification of the entire parameter vector. For an independent Gaussian channel , with and noise variance , the mutual information is . This is a different quantity from parameter Fisher information. It vanishes when the represented coefficient is zero, not merely when the corresponding constraint is absent. For a fixed teacher channel with coefficient , the input-output mutual information is fixed by the data distribution and does not turn on during training. We call the newly Fisher-positive direction in the auxiliary filtration an emergent learning direction: an independent constraint makes a previously task-flat variation locally distinguishable. Penalty-lifted architectural directions remain Fisher-null and therefore do not become learning directions in this sense (§ 4.1).
Detecting learning directions
So far the symmetry has been identified and the consequences of its breaking computed, but nothing has been said about when a breaking occurs. The chain (13) is a sequence of subgroups; it does not say what moves a system along it, nor whether every step in it is reachable. Two established phenomena answer these two questions, and in doing so give the principle empirical content. The first is the cascade of transitions driven by an penalty, which supplies a control parameter: it lets us compare which features are represented across penalized objectives and locate transitions along an ideal path of minimizers. The second is the detachment of a signal direction from a noise bulk, which supplies a precondition: in the specified spiked ensemble, spectral recovery of a population feature requires its signal to exceed the relevant threshold. This is a condition for that recovery mechanism, not a universal impossibility result for every learner. Within the stated model, they compare feature activation with statistical resolution. Because the penalty also contributes curvature of its own, we separate the modes it introduces from the learning directions (§ 4.1), and we close with a phase diagram in regularization strength and data-noise amplitude (Sec. 4.3) as a temperature analog. That both are measurable without reference to the group is what makes the principle testable outside the setting where it was derived, and we return to that in § 6.
The L2 cascade: regularization helps locate learning directions
The chain (13) is a sequence of subgroups but says nothing about when a system moves along it. A control parameter supplies that. We add an penalty of strength to the error of the full task, ,
so that in the minimal model an active minimum for sits at and is lost at the critical strength . A regularization sweep varies while keeping the data error fixed. Analytically, we compare global minimizers . In a training experiment, the optimizer, initialization, sweep rate, and relaxation time determine which minima are reached. The resulting transitions have been characterized. They are observed in general, nonlinear networks [11] (Ersoy et al. 2026), and noise-driven escape from metastable phases has been proposed as a mechanism for grokking in the models studied in [12]. In the aligned deep-linear setting, the transitions are continuous for one hidden layer and first order for two or more [14, 27]; depth alone does not establish this classification for arbitrary nonlinear architectures. In the aligned separable setting with sufficient width and distinct strengths, lowering activates features in descending order of ; increasing suppresses them in the reverse order (Appendix § 9). Along this path the data error is fixed, while the penalized objective changes and is not invariant under the full contingent group, so it does not implement the explicit contingent symmetry reduction of the filtration; it selects which features are represented. Its phase with features learned corresponds to level of the filtration: the same features, with active coefficients in this two-factor model. For each factor pair the penalty reduces the reciprocal-scaling symmetry to . This surviving sign symmetry, , is architectural because it preserves ; it is not the contingent freedom to change an unconstrained coefficient. A nonzero minimum spontaneously selects one of its two sign-related representatives: above the minimum is invariant under it, below it the two minima are exchanged by it. Each transition can therefore be labelled by the same added feature as a step of the auxiliary filtration (§ 3.2); it is not itself that explicit change of task symmetry. This is the first thing the cascade establishes about the principle: the feature ordering has a corresponding sequence of transitions in this solvable model, and the locations are computable in the solvable case and measurable in general. The second is that, for a uniform penalty, the ordering is forced rather than incidental. Because the then order the , the levels are reached in a definite sequence, and the continuous sweep of global minimizers cannot skip a feature threshold in this separable model, which is the content of § 5.3, seen from the side of the control parameter. We use quasi-static global-minimum path for this ideal comparison of global minimizers at successive . It need not be reproduced by slowly continuing one training run: following a local minimum can retain a metastable state and produce hysteresis. At finite Langevin temperature a trajectory can also visit both sign-related minima; the coefficient is invariant under this sign exchange. The penalty also lifts directions that remain Fisher-null, and the minimal model shows both kinds of mass side by side. At an active balanced minimum the reparametrization direction of Eq. (11) acquires curvature in (Figure 3), yet , so it remains a null direction of the Fisher metric: a pseudo-Goldstone mode of the symmetry that the penalty breaks explicitly. Before the feature is learned, at , the Hessian of has eigenvalues while the Fisher metric vanishes; the smaller one goes soft exactly at , and below it the direction becomes Fisher-positive. The penalty-lifted gauge mode is thus Fisher-null, whereas the normal of an active feature is Fisher-positive. This does not identify all penalty curvature with Fisher-null directions. Continuous hidden-unit rotations in the full matrix model add a symmetry-generated flat sector; the full classification is in § 5.2. Table 1 keeps the unregularized toy result, its penalized counterpart, and the full matrix calculation separate.
Figure 3. Regularization shrinks and ultimately suppresses each represented feature. Each panel shows in the factor plane , with all feature weights . The symbol in the panels denotes the regularization strength used in the text. The top row shows feature 1 () and the bottom row feature 2 (). Regularization increases from left to right, with in the top row and in the bottom row, as labelled in each panel. Colours show the loss above the panel’s minimum on a logarithmic scale shared by all panels, so dark regions lie close to a minimum. The black dashed hyperbola is the unregularized exact-fit set. For (left), the white dot marks the positive regularized minimum ; the sign-related minimum appears as the second dark well in the opposite quadrant. At the marked minimum, the normal (red or orange) has eigenvalue and positive Fisher norm. The gauge tangent (green) has eigenvalue but remains Fisher-null. At (middle), the two minima merge at the origin and one Hessian eigenvalue vanishes. For (right), the origin is the unique minimum and the feature coefficient is zero. Reading from right to left therefore illustrates continuous feature activation in this two-factor model.
| Setting | Symmetry or control | Hessian and Fisher statement |
| Bilinear partial tasks (§ 3) | An added constraint reduces contingent freedom; reciprocal rescalings survive. | At a common regular exact fit, ; each independent scalar constraint increases both ranks by one. |
| Bilinear (§ 4.1) | The full task is fixed. Positive leaves the architectural sign symmetry per feature. | At an active minimum, the gauge tangent has eigenvalue and zero Fisher norm. At the origin, onset eigenvalues are while Fisher information is zero. |
| Full two-factor matrix (§ 5.2) | The architectural group retained by the norm penalty is . Hidden-unit mixing is allowed. | Nonzero orbit tangents give flat modes of . The four-sector formulas apply under the stated one-hidden-layer assumptions. |
Table 1. The three settings and the curvature statements established in each.
Noise tuning: BBP transitions give a smooth filtration
The second detector asks a different question. A spectral estimator can recover a population feature only when its direction remains resolvable against the fluctuations of the empirical cross-covariance. The threshold below concerns this estimator in a specified large-matrix ensemble. Random matrix theory makes this sharp. For a linear network trained with squared loss on empirically whitened inputs, the parameter-dependent part of the loss depends on the targets through the empirical input–output cross-covariance , an matrix estimated from samples. For an empirically whitened fixed design , with , and labels with independent Gaussian noise entries of variance , the conditional cross-covariance is a spiked matrix,
Here the target map specifies the conditional mean ; it equals the population cross-covariance when . Without empirical whitening, the signal term is and the conditional noise covariance also depends on . The signal map has fixed low rank, its nonzero singular values are distinct feature strengths , and its singular vectors specify the feature directions.
The noise is the fluctuation that finitely many noisy samples add: has independent zero-mean entries of variance , and the amplitude grows with the label noise and falls with the number of samples. The results below hold for large dimensions at fixed aspect ratio ; we restrict the formulas to this variance and aspect-ratio convention. Without a signal, the singular values of fill a bulk with upper edge . A spike of strength separates from it only when its signal-to-noise ratio exceeds
the BBP transition [4, 6]. Below the threshold the recovered singular vectors are asymptotically uncorrelated with the true ones; above it, let be the unit empirical left singular vector associated with spike and its true output direction. Their squared overlap is the squared cosine of their angle and converges to
rising from zero at threshold toward one [6]. The corresponding input overlap converges to . The two coincide for a square matrix, , the case used in Sec. 4.3, where feature stops being resolvable at . Here controls data noise and therefore the spectral resolution of a population feature. It differs from a dynamics-side temperature or , which controls fluctuations of the parameters on a given landscape. Vanishing overlap means that the empirical singular vector is asymptotically uncorrelated with the population feature under these assumptions; it does not imply that every estimator fails or that the data Fisher matrix vanishes. The connection to § 3 is therefore an effective modelling choice: in Sec. 4.3 we use as the weight of the corresponding feature error. This provides a soft counterpart of the exact filtration. It is not an identity between spectral overlap and Fisher information. For a uniform penalty, the ordered give the same ordering of the resolution and activation thresholds in this effective model; unequal penalty weights can make the orderings compete.
The beta-T phase diagram
The parameter is the data-noise amplitude in Eq. (20). A limited temperature analogy follows from the Gaussian likelihood, not from an identification with dynamical noise. For a data set
of independent labels with noise variance and mean half-squared empirical error , Bayes’ rule gives
where is a specified prior [7]. This is a likelihood scale for the stated normalization. It neither derives the feature weighting below nor identifies with a Langevin temperature; in particular, it does not establish escape from a basin when a spectral overlap vanishes.
To illustrate competition between feature strength and regularization, we allow the two feature channels to carry different penalty weights . The data strengths remain ; a larger simply makes that channel more costly to represent. We choose so that regularization penalizes the stronger feature more heavily, allowing the phase boundaries to cross. The ordinary penalty is recovered by setting both weights to one. The phase diagram uses the effective loss
The targets remain ; the overlap weights the errors. Using it this way is an effective model of feature availability, not a simulation of noisy samples or a consequence of BBP alone. In particular, it differs from replacing the target by , although both choices give the same zero-activation threshold. Minimizing the penalty at fixed gives . For the minimum is
For and the minimum is . At , that factor pair is unconstrained. For , feature has a nonzero minimizing coefficient exactly while
and the four regions have the feature-subset labels of the two filtrations that impose feature 1 first or feature 2 first (§ 3.2). Their regularized minimizers do not generally satisfy the original exact constraints. At the thresholds are : this is the feature-weighted extension of § 4.1, reducing to its thresholds when . Distinct ratios give separate transitions. Raising lowers both thresholds through and bends the boundaries toward smaller : a feature the data only partly resolves is relinquished sooner. Because depends on while the penalty weights do not, the two boundaries need not keep their order. They cross when , here at , : below the crossing the penalty ordering governs and the heavily weighted feature is given up first, above it the resolution ordering governs and the weaker feature goes first. For comparison with the deterministic activation boundaries, the numerical protocol uses overdamped Langevin dynamics, discretized as
with independent Gaussian kicks at successive steps and held fixed throughout the grid. The kicks perturb the parameters during optimization, whereas changes the prescribed error weights . They are part of the numerical optimization protocol and do not implement the data noise represented by . Here “Langevin sampling” means collecting configurations from a finite-time trajectory; equilibration is not established by the protocol alone. No noisy data matrices are generated, so the numerical panel does not average over data-noise realizations.
The feature classification also has a separate threshold. For a fraction , the deterministic minimum obeys
The implemented choice therefore lowers the classification boundary to even before finite-time or noise effects. Changing a common rescales both deterministic curves by the same factor: the crossing value of is unchanged and its coordinate is multiplied by . This prediction for the minimum does not determine the threshold dependence of a finite-noise time average.
Figure 5. Feature selection under regularization and prescribed spectral-resolution weights. Two-feature bilinear model , with targets and penalty weights . Both panels vary and in the effective loss of Eq. (24). Colour identifies the feature subset satisfying the criterion used in each panel: neither feature (grey), only feature 1 (orange), only feature 2 (purple), or both (green). (a) Analytic classification of the global minimum. Feature is active when , equivalently . The panel is constructed directly from these inequalities. The dashed curves show the activation boundaries and are repeated in panel (b). (b) Classification from finite-time Langevin trajectories on the same effective loss, with fixed , , and steps. Each cell starts from . After the initial 40% of the run, products are recorded every 20 steps; feature is classified as active when their time average exceeds . There is one trajectory per cell on a grid. The Gaussian kicks perturb parameters; the resolution weights are prescribed, and no noisy datasets are sampled. The analytic activation boundaries cross at , . The respective weights vanish at and . The panels use different activity criteria: even at the deterministic minimum, the criterion gives the boundary . Parameter noise and finite relaxation time can cause further differences; the numerical panel does not establish equilibration. Grid ranges, sampling indices, and random-number settings are given in Appendix § 10.
Deep linear networks
We now allow full matrix factors. A depth- linear network has hidden layers and realizes with ; a two-factor network therefore has one hidden layer. We leave the diagonal minimal model for these linear and deep linear networks, where the contingent group can still be written down (see § 5.1), for one hidden layer the regularized Hessian admits the four-sector decomposition of § 5.2, and the regularization path selects feature subsets that can label a nested constraint filtration (§ 5.3). Nonlinear networks and other learning systems are discussed in § 6.
The contingent group of linear networks
In linear and deep linear networks the contingent group can be written down explicitly. With the features the singular pairs of , their coefficients , and a pair acting on the end-to-end map by , Eq. (5) becomes
Its elements rescale each constrained pair, and , act freely on the two unlearned complements, and may map learned directions into them (App. § 7), so lies in the parabolic subgroup that preserves the complements, and its dimension
falls with each added scalar constraint. Here counts imposed coefficients; these constraints alone do not require rank . The group preserves the coefficient functionals, rather than fixing one particular matrix, and a solution set need not be a single orbit. A point stabilizer would instead be the subgroup leaving one specified configuration unchanged; we do not use that name for . Figure 6 summarizes the constraint hierarchy and separates it from the norm-preserving architectural action.

Figure 6. Constraint hierarchies and their norm-preserving restrictions. (a) Upper row: the groups , where preserves each of the first coefficient functionals for every represented map. Reading from left to right removes constraints and enlarges the group, ; reading from right to left adds constraints. Lower row: their norm-preserving restrictions . Vertical arrows: restriction to norm-preserving transformations, not a group projection. These restrictions preserve the specified coefficients; preservation of the full data error is a separate condition. (b) Upper row: the exact constraint sets , with . Each additional equality selects a subset of the preceding set. Lower row: the balanced subsets . Both rows are therefore nested, and . A constraint set can contain multiple represented maps, matrix ranks, and symmetry orbits. Balancedness selects minimum-norm factorizations at a fixed map; it does not reduce the constraint set to a single configuration or identify it with a regularized minimum.
Regularization introduces additional massive modes
The distinction between architectural redundancy and unresolved features helps interpret the Hessian spectrum of the regularized linear network; the eigenvalues themselves require the loss calculation. Ladewig et al. [14] derived this spectrum in closed form and sorted it into four classes: massive data-dependent eigenvalues, a null-space sector, a sector from the explicitly broken symmetry, and zero eigenvalues from symmetry. For one hidden layer the eigenvalues in their Table II fall into the four sectors below; their loss, , is twice ours, so their eigenvalues are twice those quoted here. A first pass through the symmetry classification suggests only two families: flat modes from the surviving generators, massive modes from the broken ones. The distinction between nonzero symmetry-orbit tangents and perturbations that change the product only at higher order supplies the additional separation, as the onset calculation below shows. In this section, in Eq. (19) denotes the squared error for whitened inputs and input-output cross-covariance ; unlike , it fixes every coefficient of the map, not only its coefficients along the features. Before setting them out, one point of bookkeeping. The symmetries of § 2 were derived from the map and make no reference to ; the sectors below are sectors of , Eq. (19), at a balanced rank- critical point of , expressed in an aligned hidden basis. Here balanced means ; in this basis factor pairs are nonzero and the remaining hidden-unit blocks vanish. The explicit sector formulas below assume one hidden layer, , and generic distinct signal strengths; positivity of exchange modes additionally requires a stable selected feature subset. The two statements are compatible but not interchangeable, because the penalty is not invariant under everything the map is. Of the architectural group (2), only the orthogonal subgroup preserves , so only leaves invariant while all of leaves invariant; the difference is what produces the pseudo-Goldstone sector. The same restriction applies to the contingent action , which preserves the Frobenius norms only for orthogonal .
Flat modes are tangents to architectural symmetry orbits. At a stationary point, each nonzero tangent induced by a continuous symmetry of the objective is a null direction of ; linearly independent tangents, rather than abstract generators, determine the count. At a rank- critical point, hidden units are active and the other are inactive, with zero weights. Orthogonal reparametrizations (2) that rotate the active units, among themselves or into the inactive ones, change the weights but neither the end-to-end map nor the norm, so they move the configuration along an equal-loss family of critical points (minima when the point is stable); these are the flat directions, of them (App. § 8). In this one-hidden-layer setting, learning one more independent feature activates one more unit and reduces the point stabilizer from to , the matrix counterpart of the sign symmetry of § 4.1. Rotating only the inactive units has no effect at all, since their weights are zero. Away from degeneracies and thresholds, the listed contingent generators supply no additional Hessian-null directions (Appendix § 7): those acting on the unlearned complements touch only weight blocks that are zero at the critical point, the rescalings of the learned pairs act inside the architectural orbit, and those that mix a learned direction into a complement change the end-to-end map and are Fisher-positive; their Hessian curvature is determined by the stability of the critical point.
Onset modes activate an inactive feature at second order. Choose an inactive hidden unit and an unrepresented target direction . Perturb that unit’s input weights by and its output weights by . Since both blocks initially vanish, the first-order product differential is zero, while the exact product change is . For the full two-factor objective, the resulting change is
so the Hessian in is
Fisher information vanishes on these directions at the inactive configuration because . The contribution is the penalty curvature and the contribution comes from the data-error residual. The smaller eigenvalue crosses zero at . These are onset directions, not nonzero tangents of a symmetry orbit: a homogeneous action on the zero blocks leaves them zero, whereas the perturbations above create a new product. Their link to the filtration is that they change only , which a partial task omitting feature leaves unconstrained. The eigenvalues follow from the full objective, not from this freedom alone. For a fully inactive path through more than two weight matrices, the product changes at order greater than two; its quadratic curvature is then only the penalty, and first-order global onset need not have a Hessian zero crossing (§ 5.3).
Function-changing modes have positive Fisher norm. With whitened inputs and a positive-definite Gaussian output metric, a first-order change of the end-to-end map has positive Fisher norm. At a regularized point, however, the error Hessian also contains a residual term, so its eigenvalues must be computed separately. One cannot simply declare the continuous MSE symmetry trivial: acts identically on every map, and orthogonal rotations in true null-signal subspaces can also survive. Identifiability is determined by the product differential after such redundancies are accounted for. Among the function-changing directions are the exchange directions of Ladewig et al. [14], which tilt a learned mode toward an unlearned mode ; with one hidden layer and whitened inputs their eigenvalues are .
-modes are pseudo-Goldstone. The term reduces the reparametrization group from to its orthogonal subgroup, keeping only the norm-preserving maps. At the balanced critical point, the architectural tangent space splits orthogonally into tangents induced by skew-symmetric and symmetric hidden-coordinate generators. Their eigenvalues are respectively and , and both sectors are Fisher-null (Appendix § 8). Generators supported entirely on inactive hidden units induce zero tangents and are excluded from the count. Linear scaling with supports this interpretation only together with gauge-orbit alignment and vanishing Fisher norm.
The shift has a consequence for the flat sector that is worth stating, since it is easy to read “flat” as “no curvature in the error”. It is not. Let be tangent to the orbit at a critical point of . Both and are constant along that orbit, so is too and . By Eq. (19),
The exactly flat directions of the regularized loss are therefore concave directions of the unregularized error, with curvature exactly . Flatness here is a cancellation between a concave error surface and the penalty, not an absence of curvature in ; the Goldstone modes of the broken symmetry sit precisely where the error landscape curves downward, which is where the concave boundaries of the mechanism are found (Ersoy et al. (2026)). The value itself is automatic, since holds everywhere; what the symmetry argument predicts is the number of these directions, under the stated one-hidden-layer assumptions, for away from transitions and accidental degeneracies, and that they are the orbit directions of . In short: the Hessian at a rank- critical point resolves into four sectors: flat, onset, massive, and pseudo-Goldstone, and all four are organized by the symmetries classified in § 2.1: exactly flat where an architectural generator survives the penalty but is broken spontaneously by the configuration, onset where the contingent freedom of an unlearned feature is realized at second order, massive where the data has already locked one in, and pseudo-Goldstone where the regularizing field has explicitly broken an architectural generator. In the terms of Ladewig et al. [14] these are, in turn, their zero eigenvalues, null-space sector, massive data-dependent sector and sector. They trace the first to the orthogonal symmetry and the last to the explicit breaking of by the penalty; what the present account adds is the origin of the other two in the contingent freedom of the partial task, and which of the four carries information. The comparison therefore distinguishes two effects: informative task curvature and explicit gauge lifting. An onset soft mode is not itself a Goldstone mode, and a higher-order onset direction must not be identified with a nonzero symmetry-orbit tangent. The first is the introduction of a feature (§ 3.4): the mass corresponds to a nonzero Fisher eigenvalue, and the direction is an emergent learning direction. Along the one-hidden-layer cascade, the onset direction becomes soft at and the active branch then has a function-changing, Fisher-positive normal direction. This is the counterpart of the emergent learning direction in the filtration, under a different comparison of states. At the collapsed threshold , the Fisher metric still vanishes in that direction. It becomes positive on the nonzero branch for . The onset eigenvalue is not a Fisher eigenvalue. The Fisher metric vanishes along the onset direction until the feature is learned, so comes entirely from the penalty () and from the residual of the still-unlearned feature (). The vanishing is forced by symmetry: the inactive unit sits at the fixed point of the sign flip of its weights, an architectural symmetry, and at such a point the Fisher information is singular along the flipped directions [26]. The second is the introduction of the regularizer, which produces the pseudo-Goldstone -modes. They lie along the architectural orbit, on which the realized function does not change, so they remain null directions of the Fisher metric; their total curvature is , comprising from the ordinary error Hessian and from the penalty Hessian at the regularized critical point. Although its value is controlled by , this curvature does not imply statistical distinguishability: these gauge directions remain Fisher-null. In this setting, each activation changes the locally identifiable parameter subspace. For the full two-factor matrix model at a canonical balanced rank- factorization, under the positive observation metric. A rank increment can therefore create several Fisher-positive directions, including mixing directions. The one-mode-per-constraint identity of § 3.3 concerns one added scalar observation at a common regular point, not every rank transition of a full network.
Nested constraints and ordered regularized phases
For a fixed ordering and retained target values, the partial-task solution sets are nested. Separately, in the aligned separable model the regularization path selects feature subsets in an ordered sequence. This is what Figure 6 and Figure 7 depict; here we argue it must be so.

Figure 7. Nested feature constraints and the remaining freedom of representation. Schematic for a one-hidden-layer linear network of hidden layer width , with a fixed ordering of the features. Grey regions: the constraint sets of Eq. (4), nested from (outermost, unconstrained) to (innermost, all constraints). Each step inward adds one constraint and keeps the earlier ones. is a set of exact solutions, not a single configuration. Blue shading: the balanced subsets , where is the set of balanced factorizations, . Arrows: the black arrows are transformations that preserve the first coefficient functionals and act within , which need not be a single orbit. The blue arrows are the norm-preserving parts of these contingent actions. They keep both balancedness and the imposed coefficients. Vertical copies: factorizations of the same map , related by acting as (side arrows). Its orthogonal subgroup also preserves the parameter norm and balancedness. The copies show architectural redundancy, not further constraints. Green dotted path: connects representative configurations as constraints are added and ends at the balanced exact solution (magenta). The path only illustrates the filtration. It is neither a computed training trajectory nor a path of regularized minimizers; regularized minimizers can pass through the same ordered feature subsets with coefficients shifted away from the exact targets. The zero configuration lies in but satisfies none of the constraints, since every . Areas, distances, and vertical spacing are schematic, and the regions do not depict basins of attraction.
Logical argument. Consider the aligned separable deep-linear model with distinct singular values , uniform layerwise regularization, and every hidden width at least . Then the solution sets of the partial tasks form a nested family , and the quasi-static regularization path from the zero map toward the rank- target map passes through phases corresponding to them, in order and one transition per step: its rank- phase represents the first features. Only with one hidden layer is the active coefficient exactly ; at greater depth it follows the corresponding nonlinear stationary equation. The correspondence concerns feature subsets, not membership of the regularized minimizer in the exact sets .
The argument separates constraint-set inclusion, global threshold ordering, and local stability.
Containment. A solution that fits the first singular directions also fits the first . Hence the solution sets are nested, , and the symmetry groups are nested in the same order, (13). In the minimal model the modes decouple and thus one could think of differently ordered nesting. The one given here is the one taken by the investigated path.
One feature at a time. Because the modes decouple, each singular direction has its own critical strength , and, for a uniform penalty, the ordering of the orders the . As is lowered the directions turn on one at a time, in descending order: rank increases in unit steps, never by two at once for generic . Established: Ladewig et al. [14]; the analogous order in time is the classic deep-linear result of Saxe et al. [21].
The quasi-static path cannot pass early. At the rank- critical point every onset direction toward an unlearned feature has positive curvature while (with one hidden layer it is ; with more, the penalty alone), and the exchange directions are positive as well. The rank- configuration is therefore a local minimum of , and comparison of the decoupled objective values, rather than local curvature alone, determines when it ceases to be the global minimum, so the quasi-static path of § 4.1, which tracks the minimum at every , cannot leave it earlier. With one hidden layer the onset is continuous, and is also where the rank- configuration loses stability. With two or more hidden layers the stationarity condition is nonlinear and the two configurations coexist over a finite range of ([27, 12]): the rank- configuration stays locally stable below , and the path leaves it there because the rank- configuration becomes lower. This is established for the swept path; what it does not establish is that the sublevel sets of are disconnected between ranks, nor that arbitrary dynamics follow the same sequence. Those questions require separate analysis of barriers and attraction basins. For a specified deterministic training dynamics, an attraction basin is the set of initial configurations converging to a given minimum or minimum set; it is not the exact constraint set .
Constraint retention gives the nesting. The distinct global activation thresholds of the separable model give the ordered feature subsets along a continuous sweep of global minimizers. Local-minimum continuation can instead show metastability and hysteresis; neither path establishes nested attraction basins.
Discussion
Summary A partially specified learning task leaves contingent freedom; where a continuous action generates that freedom, adding a constraint reduces the corresponding symmetry. We classified the symmetries acting on such a system into three kinds, identified the contingent one, solved for it in deep linear networks, where it lies in a parabolic subgroup and its dimension falls with each coefficient constraint imposed, and distinguished its nontrivial action from transformations that are architectural or fix the configuration. Data distinguish the directions that change the observed map to first order. The generator actions and the loss expansion together organize the four curvature sectors found by Ladewig et al. [14]: at a critical point the contingent generators act trivially on the weights, act inside the architectural orbit, or are locked by the data into massive directions, so their fingerprint is the structure of the onset sector rather than an extra flat direction. At a common regular exact fit, adding one independent scalar constraint increases both the error-Hessian rank and the Fisher rank by one. Along the regularization path, the related gain of statistical distinguishability is determined by the product Jacobian on the active branch; the exact-fit Hessian-Fisher identity does not apply to a regularized minimum with nonzero residuals (Appendix § 8). Two measurable phenomena trace the breaking without requiring the group: a regularization sweep locates feature-selection events, and a spectral threshold characterizes recoverability in the specified noisy-data model.
Refining the task description. The coarse-graining interpretation separates redundancy in representation from unresolved functional distinctions. Architectural freedom changes how a function is represented; contingent freedom can change aspects of that function omitted by a partial task. Adding an independent scalar feature refines the description and, at a common regular exact solution, increases both the error-Hessian rank and the Fisher rank by one. For the independent Gaussian observations used here, the reverse operation has a direct statistical meaning: discarding the additional observation recovers the preceding partial model and removes its Fisher contribution. This is a concrete instance of the monotonicity of Fisher information under
parameter-independent processing of observations (Ay et al., 2015). The statistical statement concerns the observation models; the deterministic maps in Figure 1 organize the corresponding equivalences between parameters, functions, and retained feature values. This interpretation remains meaningful when the fibers are not single symmetry orbits. It describes a hierarchy of task observables, without constructing a spatial rescaling or renormalization-group flow. The regularization sweep provides a separate comparison of fitted states under the full task; it does not itself implement the coarse-graining maps.
Nonlinear networks: what survives without a group description. The argument of the paper can now be stated in its general form. The constraint-based account of § 2 applies more broadly than linear activations, but redundancy and incompleteness do not guarantee a continuous group action. The regularity and observation assumptions of § 2.4 are needed for the local rank identities even when no group is available. Without the linear-model structure, three explicit descriptions are lost: the closed form (30), the sector dimensions of App. § 8, and the identification of which generator is consumed at which transition. What is not lost is the phenomenology of § 4. A cascade of transitions under a swept remains observable, it is observed in nonlinear networks [11] (Ersoy et al. 2026), but identifying a transition with subgroup reduction requires establishing the corresponding action. Likewise, a spectral threshold is informative only when the data model and estimator meet its assumptions. The two probes are complementary: the cascade measures which features survive a penalty, whereas the spectral calculation describes recoverability in a specified noisy ensemble. Their quantitative relation must be tested for the model at hand. A consequence follows, and it concerns what they do not need. Neither measurement requires an explicit symmetry-group description, but the analytic spectral thresholds still require a suitable noise ensemble, scaling limit, and estimator. A regularization sweep and joint feature/Jacobian/Hessian measurements can be performed in nonlinear networks without assuming that their features are singular directions of an input-output covariance.
Beyond neural networks. The regular scalar-constraint identities of § 3.3 do not require linear activations, although the explicit toy groups and full-network spectra do use the stated model structure. The ingredients that drive the mechanism are two: a parameter space large enough to carry reparametrization redundancy, and unlearned directions along which a change can be absorbed into either the data or the parameters. Their realization depends on both the architecture and the task. Where the relevant unresolved variations can be identified, we expect a constraint-based description, and their breaking should produce the same signature: a Hessian that separates into a few massive, task-aligned modes and a large complement of flat ones, with the massive set growing each time a feature is learned. Related reorganizations of response spectra have been documented in physical learning systems, although their curvature operators need not be parameter Fisher matrices. Stern and coauthors show that learning imprints structure on the Hessian: the effective dimension of the response drops, the low-eigenvalue eigenvectors rotate to align with the task, and the cost and physical Hessians become coupled [23, 24]; supervised learning in such networks was framed as machine learning becoming “learning machines” with exactly this stiff/soft organization [22]. The relation of those stiff and soft response modes to uninformative parameter directions is a hypothesis. Calling a soft mode Goldstone additionally requires a nonzero tangent generated by an exact continuous symmetry.
Observables. The account predicts four things that can be checked. Turning on the field breaks and, through the balance condition, selects a single -orbit of balanced states within each fiber, so positive regularization selects balanced representatives from the degenerate fiber. This does not force a discontinuity of a chosen minimizer at : a balanced branch can approach a balanced member of the unregularized minimum set continuously. With one hidden layer, the Hessian at a rank- critical point should carry four eigenvalue classes with the dimensions of App. § 8: flat, , (and along directions without signal), and data-massive, including along the exchange directions. For the stated one-hidden-layer linear model this is the spectrum derived by Ladewig et al. [14]; analogous distinctions can guide nonlinear experiments, but the four exact classes and their multiplicities are not universal predictions. For one hidden layer, , and the generic aligned configurations considered above, the symmetry-generated null space has dimension and is spanned by the orbit directions of ; extra degeneracies must be checked separately. Along these directions the unregularized error is concave with curvature , Eq. (33): an algebraic consequence of . Their orbit alignment and dimension, rather than the value alone, test the symmetry interpretation. The groups in (30) have decreasing dimension, but orbit dimensions also depend on the subgroup that leaves the chosen configuration unchanged (its point stabilizer); group dimension alone does not determine phase ordering or stability. Outside the solvable case, the informative test is to track the error on each feature, the data Jacobian or expected Fisher matrix, and both the error and penalized Hessians through a transition. This can establish whether that transition has the proposed mechanism. The general constraint principle alone does not require every architecture to exhibit a discrete cascade or a covariance-based BBP threshold.
Relation to symmetry-constraint accounts. The contingent symmetry is related to, but distinct from, a broader phenomenon: mirror symmetries can define invariant fixed subspaces whose occupation is favored by sufficiently strong regularization or gradient noise under the hypotheses of Ziyin [26]. A fixed subspace need not consist entirely of stationary points. That account concerns symmetries of the loss itself; it includes architectural examples such as hidden-unit permutations, with additional assumptions for the regularization and noise conclusions. Contingent symmetries here preserve the auxiliary partial loss, but need not preserve the full loss: they are the freedoms a not-yet-fully-specified task leaves open, they form a nested chain indexed by the retained constraints. In the minimal model, their reduction under an independent scalar constraint is tied to an increase of Fisher rank at a common regular exact solution. The regularization path provides a related but distinct comparison.
Shortcomings. Two questions remain open. § 5.3 establishes constraint-set nesting and an ordered global-minimum feature sequence under separate assumptions. It does not establish nested attraction basins, nor identify those sets with metastable phases. Relating this sequence to the dynamics studied in Ersoy and Wiesner [12] requires a separate calculation of local minima, barriers, and escape dynamics. The universality claim above is only partly tested: the transitions themselves are observed in nonlinear networks (Ersoy and Wiesner [11]; Ersoy et al. (2026)), but tracking the Hessian or Fisher spectrum through training of a nonlinear network, and checking which feature-aligned directions gain statistical distinguishability, is a natural experiment not carried out here.
Outlook. The ordered feature subsets of § 5.3 offer labels for comparing the metastable states studied in Ersoy and Wiesner [12]. Whether their attraction basins have a corresponding organization remains open. Feature acquisition, phase selection, and gain of Fisher distinguishability are related in the exact models under the stated conditions; their coincidence, and its relation to memorization and generalization, must be tested in other settings. Whether an approximate version of can be written down past the linear, aligned-Gaussian setting in general nonlinear networks, where the transitions are observed (Ersoy and Wiesner [11]; Ersoy et al. (2026)) and the general symmetry-constraint machinery of Ziyin [26]; ); Ziyin et al. [28] already applies, is the key question for turning the general claim into a quantitative theory of how networks acquire features. The two phenomena of § 4 are what make that question answerable empirically rather than only analytically.
The coefficient-preserving group
Write and . Preserving for every is equivalent to
Since both sides have rank one, there is a nonzero scalar such that and . In orthonormal bases beginning with the constrained directions, the group therefore consists of
where is invertible and diagonal, , , and are unrestricted. Counting these entries gives
This counts transformations preserving coefficient functionals, not independent orbit tangents at a particular configuration. In particular, the decrease of group dimension is not the number of Fisher modes gained by imposing one scalar constraint. To describe the action used in § 5.2, additionally take a canonical aligned rank- factorization, with nonzero product block and all other factor blocks zero. The infinitesimal action on its product has the form
where are the infinitesimal lower-left blocks. Complement-only transformations fix the zero factor blocks; reciprocal rescalings of active pairs induce architectural gauge tangents; and the displayed mixing blocks change the represented map. The coefficient constraints alone do not require this canonical rank- configuration.
Norm-preserving restrictions. Intersecting with sets , makes orthogonal, and restricts to diagonal signs. This is the restriction in Figure 6. For orthogonal , invariance of the full map error for every additionally requires . Neither restriction is the architectural hidden-unit action , which preserves itself.
Hessian sectors and Fisher information
Consider one hidden layer of width , whitened inputs, and
The original population squared error differs from its first term only by a parameter-independent constant. Let be a perturbation of , , and . Expanding the product to second order gives
For with , the expected Fisher quadratic form is
Thus neither the residual term nor the penalty term belongs to the Fisher matrix; at , the unregularized error Hessian equals [16]. Now evaluate at a canonical balanced rank- critical point, whose active factor blocks are with , and whose other blocks vanish. A gauge tangent satisfies . Differentiating the gauge transformation of the error gradient and using at the critical point yields
Skew-symmetric therefore produces eigenvalue zero, and symmetric produces eigenvalue , whenever the induced tangent is nonzero. Balancedness makes these two tangent spaces orthogonal. In either case, matrices supported only on the inactive hidden block induce zero tangents and must be excluded from the count.
| Sector | Dimension | eigenvalue | Fisher norm |
| Orthogonal gauge | zero | ||
| Symmetric gauge | zero | ||
| Inactive-unit complement | zero | ||
| Function-changing | data-dependent | positive |
Table 1 (PDF p. 27).
The third sector couples inactive hidden units to the unrepresented input and output complements. Its first-order product differential is zero, and its quadratic blocks are Eq. (32). Writing , each omitted nonzero singular value supplies , each with multiplicity ; the remaining directions have eigenvalue . Here active modes are selected from the signal directions. The last sector is : the image of has arbitrary active-active and active-complement blocks, and a zero complement-complement block. Consequently
The four dimensions sum to . The table separates geometric subspaces, not necessarily distinct numerical eigenvalues; additional Hessian zero modes occur at thresholds or degeneracies. At saddles, Fisher-positive directions need not have positive Hessian eigenvalues. The detailed spectrum is treated in Ladewig et al. [14]. For the symmetric gauge sector, Eq. (41) also gives : its total curvature contains equal contributions from the error Hessian and the penalty Hessian.
Ordered thresholds in the aligned deep-linear model
For scalar factors per aligned feature and a fixed positive product , the arithmetic-geometric mean inequality gives , with equality for equal factor magnitudes. Each feature therefore reduces to
For , and the transition is continuous at . For , solving together with gives
For and , the sign of is the sign of . The latter ratio has its unique positive maximum at on , so the displayed value is the global activation threshold. The nonzero jump is first order, and increases strictly with . Thus distinct strengths and a uniform penalty order the global activation thresholds in the separable aligned model of § 5.3. This comparison does not determine which local minimum an optimizer reaches. More generally, for any nonnegative error finite at zero, a global minimizer of obeys , establishing convergence to zero as when all parameters are penalized.
Numerical construction of the figures
Exact landscapes. Figures 2 and 3 evaluate the displayed bilinear losses directly on uniform factor-space grids in , with and respectively 401 and 400 points per axis. The coefficient-space filtration panels use 401 points per axis on . The filtration uses ; an omitted feature is represented by , while an imposed feature uses . The regularized panels use strengths for feature 1 and for feature 2. They mark the positive balanced minimum ; the plotting code denotes by . No training trajectories are used for these figures.
Illustrative rough landscapes. For Figure 4, the plotting code evaluates
Figure 4. Illustrative landscapes at three levels of spectral resolution. The target coefficients are . For the square spiked-matrix model, the prescribed overlaps are . From top to bottom, , , and , giving , , and . Thus neither, only the first, and both population feature directions have nonzero asymptotic spectral overlap. The left and middle columns show the factor planes and ; the right column shows coefficient space . The star marks the target, while the red-outlined circle marks the displayed configuration . Its reported values are approximately in the first two rows and in the bottom row. White dots in the factor planes represent the corresponding coefficients. Black dashed hyperbolae mark ; white curves mark . Green arrows indicate product-preserving tangents, and red or orange arrows indicate product-changing normals. These arrows identify geometric directions, without by themselves establishing Hessian eigenvalues at the marked configurations. All three columns use the same additional rough landscape, with amplitudes , , and . This roughness is distinct from the deterministic overlap weights and is not generated by the BBP overlap formula alone. The resolution labels refer to singular-vector recovery, rather than exact fitting of the displayed coefficients. The random-field construction and grid selection of are specified in Appendix § 10.
where and is one fixed random Fourier field. Its raw form is . Directions are sampled by normalizing independent two-dimensional standard Gaussian vectors; frequency magnitudes are log-uniform on and , respectively. Phases are uniform on , and amplitudes are independent Gaussians with standard deviations and . The code uses NumPy’s default_rng(14) and normalizes this field once to zero mean and unit standard deviation on a grid over . At , is the minimum on a grid over the same window, without continuous refinement. The factor panels evaluate or on grids in . Logarithmic shading uses each panel’s excess loss , with and upper cutoff , where is the 90th percentile of that excess loss. This additional random field is a visualization model, not a sampled cross-covariance or a consequence of BBP theory. It is absent from the phase-diagram calculation below.
Analytic phase diagram. Figure 5 uses Eq. (24), with and , on logarithmically spaced grids of 100 values each: and . Since , minimization at fixed product reduces each feature to . Panel (a) classifies its global minimum using . The analytic crossing is
Finite-time numerical panel. Panel (b) uses one finite-time trajectory per grid point. At each step, the parameters undergo a gradient-descent update followed by an independent Gaussian kick, as specified in Eq. (27). This is the Euler-Maruyama discretization of overdamped Langevin dynamics ([33]). We use a fixed noise strength , a time step , and 12,000 updates. Initial parameter components are drawn independently from . The parameter-noise strength is held constant throughout the diagram; the plotted variable controls the prescribed feature weights . The gradients used are
All components are evaluated before updating the parameters. Products are recorded after updates whose zero-based loop indices satisfy and , giving 359 samples per trajectory. Feature is active when its sample mean exceeds . The implementation uses default_rng(0), traverses in the outer loop and in the inner loop, and assigns each cell its initialization followed by its Gaussian kicks. The dashed lines in both panels are the analytic activation boundaries. For the numerical panel’s threshold applied to a deterministic minimum, the reference boundaries would instead be , crossing at the same and . Consequently, criterion choice, finite time, parameter noise, and discretization can all contribute to differences between the panels. is fixed, no noisy datasets are sampled, and one trajectory per cell does not establish equilibration or sampling uncertainty.
References
- [1]A. Altland and B. Simons, Condensed Matter Field Theory, 3rd ed. (Cambridge Univ. Press, 2023).
- [2]S. Amari, Information Geometry and Its Applications (Springer, 2016).
- [4]J. Baik, G. Ben Arous, and S. Péché, Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices, Ann. Probab. 33, 1643 (2005).DOI
- [5]F. Benaych-Georges and R. R. Nadakuditi, The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices, Adv. Math. 227, 494 (2011).
- [6]F. Benaych-Georges and R. R. Nadakuditi, The singular values and vectors of low rank perturbations of large rectangular random matrices, J. Multivariate Anal. 111, 120 (2012).
- [7]C. M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006).
- [8]A. J. Bray and D. S. Dean, Statistics of critical points of Gaussian fields on large-dimensional spaces, Phys. Rev. Lett. 98, 150201 (2007).pubmed.ncbi.nlm.nih.gov/17501322
- [9]B. Efron and D. V. Hinkley, Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information, Biometrika 65, 457 (1978).DOI
- [11]I. T. Ersoy and K. Wiesner, Exploring L₂-phase transitions on error landscapes, in ICML Workshop on High-dimensional Learning Dynamics (HiLD 2025) (2025). https://openreview.net/forum?id=AkQNtAwO9u
- [12]I. T. Ersoy and K. Wiesner, Noise-driven escape from metastable phases explains grokking in deep neural networks, in ICML Workshop on High-dimensional Learning Dynamics (HiLD 2026) (2026), arXiv:2606.17120. https://openreview.net/forum?id=p3bdkPqsxvarxiv.org/abs/2606.17120
- [13]N. Goldenfeld, Lectures on Phase Transitions and the Renormalization Group (CRC Press, 1992).
- [14]B. Ladewig, I. T. Ersoy, and K. Wiesner, Cascading through the hierarchy: Regularizer-induced feature detection as phase transitions in deep linear neural networks (2026), arXiv:2608.06597. https://arxiv.org/abs/2608.06597
- [15]K. Lindsey and G. Menon, Regularization implies balancedness in the deep linear network (2026), arXiv:2511.01137v2. https://arxiv.org/abs/2511.01137v2
- [16]J. Martens, New insights and perspectives on the natural gradient method, J. Mach. Learn. Res. 21(146), 1 (2020). https://www.jmlr.org/papers/v21/17-678.html
- [17]G. Menon, The geometry of the deep linear network, in XIV Symposium on Probability and Stochastic Processes, edited by C. G. Higuera Chan, J. A. López Mimbela, S. I. López, and C. G. Pacheco, Progress in Probability, Vol. 81 (Birkhäuser, Cham, 2025), pp. 1–47. https://doi.org/10.1007/978-3-031-96118-2_1
- [18]J. Pennington and Y. Bahri, Geometry of neural network loss surfaces via random matrix theory, in Proceedings of the 34th International Conference on Machine Learning, Proc. Mach. Learn. Res. 70, 2798–2806 (2017). https://proceedings.mlr.press/v70/pennington17a.html
- [19]A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra, Grokking: Generalization beyond overfitting on small algorithmic datasets, in ICLR Workshop on Mathematical Reasoning in General Artificial Intelligence (2021). https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper.pdfarxiv.org/abs/2201.02177
- [20]Sagun, L., Evci, U., Guney, V., Dauphin, Y. & Bottou, L. Empirical Analysis of the Hessian of Over-Parametrized Neural Networks. (2018), https://openreview.net/forum?id=rJrTxwbCb
- [21]A. M. Saxe, J. L. McClelland, and S. Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, in International Conference on Learning Representations (ICLR) (2014). https://openreview.net/forum?id=_wzWkIpTDF_9C
- [22]M. Stern, D. Hexner, J. W. Rocks, and A. J. Liu, Supervised learning in physical networks: From machine learning to learning machines, Phys. Rev. X 11, 021045 (2021). https://doi.org/10.1103/PhysRevX.11.021045
- [23]M. Stern, A. J. Liu, and V. Balasubramanian, Physical effects of learning, Phys. Rev. E 109, 024311 (2024). https://doi.org/10.1103/PhysRevE.109.024311
- [24]M. Stern, M. Guzman, F. Martins, A. J. Liu, and V. Balasubramanian, Physical networks become what they learn, Phys. Rev. Lett. 134, 147402 (2025). https://doi.org/10.1103/PhysRevLett.134.147402
- [25]S. Watanabe, Algebraic Geometry and Statistical Learning Theory (Cambridge Univ. Press, 2009). https://doi.org/10.1017/CBO9780511800474
- [26]L. Ziyin, Symmetry induces structure and constraint of learning, in Proceedings of the 41st International Conference on Machine Learning, Proc. Mach. Learn. Res. 235, 62847–62866 (2024). https://proceedings.mlr.press/v235/ziyin24a.html
- [27]L. Ziyin and M. Ueda, Zeroth, first, and second-order phase transitions in deep neural networks, Phys. Rev. Res. 5, 043243 (2023). https://doi.org/10.1103/PhysRevResearch.5.043243
- [28]L. Ziyin, Y. Xu, T. Poggio, and I. Chuang, Parameter symmetry potentially unifies deep learning theory (2025), arXiv:2502.05300.
- [29]L. Ziyin, Y. Xu, and I. Chuang, Remove symmetries to control model expressivity and improve optimization, in International Conference on Learning Representations (ICLR) (2025), arXiv:2408.15495. https://proceedings.iclr.cc/paper_files/paper/2025/hash/040321891311e31134a1b3844efb6a9a-Abstract-Conference.html
- [30]B. Zhao, R. Walters, and R. Yu, Symmetry in Neural Network Parameter Spaces, Transactions on Machine Learning Research (2025), https://openreview.net/forum?id=jLPwq5QY6Iarxiv.org/abs/2506.13018
- [33]D. J. Higham, An algorithmic introduction to numerical simulation of stochastic differential equations, SIAM Review 43(3), 525–546 (2001). https://doi.org/10.1137/S0036144500378302



