betaFeedback

WorksPhysics and Astronomy

Informative Fisher Modes in Neural Networks as a Result of Explicit Symmetry Breaking

Abstract

Learning in a neural network takes place on a high dimensional, non-convex loss landscape, the Hessian curvature of which dictates the learning dynamics. Despite its high dimensionality, the large number of architectural symmetries explains some of the geometric of this landscape, in particular flat directions, i.e. near-zero modes of the Hessian. These architectural symmetries have therefore been suggested as an organizing principle of deep learning. It remains an open question, though, wether there are additional organizing principles in the remaining degrees of freedom in the learning dynamic. Here we identify task-dependent contingent symmetries as such an additional organizing principle. We show that non-flat learning directions emerge due to explicit breaking of these contingent symmetries. We isolate this symmetry in an analytically solvable minimal model. Furthermore, we show that, by tuning an L2 penalty, signatures of the symmetry breaking become measurable. As a consequence of the same mechanism, adding noise to the data drives the model through a sequence of broken symmetry recoveries in the form of Baik-Ben-Arous-Péché (BBP) transitions. We demonstrate analytically for linear and numerically for deep linear networks that the contingent symmetry group remains explicit, and that the symmetry-breaking mechanism identifies which Hessian modes carry task-specific information. These signatures are measurable without a symmetry-group description and, hence, offer a route to study symmetry breaking as a more general principle in nonlinear networks and other learning systems.

Introduction

Understanding how neural networks learn and generalize is a central problem in machine learning, statistics, and physics. Research approaches this problem through optimization, statistical inference, and the geometry of high-dimensional loss landscapes. Statistical physics provides complementary concepts, including symmetries, fluctuations, and phase transitions. A common challenge is to connect task structure with the behavior of the learning system. Here we examine how task constraints restrict model freedom and when the resulting curvature in the loss landscape represents statistical information. Deep linear networks trained from small initialization acquire dominant singular directions on different timescales [21]. In nonlinear networks, generalization can improve abruptly after a long period of apparently successful fitting [19, 12]. Physical learning systems also develop structured responses as they adapt: their task performance and mechanical or electrical response become linked through changes in the relevant curvature operators [22, 23, 24]. A tractable model can isolate each feature constraint, the freedom it removes, and the resulting curvature. How to connect task structure with the behavior of the learning system is addressed by different communities using different methodologies. These include information and differential geometry, singular learning theory, statistical physics, theory of criticality and random matrix theory (RMT). Our work builds on all of these to some degree. We summarize the key findings, relevant for this work, in the following section.

Prior work

One line of work investigated the geometry of DNN error landscapes by varying the strength of an L2L_2 penalty. Ziyin and Ueda [27] derived an onset of learning as the L2L_2 penalty decreases: the transition is continuous with one hidden layer and discontinuous at greater depth. Ladewig et al. [14] showed analytically that in deep linear networks the onset is only the first of a cascade of transitions, each associated with a singular direction of the data. In nonlinear networks, Ersoy and Wiesner [11] identified further transitions at lower L2L_2 strengths, related them to changes in error-landscape curvature, and associated each phase with distinguishable aspects of the task, which they demonstrated both for synthetic and for real data (Ersoy et al., 2026). In the same framework, metastability and noise-driven escape from local minima have also been investigated and found to be a possible mechanism for delayed generalization in nonlinear DNNs [12]. Taken together, these results reveal a sequence of learning transitions, each associated with a geometric location and a Hessian spectrum signature.

A second line of work examined Hessian spectra of error landscapes at points of changing curvature directly. Empirical studies reported many small Hessian eigenvalues together with a smaller set of large eigenvalues [20]; random-matrix approaches explained aspects of the spectral structure and its dependence on the task and the parameter settings [18, 8]. In spiked random-matrix models, the Baik-Ben-Arous-Péché (BBP) transition and related results described when signal-associated outliers separate from the bulk and when their eigenvectors correlate with the underlying signal [4, 5]. Information geometry supplies a complementary local description through Fisher information [2, 16]. Kühn and Rosenow (2026) explain near-zero Hessian eigenvalues through the weak explicit breaking of architectural symmetries by nonlinearities and analyze the corresponding Fisher geometry.

A third line of work studies symmetries in neural-network parameter space and their consequences for learning, reviewed by Zhao et al. [30]. Ziyin [26] proved that mirror symmetries define fixed subspaces preserved by gradient descent and give the Hessian a corresponding block structure at symmetric configurations. Under additional assumptions, sufficiently strong weight decay favors minima in these subspaces; when the symmetry preserves the observation distribution, Fisher information is singular at the corresponding symmetric configurations. Symmetric configurations also reduce the expressivity of the linearized model and the dimension accessible to training [29]. In a recent position paper, Ziyin et al. [28] proposed symmetry breaking and restoration as organizing principles for hierarchical learning. Architectural symmetries hold independently of the data, whereas symmetries of a loss can depend on the task. In deep linear networks, architectural reparametrizations have a tractable geometry [17], and an L2L_2 penalty selects balanced parametrizations of a fixed function in the full-rank square case [15]. The contingent freedoms studied here preserve an auxiliary partial task but need not preserve the full objective.

Our Contribution

We ask: How does imposing a task feature remove a task-preserving freedom and create a positive Fisher mode, and how does regularization affect this relation? Answering this requires distinguishing a freedom left open by a partial task from a symmetry of a fixed objective and from a degeneracy at a particular parameter setting. It also requires separating the curvature of the data error from that of the penalized objective. Our contributions are fivefold. (i) We distinguish contingent symmetries, the freedoms a partially specified task leaves open, from architectural symmetries and task invariances, and introduce the feature filtration to isolate them. (ii) At a common regular exact solution, we show that imposing one independent feature constraint removes one such freedom and increases the Fisher rank by one, while architectural freedom survives. (iii) We show that an L2L_2 penalty can make reparametrization directions massive while they remain null directions of the Fisher metric: these penalty-lifted directions do not correspond to learning a feature. (iv) We combine the regularization cascade with spectral recovery thresholds in a learnability phase diagram over regularization strength and BBP noise amplitude, whose regions have the feature-subset labels of the filtration; for a uniform penalty the two probes order the features identically, while unequal penalty weights make the orderings compete and the boundaries cross. (v) We state the observables by which the account can be tested in networks whose symmetry group is unknown. § 2 defines the framework and filtration. § 3 develops the minimal example and its spectral interpretation. § 4 examines regularization and statistical recovery as probes of the resulting structure. § 5 extends the analysis to deep linear networks, and § 6 discusses nonlinear networks, other learning systems, and open questions.

Conceptual framework and method

The principle investigated here is that learning a feature restricts the choices left open by a task. To examine this idea, we need to distinguish the types of degrees of freedom and specify how a feature enters the analysis. We construct a sequence of partial tasks and compare their admissible solutions, symmetries, error curvature, and statistical distinguishability on the parameter space. This section defines the construction and the quantities to be compared. The relation between them is worked out in the minimal model of § 3.

Architectural, task, and contingent symmetries

Let {fθ:θ∈Θ⊆RD}\{f_\theta:\theta\in\Theta\subseteq\mathbb{R}^{D}\} be the model family. A configuration θ\theta specifies the realized function fθf_\theta; the task specifies the requirements on that function. When a probability distribution is intended, we refer explicitly to the observation model pθ,kp_{\theta,k} introduced below. Consequently, two parameter settings can be equivalent for different reasons. We distinguish three sources of equivalence by what remains unchanged and by the conditions under which the equivalence holds.

Architectural symmetries. These are reparametrizations that leave the realized function unchanged:

fg⋅θ(x)=fθ(x)for every admissible x.(1)f_{g\cdot\theta}(x)=f_\theta(x)\quad\text{for every admissible }x. \tag*{(1)}

They follow from the parametrization and hold independently of the task. For a two-layer linear network fθ(x)=W2W1xf_\theta(x)=W_2W_1x, the ww coordinates of the intermediate vector W1xW_1x are its hidden units and ww is its hidden width. With end-to-end map M=W2W1M=W_2W_1, the reparametrizations include

(W2,W1)⟼(W2G,G−1W1),G∈GL⁡(w).(2)(W_2,W_1)\longmapsto(W_2G,G^{-1}W_1),\qquad G\in\operatorname{GL}(w). \tag*{(2)}

Hidden-unit permutations are another example; homogeneous activations also permit suitable rescalings. The available transformations depend on the architecture. The fibers of a factorized linear network and their group actions provide a rigorous setting for this redundancy [17, 15].

Data and task symmetries. These express distinctions that the fixed task does not require the model to resolve. For example, in a classification task an input transformation may preserve the assigned class label. Such an invariance is specified by the data distribution and the objective; it need not be a symmetry of an arbitrary model’s predictions. To induce a symmetry of the learning objective, the corresponding transformation must preserve the loss and the relevant data measure. Its existence alone therefore does not establish a flat direction in parameter space. It remains part of the task when that task is fully specified.

Contingent symmetries. These preserve the features already imposed while allowing other aspects of the realized function to change. Their defining property is that a further feature constraint can distinguish configurations that were previously equivalent. They are the contingent symmetries investigated here. In linear networks some admit descriptions through transformations of both model and data coordinates. Their essential content is the preservation of a partial task. They can change the represented function and need not survive a further feature constraint. This distinction concerns the origin of the equivalence, rather than three necessarily disjoint groups. A transformation can preserve both the architecture’s function and every partial task. We therefore identify the architectural freedom first and then ask what additional freedom the partial task leaves available.

Task descriptions as coarse-grainings. Let F={fθ:θ∈Θ}\mathcal{F} = \{f_\theta: \theta\in\Theta\} be the set of realizable functions, and define the realization map R:Θ→F\mathcal{R} : \Theta\to\mathcal{F} by R(θ)=fθ\mathcal{R}(\theta) = f_\theta. A partial task retains kk scalar feature values ci(θ)c_i(\theta), each depending only on the realized function. Its feature map is therefore well defined by

Ck(fθ)=(c1(θ),…,ck(θ)),Ck=Ck(F).C_k(f_\theta) = (c_1(\theta), \ldots, c_k(\theta)), \qquad C_k = C_k(\mathcal{F}).

For prescribed target values ηi\eta_i, the scalar error is obtained through

ℓk(z)=12∑i=1k(zi−ηi)2,Ek=ℓk∘Ck∘R.\ell_k(z) = \frac{1}{2}\sum_{i=1}^{k}(z_i-\eta_i)^2, \qquad E_k = \ell_k \circ C_k \circ\mathcal{R}.

These maps give descriptions containing no more detail than their inputs (Figure 1). Architectural transformations satisfy R(g⋅θ)=R(θ)\mathcal{R}(g \cdot\theta) = \mathcal{R}(\theta). Contingent transformations preserve Ck(R(g⋅θ))=Ck(R(θ))C_k(\mathcal{R}(g \cdot\theta)) = C_k(\mathcal{R}(\theta)) while potentially changing the realized function. The maps perform the coarse-graining; the transformations preserve the descriptions retained by those maps. Their orbits lie within the corresponding fibers, where a fiber is the set of points having the same image. A fiber need not be a single symmetry orbit. The final scalar error discards further distinctions: equal error does not generally imply equal feature values.

Successive descriptions of a model configuration

Figure 1. Successive descriptions of a model configuration. The realization map R\mathcal{R} sends a parameter configuration to its input-output function. Architectural symmetries change the parameters while leaving this function unchanged. The map CkC_k retains only the kk feature values specified by the partial task. Contingent symmetries can change the function while preserving these values; their parameter-space orbits lie within fibers of Ck∘RC_k \circ\mathcal{R}. The final map ℓk\ell_k compares the retained values with their targets and summarizes the discrepancies by a scalar error. Equal scalar errors need not correspond to equal feature values. Adding a feature replaces CkC_k by the finer description Ck+1C_{k+1}, allowing some previously equivalent functions to be distinguished.

Feature filtration as an analytical construction

In the tractable construction, the task consists of rr features: components of the target that the model has to represent, with prescribed strengths ηi\eta_i, i=1,…,ri = 1,\ldots,r. We use the word only in this sense. The model’s coefficient along feature ii is a scalar functional ci(θ)c_i(\theta); imposing feature ii means requiring ci(θ)=ηic_i(\theta) = \eta_i, and the model represents feature ii when ci(θ)≠0c_i(\theta) \ne0. Each cic_i depends on the realized function, so architectural reparametrizations leave it unchanged. The conditions ci(θ)=ηic_i(\theta) = \eta_i express what it means to represent the corresponding features correctly. For an aligned linear model, the desired linear input-output map S=∑iηiuiviTS = \sum_i \eta_i u_i v_i^{\mathsf{T}} and the represented map M=∑ipiuiviTM = \sum_i p_i u_i v_i^{\mathsf{T}} use the same fixed orthonormal input and output directions. Then ci(θ)=uiTM(θ)vi=pic_i(\theta) = u_i^{\mathsf{T}}M(\theta)v_i = p_i. This is what we mean by alignment; no such decomposition is assumed for every learning problem. Choose an ordering of the constraints and define the auxiliary errors

Ek(θ)=12∑i=1k(ci(θ)−ηi)2,k=0,…,r,(3)E_k(\theta) = \frac{1}{2}\sum_{i=1}^{k}(c_i(\theta)-\eta_i)^2, \qquad k = 0,\ldots,r, \tag*{(3)}

with E0=0E_0 = 0. The full task imposes all rr constraints; a partial task imposes only the specified subset, here the first kk. An exact fit of that partial task satisfies every retained equality, so Ek=0E_k = 0.

Assume that the complete set is jointly realizable. Its partial-task solution sets, equivalently the global minimum sets of EkE_k, are then

Mk={θ:ci(θ)=ηi for i≤k},Θ=M0⊇M1⊇⋯⊇Mr.(4)\mathcal{M}_k = \{\theta: c_i(\theta) = \eta_i \text{ for } i \le k\}, \qquad\Theta= \mathcal{M}_0 \supseteq\mathcal{M}_1 \supseteq\cdots\supseteq\mathcal{M}_r. \tag*{(4)}

This nesting corresponds to a refinement of task descriptions. Let ϕk:Ck+1→Ck\phi_k : \mathcal{C}_{k+1} \to\mathcal{C}_k discard the last feature value. Then

Ck=ϕk∘Ck+1,k=0,…,r−1.\mathcal{C}_k = \phi_k \circ\mathcal{C}_{k+1}, \qquad k = 0,\ldots,r-1.

For k=0k = 0, the retained description is the single empty tuple. Functions with identical first k+1k+1 feature values therefore have identical first kk values, whereas the converse need not hold. Discarding a feature coarse-grains the description; adding a feature refines it. The exact-solution sets are its target fibers,

Mk=(Ck∘R)−1((η1,…,ηk)).\mathcal{M}_k = (\mathcal{C}_k \circ\mathcal{R})^{-1}((\eta_1,\ldots,\eta_k)).

This nesting is part of the construction: each partial task retains the constraints already imposed and adds one further requirement. Once the coefficient functionals, their target values, and their ordering are fixed, the chain is determined. Different orderings can give different intermediate sets. These are constraint sets, not generally vector subspaces or nested sets of exactly different ranks. The remaining coefficient functionals are absent from EkE_k; their values may vary to the extent allowed by the architecture and the retained constraints. Whether these choices admit a group description, and what changes when the next constraint is imposed, are questions for the subsequent analysis. The level k=0k = 0 means that no feature constraint has yet been imposed. It does not mean that every parameter setting realizes the zero function. Similarly, an unconstrained feature need not have a zero coefficient. This distinction separates the auxiliary partial task from a model that has failed to fit part of a fixed, fully specified task. The filtration is an analytical device for isolating the effect of a feature constraint. Its ordering does not prescribe the arrival of data or the trajectory of an optimizer. Whether regularization or statistical recovery produces a corresponding sequence is examined separately in the later sections.

Symmetry, curvature, and information

For each partial task we examine which transformations preserve its features, how the error responds to parameter perturbations, and which perturbations change the associated observation distribution. These are distinct descriptions, whose relationship must be established for the model under consideration. To specify the symmetry question, fix a common group DD of admissible invertible transformations of parameter space and define

Gk={g∈D:ci(g⋅θ)=ci(θ) for all θ and i≤k}.(5)G_k = \{g \in D : c_i(g \cdot\theta) = c_i(\theta) \text{ for all } \theta\text{ and } i \le k\}. \tag*{(5)}

The orbit Gk⋅θG_k \cdot\theta is the set of configurations reached by these transformations from θ\theta. When the action is smooth, a one-parameter subgroup gtg_t has generator X(θ)=ddt(gt⋅θ)∣t=0X(\theta) = \left.\frac{d}{dt}(g_t \cdot\theta)\right|_{t=0}. The analysis will determine the groups and their orbits, and compare the independent tangent directions generated at a solution. We do not assume that an orbit is the entire minimum set or that every generator produces a nonzero tangent direction. We use explicit symmetry breaking for a change in the invariance of the objective and spontaneous symmetry breaking for the selection of a configuration with less symmetry than an unchanged objective [13, 1]. In the filtration, adding a feature constraint explicitly reduces the contingent group, Gk→Gk+1G_k \to G_{k+1}, because the objective changes from EkE_k to Ek+1E_{k+1}. A one-parameter action that leaves a configuration θ\theta unchanged has X(θ)=0X(\theta) = 0 and supplies no nonzero orbit tangent there. The distinct symmetry effects of regularization are introduced with the penalized objective in § 4.1. For the curvature calculation, define

Hk(θ)=∇θ2Ek(θ).(6)H_k(\theta) = \nabla_\theta^2 E_k(\theta). \tag*{(6)}

At a stationary point, v⊤Hkvv^{\top}H_kv for a unit parameter variation vv is the directional curvature of the error EkE_k. We call its zero-eigenvalue modes flat and its positive-eigenvalue modes massive. In the regularized sections we distinguish HE=∇2EH_E=\nabla^2E from HL=∇2LβH_L=\nabla^2L_\beta. Fisher information is a separate quadratic form. A Hessian zero mode need not arise from an exact invariance. We also use the coefficient Jacobian Jk=∂(c1,…,ck)/∂θJ_k=\partial(c_1,\ldots,c_k)/\partial\theta to describe which coefficient values change under an infinitesimal parameter variation. The relation between this Jacobian, symmetry generators, and the Hessian is derived in § 3. To specify the information calculation, associate the partial task with independent Gaussian observations

zi=ci(θ)+εi,εi∼N(0,ς2),i=1,…,k,(7)z_i=c_i(\theta)+\varepsilon_i,\qquad\varepsilon_i\sim\mathcal{N}(0,\varsigma^2),\qquad i=1,\ldots,k, \tag*{(7)}

with fixed variance. Let pθ,k(z)p_{\theta,k}(z) denote their joint density. The Fisher information is

Ik(θ)=Ez∼pθ,k[∇θlog⁡pθ,k(z)∇θlog⁡pθ,k(z)⊤].(8)I_k(\theta)=\mathbb{E}_{z\sim p_{\theta,k}}\left[\nabla_\theta\log p_{\theta,k}(z)\nabla_\theta\log p_{\theta,k}(z)^{\top}\right]. \tag*{(8)}

It measures the local statistical distinguishability of nearby parameter settings within the specified observation model [7, 2, 16]. We call the pullback quadratic form v⊤Ikvv^{\top}I_kv the Fisher metric; it may be degenerate in redundant parameter coordinates. We compare its null and positive directions with those of the error Hessian. Their agreement is to be established under the stated conditions; it is not part of the definition. The distinction matters once a penalty is added: its curvature can make a direction massive without introducing additional statistical distinguishability.

Assumptions and analysis strategy

The exact local analysis assumes jointly realizable constraints, twice continuously differentiable coefficient functionals, and regular exact solutions: points of Mk\mathcal{M}_k at which rank⁡Jk=k\operatorname{rank}J_k=k, so the retained constraint differentials are independent. Smooth group actions are used where symmetry generators are calculated. The Gaussian observation model fixes the statistical meaning of the Fisher calculation. Singular configurations, dependent constraints, and settings without an explicit group description require separate treatment. § 3 applies this construction to a two-feature factorized model and computes its minimum sets, symmetry actions, and curvature and information modes. We then keep the task fixed and examine how an L2L_2 penalty competes with its feature requirements (§ 4.1) and which curvature it adds without information (§ 4.1 and § 5.2). The BBP connection examines how finite noisy data resolve feature directions to different degrees, providing a statistical counterpart to exact feature inclusion (§ 4.2), and the two probes combine into a learnability phase diagram (Sec. 4.3). The later extensions distinguish the results established for linear networks from the proposed implications for nonlinear networks and other learning systems.

Explicit symmetry breaking produces positive Fisher information

We now realize the construction of § 2 in a model with four parameters and two features. Its purpose is to make the same event visible in three descriptions: each added feature constraint reduces the minimum set, removes a task-preserving freedom, and creates a positive Hessian and Fisher mode. The architectural freedom to represent a fixed function remains throughout. For the derivation of the explicit symmetry breaking when features are introduced and the resulting flat-to-massive transition in the spectra of the error Hessian and the Fisher metric, we use the filtration scheme introduced in § 2.2.

A minimal model and the partial tasks

The minimal model has four parameters θ=(a1,b1,a2,b2)\theta=(a_1,b_1,a_2,b_2) and two coefficients, one per feature. Its realized predictor is fθ(x)=p1x1+p2x2f_\theta(x)=p_1x_1+p_2x_2, with

ci(θ)=pi=aibi,i=1,2,(9)c_i(\theta)=p_i=a_ib_i,\qquad i=1,2, \tag*{(9)}

Take targets η1>η2>0\eta_1>\eta_2>0. Thus E1=12(p1−η1)2E_1=\frac{1}{2}(p_1-\eta_1)^2 and E2=12∑i=12(pi−ηi)2E_2=\frac{1}{2}\sum_{i=1}^{2}(p_i-\eta_i)^2. For whitened inputs, E[xxT]=I\mathbb{E}[xx^{\mathsf{T}}]=I, the latter is also the population error against the noiseless target f∗(x)=η1x1+η2x2f_*(x)=\eta_1x_1+\eta_2x_2: E2=12Ex[(fθ(x)−f∗(x))2]E_2=\frac{1}{2}\mathbb{E}_x[(f_\theta(x)-f_*(x))^2]. Here population error means an expectation over the input distribution, rather than an average over a finite sample. The diagonal factorization excludes hidden-unit mixing; the full matrix model is introduced in § 5. The architectural symmetry (2) reduces here to

(ai,bi)⟼(gai,bi/g),g∈GL(1),(10)(a_i,b_i)\longmapsto(ga_i,b_i/g),\qquad g\in\mathrm{GL}(1), \tag*{(10)}

under which pip_i is unchanged. For nonzero products its orbits are the hyperbolas aibi=consta_ib_i=\mathrm{const} in the (ai,bi)(a_i,b_i) plane, all points on a given hyperbola have the same coefficient pip_i: EkE_k is constant along the curve and the Fisher metric has a null direction tangent to it. At an exact fit of an imposed feature, the transverse direction changes pip_i and has positive error curvature. Writing ∇pi=(bi,ai)\nabla p_i=(b_i,a_i) makes the pair explicit,

n^=(bi,ai)ai2+bi2(changes pi),k^=(ai,−bi)ai2+bi2(preserves pi),(11)\hat{n}=\frac{(b_i,a_i)}{\sqrt{a_i^2+b_i^2}}\quad\text{(changes }p_i\text{)},\qquad \hat{k}=\frac{(a_i,-b_i)}{\sqrt{a_i^2+b_i^2}}\quad\text{(preserves }p_i\text{)}, \tag*{(11)}

and n^⋅k^=0\hat{n}\cdot\hat{k}=0, so the gauge tangent k^\hat{k} and the normal n^\hat{n} are orthogonal. They are respectively Hessian-flat and massive on the nonzero exact-fit hyperbola of an imposed feature; loss invariance along a curved orbit alone does not imply a zero ordinary Hessian eigenvalue away from stationarity. The two directions distinguish reparametrization along a fixed-product orbit from a change of the coefficient pip_i. The contingent symmetry appears while a feature is unconstrained. Before the second constraint is imposed, nothing in the objective depends on p2p_2: at an exact fit of feature 1, every value of p2p_2 is compatible with the partial task, so varying it leaves E1=0E_1=0. Once p2=η2p_2=\eta_2 is imposed, it does. Explicitly, let D\mathcal{D} consist of the independent rescalings (ai,bi)⟼(giai,hibi)(a_i,b_i)\longmapsto(g_i a_i,h_i b_i), gi,hi≠0g_i,h_i\ne0. The groups (5) are then

Gk={(g1,h1,g2,h2):gihi=1 for i≤k},dim⁡Gk=4−k,(12)G_k=\{(g_1,h_1,g_2,h_2):g_ih_i=1\text{ for }i\leq k\},\qquad\dim G_k=4-k, \tag*{(12)}

so G0⊃G1⊃G2G_0\supset G_1\supset G_2, with G2G_2 the architectural symmetry (10) of both features. Through a point of Mk\mathcal{M}_k with all ai,bi≠0a_i,b_i\ne0, the GkG_k-orbit has dimension 4−k4-k, that of Mk\mathcal{M}_k: it fills the solution set locally, and its tangents are exactly the null directions of the Hessian Hk=∇2EkH_k=\nabla^2E_k at that exact solution. The factor lost in the step G1→G2G_1\to G_2, the rescaling of p2p_2 by g2h2g_2h_2, is the contingent symmetry that imposing the second feature breaks. The transition from one state to the other is the elementary breaking, and Figure 2 shows both sides of it.

Figure 2. Imposing features one at a time. Each row shows the loss L=∑i=1212ρi(aibi−ηi)2L = \sum_{i=1}^{2} \frac{1}{2}\rho_i(a_i b_i-\eta_i)^2 with targets η=(1.2,0.8)\eta=(1.2, 0.8), in the factor coordinates (a1,b1)(a_1,b_1) (left) and (a2,b2)(a_2,b_2) (centre), and in the coefficient coordinates pi=aibip_i=a_i b_i (right). Top row: only feature 1 is imposed, (ρ1,ρ2)=(1,0)(\rho_1,\rho_2)=(1,0). Bottom row: both features are imposed, (ρ1,ρ2)=(1,1)(\rho_1,\rho_2)=(1,1). Colours show the error on a shared logarithmic scale. A grey panel marks a feature that is not imposed, so the loss does not depend on its factors.

Factor panels: for an imposed feature, the dashed white hyperbola aibi=ηia_i b_i=\eta_i is the set of exact minima, and the white dot is the balanced representative ai=bi=ηia_i=b_i=\sqrt{\eta_i}. The arrow PiP_i is normal to the hyperbola and changes pip_i; its Hessian eigenvalue is h(Pi)=ρi(ai2+bi2)=2ρiηih(P_i)=\rho_i(a_i^2+b_i^2)=2\rho_i\eta_i. The arrow SiS_i is tangent to the gauge orbit (ai,bi)↦(gai,bi/g)(a_i,b_i)\mapsto(g a_i,b_i/g), which leaves pip_i unchanged, so h(Si)=0h(S_i)=0. For a feature that is not imposed, both directions are unconstrained (purple), the dashed grey hyperbola is shown only for reference, and the black dot marks an arbitrary configuration.

Coefficient panels: the star marks the target η\eta and the open circle the represented coefficients p∗p^*. An imposed coefficient has curvature hp(Pi)=ρih_p(P_i)=\rho_i. A coefficient that is not imposed is a flat direction (purple double arrow), so p2p_2 is undetermined in the top row. With both features imposed, p∗p^* coincides with the target.

Reducing contingent symmetries generates a nested learning hierarchy

The contingent groups form a nested chain

G0⊇G1⊇⋯⊇Gr,(13)G_0 \supseteq G_1 \supseteq\cdots\supseteq G_r, \tag*{(13)}

maximal when no feature is imposed (k=0k=0) and minimal when all are (k=rk=r). Imposing the (k+1)(k+1)-th feature is the step Gk→Gk+1G_k \to G_{k+1}: the group loses transformations that change the new feature, and the set of realizations of the current accuracy, made up of orbits of equivalent parameter configurations, contracts to the set satisfying the additional constraint. Across the filtration this is explicit symmetry breaking in the sense of § 2.3: the objective itself loses invariance. Note that the modes are independent, so the ordering need not be unique. Instead of considering the models that capture feature 1 and feature 2 as a subset of the models that only capture feature 1, we can construct the nesting, starting with feature 2.

One independent constraint increases Hessian and Fisher rank by one

The added constraint detects a variation that previously left all retained coefficients unchanged. To count the resulting directions, consider a small parameter displacement ν\nu: the coefficient Jacobian JkJ_k maps it to the first-order changes of the first kk coefficients. At a regular exact solution, the displacements that remain tangent to the solution set are therefore precisely those annihilated by JkJ_k,

TθMk=ker⁡Jk(θ).(14)T_\theta\mathcal{M}_k = \operatorname{ker} J_k(\theta). \tag*{(14)}

If the first kk constraints have independent differentials, the local dimension of this set is D−kD-k, and a further independent scalar constraint removes one tangent direction. At an exact solution of the first kk constraints,

Hk=JkTJk,ker⁡Hk=ker⁡Jk.(15)H_k = J_k^{\mathsf{T}} J_k,\qquad\operatorname{ker} H_k = \operatorname{ker} J_k. \tag*{(15)}

To compare successive levels at the same point, take θ∈Mk+1\theta\in\mathcal{M}_{k+1} and set u=∇ck+1(θ)u = \nabla c_{k+1}(\theta). Then

Hk+1=Hk+uuT,vTHk+1v=(uTv)2for v∈ker⁡Hk.(16)H_{k+1} = H_k + uu^{\mathsf{T}},\qquad v^{\mathsf{T}} H_{k+1} v = (u^{\mathsf{T}}v)^2 \quad\text{for } v \in\operatorname{ker} H_k. \tag*{(16)}

An old flat direction acquires curvature precisely when it changes the new feature to first order. If uu is independent of the preceding constraint gradients, the number of positive Hessian eigenvalues increases by one and the remaining tangent directions stay flat; the positive eigenvectors can rotate unless the constraint gradients are orthogonal. A continuous symmetry that preserves the imposed features has a generator with JkX=0J_kX = 0, hence HkX=0H_kX = 0 at a fitted solution; after the additional constraint the same direction has curvature (uTX)2(u^{\mathsf{T}}X)^2. For the Gaussian observations (7) the expected Fisher information is

Ik=1ς2JkTJk,Hk=ς2Ikat exact fit,(17)\mathcal{I}_k = \frac{1}{\varsigma^2}J_k^{\mathsf{T}}J_k,\qquad H_k = \varsigma^2\mathcal{I}_k \quad\text{at exact fit}, \tag*{(17)}

so the curvature gained in (16) is a new direction of statistical distinguishability. In the minimal model the nonzero entries of the Jacobian row for feature ii is ∇pi=(bi,ai)\nabla p_i = (b_i,a_i), so imposing it adds exactly the massive direction n^\hat{n} of (11) and leaves k^\hat{k} flat. Away from exact fit the error Hessian also contains residual-dependent terms, and a penalty contributes curvature of its own; neither can be identified with Fisher information (§ 4.1).

Feature-induced massive modes are emergent learning directions

The result of § 3.3 identifies the same newly constrained parameter variation in the error Hessian and in the observation distribution. For a Gaussian model with fixed noise scale, the mean-squared error is the negative log-likelihood up to an additive constant and a positive factor (Bishop, [7]): minimizing the loss is maximum-likelihood estimation. Consequently, the loss Hessian is proportional to the Hessian of the negative log-likelihood, whose expectation under the model is the Fisher-information matrix (Amari, [2]; Martens, [16]); the negative log-likelihood Hessian for the observed sample is the observed information, including when evaluated at an optimum (Efron and Hinkley, [9]). For the feature observations of Eq. (7), exact fit removes the residual term. For a correctly specified Gaussian population loss, fitting the conditional mean instead makes its expectation vanish. With matching sum or average conventions,

∇2E∣exact fit=ς2I(θ),I(θ)=E[∇log⁡pθ∇log⁡pθT],(18)\nabla^2 E\big|_{\text{exact fit}} = \varsigma^2\mathcal{I}(\theta),\qquad\mathcal{I}(\theta) = \mathbb{E}\left[\nabla\log p_\theta\nabla\log p_\theta^{\mathsf{T}}\right], \tag*{(18)}

where ς2\varsigma^2 is the noise variance. On locally identifiable statistical coordinates, Fisher information defines a Riemannian metric; its pullback to redundant parameters is generally singular (Amari, [2]): for a fixed parametrization, its large-eigenvalue directions are the parameter combinations most tightly constrained locally by the data, and its null directions have zero first-order statistical distinguishability. At singular points they may still be distinguishable at higher order. This turns the flat and massive directions of the filtration into statements about information. A symmetry direction is one along which the model distribution does not change, so by (18) it is a null direction of I\mathcal{I}: the corresponding parameter combination is non-identifiable and carries zero information. Unbroken contingent and reparametrization symmetries give degenerate directions of the Fisher metric, the setting of singular learning theory [25]. In the minimal model at exact solutions with all factors nonzero they give all of them, Eq. (12); in deep linear networks they need not (§ 5.2). Conversely, a direction that becomes massive when a feature is imposed, Eq. (16), is a null direction of I\mathcal{I} turning positive: that parameter variation now changes the observation distribution to first order, so its Fisher quadratic form is positive. This is local statistical distinguishability along a direction, not unique identification of the entire parameter vector. For an independent Gaussian channel yi=pixi+ϵiy_i = p_i x_i + \epsilon_i, with xi∼N(0,1)x_i \sim\mathcal{N}(0,1) and noise variance ς2\varsigma^2, the mutual information is 12log⁡(1+pi2/ς2)\frac{1}{2}\log(1+p_i^2/\varsigma^2). This is a different quantity from parameter Fisher information. It vanishes when the represented coefficient is zero, not merely when the corresponding constraint is absent. For a fixed teacher channel with coefficient ηi\eta_i, the input-output mutual information is fixed by the data distribution and does not turn on during training. We call the newly Fisher-positive direction in the auxiliary filtration an emergent learning direction: an independent constraint makes a previously task-flat variation locally distinguishable. Penalty-lifted architectural directions remain Fisher-null and therefore do not become learning directions in this sense (§ 4.1).

Detecting learning directions

So far the symmetry has been identified and the consequences of its breaking computed, but nothing has been said about when a breaking occurs. The chain (13) is a sequence of subgroups; it does not say what moves a system along it, nor whether every step in it is reachable. Two established phenomena answer these two questions, and in doing so give the principle empirical content. The first is the cascade of transitions driven by an L2L_2 penalty, which supplies a control parameter: it lets us compare which features are represented across penalized objectives and locate transitions along an ideal path of minimizers. The second is the detachment of a signal direction from a noise bulk, which supplies a precondition: in the specified spiked ensemble, spectral recovery of a population feature requires its signal to exceed the relevant threshold. This is a condition for that recovery mechanism, not a universal impossibility result for every learner. Within the stated model, they compare feature activation with statistical resolution. Because the penalty also contributes curvature of its own, we separate the modes it introduces from the learning directions (§ 4.1), and we close with a phase diagram in regularization strength and data-noise amplitude (Sec. 4.3) as a temperature analog. That both are measurable without reference to the group is what makes the principle testable outside the setting where it was derived, and we return to that in § 6.

The L2 cascade: regularization helps locate learning directions

The chain (13) is a sequence of subgroups but says nothing about when a system moves along it. A control parameter supplies that. We add an L2L_2 penalty of strength β\beta to the error of the full task, E=ErE = E_r,

Lβ(θ)=E(θ)+β2∥θ∥2,β≥0,(19)\mathcal{L}_{\beta}(\theta) = E(\theta) + \frac{\beta}{2}\lVert\theta\rVert^2,\qquad\beta\ge0, \tag*{(19)}

so that in the minimal model an active minimum for 0<β<ηi0 < \beta< \eta_i sits at ai=bi=±ηi−βa_i = b_i = \pm\sqrt{\eta_i-\beta} and is lost at the critical strength βc(i)=ηi\beta_c^{(i)} = \eta_i. A regularization sweep varies β\beta while keeping the data error EE fixed. Analytically, we compare global minimizers θβ∈arg min⁡θLβ\theta_\beta\in\operatorname*{arg\,min}_{\theta}\mathcal{L}_\beta. In a training experiment, the optimizer, initialization, sweep rate, and relaxation time determine which minima are reached. The resulting transitions have been characterized. They are observed in general, nonlinear networks [11] (Ersoy et al. 2026), and noise-driven escape from metastable phases has been proposed as a mechanism for grokking in the models studied in [12]. In the aligned deep-linear setting, the transitions are continuous for one hidden layer and first order for two or more [14, 27]; depth alone does not establish this classification for arbitrary nonlinear architectures. In the aligned separable setting with sufficient width and distinct strengths, lowering β\beta activates features in descending order of ηi\eta_i; increasing β\beta suppresses them in the reverse order (Appendix § 9). Along this path the data error is fixed, while the penalized objective changes and is not invariant under the full contingent group, so it does not implement the explicit contingent symmetry reduction of the filtration; it selects which features are represented. Its phase with features 1,…,k1,\ldots,k learned corresponds to level kk of the filtration: the same features, with active coefficients pi=ηi−βp_i=\eta_i-\beta in this two-factor model. For each factor pair the penalty reduces the reciprocal-scaling symmetry GL(1)\mathrm{GL}(1) to O(1)≃Z2\mathrm{O}(1)\simeq\mathbb{Z}_2. This surviving sign symmetry, (ai,bi)↦(−ai,−bi)(a_i,b_i)\mapsto(-a_i,-b_i), is architectural because it preserves pip_i; it is not the contingent freedom to change an unconstrained coefficient. A nonzero minimum spontaneously selects one of its two sign-related representatives: above βc(i)\beta_c^{(i)} the minimum ai=bi=0a_i=b_i=0 is invariant under it, below it the two minima ai=bi=±ηi−βa_i=b_i=\pm\sqrt{\eta_i-\beta} are exchanged by it. Each transition can therefore be labelled by the same added feature as a step Gk→Gk+1G_k\to G_{k+1} of the auxiliary filtration (§ 3.2); it is not itself that explicit change of task symmetry. This is the first thing the cascade establishes about the principle: the feature ordering has a corresponding sequence of transitions in this solvable model, and the locations are computable in the solvable case and measurable in general. The second is that, for a uniform penalty, the ordering is forced rather than incidental. Because the ηi\eta_i then order the βc(i)\beta_c^{(i)}, the levels are reached in a definite sequence, and the continuous sweep of global minimizers cannot skip a feature threshold in this separable model, which is the content of § 5.3, seen from the side of the control parameter. We use quasi-static global-minimum path for this ideal comparison of global minimizers at successive β\beta. It need not be reproduced by slowly continuing one training run: following a local minimum can retain a metastable state and produce hysteresis. At finite Langevin temperature a trajectory can also visit both sign-related minima; the coefficient pip_i is invariant under this sign exchange. The penalty also lifts directions that remain Fisher-null, and the minimal model shows both kinds of mass side by side. At an active balanced minimum the reparametrization direction k^\hat{k} of Eq. (11) acquires curvature 2β2\beta in HLH_{\mathcal{L}} (Figure 3), yet ∇pi⋅k^=0\nabla p_i\cdot\hat{k}=0, so it remains a null direction of the Fisher metric: a pseudo-Goldstone mode of the GL(1)\mathrm{GL}(1) symmetry that the penalty breaks explicitly. Before the feature is learned, at ai=bi=0a_i=b_i=0, the Hessian of Lβ\mathcal{L}_{\beta} has eigenvalues β±ηi\beta\pm\eta_i while the Fisher metric vanishes; the smaller one goes soft exactly at βc(i)=ηi\beta_c^{(i)}=\eta_i, and below it the direction n^\hat{n} becomes Fisher-positive. The penalty-lifted gauge mode is thus Fisher-null, whereas the normal n^\hat{n} of an active feature is Fisher-positive. This does not identify all penalty curvature with Fisher-null directions. Continuous hidden-unit rotations in the full matrix model add a symmetry-generated flat sector; the full classification is in § 5.2. Table 1 keeps the unregularized toy result, its penalized counterpart, and the full matrix calculation separate.

Figure 3. Regularization shrinks and ultimately suppresses each represented feature. Each panel shows Lβ,i=12(aibi−ηi)2+β2(ai2+bi2)\mathcal{L}_{\beta,i} = \frac{1}{2}(a_i b_i-\eta_i)^2 + \frac{\beta}{2}(a_i^2+b_i^2) in the factor plane (ai,bi)(a_i,b_i), with all feature weights ρi=1\rho_i=1. The symbol λ\lambda in the panels denotes the regularization strength β\beta used in the text. The top row shows feature 1 (η1=1.2\eta_1=1.2) and the bottom row feature 2 (η2=0.8\eta_2=0.8). Regularization increases from left to right, with β=0.3,1.2,2\beta=0.3, 1.2, 2 in the top row and β=0.2,0.8,1.5\beta=0.2, 0.8, 1.5 in the bottom row, as labelled in each panel. Colours show the loss above the panel’s minimum on a logarithmic scale shared by all panels, so dark regions lie close to a minimum. The black dashed hyperbola aibi=ηia_i b_i=\eta_i is the unregularized exact-fit set. For 0<β<ηi0<\beta<\eta_i (left), the white dot marks the positive regularized minimum ai=bi=ηi−βa_i=b_i=\sqrt{\eta_i-\beta}; the sign-related minimum ai=bi=−ηi−βa_i=b_i=-\sqrt{\eta_i-\beta} appears as the second dark well in the opposite quadrant. At the marked minimum, the normal PiP_i (red or orange) has HLH_{\mathcal{L}} eigenvalue 2(ηi−β)2(\eta_i-\beta) and positive Fisher norm. The gauge tangent SiS_i (green) has HLH_{\mathcal{L}} eigenvalue 2β2\beta but remains Fisher-null. At β=ηi\beta=\eta_i (middle), the two minima merge at the origin and one Hessian eigenvalue vanishes. For β>ηi\beta>\eta_i (right), the origin is the unique minimum and the feature coefficient is zero. Reading from right to left therefore illustrates continuous feature activation in this two-factor model.

SettingSymmetry or controlHessian and Fisher statement
Bilinear partial tasks EkE_k (§ 3)An added constraint reduces contingent freedom; reciprocal rescalings survive.At a common regular exact fit, Hk=ς2IkH_k = \varsigma^2 I_k; each independent scalar constraint increases both ranks by one.
Bilinear Lβ\mathcal{L}_{\beta} (§ 4.1)The full task is fixed. Positive β\beta leaves the architectural sign symmetry Z2\mathbb{Z}_2 per feature.At an active minimum, the gauge tangent has HLH_{\mathcal{L}} eigenvalue 2β2\beta and zero Fisher norm. At the origin, onset eigenvalues are β±ηi\beta\pm\eta_i while Fisher information is zero.
Full two-factor matrix Lβ\mathcal{L}_{\beta} (§ 5.2)The architectural group retained by the norm penalty is O(w)O(w). Hidden-unit mixing is allowed.Nonzero orbit tangents give flat modes of HLH_{\mathcal{L}}. The four-sector formulas apply under the stated one-hidden-layer assumptions.

Table 1. The three settings and the curvature statements established in each.

Noise tuning: BBP transitions give a smooth filtration

The second detector asks a different question. A spectral estimator can recover a population feature only when its direction remains resolvable against the fluctuations of the empirical cross-covariance. The threshold below concerns this estimator in a specified large-matrix ensemble. Random matrix theory makes this sharp. For a linear network trained with squared loss on empirically whitened inputs, the parameter-dependent part of the loss depends on the targets through the empirical input–output cross-covariance Σ^yx=1N∑μ=1Nyμxμ⊤\hat{\Sigma}_{yx} = \frac{1}{N}\sum_{\mu=1}^{N} y_{\mu}x_{\mu}^{\top}, an nout×ninn_{\mathrm{out}} \times n_{\mathrm{in}} matrix estimated from NN samples. For an empirically whitened fixed design XX, with XX⊤/N=IXX^{\top}/N = I, and labels Y=SX+ϵY = SX + \epsilon with independent Gaussian noise entries of variance σ2\sigma^2, the conditional cross-covariance is a spiked matrix,

Σ^yx=S+TZ,S=∑iηiuivi⊤,T=σnin/N.(20)\hat{\Sigma}_{yx} = S + TZ,\qquad S = \sum_i \eta_i u_i v_i^{\top},\qquad T = \sigma\sqrt{n_{\mathrm{in}}/N}. \tag*{(20)}

Here the target map SS specifies the conditional mean E[y∣x]=Sx\mathbb{E}[y \mid x] = Sx; it equals the population cross-covariance when E[xx⊤]=I\mathbb{E}[xx^{\top}] = I. Without empirical whitening, the signal term is SΣ^xxS\hat{\Sigma}_{xx} and the conditional noise covariance also depends on Σ^xx\hat{\Sigma}_{xx}. The signal map SS has fixed low rank, its nonzero singular values are distinct feature strengths ηi\eta_i, and its singular vectors specify the feature directions.

The noise TZTZ is the fluctuation that finitely many noisy samples add: ZZ has independent zero-mean entries of variance 1/nin1/n_{\mathrm{in}}, and the amplitude TT grows with the label noise and falls with the number of samples. The results below hold for large dimensions at fixed aspect ratio q=nout/nin≤1q = n_{\mathrm{out}}/n_{\mathrm{in}} \le1; we restrict the formulas to this variance and aspect-ratio convention. Without a signal, the singular values of Σ^yx\hat{\Sigma}_{yx} fill a bulk with upper edge T(1+q)T(1+\sqrt{q}). A spike of strength η\eta separates from it only when its signal-to-noise ratio γ=η/T\gamma= \eta/T exceeds

γc=q1/4,(21)\gamma_c = q^{1/4}, \tag*{(21)}

the BBP transition [4, 6]. Below the threshold the recovered singular vectors are asymptotically uncorrelated with the true ones; above it, let u~i\tilde{u}_i be the unit empirical left singular vector associated with spike ii and uiu_i its true output direction. Their squared overlap ∣u~i⊤ui∣2|\tilde{u}_i^{\top}u_i|^2 is the squared cosine of their angle and converges to

ρ(γ)=γ4−qγ4+qγ2,(22)\rho(\gamma) = \frac{\gamma^4-q}{\gamma^4+q\gamma^2}, \tag*{(22)}

rising from zero at threshold toward one [6]. The corresponding input overlap ∣v~i⊤vi∣2|\tilde{v}_i^{\top}v_i|^2 converges to (γ4−q)/(γ4+γ2)(\gamma^4-q)/(\gamma^4+\gamma^2). The two coincide for a square matrix, q=1q = 1, the case used in Sec. 4.3, where feature ii stops being resolvable at T=ηiT = \eta_i. Here TT controls data noise and therefore the spectral resolution of a population feature. It differs from a dynamics-side temperature TeffT_{\mathrm{eff}} or TLT_L, which controls fluctuations of the parameters on a given landscape. Vanishing overlap means that the empirical singular vector is asymptotically uncorrelated with the population feature under these assumptions; it does not imply that every estimator fails or that the data Fisher matrix vanishes. The connection to § 3 is therefore an effective modelling choice: in Sec. 4.3 we use ρi(T)\rho_i(T) as the weight of the corresponding feature error. This provides a soft counterpart of the exact filtration. It is not an identity between spectral overlap and Fisher information. For a uniform penalty, the ordered ηi\eta_i give the same ordering of the resolution and activation thresholds in this effective model; unequal penalty weights can make the orderings compete.

The beta-T phase diagram

The parameter TT is the data-noise amplitude in Eq. (20). A limited temperature analogy follows from the Gaussian likelihood, not from an identification with dynamical noise. For a data set

DD of NN independent labels with noise variance σ2\sigma^2 and mean half-squared empirical error EempE_{\mathrm{emp}}, Bayes’ rule gives

p(θ∣D)∝π(θ)exp⁡[−NEemp(θ)σ2],σ2N=T2nin,(23)p(\theta\mid D) \propto\pi(\theta)\exp\left[-\frac{N E_{\mathrm{emp}}(\theta)}{\sigma^2}\right], \qquad\frac{\sigma^2}{N}=\frac{T^2}{n_{\mathrm{in}}}, \tag*{(23)}

where π(θ)\pi(\theta) is a specified prior [7]. This is a likelihood scale for the stated normalization. It neither derives the feature weighting below nor identifies TT with a Langevin temperature; in particular, it does not establish escape from a basin when a spectral overlap vanishes.

To illustrate competition between feature strength and regularization, we allow the two feature channels to carry different penalty weights si>0s_i>0. The data strengths remain ηi\eta_i; a larger sis_i simply makes that channel more costly to represent. We choose s=(3,1)\boldsymbol{s}=(3,1) so that regularization penalizes the stronger feature more heavily, allowing the phase boundaries to cross. The ordinary L2L_2 penalty is recovered by setting both weights to one. The phase diagram uses the effective loss

Lβ,T=12∑i=12ρi(T)(pi−ηi)2+β2∑i=12si(ai2+bi2),ρi(T)=(1−T2ηi2)+.(24)\mathcal{L}_{\beta,T}=\frac{1}{2}\sum_{i=1}^{2}\rho_i(T)(p_i-\eta_i)^2+\frac{\beta}{2}\sum_{i=1}^{2}s_i(a_i^2+b_i^2), \qquad\rho_i(T)=\left(1-\frac{T^2}{\eta_i^2}\right)_+. \tag*{(24)}

The targets remain ηi\eta_i; the overlap weights the errors. Using it this way is an effective model of feature availability, not a simulation of noisy samples or a consequence of BBP alone. In particular, it differs from replacing the target by ρiηi\rho_i\eta_i, although both choices give the same zero-activation threshold. Minimizing the penalty at fixed pip_i gives βsi∣pi∣\beta s_i|p_i|. For ρi>0\rho_i>0 the minimum is

pi∗=(ηi−βsiρi(T))+.(25)p_i^*=\left(\eta_i-\frac{\beta s_i}{\rho_i(T)}\right)_+. \tag*{(25)}

For ρi=0\rho_i=0 and β>0\beta>0 the minimum is ai=bi=0a_i=b_i=0. At ρi=β=0\rho_i=\beta=0, that factor pair is unconstrained. For β>0\beta>0, feature ii has a nonzero minimizing coefficient exactly while

β<βc(i)(T)=ρi(T)ηisi,(26)\beta<\beta_c^{(i)}(T)=\rho_i(T)\frac{\eta_i}{s_i}, \tag*{(26)}

and the four regions have the feature-subset labels of the two filtrations that impose feature 1 first or feature 2 first (§ 3.2). Their regularized minimizers do not generally satisfy the original exact constraints. At T=0T=0 the thresholds are ηi/si\eta_i/s_i: this is the feature-weighted extension of § 4.1, reducing to its thresholds when si=1s_i=1. Distinct ratios give separate transitions. Raising TT lowers both thresholds through ρi\rho_i and bends the boundaries toward smaller β\beta: a feature the data only partly resolves is relinquished sooner. Because ρi\rho_i depends on ηi\eta_i while the penalty weights sis_i do not, the two boundaries need not keep their order. They cross when s1/s2>η1/η2s_1/s_2>\eta_1/\eta_2, here at T≈0.64T\approx0.64, β≈0.29\beta\approx0.29: below the crossing the penalty ordering governs and the heavily weighted feature is given up first, above it the resolution ordering governs and the weaker feature goes first. For comparison with the deterministic activation boundaries, the numerical protocol uses overdamped Langevin dynamics, discretized as

θn+1=θn−Δt ∇Lβ,T(θn)+2TLΔt ξn,ξn∼N(0,I),(27)\theta_{n+1}=\theta_n-\Delta t\,\nabla\mathcal{L}_{\beta,T}(\theta_n)+\sqrt{2T_L\Delta t}\,\xi_n,\qquad\xi_n\sim\mathcal{N}(0,I), \tag*{(27)}

with independent Gaussian kicks at successive steps and TL=10−2T_L=10^{-2} held fixed throughout the (β,T)(\beta,T) grid. The kicks perturb the parameters during optimization, whereas TT changes the prescribed error weights ρi(T)\rho_i(T). They are part of the numerical optimization protocol and do not implement the data noise represented by TT. Here “Langevin sampling” means collecting configurations from a finite-time trajectory; equilibration is not established by the protocol alone. No noisy data matrices ZZ are generated, so the numerical panel does not average over data-noise realizations.

The feature classification also has a separate threshold. For a fraction 0<α<10<\alpha<1, the deterministic minimum obeys

pi∗>αηi⟺β<(1−α)βc(i)(T),ρi(T)>0.(28)p_i^*>\alpha\eta_i \quad\Longleftrightarrow\quad\beta<(1-\alpha)\beta_c^{(i)}(T), \qquad\rho_i(T)>0. \tag*{(28)}

The implemented choice α=0.2\alpha= 0.2 therefore lowers the classification boundary to 0.8βc(i)(T)0.8\beta_c^{(i)}(T) even before finite-time or noise effects. Changing a common α\alpha rescales both deterministic curves by the same factor: the crossing value of TT is unchanged and its β\beta coordinate is multiplied by 1−α1-\alpha. This prediction for the minimum does not determine the threshold dependence of a finite-noise time average.

Figure 5. Feature selection under regularization and prescribed spectral-resolution weights. Two-feature bilinear model pi=aibip_i=a_i b_i, with targets η=(1.2,0.8)\eta=(1.2,0.8) and penalty weights s=(3,1)s=(3,1). Both panels vary β\beta and TT in the effective loss Lβ,T\mathcal{L}_{\beta,T} of Eq. (24). Colour identifies the feature subset satisfying the criterion used in each panel: neither feature (grey), only feature 1 (orange), only feature 2 (purple), or both (green). (a) Analytic classification of the global minimum. Feature ii is active when pi∗>0p_i^*>0, equivalently β<βc(i)(T)=ρi(T)ηi/si\beta<\beta_c^{(i)}(T)=\rho_i(T)\eta_i/s_i. The panel is constructed directly from these inequalities. The dashed curves show the activation boundaries and are repeated in panel (b). (b) Classification from finite-time Langevin trajectories on the same effective loss, with fixed TL=10−2T_L=10^{-2}, Δt=10−3\Delta t=10^{-3}, and 1.2×1041.2\times10^4 steps. Each cell starts from θ∼N(0,0.12)\theta\sim\mathcal{N}(0,0.1^2). After the initial 40% of the run, products pip_i are recorded every 20 steps; feature ii is classified as active when their time average exceeds 0.2ηi0.2\eta_i. There is one trajectory per cell on a 100×100100\times100 grid. The Gaussian kicks perturb parameters; the resolution weights are prescribed, and no noisy datasets are sampled. The analytic activation boundaries cross at T≈0.64T\approx0.64, β≈0.29\beta\approx0.29. The respective weights vanish at T=0.8T=0.8 and T=1.2T=1.2. The panels use different activity criteria: even at the deterministic minimum, the criterion pi∗>0.2ηip_i^*>0.2\eta_i gives the boundary β=0.8βc(i)(T)\beta=0.8\beta_c^{(i)}(T). Parameter noise and finite relaxation time can cause further differences; the numerical panel does not establish equilibration. Grid ranges, sampling indices, and random-number settings are given in Appendix § 10.

Deep linear networks

We now allow full matrix factors. A depth-NN linear network has N−1N-1 hidden layers and realizes fθ(x)=Mxf_\theta(x)=Mx with M=WN⋯W1M=W_N\cdots W_1; a two-factor network therefore has one hidden layer. We leave the diagonal minimal model for these linear and deep linear networks, where the contingent group can still be written down (see § 5.1), for one hidden layer the regularized Hessian admits the four-sector decomposition of § 5.2, and the regularization path selects feature subsets that can label a nested constraint filtration (§ 5.3). Nonlinear networks and other learning systems are discussed in § 6.

The contingent group of linear networks

In linear and deep linear networks the contingent group can be written down explicitly. With the features the singular pairs (ui,vi)(u_i,v_i) of Σyx\Sigma_{yx}, their coefficients ci(M)=uiTMvic_i(M)=u_i^{\mathsf T}Mv_i, and a pair (g,h)(g,h) acting on the end-to-end map by M↦gMhTM\mapsto gMh^{\mathsf T}, Eq. (5) becomes

Gd={(g,h)∈GL⁡(nout)×GL⁡(nin):uiTgMhTvi=uiTMvi for all M and i≤d}.(29)G_d=\left\{(g,h)\in\operatorname{GL}(n_{\mathrm{out}})\times\operatorname{GL}(n_{\mathrm{in}}): u_i^{\mathsf T}gMh^{\mathsf T}v_i=u_i^{\mathsf T}Mv_i\ \text{for all }M\text{ and }i\le d\right\}. \tag*{(29)}

Its elements rescale each constrained pair, gTui=λiuig^{\mathsf T}u_i=\lambda_i u_i and hTvi=λi−1vih^{\mathsf T}v_i=\lambda_i^{-1}v_i, act freely on the two unlearned complements, and may map learned directions into them (App. § 7), so GdG_d lies in the parabolic subgroup that preserves the complements, and its dimension

dim⁡Gd=d+(nout−d)nout+(nin−d)nin(30)\dim G_d=d+(n_{\mathrm{out}}-d)n_{\mathrm{out}}+(n_{\mathrm{in}}-d)n_{\mathrm{in}} \tag*{(30)}

falls with each added scalar constraint. Here dd counts imposed coefficients; these constraints alone do not require rank M=dM=d. The group preserves the coefficient functionals, rather than fixing one particular matrix, and a solution set need not be a single orbit. A point stabilizer would instead be the subgroup leaving one specified configuration unchanged; we do not use that name for GdG_d. Figure 6 summarizes the constraint hierarchy and separates it from the norm-preserving architectural action.

Constraint-preserving groups and nested constraint sets

Figure 6. Constraint hierarchies and their norm-preserving restrictions. (a) Upper row: the groups GkG_k, where GkG_k preserves each of the first kk coefficient functionals for every represented map. Reading from left to right removes constraints and enlarges the group, Gr⊆Gr−1⊆⋯⊆G0G_r\subseteq G_{r-1}\subseteq\cdots\subseteq G_0; reading from right to left adds constraints. Lower row: their norm-preserving restrictions Ok=Gk∩[O(nout)×O(nin)]O_k=G_k\cap[\mathrm{O}(n_{\mathrm{out}})\times\mathrm{O}(n_{\mathrm{in}})]. Vertical arrows: restriction to norm-preserving transformations, not a group projection. These restrictions preserve the specified coefficients; preservation of the full data error is a separate condition. (b) Upper row: the exact constraint sets Mk={θ:ci(θ)=ηi, i≤k}M_k=\{\theta:c_i(\theta)=\eta_i,\ i\le k\}, with M0=ΘM_0=\Theta. Each additional equality selects a subset of the preceding set. Lower row: the balanced subsets M~k={(W2,W1)∈Mk:W2TW2=W1W1T}\widetilde{M}_k=\{(W_2,W_1)\in M_k:W_2^{\mathsf T}W_2=W_1W_1^{\mathsf T}\}. Both rows are therefore nested, and M~k⊆Mk\widetilde{M}_k\subseteq M_k. A constraint set can contain multiple represented maps, matrix ranks, and symmetry orbits. Balancedness selects minimum-norm factorizations at a fixed map; it does not reduce the constraint set to a single configuration or identify it with a regularized minimum.

Regularization introduces additional massive modes

The distinction between architectural redundancy and unresolved features helps interpret the Hessian spectrum of the regularized linear network; the eigenvalues themselves require the loss calculation. Ladewig et al. [14] derived this spectrum in closed form and sorted it into four classes: massive data-dependent eigenvalues, a null-space sector, a β\beta sector from the explicitly broken symmetry, and zero eigenvalues from symmetry. For one hidden layer the eigenvalues in their Table II fall into the four sectors below; their loss, ∥⋅∥2+β∥θ∥2\lVert\cdot\rVert^2+\beta\lVert\theta\rVert^2, is twice ours, so their eigenvalues are twice those quoted here. A first pass through the symmetry classification suggests only two families: flat modes from the surviving generators, massive modes from the broken ones. The distinction between nonzero symmetry-orbit tangents and perturbations that change the product only at higher order supplies the additional separation, as the onset calculation below shows. In this section, EE in Eq. (19) denotes the squared error 12∥W2W1−Σyx∥F2\frac{1}{2}\lVert W_2W_1-\Sigma_{yx}\rVert_F^2 for whitened inputs and input-output cross-covariance Σyx\Sigma_{yx}; unlike ErE_r, it fixes every coefficient of the map, not only its rr coefficients along the features. Before setting them out, one point of bookkeeping. The symmetries of § 2 were derived from the map MM and make no reference to β\beta; the sectors below are sectors of ∇2Lβ\nabla^2\mathcal{L}_\beta, Eq. (19), at a balanced rank-dd critical point of Lβ\mathcal{L}_\beta, expressed in an aligned hidden basis. Here balanced means W2TW2=W1W1TW_2^{\mathsf{T}}W_2=W_1W_1^{\mathsf{T}}; in this basis dd factor pairs are nonzero and the remaining hidden-unit blocks vanish. The explicit sector formulas below assume one hidden layer, β>0\beta>0, and generic distinct signal strengths; positivity of exchange modes additionally requires a stable selected feature subset. The two statements are compatible but not interchangeable, because the penalty is not invariant under everything the map is. Of the architectural group (2), only the orthogonal subgroup preserves ∥θ∥2\lVert\theta\rVert^2, so only O(w)O(w) leaves Lβ\mathcal{L}_\beta invariant while all of GL(w)\mathrm{GL}(w) leaves EE invariant; the difference is what produces the pseudo-Goldstone sector. The same restriction applies to the contingent action (W2,W1)↦(gW2,W1hT)(W_2,W_1)\mapsto(gW_2,W_1h^{\mathsf{T}}), which preserves the Frobenius norms only for orthogonal g,hg,h.

  • Flat modes are tangents to architectural symmetry orbits. At a stationary point, each nonzero tangent induced by a continuous symmetry of the objective is a null direction of HLH_{\mathcal{L}}; linearly independent tangents, rather than abstract generators, determine the count. At a rank-dd critical point, dd hidden units are active and the other w−dw-d are inactive, with zero weights. Orthogonal reparametrizations (2) that rotate the active units, among themselves or into the inactive ones, change the weights but neither the end-to-end map nor the norm, so they move the configuration along an equal-loss family of critical points (minima when the point is stable); these are the flat directions, dim⁡O(w)−dim⁡O(w−d)\dim O(w)-\dim O(w-d) of them (App. § 8). In this one-hidden-layer setting, learning one more independent feature activates one more unit and reduces the point stabilizer from O(w−d)O(w-d) to O(w−d−1)O(w-d-1), the matrix counterpart of the sign symmetry of § 4.1. Rotating only the inactive units has no effect at all, since their weights are zero. Away from degeneracies and thresholds, the listed contingent generators supply no additional Hessian-null directions (Appendix § 7): those acting on the unlearned complements touch only weight blocks that are zero at the critical point, the rescalings of the learned pairs act inside the architectural orbit, and those that mix a learned direction into a complement change the end-to-end map and are Fisher-positive; their Hessian curvature is determined by the stability of the critical point.

  • Onset modes activate an inactive feature at second order. Choose an inactive hidden unit and an unrepresented target direction (ub,vb)(u_b,v_b). Perturb that unit’s input weights by εvbT\varepsilon v_b^{\mathsf{T}} and its output weights by δub\delta u_b. Since both blocks initially vanish, the first-order product differential is zero, while the exact product change is ΔM=εδ ubvbT\Delta M=\varepsilon\delta\,u_bv_b^{\mathsf{T}}. For the full two-factor objective, the resulting change is

ΔLβ=β2(ε2+δ2)−ηbεδ+12ε2δ2,(31)\Delta\mathcal{L}_\beta=\frac{\beta}{2}\left(\varepsilon^2+\delta^2\right)-\eta_b\varepsilon\delta+\frac{1}{2}\varepsilon^2\delta^2, \tag*{(31)}

so the Hessian in (ε,δ)(\varepsilon,\delta) is

\begin{pmatrix} \beta& -\eta_b\\ -\eta_b & \beta \end{pmatrix}, \qquad \lambda_\pm=\beta\pm\eta_b. \tag*{(32)}\end{pmatrix}

Fisher information vanishes on these directions at the inactive configuration because DM=0DM=0. The β\beta contribution is the penalty curvature and the ±ηb\pm\eta_b contribution comes from the data-error residual. The smaller eigenvalue crosses zero at β=ηb\beta=\eta_b. These are onset directions, not nonzero tangents of a symmetry orbit: a homogeneous action on the zero blocks leaves them zero, whereas the perturbations above create a new product. Their link to the filtration is that they change only cbc_b, which a partial task omitting feature bb leaves unconstrained. The eigenvalues follow from the full objective, not from this freedom alone. For a fully inactive path through more than two weight matrices, the product changes at order greater than two; its quadratic curvature is then only the penalty, and first-order global onset need not have a Hessian zero crossing (§ 5.3).

  • Function-changing modes have positive Fisher norm. With whitened inputs and a positive-definite Gaussian output metric, a first-order change of the end-to-end map has positive Fisher norm. At a regularized point, however, the error Hessian also contains a residual term, so its eigenvalues must be computed separately. One cannot simply declare the continuous MSE symmetry trivial: (g,h)=(cI,c−1I)(g,h)=(cI,c^{-1}I) acts identically on every map, and orthogonal rotations in true null-signal subspaces can also survive. Identifiability is determined by the product differential after such redundancies are accounted for. Among the function-changing directions are the exchange directions of Ladewig et al. [14], which tilt a learned mode aa toward an unlearned mode bb; with one hidden layer and whitened inputs their eigenvalues are ηa±ηb\eta_a \pm\eta_b.

  • β\beta-modes are pseudo-Goldstone. The L2L_2 term reduces the reparametrization group from GL\mathrm{GL} to its orthogonal subgroup, keeping only the norm-preserving maps. At the balanced critical point, the architectural tangent space splits orthogonally into tangents induced by skew-symmetric and symmetric hidden-coordinate generators. Their HLH_{\mathcal{L}} eigenvalues are respectively 00 and 2β2\beta, and both sectors are Fisher-null (Appendix § 8). Generators supported entirely on inactive hidden units induce zero tangents and are excluded from the count. Linear scaling with β\beta supports this interpretation only together with gauge-orbit alignment and vanishing Fisher norm.

The shift has a consequence for the flat sector that is worth stating, since it is easy to read “flat” as “no curvature in the error”. It is not. Let v≠0v \ne0 be tangent to the O(w)O(w) orbit at a critical point θ∗\theta^* of Lβ\mathcal{L}_{\beta}. Both EE and ∥θ∥2\|\theta\|^2 are constant along that orbit, so Lβ\mathcal{L}_{\beta} is too and ∇2Lβ(v,v)=0\nabla^2\mathcal{L}_{\beta}(v,v)=0. By Eq. (19),

∇2E(v,v)=−β∥v∥2<0.(33)\nabla^2 E(v,v)=-\beta\|v\|^2<0. \tag*{(33)}

The exactly flat directions of the regularized loss are therefore concave directions of the unregularized error, with curvature exactly −β-\beta. Flatness here is a cancellation between a concave error surface and the penalty, not an absence of curvature in EE; the Goldstone modes of the broken symmetry sit precisely where the error landscape curves downward, which is where the concave boundaries of the L2L_2 mechanism are found (Ersoy et al. (2026)). The value −β-\beta itself is automatic, since ∇2Lβ=∇2E+βI\nabla^2\mathcal{L}_{\beta}=\nabla^2E+\beta I holds everywhere; what the symmetry argument predicts is the number of these directions, dim⁡O(w)−dim⁡O(w−d)\dim O(w)-\dim O(w-d) under the stated one-hidden-layer assumptions, for β>0\beta>0 away from transitions and accidental degeneracies, and that they are the orbit directions of O(w)O(w). In short: the Hessian at a rank-dd critical point resolves into four sectors: flat, onset, massive, and pseudo-Goldstone, and all four are organized by the symmetries classified in § 2.1: exactly flat where an architectural generator survives the penalty but is broken spontaneously by the configuration, onset where the contingent freedom of an unlearned feature is realized at second order, massive where the data has already locked one in, and pseudo-Goldstone where the regularizing field has explicitly broken an architectural generator. In the terms of Ladewig et al. [14] these are, in turn, their zero eigenvalues, null-space sector, massive data-dependent sector and β\beta sector. They trace the first to the orthogonal symmetry and the last to the explicit breaking of GL(w)\mathrm{GL}(w) by the penalty; what the present account adds is the origin of the other two in the contingent freedom of the partial task, and which of the four carries information. The comparison therefore distinguishes two effects: informative task curvature and explicit gauge lifting. An onset soft mode is not itself a Goldstone mode, and a higher-order onset direction must not be identified with a nonzero symmetry-orbit tangent. The first is the introduction of a feature (§ 3.4): the mass corresponds to a nonzero Fisher eigenvalue, and the direction is an emergent learning direction. Along the one-hidden-layer cascade, the onset direction becomes soft at βc\beta_c and the active branch then has a function-changing, Fisher-positive normal direction. This is the counterpart of the emergent learning direction in the filtration, under a different comparison of states. At the collapsed threshold β=ηb\beta= \eta_b, the Fisher metric still vanishes in that direction. It becomes positive on the nonzero branch for β<ηb\beta< \eta_b. The onset eigenvalue is not a Fisher eigenvalue. The Fisher metric vanishes along the onset direction until the feature is learned, so β±ηb\beta\pm\eta_b comes entirely from the penalty (β\beta) and from the residual of the still-unlearned feature (±ηb\pm\eta_b). The vanishing is forced by symmetry: the inactive unit sits at the fixed point of the sign flip of its weights, an architectural symmetry, and at such a point the Fisher information is singular along the flipped directions [26]. The second is the introduction of the regularizer, which produces the pseudo-Goldstone β\beta-modes. They lie along the architectural orbit, on which the realized function does not change, so they remain null directions of the Fisher metric; their total curvature is 2β2\beta, comprising β\beta from the ordinary error Hessian and β\beta from the penalty Hessian at the regularized critical point. Although its value is controlled by β\beta, this curvature does not imply statistical distinguishability: these gauge directions remain Fisher-null. In this setting, each activation changes the locally identifiable parameter subspace. For the full two-factor matrix model at a canonical balanced rank-dd factorization, rank⁡I=d(nout+nin−d)\operatorname{rank} \mathcal{I} = d(n_{\mathrm{out}} + n_{\mathrm{in}} - d) under the positive observation metric. A rank increment can therefore create several Fisher-positive directions, including mixing directions. The one-mode-per-constraint identity of § 3.3 concerns one added scalar observation at a common regular point, not every rank transition of a full network.

Nested constraints and ordered regularized phases

For a fixed ordering and retained target values, the partial-task solution sets are nested. Separately, in the aligned separable model the regularization path selects feature subsets in an ordered sequence. This is what Figure 6 and Figure 7 depict; here we argue it must be so.

Nested feature constraints and the remaining freedom of representation

Figure 7. Nested feature constraints and the remaining freedom of representation. Schematic for a one-hidden-layer linear network M=W2W1M = W_2W_1 of hidden layer width ww, with a fixed ordering of the features. Grey regions: the constraint sets Mk\mathcal{M}_k of Eq. (4), nested from M0=Θ\mathcal{M}_0 = \Theta (outermost, unconstrained) to Mr\mathcal{M}_r (innermost, all rr constraints). Each step inward adds one constraint and keeps the earlier ones. Mr\mathcal{M}_r is a set of exact solutions, not a single configuration. Blue shading: the balanced subsets M~k=Mk∩B\widetilde{\mathcal{M}}_k = \mathcal{M}_k \cap\mathcal{B}, where B\mathcal{B} is the set of balanced factorizations, W2TW2=W1W1TW_2^{\mathsf{T}}W_2 = W_1W_1^{\mathsf{T}}. Arrows: the black arrows GkG_k are transformations that preserve the first kk coefficient functionals and act within Mk\mathcal{M}_k, which need not be a single orbit. The blue arrows OkO_k are the norm-preserving parts of these contingent actions. They keep both balancedness and the imposed coefficients. Vertical copies: factorizations of the same map MM, related by G~=GL⁡(w)\widetilde{G} = \operatorname{GL}(w) acting as (W2,W1)↦(W2G,G−1W1)(W_2, W_1) \mapsto(W_2G, G^{-1}W_1) (side arrows). Its orthogonal subgroup O~=O⁡(w)\widetilde{O} = \operatorname{O}(w) also preserves the parameter norm and balancedness. The copies show architectural redundancy, not further constraints. Green dotted path: connects representative configurations as constraints are added and ends at the balanced exact solution θ∗∈Mr\theta^* \in\mathcal{M}_r (magenta). The path only illustrates the filtration. It is neither a computed training trajectory nor a path of regularized minimizers; regularized minimizers can pass through the same ordered feature subsets with coefficients shifted away from the exact targets. The zero configuration lies in M0\mathcal{M}_0 but satisfies none of the constraints, since every ηi>0\eta_i > 0. Areas, distances, and vertical spacing are schematic, and the regions do not depict basins of attraction.

Logical argument. Consider the aligned separable deep-linear model with distinct singular values η1>⋯>ηr>0\eta_1 > \cdots> \eta_r > 0, uniform layerwise L2L_2 regularization, and every hidden width at least rr. Then the solution sets of the partial tasks form a nested family M0⊃M1⊃⋯⊃Mr\mathcal{M}_0 \supset\mathcal{M}_1 \supset\cdots\supset\mathcal{M}_r, and the quasi-static regularization path from the zero map toward the rank-rr target map passes through phases corresponding to them, in order and one transition per step: its rank-kk phase represents the first kk features. Only with one hidden layer is the active coefficient exactly ηi−β\eta_i - \beta; at greater depth it follows the corresponding nonlinear stationary equation. The correspondence concerns feature subsets, not membership of the regularized minimizer in the exact sets Mk\mathcal{M}_k.

The argument separates constraint-set inclusion, global threshold ordering, and local stability.

  1. Containment. A solution that fits the first k+1k+1 singular directions also fits the first kk. Hence the solution sets are nested, Mk+1⊂Mk\mathcal{M}_{k+1} \subset\mathcal{M}_k, and the symmetry groups are nested in the same order, (13). In the minimal model the modes decouple and thus one could think of differently ordered nesting. The one given here is the one taken by the investigated L2L_2 path.

  1. One feature at a time. Because the modes decouple, each singular direction has its own critical strength βc(i)\beta_c^{(i)}, and, for a uniform penalty, the ordering of the ηi\eta_i orders the βc(i)\beta_c^{(i)}. As β\beta is lowered the directions turn on one at a time, in descending order: rank increases in unit steps, never by two at once for generic ηi\eta_i. Established: Ladewig et al. [14]; the analogous order in time is the classic deep-linear result of Saxe et al. [21].

  1. The quasi-static path cannot pass early. At the rank-kk critical point every onset direction toward an unlearned feature has positive curvature while β>βc(k+1)\beta> \beta_c^{(k+1)} (with one hidden layer it is β−ηb\beta-\eta_b; with more, the penalty β\beta alone), and the exchange directions are positive as well. The rank-kk configuration is therefore a local minimum of Lβ\mathcal{L}_\beta, and comparison of the decoupled objective values, rather than local curvature alone, determines when it ceases to be the global minimum, so the quasi-static path of § 4.1, which tracks the minimum at every β\beta, cannot leave it earlier. With one hidden layer the onset is continuous, and βc(k+1)\beta_c^{(k+1)} is also where the rank-kk configuration loses stability. With two or more hidden layers the stationarity condition is nonlinear and the two configurations coexist over a finite range of β\beta ([27, 12]): the rank-kk configuration stays locally stable below βc(k+1)\beta_c^{(k+1)}, and the path leaves it there because the rank-(k+1)(k+1) configuration becomes lower. This is established for the swept path; what it does not establish is that the sublevel sets of Lβ\mathcal{L}_\beta are disconnected between ranks, nor that arbitrary dynamics follow the same sequence. Those questions require separate analysis of barriers and attraction basins. For a specified deterministic training dynamics, an attraction basin is the set of initial configurations converging to a given minimum or minimum set; it is not the exact constraint set Mk\mathcal{M}_k.

Constraint retention gives the nesting. The distinct global activation thresholds of the separable model give the ordered feature subsets along a continuous sweep of global minimizers. Local-minimum continuation can instead show metastability and hysteresis; neither path establishes nested attraction basins.

Discussion

Summary A partially specified learning task leaves contingent freedom; where a continuous action generates that freedom, adding a constraint reduces the corresponding symmetry. We classified the symmetries acting on such a system into three kinds, identified the contingent one, solved for it in deep linear networks, where it lies in a parabolic subgroup and its dimension falls with each coefficient constraint imposed, and distinguished its nontrivial action from transformations that are architectural or fix the configuration. Data distinguish the directions that change the observed map to first order. The generator actions and the loss expansion together organize the four curvature sectors found by Ladewig et al. [14]: at a critical point the contingent generators act trivially on the weights, act inside the architectural orbit, or are locked by the data into massive directions, so their fingerprint is the structure of the onset sector rather than an extra flat direction. At a common regular exact fit, adding one independent scalar constraint increases both the error-Hessian rank and the Fisher rank by one. Along the regularization path, the related gain of statistical distinguishability is determined by the product Jacobian on the active branch; the exact-fit Hessian-Fisher identity does not apply to a regularized minimum with nonzero residuals (Appendix § 8). Two measurable phenomena trace the breaking without requiring the group: a regularization sweep locates feature-selection events, and a spectral threshold characterizes recoverability in the specified noisy-data model.

Refining the task description. The coarse-graining interpretation separates redundancy in representation from unresolved functional distinctions. Architectural freedom changes how a function is represented; contingent freedom can change aspects of that function omitted by a partial task. Adding an independent scalar feature refines the description and, at a common regular exact solution, increases both the error-Hessian rank and the Fisher rank by one. For the independent Gaussian observations used here, the reverse operation has a direct statistical meaning: discarding the additional observation recovers the preceding partial model and removes its Fisher contribution. This is a concrete instance of the monotonicity of Fisher information under

parameter-independent processing of observations (Ay et al., 2015). The statistical statement concerns the observation models; the deterministic maps in Figure 1 organize the corresponding equivalences between parameters, functions, and retained feature values. This interpretation remains meaningful when the fibers are not single symmetry orbits. It describes a hierarchy of task observables, without constructing a spatial rescaling or renormalization-group flow. The regularization sweep provides a separate comparison of fitted states under the full task; it does not itself implement the coarse-graining maps.

Nonlinear networks: what survives without a group description. The argument of the paper can now be stated in its general form. The constraint-based account of § 2 applies more broadly than linear activations, but redundancy and incompleteness do not guarantee a continuous group action. The regularity and observation assumptions of § 2.4 are needed for the local rank identities even when no group is available. Without the linear-model structure, three explicit descriptions are lost: the closed form (30), the sector dimensions of App. § 8, and the identification of which generator is consumed at which transition. What is not lost is the phenomenology of § 4. A cascade of transitions under a swept β\beta remains observable, it is observed in nonlinear networks [11] (Ersoy et al. 2026), but identifying a transition with subgroup reduction requires establishing the corresponding action. Likewise, a spectral threshold is informative only when the data model and estimator meet its assumptions. The two probes are complementary: the cascade measures which features survive a penalty, whereas the spectral calculation describes recoverability in a specified noisy ensemble. Their quantitative relation must be tested for the model at hand. A consequence follows, and it concerns what they do not need. Neither measurement requires an explicit symmetry-group description, but the analytic spectral thresholds still require a suitable noise ensemble, scaling limit, and estimator. A regularization sweep and joint feature/Jacobian/Hessian measurements can be performed in nonlinear networks without assuming that their features are singular directions of an input-output covariance.

Beyond neural networks. The regular scalar-constraint identities of § 3.3 do not require linear activations, although the explicit toy groups and full-network spectra do use the stated model structure. The ingredients that drive the mechanism are two: a parameter space large enough to carry reparametrization redundancy, and unlearned directions along which a change can be absorbed into either the data or the parameters. Their realization depends on both the architecture and the task. Where the relevant unresolved variations can be identified, we expect a constraint-based description, and their breaking should produce the same signature: a Hessian that separates into a few massive, task-aligned modes and a large complement of flat ones, with the massive set growing each time a feature is learned. Related reorganizations of response spectra have been documented in physical learning systems, although their curvature operators need not be parameter Fisher matrices. Stern and coauthors show that learning imprints structure on the Hessian: the effective dimension of the response drops, the low-eigenvalue eigenvectors rotate to align with the task, and the cost and physical Hessians become coupled [23, 24]; supervised learning in such networks was framed as machine learning becoming “learning machines” with exactly this stiff/soft organization [22]. The relation of those stiff and soft response modes to uninformative parameter directions is a hypothesis. Calling a soft mode Goldstone additionally requires a nonzero tangent generated by an exact continuous symmetry.

Observables. The account predicts four things that can be checked. Turning on the L2L_2 field breaks GL→O\mathrm{GL} \to\mathrm{O} and, through the balance condition, selects a single O(w)\mathrm{O}(w)-orbit of balanced states within each fiber, so positive regularization selects balanced representatives from the degenerate fiber. This does not force a discontinuity of a chosen minimizer at β=0\beta= 0: a balanced branch can approach a balanced member of the unregularized minimum set continuously. With one hidden layer, the Hessian at a rank-dd critical point should carry four eigenvalue classes with the dimensions of App. § 8: flat, 2β2\beta, β±ηb\beta\pm\eta_b (and β\beta along directions without signal), and data-massive, including ηa±ηb\eta_a \pm\eta_b along the exchange directions. For the stated one-hidden-layer linear model this is the spectrum derived by Ladewig et al. [14]; analogous distinctions can guide nonlinear experiments, but the four exact classes and their multiplicities are not universal predictions. For one hidden layer, β>0\beta> 0, and the generic aligned configurations considered above, the symmetry-generated null space has dimension dim⁡O(w)−dim⁡O(w−d)\dim\mathrm{O}(w)-\dim\mathrm{O}(w-d) and is spanned by the orbit directions of O(w)\mathrm{O}(w); extra degeneracies must be checked separately. Along these directions the unregularized error is concave with curvature −β-\beta, Eq. (33): an algebraic consequence of HL=HE+βIH_{\mathcal{L}} = H_E + \beta I. Their orbit alignment and dimension, rather than the value −β-\beta alone, test the symmetry interpretation. The groups in (30) have decreasing dimension, but orbit dimensions also depend on the subgroup that leaves the chosen configuration unchanged (its point stabilizer); group dimension alone does not determine phase ordering or stability. Outside the solvable case, the informative test is to track the error on each feature, the data Jacobian or expected Fisher matrix, and both the error and penalized Hessians through a transition. This can establish whether that transition has the proposed mechanism. The general constraint principle alone does not require every architecture to exhibit a discrete L2L_2 cascade or a covariance-based BBP threshold.

Relation to symmetry-constraint accounts. The contingent symmetry is related to, but distinct from, a broader phenomenon: mirror symmetries can define invariant fixed subspaces whose occupation is favored by sufficiently strong regularization or gradient noise under the hypotheses of Ziyin [26]. A fixed subspace need not consist entirely of stationary points. That account concerns symmetries of the loss itself; it includes architectural examples such as hidden-unit permutations, with additional assumptions for the regularization and noise conclusions. Contingent symmetries here preserve the auxiliary partial loss, but need not preserve the full loss: they are the freedoms a not-yet-fully-specified task leaves open, they form a nested chain indexed by the retained constraints. In the minimal model, their reduction under an independent scalar constraint is tied to an increase of Fisher rank at a common regular exact solution. The regularization path provides a related but distinct comparison.

Shortcomings. Two questions remain open. § 5.3 establishes constraint-set nesting and an ordered global-minimum feature sequence under separate assumptions. It does not establish nested attraction basins, nor identify those sets with metastable phases. Relating this sequence to the dynamics studied in Ersoy and Wiesner [12] requires a separate calculation of local minima, barriers, and escape dynamics. The universality claim above is only partly tested: the transitions themselves are observed in nonlinear networks (Ersoy and Wiesner [11]; Ersoy et al. (2026)), but tracking the Hessian or Fisher spectrum through training of a nonlinear network, and checking which feature-aligned directions gain statistical distinguishability, is a natural experiment not carried out here.

Outlook. The ordered feature subsets of § 5.3 offer labels for comparing the metastable states studied in Ersoy and Wiesner [12]. Whether their attraction basins have a corresponding organization remains open. Feature acquisition, phase selection, and gain of Fisher distinguishability are related in the exact models under the stated conditions; their coincidence, and its relation to memorization and generalization, must be tested in other settings. Whether an approximate version of GdG_d can be written down past the linear, aligned-Gaussian setting in general nonlinear networks, where the transitions are observed (Ersoy and Wiesner [11]; Ersoy et al. (2026)) and the general symmetry-constraint machinery of Ziyin [26]; ); Ziyin et al. [28] already applies, is the key question for turning the general claim into a quantitative theory of how networks acquire features. The two phenomena of § 4 are what make that question answerable empirically rather than only analytically.

The coefficient-preserving group

Write m=noutm=n_{\mathrm{out}} and n=ninn=n_{\mathrm{in}}. Preserving uiTMviu_i^{\mathsf T} M v_i for every M∈Rm×nM\in\mathbb{R}^{m\times n} is equivalent to

(gTui)(hTvi)T=uiviT.(34)(g^{\mathsf T}u_i)(h^{\mathsf T}v_i)^{\mathsf T}=u_i v_i^{\mathsf T}. \tag*{(34)}

Since both sides have rank one, there is a nonzero scalar λi\lambda_i such that gTui=λiuig^{\mathsf T}u_i=\lambda_i u_i and hTvi=λi−1vih^{\mathsf T}v_i=\lambda_i^{-1}v_i. In orthonormal bases beginning with the dd constrained directions, the group therefore consists of

g=(Λ0AB),h=(Λ−10CD),(35)g=\begin{pmatrix}\Lambda&0\\ A&B\end{pmatrix},\qquad h=\begin{pmatrix}\Lambda^{-1}&0\\ C&D\end{pmatrix}, \tag*{(35)}

where Λ\Lambda is invertible and diagonal, B∈GL(m−d)B\in\mathrm{GL}(m-d), D∈GL(n−d)D\in\mathrm{GL}(n-d), and A,CA,C are unrestricted. Counting these entries gives

dim⁡Gd=d+m(m−d)+n(n−d).(36)\dim G_d=d+m(m-d)+n(n-d). \tag*{(36)}

This counts transformations preserving coefficient functionals, not independent orbit tangents at a particular configuration. In particular, the decrease of group dimension is not the number of Fisher modes gained by imposing one scalar constraint. To describe the action used in § 5.2, additionally take a canonical aligned rank-dd factorization, with nonzero product block P=diag⁡(p1,…,pd)P=\operatorname{diag}(p_1,\ldots,p_d) and all other factor blocks zero. The infinitesimal action on its product has the form

δM=(0PC0TA0P0).(37)\delta M=\begin{pmatrix}0&PC_0^{\mathsf T}\\ A_0P&0\end{pmatrix}. \tag*{(37)}

where A0,C0A_0,C_0 are the infinitesimal lower-left blocks. Complement-only transformations fix the zero factor blocks; reciprocal rescalings of active pairs induce architectural gauge tangents; and the displayed mixing blocks change the represented map. The coefficient constraints alone do not require this canonical rank-dd configuration.

Norm-preserving restrictions. Intersecting GdG_d with O(m)×O(n)O(m)\times O(n) sets A=C=0A=C=0, makes B,DB,D orthogonal, and restricts Λ\Lambda to diagonal signs. This is the restriction OdO_d in Figure 6. For orthogonal g,hg,h, invariance of the full map error 12∥M−S∥F2\frac{1}{2}\lVert M-S\rVert_F^2 for every MM additionally requires gShT=SgSh^{\mathsf T}=S. Neither restriction is the architectural hidden-unit action (W2,W1)↦(W2G,G−1W1)(W_2,W_1)\mapsto(W_2G,G^{-1}W_1), which preserves MM itself.

Hessian sectors and Fisher information

Consider one hidden layer of width ww, whitened inputs, and

Lβ=12∥W2W1−S∥F2+β2(∥W2∥F2+∥W1∥F2),S=Σyx,β>0.(38)\mathcal{L}_{\beta}=\frac{1}{2}\lVert W_2W_1-S\rVert_F^2+\frac{\beta}{2}\left(\lVert W_2\rVert_F^2+\lVert W_1\rVert_F^2\right),\qquad S=\Sigma_{yx},\quad\beta>0. \tag*{(38)}

The original population squared error differs from its first term only by a parameter-independent constant. Let v=(U,V)v=(U,V) be a perturbation of (W2,W1)(W_2,W_1), R=W2W1−SR=W_2W_1-S, and JMv=UW1+W2VJ_Mv=UW_1+W_2V. Expanding the product to second order gives

vTHLv=∥JMv∥F2+2⟨R,UV⟩F+β(∥U∥F2+∥V∥F2).(39)v^{\mathsf T}H_{\mathcal{L}}v=\lVert J_Mv\rVert_F^2+2\langle R,UV\rangle_F+\beta\left(\lVert U\rVert_F^2+\lVert V\rVert_F^2\right). \tag*{(39)}

For y∣x∼N(Mx,ς2Im)y\mid x\sim\mathcal{N}(Mx,\varsigma^2I_m) with E[xxT]=In\mathbb{E}[xx^{\mathsf T}]=I_n, the expected Fisher quadratic form is

vTIv=ς−2∥JMv∥F2.(40)v^{\mathsf T}Iv=\varsigma^{-2}\lVert J_Mv\rVert_F^2. \tag*{(40)}

Thus neither the residual term nor the penalty term belongs to the Fisher matrix; at R=0R=0, the unregularized error Hessian equals ς2I\varsigma^2 I [16]. Now evaluate at a canonical balanced rank-dd critical point, whose active factor blocks are diag⁡(pa)\operatorname{diag}(\sqrt{p_a}) with pa=ηa−β>0p_a=\eta_a-\beta>0, and whose other blocks vanish. A gauge tangent VK=(W2K,−KW1)V_K=(W_2K,-KW_1) satisfies JMVK=0J_MV_K=0. Differentiating the gauge transformation of the error gradient and using ∇E=−βθ\nabla E=-\beta\theta at the critical point yields

HLVK=βVK+K⊤.(41)H_{\mathcal{L}}V_K=\beta V_K+K^\top. \tag*{(41)}

Skew-symmetric KK therefore produces eigenvalue zero, and symmetric KK produces eigenvalue 2β2\beta, whenever the induced tangent is nonzero. Balancedness makes these two tangent spaces orthogonal. In either case, matrices supported only on the inactive hidden block induce zero tangents and must be excluded from the count.

SectorDimensionHLH_{\mathcal{L}} eigenvalueFisher norm
Orthogonal gaugewd−d(d+1)/2wd-d(d+1)/200zero
Symmetric gaugewd−d(d−1)/2wd-d(d-1)/22β2\betazero
Inactive-unit complement(w−d)(m+n−2d)(w-d)(m+n-2d)β±ηb, β\beta\pm\eta_b,\ \betazero
Function-changingd(m+n−d)d(m+n-d)data-dependentpositive

Table 1 (PDF p. 27).

The third sector couples inactive hidden units to the unrepresented input and output complements. Its first-order product differential is zero, and its quadratic blocks are Eq. (32). Writing r=rank⁡Sr=\operatorname{rank}S, each omitted nonzero singular value supplies β±ηb\beta\pm\eta_b, each with multiplicity w−dw-d; the remaining (w−d)(m+n−2r)(w-d)(m+n-2r) directions have eigenvalue β\beta. Here dd active modes are selected from the rr signal directions. The last sector is (ker⁡JM)⊥(\ker J_M)^\perp: the image of JMJ_M has arbitrary active-active and active-complement blocks, and a zero complement-complement block. Consequently

rank⁡I=rank⁡JM=d(m+n−d).(42)\operatorname{rank}\mathcal{I}=\operatorname{rank}J_M=d(m+n-d). \tag*{(42)}

The four dimensions sum to w(m+n)w(m+n). The table separates geometric subspaces, not necessarily distinct numerical eigenvalues; additional Hessian zero modes occur at thresholds or degeneracies. At saddles, Fisher-positive directions need not have positive Hessian eigenvalues. The detailed spectrum is treated in Ladewig et al. [14]. For the symmetric gauge sector, Eq. (41) also gives HEVK=βVKH_E V_K=\beta V_K: its total curvature 2β2\beta contains equal contributions from the error Hessian and the penalty Hessian.

Ordered thresholds in the aligned deep-linear model

For N≥2N\geq2 scalar factors per aligned feature and a fixed positive product pp, the arithmetic-geometric mean inequality gives ∑ℓ=1Naℓ2≥Np2/N\sum_{\ell=1}^{N}a_\ell^2\geq Np^{2/N}, with equality for equal factor magnitudes. Each feature therefore reduces to

FN(p)=12(p−η)2+βN2p2/N,p≥0.(43)F_N(p)=\frac{1}{2}(p-\eta)^2+\frac{\beta N}{2}p^{2/N},\qquad p\geq0. \tag*{(43)}

For N=2N=2, p∗=(η−β)+p^*=(\eta-\beta)_+ and the transition is continuous at βc=η\beta_c=\eta. For N>2N>2, solving FN′(pc)=0F_N'(p_c)=0 together with FN(pc)=FN(0)F_N(p_c)=F_N(0) gives

pc=N−2N−1η,βc=ηN−1(N−2N−1η)(N−2)/N.(44)p_c=\frac{N-2}{N-1}\eta,\qquad\beta_c=\frac{\eta}{N-1}\left(\frac{N-2}{N-1}\eta\right)^{(N-2)/N}. \tag*{(44)}

For N>2N>2 and p>0p>0, the sign of FN(p)−FN(0)F_N(p)-F_N(0) is the sign of β−(2ηp−p2)/(Np2/N)\beta-(2\eta p-p^2)/(Np^{2/N}). The latter ratio has its unique positive maximum at pcp_c on 0<p<2η0<p<2\eta, so the displayed value is the global activation threshold. The nonzero jump is first order, and βc\beta_c increases strictly with η\eta. Thus distinct strengths and a uniform penalty order the global activation thresholds in the separable aligned model of § 5.3. This comparison does not determine which local minimum an optimizer reaches. More generally, for any nonnegative error finite at zero, a global minimizer of E+β∥θ∥2/2E + \beta\lVert\theta\rVert^2/2 obeys ∥θβ∥2≤2E(0)/β\lVert\theta_\beta\rVert^2 \le2E(0)/\beta, establishing convergence to zero as β→∞\beta\to\infty when all parameters are penalized.

Numerical construction of the figures

Exact landscapes. Figures 2 and 3 evaluate the displayed bilinear losses directly on uniform factor-space grids in [−2,2]2[-2,2]^2, with η=(1.2,0.8)\eta=(1.2,0.8) and respectively 401 and 400 points per axis. The coefficient-space filtration panels use 401 points per axis on [−0.55,2.15]2[-0.55,2.15]^2. The filtration uses (ρ1,ρ2)=(1,0),(1,1)(\rho_1,\rho_2)=(1,0),(1,1); an omitted feature is represented by (ai,bi)=(0.6,0.2)(a_i,b_i)=(0.6,0.2), while an imposed feature uses ai=bi=ηia_i=b_i=\sqrt{\eta_i}. The regularized panels use strengths (0.3,1.2,2.0)(0.3,1.2,2.0) for feature 1 and (0.2,0.8,1.5)(0.2,0.8,1.5) for feature 2. They mark the positive balanced minimum ai=bi=(ηi−β)+a_i=b_i=\sqrt{(\eta_i-\beta)_+}; the plotting code denotes β\beta by λ\lambda. No training trajectories are used for these figures.

Illustrative rough landscapes. For Figure 4, the plotting code evaluates

Figure 4. Illustrative landscapes at three levels of spectral resolution. The target coefficients are (η1,η2)=(1.2,0.8)(\eta_1,\eta_2)=(1.2,0.8). For the square spiked-matrix model, the prescribed overlaps are ρi(T)=(1−T2/ηi2)+\rho_i(T)=(1-T^2/\eta_i^2)_+. From top to bottom, T=1.68T=1.68, 1.021.02, and 0.600.60, giving (ρ1,ρ2)≈(0,0)(\rho_1,\rho_2)\approx(0,0), (0.28,0)(0.28,0), and (0.75,0.44)(0.75,0.44). Thus neither, only the first, and both population feature directions have nonzero asymptotic spectral overlap. The left and middle columns show the factor planes (a1,b1)(a_1,b_1) and (a2,b2)(a_2,b_2); the right column shows coefficient space (p1,p2)(p_1,p_2). The star marks the target, while the red-outlined circle marks the displayed configuration p∗p^*. Its reported values are approximately (0.08,−1.22)(0.08,-1.22) in the first two rows and (0.94,−0.10)(0.94,-0.10) in the bottom row. White dots in the factor planes represent the corresponding coefficients. Black dashed hyperbolae mark aibi=ηia_i b_i=\eta_i; white curves mark aibi=pi∗a_i b_i=p_i^*. Green arrows indicate product-preserving tangents, and red or orange arrows indicate product-changing normals. These arrows identify geometric directions, without by themselves establishing Hessian eigenvalues at the marked configurations. All three columns use the same additional rough landscape, with amplitudes Anoise≈5.28A_{\mathrm{noise}}\approx5.28, 0.610.61, and 0.110.11. This roughness is distinct from the deterministic overlap weights and is not generated by the BBP overlap formula alone. The resolution labels refer to singular-vector recovery, rather than exact fitting of the displayed coefficients. The random-field construction and grid selection of p∗p^* are specified in Appendix § 10.

ET(p)=12∑i=12ρi(T)(pi−ηi)2+A(T)ξ(p),A(T)=0.055+1.15(T/1.2)4.5,(45)E_T(p)=\frac{1}{2}\sum_{i=1}^{2}\rho_i(T)(p_i-\eta_i)^2+A(T)\xi(p),\qquad A(T)=0.055+1.15(T/1.2)^{4.5}, \tag*{(45)}

where ρi(T)=(1−T2/ηi2)+\rho_i(T)=(1-T^2/\eta_i^2)_+ and ξ\xi is one fixed random Fourier field. Its raw form is ∑j=185ajcos⁡(kj⋅p+ϕj)+∑j=170a~jcos⁡(k~j⋅p+ϕ~j)\sum_{j=1}^{85}a_j\cos(k_j\cdot p+\phi_j)+\sum_{j=1}^{70}\widetilde{a}_j\cos(\widetilde{k}_j\cdot p+\widetilde{\phi}_j). Directions are sampled by normalizing independent two-dimensional standard Gaussian vectors; frequency magnitudes are log-uniform on [0.8,9][0.8,9] and [7,20][7,20], respectively. Phases are uniform on [0,2π)[0,2\pi), and amplitudes are independent Gaussians with standard deviations 85−1/285^{-1/2} and 0.35 70−1/20.35\,70^{-1/2}. The code uses NumPy’s default_rng(14) and normalizes this field once to zero mean and unit standard deviation on a 220×220220\times220 grid over [−10.25,10.35]×[−10.25,10][-10.25,10.35]\times[-10.25,10]. At T=1.68,1.02,0.60T=1.68,1.02,0.60, p∗p^* is the minimum on a 360×360360\times360 grid over the same window, without continuous refinement. The factor panels evaluate ET(a1b1,p2∗)E_T(a_1b_1,p_2^*) or ET(p1∗,a2b2)E_T(p_1^*,a_2b_2) on 360×360360\times360 grids in [−5,5]2[-5,5]^2. Logarithmic shading uses each panel’s excess loss ET−min⁡ET+ϵE_T-\min E_T+\epsilon, with ϵ=max⁡(10−8,10−3q0.9)\epsilon=\max(10^{-8},10^{-3}q_{0.9}) and upper cutoff q0.9+ϵq_{0.9}+\epsilon, where q0.9q_{0.9} is the 90th percentile of that excess loss. This additional random field is a visualization model, not a sampled cross-covariance or a consequence of BBP theory. It is absent from the phase-diagram calculation below.

Analytic phase diagram. Figure 5 uses Eq. (24), with η=(1.2,0.8)\eta=(1.2,0.8) and s=(3,1)s=(3,1), on logarithmically spaced grids of 100 values each: T∈[0.05,1.60]T\in[0.05,1.60] and β∈[0.005,2]\beta\in[0.005,2]. Since ai2+bi2≥2∣aibi∣a_i^2+b_i^2\ge2\lvert a_i b_i\rvert, minimization at fixed product reduces each feature to ρi2(pi−ηi)2+βsi∣pi∣\frac{\rho_i}{2}(p_i-\eta_i)^2+\beta s_i\lvert p_i\rvert. Panel (a) classifies its global minimum using β<βc(i)(T)=ρi(T)ηi/si\beta<\beta_c^{(i)}(T)=\rho_i(T)\eta_i/s_i. The analytic crossing is

T×=72/175≃0.6414,β×=2/7≃0.2857.(46)T_\times=\sqrt{72/175}\simeq0.6414,\qquad\beta_\times=2/7\simeq0.2857. \tag*{(46)}

Finite-time numerical panel. Panel (b) uses one finite-time trajectory per grid point. At each step, the parameters undergo a gradient-descent update followed by an independent Gaussian kick, as specified in Eq. (27). This is the Euler-Maruyama discretization of overdamped Langevin dynamics ([33]). We use a fixed noise strength TL=10−2T_L=10^{-2}, a time step Δt=10−3\Delta t=10^{-3}, and 12,000 updates. Initial parameter components are drawn independently from N(0,0.12)\mathcal{N}(0,0.1^2). The parameter-noise strength TLT_L is held constant throughout the diagram; the plotted variable TT controls the prescribed feature weights ρi(T)\rho_i(T). The gradients used are

∂aiLβ,T=ρi(pi−ηi)bi+βsiai,∂biLβ,T=ρi(pi−ηi)ai+βsibi.(47)\partial_{a_i}\mathcal{L}_{\beta,T}=\rho_i(p_i-\eta_i)b_i+\beta s_i a_i,\qquad\partial_{b_i}\mathcal{L}_{\beta,T}=\rho_i(p_i-\eta_i)a_i+\beta s_i b_i. \tag*{(47)}

All components are evaluated before updating the parameters. Products are recorded after updates whose zero-based loop indices satisfy n>4800n > 4800 and n≡0(mod20)n \equiv0 \pmod{20}, giving 359 samples per trajectory. Feature ii is active when its sample mean exceeds 0.2ηi0.2\eta_i. The implementation uses default_rng(0), traverses TT in the outer loop and β\beta in the inner loop, and assigns each cell its initialization followed by its Gaussian kicks. The dashed lines in both panels are the analytic activation boundaries. For the numerical panel’s threshold applied to a deterministic minimum, the reference boundaries would instead be 0.8βc(i)(T)0.8\beta_c^{(i)}(T), crossing at the same T×T_{\times} and β=8/35\beta= 8/35. Consequently, criterion choice, finite time, parameter noise, and discretization can all contribute to differences between the panels. TLT_L is fixed, no noisy datasets are sampled, and one trajectory per cell does not establish equilibration or sampling uncertainty.

References

  1. [1]A. Altland and B. Simons, Condensed Matter Field Theory, 3rd ed. (Cambridge Univ. Press, 2023).
  2. [2]S. Amari, Information Geometry and Its Applications (Springer, 2016).
  3. [4]J. Baik, G. Ben Arous, and S. Péché, Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices, Ann. Probab. 33, 1643 (2005).DOI
  4. [5]F. Benaych-Georges and R. R. Nadakuditi, The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices, Adv. Math. 227, 494 (2011).
  5. [6]F. Benaych-Georges and R. R. Nadakuditi, The singular values and vectors of low rank perturbations of large rectangular random matrices, J. Multivariate Anal. 111, 120 (2012).
  6. [7]C. M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006).
  7. [8]A. J. Bray and D. S. Dean, Statistics of critical points of Gaussian fields on large-dimensional spaces, Phys. Rev. Lett. 98, 150201 (2007).pubmed.ncbi.nlm.nih.gov/17501322
  8. [9]B. Efron and D. V. Hinkley, Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information, Biometrika 65, 457 (1978).DOI
  9. [11]I. T. Ersoy and K. Wiesner, Exploring L₂-phase transitions on error landscapes, in ICML Workshop on High-dimensional Learning Dynamics (HiLD 2025) (2025). https://openreview.net/forum?id=AkQNtAwO9u
  10. [12]I. T. Ersoy and K. Wiesner, Noise-driven escape from metastable phases explains grokking in deep neural networks, in ICML Workshop on High-dimensional Learning Dynamics (HiLD 2026) (2026), arXiv:2606.17120. https://openreview.net/forum?id=p3bdkPqsxvarxiv.org/abs/2606.17120
  11. [13]N. Goldenfeld, Lectures on Phase Transitions and the Renormalization Group (CRC Press, 1992).
  12. [14]B. Ladewig, I. T. Ersoy, and K. Wiesner, Cascading through the hierarchy: Regularizer-induced feature detection as phase transitions in deep linear neural networks (2026), arXiv:2608.06597. https://arxiv.org/abs/2608.06597
  13. [15]K. Lindsey and G. Menon, Regularization implies balancedness in the deep linear network (2026), arXiv:2511.01137v2. https://arxiv.org/abs/2511.01137v2
  14. [16]J. Martens, New insights and perspectives on the natural gradient method, J. Mach. Learn. Res. 21(146), 1 (2020). https://www.jmlr.org/papers/v21/17-678.html
  15. [17]G. Menon, The geometry of the deep linear network, in XIV Symposium on Probability and Stochastic Processes, edited by C. G. Higuera Chan, J. A. López Mimbela, S. I. López, and C. G. Pacheco, Progress in Probability, Vol. 81 (Birkhäuser, Cham, 2025), pp. 1–47. https://doi.org/10.1007/978-3-031-96118-2_1
  16. [18]J. Pennington and Y. Bahri, Geometry of neural network loss surfaces via random matrix theory, in Proceedings of the 34th International Conference on Machine Learning, Proc. Mach. Learn. Res. 70, 2798–2806 (2017). https://proceedings.mlr.press/v70/pennington17a.html
  17. [19]A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra, Grokking: Generalization beyond overfitting on small algorithmic datasets, in ICLR Workshop on Mathematical Reasoning in General Artificial Intelligence (2021). https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper.pdfarxiv.org/abs/2201.02177
  18. [20]Sagun, L., Evci, U., Guney, V., Dauphin, Y. & Bottou, L. Empirical Analysis of the Hessian of Over-Parametrized Neural Networks. (2018), https://openreview.net/forum?id=rJrTxwbCb
  19. [21]A. M. Saxe, J. L. McClelland, and S. Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, in International Conference on Learning Representations (ICLR) (2014). https://openreview.net/forum?id=_wzWkIpTDF_9C
  20. [22]M. Stern, D. Hexner, J. W. Rocks, and A. J. Liu, Supervised learning in physical networks: From machine learning to learning machines, Phys. Rev. X 11, 021045 (2021). https://doi.org/10.1103/PhysRevX.11.021045
  21. [23]M. Stern, A. J. Liu, and V. Balasubramanian, Physical effects of learning, Phys. Rev. E 109, 024311 (2024). https://doi.org/10.1103/PhysRevE.109.024311
  22. [24]M. Stern, M. Guzman, F. Martins, A. J. Liu, and V. Balasubramanian, Physical networks become what they learn, Phys. Rev. Lett. 134, 147402 (2025). https://doi.org/10.1103/PhysRevLett.134.147402
  23. [25]S. Watanabe, Algebraic Geometry and Statistical Learning Theory (Cambridge Univ. Press, 2009). https://doi.org/10.1017/CBO9780511800474
  24. [26]L. Ziyin, Symmetry induces structure and constraint of learning, in Proceedings of the 41st International Conference on Machine Learning, Proc. Mach. Learn. Res. 235, 62847–62866 (2024). https://proceedings.mlr.press/v235/ziyin24a.html
  25. [27]L. Ziyin and M. Ueda, Zeroth, first, and second-order phase transitions in deep neural networks, Phys. Rev. Res. 5, 043243 (2023). https://doi.org/10.1103/PhysRevResearch.5.043243
  26. [28]L. Ziyin, Y. Xu, T. Poggio, and I. Chuang, Parameter symmetry potentially unifies deep learning theory (2025), arXiv:2502.05300.
  27. [29]L. Ziyin, Y. Xu, and I. Chuang, Remove symmetries to control model expressivity and improve optimization, in International Conference on Learning Representations (ICLR) (2025), arXiv:2408.15495. https://proceedings.iclr.cc/paper_files/paper/2025/hash/040321891311e31134a1b3844efb6a9a-Abstract-Conference.html
  28. [30]B. Zhao, R. Walters, and R. Yu, Symmetry in Neural Network Parameter Spaces, Transactions on Machine Learning Research (2025), https://openreview.net/forum?id=jLPwq5QY6Iarxiv.org/abs/2506.13018
  29. [33]D. J. Higham, An algorithmic introduction to numerical simulation of stochastic differential equations, SIAM Review 43(3), 525–546 (2001). https://doi.org/10.1137/S0036144500378302

Paper details

vlnc:W_5rq2x64y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents