betaFeedback

WorksComputer Science

Symmetry Breaking at the Edge of Stability via Central Flow Analysis

Abstract

Continuous symmetries in neural networks lead to conserved quantities that are invariant under gradient flow, such as layer balancedness in linear and ReLU networks. Finite-step gradient descent need not preserve these laws, particularly at the edge of stability (EOS), where the iterates oscillate around a smoother trajectory, which can be described by the central flow dynamics (Cohen et al., 2025). We study how gradient flow conservation laws evolve under the central flow description of this regime. We show that the curvature correction induces a finite-dimensional drift determined by the oscillation covariance along critical parameter directions. We thus take the first steps to analyse which conservation laws remain and are broken at the EOS according to the central flow dynamics by analysing a scalar two-layer model, two-layer matrix factorisation, and a two-layer ReLU network, where the known conservation laws are shown to be complete (Marcotte et al., 2023).

Introduction

Continuous symmetries often leave traces in optimisation dynamics. Under gradient flow, architectural symmetries of neural networks give rise to conserved quantities. For example, Arora et al. [1] build on linear networks preserving the matrix-valued balancedness quantity Ψ=U⊤U−V⊤V\Psi= U^{\top}U - V^{\top}V, while positively homogeneous networks admit corresponding neuron-wise balancedness quantities Du et al. [5]. Marcotte et al. (2023) formalised this phenomenon intrinsically to the gradient flow. Writing the model through a reparametrisation g(θ,x)=f(ϕ(θ),x)g(\theta, x) = f(\phi(\theta), x), they characterise gradient flow conservation laws through orthogonality to the family of admissible data-generated gradient vector fields Wϕ(θ):=range⁡∂ϕ(θ)⊤W_{\phi}(\theta) := \operatorname{range}\partial_{\phi}(\theta)^{\top}, making conservation laws solely architecture-dependent and showing that the maximum number of independent laws can be computed from the dimension of the model and the Lie algebra generated by WϕW_{\phi}, namely NGF(θ)=D−dim⁡Lie⁡(Wϕ)(θ)N_{\mathrm{GF}}(\theta) = D - \dim\operatorname{Lie}(W_{\phi})(\theta). For the shallow linear and ReLU architectures considered in this work, this machinery recovers the familiar balancedness quantities and certifies their completeness.

Finite-step gradient descent does not inherit this conservation guarantee. Symmetry-induced conservation laws are known to acquire finite-step corrections Kunin et al. (2021), and gradient descent on neural networks, with learning rate η\eta, commonly enters the EOS regime, where the largest Hessian eigenvalue (or a set of eigenvalues) approaches the stability threshold λmax⁡(H)≈2η\lambda_{\max}(H) \approx\frac{2}{\eta}, and the iterates oscillate Cohen et al. [2]; ); Damian et al. [4]. Cohen et al. [3] propose the central flow as a continuous-time description of the time-averaged trajectory in this oscillatory regime. This raises the question we study: Which gradient flow conservation laws survive the central flow dynamics, and which are broken by edge-of-stability oscillations?

We take the first steps to answer this question for three increasingly structured two-layer examples: a scalar factorisation, matrix factorisation, and a ReLU network. In each case, the gradient flow conservation laws can be represented as (where ⟨C,D⟩\langle C,D\rangle of two matrices CC and DD denotes their Frobenius inner product)

ΨA=⟨A,Ψ⟩,A∈A,(1)\Psi_{A}=\langle A,\Psi\rangle,\qquad A\in\mathcal{A}, \tag*{(1)}

a matrix-valued invariant Ψ\Psi, defined in § 2 against an architecture-dependent space of admissible test matrices A\mathcal{A}. More generally, conservation law survival at a fixed θ\theta is determined by the span of the symmetry-breaking directions reachable as the admissible oscillation covariance varies, whose dimension is bounded in (17); survival along a trajectory is characterised in (14).

Our viewpoint is complementary to recent analyses of symmetry breaking in the discrete dynamics. Kunin et al. (2021) study finite-step breaking through modified-equation analysis. Recent work at the edge of stability studies balancedness and implicit bias directly in discrete dynamics, including scalar or layer-aggregated conservation quantities Medeiros (2026); Hofmann [7], while Ghosh et al. [6] analyse balancing in deep matrix factorisation beyond the edge of stability. Here the object of study is instead the central flow dynamics: we identify which directions in that finite-dimensional conservation-law space are broken by the EOS curvature field. We then compare gradient flow, discrete gradient descent, and the central flow to illustrate how the predicted breaking emerges along central flow trajectories.

Background on symmetries and gradient flow conservation laws

We briefly recall the framework we use following Marcotte et al. (2023, 2024). Suppose the model admits a reparametrisation g(θ,x)=f(ϕ(θ),x)g(\theta,x)=f(\phi(\theta),x), so that, for a loss of the form L=F∘ϕL=F\circ\phi,

∇L(θ)=∂ϕ(θ)⊤∇F(ϕ(θ))∈Wϕ(θ),Wϕ(θ):=range⁡(∂ϕ(θ)⊤).(2)\nabla L(\theta)=\partial\phi(\theta)^{\top}\nabla F(\phi(\theta))\in W_{\phi}(\theta),\qquad W_{\phi}(\theta):=\operatorname{range}\left(\partial\phi(\theta)^{\top}\right). \tag*{(2)}

A C1C^{1} function hh is therefore conserved by gradient flow whenever ∇h(θ)⊥Wϕ(θ)\nabla h(\theta)\perp W_{\phi}(\theta), since h˙=−⟨∇h,∇L⟩=0\dot{h}=-\langle\nabla h,\nabla L\rangle=0. The maximal number of locally independent laws, under the assumption that the loss spans the output space and that dim⁡Lie⁡(Wϕ)\dim\operatorname{Lie}(W_{\phi}) is locally constant, is NGF(θ)=D−dim⁡Lie⁡(Wϕ)(θ)N_{\mathrm{GF}}(\theta)=D-\dim\operatorname{Lie}(W_{\phi})(\theta).

The conservation laws used below arise from continuous invariances of the loss (2). Let T:R×Ω→RDT:\mathbb{R}\times\Omega\to\mathbb{R}^{D} be a one parameter transformation on an open subset Ω\Omega of the model parameter space satisfying T(0,θ)=θT(0,\theta)=\theta, ϕ(T(ϵ,θ))=ϕ(θ)\phi(T(\epsilon,\theta))=\phi(\theta). Its infinitesimal generator Δ(θ):=ddϵT(ϵ,θ)∣ϵ=0\Delta(\theta):=\left.\frac{\mathrm{d}}{\mathrm{d}\epsilon}T(\epsilon,\theta)\right|_{\epsilon=0} satisfies ∂ϕ(θ)Δ(θ)=0\partial\phi(\theta)\Delta(\theta)=0, and hence lies in Wϕ(θ)⊥W_{\phi}(\theta)^{\perp}, leaving the loss invariant. If, in addition, Δ(θ)=∇h(θ)\Delta(\theta)=\nabla h(\theta) for some scalar potential hh, then hh is a gradient flow conservation law. Importantly, differentiating an invariance only implies that its generator lies in ker⁡∂ϕ(θ)\ker\partial\phi(\theta); it does not require every element of this kernel to arise from the particular symmetry family considered below.

Two-layer matrix factorisation. Let θ=(U,V)\theta=(U,V), U∈Rn×rU\in\mathbb{R}^{n\times r}, V∈Rm×rV\in\mathbb{R}^{m\times r}, ϕ(U,V)=UVT\phi(U,V)=UV^{\mathsf{T}}. For any A∈Rr×rA\in\mathbb{R}^{r\times r}, the transformation TA(ϵ,U,V)=(UeϵA,Ve−ϵAT)T_A(\epsilon,U,V)=(Ue^{\epsilon A},Ve^{-\epsilon A^{\mathsf{T}}}) leaves UVTUV^{\mathsf{T}} unchanged, with generator ΔA(U,V):=ddϵTA(ϵ,U,V)∣ϵ=0=(UA,−VAT)\Delta_A(U,V):=\left.\frac{\mathrm{d}}{\mathrm{d}\epsilon}T_A(\epsilon,U,V)\right|_{\epsilon=0}=(UA,-VA^{\mathsf{T}}). This vector field is a Euclidean gradient field precisely when A∈Sym⁡(r)A\in\operatorname{Sym}(r). For such AA, define Ψ:=UTU−VTV∈Sym⁡(r)\Psi:=U^{\mathsf{T}}U-V^{\mathsf{T}}V\in\operatorname{Sym}(r), ΨA:=⟨A,Ψ⟩=tr⁡(AΨ)\Psi_A:=\langle A,\Psi\rangle=\operatorname{tr}(A\Psi). Direct differentiation gives ∇ΨA=(2UA,−2VA)=2ΔA\nabla\Psi_A=(2UA,-2VA)=2\Delta_A. Thus the admissible test space for matrix factorisation gradient flow laws is AMF=Sym⁡(r)\mathcal{A}_{\mathrm{MF}}=\operatorname{Sym}(r). If (U;V)(U;V) has full rank rkrk, a complete set of locally independent laws has dimension NGF=rk(2r+1−rk)2N_{\mathrm{GF}}=\frac{rk(2r+1-rk)}{2}, Marcotte et al. (2023).

Two-layer ReLU. Consider g(θ,x)=Uσ(VTx)=∑j=1rujσ(vjTx)g(\theta,x)=U\sigma(V^{\mathsf{T}}x)=\sum_{j=1}^{r}u_j\sigma(v_j^{\mathsf{T}}x), with σ(z)=max⁡{z,0}\sigma(z)=\max\{z,0\}. Positive homogeneity gives the neuron-wise invariance (uj,vj)↦(eϵλjuj,e−ϵλjvj)(u_j,v_j)\mapsto(e^{\epsilon\lambda_j}u_j,e^{-\epsilon\lambda_j}v_j). Equivalently, within the matrix family above the admissible continuous generators are, generically, A∈Diag⁡(r)A\in\operatorname{Diag}(r). The corresponding independent conservation laws are hj=∥uj∥2−∥vj∥2h_j=\lVert u_j\rVert^2-\lVert v_j\rVert^2, j=1,…,rj=1,\ldots,r, or, for A=diag⁡(λ1,…,λr)A=\operatorname{diag}(\lambda_1,\ldots,\lambda_r), ΨA=∑j=1rλjhj\Psi_A=\sum_{j=1}^{r}\lambda_jh_j. Hence AReLU=Diag⁡(r)\mathcal{A}_{\mathrm{ReLU}}=\operatorname{Diag}(r), and NGF=rN_{\mathrm{GF}}=r. These laws form a complete set of independent gradient flow conservation laws for the two-layer ReLU architecture Marcotte et al. (2023).

The scalar model is the special case r=1r=1. Thus all three examples admit the common representation in Equation (1) but the admissible test space is architecture-dependent: A=R,Sym⁡(r),Diag⁡(r)\mathcal{A}=\mathbb{R},\operatorname{Sym}(r),\operatorname{Diag}(r), for the scalar, matrix-factorisation, and ReLU examples, respectively. § 3 shows that this distinction determines which component of the central flow perturbation can break the corresponding laws.

Symmetry breaking under the central flow

We now ask how the gradient flow conservation laws of § 2 evolve under the central flow. Cohen et al. [3] model the time-averaged trajectory of gradient descent at the edge of stability by

θ˙=−η∇L(θ)−η2∇θ⟨H(θ),Σ⟩,H(θ):=∇2L(θ).(3)\dot{\theta}=-\eta\nabla L(\theta)-\frac{\eta}{2}\nabla_{\theta}\langle H(\theta),\Sigma\rangle,\qquad H(\theta):=\nabla^2L(\theta). \tag*{(3)}

Here Σ⪰0\Sigma\succeq0 models the covariance of the fast oscillations. At the edge of stability its range is contained in the critical eigenspace

U(θ):=ker⁡(H(θ)−2ηI),k:=dim⁡U(θ),(4)\mathcal{U}(\theta):=\operatorname{ker}\left(H(\theta)-\frac{2}{\eta}I\right),\qquad k:=\dim\mathcal{U}(\theta), \tag*{(4)}

while the solution to the semidefinite complementarity problem (SDCP) selects the realised Σ(t)\Sigma(t), pinning the active eigenvalues at 2/η2/\eta. Away from the edge of stability, the selected covariance is Σ=0\Sigma=0, and (3) reduces to gradient flow. Writing an oscillatory displacement as δ\delta, with E[δ]=0\mathbb{E}[\delta]=0 and Σ=E[δδT]\Sigma=\mathbb{E}[\delta\delta^{\mathsf{T}}], the curvature contraction has the equivalent form

⟨H,Σ⟩=tr⁡(HTΣ)=E[δTHδ].(5)\langle H,\Sigma\rangle=\operatorname{tr}(H^{\mathsf{T}}\Sigma)=\mathbb{E}[\delta^{\mathsf{T}}H\delta]. \tag*{(5)}

Thus Σ\Sigma records where the finite-step dynamics oscillate, while ⟨H,Σ⟩\langle H,\Sigma\rangle measures the curvature encountered by those oscillations.

The central flow conservation law drift. Let hh be a gradient flow conservation law. Since ⟨∇h,∇L⟩=0\langle\nabla h,\nabla L\rangle=0, contracting (3) with ∇h\nabla h eliminates the gradient flow field:

h˙=−η2D∇h⟨H(θ),Σ⟩.(6)\dot{h}=-\frac{\eta}{2}D_{\nabla h}\langle H(\theta),\Sigma\rangle. \tag*{(6)}

The directional derivative in (6) is taken with Σ\Sigma held fixed. It is therefore not the total time derivative of ⟨H(θ(t)),Σ(t)⟩\langle H(\theta(t)),\Sigma(t)\rangle, which would contain an additional term involving Σ˙\dot{\Sigma}. For the laws of § 2, ∇ΨA=2ΔA\nabla\Psi_A=2\Delta_A, so

Ψ˙A=−ηDΔA⟨H(θ),Σ⟩.(7)\dot{\Psi}_A=-\eta D_{\Delta_A}\langle H(\theta),\Sigma\rangle. \tag*{(7)}

Collapse to the factor dimension. Write L=F∘ϕL=F\circ\phi. The Hessian decomposes as

H=Hp+Hc,Hp=∂ϕ⊤(∇2F)∂ϕ,Hc=∑a(∂aF)∇2ϕa.(8)H=H_{\mathrm{p}}+H_{\mathrm{c}},\qquad H_{\mathrm{p}}=\partial\phi^{\top}(\nabla^2F)\partial\phi,\qquad H_{\mathrm{c}}=\sum_a(\partial_aF)\nabla^2\phi_a. \tag*{(8)}

For the squared losses considered below, ∇2F=I\nabla^2F=I for the MF and scalar cases as well as for the ReLU case as considered in Appendix D we get

⟨Hp,Σ⟩=E[∥∂ϕ(θ)δ∥2].(9)\langle H_{\mathrm{p}},\Sigma\rangle=\mathbb{E}\left[\left\lVert\partial\phi(\theta)\delta\right\rVert^2\right]. \tag*{(9)}

In our two-layer examples, HcH_{\mathrm{c}} is invariant along the symmetry generator, so DΔA⟨Hc,Σ⟩=0D_{\Delta_A}\langle H_{\mathrm{c}},\Sigma\rangle=0. For ReLU, this statement and the explicit Hessian calculation below are local to a region in which the activation pattern is fixed. Because ∂ϕ(θ)δ\partial\phi(\theta)\delta is bilinear in the two factors, (9) reduces to r×rr\times r contractions. Writing

PU:=U⊤U,PV:=V⊤V,RU:=E[δU⊤δU],RV:=E[δV⊤δV],(10)P_U:=U^{\top}U,\qquad P_V:=V^{\top}V,\qquad R_U:=\mathbb{E}[\delta_U^{\top}\delta_U],\qquad R_V:=\mathbb{E}[\delta_V^{\top}\delta_V], \tag*{(10)}

differentiation along ΔA\Delta_A yields a linear functional of AA. For all three examples this functional is in the form of the raw symmetry-breaking matrix

B~=(PURV+RVPU)−(PVRU+RUPV)+2(N−N′),(11)\widetilde{B}=(P_UR_V+R_VP_U)-(P_VR_U+R_UP_V)+2(N-N'), \tag*{(11)}

where N:=E[(δV⊤V)(δU⊤U)]N:=\mathbb{E}[(\delta_V^{\top}V)(\delta_U^{\top}U)], N′:=E[(δU⊤U)(δV⊤V)]N':=\mathbb{E}[(\delta_U^{\top}U)(\delta_V^{\top}V)] denote the two orderings of the cross contraction. Their explicit forms, and the corresponding ReLU substitutions, are given in the examples in Apps. B–D. Importantly, every term in (11) is linear in Σ\Sigma. The raw matrix B~\widetilde{B} need not belong to the architecture’s admissible test space A\mathcal{A}. We therefore define the canonical symmetry-breaking matrix by B(θ,Σ):=PAB~(θ,Σ)∈AB(\theta,\Sigma):=P_{\mathcal{A}}\widetilde{B}(\theta,\Sigma)\in\mathcal{A}, where PAP_{\mathcal{A}} is the orthogonal projection onto A\mathcal{A}. Equation (7) then becomes

Ψ˙A=−η⟨A,B(θ,Σ)⟩,A∈A.(12)\dot{\Psi}_A=-\eta\langle A,B(\theta,\Sigma)\rangle,\qquad A\in\mathcal{A}. \tag*{(12)}

Criterion for survival of the conservation laws is given in Appendix § 5. The three examples differ in their admissible test space and in the low-dimensional contractions entering the raw breaking matrix B~\widetilde{B}. Table 1 summarises the resulting conservation laws and observable symmetry-breaking matrices; full derivations and empirical results are given in Apps. B–D.

modelA\mathcal{A}gradient flow lawsB\mathcal{B}
scalarR\mathbb{R}h=u2−v2h = u^2 - v^22(u2σvv−v2σuu)2(u^2\sigma_{vv} - v^2\sigma_{uu})
matrix factorisationSym⁡(r)\operatorname{Sym}(r)ΨA=tr⁡(AΨ)\Psi_A = \operatorname{tr}(A\Psi)sym⁡(B~)\operatorname{sym}(\widetilde{B})
two-layer ReLUDiag⁡(r)\operatorname{Diag}(r)hj=∥uj∥2−∥vj∥2h_j = \lVert u_j\rVert^2 - \lVert v_j\rVert^2PDiag⁡(r)B~ReLU⁡P_{\operatorname{Diag}(r)}\widetilde{B}_{\operatorname{ReLU}}

Table 1. The three instances of the symmetry-breaking construction. Here Ψ=UTU−VTV\Psi= U^{\mathsf{T}}U - V^{\mathsf{T}}V. The raw matrix B~\widetilde{B} is projected onto the architecture-dependent admissible test space before determining survival.

Conclusions and outlook

In this work we show how the central flow dynamics can break conservation laws that hold on gradient flows in scalar networks, matrix factorisation and two-layer ReLU networks. We identify a symmetry-breaking drift matrix, laying the ground for future analysis of more complex architectures and EOS conservation laws. The limitations of this work, that future work will address, include actual counting and completeness arguments, to identify the number of independent conservation laws that survive under central flow dynamics. Moreover, a formal distinction of uniform (for all admissible Σ(t)\Sigma(t)) and trajectory-dependent laws should be elaborated, to answer the question of whether new trajectory-dependent laws emerge at the EOS under the central flow.

Survival of gradient flow conservation laws

For the realised central flow covariance Σ(t)\Sigma(t), define

B(t):=B(θ(t),Σ(t)).(13)\mathcal{B}(t) := \mathcal{B}(\theta(t), \Sigma(t)). \tag*{(13)}

Equation (12) immediately gives

ΨA is conserved along the trajectory  ⟺  A⊥span⁡t{B(t)}.(14)\Psi_A\text{ is conserved along the trajectory} \iff A \perp\operatorname{span}_{t}\{\mathcal{B}(t)\}. \tag*{(14)}

Hence the central flow breaks conservation laws geometrically: the realised matrices B(t)\mathcal{B}(t) span the broken directions, and their orthogonal complement gives the surviving constant-AA conservation laws along this trajectory. It is also useful to characterise all breaking directions that can be generated by covariance perturbations supported on the current critical eigenspace. Since B~\widetilde{B} is linear in Σ\Sigma, we define

Mθ:Sym⁡(U(θ))⟶A,Mθ(Σ):=PAB~(θ,Σ).(15)M_\theta: \operatorname{Sym}(\mathcal{U}(\theta)) \longrightarrow\mathcal{A}, \qquad M_\theta(\Sigma) := P_{\mathcal{A}}\widetilde{B}(\theta,\Sigma). \tag*{(15)}

This is a linear extension over the span of supported covariance matrices; the actual central flow selects a ΣCF(θ)\Sigma_{\mathrm{CF}}(\theta) through its SDCP solution. Since

dim⁡Sym⁡(U(θ))=k(k+1)2,(16)\dim\operatorname{Sym}(\mathcal{U}(\theta)) = \frac{k(k+1)}{2}, \tag*{(16)}

the potential breaking space obeys

rank⁡Mθ≤min⁡{k(k+1)2,dim⁡A}.(17)\operatorname{rank} M_\theta\le\min\left\{\frac{k(k+1)}{2}, \dim\mathcal{A}\right\}. \tag*{(17)}

In particular, when k=1k = 1, all covariance perturbations supported on the critical eigenspace produce at most one breaking direction. For a single realised Σ(t)\Sigma(t), independently of kk, the instantaneous drift is always the single functional A↦⟨A,B(t)⟩A \mapsto\langle A, \mathcal{B}(t)\rangle; larger kk matters because it allows a larger set of breaking directions to be reached as the covariance changes.

Scalar factorisation

Consider ϕ(u,v)=uv\phi(u,v) = uv and L(u,v)=12(uv−y)2L(u,v) = \frac{1}{2}(uv-y)^2. Writing

ρ:=uv−y,h:=u2−v2,\rho:= uv-y, \qquad h := u^2-v^2,

we have

∇L=ρ(v,u)T,∇h=(2u,−2v)T,⟨∇h,∇L⟩=0.\nabla L = \rho(v,u)^{\mathsf{T}}, \qquad\nabla h = (2u,-2v)^{\mathsf{T}}, \qquad\langle\nabla h,\nabla L\rangle= 0.

Hence hh is, up to functions of hh, the unique gradient-flow conservation law in this r=1r = 1 case.

Central-flow drift. The Hessian and covariance are

H=(v22uv−y2uv−yu2),Σ=(σuuσuvσuvσvv).H=\begin{pmatrix} v^{2} & 2uv-y \\ 2uv-y & u^{2} \end{pmatrix},\qquad\Sigma=\begin{pmatrix} \sigma_{uu} & \sigma_{uv} \\ \sigma_{uv} & \sigma_{vv} \end{pmatrix}.

Holding Σ\Sigma fixed,

⟨H,Σ⟩=v2σuu+u2σvv+2(2uv−y)σuv,\langle H,\Sigma\rangle=v^{2}\sigma_{uu}+u^{2}\sigma_{vv}+2(2uv-y)\sigma_{uv},
∇θ⟨H,Σ⟩=(2uσvv+4vσuv, 2vσuu+4uσuv)T.\nabla_{\theta}\langle H,\Sigma\rangle=\left(2u\sigma_{vv}+4v\sigma_{uv},\ 2v\sigma_{uu}+4u\sigma_{uv}\right)^{\mathsf{T}}.

Contracting with ∇h\nabla h, the cross-covariance cancels:

D∇h⟨H,Σ⟩=4(u2σvv−v2σuu).D_{\nabla h}\langle H,\Sigma\rangle=4(u^{2}\sigma_{vv}-v^{2}\sigma_{uu}).

Therefore

h˙=−2η(u2σvv−v2σuu)(18)\dot{h}=-2\eta\left(u^{2}\sigma_{vv}-v^{2}\sigma_{uu}\right) \tag*{(18)}

Thus the scalar symmetry-breaking quantity is

B=2(u2σvv−v2σuu).\mathcal{B}=2(u^{2}\sigma_{vv}-v^{2}\sigma_{uu}).

One critical direction. Suppose the top Hessian eigenvalue is simple and the active critical eigenspace is one-dimensional. Then Σ=σ2qqT\Sigma=\sigma^{2}qq^{\mathsf{T}}, where qq is the unit top eigenvector, σ2\sigma^{2} is the variance of the oscillations along qq, and

⟨H,Σ⟩=σ2λmax⁡(H)\langle H,\Sigma\rangle=\sigma^{2}\lambda_{\max}(H)

and

D∇h⟨H,Σ⟩=σ2⟨∇h,∇λmax⁡⟩.D_{\nabla h}\langle H,\Sigma\rangle=\sigma^{2}\langle\nabla h,\nabla\lambda_{\max}\rangle.

Let

b:=2uv−y,R:=h24+b2.b:=2uv-y,\qquad R:=\sqrt{\frac{h^{2}}{4}+b^{2}}.

The largest eigenvalue is

λmax⁡=u2+v22+R.\lambda_{\max}=\frac{u^{2}+v^{2}}{2}+R.

Direct differentiation gives

∂uλmax⁡=u+12uh+2bvR,\partial_{u}\lambda_{\max}=u+\frac{\frac{1}{2}uh+2bv}{R},
∂vλmax⁡=v+−12vh+2buR,\partial_{v}\lambda_{\max}=v+\frac{-\frac{1}{2}vh+2bu}{R},

and therefore

⟨∇h,∇λmax⁡⟩=h(2+u2+v2R).\langle\nabla h,\nabla\lambda_{\max}\rangle=h\left(2+\frac{u^{2}+v^{2}}{R}\right).

Consequently,

h˙=−ησ22h(2+u2+v2R)(19)\dot{h}=-\frac{\eta\sigma^{2}}{2}h\left(2+\frac{u^{2}+v^{2}}{R}\right) \tag*{(19)}

For R>0R>0 and σ2>0\sigma^{2}>0,

hh˙≤0,h\dot{h}\leq0,

with equality if and only if h=0h=0. This is an instantaneous statement during the active oscillatory phase; it does not imply that h(t)h(t) reaches zero before Σ\Sigma reaches zero.

Stable terminal values. On the solution manifold uv=yuv = y,

λmax⁡=u2+v2=h2+4y2.\lambda_{\max} = u^2 + v^2 = \sqrt{h^2 + 4y^2}.

Any stable solution-manifold endpoint satisfies ηλmax⁡≤2\eta\lambda_{\max} \le2, and hence

∣h∞∣≤21η2−y2,1η2≥y2.(20)\lvert h_\infty\rvert\le2\sqrt{\frac{1}{\eta^2} - y^2}, \qquad\frac{1}{\eta^2} \ge y^2. \tag*{(20)}

Thus the EOS dynamics can reduce the balancedness gap and then freeze at a generically nonzero value once the oscillatory phase terminates.

Empirical analysis. Figure 1 isolates the symmetry-breaking mechanism. Gradient flow remains on the level set h=h0h = h_0, while both gradient descent (GD) and the central flow (CF) leave that level set once EOS effects become active. When reaching the EOS dynamics, the independently measured h˙\dot{h} agrees with the analytic prediction. Since h>0h > 0 in these runs and h˙<0\dot{h} < 0, the curvature correction moves the two factors toward balance, consistent with the sign relation hh˙≤0h\dot{h} \le0.

Four-panel analysis of gradient-flow breaking, central-flow drift, EOS pinning, and terminal bound

Figure 1. Scalar factorisation: central-flow breaking of the gradient-flow balancedness law. Results are shown over three random initialisations. Left: relative change of the gradient-flow invariant h=u2−v2h = u^2 - v^2. Gradient flow preserves hh up to numerical integration error, whereas finite-step GD and the hybrid central flow produce an O(1)O(1) change. The GD parameter midpoint, (θk+θk+1)/2(\theta_k + \theta_{k+1})/2, removes the dominant period-two oscillation before evaluating hh. Centre-left: pointwise test of the central-flow drift at the first EOS contact. For each seed, the directly measured h˙=⟨∇h,θ˙CF⟩\dot{h} = \langle\nabla h, \dot{\theta}_{\mathrm{CF}}\rangle agrees with the closed-form prediction of Eq. (18), with the dashed line denoting equality. Centre-right: normalised sharpness along hybrid central flow. Each trajectory follows ordinary gradient flow before EOS and, once the constraint becomes active, remains pinned at λmax⁡(H)=2/η\lambda_{\max}(H) = 2/\eta. Right: long-time value of ∣h∣\lvert h\rvert at T=200T = 200 against the stability bound 2η−2−y22\sqrt{\eta^{-2} - y^2}. The three runs lie on the identity line, showing empirically that the bound is saturated in these instances.

Two-layer matrix factorisation

Let

U∈Rn×r,V∈Rm×r,ϕ(U,V)=UVT,U \in\mathbb{R}^{n \times r}, \qquad V \in\mathbb{R}^{m \times r}, \qquad\phi(U,V) = UV^{\mathsf{T}},

with squared loss

L(U,V)=12∥UVT−Y∥2,G:=UVT−YL(U,V) = \frac{1}{2}\lVert UV^{\mathsf{T}} - Y\rVert^2,\qquad G := UV^{\mathsf{T}} - Y

where ∥⋅∥\lVert\cdot\rVert denotes the Frobenius norm. The gradient-flow invariant is

Ψ=UTU−VTV,ΨA=tr⁡(AΨ),A∈Sym⁡(r),\Psi= U^{\mathsf{T}}U - V^{\mathsf{T}}V,\qquad\Psi_A = \operatorname{tr}(A\Psi),\qquad A \in\operatorname{Sym}(r),

with ∇ΨA=2ΔA\nabla\Psi_A = 2\Delta_A and ΔA=(UA,−VA)\Delta_A = (UA,-VA).

Hessian contraction. For an oscillatory perturbation δ=(δU,δV)\delta= (\delta_U,\delta_V),

∂ϕ(θ)δ=δUVT+UδVT.\partial_\phi(\theta)\delta= \delta_U V^{\mathsf{T}} + U\delta_V^{\mathsf{T}}.

Since ϕ\phi is bilinear, the Hessian quadratic form splits as

δTHδ=∥δUVT+UδVT∥2⏟Hp+2⟨G,δUδVT⟩⏟Hc.\delta^{\mathsf{T}}H\delta= \underbrace{\lVert\delta_U V^{\mathsf{T}} + U\delta_V^{\mathsf{T}}\rVert^2}_{H_p} + \underbrace{2\langle G,\delta_U\delta_V^{\mathsf{T}}\rangle}_{H_c}.

Along the symmetry generator,

DΔA(UVT)=UAVT−UAVT=0,D_{\Delta_A}(UV^{\mathsf{T}}) = UAV^{\mathsf{T}} - UAV^{\mathsf{T}} = 0,

so GG is unchanged and therefore

DΔA⟨Hc,Σ⟩=0.D_{\Delta_A}\langle H_c,\Sigma\rangle= 0.

Only the pullback term contributes to symmetry breaking. Define

PU:=UTU,PV:=VTV,RU:=E[δUTδU],RV:=E[δVTδV].P_U := U^{\mathsf{T}}U,\qquad P_V := V^{\mathsf{T}}V,\qquad R_U := \mathbb{E}[\delta_U^{\mathsf{T}}\delta_U],\qquad R_V := \mathbb{E}[\delta_V^{\mathsf{T}}\delta_V].

Expanding the pullback quadratic form,

∥δUVT+UδVT∥2=tr⁡(PVδUTδU)+tr⁡(PUδVTδV)+2tr⁡(VδUTUδVT).\lVert\delta_U V^{\mathsf{T}} + U\delta_V^{\mathsf{T}}\rVert^2 = \operatorname{tr}(P_V\delta_U^{\mathsf{T}}\delta_U) + \operatorname{tr}(P_U\delta_V^{\mathsf{T}}\delta_V) + 2\operatorname{tr}(V\delta_U^{\mathsf{T}}U\delta_V^{\mathsf{T}}).

Hence

⟨Hp,Σ⟩=tr⁡(PVRU)+tr⁡(PURV)+2tr⁡(VM),M:=E[δUTUδVT].\langle H_p,\Sigma\rangle= \operatorname{tr}(P_VR_U) + \operatorname{tr}(P_UR_V) + 2\operatorname{tr}(VM),\qquad M := \mathbb{E}[\delta_U^{\mathsf{T}}U\delta_V^{\mathsf{T}}].

Differentiation along the symmetry. Holding Σ\Sigma fixed gives

P˙U=APU+PUA,P˙V=−(APV+PVA).\dot{P}_U = AP_U + P_UA,\qquad\dot{P}_V = -(AP_V + P_VA).

Therefore

ddt[tr⁡(PURV)+tr⁡(PVRU)]=tr⁡(A[PURV+RVPU−PVRU−RUPV]).\frac{\mathrm{d}}{\mathrm{d}t}\left[\operatorname{tr}(P_UR_V) + \operatorname{tr}(P_VR_U)\right] = \operatorname{tr}\left(A[P_UR_V + R_VP_U - P_VR_U - R_UP_V]\right).

For the cross term,

M˙=E[δUTUAδVT],\dot{M} = \mathbb{E}[\delta_U^{\mathsf{T}}UA\delta_V^{\mathsf{T}}],

so

ddt2tr⁡(VM)=2tr⁡(VM˙)+2tr⁡(V˙M).\frac{\mathrm{d}}{\mathrm{d}t} 2\operatorname{tr}(VM)=2\operatorname{tr}(V\dot{M})+2\operatorname{tr}(\dot{V}M).

Define

N:=E[(δV⊤V)(δU⊤U)],N′:=E[(δU⊤U)(δV⊤V)].N:=\mathbb{E}\left[(\delta^{\top}_{V}V)(\delta^{\top}_{U}U)\right],\qquad N^{\prime}:=\mathbb{E}\left[(\delta^{\top}_{U}U)(\delta^{\top}_{V}V)\right].

Then

tr⁡(V˙M)=−tr⁡(AN′),tr⁡(VM˙)=tr⁡(AN),\operatorname{tr}(\dot{V}M)=-\operatorname{tr}(AN^{\prime}),\qquad\operatorname{tr}(V\dot{M})=\operatorname{tr}(AN),

and consequently

ddt2tr⁡(VM)=2tr⁡(A(N−N′)).\frac{\mathrm{d}}{\mathrm{d}t}2\operatorname{tr}(VM)=2\operatorname{tr}\left(A(N-N^{\prime})\right).

Combining the layer and cross terms gives

B~=PURV+RVPU−PVRU−RUPV+2(N−N′).(21)\widetilde{\mathcal{B}}=P_{U}R_{V}+R_{V}P_{U}-P_{V}R_{U}-R_{U}P_{V}+2(N-N^{\prime}). \tag*{(21)}

Thus

Ψ˙A=−ηtr⁡(AB~).\dot{\Psi}_{A}=-\eta\operatorname{tr}(A\widetilde{\mathcal{B}}).

Projection onto the conservation-law space. Because A∈Sym⁡(r)A\in\operatorname{Sym}(r),

tr⁡(AB~)=⟨A,sym⁡(B~)⟩.\operatorname{tr}(A\widetilde{\mathcal{B}})=\langle A,\operatorname{sym}(\widetilde{\mathcal{B}})\rangle.

Hence the canonical symmetry-breaking matrix is

B=sym⁡(B~),\mathcal{B}=\operatorname{sym}(\widetilde{\mathcal{B}}),

and the matrix-valued invariant obeys

Ψ˙=−ηB=−ηsym⁡(B~).(22)\dot{\Psi}=-\eta\mathcal{B}=-\eta\operatorname{sym}(\widetilde{\mathcal{B}}). \tag*{(22)}

The cross contribution is the expected commutator

N−N′=E[δV⊤V,δU⊤U].N-N^{\prime}=\mathbb{E}\left[\delta^{\top}_{V}V,\delta^{\top}_{U}U\right].

It vanishes automatically at r=1r=1, but is generally nonzero for r>1r>1. Since a commutator need be neither symmetric nor skew-symmetric, B~\widetilde{\mathcal{B}} itself need not lie in Sym⁡(r)\operatorname{Sym}(r); only sym⁡(B~)\operatorname{sym}(\widetilde{\mathcal{B}}) is observable by the conservation laws.

At a realised time tt,

Ψ˙A(t)=0⟺A⊥B(t),\dot{\Psi}_{A}(t)=0\quad\Longleftrightarrow\quad A\perp\mathcal{B}(t),

while a constant-AA law survives along the whole trajectory precisely when

A⊥span⁡t{B(t)}.A\perp\operatorname{span}_{t}\{\mathcal{B}(t)\}.

Finally, the sufficient conditions

PU=PV,RU=RV,N=N′P_{U}=P_{V},\qquad R_{U}=R_{V},\qquad N=N^{\prime}

give B~=0\widetilde{\mathcal{B}}=0. The ordinary factor-balance condition U⊤U=V⊤VU^{\top}U=V^{\top}V alone does not imply the corresponding covariance balance or vanishing of the cross term.

Empirical analysis. Figure 2 shows that no nonzero constant test matrix A∈Sym⁡(2)A \in\operatorname{Sym}(2) remains orthogonal to the breaking direction for the entire trajectory although no additional critical Hessian directions appear (the second eigenvalue stays uniformly below the EOS threshold).

Matrix factorisation diagnostics

Figure 2. Matrix factorisation: the broken conservation-law direction over the EOS trajectory. We consider rank-r=2r=2 matrix factorisation, for which the gradient-flow invariant Ψ=U⊤U−V⊤V∈Sym⁡(2)\Psi=U^\top U-V^\top V\in\operatorname{Sym}(2) contains NGF=3N_{\mathrm{GF}}=3 independent scalar conservation laws. Results are shown over three random seeds. Top left: relative Frobenius displacement ∥Ψ(t)−Ψ(0)∥F/∥Ψ(0)∥F\lVert\Psi(t)-\Psi(0)\rVert_F/\lVert\Psi(0)\rVert_F. Gradient flow preserves Ψ\Psi up to numerical error, whereas GD and hybrid central flow produce an O(1)O(1) change; the adjacent-iterate GD parameter midpoint removes the dominant period-two oscillation before evaluating Ψ\Psi. Top right: pointwise validation of Ψ˙=−ηB\dot{\Psi}=-\eta B, with B=sym⁡(B~)B=\operatorname{sym}(\widetilde{B}). The relative discrepancy between the directly differentiated matrix invariant and the symmetry-breaking prediction is approximately 10−1010^{-10}–10−910^{-9} for all seeds. Bottom left: normalised singular values of the matrices B(t)B(t), represented in Frobenius-orthonormal coordinates of Sym⁡(2)\operatorname{Sym}(2) and stacked over the active EOS trajectory. All three singular values are nonzero well above the numerical error floor, showing that the realised breaking direction spans all of Sym⁡(2)\operatorname{Sym}(2) over the observed trajectory. Bottom right: rank-one EOS diagnostic. The second Hessian eigenvalue remains below both 2/η2/\eta and the displayed 0.95(2/η)0.95(2/\eta) safety margin throughout the active trajectory, so the rank-one central-flow approximation remains well separated from the onset of a second critical direction.

Two-layer ReLU

Consider

g(θ,x)=Uσ(VTx)=∑j=1rujσ(vjTx),g(\theta, x) = U\sigma(V^{\mathsf{T}}x) = \sum_{j=1}^{r}u_j\sigma(v_j^{\mathsf{T}}x),

with

L=12∑i∥Uσ(VTxi)−yi∥2.L = \frac{1}{2}\sum_i \left\|U\sigma(V^{\mathsf{T}}x_i)-y_i\right\|^2.

The gradient-flow conservation laws are

hj=∥uj∥2−∥vj∥2,j=1,…,r,h_j = \left\|u_j\right\|^2-\left\|v_j\right\|^2,\qquad j=1,\ldots,r,

or equivalently

ΨA=∑jλjhj,A=diag⁡(λ1,…,λr)∈Diag⁡(r).\Psi_A = \sum_j \lambda_j h_j,\qquad A=\operatorname{diag}(\lambda_1,\ldots,\lambda_r)\in\operatorname{Diag}(r).

Fixed activation pattern. For the explicit Hessian calculation, assume that the activation pattern is constant on the support of the oscillatory perturbations. For each example define

zi:=VTxi,si:=1[zi>0],Di:=diag⁡(si),z_i := V^{\mathsf{T}}x_i,\qquad s_i := \mathbf{1}[z_i>0],\qquad D_i := \operatorname{diag}(s_i),

and

πi:=σ(zi)=Dizi,ρi:=Uπi−yi.\pi_i := \sigma(z_i)=D_i z_i,\qquad\rho_i := U\pi_i-y_i.

For δ=(δU,δV)\delta=(\delta_U,\delta_V), let

αi:=DiδVTxi.\alpha_i := D_i\delta_V^{\mathsf{T}}x_i.

Within this fixed-gating region,

∂fi(θ)δ=δUπi+Uαi,\partial f_i(\theta)\delta=\delta_U\pi_i+U\alpha_i,

and the Hessian quadratic form is

δTHδ=∑i∥δUπi+Uαi∥2+2∑i⟨ρi,δUαi⟩.\delta^{\mathsf{T}}H\delta=\sum_i\left\|\delta_U\pi_i+U\alpha_i\right\|^2+2\sum_i\langle\rho_i,\delta_U\alpha_i\rangle.

For A∈Diag⁡(r)A\in\operatorname{Diag}(r), the symmetry generator satisfies

U˙=UA,V˙=−VA.\dot{U}=UA,\qquad\dot{V}=-VA.

Since AA commutes with each gate matrix DiD_i,

π˙i=−Aπi,ddt(Uπi)=UAπi−UAπi=0.\dot{\pi}_i=-A\pi_i,\qquad\frac{\mathrm{d}}{\mathrm{d}t}(U\pi_i)=UA\pi_i-UA\pi_i=0.

Thus the residual term is invariant and only the pullback term contributes. Define

PU:=U⊤U,RU:=E[δU⊤δU],P_U := U^\top U,\qquad R_U := \mathbb{E}[\delta_U^\top\delta_U],

together with

Π:=∑iπiπi⊤,RVgate:=∑iE[αiαi⊤].\Pi:= \sum_i \pi_i\pi_i^\top,\qquad R_V^{\mathrm{gate}} := \sum_i \mathbb{E}[\alpha_i\alpha_i^\top].

Expanding the pullback term gives

∑iE[∥δUπi+Uαi∥2]=tr⁡(ΠRU)+tr⁡(PURVgate)+C,\sum_i \mathbb{E}[\|\delta_U\pi_i+U\alpha_i\|^2] = \operatorname{tr}(\Pi R_U)+\operatorname{tr}(P_U R_V^{\mathrm{gate}})+C,

where

C:=2∑iE[πi⊤δU⊤Uαi].C := 2\sum_i \mathbb{E}[\pi_i^\top\delta_U^\top U\alpha_i].

The Gram terms satisfy

P˙U=APU+PUA,Π˙=−(AΠ+ΠA),\dot{P}_U = AP_U+P_UA,\qquad\dot{\Pi}=-(A\Pi+\Pi A),

so

ddt[tr⁡(PURVgate)+tr⁡(ΠRU)]=tr⁡(A[PURVgate+RVgatePU−ΠRU−RUΠ]).\frac{\mathrm{d}}{\mathrm{d}t}\left[\operatorname{tr}(P_U R_V^{\mathrm{gate}})+\operatorname{tr}(\Pi R_U)\right] = \operatorname{tr}\left(A[P_U R_V^{\mathrm{gate}}+R_V^{\mathrm{gate}}P_U-\Pi R_U-R_U\Pi]\right).

For the cross term, define

Ne:=∑iE[αiπi⊤δU⊤U],Ne′:=∑iE[δUUαiπi⊤].N_{\mathrm{e}} := \sum_i \mathbb{E}[\alpha_i\pi_i^\top\delta_U^\top U],\qquad N_{\mathrm{e}}' := \sum_i \mathbb{E}[\delta_U U\alpha_i\pi_i^\top].

Differentiating CC and using cyclicity gives

C˙=2tr⁡(A(Ne−Ne′)).\dot{C}=2\operatorname{tr}(A(N_{\mathrm{e}}-N_{\mathrm{e}}')).

Hence

B~ReLU=PURVgate+RVgatePU−ΠRU−RUΠ+2(Ne−Ne′).(23)\widetilde{\mathcal{B}}_{\mathrm{ReLU}} = P_U R_V^{\mathrm{gate}}+R_V^{\mathrm{gate}}P_U-\Pi R_U-R_U\Pi+2(N_{\mathrm{e}}-N_{\mathrm{e}}'). \tag*{(23)}

This is exactly the matrix-factorisation form under

PV↦Π,RV↦RVgate,N↦Ne,N′↦Ne′.P_V\mapsto\Pi,\qquad R_V\mapsto R_V^{\mathrm{gate}},\qquad N\mapsto N_{\mathrm{e}},\qquad N'\mapsto N_{\mathrm{e}}'.

Projection and survival. For ReLU the admissible test space is Diag⁡(r)\operatorname{Diag}(r), so

BReLU=PDiag⁡(r)B~ReLU.\mathcal{B}_{\mathrm{ReLU}}=P_{\operatorname{Diag}(r)}\widetilde{\mathcal{B}}_{\mathrm{ReLU}}.

In particular,

h˙j=−η(B~ReLU)jj.(24)\dot{h}_j=-\eta(\widetilde{\mathcal{B}}_{\mathrm{ReLU}})_{jj}. \tag*{(24)}

An individual conservation law hjh_j therefore has zero instantaneous drift exactly when

(B~ReLU)jj=0.(\widetilde{\mathcal{B}}_{\mathrm{ReLU}})_{jj}=0.

For a general diagonal test matrix A=diag⁡(λ1,…,λr)A=\operatorname{diag}(\lambda_1,\ldots,\lambda_r),

Ψ˙A=−η⟨A,BReLU⟩.\dot{\Psi}_A=-\eta\langle A,\mathcal{B}_{\mathrm{ReLU}}\rangle.

Thus individual hjh_j’s may all drift while a nontrivial linear combination ∑jλjhj\sum_j\lambda_jh_j has zero instantaneous drift.

Across activation boundaries. The matrix formula above assumes fixed gates. An exact identity for the averaged dynamics does not require this assumption. Suppose

w˙=−ηE[∇L(w+δ)],E[δ]=0,\dot{w}=-\eta\mathbb{E}[\nabla L(w+\delta)],\qquad\mathbb{E}[\delta]=0,

and let the linear generator ΔA\Delta_A satisfy the symmetry identity

⟨∇L(w),ΔA(w)⟩=0\langle\nabla L(w),\Delta_A(w)\rangle=0

wherever LL is differentiable. Since ∇ΨA=2ΔA\nabla\Psi_A=2\Delta_A,

Ψ˙A=−2ηE[⟨ΔA(w),∇L(w+δ)⟩].\dot{\Psi}_A=-2\eta\mathbb{E}[\langle\Delta_A(w),\nabla L(w+\delta)\rangle].

Using the invariance identity at w+δw+\delta together with ΔA(w+δ)=ΔA(w)+ΔA(δ)\Delta_A(w+\delta)=\Delta_A(w)+\Delta_A(\delta) gives

Ψ˙A=2ηE[⟨ΔA(δ),∇L(w+δ)⟩].(25)\dot{\Psi}_A=2\eta\mathbb{E}[\langle\Delta_A(\delta),\nabla L(w+\delta)\rangle]. \tag*{(25)}

Identity (25) is not specific to ReLU: it uses only that ΔA\Delta_A is linear, that ∇ΨA=2ΔA\nabla\Psi_A=2\Delta_A, and that LL is invariant along ΔA\Delta_A at every point w+δw+\delta where it is differentiable, so it holds equally for the scalar and matrix factorisation examples. We state it here because, for ReLU, the loss is not smooth: when LL is smooth near ww, the central flow (3) is the second-order truncation of these averaged dynamics and (23) is the corresponding explicit drift, which requires the oscillation to stay within one activation region. Identity (25) allows the oscillation to cross activation boundaries between w−δw-\delta and w+δw+\delta, only limiting the sampled points to not lie exactly on a boundary. Nevertheless, there is a dependence on the full distribution of δ\delta, whereas the central flow only looks at Σ=E[δδT]\Sigma=\mathbb{E}[\delta\delta^{\mathsf{T}}].

Empirical analysis. Figure 3 verifies the architecture-dependent projection underlying the ReLU result. The ReLU conservation laws only probe the diagonal component of the raw symmetry-breaking matrix. The directly measured h˙j\dot{h}_j agrees with −η(B~ReLU)jj-\eta(\widetilde{B}_{\mathrm{ReLU}})_{jj}. The agreement includes both positive and negative drifts, showing that the calculation predicts not only whether a law is broken but also its signed instantaneous motion. The explicit ReLU Hessian calculation is only valid while the oscillation remains inside a single activation region.

Three-panel plot showing local symmetry breaking, fixed-gate verification, and surviving combinations

Figure 3. Two-layer ReLU: local symmetry breaking inside a fixed activation region. We consider a two-layer ReLU network with r=3r = 3 hidden units and evaluate the central-flow prediction at an active rank-one EOS point for three independent seeds. The admissible test space is A=Diag⁡(r)\mathcal{A} = \operatorname{Diag}(r), so only the diagonal projection of the raw breaking matrix is observable. Left: pointwise validation of the per-neuron drift h˙j=−η(B~ReLU)jj\dot{h}_j = -\eta(\widetilde{\mathcal{B}}_{\mathrm{ReLU}})_{jj}, hj=∥uj∥2−∥vj∥2h_j = \lVert u_j\rVert^2 - \lVert v_j\rVert^2. Each marker corresponds to one neuron from one seed; the measured drift ⟨∇hj,θ˙CF⟩\langle\nabla h_j,\dot{\theta}_{\mathrm{CF}}\rangle lies on the identity line against the prediction. Centre: verification of the fixed-gating assumption. For every seed we report min⁡i,j∣zij∣∣δzij∣\min_{i,j} \frac{\lvert z_{ij}\rvert}{\lvert\delta z_{ij}\rvert}, δ=±σ2 q\delta= \pm\sqrt{\sigma^2}\,q, where zij=vjTxiz_{ij} = v_j^{\mathsf{T}}x_i. All ratios exceed one, so neither side of the rank-one oscillation crosses a ReLU activation boundary. Right: instantaneous surviving combinations. Writing b=diag⁡(B~ReLU)∈Rrb = \operatorname{diag}(\widetilde{\mathcal{B}}_{\mathrm{ReLU}}) \in\mathbb{R}^r, we construct an orthonormal basis of b⊥b^\perp, which has dimension r−1=2r - 1 = 2 whenever b≠0b \ne0. The maximum measured absolute drift among these two combinations is O(10−11)O(10^{-11}) for every seed.

Figure 4 shows what happens beyond the fixed-gate regime. When a pre-activation crosses zero, the Hessian changes discontinuously, and along the hybrid central-flow trajectories the rank-one oscillation violates the fixed-gate condition after fewer than 10 optimisation steps. Beyond that point (23) is no longer certified: the orange curves are obtained by recomputing the local Hessian in each new activation region. Gradient flow and centred GD are shown for reference. Characterising the central flow conservation laws across activation boundaries is left as a follow-up to this work.

Nine-panel plot comparing ReLU piecewise central flow with period-2-centered gradient descent across gate changes

Figure 4. Two-layer ReLU beyond fixed gates: piecewise continuation of the central flow. For each of three random seeds (rows), we track the three per-neuron gradient-flow laws hj=∥uj∥2−∥vj∥2h_j = \lVert u_j\rVert^2 - \lVert v_j\rVert^2 (columns). Gradient flow (solid blue) preserves every hjh_j, while finite-step GD breaks them. To remove the dominant period-two oscillation, the dashed green curve evaluates hjh_j at the adjacent-iterate parameter midpoint θˉk+1/2=(θkGD+θk+1GD)/2\bar{\theta}_{k+1/2} = (\theta_k^{\mathrm{GD}} + \theta_{k+1}^{\mathrm{GD}})/2. The orange curve is the hybrid central flow, continued piecewise by recomputing the local ReLU Hessian after activation-pattern changes. The vertical dotted line marks the first sampled point at which the rank-one oscillation fails the fixed-gate condition ∣zij∣>σˉ2∣qvTxi∣\lvert z_{ij}\rvert> \sqrt{\bar{\sigma}^2}\lvert q_v^{\mathsf{T}}x_i\rvert for some sample-neuron pair. Consequently, the explicit fixed-gate symmetry-breaking formula (23) is certified only to the left of this line; to the right, the orange curve is a heuristic piecewise continuation shown for illustration, with gradient flow and centred GD for reference.

Experimental settings

The git repository can be found at: https://github.com/naimaeb/CF_conserved_quantities.git and the experimental settings are given by Table 2.

scalarmatrix fact.ReLU (local)ReLU (trajectory)
ϕ\phiuvuvUVTUV^{\mathsf{T}}Uσ(VTx)U\sigma(V^{\mathsf{T}}x)Uσ(VTx)U\sigma(V^{\mathsf{T}}x)
dimensionsn=m=r=1n=m=r=1n=m=3n=m=3, r=2r=2n=1n=1, m=4m=4, r=3r=3n=1n=1, m=4m=4, r=3r=3
datay=1y=1rank-2 target YYN=24N=24, xi∼N(0,I)x_i\sim\mathcal{N}(0,I)as local
initialisationu0∈[1,1.45]u_0\in[1,1.45], v0∈[0.05,0.3]v_0\in[0.05,0.3]0.8N0.8\mathcal{N}, 0.3N0.3\mathcal{N}0.6N0.6\mathcal{N}, 0.25N0.25\mathcal{N}as local
n seeds3333
η\eta0.90, 0.95, 0.960.87, 0.12, 0.192.22, 0.69, 2.990.56, 1.62, 1.19
horizon (steps)20050—80
CF step0.020.1—0.1
NGFN_{\mathrm{GF}}1333

Table 2. Configuration of the four experiments. ReLU targets are yi=xiTw⋆y_i=x_i^{\mathsf{T}}w_\star with w⋆∼N(0,I)w_\star\sim\mathcal{N}(0,I), and the ReLU loss is a dataset mean. Note that when considering the dataset mean in the ReLU case, we need to scale the loss, the Hessian quadratic form, Π\Pi, RVgateR_V^{\mathrm{gate}}, CC, NeN_e and Ne′N'_e by 1N\frac{1}{N}.

References

  1. [1]Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. In International Conference on Learning Representations, 2019. URL https://mlanthology.org/iclr/2019/arora2019iclr-convergence/.arxiv.org/abs/1810.02281
  2. [2]Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. In International Conference on Learning Representations, 2021. URL https://mlanthology.org/iclr/2021/cohen2021iclr-gradient/.arxiv.org/abs/2103.00065
  3. [3]Jeremy Cohen, Alex Damian, Ameet Talwalkar, J Zico Kolter, and Jason D. Lee. Understanding optimization in deep learning with central flows. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=SI2rI3ZPs.arxiv.org/abs/2410.24206
  4. [4]Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. In International Conference on Learning Representations, 2023. URL https://mlanthology.org/iclr/2023/damian2023iclr-selfstabilization/.arxiv.org/abs/2209.15594
  5. [5]Simon S Du, Wei Hu, and Jason Lee. Algorithmic Regularization in Learning Deep Homogeneous Models: Layers Are Automatically Balanced. In Neural Information Processing Systems, pages 384–395, 2018. URL https://mlanthology.org/neurips/2018/du2018neurips-algorithmic/.arxiv.org/abs/1806.00900
  6. [6]Avrajit Ghosh, Soo Min Kwon, Rongrong Wang, Saiprasad Ravishankar, and Qing Qu. Learning Dynamics of Deep Matrix Factorization Beyond the Edge of Stability. In International Conference on Learning Representations, 2025. URL https://mlanthology.org/iclr/2025/ghosh2025iclr-learning/.arxiv.org/abs/2502.20531
  7. [7]Thomas Hofmann. The map behind the flow: Finite-step gradient descent as a dynamical system, 2026. URL https://arxiv.org/abs/2607.04993.

Paper details

vlnc:W_ruf4pbrj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents