betaFeedback

WorksComputer Science

Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize

Abstract

Deep neural networks demonstrate remarkable generalization despite being highly over-parameterized, challenging traditional statistical learning theories. While several theoretical paradigms such as margin-based bounds, PAC-Bayesian analyses, and neural tangent kernels provide global explanations, they often lack mechanistic insight into how generalization emerges from training dynamics. We introduce Neural Expansion, a unified geometric framework that characterizes generalization through the expansion of directionally-selective ray coverage in the input space. Using Generalization Intervals (GIs), we quantify direction-specific regions around training samples where the model's predictions remain stable. Our extensive empirical studies across diverse architectures and datasets reveal that training leads to progressive expansion of ray coverage, particularly along directions with low Jacobian singular values. This mechanism helps explain a large fraction of correctly classified test inputs, adversarial examples, and even mislabeled or out-of-distribution samples through the geometry of ray coverage rather than instance-level memorization. By connecting input-space curvature to predictive behavior, our framework provides a mechanistic foundation that complements existing global theories of deep learning.

Introduction

The remarkable generalization capability of deep neural networks presents a fundamental theoretical challenge. Despite having millions or billions of parameters, far exceeding their training dataset size, these models generalize well to unseen examples, contradicting classical learning theory predictions [36]. This generalization persists even without explicit regularization, suggesting intrinsic mechanisms beyond conventional capacity control [27, 5].

Traditional frameworks for understanding generalization, such as VC-dimension theory and uniform convergence bounds, provide vacuous guarantees for modern neural networks. While recent approaches, including spectral norm-based margins [5], PAC-Bayesian bounds [27], compression-based complexity [4], neural tangent kernels [17], simplicity bias [32], sharpness-aware training [3, 2], and input-space geometric invariance [25], have advanced our understanding, a critical question remains unaddressed: why does a specific test input receive a specific classification?

Preprint. Accepted at NeurIPS 2026.

We propose Neural Expansion, a novel geometric framework explaining how networks develop directionally-selective ray coverage around training samples during learning. Our approach is based on Generalization Intervals (GIs), which quantify directionally-sensitive regions around training points within which a model’s classification remains invariant [30]. Rather than defining complex decision boundaries in high-dimensional spaces, we analyze how networks progressively expand stable classification zones, ray coverage, in specific directions.

Neural Expansion refers to the progressive growth of this ray coverage during training, which becomes more pronounced in directions with low Jacobian singular values. Unlike margin-based approaches that focus on decision boundary distances, Neural Expansion captures direction-specific invariance properties that explain why certain test points get assigned to particular classes. Furthermore, while kernel theories (NTK) assume a fixed feature space and neural collapse focuses on terminal phases, our framework explains the dynamic formation of ray coverage throughout the entire training process.

Our key contributions include:

  • A theoretical framework that explains sample-level generalization through directional stability in input space, connecting Jacobian singular values to classification behavior

  • Empirical demonstration that neural networks form wide generalization intervals in directions with low curvature, creating expansive, overlapping ray coverage around training samples

  • Evidence that test-time classifications occur because test samples fall within the ray coverage of training samples, not because of instance-level memorization

  • A geometric perspective showing how standard generalization, adversarial vulnerability, long-tailed misclassification, and random-label learning can be interpreted through the directional structure of generalization intervals

In this work, we establish that the local, sample-level geometry of neural network behavior is not merely an empirical curiosity but a theoretically grounded consequence of gradient-descent training. This perspective is complementary to existing theories of generalization, which provide valuable insights into optimization dynamics and global complexity measures. By formally analyzing how Jacobian structure creates and expands directional corridors of stability in input space, we provide a missing link between representation geometry and classification decisions, one that applies across diverse architectures and dataset distributions.

Background and Related Work

Understanding why overparameterized neural networks generalize well remains one of the central open questions in deep learning theory [36]. Bengio et al. [8] hypothesized that effective generalization emerges when learned representations disentangle underlying factors of variation. The success of deep networks has since spurred diverse theoretical perspectives. Our work introduces a local, mechanistic framework based on Generalization Intervals (GIs) [30], focusing on individual sample-level classification behavior through Jacobian-derived directional invariance.

Classical approaches rely on complexity-based measures such as VC dimension, Rademacher complexity, and margins. Spectral norm-based bounds [5], PAC-Bayesian frameworks [27], and compression-based approaches [4] offer compelling global insights. Our GI framework provides a local geometric lens, explaining generalization through stable directional regions around training samples where predictions remain invariant.

The Neural Tangent Kernel (NTK) approximates wide networks as kernel regression around initialization [17], formalized and contrasted with richer feature-learning regimes [10, 34] and extended to explain triple descent [1]. These approaches focus on global dynamics near initialization; our work instead quantifies post-training directional stability via the Jacobian, capturing piecewise-linear generalization regions in realistic regimes.

Neural Collapse describes how class means, features, and classifier weights align near convergence [29, 15] (Wu and Papyan 2024), with recent work linking this to the average gradient outer product [6]. Our GI framework shows that classification decisions arise from stable, low-curvature regions that generalize across architectures and tasks.

Generalization has also been attributed to a simplicity bias [32], supported by double and deep double descent [7, 36, 26]. Recent work shows that seeking flat loss regions improves generalization [13], SGD implicitly minimizes complexity-promoting measures [33], and networks can benignly overfit noisy data under favorable signal-to-noise conditions [14], phenomena our framework explains geometrically via Jacobian curvature.

A Theoretical Framework for Neural Network Generalization

In this section, we develop our theoretical framework that captures the directional properties of generalization in deep neural networks. We introduce the generalization interval (GI), which captures how far we can go in a given direction without changing the class, and connect it to Jacobian singular value analysis.

Definition 1. Let f:Rd→Rcf : \mathbb{R}^{d} \to\mathbb{R}^{c} be a cc-class classifier outputting the logits for each class, x0∈Rdx_0 \in\mathbb{R}^{d} be a given input data, y∈{1,⋯ ,c}y \in\{1,\cdots,c\} be its class and d∈Rdd \in\mathbb{R}^{d} be the direction to traverse. The generalization interval (GI) of ff from the input x0x_0 in the direction of dd is defined by

GI(f,x0,d)=sup⁡{a>0∣fy(x0+td)>max⁡i≠yfi(x0+td) for all t∈[0,a]}(1)GI(f,x_0,d) = \sup\{a > 0 \mid f^{y}(x_0 + td) > \max_{i \ne y} f^{i}(x_0 + td) \text{ for all } t \in[0,a]\} \tag*{(1)}

As discussed earlier, this quantity measures how far we can go in the given direction before we reach the decision boundary. The idea is similar to the classical concept of margins, but a GI is direction-dependent: a model might exhibit large GIs in some directions and small GIs in others around the same point. Nevertheless, there is an important relationship between the margins and the GIs, which we will discuss later. The act of finding a GI can be described as shooting a ray from the given input until we reach the decision boundary. As such, it is natural to define the following:

Definition 2. The ray coverage of x0x_0 is the set of all points that lie within their respective GI bounds:

RC(x0)={x:GI(f,x0,x−x0∥x−x0∥)≥∥x−x0∥}RC(x_0) = \left\{x : GI\left(f,x_0,\frac{x-x_0}{\lVert x-x_0\rVert}\right) \ge\lVert x-x_0\rVert\right\}

Jacobian Analysis and GI Formation

To understand why neural networks exhibit directionality in ray coverages through sensitivity, we analyze the Jacobian of the network’s representation function. Let ϕ:Rd→Rm\phi: \mathbb{R}^{d} \to\mathbb{R}^{m} denote the mapping from input to the penultimate layer (i.e., the representation), and W∈Rc×mW \in\mathbb{R}^{c \times m} be the final linear classifier, such that f(x)=Wϕ(x)f(x) = W\phi(x).

The Jacobian Jϕ(x)∈Rm×dJ_{\phi}(x) \in\mathbb{R}^{m \times d} captures how the representation changes with respect to small perturbations in the input. We perform singular value decomposition:

Jϕ(x)=UΣVT=∑i=1min⁡(m,d)σiuiviT(2)J_{\phi}(x) = U\Sigma V^{T} = \sum_{i=1}^{\min(m,d)} \sigma_i u_i v_i^{T} \tag*{(2)}

where σ1≥σ2≥⋯≥σmin⁡(m,d)≥0\sigma_1 \ge\sigma_2 \ge\cdots\ge\sigma_{\min(m,d)} \ge0 are the singular values, vi∈Rdv_i \in\mathbb{R}^{d} are the right singular vectors (directions in input space), and ui∈Rmu_i \in\mathbb{R}^{m} are the left singular vectors (directions in representation space).

Our empirical analysis reveals a crucial property: we observe that, in trained neural networks, the vast majority of singular values are close to zero (See Figure 2b). For instance, in a network trained on CIFAR-10 (3,072-dimensional input), typically only 10-15 singular values are significantly non-zero, while over 99.5% of the singular value spectrum is effectively zero (Appendix § 10). This spectral structure has profound implications for generalization due to Theorem 1. For a right singular vector direction vv of JϕJ_{\phi}, ∥Jϕ(x0)v∥\lVert J_{\phi}(x_0)v\rVert is proportional to the corresponding singular value. Therefore, the theorem predicts that the GI is large for all but a few limited directions due to the directional sensitivity of the model, which is consistent with the empirical observation we will present later in Figure 2a.

Figure 2. (a) GI magnitudes across RSV directions in early- vs. late-stage models (epoch 1 vs. epoch 50). Training increases GIs selectively along high-index (low-curvature) directions. (b) Comparison of linear approximation quality along RSV directions before and after training.

Linear Approximation of the GI and the Relation to the Margin

As the GI can be described as the margins of samples with directionality, they naturally have a close relationship. To describe it, we first define the margin:

Definition 3. Let f:Rd→Rcf : \mathbb{R}^{d} \to\mathbb{R}^{c} be a cc-class classifier, x0∈Rdx_{0} \in\mathbb{R}^{d} be a given input data, and y∈{1,⋯ ,c}y \in\{1,\cdots,c\} be its class. The margin of the sample (x0,y)(x_{0}, y) is fy(x0)−max⁡i≠yfi(x0)f^{y}(x_{0}) - \max_{i \ne y} f^{i}(x_{0}) and we denote it by q0q_{0}. Here, fi(x0)∈Rf^{i}(x_{0}) \in\mathbb{R} means the iith component of f(x0)f(x_{0}).

We now have the following relation:

Theorem 1. Let x0,d∈Rdx_{0}, d \in\mathbb{R}^{d} be a given input and a given direction. Suppose y∈{1,⋯c}y \in\{1,\cdots c\} is the class of x0x_{0}. Assume that the classifier f:Rd→Rcf : \mathbb{R}^{d} \to\mathbb{R}^{c} is differentiable with an inhomogeneous directional derivative: the yyth component of Jf(x0)dJ_{f}(x_{0})d is not equal to any other component. Consider the linear approximation f(x0+ϵd)=f(x0)+ϵJf(x0)d+R(x0,ϵ,d)f(x_{0}+\epsilon d) = f(x_{0})+\epsilon J_{f}(x_{0})d+R(x_{0},\epsilon,d) where RR is the error term satisfying ∥R(x0,ϵ,d)∥1<δ\lVert R(x_{0},\epsilon,d)\rVert_{1}<\delta for some δ>0\delta>0, Jf(x0)J_{f}(x_{0}) denotes the Jacobian matrix for ff at x0x_{0} and ∥⋅∥1\lVert\cdot\rVert_{1} is the ℓ1\ell^{1}-norm. Then

q0−δ∥Jf(x0)d∥1≤GI⁡(f,x0,d)(3)\frac{q_{0}-\delta}{\lVert J_{f}(x_{0})d\rVert_{1}} \leq\operatorname{GI}(f,x_{0},d) \tag*{(3)}

The proof can be found in Appendix § 17. This result tells us that, under linear approximation, a larger margin would indicate a larger generalization interval. This partially explains why the GIs should increase under gradient descent training, as previous literature such as Lyu and Li [22] shows that a smooth approximation of the margin for a homogeneous network would monotonically increase under gradient descent. We note that, as we show later, the linear approximation holds surprisingly well (see § 5.1.3). Nonlinear effects are accounted for through the assumed residual bound δ\delta (Appendix § 11.§ 1).

However, this is not the end of the story. The lower bound has the norm of the directional derivative ∥Jf(x0)d∥\lVert J_{f}(x_{0})d\rVert, and thus, sensitivity in each direction would contribute to the size of the GIs.

GI Strictly Refines Spectral Margin Along Singular Directions

Theorem 1 establishes that the GI lower bound grows with margin and shrinks with directional sensitivity. This raises a natural question: does GI offer strictly more information than classical spectral margin theory [5], or does it merely restate the same global bound in geometric language? We show that GI strictly refines spectral margin by decomposing input space direction-by-direction in a way that spectral margin cannot.

Theorem 2 (GI–Spectral Margin Decomposition). Under the same setting as Theorem 1, let Jϕ(x0)=UΣV⊤J_{\phi}(x_{0}) = U\Sigma V^{\top} be the SVD of the penultimate-layer Jacobian with singular values σ1≥⋯≥σmin⁡(m,d)≥0\sigma_{1} \geq\cdots\geq\sigma_{\min(m,d)} \geq0 and right singular vectors vi∈Rdv_{i} \in\mathbb{R}^{d}. Then for each viv_{i}:

GI⁡(f,x0,vi)≥q0−δc∥W∥∞σi(4)\operatorname{GI}(f,x_{0},v_{i}) \geq\frac{q_{0}-\delta}{\sqrt{c}\lVert W\rVert_{\infty}\sigma_{i}} \tag*{(4)}

As σi→0\sigma_{i} \to0 with q0>δq_{0}>\delta, this bound grows without bound, while the spectral margin bound remains fixed and direction-agnostic.

The bound decomposes input space into sensitive directions (large σi\sigma_{i}) and invariant directions (small σi\sigma_{i}). Spectral margin theory captures only the former; GI captures both. Since a significant number of singular directions in trained networks have σi≈0\sigma_{i}\approx0 (§ 3.1), GI provides nontrivial directional information in precisely the regime where spectral margin is silent. The proof is in Appendix § 18.

Jacobian control of the input-loss gradient

Theorem 2 reveals that directions with large singular values yield a small lower bound on GIs. A deeper question follows: do these high-sensitivity, low-GI directions correspond to the directions where the loss landscape is sharpest? To partially answer this question, we bound the input-loss gradient in terms of the largest Jacobian singular value, connecting the loss landscape and the most sensitive direction.

Theorem 3 (GI and Loss Landscape Sharpness). Under the same setting as Theorem 1, let L(x)=−log⁡σy(f(x))\mathcal{L}(x)=-\log\sigma_y(f(x)) be the cross-entropy loss at x0x_0 with true class yy, and let ∇xL(x0)\nabla_x\mathcal{L}(x_0) be its gradient with respect to the input. Then:

∥∇xL(x0)∥2≤2∥W∥op⋅σ1(5)\left\|\nabla_x\mathcal{L}(x_0)\right\|_2 \le\sqrt{2}\left\|W\right\|_{\mathrm{op}}\cdot\sigma_1 \tag*{(5)}

where σ1\sigma_1 is the largest singular value of Jϕ(x0)J_\phi(x_0).

Equation (5) shows that input-space sharpness is controlled by the top Jacobian singular value σ1\sigma_1, the same quantity that governs GI via Theorem 2. Empirically, the largest right singular vector v1v_1 also closely approximates the unconstrained steepest ascent direction, as shown in Figure 10. This connects Neural Expansion directly to the flat minima literature [18]: low-curvature directions (large index ii, small σi\sigma_i) correspond to flat directions in the loss landscape and large GIs, while high-curvature directions (small index ii, large σi\sigma_i) correspond to sharp directions and small GIs. The proof is in Appendix § 19.

Gradient and image difference projections on top-k right singular vectors

Figure 10. Gradient & Image Δ\Delta Projections on Top-k Right Singular Vectors. Blue lines show the cumulative percentage of the loss gradient projected onto the top-k RSVs. Orange lines show the projection of the image difference vector between target and source images.

GI Quantifies the Adversarial Robustness Radius

The GI provides a direct lower bound on the size of adversarial perturbations. We now show that every successful class-changing perturbation has norm at least the minimum GI over unit directions, and derive a further lower bound in terms of the margin and Jacobian norm.

Theorem 4 (Adversarial Perturbation Radius via GI). Under the same setting as Theorem 1, let Δ∈Rd\Delta\in\mathbb{R}^d be any perturbation that changes the classification at x0x_0, i.e., arg⁡max⁡ifi(x0+Δ)≠y\arg\max_i f^i(x_0+\Delta)\ne y. Then:

∥Δ∥2≥min⁡∥d∥=1GI⁡(f,x0,d)(6)\|\Delta\|_2 \ge\min_{\|d\|=1}\operatorname{GI}(f,x_0,d) \tag*{(6)}

Furthermore, combining with Theorem 2:

∥Δ∥2≥q0−δc∥W∥opσ1(7)\|\Delta\|_2 \ge\frac{q_0-\delta}{\sqrt{c}\left\|W\right\|_{\mathrm{op}}\sigma_1} \tag*{(7)}

Equation (6) establishes a bound on the adversarial robustness radius: large GIs in every direction guarantee robustness. Equation (7) makes this quantitative in terms of observable quantities: margin q0q_0, classifier norm ∥W∥op\|W\|_{\mathrm{op}}, and the largest Jacobian singular value σ1\sigma_1. The proof is in Appendix § 20.

Experimental Setup

Full implementation details are provided in Appendix § 8.

Datasets: We evaluate on three benchmark datasets: MNIST [21], CIFAR-10 [19], and ImageNette [11]. To assess generalization under more challenging regimes, we include random label variants and long-tailed variants (following [36] and Ma et al. (2022)). Full details on these datasets and their variants are provided in Appendix § 8.§ 1.

Model Architectures: To demonstrate the universality of Neural Expansion, we evaluate three architecture families: Multi-Layer Perceptrons, Convolutional Networks (e.g., ResNet-20/50, VGG-16), and Vision Transformers (including ViT-B/16 and DeiT-S). All models are trained with SGD using dataset-appropriate hyperparameters until convergence. Full architecture and training details are provided in Appendix § 8.§ 2.

Generalization Interval Analysis: Our primary analytical tool is the Generalization Interval (GI), which quantifies directional prediction stability in input space. GIs are computed using a step-wise search algorithm (Appendix § 8.§ 4) along RSVs of the penultimate-layer Jacobian (§ 3.1), combined with ray coverage measurement toward training samples and linear approximation evaluation (§ 5.1.3). Together, these methods allow us to analyze the formation of ray coverage during training, its alignment with Jacobian structure, and its implications across standard, long-tailed, random-labeled, and adversarial test settings.

Results

GI Behavior Across Model Types and Training Scenarios

We conduct a systematic analysis of generalization interval (GI) behavior across model architectures, training scenarios, and datasets. GIs are computed along the right singular vectors (RSVs) of the input-to-penultimate-layer Jacobian. To ensure consistency, each RSV direction is scaled by a factor of 56.0, the maximum distance in the normalized 784-dimensional MNIST input space. Unless otherwise noted, all experiments use an MLP architecture (784–1024–512–10). Our results reveal consistent GI patterns across settings, pointing to a universal geometric structure underlying neural network generalization.

Temporal Evolution of Generalization Intervals During Training

We observe how GI-based ray coverage progressively develops during the training process in Figure 1. Both the mean GI and the proportion exceeding 1.0 increase steadily, strongly correlating with improvements in test accuracy (Pearson r=0.99r = 0.99 across epochs, range 0.994–0.998 over three runs). While both RSV and random directions (isotropic Gaussians, unit-normalized) exhibit GI expansion, RSV directions consistently show larger GIs and faster growth rates, particularly in later epochs. This demonstrates the network’s preferential expansion of ray coverage along low-curvature directions identified by the Jacobian SVD (Appendix § 9).

Evolution of ray coverage during training of an MNIST MLP over 10 epochs

Figure 1. Evolution of ray coverage during training of an MNIST MLP over 10 epochs. Left: Mean positive GI growth between same-class training samples, with percentage of GIs exceeding 1.0. Middle: Average GI along RSV directions. Right: Average GI along random directions.

GI Expansion in Early vs. Later Training

Figure 2a compares GI magnitudes across RSV directions between an early-stage model (epoch 1) and a fully trained model (epoch 50). In the early phase, GI values are relatively uniform, averaging around 7 across RSVs. After training, the distribution becomes skewed, GIs increase substantially along high-index (low-curvature) RSV directions and decrease along low-index (high-curvature) directions, with the average rising to 11.5. This indicates that training selectively expands ray coverage along directions associated with smaller singular values. We observe 512 non-zero singular values, corresponding to the representation layer dimension, while the remaining RSV directions are null due to zero singular values.

Linear Approximation Along Directions with Different Singular Values

Figure 2b quantifies the local linearity of the model along RSV directions by measuring the maximum distance α\alpha for which a linear approximation of the model output maintains a cosine similarity of at least 0.95 with the true output (evaluated in the penultimate-layer representation space).

In the untrained model (random initialization), linear approximation quality is uniformly low: α\alpha ranges from ~0.2 in high-curvature directions (low RSV indices) to ~0.4 in flatter directions, indicating limited local linearity across all directions. After training (49 epochs), the model exhibits a clear monotonic trend where linearity improves substantially with RSV index, starting from α≈0.4\alpha\approx0.4 and reaching α≈1.2\alpha\approx1.2 in low-curvature directions. This coincides with a slower decay in the Jacobian’s singular values, which extend further into high-index RSVs compared to the untrained case.

The sharp transition around indices 300–400 (Appendix § 11), where the singular value spectrum flattens, marks the boundary where the trained model’s local linearity begins to dominate. These results reinforce our hypothesis: the directions supporting the longest GIs also align with those where the model behaves most linearly. Training amplifies this structure, enabling extended ray-based coverage without changes in prediction.

Effect of Ray Coverage Interventions on Generalization

In this section, we show that intervening on ray coverage changes the model’s generalization ability, highlighting their close relationship beyond the post-hoc analysis we have so far. In Appendix § 15.§ 1, we show theoretically that, under what we call the no-re-entry-assumption, i.e., the assumption that the model does not predict the original class after leaving GI, an increase in GI implies a decrease in the prediction error. This empirical experiment expands on the theoretical result.

The effect of modifying ray coverage is evaluated using an auxiliary loss applied to perturbed training inputs. In the expand condition, correct classification of the perturbed input is encouraged. In the restrict condition, confident misclassification of the perturbed input is encouraged, while cross-entropy is retained at the original training input. The control condition uses cross-entropy alone. An MNIST-MLP and a Fashion-MNIST-CNN are trained under each condition using three random seeds. Full loss definitions and hyperparameters are provided in Appendix § 15.

As shown in Table 1, coverage restriction reduces test accuracy relative to the control by 12.2 percentage points on MNIST-MLP (p=0.003p = 0.003) and 7.4 percentage points on Fashion-MNIST-CNN (p≈0.008p \approx0.008). Coverage expansion increases accuracy by 1.7 and 1.1 percentage points, respectively (both p<0.008p < 0.008). Corresponding changes in GI and ray coverage are observed from the first epoch across all seeds. These intervention results are consistent with the theoretical relationship between GI expansion and population error and support a causal contribution of ray coverage to generalization.

ModelConditionTest Acc. (%)Mean GIRay Coverage (%)
MNIST MLPControl87.38 ± 0.081.872 ± 0.04578.33 ± 1.04
MNIST MLPExpand89.04 ± 0.081.943 ± 0.05481.17 ± 1.26
MNIST MLPRestrict75.18 ± 1.181.498 ± 0.02560.67 ± 0.76
Fashion-MNIST CNNControl85.27 ± 0.182.005 ± 0.01773.83 ± 0.29
Fashion-MNIST CNNExpand86.37 ± 0.292.037 ± 0.01174.33 ± 0.58
Fashion-MNIST CNNRestrict77.86 ± 1.291.492 ± 0.09254.33 ± 3.21

Table 1. Test accuracy, mean GI, and ray coverage under control, expansion, and restriction conditions. Values are reported as mean ± standard deviation over three random seeds at the final training epoch.

Ray Coverage as a Mechanism for Test-Time Classification

Building on the directional invariance captured by GIs, we show that ray coverage provides a mechanistic explanation for test-time classification decisions. This framework unifies standard generalization, random label learning, long-tailed distribution learning, and adversarial vulnerability under a single geometric perspective.

Test Sample Classification via Ray Coverage

We analyze the relationship between test sample classification and the ray coverage formed by training samples across MNIST, CIFAR-10, and ImageNette datasets using multiple architectures. For each test sample, we compute GIs along the direction to all training samples of the predicted class and report the percentage of GIs exceeding 1.0, representing how often the test sample falls within the ray coverage region of the predicted class.

We observe consistent patterns across datasets and models; Figure 3 reveals consistently high coverage across datasets and architectures for both correctly classified and misclassified samples.

Box plots showing the percentage of GIs exceeding 1.0 between test and training samples of the corresponding class across MNIST, CIFAR10, and Imagenette

Figure 3. Test Sample Classification via Ray Coverage. Box plots show the percentage of GIs exceeding 1.0 between test and training samples of the corresponding class. Both correctly classified and misclassified samples exhibit substantial ray coverage across datasets and architectures. The gray dashed line indicates the threshold for directional coverage, where a test sample is considered covered if at least 1% of training samples from the same class exhibit a GI greater than 1.0. Misclassified bars are absent for the ImageNette models because no misclassified samples appeared in the evaluated subset (k=20) of test samples per class, likely due to the limited sample size.

These findings suggest that test-time predictions arise from the geometry of class-level ray coverage rather than instance-level memorization. Even misclassified inputs often fall within the invariant region of the predicted class, supporting Neural Expansion as a general mechanism underlying classification across varying accuracy regimes. We include more details about the experiment in Appendix E. Class trajectory visualizations along train-to-test paths are shown in Appendix G. These measurements exhibit seed stability and class-discriminability (Appendices F.1, F.2).

Neural Expansion as a Unified Framework for Challenging Cases

We examine whether ray coverage remains the operative mechanism in challenging settings of traditional learning: random label learning, long-tailed distributions, and adversarial perturbations.

Random Label Learning: Even when label semantics are randomized (RANDOM-10, RANDOM-1001), we find that test samples consistently fall within ray coverage zones around training samples with the same assigned label. The plot in Figure 4a reveals nontrivial GI overlap between test and training samples of the same (randomly assigned) label, indicating that the model still forms geometric zones for each label, albeit smaller and noisier. This finding challenges the common assumption that networks simply “memorize” random labels without forming structure. (See Appendix § 16.§ 1 for dataset-specific coverage values.)

Random Label Learning, LT2 Misclassification Distribution, and LT2 GI Coverage

Figure 4. Left: Random Label Learning with GI ray coverage for RANDOM-10 and RANDOM-100 models. The gray dashed line (as defined in Figure 3) indicates the 1% directional coverage threshold. Middle: LT2 Misclassification Distribution showing percentage distribution from true class frequency to predicted class frequency. Right: LT2 GI Coverage showing average percentage of GIs exceeding 1.0 across different frequency categories.

Long-Tailed Distributions: In highly imbalanced datasets (e.g., MNIST-LT2), test samples from underrepresented classes are disproportionately misclassified into medium- or high-frequency classes (Figure 4b). The ray coverage plot in Figure 4c shows that this is not due to failure in learning low-frequency classes, but rather to the expansive coverage of frequent classes dominating the input space. Misclassifications align with the regions where dominant classes extend their influence. (Detailed statistics in Appendix § 16.§ 2.)

Adversarial Examples: We analyze how adversarial examples interact with ray coverage using three attack methods: Carlini-Wagner (CW) (Carlini and Wagner, [9]), Iterative Fast Gradient Sign Method (IFGSM) (Kurakin et al., [20]), and Projected Gradient Descent (PGD) (Madry et al., [24]). Figure 5 reveals the consistent presence of ray coverage regardless of whether adversarial perturbations successfully change the classification. In many cases, the attack appears to shift the test input from one class’s coverage zone to another’s, rather than entirely escaping the structure. This trend persists across MNIST and CIFAR-10.

Figure 5. Adversarial Examples Analysis (misclassified samples indicate successful attacks; correct classifications indicate failures). The heatmap shows the percentage of adversarial examples with GI > 1 to training samples of the predicted class across different dataset-model combinations and attack methods (CW, IFGSM, PGD).

For ImageNette-ViTB, correctly classified adversarial examples show low ray coverage. Notably, we could not compute coverage values for misclassified ImageNette examples, as they were often pushed outside our 10-class subset into the broader 1000-class ImageNet space. This illustrates how high-dimensional natural images provide more pathways for adversarial examples to escape class boundaries than constrained datasets like MNIST. Appendix § 16.§ 3 provides more details regarding the geometry of adversarial perturbations. To further examine model behavior under distribution shift, we also compute GI coverage for out-of-distribution samples (see Appendix § 16.§ 4). The anisotropic growth of GIs during training, driven by margin increase and directional restructuring of Jacobian singular values, is analyzed in Appendix § 21.

Discussion

Our results reveal a striking finding that cuts across all experimental conditions: ray coverage operates as the classification mechanism even when it should not. Networks trained on random labels still form geometric coverage zones, and misclassified samples still fall within ray coverage, just of the wrong class. This suggests that gradient descent does not merely fit labels; it consistently organizes input space into directional corridors, regardless of whether those corridors reflect semantic structure, though these corridors become significantly sparser and more fragmented without semantic supervision (17%–24% coverage under random labels versus >80% on standard datasets, Table 7). This clarifies the boundary between memorization and generalization: both operate through ray coverage, but generalizing networks build wider, overlapping corridors aligned with low-curvature directions that capture class-relevant structure.

SettingCorrectly ClassifiedMisclassified
RANDOM-10 (MNIST-MLP)17%24%
RANDOM-100 (MNIST-MLP)18%22%

Table 7. Ray coverage percentages for correctly and incorrectly classified test samples under random label supervision.

This mechanistic picture complements existing theories in ways that are non-redundant. PAC-Bayesian (Neyshabur et al., [27]) and margin-based bounds (Bartlett et al., [5]) certify that a model generalizes globally, but cannot say why a specific test point is classified correctly. NTK theory (Jacot et al., [17]) and Neural Collapse (Papyan et al., [29]) describe either the initialization regime or the terminal phase; our framework tracks the geometry continuously throughout training, filling a sample-level explanatory gap that global theories leave open.

Limitations and Future Work. GI computation remains expensive for high-dimensional inputs, and while our sampling-based approach scales to architectures such as ViTs, full coverage requires further approximation strategies. The framework is currently most natural for classification; extending it to regression, generation, or structured prediction requires reformulating GIs for settings without discrete decision boundaries. A promising future direction is to integrate these insights directly into training objectives, encouraging networks to explicitly maximize ray coverage in low-curvature directions to enhance generalization and robustness. Finally, the consistent emergence of ray coverage across model classes suggests that gradient-based learning may implicitly align with the intrinsic geometry of natural data distributions, a connection that could yield new principles for representation learning and regularization. A rigorous proof that individual Jacobian singular values restructure anisotropically during gradient descent training remains an open theoretical question; our current account in Section N relies on empirical observations.

Conclusion

We present Neural Expansion, a geometric framework that reinterprets generalization in deep neural networks through the lens of directionally-sensitive invariance pathways. Through extensive empirical analysis spanning diverse architectures, datasets, and training regimes, we uncover a coherent geometric signature: a compact set of high-curvature directions sculpts class boundaries, while an abundance of low-curvature directions carve out invariance corridors that link samples belonging to the same class. This dual structure provides a unifying account of seemingly disparate phenomena, namely standard generalization, memorization of random labels, misclassification of tail-class samples, and susceptibility to adversarial perturbations, all through a common geometric lens. The resulting picture reframes our understanding of what neural networks learn: rather than constructing intricate decision surfaces, they organically develop stable classification pathways whose collective geometry mirrors the latent structure of the data.

Detailed Experimental Setup

Datasets and Variants

We use three benchmark datasets: MNIST [21] (28 ×\times 28 grayscale, 10 classes), CIFAR-10 [19] (32 ×\times 32 RGB, 10 classes), and ImageNette [11], a 10-class subset of ImageNet with 224 ×\times 224 RGB images.

To test generalization under different distributions, we created variants including:

  • Random Label Variants: RANDOM-10 (random labels across 10 classes) and RANDOM-100 (random labels across 100 classes), following Zhang et al. [36].

  • Long-tailed Variants: MNIST-LT1 (imbalance ratio 6.5:1) and MNIST-LT2 (78.6:1), following Ma et al. (2022).

Images are normalized to [−1,1][-1, 1] using dataset-specific preprocessing. For ImageNette, we apply standard ImageNet normalization, random cropping, horizontal flipping, and color jitter.

Their respective open licenses: MIT (CIFAR-10), CC BY-SA 3.0 (MNIST), and Apache 2.0 (Imagenette, ViT).

Model Architectures

We evaluate three architecture families:

  • Multi-Layer Perceptrons (MLPs): For MNIST, we use a 784–1024–512–10 MLP with ReLU and batch normalization. For CIFAR-10, a 3072–2048–1024–10 MLP is used.

  • Convolutional Neural Networks (CNNs): We use custom CNNs for MNIST and CIFAR-10, along with pre-trained ResNet-20, ResNet-50, and VGG-16 (He et al. (2016); Simonyan and Zisserman [31]).

  • Vision Transformers (ViTs): Lightweight ViTs for MNIST (patch size 7 ×\times 7, embedding dim 64) and CIFAR-10 (patch size 8 ×\times 8, embedding dim 192). For ImageNette, we use ViT-B/16 and DeiT-S (Dosovitskiy et al. [12]).

Training Protocols

All custom models are trained using SGD until convergence with dataset-specific hyperparameters:

  • MNIST: 30 epochs, learning rate 10−3/10−210^{-3}/10^{-2}, no momentum or weight decay.

  • CIFAR-10: 50 epochs, learning rate 10−3/10−210^{-3}/10^{-2}, momentum 0.9, weight decay 10−410^{-4}.

  • ImageNette: 30 epochs, learning rate 10−410^{-4}, momentum 0.9, weight decay 10−410^{-4}.

We include accuracy for all the custom-trained models in Figure 6.

Epoch-wise accuracy curves across models and datasets

Figure 6. Epoch-wise accuracy curves across models and datasets for vanilla as well as LT1, LT2, Random-10 and Random-100 variations.

Analysis Methods

Generalization Interval Calculation: We implemented a step-wise search algorithm to compute GIs between sample pairs. For direction vector v=x2−x1\mathbf{v} = \mathbf{x}_2 - \mathbf{x}_1 (the unnormalized difference vector), we incrementally moved along both positive and negative directions (x1±ϵv\mathbf{x}_1 \pm\epsilon\mathbf{v}) using step size δ=0.001\delta= 0.001–0.010.01 until either a decision boundary was crossed or a maximum ϵ=3.0\epsilon= 3.0 was reached. Under this convention, ϵ=1.0\epsilon= 1.0 exactly reaches x2\mathbf{x}_2, and the threshold GI⁡(f,xi,xtest−xi)≥1\operatorname{GI}(f, x_i, x_{\mathrm{test}} - x_i) \ge1 in Equation (8) is dimensionally consistent with the unnormalized direction. Our vectorized implementation enabled batch processing (MNIST: 6000 pairs, CIFAR-10: 2000 pairs, ImageNette: 100 pairs). Table 2 contains the compute resources and runtime details of the GI coverage experiments.

DatasetArchitectureGPU(s)Runtime (hh:mm)Memory Used (GB)
MNIST (6000 pairs)MLPA500000:303.0
MNIST (6000 pairs)CNNA500000:555.0
MNIST (6000 pairs)ViTA500001:256.5
CIFAR-10 (2000 pairs)ResNet20A500000:404.2
CIFAR-10 (2000 pairs)VGG16-BNA500000:555.1
ImageNette (100 pairs)VGG16A500001:406.8
ImageNette (100 pairs)ResNet50A500001:306.5
ImageNette (100 pairs)ViT-B/16A500001:557.8
ImageNette (100 pairs)DeiT-SmallA500001:357.0

Table 2. Compute resources and runtime details for test-to-train GI coverage experiments using a step-wise directional search.

Jacobian Analysis: For each sample, we computed the Jacobian matrix J=∂f(x)∂xJ = \frac{\partial f(x)}{\partial x} of the penultimate layer features with respect to the input and performed SVD: J=UΣVTJ = U\Sigma V^T. We evaluated linear approximation quality by measuring cosine similarity between f(x0+αvi)f(x_0 + \alpha v_i) and f(x0)+αWσiuif(x_0) + \alpha W \sigma_i u_i for different α\alpha values, where both terms are cc-dimensional logit vectors.

Ray Coverage Analysis: For each test sample with predicted class cc, we calculated GIs to training samples of class cc and computed the percentage exceeding 1.0. This quantified test samples’ membership in training samples’ ray coverage zones, enabling comparisons between correct and incorrect classifications across datasets, architectures, and distribution variants.

GI Evolution Experimental Details

To analyze the temporal evolution of ray coverage during training, we tracked GI development in an MNIST MLP (784-1024-512-10) across 10 epochs. Figure 1 captures three key aspects of this evolution.

For train-to-train GI analysis (left panel), we calculated GIs between 100 randomly selected same-class training sample pairs for each digit class at checkpoints from epoch 0 to epoch 10. We measured both mean GI values and the percentage exceeding 1.0, which grew from 0.76 to 1.24 and 39% to 82% respectively, exhibiting strong correlation with test accuracy improvement (86.2% to 91.7%).

For directional analysis (middle and right panels), we computed GIs along RSVs of the Jacobian matrix and along random directions, using the same scaling factor of 56.0 for both to ensure fair comparison. This factor was derived from the maximum possible distance in the 784-dimensional normalized input space of MNIST. Along RSV directions, mean GI increased from 0.53 to 1.18, with the percentage exceeding 1.0 growing from 22% to 78%. In contrast, random directions showed more modest growth from 0.40 to 0.82, with the percentage exceeding 1.0 reaching only 34% by epoch 10.

Figure 7 provides further insight into the differential growth rates of GIs along RSV versus random directions. When normalized to show relative growth from the initial checkpoint, both direction types exhibit similar expansion through epoch 3 (approximately 500% growth). However, from epoch 4 onward, a clear divergence emerges, with RSV directions showing accelerated expansion. By epoch 10, RSV directions achieve 1220% growth relative to initialization, while random directions reach only 940%. This growing disparity indicates the network progressively develops anisotropic ray coverage that preferentially expands along directions aligned with the underlying data manifold.

Figure 7. Relative GI growth along RSV vs. random directions during training. Both direction types show similar growth patterns through epoch 3, after which RSV directions exhibit substantially accelerated expansion, reaching 1220% growth by epoch 10 compared to 940% for random directions.

To complement these directional and train-to-train analyses, we further examined how ray coverage evolves within the training set. At each checkpoint, we computed GIs between randomly selected same-class training sample pairs. Table 3 summarizes GI statistics separately for training pairs where the prediction remains correct versus incorrect.

EpochBatchCorrectIncorrect
CountMean GI% GI > 1Avg GIMedian GICountMean GI% GI > 1Avg GIMedian GI
00501.179848.211.16380.41501.192248.491.18160.49
0200501.449679.601.47081.38501.539784.001.56351.48
0399501.374880.061.38501.21501.337582.871.35531.28
1399501.435874.891.45231.23501.292281.241.30651.30
3200501.606679.801.62021.36501.257485.171.26561.20
4200501.674682.201.68631.49501.221385.071.22861.16
5399501.698981.771.71751.54501.151582.791.16151.13
6399501.717782.361.73521.56501.115081.841.12501.11
8200501.758483.941.77371.61501.116982.161.12601.14
9200501.788184.511.80391.65501.117282.541.12451.15
10399501.831585.431.84741.72501.116783.951.12191.14

Table 3. Summary of GI statistics across training checkpoints for correctly and incorrectly classified training sample pairs. Mean GI is the arithmetic mean of all GI values including those at the boundary (GI = 0); Avg GI is the mean computed only over sample pairs with GI > 0 (i.e., where a nonzero interval exists).

Directional Generalization Intervals and Jacobian SVD Analysis

Our experiment examines how Generalization Intervals (GIs) along Jacobian RSV directions evolve during training. The Singular Value Decomposition (SVD) decomposes the Jacobian matrix into orthogonal components, where singular values represent the network’s sensitivity along corresponding input directions. For each model (early-trained after 1 epoch and later-trained after 50 epochs), we computed Jacobians at 50 correctly classified MNIST samples, performed SVD, and measured GIs along each RSV direction with maximum perturbation of 20.0 and step size of 0.01.

The results reveal a significant transformation in directional sensitivity. Early training (epoch 1) shows a relatively uniform GI profile (∼7\sim7) across all RSV directions, whereas after extended training (epoch 50), GIs decrease along directions with large singular values (low-indexed RSVs) while dramatically increasing along directions with small singular values (high-indexed RSVs), reaching an average of ∼11.5\sim11.5. This pattern supports the theory that neural networks develop a natural split between an “information space” (directions with large singular values) where the model remains sensitive, and a “nuisance space” (directions with small singular values) where the model becomes increasingly invariant.

Linear Approximation Analysis Along Singular Vector Directions

We analyze the local linear behavior of neural networks by measuring how far a linear approximation remains valid along different input directions. For each RSV direction from the Jacobian SVD, we compute the maximum perturbation magnitude α\alpha at which the cosine similarity between the true network output and its linear approximation remains above 0.95. This threshold represents the distance over which the linear Taylor expansion of the network function serves as a reliable proxy for the true nonlinear function.

Our analysis contrasts untrained and trained MLP models on MNIST, using 50 samples (5 per class) for reliable statistics. For each sample, we compute the Jacobian at the penultimate layer, perform SVD, and measure α\alpha values along 100 evenly distributed RSV directions. The results reveal a striking transformation in local linearity through training. The untrained model (random initialization) exhibits uniformly poor linear approximations, with maximum α\alpha ranging only from 0.2-0.4 across all directions. In contrast, the trained model develops a clear monotonic relationship: directions with higher singular values (lower RSV indices) support shorter linear approximations (α≈0.4\alpha\approx0.4), while directions with lower singular values (higher RSV indices) maintain linearity over much greater distances (α≈1.2\alpha\approx1.2). The transition around indices 300-400, where the singular value spectrum flattens, corresponds to where trained models achieve significantly better linearity than untrained ones.

Remark: Theorem 1 and the Residual Bound

Theorem 1 may be interpreted as requiring the network ff is approximately linear, although approximate linearity is not required for the large, non-adversarial displacements involved in test-time classification. A residual bound is assumed. The theorem follows from the exact identity f(x0+td)=f(x0)+tJf(x0)d+R(t)f(x_0 + td) = f(x_0) + tJ_f(x_0)d + R(t), which holds for any differentiable ff without requiring the network be linear; nonlinearity is fully absorbed into the residual R(t)R(t), bounded by δ\delta over the ray. A larger nonlinear residual requires a correspondingly larger δ\delta, which reduces the magnitude of the resulting GI lower bound; it does not invalidate the theorem. Correspondingly, all GI and ray-coverage values reported in this paper are obtained through direct evaluation of the nonlinear network along each ray, without substituting a linearized network model.

Empirically, a first-order approximation is found to represent the network’s behavior over extended distances along directions associated with the largest GIs. This relationship is examined through representation analysis reported in this section. The analysis above shows that the distance α\alpha over which a linear approximation of the penultimate-layer representation ϕ\phi maintains cosine similarity ≥0.95\ge0.95 with the true representation grows from α≈0.2\alpha\approx0.2–0.40.4 (untrained, roughly uniform across directions) to α≈1.2\alpha\approx1.2 after training, specifically along the low-curvature directions with the largest GIs. Since f=Wϕf = W\phi for a fixed linear classifier head WW, a small residual in ϕ\phi implies a residual in ff scaled by ∥W∥op\lVert W\rVert_{\mathrm{op}} (the same step used in the proof of Theorem 2), relating these observations about ϕ\phi transfers to the quantity Theorem 1 explicitly bounds.

Test-to-Train Generalization Interval Experimental Protocol

Our ray coverage analysis examined whether test samples consistently fall within the GI zones of training samples. For each dataset, we computed:

Coverage⁡(xtest)=1∣Sp∣∑xi∈Sp1[GI⁡(f,xi,xtest−xi)≥1](8)\operatorname{Coverage}(x_{\mathrm{test}}) = \frac{1}{\lvert S_p\rvert}\sum_{x_i\in S_p}\mathbf{1}[\operatorname{GI}(f,x_i,x_{\mathrm{test}}-x_i)\ge1] \tag*{(8)}

where SpS_p is the set of training samples from the predicted class, and 1[⋅]\mathbf{1}[\cdot] is the indicator function. This measures the percentage of training samples whose ray coverage contains the test sample.

Dataset-specific protocols were as follows:

  • MNIST: All 10,000 test samples compared against all training samples of the predicted class (∼\sim6,000 per class), using MLP, CNN, and ViT models with step size δ=0.01\delta= 0.01.

  • CIFAR-10: 50 test samples per class compared against 200 randomly sampled training samples of the predicted class, using ResNet20 and VGG16-BN with step size δ=0.01\delta= 0.01.

  • ImageNette: 20 test samples per class compared against 100 randomly sampled training samples of the predicted class, using VGG16, ResNet50, ViT-B, and DeiT-S with step size δ=0.01\delta= 0.01.

For computational efficiency, we implemented batch processing tailored to each dataset’s dimensionality (6,000/2,000/100 sample pairs per batch for MNIST/CIFAR-10/ImageNette respectively). The consistently high coverage percentages across datasets support our Neural Expansion hypothesis, confirming that classification operates through ray coverage rather than instance-level memorization.

Robustness and Discriminative Validity of Ray Coverage

The ray-coverage measurements used throughout the paper are evaluated with respect to training variability and class specificity. Stability across sources of training randomness is examined, and class-specific structure is assessed through comparisons involving untrained networks and rays directed toward different classes.

Discriminative Baselines: Untrained Networks and Wrong-Class Rays

A 2×22\times2 study was conducted (untrained vs. trained checkpoint ×\times same-class vs. different-class rays) on MLP-MNIST, CNN-MNIST, and CNN-Fashion-MNIST, using n=100n = 100 test samples per architecture and the step-search protocol of Appendix A.4. For the different-class condition, the correct class is held fixed at the test sample’s predicted class cc, while rays are evaluated toward 3 random wrong-class training pools per test sample; this ensures a consistent comparison, since directly reusing the per-class coverage columns from Appendix E is invalid (it makes GI ≤1\le1 tautological, and empirically yields exactly 0%0\%).

ModelUntrained, sameUntrained, diff.Trained, sameTrained, diff.
MLP, MNIST46.8%30.6%88.9%1.2%
CNN, MNIST35.6%42.7%96.3%0.3%
CNN, Fashion-MNIST32.3%34.6%88.5%1.0%

Table 4. Ray coverage (%) under a 2×2 untrained/trained × same/different-class design. Training substantially increases same-class coverage, and trained same-class coverage substantially exceeds trained different-class coverage, confirming coverage is class-discriminative rather than an artifact of the measurement.

At p<10−11p < 10^{-11}, training substantially increases same-class coverage (32–47%→89–96%), and trained same-class coverage substantially exceeds trained different-class coverage (89–96% vs. 0.3–1.2%). The untrained same-vs-different contrast is nonsignificant on Fashion-MNIST (p=0.19p = 0.19) but statistically significant on MNIST (p=3.9×10−12p = 3.9 \times10^{-12} MLP, p=0.02p = 0.02 CNN), indicating a mild same-class pixel bias in untrained networks, with a substantially larger class-specific effect being observed following network training.

Statistical Robustness Across Seeds and Optimizers

The same-class ray-coverage measurement was also repeated (MNIST MLP/CNN, percentage of pairs with GI >1> 1, protocol as in Appendix § 12) across 5 random seeds and 3 optimizers (SGD, Adam, RMSprop; learning rate 0.001) for both architectures, 16 models total, recomputing coverage at several checkpoints from untrained through epoch 20 on a fixed n=50n = 50 evaluation split.

Table 5: Final-epoch ray coverage (%) is nearly seed-invariant and only mildly optimizer-sensitive, whereas substantial variability is observed among untrained baselines (13–99.96% MLP, 24–95% CNN).

ArchitectureFactornn runsMeanStdCV (%)Min–max
MNIST MLPSeed588.910.070.188.82–89.02
MNIST MLPOptimizer391.201.641.889.04–93.03
MNIST CNNSeed596.860.110.196.74–97.04
MNIST CNNOptimizer398.521.221.296.79–99.43

Table 5.

The untrained-vs-final difference is significant (p<0.05p < 0.05, Mann-Whitney U) in 15 of 16 runs, most at p<10−14p < 10^{-14}; the one exception (CNN seed 2, p=0.061p = 0.061) exhibits the same trend, with elevated coverage already observed at the untrained checkpoint for this seed (95.5%) at initialization. The untrained baseline is itself seed-dependent, while the trained endpoint converges tightly regardless of seed or optimizer (CV ≤1.8%\le1.8\%), supporting the stability of coverage measurements used for the results in the main text.

Orthogonal Tube Test: Full Protocol and Results

A class-preserving straight-line ray from a training sample to a test sample establishes stability only along that one direction; the extent of prediction stability in the surrounding neighborhood must therefore be assessed through additional directional measurements. This property is evaluated using an orthogonal “tube” probe: for 100 previously identified rays (GI >1> 1) per model on a trained MNIST-CNN and a trained Fashion-MNIST-CNN, we sample 20 directions exactly orthogonal to each ray at 5 evenly spaced points along it, apply the established step-search algorithm (Appendix § 8.§ 4) along directions perpendicular to the ray, and record the distance to the first class change (the tube radius), normalized by the ray’s own hop distance ∥xtest−xi∥\lVert x_{\mathrm{test}} - x_i \rVert. CNNs are specifically selected, whose local receptive fields make an orthogonal perturbation more likely to induce an early transition than in a fully-connected network, as a challenging case for this test.

Table 6: Orthogonal tube test: orthogonal stability radius around high-coverage rays, normalized by the ray’s own hop distance. A ratio ≥1\ge1 indicates the stable region is at least as wide as the ray is long, in every perpendicular direction tested.

ModelMedian hop distanceMedian tube radiusMedian ratio (tube/hop)
MNIST CNN17.923.11.39×
Fashion-MNIST CNN16.215.41.01×

Table 6.

On both CNNs, the typical high-coverage ray lies within a region at least as wide as the ray itself (Table 6), across every one of the 20 perpendicular directions sampled at every one of the 5 points along the ray. Ray coverage therefore reflects an extended multidimensional same-class region, not a one-dimensional artifact of the specific train-test direction measured.

Class Region Analysis Along Train-to-Test Paths

This section investigates how a model’s class predictions evolve along linear paths from training samples to test samples of the same class. We conduct this analysis using a Multi-Layer Perceptron (MLP) trained on the MNIST dataset. We define ε∈[0,εmax⁡]\varepsilon\in[0,\varepsilon_{\max}] as a scalar interpolation parameter that traces a linear trajectory from a starting input x1x_1 to a target input x2x_2 in input space. Each point along the path is given by x(ε)=x1+ε(x2−x1)x(\varepsilon) = x_1 + \varepsilon(x_2 - x_1). The generalization interval (GI) is the maximum value of ε\varepsilon for which the model’s predicted class remains consistent with the initial prediction at x1x_1. We group path analyses into two categories:

GI > 1.0: The test sample lies within the ray coverage of the training point. These paths typically show a single, uninterrupted class prediction (matching the true class) along the entire trajectory (see Figure 8).

Predicted class trajectories along paths from training to test samples

Figure 8. Predicted class trajectories along paths from training to test samples (Train →\to Test Class 6) with GI > 1.0. Each subplot shows class-stable behavior across the entire interpolation path, indicating the test point lies within the ray coverage of the training sample.

GI < 1.0: The test sample lies outside the ray coverage region. These paths frequently exhibit multiple class transitions, indicating that the model’s classification is unstable along the direction from train to test (see Figure 9).

Predicted class trajectories along paths from training to test samples

Figure 9. Predicted class trajectories along paths from training to test samples (Train →\to Test Class 6) with GI < 1.0. Multiple class transitions are observed along the interpolation, indicating unstable or fragmented class regions.

Together, these visualizations highlight a key behavioral difference: when GI > 1.0, the classification is stable and coherent between the training and test sample. In contrast, GI < 1.0 is associated with fragmented decision regions and unstable class behavior, even between samples of the same true class.

Causal Intervention Experiment: Full Protocol

Derivation of the Population Error Identity

We consider binary classification task with labels 0, 1 and fix the training set S={(xi,yi)}i=1NS = \{(x_i,y_i)\}_{i=1}^{N} sampled from the population. We denote a random sample from the population by (X,Y)∈Rd×R(X,Y) \in\mathbb{R}^{d} \times\mathbb{R}.

For each sample (X,Y)(X,Y) from the population, we assign a sample from the training set (xI,yI)(x_I,y_I) so that yI=Yy_I=Y. The random index II keeps track of which GI to keep track of at sample (X,Y)(X,Y), since GIs from multiple samples may reach the sample (X,Y)(X,Y). It is just a device to make sure that there is no ambiguity about which GI we are considering, and the choice is generally interchangeable. We now define, for B⊂Sd−1B \subset\mathbb{S}^{d-1}, where Sd−1\mathbb{S}^{d-1} is the d−1d-1-dimensional sphere in Rd\mathbb{R}^d,

μi(B)=P(I=i,X−xI∥X−xI∥∈B)(9)\mu_i(B)=P\left(I=i,\frac{X-x_I}{\lVert X-x_I\rVert}\in B\right) \tag*{(9)}

This is the probability that a random sample XX assigned to xix_i points to the set of directions BB when we see xix_i as the origin. In other words, this tells us which directions the population is most concentrated in, when seen from xix_i. We assume that the random variable R=∥X−xI∥2R=\lVert X-x_I\rVert_2 admits a density conditional to the direction U=X−xI∥X−xI∥U=\frac{X-x_I}{\lVert X-x_I\rVert}, denoted pi(r∣u)p_i(r\mid u):

P(I=i,U∈du,R∈dr)=pi(r∣u)drμi(du)(10)P(I=i,U\in du,R\in dr)=p_i(r\mid u)dr\mu_i(du) \tag*{(10)}

Now, let fn:Rn→{0,1}f_n:\mathbb{R}^n\to\{0,1\} denote the prediction model at training time nn. fn(x)f_n(x) denotes the predicted class of xx by the model. One nontrivial assumption we make is the following no-re-entry assumption:

fn(xI+RU)=yI⟺R<GI⁡(fn,xI,U),almost surely.(11)f_n(x_I+RU)=y_I\Longleftrightarrow R<\operatorname{GI}(f_n,x_I,U),\qquad\text{almost surely.} \tag*{(11)}

This statement says that, along the ray from the assigned sample xIx_I, the model predicts the correct original class yIy_I up to the GI boundary and not beyond. Thus, the prediction does not "re-enter" the class yIy_I. We note that this does not mean samples of class yIy_I do not occur beyond GI. In such a case, the model misclassifies these samples.

Now, we will consider the population error of the model: ϵn=P(fn(X)≠Y)\epsilon_n=P(f_n(X)\ne Y). We note that the statement fn(xI+RU)=yIf_n(x_I+RU)=y_I is equivalent to fn(X)=Yf_n(X)=Y by definitions of II, RR, UU, so the no-entry assumption means that

ϵn=P(R≥GI⁡(fn,xI,U))(12)\epsilon_n=P(R\geq\operatorname{GI}(f_n,x_I,U)) \tag*{(12)}
=∑i∫Sd−1∫GI⁡(fn,xi,u)∞pi(r∣u)drμi(du)(13)=\sum_i\int_{\mathbb{S}^{d-1}}\int_{\operatorname{GI}(f_n,x_i,u)}^\infty p_i(r\mid u)dr\mu_i(du) \tag*{(13)}

Therefore, the difference in error between fn+1f_{n+1} and fnf_n is

ϵn+1−ϵn=−∑i∫Sd−1∫GI⁡(fn,xi,u)GI⁡(fn+1,xi,u)pi(r∣u)drμi(du)(14)\epsilon_{n+1}-\epsilon_n=-\sum_i\int_{\mathbb{S}^{d-1}}\int_{\operatorname{GI}(f_n,x_i,u)}^{\operatorname{GI}(f_{n+1},x_i,u)}p_i(r\mid u)dr\mu_i(du) \tag*{(14)}

Remarkably, if the GI increased, i.e., GI⁡(fn,xi,u)≤GI⁡(fn+1,xi,u)\operatorname{GI}(f_n,x_i,u)\leq\operatorname{GI}(f_{n+1},x_i,u) almost surely, then the right-hand side of (14) is nonpositive, so the error decreases (or does not change). We remark that without the no-re-entry assumption, the conclusion is generally false. For example, for the population distribution of X∼Unif⁡([0,3])X\sim\operatorname{Unif}([0,3]) and Y=+1Y=+1, the single training example of (x1,y1)=(0,1)(x_1,y_1)=(0,1), and a training sequence of f0(x)=1{(x−1)(x−2)>0}f_0(x)=1\{(x-1)(x-2)>0\} and f1(x)=1{1−x>0}f_1(x)=1\{1-x>0\}, the GI at x1x_1 to the positive direction is unchanged at 1, but the population error increases because the interval [2,3][2,3], which was correctly classified, turns into incorrectly classified interval, while not affecting GI because [2,3][2,3] is disconnected with the region covered by GI, i.e., [0,1][0,1].

We will complement this theoretical observation with the experiment below.

Interventional Experiment: Design

For both the MNIST MLP (784–1024–512–10, Appendix A.2) and a two-block convolutional Fashion-MNIST CNN with BatchNorm, an auxiliary loss term is added, active in two of three training arms, that acts directly on what a GI measures: whether the prediction remains unchanged under local perturbations around a training point xx. For each (x,y)(x,y) we draw a random unit direction uu and perturb by ε(x)=0.08×∥x∥2\varepsilon(x)=0.08\times\lVert x\rVert_2, placing the step in the 2.0–2.2 range in this normalized space, comparable to standard ℓ2\ell_2 adversarial-perturbation budgets for MNIST-scale data: a local perturbation that is applied independently of another sample location. The hinge objective uses the margin at perturbed inputs, defined as the model’s output for the true class minus its highest output among the other classes:

  • Expand arm: L=CE+λ⋅ReLU(τ−margin⁡(x+εu))\mathcal{L} = \mathrm{CE} + \lambda\cdot\mathrm{ReLU}(\tau- \operatorname{margin}(x + \varepsilon u)), encouraging confident correct classification of the perturbed input.

  • Restrict arm: L=CE+λ⋅ReLU(margin⁡(x+εu)+τ)\mathcal{L} = \mathrm{CE} + \lambda\cdot\mathrm{ReLU}(\operatorname{margin}(x + \varepsilon u) + \tau), encouraging confident misclassification of perturbed inputs, while cross-entropy encourages correct classification at xx.

  • Control arm: standard cross-entropy, λ=0\lambda= 0.

We set τ=2.5\tau= 2.5 to match the mean margin measured on our own trained checkpoint (2.48 over 1000 training images), a target calibrated using observations. At every epoch we measure GI and ray coverage with our existing, unmodified step-search routine (Appendix § 8.§ 4), on a fixed pool of 200 random same-class train-to-train pairs, independent of the training seed and identical across arms. Each configuration is trained with 3 seeds, reported both individually and averaged.

Interventional Experiment: Full Results

Table 1 in the main text reports the final-epoch (epoch 10) results averaged over 3 seeds. Individually, restricting ray coverage on MNIST-MLP lowers test accuracy to 74.7%, 76.5%, and 74.3% across the 3 seeds (p=0.003p = 0.003 vs. control), while expansion yields consistently higher accuracy across seeds (std 0.08, p<10−4p < 10^{-4}). On Fashion-MNIST-CNN, restriction produces a statistically significant reduction (−7.4pp accuracy, −19.5pp ray coverage, p≈0.008p \approx0.008 on both); expanding increases test accuracy by 1.1pp (p=0.008p = 0.008), with a smaller corresponding increase in ray coverage on this architecture. In both datasets, GI and ray coverage separate in the intended direction from epoch 1 onward, on every individual seed, with no overlap between arms. The reported MNIST-MLP restrict arm’s train accuracy (73.9%) is nearly as low as its test accuracy (75.2%), indicating the effect is a slower formation of classification structure overall, rather than merely a widened train/test gap.

Ray Coverage in Challenging Cases

Random Label Experiment Protocol

We evaluate ray coverage in two randomized settings:

  • RANDOM-10: Training labels are reassigned within 10 classes.

  • RANDOM-100: Both training and test labels are reassigned across 100 classes.

Despite the lack of semantic label structure, ray coverage remains active in both settings. Using an MLP architecture (784-1024-512-nn, where nn is 10 or 100) with a GI step size of δ=0.01\delta= 0.01, we evaluated all 10,000 test samples against all training samples of the predicted class.

Although label randomization reduced ray coverage (17%-24% on average) as shown in Table 7, it remained the classification mechanism, in contrast to the higher coverage seen with semantically meaningful labels.

Long-tailed Distribution Analysis Details

We analyze neural network behavior under two long-tailed MNIST variants: MNIST-LT1 and MNIST-LT2, which differ in the severity of class imbalance (6.5:1 and 78.6:1, respectively) (see details in Table 8). In MNIST-LT2, which has a more extreme imbalance, we observe systematic migration of misclassified samples. As shown in Figure 4(b), samples from low-frequency classes tend to be absorbed into the ray coverage of higher-frequency classes.

Class0123456789Ratio
LT1471728662370201518061590134610348867306.5:1
BandHHHMMMMLLL
LT247162504202614641096761383218936078.6:1
BandHHHMMMLLLL

Table 8. Class distributions in MNIST long-tailed variants. H=High, M=Medium, L=Low frequency bands based on 33rd and 66th percentiles.

Despite the imbalance, ray coverage remains strong across all class groups, as shown in Figure 4(c), but the ray coverage from high-frequency classes appears to dominate test-time decisions. In MNIST-LT1, misclassified samples also showed similar migration patterns: low-frequency classes tended to migrate to medium (47.2%) and high (35.2%) frequencies, with only 17.6% staying within the low-frequency group. In both settings, the GI coverage remained consistently high (86.6%–94.3%) across all frequency combinations, confirming that Neural Expansion operates reliably across varying imbalance conditions.

Adversarial Examples Experiment Protocol

We conducted a systematic analysis of how adversarial examples interact with ray coverage across three datasets and architectures:

  • MNIST-MLP: Custom MLP with architecture 784-1024-512-10

  • CIFAR10-ResNet20: Pre-trained ResNet20 from PyTorch hub

  • ImageNette-ViTB: Pre-trained Vision Transformer Base (patch size 16)

For each configuration, we generated adversarial examples using three established attack methods:

  • Carlini-Wagner (CW) (Carlini and Wagner, [9]): Optimization-based attack that minimizes perturbation magnitude with parameters c=1e−2c = 1\mathrm{e}{-2}, learning rate 1e−21\mathrm{e}{-2}, and 1000 iterations.

  • Iterative Fast Gradient Sign Method (IFGSM) (Kurakin et al., [20]): Iterative gradient-based attack with ϵ=0.03\epsilon= 0.03, α=0.005\alpha= 0.005, and 10 iterations.

  • Projected Gradient Descent (PGD) (Madry et al., [24]): Extension of IFGSM with random initialization, using ϵ=0.03\epsilon= 0.03, α=0.005\alpha= 0.005, and 30 iterations.

Our dataset-specific protocols involved:

  • MNIST: 20 samples/class (total 200), compared against all training samples of predicted class

  • CIFAR-10: 10 samples/class (total 100), compared against 200 training samples of predicted class

  • ImageNette: 5 samples/class (total 50), compared against 100 training samples of predicted class

  • MNIST-MLP: Correctly classified adversarial examples show 83.9–86.6% coverage with their predicted class, while misclassified examples show 91.2–94.5%

  • CIFAR10-ResNet20: Correctly classified adversarial examples have moderate coverage (63.9–68.7%), and misclassified examples vary between 39.6–58.1% depending on attack strength

  • ImageNette-ViT-B: Correctly classified adversarial examples exhibit low ray coverage (15.8–19.6%). For misclassified examples, coverage could not be computed reliably due to class predictions falling outside the 10-class subset and into the full 1000-class ImageNet space.

Adversarial Alignment Analysis: We computed the Jacobian SVD at each source image and measured the cumulative projection of both adversarial gradients and image-to-image difference vectors onto the right singular vectors (see Figure 10). Loss gradients (blue lines) concentrate heavily in the first few singular vector directions, often reaching 80–95% cumulative projection within the top 5 directions. In contrast, natural image difference vectors (orange lines) show a more gradual accumulation, typically reaching only 20–40% in the top 5 directions. This contrast explains why adversarial perturbations can be much smaller than natural image differences: they selectively target the model’s most sensitive directions, precisely those with the shortest generalization intervals.

Our sample counts across dataset/model/attack combinations were: MNIST-MLP: CW (132 correct, 68 wrong), IFGSM (81 correct, 19 wrong), PGD (74 correct, 26 wrong); CIFAR10-ResNet20: CW (37 correct, 63 wrong), IFGSM (25 correct, 75 wrong), PGD (35 correct, 65 wrong); ImageNette-ViTB: CW (53 correct), IFGSM (47 correct), PGD (42 correct).

Out-of-Distribution Sample Analysis

Box plot of GI ray coverage for out-of-distribution samples

Figure 11. Box plot illustrating the GI ray coverage for out-of-distribution (OOD) samples from CIFAR-10, evaluated on an MLP trained on MNIST. The gray dashed line (as defined in Figure 3) indicates the 1% directional coverage threshold.

To assess whether out-of-distribution (OOD) inputs fall within learned class-preserving regions, we compute GI ray coverage for CIFAR-10 test samples evaluated on an MLP trained on MNIST. Since the model expects 28×2828 \times28 grayscale inputs, CIFAR-10 images are first transformed to grayscale 28×2828 \times28 images. We follow the MNIST evaluation protocol described in Appendix § 12, using 50 test samples per CIFAR-10 class. Despite being OOD, CIFAR-10 samples exhibit nontrivial GI coverage of ∼10%\sim10\%, showing that the model’s learned ray coverage regions extend into OOD input space. We note that since CIFAR-10 object classes have no semantic correspondence to MNIST digit classes, the “correct/incorrect” stratification here reflects only whether the predicted digit index happens to match the CIFAR-10 class index, and we report aggregate coverage values accordingly.

Potential extensions of this analysis include evaluation under distribution shift (e.g., ImageNet→\toImageNet-Sketch/-R), self-supervised representations (e.g., DINO or CLIP linear probes), and next-token prediction in language models, where the same GI and ray-coverage formalism applies directly to context-level prediction stability.

Proof of Theorem 1

By definition of the linear approximation, we have f(x0+ϵd)=f(x0)+ϵJf(x0)d+Rf(x_0+\epsilon d)=f(x_0)+\epsilon J_f(x_0)d+R. In particular, for each component, we have fi(x0+ϵd)=fi(x0)+ϵ[Jf(x0)d]i+Rif^i(x_0+\epsilon d)=f^i(x_0)+\epsilon[J_f(x_0)d]^i+R^i. If ϵ=GI⁡(f,x0,d)\epsilon=\operatorname{GI}(f,x_0,d), then since we are at the decision boundary, we must have fy(x0+ϵd)=fi(x0+ϵd)f^y(x_0+\epsilon d)=f^i(x_0+\epsilon d) where yy is the class of x0x_0 and ii is another class. Substituting the linear approximation, we obtain

fy(x0)+ϵ[Jf(x0)d]y+Ry=fi(x0)+ϵ[Jf(x0)d]i+Ri(15)f^y(x_0)+\epsilon[J_f(x_0)d]^y+R^y=f^i(x_0)+\epsilon[J_f(x_0)d]^i+R^i \tag*{(15)}

Reorganizing the equation, we obtain

ϵ([Jf(x0)d]y−[Jf(x0)d]i)=fi(x0)−fy(x0)+Ri−Ry(16)\epsilon\left([J_f(x_0)d]^y-[J_f(x_0)d]^i\right)=f^i(x_0)-f^y(x_0)+R^i-R^y \tag*{(16)}

By the inhomogeneous assumption, [Jf(x0)d]y−[Jf(x0)d]i≠0[J_f(x_0)d]^y-[J_f(x_0)d]^i\ne0 and thus

ϵ=fi(x0)−fy(x0)+Ri−Ry[Jf(x0)d]y−[Jf(x0)d]i(17)\epsilon=\frac{f^i(x_0)-f^y(x_0)+R^i-R^y}{[J_f(x_0)d]^y-[J_f(x_0)d]^i} \tag*{(17)}

By the triangle inequality,

∣ϵ∣=ϵ≥∣fi(x0)−fy(x0)∣−∣Ri−Ry∣∣[Jf(x0)d]y∣+∣[Jf(x0)d]i∣(18)|\epsilon|=\epsilon\ge\frac{\left|f^i(x_0)-f^y(x_0)\right|-\left|R^i-R^y\right|}{\left|[J_f(x_0)d]^y\right|+\left|[J_f(x_0)d]^i\right|} \tag*{(18)}

Note that, by the triangle inequality again,

∣Ri−Ry∣≤∑j=1c∣Rj∣=∥R∥1(19)\left|R^i-R^y\right|\le\sum_{j=1}^{c}|R^j|=\lVert R\rVert_1 \tag*{(19)}

and similarly ∣[Jf(x0)d]y∣+∣[Jf(x0)d]i∣≤∥Jf(x0)d∥1|[J_f(x_0)d]^y|+|[J_f(x_0)d]^i|\le\lVert J_f(x_0)d\rVert_1. Thus we have

ϵ≥∣fi(x0)−fy(x0)∣−∥R∥1∥Jf(x0)d∥1(20)\epsilon\ge\frac{\left|f^i(x_0)-f^y(x_0)\right|-\lVert R\rVert_1}{\lVert J_f(x_0)d\rVert_1} \tag*{(20)}

We note that we must have fy(x0)≥fi(x0)f^y(x_0)\ge f^i(x_0) so that x0x_0 is classified as yy. Thus

ϵ≥fy(x0)−fi(x0)−∥R∥1∥Jf(x0)d∥1(21)\epsilon\ge\frac{f^y(x_0)-f^i(x_0)-\lVert R\rVert_1}{\lVert J_f(x_0)d\rVert_1} \tag*{(21)}

Here i=i∗i=i^* denotes the particular wrong class whose logit ties fyf^y at the point x0+ϵdx_0+\epsilon d where the decision boundary is first reached. Since i∗≠yi^*\ne y is one of the c−1c-1 wrong classes, we have fi∗(x0)≤max⁡i≠yfi(x0)f^{i^*}(x_0)\le\max_{i\ne y}f^i(x_0), and therefore

fy(x0)−fi∗(x0)≥fy(x0)−max⁡i≠yfi(x0)=q0.(22)f^y(x_0)-f^{i^*}(x_0)\ge f^y(x_0)-\max_{i\ne y}f^i(x_0)=q_0. \tag*{(22)}

Since the margin is q0=fy(x0)−max⁡i≠yfi(x0)q_0=f^y(x_0)-\max_{i\ne y}f^i(x_0) and also ∥R∥1<δ\lVert R\rVert_1<\delta, we have

ϵ≥q0−δ∥Jf(x0)d∥1(23)\epsilon\ge\frac{q_0-\delta}{\lVert J_f(x_0)d\rVert_1} \tag*{(23)}

Proof of Theorem 2

We establish the bound GI⁡(f,x0,vi)≥q0−δc∥W∥op⁡σi\operatorname{GI}(f,x_0,v_i)\ge\frac{q_0-\delta}{\sqrt{c}\lVert W\rVert_{\operatorname{op}}\sigma_i} by starting from Theorem 1 and evaluating its denominator explicitly at d=vid=v_i using the SVD structure of Jϕ(x0)J_\phi(x_0).

Step 1: Apply Theorem 1 at d=vid=v_i. Setting d=vid=v_i (a unit vector by definition of the SVD) in Theorem 1:

GI⁡(f,x0,vi)≥q0−δ∥Jf(x0)vi∥1(24)\operatorname{GI}(f,x_0,v_i)\ge\frac{q_0-\delta}{\lVert J_f(x_0)v_i\rVert_1} \tag*{(24)}

It suffices to upper-bound ∥Jf(x0)vi∥1\lVert J_f(x_0)v_i\rVert_1, since a larger denominator yields a smaller (still valid) lower bound on GI. Step 2: Convert ℓ1\ell^{1} to ℓ2\ell^{2} norm. For any z∈Rcz \in\mathbb{R}^{c}, the Cauchy–Schwarz inequality gives ∥z∥1=⟨∣z∣,1⟩≤∥z∥2∥1∥2=c∥z∥2\lVert z\rVert_{1} = \langle|z|,1\rangle\le\lVert z\rVert_{2}\lVert1\rVert_{2} = \sqrt{c}\lVert z\rVert_{2}. Applying this:

∥Jf(x0)vi∥1≤c∥Jf(x0)vi∥2(25)\lVert J_f(x_0)v_i\rVert_{1} \le\sqrt{c}\lVert J_f(x_0)v_i\rVert_{2} \tag*{(25)}

Step 3: Apply the chain rule. Since f(x)=Wϕ(x)f(x) = W\phi(x), the chain rule gives Jf(x0)=WJϕ(x0)J_f(x_0) = WJ_\phi(x_0), so:

∥Jf(x0)vi∥2=∥WJϕ(x0)vi∥2(26)\lVert J_f(x_0)v_i\rVert_{2} = \lVert WJ_\phi(x_0)v_i\rVert_{2} \tag*{(26)}

Step 4: Use the SVD identity Jϕ(x0)vi=σiuiJ_\phi(x_0)v_i = \sigma_i u_i. By definition of the SVD Jϕ(x0)=UΣV⊤J_\phi(x_0) = U\Sigma V^{\top}, the right singular vectors are mapped as Jϕ(x0)vi=σiuiJ_\phi(x_0)v_i = \sigma_i u_i where ui∈Rmu_i \in\mathbb{R}^{m} is the ii-th left singular vector with ∥ui∥2=1\lVert u_i\rVert_{2} = 1. Substituting:

∥WJϕ(x0)vi∥2=∥Wσiui∥2=σi∥Wui∥2(27)\lVert WJ_\phi(x_0)v_i\rVert_{2} = \lVert W\sigma_i u_i\rVert_{2} = \sigma_i\lVert Wu_i\rVert_{2} \tag*{(27)}

Step 5: Bound by the operator norm. By definition, ∥W∥op=max⁡∥v∥=1∥Wv∥2\lVert W\rVert_{\mathrm{op}} = \max_{\lVert v\rVert=1}\lVert Wv\rVert_{2}. Since ∥ui∥2=1\lVert u_i\rVert_{2} = 1:

∥Wui∥2≤∥W∥op(28)\lVert Wu_i\rVert_{2} \le\lVert W\rVert_{\mathrm{op}} \tag*{(28)}

Step 6: Chain the bounds and conclude. Combining Steps 2–5:

∥Jf(x0)vi∥1≤c σi∥W∥op(29)\lVert J_f(x_0)v_i\rVert_{1} \le\sqrt{c}\,\sigma_i\lVert W\rVert_{\mathrm{op}} \tag*{(29)}

Substituting into Step 1:

GI⁡(f,x0,vi)≥q0−δc∥W∥opσi(30)\operatorname{GI}(f,x_0,v_i) \ge\frac{q_0-\delta}{\sqrt{c}\lVert W\rVert_{\mathrm{op}}\sigma_i} \tag*{(30)}

Proof of Theorem 3

We prove that the input-space loss gradient is bounded by 2∥W∥op⋅σ1\sqrt{2}\lVert W\rVert_{\mathrm{op}}\cdot\sigma_1.

Step 1: Express the input gradient via the chain rule. The cross-entropy loss is L(x)=−log⁡σy(f(x))\mathcal{L}(x)=-\log\sigma_y(f(x)), where σy\sigma_y denotes the softmax probability of the true class yy. Differentiating with respect to the input xx via the chain rule:

∇xL(x0)=Jf(x0)⊤g(31)\nabla_x\mathcal{L}(x_0) = J_f(x_0)^{\top}g \tag*{(31)}

where g∈Rcg \in\mathbb{R}^{c} is the gradient of the loss with respect to the logits f(x0)f(x_0), with entries gi=σi(x0)−1[i=y]g_i=\sigma_i(x_0)-1[i=y].

Step 2: Bound ∥g∥2\lVert g\rVert_{2}. Since gy=σy(x0)−1g_y=\sigma_y(x_0)-1 and gi=σi(x0)g_i=\sigma_i(x_0) for i≠yi\ne y, with σi(x0)∈[0,1]\sigma_i(x_0)\in[0,1] and ∑iσi=1\sum_i\sigma_i=1, the squared norm satisfies ∥g∥22=(σy−1)2+∑i≠yσi2≤(σy−1)2+(1−σy)2≤2(1−σy)2≤2\lVert g\rVert_{2}^{2}=(\sigma_y-1)^2+\sum_{i\ne y}\sigma_i^2\le(\sigma_y-1)^2+(1-\sigma_y)^2\le2(1-\sigma_y)^2\le2, so ∥g∥2≤2\lVert g\rVert_{2}\le\sqrt{2}.

Step 3: Bound the input gradient norm. Taking the ℓ2\ell^{2} norm and applying submultiplicativity, then using the chain rule decomposition Jf(x0)=WJϕ(x0)J_f(x_0)=WJ_\phi(x_0) and ∥Jϕ(x0)∥2=σ1\lVert J_\phi(x_0)\rVert_{2}=\sigma_1:

∥∇xL(x0)∥2≤∥W∥op⋅σ1⋅2=2∥W∥op⋅σ1(32)\lVert\nabla_x\mathcal{L}(x_0)\rVert_{2}\le\lVert W\rVert_{\mathrm{op}}\cdot\sigma_1\cdot\sqrt{2}=\sqrt{2}\lVert W\rVert_{\mathrm{op}}\cdot\sigma_1 \tag*{(32)}

This establishes (5). □

Proof of Theorem 4

We prove two claims: (1) any successful adversarial perturbation Δ\Delta must satisfy ∥Δ∥2≥min⁡∥d∥=1GI⁡(f,x0,d)\lVert\Delta\rVert_{2}\ge\min_{\lVert d\rVert=1}\operatorname{GI}(f,x_0,d), and (2) the lower bound q0−δc∥W∥opσ1\frac{q_0-\delta}{\sqrt{c}\lVert W\rVert_{\mathrm{op}}\sigma_1}.

Step 1: Decompose Δ\Delta into direction and magnitude. Any nonzero perturbation Δ∈Rd\Delta\in\mathbb{R}^{d} can be written as Δ=∥Δ∥2⋅dΔ\Delta=\lVert\Delta\rVert_{2}\cdot d_{\Delta} where dΔ=Δ/∥Δ∥2d_{\Delta}=\Delta/\lVert\Delta\rVert_{2} and ∥dΔ∥=1\lVert d_{\Delta}\rVert=1. Step 2: Derive the lower bound by contradiction. Suppose the perturbation Δ\Delta successfully changes the classification: arg max⁡ifi(x0+Δ)≠y\operatorname*{arg\,max}_i f^i(x_0+\Delta) \ne y. Suppose for contradiction that ∥Δ∥2<GI⁡(f,x0,dΔ)\lVert\Delta\rVert_2 < \operatorname{GI}(f,x_0,d_\Delta). By definition of GI, moving distance ∥Δ∥2\lVert\Delta\rVert_2 in direction dΔd_\Delta does not change the predicted class, which contradicts the assumption that the attack succeeds. Therefore:

∥Δ∥2≥GI⁡(f,x0,dΔ)(33)\lVert\Delta\rVert_2 \ge\operatorname{GI}(f,x_0,d_\Delta) \tag*{(33)}

Step 3: Optimize over the adversary’s choice of direction. Since the bound holds for any direction dΔd_\Delta, the tightest constraint the adversary faces is:

∥Δ∥2≥min⁡∥d∥=1GI⁡(f,x0,d)(34)\lVert\Delta\rVert_2 \ge\min_{\lVert d\rVert=1} \operatorname{GI}(f,x_0,d) \tag*{(34)}

This establishes Equation (6).

Step 4: Apply Theorem 1. Observe that for any unit dd, by Cauchy-Schwarz and the definition of operator norm,

∥Jf(x0)d∥1≤c∥W∥op∥Jϕd∥2≤c∥W∥opσ1(35)\lVert J_f(x_0)d\rVert_1 \le\sqrt{c}\lVert W\rVert_{\mathrm{op}}\lVert J_\phi d\rVert_2 \le\sqrt{c}\lVert W\rVert_{\mathrm{op}}\sigma_1 \tag*{(35)}

Then, applying this bound to Theorem 1, we obtain the desired bound.

Understanding GI Growth During Training

Our experiments in § 5.1 (and specifically Figure 2a, which compares epoch 1 to epoch 50) show consistently that GIs expand during training, and that this expansion is anisotropic: GIs grow most along high-index RSV directions (small σi\sigma_i) and shrink along low-index RSV directions (large σi\sigma_i). Here we provide a unified understanding of why this happens, grounded in two mechanisms that our experiments and prior work support.

The GI lower bound from Theorem 2 is:

Bi(t)=q0(t)−δc∥W(t)∥opσi(t)B_i(t) = \frac{q_0(t)-\delta}{\sqrt{c}\lVert W(t)\rVert_{\mathrm{op}}\sigma_i(t)}

This bound grows when the numerator (margin) increases and when the denominator (sensitivity) decreases. We identify two mechanisms driving this:

(i) Margin growth. Lyu and Li [22] prove that for homogeneous networks trained with cross-entropy loss, a smooth approximation of the normalized margin increases monotonically during training. This supports the numerator q0(t)q_0(t) being non-decreasing, consistent with what we observe empirically.

(ii) Anisotropic restructuring of Jacobian singular values. As seen in Figure 2(a), training does not uniformly shrink all singular values. Instead, it sharpens the anisotropy: high-index singular values (which are already small) shrink further, while low-index singular values (which are already large) may grow.

This observation provides a finer-grained directional decomposition of the aggregate findings by Novak et al. [28]. They report that the overall Jacobian Frobenius norm, ∥J∥F=∑iσi2\lVert J\rVert_F = \sqrt{\sum_i \sigma_i^2}, decreases during training. Because there are many more high-index (small σi\sigma_i) directions than low-index ones, the net effect on ∥J∥F\lVert J\rVert_F is still a decrease, even while a few low-index σi\sigma_i grow. It is also important to note a structural distinction: while Novak et al. [28] measure the input-output Jacobian Jf=∂f(x)∂xJ_f = \frac{\partial f(x)}{\partial x}, our analysis relies on the penultimate layer Jacobian Jϕ=∂ϕ(x)∂xJ_\phi= \frac{\partial\phi(x)}{\partial x}. These are related by Jf=WJϕJ_f = WJ_\phi, meaning their singular value spectra are connected but not identical. Our empirical results directly measure JϕJ_\phi, making them specifically relevant to our framework.

The consequence of this restructuring is that Bi(t)B_i(t) grows fastest in directions where σi\sigma_i is already small (high-index RSVs), because ∂Bi/∂σi∝−1/σi2\partial B_i/\partial\sigma_i \propto-1/\sigma_i^2 is largest in magnitude there. This directly explains the skewed GI distribution seen in Figure 2(a): after training, GIs are disproportionately large along high-index RSV directions. We note that a fully rigorous proof of individual σi\sigma_i decay during training remains an open theoretical question; our account here is grounded in our own empirical observations of JϕJ_\phi and the aggregate sensitivity results of Novak et al. [28].

References

  1. [1]Adlam, B. and Pennington, J. (2020). The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pages 74–84. PMLR.arxiv.org/abs/2008.06786
  2. [2]Andriushchenko, M., Bahri, D., Mobahi, H., and Flammarion, N. (2023). Sharpness-aware minimization leads to low-rank features. In Advances in Neural Information Processing Systems, volume 36.arxiv.org/abs/2305.16292
  3. [3]Andriushchenko, M. and Flammarion, N. (2022). Towards understanding sharpness-aware minimization. In Proceedings of the 39th International Conference on Machine Learning, pages 639–668. PMLR.arxiv.org/abs/2206.06232
  4. [4]Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. (2018). Stronger generalization bounds for deep nets via a compression approach. In International conference on machine learning, pages 254–263. PMLR.arxiv.org/abs/1802.05296
  5. [5]Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. (2017). Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30.arxiv.org/abs/1706.08498
  6. [6]Beaglehole, D., Súkeník, P., Mondelli, M., and Belkin, M. (2024). Average gradient outer product as a mechanism for deep neural collapse. Advances in Neural Information Processing Systems, 37:130764–130796.arxiv.org/abs/2402.13728
  7. [7]Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2018). Reconciling modern machine learning practice and the bias-variance trade-off. arXiv preprint arXiv:1812.11118 v2, 14.DOI
  8. [8]Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828.DOI
  9. [9]Carlini, N. and Wagner, D. (2017). Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE.arxiv.org/abs/1608.04644
  10. [10]Chizat, L., Oyallon, E., and Bach, F. (2019). On lazy training in differentiable programming. Advances in neural information processing systems, 32.arxiv.org/abs/1812.07956
  11. [11]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.
  12. [12]Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
  13. [13]Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. (2021). Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations.arxiv.org/abs/2010.01412
  14. [14]Frei, S., Vardi, G., Bartlett, P., and Srebro, N. (2023). Benign overfitting in linear classifiers and leaky relu networks from margin maximization. In The Thirty Sixth Annual Conference on Learning Theory, pages 3173–3228. PMLR.arxiv.org/abs/2303.01462
  15. [15]Han, X., Papyan, V., and Donoho, D. L. (2021). Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073.arxiv.org/abs/2106.02073
  16. [17]Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31.arxiv.org/abs/1806.07572
  17. [18]Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations.arxiv.org/abs/1609.04836
  18. [19]Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical report, University of Toronto.
  19. [20]Kurakin, A., Goodfellow, I., and Bengio, S. (2017). Adversarial examples in the physical world. In ICLR Workshop.arxiv.org/abs/1607.02533
  20. [21]LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
  21. [22]Lyu, K. and Li, J. (2020). Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations.arxiv.org/abs/1906.05890
  22. [24]Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.arxiv.org/abs/1706.06083
  23. [25]Movahedi, S. et al. (2025). Geometric inductive biases of deep networks: The role of data and architecture. In International Conference on Learning Representations. Spotlight.arxiv.org/abs/2410.12025
  24. [26]Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2021). Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003.arxiv.org/abs/1912.02292
  25. [27]Neyshabur, B., Bhojanapalli, S., and Srebro, N. (2017). A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564.arxiv.org/abs/1707.09564
  26. [28]Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J. (2018). Sensitivity and generalization in neural networks: an empirical study. arXiv preprint arXiv:1802.08760.arxiv.org/abs/1802.08760
  27. [29]Papyan, V., Han, X., and Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663.DOI
  28. [30]Salman, S., Zhang, C., Liu, X., and Mio, W. (2020). Towards quantifying intrinsic generalization of deep neural networks. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE.arxiv.org/abs/1910.08581
  29. [31]Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations.
  30. [32]Valle-Perez, G., Camargo, C. Q., and Louis, A. A. (2019). Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Conference on Learning Representations.
  31. [33]Vardi, G. (2023). On the implicit bias in deep-learning algorithms. Communications of the ACM, 66(6):86–93.arxiv.org/abs/2208.12591
  32. [34]Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., and Srebro, N. (2020). Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635–3673. PMLR.
  33. [36]Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations.arxiv.org/abs/1611.03530

Paper details

vlnc:W_qnv4mwqy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents