betaFeedback

WorksComputer Science

Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations

Abstract

Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model – they are called robust – while others cannot – they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work

Project page available at: https://gmw99.github.io/data-free-metrics-dont-work/.

Introduction

Understanding and interpreting deep learning models is a critical area of research, especially as the prevalence of these models increases in real-world applications. The holy grail of neural network interpretability lies in identifying computationally efficient metrics that can provide deep insights into the effectiveness of neural networks and their components. Data-free methods typify this endeavour by analysing the properties of neural network parameters without regard for the training or testing

40th Conference on Neural Information Processing Systems (NeurIPS 2026). data, with Random Matrix Theory (RMT) being suggested as ideally suited to understanding deep neural networks [7 Chap. 5] with numerous applications of RMT to deep neural networks.

Prominent among RMT-inspired data-free methods is a family of metrics that can, it is claimed, predict the quality of pre-trained models by only examining their weights. These metrics are based on the theory of Heavy-Tailed Self Regularization (HTSR) developed in [28, 29, 30] and are described in a Nature paper [31] that shows empirically that well-trained models tend to have metrics lying in ‘good’ ranges and concludes that these metrics can be used to discriminate between well-trained and poorly-trained models as well as identify model pathologies. In this paper, we will show that reversing the implication ‘good models ⇒\Rightarrow good metrics’ into ‘good metrics ⇒\Rightarrow good models’ is not only a logical fallacy, it is also empirically incorrect as we show over a range of scales and architectures that bad models can have good metrics. We also invalidate the original premise that ‘good models ⇒\Rightarrow good metrics’ by similarly showing that good models can have bad metrics. We argue that if pre-trained models do indeed exhibit ‘good’ metrics, it is, in part, due to our commonly used initialisation strategies and not because such data-free properties are a prerequisite for generalisation.

Scrutinising the claim that the metrics of [30, 31] can predict the quality of a model is important as numerous lines of research have built on this claim, with many assuming that specific data-free metric properties are fundamental for generalisation. Data-free metrics have been used in the context of layer-wise pruning ratios in LLMs [27], explaining and understanding stages of the grokking phenomenon [35], identifying layers to apply LoRA adapters to [36], adapting layer-wise learning rates [26] and weight decay values [14], informing early stopping conditions [15] and improving optimizers [33]; with data-free metrics often considered to have a unique relationship with generalisation. These metrics thus promise a lot: a better understanding of optimisation, redundancy, transitions from memorisation to generalisation, and interpretability of deep neural networks more generally. Our scrutiny of these metrics should be commensurate with these promises.

Our strategy to understand whether data-free metrics of [30, 31] are causal for performance or generalisation follows the well-known work of Dinh et al. [9] which showed that measures of flatness around local minima are not causal for generalisation. They did so by constructing functionally equivalent networks for which measures of flatness around a local minimum could be arbitrarily far apart, thereby disproving links between flatness and generalisation. We follow the same strategy, but focus on empirical results showing that functionally preserving models can have very distant metrics, thereby disproving the link between data-free metrics and model performance or training.

An experimentally useful class of functionally preserving models is provided by the work of Zhang et al. [47, 48] that identified an interesting and unexpected phenomenon in neural network layers: some layers within a network are robust, while others are critical. A robust layer can be either re-initialised or re-randomised without any noticeable effect on performance. A critical layer is a layer that cannot be re-initialised or re-randomised without dramatically affecting the performance of the network. Re-initialisation sets the layer back to its initial parameters before training, whilst re-randomisation sets the parameters of the layer to random values by re-sampling from the same distribution used for initialisation. It was observed that in some cases, re-initialisation and re-randomisation can result in significant performance differences for a given layer, with re-initialisation maintaining performance but re-randomisation significantly degrading it [48]. In other cases, re-initialisation and re-randomisation of a layer lead to a negligible difference in performance. This distinction between layer behaviours offers a core tool for testing the robustness of data-free metrics. For our purpose, re-initialisation and/or re-randomisation of robust layers will be used to build functionally preserving models for which the HTSR-based data-free metrics will take a large range of (mostly ‘bad’) values. In other words, the metrics offered by HTSR are not reparameterisation-invariant under re-initialisations or re-randomisations of robust layers and therefore only offer spurious insights into generalisation or well-trainedness. We also show that these metrics cannot distinguish between robust and critical layers, nor predict performance differences between re-initialisation and re-randomisation.

As a natural follow-up question from our layer-wise analysis of reparameterisation and data-free metrics, we investigate global data-free properties (average value of these metrics over all layers) and generalisation. Our results show that global data-free metric values can also be arbitrary for well-generalising models: neither layer-wise nor average data-free metrics relate to generalisation.

We hope that our work can push the field forward in two core ways. Firstly, by providing a strong test ground for the development of data-free metrics that are reparametrisation invariant, or at least do not differ too much on functionally preserving reparametrisations of models. This is precisely what the work of [9] led to, with the development of new sharpness metrics such as Relative Flatness [34] and Fisher-Rao norm [24] that are reparameterisation invariant and now considered state of the art sharpness metrics. Secondly, our work offers concrete evidence for the prioritisation of data-based metrics to improve efforts in interpretability and compression (at least until adequate data-free metrics are discovered).

Background

As discussed briefly above, data-free methods of interpretability aim to understand the inner workings of neural networks by studying the properties of the network parameters. Data-free approaches often focus on the matrix norm properties of layer weight matrices to understand learning or improve the performance of neural networks [46, 31, 40, 11, 39, 2, 45, 36, 42, 14]. However, [48] showed in their work that matrix norms, such as the Frobenius norm, are too coarse to understand the generalisation properties of neural networks. [30, 31] use RMT to analyse the weight matrices (excluding biases) of neural network layers through training to create their theory of heavy-tailed self-regularisation (HTSR). With this theory, they construct a set of predominately power-norm metrics related to generalisation that is applied after training to assess layer performance. Given a weight matrix W∈Rm×nW \in\mathbb{R}^{m \times n} and λi\lambda_i, 1≤i≤n1 \leq i \leq n the eigenvalues of W⊤WW^{\top}W the following metrics are defined:

  • α\alpha and weighted-α\alpha: the exponent of the power-law that best describes the distribution of the eigenvalues ρ(λ)∼λα\rho(\lambda) \sim\lambda^{\alpha} [30], and αλmax⁡\alpha\lambda_{\max}. Estimating α\alpha is not straightforward [30 3.2] and we show through several experiments in § K that the distribution of eigenvalues is typically poorly modelled by power-laws, undermining the statistical basis for α\alpha and its derivatives.

  • α\alpha-norm and log-α\alpha-norm: ∑iλiα\sum_i \lambda_i^\alpha, ln⁡(∑iλiα)\ln\left(\sum_i \lambda_i^\alpha\right)

  • Frobenius and log-Frobenius norm: ∑iλi\sum_i \lambda_i, ln⁡(∑iλi)\ln\left(\sum_i \lambda_i\right)

  • Spectral and log-spectral norm: λmax⁡\lambda_{\max}, ln⁡λmax⁡\ln\lambda_{\max}

Following in the footsteps of [9], we can already dismiss several of these metrics as potential indicators of performance or well-trainedness. For this, consider a two-layer network ϕ(x)=W2⋅ReLU⁡(W1⋅x)\phi(x) = W_2 \cdot\operatorname{ReLU}(W_1 \cdot x) and the reparametrisation maps Tb:(W1,W2)↦(bW1,b−1W2)T_b : (W_1, W_2) \mapsto(bW_1, b^{-1}W_2), b>0b > 0. It is clear that ϕ\phi is invariant under this reparametrisation, i.e. ϕ(x)=b−1W2⋅ReLU⁡(bW1⋅x)\phi(x) = b^{-1}W_2 \cdot\operatorname{ReLU}(bW_1 \cdot x). Since the spectrum of (bW⊤)bW(bW^{\top})bW is the spectrum of W⊤WW^{\top}W scaled by the constant positive factor b2b^2, it follows immediately that:

Theorem 1 Let MM be any of the metrics above except α\alpha. For any value K>0K > 0, there exists b>0b > 0 such that the value of MM applied to the layers of the network ϕ\phi reparametrised by TbT_b is greater (resp. smaller) than KK.

Thus, two functionally equivalent networks can differ under all the metrics above – except α\alpha – by an arbitrary amount. It would therefore be unreasonable to expect these metrics to have any predictive power on the quality or well-trainedness of a given layer in a network. Note that taking the average of the metrics over both layers (as done in [31]) will lead to the exact same conclusion.

However, α\alpha itself is invariant or close to invariant (depending on estimation techniques) under TbT_b. For example, the standard maximum likelihood estimator for α\alpha is given by α^=n∑iln⁡(λi/λmin⁡)\hat{\alpha} = \frac{n}{\sum_i \ln(\lambda_i/\lambda_{\min})} which is clearly invariant under rescaling the eigenvalues. Other metrics involving quotients of eigenvalues, for example stable rank, are also outside of the scope of Theorem 1. We will thus focus on the metric α\alpha in the rest of this paper (results for other metrics can be found in § G). According to [30, 31], well-trained models have α\alpha values in the optimal range [2, 6] for most of their layers, e.g. [30 Table 6, Fig. 12], [31, Figs. 6,7] with values of α<2\alpha< 2 corresponding to overfitting/memorisation and α>6\alpha> 6 to underfitting (see also https://weightwatcher.ai).

The map TbT_b defined above provides an example of reparametrisation, i.e. of a map g:Θ→Θg : \Theta\to\Theta on the space of parameters of a model, typically assumed to be a homeomorphism or at least a bijection. Note that whilst reparametrisations are often mentioned in the context of invariance properties under said reparametrisations, these are disjoint concepts, and in this work we will simply understand a reparametrisation as a homeomorphism on the space of parameters of a network without implying any invariance property. We identify a model with its defining set of parameters θ∈Θ\theta\in\Theta and say that a metric M:Θ→R+M : \Theta\to\mathbb{R}^{+} is invariant under a reparametrisation gg if M=M∘gM = M \circ g, for example α=α∘Tb\alpha= \alpha\circ T_b in the example above. Any re-initialisation or re-randomisation of the weight matrix WlW_l associated with the layer ll of a model defines a unique linear bijective (and thus homeomorphic) reparametrisation g:Θ→Θg : \Theta\to\Theta. It stands to reason that for a model θ\theta, if a reparametrisation g(θ)g(\theta) of this model is functionally indistinguishable from θ\theta, then any metric MM meant to capture model generalisation should also be (close to) invariant under gg for θ\theta, i.e. M(θ)≈M(g(θ))M(\theta) \approx M(g(\theta)).

Experimental Setup

Reparametrisation Invariance, Robust and Critical Layers: [48] demonstrated the robust and critical layer phenomenon across a range of trained architectures – MLPs, VGGs [41], ResNets [17], Transformers [44], Vision Transformers [10], MLPMixers [43] – and of datasets – MNIST [23], CIFAR10 [22], ImageNet [8] and LM1B [5].

In § 4.1 we start with the simplest model (ReLU FCN 5x512), Figure 1, and dataset (MNIST) that demonstrates this phenomenon [48] in order to systematically explore the behaviour of data-free metrics under layer re-initialisations and re-randomisations of a large number of trained models.

ReLU FCN Architecture

Figure 1. ReLU FCN Architecture.

An added benefit of the ReLU FCN 5x512 model is that it offers a clear performance contrast between re-initialisation and re-randomisation. [48] showed that residual blocks are robust to re-randomisation and attributed this to the residual layer potentially playing a lesser role in the network and thus having smaller activations than the skip connection. To analyse how effective the data-free metrics are at disambiguating between re-initialisation and re-randomisation, we analyse the correlation between data-free metrics and test accuracy using the Spearman correlation coefficient ρ\rho, coefficient of determination R2R^2, the root mean square error (RMSE) of the linear regression and Kendall’s tau measure (K-τ\tau). Where ρ\rho and K-τ\tau score of -1 indicates a very strong negative correlation, 0 indicates no correlation, and 1 indicates a very strong positive correlation and R2R^2 score of 1 indicates perfect predictions and a score of 0 indicates imperfect predictions. We use R2R^2, RMSE and Kendall’s Tau measure for this study as they are the correlation metrics used by [31] to highlight the predictive capacity of the data-free metrics. We trained 100 ReLU FCN 5x512 models, creating 100 initialisations and 100 trained models, to obtain a representative sample of possible initialisations and trained models. The model weights and biases are initialised and re-randomised from the same distribution U(−k,k)U(-\sqrt{k},\sqrt{k}) where kk is 1nfeatures\frac{1}{n_{\mathrm{features}}}, e.g. FC1 has k=1784k = \frac{1}{784}. We record the data-free metric properties of these trained models’ layers when they undergo re-initialisation and re-randomisation and observe whether these metrics:

  1. Are invariant under re-initialisation and re-randomisation, particularly for robust layers,

  2. Can distinguish critical from robust layers,

  3. Can predict the performance difference between re-initialisation and re-randomisation.

In the process we demonstrate the overall lack of predictive capacity of data-free metrics after training.

In § 4.2, we extend our analysis to large-scale pre-trained computer vision models (ResNet34 and ViT) both trained on ImageNet [38] and pre-trained large language models GPT2 and GPT2-Large evaluated on WikiText103 [32]. As we employ pre-trained open source models we do not have access to the starting initialisation and as we result we only focus on the case of re-randomisation to test the predictive capacity of the α\alpha metric at this scale, concretely questions 1 and 2. We perform re-initialisation and re-randomisation at large scale for the Pythia-1B model [4] and confirm our findings at the MNIST scale in Sections§ B. Furthermore, we provide empirical support for linking functional preservation to accuracy preservation under reparametrisations (re-initialisation and re-randomisation) in the main body for MNIST and in § C for all other models explored. In § G.1 we additionally explore the following metrics: Alpha Weighted (α^\hat{\alpha}, Log Alpha Norm, MP Soft Rank [30], Spectral Norm, Stable Rank [37] and Generalized von-Neumann Matrix Entropy [30]: which are formally defined in § G. The metrics are collected using the weightwatcher2 tool.

Average Alpha vs Generalisation. We perform two sets of experiments to investigate the (lack of) relationship between average/global alpha and generalisation. In both cases we rely on the singular value decomposition (SVD) W=UΣVTW = U\Sigma V^T of the weight matrices of the model’s layers.

In our first set of experiments (§ 4.3), we study the impact of alpha at initialisation on performance. For this, we construct a family of initialisations by power laws that interpolate the max-normalised singular values Σλ\Sigma_\lambda between a given initial sampled spectrum Σλ=1\Sigma_{\lambda=1} and two natural endpoints: the full rank spectrum given by the identity matrix II (which corresponds to λ=0\lambda= 0, α→∞\alpha\to\infty), and the rank 1 spectrum (1,0,...,0)(1,0,...,0) (which corresponds to λ=100001\lambda= 100001, α→0\alpha\to0). For each interpolation levels λ\lambda we initialise the weight matrices of each layer of the model as Wλ=UΣλVTW_\lambda= U\Sigma_\lambda V^T for some randomly initialised orthogonal matrices U,VU,V. This controlled intervention induces different values of α\alpha at initialisation which allows us to systematically study how initialisation affects both performance and the value of α\alpha after training. In particular it allows us to assess if models whose layers are all initialised with an α\alpha within the ‘good’ range [2,6][2,6] achieve better than random accuracy.

In the second experiments (§ 4.4) we extend this analysis to training with a fixed α\alpha. Concretely, we sample from the random initialisation for each layer to obtain singular values that correspond to a desired α\alpha value, Σα\Sigma_\alpha, and then construct Wα=UΣαVTW_\alpha= U\Sigma_\alpha V^T at each layer and keep Σα\Sigma_\alpha fixed throughout training. We then solely train UU and VTV^T while maintaining the unitary properties. We explore α\alpha within the ranges of 2≤α≤62 \leq\alpha\leq6, 10≤α≤1210 \leq\alpha\leq12, and 18≤α≤2018 \leq\alpha\leq20 for all layers.

Results and Discussion

In the main body we present our results for α\alpha. We show that these results generalise to other metrics in § G. In the tables below, we report the mean and the standard error of the mean (SEM) [3]. We show that alpha is not invariant under functionality-preserving reparametrisations in small-scale (MNIST) settings in § 4.1. We then show that our findings hold for other scales and modalities by exploring ImageNet models and the GPT family in § 4.2, with full re-initialisation and re-randomisation for the Pythia-1B model in § B. In § H we show that data-based metrics explain the performance difference from re-initialisation and re-randomisation through their implicit relationship with the lottery ticket hypothesis [12]. We then explore initialisation and average alpha, showing that the optimal alpha range is somewhat arbitrary in § 4.3 and unreliable in § K. Additionally, in § C we conduct a full functional similarity assessment of reparametrisation for all models to show that when accuracy is preserved so is function. Finally, we provide evidence that optimal global alpha values are not causal for well-generalising models by optimising models with arbitrary alpha values in § 4.4 and through a full SVD ablation for learning dynamics in § J.

Small Scale Analysis of Alpha Under Reparametrisation and Training

Quality of training vs alpha. The idea that well-trained layers have α∈[2,6]\alpha\in[2,6] seems to be borne out by training 100 ReLU FCN 5x512 models to good values of test accuracy, achieving α∈[2,6]\alpha\in[2,6] for every layer (see Table 1). Whilst well-trained may imply α∈[2,6]\alpha\in[2,6] (but more on this below), a simple experiment shows that the converse does not hold. We plot the empirical distribution of α\alpha for a 512x512 fully connected layer in Figure 2a, sampled from 10,000 potential initialisations. The resulting distribution shows that at initialised, untrained layer of this network, can fall, with a small but non-negligible probability, within the optimal α\alpha value range of 2 and 6. This highlights that a random layer can be considered optimal under RMT which suggests a core failing of the metric.

MetricLayerFC1FC2FC3FC4FC5FC6Test Accuracy
Alpha (α\alpha)4.820 ± 0.0254.205 ± 0.0394.126 ± 0.0384.135 ± 0.0354.193 ± 0.0343.793 ± 0.80596.822 ± 0.057

Table 1. Alpha (α\alpha) of the layers in ReLU FCN 5x512 and test accuracy of the model on MNIST.

Empirical distribution of α values and layer test accuracy versus α

Figure 2. (a) Empirical distribution of α\alpha values on a 512x512 fully connected layer, sampled from 10,000 initialisations. (b) Layer re-initialisation (🔵) and re-randomisation (🟠) test accuracy vs α\alpha. In § D we present the descriptive statistics for sub-figure (a) along with other initialisation schemes.

Reparametrisation-invariance of alpha. Next, we perform independent re-initialisations (🔵) and re-randomisations (🟠) in Figure 2b of each layer for 100 trained networks and record the impact on the networks’ test accuracy and alpha values. Figure 2b shows that a non-negligible proportion of networks whose performance is severely degraded by re-randomisation maintain a good value of α\alpha around 5. Conversely, many networks maintain good performance after re-initialisations (particularly of layers FC3-FC5) but with α\alpha values significantly outside of [2,6][2,6]. As is clear from Figure 2b, whilst all values of α\alpha start in the ‘good’ range [2,6][2,6], reparametrisation in the form of re-initialisation or re-randomisation scatters these values over a very wide range (typically deep into “undertitted” territory), irrespective of the change in accuracy incurred by this reparametrisation. Alpha is not invariant under these reparametrisations, even when accuracy is (i.e. on robust layers).

The robust vs critical phenomenon and alpha. Figure 2b shows the stark contrast in how layers respond to re-initialisation and re-randomisation. For example, we only see a large drop in test accuracy when applying re-initialisation to the Layer FC1, re-initialising other layers leaves performance almost unchanged. From this perspective, FC1 is a critical layer whilst FC2-FC6 are robust (to re-initialisation). In this experiment α\alpha cannot distinguish between these behaviours, the range of values taken by α\alpha shows no noticeable difference between critical and robust layers.

The re-initialisation vs re-randomisation phenomenon and alpha. We observe different results when re-randomisation is applied. We find that re-randomising any layer degrades accuracy to circa random accuracy on the test set, in other words none of the layers are robust to re-randomisation. Surprisingly, this is not reflected in the corresponding α\alpha values of these two conditions, as the distribution of α\alpha values is relatively similar for each layer and each condition. If α\alpha had predictive power we would observe a negative correlation between re-initialisation (corresponding to ‘good’ alphas) to re-randomisation (corresponding to ‘bad’ alphas). However, there is no difference between the α\alpha values of re-initialisation and re-randomisation, with a mean Spearman correlation coefficient and Kendall’s tau measure across layers of -0.053 and -0.035, respectively. Thus α\alpha does not distinguish between these behaviours either.

Generalisation of Reparametrisation Invariance to other Data-Free Metrics. We extend our analysis to other data-free metrics that have been defined in the literature. In spite of Theorem 1 and the lack of predictive power of the α\alpha metric described above, it is important to verify across other representative measures that data-free metrics are not invariant under reparametrisation (of robust layers), and cannot disambiguate between critical and robust layers, nor between re-initialisation and re-randomisation. When we conduct a similar analysis across six other metrics, Alpha Weighted, Log Alpha Norm, MP Soft Rank, Spectral Norm, Stable Rank and Generalized von-Neumann Matrix Entropy we find, as shown in § G Figure 9, that we can draw exactly the same conclusions with α\alpha. In § H we show that data-based metrics offer some predictive capacity in reference to criticality.

Layer re-initialisation and re-randomisation test accuracy versus data-free metrics

Figure 9. Layer re-initialisation (●) and re-randomisation (●) test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RMSERMSE is the root mean square error of the linear regression (red line), and K-τ\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

Reparametrisation, Functional Similarity and Accuracy Preservation. As we perform interventions on particular layers of specific models, preserving accuracy can be seen as a proxy for function preservation. To quantify the faithfulness of this proxy in our experiments, we employ functional and representational similarity measures [20] to quantify the output similarity of the original model and the model with the intervention (re-initialisation, Table 2, or re-randomisation, Table 3) on the test set. In Table 2 we observe that applying re-initialisation across layers leads to varying levels of functional preservation, with layers that have a low relative change in accuracy (FC2-FC5) maintaining functional similarity across the respective metrics while layers with higher accuracy deviation (FC1 and FC6) preserving less of the predictive structure of the original model. In the case of re-randomisation in Table 2, there are persistent high deviations in accuracy which is accompanied by increased divergence from the original model’s output distribution across similarity metrics. Overall, the correlation between functional similarity and model accuracy presented in Table 4 (for several functional similarity and correlation metrics) shows that increased functional similarity corresponds to increased accuracy, justifying the following statement: When accuracy is preserved under the reparametrisations offered by re-initialisation and re-randomisation predictive structure of the original model is approximately preserved too.

Model LayerΔ\Delta AccuracyΔ\Delta LossPrediction Similarity (↑\uparrow)Jensen-Shannon Distance (↓\downarrow)Procrustes Distance (↓\downarrow)KL Divergence (↓\downarrow)
FC1-39.859 ±\pm 0.9361.226 ±\pm 0.0380.574 ±\pm 0.0090.459 ±\pm 0.0030.340 ±\pm 0.0031.014 ±\pm 0.024
FC2-6.330 ±\pm 0.3280.285 ±\pm 0.0070.917 ±\pm 0.0030.272 ±\pm 0.0020.087 ±\pm 0.0010.267 ±\pm 0.004
FC3-1.899 ±\pm 0.1310.297 ±\pm 0.0030.962 ±\pm 0.0010.294 ±\pm 0.0010.053 ±\pm 0.0010.289 ±\pm 0.002
FC4-1.384 ±\pm 0.0800.317 ±\pm 0.0030.968 ±\pm 0.0010.308 ±\pm 0.0010.051 ±\pm 0.0010.314 ±\pm 0.002
FC5-1.716 ±\pm 0.1140.364 ±\pm 0.0030.963 ±\pm 0.0010.329 ±\pm 0.0010.054 ±\pm 0.0010.356 ±\pm 0.002
FC6-6.966 ±\pm 0.3320.522 ±\pm 0.0040.910 ±\pm 0.0030.383 ±\pm 0.0010.061 ±\pm 0.0010.487 ±\pm 0.003

Table 2. Functional preservation of the ReLU FCN 5x512 by layer under the re-initialisation reparametrisation. (↑\uparrow / ↓\downarrow) denotes the preferred direction for each metric.

Model LayerΔ\Delta AccuracyΔ\Delta LossPrediction Similarity (↑\uparrow)Jensen-Shannon Distance (↓\downarrow)Procrustes Distance (↓\downarrow)KL Divergence (↓\downarrow)
FC1-86.928 ±\pm 0.2173.704 ±\pm 0.0520.102 ±\pm 0.0020.721 ±\pm 0.0010.894 ±\pm 0.0032.601 ±\pm 0.020
FC2-86.692 ±\pm 0.2732.533 ±\pm 0.0190.104 ±\pm 0.0030.712 ±\pm 0.0010.806 ±\pm 0.0052.303 ±\pm 0.010
FC3-86.562 ±\pm 0.3322.348 ±\pm 0.0090.105 ±\pm 0.0030.711 ±\pm 0.0000.647 ±\pm 0.0082.237 ±\pm 0.006
FC4-87.350 ±\pm 0.4152.300 ±\pm 0.0070.097 ±\pm 0.0040.711 ±\pm 0.0000.466 ±\pm 0.0082.228 ±\pm 0.006
FC5-87.007 ±\pm 0.4772.315 ±\pm 0.0090.101 ±\pm 0.0050.711 ±\pm 0.0010.335 ±\pm 0.0072.232 ±\pm 0.006
FC6-86.848 ±\pm 0.6802.356 ±\pm 0.0120.102 ±\pm 0.0070.711 ±\pm 0.0010.221 ±\pm 0.0072.263 ±\pm 0.011

Table 3. Functional preservation of the ReLU FCN 5x512 by layer under the re-randomisation reparametrisation. (↑\uparrow / ↓\downarrow) denotes the preferred direction for each metric.

Similarity Metricρ\rhoR2R^2RMSEK-τ\tau
Prediction Similarity0.9971.0000.0040.964
Jensen-Shannon Distance-0.8690.9770.030-0.672
Procrustes Distance-0.8090.6350.189-0.604
KL Divergence-0.8640.9760.149-0.665

Table 4. Correlation between the change in accuracy due to re-initialisation (Table 2) and re-randomisation (Table 3) and the functional preservation of the ReLU FCN 5x512.

Analysis of larger models (ResNet’s, ViT’s, GPT2 and Pythia-1B) in § C confirms this conclusion.

Large Scale Analysis of Alpha Under Reparametrisation

This section explores the predictive power of alpha with respect to performance in large scale models that have been trained for high-complexity tasks using the reparametrisation by re-randomisation setup at the hundred million parameter scale. We find, in line with our small scale experiments, that alpha has no predictive capacity in this regime either. For these experiments, we use pre-trained open source models, and we therefore have no access to pre-training initialisations. As a result, we only focus on reparametrisation by re-randomisation to test the invariance and predictive capacity of the α\alpha metric. The results in this section are consistent with other data-free metrics in this experimental setup which we present in § G.2, G.3, G.4 and G.5.

ImageNet. We explore the relationship of a layer’s α\alpha value with the corresponding performance of the model on ImageNet with pre-trained ResNet34 and ViT models. The respective performance and model details are shown in Table 5; from this table it can be observed that these models are well-trained for the task of ImageNet. In line with the RMT perspective, the competitive performance of these models is aligned with a mean α\alpha value across layers within the optimal range.

ModelNumber of ParametersLossAccuracyMean Layer Alpha Value
ResNet3421.8M1.07173.3023.380 ±\pm 0.195
ViT86.6M0.84879.1364.287 ±\pm 0.231

Table 5. Model test performance on ImageNet and mean layer Alpha (α\alpha) value.

To conduct this experiment we randomise a layer independently and then record the corresponding test accuracy and α\alpha value post re-randomisation. Given the different implicit biases offered by the ResNet and ViT we believe that our selection of models is representative for the core architectural differences, allowing a robust analysis of the impacts of re-randomisation and α\alpha metrics at scale. We repeat our re-randomisation process 10 times per layer to capture stochastic variation in randomisations. In both ResNet34 and ViT, Figure 3, we find no relationship between α\alpha and test accuracy, as found in the small scale experiment. Both models can be re-randomised to α\alpha values outside of the desired range and maintain accuracy. The mean Spearman correlation coefficients for alpha and accuracy are 0.006 and -0.036 respectively, and Kendall’s tau values are 0.004 and -0.022, respectively. In the case of ResNet34 we see that the model is not so robust to re-randomisation, having a lower accuracy than the baseline; a large proportion of layers with optimal α\alpha values result in a model with essentially random accuracy, i.e. ‘good alpha does not imply good model’. The ViT is more robust to layer re-randomisation and represents the inverse case where many layers are significantly outside of the optimal α\alpha range but display a high preservation of the baseline accuracy, i.e. ‘good model does not imply good alpha’. As a result, these results confirm in large scale settings that α\alpha is not invariant under reparametrisation by re-randomisation, even on robust layers, and has very limited predictive capacity over post re-randomisation performance.

Figure 3. Layer re-randomisation Test Accuracy vs Alpha (α\alpha) on ImageNet. See stratified analysis in Appendix F.

Large Language Models. We confirm our negative results for the predictive capacity of the α\alpha metric in the vision domain under re-randomisation, by examining a different modality. We extend our exploration of the relationship between layer α\alpha values and the corresponding performance of the model to language models on WikiText103 with pre-trained GPT2 and GPT2-Large. The respective performance and model details is shown in Table 6. Both models have competitive performance with low loss and perplexity on the WikiText-103 [32] datasets and, in-line with HTSR Theory, have mean layer α\alpha values that are within the optimal range. We use the pre-trained models as the base model and then we randomise a layer and record its α\alpha value and the corresponding test perplexity and loss of the model. We do this 10 times for each layer to obtain a representative set of random layers. For both GPT2 and GPT2-Large, Figure 4, we observe that their is no relationship between the performance of the model and a layer’s α\alpha value, with a mean Spearman correlation coefficient of -0.038, -0.134 and Kendall’s tau measure of -0.025 and -0.089 for GPT2 and GPT2-Large respectively. We also find that randomising layers in the GPT2 and GPT2-Large model often has a negligible affect on the performance of the model irrespective of the α\alpha value associated with the layer. In other words, re-randomising generates functionally preserving reparametrisations of the original model, but with arbitrary values of alpha for the layer under consideration. Furthermore, when re-randomising robust layers, we observe that most of these layers lie far outside the optimal range of α\alpha. Again, we see that α\alpha is not invariant under reparametrisation by re-randomisation, even on robust layers, and has no predictive power over performance.

ModelNumber of ParametersLossAccuracyPerplexityMean Layer Alpha Value
GPT127M3.39939.09529.9413.865±0.1363.865 \pm0.136
GPT Large774M2.96743.87719.4364.013±0.0914.013 \pm0.091

Table 6. Model test performance on WikiText-103 and mean layer Alpha (α\alpha) value.

Layer re-randomisation Test Accuracy vs alpha on WikiText103 for GPT2 and GPT2-Large

Figure 4. Layer re-randomisation Test Accuracy vs α\alpha on WikiText103. See stratified analysis in Appendix F

Alpha At Initialisation & Training

The previous sections (§ 4.1 and § 4.2) showed that the α\alpha of a layer has no (causal) relationship with model performance. However, it could be argued that the metric is best interpreted as a global metric averaged across all layers, as done in [31]. Therefore, we explore the average model α\alpha and

its relationship to generalisation. In this section we report on the first of the two sets of SVD-based experiments described in § 3. All experiments are carried out on the ReLU FCN 5x512 model of § 4.1.

Alpha and Initialisation. In Figure 5 we explore the test accuracy of untrained models initialised with a specific model α\alpha following the singular value interpolation Σλ\Sigma_\lambda described in § 3. The ten black dots (●) represent standard uniform initialisations of a model, i.e. ten starting spectra Σ1\Sigma_1, that each generate 20 interpolations spectra towards (1,0,..,0)=Σ∞(1, 0, .., 0) = \Sigma_\infty (●) and another 20 interpolations towards I=Σ0I = \Sigma_0 (●). We observe that models within the ‘good’ range 2≤α≤62 \le\alpha\le6 (the grey region in Figure 5) exhibit no performance increase versus models outside of this range (● for α<2\alpha< 2 and ● for α>6\alpha> 6) although a global alpha in this range is meant to be indicative of a good, well trained model. Figure 5 thus highlights the lack of causal relationship with test accuracy. We observe negligible correlation between an untrained model’s initial α\alpha and its untrained test accuracy.

Figure 5. Test Accuracy vs Model α\alpha at initialisation.

Alpha and Training. As noted in [30, 31], models that generalise do consistently have an average model α∈[2,6]\alpha\in[2, 6], which does suggest that it is an interesting property of a model. If, as we have shown repeatedly, α\alpha is not connected to generalisation, how can we explain this empirical observation? To answer this question we explore how α\alpha changes throughout training in Figure 6, where the lines of the black dots (●) represent the standard initialisations training trajectories. The models initialised with α\alpha between 1 and 3 fail to generalise and maintain the corresponding α\alpha value throughout training. The rest of the models initialised within the good α\alpha range do generalise and stay within this range throughout training. The model’s initialised with an α>6\alpha> 6 do generalise throughout training, and witness a decrease in model alpha tending towards the ‘good’ range of 2≤α≤62 \le\alpha\le6 in training.

Figure 6. Test Accuracy vs Model α\alpha through training.

Figure 7 shows a positive correlation between the initialisation α\alpha and the trained α\alpha of the models. Models initialised using the standard uniform sampling (i.e. with Σ1\Sigma_1 initialisations) land in the ‘good’ range [2,6][2, 6], initialisations Σλ\Sigma_\lambda, λ<1\lambda< 1 land beyond α>6\alpha> 6. It is thus likely that if most trained models have an average α∈[2,6]\alpha\in[2, 6], it is at least in part due to the commonly used initialisation procedures. It should be noted that the correlation is not as strong for the interpolation between 1 and 0 which we attribute to the difficulty in estimating α\alpha reliably [6] (the reliability of the power-law assumption is explored in depth in § K, which shows inconclusive evidence for fitting a power law to the eigenvalues and when adjusting for under estimated tail samples we observe a strong monotonic increasing relationship (Figure 21). Notably, well performing models exist inside and outside of the good alpha range at the end of training (see Figure 6).

Trained model alpha versus initial model alpha

Figure 7. Trained model α\alpha vs initial model α\alpha (Left). Trained model α\alpha vs initial model α\alpha for interpolations λ=0.1\lambda= 0.1 to λ=1\lambda= 1 (Top Right) and λ=1\lambda= 1 to λ=100001\lambda= 100001 (Bottom Right).

Figure 21. Trained model α\alpha vs initial model α\alpha (Left). Trained model α\alpha vs initial model α\alpha for interpolations λ=0\lambda= 0 to λ=1\lambda= 1 (Top Right) and λ=1\lambda= 1 to λ=100001\lambda= 100001 (Bottom Right)

These results can also be interpreted as providing additional credence to the fact that SGD performs heavy-tail regularisation on the singular values throughout training, which can also explain why we observe α∈[2,6]\alpha\in[2, 6] in well-trained models. Thus, we propose that if the average alpha of well-trained models tends to lie in the ‘good’ range [2,6][2, 6] it is as a consequence of our standard initialisation procedures and regularisation from SGD, not because this metric is indicative of generalisation.

Generalisation With Fixed Alpha

To further highlight that that the average alpha is not causally related to generalisation, we train models with arbitrary fixed α\alpha values between 2≤α≤62 \leq\alpha\leq6, 10≤α≤1210 \leq\alpha\leq12, and 18≤α≤2018 \leq\alpha\leq20 using the SVD-based methodology with fixed-alpha spectra Σα\Sigma_{\alpha} described at the end of § 3. The results in Table 7 show that the models achieve no statistically meaningful difference in performance regardless of the fixed α\alpha. All models, regardless of fixed α\alpha, achieve similar performance. Therefore, α\alpha between 2 and 6 is not causally related or required for good performance. It should be noted that the model does not achieve the same performance as in § 4.1; however, this can be attributed to the fixed singular values Σα\Sigma_{\alpha}, which have a strong regularisation effect. Furthermore, this shows that despite models generally falling within good alpha values during training, this is not a prerequisite of generalisation and is a quirk of the current optimisation process afforded by SGD and similar variants. In § 3 we further this analysis by conducting a full ablation study of the interactions of the SVD factors during training and perform controlled interventions to understand the role the singular value spectra has on learning which further confirm the lack of causality for optimal α\alpha values and generalisation.

Fix Alpha RangeFC1FC2FC3FC4FC5FC6Test Accuracy
2≤α≤62 \leq\alpha\leq65.4799±0.13125.4799 \pm0.13125.0627±0.12095.0627 \pm0.12095.1361±0.18365.1361 \pm0.18364.9623±0.19574.9623 \pm0.19575.209±0.1945.209 \pm0.1945.4213±0.12245.4213 \pm0.122490.178±0.021490.178 \pm0.0214
10≤α≤1210 \leq\alpha\leq1210.7025±0.144210.7025 \pm0.144211.0621±0.210311.0621 \pm0.210310.8441±0.144310.8441 \pm0.144310.8218±0.128410.8218 \pm0.128411.0165±0.13811.0165 \pm0.13811.2001±0.202411.2001 \pm0.202490.223±0.023790.223 \pm0.0237
18≤α≤2018 \leq\alpha\leq2019.0752±0.124419.0752 \pm0.124417.4638±1.244817.4638 \pm1.244818.8759±0.188518.8759 \pm0.188518.7325±0.157318.7325 \pm0.157318.6775±0.202318.6775 \pm0.202318.8356±0.128918.8356 \pm0.128990.219±0.025390.219 \pm0.0253

Table 7. Resulting test performance of models trained with fixed Alpha (α\alpha) values. Mean and standard error of the mean derived from 10 trained models trained on MNIST.

Conclusion

We showed that RMT-based metrics cannot predict or explain model generalisation, nor the robust vs critical layer phenomenon, nor the re-initialisation vs re-randomisation phenomenon. We designed a range of experiments covering all aspects of the problem – individual layers § 4.1, 4.2 versus entire model § 4.3, 4.4, initialisation § 4.3, training § 4.3, 4.4, model performance § 4, eigenvalue power-law fitting § K – and a wide set of architectures, problems and scales – MNIST, ImageNet, LLMs – that all support our conclusion that

‘good model ⇏\not\Rightarrow good data-free metrics’ and ‘good data-free metrics ⇏\not\Rightarrow good model’

based on the fact that data-free metrics are affected in completely arbitrary ways by functionality-preserving reparametrisations of models (concretely given by the re-randomisations of robust layers). We have provided an alternative explanation for the fact that data-free metrics like α\alpha tend to fall in certain ranges for well-trained models.

Uncovering or disproving causality relations is a core scientific endeavour, often with concrete engineering consequences. Our results advocate for a reappraisal of the way that we approach interpretability of neural networks. Instead of using metrics with spurious predictive capacity, we argue that there is a need for an in-depth exploration of data-free methods that can reliably disambiguate between the robust and critical layer phenomenon, i.e. display some level of invariance under functionality-preserving reparametrisations. Additionally, we advocate for a focus on methods that consider more than just weight distributions of models and instead seek metrics which further understand the more nuanced interplay between between weights and data. We argue that data-free metrics fail because they do not capture the nuanced interplay between both the data passed through the model and the weights that process it. Concretely our recommendations are as follows: 1 Avoid the use of data-free that metrics as they only have spurious relations with model performance, 2 Ensure that novel data-free metrics are able to meaningfully capture performance differences offered by re-initialisation and re-randomisation, 3 In lieu of such data-free metrics, pursue the understanding of neural networks with metrics that consider the crucial interplay between weights and data and 4 Prioritise data-based approaches for interpretability based on their ability to distinguish between re-initialisation and re-randomisation.

Acknowledgments and Disclosure of Funding

1: Calculations were performed using the Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium. 2: This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT [18]. 3: Calculations were performed using the King’s College London HPC [19]. 4: This work was supported by UK Research and Innovation grant number EP/S023356/1, in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence3.

Limitations

In the main body of the paper we show across modalities, learning tasks and parameter scales that the alpha metric offered by Random Matrix Theory is not reparametrisation invariant under the re-randomisation and re-initialisation setting offered by critical layer theory. In § G, we extend our analysis to eight other data-free metrics across the same scales. While this is an extensive assessment of data-based metrics, we do not study all available data-free metrics. Given that data-free metrics are typically derived from similar matrix properties, one would expect that our findings should extend to all or most metrics built from neural network weights. However, caution would recommend that this be empirically confirmed for all data-free metrics. For the metrics we explore in the main body and in the appendix, we find no data-free metric that is reparametrisation invariant. In the paper we study two prominent modalities, vision and language, but do not explore data domains such as audio, as a result a natural extension of our experiments would be to confirm our findings in this data domain. From the results presented, we believe that the same findings would hold. In § H we show that data-based metrics are reparametrisation invariant, and thus we argue that they should be prioritised in future studies, but we only survey three data-based metrics (Activation Disagreement, Jenson-Shannon Divergence, and Linear CKA). As a result, future work could focus on which data-based metrics provide the strongest reparametrisation invariance - once again, we argue that this is more of an evaluative task and is outside of the scope of the core insights provided in this paper.

Pythia-1B Re-initialisation and Re-randomisation

Pythia-1B To conduct re-initialisation and re-randomisation experiments at large scale we use the Pythia-1B model [4] whose initialisations are publicly available. To gauge the impact of re-initialisation and re-randomisation on the model we use the same WikiText-103 dataset that was used in the main body for the GPT experiments. Table 8 presents the unmodified model accuracy on the dataset and the alpha fit of the layer weights within the training. Here, the model can be described as well-trained given that the mean α\alpha value is within the optimal range at the end of training.

ModelParamsLossAccMean α\alpha
Pythia1B2.57647.6804.801±0.202

Table 8. Model test performance on WikiText-103 and mean layer Alpha (α\alpha) value.

Table 9: The impact of re-initialisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of α\alpha and test accuracy.

Layer TypeAccuracyAlpha (α\alpha)% in Optimal α\alpha Rangeρ\rhoR2R^2RSMEK-τ\tau
Attention QKV42.187±0.89629.157±4.4290.0000.3880.1033.3950.283
Attention Out42.681±1.74321.945±2.7030.0000.1740.1196.5410.117
MLP 133.264±2.99625.337±3.8310.000-0.1060.04511.710-0.083
MLP 241.093±2.72928.064±4.7980.000-0.2470.5637.216-0.200

Table 9.

In Table 9 we present the impact of re-initialisation of a range of layers on the Pythia-1B model. For layers Attention QKV, Attention Out, MLP1 and MLP2 the accuracy is typically preserved under re-initialisation. Alongside this accuracy preservation, we record alpha values that are far outside of the optimal range. Moreover, when analysing the correlation between α\alpha values and accuracy, we see across correlation metrics that α\alpha has weak predictive capacity. These results agree with the existing analysis for the MNIST ReLU FCN 5x512 architecture explored in the main body.

When considering the case of re-randomisation Table 10 shows that accuracy is preserved less than in the case of re-randomisation. Similar to the results for re-initialisation we record alpha values that are far outside of the optimal range of alpha values. Given that for both re-randomisation and re-initialisation that we record alpha values that are outside of the optimal value range, yet there exists a meaningful performance impact between these different conditions highlights that alpha is not an effective metric for disambiguating these important differences between these reparametrisations, see Figure 8 for the global relationship between layer re-initialisation and re-randomisation test accuracy against α\alpha.

Layer TypeAccuracyAlpha (α\alpha)% in Optimal α\alpha Rangeρ\rhoR2R^2RSMEK-τ\tau
Attention QKV41.581 ±\pm 1.19333.577 ±\pm 6.2250.000-0.0590.0034.767-0.067
Attention Out42.437 ±\pm 1.81723.361 ±\pm 2.6510.000-0.6710.1486.710-0.450
MLP132.241 ±\pm 3.14029.251 ±\pm 4.4950.0000.1240.01012.4980.050
MLP240.954 ±\pm 2.72722.579 ±\pm 1.9610.000-0.2380.05910.58-0.150

Table 10. The impact of re-randomisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of α\alpha and test accuracy.

Figure 8. Pythia-1B layer re-initialisation (●) and re-randomisation (●) test accuracy vs α\alpha.

Overall, the results of re-initialisation and re-randomisation at the 1B parameter scale further highlights the inadequacies of α\alpha to meaningfully correspond to generalisation under functionally preserving reparametrisations.

Reparametrisation and Function Preservation

In this paper we suggest that preservation of model accuracy after layer reparametrisation (re-initialisation or re-randomisation) implies preservation of the function encoded by that model. In the main paper, we show that when comparing the original model to the re-parametrised model across a range of functional similarity and representational similarity metrics [20] accuracy is preserved when the predictive structure of the original model is also preserved.

Below we provide the corresponding functional/representational similarity analysis for the larger scale models presented in the paper. Overall, the correlation Table 11 and 12 show that functional/representational similarity holds a strong correlation with the change in accuracy observed. Furthermore, this provides strong evidence that when accuracy is preserved, the reparametrisations offered by re-randomisation and re-initialisation preserve the original function of the model.

(a) ResNet34

Metricρ\rhoR2R^2RMSEK-τ\tau
Prediction Similarity1.0000.9990.0050.989
Jensen-Shannon Distance-0.9760.9650.029-0.879
Procrustes Distance-0.7530.7220.121-0.659
Kullback-Leibler Divergence-0.9400.8570.851-0.814

Table 11. Correlation between the change in accuracy due to layer re-randomisation and the functional preservation of the model.

(b) ViT

Metricρ\rhoR2R^2RMSEK-τ\tau
Prediction Similarity0.9350.9950.0210.824
Jensen-Shannon Distance-0.8720.8660.087-0.752
Procrustes Distance-0.9230.8580.099-0.771
Kullback-Leibler Divergence-0.8770.9700.342-0.759

Table 12.

(c) GPT2

Metricρ\rhoR2R^2RMSEK-τ\tau
Prediction Similarity0.7470.9780.0300.579

Table 19.

(d) GPT2-Large

Metricρ\rhoR2R^2RMSEK-τ\tau
Prediction Similarity0.8420.9490.0300.655

Table 20.

Metricρ\rhoR2R^2RMSEK-τ\tau
Prediction Sim0.9910.9680.0360.933

Table 21. Table 12: Correlation between the change in accuracy due to re-initialisation and re-randomisation and the functional preservation of Pythia-1B.

Initialisation Scheme and Alpha (α)

In the main body of the paper, we use the default initialisation scheme offered PyTorch. In these results and Figure 2a in particular, we observe that there is a non-negligible probability of a randomly initialised layer having an α\alpha value in the ‘good’ range [2, 6]. In this section, we identify the probability of such ranges emerging at initialisation across a range of different layer types, namely: Linear Layer 512x512, Resnet34 Layer2 Conv_2 (Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) and ViT Base 16 Out Projection.

Additionally, as many different initialisation schemes are offered by PyTorch such as Xavier (normal (Xn\mathcal{X}_n), uniform (Xu\mathcal{X}_u)) [13] and Kamming (normal (Kn\mathcal{K}_n), uniform (Ku\mathcal{K}_u))[16] we have reproduced the analysis in each of the following tables to also capture these initialisations. The following tables present the descriptive statistics of each initialisation type and their relationships with alpha for the layers named above.

Initialisation Schemexˉ\bar{x}x~\tilde{x}σ\sigmaSkewKurtosisMinMax% in Optimal α\alpha Range
PyTorch default10.9859.0196.3902.0885.9923.50056.64016.640
Xu\mathcal{X}_u11.1128.9936.5872.0175.3603.39864.35916.910
Xn\mathcal{X}_n10.9609.0016.4482.0835.9383.44663.33217.510
Ku\mathcal{K}_u10.8778.9486.3402.1916.9503.35368.18016.700
Kn\mathcal{K}_n10.9939.0406.5242.2167.0383.40368.13517.120

Table 22. Table 13: Distribution statistics of α\alpha across initialisation schemes and layer types. (a) Linear Layer 512 × 512.

Initialisation Schemexˉ\bar{x}x~\tilde{x}σ\sigmaSkewKurtosisMinMax% in Optimal α\alpha Range
PyTorch default18.12613.51814.3022.83514.4553.594211.3875.990
Xu\mathcal{X}_u18.34113.94014.1462.74814.2293.494191.4226.210
Xn\mathcal{X}_n17.72913.54013.7473.58532.0473.405288.7636.510
Ku\mathcal{K}_u18.18613.79113.9862.79015.7793.367221.8066.230
Kn\mathcal{K}_n17.91913.75013.2712.3909.2033.583152.9026.350

Table 23. (b) ResNet34 Layer2 Conv_2 (Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)).

(c) ViT Base 16 Out Projection (Linear Layer 768 × 768).

Initialisation Schemexˉ\bar{x}x~\tilde{x}σ\sigmaSkewKurtosisMinMax% in Optimal α\alpha Range
PyTorch default13.00210.5877.9532.1836.5243.87376.6668.020
Xu\mathcal{X}_u13.19710.6678.1432.1616.9743.836101.3108.310
Xn\mathcal{X}_n12.97710.5317.9692.2006.8443.99074.7928.190
Ku\mathcal{K}_u13.18310.6648.0712.0995.9174.00168.7437.840
Kn\mathcal{K}_n13.18510.6188.2042.1916.6673.85788.8487.740

Table 13.

Table 13 presents summary statistics for α\alpha across different initialisation schemes for different layer types of models. These results complement that of those presented in Figure 2a by showing that initialised layers can fall within optimal value range of α\alpha with a somewhat large non-zero probability. Given that HTSR argues that such α\alpha values should only emerge at the end of training and quantify a good layer, this evidence further undermines the ability of α\alpha to effectively capture generalisation in neural networks.

Alpha (α\alpha) Threshold-Prediction Tests

The α\alpha metric can be considered a threshold based classification metric, where 2≤α≤62 \le\alpha\le6 denotes a ‘good’ layer or model with high accuracy and α<2\alpha< 2 or α>6\alpha> 6 is considered a ‘bad’ layer or model. With this perspective we analyse the performance of α\alpha of predicting a good layer, after reparametrisation offered by re-reinitialisation and re-randomisation. A ‘good’ layer is considered to be a layer that results in the model being within 10% of the original test accuracy after the reparametrisation and has an α\alpha within 2≤α≤62 \le\alpha\le6. In Table 14 we report the global (for all layers) AUC and the F1-score, Precision, True Positive Rate (TPR), False Positive Rate (FPR) of α\alpha predicting a good layer; to clarify, a good layer corresponds to the resulting model being within 10% of the original test accuracy after the reparametrisation. For the Pythia 1B model, we are not able to provide the AUC score nor the true positive rate score as none of the reparametrisations resulted in layers within the 2 and 6 alpha range.

ModelAUCF1-ScorePrecisionTrue Positive RateFalse Positive Rate
MLP0.5200.2420.1670.4390.387
ResNet340.4490.0000.0000.0000.020
ViT0.6860.0560.0291.0000.836
GPT20.4420.0220.0110.7140.901
GPT2-Large0.6030.0100.0050.8750.973
Pythia-1Bnan0.0000.000nan0.797

Table 14. Effectiveness of α\alpha at predicting a good layer 2≤α≤62 \le\alpha\le6 where model performance is within 10% of the original test accuracy after layer reparametrisation.

For all models evaluated, α\alpha achieves a poor AUC. For the ResNet34 and GPT2, this is worse than random; the MLP is slightly better than random, and for the GPT-Large and ViT it is better than random; however, still poor. Furthermore, the F1-Score and Precision are very low, with the highest F1-score and Precision of 0.242 and 0.167, respectively, in the MLP case. With regard to the true positive rate (TPR) and false positive rate (FPR) of the optimal α\alpha range, a high TPR corresponds with a high FPR, showing the metric is poor at effectively identifying good layers. In the case of ResNet, the α\alpha metric fails to identify any of the layers that correspond to a model maintaining performance, whilst flagging a small number that do not maintain performance as good layers. Overall, these results strongly corroborate the main findings in the paper.

Alpha (α\alpha) Analysis Stratified Over Layer Type

Table 15, presents the stratified analysis of the model layers by type, for the ResNet34 and ViT on ImageNet and the GPT2 and GPT2-Large on WikiText103. In the tables there is no consistent trend across the layer types, with some layers reporting a positive relationship between test accuracy and α\alpha and others a negative relationship. Which strongly supports the main findings in the paper that α\alpha values are not a reliable indicator of performance and generalisation.

Layer TypeAccuracyAlpha (α\alpha)% in Optimal α\alpha Rangeρ\rhoR2R^2RMSEK-τ\tau
Input Conv2d0.395 ± 0.02914.381 ± 1.8170.0000.1760.0080.0910.156
Conv2d_12.461 ± 0.26022.628 ± 1.5253.750-0.1780.0333.232-0.117
Conv2d_236.35 ± 1.61023.882 ± 1.6373.750-0.1400.05219.823-0.096
Downsample15.793 ± 2.4746.788 ± 0.60443.3330.6340.33511.0520.456
Output FC0.098 ± 0.00911.829 ± 2.0950.0000.4920.0360.0290.405

Table 15. Stratified layer re-randomisation Test Accuracy vs α\alpha. (a) ResNet34 on ImageNet

Layer TypeAccuracyAlpha (α\alpha)% in Optimal α\alpha Rangeρ\rhoR2R^2RMSEK-τ\tau
patch_embed.proj0.145 ± 0.00865.281 ± 4.6070.0000.4670.2410.0210.422
attn.qkv77.808 ± 0.07117.978 ± 1.2181.6670.0770.0190.7690.058
attn.proj77.469 ± 0.22313.345 ± 0.7256.667-0.0750.0042.442-0.049
mlp.fc152.805 ± 2.80616.678 ± 0.9110.0000.2390.03230.2380.162
mlp.fc271.847 ± 1.90220.088 ± 1.1571.667-0.0550.00120.824-0.036
head0.128 ± 0.01314.194 ± 3.3290.000-0.2190.0140.042-0.135

Table 15b. (b) ViT on ImageNet

Layer TypeAccuracyAlpha (α\alpha)% in Optimal α\alpha Rangeρ\rhoR2R^2RMSEK-τ\tau
wpe7.336 ± 0.03015.272 ± 2.7300.000-0.1270.1480.087-0.067
attn.c_attn36.472 ± 0.35516.644 ± 0.9260.000-0.0490.0023.887-0.036
attn.c_proj35.382 ± 0.73911.705 ± 0.5285.8330.0770.0038.0800.058
mlp.c_fc34.874 ± 0.84518.075 ± 1.0390.000-0.1200.0039.237-0.079
mlp.c_proj35.430 ± 0.77718.240 ± 0.8590.0000.1460.0008.5120.098

Table 15c. (c) GPT2 on WikiText103

Layer TypeAccuracyAlpha (α\alpha)% in Optimal α\alpha Rangeρ\rhoR2R^2RMSEK-τ\tau
wpe8.905 ± 0.07212.303 ± 1.62510.000-0.8180.7340.117-0.644
attn.c_attn43.581 ± 0.01222.934 ± 0.8380.000-0.0050.0070.222-0.002
attn.c_proj43.041 ± 0.20316.266 ± 0.5561.9440.0020.0003.8560.003
mlp.c_fc41.817 ± 0.35524.427 ± 0.8880.0000.0590.0056.7240.038
mlp.c_proj42.413 ± 0.34825.964 ± 0.9990.000-0.0580.0006.599-0.039

Table 15d. (d) GPT2-Large on WikiText103

Further Analysis on Data-Free Metrics

In this section we extend our analysis to the following data-free metrics under the existing experimental setup. WW represents the weight matrix of the layer and XX is the empirical spectral density of the correlation matrix, X=WTWX = W^{T}W, such that pemp(λ)∼λ−αp_{\mathrm{emp}}(\lambda) \sim\lambda^{-\alpha}, where λ\lambda are the eigenvalues of XX.

Alpha Weighted (α^\hat{\alpha}): αlog⁡(λmax)\alpha\log(\lambda_{max}), where λmax\lambda_{max} is the max eigenvalue from XX [30].

Log Alpha Norm: log⁡(∥X∥αα)\log(\lVert X\rVert_{\alpha}^{\alpha}), where ∥X∥αα=∑iMλiα\lVert X\rVert_{\alpha}^{\alpha} = \sum_{i}^{M} \lambda_{i}^{\alpha}, where MM is the rank of WW [30].

MP Soft Rank: is the ratio between the bulk edge of the pemp(λ)p_{\mathrm{emp}}(\lambda), λ+\lambda^{+}, and the max eigenvalue, λmax\lambda_{max}, λ+λmax\frac{\lambda^{+}}{\lambda_{max}} [30].

Spectral Norm: The max singular value of WW denoted as ∥W∥∞\lVert W\rVert_{\infty}.

Stable Rank: The ratio of the squared Frobenius Norm and the squared Spectral Norm, denoted as ∥W∥F2∥W∥22\frac{\lVert W\rVert_{F}^{2}}{\lVert W\rVert_{2}^{2}} [37].

Number of Power Law Spikes: The number of eigenvalues that are outside the bulk region of the ESD, when fit to an Marchenko-Pastur distribution [30].

Entropy: −1log⁡(M)∑ipilog⁡pi\frac{-1}{\log(M)}\sum_{i}p_{i}\log p_{i}, where MM is the rank of matrix WW and pip_{i} is σi2∑i(σi2)\frac{\sigma_{i}^{2}}{\sum_{i}(\sigma_{i}^{2})} where σ\sigma is the singular values of WW [30].

Frobenius Norm: The sum of the squared singular values of WW denoted as ∥W∥F\lVert W\rVert_{F}.

Each metric can be found below with the appropriate subsection that corresponds to our analysis of these data-free metrics.

The correlation between the other data-free metrics formalised in Section G and the associated test accuracy when layers undergo re-initialisation (🔵) or re-randomisation (🟠) is shown in Figure 9. For all metrics explored we observe approximately zero correlation between the metric values and the test accuracy, showing that other data-free metrics align with the findings presented for the alpha metric presented in the main body of the paper. In § G.2,§ G.3 we show this holds for ImageNet models as well as for language models in § G.4 and G.5.

Small Scale Analysis of Data-Free Metrics Under Re-Initialisation and Re-Randomisation

Norm-based metrics were originally shown to be too coarse a metric to measure the generalisability of the neural networks in [48]. Figure 9 strengthens these findings, highlighting that there is essentially no correlation between the a range of data-free metrics and the test accuracy of a model under re-randomisation and re-initialisation.

ResNet34 on ImageNet

In Figure 10, when using a range of data-free metrics we observe the same trend as found with α\alpha in the main body of the paper. Overall for the ResNet43 on ImageNet data-free metrics fail to predict the performance difference that is derived from re-randomisation across layers. While some correlations are higher than circa 0, i.e. (Spectral Norm) it can be clearly observed that there is strong overlap between metric values for the layers that have good accuracy and poor accuracy through re-randomisation, showing that the artifact of slightly higher correlation shows no meaningful insight for these metrics.

Figure 10. ResNet 34: Layer re-randomisation test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RSMERSME is the root mean square error of the linear regression (red line), and K-τ\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

ViT on ImageNet

To confirm trends for the ImageNet dataset on the ViT Figure 11 shows the relationship of data-free metrics value distributions under re-randomisations. Once, again in this figure we find that for all data-free metrics that models can generalise and perform poorly while expressing the same metric values under re-randomisation. As a result the values have no meaningful prediction of performance. This trend is in line with the observations we found with α\alpha in the main body of the paper. Here, we do find weak correlations for MP soft rank, spectral norm, stable rank, entropy, number of power law spikes and Frobenius, however, there is significant overlap between the metric values and associated test accuracy (i.e. for strong accuracy and random accuracy). Furthermore for all metrics the R2R^2 value is close to zero indicating that the regression model has little to no predictive power. Overall, data-free metrics can be considered a weak predictor of model performance.

Figure 11. ViT: Layer re-randomisation test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RMSERMSE is the root mean square error of the linear regression (red line), and KK-τ\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

GPT2 on WikiText103

To understand if there is generality in our findings of data-based metrics across modalities and parameters scales we observe the trends of performance and data-free metrics under re-randomisation for the GPT2 model. In Figure 12 we largely find a consistent trend that values for data-free metrics can simultaneously represent high and low performing models. This shows that analysing particular values for these data-free metrics provides little to no insight on performance. Here for all metrics there is a very weak correlation between metric values and performance. These trends align with those found with α\alpha in the main body of the paper.

Figure 12. GPT2: Layer re-randomisation test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RMSERMSE is the root mean square error of the linear regression (red line), and K-τ\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

GPT2-Large on WikiText103

In Figure 13, we explore the relationship of data-free metrics and accuracy under re-randomisation for the GPT-2 Large model. Here the trends mirror those for the GPT2 and the results with alpha presented in the main body. For the data-free metrics surveyed there is a moderate to weak correlation between the metric and the test accuracy of the model. However, in spite of this, there is no predictive capacity of the metrics when described with linear regression as shown with the R2R^2 close to zero. This can be attribute to overlapping values that correspond to high and low test accuracy. For the data-free metrics surveyed in this paper across parameter ranges in the GPT2 family we find no predictive capacity for the data-free metrics under re-randomisation.

Figure 13. GPT2-Large: Layer re-randomisation test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RSMERSME is the root mean square error of the linear regression (red line), and K−τK-\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

Pythia-1B on WikiText103

In Figure 14, we explore the relationship of data-free metrics and accuracy under re-initialisation (•) and re-randomisation (•) for the Pythia-1B model. Here the trends mirror those for the GPT2 models and the results with alpha presented in the main body. For the data-free metrics surveyed there is a moderate to weak correlation between the metric and the test accuracy of the model. However, in spite of this, there is no predictive capacity of the metrics when described with linear regression as shown with the R2R^2 close to zero. Which is due to overlapping values that correspond to high and low test accuracy. For the data-free metrics surveyed we find no predicative capacity for said metrics under re-initialisation and re-randomisation.

Figure 14. Pythia-1B: Layer re-initialisation (•) and re-randomisation (•) test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RMSERMSE is the root mean square error of the linear regression (red line), and KK-τ\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

Data-Based Metrics

To compare data-free and data-based metrics we explore if two widely used data-based metrics can distinguish the output of a re-initialised layer from that of a re-randomised layer. The outputs of the original and modified layers are represented as OlO_l and MlM_l, respectively. The model layer outputs are collected by passing through the test dataset, DtestD_{test}. The metrics explored are defined as follows, Linear CKA [21] is explored in Appendix I):

  • Activation Disagreement: The mean percentage of times that the neurons from OlO_l and MlM_l fail to agree to activate on DtestD_{test}.

  • Jensen–Shannon (JS) Divergence: The mean weighted average of the Kullback–Leibler (KL) Divergence of the softmax(OlO_l on DtestD_{test}) compared to the softmax(MlM_l on DtestD_{test}) [25].

When considering the results of the activation disagreement and JS divergence between the original model layer and each model with re-initialised and re-randomised layers in Figure 15, it becomes clear that there is a stark difference in similarity between the re-initialised and re-randomised layers compared to the unmodified layer. The figure demonstrates that the difference between re-initialisation and re-randomisation can be explained by the amount the layer disagrees on activation compared to the original non-modified layer. It is evident that when there is less disagreement (re-initialisation), the model can maintain accuracy, while a lot of disagreement (re-randomisation) leads to a drop in accuracy. This observation is further supported by a mean Spearman correlation coefficient and Kendall’s tau measure across layers of -0.776 and -0.5375, respectively for Activation Disagreement, indicating a negative relationship between activation disagreement and accuracy.

Figure 15. Layer re-initialisation (●), re-randomisation (●) and signs of re-initialisation with magnitudes of re-randomisation (●) test accuracy vs the respective metric.

Connection to the Lottery Ticket Hypothesis: For our MNIST experiments, both re-initialisation and re-randomisation are drawn from the Uniform distribution, U(−k,k)\mathcal{U}(-\sqrt{k},\sqrt{k}) where kk is 1features\frac{1}{\text{features}}. However, a layer can reduce to random accuracy under re-randomisation but not under re-initialisation, which suggests that an important factor for performance retention under re-initialisation is the signs of the weights. The signs of the weights, especially within networks that use the ReLU activation, essentially represent the pathway, or subnetworks within the model. The Lottery Ticket Hypothesis (LTH) suggests that at random initialisation, there exist subnetworks that, when trained, reaches the test accuracy of the original model that is trained with all the parameters and same compute budget [12]. To explore the role of the signs, we take the re-randomised values for a layer and apply the signs of the re-initialised layer to the re-randomised weights. We explore this using the data-based metrics that can disambiguate between re-initialised and re-randomised. We find in Figure 15 that when applying the signs of the re-initialisation to the magnitudes of the re-randomised layer (●), this layer is closer to the original layer as observed with a reduction in difference, and that there is an increase in performance when compared to the re-randomised layer (●). The results strongly suggest that for this network, the subnetworks defined at the start of training through initialisation do not dramatically change through training, providing credence to the LTH. Furthermore, this demonstrates the efficacy of data-based metrics to disambiguate between the subnetwork importance of layers in neural networks.

Linear CKA (Data-Based Metric) Analysis

In this section, we explore an additional data based metric, Linear CKA [21]. We asses the predictive power of Linear CKA to distinguish between the output of a re-initialised layer from that of a re-randomised layer. Overall we find that in line with other data-based metrics explored in the main paper that Linear CKA can disambiguate between a re-initialised and re-randomised layer. However, Linear CKA has less predictive power than the Activation Disagreement and JS-divergence metrics explored in the body of the paper, as demonstrated with lower correlation values and higher RMSE, see Figure 16. The results highlight an important caveat when considering data-based metrics, not all data-based metrics can be considered equal for predictive capacity. As a result, it remains an open challenge to find test beds for identifying and utilising the correct data-based metrics that effectively consider subnetwork relationships and disambiguate between robust and critical layers.

Figure 16. Layer re-initialisation (●) and re-randomisation (●) test accuracy vs the respective metric. ρ\rho is the Spearman correlation coefficient, RMSERMSE is the root mean square error of the linear regression (red line), and K-τ\tau is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.

SVD Ablation for Learning Dynamics

The empirical experiments in this paper have provided substantial evidence that data-free metrics may bear no relationship to the generalisation properties of neural networks. However, it can be argued that the spectral properties of weight matrices need to to change in order for a model to generalise, particularly since singular values do change substantially in standard training setups. In Figure 17 we show how the singular values in an MNIST model change across epochs.

Figure 17. The first 10 singular values for each layer of a 5 hidden layer neural network with width 512 throughout training on MNIST. The viridis colormap is used to delineate epochs. Dark-purple represents initialisation and the last epoch, epoch 30, is in bright-yellow. The mean (line) is reported with the standard error of the mean (error bars) from 10 runs.

Given, that the singular values values do change substantially through training, a plausible interpretation of these metrics is that they record properties of the neural network that are important in the training process that results in a well generalising model.

Motivated by this interpretation, we conduct experiments to establish the role of the spectral properties of the weight matrices in well generalising models. The following experiments explore how the factors in the SVD decomposition W=UΣVTW = U\Sigma V^{T} of weight matrices (i.e. the singular vectors and the singular values) affect training. We explore each factor independently by keeping the others fixed during training, and only letting the corresponding factor update throughout training - to isolate a single factors effect. This enables a systematic intervention analysis to understand what factors primarily drive generalisation and which ones require interactions. The factors are also explored jointly to understand the role of the interaction of factors on generalisation. Finally, the factors are all trained together to create an effective baseline of complete interaction.

If the model is unable to generalise without the singular values – and thus spectral properties – changing, this would support the interpretation that these metrics record property changes that are required for models to reach a minima that generalises well. Conversely, if a model can generalise without these properties changing, this would mean that these metrics are not casually relevant to finding a minima that generalises and thus has limited impact on generalisation in training.

Experimental Setup

To reduce confounding factors, we consider a linear layer where the weight matrices are parametrised via singular value decomposition, W=UΣVTW = U\Sigma V^{T}, to from three layers where UU and VTV^{T} are orthogonal, and Σ\Sigma is diagonal with non-negative entries.

This parametrization allows for the orthogonal direction factors UU and VTV^{T} and the scaling factor Σ\Sigma of the weight matrix to be manipulated and changed independently. Thereby enabling a controlled analysis of their respective roles in the optimization process and generalisation.

For all experiments we use a 5 hidden layer MLP without biases with the ReLU activation and a hidden width of 512 on the MNIST dataset with cross entropy loss, SGD with a learning rate of 0.001 and momentum of 0.9, a batch size of 128 and trained for 30 epochs. To isolate the effect of parameter interventions, the training data order, random seeds and initialisation procedures (expect where explicitly varied) are held fixed across all runs. Model biases are not such that the factors are the only aspect impacting model performance.

To ensure that UU and VTV^{T} maintained orthogonality throughout training we employed the standard PyTorch methods of matrix exponential for square matrices and the householder transformation for rectangular matrices4. This ensures that changes in the performance cannot be attributed to the loss of orthogonality.

Using random Σ\Sigma drawn from the Marchenko–Pastur distribution introduces implicit conditioning on the orthogonal directions of UU and VTV^{T}. Therefore Σ\Sigma is initialised as Σ=I\Sigma= I, where II is the identity matrix. This removes the scaling effects at initialisation and ensures that all singular values are treated equally by the optimiser. As a result, any observed differences in training dynamics can be attributed to the orthogonal directions of UU and VTV^{T} or to the values of Σ\Sigma rather than the co-founding effects that could arise from the initial unequal spectral scaling. Furthermore initialising with Σ=I\Sigma= I fixes the spectral properties across runs, facilitating a controlled comparison of the effects of the spectral properties between interventions.

Interventional Ablation Over Parametrisation Subsets

To study the interactions of the SVD factors during training, we perform controlled interventions, in which gradient updates are enabled on specific subsets of parametrisations, while the remaining parametrisations are held fixed with the initial values - allowing for a controlled and isolated optimisation setup.

For each experiment, a subset T⊂{U,Σ,VT}\mathcal{T} \subset\{U, \Sigma, V^{T}\} is assigned as a trainable factor. All factors in the complement {U Σ VT}∖T\{U\ \Sigma\ V^{T}\} \setminus\mathcal{T} are frozen and do not change throughout training.

We consider the following intervention sets:

  • Single-Factor Training T∈{{U},{Σ},{VT}}\mathcal{T} \in\{\{U\}, \{\Sigma\}, \{V^{T}\}\}

  • Two-Factor Training T∈{{UΣ},{ΣVT},{UVT}}\mathcal{T} \in\{\{U\Sigma\}, \{\Sigma V^{T}\}, \{UV^{T}\}\}

  • Full-Factor Training T∈{{UΣVT}}\mathcal{T} \in\{\{U\Sigma V^{T}\}\}

Single Factor Training: The single factor training experiments (U ΣU\ \Sigma and VTV^{T} interventions) allow for understanding of the role of the of each component in the learning process. The performance in these conditions reveals whether training progress can be achieved through spectral scaling (Σ\Sigma) or through rotations of the singular vectors (UU and VTV^{T}) alone. Failure of a factor to effectively generalise indicates that learning relies on the interaction between that factor and others. Whereas the success of a factor implies that the corresponding factor is a primary driver of the learning dynamics.

UU Intervention: To isolate the effects of the UU orthogonal directions during training, Σ\Sigma is fixed as Σ=I\Sigma= I and VTV^{T} is fixed as V0TV_{0}^{T}, where V0TV_{0}^{T} is orthogonal matrix, throughout training. For each run UU is initialised with a random orthogonal matrix and allowed to train. Keeping VTV^{T} fixed as V0TV_{0}^{T} throughout runs isolates the effects of variation in performance ensuring it cannot be attributed to changes in the initial conditions of Σ\Sigma and VTV^{T}.

Σ\Sigma Intervention: To isolate the effects of the Σ\Sigma during training, Σ\Sigma is initialised as Σ=I\Sigma= I the orthogonal matrices UU and VTV^{T} are randomly initialised as orthogonal matrices and then held fixed throughout training. Between runs the initial values of UU and VTV^{T} change to ensure that the results are not conditional on a single orthogonal direction.

VTV^{T} Intervention: To isolate the effects of the VTV^{T} orthogonal directions during training, Σ\Sigma is fixed as Σ=I\Sigma= I and UU is fixed as U0U_{0}, where U0U_{0} is orthogonal matrix, throughout training. For each run VTV^{T} is initialised with a random orthogonal matrix and allowed to train. Keeping UU fixed as U0U_{0} throughout runs isolates the effects of run variation in performance ensuring it cannot be attributed to changes in the initial conditions of Σ\Sigma and UU.

Two Factor Training: The two factor training experiments (Joint UΣU\Sigma, ΣVT\Sigma V^{T} and UVTUV^{T} interventions) allow for understanding the interactions between components and how this affects the training process and subsequent generalisation properties. Improvements in performance above the single factor cases, suggests that the interaction of the two factors is important to overall performance. Although if performance is comparable to full training this would suggests that the frozen factor does not significantly impede the training process.

Joint UΣU\Sigma Intervention: To isolate the extent to which performance improvements arise from the from the combined changes in Σ\Sigma and UU. The UΣU\Sigma parametrisation jointly optimises UU and Σ\Sigma while keeping VTV^{T} fixed as V0TV_{0}^{T} throughout training.

Joint ΣVT\Sigma V^{T} Intervention: To isolate the extent to which performance improvements arise from the from the combined changes in Σ\Sigma and VTV^{T}. The ΣVT\Sigma V^{T} parametrisation jointly optimises Σ\Sigma and VTV^{T} while keeping UU fixed as U0U_{0} throughout training.

Joint UVTUV^{T} Intervention: To isolate the extent to which performance improvements arise from the from the combined changes in UU and VTV^{T}. The UVTUV^{T} parametrisation jointly optimises UU and VTV^{T} while keeping Σ\Sigma fixed throughout training.

Full Factor Training UΣVTU\Sigma V^{T}: In the full factor training UΣVTU\Sigma V^{T} all components (UU, Σ\Sigma and VTV^{T}) are jointly optimised. This corresponds to standard training on the weight matrix WW, and thus serves as a reference for the impact of the single and two-factor interventions on the performance.

Table 16 summaries all the interventional conditions explored in this experiment. For this experiment instead of using 100 runs, 10 runs are used as the orthogonal constraint on the UU and VTV^{T} components significantly increased training times. The test accuracy, loss and metrics are recorded at initialisation and throughout training at the end of every epoch.

Trainable UUTrainable Σ\SigmaTrainable VTV^{T}Description
✓\checkmarkx\mathsf{x}x\mathsf{x}Output Alignment Only
x\mathsf{x}✓\checkmarkx\mathsf{x}Spectral Scaling Only
x\mathsf{x}x\mathsf{x}✓\checkmarkInput Alignment Only
✓\checkmark✓\checkmarkx\mathsf{x}Joint UU and Σ\Sigma (UΣU\Sigma)
x\mathsf{x}✓\checkmark✓\checkmarkJoint Σ\Sigma and VTV^{T} (ΣVT\Sigma V^{T})
✓\checkmarkx\mathsf{x}✓\checkmarkJoint UU and VTV^{T} (UVTUV^{T})
✓\checkmark✓\checkmark✓\checkmarkStandard Training (W=UΣVTW = U\Sigma V^{T})

Table 16. Summary of factor interventional conditions.

Experimental Results

The following experimental results cover the relationship between the factors and test performance (§ J.2.1) and data-free metrics (§ J.2.2) throughout training.

Factor Relationship with Model Performance

Figure 18 compares the performance throughout training when different subsets of the models parameters are updated during training. Training all components jointly, UΣVTU\Sigma V^{T}, as is done conventionally through optimisation directly on the weight matrix WW, results in the model with the highest test accuracy at the end of training of 97.499%.

Figure 18. Model test performance throughout training on MNIST. (left) accuracy and (right) loss. The dotted line (. . . ) is the train performance and the solid line (–) is the test performance, the ◦ represents a epoch. Each curve corresponds to training only one or more components of the parametrisation: UU, Σ\Sigma, VTV^{T}, UΣU\Sigma, ΣVT\Sigma V^{T}, UVTUV^{T} and UΣVTU\Sigma V^{T}. All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (hue) are reported from 10 runs.

Training only a single component (U ΣU\ \Sigma or VTV^{T}) results in consistently lower performance than UΣVTU\Sigma V^{T}. The UU parametrisation reaches a high initial test accuracy after 1 epoch training (83.811%) and then continues to improve in performance achieving a test accuracy of 93.236% at the end of training. In contrast, updating only Σ\Sigma results in a markedly slower initial improvement with slow but steady performance gains, yet remains significantly below the performance levels achieved by all other parametrisations throughout training resulting in a test accuracy of 37.246%. As found with only updating UU updating only VTV^{T} reaches a high initial test accuracy after 1 epoch training (88.628%) a 4.817% increase in test accuracy over UU parametrisation. The VTV^{T} parametrisation then steadily increases in performance reaching a test accuracy of 94.924% a 1.688% improvement over the UU parametrisation.

Jointly updating pairs of components (UΣU\Sigma, ΣVT\Sigma V^{T} and UVTUV^{T}) improves the resulting test performance relative to the single-components (U ΣU\ \Sigma and VTV^{T}). Both UΣU\Sigma and ΣVT\Sigma V^{T} parametrisations follow the similar initial training trajectories as UU and VTV^{T} respectively. After 1 epoch of training the UΣU\Sigma parametrisation achieves a test accuracy of 85.506% a 1.695% improvement over the UU parametrisation. The UΣU\Sigma parametrisation subsequently makes increased performance gains throughout training resulting in a test accuracy of 96.458%, only 1.041% of training with all components jointly (UΣVTU\Sigma V^{T}). Additionally after 1 epoch training the ΣVT\Sigma V^{T} parametrisation achieves a test accuracy of 88.898% a 0.270% improvement over the VTV^{T} parametrisation and 1.168% below UΣVTU\Sigma V^{T} parametrisation. The ΣVT\Sigma V^{T} parametrisation subsequently makes increased performance gains throughout training and closely matches the training trajectory of UΣVTU\Sigma V^{T}, resulting in a test accuracy of 97.373%, only 0.126% of training with all components jointly (UΣVTU\Sigma V^{T}). The UVTUV^{T} parametrisation, closely matches the training trajectory of VTV^{T} with the initial epoch of training reaching a test performance of 89.244% a 0.616% improvement over the VTV^{T} parametrisation and a 5.433% improvement over the UU parametrisation. The UVTUV^{T} parametrisation continues to closely match the VTV^{T} parametrisation with a resulting test performance of 95.524% a 0.600% improvement over the VTV^{T} parametrisation.

In summary, Figure 18, shows that the configurations exhibit a consistent ordering in test performance with UΣVTU\Sigma V^{T} performing the best, followed by ΣVT\Sigma V^{T}, UΣU\Sigma, UVTUV^{T}, VTV^{T} UU and Σ\Sigma. With the difference between UU and Σ\Sigma of 55.990% compared to a difference between UΣVTU\Sigma V^{T} and UU of 4.263%. Although Σ\Sigma was unable to achieve comparable or close performance to UΣVTU\Sigma V^{T} the inclusion of Σ\Sigma to UU and VTV^{T} parametrisations in UΣU\Sigma and ΣVT\Sigma V^{T} enabled improvements in test performance over UU and VTV^{T} of 3.222% and 2.449% respectively. The next section explores the relationship of data-free metrics throughout training to better understand how our intervention setup effects matrix (and data-free metric) properties during training and their relationship to accuracy.

Metric Relationship with Test Accuracy

Model test performance throughout training on MNIST against data-free metrics

Figure 19. Model test performance throughout training on MNIST against data-free metrics. (left) test accuracy and (right) test loss. The ◦ represents a epoch. The colour intensity displays training progress, lighter points correspond to earlier epochs with darker points corresponding to later epochs. Each curve corresponds to training only one or more components of the parametrisation: UU, Σ\Sigma, VTV^{T}, UΣU\Sigma, ΣVT\Sigma V^{T}, UVTUV^{T} and UΣVTU\Sigma V^{T}. All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (error bars) are reported from 10 runs.

To understand the role of each respective metric in the training process we explore the each data-free metric’s relationship with the test accuracy. Specifically focusing on how training with different model components effects the relationship each metric has with test accuracy throughout training.

Alpha (α\alpha): fig:19a compares the performance throughout training with the global Alpha α\alpha when different subsets of the models parameters are updated during training. For parametrisations UU, VTV^{T} and UVTUV^{T} the α\alpha metric stays approximately around 10510^{5}. This behaviour is expected as the singular values are constrained to 1. The slight deviation from 10510^{5} is introduction by the imperfect orthogonal constraints applied throughout training which results in slight negligible variations in Σ\Sigma when the UΣVTU\Sigma V^{T} is combined to calculate the metric.

The parametrisations that utilise Σ\Sigma exhibit different behaviour to UU, VTV^{T} and UVTUV^{T}. The UΣVTU\Sigma V^{T} parametrisation achieves an α\alpha value of 249.449 after the 1 epoch training, the α\alpha value continues to decrease throughout training while steadily improving in test performance, reaching a global α\alpha value of 34.485 which a corresponding test accuracy of 97.499%. The Σ\Sigma parametrisation reaches an α\alpha value of 259.177 within the first epoch of training correspond to a test accuracy of 10.542%. The Σ\Sigma parametrisation α\alpha value subsequently decreases, throughout training while improving in test performance reaching a value of 10.168 with a corresponding test accuracy of 37.246%. The introduction of Σ\Sigma to UU and VTV^{T} with UΣU\Sigma and ΣVT\Sigma V^{T} result in the global α\alpha values quickly deviating from the initial value after 1 epoch of training reaching an α\alpha value of 157.390 and 259.177 and test accuracies of 85.506% and 88.898% respectively. The α\alpha values continue to decrease throughout training reaching 26.220 and 33.338 while coinciding with a test accuracy of 96.458% and 97.373% for the UΣU\Sigma and ΣVT\Sigma V^{T} parametrisations respectively.

Overall, the Figure shows that optimal α\alpha metric values do not correlate with the resulting test performance of the model, as demonstrated with UU, VTV^{T} and UVTUV^{T} parametrisations achieving high accuracy while maintaining a large α\alpha value of 10510^5 and the Σ\Sigma parametrisation achieving a low α\alpha metric of 10.168 with a test accuracy of 37.246%.

Alpha Weighted (α^\hat{\alpha}): fig:19b compares the performance throughout training with the global Alpha Weighted α^\hat{\alpha} when different subsets of the models parameters are updated during training. The Figure largely shows the same relationship between the parametrisations U Σ VTU\ \Sigma\ V^T, ΣVT\Sigma V^T, UVTUV^T and UΣVTU\Sigma V^T and the α\alpha metric observed in fig:19a. With the UU, VTV^T and UVTUV^T parametrisations α^\hat{\alpha} value staying relatively fixed throughout training.

The Σ\Sigma parametrisation achieves an α^\hat{\alpha} value of 2.359 after 1 epoch training, which then slowly increases as training progresses to 2.854 at the end of training. The UΣU\Sigma and ΣVT\Sigma V^T parametrisations follow a similar behaviour with a initial α^\hat{\alpha} value of 3.338 and 3.975 after 1 epoch training respectively. With subsequent training the α^\hat{\alpha} value tends towards slowly increasing resulting in an α^\hat{\alpha} values of 4.851 and 4.602 for the UΣU\Sigma and ΣVT\Sigma V^T parametrisations respectively. In contrast the UΣVTU\Sigma V^T parametrisation initially reaches an α^\hat{\alpha} of 3.913 after 1 epoch of training and then subsequently fluctuates between increasing and decreasing values throughout training resulting in an α^\hat{\alpha} of 5.683 at the end of training.

The data in this section shows that there is not a strong correlation between the α^\hat{\alpha} metric throughout training and the resulting test performance of the model, as shown with the large test performance difference witnessed regardless of the recorded α^\hat{\alpha} value.

Entropy: fig:19c compares the performance throughout training with the global entropy when different subsets of the models parameters are updated during training. The UU, VTV^T and UVTUV^T parametrisations remain approximately constant throughout training while performance improves. The Σ\Sigma parametrisation approximately linearly decreases in entropy throughout training, with an initial entropy of 1.021 and a end value of 1.002, marking a 0.019 decrease in entropy. The UΣU\Sigma, ΣVT\Sigma V^T and UΣVTU\Sigma V^T parametrisations also decrease in entropy throughout training, however the deviation from initialisation is a lot smaller, with a decrease in entropy of 0.002, 0.001 and 0.001 for each parametrisation respectively. These findings establish that there is not a consistent correlation between model entropy and test performance that exists across parametrisations.

Lambda Max (λmax\lambda_{max}): fig:19d compares the performance throughout training with the global λmax\lambda_{max} when different subsets of the models parameters are updated during training. The UU, VTV^T and UVTUV^T parametrisations remain approximately constant with a value of 1 throughout training while performance improves. The Σ\Sigma parametrisation steadily increases the λmax\lambda_{max} value from 1.022 to 2.460 at the end of training. The initial increase in λmax\lambda_{max} is gradual as demonstrated with the change in λmax\lambda_{max} from epoch 1 to epoch 10 being 0.236, however later in training the increase in λmax\lambda_{max} is more substantial with the last 10 epochs resulting in a difference of 0.776.

The inclusion of the UU parametrisation to Σ\Sigma via the UΣU\Sigma parametrisation, results in the λmax\lambda_{max} value exhibiting the opposite behaviour to the Σ\Sigma parametrisation with a large initial increase in λmax\lambda_{max} at the start of training which then slows down as training progresses. Albeit the difference of the λmax\lambda_{max} trajectory results in a similar λmax\lambda_{max} value of 2.498 at the end of training. The inclusion of of the VTV^T parametrisation to Σ\Sigma via the ΣVT\Sigma V^T parametrisation, changes the training trajectory of λmax\lambda_{max}. The training trajectory closely mimics the training trajectory of UΣU\Sigma parametrisation however the change of λmax\lambda_{max} between epochs is consistently smaller, resulting in a λmax\lambda_{max} value of 2.248 at the end of training. The UΣVTU\Sigma V^T parametrisation λmax\lambda_{max} training trajectory closely mimics the ΣVT\Sigma V^T parametrisation. Starting with a small initial increase in λmax\lambda_{max} which then gradually increases through training resulting in a λmax\lambda_{max} value of 2.093 at the end of training. These results show that although λmax\lambda_{max} can correlate with performance increases, see UΣU\Sigma, ΣVT\Sigma V^T and UΣVTU\Sigma V^T parametrisations, it is not required to change for a model to learn a generalisable solution, see UU, VTV^T and UVTUV^T parametrisations.

Log Alpha Norm: fig:19e compares the performance throughout training with the global Log Alpha Norm when different subsets of the models parameters are updated during training. The Figure largely shows the same relationship between the parametrisations UΣVTU\Sigma V^T, UΣU\Sigma, ΣVT\Sigma V^T, UVTUV^T and UΣVTU\Sigma V^T and the α^\hat{\alpha} metric observed in fig:19b. The UU, VTV^T and UVTUV^T parametrisations Log Alpha Norm values vary significantly throughout training. This is expected, as Log Alpha Norm is defined as log⁡(∥X∥αα)\log(\|X\|_{\alpha}^{\alpha}), where ∥X∥αα=∑iMλiα\|X\|_{\alpha}^{\alpha} = \sum_{i}^{M}\lambda_i^{\alpha}, (see Section G) and therefore inherits the variability from the calculation of the α\alpha metric. The Σ\Sigma parametrisation Log Alpha Norm value quickly decreases after 1 epoch training going from a value of 14.527 to 3.012. The Log Alpha Norm value then slowly increases throughout training, with small changes between epochs, reaching a value of 3.573. The UΣU\Sigma, ΣVT\Sigma V^{T}, and UΣVTU\Sigma V^{T} parametrisations exhibit similar behaviour to the Σ\Sigma parametrisation however the initial decrease is lower going from 8.84 and 5.636 to 4.131 and 5.134 respectively. The parametrisations Log Alpha Norm then subsequently gradually increases throughout training as observed with the Σ\Sigma parametrisation. fig:19e shows that while the Log Alpha Norm can appear to correlate with improvements in test performance throughout training, this correlation is not consistently observed across all parametrisations namely, UU, VTV^{T} and UVTUV^{T}.

Frobenius Norm: fig:19f compares the performance throughout training with the global Frobenius Norm when different subsets of the models parameters are updated during training.

The UU, VTV^{T} and UVTUV^{T} parametrisations improved in test performance throughout training while the global Frobenius Norm stayed approximately constant throughout training, reflecting that the singular values are held fixed at 1 throughout training.

The Σ\Sigma parametrisation global Frobenius Norm initially decreases early in training reaching its lowest value at epoch 5 of 427.618 a 0.726 decrease from the initial value before training while corresponding to an improvement in test accuracy of 2.395%. The global Frobenius Norm subsequently increases with every epoch passing the initial global Frobenius Norm at epoch 12, with a test accuracy of 19.828% marking a 9.649% improvement from initialisation. The resultant global Frobenius Norm at the end of training is 442.849 corresponding with a test accuracy of 37.246%.

The UΣU\Sigma, ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations global Frobenius Norm follow a similar behaviour as observed with the Lambda Max λmax\lambda_{max} metric in fig:19d. The UΣU\Sigma has a large initial increase in the global Frobenius Norm value which then gradually increases throughout training to a value of 478.707 with a test accuracy of 96.458%. The ΣVT\Sigma V^{T} parametrisation follows a similar training trajectory as the UΣVTU\Sigma V^{T} parametrisation, however makes a larger initial increase during epoch 1 to a global Frobenius Norm of 432.480 in comparison UΣVTU\Sigma V^{T} reaches a value of 432.068 which is 0.412 smaller. While the UΣVTU\Sigma V^{T} parametrisation reaches a test accuracy of 90.066% whereas the ΣVT\Sigma V^{T} parametrisation reaches a test accuracy of 88.898% marking a 1.168% difference in test accuracy. Both the ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations steadily increase the global Frobenius Norm throughout training, with ΣVT\Sigma V^{T} reaching a higher value of 467.382 at the end of training marking a 0.892% improvement over the UΣVTU\Sigma V^{T} parametrisation. While the UΣVTU\Sigma V^{T} parametrisation reaches a test accuracy of 97.499% marking a 0.126% improvement over the ΣVT\Sigma V^{T} parametrisation at the end of training.

In summary fig:19f shows that for a model to generalise and achieve good test accuracy the Frobenius norm of the layer weights is not required to change, see factors UU, VTV^{T} and UVTUV^{T}. Furthermore enabling the Frobenius norm to change has limited impact on improving the overall performance of the model throughout training.

Spectral Norm: fig:19g compares the performance throughout training with the global Spectral Norm when different subsets of the models parameters are updated during training. The training trajectories of the global Spectral Norm closely matches the behaviour observed with Lambda Max (λmax\lambda_{max}) in fig:19d. The UU, VTV^{T} and UVTUV^{T} parametrisations global Spectral Norm stays relatively fixed throughout training. The Σ\Sigma parametrisation global Spectral Norm gradually increases through training reaching the highest value while achieving the lowest test accuracy. The UΣU\Sigma, ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations observe an initial large increase in the global Spectral Norm which then continues to gradually increase with UΣU\Sigma reaching the highest value. The ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations reach approximately the same global Spectral Norm value.

Number of Power Law Spikes: fig:19h compares the performance throughout training with the global Number of Power Law Spikes when different subsets of the models parameters are updated during training. As observed with the α\alpha, α^\hat{\alpha} and Log Alpha Norm metrics in Figures 19a, 19b and 19e respectively the UU, VTV^{T} and UVTUV^{T} parametrisations improved in test performance throughout training while the global Number of Power Law Spikes stayed relatively fixed throughout training, with the variation introduced by fitting a power law to singular values are all approximately 1 throughout training.

In contrast to the α\alpha, α^\hat{\alpha} and Log Alpha Norm metrics the global Number of Power Law Spikes increases from the initial value for the Σ UΣ\Sigma\ U\Sigma, ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations.

The Σ\Sigma parametrisation global Number of Power Law Spikes subsequently fluctuates throughout training with alternating between increasing and decreasing within training epochs. Eventually resulting in a Number of Power Law Spikes of 130.717 with a test accuracy of 37.246%.

The UΣU\Sigma parametrisation first epoch of training results in the largest increase in the global Number of Power Law Spikes from 90.750 to 160.667, corresponding to a test accuracy of 85.506%. In the subsequent training epochs the global Number of Power Law Spikes oscillates with a variance of 51.772 between 155.133 and 182.483, resulting in a global Number of Power Law Spikes of 162.250 with a test accuracy of 96.458% at the end of training.

The ΣVT\Sigma V^{T} parametrisation first epoch of training additional results in a large increase in the global Number of Power Law Spikes from 91.150 to 144.217, corresponding to a test accuracy of 88.898%, although a smaller increase than the UΣU\Sigma parametrisation. As observed with the UΣU\Sigma parametrisation in subsequent training epochs the global Number of Power Law Spikes oscillates between increasing and decreasing, resulting in a global Number of Power Law Spikes of 162.633 and a test accuracy of 97.373%.

The UΣVTU\Sigma V^{T} parametrisation follows a similar trend as the UΣU\Sigma and ΣVT\Sigma V^{T} parametrisations while improving in test accuracy, however with a smaller initial increase of the global Number of Power Law Spikes after 1 epoch training increasing from 82.633 to 138.267. The global Number of Power Law Spikes subsequently increases and decreases from epoch to epoch resulting in a global Number of Power Law Spikes of 171.750 and a test accuracy of 97.499% at the end of training.

These results indicate the the Number of Power Law Spikes is an inherently noisy metric with high variably from epoch to epoch, which can be attributed to the relationship it has with the fitting of α\alpha, as the number of power law spikes directly corresponds to how many samples where used to estimate α\alpha. Furthermore these results show that Number of Power Law Spikes does not need to change in order for the model to generalise.

Stable Rank: Figure 19i compares the performance throughout training with the global Stable Rank when different subsets of the models parameters are updated during training. The training trajectories of the global Stable Rank closely matches the behaviour observed with Lambda Max (λmax⁡\lambda_{\max}) and Spectral Norm in Figures 19d and 19i. However instead of increasing the Stable rank decreases throughout training. The UU, VTV^{T} and UVTUV^{T} parametrisations global Stable Rank stays relatively fixed throughout training. The Σ\Sigma parametrisation global Stable Rank gradually decreases throughout training reaching the lowest value while achieving the lowest test accuracy. The UΣU\Sigma, ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations observe an initial small decrease in the global Stable Rank which then continues to gradually decrease with UΣU\Sigma reaching the lowest value. The ΣVT\Sigma V^{T} and UΣVTU\Sigma V^{T} parametrisations reach approximately the same global Stable Rank value.

Discussion

The results from this section show that a model can achieve good test accuracy and generalise on a given task with fixed unbiased singular values (Σ=I\Sigma= \mathrm{I}) and therefore exhibit no change in the spectral properties of the layers of the model. Furthermore, when Σ\Sigma is trained on its own (UU and VTV^{T} fixed) the model does not achieve comparable performance to any other factor, moreover the inclusion of Σ\Sigma with UU and VTV^{T} only marginally improves performance over training with UU and VTV^{T} independently with an increase of 3.222% and 2.449% respectively. These results suggest that the dominate factor in learning can be attributed to the left and right singular vectors. This means that over-interpreting the spectral properties of layers can lead to erroneous and misleading conclusions surrounding generalisation, given that an that they are not required to change for a model to effectively generalise.

The Reliability of the Power Law Fit (α\alpha)

One of the fundamental assumptions of HTSR theory is that the tail of the eigenvalues fit and can be accurately described by a power law distribution, hence the α\alpha metric [30]. However, although the power law distribution can be fit to data, this does not mean it is the best or most apt distribution to describe the data [6]. Therefore the following sections § K.1, § K.2 explore the power law distribution assumption by performing a goodness of fit test comparing the power law distribution to the exponential, the log-normal, truncated power law and the stretched exponential distributions with the power-law package [1]. A p-value of 0.05 is used to disambiguate between statistically meaningful differences in the resulting test normalised likelihood ratios. A negative likelihood ratio indicates support for the alterative distribution.

Fitting a power law requires finding the minimum value of the tail xminx_{min}, which is done utilising the Kolmogorov-Smirnov (KS) distance [6]. When there exists one minimal across all possible xminx_{min} values fitting the tail is trivial. However, in practise this is not the case, and furthermore as this is fitted with a maximum likelihood estimation the selection of xminx_{min} can be susceptible to noise. [6] suggests as a rule of thumb that more than or equal to 50 samples can be used to obtain reliable estimates of distributional fit. Given the possibility of outliers in Figure 7 in § K.3 we explore if the selection of xminx_{min} can explain this occurrence and find that ensuring 50 points are used to estimate the tail that we find a stronger correlation between the initial α\alpha and the α\alpha at the end of training.

Overall this section shows that the eigenvalues of the ReLU FCN 5x512 models can be statistically significantly better described by the alternative distributions explored, namely the lognormal, truncated power law and the stretched exponential. For the large scale models there is inconclusive evidence to suggest that the power law distribution is the most appropriate distribution to describe the eigenvalues of these layers. Therefore in lieu of a clear statistical preference towards the power law distribution, as cautioned in [6], interpreting the data as fitting a power law may lead to erroneous and misleading conclusions about the nature of the data, in this case the spectral properties of the trained weight matrices. Furthermore identifying the tail accurately has inherent challenges and selecting the xminx_{min} with the smallest KS distance may result inaccurate findings.

Small Scale Models (ReLU FCN 5x512)

As all 100 ReLU FCN 5x512 models where trained on the same dataset with the same initialisation method and training strategy we ensemble the eigenvalues for each respective layer from the 100 models to provide a more representative sample of the eigenvalues of each layer. Concretely for FC1 to FC5 we have 51200 data points and FC6 we have 1000 data points. We compare the goodness of fit of the ensemble eigenvalues of each layer against the power law and alternative in Table 17. Table 17 shows that the statistically significant evidence that that eigenvalues of the layers can be better described as either a log-normal distribution, truncated power law or a stretched exponential (note negative normalised likelihood ratios and p-values < 0.5). This provides statistically significant evidence that the power-law distribution is not a suitable model for describing the eigenvalues of this model and demonstrates a clear preference to the alternatives.

LayerExponential Normalised LRExponential P-valueLognormal Normalised LRLognormal P-valueTruncated Power Law Normalised LRTruncated Power Law P-valueStretched Exponential Normalised LRStretched Exponential P-value
FC16.8380.000-3.8290.000-5.0260.000-3.1510.002
FC27.5220.000-5.5020.000-5.3030.000-5.8580.000
FC37.6330.000-5.7640.000-5.5350.000-6.1520.000
FC47.1350.000-6.0140.000-5.7670.000-6.4310.000
FC58.2240.000-5.1210.000-4.8260.000-5.4500.000
FC6-9.1150.000-4.7320.000-9.1160.000-4.8640.000

Table 17. ReLU FCN 5x512: Goodness of fit test of the power law distribution against the alternatives with corresponding normalised likelihood ratio (LR) (a negative value indicates support for the alternative) and p-value. For reliable fitting a minimum of 100 samples were required to fit the tail.

Large Scale Models

In this section we explore the goodness of fit of the power law to eigenvalues of the pretrained models explored in § 4.2 namely ResNet34, ViT, GPT2 GPT2-Large and Pythia-1B layers. We compare the goodness of fit for each layer and report how many layers either favoured each distribution compared or return an inconclusive results, where an inconclusive result is a result that reports a p-value greater than 0.05, The results for each respective model are shown in Table 18.

(a) ResNet34
ComparisonPower Law FavouredAlt FavouredInconclusive
Power Law vs Exponential4 (10.811%)4 (10.811%)29 (78.378%)
Power Law vs Log Normal0 (0.000%)10 (27.027%)27 (72.973%)
Power Law vs Truncated Power Law0 (0.000%)17 (37.838%)23 (62.162%)
Power Law vs Stretched Exponential1 (2.703%)10 (27.027%)26 (70.270%)
(b) ViT
ComparisonPower Law FavouredAlt FavouredInconclusive
Power Law vs Exponential4 (8.000%)2 (4.000%)44 (88.000%)
Power Law vs Log Normal0 (0.000%)7 (14.000%)43 (86.000%)
Power Law vs Truncated Power Law0 (0.000%)9 (18.000%)41 (82.000%)
Power Law vs Stretched Exponential0 (0.000%)7 (14.000%)43 (86.000%)
(c) GPT2
ComparisonPower Law FavouredAlt FavouredInconclusive
Power Law vs Exponential7 (14.286%)3 (6.122%)39 (79.592%)
Power Law vs Log Normal0 (0.000%)3 (6.122%)46 (93.878%)
Power Law vs Truncated Power Law0 (0.000%)7 (14.286%)42 (85.714%)
Power Law vs Stretched Exponential1 (2.041%)3 (6.122%)45 (91.837%)
(d) GPT2-Large
ComparisonPower Law FavouredAlt FavouredInconclusive
Power Law vs Exponential14 (9.655%)0 (0.000%)131 (90.345%)
Power Law vs Log Normal0 (0.000%)0 (0.000%)145 (100.000%)
Power Law vs Truncated Power Law0 (0.000%)18 (12.414%)127 (87.586%)
Power Law vs Stretched Exponential0 (0.000%)3 (2.069%)142 (97.931%)
(e) Pythia-1B
ComparisonPower Law FavouredAlt FavouredInconclusive
Power Law vs Exponential5 (7.812%)0 (0.000%)59 (92.188%)
Power Law vs Log Normal0 (0.000%)0 (0.000%)64 (100.000%)
Power Law vs Truncated Power Law0 (0.000%)2 (3.125%)62 (96.875%)
Power Law vs Stretched Exponential1 (1.562%)0 (0.000%)63 (98.438%)

Table 18. Comparison of distribution fits across layers using likelihood ratio tests. Counts show the number of layers favouring each distribution. Inconclusive indicates tests with a p-value greater than 0.05.

The Tables show that the power law distribution is minimally and rarely favoured across models and when it is favoured this typically occurs in comparison to the exponential distribution. Furthermore the majority of comparisons are inconclusive and thereby show no preference to a particular distribution. This means that there is not sufficient statistical evidence to support the use of the power law distribution to fit the tails of the eigenvalues as there exist multiple distributions that can provide comparably good fits to the data.

Selection of xmin⁡x_{\min}

In § 4.3 Figure 7 we observe that there is a strong correlation with model α\alpha at initialisation and at the end of training, however for the interpolated points of 0.1-0 appeared to be outliers to this trend achieving α\alpha values lower than the trend would expect. As [6] states that 50 samples can be used a rule of thumb for the minimum required samples to obtain reliable estimates of distributional fit, we explore in Figure 20 how many data points were used in fitting the power-law to the tail of the data (the black vertical line represents the 50 sample cut off). The Figure shows that many of the interpolation points α\alpha values are estimated with fewer than 50 points.

Figure 20. Number samples used for alpha estimation for each layer. Each dot represents a layer from a trained model

Given the results in Figure 20 and guided by the role of thumb we re-estimate the α\alpha values using the power law package [1] with a constraint of a minimum of 50 points used to estimate alpha for layers FC1-FC5, for layer FC6 α\alpha is still calculated from the 10 eigenvalues, to ensure the model average α\alpha is maintained and enable comparability with the original figure. The results are shown in Figure 21. It is important to note that λ\lambda points 0 and 10001 are not plotted as suitable xminx_{min} could not be found for the initialisation which expected given they are at the extremes of the interpolation. In this figure the trend that α\alpha at initialisation strongly correlates to the α\alpha at the end of training is substantially clearer, which provides further credence to the statement that the ’good’ range [2, 6] is a consequence of our standard initialisation regime and regularisation from SGD. This underscores that α\alpha does not necessarily emerge because this metric is indicative of generalisation.

Large Scale Model Details

The pretrained models were accessed from huggingface:

  • ResNet34: https://huggingface.co/timm/resnet34.tv_in1k

  • ViT: https://huggingface.co/timm/vit_base_patch16_224.augreg_in1k

  • GPT2: https://huggingface.co/openai-community/gpt2

  • GPT2-Large: https://huggingface.co/openai-community/gpt2-large

  • Pythia-1B: https://huggingface.co/EleutherAI/pythia-1b

Computational Resources

Model training and test evaluation was conducted using A100 and L40S GPUs and the AMD EPYC 7742 CPU.

Small Scale Analysis of Alpha Under Re-Initialisation and Re-Randomisation

To train each model took approximately 1 minute. Given 100 models were trained, training the models took approximately 100 GPU minutes. The evaluation time of the model was approximately 0.1 GPU minutes. Therefore total GPU usage was approximately 220 GPU minutes.

Large Scale Analysis of Alpha Under Reparametrisation

To evaluate the performance of the reparametrisation of each layer for each model took approximately:

  • ResNet34: 0.23 GPU minutes

  • ViT: 1 GPU Minutes

  • GPT2: 0.23 GPU minutes

  • GPT2-Large: 1 GPU Minutes

  • Pythia-1B: 1 GPU Minutes

The number of layers evaluated for each model was:

  • ResNet34: 37

  • ViT: 43

  • GPT2: 45

  • GPT2-Large: 145

  • Pythia-1B: 64

We evaluated the ResNet34, ViT, GPT2, GPT2-Large model with 10 random initialisation. For the Pythia-1B we use one initialisation and one random initialisation. This experiment took approximately: 2196.6 GPU minutes.

Alpha At Initialisation & Training With Any Alpha

Training and evaluating each model took approximately 4.16 CPU minutes. We trained a total of 410 models (41 interpolation points and 10 random seed) totalling approximately 1705.6 CPU minutes or 28.42 CPU hours.

Generalisation With Fixed Alpha

Training and evaluating and evaluating each model took approximately 90 GPU minutes. Given we explored 3 alpha conditions and averaged over 10 models the total training times was 2700 GPU minutes.

SVD Ablation for Learning Dynamics

Training with each factor took approximately:

  • UU: 12 GPU minutes

  • Σ\Sigma: 12 GPU minutes

  • VV: 80 GPU minutes

  • UΣU\Sigma: 15 GPU minutes

  • ΣV\Sigma V: 85 GPU minutes

  • UVUV: 90 GPU minutes

  • UΣVU\Sigma V: 100 GPU minutes

Given we average over 10 models the total training time was approximately 3940 GPU minutes.

Total Training and Evaluation time

The total training and evaluation time was approximately: 8928.6 GPU minutes and 1705.6 CPU minutes, or more aptly: 148.81 GPU hours and 28.43 CPU hours.

Datasets and Licences

Image Datasets

  • MNIST [23] has an CC licence.

  • ImageNet [38] ImageNet states that: ‘The data is available for free to researchers for non-commercial use.’ giving a CC BY-NC licence.

Language Datasets

  • WikiText-103 [32] has a CC BY-SA 4.0 licence.

NeurIPS Paper Checklist

  1. Claims

    Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

    Answer: [Yes]

    Justification: In § 4.1 and § 4.2 we show from low complexity datasets (MNIST) and small scale models (ReLU FCN 5x512) settings to high complexity datasets (ImageNet and WikiText-103) on million (ResNet34 and ViT) and hundreds of million (GPT and GPT Large) parameter scales across modalities (Image and Language) that weight analysis provided by random matrix theory, primarily alpha, are not reparametrisation invariant under critical layer theory. Following this in § 4.4 we show that ideas around the criticality of optimal alpha values in models are misplaced; through simple modifications to optimisation we show that well generalising models can arise for an arbitrary alpha value. In addition, initialisation plays a strong role in the alpha found at the end of training. Our empirical evidence shows that optimal alpha values are not a prerequisite of generalisation and in Appendix Section J we quantify the contribution by alpha through ablations studies using singular value decomposition. Finally, in Appendix Section H we show in small scale settings that data-based metrics provide crucial insights between robust and critical layers that fundamentally relate to existing theories of the lottery ticket hypothesis, Section H.

  1. Limitations

    Question: Does the paper discuss the limitations of the work performed by the authors?

    Answer: [Yes]

    Justification: In Appendix Section A we clearly articulate in which settings our findings hold and their extensions to metrics that are not explored in the paper. We also provide additional modalities where our findings would need to be confirmed.

  1. Theory assumptions and proofs

    Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

    Answer: [Yes]

    Justification: To analyse data-free metrics in Theorem 1 we show (through an extension of Dihn et al [9]) that for a majority of metrics values can be arbitrarily modified while preserving the same functional representation of the model. To study alpha, our paper provides a set of extensive empirical evaluations to explore the relationship between data-free metrics and generalisation. We acknowledge that formal proofs would be useful for our study of alpha, however the results presented concretely show that optimal alpha values are not causal for generalisation.

  1. Experimental result reproducibility

    Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

    Answer: [Yes]

    Justification: We provided full explanations for the range of experiments conducted in this paper with metric formalisations, model architectural details, optimisation hyperparameters and dataset details.

  1. Open access to data and code

    Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

    Answer: [Yes]

    Justification: Notebooks are provided at https://github.com/GMW99/data-free-metrics-dont-work which demonstrate the method and analysis for the MNIST and LLM cases.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings for the core of the paper are provided to a level of detail that enable reproduction of the results and allow for full understanding of their experimental context.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide large sample sizes for experiments in the paper, typically calculating means from 100 models or variations with a few instances where we calculate means with 10 models. In tables we report means and ± 1 Standard Error of The Mean to best represent uncertainty [3], we also analyse the correlation between data-free and data-based metrics and test accuracy using the Spearman correlation coefficient ρ\rho, coefficient of determination R2R^2, the root mean square error (RMSE) of the linear regression and Kendall’s tau measure (K−τK-\tau). When conducting likelihood tests we utilise a p-value of 0.05.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Appendix SectionM we provide details on the compute resources used to run the experiments in this paper, we also detail the estimated total compute required to re-run the experiments in the paper.

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: Every author has reviewed the code of ethics and can confirm that out paper confirms in every aspect with the NeurIPS Code of Ethics.

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [N/A]

Justification: Our results can be considered under foundational research and provide no association with particular applications or deployments. However, we do acknowledge that our results show that data-free metrics may not be suitable for understanding generalisation which impact interpretability efforts leveraging data-free metrics for neural networks across data domains.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: In the paper we provide no new models or scrapped datasets that have risk of misuse.

12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In the paper we have appropriately cited assets that have been used and provided url’s to specific datasets, models and tools where required. In Appendix Section N we explicitly state the licenses for each dataset used in the paper.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: Our paper does not release new assets. However, details of datasets and models used for the empirical analysis of the paper are provided in the paper.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: At no point in core method development of the paper were LLM’s used.

References

  1. [1]Jeff Alstott, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, 9(1), 2014. URL: https://doi.org/10.1371/journal.pone.0095816.
  2. [2]Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. URL: https://papers.nips.cc/paper_files/paper/2017/hash/b22b257ad0519d4500539da3c8bcf4dd-Abstract.html.arxiv.org/abs/1706.08498
  3. [3]Sarah Belia, Fiona Fidler, Jennifer Williams, and Geoff Cumming. Researchers misunderstand confidence intervals and standard error bars. Psychological methods, 10(4):389, 2005. URL: https://psycnet.apa.org/record/2005-16136-002.pubmed.ncbi.nlm.nih.gov/16392994
  4. [4]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Afiah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023-23. URL: https://proceedings.mlr.press/v202/biderman23a.html.arxiv.org/abs/2304.01373
  5. [5]Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Philipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014. URL: https://arxiv.org/abs/1312.3005, arXiv:1312.3005.
  6. [6]Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661–703, 2009. URL: https://doi.org/10.1137/070710111.
  7. [7]Romain Couillet and Zhenyu Liao. Random matrix methods for machine learning. Cambridge University Press, 2022. URL: https://www.cambridge.org/core/books/random-matrix-methods-for-machine-learning/6B681EB69E58B5F888EDB689C160C682.DOI
  8. [8]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. URL: https://doi.org/10.1109/CVPR.2009.5206848.
  9. [9]Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1019–1028. PMLR, 2017. URL: https://proceedings.mlr.press/v70/dinh17b.html.arxiv.org/abs/1703.04933
  10. [10]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=YicbFdNTTy.
  11. [11]Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. In Advances in Neural Information Processing Systems, volume 35, pages 33054–33065, 2022. URL: https://proceedings.neurips.cc/paper_files/paper/2022/hash/d5cd70b708f72673e2ebace18c37f71b-Abstract-Conference.html.
  12. [12]Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL: https://openreview.net/forum?id=rJl-b3RcF7.
  13. [13]Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256. PMLR, 2010-13. URL: https://proceedings.mlr.press/v9/glorot10a.html.
  14. [14]Di He, Songjun Tu, Ajay Jaiswal, Yulong Pei, Li Shen, Shiwei Liu, and Lu Yin. Alphadecay: Module-wise weight decay for heavy-tailed balancing in llms. International Journal of Artificial Intelligence and Robotics Research, 03, 2026. URL: https://doi.org/10.1142/S927233532650002X.
  15. [15]Jing He, Hua Jiang, Cheng Li, Siqian Xin, and Shuzhen Yang. Early-stopping for transformer model training, 2025. URL: https://arxiv.org/abs/2510.16074, arXiv:2510.16074.
  16. [16]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, pages 1026–1034. IEEE Computer Society, 2015. URL: https://doi.org/10.1109/ICCV.2015.123.
  17. [17]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016-06. URL: https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html.arxiv.org/abs/1512.03385
  18. [18]Thomas King, Simon Butcher, and Lukasz Zalewski. Apocrita - High Performance Computing Cluster for Queen Mary University of London, 2017-03. doi:10.5281/zenodo.438045.
  19. [19]King’s College London HPC. King’s computational research, engineering and technology environment (create), 2025. URL: https://doi.org/10.18742/rnvf-m076.
  20. [20]Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 57(9):1–52, 2025. URL: https://doi.org/10.1145/3728458.
  21. [21]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR, 2019-09. URL: https://proceedings.mlr.press/v97/kornblith19a.html.arxiv.org/abs/1905.00414
  22. [22]Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009. URL: http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf.
  23. [23]Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]., 1998. URL: http://yann.lecun.com/exdb/mnist.
  24. [24]Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 888–896. PMLR, 2019-16. URL: https://proceedings.mlr.press/v89/liang19a.html.
  25. [25]J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151, 1991. doi:10.1109/18.61115.DOI
  26. [26]Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yifan Zhou, Pu Ren, and Yaoying Yang. Model balancing in helps low-data training and fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1311–1331. Association for Computational Linguistics, 2024-11. URL: https://doi.org/10.18653/v1/2024.emnlp-main.78/.
  27. [27]Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL: https://openreview.net/forum?id=fHq4x2YXfv.arxiv.org/abs/2410.10912
  28. [28]Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning, pages 4284–4293. PMLR, 2019. URL: https://icml.cc/media/icml-2019/Slides/4388.pdf.arxiv.org/abs/1901.08276
  29. [29]Charles H Martin and Michael W Mahoney. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 505–513. SIAM, 2020. URL: https://epubs.siam.org/doi/abs/10.1137/1.9781611976236.57.arxiv.org/abs/1901.08278
  30. [30]Charles H. Martin and Michael W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021. URL: http://jmlr.org/papers/v22/20-410.html.
  31. [31]Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1):4122, 2021. URL: https://www.nature.com/articles/s41467-021-24025-8.DOI
  32. [32]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL: https://openreview.net/forum?id=Byj72udxe.arxiv.org/abs/1609.07843
  33. [33]Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu, and Yao-qing Yang. Htmuon: Improving muon via heavy-tailed spectral correction. arXiv preprint arXiv:2603.10067, 2026. URL: https://arxiv.org/pdf/2603.10067.
  34. [34]Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. In Advances in Neural Information Processing Systems, volume 34, pages 18420–18432, 2021. URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/995f5e03890b029865f402e83a81c29d-Paper.pdf.arxiv.org/abs/2001.00939
  35. [35]Hari Kishan Prakash and charles h martin. Grokking and generalization collapse: Insights from HTSR theory. In High-dimensional Learning Dynamics 2025, 2025. URL: https://openreview.net/forum?id=upyC7rwR1M.arxiv.org/abs/2506.04434
  36. [36]Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. Alphaflora: Assigning lora experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20511–20523, 2024. URL: https://aclanthology.org/2024.emnlp-main.1141.pdf.
  37. [37]Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. J. ACM, 54(4):21–es, 2007. URL: https://doi.org/10.1145/1255443.1255449.
  38. [38]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. doi:10.1007/s11263-015-0816-y.DOI
  39. [39]Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016. URL: https://proceedings.neurips.cc/paper_files/paper/2016/hash/ed265bc903a5097f61d3ec064d96d2e-Abstract.html.arxiv.org/abs/1602.07868
  40. [40]Amartya Sanyal, Philip H. Torr, and Puneet K. Dokania. Stable rank normalization for improved generalization in neural networks and gans. In International Conference on Learning Representations, 2020. URL: https://openreview.net/forum?id=H1enKkrFDB.arxiv.org/abs/1906.04659
  41. [41]Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL: https://arxiv.org/abs/1409.1556, arXiv:1409.1556.
  42. [42]Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix theory analysis of neural network weight matrices. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024. URL: https://openreview.net/forum?id=41kpc2Nzwc.
  43. [43]Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In Advances in Neural Information Processing Systems, volume 34, pages 24261–24272, 2021. URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Paper.pdf.arxiv.org/abs/2105.01601
  44. [44]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.arxiv.org/abs/1706.03762
  45. [45]Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23549–23588. PMLR, 2022-17. URL: https://proceedings.mlr.press/v162/wei22a.html.arxiv.org/abs/2203.06176
  46. [46]David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Yardi, Karen Livescu, Michael Maire, and Matthew R. Walter. Approaching deep learning through the spectral dynamics of weights, 2024. URL: https://arxiv.org/abs/2408.11804, arXiv:2408.11804.
  47. [47]Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL: https://openreview.net/forum?id=Sy8gdB9xx.arxiv.org/abs/1611.03530
  48. [48]Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? Journal of Machine Learning Research, 23(67):1–28, 2022. URL: http://jmlr.org/papers/v23/20-069.html.

Paper details

vlnc:W_bswfmf4d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents