WorksComputer Science
Data-Free Metrics Don't Work: The Missing Invariance Under Functionally Preserving Reparameterisations
Abstract
Data-free methods for analysing and understanding the layers of neural networks offer many metrics for quantifying notions of 'strong' versus 'weak' layers, with the promise of increased interpretability. In particular, random matrix theory (RMT) offers data-free metrics that claim predictive power over the quality of pre-trained models at a layerwise precision and the ability to identify model pathologies, indicating a unique relationship with generalisation. As a result, metrics from RMT are championed for pre-training optimisation strategies and post-training compression. We establish that RMT-based metrics are unrelated to performance or training by showing that functionally preserving reparametrisations of a pre-trained model can have arbitrary metrics. We show this across a range of architectures and scales. To create functionally preserving reparametrisations, we exploit the well-established phenomenon of criticality: some layers can be re-initialised or re-randomised without affecting the functional behaviour of the model – they are called robust – while others cannot – they are called critical. Re-initialising or re-randomising robust layers provides functionally preserving reparametrisations for which RMT-based metrics are arbitrary. Moreover, we show that relationships between metrics offered by RMT are spuriously related to generalisation. We conclude by showing that if many pre-trained models have data-free metrics in a 'good' range, it is, in part, dependent on model initialisation. Project page available at: https://gmw99.github.io/data-free-metrics-dont-work
Project page available at: https://gmw99.github.io/data-free-metrics-dont-work/.
Introduction
Understanding and interpreting deep learning models is a critical area of research, especially as the prevalence of these models increases in real-world applications. The holy grail of neural network interpretability lies in identifying computationally efficient metrics that can provide deep insights into the effectiveness of neural networks and their components. Data-free methods typify this endeavour by analysing the properties of neural network parameters without regard for the training or testing
40th Conference on Neural Information Processing Systems (NeurIPS 2026). data, with Random Matrix Theory (RMT) being suggested as ideally suited to understanding deep neural networks [7 Chap. 5] with numerous applications of RMT to deep neural networks.
Prominent among RMT-inspired data-free methods is a family of metrics that can, it is claimed, predict the quality of pre-trained models by only examining their weights. These metrics are based on the theory of Heavy-Tailed Self Regularization (HTSR) developed in [28, 29, 30] and are described in a Nature paper [31] that shows empirically that well-trained models tend to have metrics lying in ‘good’ ranges and concludes that these metrics can be used to discriminate between well-trained and poorly-trained models as well as identify model pathologies. In this paper, we will show that reversing the implication ‘good models good metrics’ into ‘good metrics good models’ is not only a logical fallacy, it is also empirically incorrect as we show over a range of scales and architectures that bad models can have good metrics. We also invalidate the original premise that ‘good models good metrics’ by similarly showing that good models can have bad metrics. We argue that if pre-trained models do indeed exhibit ‘good’ metrics, it is, in part, due to our commonly used initialisation strategies and not because such data-free properties are a prerequisite for generalisation.
Scrutinising the claim that the metrics of [30, 31] can predict the quality of a model is important as numerous lines of research have built on this claim, with many assuming that specific data-free metric properties are fundamental for generalisation. Data-free metrics have been used in the context of layer-wise pruning ratios in LLMs [27], explaining and understanding stages of the grokking phenomenon [35], identifying layers to apply LoRA adapters to [36], adapting layer-wise learning rates [26] and weight decay values [14], informing early stopping conditions [15] and improving optimizers [33]; with data-free metrics often considered to have a unique relationship with generalisation. These metrics thus promise a lot: a better understanding of optimisation, redundancy, transitions from memorisation to generalisation, and interpretability of deep neural networks more generally. Our scrutiny of these metrics should be commensurate with these promises.
Our strategy to understand whether data-free metrics of [30, 31] are causal for performance or generalisation follows the well-known work of Dinh et al. [9] which showed that measures of flatness around local minima are not causal for generalisation. They did so by constructing functionally equivalent networks for which measures of flatness around a local minimum could be arbitrarily far apart, thereby disproving links between flatness and generalisation. We follow the same strategy, but focus on empirical results showing that functionally preserving models can have very distant metrics, thereby disproving the link between data-free metrics and model performance or training.
An experimentally useful class of functionally preserving models is provided by the work of Zhang et al. [47, 48] that identified an interesting and unexpected phenomenon in neural network layers: some layers within a network are robust, while others are critical. A robust layer can be either re-initialised or re-randomised without any noticeable effect on performance. A critical layer is a layer that cannot be re-initialised or re-randomised without dramatically affecting the performance of the network. Re-initialisation sets the layer back to its initial parameters before training, whilst re-randomisation sets the parameters of the layer to random values by re-sampling from the same distribution used for initialisation. It was observed that in some cases, re-initialisation and re-randomisation can result in significant performance differences for a given layer, with re-initialisation maintaining performance but re-randomisation significantly degrading it [48]. In other cases, re-initialisation and re-randomisation of a layer lead to a negligible difference in performance. This distinction between layer behaviours offers a core tool for testing the robustness of data-free metrics. For our purpose, re-initialisation and/or re-randomisation of robust layers will be used to build functionally preserving models for which the HTSR-based data-free metrics will take a large range of (mostly ‘bad’) values. In other words, the metrics offered by HTSR are not reparameterisation-invariant under re-initialisations or re-randomisations of robust layers and therefore only offer spurious insights into generalisation or well-trainedness. We also show that these metrics cannot distinguish between robust and critical layers, nor predict performance differences between re-initialisation and re-randomisation.
As a natural follow-up question from our layer-wise analysis of reparameterisation and data-free metrics, we investigate global data-free properties (average value of these metrics over all layers) and generalisation. Our results show that global data-free metric values can also be arbitrary for well-generalising models: neither layer-wise nor average data-free metrics relate to generalisation.
We hope that our work can push the field forward in two core ways. Firstly, by providing a strong test ground for the development of data-free metrics that are reparametrisation invariant, or at least do not differ too much on functionally preserving reparametrisations of models. This is precisely what the work of [9] led to, with the development of new sharpness metrics such as Relative Flatness [34] and Fisher-Rao norm [24] that are reparameterisation invariant and now considered state of the art sharpness metrics. Secondly, our work offers concrete evidence for the prioritisation of data-based metrics to improve efforts in interpretability and compression (at least until adequate data-free metrics are discovered).
Background
As discussed briefly above, data-free methods of interpretability aim to understand the inner workings of neural networks by studying the properties of the network parameters. Data-free approaches often focus on the matrix norm properties of layer weight matrices to understand learning or improve the performance of neural networks [46, 31, 40, 11, 39, 2, 45, 36, 42, 14]. However, [48] showed in their work that matrix norms, such as the Frobenius norm, are too coarse to understand the generalisation properties of neural networks. [30, 31] use RMT to analyse the weight matrices (excluding biases) of neural network layers through training to create their theory of heavy-tailed self-regularisation (HTSR). With this theory, they construct a set of predominately power-norm metrics related to generalisation that is applied after training to assess layer performance. Given a weight matrix and , the eigenvalues of the following metrics are defined:
and weighted-: the exponent of the power-law that best describes the distribution of the eigenvalues [30], and . Estimating is not straightforward [30 3.2] and we show through several experiments in § K that the distribution of eigenvalues is typically poorly modelled by power-laws, undermining the statistical basis for and its derivatives.
-norm and log--norm: ,
Frobenius and log-Frobenius norm: ,
Spectral and log-spectral norm: ,
Following in the footsteps of [9], we can already dismiss several of these metrics as potential indicators of performance or well-trainedness. For this, consider a two-layer network and the reparametrisation maps , . It is clear that is invariant under this reparametrisation, i.e. . Since the spectrum of is the spectrum of scaled by the constant positive factor , it follows immediately that:
Theorem 1 Let be any of the metrics above except . For any value , there exists such that the value of applied to the layers of the network reparametrised by is greater (resp. smaller) than .
Thus, two functionally equivalent networks can differ under all the metrics above – except – by an arbitrary amount. It would therefore be unreasonable to expect these metrics to have any predictive power on the quality or well-trainedness of a given layer in a network. Note that taking the average of the metrics over both layers (as done in [31]) will lead to the exact same conclusion.
However, itself is invariant or close to invariant (depending on estimation techniques) under . For example, the standard maximum likelihood estimator for is given by which is clearly invariant under rescaling the eigenvalues. Other metrics involving quotients of eigenvalues, for example stable rank, are also outside of the scope of Theorem 1. We will thus focus on the metric in the rest of this paper (results for other metrics can be found in § G). According to [30, 31], well-trained models have values in the optimal range [2, 6] for most of their layers, e.g. [30 Table 6, Fig. 12], [31, Figs. 6,7] with values of corresponding to overfitting/memorisation and to underfitting (see also https://weightwatcher.ai).
The map defined above provides an example of reparametrisation, i.e. of a map on the space of parameters of a model, typically assumed to be a homeomorphism or at least a bijection. Note that whilst reparametrisations are often mentioned in the context of invariance properties under said reparametrisations, these are disjoint concepts, and in this work we will simply understand a reparametrisation as a homeomorphism on the space of parameters of a network without implying any invariance property. We identify a model with its defining set of parameters and say that a metric is invariant under a reparametrisation if , for example in the example above. Any re-initialisation or re-randomisation of the weight matrix associated with the layer of a model defines a unique linear bijective (and thus homeomorphic) reparametrisation . It stands to reason that for a model , if a reparametrisation of this model is functionally indistinguishable from , then any metric meant to capture model generalisation should also be (close to) invariant under for , i.e. .
Experimental Setup
Reparametrisation Invariance, Robust and Critical Layers: [48] demonstrated the robust and critical layer phenomenon across a range of trained architectures – MLPs, VGGs [41], ResNets [17], Transformers [44], Vision Transformers [10], MLPMixers [43] – and of datasets – MNIST [23], CIFAR10 [22], ImageNet [8] and LM1B [5].
In § 4.1 we start with the simplest model (ReLU FCN 5x512), Figure 1, and dataset (MNIST) that demonstrates this phenomenon [48] in order to systematically explore the behaviour of data-free metrics under layer re-initialisations and re-randomisations of a large number of trained models.

Figure 1. ReLU FCN Architecture.
An added benefit of the ReLU FCN 5x512 model is that it offers a clear performance contrast between re-initialisation and re-randomisation. [48] showed that residual blocks are robust to re-randomisation and attributed this to the residual layer potentially playing a lesser role in the network and thus having smaller activations than the skip connection. To analyse how effective the data-free metrics are at disambiguating between re-initialisation and re-randomisation, we analyse the correlation between data-free metrics and test accuracy using the Spearman correlation coefficient , coefficient of determination , the root mean square error (RMSE) of the linear regression and Kendall’s tau measure (K-). Where and K- score of -1 indicates a very strong negative correlation, 0 indicates no correlation, and 1 indicates a very strong positive correlation and score of 1 indicates perfect predictions and a score of 0 indicates imperfect predictions. We use , RMSE and Kendall’s Tau measure for this study as they are the correlation metrics used by [31] to highlight the predictive capacity of the data-free metrics. We trained 100 ReLU FCN 5x512 models, creating 100 initialisations and 100 trained models, to obtain a representative sample of possible initialisations and trained models. The model weights and biases are initialised and re-randomised from the same distribution where is , e.g. FC1 has . We record the data-free metric properties of these trained models’ layers when they undergo re-initialisation and re-randomisation and observe whether these metrics:
Are invariant under re-initialisation and re-randomisation, particularly for robust layers,
Can distinguish critical from robust layers,
Can predict the performance difference between re-initialisation and re-randomisation.
In the process we demonstrate the overall lack of predictive capacity of data-free metrics after training.
In § 4.2, we extend our analysis to large-scale pre-trained computer vision models (ResNet34 and ViT) both trained on ImageNet [38] and pre-trained large language models GPT2 and GPT2-Large evaluated on WikiText103 [32]. As we employ pre-trained open source models we do not have access to the starting initialisation and as we result we only focus on the case of re-randomisation to test the predictive capacity of the metric at this scale, concretely questions 1 and 2. We perform re-initialisation and re-randomisation at large scale for the Pythia-1B model [4] and confirm our findings at the MNIST scale in Sections§ B. Furthermore, we provide empirical support for linking functional preservation to accuracy preservation under reparametrisations (re-initialisation and re-randomisation) in the main body for MNIST and in § C for all other models explored. In § G.1 we additionally explore the following metrics: Alpha Weighted (, Log Alpha Norm, MP Soft Rank [30], Spectral Norm, Stable Rank [37] and Generalized von-Neumann Matrix Entropy [30]: which are formally defined in § G. The metrics are collected using the weightwatcher2 tool.
Average Alpha vs Generalisation. We perform two sets of experiments to investigate the (lack of) relationship between average/global alpha and generalisation. In both cases we rely on the singular value decomposition (SVD) of the weight matrices of the model’s layers.
In our first set of experiments (§ 4.3), we study the impact of alpha at initialisation on performance. For this, we construct a family of initialisations by power laws that interpolate the max-normalised singular values between a given initial sampled spectrum and two natural endpoints: the full rank spectrum given by the identity matrix (which corresponds to , ), and the rank 1 spectrum (which corresponds to , ). For each interpolation levels we initialise the weight matrices of each layer of the model as for some randomly initialised orthogonal matrices . This controlled intervention induces different values of at initialisation which allows us to systematically study how initialisation affects both performance and the value of after training. In particular it allows us to assess if models whose layers are all initialised with an within the ‘good’ range achieve better than random accuracy.
In the second experiments (§ 4.4) we extend this analysis to training with a fixed . Concretely, we sample from the random initialisation for each layer to obtain singular values that correspond to a desired value, , and then construct at each layer and keep fixed throughout training. We then solely train and while maintaining the unitary properties. We explore within the ranges of , , and for all layers.
Results and Discussion
In the main body we present our results for . We show that these results generalise to other metrics in § G. In the tables below, we report the mean and the standard error of the mean (SEM) [3]. We show that alpha is not invariant under functionality-preserving reparametrisations in small-scale (MNIST) settings in § 4.1. We then show that our findings hold for other scales and modalities by exploring ImageNet models and the GPT family in § 4.2, with full re-initialisation and re-randomisation for the Pythia-1B model in § B. In § H we show that data-based metrics explain the performance difference from re-initialisation and re-randomisation through their implicit relationship with the lottery ticket hypothesis [12]. We then explore initialisation and average alpha, showing that the optimal alpha range is somewhat arbitrary in § 4.3 and unreliable in § K. Additionally, in § C we conduct a full functional similarity assessment of reparametrisation for all models to show that when accuracy is preserved so is function. Finally, we provide evidence that optimal global alpha values are not causal for well-generalising models by optimising models with arbitrary alpha values in § 4.4 and through a full SVD ablation for learning dynamics in § J.
Small Scale Analysis of Alpha Under Reparametrisation and Training
Quality of training vs alpha. The idea that well-trained layers have seems to be borne out by training 100 ReLU FCN 5x512 models to good values of test accuracy, achieving for every layer (see Table 1). Whilst well-trained may imply (but more on this below), a simple experiment shows that the converse does not hold. We plot the empirical distribution of for a 512x512 fully connected layer in Figure 2a, sampled from 10,000 potential initialisations. The resulting distribution shows that at initialised, untrained layer of this network, can fall, with a small but non-negligible probability, within the optimal value range of 2 and 6. This highlights that a random layer can be considered optimal under RMT which suggests a core failing of the metric.
| Metric | Layer | FC1 | FC2 | FC3 | FC4 | FC5 | FC6 | Test Accuracy |
| Alpha () | 4.820 ± 0.025 | 4.205 ± 0.039 | 4.126 ± 0.038 | 4.135 ± 0.035 | 4.193 ± 0.034 | 3.793 ± 0.805 | 96.822 ± 0.057 |
Table 1. Alpha () of the layers in ReLU FCN 5x512 and test accuracy of the model on MNIST.

Figure 2. (a) Empirical distribution of values on a 512x512 fully connected layer, sampled from 10,000 initialisations. (b) Layer re-initialisation (🔵) and re-randomisation (🟠) test accuracy vs . In § D we present the descriptive statistics for sub-figure (a) along with other initialisation schemes.
Reparametrisation-invariance of alpha. Next, we perform independent re-initialisations (🔵) and re-randomisations (🟠) in Figure 2b of each layer for 100 trained networks and record the impact on the networks’ test accuracy and alpha values. Figure 2b shows that a non-negligible proportion of networks whose performance is severely degraded by re-randomisation maintain a good value of around 5. Conversely, many networks maintain good performance after re-initialisations (particularly of layers FC3-FC5) but with values significantly outside of . As is clear from Figure 2b, whilst all values of start in the ‘good’ range , reparametrisation in the form of re-initialisation or re-randomisation scatters these values over a very wide range (typically deep into “undertitted” territory), irrespective of the change in accuracy incurred by this reparametrisation. Alpha is not invariant under these reparametrisations, even when accuracy is (i.e. on robust layers).
The robust vs critical phenomenon and alpha. Figure 2b shows the stark contrast in how layers respond to re-initialisation and re-randomisation. For example, we only see a large drop in test accuracy when applying re-initialisation to the Layer FC1, re-initialising other layers leaves performance almost unchanged. From this perspective, FC1 is a critical layer whilst FC2-FC6 are robust (to re-initialisation). In this experiment cannot distinguish between these behaviours, the range of values taken by shows no noticeable difference between critical and robust layers.
The re-initialisation vs re-randomisation phenomenon and alpha. We observe different results when re-randomisation is applied. We find that re-randomising any layer degrades accuracy to circa random accuracy on the test set, in other words none of the layers are robust to re-randomisation. Surprisingly, this is not reflected in the corresponding values of these two conditions, as the distribution of values is relatively similar for each layer and each condition. If had predictive power we would observe a negative correlation between re-initialisation (corresponding to ‘good’ alphas) to re-randomisation (corresponding to ‘bad’ alphas). However, there is no difference between the values of re-initialisation and re-randomisation, with a mean Spearman correlation coefficient and Kendall’s tau measure across layers of -0.053 and -0.035, respectively. Thus does not distinguish between these behaviours either.
Generalisation of Reparametrisation Invariance to other Data-Free Metrics. We extend our analysis to other data-free metrics that have been defined in the literature. In spite of Theorem 1 and the lack of predictive power of the metric described above, it is important to verify across other representative measures that data-free metrics are not invariant under reparametrisation (of robust layers), and cannot disambiguate between critical and robust layers, nor between re-initialisation and re-randomisation. When we conduct a similar analysis across six other metrics, Alpha Weighted, Log Alpha Norm, MP Soft Rank, Spectral Norm, Stable Rank and Generalized von-Neumann Matrix Entropy we find, as shown in § G Figure 9, that we can draw exactly the same conclusions with . In § H we show that data-based metrics offer some predictive capacity in reference to criticality.

Figure 9. Layer re-initialisation (●) and re-randomisation (●) test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and K- is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Reparametrisation, Functional Similarity and Accuracy Preservation. As we perform interventions on particular layers of specific models, preserving accuracy can be seen as a proxy for function preservation. To quantify the faithfulness of this proxy in our experiments, we employ functional and representational similarity measures [20] to quantify the output similarity of the original model and the model with the intervention (re-initialisation, Table 2, or re-randomisation, Table 3) on the test set. In Table 2 we observe that applying re-initialisation across layers leads to varying levels of functional preservation, with layers that have a low relative change in accuracy (FC2-FC5) maintaining functional similarity across the respective metrics while layers with higher accuracy deviation (FC1 and FC6) preserving less of the predictive structure of the original model. In the case of re-randomisation in Table 2, there are persistent high deviations in accuracy which is accompanied by increased divergence from the original model’s output distribution across similarity metrics. Overall, the correlation between functional similarity and model accuracy presented in Table 4 (for several functional similarity and correlation metrics) shows that increased functional similarity corresponds to increased accuracy, justifying the following statement: When accuracy is preserved under the reparametrisations offered by re-initialisation and re-randomisation predictive structure of the original model is approximately preserved too.
| Model Layer | Accuracy | Loss | Prediction Similarity () | Jensen-Shannon Distance () | Procrustes Distance () | KL Divergence () |
| FC1 | -39.859 0.936 | 1.226 0.038 | 0.574 0.009 | 0.459 0.003 | 0.340 0.003 | 1.014 0.024 |
| FC2 | -6.330 0.328 | 0.285 0.007 | 0.917 0.003 | 0.272 0.002 | 0.087 0.001 | 0.267 0.004 |
| FC3 | -1.899 0.131 | 0.297 0.003 | 0.962 0.001 | 0.294 0.001 | 0.053 0.001 | 0.289 0.002 |
| FC4 | -1.384 0.080 | 0.317 0.003 | 0.968 0.001 | 0.308 0.001 | 0.051 0.001 | 0.314 0.002 |
| FC5 | -1.716 0.114 | 0.364 0.003 | 0.963 0.001 | 0.329 0.001 | 0.054 0.001 | 0.356 0.002 |
| FC6 | -6.966 0.332 | 0.522 0.004 | 0.910 0.003 | 0.383 0.001 | 0.061 0.001 | 0.487 0.003 |
Table 2. Functional preservation of the ReLU FCN 5x512 by layer under the re-initialisation reparametrisation. ( / ) denotes the preferred direction for each metric.
| Model Layer | Accuracy | Loss | Prediction Similarity () | Jensen-Shannon Distance () | Procrustes Distance () | KL Divergence () |
| FC1 | -86.928 0.217 | 3.704 0.052 | 0.102 0.002 | 0.721 0.001 | 0.894 0.003 | 2.601 0.020 |
| FC2 | -86.692 0.273 | 2.533 0.019 | 0.104 0.003 | 0.712 0.001 | 0.806 0.005 | 2.303 0.010 |
| FC3 | -86.562 0.332 | 2.348 0.009 | 0.105 0.003 | 0.711 0.000 | 0.647 0.008 | 2.237 0.006 |
| FC4 | -87.350 0.415 | 2.300 0.007 | 0.097 0.004 | 0.711 0.000 | 0.466 0.008 | 2.228 0.006 |
| FC5 | -87.007 0.477 | 2.315 0.009 | 0.101 0.005 | 0.711 0.001 | 0.335 0.007 | 2.232 0.006 |
| FC6 | -86.848 0.680 | 2.356 0.012 | 0.102 0.007 | 0.711 0.001 | 0.221 0.007 | 2.263 0.011 |
Table 3. Functional preservation of the ReLU FCN 5x512 by layer under the re-randomisation reparametrisation. ( / ) denotes the preferred direction for each metric.
| Similarity Metric | RMSE | K- | ||
| Prediction Similarity | 0.997 | 1.000 | 0.004 | 0.964 |
| Jensen-Shannon Distance | -0.869 | 0.977 | 0.030 | -0.672 |
| Procrustes Distance | -0.809 | 0.635 | 0.189 | -0.604 |
| KL Divergence | -0.864 | 0.976 | 0.149 | -0.665 |
Table 4. Correlation between the change in accuracy due to re-initialisation (Table 2) and re-randomisation (Table 3) and the functional preservation of the ReLU FCN 5x512.
Analysis of larger models (ResNet’s, ViT’s, GPT2 and Pythia-1B) in § C confirms this conclusion.
Large Scale Analysis of Alpha Under Reparametrisation
This section explores the predictive power of alpha with respect to performance in large scale models that have been trained for high-complexity tasks using the reparametrisation by re-randomisation setup at the hundred million parameter scale. We find, in line with our small scale experiments, that alpha has no predictive capacity in this regime either. For these experiments, we use pre-trained open source models, and we therefore have no access to pre-training initialisations. As a result, we only focus on reparametrisation by re-randomisation to test the invariance and predictive capacity of the metric. The results in this section are consistent with other data-free metrics in this experimental setup which we present in § G.2, G.3, G.4 and G.5.
ImageNet. We explore the relationship of a layer’s value with the corresponding performance of the model on ImageNet with pre-trained ResNet34 and ViT models. The respective performance and model details are shown in Table 5; from this table it can be observed that these models are well-trained for the task of ImageNet. In line with the RMT perspective, the competitive performance of these models is aligned with a mean value across layers within the optimal range.
| Model | Number of Parameters | Loss | Accuracy | Mean Layer Alpha Value |
| ResNet34 | 21.8M | 1.071 | 73.302 | 3.380 0.195 |
| ViT | 86.6M | 0.848 | 79.136 | 4.287 0.231 |
Table 5. Model test performance on ImageNet and mean layer Alpha () value.
To conduct this experiment we randomise a layer independently and then record the corresponding test accuracy and value post re-randomisation. Given the different implicit biases offered by the ResNet and ViT we believe that our selection of models is representative for the core architectural differences, allowing a robust analysis of the impacts of re-randomisation and metrics at scale. We repeat our re-randomisation process 10 times per layer to capture stochastic variation in randomisations. In both ResNet34 and ViT, Figure 3, we find no relationship between and test accuracy, as found in the small scale experiment. Both models can be re-randomised to values outside of the desired range and maintain accuracy. The mean Spearman correlation coefficients for alpha and accuracy are 0.006 and -0.036 respectively, and Kendall’s tau values are 0.004 and -0.022, respectively. In the case of ResNet34 we see that the model is not so robust to re-randomisation, having a lower accuracy than the baseline; a large proportion of layers with optimal values result in a model with essentially random accuracy, i.e. ‘good alpha does not imply good model’. The ViT is more robust to layer re-randomisation and represents the inverse case where many layers are significantly outside of the optimal range but display a high preservation of the baseline accuracy, i.e. ‘good model does not imply good alpha’. As a result, these results confirm in large scale settings that is not invariant under reparametrisation by re-randomisation, even on robust layers, and has very limited predictive capacity over post re-randomisation performance.
Figure 3. Layer re-randomisation Test Accuracy vs Alpha () on ImageNet. See stratified analysis in Appendix F.
Large Language Models. We confirm our negative results for the predictive capacity of the metric in the vision domain under re-randomisation, by examining a different modality. We extend our exploration of the relationship between layer values and the corresponding performance of the model to language models on WikiText103 with pre-trained GPT2 and GPT2-Large. The respective performance and model details is shown in Table 6. Both models have competitive performance with low loss and perplexity on the WikiText-103 [32] datasets and, in-line with HTSR Theory, have mean layer values that are within the optimal range. We use the pre-trained models as the base model and then we randomise a layer and record its value and the corresponding test perplexity and loss of the model. We do this 10 times for each layer to obtain a representative set of random layers. For both GPT2 and GPT2-Large, Figure 4, we observe that their is no relationship between the performance of the model and a layer’s value, with a mean Spearman correlation coefficient of -0.038, -0.134 and Kendall’s tau measure of -0.025 and -0.089 for GPT2 and GPT2-Large respectively. We also find that randomising layers in the GPT2 and GPT2-Large model often has a negligible affect on the performance of the model irrespective of the value associated with the layer. In other words, re-randomising generates functionally preserving reparametrisations of the original model, but with arbitrary values of alpha for the layer under consideration. Furthermore, when re-randomising robust layers, we observe that most of these layers lie far outside the optimal range of . Again, we see that is not invariant under reparametrisation by re-randomisation, even on robust layers, and has no predictive power over performance.
| Model | Number of Parameters | Loss | Accuracy | Perplexity | Mean Layer Alpha Value |
| GPT | 127M | 3.399 | 39.095 | 29.941 | |
| GPT Large | 774M | 2.967 | 43.877 | 19.436 |
Table 6. Model test performance on WikiText-103 and mean layer Alpha () value.

Figure 4. Layer re-randomisation Test Accuracy vs on WikiText103. See stratified analysis in Appendix F
Alpha At Initialisation & Training
The previous sections (§ 4.1 and § 4.2) showed that the of a layer has no (causal) relationship with model performance. However, it could be argued that the metric is best interpreted as a global metric averaged across all layers, as done in [31]. Therefore, we explore the average model and
its relationship to generalisation. In this section we report on the first of the two sets of SVD-based experiments described in § 3. All experiments are carried out on the ReLU FCN 5x512 model of § 4.1.
Alpha and Initialisation. In Figure 5 we explore the test accuracy of untrained models initialised with a specific model following the singular value interpolation described in § 3. The ten black dots (●) represent standard uniform initialisations of a model, i.e. ten starting spectra , that each generate 20 interpolations spectra towards (●) and another 20 interpolations towards (●). We observe that models within the ‘good’ range (the grey region in Figure 5) exhibit no performance increase versus models outside of this range (● for and ● for ) although a global alpha in this range is meant to be indicative of a good, well trained model. Figure 5 thus highlights the lack of causal relationship with test accuracy. We observe negligible correlation between an untrained model’s initial and its untrained test accuracy.
Figure 5. Test Accuracy vs Model at initialisation.
Alpha and Training. As noted in [30, 31], models that generalise do consistently have an average model , which does suggest that it is an interesting property of a model. If, as we have shown repeatedly, is not connected to generalisation, how can we explain this empirical observation? To answer this question we explore how changes throughout training in Figure 6, where the lines of the black dots (●) represent the standard initialisations training trajectories. The models initialised with between 1 and 3 fail to generalise and maintain the corresponding value throughout training. The rest of the models initialised within the good range do generalise and stay within this range throughout training. The model’s initialised with an do generalise throughout training, and witness a decrease in model alpha tending towards the ‘good’ range of in training.
Figure 6. Test Accuracy vs Model through training.
Figure 7 shows a positive correlation between the initialisation and the trained of the models. Models initialised using the standard uniform sampling (i.e. with initialisations) land in the ‘good’ range , initialisations , land beyond . It is thus likely that if most trained models have an average , it is at least in part due to the commonly used initialisation procedures. It should be noted that the correlation is not as strong for the interpolation between 1 and 0 which we attribute to the difficulty in estimating reliably [6] (the reliability of the power-law assumption is explored in depth in § K, which shows inconclusive evidence for fitting a power law to the eigenvalues and when adjusting for under estimated tail samples we observe a strong monotonic increasing relationship (Figure 21). Notably, well performing models exist inside and outside of the good alpha range at the end of training (see Figure 6).

Figure 7. Trained model vs initial model (Left). Trained model vs initial model for interpolations to (Top Right) and to (Bottom Right).
Figure 21. Trained model vs initial model (Left). Trained model vs initial model for interpolations to (Top Right) and to (Bottom Right)
These results can also be interpreted as providing additional credence to the fact that SGD performs heavy-tail regularisation on the singular values throughout training, which can also explain why we observe in well-trained models. Thus, we propose that if the average alpha of well-trained models tends to lie in the ‘good’ range it is as a consequence of our standard initialisation procedures and regularisation from SGD, not because this metric is indicative of generalisation.
Generalisation With Fixed Alpha
To further highlight that that the average alpha is not causally related to generalisation, we train models with arbitrary fixed values between , , and using the SVD-based methodology with fixed-alpha spectra described at the end of § 3. The results in Table 7 show that the models achieve no statistically meaningful difference in performance regardless of the fixed . All models, regardless of fixed , achieve similar performance. Therefore, between 2 and 6 is not causally related or required for good performance. It should be noted that the model does not achieve the same performance as in § 4.1; however, this can be attributed to the fixed singular values , which have a strong regularisation effect. Furthermore, this shows that despite models generally falling within good alpha values during training, this is not a prerequisite of generalisation and is a quirk of the current optimisation process afforded by SGD and similar variants. In § 3 we further this analysis by conducting a full ablation study of the interactions of the SVD factors during training and perform controlled interventions to understand the role the singular value spectra has on learning which further confirm the lack of causality for optimal values and generalisation.
| Fix Alpha Range | FC1 | FC2 | FC3 | FC4 | FC5 | FC6 | Test Accuracy |
Table 7. Resulting test performance of models trained with fixed Alpha () values. Mean and standard error of the mean derived from 10 trained models trained on MNIST.
Conclusion
We showed that RMT-based metrics cannot predict or explain model generalisation, nor the robust vs critical layer phenomenon, nor the re-initialisation vs re-randomisation phenomenon. We designed a range of experiments covering all aspects of the problem – individual layers § 4.1, 4.2 versus entire model § 4.3, 4.4, initialisation § 4.3, training § 4.3, 4.4, model performance § 4, eigenvalue power-law fitting § K – and a wide set of architectures, problems and scales – MNIST, ImageNet, LLMs – that all support our conclusion that
‘good model good data-free metrics’ and ‘good data-free metrics good model’
based on the fact that data-free metrics are affected in completely arbitrary ways by functionality-preserving reparametrisations of models (concretely given by the re-randomisations of robust layers). We have provided an alternative explanation for the fact that data-free metrics like tend to fall in certain ranges for well-trained models.
Uncovering or disproving causality relations is a core scientific endeavour, often with concrete engineering consequences. Our results advocate for a reappraisal of the way that we approach interpretability of neural networks. Instead of using metrics with spurious predictive capacity, we argue that there is a need for an in-depth exploration of data-free methods that can reliably disambiguate between the robust and critical layer phenomenon, i.e. display some level of invariance under functionality-preserving reparametrisations. Additionally, we advocate for a focus on methods that consider more than just weight distributions of models and instead seek metrics which further understand the more nuanced interplay between between weights and data. We argue that data-free metrics fail because they do not capture the nuanced interplay between both the data passed through the model and the weights that process it. Concretely our recommendations are as follows: 1 Avoid the use of data-free that metrics as they only have spurious relations with model performance, 2 Ensure that novel data-free metrics are able to meaningfully capture performance differences offered by re-initialisation and re-randomisation, 3 In lieu of such data-free metrics, pursue the understanding of neural networks with metrics that consider the crucial interplay between weights and data and 4 Prioritise data-based approaches for interpretability based on their ability to distinguish between re-initialisation and re-randomisation.
Acknowledgments and Disclosure of Funding
1: Calculations were performed using the Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium. 2: This research utilised Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT [18]. 3: Calculations were performed using the King’s College London HPC [19]. 4: This work was supported by UK Research and Innovation grant number EP/S023356/1, in the UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence3.
Limitations
In the main body of the paper we show across modalities, learning tasks and parameter scales that the alpha metric offered by Random Matrix Theory is not reparametrisation invariant under the re-randomisation and re-initialisation setting offered by critical layer theory. In § G, we extend our analysis to eight other data-free metrics across the same scales. While this is an extensive assessment of data-based metrics, we do not study all available data-free metrics. Given that data-free metrics are typically derived from similar matrix properties, one would expect that our findings should extend to all or most metrics built from neural network weights. However, caution would recommend that this be empirically confirmed for all data-free metrics. For the metrics we explore in the main body and in the appendix, we find no data-free metric that is reparametrisation invariant. In the paper we study two prominent modalities, vision and language, but do not explore data domains such as audio, as a result a natural extension of our experiments would be to confirm our findings in this data domain. From the results presented, we believe that the same findings would hold. In § H we show that data-based metrics are reparametrisation invariant, and thus we argue that they should be prioritised in future studies, but we only survey three data-based metrics (Activation Disagreement, Jenson-Shannon Divergence, and Linear CKA). As a result, future work could focus on which data-based metrics provide the strongest reparametrisation invariance - once again, we argue that this is more of an evaluative task and is outside of the scope of the core insights provided in this paper.
Pythia-1B Re-initialisation and Re-randomisation
Pythia-1B To conduct re-initialisation and re-randomisation experiments at large scale we use the Pythia-1B model [4] whose initialisations are publicly available. To gauge the impact of re-initialisation and re-randomisation on the model we use the same WikiText-103 dataset that was used in the main body for the GPT experiments. Table 8 presents the unmodified model accuracy on the dataset and the alpha fit of the layer weights within the training. Here, the model can be described as well-trained given that the mean value is within the optimal range at the end of training.
| Model | Params | Loss | Acc | Mean |
| Pythia | 1B | 2.576 | 47.680 | 4.801±0.202 |
Table 8. Model test performance on WikiText-103 and mean layer Alpha () value.
Table 9: The impact of re-initialisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of and test accuracy.
| Layer Type | Accuracy | Alpha () | % in Optimal Range | RSME | K- | ||
| Attention QKV | 42.187±0.896 | 29.157±4.429 | 0.000 | 0.388 | 0.103 | 3.395 | 0.283 |
| Attention Out | 42.681±1.743 | 21.945±2.703 | 0.000 | 0.174 | 0.119 | 6.541 | 0.117 |
| MLP 1 | 33.264±2.996 | 25.337±3.831 | 0.000 | -0.106 | 0.045 | 11.710 | -0.083 |
| MLP 2 | 41.093±2.729 | 28.064±4.798 | 0.000 | -0.247 | 0.563 | 7.216 | -0.200 |
Table 9.
In Table 9 we present the impact of re-initialisation of a range of layers on the Pythia-1B model. For layers Attention QKV, Attention Out, MLP1 and MLP2 the accuracy is typically preserved under re-initialisation. Alongside this accuracy preservation, we record alpha values that are far outside of the optimal range. Moreover, when analysing the correlation between values and accuracy, we see across correlation metrics that has weak predictive capacity. These results agree with the existing analysis for the MNIST ReLU FCN 5x512 architecture explored in the main body.
When considering the case of re-randomisation Table 10 shows that accuracy is preserved less than in the case of re-randomisation. Similar to the results for re-initialisation we record alpha values that are far outside of the optimal range of alpha values. Given that for both re-randomisation and re-initialisation that we record alpha values that are outside of the optimal value range, yet there exists a meaningful performance impact between these different conditions highlights that alpha is not an effective metric for disambiguating these important differences between these reparametrisations, see Figure 8 for the global relationship between layer re-initialisation and re-randomisation test accuracy against .
| Layer Type | Accuracy | Alpha () | % in Optimal Range | RSME | K- | ||
| Attention QKV | 41.581 1.193 | 33.577 6.225 | 0.000 | -0.059 | 0.003 | 4.767 | -0.067 |
| Attention Out | 42.437 1.817 | 23.361 2.651 | 0.000 | -0.671 | 0.148 | 6.710 | -0.450 |
| MLP1 | 32.241 3.140 | 29.251 4.495 | 0.000 | 0.124 | 0.010 | 12.498 | 0.050 |
| MLP2 | 40.954 2.727 | 22.579 1.961 | 0.000 | -0.238 | 0.059 | 10.58 | -0.150 |
Table 10. The impact of re-randomisation across a range of layers within the Pythia-1B model, alongside the correlation statistics of and test accuracy.
Figure 8. Pythia-1B layer re-initialisation (●) and re-randomisation (●) test accuracy vs .
Overall, the results of re-initialisation and re-randomisation at the 1B parameter scale further highlights the inadequacies of to meaningfully correspond to generalisation under functionally preserving reparametrisations.
Reparametrisation and Function Preservation
In this paper we suggest that preservation of model accuracy after layer reparametrisation (re-initialisation or re-randomisation) implies preservation of the function encoded by that model. In the main paper, we show that when comparing the original model to the re-parametrised model across a range of functional similarity and representational similarity metrics [20] accuracy is preserved when the predictive structure of the original model is also preserved.
Below we provide the corresponding functional/representational similarity analysis for the larger scale models presented in the paper. Overall, the correlation Table 11 and 12 show that functional/representational similarity holds a strong correlation with the change in accuracy observed. Furthermore, this provides strong evidence that when accuracy is preserved, the reparametrisations offered by re-randomisation and re-initialisation preserve the original function of the model.
(a) ResNet34
| Metric | RMSE | K- | ||
| Prediction Similarity | 1.000 | 0.999 | 0.005 | 0.989 |
| Jensen-Shannon Distance | -0.976 | 0.965 | 0.029 | -0.879 |
| Procrustes Distance | -0.753 | 0.722 | 0.121 | -0.659 |
| Kullback-Leibler Divergence | -0.940 | 0.857 | 0.851 | -0.814 |
Table 11. Correlation between the change in accuracy due to layer re-randomisation and the functional preservation of the model.
(b) ViT
| Metric | RMSE | K- | ||
| Prediction Similarity | 0.935 | 0.995 | 0.021 | 0.824 |
| Jensen-Shannon Distance | -0.872 | 0.866 | 0.087 | -0.752 |
| Procrustes Distance | -0.923 | 0.858 | 0.099 | -0.771 |
| Kullback-Leibler Divergence | -0.877 | 0.970 | 0.342 | -0.759 |
Table 12.
(c) GPT2
| Metric | RMSE | K- | ||
| Prediction Similarity | 0.747 | 0.978 | 0.030 | 0.579 |
Table 19.
(d) GPT2-Large
| Metric | RMSE | K- | ||
| Prediction Similarity | 0.842 | 0.949 | 0.030 | 0.655 |
Table 20.
| Metric | RMSE | K- | ||
| Prediction Sim | 0.991 | 0.968 | 0.036 | 0.933 |
Table 21. Table 12: Correlation between the change in accuracy due to re-initialisation and re-randomisation and the functional preservation of Pythia-1B.
Initialisation Scheme and Alpha (α)
In the main body of the paper, we use the default initialisation scheme offered PyTorch. In these results and Figure 2a in particular, we observe that there is a non-negligible probability of a randomly initialised layer having an value in the ‘good’ range [2, 6]. In this section, we identify the probability of such ranges emerging at initialisation across a range of different layer types, namely: Linear Layer 512x512, Resnet34 Layer2 Conv_2 (Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) and ViT Base 16 Out Projection.
Additionally, as many different initialisation schemes are offered by PyTorch such as Xavier (normal (), uniform ()) [13] and Kamming (normal (), uniform ())[16] we have reproduced the analysis in each of the following tables to also capture these initialisations. The following tables present the descriptive statistics of each initialisation type and their relationships with alpha for the layers named above.
| Initialisation Scheme | Skew | Kurtosis | Min | Max | % in Optimal Range | |||
| PyTorch default | 10.985 | 9.019 | 6.390 | 2.088 | 5.992 | 3.500 | 56.640 | 16.640 |
| 11.112 | 8.993 | 6.587 | 2.017 | 5.360 | 3.398 | 64.359 | 16.910 | |
| 10.960 | 9.001 | 6.448 | 2.083 | 5.938 | 3.446 | 63.332 | 17.510 | |
| 10.877 | 8.948 | 6.340 | 2.191 | 6.950 | 3.353 | 68.180 | 16.700 | |
| 10.993 | 9.040 | 6.524 | 2.216 | 7.038 | 3.403 | 68.135 | 17.120 |
Table 22. Table 13: Distribution statistics of across initialisation schemes and layer types. (a) Linear Layer 512 × 512.
| Initialisation Scheme | Skew | Kurtosis | Min | Max | % in Optimal Range | |||
| PyTorch default | 18.126 | 13.518 | 14.302 | 2.835 | 14.455 | 3.594 | 211.387 | 5.990 |
| 18.341 | 13.940 | 14.146 | 2.748 | 14.229 | 3.494 | 191.422 | 6.210 | |
| 17.729 | 13.540 | 13.747 | 3.585 | 32.047 | 3.405 | 288.763 | 6.510 | |
| 18.186 | 13.791 | 13.986 | 2.790 | 15.779 | 3.367 | 221.806 | 6.230 | |
| 17.919 | 13.750 | 13.271 | 2.390 | 9.203 | 3.583 | 152.902 | 6.350 |
Table 23. (b) ResNet34 Layer2 Conv_2 (Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)).
(c) ViT Base 16 Out Projection (Linear Layer 768 × 768).
| Initialisation Scheme | Skew | Kurtosis | Min | Max | % in Optimal Range | |||
| PyTorch default | 13.002 | 10.587 | 7.953 | 2.183 | 6.524 | 3.873 | 76.666 | 8.020 |
| 13.197 | 10.667 | 8.143 | 2.161 | 6.974 | 3.836 | 101.310 | 8.310 | |
| 12.977 | 10.531 | 7.969 | 2.200 | 6.844 | 3.990 | 74.792 | 8.190 | |
| 13.183 | 10.664 | 8.071 | 2.099 | 5.917 | 4.001 | 68.743 | 7.840 | |
| 13.185 | 10.618 | 8.204 | 2.191 | 6.667 | 3.857 | 88.848 | 7.740 |
Table 13.
Table 13 presents summary statistics for across different initialisation schemes for different layer types of models. These results complement that of those presented in Figure 2a by showing that initialised layers can fall within optimal value range of with a somewhat large non-zero probability. Given that HTSR argues that such values should only emerge at the end of training and quantify a good layer, this evidence further undermines the ability of to effectively capture generalisation in neural networks.
Alpha () Threshold-Prediction Tests
The metric can be considered a threshold based classification metric, where denotes a ‘good’ layer or model with high accuracy and or is considered a ‘bad’ layer or model. With this perspective we analyse the performance of of predicting a good layer, after reparametrisation offered by re-reinitialisation and re-randomisation. A ‘good’ layer is considered to be a layer that results in the model being within 10% of the original test accuracy after the reparametrisation and has an within . In Table 14 we report the global (for all layers) AUC and the F1-score, Precision, True Positive Rate (TPR), False Positive Rate (FPR) of predicting a good layer; to clarify, a good layer corresponds to the resulting model being within 10% of the original test accuracy after the reparametrisation. For the Pythia 1B model, we are not able to provide the AUC score nor the true positive rate score as none of the reparametrisations resulted in layers within the 2 and 6 alpha range.
| Model | AUC | F1-Score | Precision | True Positive Rate | False Positive Rate |
| MLP | 0.520 | 0.242 | 0.167 | 0.439 | 0.387 |
| ResNet34 | 0.449 | 0.000 | 0.000 | 0.000 | 0.020 |
| ViT | 0.686 | 0.056 | 0.029 | 1.000 | 0.836 |
| GPT2 | 0.442 | 0.022 | 0.011 | 0.714 | 0.901 |
| GPT2-Large | 0.603 | 0.010 | 0.005 | 0.875 | 0.973 |
| Pythia-1B | nan | 0.000 | 0.000 | nan | 0.797 |
Table 14. Effectiveness of at predicting a good layer where model performance is within 10% of the original test accuracy after layer reparametrisation.
For all models evaluated, achieves a poor AUC. For the ResNet34 and GPT2, this is worse than random; the MLP is slightly better than random, and for the GPT-Large and ViT it is better than random; however, still poor. Furthermore, the F1-Score and Precision are very low, with the highest F1-score and Precision of 0.242 and 0.167, respectively, in the MLP case. With regard to the true positive rate (TPR) and false positive rate (FPR) of the optimal range, a high TPR corresponds with a high FPR, showing the metric is poor at effectively identifying good layers. In the case of ResNet, the metric fails to identify any of the layers that correspond to a model maintaining performance, whilst flagging a small number that do not maintain performance as good layers. Overall, these results strongly corroborate the main findings in the paper.
Alpha () Analysis Stratified Over Layer Type
Table 15, presents the stratified analysis of the model layers by type, for the ResNet34 and ViT on ImageNet and the GPT2 and GPT2-Large on WikiText103. In the tables there is no consistent trend across the layer types, with some layers reporting a positive relationship between test accuracy and and others a negative relationship. Which strongly supports the main findings in the paper that values are not a reliable indicator of performance and generalisation.
| Layer Type | Accuracy | Alpha () | % in Optimal Range | RMSE | K- | ||
| Input Conv2d | 0.395 ± 0.029 | 14.381 ± 1.817 | 0.000 | 0.176 | 0.008 | 0.091 | 0.156 |
| Conv2d_1 | 2.461 ± 0.260 | 22.628 ± 1.525 | 3.750 | -0.178 | 0.033 | 3.232 | -0.117 |
| Conv2d_2 | 36.35 ± 1.610 | 23.882 ± 1.637 | 3.750 | -0.140 | 0.052 | 19.823 | -0.096 |
| Downsample | 15.793 ± 2.474 | 6.788 ± 0.604 | 43.333 | 0.634 | 0.335 | 11.052 | 0.456 |
| Output FC | 0.098 ± 0.009 | 11.829 ± 2.095 | 0.000 | 0.492 | 0.036 | 0.029 | 0.405 |
Table 15. Stratified layer re-randomisation Test Accuracy vs . (a) ResNet34 on ImageNet
| Layer Type | Accuracy | Alpha () | % in Optimal Range | RMSE | K- | ||
| patch_embed.proj | 0.145 ± 0.008 | 65.281 ± 4.607 | 0.000 | 0.467 | 0.241 | 0.021 | 0.422 |
| attn.qkv | 77.808 ± 0.071 | 17.978 ± 1.218 | 1.667 | 0.077 | 0.019 | 0.769 | 0.058 |
| attn.proj | 77.469 ± 0.223 | 13.345 ± 0.725 | 6.667 | -0.075 | 0.004 | 2.442 | -0.049 |
| mlp.fc1 | 52.805 ± 2.806 | 16.678 ± 0.911 | 0.000 | 0.239 | 0.032 | 30.238 | 0.162 |
| mlp.fc2 | 71.847 ± 1.902 | 20.088 ± 1.157 | 1.667 | -0.055 | 0.001 | 20.824 | -0.036 |
| head | 0.128 ± 0.013 | 14.194 ± 3.329 | 0.000 | -0.219 | 0.014 | 0.042 | -0.135 |
Table 15b. (b) ViT on ImageNet
| Layer Type | Accuracy | Alpha () | % in Optimal Range | RMSE | K- | ||
| wpe | 7.336 ± 0.030 | 15.272 ± 2.730 | 0.000 | -0.127 | 0.148 | 0.087 | -0.067 |
| attn.c_attn | 36.472 ± 0.355 | 16.644 ± 0.926 | 0.000 | -0.049 | 0.002 | 3.887 | -0.036 |
| attn.c_proj | 35.382 ± 0.739 | 11.705 ± 0.528 | 5.833 | 0.077 | 0.003 | 8.080 | 0.058 |
| mlp.c_fc | 34.874 ± 0.845 | 18.075 ± 1.039 | 0.000 | -0.120 | 0.003 | 9.237 | -0.079 |
| mlp.c_proj | 35.430 ± 0.777 | 18.240 ± 0.859 | 0.000 | 0.146 | 0.000 | 8.512 | 0.098 |
Table 15c. (c) GPT2 on WikiText103
| Layer Type | Accuracy | Alpha () | % in Optimal Range | RMSE | K- | ||
| wpe | 8.905 ± 0.072 | 12.303 ± 1.625 | 10.000 | -0.818 | 0.734 | 0.117 | -0.644 |
| attn.c_attn | 43.581 ± 0.012 | 22.934 ± 0.838 | 0.000 | -0.005 | 0.007 | 0.222 | -0.002 |
| attn.c_proj | 43.041 ± 0.203 | 16.266 ± 0.556 | 1.944 | 0.002 | 0.000 | 3.856 | 0.003 |
| mlp.c_fc | 41.817 ± 0.355 | 24.427 ± 0.888 | 0.000 | 0.059 | 0.005 | 6.724 | 0.038 |
| mlp.c_proj | 42.413 ± 0.348 | 25.964 ± 0.999 | 0.000 | -0.058 | 0.000 | 6.599 | -0.039 |
Table 15d. (d) GPT2-Large on WikiText103
Further Analysis on Data-Free Metrics
In this section we extend our analysis to the following data-free metrics under the existing experimental setup. represents the weight matrix of the layer and is the empirical spectral density of the correlation matrix, , such that , where are the eigenvalues of .
Alpha Weighted (): , where is the max eigenvalue from [30].
Log Alpha Norm: , where , where is the rank of [30].
MP Soft Rank: is the ratio between the bulk edge of the , , and the max eigenvalue, , [30].
Spectral Norm: The max singular value of denoted as .
Stable Rank: The ratio of the squared Frobenius Norm and the squared Spectral Norm, denoted as [37].
Number of Power Law Spikes: The number of eigenvalues that are outside the bulk region of the ESD, when fit to an Marchenko-Pastur distribution [30].
Entropy: , where is the rank of matrix and is where is the singular values of [30].
Frobenius Norm: The sum of the squared singular values of denoted as .
Each metric can be found below with the appropriate subsection that corresponds to our analysis of these data-free metrics.
The correlation between the other data-free metrics formalised in Section G and the associated test accuracy when layers undergo re-initialisation (🔵) or re-randomisation (🟠) is shown in Figure 9. For all metrics explored we observe approximately zero correlation between the metric values and the test accuracy, showing that other data-free metrics align with the findings presented for the alpha metric presented in the main body of the paper. In § G.2,§ G.3 we show this holds for ImageNet models as well as for language models in § G.4 and G.5.
Small Scale Analysis of Data-Free Metrics Under Re-Initialisation and Re-Randomisation
Norm-based metrics were originally shown to be too coarse a metric to measure the generalisability of the neural networks in [48]. Figure 9 strengthens these findings, highlighting that there is essentially no correlation between the a range of data-free metrics and the test accuracy of a model under re-randomisation and re-initialisation.
ResNet34 on ImageNet
In Figure 10, when using a range of data-free metrics we observe the same trend as found with in the main body of the paper. Overall for the ResNet43 on ImageNet data-free metrics fail to predict the performance difference that is derived from re-randomisation across layers. While some correlations are higher than circa 0, i.e. (Spectral Norm) it can be clearly observed that there is strong overlap between metric values for the layers that have good accuracy and poor accuracy through re-randomisation, showing that the artifact of slightly higher correlation shows no meaningful insight for these metrics.
Figure 10. ResNet 34: Layer re-randomisation test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and K- is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
ViT on ImageNet
To confirm trends for the ImageNet dataset on the ViT Figure 11 shows the relationship of data-free metrics value distributions under re-randomisations. Once, again in this figure we find that for all data-free metrics that models can generalise and perform poorly while expressing the same metric values under re-randomisation. As a result the values have no meaningful prediction of performance. This trend is in line with the observations we found with in the main body of the paper. Here, we do find weak correlations for MP soft rank, spectral norm, stable rank, entropy, number of power law spikes and Frobenius, however, there is significant overlap between the metric values and associated test accuracy (i.e. for strong accuracy and random accuracy). Furthermore for all metrics the value is close to zero indicating that the regression model has little to no predictive power. Overall, data-free metrics can be considered a weak predictor of model performance.
Figure 11. ViT: Layer re-randomisation test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and - is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
GPT2 on WikiText103
To understand if there is generality in our findings of data-based metrics across modalities and parameters scales we observe the trends of performance and data-free metrics under re-randomisation for the GPT2 model. In Figure 12 we largely find a consistent trend that values for data-free metrics can simultaneously represent high and low performing models. This shows that analysing particular values for these data-free metrics provides little to no insight on performance. Here for all metrics there is a very weak correlation between metric values and performance. These trends align with those found with in the main body of the paper.
Figure 12. GPT2: Layer re-randomisation test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and K- is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
GPT2-Large on WikiText103
In Figure 13, we explore the relationship of data-free metrics and accuracy under re-randomisation for the GPT-2 Large model. Here the trends mirror those for the GPT2 and the results with alpha presented in the main body. For the data-free metrics surveyed there is a moderate to weak correlation between the metric and the test accuracy of the model. However, in spite of this, there is no predictive capacity of the metrics when described with linear regression as shown with the close to zero. This can be attribute to overlapping values that correspond to high and low test accuracy. For the data-free metrics surveyed in this paper across parameter ranges in the GPT2 family we find no predictive capacity for the data-free metrics under re-randomisation.
Figure 13. GPT2-Large: Layer re-randomisation test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Pythia-1B on WikiText103
In Figure 14, we explore the relationship of data-free metrics and accuracy under re-initialisation (•) and re-randomisation (•) for the Pythia-1B model. Here the trends mirror those for the GPT2 models and the results with alpha presented in the main body. For the data-free metrics surveyed there is a moderate to weak correlation between the metric and the test accuracy of the model. However, in spite of this, there is no predictive capacity of the metrics when described with linear regression as shown with the close to zero. Which is due to overlapping values that correspond to high and low test accuracy. For the data-free metrics surveyed we find no predicative capacity for said metrics under re-initialisation and re-randomisation.
Figure 14. Pythia-1B: Layer re-initialisation (•) and re-randomisation (•) test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and - is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
Data-Based Metrics
To compare data-free and data-based metrics we explore if two widely used data-based metrics can distinguish the output of a re-initialised layer from that of a re-randomised layer. The outputs of the original and modified layers are represented as and , respectively. The model layer outputs are collected by passing through the test dataset, . The metrics explored are defined as follows, Linear CKA [21] is explored in Appendix I):
Activation Disagreement: The mean percentage of times that the neurons from and fail to agree to activate on .
Jensen–Shannon (JS) Divergence: The mean weighted average of the Kullback–Leibler (KL) Divergence of the softmax( on ) compared to the softmax( on ) [25].
When considering the results of the activation disagreement and JS divergence between the original model layer and each model with re-initialised and re-randomised layers in Figure 15, it becomes clear that there is a stark difference in similarity between the re-initialised and re-randomised layers compared to the unmodified layer. The figure demonstrates that the difference between re-initialisation and re-randomisation can be explained by the amount the layer disagrees on activation compared to the original non-modified layer. It is evident that when there is less disagreement (re-initialisation), the model can maintain accuracy, while a lot of disagreement (re-randomisation) leads to a drop in accuracy. This observation is further supported by a mean Spearman correlation coefficient and Kendall’s tau measure across layers of -0.776 and -0.5375, respectively for Activation Disagreement, indicating a negative relationship between activation disagreement and accuracy.
Figure 15. Layer re-initialisation (●), re-randomisation (●) and signs of re-initialisation with magnitudes of re-randomisation (●) test accuracy vs the respective metric.
Connection to the Lottery Ticket Hypothesis: For our MNIST experiments, both re-initialisation and re-randomisation are drawn from the Uniform distribution, where is . However, a layer can reduce to random accuracy under re-randomisation but not under re-initialisation, which suggests that an important factor for performance retention under re-initialisation is the signs of the weights. The signs of the weights, especially within networks that use the ReLU activation, essentially represent the pathway, or subnetworks within the model. The Lottery Ticket Hypothesis (LTH) suggests that at random initialisation, there exist subnetworks that, when trained, reaches the test accuracy of the original model that is trained with all the parameters and same compute budget [12]. To explore the role of the signs, we take the re-randomised values for a layer and apply the signs of the re-initialised layer to the re-randomised weights. We explore this using the data-based metrics that can disambiguate between re-initialised and re-randomised. We find in Figure 15 that when applying the signs of the re-initialisation to the magnitudes of the re-randomised layer (●), this layer is closer to the original layer as observed with a reduction in difference, and that there is an increase in performance when compared to the re-randomised layer (●). The results strongly suggest that for this network, the subnetworks defined at the start of training through initialisation do not dramatically change through training, providing credence to the LTH. Furthermore, this demonstrates the efficacy of data-based metrics to disambiguate between the subnetwork importance of layers in neural networks.
Linear CKA (Data-Based Metric) Analysis
In this section, we explore an additional data based metric, Linear CKA [21]. We asses the predictive power of Linear CKA to distinguish between the output of a re-initialised layer from that of a re-randomised layer. Overall we find that in line with other data-based metrics explored in the main paper that Linear CKA can disambiguate between a re-initialised and re-randomised layer. However, Linear CKA has less predictive power than the Activation Disagreement and JS-divergence metrics explored in the body of the paper, as demonstrated with lower correlation values and higher RMSE, see Figure 16. The results highlight an important caveat when considering data-based metrics, not all data-based metrics can be considered equal for predictive capacity. As a result, it remains an open challenge to find test beds for identifying and utilising the correct data-based metrics that effectively consider subnetwork relationships and disambiguate between robust and critical layers.
Figure 16. Layer re-initialisation (●) and re-randomisation (●) test accuracy vs the respective metric. is the Spearman correlation coefficient, is the root mean square error of the linear regression (red line), and K- is the Kendall’s tau measure, all with respect to the relationship between test accuracy and the metric.
SVD Ablation for Learning Dynamics
The empirical experiments in this paper have provided substantial evidence that data-free metrics may bear no relationship to the generalisation properties of neural networks. However, it can be argued that the spectral properties of weight matrices need to to change in order for a model to generalise, particularly since singular values do change substantially in standard training setups. In Figure 17 we show how the singular values in an MNIST model change across epochs.
Figure 17. The first 10 singular values for each layer of a 5 hidden layer neural network with width 512 throughout training on MNIST. The viridis colormap is used to delineate epochs. Dark-purple represents initialisation and the last epoch, epoch 30, is in bright-yellow. The mean (line) is reported with the standard error of the mean (error bars) from 10 runs.
Given, that the singular values values do change substantially through training, a plausible interpretation of these metrics is that they record properties of the neural network that are important in the training process that results in a well generalising model.
Motivated by this interpretation, we conduct experiments to establish the role of the spectral properties of the weight matrices in well generalising models. The following experiments explore how the factors in the SVD decomposition of weight matrices (i.e. the singular vectors and the singular values) affect training. We explore each factor independently by keeping the others fixed during training, and only letting the corresponding factor update throughout training - to isolate a single factors effect. This enables a systematic intervention analysis to understand what factors primarily drive generalisation and which ones require interactions. The factors are also explored jointly to understand the role of the interaction of factors on generalisation. Finally, the factors are all trained together to create an effective baseline of complete interaction.
If the model is unable to generalise without the singular values – and thus spectral properties – changing, this would support the interpretation that these metrics record property changes that are required for models to reach a minima that generalises well. Conversely, if a model can generalise without these properties changing, this would mean that these metrics are not casually relevant to finding a minima that generalises and thus has limited impact on generalisation in training.
Experimental Setup
To reduce confounding factors, we consider a linear layer where the weight matrices are parametrised via singular value decomposition, , to from three layers where and are orthogonal, and is diagonal with non-negative entries.
This parametrization allows for the orthogonal direction factors and and the scaling factor of the weight matrix to be manipulated and changed independently. Thereby enabling a controlled analysis of their respective roles in the optimization process and generalisation.
For all experiments we use a 5 hidden layer MLP without biases with the ReLU activation and a hidden width of 512 on the MNIST dataset with cross entropy loss, SGD with a learning rate of 0.001 and momentum of 0.9, a batch size of 128 and trained for 30 epochs. To isolate the effect of parameter interventions, the training data order, random seeds and initialisation procedures (expect where explicitly varied) are held fixed across all runs. Model biases are not such that the factors are the only aspect impacting model performance.
To ensure that and maintained orthogonality throughout training we employed the standard PyTorch methods of matrix exponential for square matrices and the householder transformation for rectangular matrices4. This ensures that changes in the performance cannot be attributed to the loss of orthogonality.
Using random drawn from the Marchenko–Pastur distribution introduces implicit conditioning on the orthogonal directions of and . Therefore is initialised as , where is the identity matrix. This removes the scaling effects at initialisation and ensures that all singular values are treated equally by the optimiser. As a result, any observed differences in training dynamics can be attributed to the orthogonal directions of and or to the values of rather than the co-founding effects that could arise from the initial unequal spectral scaling. Furthermore initialising with fixes the spectral properties across runs, facilitating a controlled comparison of the effects of the spectral properties between interventions.
Interventional Ablation Over Parametrisation Subsets
To study the interactions of the SVD factors during training, we perform controlled interventions, in which gradient updates are enabled on specific subsets of parametrisations, while the remaining parametrisations are held fixed with the initial values - allowing for a controlled and isolated optimisation setup.
For each experiment, a subset is assigned as a trainable factor. All factors in the complement are frozen and do not change throughout training.
We consider the following intervention sets:
Single-Factor Training
Two-Factor Training
Full-Factor Training
Single Factor Training: The single factor training experiments ( and interventions) allow for understanding of the role of the of each component in the learning process. The performance in these conditions reveals whether training progress can be achieved through spectral scaling () or through rotations of the singular vectors ( and ) alone. Failure of a factor to effectively generalise indicates that learning relies on the interaction between that factor and others. Whereas the success of a factor implies that the corresponding factor is a primary driver of the learning dynamics.
Intervention: To isolate the effects of the orthogonal directions during training, is fixed as and is fixed as , where is orthogonal matrix, throughout training. For each run is initialised with a random orthogonal matrix and allowed to train. Keeping fixed as throughout runs isolates the effects of variation in performance ensuring it cannot be attributed to changes in the initial conditions of and .
Intervention: To isolate the effects of the during training, is initialised as the orthogonal matrices and are randomly initialised as orthogonal matrices and then held fixed throughout training. Between runs the initial values of and change to ensure that the results are not conditional on a single orthogonal direction.
Intervention: To isolate the effects of the orthogonal directions during training, is fixed as and is fixed as , where is orthogonal matrix, throughout training. For each run is initialised with a random orthogonal matrix and allowed to train. Keeping fixed as throughout runs isolates the effects of run variation in performance ensuring it cannot be attributed to changes in the initial conditions of and .
Two Factor Training: The two factor training experiments (Joint , and interventions) allow for understanding the interactions between components and how this affects the training process and subsequent generalisation properties. Improvements in performance above the single factor cases, suggests that the interaction of the two factors is important to overall performance. Although if performance is comparable to full training this would suggests that the frozen factor does not significantly impede the training process.
Joint Intervention: To isolate the extent to which performance improvements arise from the from the combined changes in and . The parametrisation jointly optimises and while keeping fixed as throughout training.
Joint Intervention: To isolate the extent to which performance improvements arise from the from the combined changes in and . The parametrisation jointly optimises and while keeping fixed as throughout training.
Joint Intervention: To isolate the extent to which performance improvements arise from the from the combined changes in and . The parametrisation jointly optimises and while keeping fixed throughout training.
Full Factor Training : In the full factor training all components (, and ) are jointly optimised. This corresponds to standard training on the weight matrix , and thus serves as a reference for the impact of the single and two-factor interventions on the performance.
Table 16 summaries all the interventional conditions explored in this experiment. For this experiment instead of using 100 runs, 10 runs are used as the orthogonal constraint on the and components significantly increased training times. The test accuracy, loss and metrics are recorded at initialisation and throughout training at the end of every epoch.
| Trainable | Trainable | Trainable | Description |
| Output Alignment Only | |||
| Spectral Scaling Only | |||
| Input Alignment Only | |||
| Joint and () | |||
| Joint and () | |||
| Joint and () | |||
| Standard Training () |
Table 16. Summary of factor interventional conditions.
Experimental Results
The following experimental results cover the relationship between the factors and test performance (§ J.2.1) and data-free metrics (§ J.2.2) throughout training.
Factor Relationship with Model Performance
Figure 18 compares the performance throughout training when different subsets of the models parameters are updated during training. Training all components jointly, , as is done conventionally through optimisation directly on the weight matrix , results in the model with the highest test accuracy at the end of training of 97.499%.
Figure 18. Model test performance throughout training on MNIST. (left) accuracy and (right) loss. The dotted line (. . . ) is the train performance and the solid line (–) is the test performance, the ◦ represents a epoch. Each curve corresponds to training only one or more components of the parametrisation: , , , , , and . All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (hue) are reported from 10 runs.
Training only a single component ( or ) results in consistently lower performance than . The parametrisation reaches a high initial test accuracy after 1 epoch training (83.811%) and then continues to improve in performance achieving a test accuracy of 93.236% at the end of training. In contrast, updating only results in a markedly slower initial improvement with slow but steady performance gains, yet remains significantly below the performance levels achieved by all other parametrisations throughout training resulting in a test accuracy of 37.246%. As found with only updating updating only reaches a high initial test accuracy after 1 epoch training (88.628%) a 4.817% increase in test accuracy over parametrisation. The parametrisation then steadily increases in performance reaching a test accuracy of 94.924% a 1.688% improvement over the parametrisation.
Jointly updating pairs of components (, and ) improves the resulting test performance relative to the single-components ( and ). Both and parametrisations follow the similar initial training trajectories as and respectively. After 1 epoch of training the parametrisation achieves a test accuracy of 85.506% a 1.695% improvement over the parametrisation. The parametrisation subsequently makes increased performance gains throughout training resulting in a test accuracy of 96.458%, only 1.041% of training with all components jointly (). Additionally after 1 epoch training the parametrisation achieves a test accuracy of 88.898% a 0.270% improvement over the parametrisation and 1.168% below parametrisation. The parametrisation subsequently makes increased performance gains throughout training and closely matches the training trajectory of , resulting in a test accuracy of 97.373%, only 0.126% of training with all components jointly (). The parametrisation, closely matches the training trajectory of with the initial epoch of training reaching a test performance of 89.244% a 0.616% improvement over the parametrisation and a 5.433% improvement over the parametrisation. The parametrisation continues to closely match the parametrisation with a resulting test performance of 95.524% a 0.600% improvement over the parametrisation.
In summary, Figure 18, shows that the configurations exhibit a consistent ordering in test performance with performing the best, followed by , , , and . With the difference between and of 55.990% compared to a difference between and of 4.263%. Although was unable to achieve comparable or close performance to the inclusion of to and parametrisations in and enabled improvements in test performance over and of 3.222% and 2.449% respectively. The next section explores the relationship of data-free metrics throughout training to better understand how our intervention setup effects matrix (and data-free metric) properties during training and their relationship to accuracy.
Metric Relationship with Test Accuracy

Figure 19. Model test performance throughout training on MNIST against data-free metrics. (left) test accuracy and (right) test loss. The ◦ represents a epoch. The colour intensity displays training progress, lighter points correspond to earlier epochs with darker points corresponding to later epochs. Each curve corresponds to training only one or more components of the parametrisation: , , , , , and . All other parameters are held fixed during training. Performance is evaluated on the at the end of each epoch. The mean (line) and standard error from the mean (error bars) are reported from 10 runs.
To understand the role of each respective metric in the training process we explore the each data-free metric’s relationship with the test accuracy. Specifically focusing on how training with different model components effects the relationship each metric has with test accuracy throughout training.
Alpha (): fig:19a compares the performance throughout training with the global Alpha when different subsets of the models parameters are updated during training. For parametrisations , and the metric stays approximately around . This behaviour is expected as the singular values are constrained to 1. The slight deviation from is introduction by the imperfect orthogonal constraints applied throughout training which results in slight negligible variations in when the is combined to calculate the metric.
The parametrisations that utilise exhibit different behaviour to , and . The parametrisation achieves an value of 249.449 after the 1 epoch training, the value continues to decrease throughout training while steadily improving in test performance, reaching a global value of 34.485 which a corresponding test accuracy of 97.499%. The parametrisation reaches an value of 259.177 within the first epoch of training correspond to a test accuracy of 10.542%. The parametrisation value subsequently decreases, throughout training while improving in test performance reaching a value of 10.168 with a corresponding test accuracy of 37.246%. The introduction of to and with and result in the global values quickly deviating from the initial value after 1 epoch of training reaching an value of 157.390 and 259.177 and test accuracies of 85.506% and 88.898% respectively. The values continue to decrease throughout training reaching 26.220 and 33.338 while coinciding with a test accuracy of 96.458% and 97.373% for the and parametrisations respectively.
Overall, the Figure shows that optimal metric values do not correlate with the resulting test performance of the model, as demonstrated with , and parametrisations achieving high accuracy while maintaining a large value of and the parametrisation achieving a low metric of 10.168 with a test accuracy of 37.246%.
Alpha Weighted (): fig:19b compares the performance throughout training with the global Alpha Weighted when different subsets of the models parameters are updated during training. The Figure largely shows the same relationship between the parametrisations , , and and the metric observed in fig:19a. With the , and parametrisations value staying relatively fixed throughout training.
The parametrisation achieves an value of 2.359 after 1 epoch training, which then slowly increases as training progresses to 2.854 at the end of training. The and parametrisations follow a similar behaviour with a initial value of 3.338 and 3.975 after 1 epoch training respectively. With subsequent training the value tends towards slowly increasing resulting in an values of 4.851 and 4.602 for the and parametrisations respectively. In contrast the parametrisation initially reaches an of 3.913 after 1 epoch of training and then subsequently fluctuates between increasing and decreasing values throughout training resulting in an of 5.683 at the end of training.
The data in this section shows that there is not a strong correlation between the metric throughout training and the resulting test performance of the model, as shown with the large test performance difference witnessed regardless of the recorded value.
Entropy: fig:19c compares the performance throughout training with the global entropy when different subsets of the models parameters are updated during training. The , and parametrisations remain approximately constant throughout training while performance improves. The parametrisation approximately linearly decreases in entropy throughout training, with an initial entropy of 1.021 and a end value of 1.002, marking a 0.019 decrease in entropy. The , and parametrisations also decrease in entropy throughout training, however the deviation from initialisation is a lot smaller, with a decrease in entropy of 0.002, 0.001 and 0.001 for each parametrisation respectively. These findings establish that there is not a consistent correlation between model entropy and test performance that exists across parametrisations.
Lambda Max (): fig:19d compares the performance throughout training with the global when different subsets of the models parameters are updated during training. The , and parametrisations remain approximately constant with a value of 1 throughout training while performance improves. The parametrisation steadily increases the value from 1.022 to 2.460 at the end of training. The initial increase in is gradual as demonstrated with the change in from epoch 1 to epoch 10 being 0.236, however later in training the increase in is more substantial with the last 10 epochs resulting in a difference of 0.776.
The inclusion of the parametrisation to via the parametrisation, results in the value exhibiting the opposite behaviour to the parametrisation with a large initial increase in at the start of training which then slows down as training progresses. Albeit the difference of the trajectory results in a similar value of 2.498 at the end of training. The inclusion of of the parametrisation to via the parametrisation, changes the training trajectory of . The training trajectory closely mimics the training trajectory of parametrisation however the change of between epochs is consistently smaller, resulting in a value of 2.248 at the end of training. The parametrisation training trajectory closely mimics the parametrisation. Starting with a small initial increase in which then gradually increases through training resulting in a value of 2.093 at the end of training. These results show that although can correlate with performance increases, see , and parametrisations, it is not required to change for a model to learn a generalisable solution, see , and parametrisations.
Log Alpha Norm: fig:19e compares the performance throughout training with the global Log Alpha Norm when different subsets of the models parameters are updated during training. The Figure largely shows the same relationship between the parametrisations , , , and and the metric observed in fig:19b. The , and parametrisations Log Alpha Norm values vary significantly throughout training. This is expected, as Log Alpha Norm is defined as , where , (see Section G) and therefore inherits the variability from the calculation of the metric. The parametrisation Log Alpha Norm value quickly decreases after 1 epoch training going from a value of 14.527 to 3.012. The Log Alpha Norm value then slowly increases throughout training, with small changes between epochs, reaching a value of 3.573. The , , and parametrisations exhibit similar behaviour to the parametrisation however the initial decrease is lower going from 8.84 and 5.636 to 4.131 and 5.134 respectively. The parametrisations Log Alpha Norm then subsequently gradually increases throughout training as observed with the parametrisation. fig:19e shows that while the Log Alpha Norm can appear to correlate with improvements in test performance throughout training, this correlation is not consistently observed across all parametrisations namely, , and .
Frobenius Norm: fig:19f compares the performance throughout training with the global Frobenius Norm when different subsets of the models parameters are updated during training.
The , and parametrisations improved in test performance throughout training while the global Frobenius Norm stayed approximately constant throughout training, reflecting that the singular values are held fixed at 1 throughout training.
The parametrisation global Frobenius Norm initially decreases early in training reaching its lowest value at epoch 5 of 427.618 a 0.726 decrease from the initial value before training while corresponding to an improvement in test accuracy of 2.395%. The global Frobenius Norm subsequently increases with every epoch passing the initial global Frobenius Norm at epoch 12, with a test accuracy of 19.828% marking a 9.649% improvement from initialisation. The resultant global Frobenius Norm at the end of training is 442.849 corresponding with a test accuracy of 37.246%.
The , and parametrisations global Frobenius Norm follow a similar behaviour as observed with the Lambda Max metric in fig:19d. The has a large initial increase in the global Frobenius Norm value which then gradually increases throughout training to a value of 478.707 with a test accuracy of 96.458%. The parametrisation follows a similar training trajectory as the parametrisation, however makes a larger initial increase during epoch 1 to a global Frobenius Norm of 432.480 in comparison reaches a value of 432.068 which is 0.412 smaller. While the parametrisation reaches a test accuracy of 90.066% whereas the parametrisation reaches a test accuracy of 88.898% marking a 1.168% difference in test accuracy. Both the and parametrisations steadily increase the global Frobenius Norm throughout training, with reaching a higher value of 467.382 at the end of training marking a 0.892% improvement over the parametrisation. While the parametrisation reaches a test accuracy of 97.499% marking a 0.126% improvement over the parametrisation at the end of training.
In summary fig:19f shows that for a model to generalise and achieve good test accuracy the Frobenius norm of the layer weights is not required to change, see factors , and . Furthermore enabling the Frobenius norm to change has limited impact on improving the overall performance of the model throughout training.
Spectral Norm: fig:19g compares the performance throughout training with the global Spectral Norm when different subsets of the models parameters are updated during training. The training trajectories of the global Spectral Norm closely matches the behaviour observed with Lambda Max () in fig:19d. The , and parametrisations global Spectral Norm stays relatively fixed throughout training. The parametrisation global Spectral Norm gradually increases through training reaching the highest value while achieving the lowest test accuracy. The , and parametrisations observe an initial large increase in the global Spectral Norm which then continues to gradually increase with reaching the highest value. The and parametrisations reach approximately the same global Spectral Norm value.
Number of Power Law Spikes: fig:19h compares the performance throughout training with the global Number of Power Law Spikes when different subsets of the models parameters are updated during training. As observed with the , and Log Alpha Norm metrics in Figures 19a, 19b and 19e respectively the , and parametrisations improved in test performance throughout training while the global Number of Power Law Spikes stayed relatively fixed throughout training, with the variation introduced by fitting a power law to singular values are all approximately 1 throughout training.
In contrast to the , and Log Alpha Norm metrics the global Number of Power Law Spikes increases from the initial value for the , and parametrisations.
The parametrisation global Number of Power Law Spikes subsequently fluctuates throughout training with alternating between increasing and decreasing within training epochs. Eventually resulting in a Number of Power Law Spikes of 130.717 with a test accuracy of 37.246%.
The parametrisation first epoch of training results in the largest increase in the global Number of Power Law Spikes from 90.750 to 160.667, corresponding to a test accuracy of 85.506%. In the subsequent training epochs the global Number of Power Law Spikes oscillates with a variance of 51.772 between 155.133 and 182.483, resulting in a global Number of Power Law Spikes of 162.250 with a test accuracy of 96.458% at the end of training.
The parametrisation first epoch of training additional results in a large increase in the global Number of Power Law Spikes from 91.150 to 144.217, corresponding to a test accuracy of 88.898%, although a smaller increase than the parametrisation. As observed with the parametrisation in subsequent training epochs the global Number of Power Law Spikes oscillates between increasing and decreasing, resulting in a global Number of Power Law Spikes of 162.633 and a test accuracy of 97.373%.
The parametrisation follows a similar trend as the and parametrisations while improving in test accuracy, however with a smaller initial increase of the global Number of Power Law Spikes after 1 epoch training increasing from 82.633 to 138.267. The global Number of Power Law Spikes subsequently increases and decreases from epoch to epoch resulting in a global Number of Power Law Spikes of 171.750 and a test accuracy of 97.499% at the end of training.
These results indicate the the Number of Power Law Spikes is an inherently noisy metric with high variably from epoch to epoch, which can be attributed to the relationship it has with the fitting of , as the number of power law spikes directly corresponds to how many samples where used to estimate . Furthermore these results show that Number of Power Law Spikes does not need to change in order for the model to generalise.
Stable Rank: Figure 19i compares the performance throughout training with the global Stable Rank when different subsets of the models parameters are updated during training. The training trajectories of the global Stable Rank closely matches the behaviour observed with Lambda Max () and Spectral Norm in Figures 19d and 19i. However instead of increasing the Stable rank decreases throughout training. The , and parametrisations global Stable Rank stays relatively fixed throughout training. The parametrisation global Stable Rank gradually decreases throughout training reaching the lowest value while achieving the lowest test accuracy. The , and parametrisations observe an initial small decrease in the global Stable Rank which then continues to gradually decrease with reaching the lowest value. The and parametrisations reach approximately the same global Stable Rank value.
Discussion
The results from this section show that a model can achieve good test accuracy and generalise on a given task with fixed unbiased singular values () and therefore exhibit no change in the spectral properties of the layers of the model. Furthermore, when is trained on its own ( and fixed) the model does not achieve comparable performance to any other factor, moreover the inclusion of with and only marginally improves performance over training with and independently with an increase of 3.222% and 2.449% respectively. These results suggest that the dominate factor in learning can be attributed to the left and right singular vectors. This means that over-interpreting the spectral properties of layers can lead to erroneous and misleading conclusions surrounding generalisation, given that an that they are not required to change for a model to effectively generalise.
The Reliability of the Power Law Fit ()
One of the fundamental assumptions of HTSR theory is that the tail of the eigenvalues fit and can be accurately described by a power law distribution, hence the metric [30]. However, although the power law distribution can be fit to data, this does not mean it is the best or most apt distribution to describe the data [6]. Therefore the following sections § K.1, § K.2 explore the power law distribution assumption by performing a goodness of fit test comparing the power law distribution to the exponential, the log-normal, truncated power law and the stretched exponential distributions with the power-law package [1]. A p-value of 0.05 is used to disambiguate between statistically meaningful differences in the resulting test normalised likelihood ratios. A negative likelihood ratio indicates support for the alterative distribution.
Fitting a power law requires finding the minimum value of the tail , which is done utilising the Kolmogorov-Smirnov (KS) distance [6]. When there exists one minimal across all possible values fitting the tail is trivial. However, in practise this is not the case, and furthermore as this is fitted with a maximum likelihood estimation the selection of can be susceptible to noise. [6] suggests as a rule of thumb that more than or equal to 50 samples can be used to obtain reliable estimates of distributional fit. Given the possibility of outliers in Figure 7 in § K.3 we explore if the selection of can explain this occurrence and find that ensuring 50 points are used to estimate the tail that we find a stronger correlation between the initial and the at the end of training.
Overall this section shows that the eigenvalues of the ReLU FCN 5x512 models can be statistically significantly better described by the alternative distributions explored, namely the lognormal, truncated power law and the stretched exponential. For the large scale models there is inconclusive evidence to suggest that the power law distribution is the most appropriate distribution to describe the eigenvalues of these layers. Therefore in lieu of a clear statistical preference towards the power law distribution, as cautioned in [6], interpreting the data as fitting a power law may lead to erroneous and misleading conclusions about the nature of the data, in this case the spectral properties of the trained weight matrices. Furthermore identifying the tail accurately has inherent challenges and selecting the with the smallest KS distance may result inaccurate findings.
Small Scale Models (ReLU FCN 5x512)
As all 100 ReLU FCN 5x512 models where trained on the same dataset with the same initialisation method and training strategy we ensemble the eigenvalues for each respective layer from the 100 models to provide a more representative sample of the eigenvalues of each layer. Concretely for FC1 to FC5 we have 51200 data points and FC6 we have 1000 data points. We compare the goodness of fit of the ensemble eigenvalues of each layer against the power law and alternative in Table 17. Table 17 shows that the statistically significant evidence that that eigenvalues of the layers can be better described as either a log-normal distribution, truncated power law or a stretched exponential (note negative normalised likelihood ratios and p-values < 0.5). This provides statistically significant evidence that the power-law distribution is not a suitable model for describing the eigenvalues of this model and demonstrates a clear preference to the alternatives.
| Layer | Exponential Normalised LR | Exponential P-value | Lognormal Normalised LR | Lognormal P-value | Truncated Power Law Normalised LR | Truncated Power Law P-value | Stretched Exponential Normalised LR | Stretched Exponential P-value |
| FC1 | 6.838 | 0.000 | -3.829 | 0.000 | -5.026 | 0.000 | -3.151 | 0.002 |
| FC2 | 7.522 | 0.000 | -5.502 | 0.000 | -5.303 | 0.000 | -5.858 | 0.000 |
| FC3 | 7.633 | 0.000 | -5.764 | 0.000 | -5.535 | 0.000 | -6.152 | 0.000 |
| FC4 | 7.135 | 0.000 | -6.014 | 0.000 | -5.767 | 0.000 | -6.431 | 0.000 |
| FC5 | 8.224 | 0.000 | -5.121 | 0.000 | -4.826 | 0.000 | -5.450 | 0.000 |
| FC6 | -9.115 | 0.000 | -4.732 | 0.000 | -9.116 | 0.000 | -4.864 | 0.000 |
Table 17. ReLU FCN 5x512: Goodness of fit test of the power law distribution against the alternatives with corresponding normalised likelihood ratio (LR) (a negative value indicates support for the alternative) and p-value. For reliable fitting a minimum of 100 samples were required to fit the tail.
Large Scale Models
In this section we explore the goodness of fit of the power law to eigenvalues of the pretrained models explored in § 4.2 namely ResNet34, ViT, GPT2 GPT2-Large and Pythia-1B layers. We compare the goodness of fit for each layer and report how many layers either favoured each distribution compared or return an inconclusive results, where an inconclusive result is a result that reports a p-value greater than 0.05, The results for each respective model are shown in Table 18.
| (a) ResNet34 | |||
| Comparison | Power Law Favoured | Alt Favoured | Inconclusive |
| Power Law vs Exponential | 4 (10.811%) | 4 (10.811%) | 29 (78.378%) |
| Power Law vs Log Normal | 0 (0.000%) | 10 (27.027%) | 27 (72.973%) |
| Power Law vs Truncated Power Law | 0 (0.000%) | 17 (37.838%) | 23 (62.162%) |
| Power Law vs Stretched Exponential | 1 (2.703%) | 10 (27.027%) | 26 (70.270%) |
| (b) ViT | |||
| Comparison | Power Law Favoured | Alt Favoured | Inconclusive |
| Power Law vs Exponential | 4 (8.000%) | 2 (4.000%) | 44 (88.000%) |
| Power Law vs Log Normal | 0 (0.000%) | 7 (14.000%) | 43 (86.000%) |
| Power Law vs Truncated Power Law | 0 (0.000%) | 9 (18.000%) | 41 (82.000%) |
| Power Law vs Stretched Exponential | 0 (0.000%) | 7 (14.000%) | 43 (86.000%) |
| (c) GPT2 | |||
| Comparison | Power Law Favoured | Alt Favoured | Inconclusive |
| Power Law vs Exponential | 7 (14.286%) | 3 (6.122%) | 39 (79.592%) |
| Power Law vs Log Normal | 0 (0.000%) | 3 (6.122%) | 46 (93.878%) |
| Power Law vs Truncated Power Law | 0 (0.000%) | 7 (14.286%) | 42 (85.714%) |
| Power Law vs Stretched Exponential | 1 (2.041%) | 3 (6.122%) | 45 (91.837%) |
| (d) GPT2-Large | |||
| Comparison | Power Law Favoured | Alt Favoured | Inconclusive |
| Power Law vs Exponential | 14 (9.655%) | 0 (0.000%) | 131 (90.345%) |
| Power Law vs Log Normal | 0 (0.000%) | 0 (0.000%) | 145 (100.000%) |
| Power Law vs Truncated Power Law | 0 (0.000%) | 18 (12.414%) | 127 (87.586%) |
| Power Law vs Stretched Exponential | 0 (0.000%) | 3 (2.069%) | 142 (97.931%) |
| (e) Pythia-1B | |||
| Comparison | Power Law Favoured | Alt Favoured | Inconclusive |
| Power Law vs Exponential | 5 (7.812%) | 0 (0.000%) | 59 (92.188%) |
| Power Law vs Log Normal | 0 (0.000%) | 0 (0.000%) | 64 (100.000%) |
| Power Law vs Truncated Power Law | 0 (0.000%) | 2 (3.125%) | 62 (96.875%) |
| Power Law vs Stretched Exponential | 1 (1.562%) | 0 (0.000%) | 63 (98.438%) |
Table 18. Comparison of distribution fits across layers using likelihood ratio tests. Counts show the number of layers favouring each distribution. Inconclusive indicates tests with a p-value greater than 0.05.
The Tables show that the power law distribution is minimally and rarely favoured across models and when it is favoured this typically occurs in comparison to the exponential distribution. Furthermore the majority of comparisons are inconclusive and thereby show no preference to a particular distribution. This means that there is not sufficient statistical evidence to support the use of the power law distribution to fit the tails of the eigenvalues as there exist multiple distributions that can provide comparably good fits to the data.
Selection of
In § 4.3 Figure 7 we observe that there is a strong correlation with model at initialisation and at the end of training, however for the interpolated points of 0.1-0 appeared to be outliers to this trend achieving values lower than the trend would expect. As [6] states that 50 samples can be used a rule of thumb for the minimum required samples to obtain reliable estimates of distributional fit, we explore in Figure 20 how many data points were used in fitting the power-law to the tail of the data (the black vertical line represents the 50 sample cut off). The Figure shows that many of the interpolation points values are estimated with fewer than 50 points.
Figure 20. Number samples used for alpha estimation for each layer. Each dot represents a layer from a trained model
Given the results in Figure 20 and guided by the role of thumb we re-estimate the values using the power law package [1] with a constraint of a minimum of 50 points used to estimate alpha for layers FC1-FC5, for layer FC6 is still calculated from the 10 eigenvalues, to ensure the model average is maintained and enable comparability with the original figure. The results are shown in Figure 21. It is important to note that points 0 and 10001 are not plotted as suitable could not be found for the initialisation which expected given they are at the extremes of the interpolation. In this figure the trend that at initialisation strongly correlates to the at the end of training is substantially clearer, which provides further credence to the statement that the ’good’ range [2, 6] is a consequence of our standard initialisation regime and regularisation from SGD. This underscores that does not necessarily emerge because this metric is indicative of generalisation.
Large Scale Model Details
The pretrained models were accessed from huggingface:
ResNet34: https://huggingface.co/timm/resnet34.tv_in1k
ViT: https://huggingface.co/timm/vit_base_patch16_224.augreg_in1k
GPT2: https://huggingface.co/openai-community/gpt2
GPT2-Large: https://huggingface.co/openai-community/gpt2-large
Pythia-1B: https://huggingface.co/EleutherAI/pythia-1b
Computational Resources
Model training and test evaluation was conducted using A100 and L40S GPUs and the AMD EPYC 7742 CPU.
Small Scale Analysis of Alpha Under Re-Initialisation and Re-Randomisation
To train each model took approximately 1 minute. Given 100 models were trained, training the models took approximately 100 GPU minutes. The evaluation time of the model was approximately 0.1 GPU minutes. Therefore total GPU usage was approximately 220 GPU minutes.
Large Scale Analysis of Alpha Under Reparametrisation
To evaluate the performance of the reparametrisation of each layer for each model took approximately:
ResNet34: 0.23 GPU minutes
ViT: 1 GPU Minutes
GPT2: 0.23 GPU minutes
GPT2-Large: 1 GPU Minutes
Pythia-1B: 1 GPU Minutes
The number of layers evaluated for each model was:
ResNet34: 37
ViT: 43
GPT2: 45
GPT2-Large: 145
Pythia-1B: 64
We evaluated the ResNet34, ViT, GPT2, GPT2-Large model with 10 random initialisation. For the Pythia-1B we use one initialisation and one random initialisation. This experiment took approximately: 2196.6 GPU minutes.
Alpha At Initialisation & Training With Any Alpha
Training and evaluating each model took approximately 4.16 CPU minutes. We trained a total of 410 models (41 interpolation points and 10 random seed) totalling approximately 1705.6 CPU minutes or 28.42 CPU hours.
Generalisation With Fixed Alpha
Training and evaluating and evaluating each model took approximately 90 GPU minutes. Given we explored 3 alpha conditions and averaged over 10 models the total training times was 2700 GPU minutes.
SVD Ablation for Learning Dynamics
Training with each factor took approximately:
: 12 GPU minutes
: 12 GPU minutes
: 80 GPU minutes
: 15 GPU minutes
: 85 GPU minutes
: 90 GPU minutes
: 100 GPU minutes
Given we average over 10 models the total training time was approximately 3940 GPU minutes.
Total Training and Evaluation time
The total training and evaluation time was approximately: 8928.6 GPU minutes and 1705.6 CPU minutes, or more aptly: 148.81 GPU hours and 28.43 CPU hours.
Datasets and Licences
Image Datasets
MNIST [23] has an CC licence.
ImageNet [38] ImageNet states that: ‘The data is available for free to researchers for non-commercial use.’ giving a CC BY-NC licence.
Language Datasets
WikiText-103 [32] has a CC BY-SA 4.0 licence.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: In § 4.1 and § 4.2 we show from low complexity datasets (MNIST) and small scale models (ReLU FCN 5x512) settings to high complexity datasets (ImageNet and WikiText-103) on million (ResNet34 and ViT) and hundreds of million (GPT and GPT Large) parameter scales across modalities (Image and Language) that weight analysis provided by random matrix theory, primarily alpha, are not reparametrisation invariant under critical layer theory. Following this in § 4.4 we show that ideas around the criticality of optimal alpha values in models are misplaced; through simple modifications to optimisation we show that well generalising models can arise for an arbitrary alpha value. In addition, initialisation plays a strong role in the alpha found at the end of training. Our empirical evidence shows that optimal alpha values are not a prerequisite of generalisation and in Appendix Section J we quantify the contribution by alpha through ablations studies using singular value decomposition. Finally, in Appendix Section H we show in small scale settings that data-based metrics provide crucial insights between robust and critical layers that fundamentally relate to existing theories of the lottery ticket hypothesis, Section H.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix Section A we clearly articulate in which settings our findings hold and their extensions to metrics that are not explored in the paper. We also provide additional modalities where our findings would need to be confirmed.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: To analyse data-free metrics in Theorem 1 we show (through an extension of Dihn et al [9]) that for a majority of metrics values can be arbitrarily modified while preserving the same functional representation of the model. To study alpha, our paper provides a set of extensive empirical evaluations to explore the relationship between data-free metrics and generalisation. We acknowledge that formal proofs would be useful for our study of alpha, however the results presented concretely show that optimal alpha values are not causal for generalisation.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided full explanations for the range of experiments conducted in this paper with metric formalisations, model architectural details, optimisation hyperparameters and dataset details.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Notebooks are provided at https://github.com/GMW99/data-free-metrics-dont-work which demonstrate the method and analysis for the MNIST and LLM cases.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings for the core of the paper are provided to a level of detail that enable reproduction of the results and allow for full understanding of their experimental context.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide large sample sizes for experiments in the paper, typically calculating means from 100 models or variations with a few instances where we calculate means with 10 models. In tables we report means and ± 1 Standard Error of The Mean to best represent uncertainty [3], we also analyse the correlation between data-free and data-based metrics and test accuracy using the Spearman correlation coefficient , coefficient of determination , the root mean square error (RMSE) of the linear regression and Kendall’s tau measure (). When conducting likelihood tests we utilise a p-value of 0.05.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In Appendix SectionM we provide details on the compute resources used to run the experiments in this paper, we also detail the estimated total compute required to re-run the experiments in the paper.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: Every author has reviewed the code of ethics and can confirm that out paper confirms in every aspect with the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification: Our results can be considered under foundational research and provide no association with particular applications or deployments. However, we do acknowledge that our results show that data-free metrics may not be suitable for understanding generalisation which impact interpretability efforts leveraging data-free metrics for neural networks across data domains.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification: In the paper we provide no new models or scrapped datasets that have risk of misuse.
12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the paper we have appropriately cited assets that have been used and provided url’s to specific datasets, models and tools where required. In Appendix Section N we explicitly state the licenses for each dataset used in the paper.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification: Our paper does not release new assets. However, details of datasets and models used for the empirical analysis of the paper are provided in the paper.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification: At no point in core method development of the paper were LLM’s used.
References
- [1]Jeff Alstott, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, 9(1), 2014. URL: https://doi.org/10.1371/journal.pone.0095816.
- [2]Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. URL: https://papers.nips.cc/paper_files/paper/2017/hash/b22b257ad0519d4500539da3c8bcf4dd-Abstract.html.arxiv.org/abs/1706.08498
- [3]Sarah Belia, Fiona Fidler, Jennifer Williams, and Geoff Cumming. Researchers misunderstand confidence intervals and standard error bars. Psychological methods, 10(4):389, 2005. URL: https://psycnet.apa.org/record/2005-16136-002.pubmed.ncbi.nlm.nih.gov/16392994
- [4]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Afiah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023-23. URL: https://proceedings.mlr.press/v202/biderman23a.html.arxiv.org/abs/2304.01373
- [5]Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Philipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014. URL: https://arxiv.org/abs/1312.3005, arXiv:1312.3005.
- [6]Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661–703, 2009. URL: https://doi.org/10.1137/070710111.
- [7]Romain Couillet and Zhenyu Liao. Random matrix methods for machine learning. Cambridge University Press, 2022. URL: https://www.cambridge.org/core/books/random-matrix-methods-for-machine-learning/6B681EB69E58B5F888EDB689C160C682.DOI
- [8]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. URL: https://doi.org/10.1109/CVPR.2009.5206848.
- [9]Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1019–1028. PMLR, 2017. URL: https://proceedings.mlr.press/v70/dinh17b.html.arxiv.org/abs/1703.04933
- [10]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=YicbFdNTTy.
- [11]Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. In Advances in Neural Information Processing Systems, volume 35, pages 33054–33065, 2022. URL: https://proceedings.neurips.cc/paper_files/paper/2022/hash/d5cd70b708f72673e2ebace18c37f71b-Abstract-Conference.html.
- [12]Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL: https://openreview.net/forum?id=rJl-b3RcF7.
- [13]Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256. PMLR, 2010-13. URL: https://proceedings.mlr.press/v9/glorot10a.html.
- [14]Di He, Songjun Tu, Ajay Jaiswal, Yulong Pei, Li Shen, Shiwei Liu, and Lu Yin. Alphadecay: Module-wise weight decay for heavy-tailed balancing in llms. International Journal of Artificial Intelligence and Robotics Research, 03, 2026. URL: https://doi.org/10.1142/S927233532650002X.
- [15]Jing He, Hua Jiang, Cheng Li, Siqian Xin, and Shuzhen Yang. Early-stopping for transformer model training, 2025. URL: https://arxiv.org/abs/2510.16074, arXiv:2510.16074.
- [16]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, pages 1026–1034. IEEE Computer Society, 2015. URL: https://doi.org/10.1109/ICCV.2015.123.
- [17]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016-06. URL: https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html.arxiv.org/abs/1512.03385
- [18]Thomas King, Simon Butcher, and Lukasz Zalewski. Apocrita - High Performance Computing Cluster for Queen Mary University of London, 2017-03. doi:10.5281/zenodo.438045.
- [19]King’s College London HPC. King’s computational research, engineering and technology environment (create), 2025. URL: https://doi.org/10.18742/rnvf-m076.
- [20]Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 57(9):1–52, 2025. URL: https://doi.org/10.1145/3728458.
- [21]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR, 2019-09. URL: https://proceedings.mlr.press/v97/kornblith19a.html.arxiv.org/abs/1905.00414
- [22]Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009. URL: http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf.
- [23]Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]., 1998. URL: http://yann.lecun.com/exdb/mnist.
- [24]Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 888–896. PMLR, 2019-16. URL: https://proceedings.mlr.press/v89/liang19a.html.
- [25]J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151, 1991. doi:10.1109/18.61115.DOI
- [26]Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yifan Zhou, Pu Ren, and Yaoying Yang. Model balancing in helps low-data training and fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1311–1331. Association for Computational Linguistics, 2024-11. URL: https://doi.org/10.18653/v1/2024.emnlp-main.78/.
- [27]Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL: https://openreview.net/forum?id=fHq4x2YXfv.arxiv.org/abs/2410.10912
- [28]Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning, pages 4284–4293. PMLR, 2019. URL: https://icml.cc/media/icml-2019/Slides/4388.pdf.arxiv.org/abs/1901.08276
- [29]Charles H Martin and Michael W Mahoney. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 505–513. SIAM, 2020. URL: https://epubs.siam.org/doi/abs/10.1137/1.9781611976236.57.arxiv.org/abs/1901.08278
- [30]Charles H. Martin and Michael W. Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73, 2021. URL: http://jmlr.org/papers/v22/20-410.html.
- [31]Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1):4122, 2021. URL: https://www.nature.com/articles/s41467-021-24025-8.DOI
- [32]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL: https://openreview.net/forum?id=Byj72udxe.arxiv.org/abs/1609.07843
- [33]Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu, and Yao-qing Yang. Htmuon: Improving muon via heavy-tailed spectral correction. arXiv preprint arXiv:2603.10067, 2026. URL: https://arxiv.org/pdf/2603.10067.
- [34]Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. In Advances in Neural Information Processing Systems, volume 34, pages 18420–18432, 2021. URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/995f5e03890b029865f402e83a81c29d-Paper.pdf.arxiv.org/abs/2001.00939
- [35]Hari Kishan Prakash and charles h martin. Grokking and generalization collapse: Insights from HTSR theory. In High-dimensional Learning Dynamics 2025, 2025. URL: https://openreview.net/forum?id=upyC7rwR1M.arxiv.org/abs/2506.04434
- [36]Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. Alphaflora: Assigning lora experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20511–20523, 2024. URL: https://aclanthology.org/2024.emnlp-main.1141.pdf.
- [37]Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. J. ACM, 54(4):21–es, 2007. URL: https://doi.org/10.1145/1255443.1255449.
- [38]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. doi:10.1007/s11263-015-0816-y.DOI
- [39]Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016. URL: https://proceedings.neurips.cc/paper_files/paper/2016/hash/ed265bc903a5097f61d3ec064d96d2e-Abstract.html.arxiv.org/abs/1602.07868
- [40]Amartya Sanyal, Philip H. Torr, and Puneet K. Dokania. Stable rank normalization for improved generalization in neural networks and gans. In International Conference on Learning Representations, 2020. URL: https://openreview.net/forum?id=H1enKkrFDB.arxiv.org/abs/1906.04659
- [41]Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL: https://arxiv.org/abs/1409.1556, arXiv:1409.1556.
- [42]Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix theory analysis of neural network weight matrices. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024. URL: https://openreview.net/forum?id=41kpc2Nzwc.
- [43]Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In Advances in Neural Information Processing Systems, volume 34, pages 24261–24272, 2021. URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Paper.pdf.arxiv.org/abs/2105.01601
- [44]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.arxiv.org/abs/1706.03762
- [45]Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23549–23588. PMLR, 2022-17. URL: https://proceedings.mlr.press/v162/wei22a.html.arxiv.org/abs/2203.06176
- [46]David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Yardi, Karen Livescu, Michael Maire, and Matthew R. Walter. Approaching deep learning through the spectral dynamics of weights, 2024. URL: https://arxiv.org/abs/2408.11804, arXiv:2408.11804.
- [47]Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL: https://openreview.net/forum?id=Sy8gdB9xx.arxiv.org/abs/1611.03530
- [48]Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? Journal of Machine Learning Research, 23(67):1–28, 2022. URL: http://jmlr.org/papers/v23/20-069.html.














