WorksComputer Science
A Class of Rényi-based Concentration Metrics for Kolmogorov-Arnold Geometry in Multilayer Perceptrons
Abstract
The interpretability of artificial intelligence models has become a central concern, especially for black-box models such as multilayer perceptrons (MLPs). Kolmogorov-Arnold Geometry has recently emerged as a framework for analysing the internal representations of these networks, by examining how the hidden layer locally responds to changes in the input through the minors of its Jacobian. The degree to which a few of these minors dominate the rest, known as their concentration, has so far been measured with the Participation Ratio. Although this metric has useful properties, its maximum value depends on the size of the network, so a given value cannot be read as high or low concentration without reference to the architecture. To overcome this limitation, in this paper we propose the -RC metrics, a family of Rényi-based concentration measures that compare the normalized magnitudes of the minors with the uniform distribution through the Rényi divergence of order . All the metrics in the family are bounded between 0 and 1 regardless of the architecture, with 0 indicating that the magnitude is evenly spread over the minors and 1 that it is carried by a single one. Moreover, the metrics are non-decreasing in , and for = 2 they reduce to an explicit transformation of the Participation Ratio. Computational experiments with single-hidden-layer MLPs trained on a synthetic dataset show the benefits of our proposal.
Introduction
Understanding how neural networks internally represent information remains one of the central challenges in modern machine learning. While deep neural networks have achieved remarkable performance across a wide range of applications, their internal representations are still only partially understood. Developing mathematical tools capable of characterizing these representations is therefore an important step towards explaining why neural networks generalize successfully and under which conditions they exhibit efficient learning [6].
A recent line of research has revisited the celebrated Kolmogorov–Arnold representation theorem (Kolmogorov, 1957), originally proved by Kolmogorov and later refined by Arnold, which states that every continuous multivariate function on the multidimensional unit cube can be represented as a finite superposition of continuous univariate functions. Although this theorem is mathematically elegant, its practical impact on machine learning remained limited for decades due to the highly irregular nature of the functions involved in the representation [5, 8].
Interest in the theorem has recently been revived by the introduction of Kolmogorov–Arnold Networks (KANs) [9], a novel neural network architecture inspired by the representation theorem. Beyond their architectural design, these works have stimulated a renewed interest in understanding the geometric structure underlying the Kolmogorov–Arnold representation itself.
In this context, Freedman introduced the notion of Kolmogorov–Arnold Geometry (KAG), a geometric framework derived from the inner map appearing in the proof of the Kolmogorov–Arnold theorem [3]. The central idea is that the local geometric properties of this map provide valuable information about how different variables interact within the representation. More importantly, the author hypothesized that similar geometric patterns may emerge spontaneously in neural network architectures that are not explicitly based on the Kolmogorov–Arnold construction, particularly in single-hidden-layer multilayer perceptrons (MLPs).
To quantify these geometric properties, [4] proposed several metrics based on the Jacobian of the inner map and its exterior powers, whose entries are the minors of the Jacobian. We say that these minors are concentrated when a few of them carry most of the total magnitude while the remaining ones contribute little. For instance, the magnitudes of the matrix of minors are more concentrated than those of , although both add up to the same total. Among the proposed metrics, the Participation Ratio (PR) has received particular attention as a measure of this concentration, with lower values of the PR indicating a higher concentration of the minors. Experimental evidence suggests that, under certain training regimes, an increase in concentration with respect to initialization is associated with better model performance. Interestingly, this relationship appears only in specific settings, often referred to as the Goldilocks regime, while it disappears in others. Although the reasons for this phenomenon are not yet fully understood, these observations suggest that the PR captures relevant aspects of the Kolmogorov–Arnold Geometry emerging in multilayer perceptrons.
Despite these promising results, the interpretation of the Participation Ratio depends on the architecture under study. While its minimum value, attained when a single minor carries all the magnitude, is the same for every network, its maximum value, attained when the magnitude is uniformly spread over all minors, grows with the number of hidden neurons. Consequently, the same PR value may lie close to the maximum for a narrow network and far from it for a wider one, so its position within the attainable range, and therefore its interpretation in terms of concentration, changes with the architecture. (Freedman & Mulligan, 2025) report the ratio between the PR of the trained model and its value at initialization. This ratio measures how concentration changes during training for a given network, but it does not place the PR on a scale whose extremes have a fixed meaning.
The main objective of this paper is to address this limitation. We introduce a family of normalized concentration metrics based on the Rényi entropy of order , applied to the probability distribution obtained by normalizing the magnitudes of the minors. The minimum and maximum values of these metrics are 0 and 1, regardless of the architecture, and so is their interpretation: a value of 0 corresponds, for every order , to minimum concentration, with the magnitude uniformly spread over all minors, and a value of 1 to maximum concentration, with all the magnitude carried by a single minor. The proposed metric is monotonically non-decreasing with respect to , and the order determines the weight given to the different parts of the underlying distribution: small values of take into account the proportion of non-zero entries, whereas large values emphasize its mode. In addition, for the metric can be written as a function of the Participation Ratio, establishing a direct theoretical connection between both quantities.
Finally, we illustrate the behaviour of the proposed metric through a series of computational experiments on single-hidden-layer MLPs trained on a continuous version of the XOR problem. We show that, unlike the PR, the proposed metric places the concentration of the minors on a scale whose extremes have the same meaning for every architecture, and we analyse how the choice of affects the resulting values.
The Kolmogorov–Arnold Geometry
This section reviews the main concepts underlying the notion of Kolmogorov–Arnold Geometry (KAG), recently introduced by Freedman [3] as a geometric framework inspired by the proof of the Kolmogorov–Arnold representation theorem. We first describe the geometric intuition behind this concept in Section 2.1 and then present the mathematical objects used to quantify it, namely the Jacobian and its exterior powers, in Section 2.2. Section 2.3 particularizes these objects to single-hidden-layer MLPs, the architecture studied in this work, by decomposing the network into an inner and an outer map and computing the minors of the Jacobian of its inner map. Finally, Section 2.4 describes a particular metric proposed by [4], called the Participation Ratio, which measures the concentration of these minors.
From the Kolmogorov–Arnold theorem to Kolmogorov–Arnold Geometry
The Kolmogorov–Arnold representation theorem is one of the most celebrated results in approximation theory. Roughly speaking, it states that every continuous multivariate function on the multidimensional unit cube can be represented as a finite superposition of continuous univariate functions. While several variants and refinements of this theorem have been proposed in the literature, in this work we consider the formulation by [10], which can be stated as follows:
Theorem 1 (Kolmogorov–Arnold Theorem). Let with . There exist real numbers and Lipschitz (and thus continuous) functions , , with , such that for every continuous function , there exists a continuous function satisfying, for each ,
Furthermore, the constants and the functions can be chosen independently of the function .
We omit the formal proof of Theorem 1 here. However, the interested reader is referred to [10] for a detailed proof and discussion.
Theorem 1 can be read as a decomposition of into two maps. The inner map is defined componentwise by
and the outer map by
With this notation, we define the KA representation , and Theorem 1 states that on . Since the constants and the functions do not depend on , the inner map is universal: the same map serves for every continuous function, and only the outer map, through , depends on the target function.
Regarding its regularity, each component is a finite linear combination of Lipschitz functions, and hence Lipschitz. Therefore is a Lipschitz map, and by Rademacher’s theorem it is differentiable almost everywhere in with respect to the Lebesgue measure. Its Jacobian matrix, and hence all the quantities derived from it in Section 2.2, are therefore defined almost everywhere, but not necessarily at every point. Throughout this work, the Jacobian of is considered only at points where is differentiable.
Beyond its theoretical guarantee of universal approximation, Theorem 1 admits a compelling geometric interpretation through the structure of the inner map . The recent work of [4] characterizes the local geometric properties of this map, a framework referred to as Kolmogorov–Arnold geometry (KAG). It does so through differential quantities, mainly the Jacobian matrix and its exterior powers (Section 2.2), which provide information about how the hidden representation varies with respect to the input variables.
Jacobians, exterior powers and minor concentration
Consider the inner map defined in (2), where denotes the input dimension and the number of hidden coordinates. Given a point at which is differentiable (see Section 2.1), the Jacobian matrix describes the local sensitivity of each hidden coordinate with respect to the input variables at . In other words, each row of the Jacobian characterizes how the corresponding hidden coordinate locally reacts to infinitesimal perturbations of the input. Specifically, the -th entry of the Jacobian matrix is the partial derivative of the -th component function with respect to the -th input variable evaluated at , namely
An important consequence of the proof of the Kolmogorov–Arnold theorem (Theorem 1) is that, in the ideal construction, many rows of the Jacobian vanish. Geometrically, this means that several hidden neurons become locally insensitive to perturbations of the input, producing a highly structured differential representation.
However, analysing individual neurons is not sufficient to characterize the complete geometry. It is also necessary to study how groups of hidden neurons jointly respond to variations of the input. To this end, [4] considered the exterior powers of the Jacobian matrix. The entries of the -th exterior power correspond to the minors of order , which quantify the joint variation of subsets of hidden neurons with respect to subsets of input variables.
To formalize this notion, we introduce the definition of the -th exterior power of a matrix, which collects all possible minors into a single operator.
Definition 1. Let and let . Define the ordered index sets of cardinality as
and similarly for subsets of of size . For any pair of index sets and , let denote the submatrix formed by selecting the rows indexed by and the columns indexed by .
The -th exterior power of , denoted by , is the matrix whose entry corresponding to the row index and column index
is given by the determinant of :
Applying Definition 1 to the Jacobian matrix yields its -th exterior power , whose entries admit a natural geometric interpretation in terms of volume changes. For a linear map given by , with , an infinitesimal volume element is mapped to an element of volume . For a nonlinear map, the same holds locally around a point , with replaced by the Jacobian matrix at . This is the Jacobian determinant that appears in the change of variables formula for multiple integrals.
Minors extend this idea to rectangular Jacobians and to subsets of variables. The minor is the factor by which the oriented -dimensional volume of an infinitesimal parallelepiped spanned by the input directions changes when it is mapped by and projected onto the hidden coordinates . Its absolute value measures the magnitude of the volume change, while its sign indicates orientation. For , the minors reduce to the entries of the Jacobian, and if and , the only minor is the full Jacobian determinant, a case that does not arise in this work, where . For instance, for a map with two inputs and three hidden coordinates, the three minors of order two are the oriented areas of the projections of the image of an infinitesimal square onto the planes spanned by the hidden coordinates , and .
The distribution of these minors plays a fundamental role in the characterization of the Kolmogorov–Arnold Geometry. When only a relatively small number of minors concentrates most of the total magnitude, the geometry is said to exhibit a high degree of concentration. Conversely, if the magnitude is distributed more uniformly among the minors, the corresponding geometric structure is considered less pronounced.
MLP and its Kolmogorov–Arnold Geometry
A fundamental question raised by [4] is whether Kolmogorov–Arnold geometry emerges spontaneously during the standard training of conventional neural network architectures. Unlike the construction underlying Theorem 1, where the inner functions are explicitly designed, standard multilayer perceptrons (MLPs) do not impose any KA structure a priori. In this context, the emergence of KA geometry does not refer to the appearance of exact zeros. It refers to the development, during training, of patterns similar to those of the KA construction: Jacobian rows that are approximately zero, and a distribution of minors in which a few of them carry most of the total magnitude. These patterns are assessed by comparing the trained model with its initialization.
To evaluate whether a trained MLP exhibits a geometric structure compatible with the KA framework, the network is first decomposed into an inner map and an outer map. In the KA representation (Theorem 1), the inner map embeds the input domain into a higher-dimensional space, where a relatively simple outer map reconstructs the target function.
We consider a single-hidden-layer MLP (shallow MLP) with inputs, hidden neurons and a single linear output neuron. Let denote the input vector. The network is parameterized by:
an input weight matrix and a bias vector ;
an output weight vector and a scalar bias ;
a smooth activation function . In this work we use the GeLU activation, , where denotes the cumulative distribution function of the standard normal distribution. We denote by its componentwise extension, for .
The input–output mapping is then given by
Following the KA framework, we decompose into two stages:
Inner map :
where is the pre-activation vector.
Outer map : the affine map acting on the hidden representation ,
With this notation, , in direct analogy with the decomposition of § 2.1.
Since GeLU is smooth, is differentiable at every point. Its Jacobian is therefore defined everywhere, unlike that of , which, as discussed in § 2.1, is only guaranteed to exist almost everywhere. For non-differentiable activations such as ReLU, the Jacobian may not exist at points where some pre-activation crosses zero, and a convention would be required to compute the metrics. Using the tools of Section 2.2, the entries of the Jacobian matrix at a point are:
where is the pre-activation of the -th hidden neuron and is the derivative of the scalar activation function. For GeLU,
Equivalently, , where .
In an MLP, row of is approximately zero for one of two reasons. Either the weight vector is close to zero (for instance, due to weight sparsity or dead units), or is close to zero at , as happens with GeLU for sufficiently negative pre-activations. The first mechanism affects the row at every input, whereas the second depends on . It is this input-dependent mechanism that resembles the local insensitivity of hidden coordinates in the KA construction.
Finally, for , the entries of the -th exterior power of the MLP Jacobian, (Definition 1), are
where and . The second equality follows from the multilinearity of the determinant with respect to its rows, and shows that a hidden neuron with makes every minor that involves row approximately zero. These exterior powers quantify how the inner map of the MLP locally transforms -dimensional volume elements, providing the basis for measuring minor concentration in neural networks. For simplicity, throughout the rest of this work we denote by .
Participation Ratio
Building on the exterior powers introduced in § 2.3, we now quantify the concentration of the minors of the inner map defined in Equation (8). To this end, [4] introduced the Participation Ratio (PR), one of the main quantitative tools for detecting the spontaneous emergence of KA geometry in multilayer perceptrons. The PR measures whether the total magnitude of the minors in is concentrated in a few of them or spread across many.
Definition 2 (Participation Ratio). Let be an input point such that
. The order- Participation Ratio at is defined as
where
is the entrywise norm and
is the entrywise (Frobenius) norm. For an evaluation dataset , the global Participation Ratio is defined as the mean over all points of the dataset,
Let denote the total number of minors of order . Viewing as a vector with entries, the inequalities hold, the second one being a consequence of the Cauchy–Schwarz inequality. Hence
The lower bound is attained if and only if exactly one minor is non-zero, and the upper bound if and only if all minors have the same absolute value. Note that, while the lower bound is the same for every architecture, the upper bound depends on , and hence on the number of hidden neurons . We return to this point in § 3.
The value of can therefore be interpreted as follows:
Values close to 1 indicate high concentration: a few minors carry most of the total magnitude, and when a single minor carries all of it. This is the kind of pattern expected from the KA construction.
Values close to indicate low concentration: the magnitude is spread almost uniformly across the minors, as expected in the absence of KA structure.
Finally, observe that if , both norms vanish and Equation (13) is not defined. This can happen even when the Jacobian itself is non-zero, for instance when the rank of is smaller than . For such points we adopt the convention , which corresponds to the absence of concentration and is consistent with the convention that will be adopted for the -RC metrics in Definition 3.
To illustrate this metric in practice, the following example demonstrates how the Participation Ratio is computed for two contrasting scenarios.
Example 1 (Calculation of the Participation Ratio). Consider two cases with minors of order evaluated at a point , whose absolute values are
Both matrices have the same norm, , but their norms differ: and . Therefore
In the first case the magnitude is spread uniformly across all four minors, and the PR attains its maximum value . In the second case all the magnitude is concentrated in a single minor, and the PR attains its minimum value 1.
A novel improved metric
In this section we propose a novel metric that circumvents the limitations of the Participation Ratio metric to assess the concentration of the matrix .
This section is organized in two parts. Section 3.1 revisits the Participation Ratio and illustrates, through a simple example, why its dependence on the network architecture prevents comparing minor concentration across networks of different width. Section 3.2 then introduces the proposed Rényi-based concentration metric: it first normalizes the exterior power values, recalls the Rényi divergence and its main properties, and formally defines the resulting class of -RC metrics. It then shows that the particular case is directly related to the Participation Ratio, and closes with an illustrative example.
Motivating example
As established in the previous section, the Participation Ratio proposed by [4] possesses several appealing properties for quantifying higher-order interactions. It offers an intuitive geometric interpretation of active minors, has well-defined theoretical bounds (), and captures the degree of minor concentration at an individual input point .
Despite these strengths, the Participation Ratio suffers from a fundamental limitation: its range and its interpretation in the upper extreme limit are architecture-dependent. Because the theoretical upper bound depends explicitly on the number of hidden units , the scale of varies dynamically with the network capacity. Consequently, the same numerical PR value can carry completely opposite geometric meanings depending on the size of the network.
To clearly illustrate this scale-dependency issue, consider the following example.
Example 2 (Incompatibility of PR across architectural scales). Suppose we evaluate the order- minor concentration of two trained neural networks with different hidden layer widths at an input point :
Network A (Smaller architecture, ): The matrix of order- minors is given by:
where total minors exist. Computing the norms yields and , yielding
Here, activity is uniformly distributed across all possible minors (), representing maximum dispersion (zero geometric concentration).
Network B (Larger architecture, ): The matrix of order- minors is given by:
where total minors exist. Computing the norms yields and , giving
Here, activity is concentrated in only out of minors, representing strong geometric concentration relative to the network’s capacity (far below its maximum bound of ).
Despite operating under drastically different regimes, total uniform dispersion in Network A versus localized specialization in Network B, both examples yield the exact same Participation Ratio (). Therefore, raw PR values fail to provide a meaningful basis for comparison across different scenarios.
The main motivation of this work is finding a concentration metric more flexible than PR with a simpler interpretation across architectures. To address both issues, we propose an alternative metric based on the Rényi entropy of order . This metric is normalized so that its extreme values have the same meaning for every architecture, regardless of . On the one hand, for every order , the value corresponds to a uniform distribution of the magnitude across all minors, whereas the value is associated with a single non-zero minor. Concentration is thus measured relative to the maximum dispersion attainable by each network, rather than on a scale that changes with its width. It also has a parameter which provides flexibility in the interpretation of such concentration. In the particular case of this new metric is directly expressible as a function of Freedman’s original Participation Ratio.
Rényi-based concentration metric
In Section 3.1, we analysed the PR as a measure of activity distribution across minors. However, as commented above, PR exhibits a fundamental quantitative limitation when evaluating minor concentration across different multilayer perceptron architectures. In particular, the maximum value of is given by , where denotes the total number of minors of order . Consequently, changing the width of a hidden layer alters the baseline scale of PR, making direct comparisons between different network architectures impossible.
To overcome this limitation, we propose a novel concentration metric based on Rényi entropy of order [11]. This metric maps the activity distribution to a normalized concentration score bounded within the interval for any network architecture. Also, the value zero is interpreted as minimum concentration (uniform distribution) and the value one always as maximum concentration (only one non-zero element).
Normalization of Exterior Power Values
To apply an information-theoretic formulation, we first must interpret the matrix of minors as a discrete probability distribution. This point of view is key for our goals, since the appropriate normalization of the absolute values of the entries of can be viewed as a two-dimensional discrete Probability Mass Function (PMF).
Given an input point , let be the total number of minors of order , indexed by the multi-index sets and . Normalizing the absolute values of the entries of by its -norm yields the relative weight associated with the minor :
This collection can be interpreted as a discrete PMF over the entries of the exterior power. If all minors possess equal magnitude, reduces to the uniform two-dimensional PMF with entries . Conversely, if a single minor accumulates all magnitude, the two-dimensional PMF becomes degenerate.
Rényi Divergence and Its Properties
The Rényi divergence relies on the Rényi entropy. The Rényi entropy of order generalizes the Shannon entropy of a discrete probability distribution with entries. When and the Rényi entropy is defined from the set of probabilities as:
where unless (in such a case , by definition). Note that is the Hartley entropy, with denoting the support of . Otherwise, when it is defined as the limit
Specifically, is the classical Shannon entropy, i.e. . The case corresponds to the min-entropy where the metric is determined by the information associated with the most likely outcome. Finally, note that the case is widely used in practice and is known as the collision entropy.
The parameter regulates the weight assigned to outcomes of varying probabilities. The following cases are relevant for our discussion:
Extreme choices ( and ): For , all non-zero entries are weighted equally regardless of magnitude. For , only
the maximal element (that is to say, the mode of the discrete probability distribution) influences the value.
Choice of : In this case we have the Shannon entropy as an intermediate choice.
Choice of : The intermediate case yields the entropy:
The choice of provides a balanced weighting across the distribution while establishing a direct analytical bridge to the Participation Ratio, as we will describe later.
Let us define the Rényi divergence of order (where ) of two discrete probability distributions and with outcomes and . For the Rényi divergence is defined as (van Erven & Harremoës, 2014, Equation (1)):
and for the remaining cases, i.e. , it is defined as the limit of Equation (20),
By taking limits using L’Hôpital’s Rule or evaluating directly, the expressions in Table 1 are obtained for these three special cases: the min-divergence (related to the Hartley entropy) for , the Kullback–Leibler divergence for , and the max-divergence for .
| Parameter () | Expression | Name |
| Min-divergence/Hartley | ||
| Kullback–Leibler () | ||
| Max-divergence |
Table 1. Special cases of the Rényi divergence.
The Rényi divergence is not a distance in the strict sense since, in general, it is neither symmetric nor satisfies the triangle inequality. However, it is nonnegative for every order , and for it also satisfies the identity of indiscernibles, that is, it vanishes if and only if both distributions coincide. For this property holds in a weaker form, since the divergence vanishes if and only if both distributions have the same support. These properties, established in Proposition 1, are what we need for our objectives.
Proposition 1. For any and any discrete probability distributions and with outcomes and , the Rényi divergence is nonnegative, i.e. . Besides, for any , if and only if . The case is special and the condition becomes if and only if .
Proof. For , the proof is found in [2 Theorem 8].
For , the Rényi divergence is the Kullback–Leibler divergence and the proof is found in [1].
From Table 1, the nonnegativity of the min-divergence () is clear since and consequently . In this case, , using Table 1 we have that, if , then all the probability mass of lies on the same indices as that of , and therefore , so that . On the other hand, if , then , so that for every , that is, $\operatorname{supp}(q) \subseteq , which together with the hypothesis gives .
The expression of the Rényi divergence takes a simplified form when the second distribution is the uniform distribution , which can be written in terms of the Rényi entropy, as stated in the following proposition.
Proposition 2. For any and any discrete probability distribution with outcomes, let us consider the uniform distribution , then
Proof. As in Proposition 1, for the proof is found in [2 Equation (2], while for it is found in [1].
For the case , from Table 1 we obtain
Finally, for the case and again from Table 1 we obtain
With the aid of Proposition 2, the Rényi divergence of order allows us to quantify the distance (not a metric distance) between a certain distribution and the uniform . Specifically, for a fixed we have two extreme situations:
Minimum concentration: When (completely uniform distribution) then , yielding (a consequence of Proposition 1). For this is the only case with , whereas for the divergence vanishes for every with , that is, for every without zero entries.
Maximum concentration: When is degenerate ( for some and elsewhere) it is straightforward to check that , reaching the upper bound .
In what follows we will always use the Rényi divergence comparing a probability distribution with the uniform distribution . Therefore, we will simply denote this measure by , omitting and , that is, , where the distribution will be clear from the context.
Formal Definition of the Class of -RC Metrics
Let us formally define the proposed class of Rényi-based Concentration (RC) metrics:
Definition 3 (Class of -RC metrics). Let us fix . Let be an input point, and let denote the -th exterior power matrix with total entries. The pointwise -RC metric of order at , denoted by , is defined as follows:
*where represents the Rényi entropy associated with the probability distribution in Equation (16) and the Rényi divergence of the same distribution with respect to the uniform. Furthermore, for an evaluation dataset , the dataset-level -RC, , or simply , is defined as the sample mean across :
By construction, . For , if and only if the minor magnitudes are uniformly distributed, whereas if and only if no minor vanishes, that is, (Proposition 1). For every , if and only if all the magnitude is concentrated in a single minor (by convention, if ).
Note that, from this point on, the quantities and depend on the order and on the input point . Indeed, both are built from the probability distribution in Equation (16), which in turn is obtained from the exterior power of the Jacobian. This is the reason why the Rényi divergence and the Rényi entropy appear in the definition of the -RC metrics with the notation and , respectively.
Within this class of -RC metrics there are four special cases to be highlighted for a fixed . The 0-RC metric is based on the min-divergence (or Hartley) and only depends on the number of non-zero entries of . At the opposite extreme is the -RC metric, which is related to the modal mass (probability at the mode). In an intermediate situation we have the 1-RC metric that corresponds to the KL-divergence (and the Shannon entropy), and also the 2-RC metric that is related to the collision entropy. The 2-RC metric is directly connected to the PR as shown below.
The class of -RC metrics has a very nice behaviour with respect to the parameter . Specifically, all these metrics are arranged in a monotonic way with respect to , while their range is restricted to the interval . This result is collected in the following proposition.
Proposition 3 (Monotonicity of the class of -RC metrics). Let us fix the parameter with and , where is the number of hidden neurons of the MLP defined in § 2.3 and its input dimension, and let with . Let denote the pointwise -RC metric of order at , defined in Equation (25) from the probability distribution in Equation (16). Let with . Then , with equality if and only if is uniform on its support.
Proof. The inequality follows directly from the monotonicity of the Rényi divergence van Erven & Harremoës, [2] and Equation (25).
Regarding the equality, if is uniform on its support, then for every , so does not depend on and, in particular, .
Conversely, for two orders in , the proof of van Erven & Harremoës, [2] shows that the equality holds if and only if is uniform on its support. If or is , or , the equality and the monotonicity make constant on , which contains two such orders, and the same conclusion follows.
Connection of the 2-RC Metric with the Participation Ratio
A key analytical property of is its exact closed-form relation with . Using Equation (16), we can express the sum of squared probabilities as follows:
Taking the logarithm in Equation (27) yields:
Substituting back into from Equation (25) leads to:
This identity demonstrates that the 2-RC metric logarithmically scales and normalizes PR, removing its range dependency on architecture size while preserving its intuitive geometric interpretation.
Illustrative Example with the 2-RC Metric
To demonstrate how improves the metric interpretation with different architectures, we go back to Example 2 and consider two networks evaluated at an input point :
Network A ( available minors): The exterior power has a Participation Ratio .
The 2-RC metric is:
Activity is evenly split across all 2 available minors, resulting in zero concentration.
Network B ( available minors): The exterior power has 2 non-zero entries equal to 1 and 14 zero entries. Here (identical to Network A). However, the Rényi concentration is:
While PR assigns both networks the same value (), assigns 0.75 to Network B, accurately reflecting that its activity is concentrated within a small fraction (2/16) of its total capacity.
Computational experiments
This section illustrates the behaviour of the proposed concentration metric on single-hidden-layer MLPs trained on a continuous version of the XOR problem. We first describe the experimental setup in § 4.1 and the predictive performance of the trained models in § 4.2. We then analyse the concentration of the minors at a fixed input point for every combination of width and order (§ 4.3) and, finally, its evolution along training for the largest configuration, and (§ 4.4). The main findings are discussed in § 4.5.
Experimental setup
We consider the regression problem studied by [4], in which the target is the continuous XOR function
with input dimension . On the sign of is the product of the signs of the coordinates of , so that, as in its Boolean counterpart, the output depends jointly on all the input variables. For each random seed, we generate a training set with points and a test set with points, independently of each other. In both sets, each input is drawn independently from the uniform distribution on , and the corresponding target is computed without noise as . The superscript therefore indexes the points of the set under consideration, which will be clear from the context.
Following the formulation of § 2.3, we train single-hidden-layer MLPs with two hidden widths, . Here is the multivariate activation function, which applies the univariate activation to each component of its argument. As univariate activation we use the exact GeLU, , detailed in § 2.3. The weight matrices are initialized as random orthogonal matrices [12]. Specifically, is a random matrix with orthonormal columns and is a random vector of unit norm, both obtained from the QR decomposition of a matrix with independent standard Gaussian entries. The biases and are initialized to zero. The models are trained by minimizing the mean squared error with the Adam optimizer [7], with learning rate 0.01, and mini-batches of 250 points reshuffled at every epoch, for a fixed budget of 500 epochs and without early stopping or explicit regularization. This batch size coincides with the critical batch size reported by [4] for this problem. Each width is trained with the four seeds , which determine the dataset, the initialization and the order of the mini-batches. For a given seed, both widths are trained and evaluated on exactly the same training and test points, with the same mini-batch order. All the quantities described below are measured at initialization (epoch 0) and at the end of each of the 500 epochs, that is, on 501 states of each network.
Predictive performance is evaluated with the coefficient of determination, , ([4 Equation (III.5]). Given a set of points, it is defined as:
where is the prediction of the model for the input , and is the sample mean of the targets in . We denote by and the values obtained with () and (), respectively, so that each of them uses the mean of the targets of its own set. A perfect predictor yields , the constant predictor yields , and negative values correspond to predictions worse than that constant.
The concentration of the minors is evaluated with the class of -RC metrics of Definition 3, computed from the distribution of Equation (16) and averaged over a dataset as in Equation (26). We recall that takes values in and is non-decreasing in (Proposition 3), that the case corresponds to the Shannon entropy and the case to the collision entropy, and that the 2-RC metric is related to the Participation Ratio at every point through Equation (29). The results obtained for can therefore be read as a normalized, logarithmic version of the Participation Ratio.
We evaluate four orders, . The -RC metric is the upper limit of the class and only depends on the largest minor. The orders and correspond to the Shannon and collision entropies, respectively, the latter linked to the Participation Ratio through Equation (29). The order gives more weight to the small minors than . It is the only order for which the Rényi divergence is symmetric, and it is a function of the so-called Hellinger distance between and the uniform distribution ([2]). Since is non-decreasing in (Proposition 3), the values for and bound, at every point, those of any order between them.
The limit is not evaluated, because for the MLP of § 2.3 it carries no pointwise information. By Proposition 1 and Definition 3, only depends on which minors are non-zero, and it vanishes whenever no minor is zero. By Equation (12), a minor of the MLP vanishes if and only if or for some . The first condition does not depend on . The derivative of GeLU only vanishes at the isolated point , so the second condition only holds when lies on one of the hyperplanes , which form a set of zero measure. Hence, the support of is the same at almost every and is determined by , and so is . In particular, at almost every as soon as no minor of vanishes, as is generically the case for random and trained weights. In our experiments, no computed minor is zero at any of the evaluated points, epochs and networks, so that in all of them.
All the minors of orders are computed at every point, without sampling. For , the number of minors of each order, , is , and for , and , and for . As in Equation (26), the concentration is first computed at each point and then averaged, never computed from averaged minors. The computation exploits the factorization of the minors of the MLP Jacobian given in Equation (12), , so that the determinants are computed once per network state and reused for all the points. All the computations are carried out in double precision. The derivatives and the minors are handled in logarithmic scale, so that even the smallest minors are represented without underflow and with full relative precision.
Regarding the statistics reported below, is summarized by its mean standard deviation across the four seeds. For the concentration, the four networks of a given width are evaluated on their own test sets, which gives test points in total. The concentration is summarized by the mean standard deviation of the pointwise values over these 4096 test points, so that this dispersion combines the variability between points and between tween seeds. Note that these 4096 test points, obtained by pooling the test sets of the four seeds (), should not be confused with the 4096 training points of each dataset (). They are different sets of points, and the two numbers coincide only numerically. The implementation code, the synthetic datasets, the weights of the trained networks at every epoch and all the computed values of are publicly available on GitHub at https://github.com/JuanCarlosFdezMlg/renyi-concentration-kag.
Predictive performance
Table 2 reports the coefficient of determination given by Equation (33) of both architectures on the training and test sets, at initialization and at the end of training.
| Epoch | Set | ||
| 0 | Training (4096) | ||
| 0 | Test (1024) | ||
| 500 | Training (4096) | ||
| 500 | Test (1024) |
Table 2. Coefficient of determination (mean standard deviation across the four seeds) of the MLPs with and hidden neurons. The number of points of each set is given in parentheses.
At initialization, both architectures yield negative values of , that is, their predictions are worse than the constant prediction . After 500 epochs, the network with reaches , whereas the network with remains at . At the end of training, the values obtained on the training and test sets are almost identical, with differences below 0.017 for both widths, so there is no sign of overfitting. Since both sets lead to the same conclusions, from now on we focus exclusively on the test sets, and all the quantities reported in the remainder of this section are computed on the test points only. The narrow network is also much more sensitive to the seed. Its test values range from 0.3869 to 0.7592, while those of the wide network range from 0.8519 to 0.9660. This is consistent with the larger number of parameters of the wider network, whose greater capacity tends to make the result of training less dependent on the initialization and on the order of the mini-batches.
Pointwise concentration at a fixed input
| Metric | Value at | Mean SD | Value at | Mean SD | |
| Epoch 0 (initialization) | |||||
| 1 | 0.0333 | 0.0695 0.0209 | 0.0305 | 0.0392 0.0037 | |
| 0.0613 | 0.1202 0.0353 | 0.0506 | 0.0687 0.0079 | ||
| 0.1062 | 0.1908 0.0538 | 0.0786 | 0.1122 0.0160 | ||
| 0.3083 | 0.3998 0.0708 | 0.2304 | 0.3037 0.0453 | ||
| 4.1385 | 3.6306 0.3008 | 8.1882 | 7.5901 0.2748 | ||
| 2 | 0.0406 | 0.0742 0.0265 | 0.0268 | 0.0373 0.0050 | |
| 0.0713 | 0.1294 0.0449 | 0.0464 | 0.0667 0.0098 | ||
| 0.1177 | 0.2058 0.0664 | 0.0744 | 0.1120 0.0186 | ||
| 0.3155 | 0.4179 0.0755 | 0.2035 | 0.3049 0.0389 | ||
| 7.0609 | 5.8711 0.8301 | 29.3957 | 25.6831 1.7130 | ||
| 3 | 0.0504 | 0.0934 0.0411 | 0.0272 | 0.0393 0.0061 | |
| 0.0946 | 0.1639 0.0689 | 0.0476 | 0.0714 0.0121 | ||
| 0.1642 | 0.2559 0.0994 | 0.0770 | 0.1216 0.0226 | ||
| 0.3937 | 0.4605 0.1076 | 0.2199 | 0.3201 0.0392 | ||
| 5.3777 | 4.5578 0.8580 | 50.7623 | 42.1749 3.9628 | ||
| Epoch 500 (end of training) | |||||
| 1 | 0.0961 | 0.1401 0.0585 | 0.2042 | 0.2478 0.0373 | |
| 0.1674 | 0.2195 0.0927 | 0.2687 | 0.3163 0.0426 | ||
| 0.2499 | 0.3018 0.1230 | 0.3303 | 0.3752 0.0444 | ||
| 0.4051 | 0.4525 0.1331 | 0.4433 | 0.4838 0.0442 | ||
| 3.2937 | 3.0888 0.5711 | 4.6109 | 4.1834 0.4262 | ||
| 2 | 0.1139 | 0.2062 0.0579 | 0.2647 | 0.3318 0.0534 | |
| 0.1811 | 0.3105 0.0813 | 0.3523 | 0.4352 0.0644 | ||
| 0.2587 | 0.4084 0.0950 | 0.4435 | 0.5204 0.0624 | ||
| 0.4439 | 0.5641 0.0827 | 0.5891 | 0.6412 0.0528 | ||
| 5.1674 | 3.7889 0.7717 | 7.6340 | 5.9160 1.3604 |
Table 4.3.
We first examine the concentration of the minors at an individual input point. We take as the first test point of seed 0, , with target . Since the datasets are shared across widths, is the same point for and . Table 3 reports for the networks of seed 0, for all the widths, orders and values of , at initialization and at the end of training, together with the Participation Ratio, , of Section 2.4. Next to each value, the table also reports the mean standard deviation of the corresponding metric over the 4096 test points of the four seeds, which places the value at within the distribution of the test set. Since the minors are computed exhaustively, is obtained at each point from through Equation (29).
Several features of Table 3 are worth highlighting. First, the monotonicity of with respect to is observed in every block, both at and on average. For every width, order and epoch, . The order therefore does not modify the information contained in the minors, but only the part of their distribution that is being summarized.
| Metric | Value at | Mean SD | Value at | Mean SD | |
| 3 | 0.1346 | 0.3323 | |||
| 3 | 0.2066 | 0.4323 | |||
| 3 | 0.2890 | 0.5281 | |||
| 3 | 0.4653 | 0.6770 | |||
| 3 | 4.1828 | 7.4476 |
Table 3. -RC metrics and Participation Ratio at the test point for the networks of seed 0, and their mean standard deviation over the 4096 test points of the four seeds.
Second, training increases the concentration of every network, for every order and every value of , both at and on average, and the Participation Ratio decreases accordingly in every case. Within each network, the increase grows with the order of the minors. For , the mean concentration of the network with grows from 0.1122 to 0.3752 for , from 0.1120 to 0.5204 for and from 0.1216 to 0.6384 for . For the network with , it grows from 0.1908 to 0.3018, from 0.2058 to 0.4084 and from 0.2559 to 0.4673, respectively.
Third, the rows of contain the same pointwise information as those of , since both are related by the monotone transformation of Equation (29). The difference lies in how each value is read. A value of the Participation Ratio can only be interpreted together with its upper bound , which depends on and . This bound is 4.90, 9.17 and 7.48 for , and 9.80, 38.57 and 70.43 for , for , respectively. In contrast, directly expresses the position of the same value between the two extremes of its own range. For instance, for and , decreases from 50.7623 to 7.4476 out of a maximum of 70.43, which corresponds to an increase of from 0.0770 to 0.5281. Similarly, for and , decreases from 5.3777 to 4.1828 out of a maximum of 7.48, and increases from 0.1642 to 0.2890.
Finally, in every row of the table the point is less concentrated than the average test point, with a lower value of and a higher value of , both at initialization and at the end of training. The value at is therefore not representative of the whole test set by itself, which illustrates the pointwise heterogeneity summarized by the dataset-level metric of Equation (26) and by its standard deviation.
Evolution of the concentration during training
Although the previous analysis could be extended to every width and every order along all the epochs, from now on we restrict the study to and for the sake of brevity. This is the configuration with the best predictive performance and the largest number of minors, . For every epoch, the mean and the variance of are computed over the 4096 test points of the four seeds. Figure 1 shows the evolution of the mean concentration for the four values of , with a common vertical scale. At initialization, the mean values are 0.0393, 0.0714, 0.1216 and 0.3201 for , 1, 2 and , respectively. The concentration increases rapidly during the first 60 epochs. For instance, the mean of exceeds 0.3 at epoch 31 and 0.5 at epoch 52. The growth then slows down until a maximum is reached between epochs 210 and 240, with values 0.4763 for and 0.5778 for (both at epoch 211), 0.6649 for (epoch 238) and 0.7763 for (epoch 212). From then on, the concentration decreases slightly, and at epoch 500 the means are 0.4198, 0.5426, 0.6384 and 0.7556, respectively. As expected from the monotonicity in , the curves never cross, and the gap between and remains of the same magnitude throughout training, from 0.25 at initialization to 0.21 at the end.
Figure 1. Mean of over the test points during training for .
The maximum of the concentration roughly coincides with that of the number of hidden neurons whose input weights are very small, below times the largest entry of . On average over the four seeds, there is no such neuron before epoch 65, their number reaches 13.25 out of 32 at epoch 227, and then decreases to 5.25 at epoch 500. By Equation (12), the minors that involve these neurons become negligible compared with the others, so that the magnitude concentrates on the minors of the remaining neurons.
The slight decrease of the concentration after its maximum agrees with the observations of Freedman & Mulligan [4], who report an extremum of their minor concentration metrics during training near a training of about 0.85, away from the point of peak performance (Figure 9 of that work). In our experiments, the maxima of are reached at values of between 0.84 and 0.86. As those authors point out, the cause of this extremum is not yet understood. We have checked that it is not caused by hidden neurons becoming active again. The fraction of hidden neurons that are locally inactive at the test points, with , grows from 0.1% at initialization to 11.4% at epoch 500, where it reaches its maximum.
Figure 2 relates this evolution to the predictive performance of the same networks. Each panel corresponds to a value of and contains one point per epoch. The vertical axis is the mean of the four seeds, and the colour of each point is the mean of at that epoch, on a magma scale fixed to and common to the four panels. The halo around each point has an area proportional to the variance of at that epoch, relative to the maximum variance reached in the panel. The legend of each panel shows the halos that correspond to one quarter, one half and the whole of that maximum variance. For instance, the maximum variance for is 0.0075, so its legend shows 0.0019, 0.0037 and 0.0075. The halo therefore encodes the dispersion of the concentration, not an interval for .

Figure 2. Joint evolution of the predictive performance and the concentration of the minors for and on the test sets.
During the first epochs, the mean remains close to zero, with at epoch 20 and 0.0456 at epoch 30, and then rises sharply. It exceeds 0.25 at epoch 38, 0.5 at epoch 82, 0.75 at epoch 158 and 0.9 at epoch 338, reaching 0.9274 at epoch 500. The change of colour of the points, from dark to bright tones, takes place along this rise. It is worth noting that the concentration starts to grow before departs from zero. The mean of increases from 0.1216 at initialization to more than 0.2 at epoch 19, while the predictions are still no better than the constant . In the final phase, in contrast, keeps increasing, from 0.8282 at epoch 200 to 0.9274 at epoch 500, while the concentration decreases slightly. As already observed by [4], the relationship between both quantities is therefore not monotone along the whole training.
The halos show that the dispersion of the concentration also grows during training. For , the standard deviation increases from 0.0226 at initialization to 0.0814 at epoch 500, and the maximum variance for , and is reached at epoch 486. For , the maximum variance is reached earlier, at epoch 111.
Discussion
The experiments illustrate how the proposed metric is read in practice. Each value of lies in and expresses the position of the distribution of minors between the two extremes attainable for its own number of minors, namely the uniform distribution and a single dominant minor. For the network with and , for instance, this shows that the mean of evolves from 0.1216, close to the uniform distribution, to 0.6384, an intermediate level of concentration. The Participation Ratio carries the same pointwise information for , but its values can only be interpreted together with the bound of each configuration.
Regarding the relationship with predictive performance in the network with , the growth of the concentration accompanies, and slightly precedes, the onset of learning. However, the concentration is not a monotone function of the performance, since it decreases slightly in the last phase of training while keeps improving. This agrees with the observation of [4] that the link between Kolmogorov–Arnold geometry and performance depends on the training regime.
The effect of the order can be interpreted through the factorization of Equation (12). Each minor of order contains the product of the derivatives of hidden neurons, so that when some of these derivatives become small at a point, the minors that involve the corresponding neurons are suppressed multiplicatively. This effect is amplified as grows, which is consistent with the larger increase in concentration observed for higher orders within each network. Even so, the concentration remains far from its maximum value. For and , the mean of the -RC metric never exceeds 0.7763 and is 0.7556 at epoch 500, so even the most mode-sensitive member of the class stays well below 1 and the magnitude is never carried by a single minor. At the other end of the evaluated orders, the mean of , which gives more weight to the small minors, never exceeds 0.4763 and is 0.4198 at epoch 500. The magnitude therefore concentrates on a subset of the minors without collapsing onto a single one, and no minor vanishes. This intermediate level of concentration is consistent with the observation of [4] that the proportion of zero rows in trained MLPs is well below that of the ideal Kolmogorov–Arnold construction.
Concerning the choice of , the qualitative conclusions above, namely the increase of the concentration with training and its larger increase for higher orders within each network, hold for the four evaluated orders. The order determines which part of the distribution of minors is summarized. The order gives more weight to the small minors, the Shannon () and collision () entropies provide intermediate summaries, the latter directly linked to the Participation Ratio, and the limit only depends on the largest minor. By monotonicity, the orders and delimit, at every point, the values of any order between them. The limit , which only depends on which minors vanish, is constant for the MLP with GeLU, as discussed in § 4.1.
Conclusions and Future Research
This paper has introduced the class of -RC metrics to measure the concentration of the minors of the Jacobian of the inner map, a central notion of Kolmogorov–Arnold Geometry. The metrics are built from the Rényi divergence of order between the normalized distribution of the minors and the uniform distribution, divided by its maximum value . As a result, every member of the class takes values in , and both extremes have the same meaning for every architecture. The value 0 corresponds, for every , to a uniform distribution of the minors, and the value 1 to a single minor carrying all the magnitude. The class is non-decreasing in , with equality between two orders only when the minors are uniform on their support, and the order selects which part of the distribution is summarized, from the small minors for to the largest minor for . For the metric is a closed-form transformation of the Participation Ratio, so it keeps the pointwise information of the Participation Ratio while removing its dependence on the number of minors.
The experiments with single-hidden-layer MLPs trained on a continuous XOR problem illustrate how the class is read in practice. Training increases the concentration for every width, every order and every value of , and the increase grows with . For the network with hidden neurons and , the mean of grows from 0.12 at initialization to a maximum of 0.66, and it starts to increase before the coefficient of determination departs from zero. The relationship with the predictive performance is not monotone, since the concentration decreases slightly in the last phase of training while the performance keeps improving, in agreement with the observations of [4]. The concentration reaches intermediate values, with the mean of below 0.78 throughout training, so the magnitude concentrates on a subset of the minors without collapsing onto a single one.
The proposed family of metrics has been applied to networks in which all the minors can be enumerated. Several directions remain open. The number of minors grows combinatorially with the width, the input dimension and the order , so the exact computation of the metrics is not feasible for realistic networks, and efficient estimation methods are needed. The early growth of the concentration also suggests that the metrics could be used not only to monitor training but also to guide it, for instance by adding a concentration term to the loss function or by interleaving gradient steps on the concentration, as proposed by [3].
CRediT authorship contribution statement
Juan Carlos Fernández: Conceptualization, Methodology, Software, Writing – original draft, Writing – review & editing. Asunción Jiménez-Cordero: Conceptualization, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing. José Francisco Paris: Conceptualization, Methodology, Supervision, Validation, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The authors thankfully acknowledge the computer resources (Picasso Supercomputer), technical expertise and assistance provided by the SCBI (Supercomputing and Bioinformatics) center of the University of Málaga. J. F. Paris is partially supported by the project PAR 11/2023 of the University of Málaga.
Data availability
The implementation code, the synthetic datasets, the weights of the trained networks at every epoch and all the computed values of are publicly available on GitHub at https://github.com/JuanCarlosFdezMlg/renyi-concentration-kag.
Declaration of generative AI and AI assisted technologies in the manuscript preparation process
During the preparation of this work the authors used Claude (Anthropic) and Codex (OpenAI) in order to assist with the implementation of the code for the computational experiments, to draft and revise parts of the text, and to improve the language and readability of the manuscript. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
References
- [1]Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. Wiley Series in Telecommunications and Signal Processing (2nd ed.). Hoboken, NJ: Wiley-Interscience. doi:10.1002/047174882X.DOI
- [2]van Erven, T., & Harremoës, P. (2014). Rényi divergence and Kullback–Leibler divergence. IEEE Transactions on Information Theory, 60, 3797–3820. doi:10.1109/TIT.2014.2320500.DOI
- [3]Freedman, M. H. (2024). The proof of Kolmogorov–Arnold may illuminate neural network learning. arXiv preprint arXiv:2410.08451. doi:10.48550/arXiv.2410.08451.DOI
- [4]Freedman, M. H., & Mulligan, M. (2025). Spontaneous Kolmogorov–Arnold geometry in shallow MLPs. arXiv preprint arXiv:2509.12326. doi:10.48550/arXiv.2509.12326.DOI
- [5]Girosi, F., & Poggio, T. (1989). Representation properties of networks: Kolmogorov’s theorem is irrelevant. Neural Computation, 1, 465–469. doi:10.1162/neco.1989.1.4.465.DOI
- [6]Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2021). Explainability in deep reinforcement learning. Knowledge-Based Systems, 214, 106685. doi:10.1016/j.knosys.2020.106685.DOI
- [7]Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations. Kolmogorov, A. N. (1957). On the representation of continuous functions of several variables by superpositions of continuous functions of one variable and addition. Doklady Akademii Nauk SSSR, 114, 953–956.
- [8]Kůrková, V. (1991). Kolmogorov’s theorem is relevant. Neural Computation, 3, 617–622. doi:10.1162/neco.1991.3.4.617.DOI
- [9]Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T. Y., & Tegmark, M. (2025). KAN: Kolmogorov–Arnold networks. In International Conference on Learning Representations (pp. 70367–70413).DOI
- [10]Morris, S. (2021). Hilbert 13: Are there any genuine continuous multivariate real-valued functions? Bulletin of the American Mathematical Society, 58, 107–118. doi:10.1090/bull/1698.DOI
- [11]Rényi, A. (1961). On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics (pp. 547–561). University of California Press.
- [12]Saxe, A. M., McClelland, J. L., & Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations.
