betaFeedback

WorksComputer Science

A Class of Rényi-based Concentration Metrics for Kolmogorov-Arnold Geometry in Multilayer Perceptrons

Abstract

The interpretability of artificial intelligence models has become a central concern, especially for black-box models such as multilayer perceptrons (MLPs). Kolmogorov-Arnold Geometry has recently emerged as a framework for analysing the internal representations of these networks, by examining how the hidden layer locally responds to changes in the input through the minors of its Jacobian. The degree to which a few of these minors dominate the rest, known as their concentration, has so far been measured with the Participation Ratio. Although this metric has useful properties, its maximum value depends on the size of the network, so a given value cannot be read as high or low concentration without reference to the architecture. To overcome this limitation, in this paper we propose the αα-RC metrics, a family of Rényi-based concentration measures that compare the normalized magnitudes of the minors with the uniform distribution through the Rényi divergence of order αα. All the metrics in the family are bounded between 0 and 1 regardless of the architecture, with 0 indicating that the magnitude is evenly spread over the minors and 1 that it is carried by a single one. Moreover, the metrics are non-decreasing in αα, and for αα = 2 they reduce to an explicit transformation of the Participation Ratio. Computational experiments with single-hidden-layer MLPs trained on a synthetic dataset show the benefits of our proposal.

Introduction

Understanding how neural networks internally represent information remains one of the central challenges in modern machine learning. While deep neural networks have achieved remarkable performance across a wide range of applications, their internal representations are still only partially understood. Developing mathematical tools capable of characterizing these representations is therefore an important step towards explaining why neural networks generalize successfully and under which conditions they exhibit efficient learning [6].

A recent line of research has revisited the celebrated Kolmogorov–Arnold representation theorem (Kolmogorov, 1957), originally proved by Kolmogorov and later refined by Arnold, which states that every continuous multivariate function on the multidimensional unit cube can be represented as a finite superposition of continuous univariate functions. Although this theorem is mathematically elegant, its practical impact on machine learning remained limited for decades due to the highly irregular nature of the functions involved in the representation [5, 8].

Interest in the theorem has recently been revived by the introduction of Kolmogorov–Arnold Networks (KANs) [9], a novel neural network architecture inspired by the representation theorem. Beyond their architectural design, these works have stimulated a renewed interest in understanding the geometric structure underlying the Kolmogorov–Arnold representation itself.

In this context, Freedman introduced the notion of Kolmogorov–Arnold Geometry (KAG), a geometric framework derived from the inner map appearing in the proof of the Kolmogorov–Arnold theorem [3]. The central idea is that the local geometric properties of this map provide valuable information about how different variables interact within the representation. More importantly, the author hypothesized that similar geometric patterns may emerge spontaneously in neural network architectures that are not explicitly based on the Kolmogorov–Arnold construction, particularly in single-hidden-layer multilayer perceptrons (MLPs).

To quantify these geometric properties, [4] proposed several metrics based on the Jacobian of the inner map and its exterior powers, whose entries are the k×kk \times k minors of the Jacobian. We say that these minors are concentrated when a few of them carry most of the total magnitude while the remaining ones contribute little. For instance, the magnitudes of the matrix of minors (9001)\begin{pmatrix} 9 & 0 \\ 0 & 1 \end{pmatrix} are more concentrated than those of (2.52.52.52.5)\begin{pmatrix} 2.5 & 2.5 \\ 2.5 & 2.5 \end{pmatrix}, although both add up to the same total. Among the proposed metrics, the Participation Ratio (PR) has received particular attention as a measure of this concentration, with lower values of the PR indicating a higher concentration of the minors. Experimental evidence suggests that, under certain training regimes, an increase in concentration with respect to initialization is associated with better model performance. Interestingly, this relationship appears only in specific settings, often referred to as the Goldilocks regime, while it disappears in others. Although the reasons for this phenomenon are not yet fully understood, these observations suggest that the PR captures relevant aspects of the Kolmogorov–Arnold Geometry emerging in multilayer perceptrons.

Despite these promising results, the interpretation of the Participation Ratio depends on the architecture under study. While its minimum value, attained when a single minor carries all the magnitude, is the same for every network, its maximum value, attained when the magnitude is uniformly spread over all minors, grows with the number of hidden neurons. Consequently, the same PR value may lie close to the maximum for a narrow network and far from it for a wider one, so its position within the attainable range, and therefore its interpretation in terms of concentration, changes with the architecture. (Freedman & Mulligan, 2025) report the ratio between the PR of the trained model and its value at initialization. This ratio measures how concentration changes during training for a given network, but it does not place the PR on a scale whose extremes have a fixed meaning.

The main objective of this paper is to address this limitation. We introduce a family of normalized concentration metrics based on the Rényi entropy of order α\alpha, applied to the probability distribution obtained by normalizing the magnitudes of the minors. The minimum and maximum values of these metrics are 0 and 1, regardless of the architecture, and so is their interpretation: a value of 0 corresponds, for every order α>0\alpha> 0, to minimum concentration, with the magnitude uniformly spread over all minors, and a value of 1 to maximum concentration, with all the magnitude carried by a single minor. The proposed metric is monotonically non-decreasing with respect to α\alpha, and the order α\alpha determines the weight given to the different parts of the underlying distribution: small values of α\alpha take into account the proportion of non-zero entries, whereas large values emphasize its mode. In addition, for α=2\alpha= 2 the metric can be written as a function of the Participation Ratio, establishing a direct theoretical connection between both quantities.

Finally, we illustrate the behaviour of the proposed metric through a series of computational experiments on single-hidden-layer MLPs trained on a continuous version of the XOR problem. We show that, unlike the PR, the proposed metric places the concentration of the minors on a scale whose extremes have the same meaning for every architecture, and we analyse how the choice of α\alpha affects the resulting values.

The Kolmogorov–Arnold Geometry

This section reviews the main concepts underlying the notion of Kolmogorov–Arnold Geometry (KAG), recently introduced by Freedman [3] as a geometric framework inspired by the proof of the Kolmogorov–Arnold representation theorem. We first describe the geometric intuition behind this concept in Section 2.1 and then present the mathematical objects used to quantify it, namely the Jacobian and its exterior powers, in Section 2.2. Section 2.3 particularizes these objects to single-hidden-layer MLPs, the architecture studied in this work, by decomposing the network into an inner and an outer map and computing the minors of the Jacobian of its inner map. Finally, Section 2.4 describes a particular metric proposed by [4], called the Participation Ratio, which measures the concentration of these minors.

From the Kolmogorov–Arnold theorem to Kolmogorov–Arnold Geometry

The Kolmogorov–Arnold representation theorem is one of the most celebrated results in approximation theory. Roughly speaking, it states that every continuous multivariate function on the multidimensional unit cube can be represented as a finite superposition of continuous univariate functions. While several variants and refinements of this theorem have been proposed in the literature, in this work we consider the formulation by [10], which can be stated as follows:

Theorem 1 (Kolmogorov–Arnold Theorem). Let n∈Nn \in\mathbb{N} with n≥2n \ge2. There exist real numbers λ1,λ2,…,λn\lambda_1,\lambda_2,\ldots,\lambda_n and Lipschitz (and thus continuous) functions φj:[0,1]→R\varphi_j : [0,1] \to\mathbb{R}, j=1,…,mj = 1,\ldots,m, with m≥2n+1m \ge2n+1, such that for every continuous function f:[0,1]n→Rf : [0,1]^n \to\mathbb{R}, there exists a continuous function g:R→Rg : \mathbb{R} \to\mathbb{R} satisfying, for each (x1,x2,…,xn)∈[0,1]n(x_1,x_2,\ldots,x_n) \in[0,1]^n,

f(x1,x2,…,xn)=∑j=1mg(λ1φj(x1)+⋯+λnφj(xn)).(1)f(x_1,x_2,\ldots,x_n) = \sum_{j=1}^{m} g(\lambda_1\varphi_j(x_1)+\cdots+\lambda_n\varphi_j(x_n)). \tag*{(1)}

Furthermore, the constants λi\lambda_i and the functions φj\varphi_j can be chosen independently of the function ff.

We omit the formal proof of Theorem 1 here. However, the interested reader is referred to [10] for a detailed proof and discussion.

Theorem 1 can be read as a decomposition of ff into two maps. The inner map ΦKA:[0,1]n→Rm\Phi^{\mathrm{KA}} : [0,1]^n \to\mathbb{R}^m is defined componentwise by

ΦjKA(x)=∑i=1nλiφj(xi),j=1,…,m,(2)\Phi_j^{\mathrm{KA}}(x) = \sum_{i=1}^{n} \lambda_i\varphi_j(x_i), \qquad j = 1,\ldots,m, \tag*{(2)}

and the outer map gKA:Rm→Rg^{\mathrm{KA}}:\mathbb{R}^{m}\to\mathbb{R} by

gKA(h)=∑j=1mg(hj),h=(h1,…,hm)T∈Rm.(3)g^{\mathrm{KA}}(h)=\sum_{j=1}^{m}g(h_j),\qquad h=(h_1,\ldots,h_m)^{T}\in\mathbb{R}^{m}. \tag*{(3)}

With this notation, we define the KA representation fKA:=gKA∘ΦKAf^{\mathrm{KA}}:=g^{\mathrm{KA}}\circ\Phi^{\mathrm{KA}}, and Theorem 1 states that fKA=ff^{\mathrm{KA}}=f on [0,1]n[0,1]^n. Since the constants λi\lambda_i and the functions φj\varphi_j do not depend on ff, the inner map ΦKA\Phi^{\mathrm{KA}} is universal: the same map serves for every continuous function, and only the outer map, through gg, depends on the target function.

Regarding its regularity, each component ΦjKA\Phi_j^{\mathrm{KA}} is a finite linear combination of Lipschitz functions, and hence Lipschitz. Therefore ΦKA\Phi^{\mathrm{KA}} is a Lipschitz map, and by Rademacher’s theorem it is differentiable almost everywhere in [0,1]n[0,1]^n with respect to the Lebesgue measure. Its Jacobian matrix, and hence all the quantities derived from it in Section 2.2, are therefore defined almost everywhere, but not necessarily at every point. Throughout this work, the Jacobian of ΦKA\Phi^{\mathrm{KA}} is considered only at points where ΦKA\Phi^{\mathrm{KA}} is differentiable.

Beyond its theoretical guarantee of universal approximation, Theorem 1 admits a compelling geometric interpretation through the structure of the inner map ΦKA\Phi^{\mathrm{KA}}. The recent work of [4] characterizes the local geometric properties of this map, a framework referred to as Kolmogorov–Arnold geometry (KAG). It does so through differential quantities, mainly the Jacobian matrix and its exterior powers (Section 2.2), which provide information about how the hidden representation varies with respect to the input variables.

Jacobians, exterior powers and minor concentration

Consider the inner map ΦKA:[0,1]n→Rm\Phi^{\mathrm{KA}}:[0,1]^n\to\mathbb{R}^{m} defined in (2), where nn denotes the input dimension and mm the number of hidden coordinates. Given a point x~∈[0,1]n\tilde{x}\in[0,1]^n at which ΦKA\Phi^{\mathrm{KA}} is differentiable (see Section 2.1), the Jacobian matrix JΦKA(x~)∈Mm×n(R)J_{\Phi^{\mathrm{KA}}}(\tilde{x})\in\mathcal{M}_{m\times n}(\mathbb{R}) describes the local sensitivity of each hidden coordinate with respect to the input variables at x~\tilde{x}. In other words, each row of the Jacobian characterizes how the corresponding hidden coordinate locally reacts to infinitesimal perturbations of the input. Specifically, the (j,i)(j,i)-th entry of the Jacobian matrix is the partial derivative of the jj-th component function with respect to the ii-th input variable evaluated at x~\tilde{x}, namely

[JΦKA(x~)]ji≔∂ΦjKA(x~)∂xi,j=1,…,m,i=1,…,n.(4)[J_{\Phi_{\mathrm{KA}}}(\tilde{x})]_{ji} \coloneqq\frac{\partial\Phi^{\mathrm{KA}}_j(\tilde{x})}{\partial x_i}, \qquad j=1,\ldots,m,\quad i=1,\ldots,n. \tag*{(4)}

An important consequence of the proof of the Kolmogorov–Arnold theorem (Theorem 1) is that, in the ideal construction, many rows of the Jacobian vanish. Geometrically, this means that several hidden neurons become locally insensitive to perturbations of the input, producing a highly structured differential representation.

However, analysing individual neurons is not sufficient to characterize the complete geometry. It is also necessary to study how groups of hidden neurons jointly respond to variations of the input. To this end, [4] considered the exterior powers of the Jacobian matrix. The entries of the kk-th exterior power correspond to the minors of order kk, which quantify the joint variation of subsets of hidden neurons with respect to subsets of input variables.

To formalize this notion, we introduce the definition of the kk-th exterior power of a matrix, which collects all possible k×kk \times k minors into a single operator.

Definition 1. Let A∈Mm×n(R)A \in\mathcal{M}_{m \times n}(\mathbb{R}) and let 1≤k≤min⁡(m,n)1 \le k \le\min(m,n). Define the ordered index sets of cardinality kk as

Jk,m={J=(j1,j2,…,jk)∈{1,…,m}k:1≤j1<j2<⋯<jk≤m},(5)\mathcal{J}_{k,m} = \left\{J=(j_1,j_2,\ldots,j_k)\in\{1,\ldots,m\}^{k}:1\le j_1<j_2<\cdots<j_k\le m\right\}, \tag*{(5)}

and similarly Ik,n\mathcal{I}_{k,n} for subsets of {1,…,n}\{1,\ldots,n\} of size kk. For any pair of index sets J∈Jk,mJ \in\mathcal{J}_{k,m} and I∈Ik,nI \in\mathcal{I}_{k,n}, let AJ,I∈Mk×k(R)A_{J,I} \in\mathcal{M}_{k \times k}(\mathbb{R}) denote the submatrix formed by selecting the rows indexed by JJ and the columns indexed by II.

The kk-th exterior power of AA, denoted by PAk∈M(mk)×(nk)(R)P_A^k \in\mathcal{M}_{\binom{m}{k} \times\binom{n}{k}}(\mathbb{R}), is the matrix whose entry corresponding to the row index J∈Jk,mJ \in\mathcal{J}_{k,m} and column index I∈Ik,nI \in\mathcal{I}_{k,n}

is given by the determinant of AJ,IA_{J,I}:

[PAk]J,I=det⁡(AJ,I)=det⁡(Aj1,i1Aj1,i2⋯Aj1,ikAj2,i1Aj2,i2⋯Aj2,ik⋮⋮⋱⋮Ajk,i1Ajk,i2⋯Ajk,ik)(6)\left[P_A^k\right]_{J,I}=\det(A_{J,I})=\det\begin{pmatrix} A_{j_1,i_1} & A_{j_1,i_2} & \cdots& A_{j_1,i_k} \\ A_{j_2,i_1} & A_{j_2,i_2} & \cdots& A_{j_2,i_k} \\ \vdots& \vdots& \ddots& \vdots\\ A_{j_k,i_1} & A_{j_k,i_2} & \cdots& A_{j_k,i_k} \end{pmatrix} \tag*{(6)}

Applying Definition 1 to the Jacobian matrix JΦ(x)J_{\Phi}(x) yields its kk-th exterior power PJΦ(x)kP^k_{J_{\Phi}(x)}, whose entries admit a natural geometric interpretation in terms of volume changes. For a linear map T:Rn→RnT:\mathbb{R}^n \to\mathbb{R}^n given by T(x)=AxT(x)=Ax, with A∈Mn×n(R)A \in\mathcal{M}_{n \times n}(\mathbb{R}), an infinitesimal volume element dVdV is mapped to an element of volume ∣det⁡(A)∣dV\lvert\det(A)\rvert dV. For a nonlinear map, the same holds locally around a point x~\tilde{x}, with AA replaced by the Jacobian matrix at x~\tilde{x}. This is the Jacobian determinant that appears in the change of variables formula for multiple integrals.

Minors extend this idea to rectangular Jacobians and to subsets of variables. The minor [PJΦ(x)k]J,I\left[P^k_{J_{\Phi}(x)}\right]_{J,I} is the factor by which the oriented kk-dimensional volume of an infinitesimal parallelepiped spanned by the input directions I=(i1,…,ik)I=(i_1,\ldots,i_k) changes when it is mapped by JΦ(x)J_{\Phi}(x) and projected onto the hidden coordinates J=(j1,…,jk)J=(j_1,\ldots,j_k). Its absolute value measures the magnitude of the volume change, while its sign indicates orientation. For k=1k=1, the minors reduce to the entries of the Jacobian, and if m=nm=n and k=nk=n, the only minor is the full Jacobian determinant, a case that does not arise in this work, where m>nm>n. For instance, for a map with two inputs and three hidden coordinates, the three minors of order two are the oriented areas of the projections of the image of an infinitesimal square onto the planes spanned by the hidden coordinates (1,2)(1,2), (1,3)(1,3) and (2,3)(2,3).

The distribution of these minors plays a fundamental role in the characterization of the Kolmogorov–Arnold Geometry. When only a relatively small number of minors concentrates most of the total magnitude, the geometry is said to exhibit a high degree of concentration. Conversely, if the magnitude is distributed more uniformly among the minors, the corresponding geometric structure is considered less pronounced.

MLP and its Kolmogorov–Arnold Geometry

A fundamental question raised by [4] is whether Kolmogorov–Arnold geometry emerges spontaneously during the standard training of conventional neural network architectures. Unlike the construction underlying Theorem 1, where the inner functions are explicitly designed, standard multilayer perceptrons (MLPs) do not impose any KA structure a priori. In this context, the emergence of KA geometry does not refer to the appearance of exact zeros. It refers to the development, during training, of patterns similar to those of the KA construction: Jacobian rows that are approximately zero, and a distribution of minors in which a few of them carry most of the total magnitude. These patterns are assessed by comparing the trained model with its initialization.

To evaluate whether a trained MLP exhibits a geometric structure compatible with the KA framework, the network is first decomposed into an inner map and an outer map. In the KA representation (Theorem 1), the inner map embeds the input domain into a higher-dimensional space, where a relatively simple outer map reconstructs the target function.

We consider a single-hidden-layer MLP (shallow MLP) with nn inputs, mm hidden neurons and a single linear output neuron. Let x=(x1,…,xn)T∈Rn\mathbf{x}=(x_1,\ldots,x_n)^T\in\mathbb{R}^n denote the input vector. The network is parameterized by:

  • an input weight matrix A∈Mm×n(R)A\in\mathcal{M}_{m\times n}(\mathbb{R}) and a bias vector a∈Rma\in\mathbb{R}^m;

  • an output weight vector B∈M1×m(R)B\in\mathcal{M}_{1\times m}(\mathbb{R}) and a scalar bias b∈Rb\in\mathbb{R};

  • a smooth activation function σ:R→R\sigma:\mathbb{R}\to\mathbb{R}. In this work we use the GeLU activation, σ(t)=tF(t)\sigma(t)=tF(t), where FF denotes the cumulative distribution function of the standard normal distribution. We denote by σ:Rm→Rm\boldsymbol{\sigma}:\mathbb{R}^m\to\mathbb{R}^m its componentwise extension, σ(z)=(σ(z1),…,σ(zm))T\boldsymbol{\sigma}(\mathbf{z})=(\sigma(z_1),\ldots,\sigma(z_m))^T for z=(z1,…,zm)T∈Rm\mathbf{z}=(z_1,\ldots,z_m)^T\in\mathbb{R}^m.

The input–output mapping fMLP:Rn→Rf^{\mathrm{MLP}} : \mathbb{R}^{n} \to\mathbb{R} is then given by

fMLP(x)=B σ(Ax+a)+b.(7)f^{\mathrm{MLP}}(x) = B\,\boldsymbol{\sigma}(Ax+a)+b. \tag*{(7)}

Following the KA framework, we decompose fMLPf^{\mathrm{MLP}} into two stages:

  1. Inner map ΦMLP\Phi^{\mathrm{MLP}}:

ΦMLP:Rn⟶Rm,ΦMLP(x)=σ(z(x)),(8)\Phi^{\mathrm{MLP}} : \mathbb{R}^{n} \longrightarrow\mathbb{R}^{m}, \qquad\Phi^{\mathrm{MLP}}(x)=\boldsymbol{\sigma}(z(x)), \tag*{(8)}

where z(x)=Ax+a∈Rmz(x)=Ax+a\in\mathbb{R}^{m} is the pre-activation vector.

  1. Outer map gMLPg^{\mathrm{MLP}}: the affine map acting on the hidden representation h=ΦMLP(x)h=\Phi^{\mathrm{MLP}}(x),

gMLP:Rm⟶R,gMLP(h)=Bh+b.(9)g^{\mathrm{MLP}} : \mathbb{R}^{m} \longrightarrow\mathbb{R}, \qquad g^{\mathrm{MLP}}(h)=Bh+b. \tag*{(9)}

With this notation, fMLP=gMLP∘ΦMLPf^{\mathrm{MLP}}=g^{\mathrm{MLP}}\circ\Phi^{\mathrm{MLP}}, in direct analogy with the decomposition fKA=gKA∘ΦKAf^{\mathrm{KA}}=g^{\mathrm{KA}}\circ\Phi^{\mathrm{KA}} of § 2.1.

Since GeLU is smooth, ΦMLP\Phi^{\mathrm{MLP}} is differentiable at every point. Its Jacobian is therefore defined everywhere, unlike that of ΦKA\Phi^{\mathrm{KA}}, which, as discussed in § 2.1, is only guaranteed to exist almost everywhere. For non-differentiable activations such as ReLU, the Jacobian may not exist at points where some pre-activation crosses zero, and a convention would be required to compute the metrics. Using the tools of Section 2.2, the entries of the Jacobian matrix JΦMLP(x)∈Mm×n(R)J_{\Phi^{\mathrm{MLP}}}(x)\in\mathcal{M}_{m\times n}(\mathbb{R}) at a point xx are:

[JΦMLP(x)]ji≔∂ΦjMLP(x)∂xi=σ′(zj(x))Aji,j=1,…,m,i=1,…,n,(10)\left[J_{\Phi^{\mathrm{MLP}}}(x)\right]_{ji}\coloneqq\frac{\partial\Phi^{\mathrm{MLP}}_j(x)}{\partial x_i}=\sigma'(z_j(x))A_{ji},\qquad j=1,\ldots,m,\quad i=1,\ldots,n, \tag*{(10)}

where zj(x)=∑i=1nAjixi+ajz_j(x)=\sum_{i=1}^{n}A_{ji}x_i+a_j is the pre-activation of the jj-th hidden neuron and σ′\sigma' is the derivative of the scalar activation function. For GeLU,

σ′(t)=Φ(t)+t12πe−t2/2.(11)\sigma'(t)=\Phi(t)+t\frac{1}{\sqrt{2\pi}}e^{-t^2/2}. \tag*{(11)}

Equivalently, JΦMLP(x)=D(x)AJ_{\Phi^{\mathrm{MLP}}}(x)=D(x)A, where D(x)=diag⁡(σ′(z1(x)),…,σ′(zm(x)))D(x)=\operatorname{diag}\left(\sigma'(z_1(x)),\ldots,\sigma'(z_m(x))\right).

In an MLP, row jj of JΦMLP(x)J_{\Phi^{\mathrm{MLP}}}(x) is approximately zero for one of two reasons. Either the weight vector Aj⋅A_{j\cdot} is close to zero (for instance, due to weight sparsity or dead units), or σ′(zj(x))\sigma'(z_j(x)) is close to zero at xx, as happens with GeLU for sufficiently negative pre-activations. The first mechanism affects the row at every input, whereas the second depends on xx. It is this input-dependent mechanism that resembles the local insensitivity of hidden coordinates in the KA construction.

Finally, for 1≤k≤min⁡(m,n)1 \le k \le\min(m,n), the entries of the kk-th exterior power of the MLP Jacobian, PJΦMLP(x)k∈M(mk)×(nk)(R)P^k_{J_{\Phi^{\mathrm{MLP}}(x)}} \in\mathcal{M}^{\binom{m}{k} \times\binom{n}{k}}(\mathbb{R}) (Definition 1), are

[PJΦMLP(x)k]J,I=det⁡(σ′(zj1(x))Aj1,i1⋯σ′(zj1(x))Aj1,ik⋮⋱⋮σ′(zjk(x))Ajk,i1⋯σ′(zjk(x))Ajk,ik)=(∏l=1kσ′(zjl(x)))det⁡(AJ,I),(12)\begin{aligned} \left[P^k_{J_{\Phi^{\mathrm{MLP}}(x)}}\right]_{J,I} = \det\begin{pmatrix} \sigma'(z_{j_1}(x))A_{j_1,i_1} & \cdots& \sigma'(z_{j_1}(x))A_{j_1,i_k} \\ \vdots& \ddots& \vdots\\ \sigma'(z_{j_k}(x))A_{j_k,i_1} & \cdots& \sigma'(z_{j_k}(x))A_{j_k,i_k} \end{pmatrix} \\ &= \left(\prod_{l=1}^{k}\sigma'(z_{j_l}(x))\right)\det(A_{J,I}), \tag*{(12)} \end{aligned}

where J=(j1,…,jk)∈Jk,mJ=(j_1,\ldots,j_k)\in\mathcal{J}_{k,m} and I=(i1,…,ik)∈Ik,nI=(i_1,\ldots,i_k)\in\mathcal{I}_{k,n}. The second equality follows from the multilinearity of the determinant with respect to its rows, and shows that a hidden neuron with σ′(zj(x))≈0\sigma'(z_j(x))\approx0 makes every minor that involves row jj approximately zero. These exterior powers quantify how the inner map of the MLP locally transforms kk-dimensional volume elements, providing the basis for measuring minor concentration in neural networks. For simplicity, throughout the rest of this work we denote PJΦMLP(x)kP^k_{J_{\Phi^{\mathrm{MLP}}(x)}} by J(k)(x)J^{(k)}(x).

Participation Ratio

Building on the exterior powers introduced in § 2.3, we now quantify the concentration of the minors of the inner map ΦMLP\Phi^{\mathrm{MLP}} defined in Equation (8). To this end, [4] introduced the Participation Ratio (PR), one of the main quantitative tools for detecting the spontaneous emergence of KA geometry in multilayer perceptrons. The PR measures whether the total magnitude of the minors in J(k)(x)J^{(k)}(x) is concentrated in a few of them or spread across many.

Definition 2 (Participation Ratio). Let x∈Rnx\in\mathbb{R}^n be an input point such that

J(k)(x)≠0J^{(k)}(x) \ne0. The order-kk Participation Ratio at xx is defined as

PR(k)(x):=∥J(k)(x)∥1∥J(k)(x)∥2,(13)\mathrm{PR}^{(k)}(x) := \frac{\lVert J^{(k)}(x)\rVert_1}{\lVert J^{(k)}(x)\rVert_2}, \tag*{(13)}

where

∥J(k)(x)∥1=∑J∈Jk,m∑I∈Ik,n∣JJ,I(k)(x)∣\lVert J^{(k)}(x)\rVert_1 = \sum_{J\in\mathcal{J}_{k,m}} \sum_{I\in\mathcal{I}_{k,n}} \left|J^{(k)}_{J,I}(x)\right|

is the entrywise ℓ1\ell_1 norm and

∥J(k)(x)∥2=(∑J∈Jk,m∑I∈Ik,n∣JJ,I(k)(x)∣2)1/2\lVert J^{(k)}(x)\rVert_2 = \left(\sum_{J\in\mathcal{J}_{k,m}} \sum_{I\in\mathcal{I}_{k,n}} \left|J^{(k)}_{J,I}(x)\right|^2\right)^{1/2}

is the entrywise ℓ2\ell_2 (Frobenius) norm. For an evaluation dataset D={x(1),…,x(S)}\mathcal{D} = \{x^{(1)}, \ldots, x^{(S)}\}, the global Participation Ratio is defined as the mean over all points of the dataset,

PR(k):=1S∑s=1SPR(k)(x(s)).(14)\mathrm{PR}^{(k)} := \frac{1}{S}\sum_{s=1}^{S}\mathrm{PR}^{(k)}(x^{(s)}). \tag*{(14)}

Let Nk=(mk)(nk)N_k = \binom{m}{k}\binom{n}{k} denote the total number of minors of order kk. Viewing J(k)(x)J^{(k)}(x) as a vector vv with NkN_k entries, the inequalities ∥v∥2≤∥v∥1≤Nk∥v∥2\lVert v\rVert_2 \le\lVert v\rVert_1 \le\sqrt{N_k}\lVert v\rVert_2 hold, the second one being a consequence of the Cauchy–Schwarz inequality. Hence

1≤PR(k)(x)≤Nk.(15)1 \le\mathrm{PR}^{(k)}(x) \le\sqrt{N_k}. \tag*{(15)}

The lower bound is attained if and only if exactly one minor is non-zero, and the upper bound if and only if all minors have the same absolute value. Note that, while the lower bound is the same for every architecture, the upper bound depends on NkN_k, and hence on the number of hidden neurons mm. We return to this point in § 3.

The value of PR(k)(x)\mathrm{PR}^{(k)}(x) can therefore be interpreted as follows:

  • Values close to 1 indicate high concentration: a few minors carry most of the total magnitude, and PR(k)(x)=1\mathrm{PR}^{(k)}(x) = 1 when a single minor carries all of it. This is the kind of pattern expected from the KA construction.

  • Values close to Nk\sqrt{N_k} indicate low concentration: the magnitude is spread almost uniformly across the minors, as expected in the absence of KA structure.

Finally, observe that if J(k)(x)=0J^{(k)}(x)=0, both norms vanish and Equation (13) is not defined. This can happen even when the Jacobian itself is non-zero, for instance when the rank of JΦMLP(x)J_{\Phi_{\mathrm{MLP}}}(x) is smaller than kk. For such points we adopt the convention PR(k)(x)=Nk\mathrm{PR}^{(k)}(x)=\sqrt{N_k}, which corresponds to the absence of concentration and is consistent with the convention that will be adopted for the α\alpha-RC metrics in Definition 3.

To illustrate this metric in practice, the following example demonstrates how the Participation Ratio is computed for two contrasting scenarios.

Example 1 (Calculation of the Participation Ratio). Consider two cases with Nk=4N_k=4 minors of order kk evaluated at a point xx, whose absolute values are

∣JA(k)(x)∣=(1111),∣JB(k)(x)∣=(4000).\lvert J_A^{(k)}(x)\rvert= \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}, \qquad \lvert J_B^{(k)}(x)\rvert= \begin{pmatrix} 4 & 0 \\ 0 & 0 \end{pmatrix}.

Both matrices have the same ℓ1\ell_1 norm, ∥JA(k)(x)∥1=∥JB(k)(x)∥1=4\lVert J_A^{(k)}(x)\rVert_1=\lVert J_B^{(k)}(x)\rVert_1=4, but their ℓ2\ell_2 norms differ: ∥JA(k)(x)∥2=2\lVert J_A^{(k)}(x)\rVert_2=2 and ∥JB(k)(x)∥2=4\lVert J_B^{(k)}(x)\rVert_2=4. Therefore

PRA(k)(x)=42=2=4,PRB(k)(x)=44=1.\mathrm{PR}_A^{(k)}(x)=\frac{4}{2}=2=\sqrt{4}, \qquad \mathrm{PR}_B^{(k)}(x)=\frac{4}{4}=1.

In the first case the magnitude is spread uniformly across all four minors, and the PR attains its maximum value Nk=2\sqrt{N_k}=2. In the second case all the magnitude is concentrated in a single minor, and the PR attains its minimum value 1.

A novel improved metric

In this section we propose a novel metric that circumvents the limitations of the Participation Ratio metric to assess the concentration of the matrix J(k)(x)J^{(k)}(x).

This section is organized in two parts. Section 3.1 revisits the Participation Ratio and illustrates, through a simple example, why its dependence on the network architecture prevents comparing minor concentration across networks of different width. Section 3.2 then introduces the proposed Rényi-based concentration metric: it first normalizes the exterior power values, recalls the Rényi divergence and its main properties, and formally defines the resulting class of α\alpha-RC metrics. It then shows that the particular case α=2\alpha= 2 is directly related to the Participation Ratio, and closes with an illustrative example.

Motivating example

As established in the previous section, the Participation Ratio PR(k)(x)\mathrm{PR}^{(k)}(x) proposed by [4] possesses several appealing properties for quantifying higher-order interactions. It offers an intuitive geometric interpretation of active minors, has well-defined theoretical bounds (1≤PR(k)(x)≤Nk1 \le\mathrm{PR}^{(k)}(x) \le\sqrt{N_k}), and captures the degree of minor concentration at an individual input point xx.

Despite these strengths, the Participation Ratio suffers from a fundamental limitation: its range and its interpretation in the upper extreme limit are architecture-dependent. Because the theoretical upper bound Nk=(mk)(nk)\sqrt{N_k} = \sqrt{\binom{m}{k}\binom{n}{k}} depends explicitly on the number of hidden units mm, the scale of PR(k)(x)\mathrm{PR}^{(k)}(x) varies dynamically with the network capacity. Consequently, the same numerical PR value can carry completely opposite geometric meanings depending on the size of the network.

To clearly illustrate this scale-dependency issue, consider the following example.

Example 2 (Incompatibility of PR across architectural scales). Suppose we evaluate the order-kk minor concentration of two trained neural networks with different hidden layer widths at an input point xx:

  • Network A (Smaller architecture, Nk=2N_k = 2): The matrix of order-kk minors is given by:

JA(k)(x)=(11),J_A^{(k)}(x) = \begin{pmatrix} 1 & 1 \end{pmatrix},

where Nk=2N_k = 2 total minors exist. Computing the norms yields ∥JA(k)(x)∥1=2\lVert J_A^{(k)}(x) \rVert_1 = 2 and ∥JA(k)(x)∥2=12+12=2\lVert J_A^{(k)}(x) \rVert_2 = \sqrt{1^2 + 1^2} = \sqrt{2}, yielding

PRA(k)(x)=22=2.\mathrm{PR}_A^{(k)}(x) = \frac{2}{\sqrt{2}} = \sqrt{2}.

Here, activity is uniformly distributed across all possible minors (PRA(k)(x)=Nk\mathrm{PR}_{A}^{(k)}(x) = \sqrt{N_k}), representing maximum dispersion (zero geometric concentration).

  • Network B (Larger architecture, Nk=16N_k = 16): The matrix of order-kk minors is given by:

JB(k)(x)=(1100000000000000),J_{B}^{(k)}(x) = \begin{pmatrix} 1 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix},

where Nk=16N_k = 16 total minors exist. Computing the norms yields ∥JB(k)(x)∥1=2\lVert J_{B}^{(k)}(x)\rVert_1 = 2 and ∥JB(k)(x)∥2=12+12=2\lVert J_{B}^{(k)}(x)\rVert_2 = \sqrt{1^2 + 1^2} = \sqrt{2}, giving

PRB(k)(x)=22=2.\mathrm{PR}_{B}^{(k)}(x) = \frac{2}{\sqrt{2}} = \sqrt{2}.

Here, activity is concentrated in only 22 out of 1616 minors, representing strong geometric concentration relative to the network’s capacity (far below its maximum bound of 16=4\sqrt{16} = 4).

Despite operating under drastically different regimes, total uniform dispersion in Network A versus localized specialization in Network B, both examples yield the exact same Participation Ratio (PR(k)(x)=2\mathrm{PR}^{(k)}(x) = \sqrt{2}). Therefore, raw PR values fail to provide a meaningful basis for comparison across different scenarios.

The main motivation of this work is finding a concentration metric more flexible than PR with a simpler interpretation across architectures. To address both issues, we propose an alternative metric based on the Rényi entropy of order α\alpha. This metric is normalized so that its extreme values have the same meaning for every architecture, regardless of NkN_k. On the one hand, for every order α>0\alpha> 0, the value 00 corresponds to a uniform distribution of the magnitude across all minors, whereas the value 11 is associated with a single non-zero minor. Concentration is thus measured relative to the maximum dispersion attainable by each network, rather than on a scale that changes with its width. It also has a parameter α\alpha which provides flexibility in the interpretation of such concentration. In the particular case of α=2\alpha= 2 this new metric is directly expressible as a function of Freedman’s original Participation Ratio.

Rényi-based concentration metric

In Section 3.1, we analysed the PR as a measure of activity distribution across minors. However, as commented above, PR exhibits a fundamental quantitative limitation when evaluating minor concentration across different multilayer perceptron architectures. In particular, the maximum value of PR(k)(x)\mathrm{PR}^{(k)}(x) is given by Nk\sqrt{N_k}, where Nk=(mk)(nk)N_k = \binom{m}{k}\binom{n}{k} denotes the total number of minors of order kk. Consequently, changing the width mm of a hidden layer alters the baseline scale of PR, making direct comparisons between different network architectures impossible.

To overcome this limitation, we propose a novel concentration metric based on Rényi entropy of order α\alpha [11]. This metric maps the activity distribution to a normalized concentration score bounded within the interval [0,1][0,1] for any network architecture. Also, the value zero is interpreted as minimum concentration (uniform distribution) and the value one always as maximum concentration (only one non-zero element).

Normalization of Exterior Power Values

To apply an information-theoretic formulation, we first must interpret the matrix of minors J(k)(x)J^{(k)}(x) as a discrete probability distribution. This point of view is key for our goals, since the appropriate normalization of the absolute values of the entries of J(k)(x)J^{(k)}(x) can be viewed as a two-dimensional discrete Probability Mass Function (PMF).

Given an input point x∈Rnx \in\mathbb{R}^{n}, let Nk=(mk)(nk)N_k = \binom{m}{k}\binom{n}{k} be the total number of minors of order kk, indexed by the multi-index sets J∈Jk,m\mathcal{J} \in\mathcal{J}_{k,m} and I∈Ik,nI \in\mathcal{I}_{k,n}. Normalizing the absolute values of the entries of J(k)(x)J^{(k)}(x) by its ℓ1\ell_1-norm yields the relative weight pJ,I(k)(x)p_{J,I}^{(k)}(x) associated with the minor JJ,I(k)(x)J_{J,I}^{(k)}(x):

pJ,I(k)(x)=∣JJ,I(k)(x)∣∥J(k)(x)∥1=∣JJ,I(k)(x)∣∑J′∈Jk,m∑I′∈Ik,n∣JJ′,I′(k)(x)∣(16)p_{J,I}^{(k)}(x)=\frac{\left|J_{J,I}^{(k)}(x)\right|}{\left\lVert J^{(k)}(x)\right\rVert_1}=\frac{\left|J_{J,I}^{(k)}(x)\right|}{\sum_{J'\in\mathcal{J}_{k,m}}\sum_{I'\in\mathcal{I}_{k,n}}\left|J_{J',I'}^{(k)}(x)\right|} \tag*{(16)}

This collection can be interpreted as a discrete PMF p(k)(x)={pJ,I(k)(x)}J∈Jk,m,I∈Ik,np^{(k)}(x)=\{p_{J,I}^{(k)}(x)\}_{J\in\mathcal{J}_{k,m},I\in\mathcal{I}_{k,n}} over the NkN_k entries of the exterior power. If all minors possess equal magnitude, p(k)(x)p^{(k)}(x) reduces to the uniform two-dimensional PMF with entries 1/Nk1/N_k. Conversely, if a single minor accumulates all magnitude, the two-dimensional PMF becomes degenerate.

Rényi Divergence and Its Properties

The Rényi divergence relies on the Rényi entropy. The Rényi entropy of order α∈[0,∞)∪{∞}\alpha\in[0,\infty)\cup\{\infty\} generalizes the Shannon entropy of a discrete probability distribution pp with NN entries. When α≥0\alpha\ge0 and α≠1\alpha\ne1 the Rényi entropy is defined from the set of probabilities {pi}i=1N\{p_i\}_{i=1}^N as:

Hα(p)=11−αlog⁡∑i=1Npiα,(17)H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^{N}p_i^\alpha, \tag*{(17)}

where pi0=1p_i^0=1 unless pi=0p_i=0 (in such a case 00=00^0=0, by definition). Note that H0(p)=log⁡(∣supp⁡(p)∣)H_0(p)=\log(|\operatorname{supp}(p)|) is the Hartley entropy, with supp⁡(p)={i:pi>0}\operatorname{supp}(p)=\{i:p_i>0\} denoting the support of pp. Otherwise, when α∈{1,∞}\alpha\in\{1,\infty\} it is defined as the limit

Hα(p)=lim⁡γ→αHγ(p).(18)H_\alpha(p)=\lim_{\gamma\to\alpha}H_\gamma(p). \tag*{(18)}

Specifically, H1(p)H_1(p) is the classical Shannon entropy, i.e. H1(p)=HSh(p)=−∑i=1Npilog⁡piH_1(p)=H_{\mathrm{Sh}}(p)=-\sum_{i=1}^{N}p_i\log p_i. The case α=∞\alpha=\infty corresponds to the min-entropy H∞(p)=−log⁡(max⁡1≤i≤Npi)H_\infty(p)=-\log(\max_{1\le i\le N}p_i) where the metric is determined by the information associated with the most likely outcome. Finally, note that the case α=2\alpha=2 is widely used in practice and is known as the collision entropy.

The parameter α\alpha regulates the weight assigned to outcomes of varying probabilities. The following cases are relevant for our discussion:

  • Extreme choices (α=0\alpha=0 and α=∞\alpha=\infty): For α=0\alpha=0, all non-zero entries are weighted equally regardless of magnitude. For α=∞\alpha=\infty, only

the maximal element (that is to say, the mode of the discrete probability distribution) influences the value.

  • Choice of α=1\alpha= 1: In this case we have the Shannon entropy as an intermediate choice.

  • Choice of α=2\alpha= 2: The intermediate case α=2\alpha= 2 yields the entropy:

H2(p)=−log⁡∑i=1Npi2.(19)H_2(p) = -\log\sum_{i=1}^{N} p_i^2. \tag*{(19)}

The choice of α=2\alpha= 2 provides a balanced weighting across the distribution while establishing a direct analytical bridge to the Participation Ratio, as we will describe later.

Let us define the Rényi divergence of order α\alpha (where α∈[0,∞)∪{∞}\alpha\in[0,\infty) \cup\{\infty\}) of two discrete probability distributions pp and qq with NN outcomes and supp⁡(p)⊆supp⁡(q)\operatorname{supp}(p) \subseteq\operatorname{supp}(q). For α∈(0,1)∪(1,∞)\alpha\in(0,1) \cup(1,\infty) the Rényi divergence is defined as (van Erven & Harremoës, 2014, Equation (1)):

Dα(p∥q)=11−αlog⁡∑i=1Npiαqiα−1,(20)D_\alpha(p \parallel q) = \frac{1}{1-\alpha}\log\sum_{i=1}^{N} \frac{p_i^\alpha}{q_i^{\alpha-1}}, \tag*{(20)}

and for the remaining cases, i.e. α∈{0,1,∞}\alpha\in\{0,1,\infty\}, it is defined as the limit of Equation (20),

Dα(p∥q)=lim⁡γ→αDγ(p∥q).(21)D_\alpha(p \parallel q) = \lim_{\gamma\to\alpha} D_\gamma(p \parallel q). \tag*{(21)}

By taking limits using L’Hôpital’s Rule or evaluating directly, the expressions in Table 1 are obtained for these three special cases: the min-divergence (related to the Hartley entropy) for α=0\alpha= 0, the Kullback–Leibler divergence for α=1\alpha= 1, and the max-divergence for α=∞\alpha= \infty.

Parameter (α\alpha)ExpressionName
α=0\alpha= 0D0(p∥q)=−log⁡(∑i:pi>0qi)D_0(p \parallel q) = -\log\left(\sum_{i:p_i>0} q_i\right)Min-divergence/Hartley
α=1\alpha= 1D1(p∥q)=∑i=1Npilog⁡(piqi)D_1(p \parallel q) = \sum_{i=1}^{N} p_i \log\left(\frac{p_i}{q_i}\right)Kullback–Leibler (DKLD_{\mathrm{KL}})
α=∞\alpha= \inftyD∞(p∥q)=log⁡(max⁡1≤i≤Npiqi)D_\infty(p \parallel q) = \log\left(\max_{1 \le i \le N} \frac{p_i}{q_i}\right)Max-divergence

Table 1. Special cases of the Rényi divergence.

The Rényi divergence is not a distance in the strict sense since, in general, it is neither symmetric nor satisfies the triangle inequality. However, it is nonnegative for every order α\alpha, and for α>0\alpha> 0 it also satisfies the identity of indiscernibles, that is, it vanishes if and only if both distributions coincide. For α=0\alpha= 0 this property holds in a weaker form, since the divergence vanishes if and only if both distributions have the same support. These properties, established in Proposition 1, are what we need for our objectives.

Proposition 1. For any α∈[0,∞)∪{∞}\alpha\in[0, \infty) \cup\{\infty\} and any discrete probability distributions pp and qq with NN outcomes and supp⁡(p)⊆supp⁡(q)\operatorname{supp}(p) \subseteq\operatorname{supp}(q), the Rényi divergence is nonnegative, i.e. Dα(p∥q)≥0D_\alpha(p \parallel q) \ge0. Besides, for any α∈(0,∞)∪{∞}\alpha\in(0, \infty) \cup\{\infty\}, Dα(p∥q)=0D_\alpha(p \parallel q) = 0 if and only if p=qp = q. The case α=0\alpha= 0 is special and the condition becomes D0(p∥q)=0D_0(p \parallel q) = 0 if and only if supp⁡(p)=supp⁡(q)\operatorname{supp}(p) = \operatorname{supp}(q).

Proof. For α∈(0,1)∪(1,∞)∪{∞}\alpha\in(0,1) \cup(1,\infty) \cup\{\infty\}, the proof is found in [2 Theorem 8].

For α=1\alpha= 1, the Rényi divergence is the Kullback–Leibler divergence and the proof is found in [1].

From Table 1, the nonnegativity of the min-divergence (α=0\alpha= 0) is clear since supp⁡(p)⊆supp⁡(q)\operatorname{supp}(p) \subseteq\operatorname{supp}(q) and consequently ∑i:pi>0qi∈(0,1]\sum_{i:p_i>0} q_i \in(0,1]. In this case, α=0\alpha= 0, using Table 1 we have that, if supp⁡(p)=supp⁡(q)\operatorname{supp}(p) = \operatorname{supp}(q), then all the probability mass of qq lies on the same indices as that of pp, and therefore ∑i:pi>0qi=1\sum_{i:p_i>0} q_i = 1, so that D0(p∥q)=−log⁡(1)=0D_0(p \parallel q) = -\log(1) = 0. On the other hand, if D0(p∥q)=0D_0(p \parallel q) = 0, then ∑i:pi>0qi=1\sum_{i:p_i>0} q_i = 1, so that qi=0q_i = 0 for every i∉supp⁡(p)i \notin\operatorname{supp}(p), that is, $\operatorname{supp}(q) \subseteq supp⁡(p)\operatorname{supp}(p), which together with the hypothesis supp⁡(p)⊆supp⁡(q)\operatorname{supp}(p) \subseteq\operatorname{supp}(q) gives supp⁡(p)=supp⁡(q)\operatorname{supp}(p) = \operatorname{supp}(q).

The expression of the Rényi divergence takes a simplified form when the second distribution is the uniform distribution u=(1/N,…,1/N)u = (1/N,\ldots,1/N), which can be written in terms of the Rényi entropy, as stated in the following proposition.

Proposition 2. For any α∈[0,∞)∪{∞}\alpha\in[0,\infty) \cup\{\infty\} and any discrete probability distribution pp with NN outcomes, let us consider the uniform distribution u=(1/N,…,1/N)u = (1/N,\ldots,1/N), then

Dα(p∥u)=log⁡N−Hα(p).(22)D_{\alpha}(p \parallel u) = \log N - H_{\alpha}(p). \tag*{(22)}

Proof. As in Proposition 1, for α∈(0,1)∪(1,∞)\alpha\in(0,1) \cup(1,\infty) the proof is found in [2 Equation (2], while for α=1\alpha= 1 it is found in [1].

For the case α=0\alpha= 0, from Table 1 we obtain

D0(p∥u)=−log⁡(∑i:pi>01N)=−log⁡(∣supp⁡(p)∣1N)=log⁡N−H0(p).(23)D_{0}(p \parallel u) = -\log\left(\sum_{i:p_i>0}\frac{1}{N}\right) = -\log\left(|\operatorname{supp}(p)|\frac{1}{N}\right) = \log N - H_{0}(p). \tag*{(23)}

Finally, for the case α=∞\alpha= \infty and again from Table 1 we obtain

D∞(p∥u)=log⁡(max⁡1≤i≤Npi1/N)=log⁡(Nmax⁡1≤i≤Npi)=log⁡N−H∞(p).(24)D_{\infty}(p \parallel u) = \log\left(\max_{1\leq i\leq N}\frac{p_i}{1/N}\right) = \log\left(N\max_{1\leq i\leq N}p_i\right) = \log N - H_{\infty}(p). \tag*{(24)}

With the aid of Proposition 2, the Rényi divergence of order α∈[0,∞)∪{∞}\alpha\in[0,\infty) \cup\{\infty\} allows us to quantify the distance (not a metric distance) between a certain distribution pp and the uniform uu. Specifically, for a fixed α∈[0,∞)∪{∞}\alpha\in[0,\infty) \cup\{\infty\} we have two extreme situations:

  1. Minimum concentration: When p=up = u (completely uniform distribution) then Hα(p)=log⁡NH_{\alpha}(p) = \log N, yielding Dα(p∥u)=0D_{\alpha}(p \parallel u) = 0 (a consequence of Proposition 1). For α>0\alpha> 0 this is the only case with Dα(p∥u)=0D_{\alpha}(p \parallel u) = 0, whereas for α=0\alpha= 0 the divergence vanishes for every pp with supp⁡(p)=supp⁡(u)\operatorname{supp}(p) = \operatorname{supp}(u), that is, for every pp without zero entries.

  1. Maximum concentration: When pp is degenerate (pi=1p_i = 1 for some ii and 00 elsewhere) it is straightforward to check that Hα(p)=0H_\alpha(p) = 0, reaching the upper bound Dα(p∥u)=log⁡ND_\alpha(p \parallel u) = \log N.

In what follows we will always use the Rényi divergence comparing a probability distribution pp with the uniform distribution q=uq = u. Therefore, we will simply denote this measure by DαD_\alpha, omitting pp and uu, that is, Dα=˙Dα(p∥u)D_\alpha\mathrel{\dot=} D_\alpha(p \parallel u), where the distribution pp will be clear from the context.

Formal Definition of the Class of α\alpha-RC Metrics

Let us formally define the proposed class of Rényi-based Concentration (RC) metrics:

Definition 3 (Class of α\alpha-RC metrics). Let us fix α∈[0,∞)∪{∞}\alpha\in[0,\infty)\cup\{\infty\}. Let x∈Rnx \in\mathbb{R}^n be an input point, and let J(k)(x)J^{(k)}(x) denote the kk-th exterior power matrix with Nk=(nk)(nk)N_k = \binom{n}{k}\binom{n}{k} total entries. The pointwise α\alpha-RC metric of order kk at xx, denoted by Cα(k)(x)C_\alpha^{(k)}(x), is defined as follows:

Cα(k)(x)≔Dα(k)(x)log⁡Nk=1−Hα(k)(x)log⁡Nk,(25)C_\alpha^{(k)}(x) \coloneqq\frac{D_\alpha^{(k)}(x)}{\log N_k} = 1 - \frac{H_\alpha^{(k)}(x)}{\log N_k}, \tag*{(25)}

*where Hα(k)(x)H_\alpha^{(k)}(x) represents the Rényi entropy associated with the probability distribution p(k)(x)p^{(k)}(x) in Equation (16) and Dα(k)(x)D_\alpha^{(k)}(x) the Rényi divergence of the same distribution with respect to the uniform. Furthermore, for an evaluation dataset D={x(1),…,x(S)}\mathcal{D} = \{x^{(1)},\ldots,x^{(S)}\}, the dataset-level α\alpha-RC, Cα(k)(D)C_\alpha^{(k)}(\mathcal{D}), or simply Cα(k)C_\alpha^{(k)}, is defined as the sample mean across D\mathcal{D}:

Cα(k)≔1S∑s=1SCα(k)(x(s)).(26)C_\alpha^{(k)} \coloneqq\frac{1}{S}\sum_{s=1}^{S}C_\alpha^{(k)}(x^{(s)}). \tag*{(26)}

By construction, 0≤Cα(k)≤10 \le C_\alpha^{(k)} \le1. For α>0\alpha> 0, Cα(k)(x)=0C_\alpha^{(k)}(x) = 0 if and only if the minor magnitudes are uniformly distributed, whereas C0(k)(x)=0C_0^{(k)}(x) = 0 if and only if no minor vanishes, that is, ∣supp⁡(p(k)(x))∣=Nk\lvert\operatorname{supp}(p^{(k)}(x))\rvert= N_k (Proposition 1). For every α\alpha, Cα(k)(x)=1C_\alpha^{(k)}(x) = 1 if and only if all the magnitude is concentrated in a single minor (by convention, Cα(k)(x)=0C_\alpha^{(k)}(x) = 0 if J(k)(x)=0J^{(k)}(x) = 0).

Note that, from this point on, the quantities DαD_{\alpha} and HαH_{\alpha} depend on the order kk and on the input point xx. Indeed, both are built from the probability distribution p(k)(x)p^{(k)}(x) in Equation (16), which in turn is obtained from the exterior power J(k)(x)J^{(k)}(x) of the Jacobian. This is the reason why the Rényi divergence and the Rényi entropy appear in the definition of the α\alpha-RC metrics with the notation Dα(k)(x)D_{\alpha}^{(k)}(x) and Hα(k)(x)H_{\alpha}^{(k)}(x), respectively.

Within this class of α\alpha-RC metrics there are four special cases to be highlighted for a fixed x∈Rnx \in\mathbb{R}^{n}. The 0-RC metric is based on the min-divergence (or Hartley) and only depends on the number of non-zero entries of J(k)(x)J^{(k)}(x). At the opposite extreme is the ∞\infty-RC metric, which is related to the modal mass (probability at the mode). In an intermediate situation we have the 1-RC metric that corresponds to the KL-divergence (and the Shannon entropy), and also the 2-RC metric that is related to the collision entropy. The 2-RC metric is directly connected to the PR as shown below.

The class of α\alpha-RC metrics has a very nice behaviour with respect to the parameter α\alpha. Specifically, all these metrics are arranged in a monotonic way with respect to α\alpha, while their range is restricted to the interval [0,1][0,1]. This result is collected in the following proposition.

Proposition 3 (Monotonicity of the class of α\alpha-RC metrics). Let us fix the parameter kk with 1≤k≤min⁡(m,n)1 \leq k \leq\min(m,n) and Nk≥2N_k \geq2, where mm is the number of hidden neurons of the MLP defined in § 2.3 and nn its input dimension, and let x∈Rnx \in\mathbb{R}^{n} with J(k)(x)≠0J^{(k)}(x) \neq0. Let Cα(k)(x)C_{\alpha}^{(k)}(x) denote the pointwise α\alpha-RC metric of order kk at xx, defined in Equation (25) from the probability distribution p(k)(x)p^{(k)}(x) in Equation (16). Let α1,α2∈[0,∞)∪{∞}\alpha_1,\alpha_2 \in[0,\infty) \cup\{\infty\} with α1<α2\alpha_1 < \alpha_2. Then Cα1(k)(x)≤Cα2(k)(x)C_{\alpha_1}^{(k)}(x) \leq C_{\alpha_2}^{(k)}(x), with equality if and only if p(k)(x)p^{(k)}(x) is uniform on its support.

Proof. The inequality follows directly from the monotonicity of the Rényi divergence van Erven & Harremoës, [2] and Equation (25).

Regarding the equality, if p(k)(x)p^{(k)}(x) is uniform on its support, then Hα(k)(x)=log⁡∣supp⁡(p(k)(x))∣H_{\alpha}^{(k)}(x) = \log|\operatorname{supp}(p^{(k)}(x))| for every α\alpha, so Cα(k)(x)C_{\alpha}^{(k)}(x) does not depend on α\alpha and, in particular, Cα1(k)(x)=Cα2(k)(x)C_{\alpha_1}^{(k)}(x) = C_{\alpha_2}^{(k)}(x).

Conversely, for two orders in (0,1)∪(1,∞)(0,1) \cup(1,\infty), the proof of van Erven & Harremoës, [2] shows that the equality holds if and only if p(k)(x)p^{(k)}(x) is uniform on its support. If α1\alpha_1 or α2\alpha_2 is 00, 11 or ∞\infty, the equality and the monotonicity make DαD_\alpha constant on [α1,α2][\alpha_1,\alpha_2], which contains two such orders, and the same conclusion follows.

Connection of the 2-RC Metric with the Participation Ratio

A key analytical property of C2(k)(x)C_2^{(k)}(x) is its exact closed-form relation with PR⁡(k)(x)=∥J(k)(x)∥1∥J(k)(x)∥2\operatorname{PR}^{(k)}(x) = \frac{\lVert J^{(k)}(x)\rVert_1}{\lVert J^{(k)}(x)\rVert_2}. Using Equation (16), we can express the sum of squared probabilities as follows:

∑J∈Jk,m∑I∈Ik,n(pJ,I(k)(x))2=∑J,I∣JJ,I(k)(x)∣2∥J(k)(x)∥12=∥J(k)(x)∥22∥J(k)(x)∥12=1(PR⁡(k)(x))2.(27)\sum_{J \in\mathcal{J}_{k,m}} \sum_{I \in\mathcal{I}_{k,n}} \left(p_{J,I}^{(k)}(x)\right)^2 = \sum_{J,I} \frac{\left\lvert J_{J,I}^{(k)}(x)\right\rvert^2}{\lVert J^{(k)}(x)\rVert_1^2} = \frac{\lVert J^{(k)}(x)\rVert_2^2}{\lVert J^{(k)}(x)\rVert_1^2} = \frac{1}{\left(\operatorname{PR}^{(k)}(x)\right)^2}. \tag*{(27)}

Taking the logarithm in Equation (27) yields:

H2(k)(x)=−log⁡(1(PR⁡(k)(x))2)=2log⁡(PR⁡(k)(x)).(28)H_2^{(k)}(x) = -\log\left(\frac{1}{\left(\operatorname{PR}^{(k)}(x)\right)^2}\right) = 2\log\left(\operatorname{PR}^{(k)}(x)\right). \tag*{(28)}

Substituting H2(k)(x)H_2^{(k)}(x) back into C2(k)(x)C_2^{(k)}(x) from Equation (25) leads to:

C2(k)(x)=1−2log⁡(PR⁡(k)(x))log⁡Nk.(29)C_2^{(k)}(x) = 1 - \frac{2\log\left(\operatorname{PR}^{(k)}(x)\right)}{\log N_k}. \tag*{(29)}

This identity demonstrates that the 2-RC metric C2(k)C_2^{(k)} logarithmically scales and normalizes PR, removing its range dependency on architecture size while preserving its intuitive geometric interpretation.

Illustrative Example with the 2-RC Metric

To demonstrate how C2(k)C_2^{(k)} improves the metric interpretation with different architectures, we go back to Example 2 and consider two networks evaluated at an input point xx:

  • Network A (Nk=2N_k = 2 available minors): The exterior power JA(k)(x)=(11)J_A^{(k)}(x) = \begin{pmatrix} 1 & 1 \end{pmatrix} has a Participation Ratio PR⁡A(k)(x)=22=2\operatorname{PR}_A^{(k)}(x) = \frac{2}{\sqrt{2}} = \sqrt{2}.

The 2-RC metric is:

C2,A(k)(x)=1−2log⁡(2)log⁡2=1−log⁡2log⁡2=0.(30)C_{2,A}^{(k)}(x)=1-\frac{2\log(\sqrt{2})}{\log2}=1-\frac{\log2}{\log2}=0. \tag*{(30)}

Activity is evenly split across all 2 available minors, resulting in zero concentration.

  • Network B (Nk=16N_k=16 available minors): The exterior power JB(k)(x)J_B^{(k)}(x) has 2 non-zero entries equal to 1 and 14 zero entries. Here PRB(k)(x)=22=2\mathrm{PR}_B^{(k)}(x)=\frac{2}{\sqrt{2}}=\sqrt{2} (identical to Network A). However, the Rényi concentration is:

C2,B(k)(x)=1−2log⁡(2)log⁡16=1−log⁡24log⁡2=0.75.(31)C_{2,B}^{(k)}(x)=1-\frac{2\log(\sqrt{2})}{\log16}=1-\frac{\log2}{4\log2}=0.75. \tag*{(31)}

While PR assigns both networks the same value (2\sqrt{2}), C2(k)C_2^{(k)} assigns 0.75 to Network B, accurately reflecting that its activity is concentrated within a small fraction (2/16) of its total capacity.

Computational experiments

This section illustrates the behaviour of the proposed concentration metric on single-hidden-layer MLPs trained on a continuous version of the XOR problem. We first describe the experimental setup in § 4.1 and the predictive performance of the trained models in § 4.2. We then analyse the concentration of the minors at a fixed input point for every combination of width mm and order kk (§ 4.3) and, finally, its evolution along training for the largest configuration, m=32m=32 and k=3k=3 (§ 4.4). The main findings are discussed in § 4.5.

Experimental setup

We consider the regression problem studied by [4], in which the target is the continuous XOR function

fXOR(x)=∏i=1nsin⁡(πxi),x∈[−1,1]n,(32)f^{\mathrm{XOR}}(x)=\prod_{i=1}^{n}\sin(\pi x_i),\qquad x\in[-1,1]^n, \tag*{(32)}

with input dimension n=3n = 3. On [−1,1]3[-1,1]^3 the sign of fxor(x)f^{\mathrm{xor}}(x) is the product of the signs of the coordinates of xx, so that, as in its Boolean counterpart, the output depends jointly on all the input variables. For each random seed, we generate a training set Dtrain={(x(s),y(s))}s=1Strain\mathcal{D}_{\mathrm{train}} = \{(x^{(s)}, y^{(s)})\}_{s=1}^{S_{\mathrm{train}}} with Strain=4096S_{\mathrm{train}} = 4096 points and a test set Dtest={(x(s),y(s))}s=1Stest\mathcal{D}_{\mathrm{test}} = \{(x^{(s)}, y^{(s)})\}_{s=1}^{S_{\mathrm{test}}} with Stest=1024S_{\mathrm{test}} = 1024 points, independently of each other. In both sets, each input x(s)∈[−1,1]3x^{(s)} \in[-1,1]^3 is drawn independently from the uniform distribution on [−1,1]3[-1,1]^3, and the corresponding target is computed without noise as y(s)=fxor(x(s))y^{(s)} = f^{\mathrm{xor}}(x^{(s)}). The superscript (s)(s) therefore indexes the points of the set under consideration, which will be clear from the context.

Following the formulation of § 2.3, we train single-hidden-layer MLPs fMLP(x)=B σ(Ax+a)+bf^{\mathrm{MLP}}(x) = B \, \sigma(Ax + a) + b with two hidden widths, m∈{8,32}m \in\{8, 32\}. Here σ:Rm→Rm\boldsymbol{\sigma} : \mathbb{R}^{m} \to\mathbb{R}^{m} is the multivariate activation function, which applies the univariate activation σ:R→R\sigma: \mathbb{R} \to\mathbb{R} to each component of its argument. As univariate activation we use the exact GeLU, σ(t)=tF(t)\sigma(t) = t F(t), detailed in § 2.3. The weight matrices are initialized as random orthogonal matrices [12]. Specifically, A∈Mm×n(R)A \in\mathcal{M}_{m \times n}(\mathbb{R}) is a random matrix with orthonormal columns and B∈M1×m(R)B \in\mathcal{M}_{1 \times m}(\mathbb{R}) is a random vector of unit norm, both obtained from the QR decomposition of a matrix with independent standard Gaussian entries. The biases aa and bb are initialized to zero. The models are trained by minimizing the mean squared error with the Adam optimizer [7], with learning rate 0.01, (β1,β2)=(0.9,0.999)(\beta_1, \beta_2) = (0.9, 0.999) and mini-batches of 250 points reshuffled at every epoch, for a fixed budget of 500 epochs and without early stopping or explicit regularization. This batch size coincides with the critical batch size reported by [4] for this problem. Each width is trained with the four seeds {0,1,2,3}\{0, 1, 2, 3\}, which determine the dataset, the initialization and the order of the mini-batches. For a given seed, both widths are trained and evaluated on exactly the same training and test points, with the same mini-batch order. All the quantities described below are measured at initialization (epoch 0) and at the end of each of the 500 epochs, that is, on 501 states of each network.

Predictive performance is evaluated with the coefficient of determination, R2R^2, ([4 Equation (III.5]). Given a set D={(x(s),y(s))}s=1S\mathcal{D} = \{(x^{(s)}, y^{(s)})\}_{s=1}^{S} of SS points, it is defined as:

R2=1−∑s=1S(y(s)−y^(s))2∑s=1S(y(s)−yˉ)2(33)R^2 = 1 - \frac{\sum_{s=1}^{S}\left(y^{(s)}-\hat{y}^{(s)}\right)^2}{\sum_{s=1}^{S}\left(y^{(s)}-\bar{y}\right)^2} \tag*{(33)}

where y^(s)=fMLP(x(s))\hat{y}^{(s)} = f^{\mathrm{MLP}}(x^{(s)}) is the prediction of the model for the input x(s)x^{(s)}, and yˉ=1S∑s=1Sy(s)\bar{y} = \frac{1}{S}\sum_{s=1}^{S}y^{(s)} is the sample mean of the targets in D\mathcal{D}. We denote by Rtrain2R^2_{\mathrm{train}} and Rtest2R^2_{\mathrm{test}} the values obtained with D=Dtrain\mathcal{D} = \mathcal{D}_{\mathrm{train}} (S=StrainS = S_{\mathrm{train}}) and D=Dtest\mathcal{D} = \mathcal{D}_{\mathrm{test}} (S=StestS = S_{\mathrm{test}}), respectively, so that each of them uses the mean of the targets of its own set. A perfect predictor yields R2=1R^2 = 1, the constant predictor y^(s)=yˉ\hat{y}^{(s)} = \bar{y} yields R2=0R^2 = 0, and negative values correspond to predictions worse than that constant.

The concentration of the minors is evaluated with the class of α\alpha-RC metrics Cα(k)(x)C_{\alpha}^{(k)}(x) of Definition 3, computed from the distribution p(k)(x)p^{(k)}(x) of Equation (16) and averaged over a dataset as in Equation (26). We recall that Cα(k)(x)C_{\alpha}^{(k)}(x) takes values in [0,1][0,1] and is non-decreasing in α\alpha (Proposition 3), that the case α=1\alpha= 1 corresponds to the Shannon entropy and the case α=2\alpha= 2 to the collision entropy, and that the 2-RC metric is related to the Participation Ratio at every point through Equation (29). The results obtained for α=2\alpha= 2 can therefore be read as a normalized, logarithmic version of the Participation Ratio.

We evaluate four orders, α∈{1/2,1,2,∞}\alpha\in\{1/2, 1, 2, \infty\}. The ∞\infty-RC metric is the upper limit of the class and only depends on the largest minor. The orders α=1\alpha= 1 and α=2\alpha= 2 correspond to the Shannon and collision entropies, respectively, the latter linked to the Participation Ratio through Equation (29). The order α=1/2\alpha= 1/2 gives more weight to the small minors than α=1\alpha= 1. It is the only order for which the Rényi divergence is symmetric, and it is a function of the so-called Hellinger distance between p(k)(x)p^{(k)}(x) and the uniform distribution ([2]). Since Cα(k)(x)C_{\alpha}^{(k)}(x) is non-decreasing in α\alpha (Proposition 3), the values for α=1/2\alpha= 1/2 and α=∞\alpha= \infty bound, at every point, those of any order between them.

The limit α=0\alpha= 0 is not evaluated, because for the MLP of § 2.3 it carries no pointwise information. By Proposition 1 and Definition 3, C0(k)(x)C_0^{(k)}(x) only depends on which minors are non-zero, and it vanishes whenever no minor is zero. By Equation (12), a minor of the MLP vanishes if and only if det⁡(AJ,I)=0\det(A_{J,I})=0 or σ′(zj(x))=0\sigma^\prime(z_j(x))=0 for some j∈Jj \in J. The first condition does not depend on xx. The derivative of GeLU only vanishes at the isolated point t∗≈−0.7518t^* \approx-0.7518, so the second condition only holds when xx lies on one of the hyperplanes zj(x)=t∗z_j(x)=t^*, which form a set of zero measure. Hence, the support of p(k)(x)p^{(k)}(x) is the same at almost every xx and is determined by AA, and so is C0(k)(x)C_0^{(k)}(x). In particular, C0(k)(x)=0C_0^{(k)}(x)=0 at almost every xx as soon as no k×kk \times k minor of AA vanishes, as is generically the case for random and trained weights. In our experiments, no computed minor is zero at any of the evaluated points, epochs and networks, so that C0(k)(x)=0C_0^{(k)}(x)=0 in all of them.

All the minors of orders k∈{1,2,3}k \in\{1,2,3\} are computed at every point, without sampling. For n=3n=3, the number of minors of each order, Nk=(mk)(nk)N_k=\binom{m}{k}\binom{n}{k}, is N1=24N_1=24, N2=84N_2=84 and N3=56N_3=56 for m=8m=8, and N1=96N_1=96, N2=1488N_2=1488 and N3=4960N_3=4960 for m=32m=32. As in Equation (26), the concentration is first computed at each point and then averaged, never computed from averaged minors. The computation exploits the factorization of the minors of the MLP Jacobian given in Equation (12), [J(k)(x)]J,I=(∏j∈Jσ′(zj(x)))det⁡(AJ,I)[J^{(k)}(x)]_{J,I}=\left(\prod_{j\in J}\sigma^\prime(z_j(x))\right)\det(A_{J,I}), so that the determinants det⁡(AJ,I)\det(A_{J,I}) are computed once per network state and reused for all the points. All the computations are carried out in double precision. The derivatives σ′(zj(x))\sigma^\prime(z_j(x)) and the minors are handled in logarithmic scale, so that even the smallest minors are represented without underflow and with full relative precision.

Regarding the statistics reported below, R2R^2 is summarized by its mean ±\pm standard deviation across the four seeds. For the concentration, the four networks of a given width are evaluated on their own test sets, which gives 4×1024=40964 \times1024=4096 test points in total. The concentration is summarized by the mean ±\pm standard deviation of the pointwise values Cα(k)(x)C_\alpha^{(k)}(x) over these 4096 test points, so that this dispersion combines the variability between points and between tween seeds. Note that these 4096 test points, obtained by pooling the test sets of the four seeds (4×10244 \times1024), should not be confused with the 4096 training points of each dataset (Strain=4096S_{\mathrm{train}} = 4096). They are different sets of points, and the two numbers coincide only numerically. The implementation code, the synthetic datasets, the weights of the trained networks at every epoch and all the computed values of Cα(k)(x)C_{\alpha}^{(k)}(x) are publicly available on GitHub at https://github.com/JuanCarlosFdezMlg/renyi-concentration-kag.

Predictive performance

Table 2 reports the coefficient of determination given by Equation (33) of both architectures on the training and test sets, at initialization and at the end of training.

EpochSetm=8m = 8m=32m = 32
0Training (4096)−0.2561±0.1741-0.2561 \pm0.1741−0.0836±0.0611-0.0836 \pm0.0611
0Test (1024)−0.2358±0.1581-0.2358 \pm0.1581−0.1056±0.0800-0.1056 \pm0.0800
500Training (4096)0.6175±0.16860.6175 \pm0.16860.9322±0.04800.9322 \pm0.0480
500Test (1024)0.6011±0.16320.6011 \pm0.16320.9274±0.05160.9274 \pm0.0516

Table 2. Coefficient of determination R2R^2 (mean ±\pm standard deviation across the four seeds) of the MLPs with m=8m = 8 and m=32m = 32 hidden neurons. The number of points of each set is given in parentheses.

At initialization, both architectures yield negative values of R2R^2, that is, their predictions are worse than the constant prediction yˉ\bar{y}. After 500 epochs, the network with m=32m = 32 reaches Rtest2=0.9274±0.0516R^2_{\mathrm{test}} = 0.9274 \pm0.0516, whereas the network with m=8m = 8 remains at Rtest2=0.6011±0.1632R^2_{\mathrm{test}} = 0.6011 \pm0.1632. At the end of training, the values obtained on the training and test sets are almost identical, with differences below 0.017 for both widths, so there is no sign of overfitting. Since both sets lead to the same conclusions, from now on we focus exclusively on the test sets, and all the quantities reported in the remainder of this section are computed on the test points only. The narrow network is also much more sensitive to the seed. Its test values range from 0.3869 to 0.7592, while those of the wide network range from 0.8519 to 0.9660. This is consistent with the larger number of parameters of the wider network, whose greater capacity tends to make the result of training less dependent on the initialization and on the order of the mini-batches.

Pointwise concentration at a fixed input

kkMetricm=8m=8 Value at x∗x^{*}m=8m=8 Mean ±\pm SDm=32m=32 Value at x∗x^{*}m=32m=32 Mean ±\pm SD
Epoch 0 (initialization)
1C1/2(1)C_{1/2}^{(1)}0.03330.0695 ±\pm 0.02090.03050.0392 ±\pm 0.0037
C1(1)C_{1}^{(1)}0.06130.1202 ±\pm 0.03530.05060.0687 ±\pm 0.0079
C2(1)C_{2}^{(1)}0.10620.1908 ±\pm 0.05380.07860.1122 ±\pm 0.0160
C∞(1)C_{\infty}^{(1)}0.30830.3998 ±\pm 0.07080.23040.3037 ±\pm 0.0453
PR⁡(1)\operatorname{PR}^{(1)}4.13853.6306 ±\pm 0.30088.18827.5901 ±\pm 0.2748
2C1/2(2)C_{1/2}^{(2)}0.04060.0742 ±\pm 0.02650.02680.0373 ±\pm 0.0050
C1(2)C_{1}^{(2)}0.07130.1294 ±\pm 0.04490.04640.0667 ±\pm 0.0098
C2(2)C_{2}^{(2)}0.11770.2058 ±\pm 0.06640.07440.1120 ±\pm 0.0186
C∞(2)C_{\infty}^{(2)}0.31550.4179 ±\pm 0.07550.20350.3049 ±\pm 0.0389
PR⁡(2)\operatorname{PR}^{(2)}7.06095.8711 ±\pm 0.830129.395725.6831 ±\pm 1.7130
3C1/2(3)C_{1/2}^{(3)}0.05040.0934 ±\pm 0.04110.02720.0393 ±\pm 0.0061
C1(3)C_{1}^{(3)}0.09460.1639 ±\pm 0.06890.04760.0714 ±\pm 0.0121
C2(3)C_{2}^{(3)}0.16420.2559 ±\pm 0.09940.07700.1216 ±\pm 0.0226
C∞(3)C_{\infty}^{(3)}0.39370.4605 ±\pm 0.10760.21990.3201 ±\pm 0.0392
PR⁡(3)\operatorname{PR}^{(3)}5.37774.5578 ±\pm 0.858050.762342.1749 ±\pm 3.9628
Epoch 500 (end of training)
1C1/2(1)C_{1/2}^{(1)}0.09610.1401 ±\pm 0.05850.20420.2478 ±\pm 0.0373
C1(1)C_{1}^{(1)}0.16740.2195 ±\pm 0.09270.26870.3163 ±\pm 0.0426
C2(1)C_{2}^{(1)}0.24990.3018 ±\pm 0.12300.33030.3752 ±\pm 0.0444
C∞(1)C_{\infty}^{(1)}0.40510.4525 ±\pm 0.13310.44330.4838 ±\pm 0.0442
PR⁡(1)\operatorname{PR}^{(1)}3.29373.0888 ±\pm 0.57114.61094.1834 ±\pm 0.4262
2C1/2(2)C_{1/2}^{(2)}0.11390.2062 ±\pm 0.05790.26470.3318 ±\pm 0.0534
C1(2)C_{1}^{(2)}0.18110.3105 ±\pm 0.08130.35230.4352 ±\pm 0.0644
C2(2)C_{2}^{(2)}0.25870.4084 ±\pm 0.09500.44350.5204 ±\pm 0.0624
C∞(2)C_{\infty}^{(2)}0.44390.5641 ±\pm 0.08270.58910.6412 ±\pm 0.0528
PR⁡(2)\operatorname{PR}^{(2)}5.16743.7889 ±\pm 0.77177.63405.9160 ±\pm 1.3604

Table 4.3.

We first examine the concentration of the minors at an individual input point. We take as x∗x^* the first test point of seed 0, x∗=(0.1034,0.1249,−0.0206)x^*=(0.1034, 0.1249, -0.0206), with target fxor(x∗)=−0.0079f^{\mathrm{xor}}(x^*)=-0.0079. Since the datasets are shared across widths, x∗x^* is the same point for m=8m=8 and m=32m=32. Table 3 reports Cα(k)(x∗)C_{\alpha}^{(k)}(x^*) for the networks of seed 0, for all the widths, orders kk and values of α\alpha, at initialization and at the end of training, together with the Participation Ratio, PR(k)(x∗)\mathrm{PR}^{(k)}(x^*), of Section 2.4. Next to each value, the table also reports the mean ±\pm standard deviation of the corresponding metric over the 4096 test points of the four seeds, which places the value at x∗x^* within the distribution of the test set. Since the minors are computed exhaustively, PR(k)\mathrm{PR}^{(k)} is obtained at each point from C2(k)C_{2}^{(k)} through Equation (29).

Several features of Table 3 are worth highlighting. First, the monotonicity of Cα(k)C_{\alpha}^{(k)} with respect to α\alpha is observed in every block, both at x∗x^* and on average. For every width, order and epoch, C1/2(k)≤C1(k)≤C2(k)≤C∞(k)C_{1/2}^{(k)} \leq C_{1}^{(k)} \leq C_{2}^{(k)} \leq C_{\infty}^{(k)}. The order α\alpha therefore does not modify the information contained in the minors, but only the part of their distribution that is being summarized.

kkMetricm=8m=8 Value at x∗x^*m=8m=8 Mean ±\pm SDm=32m=32 Value at x∗x^*m=32m=32 Mean ±\pm SD
3C1/2(3)C_{1/2}^{(3)}0.13460.2619±0.08030.2619 \pm0.08030.33230.4198±0.07080.4198 \pm0.0708
3C1(3)C_{1}^{(3)}0.20660.3695±0.09850.3695 \pm0.09850.43230.5426±0.08570.5426 \pm0.0857
3C2(3)C_{2}^{(3)}0.28900.4673±0.10800.4673 \pm0.10800.52810.6384±0.08140.6384 \pm0.0814
3C∞(3)C_{\infty}^{(3)}0.46530.6313±0.09500.6313 \pm0.09500.67700.7556±0.06090.7556 \pm0.0609
3PR(3)\mathrm{PR}^{(3)}4.18282.9900±0.63702.9900 \pm0.63707.44764.9413±1.69724.9413 \pm1.6972

Table 3. α\alpha-RC metrics Cα(k)C_{\alpha}^{(k)} and Participation Ratio PR(k)\mathrm{PR}^{(k)} at the test point x∗x^* for the networks of seed 0, and their mean ±\pm standard deviation over the 4096 test points of the four seeds.

Second, training increases the concentration of every network, for every order kk and every value of α\alpha, both at x∗x^* and on average, and the Participation Ratio decreases accordingly in every case. Within each network, the increase grows with the order of the minors. For α=2\alpha=2, the mean concentration of the network with m=32m=32 grows from 0.1122 to 0.3752 for k=1k=1, from 0.1120 to 0.5204 for k=2k=2 and from 0.1216 to 0.6384 for k=3k=3. For the network with m=8m=8, it grows from 0.1908 to 0.3018, from 0.2058 to 0.4084 and from 0.2559 to 0.4673, respectively.

Third, the rows of PR⁡(k)\operatorname{PR}^{(k)} contain the same pointwise information as those of C2(k)C_{2}^{(k)}, since both are related by the monotone transformation of Equation (29). The difference lies in how each value is read. A value of the Participation Ratio can only be interpreted together with its upper bound Nk\sqrt{N_k}, which depends on mm and kk. This bound is 4.90, 9.17 and 7.48 for m=8m=8, and 9.80, 38.57 and 70.43 for m=32m=32, for k=1,2,3k=1,2,3, respectively. In contrast, C2(k)C_{2}^{(k)} directly expresses the position of the same value between the two extremes of its own range. For instance, for m=32m=32 and k=3k=3, PR⁡(3)(x∗)\operatorname{PR}^{(3)}(x^*) decreases from 50.7623 to 7.4476 out of a maximum of 70.43, which corresponds to an increase of C2(3)(x∗)C_{2}^{(3)}(x^*) from 0.0770 to 0.5281. Similarly, for m=8m=8 and k=3k=3, PR⁡(3)(x∗)\operatorname{PR}^{(3)}(x^*) decreases from 5.3777 to 4.1828 out of a maximum of 7.48, and C2(3)(x∗)C_{2}^{(3)}(x^*) increases from 0.1642 to 0.2890.

Finally, in every row of the table the point x∗x^* is less concentrated than the average test point, with a lower value of Cα(k)C_{\alpha}^{(k)} and a higher value of PR⁡(k)\operatorname{PR}^{(k)}, both at initialization and at the end of training. The value at x∗x^* is therefore not representative of the whole test set by itself, which illustrates the pointwise heterogeneity summarized by the dataset-level metric of Equation (26) and by its standard deviation.

Evolution of the concentration during training

Although the previous analysis could be extended to every width and every order along all the epochs, from now on we restrict the study to m=32m=32 and k=3k=3 for the sake of brevity. This is the configuration with the best predictive performance and the largest number of minors, N3=4960N_3=4960. For every epoch, the mean and the variance of Cα(3)C_{\alpha}^{(3)} are computed over the 4096 test points of the four seeds. Figure 1 shows the evolution of the mean concentration for the four values of α\alpha, with a common vertical scale. At initialization, the mean values are 0.0393, 0.0714, 0.1216 and 0.3201 for α=1/2\alpha= 1/2, 1, 2 and ∞\infty, respectively. The concentration increases rapidly during the first 60 epochs. For instance, the mean of C2(3)C_{2}^{(3)} exceeds 0.3 at epoch 31 and 0.5 at epoch 52. The growth then slows down until a maximum is reached between epochs 210 and 240, with values 0.4763 for α=1/2\alpha= 1/2 and 0.5778 for α=1\alpha= 1 (both at epoch 211), 0.6649 for α=2\alpha= 2 (epoch 238) and 0.7763 for α=∞\alpha= \infty (epoch 212). From then on, the concentration decreases slightly, and at epoch 500 the means are 0.4198, 0.5426, 0.6384 and 0.7556, respectively. As expected from the monotonicity in α\alpha, the curves never cross, and the gap between α=1\alpha= 1 and α=∞\alpha= \infty remains of the same magnitude throughout training, from 0.25 at initialization to 0.21 at the end.

Figure 1. Mean of Cα(3)C_{\alpha}^{(3)} over the test points during training for m=32m = 32.

The maximum of the concentration roughly coincides with that of the number of hidden neurons whose input weights are very small, below 10−810^{-8} times the largest entry of AA. On average over the four seeds, there is no such neuron before epoch 65, their number reaches 13.25 out of 32 at epoch 227, and then decreases to 5.25 at epoch 500. By Equation (12), the minors that involve these neurons become negligible compared with the others, so that the magnitude concentrates on the minors of the remaining neurons.

The slight decrease of the concentration after its maximum agrees with the observations of Freedman & Mulligan [4], who report an extremum of their minor concentration metrics during training near a training R2R^2 of about 0.85, away from the point of peak performance (Figure 9 of that work). In our experiments, the maxima of Cα(3)C_{\alpha}^{(3)} are reached at values of Rtest2R_{\mathrm{test}}^2 between 0.84 and 0.86. As those authors point out, the cause of this extremum is not yet understood. We have checked that it is not caused by hidden neurons becoming active again. The fraction of hidden neurons that are locally inactive at the test points, with ∣σ′(zj(x))∣<0.05|\sigma'(z_j(x))| < 0.05, grows from 0.1% at initialization to 11.4% at epoch 500, where it reaches its maximum.

Figure 2 relates this evolution to the predictive performance of the same networks. Each panel corresponds to a value of α\alpha and contains one point per epoch. The vertical axis is the mean Rtest2R_{\mathrm{test}}^2 of the four seeds, and the colour of each point is the mean of Cα(3)C_{\alpha}^{(3)} at that epoch, on a magma scale fixed to [0,1][0,1] and common to the four panels. The halo around each point has an area proportional to the variance of Cα(3)C_{\alpha}^{(3)} at that epoch, relative to the maximum variance reached in the panel. The legend of each panel shows the halos that correspond to one quarter, one half and the whole of that maximum variance. For instance, the maximum variance for α=2\alpha= 2 is 0.0075, so its legend shows 0.0019, 0.0037 and 0.0075. The halo therefore encodes the dispersion of the concentration, not an interval for R2R^2.

Joint evolution of the predictive performance and the concentration of the minors for $m=32$ and $k=3$ on the test sets

Figure 2. Joint evolution of the predictive performance and the concentration of the minors for m=32m = 32 and k=3k = 3 on the test sets.

During the first epochs, the mean Rtest2R^2_{\mathrm{test}} remains close to zero, with Rtest2=−0.0061R^2_{\mathrm{test}} = -0.0061 at epoch 20 and 0.0456 at epoch 30, and then rises sharply. It exceeds 0.25 at epoch 38, 0.5 at epoch 82, 0.75 at epoch 158 and 0.9 at epoch 338, reaching 0.9274 at epoch 500. The change of colour of the points, from dark to bright tones, takes place along this rise. It is worth noting that the concentration starts to grow before Rtest2R^2_{\mathrm{test}} departs from zero. The mean of C2(3)C^{(3)}_2 increases from 0.1216 at initialization to more than 0.2 at epoch 19, while the predictions are still no better than the constant yˉ\bar{y}. In the final phase, in contrast, Rtest2R^2_{\mathrm{test}} keeps increasing, from 0.8282 at epoch 200 to 0.9274 at epoch 500, while the concentration decreases slightly. As already observed by [4], the relationship between both quantities is therefore not monotone along the whole training.

The halos show that the dispersion of the concentration also grows during training. For α=2\alpha= 2, the standard deviation increases from 0.0226 at initialization to 0.0814 at epoch 500, and the maximum variance for α=1/2\alpha= 1/2, α=1\alpha= 1 and α=2\alpha= 2 is reached at epoch 486. For α=∞\alpha= \infty, the maximum variance is reached earlier, at epoch 111.

Discussion

The experiments illustrate how the proposed metric is read in practice. Each value of Cα(k)C_{\alpha}^{(k)} lies in [0,1][0,1] and expresses the position of the distribution of minors between the two extremes attainable for its own number of minors, namely the uniform distribution and a single dominant minor. For the network with m=32m = 32 and k=3k = 3, for instance, this shows that the mean of C2(3)C_{2}^{(3)} evolves from 0.1216, close to the uniform distribution, to 0.6384, an intermediate level of concentration. The Participation Ratio carries the same pointwise information for α=2\alpha= 2, but its values can only be interpreted together with the bound Nk\sqrt{N_k} of each configuration.

Regarding the relationship with predictive performance in the network with m=32m = 32, the growth of the concentration accompanies, and slightly precedes, the onset of learning. However, the concentration is not a monotone function of the performance, since it decreases slightly in the last phase of training while Rtest2R^2_{\mathrm{test}} keeps improving. This agrees with the observation of [4] that the link between Kolmogorov–Arnold geometry and performance depends on the training regime.

The effect of the order kk can be interpreted through the factorization of Equation (12). Each minor of order kk contains the product of the derivatives σ′(zj(x))\sigma'(z_j(x)) of kk hidden neurons, so that when some of these derivatives become small at a point, the minors that involve the corresponding neurons are suppressed multiplicatively. This effect is amplified as kk grows, which is consistent with the larger increase in concentration observed for higher orders within each network. Even so, the concentration remains far from its maximum value. For m=32m = 32 and k=3k = 3, the mean of the ∞\infty-RC metric C∞(3)C_{\infty}^{(3)} never exceeds 0.7763 and is 0.7556 at epoch 500, so even the most mode-sensitive member of the class stays well below 1 and the magnitude is never carried by a single minor. At the other end of the evaluated orders, the mean of C1/2(3)C_{1/2}^{(3)}, which gives more weight to the small minors, never exceeds 0.4763 and is 0.4198 at epoch 500. The magnitude therefore concentrates on a subset of the minors without collapsing onto a single one, and no minor vanishes. This intermediate level of concentration is consistent with the observation of [4] that the proportion of zero rows in trained MLPs is well below that of the ideal Kolmogorov–Arnold construction.

Concerning the choice of α\alpha, the qualitative conclusions above, namely the increase of the concentration with training and its larger increase for higher orders within each network, hold for the four evaluated orders. The order α\alpha determines which part of the distribution of minors is summarized. The order α=1/2\alpha= 1/2 gives more weight to the small minors, the Shannon (α=1\alpha= 1) and collision (α=2\alpha= 2) entropies provide intermediate summaries, the latter directly linked to the Participation Ratio, and the limit α=∞\alpha= \infty only depends on the largest minor. By monotonicity, the orders α=1/2\alpha= 1/2 and α=∞\alpha= \infty delimit, at every point, the values of any order between them. The limit α=0\alpha= 0, which only depends on which minors vanish, is constant for the MLP with GeLU, as discussed in § 4.1.

Conclusions and Future Research

This paper has introduced the class of α\alpha-RC metrics to measure the concentration of the minors of the Jacobian of the inner map, a central notion of Kolmogorov–Arnold Geometry. The metrics are built from the Rényi divergence of order α\alpha between the normalized distribution of the minors and the uniform distribution, divided by its maximum value log⁡Nk\log N_k. As a result, every member of the class takes values in [0,1][0,1], and both extremes have the same meaning for every architecture. The value 0 corresponds, for every α>0\alpha> 0, to a uniform distribution of the minors, and the value 1 to a single minor carrying all the magnitude. The class is non-decreasing in α\alpha, with equality between two orders only when the minors are uniform on their support, and the order α\alpha selects which part of the distribution is summarized, from the small minors for α=1/2\alpha= 1/2 to the largest minor for α=∞\alpha= \infty. For α=2\alpha= 2 the metric is a closed-form transformation of the Participation Ratio, so it keeps the pointwise information of the Participation Ratio while removing its dependence on the number of minors.

The experiments with single-hidden-layer MLPs trained on a continuous XOR problem illustrate how the class is read in practice. Training increases the concentration for every width, every order kk and every value of α\alpha, and the increase grows with kk. For the network with m=32m = 32 hidden neurons and k=3k = 3, the mean of C2(3)C_{2}^{(3)} grows from 0.12 at initialization to a maximum of 0.66, and it starts to increase before the coefficient of determination departs from zero. The relationship with the predictive performance is not monotone, since the concentration decreases slightly in the last phase of training while the performance keeps improving, in agreement with the observations of [4]. The concentration reaches intermediate values, with the mean of C∞(3)C_{\infty}^{(3)} below 0.78 throughout training, so the magnitude concentrates on a subset of the minors without collapsing onto a single one.

The proposed family of metrics has been applied to networks in which all the minors can be enumerated. Several directions remain open. The number of minors Nk=(mk)(nk)N_k = \binom{m}{k}\binom{n}{k} grows combinatorially with the width, the input dimension and the order kk, so the exact computation of the metrics is not feasible for realistic networks, and efficient estimation methods are needed. The early growth of the concentration also suggests that the metrics could be used not only to monitor training but also to guide it, for instance by adding a concentration term to the loss function or by interleaving gradient steps on the concentration, as proposed by [3].

CRediT authorship contribution statement

Juan Carlos Fernández: Conceptualization, Methodology, Software, Writing – original draft, Writing – review & editing. Asunción Jiménez-Cordero: Conceptualization, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing. José Francisco Paris: Conceptualization, Methodology, Supervision, Validation, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors thankfully acknowledge the computer resources (Picasso Supercomputer), technical expertise and assistance provided by the SCBI (Supercomputing and Bioinformatics) center of the University of Málaga. J. F. Paris is partially supported by the project PAR 11/2023 of the University of Málaga.

Data availability

The implementation code, the synthetic datasets, the weights of the trained networks at every epoch and all the computed values of Cα(k)(x)C_{\alpha}^{(k)}(x) are publicly available on GitHub at https://github.com/JuanCarlosFdezMlg/renyi-concentration-kag.

Declaration of generative AI and AI assisted technologies in the manuscript preparation process

During the preparation of this work the authors used Claude (Anthropic) and Codex (OpenAI) in order to assist with the implementation of the code for the computational experiments, to draft and revise parts of the text, and to improve the language and readability of the manuscript. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

References

  1. [1]Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory. Wiley Series in Telecommunications and Signal Processing (2nd ed.). Hoboken, NJ: Wiley-Interscience. doi:10.1002/047174882X.DOI
  2. [2]van Erven, T., & Harremoës, P. (2014). Rényi divergence and Kullback–Leibler divergence. IEEE Transactions on Information Theory, 60, 3797–3820. doi:10.1109/TIT.2014.2320500.DOI
  3. [3]Freedman, M. H. (2024). The proof of Kolmogorov–Arnold may illuminate neural network learning. arXiv preprint arXiv:2410.08451. doi:10.48550/arXiv.2410.08451.DOI
  4. [4]Freedman, M. H., & Mulligan, M. (2025). Spontaneous Kolmogorov–Arnold geometry in shallow MLPs. arXiv preprint arXiv:2509.12326. doi:10.48550/arXiv.2509.12326.DOI
  5. [5]Girosi, F., & Poggio, T. (1989). Representation properties of networks: Kolmogorov’s theorem is irrelevant. Neural Computation, 1, 465–469. doi:10.1162/neco.1989.1.4.465.DOI
  6. [6]Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2021). Explainability in deep reinforcement learning. Knowledge-Based Systems, 214, 106685. doi:10.1016/j.knosys.2020.106685.DOI
  7. [7]Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations. Kolmogorov, A. N. (1957). On the representation of continuous functions of several variables by superpositions of continuous functions of one variable and addition. Doklady Akademii Nauk SSSR, 114, 953–956.
  8. [8]Kůrková, V. (1991). Kolmogorov’s theorem is relevant. Neural Computation, 3, 617–622. doi:10.1162/neco.1991.3.4.617.DOI
  9. [9]Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T. Y., & Tegmark, M. (2025). KAN: Kolmogorov–Arnold networks. In International Conference on Learning Representations (pp. 70367–70413).DOI
  10. [10]Morris, S. (2021). Hilbert 13: Are there any genuine continuous multivariate real-valued functions? Bulletin of the American Mathematical Society, 58, 107–118. doi:10.1090/bull/1698.DOI
  11. [11]Rényi, A. (1961). On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics (pp. 547–561). University of California Press.
  12. [12]Saxe, A. M., McClelland, J. L., & Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations.

Paper details

vlnc:W_2usfram8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude, Codex

View statement
Looking for related papers…
Contents