WorksMathematics
Equilibrium measures for anisotropic exponential interaction energies with applications to transformers
Abstract
We study the minimization of anisotropic exponential interaction energies over probability measures on the sphere, motivated by mean-field models of self-attention in transformer architectures. For every symmetric positive definite interaction matrix, we relate the minimizer to a corresponding anisotropic Brownian motion and use the Feynman–Kac formula and elliptic regularity theory to show that it admits a smooth, strictly positive density on the sphere. We further show that it is the only stationary measure with full support for the dynamics considered. We use the stochastic viewpoint to derive an algorithm for sampling from the minimizer. In dimension two, we also propose a numerical scheme for approximating its density based on a Mathieu series expansion.
Introduction
The equilibrium behavior of interacting particle systems is a classical topic in statistical physics, including a variety of different phenomena such as clustering, screening, or crystallization (cf. [42, 31, 20, 35]). Analytical insights for large particle systems can be gained from a mean-field perspective, which opens the way to methods in calculus of variations (in the space of measures) or partial differential equations (cf. [25, 7, 8, 38, 11, 10, 37]), recently also extending the range of applications far beyond physics, e.g. to systems in biology or social behavior. A recent unexpected development in the field is the study of large-scale behavior in transformer architectures, widely used in modern machine learning, through interacting particle systems. This perspective helps explain some of the rich behavior observed in computations. The normalization mechanisms in these architectures naturally lead to problems on the sphere, whereas most of particle systems in other settings are studied in the full space. Moreover, even with a strong level of simplification relative to realistic transformer architectures, the anisotropy of the interaction remains important, distinguishing these problems from the predominant isotropic settings in the study of equilibrium measures.
In this paper, our main object of interest is the following energy functional on the space of Borel probability measures on the unit sphere , ,
where the interaction kernel is given by with a symmetric and positive definite matrix .
Variational problems of this form, with various interaction kernels , arise in several areas, including aggregation and swarming models, where attractive and repulsive interactions determine the shape of equilibrium configurations, e.g. Carrillo et al. [12], models of collective alignment on the sphere, e.g. Frouvelle and Liu [16], and in particular mean-field self-attention models in transformer architectures, e.g. Sander et al. [32]; Geshkovski et al. [17] and subsequent works (due to the large and ever expanding literature on transformers we do not attempt a review; we restrict attention to works most directly relevant to our setting). For with , it is known Borodachov et al. [5] that the unique minimizer is the uniform distribution on the sphere; this has also been noted by Geshkovski et al. [17] in the context of transformers. In the anisotropic case, the properties of the minimizer depend heavily on the eigenvalues of the matrix [9]. When is not positive definite, the minimizer may be supported on a proper subspace spanned by eigenvectors of , and is therefore singular with respect to the surface measure. For , however, it was conjectured that the minimizer has full support. In this paper we prove this conjecture using techniques from stochastic analysis and elliptic PDE theory.
Main contributions and structure of the paper. Our main contributions are as follows.
We show that the minimizer of (1) is the exit distribution of a certain stochastic process (killed anisotropic Brownian motion; § 3.1) and, furthermore, that it has a smooth and strictly positive density with respect to the surface measure on the sphere (§ 3.2), thus resolving the full-support conjecture of Burger et al. [9].
We show that, for the dynamics considered, the minimizer is the only stationary measure having full support and that every other stationary measure is singular with respect to the surface measure (§ 4.1).
We analyze the stability of the minimizer (in the maximum mean discrepancy induced by the interaction kernel) with respect to perturbations in the matrix (§ 4.2).
We propose an algorithm for sampling from the minimizer using the stochastic process introduced in § 3.1 (§ 5.1).
For , we propose a numerical scheme for approximating the minimizing density based on a Mathieu series expansion (§ 5.2).
In § 2 we collect the properties of the variational problem that will be used in the rest of the paper. Our notation is summarized below.
Notation. Throughout the paper, with is a symmetric positive definite matrix, i.e. and . For we denote by the diagonal matrix with for all . We use for the Euclidean norm and and for the unit sphere and the open unit ball, respectively. For matrices, we denote by , , , and respectively the identity, determinant, trace, and operator norm induced by . We write . For a twice continuously differentiable function , we denote by its Hessian. Let and denote the intrinsic gradient and divergence.
We denote by the space of Borel probability measures over , by the surface measure, and by the normalized surface measure. We write for the Dirac measure at , for the support of , and for its push-forward under , defined by . We write when the measures and are mutually singular, that is, concentrated on disjoint Borel sets. The spaces of real-valued continuous and smooth functions on are denoted by and , respectively. We write for the space of twice continuously differentiable functions on with compact support. Furthermore, we denote by the Hilbert space of Lebesgue functions on with respect to the surface measure , equipped with the inner product
For a sequence and , weak convergence, written , means that
For a finite signed Borel measure on , we denote its total variation norm by
The Wasserstein distance (of order 2) is defined by
where is the set of Borel probability measures on with marginals and , and is the geodesic distance. Since is compact, the Wasserstein distance metrizes the weak topology [45 Corollary 6.13].
Properties of the variational problem
In this section we collect some results about the variational problem associated with the energy that will be used throughout the paper. We consider
where, as before, is symmetric and positive definite. We first establish existence and uniqueness of a global minimizer. We then introduce the Euler–Lagrange criterion, which will guide the construction of the minimizer in § 3. Finally, we relate the energy gap to a distance between probability measures induced by the interaction kernel.
Proposition 1 (Existence and uniqueness). The problem (2) admits a unique minimizer, which we denote by .
Proof. The space is weakly compact, and continuity of on makes weakly continuous. Hence the minimum is attained. By Burger et al. [9], the energy is strictly convex on , that is
Therefore the minimizer is unique. □
Euler–Lagrange optimality criterion. For a finite signed Borel measure on , define its potential by
where we simplify notation by suppressing the dependence on .
Proposition 2 (Euler–Lagrange criterion (Bilyk et al., [4]).) For ,
In particular, by uniqueness of the minimizer, any probability measure whose potential is constant on is equal to .
The criterion says that the support of an equilibrium measure lies where its potential is smallest. A measure with constant potential on the whole sphere therefore provides a particularly convenient candidate: once it is known to be a probability measure, minimality follows. This observation will be the starting point of the proofs of main results in § 3.
Interpretation using reproducing kernel Hilbert spaces. We next show that the interaction kernel is strictly positive definite whenever is symmetric and positive definite. Set
The identity
shows that is strictly positive definite: the Gaussian kernel is strictly positive definite (Sriperumbudur et al., [41]), the map is injective, and is strictly positive.
Thus there exists a unique reproducing kernel Hilbert spaces (RKHS) with reproducing kernel (Aronszajn, [2]), whose canonical feature map is
For a finite signed Borel measure , its potential
is its kernel mean embedding (Simon-Gabriel and Schölkopf, [39]). The Bochner integral (7) is well defined since is bounded on . The reproducing property gives, for finite signed Borel measures ,
so that
For , the associated maximum mean discrepancy (Gretton et al., [19]) satisfies
Proposition 3. The distance metrizes the weak topology on ; that is, for and ,
Proof. We use Gaussian factorization of the kernel given in (6). For a finite signed Borel measure on , define the finite signed measure on . Substituting this into (8) yields
The Gaussian kernel is integrally strictly positive definite (Sriperumbudur et al. [41]), so this integral is strictly positive whenever . Since is invertible and is strictly positive on , the map is injective. Thus for every .
In particular, the kernel mean embedding is injective on , so is characteristic. Since is continuous and is compact, Simon-Gabriel et al. [40] implies that metrizes the weak topology on .
We now use this norm to relate the energy gap to the distance from the minimizer.
Proposition 4 (Energy gap). For every ,
Proof. Writing and using (9), we obtain
By linearity of the kernel mean embedding and the reproducing property, the cross-term is
Therefore,
where we used and . The integral there is nonnegative by the Euler–Lagrange criterion (5), which proves the inequality.
Remark 1. If , as will be proved further in Corollary 8, then (5) gives on . The integral in (10) therefore vanishes, yielding an exact equality
In particular, for every ,
Together with Proposition 3 and weak continuity of , this gives the following characterization of convergence to the minimizer.
Corollary 5 (Convergence of approximate minimizers). For a sequence , the following are equivalent:
Proof. The lower bound in (10) gives the first implication, and Proposition 3 gives the second. Conversely, weak convergence to implies convergence of the energies by weak continuity of .
Consequently, a sequence of probability measures has energies converging to the minimum if and only if it converges weakly to . These observations will allow us to translate energy estimates into approximation and stability statements.
Properties of the minimizer
We are now ready to state the main results. We show that is the distribution of the exit point of a certain stochastic process and use this representation to establish the existence of a smooth, strictly positive density on , resolving conjecture of Burger et al., [9]. This representation also inspires the numerical sampling method discussed in § 5.1.
A probabilistic representation
We start with an informal sketch of the construction; a precise formulation follows below. Consider the ellipsoid , obtained by applying to the unit ball, and start an isotropic Brownian motion at the origin. Let denote its exit time, that is, the first time the path reaches the boundary of the ellipsoid, and let be its exit location. Independently of the Brownian motion, assign to each path an exponentially distributed killing time, at which the path is deleted. Retain the exit locations of those paths which reach the boundary before being killed. Theorem 6 below states that these locations define a measure with a smooth, strictly positive density on the boundary of the ellipsoid. Mapping this measure back to the sphere by and normalizing its mass gives precisely .
While passing from the sphere to the boundary of an ellipsoid and back gives a clean geometric intuition, for the formal argument, we work directly in the unit ball with boundary. Instead of varying the domain, we use an anisotropic Brownian motion, with different rates of diffusion along the eigendirections of .
Tools and notation. Let be a probability space supporting a standard Brownian motion in , started at the origin, and write for expectation under . Here a -algebra on is a collection of subsets containing and closed under complements and countable unions. A filtration is an increasing family of sub--algebras of . We denote by the natural filtration of , where is the smallest -algebra making all , , measurable. A process is adapted to if is -measurable for every . An event is a measurable set . For an introduction to the theoretical tools used in the proof, such as stochastic processes, martingales, Brownian motion, and stochastic differential equations (SDE), we refer the reader to Baldi [3]; Williams [47]; Le Gall [24].
Theorem 6 (Conditional exit measure). Consider a stochastic process on given by . Define
Let be independent of the Brownian path, so that for . Then,
Proof. Let denote the collection of all subsets of -null sets in , and set
Then is the completed natural filtration of . Since is invertible, and generate the same natural filtration. Thus is also the completed natural filtration of . Since , the first coordinate of is a non-degenerate real Brownian motion, which leaves almost surely (Le Gall, [24]), therefore . and has continuous paths, so
The process is adapted to , therefore is a -stopping time.
Define event . Conditioning on and using the independence of , we get
since almost surely. By (12),
Let be given by
Since is measurable, is a Borel measure on , with . Hence is a Borel probability measure on , and
By Proposition 1 and the linearity of , it is enough to show that is constant on . Since and are -measurable, conditioning on gives
where the last equality follows from the independence of and , together with . Hence, by the standard machine (Williams, [47]), for every bounded Borel on ,
Fix . Taking gives
Set . Since , is a real Brownian motion, and since is independent of for , it is a -Brownian motion. Hence
is a -martingale; see, e.g., Durrett [14], Le Gall [24]. Since and has continuous paths, for all , so
Thus the stopped martingale is bounded, hence uniformly integrable. Applying the optional stopping theorem [24 Theorem 3.22] to this martingale gives
Putting (15) into (14) yields , independent of . Consequently, by linearity,
The claim follows by Proposition 1.
Remark 2. The process is an anisotropic Brownian motion, with a covariance matrix
In particular, if is a unit eigenvector of with eigenvalue , then has variance . Thus diffusion is faster in directions corresponding to smaller eigenvalues of .
This identity gives a comparison between interaction matrices. Let be symmetric and suppose that . Construct
using the same Brownian path. Then, for every ,
Whenever reaches the sphere, has already reached it. Their exit times therefore satisfy almost surely, and hence
By (16) and Proposition 2,
In particular, the condition holds when and admit a common orthonormal eigenbasis and each eigenvalue of is at most the corresponding eigenvalue of . The comparison thus follows by coupling the paths, without computing either minimizing density.
Smoothness and positivity of the density
In this section we show that the measure admits a density with respect to the surface measure, and obtain an expression via the Dirichlet Green function of a certain elliptic operator built from the generator of the diffusion process . The argument combines the Feynman–Kac formula with Green’s representation formula. Using elliptic regularity and Hopf’s boundary point lemma we then prove smoothness and strict positivity of the density, and hence full support of the minimizer. For a classical treatment of the elliptic theory used in the proof, we refer to Gilbarg and Trudinger [18].
Theorem 7. The measure admits a density with respect to the normalized surface measure , given by
The function is smooth and strictly positive on .
Proof. The generator of the process is defined on by
By Itô’s formula (Karatzas and Shreve, [21]),
where we used .
For , let be the unique solution of
whose existence follows from Gilbarg and Trudinger [18]. Since by Karatzas and Shreve [21], applying the Feynman–Kac representation (Karatzas and Shreve, [21]) with operator , killing rate , and zero source term gives
Set . The operator is uniformly elliptic, since
and it is formally self-adjoint because is symmetric. On , the outward unit normal is . Set , and write
for the conormal derivative. Let denote the negative of the Dirichlet Green function of in with pole at the origin, so that
Its existence and Green’s representation formula are shown in, e.g., Véron [44]. Since , this representation gives
Combining (13) with the Feynman–Kac representation (18) and (19) yields, for every ,
Since is dense in , this identifies , where . Elliptic regularity away from the pole gives (Gilbarg and Trudinger, [18]). In particular, . Moreover, the maximum principle gives in , while on . Hopf’s boundary lemma, applied near each boundary point, therefore gives
see Gilbarg and Trudinger [18]. Since on , i.e., constant, its tangential derivatives vanish there. Decomposing the ordinary gradient into its tangential and normal components therefore gives
Consequently,
Write , so that . Since , Theorem 6 gives
Thus has the strictly positive smooth density with respect to , and .
Remark 3. The function is the Poisson kernel at the origin for the operator in , with respect to surface measure . Thus the density of is obtained by normalizing this kernel. For a general discussion of Poisson kernels and their relation to Dirichlet problems, we refer to Véron [44].
Resolution of the conjecture. The following immediate consequence of the result above resolves the conjecture of Burger et al. [9].
Corollary 8. The density of is bounded away from zero on :
In particular, .
Proof. The continuous, strictly positive function attains a positive minimum on the compact set .
We now formulate the following conjecture concerning the minimizers of .
Conjecture. The minimum of on is attained at every unit vector belonging to the eigenspace associated with the largest eigenvalue of .
The probabilistic analysis in § 3.1 provides intuition for this conjecture. Since the law of the diffusion is invariant under orthogonal transformations within each eigenspace of , the density takes the same value at all unit vectors in that eigenspace. However, despite the apparent simplicity of the statement, we have not been able to establish it rigorously.
Energy landscape
A standard approach to the minimization problem (2) is to equip with a suitable metric and consider the corresponding gradient flow of . Different choices of metric lead to different descent dynamics. For the Wasserstein distance , the evolution satisfies
see, for instance, Villani [45]; Santambrogio (2017). A closely related evolution is the normalized repulsive self-attention equation studied by Geshkovski et al. [17] and Burger et al. [9],
This evolution is a gradient flow of the same energy in a modified transport geometry with nonlocal mobility (induced by a modified Wasserstein distance). Both equations are understood in the distributional sense.
The existence of a unique global minimizer does not by itself ensure convergence of either flow to it. We therefore study stationary measures of and show that every stationary measure except for is singular with respect to the surface measure. We also remark that these measures can’t be local minimizers with respect to the above Wasserstein-type distances. We then study the stability of under perturbations of the interaction matrix.
Stationary measures with respect to Wasserstein flows
We say that a probability measure is stationary if
Indeed, testing the respective stationary equations against gives
and,
Since is strictly positive, either identity implies (22). Conversely, (22) implies that both and vanish -almost everywhere, so satisfies both stationary equations (20) and (21). Hence the constant curve solves either evolution. Smoothness of on the compact sphere and the bound make both velocity fields Lipschitz in space and in the measure argument for . The corresponding evoluations are therefore unique, so a solution initialized at remains there. This characterization also agrees with stationarity under transport variations; see Burger et al. [9].
The potential is constant on , so is stationary. Other stationary measures exist. For example, if is an eigenvector of , then
so is stationary. We now show that every stationary measure distinct from has support of zero surface measure. Thus is the only stationary probability measure having a density with respect to .
Proposition 9 (Singularity of non-minimizing stationary measures). Let satisfy (22). If , then is contained in the zero set of a nonzero real-analytic function on . In particular,
Consequently, is the only stationary probability measure absolutely continuous with respect to . For , every stationary measure distinct from has finite support.
Proof. We first verify that extends to a real-analytic function on . For , expansion of the exponential gives
The exchange of the sum and the integral is justified by
Thus the power series converges absolutely and uniformly for in bounded sets, proving analyticity.
On the sphere,
It follows that given by
is real-analytic. By (22), -almost everywhere. Continuity of then gives
If , connectedness of implies that is constant on . Proposition 2 then gives .
Suppose now that . Then . Since is a connected real-analytic manifold, Langrenez et al. [23] gives
Hence , which proves singularity. Since has a smooth, strictly positive density by Theorem 7, it is the only absolutely continuous stationary probability measure.
For , the zeros of a nonzero real-analytic function on the circle are isolated by the one-dimensional identity theorem; see also Álvarez-López et al. [1]. Compactness of therefore makes the zero set finite, and the same holds for .
Remark 4. The fact that absolute continuity is a distinguishing property of the minimizer among all stationary points has an important consequence. If an evolutionary process (e.g., a gradient flow) converges weakly to an absolutely continuous stationary measure, then its limit must be . In particular, a uniform bound on the relative entropy with respect to ensures absolute continuity of every weak limit, by lower semicontinuity of relative entropy. Note, however, that preservation of absolute continuity at each finite time alone does not suffice to exclude a singular limit.
Remark 5 (Absence of local minimizers). Although the associated Wasserstein-type gradient flows admit non-minimizing stationary measures, there are no local minimizers except for . Indeed, take any metric that metrizes weak convergence of measures (e.g., one of the Wasserstein-type distances considered in the previous section) and define an open ball
Assume , , is a local minimizer of with respect to , that is, there exists an such that
Consider the segment
The map is continuous with respect to weak convergence of measures, and . Thus, an initial segment is contained in : there exists such that for every . By local minimality and strict convexity (3), we have
Subtracting and dividing by gives , which contradicts the global minimality of , hence .
Stability with respect to the interaction matrix
We now study the dependence of on the interaction matrix . The lower bound in (10) allows us to control the distance between two minimizers by comparing their energies udner the two interaction kernels.
Proposition 10 (Stability with respect to the interaction matrix). Let be symmetric, and suppose that
Then
In particular, if and is a sequence of symmetric positive definite matrices such that , then .
Proof. For every , the mean value theorem gives
Integrating against any , we obtain
By minimality of for ,
Applying the lower bound in (10) proves (23).
Now suppose that . The sequence is bounded in operator norm, so we may choose such that and for every . Therefore,
Proposition 3 gives the claimed weak convergence.
Numerical methods
In this section we use the derived representations of the minimizer to construct a sampling scheme in arbitrary dimension and, in dimension two, to approximate the unnormalized density by an expansion in Mathieu functions. The code for reproducing the numerical experiments is available at github.com/TimRoith/EquilibriumTransformers.
Sampling from the minimizer
Theorem 6 gives a construction which allows to sample from the minimizer without evaluating the density. As explained in § 3.1, one may run in , starting at the origin, and kill it at an independent exponential time of rate one. Mapping a surviving exit back by gives a sample with law . We approximate these exits by a Walk on Spheres (WoS) [29, 15, 34].
Survival during one step. Suppose that the process starts at the center of a ball of radius contained in . Rotational invariance makes the exit direction uniform and independent of the exit time. For the generator and killing rate one, the probability of reaching the boundary of this ball before killing is
Here denotes the modified Bessel function of the first kind of order . By Rasila and Sottinen [30], is the total mass of the centered-ball panharmonic measure. The strong Markov property of the exponentially killed process allows these steps to be repeated.
Remark 6 (Numerical evaluation). For numerical evaluation, it is convenient to use the exponentially scaled modified Bessel function
Multiplying the numerator and denominator in (24) by gives, for ,
Evaluating directly avoids overflow due to the exponential growth of for large arguments. The value is defined by continuity.
For , the radius
gives an admissible ball. Indeed, for every ,
Thus . We use a dimensionless tolerance on , and project to the sphere when this quantity is at most .
Algorithm 1 One walk-on-spheres trial for an approximate sample from .
Require: Symmetric , tolerance
while true do
if then
return
Draw uniformly on
if then
return Killed
Draw , independently
Convergence as the tolerance vanishes. We next show that the law of an accepted output converges to as the stopping tolerance tends to zero.
Proposition 11 (Convergence of the sampling law). Let be symmetric. For , denote by the law of the output of Algorithm 1, conditional on the trial being successful. Then
Remark 7. Convergence of the classical walk on spheres is recalled in Kyprianou et al. [22]. Related results for walks with killing and for the error due to the boundary stopping tolerance are given by Bossy et al. [6] and Mascagni and Hwang [26], respectively. These results do not state convergence of the conditional sampling law for the anisotropic domain (i.e., ellipsoids) and stopping rule used here. We therefore give a full proof.
Proof. Throughout the proof, convergence of random variables is understood to hold almost surely.
Let denote the time along the Brownian path corresponding to the -th jump of the walk without killing. Set and, whenever , define
If , set . By the strong Markov property and the local exit law, the positions follow the jump rule of Algorithm 1, with killing omitted. The choice of ensures that , so almost surely. Moreover, continuity of gives
By the definition of the successive exit times,
Thus , and .
Define the number of jumps needed to reach the -neighbourhood of the boundary, along a given trajectory of the walk without killing, i.e.,
Then , and
So that
At each step, the exit location is independent of the exit time, and is the probability of surviving until that exit. The strong Markov property of the killed process therefore implies that the accepted output has the law of conditional on . Its acceptance probability satisfies
Consequently, for every , dominated convergence and Theorem 6 yield
recovering the claim.
Note that the acceptance probability converges to . Combining (16) with (5), we obtain the following relation between the normalization, the sampling cost, and the minimum:
Sampling error and the energy gap. Independent repetitions, retaining only successful trials, yield a sequence of independent random variables with common law . By the strong law of large numbers for empirical measures [43], for each fixed ,
We now compare the empirical energy of the accepted outputs with the minimum energy.
Proof. Independence gives
Consequently, Proposition 4 yields
where the last step uses on and .
The first term in (28) accounts for the stopping tolerance, and the second for the finite number of accepted samples. Since as , the first term tends to zero. Taking and therefore gives convergence in mean squared , and hence convergence of the expected energy to its minimum. Increasing at fixed does not remove the stopping-tolerance error. An explicit rate in would require a quantitative estimate of that term.
For a numerical comparison, we apply the projected gradient descent scheme from Geshkovski et al. [17], which optimizes the configuration of an ensemble . Set . Starting from an initial ensemble , we iterate
where for is applied to each particle and is a fixed step size. We compare the resulting energy with that of the empirical measure formed from independent accepted outputs of Algorithm 1. Figure 1 reports the two energies and their relative difference for increasing values of . Over the range shown, the relative difference decreases approximately as .
Figure 1. Comparison of Algorithm 1 with projected gradient descent (29), for a fixed randomly generated matrix in dimension . Left: particle energies at equal ensemble sizes . Right: the sampling energy minus the gradient descent energy, divided by the latter. The dashed line is proportional to .
Density approximation using Mathieu functions
In this section we use the Dirichlet representation to expand the density of the unnormalized measure (13) in an orthogonal basis. In dimension two, separation of variables in elliptic coordinates gives an explicit expansion in Mathieu functions.
Throughout this section, we assume that is given in diagonal form. Since is invariant under coordinate reflections by Burger et al. [9], its unnormalized density satisfies , where
is the space of functions on the sphere invariant under coordinate reflections. For every , the matrix is orthogonal and therefore preserves surface measure on . Consequently, defines a linear isometry of . The fixed-point space of each such isometry is a closed linear subspace of . Hence their intersection, , is also a closed linear subspace of .
For an orthogonal basis of we write the orthogonal expansion of and, for each integer , its truncation as
Since is an orthogonal basis of , we have
In particular, the associated measures as . For any $n\in\mathbb{N}_0, the Dirichlet problem (17) with boundary values admits a unique solution . It directly follows from (19) that . This yields
To evaluate the coefficients, we need solutions with a given boundary data. In dimension , pairs of boundary values and solutions are available through Mathieu functions, see Meixner and Schäfke [28]; Whittaker and Watson [46]. These functions arise by separation of variables in the modified Helmholtz equation,
More precisely, let us assume that the interaction matrix is given in diagonal form with . For , we consider the elliptical coordinate transform
We note that for and it holds that . We consider solutions of (31) of the form
where and denote the angular and modified Mathieu functions, respectively. They satisfy Mathieu’s equation and its modified form (Wolf, [48], Sections 28.2, 28.20):
Here, and depends on . Accordingly, the functions depend on , too, however we suppress the dependence on for brevity. We refer to (McLachlan, [27], Chapter 2) for a detailed introduction. The functions defined by
form an orthogonal basis of , which can be seen as follows. As shown in (McLachlan, [27], Chapter 2), the set is an orthogonal basis of , where are further solutions of Mathieu’s equation, substituting with different values . In fact, consists precisely of those basis elements satisfying the symmetry conditions and , which correspond to the symmetries defining . In contrast, is odd and for every . Hence the corresponding functions on are orthogonal to .
We next compute and , which determine the coefficients in (30). To this end, let denote the restriction of to the ellipse . It directly follows that solves the Dirichlet problem
Equivalently, since , the function solves initial Dirichlet problem on (17) with boundary function . With the conventional definition of Mathieu functions (McLachlan, [27], Section 2.21), it holds that . Therefore,
Figure 2: Mathieu approximation for matrices with varying .
Note that : otherwise, the function would solve the Dirichlet problem with boundary data , contradicting uniqueness.
Numerical evaluation involves two approximations. We first truncate the series in (30) at order , obtaining
Since , it follows from [36 Theorem 4.1] that, for every
where is independent of and denotes the norm of the Sobolev space . In particular, this yields uniform convergence of the approximation as .
The evaluation of is also done approximately by a truncated Fourier series as in [13 eq. 16–20], where the Fourier coefficients are calculated by the term-recurrences specified therein. In Figure 2 we visualize the approximations obtained this way for different interaction matrices . We evaluate the energy of an empirical approximation supported on equally spaced points of . For , set
Figure 2. (a) Visualization of the approximation . (b) Energies of Mathieu approximation compared to the ones obtained with (29) for .
and define
We then report . By Corollary 8, is bounded away from zero. Uniform convergence therefore implies the existence of such that on for every . In the implementation, we start with and double until all evaluated weights are positive. This condition ensures that the normalized discrete measure is a probability measure; it does not provide an estimate of the truncation error.
Acknowledgments
The authors thank the organizers of the 2nd Workshop on Machine Learning in Infinite Dimensions, held at ETH Zürich on 8–11 September 2025, where this work originated. Y.K. is grateful to the University of Genova for supporting his research stay in Autumn 2026, when part of this work was undertaken.
O.K. acknowledges financial support of the European Commission (Horizon Europe grant ELIAS 101120237). M.B., S.K. and T.R. acknowledge support from DESY (Hamburg, Germany), a member of the Helmholtz Association HGF. This work was carried in part while T.R. was a member of DESY and was supported in part through the Maxwell computational resources operated at Deutsches Elektronen-Synchrotron DESY, Hamburg, Germany. M.B. and T.R. acknowledge funding by the German Federal Ministry of Research, Technology and Space (BMFTR) under grant agreement No. 16IS24072B and 16IS24072D (COMFORT). M.B. and S.K. acknowledge support from the German Research Foundation, project BU 2327/19-2. T.R. acknowledges the support of COST Action CA24136 - InterCoML, the Munich Center for Machine Learning and the ERC Advanced Grant NEITALG, grant agreement No. 101198055. Funded by the European Union. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
Tool and computational resource disclosure. Generative AI was used for literature review, assistance with code for the numerical experiments, as well as language and grammar corrections.
References
- [1]Antonio Álvarez-López, Borjan Geshkovski, and Domènec Ruiz-Balet. Perceptrons and localization of attention’s mean-field landscape. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 2180–2206. PMLR, 2026. URL https://proceedings.mlr.press/v306/alvarez-lopez26a.html.
- [2]Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68(3):337–404, 1950. doi: 10.1090/S0002-9947-1950-0051437-7.DOI
- [3]Paolo Baldi. Stochastic Calculus: An Introduction Through Theory and Exercises. Universitext. Springer, Cham, 2017. ISBN 978-3-319-62225-5. doi: 10.1007/978-3-319-62226-2.DOI
- [4]Dmitriy Bilyk, Ryan W. Matzke, and Oleksandr Vlasiuk. Positive definiteness and the Stolarsky invariance principle. Journal of Mathematical Analysis and Applications, 513(2):126220, 2022. doi: 10.1016/j.jmaa.2022.126220. URL https://doi.org/10.1016/j.jmaa.2022.126220.
- [5]Sergiy V. Borodachov, Douglas P. Hardin, and Edward B. Saff. Discrete Energy on Rectifiable Sets. Springer Monographs in Mathematics. Springer, New York, 2019. doi: 10.1007/978-0-387-84808-2. URL https://doi.org/10.1007/978-0-387-84808-2.
- [6]Mireille Bossy, Nicolas Champagnat, Sylvain Maire, and Denis Talay. Probabilistic interpretation and random walk on spheres algorithms for the Poisson–Boltzmann equation in molecular dynamics. ESAIM: Mathematical Modelling and Numerical Analysis, 44(5):997–1048, 2010. doi: 10.1051/m2an/2010050.DOI
- [7]Martin Burger and Marco Di Francesco. Large time behavior of nonlocal aggregation models with nonlinear diffusion. Networks and Heterogeneous Media, 3(4):749–785, 2008.DOI
- [8]Martin Burger, Razvan Fetecau, and Yanghong Huang. Stationary states and asymptotic behavior of aggregation models with nonlinear local repulsion. SIAM Journal on Applied Dynamical Systems, 13(1):397–424, 2014.DOI
- [9]Martin Burger, Samira Kabri, Yury Korolev, Tim Roith, and Lukas Weigand. Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 383(2298):20240233, 2025. doi: 10.1098/rsta.2024.0233. URL https://doi.org/10.1098/rsta.2024.0233.
- [10]José A. Cañizo and Alejandro Ramos-Lora. Discrete minimizers of the interaction energy in collective behavior: a brief numerical and analytic review. In José Antonio Carrillo and Eitan Tadmor, editors, Active Particles, Volume 4: Theory, Models, Applications, Modeling and Simulation in Science, Engineering and Technology, pages 55–78. Birkhäuser, Cham, 2024. doi: 10.1007/978-3-031-73423-6_2.DOI
- [11]José A. Carrillo, Alessio Figalli, and Francesco S. Patacchini. Geometry of minimizers for the interaction energy with mildly repulsive potentials. Annales de l’Institut Henri Poincaré C, Analyse non linéaire, 34(5):1299–1308, 2017. doi: 10.1016/j.anihpc.2016.10.004.
- [12]José Antonio Carrillo, Michel Chipot, and Yanghong Huang. On global minimizers of repulsive-attractive power-law interaction energies. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 372(2028):20130399, 2014. doi: 10.1098/rsta.2013.0399. URL https://doi.org/10.1098/rsta.2013.0399.
- [13]Lorea Chaos-Cador and E. Ley-Koo. Mathieu functions revisited: matrix evaluation and generating functions. Revista Mexicana de Física, 48(1):67–75, 2002.
- [14]Rick Durrett. Probability: Theory and Examples, volume 49 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, fifth edition, 2019. ISBN 978-1-108-47368-2.
- [15]Boris Stepanovich Elepov and Guennady Alekseevich Mikhailov. Solution of the Dirichlet problem for the equation Δu − cu = −q by a model of “walks on spheres”. USSR Computational Mathematics and Mathematical Physics, 9(3):194–204, 1969.DOI
- [16]Amic Frouvelle and Jian-Guo Liu. Dynamics in a kinetic model of oriented particles with phase transition. SIAM Journal on Mathematical Analysis, 44(2):791–826, 2012. doi: 10.1137/110823912. URL https://doi.org/10.1137/110823912.
- [17]Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479, 2025. doi: 10.1090/bull/1863. URL https://doi.org/10.1090/bull/1863.
- [18]David Gilbarg and Neil S. Trudinger. Elliptic Partial Differential Equations of Second Order. Classics in Mathematics. Springer, Berlin, Heidelberg, second edition, 2001. doi: 10.1007/978-3-642-61798-0. URL https://doi.org/10.1007/978-3-642-61798-0.
- [19]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012. URL https://jmlr.org/papers/v13/gretton12a.html.DOI
- [20]Douglas P Hardin and Edward B Saff. Minimal riesz energy point configurations for rectifiable d-dimensional manifolds. Advances in Mathematics, 193(1):174–204, 2005.
- [21]Ioannis Karatzas and Steven E. Shreve. Brownian Motion and Stochastic Calculus, volume 113 of Graduate Texts in Mathematics. Springer, New York, NY, second edition, 1991. ISBN 978-0-387-97655-6. doi: 10.1007/978-1-4612-0949-2.
- [22]Andreas E. Kyprianou, Ana Osojnik, and Tony Shardlow. Unbiased ‘walk-on-spheres’ Monte Carlo methods for the fractional Laplacian. IMA Journal of Numerical Analysis, 38(3):1550–1578, 2018. doi: 10.1093/imanum/drx042.DOI
- [23]Christopher Langrenez, Wilfred Salmon, Stephan De Bièvre, Jonathan J. Thio, Christopher K. Long, and David R. M. Arvidsson-Shukur. Almost sure minimality of the set of experiments with classical Kirkwood–Dirac representations. Physical Review A, 113(6):062215, June 2026. doi: 10.1103/v7z4-qsz8.DOI
- [24]Jean-François Le Gall. Brownian Motion, Martingales, and Stochastic Calculus, volume 274 of Graduate Texts in Mathematics. Springer, Cham, 2016. ISBN 978-3-319-31088-6. doi: 10.1007/978-3-319-31089-3.
- [25]J Maćkowiak. On the mean field description of systems of point particles in equilibrium. Physica A: Statistical Mechanics and its Applications, 110(1-2):302–320, 1982.DOI
- [26]Michael Mascagni and Chi-Ok Hwang. ϵ-shell error analysis for “Walk On Spheres” algorithms. Mathematics and Computers in Simulation, 63(2):93–104, 2003. doi: 10.1016/S0378-4754(03)00038-7.DOI
- [27]Norman W. McLachlan. Theory and Application of Mathieu Functions. Oxford University Press, Oxford, 1951.
- [28]Josef Meixner and Friedrich Wilhelm Schäfke. Mathieusche Funktionen und Sphäroidfunktionen: Mit Anwendungen auf physikalische und technische Probleme, volume 71 of Die Grundlehren der mathematischen Wissenschaften. Springer, Berlin, Heidelberg, 1954. doi: 10.1007/978-3-662-00941-3.DOI
- [29]Mervin E. Muller. Some continuous Monte Carlo methods for the Dirichlet problem. The Annals of Mathematical Statistics, 27(3):569–589, 1956. doi: 10.1214/aoms/1177728169.DOI
- [30]Antti Rasila and Tommi Sottinen. Yukawa potential, panharmonic measure and Brownian motion. Axioms, 7(2):28, 2018. doi: 10.3390/axioms7020028. URL https://doi.org/10.3390/axioms7020028.
- [31]Edward B Saff and Amo BJ Kuijlaars. Distributing many points on a sphere. Mathematical Intelligencer, 19(1):5, 1997.DOI
- [32]Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 3515–3530. PMLR, 2022. URL https://proceedings.mlr.press/v151/sander22a.html.arxiv.org/abs/2110.11773
- [34]Rohan Sawhney and Keenan Crane. Monte Carlo geometry processing: A grid-free approach to PDE-based methods on volumetric domains. ACM Transactions on Graphics, 39(4), 2020. doi: 10.1145/3386569.3392374.DOI
- [35]Sylvia Serfaty. Ginzburg–landau vortices, coulomb gases, and abrikosov lattices. Comptes Rendus. Physique, 15(6):539–546, 2014.DOI
- [36]Jie Shen and Li-Lian Wang. On spectral approximations in elliptical geometries using Mathieu functions. Mathematics of Computation, 78(266):815–844, 2009.DOI
- [37]Ruiwen Shu. Wasserstein-infinity stability and mean field limit of discrete interaction energy minimizers. arXiv preprint arXiv:2407.18395, 2024. URL https://arxiv.org/abs/2407.18395.
- [38]Robert Simione, Dejan Slepčev, and Ihsan Topaloglu. Existence of ground states of nonlocal-interaction energies. Journal of Statistical Physics, 159(4):972–986, 2015.
- [39]Carl-Johann Simon-Gabriel and Bernhard Schölkopf. Kernel distribution embeddings: Universal kernels, characteristic kernels and kernel metrics on distributions. Journal of Machine Learning Research, 19(44):1–29, 2018. URL https://jmlr.org/papers/v19/16-291.html.
- [40]Carl-Johann Simon-Gabriel, Alessandro Barp, Bernhard Schölkopf, and Lester Mackey. Metrizing weak convergence with maximum mean discrepancies. Journal of Machine Learning Research, 24 (184):1–20, 2023. URL https://jmlr.org/papers/v24/21-0599.html.
- [41]Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R. G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12(70):2389–2410, 2011. URL https://jmlr.org/papers/v12/sriperumbudur11a.html.arxiv.org/abs/1003.0887
- [42]PAJ Tindemans and HW Capel. An exact calculation of the free energy in systems with separable interactions. Physica, 72(3):433–464, 1974.DOI
- [43]V. S. Varadarajan. On the convergence of sample probability distributions. Sankhyā: The Indian Journal of Statistics, 19(1/2):23–26, 1958. URL https://www.jstor.org/stable/25048365.
- [44]Laurent Véron. Elliptic equations involving measures. In Michel Chipot and Pavol Quittner, editors, Handbook of Differential Equations: Stationary Partial Differential Equations, volume 1, pages 593–712. Elsevier/North-Holland, Amsterdam, 2004. ISBN 978-0-444-51126-3. doi: 10.1016/S1874-5733(04)80010-X.DOI
- [45]Cédric Villani. Optimal Transport: Old and New, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer, 2009. doi: 10.1007/978-3-540-71050-9.
- [46]Edmund Taylor Whittaker and George Neville Watson. A Course of Modern Analysis. Cambridge University Press, Cambridge, fourth edition, 1927.
- [47]David Williams. Probability with Martingales. Cambridge Mathematical Textbooks, Cambridge University Press, Cambridge, 1991. ISBN 978-0-521-40605-5. doi: 10.1017/CBO9780511813658.DOI
- [48]Gerhard Wolf. Mathieu functions and Hill’s equation. In F. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, editors, NIST Digital Library of Mathematical Functions, chapter 28. National Institute of Standards and Technology, 2026. URL https://dlmf.nist.gov/28. Release 1.2.8 of 2026-09-15.


