Introduction
Operator learning requires neural networks that approximate maps between functions. The input functions of the target operator are continuum objects that cannot be represented exactly on a computer. Furthermore, in scientific applications using real-world data, we generally do not have complete knowledge of the input function f f f , but rather only a finite collection of (sometimes noisy) samples from the graph of a function ( x , f ( x ) ) (x, f(x)) ( x , f ( x )) . To accurately model the continuum target operator, a neural network should not be sensitive to small changes in the discretization of the input. The output of a “discretization invariant” network converges as the discretization of the input function is refined. Quantitative guarantees about the trained network, however, require stability bounds, not merely convergence under discretization refinement. This leads to the following three questions:
What discretization stability properties are relevant for operator learning?
What network architectures satisfy these stability properties?
How can these architectures be characterized and systematically explored?
Our approach is to lift operators between functions to operators between measures. This provides a unified viewpoint of continuum inputs (functions) and their noisy discretizations. The necessary stability properties can then be stated as a Hölder stability with respect to a Wasserstein metric. Uniform discretization stability controls the behavior of learned operator networks under different discretizations of the input functions. For example, it provides quantitative guarantees for networks that are trained at one resolution or on a uniform grid, but evaluated on non-uniform point clouds sampled at a different resolution. It also provides a unified framework for the analysis of errors arising from both discretization and noisy measurements.
We then turn to the second question, and introduce Markov kernel networks , a shared generalization of transformers (Vaswani et al., 2017), FNO [22 ] , Flowers [26 ] , and other operator networks. These networks alternate nonlinear integral operators built using Markov kernels to mix information spatially with channel-wise mixing operators to mix information pointwise in the embedding space. When the individual components of these networks satisfy relevant stability properties, the entire network becomes uniformly discretization stable. Conversely, our main theorem shows that every uniformly discretization stable operator can be written exactly (not approximately) as a stable Markov kernel network.
With the design space of discretization-stable architectures fixed by the representation theorem, we propose a variational approach to explore the design space. We investigate Markov kernels that minimize regularized objectives; the induced read behavior is determined by the choice of regularizer and cost function. We provide sufficient conditions on the regularizers and costs so that the resulting networks are guaranteed to be uniformly discretization stable, and identify the regularizers corresponding to the popular network architectures discussed earlier. We aim to provide a principled framework for adaptive [6 ] and hybrid [1 ] architectures, and a useful belief space for agentic (Toscano et al., 2026) approaches to operator learning.
Prior work
Discretization invariance was initially explored in the context of ill-posed inverse problems. The term was first defined in Lassas & Siltanen [19 ] , with important antecedents in Lasanen [18 ] ; D’Ambrogi et al. [7 ] ; Lehtinen et al. [21 ] ; see Lassas et al. [20 ] and Stuart [30 ] for additional historical remarks. It was popularized in operator learning by (earlier preprints of) the seminal work Kovachki et al. [16 ] , which includes analysis of operators that converge under discretization refinement. Various notions of quantitative discretization invariance and stability bounds have been developed in Wang & Golland (2023); Bartolucci et al. [2 ] ; Lanthaler et al. [17 ] ; de Bie et al. [8 ] ; Ruiz et al. [29 ] ; Gao et al. [13 ] ; Subedi & Tewari [31 ] , and Bendahi et al. [3 ] . Stability of transformers was investigated in Castin et al. [4 ] . The notion of uniform discretization stability we propose in § 2 has roots in Transportation-L p L^p L p theory [14 , 33 ] ; we are unaware of prior work using these Hölder bounds in the context of operator learning, but Furuya et al. [12 ] discusses a similar Lipschitz bound for transformers. This paper builds on both the stability bound and “graph measure” viewpoint of that work.
The mindset for our representation theorem and variational characterizations is inspired by Unser (2019), which proved that optimal activation functions (in a total-variation sense) for multi-layer perceptrons take the form of linear splines with adaptive knot locations. Our presentation of Markov kernel networks directly mirrors measure-theoretic transformers [11 , 12 ] . Niculae & Blondel [27 ] characterized softmax attention as a minimizer of a KL-regularized objective, which was further explored in Martins et al. [25 ] ; Terjék [32 ] .
Paper outline and contributions
In § 2 , we propose uniform discretization-stability , a Hölder stability property, as a quantitative notion of discretization invariance.
In § 3 , we define Markov kernel networks , an abstract architecture that generalizes transformers, neural operators, Flowers, and others. Our main results are the representation theorems 3.6 and 3.7 establishing an equivalence between uniformly discretization-stable operators and stable Markov kernel networks. This suggests Markov kernel networks provide a natural design space for uniformly discretization-stable operator learning.
§ 4 explores the design space of Markov kernel networks using a variational characterization of discretization stable networks. We investigate Markov kernels that minimize a particular regularized objective function, and show how certain regularizers determine different read behavior, including those corresponding to existing architectures and hybrids. We establish stability bounds for variationally defined Markov kernels.
Section 5 discusses some limitations of our results. The appendices contain background and notation used throughout the document (Appendix § 6 ), proofs (Appendix § 7 ), a universal approximation result for Markov kernel networks (Appendix § 8 ), constructions for popular architectures as Markov kernel networks (Appendix § 9 ), and an additional discussion of hybrid architectures defined by variational Markov reads (Appendix § 10 ).
Uniform discretization stability
The target operator maps functions to functions, but in practice the network receives samples of the input function and their locations (Figure 1 ). We represent both these samples and the continuum input as probability measures on a space of functions graphs [12 ] . The network then maps an input measure to an output function, allowing us to quantify how changes in sampling affect its predictions.
Figure 1. Operator learning setup. The target operator is a continuum object f ↦ A [ f ] f \mapsto A[f] f ↦ A [ f ] . A neural network N \mathcal{N} N takes as input a measure μ \mu μ representing a noisy sample of points from the graph of f f f (denoted γ f \gamma_f γ f ), and outputs a function that can be evaluated at any point in the domain X X X . The network N \mathcal{N} N is uniformly discretization stable (Definition 2.1 ) if it satisfies a Hölder stability property that provides quantitative guarantees that N ( μ , ⋅ ) \mathcal{N}(\mu,\cdot) N ( μ , ⋅ ) converges to N ( γ f , ⋅ ) \mathcal{N}(\gamma_f,\cdot) N ( γ f , ⋅ ) , the output had the network been provided with the ideal (continuum) input γ f \gamma_f γ f .
Let ( X , ρ X ) (X,\rho_X) ( X , ρ X ) be a compact metric space with at least two elements, and Y ⊂ R d o u t Y \subset\mathbb{R}^{d_{\mathrm{out}}} Y ⊂ R d out be non-empty, compact, and convex. Define the graph metric space G : = X × Y G := X \times Y G := X × Y equipped with the metric
ρ G : G × G → R ≥ 0 , ρ G ( ( x 1 , y 1 ) , ( x 2 , y 2 ) ) : = ρ X ( x 1 , x 2 ) 2 + ∣ y 1 − y 2 ∣ 2 . (1) \rho_G : G \times G \to\mathbb{R}_{\ge0}, \qquad\rho_G((x_1,y_1),(x_2,y_2)) := \sqrt{\rho_X(x_1,x_2)^2 + \lvert y_1-y_2\rvert^2}. \tag*{(1)} ρ G : G × G → R ≥ 0 , ρ G (( x 1 , y 1 ) , ( x 2 , y 2 )) := ρ X ( x 1 , x 2 ) 2 + ∣ y 1 − y 2 ∣ 2 . (1)
Choose a Radon probability measure λ ∈ P ( X ) \lambda\in\mathcal{P}(X) λ ∈ P ( X ) with supp λ = X \operatorname{supp}\lambda= X supp λ = X . Let L 2 ( X , Y ) L^2(X,Y) L 2 ( X , Y ) be the space of Radon-measurable functions f : X → Y f : X \to Y f : X → Y that are square integrable with respect to λ \lambda λ . Observe that since Y Y Y is compact, L 2 ( X , Y ) ⊂ L ∞ ( X , Y ) L^2(X,Y) \subset L^\infty(X,Y) L 2 ( X , Y ) ⊂ L ∞ ( X , Y ) . In this paper, we consider operators A : L 2 ( X , Y ) → L 2 ( X , Y ) A : L^2(X,Y) \to L^2(X,Y) A : L 2 ( X , Y ) → L 2 ( X , Y ) . All maps in the paper are Borel measurable.
We identify a function f ∈ L 2 ( X , Y ) f \in L^2(X,Y) f ∈ L 2 ( X , Y ) with its graph measure [12 ] ,
γ f ∈ P ( G ) , γ f : = ( id , f ) # λ , ( id , f ) : X → X × Y . (2) \gamma_f \in\mathcal{P}(G), \qquad\gamma_f := (\operatorname{id},f)_\#\lambda,\qquad(\operatorname{id},f) : X \to X \times Y. \tag*{(2)} γ f ∈ P ( G ) , γ f := ( id , f ) # λ , ( id , f ) : X → X × Y . (2)
(See Appendix § 6 for the definition of the pushforward operator “# \# # ”.) The map f ↦ γ f f \mapsto\gamma_f f ↦ γ f is injective. For helpful intuition, when f f f is continuous (so it is defined pointwise and has a closed graph) we have supp γ f = { ( x , f ( x ) ) : x ∈ X } \operatorname{supp}\gamma_f = \{(x,f(x)) : x \in X\} supp γ f = {( x , f ( x )) : x ∈ X } .
Computer implementations of neural networks for function-to-function maps generally do not accept literal functions f ∈ L 2 ( X , Y ) f \in L^2(X,Y) f ∈ L 2 ( X , Y ) , but instead f f f is approximated by a finite collection of points on γ f \gamma_f γ f , or only near γ f \gamma_f γ f if measurements include noise. We model these sampled points as a mixture of Dirac measures. Numerically, f ∈ L 2 ( X , Y ) f \in L^2(X,Y) f ∈ L 2 ( X , Y ) is approximated by sampling a finite collection of domain points { x i } i = 1 N \{x_i\}_{i=1}^{N} { x i } i = 1 N and forming an empirical approximation of γ f \gamma_f γ f ,
μ ∈ P ( G ) , μ = ∑ i = 1 N w i δ ( x i , y i ) , w i ≥ 0 , ∑ i = 1 N w i = 1. (3) \mu\in\mathcal{P}(G), \qquad\mu= \sum_{i=1}^{N} w_i\delta_{(x_i,y_i)}, \qquad w_i \ge0,\qquad\sum_{i=1}^{N} w_i = 1. \tag*{(3)} μ ∈ P ( G ) , μ = i = 1 ∑ N w i δ ( x i , y i ) , w i ≥ 0 , i = 1 ∑ N w i = 1. (3)
When the samples ( x i , y i ) (x_i, y_i) ( x i , y i ) lie on the graph of a function f f f , we have y i = f ( x i ) y_i = f(x_i) y i = f ( x i ) . The mixture weights w i w_i w i allow the samples to approximate a graph measure even when the x i x_i x i locations are not distributed uniformly with respect to λ \lambda λ . This formulation captures both discretizations on uniform grids and irregularly sampled point clouds. Provided with samples μ \mu μ to represent the input function f f f , an operator network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(G) \times X \to Y N : P ( G ) × X → Y can then be evaluated at an arbitrary x ∈ X x \in X x ∈ X . Informally, a trained neural network N \mathcal{N} N is a good approximation of A A A when
μ ≈ γ f ⟹ N ( μ , ⋅ ) ≈ A [ f ] for all f ∈ L 2 ( X , Y ) . (4) \mu\approx\gamma_f \quad\Longrightarrow\quad\mathcal{N}(\mu,\cdot) \approx A[f] \qquad\text{for all } f \in L^2(X,Y). \tag*{(4)} μ ≈ γ f ⟹ N ( μ , ⋅ ) ≈ A [ f ] for all f ∈ L 2 ( X , Y ) . (4)
To make the above precise, we require a metric on measures to compare the graph measure γ f \gamma_f γ f to the noisy samples μ \mu μ . For this, we use the Wasserstein-1 optimal transport metric (Appendix § 6 .§ 1 ).
We now formulate uniform discretization stability as a Hölder-stability property for neural networks.
Definition 2.1 (Uniform discretization stability). A network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(G) \times X \to Y N : P ( G ) × X → Y is uniformly discretization stable on a set S ⊆ P ( G ) S \subseteq\mathcal{P}(G) S ⊆ P ( G ) if there exist constants C > 0 C > 0 C > 0 , β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] such that
sup x ∈ X ∣ N ( μ , x ) − N ( ν , x ) ∣ ≤ C W 1 ( μ , ν ) β for all μ , ν ∈ S . (5) \sup_{x \in X} \left|\mathcal{N}(\mu,x)-\mathcal{N}(\nu,x)\right| \le C W_1(\mu,\nu)^\beta\qquad\text{for all } \mu,\nu\in S. \tag*{(5)} x ∈ X sup ∣ N ( μ , x ) − N ( ν , x ) ∣ ≤ C W 1 ( μ , ν ) β for all μ , ν ∈ S . (5)
Returning now to the approximation criterion (4), using the triangle inequality and the fact λ ( X ) = 1 \lambda(X)=1 λ ( X ) = 1 ,
∥ N ( μ , ⋅ ) − A [ f ] ∥ L 2 ( X , R d o u t ) ≤ ∥ N ( γ f , ⋅ ) − A [ f ] ∥ L 2 ( X , R d o u t ) + ∥ N ( μ , ⋅ ) − N ( γ f , ⋅ ) ∥ ∞ . (6) \left\|\mathcal{N}(\mu,\cdot)-A[f]\right\|_{L^2(X,\mathbb{R}^{d_{\mathrm{out}}})}
\le
\left\|\mathcal{N}(\gamma_f,\cdot)-A[f]\right\|_{L^2(X,\mathbb{R}^{d_{\mathrm{out}}})}
+
\left\|\mathcal{N}(\mu,\cdot)-\mathcal{N}(\gamma_f,\cdot)\right\|_\infty. \tag*{(6)} ∥ N ( μ , ⋅ ) − A [ f ] ∥ L 2 ( X , R d out ) ≤ ∥ N ( γ f , ⋅ ) − A [ f ] ∥ L 2 ( X , R d out ) + ∥ N ( μ , ⋅ ) − N ( γ f , ⋅ ) ∥ ∞ . (6)
The first term on the right is the approximation error of an ideal network given all information about the input function f f f ; this quantity is controlled by training the network. The second term is the error introduced by the discretization μ \mu μ . It is largely determined by the neural network architecture itself, and is not directly controlled by training. Discretization stable architectures satisfying (DS) control the discretization error, so they have the possibility to be trained to satisfy the desired property (4).
In the next section, we will show that all such networks have a particular representation as stable Markov kernel networks. This allows us to view, for example, softmax attention, and SelfWarps ([26 ] ) as two particular choices in the same space of discretization stable architectures.
Remark. Uniform discretization stability is a stronger condition than convergence under resolution refinement. In particular, (DS) allows a unified analysis of both resolution refinement and stability to noise in the measurements of f f f , both of which are important in applications. Furthermore, for sufficiently regular classes of target operators, it allows us to obtain quantitative guarantees for the performance of a network trained at one resolution but evaluated at another, which is important for scientific applications where large quantities of high-resolution training data may be prohibitively expensive. Indeed, if N \mathcal{N} N is uniformly discretization stable, then
∥ N ( μ t e s t , ⋅ ) − A [ f ] ∥ L 2 ( X , R d o u t ) ≤ ∥ N ( μ t r a i n , ⋅ ) − A [ f ] ∥ L 2 ( X , R d o u t ) + C W 1 ( μ t e s t , μ t r a i n ) β . (7) \left\|\mathcal{N}(\mu_{\mathrm{test}},\cdot)-A[f]\right\|_{L^2(X,\mathbb{R}^{d_{\mathrm{out}}})}
\le
\left\|\mathcal{N}(\mu_{\mathrm{train}},\cdot)-A[f]\right\|_{L^2(X,\mathbb{R}^{d_{\mathrm{out}}})}
+
C W_1(\mu_{\mathrm{test}},\mu_{\mathrm{train}})^\beta. \tag*{(7)} ∥ N ( μ test , ⋅ ) − A [ f ] ∥ L 2 ( X , R d out ) ≤ ∥ N ( μ train , ⋅ ) − A [ f ] ∥ L 2 ( X , R d out ) + C W 1 ( μ test , μ train ) β . (7)
where μ t r a i n \mu_{\mathrm{train}} μ train and μ t e s t \mu_{\mathrm{test}} μ test are discretizations of f f f at the training and test resolutions, respectively. Simple convergence under resolution refinement provides no such guarantees.
Remark. Definition 2.1 does not impose additional regularity requirements of the target operator A A A or the input functions f f f . See the discussion of universal approximation in Appendix § 8 .
Non-collapsing measures S S S
The set S ⊂ P ( G ) S \subset\mathcal{P}(G) S ⊂ P ( G ) in Definition 2.1 need not contain all probability measures to provide a useful notion of discretization stability. Our main result, Theorem 3.7, holds for admissible classes that are relevant for operator learning, but not all possible inputs in P ( G ) \mathcal{P}(G) P ( G ) (see Example 3.8).
Figure 2. A measure that is not in any admissible set S S S . Since ( π X ) # μ ≠ μ = δ a (\pi_X)_\#\mu\ne\mu= \delta_a ( π X ) # μ = μ = δ a , every set containing μ \mu μ , or a sequence converging to μ \mu μ , fails the admissibility criterion (7).
A set of measurements at only a single value of x x x does not provide much information about an input function f f f , and we are not concerned with the behavior of networks on such pathological inputs. To avoid this pathology, we define a condition on admissible sets of measures to ensure there is enough variation in the X X X -marginal.
Definition 2.2 (Admissible sets of non-collapsing measures). A non-empty set S ⊂ P ( G ) \mathcal{S} \subset\mathcal{P}(\mathcal{G}) S ⊂ P ( G ) is admissible if there exists ϵ > 0 \epsilon> 0 ϵ > 0 such that
inf μ ∈ S inf a ∈ X W 1 ( ( π X ) # μ , δ a ) > ϵ (8) \inf_{\mu\in\mathcal{S}}\inf_{a \in\mathcal{X}} W_1\left((\pi_X)_\#\mu,\delta_a\right) > \epsilon \tag*{(8)} μ ∈ S inf a ∈ X inf W 1 ( ( π X ) # μ , δ a ) > ϵ (8) where π X : ( x , y ) ↦ x \pi_X : (x,y) \mapsto x π X : ( x , y ) ↦ x is projection onto the X X X component.
Crucially, S \mathcal{S} S can contain both graph measures γ f \gamma_f γ f and empirical measures μ \mu μ built from finitely many samples, even in the presence of a large amount of noise, as long as the x x x coordinates do not collapse.
Markov kernel networks
We now define an abstract architecture for operator learning called Markov kernel networks , and show they exactly represent all uniformly discretization stable operators on admissible input classes.
Definitions
Let M M M be a metric space. A nonlinear Markov kernel is a map K : P 1 ( M ) × M → P 1 ( M ) K : \mathcal{P}_1(M) \times M \to\mathcal{P}_1(M) K : P 1 ( M ) × M → P 1 ( M ) . Markov kernels form the backbone of the “in-context maps” [11 ] that define the hidden layers of a Markov kernel network. An in-context map uses a state encoded as a probability measure, called a “context”, to determine how to update a single point.
Definition 3.1 (Markov kernel in-context map). Let E E E , E ′ E' E ′ be compact metric spaces. A Markov kernel in-context map Γ : P ( E ) × E → E ′ \Gamma: \mathcal{P}(E) \times E \to E' Γ : P ( E ) × E → E ′ with H > 0 H > 0 H > 0 heads is of the form
r h ( μ , z ) = ∫ E v h ( w ) K h [ μ , z ] ( d w ) , h = 1 , … , H (9) r_h(\mu,z) = \int_E v_h(w)K_h[\mu,z](\mathrm{d}w), \qquad h = 1,\ldots,H \tag*{(9)} r h ( μ , z ) = ∫ E v h ( w ) K h [ μ , z ] ( d w ) , h = 1 , … , H (9) Γ ( μ , z ) = m ( z , r 1 ( μ , z ) , … , r H ( μ , z ) ) . (10) \Gamma(\mu,z) = m\left(z,r_1(\mu,z),\ldots,r_H(\mu,z)\right). \tag*{(10)} Γ ( μ , z ) = m ( z , r 1 ( μ , z ) , … , r H ( μ , z ) ) . (10) For h = 1 , … , H h = 1,\ldots,H h = 1 , … , H , the maps K h : P ( E ) × E → P ( E ) K_h : \mathcal{P}(E) \times E \to\mathcal{P}(E) K h : P ( E ) × E → P ( E ) are Markov kernels, the bounded maps v h : E → R d h v_h : E \to\mathbb{R}^{d_h} v h : E → R d h are called value maps , and m : E × ∏ h = 1 H R d h → E ′ m : E \times\prod_{h=1}^{H}\mathbb{R}^{d_h} \to E' m : E × ∏ h = 1 H R d h → E ′ is called a channel mixer . Each r h : P ( E ) × E → R d h r_h : \mathcal{P}(E) \times E \to\mathbb{R}^{d_h} r h : P ( E ) × E → R d h is called a read head . Additionally, we require the output of the Markov kernels to be supported on the input context μ \mu μ :
supp K h [ μ , z ] ⊂ supp μ for all μ ∈ P ( E ) , z ∈ E , h = 1 , … , H . (11) \operatorname{supp} K_h[\mu,z] \subset\operatorname{supp}\mu\qquad\text{for all } \mu\in\mathcal{P}(E),\ z \in E,\ h = 1,\ldots,H. \tag*{(11)} supp K h [ μ , z ] ⊂ supp μ for all μ ∈ P ( E ) , z ∈ E , h = 1 , … , H . (11)
Remark. The support condition (10) says that we may only access information provided in the current context, which is derived from the sampled points on the graph of the input function f f f . We have no access to information about f f f outside the sampled points. The read heads mix information spatially across the embedding space E E E ; conversely, the channel-mixers are pointwise in E E E .
We use Markov kernel in-context maps to define neural network layers. The input to the layer is a current context that was output from the previous layer, and was ultimately derived from the sampled points of the input function. The layer updates the context by applying the in-context map to each point; that is, the context is pushed-forward by the in-context map.
Definition 3.2 (Markov kernel layer). A Markov kernel layer B : P ( E ) → P ( E ′ ) B : \mathcal{P}(E) \to\mathcal{P}(E') B : P ( E ) → P ( E ′ ) determined by the Markov kernel in-context map Γ : P ( E ) × E → E ′ \Gamma: \mathcal{P}(E) \times E \to E' Γ : P ( E ) × E → E ′ is defined by
B Γ ( μ ) = Γ ( μ , ⋅ ) # μ . (12) B_\Gamma(\mu) = \Gamma(\mu,\cdot)_\#\mu. \tag*{(12)} B Γ ( μ ) = Γ ( μ , ⋅ ) # μ . (12)
Appendix § 6 .§ 2 contains an explicit construction of a softmax attention layer from a transformer using our notation to provide a link to a familiar setting.
We can now define the full Markov kernel network abstract architecture. A Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(\mathcal{G}) \times\mathcal{X} \to Y N : P ( G ) × X → Y encodes the input context into a lifted embedding space, updates the context using Markov kernel layers, then decodes the final context at a query point to obtain the output.
Definition 3.3 (Markov kernel network). Let L > 0 L > 0 L > 0 , and E ℓ E_\ell E ℓ be a compact metric space for ℓ = 0 , … , L \ell= 0,\ldots,L ℓ = 0 , … , L . A Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(\mathcal{G}) \times\mathcal{X} \to Y N : P ( G ) × X → Y is of the form
μ 0 : = q G # μ , \mu_0 := q_{G\#}\mu, μ 0 := q G # μ , μ L : = ( B L − 1 ∘ ⋯ ∘ B 0 ) ( μ 0 ) , μ ∈ P ( G ) , x ∈ X (13) \mu_L := (B_{L-1} \circ\cdots\circ B_0)(\mu_0), \qquad\mu\in\mathcal{P}(\mathcal{G}),\ x \in\mathcal{X} \tag*{(13)} μ L := ( B L − 1 ∘ ⋯ ∘ B 0 ) ( μ 0 ) , μ ∈ P ( G ) , x ∈ X (13) N ( μ , x ) : = Γ p ( μ L , q X ( x ) ) , \mathcal{N}(\mu,x) := \Gamma_p(\mu_L,q_X(x)), N ( μ , x ) := Γ p ( μ L , q X ( x )) , where
q G : G → E 0 q_G : G \to E_0 q G : G → E 0 and q X : X → E L q_X : X \to E_L q X : X → E L are pointwise encoders that lift inputs to hidden embedding spaces. We note that q X = q X ( x ) q_X = q_X(x) q X = q X ( x ) is independent of the context μ \mu μ .
B ℓ : P ( E ℓ ) → P ( E ℓ + 1 ) B_\ell: \mathcal{P}(E_\ell) \to\mathcal{P}(E_{\ell+1}) B ℓ : P ( E ℓ ) → P ( E ℓ + 1 ) are Markov kernel layers for ℓ = 0 , … , L − 1 \ell= 0, \ldots, L - 1 ℓ = 0 , … , L − 1 .
Γ p : P ( E L ) × E L → Y \Gamma_p : \mathcal{P}(E_L) \times E_L \to Y Γ p : P ( E L ) × E L → Y is a decoder in the form of a Markov kernel in-context map.
Different concrete architectures are obtained by specifying the Markov kernels K K K , value maps v v v , and channel mixers m m m (all of these for both the hidden context layers B ℓ B_\ell B ℓ and the decoder Γ p \Gamma_p Γ p ), as well as the encoders q G q_G q G and q X q_X q X . We give the example for a transformer below; additional examples with full constructions (including, for example, assorted normalization operations) can be found in Appendix D.
Example 3.4 (Transformers). Consider the simple absolute positional encoding z : = ( x , a ) ∈ R d z := (x, a) \in\mathbb{R}^{d} z := ( x , a ) ∈ R d , d = d x + d a d = d_x + d_a d = d x + d a , where x ∈ R d x x \in\mathbb{R}^{d_x} x ∈ R d x and a ∈ R d a a \in\mathbb{R}^{d_a} a ∈ R d a is the current hidden features. We preserve spatial coordinates x x x in each token. Suppressing the layer index, each head h = 1 , … , H h = 1, \ldots, H h = 1 , … , H uses a softmax attention kernel
K h [ μ , z ] ( d w ) = exp s h ( z , w ) ∫ E exp s h ( z , w ~ ) μ ( d w ~ ) μ ( d w ) , s h ( z , w ) : = ⟨ Q h z , K k e y , h w ⟩ k h . (14) K_h[\mu,z](\mathrm{d}w) = \frac{\exp s_h(z,w)}{\int_E \exp s_h(z,\widetilde{w}) \,\mu(\mathrm{d}\widetilde{w})}\,\mu(\mathrm{d}w), \qquad s_h(z,w) := \frac{\langle Q_h z,K_{\mathrm{key},h}w\rangle}{\sqrt{k_h}}. \tag*{(14)} K h [ μ , z ] ( d w ) = ∫ E exp s h ( z , w ) μ ( d w ) exp s h ( z , w ) μ ( d w ) , s h ( z , w ) := k h ⟨ Q h z , K key , h w ⟩ . (14)
Here, Q h , K k e y , h ∈ R k h × d Q_h,K_{\mathrm{key},h} \in\mathbb{R}^{k_h \times d} Q h , K key , h ∈ R k h × d , V h ∈ R d a × d V_h \in\mathbb{R}^{d_a \times d} V h ∈ R d a × d are query, key, and value matrices, respectively, and k h k_h k h is the query/key dimension. Combining these K h K_h K h with linear value maps v h ( w ) : = V h w v_h(w) := V_h w v h ( w ) := V h w yields each read head r h r_h r h according to equation (8). The channel mixers include skip connections and feed-forward MLPs F : R d → R d a F : \mathbb{R}^{d} \to\mathbb{R}^{d_a} F : R d → R d a :
m ( ( x , a ) , r 1 , … , r H ) = ( x , F ( x , a + ∑ h = 1 H r h ) ) . (15) m((x,a),r_1,\ldots,r_H) = \left(x,F\left(x,a+\sum_{h=1}^{H}r_h\right)\right). \tag*{(15)} m (( x , a ) , r 1 , … , r H ) = ( x , F ( x , a + h = 1 ∑ H r h ) ) . (15)
This construction is a simplified function graph transformer layer [12 ] ; other transformer variants for operator learning can be constructed in a similar manner.
Popular architectures can largely be classified by how information is routed through the network by their Markov kernels. Transformers are distinguished by their specific form of scaled dot-product attention. Nonlinear neural operators can be realized by multiple heads and a signed-measure decomposition of a nonlinear integral kernel k ( μ , z , w ) μ ( d w ) k(\mu,z,w)\,\mu(\mathrm{d}w) k ( μ , z , w ) μ ( d w ) ; Fourier neural operators Li et al. [22 ] can be realized by the simple choice K [ μ , z ] = μ K[\mu,z] = \mu K [ μ , z ] = μ with spectral computations occurring in v v v and m m m . Flowers, on the other hand, uses Markov kernels that output a Dirac measure. Furthermore, branch-and-trunk style networks, such as DeepONet (Chen & Chen, [5 ] ; Lu et al., 2019), also fit into the Markov kernel network abstract architecture. The distinguishing features of these networks are the forms of their decoders Γ p \Gamma_p Γ p .
A representation theorem for uniformly discretization-stable operators
We now define stable Markov kernel networks as the subset of Definition 3.3 whose individual components all satisfy certain stability properties, and show this guarantees the entire network is uniformly discretization stable. We then state our main result that every operator that is uniformly discretization stable on a class of input measures satisfying Definition 2.2 is exactly representable as a stable Markov kernel network.
Definition 3.5 (Stable Markov kernel network). A Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(G) \times X \to Y N : P ( G ) × X → Y is stable if the following hold:
The encoders q G q_G q G , q X q_X q X , value maps v h v_h v h , and channel mixers m m m are Lipschitz.
In every hidden layer B ℓ B_\ell B ℓ , every Markov kernel K h : P ( E ℓ ) × E ℓ → P ( E ℓ ) K_h : \mathcal{P}(E_\ell) \times E_\ell\to\mathcal{P}(E_\ell) K h : P ( E ℓ ) × E ℓ → P ( E ℓ ) is Hölder stable in both arguments: there exists C > 0 C > 0 C > 0 and β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] such that
W 1 ( K h [ μ , w ] , K h [ ν , z ] ) ≤ C ( W 1 ( μ , ν ) + ρ E ℓ ( w , z ) ) β for all μ , ν ∈ P ( E ℓ ) , w , z ∈ E ℓ . (16) W_1\left(K_h[\mu,w],K_h[\nu,z]\right) \le C\left(W_1(\mu,\nu)+\rho_{E_\ell}(w,z)\right)^\beta\text{ for all } \mu,\nu\in\mathcal{P}(E_\ell),\ w,z \in E_\ell. \tag*{(16)} W 1 ( K h [ μ , w ] , K h [ ν , z ] ) ≤ C ( W 1 ( μ , ν ) + ρ E ℓ ( w , z ) ) β for all μ , ν ∈ P ( E ℓ ) , w , z ∈ E ℓ . (16)
In the decoder Γ p \Gamma_p Γ p , every Markov kernel K h : P ( E L ) × E L → P ( E L ) K_h : \mathcal{P}(E_L) \times E_L \to\mathcal{P}(E_L) K h : P ( E L ) × E L → P ( E L ) is Hölder stable in the context: there exists C > 0 C > 0 C > 0 and β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] such that
W 1 ( K h [ μ , z ] , K h [ ν , z ] ) ≤ C W 1 ( μ , ν ) β for all μ , ν ∈ P ( E L ) , z ∈ E L . (17) W_1(K_h[\mu,z],K_h[\nu,z]) \leq C W_1(\mu,\nu)^\beta\qquad\text{for all } \mu,\nu\in\mathcal{P}(E_L),\ z \in E_L. \tag*{(17)} W 1 ( K h [ μ , z ] , K h [ ν , z ]) ≤ C W 1 ( μ , ν ) β for all μ , ν ∈ P ( E L ) , z ∈ E L . (17)
Remark. In particular, the decoder Γ p \Gamma_p Γ p is not required to be continuous with respect to its second argument z = q X ( x ) z = q_X(x) z = q X ( x ) .
Theorem 3.6. Every stable Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(G) \times X \to Y N : P ( G ) × X → Y is uniformly discretization stable on S = P ( G ) S = \mathcal{P}(G) S = P ( G ) .
Our main result gives a constructive extension of operators that are uniformly discretization stable on admissible sets of non-collapsing measures to uniformly discretization stable operators on the entire graph space, with the extension being a stable Markov kernel network.
Theorem 3.7. Let S ⊂ P ( G ) S \subset\mathcal{P}(G) S ⊂ P ( G ) be an admissible set of non-collapsing measures, and N ∗ : S × X → Y \mathcal{N}^* : S \times X \to Y N ∗ : S × X → Y be uniformly discretization stable on S S S . Then there exists a stable Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(G) \times X \to Y N : P ( G ) × X → Y such that N ∣ S × X = N ∗ \mathcal{N}|_{S \times X} = \mathcal{N}^* N ∣ S × X = N ∗ .
The main idea in the proof is to normalize spatial features and calibrate the decoder reads using techniques from de Carvalho et al. [9 ] . The proof is in Appendix B.
Example 3.8 (Counterexample when S = P ( G ) S = \mathcal{P}(G) S = P ( G ) ). We cannot use stable Markov kernel networks to represent all uniformly discretization stable operators without restricting the domain of input measures S S S . Let X = Y = [ 0 , 1 ] X = Y = [0,1] X = Y = [ 0 , 1 ] equipped with the Lebesgue measure λ \lambda λ , G = X × Y G = X \times Y G = X × Y and consider N ∗ : P ( G ) × X → Y \mathcal{N}^* : \mathcal{P}(G) \times X \to Y N ∗ : P ( G ) × X → Y defined by
N ∗ ( μ , x ) = ∫ G x ~ μ ( d x ~ , d y ) . (18) \mathcal{N}^*(\mu,x) = \sqrt{\int_G \widetilde{x}\,\mu(\mathrm{d}\widetilde{x},\mathrm{d}y)}. \tag*{(18)} N ∗ ( μ , x ) = ∫ G x μ ( d x , d y ) . (18)
This operator is uniformly discretization stable on the entire domain S = P ( G ) S = \mathcal{P}(G) S = P ( G ) . It is constant on graph measures, N ∗ ( γ f , x ) = 1 / 2 \mathcal{N}^*(\gamma_f,x) = 1/\sqrt{2} N ∗ ( γ f , x ) = 1/ 2 for all f ∈ L 2 ( X , Y ) f \in L^2(X,Y) f ∈ L 2 ( X , Y ) . However, for Dirac inputs δ ( t , 0 ) \delta_{(t,0)} δ ( t , 0 ) , t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] , the map t ↦ N ∗ ( δ ( t , 0 ) , x ) = t t \mapsto\mathcal{N}^*(\delta_{(t,0)},x) = \sqrt{t} t ↦ N ∗ ( δ ( t , 0 ) , x ) = t is not Lipschitz. On the other hand, every stable Markov kernel network N \mathcal{N} N forces the map t ↦ N ( δ ( t , 0 ) , x ) t \mapsto\mathcal{N}(\delta_{(t,0)},x) t ↦ N ( δ ( t , 0 ) , x ) to be Lipschitz (the support condition (11) forces every read head for Dirac inputs to be r h ( δ z , z ) = v h ( z ) r_h(\delta_z,z) = v_h(z) r h ( δ z , z ) = v h ( z ) , and the stable network becomes a composition of Lipschitz maps).
Example 3.8 illustrates the importance of the non-collapse condition (8) on the domain S S S in Theorem 3.7. We can match the behavior of N ∗ \mathcal{N}^* N ∗ with a stable Markov kernel network on any reasonable discretized input measure, even very noisy approximations of graph measures, but not its behavior when the spatial marginal collapses. However, the theorem shows that when we restrict the domain of N ∗ \mathcal{N}^* N ∗ to an admissible class S S S , we can construct a stable Markov network that matches the behavior of N ∗ \mathcal{N}^* N ∗ exactly on S S S , and the resulting network is uniformly discretization stable on all of P ( G ) \mathcal{P}(G) P ( G ) .
We close this section by characterizing the operators represented by stable Markov kernel networks.
Corollary 3.9 (Operator characterization of stable Markov kernel networks). Let A : L 2 ( X , Y ) → L 2 ( X , Y ) A : L^2(X,Y) \to L^2(X,Y) A : L 2 ( X , Y ) → L 2 ( X , Y ) be a possibly non-linear operator mapping functions to functions. Then there exists C > 0 C > 0 C > 0 , β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] such that
∥ A [ f ] − A [ g ] ∥ L ∞ ( X , R d o u t ) ≤ C W 1 ( γ f , γ g ) β for all f , g ∈ L 2 ( X , Y ) (19) \|A[f] - A[g]\|_{L^\infty(X,\mathbb{R}^{d_{\mathrm{out}}})} \leq C W_1(\gamma_f,\gamma_g)^\beta\qquad\text{for all } f,g \in L^2(X,Y) \tag*{(19)} ∥ A [ f ] − A [ g ] ∥ L ∞ ( X , R d out ) ≤ C W 1 ( γ f , γ g ) β for all f , g ∈ L 2 ( X , Y ) (19)
if and only if there exists a stable Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(G) \times X \to Y N : P ( G ) × X → Y such that
A [ f ] ( x ) = N ( γ f , x ) for all f ∈ L 2 ( X , Y ) and λ -a.e. x ∈ X . (20) A[f](x) = \mathcal{N}(\gamma_f,x) \qquad\text{for all } f \in L^2(X,Y) \text{ and } \lambda\text{-a.e. } x \in X. \tag*{(20)} A [ f ] ( x ) = N ( γ f , x ) for all f ∈ L 2 ( X , Y ) and λ -a.e. x ∈ X . (20)
Variational principles for Markov kernel networks
Theorems 3.6 and 3.7 define the design space of uniformly discretization stable neural networks to be the stable Markov kernel networks, and Definition 3.5 gives stability properties of the individual components K h K_h K h , v h v_h v h , m m m , q G q_G q G , q X q_X q X in such networks. Exploring new architectures in the design space becomes a question of specifying parameterizations or fixed implementations of these components.
We propose one approach for designing the Markov kernels K h K_h K h in (9) based on variational principles. We fix q G q_G q G , q X q_X q X , and every v h v_h v h and m m m . The proof of Theorem 3.7 shows this can be done in a manner that does not diminish the expressive power of the network itself.
Our approach is inspired by the variational characterization of softmax from Niculae & Blondel [27 ] . For key, query matrices K k e y , Q K_{\mathrm{key}}, Q K key , Q and temperature τ > 0 \tau> 0 τ > 0 , the minimizer of
arg min κ ∈ P ( E ) { − ∫ E ⟨ Q z , K k e y w ⟩ κ ( d w ) + τ KL ( κ ∥ μ ) } \underset{\kappa\in\mathcal{P}(\mathrm{E})}{\operatorname{arg\,min}}\left\{-\int_{\mathrm{E}} \langle Qz,K_{\mathrm{key}}w\rangle\kappa(\mathrm{d}w)+\tau\operatorname{KL}(\kappa\|\mu)\right\} κ ∈ P ( E ) arg min { − ∫ E ⟨ Q z , K key w ⟩ κ ( d w ) + τ KL ( κ ∥ μ ) }
is the classical softmax attention kernel
K [ μ , z ] ( d w ) = exp ( ⟨ Q z , K k e y w ⟩ / τ ) ∫ E exp ( ⟨ Q z , K k e y w ′ ⟩ / τ ) μ ( d w ′ ) μ ( d w ) . K[\mu,z](\mathrm{d}w)=\frac{\exp(\langle Qz,K_{\mathrm{key}}w\rangle/\tau)}{\int_{\mathrm{E}}\exp(\langle Qz,K_{\mathrm{key}}w'\rangle/\tau)\,\mu(\mathrm{d}w')}\,\mu(\mathrm{d}w). K [ μ , z ] ( d w ) = ∫ E exp (⟨ Q z , K key w ′ ⟩ / τ ) μ ( d w ′ ) exp (⟨ Q z , K key w ⟩ / τ ) μ ( d w ) .
We consider a general cost c : P ( E ) × E × E → R c:\mathcal{P}(\mathrm{E})\times\mathrm{E}\times\mathrm{E}\to\mathbb{R} c : P ( E ) × E × E → R and a regularizer R : P ( E ) × P ( E ) → [ 0 , + ∞ ] \mathcal{R}:\mathcal{P}(\mathrm{E})\times\mathcal{P}(\mathrm{E})\to[0,+\infty] R : P ( E ) × P ( E ) → [ 0 , + ∞ ] . Set
K [ μ , z ] ∈ arg min κ ∈ P ( E ) supp κ ⊆ supp μ J μ , z ( κ ) , J μ , z ( κ ) : = ∫ E c ( μ , z , w ) κ ( d w ) + R ( κ , μ ) . (21) K[\mu,z]\in\underset{\substack{\kappa\in\mathcal{P}(\mathrm{E})\\ \operatorname{supp}\kappa\subseteq\operatorname{supp}\mu}}{\operatorname{arg\,min}}\;J_{\mu,z}(\kappa),\qquad J_{\mu,z}(\kappa):=\int_{\mathrm{E}}c(\mu,z,w)\kappa(\mathrm{d}w)+\mathcal{R}(\kappa,\mu). \tag*{(21)} K [ μ , z ] ∈ κ ∈ P ( E ) supp κ ⊆ supp μ arg min J μ , z ( κ ) , J μ , z ( κ ) := ∫ E c ( μ , z , w ) κ ( d w ) + R ( κ , μ ) . (21)
Here, c ( μ , z , w ) c(\mu,z,w) c ( μ , z , w ) is the cost of reading from location w w w based on the input context μ \mu μ and query z z z . The regularizer R \mathcal{R} R penalizes large deviations from μ \mu μ . The support condition restricts reads to the context μ \mu μ , so when a minimizer K K K exists, it provides legal read behavior for a Markov kernel network consistent with Definition 3.1 .
From this point of view, the design space is ( c , R ) (c,\mathcal{R}) ( c , R ) -space. We ask: what are the conditions on costs and regularizers that guarantee the associated kernel is sufficiently stable? We can answer this question for a large class of regularizers constructed from f f f -divergences and Wasserstein distances.
Regularizers from f f f -divergences
An f f f -divergence D φ : P ( E ) × P ( E ) → [ 0 , + ∞ ] D_\varphi:\mathcal{P}(\mathrm{E})\times\mathcal{P}(\mathrm{E})\to[0,+\infty] D φ : P ( E ) × P ( E ) → [ 0 , + ∞ ] with generator φ : [ 0 , + ∞ ) → R ∪ { + ∞ } \varphi:[0,+\infty)\to\mathbb{R}\cup\{+\infty\} φ : [ 0 , + ∞ ) → R ∪ { + ∞ } is of the form
D φ ( κ ∥ μ ) = { ∫ E φ ( d κ d μ ) d μ , κ ≪ μ , ∞ , κ ≪̸ μ . (22) D_\varphi(\kappa\|\mu)=
\begin{cases}
\int_{\mathrm{E}}\varphi\left(\frac{\mathrm{d}\kappa}{\mathrm{d}\mu}\right)\mathrm{d}\mu, & \kappa\ll\mu,\\
\infty, & \kappa\not\ll\mu.
\end{cases} \tag*{(22)} D φ ( κ ∥ μ ) = { ∫ E φ ( d μ d κ ) d μ , ∞ , κ ≪ μ , κ ≪ μ . (22)
where φ \varphi φ is convex and φ ( 1 ) = 0 \varphi(1)=0 φ ( 1 ) = 0 . We further define φ ( 0 ) = lim t → 0 φ ( t ) \varphi(0)=\lim_{t\to0}\varphi(t) φ ( 0 ) = lim t → 0 φ ( t ) , which could be infinite. This class covers many common divergences, including KL, Hellinger, Jensen-Shannon, and χ 2 \chi^2 χ 2 . Not all f f f -divergence regularizers lead to Markov kernels with our required stability properties; the next proposition gives sufficient conditions on the f f f -divergence regularizer and cost function for this to happen.
Proposition 4.1 (Kernel stability for f f f -divergence regularizers). Let c : P ( E ) × E × E → R c:\mathcal{P}(\mathrm{E})\times\mathrm{E}\times\mathrm{E}\to\mathbb{R} c : P ( E ) × E × E → R satisfy
∣ c ( μ , z , w ) ∣ ≤ C , |c(\mu,z,w)|\le C, ∣ c ( μ , z , w ) ∣ ≤ C , ∣ c ( μ , z , w ) − c ( μ ′ , z ′ , w ′ ) ∣ ≤ L ρ E ( w , w ′ ) + L ( ρ E ( z , z ′ ) + W 1 ( μ , μ ′ ) ) β (23) |c(\mu,z,w)-c(\mu',z',w')|\le L\rho_{\mathrm{E}}(w,w')+L\left(\rho_{\mathrm{E}}(z,z')+W_1(\mu,\mu')\right)^\beta \tag*{(23)} ∣ c ( μ , z , w ) − c ( μ ′ , z ′ , w ′ ) ∣ ≤ L ρ E ( w , w ′ ) + L ( ρ E ( z , z ′ ) + W 1 ( μ , μ ′ ) ) β (23) for some C , L > 0 C,L>0 C , L > 0 , β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] . Let D φ D_\varphi D φ have generator φ : [ 0 , + ∞ ) → [ 0 , + ∞ ] \varphi:[0,+\infty)\to[0,+\infty] φ : [ 0 , + ∞ ) → [ 0 , + ∞ ] that is lower semicontinuous, strictly convex, and finite and C 1 C^1 C 1 on ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) , with φ ( 1 ) = φ ′ ( 1 ) = 0 \varphi(1)=\varphi'(1)=0 φ ( 1 ) = φ ′ ( 1 ) = 0 and lim r → + ∞ φ ′ ( r ) = + ∞ \lim_{r\to+\infty}\varphi'(r)=+\infty lim r → + ∞ φ ′ ( r ) = + ∞ . Further suppose the function
h ( t ) = { 0 , t ≤ φ ′ ( 0 + ) , ( φ ′ ) − 1 ( t ) , t > φ ′ ( 0 + ) (24) h(t)=
\begin{cases}
0, & t\le\varphi'(0+),\\
(\varphi')^{-1}(t), & t>\varphi'(0+)
\end{cases} \tag*{(24)} h ( t ) = { 0 , ( φ ′ ) − 1 ( t ) , t ≤ φ ′ ( 0 + ) , t > φ ′ ( 0 + ) (24) is Hölder stable on [ − 2 C , 2 C ] [-2C,2C] [ − 2 C , 2 C ] . Then (20) with cost function c c c and regularizer R = D φ \mathcal{R}=D_\varphi R = D φ has a unique minimizer K K K satisfying the stability condition (15). Moreover, the kernel takes the form
K [ μ , z ] ( d w ) = h ( b μ , z − c ( μ , z , w ) ) μ ( d w ) , ∫ E h ( b μ , z − c ( μ , z , w ) ) μ ( d w ) = 1 , (25) K[\mu,z](\mathrm{d}w)=h(b_{\mu,z}-c(\mu,z,w))\mu(\mathrm{d}w),\qquad\int_{\mathrm{E}}h(b_{\mu,z}-c(\mu,z,w))\mu(\mathrm{d}w)=1, \tag*{(25)} K [ μ , z ] ( d w ) = h ( b μ , z − c ( μ , z , w )) μ ( d w ) , ∫ E h ( b μ , z − c ( μ , z , w )) μ ( d w ) = 1 , (25) where b μ , z ∈ [ − C , C ] b_{\mu,z}\in[-C,C] b μ , z ∈ [ − C , C ] is the unique normalizing constant.
A proof is in Appendix § 7 .§ 2 . Table 1 gives a non-exhaustive list of discretization stable Markov kernels generated in this way. We further note that the Pearson χ 2 \chi^2 χ 2 regularizer corresponds to the construction used in the proof of Theorem 3.7 .
Table 1. Examples of f f f -divergences consistent with the hypotheses of Proposition 4.1 . W 0 W_0 W 0 denotes the principal real Lambert W W W function. The associated stable read kernel K K K is given by Equation (25) . A dash in the last column indicates we are unaware of previous investigations of these attention mechanisms.
Regularizers from projected Wasserstein distances
Regularizers constructed from f f f -divergences cover a range of Markov kernel read behavior, but they do not recover the SelfWarp behavior from Flowers. For this, we turn to regularizers built from projected Wasserstein distances. Let π X : E → X \pi_X:E\to X π X : E → X be fixed, continuous and surjective. Define
R ( κ , μ ) : = 1 2 W 2 ( π X # κ , π X # μ ) 2 . (26) \mathcal{R}(\kappa,\mu):=\frac{1}{2}W_2(\pi_{X\#}\kappa,\pi_{X\#}\mu)^2. \tag*{(26)} R ( κ , μ ) := 2 1 W 2 ( π X # κ , π X # μ ) 2 . (26)
The function π X \pi_X π X should be interpreted as a projection. The regularizer above penalizes reads in an optimal transport distance on the spatial domain X X X .
Proposition 4.2. Let R \mathcal{R} R be given by equation (26) . If ( μ , z , w ) ↦ c ( μ , z , w ) (\mu,z,w)\mapsto c(\mu,z,w) ( μ , z , w ) ↦ c ( μ , z , w ) is continuous in w w w , then there exists a Markov kernel K K K such that K [ μ , z ] K[\mu,z] K [ μ , z ] satisfies (20) for all μ \mu μ and z z z .
To recover SelfWarp on continuous graph contexts γ f \gamma_f γ f , let τ h ( x , a ) = x + g h ( a ) ∈ X \tau_h(x,a)=x+g_h(a)\in X τ h ( x , a ) = x + g h ( a ) ∈ X , with periodic wrapping when appropriate. For B > diam ( X ) B>\operatorname{diam}(X) B > diam ( X ) , choose
c h ( μ , z , w ) = B ρ X ( π X ( w ) , τ h ( z ) ) . c_h(\mu,z,w)=B\rho_X\left(\pi_X(w),\tau_h(z)\right). c h ( μ , z , w ) = B ρ X ( π X ( w ) , τ h ( z ) ) .
The unique minimizing kernel is K h [ γ f , z ] = δ ( τ h ( z ) , f ( τ h ( z ) ) ) K_h[\gamma_f,z]=\delta_{(\tau_h(z),f(\tau_h(z)))} K h [ γ f , z ] = δ ( τ h ( z ) , f ( τ h ( z ))) . With value map v h ( y , b ) = V h b v_h(y,b)=V_hb v h ( y , b ) = V h b , this gives the SelfWarp read V h f ( x + g h ( f ( x ) ) ) V_hf(x+g_h(f(x))) V h f ( x + g h ( f ( x ))) at z = ( x , f ( x ) ) z=(x,f(x)) z = ( x , f ( x )) . The precise statement and proof are in Appendix B.3.
Remark. Our final question is: can we construct a single parameterized family of regularizers such that the read behaviors of FNO, transformers, and Flowers are recovered in different parameter regimes? We show this is possible in Appendix E, which constructs a convex space of regularizers as a linear combination of spatial Wasserstein and f f f -divergences such that the resulting Markov kernel reads are discretization stable. In future work, we aim to explore this space using agents with the goal of adaptively choosing architectures suited to a particular operator learning problem at hand.
Limitations
The representation theorem presented here does not specify how to choose subclasses of Markov kernel networks suitable for a particular task, nor does it determine the most efficient implementations. For example, although it is possible to implement the GroupNorm operation as part of a Markov kernel layer (see Equation (81) in Section D.5), it is another question to choose an efficient implementation in code. The practical importance of depth is not reflected in the proof of the representation theorem, which requires only two layers. Additionally, the kernels K h K_h K h in the representation might be complicated.
Acknowledgments
I.D. was partially supported by the European Research Council Consolidator Grant 101232533 (PhaseShift). The research of M.L. was partially supported by European Research Council Advanced Grant 101097198 (PDE-Inverse) and the Research Council of Finland, grant 359182 (FAME Flagship). MVdH gratefully acknowledges the support of the Department of Energy BES, under grant DE-SC0020345, Oxy and the Simons Foundation under the MATH + X Program. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the other funding organizations.
AI Use Statement
The main portion of this document was written by humans; there is no LLM-generated text appearing on any page preceding this statement, except for light line-level edits. The appendices interleave text generated by ChatGPT 6 Astra and humans; all LLM-generated text was reviewed and edited by the authors for clarity and correctness. We also used ChatGPT and Claude for non-writing tasks, such as literature searches and exploring mathematical ideas.
Notation and background
Table 2.
A Borel probability measure μ \mu μ on a metric space M is Radon if
μ ( A ) = sup { μ ( K ) : K ⊆ A is compact } \mu(A) = \sup\{\mu(K) : K \subseteq A \text{ is compact}\} μ ( A ) = sup { μ ( K ) : K ⊆ A is compact }
for every Borel set A A A . Every Borel probability measure on a compact metric space is Radon.
For a Borel map F : M → M ′ F : \mathrm{M} \to\mathrm{M}' F : M → M ′ , the pushforward of μ \mu μ is
F # μ ( B ) = μ ( F − 1 ( B ) ) , B ⊆ M ′ Borel . (27) F_{\#}\mu(B) = \mu(F^{-1}(B)), \qquad B \subseteq\mathrm{M}'\text{ Borel}. \tag*{(27)} F # μ ( B ) = μ ( F − 1 ( B )) , B ⊆ M ′ Borel . (27)
Equivalently, ∫ ϕ d ( F # μ ) = ∫ ϕ ∘ F d μ \int\phi\,\mathrm{d}(F_{\#}\mu) = \int\phi\circ F \,\mathrm{d}\mu ∫ ϕ d ( F # μ ) = ∫ ϕ ∘ F d μ for every bounded Borel function ϕ \phi ϕ .
Definition A.1 (Wasserstein distances). For 1 ≤ p < ∞ 1 \le p < \infty 1 ≤ p < ∞ , let P p ( M ) \mathcal{P}_p(\mathrm{M}) P p ( M ) denote the Radon probability measures satisfying ∫ ρ M ( z , z 0 ) p μ ( d z ) < ∞ \int\rho_{\mathrm{M}}(z,z_0)^p \,\mu(\mathrm{d}z) < \infty ∫ ρ M ( z , z 0 ) p μ ( d z ) < ∞ for one, and hence every, z 0 ∈ M z_0 \in\mathrm{M} z 0 ∈ M . For μ , ν ∈ P p ( M ) \mu,\nu\in\mathcal{P}_p(\mathrm{M}) μ , ν ∈ P p ( M ) , define
W p ( μ , ν ) : = ( inf π ∈ Π ( μ , ν ) ∫ M × M ρ M ( z , w ) p π ( d z , d w ) ) 1 / p , (28) W_p(\mu,\nu) := \left(\inf_{\pi\in\Pi(\mu,\nu)} \int_{\mathrm{M}\times\mathrm{M}} \rho_{\mathrm{M}}(z,w)^p \,\pi(\mathrm{d}z,\mathrm{d}w)\right)^{1/p}, \tag*{(28)} W p ( μ , ν ) := ( π ∈ Π ( μ , ν ) inf ∫ M × M ρ M ( z , w ) p π ( d z , d w ) ) 1/ p , (28) where Π ( μ , ν ) \Pi(\mu,\nu) Π ( μ , ν ) is the set of probability measures on M × M \mathrm{M} \times\mathrm{M} M × M with marginals μ \mu μ and ν \nu ν .
On a compact metric space, P p ( M ) = P ( M ) \mathcal{P}_p(\mathrm{M}) = \mathcal{P}(\mathrm{M}) P p ( M ) = P ( M ) and W p W_p W p metrizes weak convergence for every finite p p p (Villani, 2009, Chapter 6). This applies to G and the embedding spaces E ℓ \mathrm{E}_{\ell} E ℓ throughout the paper.
We regard L 2 ( X , Y ) L^2(\mathrm{X},\mathrm{Y}) L 2 ( X , Y ) as equivalence classes modulo equality λ \lambda λ -almost everywhere. Borel representatives are used when pointwise values are needed. The graph measure is independent of this choice, and γ f = γ g \gamma_f = \gamma_g γ f = γ g if and only if f = g f = g f = g λ \lambda λ -almost everywhere. An L ∞ L^\infty L ∞ norm is an essential supremum; the supremum in (DS) is over every x ∈ X x \in\mathrm{X} x ∈ X .
Sequences and empirical measures
A sequence z 1 , … , z N z_1,\ldots,z_N z 1 , … , z N determines the empirical measure μ = N − 1 ∑ i δ z i \mu= N^{-1}\sum_i \delta_{z_i} μ = N − 1 ∑ i δ z i . For fixed N N N , this identifies sequences up to permutation. For two equally weighted empirical measures on a metric space,
W 1 ( 1 N ∑ i δ z i , 1 N ∑ i δ w i ) = min σ ∈ S N 1 N ∑ i ρ M ( z i , w σ ( i ) ) . W_1\left(\frac{1}{N}\sum_i\delta_{z_i},\frac{1}{N}\sum_i\delta_{w_i}\right)=\min_{\sigma\in S_N}\frac{1}{N}\sum_i\rho_{\mathrm{M}}(z_i,w_{\sigma(i)}). W 1 ( N 1 i ∑ δ z i , N 1 i ∑ δ w i ) = σ ∈ S N min N 1 i ∑ ρ M ( z i , w σ ( i ) ) .
The same measure can arise from different sequence lengths by repeating every token. A map defined on measures must give the same result under such repetitions. The attention formula below has this property.
Softmax attention as a context pushforward
Consider tokens z 1 , … , z N z_1,\ldots,z_N z 1 , … , z N in a compact subset E \mathrm{E} E of a Euclidean space. For query, key, value, and output matrices Q Q Q , K k e y K_{\mathrm{key}} K key , V V V , W W W , put
s ( z , w ) = ⟨ Q z , K k e y w ⟩ k , K [ μ , z ] ( d w ) = e s ( z , w ) ∫ E e s ( z , u ) μ ( d u ) μ ( d w ) , s(z,w)=\frac{\langle Qz,K_{\mathrm{key}}w\rangle}{\sqrt{k}},\qquad K[\mu,z](\mathrm{d}w)=\frac{e^{s(z,w)}}{\int_{\mathrm{E}}e^{s(z,u)}\,\mu(\mathrm{d}u)}\mu(\mathrm{d}w), s ( z , w ) = k ⟨ Q z , K key w ⟩ , K [ μ , z ] ( d w ) = ∫ E e s ( z , u ) μ ( d u ) e s ( z , w ) μ ( d w ) ,
where k k k is the query/key dimension. The value map is v ( w ) = V w v(w)=Vw v ( w ) = V w , and the mixer m ( z , r ) = z + W r m(z,r)=z+Wr m ( z , r ) = z + W r includes a residual connection. For μ = N − 1 ∑ j δ z j \mu=N^{-1}\sum_j\delta_{z_j} μ = N − 1 ∑ j δ z j , this gives
Γ ( μ , z i ) = z i + W ∑ j = 1 N e s ( z i , z j ) ∑ q = 1 N e s ( z i , z q ) V z j , B Γ ( μ ) = 1 N ∑ i = 1 N δ Γ ( μ , z i ) . (29) \Gamma(\mu,z_i)=z_i+W\sum_{j=1}^{N}\frac{e^{s(z_i,z_j)}}{\sum_{q=1}^{N}e^{s(z_i,z_q)}}Vz_j,\qquad B_{\Gamma}(\mu)=\frac{1}{N}\sum_{i=1}^{N}\delta_{\Gamma(\mu,z_i)}. \tag*{(29)} Γ ( μ , z i ) = z i + W j = 1 ∑ N ∑ q = 1 N e s ( z i , z q ) e s ( z i , z j ) V z j , B Γ ( μ ) = N 1 i = 1 ∑ N δ Γ ( μ , z i ) . (29)
This is the usual single-head softmax attention update, written as B Γ ( μ ) = Γ ( μ , ⋅ ) # μ B_{\Gamma}(\mu)=\Gamma(\mu,\cdot)_{\#}\mu B Γ ( μ ) = Γ ( μ , ⋅ ) # μ . For a weighted context ∑ j ω j δ z j \sum_j\omega_j\delta_{z_j} ∑ j ω j δ z j , the attention weights are proportional to ω j e s ( z i , z j ) \omega_j e^{s(z_i,z_j)} ω j e s ( z i , z j ) . Section D.2 gives the full block, including multiple heads, normalization, and the feed-forward map.
Proofs
Section 3
Theorem 3.6. Every stable Markov kernel network N : P ( G ) × X → Y \mathcal{N} : \mathcal{P}(\mathrm{G}) \times\mathcal{X} \to\mathcal{Y} N : P ( G ) × X → Y is uniformly discretization stable on S = P ( G ) \mathcal{S} = \mathcal{P}(\mathrm{G}) S = P ( G ) .
Proof. We first derive stability of the in-context maps from the component assumptions in Definition 3.5. Consider a hidden layer or the decoder, with input space E E E , output space E ′ E' E ′ , and H H H heads. Write A h > 0 A_h > 0 A h > 0 and β h ∈ ( 0 , 1 ] \beta_h \in(0,1] β h ∈ ( 0 , 1 ] for its kernel stability constants, and let V h > 0 V_h > 0 V h > 0 and M > 0 M > 0 M > 0 be Lipschitz constants for its value maps and mixer, using the sum product metric for the mixer. Set
β = min 1 ≤ h ≤ H β h , D = max { 1 , 2 diam ( E ) } . \beta= \min_{1 \le h \le H} \beta_h, \qquad D = \max\{1, 2\operatorname{diam}(E)\}. β = 1 ≤ h ≤ H min β h , D = max { 1 , 2 diam ( E )} .
Since every distance in the kernel bounds is at most D D D , the inequality t β h ≤ D β h − β t β t^{\beta_h} \le D^{\beta_h-\beta}t^\beta t β h ≤ D β h − β t β allows us to use the common exponent β \beta β after enlarging the constants A h A_h A h .
For probability measures α , α ′ ∈ P ( E ) \alpha,\alpha' \in\mathcal{P}(E) α , α ′ ∈ P ( E ) and any coupling π \pi π of them, Lipschitz continuity of v h v_h v h gives
∥ ∫ v h d α − ∫ v h d α ′ ∥ ≤ ∫ ∥ v h ( u ) − v h ( v ) ∥ π ( d u , d v ) ≤ V h ∫ ρ E ( u , v ) π ( d u , d v ) . \left\lVert\int v_h\,\mathrm{d}\alpha- \int v_h\,\mathrm{d}\alpha' \right\rVert\le\int\lVert v_h(u)-v_h(v)\rVert\,\pi(\mathrm{d}u,\mathrm{d}v) \le V_h \int\rho_E(u,v)\,\pi(\mathrm{d}u,\mathrm{d}v). ∫ v h d α − ∫ v h d α ′ ≤ ∫ ∥ v h ( u ) − v h ( v )∥ π ( d u , d v ) ≤ V h ∫ ρ E ( u , v ) π ( d u , d v ) .
Taking the infimum over couplings, each read therefore satisfies
∥ r h ( μ , w ) − r h ( ν , z ) ∥ ≤ V h W 1 ( K h [ μ , w ] , K h [ ν , z ] ) . \lVert r_h(\mu,w)-r_h(\nu,z)\rVert\le V_h W_1\left(K_h[\mu,w],K_h[\nu,z]\right). ∥ r h ( μ , w ) − r h ( ν , z )∥ ≤ V h W 1 ( K h [ μ , w ] , K h [ ν , z ] ) .
For a hidden layer, put t = W 1 ( μ , ν ) + ρ E ( w , z ) t = W_1(\mu,\nu)+\rho_E(w,z) t = W 1 ( μ , ν ) + ρ E ( w , z ) . Joint kernel stability and the Lipschitz mixer imply
ρ E ′ ( Γ ( μ , w ) , Γ ( ν , z ) ) ≤ M ( ρ E ( w , z ) + ∑ h = 1 H V h A h t β ) ≤ M ( D 1 − β + ∑ h = 1 H V h A h ) t β . \begin{aligned}
\rho_{E'}\left(\Gamma(\mu,w),\Gamma(\nu,z)\right) \le M\left(\rho_E(w,z)+\sum_{h=1}^{H}V_hA_ht^\beta\right) \\
&\le M\left(D^{1-\beta}+\sum_{h=1}^{H}V_hA_h\right)t^\beta.
\end{aligned} ρ E ′ ( Γ ( μ , w ) , Γ ( ν , z ) ) ≤ M ( ρ E ( w , z ) + h = 1 ∑ H V h A h t β ) ≤ M ( D 1 − β + h = 1 ∑ H V h A h ) t β .
Here t ≤ D t \le D t ≤ D and β ≤ 1 \beta\le1 β ≤ 1 imply t ≤ D 1 − β t β t \le D^{1-\beta}t^\beta t ≤ D 1 − β t β . For the decoder, the same argument at a fixed query gives
ρ E ′ ( Γ ( μ , z ) , Γ ( ν , z ) ) ≤ M ( ∑ h = 1 H V h A h ) W 1 ( μ , ν ) β , \rho_{E'}\left(\Gamma(\mu,z),\Gamma(\nu,z)\right) \le M\left(\sum_{h=1}^{H}V_hA_h\right)W_1(\mu,\nu)^\beta, ρ E ′ ( Γ ( μ , z ) , Γ ( ν , z ) ) ≤ M ( h = 1 ∑ H V h A h ) W 1 ( μ , ν ) β ,
uniformly in z z z . Applying these estimates to each layer yields joint stability constants C ℓ > 0 C_\ell> 0 C ℓ > 0 , β ℓ ∈ ( 0 , 1 ] \beta_\ell\in(0,1] β ℓ ∈ ( 0 , 1 ] for Γ ℓ \Gamma_\ell Γ ℓ and context-only stability constants C p > 0 C_p > 0 C p > 0 , β p ∈ ( 0 , 1 ] \beta_p \in(0,1] β p ∈ ( 0 , 1 ] for Γ p \Gamma_p Γ p .
We next show that each context update B ℓ B_\ell B ℓ is Hölder continuous in W 1 W_1 W 1 . Fix η , ξ ∈ P ( E ℓ ) \eta,\xi\in\mathcal{P}(E_\ell) η , ξ ∈ P ( E ℓ ) , write d = W 1 ( η , ξ ) d = W_1(\eta,\xi) d = W 1 ( η , ξ ) , and choose an optimal coupling π \pi π of η \eta η and ξ \xi ξ . Such a coupling exists because E ℓ E_\ell E ℓ is compact. The pushforward of π \pi π under
( w , z ) ⟼ ( Γ ℓ ( η , w ) , Γ ℓ ( ξ , z ) ) (w,z) \longmapsto\left(\Gamma_\ell(\eta,w),\Gamma_\ell(\xi,z)\right) ( w , z ) ⟼ ( Γ ℓ ( η , w ) , Γ ℓ ( ξ , z ) )
is a coupling of B ℓ ( η ) B_\ell(\eta) B ℓ ( η ) and B ℓ ( ξ ) B_\ell(\xi) B ℓ ( ξ ) . Consequently, joint stability gives
W 1 ( B ℓ ( η ) , B ℓ ( ξ ) ) ≤ C ℓ ( ∫ ( d + ρ E ℓ ( w , z ) ) β ℓ π ( d w , d z ) ) ≤ C ℓ ( d + ∫ ρ E ℓ ( w , z ) π ( d w , d z ) ) β ℓ = 2 β ℓ C ℓ d β ℓ . \begin{aligned}
W_1(B_\ell(\eta),B_\ell(\xi)) \le C_\ell\left(\int\left(d+\rho_{E_\ell}(w,z)\right)^{\beta_\ell}\,\pi(\mathrm{d}w,\mathrm{d}z)\right) \\
&\le C_\ell\left(d+\int\rho_{E_\ell}(w,z)\,\pi(\mathrm{d}w,\mathrm{d}z)\right)^{\beta_\ell} \\
&= 2^{\beta_\ell}C_\ell d^{\beta_\ell}.
\end{aligned} W 1 ( B ℓ ( η ) , B ℓ ( ξ )) ≤ C ℓ ( ∫ ( d + ρ E ℓ ( w , z ) ) β ℓ π ( d w , d z ) ) ≤ C ℓ ( d + ∫ ρ E ℓ ( w , z ) π ( d w , d z ) ) β ℓ = 2 β ℓ C ℓ d β ℓ .
The second inequality is Jensen’s inequality for the concave function t ↦ t β ℓ t \mapsto t^{\beta_\ell} t ↦ t β ℓ , and the last equality uses optimality of π \pi π .
Now fix arbitrary inputs μ , ν ∈ P ( G ) \mu,\nu\in\mathcal{P}(\mathrm{G}) μ , ν ∈ P ( G ) , and denote their successive encoded contexts by μ ℓ , ν ℓ \mu_\ell,\nu_\ell μ ℓ , ν ℓ . If Q > 0 Q > 0 Q > 0 is a Lipschitz constant for q G q_{\mathrm{G}} q G , then
W 1 ( μ 0 , ν 0 ) = W 1 ( q G # μ , q G # ν ) ≤ Q W 1 ( μ , ν ) . W_1(\mu_0,\nu_0) = W_1(q_{\mathrm{G}\#}\mu,q_{\mathrm{G}\#}\nu) \le QW_1(\mu,\nu). W 1 ( μ 0 , ν 0 ) = W 1 ( q G # μ , q G # ν ) ≤ Q W 1 ( μ , ν ) .
Iterating the context-update estimate therefore yields a constant A > 0 A > 0 A > 0 , depending only on the network, such that
W 1 ( μ L , ν L ) ≤ A W 1 ( μ , ν ) θ , θ = ∏ ℓ = 0 L − 1 β ℓ . W_1(\mu_L,\nu_L) \le A W_1(\mu,\nu)^\theta,\qquad\theta= \prod_{\ell=0}^{L-1}\beta_\ell. W 1 ( μ L , ν L ) ≤ A W 1 ( μ , ν ) θ , θ = ℓ = 0 ∏ L − 1 β ℓ .
At each output location x x x , both evaluations use the same query q X ( x ) q_X(x) q X ( x ) . The decoder’s stability bound is uniform over these queries, so
sup x ∈ X ∣ N ( μ , x ) − N ( ν , x ) ∣ = sup x ∈ X ∣ Γ p ( μ L , q X ( x ) ) − Γ p ( ν L , q X ( x ) ) ∣ ≤ C p W 1 ( μ L , ν L ) β p ≤ C p A β p W 1 ( μ , ν ) β p θ . \begin{aligned}
\sup_{x\in X} \left|\mathcal{N}(\mu,x)-\mathcal{N}(\nu,x)\right|
= \sup_{x\in X} \left|\Gamma_p(\mu_L,q_X(x))-\Gamma_p(\nu_L,q_X(x))\right| \\
&\le C_p W_1(\mu_L,\nu_L)^{\beta_p} \\
&\le C_p A^{\beta_p} W_1(\mu,\nu)^{\beta_p\theta}.
\end{aligned} x ∈ X sup ∣ N ( μ , x ) − N ( ν , x ) ∣ = x ∈ X sup ∣ Γ p ( μ L , q X ( x )) − Γ p ( ν L , q X ( x )) ∣ ≤ C p W 1 ( μ L , ν L ) β p ≤ C p A β p W 1 ( μ , ν ) β p θ .
Since β p θ ∈ ( 0 , 1 ] \beta_p\theta\in(0,1] β p θ ∈ ( 0 , 1 ] and μ , ν \mu,\nu μ , ν were arbitrary, this is uniform discretization stability on P ( G ) \mathcal{P}(\mathrm{G}) P ( G ) .
Theorem 3.7. Let S ⊂ P ( G ) \mathcal{S} \subset\mathcal{P}(\mathrm{G}) S ⊂ P ( G ) be an admissible set of non-collapsing measures, and N ∗ : S × X → Y \mathcal{N}^*:\mathcal{S}\times X \to Y N ∗ : S × X → Y be uniformly discretization stable on S \mathcal{S} S . Then there exists a stable Markov kernel network N : P ( G ) × X → Y \mathcal{N}:\mathcal{P}(\mathrm{G})\times X \to Y N : P ( G ) × X → Y such that N ∣ S × X = N ∗ \left.\mathcal{N}\right|_{\mathcal{S}\times X}=\mathcal{N}^* N ∣ S × X = N ∗ .
Proof. We use one context layer to calibrate finitely many spatial values so that they have zero mean and unit total variance on S \mathcal{S} S . The decoder then recovers the target through weighted averages, followed by a fixed linear readout and projection onto Y Y Y .
Step 1: Extend the target in the context variable. Let C > 0 C > 0 C > 0 and β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] be the discretization-stability constants of N ∗ \mathcal{N}^* N ∗ , write c = d o u t c=d_{\mathrm{out}} c = d out , and choose M ≥ 1 M \ge1 M ≥ 1 such that Y ⊂ [ − M , M ] c Y \subset[-M,M]^c Y ⊂ [ − M , M ] c . Since P ( G ) \mathcal{P}(\mathrm{G}) P ( G ) is a compact metric space, S \mathcal{S} S has a countable dense sequence ( μ ( i ) ) i ≥ 1 (\mu^{(i)})_{i\ge1} ( μ ( i ) ) i ≥ 1 . For j = 1 , … , c j=1,\ldots,c j = 1 , … , c , define
U j ( μ , x ) = clip [ − M , M ] ( inf i ≥ 1 { N j ∗ ( μ ( i ) , x ) + C W 1 ( μ , μ ( i ) ) β } ) , ( μ , x ) ∈ P ( G ) × X . U_j(\mu,x)=\operatorname{clip}_{[-M,M]}\left(\inf_{i\ge1}\left\{\mathcal{N}^*_j(\mu^{(i)},x)+C W_1(\mu,\mu^{(i)})^\beta\right\}\right),\qquad(\mu,x)\in\mathcal{P}(\mathrm{G})\times X. U j ( μ , x ) = clip [ − M , M ] ( i ≥ 1 inf { N j ∗ ( μ ( i ) , x ) + C W 1 ( μ , μ ( i ) ) β } ) , ( μ , x ) ∈ P ( G ) × X .
where clip [ − M , M ] ( t ) = max { − M , min { t , M } } \operatorname{clip}_{[-M,M]}(t)=\max\{-M,\min\{t,M\}\} clip [ − M , M ] ( t ) = max { − M , min { t , M }} . This is Borel measurable because the infimum is countable. The triangle inequality for the metric W 1 β W_1^\beta W 1 β , followed by the fact that clipping is 1-Lipschitz, gives
∣ U j ( μ , x ) − U j ( ν , x ) ∣ ≤ C W 1 ( μ , ν ) β . \left|U_j(\mu,x)-U_j(\nu,x)\right|\le C W_1(\mu,\nu)^\beta. ∣ U j ( μ , x ) − U j ( ν , x ) ∣ ≤ C W 1 ( μ , ν ) β .
If μ ∈ S \mu\in\mathcal{S} μ ∈ S , discretization stability bounds every term in the infimum from below by N j ∗ ( μ , x ) \mathcal{N}^*_j(\mu,x) N j ∗ ( μ , x ) , while density provides a sequence of terms converging to that value. Thus U = N ∗ U=\mathcal{N}^* U = N ∗ on S × X \mathcal{S}\times X S × X . Globally, ∣ U ∣ ≤ c M |U|\le\sqrt{c}M ∣ U ∣ ≤ c M and U U U is Hölder continuous in the context, uniformly in x x x , with constant C U = c C C_U=\sqrt{c}C C U = c C .
Step 2: Find finitely many spatial values with uniformly positive total variance. The set T = S ‾ T=\overline{\mathcal{S}} T = S is compact and remains non-collapsing: indeed,
μ ⟼ inf a ∈ X W 1 ( ( π X ) # μ , δ a ) \mu\longmapsto\inf_{a\in X}W_1((\pi_X)_\#\mu,\delta_a) μ ⟼ a ∈ X inf W 1 (( π X ) # μ , δ a )
is 1-Lipschitz. Consequently, every μ ∈ T \mu\in T μ ∈ T has a non-Dirac spatial marginal. For each such measure, some Lipschitz function h : X → [ 0 , 1 ] h:X\to[0,1] h : X → [ 0 , 1 ] has positive variance; one may take a rescaled distance from a point in the marginal’s support. That variance remains positive in a neighborhood of μ \mu μ . A finite subcover of T T T therefore provides Lipschitz functions h 1 , … , h n : X → [ 0 , 1 ] h_1,\ldots,h_n:X\to[0,1] h 1 , … , h n : X → [ 0 , 1 ] whose variances have a strictly positive sum throughout T T T .
Write ϕ k = h k ∘ π X \phi_k=h_k\circ\pi_X ϕ k = h k ∘ π X and, for every μ ∈ P ( G ) \mu\in\mathcal{P}(\mathrm{G}) μ ∈ P ( G ) , set
a k ( μ ) = ∫ G ϕ k d μ , b k ( μ ) = ∫ G ϕ k 2 d μ − a k ( μ ) 2 , V ( μ ) = ∑ k = 1 n b k ( μ ) . a_k(\mu)=\int_{\mathrm{G}}\phi_k\,\mathrm{d}\mu,\qquad
b_k(\mu)=\int_{\mathrm{G}}\phi_k^2\,\mathrm{d}\mu-a_k(\mu)^2,\qquad
V(\mu)=\sum_{k=1}^{n}b_k(\mu). a k ( μ ) = ∫ G ϕ k d μ , b k ( μ ) = ∫ G ϕ k 2 d μ − a k ( μ ) 2 , V ( μ ) = k = 1 ∑ n b k ( μ ) .
These functions are globally Lipschitz in W 1 W_1 W 1 , since the ϕ k \phi_k ϕ k are bounded and Lipschitz. By compactness there is v ∗ > 0 v_*>0 v ∗ > 0 such that V ( μ ) ≥ v ∗ V(\mu)\ge v_* V ( μ ) ≥ v ∗ on T T T . Define
D ( μ ) = max { V ( μ ) , v ∗ } . D(\mu)=\max\{V(\mu),v_*\}. D ( μ ) = max { V ( μ ) , v ∗ } .
Step 3: Calibrate the values in a context layer. Take L = 1 L=1 L = 1 and set
R = v ∗ − 1 / 2 , E 0 = G , E 1 = G × [ − R , R ] n . (30) R=v_*^{-1/2},\qquad E_0=\mathrm{G},\qquad E_1=\mathrm{G}\times[-R,R]^n. \tag*{(30)} R = v ∗ − 1/2 , E 0 = G , E 1 = G × [ − R , R ] n . (30)
Equip E 1 E_1 E 1 with the sum of the graph metric and the Euclidean metric on the new coordinates. Choose y ∗ ∈ Y y_* \in Y y ∗ ∈ Y and use the 1-Lipschitz encoders
q G = id G , q X ( x ) = ( ( x , y ∗ ) , 0 ) . q_G=\operatorname{id}_G,\qquad q_X(x)=((x,y_*),0). q G = id G , q X ( x ) = (( x , y ∗ ) , 0 ) .
The context layer retains each original token and appends its calibrated spatial values:
Γ 0 ( μ , g ) = ( g , ( ϕ k ( g ) − a k ( μ ) D ( μ ) ) k = 1 n ) . \Gamma_0(\mu,g)=\left(g,\left(\frac{\phi_k(g)-a_k(\mu)}{\sqrt{D(\mu)}}\right)_{k=1}^{n}\right). Γ 0 ( μ , g ) = ( g , ( D ( μ ) ϕ k ( g ) − a k ( μ ) ) k = 1 n ) .
This is a Markov kernel in-context map with one head: use the uniform kernel K ^ [ μ , g ] = μ \widehat{K}[\mu,g]=\mu K [ μ , g ] = μ and the value map
v ^ ( w ) = ( ϕ k ( w ) , ϕ k ( w ) 2 ) k = 1 n . \widehat{v}(w)=\left(\phi_k(w),\phi_k(w)^2\right)_{k=1}^{n}. v ( w ) = ( ϕ k ( w ) , ϕ k ( w ) 2 ) k = 1 n .
Its read supplies all first and second moments needed to compute a k a_k a k and D D D , so the mixer can apply the displayed formula using only the token and this read. To define the mixer on arbitrary read vectors, first clip every moment coordinate to [ 0 , 1 ] [0,1] [ 0 , 1 ] . Actual reads are unchanged, and the resulting feature coordinates always lie in [ − R , R ] [-R,R] [ − R , R ] .
The uniform kernel K ^ \widehat{K} K is jointly 1-Lipschitz, and v ^ \widehat{v} v is Lipschitz because the ϕ k \phi_k ϕ k are bounded and Lipschitz. The mixer is globally Lipschitz as well: after clipping, its moment coordinates lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , squaring is Lipschitz on this interval, and t ↦ max { t , v ∗ } − 1 / 2 t\mapsto\max\{t,v_*\}^{-1/2} t ↦ max { t , v ∗ } − 1/2 is bounded and Lipschitz. Its feature numerators are bounded and Lipschitz in the token and clipped moments. Thus the hidden layer satisfies the component assumptions of Definition 3.5 on its full domain.
Write π : E 1 → G \pi:E_1\to G π : E 1 → G for projection onto the retained token and ψ k : E 1 → [ − R , R ] \psi_k:E_1\to[-R,R] ψ k : E 1 → [ − R , R ] for the k k k th new coordinate. For any η ∈ P ( E 1 ) \eta\in\mathcal{P}(E_1) η ∈ P ( E 1 ) , define
a ˉ k ( η ) = ∫ E 1 ψ k d η , T ( η ) = ∑ k = 1 n ( ∫ E 1 ψ k 2 d η − a ˉ k ( η ) 2 ) . \bar{a}_k(\eta)=\int_{E_1}\psi_k\,\mathrm{d}\eta,\qquad T(\eta)=\sum_{k=1}^{n}\left(\int_{E_1}\psi_k^2\,\mathrm{d}\eta-\bar{a}_k(\eta)^2\right). a ˉ k ( η ) = ∫ E 1 ψ k d η , T ( η ) = k = 1 ∑ n ( ∫ E 1 ψ k 2 d η − a ˉ k ( η ) 2 ) .
For the updated context η = B 0 ( μ ) = Γ 0 ( μ , ⋅ ) # μ \eta=B_0(\mu)=\Gamma_0(\mu,\cdot)_{\#}\mu η = B 0 ( μ ) = Γ 0 ( μ , ⋅ ) # μ , direct integration gives
π # η = μ , a ˉ k ( η ) = 0 , T ( η ) = V ( μ ) D ( μ ) . \pi_{\#}\eta=\mu,\qquad\bar{a}_k(\eta)=0,\qquad T(\eta)=\frac{V(\mu)}{D(\mu)}. π # η = μ , a ˉ k ( η ) = 0 , T ( η ) = D ( μ ) V ( μ ) .
Thus the new coordinates have zero mean for every input, and their total variance is one whenever μ ∈ S \mu\in\mathcal{S} μ ∈ S . This completes the calibration.
Step 4: Decode with weighted averages and a fixed readout. For arbitrary η ∈ P ( E 1 ) \eta\in\mathcal{P}(E_1) η ∈ P ( E 1 ) and z ∈ E 1 z\in E_1 z ∈ E 1 , set
θ j ( η , z ) = U j ( π # η , π X ( π z ) ) 4 M R , j = 1 , … , c . \theta_j(\eta,z)=\frac{U_j(\pi_{\#}\eta,\pi_X(\pi z))}{4MR},\qquad j=1,\ldots,c. θ j ( η , z ) = 4 M R U j ( π # η , π X ( π z )) , j = 1 , … , c .
Use c n cn c n decoder heads, indexed by ( j , k ) (j,k) ( j , k ) , with
K j k [ η , z ] ( d w ) = [ 1 + θ j ( η , z ) ( ψ k ( w ) − a ˉ k ( η ) ) ] η ( d w ) , v j k ( w ) = ψ k ( w ) . K_{jk}[\eta,z](\mathrm{d}w)=\left[1+\theta_j(\eta,z)\left(\psi_k(w)-\bar{a}_k(\eta)\right)\right]\eta(\mathrm{d}w),\qquad v_{jk}(w)=\psi_k(w). K j k [ η , z ] ( d w ) = [ 1 + θ j ( η , z ) ( ψ k ( w ) − a ˉ k ( η ) ) ] η ( d w ) , v j k ( w ) = ψ k ( w ) .
Since ∣ θ j ∣ ≤ 1 / ( 4 R ) |\theta_j|\leq1/(4R) ∣ θ j ∣ ≤ 1/ ( 4 R ) and ∣ ψ k − a ˉ k ∣ ≤ 2 R |\psi_k-\bar{a}_k|\leq2R ∣ ψ k − a ˉ k ∣ ≤ 2 R , the densities lie in [ 1 / 2 , 3 / 2 ] [1/2,3/2] [ 1/2 , 3/2 ] . Subtracting a ˉ k ( η ) \bar{a}_k(\eta) a ˉ k ( η ) makes them integrate to one even for contexts outside the image of B 0 B_0 B 0 . These are Borel probability kernels supported on supp η \operatorname{supp}\eta supp η , and the value maps are bounded and Lipschitz. Their reads satisfy
r j k ( η , z ) = a ˉ k ( η ) + θ j ( η , z ) ( ∫ ψ k 2 d η − a ˉ k ( η ) 2 ) . r_{jk}(\eta,z)=\bar{a}_k(\eta)+\theta_j(\eta,z)\left(\int\psi_k^2\,\mathrm{d}\eta-\bar{a}_k(\eta)^2\right). r j k ( η , z ) = a ˉ k ( η ) + θ j ( η , z ) ( ∫ ψ k 2 d η − a ˉ k ( η ) 2 ) .
Let Π Y : R c → Y \Pi_Y:\mathbb{R}^c\to Y Π Y : R c → Y be nearest-point projection, which is 1-Lipschitz because Y Y Y is non-empty, compact, and convex. The decoder mixer is simply
m p ( z , ( r j k ) j , k ) = Π Y ( 4 M R ( ∑ k = 1 n r j k ) j = 1 c ) . m_p\left(z,(r_{jk})_{j,k}\right)=\Pi_Y\left(4MR\left(\sum_{k=1}^{n}r_{jk}\right)_{j=1}^{c}\right). m p ( z , ( r j k ) j , k ) = Π Y 4 M R ( k = 1 ∑ n r j k ) j = 1 c .
This mixer is globally Lipschitz, being a fixed linear map followed by the 1-Lipschitz projection. There is no reference head or variance normalization in this mixer. Indeed, writing A ( η ) = ∑ k a ˉ k ( η ) A(\eta)=\sum_k\bar{a}_k(\eta) A ( η ) = ∑ k a ˉ k ( η ) and 1 c = ( 1 , … , 1 ) ∈ R c \mathbf{1}_c=(1,\ldots,1)\in\mathbb{R}^c 1 c = ( 1 , … , 1 ) ∈ R c , the read identities give
Γ p ( η , z ) = Π Y ( 4 M R A ( η ) 1 c + T ( η ) U ( π # η , π X ( π z ) ) ) . \Gamma_p(\eta,z)=\Pi_Y\left(4MR A(\eta)\mathbf{1}_c+T(\eta)U(\pi_{\#}\eta,\pi_X(\pi z))\right). Γ p ( η , z ) = Π Y ( 4 M R A ( η ) 1 c + T ( η ) U ( π # η , π X ( π z )) ) .
It remains to verify context-only stability of the decoder kernels themselves. Fix j j j , k k k and z z z , and write
F η , z ( w ) = 1 + θ j ( η , z ) ( ψ k ( w ) − a ˉ k ( η ) ) , d = W 1 ( η , ξ ) , Δ = diam ( E 1 ) . F_{\eta,z}(w)=1+\theta_j(\eta,z)(\psi_k(w)-\bar{a}_k(\eta)),\qquad d=W_1(\eta,\xi),\qquad\Delta=\operatorname{diam}(E_1). F η , z ( w ) = 1 + θ j ( η , z ) ( ψ k ( w ) − a ˉ k ( η )) , d = W 1 ( η , ξ ) , Δ = diam ( E 1 ) .
Since π \pi π and ψ k \psi_k ψ k are 1-Lipschitz, Step 1 and the averaging estimate in the proof of Theorem 3.6 give
∣ θ j ( η , z ) − θ j ( ξ , z ) ∣ ≤ C 4 M d β , ∣ a ˉ k ( η ) − a ˉ k ( ξ ) ∣ ≤ d . |\theta_j(\eta,z)-\theta_j(\xi,z)|\leq\frac{C}{4M}d^\beta,\qquad|\bar{a}_k(\eta)-\bar{a}_k(\xi)|\leq d. ∣ θ j ( η , z ) − θ j ( ξ , z ) ∣ ≤ 4 M C d β , ∣ a ˉ k ( η ) − a ˉ k ( ξ ) ∣ ≤ d .
Together with ∣ θ j ∣ ≤ 1 / ( 4 R ) |\theta_j|\leq1/(4R) ∣ θ j ∣ ≤ 1/ ( 4 R ) and ∣ ψ k − a ˉ k ∣ ≤ 2 R |\psi_k-\bar{a}_k|\leq2R ∣ ψ k − a ˉ k ∣ ≤ 2 R , these imply
∥ F η , z ∥ ∞ ≤ 3 2 , Lip w ( F η , z ) ≤ 1 4 R , ∥ F η , z − F ξ , z ∥ ∞ ≤ C 2 M d β + d 4 R . \|F_{\eta,z}\|_\infty\leq\frac{3}{2},\qquad\operatorname{Lip}_w(F_{\eta,z})\leq\frac{1}{4R},\qquad\|F_{\eta,z}-F_{\xi,z}\|_\infty\leq\frac{C}{2M}d^\beta+\frac{d}{4R}. ∥ F η , z ∥ ∞ ≤ 2 3 , Lip w ( F η , z ) ≤ 4 R 1 , ∥ F η , z − F ξ , z ∥ ∞ ≤ 2 M C d β + 4 R d .
For a 1-Lipschitz test function f f f normalized to vanish at a fixed point of E 1 E_1 E 1 , we have ∥ f ∥ ∞ ≤ Δ \|f\|_\infty\leq\Delta ∥ f ∥ ∞ ≤ Δ and Lip ( f F η , z ) ≤ 3 / 2 + Δ / ( 4 R ) \operatorname{Lip}(fF_{\eta,z})\leq3/2+\Delta/(4R) Lip ( f F η , z ) ≤ 3/2 + Δ/ ( 4 R ) . Therefore
∣ ∫ f d K j k [ η , z ] − ∫ f d K j k [ ξ , z ] ∣ ≤ ∣ ∫ f F η , z d ( η − ξ ) ∣ + ∣ ∫ f ( F η , z − F ξ , z ) d ξ ∣ ≤ ( 3 2 + Δ 2 R ) d + Δ C 2 M d β . \begin{aligned}
\left|\int f\,\mathrm{d}K_{jk}[\eta,z]-\int f\,\mathrm{d}K_{jk}[\xi,z]\right|
\leq\left|\int fF_{\eta,z}\,\mathrm{d}(\eta-\xi)\right|+\left|\int f(F_{\eta,z}-F_{\xi,z})\,\mathrm{d}\xi\right| \\
&\leq\left(\frac{3}{2}+\frac{\Delta}{2R}\right)d+\frac{\Delta C}{2M}d^\beta.
\end{aligned} ∫ f d K j k [ η , z ] − ∫ f d K j k [ ξ , z ] ≤ ∫ f F η , z d ( η − ξ ) + ∫ f ( F η , z − F ξ , z ) d ξ ≤ ( 2 3 + 2 R Δ ) d + 2 M Δ C d β .
Kantorovich–Rubinstein duality, followed by d ≤ Δ d\leq\Delta d ≤ Δ and β ≤ 1 \beta\leq1 β ≤ 1 , now gives
W 1 ( K j k [ η , z ] , K j k [ ξ , z ] ) ≤ C K W 1 ( η , ξ ) β . W_1\bigl(K_{jk}[\eta,z],K_{jk}[\xi,z]\bigr)\leq C_KW_1(\eta,\xi)^\beta. W 1 ( K j k [ η , z ] , K j k [ ξ , z ] ) ≤ C K W 1 ( η , ξ ) β .
for a constant C K C_K C K independent of z z z . This holds for all η , ξ ∈ P ( E 1 ) \eta,\xi\in\mathcal{P}(E_1) η , ξ ∈ P ( E 1 ) , including contexts outside the image of B 0 B_0 B 0 . All components therefore satisfy Definition 3.5, and Theorem 3.6 gives uniform discretization stability of the assembled network on its full domain.
Finally, on every context η = B 0 ( μ ) \eta=B_0(\mu) η = B 0 ( μ ) the term A ( η ) A(\eta) A ( η ) vanishes, so
N ( μ , x ) = Γ p ( B 0 ( μ ) , q X ( x ) ) = Π Y ( V ( μ ) D ( μ ) U ( μ , x ) ) . \mathcal{N}(\mu,x)=\Gamma_p\bigl(B_0(\mu),q_X(x)\bigr)=\Pi_Y\left(\frac{V(\mu)}{D(\mu)}U(\mu,x)\right). N ( μ , x ) = Γ p ( B 0 ( μ ) , q X ( x ) ) = Π Y ( D ( μ ) V ( μ ) U ( μ , x ) ) .
For μ ∈ S \mu\in S μ ∈ S , the ratio V / D V/D V / D is one and U ( μ , x ) = N ∗ ( μ , x ) ∈ Y U(\mu,x)=\mathcal{N}^*(\mu,x)\in Y U ( μ , x ) = N ∗ ( μ , x ) ∈ Y , proving the required equality. Only the decoder kernels use the query dependence of U U U ; no spatial regularity of the target is needed. □
Corollary 3.9 (Operator characterization of stable Markov kernel networks). Let A : L 2 ( X , Y ) → L 2 ( X , Y ) A:L^2(X,Y)\to L^2(X,Y) A : L 2 ( X , Y ) → L 2 ( X , Y ) be a possibly non-linear operator mapping functions to functions. Then there exists C > 0 C>0 C > 0 , β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] such that
∥ A [ f ] − A [ g ] ∥ L ∞ ( X , R d o u t ) ≤ C W 1 ( γ f , γ g ) β for all f , g ∈ L 2 ( X , Y ) (31) \|A[f]-A[g]\|_{L^\infty(X,\mathbb{R}^{d_{\mathrm{out}}})}\leq CW_1(\gamma_f,\gamma_g)^\beta\qquad\text{for all }f,g\in L^2(X,Y) \tag*{(31)} ∥ A [ f ] − A [ g ] ∥ L ∞ ( X , R d out ) ≤ C W 1 ( γ f , γ g ) β for all f , g ∈ L 2 ( X , Y ) (31) if and only if there exists a stable Markov kernel network N : P ( G ) × X → Y \mathcal{N}:\mathcal{P}(\mathbb{G})\times X\to Y N : P ( G ) × X → Y such that
A [ f ] ( x ) = N ( γ f , x ) for all f ∈ L 2 ( X , Y ) and λ -a.e. x ∈ X . (32) A[f](x)=\mathcal{N}(\gamma_f,x)\qquad\text{for all }f\in L^2(X,Y)\text{ and }\lambda\text{-a.e. }x\in X. \tag*{(32)} A [ f ] ( x ) = N ( γ f , x ) for all f ∈ L 2 ( X , Y ) and λ -a.e. x ∈ X . (32)
Proof. Suppose first that the displayed Hölder estimate holds, and set
S ~ = { γ f : f ∈ L 2 ( X , Y ) } . \widetilde{S}=\{\gamma_f:f\in L^2(X,Y)\}. S = { γ f : f ∈ L 2 ( X , Y )} .
This set is non-empty because Y Y Y is non-empty. Every μ ∈ S \mu\in S μ ∈ S has spatial marginal ( π X ) # μ = λ (\pi_X)_\#\mu=\lambda ( π X ) # μ = λ . Since λ \lambda λ has full support and X X X contains at least two points, λ \lambda λ is not a Dirac measure. The function
a ⟼ W 1 ( λ , δ a ) = ∫ X ρ X ( x , a ) λ ( d x ) a\longmapsto W_1(\lambda,\delta_a)=\int_X\rho_X(x,a)\,\lambda(\mathrm{d}x) a ⟼ W 1 ( λ , δ a ) = ∫ X ρ X ( x , a ) λ ( d x )
is continuous and strictly positive on the compact space X X X . Its minimum c λ c_\lambda c λ is therefore positive, and
inf μ ∈ S , a ∈ X W 1 ( ( π X ) # μ , δ a ) = c λ > 0. \inf_{\mu\in S,\ a\in X}W_1\bigl((\pi_X)_\#\mu,\delta_a\bigr)=c_\lambda>0. μ ∈ S , a ∈ X inf W 1 ( ( π X ) # μ , δ a ) = c λ > 0.
Taking ϵ = c λ / 2 \epsilon=c_\lambda/2 ϵ = c λ /2 shows that S S S is an admissible set of non-collapsing measures.
Since Y Y Y is compact, every output of A A A belongs to L ∞ ( X , R d o u t ) L^\infty(X,\mathbb{R}^{d_{\mathrm{out}}}) L ∞ ( X , R d out ) . Define
Q : S → L ∞ ( X , R d o u t ) , Q ( γ f ) = A [ f ] . Q:S\to L^\infty(X,\mathbb{R}^{d_{\mathrm{out}}}),\qquad Q(\gamma_f)=A[f]. Q : S → L ∞ ( X , R d out ) , Q ( γ f ) = A [ f ] .
This is well-defined because equality of graph measures implies equality of the input L 2 L^2 L 2 classes. The assumed estimate states that
∥ Q ( μ ) − Q ( ν ) ∥ L ∞ ≤ C W 1 ( μ , ν ) β ( μ , ν ∈ S ) . \lVert Q(\mu)-Q(\nu)\rVert_{L^\infty} \le C W_1(\mu,\nu)^\beta\qquad(\mu,\nu\in S). ∥ Q ( μ ) − Q ( ν ) ∥ L ∞ ≤ C W 1 ( μ , ν ) β ( μ , ν ∈ S ) .
We next choose representatives of these output classes so that this estimate holds pointwise and the resulting evaluation map is jointly Borel measurable.
As a subspace of the compact metric space ( P ( G ) , W 1 ) (\mathcal{P}(G),W_1) ( P ( G ) , W 1 ) , the set S S S is separable. Choose a dense sequence ( μ i ) i ≥ 1 (\mu_i)_{i\geq1} ( μ i ) i ≥ 1 in S S S and Borel representatives u i : X → Y u_i:X\to Y u i : X → Y of Q ( μ i ) Q(\mu_i) Q ( μ i ) . The set
E = ⋃ i , j ≥ 1 { x ∈ X : ∣ u i ( x ) − u j ( x ) ∣ > C W 1 ( μ i , μ j ) β } E=\bigcup_{i,j\geq1}\left\{x\in X:\lvert u_i(x)-u_j(x)\rvert>CW_1(\mu_i,\mu_j)^\beta\right\} E = i , j ≥ 1 ⋃ { x ∈ X : ∣ u i ( x ) − u j ( x )∣ > C W 1 ( μ i , μ j ) β }
is Borel and λ \lambda λ -null. Fix y ∗ ∈ Y y_*\in Y y ∗ ∈ Y and redefine every u i u_i u i to equal y ∗ y_* y ∗ on E E E . These remain Borel representatives of the same classes and now satisfy
∣ u i ( x ) − u j ( x ) ∣ ≤ C W 1 ( μ i , μ j ) β for all i , j ≥ 1 , x ∈ X . \lvert u_i(x)-u_j(x)\rvert\leq CW_1(\mu_i,\mu_j)^\beta\qquad\text{for all }i,j\geq1,\ x\in X. ∣ u i ( x ) − u j ( x )∣ ≤ C W 1 ( μ i , μ j ) β for all i , j ≥ 1 , x ∈ X .
For k ≥ 1 k\geq1 k ≥ 1 and μ ∈ S \mu\in S μ ∈ S , let
i k ( μ ) = min { i ≥ 1 : W 1 ( μ , μ i ) < 2 − k } . i_k(\mu)=\min\{i\geq1:W_1(\mu,\mu_i)<2^{-k}\}. i k ( μ ) = min { i ≥ 1 : W 1 ( μ , μ i ) < 2 − k } .
Density makes this index well-defined, and it is Borel measurable because each set on which it equals a given index is a relative open ball with finitely many earlier relative open balls removed. Moreover,
sup x ∈ X ∣ u i k ( μ ) ( x ) − u i ℓ ( μ ) ( x ) ∣ ≤ C ( 2 − k + 2 − ℓ ) β . \sup_{x\in X}\lvert u_{i_k(\mu)}(x)-u_{i_\ell(\mu)}(x)\rvert\leq C(2^{-k}+2^{-\ell})^\beta. x ∈ X sup ∣ u i k ( μ ) ( x ) − u i ℓ ( μ ) ( x )∣ ≤ C ( 2 − k + 2 − ℓ ) β .
Thus the limit
N ∗ ( μ , x ) = lim k → ∞ u i k ( μ ) ( x ) \mathcal{N}^*(\mu,x)=\lim_{k\to\infty}u_{i_k(\mu)}(x) N ∗ ( μ , x ) = k → ∞ lim u i k ( μ ) ( x )
exists uniformly in x x x and takes values in the closed set Y Y Y . Each approximating map is jointly Borel measurable, so N ∗ \mathcal{N}^* N ∗ is jointly Borel measurable as well. Passing to the limit in the pairwise pointwise bounds gives
∣ N ∗ ( μ , x ) − N ∗ ( ν , x ) ∣ ≤ C W 1 ( μ , ν ) β ( μ , ν ∈ S , x ∈ X ) . \lvert\mathcal{N}^*(\mu,x)-\mathcal{N}^*(\nu,x)\rvert\leq CW_1(\mu,\nu)^\beta\qquad(\mu,\nu\in S,\ x\in X). ∣ N ∗ ( μ , x ) − N ∗ ( ν , x )∣ ≤ C W 1 ( μ , ν ) β ( μ , ν ∈ S , x ∈ X ) .
Finally, Q ( μ i k ( μ ) ) → Q ( μ ) Q(\mu_{i_k(\mu)})\to Q(\mu) Q ( μ i k ( μ ) ) → Q ( μ ) in L ∞ L^\infty L ∞ , while their chosen representatives converge uniformly to N ∗ ( μ , ⋅ ) \mathcal{N}^*(\mu,\cdot) N ∗ ( μ , ⋅ ) . Hence N ∗ ( μ , ⋅ ) \mathcal{N}^*(\mu,\cdot) N ∗ ( μ , ⋅ ) represents Q ( μ ) Q(\mu) Q ( μ ) .
The map N ∗ \mathcal{N}^* N ∗ is therefore uniformly discretization stable on S S S . Theorem 3.7 provides a stable Markov kernel network N \mathcal{N} N on P ( G ) × X \mathcal{P}(G)\times X P ( G ) × X extending N ∗ \mathcal{N}^* N ∗ . For every f ∈ L 2 ( X , Y ) f\in L^2(X,Y) f ∈ L 2 ( X , Y ) , we consequently have N ( γ f , x ) = A [ f ] ( x ) \mathcal{N}(\gamma_f,x)=A[f](x) N ( γ f , x ) = A [ f ] ( x ) for λ \lambda λ -almost every x x x .
Conversely, suppose such a stable Markov kernel network N \mathcal{N} N exists. By Theorem 3.6, there are constants C > 0 C>0 C > 0 and β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] such that
∥ A [ f ] − A [ g ] ∥ L ∞ = ∥ N ( γ f , ⋅ ) − N ( γ g , ⋅ ) ∥ L ∞ ≤ sup x ∈ X ∣ N ( γ f , x ) − N ( γ g , x ) ∣ ≤ C W 1 ( γ f , γ g ) β \begin{aligned}
\lVert A[f]-A[g]\rVert_{L^\infty}
&=\lVert\mathcal{N}(\gamma_f,\cdot)-\mathcal{N}(\gamma_g,\cdot)\rVert_{L^\infty}\\
&\leq\sup_{x\in X}\lvert\mathcal{N}(\gamma_f,x)-\mathcal{N}(\gamma_g,x)\rvert\\
&\leq CW_1(\gamma_f,\gamma_g)^\beta
\end{aligned} ∥ A [ f ] − A [ g ] ∥ L ∞ = ∥ N ( γ f , ⋅ ) − N ( γ g , ⋅ ) ∥ L ∞ ≤ x ∈ X sup ∣ N ( γ f , x ) − N ( γ g , x )∣ ≤ C W 1 ( γ f , γ g ) β
for all f , g ∈ L 2 ( X , Y ) f,g\in L^2(X,Y) f , g ∈ L 2 ( X , Y ) , as required. □
Section 4.1: stable variational reads
Regularized attention is a familiar instance of (20) [27 , 25 ] . A broader family uses f f f -divergences. We denote the convex generator by φ \varphi φ to distinguish it from an input field. For a superlinear generator, its divergence and the corresponding regularizer are
D φ ( κ ∥ μ ) = { ∫ φ ( p ) d μ , κ = p μ , + ∞ , κ ≪̸ μ , R φ , τ = τ D φ , τ > 0. (33) D_\varphi(\kappa\mathbin{\|}\mu)=
\begin{cases}
\int\varphi(p)\,\mathrm{d}\mu, & \kappa=p\mu,\\
+\infty, & \kappa\not\ll\mu,
\end{cases}
\qquad
R_{\varphi,\tau}=\tau D_\varphi,\quad\tau>0. \tag*{(33)} D φ ( κ ∥ μ ) = { ∫ φ ( p ) d μ , + ∞ , κ = p μ , κ ≪ μ , R φ , τ = τ D φ , τ > 0. (33)
The integral has its extended meaning. The optimizer is obtained by applying the inverse derivative of φ \varphi φ to a shifted negative cost [32 ] . Quantitative regularity of this scalar response gives a stability certificate even when the reference context changes.
Proposition B.1 (Stable divergence-regularized kernels). Let E E E be compact and τ > 0 \tau> 0 τ > 0 . Let φ : [ 0 , ∞ ) → [ 0 , ∞ ] \varphi: [0,\infty) \to[0,\infty] φ : [ 0 , ∞ ) → [ 0 , ∞ ] be lower semicontinuous, strictly convex, finite and C 1 C^1 C 1 on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) , with φ ( 1 ) = φ ′ ( 1 ) = 0 \varphi(1) = \varphi'(1) = 0 φ ( 1 ) = φ ′ ( 1 ) = 0 and lim r → ∞ φ ′ ( r ) = + ∞ \lim_{r\to\infty}\varphi'(r) = +\infty lim r → ∞ φ ′ ( r ) = + ∞ . Define its scalar response by
h ( u ) = { 0 , u ≤ φ ′ ( 0 + ) , ( φ ′ ) − 1 ( u ) , u > φ ′ ( 0 + ) . (34) h(u) =
\begin{cases}
0, & u \le\varphi'(0+),\\
(\varphi')^{-1}(u), & u > \varphi'(0+).
\end{cases} \tag*{(34)} h ( u ) = { 0 , ( φ ′ ) − 1 ( u ) , u ≤ φ ′ ( 0 + ) , u > φ ′ ( 0 + ) . (34) where the first case is absent if φ ′ ( 0 + ) = − ∞ \varphi'(0+) = -\infty φ ′ ( 0 + ) = − ∞ . Suppose the Borel cost c c c satisfies, for some M , L , H ≥ 0 M,L,H \ge0 M , L , H ≥ 0 and β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] ,
∣ c ( μ , z , w ) ∣ ≤ M , (35) |c(\mu,z,w)| \le M, \tag*{(35)} ∣ c ( μ , z , w ) ∣ ≤ M , (35) ∣ c ( μ , z , w ) − c ( ν , z ′ , w ′ ) ∣ ≤ L ρ E ( w , w ′ ) + H ( W 1 ( μ , ν ) + ρ E ( z , z ′ ) ) β . |c(\mu,z,w)-c(\nu,z',w')| \le L\rho_E(w,w') + H\left(W_1(\mu,\nu)+\rho_E(z,z')\right)^\beta. ∣ c ( μ , z , w ) − c ( ν , z ′ , w ′ ) ∣ ≤ L ρ E ( w , w ′ ) + H ( W 1 ( μ , ν ) + ρ E ( z , z ′ ) ) β . Assume that h h h is γ \gamma γ -Hölder on I M = [ − 2 M / τ , 2 M / τ ] I_M = [-2M/\tau,2M/\tau] I M = [ − 2 M / τ , 2 M / τ ] , for some γ ∈ ( 0 , 1 ] \gamma\in(0,1] γ ∈ ( 0 , 1 ] . Then (20) with R = R φ , τ \mathcal{R} = \mathcal{R}_{\varphi,\tau} R = R φ , τ has a unique minimizer, defining a supported Borel Markov kernel
K [ μ , z ] ( d w ) = h ( b μ , z − c ( μ , z , w ) / τ ) μ ( d w ) , (36) K[\mu,z](dw) = h\left(b_{\mu,z} - c(\mu,z,w)/\tau\right)\mu(dw), \tag*{(36)} K [ μ , z ] ( d w ) = h ( b μ , z − c ( μ , z , w ) / τ ) μ ( d w ) , (36) where b μ , z ∈ [ − M / τ , M / τ ] b_{\mu,z} \in[-M/\tau,M/\tau] b μ , z ∈ [ − M / τ , M / τ ] is the unique scalar making the density integrate to one. This kernel satisfies
W 1 ( K [ μ , z ] , K [ ν , z ′ ] ) ≤ C ( W 1 ( μ , ν ) + ρ E ( z , z ′ ) ) β γ . (37) W_1\bigl(K[\mu,z],K[\nu,z']\bigr) \le C\left(W_1(\mu,\nu)+\rho_E(z,z')\right)^{\beta\gamma}. \tag*{(37)} W 1 ( K [ μ , z ] , K [ ν , z ′ ] ) ≤ C ( W 1 ( μ , ν ) + ρ E ( z , z ′ ) ) β γ . (37) Here C C C depends only on the displayed bounds, diam E \operatorname{diam} E diam E , h ( 2 M / τ ) h(2M/\tau) h ( 2 M / τ ) , and the Hölder constant of h h h on I M I_M I M . If (31) is assumed only for z ′ = z z'=z z ′ = z , uniformly in z z z , then the conclusion holds for z ′ = z z'=z z ′ = z with the same uniformity.
The normalization φ ′ ( 1 ) = 0 \varphi'(1)=0 φ ′ ( 1 ) = 0 is harmless: subtracting a multiple of p − 1 p-1 p − 1 does not change the divergence between probability measures. The response condition is a quantitative convexity requirement. For example, a lower bound φ ′ ′ ( p ) ≥ m B > 0 \varphi''(p) \ge m_B > 0 φ ′′ ( p ) ≥ m B > 0 for 0 < p ≤ B 0 < p \le B 0 < p ≤ B , for every finite B B B , suffices for γ = 1 \gamma=1 γ = 1 , without requiring global strong convexity.
Proposition 4.1 is the case τ = 1 \tau=1 τ = 1 , M = C M=C M = C , and H = L H=L H = L of this result. If the scalar response has exponent γ \gamma γ , the resulting kernel has exponent β γ \beta\gamma β γ .
Proof. Write s = ( μ , z ) s=(\mu,z) s = ( μ , z ) , t = ( ν , z ′ ) t=(\nu,z') t = ( ν , z ′ ) , c s ( w ) = c ( μ , z , w ) c_s(w)=c(\mu,z,w) c s ( w ) = c ( μ , z , w ) , and
D = diam E , a = W 1 ( μ , ν ) , δ = a + ρ E ( z , z ′ ) , ε = L a + H δ β . D=\operatorname{diam} E,\qquad a=W_1(\mu,\nu),\qquad\delta=a+\rho_E(z,z'),\qquad\varepsilon=La+H\delta^\beta. D = diam E , a = W 1 ( μ , ν ) , δ = a + ρ E ( z , z ′ ) , ε = L a + H δ β .
Let A A A be a γ \gamma γ -Hölder constant of h h h on I M I_M I M , and set B = h ( 2 M / τ ) B=h(2M/\tau) B = h ( 2 M / τ ) .
Normalization and optimality. The assumptions on φ \varphi φ imply that h h h is continuous and nondecreasing, strictly increasing wherever positive, with h ( 0 ) = 1 h(0)=1 h ( 0 ) = 1 , lim u → − ∞ h ( u ) = 0 \lim_{u\to-\infty}h(u)=0 lim u → − ∞ h ( u ) = 0 , and lim u → ∞ h ( u ) = ∞ \lim_{u\to\infty}h(u)=\infty lim u → ∞ h ( u ) = ∞ . Thus
F s ( b ) : = ∫ h ( b − c s / τ ) d μ F_s(b) := \int h(b-c_s/\tau)\,d\mu F s ( b ) := ∫ h ( b − c s / τ ) d μ
is continuous, goes from zero to infinity, and is strictly increasing wherever positive. It has a unique root F s ( b s ) = 1 F_s(b_s)=1 F s ( b s ) = 1 . Since ∣ c s ∣ ≤ M |c_s| \le M ∣ c s ∣ ≤ M and h ( 0 ) = 1 h(0)=1 h ( 0 ) = 1 ,
− M / τ ≤ b s ≤ M / τ , 0 ≤ p s ( w ) : = h ( b s − c s ( w ) / τ ) ≤ B . -M/\tau\le b_s \le M/\tau,\qquad0 \le p_s(w):=h(b_s-c_s(w)/\tau) \le B. − M / τ ≤ b s ≤ M / τ , 0 ≤ p s ( w ) := h ( b s − c s ( w ) / τ ) ≤ B .
The candidate p s μ p_s\mu p s μ has finite objective. If φ ( 0 ) < ∞ \varphi(0)<\infty φ ( 0 ) < ∞ , the generator is bounded on [ 0 , B ] [0,B] [ 0 , B ] ; if φ ( 0 ) = ∞ \varphi(0)=\infty φ ( 0 ) = ∞ , then φ ′ ( 0 + ) = − ∞ \varphi'(0+)=-\infty φ ′ ( 0 + ) = − ∞ and p s ≥ h ( − 2 M / τ ) > 0 p_s \ge h(-2M/\tau)>0 p s ≥ h ( − 2 M / τ ) > 0 , so it is evaluated on a compact subinterval of ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) .
Extend φ \varphi φ by + ∞ +\infty + ∞ to negative arguments. By the definition of h h h , the scalar b s − c s ( w ) / τ b_s-c_s(w)/\tau b s − c s ( w ) / τ belongs to ∂ φ ( p s ( w ) ) \partial\varphi(p_s(w)) ∂ φ ( p s ( w )) , including when p s ( w ) = 0 p_s(w)=0 p s ( w ) = 0 . For any probability density q q q of finite divergence, the supporting-line inequality gives
φ ( q ) − φ ( p s ) ≥ ( b s − c s / τ ) ( q − p s ) . \varphi(q)-\varphi(p_s) \ge(b_s-c_s/\tau)(q-p_s). φ ( q ) − φ ( p s ) ≥ ( b s − c s / τ ) ( q − p s ) .
Integrating, using ∫ ( q − p s ) d μ = 0 \int(q-p_s)\,d\mu=0 ∫ ( q − p s ) d μ = 0 , and adding the linear cost proves J s ( q μ ) ≥ J s ( p s μ ) J_s(q\mu) \ge J_s(p_s\mu) J s ( q μ ) ≥ J s ( p s μ ) . Strict convexity forces q = p s q=p_s q = p s almost everywhere in the equality case. All other measures have infinite objective, since c s c_s c s is bounded. This proves existence and uniqueness, and absolute continuity gives the support constraint. For fixed b b b , s ↦ F s ( b ) s\mapsto F_s(b) s ↦ F s ( b ) is Borel, and
{ s : b s < b } = { s : F s ( b ) > 1 } . \{s:b_s<b\}=\{s:F_s(b)>1\}. { s : b s < b } = { s : F s ( b ) > 1 } .
Thus the normalizer is Borel, and parameter-dependent integration proves the Markov property.
Stability without estimating the normalizer. Let π \pi π be an optimal coupling of μ , ν \mu,\nu μ , ν . The cost assumptions imply
∫ ∣ c s ( w ) − c t ( v ) ∣ π ( d w , d v ) ≤ ε . \int\lvert c_s(w)-c_t(v)\rvert\,\pi(\mathrm{d}w,\mathrm{d}v)\leq\varepsilon. ∫ ∣ c s ( w ) − c t ( v )∣ π ( d w , d v ) ≤ ε .
On this coupling define q 0 ( v ) = h ( b s − c t ( v ) / τ ) q_0(v)=h(b_s-c_t(v)/\tau) q 0 ( v ) = h ( b s − c t ( v ) / τ ) . Since h h h is nondecreasing, q 0 − p t q_0-p_t q 0 − p t has the same sign everywhere. Both p s p_s p s and p t p_t p t integrate to one against π \pi π , so
∫ ∣ q 0 − p t ∣ d π = ∣ ∫ q 0 d π − 1 ∣ ≤ ∫ ∣ q 0 − p s ∣ d π . \int\lvert q_0-p_t\rvert\,\mathrm{d}\pi
=
\left\lvert\int q_0\,\mathrm{d}\pi-1\right\rvert
\leq
\int\lvert q_0-p_s\rvert\,\mathrm{d}\pi. ∫ ∣ q 0 − p t ∣ d π = ∫ q 0 d π − 1 ≤ ∫ ∣ q 0 − p s ∣ d π .
Every argument of h h h lies in I M I_M I M . The triangle inequality, the response bound, and concavity of r ↦ r γ r\mapsto r^\gamma r ↦ r γ therefore give
∫ ∣ p s ( w ) − p t ( v ) ∣ d π ≤ 2 A τ − γ ∫ ∣ c s ( w ) − c t ( v ) ∣ γ d π ≤ 2 A ( ε / τ ) γ . (38) \begin{aligned}
\int\lvert p_s(w)-p_t(v)\rvert\,\mathrm{d}\pi
&\leq2A\tau^{-\gamma}\int\lvert c_s(w)-c_t(v)\rvert^\gamma\,\mathrm{d}\pi\\
&\leq2A(\varepsilon/\tau)^\gamma. \tag*{(38)}
\end{aligned} ∫ ∣ p s ( w ) − p t ( v )∣ d π ≤ 2 A τ − γ ∫ ∣ c s ( w ) − c t ( v ) ∣ γ d π ≤ 2 A ( ε / τ ) γ . (38)
For any 1-Lipschitz test function v v v , subtract the midpoint of its range so that ∣ v ∣ ≤ D / 2 \lvert v\rvert\leq D/2 ∣ v ∣ ≤ D /2 . This does not change its integral against a difference of probability measures. Since p s ≤ B p_s\leq B p s ≤ B ,
∣ ∫ v p s d μ − ∫ v p t d ν ∣ ≤ B a + D 2 ∫ ∣ p s ( w ) − p t ( v ′ ) ∣ π ( d w , d v ′ ) . \left\lvert\int v p_s\,\mathrm{d}\mu-\int v p_t\,\mathrm{d}\nu\right\rvert
\leq Ba+\frac{D}{2}\int\lvert p_s(w)-p_t(v')\rvert\,\pi(\mathrm{d}w,\mathrm{d}v'). ∫ v p s d μ − ∫ v p t d ν ≤ B a + 2 D ∫ ∣ p s ( w ) − p t ( v ′ )∣ π ( d w , d v ′ ) .
Kantorovich–Rubinstein duality yields the quantitative bound
W 1 ( K s , K t ) ≤ B a + D A τ − γ ( L a + H δ β ) γ . (39) W_1(K_s,K_t)\leq Ba+DA\tau^{-\gamma}(La+H\delta^\beta)^\gamma. \tag*{(39)} W 1 ( K s , K t ) ≤ B a + D A τ − γ ( L a + H δ β ) γ . (39)
If D = 0 D=0 D = 0 , all the measures coincide. Otherwise a ≤ δ ≤ 2 D a\leq\delta\leq2D a ≤ δ ≤ 2 D , so boundedness converts the right-hand side into C δ β γ C\delta^{\beta\gamma} C δ β γ . Taking z ′ = z z'=z z ′ = z throughout proves the fixed-query version under only its stated cost hypotheses.
KL, Pearson, and Tsallis examples
Corollary B.2 (KL and Pearson reads). Under the cost assumptions of Proposition B.1, the regularizers
R K L , τ ( κ , μ ) = τ KL ( κ ∥ μ ) , R P e a r s o n , τ ( κ , μ ) = τ 2 χ 2 ( κ ∥ μ ) (40) R_{\mathrm{KL},\tau}(\kappa,\mu)=\tau\operatorname{KL}(\kappa\Vert\mu),
\qquad
R_{\mathrm{Pearson},\tau}(\kappa,\mu)=\frac{\tau}{2}\chi^2(\kappa\Vert\mu) \tag*{(40)} R KL , τ ( κ , μ ) = τ KL ( κ ∥ μ ) , R Pearson , τ ( κ , μ ) = 2 τ χ 2 ( κ ∥ μ ) (40) yield unique kernels with densities
p μ , z K L ( w ) = e − c ( μ , z , w ) / τ ∫ e − c ( μ , z , v ) / τ μ ( d v ) , p^{\mathrm{KL}}_{\mu,z}(w)=
\frac{e^{-c(\mu,z,w)/\tau}}
{\int e^{-c(\mu,z,v)/\tau}\,\mu(\mathrm{d}v)}, p μ , z KL ( w ) = ∫ e − c ( μ , z , v ) / τ μ ( d v ) e − c ( μ , z , w ) / τ , p μ , z P e a r s o n ( w ) = ( b μ , z P e a r s o n − c ( μ , z , w ) / τ ) + . (41) p^{\mathrm{Pearson}}_{\mu,z}(w)=
\left(b^{\mathrm{Pearson}}_{\mu,z}-c(\mu,z,w)/\tau\right)_+. \tag*{(41)} p μ , z Pearson ( w ) = ( b μ , z Pearson − c ( μ , z , w ) / τ ) + . (41) where b μ , z P e a r s o n b^{\mathrm{Pearson}}_{\mu,z} b μ , z Pearson normalizes the second density. Both satisfy (33) with exponent β \beta β .
Indeed, φ K L ( p ) = p log p − p + 1 \varphi_{\mathrm{KL}}(p)=p\log p-p+1 φ KL ( p ) = p log p − p + 1 gives h ( u ) = e u h(u)=e^u h ( u ) = e u , while φ P e a r s o n ( p ) = ( p − 1 ) 2 / 2 \varphi_{\mathrm{Pearson}}(p)=(p-1)^2/2 φ Pearson ( p ) = ( p − 1 ) 2 /2 gives h ( u ) = ( 1 + u ) + h(u)=(1+u)_+ h ( u ) = ( 1 + u ) + . Both responses are Lipschitz on bounded intervals. These are the Gibbs and positive-part transformations underlying softmax and sparsemax [24 , 25 ] . Here χ 2 ( κ ∥ μ ) = ∫ ( p − 1 ) 2 d μ \chi^2(\kappa\Vert\mu)=\int(p-1)^2\,\mathrm{d}\mu χ 2 ( κ ∥ μ ) = ∫ ( p − 1 ) 2 d μ for κ = p μ \kappa=p\mu κ = p μ , and is infinite otherwise; the factor 1 / 2 1/2 1/2 belongs to the regularizer.
The generators in Corollary B.2 have the claimed scalar responses, both locally Lipschitz. For KL, normalization gives e b s = 1 / ∫ e − c s / τ d μ e^{b_s}=1/\int e^{-c_s/\tau}\,\mathrm{d}\mu e b s = 1/ ∫ e − c s / τ d μ . For Pearson, set b s P e a r s o n = 1 + b s b_s^{\mathrm{Pearson}}=1+b_s b s Pearson = 1 + b s to obtain the stated positive-part density. In the notation of the proof, B K L = e 2 M / τ B_{\mathrm{KL}}=e^{2M/\tau} B KL = e 2 M / τ and A K L = B K L A_{\mathrm{KL}}=B_{\mathrm{KL}} A KL = B KL , while B P e a r s o n = 1 + 2 M / τ B_{\mathrm{Pearson}}=1+2M/\tau B Pearson = 1 + 2 M / τ and A P e a r s o n = 1 A_{\mathrm{Pearson}}=1 A Pearson = 1 . Equation (39) therefore gives
W 1 ( K s K L , K t K L ) ≤ B K L a + D B K L τ ε , (42) W_1(K_s^{\mathrm{KL}},K_t^{\mathrm{KL}})
\leq B_{\mathrm{KL}}a+\frac{DB_{\mathrm{KL}}}{\tau}\varepsilon, \tag*{(42)} W 1 ( K s KL , K t KL ) ≤ B KL a + τ D B KL ε , (42) W 1 ( K s P e a r s o n , K t P e a r s o n ) ≤ B P e a r s o n a + D τ ε . (43) W_1(K_s^{\mathrm{Pearson}},K_t^{\mathrm{Pearson}})
\leq B_{\mathrm{Pearson}}a+\frac{D}{\tau}\varepsilon. \tag*{(43)} W 1 ( K s Pearson , K t Pearson ) ≤ B Pearson a + τ D ε . (43)
No lower bound on the density or on positive empirical atom masses is used.
More generally, the power, or Tsallis, generators and their responses are
φ q ( p ) = p q − q p + q − 1 q ( q − 1 ) , h q ( u ) = [ 1 + ( q − 1 ) u ] + 1 / ( q − 1 ) , q > 1. (44) \varphi_q(p)=\frac{p^q-qp+q-1}{q(q-1)},\qquad h_q(u)=\left[1+(q-1)u\right]_+^{1/(q-1)},\qquad q>1. \tag*{(44)} φ q ( p ) = q ( q − 1 ) p q − q p + q − 1 , h q ( u ) = [ 1 + ( q − 1 ) u ] + 1/ ( q − 1 ) , q > 1. (44)
The density formula is the continuous entmax transformation [25 Proposition 8] . For 1 < q ≤ 2 1<q\le2 1 < q ≤ 2 the response is locally Lipschitz, so the kernel retains exponent β \beta β . For q > 2 q>2 q > 2 , it is 1 / ( q − 1 ) 1/(q-1) 1/ ( q − 1 ) -Hölder, giving exponent β / ( q − 1 ) \beta/(q-1) β / ( q − 1 ) . These are sufficient exponents, not claims of optimality. The limit q → 1 q\to1 q → 1 gives KL, and q = 2 q=2 q = 2 gives our Pearson normalization.
A direct sufficient curvature condition for γ = 1 \gamma=1 γ = 1 is φ ′ ′ ( p ) ≥ m B > 0 \varphi^{\prime\prime}(p)\ge m_B>0 φ ′′ ( p ) ≥ m B > 0 for 0 < p ≤ B 0<p\le B 0 < p ≤ B . The mean value theorem then bounds the inverse derivative by 1 / m B 1/m_B 1/ m B ; the same bound holds across a finite zero threshold by taking a limit from positive densities. If h ( I M ) h(I_M) h ( I M ) is bounded away from zero, the curvature bound is needed only on that smaller density interval. In particular, global strong convexity is unnecessary for KL.
The remaining responses in Table 1 . Differentiating the shifted-entropy and KL-plus-Pearson generators gives, respectively, log ( ( r + a ) / ( 1 + a ) ) \log((r+a)/(1+a)) log (( r + a ) / ( 1 + a )) and log r + a ( r − 1 ) \log r+a(r-1) log r + a ( r − 1 ) . Inverting these derivatives, with truncation at zero when necessary, gives the displayed responses. The centered-power derivative is sgn ( r − 1 ) ∣ r − 1 ∣ p − 1 \operatorname{sgn}(r-1)|r-1|^{p-1} sgn ( r − 1 ) ∣ r − 1 ∣ p − 1 , so its response is locally Hölder with exponent min { 1 , 1 / ( p − 1 ) } \min\{1,1/(p-1)\} min { 1 , 1/ ( p − 1 )} . The derivatives of the quadratic-plus-reverse-KL and Jeffreys generators are ( r − r − 1 ) / 2 (r-r^{-1})/2 ( r − r − 1 ) /2 and log r + 1 − r − 1 \log r+1-r^{-1} log r + 1 − r − 1 , yielding the stated formulas. These last two generators have value + ∞ +\infty + ∞ at zero, as allowed in the proposition. Apart from the two power families, all responses in the table are Lipschitz on bounded intervals: their generators have strictly positive continuous second derivative on the positive density range, and the shifted-entropy response is also Lipschitz across its zero threshold. All listed generators are normalized, strictly convex, and superlinear.
Why arbitrary divergences are not covered. Even smooth strict convexity does not prevent a support discontinuity when the generator has finite asymptotic slope. Consider reverse KL, with φ ( p ) = − log p + p − 1 \varphi(p)=-\log p+p-1 φ ( p ) = − log p + p − 1 , on E = { 0 , 1 } E=\{0,1\} E = { 0 , 1 } , and take
μ e = ( 1 − e ) δ 0 + e δ 1 , c ( w ) = − C w , C > τ . \mu_e=(1-e)\delta_0+e\delta_1,\qquad c(w)=-Cw,\qquad C>\tau. μ e = ( 1 − e ) δ 0 + e δ 1 , c ( w ) = − C w , C > τ .
For 0 < e < 1 0<e<1 0 < e < 1 , a row with mass x x x at 1 1 1 has objective − C x + τ KL ( μ e ∥ κ ) -Cx+\tau\operatorname{KL}(\mu_e\Vert\kappa) − C x + τ KL ( μ e ∥ κ ) . Its unique minimizer satisfies
C x ( 1 − x ) = τ ( x − e ) , x = C − τ + ( C − τ ) 2 + 4 C τ e 2 C ⟶ 1 − τ C > 0. Cx(1-x)=\tau(x-e),\qquad x=\frac{C-\tau+\sqrt{(C-\tau)^2+4C\tau e}}{2C}\longrightarrow1-\frac{\tau}{C}>0. C x ( 1 − x ) = τ ( x − e ) , x = 2 C C − τ + ( C − τ ) 2 + 4 C τ e ⟶ 1 − C τ > 0.
At e = 0 e=0 e = 0 , the support constraint forces the row δ 0 \delta_0 δ 0 . Thus no context-continuous selection exists for this bounded Lipschitz cost. Here φ ′ ( p ) → 1 \varphi'(p)\to1 φ ′ ( p ) → 1 , violating the proposition’s superlinearity assumption.
Nor do superlinearity and strict convexity alone ensure a Hölder rate. For example, choose a strictly increasing continuous derivative with
φ ′ ( 1 + r ) = sgn ( r ) e − 1 / ∣ r ∣ ( 0 < ∣ r ∣ ≤ 1 / 2 ) , φ ′ ( 1 ) = 0 , \varphi'(1+r)=\operatorname{sgn}(r)e^{-1/|r|}\quad(0<|r|\le1/2),\qquad\varphi'(1)=0, φ ′ ( 1 + r ) = sgn ( r ) e − 1/∣ r ∣ ( 0 < ∣ r ∣ ≤ 1/2 ) , φ ′ ( 1 ) = 0 ,
and extend it strictly increasingly to ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) with limit + ∞ +\infty + ∞ at infinity; integrating from 1 1 1 , with the lower-semicontinuous extension at zero, gives a normalized, strictly convex, superlinear generator. On E = [ 0 , 1 ] E=[0,1] E = [ 0 , 1 ] , fix μ = ( δ 0 + δ 1 ) / 2 \mu=(\delta_0+\delta_1)/2 μ = ( δ 0 + δ 1 ) /2 and take the bounded Lipschitz cost c ( z , w ) = τ min { z , e − 3 } ( 2 w − 1 ) c(z,w)=\tau\min\{z,e^{-3}\}(2w-1) c ( z , w ) = τ min { z , e − 3 } ( 2 w − 1 ) . For 0 < z ≤ e − 3 0<z\le e^{-3} 0 < z ≤ e − 3 the unique optimal densities at 0 , 1 0,1 0 , 1 are 1 + r ( z ) , 1 − r ( z ) 1+r(z),1-r(z) 1 + r ( z ) , 1 − r ( z ) , where r ( z ) = 1 / log ( 1 / z ) r(z)=1/\log(1/z) r ( z ) = 1/ log ( 1/ z ) . Hence W 1 ( K [ μ , z ] , K [ μ , 0 ] ) = r ( z ) / 2 W_1(K[\mu,z],K[\mu,0])=r(z)/2 W 1 ( K [ μ , z ] , K [ μ , 0 ]) = r ( z ) /2 , which obeys no positive Hölder exponent at zero. This explains the quantitative response assumption.
Empirical normalization. For μ = ∑ i m i δ w i \mu=\sum_i m_i\delta_{w_i} μ = ∑ i m i δ w i with m i > 0 m_i>0 m i > 0 , write k i = m i p i k_i=m_ip_i k i = m i p i and c i = c ( μ , z , w i ) c_i=c(\mu,z,w_i) c i = c ( μ , z , w i ) . Pearson minimization is exactly
min k ∈ Δ n { ∑ i c i k i + τ 2 ∑ i ( k i − m i ) 2 m i } . \min_{k\in\Delta_n}\left\{\sum_i c_i k_i+\frac{\tau}{2}\sum_i\frac{(k_i-m_i)^2}{m_i}\right\}. k ∈ Δ n min { i ∑ c i k i + 2 τ i ∑ m i ( k i − m i ) 2 } .
For m i = 1 / n m_i=1/n m i = 1/ n , the quadratic term differs from ( τ n / 2 ) ∑ i k i 2 (\tau n/2)\sum_i k_i^2 ( τ n /2 ) ∑ i k i 2 by a constant on Δ n \Delta_n Δ n . Completing the square gives the projection of − c / ( τ n ) -c/(\tau n) − c / ( τ n ) . Both quantitative estimates above are independent of the smallest positive m i m_i m i .
The fixed Pearson regularizer in the representation theorem
Use the notation of the proof of Theorem 3.7. The calibration head K ^ [ μ , g ] = μ \widehat{K}[\mu,g]=\mu K [ μ , g ] = μ is the unique optimizer for c = 0 c=0 c = 0 and R = 1 2 χ 2 \mathcal{R}=\frac{1}{2}\chi^2 R = 2 1 χ 2 . For the decoder, ψ k \psi_k ψ k is 1 1 1 -Lipschitz, ∣ ψ k ∣ ≤ R |\psi_k|\le R ∣ ψ k ∣ ≤ R , and
θ j ( η , z ) = U j ( π # η , π X ( π z ) ) 4 M R , ∣ θ j ∣ ≤ 1 4 R . \theta_j(\eta,z)=\frac{U_j(\pi_{\#}\eta,\pi_X(\pi z))}{4MR},\qquad|\theta_j|\le\frac{1}{4R}. θ j ( η , z ) = 4 M R U j ( π # η , π X ( π z )) , ∣ θ j ∣ ≤ 4 R 1 .
The decoder costs and corresponding densities are
c j k ( η , z , w ) = − θ j ( η , z ) ψ k ( w ) , (45) c_{jk}(\eta,z,w)=-\theta_j(\eta,z)\psi_k(w), \tag*{(45)} c j k ( η , z , w ) = − θ j ( η , z ) ψ k ( w ) , (45)
p j k ∗ ( η , z , w ) = 1 + θ j ( η , z ) ( ψ k ( w ) − ∫ ψ k d η ) ∈ [ 1 / 2 , 3 / 2 ] . (46) p^*_{jk}(\eta,z,w)=1+\theta_j(\eta,z)\left(\psi_k(w)-\int\psi_k\,d\eta\right)\in[1/2,3/2]. \tag*{(46)} p j k ∗ ( η , z , w ) = 1 + θ j ( η , z ) ( ψ k ( w ) − ∫ ψ k d η ) ∈ [ 1/2 , 3/2 ] . (46)
Here ∣ U j ∣ ≤ M |U_j|\le M ∣ U j ∣ ≤ M and U j U_j U j is β \beta β -Hölder in its context, uniformly at every fixed spatial query. Since π \pi π is 1 1 1 -Lipschitz, the costs in (45) obey
∣ c j k ∣ ≤ 1 4 , Lip w ( c j k ) ≤ 1 4 R , sup w ∣ c j k ( η , z , w ) − c j k ( ξ , z , w ) ∣ ≤ C 4 M W 1 ( η , ξ ) β . |c_{jk}|\le\frac{1}{4},\qquad\operatorname{Lip}_w(c_{jk})\le\frac{1}{4R},\qquad\sup_w|c_{jk}(\eta,z,w)-c_{jk}(\xi,z,w)|\le\frac{C}{4M}W_1(\eta,\xi)^\beta. ∣ c j k ∣ ≤ 4 1 , Lip w ( c j k ) ≤ 4 R 1 , w sup ∣ c j k ( η , z , w ) − c j k ( ξ , z , w ) ∣ ≤ 4 M C W 1 ( η , ξ ) β .
They are Borel and satisfy the fixed-query hypotheses globally. The density p j k ∗ p^*_{jk} p j k ∗ in (46) integrates to one and lies in [ 1 / 2 , 3 / 2 ] [1/2,3/2] [ 1/2 , 3/2 ] . Taking b P e a r s o n = 1 − θ j ∫ ψ k d η b^{\mathrm{Pearson}}=1-\theta_j\int\psi_k\,d\eta b Pearson = 1 − θ j ∫ ψ k d η in (37) gives precisely this density. Alternatively, for every probability density q ∈ L 2 ( η ) q\in L^2(\eta) q ∈ L 2 ( η ) ,
J η , z ( q η ) − J η , z ( p j k ∗ η ) = 1 2 ∫ ( q − p j k ∗ ) 2 d η , J_{\eta,z}(q\eta)-J_{\eta,z}(p^*_{jk}\eta)=\frac{1}{2}\int(q-p^*_{jk})^2\,d\eta, J η , z ( q η ) − J η , z ( p j k ∗ η ) = 2 1 ∫ ( q − p j k ∗ ) 2 d η ,
because c j k + p j k ∗ − 1 = − θ j ∫ ψ k d η c_{jk}+p^*_{jk}-1=-\theta_j\int\psi_k\,d\eta c j k + p j k ∗ − 1 = − θ j ∫ ψ k d η is constant in the read token. All other measures have infinite objective. The variational heads therefore agree with the original construction on every context, so the exact representation and its global stability are unchanged. This concerns the representing network constructed in that theorem, not a variational realization of every prescribed supported kernel.
Projected Wasserstein regularization and SelfWarp reads
Proof of Proposition 4.2. Write F ( μ ) = P ( supp μ ) F(\mu)=\mathcal{P}(\operatorname{supp}\mu) F ( μ ) = P ( supp μ ) . Since E \mathrm{E} E is compact, this is a nonempty compact subset of P ( E ) \mathcal{P}(\mathrm{E}) P ( E ) . For each fixed ( μ , z ) (\mu,z) ( μ , z ) , the finite-valued cost is continuous and bounded in w w w , so its integral is continuous in κ \kappa κ . Continuous pushforward and continuity of W 2 W_2 W 2 on probabilities over compact X X X show that J μ , z J_{\mu,z} J μ , z is continuous in κ \kappa κ . It therefore attains its minimum on F ( μ ) F(\mu) F ( μ ) .
To obtain a Borel choice of minimizers, first observe that F F F is lower hemicontinuous. Given κ ∈ F ( μ ) \kappa\in F(\mu) κ ∈ F ( μ ) and an open neighborhood U U U of κ \kappa κ , choose a finitely supported probability ∑ i a i δ w i ∈ U \sum_i a_i\delta_{w_i}\in U ∑ i a i δ w i ∈ U with w i ∈ supp μ w_i\in\operatorname{supp}\mu w i ∈ supp μ . There are open neighborhoods O i O_i O i of w i w_i w i such that ∑ i a i δ w i ′ ∈ U \sum_i a_i\delta_{w_i'}\in U ∑ i a i δ w i ′ ∈ U whenever w i ′ ∈ O i w_i'\in O_i w i ′ ∈ O i . Each μ ( O i ) > 0 \mu(O_i)>0 μ ( O i ) > 0 . By the Portmanteau theorem, every sufficiently nearby μ ′ \mu' μ ′ has μ ′ ( O i ) > 0 \mu'(O_i)>0 μ ′ ( O i ) > 0 , and hence a support point w i ′ ∈ O i w_i'\in O_i w i ′ ∈ O i . Thus F ( μ ′ ) ∩ U ≠ ∅ F(\mu')\cap U\ne\varnothing F ( μ ′ ) ∩ U = ∅ . In particular, F F F is a measurable compact-valued correspondence.
By the standing measurability convention, c c c is jointly Borel. Parameter-dependent integration therefore makes ( μ , z , κ ) ↦ J μ , z ( κ ) (\mu,z,\kappa)\mapsto J_{\mu,z}(\kappa) ( μ , z , κ ) ↦ J μ , z ( κ ) Borel; it is continuous in κ \kappa κ for fixed ( μ , z ) (\mu,z) ( μ , z ) . The measurable minimum theorem gives a measurable nonempty compact-valued argmin correspondence. The measurable selection theorem [15 Theorem 1] supplies a Borel selector K [ μ , z ] K[\mu,z] K [ μ , z ] . Since K [ μ , z ] ∈ F ( μ ) K[\mu,z]\in F(\mu) K [ μ , z ] ∈ F ( μ ) , this is a supported Markov kernel satisfying (20). No uniqueness or continuity of the selection is asserted. ∎
Proposition B.3 (Exact recovery of SelfWarp reads). Let ( X , ρ X ) (X,\rho_X) ( X , ρ X ) be a compact metric space, let A ⊂ R m A\subset\mathbb{R}^m A ⊂ R m be compact, and set E = X × A \mathrm{E}=X\times A E = X × A with spatial projection π X \pi_X π X . Let λ ∈ P ( X ) \lambda\in\mathcal{P}(X) λ ∈ P ( X ) have full support. Fix a Borel address map τ h : E → X \tau_h:\mathrm{E}\to X τ h : E → X and a constant B > D : = diam ( X ) B>D:=\operatorname{diam}(X) B > D := diam ( X ) . Use the regularizer (25) and the cost
c h ( μ , z , w ) = B ρ X ( π X ( w ) , τ h ( z ) ) . (47) c_h(\mu,z,w)=B\rho_X(\pi_X(w),\tau_h(z)). \tag*{(47)} c h ( μ , z , w ) = B ρ X ( π X ( w ) , τ h ( z )) . (47) For every continuous f : X → A f:X\to A f : X → A , write f ( y ) = ( y , f ( y ) ) f(y)=(y,f(y)) f ( y ) = ( y , f ( y )) and γ f = ( j f ) # λ \gamma_f=(j_f)_{\#}\lambda γ f = ( j f ) # λ . Then, for every z ∈ E z\in\mathrm{E} z ∈ E , the unique minimizer in (20) is
K h [ γ f , z ] = δ j f ( τ h ( z ) ) = δ ( τ h ( z ) , f ( τ h ( z ) ) ) . (48) K_h[\gamma_f,z]=\delta_{j_f(\tau_h(z))}=\delta_{(\tau_h(z),f(\tau_h(z)))}. \tag*{(48)} K h [ γ f , z ] = δ j f ( τ h ( z )) = δ ( τ h ( z ) , f ( τ h ( z ))) . (48)
In particular, on a compact Euclidean domain or torus, take τ h ( x , a ) = x + g h ( a ) \tau_h(x,a)=x+g_h(a) τ h ( x , a ) = x + g h ( a ) , assuming the address lies in X X X or using periodic wrapping. With the value map v h ( y , b ) = V h b v_h(y,b)=V_hb v h ( y , b ) = V h b , the resulting read is
r h ( γ f , ( x , f ( x ) ) ) = V h f ( x + g h ( f ( x ) ) ) , (49) r_h(\gamma_f,(x,f(x)))=V_hf\left(x+g_h(f(x))\right), \tag*{(49)} r h ( γ f , ( x , f ( x ))) = V h f ( x + g h ( f ( x )) ) , (49)
which is the continuum SelfWarp head of Muser et al. [26 ] , Appendix C.2.
Proof. Fix f f f , z z z , and write a = τ h ( z ) a=\tau_h(z) a = τ h ( z ) . Continuity of f f f and full support of λ \lambda λ give supp γ f = j f ( X ) \operatorname{supp}\gamma_f=j_f(X) supp γ f = j f ( X ) , so κ a : = δ j f ( a ) \kappa_a:=\delta_{j_f(a)} κ a := δ j f ( a ) is feasible. For any feasible κ \kappa κ , set ν = ( π X ) # κ \nu=(\pi_X)_\#\kappa ν = ( π X ) # κ , and let q q q be an optimal coupling of ν \nu ν and λ \lambda λ for W 2 W_2 W 2 . Since both distances below are bounded by D D D , the reverse triangle inequality gives
1 2 ( ρ X ( y , t ) 2 − ρ X ( a , t ) 2 ) ≥ − D ρ X ( y , a ) . \frac{1}{2}\left(\rho_X(y,t)^2-\rho_X(a,t)^2\right)\geq-D\rho_X(y,a). 2 1 ( ρ X ( y , t ) 2 − ρ X ( a , t ) 2 ) ≥ − D ρ X ( y , a ) .
Integrating against q ( d y , d t ) q(dy,dt) q ( d y , d t ) yields
1 2 W 2 ( ν , λ ) 2 − 1 2 W 2 ( δ a , λ ) 2 ≥ − D ∫ X ρ X ( y , a ) ν ( d y ) . \frac{1}{2}W_2(\nu,\lambda)^2-\frac{1}{2}W_2(\delta_a,\lambda)^2\geq-D\int_X\rho_X(y,a)\,\nu(dy). 2 1 W 2 ( ν , λ ) 2 − 2 1 W 2 ( δ a , λ ) 2 ≥ − D ∫ X ρ X ( y , a ) ν ( d y ) .
Hence the objective corresponding to (47) satisfies
J γ f , z ( κ ) − J γ f , z ( κ a ) ≥ ( B − D ) ∫ X ρ X ( y , a ) ν ( d y ) . (50) J_{\gamma_f,z}(\kappa)-J_{\gamma_f,z}(\kappa_a)\geq(B-D)\int_X\rho_X(y,a)\,\nu(dy). \tag*{(50)} J γ f , z ( κ ) − J γ f , z ( κ a ) ≥ ( B − D ) ∫ X ρ X ( y , a ) ν ( d y ) . (50)
As B > D B>D B > D , every minimizer has spatial marginal δ a \delta_a δ a . The graph j f ( X ) j_f(X) j f ( X ) contains exactly one token above a a a , so its only supported probability measure with that marginal is κ a \kappa_a κ a . This proves uniqueness and (48) ; integrating the value map proves (49) . The cost is Borel and continuous in w w w , so every minimizing Markov kernel supplied by Proposition 4.2 has the stated restriction to continuous graph contexts.
Scope of the recovery. For a general context μ \mu μ , if a = τ h ( z ) ∈ π X ( supp μ ) a=\tau_h(z)\in\pi_X(\operatorname{supp}\mu) a = τ h ( z ) ∈ π X ( supp μ ) , the same argument with ( π X ) # μ (\pi_X)_\#\mu ( π X ) # μ in place of λ \lambda λ shows that the full set of minimizers is
P ( supp μ ∩ π X − 1 ( { a } ) ) . \mathcal{P}\left(\operatorname{supp}\mu\cap\pi_X^{-1}(\{a\})\right). P ( supp μ ∩ π X − 1 ({ a }) ) .
Thus the spatial cost alone does not select between multiple observed feature values at the same address. Proposition B.3 recovers the continuum graph-context reads; the finite-grid interpolation rows and boundary conventions are described separately in Section D.5. No stability assertion for arbitrary noisy contexts follows from this exact-recovery statement.
Approximation by uniformly discretization stable operators
Uniform discretization stability does not prevent approximation of continuous operators on compact sets of input functions.
Proposition C.1. Let D ⊂ L 2 ( X , Y ) \mathcal{D}\subset L^2(X,Y) D ⊂ L 2 ( X , Y ) be compact. For every continuous A : D → L 2 ( X , Y ) A:\mathcal{D}\to L^2(X,Y) A : D → L 2 ( X , Y ) and ϵ > 0 \epsilon>0 ϵ > 0 , there exists N : P ( G ) × X → Y \mathcal{N}:\mathcal{P}(\mathcal{G})\times X\to Y N : P ( G ) × X → Y , uniformly discretization stable on P ( G ) \mathcal{P}(\mathcal{G}) P ( G ) , such that
sup f ∈ D ∥ N ( γ f , ⋅ ) − A [ f ] ∥ L 2 ( X , R d o u t ) < ϵ . (51) \sup_{f\in\mathcal{D}}\left\|\mathcal{N}(\gamma_f,\cdot)-A[f]\right\|_{L^2(X,\mathbb{R}^{d_{\mathrm{out}}})}<\epsilon. \tag*{(51)} f ∈ D sup ∥ N ( γ f , ⋅ ) − A [ f ] ∥ L 2 ( X , R d out ) < ϵ . (51)
Proof. We approximate A A A by weighted averages of finitely many of its outputs, with weights that depend Lipschitz continuously on the input graph measure. If D \mathcal{D} D is empty, any constant map suffices, so assume D ≠ ∅ \mathcal{D}\ne\varnothing D = ∅ .
First, the graph embedding is continuous because coupling the graph measures at the same spatial point gives
W 1 ( γ f , γ g ) ≤ ∫ X ∣ f ( x ) − g ( x ) ∣ λ ( d x ) ≤ ∥ f − g ∥ L 2 . W_1(\gamma_f,\gamma_g)\leq\int_X|f(x)-g(x)|\,\lambda(dx)\leq\|f-g\|_{L^2}. W 1 ( γ f , γ g ) ≤ ∫ X ∣ f ( x ) − g ( x ) ∣ λ ( d x ) ≤ ∥ f − g ∥ L 2 .
It is also injective. Since D \mathcal{D} D is compact, this embedding is therefore a homeomorphism onto the compact set
K = { γ f : f ∈ D } ⊂ P ( G ) . \mathcal{K}=\{\gamma_f:f\in\mathcal{D}\}\subset\mathcal{P}(\mathcal{G}). K = { γ f : f ∈ D } ⊂ P ( G ) .
Consequently, the induced map A ~ : K → L 2 ( X , Y ) \widetilde{A}:\mathcal{K}\to L^2(X,Y) A : K → L 2 ( X , Y ) , defined by A ~ ( γ f ) = A [ f ] \widetilde{A}(\gamma_f)=A[f] A ( γ f ) = A [ f ] , is uniformly continuous. Choose δ > 0 \delta>0 δ > 0 such that
W 1 ( γ f , γ g ) < δ ⟹ ∥ A [ f ] − A [ g ] ∥ L 2 < ϵ / 2 for f , g ∈ D . W_1(\gamma_f,\gamma_g)<\delta\quad\Longrightarrow\quad\lVert A[f]-A[g]\rVert_{L^2}<\epsilon/2 \qquad\text{for } f,g\in\mathcal{D}. W 1 ( γ f , γ g ) < δ ⟹ ∥ A [ f ] − A [ g ] ∥ L 2 < ϵ /2 for f , g ∈ D .
Choose a finite δ / 2 \delta/2 δ /2 -net μ i = γ f i \mu_i=\gamma_{f_i} μ i = γ f i , i = 1 , … , m i=1,\ldots,m i = 1 , … , m , in K \mathcal{K} K . For each i i i , choose a Borel representative h i : X → Y h_i:X\to Y h i : X → Y of A [ f i ] A[f_i] A [ f i ] . Also fix y ∗ ∈ Y y_*\in Y y ∗ ∈ Y and set h 0 ( x ) = y ∗ h_0(x)=y_* h 0 ( x ) = y ∗ .
We now define the weights on all of P ( G ) \mathcal{P}(\mathrm{G}) P ( G ) . Put
b i ( μ ) = max { δ − W 1 ( μ , μ i ) , 0 } , B ( μ ) = max { ∑ i = 1 m b i ( μ ) , δ / 2 } , b_i(\mu)=\max\{\delta-W_1(\mu,\mu_i),0\},\qquad B(\mu)=\max\left\{\sum_{i=1}^{m}b_i(\mu),\delta/2\right\}, b i ( μ ) = max { δ − W 1 ( μ , μ i ) , 0 } , B ( μ ) = max { i = 1 ∑ m b i ( μ ) , δ /2 } ,
and set
w i ( μ ) = b i ( μ ) B ( μ ) ( i = 1 , … , m ) , w 0 ( μ ) = 1 − ∑ i = 1 m w i ( μ ) . w_i(\mu)=\frac{b_i(\mu)}{B(\mu)}\quad(i=1,\ldots,m),\qquad w_0(\mu)=1-\sum_{i=1}^{m}w_i(\mu). w i ( μ ) = B ( μ ) b i ( μ ) ( i = 1 , … , m ) , w 0 ( μ ) = 1 − i = 1 ∑ m w i ( μ ) .
All these weights are nonnegative and sum to one. They are globally Lipschitz: the functions b i b_i b i are bounded and Lipschitz, and the Lipschitz denominator B B B is bounded below by δ / 2 \delta/2 δ /2 . If μ ∈ K \mu\in\mathcal{K} μ ∈ K , the net property gives ∑ i b i ( μ ) ≥ δ / 2 \sum_i b_i(\mu)\geq\delta/2 ∑ i b i ( μ ) ≥ δ /2 , so w 0 ( μ ) = 0 w_0(\mu)=0 w 0 ( μ ) = 0 . Moreover, w i ( μ ) > 0 w_i(\mu)>0 w i ( μ ) > 0 implies W 1 ( μ , μ i ) < δ W_1(\mu,\mu_i)<\delta W 1 ( μ , μ i ) < δ .
Define
N ( μ , x ) = ∑ i = 0 m w i ( μ ) h i ( x ) . \mathcal{N}(\mu,x)=\sum_{i=0}^{m}w_i(\mu)h_i(x). N ( μ , x ) = i = 0 ∑ m w i ( μ ) h i ( x ) .
This map is Borel measurable and takes values in Y Y Y by convexity. Since the h i h_i h i are uniformly bounded and the weights are Lipschitz, there is a constant C ϵ > 0 C_\epsilon>0 C ϵ > 0 such that, for all μ , ν ∈ P ( G ) \mu,\nu\in\mathcal{P}(\mathrm{G}) μ , ν ∈ P ( G ) ,
sup x ∈ X ∣ N ( μ , x ) − N ( ν , x ) ∣ ≤ ∑ i = 0 m ( sup x ∈ X ∣ h i ( x ) ∣ ) ∣ w i ( μ ) − w i ( ν ) ∣ ≤ C ϵ W 1 ( μ , ν ) . \sup_{x\in X}\left|\mathcal{N}(\mu,x)-\mathcal{N}(\nu,x)\right|
\leq\sum_{i=0}^{m}\left(\sup_{x\in X}|h_i(x)|\right)|w_i(\mu)-w_i(\nu)|
\leq C_\epsilon W_1(\mu,\nu). x ∈ X sup ∣ N ( μ , x ) − N ( ν , x ) ∣ ≤ i = 0 ∑ m ( x ∈ X sup ∣ h i ( x ) ∣ ) ∣ w i ( μ ) − w i ( ν ) ∣ ≤ C ϵ W 1 ( μ , ν ) .
Thus N \mathcal{N} N is uniformly discretization stable on P ( G ) \mathcal{P}(\mathrm{G}) P ( G ) with β = 1 \beta=1 β = 1 ; the constant C ϵ C_\epsilon C ϵ may depend on the requested accuracy.
Finally, fix f ∈ D f\in\mathcal{D} f ∈ D . At μ = γ f \mu=\gamma_f μ = γ f , the weight w 0 w_0 w 0 vanishes and each active weight w i w_i w i corresponds to an output A [ f i ] A[f_i] A [ f i ] within ϵ / 2 \epsilon/2 ϵ /2 of A [ f ] A[f] A [ f ] in L 2 L^2 L 2 . Therefore
∥ N ( γ f , ⋅ ) − A [ f ] ∥ L 2 ≤ ∑ i = 1 m w i ( γ f ) ∥ A [ f i ] − A [ f ] ∥ L 2 ≤ ϵ / 2. \lVert\mathcal{N}(\gamma_f,\cdot)-A[f]\rVert_{L^2}
\leq\sum_{i=1}^{m}w_i(\gamma_f)\lVert A[f_i]-A[f]\rVert_{L^2}
\leq\epsilon/2. ∥ N ( γ f , ⋅ ) − A [ f ] ∥ L 2 ≤ i = 1 ∑ m w i ( γ f ) ∥ A [ f i ] − A [ f ] ∥ L 2 ≤ ϵ /2.
Taking the supremum over f ∈ D f\in\mathcal{D} f ∈ D gives the required bound, since ϵ / 2 < ϵ \epsilon/2<\epsilon ϵ /2 < ϵ .
The approximating map can also be chosen to be a stable Markov kernel network. For nonempty D \mathcal{D} D , all measures in K \mathcal{K} K have spatial marginal λ \lambda λ , so K \mathcal{K} K is admissible by the same argument used in the operator-characterization corollary. Applying Theorem 3.7 to the restriction of the constructed map to K × X \mathcal{K}\times X K × X gives a stable Markov kernel network with the same approximation guarantee. For empty D \mathcal{D} D , a constant Markov kernel network suffices. This conclusion concerns the abstract architecture of Definition 3.3 .
Concrete subclasses of Markov kernel networks
We give explicit realizations of transformers, neural operators, DeepONets, and Flowers in Definition 3.3 . In each case, the Markov kernels specify where a head reads, the value maps specify what it reads, and the channel mixer combines the reads with the query token. The constructions reproduce the stated architectures exactly on their specified input contexts. They do not assert discretization stability: in particular, point evaluation may be unstable in the graph-Wasserstein metric.
Conventions, context updates, and output queries
We work with fixed parameters and finitely many layers on a compact metric spatial domain X X X with at least two points, as in the main text. Hidden tokens have the form z = ( x , a ) ∈ E ℓ = X × A ℓ z=(x,a)\in E_\ell=X\times A_\ell z = ( x , a ) ∈ E ℓ = X × A ℓ , where A ℓ ⊂ R d ℓ A_\ell\subset\mathbb{R}^{d_\ell} A ℓ ⊂ R d ℓ is a compact box containing the feature values under consideration. Thus these are bounded-state realizations, rather than claims about an architecture on an entire unbounded function space. A pointwise input lifting is encoded as q G ( x , y ) = ( x , e θ ( x , y ) ) q_G(x,y)=(x,e^\theta(x,y)) q G ( x , y ) = ( x , e θ ( x , y )) . The coordinate x x x is retained as a spatial label; it need not itself be supplied as a feature to every learned map. Continuum realizations using point reads below assume continuous hidden fields.
For a continuous hidden field u : X → A u:X\to A u : X → A and a fixed probability measure λ \lambda λ with supp λ = X \operatorname{supp}\lambda=X supp λ = X , write
j u ( x ) = ( x , u ( x ) ) , γ u = ( j u ) # λ . (52) j_u(x)=(x,u(x)),\qquad\gamma_u=(j_u)_\#\lambda. \tag*{(52)} j u ( x ) = ( x , u ( x )) , γ u = ( j u ) # λ . (52)
If Γ ( γ u , ( x , u ( x ) ) ) = ( x , u + ( x ) ) \Gamma(\gamma_u,(x,u(x)))=(x,u^+(x)) Γ ( γ u , ( x , u ( x ))) = ( x , u + ( x )) , its context update is exactly B Γ ( γ u ) = ( id , u + ) # λ B_\Gamma(\gamma_u)=(\operatorname{id},u^+)_\#\lambda B Γ ( γ u ) = ( id , u + ) # λ . Likewise, for an empirical context μ = ∑ i ω i δ ( x i , a i ) \mu=\sum_i\omega_i\delta_{(x_i,a_i)} μ = ∑ i ω i δ ( x i , a i ) , with positive weights summing to one, the pushforward updates all features simultaneously while retaining the labels and weights:
B Γ ( μ ) = ∑ i ω i δ Γ ( μ , ( x i , a i ) ) . B_\Gamma(\mu)=\sum_i\omega_i\delta_{\Gamma(\mu,(x_i,a_i))}. B Γ ( μ ) = i ∑ ω i δ Γ ( μ , ( x i , a i )) .
Empirical formulas below reproduce the corresponding discrete computation; they are not assertions that a quadrature sum equals its continuum integral.
Supported point reads. Continuity and full spatial support imply supp γ u = j u ( X ) \operatorname{supp}\gamma_u=j_u(X) supp γ u = j u ( X ) . The graph law therefore determines u u u , and the addressed read
D y [ γ u ] : = δ ( y , u ( y ) ) , y ∈ X , (53) D_y[\gamma_u]:=\delta_{(y,u(y))},\qquad y\in X, \tag*{(53)} D y [ γ u ] := δ ( y , u ( y )) , y ∈ X , (53)
is supported on the context. For a finite context with distinct spatial labels, D x i [ μ ] = δ ( x i , a i ) D_{x_i}[\mu]=\delta_{(x_i,a_i)} D x i [ μ ] = δ ( x i , a i ) is similarly valid at a supplied label. An arbitrary off-grid point requires a specified reconstruction rule. These reads are Borel maps on their stated domains. Indeed, on continuous graph contexts, u ↦ γ u u\mapsto\gamma_u u ↦ γ u is a continuous injection from the Polish space C ( X , A ) C(X,A) C ( X , A ) into P ( E ) \mathcal{P}(E) P ( E ) ; its image and inverse are Borel, and point evaluation is jointly continuous in ( u , y ) (u,y) ( u , y ) . For empirical contexts, normalizing the restriction of μ \mu μ to { y } × A \{y\}\times A { y } × A whenever this fiber has positive mass gives a jointly Borel supported read; it equals the stated Dirac measure when the feature at y y y is unique.
A final pointwise projection Q Q Q is consequently implemented by
q X ( x ) = ( x , a ∗ ) , K p [ μ , z ] = D π X z [ μ ] , v p ( y , b ) = b , m p ( z , r ) = Q ( r ) , (54) \begin{aligned}
q_X(x)&=(x,a_*),\qquad K_p[\mu,z]=D_{\pi_X z}[\mu],\\
v_p(y,b)&=b,\qquad m_p(z,r)=Q(r), \tag*{(54)}
\end{aligned} q X ( x ) v p ( y , b ) = ( x , a ∗ ) , K p [ μ , z ] = D π X z [ μ ] , = b , m p ( z , r ) = Q ( r ) , (54)
where a ∗ a_* a ∗ is any fixed feature vector in the final box. In particular, the encoder does not need access to the unknown output feature: the decoder kernel obtains it from the context. On a grid this realizes the projection at existing labels. Off-grid reconstruction requires its own supported reads and mixer, as illustrated for Flowers below.
Domains of the component maps. Each construction specifies the contexts on which equality with the original architecture is claimed. To obtain kernels on all of P ( E ) × E \mathcal{P}(E)\times E P ( E ) × E , use K [ μ , z ] = μ K[\mu,z]=\mu K [ μ , z ] = μ whenever a stated domain or mass condition fails, including an absent graph, sensor, or positive denominator. The relevant graph and fixed-grid context classes are Borel, so this gives supported Borel extensions. Values are bounded on the compact feature spaces. Since probability reads lie in the closed convex hulls of their value ranges, the mixers can be extended to arbitrary read vectors by clipping to bounding boxes and then clipping their outputs to sufficiently large feature boxes. Such extensions leave the stated realizations unchanged. For a prescribed compact convex output set Y Y Y , projection onto Y Y Y likewise leaves represented outputs in Y Y Y unchanged. None of these extensions is claimed to preserve stability automatically.
We realize the coordinate-preserving attention and pointwise MLP layers of Furuya et al. [12 ] , Section 4, Eqs. (21)–(24). Here X ⊂ R d x X\subset\mathbb{R}^{d_x} X ⊂ R d x , and hidden tokens have the form z = ( x , a ) z=(x,a) z = ( x , a ) with a ∈ R d a \in\mathbb{R}^d a ∈ R d . The spatial coordinate is passed unchanged to the output, while the query, key, value, and pointwise feature maps may all depend on the full token. The input lifting can be the identity q G ( x , y ) = ( x , y ) q_G(x,y)=(x,y) q G ( x , y ) = ( x , y ) , or a coordinate-preserving pointwise embedding q G ( x , y ) = ( x , e θ ( x , y ) ) q_G(x,y)=(x,e^\theta(x,y)) q G ( x , y ) = ( x , e θ ( x , y )) . No separate additive positional embedding is needed for this construction.
Suppress the layer index. Let d d d , p p p , and d ′ d' d ′ be the feature dimensions before attention, after attention, and after the following MLP, respectively. For head h h h , let k h k_h k h be the query/key dimension and d h d_h d h the value dimension. Choose
Q h θ , K k e y , h θ ∈ R k h × ( d x + d ) , V h θ ∈ R d h × ( d x + d ) , Q_h^\theta,K_{\mathrm{key},h}^\theta\in\mathbb{R}^{k_h \times(d_x+d)}, \qquad V_h^\theta\in\mathbb{R}^{d_h \times(d_x+d)}, Q h θ , K key , h θ ∈ R k h × ( d x + d ) , V h θ ∈ R d h × ( d x + d ) ,
W h θ ∈ R p × d h , W l o c θ ∈ R p × ( d x + d ) . (55) W_h^\theta\in\mathbb{R}^{p \times d_h}, \qquad W_{\mathrm{loc}}^\theta\in\mathbb{R}^{p \times(d_x+d)}. \tag*{(55)} W h θ ∈ R p × d h , W loc θ ∈ R p × ( d x + d ) . (55)
The subscript key distinguishes the attention key matrix from a Markov kernel. The local matrix W l o c θ W_{\mathrm{loc}}^\theta W loc θ corresponds to W L W^L W L in the cited construction. For z = ( x , a ) z=(x,a) z = ( x , a ) and w = ( y , b ) w=(y,b) w = ( y , b ) , define
s h θ ( z , w ) : = ⟨ Q h θ z , K k e y , h θ w ⟩ k h . (56) s_h^\theta(z,w):=\frac{\left\langle Q_h^\theta z,K_{\mathrm{key},h}^\theta w\right\rangle}{\sqrt{k_h}}. \tag*{(56)} s h θ ( z , w ) := k h ⟨ Q h θ z , K key , h θ w ⟩ . (56)
K h [ μ , z ] ( d w ) : = exp s h θ ( z , w ) ∫ E exp s h θ ( z , w ~ ) μ ( d w ~ ) μ ( d w ) , v h ( w ) : = V h θ w . (57) K_h[\mu,z](\mathrm{d}w):=\frac{\exp s_h^\theta(z,w)}{\int_{\mathcal{E}}\exp s_h^\theta(z,\widetilde{w})\mu(\mathrm{d}\widetilde{w})}\mu(\mathrm{d}w), \qquad v_h(w):=V_h^\theta w. \tag*{(57)} K h [ μ , z ] ( d w ) := ∫ E exp s h θ ( z , w ) μ ( d w ) exp s h θ ( z , w ) μ ( d w ) , v h ( w ) := V h θ w . (57)
Each row is a probability measure supported on supp μ \operatorname{supp}\mu supp μ . The read r h r_h r h has dimension d h d_h d h , and ∑ h W h θ r h ∈ R p \sum_h W_h^\theta r_h \in\mathbb{R}^p ∑ h W h θ r h ∈ R p is the output projection of the concatenated head values. The matrices W h θ W_h^\theta W h θ are learned channel maps; preservation of the spatial coordinate is imposed separately by the mixer. The scaling involves the query/key dimension k h k_h k h , which need not equal the value dimension d h d_h d h .
The coordinate-preserving pointwise MLP has the form
F θ ( x , t ) = ( x , H θ ( x , t ) ) , H θ : R d x + p → R d ′ , (58) F^\theta(x,t)=(x,H^\theta(x,t)), \qquad H^\theta:\mathbb{R}^{d_x+p}\to\mathbb{R}^{d'}, \tag*{(58)} F θ ( x , t ) = ( x , H θ ( x , t )) , H θ : R d x + p → R d ′ , (58)
where H θ H^\theta H θ is a feed-forward MLP acting on the full position/feature pair. For example, one may take H θ ( x , t ) = A 2 θ σ ( A 1 θ ( x , t ) + b 1 θ ) + b 2 θ H^\theta(x,t)=A_2^\theta\sigma(A_1^\theta(x,t)+b_1^\theta)+b_2^\theta H θ ( x , t ) = A 2 θ σ ( A 1 θ ( x , t ) + b 1 θ ) + b 2 θ , with A 1 θ ∈ R d f f × ( d x + p ) A_1^\theta\in\mathbb{R}^{d_{\mathrm{ff}}\times(d_x+p)} A 1 θ ∈ R d ff × ( d x + p ) , A 2 θ ∈ R d ′ × d f f A_2^\theta\in\mathbb{R}^{d'\times d_{\mathrm{ff}}} A 2 θ ∈ R d ′ × d ff , b 1 θ ∈ R d f f b_1^\theta\in\mathbb{R}^{d_{\mathrm{ff}}} b 1 θ ∈ R d ff , b 2 θ ∈ R d ′ b_2^\theta\in\mathbb{R}^{d'} b 2 θ ∈ R d ′ , and coordinatewise ReLU activation σ \sigma σ . Combining attention with this pointwise map gives the channel mixer
t = W l o c θ z + ∑ h = 1 H W h θ r h , m ( z , r 1 , … , r H ) = ( x , H θ ( x , t ) ) , z = ( x , a ) . (59) t=W_{\mathrm{loc}}^\theta z+\sum_{h=1}^{H}W_h^\theta r_h,\qquad m(z,r_1,\ldots,r_H)=(x,H^\theta(x,t)),\qquad z=(x,a). \tag*{(59)} t = W loc θ z + h = 1 ∑ H W h θ r h , m ( z , r 1 , … , r H ) = ( x , H θ ( x , t )) , z = ( x , a ) . (59)
The attention map is ( x , a ) ↦ ( x , t ) (x,a)\mapsto(x,t) ( x , a ) ↦ ( x , t ) , and (55) applies F θ F^\theta F θ to its output. Since F θ F^\theta F θ is independent of the context, the two operations form a single Markov kernel layer. If p = d p=d p = d , the choice W l o c θ ( x , a ) = a W_{\mathrm{loc}}^\theta(x,a)=a W loc θ ( x , a ) = a gives the usual feature residual connection. If also d ′ = d d'=d d ′ = d , an MLP residual can be written H θ ( x , t ) = t + f θ ( x , t ) H^\theta(x,t)=t+f^\theta(x,t) H θ ( x , t ) = t + f θ ( x , t ) . The displayed construction has no layer normalization or dropout, as in the cited attention/MLP formulas.
For μ = ∑ j = 1 n ω j δ z j \mu=\sum_{j=1}^{n}\omega_j\delta_{z_j} μ = ∑ j = 1 n ω j δ z j , with z j = ( x j , a j ) z_j=(x_j,a_j) z j = ( x j , a j ) , ω j > 0 \omega_j>0 ω j > 0 , and ∑ j ω j = 1 \sum_j\omega_j=1 ∑ j ω j = 1 , the heads give
r h ( μ , z i ) = ∑ j = 1 n α i j h V h θ z j , α i j h = ω j exp s h θ ( z i , z j ) ∑ q = 1 n ω q exp s h θ ( z i , z q ) . (60) r_h(\mu,z_i)=\sum_{j=1}^{n}\alpha_{ij}^h V_h^\theta z_j,\qquad\alpha_{ij}^h=\frac{\omega_j\exp s_h^\theta(z_i,z_j)}{\sum_{q=1}^{n}\omega_q\exp s_h^\theta(z_i,z_q)}. \tag*{(60)} r h ( μ , z i ) = j = 1 ∑ n α ij h V h θ z j , α ij h = ∑ q = 1 n ω q exp s h θ ( z i , z q ) ω j exp s h θ ( z i , z j ) . (60)
Uniform weights recover ordinary rowwise softmax; general weights give its quadrature-weighted counterpart. If m ( z i , r 1 , … , r H ) = ( x i , a i + ) m(z_i,r_1,\ldots,r_H)=(x_i,a_i^+) m ( z i , r 1 , … , r H ) = ( x i , a i + ) , the bulk update is
B Γ ( μ ) = ∑ i = 1 n ω i δ ( x i , a i + ) . (61) B_{\Gamma}(\mu)=\sum_{i=1}^{n}\omega_i\delta_{(x_i,a_i^+)}. \tag*{(61)} B Γ ( μ ) = i = 1 ∑ n ω i δ ( x i , a i + ) . (61)
Thus every token is updated from the same previous context, and the pushforward retains its spatial coordinate and sampling weight. In particular, a graph context γ u \gamma_u γ u is sent to γ u + \gamma_{u^+} γ u + , where u + ( x ) u^+(x) u + ( x ) is the feature component of F ( γ u , ( x , u ( x ) ) ) F(\gamma_u,(x,u(x))) F ( γ u , ( x , u ( x ))) . Reordering complete position/feature tokens merely reorders their outputs; the dependence of the learned maps on the spatial coordinates supplies the positional information.
On a finite sequence, causal and padding masks are included by multiplying the exponential in both the numerator and denominator of (53) by the prescribed mask M h ( z , w ) ∈ { 0 , 1 } M_h(z,w)\in\{0,1\} M h ( z , w ) ∈ { 0 , 1 } . Every represented query must have at least one permitted source token. This is again a supported probability row.
Readouts and propagated queries. For a final pointwise output map O θ O^\theta O θ , the supported graph read D x [ μ L ] D_x[\mu_L] D x [ μ L ] from (53) , followed by the feature projection and O θ O^\theta O θ , realizes the continuum output O θ ( u L ( x ) ) O^\theta(u_L(x)) O θ ( u L ( x )) ; on an empirical context this is the native readout at supplied locations. A separate final attention decoder can use a context-independent query encoder q X ( x ) = ( x , ψ θ ( x ) ) ∈ E L q_X(x)=(x,\psi^\theta(x))\in E_L q X ( x ) = ( x , ψ θ ( x )) ∈ E L and full-token scores and values
s p , h θ ( z , w ) = ⟨ Q p , h θ z , K k e y , p , h θ w ⟩ k p , h , v p , h ( w ) = V p , h θ w , z = q X ( x ) . (62) s^\theta_{p,h}(z,w)=\frac{\langle Q^\theta_{p,h}z,K^\theta_{\mathrm{key},p,h}w\rangle}{\sqrt{k_{p,h}}},\qquad v_{p,h}(w)=V^\theta_{p,h}w,\qquad z=q_X(x). \tag*{(62)} s p , h θ ( z , w ) = k p , h ⟨ Q p , h θ z , K key , p , h θ w ⟩ , v p , h ( w ) = V p , h θ w , z = q X ( x ) . (62)
Insert this score into (57) on E L E_L E L . The decoder mixer combines these reads with the query token and applies the output projection. The query need not be an atom of the source context.
The fixed-query-value construction of Furuya et al. [12 ] uses an additional query trajectory. With the identity input lifting, it starts from z 0 q = ( x , a ∗ ) z^q_0=(x,a_*) z 0 q = ( x , a ∗ ) and propagates
z ℓ + 1 q = Γ ℓ ( μ ℓ , z ℓ q ) , μ ℓ + 1 = Γ ℓ ( μ ℓ , ⋅ ) # μ ℓ , (63) z^q_{\ell+1}=\Gamma_\ell(\mu_\ell,z^q_\ell),\qquad\mu_{\ell+1}=\Gamma_\ell(\mu_\ell,\cdot)_{\#}\mu_\ell, \tag*{(63)} z ℓ + 1 q = Γ ℓ ( μ ℓ , z ℓ q ) , μ ℓ + 1 = Γ ℓ ( μ ℓ , ⋅ ) # μ ℓ , (63)
with the query initialized independently of the source feature at x x x and never inserted into the source context. Its final feature can differ from that of a source token at the same position. The identification of hidden layers above is exact; reproducing that additional query convention requires retaining its trajectory, rather than identifying it with either of the final-context readouts just described.
Neural operators
On a common spatial domain, with the identity local coordinate map and a normalized integration measure, the nonlinear integral layers of Kovachki et al. [16 ] have the form
u + ( x ) = σ ( W θ u ( x ) + b θ ( x ) + ∫ X κ θ [ u ] ( x , y ) u ( y ) λ ( d y ) ) . (64) u^+(x)=\sigma\left(W^\theta u(x)+b^\theta(x)+\int_X\kappa^\theta[u](x,y)u(y)\,\lambda(\mathrm{d}y)\right). \tag*{(64)} u + ( x ) = σ ( W θ u ( x ) + b θ ( x ) + ∫ X κ θ [ u ] ( x , y ) u ( y ) λ ( d y ) ) . (64)
Here u ( x ) ∈ R d u(x)\in\mathbb{R}^d u ( x ) ∈ R d , the updated feature lies in R p \mathbb{R}^p R p , W θ ∈ R p × d W^\theta\in\mathbb{R}^{p\times d} W θ ∈ R p × d , and the integral kernel is matrix-valued. Its dependence on the current features, or on retained input channels, is permitted. In graph-context notation, write the integral as
I i θ ( μ , z ) = ∑ j = 1 d ∫ E k i j θ ( μ , z , w ) b j μ ( d w ) , z = ( x , a ) , w = ( y , b ) . (65) I^\theta_i(\mu,z)=\sum_{j=1}^{d}\int_E k^\theta_{ij}(\mu,z,w)b_j\,\mu(\mathrm{d}w),\qquad z=(x,a),\qquad w=(y,b). \tag*{(65)} I i θ ( μ , z ) = j = 1 ∑ d ∫ E k ij θ ( μ , z , w ) b j μ ( d w ) , z = ( x , a ) , w = ( y , b ) . (65)
For example, k θ ( γ u , ( x , u ( x ) ) , ( y , u ( y ) ) ) = κ θ [ u ] ( x , y ) k^\theta(\gamma_u,(x,u(x)),(y,u(y)))=\kappa^\theta[u](x,y) k θ ( γ u , ( x , u ( x )) , ( y , u ( y ))) = κ θ [ u ] ( x , y ) . We take k θ : P ( E ) × E × E → R p × d k^\theta:\mathcal{P}(E)\times E\times E\to\mathbb{R}^{p\times d} k θ : P ( E ) × E × E → R p × d Borel measurable, allowing context dependence subject to the mass bound below. The signed matrix entries need not themselves be probability densities. A fixed finite integration measure ν = c λ \nu=c\lambda ν = c λ is included by absorbing c = ν ( X ) c=\nu(X) c = ν ( X ) into the kernel.
Choose continuous ϕ 0 , ϕ 1 : X → [ 0 , 1 ] \phi_0,\phi_1:X\to[0,1] ϕ 0 , ϕ 1 : X → [ 0 , 1 ] and nonzero continuous ρ 0 , ρ 1 : X → [ 0 , ∞ ) \rho_0,\rho_1:X\to[0,\infty) ρ 0 , ρ 1 : X → [ 0 , ∞ ) such that
ϕ 0 + ϕ 1 = 1 , ϕ s ρ s = 0 , s ∈ { 0 , 1 } . (66) \phi_0+\phi_1=1,\qquad\phi_s\rho_s=0,\qquad s\in\{0,1\}. \tag*{(66)} ϕ 0 + ϕ 1 = 1 , ϕ s ρ s = 0 , s ∈ { 0 , 1 } . (66)
Such functions exist because X X X has at least two points: choose a function with plateaus 0 and 1 on disjoint neighborhoods, its complement, and bumps inside their respective zero regions. The partition splits each value into two pieces. Each piece then has a region where additional probability mass contributes zero to its read.
Lemma D.1 (Markov realization of signed integral layers). Let the feature range be compact and fix the parameters θ \theta θ . Suppose that, on the represented contexts and queries, the positive and negative parts of every entry satisfy
a i j ± ( μ , z ) : = ∫ E ( k i j θ ) ± ( μ , z , w ) μ ( d w ) ≤ M (67) a^{\pm}_{ij}(\mu,z):=\int_E\left(k^\theta_{ij}\right)^{\pm}(\mu,z,w)\,\mu(\mathrm{d}w)\le M \tag*{(67)} a ij ± ( μ , z ) := ∫ E ( k ij θ ) ± ( μ , z , w ) μ ( d w ) ≤ M (67) for some fixed M > 0 M>0 M > 0 . For the functions in (66) , also assume c s ( μ ) : = ∫ E ρ s ( y ) μ ( d w ) > 0 c_s(\mu):=\int_E\rho_s(y)\,\mu(\mathrm{d}w)>0 c s ( μ ) := ∫ E ρ s ( y ) μ ( d w ) > 0 for s = 0 , 1 s=0,1 s = 0 , 1 . Then (65) is recovered exactly from 4 p d 4pd 4 p d scalar Markov read heads, each absolutely continuous with respect to μ \mu μ , by a linear channel mixer.
Proof. Define the completion probabilities
ν s [ μ ] ( d w ) : = ρ s ( y ) c s ( μ ) μ ( d w ) . (68) \nu_s[\mu](\mathrm{d}w) := \frac{\rho_s(y)}{c_s(\mu)} \mu(\mathrm{d}w). \tag*{(68)} ν s [ μ ] ( d w ) := c s ( μ ) ρ s ( y ) μ ( d w ) . (68)
For the full-support graph contexts of Section D.1, c s ( γ u ) = ∫ X ρ s d λ > 0 c_s(\gamma_u) = \int_X \rho_s \,\mathrm{d}\lambda> 0 c s ( γ u ) = ∫ X ρ s d λ > 0 is independent of u u u .
Index the heads by h = ( i , j , s , ϵ ) h=(i,j,s,\epsilon) h = ( i , j , s , ϵ ) , where ϵ ∈ { + , − } \epsilon\in\{+,-\} ϵ ∈ { + , − } . Set
K i j , s , ϵ [ μ , z ] ( d w ) : = ( k i j θ ) ϵ ( μ , z , w ) M μ ( d w ) + ( 1 − a i j ϵ ( μ , z ) M ) ν s [ μ ] ( d w ) , (69) K_{ij,s,\epsilon}[\mu,z](\mathrm{d}w) := \frac{(k_{ij}^{\theta})^{\epsilon}(\mu,z,w)}{M}\mu(\mathrm{d}w)+\left(1-\frac{a_{ij}^{\epsilon}(\mu,z)}{M}\right)\nu_s[\mu](\mathrm{d}w), \tag*{(69)} K ij , s , ϵ [ μ , z ] ( d w ) := M ( k ij θ ) ϵ ( μ , z , w ) μ ( d w ) + ( 1 − M a ij ϵ ( μ , z ) ) ν s [ μ ] ( d w ) , (69)
v i j , s , ϵ ( w ) : = ϕ s ( y ) b j . (70) v_{ij,s,\epsilon}(w):=\phi_s(y)b_j. \tag*{(70)} v ij , s , ϵ ( w ) := ϕ s ( y ) b j . (70)
The two nonnegative summands in (65) have masses a i j ϵ / M a_{ij}^{\epsilon}/M a ij ϵ / M and 1 − a i j ϵ / M 1-a_{ij}^{\epsilon}/M 1 − a ij ϵ / M , respectively. Hence K i j , s , ϵ K_{ij,s,\epsilon} K ij , s , ϵ is a probability measure supported on supp μ \operatorname{supp}\mu supp μ . The bounded value map depends only on w w w . Since it vanishes ν s [ μ ] \nu_s[\mu] ν s [ μ ] -almost everywhere, its read is
r i j , s , ϵ ( μ , z ) = 1 M ∫ X ( k i j θ ) ϵ ( μ , z , w ) ϕ s ( y ) b j μ ( d w ) . (71) r_{ij,s,\epsilon}(\mu,z)=\frac{1}{M}\int_X (k_{ij}^{\theta})^{\epsilon}(\mu,z,w)\phi_s(y)b_j\,\mu(\mathrm{d}w). \tag*{(71)} r ij , s , ϵ ( μ , z ) = M 1 ∫ X ( k ij θ ) ϵ ( μ , z , w ) ϕ s ( y ) b j μ ( d w ) . (71)
Summing over the partition and subtracting the signed parts gives
I i θ ( μ , z ) = M ∑ j = 1 d ∑ s = 0 1 ( r i j , s , + ( μ , z ) − r i j , s , − ( μ , z ) ) . (72) I_i^{\theta}(\mu,z)=M\sum_{j=1}^{d}\sum_{s=0}^{1}\left(r_{ij,s,+}(\mu,z)-r_{ij,s,-}(\mu,z)\right). \tag*{(72)} I i θ ( μ , z ) = M j = 1 ∑ d s = 0 ∑ 1 ( r ij , s , + ( μ , z ) − r ij , s , − ( μ , z ) ) . (72)
Thus the channel mixer for (60) is
m θ ( ( x , a ) , r ) : = ( x , σ ( W θ a + b θ ( x ) + J ( r ) ) ) , m^{\theta}((x,a),r):=\left(x,\sigma(W^{\theta}a+b^{\theta}(x)+J(r))\right), m θ (( x , a ) , r ) := ( x , σ ( W θ a + b θ ( x ) + J ( r )) ) ,
J i ( r ) : = M ∑ j = 1 d ∑ s = 0 1 ( r i j , s , + − r i j , s , − ) . (73) J_i(r):=M\sum_{j=1}^{d}\sum_{s=0}^{1}\left(r_{ij,s,+}-r_{ij,s,-}\right). \tag*{(73)} J i ( r ) := M j = 1 ∑ d s = 0 ∑ 1 ( r ij , s , + − r ij , s , − ) . (73)
In particular, the masses a i j ± a_{ij}^{\pm} a ij ± remain inside the kernels; the mixer does not require them as additional inputs. Preserving x x x makes the context pushforward precisely the graph law of u + u^+ u + . Applying this construction at every layer, together with the lifting of Section D.1 and the decoder (50), recovers the complete network whenever the represented hidden fields remain continuous. Usual continuous local kernels and pointwise maps have this property. Borel kernels alone need not preserve it. Any original-input channels used by the kernels are retained through the layers.
The pointwise bound ∣ k i j θ ∣ ≤ M |k_{ij}^{\theta}|\le M ∣ k ij θ ∣ ≤ M is sufficient for (63), but a uniform row-mass bound is all the proof uses. For the usual continuous local kernel parameterizations, compact feature ranges give a pointwise bound. General dependence on the entire context requires the mass bound separately. The same identities hold for a weighted empirical context μ = ∑ n ω n δ ( x n , u n ) \mu=\sum_n\omega_n\delta_{(x_n,u_n)} μ = ∑ n ω n δ ( x n , u n ) whenever (63) holds and ∑ n ω n ρ s ( x n ) > 0 \sum_n\omega_n\rho_s(x_n)>0 ∑ n ω n ρ s ( x n ) > 0 for both s s s . They then reproduce the corresponding quadrature layer exactly. The completion condition holds eventually along spatial quadratures converging weakly to λ \lambda λ , but need not hold on arbitrary coarse samples. These are representation statements; no discretization-stability assertion follows from the signed decomposition alone.
Fourier neural operators
The finite-mode spectral layers of Fourier neural operators (Li et al. [22 ] ) admit a simpler realization. Let X = T d x X=\mathbb{T}^{d_x} X = T d x with normalized volume λ \lambda λ , and let Λ ⊂ Z d x \Lambda\subset\mathbb{Z}^{d_x} Λ ⊂ Z d x be the finite set of retained modes. With R k θ ∈ C p × d R_k^{\theta}\in\mathbb{C}^{p\times d} R k θ ∈ C p × d , their spectral contribution is
S θ [ u ] ( x ) = ∑ k ∈ Λ e 2 π i k ⋅ x R k θ u ^ ( k ) , u ^ ( k ) = ∫ X e − 2 π i k ⋅ y u ( y ) λ ( d y ) . (74) \mathcal{S}^{\theta}[u](x)=\sum_{k\in\Lambda}e^{2\pi i k\cdot x}R_k^{\theta}\widehat{u}(k),\qquad\widehat{u}(k)=\int_X e^{-2\pi i k\cdot y}u(y)\,\lambda(\mathrm{d}y). \tag*{(74)} S θ [ u ] ( x ) = k ∈ Λ ∑ e 2 π ik ⋅ x R k θ u ( k ) , u ( k ) = ∫ X e − 2 π ik ⋅ y u ( y ) λ ( d y ) . (74)
Take the usual conjugate symmetry of the modes and multipliers so that S θ [ u ] \mathcal{S}^{\theta}[u] S θ [ u ] is real. A single Markov head with
K [ μ , z ] = μ , v ( y , b ) = ( cos ( 2 π k ⋅ y ) b , sin ( 2 π k ⋅ y ) b ) k ∈ Λ (75) K[\mu,z]=\mu,\qquad v(y,b)=\left(\cos(2\pi k\cdot y)b,\sin(2\pi k\cdot y)b\right)_{k\in\Lambda} \tag*{(75)} K [ μ , z ] = μ , v ( y , b ) = ( cos ( 2 π k ⋅ y ) b , sin ( 2 π k ⋅ y ) b ) k ∈ Λ (75)
returns 2 d ∣ Λ ∣ 2d|\Lambda| 2 d ∣Λ∣ real Fourier moments r = ( C k , S k ) k ∈ Λ r=(C_k,S_k)_{k\in\Lambda} r = ( C k , S k ) k ∈ Λ , with u ^ ( k ) = C k − i S k \widehat{u}(k)=C_k-iS_k u ( k ) = C k − i S k . The channel mixer is
m θ ( ( x , a ) , r ) : = ( x , σ ( W θ a + b θ ( x ) + F θ ( x , r ) ) ) , m^\theta((x,a),r):=\left(x,\sigma(W^\theta a+b^\theta(x)+F^\theta(x,r))\right), m θ (( x , a ) , r ) := ( x , σ ( W θ a + b θ ( x ) + F θ ( x , r )) ) ,
F θ ( x , r ) : = Re ∑ k ∈ Λ e 2 π i k ⋅ x R k θ ( C k − i S k ) . (76) F^\theta(x,r):=\operatorname{Re}\sum_{k\in\Lambda}e^{2\pi i k\cdot x}R^\theta_k(C_k-iS_k). \tag*{(76)} F θ ( x , r ) := Re k ∈ Λ ∑ e 2 π ik ⋅ x R k θ ( C k − i S k ) . (76)
Complex notation in this expression abbreviates a real linear map on the sine and cosine moments. Thus the Markov kernel performs an unweighted context read, while the value map collects the Fourier moments and the mixer applies the learned multipliers and synthesizes the result at x x x . The number of read channels depends on the retained modes, not on the spatial resolution.
On a uniform periodic grid, μ = N − 1 ∑ n = 1 N δ ( x n , u n ) \mu=N^{-1}\sum_{n=1}^{N}\delta(x_n,u_n) μ = N − 1 ∑ n = 1 N δ ( x n , u n ) gives normalized discrete Fourier coefficients, and (72) agrees with the corresponding FFT layer when the forward and inverse normalization conventions are matched. Retained modes must be represented with the same grid-frequency and conjugate-symmetry conventions. Other positive quadrature weights give the associated quadrature Fourier layer; equality to the continuum coefficients requires a separate quadrature argument. The formulas use the actual normalized-volume integration measure, or its declared discrete quadrature. Other fixed domain volumes or transform conventions require their known normalization factors. The local term, pointwise maps, and final projection are handled as in Section D.1.
DeepONets
A fixed-sensor DeepONet pairs coefficients from a branch network with features from a trunk network (Lu et al., 2019, Section 2.1). For scalar output, sensors s 1 , … , s S ∈ X s_1,\ldots,s_S\in X s 1 , … , s S ∈ X , and branch/trunk width P P P , its map is
N ( γ f , x ) = ⟨ B θ ( f ( s 1 ) , … , f ( s S ) ) , T θ ( x ) ⟩ + b 0 θ , (77) \mathcal{N}(\gamma_f,x)=\left\langle B^\theta(f(s_1),\ldots,f(s_S)),T^\theta(x)\right\rangle+b^\theta_0, \tag*{(77)} N ( γ f , x ) = ⟨ B θ ( f ( s 1 ) , … , f ( s S )) , T θ ( x ) ⟩ + b 0 θ , (77)
where B θ : A S → R P B^\theta:A^S\to\mathbb{R}^P B θ : A S → R P and T θ : X → R P T^\theta:X\to\mathbb{R}^P T θ : X → R P are the branch and trunk maps. Vector outputs are obtained by applying this construction to each output component. We use the common input/output domain of Definition 3.3 .
Take the identity lifting q G ( x , a ) = ( x , a ) q_G(x,a)=(x,a) q G ( x , a ) = ( x , a ) . To satisfy L > 0 L>0 L > 0 in that definition, use one identity context layer: K 0 [ μ , z ] = μ K_0[\mu,z]=\mu K 0 [ μ , z ] = μ , v 0 ≡ 0 v_0\equiv0 v 0 ≡ 0 , and m 0 ( z , r ) = z m_0(z,r)=z m 0 ( z , r ) = z give B 0 ( μ ) = μ B_0(\mu)=\mu B 0 ( μ ) = μ . All sensor extraction and branch–trunk synthesis can then occur in the decoder. Using the supported evaluation rows (49), choose
K p , j [ μ , ( x , a ) ] = D s j [ μ ] , v p , j ( y , b ) = b , j = 1 , … , S . (78) K_{p,j}[\mu,(x,a)]=D_{s_j}[\mu],\qquad v_{p,j}(y,b)=b,\qquad j=1,\ldots,S. \tag*{(78)} K p , j [ μ , ( x , a )] = D s j [ μ ] , v p , j ( y , b ) = b , j = 1 , … , S . (78)
On a continuous graph context γ f \gamma_f γ f , the j j j th read is exactly r j = f ( s j ) r_j=f(s_j) r j = f ( s j ) . The decoder mixer
m p ( ( x , a ) , r 1 , … , r S ) = ⟨ B θ ( r 1 , … , r S ) , T θ ( x ) ⟩ + b 0 θ (79) m_p((x,a),r_1,\ldots,r_S)=\left\langle B^\theta(r_1,\ldots,r_S),T^\theta(x)\right\rangle+b^\theta_0 \tag*{(79)} m p (( x , a ) , r 1 , … , r S ) = ⟨ B θ ( r 1 , … , r S ) , T θ ( x ) ⟩ + b 0 θ (79)
therefore gives (73). It ignores the placeholder feature a a a and uses x x x in the trunk. Signed branch and trunk outputs are permitted because the probability constraint applies to the read measures.
For an empirical input, the same realization is exact when every sensor location occurs in the context with its observed value. The output query x x x may be any point of X X X , since its dependence enters through the trunk and does not require a sample at x x x . If a sensor is absent, its value requires a declared reconstruction rule. The fixed-sensor realization itself provides no such reconstruction and no uniform graph-W 1 W_1 W 1 stability assertion.
Flowers
The SelfWarp heads, residual blocks, and multiscale composition of Flowers are specified in Muser et al. (2026, Appendix C). We first represent a block using the graph-context conventions of Section D.1, with u : X → R C i n u:X\to\mathbb{R}^{C_{\mathrm{in}}} u : X → R C in continuous and bounded. Here X X X is a compact Euclidean domain or a torus. Let C o u t = H C h C_{\mathrm{out}}=HC_h C out = H C h , and write the value projection in head blocks as V h ∈ R C h × C i n V_h\in\mathbb{R}^{C_h\times C_{\mathrm{in}}} V h ∈ R C h × C in . The displacement maps q h : R C i n → R d x q_h:\mathbb{R}^{C_{\mathrm{in}}}\to\mathbb{R}^{d_x} q h : R C in → R d x use all input channels. For z = ( x , a ) z=(x,a) z = ( x , a ) , let τ h ( z ) = x + q h ( a ) \tau_h(z)=x+q_h(a) τ h ( z ) = x + q h ( a ) , interpreted with periodic wrapping, or assume that this address lies in X X X . Set
K h [ γ u , z ] = D τ h ( z ) [ γ u ] = δ ( τ h ( z ) , u ( τ h ( z ) ) ) , v h ( y , b ) = V h b . (80) K_h[\gamma_u,z]=D_{\tau_h(z)}[\gamma_u]=\delta(\tau_h(z),u(\tau_h(z))),\qquad v_h(y,b)=V_hb. \tag*{(80)} K h [ γ u , z ] = D τ h ( z ) [ γ u ] = δ ( τ h ( z ) , u ( τ h ( z ))) , v h ( y , b ) = V h b . (80)
Then r h ( γ u , ( x , u ( x ) ) ) = V h u ( x + g h ( u ( x ) ) ) r_h(\gamma_u,(x,u(x)))=V_hu(x+g_h(u(x))) r h ( γ u , ( x , u ( x ))) = V h u ( x + g h ( u ( x ))) . Thus the mixer
m w a r p ( ( x , a ) , r 1 , … , r H ) = ( x , P a + ⨁ h = 1 H r h ) m_{\mathrm{warp}}((x,a),r_1,\ldots,r_H)=\left(x,Pa+\bigoplus_{h=1}^{H}r_h\right) m warp (( x , a ) , r 1 , … , r H ) = ( x , P a + h = 1 ⨁ H r h )
realizes SelfWarp plus its pointwise residual projection P ∈ R C o u t × C i n P\in\mathbb{R}^{C_{\mathrm{out}}\times C_{\mathrm{in}}} P ∈ R C out × C in . For affine projections, value biases are included in v h v_h v h and residual biases in the mixer. The output retains the target coordinate x x x . Since λ \lambda λ has full support, the point selected in (80) belongs to supp γ u \operatorname{supp}\gamma_u supp γ u .
Normalization. Write b b b for the field produced by this first layer. GroupNorm uses moments of b b b across space and channel groups, so it is a second Markov kernel layer. For every group G ⊂ { 1 , … , C o u t } G\subset\{1,\ldots,C_{\mathrm{out}}\} G ⊂ { 1 , … , C out } , take K G [ γ b , z ] = γ b K_G[\gamma_b,z]=\gamma_b K G [ γ b , z ] = γ b and
v G ( y , b ) = ( 1 ∣ G ∣ ∑ c ∈ G b c , 1 ∣ G ∣ ∑ c ∈ G b c 2 ) . v_G(y,b)=\left(\frac{1}{|G|}\sum_{c\in G}b_c,\frac{1}{|G|}\sum_{c\in G}b_c^2\right). v G ( y , b ) = ( ∣ G ∣ 1 c ∈ G ∑ b c , ∣ G ∣ 1 c ∈ G ∑ b c 2 ) .
The resulting read gives the group mean M G M_G M G and second moment S G S_G S G . Its mixer preserves x x x and maps each feature channel c ∈ G c\in G c ∈ G to
GELU ( γ c b c − M G max { S G − M G 2 , 0 } + ε + β c ) , ε > 0. (81) \operatorname{GELU}\left(\gamma_c\frac{b_c-M_G}{\sqrt{\max\{S_G-M_G^2,0\}+\varepsilon}}+\beta_c\right),\qquad\varepsilon>0. \tag*{(81)} GELU ( γ c max { S G − M G 2 , 0 } + ε b c − M G + β c ) , ε > 0. (81)
For actual moment reads S G − M G 2 ≥ 0 S_G-M_G^2\ge0 S G − M G 2 ≥ 0 ; the maximum only extends the mixer to arbitrary read vectors. This reproduces the order residual sum, GroupNorm, then GELU. On a grid use K G = n − 1 ∑ j = 1 n δ z j K_G=n^{-1}\sum_{j=1}^{n}\delta_{z_j} K G = n − 1 ∑ j = 1 n δ z j to obtain the required uniform spatial average regardless of context weights. When supplied as input, conditioning variables for FiLM can be retained in the token and used to determine γ c , β c \gamma_c,\beta_c γ c , β c . Stacking these two-layer realizations and using the decoder (50) gives the corresponding single-scale network, including its pointwise lifting and final projection.
Interpolation and boundaries. On a fixed grid with tokens z j = ( x j , a j ) z_j=(x_j,a_j) z j = ( x j , a j ) , let b j ( y ) ≥ 0 b_j(y)\ge0 b j ( y ) ≥ 0 be the sampler’s multilinear interpolation weights. For constant-preserving boundary conventions, the implemented warp has
K h [ μ , z ] = ∑ j b j ( τ h ( z ) ) δ z j , ∑ j b j ( τ h ( z ) ) = 1. K_h[\mu,z]=\sum_j b_j(\tau_h(z))\delta_{z_j},\qquad\sum_j b_j(\tau_h(z))=1. K h [ μ , z ] = j ∑ b j ( τ h ( z )) δ z j , j ∑ b j ( τ h ( z )) = 1.
These rows are supported on the sampled tokens. With zero padding, the sum s h ( z ) : = ∑ j b j ( τ h ( z ) ) s_h(z):=\sum_j b_j(\tau_h(z)) s h ( z ) := ∑ j b j ( τ h ( z )) can be less than one. For s h ( z ) > 0 s_h(z)>0 s h ( z ) > 0 , divide this row by s h ( z ) s_h(z) s h ( z ) and multiply the resulting read by s h ( z ) s_h(z) s h ( z ) in m w a r p m_{\mathrm{warp}} m warp ; when s h ( z ) = 0 s_h(z)=0 s h ( z ) = 0 , choose K h [ μ , z ] = μ K_h[\mu,z]=\mu K h [ μ , z ] = μ and multiply the read by zero. The factor s h ( z ) s_h(z) s h ( z ) is determined by the fixed sampler geometry and the local source address, so the mixer can compute it from z z z . This implements zero padding without adding unsupported zero-valued tokens.
The full multiscale network. The U-Net in Muser et al. [26 ] additionally uses strided and transposed convolutions and stored encoder skips. To represent it on its finite grid hierarchy Λ 0 , … , Λ J \Lambda_0,\ldots,\Lambda_J Λ 0 , … , Λ J , retain every finest-grid label i ∈ Λ 0 i\in\Lambda_0 i ∈ Λ 0 throughout the MKN. Choose surjections c ℓ : Λ 0 → Λ ℓ c_\ell:\Lambda_0\to\Lambda_\ell c ℓ : Λ 0 → Λ ℓ , with c 0 = id c_0=\operatorname{id} c 0 = id , and representatives ι ℓ ( y ) ∈ c ℓ − 1 ( y ) \iota_\ell(y)\in c_\ell^{-1}(y) ι ℓ ( y ) ∈ c ℓ − 1 ( y ) . At scale ℓ \ell ℓ , token i i i stores the active feature at c ℓ ( i ) c_\ell(i) c ℓ ( i ) and the previously saved skip features at their corresponding coordinates. Warp addresses use the physical coordinate c ℓ ( i ) c_\ell(i) c ℓ ( i ) at that scale; the original label i i i is retained only to identify the carrier token. A read from a coarse site y y y selects the existing token z ι ℓ ( y ) z_{\iota_\ell(y)} z ι ℓ ( y ) and extracts its active channels. Consequently, each finite convolution stencil is represented by Dirac heads followed by the original, possibly signed, filter matrices in the mixer; the same construction realizes transposed convolutions and changes the active scale while retaining all finest-grid labels. Out-of-domain stencil terms receive zero coefficients. Coarse-grid interpolation uses the preceding formula with these representative tokens. GroupNorm uses the probability row
K n o r m , ℓ [ μ , z ] = 1 ∣ Λ ℓ ∣ ∑ y ∈ Λ ℓ δ z ι ℓ ( y ) , K_{\mathrm{norm},\ell}[\mu,z]=\frac{1}{|\Lambda_\ell|}\sum_{y\in\Lambda_\ell}\delta_{z_{\iota_\ell(y)}}, K norm , ℓ [ μ , z ] = ∣ Λ ℓ ∣ 1 y ∈ Λ ℓ ∑ δ z ι ℓ ( y ) ,
which gives exactly the uniform coarse-grid moments, irrespective of the sizes of the fibers of c ℓ c_\ell c ℓ . Convolutional biases and the ReLU following each down/up convolution are applied in the mixer. Saved skip slots are copied unchanged through the active-channel operations; copying and concatenation are pointwise mixers, performed at the original architecture’s stages. The encoder is q G ( x , a ) = ( x , Lift ( a ⊕ x ) ) q_G(x,a)=(x,\operatorname{Lift}(a\mathbin{\oplus}x)) q G ( x , a ) = ( x , Lift ( a ⊕ x )) , with empty skip slots appended; the final supported point read at a finest-grid query, followed by the original pointwise projection, is Γ p \Gamma_p Γ p . Hence the complete finite-grid Flowers architecture is an MKN. Retaining the finest-grid labels avoids requiring a pushforward layer to create new tokens during upsampling. These exact realizations do not by themselves assert discretization stability of the learned point reads or of the multiscale network.
Hybrid architectures through variational Markov reads
We construct a convex family of regularizers that recovers attention, uniform reads, and continuum SelfWarp at different parameter values. The read kernels satisfy a common Hölder bound on a fixed class of regular graph contexts. We also give a grid approximation bound and explain why the graph-context restriction is needed.
Graph contexts and costs
Let X = [ 0 , 1 ] d X=[0,1]^d X = [ 0 , 1 ] d , with d ≥ 1 d\ge1 d ≥ 1 and normalized Lebesgue probability measure λ = d x \lambda=dx λ = d x . Let V V V be a nonempty compact convex subset of a Euclidean space, and put E = X × V E=X\times V E = X × V with its Euclidean product metric. For L u ≥ 0 L_u\ge0 L u ≥ 0 , define
U L u : = { u : X → V : Lip ( u ) ≤ L u } , j u ( y ) : = ( y , u ( y ) ) , γ u : = ( j u ) # λ . \mathcal{U}_{L_u}:=\{u:X\to V:\operatorname{Lip}(u)\le L_u\},\qquad j_u(y):=(y,u(y)),\qquad\gamma_u:=(j_u)_\#\lambda. U L u := { u : X → V : Lip ( u ) ≤ L u } , j u ( y ) := ( y , u ( y )) , γ u := ( j u ) # λ .
Every probability measure supported on supp γ u = j u ( X ) \operatorname{supp}\gamma_u=j_u(X) supp γ u = j u ( X ) has the unique form κ = ( j u ) # ν \kappa=(j_u)_\#\nu κ = ( j u ) # ν , with ν ∈ P ( X ) \nu\in\mathcal{P}(X) ν ∈ P ( X ) . Consequently,
1 2 W 2 2 ( ( π X ) # κ , ( π X ) # γ u ) = 1 2 W 2 2 ( ν , λ ) , KL ( κ ∥ γ u ) = KL ( ν ∥ λ ) . \frac{1}{2}W_2^2((\pi_X)_\#\kappa,(\pi_X)_\#\gamma_u)=\frac{1}{2}W_2^2(\nu,\lambda),\qquad\operatorname{KL}(\kappa\mathbin{\|}\gamma_u)=\operatorname{KL}(\nu\mathbin{\|}\lambda). 2 1 W 2 2 (( π X ) # κ , ( π X ) # γ u ) = 2 1 W 2 2 ( ν , λ ) , KL ( κ ∥ γ u ) = KL ( ν ∥ λ ) .
The KL identity follows because j u j_u j u is a homeomorphism onto its graph; it also holds when the value is infinite.
To specify a common cost class, set
T L u : = { ( γ u , z , j u ( y ) ) : u ∈ U L u , z ∈ E , y ∈ X } . \mathcal{T}_{L_u}:=\{(\gamma_u,z,j_u(y)):u\in\mathcal{U}_{L_u},\ z\in E,\ y\in X\}. T L u := {( γ u , z , j u ( y )) : u ∈ U L u , z ∈ E , y ∈ X } .
Fix M , L c , L y ≥ 0 M,L_c,L_y\ge0 M , L c , L y ≥ 0 and β ∈ ( 0 , 1 ] \beta\in(0,1] β ∈ ( 0 , 1 ] . Let C \mathcal{C} C consist of the continuous real costs on T L u \mathcal{T}_{L_u} T L u satisfying
∣ c ( γ u , z , j u ( y ) ) ∣ ≤ M , \lvert c(\gamma_u,z,j_u(y))\rvert\le M, ∣ c ( γ u , z , j u ( y ))∣ ≤ M ,
∣ c ( γ u , z , j u ( y ) ) − c ( γ v , z ′ , j v ( y ′ ) ) ∣ ≤ L c ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) β + L y ∣ y − y ′ ∣ . (82) \lvert c(\gamma_u,z,j_u(y))-c(\gamma_v,z',j_v(y'))\rvert\le L_c\left(W_1(\gamma_u,\gamma_v)+\rho_E(z,z')\right)^\beta+L_y\lvert y-y'\rvert. \tag*{(82)} ∣ c ( γ u , z , j u ( y )) − c ( γ v , z ′ , j v ( y ′ ))∣ ≤ L c ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) β + L y ∣ y − y ′ ∣ . (82)
A stable Wasserstein–KL family
Fix τ > 0 \tau>0 τ > 0 . For α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] , define
J α , c ; u , z ( ν ) : = ∫ X c ( γ u , z , j u ( y ) ) ν ( d y ) + 1 − α 2 W 2 2 ( λ , ν ) + α τ KL ( ν ∥ λ ) , ν ∈ P ( X ) . (83) J_{\alpha,c;u,z}(\nu):=\int_X c(\gamma_u,z,j_u(y))\,\nu(dy)+\frac{1-\alpha}{2}W_2^2(\lambda,\nu)+\alpha\tau\operatorname{KL}(\nu\mathbin{\|}\lambda),\qquad\nu\in\mathcal{P}(X). \tag*{(83)} J α , c ; u , z ( ν ) := ∫ X c ( γ u , z , j u ( y )) ν ( d y ) + 2 1 − α W 2 2 ( λ , ν ) + α τ KL ( ν ∥ λ ) , ν ∈ P ( X ) . (83)
At α = 0 \alpha=0 α = 0 the entropy term is omitted, rather than interpreted as 0 ⋅ ( + ∞ ) 0\cdot(+\infty) 0 ⋅ ( + ∞ ) .
Theorem E.1 (A convex class of stable variational reads). For every c ∈ C c\in\mathcal{C} c ∈ C , α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] , u ∈ U L u u\in\mathcal{U}_{L_u} u ∈ U L u , and z ∈ E z\in E z ∈ E , the objective (79) has a unique minimizer ν α , c ; u , z \nu_{\alpha,c;u,z} ν α , c ; u , z . Define the supported graph-token read
K α , c [ γ u , z ] : = ( j u ) # ν α , c ; u , z . (84) K_{\alpha,c}[\gamma_u,z]:=(j_u)_\#\nu_{\alpha,c;u,z}. \tag*{(84)} K α , c [ γ u , z ] := ( j u ) # ν α , c ; u , z . (84) Set
η : = min { β 3 , 1 d + 1 } , ω ( 0 ) : = 0 , ω ( h ) : = h log ( e / h ) ( 0 < h ≤ 1 ) . \eta:=\min\left\{\frac{\beta}{3},\frac{1}{d+1}\right\},\qquad\omega(0):=0,\qquad\omega(h):=h\log(e/h)\quad(0<h\le1). η := min { 3 β , d + 1 1 } , ω ( 0 ) := 0 , ω ( h ) := h log ( e / h ) ( 0 < h ≤ 1 ) . There is a constant A A A , depending only on d d d , L u L_u L u , M M M , L c L_c L c , L y L_y L y , β \beta β , τ \tau τ , and diam ( V ) \operatorname{diam}(V) diam ( V ) , such that
W 1 ( K α , c [ γ u , z ] , K α ~ , c ~ [ γ v , z ′ ] ) ≤ A [ ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) η + ( ∥ c − c ~ ∥ ∞ , T L u + ω ( ∣ α − α ~ ∣ ) ) 1 / 3 ] . (85) W_1\left(K_{\alpha,c}[\gamma_u,z],K_{\widetilde{\alpha},\widetilde{c}}[\gamma_{v},z']\right)
\le A\left[\left(W_1(\gamma_u,\gamma_v)+\rho_E(z,z')\right)^\eta+\left(\lVert c-\widetilde{c}\rVert_{\infty,\mathcal{T}_{L_u}}+\omega(\lvert\alpha-\widetilde{\alpha}\rvert)\right)^{1/3}\right]. \tag*{(85)} W 1 ( K α , c [ γ u , z ] , K α , c [ γ v , z ′ ] ) ≤ A [ ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) η + ( ∥ c − c ∥ ∞ , T L u + ω (∣ α − α ∣) ) 1/3 ] . (85)
In particular, each fixed design defines a Hölder-stable Borel Markov kernel on this graph-context class. The design space C × [ 0 , 1 ] \mathcal{C} \times[0,1] C × [ 0 , 1 ] is convex and compact for uniform convergence of costs and ordinary convergence of α \alpha α .
The exponents are not claimed optimal. The estimate controls read kernels on the stated graph class; applying it through multiple layers requires preservation of the spatial marginal and the uniform graph regularity. It does not supply the all-context kernel bound in Definition 3.5.
Proof of Theorem E.1
The main step is cubic transport growth under linear interpolation of measures ; entropy adds quadratic growth. Together they control the spatial minimizer uniformly. We then lift to graph tokens and compare coefficients using finite-entropy histograms, since a point read itself has infinite entropy.
Throughout this appendix, D = d D=\sqrt{d} D = d and spatial Wasserstein distances use Euclidean distance on X = [ 0 , 1 ] d X=[0,1]^d X = [ 0 , 1 ] d . For the two penalties, write F ( ν ) = 1 2 W 2 2 ( λ , ν ) F(\nu)=\frac{1}{2}W_2^2(\lambda,\nu) F ( ν ) = 2 1 W 2 2 ( λ , ν ) and H ( ν ) = KL ( ν ∥ λ ) H(\nu)=\operatorname{KL}(\nu\mathbin{\|}\lambda) H ( ν ) = KL ( ν ∥ λ ) . Set
q d = ( 12 d D ) − 1 , m d , τ = min { q d , 2 τ / D 3 } > 0. (86) q_d=(12dD)^{-1}, \qquad m_{d,\tau}=\min\{q_d,2\tau/D^3\}>0. \tag*{(86)} q d = ( 12 d D ) − 1 , m d , τ = min { q d , 2 τ / D 3 } > 0. (86)
For the spatial proof, c ∈ C ( X ) c\in C(X) c ∈ C ( X ) denotes a single continuous function of position, and ν α , c \nu_{\alpha,c} ν α , c denotes its minimizing spatial measure. The corresponding token read is its pushforward through j u j_u j u . Write
J α , c ( ν ) = ∫ X c d ν + ( 1 − α ) F ( ν ) + α τ H ( ν ) , V ( c ) = min ν J 0 , c ( ν ) . J_{\alpha,c}(\nu)=\int_X c\,\mathrm{d}\nu+(1-\alpha)F(\nu)+\alpha\tau H(\nu), \qquad V(c)=\min_{\nu}J_{0,c}(\nu). J α , c ( ν ) = ∫ X c d ν + ( 1 − α ) F ( ν ) + α τ H ( ν ) , V ( c ) = ν min J 0 , c ( ν ) .
We use standard compact Kantorovich duality and standard Wasserstein facts; the estimates needed below are proved explicitly.
Transport provides quantitative growth
Growth at zero entropy. For every c ∈ C ( X ) c\in C(X) c ∈ C ( X ) , J 0 , c J_{0,c} J 0 , c has a unique minimizer ν c \nu_c ν c . For every ν ∈ P ( X ) \nu\in\mathcal{P}(X) ν ∈ P ( X ) ,
J 0 , c ( ν ) − V ( c ) ≥ q d W 1 ( ν , ν c ) 3 . (87) J_{0,c}(\nu)-V(c)\ge q_d W_1(\nu,\nu_c)^3. \tag*{(87)} J 0 , c ( ν ) − V ( c ) ≥ q d W 1 ( ν , ν c ) 3 . (87)
Consequently, if ν b = ( 1 − b ) ν 0 + b ν 1 \nu_b=(1-b)\nu_0+b\nu_1 ν b = ( 1 − b ) ν 0 + b ν 1 and r = W 1 ( ν 0 , ν 1 ) r=W_1(\nu_0,\nu_1) r = W 1 ( ν 0 , ν 1 ) , then
( 1 − b ) F ( ν 0 ) + b F ( ν 1 ) − F ( ν b ) ≥ q d b ( 1 − b ) ( b 2 + ( 1 − b ) 2 ) r 3 . (88) (1-b)F(\nu_0)+bF(\nu_1)-F(\nu_b)\ge q_d b(1-b)\left(b^2+(1-b)^2\right)r^3. \tag*{(88)} ( 1 − b ) F ( ν 0 ) + b F ( ν 1 ) − F ( ν b ) ≥ q d b ( 1 − b ) ( b 2 + ( 1 − b ) 2 ) r 3 . (88)
Proof. Identify the optimizer. Define on all of R d \mathbb{R}^d R d
Φ c ( x ) = sup y ∈ X { x ⋅ y − 1 2 ∣ y ∣ 2 − c ( y ) } . \Phi_c(x)=\sup_{y\in X}\left\{x\cdot y-\frac{1}{2}|y|^2-c(y)\right\}. Φ c ( x ) = y ∈ X sup { x ⋅ y − 2 1 ∣ y ∣ 2 − c ( y ) } .
This convex Lipschitz function is differentiable almost everywhere, and its gradients lie in X X X . At a differentiability point its maximizing destination is unique and equals T c ( x ) = ∇ Φ c ( x ) T_c(x)=\nabla\Phi_c(x) T c ( x ) = ∇ Φ c ( x ) . Define T c T_c T c arbitrarily on the null exceptional set. For every source–destination pair,
1 2 ∣ x − y ∣ 2 + c ( y ) ≥ 1 2 ∣ x ∣ 2 − Φ c ( x ) . \frac{1}{2}|x-y|^2+c(y)\ge\frac{1}{2}|x|^2-\Phi_c(x). 2 1 ∣ x − y ∣ 2 + c ( y ) ≥ 2 1 ∣ x ∣ 2 − Φ c ( x ) .
The coupling ( id , T c ) # λ (\operatorname{id},T_c)_\#\lambda ( id , T c ) # λ attains equality. Integrating against an optimal coupling for any candidate row gives
ν c = ( T c ) # λ , V ( c ) = ∫ X ( 1 2 ∣ x ∣ 2 − Φ c ( x ) ) d λ . (89) \nu_c=(T_c)_\#\lambda,\qquad V(c)=\int_X\left(\frac{1}{2}|x|^2-\Phi_c(x)\right)\,\mathrm{d}\lambda. \tag*{(89)} ν c = ( T c ) # λ , V ( c ) = ∫ X ( 2 1 ∣ x ∣ 2 − Φ c ( x ) ) d λ . (89)
Every minimizing coupling must have destination T c ( x ) T_c(x) T c ( x ) for almost every source, proving uniqueness.
Control the response to a cost perturbation. If Φ , Ψ \Phi,\Psi Φ , Ψ are smooth convex functions with gradients in X X X , put u = Φ − Ψ u=\Phi-\Psi u = Φ − Ψ and e = ∥ u ∥ L ∞ ( X ) e=\|u\|_{L^\infty(X)} e = ∥ u ∥ L ∞ ( X ) . Integration by parts on the cube gives
∫ X ∣ ∇ u ∣ 2 d x = ∫ ∂ X u ∂ n u d S − ∫ X u ( Δ Φ − Δ Ψ ) d x ≤ 4 d e . \int_X|\nabla u|^2\,\mathrm{d}x=\int_{\partial X}u\,\partial_nu\,\mathrm{d}S-\int_Xu(\Delta\Phi-\Delta\Psi)\,\mathrm{d}x\le4de. ∫ X ∣∇ u ∣ 2 d x = ∫ ∂ X u ∂ n u d S − ∫ X u ( ΔΦ − ΔΨ ) d x ≤ 4 d e .
Indeed, each gradient component belongs to [ 0 , 1 ] [0,1] [ 0 , 1 ] , so the boundary integral of ∣ ∂ n u ∣ |\partial_nu| ∣ ∂ n u ∣ is at most 2 d 2d 2 d . Convexity and the divergence theorem give 0 ≤ ∫ X Δ Φ ≤ d 0\le\int_X\Delta\Phi\le d 0 ≤ ∫ X ΔΦ ≤ d and the same bound for Ψ \Psi Ψ . For Φ c , Φ ˉ c , \Phi_c,\bar{\Phi}_c, Φ c , Φ ˉ c , global mollification preserves convexity, the gradient range, and the bound ∥ Φ c − Φ ~ c ∥ L ∞ ( R d ) ≤ ∥ c − c ~ ∥ ∞ \lVert\Phi_c-\widetilde{\Phi}_c\rVert_{L^\infty(\mathbb{R}^d)} \le\lVert c-\widetilde{c}\rVert_\infty ∥ Φ c − Φ c ∥ L ∞ ( R d ) ≤ ∥ c − c ∥ ∞ . Dominated convergence of the mollified gradients therefore yields
W 1 ( ν c , ν c ~ ) ≤ W 2 ( ν c , ν c ~ ) ≤ 2 d ∥ c − c ~ ∥ ∞ 1 / 2 . (90) W_1(\nu_c,\nu_{\widetilde{c}}) \le W_2(\nu_c,\nu_{\widetilde{c}}) \le2\sqrt{d}\lVert c-\widetilde{c}\rVert_\infty^{1/2}. \tag*{(90)} W 1 ( ν c , ν c ) ≤ W 2 ( ν c , ν c ) ≤ 2 d ∥ c − c ∥ ∞ 1/2 . (90)
Turn response control into growth. Set r = W 1 ( ν , ν c ) r=W_1(\nu,\nu_c) r = W 1 ( ν , ν c ) and choose a 1-Lipschitz Kantorovich–Rubinstein test h h h with ∫ h d ( ν − ν c ) = r \int h\,\mathrm{d}(\nu-\nu_c)=r ∫ h d ( ν − ν c ) = r and ∥ h ∥ ∞ ≤ D \lVert h\rVert_\infty\le D ∥ h ∥ ∞ ≤ D . Testing the minimizers at neighboring costs in each other’s objectives, then using (86), gives
d d t V ( c − t h ) = − ∫ h d ν c − t h . \frac{\mathrm{d}}{\mathrm{d}t}V(c-th)=-\int h\,\mathrm{d}\nu_{c-th}. d t d V ( c − t h ) = − ∫ h d ν c − t h .
The derivative is continuous by (86). Integrating it and using Lip ( h ) ≤ 1 \operatorname{Lip}(h)\le1 Lip ( h ) ≤ 1 gives
V ( c − t h ) ≥ V ( c ) − t ∫ h d ν c − 4 3 d D t 3 / 2 . V(c-th)\ge V(c)-t\int h\,\mathrm{d}\nu_c-\frac{4}{3}\sqrt{dD}\,t^{3/2}. V ( c − t h ) ≥ V ( c ) − t ∫ h d ν c − 3 4 d D t 3/2 .
On the other hand, V ( c − t h ) ≤ J 0 , c ( ν ) − t ∫ h d ν V(c-th)\le J_{0,c}(\nu)-t\int h\,\mathrm{d}\nu V ( c − t h ) ≤ J 0 , c ( ν ) − t ∫ h d ν . Hence
J 0 , c ( ν ) − V ( c ) ≥ t r − 4 3 d D t 3 / 2 . J_{0,c}(\nu)-V(c)\ge tr-\frac{4}{3}\sqrt{dD}\,t^{3/2}. J 0 , c ( ν ) − V ( c ) ≥ t r − 3 4 d D t 3/2 .
Choosing t = r 2 / ( 4 d D ) t=r^2/(4dD) t = r 2 / ( 4 d D ) proves (83); the case r = 0 r=0 r = 0 follows from minimality.
Finally, compact Kantorovich duality supplies a continuous destination potential b ∗ b_* b ∗ supporting F F F at ν b \nu_b ν b : F ( ν ) ≥ F ( ν b ) + ∫ b ∗ d ( ν − ν b ) F(\nu)\ge F(\nu_b)+\int b_*\,\mathrm{d}(\nu-\nu_b) F ( ν ) ≥ F ( ν b ) + ∫ b ∗ d ( ν − ν b ) . Thus ν b \nu_b ν b minimizes J 0 , − b ∗ J_{0,-b_*} J 0 , − b ∗ . Apply (83) to ν 0 , ν 1 \nu_0,\nu_1 ν 0 , ν 1 and add with weights 1 − b , b 1-b,b 1 − b , b . The linear terms cancel, while the dual norm formula for W 1 W_1 W 1 gives W 1 ( ν 0 , ν b ) = b r W_1(\nu_0,\nu_b)=br W 1 ( ν 0 , ν b ) = b r and W 1 ( ν 1 , ν b ) = ( 1 − b ) r W_1(\nu_1,\nu_b)=(1-b)r W 1 ( ν 1 , ν b ) = ( 1 − b ) r . This proves (84).
Existence for every α \alpha α follows from weak lower semicontinuity on compact P ( X ) \mathcal{P}(X) P ( X ) and the finite competitor λ \lambda λ . When the endpoint entropies are finite, linear interpolation has the identity
( 1 − b ) H ( ν 0 ) + b H ( ν 1 ) − H ( ν b ) = ( 1 − b ) KL ( ν 0 ∥ ν b ) + b KL ( ν 1 ∥ ν b ) . (1-b)H(\nu_0)+bH(\nu_1)-H(\nu_b)=(1-b)\operatorname{KL}(\nu_0\Vert\nu_b)+b\operatorname{KL}(\nu_1\Vert\nu_b). ( 1 − b ) H ( ν 0 ) + b H ( ν 1 ) − H ( ν b ) = ( 1 − b ) KL ( ν 0 ∥ ν b ) + b KL ( ν 1 ∥ ν b ) .
Pinsker’s inequality and W 1 ≤ ( D / 2 ) ∥ ⋅ ∥ T V W_1\le(D/2)\lVert\cdot\rVert_{\mathrm{TV}} W 1 ≤ ( D /2 ) ∥ ⋅ ∥ TV , with ∥ ⋅ ∥ T V \lVert\cdot\rVert_{\mathrm{TV}} ∥ ⋅ ∥ TV the total variation norm, bound this below by 2 b ( 1 − b ) W 1 ( ν 0 , ν 1 ) 2 / D 2 2b(1-b)W_1(\nu_0,\nu_1)^2/D^2 2 b ( 1 − b ) W 1 ( ν 0 , ν 1 ) 2 / D 2 . Combine this with (84), take ν 0 \nu_0 ν 0 to be a minimizer, divide the interpolation deficit by b b b , and let b ↓ 0 b\downarrow0 b ↓ 0 . For every finite-objective competitor, with r = W 1 ( ν , ν α , c ) r=W_1(\nu,\nu_{\alpha,c}) r = W 1 ( ν , ν α , c ) , this proves
J α , c ( ν ) − J α , c ( ν α , c ) ≥ ( 1 − α ) q d r 3 + 2 α τ D 2 r 2 ≥ m d , τ r 3 . (91) J_{\alpha,c}(\nu)-J_{\alpha,c}(\nu_{\alpha,c})\ge(1-\alpha)q_d r^3+\frac{2\alpha\tau}{D^2}r^2\ge m_{d,\tau}r^3. \tag*{(91)} J α , c ( ν ) − J α , c ( ν α , c ) ≥ ( 1 − α ) q d r 3 + D 2 2 α τ r 2 ≥ m d , τ r 3 . (91)
At α = 0 \alpha=0 α = 0 no entropy assertion is used. The last inequality uses r ≤ D r\le D r ≤ D ; it also proves uniqueness. Apply (87) in both directions to the minimizers for c c c and c ~ \widetilde{c} c . Adding cancels the regularizer, so that
W 1 ( ν α , c , ν α , c ~ ) ≤ m d , τ − 1 / 3 ∥ c − c ~ ∥ ∞ 1 / 3 , 0 ≤ α ≤ 1. (92) W_1(\nu_{\alpha,c},\nu_{\alpha,\widetilde{c}})\le m_{d,\tau}^{-1/3}\lVert c-\widetilde{c}\rVert_\infty^{1/3},\qquad0\le\alpha\le1. \tag*{(92)} W 1 ( ν α , c , ν α , c ) ≤ m d , τ − 1/3 ∥ c − c ∥ ∞ 1/3 , 0 ≤ α ≤ 1. (92)
This proves the spatial estimate with no positive lower bound on the entropy coefficient.
Continuity in the entropy coefficient. If Lip ( c ) ≤ L y \operatorname{Lip}(c)\le L_y Lip ( c ) ≤ L y , then
W 1 ( ν α , c , ν α ~ , c ) ≤ C d , L y , τ ω ( ∣ α − α ~ ∣ ) 1 / 3 for all α , α ~ ∈ [ 0 , 1 ] . (93) W_1(\nu_{\alpha,c},\nu_{\widetilde{\alpha},c})\le C_{d,L_y,\tau}\omega(|\alpha-\widetilde{\alpha}|)^{1/3}\quad\text{for all }\alpha,\widetilde{\alpha}\in[0,1]. \tag*{(93)} W 1 ( ν α , c , ν α , c ) ≤ C d , L y , τ ω ( ∣ α − α ∣ ) 1/3 for all α , α ∈ [ 0 , 1 ] . (93)
Proof. Approximate the singular endpoint. Divide X X X into q d q^d q d cubes and let S q ν S_q\nu S q ν spread each cell’s mass uniformly within that cell. Then
W 1 ( S q ν , ν ) ≤ D / q , H ( S q ν ) ≤ d log q , H ( S q ν ) ≤ H ( ν ) (94) W_1(S_q\nu,\nu)\le D/q,\qquad H(S_q\nu)\le d\log q,\qquad H(S_q\nu)\le H(\nu) \tag*{(94)} W 1 ( S q ν , ν ) ≤ D / q , H ( S q ν ) ≤ d log q , H ( S q ν ) ≤ H ( ν ) (94)
when the last right-hand side is finite. The entropy statements follow from the cell probabilities and Jensen’s inequality. Also
∣ F ( ν ) − F ( ν ′ ) ∣ ≤ D W 1 ( ν , ν ′ ) . (95) \lvert F(\nu)-F(\nu')\rvert\le DW_1(\nu,\nu'). \tag*{(95)} ∣ F ( ν ) − F ( ν ′ )∣ ≤ D W 1 ( ν , ν ′ ) . (95)
To see this, glue an optimal coupling from λ \lambda λ to ν \nu ν to a coupling between ν \nu ν and ν ′ \nu' ν ′ , and use that ∣ x − y ∣ 2 / 2 \lvert x-y\rvert^2/2 ∣ x − y ∣ 2 /2 is D D D -Lipschitz in y y y on X X X ; then interchange the targets. Set C 0 = D ( L y + D ) C_0=D(L_y+D) C 0 = D ( L y + D ) and V ( c ) = J 0 , c ( ν c ) V(c)=J_{0,c}(\nu_c) V ( c ) = J 0 , c ( ν c ) . Testing S q ν c S_q\nu_c S q ν c in the α \alpha α -problem, using 0 ≤ F ≤ D 2 / 2 0\le F\le D^2/2 0 ≤ F ≤ D 2 /2 and H ≥ 0 H\ge0 H ≥ 0 , gives
J 0 , c ( ν α , c ) − V ( c ) + α τ H ( ν α , c ) ≤ C 0 / q + α τ d log q + α D 2 / 2. (96) J_{0,c}(\nu_{\alpha,c})-V(c)+\alpha\tau H(\nu_{\alpha,c})\le C_0/q+\alpha\tau d\log q+\alpha D^2/2. \tag*{(96)} J 0 , c ( ν α , c ) − V ( c ) + α τ H ( ν α , c ) ≤ C 0 / q + α τ d log q + α D 2 /2. (96)
For α > 0 \alpha>0 α > 0 , take q = ⌈ 1 / α ⌉ q=\lceil1/\alpha\rceil q = ⌈ 1/ α ⌉ . The transport growth estimate and the nonnegativity of J 0 , c − V ( c ) J_{0,c}-V(c) J 0 , c − V ( c ) imply, respectively,
W 1 ( ν α , c , ν c ) 3 ≤ α q d ( C 0 + D 2 / 2 + τ d log ( 2 / α ) ) . (97) W_1(\nu_{\alpha,c},\nu_c)^3\le\frac{\alpha}{q d}\left(C_0+D^2/2+\tau d\log(2/\alpha)\right). \tag*{(97)} W 1 ( ν α , c , ν c ) 3 ≤ q d α ( C 0 + D 2 /2 + τ d log ( 2/ α ) ) . (97)
H ( ν α , c ) ≤ B 0 + d log ( 2 / α ) , B 0 = ( C 0 + D 2 / 2 ) / τ . (98) H(\nu_{\alpha,c})\le B_0+d\log(2/\alpha),\qquad B_0=(C_0+D^2/2)/\tau. \tag*{(98)} H ( ν α , c ) ≤ B 0 + d log ( 2/ α ) , B 0 = ( C 0 + D 2 /2 ) / τ . (98)
Compare two positive coefficients. By symmetry assume 0 ≤ α ≤ α ~ ≤ 1 0\le\alpha\le\tilde{\alpha}\le1 0 ≤ α ≤ α ~ ≤ 1 , and put h = α ~ − α h=\tilde{\alpha}-\alpha h = α ~ − α . For α ≥ h > 0 \alpha\ge h>0 α ≥ h > 0 , write J α ~ , c = J α , c + h ( τ H − F ) J_{\tilde{\alpha},c}=J_{\alpha,c}+h(\tau H-F) J α ~ , c = J α , c + h ( τ H − F ) and test each minimizer in the other’s objective. (91) bounds the sum of the two gaps below by 2 m d , τ W 1 ( ν α , c , ν α ~ , c ) 3 2m_{d,\tau}W_1(\nu_{\alpha,c},\nu_{\tilde{\alpha},c})^3 2 m d , τ W 1 ( ν α , c , ν α ~ , c ) 3 . The sum equals
h [ ( τ H − F ) ( ν α , c ) − ( τ H − F ) ( ν α ~ , c ) ] ≤ h [ τ B 0 + τ d log ( 2 / h ) + D 2 / 2 ] , h\left[(\tau H-F)(\nu_{\alpha,c})-(\tau H-F)(\nu_{\tilde{\alpha},c})\right]\le h\left[\tau B_0+\tau d\log(2/h)+D^2/2\right], h [ ( τ H − F ) ( ν α , c ) − ( τ H − F ) ( ν α ~ , c ) ] ≤ h [ τ B 0 + τ d log ( 2/ h ) + D 2 /2 ] ,
by (94). This proves the required modulus in this regime. If 0 < α < h 0<\alpha<h 0 < α < h and h < 1 / 4 h<1/4 h < 1/4 , both coefficients are below 2 h 2h 2 h ; compare each row to ν c \nu_c ν c using (93). If h ≥ 1 / 4 h\ge1/4 h ≥ 1/4 , use W 1 ≤ D W_1\le D W 1 ≤ D . The case h = 0 h=0 h = 0 is immediate.
Architecture examples
First control the graph lift. Put δ = W 1 ( γ u , γ v ) \delta=W_1(\gamma_u,\gamma_v) δ = W 1 ( γ u , γ v ) and L j = 1 + L u 2 L_j=\sqrt{1+L_u^2} L j = 1 + L u 2 . A coupling of the graph measures projects to a coupling of λ \lambda λ with itself. At a paired location ( x , y ) (x,y) ( x , y ) ,
∣ u ( x ) − v ( x ) ∣ ≤ ∣ u ( x ) − v ( y ) ∣ + L u ∣ x − y ∣ ≤ L j ρ E ( j u ( x ) , j v ( y ) ) . |u(x)-v(x)|\le|u(x)-v(y)|+L_u|x-y|\le L_j\rho_E(j_u(x),j_v(y)). ∣ u ( x ) − v ( x ) ∣ ≤ ∣ u ( x ) − v ( y ) ∣ + L u ∣ x − y ∣ ≤ L j ρ E ( j u ( x ) , j v ( y )) .
Integration and minimization give ∫ X ∣ u − v ∣ d λ ≤ L j δ \int_X|u-v|\,d\lambda\le L_j\delta ∫ X ∣ u − v ∣ d λ ≤ L j δ . Put M u v = ∥ u − v ∥ ∞ M_{uv}=\|u-v\|_\infty M uv = ∥ u − v ∥ ∞ and B ∗ = 1 + 2 L u + diam V B_* =1+2L_u+\operatorname{diam}V B ∗ = 1 + 2 L u + diam V . At a maximizer, ∣ u − v ∣ ≥ M u v / 2 |u-v|\ge M_{uv}/2 ∣ u − v ∣ ≥ M uv /2 within distance M u v / ( 2 B ∗ ) ≤ 1 / 2 M_{uv}/(2B_*)\le1/2 M uv / ( 2 B ∗ ) ≤ 1/2 . The intersection of that ball with the cube has volume bounded below by a dimensional constant times its radius to the power d d d . Thus
∥ u − v ∥ ∞ ≤ C G δ 1 / ( d + 1 ) , C G = C G ( d , L u , diam V ) . (99) \|u-v\|_\infty\le C_G\delta^{1/(d+1)},\qquad C_G=C_G(d,L_u,\operatorname{diam}V). \tag*{(99)} ∥ u − v ∥ ∞ ≤ C G δ 1/ ( d + 1 ) , C G = C G ( d , L u , diam V ) . (99)
For fixed ( α , c ) (\alpha,c) ( α , c ) , apply (88) to the two functions y ↦ c ( γ u , z , j u ( y ) ) y\mapsto c(\gamma_u,z,j_u(y)) y ↦ c ( γ u , z , j u ( y )) and y ↦ c ( γ v , z ′ , j v ( y ) ) y\mapsto c(\gamma_v,z',j_v(y)) y ↦ c ( γ v , z ′ , j v ( y )) . Couple their minimizing spatial measures, then change the graph lift at the same position. Equations (78) and (95) yield
W 1 ( K α , c [ γ u , z ] , K α , c [ γ v , z ′ ] ) ≤ L j ( L c / m d , τ ) 1 / 3 ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) β / 3 + C G δ 1 / ( d + 1 ) . W_1(K_{\alpha,c}[\gamma_u,z],K_{\alpha,c}[\gamma_v,z'])
\le L_j(L_c/m_{d,\tau})^{1/3}\left(W_1(\gamma_u,\gamma_v)+\rho_E(z,z')\right)^{\beta/3}+C_G\delta^{1/(d+1)}. W 1 ( K α , c [ γ u , z ] , K α , c [ γ v , z ′ ]) ≤ L j ( L c / m d , τ ) 1/3 ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) β /3 + C G δ 1/ ( d + 1 ) .
All input distances are bounded, so this is at most a constant times ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) η (W_1(\gamma_u,\gamma_v)+\rho_E(z,z'))^\eta ( W 1 ( γ u , γ v ) + ρ E ( z , z ′ ) ) η . At fixed ( γ v , z ′ ) (\gamma_v,z') ( γ v , z ′ ) , combine (88), the coefficient estimate (89), and the Lipschitz pushforward bound to control the change from ( α , c ) (\alpha,c) ( α , c ) to ( α ~ , c ~ ) (\tilde{\alpha},\tilde{c}) ( α ~ , c ~ ) . The triangle inequality with intermediate row K α , c [ γ v , z ′ ] K_{\alpha,c}[\gamma_v,z'] K α , c [ γ v , z ′ ] proves (81).
The class U L u U_{L_u} U L u is compact in the uniform norm by Arzelà–Ascoli; u ↦ γ u u\mapsto\gamma_u u ↦ γ u is continuous and injective, since λ \lambda λ has full support and u u u is continuous. Hence the set of admissible triples ( γ u , z , j u ( y ) ) (\gamma_u,z,j_u(y)) ( γ u , z , j u ( y )) is compact. The bounds in (78) make C \mathcal C C convex, closed, bounded, and equicontinuous there, so it is compact. Existence and uniqueness were proved in (87); pushing the spatial optimizer through j u j_u j u gives the unique token row. Continuity implies measurability. Compactness is asserted for the cost–coefficient coordinates, not for uniform values of extended-valued regularizers at $\alpha=0.
For graph contexts, write
F μ ( κ ) : = 1 2 W 2 2 ( ( π X ) # κ , ( π X ) # μ ) , R α ( κ , μ ) : = ( 1 − α ) F μ ( κ ) + α τ KL ( κ ∥ μ ) . F_\mu(\kappa):=\tfrac12W_2^2((\pi_X)_\#\kappa,(\pi_X)_\#\mu),\qquad R_\alpha(\kappa,\mu):=(1-\alpha)F_\mu(\kappa)+\alpha\tau\operatorname{KL}(\kappa\|\mu). F μ ( κ ) := 2 1 W 2 2 (( π X ) # κ , ( π X ) # μ ) , R α ( κ , μ ) := ( 1 − α ) F μ ( κ ) + α τ KL ( κ ∥ μ ) .
tbl:2 records the three architecture reads in this family and the separate Pearson construction from the representation theorem. The latter is proved in Section B.2.2; its decoder costs need not belong to the joint cost class C \mathcal C C .
caption: tbl:2 : Variational read constructions. The first three rows belong to the Wasserstein–KL family on the stated graph class, with an admissible Hölder address a ( μ , z ) = π X ( z ) + g ( μ , z ) ∈ X a(\mu,z)=\pi_X(z)+g(\mu,z)\in X a ( μ , z ) = π X ( z ) + g ( μ , z ) ∈ X for SelfWarp. The last row is the separate Pearson realization; its calibration head has zero cost.
At α = 1 \alpha=1 α = 1 , put Z = ∫ X e − c / τ d λ Z=\int_X e^{-c/\tau}\,d\lambda Z = ∫ X e − c / τ d λ . Direct expansion gives
∫ X c d ν + τ H ( ν ) = τ KL ( ν ∥ Z − 1 e − c / τ λ ) − τ log Z . \int_X c\,d\nu+\tau H(\nu)=\tau\operatorname{KL}\left(\nu\mathbin\Vert Z^{-1}e^{-c/\tau}\lambda\right)-\tau\log Z. ∫ X c d ν + τ H ( ν ) = τ KL ( ν ∥ Z − 1 e − c / τ λ ) − τ log Z .
This proves the normalized attention row in Table 2, including extended values outside the entropy domain. For the SelfWarp cost in the table, (95) gives
∫ X B ∣ y − π X z − g ( γ u , z ) ∣ ν ( d y ) + F ( ν ) − F ( δ π X z + g ( γ u , z ) ) ≥ ( B − D ) W 1 ( ν , δ π X z + g ( γ u , z ) ) . \int_X B\lvert y-\pi_Xz-g(\gamma_u,z)\rvert\,\nu(dy)+F(\nu)-F\left(\delta_{\pi_Xz+g(\gamma_u,z)}\right)
\ge(B-D)W_1\left(\nu,\delta_{\pi_Xz+g(\gamma_u,z)}\right). ∫ X B ∣ y − π X z − g ( γ u , z )∣ ν ( d y ) + F ( ν ) − F ( δ π X z + g ( γ u , z ) ) ≥ ( B − D ) W 1 ( ν , δ π X z + g ( γ u , z ) ) .
Therefore B > D B>D B > D suffices for the unique spatial optimizer δ π X z + g ( γ u , z ) \delta_{\pi_Xz+g(\gamma_u,z)} δ π X z + g ( γ u , z ) . It need not be necessary for a particular address. For zero cost, both F F F and H H H are nonnegative and vanish only at λ \lambda λ , so every α \alpha α gives K [ γ u , z ] = γ u K[\gamma_u,z]=\gamma_u K [ γ u , z ] = γ u . At α = 0 \alpha=0 α = 0 , the spatial optimizer also has the transport-map description (85). The point-read identification agrees with Proposition B.3.
A bounded Lipschitz attention score in ( z , w ) (z,w) ( z , w ) is spatially Lipschitz after substitution w = j u ( y ) w=j_u(y) w = j u ( y ) . (99) controls its dependence on u u u , giving (78), for example with β = 1 / ( d + 1 ) \beta=1/(d+1) β = 1/ ( d + 1 ) . For the warp costs,
∣ B ∣ y − π X z − g ( γ u , z ) ∣ − B ∣ y ′ − π X z ′ − g ( γ v , z ′ ) ∣ ∣ ≤ B ( ∣ y − y ′ ∣ + ρ E ( z , z ′ ) + ∣ g ( γ u , z ) − g ( γ v , z ′ ) ∣ ) . \left|B\lvert y-\pi_Xz-g(\gamma_u,z)\rvert-B\lvert y'-\pi_Xz'-g(\gamma_v,z')\rvert\right|
\le B\left(\lvert y-y'\rvert+\rho_E(z,z')+\lvert g(\gamma_u,z)-g(\gamma_v,z')\rvert\right). ∣ B ∣ y − π X z − g ( γ u , z )∣ − B ∣ y ′ − π X z ′ − g ( γ v , z ′ )∣ ∣ ≤ B ( ∣ y − y ′ ∣ + ρ E ( z , z ′ ) + ∣ g ( γ u , z ) − g ( γ v , z ′ )∣ ) .
A Hölder displacement map therefore gives an admissible cost. Choose a common exponent no larger than 1 / ( d + 1 ) 1/(d+1) 1/ ( d + 1 ) and the displacement exponent, and common bounds covering both families. Then the endpoints and their straight interpolation belong to C × [ 0 , 1 ] \mathcal{C}\times[0,1] C × [ 0 , 1 ] .
Grid consistency and the obstruction from changing supports
The graph theorem does not by itself certify optimization on empirical contexts. The following approximation statement supplies a separate numerical guarantee. Partition X X X into cubes Q i Q_i Q i of side h = 1 / q h=1/q h = 1/ q , with centers x i x_i x i and weights w i = q − d w_i=q^{-d} w i = q − d . Let λ h = ∑ i w i δ x i \lambda_h=\sum_i w_i\delta_{x_i} λ h = ∑ i w i δ x i . For a probability vector p = ( p i ) p=(p_i) p = ( p i ) , set ν h ( p ) = ∑ i p i δ x i \nu_h(p)=\sum_i p_i\delta_{x_i} ν h ( p ) = ∑ i p i δ x i and
J α , c ^ h ( p ) = ∑ i c ^ i p i + 1 − α 2 W 2 2 ( λ h , ν h ( p ) ) + α τ ∑ i p i log ( p i / w i ) . (100) \mathcal{J}^{h}_{\alpha,\widehat c}(p)=\sum_i\widehat c_i p_i+\frac{1-\alpha}{2}W_2^2(\lambda_h,\nu_h(p))+\alpha\tau\sum_i p_i\log(p_i/w_i). \tag*{(100)} J α , c h ( p ) = i ∑ c i p i + 2 1 − α W 2 2 ( λ h , ν h ( p )) + α τ i ∑ p i log ( p i / w i ) . (100)
This is a convex problem. Its entropy acts on the destination row, not on the transport coupling.
Uniform grid consistency. Suppose Lip ( c ) ≤ L y \operatorname{Lip}(c)\le L_y Lip ( c ) ≤ L y , max i ∣ c ^ i − c ( x i ) ∣ ≤ e c \max_i\lvert\widehat c_i-c(x_i)\rvert\le e_c max i ∣ c i − c ( x i )∣ ≤ e c , and p h p_h p h has objective gap at most e s o l v e e_{\mathrm{solve}} e solve in (100) . Then, for all α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] ,
W 1 ( ν h ( p h ) , ν α , c ) ≤ D h + [ ( 3 D L y + 5 D 2 ) h + 2 e c + e s o l v e m d , τ ] 1 / 3 . (101) W_1(\nu_h(p_h),\nu_{\alpha,c})\le Dh+\left[\frac{(3DL_y+5D^2)h+2e_c+e_{\mathrm{solve}}}{m_{d,\tau}}\right]^{1/3}. \tag*{(101)} W 1 ( ν h ( p h ) , ν α , c ) ≤ D h + [ m d , τ ( 3 D L y + 5 D 2 ) h + 2 e c + e solve ] 1/3 . (101)
For exact samples of u ∈ U L u u\in\mathcal{U}_{L_u} u ∈ U L u , pushing the row to j u ( x i ) j_u(x_i) j u ( x i ) multiplies this bound by at most 1 + L u 2 \sqrt{1+L_u^2} 1 + L u 2 .
Proof. Lift p p p to the histogram L h p = ∑ i ( p i / w i ) 1 Q i λ L_hp = \sum_i (p_i/w_i)1_{Q_i}\lambda L h p = ∑ i ( p i / w i ) 1 Q i λ . Its entropy equals the discrete entropy, and W 1 ( L h p , ν h ( p ) ) ≤ D h W_1(L_hp,\nu_h(p)) \le D_h W 1 ( L h p , ν h ( p )) ≤ D h . Coupling source and destination points to their cell centers shows that the transport terms differ by at most 2 D h 2 2D_h^2 2 D h 2 ; the cost terms differ by at most D L y h + e c D L_y h + e_c D L y h + e c . Thus the continuum and discrete objectives differ by at most C 1 h + e c C_1h + e_c C 1 h + e c , where C 1 = D L y + 2 D 2 C_1 = D L_y + 2D^2 C 1 = D L y + 2 D 2 .
Take as a discrete competitor the cell masses of ν α , c \nu_{\alpha,c} ν α , c . Its histogram is S q ν α , c S_q\nu_{\alpha,c} S q ν α , c , whose continuum objective is at most the minimum plus C 0 h C_0h C 0 h , with C 0 = D L y + D 2 C_0 = D L_y + D^2 C 0 = D L y + D 2 , by (94) and (95) . When α = 0 \alpha= 0 α = 0 no entropy comparison is needed. Comparing the discrete optimizer to this competitor gives a continuum objective gap for L h p h L_hp_h L h p h of at most
( C 0 + 2 C 1 ) h + 2 e c + e s o l v e . (C_0 + 2C_1)h + 2e_c + e_{\mathrm{solve}}. ( C 0 + 2 C 1 ) h + 2 e c + e solve .
Apply (91) and add the lifting distance D h D_h D h . This proves (101) , for every approximate minimizer, including nonunique discrete minimizers at α = 0 \alpha= 0 α = 0 .
Why non-collapse is insufficient. Take the ambient interval E = X = [ 0 , 2 ] E = X = [0,2] E = X = [ 0 , 2 ] , π X = id \pi_X = \operatorname{id} π X = id , the fixed cost c ( w ) = − 3 w c(w) = -3w c ( w ) = − 3 w , and
μ ε = 1 − ε 2 ( δ 0 + δ 1 ) + ε δ 2 , 0 ≤ ε ≤ 1 4 . \mu_\varepsilon= \frac{1-\varepsilon}{2}(\delta_0+\delta_1)+\varepsilon\delta_2,\qquad0\le\varepsilon\le\frac14. μ ε = 2 1 − ε ( δ 0 + δ 1 ) + ε δ 2 , 0 ≤ ε ≤ 4 1 .
These measures remain a positive distance from every Dirac measure and W 1 ( μ ε , μ 0 ) = 3 ε / 2 W_1(\mu_\varepsilon,\mu_0)=3\varepsilon/2 W 1 ( μ ε , μ 0 ) = 3 ε /2 . At α = 0 \alpha=0 α = 0 , the source–destination objective is − 3 y + ∣ x − y ∣ 2 / 2 -3y+\lvert x-y\rvert^2/2 − 3 y + ∣ x − y ∣ 2 /2 , strictly decreasing in the available destination y ∈ [ 0 , 2 ] y\in[0,2] y ∈ [ 0 , 2 ] . Its unique minimizing row is therefore δ 2 \delta_2 δ 2 for ε > 0 \varepsilon>0 ε > 0 , but δ 1 \delta_1 δ 1 for ε = 0 \varepsilon=0 ε = 0 . Thus the input distance tends to zero while the distance between the minimizing reads remains one. The theorem therefore cannot extend to arbitrary non-collapsing contexts. The grid bound concerns a specified approximation of the continuum problem.