Research Thesis
in partial fulfillment of the requirements for the degree of Doctor of Philosophy
Submitted to the Senate of the Technion — Israel Institute of Technology
Tishrei 5787 Haifa September 2026
The Research Thesis Was Done in the Interdisciplinary Program in Applied Mathematics Under the Supervision of Prof. Nadav Dym.
Parts of the research presented in this thesis have appeared in the publications listed below. For each publication, its status and the author’s contribution are stated.
Yonatan Sverdlov and Nadav Dym. On the expressive power of sparse geometric MPNNs. In International Conference on Learning Representations (ICLR), 2025. Status: Published at ICLR 2025. Yonatan Sverdlov’s contribution: Yonatan was involved in the work throughout the project, from its initial stages to the final version. He contributed to the development and proofs of the theoretical results, developed and implemented the code, and contributed substantially to the scientific writing of the paper.
Yonatan Sverdlov, Ido Springer, and Nadav Dym. Revisiting multi-permutation equivariance through the lens of irreducible representations. In International Conference on Learning Representations (ICLR), 2025. Status: Published at ICLR 2025. Yonatan Sverdlov and Ido Springer contributed equally to this work. Yonatan contributed to the development and proofs of the theoretical results, developed and implemented the code, and contributed substantially to the scientific writing of the paper.
Soutrik Sarangi, Yonatan Sverdlov, Nadav Dym, and Abir De. Monotone and separable set functions: characterizations and neural models. In Advances in Neural Information Processing Systems (NeurIPS), 2025. Status: Published at NeurIPS 2025. Soutrik Sarangi and Yonatan Sverdlov contributed equally to this work. Yonatan was substantially involved in both the theoretical and computational aspects of the work. He contributed to the development and proofs of the theoretical results, to the development and implementation of the code, and to the scientific writing of the paper.
Yonatan Sverdlov, Yair Davidson, Nadav Dym, and Tal Amir. FSW-GNN: a bi-Lipschitz WL-equivalent graph neural network. In Learning on Graphs Conference (LoG), 2025. Status: Published at LoG 2025. Yonatan was substantially involved in both the theoretical and computational aspects of the work. He contributed to the development and proofs of the theoretical results, to the development and implementation of the code, and to the scientific writing of the paper.
The author of this thesis states that the research, including the collection, processing and presentation of data, addressing and comparing to previous research, etc., was done entirely in an honest way, as expected from scientific research that is conducted according to the ethical standards of the academic world. Also, reporting the research and its results in this thesis was done in an honest and complete manner, according to the same standards.
The generous financial help of the Technion and the Israeli Science Foundation grant no. 272/23 is gratefully acknowledged.
Notation and Abbreviations
Table 1.
G \mathcal{G} G
A symmetry group acting on a feature space in the equivariance chapters. In the FSW-GNN paper, G \mathcal{G} G is locally reused for a graph space
g ⋅ x g \cdot x g ⋅ x
Action of a group element g ∈ G g \in\mathcal{G} g ∈ G on an element x x x of the representation space
U , V , W \mathcal{U}, \mathcal{V}, \mathcal{W} U , V , W
Finite-dimensional vector spaces carrying group representations
V G \mathcal{V}^{\mathcal{G}} V G
Fixed-point subspace of a representation: vectors v ∈ V v \in\mathcal{V} v ∈ V satisfying g ⋅ v = v g \cdot v = v g ⋅ v = v for all g ∈ G g \in\mathcal{G} g ∈ G
S n S_n S n
Symmetric group of permutations of n n n elements
G ≀ S k \mathcal{G} \wr S_k G ≀ S k
Wreath-product symmetry for a set of k k k unaligned objects, each carrying a G \mathcal{G} G -symmetry
P P P
Permutation matrix representing a relabeling of elements or graph vertices
O ( d ) O(d) O ( d )
Group of orthogonal transformations of R d \mathbb{R}^{d} R d , including rotations and reflections
x ↦ Q x + t x \mapsto Qx + t x ↦ Q x + t
Rigid transformation with Q ∈ O ( d ) Q \in O(d) Q ∈ O ( d ) and translation t ∈ R d t \in\mathbb{R}^{d} t ∈ R d
G ≅ H G \cong H G ≅ H
Geometric graphs G G G and H H H differ only by vertex relabeling and a rigid transformation
F ( G ) F(G) F ( G )
Invariant finite-dimensional representation of a graph G G G
ρ ( G , H ) \rho(G,H) ρ ( G , H )
Metric or pseudometric measuring the distance between two input graphs
c , C c, C c , C
Bi-Lipschitz constants satisfying 0 < c ≤ C < ∞ 0 < c \le C < \infty 0 < c ≤ C < ∞ and c ρ ( G , H ) ≤ ∥ F ( G ) − F ( H ) ∥ ≤ C ρ ( G , H ) c\rho(G,H) \le\lVert F(G) - F(H) \rVert\le C\rho(G,H) c ρ ( G , H ) ≤ ∥ F ( G ) − F ( H )∥ ≤ C ρ ( G , H )
G ≤ N ( Ω ) \mathcal{G}_{\le N}(\Omega) G ≤ N ( Ω )
Space of vertex-featured graphs with at most N N N vertices and node features in Ω \Omega Ω ; used in the FSW-GNN chapter
G ≡ W L H G \equiv_{\mathrm{WL}} H G ≡ WL H
Graphs G G G and H H H are not distinguished by the corresponding Weisfeiler–Leman procedure
DS
doubly stochastic graph metric (doubly stochastic graph metric)
DWS
Deep Weight Space (Deep Weight Space)
E-GGNN
equivariant geometric graph neural network (equivariant geometric graph neural network)
EGenNet
equivariant geometric message-passing architecture introduced for generic separation (equivariant geometric message-passing architecture introduced for generic separation)
FSW
Fourier Sliced-Wasserstein (Fourier Sliced-Wasserstein)
FSW-GNN
graph neural network using Fourier Sliced-Wasserstein aggregation (graph neural network using Fourier Sliced-Wasserstein aggregation)
GNN
graph neural network (graph neural network)
Preamble
Background: Symmetries, Sets, and Graphs
In this section, we introduce the main notions used throughout the thesis for learning on structured data. We first formalize symmetry through group actions and define invariant and equivariant functions. We then review permutation-invariant set representations, message-passing neural networks, and their connection to the Weisfeiler–Leman test. Finally, we discuss stronger notions of expressivity based on separation and quantitative preservation of information.
Learning with Symmetries
Let G \mathcal{G} G be a group acting on a set X \mathcal{X} X . A group action is a map
G × X → X , ( g , x ) ↦ g ⋅ x , \mathcal{G} \times\mathcal{X} \to\mathcal{X}, \qquad(g,x) \mapsto g \cdot x, G × X → X , ( g , x ) ↦ g ⋅ x ,
satisfying
e ⋅ x = x , g 1 ⋅ ( g 2 ⋅ x ) = ( g 1 g 2 ) ⋅ x , e \cdot x = x, \qquad g_1 \cdot(g_2 \cdot x) = (g_1g_2) \cdot x, e ⋅ x = x , g 1 ⋅ ( g 2 ⋅ x ) = ( g 1 g 2 ) ⋅ x ,
for every x ∈ X x \in\mathcal{X} x ∈ X and g 1 , g 2 ∈ G g_1, g_2 \in\mathcal{G} g 1 , g 2 ∈ G .
The orbit of x x x is
O ( x ) = { g ⋅ x : g ∈ G } . \mathcal{O}(x) = \{g \cdot x : g \in\mathcal{G}\}. O ( x ) = { g ⋅ x : g ∈ G } .
In many learning problems, elements in the same orbit correspond to different representations of the same underlying object.
Suppose that G \mathcal{G} G acts on both an input space X \mathcal{X} X and an output space Y \mathcal{Y} Y . A function f : X → Y f : \mathcal{X} \to\mathcal{Y} f : X → Y is called G \mathcal{G} G -equivariant if
f ( g ⋅ x ) = g ⋅ f ( x ) f(g \cdot x) = g \cdot f(x) f ( g ⋅ x ) = g ⋅ f ( x )
for every g ∈ G g \in\mathcal{G} g ∈ G and x ∈ X x \in\mathcal{X} x ∈ X .
It is called G \mathcal{G} G -invariant if
f ( g ⋅ x ) = f ( x ) . f(g \cdot x) = f(x). f ( g ⋅ x ) = f ( x ) .
for every g ∈ G g \in\mathcal{G} g ∈ G and x ∈ X x \in\mathcal{X} x ∈ X . Thus, an invariant function gives the same output for all equivalent representations of an object, whereas an equivariant function transforms its output consistently with the input. Building these constraints directly into an architecture is a standard way to encode known structure and reduce the need to learn symmetry from data [CW16; RSP17].
A typical example is graph learning. Relabeling the nodes of a graph should not affect a graph-level prediction, such as graph classification, and hence such a prediction should be invariant. On the other hand, node-level predictions should be permuted together with the nodes, and are therefore equivariant.
Symmetry, however, specifies only which inputs must be treated as equivalent. It does not by itself determine whether an architecture can separate inputs that are not related by the symmetry. This distinction between symmetry and expressive power will be central throughout the thesis.
Sets, Multisets, and Permutation-Invariant Embeddings
Consider a collection of vectors
X = ( x 1 , … , x n ) , x i ∈ Ω ⊆ R d . X = (x_1, \ldots, x_n), \qquad x_i \in\Omega\subseteq\mathbb{R}^d. X = ( x 1 , … , x n ) , x i ∈ Ω ⊆ R d .
The symmetric group S n S_n S n acts by permuting the elements:
π ⋅ X = ( x π − 1 ( 1 ) , … , x π − 1 ( n ) ) . \pi\cdot X = (x_{\pi^{-1}(1)}, \ldots, x_{\pi^{-1}(n)}). π ⋅ X = ( x π − 1 ( 1 ) , … , x π − 1 ( n ) ) .
If the ordering carries no information, a function of X X X should be invariant to this action.
We use
S = { { x 1 , … , x n } } S = \{\{x_1, \ldots, x_n\}\} S = {{ x 1 , … , x n }}
to denote a multiset : an unordered collection in which repetitions are allowed. Multisets arise naturally throughout this thesis, particularly when aggregating the features of neighboring nodes in a graph. Let Ω \Omega Ω be a ground set, and denote by M ( Ω ) \mathcal{M}(\Omega) M ( Ω ) the collection of all finite multisets with elements in Ω \Omega Ω .
A multiset embedding is a map
Φ : M ( Ω ) → R m . \Phi: \mathcal{M}(\Omega) \to\mathbb{R}^m. Φ : M ( Ω ) → R m .
Equivalently, multiset functions correspond to permutation-invariant functions on multisets.
A basic and widely used construction is the DeepSets from [ZKR+17]
Φ ( S ) = ρ ( ∑ x ∈ S ϕ ( x ) ) \Phi(S) = \rho\left(\sum_{x \in S} \phi(x)\right) Φ ( S ) = ρ ( x ∈ S ∑ ϕ ( x ) )
where ϕ \phi ϕ and ρ \rho ρ are learned functions. More generally, one can use any permutation-invariant aggregation operator,
Φ ( S ) = ρ ( AGG { { ϕ ( x ) : x ∈ S } } ) . \Phi(S) = \rho\left(\operatorname{AGG}\{\{\phi(x) : x \in S\}\}\right). Φ ( S ) = ρ ( AGG {{ ϕ ( x ) : x ∈ S }} ) .
Graphs and Message Passing Neural Networks
A vertex-featured graph is represented as
G = ( V , E , X ) , G = (V, E, X), G = ( V , E , X ) ,
where
V = { v 1 , … , v n } V = \{v_1, \ldots, v_n\} V = { v 1 , … , v n }
is the vertex set,
E ⊆ { { v i , v j } : i , j ∈ [ n ] } E \subseteq\{\{v_i, v_j\} : i, j \in[n]\} E ⊆ {{ v i , v j } : i , j ∈ [ n ]}
is the edge set, and
X = ( x 1 , … , x n ) , x i ∈ R d , X = (x_1, \ldots, x_n), \qquad x_i \in\mathbb{R}^d, X = ( x 1 , … , x n ) , x i ∈ R d ,
contains the node features.
We denote the neighborhood of a node v v v by
N ( v ) = { u ∈ V : { u , v } ∈ E } . \mathcal{N}(v) = \{u \in V : \{u, v\} \in E\}. N ( v ) = { u ∈ V : { u , v } ∈ E } .
Equivalently, a graph may be represented by an adjacency matrix A ∈ { 0 , 1 } n × n A \in\{0,1\}^{n \times n} A ∈ { 0 , 1 } n × n and a node-feature matrix X X X . Given a permutation matrix P P P , a relabeling of the vertices transforms the graph representation according to
( A , X ) ↦ ( P A P T , P X ) . (A, X) \mapsto(PAP^{\mathsf{T}}, PX). ( A , X ) ↦ ( P A P T , P X ) .
Two graphs are isomorphic when their representations are related by such a relabeling. A graph-level function should therefore satisfy
F ( P A P T , P X ) = F ( A , X ) , F(PAP^{\mathsf{T}}, PX) = F(A, X), F ( P A P T , P X ) = F ( A , X ) ,
whereas a node-level function should satisfy
F ( P A P T , P X ) = P F ( A , X ) . F(PAP^{\mathsf{T}}, PX) = PF(A, X). F ( P A P T , P X ) = P F ( A , X ) .
Message passing. Message Passing Neural Networks (MPNNs) construct node representations by iteratively aggregating information from neighboring nodes [GSR+17].
Starting from
h v ( 0 ) = x v , h_v^{(0)} = x_v, h v ( 0 ) = x v ,
a general message-passing layer has the form
m v ( t ) = AGG t ( { { M t ( h v ( t − 1 ) , h u ( t − 1 ) ) : u ∈ N ( v ) } } ) , m_v^{(t)} = \operatorname{AGG}_t\left(\left\{ \left\{ M_t\left(h_v^{(t-1)}, h_u^{(t-1)}\right) : u \in\mathcal{N}(v) \right\} \right\}\right), m v ( t ) = AGG t ( { { M t ( h v ( t − 1 ) , h u ( t − 1 ) ) : u ∈ N ( v ) } } ) ,
followed by
h v ( t ) = U t ( h v ( t − 1 ) , m v ( t ) ) . h_v^{(t)} = U_t\left(h_v^{(t-1)}, m_v^{(t)}\right). h v ( t ) = U t ( h v ( t − 1 ) , m v ( t ) ) .
The aggregation operator is permutation invariant, ensuring that the overall network is equivariant to node relabeling.
For graph-level tasks, the final node representations are combined using an invariant readout,
h G = READOUT ( { { h v ( T ) : v ∈ V } } ) . h_G = \operatorname{READOUT}\left(\left\{ \left\{ h_v^{(T)} : v \in V \right\} \right\}\right). h G = READOUT ( { { h v ( T ) : v ∈ V } } ) .
After T T T iterations, the representation of a node can depend only on its T T T -hop neighborhood.
Examples of MPNN architectures. Many popular graph neural networks are special cases of the message-passing framework.
h v ( t ) = σ ( W t ∑ u ∈ N ( v ) ∪ { v } 1 d ~ v d ~ u h u ( t − 1 ) ) , h_v^{(t)} = \sigma\left(W_t \sum_{u \in\mathcal{N}(v)\cup\{v\}} \frac{1}{\sqrt{\widetilde{d}_v\widetilde{d}_u}}h_u^{(t-1)}\right), h v ( t ) = σ W t u ∈ N ( v ) ∪ { v } ∑ d v d u 1 h u ( t − 1 ) ,
where d ~ v \widetilde{d}_v d v is the degree after adding self-loops.
h v ( t ) = σ ( W t [ h v ( t − 1 ) ∥ AGG ( { { h u ( t − 1 ) : u ∈ N ( v ) } } ) ] ) . h_v^{(t)} = \sigma\left(W_t\left[h_v^{(t-1)} \mathbin\Vert\operatorname{AGG}\left(\left\{ \left\{h_u^{(t-1)} : u \in\mathcal{N}(v)\right\}\right\}\right)\right]\right). h v ( t ) = σ ( W t [ h v ( t − 1 ) ∥ AGG ( { { h u ( t − 1 ) : u ∈ N ( v ) } } ) ] ) .
h v ( t ) = σ ( ∑ u ∈ N ( v ) ∪ { v } α v u ( t ) W t h u ( t − 1 ) ) , h_v^{(t)} = \sigma\left(\sum_{u \in\mathcal{N}(v)\cup\{v\}} \alpha_{vu}^{(t)} W_t h_u^{(t-1)}\right), h v ( t ) = σ u ∈ N ( v ) ∪ { v } ∑ α v u ( t ) W t h u ( t − 1 ) ,
where α v u ( t ) \alpha_{vu}^{(t)} α v u ( t ) is computed from the features of v v v and u u u .
h v ( t ) = M L P t ( ( 1 + ϵ t ) h v ( t − 1 ) + ∑ u ∈ N ( v ) h u ( t − 1 ) ) h_v^{(t)} = \mathrm{MLP}_t\left((1+\epsilon_t)h_v^{(t-1)}+\sum_{u\in\mathcal{N}(v)}h_u^{(t-1)}\right) h v ( t ) = MLP t ( 1 + ϵ t ) h v ( t − 1 ) + u ∈ N ( v ) ∑ h u ( t − 1 )
GIN was designed to match the distinguishing power of the one-dimensional Weisfeiler–Leman test.
The Weisfeiler–Leman test. The expressive power of MPNNs is closely related to the one-dimensional Weisfeiler–Leman (1-WL) graph isomorphism test [WL68; XHLJ19; MRF+19].
The 1-WL algorithm assigns a color c v ( t ) c_v^{(t)} c v ( t ) to every node and updates it according to
c v ( t ) = HASH ( c v ( t − 1 ) , { { c u ( t − 1 ) : u ∈ N ( v ) } } ) , c_v^{(t)} = \operatorname{HASH}\left(c_v^{(t-1)},\{\{c_u^{(t-1)}:u\in\mathcal{N}(v)\}\}\right), c v ( t ) = HASH ( c v ( t − 1 ) , {{ c u ( t − 1 ) : u ∈ N ( v )}} ) ,
where the hash function is assumed to be injective.
Two graphs are distinguished if, at some iteration, the multisets of node colors differ. We write
G ≡ W L H G \equiv_{\mathrm{WL}} H G ≡ WL H
when 1-WL does not distinguish G G G and H H H .
The similarity with message passing is direct: both procedures repeatedly combine the current representation of a node with the multiset of representations in its neighborhood. Standard MPNNs cannot distinguish graphs that 1-WL fails to distinguish. Conversely, MPNNs with sufficiently expressive and injective aggregation and update functions can attain the distinguishing power of 1-WL [38 MRF+19] .
This connection provides a natural benchmark for the expressive power of message-passing architectures.
From Symmetry to Expressivity
Invariance specifies which inputs a representation should identify, but does not guarantee that it preserves all other distinctions. This is important for approximation: if a representation maps two nonequivalent inputs to the same point, then no subsequent function of that representation can distinguish between them.
Let ∼ \sim ∼ denote the equivalence relation induced by the relevant symmetry, and let
F : X → R m F:\mathcal{X}\to\mathbb{R}^{m} F : X → R m
be an invariant representation. We say that F F F separates inputs up to the symmetry if
F ( x ) = F ( y ) ⟹ x ∼ y . F(x) = F(y) \implies x \sim y. F ( x ) = F ( y ) ⟹ x ∼ y .
Equivalently,
x ≁ y ⟹ F ( x ) ≠ F ( y ) . x \not\sim y \implies F(x) \ne F(y). x ∼ y ⟹ F ( x ) = F ( y ) .
Thus, separation requires the representation to identify only inputs that are equivalent under the prescribed symmetry.
For graphs, the relevant equivalence relation may be graph isomorphism or a coarser relation such as 1-WL equivalence. Accordingly, the expressive power of a graph architecture is commonly measured by which pairs of nonequivalent graphs it is able to distinguish.
Geometric and Multi-Permutation Symmetries
The symmetry framework above also extends beyond ordinary vertex relabeling. Two examples are particularly relevant to this thesis.
Geometric graphs and point clouds. For geometric data with positions x i ∈ R d x_i \in\mathbb{R}^d x i ∈ R d , one often requires invariance or equivariance not only to permutations but also to rigid transformations,
x i ↦ Q x i + t , Q ∈ O ( d ) , t ∈ R d . x_i \mapsto Qx_i + t,\qquad Q \in O(d),\quad t \in\mathbb{R}^d. x i ↦ Q x i + t , Q ∈ O ( d ) , t ∈ R d .
This is the basic symmetry principle behind many geometric and Euclidean equivariant neural networks [TSK+18; SHW21]. Accordingly, two geometric graphs should be regarded as equivalent when they differ only by a relabeling of their vertices and a rigid motion of their coordinates. Chapter 3 studies when sparse geometric message-passing architectures can separate such objects, with particular attention to generic completeness and to the role of the underlying graph structure.
Multiple permutation symmetries.
Some structured inputs contain several unordered axes. For example, for
X ∈ R n × m × d , X \in\mathbb{R}^{n \times m \times d}, X ∈ R n × m × d ,
the product group
S n × S m S_n \times S_m S n × S m
may act independently on the first two indices:
( ( π , σ ) ⋅ X ) i , j , : = X π − 1 ( i ) , σ − 1 ( j ) , : . ((\pi,\sigma) \cdot X)_{i,j,:} = X_{\pi^{-1}(i),\sigma^{-1}(j),:}. (( π , σ ) ⋅ X ) i , j , : = X π − 1 ( i ) , σ − 1 ( j ) , : .
Linear layers equivariant to such actions can be characterized through parameter-sharing arguments [RSP17; HGLR18]. Chapter 4 develops a complementary representation-theoretic approach based on irreducible decompositions and Schur’s lemma, and applies it to several settings with multiple permutation symmetries.
Thesis Overview
The main body of this thesis is organized around four research works, presented in Chapters 3–6. Each of these chapters contains the corresponding paper and develops one of the main contributions of the thesis in full detail. Chapter 2 provides a more concise and self-contained account of the main ideas and methodology underlying these works, and may therefore serve as an overview before turning to the individual papers.
Chapter 3 studies the expressive power of message-passing neural networks on sparse geometric graphs. Chapter 4 develops a representation-theoretic approach to the characterization of equivariant linear layers. Chapter 5 studies set representations that preserve set inclusion, while Chapter 6 introduces FSW-GNN, a WL-equivalent MPNN with strong theoretical guarantees and empirical performance on long-range graph-learning tasks.
Taken together, these works study how neural representations can respect the symmetries of structured data while preserving progressively richer forms of information.
Research Methods
This chapter provides a concise overview of the four papers comprising the thesis. Each section summarizes the main problem, methodology, and principal results of one paper. Specifically, Section 2.1 summarizes the first paper presented in Chapter 3, Section 2.2 summarizes the second paper presented in Chapter 4, Section 2.3 summarizes the third paper presented in Chapter 5, and Section 2.4 summarizes the fourth paper presented in Chapter 6.
Geometric Graphs and Generic Separation
This section summarizes the first paper of the thesis, presented in Chapter 3. The main question is simple: when can a geometric message-passing network distinguish two geometric graphs that are not related by the natural symmetries of the problem?
Geometric Graphs
A geometric graph is a graph whose vertices are embedded in Euclidean space. We write
G = ( A , X ) , G = (A, X), G = ( A , X ) ,
where A ∈ { 0 , 1 } n × n A \in\{0,1\}^{n \times n} A ∈ { 0 , 1 } n × n is the adjacency matrix and
X = ( x 1 , … , x n ) , x i ∈ R d , X = (x_1,\ldots,x_n), \qquad x_i \in\mathbb{R}^d, X = ( x 1 , … , x n ) , x i ∈ R d ,
contains the vertex positions. Such graphs admit several natural symmetries. Relabeling the vertices by a permutation matrix P P P transforms the representation as
( A , X ) ↦ ( P A P T , P X ) , (A, X) \mapsto(PAP^{\mathsf{T}}, PX), ( A , X ) ↦ ( P A P T , P X ) ,
while a rigid transformation acts on the coordinates as
x i ↦ Q x i + t , x_i \mapsto Qx_i + t, x i ↦ Q x i + t ,
where Q ∈ O ( d ) Q \in O(d) Q ∈ O ( d ) and t ∈ R d t \in\mathbb{R}^d t ∈ R d . Two geometric graphs are considered equivalent if one can be obtained from the other by these transformations.
Our results are stated for generic geometric graphs. Informally, genericity rules out special degenerate configurations and holds for almost every choice of vertex positions.
Another important notion is global rigidity . A geometric graph is globally rigid if its edge lengths determine the entire realization, up to a rigid transformation. Thus, if a graph is globally rigid, knowing the lengths of its edges is enough to reconstruct its geometry up to the symmetries that we wish to ignore.
Message Passing on Geometric Graphs
We study two families of geometric message-passing networks.
An invariant geometric GNN (I-GGNN) propagates only invariant scalar features. Geometric information enters the network through edge lengths:
s i ( t + 1 ) = f t ( { ( s j ( t ) , ∥ x i − x j ∥ ) : j ∈ N ( i ) } ) . s_i^{(t+1)} = f_t\left(\left\{\left(s_j^{(t)}, \lVert x_i-x_j\rVert\right): j \in\mathcal{N}(i)\right\}\right). s i ( t + 1 ) = f t ( { ( s j ( t ) , ∥ x i − x j ∥ ) : j ∈ N ( i ) } ) .
As a result, an I-GGNN does not retain directional information during message passing.
An equivariant geometric GNN (E-GGNN), in contrast, may propagate vector-valued features. Its messages can depend on relative positions x i − x j x_i-x_j x i − x j , allowing directional information to be preserved:
v i ( t + 1 ) = f t ( { ( v j ( t ) , x i − x j ) : j ∈ N ( i ) } ) . v_i^{(t+1)} = f_t\left(\left\{\left(v_j^{(t)}, x_i-x_j\right): j \in\mathcal{N}(i)\right\}\right). v i ( t + 1 ) = f t ( { ( v j ( t ) , x i − x j ) : j ∈ N ( i ) } ) .
The message-passing layers are required to be permutation equivariant and rotation equivariant, while the final graph-level output is invariant to vertex permutations and rigid transformations.
To study the limitations of the message-passing framework itself, rather than those of a particular choice of neural networks, we consider maximally expressive I-GGNNs and E-GGNNs. Roughly speaking, all operations are assumed to preserve as much information as their required symmetries allow. In particular:
the neighborhood aggregation does not identify two different multisets unless this is forced by the relevant symmetry;
equivariant functions do not identify two inputs unless they are related by the corresponding group action;
invariantization identifies only inputs related by the symmetry; and
the final readout is injective on multisets of node features.
These assumptions give the strongest possible expressive power within each message-passing framework and allow us to ask which limitations are inherent to invariant or equivariant message passing itself.
Main Expressivity Results
Our main theoretical result gives a complete generic characterization for both classes of networks.
For I-GGNNs, the relevant graph property is generic global rigidity . The reason is intuitive: an I-GGNN observes the geometry only through the lengths of graph edges. If two different realizations have exactly the same edge lengths, then the network receives the same geometric information from both and cannot distinguish them.
We prove that a maximally expressive I-GGNN can generically identify a geometric graph precisely when the underlying graph is generically globally rigid:
I-GGNN generic identification ⟺ generic global rigidity . \text{I-GGNN generic identification} \Longleftrightarrow\text{generic global rigidity}. I-GGNN generic identification ⟺ generic global rigidity .
For E-GGNNs, the situation is different. Since vector features can preserve directional information, the network is not limited to edge lengths. We show that global rigidity is no longer required: the only necessary graph property is connectivity.
Indeed, if the graph is disconnected, its connected components can be moved independently and message passing cannot communicate between them. On the other hand, for a connected graph, a maximally expressive E-GGNN with sufficient depth can generically identify the entire geometric graph. Hence,
E-GGNN generic identification ⟺ connectivity . \text{E-GGNN generic identification} \Longleftrightarrow\text{connectivity}. E-GGNN generic identification ⟺ connectivity .
This gives a sharp distinction between the two frameworks. Invariant message passing requires the considerably stronger condition of global rigidity, whereas equivariant message passing requires only connectivity.
EGenNet
The previous results describe what a maximally expressive E-GGNN could achieve. We next ask whether these requirements can actually be realized by a concrete and practical neural architecture.
To this end, we introduce EGenNet . The architecture is designed as an explicit implementation of the E-GGNN framework described above. Its message-passing layers are permutation and rotation equivariant, translation is handled through relative coordinates, and its final representation is invariant to the symmetries of the geometric graph.
More importantly, EGenNet satisfies the maximal-expressivity requirements: its aggregation, equivariant transformations, invariantization, and final readout can all be chosen so that they do not lose information beyond what is required by the corresponding symmetries.
We prove that EGenNet therefore attains the generic expressive power predicted by our characterization of E-GGNNs. In particular, with sufficient depth it generically distinguishes connected geometric graphs exactly up to geometric isomorphism.
Thus, EGenNet connects the abstract expressivity result with an explicit neural architecture: the theoretical advantage of equivariant message passing is not merely an idealized property of the framework, but can be realized by a concrete model. We further validate the architecture empirically on synthetic separation tasks and molecular learning benchmarks.
Characterizing Equivariant Linear Layers
A recurring problem in equivariant deep learning is the following. Suppose that a symmetry group acts linearly on a vector space V \mathcal{V} V . We would like to characterize all linear maps
T : V → V T : \mathcal{V} \to\mathcal{V} T : V → V
that respect the symmetry, namely,
T ( g ⋅ x ) = g ⋅ T ( x ) ∀ g ∈ G , x ∈ V . T(g \cdot x) = g \cdot T(x) \qquad\forall g \in\mathcal{G},\ x \in\mathcal{V}. T ( g ⋅ x ) = g ⋅ T ( x ) ∀ g ∈ G , x ∈ V .
Once these linear maps are characterized, they can be used as the basic linear layers of an equivariant neural network.
The traditional approach. For permutation symmetries, this problem has traditionally been addressed through parameter sharing . One begins with a general linear map and uses the equivariance constraints to determine which entries of its parameter tensor must be equal.
Equivalently, one can study the orbits induced by the group on the input and output indices: parameters associated with indices that belong to the same orbit must be shared. This approach leads, for example, to the classical characterizations of DeepSets and higher-order invariant graph networks.
The parameter-sharing viewpoint is effective, but the required bookkeeping becomes increasingly involved when several permutation groups act simultaneously, or when the group acts on several interacting tensors.
Our approach. In Chapter 4, we study the same characterization problem from a different perspective. Instead of working directly with individual entries of the parameter tensor, we decompose the underlying vector space into minimal subspaces that are preserved by the symmetry group.
Let G G G be a finite group acting linearly on a finite-dimensional real vector space V V V .
A subspace
is called invariant if
g ⋅ u ∈ U ∀ g ∈ G , u ∈ U . g \cdot u \in U \qquad\forall g \in G,\, u \in U. g ⋅ u ∈ U ∀ g ∈ G , u ∈ U .
An invariant subspace is called irreducible if it contains no nonzero proper invariant subspace.
For the finite groups considered here, the space can be decomposed as a direct sum
V = V 1 ⊕ ⋯ ⊕ V s , V = V_1 \oplus\cdots\oplus V_s, V = V 1 ⊕ ⋯ ⊕ V s ,
where each V i V_i V i is irreducible.
The main idea is that after this decomposition has been found, the possible linear equivariant maps between the different components are extremely restricted.
A Simple Example
Consider the action on R 2 \mathbb{R}^2 R 2 that exchanges the two coordinates,
( x 1 , x 2 ) ↦ ( x 2 , x 1 ) . (x_1,x_2) \mapsto(x_2,x_1). ( x 1 , x 2 ) ↦ ( x 2 , x 1 ) .
There are two natural invariant subspaces:
U + = span { ( 1 , 1 ) } , U − = span { ( 1 , − 1 ) } . U_+ = \operatorname{span}\{(1,1)\}, \qquad U_- = \operatorname{span}\{(1,-1)\}. U + = span {( 1 , 1 )} , U − = span {( 1 , − 1 )} .
Hence
R 2 = U + ⊕ U − . \mathbb{R}^2 = U_+ \oplus U_-. R 2 = U + ⊕ U − .
The symmetry acts trivially on U + U_+ U + , while it acts by a sign change on U − U_- U − . These are therefore two different symmetry types.
An equivariant linear map cannot arbitrarily mix the two components. It can only scale each component independently:
T ( x + + x − ) = a x + + b x − . T(x_+ + x_-) = ax_+ + bx_-. T ( x + + x − ) = a x + + b x − .
This elementary example already illustrates the basic principle of our approach: first decompose the space according to how the symmetry acts, and then determine which interactions between the resulting components are possible.
Schur’s Lemma
This principle is formalized by Schur’s lemma.
Let U \mathcal{U} U and W \mathcal{W} W be irreducible invariant subspaces. We say that they are of the same symmetry type if there exists an equivariant linear bijection
L : U → W L:\mathcal{U}\to\mathcal{W} L : U → W
Schur’s lemma states that an equivariant linear map
T : U → W T:\mathcal{U}\to\mathcal{W} T : U → W
satisfies the following:
for some scalar λ ∈ R \lambda\in\mathbb{R} λ ∈ R and a fixed equivariant isomorphism L L L .
Technically, the second statement over the real numbers requires the irreducible spaces to be absolutely irreducible ; all irreducible components used in the constructions below satisfy this property.
Consequently, suppose that
V = ⨁ i = 1 s V i \mathcal{V}=\bigoplus_{i=1}^{s}\mathcal{V}_i V = i = 1 ⨁ s V i
is an irreducible decomposition. A linear equivariant map is completely determined by the pairs of components that are of the same symmetry type.
If the distinct symmetry types occur with multiplicities
α 1 , … , α k , \alpha_1,\ldots,\alpha_k, α 1 , … , α k ,
then the dimension of the space of equivariant linear maps is
∑ i = 1 k α i 2 . \sum_{i=1}^{k}\alpha_i^2. i = 1 ∑ k α i 2 .
Thus, the characterization of an equivariant layer reduces to two main steps:
decompose the feature space into irreducible invariant subspaces;
determine which of these subspaces are of the same symmetry type.
Schur’s lemma then provides the form of all possible linear equivariant mappings automatically.
The Classical Permutation Example: DeepSets
We first apply this approach to the classical action of the symmetric group S n S_n S n on
V = R n \mathcal{V} = \mathbb{R}^n V = R n
by permutation of coordinates. There are two irreducible invariant subspaces:
S = span { 1 n } , \mathbf{S} = \operatorname{span}\{\mathbf{1}_n\}, S = span { 1 n } ,
and
V ( n ) = { x ∈ R n : ∑ i = 1 n x i = 0 } . \mathbf{V}(n) = \left\{x \in\mathbb{R}^n : \sum_{i=1}^{n} x_i = 0\right\}. V ( n ) = { x ∈ R n : i = 1 ∑ n x i = 0 } .
Every vector can be uniquely decomposed as
x = x ˉ 1 n + ( x − x ˉ 1 n ) , x ˉ = 1 n ∑ i = 1 n x i . x = \bar{x}\mathbf{1}_n + (x-\bar{x}\mathbf{1}_n), \qquad\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i. x = x ˉ 1 n + ( x − x ˉ 1 n ) , x ˉ = n 1 i = 1 ∑ n x i .
The two subspaces are not of the same symmetry type. Therefore Schur’s lemma implies that a linear permutation-equivariant map has two free parameters: one acting on the constant component and one on the zero-sum component.
Equivalently, every such map has the form
T ( x ) = a x + b ( ∑ i = 1 n x i ) 1. T(x) = ax + b\left(\sum_{i=1}^{n}x_i\right)\mathbf{1}. T ( x ) = a x + b ( i = 1 ∑ n x i ) 1 .
This recovers the classical linear equivariant layer underlying DeepSets. The usual two-parameter characterization, commonly derived through parameter sharing, therefore follows immediately from the decomposition
R n = S ⊕ V ( n ) . \mathbb{R}^n = \mathbf{S} \oplus\mathbf{V}(n). R n = S ⊕ V ( n ) .
Second-Order Graph Equivariance: 2-IGN
A more involved example is the second-order graph setting underlying 2-IGN. Here
V = R n × n , \mathcal{V} = \mathbb{R}^{n \times n}, V = R n × n ,
and a permutation τ ∈ S n \tau\in S_n τ ∈ S n acts simultaneously on the two indices:
( τ ⋅ X ) i j = X τ − 1 ( i ) , τ − 1 ( j ) . (\tau\cdot X)_{ij} = X_{\tau^{-1}(i),\tau^{-1}(j)}. ( τ ⋅ X ) ij = X τ − 1 ( i ) , τ − 1 ( j ) .
In matrix notation,
X ↦ P X P T , X \mapsto PXP^{\mathsf{T}}, X ↦ P X P T ,
where P P P is the permutation matrix associated with τ \tau τ .
For n ≥ 4 n \ge4 n ≥ 4 , we show that R n × n \mathbb{R}^{n \times n} R n × n decomposes into seven irreducible invariant subspaces.
Two of these components are of the same symmetry type, three further components are of a second common type, and the remaining two components each occur once. The multiplicities are therefore
2 , 3 , 1 , 1. 2,\quad3,\quad1,\quad1. 2 , 3 , 1 , 1.
Schur’s lemma immediately gives
2 2 + 3 2 + 1 2 + 1 2 = 15 2^2 + 3^2 + 1^2 + 1^2 = 15 2 2 + 3 2 + 1 2 + 1 2 = 15
free parameters.
Thus the space of second-order permutation-equivariant linear layers is 15-dimensional, recovering exactly the classical 2-IGN characterization.
The significance here is methodological. The fifteen parameters are usually obtained by studying fifteen different parameter-sharing patterns in a four-index tensor. Our derivation instead follows from the structure of the seven invariant subspaces and their isomorphism relations.
Independent permutations of rows and columns.
Consider the action of S n × S m S_n \times S_m S n × S m on R n × m \mathbb{R}^{n \times m} R n × m , where S n S_n S n permutes the rows and S m S_m S m permutes the columns:
( P , Q ) ⋅ X = P X Q T . (P,Q) \cdot X = P X Q^{\mathsf{T}}. ( P , Q ) ⋅ X = P X Q T .
Under this action, the matrix space decomposes into four natural invariant subspaces. The first is the one-dimensional space of constant matrices,
U 1 = { ( c c ⋯ c c c ⋯ c ⋮ ⋮ ⋱ ⋮ c c ⋯ c ) : c ∈ R } . U_1 = \left\{
\begin{pmatrix}
c & c & \cdots& c \\
c & c & \cdots& c \\
\vdots& \vdots& \ddots& \vdots\\
c & c & \cdots& c
\end{pmatrix}
: c \in\mathbb{R}
\right\}. U 1 = ⎩ ⎨ ⎧ c c ⋮ c c c ⋮ c ⋯ ⋯ ⋱ ⋯ c c ⋮ c : c ∈ R ⎭ ⎬ ⎫ .
The second consists of matrices whose rows are all identical, with the common row having zero sum,
U 2 = { ( − x T − x T ⋮ − x T ) : x ∈ R m , ∑ i = 1 m x i = 0 } , U_2 = \left\{
\begin{pmatrix}
-x^{\mathsf{T}} \\
-x^{\mathsf{T}} \\
\vdots\\
-x^{\mathsf{T}}
\end{pmatrix}
: x \in\mathbb{R}^{m},\ \sum_{i=1}^{m} x_i = 0
\right\}, U 2 = ⎩ ⎨ ⎧ − x T − x T ⋮ − x T : x ∈ R m , i = 1 ∑ m x i = 0 ⎭ ⎬ ⎫ ,
and therefore has dimension m − 1 m-1 m − 1 . The third consists of matrices whose columns are all identical,
U 3 = { [ y y ⋯ y ] : y ∈ R n , ∑ i = 1 n y i = 0 } . U_3=\left\{\begin{bmatrix} y & y & \cdots& y \end{bmatrix}: y\in\mathbb{R}^{n},\ \sum_{i=1}^{n}y_i=0\right\}. U 3 = { [ y y ⋯ y ] : y ∈ R n , i = 1 ∑ n y i = 0 } .
and has dimension n − 1 n-1 n − 1 . Finally,
U 4 = { Z ∈ R n × m : Z 1 m = 0 , 1 n T Z = 0 } U_4=\left\{Z\in\mathbb{R}^{n\times m}:Z\mathbf{1}_m=0,\ \mathbf{1}_n^{\mathsf{T}}Z=0\right\} U 4 = { Z ∈ R n × m : Z 1 m = 0 , 1 n T Z = 0 }
is the space of matrices whose row sums and column sums are all zero, and has dimension ( n − 1 ) ( m − 1 ) (n-1)(m-1) ( n − 1 ) ( m − 1 ) . Hence
R n × m = U 1 ⊕ U 2 ⊕ U 3 ⊕ U 4 . \mathbb{R}^{n\times m}=U_1\oplus U_2\oplus U_3\oplus U_4. R n × m = U 1 ⊕ U 2 ⊕ U 3 ⊕ U 4 .
These four subspaces are irreducible and pairwise non-isomorphic S n × S m S_n\times S_m S n × S m -representations. Therefore, by Schur’s lemma, every linear S n × S m S_n\times S_m S n × S m -equivariant layer acts by an independent scalar on each component: if
X = X 00 + X 01 + X 10 + X 11 , X a b ∈ U a b , X=X_{00}+X_{01}+X_{10}+X_{11},\qquad X_{ab}\in U_{ab}, X = X 00 + X 01 + X 10 + X 11 , X ab ∈ U ab ,
then
T ( X ) = λ 00 X 00 + λ 01 X 01 + λ 10 X 10 + λ 11 X 11 . T(X)=\lambda_{00}X_{00}+\lambda_{01}X_{01}+\lambda_{10}X_{10}+\lambda_{11}X_{11}. T ( X ) = λ 00 X 00 + λ 01 X 01 + λ 10 X 10 + λ 11 X 11 .
Equivalently, such a layer is determined by four parameters and can be written entrywise as
( T X ) i j = a X i j + b ∑ j ′ = 1 m X i j ′ + c ∑ i ′ = 1 n X i ′ j + d ∑ i ′ = 1 n ∑ j ′ = 1 m X i ′ j ′ . (TX)_{ij}=aX_{ij}+b\sum_{j'=1}^{m}X_{ij'}+c\sum_{i'=1}^{n}X_{i'j}+d\sum_{i'=1}^{n}\sum_{j'=1}^{m}X_{i'j'}. ( T X ) ij = a X ij + b j ′ = 1 ∑ m X i j ′ + c i ′ = 1 ∑ n X i ′ j + d i ′ = 1 ∑ n j ′ = 1 ∑ m X i ′ j ′ .
Wreath-Product Equivariance
The direct-product example above assumes that the same collection of row and column permutations acts on a single object. A different symmetry appears when the input itself is a set of unaligned symmetric objects . Suppose that each object belongs to a representation space V V V of a finite group G G G , and consider a k k k -tuple
( v 1 , … , v k ) ∈ V k . (v_1,\ldots,v_k)\in V^k. ( v 1 , … , v k ) ∈ V k .
Each object may be transformed independently by an element of G G G , while the k k k objects themselves may also be permuted. The resulting symmetry group is the wreath product
G ≀ S k = G k ⋊ S k , G\wr S_k=G^k\rtimes S_k, G ≀ S k = G k ⋊ S k ,
where S k S_k S k acts on G k \mathcal{G}^k G k by permuting its k k k factors [WAR20; MLCF20]. With the standard convention, its action on V k \mathcal{V}^k V k can be written as
( ( g 1 , … , g k ) , τ ) ⋅ ( v 1 , … , v k ) = ( g 1 ⋅ v τ − 1 ( 1 ) , … , g k ⋅ v τ − 1 ( k ) ) . ((g_1,\ldots,g_k),\tau)\cdot(v_1,\ldots,v_k)=(g_1\cdot v_{\tau^{-1}(1)},\ldots,g_k\cdot v_{\tau^{-1}(k)}). (( g 1 , … , g k ) , τ ) ⋅ ( v 1 , … , v k ) = ( g 1 ⋅ v τ − 1 ( 1 ) , … , g k ⋅ v τ − 1 ( k ) ) .
Thus the symmetry contains two distinct operations: independent “alignments” inside each object and permutation of the objects themselves. This setting appears, for example, in collections of graphs, collections of neural-network weight spaces, hierarchical data, and alignment problems.
A simple way to obtain a wreath-equivariant layer is to apply the same G \mathcal{G} G -equivariant map independently to every element,
L S i a m ( v 1 , … , v k ) = ( L ^ ( v 1 ) , … , L ^ ( v k ) ) . L_{\mathrm{Siam}}(v_1,\ldots,v_k)=(\widehat{L}(v_1),\ldots,\widehat{L}(v_k)). L Siam ( v 1 , … , v k ) = ( L ( v 1 ) , … , L ( v k )) .
We refer to such a layer as Siamese . The main question studied in Chapter 4 is whether this exhausts all linear G ≀ S k \mathcal{G}\wr S_k G ≀ S k -equivariant maps. In general, it does not.
The characterization is controlled by the fixed-point subspace
V f i x e d = { v ∈ V : g ⋅ v = v for every g ∈ G } . \mathcal{V}_{\mathrm{fixed}}=\{v\in\mathcal{V}:g\cdot v=v\text{ for every }g\in\mathcal{G}\}. V fixed = { v ∈ V : g ⋅ v = v for every g ∈ G } .
Let
s = dim V f i x e d , s=\dim\mathcal{V}_{\mathrm{fixed}}, s = dim V fixed ,
and choose a basis e 1 , … , e s e_1,\ldots,e_s e 1 , … , e s of this subspace. Since G \mathcal{G} G is finite, V \mathcal{V} V admits a G \mathcal{G} G -invariant inner product; it can be obtained by averaging any inner product over the group [FH13].
Our characterization shows that every linear G ≀ S k \mathcal{G}\wr S_k G ≀ S k -equivariant map L : V k → V k L:\mathcal{V}^k\to\mathcal{V}^k L : V k → V k has the form
L ( v 1 , … , v k ) = ( L ^ ( v 1 ) , … , L ^ ( v k ) ) + ∑ i , j = 1 s a i j ( ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j , … , ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j ) . \begin{aligned}
L(v_1,\ldots,v_k)&=(\widehat{L}(v_1),\ldots,\widehat{L}(v_k))\\
&\quad+\sum_{i,j=1}^{s}a_{ij}\left(\sum_{\ell=1}^{k}\langle v_\ell,e_i\rangle e_j,\ldots,\sum_{\ell=1}^{k}\langle v_\ell,e_i\rangle e_j\right).
\end{aligned} L ( v 1 , … , v k ) = ( L ( v 1 ) , … , L ( v k )) + i , j = 1 ∑ s a ij ( ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j , … , ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j ) .
where L ^ : V → V \widehat{L}:\mathcal{V}\to\mathcal{V} L : V → V is an arbitrary linear G \mathcal{G} G -equivariant map and a i j ∈ R a_{ij}\in\mathbb{R} a ij ∈ R . The first term is the Siamese part. The second term couples different elements of the set: it extracts invariant scalar information from all k k k elements, aggregates that information globally, and broadcasts it back to every element in a fixed direction. These are precisely the additional non-Siamese interactions.
The representation-theoretic reason for this form is particularly simple. Every non-trivial irreducible component of V \mathcal{V} V remains irreducible when lifted to V k \mathcal{V}^k V k under the wreath-product action. In contrast, each one-dimensional trivial component produces the usual two irreducible S k S_k S k components: a constant component and a zero-sum component. Schur’s lemma therefore creates an additional family of interactions only through the trivial part of V \mathcal{V} V .
Consequently, beyond the Siamese equivariant maps there are exactly s 2 s^2 s 2 independent non-Siamese linear maps. This recovers the previously studied transitive case, where s = 1 s = 1 s = 1 and there is only one additional interaction, but gives substantially richer layers when the group action has several independent invariant directions. For example, for graph matrices V = R n × n \mathcal{V} = \mathbb{R}^{n \times n} V = R n × n under simultaneous node permutations, s = 2 s = 2 s = 2 , giving four non-Siamese maps. For deep weight spaces, the dimension of the fixed-point subspace can be much larger, yielding a correspondingly large family of additional equivariant interactions.
Monotone and Separable Set Representations
This section summarizes the third paper of the thesis, presented in Chapter 5. A central goal in learning on sets is to construct permutation-invariant representations that preserve the information contained in the input set. DeepSets [ZKR+17] provides a fundamental framework for this purpose, using representations of the form
F ( S ) = ρ ( ∑ x ∈ S ϕ ( x ) ) . F(S) = \rho\left(\sum_{x \in S} \phi(x)\right). F ( S ) = ρ ( x ∈ S ∑ ϕ ( x ) ) .
Much of the classical theory asks whether such representations are expressive enough to distinguish different sets. Our work asks for a stronger property: can the representation preserve not only whether two sets are different, but also whether one is contained in the other?
Monotone and Separable Set Functions
Let V V V be a ground set. We seek a representation
F ( S ) ∈ R m F(S) \in\mathbb{R}^{m} F ( S ) ∈ R m
for finite multisets S ⊆ V S \subseteq V S ⊆ V , such that multiset containment is represented exactly by the coordinatewise order:
S ⊆ T ⟺ F ( S ) ≤ F ( T ) . S \subseteq T \iff F(S) \leq F(T). S ⊆ T ⟺ F ( S ) ≤ F ( T ) .
We call F F F monotone if
S ⊆ T ⟹ F ( S ) ≤ F ( T ) , S \subseteq T \implies F(S) \leq F(T), S ⊆ T ⟹ F ( S ) ≤ F ( T ) ,
and separable if
F ( S ) ≤ F ( T ) ⟹ S ⊆ T . F(S) \leq F(T) \implies S \subseteq T. F ( S ) ≤ F ( T ) ⟹ S ⊆ T .
A function satisfying both properties is called a monotone and separable (MAS) function.
This requirement is stronger than ordinary injectivity. An injective representation only distinguishes different sets; a MAS representation preserves the entire partial order induced by containment.
Finite Ground Sets
We first assume that the ground set is finite,
while placing no bound on the cardinality of the input multisets.
In this setting we obtain an exact result. The smallest dimension in which a MAS representation can exist is precisely
m ∗ ( V , ∞ ) = n . m^{*}(V,\infty) = n. m ∗ ( V , ∞ ) = n .
The upper bound is simple: use one coordinate for each element of V V V , and let F ( S ) F(S) F ( S ) record its multiplicity in S S S . Coordinatewise comparison then coincides exactly with multiset containment. Our lower bound shows that no representation into fewer than n n n coordinates can satisfy both monotonicity and separability for multisets of arbitrary size.
Infinite Ground Sets
The situation changes fundamentally when the ground set is infinite, as is typically the case in machine learning, for example when V = R d V = \mathbb{R}^{d} V = R d .
We prove that for every k ≥ 2 k \ge2 k ≥ 2 , there is no finite-dimensional MAS representation on multisets of cardinality at most k k k :
∣ V ∣ = ∞ , k ≥ 2 ⇒ no finite-dimensional MAS representation exists. |V| = \infty,\quad k \ge2 \quad\Rightarrow\quad\text{no finite-dimensional MAS representation exists.} ∣ V ∣ = ∞ , k ≥ 2 ⇒ no finite-dimensional MAS representation exists.
Thus, exact preservation of containment is too strong to be achieved in finite dimension on general continuous domains.
Bounded Multiset Cardinality
For a finite ground set, the dimension n n n may still be prohibitively large. We therefore ask whether the situation improves if every input multiset contains at most k k k elements.
It does. We prove the upper bound
m ∗ ( V , k ) ≤ min { n , ( k + 2 ) k + 2 log n } . m^{*}(V,k) \le\min\left\{n,\,(k+2)^{k+2}\log n\right\}. m ∗ ( V , k ) ≤ min { n , ( k + 2 ) k + 2 log n } .
Thus, for fixed k k k , the required dimension can grow only logarithmically with the size of the ground set rather than linearly.
We complement this with lower bounds. For k ≥ 2 k \ge2 k ≥ 2 ,
m ∗ ( V , k ) ≥ log 2 ( log 3 n ) m^*(V,k) \ge\log_{2}(\log_{3} n) m ∗ ( V , k ) ≥ log 2 ( log 3 n )
and, when k ≤ ( n − 1 ) / 2 k \le(n-1)/2 k ≤ ( n − 1 ) /2 ,
m ∗ ( V , k ) ≥ 2 k m^*(V,k) \ge2k m ∗ ( V , k ) ≥ 2 k
Hence, in this regime,
m ∗ ( V , k ) ≥ max { 2 k , log 2 ( log 3 n ) } . m^*(V,k) \ge\max\{2k,\log_{2}(\log_{3} n)\}. m ∗ ( V , k ) ≥ max { 2 k , log 2 ( log 3 n )} .
These results show that bounding the input size dramatically reduces the dimension required for containment-preserving representations, while some dependence on both k k k and n n n remains unavoidable.
Approximate Containment
Exact containment is a Boolean notion, whereas in learning problems the set elements are often noisy or continuous. We therefore also introduce an asymmetric distance that measures how far a set S S S is from being contained in another set T T T .
For V ⊆ R d V \subseteq\mathbb{R}^{d} V ⊆ R d and ∣ S ∣ ≤ ∣ T ∣ |S| \le|T| ∣ S ∣ ≤ ∣ T ∣ , we define
d ⊆ ( S , T ) = min T ′ ⊆ T ∣ T ′ ∣ = ∣ S ∣ EMD ( S , T ′ ) . d_{\subseteq}(S,T)=\min_{\substack{T'\subseteq T\\|T'|=|S|}} \operatorname{EMD}(S,T'). d ⊆ ( S , T ) = T ′ ⊆ T ∣ T ′ ∣ = ∣ S ∣ min EMD ( S , T ′ ) .
where EMD denotes the Earth Mover Distance.
The asymmetry is intentional. We are not asking how similar S S S and T T T are, but rather how close S S S is to some subset of T T T . In particular,
d ⊆ ( S , T ) = 0 ⟺ S ⊆ T . d_{\subseteq}(S,T)=0 \Longleftrightarrow S\subseteq T. d ⊆ ( S , T ) = 0 ⟺ S ⊆ T .
If S S S is not exactly contained in T T T , but each of its elements can be matched closely to elements of T T T , then d ⊆ ( S , T ) d_{\subseteq}(S,T) d ⊆ ( S , T ) is small.
This allows us to strengthen the purely Boolean notion of separation. We introduce a lower Hölder separability condition, under which the amount by which the embeddings violate the coordinatewise order is quantitatively controlled from below by d ⊆ ( S , T ) d_{\subseteq}(S,T) d ⊆ ( S , T ) . Thus, sets that are farther from satisfying containment are also easier to separate in the embedding space. This extends ideas of Hölder stability for multiset representations [DD24] to the asymmetric set-containment setting.
MASNet
The impossibility result for infinite ground sets motivates relaxing exact MAS while preserving its most useful structural property. We introduce a parametric notion called weakly MAS . A family F ( S ; w ) F(S; w) F ( S ; w ) is weakly MAS if
for every parameter choice w w w , F ( ⋅ ; w ) F(\cdot; w) F ( ⋅ ; w ) is monotone; and
whenever S ⊈ T S \not\subseteq T S ⊆ T , there exists a parameter choice w w w for which F ( S ; w ) ≰ F ( T ; w ) F(S; w) \not\leq F(T; w) F ( S ; w ) ≤ F ( T ; w ) .
Thus, monotonicity is guaranteed for every model in the family, while different parameter choices may separate different pairs of non-contained sets.
Based on this relaxation, we introduce MASNet , a DeepSets-style architecture of the form
MASNet ( S ) = M 2 ( ∑ x ∈ S σ ( M 1 ( x ) ) ) . \operatorname{MASNet}(S) = M_2\left(\sum_{x \in S} \sigma\left(M_1(x)\right)\right). MASNet ( S ) = M 2 ( x ∈ S ∑ σ ( M 1 ( x ) ) ) .
The construction is deliberately simple. The element-wise network M 1 M_1 M 1 maps each input element to a latent representation, the non-negative activation σ \sigma σ produces contributions that are summed over the set, and M 2 M_2 M 2 is chosen to be a monotone vector-to-vector map. Consequently, adding elements to the input can only increase the resulting representation, so monotonicity holds by construction.
The choice of σ \sigma σ determines the separation properties. Using hat-shaped activations yields a weakly MAS model and, under the assumptions developed in the paper, also provides lower Hölder separability with respect to d ⊆ d_{\subseteq} d ⊆ . Alternatively, a two-layer ReLU realization of M 1 M_1 M 1 can also produce a weakly MAS model.
Hence, MASNet retains exact monotonicity while replacing the impossible requirement of a single finite-dimensional MAS embedding on continuous domains by a parametric separation guarantee. The resulting inductive bias is particularly suited to exact and approximate set-containment problems, where our experiments show improvements over standard set architectures such as DeepSets and Set Transformers.
Bi-Lipschitz Message Passing and FSW-GNN
The previous section focused on whether a neural architecture respects a given symmetry. For graph neural networks, however, respecting permutation symmetry is only part of the problem. Even an expressive MPNN may gradually lose information as messages are repeatedly aggregated and propagated across the graph.
This issue becomes particularly apparent in long-range graph learning tasks, where solving the problem requires many message-passing iterations.
Long-Range Information and Oversquashing
Recall that after t t t message-passing iterations, the feature of a node can depend on information from nodes at distance at most t t t . Therefore, a task whose relevant information is r r r hops away requires at least r r r message-passing iterations.
In principle, increasing the depth should allow information to travel farther. In practice, however, deep MPNNs often become difficult to train and fail on long-range tasks.
One important explanation for the difficulty of long-range message passing is over-squashing . As the receptive field of a node grows with depth, information from an increasingly large number of distant nodes must be compressed into a fixed-dimensional representation and propagated through relatively few intermediate nodes. Consequently, information originating far away may have only a weak influence on the final representation, even when the network is sufficiently deep for that information to reach the target node.
A useful example is the Ring benchmark. Each graph is a cycle with a designated source node and target node at graph distance r r r , as illustrated in Figure 2.1 . The target is assigned one of five possible labels, and the task is to predict this label at the source node. Thus, solving the task requires information to travel over r r r message-passing steps. As r r r increases, this becomes increasingly difficult for standard MPNNs, as shown in Figure 2.1 . Since there are five possible labels, random guessing achieves 20% accuracy. The comparison is shown in Figure 2.1 .
Figure 2.1. Performance of standard MPNNs on the Ring long-range task. As the required message-passing depth increases, the models gradually fail to propagate information across the graph. The Ring topology is illustrated on the right.
This behavior is particularly interesting because it cannot be explained solely through the usual binary notion of expressive power. Some standard MPNNs, such as GIN, are theoretically as expressive as the 1-WL test. Nevertheless, being able to separate two inputs in principle does not imply that their learned representations remain numerically well separated.
This motivates a quantitative question:
How much of the distance between different inputs is preserved by the network?
WL-Compatible Metrics on Graphs
Binary expressivity alone does not quantify how much information is retained by an embedding. To formulate a quantitative notion, we first need a distance on the graph domain. Let G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) denote vertex-featured graphs with at most N N N vertices and features in Ω ⊆ R d \Omega\subseteq\mathbb{R}^{d} Ω ⊆ R d . The natural equivalence relation for an MPNN is not graph isomorphism but 1-WL equivalence: no MPNN can distinguish a pair of graphs that 1-WL fails to distinguish [38 MRF^{+}19] . We therefore use metrics whose zero sets agree exactly with this limitation.
A WL metric is a function
ρ : G ≤ N ( Ω ) × G ≤ N ( Ω ) ⟶ R ≥ 0 \rho: \mathcal{G}_{\leq N}(\Omega) \times\mathcal{G}_{\leq N}(\Omega) \longrightarrow\mathbb{R}_{\geq0} ρ : G ≤ N ( Ω ) × G ≤ N ( Ω ) ⟶ R ≥ 0
that is symmetric, satisfies the triangle inequality, and obeys
ρ ( G , H ) = 0 ⟺ G ≡ W L H . \rho(G,H) = 0 \Longleftrightarrow G \equiv_{\mathrm{WL}} H. ρ ( G , H ) = 0 ⟺ G ≡ WL H .
Strictly speaking, this is a pseudometric on the space of labeled graph representations, since distinct graphs may have zero distance. Equivalently, it is an ordinary metric on the quotient space obtained by identifying all WL-equivalent graphs. The point is that a WL metric refines the binary WL test: it still regards exactly the same pairs as indistinguishable, but it also assigns a magnitude to the difference between distinguishable pairs.
We use two such metrics, which capture graph difference from complementary view-points.
The Doubly Stochastic Metric
The Doubly Stochastic (DS) metric originates from a convex relaxation of graph isomorphism [Gro20; Bök21]. For two featureless graphs with the same number n n n of vertices and adjacency matrices A A A and A ~ \widetilde{A} A , isomorphism requires a permutation matrix P P P satisfying
A P = P A ~ . AP = P\widetilde{A}. A P = P A .
Replacing the discrete set of permutation matrices by its convex hull, the set D n \mathcal{D}_n D n of doubly stochastic matrices, leads to the discrepancy
min S ∈ D n ∥ A S − S A ~ ∥ . \min_{S \in\mathcal{D}_n} \lVert AS - S\widetilde{A}\rVert. S ∈ D n min ∥ A S − S A ∥ .
A classical characterization of fractional graph isomorphism implies that this quantity vanishes exactly when the two graphs are 1-WL equivalent [Gro21]. Thus the relaxation is closely aligned with the expressive power of message passing rather than with exact graph isomorphism.
For the setting of Chapter 6, graphs may have different numbers of vertices and continuous vertex features. We therefore use an extended DS metric. Let
G = ( A , X ) , G ~ = ( A ~ , X ~ ) , G = (A, X), \qquad\widetilde{G} = (\widetilde{A}, \widetilde{X}), G = ( A , X ) , G = ( A , X ) ,
with n n n and n ~ \widetilde{n} n vertices, respectively, and define Π ( n , n ~ ) \Pi(n,\widetilde{n}) Π ( n , n ) as the set of nonnegative n × n ~ n \times\widetilde{n} n × n matrices whose row sums are 1 / n 1/n 1/ n and whose column sums are 1 / n ~ 1/\widetilde{n} 1/ n . We set
ρ D S ( G , G ~ ) = ∣ n − n ~ ∣ + min S ∈ Π ( n , n ~ ) ( ∥ A S − S A ~ ∥ 1 + ∑ i = 1 n ∑ j = 1 n ~ S i j ∥ x i − x ~ j ∥ 1 ) . \rho_{\mathrm{DS}}(G,\widetilde{G}) = \lvert n - \widetilde{n}\rvert
+ \min_{S \in\Pi(n,\widetilde{n})}
\left(
\lVert AS - S\widetilde{A}\rVert_1
+ \sum_{i=1}^{n}\sum_{j=1}^{\widetilde{n}} S_{ij}\lVert x_i - \widetilde{x}_j\rVert_1
\right). ρ DS ( G , G ) = ∣ n − n ∣ + S ∈ Π ( n , n ) min ( ∥ A S − S A ∥ 1 + i = 1 ∑ n j = 1 ∑ n S ij ∥ x i − x j ∥ 1 ) .
The first term penalizes different graph cardinalities, the matrix term measures disagreement between the two adjacency structures under a soft vertex correspondence, and the final term measures feature mismatch under the same correspondence. We prove that this extension is again a WL metric.
Tree Mover’s Distance
The second metric is the Tree Mover’s Distance (TMD) of [CJ22]. Its construction follows the computation-tree interpretation of message passing. After t t t rounds, the representation at a vertex depends on the rooted depth-t t t computation tree obtained by recursively unfolding its neighborhood. TMD defines a distance between two such rooted trees by comparing the root features and then optimally matching their child subtrees; the matching cost is itself defined recursively. A graph-level distance is obtained by an optimal-transport matching of the rooted computation trees of all vertices.
This construction mirrors the WL recursion. In particular, the tree distance between two rooted computation trees is zero precisely when WL assigns the corresponding roots the same color after the relevant number of iterations. Under the standard assumptions on the feature domain, TMD therefore induces a WL metric at graph level [CJ22]. Whereas the DS metric is based on a soft global correspondence between adjacency matrices, TMD measures graph difference through the local computation trees actually seen by message passing.
Bi-Lipschitz Embeddings and Distortion
Once a WL metric ρ \rho ρ is fixed, we can ask whether a graph embedding
F : G ≤ N ( Ω ) → R m F : \mathcal{G}_{\leq N}(\Omega) \to\mathbb{R}^{m} F : G ≤ N ( Ω ) → R m
preserves that geometry. We say that F F F is bi-Lipschitz with respect to ρ \rho ρ if there exist constants
0 < c ≤ C < ∞ 0 < c \leq C < \infty 0 < c ≤ C < ∞
such that
c ρ ( G , H ) ≤ ∥ F ( G ) − F ( H ) ∥ ≤ C ρ ( G , H ) c \rho(G,H) \leq\lVert F(G) - F(H) \rVert\leq C \rho(G,H) c ρ ( G , H ) ≤ ∥ F ( G ) − F ( H )∥ ≤ C ρ ( G , H )
for every pair G G G , H H H in the considered domain. The upper inequality provides stability: nearby inputs cannot be amplified arbitrarily. The lower inequality provides quantitative separation: inputs that are far apart according to the WL geometry cannot collapse to nearly the same representation. The ratio C c \frac{C}{c} c C upper-bounds the metric distortion of the embedding.
This is strictly stronger than WL-equivalent separation. A WL-equivalent embedding only guarantees
F ( G ) = F ( H ) ⟺ G ≡ W L H ; F(G) = F(H) \Longleftrightarrow G \equiv_{\mathrm{WL}} H; F ( G ) = F ( H ) ⟺ G ≡ WL H ;
it places no lower bound on the distance between the embeddings of two non-equivalent graphs. Consequently, a sequence of distinguishable graph pairs can be mapped arbitrarily close together while the model remains fully WL-expressive. Bi-Lipschitzness rules out precisely this form of numerical collapse.
The Aggregation Bottleneck
A central source of oversquashing lies in the aggregation step itself.
At every message-passing iteration, a node compresses a multiset of neighboring features,
{ { h u ( t − 1 ) : u ∈ N ( v ) } } , \{\{h_u^{(t-1)} : u \in\mathcal{N}(v)\}\}, {{ h u ( t − 1 ) : u ∈ N ( v )}} ,
into a single finite-dimensional vector. Standard MPNNs commonly use simple aggregation operations such as mean, maximum, or summation. Mean and maximum aggregation are not injective on general multisets and therefore necessarily discard information. Sum aggregation can be sufficiently expressive to obtain WL-equivalent MPNNs [2 ] , but it’s provably not a bi-Lipschitz embedding of multiset space.
Consequently, even when an aggregation rule is expressive enough to distinguish the required multisets theoretically, it may strongly distort their distances. Repeating such aggregation over many layers provides a possible mechanism by which information becomes increasingly compressed in deep MPNNs.
This observation motivates replacing standard aggregation by a multiset embedding that not only distinguishes multisets but also preserves their geometry.
FSW Aggregation
Our construction is based on the Fourier Sliced-Wasserstein (FSW) embedding , a permutation-invariant embedding of multisets into Euclidean space introduced by Amir and Dym [AD24].
At a high level, FSW first projects the elements of a multiset onto several one-dimensional directions. The projected values are sorted and interpreted through their quantile functions, from which Fourier-type features are extracted. The resulting finite-dimensional representation is independent of the ordering of the input elements and naturally accommodates multisets of different cardinalities.
Most importantly for our purpose, the FSW embedding is bi-Lipschitz with respect to a natural Wasserstein metric on multisets.
We therefore use FSW as the neighborhood aggregation mechanism of an MPNN. At iteration t t t , we compute
q v ( t ) = FSW t ( { { h u ( t − 1 ) : u ∈ N ( v ) } } ) , q_v^{(t)} = \operatorname{FSW}_t\left(\left\{\left\{h_u^{(t-1)} : u \in\mathcal{N}(v)\right\}\right\}\right), q v ( t ) = FSW t ( { { h u ( t − 1 ) : u ∈ N ( v ) } } ) ,
and update the node representation by first concatenating the previous node state with the aggregated message, and then projecting the result back to the hidden dimension:
h v ( t ) = Φ t ( [ h v ( t − 1 ) ∥ q v ( t ) ] ) , h_v^{(t)} = \Phi_t\left(\left[h_v^{(t-1)} \mathbin{\|} q_v^{(t)}\right]\right), h v ( t ) = Φ t ( [ h v ( t − 1 ) ∥ q v ( t ) ] ) ,
where ∥ \| ∥ denotes concatenation and Φ t \Phi_t Φ t is a learned projection back to the hidden dimension.
A final FSW aggregation over all node representations is used for graph-level prediction. We refer to the resulting architecture as FSW-GNN .
A Bi-Lipschitz WL-Equivalent MPNN
We show that FSW-GNN can attain the full distinguishing power available to standard message-passing networks. For graphs with at most N N N vertices and d d d -dimensional input features, an FSW-GNN with T = N T=N T = N iterations and sufficiently large hidden dimension is, for almost every choice of its parameters, equivalent in expressive power to 1-WL. The proof uses the Finite Witness Theorem for multiset embeddings [2 ] .
The stronger result concerns metric preservation. Let Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d be compact. Under the same assumptions that guarantee WL-equivalence, FSW-GNN is bi-Lipschitz with respect to ρ D S \rho_{\mathrm{DS}} ρ DS on G ≤ N ( Ω ) G_{\leq N}(\Omega) G ≤ N ( Ω ) . If, additionally, 0 ∉ Ω 0 \notin\Omega 0 ∈ / Ω , it is also bi-Lipschitz with respect to TMD.
Long-Range Behavior
The motivation for bi-Lipschitz message passing is reflected particularly clearly in long-range experiments.
On the Ring task, we vary the problem radius from
2 ≤ r ≤ 15. 2 \leq r \leq15. 2 ≤ r ≤ 15.
FSW-GNN achieves perfect accuracy throughout the entire tested range, as shown in Figure 2.3 .
Figure 2.3. Performance on the Ring long-range task, including FSW-GNN.
We observe the same qualitative behavior on additional graph-transfer tasks. The different graph topologies considered in these experiments—CliquePath, Ring, and CrossRing—are illustrated in Figure 2.2 . In each case, information must be propagated from a source node to a target node located r r r steps away.
Figure 2.2. Additional graph-transfer topologies used in the long-range experiments.
On all three benchmarks, FSW-GNN is the only tested method to maintain 100% accuracy for all tested radii up to r = 15 r = 15 r = 15 , as shown in Figure 2.4 .
Figure 2.4. Performance on the CliquePath and CrossRing long-range graph-transfer tasks. FSW-GNN maintains 100% accuracy for all tested radii up to r = 15 r = 15 r = 15 .
The competing MPNNs begin to fail considerably earlier, with the precise failure point depending on the graph topology.
FSW-GNN is not completely immune to the difficulties of very deep message passing. Its performance eventually deteriorates when the required radius becomes sufficiently large; in our experiments this occurs only at substantially larger radii, beyond approximately r = 20 r = 20 r = 20 .
Thus, our claim is not that bi-Lipschitz aggregation eliminates oversquashing altogether. Rather, both the theoretical guarantees and the experiments indicate that FSW-GNN preserves useful information through many more message-passing iterations, and consequently fails substantially later than standard MPNNs on long-range problems.
On the Expressive Power of Sparse Geometric MPNNs
Yonatan Sverdlov; Nadav Dym
Motivated by applications in chemistry and other sciences, we study the expressive power of message-passing neural networks for geometric graphs, whose node features correspond to 3-dimensional positions. Recent work has shown that such models can separate generic pairs of non-isomorphic geometric graphs, though they may fail to separate some rare and complicated instances. However, these results assume a fully connected graph, where each node possesses complete knowledge of all other nodes. In contrast, often, in applications, every node only possesses knowledge of a small number of nearest neighbors. This paper shows that generic pairs of non-isomorphic geometric graphs can be separated by message-passing networks with rotation equivariant features as long as the underlying graph is connected. When only invariant intermediate features are allowed, generic separation is guaranteed for generically globally rigid graphs. We introduce a simple architecture, EGenNet, which achieves our theoretical guarantees and compares favorably with alternative architectures on synthetic and chemical benchmarks. Our code is available at GitHub.
Chapter 3
Introduction
Geometric graphs are graphs whose nodes are a ’position’ vector in R d \mathbb{R}^{d} R d (typically d = 3 d = 3 d = 3 ) and whose symmetries include the permutation symmetries of combinatorial graphs and translation and rotation of the node positions. Geometric graphs arise naturally in learning applications for chemistry, physical dynamics, and computer vision as natural models for molecules, particle systems, and 3D point clouds. These applications motivated the introduction of many learning models for geometric graphs, which were often inspired by ’standard’ graph neural networks (GNNs) for combinatorial graphs [SHW21; HCW+24; GBG21]. Subsequently, several theoretical works aimed primarily at understanding the expressive power and limitations of GNNs for geometric graphs.
GNNs for geometric graphs produce global graph features invariant to geometric graph symmetries. The expressive power of GNNs is typically assessed primarily by their ability to assign different global features to pairs of geometric graphs that are not geometrically isomorphic.
Recent research on this problem has uncovered several interesting results, mainly assuming that the graphs are full (an edge connects each pair of nodes). Under the full graph assumptions, several models for geometric GNNs are complete , that is, capable of separating all pairs of non-isomorphic graphs. However, this typically comes at a relatively high computational price. Examples of complete models include ( d − 1 ) (d-1) ( d − 1 ) -WL-based GNNs [DKR+23; HAGD23; LWHZ24], GNNs based on sub-graph aggregation [LWKZ24], and message passing models based on arbitrarily high-dimensional irreducible representations [TSK+18; GBG21; DM20; FBMD22]. This paper focuses on more efficient geometric GNNs: message-passing neural networks (MPNN) based on simple invariant and equivariant features. In [PC22; LWHZ24], it was shown that, even under the full graph assumption, invariant MPNN-based networks are not complete. In contrast, Li, Wang, Kang, and Zhang [LWKZ24] and Hordan, Amir, Gortler, and Dym [HAGD23] proved that these models are ’generically complete,’ which means that, under the full graph assumption, they are capable of separating most geometric graphs and only fail on a Lebesgue measure zero set of corner cases, which are difficult to separate.
In practical applications, the full graph assumption is often not fulfilled: to improve computational efficiency, geometric GNNs often operate on sparse graphs, where each node is only connected to a few nearest neighbors. Accordingly, the first question we address in this paper is:
Question 1. For which graphs, other than the full graphs, are geometric message-passing neural networks generically complete?
Our answer to this question is inspired by the results of [JBM+23]. They divided Message-passing-based networks for geometric graphs into Equivariant Geometric Graph Neural Networks (E-GGNNs), which use rotation equivariant hidden features, and Invariant Geometric Graph Neural Networks (I-GGNNs), which use rotation invariant hidden features, and showed that E-GGNNs have substantial advantages over I-GGNNs in terms of their expressive power.
Our answer to Question 1 is also strongly related to this dichotomy. We prove in Theorem 3.4.1 that E-GGNNs are generically complete for a given graph if and only if the graph is connected. This is visualized in Figure 3.1 . E-GGNNs will not be able to differentiate between the pair of geometric graphs in Figure 3.1 (A) as the underlying graph is disconnected but will be able to differentiate between the pairs in (B) and (C). In contrast, I-GGNNs will not be able to differentiate between the non-isomorphic pairs in Figure 3.1 (A)–(B), as the distances across all graph edges are the same. On the other hand, in (C), the distances along graph edges determine the positions of all nodes up to rotation and translation. In rigidity theory, such graphs are called ’generically globally rigid.’ In Theorem 3.3.1, we prove that I-GGNNs are generically complete if and only if the underlying graph is generically globally rigid. These two theorems give a complete answer to Question 1.
Figure 3.1. (A) and (B) are pairs of non-isomorphic geometric graphs with the same distances along edges. I-GGNN cannot distinguish such pairs, while E-GGNN can (generically) if the graph is connected, as in (B). Subplot (C) depicts an example of globally rigid graphs, where non-isomorphic geometric graphs do not share the same distances across edges (in the figure, the edge between pink and light green). Such examples can be separated by both I-GGNN and E-GGNN.
To guarantee generic separation, our theorems make the standard assumption [JBM+23] that the I-GGNN and E-GGNN we consider are maximally expressive, meaning that all aggregation and readout functions used throughout the networks are injective. Recent work has shown that relatively standard and simple I-GGNN can fulfill this assumption [2 ] . However, for E-GGNN, it is unclear how this assumption can be fulfilled. Accordingly, our second question is
Question 2. Can we devise a simple E-GGNN architecture that is maximally expressive hence, generically complete on connected graphs?
We answer this question affirmatively by devising a simple architecture, which we name EGenNet, and proving in Theorem 3.5.1 that it is maximally expressive (for generic graphs). This architecture is based on a simple message-passing architecture that resembles EGNN [SHW21] but uses multiple equivariant channels as in [LKG+23].
We experimentally find that EGenNet is highly successful in synthetic separation tasks and on several learning tasks involving medium-sized molecules, with a performance comparable to, or better than, state-of-the-art methods. These results indicate that simple, generically complete methods like EGenNet are sufficient for the tasks we considered. However, we acknowledge that there may be other tasks, perhaps involving molecules with non-trivial symmetry groups, where more complex geometric graph neural networks that offer full separation could be beneficial.
Contributions To summarize, our main results are
Appendix See the appendix for all full proofs and related work.
Setup: Geometric Graph Neural Networks and Rigidity Theory
Geometric Graphs. In our discussion of geometric graphs and GNNs, we loosely follow the definitions of [JBM+23]. We define a geometric graph as a pair G = ( A , X ) \mathcal{G} = (A, X) G = ( A , X ) . The matrix A ∈ { 0 , 1 } n × n A \in\{0,1\}^{n \times n} A ∈ { 0 , 1 } n × n is an adjacency matrix, which fully defines a combinatorial graph. The neighborhood of a node i i i is denoted by N i \mathcal{N}_i N i . Accordingly, we sometimes use statements like ’the graph A A A .’ We assume for simplicity that the graph nodes are the integers 1 , … , n 1,\ldots,n 1 , … , n . The matrix X = ( x 1 , … , x n ) ∈ R n × d X = (x_1,\ldots,x_n) \in\mathbb{R}^{n \times d} X = ( x 1 , … , x n ) ∈ R n × d denotes node positions. We denote the set of all geometric graphs with coordinates in R d \mathbb{R}^d R d by G ( d ) \mathcal{G}(d) G ( d ) , and the subset of graphs with ≤ N \le N ≤ N nodes by G N ( d ) \mathcal{G}_N(d) G N ( d ) . In Appendix H, we discuss incorporating additional node and edge features into our model and theoretical results.
The symmetry group of geometric graphs is the product of the permutation group S n S_n S n and the group of rigid motions E ( d ) \mathcal{E}(d) E ( d ) . The action of a permutation matrix P P P on a geometric graph is given by P G : = ( P A P T , P X ) P\mathcal{G} := (PAP^{\mathsf{T}}, PX) P G := ( P A P T , P X ) . A rigid motion in E ( d ) \mathcal{E}(d) E ( d ) is a rotation Q ∈ O ( d ) Q \in O(d) Q ∈ O ( d ) and a translation vector t ∈ R d t \in\mathbb{R}^d t ∈ R d . They act on X X X by applying the rotation and translation to all its coordinates, x i ↦ Q x i + t x_i \mapsto Qx_i + t x i ↦ Q x i + t .
Two geometric graphs G \mathcal{G} G and H \mathcal{H} H are geometrically isomorphic if there exists a transformation in S n × E ( d ) S_n \times\mathcal{E}(d) S n × E ( d ) which can be applied to G \mathcal{G} G to obtain H \mathcal{H} H . We will say that a function F F F defined on G ( d ) \mathcal{G}(d) G ( d ) is invariant if F ( G ) = F ( H ) F(\mathcal{G}) = F(\mathcal{H}) F ( G ) = F ( H ) for all G \mathcal{G} G and H \mathcal{H} H which are geometrically isomorphic.
The expressive power of an invariant model F F F is closely related to its ability to separate pairs G , H \mathcal{G}, \mathcal{H} G , H that are not geometrically isomorphic: clearly, if F F F cannot separate G \mathcal{G} G from H \mathcal{H} H , then F F F will not be a good approximation for functions f f f for which f ( G ) ≠ f ( H ) f(\mathcal{G}) \ne f(\mathcal{H}) f ( G ) = f ( H ) . Conversely, if F F F can separate all pairs of geometric graphs, then it can be used to approximate all continuous invariant [DG24; HAGD23] and equivariant [VHS+21; HAD24] functions. In Appendix F, we discuss the connection between approximation and separation in the context of our results.
Constructing an F F F which can separate all geometric graphs is at least as challenging as distinguishing all combinatorial graphs (by setting all node features to zero). Without any assumptions on the point cloud, achieving this is infeasible. To address this, we employ the concept of generic geometric graphs which originates from rigidity theory.
Generic separation and Rigidity Theory As discussed in the introduction, our paper focuses on generic separation. The results we present in this section are primarily influenced by rigidity theory, so we also use this field’s common notion of ’generic’. We will say that X ∈ R d × n X \in\mathbb{R}^{d \times n} X ∈ R d × n is generic if it is not the root of any non-zero multivariate polynomial p : R d × n → R p : \mathbb{R}^{d \times n} \to\mathbb{R} p : R d × n → R with rational coefficients. The genericity assumption guarantees that X X X avoids many degeneracies, making separation easier. For example, for generic X X X all pairs of points have different norms, and every d d d tuple of points x i 1 , … , x i d x_{i_1}, \ldots, x_{i_d} x i 1 , … , x i d is full rank because ∥ x i ∥ 2 − ∥ x j ∥ 2 \lVert x_i \rVert^2 - \lVert x_j \rVert^2 ∥ x i ∥ 2 − ∥ x j ∥ 2 and det [ x i 1 , … , x i d ] \det[x_{i_1}, \ldots, x_{i_d}] det [ x i 1 , … , x i d ] are polynomials with natural coefficients. We note that Lebesgue almost all point clouds in R d × n \mathbb{R}^{d \times n} R d × n are generic [Mit20].
A globally rigid graph is a central topic in rigidity theory:
Definition 3.2.1 (Globally rigid). ([GGJ22]) A geometric graph G = ( A , X ) \mathcal{G} = (A, X) G = ( A , X ) is globally rigid in R d \mathbb{R}^d R d , if the only X ′ ∈ R n × d X' \in\mathbb{R}^{n \times d} X ′ ∈ R n × d which satisfy that ∥ x i − x j ∥ = ∥ x i ′ − x j ′ ∥ \lVert x_i - x_j \rVert= \lVert x'_i - x'_j \rVert ∥ x i − x j ∥ = ∥ x i ′ − x j ′ ∥ for all edges ( i , j ) (i,j) ( i , j ) in A A A , are those that are related to X X X by a rigid motion.
A combinatorial graph A A A is generically globally rigid in R d \mathbb{R}^d R d , if the geometric graph G = ( A , X ) \mathcal{G} = (A, X) G = ( A , X ) is globally rigid in R d \mathbb{R}^d R d for every generic X ∈ R n × d X \in\mathbb{R}^{n \times d} X ∈ R n × d .
The geometric graphs in Figure 3.1 (A)-(B) are not globally rigid, as they have the same distances along edges but are not related by a rigid motion. The underlying combinatorial graph, disconnected in (A) or line graph in (B), is not generically globally rigid. In contrast, the underlying combinatorial graph in (C) is generically globally rigid. Another simple example of a generically globally rigid graph is the full graph. Additional non-trivial examples can be found in [JT22; CW05]. A necessary condition for generic global rigidity of a graph with n ≥ d + 2 n \ge d+2 n ≥ d + 2 nodes in R d \mathbb{R}^d R d is to be d + 1 d+1 d + 1 connected1 , which explains why the graphs in (A)-(B) are not generically globally rigid.
Note that global rigidity definitions focus on reconstructing X X X when A A A is known. In the study of GGNNs, we are interested in identifying both X X X and A A A :
Definition 3.2.2 . Let d d d be a natural number. Following [LWKZ24], we will say that an invariant function F F F defined on G ( d ) \mathcal{G}(d) G ( d ) , identifies a geometric graph G \mathcal{G} G , if for every G ^ ∈ G ( d ) \widehat{\mathcal{G}} \in\mathcal{G}(d) G ∈ G ( d ) , we have that F ( G ) = F ( G ^ ) F(\mathcal{G}) = F(\widehat{\mathcal{G}}) F ( G ) = F ( G ) if and only if G \mathcal{G} G and G ^ \widehat{\mathcal{G}} G are geometrically isomorphic.
We say that F F F generically identifies a combinatorial graph A A A in R d \mathbb{R}^d R d , if F F F can identify G = ( A , X ) \mathcal{G} = (A, X) G = ( A , X ) for every generic X ∈ R n × d X \in\mathbb{R}^{n \times d} X ∈ R n × d .
We say that F F F generically fails to identify a combinatorial graph A A A in R d \mathbb{R}^d R d , if F F F does not identify G = ( A , X ) \mathcal{G} = (A, X) G = ( A , X ) for every generic X ∈ R n × d X \in\mathbb{R}^{n \times d} X ∈ R n × d .
Note that the fact that F F F does not generically identify A A A only guarantees that there exists a generic X X X so that F F F fails to identify ( A , X ) (A, X) ( A , X ) . The statement ‘F F F generically fails to identify’ is, a priori, significantly stronger. Also note that invariant F F F separates all non-isomorphic graphs (is complete ) if and only if it identifies all graphs. As discussed in the introduction, our goal will be to classify which graphs are generically identifiable by standard message-passing-based networks for geometric graphs, which can be classified as either I-GGNN or E-GGNN. We will now define these concepts.
E-GGNN In the context of combinatorial graphs (with no geometric information), Gilmer, Schoenholz, Riley, Vinyals, and Dahl [GSR+17] and Jogl, Thiessen, and Gärtner [JTG23] and Veličković [Vel22] showed that many famous graph neural networks are instantiations of MPNNs. The goal of [JBM+23] was to show, analogously, that many neural networks for geometric graphs follow a generalized MPNN framework. Loosely following [JBM+23], we define Equivariant Geometric Graph Neural Networks (E-GGNN) to be functions defined by a sequence of layers that propagates vector features from iteration t t t to t + 1 t+1 t + 1 via a learnable function f ( t ) f_{(t)} f ( t ) (at initialization we set v j ( 0 ) = 0 v_j^{(0)}=0 v j ( 0 ) = 0 ).
v i ( t + 1 ) = f ( t ) ( { { v j ( t ) , x i − x j ∣ j ∈ N i } } ) . (1) v_i^{(t+1)}=f_{(t)}\left(\left\{\left\{v_j^{(t)},\boldsymbol{x}_i-\boldsymbol{x}_j\mid j\in\mathcal{N}_i\right\}\right\}\right). \tag*{(1)} v i ( t + 1 ) = f ( t ) ( { { v j ( t ) , x i − x j ∣ j ∈ N i } } ) . (1)
To ensure the equivariance of the construction, the function f ( t ) f_{(t)} f ( t ) is required to be rotation equivariant. Translation invariance is implicitly guaranteed by the translation invariance of x i − x j \boldsymbol{x}_i-\boldsymbol{x}_j x i − x j , and the requirement that f ( t ) f_{(t)} f ( t ) is defined on multi-sets implicitly enforces permutation equivariance. We will often refer to f ( t ) f_{(t)} f ( t ) as an aggregation function .
We note that for the sake of simplicity, we don’t require a combine step that incorporates v i ( t ) v_i^{(t)} v i ( t ) in the process of computing v i ( t + 1 ) v_i^{(t+1)} v i ( t + 1 ) since we find that this step has little influence on our theoretical and empirical results. We also note that the vector v i ( t ) v_i^{(t)} v i ( t ) is not necessarily d d d dimensional, and the action of O ( d ) O(d) O ( d ) on v i ( t ) v_i^{(t)} v i ( t ) is allowed to change from layer to layer. For example, in the EGenNet architecture we will introduce later on, each v i ( t ) v_i^{(t)} v i ( t ) will be a d × m d\times m d × m matrix, and the action of O ( d ) O(d) O ( d ) will be multiplication on each of the m m m coordinates.
After T T T iterations of E-GGNNs, a permutation, rotation, and translation invariant feature vector can be obtained via a two-step process involving a rotation-invariant function f i n v f_{\mathrm{inv}} f inv and a multi-set ReadOut function:
s i = f i n v ( v i ( T ) ) , s g l o b a l = ReadOut { { s 1 , … , s n } } (2) s_i=f_{\mathrm{inv}}\left(v_i^{(T)}\right),\qquad s^{\mathrm{global}}=\operatorname{ReadOut}\left\{\left\{s_1,\ldots,s_n\right\}\right\} \tag*{(2)} s i = f inv ( v i ( T ) ) , s global = ReadOut { { s 1 , … , s n } } (2)
As shown by [JBM+23], many popular geometric graph neural networks can be seen as instances of E-GGNNs with a specific choice of functions f ( t ) f_{(t)} f ( t ) and readout functions. Examples include EGNN [SHW21], TFN [TSK+18], GVP [JES+20], and MACE [BKS+22]. In § 5.4.1 , we present EGenNet, our instantiation of this framework.
I-GGNNs An important subset of E-GGNNs are invariant GGNNs (I-GGNNs). These networks only maintain scalar features s i ( t ) s_i^{(t)} s i ( t ) and replace x i − x j \boldsymbol{x}_i-\boldsymbol{x}_j x i − x j with its norm to obtain
s i ( t + 1 ) = f ( t ) ( { { s j ( t ) , ∥ x i − x j ∥ ∣ j ∈ N i } } ) (3) s_i^{(t+1)}=f_{(t)}\left(\left\{\left\{s_j^{(t)},\left\|\boldsymbol{x}_i-\boldsymbol{x}_j\right\| \mid j\in\mathcal{N}_i\right\}\right\}\right) \tag*{(3)} s i ( t + 1 ) = f ( t ) ( { { s j ( t ) , ∥ x i − x j ∥ ∣ j ∈ N i } } ) (3)
A global invariant feature is obtained via a permutation invariant readout function as in equation (2) . An advantage of I-GGNNs is that there are no rotation equivariance constraints on f ( t ) f(t) f ( t ) . Therefore, any standard message-passing neural network that can process continuous edge weights can be employed as an I-GGNN by setting the edge weights to ∥ x i j ∥ \lVert x_{ij}\rVert ∥ x ij ∥ . Examples of I-GGNNs in the literature include a variant of EGNN used for invariant tasks [SHW21], SchNet [SSK+18], DimeNet [GGG20] and SphereNet [LWL+21].2 Our definition of I-GGNN follows the definition from [PC22; LWHZ24]. In Appendix § 5.13 , we explain that the definition of I-GGNN [JBM+23] has a seemingly minor difference that renders it significantly stronger.
Maximally expressive E-GGNN and I-GGNNs. The separation power of E-GGNN and I-GGNN architectures depends on the specific instance used. For example, if we choose the final readout function as the zero function, all geometric graphs will be assigned the same value. To address this, it is customary to assume that all functions f ( t ) f(t) f ( t ) , as well as f i n v f_{\mathrm{inv}} f inv and the readout function, are injective, up to the ambiguities these functions have by construction. Following [JBM+23], we will call E-GGNN (respectively I-GGNN) architectures, which are composed of injective functions, maximally expressive E-GGNN (respectively I-GGNN) architectures. We note that the existence of the injective functions necessary for maximal expressivity follows from set-theoretic considerations, similar to those used by [WHB+24]. The question of whether practical differentiable aggregation functions are injective is discussed in § 5.4 .
Expressive power of I-GGNN
Our next goal is to analyze the separation abilities of I-GGNN.
Theorem 3.3.1. [expressive power of I-GGNN] Let d d d be a natural number. Let F F F be an I-GGNN. Let A A A be a graph not generically globally rigid on R d \mathbb{R}^{d} R d . Then, F F F generically fails to identify A A A .
Conversely, if A A A is generically globally rigid on R d \mathbb{R}^{d} R d and F F F is a maximally expressive I-GGNN with depth T = 1 T=1 T = 1 , then F F F generically identifies A A A .
Proof idea. if A A A is not globally rigid, then according to [GHT10], for every generic X X X there exists X ′ X' X ′ for which distances along edges are preserved, but X X X , X ′ X' X ′ are not related by a rigid motion. It’s easy to see any I-GGNN, which is based only on distances along edges, can’t separate such graphs.
On the other hand, generic geometric graphs have distinct pairwise distances. The full proof (see Appendix § 5.15 ) shows that this can be used to reconstruct the combinatorial graph A A A . By global rigidity, we can then reconstruct X X X , up to rigid motion.
Note that Theorem 3.3.1 is a generalization of [HAGD23] that showed maximally expressive I-GGNN generically identify full graphs. We now showed this is true for all generically globally rigid graphs.
I-GGNN generic separation by power graph preprocessing A natural strategy to overcome the lack of generic separation for graphs that are not generically globally rigid, as proven in Theorem 3.3.1, is to replace these graphs with graphs that are generically globally rigid. This replacement can be done using power graphs: For a given graph G \mathcal{G} G and natural number k k k , the power graph G k \mathcal{G}^{k} G k is defined to have the same nodes as the original graph G \mathcal{G} G . A pair of nodes are connected by an edge in the power graph if and only if there is a path between them in the original graph of length ≤ k \leq k ≤ k .
In [JT22; CW05], it was shown that if the original graph G \mathcal{G} G is connected, then the power graph G d + 1 \mathcal{G}^{d+1} G d + 1 is generically globally rigid in R d \mathbb{R}^{d} R d . Accordingly, when the input graph G \mathcal{G} G is connected but not generically globally rigid, our theory suggests that applying I-GGNN not to the original graph but to its d + 1 d+1 d + 1 power may benefit separation. An experiment illustrating this idea is presented in Table 3.1 .
Table 3.1. E-GGNN models like EGenNet can separate the two pairs of connected graph in Figure 3.2 . I-GGNN models cannot separate unless power graphs are considered.
Expressive power of E-GGNN
When using E-GGNN rather than I-GGNN, we will have generic separation if and only if the graph is connected.
Theorem 3.4.1. [expressive power of E-GGNN] Let d d d be a natural number. Let F F F be an E-GGNN. Let A A A be a disconnected graph. Then, F F F generically fails to identify A A A . Conversely, if A A A is connected and F F F is a maximally expressive E-GGNN with depth T ≥ d + 1 T \geq d+1 T ≥ d + 1 , then F F F generically identifies A A A .
Proof idea. Suppose a geometric graph G \mathcal{G} G isn’t connected. We can apply two distinct rigid motions to two connected components and obtain a new geometric graph, G ′ \mathcal{G}' G ′ , which cannot be separated from G \mathcal{G} G using E-GGNN. This is illustrated in Figure 3.1 (A). Conversely, in the appendix Theorem G, we show that the features s i s_i s i obtained after d + 1 d+1 d + 1 iterations of a maximally expressive E-GGNN can encode the multi-set of all distances ∥ x i − x j ∥ \lVert x_i-x_j\rVert ∥ x i − x j ∥ for all j j j which are d + 1 d+1 d + 1 -hop neighbors of i i i (a similar claim was shown by [JBM+23]). Thus, d + 1 d+1 d + 1 iterations of maximally expressive E-GGNN can simulate the application of a single iteration of an I-GGNN on the d + 1 d+1 d + 1 power graph. The proof then follows from the fact that the ( d + 1 ) (d+1) ( d + 1 ) power graph is generically globally rigid [JT22; CW05].
An immediate result of this theorem is that generic graph separation is a local problem with a problem radius of at most d + 1 d+1 d + 1 [AY20].3 We note that the generic assumption allows us to reconstruct the coordinates of the nodes and the combinatorial graph. Without the generic assumption, even models based on 2 2 2 -WL that are complete on full graphs cannot reconstruct the combinatorial graph. Setting all positions to zero nullifies the geometric information, and 2 2 2 -WL is insufficient to distinguish all combinatorial graphs.
Building a generically maximally expressive E-GGNN
To obtain generic separation guarantees as in Theorems 3.3.1, 3.4.1 for a practical instantiation of E-GGNN (or I-GGNN), we will need to show that the instantiation is fully expressive. This type of question was studied rather extensively for combinatorial graphs [2 ACI+22] , and these results can be used to show that some simple realizations of I-GGNN are maximally expressive. We discuss this in more detail in Appendix C.
For E-GGNN, the requirement that f ( t ) f_{(t)} f ( t ) is simultaneously injective (up to permutation) and equivariant to rotations seems to be challenging to attain with a simple E-GGNN construction, and indeed, [JBM+23] who suggested the notions of maximally expressive E-GGNN, did not give any indication of how they could be constructed in practice. Moreover, in Appendix D, we show that a maximally expressive E-GGNN can separate all pairs of full graphs and separate geometrically all pairs of connected graphs4 , indicating that constructing such a network may be computationally expensive, as currently, such guarantees are only achieved by networks of relatively high complexity based on 2 2 2 -WL subgraph aggregation techniques.
In our setting, however, the problem is less severe since, from the outset, we are only interested in analyzing network behavior on generic graphs. Accordingly, we would like to require our E-GGNN to be maximally expressive only for generic graphs. It turns out that this goal can be achieved with a straightforward E-GGNN architecture, which we name EGenNet. Our next step will be to present this architecture.
The EGenNet architecture
Our goal in this section is to build a maximally expressive EGNN. For this, we need to implement injective readout and aggregation functions f ( t ) f_{(t)} f ( t ) , f i n v f_{\mathrm{inv}} f inv , ReadOut (see equations (1) –(2) ). The EGenNet architecture we suggest resembles an EGNN architecture [SHW21] with multiple equivariant channels as in [LKG+23]. The EGenNet architecture depends on two hyper-parameters: depth T T T (the term ‘depth’ refers to the number of MPNN iterations) and channel number C C C . The input to the architecture is a geometric graph ( A , X ) (\mathbf{A},\mathbf{X}) ( A , X ) .
Aggregation The architectures then maintains, at each iteration t t t and node i i i , an ’equivariant’ node feature with C C C channels
v i ( t ) = ( v i , 1 ( t ) , … , v i , C ( t ) ) , v i , c ( t ) ∈ R d , c = 1 , … , C (4) v_i^{(t)} = (v_{i,1}^{(t)}, \ldots, v_{i,C}^{(t)}), \quad v_{i,c}^{(t)} \in\mathbb{R}^{d}, c = 1, \ldots, C \tag*{(4)} v i ( t ) = ( v i , 1 ( t ) , … , v i , C ( t ) ) , v i , c ( t ) ∈ R d , c = 1 , … , C (4)
These feature values are initialized to zero and are updated via the rotation and permutation equivariant aggregation formula
v i , q ( t + 1 ) = ∑ j ∈ N i ( ϕ ( t , q , 0 ) ( ∥ x i − x j ∥ , ∥ v j ∥ ) ( x i − x j ) + ∑ c = 1 C ϕ ( t , q , c ) ( ∥ x i − x j ∥ , ∥ v j ∥ ) v j , c ( t ) ) (5) v_{i,q}^{(t+1)} = \sum_{j \in\mathcal{N}_i} \left( \phi^{(t,q,0)}(\lVert x_i - x_j \rVert, \lVert v_j \rVert)(x_i - x_j) + \sum_{c=1}^{C} \phi^{(t,q,c)}(\lVert x_i - x_j \rVert, \lVert v_j \rVert)v_{j,c}^{(t)} \right) \tag*{(5)} v i , q ( t + 1 ) = j ∈ N i ∑ ( ϕ ( t , q , 0 ) (∥ x i − x j ∥ , ∥ v j ∥) ( x i − x j ) + c = 1 ∑ C ϕ ( t , q , c ) (∥ x i − x j ∥ , ∥ v j ∥) v j , c ( t ) ) (5)
where ∥ v j ∥ \lVert v_j \rVert ∥ v j ∥ is the C C C dimensional vector containing the norms of the vectors v j , c v_{j,c} v j , c , c = 1 , … , C c = 1, \ldots, C c = 1 , … , C .
Construction of f i n v f_{\mathrm{inv}} f inv and ReadOut After T T T iterations, we obtain rotation and permutation invariant node features s i ∈ R C s_i \in\mathbb{R}^{C} s i ∈ R C and global feature s g l o b a l ∈ R C s^{\mathrm{global}} \in\mathbb{R}^{C} s global ∈ R C via
s i , q = ∥ ∑ c = 1 C θ c , q v i , c ( T ) ∥ 2 , s q g l o b a l = ∑ i = 1 n ϕ g l o b a l ( q ) ( s i ) , q = 1 , … , C (6) s_{i,q} = \left\lVert\sum_{c=1}^{C} \theta_{c,q}v_{i,c}^{(T)} \right\rVert^2,\qquad s_q^{\mathrm{global}} = \sum_{i=1}^{n} \phi_{\mathrm{global}}^{(q)}(s_i),\quad q = 1,\ldots,C \tag*{(6)} s i , q = c = 1 ∑ C θ c , q v i , c ( T ) 2 , s q global = i = 1 ∑ n ϕ global ( q ) ( s i ) , q = 1 , … , C (6)
Where θ \theta θ is a learnable matrix, and the functions ϕ g l o b a l ( q ) , ϕ ( t , q , c ) \phi_{\mathrm{global}}^{(q)}, \phi^{(t,q,c)} ϕ global ( q ) , ϕ ( t , q , c ) are all fully connected neural networks with a single layer, an output dimension of 1, and an analytic non-polynomial activation (in our implementation, TanH activation). The complexity of EGenNet is linear in N ⋅ C N \cdot C N ⋅ C , as we discuss in Appendix § 5.15 .2.
The next theorem shows that EGenNet is generically maximally expressive:
Theorem 3.5.1. Let d , N , T d,N,T d , N , T be natural numbers. Let F F F be a maximally expressive E-GGNN of depth T T T , and let F θ F_\theta F θ denote the EGenNet architecture with depth T T T and C = 2 N d + 1 C = 2Nd + 1 C = 2 N d + 1 channels. Then, for Lebesgue almost every choice of network parameters θ \theta θ , we have that for all generic G , G ^ ∈ G ^ N ( d ) G,\widehat{G} \in\widehat{\mathcal{G}}_N(d) G , G ∈ G N ( d ) ,
F ( G ) = F ( G ^ ) ⟺ F θ ( G ) = F θ ( G ^ ) F(G) = F(\widehat{G}) \Longleftrightarrow F_\theta(G) = F_\theta(\widehat{G}) F ( G ) = F ( G ) ⟺ F θ ( G ) = F θ ( G )
Proof idea. The proof relies on the methodology developed by Dym and Gortler [DG24] and Amir, Gortler, Avni, Ravina, and Dym [2 ] and Hordan, Amir, Gortler, and Dym [HAGD23]. We note that each step in EGenNet employs C C C identical analytic functions indexed by q = 1 , … , C q = 1, \ldots, C q = 1 , … , C , with independent parameters. Dym and Gortler [DG24] and Amir, Gortler, Avni, Ravina, and Dym [2 ] showed that in this setting, if every pair of non-isomorphic inputs can be separated by some parameter of the original function, then C = 2 N d + 1 C = 2Nd + 1 C = 2 N d + 1 copies of this function with random parameters will suffice to separate all pairs uniformly . The proof is then reduced to showing that any pair of non-isomorphic inputs can be separated by some parameter vector, which we prove in the appendix G.1.
Joining this theorem with our theorem on maximally expressive E-GGNN, we deduce
Corollary 3.5.2. Let d , N d,N d , N be natural numbers. let F θ F_\theta F θ denote the EGenNet architecture with depth T = d + 1 T=d+1 T = d + 1 and C = 2 N d + 1 C=2Nd+1 C = 2 N d + 1 channels. Then for Lebesgue almost every θ \theta θ , for every pair of graphs G , G ′ ∈ G N ( d ) \mathcal{G},\mathcal{G}' \in\mathcal{G}_N(d) G , G ′ ∈ G N ( d ) which are generic and connected,
F θ ( G ) = F θ ( G ^ ) ⟺ G , G ^ are geometrically isomorphic F_\theta(\mathcal{G})=F_\theta(\hat{\mathcal{G}}) \Longleftrightarrow\mathcal{G},\hat{\mathcal{G}} \text{ are geometrically isomorphic} F θ ( G ) = F θ ( G ^ ) ⟺ G , G ^ are geometrically isomorphic
Note that by [JBM+23], theorems 20, 22, the dimension of any injective embedding must be at least n d − d ( d − 1 ) 2 nd-\frac{d(d-1)}{2} n d − 2 d ( d − 1 ) , showing our embedding dimension 2 n d + 1 2nd+1 2 n d + 1 is optimal up to factor of 2 (when n ≫ d n \gg d n ≫ d ).
Experiments
In the experiments section, we have three main objectives. First, we aim to demonstrate that invariant models are indeed strictly less expressive than equivariant models and that power graphs help to mitigate this gap. Second, we seek to show that our model can distinguish challenging graph examples, including those requiring multiple layers for separation, thereby avoiding the bottleneck phenomenon encountered in other geometric models. Finally, we aim to demonstrate the improved performance of our model on real-world benchmarks.
Separation experiments
Enhancing Model Expressiveness: Power Graphs in Invariant vs. Equivariant Comparisons We begin with experiments to corroborate our theoretical findings. We consider examples where the graph A A A is a line graph, which is connected but not generically globally rigid. We choose two pairs of geometric line graphs, Pair a and Pair b, depicted in Figure 3.2 . These pairs have identical distances along graph edges but are not isomorphic. Therefore, they cannot be distinguished by any I-GGNN but (at least when generic) can be distinguished by E-GGNN and our EGenNet architectures. Following the protocol from [JBM+23], we construct binary classification problems from Pair a and Pair b and apply our E-GGNN architecture EGenNet, as well as three popular I-GGNN architectures: SchNet [SSK+18], DimeNet [GGG20], and SphereNet [LWL+21].
Figure 3.2. From Left to Right: k k k -chain pair graphs that are non-generic and require ∼ k / 2 \sim k/2 ∼ k /2 blocks for separation, Pair a in which 2-power graph is enough for I-GGNN to distinguish and Pair b in which 3-power graph is necessary and sufficient for separation.
The results are shown in the first column in Table 3.1 . As expected, EGenNet perfectly separates pairs a and b, while SchNet fails to separate both. Note that DimeNet and SphereNet separate Pair a but not Pair b. This is because DimeNet and SphereNet compute angles at edges, which utilize information not contained in the graph G \mathcal{G} G but only in the power graph G 2 \mathcal{G}^2 G 2 . Indeed, distances between two-hop neighbors are sufficient for discriminating between the two geometric graphs in Pair a but not in Pair b, where the distance between the three-hop red and orange nodes is needed for separation. Next, as we suggested in § 5.2 , we apply the same algorithms to the powers of the original graph. As shown in Table 3.1 , this improves the separation capabilities of the I-GGNN models. All invariant models succeed when the third power is taken, as expected since a connected graph’s d + 1 d + 1 d + 1 power (here d = 2 d = 2 d = 2 ) is generically globally rigid.
Unpacking the Bottleneck: Layer Requirements for Complex Graph Separation
k-chains Next, we consider the k k k chain pairs suggested in [JBM+23] and illustrated in the left of Figure 3.2 . By design, this is a pair of graphs with k k k nodes, which can only be separated by k − 1 k - 1 k − 1 hop distances. In [JBM+23], it was shown that a depth of at least ⌊ k / 2 ⌋ + 1 \lfloor k/2 \rfloor+ 1 ⌊ k /2 ⌋ + 1 is necessary to achieve separation by E-GGNN in this setting. Note that as most nodes in the chain graphs are on the same line, this point cloud is not generic, so the fact that for large k k k , more than d + 1 d + 1 d + 1 iterations are needed does not contradict our theory. In our experiments, we include two variations of SchNet [SSK+18]: full graph and global feature. Those variations use all pairwise distances, thus showing better results than other invariant models. In the first experiment, we set k = 4 k = 4 k = 4 . Theoretically, 3 blocks are needed for propagating the information, and 2 blocks are insufficient. As shown in Appendix § 5.2 , we succeed with a probability of 1 over 10 trials with 3 blocks. For checking long-term dependencies, we choose k = 12 k = 12 k = 12 again; 7 blocks are needed, and 6 is insufficient. We examined all models with 7, 8, 9, 10, 11 blocks, each 10 times. As shown in 3.4, we are the only ones succeeding with 7 blocks, and all other models need a much higher number of blocks. This may suggest our model is less influenced by over-squashing [AY20]. In the appendix, Table 3.7 , we add a time comparison table for all models using the 128 dimensional feature.4
Table 3.7. Runtime comparison on the 12-chain problem. Lower values indicate faster execution.
Hard examples In Table 3.5 in the appendix, we show that our model can also perfectly separate challenging examples from [PWB+20; PC22], using the experiment setup taken from [HAGD23]. These geometric graph pairs cannot be separated by I-GGNN, even on full graphs, and are also difficult to separate using equivariant methods like EGNN.
Table 3.5. Accuracy on challenging point-cloud benchmarks [PC22; PWB+20]. Higher values are better; 0.5 corresponds to chance-level performance.
Chemical property experiments Finally, we check our model’s ability to learn regression-invariant real-world chemical properties, a critical task in computational chemistry and material science. Learning chemical properties accurately is essential for predicting the behaviors of compounds, enabling advancements in drug discovery, material design, and environmental science. By effectively capturing these properties, the model demonstrates its potential to assist in solving complex chemical problems, where precise property prediction can reduce experimental costs and guide targeted research. As in [ZHA+24], we compare to the following models: Random Forest [Bre01], LSTM [HS97], Transformer [VSP+17], GIN [XHLJ18], GIN+VN [HFZ+20], ChemProp [HGC+23], GraphGPS [RGD+22], SchNet [SSK+18], DimeNet++ [GGMG20], GemNet [GBG21], PaiNN [SUG21], ClofNet [DZD+22], LEFTNet [DWF+24], and Marcel [ZHA+24]. Our task is learning chemical property prediction on three datasets: Drugs [AG22], Kraken [GdPF+22], and BDE [MSH+18]. Drugs is the largest and most challenging dataset, containing 560K samples; BDE is a medium-sized dataset with 80K samples; and Kraken is the smallest, with 22K samples. The baseline model performance, data split, and training procedure are taken from [ZHA+24]. In particular, we ran each experiment three times and reported the model’s test error with the best validation performance. In Table 3.6 in the appendix, we report the mean and standard deviation.
Table 3.6. Raw performance data (mean ± \pm ± standard deviation) of 1D, 2D, 3D, and conformer ensemble MRL models all taken from [ZHA+24] in terms of absolute test error. Our model is in the 3D category, named EGenNet.
As we see in Table 3.2 , we show comparable or improved results in all tasks. For example, in BDE, we have an improvement of 78%, in Kraken property burL 33%, and in the property L 48%. Remarkably, our method outperforms competing more complex methods like GemNet, which employs high-order irreducible representations. This can be explained by the generic completeness of our method, coupled with results we will show in the appendix Table 3.8 , which show that these datasets have a small number of norm-repetitions, indicating that they are ‘close to generic’ .
Table 3.2. Performance of 1D, 2D, and 3D baseline MRL models and the best results from ensemble learning strategies on 3D GNNs. The metric used is the Mean Absolute Error (MAE, ↓ \downarrow ↓ ). The bold indicates the best-performing model, while <u>underlined</u> denotes the second-best.
Table 3.8. Fraction of repeated norms at different thresholds across datasets.
Limitation and Future Work
In this work, we characterized the generic expressive power of I-GGNN and E-GGNN, showed that EGenNet is a generically maximally expressive E-GGNN, and showed the effectiveness of this architecture on chemical regression tasks. A limitation of our work is that it may struggle with symmetric and non-generic input. Future work could consider assessing the expressive power of non-MPNN models like geometric 2-WL on sparse graphs to address such input efficiently.
Appendix
This section discusses related work that is not discussed in the main text.
Equivariant models In the context of learning on geometric graphs, models should be equivariant to the joint action of permutations, rotations, and translations. Besides the models discussed in the paper, some prominent models with this equivariant structure include Cormorant [AHK19], Vector Neurons [DLD+21], and SE(3 3 3 ) transformers [FWFW20].
Completeness, universality and generic completeness The main text discusses related works on completeness focused on methods coming from the geometric learning community. Other complete methods, generally using similar ideas, appeared independently in the more chemistry-oriented community[Kur23; WK23; Sha16; NPC20]. Completeness in 2 D 2D 2 D was achieved in [BKF22], and completeness with respect to rigid motions (without permutations) was achieved by Comenet [WLL+22]. Comenet also uses some genericity assumptions (e.g., that there is a well-defined notion of nearest neighbors).
The main text discussed how generic completeness can be achieved using I-GGNN when the graph is globally rigid. A similar result can be obtained when considering the list of all distances [GGJ22]. Generic completeness for full graphs can also be obtained using more complex distance-based features [WMP+22] or shape principal axes [PAS+21] (under the generic assumption that the principal eigenvalues are distinct).
Comparison with [WZ22] Generic completeness without the full graph assumption, employing only the assumption that local neighborhoods are not symmetric, is discussed by [WZ22]. However, there are notable differences in key aspects of the results. Specifically, Theorem 4.2 in [WZ22] demonstrates that their proposed equivariant model can identify generic molecules using L L L message-passing iterations, where L L L is the graph’s diameter. In contrast, our Theorem 3.4.1 requires only d + 1 d+1 d + 1 iterations (where d = 3 d=3 d = 3 in typical applications, resulting in a total of just four iterations).
Moreover, while Theorem 4.2 in [WZ22] focuses on recovering the 3 D 3D 3 D structure, it does not address the combinatorial structure. Our Theorem 3.4.1, however, guarantees the recovery of both combinatorial and geometric structures in the generic case.
Additionally, our Theorem 3.3.1 precisely describes the conditions under which invariant models are, and are not, generically complete—a result not covered in [WZ22].
Implementation details
Here, we detail the hyper-parameter choice. For all experiments, we use AdamW optimizer [LH17]. We use a varying weight decay of 1 e − 4 1\mathrm{e}^{-4} 1 e − 4 , 1 e − 5 1\mathrm{e}^{-5} 1 e − 5 and learning rate of 1 e − 4 1\mathrm{e}^{-4} 1 e − 4 . Reduce On Plateau scheduler is used with a rate decrease of 0.75 with the patience of 15 epochs. We use two blocks with 40 channels in the challenging point cloud separation experiments. For k k k -chain experiments, we repeat each experiment 10 times, with 10 different seeds, and set the number of channels to be 60 and use a varying number of blocks, as detailed in the table. For power graph experiments, we repeat each experiment 10 times, with 10 different seeds, and set the number of channels to be 60 and 3 blocks. For chemical property experiments, the entire procedure is taken from [ZHA+24]: each dataset is partitioned randomly into three subsets: 70% for training, 10% for validation, and 20% for test. We set the number of channels to be 256 and used 6 blocks. Each model is trained over 1,500 epochs. Experiments are repeated three times for all eight regression targets, and the results reported correspond to the model that performs best on the validation set in terms of Mean Absolute Error (MAE). In contrast to the experiments in [ZHA+24], we didn’t consider the EE dataset as it’s currently not public on GitHub. We utilize PyTorch and PyTorch-Geometric to implement all deep learning models. All experiments are conducted on servers with NVIDIA A40 GPUs, each with 48GB of memory. All dataset details, including statistics, can be found in [ZHA+24].
Building perfect I-GGNN
Here, we describe in detail how to construct a perfect I-GGNN. We will assume that we consider geometric graphs with up to N N N nodes. The main ingredients needed are aggregation functions and ReadOut function that are injective on the space of all multisets of size ≤ N \le N ≤ N . [2 ] proved that the sum of projections and analytic activation can be used to construct such layers. Formally, let M \mathcal{M} M be a semi-analytic set of dimension D D D , and N N N vectors in R d \mathbb{R}^{d} R d denoted by X \mathbf{X} X . let A ∈ R ( 2 D + 1 ) × d A \in\mathbb{R}^{(2D+1)\times d} A ∈ R ( 2 D + 1 ) × d , b ∈ R 2 D + 1 b \in\mathbb{R}^{2D+1} b ∈ R 2 D + 1 , σ : R → R \sigma: \mathbb{R} \to\mathbb{R} σ : R → R analytic non-polynomial activation, then for Lebesgue almost every A , b A,b A , b
f A , b ( X ) = ∑ i = 1 N σ ( A ⋅ X i + b ) f_{A,b}(\mathbf{X}) = \sum_{i=1}^{N} \sigma(A \cdot X_i + b) f A , b ( X ) = i = 1 ∑ N σ ( A ⋅ X i + b )
is injective up to permutation. Thus, the composition of such blocks and such ReadOut function with dimension D = d ⋅ N D = d \cdot N D = d ⋅ N yields a maximally expressive model on all input domain. Here, D D D is the intrinsic dimension of all features, ultimately produced by the D D D dimensional input space.
Separation Power of Fully Maximally expressive E-GGNN
In the main text, we proposed EGenNet a simple generically maximal expressive E-GGNN. In this section, we show that if we have a maximally expressive E-GGNN on all input domain, we can reconstruct a full graph up to group action, and if the graph is connected, we can reconstruct its point cloud. This indicates that fully maximally expressive E-GGNN is as challenging as achieving completeness, which generally requires complex methods like 2-WL-based GNNs.
Theorem D.1. Let G = ( A , X ) \mathcal{G}=(\mathbf{A},\mathbf{X}) G = ( A , X ) be a geometric graph, where A \mathbf{A} A is the full graph on n n n nodes, and X \mathbf{X} X is an arbitrary d × n d \times n d × n matrix, then one iteration of maximally expressive E-GGNN can reconstruct X \mathbf{X} X .
Proof. Assume we run one iteration of E-GGNN on our geometric graph, then for each node i i i , we know up to some rotation Q i Q_i Q i
{ { Q i ⋅ ( x i − x j ) ∣ j ∈ [ n ] } } \left\{\left\{Q_i \cdot(x_i-x_j) \mid j \in[n]\right\}\right\} { { Q i ⋅ ( x i − x j ) ∣ j ∈ [ n ] } }
Choosing i = 1 i=1 i = 1 , and x i = 0 x_i=0 x i = 0 , we have all other positions Q i ⋅ x j Q_i \cdot x_j Q i ⋅ x j , j ≠ i j \ne i j = i , thus up to translation (as we set x i x_i x i ), rotation Q i Q_i Q i and node permutation, we can reconstruct our point cloud. □ \square □
We now show that using n n n iterations of E-GGNN can reconstruct the point cloud of each connected graph.
Theorem D.2. Assuming a connected geometric graph G = ( A , X ) \mathcal{G}=(\mathbf{A},\mathbf{X}) G = ( A , X ) with n n n nodes, then n n n iteration of maximally expressive E-GGNN can reconstruct the point cloud.
Proof. Assume we run n n n iterations of a maximally expressive E-GGNN on our connected graph. By the injectivity of the aggregation functions, we can recursively reconstruct from v i ( t ) v_i^{(t)} v i ( t ) , up to a rotation Q i Q_i Q i , all relative displacement vectors appearing in the t t t -hop computation tree rooted at node i i i . In particular, for every node j j j whose graph distance from i i i is at most t t t , we can reconstruct Q i ( x i − x j ) Q_i(x_i-x_j) Q i ( x i − x j ) by summing the relative displacement vectors along a path from i i i to j j j . Since the graph is connected and its diameter is at most n − 1 n-1 n − 1 , after n n n iterations this allows us to reconstruct all node positions relative to x i x_i x i , up to a common rotation Q i Q_i Q i . Hence, up to translation, rotation, and node permutation, we can reconstruct the point cloud. □ \square □
I-GGNN+
Here, we present the definition of I-GGNN presented by [JBM+23], denoted by I-GGNN+, and show that it can separate all geometric full graphs. After, we explain why this model is strictly stronger than our I-GGNN.
[JBM+23] defines (neglecting the update step also used there)
v i ( t + 1 ) = f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ N i } ) . (7) v_i^{(t+1)} = f^t\left(\left\{\left\{v_j^{(t)},x_i-x_j\right\},\mid j\in\mathcal{N}_i\right\}\right). \tag*{(7)} v i ( t + 1 ) = f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ N i } ) . (7)
Such that f t f^t f t holds that for all Q ∈ O ( d ) Q\in O(d) Q ∈ O ( d )
f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ N i } ) = f t ( { { v j ( t ) , Q ⋅ ( x i − x j ) } , ∣ j ∈ N i } ) f^t\left(\left\{\left\{v_j^{(t)},x_i-x_j\right\},\mid j\in\mathcal{N}_i\right\}\right)=f^t\left(\left\{\left\{v_j^{(t)},Q\cdot(x_i-x_j)\right\},\mid j\in\mathcal{N}_i\right\}\right) f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ N i } ) = f t ( { { v j ( t ) , Q ⋅ ( x i − x j ) } , ∣ j ∈ N i } )
I-GGNN+ is maximally expressive if f t f^t f t holds:
f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ N i } ) = f t ( { { v j ( t ) , x ^ i − x ^ j } , ∣ j ∈ N i } ) ⟺ f^t\left(\left\{\left\{v_j^{(t)},x_i-x_j\right\},\mid j\in\mathcal{N}_i\right\}\right)=f^t\left(\left\{\left\{v_j^{(t)},\hat{x}_i-\hat{x}_j\right\},\mid j\in\mathcal{N}_i\right\}\right)\Longleftrightarrow f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ N i } ) = f t ( { { v j ( t ) , x ^ i − x ^ j } , ∣ j ∈ N i } ) ⟺
∃ Q ∈ O ( 3 ) : { { v j ( t ) , x i − x j } , ∣ j ∈ N i } = { { v j ( t ) , Q ⋅ ( x ^ i − x ^ j ) } , ∣ j ∈ N i } \exists Q\in O(3):\left\{\left\{v_j^{(t)},x_i-x_j\right\},\mid j\in\mathcal{N}_i\right\}=\left\{\left\{v_j^{(t)},Q\cdot(\hat{x}_i-\hat{x}_j)\right\},\mid j\in\mathcal{N}_i\right\} ∃ Q ∈ O ( 3 ) : { { v j ( t ) , x i − x j } , ∣ j ∈ N i } = { { v j ( t ) , Q ⋅ ( x ^ i − x ^ j ) } , ∣ j ∈ N i }
We now prove that if we run one iteration of maximally expressive I-GGNN+ on a full graph, we can reconstruct it.
Theorem E.1. Let G = ( A , X ) \mathcal{G}=(\mathbf{A},\mathbf{X}) G = ( A , X ) be a geometric graph, where A \mathbf{A} A is the full graph on n n n nodes, and X \mathbf{X} X is an arbitrary d × n d\times n d × n matrix, then one iteration of maximally expressive I-GGNN+ can reconstruct X \mathbf{X} X .
Proof. Assume we run one iteration of I-GGNN+. Then, for each node i i i , we have
f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ [ n ] } ) f^t\left(\left\{\left\{v_j^{(t)},x_i-x_j\right\},\mid j\in[n]\right\}\right) f t ( { { v j ( t ) , x i − x j } , ∣ j ∈ [ n ] } ) then up to some rotation Q i ∈ O ( 3 ) Q_i\in O(3) Q i ∈ O ( 3 ) , we know
{ { Q i ⋅ ( x i − x j ) } , ∣ j ∈ [ n ] } \left\{\left\{Q_i\cdot(x_i-x_j)\right\},\mid j\in[n]\right\} { { Q i ⋅ ( x i − x j ) } , ∣ j ∈ [ n ] } Then, we can reconstruct the point cloud by Theorem D.1.
It’s easy to see that I-GGNN+ can simulate our I-GGNN as the norm function is invariant under multiplication by Q ∈ O ( 3 ) Q\in O(3) Q ∈ O ( 3 ) . But [PWB+20; PC22] gave examples of full-graphs each I-GGNN can’t separate, but by what we proved above, maximally expressive I-GGNN+ can separate them. Thus, I-GGNN+ is strictly more powerful than I-GGNN in its expressive power.
Approximation and separation
It is known that a tight connection exists between the approximation of invariant functions and complete invariant models [HAGD23].
In the context of our paper, we discussed generic completeness. In particular, we showed in Corollary 3.5.2 that EGenNet with random parameters is complete on G g e n e r i c ( d , N ) \mathcal{G}_{\mathrm{generic}}(d,N) G generic ( d , N ) , which denotes the set of all geometric graphs G = ( A , X ) \mathcal{G}=(\mathbf{A},\mathbf{X}) G = ( A , X ) with exactly N N N nodes, A \mathbf{A} A connected, and X ∈ R d × n \mathbf{X} \in\mathbb{R}^{d \times n} X ∈ R d × n generic. Accordingly, we can obtain the following theorem:
Theorem F.1. Let K ⊆ G g e n e r i c ( d , N ) K \subseteq\mathcal{G}_{\mathrm{generic}}(d,N) K ⊆ G generic ( d , N ) be a compact set, f s e p : K → R m f^{\mathrm{sep}}: K \to\mathbb{R}^{m} f sep : K → R m a continuous function invariant to rotations, translations, and permutations. Then f s e p f^{\mathrm{sep}} f sep is injective on K K K (up to symmetries) if and only if every continuous invariant f : K → R M f: K \to\mathbb{R}^{M} f : K → R M can be approximated uniformly on K K K using functions of the form N ( f s e p ) \mathcal{N}(f^{\mathrm{sep}}) N ( f sep ) , where N \mathcal{N} N is a fully connected neural network.
Proof. Assume, on the one hand, f s e p f^{\mathrm{sep}} f sep is injective on K K K (up to symmetries). Given a continuous f : K → R M f: K \to\mathbb{R}^{M} f : K → R M we want to approximate, then, by [HAGD23], ∀ ϵ > 0 ∃ \forall\epsilon> 0\ \exists ∀ ϵ > 0 ∃ neural network N \mathcal{N} N :
sup G ∈ K ∣ f ( G ) − N ( f s e p ( G ) ) ∣ < ϵ \sup_{\mathcal{G} \in K} \lvert f(\mathcal{G})-\mathcal{N}(f^{\mathrm{sep}}(\mathcal{G}))\rvert< \epsilon G ∈ K sup ∣ f ( G ) − N ( f sep ( G ))∣ < ϵ
On the other hand, assume by contradiction there exists G 1 = ( A 1 , X 1 ) \mathcal{G}_1=(\mathbf{A}_1,\mathbf{X}_1) G 1 = ( A 1 , X 1 ) and G 2 = ( A 2 , X 2 ) \mathcal{G}_2=(\mathbf{A}_2,\mathbf{X}_2) G 2 = ( A 2 , X 2 ) which are not related by translation, rotation or permutation, such that f s e p ( G 1 ) = f s e p ( G 2 ) f^{\mathrm{sep}}(\mathcal{G}_1)=f^{\mathrm{sep}}(\mathcal{G}_2) f sep ( G 1 ) = f sep ( G 2 ) .
Define f ( G ) = inf g ∈ G ∥ ( A , X ) − g ⋅ ( A 1 , X 1 ) ∥ f(\mathcal{G})=\inf_{g\in\mathcal{G}}\lVert(\mathbf{A},\mathbf{X})-g\cdot(\mathbf{A}_1,\mathbf{X}_1)\rVert f ( G ) = inf g ∈ G ∥( A , X ) − g ⋅ ( A 1 , X 1 )∥ , where G = ( A , X ) \mathcal{G}=(\mathbf{A},\mathbf{X}) G = ( A , X ) , and G \mathcal{G} G is the group of permutations, rotations, and translation. Note that this minimum is obtained as the product group of rotation and permutation is compact, and the translation can be bounded by the sum of the norms of the two graphs. Then, according to the Tychonoff’s theorem, our group is compact, and we take a minimum continuous function over a compact set. Note that f f f is an invariant, continuous function such that f ( G 1 ) = 0 f(\mathcal{G}_1)=0 f ( G 1 ) = 0 , and f ( G 2 ) > 0 f(\mathcal{G}_2)>0 f ( G 2 ) > 0 , but for each neural network N \mathcal{N} N , N ( f s e p ( G 1 ) ) = N ( f s e p ( G 2 ) ) \mathcal{N}(f^{\mathrm{sep}}(\mathcal{G}_1))=\mathcal{N}(f^{\mathrm{sep}}(\mathcal{G}_2)) N ( f sep ( G 1 )) = N ( f sep ( G 2 )) , and we can’t approximate our desired function f f f with ϵ = f ( G 2 ) 2 \epsilon=\frac{f(\mathcal{G}_2)}{2} ϵ = 2 f ( G 2 ) accuracy, yielding a contradiction.
Proofs
In this section, we restate and prove theorems that have not been fully proved in the main text.
Theorem 3.3.1. [expressive power of I-GGNN] Let d d d be a natural number. Let F F F be an I-GGNN. Let A \mathbf{A} A be a graph not generically globally rigid on R d \mathbb{R}^{d} R d . Then, F F F generically fails to identify A \mathbf{A} A .
Conversely, if A \mathbf{A} A is generically globally rigid on R d \mathbb{R}^{d} R d and F F F is a maximally expressive I-GGNN with depth T = 1 T=1 T = 1 , then F F F generically identifies A \mathbf{A} A .
Proof. Let G = ( A , X ) \mathcal{G}=(\mathbf{A},\mathbf{X}) G = ( A , X ) be a geometric graph, and assume that X \mathbf{X} X is generic. First, we assume that A \mathbf{A} A is generically globally rigid. Assume that there is some geometric graph G ′ \mathcal{G}' G ′ which is assigned the same global feature as G \mathcal{G} G after a single iteration. We need to show the two graphs are geometrically isomorphic. The multisets of s i s_i s i and s i ′ s'_i s i ′ features in equation (2) are equal, and in particular, both graphs have the same number of nodes n n n , and by relabeling if necessary, we can assume that s i = s i ′ s_i=s'_i s i = s i ′ for all i = 1 , … , n i=1,\ldots,n i = 1 , … , n . This equation, in turn, implies that for every node i i i ,
{ { ∥ x i j ∥ , j ∈ N i } } = { { ∥ x i j ′ ∥ , j ∈ N i ′ } } \{\{\lVert x_{ij}\rVert,j\in\mathcal{N}_i\}\}=\{\{\lVert x'_{ij}\rVert,j\in\mathcal{N}'_i\}\} {{∥ x ij ∥ , j ∈ N i }} = {{∥ x ij ′ ∥ , j ∈ N i ′ }}
In particular, the i i i ’th node in G \mathcal{G} G and G ′ \mathcal{G}' G ′ have the same degree, so both graphs have the same number of edges. For any fixed ( i , j ) (i,j) ( i , j ) which is an edge of G \mathcal{G} G , the distance ∥ x i j ∥ \lVert x_{ij}\rVert ∥ x ij ∥ will appear exactly once in the multiset corresponding to s i ′ s'_i s i ′ and s j ′ s'_j s j ′ and will not appear in the other multisets. This observation implies that ( i , j ) (i,j) ( i , j ) is also an edge in the graph G ′ \mathcal{G}' G ′ . Since the number of edges in both graphs is the same, we deduce that A = A ′ A=A' A = A ′ . By global rigidity, we deduce that all pairwise distances are the same, and by [SHW21], the two graphs are geometrically isomorphic.
We mention a theorem proved in [GHT10] for the other direction. [GHT10] proved A A A is globally rigid if exists generic X X X such that ( A , X ) (A,X) ( A , X ) is globally rigid. Thus if A A A is not globally rigid, for every generic X X X , there exists X ′ X' X ′ for whom distances along edges are preserved, but X X X and X ′ X' X ′ aren’t geometrically isomorphic. By immediate induction on the number of layers, it can be shown that any I-GGNN cannot separate any pair of geometric graphs whose distances across edges are preserved. This concludes the proof.
Theorem 3.4.1. [expressive power of E-GGNN] Let d d d be a natural number. Let F F F be an E-GGNN. Let A A A be a disconnected graph. Then, F F F generically fails to identify A A A . Conversely, if A A A is connected and F F F is a maximally expressive E-GGNN with depth T ≥ d + 1 T\ge d+1 T ≥ d + 1 , then F F F generically identifies A A A .
Proof. Assume A A A is not connected. Let X ∈ R n × d X\in\mathbb{R}^{n\times d} X ∈ R n × d . Let V 1 V_1 V 1 be a connected component of the graph, and let G ′ \mathcal{G}' G ′ be the geometric graph whose adjacency matrix is A ′ = A A'=A A ′ = A , and where X ′ X' X ′ is obtained from X X X by applying a non-trivial translation t ∈ R d t\in\mathbb{R}^{d} t ∈ R d to all nodes in V 1 V_1 V 1 : x ^ i = x i + t \hat{x}_i=x_i+t x ^ i = x i + t for all i ∈ V 1 i\in V_1 i ∈ V 1 , and setting x ^ j = x j \hat{x}_j=x_j x ^ j = x j for all j ∉ V 1 j\notin V_1 j ∈ / V 1 . We choose t t t with sufficiently large norm so that the two graphs are not geometrically isomorphic. However, since there are no edges between V 1 V_1 V 1 and the other connected components, for every edge ( i , j ) (i,j) ( i , j ) we have x ^ i − x ^ j = x i − x j \hat{x}_i-\hat{x}_j=x_i-x_j x ^ i − x ^ j = x i − x j . Therefore, for any E-GGNN, we can easily see recursively that all features v i ( t ) v_i^{(t)} v i ( t ) constructed from X X X and X ′ X' X ′ , respectively, will be the same, and consequently, the E-GGNN will assign the same global features to G \mathcal{G} G and G ′ \mathcal{G}' G ′ .
Now assume A A A is connected, and we are given two geometric graphs G \mathcal{G} G and G ^ \hat{\mathcal{G}} G ^ in ( G g e n e r i c ( d , N ) , G N ( d ) ) (\mathcal{G}_{\mathrm{generic}}(d,N),\mathcal{G}_{N}(d)) ( G generic ( d , N ) , G N ( d )) . By assumption, X X X is generic. Assume that the E-GGNN assigns the same value of G \mathcal{G} G and G ^ \hat{\mathcal{G}} G ^ . We need to show that the two graphs are geometrically isomorphic. First, we note that since the two graphs have the same global features, they have the same number of nodes n n n , and by applying relabeling if necessary, we can assume that sume that
s i = s ^ i , ∀ i = 1 , … , n s_i=\hat{s}_i,\quad\forall i=1,\ldots,n s i = s ^ i , ∀ i = 1 , … , n
which implies that for every node i i i there is some rotation R i ∈ O ( d ) R_i\in O(d) R i ∈ O ( d ) such that v i ( T ) = R i v ^ i ( T ) \mathbf{v}_i^{(T)}=R_i\hat{\mathbf{v}}_i^{(T)} v i ( T ) = R i v ^ i ( T ) . By rotation equivariance, we know that R i v i ( T ) R_i\mathbf{v}_i^{(T)} R i v i ( T ) is the feature that we would have obtained by applying the maximally expressive E-GGNN to R i ⋅ X ^ R_i\cdot\hat{X} R i ⋅ X ^ . We deduce that
{ { v j ( T − 1 ) , x i − x j ∣ j ∈ N i } } = { { R i v ^ j ( T − 1 ) , R i ( x ^ i − x ^ j ) ∣ j ∈ N ^ i } } \{\{\mathbf{v}_j^{(T-1)},x_i-x_j\mid j\in\mathcal{N}_i\}\}=\{\{R_i\hat{\mathbf{v}}_j^{(T-1)},R_i(\hat{x}_i-\hat{x}_j)\mid j\in\hat{\mathcal{N}}_i\}\} {{ v j ( T − 1 ) , x i − x j ∣ j ∈ N i }} = {{ R i v ^ j ( T − 1 ) , R i ( x ^ i − x ^ j ) ∣ j ∈ N ^ i }}
The equality of multisets above implies that the node i i i has the same degree in A A A and A ^ \hat{A} A ^ . Since this is true for all nodes, A A A and A ^ \hat{A} A ^ have the same number of edges. Moreover
{ { ∥ x i − x j ∥ , j ∈ N i } } = { { ∥ x ^ i − x ^ j ∥ , j ∈ N ^ i } } \{\{\lVert x_i-x_j\rVert,j\in\mathcal{N}_i\}\}=\{\{\lVert\hat{x}_i-\hat{x}_j\rVert,j\in\hat{\mathcal{N}}_i\}\} {{∥ x i − x j ∥ , j ∈ N i }} = {{∥ x ^ i − x ^ j ∥ , j ∈ N ^ i }}
Since X X X is generic, for every edge ( i , j ) ∈ A (i,j)\in A ( i , j ) ∈ A the distance ∥ x i − x j ∥ \lVert x_i-x_j\rVert ∥ x i − x j ∥ will appear only in the 1-hop multi-sets corresponding to the i i i and j j j nodes. Thus, they will appear only once in the 1-hop multisets corresponding to the second graph’s i i i and j j j nodes. This implies that ( i , j ) (i,j) ( i , j ) is an edge in A ^ \hat{A} A ^ too, and
R i ( x ^ i − x ^ j ) = x i − x j , ∀ j ∈ N i . R_i(\hat{x}_i-\hat{x}_j)=x_i-x_j,\quad\forall j\in\mathcal{N}_i. R i ( x ^ i − x ^ j ) = x i − x j , ∀ j ∈ N i .
Since the two graphs have the same number of edges, it follows that A = A ^ A=\hat{A} A = A ^ . We now show recursively that for all 1 ≤ t ≤ T 1\leq t\leq T 1 ≤ t ≤ T , we have v j T − t = R i v ^ j T − t \mathbf{v}_j^{T-t}=R_i\hat{\mathbf{v}}_j^{T-t} v j T − t = R i v ^ j T − t and x i − x j = R i ( x ^ i − x ^ j ) x_i-x_j=R_i(\hat{x}_i-\hat{x}_j) x i − x j = R i ( x ^ i − x ^ j ) for all j j j that is a t t t -hop neighbor of i i i (we say j j j is an t t t -hop neighbor of i i i , if there is a path of length at most t t t from i i i to j j j ).
We have already proved the claim for t = 1 t=1 t = 1 . Now we assume correctness for t t t and show for t + 1 t+1 t + 1 . Since by the induction hypothesis we know that v j T − t = R i v ^ j T − t \mathbf{v}_j^{T-t}=R_i\hat{\mathbf{v}}_j^{T-t} v j T − t = R i v ^ j T − t for every j j j which is t t t hop neighbor of i i i , we deduce the multi-set equality
{ { v k T − ( t + 1 ) , x j − x k ∣ k ∈ N j } } = { { R i v ^ k T − ( t + 1 ) , R i ( x ^ j − x ^ k ) ∣ k ∈ N j } } \{\{\mathbf{v}_k^{T-(t+1)},x_j-x_k\mid k\in\mathcal{N}_j\}\}=\{\{R_i\hat{\mathbf{v}}_k^{T-(t+1)},R_i(\hat{x}_j-\hat{x}_k)\mid k\in\mathcal{N}_j\}\} {{ v k T − ( t + 1 ) , x j − x k ∣ k ∈ N j }} = {{ R i v ^ k T − ( t + 1 ) , R i ( x ^ j − x ^ k ) ∣ k ∈ N j }}
Since we know that the pairwise norms of X X X are all distinct due to the genericity of X X X , and also that for every edge ( j , k ) (j,k) ( j , k ) we have the norm equality ∥ x j − x k ∥ = ∥ x ^ j − x ^ k ∥ \lVert x_j-x_k\rVert=\lVert\hat{x}_j-\hat{x}_k\rVert ∥ x j − x k ∥ = ∥ x ^ j − x ^ k ∥ , we see that necessarily
v k T − ( t + 1 ) = R i v ^ k T − ( t + 1 ) , and x j − x k = R i ( x ^ j − x ^ k ) . \mathbf{v}_k^{T-(t+1)}=R_i\hat{\mathbf{v}}_k^{T-(t+1)},\text{ and }x_j-x_k=R_i(\hat{x}_j-\hat{x}_k). v k T − ( t + 1 ) = R i v ^ k T − ( t + 1 ) , and x j − x k = R i ( x ^ j − x ^ k ) .
Now, every k k k , an t + 1 t+1 t + 1 hop neighbor of i i i , is connected by an edge to a node j j j , an t t t hop neighbor of i i i . By the induction assumption we know that x i − x j = R i ( x ^ i − x ^ j ) x_i-x_j=R_i(\hat{x}_i-\hat{x}_j) x i − x j = R i ( x ^ i − x ^ j ) . We deduce that
x i − x k = x i − x j + x j − x k = R i ( x ^ i − x ^ j + x ^ j − x ^ k ) = R i ( x ^ i − x ^ k ) x_i-x_k=x_i-x_j+x_j-x_k=R_i(\hat{x}_i-\hat{x}_j+\hat{x}_j-\hat{x}_k)=R_i(\hat{x}_i-\hat{x}_k) x i − x k = x i − x j + x j − x k = R i ( x ^ i − x ^ j + x ^ j − x ^ k ) = R i ( x ^ i − x ^ k )
for all t + 1 t+1 t + 1 hop neighbors of i i i . Continuing with this argument inductively, we see that for all d + 1 d+1 d + 1 hop neighbors k k k of i i i , we obtain
x i − x k = R i ( x ^ i − x ^ k ) x_i-x_k=R_i(\hat{x}_i-\hat{x}_k) x i − x k = R i ( x ^ i − x ^ k )
and in particular
∥ x i − x k ∥ = ∥ x ^ i − x ^ k ∥ \lVert x_i-x_k\rVert=\lVert\hat{x}_i-\hat{x}_k\rVert ∥ x i − x k ∥ = ∥ x ^ i − x ^ k ∥
for all d + 1 d+1 d + 1 hop neighbors. By global generic rigidity of the d + 1 d+1 d + 1 power graph of a connected graph [JT22; CW05], it follows that the two graphs are geometrically isomorphic. Thus, as previously, we reconstructed the combinatorial graph, and we now reconstructed the geometric graphs; we are done.
Proof of Maximal Expressivity of EGenNet
This subsection is devoted to proving Theorem 3.5.1 on the generic maximal expressivity of EGenNet. This proof will require some background on the finite witness theorem from [2 ] :
Background on the finite witness theorem. Before we prove the theorem, we review some definitions and results from [2 ] , which we will use for the proof. The main tool we require from [2 ] is Theorem A.2 from that paper, the finite witness theorem. We state it here in the special case where the parameter space is Euclidean, and the function F F F is analytic in the parameters:
Theorem G.1 (Finite Witness Theorem, [2 ] ). Let M ⊆ R p M\subseteq\mathbb{R}^{p} M ⊆ R p be a σ \sigma σ -sub-analytic set of dimension D D D . Let F : M × R q → R F:M\times\mathbb{R}^{q}\to\mathbb{R} F : M × R q → R be a σ \sigma σ -sub-analytic function, and assume that F ( z ; θ ) F(z;\theta) F ( z ; θ ) is analytic as a function of θ \theta θ for all fixed z ∈ M z\in M z ∈ M . Define the set
N = { z ∈ M ∣ F ( z ; θ ) = 0 , ∀ θ ∈ R q } . \mathcal{N}=\{z\in M\mid F(z;\theta)=0,\ \forall\theta\in\mathbb{R}^{q}\}. N = { z ∈ M ∣ F ( z ; θ ) = 0 , ∀ θ ∈ R q } . Then for Lebesgue almost every ( θ ( 1 ) , … , θ ( D + 1 ) ) ∈ R q × ( D + 1 ) (\theta^{(1)},\ldots,\theta^{(D+1)})\in\mathbb{R}^{q\times(D+1)} ( θ ( 1 ) , … , θ ( D + 1 ) ) ∈ R q × ( D + 1 ) ,
N = { z ∈ M ∣ F ( z ; θ ( i ) ) = 0 , ∀ i = 1 , … , D + 1 } . (8) \mathcal{N}=\{z\in M\mid F(z;\theta^{(i)})=0,\ \forall i=1,\ldots,D+1\}. \tag*{(8)} N = { z ∈ M ∣ F ( z ; θ ( i ) ) = 0 , ∀ i = 1 , … , D + 1 } . (8)
Remark: We note that this theorem also holds when the output of F F F is a vector in R d \mathbb{R}^{d} R d , which is what we will use in our proofs below. To see the theorem holds for a function F ^ : M × R q → R d \hat{F}:M\times\mathbb{R}^{q}\to\mathbb{R}^{d} F ^ : M × R q → R d , we can define a new parametric function F F F with additional parameters a ∈ R d a\in\mathbb{R}^{d} a ∈ R d ,
F ( z ; θ , a ) = ⟨ F ^ ( z ; θ ) , a ⟩ F(z;\theta,a)=\langle\hat{F}(z;\theta),a\rangle F ( z ; θ , a ) = ⟨ F ^ ( z ; θ ) , a ⟩
and use the theorem for F F F , which has scalar output, to deduce the theorem for F ^ \hat{F} F ^ .5
The finite witness theorem assumes the domain is a σ \sigma σ sub-analytic set. The full definition of σ \sigma σ -sub-analytic sets is rather involved, and we refer the reader to Appendix A in [2 ] for a full definition. Rather than giving a full definition here, we will recall the properties of these sets, which we will need.
The class of σ \sigma σ subanalytic sets in R d \mathbb{R}^{d} R d is rather large. In particular, it contains all open sets in R d \mathbb{R}^{d} R d , all semi-algebraic sets in R d \mathbb{R}^{d} R d (sets defined by polynomial equalities and inequalities, like spheres and linear sub-spaces), as well as sets defined by analytic equality and inequality constraints. The class of σ \sigma σ subanalytic sets is closed under countable unions and finite intersections.
σ \sigma σ -sub-analytic sets are always a countable union of smooth manifolds. The dimension of a σ \sigma σ sub-analytic set is the maximal dimension of the manifolds composing it.
A σ \sigma σ -sub-analytic function is a function whose graph is a σ \sigma σ sub-analytic set. Analytic functions are, in particular, σ \sigma σ -sub-analytic. When a σ \sigma σ sub-analytic function is applied to a σ \sigma σ sub-analytic set, the image is a σ \sigma σ sub-analytic set whose dimension is no larger than the dimension of the σ \sigma σ sub-analytic domain [2 ] .
We can now restate and prove Theorem 3.5.1:
Theorem 3.5.1. Let d , N , T d,N,T d , N , T be natural numbers. Let F F F be a maximally expressive E-GGNN of depth T T T , and let F θ F_{\theta} F θ denote the EGenNet architecture with depth T T T and C = 2 N d + 1 C=2Nd+1 C = 2 N d + 1 channels. Then, for Lebesgue almost every choice of network parameters θ \theta θ , we have that for all generic G , G ^ ∈ G N ( d ) \mathcal{G},\widehat{\mathcal{G}}\in\mathbb{G}_{N}(d) G , G ∈ G N ( d ) ,
F ( G ) = F ( G ^ ) ⟺ F θ ( G ) = F θ ( G ^ ) F(\mathcal{G})=F(\widehat{\mathcal{G}})\Longleftrightarrow F_{\theta}(\mathcal{G})=F_{\theta}(\widehat{\mathcal{G}}) F ( G ) = F ( G ) ⟺ F θ ( G ) = F θ ( G )
Proof. In the proof, we consider the set of all generic geometric graphs in G N ( d ) \mathbb{G}_{N}(d) G N ( d ) . We will prove the theorem for an even larger set: the set G N d i s t i n c t ( d ) \mathbb{G}_{N}^{\mathrm{distinct}}(d) G N distinct ( d ) of all geometric graphs in G N ( d ) \mathbb{G}_{N}(d) G N ( d ) , which additionally satisfies that for any distinct edges ( i , j ) (i,j) ( i , j ) and ( s , t ) (s,t) ( s , t ) in the graph, 0 < ∥ x i − x j ∥ ≠ ∥ x s − x t ∥ 0<\lVert x_i-x_j\rVert\ne\lVert x_s-x_t\rVert 0 < ∥ x i − x j ∥ = ∥ x s − x t ∥ . Since polynomial inequalities with integer coefficients define it, we note that G N d i s t i n c t ( d ) \mathbb{G}_{N}^{\mathrm{distinct}}(d) G N distinct ( d ) contains all generic graphs with ≤ N \le N ≤ N nodes. The advantage of considering G N d i s t i n c t ( d ) \mathbb{G}_{N}^{\mathrm{distinct}}(d) G N distinct ( d ) is that the set of all geometric graphs in G N d i s t i n c t ( d ) \mathbb{G}_{N}^{\mathrm{distinct}}(d) G N distinct ( d ) of the same cardinality n ≤ N n\le N n ≤ N can be thought of as a subset of R n × n ⊕ R n × d \mathbb{R}^{n\times n}\oplus\mathbb{R}^{n\times d} R n × n ⊕ R n × d . Since a constant number of polynomial inequalities defines it, it is a semi-algebraic set, and hence, in particular, σ \sigma σ -subanalytic, so the finite witness theorem can be applied. In contrast, it is unclear that the set of all generic points of fixed dimension is a σ \sigma σ subanalytic set.
Our goal is to show that for the networks described in the theorem, for almost every choice of parameter values, all functions described in the procedure are injective. To be more accurate, let us introduce some notation given an arbitrary pair of geometric graphs
G = ( A , X ) , G ^ = ( A ^ , X ^ ) \mathcal{G}=(A,X),\qquad\widehat{\mathcal{G}}=(\widehat{A},\widehat{X}) G = ( A , X ) , G = ( A , X )
in our domain G N d i s t i n c t ( d ) \mathbb{G}_{N}^{\mathrm{distinct}}(d) G N distinct ( d ) , we denote the features corresponding to the first graph by v i ( t ) v_i^{(t)} v i ( t ) , s i s_i s i and s g l o b a l s^{\mathrm{global}} s global as in the main text, and the features corresponding to the second graph by v ^ i ( t ) \hat{v}_i^{(t)} v ^ i ( t ) , s ^ i \hat{s}_i s ^ i and s ^ g l o b a l \hat{s}^{\mathrm{global}} s ^ global . The neighborhood of a node k k k in G ^ \hat{G} G ^ is denoted by N ^ k \hat{\mathcal{N}}_k N ^ k . The number of nodes in the two graphs is denoted by n n n and m m m , respectively.
We need to show that for all t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 and all nodes i i i of G \mathcal{G} G and k k k of G ^ \hat{\mathcal{G}} G ^ ,
v i ( t + 1 ) = v ^ k ( t + 1 ) if and only if { { ( x i − x j , v j ( t ) ) , j ∈ N i } } = { { ( x ^ k − x ^ j , v ^ j ( t ) ) ∣ j ∈ N ^ k } } (9) v_i^{(t+1)}=\hat{v}_k^{(t+1)}\ \text{if and only if}\ \{\{(x_i-x_j,v_j^{(t)}),j\in\mathcal{N}_i\}\}=\{\{(\hat{x}_k-\hat{x}_j,\hat{v}_j^{(t)})\mid j\in\hat{\mathcal{N}}_k\}\} \tag*{(9)} v i ( t + 1 ) = v ^ k ( t + 1 ) if and only if {{( x i − x j , v j ( t ) ) , j ∈ N i }} = {{( x ^ k − x ^ j , v ^ j ( t ) ) ∣ j ∈ N ^ k }} (9)
s i = s ^ k if and only if ∃ Q ∈ O ( d ) , Q v i ( T ) = v ^ k ( T ) (10) s_i=\hat{s}_k\ \text{if and only if}\ \exists Q\in O(d),\ Qv_i^{(T)}=\hat{v}_k^{(T)} \tag*{(10)} s i = s ^ k if and only if ∃ Q ∈ O ( d ) , Q v i ( T ) = v ^ k ( T ) (10)
s g l o b a l = s ^ g l o b a l if and only if { { s i ∣ i = 1 , … , n } } = { { s ^ i ∣ i = 1 , … , m } } (11) s^{\mathrm{global}}=\hat{s}^{\mathrm{global}}\ \text{if and only if}\ \{\{s_i\mid i=1,\ldots,n\}\}=\{\{\hat{s}_i\mid i=1,\ldots,m\}\} \tag*{(11)} s global = s ^ global if and only if {{ s i ∣ i = 1 , … , n }} = {{ s ^ i ∣ i = 1 , … , m }} (11)
Our first step for reaching this result is using the finite witness theorem, similar to what was done in [HAGD23]. We note that the features we are interested in are composed of C = 2 N d + 1 C=2Nd+1 C = 2 N d + 1 coordinates, denoted by q = 1 , … , C q=1,\ldots,C q = 1 , … , C , which are of the general form
v i , q ( t + 1 ) − v ^ i , q ( t + 1 ) = F t ( X , V ( t ) , X ^ , V ^ ( t ) ; α q , t ) , t = 0 , … , T − 1 (12) v_{i,q}^{(t+1)}-\hat{v}_{i,q}^{(t+1)}=F_t(X,V^{(t)},\hat{X},\hat{V}^{(t)};\alpha_{q,t}),\quad t=0,\ldots,T-1 \tag*{(12)} v i , q ( t + 1 ) − v ^ i , q ( t + 1 ) = F t ( X , V ( t ) , X ^ , V ^ ( t ) ; α q , t ) , t = 0 , … , T − 1 (12)
s i , q ( t + 1 ) − s ^ i , q ( t + 1 ) = F T ( V ( T ) , V ^ ( t ) ; α q , T ) (13) s_{i,q}^{(t+1)}-\hat{s}_{i,q}^{(t+1)}=F_T(V^{(T)},\hat{V}^{(t)};\alpha_{q,T}) \tag*{(13)} s i , q ( t + 1 ) − s ^ i , q ( t + 1 ) = F T ( V ( T ) , V ^ ( t ) ; α q , T ) (13)
s q g l o b a l − s ^ g g l o b a l = F T + 1 ( s 1 , … , s n , s ^ 1 , … , s ^ m ; α q , T + 1 ) (14) s_q^{\mathrm{global}}-\hat{s}_g^{\mathrm{global}}=F_{T+1}(s_1,\ldots,s_n,\hat{s}_1,\ldots,\hat{s}_m;\alpha_{q,T+1}) \tag*{(14)} s q global − s ^ g global = F T + 1 ( s 1 , … , s n , s ^ 1 , … , s ^ m ; α q , T + 1 ) (14)
where α q , t \alpha_{q,t} α q , t denote the parameters of the appropriate function, V ( t ) V^{(t)} V ( t ) is the concatenation of all n n n features v i ( t ) v_i^{(t)} v i ( t ) , and V ^ ( t ) \hat{V}^{(t)} V ^ ( t ) is the concatenation of all m m m features v ^ k ( t ) \hat{v}_k^{(t)} v ^ k ( t ) . In essence, we claim that by the finite witness theorem, for almost every choice of network parameters, we will have the following equalities for all geometric graphs G , G ^ \mathcal{G},\hat{\mathcal{G}} G , G ^ satisfying the theorem’s assumptions:
v i ( t + 1 ) − v ^ k ( t + 1 ) = 0 if and only if F t ( X , V ( t ) , X ^ , V ^ ( t ) ; α t ) = 0 , ∀ α (15) v_i^{(t+1)}-\hat{v}_k^{(t+1)}=0\ \text{if and only if}\ F_t(X,V^{(t)},\hat{X},\hat{V}^{(t)};\alpha_t)=0,\ \forall\alpha \tag*{(15)} v i ( t + 1 ) − v ^ k ( t + 1 ) = 0 if and only if F t ( X , V ( t ) , X ^ , V ^ ( t ) ; α t ) = 0 , ∀ α (15)
s i − s ^ k = 0 if and only if F T ( V ( T ) , V ^ ( t ) ; α ) , ∀ α (16) s_i-\hat{s}_k=0\ \text{if and only if}\ F_T(V^{(T)},\hat{V}^{(t)};\alpha),\ \forall\alpha \tag*{(16)} s i − s ^ k = 0 if and only if F T ( V ( T ) , V ^ ( t ) ; α ) , ∀ α (16)
s g l o b a l − s ^ g l o b a l = 0 if and only if F T + 1 ( s 1 , … , s n , s ^ 1 , … , s ^ m ; α ) = 0 , ∀ α (17) s^{\mathrm{global}}-\hat{s}^{\mathrm{global}}=0\ \text{if and only if}\ F_{T+1}(s_1,\ldots,s_n,\hat{s}_1,\ldots,\hat{s}_m;\alpha)=0,\quad\forall\alpha \tag*{(17)} s global − s ^ global = 0 if and only if F T + 1 ( s 1 , … , s n , s ^ 1 , … , s ^ m ; α ) = 0 , ∀ α (17)
In other words, the finite witness theorem enables us to reduce the problem to the problem of showing that some choice of network parameter exists, which enables separation.
Some explanation is needed as to the details of how the finite witness theorem is applied. It is convenient to think of the network parameters being chosen recursively. For t = 0 t=0 t = 0 , the finite witness theorem allows us to choose the parameters α q , 0 \alpha_{q,0} α q , 0 , q = 1 , … , C q=1,\ldots,C q = 1 , … , C in equation 3.12 so that the theorem applies. Formally, to do this, we need to split into a finite number of cases: once the number of nodes n n n and m m m in the two graphs are specified, as well as the combinatorial graphs A A A , A ^ \hat{A} A ^ and the nodes i , k i,k i , k , we can think of F 0 F_0 F 0 as a function which is analytic in the parameters, and σ \sigma σ -subanalytic on its domain (the norms used in the network are semi-algebraic functions, and hence σ \sigma σ -subanalytic functions. All other functions used in the network are analytic). The function F 0 F_0 F 0 operates on the domain of pairs X ∈ R d × n X\in\mathbb{R}^{d\times n} X ∈ R d × n , X ^ ∈ R d × m \hat{X}\in\mathbb{R}^{d\times m} X ^ ∈ R d × m which satisfy a finite number of polynomial inequalities. This domain is semi-algebraic and has dimensions d ( n + m ) ≤ 2 N d d(n+m) \le2Nd d ( n + m ) ≤ 2 N d . Therefore, we can apply the finite witness theorem with C = 2 N d + 1 C = 2Nd+1 C = 2 N d + 1 . Next, once the parameters α q , t \alpha_{q,t} α q , t are chosen for all t < t ′ t < t' t < t ′ , we can choose the parameters α q , t ′ \alpha_{q,t'} α q , t ′ again using the finite witness theorem. Central to the usage here is that, once the previous parameters were fixed, the features in the t ′ t' t ′ step are an image of a σ \sigma σ -subanalytic function applied to a pair ( X , X ^ ) (X,\widehat{X}) ( X , X ) . Therefore, the intrinsic dimension of the t ′ t' t ′ generation features is no larger than 2 N d = C − 1 2Nd = C-1 2 N d = C − 1 (for more on this argument, see [HAGD23]). This concludes the recursive definition of parameter choice. By Fubini’s theorem, one can verify that, in fact, for Lebesgue almost every parameter will be ’good.’ It remains to show that
The right hand side of equation (9) holds if and only if the right hand side of equation (15) holds
The right hand side of equation 3.9 holds if and only if the right hand side of equation 3.15 holds (18) \text{The right hand side of equation 3.9 holds if and only if the right hand side of equation 3.15 holds} \tag*{(18)} The right hand side of equation 3.9 holds if and only if the right hand side of equation 3.15 holds (18)
The right hand side of equation (10) holds if and only if the right hand side of equation (16) holds
The right hand side of equation 3.10 holds if and only if the right hand side of equation 3.16 holds (19) \text{The right hand side of equation 3.10 holds if and only if the right hand side of equation 3.16 holds} \tag*{(19)} The right hand side of equation 3.10 holds if and only if the right hand side of equation 3.16 holds (19)
The right hand side of equation (11) holds if and only if the right hand side of equation (17) holds
The right hand side of equation 3.11 holds if and only if the right hand side of equation 3.17 holds (20) \text{The right hand side of equation 3.11 holds if and only if the right hand side of equation 3.17 holds} \tag*{(20)} The right hand side of equation 3.11 holds if and only if the right hand side of equation 3.17 holds (20)
We now reformulate these three claims as three lemmas.
We use some fixed analytic non-polynomial function σ \sigma σ in the following lemmas. We say that ϕ \phi ϕ is a basic analytic network if it is of the form
ϕ ( s ; a , b ) = σ ( a ⋅ s + b ) \phi(s; a, b) = \sigma(a \cdot s + b) ϕ ( s ; a , b ) = σ ( a ⋅ s + b )
for some a ∈ R d a \in\mathbb{R}^{d} a ∈ R d , b ∈ R b \in\mathbb{R} b ∈ R .
Our first lemma will show 3.18, where z j z_j z j will assume the role of x i − x j x_i-x_j x i − x j .
Lemma G.2. Let z 1 , … , z n z_1,\ldots,z_n z 1 , … , z n and z ^ 1 , … , z ^ m \widehat{z}_1,\ldots,\widehat{z}_m z 1 , … , z m be non-zero vectors in R d \mathbb{R}^{d} R d . Additionally assume that ∥ z 1 ∥ < ∥ z 2 ∥ < ⋯ < ∥ z n ∥ \lVert z_1\rVert< \lVert z_2\rVert< \cdots< \lVert z_n\rVert ∥ z 1 ∥ < ∥ z 2 ∥ < ⋯ < ∥ z n ∥ and ∥ z ^ 1 ∥ < ∥ z ^ 2 ∥ < ⋯ < ∥ z ^ m ∥ \lVert\widehat{z}_1\rVert< \lVert\widehat{z}_2\rVert< \cdots< \lVert\widehat{z}_m\rVert ∥ z 1 ∥ < ∥ z 2 ∥ < ⋯ < ∥ z m ∥ .
Let v 1 , … , v n v_1,\ldots,v_n v 1 , … , v n and v ^ 1 , … , v ^ m \widehat{v}_1,\ldots,\widehat{v}_m v 1 , … , v m be matrices in R d × C \mathbb{R}^{d \times C} R d × C , and denote v i = ( v i , 1 , … , v i , C ) v_i=(v_{i,1},\ldots,v_{i,C}) v i = ( v i , 1 , … , v i , C ) and v ^ = ( v ^ i , 1 , … , v ^ i , C ) \widehat{v}=(\widehat{v}_{i,1},\ldots,\widehat{v}_{i,C}) v = ( v i , 1 , … , v i , C ) . If for all basic analytic neural networks ϕ ( 0 ) , … , ϕ ( C ) \phi^{(0)},\ldots,\phi^{(C)} ϕ ( 0 ) , … , ϕ ( C ) we have that
∑ j = 1 n ( ϕ ( 0 ) ( ∥ z j ∥ ) ( z j ) + ∑ c = 1 C ϕ ( c ) ( ∥ z j ∥ ) v j c ) − ∑ j = 1 m ( ϕ ( 0 ) ( ∥ z ^ j ∥ ) ( z ^ j ) + ∑ c = 1 C ϕ ( c ) ( ∥ z ^ j ∥ ) v ^ j c ) = 0 (21) \sum_{j=1}^{n}\left(\phi^{(0)}(\lVert z_j\rVert)(z_j)+\sum_{c=1}^{C}\phi^{(c)}(\lVert z_j\rVert)v_{jc}\right)-\sum_{j=1}^{m}\left(\phi^{(0)}(\lVert\widehat{z}_j\rVert)(\widehat{z}_j)+\sum_{c=1}^{C}\phi^{(c)}(\lVert\widehat{z}_j\rVert)\widehat{v}_{jc}\right)=0 \tag*{(21)} j = 1 ∑ n ( ϕ ( 0 ) (∥ z j ∥) ( z j ) + c = 1 ∑ C ϕ ( c ) (∥ z j ∥) v j c ) − j = 1 ∑ m ( ϕ ( 0 ) (∥ z j ∥) ( z j ) + c = 1 ∑ C ϕ ( c ) (∥ z j ∥) v j c ) = 0 (21) then
{ { ( z 1 , v 1 ) , … , ( z n , v n ) } } = { { ( z ^ 1 , v ^ 1 ) , … , ( z ^ m , v ^ m ) } } \{\{(z_1,v_1),\ldots,(z_n,v_n)\}\}=\{\{(\widehat{z}_1,\widehat{v}_1),\ldots,(\widehat{z}_m,\widehat{v}_m)\}\} {{( z 1 , v 1 ) , … , ( z n , v n )}} = {{( z 1 , v 1 ) , … , ( z m , v m )}} We note that this lemma is slightly stronger than we need since, in contrast with the original MPNN, we do not use the norms of v i , c v_{i,c} v i , c as features.
Our second lemma will show 3.19
Lemma G.3. Let v = ( v 1 , … , v C ) ∈ R d × C \mathbf{v}=(v_1,\ldots,v_C)\in\mathbb{R}^{d\times C} v = ( v 1 , … , v C ) ∈ R d × C and v ^ = ( v ^ 1 , … , v ^ C ) ∈ R d × C \hat{\mathbf{v}}=(\hat{v}_1,\ldots,\hat{v}_C)\in\mathbb{R}^{d\times C} v ^ = ( v ^ 1 , … , v ^ C ) ∈ R d × C . If for all θ ∈ R C \theta\in\mathbb{R}^{C} θ ∈ R C we have that
∥ ∑ c = 1 C θ c v c ∥ = ∥ ∑ c = 1 C θ c v ^ c ∥ \left\lVert\sum_{c=1}^{C}\theta_c v_c\right\rVert=\left\lVert\sum_{c=1}^{C}\theta_c\hat{v}_c\right\rVert c = 1 ∑ C θ c v c = c = 1 ∑ C θ c v ^ c then there exists an Q ∈ O ( d ) Q\in O(d) Q ∈ O ( d ) such that Q v c = v ^ c Qv_c=\hat{v}_c Q v c = v ^ c for all c = 1 , … , C c=1,\ldots,C c = 1 , … , C .
For proof of this lemma, see Section 3.2 in [DG24].
Our third lemma will show 3.20
Lemma G.4. Let s 1 , … , s n s_1,\ldots,s_n s 1 , … , s n and s ^ 1 , … , s ^ m \hat{s}_1,\ldots,\hat{s}_m s ^ 1 , … , s ^ m be vectors in R d \mathbb{R}^{d} R d . If for all basic analytic ϕ \phi ϕ , we have that ∑ i = 1 n ϕ ( s i ) = ∑ i = 1 m ϕ ( s ^ i ) \sum_{i=1}^{n}\phi(s_i)=\sum_{i=1}^{m}\phi(\hat{s}_i) ∑ i = 1 n ϕ ( s i ) = ∑ i = 1 m ϕ ( s ^ i ) then { { s 1 , … , s n } } = { { s ^ 1 , … , s ^ m } } \{\{s_1,\ldots,s_n\}\}=\{\{\hat{s}_1,\ldots,\hat{s}_m\}\} {{ s 1 , … , s n }} = {{ s ^ 1 , … , s ^ m }} .
For proof of this lemma, see Proposition 3.2 in [2 ] .
It remains to prove Lemma G.2. We note that if equation 3.21 holds for all basic analytic networks ϕ ( c ) \phi^{(c)} ϕ ( c ) , c = 0 , … , C c=0,\ldots,C c = 0 , … , C , this equality is also true if each ϕ ( c ) \phi^{(c)} ϕ ( c ) is a linear combination of a finite number of shallow neural networks. Next, since we are only considering the values of ϕ ( c ) \phi^{(c)} ϕ ( c ) on a fixed finite collection of inputs, and since linear combinations of shallow neural networks with analytic (or even continuous) non-polynomial functions can approximate any continuous function uniformly on a compact set (and in particular on a finite set) [Pin99], it follows that equation G.2 holds for all continuous ϕ ( c ) \phi^{(c)} ϕ ( c ) .
Next, we consider the norm multisets:
S = { { ∥ z 1 ∥ , … , ∥ z n ∥ } } , S ^ = { { ∥ z ^ 1 ∥ , … , ∥ z ^ m ∥ } } . S=\{\{\lVert z_1\rVert,\ldots,\lVert z_n\rVert\}\},\qquad\hat{S}=\{\{\lVert\hat{z}_1\rVert,\ldots,\lVert\hat{z}_m\rVert\}\}. S = {{∥ z 1 ∥ , … , ∥ z n ∥}} , S ^ = {{∥ z ^ 1 ∥ , … , ∥ z ^ m ∥}} .
We claim that S = S ^ S=\hat{S} S = S ^ . We first show that S ⊆ S ^ S\subseteq\hat{S} S ⊆ S ^ . Fix some s ∈ { 1 , … , n } s\in\{1,\ldots,n\} s ∈ { 1 , … , n } , we need to show that ∥ z s ∥ ∈ S ^ \lVert z_s\rVert\in\hat{S} ∥ z s ∥ ∈ S ^ . We define a continuous function f s f^s f s such that f s ( ∥ z s ∥ ) = 1 f^s(\lVert z_s\rVert)=1 f s (∥ z s ∥) = 1 and f s ( ∥ z i ∥ ) = 0 = f s ( ∥ z ^ j ∥ ) f^s(\lVert z_i\rVert)=0=f^s(\lVert\hat{z}_j\rVert) f s (∥ z i ∥) = 0 = f s (∥ z ^ j ∥) for all i ≠ s i\ne s i = s , and for all j = 1 , … , m j=1,\ldots,m j = 1 , … , m such that the norms ∥ z ^ j ∥ \lVert\hat{z}_j\rVert ∥ z ^ j ∥ and ∥ z s ∥ \lVert z_s\rVert ∥ z s ∥ are not equal. We claim that there exists j j j for which the norms are equal. Otherwise, we choose the continuous functions ϕ ( c ) \phi^{(c)} ϕ ( c ) , c = 1 , … , C c=1,\ldots,C c = 1 , … , C to be the zero functions, and choose ϕ ( 0 ) \phi^{(0)} ϕ ( 0 ) to be f s f^s f s . This will lead to the equation z s = 0 z_s=0 z s = 0 in G.2, which contradicts the assumption that z s ≠ 0 z_s\ne0 z s = 0 , so we see that S ⊆ S ^ S\subseteq\hat{S} S ⊆ S ^ . A symmetric argument shows that S ^ ⊆ S \hat{S}\subseteq S S ^ ⊆ S and so S = S ^ S=\hat{S} S = S ^ . We deduce that n = m n=m n = m and ∥ z i ∥ = ∥ z ^ i ∥ \lVert z_i\rVert=\lVert\hat{z}_i\rVert ∥ z i ∥ = ∥ z ^ i ∥ for all i = 1 , … , n i=1,\ldots,n i = 1 , … , n . We can now conclude the proof: fix an arbitrary s ∈ { 1 , … , n } s\in\{1,\ldots,n\} s ∈ { 1 , … , n } . Choose ϕ ( 0 ) = f s \phi^{(0)}=f^s ϕ ( 0 ) = f s and take the other ϕ ( c ) \phi^{(c)} ϕ ( c ) to be zero. Then G.2 gives
z s − z ^ s = 0 z_s-\hat{z}_s=0 z s − z ^ s = 0
Similarly, for an arbitrary index q ∈ { 1 , … , C } q\in\{1,\ldots,C\} q ∈ { 1 , … , C } , we can choose ϕ ( q ) = f s \phi^{(q)}=f^s ϕ ( q ) = f s and ϕ ( c ) = 0 \phi^{(c)}=0 ϕ ( c ) = 0 for all c ≠ q c\ne q c = q in { 0 , 1 , … , C } \{0,1,\ldots,C\} { 0 , 1 , … , C } . In this case G.2 gives
v j q − v ^ j q = 0. v_{jq}-\hat{v}_{jq}=0. v j q − v ^ j q = 0.
This concludes the proof of the lemma and the proof of the theorem.
Complexity
Running EGenNet for T T T iterations with C C C channels requires O ( C ⋅ d ⋅ deg i ) O(C \cdot d \cdot\operatorname{deg}_i) O ( C ⋅ d ⋅ deg i ) elementary arithmetic operations for every node i i i , at every iteration t t t . Overall, this requires O ( C ⋅ d ⋅ N ⋅ d a v g ⋅ T ) O(C \cdot d \cdot N \cdot d_{\mathrm{avg}} \cdot T) O ( C ⋅ d ⋅ N ⋅ d avg ⋅ T ) to run EGenNet, where d a v g d_{\mathrm{avg}} d avg denotes the average degree in the graph. This is linear in N N N , which dominates the other terms in the complexity bound. If we set C = 2 N d + 1 C = 2N^{d+1} C = 2 N d + 1 as required for maximal expressivity in Theorem 3.5.1, then the complexity will be quadratic in N N N , which is less by a factor of n n n than [DKR+23; LWHZ24], and by a factor of n 2 n^2 n 2 than [HAGD23]. We also note that the cardinality in the finite witness theorem, the main tool for our proof, depends on the intrinsic dimension of the input and not the ambient dimension. Thus, under the standard ’manifold assumption’ [Cay+08], which asserts that data in learning tasks typically resides in a manifold of low dimension r r r , embedded in a high dimensional Euclidean space, the proof can be adapted to show that the number of channels necessary for maximal expressivity is only 2 r + 1 2r + 1 2 r + 1 . Thus, under the manifold assumption, the complexity of EGenNet is linear in the number of nodes N N N .
Adding node and edge features
This section explains how we incorporate node and edge features into our architecture and theory.
A featured geometric graph is a 4-tuple G = ( A 1 , X 1 , F 1 , E 1 ) \mathcal{G} = (A_1, X_1, F_1, E_1) G = ( A 1 , X 1 , F 1 , E 1 ) where as before A A A is an n n n by n n n adjacency matrix, and X X X is a n n n by d d d matrix denoting node positions. F F F is a n n n by d n d_n d n matrix denoting node features (without the action of rigid motions), and E E E is a n × n × d e n \times n \times d_e n × n × d e tensor of edge features. We denote the feature at a node i i i and edge ( i , j ) (i,j) ( i , j ) by f i f_i f i (a column of F F F ) and f i , j f_{i,j} f i , j , respectively.
We incorporate edge and node features into EGenNet by slightly modifying the aggregation step to
v i , q ( t + 1 ) = ∑ j ∈ N i ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) ⋅ ( ϕ ( t , q , 0 ) ( ∥ x i − x j ∥ , ∥ v j ∥ ) ( x i − x j ) + ∑ c = 1 k ϕ ( t , q , c ) ( ∥ x i − x j ∥ , ∥ v j ∥ ) v j c ( t ) ) v_{i,q}^{(t+1)} = \sum_{j \in\mathcal{N}_i} \psi_n^{(t,q)}(f_i) \cdot\psi_e^{(t,q)}(f_{i,j}) \cdot\left(\phi^{(t,q,0)}(\lVert x_i-x_j\rVert,\lVert v_j\rVert)(x_i-x_j) + \sum_{c=1}^{k} \phi^{(t,q,c)}(\lVert x_i-x_j\rVert,\lVert v_j\rVert)v_{jc}^{(t)}\right) v i , q ( t + 1 ) = j ∈ N i ∑ ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) ⋅ ( ϕ ( t , q , 0 ) (∥ x i − x j ∥ , ∥ v j ∥) ( x i − x j ) + c = 1 ∑ k ϕ ( t , q , c ) (∥ x i − x j ∥ , ∥ v j ∥) v j c ( t ) )
In our analysis we will take ψ n ( t , q ) : R d n → R \psi_n^{(t,q)} : \mathbb{R}^{d_n} \to\mathbb{R} ψ n ( t , q ) : R d n → R and ψ e ( t , q ) : R d e → R \psi_e^{(t,q)} : \mathbb{R}^{d_e} \to\mathbb{R} ψ e ( t , q ) : R d e → R to be simple linear functions
ψ n ( t , q ) ( f i ) = a n ( t , q ) ⋅ f i + b n ( t , q ) , ψ e ( t , q ) ( f i , j ) = a e ( t , q ) ⋅ f i , j + b q ( t , q ) \psi_n^{(t,q)}(f_i) = a_n^{(t,q)} \cdot f_i + b_n^{(t,q)}, \qquad\psi_e^{(t,q)}(f_{i,j}) = a_e^{(t,q)} \cdot f_{i,j} + b_q^{(t,q)} ψ n ( t , q ) ( f i ) = a n ( t , q ) ⋅ f i + b n ( t , q ) , ψ e ( t , q ) ( f i , j ) = a e ( t , q ) ⋅ f i , j + b q ( t , q )
In practice, in our code, we use a slightly more complex function involving radial basis functions as in [LWHZ24].
To extend our analysis to the case of node and edge features, we first extend our definition of isomorphism of geometric graphs to include edge and node features.
Definition H.1. Given two featured geometric graphs
G = ( A 1 , X 1 , F 1 , E 1 ) , H = ( A 2 , X 2 , F 2 , E 2 ) \mathcal{G} = (A_1, X_1, F_1, E_1), \quad\mathcal{H} = (A_2, X_2, F_2, E_2) G = ( A 1 , X 1 , F 1 , E 1 ) , H = ( A 2 , X 2 , F 2 , E 2 ) G \mathcal{G} G and H \mathcal{H} H are featured geometrically isomorphic if there exists a permutation matrix P P P , a rotation matrix Q Q Q , and a translation t t t such that:
A 2 = P ⋅ A 1 ⋅ P T A_2 = P \cdot A_1 \cdot P^{T} A 2 = P ⋅ A 1 ⋅ P T .
X 2 = P ⋅ X 1 ⋅ Q + t X_2 = P \cdot X_1 \cdot Q + t X 2 = P ⋅ X 1 ⋅ Q + t
F 2 = P ⋅ F 1 F_2 = P \cdot F_1 F 2 = P ⋅ F 1
E 2 = P ⋅ E 1 ⋅ P T E_2 = P \cdot E_1 \cdot P^{T} E 2 = P ⋅ E 1 ⋅ P T
where P ⋅ E 1 ⋅ P T P \cdot E_1 \cdot P^{T} P ⋅ E 1 ⋅ P T should be understood as operating element-wise on the n e n_e n e matrices of dimension n × n n \times n n × n which compose the tensor E E E .
We can then prove the following theorem: a generalization of Theorem 3.5.2 to the featured graph case.
Theorem H.2. Let N N N , d d d , d e d_e d e , d n d_n d n be natural numbers. Let F θ F_\theta F θ denote the EGenNet architecture for featured geometric graphs, with depth T = d + 1 T = d+1 T = d + 1 and C = 2 N d + d e + d n + 1 C = 2Nd+d_e+d_n+1 C = 2 N d + d e + d n + 1 channels. Then for Lebesgue almost every θ \theta θ , we have for all featured graphs G , G ^ ∈ G N ( d ) \mathcal{G}, \widehat{\mathcal{G}} \in\mathcal{G}_N(d) G , G ∈ G N ( d ) which are generic and connected.
F θ ( G ) = F θ ( G ^ ) ⟺ G , G ^ are featured geometrically isomorphic F_\theta(\mathcal{G}) = F_\theta(\widehat{\mathcal{G}}) \Longleftrightarrow\mathcal{G},\widehat{\mathcal{G}}\text{ are featured geometrically isomorphic} F θ ( G ) = F θ ( G ) ⟺ G , G are featured geometrically isomorphic We note that the assumption that the geometric graph is generic is only on the position coordinates X X X . The statement will hold for all node and edge features in this case.
Proof. We explain how to modify the proof of Theorem 3.5.1 to obtain this result. First, we note that the number of channels was adapted to take into account the higher dimensionality of the data now that node and edge features are included. The finite witness theorem can be applied with this adapted dimension, as explained in Theorem 3.5.1. Thus, we only need to show that, for given featured graphs G ^ , G \widehat{\mathcal{G}}, \mathcal{G} G , G satisfying the conditions of the theorem, if for all θ \theta θ we have that F θ ( G ) = F θ ( G ^ ) F_\theta(\mathcal{G}) = F_\theta(\widehat{\mathcal{G}}) F θ ( G ) = F θ ( G ) , then G \mathcal{G} G and G ^ \widehat{\mathcal{G}} G are featured geometrically isomorphic.
Now, let us assume that, for all θ \theta θ we have F θ ( G ) = F θ ( G ^ ) F_\theta(\mathcal{G}) = F_\theta(\widehat{\mathcal{G}}) F θ ( G ) = F θ ( G ) . We can obtain all the results we had for the case where there were no additional features by choosing the parameters
a n ( t , q ) = 0 , b n ( t , q ) = 1 , a e ( t , q ) = 0 , b e ( t , q ) = 1 a_n^{(t,q)} = 0, b_n^{(t,q)} = 1, a_e^{(t,q)} = 0, b_e^{(t,q)} = 1 a n ( t , q ) = 0 , b n ( t , q ) = 1 , a e ( t , q ) = 0 , b e ( t , q ) = 1
so that ψ n ( t , q ) = 1 = ψ e ( t , q ) \psi_n^{(t,q)} = 1 = \psi_e^{(t,q)} ψ n ( t , q ) = 1 = ψ e ( t , q ) , and the effect of node and edge features on the architectures is ’cancelled out’. In particular, we can deduce that, after an appropriate permutation and rigid motion are applied to the data, the graphs have the same number of nodes n n n , and for all node i i i and iteration t t t , we have
v i ( t ) = v ^ i ( t ) , x i = x ^ i v_i^{(t)} = \hat{v}_i^{(t)}, x_i = \hat{x}_i v i ( t ) = v ^ i ( t ) , x i = x ^ i
Now, by setting ϕ ( t , q , c ) = 0 \phi^{(t,q,c)} = 0 ϕ ( t , q , c ) = 0 for c ≥ 1 c \ge1 c ≥ 1 , and considering ϕ ( t , q , 0 ) \phi^{(t,q,0)} ϕ ( t , q , 0 ) which depend only on the first coordinate, we can deduce that
∑ j ∈ N i [ ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) − ψ n ( t , q ) ( f ^ i ) ⋅ ψ e ( t , q ) ( f ^ i , j ) ] ⋅ ( ϕ ( t , q , 0 ) ( ∥ x i − x j ∥ ) ( x i − x j ) ) = 0 \sum_{j \in\mathcal{N}_i} \left[\psi_n^{(t,q)}(f_i) \cdot\psi_e^{(t,q)}(f_{i,j}) - \psi_n^{(t,q)}(\hat{f}_i) \cdot\psi_e^{(t,q)}(\hat{f}_{i,j})\right] \cdot\left(\phi^{(t,q,0)}(\lVert x_i-x_j\rVert)(x_i-x_j)\right) = 0 j ∈ N i ∑ [ ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) − ψ n ( t , q ) ( f ^ i ) ⋅ ψ e ( t , q ) ( f ^ i , j ) ] ⋅ ( ϕ ( t , q , 0 ) (∥ x i − x j ∥) ( x i − x j ) ) = 0
for all choices of parameters of the functions ψ n ( t , q ) \psi_n^{(t,q)} ψ n ( t , q ) , ψ e ( t , q ) \psi_e^{(t,q)} ψ e ( t , q ) and ϕ ( t , q , 0 ) \phi^{(t,q,0)} ϕ ( t , q , 0 ) . As we showed in the proof of Theorem 3.5.1, this equation implies that this equation will also hold for any continuous ϕ ( t , q , 0 ) \phi^{(t,q,0)} ϕ ( t , q , 0 ) (and any choice of parameters of the other functions) since the span of all such basic analytic functions is dense. In particular, for fixed i , j i,j i , j , we can choose ϕ ( t , q , 0 ) \phi^{(t,q,0)} ϕ ( t , q , 0 ) to be a continuous function with ϕ ( t , q , 0 ) ( ∥ x i − x j ∥ ) = 1 \phi^{(t,q,0)}(\lVert x_i-x_j\rVert) = 1 ϕ ( t , q , 0 ) (∥ x i − x j ∥) = 1 and ϕ ( t , q , 0 ) ( ∥ x i − x k ∥ ) = 0 \phi^{(t,q,0)}(\lVert x_i-x_k\rVert) = 0 ϕ ( t , q , 0 ) (∥ x i − x k ∥) = 0 if k ≠ j k \ne j k = j . Inserting this into the last equality we obtain
[ ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) − ψ n ( t , q ) ( f ^ i ) ⋅ ψ e ( t , q ) ( f ^ i , j ) ] ⋅ ( x i − x j ) = 0 \left[\psi_n^{(t,q)}(f_i) \cdot\psi_e^{(t,q)}(f_{i,j}) - \psi_n^{(t,q)}(\hat{f}_i) \cdot\psi_e^{(t,q)}(\hat{f}_{i,j})\right] \cdot(x_i-x_j) = 0 [ ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) − ψ n ( t , q ) ( f ^ i ) ⋅ ψ e ( t , q ) ( f ^ i , j ) ] ⋅ ( x i − x j ) = 0
by genericity x i − x j ≠ 0 x_i-x_j \ne0 x i − x j = 0 , and so we deduce that
ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) − ψ n ( t , q ) ( f ^ i ) ⋅ ψ e ( t , q ) ( f ^ i , j ) = 0 \psi_n^{(t,q)}(f_i) \cdot\psi_e^{(t,q)}(f_{i,j}) - \psi_n^{(t,q)}(\hat{f}_i) \cdot\psi_e^{(t,q)}(\hat{f}_{i,j}) = 0 ψ n ( t , q ) ( f i ) ⋅ ψ e ( t , q ) ( f i , j ) − ψ n ( t , q ) ( f ^ i ) ⋅ ψ e ( t , q ) ( f ^ i , j ) = 0
for every choice of function parameters. Setting a n ( t , q ) = 0 a_n^{(t,q)} = 0 a n ( t , q ) = 0 , b n ( t , q ) = 1 b_n^{(t,q)} = 1 b n ( t , q ) = 1 , b e ( t , q ) = 0 b_e^{(t,q)} = 0 b e ( t , q ) = 0 we obtain that
a e ⋅ ( f i j − f ^ i j ) = 0 , ∀ a e ∈ R d e a_e \cdot(f_{ij} - \hat{f}_{ij}) = 0,\quad\forall a_e \in\mathbb{R}^{d_e} a e ⋅ ( f ij − f ^ ij ) = 0 , ∀ a e ∈ R d e
and therefore f i j − f ^ i j = 0 f_{ij} - \hat{f}_{ij} = 0 f ij − f ^ ij = 0 . Similarly, setting a e ( t , q ) = 0 a_e^{(t,q)} = 0 a e ( t , q ) = 0 , b e ( t , q ) = 1 b_e^{(t,q)} = 1 b e ( t , q ) = 1 , b n ( t , q ) = 0 b_n^{(t,q)} = 0 b n ( t , q ) = 0 we obtain that
a n ⋅ ( f i − f ^ i ) = 0 , ∀ a n ∈ R d n a_n \cdot(f_i - \hat{f}_i) = 0,\quad\forall a_n \in\mathbb{R}^{d_n} a n ⋅ ( f i − f ^ i ) = 0 , ∀ a n ∈ R d n
and therefore f i − f ^ i = 0 f_i - \hat{f}_i = 0 f i − f ^ i = 0 . This concludes the proof of the theorem.
Additional Empirical Results
k-chain results Here, we show the results of the k k k -chain experiment for k = 4 k = 4 k = 4 . As we can see, only our method EGenNet and GVP [JES+20] succeed using the minimal number of blocks. This experiment illustrates our model’s ability to distinguish very sparse graphs. Next, we show results for k = 12 k = 12 k = 12 . As can be seen, we are the only ones succeeding with the minimal number of blocks, and other models need a higher number
Table 3.3: Separation Learning of 4-chain paths Figure 3.2. We show the learned accuracy over ten trials for each model, depending on the number of blocks. We show the minimal needed number of blocks (3) is sufficient for separation, and only [JES+20] also succeeded (among equivariant models working on the sparse graph) all 10 trials using 3 blocks.
Table 3.3. Separation Learning of 4-chain paths Figure 3.2. We show the learned accuracy over ten trials for each model, depending on the number of blocks. We show the minimal needed number of blocks (3) is sufficient for separation, and only [JES+20] also succeeded (among equivariant models working on the sparse graph) all 10 trials using 3 blocks.
of blocks to succeed in distinguishing.
Hard examples This section presents the results mentioned in § 5.5.1 . We use a dataset of four difficult-to-separate pairs of point clouds. Three (Hard1,Hard2,Hard3) are taken from [PWB+20]. The last pair, ’Harder,’ is taken from [PC22] and is an example of a non-isomorphic pair that I-GGNN models cannot distinguish. We show we succeeded in separating all tuples with a probability of 1. The tested models are EGENNET, GramNet [HAGD23], GeoEGNN [HAGD23], EGNN [SHW21], LinearEGNN, MACE [BKS+22], TFN [TSK+18], DimeNet [GGG20], GVPGNN [JES+20] and are taken from
Table 3.4 : Separation Learning of 12-chain paths Figure 3.2 . We show the learned accuracy over ten trials for each model depending on the number of blocks. As we can see, we are the only ones that succeed with the minimal number of blocks (7), and other models need many more layers to show high Performance.
Table 3.4. Separation Learning of 12-chain paths Figure 3.2. We show the learned accuracy over ten trials for each model depending on the number of blocks. As we can see, we are the only ones that succeed with the minimal number of blocks (7), and other models need many more layers to show high Performance.
[HAGD23].
Chemical property experiments with full statistics the table for the chemical property prediction tasks in the main text, Table 3.6 , was produced using the protocol from [ZHA+24]. In particular, each method was run with three different seeds. The seed that did best on the validation set was chosen, and its results on the test set were reported.
Table 3.6 presents each task’s mean and standard deviation over 3 seeds. As we can see, in most tasks, the standard deviation is rather low, and the results are qualitatively similar to those obtained in Table 3.2 . In particular, in the tasks in the Kraken and BDE datasets, we outperform the competitors, often by a large margin. In the Drugs 75K, we obtain comparable, but somewhat higher, results than the best method (while in Table 3.2 , in two of three Drugs-75K cases, our results were the best by a small margin).
Time measurements To compare our running time with other models, we measured the execution time for each model on the 12-chain task with all models using 128 hidden feature size. The results 3.7 show that our running time is on par with the competitors.
Norm Accuracies To heuristically estimate ‘how generic’ the molecules in the chemical property datasets [AG22; GdPF+22; MSH+18] we considered are, we computed all norms of atoms in each (centralized) molecule and counted the number of unique norms relative to the total number of points. We considered a norm unique if its distance from all other norms exceeded a specified threshold μ \mu μ , and considered various threshold values. The results 3.8 indicate that the norms exhibit considerable variation, suggesting that the molecules are close to being generic. This supports the conjecture stated in the main text that the surprising success of our simple method, compared to more expressive methods, is related to the near-genericity of the molecules in these datasets.
Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representations
Yonatan Sverdlov; Ido Springer; Nadav Dym
This paper explores the characterization of equivariant linear layers for representations of permutations and related groups. Unlike traditional approaches, which address these problems using parameter-sharing, we consider an alternative methodology based on irreducible representations and Schur’s lemma. Using this methodology, we obtain an alternative derivation for existing models like DeepSets, 2-IGN graph equivariant networks, and Deep Weight Space (DWS) networks. The derivation for DWS networks is significantly simpler than that of previous results. Next, we extend our approach to unaligned symmetric sets, where equivariance to the wreath product of groups is required. Previous works have addressed this problem in a rather restrictive setting, in which almost all wreath equivariant layers are Siamese. In contrast, we give a full characterization of layers in this case and show that there is a vast number of additional non-Siamese layers in some settings. We also show empirically that these additional non-Siamese layers can improve performance in tasks like graph anomaly detection, weight space alignment, and learning Wasserstein distances. Our code is available at GitHub.
Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representations
Introduction
Learning with symmetries has recently attracted great attention in machine learning. In this learning setting, a group acts on an input space, and the hypothesis mappings are restricted to be equivariant with respect to the group action. Motivated by the structure of fully connected neural networks, group equivariant models are often defined by a composition of parametric linear equivariant functions and non-parametric non-linear activation functions [CW16]. Thus, characterizing all equivariant layers for a given group action is fundamental for understanding and designing equivariant deep neural networks. Indeed, this question has attracted a considerable amount of attention in various scenarios (e.g., [FWW21; KT18; CGW19; Pea23a; Pea23b]).
For permutation groups, the standard strategy for characterizing equivariant layers is through studying parameter-sharing [RSP17]. Perhaps the first result in this direction was given by [ZKR+17], who showed that only two parameters are required to characterize all permutation equivariant layers on a set of scalars. Another famous example is [MBSL18], who described the specific parameter-sharing scheme for graph equivariant networks and other tensor representations of the symmetric group. Recently, parameter-sharing has been applied in [NSA+23; ZYB+24] to characterize all equivariant layers of weight space neural functionals. These neural networks operate on weights of another neural network, a problem with a complex multi-permutation equivariant structure.
In this paper, we revisit equivariant linear layer characterization for permutation groups from the perspective of irreducible representations. Given a group G \mathcal{G} G acting on a linear space V \mathcal{V} V , we explore linear equivariant mappings f : V → V f:\mathcal{V}\to\mathcal{V} f : V → V by decomposing V \mathcal{V} V to its minimal invariant components, which are irreducible representations of G \mathcal{G} G . Once this decomposition is performed, a basic result in representation theory called Schur’s lemma automatically provides us with a full characterization of all equivariant mappings.
As a first step, we use our method to get an alternative derivation for known equivariant layer characterization at the basis of the DeepSets [ZKR+17], 2-IGN [MBSL18], and DWSNets [NSA+23] architectures. Notably, for the Deep Weight Space problem, our derivation is significantly simpler than the cumbersome derivations previously obtained by parameter sharing [NSA+23; ZYB+24].
Next, we consider the problem of sets of unaligned symmetric elements, where equivariance to the joint action of a group G \mathcal{G} G and the permutation group S k S_k S k is required (also known as the wreath product G ≀ S k \mathcal{G}\wr S_k G ≀ S k ). This setting arises naturally when considering alignment, distance prediction, or anomaly detection of sets of unaligned symmetric elements. It was studied by [WAR20] under the assumption that G \mathcal{G} G acts transitively, in which case Siamese Networks capture an overwhelming majority of the equivariant layers. Indeed, in practice, Siamese networks are often employed [NSF+24; CW24]. In contrast, we give a full characterization of the equivariant layers in the general setting and show that in some settings (e.g., alignment of weights spaces), there can be a large number of non-Siamese equivariant layers. Empirically, we show that these additional layers improve performance on a synthetic graph anomaly detection task and the deep weight space alignment task discussed in [NSF+24]. In summary, our contributions are the following:
We give an alternative irreducible-based derivation for the DeepSets [ZKR+17], 2-IGN [MBSL18], and DWSNets [NSA+23] architectures. The DWSnets derivation is significantly simpler than previous approaches.
We empirically validate the importance of our equivariant layers for sets of unaligned symmetric elements by showing their superiority on a synthetic graph anomaly detection task, Wasserstein set distance computation, and deep weight space alignment task.
Related work
Learning Sets of Symmetric Elements. When dealing with multiple occurrences of symmetric objects as a set, additional symmetries arise as the set elements can be permuted. In this case, the layers are required to be equivariant under the permutation of elements in addition to the original element group action. [MLCF20] suggested a new layer called Deep Sets for Symmetric Elements (DSS) that operates on a set of symmetric objects. They showed that incorporating DSS layers is strictly more expressive than using Siamese networks. The DSS framework was also used in subgraph aggregation networks [BFL+21], where the network should be invariant to the order of the sub-graphs. In the main setting of [MLCF20], the same group element g ∈ G g \in G g ∈ G acts uniformly on all set objects. However, the more challenging setting is where we allow different elements g i ∈ G g_i \in G g i ∈ G to act on the objects. This corresponds to the action of the restricted wreath product G ≀ S k G \wr S_k G ≀ S k on the set. [MLCF20] deal with this setting as well, but only when G G G is a subgroup of S n S_n S n and the action of G G G is transitive. Later, [WAR20] characterized all linear mappings for general wreath products, however they also assumed that the group actions are transitive.
Equivariant Characterization Via Irreducibles. Using irreducible representations to characterize equivariant layers is a popular approach for rotation-equivariant networks [TSK+18; AHK19; DM]. For permutation groups, the primary work which considered this approach, to the best of our knowledge, is [HSK20]. They suggest using the known characterization of permutation irreducible representations with Young diagrams [FH13] to characterize permutation-equivariant layers and explain how to reconstruct the results from DeepSets and 2-IGN. Our approach for these results is similar but gives a more explicit derivation of the DeepSet and 2-IGN layers. More importantly, our analysis covers deep weight spaces and sets of unaligned symmetric elements, not discussed in this paper.
preliminaries
Let V \mathcal{V} V be a finite-dimensional vector space over the field F = C \mathbb{F} = \mathbb{C} F = C or R \mathbb{R} R . Let G \mathcal{G} G be a finite group acting linearly on V \mathcal{V} V . We will sometimes say that V \mathcal{V} V is a representation of G \mathcal{G} G . We will say V \mathcal{V} V is a trivial representation of G \mathcal{G} G if the action of G \mathcal{G} G on V \mathcal{V} V is the trivial action g v = v gv = v g v = v for all g ∈ G g \in\mathcal{G} g ∈ G and v ∈ V v \in\mathcal{V} v ∈ V .
A subspace U ⊆ V \mathcal{U} \subseteq\mathcal{V} U ⊆ V is invariant to G \mathcal{G} G if g ⋅ u ∈ U g \cdot u \in\mathcal{U} g ⋅ u ∈ U for all g ∈ G g \in\mathcal{G} g ∈ G and u ∈ U u \in\mathcal{U} u ∈ U . We say an invariant subspace U \mathcal{U} U is irreducible if it does not strictly contain an invariant subspace except the zero space. By Maschke’s theorem, each finite-dimensional space V \mathcal{V} V can be decomposed into a direct sum of irreducible invariant spaces [FH13].
A mapping T : V → V T : \mathcal{V} \to\mathcal{V} T : V → V is equivariant if T ( g ⋅ v ) = g ⋅ T ( v ) T(g \cdot v) = g \cdot T(v) T ( g ⋅ v ) = g ⋅ T ( v ) for every v ∈ V , g ∈ G v \in\mathcal{V}, g \in\mathcal{G} v ∈ V , g ∈ G . Two representations V , U \mathcal{V},\mathcal{U} V , U of G \mathcal{G} G are isomorphic if there is an equivariant linear bijection L V , U : V → U L_{\mathcal{V},\mathcal{U}} : \mathcal{V} \to\mathcal{U} L V , U : V → U , which we notate by V ≅ U \mathcal{V} \cong\mathcal{U} V ≅ U .
The following lemma, named Schur’s lemma, is a key component of our work. Schur’s lemma describes equivariant mappings between irreducible representations:
Schur’s lemma. Let V , W \mathcal{V}, \mathcal{W} V , W be finite-dimensional irreducible representations of G \mathcal{G} G over F \mathbb{F} F , and let T : V → W T : \mathcal{V} \to\mathcal{W} T : V → W be a G \mathcal{G} G -equivariant linear map.1 Then:
Either T T T is an isomorphism or T = 0 T = 0 T = 0 .
If L : V → W L : \mathcal{V} \to\mathcal{W} L : V → W is an isomorphism, and F = C \mathbb{F} = \mathbb{C} F = C , then T = λ L T = \lambda L T = λ L for λ ∈ C \lambda\in\mathbb{C} λ ∈ C .
In this paper, we will only consider representations over R \mathbb{R} R , which are the common setting in applications. In all cases, we will discuss, our real irreducible representations are absolutely irreducible . This means that their natural extension to complex vector spaces is irreducible. In this case, if L : V → W L : \mathcal{V} \to\mathcal{W} L : V → W is an isomorphism, F = R \mathbb{F} = \mathbb{R} F = R , and V \mathcal{V} V is absolutely irreducible , then T = λ L T = \lambda L T = λ L for λ ∈ R \lambda\in\mathbb{R} λ ∈ R [Boa]. In Appendix B we elaborate more on this topic. We note that when V \mathcal{V} V and W \mathcal{W} W are isomorphic irreducible real representations and V \mathcal{V} V is not absolutely irreducible, the space Hom G ( V , W ) \operatorname{Hom}_{\mathcal{G}}(\mathcal{V}, \mathcal{W}) Hom G ( V , W ) of equivariant linear mappings is either 2 or 4 dimensional [Poo16]. In this setting, using an automatic computational method to find all 2-4 equivariant layers may be beneficial [FWW21].
Layer characterization using Schur’s Lemma Equivariant layers from a representation V \mathcal{V} V to itself can be characterized using Schur’s lemma via two steps. The first step is decomposition, where we identify the decomposition of V \mathcal{V} V into irreducible representations V = ⨁ i = 1 s V i \mathcal{V} = \bigoplus_{i=1}^{s} \mathcal{V}_i V = ⨁ i = 1 s V i . Any x ∈ V x \in\mathcal{V} x ∈ V can be written uniquely as a sum x = ∑ i = 1 s x i x = \sum_{i=1}^{s} x_i x = ∑ i = 1 s x i where each x i x_i x i is in V i \mathcal{V}_i V i . The decomposition step also requires an algorithm to compute this decomposition. Next, we will need to identify which of the space pairs V i \mathcal{V}_i V i and V j \mathcal{V}_j V j are isomorphic, and if they are, specify an isomorphism L i j L_{ij} L ij . Finally, since we are working over the reals, we will need to verify that all irreducible representations are absolutely irreducible.
Once the decomposition step is carried out, the second step uses Schur’s lemma. Equivariant mappings T : V → V T:\mathcal{V}\to\mathcal{V} T : V → V can be written as T = ∑ i , j = 1 s T i j T=\sum_{i,j=1}^{s}T_{ij} T = ∑ i , j = 1 s T ij , where T i j : V i → V j T_{ij}:\mathcal{V}_i\to\mathcal{V}_j T ij : V i → V j are equivariant. By Schur’s lemma if V i \mathcal{V}_i V i and V j \mathcal{V}_j V j are isomorphic then T i j = λ i j L i j T_{ij}=\lambda_{ij}L_{ij} T ij = λ ij L ij for some λ i j ∈ R \lambda_{ij}\in\mathbb{R} λ ij ∈ R , and otherwise T i j = 0 T_{ij}=0 T ij = 0 . All in all, we obtain that T T T is equivariant if and only if for some choice of the scalar λ i j \lambda_{ij} λ ij we have
T ( x 1 + … + x s ) = ∑ ( i , j ) , V i ≅ V j λ i j L i j ( x i ) . (22) T(x_1+\ldots+x_s)=\sum_{\substack{(i,j),\mathcal{V}_i\cong\mathcal{V}_j}}\lambda_{ij}L_{ij}(x_i). \tag*{(22)} T ( x 1 + … + x s ) = ( i , j ) , V i ≅ V j ∑ λ ij L ij ( x i ) . (22)
The number of parameters in the expression above depends on the isomorphism relations between the irreducibles. We can assume without loss of generality that the first 1 ≤ t ≤ s 1\leq t\leq s 1 ≤ t ≤ s irreducible spaces are not isomorphic, and by identifying isomorphic irreducibles, we can rewrite the decomposition above as V = ⨁ j = 1 t V j ⊕ α j \mathcal{V}=\bigoplus_{j=1}^{t}\mathcal{V}_j^{\oplus\alpha_j} V = ⨁ j = 1 t V j ⊕ α j (Where V j ⊕ α j \mathcal{V}_j^{\oplus\alpha_j} V j ⊕ α j is defined to be the direct sum of α j \alpha_j α j copies of V j \mathcal{V}_j V j ) where α 1 + … + α t = s \alpha_1+\ldots+\alpha_t=s α 1 + … + α t = s . The number of parameters λ i j \lambda_{ij} λ ij in the decomposition is then ∑ i = 1 t α i 2 \sum_{i=1}^{t}\alpha_i^2 ∑ i = 1 t α i 2 . This process can easily be generalized to compute mappings between representations V \mathcal{V} V and W \mathcal{W} W by decomposing both spaces into irreducibles.
We note that the cornerstones of this methodology: decomposition into irreducibles and Schur’s Lemma, are applicable for all finite dimensional representations of finite groups (and also for compact infinite groups like S O ( d ) SO(d) S O ( d ) ). The main challenge in this approach is characterizing and computing the decomposition into irreducibles. This needs to be done on a case to case basis. Much of the remainder of the paper will be devoted to computing these decompositions for important equivariant learning scenarios.
Computing linear equivariant layers
We now demonstrate how to derive the results in [ZKR+17; MBSL18; NSA+23] via decomposition into irreducibles.
Deep Sets
We begin with the simple case of the action of the permutation group S n S_n S n on V = R n \mathcal{V}=\mathbb{R}^n V = R n by permutation of elements (we assume n ≥ 2 n\geq2 n ≥ 2 ). Here, there are two non-zero invariant spaces:
S = { α ⋅ 1 n ∣ α ∈ R } , V ( n ) = { x ∈ R n : ∑ i = 1 n x i = 0 } . (23) S=\{\alpha\cdot\mathbf{1}_n\mid\alpha\in\mathbb{R}\},\qquad\mathcal{V}(n)=\left\{x\in\mathbb{R}^n:\sum_{i=1}^{n}x_i=0\right\}. \tag*{(23)} S = { α ⋅ 1 n ∣ α ∈ R } , V ( n ) = { x ∈ R n : i = 1 ∑ n x i = 0 } . (23)
As the first space is one-dimensional, it’s irreducible. By [HOT06], the second one is also irreducible. These two spaces are not isomorphic since the action of S n S_n S n on S S S is trivial, but the action of S n S_n S n on V ( n ) \mathcal{V}(n) V ( n ) is not (also they do not have the same dimension when n ≥ 3 n\geq3 n ≥ 3 ).
The decomposition of x ∈ R n x\in\mathbb{R}^n x ∈ R n to a sum of elements from the irreducible spaces can be computed as
x = x ˉ 1 n + ( x − x ˉ 1 n ) x = \bar{x}1_n + (x - \bar{x}1_n) x = x ˉ 1 n + ( x − x ˉ 1 n )
where x ˉ \bar{x} x ˉ is the average of x x x , and 1 n 1_n 1 n is the all one vector. By Equation (22) , Schur’s lemma, and the fact that all irreducibles of the permutation group are absolutely irreducible (see Appendix B), the linear equivariant mappings T : V → V T : \mathcal{V} \to\mathcal{V} T : V → V are characterized by two parameters a , b ∈ R a,b \in\mathbb{R} a , b ∈ R , that is:
T x = a x ˉ 1 n + b ( x − x ˉ 1 n ) . Tx = a\bar{x}1_n + b(x - \bar{x}1_n). T x = a x ˉ 1 n + b ( x − x ˉ 1 n ) .
This is exactly the result described in DeepSets [ZKR+17].
Equivariant graph layers
We next consider the setting where V = R n × n \mathcal{V} = \mathbb{R}^{n \times n} V = R n × n , G = S n \mathcal{G} = S_n G = S n and the group action is defined by ( τ ⋅ X ) i j = X τ − 1 ( i ) , τ − 1 ( j ) (\tau\cdot X)_{ij} = X_{\tau^{-1}(i),\tau^{-1}(j)} ( τ ⋅ X ) ij = X τ − 1 ( i ) , τ − 1 ( j ) . This setting is natural for graph neural networks, as two adjacency matrices A , B ∈ R n × n A,B \in\mathbb{R}^{n \times n} A , B ∈ R n × n represent isomorphic graphs if and only if A = τ ⋅ B A = \tau\cdot B A = τ ⋅ B for some permutation τ \tau τ . This setting was considered in [MBSL18] using a parameter sharing scheme, and we show how to obtain similar results using irreducibles (note that [MBSL18] also discussed general mappings between k k k -order and ℓ \ell ℓ -order tensors. In contrast, we only discuss the k = ℓ = 2 k = \ell= 2 k = ℓ = 2 case).
We claim that, when n ≥ 4 n \ge4 n ≥ 4 , the space R n × n \mathbb{R}^{n \times n} R n × n can be written as a sum of seven irreducible permutation invariant sub-spaces. The behavior for n < 4 n < 4 n < 4 is discussed in Appendix C.3.
The first two spaces are one-dimensional representations on which the action of S n S_n S n is trivial: diagonal matrices with identical diagonal entries and matrices with zero diagonal and identical off-diagonal entries:
V 0 = { a ⋅ I n ∣ a ∈ R } , V 1 = { a ⋅ ( 1 n × n − I n ) ∣ a ∈ R } . (24) \mathcal{V}_0 = \{a \cdot I_n \mid a \in\mathbb{R}\}, \qquad\mathcal{V}_1 = \{a \cdot(1_{n \times n} - I_n) \mid a \in\mathbb{R}\}. \tag*{(24)} V 0 = { a ⋅ I n ∣ a ∈ R } , V 1 = { a ⋅ ( 1 n × n − I n ) ∣ a ∈ R } . (24)
Next, we have three spaces of dimension n − 1 n - 1 n − 1 which are isomorphic to V ( n ) \mathcal{V}(n) V ( n ) from equation 4.2. The first space is the space V 2 \mathcal{V}_2 V 2 of diagonal matrices whose diagonal sums to zero and the next two n − 1 n - 1 n − 1 dimensional spaces are the space of matrices whose rows (respectively columns) are constant, and columns (respectively rows) sum to zero:
V 3 = { r 1 n T ∣ ∑ i = 1 n r i = 0 } , V 4 = { 1 n c T ∣ ∑ i = 1 n c i = 0 } \mathcal{V}_3 = \{r1_n^{T} \mid\sum_{i=1}^{n} r_i = 0\}, \qquad\mathcal{V}_4 = \{1_n c^{T} \mid\sum_{i=1}^{n} c_i = 0\} V 3 = { r 1 n T ∣ i = 1 ∑ n r i = 0 } , V 4 = { 1 n c T ∣ i = 1 ∑ n c i = 0 }
Finally, there are two larger irreducible spaces:
V 5 = { A ∣ A = − A T , A 1 n = 0 n } , V 6 = { A ∣ A = A T , A 1 n = 0 n , A i i = 0 , ∀ i = 1 , … , n } \mathcal{V}_5 = \{A \mid A = -A^{T}, A1_n = 0_n\}, \qquad\mathcal{V}_6 = \{A \mid A = A^{T}, A1_n = 0_n, A_{ii} = 0, \forall i = 1,\ldots,n\} V 5 = { A ∣ A = − A T , A 1 n = 0 n } , V 6 = { A ∣ A = A T , A 1 n = 0 n , A ii = 0 , ∀ i = 1 , … , n }
The dimension of V 5 \mathcal{V}_5 V 5 is ( n − 1 2 ) = n 2 − 3 n 2 + 1 \binom{n-1}{2} = \frac{n^2 - 3n}{2} + 1 ( 2 n − 1 ) = 2 n 2 − 3 n + 1 , and the dimension of V 6 \mathcal{V}_6 V 6 is ( n − 1 2 ) − 1 = \binom{n-1}{2} - 1 = ( 2 n − 1 ) − 1 = n 2 − 3 n 2 \frac{n^2-3n}{2} 2 n 2 − 3 n . So, we have the following decomposition:
V = V 0 ⊕ V 1 ⊕ V 2 ⊕ V 3 ⊕ V 4 ⊕ V 5 ⊕ V 6 ≅ V 0 ⊕ 2 ⊕ V 3 ⊕ 3 ⊕ V 5 ⊕ V 6 \mathcal{V} = \mathcal{V}_0 \oplus\mathcal{V}_1 \oplus\mathcal{V}_2 \oplus\mathcal{V}_3 \oplus\mathcal{V}_4 \oplus\mathcal{V}_5 \oplus\mathcal{V}_6 \cong\mathcal{V}_0^{\oplus2} \oplus\mathcal{V}_3^{\oplus3} \oplus\mathcal{V}_5 \oplus\mathcal{V}_6 V = V 0 ⊕ V 1 ⊕ V 2 ⊕ V 3 ⊕ V 4 ⊕ V 5 ⊕ V 6 ≅ V 0 ⊕ 2 ⊕ V 3 ⊕ 3 ⊕ V 5 ⊕ V 6
Accordingly, using Schur’s lemma, each linear mapping can be characterized by 3 2 + 2 2 + 1 + 1 = 15 3^2 + 2^2 + 1 + 1 = 15 3 2 + 2 2 + 1 + 1 = 15 parameters, the same result from [MBSL18], so we have the expected number of parameters. In appendix C, we give a formal proof of these arguments and present an algorithm to decompose an input matrix X \mathbf{X} X into its seven irreducible spaces with linear complexity in the matrix size n 2 n^2 n 2 . These results are summarized in the following theorem.
Theorem 4.4.1. For all n ≥ 4 n \ge4 n ≥ 4 , the space R n × n \mathbb{R}^{n \times n} R n × n can be written as a direct sum of the spaces V 0 , … , V 6 \mathcal{V}_0,\ldots,\mathcal{V}_6 V 0 , … , V 6 . These spaces are invariant and irreducible, and the isomorphism relations between them are given by V 0 ≅ V 1 \mathcal{V}_0 \cong\mathcal{V}_1 V 0 ≅ V 1 , V 2 ≅ V 3 ≅ V 4 \mathcal{V}_2 \cong\mathcal{V}_3 \cong\mathcal{V}_4 V 2 ≅ V 3 ≅ V 4 .
Deep weight spaces
Recently, there has been growing interest in devising neural operators: neural networks that operate on an input, which is itself a neural network. This type of problem is of interest for various tasks involving post-processing and synthesis of multiple trained neural networks, as well as for processing Implicit Neural Representations (INRs), which are a popular alternative for representing certain standard data structures. (see e.g. [KBP24] for further discussion).
A key concept in the design of ’neural operators’ has been the requirement that they are equivariant to the input neural data [KBP24; KKZ+24; ZFH24; ZYJ+23; LML+24]. In this section we consider the setting discussed in [NSA+23; ZYB+24], where the neural data is a collection of weights and biases ( W m , b m ) m = 1 M (W_m,b_m)_{m=1}^{M} ( W m , b m ) m = 1 M representing an MLP of depth M M M , and a neural operator is constructed by composing standard activation with linear mappings which are equivariant with respect to the multi-permutation action we will now describe:
The output of an MLP architecture is invariant to the permutation of its hidden neurons. For example, the MLP defined by W 2 ⋅ ReLU ⋅ ( W 1 x ) W_2 \cdot\operatorname{ReLU} \cdot(W_1x) W 2 ⋅ ReLU ⋅ ( W 1 x ) will remain the same function if we replace weights W 2 , W 1 W_2,W_1 W 2 , W 1 with the weights W 2 P , P T W 1 W_2P,P^{T}W_1 W 2 P , P T W 1 .
To define the symmetries of learning on MLPs in full generality, we will adapt the notation from [NSA+23]. We consider the space of MLP parameters with a given fixed depth M M M and layer dimensions d 0 , … , d M d_0,\ldots,d_M d 0 , … , d M (M M M and all d j d_j d j are assumed to be larger than one). These are parameterized by the vector space V = ⨁ m = 1 M ( W m ⊕ B m ) \mathcal{V} = \bigoplus_{m=1}^{M}(W_m \oplus B_m) V = ⨁ m = 1 M ( W m ⊕ B m ) where W m : = R d m × d m − 1 W_m := \mathbb{R}^{d_m \times d_{m-1}} W m := R d m × d m − 1 and B m : = R d m B_m := \mathbb{R}^{d_m} B m := R d m represent the weights and biases of the m m m -th layer. The symmetry group of the weight space is the direct product of symmetric groups G = S d 1 × ⋯ × S d M − 1 \mathcal{G} = S_{d_1} \times\cdots\times S_{d_{M-1}} G = S d 1 × ⋯ × S d M − 1 . An element g = ( τ 1 , … , τ M − 1 ) g=(\tau_1,\ldots,\tau_{M-1}) g = ( τ 1 , … , τ M − 1 ) in the group acts on an element v = [ W m , b m ] m ∈ [ M ] v=[W_m,b_m]_{m\in[M]} v = [ W m , b m ] m ∈ [ M ] from V \mathcal{V} V as follows:
ρ ( g ) v = [ W m ′ , b m ′ ] m ∈ [ M ] , (25) \rho(g)v = [W'_m,b'_m]_{m\in[M]}, \tag*{(25)} ρ ( g ) v = [ W m ′ , b m ′ ] m ∈ [ M ] , (25)
W 1 ′ = P τ 1 T W 1 , b 1 ′ = P τ 1 T b 1 , (26) W'_1 = P_{\tau_1}^{T}W_1,\quad b'_1 = P_{\tau_1}^{T}b_1, \tag*{(26)} W 1 ′ = P τ 1 T W 1 , b 1 ′ = P τ 1 T b 1 , (26)
W m ′ = P τ m T W m P τ m − 1 , b m ′ = P τ m T b m , m ∈ [ 2 , M − 1 ] (27) W'_m = P_{\tau_m}^{T}W_mP_{\tau_{m-1}},\quad b'_m = P_{\tau_m}^{T}b_m,\quad m\in[2,M-1] \tag*{(27)} W m ′ = P τ m T W m P τ m − 1 , b m ′ = P τ m T b m , m ∈ [ 2 , M − 1 ] (27)
W M ′ = W M P τ M − 1 , b M ′ = b M . (28) W'_M = W_MP_{\tau_{M-1}},\quad b'_M = b_M. \tag*{(28)} W M ′ = W M P τ M − 1 , b M ′ = b M . (28)
where P τ m ∈ R d m × d m P_{\tau_m}\in\mathbb{R}^{d_m\times d_m} P τ m ∈ R d m × d m is the permutation matrix of τ m ∈ S d m \tau_m\in S_{d_m} τ m ∈ S d m .
Previous work [NSA+23; ZYB+24] has already characterized all linear equivariant functions from V \mathcal{V} V to itself. However, this characterization requires tedious bookkeeping and division into a large number of different cases. Here we show how to decompose V \mathcal{V} V into irreducibles in a rather straightforward way, and as a result obtain an (arguably) simpler characterization of all linear equivariant layers.
We claim that V \mathcal{V} V is a direct sum of multiple copies of 2 M − 2 2M-2 2 M − 2 irreducible representations of G \mathcal{G} G :
The first irreducible representation is the trivial scalar representation S = R \mathbf{S}=\mathbb{R} S = R with the trivial action g x = x gx=x g x = x .
The next M − 1 M-1 M − 1 representations are the vector representations V m \mathbf{V}_m V m , m = 1 , … , M − 1 m=1,\ldots,M-1 m = 1 , … , M − 1 of vectors in R d m \mathbb{R}^{d_m} R d m which sum to zero (the spaces V ( d m ) \mathbf{V}(d_m) V ( d m ) from equation 4.2), with the action of g g g being permutation of the vector by the m m m -th permutation τ m \tau_m τ m .
The last M − 2 M-2 M − 2 representations are representations M m \mathbf{M}_m M m , m = 2 , … , M − 1 m=2,\ldots,M-1 m = 2 , … , M − 1 of d m × d m − 1 d_m\times d_{m-1} d m × d m − 1 matrices whose rows and columns sum to zero, where the action of g g g on a matrix W W W in this space is given by P τ m T W P τ m − 1 P_{\tau_m}^{T}WP_{\tau_{m-1}} P τ m T W P τ m − 1 .
The linear equivariant layers from V \mathcal{V} V to itself can be inferred from the following theorem:
Theorem 4.4.2. The spaces S , V 1 , … , V M − 1 , M 2 , … , M M − 1 \mathbf{S},\mathbf{V}_1,\ldots,\mathbf{V}_{M-1},\mathbf{M}_2,\ldots,\mathbf{M}_{M-1} S , V 1 , … , V M − 1 , M 2 , … , M M − 1 are absolutely irreducible and pairwise non-isomorphic.
If M ≥ 3 M\ge3 M ≥ 3 , then
V ≃ S ⊕ α ⊕ ( ⨁ m = 1 M − 1 V m ⊕ β m ) ⊕ ( ⨁ m = 2 M − 1 M m ) , \mathcal{V}\simeq\mathbf{S}^{\oplus\alpha}\oplus\left(\bigoplus_{m=1}^{M-1}\mathbf{V}_m^{\oplus\beta_m}\right)\oplus\left(\bigoplus_{m=2}^{M-1}\mathbf{M}_m\right), V ≃ S ⊕ α ⊕ ( m = 1 ⨁ M − 1 V m ⊕ β m ) ⊕ ( m = 2 ⨁ M − 1 M m ) , where
α = d 0 + 2 d M + 2 M − 3 , β 1 = d 0 + 2 , β M − 1 = d M + 2 , β m = 3 , m = 2 , … , M − 2. (29) \alpha=d_0+2d_M+2M-3,\qquad\beta_1=d_0+2,\qquad\beta_{M-1}=d_M+2,\qquad\beta_m=3,\quad m=2,\ldots,M-2. \tag*{(29)} α = d 0 + 2 d M + 2 M − 3 , β 1 = d 0 + 2 , β M − 1 = d M + 2 , β m = 3 , m = 2 , … , M − 2. (29) Consequently, the space of equivariant linear mappings from V \mathcal{V} V to itself has dimension
α 2 + ∑ m = 1 M − 1 β m 2 + ( M − 2 ) . \alpha^2+\sum_{m=1}^{M-1}\beta_m^2+(M-2). α 2 + m = 1 ∑ M − 1 β m 2 + ( M − 2 ) . If M = 2 M=2 M = 2 , then
V ≃ S ⊕ α ⊕ V 1 ⊕ β 1 , \mathcal{V}\simeq\mathbf{S}^{\oplus\alpha}\oplus\mathbf{V}_1^{\oplus\beta_1}, V ≃ S ⊕ α ⊕ V 1 ⊕ β 1 ,
where
α = d 0 + 2 d 2 + 1 , β 1 = d 0 + d 2 + 1 , \alpha= d_{0} + 2d_{2} + 1,\qquad\beta_{1} = d_{0} + d_{2} + 1, α = d 0 + 2 d 2 + 1 , β 1 = d 0 + d 2 + 1 ,
and hence the space of equivariant linear mappings from V \mathcal{V} V to itself has dimension
α 2 + β 1 2 . \alpha^{2} + \beta_{1}^{2}. α 2 + β 1 2 .
Proof. The spaces S , V 1 , … , V M − 1 , M 2 , … , M M − 1 \mathbf{S}, \mathbf{V}_{1}, \ldots, \mathbf{V}_{M-1}, \mathbf{M}_{2}, \ldots, \mathbf{M}_{M-1} S , V 1 , … , V M − 1 , M 2 , … , M M − 1 are not isomorphic (even when they have the same dimension). The action of G \mathcal{G} G on S \mathbf{S} S is trivial while the action on the other spaces is not. The other spaces are pairwise non-isomorphic because different factors of G \mathcal{G} G act non-trivially on them: V m \mathbf{V}_{m} V m is acted on non-trivially only by S d m S_{d_{m}} S d m , whereas M m \mathbf{M}_{m} M m is acted on non-trivially by S d m − 1 S_{d_{m-1}} S d m − 1 and S d m S_{d_{m}} S d m . Hence, these representations have different kernels and therefore cannot be isomorphic. The spaces S \mathbf{S} S and V m \mathbf{V}_{m} V m are absolutely irreducible, as we discussed in Subsection 4.4.1. We will explain why M m \mathbf{M}_{m} M m are absolutely irreducible in Appendix D.
Obtaining the decomposition of V \mathcal{V} V is rather straightforward. First, following [NSA+23], we identify the weight spaces W m \mathcal{W}_{m} W m and bias spaces B m \mathcal{B}_{m} B m with subspaces W ^ m \widehat{\mathcal{W}}_{m} W m and B ^ m \widehat{\mathcal{B}}_{m} B m of V \mathcal{V} V by zero padding, and note that each one of these subspaces is G \mathcal{G} G -invariant and that their direct sum gives us the full parameter space V \mathcal{V} V . We can then decompose each one of these spaces into irreducibles to obtain
B ^ m ≅ S ⊕ V m , m = 1 , … , M − 1 (30) \widehat{\mathcal{B}}_{m} \cong\mathbf{S} \oplus\mathbf{V}_{m},\qquad m=1,\ldots,M-1 \tag*{(30)} B m ≅ S ⊕ V m , m = 1 , … , M − 1 (30)
B ^ M ≅ S ⊕ d M (31) \widehat{\mathcal{B}}_{M} \cong\mathbf{S}^{\oplus d_{M}} \tag*{(31)} B M ≅ S ⊕ d M (31)
W ^ 1 ≅ S ⊕ d 0 ⊕ V 1 ⊕ d 0 (32) \widehat{\mathcal{W}}_{1} \cong\mathbf{S}^{\oplus d_{0}} \oplus\mathbf{V}_{1}^{\oplus d_{0}} \tag*{(32)} W 1 ≅ S ⊕ d 0 ⊕ V 1 ⊕ d 0 (32)
W ^ m ≅ S ⊕ V m − 1 ⊕ V m ⊕ M m , m = 2 , … , M − 1 (33) \widehat{\mathcal{W}}_{m} \cong\mathbf{S} \oplus\mathbf{V}_{m-1} \oplus\mathbf{V}_{m} \oplus\mathbf{M}_{m},\qquad m=2,\ldots,M-1 \tag*{(33)} W m ≅ S ⊕ V m − 1 ⊕ V m ⊕ M m , m = 2 , … , M − 1 (33)
W ^ M ≅ S ⊕ d M ⊕ V M − 1 ⊕ d M (34) \widehat{\mathcal{W}}_{M} \cong\mathbf{S}^{\oplus d_{M}} \oplus\mathbf{V}_{M-1}^{\oplus d_{M}} \tag*{(34)} W M ≅ S ⊕ d M ⊕ V M − 1 ⊕ d M (34)
The multiplicities of each irreducible in V \mathcal{V} V , specified in equation (29) , can now be found by simply counting how many times each irreducible appears in the decomposition above.
The decomposition in equation 4.6 can actually be easily obtained from the ’deepsets’ decomposition of the action of S n S_{n} S n on R n \mathbb{R}^{n} R n described previously: the action of G \mathcal{G} G on B ^ m \widehat{\mathcal{B}}_{m} B m with m < M m<M m < M is isomorphic to the action of S d m S_{d_{m}} S d m on R d m \mathbb{R}^{d_{m}} R d m and hence we get the exact ’DeepSets’ decomposition from § 7.4.1 . The action of G \mathcal{G} G on B ^ M \widehat{\mathcal{B}}_{M} B M is trivial and hence can be written as a direct sum of d M d_{M} d M trivial one dimensional S \mathbf{S} S spaces. The action on W ^ 1 \widehat{\mathcal{W}}_{1} W 1 (and W ^ M \widehat{\mathcal{W}}_{M} W M ) multiplies a matrix by a permutation from the left (right), and hence can be seen as a direct sum of ‘deep-sets’ actions on the columns (rows) of the matrix. Finally, the action of G \mathcal{G} G on W ^ m \widehat{\mathcal{W}}_{m} W m for m = 2 , … , M − 1 m=2,\ldots,M-1 m = 2 , … , M − 1 is a tensor product of the natural action of S d m − 1 S_{d_{m-1}} S d m − 1 on R d m − 1 \mathbb{R}^{d_{m-1}} R d m − 1 and the action of S d m S_{d_{m}} S d m on R d m \mathbb{R}^{d_{m}} R d m , and therefore its irreducible decomposition can be obtained by taking tensor products of the irreducible representations of R d m \mathbb{R}^{d_{m}} R d m and R d m − 1 \mathbb{R}^{d_{m-1}} R d m − 1 , as explained in more detail in the full proof.
We note that equation 4.6 includes almost all information required to compute linear equivariant maps from V \mathcal{V} V to itself. All that is left is the decomposition algorithm for writing each ( W m , B m ) m ∈ [ M ] (\mathcal{W}_m,\mathcal{B}_m)_{m\in[M]} ( W m , B m ) m ∈ [ M ] as a direct sum of elements in the irreducible decomposition. This can be done independently for each subspace W m \mathcal{W}_m W m and B m \mathcal{B}_m B m . The decomposition for W ^ m \widehat{\mathcal{W}}_m W m in equation 4.6d is not immediate and we will explain it in Appendix D.1.
The decomposition in equation 4.6 and equation 4.5 also provides substantial additional information. For example, we can immediately see that there are α \alpha α invariant maps from V \mathcal{V} V to R = S \mathbb{R}=\mathcal{S} R = S , which correspond to the number of copies of the trivial representation S \mathcal{S} S in V \mathcal{V} V . Moreover, if we are interested in the equivariant maps from a bias space B ^ i \widehat{\mathcal{B}}_i B i (or weight space W ^ i \widehat{\mathcal{W}}_i W i ) to another bias space B ^ j \widehat{\mathcal{B}}_j B j (or weight space W ^ j \widehat{\mathcal{W}}_j W j ), we can easily infer the equivariant mappings from the decomposition in equation 4.6. For example, when i = j = M i=j=M i = j = M there will be d M 2 d_M^2 d M 2 mappings since B ^ M \widehat{\mathcal{B}}_M B M consists of d M d_M d M isomorphic representations, and when i < j = M i<j=M i < j = M there will be d M d_M d M mappings since S \mathcal{S} S appears a single time in B ^ i \widehat{\mathcal{B}}_i B i and d M d_M d M times in B ^ M \widehat{\mathcal{B}}_M B M . Continuing in this way we can reconstruct all the different bias-to-bias, bias-to-weight, weight-to-bias, and weight-to-weight cases analyzed in Tables 5-8 of [NSA+23], and Tables 8-11 in [ZYB+24]. More importantly, these tables which were necessary for implementing weight space layers in previous work, are not necessary when implementing these layers using Schur’s lemma as we suggest.
Sets of unaligned symmetric elements
Next, we consider the setting where our data is a k k k -tuple of ’unaligned objects’ ( v 1 , … , v k ) (v_1,\ldots,v_k) ( v 1 , … , v k ) , each coming from a representation V \mathcal{V} V of G \mathcal{G} G , and we want to learn functions which are equivariant with respect to the joint action of a k k k -tuple of group elements ( g 1 , … , g k ) (g_1,\ldots,g_k) ( g 1 , … , g k ) on each coordinate independently, and to a permutation τ ∈ S k \tau\in S_k τ ∈ S k of the k k k -tuple.
We define
G ≀ S k : = G k ⋊ S k , \mathcal{G}\wr S_k:=\mathcal{G}^k\rtimes S_k, G ≀ S k := G k ⋊ S k ,
where S k S_k S k acts on G k \mathcal{G}^k G k by permuting its coordinates. Writing an element of G ≀ S k \mathcal{G}\wr S_k G ≀ S k as ( ( g 1 , … , g k ) , τ ) ((g_1,\ldots,g_k),\tau) (( g 1 , … , g k ) , τ ) , its action on V k \mathcal{V}^k V k is given by
( ( g 1 , … , g k ) , τ ) ⋅ ( v 1 , … , v k ) = ( g 1 ⋅ v τ − 1 ( 1 ) , … , g k ⋅ v τ − 1 ( k ) ) . (35) ((g_1,\ldots,g_k),\tau)\cdot(v_1,\ldots,v_k)=(g_1\cdot v_{\tau^{-1}(1)},\ldots,g_k\cdot v_{\tau^{-1}(k)}). \tag*{(35)} (( g 1 , … , g k ) , τ ) ⋅ ( v 1 , … , v k ) = ( g 1 ⋅ v τ − 1 ( 1 ) , … , g k ⋅ v τ − 1 ( k ) ) . (35)
The group for which this action is defined is also called the restricted wreath product of G \mathcal{G} G and S k S_k S k . For more details, see [WAR20].
This type of ’wreath-equivariant-structure’ arises in several settings. One is ’alignment problems’, where our goal is, given a pair of elements ( v 1 , v 2 ) ∈ V k (v_1,v_2)\in\mathcal{V}^k ( v 1 , v 2 ) ∈ V k , k = 2 k=2 k = 2 , to find the group element g ∗ = g ∗ ( v 1 , v 2 ) g^*=g^*(v_1,v_2) g ∗ = g ∗ ( v 1 , v 2 ) which makes v 1 v_1 v 1 ’as similar as possible’ to v 2 v_2 v 2 . This task is equivariant to application of G \mathcal{G} G elements to each coordinate because if g ∗ v 1 ≈ v 2 g^*v_1\approx v_2 g ∗ v 1 ≈ v 2 then g 2 g ∗ g 1 − 1 ( g 1 v 1 ) ≈ g 2 v 2 g_2g^*g_1^{-1}(g_1v_1)\approx g_2v_2 g 2 g ∗ g 1 − 1 ( g 1 v 1 ) ≈ g 2 v 2 . Similarly, this task is equivariant to permuting v 1 v_1 v 1 and v 2 v_2 v 2 , because if g ∗ v 1 ≈ v 2 g^*v_1\approx v_2 g ∗ v 1 ≈ v 2 then ( g ∗ ) − 1 v 2 ≈ v 1 (g^*)^{-1}v_2\approx v_1 ( g ∗ ) − 1 v 2 ≈ v 1 . For a more detailed derivation, see [CW24; NSF+24], which discussed these problems for sets and weight spaces, respectively. Additional examples of ’wreath-equivariant-problems’ are the anomaly detection problem discussed in the experimental section, and problems with hierarchical structures as discussed in [WAR20].
Wreath-equivariant layers. Our aim is to characterize all G ≀ S k \mathcal{G} \wr S_k G ≀ S k equivariant mappings from V k \mathcal{V}^k V k to itself. We note that any linear G \mathcal{G} G -equivariant mapping L ^ : V → V \hat{L}:\mathcal{V}\to\mathcal{V} L ^ : V → V induces a ’Siamese’ G ≀ S k \mathcal{G}\wr S_k G ≀ S k equivariant mapping defined by
L ( v 1 , … , v k ) = ( L ^ ( v 1 ) , … , L ^ ( v k ) ) . L(v_1,\ldots,v_k)=(\hat{L}(v_1),\ldots,\hat{L}(v_k)). L ( v 1 , … , v k ) = ( L ^ ( v 1 ) , … , L ^ ( v k )) .
The interesting question is how many additional mappings are present. This problem was previously studied in [MLCF20; WAR20] when G \mathcal{G} G is a finite group acting on R n \mathbb{R}^n R n transitively by permutations (this means that the action of G \mathcal{G} G on [ n ] [n] [ n ] has a single orbit). In this setting, the equivariant mappings are composed of the Siamese mappings and a single additional non-Siamese mapping. However, the transitivity assumption does not hold in many examples of interest, such as the graph and weight space examples discussed in this paper. In our analysis, we will relax the transitivity assumption, and allow G \mathcal{G} G to be a general finite group. In some cases, this will lead to a substantial number of non-Siamese networks.
To characterize G ≀ S k \mathcal{G}\wr S_k G ≀ S k equivariant functions, we first aim to characterize all invariant irreducible sub-spaces of V k \mathcal{V}^k V k , assuming we know all irreducible sub-spaces of V \mathcal{V} V . An important role will be played by trivial representations : representations S \mathsf{S} S of G \mathcal{G} G such that g v = v gv=v g v = v for all g ∈ G g\in\mathcal{G} g ∈ G and v ∈ S v\in\mathsf{S} v ∈ S .
Theorem 4.5.1. Let k ≥ 2 k\ge2 k ≥ 2 , and let V \mathcal{V} V be a real representation of G \mathcal{G} G , with irreducible decomposition
V = ( ⨁ i = 1 s S i ) ⊕ ( ⨁ j = 1 t V j ) (36) \mathcal{V}=\left(\bigoplus_{i=1}^{s}\mathsf{S}_i\right)\oplus\left(\bigoplus_{j=1}^{t}\mathcal{V}_j\right) \tag*{(36)} V = ( i = 1 ⨁ s S i ) ⊕ ( j = 1 ⨁ t V j ) (36) where S i \mathsf{S}_i S i are trivial representations and V j \mathcal{V}_j V j are non-trivial irreducible representations. Then an irreducible decomposition for V k \mathcal{V}^k V k with respect to the action of G ≀ S k \mathcal{G}\wr S_k G ≀ S k is given by
V k = ( ⨁ i = 1 s S i , 0 k ) ⊕ ( ⨁ i = 1 s S i , 1 k ) ⊕ ( ⨁ j = 1 t V j k ) , \mathcal{V}^k=\left(\bigoplus_{i=1}^{s}\mathsf{S}_{i,0}^k\right)\oplus\left(\bigoplus_{i=1}^{s}\mathsf{S}_{i,1}^k\right)\oplus\left(\bigoplus_{j=1}^{t}\mathcal{V}_j^k\right), V k = ( i = 1 ⨁ s S i , 0 k ) ⊕ ( i = 1 ⨁ s S i , 1 k ) ⊕ ( j = 1 ⨁ t V j k ) , where
S i , 0 k = { ( s 1 , … , s k ) ∈ S i k : ∑ ℓ = 1 k s ℓ = 0 } , S i , 1 k = { ( s , … , s ) ∈ S i k } . \mathsf{S}_{i,0}^k=\left\{(s_1,\ldots,s_k)\in\mathsf{S}_i^k:\sum_{\ell=1}^{k}s_\ell=0\right\},\qquad
\mathsf{S}_{i,1}^k=\left\{(s,\ldots,s)\in\mathsf{S}_i^k\right\}. S i , 0 k = { ( s 1 , … , s k ) ∈ S i k : ℓ = 1 ∑ k s ℓ = 0 } , S i , 1 k = { ( s , … , s ) ∈ S i k } . Proof. The fact that V i \mathcal{V}_i V i is G \mathcal{G} G invariant implies easily that V i k \mathcal{V}_i^k V i k is G ≀ S k \mathcal{G}\wr S_k G ≀ S k also invariant. In the appendix E.1, we will show that if the action of G \mathcal{G} G on V \mathcal{V} V is not trivial then V k \mathcal{V}^k V k is irreducible representation of G ≀ S k \mathcal{G}\wr S_k G ≀ S k . In contrast, for the spaces S i k \mathsf{S}_i^k S i k the action of G \mathcal{G} G is trivial, so this representation can be identified with the standard representation R k \mathbb{R}^k R k of S k S_k S k . The decomposition to S i , 0 k \mathsf{S}_{i,0}^k S i , 0 k and S i , 1 k \mathsf{S}_{i,1}^k S i , 1 k then follows from the ’DeepSets decomposition’ discussed in Section 4.4.1.
To count the number of linear equivariant mappings L : V k → V k L:\mathcal{V}^{k}\to\mathcal{V}^{k} L : V k → V k , we note that
S i , 0 k ≅ S j , 0 k , S i , 1 k ≅ S j , 1 k , S i , 0 k ≇ S i , 1 k , ∀ 1 ≤ i < j ≤ s S_{i,0}^{k}\cong S_{j,0}^{k},\quad S_{i,1}^{k}\cong S_{j,1}^{k},\quad S_{i,0}^{k}\not\cong S_{i,1}^{k},\quad\forall1\leq i<j\leq s S i , 0 k ≅ S j , 0 k , S i , 1 k ≅ S j , 1 k , S i , 0 k ≅ S i , 1 k , ∀1 ≤ i < j ≤ s
Thus, the dimension of the space of linear equivariant mappings from the “trivial part” of V k \mathcal{V}^{k} V k , that is,
( ⨁ i = 1 s S i , 0 k ) ⊕ ( ⨁ i = 1 s S i , 1 k ) , \left(\bigoplus_{i=1}^{s}S_{i,0}^{k}\right)\oplus\left(\bigoplus_{i=1}^{s}S_{i,1}^{k}\right), ( i = 1 ⨁ s S i , 0 k ) ⊕ ( i = 1 ⨁ s S i , 1 k ) ,
to itself is 2 s 2 2s^{2} 2 s 2 , while the number of corresponding Siamese mappings, induced by linear equivariant mappings from the “trivial part” of V \mathcal{V} V to itself, is s 2 s^{2} s 2 . As a result, we obtain s 2 s^{2} s 2 additional non-Siamese G ∘ S k \mathcal{G}\circ S_{k} G ∘ S k -equivariant basis maps.
Examples. We found that the number of non-Siamese layers is s 2 s^{2} s 2 , where s s s is the number of trivial representations in V \mathcal{V} V . Let us consider the implications for the three examples we have discussed earlier:
Deep-sets. In the DeepSets setting where V = R n \mathcal{V}=\mathbb{R}^{n} V = R n , G = S n \mathcal{G}=S_{n} G = S n , there is a single trivial representation of constant vectors, and therefore in this setting there is a unique non-Siamese layer for V k \mathcal{V}^{k} V k . Indeed, this is the case for any group acting transitively by permutations on R n \mathbb{R}^{n} R n , and thus we obtain the results for transitive actions from [MLCF20; WAR20].
Graphs. In the graph setting where V = R n × n \mathcal{V}=\mathbb{R}^{n\times n} V = R n × n , G = S n \mathcal{G}=S_{n} G = S n there are s = 2 s=2 s = 2 trivial representations (see equation (24) ), and therefore s 2 = 4 s^{2}=4 s 2 = 4 non-Siamese layers.
Weight Spaces. In the weight space examples the number s s s of trivial representations is rather large, α = d 0 + 2 d M + 2 M − 3 \alpha=d_{0}+2d_{M}+2M-3 α = d 0 + 2 d M + 2 M − 3 from equation (29) , and there are α 2 \alpha^{2} α 2 non-Siamese mappings.
In the next theorem we give an explicit characterization of the non-Siamese layers of V k \mathcal{V}^{k} V k . For this characterization, we note that the direct sum of all trivial representations of V \mathcal{V} V (as in equation (36) ) is the vector space V f i x e d = { v ∈ V ∣ g v = v , ∀ g ∈ G } \mathcal{V}_{fixed}=\{v\in\mathcal{V}\mid gv=v,\ \forall g\in\mathcal{G}\} V f i x e d = { v ∈ V ∣ g v = v , ∀ g ∈ G } . Alternatively, every basis e 1 , … , e s e_{1},\ldots,e_{s} e 1 , … , e s of V f i x e d \mathcal{V}_{fixed} V f i x e d defines a decomposition of V f i x e d \mathcal{V}_{fixed} V f i x e d into one-dimensional irreducible invariant sub-spaces S i = { c e i ∣ c ∈ R } S_{i}=\{ce_{i}\mid c\in\mathbb{R}\} S i = { c e i ∣ c ∈ R } . Accordingly, our characterization of non-Siamese layers is based on such bases:
Theorem 4.5.2. Let V \mathcal{V} V be a real representation of a finite group G \mathcal{G} G . and let e 1 , … , e s e_{1},\ldots,e_{s} e 1 , … , e s be a basis for the subspace V f i x e d \mathcal{V}_{fixed} V f i x e d . Let ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ be a G \mathcal{G} G invariant inner product on V \mathcal{V} V . Then every linear equivariant map L : V k → V k L:\mathcal{V}^{k}\to\mathcal{V}^{k} L : V k → V k is of the form
L ( v 1 , … , v k ) = ∑ i , j = 1 s a i j ( ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j , … , ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j ) + ( L ^ ( v 1 ) , … , L ^ ( v k ) ) (37) L(v_{1},\ldots,v_{k})=\sum_{i,j=1}^{s}a_{ij}\left(\sum_{\ell=1}^{k}\langle v_{\ell},e_{i}\rangle e_{j},\ldots,\sum_{\ell=1}^{k}\langle v_{\ell},e_{i}\rangle e_{j}\right)+\left(\widehat{L}(v_{1}),\ldots,\widehat{L}(v_{k})\right) \tag*{(37)} L ( v 1 , … , v k ) = i , j = 1 ∑ s a ij ( ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j , … , ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j ) + ( L ( v 1 ) , … , L ( v k ) ) (37)
where L ^ : V → V \hat{L}: \mathcal{V} \to\mathcal{V} L ^ : V → V is a linear equivariant map, and a i j a_{ij} a ij are real numbers. Conversely, every linear mapping of the form defined in equation 4.9 is equivariant.
The theorem is proven in Appendix E. We note that an invariant inner product on V \mathcal{V} V is an inner product satisfying ⟨ g v , g u ⟩ = ⟨ v , u ⟩ \langle gv, gu\rangle= \langle v, u\rangle ⟨ g v , g u ⟩ = ⟨ v , u ⟩ for all g ∈ G g \in\mathcal{G} g ∈ G and u , v ∈ V u, v \in\mathcal{V} u , v ∈ V . When G \mathcal{G} G is finite, an invariant inner product always exists: it can be obtained by starting from an arbitrary inner product and then averaging over the group [FH13]. In the examples we consider in this paper, the group G \mathcal{G} G acts by permutations. In this case, the standard ℓ 2 \ell_2 ℓ 2 inner product is invariant. Implementing the non-Siamese layers defined in equation 4.5.2 only requires finding a basis for V f i x e d \mathcal{V}_{fixed} V f i x e d . In particular, if V \mathcal{V} V is one of the spaces discussed previously, the basis e i e_i e i is just a choice of an element from each of the trivial spaces S i S_i S i in the decomposition of V \mathcal{V} V .
Beyond S k S_k S k . So far, we have considered the action of G ≀ S k \mathcal{G} \wr S_k G ≀ S k where G \mathcal{G} G is finite. We can also generalize these results to the setting G ≀ H \mathcal{G} \wr H G ≀ H , where H H H is a subgroup of S k S_k S k which acts transitively on { 1 , … , k } \{1,\ldots,k\} { 1 , … , k } . By doing this, we are generalizing the results in [WAR20], which assumes that both G \mathcal{G} G and H H H act transitively.
The full generalization is described in Appendix E.1. The general idea is this: Since G \mathcal{G} G acts trivially on V f i x e d \mathcal{V}_{fixed} V f i x e d , the action of G ≀ H \mathcal{G} \wr H G ≀ H on V f i x e d k \mathcal{V}_{fixed}^{k} V f i x e d k can be identified with the action of H H H on V f i x e d k \mathcal{V}_{fixed}^{k} V f i x e d k . In turn, V f i x e d k \mathcal{V}_{fixed}^{k} V f i x e d k is a direct sum of s s s different copies of R k \mathbb{R}^{k} R k . Accordingly, the number of H H H equivariant maps from V f i x e d k \mathcal{V}_{fixed}^{k} V f i x e d k to itself is s 2 ⋅ h s^2 \cdot h s 2 ⋅ h , where h h h is the number of H H H equivariant maps from R k \mathbb{R}^{k} R k to itself. One of these h h h maps is the identity map, which is Siamese; therefore, the total number of non-Siamese equivariant maps is ( h − 1 ) ⋅ s 2 (h-1)\cdot s^2 ( h − 1 ) ⋅ s 2 . In the case where H = S k H=S_k H = S k , we have that h = 2 h=2 h = 2 , and therefore, in our analysis above, we have found s 2 s^2 s 2 non-Siamese maps.
Experiments
In this section, we consider problems with a wreath-equivariant structure. We compare networks implementing the complete basis of equivariant layers that we have found, with networks that only use Siamese layers or combine a partial list of non-Siamese layers suggested in previous works. Implementation details of all experiments are described in section A.
Graph Anomaly Detection. We begin with a synthetic wreath-equivariant task in which Siamese networks will fail by design: we consider a graph anomaly detection problem, where the input is k k k graphs with n n n nodes, ( G 1 , … , G k ) (\mathcal{G}_1,\ldots,\mathcal{G}_k) ( G 1 , … , G k ) , where most graphs are similar, and one is an ”anomaly.” The output is a k k k dimensional probability vector, where the i i i -th entry denotes the probability that G i \mathcal{G}_i G i is an anomaly. This task is G ≀ S k \mathcal{G}\wr S_k G ≀ S k equivariant (where G = S n \mathcal{G}=S_n G = S n ), where the action on the input space is as in equation (35) , and the action on the output space is just permuting the entries of the probability vector.
We generate data for this problem as follows: We randomly generate two graphs G \mathcal{G} G and G ^ \widehat{\mathcal{G}} G by sampling each entry of their adjacency matrices independently from a Bernoulli distribution. We take k − 1 k-1 k − 1 copies of the graphs ( G 1 , … , G k − 1 ) (\mathcal{G}_1,\ldots,\mathcal{G}_{k-1}) ( G 1 , … , G k − 1 ) to be permuted copies of G \mathcal{G} G , and one of them to be G ^ \widehat{\mathcal{G}} G and insert it in a random location. We add Gaussian noise with standard deviation η \eta η to the permuted copies of G \mathcal{G} G and to the anomalous graph G ^ \widehat{\mathcal{G}} G .
We consider equivariant models for this task, composed of several linear wreath-equivariant layers and point-wise ReLU activations. The final layer is a point-wise summation of each of the k k k graphs to obtain a final vector in R k \mathbb{R}^{k} R k , followed by a softmax layer to obtain a probability vector. For the linear wreath-equivariant layers, we consider several alternatives: Siamese layers only, adding the single additional non-Siamese layer suggested in [WAR20; MLCF20] denoted by DSS, and the full model we suggest, which has four non-Siamese layers (we name our model SchurNet). The Siamese layers are implemented using the decomposition computed in Subsection 4.4.2.
As shown in Table 4.1 , the Siamese model achieves only 48% accuracy without noise, and its performance further decreases to 28% when noise is added. In contrast, DSS achieves 97.5% and 92.0% accuracy, while SchurNet achieves perfect accuracy in the noiseless setting and 97.0% accuracy under noise. This is to be expected since Siamese features can be useful to differentiate between G \mathcal{G} G and G ^ \widehat{\mathcal{G}} G , but not to determine how many times each one of them occurs in a k k k -tuple. In contrast, networks with non-Siamese layers attain significantly better performance, whereas our method, which contains the maximal number of non-Siamese layers, attains the best performance.
Table 4.1. Anomaly detection accuracy under different noise levels.
Wasserstein Distance Computation We next consider the task of learning the Wasserstein distance, as discussed in [AD24; CW24; HKD+24]. The Wasserstein distance between two unordered multisets of vectors in R d \mathbb{R}^{d} R d , denoted by { x 1 , … , x n } \{x_1,\ldots,x_n\} { x 1 , … , x n } and { y 1 , … , y n } \{y_1,\ldots,y_n\} { y 1 , … , y n } , is defined to be the minimal distance between the sets, under the optimal permutation giving the best alignment between them (as in the alignment problem discussed in the beginning of § 7.6 ). Computing Wasserstein distances can be computationally intensive, so this task aims to devise a neural network to learn the distance instead. Learning Wasserstein distances is a S n ≀ S 2 \mathcal{S}_n \wr\mathcal{S}_2 S n ≀ S 2 invariant task. In [CW24], a Siamese approach named NProductNet is suggested to address this problem. We compare SchurNet to [CW24] to demonstrate that adding our non-siamese layers enhances the performance of standard Siamese layers. We note that [AD24] achieved state-ofthe-art results for this task with a very different method, which does not follow the conventional paradigm of a stack of linear layers and non-linear activation functions. Therefore, we have no direct way of adding our non-siamese layers to their method for comparison. In our experiments, we add the additional d 2 d^2 d 2 non-Siamese layers to their implementation and compare performance on the datasets addressed in their paper. The results, reported in Table 4.2 , show that adding non-Siamese layers improves performance in most cases. We note that each dataset has two settings: the top row corresponds to test sets whose cardinalities are within the range seen during training, while the bottom row evaluates generalization to unseen set sizes.
Table 4.2. Comparison of SchurNet and NProductNet. The best result in each setting is shown in bold, and the second-best is underlined.
Weight Space Alignment. We consider the alignment task discussed above in the setting where v 1 , v 2 v_1,v_2 v 1 , v 2 are elements in weight spaces, and the task is to find the group element that optimally aligns v 1 v_1 v 1 and v 2 v_2 v 2 . One interesting application of this problem is model merging: in [AHS], it was shown that linear interpolation of v 1 , v 2 v_1,v_2 v 1 , v 2 gives a new network whose performance is considerably worse, but interpolation after alignment leads to much better results.
The weight space alignment problem was considered in [AHS; TCD+20; PMD+23]. Recently, [NSF+24] outperformed these methods using a learning approach based on the DWS layers from [NSA+23], applied to v 1 , v 2 v_1,v_2 v 1 , v 2 in a Siamese fashion. This experiment aims to check whether adding non-Siamese layers improves their results. We take only part of the described layers in Theorem 4.5.2 layers: the mappings from each weight/bias space to itself, excluding, for example, non-Siamese weight to other weight or weight to bias. This added fewer parameters and generalized better than taking all mappings.
We ran two sets of experiments from [NSF+24]: one on MLPs trained on MNIST [LCB98] and one on MLPs trained on the CIFAR10 [KH+09] dataset. We trained both models for 100 epochs and reported the test accuracy and the reconstruction loss. We ran several hyper-parameter configurations for both methods to ensure a fair comparison and reported the results with the best reconstruction loss. As we can see, our model with non-Siamese layers outperforms the Siamese network from [NSF+24] in all settings.
Table 4.3. Comparison of SchurNet and Siamese models on MNIST and CIFAR10.
Conclusion. In this paper, we revisited the idea of using irreducible representations instead of parameter-sharing to characterize equivariant linear layers for representations of permutations and related groups. Using this approach, we obtained alternative derivations for the characterizations of equivariant layers from DeepSets and 2-IGN, and a significantly simplified derivation of deep weight spaces equivariant layers. We have also obtained a previously unknown characterization for the linear equivariant layers of wreath products G ≀ S k \mathcal{G} \wr S_k G ≀ S k , and showed the benefits of using the full characterization for several wreath-equivariant tasks. In general, looking forward to other yet unknown applications, what could be the benefit of using the irreducible approach? One answer is reduced book-keeping, as exemplified in the deep weight space example. In general, the number of equivariant layers can be, in the worst case, quadratic in the number of irreducibles. When this happens, the irreducible approach is expected to lead to simpler characterizations that are easier to implement. Additionally, networks like k k k -IGN [MBSL18], which are based on tensor representations, use intermediate features in n k n^k n k , and using irreducibles could lead to new equivariant models with intermediate irreducible features of lower dimensions. Finding the irreducible decompositions of k k k -IGN when k > 2 k > 2 k > 2 and the potential applications of this decomposition is an interesting avenue for future work.
Appendix
Implementation details
In this section, we detail the hyper-parameters used through training. In all our models we used Adam or AdamW optimizers.
Graph Anomaly Detection In this set of experiments, we ran each model with 27 different hyper-parameter configurations and reported the best result for each model. We searched over different learning rates, weight decays, and learning-rate decay factors. In our model, we used four layers, each with a hidden dimension of 256 and ReLU activation.
Deep weight space alignment In this section we took the model of [NSF+24] and added our non-Siamese common layers. We searched for each model and dataset over four hyper-parameters and for each reported the best performance. We trained for 100 epochs and reported the reconstruction loss.
Wasserstein Distance computation In this set we took the model of [CW24] and added the non-Siamese layers. We trained for 200 epochs and reported the relative absolute mean error. In most of the experiments we used a learning rate of 1 e − 4 1\mathrm{e}^{-4} 1 e − 4 and weight decay of 1 e − 5 1\mathrm{e}^{-5} 1 e − 5 .
Absolutely Irreducible Real Representations
Here we give some more details on the concept of absolutely irreducible real representations.
As discussed in the main text, a real irreducible representation V \mathcal{V} V is called absolutely irreducible if its complexification is irreducible over C \mathbb{C} C . The complexification is denoted by c V c\mathcal{V} c V . It can be defined by choosing a basis v 1 , … , v n v_1,\ldots,v_n v 1 , … , v n of V \mathcal{V} V and then defining
c V = { ∑ i = 1 n c i v i , c i ∈ C } . c\mathcal{V} = \left\{ \sum_{i=1}^{n} c_i v_i,\quad c_i \in\mathbb{C} \right\}. c V = { i = 1 ∑ n c i v i , c i ∈ C } .
Let T : V → V T:\mathcal{V}\to\mathcal{V} T : V → V be an equivariant non-zero mapping, and assume V \mathcal{V} V is absolutely irreducible. Then T v = λ v T v=\lambda v T v = λ v for some real λ \lambda λ . This is because T T T can be linearly extended to a non-zero linear mapping T : c V → c V T:c\mathcal{V}\to c\mathcal{V} T : c V → c V . Similarly, the group action of G \mathcal{G} G which is linear can also be extended to c V c\mathcal{V} c V . The extension of T T T to c V c\mathcal{V} c V will be equivariant, and therefore, since c V c\mathcal{V} c V is irreducible by assumption, Schur’s Lemma for complex representations implies that T = λ I T=\lambda I T = λ I for λ ∈ C \lambda\in\mathbb{C} λ ∈ C . Since the original map T T T maps V \mathcal{V} V into itself, for every non-zero v ∈ V v\in\mathcal{V} v ∈ V we have T v = λ v ∈ V Tv=\lambda v\in\mathcal{V} T v = λ v ∈ V , which implies that λ ∈ R \lambda\in\mathbb{R} λ ∈ R .
Real irreducible permutation representations are absolutely irreducible. Classical representation theory results (see e.g., [FH13]) characterize all complex irreducible representations of the permutation group, up to isomorphism. In this process, they show that all these representations can be defined over the rational numbers. This means that, for any complex irreducible representation U \mathcal{U} U of the permutation group S n S_n S n , there exists a basis in which for every g ∈ S n g\in S_n g ∈ S n , the matrix representing the linear mapping defined by g g g will be rational.
As explained, e.g., in [Web], this fact implies that every irreducible s s s dimensional real representation V \mathcal{V} V is absolutely irreducible. Indeed, if c V c\mathcal{V} c V were not irreducible over C \mathbb{C} C , then c V = V 1 ⊕ V 2 cV = V_1 \oplus V_2 c V = V 1 ⊕ V 2 where V 1 V_1 V 1 is an invariant (complex) irreducible subspace, and the dimension t t t of V 1 V_1 V 1 satisfies 0 < t < s 0 < t < s 0 < t < s . In an appropriate basis u 1 , … , u t u_1,\ldots,u_t u 1 , … , u t of V 1 V_1 V 1 we will have that the matrix representing each g ∈ S n g \in S_n g ∈ S n is rational. We can write each u j u_j u j as
u j = ∑ k = 1 s ( a j k + i b j k ) v k . u_j = \sum_{k=1}^{s}(a_{jk} + ib_{jk})v_k. u j = k = 1 ∑ s ( a j k + i b j k ) v k .
Consider the real subspaces
A = span R { ∑ k = 1 s a j k v k : j = 1 , … , t } , B = span R { ∑ k = 1 s b j k v k : j = 1 , … , t } . A = \operatorname{span}_{\mathbb{R}}\left\{\sum_{k=1}^{s} a_{jk}v_k : j = 1,\ldots,t\right\}, \qquad B = \operatorname{span}_{\mathbb{R}}\left\{\sum_{k=1}^{s} b_{jk}v_k : j = 1,\ldots,t\right\}. A = span R { k = 1 ∑ s a j k v k : j = 1 , … , t } , B = span R { k = 1 ∑ s b j k v k : j = 1 , … , t } .
Since the matrices representing the action of G G G on V 1 V_1 V 1 in the basis u 1 , … , u t u_1,\ldots,u_t u 1 , … , u t have real entries, both A A A and B B B are G G G -invariant real subspaces of V V V . Moreover, at least one of them is non-zero, while both have dimension at most t < s t < s t < s . Thus V V V contains a non-zero proper invariant real subspace, contradicting its irreducibility.
graph networks irreducible decomposition
In the main text, we claim that, for n ≥ 4 n \ge4 n ≥ 4 , the representation R n × n \mathbb{R}^{n \times n} R n × n of the permutation group S n S_n S n can be decomposed into a direct sum of seven irreducible representations:
V 0 = { a ⋅ I n ∣ a ∈ R } V_0 = \{a \cdot I_n \mid a \in\mathbb{R}\} V 0 = { a ⋅ I n ∣ a ∈ R }
V 1 = { a ⋅ ( 1 n × n − I n ) ∣ a ∈ R } V_1 = \{a \cdot(1_{n \times n} - I_n) \mid a \in\mathbb{R}\} V 1 = { a ⋅ ( 1 n × n − I n ) ∣ a ∈ R }
V 2 = { Diag ( α 1 , … , α n ) ∣ ∑ i = 1 n α i = 0 } V_2 = \{\operatorname{Diag}(\alpha_1,\ldots,\alpha_n) \mid\sum_{i=1}^{n}\alpha_i = 0\} V 2 = { Diag ( α 1 , … , α n ) ∣ i = 1 ∑ n α i = 0 }
V 3 = { r 1 n T ∣ ∑ i = 1 n r i = 0 } V_3 = \{r1_n^T \mid\sum_{i=1}^{n}r_i = 0\} V 3 = { r 1 n T ∣ i = 1 ∑ n r i = 0 }
V 4 = { 1 n c T ∣ ∑ i = 1 n c i = 0 } V_4 = \{1_n c^T \mid\sum_{i=1}^{n}c_i = 0\} V 4 = { 1 n c T ∣ i = 1 ∑ n c i = 0 }
V 5 = { A ∣ A = − A T , A 1 n = 0 } V_5 = \{A \mid A = -A^T,\ A1_n = 0\} V 5 = { A ∣ A = − A T , A 1 n = 0 }
V 6 = { A ∣ A = A T , A 1 n = 0 , A i i = 0 , ∀ i = 1 , … , n } . V_6 = \{A \mid A = A^T,\ A1_n = 0,\ A_{ii} = 0,\ \forall i = 1,\ldots,n\}. V 6 = { A ∣ A = A T , A 1 n = 0 , A ii = 0 , ∀ i = 1 , … , n } .
We begin by giving an algorithm describing how every n n n by n n n matrix can be decomposed into a sum of matrices from these spaces, with a computational complexity of O ( n 2 ) O(n^2) O ( n 2 ) . This is necessary for a 2-IGN implementation which is based on irreducibles, and is also a first step towards a full proof of Theorem 4.4.1. This full proof will be presented after the algorithm.
Decomposition algorithm
We will give an algorithm to decompose a matrix A ∈ R n × n A \in\mathbb{R}^{n \times n} A ∈ R n × n into a sum of matrices from these seven sub-spaces. We assume n ≥ 4 n \ge4 n ≥ 4 (actually, when n = 3 n = 3 n = 3 the algorithm works as well, but in this case V 6 \mathcal{V}_6 V 6 is zero dimensional as discussed in the main text).
The first step of the algorithm is to write
A = B + ( 0 a ⋯ a a 0 ⋯ a ⋮ ⋮ ⋱ ⋮ a a ⋯ 0 ) + ( b b ⋮ b ) A = B +
\begin{pmatrix}
0 & a & \cdots& a \\
a & 0 & \cdots& a \\
\vdots& \vdots& \ddots& \vdots\\
a & a & \cdots& 0
\end{pmatrix}
+
\begin{pmatrix}
b \\
b \\
\vdots\\
b
\end{pmatrix} A = B + 0 a ⋮ a a 0 ⋮ a ⋯ ⋯ ⋱ ⋯ a a ⋮ 0 + b b ⋮ b
where the two matrices from the right are in V 0 \mathcal{V}_0 V 0 and V 1 \mathcal{V}_1 V 1 , and a a a and b b b are the average of off-diagonal and diagonal elements of A A A , respectively, so that the diagonal and off-diagonal elements of B B B both sum to zero. It remains to find a decomposition for B B B . The crucial part of this is the following lemma
Lemma C.1. Let n > 2 n > 2 n > 2 be a natural number, and let B ∈ R n × n B \in\mathbb{R}^{n \times n} B ∈ R n × n be a matrix whose diagonal elements sum to zero, and off-diagonal elements sum to zero. Then there exists a matrix C C C in
V ^ = { C ∈ R n × n ∣ C 1 n = 0 , C T 1 n = 0 and C i i = 0 , ∀ i = 1 , … , n } (38) \widehat{\mathcal{V}} = \{C \in\mathbb{R}^{n \times n} \mid C1_n = 0,\ C^{T}1_n = 0 \text{ and } C_{ii} = 0,\ \forall i = 1,\ldots,n\} \tag*{(38)} V = { C ∈ R n × n ∣ C 1 n = 0 , C T 1 n = 0 and C ii = 0 , ∀ i = 1 , … , n } (38) and vectors r , c , d ∈ R n r,c,d \in\mathbb{R}^{n} r , c , d ∈ R n which all sum to zero, such that
C = B + r 1 n T + 1 n c T + diag ( d ) . (39) C = B + r1_n^{T} + 1_nc^{T} + \operatorname{diag}(d). \tag*{(39)} C = B + r 1 n T + 1 n c T + diag ( d ) . (39) Moreover, this decomposition of B B B is unique.
Stated differently, the matrix B B B will be a linear combination of matrices r 1 n T r1_n^{T} r 1 n T , 1 n c T 1_nc^{T} 1 n c T , and diag ( d ) \operatorname{diag}(d) diag ( d ) , which are in V 3 \mathcal{V}_3 V 3 , V 4 \mathcal{V}_4 V 4 , and V 2 \mathcal{V}_2 V 2 , respectively, and the matrix C C C , which is in V ^ \widehat{\mathcal{V}} V . Once we prove the lemma, we will conclude by showing that a matrix in V ^ \widehat{\mathcal{V}} V can be written as a sum of matrices in V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 , which will conclude the argument.
Proof of the Lemma. Let us denote the sum of the i i i -th row and column of B B B by r i B r_i^B r i B and c i B c_i^B c i B respectively. Denote d i B = B i i d_i^B = B_{ii} d i B = B ii . Note that the vectors r B r^B r B , c B c^B c B , and d B d^B d B all sum to zero by our assumption on B B B .
We need to show there exist vectors r , d , c ∈ R n r,d,c \in\mathbb{R}^{n} r , d , c ∈ R n satisfying equation 4.11 for an appropriate C ∈ V ^ C \in\widehat{\mathcal{V}} C ∈ V . This will occur if and only if the following equations are satisfied
∑ i = 1 n r i = 0 \sum_{i=1}^{n} r_i = 0 i = 1 ∑ n r i = 0
∑ i = 1 n c i = 0 \sum_{i=1}^{n} c_i = 0 i = 1 ∑ n c i = 0
∑ i = 1 n d i = 0 \sum_{i=1}^{n} d_i = 0 i = 1 ∑ n d i = 0
r i + c i + d i = − d i B , ∀ i = 1 , … , n r_i + c_i + d_i = -d_i^{B}, \quad\forall i = 1,\ldots,n r i + c i + d i = − d i B , ∀ i = 1 , … , n
n r i + ∑ j = 1 n c j + d i = − r i B , ∀ i = 1 , … , n nr_i + \sum_{j=1}^{n} c_j + d_i = -r_i^{B}, \quad\forall i = 1,\ldots,n n r i + j = 1 ∑ n c j + d i = − r i B , ∀ i = 1 , … , n
n c i + ∑ j = 1 n r j + d i = − c i B , ∀ i = 1 , … , n . nc_i + \sum_{j=1}^{n} r_j + d_i = -c_i^{B}, \quad\forall i = 1,\ldots,n. n c i + j = 1 ∑ n r j + d i = − c i B , ∀ i = 1 , … , n .
Here the first three constraints are the requirements that the vectors r r r , c c c , d d d all sum to zero, and the last three constraints (each of which actually represents n n n constraints) follow from our requirement that the diagonal elements of C C C will be zero and that the rows and columns of C C C sum to zero.
Next, we note that since the vectors r r r , c c c , d d d , r B r^{B} r B , c B c^{B} c B , d B d^{B} d B all sum to zero, if the last three equations above are satisfied for i = 1 , … , n − 1 i = 1,\ldots,n-1 i = 1 , … , n − 1 , they will automatically be satisfied for i = n i = n i = n . That is, these equations are equivalent to
∑ i = 1 n r i = 0 \sum_{i=1}^{n} r_i = 0 i = 1 ∑ n r i = 0
∑ i = 1 n c i = 0 \sum_{i=1}^{n} c_i = 0 i = 1 ∑ n c i = 0
∑ i = 1 n d i = 0 \sum_{i=1}^{n} d_i = 0 i = 1 ∑ n d i = 0
r i + c i + d i = − d i B , ∀ i = 1 , … , n − 1 r_i + c_i + d_i = -d_i^{B}, \quad\forall i = 1,\ldots,n-1 r i + c i + d i = − d i B , ∀ i = 1 , … , n − 1
n r i + d i = − r i B , ∀ i = 1 , … , n − 1 nr_i + d_i = -r_i^{B}, \quad\forall i = 1,\ldots,n-1 n r i + d i = − r i B , ∀ i = 1 , … , n − 1
n c i + d i = − c i B , ∀ i = 1 , … , n − 1 nc_i + d_i = -c_i^{B}, \quad\forall i = 1,\ldots,n-1 n c i + d i = − c i B , ∀ i = 1 , … , n − 1
This gives us a 3 n 3n 3 n linear equations system in 3 n 3n 3 n variables. We will show this equation has a unique solution and compute it explicitly. First, we eliminate the variables r n r_n r n , c n c_n c n and d 1 , … , d n d_1,\ldots,d_n d 1 , … , d n by setting
d i = − d i B − r i − c i , i = 1 , … , n − 1 (40) d_i = -d_i^{B} - r_i - c_i, \quad i = 1,\ldots,n-1 \tag*{(40)} d i = − d i B − r i − c i , i = 1 , … , n − 1 (40)
and
r n = − ∑ j = 1 n − 1 r j , c n = − ∑ j = 1 n − 1 c j , d n = − ∑ j = 1 n − 1 d j (41) r_n=-\sum_{j=1}^{n-1}r_j,\qquad c_n=-\sum_{j=1}^{n-1}c_j,\qquad d_n=-\sum_{j=1}^{n-1}d_j \tag*{(41)} r n = − j = 1 ∑ n − 1 r j , c n = − j = 1 ∑ n − 1 c j , d n = − j = 1 ∑ n − 1 d j (41)
This ensures that the first four equations above are satisfied, and we are left with the task of choosing r 1 , … , r n − 1 , c 1 , … , c n − 1 r_1,\ldots,r_{n-1},c_1,\ldots,c_{n-1} r 1 , … , r n − 1 , c 1 , … , c n − 1 satisfying the last two equations (which are actually 2 n − 2 2n-2 2 n − 2 equations). By replacing the eliminated variables d i d_i d i we obtain the equations
( n − 1 ) r i − c i = − r i B + d i B , ∀ i = 1 , … , n − 1 ( n − 1 ) c i − r i = − c i B + d i B , ∀ i = 1 , … , n − 1 \begin{aligned}
(n-1)r_i-c_i&=-r_i^{B}+d_i^{B}, &&\forall i=1,\ldots,n-1\\
(n-1)c_i-r_i&=-c_i^{B}+d_i^{B}, &&\forall i=1,\ldots,n-1
\end{aligned} ( n − 1 ) r i − c i ( n − 1 ) c i − r i = − r i B + d i B , = − c i B + d i B , ∀ i = 1 , … , n − 1 ∀ i = 1 , … , n − 1
For every i i i , there are two equations in r i r_i r i and c i c_i c i . Since we assume n ≥ 4 n\geq4 n ≥ 4 , this equation has a unique solution
( r i c i ) = 1 n 2 − 2 n ( n d i B − ( n − 1 ) r i B − c i B n d i B − ( n − 1 ) c i B − r i B ) \begin{pmatrix}r_i\\c_i\end{pmatrix}
=\frac{1}{n^2-2n}
\begin{pmatrix}
nd_i^{B}-(n-1)r_i^{B}-c_i^{B}\\
nd_i^{B}-(n-1)c_i^{B}-r_i^{B}
\end{pmatrix} ( r i c i ) = n 2 − 2 n 1 ( n d i B − ( n − 1 ) r i B − c i B n d i B − ( n − 1 ) c i B − r i B )
And we can derive
( r i c i ) = 1 n 2 − 2 n ( n − 1 1 1 n − 1 ) ( − r i B + d i B − c i B + d i B ) \begin{pmatrix}r_i\\c_i\end{pmatrix}
=\frac{1}{n^2-2n}
\begin{pmatrix}
n-1 & 1\\
1 & n-1
\end{pmatrix}
\begin{pmatrix}
-r_i^{B}+d_i^{B}\\
-c_i^{B}+d_i^{B}
\end{pmatrix} ( r i c i ) = n 2 − 2 n 1 ( n − 1 1 1 n − 1 ) ( − r i B + d i B − c i B + d i B )
We can then reconstruct d 1 , … , d n − 1 d_1,\ldots,d_{n-1} d 1 , … , d n − 1 from equation 4.12 and c n , r n , d n c_n,r_n,d_n c n , r n , d n from equation 4.13. This concludes the proof.
Given the proof, it remains to decompose a matrix C ∈ V ^ C\in\widehat{\mathcal{V}} C ∈ V into a sum of two matrices in V 5 , V 6 \mathcal{V}_5,\mathcal{V}_6 V 5 , V 6 . This can be simply done by writing
C = 1 2 ( C + C T ) + 1 2 ( C − C T ) C=\frac{1}{2}(C+C^{T})+\frac{1}{2}(C-C^{T}) C = 2 1 ( C + C T ) + 2 1 ( C − C T )
and noting that the matrices 1 2 ( C − C T ) \frac{1}{2}(C-C^{T}) 2 1 ( C − C T ) and 1 2 ( C + C T ) \frac{1}{2}(C+C^{T}) 2 1 ( C + C T ) are in V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 .
isomorphism types
We now restate and prove the theorem on 2-IGN stated in the main text
Theorem 4.4.1. For all n ≥ 4 n\geq4 n ≥ 4 , the space R n × n \mathbb{R}^{n\times n} R n × n can be written as a direct sum of the spaces V 0 , … , V 6 \mathcal{V}_0,\ldots,\mathcal{V}_6 V 0 , … , V 6 . These spaces are invariant and irreducible, and the isomorphism relations between them are given by V 0 ≅ V 1 \mathcal{V}_0\cong\mathcal{V}_1 V 0 ≅ V 1 , V 2 ≅ V 3 ≅ V 4 \mathcal{V}_2\cong\mathcal{V}_3\cong\mathcal{V}_4 V 2 ≅ V 3 ≅ V 4 .
Proof. The decomposition given in Subsection C.1 is unique when n ≥ 4 n\geq4 n ≥ 4 , and thus we see that
R n × n = V 0 ⊕ … ⊕ V 6 . (42) \mathbb{R}^{n\times n}=\mathcal{V}_0\oplus\ldots\oplus\mathcal{V}_6. \tag*{(42)} R n × n = V 0 ⊕ … ⊕ V 6 . (42)
It is also clear that all the spaces V i \mathcal{V}_i V i are invariant, that V 0 , V 1 \mathcal{V}_0,\mathcal{V}_1 V 0 , V 1 are isomorphic to the irreducible trivial space, and V 2 , V 3 , V 4 \mathcal{V}_2,\mathcal{V}_3,\mathcal{V}_4 V 2 , V 3 , V 4 are isomorphic to the irreducible space V ( n ) \mathcal{V}(n) V ( n ) . It remains to show: (a) That V 5 , V 6 \mathcal{V}_5,\mathcal{V}_6 V 5 , V 6 are not zero-dimensional, (b) that they are irreducible, and (c) that both V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 are not isomorphic to any of the other spaces.
We handle (a) by computing the dimension of V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 . We note that a matrix A ∈ V 5 A \in\mathcal{V}_5 A ∈ V 5 can be parameterized by freely choosing all upper diagonal elements A i j A_{ij} A ij with j − i > 1 j-i>1 j − i > 1 . The elements A j i A_{ji} A j i with j − i > 1 j-i>1 j − i > 1 are then determined by the constraint A j i = − A i j A_{ji}=-A_{ij} A j i = − A ij , and then the coordinates A 12 = − A 21 , A 23 = − A 32 , … A_{12}=-A_{21}, A_{23}=-A_{32},\ldots A 12 = − A 21 , A 23 = − A 32 , … are determined recursively by the constraint that the first n − 1 n-1 n − 1 rows of A A A will sum to zero. The obtained matrix A A A satisfies A = − A T A=-A^T A = − A T and its first n − 1 n-1 n − 1 rows sum to zero. It follows from A = − A T A=-A^T A = − A T that the sum of all elements of A A A is zero and hence the last row of A A A must also sum to zero and A A A is indeed in V 5 \mathcal{V}_5 V 5 .
To summarize, since V 5 \mathcal{V}_5 V 5 is parameterized by the upper diagonal elements A i j A_{ij} A ij with j − i > 1 j-i>1 j − i > 1 we deduce that
dim ( V 5 ) = ( n 2 − n ) / 2 − ( n − 1 ) = n 2 − 3 n 2 + 1. \dim(\mathcal{V}_5)=(n^2-n)/2-(n-1)=\frac{n^2-3n}{2}+1. dim ( V 5 ) = ( n 2 − n ) /2 − ( n − 1 ) = 2 n 2 − 3 n + 1.
We compute the dimension of V 6 \mathcal{V}_6 V 6 by noting that V ^ \widehat{\mathcal{V}} V from equation (38) is a direct sum of V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 . The space V ^ \widehat{\mathcal{V}} V can be parameterized by choosing an ( n − 1 ) × ( n − 1 ) (n-1)\times(n-1) ( n − 1 ) × ( n − 1 ) matrix A A A with zero on the diagonal, and whose elements sum to zero. There is then a unique extension of this matrix to an n n n by n n n matrix A ^ \widehat{A} A which is in V ^ \widehat{\mathcal{V}} V and satisfies A ^ i j = A i j \widehat{A}_{ij}=A_{ij} A ij = A ij for all $1\le i,j\le n-1. Accordingly, the dimension of V ^ \widehat{\mathcal{V}} V is ( n − 1 ) 2 − ( n − 1 ) − 1 = n 2 − 3 n + 1 (n-1)^2-(n-1)-1=n^2-3n+1 ( n − 1 ) 2 − ( n − 1 ) − 1 = n 2 − 3 n + 1 and
dim ( V 6 ) = dim ( V ^ ) − dim ( V 5 ) = n 2 − 3 n 2 . \dim(\mathcal{V}_6)=\dim(\widehat{\mathcal{V}})-\dim(\mathcal{V}_5)=\frac{n^2-3n}{2}. dim ( V 6 ) = dim ( V ) − dim ( V 5 ) = 2 n 2 − 3 n .
It follows that when n ≥ 4 n\ge4 n ≥ 4 , both V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 have positive dimension. Moreover, V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 are not isomorphic as they do not have the same dimension. The same argument shows that V 6 \mathcal{V}_6 V 6 is not isomorphic to any of the other spaces V i \mathcal{V}_i V i , for i = 0 , … , 4 i=0,\ldots,4 i = 0 , … , 4 , since they do not have the same dimension for any n n n . In contrast V 5 \mathcal{V}_5 V 5 does have the same dimension as V 2 \mathcal{V}_2 V 2 when n = 4 n=4 n = 4 , as we get dim ( V 5 ) = 3 = dim ( V 2 ) \dim(\mathcal{V}_5)=3=\dim(\mathcal{V}_2) dim ( V 5 ) = 3 = dim ( V 2 ) . Nonetheless, the spaces are still not isomorphic since the action of S n S_n S n on the spaces is different. For example, the subspace of fixed points of the permutation τ = ( 1 , 2 ) \tau=(1,2) τ = ( 1 , 2 ) in the space V 2 ≅ V ( n ) \mathcal{V}_2\cong\mathcal{V}(n) V 2 ≅ V ( n ) is all n = 4 n=4 n = 4 dimensional vectors x x x which sum to zero and satisfy x 1 = x 2 x_1=x_2 x 1 = x 2 . This space is two dimensional. In contrast, one can verify that the subspace of fixed points of τ \tau τ in V 5 \mathcal{V}_5 V 5 is one dimensional. Hence they cannot be isomorphic.
It remains to show that V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 are indeed irreducible. Let V ~ 5 ⊆ V 5 \widetilde{\mathcal{V}}_5\subseteq\mathcal{V}_5 V 5 ⊆ V 5 and V ~ 6 ⊆ V 6 \widetilde{\mathcal{V}}_6\subseteq\mathcal{V}_6 V 6 ⊆ V 6 be non-zero invariant subspaces. We need to show that V ~ j = V j \widetilde{\mathcal{V}}_j=\mathcal{V}_j V j = V j for j = 5 , 6 j=5,6 j = 5 , 6 . To show this, we consider the space
U = V 0 ⊕ V 1 ⊕ ⋯ ⊕ V 4 ⊕ V ~ 5 ⊕ V ~ 6 . \mathcal{U}=\mathcal{V}_0\oplus\mathcal{V}_1\oplus\cdots\oplus\mathcal{V}_4\oplus\widetilde{\mathcal{V}}_5\oplus\widetilde{\mathcal{V}}_6. U = V 0 ⊕ V 1 ⊕ ⋯ ⊕ V 4 ⊕ V 5 ⊕ V 6 .
It is sufficient to show that U = R n × n \mathcal{U}=\mathbb{R}^{n\times n} U = R n × n . Since U \mathcal{U} U contains all diagonal matrices which are given by V ~ 0 ⊕ V ~ 2 \widetilde{V}_0\oplus\widetilde{V}_2 V 0 ⊕ V 2 , and since it is an invariant subspace, it is sufficient to show that the matrix E 1 , 2 E^{1,2} E 1 , 2 which is zero in all coordinates except for a single coordinate E 1 , 2 1 , 2 = 1 E^{1,2}_{1,2}=1 E 1 , 2 1 , 2 = 1 , is in U \mathcal{U} U .
To show that E 1 , 2 E^{1,2} E 1 , 2 is in U \mathcal{U} U , let us choose a non-zero matrix A ∈ V ~ 5 A\in\widetilde{V}_5 A ∈ V 5 and B ∈ V ~ 6 B\in\widetilde{V}_6 B ∈ V 6 . Both matrices have a non-zero off diagonal matrix element, and by applying an appropriate permutation and rescaling we can assume that A 12 = 1 = B 12 A_{12}=1=B_{12} A 12 = 1 = B 12 . Next, let us average over the group
stab ( 1 , 2 ) = { τ ∈ S n , τ ( 1 ) = 1 , τ ( 2 ) = 2 } \operatorname{stab}(1,2)=\{\tau\in S_n,\tau(1)=1,\tau(2)=2\} stab ( 1 , 2 ) = { τ ∈ S n , τ ( 1 ) = 1 , τ ( 2 ) = 2 }
to obtain new matrices
A ^ = 1 ∣ stab ( 1 , 2 ) ∣ ∑ τ ∈ stab ( 1 , 2 ) τ ⋅ A , B ^ = 1 ∣ stab ( 1 , 2 ) ∣ ∑ τ ∈ stab ( 1 , 2 ) τ ⋅ B \widehat{A}=\frac{1}{|\operatorname{stab}(1,2)|}\sum_{\tau\in\operatorname{stab}(1,2)}\tau\cdot A,\qquad\widehat{B}=\frac{1}{|\operatorname{stab}(1,2)|}\sum_{\tau\in\operatorname{stab}(1,2)}\tau\cdot B A = ∣ stab ( 1 , 2 ) ∣ 1 τ ∈ stab ( 1 , 2 ) ∑ τ ⋅ A , B = ∣ stab ( 1 , 2 ) ∣ 1 τ ∈ stab ( 1 , 2 ) ∑ τ ⋅ B
Note that A ^ \widehat{A} A (respectively B ^ \widehat{B} B ) is in V ~ 5 \widetilde{V}_5 V 5 (respectively V ~ 6 \widetilde{V}_6 V 6 ) and satisfy A ^ 12 = 1 = B ^ 12 \widehat{A}_{12}=1=\widehat{B}_{12} A 12 = 1 = B 12 and
A ^ 1 j = A ^ 1 k , B ^ 1 j = B ^ 1 k ∀ j , k > 2 \widehat{A}_{1j}=\widehat{A}_{1k},\widehat{B}_{1j}=\widehat{B}_{1k}\ \forall j,k>2 A 1 j = A 1 k , B 1 j = B 1 k ∀ j , k > 2
and
A ^ j k = A ^ s t , B ^ j k = B ^ s t \widehat{A}_{jk}=\widehat{A}_{st},\widehat{B}_{jk}=\widehat{B}_{st} A j k = A s t , B j k = B s t
for all j , k , s , t j,k,s,t j , k , s , t which are all larger than 2, and such that j ≠ k , s ≠ t j\ne k,s\ne t j = k , s = t . Defining for convenience
a = 1 n − 2 , b = 2 ( n − 2 ) ( n − 3 ) , a=\frac{1}{n-2},b=\frac{2}{(n-2)(n-3)}, a = n − 2 1 , b = ( n − 2 ) ( n − 3 ) 2 ,
it follows that
A ^ = ( 0 1 − a ⋯ − a − 1 0 a ⋯ a a − a 0 ⋯ 0 ⋮ ⋮ ⋮ ⋱ ⋮ a − a 0 ⋯ 0 ) B ^ = ( 0 1 − a ⋯ − a 1 0 − a ⋯ − a − a − a b ⋯ b ⋮ ⋮ ⋮ ⋱ ⋮ − a − a b ⋯ b ) − diag ( 0 , 0 , b , … , b ) \widehat{A}=\begin{pmatrix}
0 & 1 & -a & \cdots& -a\\
-1 & 0 & a & \cdots& a\\
a & -a & 0 & \cdots& 0\\
\vdots& \vdots& \vdots& \ddots& \vdots\\
a & -a & 0 & \cdots& 0
\end{pmatrix}\qquad
\widehat{B}=\begin{pmatrix}
0 & 1 & -a & \cdots& -a\\
1 & 0 & -a & \cdots& -a\\
-a & -a & b & \cdots& b\\
\vdots& \vdots& \vdots& \ddots& \vdots\\
-a & -a & b & \cdots& b
\end{pmatrix}-\operatorname{diag}(0,0,b,\ldots,b) A = 0 − 1 a ⋮ a 1 0 − a ⋮ − a − a a 0 ⋮ 0 ⋯ ⋯ ⋯ ⋱ ⋯ − a a 0 ⋮ 0 B = 0 1 − a ⋮ − a 1 0 − a ⋮ − a − a − a b ⋮ b ⋯ ⋯ ⋯ ⋱ ⋯ − a − a b ⋮ b − diag ( 0 , 0 , b , … , b )
It is not difficult to show that any matrix which is constant along the rows is in V ~ 0 ⊕ ⋯ ⊕ V ~ 4 ⊆ U \widetilde{V}_0\oplus\cdots\oplus\widetilde{V}_4\subseteq\mathcal{U} V 0 ⊕ ⋯ ⊕ V 4 ⊆ U . The same is true for any matrix which is constant along the columns. Let C ( i ) C(i) C ( i ) denote the matrix with C i j ( i ) = 1 C_{ij}(i)=1 C ij ( i ) = 1 for all j j j and C k j ( i ) = 0 C_{kj}(i)=0 C k j ( i ) = 0 for all j j j and all k ≠ i k \ne i k = i . Then
A ~ : = A ^ + a C ( 1 ) − a C ( 2 ) − a C T ( 1 ) + a C T ( 2 ) = ( 0 1 + 2 a 0 ⋯ 0 − 1 − 2 a 0 0 ⋯ 0 0 0 0 ⋯ 0 ⋮ 0 0 0 ⋯ 0 ) \widetilde{A}:=\widehat{A}+aC(1)-aC(2)-aC^{T}(1)+aC^{T}(2)
=
\begin{pmatrix}
0 & 1+2a & 0 & \cdots& 0 \\
-1-2a & 0 & 0 & \cdots& 0 \\
0 & 0 & 0 & \cdots& 0 \\
\vdots& & & & \\
0 & 0 & 0 & \cdots& 0
\end{pmatrix} A := A + a C ( 1 ) − a C ( 2 ) − a C T ( 1 ) + a C T ( 2 ) = 0 − 1 − 2 a 0 ⋮ 0 1 + 2 a 0 0 0 0 0 0 0 ⋯ ⋯ ⋯ ⋯ 0 0 0 0
B ~ : = B ^ + d i a g ( 0 , 0 , b , … , b ) − b 1 n × n + ( a + b ) ( C ( 1 ) + C ( 2 ) + C T ( 1 ) + C T ( 2 ) ) = ( 2 a + b 1 + 2 a + b 0 ⋯ 0 1 + 2 a + b 2 a + b 0 ⋯ 0 0 0 0 ⋯ 0 ⋮ 0 0 0 ⋯ 0 ) \widetilde{B}:=\widehat{B}+d i a g(0,0,b,\ldots,b)-b\mathbf{1}_{n\times n}+(a+b)(C(1)+C(2)+C^{T}(1)+C^{T}(2))
=
\begin{pmatrix}
2a+b & 1+2a+b & 0 & \cdots& 0 \\
1+2a+b & 2a+b & 0 & \cdots& 0 \\
0 & 0 & 0 & \cdots& 0 \\
\vdots& & & & \\
0 & 0 & 0 & \cdots& 0
\end{pmatrix} B := B + d ia g ( 0 , 0 , b , … , b ) − b 1 n × n + ( a + b ) ( C ( 1 ) + C ( 2 ) + C T ( 1 ) + C T ( 2 )) = 2 a + b 1 + 2 a + b 0 ⋮ 0 1 + 2 a + b 2 a + b 0 0 0 0 0 0 ⋯ ⋯ ⋯ ⋯ 0 0 0 0
where both A ~ \widetilde{A} A and B ~ \widetilde{B} B are in U \mathcal{U} U . Since a , b a,b a , b are positive numbers, we can take the following linear combination which zeros out the ( 2 , 1 ) (2,1) ( 2 , 1 ) entry and sets the ( 1 , 2 ) (1,2) ( 1 , 2 ) entry to one:
1 2 ( 1 1 + 2 a A ~ + 1 1 + 2 a + b B ~ ) = ( c 1 0 ⋯ 0 0 d 0 ⋯ 0 0 0 0 ⋯ 0 ⋮ 0 0 0 ⋯ 0 ) \frac{1}{2}\left(\frac{1}{1+2a}\widetilde{A}+\frac{1}{1+2a+b}\widetilde{B}\right)
=
\begin{pmatrix}
c & 1 & 0 & \cdots& 0 \\
0 & d & 0 & \cdots& 0 \\
0 & 0 & 0 & \cdots& 0 \\
\vdots& & & & \\
0 & 0 & 0 & \cdots& 0
\end{pmatrix} 2 1 ( 1 + 2 a 1 A + 1 + 2 a + b 1 B ) = c 0 0 ⋮ 0 1 d 0 0 0 0 0 0 ⋯ ⋯ ⋯ ⋯ 0 0 0 0
for appropriate c , d c,d c , d . Since diagonal matrices are in U \mathcal{U} U , we can remove the diagonal matrix d i a g ( c , d , 0 , … , 0 ) diag(c,d,0,\ldots,0) d ia g ( c , d , 0 , … , 0 ) to finally obtain that the matrix E 1 , 2 E^{1,2} E 1 , 2 is in U \mathcal{U} U . □ \square □
Small n n n
As discussed in [FWW21; Pea22], when n < 4 n<4 n < 4 the dimension of the space of equivariant mappings is smaller than 15 15 15 : it is 14 14 14 for n = 3 n=3 n = 3 and 8 8 8 for n = 2 n=2 n = 2 . We now explain this according to our derivation.
For n = 3 n=3 n = 3 , the space V 6 \mathcal{V}_6 V 6 is zero-dimensional. In this case, there are only 2 2 + 3 2 + 1 = 14 2^2+3^2+1=14 2 2 + 3 2 + 1 = 14 equivariant linear mappings. When n = 2 n=2 n = 2 , both V 5 \mathcal{V}_5 V 5 and V 6 \mathcal{V}_6 V 6 are zero dimensional. Additionally, we can directly verify that in this case, we can ”drop” V 4 \mathcal{V}_4 V 4 as well because R 2 × 2 = V 0 ⊕ V 1 ⊕ V 2 ⊕ V 3 \mathbb{R}^{2\times2}=\mathcal{V}_0\oplus\mathcal{V}_1\oplus\mathcal{V}_2\oplus\mathcal{V}_3 R 2 × 2 = V 0 ⊕ V 1 ⊕ V 2 ⊕ V 3 . Since V 0 ≅ V 1 \mathcal{V}_0\cong\mathcal{V}_1 V 0 ≅ V 1 and V 2 ≅ V 3 \mathcal{V}_2\cong\mathcal{V}_3 V 2 ≅ V 3 we have 2 2 + 2 2 = 8 2^2+2^2=8 2 2 + 2 2 = 8 equivariant linear mappings.
Proof for Deep Weight Space decomposition
In this section we fill in some details on the tensor product of representations which were omitted from the proof of Theorem 4.4.2 in the main text.
At the base of this discussion is the following lemma [Ser77]:
Lemma D.1. Let G 1 , G 2 G_1,G_2 G 1 , G 2 be finite groups, and let V 1 , V 2 V_1,V_2 V 1 , V 2 be (real or complex) finite dimensional absolutely irreducible representations, with an action denoted by ρ 1 , ρ 2 \rho_1,\rho_2 ρ 1 , ρ 2 . Then the action ρ 1 ⊗ ρ 2 \rho_1\otimes\rho_2 ρ 1 ⊗ ρ 2 on V 1 ⊗ V 2 V_1\otimes V_2 V 1 ⊗ V 2 is an absolutely irreducible representation of G 1 × G 2 G_1\times G_2 G 1 × G 2 .
Firstly, we explain the decomposition of W ^ m \widehat{W}_m W m into irreducibles. We note that the representation W ^ m \widehat{W}_m W m of G G G can be thought of as a representation of S d m − 1 × S d m S_{d_{m-1}}\times S_{d_m} S d m − 1 × S d m , which is a tensor product of the representation R d m − 1 \mathbb{R}^{d_{m-1}} R d m − 1 of S d m − 1 S_{d_{m-1}} S d m − 1 and the representation R d m \mathbb{R}^{d_m} R d m of S d m S_{d_m} S d m . Therefore [Ser77], its irreducible decomposition is given by the tensor product of the irreducible decomposition of its factors, namely
W ^ m ≅ R d m − 1 ⊗ R d m ≅ ( S ⊕ V ( d m − 1 ) ) ⊗ ( S ⊕ V ( d m ) ) ≅ ( S ⊗ S ) ⊕ ( S ⊗ V ( d m ) ) ⊕ ( V ( d m − 1 ) ⊗ S ) ⊕ ( V ( d m − 1 ) ⊗ V ( d m ) ) ≅ S ⊕ V ( d m ) ⊕ V ( d m − 1 ) ⊕ M m . \begin{aligned}
\widehat{W}_m &\cong\mathbb{R}^{d_{m-1}}\otimes\mathbb{R}^{d_m}\cong(S\oplus V(d_{m-1}))\otimes(S\oplus V(d_m))\\
&\cong(S\otimes S)\oplus(S\otimes V(d_m))\oplus(V(d_{m-1})\otimes S)\oplus(V(d_{m-1})\otimes V(d_m))\\
&\cong S\oplus V(d_m)\oplus V(d_{m-1})\oplus M_m.
\end{aligned} W m ≅ R d m − 1 ⊗ R d m ≅ ( S ⊕ V ( d m − 1 )) ⊗ ( S ⊕ V ( d m )) ≅ ( S ⊗ S ) ⊕ ( S ⊗ V ( d m )) ⊕ ( V ( d m − 1 ) ⊗ S ) ⊕ ( V ( d m − 1 ) ⊗ V ( d m )) ≅ S ⊕ V ( d m ) ⊕ V ( d m − 1 ) ⊕ M m .
All components in the decomposition are irreducible as a tensor product of irreducibles.
We also need to show that all irreducibles in the decomposition are absolutely irreducible. For S S S this is obvious. For all representations V ( d m ) V(d_m) V ( d m ) this follows from the fact that these representations can be identified with representations of the permutation group S d m S_{d_m} S d m , which are always absolutely irreducible. To see that M m M_m M m is absolutely irreducible, we note that M m M_m M m is the set of d m × d m − 1 d_m\times d_{m-1} d m × d m − 1 real matrices whose rows and columns all sum to zero, and its complexification C M m \mathbb{C}M_m C M m is the set of complex matrices whose rows and columns all sum to zero. C M m \mathbb{C}M_m C M m is the tensor product of C V ( d m − 1 ) \mathbb{C}V(d_{m-1}) C V ( d m − 1 ) and C V ( d m ) \mathbb{C}V(d_m) C V ( d m ) , the complexification of V ( d m − 1 ) V(d_{m-1}) V ( d m − 1 ) and V ( d m ) V(d_m) V ( d m ) , respectively. Since these spaces are irreducible, so is their tensor product.
Decomposition of weight spaces
We explain here how each matrix A ∈ W m A\in W_m A ∈ W m , m = 2 , … , M − 1 m=2,\ldots,M-1 m = 2 , … , M − 1 can be written as a sum of elements from its irreducible decomposition. For convenience denote a = d m , b = d m − 1 a=d_m,b=d_{m-1} a = d m , b = d m − 1 . We want to write A A A as a sum of elements from the spaces
S ≅ { c 1 a × b } , V m − 1 ≅ { 1 a c T ∣ c ∈ R b , c T 1 b = 0 } , V m ≅ { r 1 b T ∣ r ∈ R a , r T 1 a = 0 } S\cong\{c\mathbf{1}_{a\times b}\},\qquad V_{m-1}\cong\{\mathbf{1}_a c^T\mid c\in\mathbb{R}^b,c^T\mathbf{1}_b=0\},\qquad V_m\cong\{r\mathbf{1}_b^T\mid r\in\mathbb{R}^a,r^T\mathbf{1}_a=0\} S ≅ { c 1 a × b } , V m − 1 ≅ { 1 a c T ∣ c ∈ R b , c T 1 b = 0 } , V m ≅ { r 1 b T ∣ r ∈ R a , r T 1 a = 0 }
and
M m = { C ∈ R a × b , C 1 b = 0 b , C T 1 a = 0 a } . M_m=\{C\in\mathbb{R}^{a\times b},C\mathbf{1}_b=0_b,C^T\mathbf{1}_a=0_a\}. M m = { C ∈ R a × b , C 1 b = 0 b , C T 1 a = 0 a } .
As a first step, we take our given A ∈ V m A \in\mathcal{V}_m A ∈ V m and write it as
A = A ˉ 1 a × b + ( A − A ˉ 1 a × b ) A = \bar{A}1_{a\times b} + (A - \bar{A}1_{a\times b}) A = A ˉ 1 a × b + ( A − A ˉ 1 a × b )
where A ˉ \bar{A} A ˉ is the average of all elements of A A A , and A ˉ 1 a × b \bar{A}1_{a\times b} A ˉ 1 a × b is in S \mathbf{S} S . Next, we need to decompose ( A − A ˉ 1 a × b ) (A - \bar{A}1_{a\times b}) ( A − A ˉ 1 a × b ) which we denote by B B B . Note that ∑ i j B i j = 0 \sum_{ij} B_{ij} = 0 ∑ ij B ij = 0 .
Let r r r and c c c be the vectors describing the average of the rows and columns of B B B , respectively:
r = 1 b B 1 b , c = 1 a B T 1 a r = \frac{1}{b}B1_b,\qquad c = \frac{1}{a}B^{T}1_a r = b 1 B 1 b , c = a 1 B T 1 a
Note that r T 1 a = 1 b ∑ i j B i j = 0 r^{T}1_a = \frac{1}{b}\sum_{ij} B_{ij} = 0 r T 1 a = b 1 ∑ ij B ij = 0 and similarly c T 1 b = 0 c^{T}1_b = 0 c T 1 b = 0 and thus 1 a c T ∈ V m − 1 1_a c^{T} \in\mathcal{V}_{m-1} 1 a c T ∈ V m − 1 and r 1 b T ∈ V m r1_b^{T} \in\mathcal{V}_m r 1 b T ∈ V m . Now let us define
C = B − 1 a c T − r 1 b T C = B - 1_a c^{T} - r1_b^{T} C = B − 1 a c T − r 1 b T
We can directly verify that C 1 b = 0 b C1_b = 0_b C 1 b = 0 b and C T 1 a = 0 a C^{T}1_a = 0_a C T 1 a = 0 a . Therefore C C C is in M m \mathbf{M}_m M m and we obtained the decomposition we wanted.
E Proofs for Wreath Products
To conclude the proof of Theorem 4.5.1 , we need to prove the following lemma
Lemma E.1. Let V \mathcal{V} V be an irreducible representation of G G G , and assume the action of G G G on V \mathcal{V} V is not trivial. Then V k \mathcal{V}^{k} V k is an irreducible representation of G ≀ S k G \wr S_{k} G ≀ S k .
Proof. The space V k \mathcal{V}^{k} V k is invariant. Let W ⊆ V k \mathcal{W} \subseteq\mathcal{V}^{k} W ⊆ V k be a non-zero G ≀ S k G \wr S_{k} G ≀ S k invariant subspace. We need to show that W = V k \mathcal{W} = \mathcal{V}^{k} W = V k .
We first claim that there exists an element in W \mathcal{W} W of the form ( u , 0 , … , 0 ) (u,0,\ldots,0) ( u , 0 , … , 0 ) with u ≠ 0 u \ne0 u = 0 . Let ( v 1 , v 2 , … , v k ) (v_{1},v_{2},\ldots,v_{k}) ( v 1 , v 2 , … , v k ) be a non-zero element in W \mathcal{W} W . By applying a permutation if necessary, we can assume without loss of generality that the first element is non-zero. If v i = 0 v_{i}=0 v i = 0 , ∀ 2 ≤ i ≤ k \forall2 \le i \le k ∀2 ≤ i ≤ k , we are done. Otherwise, there exists some g ∈ G g \in G g ∈ G such that g v 1 ≠ v 1 g v_{1} \ne v_{1} g v 1 = v 1 and so
( g ⋅ v 1 , v 2 , … , v k ) − ( v 1 , v 2 , … , v k ) = ( g v 1 − v 1 , 0 , … , 0 ) ∈ W (g \cdot v_{1},v_{2},\ldots,v_{k})-(v_{1},v_{2},\ldots,v_{k})=(g v_{1}-v_{1},0,\ldots,0)\in\mathcal{W} ( g ⋅ v 1 , v 2 , … , v k ) − ( v 1 , v 2 , … , v k ) = ( g v 1 − v 1 , 0 , … , 0 ) ∈ W
The space { v ∈ V ∣ ( v , 0 , … , 0 ) ∈ W } \{v \in\mathcal{V}\mid(v,0,\ldots,0)\in\mathcal{W}\} { v ∈ V ∣ ( v , 0 , … , 0 ) ∈ W } is invariant under the action of G G G , and we just saw it is not zero. Since V \mathcal{V} V is irreducible, this space equals V \mathcal{V} V , and thus W \mathcal{W} W contains V ⊕ 0 … ⊕ 0 \mathcal{V}\oplus0\ldots\oplus0 V ⊕ 0 … ⊕ 0 . By S k S_{k} S k invariance it follows that W \mathcal{W} W contains all k k k -tuples with a single non-zero entry, and since such tuples span all of V k \mathcal{V}^{k} V k we deduce that W = V k \mathcal{W}=\mathcal{V}^{k} W = V k . □
Theorem 4.5.2. Let V \mathcal{V} V be a real representation of a finite group G G G . and let e 1 , … , e s e_{1},\ldots,e_{s} e 1 , … , e s be a basis for the subspace V f i x e d \mathcal{V}_{fixed} V f i x e d . Let ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ be a G G G invariant inner product on V \mathcal{V} V . Then every linear equivariant map L : V k → V k L:\mathcal{V}^{k}\to\mathcal{V}^{k} L : V k → V k is of the form
L ( v 1 , … , v k ) = ∑ i , j = 1 s a i j ( ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j , … , ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j ) + ( L ^ ( v 1 ) , … , L ^ ( v k ) ) (43) L(v_{1},\ldots,v_{k})=\sum_{i,j=1}^{s}a_{ij}\left(\sum_{\ell=1}^{k}\langle v_{\ell},e_{i}\rangle e_{j},\ldots,\sum_{\ell=1}^{k}\langle v_{\ell},e_{i}\rangle e_{j}\right)+(\widehat{L}(v_{1}),\ldots,\widehat{L}(v_{k})) \tag*{(43)} L ( v 1 , … , v k ) = i , j = 1 ∑ s a ij ( ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j , … , ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j ) + ( L ( v 1 ) , … , L ( v k )) (43) where L ^ : V → V \widehat{L}:\mathcal{V}\to\mathcal{V} L : V → V is a linear equivariant map, and a i j a_{ij} a ij are real numbers. Conversely, every linear mapping of the form defined in equation 4.9 is equivariant.
Proof. It is straightforward to check that every mapping of the form equation 4.9 is equivariant. We would like to prove the converse statement.
It is also straightforward to verify that the space spanned by the non-Siamese mappings
( v 1 , … , v k ) ↦ ( ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j , … , ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j ) (v_{1},\ldots,v_{k})\mapsto\left(\sum_{\ell=1}^{k}\langle v_{\ell},e_{i}\rangle e_{j},\ldots,\sum_{\ell=1}^{k}\langle v_{\ell},e_{i}\rangle e_{j}\right) ( v 1 , … , v k ) ↦ ( ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j , … , ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j )
does not depend on the choice of the basis e 1 , … , e s e_{1},\ldots,e_{s} e 1 , … , e s for V f i x e d \mathcal{V}_{fixed} V f i x e d . In particular, we can assume without loss of generality that e 1 , … , e s e_{1},\ldots,e_{s} e 1 , … , e s are an orthonormal basis for V f i x e d \mathcal{V}_{fixed} V f i x e d .
Note that V \mathcal{V} V can be written as a direct sum of two G G G invariant spaces V = V f i x e d ⊕ V ⊥ \mathcal{V}=\mathcal{V}_{fixed}\oplus\mathcal{V}_{\perp} V = V f i x e d ⊕ V ⊥ , where V ⊥ \mathcal{V}_{\perp} V ⊥ is the space of vectors in V \mathcal{V} V which are orthogonal to V f i x e d \mathcal{V}_{fixed} V f i x e d with respect to the given G \mathcal{G} G invariant inner product. It follows that V k = V f i x e d k ⊕ V ⊥ k \mathcal{V}^{k} = \mathcal{V}^{k}_{fixed} \oplus\mathcal{V}^{k}_{\perp} V k = V f i x e d k ⊕ V ⊥ k is a decomposition of V k \mathcal{V}^{k} V k into two G k \mathcal{G}^{k} G k invariant spaces.
In this situation, we know that any linear equivariant L : V k → V k L : \mathcal{V}^{k} \to\mathcal{V}^{k} L : V k → V k can be written as L = L f i x e d + L ⊥ L = L_{fixed} + L_{\perp} L = L f i x e d + L ⊥ , where L f i x e d : V f i x e d k → V f i x e d k L_{fixed} : \mathcal{V}^{k}_{fixed} \to\mathcal{V}^{k}_{fixed} L f i x e d : V f i x e d k → V f i x e d k and L ⊥ : V ⊥ k → V ⊥ k L_{\perp} : \mathcal{V}^{k}_{\perp} \to\mathcal{V}^{k}_{\perp} L ⊥ : V ⊥ k → V ⊥ k are equivariant. We know that there are no non-zero linear equivariant maps from V f i x e d k \mathcal{V}^{k}_{fixed} V f i x e d k to V ⊥ k \mathcal{V}^{k}_{\perp} V ⊥ k or vice versa, since V f i x e d k \mathcal{V}^{k}_{fixed} V f i x e d k is a direct sum of trivial G k \mathcal{G}^{k} G k -representations, whereas V ⊥ k \mathcal{V}^{k}_{\perp} V ⊥ k contains no trivial G k \mathcal{G}^{k} G k -subrepresentation.
Let L : V ⊥ k → V ⊥ k L : \mathcal{V}^{k}_{\perp} \to\mathcal{V}^{k}_{\perp} L : V ⊥ k → V ⊥ k be a linear equivariant map. We want to show that it is a ’Siamese mapping’. Denote
L ( v 1 , … , v k ) = ( L 1 ( v 1 , … , v k ) , … , L k ( v 1 , … , v k ) ) . L(v_{1}, \ldots, v_{k}) = (L_{1}(v_{1}, \ldots, v_{k}), \ldots, L_{k}(v_{1}, \ldots, v_{k})). L ( v 1 , … , v k ) = ( L 1 ( v 1 , … , v k ) , … , L k ( v 1 , … , v k )) .
We define L ^ : V ⊥ → V ⊥ \hat{L} : \mathcal{V}_{\perp} \to\mathcal{V}_{\perp} L ^ : V ⊥ → V ⊥ by
L ^ ( v ) = L 1 ( v , 0 , … , 0 ) . \hat{L}(v) = L_{1}(v, 0, \ldots, 0). L ^ ( v ) = L 1 ( v , 0 , … , 0 ) .
Note that L ^ \hat{L} L ^ is G \mathcal{G} G equivariant because
L ^ ( g v ) = L 1 ( g v , 0 , … , 0 ) = g L 1 ( v , 0 , … , 0 ) = g L ^ ( v ) , ∀ g ∈ G . \hat{L}(gv) = L_{1}(gv, 0, \ldots, 0) = gL_{1}(v, 0, \ldots, 0) = g\hat{L}(v), \quad\forall g \in\mathcal{G}. L ^ ( g v ) = L 1 ( g v , 0 , … , 0 ) = g L 1 ( v , 0 , … , 0 ) = g L ^ ( v ) , ∀ g ∈ G .
Next, note that for every v 2 , … , v k ∈ V ⊥ v_{2}, \ldots, v_{k} \in\mathcal{V}_{\perp} v 2 , … , v k ∈ V ⊥ , we have that
L 1 ( 0 , v 2 , … , v k ) = L 1 ( g ⋅ 0 , v 2 , … , v k ) = g ⋅ L 1 ( 0 , v 2 , … , v k ) . L_{1}(0, v_{2}, \ldots, v_{k}) = L_{1}(g \cdot0, v_{2}, \ldots, v_{k}) = g \cdot L_{1}(0, v_{2}, \ldots, v_{k}). L 1 ( 0 , v 2 , … , v k ) = L 1 ( g ⋅ 0 , v 2 , … , v k ) = g ⋅ L 1 ( 0 , v 2 , … , v k ) .
Thus,
L 1 ( 0 , v 2 , … , v k ) ∈ V f i x e d . L_{1}(0, v_{2}, \ldots, v_{k}) \in\mathcal{V}_{fixed}. L 1 ( 0 , v 2 , … , v k ) ∈ V f i x e d .
On the other hand, since L : V ⊥ k → V ⊥ k L : \mathcal{V}^{k}_{\perp} \to\mathcal{V}^{k}_{\perp} L : V ⊥ k → V ⊥ k , we also have
L 1 ( 0 , v 2 , … , v k ) ∈ V ⊥ . L_{1}(0, v_{2}, \ldots, v_{k}) \in\mathcal{V}_{\perp}. L 1 ( 0 , v 2 , … , v k ) ∈ V ⊥ .
Therefore,
L 1 ( 0 , v 2 , … , v k ) ∈ V f i x e d ∩ V ⊥ = { 0 } . L_{1}(0, v_{2}, \ldots, v_{k}) \in\mathcal{V}_{fixed} \cap\mathcal{V}_{\perp} = \{0\}. L 1 ( 0 , v 2 , … , v k ) ∈ V f i x e d ∩ V ⊥ = { 0 } .
We deduce that
L 1 ( v 1 , … , v k ) = L 1 ( v 1 , 0 , … , 0 ) = L ^ ( v 1 ) . L_{1}(v_{1}, \ldots, v_{k}) = L_{1}(v_{1}, 0, \ldots, 0) = \hat{L}(v_{1}). L 1 ( v 1 , … , v k ) = L 1 ( v 1 , 0 , … , 0 ) = L ^ ( v 1 ) .
By S k S_{k} S k equivariance we can deduce that
L j ( v 1 , … , v k ) = L ^ ( v j ) , L_{j}(v_{1}, \ldots, v_{k}) = \hat{L}(v_{j}), L j ( v 1 , … , v k ) = L ^ ( v j ) ,
and we can also show, as we did for L 1 L_{1} L 1 , that
L j ( v 1 , … , v k ) = L j ( 0 , … , 0 , v j , 0 , … , 0 ) . L_{j}(v_{1}, \ldots, v_{k}) = L_{j}(0, \ldots, 0, v_{j}, 0, \ldots, 0). L j ( v 1 , … , v k ) = L j ( 0 , … , 0 , v j , 0 , … , 0 ) .
Thus L L L is a Siamese mapping induced by L ^ \hat{L} L ^ , that is,
L ( v 1 , … , v k ) = ( L ^ ( v 1 ) , … , L ^ ( v k ) ) . L(v_1,\ldots,v_k)=\left(\hat{L}(v_1),\ldots,\hat{L}(v_k)\right). L ( v 1 , … , v k ) = ( L ^ ( v 1 ) , … , L ^ ( v k ) ) .
Now, let us consider linear equivariant mappings from V f i x e d k \mathcal{V}_{fixed}^k V f i x e d k to itself. Denote
S i = { c e i : c ∈ R } , \mathbf{S}_i=\{ce_i:c\in\mathbb{R}\}, S i = { c e i : c ∈ R } ,
and note that an irreducible decomposition of V f i x e d \mathcal{V}_{fixed} V f i x e d is given by
V f i x e d = S 1 ⊕ ⋯ ⊕ S s . \mathcal{V}_{fixed}=\mathbf{S}_1\oplus\cdots\oplus\mathbf{S}_s. V f i x e d = S 1 ⊕ ⋯ ⊕ S s .
Accordingly, V f i x e d k \mathcal{V}_{fixed}^k V f i x e d k can be written as a direct sum of the invariant subspaces S i k \mathbf{S}_i^k S i k for i = 1 , … , s i=1,\ldots,s i = 1 , … , s . Since G k G^k G k acts trivially on V f i x e d k \mathcal{V}_{fixed}^k V f i x e d k , the action of G ≀ S k G\wr S_k G ≀ S k on each S i k \mathbf{S}_i^k S i k reduces to the action of S k S_k S k , which is isomorphic to the standard permutation action of S k S_k S k on R k \mathbb{R}^k R k . The isomorphism
ψ i : S i k → R k \psi_i:\mathbf{S}_i^k\to\mathbb{R}^k ψ i : S i k → R k
is given by
ψ i ( v 1 , … , v k ) = ( ⟨ v 1 , e i ⟩ , … , ⟨ v k , e i ⟩ ) . \psi_i(v_1,\ldots,v_k)=\left(\langle v_1,e_i\rangle,\ldots,\langle v_k,e_i\rangle\right). ψ i ( v 1 , … , v k ) = ( ⟨ v 1 , e i ⟩ , … , ⟨ v k , e i ⟩ ) .
Note that, since we assumed without loss of generality that e 1 , … , e s e_1,\ldots,e_s e 1 , … , e s is an orthonormal basis, ψ i \psi_i ψ i is zero on all other components of V f i x e d k \mathcal{V}_{fixed}^k V f i x e d k . Similarly, we define an isomorphism
ψ ~ i : R k → S i k \tilde{\psi}_i:\mathbb{R}^k\to\mathbf{S}_i^k ψ ~ i : R k → S i k
by
ψ ~ i ( x 1 , … , x k ) = ( x 1 e i , … , x k e i ) . \tilde{\psi}_i(x_1,\ldots,x_k)=(x_1e_i,\ldots,x_ke_i). ψ ~ i ( x 1 , … , x k ) = ( x 1 e i , … , x k e i ) .
As discussed in § 7.4.1 , an equivariant mapping from R k \mathbb{R}^k R k to itself can be written as a linear combination of the identity mapping x ↦ x x\mapsto x x ↦ x (which is a Siamese mapping), and the mapping
T x = ( ∑ ℓ = 1 k x ℓ , … , ∑ ℓ = 1 k x ℓ ) . Tx=\left(\sum_{\ell=1}^{k}x_\ell,\ldots,\sum_{\ell=1}^{k}x_\ell\right). T x = ( ℓ = 1 ∑ k x ℓ , … , ℓ = 1 ∑ k x ℓ ) .
Accordingly, an equivariant mapping from S i k \mathbf{S}_i^k S i k to S j k \mathbf{S}_j^k S j k is a linear combination of a Siamese map and the mapping ψ ~ j ∘ T ∘ ψ i \tilde{\psi}_j\circ T\circ\psi_i ψ ~ j ∘ T ∘ ψ i , which is given by
ψ ~ j ∘ T ∘ ψ i ( v 1 , … , v k ) = ( ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j , … , ∑ ℓ = 1 k ⟨ v ℓ , e i ⟩ e j ) . \tilde{\psi}_j\circ T\circ\psi_i(v_1,\ldots,v_k)=\left(\sum_{\ell=1}^{k}\langle v_\ell,e_i\rangle e_j,\ldots,\sum_{\ell=1}^{k}\langle v_\ell,e_i\rangle e_j\right). ψ ~ j ∘ T ∘ ψ i ( v 1 , … , v k ) = ( ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j , … , ℓ = 1 ∑ k ⟨ v ℓ , e i ⟩ e j ) .
This concludes the proof.
Beyond S k S_k S k
We now generalize the results from Theorem 4.5.2 to the case where S k S_k S k is replaced with any subgroup H ≤ S k H \leq S_k H ≤ S k which acts on { 1 , … , k } \{1,\ldots,k\} { 1 , … , k } transitively:
Theorem E.2 (Generalization of Theorem 4.5.2). Let V \mathcal{V} V be a real representation of a finite group G \mathcal{G} G , and let e 1 , … , e s e_1,\ldots,e_s e 1 , … , e s be a basis of the subspace V f i x e d \mathcal{V}_{\mathrm{fixed}} V fixed . Let ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ be a G \mathcal{G} G -invariant inner product on V \mathcal{V} V .
Let H H H be a subgroup of S k S_k S k which acts on { 1 , … , k } \{1,\ldots,k\} { 1 , … , k } transitively. Let T 1 , T 2 , … , T h T_1,T_2,\ldots,T_h T 1 , T 2 , … , T h be a basis for the H H H -equivariant mappings from R k \mathbb{R}^k R k to itself, where T h T_h T h is the identity mapping.
Then every G ⋉ H \mathcal{G}\ltimes H G ⋉ H linear equivariant map L : V k → V k L:\mathcal{V}^k\to\mathcal{V}^k L : V k → V k is of the form
L ( v 1 , … , v k ) = ∑ ℓ = 1 h ∑ i , j = 1 s a i j ℓ ⋅ ( ψ ~ j ∘ T ℓ ∘ ψ i ) ( v 1 , … , v k ) + ( L ^ ( v 1 ) , … , L ^ ( v k ) ) L(v_1,\ldots,v_k)=\sum_{\ell=1}^{h}\sum_{i,j=1}^{s}a_{ij\ell}\cdot(\widetilde{\psi}_j\circ T_\ell\circ\psi_i)(v_1,\ldots,v_k)+\left(\widehat{L}(v_1),\ldots,\widehat{L}(v_k)\right) L ( v 1 , … , v k ) = ℓ = 1 ∑ h i , j = 1 ∑ s a ij ℓ ⋅ ( ψ j ∘ T ℓ ∘ ψ i ) ( v 1 , … , v k ) + ( L ( v 1 ) , … , L ( v k ) ) where L ^ : V → V \widehat{L}:\mathcal{V}\to\mathcal{V} L : V → V is a linear equivariant map, and a i j ℓ a_{ij\ell} a ij ℓ are real numbers. Conversely, every linear mapping of the form defined in is equivariant.
The proof is identical to the proof of Theorem 4.5.2.
References
Monotone and Separable Set Functions: Characterizations and Neural Models
Soutrik Sarangi; Yonatan Sverdlov; Nadav Dym; Abir De
Motivated by applications for set containment problems, we consider the following fundamental problem: can we design set-to-vector functions so that the natural partial order on sets is preserved, namely S ⊆ T S \subseteq T S ⊆ T if and only if F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) . We call functions satisfying this property Monotone and Separating (MAS) set functions. We establish lower and upper bounds for the vector dimension necessary to obtain MAS functions, as a function of the cardinality of the multisets and the underlying ground set. In the important case of an infinite ground set, we show that MAS functions do not exist, but provide a model called MASNet which provably enjoys a relaxed MAS property we name “weakly MAS” and is stable in the sense of Holder continuity. We also show that MAS functions can be used to construct universal models that are monotone by construction and can approximate all monotone set functions. Experimentally, we consider a variety of set containment tasks. The experiments show the benefit of using our MASNet model, in comparison with standard set models which do not incorporate set containment as an inductive bias. Our implementation is available in https://github.com/structlearning/MASNET.
Chapter 5
Introduction
A multiset { { x 1 , … , x n } } \{\{x_1,\ldots,x_n\}\} {{ x 1 , … , x n }} is an unordered collection of vectors, where order does not matter (like sets) and repetitions are allowed (unlike sets). In recent years, there has been increased interest in neural networks that can map multisets to vectors, with applications for physical simulations [HLL23], processing point clouds [29 ] , and graph neural networks [38 ] . Another important application of multiset-to-vector maps is set-containment search [30 , 19 , 16 ] . Here the goal is to check whether a given multiset S S S is (approximately) a subset of T T T , and this is often carried out by learning a multiset-to-vector mapping F F F , and then checking whether F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) (element-wise), in which case, one deduces that S S S is a subset of T T T . In this paper, we look into this problem from a theoretical perspective.
Monotone and Separable functions We begin our analysis with some definitions: we say that a function F F F mapping sets to vectors is monotone if S ⊆ T ⊆ V S \subseteq T \subseteq V S ⊆ T ⊆ V implies that F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) , and we will say that F F F is separable if F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) implies that S ⊆ T S \subseteq T S ⊆ T . When F F F is simultaneously monotone and separable , we call it a MAS function. When F F F is a MAS function, we can safely test whether F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) , and this will be fully equivalent to checking whether S ⊆ T S \subseteq T S ⊆ T .
Motivation for MAS functions It is natural to ask why we’re interested in functions that are MAS instead of just being monotone or just separable. For this, we begin with a couple of real-life applications based on the set-containment task: (I) In recommendation system design, one problem is to recommend item having a particular set of features, S S S . Here, we represent each item as a set of corresponding features T T T , and the problem is to find all sets T T T such that approximately, S ⊂ T S \subset T S ⊂ T . (II) Text entailment where we are given a small query sentence q q q and the goal is to find the set of corpus items c c c from a large corpus C C C , where q ⟹ c q \Longrightarrow c q ⟹ c or c c c entails q q q . Here, q q q and c c c are typically represented as sets of contextual embeddings (S S S for q q q and T T T for c c c ) and the entailment problem can be cast as the problem of checking if S ⊂ T S \subset T S ⊂ T .
In a neural network setting, such problems require us to design a function F F F such that the ”order” between F ( S ) F(S) F ( S ) and F ( T ) F(T) F ( T ) can serve as a boolean test for whether S S S is a subset of T T T . For this, F F F needs to satisfy two conditions: (A) F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) implies S ⊂ T S \subset T S ⊂ T (B) F ( S ) ≰ F ( T ) F(S) \nleq F(T) F ( S ) ≰ F ( T ) implies S ⊈ T S \nsubseteq T S ⊈ T . Here, we like to emphasize that, for high accuracy, F F F needs to satisfy both conditions A and B as above. Monotone functions satisfy condition (B) without necessarily satisfying condition (A). As a result, if we predict S ⊂ T S \subset T S ⊂ T based on F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) when F F F is monotone but not separable, it will give large number of false positives. Similarly, if F F F is separable but not monotone, then F F F satisfies condition (A) without satisfying condition (B). Thus, using such for checking S ⊂ T S \subset T S ⊂ T results in a large number of false negatives. Hence, both monotonicity and separability are necessary to develop an accurate test for set-containment
Related works Monotone set functions have been widely studied in the theory of capacities [11 ] , fuzzy measures [3 CD84] , game theory and economics [20 ] , combinatorial auctions [24 , 15 , 18 ] , and learning theory [BCIW12; YDC+17; BR88; OS; Bsh95; LZL25]. These works primarily focus on scalar-valued set functions or vector-valued models without structural order constraints.
To the best of our knowledge, jointly monotone and separable multiset (MAS) functions, i.e., multiset-to-vector maps that preserve the partial order induced by multiset containment—have not been previously formalized. Classical results on partially ordered sets, such as order dimension [14 ] , provide relevant mathematical background but do not address learnable or differentiable constructions. Our goal is to characterize the existence of MAS functions and develop models that satisfy MAS constraints by construction for multiset containment tasks.
This work is also related to injective multiset representations. Foundational results on permutation-invariant functions include [ZKR+17; XHLJ19], with subsequent work analyzing the latent dimension required for injectivity [2 , 33 , 37 ] and proposing differentiable injective or bi-Lipschitz embeddings [1 , 6 , 31 ] . In this work, we study MAS functions, which impose strictly stronger constraints than injectivity (see Subsection 5.2.1).
An additional goal of this work is stability: in most learning scenarios, we are looking for S S S , which is only approximately a subset of T T T . Accordingly, we would like to design functions F F F that are not only MAS but also stable, in the sense that when S S S is approximately a subset of T T T , then F ( S ) F(S) F ( S ) is approximately dominated by F ( T ) F(T) F ( T ) .
Summary of our goal In summary, in this work, we aim to characterize (MAS) set functions that output finite-dimensional set representations and subsequently design neural networks for such functions. At a high level, we seek to address the following questions: (1) What are the conditions for existence of a finite dimensional MAS functions? (2) What are the permissible relaxations to monotonicity and separability if the existential conditions identified in (1) are not satisfied? (3) What are the possible neural architectures for these functions? (4) Are these functions stable? We expect that the design of trainable models with built-in guarantees of monotonicity, separability, and stability will significantly enhance the inductive bias for set containment applications.
Main Results
We address the goal specified in the previous subsection by providing a detailed characterization and neural architecture for monotone and separable set functions. Our main results are as follows:
Existential characterization of MAS functions Our first objective is to determine whether it is possible to obtain a MAS set function with finite output dimension m m m . We begin our analysis in the case where set elements are assumed to be taken from a finite ground set V V V , and show that in this case a MAS function exists if, and only if, m ≥ ∣ V ∣ m \ge|V| m ≥ ∣ V ∣ . Next, we add the assumption that the cardinality of the input multisets is bounded by k k k , and provide lower and upper bounds for m m m in this case. When the ground set is infinite, we show that MAS set functions do not exist.
Weakly MAS functions In most applications, the ground set is V = R D V = \mathbb{R}^{D} V = R D , and our analysis shows that in this case, MAS functions do not exist in general. To address this, we introduce the notion of weakly MAS functions by extending the set functions to parametric set functions F ( S , w ) F(S,w) F ( S , w ) , where w w w is a parameter. Weakly MAS functions require that (i) F ( ⋅ , w ) F(\cdot,w) F ( ⋅ , w ) is monotone, for all w w w , and (ii) for any S ⊄ T S \not\subset T S ⊂ T , there exists at least one w ∈ W w \in\mathcal{W} w ∈ W such that F ( ⋅ , w ) F(\cdot,w) F ( ⋅ , w ) separates S S S and T T T . We explain how, by choosing suitable activations, it is possible to construct simple deep sets [ZKR+17] models which are weakly MAS, and name the resulting model MASNet.
Stability We define the notion of stability as discussed above, separability notions are Boolean and fail to capture graded distance. We would like to guarantee that F ( T ) F(T) F ( T ) is ’almost’ larger than F ( S ) F(S) F ( S ) when S S S is ’almost’ a subset of T T T . To address this, we propose a novel asymmetric set distance and present a Holder separability condition [13 ] which ensures stability in terms of this asymmetric distance.
Monotone Functions Finally, we show that MAS functions can be used to provide universal models for computing set-to-vector functions. Thus, MAS functions may be a useful concept also outside of the set containment setting.
Experiments We provide a number of experiments showing that for set containment problems, using our weakly MAS model leads to stronger results in comparison with standard multiset models like DeepSets and SetTransformer, which do not incorporate MAS considerations as an inductive bias.
Summary of contributions Our main contributions in this paper are: (1) We introduce the novel notion of monotone and separating (MAS) multiset functions. (2) We discuss lower and upper bounds for the embedding dimension for MAS functions, and suggest a model MASNet, which is weakly MAS. (3) We show the stability of MASNet. (4) Experimentally, we prove the effectiveness of MASNet for set containment tasks.
Existential characterization of MAS functions
We begin by stating the notation we will use for our paper.
Notation We denote by V V V the ground set and by P < ∞ ( V ) \mathcal{P}_{<\infty}(V) P < ∞ ( V ) the collection of all multisets from V V V with finite cardinality S = { { x 1 , ⋯ , x s } } S = \{\{x_1,\cdots,x_s\}\} S = {{ x 1 , ⋯ , x s }} where x i ∈ V x_i \in V x i ∈ V . We denote the space of all multisets with at most k k k elements by P ≤ k ( V ) \mathcal{P}_{\leq k}(V) P ≤ k ( V ) . We use [ d ] [d] [ d ] to denote the set { 1 , ⋯ , d } \{1,\cdots,d\} { 1 , ⋯ , d } . Given two vectors v , u ∈ R d v,u \in\mathbb{R}^{d} v , u ∈ R d , we say v ≤ u v \leq u v ≤ u if v [ i ] ≤ u [ i ] v[i] \leq u[i] v [ i ] ≤ u [ i ] for all i ∈ [ d ] i \in[d] i ∈ [ d ] . Also, for an element v ∈ V v \in V v ∈ V and a multiset S ∈ P < ∞ ( V ) S \in\mathcal{P}_{<\infty}(V) S ∈ P < ∞ ( V ) , we use c S ( v ) c_S(v) c S ( v ) to denote the number of times v v v occurs in S S S . We say S ⊆ T S \subseteq T S ⊆ T if c S ( v ) ≤ c T ( v ) c_S(v) \leq c_T(v) c S ( v ) ≤ c T ( v ) for all v ∈ V v \in V v ∈ V .
Proofs All proofs for the results stated in the paper are provided in Appendix B.
In this section, our goal is to understand when an MAS function F : P ≤ k ( V ) → R m F : \mathcal{P}_{\leq k}(V) \to\mathbb{R}^{m} F : P ≤ k ( V ) → R m or F : P < ∞ ( V ) → R m F : \mathcal{P}_{<\infty}(V) \to\mathbb{R}^{m} F : P < ∞ ( V ) → R m exists, and if it does, what is the smallest m m m for which such a mapping exists. We denote this minimal dimension as m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) , and m ⋆ ( V , ∞ ) m^{\star}(V,\infty) m ⋆ ( V , ∞ ) , respectively.
Prologue: Relation to injectivity
Our inquiry is related to the notion of injective multiset functions, as every MAS function F F F is, in particular, injective. Indeed, suppose for two sets S , T S,T S , T we have F ( S ) = F ( T ) F(S) = F(T) F ( S ) = F ( T ) . Then, we have both F ( S ) ≤ F ( T ) F(S) \leq F(T) F ( S ) ≤ F ( T ) and F ( T ) ≤ F ( S ) F(T) \leq F(S) F ( T ) ≤ F ( S ) . Hence, we have both S ⊆ T S \subseteq T S ⊆ T and T ⊆ S T \subseteq S T ⊆ S by separability, which implies S = T S = T S = T . Thus, injectivity follows from separability.
Injective multiset functions f : P ≤ k ( V ) → R m f : \mathcal{P}_{\leq k}(V) \to\mathbb{R}^{m} f : P ≤ k ( V ) → R m are attainable even for m = 1 m = 1 m = 1 , provided that V V V is finite or even infinite [ZKR+17; XHLJ19; AGA+23; AD25]. As we will see, attaining the strong condition of MAS multiset functions requires high dimension, and in some cases (infinite V V V ), it does not even exist. This can be seen even in the following very simple example:
Example 5.2.1. Assume the ground set consists of only two elements V = { 0 , 1 } V = \{0,1\} V = { 0 , 1 } , and we only look for multisets of cardinality ≤ k = 1 \leq k = 1 ≤ k = 1 . In this case, there are only three multisets in the space P ≤ 1 ( V ) \mathcal{P}_{\leq1}(V) P ≤ 1 ( V ) : the empty set, S = { 0 } S = \{0\} S = { 0 } , and T = { 1 } T = \{1\} T = { 1 } . As discussed previously, there exists a multiset function F : P ≤ 1 ( V ) → R F : \mathcal{P}_{\leq1}(V) \to\mathbb{R} F : P ≤ 1 ( V ) → R which is injective. However, no such function can be separated. This is because neither S S S nor T T T is a subset of the other. However, because real numbers are totally ordered, we have either F ( S ) ≤ F ( T ) F(S) \leq F(T) F ( S ) ≤ F ( T ) or F ( T ) ≤ F ( S ) F(T) \leq F(S) F ( T ) ≤ F ( S ) , violating the separability condition.
We note that the monotonicity can be satisfied by using functions of the form
F ( S ) = ∑ x ∈ S f ( x ) F(S) = \sum_{x \in S} f(x) F ( S ) = x ∈ S ∑ f ( x )
for non-negative f f f . Thus, achieving separation is harder than achieving monotonicity.
The key reason for the non-existence of a scalar MAS set function is that multisets (or sets) are not totally ordered, but scalars are. However, this issue does not arise for multidimensional set functions, where vectors, similar to sets, are partially ordered. So, when mapping to vectors, when and how can MAS functions be constructed? We will now discuss this.
Existence of MAS functions: finite ground set, unbounded cardinality
We study MAS functions F : P < ∞ ( V ) → R m F : \mathcal{P}_{<\infty}(V) \to\mathbb{R}^{m} F : P < ∞ ( V ) → R m , where input multisets may have an arbitrarily large finite cardinality, and the ground set is finite ∣ V ∣ = n |V| = n ∣ V ∣ = n . In this case, we can show that the smallest possible dimension m ⋆ ( V , k ) m^\star(V,k) m ⋆ ( V , k ) of a MAS function is exactly n n n :
Theorem 5.2.2 (Dimension of MAS Function). For a finite ground set V V V of size n n n , there exists a MAS function F : P < ∞ ( V ) → R n F:\mathcal{P}_{<\infty}(V)\to\mathbb{R}^n F : P < ∞ ( V ) → R n . In addition, any MAS function must have a dimension of at least n n n . In other words, m ⋆ ( V , ∞ ) = n m^\star(V,\infty)=n m ⋆ ( V , ∞ ) = n .
Proof Sketch The construction of a monotone embedding with n = ∣ V ∣ n=|V| n = ∣ V ∣ simply uses one-hot encoding. Namely, we identify V V V with [ n ] [n] [ n ] . For every S ∈ P < ∞ ( V ) S\in\mathcal{P}_{<\infty}(V) S ∈ P < ∞ ( V ) , we define
F ( S ) = ∑ s ∈ S e s ∈ R n (44) F(S)=\sum_{s\in S}e_s\in\mathbb{R}^n \tag*{(44)} F ( S ) = s ∈ S ∑ e s ∈ R n (44)
where e s ∈ R n e_s\in\mathbb{R}^n e s ∈ R n is the vector with e s [ s ] = 1 e_s[s]=1 e s [ s ] = 1 and e s [ j ] = 0 e_s[j]=0 e s [ j ] = 0 for all j ≠ s . I t ’ s c l e a r j\ne s. It’s clear j = s . I t ’ sc l e a r F$ satisfies all conditions. For the second part, assume we have an embedding of dimension m m m , for every output dimension i ∈ [ m ] i\in[m] i ∈ [ m ] , there is a “maximal singleton element” v i ⋆ ∈ V v_i^\star\in V v i ⋆ ∈ V such that F ( { v i ⋆ } ) [ i ] ≥ F ( { v } ) [ i ] F(\{v_i^\star\})[i]\ge F(\{v\})[i] F ({ v i ⋆ }) [ i ] ≥ F ({ v }) [ i ] , ∀ v ∈ V \forall v\in V ∀ v ∈ V . The value of the set function F F F applied to T = { v 1 ⋆ , … , v m ⋆ } T=\{v_1^\star,\ldots,v_m^\star\} T = { v 1 ⋆ , … , v m ⋆ } will dominate any singleton, due to the monotonicity of F F F . However, if m < ∣ V ∣ m<|V| m < ∣ V ∣ , then we can select a u ∈ V ∖ T u\in V\setminus T u ∈ V ∖ T and then F ( { u } ) ≤ F ( T ) F(\{u\})\le F(T) F ({ u }) ≤ F ( T ) , which contradicts separability.
Existence of MAS functions: finite ground set, finite cardinality
We now consider the case where the ground set V V V is finite, as before, but now the multisets have bounded cardinality k < ∣ V ∣ k<|V| k < ∣ V ∣ . Indeed, in many practical applications, input multiset cardinality is much smaller than ∣ V ∣ |V| ∣ V ∣ , even when ∣ V ∣ |V| ∣ V ∣ is large. In this setting, we show that we can get a lower embedding dimension m m m than in Theorem 5.2.2. In fact, we show the minimal output dimension m m m of a MAS function F : P ≤ k ( V ) → R m F:\mathcal{P}_{\le k}(V)\to\mathbb{R}^m F : P ≤ k ( V ) → R m can scale logarithmically with the size of the ground set ∣ V ∣ |V| ∣ V ∣ . However, this comes at the price of an exponential dependence on k k k :
Theorem 5.2.3 (Upper Bound on m ⋆ ( V , k ) m^\star(V,k) m ⋆ ( V , k ) ). Let k < n k<n k < n be natural numbers, and let V V V be a ground set with ∣ V ∣ = n |V|=n ∣ V ∣ = n . Then there exists a MAS function F : P ≤ k ( V ) → R m F:\mathcal{P}_{\le k}(V)\to\mathbb{R}^m F : P ≤ k ( V ) → R m with embedding dimension m = ( k + 2 ) k + 2 log ( n ) m=(k+2)^{k+2}\log(n) m = ( k + 2 ) k + 2 log ( n ) .
Proof sketch We construct the MAS function by taking the one-hot embedding F : P ≤ k ( V ) → R n F:\mathcal{P}_{\le k}(V)\to\mathbb{R}^n F : P ≤ k ( V ) → R n from (44) , and then applying m m m random projections defined by vectors in R n \mathbb{R}^n R n with non-negative entries. The non-negativity ensures monotonicity, and we prove that for the stated value of m m m , the probability of achieving a MAS function using this procedure is strictly positive.
The theorem shows that m ⋆ ( V , k ) ≤ ( k + 2 ) k + 2 log ( n ) m^\star(V,k)\le(k+2)^{k+2}\log(n) m ⋆ ( V , k ) ≤ ( k + 2 ) k + 2 log ( n ) . We now give two lower bounds on the embedding dimension: we show that m m m must depend at least linearly on k k k , and at least double logarithmically on $n:
Theorem 5.2.4. (Lower bounds on m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) ). Let k ≥ 2 k \ge2 k ≥ 2 and n n n be natural numbers, and let V V V be a ground set with ∣ V ∣ = n |V| = n ∣ V ∣ = n . Then the smallest possible dimension m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) of a MAS function satisfies m ⋆ ( V , k ) ≥ log 2 ( log 3 n ) m^{\star}(V,k) \ge\log_{2}(\log_{3} n) m ⋆ ( V , k ) ≥ log 2 ( log 3 n ) . Moreover, if k ≤ n − 1 2 k \le\frac{n-1}{2} k ≤ 2 n − 1 then m ⋆ ( V , k ) ≥ 2 k m^{\star}(V,k) \ge2k m ⋆ ( V , k ) ≥ 2 k
Proof Sketch. To obtain the first lower bound, we consider the sequence of singleton-set embeddings M : = ( F ( { 1 } ) , … , F ( { n } ) ) \mathcal{M} := (F(\{1\}), \ldots, F(\{n\})) M := ( F ({ 1 }) , … , F ({ n })) . By the Erdős-Szekers theorem [17 ] , every scalar sequence with n n n elements has a monotone subsequence of length ∼ n 1 2 \sim n^{\frac{1}{2}} ∼ n 2 1 . Applying this recursively to vectors of length m m m gets us a monotone subsequence in all m m m coordinates of the vector, of length ∼ n 1 2 m \sim n^{\frac{1}{2^{m}}} ∼ n 2 m 1 . If this subsequence is of length ≥ 3 \ge3 ≥ 3 then we will have three distinct elements v 1 , v 2 , v 3 ∈ V v_{1}, v_{2}, v_{3} \in V v 1 , v 2 , v 3 ∈ V such that F ( { v 1 } ) [ i ] ≤ F ( { v 2 } ) [ i ] ≤ F ( { v 3 } ) [ i ] F(\{v_{1}\})[i] \le F(\{v_{2}\})[i] \le F(\{v_{3}\})[i] F ({ v 1 }) [ i ] ≤ F ({ v 2 }) [ i ] ≤ F ({ v 3 }) [ i ] or F ( { v 3 } ) [ i ] ≤ F ( { v 2 } ) [ i ] ≤ F ( { v 1 } ) [ i ] F(\{v_{3}\})[i] \le F(\{v_{2}\})[i] \le F(\{v_{1}\})[i] F ({ v 3 }) [ i ] ≤ F ({ v 2 }) [ i ] ≤ F ({ v 1 }) [ i ] , ∀ i ∈ [ m ] \forall i \in[m] ∀ i ∈ [ m ] . Monotonicity implies that F ( { v 2 } ) [ i ] F(\{v_{2}\})[i] F ({ v 2 }) [ i ] is dominated by F ( { v 1 , v 3 } ) [ i ] F(\{v_{1},v_{3}\})[i] F ({ v 1 , v 3 }) [ i ] , for all i ∈ [ m ] i \in[m] i ∈ [ m ] , thus contradicting separability. It follows that MAS existence can only happen when n 1 2 m ≤ 2 n^{\frac{1}{2^{m}}} \le2 n 2 m 1 ≤ 2 , which leads to the double logarithmic lower bound.
For the lower bound in terms of k k k , the argument is similar to the proof of Theorem 5.2.2, where we select a ”maximal singleton element” v i ⋆ v_{i}^{\star} v i ⋆ for every dimension i ∈ [ m ] i \in[m] i ∈ [ m ] . Since m < ∣ V ∣ − 1 m < |V| - 1 m < ∣ V ∣ − 1 , we can find elements { u 1 , u 2 } \{u_{1},u_{2}\} { u 1 , u 2 } disjoint from the collection ⋃ i = 1 m { v i ⋆ } \bigcup_{i=1}^{m}\{v_{i}^{\star}\} ⋃ i = 1 m { v i ⋆ } , and WLOG F ( { u 1 } ) [ i ] ≥ F ( { u 2 } ) [ i ] F(\{u_{1}\})[i] \ge F(\{u_{2}\})[i] F ({ u 1 }) [ i ] ≥ F ({ u 2 }) [ i ] for at least half of the indices in [ m ] [m] [ m ] . We then construct S = { u 2 } S = \{u_{2}\} S = { u 2 } and T T T to be the union of u 1 u_{1} u 1 and all the v i ⋆ v_{i}^{\star} v i ⋆ for all dimensions i i i , where F ( { u 1 } ) [ i ] ≤ F ( { u 2 } ) [ i ] F(\{u_{1}\})[i] \le F(\{u_{2}\})[i] F ({ u 1 }) [ i ] ≤ F ({ u 2 }) [ i ] . Then monotonicity implies F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) , but S ⊈ T S \nsubseteq T S ⊈ T . This implies k ≤ ∣ T ∣ − 1 ≤ m / 2 k \le|T| - 1 \le m/2 k ≤ ∣ T ∣ − 1 ≤ m /2 , as otherwise separability of F F F in P ≤ k ( V ) \mathcal{P}_{\le k}(V) P ≤ k ( V ) is violated.
Non-existence of MAS functions: for infinite ground set
For most practical applications, we deal with infinite (often uncountable) ground sets, for example V = R d V = \mathbb{R}^{d} V = R d . Thus, we would like to analyze whether MAS functions exist when ∣ V ∣ = ∞ |V| = \infty ∣ V ∣ = ∞ . The answer to this question is negative:
Corollary 5.2.5. Given a ground set V V V with ∣ V ∣ = ∞ |V| = \infty ∣ V ∣ = ∞ , and k ≥ 2 k \ge2 k ≥ 2 , there does not exist a MAS function F : P ≤ k ( V ) → R m F : \mathcal{P}_{\le k}(V) \to\mathbb{R}^{m} F : P ≤ k ( V ) → R m for any finite m ∈ N m \in\mathbb{N} m ∈ N .
This corollary is a simple consequence of the result in Theorem 5.2.4 that the embedding dimension cannot be larger than log log ∣ V ∣ \log\log|V| log log ∣ V ∣ when V V V is finite. We note that this result makes the minimal assumption k ≥ 2 k \ge2 k ≥ 2 . In Appendix B we show this assumption is necessary, and that in the degenerate case k = 1 k = 1 k = 1 there is a MAS function F : P ≤ 1 ( V ) → R 2 F : \mathcal{P}_{\le1}(V) \to\mathbb{R}^{2} F : P ≤ 1 ( V ) → R 2 for V = [ − 1 , 1 ] V = [-1,1] V = [ − 1 , 1 ] .
Summary In this section, we provided lower and upper bounds for the smallest value m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) for which an MAS function F : P ≤ k ( V ) → R m F : \mathcal{P}_{\le k}(V) \to\mathbb{R}^{m} F : P ≤ k ( V ) → R m exists. Our results are summarized in tbl:5.1 . We note that some of these results require weak assumptions, which are stated in the theorems but not in the table.
Table 5. This table summarizes the lower and upper bounds for the smallest value m ⋆ ( V , k ) m^\star(V,k) m ⋆ ( V , k ) for which an MAS function F : P < k ( V ) → R m F:\mathcal{P}_{<k}(V)\to\mathbb{R}^m F : P < k ( V ) → R m exists, where k k k is maximal set cardinality and n n n denotes the cardinality of the ground set V V V .
Relaxations of MAS functions
Our definitions so far did not address differential parameterized set functions F ( S , w ) F(S,w) F ( S , w ) , which are the natural object of interest for set learning applications. One natural way to address this could be to require that there exists w w w such that F ( ∙ , w ) F(\bullet,w) F ( ∙ , w ) is a MAS function; however, since our theoretical results show that MAS functions do not exist when ∣ V ∣ = ∞ |V|=\infty ∣ V ∣ = ∞ , this requirement is too restrictive. Instead, we retain monotonicity as a hard constraint for every parameter w ∈ W w\in\mathcal{W} w ∈ W , while treating separability as an existential condition over the parameter space.
The notion of weakly MAS functions
Formally, given a ground set V V V , and a probability space ( W , B , μ ) (\mathcal{W},\mathcal{B},\mu) ( W , B , μ ) , we consider parametric set functions F : P < ∞ ( V ) × W → R m F:\mathcal{P}_{<\infty}(V)\times\mathcal{W}\to\mathbb{R}^m F : P < ∞ ( V ) × W → R m , where F ( S , ∙ ) F(S,\bullet) F ( S , ∙ ) is measurable for all S S S . We then define weakly MAS functions as:
Definition 5.3.1 (weakly MAS function). The set function F : P < ∞ ( V ) × W → R F:\mathcal{P}_{<\infty}(V)\times\mathcal{W}\to\mathbb{R} F : P < ∞ ( V ) × W → R is a weakly MAS function if the following two conditions are satisfied:
(1) Pointwise monotonicity For any w ∈ W w\in W w ∈ W and S ⊆ T S\subseteq T S ⊆ T , we have that F ( S , w ) ≤ F ( T , w ) F(S,w)\le F(T,w) F ( S , w ) ≤ F ( T , w ) .
(2) Weak separability If S ⊈ T S\not\subseteq T S ⊆ T , then there exists w ∈ W w\in\mathcal{W} w ∈ W such that F ( S ; w ) > F ( T ; w ) F(S;w)>F(T;w) F ( S ; w ) > F ( T ; w ) .
Which condition, monotonicity or separability, should be relaxed? In our definition, we relax the separability requirement so that different multiset pairs can be separated by different parameters, while we maintain strict requirements of pointwise monotonicity. This is because, as discussed in Example 5.2.1, constructing monotone set functions is straightforward, even in one dimension. In contrast, separability is a much stronger and more difficult condition to satisfy. This is a key reason why MAS functions do not exist when ∣ V ∣ = ∞ |V|=\infty ∣ V ∣ = ∞ . Indeed, as shown in Appendix B, there can be no continuous separable set function even without the monotonicity assumption when the ground set is V = R D V=\mathbb{R}^D V = R D . Therefore, separability is the natural condition for relaxation. Moreover, monotonicity remains a useful and often necessary constraint in many applications.
Are scalar deep sets weakly MAS? Equipped with the new notion of weakly MAS functions, we ask the question of how to construct such a class of functions. As a first step to achieve this goal, we look into a simple instance of DeepSets [ZKR+17]. DeepSets defines a set function by applying an ’inner’ MLP M 1 M_1 M 1 to each set element, summing over the result, and then applying an ’outer’ MLP M 2 M_2 M 2 . We consider the case where M 1 M_1 M 1 is a shallow MLP M 1 ( x ) = σ ( A x + b ) M_1(x)=\sigma(Ax+b) M 1 ( x ) = σ ( A x + b ) which gives us models of the form
F ( S ; ( A , b ) ) = M 2 ( ∑ x ∈ S σ ( A x + b ) ) (45) F(S;(A,b))=M_2\left(\sum_{x\in S}\sigma(Ax+b)\right) \tag*{(45)} F ( S ; ( A , b )) = M 2 ( x ∈ S ∑ σ ( A x + b ) ) (45)
To ensure monotonicity for every parameter choice A , b A,b A , b , we will require that M 2 M_2 M 2 is a monotone vector-to-vector function, and the activation σ \sigma σ is non-negative. While these choices automatically ensure monotonicity, they may not lead to weak MAS functions as for many activations there will be no separation:
Proposition 5.3.2. If the deep set model in (5.2) is implemented with a vector-to-vector monotonously increasing function M 2 M_2 M 2 and a non-negative activation σ \sigma σ , then F ( ∙ ; ( A , b ) ) F(\bullet;(A,b)) F ( ∙ ; ( A , b )) is monotone for every ( A , b ) (A,b) ( A , b ) . If in addition σ \sigma σ is monotone (increasing or decreasing), then there exists S ⊈ T S\not\subseteq T S ⊆ T with F ( S ; ( A , b ) ) ≤ F ( T ; ( A , b ) ) F(S;(A,b))\leq F(T;(A,b)) F ( S ; ( A , b )) ≤ F ( T ; ( A , b )) for all A , b A,b A , b .
Proof idea The monotonicity is rather straightforward. To prove lack of separation when σ \sigma σ is monotone, choose x ≠ y x\neq y x = y , z = 1 2 ( x + y ) z=\frac{1}{2}(x+y) z = 2 1 ( x + y ) , S = { z } S=\{z\} S = { z } , T = { x , y } T=\{x,y\} T = { x , y } , and then A z + b Az+b A z + b is the average of A x + b Ax+b A x + b and A y + b Ay+b A y + b , which can be used to show that F ( T ; ( A , b ) ) ≥ F ( S ; ( A , b ) ) F(T;(A,b))\geq F(S;(A,b)) F ( T ; ( A , b )) ≥ F ( S ; ( A , b )) for all A , b A,b A , b .
Are Set Transformers weakly MAS? Now we ask the question if Set Transformers [23 ] are weakly MAS. We notice in the following that Set Transformers are not even monotone in the following result. For this, we consider Set Transformer with sum-pooling, namely F ( S ; W Q , W K , W V ) = SumPool ( Attn ( S ) ) F(S;W_Q,W_K,W_V)=\operatorname{SumPool}(\operatorname{Attn}(S)) F ( S ; W Q , W K , W V ) = SumPool ( Attn ( S )) . Explicitly, this is defined for a given multiset X = { x 1 , ⋯ , x n } X=\{x_1,\cdots,x_n\} X = { x 1 , ⋯ , x n } by first defining query, key, values per-point: q i = W Q x i \mathbf{q}_i=W_Qx_i q i = W Q x i , k i = W K x i \mathbf{k}_i=W_Kx_i k i = W K x i , v i = W V x i \mathbf{v}_i=W_Vx_i v i = W V x i , and then taking a weighted average, defined by weights:
α i , j = e q i ⋅ k j ∑ s e q i ⋅ k s \alpha_{i,j}=\frac{e^{\mathbf{q}_i\cdot\mathbf{k}_j}}{\sum_{s}e^{\mathbf{q}_i\cdot\mathbf{k}_s}} α i , j = ∑ s e q i ⋅ k s e q i ⋅ k j
to obtain a permutation-invariant function
F ( S ; W Q , W K , W V ) : = ∑ i , j α i , j v i F(S;W_Q,W_K,W_V):=\sum_{i,j}\alpha_{i,j}\mathbf{v}_i F ( S ; W Q , W K , W V ) := i , j ∑ α i , j v i
Proposition 5.3.3. Let V = R d V=\mathbb{R}^{d} V = R d and let F : P < ∞ ( V ) → R D F:\mathcal{P}_{<\infty}(V)\to\mathbb{R}^{D} F : P < ∞ ( V ) → R D be a Set Transformer defined by F ( S ; W Q , W K , W V ) : = SumPool ( Attn ( S ) ) F(S;W_Q,W_K,W_V):=\operatorname{SumPool}(\operatorname{Attn}(S)) F ( S ; W Q , W K , W V ) := SumPool ( Attn ( S )) . Then F F F is not a point-wise monotone function.
Proof Idea We consider full rank matrices W Q = W K W_Q=W_K W Q = W K and any non-zero W V W_V W V . We choose x 1 ∈ R d \mathbf{x}_1\in\mathbb{R}^{d} x 1 ∈ R d such that v 1 : = W V x 1 ≠ 0 \mathbf{v}_1:=W_V\mathbf{x}_1\neq0 v 1 := W V x 1 = 0 , and consider the sets S = { x 1 } S=\{\mathbf{x}_1\} S = { x 1 } , T = { x 1 , 0 d } T=\{\mathbf{x}_1,\mathbf{0}_d\} T = { x 1 , 0 d } . Then, F ( S ; W Q , W K , W V ) = v 1 F(S;W_Q,W_K,W_V)=\mathbf{v}_1 F ( S ; W Q , W K , W V ) = v 1 and F ( T ; W Q , W K , W V ) = ( α 1 , 1 + α 1 , 2 ) v 1 F(T;W_Q,W_K,W_V)=(\alpha_{1,1}+\alpha_{1,2})\mathbf{v}_1 F ( T ; W Q , W K , W V ) = ( α 1 , 1 + α 1 , 2 ) v 1 . Note that, α 1 , 1 + α 1 , 2 > 1 \alpha_{1,1}+\alpha_{1,2}>1 α 1 , 1 + α 1 , 2 > 1 and thus, for any negative v 1 [ j ] \mathbf{v}_1[j] v 1 [ j ] , we have F ( T ; W Q , W K , W V ) [ j ] < F ( S ; W Q , W K , W V ) [ j ] F(T;W_Q,W_K,W_V)[j]<F(S;W_Q,W_K,W_V)[j] F ( T ; W Q , W K , W V ) [ j ] < F ( S ; W Q , W K , W V ) [ j ] and thus, monotonicity is violated.
Thus, unlike Deep Sets, it’s not obvious how to make Set Transformers monotone, let alone weakly separable. Thus, in the following sections, we shall focus on obtaining weakly-MAS functions from DeepSet-like models only.
The Hat activation class
Note that most commonly used non-negative activation functions, such as ReLU or sigmoid, are monotonically non-decreasing and therefore, from Proposition 5.3.2 , using them as σ \sigma σ above will fail weak separability. Here, we consider a novel class of activation functions we call ’Hat Activations’, which will all make F F F in Eq. (45) weakly MAS.
Definition 5.3.4 (The Hat activation function). We call a function σ : R → R \sigma: \mathbb{R} \to\mathbb{R} σ : R → R a hat activation if it is (a) non-negative (b) compactly supported (c) not identically zero and (d) continuous.
Examples of hat functions are the third and fourth functions in Figure 5.1 . When σ \sigma σ is a hat function, Equation (45) is weakly MAS, even when the output of F F F is taken to be scalar. For this result, we will need to require that M 2 M_2 M 2 is strictly monotone. This can be handled by simply setting M 2 M_2 M 2 to be the identity.
Figure 5.1. Using a multiset model as in (5.2) with activations which are not always non-negative, like σ = T a n h \sigma= \mathrm{Tanh} σ = Tanh will not be monotone. R e L U \mathrm{ReLU} ReLU will be monotone, but to be weakly MAS, two layers are required. T R I \mathrm{TRI} TRI and more general hat functions are weakly MAS even with a single layer.
Proposition 5.3.5. If M 2 M_2 M 2 is strictly monotone increasing, and σ \sigma σ is a hat activation function, then M 2 ( ∑ x ∈ S σ ( a ⊤ x + b ) ) M_2\left(\sum_{x \in S}\sigma(a^\top x+b)\right) M 2 ( ∑ x ∈ S σ ( a ⊤ x + b ) ) is a weakly MAS function.
Proof idea If S S S is not a subset of T T T , there is an element s s s whose multiplicity in S S S is larger than its multiplicity in T T T . We can then choose a , b a,b a , b so that s s s is in the support of σ ( a x + b ) \sigma(ax+b) σ ( a x + b ) but the elements of T T T are not.
Slightly larger ReLU networks
So far, we have considered simple models as in (5.2) which apply a single linear layer before applying activation and summation. In this case, we saw that ReLU and other activations do not create weakly MAS functions, and defined the notion of hat activations, which are weakly MAS functions. But what happens if we use ReLU activations but allow deeper networks?
We can use our previous analysis to show that very small ReLU networks with two layers can already be MAS functions. To see this, we consider a candidate from the hat function class, namely the TRI function from Figure 5.1 , and observe that: TRI ( x ) = ReLU ( ReLU ( 2 x ) + ReLU ( 2 x − 2 ) − ReLU ( 4 x − 2 ) ) \operatorname{TRI}(x)=\operatorname{ReLU}(\operatorname{ReLU}(2x)+\operatorname{ReLU}(2x-2)-\operatorname{ReLU}(4x-2)) TRI ( x ) = ReLU ( ReLU ( 2 x ) + ReLU ( 2 x − 2 ) − ReLU ( 4 x − 2 )) We use this to deduce the following result:
Proposition 5.3.6. Given V ⊆ R d V \subseteq\mathbb{R}^{d} V ⊆ R d , we consider affine transformations A 2 : R d → R 3 , A 1 : R 3 → R \mathcal{A}_2:\mathbb{R}^{d}\to\mathbb{R}^{3},\mathcal{A}_1:\mathbb{R}^{3}\to\mathbb{R} A 2 : R d → R 3 , A 1 : R 3 → R . Thus, here A 2 ( t ) : = A 2 t + b 2 \mathcal{A}_2(t):=A_2t+b_2 A 2 ( t ) := A 2 t + b 2 and A 1 ( z ) = a 1 T z + b 1 \mathcal{A}_1(z)=a_1^{\mathsf{T}}z+b_1 A 1 ( z ) = a 1 T z + b 1 , where A 2 ∈ R 3 × d , b 2 ∈ R 3 , a 1 ∈ R 3 , b 1 ∈ R A_2\in\mathbb{R}^{3\times d},b_2\in\mathbb{R}^{3},a_1\in\mathbb{R}^{3},b_1\in\mathbb{R} A 2 ∈ R 3 × d , b 2 ∈ R 3 , a 1 ∈ R 3 , b 1 ∈ R are the respective parameters. Then, the set functions of the form
F : P < ∞ ( V ) → R , F ( S ; A 1 , A 2 ) = M 2 ( ∑ x ∈ S ReLU ∘ A 1 ∘ ReLU ∘ A 2 ( x ) ) (46) F:\mathcal{P}_{<\infty}(V)\to\mathbb{R},\quad F(S;\mathcal{A}_1,\mathcal{A}_2)=M_2\left(\sum_{x\in S}\operatorname{ReLU}\circ\mathcal{A}_1\circ\operatorname{ReLU}\circ\mathcal{A}_2(x)\right) \tag*{(46)} F : P < ∞ ( V ) → R , F ( S ; A 1 , A 2 ) = M 2 ( x ∈ S ∑ ReLU ∘ A 1 ∘ ReLU ∘ A 2 ( x ) ) (46) are weakly MAS functions.
Our results so far are summarized in Figure 5.1 . If σ \sigma σ is not a non-negative function, then F F F will not be monotone. We can attain weakly MAS functions with a two-layer ReLU network, or with a one-layer network with hat activations.
Asymmetric distance induced Hölder separability
Why weak separability is not sufficient While weak separability is a key relaxation for weakly MAS functions (Definition 5.3.1), it has two key limitations: (1) Separation should ideally hold over a non-negligible subset of W \mathcal{W} W , not just a single point w w w ; (2) it treats separability as a boolean condition, ignoring approximate containment. We would like to guarantee that if S S S is almost a subset of T T T , then F ( S , w ) F(S,w) F ( S , w ) will be close to dominated by F ( T , w ) F(T,w) F ( T , w ) . This motivates us to create an asymmetric (pseudo) metric to quantify the size of set difference and create stronger notions of separability based on that.
Asymmetric set distance For a ground set V ⊆ R d V\subseteq\mathbb{R}^{d} V ⊆ R d and two sets S , T ∈ P < ∞ ( V ) S,T\in\mathcal{P}_{<\infty}(V) S , T ∈ P < ∞ ( V ) such that ∣ S ∣ ≤ ∣ T ∣ |S|\leq|T| ∣ S ∣ ≤ ∣ T ∣ . We define the asymmetric distance from S S S to T T T as the Earth Mover Distance (EMD) between S S S and the subset of T T T that is closest to S S S with the same cardinality as S S S .
Δ ( S , T ) = min T ′ ⊆ T ∣ S ∣ = ∣ T ′ ∣ EMD ( S , T ′ ) (47) \Delta(S,T)=\min_{\substack{T'\subseteq T\\|S|=|T'|}}\operatorname{EMD}(S,T') \tag*{(47)} Δ ( S , T ) = T ′ ⊆ T ∣ S ∣ = ∣ T ′ ∣ min EMD ( S , T ′ ) (47)
Δ ( ∙ , ∙ ) \Delta(\bullet,\bullet) Δ ( ∙ , ∙ ) captures graded set containment: if S ⊈ T S\not\subseteq T S ⊆ T , but is ”very close” to a subset of T T T in EMD metric, then Δ ( S , T ) \Delta(S,T) Δ ( S , T ) is small. Moreover, Δ ( S , T ) = 0 ⟺ S ⊆ T \Delta(S,T)=0\Longleftrightarrow S\subseteq T Δ ( S , T ) = 0 ⟺ S ⊆ T . Based on Δ \Delta Δ , we propose a newer separability notion for parametric functions.
Lower Hölder separability of set functions Davidson and Dym [13 ] introduced Hölder continuity through expectation over parameters. Unlike their symmetric EMD, we define Hölder separability using the asymmetric distance Δ ( 5.4 ) \Delta(5.4) Δ ( 5.4 ) .
Definition 5.3.7 (Lower Hölder separability). Let V ⊆ R d V \subseteq\mathbb{R}^{d} V ⊆ R d be the ground set and ( W , B , μ ) (\mathcal{W},\mathcal{B},\mu) ( W , B , μ ) be a probability space, and a constant λ > 0 \lambda> 0 λ > 0 . A parametric set function F ( ⋅ , w ) : P ≤ k ( V ) → R m F(\cdot,w):\mathcal{P}_{\leq k}(V) \to\mathbb{R}^{m} F ( ⋅ , w ) : P ≤ k ( V ) → R m with w ∼ μ ( ⋅ ) w \sim\mu(\cdot) w ∼ μ ( ⋅ ) is λ \lambda λ lower Hölder separable if there exists c > 0 c > 0 c > 0 such that:
E w ∼ μ ( ⋅ ) [ ∥ F ( S ; w ) − F ( T ; w ) ∥ + 1 ] ≥ c ⋅ Δ ( S , T ) λ , for all S , T ∈ P ≤ k ( V ) (48) \mathbb{E}_{w\sim\mu(\cdot)}\left[\lVert F(S;w)-F(T;w)\rVert_{+}^{1}\right] \geq c\cdot\Delta(S,T)^{\lambda},\quad\text{for all }S,T\in\mathcal{P}_{\leq k}(V) \tag*{(48)} E w ∼ μ ( ⋅ ) [ ∥ F ( S ; w ) − F ( T ; w ) ∥ + 1 ] ≥ c ⋅ Δ ( S , T ) λ , for all S , T ∈ P ≤ k ( V ) (48)
Hölder separable Set functions
Hölder separability using Hat activation Proposition 5.3.5 shows how to construct weakly MAS functions using Hat activations. We now show that these constructions are also lower Hölder, under additional weak assumptions on the hat functions and M 2 M_{2} M 2 :
Theorem 5.3.8 (F F F is lower Hölder). Let V ⊂ R d V \subset\mathbb{R}^{d} V ⊂ R d be a compact set, and σ \sigma σ a Hat activation function which is piecewise continuously differentiable supported in some interval [ γ 1 , γ 2 ] [\gamma_{1},\gamma_{2}] [ γ 1 , γ 2 ] , and satisfying the condition: lim t → γ 1 + d σ d t > 0 \lim_{t\to\gamma_{1}^{+}}\frac{d\sigma}{dt}>0 lim t → γ 1 + d t d σ > 0 . Let M 2 : R → R M_{2}:\mathbb{R}\to\mathbb{R} M 2 : R → R be a lower Lipschitz function. Consider the function F ( S ; ( a , b , c ) ) = M 2 ( ∑ x ∈ S σ ( a ⊤ x + b c ) ) F(S;(a,b,c))=M_{2}\left(\sum_{x\in S}\sigma\left(\frac{a^{\top}x+b}{c}\right)\right) F ( S ; ( a , b , c )) = M 2 ( ∑ x ∈ S σ ( c a ⊤ x + b ) ) , where the multisets S S S come from P ≤ k ( V ) \mathcal{P}_{\leq k}(V) P ≤ k ( V ) , and a ∼ Unif ( S d − 1 ) a\sim\operatorname{Unif}(\mathbb{S}^{d-1}) a ∼ Unif ( S d − 1 ) , b ∼ Unif ( [ − 1 , 1 ] ) b\sim\operatorname{Unif}([-1,1]) b ∼ Unif ([ − 1 , 1 ]) , c ∼ Unif ( ( 0 , 2 ] ) c\sim\operatorname{Unif}((0,2]) c ∼ Unif (( 0 , 2 ]) . Then, F ( ⋅ , ( a , b , c ) ) F(\cdot,(a,b,c)) F ( ⋅ , ( a , b , c )) is Monotone Hölder separable with exponent λ = 2 \lambda=2 λ = 2 .
Probability of successful separation Given S , T ⊆ V S,T\subseteq V S , T ⊆ V , we can consider [ F ( S ; ( a , b , c ) ) − F ( T ; ( a , b , c ) ) ] + [F(S;(a,b,c))-F(T;(a,b,c))]_{+} [ F ( S ; ( a , b , c )) − F ( T ; ( a , b , c )) ] + to be a real-valued non-negative random variable, where the randomness is over the parameter space ( a , b , c ) ∈ W = R d × R × R (a,b,c)\in\mathcal{W}=\mathbb{R}^{d}\times\mathbb{R}\times\mathbb{R} ( a , b , c ) ∈ W = R d × R × R equipped with a probability measure. The above result on lower Hölder stability gives us a lower bound on the probability of a parameter tuple ( a , b , c ) (a,b,c) ( a , b , c ) separating two non-subsets S , T S,T S , T proportional to the set distance Δ ( S , T ) \Delta(S,T) Δ ( S , T ) . Moreover, the functions we have considered so far are scalar-valued set functions. By considering independent copies of the parameters across multiple dimensions, we can increase the probability of separation, since F ( S ) , F ( T ) F(S),F(T) F ( S ) , F ( T ) are not separated iff F ( S ) [ i ] ≤ F ( T ) [ i ] F(S)[i]\leq F(T)[i] F ( S ) [ i ] ≤ F ( T ) [ i ] across each embedding dimension i i i . We formalize the above observations as follows:
Theorem 5.3.9 (Probability bounds on separation). Let V ⊆ R d V\subseteq\mathbb{R}^{d} V ⊆ R d and σ \sigma σ be as in Theorem 5.3.8, and let A ∈ R m × d A\in\mathbb{R}^{m\times d} A ∈ R m × d , b ∈ R m b\in\mathbb{R}^{m} b ∈ R m , c ∈ R m c\in\mathbb{R}^{m} c ∈ R m whose m m m columns (respectively entries) are drawn independently from the distribution on a j , b j , c j a_j,b_j,c_j a j , b j , c j described in Theorem 5.3.8, and consider the function
F ( S ; A , b , c ) = ∑ x ∈ S σ ( c − 1 ⊙ ( A x + b ) ) . (49) F(S;A,b,c)=\sum_{x\in S}\sigma\left(c^{-1}\odot(Ax+b)\right). \tag*{(49)} F ( S ; A , b , c ) = x ∈ S ∑ σ ( c − 1 ⊙ ( A x + b ) ) . (49) Then there exists C > 0 C>0 C > 0 , so that for all S ⊈ T S\not\subseteq T S ⊆ T , P ( F ( S ) ≤ F ( T ) ) ≤ ( 1 − C Δ 2 ( S , T ) ) m \mathbb{P}(F(S)\leq F(T))\leq\left(1-C\Delta^{2}(S,T)\right)^{m} P ( F ( S ) ≤ F ( T )) ≤ ( 1 − C Δ 2 ( S , T ) ) m .
In Equation (5.6) c − 1 c^{-1} c − 1 stands for the elementwise inverse and ⊙ \odot ⊙ stands for elementwise multiplication.
Note that the probability of failed separation goes to zero exponentially as m m m increases, and also becomes smaller as Δ ( S , T ) \Delta(S,T) Δ ( S , T ) increases. This supports the idea that a larger measure of parameters separates sets with larger asymmetric distance and that separability becomes easier with larger embedding dimensions (which we saw previously in the existential results).
Hölder separable set functions using ReLU As seen in Proposition 5.3.6, one can construct weakly MAS functions with a two layer neural network with ReLU activation. Our proof used that two layer ReLU networks can basically ”simulate” a Hat function. Under the light of the above results on Hölder separability, one can argue that two-layer ReLU networks would also have such guarantees. Also, we show in Appendix B that under certain structure on the ground set V V V (for example, V V V being a hypersphere), even with one layer ReLU networks, functions of the form in Equation (45) will also have Lower Hölder separability guarantees.
Upper Lipschitz bounds In Appendix. B we complement our results on lower Holder stability by showing that the construction in Theorem 5.3.9 is also upper Lipschitz in an appropriate sense.
Monotone Universality and the role of M 2 M_2 M 2
Besides set containment-based applications, there are other interesting scenarios where one would like to construct monotone multiset-to-vector functions. The following theorem shows that on finite ground sets, the combination of a MAS function and a universal monotone vector-to-vector (such as [32 ] ) can approximate all monotone multiset-to-vector functions.
Theorem 5.3.10 (Universality). Let V V V be a finite ground set, and let F : P ≤ k ( V ) → R m F : \mathcal{P}_{\le k}(V) \to\mathbb{R}^m F : P ≤ k ( V ) → R m be a MAS function. Then for every multiset-to-vector monotone function f : P ≤ k ( V ) → R s f : \mathcal{P}_{\le k}(V) \to\mathbb{R}^s f : P ≤ k ( V ) → R s , there exists a vector-to-vector monotone function M : R m → R s M : \mathbb{R}^m \to\mathbb{R}^s M : R m → R s such that F ( S ) = M ∘ f ( S ) F(S) = M \circ f(S) F ( S ) = M ∘ f ( S ) .
MASNet: Neural Modeling of MAS functions
Now, our goal is to leverage our theoretical analysis to design neural network-based multiset-to-vector models that preserve monotonicity and weak separability. Motivated by the formulation in DeepSets and our analysis so far, we wish to design neural set functions of the form:
MASNet ( S ) = M θ 2 ( ∑ x ∈ S σ ( M θ 1 ( x ) ) ) (50) \operatorname{MASNet}(S) = M_{\theta_2}\left(\sum_{x \in S} \sigma\left(M_{\theta_1}(x)\right)\right) \tag*{(50)} MASNet ( S ) = M θ 2 ( x ∈ S ∑ σ ( M θ 1 ( x ) ) ) (50)
In our analysis, we required M θ 2 M_{\theta_2} M θ 2 to be a monotone vector-to-vector function. In most of our experiments, we enforce this simply by choosing M 2 ( x ) = x M_2(x) = x M 2 ( x ) = x , but this can also be enforced by using monotone activations and non-negative parameters.
MASNet-Hat To enforce weakly MAS in Equation (50) , our results from Prop. 5.3.5 and Theorem 5.3.8 suggest using a Hat activation as σ \sigma σ . In our experiment, we do it by not choosing σ \sigma σ to be a specific hat activation, but rather the parametric form:
σ α , β , γ ( x ) = ReLU ( x − α γ ⋅ β ) + ReLU ( x − ( α + β ) ( 1 − γ ) ⋅ β ) − ReLU ( x − ( α + γ ⋅ β ) γ ⋅ ( 1 − γ ) ⋅ β ) (51) \sigma_{\alpha,\beta,\gamma}(x) = \operatorname{ReLU}\left(\frac{x-\alpha}{\gamma\cdot\beta}\right) + \operatorname{ReLU}\left(\frac{x-(\alpha+\beta)}{(1-\gamma)\cdot\beta}\right) - \operatorname{ReLU}\left(\frac{x-(\alpha+\gamma\cdot\beta)}{\gamma\cdot(1-\gamma)\cdot\beta}\right) \tag*{(51)} σ α , β , γ ( x ) = ReLU ( γ ⋅ β x − α ) + ReLU ( ( 1 − γ ) ⋅ β x − ( α + β ) ) − ReLU ( γ ⋅ ( 1 − γ ) ⋅ β x − ( α + γ ⋅ β ) ) (51)
For all α , β > 0 \alpha, \beta> 0 α , β > 0 , and γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , σ α , β , γ \sigma_{\alpha,\beta,\gamma} σ α , β , γ is a hat function (Definition 5.3.4 ) with support [ α , α + β ] [\alpha, \alpha+\beta] [ α , α + β ] and peak at α + m β \alpha+m\beta α + m β . Examples include the third and fourth functions in Figure 5.1 . Applying this to an m m m -dimensional output with independent α , β , γ \alpha,\beta,\gamma α , β , γ per dimension yields 3 m 3m 3 m parameters total.
MASNet with ReLU Motivated by our theoretical results from Theorem 5.3.6 of 2-Layer ReLU networks being weakly MAS and from our discussion of ReLU networks being Hölder separable (under appropriate assumptions) in § 8.5.2 , we propose using ReLU in MASNet, which case M θ 1 M_{\theta_1} M θ 1 will have to be a network with at least 2 layers to ensure weakly MAS as in Theorem 5.3.6. This gives one more way to design MASNet and we refer to this option as MASNet-ReLU.
Other variants of MASNet Definition 5.3.4 establishes a general hat function class extending beyond the piecewise linear functions as in (51) . We present MASNet-INT in Appendix C, which achieves universal approximation of the Hat class by modeling its derivative via neural networks and approximating the integral. Additionally, using the specific hat activation TRI (Figure 5.1 ) as σ \sigma σ yields an alternative formulation called MASNet-TRI.
Which MASNet to choose and when? We have given multiple recipes of MASNet in the above discussion, and the question arises: which one to choose? We show via our experiments: For parameter-constrained scenarios requiring single-layer M θ 1 M_{\theta_1} M θ 1 networks, MASNet-Hat provides optimal separability (both theoretically and empirically). For deeper M θ 1 M_{\theta_1} M θ 1 (≥ 2 \ge2 ≥ 2 layers), both MASNet-ReLU and Hat-based variants perform similarly, with MASNet-ReLU often providing more stable training and better performance overall.
Experiments
We evaluate the MASNet variants (§ 8.6 ) on synthetic, text and point-cloud datasets to characterize monotonicity and separability. Specifically, we focus on (exact and approximate) set containment. Given sets S S S and T T T , with binary label y ( S , T ) ∈ { 0 , 1 } y(S,T) \in\{0,1\} y ( S , T ) ∈ { 0 , 1 } indicating (exact or approximate) set containment, we evaluate how accurately MASNet predicts y ( S , T ) y(S,T) y ( S , T ) .
Set Containment
We perform experiments on a synthetically generated dataset, four text datasets, and one image dataset. In each case, we split the dataset into 5:2:2 train, test, and dev folds. We minimize the following fixed-margin hinge loss that enforces vector dominance, to train the parameters of MASNet.
∑ S , T ( 1 − y ( S , T ) ) min i ∈ [ m ] [ F ( S ) [ i ] − F ( T ) [ i ] + δ ] + + y ( S , T ) max i ∈ [ m ] [ F ( S ) [ i ] − F ( T ) [ i ] + δ ] + (52) \sum_{S,T}(1-y(S,T))\min_{i\in[m]}[F(S)[i]-F(T)[i]+\delta]_{+}+y(S,T)\max_{i\in[m]}[F(S)[i]-F(T)[i]+\delta]_{+} \tag*{(52)} S , T ∑ ( 1 − y ( S , T )) i ∈ [ m ] min [ F ( S ) [ i ] − F ( T ) [ i ] + δ ] + + y ( S , T ) i ∈ [ m ] max [ F ( S ) [ i ] − F ( T ) [ i ] + δ ] + (52)
Analysis on synthetic datasets To generate our synthetic dataset, we first sample a target set T ⊂ R d T \subset\mathbb{R}^{d} T ⊂ R d , where each element x ∈ T x \in T x ∈ T is drawn from N ( 0 , I ) \mathcal{N}(0,\mathbf{I}) N ( 0 , I ) , i.i.d. Given T T T , we compute S S S with a fixed size ∣ S ∣ = s |S|=s ∣ S ∣ = s as follows. We first obtain S S S with positive labels y ( S , T ) = 1 y(S,T)=1 y ( S , T ) = 1 by drawing a subset from T T T , uniformly at random, without replacement. To generate S S S with y ( S , T ) = 0 y(S,T)=0 y ( S , T ) = 0 , we sample s s s from N ( 0 , I ) \mathcal{N}(0,\mathbf{I}) N ( 0 , I ) independently. We chose d = 4 d=4 d = 4 and the set embeddings have m = 256 m=256 m = 256 dimensions. The class-ratio of subsets to non-subsets was taken to be 1:1.
We compare two variants of MASNet—MASNet-ReLU (M-ReLU) and MASNet-Hat (M-Hat)—against Deep Sets (DS) [ZKR+17] and Set Transformer (ST) [23 ] . Table 5.2 summarizes the results for varying values of ∣ S ∣ |S| ∣ S ∣ and ∣ T ∣ |T| ∣ T ∣ . We observe that: (1) Both variants of our method outperform the baselines; (2) the baselines degrade significantly as ∣ T ∣ |T| ∣ T ∣ increases (with fixed ∣ S ∣ |S| ∣ S ∣ ), as predicting separability becomes harder as the gap between ∣ T ∣ |T| ∣ T ∣ and ∣ S ∣ |S| ∣ S ∣ increases; (3) The performance of M-ReLU and M-Hat is comparable. We now compare 1-layer pointwise models ending in ReLU and Hat activation respectively, followed by aggregation: we call them 1layer-ReLU and 1layerM-Hat. We see in Figure 5.2 that: (1) 1layerM-Hat performs significantly better than 1layer-ReLU, validating the results from Propn. 5.3.2. (2) acc. declines for both as ∣ T ∣ |T| ∣ T ∣ increases, consistent with the intuition that larger ∣ T ∖ S ∣ |T\setminus S| ∣ T ∖ S ∣ gaps make the task harder.
Table 5.2. Accuracy on the synthetic dataset.
Figure 5.2. Acc vs ∣ T ∣ |T| ∣ T ∣ for 1-layer MLP
Set containment on Text datasets We evaluate on MSWEB, MSNBC, and Amazon Registry datasets, which exhibit natural set containment from user behavior. Each datapoint is a wordbag; we compute BERT embeddings for all unique words to form the ground set. ( S , T ) (S,T) ( S , T ) pairs are sampled from these (Appendix D), and labeled via set containment. For inexact containment, small
Table 5.3. Accuracy on text datasets with a 1:1 class ratio.
Gaussian noise is added to S S S without changing y ( S , T ) y(S,T) y ( S , T ) . We now compare the variants of MASNet: MASNet-ReLU and MASNet-Hat against several baselines, DeepSets [ZKR+17], Set Transformers [23 ] , FlexSubNet [12 ] , Neural SFE [22 ] , for inexact set containment on the datasets: MSWEB, MSNBC, Amazon-Feeding, Amazon-Bedding. Gaussian noise with 0.01 std was used to generate noisy pairs. The class ratio of subsets to non-subsets was taken to be 1:1. Here, set embeddings have dim m = 50 m = 50 m = 50 , and set elements have dim d = 768 d = 768 d = 768 . The results in table 5.3 show: (1) MASNet-ReLU and MASNet-Hat have similar acc., with MASNet-ReLU being marginally better. (2) Both are generally better than baselines.
Set containment on point clouds We use ModelNet40 [WSK+15], a dataset of 3D CAD models across
40 categories. For each object in category C 1 C_1 C 1 , we sample 1024 points to form T T T . For true subsets, we sub-sample S ⊆ T S \subseteq T S ⊆ T and for non-subsets, S S S is sampled from a different category C 2 C_2 C 2 . Small white noise(std=0.005) is added to S S S for the task of inexact containment. We use a pointcloud encoder(such as pointnet) followed by a set-to-vector embedding model to produce m = 50 m = 50 m = 50 dimensional embeddings, and each point in the dataset is a 3D coordinate with d = 3 d = 3 d = 3 dimensions. The class ratio of subsets to non-subsets was taken to be 1:1.
Table 5.4. Point-cloud accuracy for different set sizes ∣ S ∣ |S| ∣ S ∣ , using a 1:1 class ratio.
Results are in table 5.4. We observe that: (1) MASNet-ReLU and MASNet-Hat are generally better than baselines, with MASNet-ReLU being slightly better. (2) The accuracy of MASNet decreases as ∣ S ∣ |S| ∣ S ∣ decreases, with the separability becoming hard for monotone models.
Monotone Set function approximation
In this experiment, we examine if a scalar variant of MASNet can approximate a monotone function F ∗ F^* F ∗ , by getting trained from set, value pairs in the form of { ( S , F ∗ ( S ) ) } \{(S, F^*(S))\} {( S , F ∗ ( S ))} . Once trained, we measure ∣ M A S ( S ) − F ∗ ( S ) ∣ |MAS(S) - F^*(S)| ∣ M A S ( S ) − F ∗ ( S ) ∣ for each S S S and average over them to compute MAE. We compare MASNet against baselines: DeepSets, Set Transformer. We trained using MSE loss, and report test MAE results in Table 5.5 where we see: (1) MASNet-Hat is best performing, followed by MASNet-ReLU. (2) Both are better than baselines.
Table 5.5. Monotone function approximation.
Conclusion and Future Work
In this work, we study the design of set functions that are useful in set containment through Monotone and Separating (MAS) properties. We derive bounds on the embedding dimensions required for MAS functions and show their nonexistence in infinite domains. To address this, we introduce a relaxed model, MASNet, which satisfies a weak MAS property and is provably stable. Experiments demonstrate that MASNet outperforms standard models on set containment tasks by leveraging monotonicity as an inductive bias. It would be interesting to check if our method can be used for applications in graphs, e.g. , subgraph matching. Another potential direction is to consider other possible relaxations of separability.
Limitations
Our work provides a theoretical analysis of monotonicity and separability for set functions in the context of set containment, and introduces neural models that outperform standard set-based architectures such as DeepSets and Set Transformers on set containment tasks. Our work has the following limitations, addressing which are interesting directions for future work:
After proving the impossibility of constructing MAS functions over infinite ground sets, we proposed parametric set functions defined with respect to a probability measure over the parameter space. We provided probabilistic guarantees: for non-subsets S , T S, T S , T , if their asymmetric distance Δ ( S , T ) \Delta(S, T) Δ ( S , T ) is large, then a randomly sampled function from the parameter space will separate them with high probability. However, these guarantees hold only in the randomized setting—analogous to guarantees for neural networks at random initialization—and do not extend to models after training via gradient-based optimization.
Our universality result, which shows that MAS functions composed with monotone vector-to-vector mappings can approximate all monotone set-to-vector func-
tions, was only established for finite ground sets. Extending this result to infinite ground sets remains an open problem.
;
Appendix
This work lies at the intersection of representation learning for sets and multisets, monotone and submodular set functions, and theoretical limits of learning under constraints on some partial order. Below we review related literature across several communities, such as machine learning, optimization, theoretical computer science, fuzzy measures, and physics. We highlight how these threads connect to our problem setting.
Multiset to vector functions, and permutation invariant functions (including symmetric and anti-symmetric polynomials) have been used in a bunch of real life modelling tasks. Anti symmetric polynomials in the context of quantum many-body physics were used in [HLL23]. For point cloud datasets, such set-to-vector embeddings are widely used such as in [29 ] . In the context of search and retrieval, set-containment was directly used in [30 ] . On a somewhat related aspect, [19 ] deals with monotone ranking functions on a vector. This is specifically a monotone function from multisets to reals if the vector P P P denotes the weight/frequency of each object in the set. Moreover, since this is a vector-to-real monotone function, composing a set-to-vector monotone function with a monotonic ranking function preserves monotonicity. These are relevant to the recent usage of set containment in the context of information retrieval. Another related usage is in vector search [16 ] , where relevance based retrieval is done after converting objects into high-dimensional embeddings.
More specifically monotone (multi)set to vector functions have been of interest in the form of theory of capacities [11 ] which is a probability measure based on monotonicity. In fuzzy set theory, where partial membership is allowed, any set-to-vector monotone function is equivalent to a vector-to-vector monotone function. Measures and integrals on such structures have been explored in the form of Choquet integrals [3 CD84] etc. In the context of game theory and economics, monotone and submodular functions are used to model diminishing marginal utilities and in combinatorial auctions, such as in [24 , 15 , 18 ] . In learning theory, understanding PAC learning and hardness of vector-to-vector monotone functions have always received a lot of interest. And using the membership vector of multisets, this is basically equivalent to learning multiset-to-vector monotone functions. Some works such as [7 ] uses set containment and monotone set functions as a tool in the proof. Specifically, they perform a reduction of neural network training to the set splitting problem to prove the hardness result. This is the task of partitioning a set S = S 1 ∪ S 2 S = S_1 \cup S_2 S = S 1 ∪ S 2 such that for a collection of subsets C = { C i : C i ⊆ S , ∀ i } \mathcal{C} = \{C_i : C_i \subseteq S, \forall i\} C = { C i : C i ⊆ S , ∀ i } we must have C i ⊈ S 1 C_i \not\subseteq S_1 C i ⊆ S 1 and C i ⊈ S 2 C_i \not\subseteq S_2 C i ⊆ S 2 , ∀ i \forall i ∀ i . This is basically preventing set containment for a subset family. More direct applications include [39 ] which discusses the challenges of learning vector to vector monotonic functions or [5 ] that discusses learning monotone valuation functions. Works like [8 , 28 ] deal with learning monotone boolean functions to any constant accuracy, under the uniform distribution in polynomial time. Since any boolean vector is basically one-hot encoding of a set on a fixed size vocabulary, this paper essentially describes learning monotone set functions. Works on monotone learning such as [26 ] describe learning processes in which expected performance consistently improves as the amount of training data increases. Thus, a trained model is a monotone set function from the training dataset to reals, where the output is a valuation metric.
So far, we have discussed works that have used multiset to vector embeddings and analyzed them. We have taken special care to discuss works that have involved the structure of monotonicity and applied it to various domains. However, to the best of our knowledge, there has been no previous work that specifically deals with partial order preserving set-to-vector maps. One related work is [14 ] , which has studied the order dimension of a partially ordered structure. Now, note that partial order preserving sets are injective as discussed before, and injectivity is in fact a weaker condition that the MAS property. Here, we discuss a number of works on injective functions on multisets. The paper [2 ] shows that moments of neural networks do define injective multiset functions, provided that an analytic non-polynomial activation is used. They also state and prove a finite witness theorem, which is of independent interest. There are a series of works such as [1 , 6 , 31 ] that design bi-Lipschitz(hence, injective) embeddings on sets and graphs. On the other hand, works like [33 , 37 ] investigate the impact of latent dimension on the expressive power of set functions such as DeepSets.
Proofs of the technical results
Proofs of existential results on MAS functions
Theorem 5.2.2 (Dimension of MAS Function). For a finite ground set V V V of size n n n , there exists a MAS function F : P < ∞ ( V ) → R n F:\mathcal{P}_{<\infty}(V)\to\mathbb{R}^{n} F : P < ∞ ( V ) → R n . In addition, any MAS function must have a dimension of at least n n n . In other words, m ⋆ ( V , ∞ ) = n m^{\star}(V,\infty)=n m ⋆ ( V , ∞ ) = n .
Proof. The construction of a monotone embedding with n = ∣ V ∣ n=|V| n = ∣ V ∣ simply uses one-hot encoding. Namely, we identify V V V with [ n ] [n] [ n ] . For every S ∈ P < ∞ ( V ) S\in\mathcal{P}_{<\infty}(V) S ∈ P < ∞ ( V ) , we define
F ( S ) = ∑ s ∈ S e s ∈ R n (53) F(S)=\sum_{s\in S}e_s\in\mathbb{R}^{n} \tag*{(53)} F ( S ) = s ∈ S ∑ e s ∈ R n (53)
where e s ∈ R n e_s\in\mathbb{R}^{n} e s ∈ R n is the vector with e s [ s ] = 1 e_s[s]=1 e s [ s ] = 1 and e s [ j ] = 0 e_s[j]=0 e s [ j ] = 0 for all j ≠ s j\ne s j = s . We now show it satisfies all conditions: Let S ⊆ T S\subseteq T S ⊆ T . Then, by definition, ∀ v ∈ V , c S ( v ) ≤ c T ( v ) \forall v\in V,c_S(v)\le c_T(v) ∀ v ∈ V , c S ( v ) ≤ c T ( v ) , then F ( S ) = ∑ s ∈ S e s ≤ ∑ t ∈ T e t = F ( T ) F(S)=\sum_{s\in S}e_s\le\sum_{t\in T}e_t=F(T) F ( S ) = ∑ s ∈ S e s ≤ ∑ t ∈ T e t = F ( T ) thus F ( S ) ≤ F ( T ) F(S)\le F(T) F ( S ) ≤ F ( T ) . On the other hand, if F ( S ) ≤ F ( T ) F(S)\le F(T) F ( S ) ≤ F ( T ) then ∀ v ∈ V , c S ( v ) ≤ c T ( v ) \forall v\in V,c_S(v)\le c_T(v) ∀ v ∈ V , c S ( v ) ≤ c T ( v ) , and thus S ⊆ T S\subseteq T S ⊆ T .
For the second direction, assume there exists MAS F : V → R m F:V\to\mathbb{R}^{m} F : V → R m for m ≤ n − 1 m\le n-1 m ≤ n − 1 . For every output dimension i ∈ [ m ] i\in[m] i ∈ [ m ] , there is a ”maximal singleton element” v i ⋆ ∈ V v_i^{\star}\in V v i ⋆ ∈ V such that F ( { v i ⋆ } ) [ i ] ≥ F ( { v } ) [ i ] , ∀ v ∈ V F(\{v_i^{\star}\})[i]\ge F(\{v\})[i],\forall v\in V F ({ v i ⋆ }) [ i ] ≥ F ({ v }) [ i ] , ∀ v ∈ V . The value of the set function over the union of such v i ⋆ v_i^{\star} v i ⋆ -s across i ∈ [ m ] i\in[m] i ∈ [ m ] will dominate any singleton that is disjoint from this collection, thanks to the monotonicity of F F F . As m < ∣ V ∣ m<|V| m < ∣ V ∣ , then T : = ⋃ i = 1 m { v i ⋆ } T:=\bigcup_{i=1}^{m}\{v_i^{\star}\} T := ⋃ i = 1 m { v i ⋆ } does not cover V V V . Thus, we can select an element disjoint from this collection named s ∈ V s\in V s ∈ V , which gives disjoint sets S = { s } , T S=\{s\},T S = { s } , T with F ( S ) ≤ F ( T ) F(S)\le F(T) F ( S ) ≤ F ( T ) , which contradicts separability. □ \square □
Theorem 5.2.3 (Upper Bound on m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) ). Let k < n k<n k < n be natural numbers, and let V V V be a ground set with ∣ V ∣ = n |V|=n ∣ V ∣ = n . Then there exists a MAS function F : P ≤ k ( V ) → R m F:\mathcal{P}_{\le k}(V)\to\mathbb{R}^{m} F : P ≤ k ( V ) → R m with embedding dimension m = ( k + 2 ) k + 2 log ( n ) m=(k+2)^{k+2}\log(n) m = ( k + 2 ) k + 2 log ( n ) .
Proof. For convenience of notation, in this proof we will assume without loss of generality that V = [ n ] V=[n] V = [ n ] . As discussed in the main text, the function F a 1 , … , a m F_{a_1,\ldots,a_m} F a 1 , … , a m will be weakly monotone for any choice of non-negative vectors. Our goal is to show that there exists a set of parameters a 1 , … , a m ∈ [ 0 , 1 ] n a_1,\ldots,a_m\in[0,1]^n a 1 , … , a m ∈ [ 0 , 1 ] n , such that the obtained F a 1 , … , a m F_{a_1,\ldots,a_m} F a 1 , … , a m is a monotone embedding, which means that, if S ⊈ T S\not\subseteq T S ⊆ T , then there exists some j j j such that h j ( S ) > h j ( T ) h_j(S)>h_j(T) h j ( S ) > h j ( T ) , or in other words ⟨ f ( S ) , a j ⟩ > ⟨ f ( T ) , a j ⟩ \langle f(S),a_j\rangle>\langle f(T),a_j\rangle ⟨ f ( S ) , a j ⟩ > ⟨ f ( T ) , a j ⟩ . We note that to prove this, it is sufficient to consider pairs ( S , T ) (S,T) ( S , T ) which we will call extreme pairs . This means that S S S consists of a single element x ∈ V x\in V x ∈ V , and T T T consists of k k k elements in V V V but does not contain x x x (but repetitions are allowed). Indeed, if the claim is true for such pairs, and S , T ∈ P ≤ k ( V ) S,T\in\mathcal{P}_{\le k}(V) S , T ∈ P ≤ k ( V ) such that S ⊄ T S\not\subset T S ⊂ T , then there exists an element x x x which is in S S S more times than it’s in T T T . First, remove all copies of x x x from T T T , and we know that at least one copy of x x x will be in S S S , note that the difference between the function values remains the same. Then, we will ’replace’ S S S with a smaller set S ~ = { x } \widetilde{S}=\{x\} S = { x } , and T ~ \widetilde{T} T from adding more elements to T T T from V ∖ { x } V\setminus\{x\} V ∖ { x } to obtain a new set T ~ \widetilde{T} T of maximal cardinality k. We then will have that ( S ~ , T ~ ) (\widetilde{S},\widetilde{T}) ( S , T ) is an extreme pair, and so for some j j j we will have that h j ( S ~ ) > h j ( T ~ ) h_j(\widetilde{S}) > h_j(\widetilde{T}) h j ( S ) > h j ( T ) . Therefore, using the weak monotonicity of the function h j h_j h j , we will have
h j ( S ) ≥ h j ( S ~ ) > h j ( T ~ ) ≥ h j ( T ) . h_j(S) \ge h_j(\widetilde{S}) > h_j(\widetilde{T}) \ge h_j(T). h j ( S ) ≥ h j ( S ) > h j ( T ) ≥ h j ( T ) .
Given an extreme pair ( S , T ) (S,T) ( S , T ) we consider the set of all ’bad’ a a a , namely
B S , T : = { a ∈ [ 0 , 1 ] n ∣ ⟨ f ( S ) , a ⟩ > ⟨ f ( T ) , a ⟩ } . B_{S,T} := \{a \in[0,1]^n \mid\langle f(S),a\rangle> \langle f(T),a\rangle\}. B S , T := { a ∈ [ 0 , 1 ] n ∣ ⟨ f ( S ) , a ⟩ > ⟨ f ( T ) , a ⟩} .
Our first goal will be to bound the Lebesgue measure of this set. Recall that S S S is a singleton S = { s } S = \{s\} S = { s } , and T T T does not contain this singleton. Accordingly,
B S , T = { a ∈ [ 0 , 1 ] n ∣ a s ≤ ∑ t ∈ T a t } . B_{S,T} = \{a \in[0,1]^n \mid a_s \le\sum_{t\in T} a_t\}. B S , T = { a ∈ [ 0 , 1 ] n ∣ a s ≤ t ∈ T ∑ a t } .
We will compute the probability of the complement of this set, namely vectors a a a such that a s > ∑ t ∈ T a t a_s > \sum_{t\in T} a_t a s > ∑ t ∈ T a t . First denote by
E S , T = { a ∈ [ 0 , 1 ] n ∣ ∀ t ∈ T , a t < a s k } . E_{S,T} = \{a \in[0,1]^n \mid\forall t \in T, a_t < \frac{a_s}{k}\}. E S , T = { a ∈ [ 0 , 1 ] n ∣ ∀ t ∈ T , a t < k a s } .
Note, that
E S , T ⊆ { a ∈ [ 0 , 1 ] n ∣ a s > ∑ t ∈ T a t } . E_{S,T} \subseteq\{a \in[0,1]^n \mid a_s > \sum_{t\in T} a_t\}. E S , T ⊆ { a ∈ [ 0 , 1 ] n ∣ a s > t ∈ T ∑ a t } .
So let’s compute the measure of E S , T E_{S,T} E S , T :
Leb ( E S , T ) = Leb ( { a ∈ [ 0 , 1 ] n ∣ ∀ t ∈ T , a t ≤ a s k } ) = ∫ 0 1 ( a s k ) k d a s = 1 k k ⋅ ( k + 1 ) ≥ 1 ( k + 1 ) k + 1 . \operatorname{Leb}(E_{S,T}) = \operatorname{Leb}\left(\{a \in[0,1]^n \mid\forall t \in T, a_t \le\frac{a_s}{k}\}\right) = \int_0^1 \left(\frac{a_s}{k}\right)^k da_s = \frac{1}{k^k \cdot(k+1)} \ge\frac{1}{(k+1)^{k+1}}. Leb ( E S , T ) = Leb ( { a ∈ [ 0 , 1 ] n ∣ ∀ t ∈ T , a t ≤ k a s } ) = ∫ 0 1 ( k a s ) k d a s = k k ⋅ ( k + 1 ) 1 ≥ ( k + 1 ) k + 1 1 .
So,
Leb ( B S , T ) ≤ 1 − Leb ( E S , T ) ≤ 1 − 1 ( k + 1 ) k + 1 . \operatorname{Leb}(B_{S,T}) \le1 - \operatorname{Leb}(E_{S,T}) \le1 - \frac{1}{(k+1)^{k+1}}. Leb ( B S , T ) ≤ 1 − Leb ( E S , T ) ≤ 1 − ( k + 1 ) k + 1 1 .
Now, consider the set
B S , T m = { a 1 , … , a m ∈ B S , T } , B^m_{S,T} = \{a_1,\ldots,a_m \in B_{S,T}\}, B S , T m = { a 1 , … , a m ∈ B S , T } ,
Then we have
Leb ( B S , T m ) ≤ ( 1 − 1 ( k + 1 ) k + 1 ) m \operatorname{Leb}(B^m_{S,T}) \le\left(1-\frac{1}{(k+1)^{k+1}}\right)^m Leb ( B S , T m ) ≤ ( 1 − ( k + 1 ) k + 1 1 ) m
The vectors a 1 , … , a m a_1,\ldots,a_m a 1 , … , a m will not define a monotone embedding, if there exists an extreme pair S , T S,T S , T which is in B S , T m B^m_{S,T} B S , T m . The probability of this happening can be bounded by a union bound. There are n ⋅ ( n − 1 k ) n \cdot\binom{n-1}{k} n ⋅ ( k n − 1 ) extreme pairs, and for simplicity we will replace this number with a larger but simpler expression n k + 2 > n ⋅ ( n − 1 k ) n^{k+2} > n \cdot\binom{n-1}{k} n k + 2 > n ⋅ ( k n − 1 ) . The union bound will then give us
Leb { a 1 , … , a m do not define a monotone embedding } ≤ n k + 2 ( 1 − 1 ( k + 1 ) k + 1 ) m \operatorname{Leb}\{a_1,\ldots,a_m \text{ do not define a monotone embedding}\}\le n^{k+2}\left(1-\frac{1}{(k+1)^{k+1}}\right)^m Leb { a 1 , … , a m do not define a monotone embedding } ≤ n k + 2 ( 1 − ( k + 1 ) k + 1 1 ) m
It is sufficient to show that this expression is smaller than 1, for our m m m , which, by taking a logarithm, is equivalent to requiring that
( k + 2 ) ln ( n ) + m ln ( 1 − 1 ( k + 1 ) k + 1 ) < 0 (k+2)\ln(n)+m\ln\left(1-\frac{1}{(k+1)^{k+1}}\right)<0 ( k + 2 ) ln ( n ) + m ln ( 1 − ( k + 1 ) k + 1 1 ) < 0
Using Taylor’s expansion it can be shown that ln ( 1 − x ) < − x \ln(1-x)<-x ln ( 1 − x ) < − x for x ∈ ( 0 , 1 ) x\in(0,1) x ∈ ( 0 , 1 ) , and as a result it is sufficient to choose m m m so that
( k + 2 ) ln ( n ) − m ( k + 1 ) k + 1 < 0 (k+2)\ln(n)-\frac{m}{(k+1)^{k+1}}<0 ( k + 2 ) ln ( n ) − ( k + 1 ) k + 1 m < 0
or equivalently
m > ( k + 1 ) k + 1 ⋅ ( k + 2 ) ⋅ ln ( n ) . m>(k+1)^{k+1}\cdot(k+2)\cdot\ln(n). m > ( k + 1 ) k + 1 ⋅ ( k + 2 ) ⋅ ln ( n ) .
For convenience we slightly enlarge this lower bound to obtain m ≥ ( k + 2 ) k + 2 ln ( n ) m\ge(k+2)^{k+2}\ln(n) m ≥ ( k + 2 ) k + 2 ln ( n )
Theorem 5.2.4 (Lower bounds on m ⋆ ( V , k ) m^\star(V,k) m ⋆ ( V , k ) ). Let k ≥ 2 k\ge2 k ≥ 2 and n n n be natural numbers, and let V V V be a ground set with ∣ V ∣ = n |V|=n ∣ V ∣ = n . Then the smallest possible dimension m ⋆ ( V , k ) m^\star(V,k) m ⋆ ( V , k ) of a MAS function satisfies m ⋆ ( V , k ) ≥ log 2 ( log 3 n ) m^\star(V,k)\ge\log_2(\log_3 n) m ⋆ ( V , k ) ≥ log 2 ( log 3 n ) . Moreover, if k ≤ n − 1 2 k\le\frac{n-1}{2} k ≤ 2 n − 1 then $m^\star(V,k)\ge 2k
Proof. The proof is partially inspired by the proof in [14 ] .
**Lower bound on k k k : ** We want to show the lower bound of $m^\star(V,k)\ge \min(2k,n-2). As in the proof of Theorem 5.2.2, we select a ”maximal singleton element” v i ⋆ ∈ V v_i^\star\in V v i ⋆ ∈ V for every dimension i ∈ { 1 , 2 , ⋯ , m ⋆ ( V , k ) } i\in\{1,2,\cdots,m^\star(V,k)\} i ∈ { 1 , 2 , ⋯ , m ⋆ ( V , k )} . Thus we must have:
for each i ∈ [ m ⋆ ( V , k ) ] : F ( { v i ⋆ } ) [ i ] ≥ F ( { v } ) [ i ] , ∀ v ∈ V \text{for each }i\in[m^\star(V,k)]:\quad F(\{v_i^\star\})[i]\ge F(\{v\})[i],\ \forall v\in V for each i ∈ [ m ⋆ ( V , k )] : F ({ v i ⋆ }) [ i ] ≥ F ({ v }) [ i ] , ∀ v ∈ V
Now, if m ⋆ ( V , k ) ≤ ∣ V ∣ − 2 m^\star(V,k)\le|V|-2 m ⋆ ( V , k ) ≤ ∣ V ∣ − 2 , we can find two elements { u 1 , u 2 } \{u_1,u_2\} { u 1 , u 2 } disjoint from the collection V : = ⋃ i = 1 m ⋆ ( V , k ) { v i ⋆ } \mathcal{V}:=\bigcup_{i=1}^{m^\star(V,k)}\{v_i^\star\} V := ⋃ i = 1 m ⋆ ( V , k ) { v i ⋆ } . Without loss of generality, we can assume that F ( { u 1 } ) [ i ] ≥ F ( { u 2 } ) [ i ] F(\{u_1\})[i]\ge F(\{u_2\})[i] F ({ u 1 }) [ i ] ≥ F ({ u 2 }) [ i ] for most indices in [ m ⋆ ( V , k ) ] [m^\star(V,k)] [ m ⋆ ( V , k )] . We can then construct a set T T T containing u 1 u_1 u 1 and all the v i ⋆ v_i^\star v i ⋆ for all indices i i i for which F ( { u 1 } ) [ i ] < F ( { u 2 } ) [ i ] F(\{u_1\})[i]<F(\{u_2\})[i] F ({ u 1 }) [ i ] < F ({ u 2 }) [ i ] , thus we define:
T : = { u 1 } ∪ { v i ⋆ : F ( { u 1 } ) [ j ] < F ( { u 2 } ) [ j ] } T:=\{u_1\}\cup\{v_i^\star:F(\{u_1\})[j]<F(\{u_2\})[j]\} T := { u 1 } ∪ { v i ⋆ : F ({ u 1 }) [ j ] < F ({ u 2 }) [ j ]}
We then have that:
F ( { u 2 } ) [ i ] ≤ max { F ( { u 1 } ) [ i ] , F ( { v i ⋆ } ) [ i ] } ≤ F ( T ) [ i ] , ∀ i ∈ [ m ⋆ ( V , k ) ] F(\{u_2\})[i]\le\max\{F(\{u_1\})[i],F(\{v_i^\star\})[i]\}\le F(T)[i],\ \forall i\in[m^\star(V,k)] F ({ u 2 }) [ i ] ≤ max { F ({ u 1 }) [ i ] , F ({ v i ⋆ }) [ i ]} ≤ F ( T ) [ i ] , ∀ i ∈ [ m ⋆ ( V , k )]
where the last inequality uses the monotonicity of F F F . This implies that F ( S ) ≤ F ( T ) F(S) \le F(T) F ( S ) ≤ F ( T ) . However, S S S is not a subset of T T T . It follows that k ≤ ∣ T ∣ − 1 ≤ m ⋆ ( V , k ) / 2 k \le|T| - 1 \le m^{\star}(V,k)/2 k ≤ ∣ T ∣ − 1 ≤ m ⋆ ( V , k ) /2 , as otherwise separability of F F F in P ≤ k ( V ) \mathcal{P}_{\le k}(V) P ≤ k ( V ) is violated. Hence, we get that: m ⋆ ( V , k ) ≥ 2 k m^{\star}(V,k) \ge2k m ⋆ ( V , k ) ≥ 2 k
Lower bound on n n n : For this proof, we will use the Erdős-Szekeres theorem, which in particular states, for a natural N ≥ 2 N \ge2 N ≥ 2 , that a sequence of real numbers with ≥ N 2 \ge N^2 ≥ N 2 elements has a subsequence of cardinality N N N which is monotone (either monotonously increasing or monotonously decreasing). Assume we have F : P ≤ k ( V ) → R m F : \mathcal{P}_{\le k}(V) \to\mathbb{R}^m F : P ≤ k ( V ) → R m a monotone embedding, where V = [ n ] V = [n] V = [ n ] . Assume by way of contradiction that:
log 2 ( log 3 n ) > m or equivalently n > 3 2 m \log_2(\log_3 n) > m \text{ or equivalently } n > 3^{2^m} log 2 ( log 3 n ) > m or equivalently n > 3 2 m
Consider the first coordinate and look at the sequence
F ( { 1 } ) [ 1 ] , … , F ( { n } ) [ 1 ] F(\{1\})[1], \ldots, F(\{n\})[1] F ({ 1 }) [ 1 ] , … , F ({ n }) [ 1 ]
This is a real-valued sequence, and so there is a monotone subsequence a 1 < a 2 < … < a ℓ a_1 < a_2 < \ldots< a_\ell a 1 < a 2 < … < a ℓ of the original sequence 1 , … , n 1, \ldots, n 1 , … , n , with ℓ = 3 2 m = 3 2 m − 1 \ell= \sqrt{3^{2^m}} = 3^{2^{m-1}} ℓ = 3 2 m = 3 2 m − 1 , such that:
F ( a 1 ) [ 1 ] ≤ F ( a 2 ) [ 1 ] ≤ … ≤ F ( a ℓ ) [ 1 ] or F ( a 1 ) [ 1 ] ≥ F ( a 2 ) [ 1 ] ≥ … ≥ F ( a ℓ ) [ 1 ] F(a_1)[1] \le F(a_2)[1] \le\ldots\le F(a_\ell)[1] \text{ or } F(a_1)[1] \ge F(a_2)[1] \ge\ldots\ge F(a_\ell)[1] F ( a 1 ) [ 1 ] ≤ F ( a 2 ) [ 1 ] ≤ … ≤ F ( a ℓ ) [ 1 ] or F ( a 1 ) [ 1 ] ≥ F ( a 2 ) [ 1 ] ≥ … ≥ F ( a ℓ ) [ 1 ]
Next, we consider the second coordinate of this subsequence, namely F ( a 1 ) [ 2 ] , … , F ( a ℓ ) [ 2 ] F(a_1)[2], \ldots, F(a_\ell)[2] F ( a 1 ) [ 2 ] , … , F ( a ℓ ) [ 2 ] and obtain a new subsequence { a j } \{a_j\} { a j } such that both the sequences: { F ( a j ) [ 1 ] } \{F(a_j)[1]\} { F ( a j ) [ 1 ]} and { F ( a j ) [ 2 ] } \{F(a_j)[2]\} { F ( a j ) [ 2 ]} are ordered monotonically, and the size of the subsequence is the square root of the previous one, namely 3 2 m − 2 3^{2^{m-2}} 3 2 m − 2 . After doing this m m m times, we have a subsequence of size 3 3 3 . Namely, we have three distinct elements u , v , w ∈ V u, v, w \in V u , v , w ∈ V such that for all i = 1 , … , m i = 1, \ldots, m i = 1 , … , m ,
either F ( { u } ) [ i ] ≤ F ( { v } ) [ i ] ≤ F ( { w } ) [ i ] or F ( { u } ) [ i ] ≥ F ( { v } ) [ i ] ≥ F ( { w } ) [ i ] \text{either } F(\{u\})[i] \le F(\{v\})[i] \le F(\{w\})[i] \text{ or } F(\{u\})[i] \ge F(\{v\})[i] \ge F(\{w\})[i] either F ({ u }) [ i ] ≤ F ({ v }) [ i ] ≤ F ({ w }) [ i ] or F ({ u }) [ i ] ≥ F ({ v }) [ i ] ≥ F ({ w }) [ i ]
It follows that for all i i i , we have:
F ( { v } ) [ i ] ≤ max ( F ( { u } ) [ i ] , F ( { w } ) [ i ] ) ≤ F ( { u , w } ) [ i ] F(\{v\})[i] \le\max\left(F(\{u\})[i], F(\{w\})[i]\right) \le F(\{u,w\})[i] F ({ v }) [ i ] ≤ max ( F ({ u }) [ i ] , F ({ w }) [ i ] ) ≤ F ({ u , w }) [ i ]
Thus, F ( { v } ) ≤ F ( { u , w } ) F(\{v\}) \le F(\{u,w\}) F ({ v }) ≤ F ({ u , w }) , which is a contradiction since { v } \{v\} { v } is not contained in { u , w } \{u,w\} { u , w } .
A refined lower bound: Now, we may extend the above proof idea to generalize the lower bound on m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) , which is useful especially when ∣ V ∣ > > k |V| >> k ∣ V ∣ >> k . In such cases, the following gives a tighter lower bound on m ⋆ ( V , k ) m^{\star}(V,k) m ⋆ ( V , k ) which is quadratic in k k k .
Lemma B.1. For all ℓ ∈ [ k ] \ell\in[k] ℓ ∈ [ k ] , we have m ⋆ ( V , k ) ≥ min ( ∣ V ∣ − ℓ , ℓ k + 2 ℓ − ℓ 2 ) m^{\star}(V,k) \ge\min\left(|V|-\ell,\ell k+2\ell-\ell^2\right) m ⋆ ( V , k ) ≥ min ( ∣ V ∣ − ℓ , ℓ k + 2 ℓ − ℓ 2 ) . More specifically, if ℓ = k + 2 2 \ell= \frac{k+2}{2} ℓ = 2 k + 2 , then m ⋆ ( V , k ) ≥ min { ∣ V ∣ − k + 2 2 , ( k + 2 2 ) 2 } m^{\star}(V,k) \ge\min\left\{|V|-\frac{k+2}{2},\left(\frac{k+2}{2}\right)^2\right\} m ⋆ ( V , k ) ≥ min { ∣ V ∣ − 2 k + 2 , ( 2 k + 2 ) 2 }
Proof. Consider any ℓ ∈ [ k ] \ell\in[k] ℓ ∈ [ k ] . Like in the proof just before, we consider a ”maximal singleton element” v i ∗ ∈ V v_i^* \in V v i ∗ ∈ V for every dimension i ∈ { 1 , 2 , ⋯ , m ∗ ( V , k ) } i \in\{1,2,\cdots,m^*(V,k)\} i ∈ { 1 , 2 , ⋯ , m ∗ ( V , k )} and obtain the collection V : = ⋃ i = 1 m ∗ ( V , k ) { v i ∗ } \mathcal{V} := \bigcup_{i=1}^{m^*(V,k)} \{v_i^*\} V := ⋃ i = 1 m ∗ ( V , k ) { v i ∗ } . Let us assume m ∗ ( V , k ) ≤ ∣ V ∣ − ℓ m^*(V,k) \leq|V|-\ell m ∗ ( V , k ) ≤ ∣ V ∣ − ℓ . Thus, we can produce the collection U : = { u 1 , u 2 , ⋯ , u ℓ } \mathcal{U} := \{u_1,u_2,\cdots,u_\ell\} U := { u 1 , u 2 , ⋯ , u ℓ } disjoint from V \mathcal{V} V , i.e V ∩ U = ∅ \mathcal{V} \cap\mathcal{U} = \emptyset V ∩ U = ∅ . For each u q ∈ U u_q \in\mathcal{U} u q ∈ U , let m q m_q m q be the number of co-ordinates j ∈ [ m ∗ ( V , k ) ] j \in[m^*(V,k)] j ∈ [ m ∗ ( V , k )] such that F ( { u q } ) [ j ] > F ( { u r } ) [ j ] , ∀ r ∈ [ ℓ ] ∖ { q } F(\{u_q\})[j] > F(\{u_r\})[j], \forall r \in[\ell] \setminus\{q\} F ({ u q }) [ j ] > F ({ u r }) [ j ] , ∀ r ∈ [ ℓ ] ∖ { q } . Clearly, ∑ q ∈ [ ℓ ] m q ≤ m ∗ ( V , k ) \sum_{q\in[\ell]}m_q \leq m^*(V,k) ∑ q ∈ [ ℓ ] m q ≤ m ∗ ( V , k ) , thus, ∃ ℓ 0 ∈ [ ℓ ] \exists\ell_0 \in[\ell] ∃ ℓ 0 ∈ [ ℓ ] such that m ℓ 0 ≤ m ∗ ( V , k ) ℓ m_{\ell_0} \leq\frac{m^*(V,k)}{\ell} m ℓ 0 ≤ ℓ m ∗ ( V , k ) . Like before, we consider the sets S = { u ℓ 0 } S=\{u_{\ell_0}\} S = { u ℓ 0 } . We construct T T T as follows: we select U ∖ { u ℓ 0 } \mathcal{U}\setminus\{u_{\ell_0}\} U ∖ { u ℓ 0 } and take union with all { v j ∗ } \{v_j^*\} { v j ∗ } for each dimension j j j in which F ( { u ℓ 0 } ) [ j ] > F ( { u r } ) [ j ] , ∀ r ∈ [ ℓ ] ∖ { ℓ 0 } F(\{u_{\ell_0}\})[j] > F(\{u_r\})[j], \forall r \in[\ell]\setminus\{\ell_0\} F ({ u ℓ 0 }) [ j ] > F ({ u r }) [ j ] , ∀ r ∈ [ ℓ ] ∖ { ℓ 0 } . Since { v j ∗ } \{v_j^*\} { v j ∗ } is the maximal singleton for the dimension j j j , it follows that F ( { u ℓ 0 } ) [ j ] ≤ F ( { v j ∗ } ) [ j ] F(\{u_{\ell_0}\})[j] \leq F(\{v_j^*\})[j] F ({ u ℓ 0 }) [ j ] ≤ F ({ v j ∗ }) [ j ] . Hence, if we define:
T = ( U ∖ { u ℓ 0 } ) ∪ { v j ∗ : F ( { u ℓ 0 } ) [ j ] > F ( { u r } ) , ∀ r ∈ [ ℓ ] ∖ { ℓ 0 } } T = \left(\mathcal{U}\setminus\{u_{\ell_0}\}\right) \cup\left\{v_j^* : F(\{u_{\ell_0}\})[j] > F(\{u_r\}), \forall r \in[\ell]\setminus\{\ell_0\}\right\} T = ( U ∖ { u ℓ 0 } ) ∪ { v j ∗ : F ({ u ℓ 0 }) [ j ] > F ({ u r }) , ∀ r ∈ [ ℓ ] ∖ { ℓ 0 } }
By the choice of ℓ 0 \ell_0 ℓ 0 , we have that ∣ { v j ∗ : F ( { u ℓ 0 } ) [ j ] > F ( { u r } ) , ∀ r ∈ [ ℓ ] ∖ { ℓ 0 } } ∣ ≤ m ∗ ( V , k ) ℓ \left|\left\{v_j^*:F(\{u_{\ell_0}\})[j] > F(\{u_r\}),\forall r \in[\ell]\setminus\{\ell_0\}\right\}\right| \leq\frac{m^*(V,k)}{\ell} { v j ∗ : F ({ u ℓ 0 }) [ j ] > F ({ u r }) , ∀ r ∈ [ ℓ ] ∖ { ℓ 0 } } ≤ ℓ m ∗ ( V , k ) . Now, we must have that, ∀ j ∈ [ m ∗ ( V , k ) ] , ∃ t ∈ T \forall j \in[m^*(V,k)], \exists t \in T ∀ j ∈ [ m ∗ ( V , k )] , ∃ t ∈ T such that F ( { t } ) [ j ] ≥ F ( { u ℓ 0 } ) [ j ] F(\{t\})[j] \geq F(\{u_{\ell_0}\})[j] F ({ t }) [ j ] ≥ F ({ u ℓ 0 }) [ j ] . By monotonicity of F F F , we thus have that F ( { u ℓ 0 } ) [ j ] ≤ F ( T ) [ j ] , ∀ j ∈ [ m ∗ ( V , k ) ] F(\{u_{\ell_0}\})[j] \leq F(T)[j], \forall j \in[m^*(V,k)] F ({ u ℓ 0 }) [ j ] ≤ F ( T ) [ j ] , ∀ j ∈ [ m ∗ ( V , k )] which implies F ( S ) ≤ F ( T ) F(S) \leq F(T) F ( S ) ≤ F ( T ) . But by design, S ∩ T = ∅ S \cap T = \emptyset S ∩ T = ∅ , which contradicts separability of F F F . Thus, we must have that, ∣ T ∣ > k |T| > k ∣ T ∣ > k , i.e ∣ T ∣ − 1 ≥ k |T|-1 \geq k ∣ T ∣ − 1 ≥ k . But as stated before, ∣ T ∣ ≤ m ∗ ( V , k ) ℓ + ℓ − 1 |T| \leq\frac{m^*(V,k)}{\ell}+\ell-1 ∣ T ∣ ≤ ℓ m ∗ ( V , k ) + ℓ − 1 . Combining these two, we get that: m ∗ ( V , k ) ℓ ≥ k + 2 − ℓ \frac{m^*(V,k)}{\ell} \geq k+2-\ell ℓ m ∗ ( V , k ) ≥ k + 2 − ℓ , which implies m ∗ ( V , k ) ≥ k ℓ + 2 ℓ − ℓ 2 m^*(V,k) \geq k\ell+2\ell-\ell^2 m ∗ ( V , k ) ≥ k ℓ + 2 ℓ − ℓ 2 , thus proving our result.
Now, we already have a sufficient upper bound on m ∗ ( V , k ) m^*(V,k) m ∗ ( V , k ) from Theorem 5.2.3 that’s O ( k k + 2 log ( n ) ) O(k^{k+2}\log(n)) O ( k k + 2 log ( n )) . Now, when n > > k n >> k n >> k we have that O ( k k + 2 log ( n ) ) < n − O ( k ) O(k^{k+2}\log(n)) < n-O(k) O ( k k + 2 log ( n )) < n − O ( k ) , in which case we may apply ℓ = k + 2 2 \ell=\frac{k+2}{2} ℓ = 2 k + 2 in the above lemma. And since we already know that O ( k k + 2 log ( n ) ) < n − O ( k ) O(k^{k+2}\log(n)) < n-O(k) O ( k k + 2 log ( n )) < n − O ( k ) , we thus get that m ∗ ( V , k ) < n − O ( k ) m^*(V,k) < n-O(k) m ∗ ( V , k ) < n − O ( k ) . Thus, in this case the above lemma gives us m ∗ ( V , k ) ≥ ( k + 2 2 ) 2 m^*(V,k) \geq\left(\frac{k+2}{2}\right)^2 m ∗ ( V , k ) ≥ ( 2 k + 2 ) 2 , which is a tighter quadratic lower bound in k k k .
When k = 1 k=1 k = 1 In the degenerate case where only a single set element is allowed, k = 1 k=1 k = 1 , it is possible to construct a MAS function even for the uncountable ground set V = [ − 1 , 1 ] V=[-1,1] V = [ − 1 , 1 ] by defining F : P ≤ 1 ( [ − 1 , 1 ] ) → R 2 F:\mathcal{P}_{\leq1}([-1,1])\to\mathbb{R}^2 F : P ≤ 1 ([ − 1 , 1 ]) → R 2 via
F ( S ) = ( − 1 , − 1 ) if S = ∅ and F ( S ) = ( − x , x ) if S = { x } , (54) F(S)=(-1,-1)\text{ if }S=\emptyset\qquad\text{and}\qquad F(S)=(-x,x)\text{ if }S=\{x\}, \tag*{(54)} F ( S ) = ( − 1 , − 1 ) if S = ∅ and F ( S ) = ( − x , x ) if S = { x } , (54)
Separable embeddings (even non monotone) don’t exist: Now, as discussed in the main text, we give a justification here on why we choose to relax separability and not monotonicity in the definition of weakly MAS embeddings. Here, we show that, with the added assumptions of injectivity and continuity, even non-monotone set-to-vector embeddings do not exist in any dimension for uncountable ground sets.
Theorem B.2. There does not exist a continuous injective separable set function taking values in R m \mathbb{R}^m R m for any m m m when the ground set V V V is an open subset of $\mathbb{R}^d
Proof. We restrict ourselves to P = n ( V ) \mathcal{P}_{=n}(V) P = n ( V ) , i.e, only subsets of size n n n . Since F F F is continuous and permutation-invariant, according to Theorem-7 of [ZKR+17], f ( { x 1 , ⋯ , x n } ) = ρ ( ∑ i = 1 n ϕ ( x i ) ) f(\{x_1,\cdots,x_n\})=\rho(\sum_{i=1}^{n}\phi(x_i)) f ({ x 1 , ⋯ , x n }) = ρ ( ∑ i = 1 n ϕ ( x i )) , where ρ , ϕ \rho,\phi ρ , ϕ are continuous and bijective(given in condition). Now, consider the space S ⊂ [ 0 , 1 ] n \mathcal{S}\subset[0,1]^n S ⊂ [ 0 , 1 ] n , S = { ( x 1 , ⋯ , x n ) : x 1 < x 2 < x 3 < ⋯ < x n ; 0 < x i < 1 , ∀ i ∈ [ n ] } \mathcal{S}=\{(x_1,\cdots,x_n):x_1<x_2<x_3<\cdots<x_n;0<x_i<1,\forall i\in[n]\} S = {( x 1 , ⋯ , x n ) : x 1 < x 2 < x 3 < ⋯ < x n ; 0 < x i < 1 , ∀ i ∈ [ n ]} . Note that S \mathcal{S} S is open in [ 0 , 1 ] n [0,1]^n [ 0 , 1 ] n (to see this, let ϵ = min i ∈ [ n − 1 ] ( x i + 1 − x i ) \epsilon=\min_{i\in[n-1]}(x_{i+1}-x_i) ϵ = min i ∈ [ n − 1 ] ( x i + 1 − x i ) . The ball of radius ϵ 2 \frac{\epsilon}{2} 2 ϵ is contained in the domain). Since F : S → R n F:\mathcal{S}\to\mathbb{R}^n F : S → R n is an injective, continuous map. Thus, by Invariance of Domain Theorem([Mun16]), F ( S ) F(\mathcal{S}) F ( S ) is open in R n \mathbb{R}^n R n as well. Note that each element of S \mathcal{S} S uniquely corresponds to an element of P = n ( V ) \mathcal{P}_{=n}(V) P = n ( V ) . Thus, F ( P = n ( V ) ) F(\mathcal{P}_{=n}(V)) F ( P = n ( V )) is open in R n \mathbb{R}^n R n . Now, consider any set A = { a 1 , ⋯ , a n } A=\{a_1,\cdots,a_n\} A = { a 1 , ⋯ , a n } . Since F F F is an open map, we have that ∃ δ > 0 \exists\delta>0 ∃ δ > 0 such that B ( F ( A ) , δ ) ⊆ F ( P = n ( V ) ) B(F(A),\delta)\subseteq F(\mathcal{P}_{=n}(V)) B ( F ( A ) , δ ) ⊆ F ( P = n ( V )) . Thus, ∃ B ∈ P = n ( V ) \exists B\in\mathcal{P}_{=n}(V) ∃ B ∈ P = n ( V ) such that F ( B ) i = F ( A ) i + δ 2 n , ∀ i ∈ [ n ] F(B)_i=F(A)_i+\frac{\delta}{2n},\ \forall i\in[n] F ( B ) i = F ( A ) i + 2 n δ , ∀ i ∈ [ n ] . But then, F ( B ) > F ( A ) F(B)>F(A) F ( B ) > F ( A ) implies A ⊂ B A\subset B A ⊂ B but ∣ A ∣ = ∣ B ∣ = n |A|=|B|=n ∣ A ∣ = ∣ B ∣ = n by construction. This gives a contradiction. □ \square □
Proofs of results on weakly MAS functions
Proposition 6. If the deep set model in (5.2) is implemented with a vector-to-vector monotonously increasing function M 2 M_2 M 2 and a non-negative activation σ \sigma σ , then F ( ∙ ; ( A , b ) ) F(\bullet;(A,b)) F ( ∙ ; ( A , b )) is monotone for every ( A , b ) (A,b) ( A , b ) . If in addition σ \sigma σ is monotone (increasing or decreasing), then there exists S ⊈ T S\nsubseteq T S ⊈ T with F ( S ; ( A , b ) ) ≤ F ( T ; ( A , b ) ) F(S;(A,b))\leq F(T;(A,b)) F ( S ; ( A , b )) ≤ F ( T ; ( A , b )) for all A , b A,b A , b .
Proof. Let x < y x<y x < y and consider the two sets:
S = { x + y 2 } , T = { x , y } S=\left\{\frac{x+y}{2}\right\},\ T=\{x,y\} S = { 2 x + y } , T = { x , y } And we claim that for all A , b A,b A , b , although S ⊈ T S\nsubseteq T S ⊈ T , F ( S , ( A , b ) ) ≤ F ( T , ( A , b ) ) F(S,(A,b))\leq F(T,(A,b)) F ( S , ( A , b )) ≤ F ( T , ( A , b )) . Note that as x < x + y 2 < y x<\frac{x+y}{2}<y x < 2 x + y < y , then it must have been that
A x + b ≤ A ⋅ x + y 2 + b ≤ A y + b Ax+b\leq A\cdot\frac{x+y}{2}+b\leq Ay+b A x + b ≤ A ⋅ 2 x + y + b ≤ A y + b Or
A y + b ≤ A ⋅ x + y 2 + b ≤ A x + b Ay+b\leq A\cdot\frac{x+y}{2}+b\leq Ax+b A y + b ≤ A ⋅ 2 x + y + b ≤ A x + b Then, by (weakly) monotonicity of the activation σ \sigma σ and of M 2 M_2 M 2 , it must be that
σ ( A ⋅ x + y 2 + b ) ≤ σ ( A ⋅ x + b ) \sigma\left(A\cdot\frac{x+y}{2}+b\right)\leq\sigma(A\cdot x+b) σ ( A ⋅ 2 x + y + b ) ≤ σ ( A ⋅ x + b ) Or that
σ ( A ⋅ x + y 2 + b ) ≤ σ ( A ⋅ y + b ) \sigma\left(A\cdot\frac{x+y}{2}+b\right)\leq\sigma(A\cdot y+b) σ ( A ⋅ 2 x + y + b ) ≤ σ ( A ⋅ y + b )
In any case, it’s true that
σ ( A ⋅ x + y 2 ) ≤ σ ( A ⋅ x + b ) + σ ( A ⋅ y + b ) \sigma\left(A \cdot\frac{x+y}{2}\right) \le\sigma(A \cdot x+b)+\sigma(A \cdot y+b) σ ( A ⋅ 2 x + y ) ≤ σ ( A ⋅ x + b ) + σ ( A ⋅ y + b )
Then, by monotonicity of M 2 M_2 M 2 ,
F ( S , ( A , b ) ) = M 2 ⋅ ( σ ( A ⋅ x + y 2 + b ) ) ≤ M 2 ⋅ ( σ ( A ⋅ x + b ) + σ ( A ⋅ y + b ) ) = F ( T , ( A , b ) ) F(S,(A,b))=M_2\cdot\left(\sigma\left(A\cdot\frac{x+y}{2}+b\right)\right)\le M_2\cdot\left(\sigma(A\cdot x+b)+\sigma(A\cdot y+b)\right)=F(T,(A,b)) F ( S , ( A , b )) = M 2 ⋅ ( σ ( A ⋅ 2 x + y + b ) ) ≤ M 2 ⋅ ( σ ( A ⋅ x + b ) + σ ( A ⋅ y + b ) ) = F ( T , ( A , b ))
Thus,
F ( S , ( A , b ) ) ≤ F ( T , ( A , b ) ) F(S,(A,b))\le F(T,(A,b)) F ( S , ( A , b )) ≤ F ( T , ( A , b ))
Proposition B.3. Let V = R d V=\mathbb{R}^d V = R d and let F : P < ∞ ( V ) → R D F:\mathcal{P}_{<\infty}(V)\to\mathbb{R}^D F : P < ∞ ( V ) → R D be a Set Transformer defined by F ( S ; W Q , W K , W V ) : = SumPool ( Attn ( S ) ) F(S;W_Q,W_K,W_V):=\operatorname{SumPool}(\operatorname{Attn}(S)) F ( S ; W Q , W K , W V ) := SumPool ( Attn ( S )) . Then F F F is not a point-wise monotone function.
Proof. We need to come up with a class of tuples of parameters ( W Q , W K , W V ) (W_Q,W_K,W_V) ( W Q , W K , W V ) such that the function S ↦ F ( S ; W Q , W K , W V ) S\mapsto F(S;W_Q,W_K,W_V) S ↦ F ( S ; W Q , W K , W V ) is not monotone. Choose W Q = W K = W W_Q=W_K=W W Q = W K = W such that W W W is full-rank. Choose any non-zero W V W_V W V . Now, choose some vector x 1 x_1 x 1 such that W V x 1 W_Vx_1 W V x 1 is non-zero. By swapping x 1 x_1 x 1 with − x 1 -x_1 − x 1 if necessary, we can choose some index j j j such that for v 1 = W V x 1 v_1=W_Vx_1 v 1 = W V x 1 , the j j j -th index is negative, i.e. v 1 [ j ] < 0 v_1[j]<0 v 1 [ j ] < 0 . By assumption we also have that q 1 = k 1 q_1=k_1 q 1 = k 1 is a non-zero vector, and thus has a positive norm. Now let S S S be the singleton set S = { x 1 } S=\{x_1\} S = { x 1 } , which is a subset of T = { x 1 , 0 d } T=\{x_1,0_d\} T = { x 1 , 0 d } where 0 d 0_d 0 d is the d d d -dimensional all-zero vector. Note that, F ( S ; W Q , W K , W V ) = v 1 F(S;W_Q,W_K,W_V)=v_1 F ( S ; W Q , W K , W V ) = v 1 . And since q 2 , k 2 , v 2 q_2,k_2,v_2 q 2 , k 2 , v 2 are all zero, we have: F ( T ; W Q , W K , W V ) = ( α 1 , 1 + α 1 , 2 ) v 1 F(T;W_Q,W_K,W_V)=(\alpha_{1,1}+\alpha_{1,2})v_1 F ( T ; W Q , W K , W V ) = ( α 1 , 1 + α 1 , 2 ) v 1 . Note that,
α 1 , 1 α 1 , 1 + α 1 , 2 = e ∥ q 1 ∥ 2 e ∥ q 1 ∥ 2 + 1 + 1 2 > 1. \frac{\alpha_{1,1}}{\alpha_{1,1}+\alpha_{1,2}}=\frac{e^{\lVert q_1\rVert^2}}{e^{\lVert q_1\rVert^2+1}}+\frac{1}{2}>1. α 1 , 1 + α 1 , 2 α 1 , 1 = e ∥ q 1 ∥ 2 + 1 e ∥ q 1 ∥ 2 + 2 1 > 1. Hence, it follows that F ( T ; W Q , W K , W V ) [ j ] < F ( S ; W Q , W K , W V ) [ j ] F(T;W_Q,W_K,W_V)[j]<F(S;W_Q,W_K,W_V)[j] F ( T ; W Q , W K , W V ) [ j ] < F ( S ; W Q , W K , W V ) [ j ] and thus, the non-monotonicity follows
Proposition 8. If M 2 M_2 M 2 is strictly monotone increasing, and σ \sigma σ is a hat activation function, then F ( S , ( a , b ) ) = M 2 ( ∑ x ∈ S σ ( a T x + b ) ) F(S,(a,b))=M_2\left(\sum_{x\in S}\sigma(a^\mathsf{T}x+b)\right) F ( S , ( a , b )) = M 2 ( ∑ x ∈ S σ ( a T x + b ) ) is a weakly MAS function.
Proof. It’s clear the function is a set function. Let S ⊆ T S\subseteq T S ⊆ T . Since σ \sigma σ is positive, then for any a , b a,b a , b ,
∑ x ∈ S σ ( a T x + b ) ≤ ∑ x ∈ S σ ( a T x + b ) + ∑ x ∈ T ∖ S σ ( a T x + b ) = ∑ x ∈ T σ ( a T x + b ) \sum_{x\in S}\sigma(a^\mathsf{T}x+b)\le\sum_{x\in S}\sigma(a^\mathsf{T}x+b)+\sum_{x\in T\setminus S}\sigma(a^\mathsf{T}x+b)=\sum_{x\in T}\sigma(a^\mathsf{T}x+b) x ∈ S ∑ σ ( a T x + b ) ≤ x ∈ S ∑ σ ( a T x + b ) + x ∈ T ∖ S ∑ σ ( a T x + b ) = x ∈ T ∑ σ ( a T x + b ) So
F ( S , ( a , b ) ) ≤ F ( T , ( a , b ) ) F(S,(a,b))\le F(T,(a,b)) F ( S , ( a , b )) ≤ F ( T , ( a , b )) Denote by [ s 1 , s 2 ] [s_1,s_2] [ s 1 , s 2 ] the support of σ \sigma σ , and choose some point in it such that s ∈ [ s 1 , s 2 ] s\in[s_1,s_2] s ∈ [ s 1 , s 2 ] : σ ( s ) > 0 \sigma(s)>0 σ ( s ) > 0 . Now, let S ⊈ T S\not\subseteq T S ⊆ T . Then there exists z ∈ V : C S ( z ) > C T ( z ) z\in V:C_S(z)>C_T(z) z ∈ V : C S ( z ) > C T ( z ) . Now, remove z z z
from both sets C T ( z ) C_T(z) C T ( z ) times. This subtracts from both function values the same amount. Now, we have that z ∈ S z \in S z ∈ S , z ∉ T z \notin T z ∈ / T . Denote by ϵ = min t ∈ T ∣ z − t ∣ 2 > 0 \epsilon= \frac{\min_{t \in T} |z-t|}{2} > 0 ϵ = 2 m i n t ∈ T ∣ z − t ∣ > 0 . Choose
a = s 2 − s 1 ϵ , b = − ( a ⋅ z − s ) a = \frac{s_2-s_1}{\epsilon}, b = -(a \cdot z - s) a = ϵ s 2 − s 1 , b = − ( a ⋅ z − s )
We claim that ∀ t ∈ T , σ ( a ⋅ t + b ) = 0 \forall t \in T, \sigma(a \cdot t + b) = 0 ∀ t ∈ T , σ ( a ⋅ t + b ) = 0 . Assume it’s not zero, thus s 1 < a ⋅ t + b < s 2 s_1 < a \cdot t + b < s_2 s 1 < a ⋅ t + b < s 2 . Thus
s 1 < s 2 − s 1 ϵ ⋅ t − a ⋅ z + s < s 2 s_1 < \frac{s_2-s_1}{\epsilon} \cdot t - a \cdot z + s < s_2 s 1 < ϵ s 2 − s 1 ⋅ t − a ⋅ z + s < s 2
Thus,
s 1 − s < s 2 − s 1 ϵ ⋅ t − s 2 − s 1 ϵ ⋅ z < s 2 − s s_1-s < \frac{s_2-s_1}{\epsilon} \cdot t - \frac{s_2-s_1}{\epsilon} \cdot z < s_2-s s 1 − s < ϵ s 2 − s 1 ⋅ t − ϵ s 2 − s 1 ⋅ z < s 2 − s
Thus,
s 1 − s s 2 − s 1 ⋅ ϵ < t − z < ϵ ⋅ s 2 − s s 2 − s 1 \frac{s_1-s}{s_2-s_1} \cdot\epsilon< t-z < \epsilon\cdot\frac{s_2-s}{s_2-s_1} s 2 − s 1 s 1 − s ⋅ ϵ < t − z < ϵ ⋅ s 2 − s 1 s 2 − s
But,
∣ s 1 − s s 2 − s 1 ∣ ≤ 1 \left|\frac{s_1-s}{s_2-s_1}\right| \leq1 s 2 − s 1 s 1 − s ≤ 1
∣ s 2 − s s 2 − s 1 ∣ ≤ 1 \left|\frac{s_2-s}{s_2-s_1}\right| \leq1 s 2 − s 1 s 2 − s ≤ 1
Thus,
− ϵ ≤ t − z ≤ ϵ -\epsilon\leq t-z \leq\epsilon − ϵ ≤ t − z ≤ ϵ
A contradiction to the definition of ϵ \epsilon ϵ . Thus, F ( T , ( a , b ) ) = M 2 ( 0 ) F(T,(a,b)) = M_2(0) F ( T , ( a , b )) = M 2 ( 0 ) . But
σ ( a ⋅ z + b ) = σ ( a ⋅ z − a ⋅ z + s 1 ) = σ ( s 1 ) > 0 \sigma(a \cdot z+b) = \sigma(a \cdot z-a \cdot z+s_1) = \sigma(s_1) > 0 σ ( a ⋅ z + b ) = σ ( a ⋅ z − a ⋅ z + s 1 ) = σ ( s 1 ) > 0
Thus
F ( S , ( a , b ) ) = M 2 ( ∑ x ∈ S σ ( a T x + b ) ) ≥ M 2 ( σ ( s 1 ) ) > M 2 ( 0 ) = F ( T , ( a , b ) ) F(S,(a,b)) = M_2\left(\sum_{x \in S} \sigma(a^\mathsf{T}x+b)\right) \geq M_2(\sigma(s_1)) > M_2(0) = F(T,(a,b)) F ( S , ( a , b )) = M 2 ( x ∈ S ∑ σ ( a T x + b ) ) ≥ M 2 ( σ ( s 1 )) > M 2 ( 0 ) = F ( T , ( a , b ))
And we are done.
Proposition B.4. Given V ⊆ R d V \subseteq\mathbb{R}^d V ⊆ R d , we consider affine transformations A 2 : R d → R 3 , A 1 : R 3 → R \mathcal{A}_2 : \mathbb{R}^d \to\mathbb{R}^3, \mathcal{A}_1 : \mathbb{R}^3 \to\mathbb{R} A 2 : R d → R 3 , A 1 : R 3 → R . Thus, here A 2 ( t ) : = A 2 t + b 2 \mathcal{A}_2(t) := A_2t+b_2 A 2 ( t ) := A 2 t + b 2 and A 1 ( z ) = a 1 T z + b 1 \mathcal{A}_1(z) = a_1^\mathsf{T}z+b_1 A 1 ( z ) = a 1 T z + b 1 , where A 2 ∈ R 3 × d , b 2 ∈ R 3 , a 1 ∈ R 3 , b 1 ∈ R A_2 \in\mathbb{R}^{3 \times d}, b_2 \in\mathbb{R}^3, a_1 \in\mathbb{R}^3, b_1 \in\mathbb{R} A 2 ∈ R 3 × d , b 2 ∈ R 3 , a 1 ∈ R 3 , b 1 ∈ R are the respective parameters. Then, the set functions of the form
F : P < ∞ ( V ) → R , F ( S ; A 1 , A 2 ) = M 2 ( ∑ x ∈ S ReLU ∘ A 1 ∘ ReLU ∘ A 2 ( x ) ) (55) F : \mathcal{P}_{<\infty}(V) \to\mathbb{R}, F(S; \mathcal{A}_1,\mathcal{A}_2) = M_2\left(\sum_{x \in S} \operatorname{ReLU} \circ\mathcal{A}_1 \circ\operatorname{ReLU} \circ\mathcal{A}_2(x)\right) \tag*{(55)} F : P < ∞ ( V ) → R , F ( S ; A 1 , A 2 ) = M 2 ( x ∈ S ∑ ReLU ∘ A 1 ∘ ReLU ∘ A 2 ( x ) ) (55)
are weakly MAS functions.
Proof. Consider the function TRI as in the third plot of Figure 5.1 . We know that TRI belongs to the class of Hat functions. From 5.3.5 we know that, for any S ⊈ T ∈ P < ∞ ( V ) S \nsubseteq T \in\mathcal{P}_{<\infty}(V) S ⊈ T ∈ P < ∞ ( V ) , ∃ a ∈ R d \exists a \in\mathbb{R}^{d} ∃ a ∈ R d and b ∈ R b \in\mathbb{R} b ∈ R such that ∑ x ∈ S TRI ( a ⊤ x + b ) > ∑ y ∈ T TRI ( a ⊤ y + b ) \sum_{x \in S} \operatorname{TRI}(a^{\top}x+b) > \sum_{y \in T} \operatorname{TRI}(a^{\top}y+b) ∑ x ∈ S TRI ( a ⊤ x + b ) > ∑ y ∈ T TRI ( a ⊤ y + b ) . Now, consider the following parameters:
A = [ a , a , a ] ⊤ ∈ R 3 × d , b = [ b + 1 , b − 1 , b ] ⊤ ∈ R 3 A = [a,a,a]^{\top} \in\mathbb{R}^{3 \times d}, \quad b = [b+1,b-1,b]^{\top} \in\mathbb{R}^{3} A = [ a , a , a ] ⊤ ∈ R 3 × d , b = [ b + 1 , b − 1 , b ] ⊤ ∈ R 3
a 1 = [ 1 , 1 , − 2 ] ⊤ ∈ R 3 , and b 1 = 0 (56) a_{1} = [1,1,-2]^{\top} \in\mathbb{R}^{3}, \quad\text{and } b_{1}=0 \tag*{(56)} a 1 = [ 1 , 1 , − 2 ] ⊤ ∈ R 3 , and b 1 = 0 (56)
Then, for the above values of parameters, we have:
ReLU ∘ A 2 ( x ) = [ ReLU ( ( a ⊤ x + b ) + 1 ) , ReLU ( ( a ⊤ x + b ) − 1 ) , ReLU ( a ⊤ x + b ) ] \operatorname{ReLU} \circ A_{2}(x) = \left[\operatorname{ReLU}\left((a^{\top}x+b)+1\right),\operatorname{ReLU}\left((a^{\top}x+b)-1\right),\operatorname{ReLU}\left(a^{\top}x+b\right)\right] ReLU ∘ A 2 ( x ) = [ ReLU ( ( a ⊤ x + b ) + 1 ) , ReLU ( ( a ⊤ x + b ) − 1 ) , ReLU ( a ⊤ x + b ) ]
Thus, we have:
a 1 ⊤ ( ReLU ∘ A 2 ( x ) ) + b 1 = ReLU ( ( a ⊤ x + b ) + 1 ) + ReLU ( ( a ⊤ x + b ) − 1 ) − 2 ReLU ( a ⊤ x + b ) = TRI ( a ⊤ x + b ) \begin{aligned}
a_{1}^{\top}\left(\operatorname{ReLU} \circ A_{2}(x)\right)+b_{1}
&= \operatorname{ReLU}\left((a^{\top}x+b)+1\right)+\operatorname{ReLU}\left((a^{\top}x+b)-1\right)-2\operatorname{ReLU}\left(a^{\top}x+b\right) \\
&= \operatorname{TRI}\left(a^{\top}x+b\right)
\end{aligned} a 1 ⊤ ( ReLU ∘ A 2 ( x ) ) + b 1 = ReLU ( ( a ⊤ x + b ) + 1 ) + ReLU ( ( a ⊤ x + b ) − 1 ) − 2 ReLU ( a ⊤ x + b ) = TRI ( a ⊤ x + b )
Thus we have, ReLU ∘ A 1 ∘ ReLU ∘ A 2 ( x ) = ReLU ∘ TRI ( a ⊤ x + b ) = TRI ( a ⊤ x + b ) \operatorname{ReLU} \circ A_{1} \circ\operatorname{ReLU} \circ A_{2}(x)=\operatorname{ReLU} \circ\operatorname{TRI}(a^{\top}x+b)=\operatorname{TRI}(a^{\top}x+b) ReLU ∘ A 1 ∘ ReLU ∘ A 2 ( x ) = ReLU ∘ TRI ( a ⊤ x + b ) = TRI ( a ⊤ x + b ) . Hence, F ( S ) = ∑ x ∈ S TRI ( a ⊤ x + b ) > ∑ y ∈ T TRI ( a ⊤ y + b ) = F ( T ) F(S)=\sum_{x\in S}\operatorname{TRI}(a^{\top}x+b)>\sum_{y\in T}\operatorname{TRI}(a^{\top}y+b)=F(T) F ( S ) = ∑ x ∈ S TRI ( a ⊤ x + b ) > ∑ y ∈ T TRI ( a ⊤ y + b ) = F ( T ) and weak separability is proved. Since M 2 M_{2} M 2 is monotone increasing and ReLU is non-negative, it follows that F F F is monotone as well, hence weakly MAS.
Hölder separability of MAS functions
Theorem 5.3.8 (F F F is lower Hölder). Let V ⊂ R d V \subset\mathbb{R}^{d} V ⊂ R d be a compact set, and σ \sigma σ a Hat activation function which is piecewise continuously differentiable supported in some interval [ γ 1 , γ 2 ] [\gamma_{1},\gamma_{2}] [ γ 1 , γ 2 ] , and satisfying the condition: lim t → γ 1 + d σ d t > 0 \lim_{t\to\gamma_{1}^{+}}\frac{\mathrm{d}\sigma}{\mathrm{d}t}>0 lim t → γ 1 + d t d σ > 0 . Let M 2 : R → R M_{2}:\mathbb{R}\to\mathbb{R} M 2 : R → R be a lower Lipschitz function. Consider the function F ( S ; ( a , b , c ) ) = M 2 ( ∑ x ∈ S σ ( a ⊤ x + b c ) ) F(S;(a,b,c))=M_{2}\left(\sum_{x\in S}\sigma\left(\frac{a^{\top}x+b}{c}\right)\right) F ( S ; ( a , b , c )) = M 2 ( ∑ x ∈ S σ ( c a ⊤ x + b ) ) , where the multisets S S S come from P ≤ k ( V ) \mathcal{P}_{\leq k}(V) P ≤ k ( V ) , and a ∼ Unif ( S d − 1 ) a\sim\operatorname{Unif}(\mathbb{S}^{d-1}) a ∼ Unif ( S d − 1 ) , b ∼ Unif ( [ − 1 , 1 ] ) b\sim\operatorname{Unif}([-1,1]) b ∼ Unif ([ − 1 , 1 ]) , c ∼ Unif ( ( 0 , 2 ] ) c\sim\operatorname{Unif}((0,2]) c ∼ Unif (( 0 , 2 ]) . Then, F ( ∙ , ( a , b , c ) ) F(\bullet,(a,b,c)) F ( ∙ , ( a , b , c )) is Monotone Hölder separable with exponent λ = 2 \lambda=2 λ = 2 .
Proof. We consider V ⊂ R d V\subset\mathbb{R}^{d} V ⊂ R d to be a compact set with maximum norm of 1. Also, we note that, if σ ( x ) \sigma(x) σ ( x ) is a Hat function with support in [ γ 1 , γ 2 ] [\gamma_{1},\gamma_{2}] [ γ 1 , γ 2 ] , then σ ( x − γ 1 γ 2 − γ 1 ) \sigma\left(\frac{x-\gamma_{1}}{\gamma_{2}-\gamma_{1}}\right) σ ( γ 2 − γ 1 x − γ 1 ) is a Hat function with support in ( 0 , 1 ) (0,1) ( 0 , 1 ) . For this proof, we thus consider σ \sigma σ to have support in [ 0 , 1 ] [0,1] [ 0 , 1 ] . For support in general [ γ 1 , γ 2 ] [\gamma_{1},\gamma_{2}] [ γ 1 , γ 2 ] and a general norm bound of V V V , the distributions on b , c b,c b , c need to be scaled and shifted to get same Lower Hölder results with the Hölder constants scaled appropriately. 1 We recall the definition of Δ ( S , T ) \Delta(S,T) Δ ( S , T ) from equation (47) . Let S = { x 1 , ⋯ , x M } S = \{x_1,\cdots,x_M\} S = { x 1 , ⋯ , x M } and let T = { y 1 , ⋯ , y N } T = \{y_1,\cdots,y_N\} T = { y 1 , ⋯ , y N } , where S , T ⊂ V S,T \subset V S , T ⊂ V . Clearly by the definition of Δ ( S , T ) \Delta(S,T) Δ ( S , T ) , it is sufficient to consider ∣ S ∣ ≤ ∣ T ∣ \lvert S\rvert\le\lvert T\rvert ∣ S ∣ ≤ ∣ T ∣ , i.e. M ≤ N ≤ k M \le N \le k M ≤ N ≤ k .
Using the definition of EMD gives us the following equivalent definition of Δ ( S , T ) \Delta(S,T) Δ ( S , T ) :
If Ω M , N : = { τ : [ M ] → [ N ] , τ is injective } \Omega_{M,N} := \{\tau: [M] \to[N], \tau\text{ is injective}\} Ω M , N := { τ : [ M ] → [ N ] , τ is injective } , then Δ ( S , T ) = min τ ∈ Ω M , N ∑ i = 1 M ∥ x i − y τ ( i ) ∥ 2 \Delta(S,T) = \min_{\tau\in\Omega_{M,N}} \sum_{i=1}^{M} \lVert x_i-y_{\tau(i)}\rVert_2 Δ ( S , T ) = min τ ∈ Ω M , N ∑ i = 1 M ∥ x i − y τ ( i ) ∥ 2 . Let τ ∗ \tau^* τ ∗ be the optimal coupling between [ M ] [M] [ M ] and [ N ] [N] [ N ] , i.e. τ ∗ = arg min τ ∈ Ω M , N ∑ i = 1 M ∥ x i − y τ ( i ) ∥ 2 \tau^* = \arg\min_{\tau\in\Omega_{M,N}} \sum_{i=1}^{M}\lVert x_i-y_{\tau(i)}\rVert_2 τ ∗ = arg min τ ∈ Ω M , N ∑ i = 1 M ∥ x i − y τ ( i ) ∥ 2 , which means Δ ( S , T ) : = ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 \Delta(S,T) := \sum_{i=1}^{M}\lVert x_i-y_{\tau^*(i)}\rVert_2 Δ ( S , T ) := ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 . Now, ∀ i ∈ [ M ] \forall i \in[M] ∀ i ∈ [ M ] , we define Δ i : = ∥ x i − y τ ∗ ( i ) ∥ 2 \Delta_i := \lVert x_i-y_{\tau^*(i)}\rVert_2 Δ i := ∥ x i − y τ ∗ ( i ) ∥ 2 . For any x ∈ R d x \in\mathbb{R}^d x ∈ R d , r > 0 r>0 r > 0 , let B ( x , r ) B(x,r) B ( x , r ) denote the Euclidean ball centered at x x x with radius r r r . We define Ω i : = { ℓ ∈ [ M ] : y τ ∗ ( ℓ ) ∈ B ( x i , Δ i ) } \Omega_i := \{\ell\in[M] : y_{\tau^*(\ell)}\in B(x_i,\Delta_i)\} Ω i := { ℓ ∈ [ M ] : y τ ∗ ( ℓ ) ∈ B ( x i , Δ i )} , ∀ i ∈ [ M ] \forall i\in[M] ∀ i ∈ [ M ] .
With these notations, we make the following observation:
Claim-1. Consider any r ∈ ( 0 , k ) r \in(0,k) r ∈ ( 0 , k ) . Then, ∃ i 0 ∈ [ M ] \exists i_0 \in[M] ∃ i 0 ∈ [ M ] such that ∑ ℓ ∈ Ω i 0 Δ ℓ ≤ Δ i 0 r \sum_{\ell\in\Omega_{i_0}}\Delta_\ell\le\Delta_{i_0}r ∑ ℓ ∈ Ω i 0 Δ ℓ ≤ Δ i 0 r and Δ i 0 ≥ r k k k + 1 Δ ( S , T ) \Delta_{i_0} \ge\frac{r^k}{k^{k+1}}\Delta(S,T) Δ i 0 ≥ k k + 1 r k Δ ( S , T ) .
Proof. For the sake of contradiction, we assume otherwise, i.e ∀ i ∈ [ M ] \forall i \in[M] ∀ i ∈ [ M ] , we have: either Δ i ≤ r k k k + 1 Δ ( S , T ) \Delta_i \le\frac{r^k}{k^{k+1}}\Delta(S,T) Δ i ≤ k k + 1 r k Δ ( S , T ) or ∑ ℓ ∈ Ω i Δ ℓ ≥ Δ i r \sum_{\ell\in\Omega_i}\Delta_\ell\ge\Delta_i r ∑ ℓ ∈ Ω i Δ ℓ ≥ Δ i r . WLOG, assume the following order on Δ i \Delta_i Δ i ’s: Δ 1 ≥ Δ 2 ≥ ⋯ ≥ Δ M \Delta_1 \ge\Delta_2 \ge\cdots\ge\Delta_M Δ 1 ≥ Δ 2 ≥ ⋯ ≥ Δ M . We build the following directed graph G \mathcal{G} G on [ M ] [M] [ M ] . Start with the node v 0 = 1 v_0=1 v 0 = 1 and add one node to G \mathcal{G} G at a time. At step t t t , we select v t : = arg max u ∈ Ω v t − 1 Δ u v_t := \arg\max_{u\in\Omega_{v_{t-1}}}\Delta_u v t := arg max u ∈ Ω v t − 1 Δ u and add the edge ( v t − 1 , v t ) (v_{t-1},v_t) ( v t − 1 , v t ) to G \mathcal{G} G along with the new node v t v_t v t .
Note that, by definition of Δ ( S , T ) \Delta(S,T) Δ ( S , T ) we have that: Δ ( S , T ) = ∑ i = 1 M Δ i \Delta(S,T) = \sum_{i=1}^{M}\Delta_i Δ ( S , T ) = ∑ i = 1 M Δ i . This, implies that: Δ v 0 ≥ Δ ( S , T ) M \Delta_{v_0} \ge\frac{\Delta(S,T)}{M} Δ v 0 ≥ M Δ ( S , T ) , as we take argmax over entire [ M ] [M] [ M ] . Thus, Δ 1 ≥ Δ ( S , T ) / M ≥ Δ ( S , T ) / k \Delta_1 \ge\Delta(S,T)/M \ge\Delta(S,T)/k Δ 1 ≥ Δ ( S , T ) / M ≥ Δ ( S , T ) / k . Thus, by the contradictory assumption, we must have: ∑ u ∈ Ω v 0 Δ u ≥ Δ 1 r \sum_{u\in\Omega_{v_0}}\Delta_u \ge\Delta_1r ∑ u ∈ Ω v 0 Δ u ≥ Δ 1 r . Since v 1 = arg max u ∈ Ω v 0 Δ u v_1 = \arg\max_{u\in\Omega_{v_0}}\Delta_u v 1 = arg max u ∈ Ω v 0 Δ u , it thus follows that: Δ v 1 ≥ r k Δ 1 ≥ r k 2 Δ ( S , T ) \Delta_{v_1} \ge\frac{r}{k}\Delta_1 \ge\frac{r}{k^2}\Delta(S,T) Δ v 1 ≥ k r Δ 1 ≥ k 2 r Δ ( S , T ) . Following the similar argument, we thus have, Δ v 2 ≥ r k Δ v 1 \Delta_{v_2} \ge\frac{r}{k}\Delta_{v_1} Δ v 2 ≥ k r Δ v 1 and so on. Thus, ∀ t ∈ N \forall t \in\mathbb{N} ∀ t ∈ N we must have: ∑ u ∈ Ω v t Δ u ≥ Δ v t r \sum_{u\in\Omega_{v_t}}\Delta_u \ge\Delta_{v_t}r ∑ u ∈ Ω v t Δ u ≥ Δ v t r . Thus, for v t + 1 v_{t+1} v t + 1 , defined by: arg max u ∈ Ω v t Δ u \arg\max_{u\in\Omega_{v_t}}\Delta_u arg max u ∈ Ω v t Δ u , we must have: Δ v t + 1 ≥ r k Δ v t \Delta_{v_{t+1}} \ge\frac{r}{k}\Delta_{v_t} Δ v t + 1 ≥ k r Δ v t . Since this holds for every t ∈ N t \in\mathbb{N} t ∈ N and since r > 0 r>0 r > 0 by hypothesis, we must have: Δ v t ≥ r k Δ v t − 1 ≥ ( r k ) 2 Δ v t − 1 ≥ ⋯ ≥ ( r k ) t Δ v 0 \Delta_{v_t} \ge\frac{r}{k}\Delta_{v_{t-1}} \ge\left(\frac{r}{k}\right)^2\Delta_{v_{t-1}} \ge\cdots\ge\left(\frac{r}{k}\right)^t\Delta_{v_0} Δ v t ≥ k r Δ v t − 1 ≥ ( k r ) 2 Δ v t − 1 ≥ ⋯ ≥ ( k r ) t Δ v 0 .
As we work with finite M , N M,N M , N either G \mathcal{G} G is a DAG and this process terminates; or G \mathcal{G} G has a cycle. In former case, at the termination step t t t_t t t we cannot add any more nodes, thus Ω t t = ∅ \Omega_{t_t}=\varnothing Ω t t = ∅ . Since all nodes are unique in this case, we must have t t ≤ k t_t\le k t t ≤ k . So, Δ t t ≥ ( r k ) t t Δ v 0 ≥ 1 k ( r k ) t t Δ ( S , T ) \Delta_{t_t} \ge\left(\frac{r}{k}\right)^{t_t}\Delta_{v_0} \ge\frac{1}{k}\left(\frac{r}{k}\right)^{t_t}\Delta(S,T) Δ t t ≥ ( k r ) t t Δ v 0 ≥ k 1 ( k r ) t t Δ ( S , T ) . Since r k ∈ ( 0 , 1 ) \frac{r}{k}\in(0,1) k r ∈ ( 0 , 1 ) by hypothesis and since t t ≤ k t_t\le k t t ≤ k , we thus have: Δ t t ≥ 1 k ( r k ) k Δ ( S , T ) \Delta_{t_t} \ge\frac{1}{k}\left(\frac{r}{k}\right)^k\Delta(S,T) Δ t t ≥ k 1 ( k r ) k Δ ( S , T ) , and hence done with this case.
For the second case, consider the smallest cycle in G \mathcal{G} G . Let this cycle of length p p p be C = { v β → v β + 1 → ⋯ → v β + p − 1 → v β } C = \{v_\beta\to v_{\beta+1}\to\cdots\to v_{\beta+p-1}\to v_\beta\} C = { v β → v β + 1 → ⋯ → v β + p − 1 → v β } for some β ≥ 0 \beta\ge0 β ≥ 0 . Then, we have: β + i + 1 ∈ Ω β + i \beta+i+1\in\Omega_{\beta+i} β + i + 1 ∈ Ω β + i , ∀ i ∈ { 0 , ⋯ , p − 1 } \forall i\in\{0,\cdots,p-1\} ∀ i ∈ { 0 , ⋯ , p − 1 } , where addition of indices is modulo p p p . Thus we have: ∥ x v β + i − y τ ∗ ( v β + i + 1 ) ∥ 2 < ∥ x v β + i − y τ ∗ ( v β + i ) ∥ 2 \lVert x_{v_{\beta+i}}-y_{\tau^*(v_{\beta+i+1})}\rVert_2 < \lVert x_{v_{\beta+i}}-y_{\tau^*(v_{\beta+i})}\rVert_2 ∥ x v β + i − y τ ∗ ( v β + i + 1 ) ∥ 2 < ∥ x v β + i − y τ ∗ ( v β + i ) ∥ 2 , ∀ i ∈ { 0 , ⋯ , p − 1 } \forall i\in\{0,\cdots,p-1\} ∀ i ∈ { 0 , ⋯ , p − 1 } . Taking sum, we have the following: ∥ x v β − y τ ∗ ( v β + 1 ) ∥ 2 + ∥ x v β + 1 − y τ ∗ ( v β + 2 ) ∥ 2 + ⋯ + ∥ x v β + p − 1 − y τ ∗ ( v β ) ∥ 2 < ∑ j = 0 p − 1 ∥ x v β + j − y τ ∗ ( v β + j ) ∥ 2 \lVert x_{v_\beta}-y_{\tau^*(v_{\beta+1})}\rVert_2+\lVert x_{v_{\beta+1}}-y_{\tau^*(v_{\beta+2})}\rVert_2+\cdots+\lVert x_{v_{\beta+p-1}}-y_{\tau^*(v_\beta)}\rVert_2 < \sum_{j=0}^{p-1}\lVert x_{v_{\beta+j}}-y_{\tau^*(v_{\beta+j})}\rVert_2 ∥ x v β − y τ ∗ ( v β + 1 ) ∥ 2 + ∥ x v β + 1 − y τ ∗ ( v β + 2 ) ∥ 2 + ⋯ + ∥ x v β + p − 1 − y τ ∗ ( v β ) ∥ 2 < ∑ j = 0 p − 1 ∥ x v β + j − y τ ∗ ( v β + j ) ∥ 2 . Hence, switching from the coupling i ↦ τ ∗ ( i ) i\mapsto\tau^*(i) i ↦ τ ∗ ( i ) to i ↦ τ ∗ ( i + 1 ) i\mapsto\tau^*(i+1) i ↦ τ ∗ ( i + 1 ) for i ∈ { v β , ⋯ , v β + p − 2 } i\in\{v_\beta,\cdots,v_{\beta+p-2}\} i ∈ { v β , ⋯ , v β + p − 2 } and from v β + p − 1 ↦ τ ∗ ( v β + p − 1 ) v_\beta+p-1\mapsto\tau^*(v_\beta+p-1) v β + p − 1 ↦ τ ∗ ( v β + p − 1 ) to v β + p − 1 ↦ τ ∗ ( v β ) v_\beta+p-1\mapsto\tau^*(v_\beta) v β + p − 1 ↦ τ ∗ ( v β ) strictly reduces the value of the sum ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 \sum_{i=1}^{M}\lVert x_i-y_{\tau^*(i)}\rVert_2 ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 , but we started with the optimal coupling τ ∗ \tau^* τ ∗ . This gives the reqd. contradiction. □ \square □
We also make the following observation regarding this ”optimal coupling” τ ∗ \tau^* τ ∗ :
Observation-2: For any given i ∈ [ M ] i \in[M] i ∈ [ M ] , if y j ∈ T ∩ Ω i y_j \in T \cap\Omega_i y j ∈ T ∩ Ω i for some j ∈ [ N ] j \in[N] j ∈ [ N ] , then ∃ ℓ ∈ [ M ] \exists\ell\in[M] ∃ ℓ ∈ [ M ] such that j = τ ∗ ( ℓ ) j = \tau^{*}(\ell) j = τ ∗ ( ℓ )
Proof. The proof follows from the following observation: if y j ∈ Ω i y_j \in\Omega_i y j ∈ Ω i for any i ∈ [ M ] i \in[M] i ∈ [ M ] , then we must have: ∥ x i − y j ∥ 2 ≤ Δ i = ∥ x i − y τ ( i ) ∥ 2 \lVert x_i-y_j\rVert_2 \leq\Delta_i = \lVert x_i-y_{\tau(i)}\rVert_2 ∥ x i − y j ∥ 2 ≤ Δ i = ∥ x i − y τ ( i ) ∥ 2 . Thus, if y j y_j y j was ”free”, i.e if ∄ ℓ ∈ [ M ] : j = τ ∗ ( ℓ ) \nexists\ell\in[M]:j=\tau^{*}(\ell) ∄ ℓ ∈ [ M ] : j = τ ∗ ( ℓ ) , then we can switch from i ↦ τ ∗ ( i ) i \mapsto\tau^{*}(i) i ↦ τ ∗ ( i ) to i ↦ j i \mapsto j i ↦ j , and this would reduce the sum ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 \sum_{i=1}^{M}\lVert x_i-y_{\tau^{*}(i)}\rVert_2 ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 , but τ ∗ \tau^{*} τ ∗ was the optimal coupling. This gives a contradiction
Now, as per the starting discussion, we consider supp ( σ ) = [ 0 , 1 ] \operatorname{supp}(\sigma)=[0,1] supp ( σ ) = [ 0 , 1 ] , and the proof for general support of [ γ 1 , γ 2 ] [\gamma_1,\gamma_2] [ γ 1 , γ 2 ] requires the scaling and shifting as mentioned before.
By assumption we have, lim t → 0 + σ ′ ( t ) > 0 \lim_{t\to0^{+}}\sigma'(t)>0 lim t → 0 + σ ′ ( t ) > 0 . We define κ 1 : = lim t → 0 + σ ′ ( t ) 2 \kappa_1:=\frac{\lim_{t\to0^{+}}\sigma'(t)}{2} κ 1 := 2 l i m t → 0 + σ ′ ( t ) . Since σ \sigma σ is piecewise continuously differentiable as per assumption, we must have σ ′ ( t ) ≥ κ 1 , ∀ t ∈ ( 0 , ω ) \sigma'(t)\geq\kappa_1,\ \forall t\in(0,\omega) σ ′ ( t ) ≥ κ 1 , ∀ t ∈ ( 0 , ω ) for some ω ∈ ( 0 , 1 ) \omega\in(0,1) ω ∈ ( 0 , 1 ) by the continuity of σ ′ \sigma' σ ′ . Note that, by Lagrange’s Mean Value Theorem, we have that:
∀ x , y ∈ ( 0 , ω ) , ∣ σ ( x ) − σ ( y ) ∣ ∣ x − y ∣ = σ ′ ( z ) , where z ∈ ( x , y ) ⟹ ∀ x , y ∈ ( 0 , ω ) , ∣ σ ( x ) − σ ( y ) ∣ ≥ κ 1 ∣ x − y ∣ , since z ∈ ( 0 , ω ) ⟹ lim y → 0 + ∣ σ ( x ) − σ ( y ) ∣ ≥ κ 1 ∣ x ∣ , as σ is cts. ⟹ σ ( x ) ≥ κ 1 x , ∀ x ∈ ( 0 , ω ) , as σ ≥ 0 \begin{aligned}
&\forall x,y\in(0,\omega),\frac{|\sigma(x)-\sigma(y)|}{|x-y|}=\sigma'(z),\text{ where }z\in(x,y)\\
&\implies\forall x,y\in(0,\omega),|\sigma(x)-\sigma(y)|\geq\kappa_1|x-y|,\text{ since }z\in(0,\omega)\\
&\implies\lim_{y\to0^{+}}|\sigma(x)-\sigma(y)|\geq\kappa_1|x|,\text{ as }\sigma\text{ is cts.}\\
&\implies\sigma(x)\geq\kappa_1x,\ \forall x\in(0,\omega),\text{ as }\sigma\geq0
\end{aligned} ∀ x , y ∈ ( 0 , ω ) , ∣ x − y ∣ ∣ σ ( x ) − σ ( y ) ∣ = σ ′ ( z ) , where z ∈ ( x , y ) ⟹ ∀ x , y ∈ ( 0 , ω ) , ∣ σ ( x ) − σ ( y ) ∣ ≥ κ 1 ∣ x − y ∣ , since z ∈ ( 0 , ω ) ⟹ y → 0 + lim ∣ σ ( x ) − σ ( y ) ∣ ≥ κ 1 ∣ x ∣ , as σ is cts. ⟹ σ ( x ) ≥ κ 1 x , ∀ x ∈ ( 0 , ω ) , as σ ≥ 0
Also, σ \sigma σ is upper Lipschitz with constant κ 2 > 0 \kappa_2>0 κ 2 > 0 per definition of Hat fn. Now, consider the i 0 i_0 i 0 coming from the Claim as proved above. We define ω ′ : = min ( ω 1 − ω , 1 ) \omega':=\min\left(\frac{\omega}{1-\omega},1\right) ω ′ := min ( 1 − ω ω , 1 ) . Let P ∈ N P\in\mathbb{N} P ∈ N be such that κ 1 P < 1 2 \frac{\kappa_1}{P}<\frac{1}{2} P κ 1 < 2 1 and κ 1 κ 2 P < k \frac{\kappa_1}{\kappa_2P}<k κ 2 P κ 1 < k and ω ′ > 2 P \omega'>\frac{2}{P} ω ′ > P 2 . Note that, such a P ∈ N P\in\mathbb{N} P ∈ N exists, since taking P → ∞ P\to\infty P → ∞ satisfies all the 3 conditions.
Now, for a given a ∈ S d − 1 a\in S^{d-1} a ∈ S d − 1 , consider the sets S a = { a ⊤ x : x ∈ S } , T a = { a ⊤ y : y ∈ T } S_a=\{a^{\top}x:x\in S\},\ T_a=\{a^{\top}y:y\in T\} S a = { a ⊤ x : x ∈ S } , T a = { a ⊤ y : y ∈ T } . Note that, since we have ∥ x i ∥ , ∥ y j ∥ ≤ 1 , ∀ i ∈ [ M ] , j ∈ [ N ] \lVert x_i\rVert,\lVert y_j\rVert\leq1,\ \forall i\in[M],j\in[N] ∥ x i ∥ , ∥ y j ∥ ≤ 1 , ∀ i ∈ [ M ] , j ∈ [ N ] as per earlier assumption, we get S a , T a ⊂ ( − 1 , 1 ) S_a,T_a\subset(-1,1) S a , T a ⊂ ( − 1 , 1 ) . For given a ∈ S d − 1 a\in S^{d-1} a ∈ S d − 1 , we define the ”optimal coupling” between the real-valued sets S a , T a S_a,T_a S a , T a as: τ a ∗ : = arg min τ ∈ Ω M , N ∑ i = 1 M ∣ a ⊤ x i − a ⊤ y τ ( i ) ∣ \tau_a^{*}:=\arg\min_{\tau\in\Omega_{M,N}}\sum_{i=1}^{M}|a^{\top}x_i-a^{\top}y_{\tau(i)}| τ a ∗ := arg min τ ∈ Ω M , N ∑ i = 1 M ∣ a ⊤ x i − a ⊤ y τ ( i ) ∣ . Based on this, we define Δ i a : = ∣ a ⊤ x i − a ⊤ y τ a ∗ ( i ) ∣ \Delta_i^a:=|a^{\top}x_i-a^{\top}y_{\tau_a^{*}(i)}| Δ i a := ∣ a ⊤ x i − a ⊤ y τ a ∗ ( i ) ∣ and Ω i a = { ℓ ∈ [ M ] : y τ a ∗ ( ℓ ) ∈ B ( a ⊤ x i , Δ i a ) } \Omega_i^a=\{\ell\in[M]:y_{\tau_a^{*}(\ell)}\in B(a^{\top}x_i,\Delta_i^a)\} Ω i a = { ℓ ∈ [ M ] : y τ a ∗ ( ℓ ) ∈ B ( a ⊤ x i , Δ i a )} . We have: Δ ( S a , T a ) : = ∑ i = 1 M Δ i a \Delta(S_a,T_a):=\sum_{i=1}^{M}\Delta_i^a Δ ( S a , T a ) := ∑ i = 1 M Δ i a . Let i 0 ∈ [ M ] i_0\in[M] i 0 ∈ [ M ] be the index that comes from applying the claim to the sets S a , T a S_a,T_a S a , T a . Now, we define the regions B a : = ( a ⊤ x i 0 − ω ′ Δ i 0 a , a ⊤ x i 0 − Δ i 0 a P ) B_a:=(a^{\top}x_{i_0}-\omega'\Delta_{i_0}^a,a^{\top}x_{i_0}-\frac{\Delta_{i_0}^a}{P}) B a := ( a ⊤ x i 0 − ω ′ Δ i 0 a , a ⊤ x i 0 − P Δ i 0 a ) and for a given pair ( a ∈ S d − 1 , b ∈ R ) (a\in S^{d-1},b\in\mathbb{R}) ( a ∈ S d − 1 , b ∈ R ) , we consider C a , b : = ( a ⊤ x i 0 − b ω , a ⊤ x i 0 + Δ i 0 a − b ) C_{a,b}:=(\frac{a^{\top}x_{i_0}-b}{\omega},a^{\top}x_{i_0}+\Delta_{i_0}^a-b) C a , b := ( ω a ⊤ x i 0 − b , a ⊤ x i 0 + Δ i 0 a − b ) . Thus, we have:
( a T x i 0 + Δ i 0 a − b ) − ( a T x i 0 − b ω ) = ( 1 − ω ) ( b − a T x i 0 ) + ω Δ i 0 a ω = 1 − ω ω ( ( b − a T x i 0 ) + ω 1 − ω Δ i 0 a ) ≥ 1 − ω ω ( b − ( a T x i 0 − ω ′ Δ i 0 a ) ) > 0 , ∀ b > a T x i 0 − ω ′ Δ i 0 a \begin{aligned}
\left(a^{\mathsf T}x_{i_0}+\Delta^{a}_{i_0}-b\right)-\left(\frac{a^{\mathsf T}x_{i_0}-b}{\omega}\right)
&=\frac{(1-\omega)(b-a^{\mathsf T}x_{i_0})+\omega\Delta^{a}_{i_0}}{\omega}\\
&=\frac{1-\omega}{\omega}\left((b-a^{\mathsf T}x_{i_0})+\frac{\omega}{1-\omega}\Delta^{a}_{i_0}\right)\\
&\ge\frac{1-\omega}{\omega}\left(b-\left(a^{\mathsf T}x_{i_0}-\omega'\Delta^{a}_{i_0}\right)\right)>0,\ \forall b>a^{\mathsf T}x_{i_0}-\omega'\Delta^{a}_{i_0}
\end{aligned} ( a T x i 0 + Δ i 0 a − b ) − ( ω a T x i 0 − b ) = ω ( 1 − ω ) ( b − a T x i 0 ) + ω Δ i 0 a = ω 1 − ω ( ( b − a T x i 0 ) + 1 − ω ω Δ i 0 a ) ≥ ω 1 − ω ( b − ( a T x i 0 − ω ′ Δ i 0 a ) ) > 0 , ∀ b > a T x i 0 − ω ′ Δ i 0 a
The above analysis shows that C a , b ≠ ∅ \mathcal{C}_{a,b}\ne\varnothing C a , b = ∅ , ∀ b ∈ B a \forall b\in B_a ∀ b ∈ B a , so the intervals are well defined. Now, if b ∈ B a b\in B_a b ∈ B a and y < a T x i 0 − Δ i 0 a y<a^{\mathsf T}x_{i_0}-\Delta^{a}_{i_0} y < a T x i 0 − Δ i 0 a then we have: y − b < a T x i 0 − Δ i 0 a − a T x i 0 + ω ′ Δ i 0 a = ( ω ′ − 1 ) Δ i 0 a ≤ 0 y-b<a^{\mathsf T}x_{i_0}-\Delta^{a}_{i_0}-a^{\mathsf T}x_{i_0}+\omega'\Delta^{a}_{i_0}=(\omega'-1)\Delta^{a}_{i_0}\le0 y − b < a T x i 0 − Δ i 0 a − a T x i 0 + ω ′ Δ i 0 a = ( ω ′ − 1 ) Δ i 0 a ≤ 0 by definition of ω ′ \omega' ω ′ . On the other hand, if b ∈ B a b\in B_a b ∈ B a and c ∈ C a , b c\in C_{a,b} c ∈ C a , b , for given any y > a T x i 0 + Δ i 0 a y>a^{\mathsf T}x_{i_0}+\Delta^{a}_{i_0} y > a T x i 0 + Δ i 0 a we have y − b c ≥ a T x i 0 − b + Δ i 0 a a T x i 0 − b + Δ i 0 a = 1 \frac{y-b}{c}\ge\frac{a^{\mathsf T}x_{i_0}-b+\Delta^{a}_{i_0}}{a^{\mathsf T}x_{i_0}-b+\Delta^{a}_{i_0}}=1 c y − b ≥ a T x i 0 − b + Δ i 0 a a T x i 0 − b + Δ i 0 a = 1 . Combining these, we get that if ∣ y − a T x i 0 ∣ ≥ Δ i 0 a \left|y-a^{\mathsf T}x_{i_0}\right|\ge\Delta^{a}_{i_0} y − a T x i 0 ≥ Δ i 0 a , then y − b c ∉ ( 0 , 1 ) \frac{y-b}{c}\notin(0,1) c y − b ∈ / ( 0 , 1 ) , ∀ b ∈ B a , c ∈ C a , b \forall b\in B_a,c\in C_{a,b} ∀ b ∈ B a , c ∈ C a , b . Thus, conditioning on a given a ∈ S d − 1 a\in\mathcal{S}^{d-1} a ∈ S d − 1 , we can write the conditional expectation taken over b , c b,c b , c as follows:
E b , c [ ∑ x ∈ S σ ( a T x − b c ) − ∑ y ∈ T σ ( a T y − b c ) ] + ≥ E b , c [ ( ∑ x ∈ S σ ( a T x − b c ) − ∑ y ∈ T σ ( a T y − b c ) ) 1 { b ∈ B a , c ∈ C a , b } ] + , using non-negative RV ≥ 1 4 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 ∫ c = a T x i 0 − b ω a T x i 0 + Δ i 0 a − b [ σ ( a T x i 0 − b c ) − ∑ ℓ ∈ Ω i 0 a ∣ σ ( a T y π a ( ℓ ) − b c ) − σ ( a T x ℓ − b c ) ∣ ] + d c d b ≥ 1 4 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 ∫ c = a T x i 0 − b ω a T x i 0 + Δ i 0 a − b [ κ 1 a T x i 0 − b c − κ 2 Δ i 0 a P κ 2 c ] + d c d b , from Lipschitz cts. σ = κ 1 4 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 [ a T x i 0 − b − Δ i 0 a P ] ∫ c = a T x i 0 − b ω a T x i 0 + Δ i 0 a − b d c c d b , as ReLU ( r x ) = r ReLU ( x ) = κ 1 4 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 − Δ i 0 a P ( a T x i 0 − b − Δ i 0 a P ) ln ( ω ( a T x i 0 + Δ i 0 a − b ) a T x i 0 − b ) d b = κ 1 4 ∫ z = Δ i 0 a P ω ′ Δ i 0 a ( z − Δ i 0 a P ) ln ( ω ( z + Δ i 0 a ) z ) d z , putting z : = a T x i 0 − b ≥ κ 1 4 ∫ z = Δ i 0 a P ω ′ Δ i 0 a 2 ( z − Δ i 0 a P ) ln ( ω ( 1 + Δ i 0 a z ) ) d z , using non-negativity of integrand ≥ κ 1 4 ln ( ω ( 1 + 2 ω ′ ) ) ∫ z = Δ i 0 a P ω ′ Δ i 0 a 2 ( z − Δ i 0 a P ) d z ≥ κ 1 8 ln ( 2 − ω ) ( ω ′ 2 − 1 P ) 2 ( Δ i 0 a ) 2 , using ω ′ ≤ ω 1 − ω ≥ C ⋅ 1 k ( κ 1 κ 2 P ) k ( Δ a ( S , T ) ) 2 , where C > 0 as ω ∈ ( 0 , 1 ) and P ≥ 2 ω ′ by earlier choice \begin{aligned}
\mathbb{E}_{b,c}\left[\sum_{x\in S}\sigma\left(\frac{a^{\mathsf T}x-b}{c}\right)-\sum_{y\in T}\sigma\left(\frac{a^{\mathsf T}y-b}{c}\right)\right]_{+}
&\ge\mathbb{E}_{b,c}\left[\left(\sum_{x\in S}\sigma\left(\frac{a^{\mathsf T}x-b}{c}\right)-\sum_{y\in T}\sigma\left(\frac{a^{\mathsf T}y-b}{c}\right)\right)\mathbf{1}_{\{b\in B_a,c\in C_{a,b}\}}\right]_{+},\ \text{using non-negative RV}\\
&\ge\frac{1}{4}\int_{b=a^{\mathsf T}x_{i_0}-\omega'\Delta^{a}_{i_0}}^{a^{\mathsf T}x_{i_0}}\int_{c=\frac{a^{\mathsf T}x_{i_0}-b}{\omega}}^{a^{\mathsf T}x_{i_0}+\Delta^{a}_{i_0}-b}
\left[\sigma\left(\frac{a^{\mathsf T}x_{i_0}-b}{c}\right)-\sum_{\ell\in\Omega^{a}_{i_0}}\left|\sigma\left(\frac{a^{\mathsf T}y_{\pi_a(\ell)}-b}{c}\right)-\sigma\left(\frac{a^{\mathsf T}x_{\ell}-b}{c}\right)\right|\right]_{+}\,dc\,db\\
&\ge\frac{1}{4}\int_{b=a^{\mathsf T}x_{i_0}-\omega'\Delta^{a}_{i_0}}^{a^{\mathsf T}x_{i_0}}\int_{c=\frac{a^{\mathsf T}x_{i_0}-b}{\omega}}^{a^{\mathsf T}x_{i_0}+\Delta^{a}_{i_0}-b}
\left[\kappa_1\frac{a^{\mathsf T}x_{i_0}-b}{c}-\kappa_2\frac{\Delta^{a}_{i_0}}{P_{\kappa_2}c}\right]_{+}\,dc\,db,\ \text{from Lipschitz cts. }\sigma\\
&=\frac{\kappa_1}{4}\int_{b=a^{\mathsf T}x_{i_0}-\omega'\Delta^{a}_{i_0}}^{a^{\mathsf T}x_{i_0}}\left[a^{\mathsf T}x_{i_0}-b-\frac{\Delta^{a}_{i_0}}{P}\right]\int_{c=\frac{a^{\mathsf T}x_{i_0}-b}{\omega}}^{a^{\mathsf T}x_{i_0}+\Delta^{a}_{i_0}-b}\frac{dc}{c}\,db,\ \text{as ReLU}(rx)=r\operatorname{ReLU}(x)\\
&=\frac{\kappa_1}{4}\int_{b=a^{\mathsf T}x_{i_0}-\omega'\Delta^{a}_{i_0}}^{a^{\mathsf T}x_{i_0}-\frac{\Delta^{a}_{i_0}}{P}}\left(a^{\mathsf T}x_{i_0}-b-\frac{\Delta^{a}_{i_0}}{P}\right)\ln\left(\frac{\omega(a^{\mathsf T}x_{i_0}+\Delta^{a}_{i_0}-b)}{a^{\mathsf T}x_{i_0}-b}\right)\,db\\
&=\frac{\kappa_1}{4}\int_{z=\frac{\Delta^{a}_{i_0}}{P}}^{\omega'\Delta^{a}_{i_0}}\left(z-\frac{\Delta^{a}_{i_0}}{P}\right)\ln\left(\frac{\omega(z+\Delta^{a}_{i_0})}{z}\right)\,dz,\ \text{putting }z:=a^{\mathsf T}x_{i_0}-b\\
&\ge\frac{\kappa_1}{4}\int_{z=\frac{\Delta^{a}_{i_0}}{P}}^{\frac{\omega'\Delta^{a}_{i_0}}{2}}\left(z-\frac{\Delta^{a}_{i_0}}{P}\right)\ln\left(\omega\left(1+\frac{\Delta^{a}_{i_0}}{z}\right)\right)\,dz,\ \text{using non-negativity of integrand}\\
&\ge\frac{\kappa_1}{4}\ln\left(\omega\left(1+\frac{2}{\omega'}\right)\right)\int_{z=\frac{\Delta^{a}_{i_0}}{P}}^{\frac{\omega'\Delta^{a}_{i_0}}{2}}\left(z-\frac{\Delta^{a}_{i_0}}{P}\right)\,dz \\
&\ge\frac{\kappa_1}{8}\ln(2-\omega)\left(\frac{\omega'}{2}-\frac{1}{P}\right)^2\left(\Delta^a_{i0}\right)^2,\ \text{using }\omega'\le\frac{\omega}{1-\omega} \\
&\ge C\cdot\frac{1}{k}\left(\frac{\kappa_1}{\kappa_2P}\right)^k\left(\Delta^a(S,T)\right)^2,\ \text{where }C>0\text{ as }\omega\in(0,1)\text{ and }P\ge\frac{2}{\omega'}\text{ by earlier choice}
\end{aligned} E b , c x ∈ S ∑ σ ( c a T x − b ) − y ∈ T ∑ σ ( c a T y − b ) + ≥ E b , c x ∈ S ∑ σ ( c a T x − b ) − y ∈ T ∑ σ ( c a T y − b ) 1 { b ∈ B a , c ∈ C a , b } + , using non-negative RV ≥ 4 1 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 ∫ c = ω a T x i 0 − b a T x i 0 + Δ i 0 a − b σ ( c a T x i 0 − b ) − ℓ ∈ Ω i 0 a ∑ σ ( c a T y π a ( ℓ ) − b ) − σ ( c a T x ℓ − b ) + d c d b ≥ 4 1 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 ∫ c = ω a T x i 0 − b a T x i 0 + Δ i 0 a − b [ κ 1 c a T x i 0 − b − κ 2 P κ 2 c Δ i 0 a ] + d c d b , from Lipschitz cts. σ = 4 κ 1 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 [ a T x i 0 − b − P Δ i 0 a ] ∫ c = ω a T x i 0 − b a T x i 0 + Δ i 0 a − b c d c d b , as ReLU ( r x ) = r ReLU ( x ) = 4 κ 1 ∫ b = a T x i 0 − ω ′ Δ i 0 a a T x i 0 − P Δ i 0 a ( a T x i 0 − b − P Δ i 0 a ) ln ( a T x i 0 − b ω ( a T x i 0 + Δ i 0 a − b ) ) d b = 4 κ 1 ∫ z = P Δ i 0 a ω ′ Δ i 0 a ( z − P Δ i 0 a ) ln ( z ω ( z + Δ i 0 a ) ) d z , putting z := a T x i 0 − b ≥ 4 κ 1 ∫ z = P Δ i 0 a 2 ω ′ Δ i 0 a ( z − P Δ i 0 a ) ln ( ω ( 1 + z Δ i 0 a ) ) d z , using non-negativity of integrand ≥ 4 κ 1 ln ( ω ( 1 + ω ′ 2 ) ) ∫ z = P Δ i 0 a 2 ω ′ Δ i 0 a ( z − P Δ i 0 a ) d z ≥ 8 κ 1 ln ( 2 − ω ) ( 2 ω ′ − P 1 ) 2 ( Δ i 0 a ) 2 , using ω ′ ≤ 1 − ω ω ≥ C ⋅ k 1 ( κ 2 P κ 1 ) k ( Δ a ( S , T ) ) 2 , where C > 0 as ω ∈ ( 0 , 1 ) and P ≥ ω ′ 2 by earlier choice
Thus, we have shown that: E b , c [ [ F ( S ) − F ( T ) ] + ∣ a ] ≥ C ′ ⋅ ( Δ a ( S , T ) ) 2 \mathbb{E}_{b,c}\left[\left[F(S)-F(T)\right]_+ \mid a\right]\ge C'\cdot\left(\Delta^a(S,T)\right)^2 E b , c [ [ F ( S ) − F ( T ) ] + ∣ a ] ≥ C ′ ⋅ ( Δ a ( S , T ) ) 2 . By law of total expectation, we have:
E a , b , c [ F ( S ) − F ( T ) ] + = E a E b , c [ [ F ( S ) − F ( T ) ] + ∣ a ] ≥ C ′ E a ( Δ a ( S , T ) ) 2 ≥ C ′ ( E a Δ a ( S , T ) ) 2 , by Jensen’s ineq. \begin{aligned}
\mathbb{E}_{a,b,c}\left[F(S)-F(T)\right]_+=\mathbb{E}_a\mathbb{E}_{b,c}\left[\left[F(S)-F(T)\right]_+\mid a\right] \\
&\ge C'\mathbb{E}_a\left(\Delta^a(S,T)\right)^2\ge C'\left(\mathbb{E}_a\Delta^a(S,T)\right)^2,\ \text{by Jensen’s ineq.}
\end{aligned} E a , b , c [ F ( S ) − F ( T ) ] + = E a E b , c [ [ F ( S ) − F ( T ) ] + ∣ a ] ≥ C ′ E a ( Δ a ( S , T ) ) 2 ≥ C ′ ( E a Δ a ( S , T ) ) 2 , by Jensen’s ineq.
Thus, to show Lower Hölder separability with exponent λ = 2 \lambda=2 λ = 2 , it remains to show that, E a ∼ Unif ( S d − 1 ) Δ a ( S , T ) ≥ c 1 ⋅ Δ ( S , T ) \mathbb{E}_{a\sim\operatorname{Unif}(\mathbb{S}^{d-1})}\Delta^a(S,T)\ge c_1\cdot\Delta(S,T) E a ∼ Unif ( S d − 1 ) Δ a ( S , T ) ≥ c 1 ⋅ Δ ( S , T ) , where c 1 > 0 c_1>0 c 1 > 0 is a constant. Now, we make the following observation:
Observation-3. Suppose we’re given ℓ ∈ N \ell\in\mathbb{N} ℓ ∈ N and ℓ \ell ℓ many non-zero vectors u 1 , u 2 , … , u ℓ ∈ R d u_1,u_2,\ldots,u_\ell\in\mathbb{R}^d u 1 , u 2 , … , u ℓ ∈ R d . If a ∼ S d − 1 a\sim\mathbb{S}^{d-1} a ∼ S d − 1 , then ∃ δ ( ℓ ) > 0 \exists\delta(\ell)>0 ∃ δ ( ℓ ) > 0 such that: P { ∣ a T u i ∣ ≥ δ ( ℓ ) ∥ u i ∥ , ∀ i ∈ [ ℓ ] } ≥ 1 2 \mathbb{P}\left\{\left|a^\mathsf{T}u_i\right|\ge\delta(\ell)\lVert u_i\rVert,\forall i\in[\ell]\right\}\ge\frac{1}{2} P { a T u i ≥ δ ( ℓ ) ∥ u i ∥ , ∀ i ∈ [ ℓ ] } ≥ 2 1
Proof. Firstly, we note that, for a fixed i ∈ [ ℓ ] i\in[\ell] i ∈ [ ℓ ] , ∣ a T u i ∣ \left|a^\mathsf{T}u_i\right| a T u i is a continuous non-negative real-valued random variable. Thus, ∃ t i ∈ R + \exists t_i\in\mathbb{R}_+ ∃ t i ∈ R + such that P { ∣ a T u i ∣ ≥ t i } ≥ 1 2 \mathbb{P}\left\{\left|a^\mathsf{T}u_i\right|\ge t_i\right\}\ge\frac{1}{2} P { a T u i ≥ t i } ≥ 2 1 . Now, Choosing δ i : = t i ∥ u i ∥ > 0 \delta_i:=\frac{t_i}{\lVert u_i\rVert}>0 δ i := ∥ u i ∥ t i > 0 gives us: P { ∣ a T u i ∣ ≥ δ i ∥ u i ∥ } ≥ 1 2 , ∀ i ∈ [ ℓ ] \mathbb{P}\left\{\left|a^\mathsf{T}u_i\right|\ge\delta_i\lVert u_i\rVert\right\}\ge\frac{1}{2},\forall i\in[\ell] P { a T u i ≥ δ i ∥ u i ∥ } ≥ 2 1 , ∀ i ∈ [ ℓ ] . Since δ i > 0 , ∀ i ∈ [ ℓ ] \delta_i>0,\forall i\in[\ell] δ i > 0 , ∀ i ∈ [ ℓ ] and ℓ \ell ℓ is finite, we thus have: δ : = min i ∈ [ ℓ ] δ i > 0 \delta:=\min_{i\in[\ell]}\delta_i>0 δ := min i ∈ [ ℓ ] δ i > 0 . For this particular choice of δ \delta δ , we have that P { ∣ a T u i ∣ ≥ δ ∥ u i ∥ } ≥ 1 2 , ∀ i ∈ [ ℓ ] \mathbb{P}\left\{\left|a^\mathsf{T}u_i\right|\ge\delta\lVert u_i\rVert\right\}\ge\frac{1}{2},\forall i\in[\ell] P { a T u i ≥ δ ∥ u i ∥ } ≥ 2 1 , ∀ i ∈ [ ℓ ] , and the observation is proved. □ \square □
. Now, consider all ℓ 0 : = ( N M ) M ! \ell_0:=\binom{N}{M}M! ℓ 0 := ( M N ) M ! vectors ( x i − y τ ( i ) ) ∈ R d (x_i-y_{\tau(i)})\in\mathbb{R}^d ( x i − y τ ( i ) ) ∈ R d , where τ \tau τ runs over all injective functions from [ M ] → [ N ] [M]\to[N] [ M ] → [ N ] . For this particular choice of ℓ 0 \ell_0 ℓ 0 , we get from the last observation that, ∃ δ ( N , M ) > 0 \exists\delta(N,M)>0 ∃ δ ( N , M ) > 0 such that P { ∣ a T ( x i − y τ ( i ) ) ∣ ≥ δ ( N , M ) ∥ x i − y τ ( i ) ∥ } ≥ 1 2 \mathbb{P}\left\{\left|a^\mathsf{T}\left(x_i-y_{\tau(i)}\right)\right|\ge\delta(N,M)\lVert x_i-y_{\tau(i)}\rVert\right\}\ge\frac{1}{2} P { a T ( x i − y τ ( i ) ) ≥ δ ( N , M ) ∥ x i − y τ ( i ) ∥ } ≥ 2 1 , for all possible injective τ : [ M ] → [ N ] \tau:[M]\to[N] τ : [ M ] → [ N ] , we call this set of a ∈ S d − 1 a\in\mathbb{S}^{d-1} a ∈ S d − 1 to be A ⊆ S d − 1 A\subseteq\mathbb{S}^{d-1} A ⊆ S d − 1 . Now, given a fixed a ∈ S d − 1 a\in\mathbb{S}^{d-1} a ∈ S d − 1 , we have a particular τ a : [ M ] → [ N ] \tau_a:[M]\to[N] τ a : [ M ] → [ N ] that gives Δ a ( S , T ) = ∑ i = 2 M ∣ a T ( x i − y τ a ( i ) ) ∣ \Delta^a(S,T)=\sum_{i=2}^{M}\left|a^\mathsf{T}\left(x_i-y_{\tau_a(i)}\right)\right| Δ a ( S , T ) = ∑ i = 2 M a T ( x i − y τ a ( i ) ) . Also, let τ ∗ : [ M ] → [ N ] \tau^*:[M]\to[N] τ ∗ : [ M ] → [ N ] be the ”optimal coupling” such that Δ ( S , T ) = ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ \Delta(S,T)=\sum_{i=1}^{M}\lVert x_i-y_{\tau^*(i)}\rVert Δ ( S , T ) = ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ . Now, we can write:
E a ∼ Unif ( S d − 1 ) Δ a ( S , T ) = E a ∼ Unif ( S d − 1 ) [ ∑ i = 1 M ∣ a ⊤ ( x i − y τ a ( i ) ) ∣ ] ≥ E a ∼ Unif ( S d − 1 ) [ ∑ i = 1 M ∣ a ⊤ ( x i − y τ a ( i ) ) ∣ ∣ a ∈ A ] P { a ∈ A } ≥ δ ( N , M ) 2 E a [ ∑ i = 1 M ∥ x i − y τ a ( i ) ∥ 2 ] ≥ δ ( N , M ) 2 E a [ ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 ] , as τ ∗ is optimal coupling = δ ( N , M ) 2 ( ∑ i = 1 M ∥ x i − y τ ∗ ( i ) ∥ 2 ) = δ ( N , M ) 2 Δ ( S , T ) \begin{aligned}
\mathbb{E}_{a\sim\operatorname{Unif}(\mathbb{S}^{d-1})}\Delta^a(S,T)
&=\mathbb{E}_{a\sim\operatorname{Unif}(\mathbb{S}^{d-1})}\left[\sum_{i=1}^{M}\left|a^\top(x_i-y_{\tau_a(i)})\right|\right]\\
&\geq\mathbb{E}_{a\sim\operatorname{Unif}(\mathbb{S}^{d-1})}\left[\sum_{i=1}^{M}\left|a^\top(x_i-y_{\tau_a(i)})\right|\mid a\in A\right]\mathbb{P}\{a\in A\}\\
&\geq\frac{\delta(N,M)}{2}\mathbb{E}_a\left[\sum_{i=1}^{M}\lVert x_i-y_{\tau_a(i)}\rVert_2\right]\\
&\geq\frac{\delta(N,M)}{2}\mathbb{E}_a\left[\sum_{i=1}^{M}\lVert x_i-y_{\tau^*(i)}\rVert_2\right],\text{ as }\tau^*\text{ is optimal coupling}\\
&=\frac{\delta(N,M)}{2}\left(\sum_{i=1}^{M}\lVert x_i-y_{\tau^*(i)}\rVert_2\right)=\frac{\delta(N,M)}{2}\Delta(S,T)
\end{aligned} E a ∼ Unif ( S d − 1 ) Δ a ( S , T ) = E a ∼ Unif ( S d − 1 ) [ i = 1 ∑ M a ⊤ ( x i − y τ a ( i ) ) ] ≥ E a ∼ Unif ( S d − 1 ) [ i = 1 ∑ M a ⊤ ( x i − y τ a ( i ) ) ∣ a ∈ A ] P { a ∈ A } ≥ 2 δ ( N , M ) E a [ i = 1 ∑ M ∥ x i − y τ a ( i ) ∥ 2 ] ≥ 2 δ ( N , M ) E a [ i = 1 ∑ M ∥ x i − y τ ∗ ( i ) ∥ 2 ] , as τ ∗ is optimal coupling = 2 δ ( N , M ) ( i = 1 ∑ M ∥ x i − y τ ∗ ( i ) ∥ 2 ) = 2 δ ( N , M ) Δ ( S , T )
Hence, F F F is Hölder separable with constant C ′ δ ( N , M ) 2 \frac{C'\delta(N,M)}{2} 2 C ′ δ ( N , M ) and exponent λ = 2 \lambda=2 λ = 2 .
Theorem 5.3.9 (Probability bounds on separation). Let V ⊆ R d V\subseteq\mathbb{R}^d V ⊆ R d and σ \sigma σ be as in Theorem 5.3.8, and let A ∈ R m × d A\in\mathbb{R}^{m\times d} A ∈ R m × d , b ∈ R m b\in\mathbb{R}^m b ∈ R m , c ∈ R m c\in\mathbb{R}^m c ∈ R m whose m m m columns (respectively entries) are drawn independently from the distribution on a j , b j , c j a_j,b_j,c_j a j , b j , c j described in Theorem 5.3.8, and consider the function
F ( S ; A , b , c ) = ∑ x ∈ S σ ( c − 1 ⊙ ( A x + b ) ) . (57) F(S;A,b,c)=\sum_{x\in S}\sigma\left(c^{-1}\odot(Ax+b)\right). \tag*{(57)} F ( S ; A , b , c ) = x ∈ S ∑ σ ( c − 1 ⊙ ( A x + b ) ) . (57) Then there exists C > 0 C>0 C > 0 , so that for all S ⊈ T S\nsubseteq T S ⊈ T , P ( F ( S ) ≤ F ( T ) ) ≤ ( 1 − C Δ 2 ( S , T ) ) m \mathbb{P}(F(S)\leq F(T))\leq\left(1-C\Delta^2(S,T)\right)^m P ( F ( S ) ≤ F ( T )) ≤ ( 1 − C Δ 2 ( S , T ) ) m .
Proof. Denote by f f f the scalar-valued set function for a co-ordinate of F F F ; we proved it’s a lower Holder function. Let S ⊈ T S\nsubseteq T S ⊈ T , then by lower continuity we know that,
E w ∼ μ ( ⋅ ) ∥ [ f ( S ; w ) − f ( T ; w ) ] + ∥ 1 ≥ C ⋅ Δ ( S , T ) 2 , for all S , T ∈ P ≤ k ( V ) (58) \mathbb{E}_{w\sim\mu(\cdot)}\lVert[f(S;w)-f(T;w)]_+\rVert_1\geq C\cdot\Delta(S,T)^2,\quad\text{for all }S,T\in\mathcal{P}_{\leq k}(V) \tag*{(58)} E w ∼ μ ( ⋅ ) ∥[ f ( S ; w ) − f ( T ; w ) ] + ∥ 1 ≥ C ⋅ Δ ( S , T ) 2 , for all S , T ∈ P ≤ k ( V ) (58)
We know that
c ⋅ Δ ( S , T ) 2 ≤ E w ∼ μ ( ⋅ ) ∥ [ f ( S ; w ) − f ( T ; w ) ] + ∥ 1 ≤ P ( f ( S ; w ) ≥ f ( T ; w ) ) ⋅ sup w ∈ W ∣ f ( S ; w ) ≥ f ( T ; w ) ∣ c\cdot\Delta(S,T)^2\leq\mathbb{E}_{w\sim\mu(\cdot)}\lVert[f(S;w)-f(T;w)]_+\rVert_1
\leq\mathbb{P}(f(S;w)\geq f(T;w))\cdot\sup_{w\in\mathcal{W}}|f(S;w)\geq f(T;w)| c ⋅ Δ ( S , T ) 2 ≤ E w ∼ μ ( ⋅ ) ∥[ f ( S ; w ) − f ( T ; w ) ] + ∥ 1 ≤ P ( f ( S ; w ) ≥ f ( T ; w )) ⋅ w ∈ W sup ∣ f ( S ; w ) ≥ f ( T ; w ) ∣
Now, note that as f f f is a bounded function, denote by M M M the supremum of f f f over all sets S S S and weights w ∈ W w\in\mathcal{W} w ∈ W . Thus by the triangle equality,
c ⋅ Δ ( S , T ) 2 ≤ 2 M ⋅ P ( f ( S ; w ) ≥ f ( T ; w ) ) c\cdot\Delta(S,T)^2\leq2M\cdot\mathbb{P}(f(S;w)\geq f(T;w)) c ⋅ Δ ( S , T ) 2 ≤ 2 M ⋅ P ( f ( S ; w ) ≥ f ( T ; w ))
Thus, we obtain that
c ⋅ Δ ( S , T ) 2 2 ⋅ M ≤ P ( f ( S ; w ) ≥ f ( T ; w ) ) \frac{c \cdot\Delta(S,T)^2}{2 \cdot M} \leq\mathbb{P}(f(S;w) \geq f(T;w)) 2 ⋅ M c ⋅ Δ ( S , T ) 2 ≤ P ( f ( S ; w ) ≥ f ( T ; w ))
Thus, the complement satisfies:
P ( f ( S ; w ) < f ( T ; w ) ) ≤ 1 − c ⋅ Δ ( S , T ) 2 2 ⋅ M \mathbb{P}(f(S;w) < f(T;w)) \leq1 - \frac{c \cdot\Delta(S,T)^2}{2 \cdot M} P ( f ( S ; w ) < f ( T ; w )) ≤ 1 − 2 ⋅ M c ⋅ Δ ( S , T ) 2
Now, taking m m m independent copies of f f f , namely F F F , we have that
P ( ∀ i ∈ [ m ] , F ( S ; w ) i < F ( T ; w ) i ) ≤ ( 1 − c ⋅ Δ ( S , T ) 2 2 ⋅ M ) m \mathbb{P}(\forall i \in[m], F(S;w)_i < F(T;w)_i) \leq\left(1 - \frac{c \cdot\Delta(S,T)^2}{2 \cdot M}\right)^m P ( ∀ i ∈ [ m ] , F ( S ; w ) i < F ( T ; w ) i ) ≤ ( 1 − 2 ⋅ M c ⋅ Δ ( S , T ) 2 ) m
Denoting by C : = c 2 M C := \frac{c}{2M} C := 2 M c yield the desired:
P ( F ( S ; w ) < F ( T ; w ) ) ≤ ( 1 − C ⋅ Δ ( S , T ) 2 ) m \mathbb{P}(F(S;w) < F(T;w)) \leq(1 - C \cdot\Delta(S,T)^2)^m P ( F ( S ; w ) < F ( T ; w )) ≤ ( 1 − C ⋅ Δ ( S , T ) 2 ) m
Lower Hölder separability of ReLU networks So far, we have only given results about Lower Hölder separability of MAS functions with Hat activations. However, as seen in Proposition 5.3.6 , we have seen that a two layer linear network with ReLU activation is weakly MAS, and as observed from the proof technique, such a 2 layer ReLU net can ”simulate” a hat activation. In our arguments, we have shown Lower Hölder separability of Hat Activation based parametric set functions by finding a separating parameter point w ∈ W w \in\mathcal{W} w ∈ W for a given pair of sets ( S , T ) (S,T) ( S , T ) , and constructing an open set of non-zero measure, W ′ ⊆ W \mathcal{W}' \subseteq\mathcal{W} W ′ ⊆ W around w w w . We lower bounded the expectation of the non-negative Random variable in the definition of Lower Hölder separability(in Equation (48) ) by restricting the expectation only to W ′ \mathcal{W}' W ′ and showing that it is proportional to Δ ( S , T ) λ \Delta(S,T)^\lambda Δ ( S , T ) λ . One can extend this exact same idea to construct the open set of parameters for the two later ReLU networks from the separable parameter point that we get from proof of Proposition 5.3.6 .
Also, as obtained from Proposition 5.3.2 , we see that one layer ReLU networks are not even weakly MAS. But a key step in that proof uses that one can find 3 collinear elements in the ground set V V V and construct sets with them to obtain the necessary counter-example. Thus, a natural question is to ask: whether shallow 1-layer networks with ReLU activation are also weakly MAS with some additional assumptions on the domain. We show in the following result that, with the additional assumption of V V V being a hypersphere, we may guarantee Lower Hölder separability for even shallow ReLU networks.
Theorem B.5. Given a ground set V ⊆ S d − 1 V \subseteq S^{d-1} V ⊆ S d − 1 . We define F : P ≤ k ( V ) → R F : \mathcal{P}_{\leq k}(V) \to\mathbb{R} F : P ≤ k ( V ) → R as F ( S ) = ∑ x ∈ S ReLU ( a ⊤ x + b ) F(S) = \sum_{x \in S} \operatorname{ReLU}(a^\top x + b) F ( S ) = ∑ x ∈ S ReLU ( a ⊤ x + b ) , where a ∼ Unif ( S d − 1 ) a \sim\operatorname{Unif}(S^{d-1}) a ∼ Unif ( S d − 1 ) and b ∼ Unif ( − 1 , 1 ) b \sim\operatorname{Unif}(-1,1) b ∼ Unif ( − 1 , 1 ) . Then, for
any S , T ∈ P < ∞ ( V ) S,T \in\mathcal{P}_{<\infty}(V) S , T ∈ P < ∞ ( V ) , we have: E a , b [ F ( S ) − F ( T ) ] + ≥ C ( k ) Δ ( S , T ) ( d + 3 ) 2 k \mathbb{E}_{a,b}[F(S)-F(T)]_{+} \ge C(k)\Delta(S,T)^{(d+3)2^{k}} E a , b [ F ( S ) − F ( T ) ] + ≥ C ( k ) Δ ( S , T ) ( d + 3 ) 2 k for some constant C ( k ) > 0 C(k)>0 C ( k ) > 0 depending only on k k k .
Proof. Let S = { x 1 , ⋯ , x M } S=\{x_1,\cdots,x_M\} S = { x 1 , ⋯ , x M } and T = { y 1 , ⋯ , y N } T=\{y_1,\cdots,y_N\} T = { y 1 , ⋯ , y N } , where M ≤ N ≤ k M\le N\le k M ≤ N ≤ k . And according to the definition of Δ ( S , T ) \Delta(S,T) Δ ( S , T ) from Equation (47) we can equivalently write
Δ ( S , T ) = min π ∈ S N ∑ i = 1 M ∥ x i − y π ( i ) ∥ 2 . \Delta(S,T)=\min_{\pi\in S_N}\sum_{i=1}^{M}\lVert x_i-y_{\pi(i)}\rVert_2. Δ ( S , T ) = π ∈ S N min i = 1 ∑ M ∥ x i − y π ( i ) ∥ 2 .
Let π ∗ \pi^* π ∗ be the optimal solution of the above problem, then we can define the map τ : [ M ] → [ N ] \tau:[M]\to[N] τ : [ M ] → [ N ] as follows: τ ( i ) = π ∗ ( i ) , ∀ i ∈ [ M ] \tau(i)=\pi^*(i),\ \forall i\in[M] τ ( i ) = π ∗ ( i ) , ∀ i ∈ [ M ] . Essentially, x i x_i x i and y τ ( i ) y_{\tau(i)} y τ ( i ) are coupled in the optimal alignment. Now, for any given i ∈ [ M ] i\in[M] i ∈ [ M ] , let Δ i : = ∥ x i − y τ ( i ) ∥ 2 \Delta_i:=\lVert x_i-y_{\tau(i)}\rVert_2 Δ i := ∥ x i − y τ ( i ) ∥ 2 . Then, consider the open Euclidean ball B ( x i , Δ i ) \mathcal{B}(x_i,\Delta_i) B ( x i , Δ i ) . Then if j ∈ [ N ] j\in[N] j ∈ [ N ] such that y j ∈ B ( x i , Δ i ) y_j\in\mathcal{B}(x_i,\Delta_i) y j ∈ B ( x i , Δ i ) then, j = τ ( ℓ ) j=\tau(\ell) j = τ ( ℓ ) for some ℓ ∈ [ M ] \ell\in[M] ℓ ∈ [ M ] . Otherwise switching from i ↦ τ ( i ) i\mapsto\tau(i) i ↦ τ ( i ) to i ↦ j i\mapsto j i ↦ j reduces the cost, which violates optimal alignment. Let Ω i : = { ℓ ∈ [ M ] : y τ ( ℓ ) ∈ B ( x i , Δ i ) } \Omega_i:=\{\ell\in[M]:y_{\tau(\ell)}\in\mathcal{B}(x_i,\Delta_i)\} Ω i := { ℓ ∈ [ M ] : y τ ( ℓ ) ∈ B ( x i , Δ i )} . Here, we make the following claim:
Claim B.6. ∃ α ∈ [ M ] \exists\alpha\in[M] ∃ α ∈ [ M ] such that ∑ j ∈ Ω i ∥ y j − x τ − 1 ( j ) ∥ 2 ≤ Δ α 2 16 \sum_{j\in\Omega_i}\lVert y_j-x_{\tau^{-1}(j)}\rVert_2\le\frac{\Delta_\alpha^2}{16} ∑ j ∈ Ω i ∥ y j − x τ − 1 ( j ) ∥ 2 ≤ 16 Δ α 2 and Δ α ≥ 16 Δ 2 ( S , T ) ( 16 k ) 2 k \Delta_\alpha\ge\frac{16\Delta^2(S,T)}{(16k)^{2^k}} Δ α ≥ ( 16 k ) 2 k 16 Δ 2 ( S , T ) .
Proof. Suppose not. WLOG, assume the following order Δ 1 ≥ Δ 2 ≥ ⋯ ≥ Δ M \Delta_1\ge\Delta_2\ge\cdots\ge\Delta_M Δ 1 ≥ Δ 2 ≥ ⋯ ≥ Δ M . We build the following directed graph G G G on [ M ] [M] [ M ] . Start with the node v 0 = 1 v_0=1 v 0 = 1 and add one node at a time. At step t t t , we select v t : = arg max v ∈ Ω v t − 1 Δ v v_t:=\operatorname*{arg\,max}_{v\in\Omega_{v_{t-1}}}\Delta_v v t := arg max v ∈ Ω v t − 1 Δ v and add the edge ( v t − 1 , v t ) (v_{t-1},v_t) ( v t − 1 , v t ) . By the assumption, Δ v t 16 k ≥ ( Δ v t − 1 2 ) 2 ≥ ⋯ ≥ ( Δ v 0 ( 16 k ) 2 ) 2 k \frac{\Delta_{v_t}}{16k}\ge\left(\frac{\Delta_{v_{t-1}}}{2}\right)^2\ge\cdots\ge\left(\frac{\Delta_{v_0}}{(16k)^{2}}\right)^{2^k} 16 k Δ v t ≥ ( 2 Δ v t − 1 ) 2 ≥ ⋯ ≥ ( ( 16 k ) 2 Δ v 0 ) 2 k . Since we work with finite M M M , either G G G is a DAG and this process terminates or G G G has a cycle. In former case at termination step T T T we cannot add any more nodes, thus Ω T = ∅ \Omega_T=\varnothing Ω T = ∅ , and T ≤ k T\le k T ≤ k . So, Δ T ≥ 16 k ( 16 k ) 2 T ≥ 16 Δ 2 ( S , T ) ( 16 k ) 2 k \Delta_T\ge\frac{16^k}{(16k)^{2^T}}\ge\frac{16\Delta^2(S,T)}{(16k)^{2^k}} Δ T ≥ ( 16 k ) 2 T 1 6 k ≥ ( 16 k ) 2 k 16 Δ 2 ( S , T ) and we’re done. Otherwise, consider the smallest cycle in G G G . Let this cycle of length p p p be C = { v β → v β + 1 → ⋯ → v β + p − 1 → v β } C=\{v_\beta\to v_{\beta+1}\to\cdots\to v_{\beta+p-1}\to v_\beta\} C = { v β → v β + 1 → ⋯ → v β + p − 1 → v β } . Then, we have ∥ x v β + i − y τ ( v β + i + 1 ) ∥ 2 < ∥ x v β + i − y τ ( v β + i ) ∥ 2 , ∀ i ∈ { 0 , ⋯ , p − 1 } \lVert x_{v_{\beta+i}}-y_{\tau(v_{\beta+i+1})}\rVert_2<\lVert x_{v_{\beta+i}}-y_{\tau(v_{\beta+i})}\rVert_2,\ \forall i\in\{0,\cdots,p-1\} ∥ x v β + i − y τ ( v β + i + 1 ) ∥ 2 < ∥ x v β + i − y τ ( v β + i ) ∥ 2 , ∀ i ∈ { 0 , ⋯ , p − 1 } and addition of sub-indices is modulo p p p . Thus, we have the following: ∥ x v β − y τ ( v β + 1 ) ∥ 2 + ∥ x v β + 1 − y τ ( v β + 2 ) ∥ 2 + ⋯ + ∥ x v β + p − 1 − y τ ( v β ) ∥ 2 < ∑ j = 0 p − 1 ∥ x v β + i − y τ ( v β + j ) ∥ 2 \lVert x_{v_\beta}-y_{\tau(v_{\beta+1})}\rVert_2+\lVert x_{v_{\beta+1}}-y_{\tau(v_{\beta+2})}\rVert_2+\cdots+\lVert x_{v_{\beta+p-1}}-y_{\tau(v_\beta)}\rVert_2<\sum_{j=0}^{p-1}\lVert x_{v_{\beta+i}}-y_{\tau(v_{\beta+j})}\rVert_2 ∥ x v β − y τ ( v β + 1 ) ∥ 2 + ∥ x v β + 1 − y τ ( v β + 2 ) ∥ 2 + ⋯ + ∥ x v β + p − 1 − y τ ( v β ) ∥ 2 < ∑ j = 0 p − 1 ∥ x v β + i − y τ ( v β + j ) ∥ 2 . Hence, switching from the coupling i ↦ τ ( i ) i\mapsto\tau(i) i ↦ τ ( i ) to i ↦ τ ( i + 1 ) i\mapsto\tau(i+1) i ↦ τ ( i + 1 ) for i ∈ { v β , ⋯ , v β + p − 2 } i\in\{v_\beta,\cdots,v_{\beta+p-2}\} i ∈ { v β , ⋯ , v β + p − 2 } and to v β + p − 1 ↦ v β v_\beta+p-1\mapsto v_\beta v β + p − 1 ↦ v β strictly reduces the cost, but we started with the optimal coupling. This gives a contradiction.
Consider α \alpha α as in the above claim. Then we define the region: A : = { a ∈ S d − 1 : ∥ a − x α ∥ < Δ α 4 } \mathcal{A}:=\{a\in\mathbb{S}^{d-1}:\lVert a-x_\alpha\rVert<\frac{\Delta_\alpha}{4}\} A := { a ∈ S d − 1 : ∥ a − x α ∥ < 4 Δ α } . Then, ∀ a ∈ A \forall a\in\mathcal{A} ∀ a ∈ A , ∥ a − x α ∥ 2 < ∥ a − y τ ( α ) ∥ 2 \lVert a-x_\alpha\rVert_2<\lVert a-y_{\tau(\alpha)}\rVert_2 ∥ a − x α ∥ 2 < ∥ a − y τ ( α ) ∥ 2 . Thus, a ⊤ x α > a ⊤ y τ ( α ) a^\top x_\alpha>a^\top y_{\tau(\alpha)} a ⊤ x α > a ⊤ y τ ( α ) . Also, consider the region B a : = { b ∈ R : − 1 + Δ α 2 16 + Δ α 2 64 + ∥ a − x ∥ 2 2 ≤ b ≤ − 1 + Δ α 2 8 } \mathcal{B}_a:=\left\{b\in\mathbb{R}:-1+\frac{\Delta_\alpha^2}{16}+\frac{\Delta_\alpha^2}{64}+\frac{\lVert a-x\rVert^2}{2}\le b\le-1+\frac{\Delta_\alpha^2}{8}\right\} B a := { b ∈ R : − 1 + 16 Δ α 2 + 64 Δ α 2 + 2 ∥ a − x ∥ 2 ≤ b ≤ − 1 + 8 Δ α 2 } . For any given a ∈ A a\in\mathcal{A} a ∈ A , we have: ∥ a − x ∥ 2 2 ≤ Δ α 2 32 \frac{\lVert a-x\rVert^2}{2}\le\frac{\Delta_\alpha^2}{32} 2 ∥ a − x ∥ 2 ≤ 32 Δ α 2 , thus − 1 + Δ α 2 16 + Δ α 2 64 + ∥ a − x ∥ 2 2 ≤ − 1 + 7 64 Δ α 2 ≤ − 1 + Δ α 2 8 -1+\frac{\Delta_\alpha^2}{16}+\frac{\Delta_\alpha^2}{64}+\frac{\lVert a-x\rVert^2}{2}\le-1+\frac{7}{64}\Delta_\alpha^2\le-1+\frac{\Delta_\alpha^2}{8} − 1 + 16 Δ α 2 + 64 Δ α 2 + 2 ∥ a − x ∥ 2 ≤ − 1 + 64 7 Δ α 2 ≤ − 1 + 8 Δ α 2 , so B a ≠ ∅ \mathcal{B}_a\ne\varnothing B a = ∅ . Also note that, ∀ a ∈ A , b ∈ B a \forall a\in\mathcal{A},b\in\mathcal{B}_a ∀ a ∈ A , b ∈ B a , we have:
[ F ( S ) − F ( T ) ] + ≥ [ ReLU ( a ⊤ x α + b ) − ( ∑ β ∈ Ω α ReLU ( a ⊤ y τ ( β ) ) − ReLU ( a ⊤ x β ) ) ] + ≥ [ ReLU ( a ⊤ x α + b ) − ∑ β ∈ Ω α ∣ ReLU ( a ⊤ y τ ( β ) ) − ReLU ( a ⊤ x β ) ∣ ] + \begin{aligned}
[F(S)-F(T)]_+\ge\left[\operatorname{ReLU}(a^\top x_\alpha+b)-\left(\sum_{\beta\in\Omega_\alpha}\operatorname{ReLU}(a^\top y_{\tau(\beta)})-\operatorname{ReLU}(a^\top x_\beta)\right)\right]_+ \\
&\ge\left[\operatorname{ReLU}(a^\top x_\alpha+b)-\sum_{\beta\in\Omega_\alpha}\left|\operatorname{ReLU}(a^\top y_{\tau(\beta)})-\operatorname{ReLU}(a^\top x_\beta)\right|\right]_+
\end{aligned} [ F ( S ) − F ( T ) ] + ≥ ReLU ( a ⊤ x α + b ) − β ∈ Ω α ∑ ReLU ( a ⊤ y τ ( β ) ) − ReLU ( a ⊤ x β ) + ≥ ReLU ( a ⊤ x α + b ) − β ∈ Ω α ∑ ReLU ( a ⊤ y τ ( β ) ) − ReLU ( a ⊤ x β ) +
≥ [ ReLU ( a ⊤ x α + b ) − ∑ β ∈ Ω α ∣ a ⊤ ( x − y ) ∣ ] + ≥ [ ReLU ( a ⊤ x α + b ) − ∑ β ∈ Ω α ∥ x β − y τ ( β ) ∥ ] + ≥ [ a ⊤ x α + b − Δ α 2 16 ] + = [ 1 + b − ∥ a − x ∥ 2 2 − Δ α 2 16 ] + ≥ Δ α 2 64 \begin{aligned}
&\ge\left[\operatorname{ReLU}\left(a^{\top}x_{\alpha}+b\right)-\sum_{\beta\in\Omega_{\alpha}}\left|a^{\top}(x-y)\right|\right]_{+}\\
&\ge\left[\operatorname{ReLU}\left(a^{\top}x_{\alpha}+b\right)-\sum_{\beta\in\Omega_{\alpha}}\left\|x_{\beta}-y_{\tau(\beta)}\right\|\right]_{+}\\
&\ge\left[a^{\top}x_{\alpha}+b-\frac{\Delta_{\alpha}^{2}}{16}\right]_{+}
=\left[1+b-\frac{\|a-x\|^{2}}{2}-\frac{\Delta_{\alpha}^{2}}{16}\right]_{+}
\ge\frac{\Delta_{\alpha}^{2}}{64}
\end{aligned} ≥ ReLU ( a ⊤ x α + b ) − β ∈ Ω α ∑ a ⊤ ( x − y ) + ≥ ReLU ( a ⊤ x α + b ) − β ∈ Ω α ∑ x β − y τ ( β ) + ≥ [ a ⊤ x α + b − 16 Δ α 2 ] + = [ 1 + b − 2 ∥ a − x ∥ 2 − 16 Δ α 2 ] + ≥ 64 Δ α 2
Thus, we have:
E a , b [ F ( S ) − F ( T ) ] + ≥ E a , b [ ( F ( S ) − F ( T ) ) 1 { a ∈ A , b ∈ B a } ] + ≥ Δ α 2 64 P { a ∈ A , b ∈ B a } = Δ α 2 64 S d − 1 ∫ a ∈ S d − 1 : a 1 ≥ 1 − Δ α 2 32 1 2 ( 3 64 Δ α 2 − ( a 1 − 1 ) ) d a ≥ Δ α 4 2 13 S d − 1 ∫ a ∈ S d − 1 : a 1 ≥ 1 − Δ α 2 32 d a = Δ α 4 2 13 ∫ 1 − Δ α 2 32 1 ( 1 − x 2 ) d − 3 2 d x ≥ Δ α 6 2 18 ( Δ α 2 32 ) d − 3 2 ( 2 − Δ α 2 32 ) d − 3 2 ≥ Δ α d + 3 2 23 \begin{aligned}
\mathbb{E}_{a,b}\left[F(S)-F(T)\right]_{+}
&\ge\mathbb{E}_{a,b}\left[(F(S)-F(T))\mathbf{1}_{\{a\in\mathcal{A},b\in\mathcal{B}_{a}\}}\right]_{+}\\
&\ge\frac{\Delta_{\alpha}^{2}}{64}\mathbb{P}\{a\in\mathcal{A},b\in\mathcal{B}_{a}\}\\
&=\frac{\Delta_{\alpha}^{2}}{64S_{d-1}}\int_{a\in S^{d-1}:a_{1}\ge1-\frac{\Delta_{\alpha}^{2}}{32}}\frac{1}{2}\left(\frac{3}{64}\Delta_{\alpha}^{2}-(a_{1}-1)\right)\,da\\
&\ge\frac{\Delta_{\alpha}^{4}}{2^{13}S_{d-1}}\int_{a\in S^{d-1}:a_{1}\ge1-\frac{\Delta_{\alpha}^{2}}{32}}da\\
&=\frac{\Delta_{\alpha}^{4}}{2^{13}}\int_{1-\frac{\Delta_{\alpha}^{2}}{32}}^{1}(1-x^{2})^{\frac{d-3}{2}}\,dx
\ge\frac{\Delta_{\alpha}^{6}}{2^{18}}\left(\frac{\Delta_{\alpha}^{2}}{32}\right)^{\frac{d-3}{2}}\left(2-\frac{\Delta_{\alpha}^{2}}{32}\right)^{\frac{d-3}{2}}\\
&\ge\frac{\Delta_{\alpha}^{d+3}}{2^{23}}
\end{aligned} E a , b [ F ( S ) − F ( T ) ] + ≥ E a , b [ ( F ( S ) − F ( T )) 1 { a ∈ A , b ∈ B a } ] + ≥ 64 Δ α 2 P { a ∈ A , b ∈ B a } = 64 S d − 1 Δ α 2 ∫ a ∈ S d − 1 : a 1 ≥ 1 − 32 Δ α 2 2 1 ( 64 3 Δ α 2 − ( a 1 − 1 ) ) d a ≥ 2 13 S d − 1 Δ α 4 ∫ a ∈ S d − 1 : a 1 ≥ 1 − 32 Δ α 2 d a = 2 13 Δ α 4 ∫ 1 − 32 Δ α 2 1 ( 1 − x 2 ) 2 d − 3 d x ≥ 2 18 Δ α 6 ( 32 Δ α 2 ) 2 d − 3 ( 2 − 32 Δ α 2 ) 2 d − 3 ≥ 2 23 Δ α d + 3
Now, using the bounds from B.6, we get that, Δ α ≥ C ( k ) Δ 2 k ( S , T ) \Delta_{\alpha}\ge C(k)\Delta^{2^{k}}(S,T) Δ α ≥ C ( k ) Δ 2 k ( S , T ) . Thus we have that, E [ F ( S ) − F ( T ) ] + ≥ C 1 ( k ) Δ ( S , T ) ( d + 3 ) 2 k \mathbb{E}[F(S)-F(T)]_{+}\ge C_{1}(k)\Delta(S,T)^{(d+3)2^{k}} E [ F ( S ) − F ( T ) ] + ≥ C 1 ( k ) Δ ( S , T ) ( d + 3 ) 2 k , and done □
Upper Lipschitz property of weakly MAS functions
We now show that, the MAS functions obtained by applying one layer neural network with Hat activation for each element of the set, followed by sum aggregation are also Upper Lipschitz continuous in expectation with respect to the augmented Wasserstein distance, following the framework of Davidson and Dym [13 ] . This shows stability of such MAS embeddings. We say that a parametric set function F : P ≤ k ( V ) × W → R m F:\mathcal{P}_{\leq k}(V)\times\mathcal{W}\to\mathbb{R}^{m} F : P ≤ k ( V ) × W → R m is Upper Lipschitz in expectation if ∃ C > 0 \exists C>0 ∃ C > 0 such that:
E w ∈ W ∥ F ( S ; w ) − F ( T ; w ) ∥ 1 ≤ C ⋅ W ( k ) ( S , T ) , ∀ S , T ∈ P ≤ k ( V ) (59) \mathbb{E}_{w\in\mathcal{W}}\|F(S;w)-F(T;w)\|_{1}\leq C\cdot\mathcal{W}^{(k)}(S,T),\quad\forall S,T\in\mathcal{P}_{\leq k}(V) \tag*{(59)} E w ∈ W ∥ F ( S ; w ) − F ( T ; w ) ∥ 1 ≤ C ⋅ W ( k ) ( S , T ) , ∀ S , T ∈ P ≤ k ( V ) (59)
Where W ( k ) ( ⋅ , ∗ ) \mathcal{W}^{(k)}(\cdot,\ast) W ( k ) ( ⋅ , ∗ ) is the augmented-Wasserstein metric for sets in P ≤ k ( V ) \mathcal{P}_{\leq k}(V) P ≤ k ( V ) , where a padding z z z is added for k − ∣ S ∣ k-|S| k − ∣ S ∣ times to any multiset S S S of size less than k k k . We work with a compact ground set V V V , and we pick a padding z z z that has a positive distance from V V V . For the following proofs, we work with V ⊆ R d V\subseteq\mathbb{R}^{d} V ⊆ R d which is norm bounded by 1, and we choose the padding element z ∈ R d z\in\mathbb{R}^{d} z ∈ R d such that ∥ z ∥ ≥ 3 \|z\|\geq3 ∥ z ∥ ≥ 3 . For a general V V V whose norm is bounded by B B B , we can scale the padding z z z by B B B . With this, we show that, real valued functions F : P ≤ k ( V ) → R F:\mathcal{P}_{\leq k}(V)\to\mathbb{R} F : P ≤ k ( V ) → R given by F ( S ) = ∑ x ∈ S σ ( a ⊤ x − b c ) F(S)=\sum_{x\in S}\sigma\left(\frac{a^{\top}x-b}{c}\right) F ( S ) = ∑ x ∈ S σ ( c a ⊤ x − b ) are Upper Lipschitz w.r.t W k \mathsf{W}_k W k . Now, if we take independent parametric copies across m m m output dimensions and if M 2 M_2 M 2 is a vector-to-vector Lipschitz function, then functions of the form M 2 ∘ F M_2 \circ F M 2 ∘ F , where F F F has the form in Equation (5.6), are Upper Lipschitz as well. This we state in the following theorem:
Theorem B.7 (F F F is upper Lipschitz). Let σ \sigma σ belongs to the class of Hat activations, and V ⊆ R d V \subseteq\mathbb{R}^{d} V ⊆ R d be a compact ground set such that sup v ∈ V ∥ v ∥ ≤ 1 \sup_{v \in V}\lVert v\rVert\le1 sup v ∈ V ∥ v ∥ ≤ 1 . We consider F : P ≤ k ( V ) → R F:\mathcal{P}_{\leq k}(V) \to\mathbb{R} F : P ≤ k ( V ) → R , F ( S ) = ∑ x ∈ S σ ( a T x − b c ) F(S)=\sum_{x\in S}\sigma\left(\frac{a^{\mathsf{T}}x-b}{c}\right) F ( S ) = ∑ x ∈ S σ ( c a T x − b ) , with the distributions a ∼ Unif ( S d − 1 ) a \sim\operatorname{Unif}(\mathbb{S}^{d-1}) a ∼ Unif ( S d − 1 ) , b ∼ Unif ( − 1 , 1 ) b \sim\operatorname{Unif}(-1,1) b ∼ Unif ( − 1 , 1 ) , c ∼ Unif ( 0 , 2 ) c \sim\operatorname{Unif}(0,2) c ∼ Unif ( 0 , 2 ) . Then ∃ \exists ∃ a constant C > 0 C > 0 C > 0 , such that for any S , T ∈ P ≤ k ( V ) S,T \in\mathcal{P}_{\leq k}(V) S , T ∈ P ≤ k ( V ) :
E a , b , c ∣ F ( S ) − F ( T ) ∣ ≤ C ⋅ W ( k ) ( S , T ) \mathbb{E}_{a,b,c}\lvert F(S)-F(T)\rvert\leq C \cdot\mathsf{W}^{(k)}(S,T) E a , b , c ∣ F ( S ) − F ( T )∣ ≤ C ⋅ W ( k ) ( S , T )
Proof. Consider S = { x 1 , ⋯ , x M } , T = { y 1 , ⋯ , y N } S=\{x_1,\cdots,x_M\},T=\{y_1,\cdots,y_N\} S = { x 1 , ⋯ , x M } , T = { y 1 , ⋯ , y N } . Since ∣ F ( S ) − F ( T ) ∣ \lvert F(S)-F(T)\rvert ∣ F ( S ) − F ( T )∣ is symmetric in S , T S,T S , T , we consider WLOG that M ≤ N M \leq N M ≤ N . As in previous sections, we use τ \tau τ to denote an injective function from [ M ] → [ N ] [M]\to[N] [ M ] → [ N ] , and we use Ω N , M \Omega_{N,M} Ω N , M to denote the set of all injective functions from [ M ] → [ N ] [M]\to[N] [ M ] → [ N ] . Thus, we can write the augmented Wasserstein distance on P ≤ k ( V ) \mathcal{P}_{\leq k}(V) P ≤ k ( V ) with padding z z z as:
W ( k ) ( S , T ) = arg min τ ∈ Ω N , M { ∑ i = 1 M ∥ x i − y τ ( i ) ∥ + ∑ j ∉ τ ( i ) ∥ y j − z ∥ } \mathsf{W}^{(k)}(S,T)=\underset{\tau\in\Omega_{N,M}}{\arg\min}\left\{\sum_{i=1}^{M}\lVert x_i-y_{\tau(i)}\rVert+\sum_{j\notin\tau(i)}\lVert y_j-z\rVert\right\} W ( k ) ( S , T ) = τ ∈ Ω N , M arg min ⎩ ⎨ ⎧ i = 1 ∑ M ∥ x i − y τ ( i ) ∥ + j ∈ / τ ( i ) ∑ ∥ y j − z ∥ ⎭ ⎬ ⎫
Let τ ∗ \tau^* τ ∗ be the argmin of the above expression that gives the W k ( S , T ) \mathsf{W}_k(S,T) W k ( S , T ) Now, we can write:
E a , b , c ∣ F ( S ) − F ( T ) ∣ = E a , b , c ∣ ∑ i = 1 M σ ( a T x i − b c ) − σ ( a T y τ ∗ ( i ) − b c ) − ∑ j ≠ τ ∗ ( i ) σ ( a T y j − b c ) ∣ ≤ ∑ i = 1 M E a , b , c ∣ σ ( a T x i − b c ) − σ ( a T y τ ∗ ( i ) − b c ) ∣ + ∑ j ≠ τ ∗ ( i ) ∣ E a , b , c ∣ σ ( a T y j − b c ) ∣ \begin{aligned}
\mathbb{E}_{a,b,c}\lvert F(S)-F(T)\rvert=\mathbb{E}_{a,b,c}\left\lvert\sum_{i=1}^{M}\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\tau^*(i)}-b}{c}\right)-\sum_{j\ne\tau^*(i)}\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)\right\rvert\\
&\leq\sum_{i=1}^{M}\mathbb{E}_{a,b,c}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\tau^*(i)}-b}{c}\right)\right\rvert+\sum_{j\ne\tau^*(i)}\left\lvert\mathbb{E}_{a,b,c}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)\right\rvert\right.
\end{aligned} E a , b , c ∣ F ( S ) − F ( T )∣ = E a , b , c i = 1 ∑ M σ ( c a T x i − b ) − σ ( c a T y τ ∗ ( i ) − b ) − j = τ ∗ ( i ) ∑ σ ( c a T y j − b ) ≤ i = 1 ∑ M E a , b , c σ ( c a T x i − b ) − σ ( c a T y τ ∗ ( i ) − b ) + j = τ ∗ ( i ) ∑ E a , b , c σ ( c a T y j − b )
We now separately analyze coupled and un-coupled terms: i.e terms of the form E ∣ σ ( a T y j − b c ) ∣ \mathbb{E}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)\right\rvert E σ ( c a T y j − b ) and E ∣ σ ( a T x i − b c ) − σ ( a T y τ ∗ ( i ) − b c ) ∣ \mathbb{E}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\tau^*(i)}-b}{c}\right)\right\rvert E σ ( c a T x i − b ) − σ ( c a T y τ ∗ ( i ) − b ) in the following analysis. For the proof that follows, we consider supp ( σ ) ⊆ [ 0 , 1 ] \operatorname{supp}(\sigma)\subseteq[0,1] supp ( σ ) ⊆ [ 0 , 1 ] . For support in [ γ 1 , γ 2 ] [\gamma_1,\gamma_2] [ γ 1 , γ 2 ] , the proof follows similarly by shifting the distributions of b , c b,c b , c by γ 1 \gamma_1 γ 1 and scaling by γ 2 − γ 1 \gamma_2-\gamma_1 γ 2 − γ 1 .
Computing E ∣ σ ( a T y j − b c ) ∣ \mathbb{E}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)\right\rvert E σ ( c a T y j − b ) : Note that, σ ( a T y j − b c ) = 0 , ∀ b ≥ a T y j \sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)=0,\forall b\geq a^{\mathsf{T}}y_j σ ( c a T y j − b ) = 0 , ∀ b ≥ a T y j . Also, for a given b ∈ ( − 1 , a T y j ) b\in(-1,a^{\mathsf{T}}y_j) b ∈ ( − 1 , a T y j ) we have that:
∀ c ∈ ( 0 , a T y j − b ) , a T y j − b c ≥ 1 ⟹ σ ( a T y j − b c ) = 0 \forall c\in(0,a^{\mathsf{T}}y_j-b),\frac{a^{\mathsf{T}}y_j-b}{c}\geq1\Longrightarrow\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)=0 ∀ c ∈ ( 0 , a T y j − b ) , c a T y j − b ≥ 1 ⟹ σ ( c a T y j − b ) = 0
Thus, conditioned on a given a ∈ S d − 1 a \in\mathcal{S}^{d-1} a ∈ S d − 1 , we can write the conditional expectation over b , c b,c b , c as follows:
E b , c ∣ σ ( a T y j − b c ) | a ∣ = 1 4 ∫ b = − 1 a T y j ∫ c = a T y j − b 2 ∣ σ ( a T y j − b c ) − σ ( 0 ) ∣ d c d b , \mathbb{E}_{b,c}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)\middle|a\right\rvert
= \frac{1}{4}\int_{b=-1}^{a^{\mathsf{T}}y_j}\int_{c=a^{\mathsf{T}}y_j-b}^{2}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)-\sigma(0)\right\rvert\,dc\,db, E b , c σ ( c a T y j − b ) a = 4 1 ∫ b = − 1 a T y j ∫ c = a T y j − b 2 σ ( c a T y j − b ) − σ ( 0 ) d c d b ,
using σ ( 0 ) = 0 \sigma(0)=0 σ ( 0 ) = 0
≤ κ 2 4 ∫ b = − 1 a T y j ∫ c = a T y j − b 2 ( a T y j − b c ) d c d b , \leq\frac{\kappa_2}{4}\int_{b=-1}^{a^{\mathsf{T}}y_j}\int_{c=a^{\mathsf{T}}y_j-b}^{2}\left(\frac{a^{\mathsf{T}}y_j-b}{c}\right)\,dc\,db, ≤ 4 κ 2 ∫ b = − 1 a T y j ∫ c = a T y j − b 2 ( c a T y j − b ) d c d b ,
as σ \sigma σ is a Hat activation, it’s Lipschitz
= κ 2 4 ∫ b = − 1 a T y j ( a T y j − b ) ln ( 2 a T y j − b ) d b = κ 2 4 ∫ a T y j + 1 0 z ln ( 2 z ) ( − d z ) , z : = a T y j − b = κ 2 4 ∫ 0 a T y j + 1 z ln ( 2 z ) d z = κ 2 4 ⋅ a T y j + 1 2 ( a T y j + 1 2 + ( a T y j + 1 ) ln ( 2 a T y j + 1 ) ) ≤ κ 2 4 ⋅ ( a T y j + a T a ) = κ 2 4 ∣ a T ( y j + a ) ∣ ≤ κ 2 4 ∥ y j + a ∥ ≤ κ 2 4 ∥ y j − z ∥ \begin{aligned}
= \frac{\kappa_2}{4}\int_{b=-1}^{a^{\mathsf{T}}y_j}(a^{\mathsf{T}}y_j-b)\ln\left(\frac{2}{a^{\mathsf{T}}y_j-b}\right)\,db
= \frac{\kappa_2}{4}\int_{a^{\mathsf{T}}y_j+1}^{0}z\ln\left(\frac{2}{z}\right)(-dz),\ z:=a^{\mathsf{T}}y_j-b \\
&= \frac{\kappa_2}{4}\int_{0}^{a^{\mathsf{T}}y_j+1}z\ln\left(\frac{2}{z}\right)\,dz
= \frac{\kappa_2}{4}\cdot\frac{a^{\mathsf{T}}y_j+1}{2}\left(\frac{a^{\mathsf{T}}y_j+1}{2}+\left(a^{\mathsf{T}}y_j+1\right)\ln\left(\frac{2}{a^{\mathsf{T}}y_j+1}\right)\right) \\
&\leq\frac{\kappa_2}{4}\cdot\left(a^{\mathsf{T}}y_j+a^{\mathsf{T}}a\right)
= \frac{\kappa_2}{4}\left\lvert a^{\mathsf{T}}(y_j+a)\right\rvert
\leq\frac{\kappa_2}{4}\lVert y_j+a\rVert
\leq\frac{\kappa_2}{4}\lVert y_j-z\rVert
\end{aligned} = 4 κ 2 ∫ b = − 1 a T y j ( a T y j − b ) ln ( a T y j − b 2 ) d b = 4 κ 2 ∫ a T y j + 1 0 z ln ( z 2 ) ( − d z ) , z := a T y j − b = 4 κ 2 ∫ 0 a T y j + 1 z ln ( z 2 ) d z = 4 κ 2 ⋅ 2 a T y j + 1 ( 2 a T y j + 1 + ( a T y j + 1 ) ln ( a T y j + 1 2 ) ) ≤ 4 κ 2 ⋅ ( a T y j + a T a ) = 4 κ 2 a T ( y j + a ) ≤ 4 κ 2 ∥ y j + a ∥ ≤ 4 κ 2 ∥ y j − z ∥
Now, taking an expectation over a ∼ Unif ( S d − 1 ) a\sim\operatorname{Unif}(\mathcal{S}^{d-1}) a ∼ Unif ( S d − 1 ) , the inequality is preserved and we get that, for ”isolated ” y j y_j y j ’s, E σ ( a T − b c ) ≤ κ 2 4 ∥ y j − z ∥ \mathbb{E}_{\sigma}\left(\frac{a^{\mathsf{T}}-b}{c}\right)\leq\frac{\kappa_2}{4}\lVert y_j-z\rVert E σ ( c a T − b ) ≤ 4 κ 2 ∥ y j − z ∥
Thus for ”isolated” elements from the larger set that get coupled with the padded element z z z while computing augmented-W( k ) ^{(k)} ( k ) , we have shown that they are upper bounded by their distance from the padding z z z . Hence, it remains to show a similar Lipschitz upper bound for the elements x 1 , ⋯ x m ∈ S 1 x_1,\cdots x_m\in S_1 x 1 , ⋯ x m ∈ S 1 which have a corresponding y i ( 1 ) , ⋯ , y i ( m ) ∈ S 2 y_{i(1)},\cdots,y_{i(m)}\in S_2 y i ( 1 ) , ⋯ , y i ( m ) ∈ S 2 in the optimal coupling
Computing E ∣ σ ( a T x i − b c ) − σ ( a T y π ∗ ( i ) − b c ) ∣ \mathbb{E}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\pi^*(i)}-b}{c}\right)\right\rvert E σ ( c a T x i − b ) − σ ( c a T y π ∗ ( i ) − b ) :
Now, we have, for any i ∈ [ M ] i\in[M] i ∈ [ M ] , and a given a ∈ S d − 1 a\in\mathcal{S}^{d-1} a ∈ S d − 1 the conditional expectation E b , c ∣ σ ( a T x i − b c ) − σ ( a T y π ∗ ( i ) − b c ) ∣ \mathbb{E}_{b,c}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\pi^*(i)}-b}{c}\right)\right\rvert E b , c σ ( c a T x i − b ) − σ ( c a T y π ∗ ( i ) − b ) is symmetric in x i x_i x i and y π ∗ ( i ) y_{\pi^*(i)} y π ∗ ( i ) . Thus, WLOG we can assume that, a T x i ≥ a T y π ∗ ( i ) a^{\mathsf{T}}x_i\geq a^{\mathsf{T}}y_{\pi^*(i)} a T x i ≥ a T y π ∗ ( i ) . Note that we then have:
∣ σ ( a T x i − b c ) − σ ( a T y π ∗ ( i ) − b c ) ∣ = { ∣ σ ( a T x i − b c ) ∣ , ∀ b ∈ ( a T y π ∗ ( i ) , a T x i ) , ∀ c ∈ ( a T x i − b , 2 ) ∣ σ ( a T x i − b c ) − σ ( a T y π ∗ ( i ) − b c ) ∣ , ∀ b ∈ ( − 1 , a T y π ∗ ( i ) ) , ∀ c ∈ ( a T y π ∗ ( i ) − b , 2 ) 0 , otherwise \left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\pi^*(i)}-b}{c}\right)\right\rvert
=
\begin{cases}
\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)\right\rvert, & \forall b\in(a^{\mathsf{T}}y_{\pi^*(i)},a^{\mathsf{T}}x_i),\ \forall c\in(a^{\mathsf{T}}x_i-b,2)\\
\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\pi^*(i)}-b}{c}\right)\right\rvert, & \forall b\in(-1,a^{\mathsf{T}}y_{\pi^*(i)}),\ \forall c\in(a^{\mathsf{T}}y_{\pi^*(i)}-b,2)\\
0, & \text{otherwise}
\end{cases} σ ( c a T x i − b ) − σ ( c a T y π ∗ ( i ) − b ) = ⎩ ⎨ ⎧ σ ( c a T x i − b ) , σ ( c a T x i − b ) − σ ( c a T y π ∗ ( i ) − b ) , 0 , ∀ b ∈ ( a T y π ∗ ( i ) , a T x i ) , ∀ c ∈ ( a T x i − b , 2 ) ∀ b ∈ ( − 1 , a T y π ∗ ( i ) ) , ∀ c ∈ ( a T y π ∗ ( i ) − b , 2 ) otherwise
Hence, conditioned on a specific a ∈ S d − 1 a\in\mathcal{S}^{d-1} a ∈ S d − 1 , we can write the conditional expectation over b , c b,c b , c as:
E b , c ∣ σ ( a T x i − b c ) − σ ( a T y π ∗ ( i ) − b c ) ∣ = ∫ b = a T y π ∗ ( i ) a T x i ∫ c = a T x i − b 2 σ ( a T x i − b c ) d c d b + ∫ b = − 1 a T y T ∗ ( i ) ∫ c = a T y T ∗ ( i ) − b 2 ∣ σ ( a T x i − b c ) − σ ( a T y T ∗ ( i ) − b c ) ∣ d c d b ≤ κ 2 4 ∫ b = a T y T ∗ ( i ) a T x i ( a T x i − b ) ln ( 2 a T x i − b ) d b + κ 2 4 ∣ a T ( x i − y T ∗ ( i ) ) ∣ ∫ b = − 1 a T y T ∗ ( i ) ln ( 2 a T y T ∗ ( i ) − b ) d b ≤ κ 2 4 ( ∫ 0 a T ( x i − y T ∗ ( i ) ) z ln ( 2 z ) d z + ∣ a T ( x i − y T ∗ ( i ) ) ∣ ∫ 0 a T ( y T ∗ ( i ) + 1 ) ln ( 2 z ) d z ) ≤ κ 2 4 ( ∣ a T ( x i − y T ∗ ( i ) ) ∣ + 2 ∣ a T ( x i − y T ∗ ( i ) ) ∣ ) < κ 2 ∣ a T ( x i − y T ∗ ( i ) ) ∣ ≤ κ 2 ∥ x i − y T ∗ ( i ) ∥ , by Cauchy Schwarz \begin{aligned}
\mathbb{E}_{b,c}\left\lvert\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\pi^*(i)}-b}{c}\right)\right\rvert
=\int_{b=a^{\mathsf{T}}y_{\pi^*(i)}}^{a^{\mathsf{T}}x_i}\int_{c=a^{\mathsf{T}}x_i-b}^{2}\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)\,dc\,db \\
&+\int_{b=-1}^{a^{\mathsf{T}}y_{\mathcal{T}^*(i)}}\int_{c=a^{\mathsf{T}}y_{\mathcal{T}^*(i)}-b}^{2}\left|\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\mathcal{T}^*(i)}-b}{c}\right)\right|\,dc\,db \\
\le\frac{\kappa_2}{4}\int_{b=a^{\mathsf{T}}y_{\mathcal{T}^*(i)}}^{a^{\mathsf{T}}x_i}(a^{\mathsf{T}}x_i-b)\ln\left(\frac{2}{a^{\mathsf{T}}x_i-b}\right)\,db \\
+\frac{\kappa_2}{4}\left|a^{\mathsf{T}}(x_i-y_{\mathcal{T}^*(i)})\right|\int_{b=-1}^{a^{\mathsf{T}}y_{\mathcal{T}^*(i)}}\ln\left(\frac{2}{a^{\mathsf{T}}y_{\mathcal{T}^*(i)}-b}\right)\,db \\
\le\frac{\kappa_2}{4}\left(\int_{0}^{a^{\mathsf{T}}(x_i-y_{\mathcal{T}^*(i)})}z\ln\left(\frac{2}{z}\right)\,dz+\left|a^{\mathsf{T}}(x_i-y_{\mathcal{T}^*(i)})\right|\int_{0}^{a^{\mathsf{T}}(y_{\mathcal{T}^*(i)}+1)}\ln\left(\frac{2}{z}\right)\,dz\right) \\
\le\frac{\kappa_2}{4}\left(\left|a^{\mathsf{T}}(x_i-y_{\mathcal{T}^*(i)})\right|+2\left|a^{\mathsf{T}}(x_i-y_{\mathcal{T}^*(i)})\right|\right)<\kappa_2\left|a^{\mathsf{T}}(x_i-y_{\mathcal{T}^*(i)})\right| \\
\le\kappa_2\|x_i-y_{\mathcal{T}^*(i)}\|,\ \text{by Cauchy Schwarz}
\end{aligned} E b , c σ ( c a T x i − b ) − σ ( c a T y π ∗ ( i ) − b ) = ∫ b = a T y π ∗ ( i ) a T x i ∫ c = a T x i − b 2 σ ( c a T x i − b ) d c d b ≤ 4 κ 2 ∫ b = a T y T ∗ ( i ) a T x i ( a T x i − b ) ln ( a T x i − b 2 ) d b + 4 κ 2 a T ( x i − y T ∗ ( i ) ) ∫ b = − 1 a T y T ∗ ( i ) ln ( a T y T ∗ ( i ) − b 2 ) d b ≤ 4 κ 2 ( ∫ 0 a T ( x i − y T ∗ ( i ) ) z ln ( z 2 ) d z + a T ( x i − y T ∗ ( i ) ) ∫ 0 a T ( y T ∗ ( i ) + 1 ) ln ( z 2 ) d z ) ≤ 4 κ 2 ( a T ( x i − y T ∗ ( i ) ) + 2 a T ( x i − y T ∗ ( i ) ) ) < κ 2 a T ( x i − y T ∗ ( i ) ) ≤ κ 2 ∥ x i − y T ∗ ( i ) ∥ , by Cauchy Schwarz + ∫ b = − 1 a T y T ∗ ( i ) ∫ c = a T y T ∗ ( i ) − b 2 σ ( c a T x i − b ) − σ ( c a T y T ∗ ( i ) − b ) d c d b
Thus taking one more expectation over a ∼ S d − 1 a\sim\mathbb{S}^{d-1} a ∼ S d − 1 , and summing over all i ∈ [ M ] i\in[M] i ∈ [ M ] , we get:
E a , b , c ∑ i = 1 M ∣ σ ( a T x i − b c ) − σ ( a T y T ∗ ( i ) − b c ) ∣ ≤ κ 2 ∑ i = 1 M ∥ x i − y T ∗ ( i ) ∥ \mathbb{E}_{a,b,c}\sum_{i=1}^{M}\left|\sigma\left(\frac{a^{\mathsf{T}}x_i-b}{c}\right)-\sigma\left(\frac{a^{\mathsf{T}}y_{\mathcal{T}^*(i)}-b}{c}\right)\right|\le\kappa_2\sum_{i=1}^{M}\|x_i-y_{\mathcal{T}^*(i)}\| E a , b , c i = 1 ∑ M σ ( c a T x i − b ) − σ ( c a T y T ∗ ( i ) − b ) ≤ κ 2 i = 1 ∑ M ∥ x i − y T ∗ ( i ) ∥
Thus, combining our computations for ”isolated” y j y_j y j ’s and ”coupled” ( x i , y T ∗ ( i ) ) (x_i,y_{\mathcal{T}^*(i)}) ( x i , y T ∗ ( i ) ) ’s we get that:
E a , b , c ∣ F ( S ) − F ( T ) ∣ ≤ κ 2 4 ∑ j ≠ T ∗ ( i ) ∥ y j − z ∥ + κ 2 ∑ i = 1 M ∥ x i − y T ∗ ( i ) ∥ ≤ κ 2 W ( k ) ( S , T ) \mathbb{E}_{a,b,c}|F(S)-F(T)|\le\frac{\kappa_2}{4}\sum_{j\ne\mathcal{T}^*(i)}\|y_j-z\|+\kappa_2\sum_{i=1}^{M}\|x_i-y_{\mathcal{T}^*(i)}\|\le\kappa_2\mathsf{W}^{(k)}(S,T) E a , b , c ∣ F ( S ) − F ( T ) ∣ ≤ 4 κ 2 j = T ∗ ( i ) ∑ ∥ y j − z ∥ + κ 2 i = 1 ∑ M ∥ x i − y T ∗ ( i ) ∥ ≤ κ 2 W ( k ) ( S , T )
And our proof of F F F being upper Lipschitz is complete. □ \square □
Universal approximators with MAS functions
Theorem 5.3.10 (Universality). Let V V V be a finite ground set, and let F : P ≤ k ( V ) → R m F:\mathcal{P}_{\le k}(V)\to\mathbb{R}^{m} F : P ≤ k ( V ) → R m be a MAS function. Then for every multiset-to-vector monotone function f : P ≤ k ( V ) → R s f:\mathcal{P}_{\le k}(V)\to\mathbb{R}^{s} f : P ≤ k ( V ) → R s , there exists a vector-to-vector monotone function M : R m → R s M:\mathbb{R}^{m}\to\mathbb{R}^{s} M : R m → R s such that F ( S ) = M ∘ f ( S ) F(S)=M\circ f(S) F ( S ) = M ∘ f ( S ) .
Proof. A MAS function is in particular invertible, and therefore we can write f ( S ) = f ∘ F − 1 ∘ F ( S ) f(S)=f\circ F^{-1}\circ F(S) f ( S ) = f ∘ F − 1 ∘ F ( S ) . On the image of F F F we can define the function M = f ∘ F − 1 M=f\circ F^{-1} M = f ∘ F − 1 which is a vector-to-vector functions, and it is monotone on the image of F F F , because if v = F ( S ) , u = F ( T ) v=F(S),u=F(T) v = F ( S ) , u = F ( T ) for some sets S , T S,T S , T , and if v ≤ u v\le u v ≤ u , then by separability S ⊆ T S\subseteq T S ⊆ T , and therefore
M ( v ) = f ∘ F − 1 ( v ) = f ( S ) ≤ f ( T ) = f ∘ F − 1 ( u ) = M ( u ) M(v)=f\circ F^{-1}(v)=f(S)\le f(T)=f\circ F^{-1}(u)=M(u) M ( v ) = f ∘ F − 1 ( v ) = f ( S ) ≤ f ( T ) = f ∘ F − 1 ( u ) = M ( u )
Finally, we need to show that M M M can be extended to a monotone function on all of R m \mathbb{R}^{m} R m . We accomplish this by defining
M ( v ) = max { M ( u ) ∣ u ≤ v , u ∈ Image ( F ) } . M(v) = \max\{M(u) \mid u \leq v, u \in\operatorname{Image}(F)\}. M ( v ) = max { M ( u ) ∣ u ≤ v , u ∈ Image ( F )} .
Details about our model
Architecture details:
In all 4 of MASNet models, we use an elementwise neural network N N θ : R d → R m NN_{\theta}: \mathbb{R}^{d} \to\mathbb{R}^{m} N N θ : R d → R m , where the ground set V ⊆ R d V \subseteq\mathbb{R}^{d} V ⊆ R d and the output is in R m \mathbb{R}^{m} R m . We follow that with a sum aggregation, followed by a monotone vector-to-vector neural network M 2 , ϕ : R m → R m M_{2,\phi}: \mathbb{R}^{m} \to\mathbb{R}^{m} M 2 , ϕ : R m → R m . For the embedding Neural Network, we use a 1 hidden layer NN with ReLU activation in the hidden layers and ReLU or Hat activation in the output, depending on whether we’re using MASNet-ReLU or Hat activation based MASNet. To make M 2 , ϕ M_{2,\phi} M 2 , ϕ monotone, we use non-negative weights by taking the absolute value of parameters before applying the linear transformation, and use all monotonic activations like ReLU in the intermediate layers. Now, as discussed before, we give details of our model MASNet-INT.
MASNet-INT
In this appendix we design universal approximators for the class of Hat functions to learn σ \sigma σ . We parametrize For that, we derive equivalent conditions on the derivatives of σ \sigma σ (per embedding dimension) and use the techniques of learning functions by modelling the derivatives using neural networks and then using a numerical integration as in Wehenkel and Louppe [34 ] .
Lemma C.1. If σ : R → R ≥ 0 \sigma: \mathbb{R} \to\mathbb{R}_{\geq0} σ : R → R ≥ 0 belongs to the hat activation class, then σ ′ \sigma' σ ′ satisfies the following conditions for some positive constants c , C > 0 c, C > 0 c , C > 0 and α ∈ R \alpha\in\mathbb{R} α ∈ R , β > 0 \beta> 0 β > 0 , γ ∈ ( 0 , 1 ) \gamma\in(0, 1) γ ∈ ( 0 , 1 ) :
supp ( σ ′ ) ⊆ [ α , α + β ] \operatorname{supp}(\sigma') \subseteq[\alpha, \alpha+ \beta] supp ( σ ′ ) ⊆ [ α , α + β ]
∫ α α + β σ ′ ( x ) d x = 0 \int_{\alpha}^{\alpha+\beta} \sigma'(x)\,\mathrm{d}x = 0 ∫ α α + β σ ′ ( x ) d x = 0
σ ′ ( x ) ≤ C \sigma'(x) \leq C σ ′ ( x ) ≤ C , ∀ x ∈ R \forall x \in\mathbb{R} ∀ x ∈ R
σ ′ ( x ) ≥ c \sigma'(x) \geq c σ ′ ( x ) ≥ c , ∀ x ∈ ( α , α + γ ⋅ β ) \forall x \in(\alpha, \alpha+ \gamma\cdot\beta) ∀ x ∈ ( α , α + γ ⋅ β )
The above 4 conditions, along with σ ( α ) = 0 \sigma(\alpha) = 0 σ ( α ) = 0 , are the necessary and sufficient conditions that characterize the hat activation class.
Proof. We first prove that, if σ \sigma σ belongs to the hat activation class as defined in 5.3.4, then conditions 1-4 are satisfied. Note that, if σ \sigma σ is a Hat activation, then σ \sigma σ has compact support in some [ α , α + β ] [\alpha, \alpha+ \beta] [ α , α + β ] by definition, and it’s piecewise continuously differentiable. Thus, outside [ α , α + β ] [\alpha, \alpha+\beta] [ α , α + β ] , we have σ ≡ 0 ⇒ σ ′ ≡ 0 \sigma\equiv0 \Rightarrow\sigma' \equiv0 σ ≡ 0 ⇒ σ ′ ≡ 0 . Hence, supp ( σ ′ ) ⊆ [ α , α + β ] \operatorname{supp}(\sigma') \subseteq[\alpha, \alpha+\beta] supp ( σ ′ ) ⊆ [ α , α + β ] . We also have that, 0 = σ ( α ) = σ ( α + β ) ⇒ σ ( α + β ) − σ ( α ) = 0 ⇒ ∫ α α + β σ ′ ( t ) d t = 0 0=\sigma(\alpha)=\sigma(\alpha+\beta) \Rightarrow\sigma(\alpha+\beta)-\sigma(\alpha)=0 \Rightarrow\int_{\alpha}^{\alpha+\beta}\sigma'(t)\,\mathrm{d}t=0 0 = σ ( α ) = σ ( α + β ) ⇒ σ ( α + β ) − σ ( α ) = 0 ⇒ ∫ α α + β σ ′ ( t ) d t = 0 . Finally, σ \sigma σ being Lipschitz is equivalent to ∣ σ ′ ( t ) ∣ ≤ C |\sigma'(t)|\le C ∣ σ ′ ( t ) ∣ ≤ C for some C > 0 C>0 C > 0 and by continuity of σ ′ \sigma' σ ′ in ( α , α + β ) (\alpha,\alpha+\beta) ( α , α + β ) we must have that 2 c : = lim t → α + σ ′ ( t ) > 0 ⇒ σ ′ ( t ) ≥ c , ∀ t ∈ ( α , α + γ ⋅ β ) 2c:=\lim_{t\to\alpha+}\sigma'(t)>0 \Rightarrow\sigma'(t)\ge c,\ \forall t\in(\alpha,\alpha+\gamma\cdot\beta) 2 c := lim t → α + σ ′ ( t ) > 0 ⇒ σ ′ ( t ) ≥ c , ∀ t ∈ ( α , α + γ ⋅ β ) for some γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) . Thus, conditions 1-4 are implied by σ \sigma σ belonging to the Hat activation class.
Moreover, if conditions 1-4 are satisfied, and σ ( α ) = 0 \sigma(\alpha)=0 σ ( α ) = 0 , then condition-2 implies σ ( α + β ) = 0 \sigma(\alpha+\beta)=0 σ ( α + β ) = 0 . This, along with condition-1 implies that σ ( t ) = 0 , ∀ t ∉ [ α , α + β ] \sigma(t)=0,\ \forall t\notin[\alpha,\alpha+\beta] σ ( t ) = 0 , ∀ t ∈ / [ α , α + β ] . Condition-4 immediately implies that σ ( α + γ ⋅ β ) ≥ c ⋅ γ β > 0 \sigma(\alpha+\gamma\cdot\beta)\ge c\cdot\gamma\beta>0 σ ( α + γ ⋅ β ) ≥ c ⋅ γ β > 0 , thus σ ≢ 0 \sigma\not\equiv0 σ ≡ 0 . On the other hand, we get that σ ′ ≤ C \sigma'\le C σ ′ ≤ C implies by LMVT that, ∣ σ ( x ) − σ ( y ) ∣ ≤ C ⋅ ∣ x − y ∣ |\sigma(x)-\sigma(y)|\le C\cdot|x-y| ∣ σ ( x ) − σ ( y ) ∣ ≤ C ⋅ ∣ x − y ∣ . Hence, σ \sigma σ belongs to the class of Hat activations following the definition from 5.3.4. □ \square □
Neural Parametrization of MASNet-INT: Consider h θ 1 ( ⋅ ) , h ϕ 2 ( ⋅ ) h_\theta^1(\cdot),h_\phi^2(\cdot) h θ 1 ( ⋅ ) , h ϕ 2 ( ⋅ ) to be non-negative + bounded, and bounded fully connected Neural Networks, respectively, of one hidden layer each. Also consider the trainable support parameters to be α ∈ R m , β ∈ R + m , γ ∈ ( 0 , 1 ) m \boldsymbol{\alpha}\in\mathbb{R}^m,\boldsymbol{\beta}\in\mathbb{R}_+^m,\boldsymbol{\gamma}\in(0,1)^m α ∈ R m , β ∈ R + m , γ ∈ ( 0 , 1 ) m where m m m is the output dimension and α , β , γ \boldsymbol{\alpha},\boldsymbol{\beta},\boldsymbol{\gamma} α , β , γ are the support parameters, aggregated for all output dimensions. Now, we define an integral based model of parametric hat functions. Let Θ = ( θ , ϕ , α , β , m ) \Theta=(\theta,\phi,\boldsymbol{\alpha},\boldsymbol{\beta},m) Θ = ( θ , ϕ , α , β , m ) be the parameter space.
σ Θ ( x ) = ∫ α x h θ 1 ( z ) 1 { α ≤ z ≤ α + γ β } d z − [ ∫ α α + γ β h θ 1 ( z ) d z ∫ α + γ β α + β h ϕ 2 ( z ) d z ] ( ∫ α + γ β x h ϕ 2 ( z ) 1 { α + γ β ≤ z ≤ α + β } d z ) (60) \sigma_\Theta(x)=\int_{\alpha}^{x}h_\theta^1(z)\mathbf{1}_{\{\alpha\le z\le\alpha+\gamma\beta\}}\,\mathrm{d}z-\left[\frac{\int_{\alpha}^{\alpha+\gamma\beta}h_\theta^1(z)\,\mathrm{d}z}{\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_\phi^2(z)\,\mathrm{d}z}\right]\left(\int_{\alpha+\gamma\beta}^{x}h_\phi^2(z)\mathbf{1}_{\{\alpha+\gamma\beta\le z\le\alpha+\beta\}}\,\mathrm{d}z\right) \tag*{(60)} σ Θ ( x ) = ∫ α x h θ 1 ( z ) 1 { α ≤ z ≤ α + γ β } d z − ∫ α + γ β α + β h ϕ 2 ( z ) d z ∫ α α + γ β h θ 1 ( z ) d z ( ∫ α + γ β x h ϕ 2 ( z ) 1 { α + γ β ≤ z ≤ α + β } d z ) (60)
In the above formulation, in addition to the support parameters we also learn the function itself rather than performing a linear interpolation, this makes the above class a universal approximator of hat functions.
Lemma C.2 (Universal Hat approximator). The family σ Θ \sigma_\Theta σ Θ defined in Equation (60) is an universal approximator of the class of Hat functions having support in [ α , α + β ] [\alpha,\alpha+\beta] [ α , α + β ] with σ ′ ( t ) > 0 , ∀ t ∈ ( α , α + γ β ) \sigma'(t)>0,\ \forall t\in(\alpha,\alpha+\gamma\beta) σ ′ ( t ) > 0 , ∀ t ∈ ( α , α + γ β )
Proof. We show that the given parametric model is an universal approximator of Hat functions using their equivalent formulation in Lemma C.1. Let σ 1 \sigma_1 σ 1 be σ \sigma σ restricted to the interval [ α , α + γ β ] [\alpha,\alpha+\gamma\beta] [ α , α + γ β ] and σ 2 \sigma_2 σ 2 be σ \sigma σ restricted to the interval [ α + γ β , α + β ] [\alpha+\gamma\beta,\alpha+\beta] [ α + γ β , α + β ] . Using Theorem-1 from Lu, Pu, Wang, Hu, and Wang [25 ] , we can get ReLU neural network h θ 1 : R → R h_\theta^1:\mathbb{R}\to\mathbb{R} h θ 1 : R → R and h ϕ 2 : R → R h_\phi^2:\mathbb{R}\to\mathbb{R} h ϕ 2 : R → R of width 5 each such that they approximate the derivatives of the restricted functions σ 1 ′ \sigma_1' σ 1 ′ and σ 2 ′ \sigma_2' σ 2 ′ . Thus, the following hold for any ϵ > 0 \epsilon>0 ϵ > 0 and any constant c ∈ R c\in\mathbb{R} c ∈ R :
∫ α α + γ β ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z ≤ ϵ and ∫ α + γ β α + β ∣ σ 2 ′ ( t ) + c ⋅ h ϕ 2 ( t ) ∣ d t ≤ ϵ (61) \int_{\alpha}^{\alpha+\gamma\beta}\left|\sigma_1'(z)-h_\theta^1(z)\right|\,\mathrm{d}z\le\epsilon\quad\text{and}\quad\int_{\alpha+\gamma\beta}^{\alpha+\beta}\left|\sigma_2'(t)+c\cdot h_\phi^2(t)\right|\,\mathrm{d}t\le\epsilon \tag*{(61)} ∫ α α + γ β σ 1 ′ ( z ) − h θ 1 ( z ) d z ≤ ϵ and ∫ α + γ β α + β σ 2 ′ ( t ) + c ⋅ h ϕ 2 ( t ) d t ≤ ϵ (61)
We now observe that, if x ∈ ( − ∞ , α ) x\in(-\infty,\alpha) x ∈ ( − ∞ , α ) , both the indicator functions 1 { α ≤ z ≤ α + γ β } \mathbf{1}_{\{\alpha\le z\le\alpha+\gamma\beta\}} 1 { α ≤ z ≤ α + γ β } and 1 { α + γ β ≤ z ≤ α + β } 1_{\{\alpha+\gamma\beta\le z\le\alpha+\beta\}} 1 { α + γ β ≤ z ≤ α + β } as in the integrands of Equation (5.15) evaluate to 0, thus σ Θ \sigma_{\Theta} σ Θ exactly coincides with σ \sigma σ . On the other hand, if x ∈ ( α + β , ∞ ) x\in(\alpha+\beta,\infty) x ∈ ( α + β , ∞ ) then we would have the first integral of 5.15 evaluate to ∫ α α + β h θ 1 ( z ) d z \int_{\alpha}^{\alpha+\beta}h_{\theta}^{1}(z)\,dz ∫ α α + β h θ 1 ( z ) d z (due to presence of the indicator 1 { α ≤ z ≤ α + γ β } 1_{\{\alpha\le z\le\alpha+\gamma\beta\}} 1 { α ≤ z ≤ α + γ β } and the second expression evaluates to − [ ∫ α α + β h θ 1 ( z ) d z ∫ α + γ β α + β h ϕ 2 ( z ) d z ] ( ∫ α + γ β α + β h ϕ 2 ( z ) d z ) = − ∫ α α + γ β h θ 1 ( z ) d z -\left[\frac{\int_{\alpha}^{\alpha+\beta}h_{\theta}^{1}(z)\,dz}{\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_{\phi}^{2}(z)\,dz}\right]\left(\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_{\phi}^{2}(z)\,dz\right)=-\int_{\alpha}^{\alpha+\gamma\beta}h_{\theta}^{1}(z)\,dz − [ ∫ α + γ β α + β h ϕ 2 ( z ) d z ∫ α α + β h θ 1 ( z ) d z ] ( ∫ α + γ β α + β h ϕ 2 ( z ) d z ) = − ∫ α α + γ β h θ 1 ( z ) d z . Thus, both the expressions sum to 0 and σ Θ \sigma_{\Theta} σ Θ coincides with σ \sigma σ for x ∈ ( α + β , ∞ ) x\in(\alpha+\beta,\infty) x ∈ ( α + β , ∞ ) as well.
Now we shall show that, for any x ∈ ( α , α + β ) x\in(\alpha,\alpha+\beta) x ∈ ( α , α + β ) we must also have ∣ σ Θ ( x ) − σ ( x ) ∣ ≤ ϵ |\sigma_{\Theta}(x)-\sigma(x)|\le\epsilon ∣ σ Θ ( x ) − σ ( x ) ∣ ≤ ϵ , implying convergence in sup norm. We consider the following cases:
Case-I: When x ∈ ( α , α + γ β ) x\in(\alpha,\alpha+\gamma\beta) x ∈ ( α , α + γ β ) , we have that:
∣ σ ( x ) − σ Θ ( x ) ∣ = ∣ ∫ α x σ 1 ′ ( z ) d z − ∫ α x h θ 1 ( z ) d z ∣ ≤ ∫ α x ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z , by triangle ineq. ≤ ∫ α α + β γ ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z , by non-negativity of integrand ≤ ϵ , by Equation (5.16) \begin{aligned}
|\sigma(x)-\sigma_{\Theta}(x)|
&=\left|\int_{\alpha}^{x}\sigma_{1}'(z)\,dz-\int_{\alpha}^{x}h_{\theta}^{1}(z)\,dz\right|\\
&\le\int_{\alpha}^{x}|\sigma_{1}'(z)-h_{\theta}^{1}(z)|\,dz,\text{ by triangle ineq.}\\
&\le\int_{\alpha}^{\alpha+\beta\gamma}|\sigma_{1}'(z)-h_{\theta}^{1}(z)|\,dz,\text{ by non-negativity of integrand}\\
&\le\epsilon,\text{ by Equation (5.16)}
\end{aligned} ∣ σ ( x ) − σ Θ ( x ) ∣ = ∫ α x σ 1 ′ ( z ) d z − ∫ α x h θ 1 ( z ) d z ≤ ∫ α x ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z , by triangle ineq. ≤ ∫ α α + β γ ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z , by non-negativity of integrand ≤ ϵ , by Equation (5.16)
∣ σ ( x ) − σ Θ ( x ) ∣ = ∣ ∫ α x σ ′ ( z ) d z − ∫ α α + γ β h θ 1 ( z ) d z − ∫ α + γ β x h ϕ 2 ( z ) d z ∣ = ∣ ∫ α α + γ β ( σ 1 ′ ( z ) − h θ 1 ( z ) ) d z + ∫ α + γ β x ( σ 2 ′ ( z ) + [ ∫ α α + γ β h θ 1 ( z ) d z ∫ α + γ β α + β h ϕ 2 ( z ) d z ] h ϕ 2 ( z ) ) d z ∣ ≤ ∫ α α + γ β ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z + ∫ α + γ β x ∣ σ 2 ′ ( z ) + [ ∫ α α + γ β h θ 1 ( z ) d z ∫ α + γ β α + β h ϕ 2 ( z ) d z ] h ϕ 2 ( z ) ∣ d z , tri. ineq. ≤ ϵ + ∫ α + γ β α + β ∣ σ 2 ′ ( z ) + [ ∫ α α + γ β h θ 1 ( z ) d z ∫ α + γ β α + β h ϕ 2 ( z ) d z ] h ϕ 2 ( z ) ∣ d z , by 5.16 on 1st term & tri. ineq. on 2nd ≤ 2 ϵ \begin{aligned}
|\sigma(x)-\sigma_{\Theta}(x)|
&=\left|\int_{\alpha}^{x}\sigma'(z)\,dz-\int_{\alpha}^{\alpha+\gamma\beta}h_{\theta}^{1}(z)\,dz-\int_{\alpha+\gamma\beta}^{x}h_{\phi}^{2}(z)\,dz\right|\\
&=\left|\int_{\alpha}^{\alpha+\gamma\beta}\left(\sigma_{1}'(z)-h_{\theta}^{1}(z)\right)\,dz+\int_{\alpha+\gamma\beta}^{x}\left(\sigma_{2}'(z)+\left[\frac{\int_{\alpha}^{\alpha+\gamma\beta}h_{\theta}^{1}(z)\,dz}{\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_{\phi}^{2}(z)\,dz}\right]h_{\phi}^{2}(z)\right)\,dz\right|\\
&\le\int_{\alpha}^{\alpha+\gamma\beta}|\sigma_{1}'(z)-h_{\theta}^{1}(z)|\,dz+\int_{\alpha+\gamma\beta}^{x}\left|\sigma_{2}'(z)+\left[\frac{\int_{\alpha}^{\alpha+\gamma\beta}h_{\theta}^{1}(z)\,dz}{\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_{\phi}^{2}(z)\,dz}\right]h_{\phi}^{2}(z)\right|\,dz,\text{ tri. ineq.}\\
&\le\epsilon+\int_{\alpha+\gamma\beta}^{\alpha+\beta}\left|\sigma_{2}'(z)+\left[\frac{\int_{\alpha}^{\alpha+\gamma\beta}h_{\theta}^{1}(z)\,dz}{\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_{\phi}^{2}(z)\,dz}\right]h_{\phi}^{2}(z)\right|\,dz,\text{ by 5.16 on 1st term \& tri. ineq. on 2nd}\\
&\le2\epsilon
\end{aligned} ∣ σ ( x ) − σ Θ ( x ) ∣ = ∫ α x σ ′ ( z ) d z − ∫ α α + γ β h θ 1 ( z ) d z − ∫ α + γ β x h ϕ 2 ( z ) d z = ∫ α α + γ β ( σ 1 ′ ( z ) − h θ 1 ( z ) ) d z + ∫ α + γ β x σ 2 ′ ( z ) + ∫ α + γ β α + β h ϕ 2 ( z ) d z ∫ α α + γ β h θ 1 ( z ) d z h ϕ 2 ( z ) d z ≤ ∫ α α + γ β ∣ σ 1 ′ ( z ) − h θ 1 ( z ) ∣ d z + ∫ α + γ β x σ 2 ′ ( z ) + ∫ α + γ β α + β h ϕ 2 ( z ) d z ∫ α α + γ β h θ 1 ( z ) d z h ϕ 2 ( z ) d z , tri. ineq. ≤ ϵ + ∫ α + γ β α + β σ 2 ′ ( z ) + ∫ α + γ β α + β h ϕ 2 ( z ) d z ∫ α α + γ β h θ 1 ( z ) d z h ϕ 2 ( z ) d z , by 5.16 on 1st term & tri. ineq. on 2nd ≤ 2 ϵ
where last inequality is obtained c = ∫ α α + γ β h θ 1 ( z ) d z ∫ α + γ β α + β h ϕ 2 ( z ) d z c=\frac{\int_{\alpha}^{\alpha+\gamma\beta}h_{\theta}^{1}(z)\,dz}{\int_{\alpha+\gamma\beta}^{\alpha+\beta}h_{\phi}^{2}(z)\,dz} c = ∫ α + γ β α + β h ϕ 2 ( z ) d z ∫ α α + γ β h θ 1 ( z ) d z in the universal approximator for σ 2 ′ \sigma_{2}' σ 2 ′ in (61) . Thus, just we see that, for any ϵ > 0 \epsilon>0 ϵ > 0 and α ∈ R , β > 0 , γ ∈ ( 0 , 1 ) \alpha\in\mathbb{R},\beta>0,\gamma\in(0,1) α ∈ R , β > 0 , γ ∈ ( 0 , 1 ) we can find a universal approximator for the Hat function class
Parametrizing $\alpha,\beta,\gamma We have provided a universal approximator for the Hat function class given α , β , γ \alpha,\beta,\gamma α , β , γ but we’d also want to have α , β , γ \alpha,\beta,\gamma α , β , γ as learnable parameters. This is valid for both MASNet-Hat and MASNet-INT, as in both cases we seek to learn the support parameters. Now, there is no constraint on α \alpha α , so we can directly initialize α ∈ R 3 m \alpha\in\mathbb{R}^{3m} α ∈ R 3 m and learn it through gradient descent. However, we have a constraint on β \beta β that β > 0 \beta> 0 β > 0 . For this, we first initialize β 0 ∈ R m \beta_0 \in\mathbb{R}^{m} β 0 ∈ R m and we obtain β ∈ R m \beta\in\mathbb{R}^{m} β ∈ R m by applying co-ordinatewise positive transformation on β 0 \beta_0 β 0 . This we do in two ways, depending on which works better on a given task: (I) by simply taking β 0 ↦ ∣ β 0 ∣ \beta_0 \mapsto\lvert\beta_0\rvert β 0 ↦ ∣ β 0 ∣ , where the absolute value is taken pointwise. (II) by taking the pointwise transformation β 0 ↦ ELU ( β 0 ; υ ) + υ \beta_0 \mapsto\operatorname{ELU}(\beta_0;\upsilon)+\upsilon β 0 ↦ ELU ( β 0 ; υ ) + υ , where ELU ( ⋅ ; υ ) \operatorname{ELU}(\cdot;\upsilon) ELU ( ⋅ ; υ ) is the Exponential Linear Unit with hyperparameter υ > 0 \upsilon> 0 υ > 0 . Note that, the ELU function is given by:
ELU ( x ; υ ) = { x , if x > 0 , υ ( e x − 1 ) , if x ≤ 0 \operatorname{ELU}(x;\upsilon)=
\begin{cases}
x, & \text{if } x>0,\\
\upsilon(e^x-1), & \text{if } x\leq0
\end{cases} ELU ( x ; υ ) = { x , υ ( e x − 1 ) , if x > 0 , if x ≤ 0
For parametrizing γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , we similarly initialize γ 0 ∈ R m \gamma_0 \in\mathbb{R}^{m} γ 0 ∈ R m and then apply a pointwise transformation that takes each co-ordinate to ( 0 , 1 ) (0,1) ( 0 , 1 ) . We use tempered sigmoid function: γ 0 ↦ Sigmoid ( τ ⋅ γ 0 ) \gamma_0 \mapsto\operatorname{Sigmoid}(\tau\cdot\gamma_0) γ 0 ↦ Sigmoid ( τ ⋅ γ 0 ) where τ > 0 \tau> 0 τ > 0 is a hyperparameter
Designing soft indicator functions: In the integrands for MASNet-INT, we have some indicator functions of the form 1 { a ≤ ⋅ ≤ b } ( x ) = \mathbf{1}_{\{a\leq\cdot\leq b\}}(x)= 1 { a ≤ ⋅ ≤ b } ( x ) =
{ 1 , if x ∈ [ a , b ] , 0 , o.w. \begin{cases}
1, & \text{if } x\in[a,b],\\
0, & \text{o.w.}
\end{cases} { 1 , 0 , if x ∈ [ a , b ] , o.w.
We want these indicator functions to have non zero gradients as we intend to learn the ”support parameters” a , b a,b a , b of such indicator function. If we chose binary indicators, then that is not differentiable everywhere and have gradient 0 in most places, making it difficult to learn a , b a,b a , b . Thus, we design a ”soft” indicator function with the help of tempered sigmoid with hyperparameter τ S \tau_S τ S as follows:
1 { a ≤ ⋅ ≤ b } ( x ) = Sigmoid ( τ S ⋅ ( x − a ) ) ⋅ Sigmoid ( τ S ⋅ ( b − x ) ) \mathbf{1}_{\{a\leq\cdot\leq b\}}(x)=\operatorname{Sigmoid}\left(\tau_S\cdot(x-a)\right)\cdot\operatorname{Sigmoid}\left(\tau_S\cdot(b-x)\right) 1 { a ≤ ⋅ ≤ b } ( x ) = Sigmoid ( τ S ⋅ ( x − a ) ) ⋅ Sigmoid ( τ S ⋅ ( b − x ) )
Making MASNet Mini Batch Consistent
There has been a chain of works to make set-based neural models to be mini batch consistent in [9 , 35 ] which have applications in large scale machine learning. The goal of these Mini Batch Consistent (MBC) works is to design a set function F F F , which, if applied on any subsets from partitions of a set S S S and pooled through an activation g g g , should return F ( S ) F(S) F ( S ) . Thus, at a high level, MBC functions analyse and design functions for the following batch consistency condition: g ( F ( S 1 ) , ⋯ , F ( S n ) ) = F ( S ) g(F(S_1),\cdots,F(S_n))=F(S) g ( F ( S 1 ) , ⋯ , F ( S n )) = F ( S ) . MBC methods achieve this by imposing certain restrictions on attention-based set architectures. Similar to this, our work also puts restrictions on the inner embedding transformation and the outer vector-to-vector transformations. It is thus a natural question to ask, if MASNet can be made minibatch consistent. Since set containment is the major theme of our work, with potential applications in retrieval and recommendation systems, processing huge sets at once can be a key bottleneck, and it would be nice to make MASNet process over mini-batches and then design some aggregation mechanism to make the aggregated embedding equal to the embedding of the entire set.
If we consider a partition of a given set S = ⋃ i = 1 k S i S = \bigcup_{i=1}^{k} S_i S = ⋃ i = 1 k S i , where S i S_i S i ’s are all disjoint, then MASNet ( S i ) ≤ MASNet ( S ) \operatorname{MASNet}(S_i) \leq\operatorname{MASNet}(S) MASNet ( S i ) ≤ MASNet ( S ) , by monotonicity. But, we can try to find an appropriate pooling mechanism over the subset embeddings to make the aggregated embedding same as that of the entire set. For example, if the outer transformation M θ 2 M_{\theta_2} M θ 2 , as in (50) was invertible, and then we may define a particular permutation-invariant pooling function g g g such that, g ( X 1 , ⋯ , X k ) = M θ 2 ( ∑ i = 1 k M θ 2 − 1 ( X i ) ) g(X_1,\cdots,X_k) = M_{\theta_2}\left(\sum_{i=1}^{k}M_{\theta_2}^{-1}(X_i)\right) g ( X 1 , ⋯ , X k ) = M θ 2 ( ∑ i = 1 k M θ 2 − 1 ( X i ) ) , then we’d have:
g ( MASNet ( S 1 ) , ⋯ , MASNet ( S k ) ) = M θ 2 ( M θ 2 − 1 ( MASNet ( S i ) ) ) = M θ 2 ( ∑ i = 1 k ∑ x ∈ S i σ ∘ M θ 1 ( x ) ) M θ 2 ( ∑ x ∈ S σ ∘ M θ 1 ( x ) ) = MASNet ( S ) . \begin{aligned}
g(\operatorname{MASNet}(S_1),\cdots,\operatorname{MASNet}(S_k)) &= M_{\theta_2}\left(M_{\theta_2}^{-1}(\operatorname{MASNet}(S_i))\right) \\
&= M_{\theta_2}\left(\sum_{i=1}^{k}\sum_{x\in S_i}\sigma\circ M_{\theta_1}(x)\right) \\
M_{\theta_2}\left(\sum_{x\in S}\sigma\circ M_{\theta_1}(x)\right) &= \operatorname{MASNet}(S).
\end{aligned} g ( MASNet ( S 1 ) , ⋯ , MASNet ( S k )) M θ 2 ( x ∈ S ∑ σ ∘ M θ 1 ( x ) ) = M θ 2 ( M θ 2 − 1 ( MASNet ( S i )) ) = M θ 2 ( i = 1 ∑ k x ∈ S i ∑ σ ∘ M θ 1 ( x ) ) = MASNet ( S ) .
his is irrespective of how we partition S S S into S 1 , ⋯ S k S_1,\cdots S_k S 1 , ⋯ S k . And this would enable us to process the subsets independently and aggregate them accordingly.
For our purposes, we have needed the outer transformation M θ 2 M_{\theta_2} M θ 2 to be a vector-to-vector monotone function in (50) . But that doesn’t guarantee injectivity or invertibility. Thus, we need to restrict the choice of M θ 2 M_{\theta_2} M θ 2 to be monotone and invertible for MASNet to be MBC consistent. One possible choice is to use ideas from relevant works like [4 , 34 ] to design monotone and invertible neural networks, and analyse the expressibility properties of the resulting set model. This forms an interesting direction for future work.
Additional details about experiments and more experiments
Hardware details:
All experiments were performed in a compute server, running the OS of GNU Linux Version 12, equipped with a 16 core Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz CPU architecture and equipped with a cluster of 6 NVIDIA RTX A6000 GPUs with a memory of 49GB each.
Dataset details and more experiments
We now give an account of preparation details of each dataset and accounts on more experiments on each.
Synthetic datasets and related experiments: We begin with a controlled synthetic setting where we generate pairs ( S , T ) (S,T) ( S , T ) such that k S < k T k_S < k_T k S < k T .
Each pair is labeled either positive or negative. For positive examples, we first sample k T k_T k T vectors in R d \mathbb{R}^{d} R d from N ( 0 , I d ) \mathcal{N}(0,I_d) N ( 0 , I d ) , which form the set T T T . A subset of size k S k_S k S is then uniformly sampled from T T T to obtain S S S . For negative examples, we again sample T T T as above, but then independently draw k S k_S k S vectors from R d \mathbb{R}^{d} R d to form S S S . To simulate a real world scenario of information loss, we inject gaussian noise into the vectors of S S S to get S ′ S' S ′ but keep the boolean label unchanged.
On this, we provide two more plots: Figure 5.3 and Figure 5.4 that compares the accuracies of MASNet vs baselines, by varying the gap between target set size ∣ T ∣ |T| ∣ T ∣ and query set size ∣ S ∣ |S| ∣ S ∣ . Note that the more this gap is, the more it becomes challenging to separate two non-subsets. In all cases, the embedding MLP M θ 1 M_{\theta_1} M θ 1 from Equation (50) is comprising of ≥ 2 \ge2 ≥ 2 layers.
Figure 5.3. Acc vs ∣ T ∣ |T| ∣ T ∣ for all models ≥ 2 \ge2 ≥ 2 layer MLP
Figure 5.4. Acc vs ∣ S ∣ |S| ∣ S ∣ for all models ≥ 2 \ge2 ≥ 2 layer MLP
Text-based datasets and related experiments: We evaluate our models on three text based real-world datasets: MSWEB, MSNBC, and the Amazon Baby Registry datasets, all of which reflect naturally occurring set containment structures coming out of user behavior in websites and recommendation engines.
MSWEB and MSNBC: These datasets consist of user activity logs from www.microsoft.com and www.msnbc.com , respectively. Each user session is treated as a bag of page identifiers, which are embedded into 768-dimensional vectors using a pre-trained BERT model. We construct query-target pairs ( S , T ) (S,T) ( S , T ) by sampling S S S from a session V V V , and letting T = V ∖ S T = V \setminus S T = V ∖ S . A pair is labeled 1 if S ⊆ T S \subseteq T S ⊆ T , and 0 otherwise. To simulate noisy real-world conditions, Gaussian noise is added to each element in S S S while preserving the label. The positive-to-negative ratio was taken to be 0.1, which is the number of labels 1 to labels 0 in the entire dataset.
Amazon Baby Registry: This dataset contains subsets of products selected by customers, each tagged by category (e.g., “toys” or “feeding”). Product descriptions are embedded using BERT. We filter out sets of size < 2 < 2 < 2 or > 30 > 30 > 30 , and for each valid subset S S S , we generate T ⊇ S T \supseteq S T ⊇ S of size 30 by sampling additional items from the same category. For negative samples, T T T is sampled randomly and verified to satisfy S ⊈ T S \not\subseteq T S ⊆ T .
Noise is added to S S S in both cases. This models scenarios such as predicting whether a set of displayed products T T T includes a customer’s interest S S S . On these datasets, we provide three sets of additional experiments. In the first table, we perform the same set-containment task as done in the main text, but with a different negative-to-positive ratio of 90 : 10.
Note that, due to the inductive bias of monotonicity, all the positive examples are correctly classified by design is the case of MASNet. Thus, it only has to learn to identify the negative examples. Thus, if the test set has a higher proportion of negative examples, then it throws a model a tougher task to learn. This is shown in the following Table 5.6 . Other than the modified class ratio, we provide two additional tables: in the first one, we compare shallow(1 layer) vs deep (≥ 2 \ge2 ≥ 2 layers) embedding MLP M θ 1 M_{\theta_1} M θ 1 in MASNet, which is shown in Table 5.7 . White noise of std 0.1 added for inexact set containment.
Table 5.6. Set containment in text datasets with 90:10 ratio
Table 5.7. Comparison of shallow vs. deep variants of MASNet across datasets.
Ablation studies on text datasets: From the definition of MASNet in (50) , we see that, the formulation is quite similar to Deepsets, which is of the form F ( S ) = M 2 ( ∑ x ∈ S ReLU ( a ⊤ M 1 ( x ) + b ) ) F(S) = M_2\left(\sum_{x \in S} \operatorname{ReLU}(a^\top M_1(x) + b)\right) F ( S ) = M 2 ( ∑ x ∈ S ReLU ( a ⊤ M 1 ( x ) + b ) ) . As shown earlier, using ReLU in the last layer of an elementwise function with a universally approximating M 1 M_1 M 1 is an instance of MASNet.
But it is different from DeepSets in a key point, namely:
For set containment tasks, we don’t have an outer M 2 M_2 M 2 , which DeepSets have; for universal approximation tasks, we use a monotonically increasing M 2 M_2 M 2 that we enforce by taking positive weights and increasing ac-
Table 5.8. Ablation study
tivation functions. Also, for Hat function based MASNET models, • we use a re-parametrization in which we perform a division based scaling. We now show the effect of the outer monotonic M 2 M_2 M 2 on DeepSets and division based re-parametrization on MAS-Net-Hat in the ablation study in table 5.8. White noise of std 0.1 added for inexact containment.
Pointcloud datasets and related experiments ModelNet40 [WSK+15] is a benchmark dataset of 12,311 CAD models across 40 object categories, with each object represented as a 3D point cloud. We frame our task as checking if a given point cloud S S S is a segment of a target point cloud T T T .
Firstly, we choose an object from an object category C 1 C_1 C 1 , and randomly sample 1024 points from that object to get the target point cloud T T T . To obtain a positive sample (i.e true subset) from T T T , we first sample a random center point from T T T , and then extract S S S using a hybrid approach: selecting the nearest point to the center, a few local neighbors via k-NN, and the rest via importance-weighted sampling (inverse-distance from center with noise). This makes sure that we’re selecting a true local region from the point cloud, which is in-line with the actual task of detecting whether a given segment of an object is contained in a target object. For a negative sample(non-subset), we sample S S S from an object of a different category C 2 C_2 C 2 .
We first give the accuracy table for Point-Net encoder, but with the modified negative-to-positive class ratio of 90:10 to make the task harder for MASNet, which correctly classifies the actual subsets by the inductive bias of monotonicity. The numbers with the modified class-ratio are given in Table 5.9 :
Table 5.9. Performance on Point cloud for different values of ∣ S ∣ |S| ∣ S ∣ , for 90:10 class ratio
We also give the accuracy numbers for DGCNN encoder, in Table 5.10
Table 5.10. Point cloud with DGCNN for different values of ∣ S ∣ |S| ∣ S ∣ .
Details on Linear assignment problem In this problem, we are given a positive matrix M ∈ R n × m M \in\mathbb{R}^{n \times m} M ∈ R n × m , n ≤ m n \le m n ≤ m , where M i , j M_{i,j} M i , j represents the salary that a ’worker’ will be paid to do a ’job’ j j j . The goal of the task is to maximize the average salary obtained by all workers. This is done by finding the optimal assignment π ∈ S n , m \pi\in S_{n,m} π ∈ S n , m which maps a worker i ∈ [ n ] i \in[n] i ∈ [ n ] to a ’job’ π ( i ) ∈ [ m ] \pi(i) \in[m] π ( i ) ∈ [ m ] , where by construction each worker can be mapped to at most one job, so π ( i ) ∈ { 0 , 1 } \pi(i) \in\{0,1\} π ( i ) ∈ { 0 , 1 } . This gives us the following maximization problem:
F ( M ) = 1 n max π ∈ S n , m ∑ i = 1 n M i , π ( i ) . F(M) = \frac{1}{n} \max_{\pi\in S_{n,m}} \sum_{i=1}^{n} M_{i,\pi(i)}. F ( M ) = n 1 π ∈ S n , m max i = 1 ∑ n M i , π ( i ) .
Thinking of the matrix M M M as a set of m m m columns M = [ M 1 , … , M m ] M = [M_1,\ldots,M_m] M = [ M 1 , … , M m ] , we see that the function F F F is permutation invariant and monotone. Accordingly, our goal will be to evaluate our MASNet model and baselines.
References
FSW-GNN: A Bi-Lipschitz WL-Equivalent Graph Neural Network
Yonatan Sverdlov; Yair Davidson; Nadav Dym; Tal Amir
Famously, the ability of Message Passing Neural Networks (MPNN) to distinguish between graphs is limited to graphs separable by the Weisfeiler-Leman (WL) graph isomorphism test, and the strongest MPNNs, in terms of separation power, are WL-equivalent. However, it was demonstrated that the quality of separation provided by standard WL-equivalent MPNNs can be very low, resulting in WL-separable graphs being mapped to very similar, hardly distinguishable outputs. This phenomenon can be explained by the recent observation that standard MPNNs are not lower-Lipschitz. This paper addresses this issue by introducing FSW-GNN, the first MPNN that is fully bi-Lipschitz with respect to standard WL-equivalent graph metrics. Empirically, we show that our MPNN is competitive with standard MPNNs for several graph learning tasks and is far more accurate in long-range tasks, due to its ability to avoid oversmoothing and oversquashing. Our code is available at https://github.com/yonatansverdlov/Over-squashing .
Chapter 6
FSW-GNN: A Bi-Lipschitz WL-Equivalent Graph Neural Network
Introduction
Graph neural networks are a central research topic in contemporary machine learning. As shown by Gilmer, Schoenholz, Riley, Vinyals, and Dahl [GSR+17], many of the most popular models can be seen as instantiations of Message Passing Neural Networks (MPNNs).
A well-known limitation of MPNNs is that they cannot differentiate between all distinct pairs of graphs. In fact, a pair of distinct graphs that cannot be separated by the Weisfeiler-Leman (WL) graph isomorphism test will not be separated by any MPNN [38 ] . Accordingly, the most expressive MPNNs are those that are WL-equivalent , which means they can separate all pairs of graphs that are separable by WL. WL-equivalent MPNNs were proposed in the seminal works of Xu, Hu, Leskovec, and Jegelka [38 ] and Morris, Ritzert, Fey, Hamilton, Lenssen, Rattan, and Grohe [MRF+19], and the complexity of these constructions was later improved in [ACI+22; AGA+23].
While separation is theoretically guaranteed with WL-equivalent MPNNs, in some cases, their separation in practice is so weak that it cannot be observed with 32-bit floating-point numbers; see [BKR24a]. Moreover, using many MPNN iterations often leads to almost-identical node features (oversmoothing), or features that are barely affected by changes in far-off nodes (oversquashing). These observations motivate the development of quantitative estimates of MPNN separation by means of bi-Lipschitz stability guarantees. These guarantees would ensure that Euclidean distances in the MPNN feature space are neither much larger nor much smaller than distances in the original graph space, which are defined by a suitable metric on graphs. Consequently, as we shall see, detrimental phenomena like oversmoothing and oversquashing can be fundamentally mitigated.
This paper introduces a novel MPNN, called FSW-GNN (Fourier Sliced-Wasserstein GNN ), which is bi-Lipschitz with respect to two WL-equivalent graph metrics: (a) the Doubly Stochastic (DS) metric of [Gro20], and (b) the Tree Mover’s Distance (TMD) metric of Chuang and Jegelka [CJ22]. Empirically, we show that FSW-GNN performs comparably or better than prevalent MPNNs on standard learning tasks, and achieves significantly superior performance in long-range tasks, i.e., tasks that require a large number of message-passing iterations (for example see Figure 6.1 ). This can be attributed to the bi-Lipschitzness of FSW-GNN, in contrast to standard MPNNs, which are not bi-Lipschitz [DD24].
Figure 6.1. FSW-GNN handles tasks with large radius better than standard MPNNs, which are more prone to oversmoothing and oversquashing. Task taken from [AY21]
Related Works
Bi-Lipschitzness Bi-Lipschitzness arises naturally in many domains, including frames [Bal97], phase retrieval [BCMN14; CDDL21], group-invariant learning [CCC20; CIM24] and multisets [2 AD24] . In the context of MPNNs, upper Lipschitzness was discussed in [CJ22; Lev23]. A recent survey by Morris, Dym, Maron, Ceylan, Frasca, Levie, Lim, Bronstein, Grohe, and Jegelka [MDM+24] identifies bi-Lipschitzness as a significant future goal for theoretical GNN research.
SortMPNN First steps towards a Bi-Lipschitz MPNN have recently been made by Davidson and Dym [DD24]. Their work analyses a weaker notion of Lipschitz and Hölder guarantees—in expectation over the model parameters. They show that essentially all popular MPNN models fail to be lower-Lipschitz, but are lower-Hölder in expectation, with an exponent that grows worse as the MPNN depth increases. In contrast, they propose a novel MPNN, called SortMPNN, and prove that it satisfies this weaker notion of bi-Lipschitzness in expectation.
As we discuss below, the FSW-GNN satisfies the standard, stronger, notion of bi-Lipschitzness. Interestingly, the proof technique we present here for FSW-GNN also applies to SortMPNN, allowing us to establish that SortMPNN is bi-Lipschitz as well. Nonetheless, SortMPNN has a key limitation in its message aggregation mechanism: it handles neighborhoods of different sizes by padding them to a predetermined maximal size. This requires a priori knowledge of the maximal neighborhood size in all future input graphs—an inherent constraint that significantly limits its applicability. In contrast, FSW-GNN does not share this constraint, since it treats vertex neighborhoods as distributions. Moreover, due to this padding-based approach, SortMPNN only accommodates neighborhoods that are multisets, thus making it unsuitable for graphs with non-integer edge weights. In contrast, FSW-GNN naturally supports edge weights.
MPNNs with advanced pooling mechanisms In addition to SortMPNN, our approach is conceptually related to other MPNNs that replace the basic max- or sum-pooling, used for message aggregation, with more advanced pooling mechanisms, such as sorting [6 ZCNC18] , standard deviation [CCB+20], or Sliced-Wasserstein embeddings via template distributions [KNRH21]. However, these methods lack the bi-Lipschitzness guarantees that our model provides.
WL-equivalent metrics Bi-Lipschitz analysis of MPNNs requires a WL-equivalent graph metric. Several such metrics have been proposed, with notable examples being the Doubly-Stochastic (DS) metric (also known as the tree metric ) [Gro20]; the Tree Mover’s Distance (TMD) [CJ22]; and the WL metric [CLM+22]. In this paper, we prove that the graph embeddings computed by our FSW-GNN model are bi-Lipschitz with respect to both the DS and TMD metrics. This analysis applies to graphs of bounded size with continuous, bounded node-features. Weaker notions of equivalence between these metrics, in the context of graphs with unbounded cardinality and without node features, are discussed in [BLH+24; Bök21].
Problem Setting
In this section, we outline the problem setting, first providing the theoretical background of the problem and then stating our objectives.
Vertex-featured graphs Our main objects of study are graphs with vertex features, represented as triplets G = ( V , E , X ) G=(V,E,X) G = ( V , E , X ) , where V = { v i } i = 1 n V=\{v_i\}_{i=1}^{n} V = { v i } i = 1 n is the set of vertices, E ⊆ { { v i , v j } ∣ i , j ∈ [ n ] } E\subseteq\{\{v_i,v_j\}\mid i,j\in[n]\} E ⊆ {{ v i , v j } ∣ i , j ∈ [ n ]} are the undirected edges in G G G , and X = [ x 1 , … , x n ] X=[x_1,\ldots,x_n] X = [ x 1 , … , x n ] is a matrix containing the vertex feature vectors x i ∈ Ω x_i\in\Omega x i ∈ Ω , with Ω ⊆ R d \Omega\subseteq\mathbb{R}^{d} Ω ⊆ R d being the feature domain . We denote by G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) the set of all vertex-featured graphs with at most N N N vertices and corresponding features in Ω \Omega Ω . Throughout the paper, we use { } \{\} { } to denote multisets. We note that our results readily extend to graphs with edge features.
Weisfeiler-Leman Graph Isomorphism test Two graphs are isomorphic if they are identical up to relabeling of their nodes. Perhaps surprisingly, the problem of determining whether two given graphs are isomorphic is rather challenging. To date, no known algorithm can solve it in polynomial time [Bab16]. However, there exist various heuristics that provide an incomplete but often adequate method to test whether a given pair of graphs is isomorphic. The most notable example is the Weisfeiler-Leman (WL) graph isomorphism test.
The WL test can be described as assigning to each graph G = ( V , E , X ) G=(V,E,X) G = ( V , E , X ) a feature vector c G T c_G^T c G T according to the formula
c v 0 : = X v , v ∈ V ; for 1 ≤ t ≤ T : c v t : = Combine ( c v t − 1 , Aggregate ( { c u t − 1 ∣ u ∈ N v } ) ) ; c G T : = Readout ( { c v 1 T , … , c v n T } ) , (62) \begin{aligned}
c_v^0 &:= X_v,\quad v\in V;\quad\text{for }1\leq t\leq T: \\
c_v^t &:= \operatorname{Combine}\left(c_v^{t-1},\operatorname{Aggregate}\left(\{c_u^{t-1}\mid u\in\mathcal{N}_v\}\right)\right); \\
c_G^T &:= \operatorname{Readout}\left(\{c_{v_1}^T,\ldots,c_{v_n}^T\}\right), \tag*{(62)}
\end{aligned} c v 0 c v t c G T := X v , v ∈ V ; for 1 ≤ t ≤ T : := Combine ( c v t − 1 , Aggregate ( { c u t − 1 ∣ u ∈ N v } ) ) ; := Readout ( { c v 1 T , … , c v n T } ) , (62)
where Aggregate and Readout are functions that injectively map multisets of vectors in Euclidean space into another Euclidean space, Combine is an injective function from one Euclidean space to another, and N v \mathcal{N}_v N v denotes the neighborhood of the vertex v v v in G G G .
Definition (WL graph equivalence). Two vertex-featured graphs G G G and G ~ \widetilde{G} G are said to be WL-equivalent, denoted by G ∼ W L G ~ G \mathrel{\mathop{\sim}\limits^{\mathrm{WL}}} \widetilde{G} G ∼ WL G , if c G T = c G ~ T c_G^T = c_{\widetilde{G}}^T c G T = c G T for all T ≥ 0 T \ge0 T ≥ 0 . Otherwise, they are said to be WL-separable.
It is a known fact [Gro21; MLM+23] that for G , G ~ ∈ G ≤ N ( R d ) G,\widetilde{G} \in\mathcal{G}_{\leq N}(\mathbb{R}^d) G , G ∈ G ≤ N ( R d ) , if the equality c G T = c G ~ T c_G^T = c_{\widetilde{G}}^T c G T = c G T is satisfied for T = N T=N T = N , then it is satisfied for all T ≥ 0 T \ge0 T ≥ 0 , and thus G ∼ W L G ~ G \mathrel{\mathop{\sim}\limits^{\mathrm{WL}}} \widetilde{G} G ∼ WL G .
While the WL test can separate most pairs of non-isomorphic graphs, there exist examples of non-isomorphic graph pairs that WL cannot separate; see [Zop22].
Message passing neural networks Message Passing Neural Networks (MPNNs) operate on a similar principle to the WL test, but with the purpose of performing predictions on single graphs rather than determining if pairs of them are isomorphic. Their core mechanism is the message-passing procedure, which maintains a hidden feature for each vertex and iteratively updates it as a function of the neighbors’ features. This process is outlined as follows:
Initialization: The hidden feature h v ( 0 ) h_v^{(0)} h v ( 0 ) of each node is initialized by its input feature x v x_v x v .
Message aggregation: Each node v ∈ V v \in V v ∈ V aggregates messages from its neighbors by
m v ( t ) : = Aggregate ( { h u ( t − 1 ) ∣ u ∈ N v } ) (63) m_v^{(t)} := \operatorname{Aggregate}\left(\left\{h_u^{(t-1)} \mid u \in\mathcal{N}_v\right\}\right) \tag*{(63)} m v ( t ) := Aggregate ( { h u ( t − 1 ) ∣ u ∈ N v } ) (63)
Where Aggregate is a multiset-to-vector function.
Update step: Each node updates its own hidden feature according to its aggregated messages and its previous hidden feature, using a vector-to-vector update function:
h v ( t ) : = Update ( m v ( t ) , h v ( t − 1 ) ) , (64) h_v^{(t)} := \operatorname{Update}\left(m_v^{(t)}, h_v^{(t-1)}\right), \tag*{(64)} h v ( t ) := Update ( m v ( t ) , h v ( t − 1 ) ) , (64)
Readout: After T T T iterations of steps 2-3, a graph-level feature h G h_G h G is computed by a multiset-to-vector readout function:
h G : = Readout ( { h v ( T ) ∣ v ∈ V } ) . h_G := \operatorname{Readout}\left(\left\{h_v^{(T)} \mid v \in V\right\}\right). h G := Readout ( { h v ( T ) ∣ v ∈ V } ) .
Numerous MPNNs were proposed in recent years, including GIN [38 ] , GraphSAGE [HYL17], GAT [VCC+18], and GCN [KW17], the main differences between them being the specific choices of the aggregation, update, and readout functions.
An MPNN computes an embedding of a graph G G G to a vector F ( G ) = h G F(G)=h_G F ( G ) = h G . The obtained embedding is often further processed by standard machine-learning tools for vectors, such as multi-layer perceptrons (MLPs), to obtain a final graph prediction. The ability of such a model to approximate functions on graphs is closely related to the separation properties of F F F : if F F F can differentiate between any pair of non-isomorphic graphs, then a model of the form M L P ∘ F \mathrm{MLP} \circ F MLP ∘ F would be able to approximate any function on graphs [CVCB19].
Unfortunately, an MPNN cannot separate any pair of WL-equivalent graphs, even if they are not truly isomorphic [38 MRF+19] . Accordingly, the best we can hope for from an MPNN, in terms of separation, is WL equivalence : for every pair of graphs G , G ′ ∈ G ≤ N ( Ω ) G, G' \in\mathcal{G}_{\leq N}(\Omega) G , G ′ ∈ G ≤ N ( Ω ) , F ( G ) = F ( G ′ ) F(G) = F(G') F ( G ) = F ( G ′ ) if and only if G ∼ W L G ′ G \mathrel{\overset{\mathrm{WL}}{\sim}} G' G ∼ WL G ′ . While MPNNs based on max- or mean-pooling cannot be WL-equivalent [38 ] , it is possible to construct WL-equivalent MPNNs based on sum-pooling, as discussed in [38 MRF+19, ACI+22, 2 BKR24b] . Theoretically, a properly tuned graph model based on a WL-equivalent MPNN should be capable of perfectly solving any binary classification task, provided that no two WL-equivalent graphs have different ground-truth labels. However, this separation does not always manifest in practice. One reason is that WL-equivalent functions may map two input graphs that are far apart in the input space to outputs that are numerically indistinguishable in the output Euclidean space. In fact, Davidson and Dym [DD24] provide an example of graph pairs that are not WL-equivalent, yet are mapped to near-identical outputs by standard sum-based MPNNs. Consequently, these MPNNs fail on binary classification tasks for such graphs.
This paper aims to address this limitation by devising an MPNN whose embeddings preserve distances in the bi-Lipschitz sense. To state our goal formally, we first need to define a notion of distance on the input space of graphs.
WL metric for graphs WL metrics quantify the extent to which two graphs are not WL-equivalent:
Definition 6.2.1 (WL metric). A WL metric on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) is a function ρ : G ≤ N ( Ω ) × G ≤ N ( Ω ) → R ≥ 0 \rho: \mathcal{G}_{\leq N}(\Omega) \times\mathcal{G}_{\leq N}(\Omega) \to\mathbb{R}_{\geq0} ρ : G ≤ N ( Ω ) × G ≤ N ( Ω ) → R ≥ 0 that satisfies the standard requirements for a metric, with the exception that ρ ( G 1 , G 2 ) = 0 \rho(G_1,G_2) = 0 ρ ( G 1 , G 2 ) = 0 if and only if G 1 ∼ W L G 2 G_1 \mathrel{\overset{\mathrm{WL}}{\sim}} G_2 G 1 ∼ WL G 2 .
For convenience, we use the term WL metric, despite the fact that strictly speaking, WL metrics are pseudometrics on G ≤ N ( R d ) \mathcal{G}_{\leq N}(\mathbb{R}^d) G ≤ N ( R d ) .
Tree Mover’s Distance The first WL metric we consider is the Tree Mover’s Distance (TMD), defined in [CJ22]. This metric is based on building computation trees T v ( t ) T_v^{(t)} T v ( t ) , which simulate the WL procedure used to create the node features h v ( t ) h_v^{(t)} h v ( t ) , and calculating distances
TD ( T v ( t ) , T u ( t ) ) (65) \operatorname{TD}(T_v^{(t)}, T_u^{(t)}) \tag*{(65)} TD ( T v ( t ) , T u ( t ) ) (65)
recursively between sub-trees using optimal transport. These node-level distances are zero if, and only if, the features c v ( t ) c_v^{(t)} c v ( t ) and c u ( t ) c_u^{(t)} c u ( t ) , constructed by the WL test, are equal. A graph-level distance TMD ( G , G ′ ) \operatorname{TMD}(G,G') TMD ( G , G ′ ) is obtained by aggregating all node-level distances. For the full definition, see Section C and [CJ22]. Under the assumption that the feature domain Ω \Omega Ω does not contain the zero vector, Chuang and Jegelka [CJ22] proved that TMD ( G , G ′ ) \operatorname{TMD}(G,G') TMD ( G , G ′ ) is a WL-metric.
The second WL metric we consider is the DS metric [Gro21]. Originally, this metric was defined only for featureless graphs of fixed cardinality. In the next section, we extend this metric to the more general case of G ≤ N ( R d ) \mathcal{G}_{\leq N}(\mathbb{R}^{d}) G ≤ N ( R d ) .
Bi-Lipschitzness. Once a WL-metric is defined to measure distances between graphs, one can bound the distortion incurred by a graph embedding with respect to that metric, using the notion of bi-Lipschitzness:
Definition (Bi-Lipschitz embedding). Let ρ \rho ρ be a WL-metric on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) . An embedding E : G ≤ N ( Ω ) → R m E:\mathcal{G}_{\leq N}(\Omega)\to\mathbb{R}^{m} E : G ≤ N ( Ω ) → R m is said to be bi-Lipschitz with respect to ρ \rho ρ on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) if there exist constants 0 < c ≤ C < ∞ 0<c\leq C<\infty 0 < c ≤ C < ∞ such that ∀ G 1 , G 2 ∈ G ≤ N ( Ω ) \forall G_{1},G_{2}\in\mathcal{G}_{\leq N}(\Omega) ∀ G 1 , G 2 ∈ G ≤ N ( Ω ) ,
c ⋅ ρ ( G 1 , G 2 ) ≤ ∥ E ( G 1 ) − E ( G 2 ) ∥ 2 ≤ C ⋅ ρ ( G 1 , G 2 ) . (66) c\cdot\rho(G_{1},G_{2})\leq\lVert E(G_{1})-E(G_{2})\rVert_{2}\leq C\cdot\rho(G_{1},G_{2}). \tag*{(66)} c ⋅ ρ ( G 1 , G 2 ) ≤ ∥ E ( G 1 ) − E ( G 2 ) ∥ 2 ≤ C ⋅ ρ ( G 1 , G 2 ) . (66) If E E E only satisfies the left- or right-hand side of Equation (66) , it is said to be lower- or upper-Lipschitz respectively.
Bi-Lipschitzness ensures that the embedding maps the input space G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) into the output Euclidean space with bounded distortion, with the ratio C c \frac{C}{c} c C serving as an upper bound on the distortion, akin to the condition number of a matrix.
Main Contributions
In this section, we discuss our main contributions. We begin by defining our generalized DS metric for vertex-featured graphs. We then discuss our proposed MPNN and show that it is bi-Lipschitz with respect to the DS and TMD metrics.
The DS metric. The DS metric originates from a relaxation of the graph isomorphism problem . Two graphs G G G and G ~ \widetilde{G} G , each with n n n vertices, and corresponding adjacency matrices A A A , A ~ \widetilde{A} A are isomorphic if and only if there exists a permutation matrix P P P such that A P = P A ~ AP=P\widetilde{A} A P = P A . Since checking whether graphs are isomorphic is intractable, an approximate solution can be sought by considering the equation A S = S A ~ AS=S\widetilde{A} A S = S A with S ∈ D n S\in D_{n} S ∈ D n , the collection of n × n n\times n n × n doubly stochastic matrices , which is the convex hull of all permutation matrices. Remarkably, this equation admits a solution S ∈ D n S\in D_{n} S ∈ D n if and only if the graphs G G G and G ~ \widetilde{G} G are WL-equivalent [SU13]. Accordingly, a WL-metric between featureless graphs with the same number of vertices n n n can be defined by the minimization problem
ρ D S ( G , G ~ ) = min S ∈ D n ∥ A S − S A ~ ∥ , (67) \rho_{\mathrm{DS}}(G,\widetilde{G})=\min_{S\in D_{n}}\lVert AS-S\widetilde{A}\rVert, \tag*{(67)} ρ DS ( G , G ) = S ∈ D n min ∥ A S − S A ∥ , (67)
where ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ could denote any p p p norm. The fact that this is indeed a pseudometric was established in [BI18]. The optimization problem Equation (67) can be solved by off-the-shelf convex optimization solvers and was considered as a method for finding the correspondence between two graphs in several papers, including [ABK15; LFF+16; Dym18; DML17; BTM18].
The idea of using the DS metric for MPNN stability analysis was introduced in [Gro20] and further discussed by Böker [Bök21]. To apply this idea to our setting, we need to adapt this metric to vertex-featured graphs with varying numbers of vertices. We do this by augmenting it as follows:
ρ D S ( G , G ~ ) = ∣ n − n ~ ∣ + min S ∈ Π ( n , n ~ ) ( ∥ A S − S A ~ ∥ 1 + ∑ i ∈ [ n ] , j ∈ [ n ~ ] S i j ∥ x i − x ~ j ∥ 1 ) (68) \rho_{\mathrm{DS}}(G,\widetilde{G}) = \lvert n-\widetilde{n}\rvert+ \min_{S\in\Pi(n,\widetilde{n})} \left( \lVert AS-S\widetilde{A}\rVert_{1} + \sum_{\substack{i\in[n],j\in[\widetilde{n}]}} S_{ij}\lVert x_i-\widetilde{x}_j\rVert_{1} \right) \tag*{(68)} ρ DS ( G , G ) = ∣ n − n ∣ + S ∈ Π ( n , n ) min ∥ A S − S A ∥ 1 + i ∈ [ n ] , j ∈ [ n ] ∑ S ij ∥ x i − x j ∥ 1 (68)
where n n n and n ~ \widetilde{n} n denote the number of vertices in G G G and G ~ \widetilde{G} G , x i x_i x i and x ~ j \widetilde{x}_j x j denote the vertex features of G G G and G ~ \widetilde{G} G , and Π ( n , n ~ ) \Pi(n,\widetilde{n}) Π ( n , n ) is the set of n × n ~ n\times\widetilde{n} n × n matrices S S S with non-negative entries, whose rows and columns sum to 1 n \frac{1}{n} n 1 and 1 n ~ \frac{1}{\widetilde{n}} n 1 , respectively.
Theorem 6.3.1. (Proof in Section D.1) ρ D S \rho_{\mathrm{DS}} ρ DS is a W L WL W L -equivalent metric on G ≤ N ( R d ) G_{\leq N}(\mathbb{R}^{d}) G ≤ N ( R d ) .
Bi-Lipschitz MPNN We now present our main contribution: a novel MPNN that is not only WL-equivalent, but also bi-Lipschitz, with respect to both ρ D S \rho_{\mathrm{DS}} ρ DS and TMD.
FSW Embedding The core innovation in our MPNN lies in its message aggregation method. Specifically, we employ the Fourier Sliced-Wasserstein (FSW) Embedding —a method for embedding multisets of vectors into Euclidean space, proposed by Amir and Dym [AD24], where it was also shown to be bi-Lipschitz. This property makes it plausible, a priori , that an MPNN based on FSW aggregation will be bi-Lipschitz for graphs . In this work, we formally establish that this is indeed the case. We begin by describing the FSW embedding and then introduce our FSW-GNN architecture.
The FSW embedding maps an input multiset X = { x 1 , … , x n } X=\{x_1,\ldots,x_n\} X = { x 1 , … , x n } , where x 1 , … , x n ∈ R d x_1,\ldots,x_n\in\mathbb{R}^{d} x 1 , … , x n ∈ R d , to an output vector z = ( z 1 , … , z m ) ∈ R m z=(z_1,\ldots,z_m)\in\mathbb{R}^{m} z = ( z 1 , … , z m ) ∈ R m . It is denoted by
z = E F S W ( X ; ( v k , ξ k ) k = 1 m − 1 ) , X = { x 1 , … , x n } . z=E_{\mathrm{FSW}}\left(X;(v_k,\xi_k)_{k=1}^{m-1}\right), \quad X=\{x_1,\ldots,x_n\}. z = E FSW ( X ; ( v k , ξ k ) k = 1 m − 1 ) , X = { x 1 , … , x n } .
In addition to the input multiset X X X , it depends on parameters ( v k , ξ k ) k = 1 m − 1 (v_k,\xi_k)_{k=1}^{m-1} ( v k , ξ k ) k = 1 m − 1 , where each v k ∈ S d − 1 v_k\in S^{d-1} v k ∈ S d − 1 represents a direction vector and ξ k ∈ R \xi_k\in\mathbb{R} ξ k ∈ R represents a frequency. The embedding is computed in three steps: First, the direction v k v_k v k is used to project the original multiset of vectors to a multiset of scalars { ⟨ v k , x i ⟩ } i = 1 n \{\langle v_k,x_i\rangle\}_{i=1}^{n} {⟨ v k , x i ⟩ } i = 1 n , which is then sorted: y k = sort ( { ⟨ v k , x i ⟩ } i = 1 n ) y_k=\operatorname{sort}(\{\langle v_k,x_i\rangle\}_{i=1}^{n}) y k = sort ({⟨ v k , x i ⟩ } i = 1 n ) . This step is similar to the sort-type embedding used in SortMPNN, and was shown to be bi-Lipschitz on multisets of fixed size [6 ] . However, taking y k y_k y k directly as the embedding leads to an output dimension dependent on the input multiset’s size, thus making the embedding unsuitable for varying-size multisets. The next steps address and resolve this limitation.
In the second step, the vector y k \mathbf{y}_k y k is identified with a step function Q y k : [ 0 , 1 ] → R Q_{\mathbf{y}_k}:[0,1]\to\mathbb{R} Q y k : [ 0 , 1 ] → R , namely the quantile function of the multiset { ⟨ v k , x i ⟩ } i = 1 n \{\langle\mathbf{v}_k,\mathbf{x}_i\rangle\}_{i=1}^n {⟨ v k , x i ⟩ } i = 1 n ; see illustration in Figure 6.2 . Then, in the third step, the cosine transform , a variant of the Fourier transform, is applied to Q y k Q_{\mathbf{y}_k} Q y k , and sampled at the given frequency ξ k \xi_k ξ k , to obtain the final output coordinates z k \mathbf{z}_k z k , k = 1 , … , m − 1 k=1,\ldots,m-1 k = 1 , … , m − 1 . This is summarized by:
Figure 6.2. The quantile function for three different multisets
y k = ( y k 1 , … , y k n ) = sort ( ⟨ v k , x 1 ⟩ , … , ⟨ v k , x n ⟩ ) (69) \mathbf{y}_k=(y_{k1},\ldots,y_{kn})=\operatorname{sort}(\langle\mathbf{v}_k,\mathbf{x}_1\rangle,\ldots,\langle\mathbf{v}_k,\mathbf{x}_n\rangle) \tag*{(69)} y k = ( y k 1 , … , y k n ) = sort (⟨ v k , x 1 ⟩ , … , ⟨ v k , x n ⟩) (69)
Q y k ( t ) = y k i with i such that t ∈ [ i − 1 n , i n ) (70) Q_{\mathbf{y}_k}(t)=y_{ki}\ \text{with }i\text{ such that }t\in\left[\frac{i-1}{n},\frac{i}{n}\right) \tag*{(70)} Q y k ( t ) = y k i with i such that t ∈ [ n i − 1 , n i ) (70)
z k = 2 ( 1 + ξ k ) ∫ 0 1 Q y k ( t ) cos ( 2 π ξ k t ) d t (71) z_k=2(1+\xi_k)\int_0^1 Q_{\mathbf{y}_k}(t)\cos(2\pi\xi_k t)\,\mathrm{d}t \tag*{(71)} z k = 2 ( 1 + ξ k ) ∫ 0 1 Q y k ( t ) cos ( 2 π ξ k t ) d t (71)
Lastly, note that Equations (6.8) to (6.10) treat input multisets as uniform distributions over their elements, and thus are agnostic to the multiset size by design. To address this and ensure that multisets of different sizes but identical element-proportions are mapped to distinct outputs, the last output coordinate z m z_m z m is set to the cardinality ∣ X ∣ |\mathbf{X}| ∣ X ∣ of the input multiset X \mathbf{X} X . Further details appear in [AD24, Appendix A.1].
Note that the integral in Equation (6.10) has a closed-form solution, and the whole embedding can be computed with a complexity of O ( m n d + m n log n ) O(mnd+mn\log n) O ( mn d + mn log n ) , similarly to simple aggregation functions such as sum pooling. Furthermore, the embedding parameters and output dimension do not depend on n n n , making this method suitable for multisets of different sizes.
FSW-GNN The FSW-GNN model processes input graphs G = ( V , E , X ) \mathcal{G}=(V,E,X) G = ( V , E , X ) by T T T message-passing iterations:
h v ( 0 ) : = x v , q v ( t ) : = E F S W ( t ) ( { h u ( t − 1 ) ∣ u ∈ N v } ) , 1 ≤ t ≤ T , h v ( t ) : = Φ ( t ) ( [ h v ( t − 1 ) ; q v ( t ) ] ) , (72) \begin{aligned}
\mathbf{h}_v^{(0)}&:=\mathbf{x}_v,\\
\mathbf{q}_v^{(t)}&:=E_{\mathrm{FSW}}^{(t)}\left(\left\{\mathbf{h}_u^{(t-1)}\mid u\in N_v\right\}\right),\quad1\le t\le T,\\
\mathbf{h}_v^{(t)}&:=\Phi^{(t)}\left(\left[\mathbf{h}_v^{(t-1)};\mathbf{q}_v^{(t)}\right]\right), \tag*{(72)}
\end{aligned} h v ( 0 ) q v ( t ) h v ( t ) := x v , := E FSW ( t ) ( { h u ( t − 1 ) ∣ u ∈ N v } ) , 1 ≤ t ≤ T , := Φ ( t ) ( [ h v ( t − 1 ) ; q v ( t ) ] ) , (72)
where the functions E F S W ( t ) E_{\mathrm{FSW}}^{(t)} E FSW ( t ) are all instances of the FSW embedding, Φ ( t ) \Phi^{(t)} Φ ( t ) are MLPs, and [ x ; y ] [\mathbf{x};\mathbf{y}] [ x ; y ] denotes column-wise concatenation of column vectors x \mathbf{x} x and y \mathbf{y} y . Finally, a graph-level output is computed by:
h G : = Ψ ∘ E F S W G l o b ( { h v ( T ) ∣ v ∈ V } ) , (73) \mathbf{h}_{G}:=\Psi\circ E_{\mathrm{FSW}}^{\mathrm{Glob}}\left(\left\{\mathbf{h}_v^{(T)}\mid v\in V\right\}\right), \tag*{(73)} h G := Ψ ∘ E FSW Glob ( { h v ( T ) ∣ v ∈ V } ) , (73)
where, again, E F S W G l o b E_{\mathrm{FSW}}^{\mathrm{Glob}} E FSW Glob is an FSW embedding, and Ψ \Psi Ψ an MLP. The following theorem shows that, with the appropriate choice of MLP sizes and number of iterations T T T , our proposed architecture is WL equivalent:
Theorem 6.3.2. (Proof in Section D.2) Consider the FSW-GNN architecture for input graphs in G ≤ N ( R d ) \mathcal{G}_{\leq N}(\mathbb{R}^{d}) G ≤ N ( R d ) , with T = N T=N T = N iterations, where Φ ( t ) \Phi^{(t)} Φ ( t ) , Ψ \Psi Ψ are just linear functions, and all features (except for input features) are of dimension m ≥ 2 N d + 2 m \geq2Nd+2 m ≥ 2 N d + 2 . Then for almost every choice of model parameters, the graph embedding defined by the architecture is WL equivalent.
The proof of Theorem 6.3.2 is based on the theory of σ \sigma σ -subanalytic functions and the Finite Witness Theorem , introduced in [2 ] .
It is worth noting that the output dimension m m m required in practice is typically considerably lower than the one required in Theorem 6.3.2. This can be explained intuitively by the following fact: if all input graphs originate from a subset of G ≤ N ( R d ) \mathcal{G}_{\leq N}(\mathbb{R}^{d}) G ≤ N ( R d ) with intrinsic dimension D D D that is lower than the ambient dimension n ⋅ d n \cdot d n ⋅ d , then it can be shown that m = 2 D + 2 m=2D+2 m = 2 D + 2 suffices for WL-equivalence.
From separation to bi-Lipschitzness In general, WL-equivalence does not imply bi-Lipschitzness. As mentioned above, sum-based MPNN can be injective but are never bi-Lipschitz. In contrast, we shall now prove that for FSW-GNN, WL-equivalence does imply bi-Lipschitzness, under the assumption that the feature domain Ω \Omega Ω is compact:
Theorem 6.3.3. (Proof in Section D.3) Let Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d be compact. Under the assumptions of Theorem 6.3.2, the FSW-GNN is bi-Lipschitz with respect to ρ D S \rho_{\mathrm{DS}} ρ DS on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) . If, additionally, Ω \Omega Ω does not contain 0 0 0 , then the FSW-GNN is bi-Lipschitz with respect to TMD on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) .
We now give a high-level explanation of the proof idea. We rely on the following facts: (1) the output of FSW-GNN for an input graph G = ( V , E , X ) G=(V,E,X) G = ( V , E , X ) is piecewise-linear with respect to the vertex-feature matrix X X X . This follows from properties of the FSW embedding functions E F S W ( t ) E_{\mathrm{FSW}}^{(t)} E FSW ( t ) and E F S W G l o b E_{\mathrm{FSW}}^{\mathrm{Glob}} E FSW Glob used in Equations (72) and (73) . (2) both metrics ρ D S \rho_{\mathrm{DS}} ρ DS and TMD can be transformed, with bounded distortion, into metrics that are piecewise-linear, by choosing all the vector norms they employ to be the ℓ 1 \ell_1 ℓ 1 norm. The claim then follows from these observations and the following theorem, which shows that two functions that are piecewise linear and have the same zero set, are bi-Lipschitz with respect to one another:
Lemma 6.3.4. (Proof in Section D.3) Let f , g : M → R ≥ 0 f,g:M\to\mathbb{R}_{\geq0} f , g : M → R ≥ 0 be nonnegative piecewise-linear functions defined on a compact polygon M ⊂ R d M\subset\mathbb{R}^{d} M ⊂ R d . Suppose that for all x ∈ M x\in M x ∈ M , f ( x ) = 0 f(x)=0 f ( x ) = 0 if and only if g ( x ) = 0 g(x)=0 g ( x ) = 0 . Then there exist real constants c , C > 0 c,C>0 c , C > 0 such that
c ⋅ g ( x ) ≤ f ( x ) ≤ C ⋅ g ( x ) , ∀ x ∈ M . (74) c\cdot g(x)\leq f(x)\leq C\cdot g(x), \quad\forall x\in M. \tag*{(74)} c ⋅ g ( x ) ≤ f ( x ) ≤ C ⋅ g ( x ) , ∀ x ∈ M . (74)
We note that the assumption of a compact domain is essential; see Remark 6 in [BTW24]. While Theorem 6.3.4 is rather intuitive, we are not aware of it appearing previously in the literature. This lemma easily implies1 previous bi-Lipschitzness results in the literature, such as Theorem 1.4 in [BT23], Theorem 1 in [BTW24], and Theorem 3.10 in [6 ] . We believe the lemma has the potential to serve as a valuable tool for proving bi-Lipschitzness results in additional domains in the future. To illustrate this, in Appendix F we show how the lemma can be used to construct a bi-Lipschitz subgraph aggregation MPNN .
To complement Theorem 6.3.3, we now present an analogous result for node-level tasks: we show that the node features computed by FSW-GNN are bi-Lipschitz with respect to the Tree Distance discussed in Equation (65) .
Theorem 6.3.5. (Proof in Section D.3) Suppose that Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d is a compact set that does not contain zero. Under the assumptions of Theorem 6.3.2, the node features computed by FSW-GNN are bi-Lipschitz with respect to the Tree Distance metric on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) .
Oversmoothing, Oversquashing, and Bi-Lipschitz MPNN
Training deep MPNNs is one of the core challenges in graph neural networks [MDM+24]. The difficulty is often attributed to oversmoothing [RBM23] or oversquashing [AY21]. Both phenomena can be regarded as a form of metric distortion induced by the MPNN as the number of iterations grows. Oversmoothing is the phenomenon where, for large t t t ,
∥ x v ( t ) − x u ( t ) ∥ ≈ 0 , ∀ v , u ∈ V . (75) \left\|x_v^{(t)} - x_u^{(t)}\right\| \approx0,\quad\forall v,u \in V. \tag*{(75)} x v ( t ) − x u ( t ) ≈ 0 , ∀ v , u ∈ V . (75)
Oversquashing, as defined in [DGB+23], refers the phenomenon where a non-negligible change in an input node feature x v ( 0 ) x_v^{(0)} x v ( 0 ) to a new value x ^ v ( 0 ) \hat{x}_v^{(0)} x ^ v ( 0 ) results in only a negligible change in features of far-away nodes, namely,
∥ x u ( t ) − x ^ u ( t ) ∥ ≪ ∥ x v ( 0 ) − x ^ v ( 0 ) ∥ , for u far from v . (76) \left\|x_u^{(t)} - \hat{x}_u^{(t)}\right\| \ll\left\|x_v^{(0)} - \hat{x}_v^{(0)}\right\|,\quad\text{for }u\text{ far from }v. \tag*{(76)} x u ( t ) − x ^ u ( t ) ≪ x v ( 0 ) − x ^ v ( 0 ) , for u far from v . (76)
In other words, information fails to propagate adequately between distant nodes through the message-passing process.
Accordingly, an MPNN that perfectly preserves distances between node features would provably not encounter oversmoothing or oversquashing. We note that FSW-GNN is not in this position: while it is bi-Lipschitz and thus has bounded distortion, we cannot rule out the possibility that its distortion grows with t t t . Nonetheless, we conjecture that FSW-GNN should perform much better than standard MPNNs on long range tasks. This is because, as shown in [DD24], standard MPNNs are only lower Hölder in expectation, namely, the inequality
∥ x v ( t ) − x u ( t ) ∥ ≥ C t [ TD ( T v ( t ) , T u ( t ) ) ] α t \lVert x_v^{(t)} - x_u^{(t)} \rVert\ge C_t \left[\operatorname{TD}\left(\mathcal{T}_v^{(t)},\mathcal{T}_u^{(t)}\right)\right]^{\alpha_t} ∥ x v ( t ) − x u ( t ) ∥ ≥ C t [ TD ( T v ( t ) , T u ( t ) ) ] α t
holds only in expectation, with constants 1 C t \frac{1}{C_t} C t 1 and exponents α t \alpha_t α t that grow with t t t . In contrast, for FSW-GNN we have α t = 1 \alpha_t = 1 α t = 1 for all t t t , and only 1 C t \frac{1}{C_t} C t 1 may grow with t t t . We provide empirical evidence for our conjecture in § 10.5 , where we show that our bi-Lipschitz MPNN outperforms standard MPNNs on long-range tasks.
Numerical Experiments
We compare the performance of FSW-GNN with standard MPNNs and Sort-MPNN on both real-world benchmarks and synthetic long-range tasks.
Empirical distortion evaluation First, to assess the metric distortion induced by different MPNNs, we compared the distances induced by each MPNN vs. the TMD and DS metric on a particularly challenging set of graph pairs; see Section E for details.
Empirical estimates C ^ , c ^ \widehat{C},\widehat{c} C , c of the constants C , c C,c C , c of Equation (66) were computed, and the distortion estimate was taken as the ratio C ^ / c ^ \widehat{C}/\widehat{c} C / c . The results appear in Figure 6.4 . Similar results were obtained for the TMD; see Figure 6.6 in the appendix. As seen in the figures, our method yields considerably lower distortion than all competitors, which aligns with our theoretical guarantees.
Figure 6.4. Empirical distortion evaluation with respect to the Doubly-Stochastic (DS) metric
Figure 6.6. Empirical distortion evaluation with respect to the Tree Mover’s Distance (TMD)
Transductive Learning Next, we compare FSW-GNN with GCN, GAT and Sort-MPNN for nine transductive learning tasks taken from [PWC+20]. As shown in Table 6.1 , FSW-GNN outperforms the competition in six out of nine tasks. SortMPNN is a clear winner in two of the other graphs, both of which have relatively large average degrees (see Table 6.4 ). In addition, we include the state-of-the-art results known to us for each dataset: For Cora [IFSL20a]; for Citeseer [IFSL20b]; for PubMed [IFSL20c]; for Chameleon [RCD+24]; for Squirrel [KC23]; for Actor [HZWL24a]; for Cornell [EHT24]; for Texas [LHL+22]; for Wisconsin [HZWL24b]. These results are typically achieved by models that are strictly more powerful and computationally expensive than any MPNN.
Table 6.1. Node classification accuracy (%) across homophilic and heterophilic graph benchmarks. The best and second-best results among the compared models are highlighted in bold and underlined, respectively. The SOTA row reports the best previously published result.
Table 6.4. Graph statistics for transductive learning and MUTAG.
Graph classification and regression In the appendix, we include results on standard graph classification and regression tasks: the peptides-func and peptides-struct datasets of the LRGB benchmark [DRG+22], and the MUTAG and Protein datasets [MKB+20]. These results appear in Table 6.2 and Table 6.3 .
Table 6.2. Performance of different models on the LRGB results.
Table 6.3. Performance of different models on the MUTAG and Protein datasets.
Long range tasks Next, we consider several synthetic long-range tasks proposed in the oversquashing literature [AY21; DGB+23]. In these problems, one can control the problem radius r r r , which is the number of MPNN iterations needed to solve the problem.
We first consider the NeighborsMatch problem from [AY21]. In Figure 6.1 , we compare the performance of FSW-GNN with standard MPNN on the NeighborsMatch problem with 2 ≤ r ≤ 8 2 \le r \le8 2 ≤ r ≤ 8 . Our FSW-GNN achieves perfect accuracy for all values of r r r , while Sort-MPNN fails at r = 8 r = 8 r = 8 and the other competing methods falter for r ≥ 6 r \ge6 r ≥ 6 .
Next, we consider the ‘graph transfer’ tasks [DGB+23], in which a ‘source’ node feature is propagated to a ‘target’ node that is r r r steps away. All other nodes have ‘blank’ features. Here we used three graph topologies proposed by [DGB+23]: clique, ring, and crossring, with problem radii r r r varying from 2 to 15.
As shown in Figure 6.3 , FSW-GNN is the only method to achieve 100% accuracy across all three graphs and radii. Other models start failing at much smaller r r r , which depends on the graph topology. We do find that our model’s performance deteriorates when r > 20 r > 20 r > 20 .
Figure 6.3. Performance comparison of MPNN models across the CliquePath, Ring, and CrossRing graph transfer tasks as presented in [DGB+23].
Finally, we evaluate the relation between our empirical results and oversmoothing on the ring experiment, using the Mean Average Distance (MAD) [RBM23]. As seen in Figure 6.5 , all methods except for FSW-GNN and GIN exhibit oversmoothing starting from some r r r .
Figure 6.5. MAD Energy vs. Number of MPNN layers for various models on the Ring long-range task
We note that long-range issues can be alleviated by graph rewiring methods, which effectively reduce problem radii [GDBD23], or by adding global information using spectral filters [GKHG24] or graph transformers. A key distinction of FSW-GNN is that it performs well without modifying the original graph topology. Thus, this work directly addresses the problem of relaying long-range messages, rather than circumventing it by architectural modifications.
Conclusion
In this paper, we introduced FSW-GNN, the first bi-Lipschitz MPNN. Empirically, we found that FSW-GNN is highly effective, particularly for long-range problems, and appears to mitigate oversmoothing and oversquashing due to its inherent ability to preserve graph metrics.
A slight drawback of FSW-GNN is that its runtime is somewhat higher than that of standard MPNNs; see Table 6.6 . An interesting direction for future work is to obtain quantitative estimates of its bi-Lipschitz constants.
Table 6.6. Comparison of Average testing time in seconds in evaluation mode for different models on MUTAG.
Appendix
Statistics on our benchmarks
We present some statistics of the learning problems considered in the main text. In Table 6.4 , we present the statistics for each transductive dataset, and for the MUTAG dataset, including the number of nodes, edges, features, classes, average degree, and density, measuring the number of edges divided by the number of maximal edges. As we can see in the table, these datasets are very sparse.
Table 6.5 shows the same statistics for the Peptides tasks from the LRGB dataset.
Table 6.5. Graph statistics for Peptides datasets.
Implementation details
For the Transductive learning and the LRGB dataset, we looked for each learning rate in the set 1 e − 3 , 5 e − 3 1e^{-3}, 5e^{-3} 1 e − 3 , 5 e − 3 and weight decay in the set 0.0, 1 e − 5 1e^{-5} 1 e − 5 and learning rate factor of 0.7. For each such configuration we stop according to the validation and take the best configuration of the validation set. For all Over-squashing tasks we took a learning rate of 0.001 and weight decay 0, and learning rate factor of 0.7. In all datasets we took the original dataset splits to train, test and validation sets.
Relation to Tree Mover’s Distance
In this appendix we give a full description of the Tree Mover’s Distance as defined in Chuang and Jegelka [CJ22].
We first review Wasserstein distances. Recall that if ( X , d ) (X, d) ( X , d ) is a metric space, Ω ⊆ X \Omega\subseteq X Ω ⊆ X is a subset, then the Wasserstein distance can be defined on the space of multisets consisting of n n n elements in Ω \Omega Ω via
W d ( x 1 , … , x n , y 1 , … , y n ) = min τ ∈ S n ∑ j = 1 n d ( x j , y τ ( j ) ) W_d(x_1,\ldots,x_n,y_1,\ldots,y_n)=\min_{\tau\in S_n}\sum_{j=1}^{n}d(x_j,y_{\tau(j)}) W d ( x 1 , … , x n , y 1 , … , y n ) = τ ∈ S n min j = 1 ∑ n d ( x j , y τ ( j ) )
For multisets of different size, the authors of [CJ22] used an augmentation map, which, for a fixed parameter n n n , augments multisets of size r ≤ n r\le n r ≤ n by padding with n − r n-r n − r instances of an element z z z in X ∖ Ω \mathcal{X}\setminus\Omega X ∖ Ω , namely
Γ z ( x 1 , … , x r ) = ( x 1 , … , x r , x r + 1 = z , … , x n = z ) \Gamma^z(x_1,\ldots,x_r)=(x_1,\ldots,x_r,x_{r+1}=z,\ldots,x_n=z) Γ z ( x 1 , … , x r ) = ( x 1 , … , x r , x r + 1 = z , … , x n = z )
and the augmented distance on multi-sets of size up to n n n is defined by
W d ( X , X ^ ) = W d 0 ( Γ z ( X ) , Γ z ( X ^ ) ) . W_d(X,\widehat{X})=W_d^0\left(\Gamma^z(X),\Gamma^z(\widehat{X})\right). W d ( X , X ) = W d 0 ( Γ z ( X ) , Γ z ( X ) ) .
We now return to define the TMD. We consider the space of graphs G ≤ N ( Ω ) \mathcal{G}_{\le N}(\Omega) G ≤ N ( Ω ) , consisting of graphs with ≤ N \le N ≤ N nodes, with node features coming from a compact domain Ω ⊆ R d \Omega\subseteq\mathbb{R}^d Ω ⊆ R d such that 0 ∉ Ω 0\notin\Omega 0 ∈ / Ω . The TMD is defined using the notion of computation trees:
Definition C.1. (Computation Trees). Given a graph G = ( V , E , X ) \mathcal{G}=(V,E,X) G = ( V , E , X ) with node features { x v } v ∈ V \{x_v\}_{v\in V} { x v } v ∈ V , let T v 1 T_v^1 T v 1 be the rooted tree with a single node v v v , which is also the root of the tree, and node features x v x_v x v . For t ∈ N t\in\mathbb{N} t ∈ N let T v t T_v^t T v t be the depth-t t t computation tree of node v v v constructed by connecting the neighbors of the leaf nodes of T v t − 1 T_v^{t-1} T v t − 1 to the tree. Each node is assigned the same node feature it had in the original graph G \mathcal{G} G . The multiset of depth-t t t computation trees defined by G \mathcal{G} G is denoted by T G t : = { T v t } v ∈ V T_{\mathcal{G}}^t:=\{T_v^t\}_{v\in V} T G t := { T v t } v ∈ V . Additionally, for a tree T T T with root r r r , we denote by T r T_r T r the multiset of subtrees that root at the descendants of r r r .
Definition C.2. (Blank Tree). A blank tree T ‾ 0 \overline{T}_0 T 0 is a tree (graph) that contains a single node and no edge, where the node feature is the zero vector 0 0 0 .
Recall that by assumption, all node features will come from the compact set Ω \Omega Ω , and 0 ∉ Ω 0\notin\Omega 0 ∈ / Ω . Therefore, the blank tree is not included in the space of trees with features in Ω \Omega Ω , and can be used for augmentation.
We can now define the tree distance:
Definition C.3. (Tree Distance).2 The distance between two trees T a , T b T_a,T_b T a , T b with features from Ω \Omega Ω and 0 ∉ Ω 0\notin\Omega 0 ∈ / Ω , is defined recursively as
TD ( T a , T b ) : = { ∥ x r a − x r b ∥ 1 + W TD T ‾ 0 ( T r a , T r b ) if t > 1 , ∥ x r a − x r b ∥ 1 otherwise . \operatorname{TD}(T_a,T_b):=
\begin{cases}
\lVert x_{r_a}-x_{r_b}\rVert_1+W_{\operatorname{TD}}^{\overline{T}_0}(T_{r_a},T_{r_b}) & \text{if }t>1,\\
\lVert x_{r_a}-x_{r_b}\rVert_1 & \text{otherwise}.
\end{cases} TD ( T a , T b ) := { ∥ x r a − x r b ∥ 1 + W TD T 0 ( T r a , T r b ) ∥ x r a − x r b ∥ 1 if t > 1 , otherwise .
where t t t denotes the maximal depth of the trees T a \mathbf{T}_a T a and T b \mathbf{T}_b T b . Here W T D T ‾ 0 W_{\mathrm{TD}}^{\overline{\mathbf{T}}_0} W TD T 0 denotes the Wasserstein metric obtained from the metric T D \mathrm{TD} TD on the space of trees of smaller depth, with augmentation by blank trees T ‾ 0 \overline{\mathbf{T}}_0 T 0 .
Definition C.4. (Tree Mover’s Distance). Given two graphs, G a , G b G_a,G_b G a , G b and w , t ≥ 0 w,t \ge0 w , t ≥ 0 , the tree mover’s distance is defined by
T M D t ( G a , G b ) = W T D T ‾ 0 ( T G a t , T G b t ) , \mathrm{TMD}^{t}(G_a,G_b) = W_{\mathrm{TD}}^{\overline{\mathbf{T}}_0}(\mathcal{T}_{G_a}^{t},\mathcal{T}_{G_b}^{t}), TMD t ( G a , G b ) = W TD T 0 ( T G a t , T G b t ) , where T G a t \mathcal{T}_{G_a}^{t} T G a t and T G b t \mathcal{T}_{G_b}^{t} T G b t denote the multiset of all depth t t t computational trees arising from the graphs G a G_a G a and G b G_b G b , respectively. Chuang and Jegelka [CJ22] proved that T M D t ( G a , G b ) \mathrm{TMD}^{t}(G_a,G_b) TMD t ( G a , G b ) is a pseudo-metric that fails to distinguish only graphs that cannot be separated by t + 1 t+1 t + 1 iterations of the WL test. Thus, assuming that 0 ∉ Ω 0 \notin\Omega 0 ∈ / Ω , T M D t ( G a , G b ) \mathrm{TMD}^{t}(G_a,G_b) TMD t ( G a , G b ) is WL equivalent on G ≤ N ( Ω ) \mathcal{G}_{\le N}(\Omega) G ≤ N ( Ω ) whenever t + 1 ≥ N t+1 \ge N t + 1 ≥ N . In addition, it is easy to see from the definition of TMD that it satisfies the following properties:
T M D t ( ( A , α ⋅ X ) , ( B , α ⋅ Y ) ) = α ⋅ T M D t ( X , Y ) \mathrm{TMD}^{t}((\mathbf{A},\alpha\cdot\mathbf{X}),(\mathbf{B},\alpha\cdot\mathbf{Y})) = \alpha\cdot\mathrm{TMD}^{t}(\mathbf{X},\mathbf{Y}) TMD t (( A , α ⋅ X ) , ( B , α ⋅ Y )) = α ⋅ TMD t ( X , Y ) for any α ≥ 0 \alpha\ge0 α ≥ 0 .
For fixed A , B \mathbf{A},\mathbf{B} A , B , the TMD metric is piecewise linear in ( X , Y ) (\mathbf{X},\mathbf{Y}) ( X , Y ) .
These properties will be used to show that under the above assumptions, the embedding computed by FSW-GNN is bi-Lipschitz with respect to TMD.
Proofs
DS metric
Here we prove Theorem 6.3.1
Theorem 6.3.1. ρ D S \rho_{\mathrm{DS}} ρ DS is a WL-equivalent metric on G ≤ N ( R d ) \mathcal{G}_{\le N}(\mathbb{R}^{d}) G ≤ N ( R d ) .
Proof. ρ D S \rho_{\mathrm{DS}} ρ DS is symmetric, since if a pair’s minimum is obtained at S S S then the exact value is obtained for the opposite pair with S T S^{\mathsf{T}} S T and vice-versa.
To prove the triangle inequality, let ( V 1 , A , X ) , ( V 2 , B , Y ) , ( V 3 , C , Z ) (V_1,\mathbf{A},\mathbf{X}),(V_2,\mathbf{B},\mathbf{Y}),(V_3,\mathbf{C},\mathbf{Z}) ( V 1 , A , X ) , ( V 2 , B , Y ) , ( V 3 , C , Z ) be three arbitrary graphs with ∣ V i ∣ = n i \lvert V_i\rvert= n_i ∣ V i ∣ = n i , i = 1 , 2 , 3 i=1,2,3 i = 1 , 2 , 3 . Then the following holds:
α : = ρ D S ( A , B ) = ∣ n 1 − n 2 ∣ + min S ∈ Π ( n 1 , n 2 ) ∥ A S − S B ∥ 1 + ∑ i , j S i , j ∥ X i − Y j ∥ 1 \alpha:= \rho_{\mathrm{DS}}(\mathbf{A},\mathbf{B}) = \lvert n_1-n_2\rvert+ \min_{S\in\Pi(n_1,n_2)}\lVert\mathbf{A}S-S\mathbf{B}\rVert_1 + \sum_{i,j}S_{i,j}\lVert\mathbf{X}_i-\mathbf{Y}_j\rVert_1 α := ρ DS ( A , B ) = ∣ n 1 − n 2 ∣ + S ∈ Π ( n 1 , n 2 ) min ∥ A S − S B ∥ 1 + i , j ∑ S i , j ∥ X i − Y j ∥ 1 β : = ρ D S ( B , C ) = ∣ n 2 − n 3 ∣ + min S ∈ Π ( n 2 , n 3 ) ∥ B S − S C ∥ 1 + ∑ i , j S i , j ∥ Y i − Z j ∥ 1 \beta:= \rho_{\mathrm{DS}}(\mathbf{B},\mathbf{C}) = \lvert n_2-n_3\rvert+ \min_{S\in\Pi(n_2,n_3)}\lVert\mathbf{B}S-S\mathbf{C}\rVert_1 + \sum_{i,j}S_{i,j}\lVert\mathbf{Y}_i-\mathbf{Z}_j\rVert_1 β := ρ DS ( B , C ) = ∣ n 2 − n 3 ∣ + S ∈ Π ( n 2 , n 3 ) min ∥ B S − S C ∥ 1 + i , j ∑ S i , j ∥ Y i − Z j ∥ 1 γ : = ρ D S ( A , C ) = ∣ n 1 − n 3 ∣ + min S ∈ Π ( n 1 , n 3 ) ∥ A S − S C ∥ 1 + ∑ i , j S i , j ∥ X i − Z j ∥ 1 \gamma:= \rho_{\mathrm{DS}}(\mathbf{A},\mathbf{C}) = \lvert n_1-n_3\rvert+ \min_{S\in\Pi(n_1,n_3)}\lVert\mathbf{A}S-S\mathbf{C}\rVert_1 + \sum_{i,j}S_{i,j}\lVert\mathbf{X}_i-\mathbf{Z}_j\rVert_1 γ := ρ DS ( A , C ) = ∣ n 1 − n 3 ∣ + S ∈ Π ( n 1 , n 3 ) min ∥ A S − S C ∥ 1 + i , j ∑ S i , j ∥ X i − Z j ∥ 1 :::`
We want to prove γ ≤ β + α \gamma\le\beta+ \alpha γ ≤ β + α . Let S 1 , S 2 S^1, S^2 S 1 , S 2 be the minimizers of the first two equations. Define S 3 = n 2 ⋅ S 1 ⋅ S 2 S^3 = n_2 \cdot S^1 \cdot S^2 S 3 = n 2 ⋅ S 1 ⋅ S 2 and note that S 3 ∈ Π ( n 1 , n 3 ) S^3 \in\Pi(n_1,n_3) S 3 ∈ Π ( n 1 , n 3 ) because, using the notation 1 d \mathbf{1}_d 1 d for the all one d d d -dimensional vector, we have
S 3 1 n 3 = n 2 ⋅ S 1 ⋅ S 2 1 n 3 = S 1 1 n 2 = 1 n 1 1 n 1 S^3\mathbf{1}_{n_3}=n_2\cdot S^1\cdot S^2\mathbf{1}_{n_3}=S^1\mathbf{1}_{n_2}=\frac{1}{n_1}\mathbf{1}_{n_1} S 3 1 n 3 = n 2 ⋅ S 1 ⋅ S 2 1 n 3 = S 1 1 n 2 = n 1 1 1 n 1
1 n 1 T S 3 = n 2 ⋅ 1 n 1 T S 1 ⋅ S 2 = 1 n 2 T S 2 = 1 n 3 1 n 3 . \mathbf{1}_{n_1}^{T}S^3=n_2\cdot\mathbf{1}_{n_1}^{T}S^1\cdot S^2=\mathbf{1}_{n_2}^{T}S^2=\frac{1}{n_3}\mathbf{1}_{n_3}. 1 n 1 T S 3 = n 2 ⋅ 1 n 1 T S 1 ⋅ S 2 = 1 n 2 T S 2 = n 3 1 1 n 3 .
Next, we prove the following lemma (all matrix norms in this lemma are entrywise ℓ 1 \ell_1 ℓ 1 norms).
Lemma D.1. Let S ∈ Π ( n 1 , n 2 ) S \in\Pi(n_1,n_2) S ∈ Π ( n 1 , n 2 ) , then for all B ∈ R n 2 × b B \in\mathbb{R}^{n_2\times b} B ∈ R n 2 × b and A ∈ R a × n 1 A \in\mathbb{R}^{a\times n_1} A ∈ R a × n 1
∥ S B ∥ 1 ≤ 1 n 2 ∥ B ∥ 1 , ∥ A S ∥ 1 ≤ 1 n 1 ∥ A ∥ 1 \lVert SB\rVert_1 \le\frac{1}{n_2}\lVert B\rVert_1,\qquad\lVert AS\rVert_1 \le\frac{1}{n_1}\lVert A\rVert_1 ∥ S B ∥ 1 ≤ n 2 1 ∥ B ∥ 1 , ∥ A S ∥ 1 ≤ n 1 1 ∥ A ∥ 1
Proof.
∥ S B ∥ 1 = ∑ i = 1 n 1 ∑ j = 1 b ∣ ( S B ) i j ∣ ≤ ∑ i = 1 n 1 ∑ j = 1 b ∑ k = 1 n 2 S i k ∣ B k j ∣ = ∑ j = 1 b ∑ k = 1 n 2 ∣ B k j ∣ ⋅ ∑ i = 1 n 1 S i k = 1 n 2 ∥ B ∥ 1 \begin{aligned}
\lVert SB\rVert_1
&=\sum_{i=1}^{n_1}\sum_{j=1}^{b}\lvert(SB)_{ij}\rvert\\
&\le\sum_{i=1}^{n_1}\sum_{j=1}^{b}\sum_{k=1}^{n_2}S_{ik}\lvert B_{kj}\rvert\\
&=\sum_{j=1}^{b}\sum_{k=1}^{n_2}\lvert B_{kj}\rvert\cdot\sum_{i=1}^{n_1}S_{ik}\\
&=\frac{1}{n_2}\lVert B\rVert_1
\end{aligned} ∥ S B ∥ 1 = i = 1 ∑ n 1 j = 1 ∑ b ∣( S B ) ij ∣ ≤ i = 1 ∑ n 1 j = 1 ∑ b k = 1 ∑ n 2 S ik ∣ B k j ∣ = j = 1 ∑ b k = 1 ∑ n 2 ∣ B k j ∣ ⋅ i = 1 ∑ n 1 S ik = n 2 1 ∥ B ∥ 1
A symmetric argument can be used to show that ∥ A S ∥ 1 ≤ 1 n 1 ∥ A ∥ 1 . □ \lVert AS\rVert_1 \le\frac{1}{n_1}\lVert A\rVert_1. \quad\square ∥ A S ∥ 1 ≤ n 1 1 ∥ A ∥ 1 . □
Given the lemma, we now have
1 n 2 ∥ A S 3 − S 3 C ∥ 1 = ∥ A S 1 S 2 − S 1 S 2 C ∥ 1 = ∥ A S 1 S 2 − S 1 B S 2 + S 1 B S 2 − S 1 S 2 C ∥ 1 ≤ ∥ A S 1 S 2 − S 1 B S 2 ∥ 1 + ∥ S 1 B S 2 − S 1 S 2 C ∥ 1 = ∥ ( A S 1 − S 1 B ) ⋅ S 2 ∥ 1 + ∥ S 1 ⋅ ( B S 2 − S 2 C ) ∥ 1 ≤ 1 n 2 ∥ A S 1 − S 1 B ∥ 1 + 1 n 2 ⋅ ∥ B S 2 − S 2 C ∥ 1 . \begin{aligned}
\frac{1}{n_2}\lVert AS^3-S^3C\rVert_1
&=\lVert AS^1S^2-S^1S^2C\rVert_1\\
&=\lVert AS^1S^2-S^1BS^2+S^1BS^2-S^1S^2C\rVert_1\\
&\le\lVert AS^1S^2-S^1BS^2\rVert_1+\lVert S^1BS^2-S^1S^2C\rVert_1\\
&=\lVert(AS^1-S^1B)\cdot S^2\rVert_1+\lVert S^1\cdot(BS^2-S^2C)\rVert_1\\
&\le\frac{1}{n_2}\lVert AS^1-S^1B\rVert_1+\frac{1}{n_2}\cdot\lVert BS^2-S^2C\rVert_1.
\end{aligned} n 2 1 ∥ A S 3 − S 3 C ∥ 1 = ∥ A S 1 S 2 − S 1 S 2 C ∥ 1 = ∥ A S 1 S 2 − S 1 B S 2 + S 1 B S 2 − S 1 S 2 C ∥ 1 ≤ ∥ A S 1 S 2 − S 1 B S 2 ∥ 1 + ∥ S 1 B S 2 − S 1 S 2 C ∥ 1 = ∥( A S 1 − S 1 B ) ⋅ S 2 ∥ 1 + ∥ S 1 ⋅ ( B S 2 − S 2 C ) ∥ 1 ≤ n 2 1 ∥ A S 1 − S 1 B ∥ 1 + n 2 1 ⋅ ∥ B S 2 − S 2 C ∥ 1 .
Next, we show the second part is also smaller:
1 n 2 ∑ i , j S i , j 3 ⋅ ∥ X i − Z j ∥ 1 = ∑ i , j ∑ k S i , k 1 ⋅ S k , j 2 ⋅ ∥ X i − Z j ∥ 1 ≤ ∑ i , j ∑ k S i , k 1 ⋅ S k , j 2 ⋅ ( ∥ X i − Y k ∥ 1 + ∥ Y k − Z j ∥ 1 ) = ∑ i , j ∑ k S i , k 1 ⋅ S k , j 2 ⋅ ∥ X i − Y k ∥ 1 + ∑ i , j ∑ k S i , k 1 ⋅ S k , j 2 ⋅ ∥ Y k − Z j ∥ 1 . \begin{aligned}
\frac{1}{n_2}\sum_{i,j}S^3_{i,j}\cdot\lVert X_i-Z_j\rVert_1
&=\sum_{i,j}\sum_k S^1_{i,k}\cdot S^2_{k,j}\cdot\lVert X_i-Z_j\rVert_1\\
&\le\sum_{i,j}\sum_k S^1_{i,k}\cdot S^2_{k,j}\cdot\left(\lVert X_i-Y_k\rVert_1+\lVert Y_k-Z_j\rVert_1\right)\\
&=\sum_{i,j}\sum_k S^1_{i,k}\cdot S^2_{k,j}\cdot\lVert X_i-Y_k\rVert_1+\sum_{i,j}\sum_k S^1_{i,k}\cdot S^2_{k,j}\cdot\lVert Y_k-Z_j\rVert_1.
\end{aligned} n 2 1 i , j ∑ S i , j 3 ⋅ ∥ X i − Z j ∥ 1 = i , j ∑ k ∑ S i , k 1 ⋅ S k , j 2 ⋅ ∥ X i − Z j ∥ 1 ≤ i , j ∑ k ∑ S i , k 1 ⋅ S k , j 2 ⋅ ( ∥ X i − Y k ∥ 1 + ∥ Y k − Z j ∥ 1 ) = i , j ∑ k ∑ S i , k 1 ⋅ S k , j 2 ⋅ ∥ X i − Y k ∥ 1 + i , j ∑ k ∑ S i , k 1 ⋅ S k , j 2 ⋅ ∥ Y k − Z j ∥ 1 .
We now open both sums:
∑ i , j ∑ k S i , k 1 ⋅ S k , j 2 ⋅ ∥ X i − Y k ∥ 1 = ∑ i ∑ k S i , k 1 ⋅ ∥ X i − Y k ∥ 1 ⋅ ∑ j S k , j 2 = 1 n 2 ∑ i ∑ k S i , k 1 ⋅ ∥ X i − Y k ∥ 1 . \begin{aligned}
\sum_{i,j}\sum_k S^1_{i,k}\cdot S^2_{k,j}\cdot\lVert X_i-Y_k\rVert_1=\sum_i\sum_k S^1_{i,k}\cdot\lVert X_i-Y_k\rVert_1\cdot\sum_j S^2_{k,j} \\
&=\frac{1}{n_2}\sum_i\sum_k S^1_{i,k}\cdot\lVert X_i-Y_k\rVert_1.
\end{aligned} i , j ∑ k ∑ S i , k 1 ⋅ S k , j 2 ⋅ ∥ X i − Y k ∥ 1 = i ∑ k ∑ S i , k 1 ⋅ ∥ X i − Y k ∥ 1 ⋅ j ∑ S k , j 2 = n 2 1 i ∑ k ∑ S i , k 1 ⋅ ∥ X i − Y k ∥ 1 .
With the same argument, we obtain that
∑ i , j ∑ k S i , k 1 ⋅ S k , j 2 ⋅ ∥ Y k − Z j ∥ 1 = 1 n 2 ∑ k , j S k , j 2 ⋅ ∥ Y k − Z j ∥ 1 . \sum_{i,j}\sum_k S^1_{i,k}\cdot S^2_{k,j}\cdot\lVert Y_k-Z_j\rVert_1=\frac{1}{n_2}\sum_{k,j}S^2_{k,j}\cdot\lVert Y_k-Z_j\rVert_1. i , j ∑ k ∑ S i , k 1 ⋅ S k , j 2 ⋅ ∥ Y k − Z j ∥ 1 = n 2 1 k , j ∑ S k , j 2 ⋅ ∥ Y k − Z j ∥ 1 .
So overall we obtain
γ ≤ ∣ n 1 − n 3 ∣ + ∥ A S 3 − S 3 C ∥ 1 + ∑ i , j S i , j 3 ∥ X i − Z j ∥ 1 ≤ ∣ n 1 − n 2 ∣ + ∥ A S 1 − S 1 B ∥ 1 + ∑ i , j S i , j 1 ∥ X i − Y j ∥ 1 + ∣ n 2 − n 3 ∣ + ∥ B S 2 − S 2 C ∥ 1 + ∑ i , j S i , j 2 ∥ Y i − Z j ∥ 1 = α + β . \begin{aligned}
\gamma&\leq|n_1-n_3|+\lVert AS^3-S^3C\rVert_1+\sum_{i,j}S^3_{i,j}\lVert X_i-Z_j\rVert_1\\
&\leq|n_1-n_2|+\lVert AS^1-S^1B\rVert_1+\sum_{i,j}S^1_{i,j}\lVert X_i-Y_j\rVert_1+|n_2-n_3|\\
&\quad+\lVert BS^2-S^2C\rVert_1+\sum_{i,j}S^2_{i,j}\lVert Y_i-Z_j\rVert_1=\alpha+\beta.
\end{aligned} γ ≤ ∣ n 1 − n 3 ∣ + ∥ A S 3 − S 3 C ∥ 1 + i , j ∑ S i , j 3 ∥ X i − Z j ∥ 1 ≤ ∣ n 1 − n 2 ∣ + ∥ A S 1 − S 1 B ∥ 1 + i , j ∑ S i , j 1 ∥ X i − Y j ∥ 1 + ∣ n 2 − n 3 ∣ + ∥ B S 2 − S 2 C ∥ 1 + i , j ∑ S i , j 2 ∥ Y i − Z j ∥ 1 = α + β .
Now, we show that our metric ρ D S \rho_{\mathrm{DS}} ρ DS is equivalent to WL. Clearly, any pair of graphs with different numbers of vertices are distinguished both by WL and by ρ D S \rho_{\mathrm{DS}} ρ DS . Thus, in the following, we assume that the two graphs have the same number of vertices.
We begin first with some needed definitions.
Partitions Most of our techniques are inspired by [SU13]. Given G = ( A , X ) G=(A,X) G = ( A , X ) , we define a stable partition V = P 1 ∪ P 2 . . . ∪ P k \mathcal{V}=\mathcal{P}_1\cup\mathcal{P}_2...\cup\mathcal{P}_k V = P 1 ∪ P 2 ... ∪ P k if
∀ u , v ∈ P i , X u = X v (77) \forall u,v\in\mathcal{P}_i,\ X_u=X_v \tag*{(77)} ∀ u , v ∈ P i , X u = X v (77)
∀ l ∈ [ k ] , i ∈ [ k ] , u , v ∈ P i , ∣ N u ∩ P l ∣ = ∣ N v ∩ P l ∣ (78) \forall l\in[k],\ i\in[k],\ u,v\in\mathcal{P}_i,\ \lvert N_u\cap\mathcal{P}_l\rvert=\lvert N_v\cap\mathcal{P}_l\rvert \tag*{(78)} ∀ l ∈ [ k ] , i ∈ [ k ] , u , v ∈ P i , ∣ N u ∩ P l ∣ = ∣ N v ∩ P l ∣ (78)
In simple words, two nodes in the same partition must have the same feature and the same number of neighbors with the same feature. Note that the union of all singletons is a valid stable partition. We can characterize a partition as a k k k tuple, and in the i i i 'th place, we put a tuple of the common feature and a vector telling the number of neighbors in other partitions. We say that graphs have the same stable partition if, up to permuting the k k k -tuple indices, we have the same k k k -tuple.
Lemma D.2. The number of colors in 1-WL can’t increase at level n n n . In addition, all nodes with the same color at step n n n make a stable partition.
Proof. The number of colors of 1-WL can only increase as the number of iterations increases. It is also known that, if at some step t t t the number of colors does not increase, then the process is ’stuck’. Accordingly, after at most n n n iterations, the number of different colors will stay the same. Denote by P = ( P 1 , … , P s ) \mathcal{P} = (\mathcal{P}_1, \ldots, \mathcal{P}_s) P = ( P 1 , … , P s ) a partition of the nodes with the same coloring at the n n n -th iteration, and let c ∈ C c \in\mathcal{C} c ∈ C be some color, we claim that nodes in the same partition, have the same number of neighbors with color c c c . As otherwise, those nodes that now have the same coloring will have different coloring in the next n + 1 n+1 n + 1 iteration. But as the number of colors doesn’t decrease (because of the concatenation of the current color), we have at least one more color, a contradiction. So, we found a stable partition. □ \square □
Lemma D.3. The following conditions are equivalent:
G ≅ 1 - W L H \mathcal{G} \cong_{1\text{-}WL} \mathcal{H} G ≅ 1 - W L H
Both graphs have a common stable partition. That is, there is a relabeling of G \mathcal{G} G such that the same partition P 1 , … , P s \mathcal{P}_1, \ldots, \mathcal{P}_s P 1 , … , P s of V V V is a stable partition for both G \mathcal{G} G and H \mathcal{H} H , and the features inside each partition are the same, in both G \mathcal{G} G and H \mathcal{H} H .
Proof. Assume we have the same common partition. Up to renaming the names of the vertices of the nodes in the first graphs, we assume we have the same stable partition with the same parameters. Assume that a common stable partition exists P = P 1 ∪ … ∪ P k \mathcal{P} = \mathcal{P}_1 \cup\ldots\cup\mathcal{P}_k P = P 1 ∪ … ∪ P k . Then, by simple induction on the number of iterations, we will prove that all nodes in the same partition have the same color in both graphs.
Basis Nodes in the same partition have the same initial feature and, thus, have the same color (in both graphs).
Step By the induction hypothesis, all nodes in the same partition iteration T T T have the same color; and they both have the same number of neighbors with the same color, so the aggregation yields the same output. Thus, in iteration T + 1 T+1 T + 1 , those nodes also have the same color. Note that this argument is symmetric to both graphs; thus, the 1-WL test will not distinguish them. On the other hand, if G 1 , G 2 \mathcal{G}_1,\mathcal{G}_2 G 1 , G 2 have the same 1-WL embedding, then we partition the nodes to those classes with the same color at iteration n n n , denoted by P = P 1 ∪ … ∪ P k \mathcal{P} = \mathcal{P}_1 \cup\ldots\cup\mathcal{P}_k P = P 1 ∪ … ∪ P k . We have to show this partition is valid. First, by definition, those nodes u , v ∈ P i u, v \in\mathcal{P}_i u , v ∈ P i have the same node feature (by iteration 1); next, if u , v u, v u , v don’t have the same number of neighbors in P l \mathcal{P}_l P l for some l ∈ [ k ] l \in[k] l ∈ [ k ] , then their color won’t be the same at iteration n + 1 n+1 n + 1 . But, as shown in lemma D.2, the number of distinct colors can’t increase between n n n to n + 1 n+1 n + 1 . Thus, we found a common stable partition. □ \square □
Before proving the following lemma, we revise a definition from [SU13]. We define direct the direct sum of matrices S 1 S_1 S 1 and S 2 S_2 S 2 by:
S 1 ⊕ S 2 = [ S 1 0 0 S 2 ] S_1 \oplus S_2 =
\begin{bmatrix}
S_1 & 0 \\
0 & S_2
\end{bmatrix} S 1 ⊕ S 2 = [ S 1 0 0 S 2 ]
Given S ∈ R n × n S \in\mathbb{R}^{n \times n} S ∈ R n × n , we say S S S is composable if there exists P , Q , S 1 , S 2 P,Q,S_1,S_2 P , Q , S 1 , S 2 such that
S = P ⋅ ( S 1 ⊕ S 2 ) ⋅ Q S = P \cdot(S_1 \oplus S_2) \cdot Q S = P ⋅ ( S 1 ⊕ S 2 ) ⋅ Q
Such that P , Q P,Q P , Q are permutation matrices. By simple induction, we can write M M M as a direct sum of an indecomposable
S = P ⋅ ( S 1 ⊕ S 2 ⊕ ⋯ ⊕ S t ) ⋅ Q S = P \cdot(S_1 \oplus S_2 \oplus\cdots\oplus S_t) \cdot Q S = P ⋅ ( S 1 ⊕ S 2 ⊕ ⋯ ⊕ S t ) ⋅ Q
Lemma D.4. Let S ∈ Π ( n , n ) S \in\Pi(n,n) S ∈ Π ( n , n ) and assume it’s in the form of S = S 1 ⊕ ⋯ ⊕ S k S = S_1 \oplus\cdots\oplus S_k S = S 1 ⊕ ⋯ ⊕ S k such that all blocks are indecomposable and ∑ i , j S i , j ⋅ ∥ X i − Y j ∥ = 0 \sum_{i,j} S_{i,j} \cdot\lVert X_i - Y_j \rVert= 0 ∑ i , j S i , j ⋅ ∥ X i − Y j ∥ = 0 . Denote by i 1 , … , i k + 1 i_1,\ldots,i_{k+1} i 1 , … , i k + 1 the indices that define the start and the end of S 1 , … , S t S_1,\ldots,S_t S 1 , … , S t and by I t : = { i t , … , i t + 1 − 1 } I_t := \{i_t,\ldots,i_{t+1}-1\} I t := { i t , … , i t + 1 − 1 } . Then X i = Y j X_i = Y_j X i = Y j , ∀ t ∈ [ k ] , ∀ i , j ∈ I t \forall t \in[k], \forall i,j \in I_t ∀ t ∈ [ k ] , ∀ i , j ∈ I t .
Proof. Given t ∈ [ k ] t \in[k] t ∈ [ k ] , we build a bipartite graph from I t I_t I t to itself, such that two indexes i , j i,j i , j are connected if S i , j > 0 S_{i,j} > 0 S i , j > 0 . Note that this graph is connected, as otherwise, S t S_t S t could be composed into its connected components, and thus S t S_t S t would be composable. Given a path P = ( i 1 , … , i l ) P = (i_1,\ldots,i_l) P = ( i 1 , … , i l ) in the bipartite support graph, note that for every edge in the path connecting a row vertex i i i to a column vertex j j j , we have S i , j > 0 S_{i,j} > 0 S i , j > 0 . Since the metric vanishes, this implies that X i = Y j X_i = Y_j X i = Y j . Propagating these equalities along the path, we obtain X i = Y j X_i = Y_j X i = Y j for all i , j ∈ I k i,j \in I_k i , j ∈ I k , and we are done. □ \square □
Theorem D.5. Let G , H \mathcal{G},\mathcal{H} G , H be two featured graphs. Then
d ( G , H ) = 0 ⟺ G ∼ W L H d(\mathcal{G},\mathcal{H}) = 0 \Longleftrightarrow\mathcal{G} \mathrel{\overset{\mathrm{WL}}{\sim}} \mathcal{H} d ( G , H ) = 0 ⟺ G ∼ WL H
We first prove that if the two graphs are 1-WL equivalent, then this metric vanishes. We may assume they have the same stable partition as we proved above. Take P = P 1 ∪ … ∪ P t \mathcal{P} = \mathcal{P}_1 \cup\ldots\cup\mathcal{P}_t P = P 1 ∪ … ∪ P t , rename the nodes such that they come in consecutive order and denote by n 1 , n 2 , … , n k n_1,n_2,\ldots,n_k n 1 , n 2 , … , n k the sizes of the partitions. Note that by definition, ∀ u , v ∈ P i \forall u,v \in\mathcal{P}_i ∀ u , v ∈ P i , both have the same feature and number of neighbors in all P l \mathcal{P}_l P l . As in the book [SU13], define
S : = 1 n ( 1 n 1 J n 1 ⊕ ⋯ ⊕ 1 n k J n k ) . S := \frac{1}{n}\left(\frac{1}{n_1}J_{n_1} \oplus\cdots\oplus\frac{1}{n_k}J_{n_k}\right). S := n 1 ( n 1 1 J n 1 ⊕ ⋯ ⊕ n k 1 J n k ) .
and from the book [SU13], we know that A S = S B AS = SB A S = S B . As all nodes in the same partition have the same feature, X i = Y j X_i = Y_j X i = Y j , ∀ i , j ∈ I k = [ n t , n t + 1 ] \forall i,j \in I_k = [n_t,n_{t+1}] ∀ i , j ∈ I k = [ n t , n t + 1 ] , ∀ t ∈ [ k ] \forall t \in[k] ∀ t ∈ [ k ] , so as S S S is non-zero only on indexes in the same partition then also this metric vanishes, so also the sum vanishes on S S S .
For the second direction, let G = ( A , X ) , H = ( B , Y ) \mathcal{G} = (A,X), \mathcal{H} = (B,Y) G = ( A , X ) , H = ( B , Y ) be two graphs and assume there exists a matrix of the form S = P ( S 1 ⊕ ⋯ ⊕ S t ) Q S = P(S^1 \oplus\cdots\oplus S_t)Q S = P ( S 1 ⊕ ⋯ ⊕ S t ) Q such that the metric vanishes. Denote by D = S 1 ⊕ ⋯ ⊕ S t D = S^1 \oplus\cdots\oplus S_t D = S 1 ⊕ ⋯ ⊕ S t . We show that, up to relabeling the nodes of the two graphs, we can choose S S S to be a block diagonal matrix. Since the metric vanishes, we have A S = S B AS = SB A S = S B . Therefore,
0 = ∥ A P D Q − P D Q B ∥ 1 = ∥ P ⋅ ( P − 1 A P ⋅ D − D ⋅ Q B Q − 1 ) ⋅ Q ∥ 1 = ( ∗ ) ∥ P − 1 A P ⋅ D − D ⋅ Q B Q − 1 ∥ 1 , 0 = \lVert APDQ - PDQB \rVert_1
= \lVert P \cdot(P^{-1}AP \cdot D - D \cdot QBQ^{-1}) \cdot Q \rVert_1
\overset{(*)}{=} \lVert P^{-1}AP \cdot D - D \cdot QBQ^{-1} \rVert_1, 0 = ∥ A P D Q − P D QB ∥ 1 = ∥ P ⋅ ( P − 1 A P ⋅ D − D ⋅ QB Q − 1 ) ⋅ Q ∥ 1 = ( ∗ ) ∥ P − 1 A P ⋅ D − D ⋅ QB Q − 1 ∥ 1 ,
where ( ∗ ) (*) ( ∗ ) follows from the fact that permutation matrices preserve the norm.
Similarly, for the feature term, by reindexing the sum according to the permutations defined by P P P and Q Q Q , we obtain
0 = ∑ i , j ( P D Q ) i , j ⋅ ∥ X i − Y j ∥ 1 = ∑ i , j D i , j ⋅ ∥ ( P − 1 X ) i − ( Q Y ) j ∥ 1 . 0 = \sum_{i,j}(PDQ)_{i,j} \cdot\lVert X_i - Y_j \rVert_1
= \sum_{i,j}D_{i,j} \cdot\lVert(P^{-1}X)_i - (QY)_j \rVert_1. 0 = i , j ∑ ( P D Q ) i , j ⋅ ∥ X i − Y j ∥ 1 = i , j ∑ D i , j ⋅ ∥( P − 1 X ) i − ( Q Y ) j ∥ 1 .
We define two graphs
G ^ = ( P − 1 A P , P − 1 X ) ≅ G , H ^ = ( Q B Q − 1 , Q Y ) ≅ H . \widehat{G} = (P^{-1}AP, P^{-1}X) \cong G,\qquad
\widehat{H} = (QBQ^{-1}, QY) \cong H. G = ( P − 1 A P , P − 1 X ) ≅ G , H = ( QB Q − 1 , Q Y ) ≅ H .
Thus, for G ^ \widehat{G} G and H ^ \widehat{H} H , the metric vanishes using the block diagonal matrix D D D . Hence, up to isomorphism, we can assume that S = D S = D S = D is a block diagonal matrix.
Note that D D D defines a partition P 1 , … , P s \mathcal{P}_1,\ldots,\mathcal{P}_s P 1 , … , P s of the nodes, and we will prove that it is a stable partition of both graphs. From
( P − 1 A P ) D = D ( Q B Q − 1 ) , (P^{-1}AP)D = D(QBQ^{-1}), ( P − 1 A P ) D = D ( QB Q − 1 ) ,
we obtain, as in [SU13], that every two nodes u , v ∈ P k u,v \in\mathcal{P}_k u , v ∈ P k have the same number of neighbors in P l \mathcal{P}_l P l , for every l ∈ [ s ] l \in[s] l ∈ [ s ] .
Next, by Lemma D.4 , we know that X ^ i = Y ^ j \widehat{X}_i = \widehat{Y}_j X i = Y j for all i , j ∈ P k i,j \in\mathcal{P}_k i , j ∈ P k , so nodes in the same partition have the same feature. Thus, this partition is stable.
Therefore, G ^ \widehat{G} G and H ^ \widehat{H} H have exactly the same stable partition. Consequently, G G G and H H H have the same stable partition up to isomorphism, and by Lemma D.3 , the two graphs are 1-WL equivalent.
FSW-GNN: Equivalence to WL
We now prove Theorem 6.3.2 that says the FSW-GNN is equivalent to WL.
Theorem 6.3.2. Consider the FSW-GNN architecture for input graphs in G ≤ N ( R d ) \mathcal{G}_{\leq N}(\mathbb{R}^d) G ≤ N ( R d ) , with T = N T = N T = N iterations, where Φ ( t ) , Ψ \Phi^{(t)},\Psi Φ ( t ) , Ψ are just linear functions, and all features (except for input features) are of dimension m ≥ 2 N d + 2 m \geq2Nd + 2 m ≥ 2 N d + 2 . Then for almost every choice of model parameters, the graph embedding defined by the architecture is WL equivalent.
Proof. This proof employs the finite witness theorem [2 ] , and the basic methodology to use it for graph neural network injectivity, first presented by Hordan, Amir, Gortler, and Dym [HAGD24].
It is sufficient to show that the functions Φ ( t ) \Phi^{(t)} Φ ( t ) and Ψ \Psi Ψ are injective vector-to-vector functions, and that the FSW embedding maps used in the FSW-GNN are injective multiset to vector functions. The injectivity of all these functions ensures that the MPNN is WL-equivalent [XHLJ18].
By Theorem 4.1 in [AD24], if we take the FSW embedding with an output dimension of 2 N d + 1 2Nd+1 2 N d + 1 , defined on the space of multisets with at most N N N features in R d \mathbb{R}^d R d , then the embedding will be injective for almost every choice of its parameters. This injectivity is in the sense that two multisets will attain the same value if and only if they define the same distribution. This can happen when the multisets are not equal, but have different cardinalities. However, the last coordinate of the FSW embedding, which encodes the multiset cardinality, ensures that this will not be an issue. Accordingly the FSW embedding is injective on the multiset space.
We note that the output of the FSW embedding will now be node features of a higher dimension of 2 N d + 2 2Nd+2 2 N d + 2 , rather than just d d d . However, the set of all possible features obtained from the previous iteration will be a semialgebraic set of dimension ≤ N d \leq Nd ≤ N d . As long as this is the case, reapplying the FSW embedding with output dimension of 2 N d + 2 2Nd+2 2 N d + 2 will still be injective, for almost every choice of parameters.
The same argument can also be applied to the linear functions Φ ( t ) \Phi^{(t)} Φ ( t ) and Ψ \Psi Ψ used in FSW-GNN. Indeed, it is an easy consequence of the finite witness theorem [2 ] that, for a semialgebraic set of dimension at most N d Nd N d , almost every linear map into R m \mathbb{R}^m R m is injective whenever m > 2 N d m>2Nd m > 2 N d . Since m ≥ 2 N d + 2 m\geq2Nd+2 m ≥ 2 N d + 2 , the functions Φ ( t ) \Phi^{(t)} Φ ( t ) and Ψ \Psi Ψ are injective on their respective sets of attainable inputs for almost every choice of parameters. This concludes the proof.
FSW-GNN: Bi-Lipschitzness
We first prove Theorem 6.3.4.
Lemma D.6. Let f , g : M → R ≥ 0 f,g:M\to\mathbb{R}_{\geq0} f , g : M → R ≥ 0 be nonnegative piecewise-linear functions defined on a compact polygon M ⊂ R d M\subset\mathbb{R}^d M ⊂ R d . Suppose that for all x ∈ M x\in M x ∈ M , f ( x ) = 0 f(x)=0 f ( x ) = 0 if and only if g ( x ) = 0 g(x)=0 g ( x ) = 0 . Then there exist real constants c , C > 0 c,C>0 c , C > 0 such that
c ⋅ g ( x ) ≤ f ( x ) ≤ C ⋅ g ( x ) , ∀ x ∈ M . (79) c\cdot g(x)\leq f(x)\leq C\cdot g(x),\quad\forall x\in M. \tag*{(79)} c ⋅ g ( x ) ≤ f ( x ) ≤ C ⋅ g ( x ) , ∀ x ∈ M . (79)
Proof. By definition, there is a partition of M M M into a finite union of compact polygons, so that the restriction of f f f to each polygon is an affine function. g g g also has such a partition, and by taking the mutual refinement of the two partitions, we obtain a finite partition of M M M into compact polygons, such that the restriction of both f f f and g g g to each polygon is affine.
Accordingly, it is sufficient to prove the claim on a single compact polygon $P\subseteq M on which the restriction of both f f f and g g g is affine. A compact polygon has a finite number of extreme points v 1 , … , v k v_1,\ldots,v_k v 1 , … , v k , and every point in the polygon can be written as a convex combination of these points. Now let us denote
I = { i ∈ [ k ] ∣ f ( v i ) > 0 } = { i ∈ [ k ] ∣ g ( v i ) > 0 } . I=\{i\in[k]\mid f(v_i)>0\}=\{i\in[k]\mid g(v_i)>0\}. I = { i ∈ [ k ] ∣ f ( v i ) > 0 } = { i ∈ [ k ] ∣ g ( v i ) > 0 } .
We note that if I I I is empty, then f f f and g g g are identically zero on P P P , and there is nothing to prove. Otherwise, the following quantities are well defined and strictly positive:
m f = min i ∈ I f ( v i ) , M f = max i ∈ I f ( v i ) m_f=\min_{i\in I}f(v_i),\qquad M_f=\max_{i\in I}f(v_i) m f = i ∈ I min f ( v i ) , M f = i ∈ I max f ( v i )
m g = min i ∈ I g ( v i ) , M g = max i ∈ I g ( v i ) m_g=\min_{i\in I}g(v_i),\qquad M_g=\max_{i\in I}g(v_i) m g = i ∈ I min g ( v i ) , M g = i ∈ I max g ( v i )
Then for every x ∈ P x\in P x ∈ P , there exist non-negative θ 1 , … , θ k \theta_1,\ldots,\theta_k θ 1 , … , θ k which sum to 1 such that
x = ∑ i = 1 k θ i v i . x=\sum_{i=1}^{k}\theta_i v_i. x = i = 1 ∑ k θ i v i .
It follows that
f ( x ) = f ( ∑ i = 1 k θ i v i ) = ∑ i = 1 k θ i f ( v i ) = ∑ i ∈ I θ i f ( v i ) ≥ m f ∑ i ∈ I θ i = m f M g ⋅ M g ∑ i ∈ I θ i ≥ m f M g ⋅ ∑ i ∈ I θ i g ( v i ) = m f M g ⋅ g ( ∑ i = 1 k θ i v i ) = m f M g ⋅ g ( x ) . \begin{aligned}
f(x)&=f\left(\sum_{i=1}^{k}\theta_i v_i\right)=\sum_{i=1}^{k}\theta_i f(v_i)=\sum_{i\in I}\theta_i f(v_i)\\
&\ge m_f\sum_{i\in I}\theta_i=\frac{m_f}{M_g}\cdot M_g\sum_{i\in I}\theta_i\ge\frac{m_f}{M_g}\cdot\sum_{i\in I}\theta_i g(v_i)\\
&=\frac{m_f}{M_g}\cdot g\left(\sum_{i=1}^{k}\theta_i v_i\right)=\frac{m_f}{M_g}\cdot g(x).
\end{aligned} f ( x ) = f ( i = 1 ∑ k θ i v i ) = i = 1 ∑ k θ i f ( v i ) = i ∈ I ∑ θ i f ( v i ) ≥ m f i ∈ I ∑ θ i = M g m f ⋅ M g i ∈ I ∑ θ i ≥ M g m f ⋅ i ∈ I ∑ θ i g ( v i ) = M g m f ⋅ g ( i = 1 ∑ k θ i v i ) = M g m f ⋅ g ( x ) .
By reversing the roles of f f f and g g g , we obtain that for every x ∈ P x\in P x ∈ P ,
g ( x ) ≥ m g M f f ( x ) . g(x)\ge\frac{m_g}{M_f}f(x). g ( x ) ≥ M f m g f ( x ) .
This concludes the proof.
We now turn to the full proof of Theorem 6.3.3.
Theorem 6.3.3. Let Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d be compact. Under the assumptions of Theorem 6.3.2, the FSW-GNN is bi-Lipschitz with respect to ρ D S \rho_{\mathrm{DS}} ρ DS on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) . If, additionally, Ω \Omega Ω does not contain 0 \mathbf{0} 0 , then the FSW-GNN is bi-Lipschitz with respect to T M D \mathrm{TMD} TMD on G ≤ N ( Ω ) \mathcal{G}_{\leq N}(\Omega) G ≤ N ( Ω ) .
Proof. Following the reasoning in the main text, the rest of the proof is as follows: Let G , G ~ ∈ G ≤ N ( Ω ) G,\widetilde{G}\in\mathcal{G}_{\leq N}(\Omega) G , G ∈ G ≤ N ( Ω ) , G = ( V , E , X ) G=(V,E,X) G = ( V , E , X ) and G ~ = ( V ~ , E ~ , X ~ ) \widetilde{G}=(\widetilde{V},\widetilde{E},\widetilde{X}) G = ( V , E , X ) , with n n n and n ~ \widetilde{n} n vertices respectively. Since there is a finite number of choices of possible n , n ~ ≤ N n,\widetilde{n}\leq N n , n ≤ N and edges E , E ~ E,\widetilde{E} E , E . It is enough to show that once these parameters are chosen, the bi-Lipschitz ratio is bounded on all choices of X ∈ Ω n X\in\Omega^n X ∈ Ω n , X ~ ∈ Ω n ~ \widetilde{X}\in\Omega^{\widetilde{n}} X ∈ Ω n . Define the function f ( X , X ~ ) = ∥ h G − h G ~ ∥ 1 f(X,\widetilde{X})=\lVert h_G-h_{\widetilde{G}}\rVert_1 f ( X , X ) = ∥ h G − h G ∥ 1 , g ( X , X ~ ) = ρ ( G , G ~ ) g(X,\widetilde{X})=\rho(G,\widetilde{G}) g ( X , X ) = ρ ( G , G ) , where ρ \rho ρ is either ρ D S \rho_{\mathrm{DS}} ρ DS or T M D \mathrm{TMD} TMD . By the comment above, both f f f and g g g are piecewise linear. Since both FSW-GNN and the metric ρ \rho ρ are WL-equivalent,
f f f and g g g have the same zero set. Lastly, if ρ = ρ D S \rho= \rho_{\mathrm{DS}} ρ = ρ DS , since Ω \Omega Ω is compact, it is contained in a compact polygon. If ρ = T M D \rho= \mathrm{TMD} ρ = TMD , then additionally 0 ∉ Ω 0 \notin\Omega 0 ∈ / Ω , and therefore the distance from Ω \Omega Ω to 0 0 0 is strictly positive. Hence, Ω \Omega Ω is contained in a finite union of compact polygons that don’t contain 0 0 0 (e.g., by taking the set difference between a compact box containing Ω \Omega Ω and a sufficiently small ℓ ∞ \ell_\infty ℓ ∞ ball around 0 0 0 ). Thus, Lemma 3.4 guarantees the existence of Lipschitz constants c , C c,C c , C . Thus, Theorem 6.3.4 guarantees the existence of Lipschitz constants c , C c,C c , C . Finally, since there are only finitely many choices of graph sizes, adjacency matrices, vertices v , v ~ v,\tilde{v} v , v ~ , and iterations t ≤ N t \le N t ≤ N , we can take the minimum of the corresponding lower Lipschitz constants and the maximum of the upper Lipschitz constants. Thus, the constants c , C c,C c , C can be chosen uniformly over G ≤ N ( Ω ) \mathcal{G}_{\le N}(\Omega) G ≤ N ( Ω ) .
Finally, note that by equivalence of all norms on a finite dimensional space, the same bi-Lipschitz equivalence holds if we replace the one-norm in the definition of f f f with any other norm.
Theorem D.7. Suppose that Ω ⊂ R d \Omega\subset\mathbb{R}^{d} Ω ⊂ R d is a compact set that does not contain zero. Under the assumptions of Theorem 6.3.2, the node features computed by FSW-GNN are bi-Lipschitz with respect to the Tree Distance metric on G ≤ N ( Ω ) \mathcal{G}_{\le N}(\Omega) G ≤ N ( Ω ) .
Proof. Let G = ( V , A , X ) G=(V,A,X) G = ( V , A , X ) , G ~ = ( V ~ , A ~ , X ~ ) \widetilde{G}=(\widetilde{V},\widetilde{A},\widetilde{X}) G = ( V , A , X ) be two graphs in G ≤ N ( Ω ) \mathcal{G}_{\le N}(\Omega) G ≤ N ( Ω ) . Let t ∈ N t \in\mathbb{N} t ∈ N denote the number of MPNN/WL iterations applied. Let v ∈ V v \in V v ∈ V , v ~ ∈ V ~ \widetilde{v} \in\widetilde{V} v ∈ V . Let h v ( t ) h_{v}^{(t)} h v ( t ) and h v ~ ( t ) h_{\widetilde{v}}^{(t)} h v ( t ) be the output features of vertices v v v and v ~ \widetilde{v} v computed by FSW-GNN. Then TD ( T v ( t ) , T v ~ ( t ) ) = 0 \operatorname{TD}(T_{v}^{(t)},T_{\widetilde{v}}^{(t)})=0 TD ( T v ( t ) , T v ( t ) ) = 0 if and only if v v v and v ~ \widetilde{v} v are assigned the same features by the WL test, which in turn takes place if and only if h v ( t ) = h v ~ ( t ) h_{v}^{(t)}=h_{\widetilde{v}}^{(t)} h v ( t ) = h v ( t ) . Note that ∥ h v − h v ~ ∥ 1 \lVert h_{v}-h_{\widetilde{v}}\rVert_{1} ∥ h v − h v ∥ 1 is a piecewise-linear function of the input vertex features { x v } v ∈ V \{x_{v}\}_{v\in V} { x v } v ∈ V and { x ~ v ~ } v ~ ∈ V ~ \{\widetilde{x}_{\widetilde{v}}\}_{\widetilde{v}\in\widetilde{V}} { x v } v ∈ V . Moreover, TD ( T v ( t ) , T v ~ ( t ) ) \operatorname{TD}(T_{v}^{(t)},T_{\widetilde{v}}^{(t)}) TD ( T v ( t ) , T v ( t ) ) is equivalent, up to bounded distortion, to a piecewise-linear function of the input vertex features { x u } u ∈ V \{x_{u}\}_{u\in V} { x u } u ∈ V and { x ~ u ~ } u ~ ∈ V ~ \{\widetilde{x}_{\widetilde{u}}\}_{\widetilde{u}\in\widetilde{V}} { x u } u ∈ V . Lastly, similarly to the proof of Theorem 6.3.3, Ω \Omega Ω can be extended to a compact polygon that does not contain 0 \mathbf{0} 0 . Thus, by Theorem 6.3.4, there exist constants 0 < c , C < ∞ 0<c,C<\infty 0 < c , C < ∞ such that
c ⋅ TD ( T v ( t ) , T v ~ ( t ) ) ≤ ∥ h v ( t ) − h v ~ ( t ) ∥ 1 ≤ C ⋅ TD ( T v ( t ) , T v ~ ( t ) ) . c\cdot\operatorname{TD}(T_{v}^{(t)},T_{\widetilde{v}}^{(t)})\le\lVert h_{v}^{(t)}-h_{\widetilde{v}}^{(t)}\rVert_{1}\le C\cdot\operatorname{TD}(T_{v}^{(t)},T_{\widetilde{v}}^{(t)}). c ⋅ TD ( T v ( t ) , T v ( t ) ) ≤ ∥ h v ( t ) − h v ( t ) ∥ 1 ≤ C ⋅ TD ( T v ( t ) , T v ( t ) ) .
E Experiment Details
We used the Adam optimizer for all experiments.
For the empirical distortion evaluation, we used pairs of graphs G , G ~ G,\widetilde{G} G , G , each consisting of four vertices and the edges 1 − 2 − 3 − 4 − 1 1-2-3-4-1 1 − 2 − 3 − 4 − 1 . Two random vectors v 0 , Δ v ∈ R d v_{0},\Delta v\in\mathbb{R}^{d} v 0 , Δ v ∈ R d were drawn i.i.d. Gaussian and normalized to unit length. In G G G all vertex features were set to v 0 v_{0} v 0 , whereas in G ~ \widetilde{G} G they were set to v 0 + ϵ σ Δ v v_{0}+\epsilon\sigma\Delta v v 0 + ϵ σ Δ v , with σ = 1 \sigma=1 σ = 1 for v 2 , v 4 v_{2},v_{4} v 2 , v 4 and − 1 -1 − 1 for v 1 , v 3 v_{1},v_{3} v 1 , v 3 . We used ϵ = { 1 , 10 − 1 , 10 − 2 , … , 10 − 6 } \epsilon=\{1,10^{-1},10^{-2},\ldots,10^{-6}\} ϵ = { 1 , 1 0 − 1 , 1 0 − 2 , … , 1 0 − 6 } , and generated 100 pairs for each value of ϵ \epsilon ϵ , and evaluated the constants C C C , c c c for the resulting pairs. This experiment was repeated 10 times and the average distortion was taken. To ensure accurate results, we used 64-bit floating-point arithmetic in this experiment.
For the NeighborsMatch problem from [AY21], we used the protocol developed in their paper: we used their implementation for the MPNNs we compared to, with a hidden dimension of 64 for all models, searched for the best hyperparameters for each model, and reported the training accuracy. For fair comparison with rival models, we repeated each sample 100 times, as was done in [AY21].
For the Ring dataset, we used the results from [DGB+23] and trained our models with a hidden dimension of 64.
For the LRGB dataset, we trained all models under the constraint of 500K parameters. In contrast, for the MolHIV dataset, there was no restriction, and we trained the models with 40K parameters.
For the transductive learning tasks, we used a hidden dimension of 128 across all models.
Timing Here, we show the testing time for FSW-GNN, GIN, GCN, GGNN, GAT, and GraphSAGE on the MUTAG graph classification task.
Ordered subgraph aggregation
In this section, we recall sub-graph aggregation networks and show that FSW-GNN could be used to create a bi-Lipschitz embedding with respect to the metric equivalent to sub-graph WL.
Ordered subgraph 1-WL
The OSWL (Ordered sub-graph WL) is taken from [QRG+22; BFL+21]. Given a 1-WL procedure using T T T iterations denoted by W L T WL^T W L T , and deterministic graph sampling Π \Pi Π (could be node/edge deletion), we define the following three steps:
Step 1: Deterministic graph sampling (e.g. node/edge deletion) Π ( G ) = { G 1 , … , G m } \Pi(\mathcal{G}) = \{\mathcal{G}_1, \ldots, \mathcal{G}_m\} Π ( G ) = { G 1 , … , G m }
y G = ϕ ( { y 1 , … , y m } ) y_{\mathcal{G}} = \phi(\{y_1,\ldots,y_m\}) y G = ϕ ({ y 1 , … , y m })
Where W L T WL^T W L T denotes the application of T T T iterations of the WL test, and ϕ \phi ϕ is an injective multiset function.
Definition F.1. We say G 1 , G 2 \mathcal{G}_1,\mathcal{G}_2 G 1 , G 2 are equivalent up to OSWL, if O S W L T ( G 1 ) = O S W L T ( G 2 ) , ∀ T ∈ N OSWL^T(\mathcal{G}_1) = OSWL^T(\mathcal{G}_2), \forall T \in\mathbb{N} O S W L T ( G 1 ) = O S W L T ( G 2 ) , ∀ T ∈ N , and we denote this by G 1 ≅ G 2 \mathcal{G}_1 \cong\mathcal{G}_2 G 1 ≅ G 2 . As for WL, checking equality for T = ∣ G 1 ∣ T = |\mathcal{G}_1| T = ∣ G 1 ∣ is sufficient to ensure equality for all T T T .
Ordered subgraph MPNN Ordered subgraph MPNNs are very similar to OSWL. Given an MPNN and a multi-set to vector embedding ϕ \phi ϕ , we define its sub-graph version:
Step 1: Deterministic graph sampling Π ( G ) = { G 1 , … , G m } \Pi(\mathcal{G}) = \{\mathcal{G}_1,\ldots,\mathcal{G}_m\} Π ( G ) = { G 1 , … , G m }
y i = M P N N ( G i ) , G i ∈ Π ( G ) y_i = MPNN(\mathcal{G}_i), \mathcal{G}_i \in\Pi(\mathcal{G}) y i = M P N N ( G i ) , G i ∈ Π ( G )
y G = ϕ ( { y 1 , … , y m } ) y_{\mathcal{G}} = \phi(\{y_1,\ldots,y_m\}) y G = ϕ ({ y 1 , … , y m })
OSWL equivalent metric We define a metric OSWL equivalent similarly to 6.2.1:
Definition (WL metric). An OSWL metric on G = N ( Ω ) \mathcal{G}_{=N}(\Omega) G = N ( Ω ) is a function ρ : G = N ( Ω ) × G = N ( Ω ) → R ≥ 0 \rho: \mathcal{G}_{=N}(\Omega) \times\mathcal{G}_{=N}(\Omega) \to\mathbb{R}_{\ge0} ρ : G = N ( Ω ) × G = N ( Ω ) → R ≥ 0 that satisfies the following conditions for all G 1 , G 2 , G 3 ∈ G = N ( Ω ) \mathcal{G}_1,\mathcal{G}_2,\mathcal{G}_3 \in\mathcal{G}_{=N}(\Omega) G 1 , G 2 , G 3 ∈ G = N ( Ω ) :
ρ ( G 1 , G 2 ) = ρ ( G 2 , G 1 ) Symmetry ρ ( G 1 , G 3 ) ≤ ρ ( G 1 , G 2 ) + ρ ( G 2 , G 3 ) Triangle inequality ρ ( G 1 , G 2 ) = 0 ⟺ G 1 ≅ G 2 . OSWL equivalence (80) \begin{aligned}
\rho(\mathcal{G}_1,\mathcal{G}_2) &= \rho(\mathcal{G}_2,\mathcal{G}_1) && \text{Symmetry}\\
\rho(\mathcal{G}_1,\mathcal{G}_3) &\le\rho(\mathcal{G}_1,\mathcal{G}_2) + \rho(\mathcal{G}_2,\mathcal{G}_3) && \text{Triangle inequality}\\
\rho(\mathcal{G}_1,\mathcal{G}_2) &= 0 \Longleftrightarrow\mathcal{G}_1 \cong\mathcal{G}_2. && \text{OSWL equivalence} \tag*{(80)}
\end{aligned} ρ ( G 1 , G 2 ) ρ ( G 1 , G 3 ) ρ ( G 1 , G 2 ) = ρ ( G 2 , G 1 ) ≤ ρ ( G 1 , G 2 ) + ρ ( G 2 , G 3 ) = 0 ⟺ G 1 ≅ G 2 . Symmetry Triangle inequality OSWL equivalence (80)
Theorem F.2. Let a 1-WL equivalent metric d W L d_{WL} d W L , and sampling procedure, the following metric is OSWL equivalent:
d s u b ( G , H ) = min π ∈ S m ∑ i = 1 m d W L ( G i , H π ( i ) ) d_{sub}(\mathcal{G},\mathcal{H}) = \min_{\pi\in S_m} \sum_{i=1}^{m} d_{WL}(\mathcal{G}_i,\mathcal{H}_{\pi(i)}) d s u b ( G , H ) = π ∈ S m min i = 1 ∑ m d W L ( G i , H π ( i ) ) Proof. We prove only the last part, which is not trivial. Assume first that the number of nodes is the same, then:
d s u b ( G , H ) = 0 ⟺ ∃ π ∈ S m : ∀ i ∈ [ m ] : d W L ( G i , H π ( i ) ) = 0 ⟺ ⟺ ∃ π ∈ S m : ∀ i ∈ [ m ] , y i G = y π ( i ) H ϕ ( { y 1 G , … , y m G } ) = ϕ ( { y 1 H , … , y m H } ) ⟺ O S W L ( G ) = O S W L ( H ) \begin{aligned}
d_{sub}(\mathcal{G},\mathcal{H}) = 0 &\Longleftrightarrow\exists\pi\in S_m : \forall i \in[m] : d_{WL}(\mathcal{G}_i,\mathcal{H}_{\pi(i)}) = 0 \Longleftrightarrow\\
&\Longleftrightarrow\exists\pi\in S_m : \forall i \in[m], y_i^{\mathcal{G}} = y_{\pi(i)}^{\mathcal{H}}\\
\phi(\{y_1^{\mathcal{G}},\ldots,y_m^{\mathcal{G}}\}) = \phi(\{y_1^{\mathcal{H}},\ldots,y_m^{\mathcal{H}}\}) &\Longleftrightarrow OSWL(\mathcal{G}) = OSWL(\mathcal{H})
\end{aligned} d s u b ( G , H ) = 0 ϕ ({ y 1 G , … , y m G }) = ϕ ({ y 1 H , … , y m H }) ⟺ ∃ π ∈ S m : ∀ i ∈ [ m ] : d W L ( G i , H π ( i ) ) = 0 ⟺ ⟺ ∃ π ∈ S m : ∀ i ∈ [ m ] , y i G = y π ( i ) H ⟺ O S W L ( G ) = O S W L ( H )